Course information

This course provides an introduction to stochastic processes and their various applications in discrete and state space, covering: Markov Chains: Definition, higher-order transition probabilities, decomposition of state space into communication classes, strong Markov property, recurrence, and transience. Markov Chain Potential Theory: Absorption probabilities and statistics of arrival times. Invariant Distributions: Definition, existence/uniqueness theorems, different representations, the structure of the space of invariant distributions, time reversibility, and detailed balance. Asymptotic Behavior of Markov Chains: Periodicity, asymptotic distribution, and the ergodic theorem. Applications: PageRank algorithm, Markov Chain Monte Carlo (MCMC) computational method, and simulated annealing. Poisson Processes: Definition, properties, addition and thinning, and modeling with Poisson processes.