Markov process
# Markov Process
Definition: Markov process A Markov process is a stochastic process where the probability of an event only depends on the last event $$\mu(X(t)\in B| X(t_m) = x_m, …, X(t_1) = x_1) = \mu(X(t)\in B|X(t_m)=x_m)$$ This condition is called the Markov condition
like for normal probability densities, the joint probability density $p_m$ is defined by the integral $$P(B_m,t_m;…;B_1,t_1)=\int_{B_m}\text{d}x_m \cdots \int_{B_1}\text{d}x_1 p_m(x_m,t_m;…;x_1,t_1)$$ with analogue definitions for the conditional probabilities $$p_{l|k}(x_{k+l},t_{k+l};…;x_{k+1},t_{k+1}|x_k,t_k;…;x_1,t_1)=\frac{p_{k+l}(x_{k+l},t_{k+l};…;x_1,t_1)}{p_k(x_k,t_k;…;x_1,t_1)}$$ We denote with $p_{l|k}$ the conditional probability of $l$ events $x_i$ happening at times $t_i$ given that $k$ events at times $t_{1}-t_{k}$ happend.
The markov condition for the densities looks like $$p_{1|m}=p_{1|1}.$$ The probability $p_{1|1}(x,t|x^{’},t^{’})$ is the conditional transition probability from value $x^’$ to $x$ also called propagator.
Definition: Propagator We define the propagator $T(x,t|x^{’},t^{’}) = p_{1|1}(x,t|x^{’},t^{’})$ It gives the conditional probability to evolve to $x$ at $t$ given that we are currently in the state $x^’$ at $t^'$
Also $p(x,t) = p_1(x,t)$, which is just the probability of a $x$ appearing at some time $t$.
The propagator fulfills $$\int \text{d}x T(x,t|x^’,t^’)=1$$ $$\lim_{t\to t^’}T(x,t|x^’,t^’)=\delta(x-x^’)$$
We can ’evolve’ an density at time $t_0$ using the propagator $$p(x,t)=\int \text{d}x^’ T(x,t|x^’,t_0)p(x^’,t_0)$$
Definition: Homogenous stochastic process A stocahstic process in homogenous if $T$ only depends on $\delta t = t - t^’$. We can then write the propagator as $T_\tau(x|x^’)$
Definition: Stationary stochastic process A stochastic process is stationary if it is invariant under time translation under all time variables In this case $p(x,t) = p(x)$ and it is homogenous.
# Chapman-Kolmogorov equation
Chapman-Kolmogorov equation $$T(x_3,t_3|x_1,t_1) = \int\text{d}x_2 T(x_3,t_3|x_2,t_2)T(x_2,t_2|x_1,t_1)$$ Which can intuitivly interpreted as multiplying the probabilites of $1\to 2$ and $2\to 3$ and summing over all possible $2$s
For a homogenous Markov process the Chapman-Kolmogorov equation becomes $$T_{\tau+\tau^’}(x|x^’)=\int \text{d}x^{’’}T_\tau(x|x^{’’})T_{\tau^’}(x^{’’}|x^’)$$
The Chapman-Kolmogorov equation can be derived by starting from $p_3(x_3,t_3;x_2,t_2;x_1,t_1)$ and deriving $p_{1|1}(x_3,t_3|x_1,t_1)$ using the definition of the conditional probability. On an intermediate step $x_2$ is integrated out.
With a propagator and an initial density we can determin the whole hirarchy of joint probability distirbutions. So to define a Markov process we need to specify the propagator and the initial density $p(x,t_0)$
Differential Chapman-Kolmogorov equation By differentiating the Chapman-Kolmogorov equation we obtain the differential form $$\frac{\partial}{\partial t}T(x,t|x^’,t^’) = \mathcal{A}T(x,t|x^’,t^’)).$$ Here $\mathcal{A}$ is a linear operator, which acts on a density $p(x)$ called Generator $$\mathcal{A}p(x) = \lim_{\Delta t \to 0}\frac{1}{\Delta t}\left[\int \text{d}x^’ T(x,t + \Delta t|x^’,t)\rho(x^’) - \rho(x)\right]$$ The generator describes the infinitesimal time propagation
For a homogenous Markov process the propagator $T$ can be written in terms of the generator $\mathcal{A}$ as $$T_\tau(x|x^’) = \exp(\tau\mathcal{A})\delta(x-x^’),$$ which can be seen when inserting it into the differential form of the Chapman-Kolmogorov equation. Equally we get $$p(x,t) = \exp(\tau\mathcal{A})p(x,t_0)$$
The condition $\tau \geq 0$ makes the process irreversibel and therefore dictates the direction of time, since we can propagate backwards in time.
# Types of Markov processes
Different Markov processes can be distinguished by the form of their generator. Important ones are