Markov

Updated: 2019-01-03

Markov Chain vs Markov Process

Markov decision processes are an extension of Markov chains; the difference is the addition of actions (allowing choice) and rewards (giving motivation). Conversely, if only one action exists for each state and all rewards are the same (e.g., zero), a Markov decision process reduces to a Markov chain.

Recurrent vs Transient

Recurrent states

State ii is said to be persistent or recurrent if the probability of returning back to ii, having started at ii is 1

P(Xn=i for some n>0X0=i)=1P(X_n = i \text{ for some }n > 0 | X_0 = i) = 1

If you start at a Recurrent State, then you will for sure return back to that state at some point in the future.

Transient states

State ii is said to be transient if it is not guaranteed to return back to ii, having started at ior probability of returning back to ii having started at ii is less than 1

P(Xn=i for some n>0X0=i)<1P(X_n = i \text{ for some }n > 0 | X_0 = i) < 1

There is some positive probability that once you leave you will never return.