> For the complete documentation index, see [llms.txt](https://drdh.gitbook.io/rl/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://drdh.gitbook.io/rl/deep-rl-course/intro-to-rl/mdp-definition.md).

# MDP Definition

## Terminology and Notation

Let's begin with Markov Decision Process (MDP).

![MDP Notation](https://4133958719-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LigLKy0c06y4iTEtrkI%2F-LiqraEgDaO9pJSVs2A2%2F-LiqrbUC3RrlCQokBHQf%2F1562140002878.png?generation=1562141162847811\&alt=media)

We choose action $$a\_t$$ at time $$t$$ when we saw observation $$o\_t$$, and the latent state $$s\_t$$ transfered to $$s\_{t+1}$$, and we got reward $$r(s,t)$$ from the environment.

## Definitions

### Fully Observed

Markov decision process $$\mathcal{M}={\mathcal{S,A,T,r}}$$

* $$\mathcal{S}$$ : state space; states $$s\in\mathcal{S}$$ (discrete or continuous)
* $$\mathcal{A}$$ : action space; actions $$a\in\mathcal{A}$$ (discrete or continuous)
* $$\mathcal{T}$$ : transition operator, a tensor
* $$r$$ : reward function; $$r(s\_t,a\_t):\mathcal{S}\times\mathcal{A}\to \mathbb{R}$$

### Partially Observed

partially observed Markov decision process $$\mathcal{M}={\mathcal{S,A,O,T,E, r}}$$

* $$\mathcal{S}$$ : state space; states $$s\in\mathcal{S}$$ (discrete or continuous)
* $$\mathcal{A}$$ : action space; actions $$a\in\mathcal{A}$$ (discrete or continuous)
* $$\mathcal{O}$$ : observation space; observations $$o\in\mathcal{O}$$ (discrete or continuous)
* $$\mathcal{T}$$ : transition operator, a tensor
* $$\mathcal{E}$$ : emission probability $$p(o\_t|s\_t)$$
* $$r$$ : reward function; $$r(s\_t,a\_t):\mathcal{S}\times\mathcal{A}\to \mathbb{R}$$