> For the complete documentation index, see [llms.txt](https://drdh.gitbook.io/rl/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://drdh.gitbook.io/rl/deep-rl-course/actor-critic-algorithms/discount-factors.md).

# Discount factors

## Infinite cases

By now, we only discussed episodic tasks, but what about continuous/cyclical tasks? What if $$T$$ is $$\infty$$? In many cases, $$\hat{V}^\pi\_\phi$$ can get infinitely large. A simple trick will solve this problem: better to get rewards sooner than later.

We have $$\gamma$$ chances to die every step:

![new MDP](https://4133958719-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LigLKy0c06y4iTEtrkI%2F-LlWOIcWKIYnXmMSdlaE%2F-LlWOMZfOSv-9-rMdD7m%2F1565000112728.png?generation=1565002136792841\&alt=media)

So the new target is:

$$
y\_{i,t}\approx r(s\_{i,t},a\_{i,t})+\gamma \hat{V}^\pi\_\phi(s\_{i,t+1})
$$

And discount factor $$\gamma \in \[0,1]$$. (0.99 works well)

$$
\begin{aligned}
\nabla\_\theta J(\theta)&\approx\frac{1}{N}\sum\_{i=1}^N\sum\_{t=1}^T\nabla\_\theta \log \pi\_\theta(a\_{i,t}|s\_{i,t})\left(r(s\_{i,t},a\_{i,t})+\gamma \hat{V}^\pi\_\phi(s\_{i,t+1})-\hat{V}^\pi\_\phi(s\_{i,t})    \right) \\
&\approx \frac{1}{N}\sum\_{i=1}^N\sum\_{t=1}^T\nabla\_\theta \log \pi\_\theta(a\_{i,t}|s\_{i,t}) \hat{A}^\pi\_\phi(s\_{i,t},a\_{i,t})
\end{aligned}
$$

## Discount factors for policy gradient

In Monte Carlo policy gradients, we have 2 options:

option 1:

$$
\nabla\_\theta J(\theta)\approx\frac{1}{N}\sum\_{i=1}^N\sum\_{t=1}^T\nabla\_\theta \log \pi\_\theta(a\_{i,t}|s\_{i,t})\left(\sum\_{t'=t}^T \gamma^{t'-t} r(s\_{i,t'},a\_{i,t'})\right)
$$

option 2:

$$
\nabla\_\theta J(\theta)\approx\frac{1}{N}\sum\_{i=1}^N\left(\sum\_{t=1}^T\nabla\_\theta \log \pi\_\theta(a\_{i,t}|s\_{i,t})\right)\left(\sum\_{t=1}^T \gamma^{t-1} r(s\_{i,t},a\_{i,t})\right)
$$

Consider causality:

$$
\begin{aligned}
\nabla\_\theta J(\theta)&\approx\frac{1}{N}\sum\_{i=1}^N\sum\_{t=1}^T\nabla\_\theta \log \pi\_\theta(a\_{i,t}|s\_{i,t})\left(\sum\_{t'=t}^T \gamma^{t'-1} r(s\_{i,t'},a\_{i,t'})\right)\\
&\approx \frac{1}{N}\sum\_{i=1}^N\sum\_{t=1}^T\gamma^{t-1}\nabla\_\theta \log \pi\_\theta(a\_{i,t}|s\_{i,t})\left(\sum\_{t'=t}^T \gamma^{t'-t} r(s\_{i,t'},a\_{i,t'})\right)\\
\end{aligned}
$$

option 1 only changes "reward to go" and only consider reward discount from current state. But option 2 consider the whole episode. So option 2 is the true situation when the robot have some chances $$\gamma$$ to die every step. But option 1 is what we choose.

Because the reason why we use discount factor is to solve infinity problems in continuous cases, but death model(option 2) only cares the early steps of the whole episode. We want to approximate to the average reward without discount. The future rewards is more uncertain, which needs to be removed gradually.

## Actor-critic algorithms (with discount)

batch version

> batch actor-critic algorithm:
>
> repeat until converge:
>
> \====1: sample $${s\_i,a\_i }$$ from $$\pi\_\theta(a|s)$$ (run it on the robot)
>
> \====2: fit $$\hat{V}^\pi\_\phi(s)$$ to sampled reward sums (use bootstrapped estimate target)
>
> \====3: evaluate $$\hat{A}^\pi(s\_i,a\_i)=r(s\_i,a\_i)+\gamma\hat{V}^\pi\_\phi(s\_i')-\hat{V}^\pi\_\phi(s\_i)$$
>
> \====4: $$\nabla\_\theta J(\theta)\approx \sum\_i \nabla\_\theta \log \pi\_\theta (a\_i|s\_i)\hat{A}^\pi(s\_i,a\_i)$$
>
> \====5: $$\theta \leftarrow \theta+ \alpha\nabla\_\theta J(\theta)$$

online version

> online actor-critic algorithm:
>
> repeat until converge:
>
> \====1: take action $$a\sim \pi\_\theta(a|s)$$, get $$(s,a,s',r)$$
>
> \====2: update $$\hat{V}^\pi\_\phi$$ using target $$r+\gamma\hat{V}^\pi\_\phi(s')$$
>
> \====3: evaluate $$\hat{A}^\pi(s,a)=r(s,a)+\gamma\hat{V}^\pi\_\phi(s')-\hat{V}^\pi\_\phi(s)$$
>
> \====4: $$\nabla\_\theta J(\theta)\approx \nabla\_\theta \log \pi\_\theta (a|s)\hat{A}^\pi(s,a)$$
>
> \====5: $$\theta \leftarrow \theta+ \alpha\nabla\_\theta J(\theta)$$