> For the complete documentation index, see [llms.txt](https://drdh.gitbook.io/rl/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://drdh.gitbook.io/rl/deep-rl-course/actor-critic-algorithms/other-advantages.md).

# Other advantages

## Eligibility traces & n-step returns

$$\hat{V}^\pi\_\phi$$ will bring bias and reward-sum will bring variance.

**Critic**

$$
\hat{A}^\pi\_{C}=r(s\_{t},a\_{t})+\gamma \hat{V}^\pi\_\phi(s\_{t+1})-\hat{V}^\pi\_\phi(s\_{t})
$$

**+**: lower variance

**-**: higher bias if value is wrong (it always is)

**Monte Carlo**

$$
\hat{A}^\pi\_{MC}=\sum\_{t'=t}^\infty\gamma^{t'-t}r(s\_{t'},a\_{t'})-\hat{V}^\pi\_\phi(s\_{t})
$$

**+**: no bias

**-**: higher variance (because single-sample estimate)

Can we combine these two, to control bias/variance tradeoff?

![discount reward](https://4133958719-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LigLKy0c06y4iTEtrkI%2F-LlWhlLO1ma5-siFFwf1%2F-LlWjotVMr1YF9YZpV2M%2F1565006556738.png?generation=1565008023955882\&alt=media)

Reward declines due to discount factor $$\gamma$$ . We can early cut.

![n-step returns](https://4133958719-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LigLKy0c06y4iTEtrkI%2F-LlWhlLO1ma5-siFFwf1%2F-LlWjotXXnsVACS7wbJb%2F1565006522156.png?generation=1565008024012673\&alt=media)

$$
\hat{A}^\pi\_n(s\_t,a\_t)=\sum\_{t'=t}^{t+n}\gamma^{t'-t}r(s\_{t'},a\_{t'})+\gamma^n \hat{V}^\pi\_\phi(s\_{t+n})-\hat{V}^\pi\_\phi(s\_{t})
$$

Choosing $$n>1$$ often works better.

## Generalized advantage estimate

Do we have to choose just one $$n$$ ? We can cut everywhere all at once.

Use weighted combination of n-step returns:

$$w\_n\propto \lambda^{n-1}$$

$$
\begin{aligned}
\hat{A}^\pi\_{GAE}(s\_t,a\_t)&=\sum\_{n=1}^\infty w\_n \hat{A}^\pi\_n(s\_t,a\_t)    \\
&=r(s\_t,a\_t)+\gamma((1-\lambda))\hat{V}^\pi\_\phi(s\_{t+1})+\lambda(r(s\_{t+1},a\_{t+1})+\gamma((1-\lambda)\hat{V}^\pi\_\phi(s\_{t+2})+\lambda r(s\_{t+2},a\_{t+2})+\cdots))\\
&=\sum\_{t'=t}^\infty (\gamma\lambda)^{t'-t}(r(s\_{t'},a\_{t'})+\gamma\hat{V}^\pi\_\phi(s\_{t'+1})-\hat{V}^\pi\_\phi(s\_{t'}))\\
&=\sum\_{t'=t}^\infty (\gamma\lambda)^{t'-t}\delta\_{t'}
\end{aligned}
$$