> For the complete documentation index, see [llms.txt](https://drdh.gitbook.io/rl/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://drdh.gitbook.io/rl/deep-rl-course/actor-critic-algorithms/advantages.md).

# Advantages

From last chapter, policy gradient is $$\nabla\_\theta J(\theta)\approx \frac{1}{N}\sum\_i \left(\sum\_{t=1}^T \nabla\_\theta \log \pi\_\theta(a\_{i,t}|s\_{i,t})(\sum\_{t'=t}^T r(s\_{i,t'},a\_{i,t'})) \right)$$, and then update the policy's parameter $$\theta \leftarrow \theta+\alpha \nabla\_{\theta}J(\theta)$$. Here $$\hat{Q}*{i,t}^\pi=\sum*{t'=t}^T r(s\_{i,t'},a\_{i,t'})$$ is Q function or "reward to go". We knew that $$\hat{Q}*{i,t}^\pi$$ is the estimate of expected reward if we take action $$a*{i,t}$$ in state $$s\_{i,t}$$, but it just used one trajectory to estimate. Can we do better? In theory, the true expected reward-to-go is:

$$
Q(s\_t,a\_t)=\sum\_{t'=t}^T \mathbb{E}\left\[r(s\_{t'},a\_{t'})|s\_t,a\_t\right]
$$

Now, how about the baseline? In policy gradient, we use average Q, i.e. $$b\_t=\frac{1}{N}\sum\_i Q(s\_{i,t},a\_{i,t})$$. We can also use value function $$V(s\_t)=\mathbb{E}*{a\_t\sim \pi*\theta(a\_t|s\_t)}\[Q(s\_t,a\_t)]$$. So we have:

$$
\begin{aligned}
\nabla\_{\theta}J(\theta)&\approx \frac{1}{N}\sum\_{i=1}^N\sum\_{t=1}^T\nabla\_\theta\log\pi\_\theta (a\_{i,t}|s\_{i,t})(Q(s\_{i,t},a\_{i,t})-V(s\_{i,t}))\\
&\approx \frac{1}{N}\sum\_{i=1}^N\sum\_{t=1}^T\nabla\_\theta\log\pi\_\theta (a\_{i,t}|s\_{i,t})A(s\_{i,t},a\_{i,t})
\end{aligned}
$$

The better the estimate of advantage $$A(s\_t,a\_t)$$, the lower the variance.