> For the complete documentation index, see [llms.txt](https://drdh.gitbook.io/rl/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://drdh.gitbook.io/rl/deep-rl-course/actor-critic-algorithms/baselines.md).

# Baselines

## Critic as state-dependent baselines

Actor-critic

$$
\nabla\_\theta J(\theta)\approx\frac{1}{N}\sum\_{i=1}^N\sum\_{t=1}^T\nabla\_\theta \log \pi\_\theta(a\_{i,t}|s\_{i,t})\left(r(s\_{i,t},a\_{i,t})+\gamma \hat{V}^\pi\_\phi(s\_{i,t+1})-\hat{V}^\pi\_\phi(s\_{i,t})    \right)
$$

**+**: lower variance (due to critic)

**-**: not unbiased (if the critic is not perfect)

Policy gradient

$$
\nabla\_\theta J(\theta)\approx\frac{1}{N}\sum\_{i=1}^N\sum\_{t=1}^T\nabla\_\theta \log \pi\_\theta(a\_{i,t}|s\_{i,t})\left(\sum\_{t'=t}^T\gamma^{t'-t}r(s\_{i,t'},a\_{i,t'})-b\right)
$$

**+**: no bias

**-**: higher variance (because single-sample estimate)

can we use $$\hat{V}^\pi\_\phi$$ and still keep the estimate unbiased ?

$$
\nabla\_\theta J(\theta)\approx\frac{1}{N}\sum\_{i=1}^N\sum\_{t=1}^T\nabla\_\theta \log \pi\_\theta(a\_{i,t}|s\_{i,t})\left(\sum\_{t'=t}^T\gamma^{t'-t}r(s\_{i,t'},a\_{i,t'})-\hat{V}^\pi\_\phi(s\_{i,t})\right)
$$

**+**: no bias

**+**: lower variance (baseline is closer to rewards)

## Control variates: action-dependent baselines

**In theory**

$$
A^\pi(s\_t,a\_t)=Q^\pi(s\_t,a\_t)-V^\pi(s\_t)
$$

**Option 1:**

$$
\hat{A}^\pi(s\_t,a\_t)=\sum\_{t'=t}^\infty \gamma^{t'-t}r(s\_{t'},a\_{t'})-V^\pi\_\phi(s\_t)
$$

**+**: no bias

**-**: higher variance (because single-sample estimate)

**Option 2:**

$$
\hat{A}^\pi(s\_t,a\_t)=\sum\_{t'=t}^\infty \gamma^{t'-t}r(s\_{t'},a\_{t'})-Q^\pi\_\phi(s\_t,a\_t)
$$

**+**: goes to zero in expectation is critic is correct

**-**: not correct

**Combination**:

$$
\nabla\_\theta J(\theta)\approx\frac{1}{N}\sum\_{i=1}^N\sum\_{t=1}^T\nabla\_\theta \log \pi\_\theta(a\_{i,t}|s\_{i,t})\left(\hat{Q}*{i,t}- Q^\pi*\phi(s\_{i,t},a\_{i,t})\right)
\+
\frac{1}{N}\sum\_{i=1}^N\sum\_{t=1}^T\nabla\_\theta \mathbb{E}*{a\sim\pi*\theta(a\_t|s\_t)}\left\[Q^\pi\_\phi(s\_{i,t},a\_{i,t}) \right]
$$