Expectation, Variance, and Covariance

A distribution contains everything there is to know about a random variable, but we often want a few numbers that summarize it. The expectation summarizes where a distribution is centered, the variance how spread out it is, and the covariance how two random variables move together. The conditional expectation then asks what we expect once we know something about the outcome, and it is the quantity that value functions in reinforcement learning are built on.

info

This chapter covers

• the expectation of a random variable and of a function of it,
• linearity of expectation,
• variance and standard deviation,
• covariance, correlation, the variance of a sum, and the covariance matrix, and
• conditional expectation and the law of total expectation.

Expectation

The expectation (or expected value) of a random variable is the average of its values, each weighted by its probability,

$$\mathbb{E}[X] = \begin{cases} \displaystyle \sum_{x} x \, p(x) & \text{if } X \text{ is discrete,} \\[2ex] \displaystyle \int_{-\infty}^{\infty} x \, f(x) \, dx & \text{if } X \text{ is continuous.} \end{cases}$$

It is also called the mean of $X$ and written $\mu$. If you repeat the experiment many times and average the results, the average approaches $\mathbb{E}[X]$, which chapter 5 makes precise.

For a fair d20, every face has probability $1/20$, so

$$\mathbb{E}[X] = \frac{1 + 2 + \cdots + 20}{20} = 10.5.$$

The expected value is a value the die can never show. It describes the long-run average, not a typical roll.

For the crit indicator $C \sim \text{Bernoulli}(0.05)$, $\mathbb{E}[C] = 0 \times 0.95 + 1 \times 0.05 = 0.05$. In general, the expectation of the indicator of an event is the probability of that event.

Expectation of a Function

To find the expectation of $g(X)$, weight $g(x)$ by the probabilities of $X$. There's no need to work out the distribution of $g(X)$ first,

$$\mathbb{E}[g(X)] = \sum_{x} g(x) \, p(x) \qquad \text{or} \qquad \mathbb{E}[g(X)] = \int_{-\infty}^{\infty} g(x) \, f(x) \, dx.$$

For the d20, $\mathbb{E}[X^2] = (1^2 + 2^2 + \cdots + 20^2)/20 = 143.5$. This is not $(\mathbb{E}[X])^2 = 110.25$. In general $\mathbb{E}[g(X)] \neq g(\mathbb{E}[X])$ unless $g$ is linear.

Linearity

Expectation is linear. For any random variables $X$ and $Y$ and constants $a$, $b$ and $c$,

$$\mathbb{E}[aX + bY + c] = a \, \mathbb{E}[X] + b \, \mathbb{E}[Y] + c.$$

This holds whether or not $X$ and $Y$ are related, which makes it one of the most useful facts in probability. A damage roll of 2d6+3 is $D = D_1 + D_2 + 3$, where $D_1$ and $D_2$ are the two six-sided dice. Each die has mean $3.5$, so $\mathbb{E}[D] = 3.5 + 3.5 + 3 = 10$ without ever writing out the distribution of $D$.

In RL, the return satisfies $G_t = R_{t+1} + \gamma G_{t+1}$, and linearity gives $\mathbb{E}[G_t] = \mathbb{E}[R_{t+1}] + \gamma \, \mathbb{E}[G_{t+1}]$, the first step toward the Bellman equations.

Variance and Standard Deviation

The variance measures how far a random variable typically falls from its mean. It is the expected squared distance from the mean,

$$\operatorname{Var}(X) = \mathbb{E}\big[(X - \mu)^2\big] = \begin{cases} \displaystyle \sum_{x} (x - \mu)^2 \, p(x) & \text{if } X \text{ is discrete,} \\[2ex] \displaystyle \int_{-\infty}^{\infty} (x - \mu)^2 f(x) \, dx & \text{if } X \text{ is continuous.} \end{cases}$$

Expanding the square and using linearity gives a shortcut that is usually easier to compute,

$$\operatorname{Var}(X) = \mathbb{E}[X^2] - (\mathbb{E}[X])^2.$$

For the d20, $\operatorname{Var}(X) = 143.5 - 110.25 = 33.25$. Because the variance is in squared units, we often report the standard deviation $\sigma = \sqrt{\operatorname{Var}(X)}$ instead, which has the same units as $X$. A d20 roll has a standard deviation of about $5.77$.

Shifting a random variable doesn't change its spread, and scaling it scales the standard deviation by the same factor,

$$\operatorname{Var}(aX + b) = a^2 \operatorname{Var}(X).$$

Covariance and Correlation

The covariance of two random variables measures whether they tend to be above their means at the same time,

$$\operatorname{Cov}(X, Y) = \mathbb{E}\big[(X - \mu_X)(Y - \mu_Y)\big] = \mathbb{E}[XY] - \mathbb{E}[X] \, \mathbb{E}[Y].$$

It is positive when large values of $X$ come with large values of $Y$, negative when large values of one come with small values of the other, and $\operatorname{Cov}(X, X) = \operatorname{Var}(X)$. Its size depends on the units of $X$ and $Y$, so the correlation rescales it to lie between $-1$ and $1$,

$$\rho(X, Y) = \frac{\operatorname{Cov}(X, Y)}{\sigma_X \, \sigma_Y}.$$

A correlation of $\pm 1$ means $Y$ is an exact linear function of $X$.

Independent random variables, whose values don't affect each other's probabilities, have zero covariance. The converse is false, because covariance only detects linear relationships. If $X$ is equally likely to be $-1$, $0$ or $1$ and $Y = X^2$, then $\operatorname{Cov}(X, Y) = \mathbb{E}[X^3] - \mathbb{E}[X]\,\mathbb{E}[X^2] = 0$, even though $Y$ is completely determined by $X$.

Variance of a Sum

Covariance is what makes variances of sums more complicated than expectations of sums,

$$\operatorname{Var}(X + Y) = \operatorname{Var}(X) + \operatorname{Var}(Y) + 2 \operatorname{Cov}(X, Y).$$

When $X$ and $Y$ are independent the covariance term vanishes and the variances simply add. The two dice in a 2d6+3 damage roll are independent and each has variance $35/12$, so $\operatorname{Var}(D) = 35/12 + 35/12 = 35/6 \approx 5.83$. The same rule for $n$ independent random variables, $\operatorname{Var}(X_1 + \cdots + X_n) = \operatorname{Var}(X_1) + \cdots + \operatorname{Var}(X_n)$, is the reason averages become less noisy as $n$ grows, which is the subject of chapter 5.

Covariance Matrix

For a random vector $\mathbf{X} = (X_1, \dots, X_d)$, the mean vector $\boldsymbol{\mu} = \mathbb{E}[\mathbf{X}]$ collects the means of the coordinates, and the covariance matrix collects all their pairwise covariances,

$$\boldsymbol{\Sigma} = \mathbb{E}\big[(\mathbf{X} - \boldsymbol{\mu})(\mathbf{X} - \boldsymbol{\mu})^\top\big], \qquad \Sigma_{ij} = \operatorname{Cov}(X_i, X_j).$$

The diagonal holds the variances of the coordinates. A covariance matrix is always symmetric and positive semidefinite. Linear transformations act on it in a simple way,

$$\mathbb{E}[\mathbf{A}\mathbf{X} + \mathbf{b}] = \mathbf{A}\boldsymbol{\mu} + \mathbf{b}, \qquad \operatorname{Cov}(\mathbf{A}\mathbf{X} + \mathbf{b}) = \mathbf{A}\boldsymbol{\Sigma}\mathbf{A}^\top.$$

The covariance matrix is the second parameter of the multivariate normal distribution.

Conditional Expectation

Often we learn something about an outcome and want to update what we expect. The conditional expectation of $X$ given $Y = y$ is the expectation computed with the conditional distribution of $X$ once $Y = y$ is known,

$$\mathbb{E}[X \mid Y = y] = \sum_{x} x \, \Pr(X = x \mid Y = y), \qquad \Pr(X = x \mid Y = y) = \frac{\Pr(X = x, Y = y)}{\Pr(Y = y)}.$$

Back to the dragon fight. The paladin hits armor class 15 with probability $0.3$, deals 2d6+3 damage on a hit and nothing on a miss (ignoring the extra dice on a critical hit). Let $D$ be the damage and $H$ indicate a hit. Then $\mathbb{E}[D \mid H = 1] = 10$ and $\mathbb{E}[D \mid H = 0] = 0$.

Before $Y$ is observed, $\mathbb{E}[X \mid Y]$ is itself a random variable, a function of $Y$ that takes the value $\mathbb{E}[X \mid Y = y]$ when $Y = y$.

Law of Total Expectation

Averaging the conditional expectations over the possible values of $Y$ recovers the overall expectation,

$$\mathbb{E}[X] = \mathbb{E}\big[\mathbb{E}[X \mid Y]\big] = \sum_{y} \mathbb{E}[X \mid Y = y] \, \Pr(Y = y).$$

This is the law of total expectation. For the paladin, the expected damage per attack is $\mathbb{E}[D] = 10 \times 0.3 + 0 \times 0.7 = 3$.

Where this shows up

The value of a state in RL is a conditional expectation, $v_\pi(s) = \mathbb{E}_\pi[G_t \mid S_t = s]$. The Bellman equation for $v_\pi$ comes from applying the law of total expectation twice, first conditioning on the action and then on the next state and reward, together with linearity and the Markov property (see Markov Decision Processes). In supervised learning, the prediction of $Y$ from $X = x$ that minimizes the expected squared error is $\mathbb{E}[Y \mid X = x]$, which is what a regression model tries to approximate.

Recap

• The expectation is the probability-weighted average of a random variable, and $\mathbb{E}[g(X)]$ is computed by weighting $g(x)$ with the probabilities of $X$.
• Expectation is linear for any random variables, dependent or not.
• The variance is the expected squared distance from the mean, and $\operatorname{Var}(X) = \mathbb{E}[X^2] - (\mathbb{E}[X])^2$.
• Covariance measures linear co-movement, independence implies zero covariance but not the other way around, and variances of independent random variables add.
• The covariance matrix collects the pairwise covariances of a random vector.
• The conditional expectation $\mathbb{E}[X \mid Y = y]$ updates the expectation once $Y$ is known, and averaging it over $Y$ gives back $\mathbb{E}[X]$.

1. What is the expected damage of a 2d6+3 roll?

• $15$
• $9.5$
• $10$
• $7$

2. If $\operatorname{Var}(X) = 2$, what is $\operatorname{Var}(3X + 5)$?

• $23$
• $18$
• $6$
• $11$

3. Two random variables have $\operatorname{Cov}(X, Y) = 0$. What can you conclude?

• $\operatorname{Var}(X + Y) = 0$
• $X$ and $Y$ are independent
• $X$ and $Y$ have the same mean
• $X$ and $Y$ have no linear relationship, but may still be dependent

What's Next

So far we have computed expectations from known distributions. In practice the distribution is unknown and we only have samples. Sampling Distributions and the Central Limit Theorem studies how well the average of a sample estimates the expectation.