Expectation and Variance

Expected Value

The expected value (mean, expectation) is the long-run average value of a random variable.

For discrete variables:

\[E[X] = \sum_x x \cdot p(x)\]

For continuous variables:

\[E[X] = \int_{-\infty}^{\infty} x \cdot f(x) dx\]

Notation: $E[X]$, $\mathbb{E}[X]$, or $\mu_X$

Properties of Expectation

• Linearity: $E[aX + bY] = aE[X] + bE[Y]$
• Constant: $E[c] = c$
• Independence: If $X$ and $Y$ are independent, $E[XY] = E[X]E[Y]$

• Law of the unconscious statistician (LOTUS):

  \[E[g(X)] = \sum_x g(x) p(x) \quad \text{(discrete)}\] \[E[g(X)] = \int_{-\infty}^{\infty} g(x) f(x) dx \quad \text{(continuous)}\]

Variance

Variance measures the spread or dispersion around the mean:

\[\text{Var}(X) = E[(X - \mu)^2]\]

Computational formula:

\[\text{Var}(X) = E[X^2] - (E[X])^2\]

Notation: $\text{Var}(X)$, $\sigma_X^2$, or $\sigma^2$

Properties of Variance

• Constant: $\text{Var}(c) = 0$
• Scaling: $\text{Var}(aX) = a^2 \text{Var}(X)$
• Shift: $\text{Var}(X + c) = \text{Var}(X)$

• Sum of independent variables:

  \[\text{Var}(X + Y) = \text{Var}(X) + \text{Var}(Y) \quad \text{(if independent)}\]

• General sum:

  \[\text{Var}(X + Y) = \text{Var}(X) + \text{Var}(Y) + 2\text{Cov}(X, Y)\]

Standard Deviation

Standard deviation is the square root of variance:

\[\sigma = \sqrt{\text{Var}(X)}\]

Same units as $X$, more interpretable than variance.

Covariance

Covariance measures how two variables vary together:

\[\text{Cov}(X, Y) = E[(X - \mu_X)(Y - \mu_Y)]\]

Computational formula:

\[\text{Cov}(X, Y) = E[XY] - E[X]E[Y]\]

Properties:

• $\text{Cov}(X, X) = \text{Var}(X)$
• $\text{Cov}(X, Y) = \text{Cov}(Y, X)$ (symmetric)
• $\text{Cov}(aX, bY) = ab \cdot \text{Cov}(X, Y)$
• $\text{Cov}(X + Y, Z) = \text{Cov}(X, Z) + \text{Cov}(Y, Z)$

Correlation

Correlation is normalized covariance (scale-free):

\[\rho(X, Y) = \frac{\text{Cov}(X, Y)}{\sigma_X \sigma_Y}\]

Properties:

• $-1 \leq \rho \leq 1$
• $\rho = 1$: perfect positive linear relationship
• $\rho = -1$: perfect negative linear relationship
• $\rho = 0$: no linear relationship (but may have nonlinear dependence)
• Independence implies zero correlation, but zero correlation does NOT imply independence

Sample Statistics

Given data $x_1, \ldots, x_n$:

Sample mean:

\[\bar{x} = \frac{1}{n} \sum_{i=1}^n x_i\]

Sample variance (unbiased estimator):

\[s^2 = \frac{1}{n-1} \sum_{i=1}^n (x_i - \bar{x})^2\]

Sample standard deviation:

\[s = \sqrt{s^2}\]

Sample covariance:

\[\text{Cov}(X, Y) = \frac{1}{n-1} \sum_{i=1}^n (x_i - \bar{x})(y_i - \bar{y})\]

Moments

Raw moments: $\mu_k’ = E[X^k]$

Central moments: $\mu_k = E[(X - \mu)^k]$

• $\mu_1 = \mu$ (mean)
• $\mu_2 = \sigma^2$ (variance)
• $\mu_3$ relates to skewness (asymmetry)
• $\mu_4$ relates to kurtosis (tail heaviness)

Moment Generating Function (MGF)

\[M_X(t) = E[e^{tX}]\]

Key property: $k$-th derivative at $t=0$ gives $k$-th moment:

\[M_X^{(k)}(0) = E[X^k]\]

Uniqueness: If two distributions have the same MGF, they are identical.

Inequalities

Markov’s inequality (for non-negative $X$):

\[P(X \geq a) \leq \frac{E[X]}{a}\]

Chebyshev’s inequality:

\[P(|X - \mu| \geq k\sigma) \leq \frac{1}{k^2}\]

Guarantees that most mass lies within a few standard deviations of the mean.

Law of Large Numbers

Weak LLN: Sample mean converges in probability to expected value:

\[\bar{X}_n \xrightarrow{P} \mu \quad \text{as } n \to \infty\]

Strong LLN: Sample mean converges almost surely:

\[\bar{X}_n \xrightarrow{a.s.} \mu \quad \text{as } n \to \infty\]

Foundation of Monte Carlo methods and empirical risk minimization.

Central Limit Theorem

For i.i.d. variables $X_1, \ldots, X_n$ with mean $\mu$ and variance $\sigma^2$:

\[\frac{\bar{X}_n - \mu}{\sigma/\sqrt{n}} \xrightarrow{d} \mathcal{N}(0, 1)\]

As $n \to \infty$, the sample mean approaches a Normal distribution regardless of the original distribution.