Of Notes

Multivariate Gaussian Distribution

2 min read · revised 5 November 2026 · referred to by 3 notes · ·

Previsouly when we had independent observations equal to:

We can now write as:

I prefer the vector notation of vector values in this distribution because it's easier to see.

Intuitively lets start with which yields in exponential term in the form of:

By the product of exponentials:

The key step comes in transforming this into a vectorized form:

First we represent the differences as a vector of differences:

Turning it into a row vector:

Now to introduce the variance we define:

Of which the inverse is:

Now we have nicely isolated the numerator and denominator terms of the summation. We can reconstruct this fully by multiplying:

You will notie a missing square in the numertor, we fix:

This is called the Quadratic Form in Matrices.

We now have a very early form of the multivariate gaussian. What we recovered in this last term is what you would call the Mahalnobis distance: .

Level sets of the density for diagonal \(\Sigma\) with \(\sigma_1 > \sigma_2\): points at equal Mahalanobis distance \(d_M\) from \(\vec{\mu}\) lie on an axis-aligned ellipse, with semi-axes \(d_M\sigma_1\) and \(d_M\sigma_2\).
Fig. 1. Level sets of the density for diagonal with : points at equal Mahalanobis distance from lie on an axis-aligned ellipse, with semi-axes and .

Properties of Variance

The name for the inverse of the variance is yet again what we call precision:

We also note that the number of free parameters in a full variance matrix grows as . This scale is mitigated by three other kinds of variances:

Full General
Isotropic Variance
Diagonal Covariance

Eigenvalue Decomposition of Covariance Matrix

We start with the intuition of how eigenvectors and eigenvalues describe the shape of a Gaussian:

Looking at the exponential term we perform a projection of shifted by onto the eigenvectors setting .

Eigenvalue as variance along a direction, for \(\Sigma = \begin{pmatrix}2&1\\1&2\end{pmatrix}\). Projecting the density onto each eigenvector gives a one-dimensional Gaussian whose variance is the eigenvalue: wide for \(\vec{v}_2\) (\(\lambda_2 = 3\)), narrow and tall for \(\vec{v}_1\) (\(\lambda_1 = 1\)). The ellipse is the one-standard-deviation contour, with semi-axes \(\sqrt{\lambda_2}\) and \(\sqrt{\lambda_1}\).
Fig. 2. Eigenvalue as variance along a direction, for . Projecting the density onto each eigenvector gives a one-dimensional Gaussian whose variance is the eigenvalue: wide for (), narrow and tall for (). The ellipse is the one-standard-deviation contour, with semi-axes and .

Sufficient Statistics

Generally for the Gaussian:

/ or Ctrl K
search; empty, it shows notes to revise and lately read
m
mark this note to revise, or unmark it
c
cover maths, proofs and solutions to test yourself
n
uncover the next covered item
y
copy this note’s org-roam link
e
copy this note as markdown, LaTeX kept as TeX
p
the next problem to work; there h hints, s shows the solution, x passes it for today
→ ← j k space
turn the page, when reading in pages
l
on the map, trace the greatest distance, then the longest path
double-click
a formula, to copy its TeX
rest on maths
bind a part of a formula and open the wheel of notes it comes from; scroll or ↑ ↓ widens it, 1–6 goes
?
this card