Of Notes
Gaussian Integrals and the Laplace Method
Table of Contents
Week 3 of Mathematical Foundations for Complex Systems Modelling. The goal is to compute integrals of the exponential of a quadratic, and then to use them to approximate much harder integrals. Everything below is built from zero, and every change of variables is carried out step by step.
The Basic Gaussian Integral
We want to compute:
where
There is no antiderivative of
The key observation is that
Now we have two integrals which both use the letter
Next, the product of two integrals over independent variables is the double integral of the product of the integrands. To see why, fix
Applying this with
So squaring did two things at once: it gave us permission to introduce a second variable
Change of Variables to Polar Coordinates
The exponent only depends on the radius
The change of variables and its inverse are:
Substituting into the exponent directly gives
The area element transforms as
The whole plane
The
Change of Variables in the Radial Integral
We substitute
Differentiating both sides with respect to
so the measure
The bounds transform too: when
Replacing
where the remaining integral is one because
Since
Dividing both sides by
which says the function
Adding a Linear Term
Next we add a linear term
We complete the square. We expand
So our exponent equals the square minus too much by
where the last term is a constant, so it factors out of the integral.
Change of Variables: the Shift
We shift
Differentiating gives
The bounds
The square term becomes
where the last equality is the basic Gaussian integral from the first section.
In the normalized form of the lecture notes (Eq. 4.25), this is called the Gaussian linearization identity (or Gauss–Hubbard–Stratonovich transformation):
which linearizes quadratic exponents in partition function calculations: the quadratic in the exponent trades against the linear term to produce a pure exponential of
Several Dimensions
Now let
We compute:
Change of Variables: the Matrix Square Root
The matrix
Since every
where
Two facts about these matrices, which follow from multiplying out the diagonal form:
The second one holds because diagonal matrices commute:
We change variables to
For a linear change of variables
Here
where
Now we transform the exponent. Substituting
So the coupled quadratic form in
Putting the Jacobian back:
With a Linear Term
We add a linear term
Since
We multiply out
and we evaluate the three correction terms one at a time.
The first collapses because
The second is the transpose of the first. A transpose of a scalar is the same scalar, so
For the last one we use the symmetry of
Putting the four terms back together and multiplying by
So our exponent equals the square minus too much by
which is the matrix version of
Change of Variables: the Shift
We shift
Differentiating,
The domain is all of
The square term becomes
where the last equality is the multivariate Gaussian integral from the previous section.
This is the fundamental multivariate Gaussian identity.
We can now derive the density of the multivariate Gaussian from it. We look for a probability density of the form
We fix
where we used the multivariate Gaussian integral from the previous section. Solving for
Setting
The Laplace Method
We now invert the logic: instead of computing Gaussian integrals exactly, we use them to approximate integrals which are not Gaussian.
We consider:
where
Why the Minimum Dominates
The integrand is largest where
At a point where
So for
Taylor Expansion Around the Minimum
Since only
The Taylor expansion of a function around a point
We apply this to
For
For
Substituting both into the integral:
where the constant
Change of Variables: the Rescaling
The remaining integral is almost the basic Gaussian integral, except for the width
We define
Differentiating with respect to
The bounds transform into:
Since
The exponent transforms as:
where the
Putting the transformed measure, bounds and exponent together:
where the last equality is the basic Gaussian integral with
The 1D Laplace Formula
Collecting everything (lecture notes Eq. 4.17):
where
Several Dimensions
The same derivation works over a domain
The scalar Taylor expansion stacks into vector form, and again the linear term vanishes because
Change of Variables: the Rescaling
Exactly as in one dimension we define the rescaling variable
The change of variables is linear with matrix
The exponent cancels exactly as in one dimension. Substituting
Since
The result collects as:
where
Application: Stirling's Approximation
As a check that the machine works, we derive the asymptotics of the factorial.
We start from Euler's integral representation:
To check that this integral really is
The boundary term vanishes at both ends:
The integrand does not have the shape
Change of Variables: Pulling N Out
We substitute
Differentiating,
The bounds stay
Now each factor transforms separately. The power becomes
where the
To move
Now it has the shape
Finding the Minimum
We find the stationary point:
and evaluate the ingredients. The value at the minimum:
The second derivative:
Plugging into the Formula
The 1D Laplace formula gives:
Multiplying back by the prefactor
where we combined
which is James Stirling's approximation, and the factorial is exactly the kind of integral where no elementary antiderivative exists, so this is the only way to get at its large-