Of Notes
KL Divergence
Table of Contents
A very useful metric - from the 1951 paper of Solomon Kullback and Richard Leibler — used to understand how different two distributions
Definition
In the continuous case, for densities
In the discrete case the integral becomes a sum, and it is the same expectation under
Properties
, with equality exactly when (almost everywhere). in general. The order matters: it measures the cost of using when data are distributed as . Because it is not symmetric, KL is not a distance.
Non-negative by Jensen's Inequality
Jensens Inequality says that for a convex function
Take
, which is convex. Swapping to by the log identity ,The right-hand side collapses, because the
cancels and sums to :Therefore
The same steps hold with integrals in the continuous case. Since
Derivation from Entropy
KL divergence comes out of entropy: if we have two models that differ in some way, it measures the "spare" information we pay for by coding
where the last step moves the minus sign into the log.
When Support Differs
When there is some
Therefore, for KL to be finite, our distributions must satisfy: