Of Notes

KL Divergence

1 min read · revised 15 November 2026 · referred to by 6 notes · ·

A very useful metric - from the 1951 paper of Solomon Kullback and Richard Leibler — used to understand how different two distributions and might be, written .

Two distributions \(p\) (dark) and \(q\) (accent) offset from each other. KL divergence measures the gap between them, as seen from \(p\).
Fig. 1. Two distributions (dark) and (accent) offset from each other. KL divergence measures the gap between them, as seen from .

Definition

In the continuous case, for densities and ,

In the discrete case the integral becomes a sum, and it is the same expectation under :

Properties

  • , with equality exactly when (almost everywhere).
  • in general. The order matters: it measures the cost of using when data are distributed as . Because it is not symmetric, KL is not a distance.

Non-negative by Jensen's Inequality

Jensens Inequality says that for a convex function ,

  1. Take , which is convex. Swapping to by the log identity ,

  2. The right-hand side collapses, because the cancels and sums to :

  3. Therefore

The same steps hold with integrals in the continuous case. Since is strictly convex, equality in Jensen needs to be constant, which forces .

Derivation from Entropy

KL divergence comes out of entropy: if we have two models that differ in some way, it measures the "spare" information we pay for by coding with . It is the cross-entropy minus the entropy:

where the last step moves the minus sign into the log.

When Support Differs

When there is some with then the log division would have a division by 0.

Therefore, for KL to be finite, our distributions must satisfy:

/ or Ctrl K
search; empty, it shows notes to revise and lately read
m
mark this note to revise, or unmark it
c
cover maths, proofs and solutions to test yourself
n
uncover the next covered item
y
copy this note’s org-roam link
e
copy this note as markdown, LaTeX kept as TeX
p
the next problem to work; there h hints, s shows the solution, x passes it for today
→ ← j k space
turn the page, when reading in pages
l
on the map, trace the greatest distance, then the longest path
double-click
a formula, to copy its TeX
rest on maths
bind a part of a formula and open the wheel of notes it comes from; scroll or ↑ ↓ widens it, 1–6 goes
?
this card