Of Notes

Shannon Entropy

1 min read · revised 11 November 2026 · referred to by 8 notes · ·

Table of Contents

We start with surprise which if we take the expected value of, will lead us to entropy. The measure is the one named for Claude Shannon.

and for discrete distributions:

Cross Entropy

When computing the KL Divergence of two distributions we state:

Which if re-aranged to isolate (our cross entropy term):

But simply the cross entropy is for a continuous case the expected negative log of under :

/ or Ctrl K
search; empty, it shows notes to revise and lately read
m
mark this note to revise, or unmark it
c
cover maths, proofs and solutions to test yourself
n
uncover the next covered item
y
copy this note’s org-roam link
e
copy this note as markdown, LaTeX kept as TeX
p
the next problem to work; there h hints, s shows the solution, x passes it for today
→ ← j k space
turn the page, when reading in pages
l
on the map, trace the greatest distance, then the longest path
double-click
a formula, to copy its TeX
rest on maths
bind a part of a formula and open the wheel of notes it comes from; scroll or ↑ ↓ widens it, 1–6 goes
?
this card