Of Notes

Claude Shannon

1 min read · revised 16 October 2026 · referred to by 2 notes · ·

American mathematician and engineer, the father of information theory. In one 1948 paper he took "information" — previously a loose, qualitative idea — and made it something you can measure with a single real number, in bits.

  • The Shannon Entropy note carries his name, and the measure itself reappears everywhere in this KB: in the Surprisal Self Information of a single outcome, and inside the KL Divergence which is the Shannon entropy of one distribution sampled from the wrong distribution.
/ or Ctrl K
search; empty, it shows notes to revise and lately read
m
mark this note to revise, or unmark it
c
cover maths, proofs and solutions to test yourself
n
uncover the next covered item
y
copy this note’s org-roam link
e
copy this note as markdown, LaTeX kept as TeX
p
the next problem to work; there h hints, s shows the solution, x passes it for today
→ ← j k space
turn the page, when reading in pages
l
on the map, trace the greatest distance, then the longest path
double-click
a formula, to copy its TeX
rest on maths
bind a part of a formula and open the wheel of notes it comes from; scroll or ↑ ↓ widens it, 1–6 goes
?
this card