Of Notes
Claude Shannon
American mathematician and engineer, the father of information theory. In one 1948 paper he took "information" — previously a loose, qualitative idea — and made it something you can measure with a single real number, in bits.
- The Shannon Entropy note carries his name, and the measure itself reappears everywhere in this KB: in the Surprisal Self Information of a single outcome, and inside the KL Divergence which is the Shannon entropy of one distribution sampled from the wrong distribution.