Of Notes

Surprisal Self Information

1 min read · revised 17 October 2026 · referred to by 2 notes · ·

Table of Contents

This is a numerical measure of how surprising some observation is, introduced by Claude Shannon as self-information.

Naturally, the less probable some event is, the more surprising it is.

We measure this information in bits and define:

Surprisal \(-\log_2 p\): halving the probability adds exactly one bit, and a certain event (\(p = 1\)) carries none.
Fig. 1. Surprisal : halving the probability adds exactly one bit, and a certain event () carries none.

Origin of the Logarithm

The log property of lets us say that for two distinct events with a joint PDF: we want to however know:

From which we can apply the log rule and say:

We introduce the negative since which if passed into the log .

/ or Ctrl K
search; empty, it shows notes to revise and lately read
m
mark this note to revise, or unmark it
c
cover maths, proofs and solutions to test yourself
n
uncover the next covered item
y
copy this note’s org-roam link
e
copy this note as markdown, LaTeX kept as TeX
p
the next problem to work; there h hints, s shows the solution, x passes it for today
→ ← j k space
turn the page, when reading in pages
l
on the map, trace the greatest distance, then the longest path
double-click
a formula, to copy its TeX
rest on maths
bind a part of a formula and open the wheel of notes it comes from; scroll or ↑ ↓ widens it, 1–6 goes
?
this card