Of Notes
Shannon Entropy
Table of Contents
We start with surprise which if we take the expected value of, will lead us to entropy. The measure is the one named for Claude Shannon.
and for discrete distributions:
Cross Entropy
When computing the KL Divergence of two distributions we state:
Which if re-aranged to isolate
But simply the cross entropy is for a continuous case the expected negative log of