Of Notes

Bayesian Rule & Inference

4 min read · revised 16 October 2026 · referred to by 5 notes · ·

Everything is governed by Thomas Bayes' Rule:

In inference we think about as some hypothesis and as new observed data. The rule then tells us how to update our confidence in the hypothesis given data.

This can be thought about simplified as:

Prior -

this is what you believe about A before you see B, this could be that "A person has a disease" which occurs naturally with .

Likelihood -

Now our observation can mean that "the test is positive" and we know that:

which means that 99% of the people who have the disease and take the test, test positive.

Evidence -

Now we need to understand all the cases in which we would observe the given evidence or outcomes on which we are conditioning - when B would be Positive

In case of disease we need to account for:

  Sick Healthy
Positive 0.99 0.05

this gives us that:

To compute we need to account for either a true positive which is when Sick AND Positive OR Healthy AND Positive, by which we arrive at:

Posterior

The fundamental idea of a posterior is that it is a combination of the prior and likelihood.

Now we can calculate the posterior probability of given , this is an exercise for recap.


P(B) — evidence:
P(A|B) — posterior:

Derivation of Bayes Rule

We start with simple conditional probability:

You can rearrange either formulation to obtain the joint probability and plug that into the other.

And boom.

Inference

When we want to use this to infer something we write it as

where is some unknown parameter and is some data we observe. We can maybe have a coin with unknown bias and we don't know . Right before flipping the coin however we might have some belief which is our prior.

After flipping 10 times we get so we formulate which in this case would lead us to think and we need to figure out the opposite.

Once we have observed , will not vary with different since we are only observing, that is why we write that:

since the evidence which we do not include in the above formulation must encompass all possible variations of some continuous parameter we can write:

A much more in depth explanation and development can be found @ Bayesian Inference.

Decision Theory

We can extend this to deciding what to do next given some data and unknown state . To move forward we need to create some Loss:

where is some action we need to pick. With inference we get and since we do not know we cannot directly minimize , but we do have a possible distribution over possible parameters. This way we can average the loss over all possible u's:

and with this expected calculation we can wrap that into a minimization step:

This translates well into Losses.

/ or Ctrl K
search; empty, it shows notes to revise and lately read
m
mark this note to revise, or unmark it
c
cover maths, proofs and solutions to test yourself
n
uncover the next covered item
y
copy this note’s org-roam link
e
copy this note as markdown, LaTeX kept as TeX
p
the next problem to work; there h hints, s shows the solution, x passes it for today
→ ← j k space
turn the page, when reading in pages
l
on the map, trace the greatest distance, then the longest path
double-click
a formula, to copy its TeX
rest on maths
bind a part of a formula and open the wheel of notes it comes from; scroll or ↑ ↓ widens it, 1–6 goes
?
this card