Of Notes
Conjugate Prior
Table of Contents
A conjugate prior is a probability distribution chosen such that a posterior probability belongs to the same family as the prior distribution.
Suppose we have a likelihood which follows the Bernoulli Distribution of something like a coin toss:
We now want to pick a prior that will make the product with this distribution easy to later normalize or analyze. For this case we might pick a Beta Distribution for
When we combine these two distribution to achieve our posterior distribution they merge quite nicely:
Which we finally express with all the dependent parameters:
Example Conjugate Priors
| Likelihood | Unknown parameter | Conjugate Prior |
|---|---|---|
| Bernoulli | success probability |
Beta |
| Binomial | success probability |
Beta |
| Categorical / Multinomial | category probabilities |
Dirichlet |
| Poisson | rate |
Gamma |
| Exponential | rate |
Gamma |
| Geometric | success probability |
Beta |
| Negative binomial | success probability |
Beta |
| Gaussian (known variance) | mean |
Gaussian |
| Gaussian (known mean) | precision |
Gamma |
| Gaussian (known mean) | variance |
Inverse-Gamma |
| Gaussian (both unknown) | mean |
Normal-Gamma |
| Multivariate Gaussian (known covariance) | mean vector | Gaussian |
| Multivariate Gaussian (known mean) | covariance matrix | Inverse-Wishart |
| Uniform |
upper bound |
Pareto |