Of Notes

Gaussian Linear Model, Ridge Regression, and Regularization

1 min read · revised 17 October 2026 · ·

Table of Contents

Problem

Consider the Gaussian linear model for prediction in matrix form, i.e. where ( denotes transpose) are a set of observations, is a known matrix (), a vector of unknown parameters and , being the identity matrix and assumed to be invertible.

(a) [20 marks] Derive the maximum likelihood estimators and for and respectively.

(b) [10 marks] Let be the fitted values. Derive the expression of the hat matrix such that . Show that is a projection matrix (i.e. and ) and that the residuals vector and are uncorrelated.

(c) [20 marks] Now consider the ridge regression estimator for , i.e. with a regularisation parameter. Derive the explicit expression for and compute its mean square error.

(d) [10 marks] Sketch the plot of the MSE as a function of and discuss what it means in terms of the benefit of introducing some regularisation in the estimator.

(e) [30 marks] Mathematically, what is the connection between the ridge regression estimator and estimators given by Bayesian linear models? Explain what is instead the difference in the way the regularisation parameter is chosen and list one advantage and one disadvantage in each case.

(f) [10 marks] Explain how to modify the penalty term in the regularized regression problem so that the resulting estimator has the property of sparsity.

Solution

/ or Ctrl K
search; empty, it shows notes to revise and lately read
m
mark this note to revise, or unmark it
c
cover maths, proofs and solutions to test yourself
n
uncover the next covered item
y
copy this note’s org-roam link
e
copy this note as markdown, LaTeX kept as TeX
p
the next problem to work; there h hints, s shows the solution, x passes it for today
→ ← j k space
turn the page, when reading in pages
l
on the map, trace the greatest distance, then the longest path
double-click
a formula, to copy its TeX
rest on maths
bind a part of a formula and open the wheel of notes it comes from; scroll or ↑ ↓ widens it, 1–6 goes
?
this card