Of Notes
Gaussian Linear Model, Ridge Regression, and Regularization
Problem
Consider the Gaussian linear model for prediction in matrix form, i.e.
(a) [20 marks] Derive the maximum likelihood estimators
(b) [10 marks] Let
(c) [20 marks] Now consider the ridge regression estimator for
(d) [10 marks] Sketch the plot of the MSE as a function of
(e) [30 marks] Mathematically, what is the connection between the ridge regression estimator and estimators given by Bayesian linear models? Explain what is instead the difference in the way the regularisation parameter is chosen and list one advantage and one disadvantage in each case.
(f) [10 marks] Explain how to modify the penalty term in the regularized regression problem so that the resulting estimator has the property of sparsity.