Elements of Statistical Learning

Summer 2021 Examination

4 problems · 0 solved

  1. 1.Gaussian Linear Model, Ridge Regression, and Regularizationsolution

    Consider the Gaussian linear model for prediction in matrix form, i.e. where ( denotes transpose) are a set of observations, is a known matrix (), a vector of unknown parameters and , being the identity matrix and assumed to be invertible.

    (a) [20 marks] Derive the maximum likelihood estimators and for and respectively.

    (b) [10 marks] Let be the fitted values. Derive the expression of the hat matrix such that . Show that is a projection matrix (i.e. and ) and that the residuals vector and are uncorrelated.

    (c) [20 marks] Now consider the ridge regression estimator for , i.e. with a regularisation parameter. Derive the explicit expression for and compute its mean square error.

    (d) [10 marks] Sketch the plot of the MSE as a function of and discuss what it means in terms of the benefit of introducing some regularisation in the estimator.

    (e) [30 marks] Mathematically, what is the connection between the ridge regression estimator and estimators given by Bayesian linear models? Explain what is instead the difference in the way the regularisation parameter is chosen and list one advantage and one disadvantage in each case.

    (f) [10 marks] Explain how to modify the penalty term in the regularized regression problem so that the resulting estimator has the property of sparsity.

  2. 2.Discriminative vs Generative Classification and Laplace Approximationsolution

    (a) (i) [10 marks] State briefly what is meant by discriminative and generative approaches to classification. List one benefit and one problem with each approach.

    (ii) [25 marks] Consider the case of binary classification where we wish to determine if an input point belongs to one of two classes and . If we take a generative approach and find , , using class conditional densities and prior , predictions are made according to Show that can be written in terms of a logistic sigmoid where , for a suitably defined function .

    (iii) [25 marks] Assuming the class conditional density is Gaussian, for with expectation and covariance matrix , show that with

    (iv) [10 marks] Describe how using basis functions one might extend the above result to more complicated predictors for the class probabilities.

    (b) Consider the following supervised learning problem: we are given some data consisting of inputs and output labels . We wish to predict the output label of a new input point , using a discriminative method where is of the form with arbitrary .

    (i) [10 marks] Assuming that the labels are conditionally independent given and , where the -th row of is , state the likelihood of the labels given .

    (ii) [20 marks] For the case of a zero mean Gaussian prior over with covariance , explain how the Laplace approximation of the posterior is constructed and state the form of the approximate posterior.

  3. 3.Gaussian Process Definition, Valid Kernels, and GP Regressionsolution

    Consider Gaussian Processes (GPs) on inputs .

    (a) [10 marks] State the definition for a stochastic process to be a GP with mean function and covariance function . Explain what it means for the covariance function to be valid.

    (b) [10 marks] Show that the covariance function is valid, if is a symmetric positive definite matrix.

    (c) [10 marks] Show that, for an arbitrary function , the covariance function is valid.

    (d) [15 marks] Consider two independent zero-mean GPs and with covariance functions and respectively. Let , and let be the corresponding Gram matrix. Show that is a valid covariance function.

    (e) Now consider a Gaussian process regression , where is a Gaussian process. (i) [15 marks] Find the marginal distribution of . (ii) [20 marks] Consider predicting a new observation at . By considering the joint distribution of and or otherwise, find the predictive distribution of .

    (f) Let for a one-dimensional input with the kernel function and you are given one input value and corresponding observation . (i) [10 marks] Compute the predictive distribution in and discuss what happens in the limit . (ii) [10 marks] Compute the predictive distribution in and discuss what happens in the limit .

  4. 4.Mixture Distributions, EM Algorithm, and K-Means Connectionsolution

    Consider a random variable with a mixture distribution of the form where the conditional expectation of is and the conditional covariance matrix of is . Let be an i.i.d sample from this mixture model.

    (a) [20 marks] Derive the expressions of the expectation and covariance matrix of .

    (b) [25 marks] Outline how the EM algorithm can be used to estimate the parameters .

    Let us now assume that have multivariate normal distributions with mean and covariance matrix , with the identity matrix.

    (c) [35 marks] Derive the expressions of the update formulas for , and in the M-step of the algorithm.

    (d) [20 marks] Assume now that is known. Comment on what happens to the EM procedure when and what similarities there are between the resulting algorithm and the K-means clustering.

Solutions

  1. 1.Gaussian Linear Model, Ridge Regression, and Regularization

    No solution yet.

  2. 2.Discriminative vs Generative Classification and Laplace Approximation

    No solution yet.

  3. 3.Gaussian Process Definition, Valid Kernels, and GP Regression

    No solution yet.

  4. 4.Mixture Distributions, EM Algorithm, and K-Means Connection

    No solution yet.

/ or Ctrl K
search; empty, it shows notes to revise and lately read
m
mark this note to revise, or unmark it
c
cover maths, proofs and solutions to test yourself
n
uncover the next covered item
y
copy this note’s org-roam link
e
copy this note as markdown, LaTeX kept as TeX
p
the next problem to work; there h hints, s shows the solution, x passes it for today
→ ← j k space
turn the page, when reading in pages
l
on the map, trace the greatest distance, then the longest path
double-click
a formula, to copy its TeX
rest on maths
bind a part of a formula and open the wheel of notes it comes from; scroll or ↑ ↓ widens it, 1–6 goes
?
this card