Elements of Statistical Learning

January 2022 Examination

3 problems · 0 solved

  1. 1.Bayesian Linear Regression and EM Algorithmsolution

    Consider the following Bayesian linear regression model

    with known and the prior for is , with known. Now assume that you are given a data set of one training input and the associated response . (a) [25 marks] Using Bayes theorem, derive the expression of the posterior distribution for . (b) [20 marks] Explain what happens in the two limit cases where and . Discuss the interpretation of these two scenarios from the point of view of the trade off between our prior knowledge and what we learn from the data. (c) [20 marks] Describe how the E-M algorithm can be used to estimate both and from the data, via evidence approximation (with known). Define the E-step and the function to be maximised in the M-step. (d) [20 marks] Find an explicit expression for the function as a function of and , where the expectations are over the posterior of . (e) [15 marks] Maximise with respect to , and hence derive the E-M update equation for this parameter.

  2. 2.Discriminative Classification and Laplace Approximationsolution

    Consider the following supervised learning problem: we are given some data consisting of inputs and output labels . We wish to predict the output label of a new input point , using a discriminative model where is of the form with parameters and is a known function. (a) [20 marks] Assuming a loss function given by the matrix , where (= loss when true class is but we predict ) and (= loss when true class is but we predict ) and are known, derive the decision rule to classify a new input . (b) [10 marks] Assuming that the labels are conditionally independent given and , where the $i$-th row of is , state the likelihood of the labels given . Let now the function be the logistic sigmoid

    (c) [20 marks] Show that the gradient of the log-likelihood has the form , and find out the expression of . (d) [20 marks] Show that the Hessian matrix of the log-likelihood has the form , for the appropriate choice of the matrix . Show then that has a unique maximum (i.e. that its Hessian is negative definite). (e) Let's now assume the parameters to have a prior distribution which is a multivariate normal with zero mean and covariance matrix . (i) [15 marks] Show that the posterior distribution for is not normal but it is unimodal. (ii) [15 marks] Describe how to derive the Laplace approximation for this posterior distribution.

  3. 3.Hierarchical Clustering and Gaussian Mixture Modelssolution

    Consider the following measured inputs:

    (a) [25 marks] Using the Euclidean distance in and complete linkage, draw the dendrogram for these inputs and comment on the possible clustering results based on visual inspection of the dendrogram. (b) [15 marks] Describe one advantage and one disadvantage of using the complete linkage with respect to the single linkage. (c) [25 marks] Define the within-cluster sum of squares and draw a sketch of the within-cluster sum of squares for the hierarchy of clusters obtained from the dendrogram (Exact values are not needed in the sketch but you must represent the order of magnitude correctly for each grouping). What is the best number of clusters based on this measure? (d) [15 marks] What different measure can you use to choose the number of clusters, if you are interested not only in the within-cluster variance but also in the separation between different clusters? Define this measure and explain how it can be used to select the best number of clusters. (e) [20 marks] Define a Gaussian mixture model to cluster the inputs above in clusters. Assuming you have estimated the parameters of this model, explain how you can assign each input to one of the clusters.

Solutions

  1. 1.Bayesian Linear Regression and EM Algorithm

    No solution yet.

  2. 2.Discriminative Classification and Laplace Approximation

    No solution yet.

  3. 3.Hierarchical Clustering and Gaussian Mixture Models

    No solution yet.

/ or Ctrl K
search; empty, it shows notes to revise and lately read
m
mark this note to revise, or unmark it
c
cover maths, proofs and solutions to test yourself
n
uncover the next covered item
y
copy this note’s org-roam link
e
copy this note as markdown, LaTeX kept as TeX
p
the next problem to work; there h hints, s shows the solution, x passes it for today
→ ← j k space
turn the page, when reading in pages
l
on the map, trace the greatest distance, then the longest path
double-click
a formula, to copy its TeX
rest on maths
bind a part of a formula and open the wheel of notes it comes from; scroll or ↑ ↓ widens it, 1–6 goes
?
this card