Elements of Statistical Learning
January 2022 Examination
1.Bayesian Linear Regression and EM Algorithmsolution
Consider the following Bayesian linear regression model
with
known and the prior for is , with known. Now assume that you are given a data set of one training input and the associated response . (a) [25 marks] Using Bayes theorem, derive the expression of the posterior distribution for . (b) [20 marks] Explain what happens in the two limit cases where and . Discuss the interpretation of these two scenarios from the point of view of the trade off between our prior knowledge and what we learn from the data. (c) [20 marks] Describe how the E-M algorithm can be used to estimate both and from the data, via evidence approximation (with known). Define the E-step and the function to be maximised in the M-step. (d) [20 marks] Find an explicit expression for the function as a function of and , where the expectations are over the posterior of . (e) [15 marks] Maximise with respect to , and hence derive the E-M update equation for this parameter.2.Discriminative Classification and Laplace Approximationsolution
Consider the following supervised learning problem: we are given some data
consisting of inputs and output labels . We wish to predict the output label of a new input point , using a discriminative model where is of the form with parameters and is a known function. (a) [20 marks] Assuming a loss function given by the matrix , where (= loss when true class is but we predict ) and (= loss when true class is but we predict ) and are known, derive the decision rule to classify a new input . (b) [10 marks] Assuming that the labels are conditionally independent given and , where the $i$-th row of is , state the likelihood of the labels given . Let now the function be the logistic sigmoid(c) [20 marks] Show that the gradient of the log-likelihood
has the form , and find out the expression of . (d) [20 marks] Show that the Hessian matrix of the log-likelihood has the form , for the appropriate choice of the matrix . Show then that has a unique maximum (i.e. that its Hessian is negative definite). (e) Let's now assume the parameters to have a prior distribution which is a multivariate normal with zero mean and covariance matrix . (i) [15 marks] Show that the posterior distribution for is not normal but it is unimodal. (ii) [15 marks] Describe how to derive the Laplace approximation for this posterior distribution.3.Hierarchical Clustering and Gaussian Mixture Modelssolution
Consider the following measured inputs:
(a) [25 marks] Using the Euclidean distance in
and complete linkage, draw the dendrogram for these inputs and comment on the possible clustering results based on visual inspection of the dendrogram. (b) [15 marks] Describe one advantage and one disadvantage of using the complete linkage with respect to the single linkage. (c) [25 marks] Define the within-cluster sum of squares and draw a sketch of the within-cluster sum of squares for the hierarchy of clusters obtained from the dendrogram (Exact values are not needed in the sketch but you must represent the order of magnitude correctly for each grouping). What is the best number of clusters based on this measure? (d) [15 marks] What different measure can you use to choose the number of clusters, if you are interested not only in the within-cluster variance but also in the separation between different clusters? Define this measure and explain how it can be used to select the best number of clusters. (e) [20 marks] Define a Gaussian mixture model to cluster the inputs above in clusters. Assuming you have estimated the parameters of this model, explain how you can assign each input to one of the clusters.
Solutions
1.Bayesian Linear Regression and EM Algorithm
No solution yet.
2.Discriminative Classification and Laplace Approximation
No solution yet.
3.Hierarchical Clustering and Gaussian Mixture Models
No solution yet.