Elements of Statistical Learning
Summer 2023 Examination
1.Univariate Normal MLE, MAP, and Empirical Bayessolution
Consider an independent and identically distributed sample
from a univariate Normal distribution(a) [5 marks] Derive the expression of the maximum likelihood estimator for the mean
. (b) [7 marks] Assuming known and a prior distribution , derive the expression of the posterior distribution for . (c) [13 marks] Explain how an Empirical Bayes approach can be used to estimate the variance parameter (now assumed to be unknown) and and derive the expression of the marginal likelihood to be maximised.2.K-Means Clustering, Silhouette Index, and GMMsolution
You measured the following data points:
(a) [5 marks] Using the Euclidean distance in
and the starting representatives , you are asked to consider the K-means clustering with . How many iterations will be required for the 2-means algorithm to converge and what is the final clustering assignment? (b) [5 marks] Show that the 2-means algorithm would return a different clustering assignment for other choices of the starting representatives. (c) [8 marks] Using two clustering assignments obtained at point (a) and (b) from different starting representatives, compute the silhouette index for the point . Comment on how well the two assignments are clustering this point. (d) [7 marks] Define a Gaussian mixture model to cluster the inputs above in clusters. Describe one advantage of this approach over the K-means (for a generic sample, not necessarily the one considered in this question).3.Generative Classification and Decision Rulesolution
You are tasked to develop a classifier for a set of univariate observations with 3 possible classes, using the generative model
and the loss function given by the matrix
(a) [7 marks] Assuming the parameters
known, derive the expression of the decision rule to classify a new observation . (b) [10 marks] Derive the expression of the marginal distribution . (c) [8 marks] Now let . How would you classify the new observation ? Should you trust this prediction? Why?4.Gaussian Processes and Kernel Validitysolution
Consider Gaussian Processes (GPs) on inputs
. (a) [5 marks] State the definition for a stochastic process to be a GP with mean function and covariance function . Explain what it means for the covariance function to be valid. (b) [5 marks] Show that the covariance function is valid. (c) [5 marks] Show that, given two valid covariance functions and , is valid. (d) [10 marks] Now consider a Gaussian process regression , where for a one-dimensional input with the kernel functionand you are given one input value
and corresponding observation . Compute the predictive distribution in and discuss what happens to the uncertainty of the prediction in the limit .
Solutions
1.Univariate Normal MLE, MAP, and Empirical Bayes
No solution yet.
2.K-Means Clustering, Silhouette Index, and GMM
No solution yet.
3.Generative Classification and Decision Rule
No solution yet.
4.Gaussian Processes and Kernel Validity
No solution yet.