Consider the following supervised learning problem: we are given some data consisting of inputs and output labels . We wish to predict the output label of a new input point , using a discriminative model where is of the form with parameters and is a known function.
(a) [20 marks] Assuming a loss function given by the matrix , where (= loss when true class is but we predict ) and (= loss when true class is but we predict ) and are known, derive the decision rule to classify a new input .
(b) [10 marks] Assuming that the labels are conditionally independent given and , where the $i$-th row of is , state the likelihood of the labels given .
Let now the function be the logistic sigmoid
(c) [20 marks] Show that the gradient of the log-likelihood has the form , and find out the expression of .
(d) [20 marks] Show that the Hessian matrix of the log-likelihood has the form , for the appropriate choice of the matrix . Show then that has a unique maximum (i.e. that its Hessian is negative definite).
(e) Let's now assume the parameters to have a prior distribution which is a multivariate normal with zero mean and covariance matrix .
(i) [15 marks] Show that the posterior distribution for is not normal but it is unimodal.
(ii) [15 marks] Describe how to derive the Laplace approximation for this posterior distribution.