CS6140 Machine Learning Regression

Regression

based on a document by Andrew Ng

September 16, 2014

1 Linear regression

Lets consider a supervised learning example where data points are houses with 2 features (X^{1}= living area; X^{2} = number of bedrooms) and labels are the prices.

Living area (ft^{2}) #bedrooms price (1000$s)
2104  3  400
1600  3  330
2400  3  369
1416  2  232
3000  4  540
...  ...  ...

Each datapoint is thus (x_{t}, y_{t}), where x_{t} = (x_{t}^{1}, x_{t}^{2}) are the living area and the number of bedrooms respectively, and y_{t} is the price. The main problem addressed by regression is: Can we predict the price (or label) y_{t} from input features x_{t} ? Today we study linear regression, that is if we can predict the price from the features using a linear regressor

h_{w}(x) = w^{0} + w^{1}x^{1} + w^{2}x^{2}

where w = (w^{0}, w^{1}, w^{2}) are the parameters of the regression function. Within the class of linear functions (regressors) our task shall be to find the best parameters w. When there is no risk of confusion, we will drop w from the h notation, and we will assume a dummy feature x^{0} = 1 for all datapoints such that we can write

h(x) = ∑_{d=0}^{D} w^{d}x^{d}

where d iterates through input features 1,2,... ,D (in our example D = 2).

What do we mean by the best regression fit? For today, we will measure the error (or cost) of the regression function by the mean square error

J(w) = ∑_{t}(h_{w}(x_{t}) − y_{t})^{2}

and we will naturally look for the w that minimizes the error function. The regresion obtained using the square error function J is called least square regression, or least square fit. We present two methods for minimizing J: a direct linear algebra solution next, and gradient descent optimization later.

2 Least mean square via normal equations

2.1 Matrix derivatives

Let f : ℝ^{m×n} ↦ ℝ a function that takes a matrix as input and outputs a real number. We define the derivative of f with respect to the matrix A as

∇_{A}f(A) = [ \begin{matrix}\frac{∂f}{∂a_{11}} & ... & \frac{∂f}{∂a_{1n}} \\ ... & ... & ... \\ \frac{∂f}{∂a_{m1}} & ... & \frac{∂f}{∂a_{mn}} \end{matrix} ]

where a_{ij} is the element of A on row i and column j. For example, consider

A = [ \begin{matrix}a_{11} & a_{12} \\ a_{21} & a_{22} \end{matrix} ] and the function f(A) = \frac{3}{2}a_{11} + 5a_{12}^{2} + a_{21}a_{22}. Then

∇_{A}f(A) = [ \begin{matrix}\frac{3}{2} & 10a_{12} \\ a_{22} & a_{21} \end{matrix} ]

Today we will look at the trace function as f. The trace of a matrix A is the sum of the elements of the main diagonal

tr(A) = ∑_{i} a_{ii}

The following are simple and well known properties of the trace:

tr(AB) = tr(BA)
tr(A) = tr(A^{T})
tr(A + B) = tr(A) + tr(B)
tr(xA) = xtr(A)

The following properties of the trace matrix derivative are going to be useful for finding an exact regression solution:

∇_{A}tr(AB) = B^{T}  (1)
∇_{A^{T}} f(A) = (∇_{A}f(A))^{T}  (2)
∇_{A}tr(ABA^{T} C) = CAB + C^{T} AB^{T}  (3)

Combining the second and third statements above we get

∇_{A^{T}} tr(ABA^{T} C) = B^{T} A^{T} C^{T} + BA^{T} C  (4)

2.2 An exact solution for regression using linear algebra

Given the datapoints (x_{t}, y_{t}) for t = 1, 2, ..., m, with D input dimensions (features), we shall look at them in a matrix form

X = [ \begin{matrix}x_{1}^{1} & ... & x_{1}^{D} \\ ... & ... & ... \\ x_{m}^{1} & ... & x_{m}^{D} \end{matrix} ] Y = [ \begin{matrix}y_{1} \\ ... \\ y_{m} \end{matrix} ]

Then the error array asociated with our regressor h_{w}(x) = ∑_{d} w^{d}x^{d} is

E = [ \begin{matrix}h_{w}(x_{1}) − y_{1} \\ ... \\ h_{w}(x_{m}) − y_{m} \end{matrix} ] = [ \begin{matrix}x_{1}w \\ ... \\ x_{m}w \end{matrix} ] − [ \begin{matrix}y_{1} \\ ... \\ y_{m} \end{matrix} ] = Xw − Y

(we used w = (w^{0}, w^{1}, ...w^{d})^{T} as a vector column). We can now write the mean square error as

J(w) = \frac{1}{2} ∑_{t}(h_{w}(x_{t}) − y_{t})^{2} = \frac{1}{2}E^{T} E = \frac{1}{2}(Xw − Y)^{T} (Xw − Y)

Then

∇_{w}J(w) = ∇_{w} \frac{1}{2}E^{T} E = ∇_{w} \frac{1}{2}(Xw − Y)^{T} (Xw − Y)
= \frac{1}{2}∇_{w}(w^{T} X^{T} Xw − w^{T} X^{T} Y − Y^{T} Xw + Y^{T} Y)
= \frac{1}{2}∇_{w}tr(w^{T} X^{T} Xw − w^{T} X^{T} Y − Y^{T} Xw + Y^{T} Y)
= \frac{1}{2}∇_{w}(tr(w^{T} X^{T} Xw) − 2tr(Y^{T} Xw))
= \frac{1}{2}(X^{T} Xw + X^{T} Xw − 2X^{T} Y)
= X^{T} Xw − X^{T} Y

because: in the third step we have tr(x) = x, in the four step we have tr(A) = tr(A^{T}), and in the fifth step we are using equation 4 with A^{T} = w, B = B^{T} = X^{T} X, C = I.

Since we are trying to minimize J, a convex function, a sure way to find w that minimizes J is to set its derivative to zero. In doing so we obtain

X^{T} Xw = X^{T} Y or w = (X^{T} X)^{−1}X^{T} Y

This is the exact w that minimizes the mean square error.

3 Least mean square probabilistic interpretation

Why mean square error? We show now that the objective J used is a direct consequence of a very common assumption over the data. Lets look at the errors

ϵ_{t} = h(x_{t}) − y_{t}

and lets make the assumption that they are IID according to a gaussian (normal) distribution of mean μ = 0 and variance σ^{2}. That we write ϵ N (0, σ^{2}) or

p(ϵ) = \frac{1}{\sqrt{2π}σ} exp (−\frac{ϵ^{2}}{2σ^{2}} )

which implies

p(y|x; w) = \frac{1}{\sqrt{2π}σ} exp (−\frac{(wx − y)^{2}}{2σ^{2}} )

Note that above w is a parameter (array) and not a random variable. Given the input X, what is the probability of Y given the parameters w ? Equivalently, this is the likelihood that w is the correct parameter for the model

L(w) = L(w; X, Y) = p(Y |X; w)

Using the IID assumption the likelihood becomes

L(w) = ∏_{t} p(y_{t}|x_{t}); w

= ∏_{t} \frac{1}{\sqrt{2π}σ} exp (−\frac{(wx_{t} − y_{t})^{2}}{2σ^{2}} )

since we have now a probabilistic model over the data, a common way to determine the best parameters is to use maximum likelihood; in other words find w that realizes the maximum L. Instead of maximizing L we shall maximize the log likelihood log L(w) because it simplifies the math (and produces the same ”best” w)

l(w) = log L(w)
= log ∏_{t} \frac{1}{\sqrt{2π}σ} exp (−\frac{(wx_{t} − y_{t})^{2}}{2σ^{2}} )

= ∑_{t} log \frac{1}{\sqrt{2π}σ} exp (−\frac{(wx_{t} − y_{t})^{2}}{2σ^{2}} )
= m log \frac{1}{\sqrt{2π}σ} − \frac{1}{2σ^{2}} ∑_{t}(h_{w}(x_{t}) − y_{t})^{2}

Hence, the maximizing the likelihood L produces the same w as minimizing the mean square error (since the front term does not depend on w). That is to say that, if we believe the errors to be IID normally, then the maximum likelihood is obtained for the parameters w that minimizes the mean square error.

4 Classification and logistic regression

In classification, the labels y are not numeric values (like prices), but instead class labels. For today, lets assume that we have two classes denoted by 0 and 1; we call this binary classification and we write y ∈ {0, 1}.

4.1 Logistic transformation

We could, in principle try to run the linear regression we just studied, without making use of the fact that y ∈ {0, 1}. (Essentially assume y are simply real numbers). There are several problem with this approach: first, the regression assumes the data supports a linear fit, which might not be true anymore for classification problems; second, our regressor h(x) will take lots of undesirable values (like the ones far outside the interval [0,1]).

To make an explicit mapping between the real valued regressor h and the set {0,1}, we would like a function that preserves differentiability and has a easy interpretable meaning. We choose the logistic function, also called sigmoid

g(z) = \frac{1}{1 + e^{−z}}

Figure 1: Logistic function

Note that g acts like a indicator for {0,1}, but it is much more sensitive than a linear function. We can make it even more sensitive by, for example, doubling the input z before applying the function. Lets state the derivative of g, since we are going to use it later on

g′(z) = \frac{∂g(z)}{∂z}
= \frac{1}{(1 + e^{−z})^{2}} e^{−z}
= \frac{1}{1 + e^{−z}} (1 − \frac{1}{1 + e^{−z}} )
= g(z)(1 − g(z))

4.2 Logistic regression

We apply g to the linear regression function to obtain a logistic regression. Our new hyphothesis (or predictor, or regressor) becomes

h_{w}(x) = g(wx) = \frac{1}{1 + e^{−wx}} = \frac{1}{1 + e^{− ∑_{d} w^{d}x^{d}}}