X^{1} X^{2} 2$ (x_{t}, y_{t}) x_{t} =

(x_{t}^{1}, x_{t}^{2}) y_{t} regression y_{t} x_{t} linear regression h_{w}(x) = w^{0}+w^{1}x^{1}+w^{2}x^{2}

w = (w^{0}, w^{1}, w^{2}) w w h x^{0} = 1

h(x) = ∑_{d=0}^{D} w^{d}x^{d}

d D D = 2 squared error

J(w) = ∑_{t}(h_{w}(x_{t})−y_{t})^{2}

w J least square regression least square fit

J J w ∂J/∂w w

w^{j} = w^{j}−λ\frac{∂J(w)}{∂w^{j}}

λ learning rate J(x) = (x− 2)^{2}+

1 x_{0} J

\frac{∂J(x)}{∂x} = 2x−4

λ = 0.25

x_{1} = x_{0} − λ\frac{∂J(x_{0})}{∂x} = 3 − .25(2 ∗ 3 − 4) = 2.5

x_{2} = x_{1} − λ\frac{∂J(x_{1})}{∂x} = 2.5 − .25(2 ∗ 2.5 − 4) = 2.25

x_{3} = x_{2} − λ\frac{∂J(x_{2})}{∂x} = 2.25 − .25(2 ∗ 2.25 − 4) = 2.125

?? descent J

J(w) = \frac{1}{2} ∑_{t}(h_{w}(x_{t})−y_{t})^{2}

1/2 w

\frac{∂J(w)}{∂w^{j}}=\frac{∂ \frac{1}{2}(h_{w}(x) − y)^{2}}{∂w^{j}}

=(h_{w}(x) − y)\frac{∂(h_{w}(x) − y)}{∂w^{j}}

=(h_{w}(x) − y)\frac{∂(∑_{j} w^{j}x^{j} − y)}{∂w^{j}}
=(h_{w}(x) − y)x^{j}

w^{j} = w^{j}−λ(h_{w}(x)−y)x^{j}

batch gradient descent J J

w^{j} = w^{j}−λ ∑_{t}(h_{w}(x_{t})−y_{t})x_{t}^{j}

stochastic gradient decent w^{j} = w^{j}−λ(h_{w}(x_{t})−y_{t})x_{t}^{j}forallj