f(x) = 7.5 sin(2.5πx) t = f(x) + ϵ

argmin_{w∈ℝ^{m}} ∑_{i=1}^{N}(t_{i} − w^{T} x_{i})^{2}

subject to: ∑_{j=1}^{m} w_{i}^{2} ≤ c

w_{ridge} = argmin_{w∈ℝ^{m}} J(w) = ∑_{i=1}^{N}(t_{i} − w^{T} x_{i})^{2} + λ ∑_{j=1}^{m} w_{i}^{2}

∑_{j=1}^{m} w_{i}^{2} = w^{T} w = ∥w∥_{2}^{2}

w_{ridge} = argmin_{w∈ℝ^{m}} J(w) = ∑_{i=1}^{N}(t_{i} − w^{T} x_{i})^{2} + λ∥w∥_{2}^{2}

∇_{w}J(w) = ∇_{w}E^{T} E = ∇_{w}(Zw − t)^{T} (Zw − t) + ∇_{w}λw^{T} w
= ∇_{w}(w^{T} Z^{T} Zw − w^{T} Z^{T} t − t^{T} Zw + t^{T} t) + λ∇_{w}w^{T} w
= ∇_{w}(w^{T} Z^{T} Zw − w^{T} Z^{T} t − w^{T} Z^{T} t + t^{T} t) + λ∇_{w}w^{T} w
= ∇_{w}(w^{T} Z^{T} Zw − 2w^{T} Z^{T} t + t^{T} t) + λ∇_{w}w^{T} w
= 2(Z^{T} Zw − Z^{T} t) + 2λw
= 2Z^{T} Zw + 2λIw − 2Z^{T} t
= 2(Z^{T} Z + λI)w − 2Z^{T} t

(Z^{T} Z + λI)w = Z^{T} t or w_{ridge} = (Z^{T} Z + λI)^{−1}Z^{T} t

1 / N

ˆ w ŵ_{reg} = argmin_{w∈ℝ^{m}} Loss(x, w) + ∥w∥_{q}

Loss(x, w) = ∑_{i=1}^{N}(t_{i} − w_{0} − ∑_{j=1}^{m} w_{j}x_{ij})^{2}

D