最小二乘法与机器学习正规方程

每个样本都有误差ei=yi^−yi​e_i = \hat{y_i} - y_i​ei​=yi​^​−yi​​，如果直接把误差相加，会正负互相抵消，所以把误差平方：
Loss:J(θ)=∑i(yi^−yi​)2 Loss:J(\theta) = \sum_i(\hat{y_i} - y_i​)^2Loss:J(θ)=i∑​(yi​^​−yi​​)2
在机器学习中，假设有mmm个样本，把所有输入写成矩阵：X=[1x11x2⋮⋮1xm]X = \begin{bmatrix} 1 & x_1 \\ 1 & x_2 \\ \vdots & \vdots \\ 1 & x_m \end{bmatrix}X=​11⋮1​x1​x2​⋮xm​​​
参数：θ=[θ0θ1]\theta = \begin{bmatrix} \theta_0 \\ \theta_1 \end{bmatrix}θ=[θ0​θ1​​]，真实值：y=[y1y2⋮ym]y = \begin{bmatrix} y_1 \\ y_2 \\ \vdots \\ y_m \end{bmatrix}y=​y1​y2​⋮ym​​​
预测值：y^=θ0+θ1x=Xθ\hat{y} = \theta_0 + \theta_1 x = X \thetay^​=θ0​+θ1​x=Xθ
误差：ei=yi^−yi​=Xθ−ye_i = \hat{y_i} - y_i​ = X\theta - yei​=yi​^​−yi​​=Xθ−y
Loss：J(θ)=(Xθ−y)⊺(Xθ−y)=θ⊺X⊺Xθ−2y⊺Xθ+y⊺yJ(\theta) = (X\theta - y)^\intercal (X\theta - y) = \theta^\intercal X^\intercal X \theta - 2y^\intercal X \theta + y^\intercal yJ(θ)=(Xθ−y)⊺(Xθ−y)=θ⊺X⊺Xθ−2y⊺Xθ+y⊺y，其中y⊺yy^\intercal yy⊺y和θ\thetaθ没有关系，求导后消失
对θ\thetaθ求导：∂J∂θ=2X⊺Xθ−2X⊺y=0\frac{\partial J}{\partial \theta} = 2 X^\intercal X \theta - 2 X^\intercal y = 0∂θ∂J​=2X⊺Xθ−2X⊺y=0
正规方程：X⊺Xθ=X⊺yX^\intercal X \theta = X^\intercal yX⊺Xθ=X⊺y、θ=(X⊺X)−1X⊺y\theta = (X^\intercal X)^{-1} X^\intercal yθ=(X⊺X)−1X⊺y