4.多元线性回归

人工智能60

线性模型假定预测(\hat{y})是对应(x={x_1,x_2,\cdots,x_p})的属性的线性组合,即:

[\begin{align} \hat{y} &=\theta_0+\theta_1 x_1+\theta_2 x_2+ \cdots + \theta_n x_p\ & =\theta_0 x_0+\theta_1 x_1+\theta_2 x_2+ \cdots + \theta_n x_n \qquad 其中 x_0 \equiv 1 \end{align} \tag{1} ]

令 (\theta={\theta_0,\theta_1,\theta_2,\cdots,\theta_n}^T,x_b={x_0,x_1,\cdots,x_n}),则:

[\hat{y} =x_b \theta \tag{2} ]

现有数据(X={x^{(i)},x^{(2)},\cdots,x^{(N)}}^T),对应(y={y^{(1)},y^{(2)},\cdots,y^{(N)}}^T),其中(x^{(i)}={x_1^{(i)},x_2^{(i)},\cdots,x_p^{(i)}})。对了方便

记(X_b={x_b^{(i)},x_b^{(2)},\cdots,x_b^{(N)}}^T)。

定义损失函数:

[\begin{align} J(\theta)&=\sum_{i=1}^N (y^{(i)}-x_b^{(i)}\theta)^2 \ &=(y-X_b\theta)^T(y-X_b\theta) \end{align} \tag{3} ]

令(\frac{dJ(\theta)}{d\theta}=2X_b^T(X_b\theta-y)=0),得:

[X_b^TX_b\theta=X_b^Ty \tag{4} ]

假设(X^TX)可逆,则:

[\hat{\theta}=(X_b^TX_b)^{-1}X_b^Ty \tag{5} ]

相应的:

[\hat{y} =X_b(X_b^TX_b)^{-1}X_b^Ty \tag{6} ]

在以上求解过程中,如果:

针对以上问题,可以考虑使用梯度下降法来求参数,下面代码演示:

import numpy as np
def get_data(theta_true,N):
    X=np.random.normal(size=(N,len(theta_true)))
    Y=X@theta_true+np.random.normal(size=(N))
    return (X,Y)
def loss(X,Y,theta):
    return np.sum(X@theta-Y)
def loss_dtheta(X,Y,theta):
    return X.T@(X@theta-Y)
def SGDLinerModel(X,Y,theta_init,lr=0.01,error=1e-11,maxIter=10000):
    theta=theta_init
    for iter in range(0,maxIter):
        dtheta=loss_dtheta(X,Y,theta)
        new_theta=theta-lr*dtheta
        if(np.abs(loss(X,Y,theta)-loss(X,Y,new_theta))

Original: https://www.cnblogs.com/bestwangyulu/p/16390656.html
Author: Wang-YL
Title: 4.多元线性回归