Boston数据集上的梯度下降

Question

我正在尝试从头开始实现梯度下降算法，并将其用于Boston数据集。这是我到目前为止的内容：

import numpy as np
from sklearn.datasets import load_boston
import matplotlib.pyplot as plt
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error


dataset = load_boston()
X = dataset.data
y = dataset.targe

def gradient_descent(X, y, eta = 0.0001, n_iter = 1000):
    '''
    Gradient descent implementation
    eta: Learning Rate
    n_iter: Number of iteration
    '''
    b0 = b1 = 1
    costList = []
    m = len(y)
    for i in range(n_iter):
        error = y - (b0 + X*b1)
        #Updating b0 and b1
        b0 += -eta * (-2 * error.sum() / m )
        b1 += -eta * (-2 * X.dot(error).sum() / m )
        costList.append(mean_squared_error(y, (b0 + b1*X)))
    return costList


listt = gradient_descent(X, y)

但是，当我尝试计算错误时，出现以下错误：operands could not be broadcast together with shapes (506,) (506,13)其中y的形状为(506,) and shape of y_predict is (506, 13) I

发生此错误是因为y_predict的尺寸为2，而y为一维。但是，我不知道如何克服这个问题。

GD仅在使用整个数据集的一个功能时有效；但是，我想全部使用它们

Answer 1

您需要使用multivariate linear regression而不是单变量线性回归，因为线性使用的是X连续有多个值。

因此，b1应该是向量，而不是标量。

我刚刚将您的代码转换为多元线性回归。但是它很容易发散（b1会很大）。因此，也许您应该考虑使用高级线性回归，例如ridge regression。

import numpy as np
from sklearn.datasets import load_boston
import matplotlib.pyplot as plt
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error


dataset = load_boston()
X = dataset.data
y = np.expand_dims(dataset.target,axis=1) # for maxtrix mul

#new_X = (X-X.mean())/X.std()

def gradient_descent(X, y, eta = 0.000001, n_iter = 1000):
    '''
    Gradient descent implementation
    eta: Learning Rate
    n_iter: Number of iteration
    '''
    b0 = 1
    b1 = np.ones((13,1)) # b1 should be vector
    costList = []
    m = len(y)
    for _ in range(n_iter):
        # error = y - (b0 + X.dot(b1)) # it is wrong
        error = (b0 + X.dot(b1)) - y

        #Updating b0 and b1
        b0 += -eta * (error.sum() / m )
        b1 += -eta * (X.T.dot(error) / m )
        costList.append(mean_squared_error(y, (b0 + X.dot(b1))))
    return costList


listt = gradient_descent(X, y, eta=10e-5,n_iter=10)