Source

Adapted from my own handwritten and hand-typed notes from Year 1 to Year 3 of my EEE degree

Univariate linear regression predicts one numeric variable from one other variable using a straight line. Example: predict a Pokemon's Total from its HP (the two are clearly positively correlated, see Bivariate Exploration and Correlation).

Model

Where:

  • is the slope and the intercept, the parameters to learn
  • is the error that the line cannot explain

Cost Function

The line is fitted by minimising the residual sum of squares (RSS) over the train data:

Each term is (actual minus predicted) squared.

Steps in Linear Regression

The algorithm:

  1. Guess parameters and
  2. Predict Total for every point in the train data
  3. Calculate the errors compared to the actual values
  4. Choose a cost function (like the sum of squared errors)
  5. Tune and to minimise it

For the Pokemon train set the result was , and the same line is then used to predict Total on the test set.

Goodness of Fit

Mean squared error, lower is better:

Explained variance , higher is better, with :

is the fraction of the variation in Total that the line explains. Comparing train and test values shows whether the model generalises.

Code

from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error
 
linreg = LinearRegression()
linreg.fit(X_train, y_train)
print(linreg.intercept_, linreg.coef_)     # b, a
 
y_test_pred = linreg.predict(X_test)
print('R^2 test:', linreg.score(X_test, y_test))
print('MSE test:', mean_squared_error(y_test, y_test_pred))