Source
Adapted from my own handwritten and hand-typed notes from Year 1 to Year 3 of my EEE degree
Univariate linear regression predicts one numeric variable from one other variable using a straight line. Example: predict a Pokemon's Total from its HP (the two are clearly positively correlated, see Bivariate Exploration and Correlation).
Model
Where:
- is the slope and the intercept, the parameters to learn
- is the error that the line cannot explain
Cost Function
The line is fitted by minimising the residual sum of squares (RSS) over the train data:
Each term is (actual minus predicted) squared.
Steps in Linear Regression
The algorithm:
- Guess parameters and
- Predict
Totalfor every point in the train data - Calculate the errors compared to the actual values
- Choose a cost function (like the sum of squared errors)
- Tune and to minimise it
For the Pokemon train set the result was , and the same line is then used to predict Total on the test set.
Goodness of Fit
Mean squared error, lower is better:
Explained variance , higher is better, with :
is the fraction of the variation in Total that the line explains. Comparing train and test values shows whether the model generalises.
Code
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error
linreg = LinearRegression()
linreg.fit(X_train, y_train)
print(linreg.intercept_, linreg.coef_) # b, a
y_test_pred = linreg.predict(X_test)
print('R^2 test:', linreg.score(X_test, y_test))
print('MSE test:', mean_squared_error(y_test, y_test_pred))