OLS Regression and R-squared (optional)

This page continues from Interpreting a Linear Regression Model: it uses the same housing train/test split, and y_pred_housing, the trained model's predictions on the test set.

OLS with statsmodels

scikit-learn focuses on prediction. The statsmodels package fits the same model, called OLS (Ordinary Least Squares) in statistics, and reports more about the weights from a statistics perspective, such as p-values and confidence intervals:

import statsmodels.api as sm

# add_constant adds a column of 1s for the bias, just like the normal equation
X_train_const = sm.add_constant(X_train_housing)
ols_results = sm.OLS(y_train_housing, X_train_const).fit()
print(ols_results.summary())

Output:
                            OLS Regression Results
==============================================================================
Dep. Variable:     median_house_value   R-squared:                       0.514
Model:                            OLS   Adj. R-squared:                  0.513
Method:                 Least Squares   F-statistic:                     5809.
No. Observations:               16512   AIC:                         4.200e+05
Df Residuals:                   16508   BIC:                         4.200e+05
Df Model:                           3
======================================================================================
                         coef    std err          t      P>|t|      [0.025      0.975]
--------------------------------------------------------------------------------------
const              -1.777e+04   2505.219     -7.093      0.000   -2.27e+04   -1.29e+04
population             2.9753      0.578      5.152      0.000       1.843       4.107
median_income        4.34e+04    332.242    130.624      0.000    4.27e+04    4.41e+04
housing_median_age  1828.0789     52.495     34.824      0.000    1725.183    1930.975
==============================================================================

The output above is shortened. How to read it:

• coef are the bias (const) and the weights, exactly the same as scikit-learn's intercept_ and coef_ for the model trained on the training set on the previous page (-17768, 2.975, 43399, 1828).
• P>|t| is the p-value: if a feature had no association with house value at all, how likely would we be to see a weight this far from 0? All three are 0.000, so all three associations are statistically significant. Significant does not mean large or important, and it does not mean the feature causes the price. With 16,512 districts, even a tiny association is significant.
• [0.025 0.975] is the 95% confidence interval for each weight. For example, the weight for population is 2.975, and we are 95% confident the true weight is between 1.843 and 4.107.

R-squared

The table's R-squared ($R^2$, the coefficient of determination) turns the comparison with the baseline into one number:

where $\bar{y}$ is the average of $y$. The top sum is the model's squared error; the bottom sum is the squared error of the baseline that predicts $\bar{y}$ for everyone. So $R^2$ is 1 minus the model's MSE as a fraction of the baseline's MSE:

• $R^2 = 1$: a perfect model, with no errors at all
• $R^2 = 0$: no better than predicting the average for everyone
• $R^2 < 0$: worse than predicting the average, which can happen on the test set

The table's 0.514 is measured on the training set, the data the model was fit on. On the test set, scikit-learn's r2_score gives a slightly lower value:

from sklearn.metrics import r2_score

print(f'R-squared on the test set is {r2_score(y_test_housing, y_pred_housing):.3f}')

Output:
R-squared on the test set is 0.495

You can check it from the RMSEs on the previous page: $1 - (81324 / 114486)^2 \approx 0.50$. People often say the model explains about half of the variation in house values. Adj. R-squared in the table adjusts R-squared down slightly for the number of features, because adding any feature, even a useless one, never lowers R-squared on the training set.

Unlike RMSE, $R^2$ has no unit, so you can compare it across datasets, but it does not tell you how large the errors are in dollars. Report both.

What counts as a good $R^2$? There is no universal cutoff: it depends on how predictable the target is. Rough rules of thumb:

Rather than asking whether $R^2$ is high, ask whether the model beats the alternatives (a simpler model, or what you use today), whether the test $R^2$ is close to the training $R^2$ (0.495 vs. 0.514 here, so no sign of overfitting), and whether the errors are small enough for the decision.

Be suspicious of a very high $R^2$ in a noisy domain. An $R^2$ of 0.99 on house prices usually means a bug, such as data leakage: a feature that secretly contains the answer.

For more on reading this table, see Interpreting Regression Output from Princeton University Library.