Model Tuning via Grid Search | BANA409 — Machine Learning for Business

Recall that we manually created three decision models with different parameters in the model evaluation lecture:

from sklearn.tree import DecisionTreeClassifier

tree_clf1 = DecisionTreeClassifier(criterion='entropy')
tree_clf2 = DecisionTreeClassifier(criterion='entropy', max_depth=3)
tree_clf3 = DecisionTreeClassifier(criterion='gini', max_depth=6)

max_depth can be any number, e.g., 4, 5, 7, 9, 15...

criterion can be entropy, gini, logloss

Question: Which combination of those parameters can give better performance?

Answer: We don’t know the optimal combination and just need to try as much as we can as time and computing resources permit, which is essentially a grid search.

For example, we want to try max_depth being 3, 5, 7, 10, 15 and criterion being entropy or gini, then we will have 5 x2 = 10 combinations to try - you don’t want to manually create 10 models as shown above - let’s use Grid Search for this.

The process of trying different values of the parameters is often called (hyper)parameter tuning.

The following basic steps are the same as before:

# load packages
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
plt.style.use('default')

# read in titanic dataset
df = pd.read_csv('titanic.csv')

# In order to avoid any pre-processing steps to keep this simple
# we only choose three numerical features with no data issues (SibSp, Parch and Fare) to train the model
X = df[['SibSp', 'Parch', 'Fare']]
y = df['Survived']

# Split the data into a training set and a test set. 
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

Now, let’s setup the Grid Search with 5-fold CV:

from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import GridSearchCV

# initialize the decision tree without any parameters
tree_clf = DecisionTreeClassifier()

# set up the values of hyperparameters you want to evaluate
param_grid = [
    {
        'criterion': ['gini', 'entropy'], 
        'max_depth': [3, 5, 7, 10, 15],
    }
]

# set up the grid search with 5-fold cross validation using accuracy as the metric
grid_search = GridSearchCV(tree_clf, param_grid, cv=5, scoring='accuracy')

# the following will try 10 parameter combinations with 5-fold cross validation
# grid search may take a long time to complete if many combinations are explored
grid_search.fit(X_train, y_train)

Then, you can check the best performing parameter combination:

# check the best performing parameter combination
grid_search.best_params_

{'criterion': 'gini', 'max_depth': 5}

the cross validation results are stored in the following variable, which is a dictionary with many keys

# show the evaluation result details
grid_search.cv_results_.keys()

dict_keys(['mean_fit_time', 'std_fit_time', 'mean_score_time', 'std_score_time', 'param_criterion', 'param_max_depth', 'params', 'split0_test_score', 'split1_test_score', 'split2_test_score', 'split3_test_score', 'split4_test_score', 'mean_test_score', 'std_test_score', 'rank_test_score'])

we often need the best mean_test_score

# show the best mean test score
grid_search.cv_results_['mean_test_score'].max()

0.6994090416625628

This means that grid search tells us decision tree model with {'criterion': 'gini', 'max_depth': 5} parameter combination has the best accuracy of 0.6994090416625628

Grid search once finds the best parameters, it also retrain the model with the best parameters using ALL training data and store the final model in .best_estimator_

# get the best final model
tree_clf_best = grid_search.best_estimator_

the rest would be the same as before: test the model using the testing data and evaluate the performance:

# get the prediction results from the testing set
y_pred = tree_clf_best.predict(X_test)

# calculate accuracy, precision, recall, f1-score
from sklearn.metrics import accuracy_score, recall_score, precision_score, f1_score

print(f'Accuracy Score : {accuracy_score(y_test,y_pred)}')
print(f'Precision Score : {precision_score(y_test,y_pred)}')
print(f'Recall Score : {recall_score(y_test,y_pred)}')
print(f'F1 Score : {f1_score(y_test,y_pred)}')