Full Pipeline
In this lecture, we focus on developing the full automatic pipeline with grid search and cross validation to train, evaluate, and build machine learning models, which is depicted in the following flowchart:
The key steps are:
- drop the unimportant features, such as
df_train.drop(['customerID'], axis=1, inplace=True)- make sure the feature data types are correct, e.g., you can use the following statement to change object/string to numbers
# errors='coerce' means invalid parsing will be set as NaN
df['TotalCharges']=pd.to_numeric(df['TotalCharges'], errors='coerce')- Data preprocessing with row removal: if you need to remove any rows such as few rows with missing values, you need to do them BEFORE splitting the X and y in the next step.
- Separate features and target and split train and test dataset, such as
X = df.drop(['Survived'], axis=1)
y = df['Survived']
# Split the data into a training set and a test set.
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
- separate the features into numerical and categorical groups:
# select sub dataframe based on data type
df_num = X.select_dtypes(exclude ='object')
df_cat = X.select_dtypes(include ='object')
# we need the column names as lists
num_features = df_num.columns.tolist()
cat_features = df_cat.columns.tolist()- First develop two pipelines: one for numerical features and one for categorical features
- numerical feature pipeline
- step 1: impute the missing value (we will try mean and median)
- step 2: conduct standardization using z-score (default)
- categorical feature pipeline:
- step 1: impute the missing value using the most frequent value
- step 2: one-hot encode the features
- numerical feature pipeline
- Combine the two pipelines to form our preprocessor.
- Attach the model to the preprocessor to form the full pipeline
- Use GridSearch with the full pipeline to find the best model and test on the test set
- (Optional) if for Kaggle competitions, use the trained model with full pipeline to make predictions
We first prepare the training and testing datasets as we did before:
# load the package
import pandas as pd
# read titanic data
df = pd.read_csv('titanic.csv')
# Prepare the data by separating X and y and dropping unimportant features
# Only 7 features used: 'Age', 'SibSp', 'Fare', 'Parch', 'Sex', 'Embarked', 'Pclass'
X = df.drop(['Survived', 'PassengerId', 'Name', 'Ticket', 'Cabin'], axis=1)
y = df['Survived']
# Split the data into a training set and a test set.
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
Next, we divide the features into numerical and categorical groups: different types of features will go through different pre-processing steps.
# Numerical Features: ['Age', 'SibSp', 'Fare', 'Parch']
# Categorical Features:['Sex', 'Embarked', 'Pclass'
num_features = ['Age', 'SibSp', 'Fare', 'Parch']
cat_features = ['Sex', 'Embarked', 'Pclass']Build the preprocessing pipeline for numerical and categorial features respectively using Pipeline
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler, OneHotEncoder
# NOTE the step names can be arbitrary
# Create the preprocessing pipeline for numerical features
# Step 1 filling the missing values if any using mean
# Step 2 is standardization using the z-score
num_pipeline = Pipeline(
steps=[
('num_imputer', SimpleImputer()),
('scaler', StandardScaler()),
]
)
# Create the preprocessing pipelines for the categorical features
# Step 1: filling the missing values if any using the most frequent value
# Step 2: one hot encoding
cat_pipeline = Pipeline(
steps=[
('cat_imputer', SimpleImputer()),
('onehot', OneHotEncoder()),
]
)Combine the two pipelines into the preprocessor for all features:
# Assign features to the pipelines and
# Combine two pipelines to form the preprocessor
from sklearn.compose import ColumnTransformer
preprocessor = ColumnTransformer(
transformers=[
('num_pipeline', num_pipeline, num_features), # pipeline name, pipeline, features to process
('cat_pipeline', cat_pipeline, cat_features), # pipeline name, pipeline, features to process
]
)Next, we attach the model to the preprocess to form the complete pipeline:
# Specify the model to use, which is DecisionTreeClassifier in this example
# Make a full pipeline by combining preprocessor and the model
from sklearn.tree import DecisionTreeClassifier
# final decision tree (dt) pipeline
dt_pipeline = Pipeline(
steps=[
('preprocessor', preprocessor),
('tree_clf', DecisionTreeClassifier()),
]
)Setup the Grid Search with the full pipeline.
IMPORTANT: You must use the step names as the prefix followed by
two under_scores (not one)
to specify the parameter names with a “full path”
For example, if we want to refer to the strategy parameter for the SimpleImputer() function, the “full path” of the step names is as follows:
preprocessor —> num_pipeline —> num_imputer —> strategy
Therefore, we can specify that we are going to try two different strategies mean vs. median by linking the steps names with two under_scores as follows:
'preprocessor__num_pipeline__num_imputer__strategy': ['mean', 'median']See the whole grid search setup in the following.
Note that 2 (num_imputer__strategy) x 2 (criterion) x 5 (max_depth) = 20 models with different parameter settings are trained and tested in the Grid Search.
# GridSearch with 10-fold cross validation and accuracy as the metric
from sklearn.model_selection import GridSearchCV
# set up the values of hyperparameters you want to evaluate
# IMPORTANT!!!!!!!
# here you must use the step names as the prefix followed by two under_scores to specify the parameter names
# you also need to specify the "full path" of the steps
param_grid = [
{
'preprocessor__num_pipeline__num_imputer__strategy': ['mean', 'median'],
'preprocessor__cat_pipeline__cat_imputer__strategy': ['most_frequent'], # only one choice for this parameter
'tree_clf__criterion': ['gini', 'entropy'],
'tree_clf__max_depth': [3, 7, 10, 12, 15],
}
]
# set up the grid search
grid_search = GridSearchCV(dt_pipeline, param_grid, cv=10, scoring='accuracy')Now, we are ready to train the model with Grid Search and find the best performing parameters for the decision tree.
# train the model using the full pipeline
grid_search.fit(X_train, y_train)
# check the best performing parameter combination
grid_search.best_params_
{'preprocessor__cat_pipeline__cat_imputer__strategy': 'most_frequent',
'preprocessor__num_pipeline__num_imputer__strategy': 'mean',
'tree_clf__criterion': 'entropy',
'tree_clf__max_depth': 3}
# check the mean test score - see 20 scores
grid_search.cv_results_['mean_test_score']
array([0.82443271, 0.81179577, 0.79911972, 0.78658059, 0.77679969,
0.82584116, 0.80338419, 0.79628326, 0.78941706, 0.7711072 ,
0.82443271, 0.81318466, 0.80616197, 0.78798905, 0.76549296,
0.82584116, 0.80479264, 0.79348592, 0.79225352, 0.77537167])
# overall average accuracy score
grid_search.cv_results_['mean_test_score'].mean()
0.7986883802816902
# select the best model
tree_clf_best = grid_search.best_estimator_Next, we test the best model on the test set:
# final evaluation using the test data
y_pred = tree_clf_best.predict(X_test)
# calculate accuracy, precision, recall, f1-score
# y_test is the ground truth, y_pred is our model's prediction
from sklearn.metrics import accuracy_score, recall_score, precision_score, f1_score
print(f'Accuracy Score : {accuracy_score(y_test, y_pred)}')
print(f'Precision Score : {precision_score(y_test, y_pred)}')
print(f'Recall Score : {recall_score(y_test, y_pred)}')
print(f'F1 Score : {f1_score(y_test, y_pred)}')
Accuracy Score : 0.7988826815642458
Precision Score : 0.796875
Recall Score : 0.6891891891891891
F1 Score : 0.7391304347826088If we want to use this trained model to make a submission to Kaggle, you can do the following.
titanic_test = pd.read_csv('titanic_test.csv')
titanic_test.info()
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 418 entries, 0 to 417
Data columns (total 11 columns):
# Column Non-Null Count Dtype
--- ------ -------------- -----
0 PassengerId 418 non-null int64
1 Pclass 418 non-null int64
2 Name 418 non-null object
3 Sex 418 non-null object
4 Age 332 non-null float64
5 SibSp 418 non-null int64
6 Parch 418 non-null int64
7 Ticket 418 non-null object
8 Fare 417 non-null float64
9 Cabin 91 non-null object
10 Embarked 418 non-null object
dtypes: float64(2), int64(4), object(5)
memory usage: 36.0+ KB
X_train.info()
<class 'pandas.core.frame.DataFrame'>
Int64Index: 712 entries, 331 to 102
Data columns (total 7 columns):
# Column Non-Null Count Dtype
--- ------ -------------- -----
0 Pclass 712 non-null int64
1 Sex 712 non-null object
2 Age 572 non-null float64
3 SibSp 712 non-null int64
4 Parch 712 non-null int64
5 Fare 712 non-null float64
6 Embarked 710 non-null object
dtypes: float64(2), int64(3), object(2)
memory usage: 44.5+ KBWe have to make sure that titanic_test includes all columns that X_train has, which is the case. The required columns will be automatically chosen by the pipeline.
NOTE that we can directly use tree_clf_best and all preprocessing will be taken care of automatically.
# we have to make sure that titanic_test includes all columns that X_train has
y_pred_titanic = tree_clf_best.predict(titanic_test)Finally, we can generate the csv for submission:
# combine id and prediction for kaggle submission
dt_pipeline_submit = pd.DataFrame({
'PassengerId': titanic_test['PassengerId'],
'Survived': y_pred_titanic
})
# generate the csv
dt_pipeline_submit.to_csv('dt-pipeline-submit.csv', index=False)
print('csv saved! please submit the prediction csv to Kaggle.com')