Feature Importance (optional) | BANA409 — Machine Learning for Business

Feature importance scores of the features reflect how useful they are at predicting the target variable.

Feature importance scores are important, because they can

provide insight into the data and model
form the basis for dimensionality reduction and feature selection that can improve the efficiency and effectiveness of a predictive model

There are various ways of calculating feature importance scores for different models.

Decision tree models offers importance scores based on the reduction in the criterion used to split nodes, like gini or entropy.

In this lecture, we learn how to get the importance score from a decision tree model with full pipeline:

import pandas as pd
import matplotlib.pyplot as plt
plt.style.use('ggplot')
from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.compose import ColumnTransformer
from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import GridSearchCV

# load data titanic data
df = pd.read_csv('train.csv')

# remove unimportant features
X = df.drop(['Survived', 'PassengerId', 'Name', 'Ticket', 'Cabin'], axis=1) 
y = df['Survived']

# train-test split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# separate numerical and categorical features 7 in total
num_features = ['Age', 'SibSp', 'Fare', 'Parch']
cat_features = ['Sex', 'Embarked', 'Pclass']

# numerical feature pipeline
num_pipeline = Pipeline(
    steps=[
        ('num_imputer', SimpleImputer()),
        ('scaler', StandardScaler()),
        ]
)

# categorical feature pipeline
cat_pipeline = Pipeline(
    steps=[
        ('cat_imputer', SimpleImputer()),
        ('onehot', OneHotEncoder()),
    ]
)

# Assign features to the pipelines and Combine two pipelines to form the preprocessor
preprocessor = ColumnTransformer(
    transformers=[
        ('num_pipeline', num_pipeline, num_features),
        ('cat_pipeline', cat_pipeline, cat_features),
    ]
)

# make a full pipeline by combining preprocessor and the model
tree_final_pipeline = Pipeline(
    steps=[
        ('preprocessor', preprocessor),
        ('tree_clf', DecisionTreeClassifier()),
    ]
)

# set up hyperparameters we want to tune
param_grid = [
    {
        'preprocessor__num_pipeline__num_imputer__strategy': ['mean', 'median'],
        'preprocessor__cat_pipeline__cat_imputer__strategy': ['most_frequent'],
        'tree_clf__criterion': ['gini', 'entropy'], 
        'tree_clf__max_depth': [3, 5, 7, 9, 11],
    }
]

# set up the grid search 
grid_search = GridSearchCV(tree_final_pipeline, param_grid, cv=10, scoring='accuracy')

# train the model using the full pipeline
grid_search.fit(X_train, y_train)

after running the code above, we can get the best parameters, model, and the test score:

# check the best performing parameter combination
grid_search.best_params_

# select the best model
tree_clf_best = grid_search.best_estimator_

# best test score
grid_search.cv_results_['mean_test_score'].max()

Output:
0.8258411580594679

We can print out the feature importance scores (why 7 features but 12 feature importance scores?):

# feature importance: why 7 features 12 score?
tree_clf_best['tree_clf'].feature_importances_

Output:

array([0.08110601, 0.05513595, 0.0608736 , 0.        , 0.        ,
       0.54533576, 0.        , 0.        , 0.        , 0.05311821,
       0.        , 0.20443047])

Answer: it’s because the one-hot encoding! We need to find the corresponding feature names after one-hot encoding.

# get the step names, which is a dict
tree_clf_best.named_steps

Output:

{'preprocessor': ColumnTransformer(transformers=[('num_pipeline',
                                  Pipeline(steps=[('num_imputer',
                                                   SimpleImputer()),
                                                  ('scaler', StandardScaler())]),
                                  ['Age', 'SibSp', 'Fare', 'Parch']),
                                 ('cat_pipeline',
                                  Pipeline(steps=[('cat_imputer',
                                                   SimpleImputer(strategy='most_frequent')),
                                                  ('onehot', OneHotEncoder())]),
                                  ['Sex', 'Embarked', 'Pclass'])]),
 'tree_clf': DecisionTreeClassifier(criterion='entropy', max_depth=3)}

Then, we can choose the numerical features as follows:

# the following returns a list with two tuples
# one for num_pipeline, one for cat_pipeline
tree_clf_best['preprocessor'].transformers_  

Output:

[('num_pipeline',
  Pipeline(steps=[('num_imputer', SimpleImputer()), ('scaler', StandardScaler())]),
  ['Age', 'SibSp', 'Fare', 'Parch']),
 ('cat_pipeline',
  Pipeline(steps=[('cat_imputer', SimpleImputer(strategy='most_frequent')),
                  ('onehot', OneHotEncoder())]),
  ['Sex', 'Embarked', 'Pclass'])]

# this is a tuple with three elements
# ('name of the pipeline', the pipeline, the feature names pipeline is appied to)
tree_clf_best['preprocessor'].transformers_[1]  

('cat_pipeline',
 Pipeline(steps=[('cat_imputer', SimpleImputer(strategy='most_frequent')),
                 ('onehot', OneHotEncoder())]),
 ['Sex', 'Embarked', 'Pclass'])

So we can select the fitted cat_pipeline as follows:

# select the fitted cat pipeline
fitted_cat_pipeline = tree_clf_best['preprocessor'].transformers_[1][1]

# get the one-hot features as a numpy array
one_hot_features = fitted_cat_pipeline['onehot'].get_feature_names_out(cat_features)
one_hot_features

Output:

array(['Sex_female', 'Sex_male', 'Embarked_C', 'Embarked_Q', 'Embarked_S',
       'Pclass_1', 'Pclass_2', 'Pclass_3'], dtype=object)

Now we can get all feature names as follows (12 features in total):

# convert one hot features from a numpy array to a list
feature_names = num_features + list(one_hot_features)
feature_names

Output:

['Age',
 'SibSp',
 'Fare',
 'Parch',
 'Sex_female',
 'Sex_male',
 'Embarked_C',
 'Embarked_Q',
 'Embarked_S',
 'Pclass_1',
 'Pclass_2',
 'Pclass_3']

We get the importance scores:

# get importance score and generate a df 
importance_scores = tree_clf_best['tree_clf'].feature_importances_

# generate a dataframe
df_importance_score = pd.DataFrame(importance_scores, index=feature_names, columns=['importance'])

# sort the dataframe
df_importance_score.sort_values('importance', ascending=False, inplace=True)

df_importance_score

Screen Shot 2022-04-18 at 10.20.24 PM.png

Plot the importance scores:

# plot the feature importance scores
df_importance_score.plot.bar()

From these scores, we know:

the most important feature is Sex
features Parch and Embarked are not useful for the prediction and can be removed to simplify our mode

To verify, run the following code:

# remove the unimportant features and see the test score and compared with 0.8258

# load data titanic data
df = pd.read_csv('train.csv')

# remove unimportant features
X = df.drop(['Survived', 'PassengerId', 'Name', 'Ticket', 'Cabin'], axis=1)
y = df['Survived']

# train-test split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# separate numerical and categorical features
num_features = ['Age', 'SibSp', 'Fare']
cat_features = ['Sex', 'Pclass']

# numerical feature pipeline
num_pipeline = Pipeline(
    steps=[
        ('num_imputer', SimpleImputer()),
        ('scaler', StandardScaler()),
        ]
)

# categorical feature pipeline
cat_pipeline = Pipeline(
    steps=[
        ('cat_imputer', SimpleImputer()),
        ('onehot', OneHotEncoder()),
    ]
)

# Assign features to the pipelines and Combine two pipelines to form the preprocessor
preprocessor = ColumnTransformer(
    transformers=[
        ('num_pipeline', num_pipeline, num_features),
        ('cat_pipeline', cat_pipeline, cat_features),
    ]
)

# make a full pipeline by combining preprocessor and the model
tree_final_pipeline = Pipeline(
    steps=[
        ('preprocessor', preprocessor),
        ('tree_clf', DecisionTreeClassifier()),
    ]
)

# set up hyperparameters we want to tune
param_grid = [
    {
        'preprocessor__num_pipeline__num_imputer__strategy': ['mean', 'median'],
        'preprocessor__cat_pipeline__cat_imputer__strategy': ['most_frequent'],
        'tree_clf__criterion': ['gini', 'entropy'], 
        'tree_clf__max_depth': [3, 5, 7, 9, 11],
    }
]

# set up the grid search 
grid_search = GridSearchCV(tree_final_pipeline, param_grid, cv=10, scoring='accuracy')

# train the model using the full pipeline
grid_search.fit(X_train, y_train)

# get the best test score
print('the best accuracy with all features is 0.8258')
print(f"the best accuracy after feature selection is {grid_search.cv_results_['mean_test_score'].max():.4f}")