Model Evaluation In-Class Exercise

In the Heuristic Challenge you wrote rules by hand, and a good rule landed in the mid 0.70s on accuracy. Today you hand the same data to a decision tree and try to beat your rule — and you do it without Kaggle.

That is the point of the lecture's Evaluate without Kaggle? question: in a real project there is no leaderboard, so you have to be able to say how good a model is on your own, and be right. The Lakeside competition stays a rules-only leaderboard; nothing you build today gets submitted there.

Work in a notebook called model-evaluation-exercise.ipynb. The modeling choices are yours — which columns to use, how to clean them, which tree parameters to try. What is not up to you is how you evaluate. Two rules for the whole exercise:

1. The test set is locked. Set aside 20% of the data before you do anything else and look at it exactly once, in Part 3. Every decision before that — which features, which parameters — is made without it. If you peek and then change your model, your test score is no longer an honest estimate of anything — see How to Use the Test Set.
2. Every choice gets a reason. When you drop a column, fill a missing value, or pick a parameter, say why in a sentence. "It scored higher" counts only if you say which score.

The Data

You need train.csv from the competition's Data tab, or from the lakeside-booking-data.zip mirror; the columns are described on the Heuristic Challenge page. test.csv is not used today: it has no labels, so it cannot tell you how good your model is.

Before a tree can learn from this file you have to deal with two things, both covered in Data Preprocessing: several columns have missing values, and several are text categories a tree cannot split on directly. How you handle them is up to you.

The Split

Split train.csv three ways, as in Holdout Validation, using train_test_split twice with random_state=409:

• First split off the test set (20%) and put it away.
• Then split what remains into train (60% of all rows) and validation (20% of all rows).

Split before you drop any rows, so that everyone in the class holds out the same bookings and your test scores are comparable with your classmates' — whatever features each of you chose.

Part 1 — A Baseline Tree

Every tree in this part uses DecisionTreeClassifier with no parameters changed.

1a. Start small. Pick the two to four columns you think matter most for predicting a cancellation — your Lakeside rules are the evidence — and say in a sentence why each one made the cut. Fit a tree on just those columns and score it on the training data and on the validation data. Note how big the tree got with get_depth() and get_n_leaves().

1b. Now use everything. Fit the same default tree on all the columns and score it the same way.

Answer:

1. Is more features better here? Compare the two trees on training accuracy, validation accuracy and number of leaves, and explain what you see. Which kinds of columns make it easiest for a tree to memorize the training rows?
2. The lecture's remedies for overfitting include remove features. Does your comparison of the two trees back that up? Which other remedy on that list could you apply to the all-columns tree, so that it can keep its extra columns without memorizing them?

The all-columns tree is the baseline for the rest of the exercise — the one everything else has to beat.

Part 2 — Get the Score Up

Now make the tree better. You have two levers:

• The features. Which columns go in, how they are cleaned and encoded, and any new column you derive — anything you learned about this data writing your Lakeside rules is fair game.
• The tree's parameters. Anything in When to Stop? — max_depth, min_samples_leaf, min_samples_split, criterion — or others from the DecisionTreeClassifier documentation.

A single validation set is small and can be a bit lucky or unlucky, so choose between models with 5-fold cross validation (cross_validate with cv=5) on the train and validation rows together. The test set never enters it.

Keep a log: one row per attempt, with the features, the parameters, the training accuracy and the CV accuracy. Then pick one parameter — max_depth is a good choice — and try it at five or six values spread from small to large, such as 2, 4, 6, 8, 12 and 20, keeping the features and every other parameter the same. Spread them wide enough that the chart shows both a too-simple end and a too-complex end. Plot training accuracy and CV accuracy against that parameter on one chart.

Each of the five rounds gives two scores: one on the four folds the tree trained on, and one on the fold it never saw. cross_validate with return_train_score=True returns both, as in Model Evaluation and Selection — average train_score for training accuracy and test_score for CV accuracy. These are the training and testing performance from the lecture's detection rules: both low means underfitting, a big gap between them means overfitting. (Despite the name, test_score here is the held-out fold, not your locked test set.)

Your final model is the attempt with the best CV score. Answer:

1. On that chart, where does the CV curve peak, and what happens to the training curve past that point? Mark the region of your chart that is underfitting and the region that is overfitting, and say which line on the lecture's detection checklist told you. Where do your two Part 1 trees fall?
2. What moved your CV score more — the features or the parameters? By how much?

Part 3 — The One Look at the Test Set

Refit your final model on all of the train and validation rows — cross validation was for choosing, not for building — and score it once on the test set. Do the same for the Part 1 baseline, so you have both test scores side by side.

1. How close is your final test score to the CV score that chose it? What does that tell you about whether you needed Kaggle to know how good your model is?
2. Did your tree beat your best Lakeside rule? By how much?

Submission

Upload your finished notebook to Canvas for grading. Answer every question in this exercise in a markdown cell in the notebook, right after the code that produced the numbers it asks about — answers anywhere else are not graded.