Assignment 3: Model Evaluation, Full Pipeline and Feature Importance

All three parts use the Lakeside booking data from the Heuristic Challenge, and all three are scored on the same locked test set. That set is what makes the parts comparable:

• Part 1: finish the Model Evaluation In-Class Exercise, where you tuned a tree by hand.
• Part 2: build the same kind of model with a Full Pipeline and grid search, and check whether automating the search beats your hand tuning.
• Part 3: use Feature Importance to explain the Part 2 model to the hotel.

Answer every question in a markdown cell in the notebook, right after the code that produced the numbers it asks about. Answers anywhere else are not graded. The two rules from the in-class exercise hold for the whole assignment:

1. The test set is locked. Every choice is made with cross validation, and the test set is scored exactly once per model, at the end. See How to Use the Test Set.
2. Every choice gets a reason. "It scored higher" counts only if you say which score.

Part 1: The In-Class Exercise

Finish all three parts of the Model Evaluation In-Class Exercise in model-evaluation-exercise.ipynb, with every question answered. If you finished in class, check your answers and hand in the same notebook.

Part 2: A Full Pipeline for Lakeside

In the in-class exercise you cleaned the data, encoded it and tried parameters by hand, one attempt at a time. Now put all of it into a single pipeline, as in Full Pipeline, and let GridSearchCV do the search. Work in a new notebook called lakeside-full-pipeline.ipynb.

Split off the test set exactly as in the exercise (20%, random_state=409, before dropping any rows), so that it holds the same 2,709 bookings your Part 1 final model was tested on. Then build the full pipeline, find the best tree with grid search and cross validation, and score it once on the test set, with its confusion matrix. Treat canceled = 1 as the positive class.

The Questions

1. How many trees did the grid search fit in total? Show the calculation. (Hint: count the parameter combinations, then think about what cross validation does to each one, and what happens after the best combination is found.)
2. What the mistakes cost. The revenue team plans to overbook every room the model predicts will be canceled. A false positive (predicted canceled, but the guest shows up) means the hotel has to "walk" a guest to another hotel: about $400. A false negative (predicted to stay, but the guest cancels) means one empty room: about $150.
   • Which of precision and recall should Lakeside care about more, and why?
   • What is the total cost of your model's mistakes on the test set? What would it cost to predict that nobody cancels?

Part 3: Explain the Model to Lakeside

The revenue manager will not overbook rooms because a model told them to. They want to know what the predictions are based on. Use Feature Importance to explain your Part 2 model.

The Questions

1. Write to the revenue manager. In four to six sentences, explain what the model's predictions are based on and which way each factor pushes. Write it so the manager could forward it to the front desk: no "gini", no "one-hot", no "feature importance". Back it with real numbers, such as the cancellation rate of each group in the training data.
2. Compare with your rules. In Assignment 2 you said which column mattered most for your Lakeside rules. Does the tree agree? Name one column the tree relies on that your rules missed, or one your rules relied on that the tree ignores, and suggest why.
3. What importance does not tell you. The manager reads your note and proposes to stop taking bookings more than 90 days out, "since lead time drives cancellations." What would you tell them?

Submission

Upload both notebooks to Canvas:

1. model-evaluation-exercise.ipynb: Part 1.
2. lakeside-full-pipeline.ipynb: Parts 2 and 3.

Run each notebook from top to bottom before you upload it, so that every output shown was produced by the code above it.