Assignment 2: Prediction Without and With a Model

Part 1 is baseline prediction on the Lakeside booking data: rules you wrote yourself, no model. Part 2 is a decision tree on the Titanic data.

Model-free baseline prediction matters. A baseline is the bar any model has to clear, and without one you have no way of knowing whether the model was worth building.

There is no notebook to hand in. Each part ends with scored submissions on Kaggle, and what you hand in is the screenshots plus a short write-up. A score with no screenshot is not graded, and every screenshot has to show your Kaggle team name (Part 1) or your own username (Part 2).

Part 1 — The Lakeside Competition (Model-Free)

Finish the Heuristic Challenge: predict which of Lakeside Resorts Group's bookings get canceled, using rules only — no models.

Group Work, Individual Submission

Part 1 is a group exercise. Merge with your groupmates into one team on Kaggle and submit as that team. Settle the team before anyone submits: once you merge, every submission counts toward the team, the leaderboard shows only the team, and Kaggle does not let you leave it.

Everyone in the group still submits individually on Canvas: the team's screenshots plus your own write-up. Name your team and your groupmates at the top of your write-up.

What to Do

Submit at least two different rules to the competition, and make them a progression rather than two guesses:

1. A one-column rule. The single best column you can find, with a threshold you picked from the data.
2. Your best rule. Add conditions to the first rule and see what they are worth — however many it takes.

Predicting Canceled = 0 for everybody scores about 0.596 on this leaderboard. That is the floor.

What to Hand In

Write up each of your two-or-more rules with all three of these:

• The rule in plain English, as one sentence a hotel manager could act on: "predict a booking will be canceled when it was made more than 90 days out and the guest put down no deposit."
• The evidence from the data that made you believe it.
• The leaderboard score it got, and — for every rule after the first — what the change bought you compared to the previous rule, and whether that matches what you expected before you submitted.

Then close with a short section answering these:

1. Which column mattered most, and how do you know? Point at the numbers, not at your intuition about hotels.
2. Which column looked promising and turned out to be worthless? Every one of you will have at least one. Show the table that fooled you and say what you should have looked at to catch it sooner.
3. Does any column in your rule only work in combination with another? That is, is there a column whose effect reverses or disappears depending on the value of a second column? If you found one, show it. If you looked and found none, say where you looked.

Screenshots for Part 1

Three images:

• Your team's submission history on the competition's My Submissions tab, showing all of the team's scored submissions in one shot with the team name visible.
• Your team's best submission score.
• Your team's leaderboard position — the leaderboard scrolled to where your team sits, with its rank and score readable.

Your rank is not the grade. A well-argued rule that came 20th scores higher here than an unexplained rule that came 2nd.

Part 2 — The Titanic Decision Tree

Work through Decision Tree Prediction end to end on the Kaggle Titanic competition. Type the code yourself rather than pasting the lecture's complete script and running it once — the questions below assume you watched each step happen. The code stays on your machine; only the scores and the answers are handed in.

You make two submissions:

1. The default tree — DecisionTreeClassifier() with nothing changed, exactly as in the lecture.
2. A depth-limited tree — the same features, with max_depth=5: DecisionTreeClassifier(max_depth=5). The lecture sets the argument the same way in its Tree Visualization section — DecisionTreeClassifier(criterion='entropy', max_depth=3) — and When to Stop? explains what the argument does.

The Questions

Answer each one in your write-up, in your own words. Where a question asks you to show a number, a table or a tree, paste the output or a screenshot of it next to the answer.

1. Missing data is handled in opposite ways on the two sides. The lecture drops rows with missing values from train.csv and fills them in test.csv. Why can't you drop the rows in the test set? And when you fill the one missing Fare, is it more defensible to use the mean of the test set or the mean of the training set? Say which you used and why.

2. One-hot encoding is encoded twice, separately. Print X.columns and X_test.columns and compare them. What exactly would go wrong if the test set happened to contain a category the training set did not, or vice versa — and would you get an error, or a silently wrong prediction? Show a line of code that would catch the problem before you submitted.

3. Information gain by hand. Using the training data, compute the entropy of Survived, then the information gain of splitting on Sex and the information gain of splitting on Pclass. Show the frequency tables and the arithmetic. Which feature would ID3 pick as the root? Now plot your tree and look at its actual root node — does it agree? If not, give a reason it might differ.

Screenshots for Part 2

• Both submission scores, on the My Submissions tab.
• Your leaderboard position on the Titanic leaderboard, with your rank and score readable.

Submission

Upload to Canvas:

1. One write-up covering both parts — PDF — with the rules, the scores and the answers to every question.
2. The five screenshots, either embedded in the write-up or uploaded separately, named so it is obvious which is which: lakeside-submissions.png, lakeside-best-score.png, lakeside-leaderboard.png, titanic-submissions.png, titanic-leaderboard.png.

No notebook is collected. What is graded is the scores you got on Kaggle and the reasoning you give for them.