Model Evaluation
How good is a model before Kaggle tells you? We first cover the measures behind the score and how to estimate it yourself with validation, then put them to work on the Titanic data.
Topics
- Model Evaluation Basics — the confusion matrix, accuracy, precision, recall, and F1; the uneven costs of the two error types; holdout validation and k-fold cross validation; overfitting, underfitting, and their remedies.
- Model Evaluation and Selection
- Precision-Recall Trade-off and ROC/AUC (optional) — prediction probabilities and the decision threshold, the precision-recall trade-off, ROC curves and AUC for comparing models, and why they matter for imbalanced data.
- Model Evaluation In-Class Exercise — a decision tree on the Lakeside booking data, scored without Kaggle: a locked test set, and validation and 5-fold cross validation to tune it.
- Model Tuning via Grid Search
- Full Pipeline
- Feature Importance — which features a decision tree relies on, how to read the scores after one-hot encoding, and why it matters in practice: explaining a model to the people who use it, catching data leakage, and simplifying the pipeline.
Datasets
Decision Tree Prediction
We create a basic Decision Tree model with all default settings to show the key steps for building a baseline model-based prediction (no model selection, no…
Model Evaluation Basics
When you submit a prediction file to Kaggle, you get back a single number — for example, 0.76555 for the gender-based baseline and 0.77511 for a decision…