Ensemble Learning

Wisdom of the crowd: the collective opinion of a group of individuals is often better than a single expert's answer.
Ensemble Learning: the prediction of a group of predictors often outperforms the best individual predictor.

In this lecture, we cover the three types of ensemble learning — voting, bagging (including random forest), and boosting (AdaBoost, Gradient Boosting, and XGBoost) — and try each one with scikit-learn. Run the code in a notebook in your ml4biz project; all the packages, including xgboost, are already installed.

Three Types of Ensemble Learning

Voting trains different models on all the data in parallel; bagging trains one type of model on random samples in parallel; boosting trains one type of model on all the data one after another

The code for the figures on this page is in the Appendix.

Voting: an ensemble of different models, each trained on all the data
Bagging (Bootstrap Aggregating): an ensemble of one type of model trained on different random subsets of the data in parallel, such as Random Forest
Boosting: an ensemble of one type of model trained on all the data sequentially, each trying to correct its predecessor, such as AdaBoost, Gradient Boosting, and XGBoost

Why Ensembles Work: Bias and Variance

A model's prediction error on new data comes from three sources:

Prediction Error = Bias² + Variance + Noise

(Bias is squared because the formula breaks down squared error. Bias can be positive or negative, while variance and noise are already squared quantities.)

Bias is error from the model being too simple to capture the real pattern in the data, such as fitting a straight line to data that curves. A high-bias model does poorly on the training data and on new data. High bias → under-fitting.
Variance is error from the model being too sensitive to its training data: it learns the quirks of that particular sample instead of the general pattern, so training it on a slightly different sample gives a very different model. A high-variance model, such as a very deep decision tree, does well on the training data but much worse on new data. High variance → overfitting.
Noise is error in the data itself, such as typos or faulty measurements. No model can remove it; only cleaning the data can.

The trade-off: for a single model, lowering one usually raises the other. A simpler model (a shallow tree) has lower variance but higher bias; a more complex model (a deep tree) has lower bias but higher variance. The best you can do is tune the model to a point in between, such as choosing max_depth.

Ensembles work around this trade-off by combining many models so that their errors partly cancel out:

Voting combines different types of models, such as a decision tree, logistic regression, and an SVM. Because they make different assumptions, they tend to make different mistakes, and the majority vote corrects many of them.
Bagging reduces variance. It trains many low-bias, high-variance models, such as deep trees (no depth limit: scikit-learn's default max_depth=None keeps splitting until every leaf is pure or too small to split), each on a different random sample. Each one overfits in a different way, so averaging or voting cancels out much of that overfitting, while bias stays low.
Boosting reduces bias. It trains many high-bias, low-variance models, such as one-split trees ("stumps"), one after another, each fitting the errors the previous ones left. Combined, they capture patterns no single stump can.

Voting

Classification: once each model gives its prediction, we use hard or soft voting to get the final prediction.
Regression: the average of all predictions is the final prediction.

Voting: a decision tree, logistic regression, and SVM each trained on all the two-moons training data, combined by hard voting

Hard vs. Soft Voting

Hard voting (majority voting): count the votes and predict the class with the most votes
Soft voting: if the classifiers can estimate class probabilities, average the probabilities over all classifiers and predict the class with the highest average probability

Hard voting predictions:

Classifier 1 predicts class A
Classifier 2 predicts class B
Classifier 3 predicts class B

2/3 classifiers predict class B, so class B is the ensemble decision.

Soft voting predictions:

Classifier 1 predicts class A with probability 0.9
Classifier 2 predicts class B with probability 0.55 (A 0.45)
Classifier 3 predicts class B with probability 0.55 (A 0.45)

The average probability of class A across all classifiers is (0.9 + 0.45 + 0.45) / 3 = 0.6, which is above the 0.5 decision threshold, so class A is the ensemble decision. Soft voting gives more weight to confident votes.

We create some data to illustrate hard and soft voting:

import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn.datasets import make_moons
from sklearn.model_selection import train_test_split
plt.style.use('ggplot')

# make_moons generates a simple toy dataset to visualize classification algorithms
# you can try a different noise level to make classification harder or easier
X, y = make_moons(n_samples=1000, noise=0.3, random_state=42)

moon = pd.DataFrame(X, columns=['x1', 'x2'])
moon['label'] = y
sns.scatterplot(x='x1', y='x2', hue='label', data=moon)

# split train and test sets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

The two-moons dataset

We combine three different classifiers: the decision tree, logistic regression, and the support vector machine (SVM) — see the Logistic Regression and Support Vector Machine lectures. You only need to know that they make different assumptions from each other — which is exactly what voting needs.

Hard voting ensemble model:

from sklearn.ensemble import VotingClassifier
from sklearn.tree import DecisionTreeClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.svm import SVC
from sklearn.metrics import accuracy_score

# three classifiers: decision tree, logistic regression, and svm
tree_clf = DecisionTreeClassifier(random_state=42)
log_reg_clf = LogisticRegression()
svm_clf = SVC(random_state=42)

# hard voting ensemble model
voting_clf = VotingClassifier(
    estimators=[
        ('dt', tree_clf),
        ('logit', log_reg_clf),
        ('svm', svm_clf),
    ],
    voting='hard'  # voting type
)

# train and test each model using a loop
for clf in (tree_clf, log_reg_clf, svm_clf, voting_clf):
    clf.fit(X_train, y_train)
    y_pred = clf.predict(X_test)
    print(clf.__class__.__name__, accuracy_score(y_test, y_pred))

Output:
DecisionTreeClassifier 0.91
LogisticRegression 0.82
SVC 0.915
VotingClassifier 0.905

Soft voting ensemble model — soft voting needs every classifier to estimate class probabilities. An SVC doesn't by default, so we wrap it in CalibratedClassifierCV, which turns its scores into probabilities:

from sklearn.calibration import CalibratedClassifierCV

# three classifiers: decision tree, logistic regression, and svm
tree_clf = DecisionTreeClassifier(random_state=42)
log_reg_clf = LogisticRegression()
svm_clf = CalibratedClassifierCV(SVC(random_state=42), ensemble=False)  # svm that estimates class probabilities

# soft voting ensemble model
voting_clf = VotingClassifier(
    estimators=[
        ('dt', tree_clf),
        ('logit', log_reg_clf),
        ('svm', svm_clf),
    ],
    voting='soft'  # voting type
)

# train and test each model using a loop
for clf in (tree_clf, log_reg_clf, svm_clf, voting_clf):
    clf.fit(X_train, y_train)
    y_pred = clf.predict(X_test)
    print(clf.__class__.__name__, accuracy_score(y_test, y_pred))

Output:
DecisionTreeClassifier 0.91
LogisticRegression 0.82
CalibratedClassifierCV 0.915
VotingClassifier 0.925

Here soft voting (0.925) beats every individual model, while hard voting (0.905) does not beat the best one: the weak logistic regression gets a full vote in hard voting, but soft voting lets the more confident models outweigh it. An ensemble is not guaranteed to win — it works best when the models are individually good and make different mistakes.

Bagging (Bootstrap Aggregating)

Bagging uses the same type of model (e.g., a decision tree) but trains each copy on a different random subset of the training set.

Once each model makes its predictions, the results are aggregated using hard/soft voting for classification and the average for regression.

Two types of subset sampling:

bootstrapping: sampling with replacement (the same instance can be selected more than once in one subset)
pasting: sampling without replacement

Both work. Bagging traditionally uses bootstrapping, and it is the default in scikit-learn. A bootstrap sample has as many rows as the training set, but because of repeats only about 63% of them are distinct. Pasting with a similar sample size, such as max_samples=0.6 with bootstrap=False, usually gives similar results.

Bagging: four bootstrap samples drawn with replacement from 10 instances, one tree trained on each, then combined by vote or average

Out-of-bag Evaluation

With bootstrapping, some instances are picked more than once for a subset, and others are not picked at all.

If there are $m$ training instances, the probability that a given instance is not picked in one random draw is $\frac{m-1}{m}$ .

Each bootstrap sample (bag) has the same size as the training set, i.e., $m$ draws, so the probability that an instance is not picked for a bag at all is $(\frac{m-1}{m})^m$ .

When $m$ is large: $\displaystyle{\lim_{m \to \infty}{\left(\frac{m-1}{m}\right)}^m} = e^{-1} \approx 0.368$

In other words, each individual model never sees about 37% of the training data, which are called out-of-bag (OOB) instances. OOB instances can be used to validate each model without creating a separate validation set.

How many times each of 10 training instances is picked in three bootstrap samples; instances never picked are out-of-bag

The three bags here are the first three bootstrap samples from the bagging figure above. For more, see Out-of-bag error on Wikipedia.

Random Forest

A Random Forest is bagging with decision trees, plus one more source of randomness. For example, if you have 1000 data points, each tree in the forest is trained on 1000 bootstrapped samples (some samples will be repeated), and each "bag" corresponds to one tree (an estimator). In addition, each split in each tree only considers a random subset of the features. Both kinds of randomness make the trees more different from each other, which makes their combined vote more reliable — this is why it is called a random forest.

Check the meaning of the parameters:

from sklearn.ensemble import RandomForestClassifier

# n_estimators (default 100): the number of trees in the forest
# bootstrap (default True): whether bootstrap samples are used when building trees. If False, the whole dataset is used to build each tree
# oob_score: whether to use out-of-bag samples to estimate the generalization score. Only available if bootstrap=True
# max_features (default 'sqrt'): the number of features to consider at each split
# max_depth (default None): no depth limit, so each tree grows until its leaves are pure
# you can also specify other decision tree parameters such as criterion
rf_clf = RandomForestClassifier(n_estimators=200, bootstrap=True, oob_score=True,
                                criterion='entropy', random_state=42)

rf_clf.fit(X_train, y_train)
print('oob accuracy', rf_clf.oob_score_)

y_pred = rf_clf.predict(X_test)
print('random forest testing accuracy', accuracy_score(y_test, y_pred))

Output:
oob accuracy 0.90625
random forest testing accuracy 0.905

The OOB accuracy (0.906), computed from the training data alone, is close to the accuracy on the test set (0.905). Each tree here grows with no depth limit and fits its own training sample perfectly, yet the forest still does well on new data: averaging cancels out much of the overfitting. max_depth can still be tuned, for example with grid search.

Boosting

Boosting refers to any Ensemble Learning method that combines several weak predictors into a strong one.

The general idea of boosting is to train predictors sequentially, each trying to correct its predecessor.

AdaBoost

AdaBoost (Adaptive Boosting) is adaptive in the sense that each subsequent weak learner is tweaked in favor of the instances misclassified by the previous ones.

Key idea: each training instance has a weight and each weak learner has a weight, and both are updated in each iteration (training each subsequent weak learner).

The algorithm works as follows. Assume there are $m$ training instances and the labels are -1 or 1 (instead of 0 or 1):

Initialize the training instance weights uniformly as $w_i^{(1)} = \frac{1}{m}$ .
For each iteration $t$ (train one weak learner):
- fit the weak learner $h_t(x)$ to the weighted training data
- calculate the weighted error rate $\epsilon_t$ — the sum of the weights of the misclassified instances divided by the sum of all instance weights:
  $\epsilon_t = \frac{\sum_{i=1}^m \mathbf{1}[h_t(x_i) \neq y_i] \, w_i^{(t)}}{\sum_{i=1}^m w_i^{(t)}}$
  $\epsilon_t$ is 0 when all predictions are right and 1 when all are wrong.
- set a weight for the weak learner based on its accuracy ( $\eta$ is the learning rate) — the more accurate the learner, the larger its weight:
  $\alpha_t = \eta \ln \Big( \frac{1-\epsilon_t}{\epsilon_t}\Big)$
- update the instance weights: $w_i^{(t+1)} = w_i^{(t)} e^{-\alpha_t y_i h_t(x_i)}$ . For a misclassified instance, $y_i h_t(x_i) = -1$ , so its weight increases; for a correctly classified one, it decreases.
- renormalize the weights, so that $\sum_{i=1}^m w_i^{(t+1)} = 1$ .
To make predictions, AdaBoost computes the predictions of all $T$ weak learners and weights them using $\alpha_t$ . The final prediction is the one with the majority of the weighted votes:
$H(x) = \text{sign} \Big( \sum_{t=1}^T \alpha_t h_t(x) \Big)$

Boosting: three AdaBoost rounds of decision stumps, where misclassified points get bigger weights in the next round, and their weighted vote

A Worked Example

A restaurant wants to predict whether each hour of the day will be busy or quiet, so it can schedule staff. It has ten hours of history, and the only feature is the hour:

hour	11	12	13	14	15	16	17	18	19	20
actual	busy	busy	busy	quiet	quiet	quiet	busy	busy	busy	quiet

Lunch is busy, the afternoon is quiet, dinner is busy, and the late evening is quiet. No single cut-off time can separate busy from quiet, so a single stump (a one-split tree, i.e., one "before/after" rule) can't get every hour right. Let's see how AdaBoost combines three of them, with learning rate $\eta = 0.5$ and busy = +1, quiet = −1.

Each round trains exactly one stump, which is one rule (one cut-off time with a prediction on each side). Three rounds give three rules, and the final model keeps all three and lets them vote.

In each round, the stump is trained like any decision tree (see Decision Tree Basics): try every possible cut-off time (13:30 between 13 and 14, and so on) and keep the one with the highest information gain. The only difference is that each hour counts by its weight instead of counting as one.

For example, here is the entropy of all ten hours, first counting each hour once (round 1), then with round 2's weights from the table below, where the dinner hours (17–19) that rule 1 missed weigh 0.167 each and every other hour weighs 0.071:

Counting: the busy hours are 11, 12, 13, 17, 18, 19, which is 6 of 10, so $p(\text{busy}) = \frac{6}{10} = 0.6$ and $p(\text{quiet}) = 0.4$ :
$\text{Entropy} = -0.6 \log_2 0.6 - 0.4 \log_2 0.4 = 0.971$
Weighting: the busy hours weigh $3 \times 0.071 + 3 \times 0.167 = 0.714$ , the quiet hours (14, 15, 16, 20) weigh the rest, $1 - 0.714 = 0.286$ , and all ten weigh 1. So $p(\text{busy}) = 0.714$ and $p(\text{quiet}) = 0.286$ :
$\text{Entropy} = -0.714 \log_2 0.714 - 0.286 \log_2 0.286 = 0.863$

In short, $p(\text{busy})$ = weight of the busy hours ÷ weight of all hours in the node, instead of number of busy hours ÷ number of hours. Every entropy, and so every information gain, is calculated this way, which is how the weights change which split wins. For example, the cut at 19:30 sends 11–19 left (busy $0.714$ , quiet $3 \times 0.071 = 0.214$ , total $0.929$ , so $p(\text{busy}) = 0.769$ and entropy $0.779$ ) and 20:00 right (quiet only, entropy 0), so its information gain is $0.863 - 0.929 \times 0.779 - 0 = 0.139$ . Doing this for every cut:

cut	information gain, counting (round 1)	information gain, weighting (round 2)
13:30	0.281 ← chosen	0.120
16:30	0.047	0.124
19:30	0.144	0.139 ← chosen

Counting, the best cut is 13:30, which misses the dinner hours. Once those hours weigh more, the best cut moves to 19:30, which gets them right.

The table below shows the weight of each hour in each round and the rule chosen in that round; bold marks a wrong prediction.

hour	11	12	13	14	15	16	17	18	19	20
weights, round 1	0.1	0.1	0.1	0.1	0.1	0.1	0.1	0.1	0.1	0.1
rule 1: busy before 13:30	busy	busy	busy	quiet	quiet	quiet	quiet	quiet	quiet	quiet
weights, round 2	0.071	0.071	0.071	0.071	0.071	0.071	0.167	0.167	0.167	0.071
rule 2: busy before 19:30	busy	busy	busy	busy	busy	busy	busy	busy	busy	quiet
weights, round 3	0.045	0.045	0.045	0.167	0.167	0.167	0.106	0.106	0.106	0.045
rule 3: busy after 16:30	quiet	quiet	quiet	quiet	quiet	quiet	busy	busy	busy	busy

Round 1 → rule 1: busy before 13:30, quiet after. Every hour starts with the same weight, $\frac{1}{10} = 0.1$ , and this cut has the highest information gain (0.281, as computed above). It misses the dinner rush (17–19).
- Weighted error $\epsilon_1$ : the weight of the missed hours divided by the total weight, $\epsilon_1 = \frac{0.1 + 0.1 + 0.1}{1} = 0.3$ .
- Rule weight $\alpha_1 = \eta \ln \frac{1-\epsilon_1}{\epsilon_1}$ : here $1 - \epsilon_1 = 0.7$ is the weight of the hours it got right, so $\alpha_1 = 0.5 \times \ln \frac{0.7}{0.3} = 0.5 \times \ln 2.33 = 0.5 \times 0.85 = 0.42$ . The more weight a rule gets right compared with what it gets wrong, the bigger its say in the final vote.
Then AdaBoost updates each hour's weight with $w_i \leftarrow w_i \, e^{-\alpha_1 y_i h_1(x_i)}$ , where $y_i h_1(x_i)$ is $+1$ if rule 1 got hour $i$ right and $-1$ if it got it wrong:
- missed hours (17, 18, 19): multiply by $e^{0.42} = 1.53$ , so $0.1 \to 0.1528$
- correct hours (the other seven): multiply by $e^{-0.42} = 0.65$ , so $0.1 \to 0.0655$
- renormalize so the weights add up to 1 again: the total is $3 \times 0.1528 + 7 \times 0.0655 = 0.917$ , so divide every weight by 0.917. The missed hours become $0.1528 / 0.917 = 0.167$ and the others $0.0655 / 0.917 = 0.071$ .
Each missed hour now counts more than twice as much as each correct one ( $0.167$ vs. $0.071$ ) when the next rule is scored. Rounds 2 and 3 update the weights the same way.

Round 2 → rule 2: busy before 19:30, quiet after. The dinner hours now weigh the most, so the best rule is one that gets them right. It misses the quiet afternoon (14–16), but those hours weigh only 0.071 each, so $\epsilon_2 = 3 \times 0.071 = 0.21$ and $\alpha_2 = 0.65$ . Compare round 1's rule, which also gets three hours wrong — but they are the dinner hours, so its error is now $3 \times 0.167 = 0.5$ :

rule	hours it gets wrong	weighted error $\epsilon$
busy before 13:30 (round 1's rule)	17, 18, 19	$3 \times 0.167 = 0.50$
busy before 19:30	14, 15, 16	$3 \times 0.071 = 0.21$ ← chosen

Round 3 → rule 3: quiet before 16:30, busy after. Now the afternoon hours weigh the most, so the best rule is one that gets them right. It misses lunch (11–13) and 20:00, which weigh only 0.045 each, so $\epsilon_3 = 0.18$ and $\alpha_3 = 0.75$ .

Each round, the hours the last rule missed end up carrying exactly half of the total weight ( $3 \times 0.167 = 0.5$ ). The last rule would now score $\epsilon = 0.5$ , no better than a coin flip, so the next round is forced to pick a rule that fixes those hours.

To predict, each rule votes +1 (busy) or −1 (quiet), weighted by its $\alpha$ : score $= 0.42 \times \text{rule 1} + 0.65 \times \text{rule 2} + 0.75 \times \text{rule 3}$ . A positive score means busy:

hour	11	12	13	14	15	16	17	18	19	20
weighted score	0.32	0.32	0.32	−0.53	−0.53	−0.53	0.98	0.98	0.98	−0.32
final prediction	busy	busy	busy	quiet	quiet	quiet	busy	busy	busy	quiet

For example, at 11:00, rules 1 and 2 say busy and rule 3 says quiet: $0.42 + 0.65 - 0.75 = 0.32 > 0$ , so busy.

All ten hours are predicted correctly, even though each rule on its own gets three or four wrong.

from sklearn.ensemble import AdaBoostClassifier

ada_clf = AdaBoostClassifier(n_estimators=200, learning_rate=0.5, random_state=42)
ada_clf.fit(X_train, y_train)
y_pred = ada_clf.predict(X_test)
print('AdaBoostClassifier', accuracy_score(y_test, y_pred))

Output:
AdaBoostClassifier 0.925

Gradient Boosting

Gradient Boosting is similar to AdaBoost, but instead of adjusting the instance weights for each predictor, it fits each new predictor to the residual errors made by the previous predictors.

from sklearn.ensemble import GradientBoostingClassifier

gb_clf = GradientBoostingClassifier(n_estimators=200, learning_rate=0.5, random_state=42)
gb_clf.fit(X_train, y_train)
y_pred = gb_clf.predict(X_test)
print('GradientBoostingClassifier', accuracy_score(y_test, y_pred))

Output:
GradientBoostingClassifier 0.875

Gradient Boosting does worst here because a learning rate of 0.5 with 200 trees overfits this small, noisy dataset. Try learning_rate=0.1, which gives 0.895 — boosting models need tuning, for example with the grid search from the Model Evaluation lecture.

XGBoost

An optimized implementation of Gradient Boosting is called Extreme Gradient Boosting (XGBoost), which is extremely fast, scalable, and portable. XGBoost is a separate package, already installed in your ml4biz project (otherwise, install it with uv add xgboost). You can use it as follows:

import xgboost as xgb

xgb_clf = xgb.XGBClassifier(random_state=42)
xgb_clf.fit(X_train, y_train)
y_pred = xgb_clf.predict(X_test)
print('XGBoost Classifier accuracy:', accuracy_score(y_test, y_pred))

Output:
XGBoost Classifier accuracy: 0.89

Appendix: Plotting Code

The code below draws the figures on this page. Run it in a notebook in your ml4biz project; each plt.show() displays one figure.

import numpy as np
import matplotlib.pyplot as plt
from matplotlib.patches import FancyArrowPatch, FancyBboxPatch, Rectangle
from sklearn.datasets import make_moons
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.svm import SVC
from sklearn.ensemble import VotingClassifier
from sklearn.metrics import accuracy_score

plt.style.use('seaborn-v0_8-white')
BLUE, ORANGE, GREEN, PURPLE, GRAY = '#4C72B0', '#DD8452', '#55A868', '#8172B3', '#8C8C8C'


def box(ax, x, y, text, w=2.2, h=0.9, fc=BLUE, tc='white', fs=11):
    """Draw a rounded box centered at (x, y) with a label inside."""
    ax.add_patch(FancyBboxPatch((x - w / 2, y - h / 2), w, h, boxstyle='round,pad=0.05,rounding_size=0.15',
                                fc=fc, ec='none'))
    ax.text(x, y, text, ha='center', va='center', color=tc, fontsize=fs, fontweight='bold')


def arrow(ax, start, end, text=None, color='#555555'):
    """Draw an arrow from start to end, with an optional label at its middle."""
    ax.annotate('', xy=end, xytext=start,
                arrowprops=dict(arrowstyle='-|>', color=color, lw=1.5, shrinkA=0, shrinkB=0))
    if text:
        ax.text((start[0] + end[0]) / 2, (start[1] + end[1]) / 2 + 0.25, text,
                ha='center', va='bottom', fontsize=9, color='#555555', style='italic')


# ---------------------------------------------------------------------------
# Figure 1: overview of the three types of ensemble learning
# ---------------------------------------------------------------------------
fig, axes = plt.subplots(1, 3, figsize=(14, 5.2))
for ax in axes:
    ax.set_xlim(0, 10)
    ax.set_ylim(0, 10)
    ax.axis('off')

# voting: different models, all the data, in parallel
ax = axes[0]
ax.set_title('Voting', fontsize=15, fontweight='bold')
box(ax, 5, 9, 'All training data', w=4.4, fc=GRAY)
for x, name, c in [(1.7, 'Tree', BLUE), (5, 'Logistic', ORANGE), (8.3, 'SVM', GREEN)]:
    arrow(ax, (5, 8.5), (x, 6.0))
    box(ax, x, 5.5, name, w=2.6, fc=c)
    arrow(ax, (x, 5.0), (5, 2.9))
box(ax, 5, 2.4, 'Vote', w=3.0, fc='#333333')
ax.text(5, 0.8, 'different models\nsame data, trained in parallel', ha='center', fontsize=11)

# bagging: one type of model, random subsets, in parallel
ax = axes[1]
ax.set_title('Bagging', fontsize=15, fontweight='bold')
box(ax, 5, 9, 'All training data', w=4.4, fc=GRAY)
for x, k in [(1.7, 1), (5, 2), (8.3, 3)]:
    arrow(ax, (5, 8.5), (x, 7.4))
    box(ax, x, 6.95, f'Sample {k}', w=2.6, h=0.8, fc='#BBBBBB', tc='#333333', fs=10)
    arrow(ax, (x, 6.5), (x, 5.95))
    box(ax, x, 5.5, f'Tree {k}', w=2.6, fc=BLUE)
    arrow(ax, (x, 5.0), (5, 2.9))
box(ax, 5, 2.4, 'Vote / average', w=3.6, fc='#333333')
ax.text(5, 0.8, 'one type of model\nrandom subsets, trained in parallel', ha='center', fontsize=11)

# boosting: one type of model, all the data, one after another
ax = axes[2]
ax.set_title('Boosting', fontsize=15, fontweight='bold')
box(ax, 5, 9, 'All training data', w=4.4, fc=GRAY)
for x, k in [(1.7, 1), (5, 2), (8.3, 3)]:
    box(ax, x, 5.5, f'Tree {k}', w=2.6, fc=BLUE)
    arrow(ax, (x, 5.0), (5, 2.9))
arrow(ax, (5, 8.5), (1.7, 6.0))
arrow(ax, (3.0, 5.5), (3.7, 5.5))
arrow(ax, (6.3, 5.5), (7.0, 5.5))
ax.text(6.6, 6.3, 'each fixes the previous one\'s errors', ha='center', fontsize=9, color='#555555', style='italic')
box(ax, 5, 2.4, 'Weighted vote', w=3.6, fc='#333333')
ax.text(5, 0.8, 'one type of model\nsame data, trained one after another', ha='center', fontsize=11)

plt.tight_layout()
plt.show()

# ---------------------------------------------------------------------------
# Figure 2: voting on the two-moons data from this page
# ---------------------------------------------------------------------------
X, y = make_moons(n_samples=1000, noise=0.3, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

models = [('Decision tree', DecisionTreeClassifier(random_state=42)),
          ('Logistic regression', LogisticRegression()),
          ('SVM', SVC(random_state=42))]
voting = VotingClassifier(estimators=[(n, m) for n, m in models], voting='hard')

# a grid of points covering the plot, used to color each model's decision regions
xx, yy = np.meshgrid(np.linspace(-2, 3, 300), np.linspace(-1.5, 2, 300))
grid = np.c_[xx.ravel(), yy.ravel()]


def regions(ax, model, title, edge):
    """Fit a model on all training data and shade its predicted class regions."""
    model.fit(X_train, y_train)
    acc = accuracy_score(y_test, model.predict(X_test))
    ax.contourf(xx, yy, model.predict(grid).reshape(xx.shape), levels=[-0.5, 0.5, 1.5],
                colors=[BLUE, ORANGE], alpha=0.25)
    ax.scatter(X_train[:, 0], X_train[:, 1], c=np.where(y_train == 1, ORANGE, BLUE), s=6)
    ax.set_title(f'{title}\ntest accuracy {acc:.3f}', fontsize=11)
    ax.set_xticks([])
    ax.set_yticks([])
    for s in ax.spines.values():
        s.set_edgecolor(edge)
        s.set_linewidth(2.5)


fig = plt.figure(figsize=(13, 10))
ax_data = fig.add_axes([0.375, 0.73, 0.25, 0.21])
ax_models = [fig.add_axes([0.05 + 0.32 * i, 0.38, 0.25, 0.21]) for i in range(3)]
ax_vote = fig.add_axes([0.375, 0.03, 0.25, 0.21])

ax_data.scatter(X_train[:, 0], X_train[:, 1], c=np.where(y_train == 1, ORANGE, BLUE), s=6)
ax_data.set_xlim(-2, 3)
ax_data.set_ylim(-1.5, 2)
ax_data.set_title('All 800 training points', fontsize=11)
ax_data.set_xticks([])
ax_data.set_yticks([])
for ax, (name, model), c in zip(ax_models, models, [BLUE, ORANGE, GREEN]):
    regions(ax, model, name, c)
regions(ax_vote, voting, 'Hard voting ensemble', '#333333')

# arrows in figure coordinates: data -> each model -> vote
for ax in ax_models:
    x = ax.get_position().x0 + 0.125
    fig.add_artist(FancyArrowPatch(
        (0.5, 0.72), (x, 0.65), transform=fig.transFigure, arrowstyle='-|>', mutation_scale=15, color='#555555'))
    fig.add_artist(FancyArrowPatch(
        (x, 0.37), (0.5, 0.30), transform=fig.transFigure, arrowstyle='-|>', mutation_scale=15, color='#555555'))
fig.text(0.70, 0.69, 'each model is trained on all the data', fontsize=11, style='italic', color='#555555')
fig.text(0.70, 0.295, 'each point gets the class most models predict', fontsize=11, style='italic', color='#555555')
plt.show()

# ---------------------------------------------------------------------------
# Figure 3: bagging with bootstrap samples of a 10-instance training set
# ---------------------------------------------------------------------------
rng = np.random.default_rng(42)
m, n_bags = 10, 4
colors = plt.cm.tab10(np.arange(m))  # one color per training instance
bags = [np.sort(rng.choice(m, size=m, replace=True)) for _ in range(n_bags)]


def squares(ax, idx, x0, y0, per_row=5, size=0.55):
    """Draw instances as numbered colored squares, per_row squares per row."""
    for j, i in enumerate(idx):
        x, yv = x0 + (j % per_row) * size, y0 - (j // per_row) * size
        ax.add_patch(Rectangle((x, yv), size * 0.92, size * 0.92, fc=colors[i], ec='none'))
        ax.text(x + size * 0.46, yv + size * 0.46, str(i + 1), ha='center', va='center',
                fontsize=9, color='white', fontweight='bold')


fig, ax = plt.subplots(figsize=(13, 7.5))
ax.set_xlim(0, 16)
ax.set_ylim(0, 10)
ax.axis('off')

ax.text(8, 9.75, 'Training set (10 instances)', ha='center', fontsize=12, fontweight='bold')
squares(ax, range(m), 5.25, 8.9, per_row=10)
centers = [2, 6, 10, 14]
for k, (cx, bag) in enumerate(zip(centers, bags), start=1):
    arrow(ax, (8, 8.7), (cx, 7.35))
    missing = sorted(set(range(m)) - set(bag))
    ax.text(cx, 7.0, f'Bootstrap sample {k}', ha='center', fontsize=11, fontweight='bold')
    squares(ax, bag, cx - 1.375, 6.2)
    ax.text(cx, 5.25, 'not picked: ' + ', '.join(str(i + 1) for i in missing),
            ha='center', fontsize=9.5, color='#555555')
    arrow(ax, (cx, 5.0), (cx, 4.35))
    box(ax, cx, 3.9, f'Tree {k}', w=2.4)
    arrow(ax, (cx, 3.45), (8, 2.05))
ax.text(8, 7.75, 'random sampling with replacement', ha='center', fontsize=10, style='italic',
        color='#555555', bbox=dict(fc='white', ec='none'))
box(ax, 8, 1.6, 'Vote (classification) / average (regression)', w=6.6, fc='#333333')
ax.text(8, 0.55, 'Same type of model, each trained on its own bootstrap sample, all in parallel',
        ha='center', fontsize=11)
plt.show()

# ---------------------------------------------------------------------------
# Figure 4: boosting -- three rounds of AdaBoost with decision stumps
# ---------------------------------------------------------------------------
Xb, yb = make_moons(n_samples=60, noise=0.3, random_state=42)
yb = np.where(yb == 1, 1, -1)  # AdaBoost uses labels -1 and 1
w = np.full(len(yb), 1 / len(yb))  # step 1: equal instance weights
eta, rounds = 0.5, 3  # learning rate 0.5, as in the AdaBoost code on this page
xx, yy = np.meshgrid(np.linspace(-2, 3, 300), np.linspace(-1.5, 2, 300))
grid = np.c_[xx.ravel(), yy.ravel()]
score = np.zeros(len(grid))  # running sum of alpha_t * h_t(x) over the grid
train_score = np.zeros(len(yb))  # the same sum for the training points

fig, axes = plt.subplots(1, rounds + 1, figsize=(16, 4.6))
for t, ax in enumerate(axes[:rounds]):
    stump = DecisionTreeClassifier(max_depth=1, random_state=42).fit(Xb, yb, sample_weight=w)
    pred = stump.predict(Xb)
    wrong = pred != yb
    eps = w[wrong].sum() / w.sum()                 # weighted error rate
    alpha = eta * np.log((1 - eps) / eps)          # learner weight
    ax.contourf(xx, yy, stump.predict(grid).reshape(xx.shape), levels=[-2, 0, 2],
                colors=[BLUE, ORANGE], alpha=0.2)
    ax.scatter(Xb[:, 0], Xb[:, 1], s=30 * np.sqrt(w * len(w)), c=np.where(yb == 1, ORANGE, BLUE),
               edgecolors=np.where(wrong, 'black', 'white'), linewidths=1.2)  # area grows with weight
    ax.set_title(f'Tree {t + 1} (a stump)\nerror = {eps:.2f}, weight $\\alpha$ = {alpha:.2f}', fontsize=11)
    score += alpha * stump.predict(grid)
    train_score += alpha * pred
    w = w * np.exp(-alpha * yb * pred)             # mistakes get heavier, correct ones lighter
    w = w / w.sum()                                # renormalize
    if t < rounds - 1:
        ax.annotate('', xy=(1.2, 0.5), xytext=(1.03, 0.5), xycoords='axes fraction',
                    arrowprops=dict(arrowstyle='-|>', color='#555555', lw=2, mutation_scale=20))

ax = axes[-1]
ax.contourf(xx, yy, np.sign(score).reshape(xx.shape), levels=[-2, 0, 2], colors=[BLUE, ORANGE], alpha=0.2)
ax.scatter(Xb[:, 0], Xb[:, 1], s=20, c=np.where(yb == 1, ORANGE, BLUE))
acc = np.mean(np.sign(train_score) == yb)
ax.set_title(f'Final model: weighted vote of trees 1-3\ntraining accuracy {acc:.2f}', fontsize=11)
for ax in axes:
    ax.set_xticks([])
    ax.set_yticks([])
fig.suptitle('Point size = instance weight; black outline = misclassified by that tree, '
             'so it gets a bigger weight in the next round', fontsize=12, y=0.02)
plt.subplots_adjust(wspace=0.25, bottom=0.1)
plt.show()

# ---------------------------------------------------------------------------
# Figure 6: out-of-bag instances for three bootstrap samples
# ---------------------------------------------------------------------------
rng = np.random.default_rng(42)  # same seed as Figure 3, so bags 1-3 match
m, n_bags = 10, 3
counts = np.array([np.bincount(rng.choice(m, size=m, replace=True), minlength=m) for _ in range(n_bags)])

fig, ax = plt.subplots(figsize=(11, 3.9))
for b in range(n_bags):
    for i in range(m):
        c = counts[b, i]
        ax.add_patch(Rectangle((i + 0.05, n_bags - 1 - b + 0.05), 0.9, 0.9,
                               fc=ORANGE if c == 0 else BLUE, alpha=1 if c == 0 else min(1, 0.25 + 0.2 * c), ec='none'))
        ax.text(i + 0.5, n_bags - 0.5 - b, 'OOB' if c == 0 else f'×{c}', ha='center', va='center',
                fontsize=11, color='white' if c == 0 else '#222222', fontweight='bold')
    oob = (counts[b] == 0).sum()
    ax.text(m + 0.2, n_bags - 0.5 - b, f'{oob}/{m} out-of-bag', va='center', fontsize=11)
ax.set_xlim(0, m + 2.3)
ax.set_ylim(0, n_bags)
ax.set_xticks(np.arange(m) + 0.5, [f'#{i + 1}' for i in range(m)])
ax.set_yticks(np.arange(n_bags) + 0.5, [f'Bag {n_bags - b}' for b in range(n_bags)])
ax.tick_params(length=0, labelsize=11)
ax.set_xlabel('Training instance', fontsize=11)
for s in ax.spines.values():
    s.set_visible(False)

# how much is out-of-bag on average for a large training set?
big = np.random.default_rng(0)
m_big = 1000
frac = np.mean([np.mean(np.bincount(big.choice(m_big, m_big), minlength=m_big) == 0) for _ in range(500)])
ax.set_title(f'×k = picked k times; OOB = never picked. With m = {m_big} instances, '
             f'{frac:.1%} are out-of-bag on average (theory: 36.8%)', fontsize=11)
plt.tight_layout()
plt.show()

On this page