Ensemble Methods (Bagging, Boosting)

Ensemble learning combines multiple models to improve predictions. The result depends on the component models and their errors.

The idea is that diverse models make different mistakes, so if we average them out, we can improve the overall accuracy.

Combining Predictions

Two common strategies for merging individual model outputs:

Method	How it works	Best for
Majority vote	Choose the most common prediction across all models	Classification
Summation	Sum the probability scores for each class; pick the highest	Classification with probabilities

Note: Averaging is equivalent to summation for the purpose of picking the winner, so the extra division step that we need for averaging is unnecessary.

---

Bootstrap Aggregating (Bagging)

Introduction

Bagging reduces overfitting by training each model on a different random sample of the data, rather than all models seeing the same training set.

How It Works

1. From the original dataset, randomly sample with replacement to create a new dataset of the same size
2. Train one model on this new dataset
3. Repeat for every model in the ensemble where each sees a slightly different dataset
4. At inference time, aggregate all model predictions (majority vote or sum)

With replacement means the same data point can appear multiple times in one dataset, and some points may not appear at all (these are called out-of-bag samples and can be used for validation).

Effect

Because each model is trained on a different subset of the data, individual models overfit to different noise. When combined, the noise cancels out and hopefully the true signal is amplified.

---

Boosting

Introduction

Boosting adds models sequentially to improve the current combined prediction. Compare its validation performance and training cost with bagging for the specific task.

How AdaBoost Works

1. Train a base model on the original dataset with equal weights on all samples
2. Identify which samples were misclassified
3. Increase the weights of misclassified samples (making them matter more)
4. Decrease the weights of correctly classified samples
5. Train the next model on the re-weighted dataset
6. Repeat the training where each model in the sequence focuses more on what previous ones got wrong
7. Final prediction is a weighted sum of all models:

$$\text{prediction}=w_1\cdot {\text{out}}_1+w_2\cdot {\text{out}}_2+w_3\cdot {\text{out}}_3$$

where each weight (w_i) reflects that model's accuracy on its training data.

How Gradient Boosting Differs

Gradient boosting fits each new model to a signal from the current loss gradient. With squared-error regression, this signal is the residual: target minus current prediction. The new model corrects part of the remaining error.

AdaBoost's sample reweighting procedure and gradient boosting's loss-gradient procedure are different implementations of sequential model combination. scikit-learn ensemble guide.

Bagging vs. Boosting

	Bagging	Boosting
Training	Parallel (models are independent)	Sequential (each depends on previous)
Focus	Reduce variance / overfitting	Reduce bias / improve hard examples
Speed	Faster	Slower
Example algorithms	Random Forest	AdaBoost, XGBoost, LightGBM

---

Code

Compare a vote with average probabilities

import numpy as np
 
probabilities = np.array([[0.51, 0.49], [0.51, 0.49], [0.01, 0.99]])
# Rows are models; columns are classes 0 and 1 for one sample
 
individual_labels = probabilities.argmax(axis=1)
# argmax(axis=1) means we return the index of the max argument
# axis=1 here means that we look through each row, and find the
# max of the columns
 
vote = np.bincount(individual_labels, minlength=2).argmax()
# this returns [2, 1]
 
mean_probabilities = probabilities.mean(axis=0)
# average out the probabilties for each class
 
combined_label = mean_probabilities.argmax()
print(individual_labels.tolist())  # [0, 0, 1]
print(vote)                        # 0
print(mean_probabilities.round(3))  # [0.343 0.657]
print(combined_label)              # 1

Read the two combinations:

• argmax(axis=1) selects one class from each model's row
• bincount counts the class votes; the most frequent class wins
• mean(axis=0) averages corresponding class probabilities across models
• The strong class-1 probability from the third model can change the probability-based result

All models must use the same class-column order. For an ensemble of trained trees, see Random Forest.