VFlash: Fast, Accurate Visual Planning via Flash Primitive Decoding
u_Unrolled
D3: Classical ML · Boosting — From AdaBoost to Gradient Boosting, XGBoost, LightGBM, CatBoost and Stacking
Phase 2/Domain 3/Lesson 3

Boosting — From AdaBoost to Gradient Boosting, XGBoost, LightGBM, CatBoost and Stacking

15 min read·Intermediate· Unrolled Pro
a sequence of small trees, each fitted to the residual errors left by the sum of the previous ones, with the ensemble's prediction curve getting closer to the target after 1, 10 and 100 trees; beside it, a training/validation loss plot where validation loss bottoms out (early stopping) while training loss keeps falling.
Diagram— a sequence of small trees, each fitted to the residual errors left by the sum of the previous ones, with the ensemble's prediction curve getting closer to the target after 1, 10 and 100 trees; beside it, a training/validation loss plot where validation loss bottoms out (early stopping) while training loss keeps falling.

You already know: decision trees and bagging/random forests (Supervised learning); gradient descent (Optimization); loss functions and overfitting (Supervised learning, ML evaluation).

Why this matters for Atlas. Random forests average many independent deep trees to reduce variance. Boosting does the opposite: it builds small trees sequentially, each one correcting the errors of the ensemble so far, reducing bias. Framed as gradient descent in function space, it works with any differentiable loss, which is why one library handles regression, classification, ranking and survival models. This lesson builds AdaBoost and gradient boosting from scratch, explains what XGBoost, LightGBM and CatBoost add, and finishes with stacking.


1. AdaBoost: reweight the mistakes

The question. Can many weak classifiers make a strong one?

Start from nothing. Start with equal weights on all training examples. Fit a weak learner (a decision stump: one split on one feature), compute its weighted error ε, give it a vote α = ½ ln((1 − ε)/ε), then increase the weights of misclassified examples so the next stump focuses on them. The final classifier is the sign of the weighted vote. AdaBoost turns stumps barely better than chance into a strong classifier; it's equivalent to minimizing an exponential loss, which also makes it sensitive to label noise (mislabelled points get ever larger weights).

That's the preview. The full lesson, with its code and exercises, is part of Unrolled Pro.
Staff AI & Pro Tier Lesson

Unlock "Boosting — From AdaBoost to Gradient Boosting, XGBoost, LightGBM, CatBoost and Stacking"

The foundations are free forever. Unrolled Pro unlocks the complete lesson, production architectures, and 55 battle-tested ML system design case studies.

Everything included with Pro:
All 234+ Deep-Dive Lessons across transformer internals, inference, RAG & agents
55 Staff-Level ML & System Design Case Studies with mathematical capacity estimates
AI Engineer Sheet & Exercises with worked solutions, self-checks, and code
100 Landmark Research Papers guided walkthroughs & progress tracking
14-Day Money-Back Guarantee — 100% full refund if you aren't satisfied
View Plans & Unlock All Lessons

From ₹167/mo (~$2/mo) · Cancel anytime · Instant activation

100% Risk-Free: 14-day no-questions-asked money-back guarantee