You already know: precision, recall, PR-AUC, calibration and thresholds (ML evaluation); boosted trees and early stopping (Boosting); cross-validation (ML evaluation).
Why this matters for Atlas. Rare positives (churn, fraud, failing students) break naive training and evaluation; hyperparameters decide whether a model is mediocre or excellent, and the search strategy decides how much compute that costs; and explanations decide whether people trust and correctly use a model. Each has a principled approach and a set of traps. This lesson covers resampling vs reweighting vs thresholding, random and Bayesian search with proper validation, and Shapley-value explanations computed from first principles.
1. Imbalanced data: fix the decision, not just the data
The question. With only a few percent positives, how should you train and evaluate?
Start from nothing. Imbalance mainly hurts in three ways: accuracy becomes meaningless (predicting "no" scores 96%), a default 0.5 threshold rarely matches the business trade-off, and small minority classes give noisy estimates. Options: class weights (scale the loss of minority examples), resampling (oversample minorities, e.g. SMOTE, or undersample the majority), and threshold tuning on a calibrated model. For probabilistic models, the cleanest approach is usually: train on the natural distribution (or reweight), calibrate, then choose the threshold from costs (Fraud case study, D22). Resampling changes the predicted probabilities, so if you resample, recalibrate. Evaluate with PR-AUC, recall at a fixed precision, or expected cost, with stratified splits.