VFlash: Fast, Accurate Visual Planning via Flash Primitive Decoding
u_Unrolled
D3: Classical ML · Feature Engineering — Turning Raw Data Into Model Inputs
Phase 2/Domain 3/Lesson 7

Feature Engineering — Turning Raw Data Into Model Inputs

15 min read·Intermediate·Free lesson
Feature Engineering — Turning Raw Data Into Model Inputs

You already know: models consume numeric feature matrices X (Supervised learning §1); PCA can compress features (Unsupervised learning §4); time series need lag features and time-based splits (ML problem families §4).

Why this matters for Atlas. Real data arrives as logs: some fields missing, plan names as text, timestamps, IDs. A model only sees numbers. How you convert raw data into features often matters more than the choice of model — and doing it carelessly creates the two most expensive bugs in applied ML: data leakage (the model secretly sees the answer) and training-serving skew (features computed differently in production).


1. Missing values

The question. 15% of students have no "minutes per lesson" recorded. Drop them? Fill them in?

Start from nothing. Options, from simplest:

  • Drop rows — only if few are missing at random; otherwise you bias the data.
  • Impute — fill with the median (robust to outliers), the mean, the most frequent category, or a model's prediction.
  • Add a "was missing" indicator — missingness itself is often informative ("no time recorded" may mean the student never opened the lesson).

Crucially, the fill value (e.g. the median) must be computed on the training data only and then reused for validation, test and production.

a column with gaps; arrows to "median fill" (gaps filled with 23) and an extra column "minutes_missing" with 1s where the gaps were.
Diagram— a column with gaps; arrows to "median fill" (gaps filled with 23) and an extra column "minutes_missing" with 1s where the gaps were.

Worked example.

python
import numpy as np
from sklearn.impute import SimpleImputer

minutes = np.array([[12.0], [np.nan], [30.0], [8.0], [np.nan], [45.0]])
imputer = SimpleImputer(strategy="median", add_indicator=True)
print(imputer.fit_transform(minutes))
# column 0: missing values replaced by the median (21.0); column 1: 1.0 where the value was missing

Common trap. Filling missing values with 0 when 0 is a real, meaningful value ("zero minutes" ≠ "unknown").


2. Categorical features: encoding text labels as numbers

The question. How does a model use plan = "free" / "pro_monthly" / "pro_yearly" or country = "IN" / "US" / …?

Start from nothing.

  • One-hot encoding — one 0/1 column per category. Safe and simple; explodes with many categories.
  • Ordinal encoding — map categories to integers. Only correct when order is real (beginner < intermediate < advanced); otherwise it invents a fake ordering for linear models (trees cope better).
  • Target encoding — replace each category with the average label for that category. Powerful for high-cardinality features (thousands of IDs), but it uses the label, so it must be computed out-of-fold or it leaks.
  • Hashing / embeddings — for huge vocabularies (D4 learns embeddings end-to-end).
the column plan with three values transformed three ways — one-hot (three 0/1 columns), ordinal (0/1/2), target-encoded (average quit rate per plan).
Diagram— the column plan with three values transformed three ways — one-hot (three 0/1 columns), ordinal (0/1/2), target-encoded (average quit rate per plan).

Worked example.

python
import pandas as pd
from sklearn.preprocessing import OneHotEncoder, OrdinalEncoder

df = pd.DataFrame({"plan": ["free", "pro_monthly", "free", "pro_yearly"],
                   "level": ["beginner", "advanced", "intermediate", "beginner"]})

onehot = OneHotEncoder(handle_unknown="ignore", sparse_output=False).fit(df[["plan"]])
print(onehot.get_feature_names_out(), onehot.transform(pd.DataFrame({"plan": ["pro_yearly", "enterprise"]})))
# an unseen category ("enterprise") becomes all zeros instead of crashing in production

ordinal = OrdinalEncoder(categories=[["beginner", "intermediate", "advanced"]])
print(ordinal.fit_transform(df[["level"]]).ravel())          # [0. 2. 1. 0.] — real order

3. Scaling: standardization and normalization

The question. One feature is "minutes" (5–60), another "tokens" (100–100,000). Does that matter?

Start from nothing.

  • Standardization — (x − mean) / std: mean 0, standard deviation 1.
  • Min–max normalization — rescale to [0, 1].
  • Log transform — for heavy-tailed positive values (token counts, prices) before scaling.

Distance-based and gradient-based models (k-means, kNN, logistic regression, neural nets) are sensitive to scale; tree models are not (a split threshold works in any units). As with imputation, fit the scaler on training data only.

Chart, three panels: raw tokens (long right tail), after log transform (bell-ish), after standardization (centred at 0).
Chart— Chart, three panels: raw tokens (long right tail), after log transform (bell-ish), after standardization (centred at 0).

Worked example — scale changes k-nearest-neighbour results:

python
import numpy as np
from sklearn.model_selection import cross_val_score
from sklearn.neighbors import KNeighborsClassifier
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

rng = np.random.default_rng(0)
n = 1000
minutes = rng.uniform(5, 60, n)
tokens = rng.uniform(100, 100_000, n)             # huge scale, but pure noise
y = minutes > 30                                  # the label depends only on minutes
X = np.column_stack([minutes, tokens])

print("kNN raw     :", cross_val_score(KNeighborsClassifier(), X, y, cv=5).mean().round(3))
print("kNN scaled  :", cross_val_score(make_pipeline(StandardScaler(), KNeighborsClassifier()), X, y, cv=5).mean().round(3))

Without scaling, distances are dominated by the noisy large-scale feature.


4. Building features: interactions, ratios and time

The question. The raw columns are there. What new columns would make the pattern obvious to the model?

Start from nothing. Good features encode domain knowledge:

  • Ratios — hints_per_lesson, tokens_per_second: often more meaningful than raw counts.
  • Interactions — is_pro × lessons_done: "pro users who are active" behave differently.
  • Time features — hour of day, weekday, days since signup, days since last visit (recency), counts in the last 7 days (frequency). Cyclical values like hour of day can be encoded as sin/cos so 23:00 is close to 00:00.
  • Aggregates — per-user averages and counts, computed only from events before the prediction time.
a timeline of one student's events with a vertical "prediction time" line; aggregates computed only from the events to the left of the line.
Diagram— a timeline of one student's events with a vertical "prediction time" line; aggregates computed only from the events to the left of the line.

Worked example — recency/frequency features with a strict cut-off:

python
import pandas as pd

events = pd.DataFrame({
    "user": ["a", "a", "a", "b", "b", "c"],
    "ts": pd.to_datetime(["2026-10-01", "2026-10-20", "2026-11-05", "2026-10-02", "2026-10-03", "2026-11-06"]),
    "lesson_done": [1, 1, 0, 1, 1, 1],
})
prediction_time = pd.Timestamp("2026-11-01")
past = events[events.ts < prediction_time]                       # never look past the cut-off

features = past.groupby("user").agg(
    lessons_done=("lesson_done", "sum"),
    last_seen=("ts", "max"),
    active_days=("ts", "nunique"),
)
features["days_since_last_seen"] = (prediction_time - features.last_seen).dt.days
print(features.drop(columns="last_seen"))
# user "c" has no events before the cut-off, so has no row — handle as a cold-start user

5. Data leakage: when the model cheats

The question. A model scores 99% in testing and fails in production. What happened?

Start from nothing. Leakage is any information available during training that won't be available at prediction time. Common forms:

  • Target leakage — a feature derived from the outcome (cancellation_reason to predict cancellation; "refund issued" to predict fraud).
  • Temporal leakage — features computed with future data, or a random split of time-ordered data.
  • Preprocessing leakage — fitting the scaler, imputer, encoder or feature selection on the whole dataset before splitting.
  • Duplicate/group leakage — the same user (or near-duplicate documents) in both train and test.

The fix: put every learned preprocessing step inside a Pipeline that is fit on training folds only, split by time or by group when appropriate, and ask of every feature: "Would I know this value at the moment I make the prediction?"

Diagram, two pipelines: "wrong" — scale/select on all data, then split (an arrow leaks from the test set back into preprocessing); "right" — split first, then fit preprocessing inside each training fold.
Diagram— Diagram, two pipelines: "wrong" — scale/select on all data, then split (an arrow leaks from the test set back into preprocessing); "right" — split first, then fit preprocessing inside each training fold.

Worked example — feature selection leakage makes pure noise look predictive:

python
import numpy as np
from sklearn.feature_selection import SelectKBest, f_classif
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score
from sklearn.pipeline import make_pipeline

rng = np.random.default_rng(0)
X = rng.normal(size=(200, 5000))          # 5,000 random features
y = rng.integers(0, 2, 200)               # random labels: nothing to learn

# WRONG: choose the 20 "best" features using ALL rows, then cross-validate
X_leaky = SelectKBest(f_classif, k=20).fit_transform(X, y)
wrong = cross_val_score(LogisticRegression(max_iter=1000), X_leaky, y, cv=5).mean()

# RIGHT: selection happens inside each training fold
pipe = make_pipeline(SelectKBest(f_classif, k=20), LogisticRegression(max_iter=1000))
right = cross_val_score(pipe, X, y, cv=5).mean()

print(f"leaky accuracy {wrong:.2f}  vs  honest accuracy {right:.2f}   (truth: 0.50, it's noise)")

In production / interviews. Interview question: "Your churn model has 0.98 AUC offline but performs like chance in production. What do you check first?" (Leakage: feature timestamps relative to the label, preprocessing fit on all data, users appearing in both splits.)


6. Training-serving skew and feature stores

The question. The model is fine offline. In production, the same user gets a different prediction. Why?

Start from nothing. Training-serving skew is when features are computed differently at training time (often a batch SQL job over history) and at serving time (often application code on live data). Typical causes: different code paths, different time windows, unit mismatches, a missing-value default that differs, or stale data.

Defences:

  • One definition per feature, reused for training and serving — at minimum, save the fitted preprocessing pipeline with the model.
  • Feature stores (Feast, Tecton, Vertex/SageMaker feature stores) keep feature definitions in one place, compute them for offline training with point-in-time correctness, and serve the latest values online with low latency (D11).
  • Log the features actually served, and compare their distributions with the training data (monitoring, D11).
one feature definition box feeding two paths — "offline: historical, point-in-time joins → training" and "online: latest values, low latency → serving" — with a monitor comparing the two distributions.
Diagram— one feature definition box feeding two paths — "offline: historical, point-in-time joins → training" and "online: latest values, low latency → serving" — with a monitor comparing the two distributions.

Worked example — save the fitted pipeline, not just the model:

python
import pickle
import numpy as np
import pandas as pd
from sklearn.compose import ColumnTransformer
from sklearn.impute import SimpleImputer
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import Pipeline, make_pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler

rng = np.random.default_rng(1)
train = pd.DataFrame({"minutes": rng.uniform(5, 60, 500),
                      "plan": rng.choice(["free", "pro_monthly", "pro_yearly"], 500)})
train.loc[rng.random(500) < 0.1, "minutes"] = np.nan
y = (train.minutes.fillna(30) < 20) | (train.plan == "free")

preprocess = ColumnTransformer([
    ("num", make_pipeline(SimpleImputer(strategy="median"), StandardScaler()), ["minutes"]),
    ("cat", OneHotEncoder(handle_unknown="ignore"), ["plan"]),
])
model = Pipeline([("features", preprocess), ("clf", LogisticRegression())]).fit(train, y)

blob = pickle.dumps(model)                        # ship THIS: preprocessing + model together
served = pickle.loads(blob)
live_request = pd.DataFrame({"minutes": [np.nan], "plan": ["enterprise"]})   # missing + unseen
print("P(quit):", served.predict_proba(live_request)[0, 1].round(3))          # no crash, same logic as training

7. Feature selection and importance

The question. We have 300 candidate features. Should we keep them all?

Start from nothing. Fewer features → simpler, faster, less overfitting, easier monitoring. Approaches:

  • Filter — score each feature alone (correlation, mutual information). Fast, ignores interactions.
  • Embedded — L1 regularization zeroes weak features; tree models report importances.
  • Permutation importance — shuffle one feature on held-out data and measure how much the score drops. Model-agnostic and honest — but correlated features share credit, so each can look unimportant alone.

Importance says what the model relies on, not what causes the outcome.

Worked example.

python
import numpy as np
from sklearn.ensemble import RandomForestClassifier
from sklearn.inspection import permutation_importance
from sklearn.model_selection import train_test_split

rng = np.random.default_rng(2)
n = 2000
useful = rng.normal(size=n)
copy_of_useful = useful + 0.05 * rng.normal(size=n)       # highly correlated duplicate
noise = rng.normal(size=(n, 3))
X = np.column_stack([useful, copy_of_useful, noise])
y = useful + 0.3 * rng.normal(size=n) > 0

X_tr, X_te, y_tr, y_te = train_test_split(X, y, random_state=0)
rf = RandomForestClassifier(n_estimators=200, random_state=0).fit(X_tr, y_tr)
perm = permutation_importance(rf, X_te, y_te, n_repeats=10, random_state=0)
for name, imp in zip(["useful", "copy_of_useful", "noise1", "noise2", "noise3"], perm.importances_mean):
    print(f"{name:15s} {imp:.3f}")
# the two correlated features split the credit; dropping one barely hurts the model

Quick check

A churn model's top feature is 'number of cancellation-page visits', and it predicts perfectly offline but poorly in production. What's likely happening?

  1. The model is overfitting to noise
  2. Target leakage: the feature is recorded during the churn process itself (after or at the moment of the decision), so it isn't available early enough at prediction time
  3. The feature needs scaling
  4. Production data is too small
Show the answer

B. Target leakage: the feature is recorded during the churn process itself (after or at the moment of the decision), so it isn't available early enough at prediction time — Features must be computable at prediction time from past data; leaked features inflate offline metrics and vanish in production.


Check yourself

  1. Recall: Why must an imputer or scaler be fitted on training data only?
  2. Apply: A feature refund_issued predicts fraud with 99% accuracy. Use it?
  3. Predict: Does standardizing features change a decision tree's accuracy?
Answers
  1. Otherwise statistics from validation/test data leak into training, inflating scores.
  2. Only if a refund is known before the fraud decision — usually it's a consequence of fraud, i.e. target leakage.
  3. No (essentially) — tree splits are thresholds and work the same in any monotonic rescaling.

Exercise

Build a churn pipeline on synthetic event data: compute recency/frequency features with a strict prediction-time cut-off, one-hot encode the plan, impute and scale numeric features inside a ColumnTransformer, and evaluate with a time-based split. Then deliberately add a leaky feature (days_until_cancel, computed using future events) and show how much it inflates the offline score.

Solution sketch
python
import numpy as np
import pandas as pd
from sklearn.compose import ColumnTransformer
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.impute import SimpleImputer
from sklearn.metrics import roc_auc_score
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder

rng = np.random.default_rng(0)
n = 4000
df = pd.DataFrame({
    "signup_month": rng.integers(1, 13, n),
    "plan": rng.choice(["free", "pro_monthly", "pro_yearly"], n, p=[0.7, 0.2, 0.1]),
    "days_since_last_seen": rng.exponential(7, n),
    "lessons_last_30d": rng.poisson(4, n).astype(float),
})
df.loc[rng.random(n) < 0.05, "lessons_last_30d"] = np.nan
logit = 0.15 * df.days_since_last_seen - 0.3 * df.lessons_last_30d.fillna(2) + (df.plan == "free") * 0.8 - 1
df["churned"] = rng.random(n) < 1 / (1 + np.exp(-logit))
df["days_until_cancel"] = np.where(df.churned, rng.integers(1, 30, n), 999)   # LEAKY: from the future

train, test = df[df.signup_month <= 9], df[df.signup_month > 9]               # time-based split

def evaluate(cols):
    num = [c for c in cols if c != "plan"]
    pre = ColumnTransformer([("num", SimpleImputer(strategy="median"), num),
                             ("cat", OneHotEncoder(handle_unknown="ignore"), ["plan"])])
    model = Pipeline([("pre", pre), ("gb", HistGradientBoostingClassifier(random_state=0))])
    model.fit(train[cols], train.churned)
    return roc_auc_score(test.churned, model.predict_proba(test[cols])[:, 1])

honest = ["plan", "days_since_last_seen", "lessons_last_30d"]
print(f"honest AUC {evaluate(honest):.3f} | with leaky feature {evaluate(honest + ['days_until_cancel']):.3f}")

This unlocks:

  • ML evaluation — splitting and scoring so leaks are caught.
  • D11 Feature engineering infrastructure — feature stores and point-in-time joins at scale.
  • D16 Data processing & quality — data validation before features are built.

Sources: scikit-learn user guide — preprocessing, pipelines, "Common pitfalls and recommended practices" (data leakage), permutation importance; Kaufman et al., Leakage in Data Mining (2012); Sculley et al., Hidden Technical Debt in Machine Learning Systems (2015); Feast documentation — point-in-time joins.

Build Atlas · Part 2 of 9: Learning from dataUse this lesson in the project: A first model that learns: TF-IDF features, softmax regression in NumPy, and an evaluation that can't cheat.