Machine learning foundations — prediction vs explanation, validation, overfitting and leakage

Prediction vs explanation

• Explanation asks why: which factors are associated with churn, by how much, with what uncertainty? Simple, interpretable models (regression) and careful causal reasoning.
• Prediction asks what will happen for a new case: which subscribers will churn next month? Accuracy on new data matters most; complex models are acceptable if they are validated and monitored.

A model can predict well without explaining causes (it may exploit correlations), and a good explanatory model may predict modestly. State the goal before choosing methods.

When ML is useful — and when it is unnecessary

ML helps when there are many cases, a repeated decision, enough historical labelled data, and patterns too complex for rules. It is unnecessary or harmful when:

• a simple rule or SQL query answers the question ("customers inactive 90 days");
• there are few cases or the decision is one-off;
• labels are unreliable or the future will differ fundamentally from the past;
• the organisation cannot act on the predictions or maintain the model.

AI assistants (LLMs) can draft SQL and code quickly, but they can invent columns, misunderstand grain or produce subtly wrong logic. Treat generated code as an untested junior colleague's work: read it, run it on known cases, reconcile results — and never paste confidential data into tools your organisation has not approved.

Splitting data

Cross-validation (k-fold) rotates the validation fold for more stable estimates on small data. For time-ordered problems (forecasts, churn next month), split by time: train on the past, validate on later periods. Random splits leak future patterns into training.

Metrics

Regression: MAE (average absolute error, in units), RMSE (penalises large errors), MAPE (% error — undefined at zero actuals), WAPE (Σ|error| ÷ Σ actual — robust for retail), R².

Classification: confusion matrix (TP, FP, TN, FN); precision = TP ÷ (TP + FP); recall = TP ÷ (TP + FN); F1; ROC-AUC (ranking quality); calibration (do predicted 30% risks happen 30% of the time?).

Worked example — 1,000 subscribers, 100 churn. A model flags 150; 60 of them churn. TP = 60, FP = 90, FN = 40, TN = 810. Precision = 60 ÷ 150 = 40%; recall = 60 ÷ 100 = 60%; accuracy = 870 ÷ 1,000 = 87% — lower than the 90% of "nobody churns". Accuracy is misleading for rare events; choose metrics that match the business cost of errors.

Always compare with a baseline: majority class, last value, seasonal naive, a simple rule.

Overfitting and underfitting

• Underfitting: the model is too simple; poor on training and test data (high bias).
• Overfitting: the model memorises training noise; excellent on training, worse on test (high variance).

import pandas as pd, numpy as np
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier
from sklearn.metrics import roc_auc_score
subs = pd.read_csv("telecom/subscribers.csv")
X = pd.get_dummies(subs[["contract_type", "payment_method", "region", "acquisition_channel", "plan_id"]].astype(str), drop_first=True)
X["age"] = subs["age"]; y = subs["churned"]
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3, random_state=7, stratify=y)
for depth in [1, 3, 6, 12, None]:
    m = DecisionTreeClassifier(max_depth=depth, random_state=7).fit(Xtr, ytr)
    print(f"max_depth={str(depth):>4}  train AUC={roc_auc_score(ytr, m.predict_proba(Xtr)[:,1]):.3f}  test AUC={roc_auc_score(yte, m.predict_proba(Xte)[:,1]):.3f}")

max_depth=   1  train AUC=0.574  test AUC=0.579
max_depth=   3  train AUC=0.608  test AUC=0.613
max_depth=   6  train AUC=0.640  test AUC=0.617
max_depth=  12  train AUC=0.792  test AUC=0.560
max_depth=None  train AUC=0.988  test AUC=0.507

As depth grows, training AUC climbs toward 1 while test AUC stalls and then falls — the signature of overfitting. Choose complexity on validation performance, not training performance.

Data leakage

Leakage = information available during training that would not be available at prediction time. It produces excellent offline metrics and failure in production.

Prevention: define a prediction date; build every feature only from data before it; split by time; fit preprocessing inside a pipeline on training data only; be suspicious of any feature that looks "too good".

Practice

Summary

• Decide whether you need prediction or explanation; use ML only when it beats simpler approaches and can be acted on.
• Split train/validation/test — by time for temporal problems; compare with baselines.
• Choose metrics matching the business cost of errors; accuracy misleads on rare events.
• Overfitting = great training, worse test; leakage = information from the future — prevent it by design.