Regression, gradient boosting (XGBoost), model comparison and time-series forecasting

Linear regression as a predictive model

Linear (and logistic) regression are strong baselines: fast, stable, interpretable. Add features (lags, dummies, interactions) to capture more structure; regularisation (ridge, lasso) controls overfitting when there are many features.

Gradient boosting and XGBoost

Gradient boosting builds many small decision trees sequentially; each new tree fits the errors (gradients) of the current ensemble. Predictions are the sum of all trees' contributions, scaled by a learning rate.

XGBoost is a widely used, efficient implementation (similar: LightGBM, CatBoost, scikit-learn's HistGradientBoosting). Key hyperparameters:

Use early stopping on a validation set to choose the number of trees. In Python: xgboost.XGBRegressor(n_estimators=500, learning_rate=0.05, max_depth=4, subsample=0.8, early_stopping_rounds=50) with eval_set=[(X_val, y_val)]. The example below uses scikit-learn's HistGradientBoostingRegressor, which implements the same idea and is available everywhere scikit-learn is.

Forecasting monthly demand: a full backtest

Time-series terms: trend, seasonality, forecast horizon (how far ahead: 1–3 months), forecast accuracy (MAE, WAPE). We forecast monthly units per product category, horizon 1 month, with rolling-origin evaluation: for each month of 2025, train only on data before it.

import pandas as pd, numpy as np
from sklearn.linear_model import Ridge
from sklearn.ensemble import HistGradientBoostingRegressor

orders = pd.read_csv("retail/orders.csv", parse_dates=["order_date"])
items = pd.read_csv("retail/order_items.csv").merge(orders[["order_id", "order_date"]], on="order_id")
items = items.merge(pd.read_csv("retail/products.csv")[["product_id", "category"]], on="product_id")
items["month"] = items["order_date"].dt.to_period("M").dt.to_timestamp()
panel = items.groupby(["category", "month"])["quantity"].sum().rename("units").reset_index()
panel = panel.sort_values(["category", "month"])
g = panel.groupby("category")["units"]
panel["lag1"], panel["lag2"], panel["lag12"] = g.shift(1), g.shift(2), g.shift(12)
panel["roll3"] = g.shift(1).rolling(3).mean().reset_index(level=0, drop=True)
panel["month_no"] = panel["month"].dt.month
panel["t"] = (panel["month"].dt.year - 2023) * 12 + panel["month_no"]
data = pd.get_dummies(panel.dropna(), columns=["category"], dtype=float)
feats = [c for c in data.columns if c not in ("month", "units")]

res = []
for origin in pd.date_range("2025-01-01", "2025-12-01", freq="MS"):
    tr, te = data[data.month < origin], data[data.month == origin]
    preds = {
        "seasonal_naive": te["lag12"] * (tr.groupby("t")["units"].sum().iloc[-12:].sum() / tr.groupby("t")["lag12"].sum().iloc[-12:].sum()),
        "ridge": Ridge(alpha=1.0).fit(tr[feats], tr["units"]).predict(te[feats]),
        "gboost": HistGradientBoostingRegressor(max_depth=3, learning_rate=0.05, max_iter=300, random_state=0).fit(tr[feats], tr["units"]).predict(te[feats]),
    }
    for name, p in preds.items():
        res.append({"model": name, "abs_err": np.abs(te["units"].values - np.asarray(p)).sum(), "actual": te["units"].sum()})
r = pd.DataFrame(res).groupby("model")[["abs_err", "actual"]].sum()
r["WAPE_%"] = (100 * r["abs_err"] / r["actual"]).round(1)
print(r["WAPE_%"].sort_values())

model
seasonal_naive     7.8
ridge             10.7
gboost            20.0
Name: WAPE_%, dtype: float64

Reading the result. WAPE is the share of total units mis-forecast. Here the seasonal naive benchmark wins (≈8% WAPE), ridge regression is second and gradient boosting is clearly worse (≈20%). Category-level demand at Orion is dominated by a stable trend and yearly seasonality, which last year's same month × growth already captures; the boosted trees, trained on only a couple of hundred rows, overfit noise and cannot extrapolate the upward trend (trees never predict beyond the range they have seen). With only 6 categories and 36 months, a flexible model has little data to learn from — a common, honest outcome that should be reported, not hidden. Gains typically appear with more series (product-level), richer features (price, promotions, stock-outs) and longer history.

Model comparison done properly

• Same splits, same target, same metric for every model.
• Compare against baselines; report variability across folds/origins.
• Prefer the simpler model when performance is similar (cheaper, more robust, easier to explain).
• Look at errors by segment (ABC class, store, category) — an average can hide failure where it matters most.

Model limitations and business interpretation

Every model card should state: purpose, training data period, features, evaluation design and results vs baseline, known weaknesses (e.g. stock-out months understate demand; new products have no history), fairness considerations, and how predictions should be used ("order quantity suggestion reviewed by buyers", not "automatic purchase orders").

Translate accuracy into business terms: a 5-point WAPE improvement on 400k units is 20k fewer mis-forecast units — what does that mean in stock-outs avoided and inventory carried?

Deployment fundamentals

Practice

Summary

• Linear models are strong, interpretable baselines; gradient boosting captures non-linearity and interactions with careful tuning and early stopping.
• Evaluate forecasts by rolling-origin backtests against seasonal naive baselines, using WAPE/MAE.
• Report honestly — including when complex models don't win — and document limitations.
• Deploy pipelines, monitor drift and accuracy, and retrain deliberately.