Audit a machine-learning sports forecast
Audit a forecasting model the way a reviewer would: base rates, calibration by bucket, Brier and log loss against a market baseline, leakage checks, multiple model comparisons and drift — then read Walsh & Joshi (2024) with its corrigendum.
Learning objectives
- Compare a model's Brier score and log loss with both a base-rate forecast and a de-vigged market baseline.
- Read a reliability table and diagnose over- or under-confidence by bucket.
- Detect leakage by re-scoring a model with point-in-time features only.
- Explain why one test season and a best-of-many model are weak evidence, using the corrected Walsh & Joshi figures.
Start with base rates and a market baseline
The Academy's synthetic dataset has 1,200 binary events over 730 days, each with a de-vigged market probability and offered odds carrying a 5% margin. Split chronologically at day 365, it gives 600 training and 600 test events. The test base rate is 49.67% (training: 51.0%).
Brier = (1/n) Σ (p_i − y_i)² Log loss = −(1/n) Σ [y_i ln p_i + (1 − y_i) ln(1 − p_i)]| Forecast | Brier | Log loss | ECE |
|---|---|---|---|
| Constant 51.0% (training base rate) | 0.2502 | 0.6935 | — |
| Market (de-vigged) | 0.2039 | 0.5941 | 0.0386 |
| Overconfident model | 0.2201 | 0.6615 | 0.1291 |
| Overconfident, temperature 2.10 fitted on training | 0.2037 | 0.5936 | 0.0254 |
| Leaky model (backtest) | 0.0861 | 0.3206 | 0.1936 |
| Leaky model, point-in-time features only | 0.2039 | 0.5941 | 0.0386 |
The base-rate forecast is the floor any model must clear; the market is the bar that matters for betting. A model that beats the base rate but not the market has learned what the price already knows.
Calibration by bucket
| Bucket | Events | Mean forecast | Observed rate |
|---|---|---|---|
| 0.0–0.1 | 107 | 5.4% | 21.5% |
| 0.1–0.2 | 78 | 14.4% | 24.4% |
| 0.3–0.4 | 44 | 34.5% | 59.1% |
| 0.5–0.6 | 40 | 54.2% | 47.5% |
| 0.8–0.9 | 58 | 85.1% | 72.4% |
| 0.9–1.0 | 102 | 94.4% | 81.4% |
The signature of overconfidence is visible at both ends: events the model calls near-impossible happen about a fifth of the time, and its near-certainties fail almost a fifth of the time. Expected calibration error (the count-weighted gap between forecast and outcome across ten buckets) is 0.1291, more than three times the market's 0.0386. Rescaling with a single temperature fitted on the training period (T = 2.10) brings ECE down to 0.0254 and the Brier score to 0.2037.
Leakage checks
The leaky model uses a feature recorded after the event. Its backtest looks extraordinary: Brier 0.0861, and a rule that bets whenever its probability times the odds exceeds one places 287 test bets, wins all 287 and returns +102.1%. Re-scored with point-in-time features only, it collapses exactly onto the market: Brier 0.2039 and no qualifying bets at all.
- Every feature has a timestamp, and the join to each event is as of the decision time.
- Re-score the model with features truncated to the decision time; any large drop is a leak.
- Treat near-perfect results (a 100% win rate, Brier far below the market) as a bug until proven otherwise.
- Check that scaling, imputation and feature selection were fitted on training rows only.
Multiple models, betting translation and drift
The recalibrated model beats the market by 0.0002 in Brier score. Turned into a rule — bet when model probability × odds > 1 — it makes 179 test bets with 39 wins and a ROI of +0.13%. In this synthetic world we know the true average EV of those 179 bets: −1.91%. The ROI standard error at their 4.83 average odds is about 14.6 points, so the sample cannot separate the two.
Drift is the next check. Split the test year in two: in the first half (301 events) the recalibrated model scores 0.2129 against the market's 0.2139; in the second half (299 events) it scores 0.1944 against 0.1938. The ranking flips. And if 20 model variants had been compared at the 5% level, the chance that at least one looks significantly better by luck would be 64.15%.
Win rate is a separate trap. The raw overconfident model, betting whenever its probability times the odds exceeds one, places 271 test bets and wins 188 of them — a 69.4% strike rate — yet loses 20.01 units, a ROI of −7.38%. Every one of those bets is priced below 2.00 (average 1.37): overconfidence pushes the model towards short-priced favourites it rates as near-certain. For deployment, monitor a rolling Brier score against the market on new data, re-fit calibration on a fixed schedule using only past data, and set in advance the gap that would trigger withdrawal.
Worked example
Worked example · A one-page audit of the overconfident model
A vendor submits the overconfident model with a claim that it is "better calibrated than the market after tuning". You have the 1,200-event synthetic dataset and split at day 365.
- Base rate
Test outcomes: 298 of 60049.67%; constant forecast Brier 0.2502, log loss 0.6935 - Market baseline
test-period scores, market modelBrier 0.2039, log loss 0.5941, ECE 0.0386 - Raw model
test-period scores, overconfident modelBrier 0.2201, log loss 0.6615, ECE 0.1291: worse than the market - Calibrate on training only
temperature fitted on train data → T = 2.10Brier 0.2037, log loss 0.5936, ECE 0.0254 - Gain over the market
0.2037 − 0.2039; 0.5936 − 0.5941−0.0002 Brier, −0.0005 log loss: negligible - Betting translation
Bet when p × odds > 1 at the offered odds179 bets, 39 wins, ROI +0.13% (ROI s.e. ≈ 14.6 points) - Drift
Brier by half of the test year: model vs market0.2129 vs 0.2139, then 0.1944 vs 0.1938: ranking flips
After calibration the model is essentially the market with extra noise. Its gain is a fraction of a thousandth of a Brier point, its betting rule is indistinguishable from zero, and its advantage reverses within the test year. Audit verdict: calibrated, but no forecasting or betting edge demonstrated.
Practice · Watch a leaky model collapse
Open the Calibration and backtesting lab with the split at day 365. Compare the market, overconfident and leaky models on the test period, then switch the leaky model to point-in-time features. Note the Brier score, log loss and the reliability diagram each time.
Lab 4 · Calibration and backtesting
Synthetic · deterministic| Model | Train Brier | Test Brier | Test log loss | Test ECE |
|---|---|---|---|---|
| Market baseline | 0.2100 | 0.2039 | 0.5941 | 0.0386 |
| Overconfident model | 0.2272 | 0.2201 | 0.6615 | 0.1291 |
| Leaky model | 0.0887 | 0.0861 | 0.3206 | 0.1936 |
| Leaky model, deployed (no post-event feature) | – | 0.2039 | 0.5941 | 0.0386 |
| Overconfident, recalibrated (T = 2.10) | – | 0.2037 | 0.5936 | 0.0254 |
- Observed rate
- Perfect calibration
Show data table
| Forecast bucket midpoint (%) | Observed rate | Perfect calibration |
|---|---|---|
| 5% | 21% | 5% |
| 15% | 24% | 15% |
| 25% | 36% | 25% |
| 35% | 59% | 35% |
| 45% | 47% | 45% |
| 55% | 48% | 55% |
| 65% | 57% | 65% |
| 75% | 59% | 75% |
| 85% | 72% | 85% |
| 95% | 81% | 95% |
- Leakage: the leaky model scores far better than the market in the backtest, then matches the market exactly once the post-event feature is unavailable. A backtest is only as honest as its timestamps.
- Calibration: the overconfident model ranks events like the market but its probabilities are too extreme, so its log loss and ECE are worse. A single temperature fitted on the training period repairs much of it.
- Train/test separation: switch the fit to “All data” and the test score improves because the test period was used to fit — an optimistic estimate you could never achieve live.
- Win rate is not profit: backing favourites wins most of the time and still loses money after the margin.
Educational calculator. Uses only the numbers you enter or seeded synthetic data — no live prices, no staking, no real money.
Show what you should find
Market: Brier 0.2039, log loss 0.5941, ECE 0.0386. Overconfident: Brier 0.2201, log loss 0.6615, ECE 0.1291, with low buckets under-forecast and high buckets over-forecast. Leaky: Brier 0.0861 and log loss 0.3206 — far better than the market, which is the red flag. With point-in-time features the leaky model returns exactly to the market's 0.2039 / 0.5941. The 'skill' was information from after the event.
Common mistakes
- ✗ Comparing a model only with a base-rate forecast.For betting questions the baseline is the de-vigged market. Beating 0.2502 is easy; beating 0.2039 is the test.
- ✗ Celebrating a near-perfect backtest.A 287-from-287 record is a leakage alarm. Re-score with point-in-time features before anything else.
- ✗ Quoting the original Walsh & Joshi return figures.Use the corrigendum: average returns about −9.8% (calibration) and −26.8% (accuracy); the original figures were withdrawn Walsh & Joshi (2024).
- ✗ Reporting the best of many models as the expected performance.The maximum of many noisy results is biased upwards. Report the average across variants and the number tried.
When it fails
Supporting research
- Walsh & Joshi (2024): Machine learning for sports betting: Should model selection be based on accuracy or calibration?Machine Learning with Applications · Peer-reviewed empiricalCalibration-based selection did better than accuracy-based selection, but corrected average returns were negative for both; one test season.
- Brier (1950): Verification of forecasts expressed in terms of probabilityMonthly Weather Review · Peer-reviewed methodsThe Brier score as a forecast-verification measure.
- Gneiting & Raftery (2007): Strictly Proper Scoring Rules, Prediction, and EstimationJournal of the American Statistical Association · Peer-reviewed theoryWhy Brier and log loss are strictly proper scores that reward honest probabilities.
- Bailey et al. (2014): Pseudo-Mathematics and Financial Charlatanism: The Effects of Backtest Overfitting on Out-of-Sample PerformanceNotices of the American Mathematical Society · Peer-reviewed theoryWhy the best of many tested models overstates out-of-sample performance.
Check your understanding
Practice only: saved on this device, no XP and no account record.