Calibration, scoring rules and model drift
Separate calibration, discrimination and accuracy; score forecasts with Brier and log loss; read reliability diagrams and calibration by odds bucket; and monitor models for drift against a benchmark.
Learning objectives
- Compute the Brier score and log loss for a set of forecasts by hand.
- Distinguish calibration, discrimination and accuracy, and show two models with equal accuracy but different calibration.
- Read a reliability diagram and ECE, and check calibration by odds bucket with confidence intervals.
- Monitor for drift with scores relative to a benchmark rather than raw scores.
- Summarise Walsh & Joshi (2024) correctly, including its corrigendum.
Three different questions
- Calibration: do events forecast at 70% happen about 70% of the time?
- Discrimination: do events that happen tend to receive higher forecasts than events that do not?
- Accuracy: what share of yes/no calls at a 50% threshold are right?
Betting decisions compare a probability with a price, so the level of the probability matters, not just its ranking. Accuracy ignores the level entirely. The synthetic teaching data makes this concrete: the Calibration lab uses 1,200 events over 730 days, and a chronological split at day 365 gives 600 training and 600 test events. Nothing in it is real match data.
| Model | Accuracy | Brier | Log loss | ECE (10 bins) |
|---|---|---|---|---|
| Market baseline | 67.5% | 0.2039 | 0.5941 | 0.0386 |
| Overconfident model | 67.5% | 0.2201 | 0.6615 | 0.1291 |
| Overconfident, recalibrated on training data (T = 2.1) | 67.5% | 0.2037 | 0.5936 | 0.0254 |
The overconfident model ranks events exactly like the market but pushes probabilities towards 0 and 1. Its accuracy is identical; its Brier score, log loss and calibration error are clearly worse. A selection rule based on accuracy could not tell these models apart.
Proper scoring rules
Brier = (1/N) × Σ (p_i − y_i)²Log loss = −(1/N) × Σ [y_i·ln(p_i) + (1 − y_i)·ln(1 − p_i)]Both are strictly proper: your expected score is best when you report what you truly believe Gneiting & Raftery (2007). If an event really has a 60% chance, reporting 0.5, 0.6, 0.7 or 0.8 gives expected Brier scores of 0.2500, 0.2400, 0.2500 and 0.2800, and expected log losses of 0.6931, 0.6730, 0.6956 and 0.7777. Shading a forecast to look bolder can only hurt.
Log loss is harsher on overconfidence: a 90% forecast that fails costs 2.303, and a 99% forecast that fails costs 4.605. Always compare with benchmarks. On the test set, always forecasting the base rate (49.67%) scores Brier 0.2500 and log loss 0.6931. Even the true probabilities, unknowable in real life, score Brier 0.2029 — the floor is not zero, because outcomes are random.
Reliability diagrams, ECE and odds buckets
A reliability diagram groups forecasts into bins (here ten equal-width bins) and plots the mean forecast against the observed frequency. The overconfident model's top bin holds 102 forecasts averaging 94.4%, but only 81.4% happened (Wilson 95% interval 72.7%–87.7%). Its bottom bin holds 107 forecasts averaging 5.4%, of which 21.5% happened.
ECE = Σ_bins (n_b / N) × |observed_b − mean forecast_b|| Odds bucket | Events | Mean market probability | Observed rate (Wilson 95%) | ROI backing every event |
|---|---|---|---|---|
| 1.00–1.59 | 225 | 74.0% | 73.3% (67.2%–78.7%) | −5.7% |
| 1.60–2.19 | 109 | 51.9% | 50.5% (41.2%–59.7%) | −6.2% |
| 2.20–3.49 | 126 | 35.2% | 38.9% (30.8%–47.6%) | +3.7% |
| 3.50 and above | 140 | 19.3% | 20.7% (14.8%–28.2%) | +4.9% |
The two longer-odds buckets show positive ROI, yet each interval comfortably contains the market's own probability. This synthetic data has no built-in pricing bias, so those returns are noise. Real data behaves the same way: apparent single-season inefficiencies often fail to persist Winkelmann et al. (2024).
Monitoring edge decay and drift
Drift means the relationship between inputs and outcomes changes: rule changes, a new data feed, or a market that becomes sharper. Monitor scores relative to a benchmark on the same events, because raw scores move with the mix of events. In the test period the market's own Brier score was 0.2139 in the first half (301 events) and 0.1938 in the second (299), although the synthetic data contains no drift at all.
The relative score is steadier: the recalibrated model minus the market was −0.0010 in the first half and +0.0005 in the second. That is what ‘no edge, no drift’ looks like — a useful baseline for setting alarms. Recalibration fixed the model's calibration (ECE 0.0254) but added no information: a value rule backing events where model probability × odds > 1 placed 179 test bets for an ROI of +0.13%, while the uncalibrated model placed 271 bets at −7.38%.
Worked example
Worked example · Scoring four forecasts, then three models
Four hypothetical forecasts: Millford win 70% (happened), Dunmore win 40% (did not), Redcastle win 90% (did not), Ferrybank win 20% (did not). Then the synthetic test period from the calibration lab.
- Brier terms
(0.7 − 1)², (0.4 − 0)², (0.9 − 0)², (0.2 − 0)²0.09, 0.16, 0.81, 0.04 - Brier score
(0.09 + 0.16 + 0.81 + 0.04) / 4 = 1.10 / 40.275 - Log-loss terms
−ln 0.7, −ln 0.6, −ln 0.1, −ln 0.80.357, 0.511, 2.303, 0.223 - Log loss
3.393 / 40.848 — the failed 90% forecast supplies two thirds of it - Synthetic test: accuracy
Market vs overconfident model, 600 events67.5% vs 67.5% - Synthetic test: proper scores
Brier and log loss0.2039 / 0.5941 vs 0.2201 / 0.6615 - Recalibrate on training data only
temperature fitted on train data = 2.1, applied to testBrier 0.2037, log loss 0.5936, ECE 0.0254 - Does better calibration mean an edge?
Back events where p × odds > 1 in the test period179 bets, ROI +0.13%: no clear edge
Proper scores separated models that accuracy could not. Recalibration fixed the probability levels but could not add information the market lacked, so the honest conclusion is ‘calibrated, no clear edge’.
Practice · Scores, reliability and a leaky model
Open the Calibration and Backtesting lab with the split at day 365. For the test period, record Brier, log loss and ECE for the market baseline and the overconfident model, and read the top bin of each reliability diagram. Then look at the leaky model with and without point-in-time data.
Lab 4 · Calibration and backtesting
Synthetic · deterministic| Model | Train Brier | Test Brier | Test log loss | Test ECE |
|---|---|---|---|---|
| Market baseline | 0.2100 | 0.2039 | 0.5941 | 0.0386 |
| Overconfident model | 0.2272 | 0.2201 | 0.6615 | 0.1291 |
| Leaky model | 0.0887 | 0.0861 | 0.3206 | 0.1936 |
| Leaky model, deployed (no post-event feature) | – | 0.2039 | 0.5941 | 0.0386 |
| Overconfident, recalibrated (T = 2.10) | – | 0.2037 | 0.5936 | 0.0254 |
- Observed rate
- Perfect calibration
Show data table
| Forecast bucket midpoint (%) | Observed rate | Perfect calibration |
|---|---|---|
| 5% | 21% | 5% |
| 15% | 24% | 15% |
| 25% | 36% | 25% |
| 35% | 59% | 35% |
| 45% | 47% | 45% |
| 55% | 48% | 55% |
| 65% | 57% | 65% |
| 75% | 59% | 75% |
| 85% | 72% | 85% |
| 95% | 81% | 95% |
- Leakage: the leaky model scores far better than the market in the backtest, then matches the market exactly once the post-event feature is unavailable. A backtest is only as honest as its timestamps.
- Calibration: the overconfident model ranks events like the market but its probabilities are too extreme, so its log loss and ECE are worse. A single temperature fitted on the training period repairs much of it.
- Train/test separation: switch the fit to “All data” and the test score improves because the test period was used to fit — an optimistic estimate you could never achieve live.
- Win rate is not profit: backing favourites wins most of the time and still loses money after the margin.
Educational calculator. Uses only the numbers you enter or seeded synthetic data — no live prices, no staking, no real money.
Show what you should find
Test period (600 events): market Brier 0.2039, log loss 0.5941, ECE 0.0386; overconfident Brier 0.2201, log loss 0.6615, ECE 0.1291, with its top bin forecasting 94.4% against 81.4% observed. The leaky model looks brilliant (Brier 0.0861, log loss 0.3206) because it uses information recorded after the event; with point-in-time data it collapses to the market's 0.2039 and 0.5941. A score that is too good to be true usually is — see Backtesting, Leakage and Overfitting.
Common mistakes
- ✗ Selecting models by accuracy.The market and overconfident models both scored 67.5% accuracy but differed clearly in Brier (0.2039 vs 0.2201). Use proper scoring rules.
- ✗ Comparing raw Brier scores across different sets of events.The same market model scored 0.2139 and 0.1938 in two halves of one period. Compare models on identical events, or relative to a benchmark.
- ✗ Treating a low ECE as proof of a good model.ECE is noisy and binning-dependent: the true probabilities scored 0.0472 with ten bins on 600 events. Show bin counts and intervals.
- ✗ Citing Walsh & Joshi as evidence that calibrated models make money.Their corrected average returns were negative for both selection rules (about −9.8% and −26.8%).
When it fails
Supporting research
- Brier (1950): Verification of forecasts expressed in terms of probabilityMonthly Weather Review · Peer-reviewed methodsDefinition of the Brier score for probability forecasts.
- Gneiting & Raftery (2007): Strictly Proper Scoring Rules, Prediction, and EstimationJournal of the American Statistical Association · Peer-reviewed theoryStrictly proper scoring rules reward honest probability forecasts.
- Walsh & Joshi (2024): Machine learning for sports betting: Should model selection be based on accuracy or calibration?Machine Learning with Applications · Peer-reviewed empiricalCalibration-based selection beat accuracy-based selection, but corrected average returns were negative for both.
- Winkelmann et al. (2024): Are Betting Markets Inefficient? Evidence From Simulations and Real DataJournal of Sports Economics · Peer-reviewed empiricalApparent inefficiencies in single seasons often do not persist, so bucket-level ROI needs caution.
Check your understanding
Practice only: saved on this device, no XP and no account record.