Calibration and Backtesting
Score synthetic forecasts with Brier and log loss, read a reliability diagram, split by date and watch a leaky model collapse.
Lab 4 · Calibration and backtesting
Synthetic · deterministic1,200 synthetic events over two years with a known true probability, a margin-inclusive price and an outcome. Move the train/test date and compare three forecasting models.
| Model | Train Brier | Test Brier | Test log loss | Test ECE |
|---|---|---|---|---|
| Market baseline | 0.2100 | 0.2039 | 0.5941 | 0.0386 |
| Overconfident model | 0.2272 | 0.2201 | 0.6615 | 0.1291 |
| Leaky model | 0.0887 | 0.0861 | 0.3206 | 0.1936 |
| Leaky model, deployed (no post-event feature) | – | 0.2039 | 0.5941 | 0.0386 |
| Overconfident, recalibrated (T = 2.10) | – | 0.2037 | 0.5936 | 0.0254 |
- Observed rate
- Perfect calibration
Show data table
| Forecast bucket midpoint (%) | Observed rate | Perfect calibration |
|---|---|---|
| 5% | 21% | 5% |
| 15% | 24% | 15% |
| 25% | 36% | 25% |
| 35% | 59% | 35% |
| 45% | 47% | 45% |
| 55% | 48% | 55% |
| 65% | 57% | 65% |
| 75% | 59% | 75% |
| 85% | 72% | 85% |
| 95% | 81% | 95% |
Favourites (≥65%) win rate76.2%
Favourites ROI at offered odds−5.7%
Favourite bets in test181
Fitted temperature2.10
- Leakage: the leaky model scores far better than the market in the backtest, then matches the market exactly once the post-event feature is unavailable. A backtest is only as honest as its timestamps.
- Calibration: the overconfident model ranks events like the market but its probabilities are too extreme, so its log loss and ECE are worse. A single temperature fitted on the training period repairs much of it.
- Train/test separation: switch the fit to “All data” and the test score improves because the test period was used to fit — an optimistic estimate you could never achieve live.
- Win rate is not profit: backing favourites wins most of the time and still loses money after the margin.
Educational calculator. Uses only the numbers you enter or seeded synthetic data — no live prices, no staking, no real money.