Backtesting without fooling yourself
Most impressive betting backtests fail for mundane reasons: data that were not available at the time, duplicated rows, too many tries on the same history. Learn point-in-time discipline, multiple-testing corrections and walk-forward validation.
Learning objectives
- Build point-in-time datasets using as-of timestamp joins and identify look-ahead, duplicate and survivorship errors.
- Compute the family-wise false-positive rate for k tests and the matching Bonferroni threshold.
- Run a chronological or walk-forward validation and explain why in-sample results overstate performance.
- Re-run a flawed backtest after correcting its data and report how the headline changes.
Point-in-time data and timestamp joins
A backtest asks: what would have happened if I had followed this rule, knowing only what I could have known at the time? Every price and every feature therefore needs a timestamp for when it became available, and every join must be an as-of join: take the latest value recorded before the decision time, never a value from later.
quote_time ≤ decision_time < kick-off, feature_time ≤ decision_time- Look-ahead prices: quotes captured after kick-off, or closing odds used as if available in the morning.
- Look-ahead features: season averages that include the match being predicted; confirmed line-ups used for a decision made before they were published.
- Duplicate markets: the same quote stored twice, or alternative lines (−0.5 and −0.75) counted as independent bets on the same outcome.
- Survivorship: only leagues, books or teams that still exist today; strategies kept because they 'survived' earlier tests.
Multiple testing: why the best result is biased
Test one rule at the 5% level and a skill-free rule passes 5% of the time. Test many and the chance that at least one passes climbs quickly.
FWER = 1 − (1 − α)ᵏ Bonferroni: α_per test = α / kOut-of-sample and walk-forward validation
Split by date, never at random: train on earlier matches, test on later ones, and do not look at the test period until every choice is frozen. Walk-forward validation repeats this: fit on a window, test on the next block, roll forward, and report the stitched-together out-of-sample results.
Three details make walk-forward honest. Refit only with data available at each refit date, including any hyperparameters such as a time-decay rate. Leave a gap between training and test blocks when features use rolling windows, so the last training rows do not share matches with the first test rows. And once you have looked at a test block and changed the model because of it, that block has become training data: you need fresh, untouched data for the next claim.
A small synthetic illustration: on the first 365 days of the lab dataset (600 events) we tried 21 favourite thresholds from 0.30 to 0.80. The best, 0.725, returned −3.14% on 131 bets in sample. On the following 365 days it returned −5.81% on 125 bets. The selected rule looked better than its neighbours mostly because it was selected; with 21 tries, the chance of at least one 5%-level false positive would be 65.94%.
Split the synthetic data by date and compare the market baseline, an overconfident model and a leaky model.
Lab 4 · Calibration and backtesting
Synthetic · deterministic| Model | Train Brier | Test Brier | Test log loss | Test ECE |
|---|---|---|---|---|
| Market baseline | 0.2100 | 0.2039 | 0.5941 | 0.0386 |
| Overconfident model | 0.2272 | 0.2201 | 0.6615 | 0.1291 |
| Leaky model | 0.0887 | 0.0861 | 0.3206 | 0.1936 |
| Leaky model, deployed (no post-event feature) | – | 0.2039 | 0.5941 | 0.0386 |
| Overconfident, recalibrated (T = 2.10) | – | 0.2037 | 0.5936 | 0.0254 |
- Observed rate
- Perfect calibration
Show data table
| Forecast bucket midpoint (%) | Observed rate | Perfect calibration |
|---|---|---|
| 5% | 21% | 5% |
| 15% | 24% | 15% |
| 25% | 36% | 25% |
| 35% | 59% | 35% |
| 45% | 47% | 45% |
| 55% | 48% | 55% |
| 65% | 57% | 65% |
| 75% | 59% | 75% |
| 85% | 72% | 85% |
| 95% | 81% | 95% |
- Leakage: the leaky model scores far better than the market in the backtest, then matches the market exactly once the post-event feature is unavailable. A backtest is only as honest as its timestamps.
- Calibration: the overconfident model ranks events like the market but its probabilities are too extreme, so its log loss and ECE are worse. A single temperature fitted on the training period repairs much of it.
- Train/test separation: switch the fit to “All data” and the test score improves because the test period was used to fit — an optimistic estimate you could never achieve live.
- Win rate is not profit: backing favourites wins most of the time and still loses money after the margin.
Educational calculator. Uses only the numbers you enter or seeded synthetic data — no live prices, no staking, no real money.
A published profit that was one bad row
Lesson 21 studies that correction in depth. The worked example below reproduces the same failure in miniature with the synthetic rows of the replication lab.
- Profit concentration: what share of total profit comes from the top one, five and ten bets? If one row dominates, inspect it first.
- Price sanity: flag any recorded price far from the consensus at the same time (lesson 7); a tenfold gap is almost always an error.
- Uniqueness: enforce one row per event, market key, selection and quote time.
- Executability: cap every stake at the limit the book accepted at the time, and drop bets the operator later voided.
- Pre-registration: write down the rule, the data window and the success test before running the out-of-sample period.
Find the bad price, the look-ahead quotes, the duplicate and the unexecutable stake before reading the worked example.
Lab 5 · Research replication challenge
Synthetic · deterministic| ID | Match · pick | Kickoff | Quoted at | Odds | Consensus | Stake / max | Result | Your flag |
|---|---|---|---|---|---|---|---|---|
| B01 | Harbour City v Northvale · Home | 08-16 14:00 | 08-16 11:05 | 2.10 | 1.94 | 10 / 25 | Won | |
| B02 | Ashbridge v Millford · Away | 08-16 14:00 | 08-16 10:40 | 3.40 | 3.14 | 10 / 25 | Lost | |
| B03 | Kestrel Park v Dunmore · Home | 08-17 13:00 | 08-17 09:15 | 1.95 | 1.80 | 10 / 25 | Won | |
| B04 | Redcastle v Ferrybank · Draw | 08-23 14:00 | 08-23 12:10 | 2.60 | 2.40 | 10 / 25 | Lost | |
| B05 | Northvale v Glenrock · Away | 08-23 16:30 | 08-23 13:00 | 2.25 | 2.08 | 10 / 25 | Lost | |
| B06 | Millford v Harbour City · Away | 08-30 14:00 | 08-30 10:20 | 3.10 | 2.87 | 10 / 25 | Won | |
| B07 | Dunmore v Ashbridge · Home | 08-30 14:00 | 08-30 11:45 | 2.05 | 1.89 | 10 / 25 | Lost | |
| B08 | Glenrock v Kestrel Park · Away | 08-31 15:30 | 08-31 12:30 | 1.90 | 1.75 | 10 / 25 | Won | |
| B09 | Ferrybank v Redcastle · Draw | 09-13 14:00 | 09-13 09:50 | 2.80 | 2.59 | 10 / 25 | Lost | |
| B10 | Harbour City v Dunmore · Home | 09-13 16:30 | 09-13 14:00 | 2.40 | 2.22 | 10 / 25 | Won | |
| B11 | Ashbridge v Glenrock · Away | 09-20 14:00 | 09-20 10:10 | 3.60 | 3.33 | 10 / 25 | Lost | |
| B12 | Millford v Northvale · Home | 09-20 14:00 | 09-20 11:30 | 2.00 | 1.85 | 10 / 25 | Lost | |
| B13 | Redcastle v Kestrel Park · Away | 09-27 14:00 | 09-27 09:40 | 2.30 | 2.12 | 10 / 25 | Won | |
| B14 | Dunmore v Ferrybank · Draw | 09-27 16:30 | 09-27 13:20 | 2.75 | 2.54 | 10 / 25 | Lost | |
| B15 | Glenrock v Harbour City · Home | 10-04 14:00 | 10-04 10:00 | 1.98 | 1.83 | 10 / 25 | Won | |
| B16 | Northvale v Ashbridge · Away | 10-04 14:00 | 10-04 11:15 | 3.25 | 3.00 | 10 / 25 | Lost | |
| B17 | Kestrel Park v Millford · Home | 10-18 14:00 | 10-18 10:25 | 2.15 | 1.99 | 10 / 25 | Lost | |
| B18 | Ferrybank v Glenrock · Away | 10-18 16:30 | 10-18 12:50 | 2.50 | 2.31 | 10 / 25 | Won | |
| B19 | Harbour City v Redcastle · Home | 10-25 14:00 | 10-25 09:30 | 2.20 | 2.03 | 10 / 25 | Lost | |
| B20 | Ashbridge v Dunmore · Draw | 10-25 14:00 | 10-25 11:40 | 3.00 | 2.77 | 10 / 25 | Lost | |
| B21 | Millford v Kestrel Park · Away | 11-01 14:00 | 11-01 10:05 | 2.45 | 2.26 | 10 / 25 | Lost | |
| B22 | Northvale v Ferrybank · Home | 11-01 16:30 | 11-01 13:10 | 1.92 | 1.77 | 10 / 25 | Lost | |
| B23 | Kestrel Park v Ashbridge · Away | 11-08 14:00 | 11-08 10:30 | 41.00 | 4.10 | 10 / 25 | Won | |
| B24 | Redcastle v Northvale · Home | 11-08 14:00 | 11-08 14:17 | 2.90 | 2.12 | 10 / 25 | Won | |
| B25 | Glenrock v Millford · Draw | 11-15 14:00 | 11-15 16:05 | 2.70 | 2.49 | 10 / 25 | Won | |
| B26 | Millford v Harbour City · Away | 08-30 14:00 | 08-30 10:20 | 3.10 | 2.87 | 10 / 25 | Won | |
| B27 | Dunmore v Glenrock · Home | 11-22 14:00 | 11-22 12:00 | 2.65 | 2.45 | 50 / 25 | Won |
Educational calculator. Uses only the numbers you enter or seeded synthetic data — no live prices, no staking, no real money.
Worked example
Worked example · Cleaning a 161.87% backtest
The Research Replication Challenge holds 27 synthetic bets (fictional teams) selected by the rule 'bet when the recorded price is at least 7.5% above the consensus'. The 7.5% threshold was chosen after testing 40 thresholds on the same sample. Stakes are 10 units except one.
- Headline result
All 27 recorded rows27 bets, 13 wins, 310 staked, profit +501.80, ROI +161.87% - Bad price (B23)
Recorded 41.0 against a consensus of 4.10; this one win contributes 10 × 40 = 400 unitsRemove it: 26 bets, profit +101.80, ROI +33.93% - Look-ahead quotes (B24, B25)
Quoted 17 minutes and 2 hours 5 minutes after kick-offRemove both: 24 bets, 280 staked, profit +65.80, ROI +23.50% - Duplicate (B26)
Same match, selection, time and price as B06Remove it: 23 bets, 270 staked, profit +44.80, ROI +16.59% - Unexecutable stake (B27)
Stake 50 against a maximum of 25Cap at 25: 23 bets, 245 staked, profit +3.55, ROI +1.45% - The 22 rows with no planted flaw
B01–B22 only22 bets, 8 wins, profit −37.70, ROI −17.14% - Multiple testing on the threshold
1 − (1 − 0.05)^40 for independent null tests; Bonferroni 0.05 / 40Illustrative FWER 87.15%; shared-data thresholds need not be independent; per-test threshold 0.00125
Each correction is mechanical, yet together they turn +161.87% into +1.45% on 23 bets — and that small positive figure rests on a single 25-unit win (B27); the 22 unflawed rows lost 17.14%. With the threshold chosen from 40 tries, the honest conclusion is no evidence of an edge.
Practice · Watch a leaky model collapse
Open the Calibration and Backtesting lab with the chronological split at day 365. Compare test-period Brier score and log loss for the market baseline, the overconfident model and the leaky model, then switch on point-in-time features.
Lab 4 · Calibration and backtesting
Synthetic · deterministic| Model | Train Brier | Test Brier | Test log loss | Test ECE |
|---|---|---|---|---|
| Market baseline | 0.2100 | 0.2039 | 0.5941 | 0.0386 |
| Overconfident model | 0.2272 | 0.2201 | 0.6615 | 0.1291 |
| Leaky model | 0.0887 | 0.0861 | 0.3206 | 0.1936 |
| Leaky model, deployed (no post-event feature) | – | 0.2039 | 0.5941 | 0.0386 |
| Overconfident, recalibrated (T = 2.10) | – | 0.2037 | 0.5936 | 0.0254 |
- Observed rate
- Perfect calibration
Show data table
| Forecast bucket midpoint (%) | Observed rate | Perfect calibration |
|---|---|---|
| 5% | 21% | 5% |
| 15% | 24% | 15% |
| 25% | 36% | 25% |
| 35% | 59% | 35% |
| 45% | 47% | 45% |
| 55% | 48% | 55% |
| 65% | 57% | 65% |
| 75% | 59% | 75% |
| 85% | 72% | 85% |
| 95% | 81% | 95% |
- Leakage: the leaky model scores far better than the market in the backtest, then matches the market exactly once the post-event feature is unavailable. A backtest is only as honest as its timestamps.
- Calibration: the overconfident model ranks events like the market but its probabilities are too extreme, so its log loss and ECE are worse. A single temperature fitted on the training period repairs much of it.
- Train/test separation: switch the fit to “All data” and the test score improves because the test period was used to fit — an optimistic estimate you could never achieve live.
- Win rate is not profit: backing favourites wins most of the time and still loses money after the margin.
Educational calculator. Uses only the numbers you enter or seeded synthetic data — no live prices, no staking, no real money.
Show what you should find
The split gives 600 training and 600 test events. On the test period the market baseline scores Brier 0.2039 and log loss 0.5941; the overconfident model is worse (Brier 0.2201, log loss 0.6615); the leaky model looks brilliant (Brier 0.0861, log loss 0.3206) even though its calibration error (0.1936) is far worse than the market's (0.0386). With point-in-time data the leaky model falls back to exactly the market's scores (Brier 0.2039), because its 'skill' was the outcome leaking into the feature.
Common mistakes
- ✗ Joining prices to matches by date only.Use as-of joins on timestamps: the quote must precede both the decision and kick-off.
- ✗ Reporting the best of many variants without the count.Report every trial. With 20 tests at 5%, the chance of at least one false positive is 64.15%; Bonferroni requires p < 0.0025.
- ✗ Random train/test splits for time-ordered matches.Split chronologically or walk forward; random splits leak future information about team strength into training.
- ✗ Trusting the headline ROI when one row dominates profit.Check profit concentration. One erroneous 41.0 price produced 400 of 501.80 units in the example.
When it fails
Supporting research
- Bailey et al. (2014): Pseudo-Mathematics and Financial Charlatanism: The Effects of Backtest Overfitting on Out-of-Sample PerformanceNotices of the American Mathematical Society · Peer-reviewed theoryTesting many variants on one history inflates the best backtest; the number of trials must be reported.
- Winkelmann et al. (2024): Are Betting Markets Inefficient? Evidence From Simulations and Real DataJournal of Sports Economics · Peer-reviewed empiricalSignificant single-season results are likely even in efficient markets, so validation must span periods.
- Clegg & Cartlidge (2025): Not feeling the buzz: Correction study of mispricing and inefficiency in online sportsbooksInternational Journal of Forecasting · Peer-reviewed replicationA published out-of-sample profit that largely depended on one erroneous long price and did not persist after 2020.
Check your understanding
Practice only: saved on this device, no XP and no account record.