Anatomy of a false-positive backtest

The case: a published profit and its correction

Nothing about this is unusual or scandalous. Betting returns are heavy-tailed: one long-odds winner pays as much as dozens of short-odds winners, so a single bad price can dominate a sample. In the synthetic data below, one 10-unit winner at 41.00 earns 400 units — the profit of about 36 winners at 2.10. The lesson is procedural: every backtest needs row-level checks before anyone looks at the ROI.

1. Price sanity: ratio of the recorded price to an independent consensus; investigate anything far outside the usual range.
2. Timing: the quote capture time must precede kickoff (and the decision time of the rule).
3. Uniqueness: one row per match, selection, quote time and price.
4. Executability: stake no larger than the maximum the bookmaker accepted at that time.
5. Settlement: results checked against a second source, under the market's own settlement rules.

The synthetic dataset

The Replication Challenge uses 27 fictional bets. The rule: back a selection when the recorded price is at least 7.5% above the median of other bookmakers. Stakes are 10 units, and the 7.5% threshold was chosen after trying 40 thresholds on this same sample. Twenty-two rows are clean; five carry planted flaws that mirror failures in the literature.

Peeling back the flaws one at a time

Row B23 alone contributes 10 × (41 − 1) = 400 units, or 79.7% of the headline profit. That is the same shape as the real correction, where one erroneous long price explained much of the result. The remaining flaws matter too — together they account for another 98 units.

Now look harder at the corrected +1.4%. It depends on B27, a winner that still carries a 25-unit stake when every other bet is 10. At flat one-unit stakes the 23 corrected bets lose 2.12 units, a ROI of −9.2%; the 22 clean rows alone return −17.1% (−37.7 on 220 staked). The win rate is 9 of 23 (39.1%), with a Wilson 95% interval of 22.2% to 59.2% — comfortably containing the 40.1% break-even implied by the average odds of 2.49.

The threshold was also a choice

The rule's 7.5% threshold was the winner of 40 trials. If each trial were an independent test at the 5% level with no real effect, the chance that at least one looks significant would be 1 − 0.95⁴⁰ ≈ 87.1%. Real thresholds are correlated, so the exact figure is lower, but the direction is the point made by Bailey et al. (2014): the best of many backtests is biased upwards. A Bonferroni-style bar would require p < 0.05 / 40 = 0.00125.

When you write up an audit like this, report the headline and corrected figures side by side, list each changed row with its reason, give the profit attributable to each correction, and state the number of rule variants tried. Then say what the corrected sample can and cannot support. Here that is short: 23 bets cannot establish an edge in either direction, so the next step is new data, not a new threshold.

Worked example

Common mistakes

• ✗ Checking only the ROI, not the rows that produce it.Sort rows by profit contribution. If one row supplies most of the profit, verify it against an independent source before anything else.
• ✗ Deleting only losing rows that look odd.Apply the same checks to winners and losers. Here every flagged row is a winner: flawed rows that happen to win are exactly the ones that inflate a headline, so they deserve the hardest look.
• ✗ Reporting the corrected +1.4% as a smaller but real edge.With 23 bets the ROI standard error is about 25 points, and flat stakes give −9.2%. Report it as no evidence either way.
• ✗ Forgetting that the threshold was tuned on the same sample.Report the number of trials (40 here) and test the frozen rule on later data, as Clegg & Cartlidge (2025) did with post-2020 data.

When it fails

Supporting research

• Ramirez, Reade & Singleton (2023): Betting on a buzz: Mispricing and inefficiency in online sportsbooksInternational Journal of Forecasting · Peer-reviewed empiricalThe original attention-based mispricing study whose headline profitability was later corrected.
• Clegg & Cartlidge (2025): Not feeling the buzz: Correction study of mispricing and inefficiency in online sportsbooksInternational Journal of Forecasting · Peer-reviewed replicationThe correction: one erroneous outlier price drove much of the profit, returns came mainly from bookmaker mispricing, and no profits appeared after 2020.
• Bailey et al. (2014): Pseudo-Mathematics and Financial Charlatanism: The Effects of Backtest Overfitting on Out-of-Sample PerformanceNotices of the American Mathematical Society · Peer-reviewed theoryWhy a threshold chosen from many trials inflates backtest performance.

Check your understanding