A Profitable Backtest That Never Existed

A research note that is too good

Imagine receiving a short research note. Its rule is simple: back any selection whose recorded price is at least 7.5% above the median of other bookmakers. Over 27 bets it staked 310 units and made 501.8 units of profit — a return of +161.9%. The note includes the full bet log. Everything in it is fictional and synthetic, but every flaw is modelled on errors that have appeared in real research.

The right first reaction to a number like that is not excitement. It is a question: what would have to be true for this to be real? Extraordinary results need extraordinary verification.

Flaw 1: one bad price observation

Row B23 records a winning bet on an away side at 41.00. The median price of the other bookmakers at the same moment was 4.10. A tenfold gap on an ordinary football market is not an opportunity; it is almost certainly a misplaced decimal point or a scraping error, and no bookmaker offered it. That single row contributes 400 of the 501.8 units of profit — about 80% of the entire claimed return.

The defence is mechanical: compare every recorded price with a consensus, flag anything implausibly far from it, and report results with and without the largest winners. Removing B23 alone takes the claimed return from +161.9% to +33.9% — still impressive, and still not real.

Flaws 2 and 3: quotes from the future

Row B24 was quoted at 14:17 for a match that kicked off at 14:00. Row B25 was 'quoted' at 16:05 for a 14:00 kickoff — after the final whistle, almost certainly the result of joining a post-match data snapshot to the wrong table. A pre-match strategy could never have taken either price. Both won.

This is look-ahead bias: using information that was not available at decision time. It rarely looks as obvious as a timestamp after kickoff. It hides in features computed from full-season data, in closing prices used as if they were available at opening, and in team ratings updated with the result being predicted. The cure is point-in-time data: every input stamped with when it became known, and every join made only on information that existed before the decision.

Flaw 4: the same bet counted twice

Row B26 is identical to B06: same match, same selection, same quote time, same price. Duplicates appear when data is merged from several feeds, when markets are listed under two names, or when a scraper re-reads a page. Each duplicate of a winner inflates profit; each duplicate of anything shrinks the apparent uncertainty, because the sample looks larger than it is.

Flaw 5: a stake that could not be placed

Row B27 staked 50 units where the bookmaker's maximum for that market was 25. Only 25 could have been placed. Real execution adds more friction: prices move when you submit, some bets are refused, and accounts that win consistently are often limited — a problem reported by authors who traded a consensus-outlier strategy for real Kaunitz, Zhong & Kreiner (2017). A backtest that ignores limits is a backtest of a market that does not exist.

The corrected result

After removing the bad price, the two look-ahead rows and the duplicate, and capping B27 at its maximum stake, the strategy made 3.55 units on 245 staked: +1.4% on 23 bets. With average odds of about 2.5, the standard error of the return on 23 such bets is roughly 25 percentage points. The corrected result is statistically indistinguishable from zero — or from a loss.

A statistical audit of what remains

Suppose the cleaned log were taken at face value. Nine of its 23 bets won: a hit rate of 39.1%, with a 95% Wilson interval of 22.2% to 59.2%. The average recorded price is about 2.49, so a bettor with no skill at all would need roughly a 40.1% hit rate just to break even. The observed rate is not even above that line, and the interval spans everything from disastrous to excellent.

Another way to see it: if every bet had exactly break-even odds of winning, the chance of nine or more winners out of 23 would be about 62%. The corrected result is exactly what you would expect from no edge at all. Twenty-three bets cannot separate a good strategy from a bad one; that requires hundreds or thousands of independent, executable bets — and even then only if the rule was fixed before the data were examined.

The quieter flaw: choosing the rule after seeing the data

The note mentions, in passing, that the 7.5% threshold was the best of 40 thresholds tried on the same sample. For 40 independent null tests at a 5% significance level, the chance of at least one false positive is 87.1%. Thresholds on the same sample are correlated, so that number is an illustration, not their measured error rate. Selection still requires a multiple-testing adjustment or an untouched holdout.

Survivorship bias and the data you never see

A backtest is also shaped by what is missing. Bookmakers that closed, leagues that were dropped from a feed, markets that were suspended and bets that were refused all vanish from historical datasets. If the survivors differ systematically — more liquid leagues, sharper books, unrestricted accounts — results drawn from them overstate what a real bettor could have achieved. Record what you excluded and why.

The same applies to strategies. Researchers and bettors rarely publish the ideas that failed, so the strategies we hear about are a selected sample of the ones that happened to work. A single published success is therefore weaker evidence than it looks; ask how many similar ideas were tried and abandoned.

Walk-forward validation

The honest alternative to a single in-sample result is walk-forward testing: choose the rule and its parameters using only data before a date, test on the period after it, then roll forward and repeat. Every test period is genuinely out of sample, and the final figure is the concatenation of results you could have obtained in real time. If a parameter must be tuned, tune it on the training window only, and report how many variants you tried.

1. Freeze the hypothesis, data sources and parameters before looking at test results.
2. Timestamp every input and check that each is available before the decision.
3. De-duplicate and compare every price with a consensus; report results with and without the largest winners.
4. Model execution: maximum stakes, refusals, slippage and margins.
5. Count the trials: every rule, threshold and market you tested.
6. Validate forward across seasons and leagues, and accept no edge demonstrated as a result.

Findings and interpretation

Check your understanding

References

• Lawrence Clegg, John Cartlidge (2025). Not feeling the buzz: Correction study of mispricing and inefficiency in online sportsbooks.International Journal of Forecasting 41(2), 798–802 (preprint arXiv:2306.01740, 2023)A published profitable strategy that largely depended on one erroneous outlier price.Original source (doi:10.1016/j.ijforecast.2024.06.012) (opens in a new tab)
• Philip Ramirez, J. James Reade, Carl Singleton (2023). Betting on a buzz: Mispricing and inefficiency in online sportsbooks.International Journal of Forecasting 39(3), 1413–1423The original study that the correction re-examined.Original source (doi:10.1016/j.ijforecast.2022.07.011) (opens in a new tab)
• David Winkelmann, Marius Ötting, Christian Deutscher, Tomasz Makarewicz (2024). Are Betting Markets Inefficient? Evidence From Simulations and Real Data.Journal of Sports Economics 25(1), 54–97Single-season inefficiencies are expected even in efficient markets and rarely persist.Original source (doi:10.1177/15270025231204997) (opens in a new tab)
• David H. Bailey, Jonathan M. Borwein, Marcos López de Prado, Qiji Jim Zhu (2014). Pseudo-Mathematics and Financial Charlatanism: The Effects of Backtest Overfitting on Out-of-Sample Performance.Notices of the AMS 61(5), 458–471How many trials inflate the best backtest result.Original source (doi:10.1090/noti1105) (opens in a new tab)

Further reading

• Kaunitz, Zhong & Kreiner (2017): Beating the bookies with their own numbers – and how the online sports betting market is riggedExecution constraints and account restrictions in a real-money trial (preprint).
• Hegarty & Whelan (2024): Comparing two methods for testing the efficiency of sports betting marketsHow test design changes conclusions about market efficiency.