Persistent bias or single-season noise?
Weigh claims of persistent pricing bias against multi-season evidence: why significant single seasons are expected even in efficient markets, how theory and competition evidence fit together, and how to pre-register a cross-season test.
Learning objectives
- Compute family-wise error rates and binomial tail probabilities for many season-level tests.
- Explain why Winkelmann et al. (2024) expect significant single-season results even in an efficient market.
- Reconcile Whelan (2024) theory with Hegarty & Whelan (2026) evidence on market structure.
- Write a pre-registered multi-season, multi-league test with a decision rule that includes "not demonstrated".
Two different claims
"Long shots were overpriced last season" and "long shots are persistently overpriced" sound similar but need very different evidence. The first is a description of one sample. The second is a claim about the process generating prices, and it predicts that the pattern will recur across seasons and leagues that were not used to discover it.
A persistence claim therefore makes testable predictions: the sign of the effect should be stable across most seasons, its size should not collapse once the discovery sample is excluded, and it should appear where its proposed mechanism says it should — and be weaker where the mechanism is absent. If none of these predictions is stated before testing, almost any pattern of results can be read as support.
The arithmetic of many seasons
| Tests k | Example | FWER | Bonferroni threshold |
|---|---|---|---|
| 1 | One league, one season | 5.00% | 0.05 |
| 5 | Five leagues, one season | 22.62% | 0.01 |
| 14 | One league, fourteen seasons | 51.23% | 0.0036 |
| 20 | Five leagues × four bias definitions | 64.15% | 0.0025 |
| 70 | Five leagues × fourteen seasons | 97.24% | 0.00071 |
Counting significant seasons helps only if you know what to expect under the null. With 14 independent seasons and no real bias, the chance of at least two significant seasons is 15.30% and of at least three is 3.01%. With 70 league-seasons, at least three significant results happen 68.63% of the time; at least six still happen 13.72% of the time. "Six significant seasons" sounds like a pattern; it is not, at that scale.
A synthetic demonstration makes the point concrete. In the Academy's seeded dataset the market is noisy but unbiased, and backing every long shot (market probability ≤ 30%) has a true average EV of −1.94% — knowable only because the data are synthetic. Split into six blocks of 200 events, the realised ROI runs −3.0%, +49.5%, −9.7%, +25.0%, −16.6% and +4.7%. Pooled, 295 bets win 69 times for +7.2%, yet the ROI standard error at the 4.96 average odds is about 11.6 points (about 28 points per 50-bet block).
Theory, competition evidence and test design
Persistent biases are not impossible; they need a mechanism and evidence that matches it. Whelan (2024) shows theoretically that bettor disagreement plus risk-averse bookmakers can generate a favourite–longshot pattern without assuming bettors love risk. Hegarty & Whelan (2026) test a related model on European football: bookmakers facing less competition price with a favourite–longshot bias, while the competitive market shows little of it, and the model's prediction about draws is also supported.
How you test matters as much as what you test. Hegarty & Whelan (2024) compare two regression designs: tests built on normalised (de-vigged) probabilities behave well, whereas tests on raw inverse odds do not and are biased against finding a favourite–longshot bias. A null result from the wrong design is not evidence of efficiency.
Pre-registering a cross-season test
- State the hypothesis with a direction and market structure: "in league set L, selections with de-vigged probability below 25% return less than those above 60%".
- Fix the data before testing: leagues, seasons, odds source, quote timing and the de-vig method (normalised probabilities, per Hegarty & Whelan (2024)).
- Name one primary test on the pooled sample, and list secondary tests with a correction such as Bonferroni.
- Specify the decision rule in advance, including what counts as "not demonstrated" and what would count as evidence against.
- Hold out later seasons that nobody looks at until the rule is frozen.
Pooling is necessary but not magic. Take a typical long shot priced at 4.50 that wins 21% of the time. One season of 60 such bets has a ROI standard error of about 23.7 points; fourteen seasons pooled (840 bets) bring it down to about 6.3 points. Detecting a genuine 3% edge at that price would need roughly 15,256 bets. A multi-league, multi-season design is therefore not excessive caution — it is the minimum sample at which the question can be answered.
| Pattern of results | Verdict |
|---|---|
| Primary pooled test significant, sign consistent across most seasons and leagues, holds in held-out seasons | Evidence of a persistent bias (still check execution) |
| A few significant seasons, mixed signs, pooled test not significant | Not demonstrated |
| Significant in less competitive markets only | Conditional bias; no claim about competitive markets |
| Theory predicts it, data from the right design do not show it | Not demonstrated; mechanism may be weak |
Worked example
Worked example · Six significant league-seasons out of seventy
A hypothetical report tests a long-shot bias separately in five fictional leagues over fourteen seasons (70 tests at α = 5%). Six league-seasons come out significant, and none of them is significant after correction. The pooled primary test was not pre-registered.
- Expected false positives
70 × 0.053.5 under the null - Family-wise illustration
1 − 0.95^70, if the null tests are independent97.24%; dependence changes the exact rate - Tail probability
P(X ≥ 6) for X ~ Bin(70, 0.05)13.72%: six or more is unremarkable - Corrected threshold
Bonferroni: 0.05 / 70p < 0.00071 per test; no result passes - For comparison
P(X ≥ 8) for X ~ Bin(70, 0.05)2.34%: eight would begin to be surprising - Design check
Were tests run on normalised probabilities?Required before any conclusion
Six of seventy is roughly what pure chance produces. The honest verdict is persistent bias not demonstrated. The next step is not to search for the leagues that worked, but to pre-register one pooled test on normalised probabilities and run it on seasons not yet examined.
Practice · Compute the false-positive budget
You plan to test four definitions of "long shot" in five leagues (20 tests at α = 5%). Compute the family-wise error rate and the Bonferroni threshold. Then compute the chance of at least three significant seasons out of fourteen when there is no bias. Finally, explain which single test you would pre-register as primary.
Show what you should find
FWER = 1 − 0.95²⁰ ≈ 64.15%; Bonferroni threshold 0.05 / 20 = 0.0025. P(X ≥ 3) for X ~ Bin(14, 0.05) ≈ 3.01%. A sensible primary test is one pooled regression on normalised probabilities across all five leagues with one fixed long-shot definition; the other definitions become corrected secondary analyses.
Common mistakes
- ✗ Counting significant seasons without comparing to the null expectation.Compute the expected number (k × α) and a binomial tail; six of 70 has a 13.72% chance under no bias.
- ✗ Testing efficiency with raw inverse odds.Use normalised probabilities; raw inverse-odds tests are biased against finding a favourite–longshot bias Hegarty & Whelan (2024).
- ✗ Treating a theoretical mechanism as proof of a profitable pattern.Whelan (2024) explains how a bias can arise; it does not measure its size or show that backing favourites pays.
- ✗ Generalising from less competitive to competitive markets.Hegarty & Whelan (2026) find the bias concentrated where competition is weaker; say which market structure your claim covers.
When it fails
Supporting research
- Winkelmann et al. (2024): Are Betting Markets Inefficient? Evidence From Simulations and Real DataJournal of Sports Economics · Peer-reviewed empiricalSingle-season inefficiencies do not persist across fourteen seasons, and simulations show significant single-season results are likely in an efficient market.
- Whelan (2024): Risk aversion and favourite–longshot bias in a competitive fixed-odds betting marketEconomica · Peer-reviewed theoryA theoretical mechanism by which disagreement and bookmaker risk aversion can produce a favourite–longshot pattern.
- Hegarty & Whelan (2026): Market structure and prices in online betting markets: theory and evidenceOxford Economic Papers · Peer-reviewed empiricalEvidence that the bias appears in less competitive bookmaker markets and little in the competitive one.
- Hegarty & Whelan (2024): Comparing two methods for testing the efficiency of sports betting marketsSports Economics Review · Peer-reviewed methodsWhy efficiency tests should use normalised probabilities rather than raw inverse odds.
Check your understanding
Practice only: saved on this device, no XP and no account record.