Write a reproducible research report
Turn an analysis into an auditable memo: question, null hypothesis, data lineage, point-in-time joins, pre-registered rules, confidence bounds, execution assumptions, failure cases, conflicting evidence and an explicit decision — including "no edge demonstrated".
Learning objectives
- Fill in every field of a research memo template for a betting-rule study.
- Report a Wilson interval, a ROI standard error and the number of bets needed to detect the claimed edge.
- State costs, execution assumptions and the number of variants tested.
- Write a decision that follows from the evidence, including "no bet / no edge demonstrated".
Why a memo, not a screenshot
A profit chart answers "what happened in this file?". A research memo answers "what would an independent analyst find if they re-ran this from raw data?". The cases in this track — a single outlier driving a published profit Clegg & Cartlidge (2025), a corrigendum that turned positive average returns negative Walsh & Joshi (2024) — were caught because someone could re-run the work. Write every report as if that someone will be you, six months from now, with no memory of the choices you made.
The memo template
| Field | What to write |
|---|---|
| Question | One sentence: which market, which selections, which period. |
| Null hypothesis | What you expect if there is no edge (for example, ROI equals the margin-implied loss). |
| Data lineage | Sources, versions, download dates, row counts before and after cleaning, and every exclusion rule. |
| Timestamps and joins | Quote capture time versus kickoff; as-of joins; proof no feature post-dates the decision. |
| Pre-registered rule | The exact selection and staking rule, frozen before the test data were viewed. |
| Variants tried | Number of thresholds, models or leagues tested, and the correction applied. |
| Results with bounds | Bets, wins, ROI, Wilson interval, ROI standard error, bets needed to detect the edge. |
| Costs and execution | Margin, slippage, fill rate, stake caps, quote age; executed EV. |
| Failure cases | Where the rule lost most, and rows that drive most of the profit. |
| Conflicting evidence | Published studies or your own tests that point the other way. |
| Decision | Act in simulation / continue paper-tracking / no edge demonstrated / reject. |
| Replication package | Code, data hashes, random seeds and the command that regenerates every number. |
Two fields deserve more care than they usually get. Data lineage means a reader can trace every row back to a source file and every exclusion to a written rule; if 27 rows became 23, the memo says which four went and why. Point-in-time joins mean that each bet is matched to the last quote and the last feature value recorded before the decision, never to a later revision. Odds providers sometimes back-fill or overwrite columns, so store the capture timestamp alongside every price.
The conflicting evidence field is where intellectual honesty shows. If published work finds that a similar pattern did not persist across seasons, or that a similar profit came from a data error, the memo must say so and explain why this study should be read differently — or concede that it should not.
Confidence bounds you must report
Wilson: (p̂ + z²/2n ± z √(p̂(1 − p̂)/n + z²/4n²)) / (1 + z²/n)SE(ROI) = sd / √n sd = √(p(o − 1)² + (1 − p) − EV²)n ≈ (z × sd / edge)²The three numbers answer different questions. The Wilson interval asks whether the win rate is clearly above break-even. The ROI standard error says how far the realised return could be from the true one. The detection count tells you whether the sample was ever large enough to answer the question — often it was not, and the memo should say so before anyone asks.
Writing the decision
The decision field must follow mechanically from the evidence fields. Four outcomes cover most studies: act in simulation (bounds clear break-even after costs and corrections, and fresh data agree); continue paper-tracking (promising but under-powered); no edge demonstrated (bounds include break-even, or execution removes the edge); and reject (bounds sit below break-even or the hypothesis failed on held-out data).
Finally, list what would change your mind: the sample size at which the edge would be detectable, the execution assumptions that would make it vanish, and the held-out data you will check next. A memo that names its own falsification test is one other people can trust.
Execution assumptions belong in the decision, not in a footnote. In the mini-memo below, a true win rate of 49% would still sit above the 47.62% break-even at the quoted 2.10 — yet with an 80% fill and 3% slippage the executed EV would be −0.15% per intended stake. An edge that exists only at the quoted price is not an edge you can act on.
Worked example
Worked example · A complete mini-memo on a synthetic result
Question: does rule R — back the home side in a fictional league when a model's edge is at least 4% — beat break-even? Synthetic result for illustration: 400 bets, all at hypothetical odds 2.10, 205 wins, one-unit stakes. Twelve threshold variants were tried before choosing 4%.
- Null hypothesis
True win rate equals break-even 1/2.1047.62% - Headline
Profit 205 × 2.10 − 400; ROI 30.5 / 400+30.5 units, ROI +7.63% (win rate 51.25%) - Wilson 95% interval
Wilson interval for 205 of 40046.36% to 56.11%: contains break-even 47.62% - ROI standard error
sd 1.050 / √4005.25 points; approx. 95% range −2.66% to +17.91% - One-sided test
P(X ≥ 205) for X ~ Bin(400, 0.4762)8.02%: not significant at 5% even before correction - Multiple variants
1 − 0.95¹² for independent null tests; Bonferroni 0.05 / 12Illustrative FWER 45.96%; bar p < 0.0042 - Execution
executed EV: p 0.5125, odds 2.10, fill 80%, slippage 3%+3.52% per intended stake, assuming the observed win rate is true (optimistic) - Precision illustration (not power)
n ≈ (1.96 × SD / edge)²; no alternative-hypothesis power is specified729 bets for the observed ROI; 10,580 for a 2% edge - Decision
Bounds include break-even; 12 variants; execution halves the edgeNo edge demonstrated; continue paper-tracking with the rule frozen
A +7.6% ROI on 400 bets reads like a result, but the interval spans break-even, the one-sided p-value is 8%, twelve variants were tried, and realistic execution roughly halves what remains. The memo's conclusion is no edge demonstrated. Its falsification test: track the frozen rule on the next 729 synthetic bets and report the result whichever way it falls.
Practice · Check the memo's bounds in the EV lab
Open the Fair odds and EV lab with odds 2.10, estimate 51.25% and the Wilson bounds 46.36%–56.11% as the range. Read the verdict, then write the one-line decision for the memo.
Lab 1 · Fair odds and expected value
Synthetic · deterministicNo clear advantage. Your range straddles break-even. Whether the bet has value depends on which end of your estimate is right, so there is no clear advantage. A careful analyst treats this as a pass unless better evidence narrows the range.
Show data table
| True probability (%) | Expected ROI |
|---|---|
| 36% | -24% |
| 37% | -22% |
| 38% | -20% |
| 39% | -18% |
| 40% | -16% |
| 41% | -14% |
| 42% | -12% |
| 43% | -10% |
| 44% | -8% |
| 45% | -5% |
| 46% | -3% |
| 47% | -1% |
| 48% | 1% |
| 49% | 3% |
| 50% | 5% |
| 51% | 7% |
| 52% | 9% |
| 53% | 11% |
| 54% | 13% |
| 55% | 16% |
| 56% | 18% |
| 57% | 20% |
| 58% | 22% |
| 59% | 24% |
| 60% | 26% |
| 61% | 28% |
| 62% | 30% |
| 63% | 32% |
| 64% | 34% |
| 65% | 37% |
| 66% | 39% |
| 67% | 41% |
Educational calculator. Uses only the numbers you enter or seeded synthetic data — no live prices, no staking, no real money.
Show what you should find
Break-even is 47.62%. EV at the estimate is +7.63%, but the range runs from about −2.6% at 46.36% to about +17.8% at 56.11%, so the verdict is 'depends on the estimate'. Decision line: "Rule R: no edge demonstrated (Wilson interval includes break-even; 12 variants tested); continue paper-tracking for 729 further bets with the rule frozen."
Common mistakes
- ✗ Reporting ROI without an interval.Always pair ROI with a Wilson interval for the win rate and a ROI standard error; +7.63% ± 5.25 points means something very different from +7.63% alone.
- ✗ Leaving out the variants that did not work.State how many thresholds, models or leagues were tried and apply a correction; twelve variants give a 45.96% chance of a false positive.
- ✗ Ignoring execution in the conclusion.Report executed EV under explicit fill and slippage assumptions alongside the nominal figure.
- ✗ Treating "no edge demonstrated" as a failed project.It is a valid, publishable conclusion when the evidence supports it, and it prevents costly false positives.
When it fails
Supporting research
- Bailey et al. (2014): Pseudo-Mathematics and Financial Charlatanism: The Effects of Backtest Overfitting on Out-of-Sample PerformanceNotices of the American Mathematical Society · Peer-reviewed theoryWhy the number of variants tried must be reported and corrected for.
- Clegg & Cartlidge (2025): Not feeling the buzz: Correction study of mispricing and inefficiency in online sportsbooksInternational Journal of Forecasting · Peer-reviewed replicationA worked example of how a re-run on original data exposed an outlier-driven result.
- Walsh & Joshi (2024): Machine learning for sports betting: Should model selection be based on accuracy or calibration?Machine Learning with Applications · Peer-reviewed empiricalA corrigendum that reversed average returns, showing the value of releasing code and correcting the record.
- Winkelmann et al. (2024): Are Betting Markets Inefficient? Evidence From Simulations and Real DataJournal of Sports Economics · Peer-reviewed empiricalWhy single-sample significance is weak evidence and multi-season tests are needed.
Check your understanding
Practice only: saved on this device, no XP and no account record.