The backtest overfitting checklist

Twelve checks, ordered by what kills fastest. Before you trust a backtest — yours or anyone's — walk it through this list: realistic costs and execution delay first, because most edges die right there; then leakage; then multiple-testing accounting; then robustness. Each check states what to do, what failure looks like, and the machine-readable failure code an AlphaAssay verdict would attach, so your agent can run the same list programmatically. A strategy that clears all twelve is not proven — no finite test can prove an edge — but it has earned the right to face real statistics instead of real money. Copy the whole checklist as markdown below and wire it into your agent's pre-deployment routine.

Gate 1 — net edge after reality

1. Charge fees, spread and slippage

Rebuild the backtest with your venue's real fee schedule and a conservative spread. Failure looks like: the equity curve flattens the moment money touches it.

2. Delay every fill by one bar

If the edge vanishes with one bar of delay, your simulation was trading on information it did not yet have. Code: edge_collapses_at_lag1.

3. Stress costs 2×

Double fees and spread. An edge that dies here is a bet on your cost model, not on the market. Code: cost_stress.

Gate 2 — honest trial accounting

4. Count every variant you tried — including discarded ones

Parameter grids, deleted notebooks, „quick experiments": all trials. The winner was selected from all of them, so the statistics must pay for all of them.

5. Compute the Deflated Sharpe Ratio

Feed your Sharpe, history length and true trial count into the calculator. Below 0.95: not proven. Code: family_budget_exhausted.

6. Freeze the spec before the final test

Decide every parameter, then test once. Better: make it provable with a sealed public timestamp.

Gate 3 — luck versus timing

7. Race it against random twins

Generate placebo signals with the same trading profile (frequency, holding period, exposure). If they do as well, the timing was never the edge.

8. Read the placebo percentile, not the raw return

Beating 61 of 100 random twins is a coin toss with extra steps; the percentile is the honest number.

9. Delete the best month and re-run

An edge that lives in one lucky window is a story about that window. Code: jackknife.

Gate 4 — robustness

10. Split by regime and check both halves

Bull/bear, high/low volatility — an edge that only exists in one half is regime luck wearing a lab coat. Code: regime_split.

11. Wiggle every parameter ±20%

If neighbouring parameters fail, you found a coordinate, not an edge. Code: param_wiggle.

12. Ask what capacity the edge survives

Estimate the size at which your own trading moves the price you depend on. Some real edges are real and worth $40 a month.

Why this order?

Because mortality is front-loaded: most signals die at costs or the placebo race long before anything exotic gets a chance. Checking robustness on a signal that never survived costs is polishing a corpse.

Copy the checklist (markdown, free to reuse with a link back)

backtest-overfitting-checklist.md — CC BY 4.0, link to alphaassay.com

# Backtest overfitting checklist (alphaassay.com/research/backtest-overfitting-checklist)
Gate 1 — net edge:      1. charge fees+spread+slippage  2. delay fills 1 bar  3. stress costs 2x
Gate 2 — trials:        4. count ALL variants tried     5. deflated Sharpe >= 0.95  6. freeze spec before final test
Gate 3 — luck:          7. race vs matched random twins 8. read the placebo percentile  9. delete best month, re-run
Gate 4 — robustness:   10. split by regime             11. wiggle params ±20%  12. estimate capacity
Rule: a pass is a statistical trial result — never a promise of returns.

Run it as one API call

The manual checklist takes an afternoon; the battery runs the statistical parts in one call and signs the result. Free known-answer demo first — no account:

terminal

$ curl -s https://api.alphaassay.com/v1/assay/demo \
    -H "Content-Type: application/json" -d @golden_lookahead.json