Is my backtest overfit?

Statistically, probably — and that is a base rate, not an insult. When McLean and Pontiff re-tested 97 peer-reviewed return anomalies, more than half of the performance evaporated after publication; when Chordia, Goyal and Saretto generated 2.1 million strategies and corrected properly for multiple testing, almost none survived. Retail backtests receive far less scrutiny than either. So the honest question is not whether your backtest looks good — everyone's does — but whether it survives the four ways good-looking backtests are manufactured: unrealistic costs, look-ahead leakage, selection under multiple testing, and regime luck. The twelve signs below cover how your backtest was built — no numbers uploaded, no account — and point to the gate of a statistical trial that would most likely kill it, in the same failure-code vocabulary a real AlphaAssay verdict uses. Three minutes here is cheaper than a live drawdown.

The twelve signs, gate by gate

Did you charge fees, spread AND at least one bar of execution delay?

Frictionless fills are the single most common manufacturing defect. Real fills pay the spread, pay fees, and happen a bar late. Failure signature: edge_collapses_at_lag1, gate 1.

Does the edge survive doubling your cost assumptions?

An edge that dies at 2× costs is a cost-model bet, not a market edge (cost_stress).

Is the strategy's turnover realistic for its capital?

High-turnover edges evaporate with size; capacity is part of gate 1 economics, not an afterthought.

Could any input have been revised after the fact?

Earnings restatements, index membership, delisted assets quietly leak the future into the past — survivorship and restatement bias live in the data, not the code.

Does every signal use only information available at trade time?

Close prices used at the open, same-bar highs, „monthly" data stamped mid-month: classic look-ahead. If delaying execution one bar kills the edge, the answer was no.

Did you test on assets that no longer exist?

Testing today's coin list or index members silently deletes everything that died — the past looks safer than it was.

How many parameter combinations did you try before this one?

Count everything, including deleted notebooks. Forty trials manufacture a great backtest from noise — mathematically guaranteed (Bailey et al., 2014). Signature: family_budget_exhausted, gate 2.

Did you freeze the spec before the final test?

If the „final" test was re-run after a tweak, it was not final. Pre-registration exists for exactly this (how it works).

Would you have published this result if it were negative?

If not, your process has selection built in — the survivor you are looking at was chosen by outcome.

Does the edge survive deleting its best single month?

An edge that lives in one lucky window is a story about that window (jackknife, gate 3/4).

Does it work in both halves of a regime split?

A strategy that rode one bull market backtests beautifully until the regime ends (regime_split, gate 4).

Does it survive wiggling every parameter ±20%?

If lookback 20 works and 19/21 don't, you found a coordinate, not an edge (param_wiggle).

What are the four ways backtests get manufactured?

What is look-ahead bias?

Using information in simulation that was not available at the moment of the trade — restated data, same-bar prices, later index membership. The backtest looks clairvoyant because it literally was.

What is survivorship bias?

Testing on the assets that survived until today, silently excluding everything that died on the way. Mean-reversion strategies look like geniuses among survivors.

What is selection under multiple testing?

Trying many variants and keeping the best: the winner's performance is part skill, part selection — and with enough trials, all selection. The Deflated Sharpe Ratio was built to price exactly this.

What is regime luck?

An edge that only ever traded one market regime — one bull run, one volatility state — and mistakes that regime for the world.

What does a real trial add?

Self-diagnosis catches construction errors; only statistics catch luck. A signal that passes all twelve signs above still needs deflation against its true trial count and a placebo race against random twins — that is the battery, and the demo tier runs it free on known-answer test cases: golden specimens.