AlphaAssayWhy backtests flatter everyone
Run enough backtests and one of them will look brilliant. Not because you found an edge — because you rolled dice often enough. This is the single most expensive misunderstanding in retail trading, and it has nothing to do with intelligence. It is arithmetic.
The three flatterers
Look-ahead. Your simulation quietly used information that was not available at the moment of the trade — a close price used at the open, a restated earnings figure, an index membership that was decided later. The backtest looks clairvoyant because, in a literal sense, it was.
Survivorship. Test on today's coin list or today's index members and you have silently excluded everything that died on the way. The past looks safer than it was, and mean-reversion strategies look like geniuses among survivors.
Selection under multiple testing. The quiet killer. Try 40 parameter combinations, keep the best: its performance is now part skill, part selection. Bailey, Borwein, López de Prado and Zhu showed that with enough variants, a „great" backtest is mathematically guaranteed — from pure noise.
„Most claimed research findings in financial economics are likely false." — Campbell Harvey, Yan Liu & Heqing Zhu, Review of Financial Studies (2016)
The numbers are brutal
McLean and Pontiff took 97 published, peer-reviewed return anomalies and asked a simple question: what happened after publication? On average the strategies lost roughly a quarter of their returns out of sample and more than half after publication. These were the best ideas academia had — reviewed, replicated, printed. Retail backtests do not get that much scrutiny before real money follows them.
Chordia, Goyal and Saretto went further and generated about 2.1 million trading strategies systematically. After correcting properly for the number of trials, almost none survived. The haystack is essentially all hay.
What honest testing looks like
None of this means edges don't exist. It means the default answer to „my backtest looks great" must be „so does everyone's" — and the burden of proof sits with the signal, not the skeptic. Honest testing therefore does four things, in order:
1. Charge realistic costs first. Fees, spread, slippage, execution delay. Most edges end here.
2. Deflate for every attempt. The Deflated Sharpe Ratio (Bailey & López de Prado, 2014) exists
precisely to subtract the luck you bought by trying many variants.
3. Race it against placebos. If 500 random twins with the same trading profile do as well,
the timing was never the edge.
4. Attack what survives. Delay it, stress the costs, remove chunks of history, split regimes,
wiggle parameters. A real edge is inconvenient to kill.
That sequence is exactly what the AlphaAssay battery runs — with the outcome signed, so nobody (including us) can quietly rewrite it later.
The point is not pessimism. The point is that a signal that survives all of this means something — and one that fails just saved you from finding out with real money.