Was your trading edge ever real? A field guide to backtest forensics

ALPHAASSAY RESEARCH · LONGREAD · 12 MIN READ

A backtest is the most flattering document in finance. You built the rule, you chose the window, you picked the instruments that were still around to test — and then the equity curve slopes up and to the right, and it feels like discovery. Most of the time it is not discovery; it is the residue of your own hopeful search. This is a field guide to telling the two apart, written from the perspective of the thing that tries to kill a signal rather than sell it.

None of what follows is investment advice. It is a methodology audit — how to test whether a result is real, not what to trade.

Why do most backtested edges vanish when real money touches them?

Because the number that looked like skill was mostly selection. When you test enough rules, some will clear any bar by chance alone, and the backtest keeps no memory of the ones you discarded. Harvey, Liu and Zhu (2016, Review of Financial Studies) reviewed 296 published factors and argued that the conventional t-stat threshold of 2.0 is far too lenient once you account for how many factors were tried; they proposed a hurdle closer to 3.0, and even that is a floor rather than a guarantee. Bailey and López de Prado made the same point sharper with the Deflated Sharpe Ratio (2014): a Sharpe of 2.0 from a single honest test and a Sharpe of 2.0 selected as the best of a thousand tries are not the same evidence, and only one of them should survive.

The practical consequence is uncomfortable. If you do not know how many variations were tried before the winner appeared, you cannot know what the winner's Sharpe is worth — and "I only ran a few" is exactly the belief that a long, forgetful afternoon of parameter tweaking produces.

What kills a backtest first, and why are costs the usual suspect?

The most common assassin is not exotic; it is the fee schedule. A rule can have a genuinely positive gross edge and still die the moment realistic round-trip costs, slippage and market impact are subtracted — the net return flips non-positive and the whole thing was an accounting artifact of ignoring friction. Our own free demo runs a built-in 40-trade example through the validator, and the blocking finding it returns is exactly this: costs_slippage_capacity, a cost-dead sign-flip — gross edge positive, net edge gone. That verdict is fail, and the recommended agent action is do_not_trade.

What makes costs so lethal is that they scale with turnover, so the strategies that look most exciting on paper — high-frequency mean-reversion, dense signal churn — are precisely the ones with the thinnest margin against friction. Before you celebrate a curve, model fees and slippage at the size you actually intend to trade, and show that net-of-cost return stays positive at that turnover. If it does not survive its own transaction costs, nothing else about it matters.

How do you separate skill from luck after a thousand parameter sweeps?

You stop trusting the winner's raw score and start pricing in the search that produced it. Two published tools do this without hand-waving. The Deflated Sharpe Ratio (Bailey and López de Prado, 2014) discounts an observed Sharpe by the number of independent trials and the non-normality of returns, asking whether the result would still look special against the distribution of best-of-N outcomes under no edge. The Probability of Backtest Overfitting (Bailey, Borwein, López de Prado and Zhu, 2017) goes further with combinatorial purged cross-validation: it measures how often the configuration that ranked best in-sample lands at or below the median out-of-sample, which is the operational definition of a search that fooled itself.

There is even a floor on how short a backtest is allowed to be. The Minimum Backtest Length (Bailey et al., "Pseudo-Mathematics and Financial Charlatanism," 2014) makes the point that with enough trials, a high in-sample Sharpe over too few years is expected under randomness — the length of your track record is itself evidence, or the lack of it. A single deflated number will not make a bad idea good, but it will stop a lucky one from being mistaken for a skilled one.

What actually survives when you test 18,000 rules at once?

We ran that experiment on ourselves. Taking the classical technical-analysis universe in the Sullivan–Timmermann–White lineage — filter rules, moving-average crossovers, support/resistance, channel breakouts, on-balance-volume rules, momentum and confirmed breakouts, on openly documented parameter grids that sum to exactly 18,000 rules — we tested every one of them against seven crypto bluechips on daily candles. That makes 126,000 rule-market pairs, and each pair went through the same battery a paying check receives, under the same cost model of 10 basis points per side with one bar of entry lag.

The funnel reads like an actuarial table. Costs alone remove 12,274 pairs whose gross edge never survived friction, and the deflation stage then removes 113,462 more: once you price in that 18,000 rules were searched — deflated Sharpe with n_trials = 18,000 plus Benjamini–Hochberg-adjusted p-values — roughly 90% of all pairs die from the size of the search rather than from anything the market did. Stability checks (combinatorial purged cross-validation and trade-concentration) leave 247, and the random-timing placebo, 500 matched random trade-sets on the same asset per candidate, leaves 245. That is 0.19% of where we started.

The survivors deserve the fine print more than the dead do. 234 of the 245 sit on a single asset, BNB, whose decade contained an exceptional one-way drift — which makes survival there a story about one asset's regime, not evidence that technical analysis works. On Bitcoin exactly one rule in 18,000 survives, and on Ethereum, XRP, ADA and Litecoin none do. The placebo stage shows the survivors time their entries better than chance on the same asset (median placebo percentile 85), which is a weaker claim than beating the asset itself: only 41 of the 245 — 0.03% of all 126,000 pairs — end up with more money than simply holding, and the single Bitcoin survivor returned 361% against 472% for doing nothing. All 245 do beat holding risk-adjusted, because a long/flat rule sits out the crashes. Timing bought calm, not return.

The honest limits belong in every retelling of these numbers. Everything above is in-sample of one frozen window (2017/2020 through 2026-07-08), long/flat on daily candles under a single published cost model, so different assumptions produce different numbers; surviving means "not falsified on this window", never a forecast and never a recommendation, because the battery is demote-only and can only kill. For the same reason we publish the mortality and the rule grids, not the surviving rules — a funnel that ended by handing out winners would be manufacturing the next generation of selection bias.

None of this asks you to take our word for it. The whole funnel — universe, method, stage-by-stage counts and the honest-limits block — is published as a signed document at GET /v1/public/benchmark, Ed25519-verifiable offline against our published key, so you can check the numbers in this section against the machine-readable original rather than trust the prose. The plain-language summary lives on the benchmark page.

Can a strategy pass a test it should fail?

Yes, and that is why a placebo matters more than another in-sample metric. If you run your rule against synthetic worlds that were constructed to contain no exploitable edge and it still appears to make money, the "edge" is a property of your process, not of the market. AlphaAssay's free demo previews exactly this with three seeded null worlds: a Heston stochastic-volatility path (total return −2.87%), a Merton jump-diffusion path (−0.32%), and a symmetric drift-burst path (−15.85%) — each labelled no_edge by construction, volatility clusters and jumps included. The paid falsification battery runs a strategy across dozens of such worlds, because a rule that "works" where nothing can work has told you about your backtest, not about price.

This is the test almost nobody applies to themselves, and it is the one that separates a robust signal from a well-dressed coincidence. An edge worth trading survives contact with worlds that were rigged to be unbeatable; a curve-fit does not.

How do you audit a signal you did not build — or a seller's track record?

You demand provenance and treat its absence as a finding, not a footnote. A shared CSV is survivorship-selected by definition — you are seeing it because its past curve looked good — so it should be treated as in-sample until proven otherwise on genuine post-publication data. Two questions do most of the work. Could the backtest have used information unavailable at decision time (look-ahead leakage, a suspiciously loss-free curve, future data folded into a feature)? And was the universe point-in-time, including the names that were later delisted rather than the ones that happened to survive to today?

For judging any paid channel, platform or signal seller, there is a public falsification protocol — seven falsifiable tests covering provenance, survivorship, pre-registration, placebo, costs, trial accounting and the examiner itself — and assay_provider_protocol hands it to you for free, machine-readable, applied to AlphaAssay included. The honest version of "trust me" is "here is how you would catch me lying."

Why should you trust the validator that grades everyone else?

Because a validator that cannot be checked is just another opinion with better vocabulary. Every verdict AlphaAssay issues is deterministic and carries the provenance hashes to replay it, so you can reproduce the exact output rather than take it on faith, and verdicts are demote-only — evidence can lower a grade, never inflate one. The platform also grades its own accuracy in public: a signed calibration ledger that counts registrations, survivors and, as forward windows mature, the hit-rate of its own verdicts.

At the time of writing that ledger is honest about being new — it reports insufficient_history rather than a flattering number, because the track record accrues from day one and pretending otherwise would be the exact sin the whole system exists to catch. A record that ages in public is worth more than a launch-day claim, precisely because you can watch whether it holds.

The one question worth asking your own backtest

Every technique above reduces to a single self-referential question that the good quants ask before capital, not after a drawdown: was my edge ever real, or was it the shape of my own search? You can work through the checklist by hand — model costs at real size, deflate for the number of tries, run a placebo world, prove point-in-time data, keep a pre-registered forward record — or you can let a deterministic, fail-closed battery do it and hand you a verdict with the exact gate that failed first.

If you want to see the full output shape without paying or signing up, assay_demo returns a complete verdict envelope over the built-in example. If you are about to spend weeks on an idea, assay_graveyard tells you for free how often that structural family has already died — and the public digest keeps the running score, currently families: 13 · tested: 17,500 · killed: 335 · survived: 0. Zero survivors so far is not a boast about severity; given the base rates above, a young ledger is expected to look exactly like this. The totals are deliberately floored to coarse steps so that platform volume is not inferable, and per-family mortality is itemised only once a family clears k-anonymity at five distinct submitters. (The 18,000-rule benchmark above ran in an isolated store, so its 126,000 pairs are not part of these ledger totals — the two scoreboards count different things.) Test it before it costs you — not because there is a clock, but because the alternative is finding out live.

AlphaAssay is an independent statistical assay office for trading signals and backtests: deflated Sharpe with cumulative trial accounting, overfitting detection (PBO/CPCV), leakage forensics, placebo tests against matched null worlds, and pre-registration through operator-published chained commitments — returning deterministic, replayable verdicts; separately issued certificates carry Ed25519 signatures anyone can verify offline. It does not generate strategies and gives no buy/sell advice. It is a methodology audit, not investment advice. Free demo and golden test vectors at alphaassay.com; available as a hosted API and MCP server.