AlphaAssayThe $546k backtest that passed walk-forward — a case study in honest self-validation failing
In 2025, a trader on a public algotrading forum described a strategy that had turned a simulated $8,000 into $546,000 — validated the way the textbooks say: a self-built backtester, ticker-specific fees, walk-forward analysis, and six months of live trading that tracked the simulation closely. Then the account that was supposed to be at half a million stood at $3,000. Nothing in this story is sloppy, and that is exactly why it matters: every check this trader ran answered a when question — does the edge hold in later data? — while the thing that killed the account was a how many question that no tool in the stack ever asked. This page reconstructs the failure, anonymised, as the community itself diagnosed it, and names the statistics that ask the missing question.
What did this trader do right?
Almost everything the standard advice demands. A custom backtester instead of a black box — so the cost model was known. Ticker-specific fees instead of a flat guess. Walk-forward analysis instead of one in-sample fit — the strategy was re-fit on rolling windows and scored on the data after each window. And six months of live trading whose results matched the simulation — the implementation was honest, the data pipeline clean. If your checklist is „costs, out-of-sample, live confirmation", this system passed it. Most backtests never get within sight of this discipline, and the account still ended at $3,000.
What killed it anyway?
The community's post-mortem converged on one word: selection. A strategy that reaches a $546k equity curve is rarely the first thing its author tried — it is the survivor of dozens or hundreds of variants, most of them deleted and none of them counted. Walk-forward cannot see that count: it validates the winner, not the search that produced the winner — and peer-reviewed evidence ranks walk-forward as the weakest of the common false-discovery preventions (Arian, Norouzi & Seco, Knowledge-Based Systems 305, 2024). With enough trials, some variant will pass any fixed battery of when-questions by luck alone — at 45 trials, a daily Sharpe of 1.0 needs about five years of history before it stops being expected from noise. The forum verdict for this system was the generic one, and it is the base rate: the search manufactured the curve, and the validation never priced the search.
Why didn't six months of live trading catch it?
Because six months of live results is an underpowered test that feels like a decisive one. Run the arithmetic: for a modest real edge, a few hundred observations detect it well under half the time — and the same is true in reverse, a no-edge strategy can look confirmed for months, especially when one regime carries it. Live tracking answers „is the implementation faithful?" — it barely moves „is the edge real?" until far more time has passed than anyone's patience allows. That is why the battery stamps verdicts underpowered instead of letting a short confirmation window read as an acquittal.
What would the missing questions have been called?
We never saw this system and cannot re-run it — this is a reconstruction from a public account, not
a verdict. But the questions that were never asked all have names in
the failure-code register: the family's honest trial count
(deflated_out_at_n=N), the backtest length demanded by that count
(BACKTEST_TOO_SHORT_FOR_N), whether the winner of a sweep is a property of the search
rather than the market (PBO_HIGH — the
statistic, explained), and whether the out-of-sample halves of anchored folds actually make money
(wf_oos_negative). None of these is exotic; they are one battery of how-many questions
bolted onto the when-questions this trader already ran.
The detail everyone should copy
The most instructive line in the whole thread is not the loss — it is that the author, after the collapse, publicly asked for adversarial review: „if you see any blindspots … please let me know." That instinct — invite the attack instead of defending the curve — is the entire discipline in one sentence, and it echoes across the community: „I've tested hundreds of strategies … they all eventually became unprofitable." „The backtest results were starting to look too good to be true, but I couldn't spot anything wrong." „One wrong step and all your backtests will give you wrong results." The demand side is real too: users of a popular open-source backtesting library asked its author, by name, for deflated Sharpe and „tests of statistical significance that take into account the number of tests" — acknowledged, never built. Asking to be falsified is the rational move; what has been missing is somewhere to send the request.
What is the honest way to run this today?
Count every attempt against one family budget (why deflation must be cumulative), demand the backtest length your trial count implies, grade the sweep itself and not just its winner, and treat short live windows as what they are — weak evidence. If you want the whole battery of how-many questions run against a signal, that is literally the service: free known-answer specimens first, the falsification protocol if you are judging someone else's claim — and check the graveyard before you spend weeks on an idea the crowd has already buried. A fail with a named cause, at five cents, is the cheapest version of this story.