AlphaAssaySame strategy, five engines, different answers: backtest implementation risk
Run one identical strategy — same rules, same data, same cost specification — through five widely used backtesting engines, and the reported total return can differ by up to 3.71 %. That is the central measurement of a 2026 study that benchmarked five engine implementations across fifteen strategies on S&P 500 data from 2020–2024, and named the effect implementation risk: variability in backtest outcomes attributable solely to the choice of simulation engine. On a $1B portfolio the ambiguity is worth roughly $37M a year. Your Sharpe ratio can be honest, your data clean, your trial count accounted for — and the number you are staring at still depends on which engine happened to produce it.
Where does the divergence come from?
Almost entirely from transaction-cost models. In the study's zero-cost ablation, all five engines agreed exactly — to the digit. Turn realistic costs on and the divergence rises monotonically with cost intensity (Spearman ρ = 0.93). The engines do not disagree about markets; they disagree about what trading costs, and each buries that opinion in defaults. The forensic part of the study found seven previously undocumented defects across widely used engines — including one that silently divides the commission rate you pass by 100, so a user specifying 18 basis points is actually charged 0.18. Every backtest on that engine looks systematically cheaper to trade than reality.
Is this just overfitting in disguise?
No — and that is what makes it dangerous. Implementation risk is orthogonal to statistical overfitting: deflation, placebo trials and out-of-sample discipline all assume the simulator itself is telling the truth. A perfectly honest researcher with a perfectly deflated Sharpe still inherits the engine's cost model, bugs and all. It is a separate axis of backtest unreliability, and no split scheme fixes it.
What can you do about it?
The study's recommendation is blunt: validate with at least two independent engines of maximally different architecture (one event-driven, one vectorised), and audit the cost model of each against a reference specification. In practice almost nobody does this — it doubles the plumbing. Three cheaper habits catch most of the damage: read your engine's commission and slippage defaults instead of trusting them; re-run the backtest with costs set to zero and sanity-check that the delta matches your own cost arithmetic; and treat any engine-reported cost below your venue's published fee schedule as a bug until proven otherwise.
How does this relate to an AlphaAssay verdict?
Our cost_stress attack already treats the submitted cost assumptions as hostile — it
re-prices the edge under worse fees, wider spreads and delayed fills, which catches strategies that only
live inside optimistic cost settings. What it does not do is re-implement your simulator: cross-engine
replication is an open frontier for the field, not a solved gate, and we treat it that way. Until it is
solved, the honest posture is the one this page describes: assume the engine is part of the experiment,
not part of the ground truth. (Source: „implementation risk" benchmark study, arXiv:2603.20319, 2026;
figures quoted above are from the paper.)