METHODOLOGY

Every signal goes on trial. Here is the courtroom.

The battery is deterministic: same input, same verdict, bit for bit. No model moods, no black box — statistics you can name, in an order you can audit.

The four gates

GATE 1

Net edge

Does anything remain after realistic costs, slippage and delay? Most signals end here — the edge was an artifact of frictionless simulation.

GATE 2

Family deflation

How many variants did you (or the world) try before this one? We deflate the score for every attempt — luck compounds fast when you keep rolling dice.

GATE 3

Placebo trial

Your signal competes against 500 random twins with the same trading profile. If random timing does just as well, your timing wasn't the edge.

GATE 4

Capacity & robustness

Survives with chunks of history removed? Across regimes? With parameters wiggled? At the size you'd actually trade?

The verdict names the first gate your signal failed — so you know exactly what to fix, or when to stop. That is the difference between a diagnosis and an oracle.

THE GRADIENT

What a verdict actually contains.

✓died_at — the first failed gate.
✓failure_codes — machine-readable causes your agent can act on.
✓Placebo percentile — one number instead of a gut feeling.
✓Survival map — five live attacks, each marked survived or dead.
✓Search budget — how many honest tries your family has left. Ends the endless tweaking loop.
✓Session history — „your last three variants all died of costs." A learning curve, not isolated verdicts.

diagnosis — strat_4217

{
  "verdict":     "fail",
  "died_at":     "gate_2_family_deflation",
  "failure_codes": [
    "edge_collapses_at_lag1",
    "family_budget_exhausted"
  ],
  "placebo_percentile": 61,   // beats 61% of 500 random twins
  "survival_map": {
    "delay_1bar":   "dead",
    "cost_stress":  "survives",
    "jackknife":    "survives",
    "regime_split": "dead",
    "param_wiggle": "survives"
  },
  "search_budget_left": 4
}

Demote-only: evidence can only make things worse

A verdict is never upgraded after the fact. New evidence can lower a grade — a data vendor restates prices, a leak is discovered — but nothing can inflate one. That removes the strongest temptation any rating business faces. It also means: when something we certified turns out to be wrong, we say so publicly and the certificate shows as revoked on /verify.

The statistics have names

Nothing in the battery is proprietary magic. The core is the published state of the art for separating skill from luck under multiple testing:

„Most claimed research findings in financial economics are likely false."HARVEY, LIU & ZHU · REV. FINANCIAL STUDIES 2016

Published strategies lose over half their returns — ≈−26% out-of-sample, ≈−58% post-publication.McLEAN & PONTIFF · JOURNAL OF FINANCE 2016

A few dozen tries manufacture „great" backtests out of pure noise — overfitting is mathematically guaranteed.BAILEY, BORWEIN, LÓPEZ DE PRADO & ZHU · AMS 2014

Of ~2.1 million systematically tested strategies, almost none survive correct multiple-testing correction.CHORDIA, GOYAL & SARETTO · RFS 2020

the primer: why backtests flatter everyone