Every signal goes on trial. Here is the courtroom.
The battery follows a fixed gate sequence. Pure/read operations require the same explicit
as_of for reproducibility; stateful calls expose their effective timestamp and may reflect
cumulative tenant state. No model moods, no black box —
statistics you can name, in an order you can audit.
The four gates
Net edge
Does anything remain after realistic costs, slippage and delay? Most signals end here — the edge was an artifact of frictionless simulation.
Family deflation
How many variants did you (or the world) try before this one? We deflate the score for every attempt — luck compounds fast when you keep rolling dice.
Placebo trial
Your signal competes against 500 random twins with the same trading profile. If random timing does just as well, your timing wasn't the edge.
Capacity & robustness
Survives with chunks of history removed? Across regimes? With parameters wiggled? At the size you'd actually trade?
Under the hood the four gates unfold into eleven graded stages, run in trial order —
net_edge → funding_edge → family_deflation → power_honesty → significance → cpcv → walk_forward → concentration → placebo → capacity → graveyard_prior —
and the verdict names the first stage your signal failed, so you know exactly what to fix, or when to stop.
That is the difference between a diagnosis and an oracle.
What a verdict actually contains.
- ✓died_at — the first failed gate.
- ✓failure_codes — machine-readable causes your agent can act on.
- ✓Placebo percentile — one number instead of a gut feeling.
- ✓Survival map — eight live attacks, each marked survived or dead.
- ✓Search budget — how many honest tries your family has left. Ends the endless tweaking loop.
- ✓Session history — „your last three variants all died of costs." A learning curve, not isolated verdicts.
The 600 tries below are not this
caller's own: n_trials_effective is inherited from the anonymised family ledger, where
every recorded attempt at the same signal family counts against one shared budget — which is why a
first call can already fail deflation.
{
"schema": "gauntlet.v1",
"verdict": "fail",
"died_at": "family_deflation",
"failure_codes": ["deflated_out_at_n=600"],
"stages": [
{ "stage": "net_edge", "verdict": "pass",
"evidence": { "net_sharpe_annualized": 3.11, "trades": 184,
"bars": 380, "net_return_total_pct": 41.7 } },
// 380 daily bars ≈ 18 months: a high Sharpe on a short window
{ "stage": "funding_edge", "verdict": "skipped",
"evidence": { "reason": "no perpetuals in book" } },
{ "stage": "family_deflation", "verdict": "fail",
"evidence": { "dsr": 0.31, "cumulative_n": 1, "variants_in_call": 1,
"n_trials_effective": 600,
"effective_n_method": "family_ledger",
"killed_by": "deflated_out_at_n",
"family_verdict": "deflated_out" } }
// over 18 months, the best of 600 recorded family tries is expected to look
// about this good by chance alone — so 3.11 buys only dsr 0.31, not a pass
// + power_honesty, significance, cpcv, walk_forward, concentration, placebo, capacity, graveyard_prior
],
"budget": { "cumulative_n": 1, "n_trials_effective": 600 }
// cumulative_n = your own submissions; n_trials_effective counts the whole family's recorded tries
}
Demote-only: evidence can only make things worse
A verdict is never upgraded after the fact. New evidence can lower a grade — a data vendor restates prices, a leak is discovered — but nothing can inflate one. That removes the strongest temptation any rating business faces. It also means: when something we certified turns out to be wrong, we say so publicly and the certificate shows as revoked on /verify.
The statistics have names
Nothing in the battery is proprietary magic. The core is the published state of the art for separating skill from luck under multiple testing:
The same question, asked from independent directions.
The named statistics above are the skeleton. Inside the stages, several cells now ask the same question more than once, because an edge that only survives one framing was never an edge. Every field below is delivered in the verdict envelope and documented in the live tool handshake.
Deflation, from three sides
The Deflated Sharpe Ratio asks whether your score beats the best of N random tries; the Benjamini–Hochberg-adjusted p asks whether it survives the family of tests around it; and the empirical-Bayes shrinkage exhibit asks the family itself — given everything this strategy family has recorded, how much of a t-statistic this size is signal and how much is noise? The shrunk value (t_shrunk, shrink_factor) is an exhibit, never the verdict, and under three usable family trials it reports insufficient_family_history instead of guessing.
Three brackets, one mean
The significance battery puts three confidence brackets around the same mean return: the percentile bootstrap interval, the studentized bootstrap-t interval — second-order accurate, it stays honest exactly where the first one flatters — and the Newey–West HAC interval, which prices in the autocorrelation real trading returns carry (se_naive vs se_hac). When a sharper bracket contains zero while the first does not, the envelope says so (studentized_ci_contains_zero, hac_ci_contains_zero): the claimed precision was borrowed from optimistic assumptions, not earned.
Carry is not alpha
A perpetual-futures backtest that ignores the funding leg overstates its edge, because funding is a cost the exchange really collects. Given a funding-rate series, the funding_edge stage charges it against every holding period and can find two things: the profit only lived in that ignored cost (funding_erases_edge), or the price leg loses money and the "edge" is the funding income itself (edge_is_funding_carry) — a premium available to any holder of the position, crash-prone, and no evidence of timing skill.