What counts as the same strategy? Trial accounting for families

ALPHAASSAY RESEARCH · METHOD · 5 MIN READ

RSI(5) < 20, RSI(7) < 25 and RSI(9) < 30 are not three ideas. They are one idea, asked three times — and if you only count the version you kept, your statistics quietly assume the other two never happened. That assumption is how good-looking backtests are manufactured from noise: try enough cousins of one hypothesis, keep the best, and the winner's performance contains selection luck by construction. The fix has a name — trial accounting — and the unit it must be counted in is not the run, not the submission, but the strategy family.

What is a strategy family?

A family is the set of variants that share a hypothesis: the same signal logic with different lookbacks, thresholds, symbols or cosmetic rearrangements. Statistically they are highly correlated trials — knowing one result tells you most of the next. Counting them as independent experiments is the textbook error that the Deflated Sharpe Ratio exists to correct (the DSR, explained): every additional trial raises the bar a result must clear before it means anything.

Why must the count survive your retries?

Here is the failure mode of most self-run validation: each new backtest starts with an amnesiac counter. You test five variants today, sleep, test five tomorrow — and every tool involved happily treats trial #6 as trial #1. The search history lives only in your memory, where it is subject to the most reliable leak in finance: forgetting the failures. Honest deflation therefore has to be cumulative across submissions. At AlphaAssay the battery keeps that count per family, and it cannot be reset by rewording the idea: family identity comes from the signal's structure, recorded as a one-way fingerprint plus a 32-number sketch of each trial's return profile — the sketch is what lets the ledger prove two variants were near-duplicates (and count them as ~one), and it is far too coarse to reconstruct trades or rules. When a family has spent its statistical budget, the verdict says so in machine-readable form: deflated_out_at_n=N — after N effective trials, one more tweak is indistinguishable from luck (failure codes, explained).

Do twenty near-identical variants really count as twenty?

No — and the correction cuts both ways. Twenty variants of one idea are not twenty independent discoveries, so the battery deflates by an effective trial count rather than the raw one: N̂ = ρ̄ + (1 − ρ̄)·M, the estimator from Bailey and López de Prado's Deflated Sharpe appendix, where ρ̄ is the average correlation between the family's return profiles and M the raw count. Twenty near-duplicates with ρ̄ close to 1 collapse toward one effective trial; twenty genuinely different attempts stay twenty. The honesty constraint is fail-closed: only proven closeness deduplicates — a trial whose return profile was never recorded counts fully, because „probably similar" is not evidence. The verdict shows its arithmetic in the open: n_trials_effective, effective_n_method and break_even_n — the trial count at which this very result would deflate out — arrive in every budget block.

What does the budget arithmetic say about your numbers?

The thresholds are public — so ask them directly. This widget queries the live endpoint GET /v1/public/family-budget: what survival demands at your trial count, and the trial number at which your very result stops clearing the family threshold. It is pure mathematics over the published thresholds; the endpoint reads no family data and stores nothing.

The cross-sectional variance of no-edge trial Sharpes is set to 1/observations — the same neutral assumption as the DSR calculator — and both thresholds (expected-max-Sharpe deflation AND multiple-testing significance) are the ones every live verdict applies. Free, no account.

Isn't that harsh? I only submitted one variant.

It is the opposite of harsh — it is the only reading under which your pass means something. A validator that forgets your retries will eventually bless one of them, and that blessing is worthless precisely because it was inevitable. Demote-only plus cumulative family accounting is what makes a survivor rare, and rarity is the entire value of the verdict. The budget is not a paywall, either: it is a property of the mathematics, not of the pricing — the four gates spend it whether you check honestly or not. The only choice is whether anything keeps the receipts.

What should you do with a deflated-out family?

Stop digging in dead ground. deflated_out_at_n=N is not an insult; it is the most expensive lesson in quantitative finance delivered at five cents: this vein is mined out. The productive move is a genuinely different hypothesis — different information source, different mechanism, different family — not the eleventh cousin of the tenth variant. The overfitting checklist has the manual version of this discipline; the battery runs it automatically, with a memory.