How to test a trading-signal provider: the falsification protocol
Any signal provider — a paid channel, an „AI edge" platform, a friend's bot — can be put on trial in an afternoon, without their cooperation and without trusting a single screenshot. The business model creates strong selection incentives: treat the displayed history as selected until the provider supplies the complete issue history, including losers and retired calls. A track record without independently archived or pinned timestamps is an anecdote with formatting. None of that proves a given provider is dishonest — it proves that unaudited claims carry no information either way. The protocol below separates edge from selection using nothing but public tools and seven falsifiable tests. It applies to every provider. Including us.
The seven tests
1 — The provenance test
Demand proof that each historical call existed before its outcome: a cryptographic timestamp, a third-party archive, anything that cannot be produced after the fact. A screenshot of past wins fails this test by construction — not because it is fake, but because you cannot tell. Failure looks like: „trust me, we called it."
2 — The survivorship test
Ask for the complete list of signals ever issued — including retired ones, including losers. A provider that shows only survivors does not have a track record; it has a highlight reel. If the full history is „not available", the missing part is the answer.
3 — The pre-registration test
The strongest test costs one week: have the provider commit their next ten calls in advance, sealed with a public timestamp, then score them strictly on what happened afterwards. Any provider with a real edge profits from this test — it is the cheapest credibility they will ever buy. Refusal is information. (AlphaAssay pre-registration records an operator-published chained commitment, then evaluates post-cutoff data; independent timestamp trust requires an external anchor.)
4 — The placebo test
Take the provider's fills and race them against random entries with the same trading profile — same frequency, same holding period, same instruments. If random twins do as well, the timing added nothing and you are paying for coin flips with confidence. When the gauntlet's placebo stage is included, it runs the requested/configured placebo count within the runtime bounds and reports the resulting percentile; that percentile belongs to that requested gauntlet execution.
5 — The cost test
Recompute the claimed returns with your venue's real fees, spread and realistic slippage. The single
most common way a „profitable" signal service is technically telling the truth is that it is profitable
before costs — for the marketing, not for you. Machine-readable version:
no_net_edge.
6 — The trial-accounting test
Ask how many signal streams, variants and model versions the provider runs in total. One public winner out of a hundred quiet attempts is the expected outcome of luck, not evidence of skill — that is the arithmetic the Deflated Sharpe Ratio prices, and the reason honest accounting has to be cumulative (per family, not per marketing page).
7 — The examiner test
Whoever grades the signals: apply the same standard to them. Can you verify their separately issued
evidence against externally rooted platform trust? Do they state which pure/read calls require an explicit as_of, and which
stateful calls replay only a stored idempotent response? Do they disclose whether a mature outcome metric
exists, without turning insufficient history into a score? Can a grade be bought up after the fact? The
five validator questions are this test in long
form — and they are why our verdicts expose timestamps and replay identifiers, are demote-only,
optionally certifiable, while the
public calibration state explicitly remains insufficient-history.
Run it against us first
A protocol you cannot run against its author is marketing. Ours is designed to be turned around: the golden specimens are known-answer traps — send them and check that we catch the planted flaw, refuse the cherry-pick, and abstain honestly on thin data; the hosted certificate verifier checks platform trust, while a full offline check requires an independently pinned signed keyring and complete revocation-head history; calibration v0 publishes only its bucketed population and accumulating/insufficient-history state. If we ever fail your run of this protocol, that finding is exactly as valuable to you as it is expensive to us — which is the point.
For agents: the protocol as rules
Machine-usable form — drop into your agent's rules (ready-made blocks here):
Before adopting ANY external signal source (paid channel, platform, bot):
1. provenance -> reject history without independently archived/pinned timestamps
2. survivorship -> demand the full issue history incl. losers; partial = reject
3. preregister -> require 10 sealed forward calls before capital
(POST /v1/assay/register, then forward-evaluate)
4. placebo -> validate a sample of calls: POST /v1/assay/demo,
reject if placebo percentile ~50 (chance)
5. costs -> reject anything profitable only before fees/spread/slippage
(failure_code: no_net_edge)
6. trial count -> ask total streams/variants ever run; unanswered = assume
selection (deflated_out_at_n logic applies)
7. examiner -> apply 1-6 to the validator itself; pure/read work must
name as_of, stateful replay must be stored and idempotent;
issue and verify a certificate
separately when signed evidence is required; accept calibration
data only from a valid signed snapshot
Apply rule 7 to AlphaAssay too: run the golden specimens and try to catch
us being wrong before you pay anything.