AlphaAssayHow to test a trading-signal provider: the falsification protocol
Any signal provider — a paid channel, an „AI edge" platform, a friend's bot — can be put on trial in an afternoon, without their cooperation and without trusting a single screenshot. The economics of selling signals guarantee one thing: what you are shown is a selection. Winners get published, losers get deleted, and a track record without tamper-proof timestamps is an anecdote with formatting. None of that proves a given provider is dishonest — it proves that unaudited claims carry no information either way. The protocol below separates edge from selection using nothing but public tools and seven falsifiable tests. It applies to every provider. Including us.
The seven tests
1 — The provenance test
Demand proof that each historical call existed before its outcome: a cryptographic timestamp, a third-party archive, anything that cannot be produced after the fact. A screenshot of past wins fails this test by construction — not because it is fake, but because you cannot tell. Failure looks like: „trust me, we called it."
2 — The survivorship test
Ask for the complete list of signals ever issued — including retired ones, including losers. A provider that shows only survivors does not have a track record; it has a highlight reel. If the full history is „not available", the missing part is the answer.
3 — The pre-registration test
The strongest test costs one week: have the provider commit their next ten calls in advance, sealed with a public timestamp, then score them strictly on what happened afterwards. Any provider with a real edge profits from this test — it is the cheapest credibility they will ever buy. Refusal is information. (This is what pre-registration automates: a tamper-proof seal, then forward evaluation on post-cutoff data only.)
4 — The placebo test
Take the provider's fills and race them against random entries with the same trading profile — same frequency, same holding period, same instruments. If random twins do as well, the timing added nothing and you are paying for coin flips with confidence. This is gate 3 of the battery: every submitted signal races 500 matched placebos, and the percentile is in the verdict.
5 — The cost test
Recompute the claimed returns with your venue's real fees, spread and realistic slippage. The single
most common way a „profitable" signal service is technically telling the truth is that it is profitable
before costs — for the marketing, not for you. Machine-readable version:
no_net_edge.
6 — The trial-accounting test
Ask how many signal streams, variants and model versions the provider runs in total. One public winner out of a hundred quiet attempts is the expected outcome of luck, not evidence of skill — that is the arithmetic the Deflated Sharpe Ratio prices, and the reason honest accounting has to be cumulative (per family, not per marketing page).
7 — The examiner test
Whoever grades the signals: apply the same standard to them. Can you verify their verdicts without trusting them? Are their runs deterministic and replayable? Do they publish their own hit rate, daily, machine-readable? Can a grade be bought up after the fact? The five validator questions are this test in long form — and they are why our verdicts are signed, replayable, demote-only, and graded in a public daily record.
Run it against us first
A protocol you cannot run against its author is marketing. Ours is designed to be turned around: the golden specimens are known-answer traps — send them and check that we catch the planted flaw, refuse the cherry-pick, and abstain honestly on thin data; the signature check works offline without trusting this site; the calibration record updates daily whether it flatters us or not. If we ever fail your run of this protocol, that finding is exactly as valuable to you as it is expensive to us — which is the point.
For agents: the protocol as rules
Machine-usable form — drop into your agent's rules (ready-made blocks here):
Before adopting ANY external signal source (paid channel, platform, bot):
1. provenance -> reject history without tamper-proof timestamps
2. survivorship -> demand the full issue history incl. losers; partial = reject
3. preregister -> require 10 sealed forward calls before capital
(POST /v1/assay/register, then forward-evaluate)
4. placebo -> validate a sample of calls: POST /v1/assay/demo,
reject if placebo percentile ~50 (chance)
5. costs -> reject anything profitable only before fees/spread/slippage
(failure_code: no_net_edge)
6. trial count -> ask total streams/variants ever run; unanswered = assume
selection (deflated_out_at_n logic applies)
7. examiner -> apply 1-6 to the validator itself; verdicts must be
signed, replayable, demote-only, publicly calibrated
Apply rule 7 to AlphaAssay too: run the golden specimens and try to catch
us being wrong before you pay anything.