Golden specimens

Golden specimens are known-answer test cases — the standing offer to catch us being wrong before you pay us. Each is a prepared signal with a planted property and a known correct verdict.

specimen	planted property	expected verdict
golden_clean.json	a genuine, persistent edge	✓ pass
golden_lookahead.json	a whipsaw with no net-of-cost edge	✕ fail · net_edge · no_net_edge
golden_cherry.json	best-of-many parameter cherry-pick	✕ fail · family_deflation · deflated_out_at_n=50 + BACKTEST_TOO_SHORT_FOR_N=50
golden_thin.json	too little data to judge	insufficient_evidence

Every file above is a direct download, and next to each sits its full expected response (golden_*.expected.json, e.g. for the whipsaw) — post the specimen body to POST /v1/assay/demo exactly as downloaded. A demo call without a specimen body returns the built-in 40-trade example in the validate envelope instead; the gauntlet-shaped verdict above appears when you send a specimen. One deliberate exception: golden_thin has too little data for the gauntlet to run at all, so its response comes back in the validate envelope (the verdict object nested under verdict) — assert against its expected.json rather than assuming the gauntlet shape.

When you diff a live response against its expected.json, ignore exactly three paths — _meta.as_of, verdict.provenance.generated_at and explanation.provenance_attestation.generated_at (top-level provenance in the gauntlet shape) — because they are timestamps of the call itself; every other byte, including input_digest and request_hash, is deterministic and bound to the _meta.engine_version recorded inside the fixture, so a mismatch elsewhere means the engine changed and the fixture republication is owed, not that your comparison is wrong.

These four rows are the known-answer section of the public Signal Validation Benchmark — the same specimens, scored in the open.

What do the golden specimens prove?

Together, the four cover the three things you should demand of any validator: it catches real flaws (the costless whipsaw, the cherry-pick), it doesn't cry wolf (clean passes), and it admits when it cannot know (thin abstains). A validator that only ever says no is as useless as one that only says yes.

How do I use golden specimens in CI?

Run them via POST /v1/assay/demo (quickstart), free — and assert against the envelope the specimen actually returns, because two of them differ:

golden_lookahead, golden_clean, golden_cherry → gauntlet envelope: assert the top-level verdict, died_at and failure_codes.
golden_thin → validate envelope, because there is too little data for the gauntlet to run at all: assert the nested verdict.verdict (insufficient_evidence). There is no top-level died_at or failure_codes here — asserting them will fail, and that is the specimen documenting the abstention, not a defect.

Each specimen ships its full expected response next to it (*.expected.json), so the safest assertion is a comparison against that file with one exception: _meta.as_of is the only field that varies between two identical calls — everything else, including the digests, is deterministic and can be compared byte for byte.