AlphaAssayThe Probability of Backtest Overfitting (PBO), explained — with a calculator
The Probability of Backtest Overfitting (PBO) answers the question that deflation alone cannot: when you pick the best of many strategy configurations, how often does that in-sample winner turn out to be below-average out of sample? If the answer is around 50%, your selection process is a coin flip — the textbook signature of overfitting. Bailey, Borwein, López de Prado and Zhu (2017) estimate it with Combinatorially Symmetric Cross-Validation (CSCV): split the history many ways, and each time check where the in-sample champion lands out of sample. The calculator below runs the full CSCV on a performance matrix — the example data, or your own — entirely in your browser. It needs a matrix, not a single number, because overfitting is a property of the search, not of one equity curve.
The calculator
The box loads with an illustrative synthetic matrix — nine pure-noise configurations plus one with a genuine edge. At S = 8 blocks it returns PBO ≈ 0.01, the „1 real edge" row below. Swap in the „10 pure noise" example to watch PBO jump to ≈ 0.53. Paste your own matrix of strategy returns to test a real search.
What does PBO look like on known data?
| example matrix (16 obs × 10 configs, S = 8) | what it is | PBO | reading |
|---|---|---|---|
| 10 pure noise | no real edge — the best is chosen by luck | 0.53 | ≈ a coin flip out of sample — textbook overfitting |
| 1 real edge + 9 noise | one genuinely persistent signal among decoys | 0.01 | the real edge survives CSCV; overfitting is unlikely |
Both rows are computed by the calculator above from the same embedded matrices — click the example links to reproduce them. The contrast is the whole point: overfitting is not about how good the winner looks in-sample, but whether it keeps winning out of sample.
How CSCV computes PBO, step by step
Step 1 — build the matrix. Rows are observations (returns), columns are the strategy configurations you tried. Overfitting lives across columns, so you need all of them — including the ones you would have discarded.
Step 2 — split symmetrically. Cut the rows into S equal blocks (S even). For every way of
choosing S/2 blocks as in-sample, the remaining half is out-of-sample — that is
C(S, S/2) balanced splits (S = 8 → 70).
Step 3 — follow the in-sample winner. In each split, find the configuration with the best
in-sample Sharpe, then read its out-of-sample rank. Its relative rank ω ∈ (0,1)
gives the logit λ = ln(ω / (1 − ω)); λ ≤ 0 means the in-sample champion landed
in the bottom half out of sample.
Step 4 — the probability. PBO = (share of splits with λ ≤ 0). PBO near 0.5 means
your selection carries no out-of-sample information — pure overfitting; PBO near 0 means the winner keeps
winning.
How does PBO relate to the Deflated Sharpe Ratio?
They attack the same disease from opposite ends. The Deflated Sharpe Ratio prices the selection inflation in a single number when you can count your trials; PBO measures the out-of-sample consistency of the selection process itself when you have the full performance matrix. Neither is a promise of returns — both are robustness gates, and both inform gate 2 and gate 4 of the AlphaAssay battery. Related: walk-forward analysis · the overfitting checklist.