Selection bias in strategy search.

The problem

Testing many candidate strategies and reporting the best is a maximum, not a sample. The expected maximum of N draws from a null distribution grows with N, so the winning backtest looks better the harder you searched — and the appearance is entirely mechanical.

E[ max SR_null ] ≈ σ_SR · √( 2 ln N )

N

number of independent strategy configurations tested

σ_SR

standard error of the Sharpe estimate

The correction

Two things are required, and they are cheap. Record the number of configurations actually tested — including the ones abandoned early, which is where the count usually goes wrong — and control the false discovery rate across the family of tests rather than the error rate of each in isolation. Benjamini–Hochberg is adequate and does not assume independence between strategies, which matters because candidate strategies are typically correlated with one another.

The practical consequence is that a result must clear a threshold that rises with the intensity of the search that produced it. Most do not. That is the correct outcome and the reason the procedure is unpopular.

What this costs us

It makes the research process slower and the reported numbers smaller than they would otherwise be. It also means a strategy that survives has survived something, and that the number attached to it is a number rather than an artefact of persistence.

We report the search count alongside every result, in the same way we report the half-life and the duplication factor. All three are terms most likely to be flattering by accident.

What would falsify this

• Evidence that corrected thresholds systematically discard strategies that then perform out of sample. That would indicate the correction is too aggressive for correlated families and the procedure needs replacing, not abandoning.
• A search process genuinely free of selection — a single hypothesis specified in advance and tested once. Where that is the case the correction is unnecessary, and saying so is part of reporting it.
• Any demonstration that recorded search counts are unreliable in practice, which would make the whole correction unverifiable by a third party and reduce it to a claim of good faith.

Research commentary on measurement. It is not an offer, a solicitation, or investment advice, and it recommends no security, strategy or transaction.