The Anatomy of a Signal: From Market Data to Tradable Hypothesis

Generate a thousand strategies whose true expected return is exactly zero. Backtest each on ten years of daily data, keep the one with the best in-sample Sharpe ratio, and report it. On average that number is 1.46. Roughly a third of the time it exceeds 1.5. Out of sample, the strategy you selected delivers 0.01.

Figure 1 is that experiment across trial counts from one to five thousand. Nothing in it has any edge. The entire rising line is the search.

Section 01What a backtested number measures

The rising line tracks the analytic expected maximum of N estimates almost exactly, which is the point: it is not a finding about markets but about order statistics. Searching harder produces a better-looking winner whether or not anything is there to find, and the improvement is predictable enough to be plotted.

Two readings from the experiment are worth keeping. With a hundred trials and no edge whatsoever, there is a 71 per cent chance of producing an in-sample Sharpe above 1.0. With a thousand trials, a 34 per cent chance of exceeding 1.5. Those are numbers a research meeting would take seriously, and they arise here from noise alone.

This is why a reported Sharpe ratio without a trial count is close to uninterpretable. The same figure means something quite different after three attempts and after three thousand, and only the researcher knows which. Harvey, Liu and Zhu made the corresponding argument about the published factor literature, concluding that heavy mining of the cross-section requires a hurdle closer to a t-statistic of 3 than the textbook 2. Bailey, Borwein, Lopez de Prado and Zhu made the operational version: as the number of configurations grows, an impressive backtested Sharpe becomes increasingly likely under selection alone.

Section 02The harder result

The experiment has a second part that is more uncomfortable than the first. Plant five genuine signals, each with a true annualised Sharpe of 0.5, among a thousand null candidates. Rank everything by in-sample performance and take the top five.

None of the five genuine signals appears in that top five. The selected group delivers minus 0.11 out of sample. The five real signals, had they been identified, would have delivered 0.49, close to their true value.

The edge was there. The search found it, in the sense that the candidates were in the sample. Ranking by in-sample performance then discarded all of it in favour of five lucky nulls, because a true Sharpe of 0.5 estimated over five years is worth less to the ranking than a lucky draw from a thousand attempts. Search does not merely manufacture false positives. It actively buries small true effects underneath them.

Section 03The attrition after the statistics

Suppose a feature survives all of that and carries a plausible mechanism. It now meets the frictions that turn paper edge into realised edge, or into nothing. The first is cost. A strategy measured at the midpoint assumes it can trade at a price no counterparty is offering, and Issue 003 puts numbers on what that assumption is worth: at 400 basis points of gross alpha, the break-even turnover is roughly 25 times a year in large caps and 8 in small caps.

The second is decay. Expected return is a function of how many people are trading the same thing, so an edge that becomes known compresses. The third is regime change: a relationship that held across one volatility environment, one rate regime, one market structure can stop holding when those conditions change. None of the three is a modelling error. They are properties of trading in a system that adapts to being traded.

for each candidate:
    record(data_vintage, universe, feature, parameters, benchmark)
    record(all_trials, including every rejected configuration)
    estimate(out_of_sample_return, turnover, cost, capacity)
    report(sensitivity_across_periods_and_specifications)

simulated, zero true edge, 2520 daily observations split 1260/1260:

    trials     mean best in-sample Sharpe     same strategy, out of sample
         1                 -0.02                        0.06
       100                  1.12                        0.02
     1,000                  1.46                        0.01
     5,000                  1.65                       -0.03

    P(best in-sample Sharpe > 1.0 |   100 trials, no edge)  =  71%
    P(best in-sample Sharpe > 1.5 | 1,000 trials, no edge)  =  34%

Section 04The protocol is the product

If apparent edge is abundant and durable edge is rare, the scarce asset on a research desk is a process that reliably distinguishes them. That process keeps a log of every configuration tried, so the denominator is never hidden from whoever reads the result. It reserves a genuinely untouched out-of-sample period and refuses to spend it casually. It paper-trades before it risk-trades, so the gap between modelled and realised fills is measured rather than assumed. It sizes risk to the uncertainty of the estimate rather than to the attractiveness of the backtest. And it treats a fading signal as expected news.

The backtest is not the evidence. The process that produced it is the evidence.

Figure 2 is therefore an evidentiary lifecycle rather than a production line. Reconstruction establishes that the observation existed. Multiple-testing control establishes how surprising it is relative to the search. Execution modelling establishes whether it can be implemented. Live monitoring tests whether the proposed mechanism remains compatible with new data. The research object is the feature, its provenance, its feasible trading policy, and the conditions under which belief in it should be reduced.

The planted-signal result suggests one further discipline that the lifecycle diagram does not capture. If ranking by in-sample performance destroys small true effects, then a desk with a genuine prior about mechanism should be willing to advance a candidate that did not win, and to decline one that did. That is uncomfortable to operate and hard to defend in a meeting, and the alternative is a selection rule that reliably picks the wrong five.

This research is analysis and commentary for general information. It is not investment advice, an offer, or a solicitation, and it contains no price forecasts. Figure 1 is simulated under the parameters and seed declared in its notes.