Backtests Do Not Fill Orders

Take a strategy that earns 400 basis points a year before costs, and ask only one further question: how often does it trade? Under the cost model in Figure 1, that same gross alpha survives 24 times annual turnover in large-cap US equities, breaks even at about 15 times in mid caps, and is already negative past roughly 8 times in small caps. Nothing about the signal changes across those three answers. The strategy is good, marginal, or a loss depending entirely on what it trades and how often.

This is arithmetic under stated assumptions rather than a measurement, and Figure 1 says so. But the arithmetic is where a research process either builds in its execution constraint or discovers it later at cost.

Section 01Paper versus reality

The cleanest name for the gap is old. In 1988 Andre Perold described the implementation shortfall: the difference between the return of a paper portfolio, whose trades are recorded at the prices observed when the decision was made, and the return of the real portfolio that had to be traded into existence. The backtest is the paper portfolio. Perold's point was that the cost of trading is not the commission line but the whole distance between decision and execution, including the part that never appears on an invoice because it takes the form of a worse price.

Marking an order at the midpoint records a price, not an execution. No counterparty is obliged to trade there, queue priority does not exist in the simulation, and the simulated order cannot move the book it is supposed to be crossing. A researcher can spend months refining a signal and thirty seconds assuming it fills at the mid. The assumption is silent and, for anything fast, enormous.

Section 02Where the edge goes

Figure 1 makes the arithmetic explicit. Cost per unit of turnover is half the quoted spread plus a market-impact term that scales with the square root of participation, the concave form associated with Almgren and Chriss and with the empirical impact literature. Under the parameters declared in the figure, that comes to roughly 16 basis points per unit of turnover in large caps, 27 in mid caps and 47 in small caps. Multiply by turnover and subtract.

The right-hand panel inverts the question into the number a research desk should actually be comparing its backtest against. A strategy turning over eight times a year in mid caps must produce about 214 basis points gross simply to reach zero. Comparing its backtested return to zero, rather than to that hurdle, is the specific error the panel is drawn to prevent.

Two properties of the arithmetic matter more than any single parameter. Every term subtracts, so the error introduced by a mid-price backtest is one-directional rather than symmetric. And the terms are largest for high-turnover, capacity-hungry strategies, which is exactly the sort a broad specification search tends to surface. The backtest does not err at random. It flatters, and it flatters hardest the strategies that most deserve suspicion.

The sensitivity to the impact assumption is worth stating rather than hiding. Holding everything else fixed and moving the impact coefficient across a plausible range from 0.3 to 1.5 moves the large-cap break-even from 48 times turnover to 11. Moving participation from 1 to 25 per cent of average daily volume moves it from 53 times to 12. A desk that has not measured its own impact does not know which of those worlds it is in, and the difference between them is the difference between a viable strategy and a fee-paying one.

Section 03You cannot trade your way out

A natural response is to trade more cleverly, and there is real science in doing so, but it has a hard limit. Almgren and Chriss framed optimal execution as a trade-off with no free corner. Trade quickly and you minimise the risk of the price drifting away while you wait, but you maximise impact. Trade slowly and you minimise impact but accept timing risk. They separated permanent impact, the lasting price change your trading causes, from temporary impact, the transient cost of demanding immediacy, and showed that the best available outcome is a point on an efficient frontier trading expected cost against the variance of that cost. There is no schedule that escapes the trade-off, only a considered choice of where to sit on it.

The deeper consequence concerns capacity. Because impact grows with size relative to depth, edge per dollar shrinks as a strategy scales, and there is some size at which the marginal trade consumes the marginal alpha. A signal can be real and still be uninvestable at scale. That is a statement about the order book rather than about the idea, and it is why execution belongs inside the research process rather than as a haircut applied at the end.

cost per unit of turnover, in basis points:

    c(q)  =  s/2  +  k * sigma * sqrt(q)

        s      quoted spread, bp
        sigma  daily volatility of the traded name, bp
        k      impact coefficient, dimensionless
        q      order size as a fraction of average daily volume

net alpha:        a_net  =  a_gross  -  T * c(q)
break-even:       T*     =  a_gross / c(q)

implementation_shortfall = paper_return - real_return
                         = fees + spread + slippage + impact + timing

A backtest does not err at random. It flatters, and it flatters hardest the strategies that most deserve suspicion.

The research object is therefore alpha conditional on an execution policy, not raw alpha. That means estimating cost against a declared benchmark, scaling impact with participation and depth, keeping rejected and unfilled orders in the sample, and testing stressed liquidity separately from average conditions. A strategy has survived implementation only when its residual return stays credible under those assumptions. Until then the backtest has established a forecast without establishing that the forecast can be owned.

This research is analysis and commentary for general information. It is not investment advice, an offer, or a solicitation, and it contains no price forecasts. Figure 1 is simulated under the parameters declared in its notes; the cited works are the basis for the cost specification, and the interpretation is the author's.