Backtests Do Not Fill Orders

A backtest is a conditional simulation, and its most consequential condition is often the fill model. Marking an order at the midpoint records a price, not an execution: no counterparty is obliged to trade there, queue priority is absent, and the simulated order cannot alter the book it is supposed to cross. Live trading introduces spread, delay, impact, and selection effects that are economically related rather than independent. The gap between decision price and realised result is implementation shortfall, and for short-horizon strategies it can dominate the signal being tested.

Section 01Paper versus reality

The cleanest way to name this gap is old. In 1988 Andre Perold described the implementation shortfall: the difference between the return of a paper portfolio, whose trades are recorded at the prices observed when the decision was made, and the return of the real portfolio that actually had to be traded into existence. The backtest is the paper portfolio. The live book is reality. Perold's point was that the cost of trading is not the commission line; it is the whole distance between the decision and its execution, including the part that never appears on any invoice because it takes the form of a worse price. Once you measure cost that way, a strategy is not evaluated by the edge it shows but by the edge that survives being implemented.

This reframing is uncomfortable because it moves the hardest part of the work to the end, where it is easiest to wave away. A researcher can spend months refining a signal and minutes assuming it fills at the midpoint. The assumption is silent and enormous. For a slow, patient strategy it may be nearly harmless. For a fast one it can be the entire result, because the same property that makes a short-horizon signal attractive, frequent trading on small predicted moves, also makes it pay the execution tax most often.

Section 02Where the edge goes

Figure 3 follows a unit of simulated alpha through an execution ledger. Fees are explicit. Spread is the price of immediacy relative to the prevailing quote. Delay and latency change the benchmark before the order arrives. Impact is the price response associated with the order itself, while adverse selection describes fills that occur disproportionately when subsequent prices move against the liquidity provider. These labels overlap in empirical measurement, so a serious transaction-cost analysis chooses a benchmark and decomposition convention before assigning quantities. What matters is not the cosmetic precision of each bar but whether the total shortfall is estimated out of sample and at the intended scale.

Two features of the waterfall matter more than the size of any single step. The first is direction: every cost subtracts, none adds, so the error in a mid-price backtest is biased, not symmetric. The second is correlation with attractiveness: the steps are deepest for high-turnover, capacity-hungry strategies, which is exactly the kind a naive search tends to surface. The backtest does not lie at random. It flatters, and it flatters most the strategies you should most distrust.

Section 03You cannot trade your way out

A natural response is to trade more cleverly, and there is real science in doing so, but it has a hard limit. Almgren and Chriss, in their work on optimal execution, framed the problem as a trade-off with no free corner. Trade quickly and you minimise the risk that the price drifts away while you wait, but you maximise market impact. Trade slowly and you minimise impact, but you take on timing risk, the chance that the market moves against you before you are done. They distinguished permanent impact, the lasting price change your trading causes, from temporary impact, the transient cost of demanding immediacy, and showed that the best a trader can do is choose a point on an efficient frontier that trades expected cost against the variance of that cost. There is no schedule that escapes the trade-off; there is only a considered choice of where to sit on it.

The deeper consequence is about capacity. Because impact grows with size relative to depth, a strategy's edge per dollar shrinks as it scales, and there is some size at which the marginal trade eats the marginal alpha entirely. A signal can be genuine and still be uninvestable at scale, which is a statement about the order book, not the idea. This is why execution belongs inside the research process rather than as a haircut applied at the end. The questions are continuous with the signal itself: what will I actually pay to cross, how much will I move the price, and at what size does the edge disappear?

implementation_shortfall  =  paper_return - real_return
                          =  fees + spread + slippage + impact + timing

optimal execution (Almgren-Chriss):
    minimise   E[cost]  +  lambda * Var[cost]
    trade fast -> impact up,  timing risk down
    trade slow -> impact down, timing risk up
    => an efficient frontier, not a costless corner

A backtest does not lie at random. It flatters, and it flatters most the strategies you should most distrust.

The research object is therefore not raw alpha but alpha conditional on an execution policy. That means estimating cost against a declared benchmark, scaling impact with participation and depth, preserving rejected and unfilled orders in the sample, and testing stressed liquidity separately from average conditions. A strategy has survived implementation only when its residual return remains credible under those assumptions. Until then, the backtest describes a forecast without establishing that the forecast can be owned.

This research is analysis and commentary for general information. It is not investment advice, an offer, or a solicitation, and it contains no price forecasts. The figure is illustrative; the cited works are the basis for the framework, and the interpretation is the author's.