When a Model Stops Describing the Market

A model can be wrong in more than one way. Its errors may widen while the fitted relationship survives; a coefficient may drift before recent losses expose it; residuals may acquire a sustained bias. A generic regime-change alert collapses those failures into one signal and leaves the diagnosis unresolved.

A controlled experiment separates them. The same fixed production model faces three known breaks, while outcome RMSE, a rolling OLS coefficient and residual CUSUM each operate at approximately the same 5 per cent false-alert rate. Detection power is specific to the statistic. At the primary break sizes, CUSUM finds every mean shift, the coefficient monitor every coefficient break, and RMSE every volatility increase. No monitor dominates because each watches a different property of the model.

Section 01Failure has more than one meaning

A run of losses is an outcome, not proof that the model has changed. Noise puts bad runs inside a valid forecast distribution. Conversely, a coefficient can move while forecasts remain acceptable because another component offsets the change or the signal is small relative to residual noise.

Structural change is a claim about the data-generating process. Simulation can supply that label because the process is known; market data cannot. Operational model failure is broader still. A changed coefficient may matter for one decision and not another, while bad data, a shifted population or an unvalidated use can make an unchanged model unsafe. Parameter movement also says little about economic materiality until it is connected to the model's decision rule. A useful monitoring design tests chosen assumptions and tracks outcomes separately.

Section 02An experiment where the break is known

The baseline process is y[t] = 0.6x[t] + epsilon[t], with independent standard-normal signal and residual. The production coefficient remains fixed at 0.6. At observation 250 of a 500-observation monitoring period, exactly one feature changes: the intercept rises by half a residual standard deviation, the coefficient falls from 0.6 to zero, or residual volatility increases by 50 per cent. These are controlled versions of sustained forecast bias, loss of the fitted relationship and higher error scale.

Each monitor receives observations sequentially. Outcome RMSE measures trailing forecast-error scale. The parameter statistic is the absolute standardised distance between a trailing OLS estimate and 0.6. Residual CUSUM accumulates sustained signed errors, moving back towards zero when they reverse. Every value at time t uses data through t.

The diagonal of Figure 1 carries the result. At the common false-alert budget, each matched monitor detects 100 per cent of its primary break. RMSE finds the volatility increase after a conditional median of 31 observations, the beta monitor finds the coefficient break after 41, and CUSUM finds the mean shift after 43. Off the diagonal, power falls sharply: the beta monitor detects only 8.1 per cent of mean shifts, while CUSUM detects 14.1 per cent of coefficient breaks.

RMSE is broader and less diagnostic in this experiment. It detects 72.0 per cent of coefficient breaks after a conditional median of 101 observations, and 42.4 per cent of mean shifts after 121. Squaring the residual discards the sign and source that distinguish those failures. The ranking changes with the break because outcome scale, coefficient stability and signed residual bias are different quantities.

stable process:     y[t] = 0.6 x[t] + epsilon[t]
production forecast:                 0.6 x[t]
standardised residual:   z[t] = y[t] - 0.6 x[t]

outcome monitor:      sqrt(mean(z[t-w+1:t]^2))
parameter monitor:    abs(beta_hat[t-w+1:t] - 0.6) / se
residual CUSUM:       max(0, previous + or - z[t] - allowance)

5,000 stable calibration paths set each threshold
5,000 new paths measure false alerts, detection and delay
break starts at 250; horizon ends at 500; primary w = 60
seed = 20260901

Section 03Reaction speed has a price

For the full coefficient fall of 0.6, a 20-observation beta monitor detects 99.6 per cent of breaks with a median delay of 38 observations. Its apparent advantage fades when the coefficient change is 0.3. Detection falls to 42.8 per cent, compared with 83.8 per cent for the 60-observation window and 93.9 per cent for 120. Their conditional median delays are 115, 94 and 103 observations. The shortest estimate often cannot separate the smaller shift from its own sampling variation.

The RMSE monitor has the same constraint. When residual scale rises by 25 per cent, detection is 84.2 per cent at 20 observations, 97.1 at 60 and 99.1 at 120. Responsiveness depends on the size of change the monitor is required to distinguish.

Section 04A live example without a true regime label

Simulation reveals the break and permits detection accuracy to be scored. Markets provide no such label. Live monitoring instead identifies evidence that warrants investigation, with the cause and timing left as separate inferences.

The empirical example uses daily US Treasury par yields. A fixed OLS model trained on 1250 observations from 2010 to 2014 predicts the daily change in the ten-year yield from the change in the two-year yield. Its beta is 1.343, and the residual scale used to normalise RMSE is 4.07 basis points. From 1 January 2015, the model is monitored without refitting. The primary statistic is trailing 60-session RMSE divided by that training scale. Its threshold of 1.848 is the training period's 99th percentile and is frozen before monitoring begins.

The first breach occurs on 13 March 2023, after several sessions in which the two-year yield moved much more sharply than the ten-year yield. The statistic remains above its boundary for 68 sessions and peaks at 3.23 times training error on 26 May 2023. The alert is causal: 13 March is the first crossing of a pre-existing threshold. The 2020 shock never crosses that boundary. This monitor measures error in the two-year to ten-year relationship, not unconditional yield volatility.

The lower panel supplies diagnostic context. The trailing 126-session beta is 0.65 at the breach, well below the training estimate, and ends the sample at 0.73. Its decline began before March 2023, while the outcome statistic also reflects larger residual moves. The alert establishes a departure from training tolerance; the parameter path helps characterise it, but does not prove a cause or a single break date.

Causal smoothing choices produce materially different alert histories. The 20-session monitor first alerts on 15 June 2022 and fragments into 5 episodes. The 60-session version waits until March 2023 and produces one 68-session episode. At 120 sessions, the first alert is 13 July 2022, followed by 6 episodes and 208 sessions above a lower training-calibrated threshold. These are outputs from three filters, not estimates of one hidden break date.

Section 05What an alert should change

An alert should trigger a defined response. First validate the data, mappings and production inputs. Then review forecast errors and parameters against relevant benchmarks, and restrict model use temporarily where the decision risk warrants it. Recalibration follows diagnosis. An immediate refit can convert noise into a new parameter and erase the evidence needed to understand the alert.

The April 2026 Federal Reserve, OCC and FDIC guidance gives institutional support to the same separation. For banking organisations, it treats conceptual soundness, outcome analysis and ongoing monitoring as distinct parts of model validation, including whether vendor models remain fit for purpose. It neither governs trading strategies nor prescribes the statistics used here. Traditional statistical and quantitative models, along with non-generative, non-agentic AI models, fall within its stated principles; generative and agentic AI sit outside that guidance. The agencies still expect organisations to determine appropriate governance and controls for systems beyond its scope.

Operationally, each statistic should be attached to a named model assumption, an investigation threshold and a set of permitted responses. That record determines whether an alert leads to a data fix, closer review, restricted use or model replacement. The monitoring system is the connection between evidence and those decisions; the anomaly score is only one component.

This article is general research and commentary, not investment advice or a trading signal.