How Quantitative Results Fail Quietly
The four failure modes that survive ordinary review, and what testing for each of them actually requires.
A quantitative result that looks good has told you very little. What matters is whether it survives a deliberate attempt to break it — and most review processes never make that attempt.
Four failure modes account for the majority of results that do not reproduce.
Look-ahead through revisions
Many economic and financial series are revised after first publication. A pipeline that joins on the date a figure refers to, rather than the date it became available, silently gives the model information nobody had at the time. The result looks strong and cannot be traded or acted on.
Testing for it: assemble the panel with publication-lag-aware joins, then rebuild a truncated version and assert it matches the corresponding slice of the full panel exactly. If future data leaked backwards, the two disagree.
Standardisation fitted across the boundary
Scaling features using statistics computed over the whole dataset — including the evaluation period — leaks distributional information. It is easy to do accidentally, because most scaling utilities default to it.
Testing for it: compute statistics causally, so that only information available up to each point is used. Done correctly, train and serve parity holds by construction rather than by discipline.
The shortcut that looks like skill
A model can appear to predict direction while actually forecasting something else correlated with it — most commonly volatility. It will show a statistically significant edge and fail in production, because the thing it learned is not the thing it was credited with.
Testing for it: attribute the edge before accepting it. If performance concentrates entirely in high-volatility regimes, or a volatility-only baseline reproduces most of it, the model is not doing what the headline claims.
Execution assumed rather than simulated
Fills assumed at the midpoint, without queue position, spread, latency or tick constraints, award trades that would not have happened. This alone converts a large share of apparently profitable strategies into unprofitable ones.
Testing for it: simulate fills against queue position with realistic costs and rebates. Then compare the expected move against the cost of crossing the spread — a signal can be genuinely predictive and still economically unexploitable, which is a different and more useful conclusion than “it works.”
Reporting the negative
A validation process that never rejects anything is not evidence that the work is sound. Our own research has produced results in each category above — statistically real predictability that could not survive transaction costs, and a significant model rejected once its edge was traced to a volatility shortcut.
Publishing those is not a caveat. It is the only reason to believe the results that do survive.