5. Statistical Methodology Failures

5.1 The Multiple Comparisons Problem

Harvey, Liu, and Zhu (2016) demonstrated that the threshold for statistical significance in backtested strategies must be adjusted for the number of strategies tested. Their work, which examined the factor zoo in academic finance, showed that a $t$-statistic of 2.0 (the traditional threshold for significance at the 5% level) is statistically inadequate when hundreds or thousands of strategy variants have been evaluated. They proposed a minimum $t$-statistic of approximately 3.0 for newly discovered factors, to allow for the implicit multiple testing that runs through the field.

The problem is even more acute in retail backtesting, where far more variations are typically tested than in academic research and the count is almost never reported. A researcher who tests a moving average crossover strategy with twenty combinations of fast and slow periods, across ten instruments, with three different filters, has evaluated six hundred strategy variants. Presenting the best performer as “the strategy” without adjusting for multiple testing is meaningless, and yet it’s standard practice.

Bailey, Borwein, Lopez de Prado, and Zhu (2014) formalised this intuition with what they called the false strategy theorem. The expected maximum Sharpe ratio among $N$ independent zero-skill strategies grows as approximately $\sqrt{2 \ln N}$, which means that trying just ten configurations of a worthless strategy is expected to produce one with a Sharpe ratio above 1.5 in-sample, while the true out-of-sample expectation remains zero. They derived a corresponding minimum backtest length (MinBTL): with five years of daily data, no more than forty-five independent configurations should be tested before the expected maximum Sharpe ratio from pure noise reaches 1.0, and with only two years of data the budget drops to seven. Most retail traders blow past these thresholds before lunch on the first day of development. The arithmetic seems to promise that a longer history buys a bigger budget of trials, but that holds only if the extra years are exchangeable, drawn from the same market as the recent ones, and Regime Dependence and Non-Stationarity shows that for many short-term effects they were not.

The false strategy theorem describes a search across configurations of a zero-skill process, where the inflated Sharpe ratio is pure noise and the remedy is the multiple-testing correction the theorem prescribes. Any strategy whose payoff depends on the order of prices inside a bar is in a worse position. As Section 4.2 sets out, the OHLC path assumption generates a per-trade error that is structurally one-signed in the strategy’s favour, and the bias per trade grows as the bracket tightens relative to the typical bar range. A parameter search then prefers configurations with tight stops and tight targets, which is exactly the region of parameter space where the bias is largest. I measured the ordering error on one instrument, ES, using one-second data from June 2010 to June 2026 (Databento, 4,986 days). The open-position rule ordered the day’s high and low correctly on 81% of days, but when the open sat in the middle of the range it was right only about half the time, and those choppy days are the ones on which price sweeps through both a stop and a target, so the engine’s mistakes book wins that never happened. Optimisation hunts the parameter space for the settings these optimistic assumptions flatter most, so the search is mining a systematically positive artefact where the theorem assumes unbiased noise. The Bailey and López de Prado correction adjusts for the search and leaves untouched a bias that was already present before the search began. For path-sensitive strategies on coarse bar data the two effects multiply, and no multiple-testing correction on its own can recover honest performance metrics from the contaminated simulator.

Wiecki, Campbell, Lent, and Stauth (2016) provided striking empirical confirmation of this problem. Using a dataset of 888 algorithmic trading strategies developed on the Quantopian platform, each with at least six months of out-of-sample performance, they found that commonly reported backtest metrics such as the Sharpe ratio had almost no power to predict out-of-sample results ($R^2 < 0.025$). More pointedly, they found a statistically significant positive relationship between the amount of backtesting a researcher performed on a strategy and the size of the gap between in-sample and out-of-sample performance: the more a strategy was tested and refined, the worse it performed in live trading relative to its backtest.

This is the multiple comparisons problem made visible at scale, since each round of parameter adjustment is an implicit additional test that inflates in-sample performance while weakening the strategy’s ability to generalise to unseen data.

The same inflation shows up when published results are followed past the end of their original samples. McLean and Pontiff (2016) studied 97 published return predictors and found that their returns were 26% lower after the end of the original sample, a measure of in-sample overfitting, and 58% lower after publication. Chordia, Subrahmanyam, and Tong (2014) found that a combined anomaly strategy’s average return and Sharpe ratio roughly halved after decimalisation. I ran the same comparison on Ken French’s daily US data from July 1926 to August 2026,¹ splitting four well-known effects at their publication dates. One of them, follow-yesterday, is the simplest trend rule there is: long the market after an up day and short after a down day. Each cell in the table gives the annual return in % and then the Sharpe ratio.

These are paper portfolios with no trading costs and no shorting limits, so the follow-yesterday figures before 1975, when fixed brokerage commissions ended, are an upper bound on what anyone could have earned.

The Monday effect that French (1980) documented went the same way: in every decade from the 1920s to the 1980s, Monday’s excess return was 16 to 37 basis points below the average of the other days, and since 1990 the gap has been between −7 and +7 basis points. Schwert (2003) has the neatest illustration of the problem I know of. Pooled over 1885 to 2002, the weekend effect has a $t$-statistic of −10, but over 1978–2002 alone it was −1.4, so nearly all of the evidence came from markets that had already gone.

Momentum’s returns fell after publication but didn’t vanish, and the wider evidence on decay is mixed. Ilmanen et al. (2021) found no post-publication decay in value, momentum, carry, and defensive strategies across asset classes, and Jacobs and Müller (2020), looking at 241 anomalies in 39 countries, found that the United States was the only market with a reliable decline after publication.

The academic literature offers a family of methods designed specifically to correct for multiple testing in strategy evaluation, though retail back-testers have hardly heard of them. White’s Reality Check (2000) uses bootstrap resampling to test whether the best-performing strategy from a set of candidates is genuinely superior to a benchmark after accounting for the number of alternatives tested. Hansen (2005) refined this into the Superior Predictive Ability (SPA) test, which is more powerful against specific alternatives and has been extended by Romano and Wolf into stepwise procedures that go further and identify which strategies in a set keep their significance after correction. Bailey and Lopez de Prado (2017) proposed the Probability of Backtest Overfitting (PBO), estimated via Combinatorially Symmetric Cross-Validation (CSCV): the data is partitioned into multiple subsets, strategy parameters are fitted on every possible training combination, and the frequency with which the best in-sample configuration underperforms out-of-sample gives a direct estimate of the probability that the backtest is overfit. Their related work on the Deflated Sharpe Ratio (2014) adjusts the reported Sharpe ratio for the number of trials conducted, non-normality of returns, and sample length, and the result says far more than the raw Sharpe about whether the observed performance is distinguishable from chance. The common thread is that the number of strategies tested must be treated as a parameter of the evaluation and reported with it. Any researcher who has tested more than a handful of variants and doesn’t apply some form of multiple-testing correction is, in effect, reporting the expected maximum of a set of random draws (exactly the quantity the false strategy theorem estimates) and presenting it as an expected value.

For traders who won’t put in the effort to implement the formal corrections above, the practitioner literature offers a back-of-envelope heuristic: discount the reported expected return by a factor of $1 - 0.95^N$, where $N$ is the number of strategy variants tested.² It’s crude, but it’s still far better than the standard retail practice of testing 50 variants and reporting the equity curve of the best.

5.2 Insufficient Independent Observations

A ten-year daily backtest of a monthly rebalancing strategy produces only one hundred and twenty observations, and fewer if the strategy is not always in the market. The number of truly independent observations may be smaller still if signals are serially correlated, as they often are in trend-following and momentum strategies where a single trend can generate a cluster of correlated trades.

The statistical power of a test with one hundred and twenty observations to detect a real but modest edge (say, a Sharpe ratio of 0.5) is discouragingly low. The confidence intervals around estimated performance metrics are wide, and the probability of a truly profitable strategy appearing unprofitable (or vice versa) in a single sample is substantial. Yet backtests are routinely presented with a precision (“Sharpe ratio: 1.47”) that implies a level of certainty the data cannot support, as if two decimal places made a number reliable. Bailey and Lopez de Prado (2014) have argued persuasively that the Sharpe ratio, as commonly estimated from backtest data, is a deeply unreliable measure of forward-looking performance.

Lo (2002) gives the standard error of a Sharpe ratio estimated from $T$ years of independent returns as roughly $\sqrt{(1 + \frac{1}{2}SR^2)/T}$. For a strategy with a true annual Sharpe ratio of 0.5, that comes to about 0.34 after 10 years and 0.11 after 100, and it takes about 18 years before the estimate sits two standard errors from zero. A Sharpe ratio estimated from three and a half years has a standard error of about 0.6, against about 0.3 for a decade. Showing that a Sharpe ratio of 0.5 had halved, which is roughly the decay the literature reports after publication, would take well over a century of data on each side of the change. Fama and French (2021) ran into the same wall with value, whose premium fell from 0.42% a month in 1963–91 to 0.11% in 1991–2019. Even so, the difference between the two halves was only about 1.4 standard errors, and as they put it, “the culprit is volatility”.

All of this arithmetic assumes that every year is a draw from the same distribution. More years shrink the error only if the market stayed the same while they accumulated, and the next section shows that often it didn’t.

The pre-FOMC drift shows how little even a long run of events can settle. Lucca and Moench (2015) found that US stocks rose an average of 49 basis points in the 24 hours before scheduled FOMC announcements from September 1994 to March 2011, and Kurov, Wolfe, and Gilbert (2021) found that the drift had essentially disappeared after 2015. I measured it on ES for every scheduled meeting from June 2010, from 1:00pm CT the day before the statement to 11:15am CT on the statement day.

Taken at face value, the drift has come back since 2016. But the 2016–20 average depends heavily on a single day, 5 November 2020, the statement day after the presidential election; the 2016–19 median is only 3.9 basis points; and the 2021–26 $t$-statistic of 1.72 comes from 46 meetings. With every scheduled meeting for 16 years I still can’t tell whether the drift has returned or whether this is noise, so I’m leaving the question open. A claim that it has come back needs as much evidence as a claim that it has gone.

5.3 Regime Dependence and Non-Stationarity

Financial markets are non-stationary: the statistical properties of returns (their mean, variance, autocorrelation structure, and tail behaviour) change over time as market structure, participant composition, regulatory frameworks, and macroeconomic conditions evolve. A strategy optimised on a period dominated by a particular regime (low-volatility trending markets, for example) may perform entirely differently in another (choppy, mean-reverting markets with elevated volatility). A backtest that spans several regimes can show acceptable aggregate performance while hiding long stretches of severe underperformance, and those stretches are what a trader actually lives through.

5.3.1 A century of daily returns, decade by decade

Non-stationarity is easiest to see over a long span, so I measured Ken French’s daily US market returns from 1 July 1926 to 31 August 2026.¹ They are built from CRSP’s value-weighted index, which covers only the NYSE until July 1962. I annualised each period with its actual number of sessions a year, which matters before 1952, when the NYSE also traded on Saturdays and there were about 300 sessions a year. The last column of the table is the Sharpe ratio, before costs, of the follow-yesterday rule described earlier.

Volatility ranged from 9.8% a year in the 1960s to 30.2% in the 1930s, and the share of days that moved more than 2% either way ranged from 1% in the 1950s and 1960s to 18% in the 1930s, so a rule with a fixed stop, a fixed target, or a fixed position size sees a different market in each decade.

The lag-1 autocorrelation of daily returns changed even more: until the late 1990s it averaged about 0.15, reaching 0.29 in the 1970s, and since 2000 it has hovered around zero or below. Campbell, Grossman, and Wang (1993) reported a daily autocorrelation of about 0.2 for the value-weighted index, and nearly 0.3 in 1962–74. Baltussen, van Bekkum, and Da (2019) found the same switch from positive to negative across 20 indexes in 15 countries, and linked it to the growth of index futures, ETFs, and index funds.

Follow-yesterday feeds on that autocorrelation. Over the whole period it has a Sharpe ratio of 1.13, and its decade Sharpe ratios ran from 1.69 to 3.82 between the 1920s and the 1990s, with the 1930s the exception at 0.53. Since Lo and MacKinlay (1988) published the autocorrelation, the rule’s Sharpe ratio has been 0.00. Anyone who ran it on a hundred years of data would find 60 years of spectacular results, a bad patch, and a long-run average that looks excellent, and they’d have been wrong about it in every year of the last quarter century. The usual caveat applies with extra force here, because the portfolios carry no costs and no shorting limits and are rebalanced daily in a way nobody could have managed in 1930, so the follow-yesterday figures before 1975 are an upper bound.

Decades are an arbitrary unit, so I also let the data choose the break dates. I computed the lag-1 autocorrelation for each calendar year, searched for the segmentation into flat pieces of at least five years that minimises the sum of squared deviations, and chose the number of breaks by the Bayesian information criterion. It’s a simplified version of the method of Bai and Perron (1998, 2003), and it chose four breaks.

The last break falls in 1997, the year the SEC’s order handling rules took effect (from 20 January) and the NYSE cut its tick from an eighth of a dollar to a sixteenth (24 June), on the way to decimal prices in 2001. A break found in annual data is located only to within a few years, and I’m not claiming that tick size alone killed the effect, since electronic trading, index funds, and index futures all grew over the same years. What the data does show is a large break that lines up with a change in how the market worked, and a run of bad luck wouldn’t have produced a sign change that has lasted 30 years.

The distinction I find most useful is between edges that come from people and edges that come from plumbing. Premia for bearing risk and patterns rooted in how investors behave have survived changes of market, and momentum, the clearest example, still earned 4% a year over the last decade. Edges that come from costs, tick sizes, session hours, stale prices, or a particular kind of counterparty go when those do. The day-to-day trend needed slow price discovery and high trading costs, and both have gone.

Changes to sessions and venues are structural breaks in the data as well as in the market. Until 1952 the NYSE traded on Saturday mornings, and a weekday-only calendar turns a significant Monday effect in the years before that into noise, as Session Hours Are Dated Facts shows.

5.3.2 Longer samples and one-way breaks

The failure to account for regime dependence is related to the broader problem of overfitting. A strategy with enough free parameters can be fitted to any historical data set, including one containing several regimes, but the fitted model captures the particular sequence of regimes in the sample rather than any stable underlying relationship. Regimes are usually pictured as a cycle, with trending and choppy markets taking turns, so that a long enough sample contains every kind and a strategy that survives them all can be trusted. The breaks above are of a different kind: one-way changes inside the sample that never reversed. The case for the longest history rests on the square-root rule, which assumes that every year is a draw from the same distribution. If the distribution changed in 1975, 1997, or 2015, adding the years before the change moves the estimate towards the wrong answer, with growing confidence. It’s the same failure that Section 4.2 describes for the OHLC path error: variance vanishes, but bias persists. Pesaran and Timmermann (2007), who worked out how to choose an estimation window when there are breaks, concluded that it usually pays to include some data from before a break, because it reduces variance by more than it adds bias, but not all of it, and that the right amount has to be found by testing out of sample.

Walk-forward analysis and out-of-sample testing can reduce this problem but can’t remove it, particularly when a one-way break falls inside the test period or the out-of-sample window is short. Worse, these techniques can only validate a strategy against conditions that have already occurred in the historical record, and they offer no protection against structural changes still to come or against events without precedent. The following section looks at the limits of these widely trusted validation methods.

Cryptocurrency markets show regime dependence in an especially stark form. Regulatory changes in crypto can be abrupt and binary: instruments declared securities retroactively, exchanges banned from entire jurisdictions overnight, leverage limits imposed with little warning. A crypto backtest spanning 2019 to 2025 assumes a degree of regulatory continuity that simply did not exist during that period. Even the regulated end of the market changed its calendar inside such a sample: CME crypto futures have traded around the clock since 29 May 2026, with weekend trades carrying Monday’s trade date, so a Monday daily bar now contains Saturday’s trading. More broadly, crypto amplifies almost every failure mode discussed in this section, since regime shifts are faster, leverage is higher, liquidity cliffs are more extreme, and the institutional structure of the market itself is less stable than in any traditional asset class. If the arguments in this paper apply to regulated futures and equities, they apply to crypto with considerably greater force.

5.4 The Parameter Plateau Illusion

A widely taught principle of strategy optimisation is that robust strategies should show “stable regions” or “plateaux” in their parameter space: zones where performance degrades only gradually as parameters are varied, in contrast to narrow spikes where a single parameter value produces good results but neighbouring values do not. The intuition is sound, since a strategy whose performance is insensitive to small parameter changes is more likely to generalise than one whose performance depends on a precise setting. However, the standard method of identifying these plateaux contains a subtle mathematical artefact that is almost never discussed.

Consider a moving average crossover strategy optimised over a lookback range of 10 to 200 bars in steps of 10. On the surface, each step is an equal increment of 10 additional bars of data, but the proportional change in the information available to the indicator varies enormously across the range. Stepping from 10 bars to 20 bars doubles the lookback window, a 100% increase in the data the indicator considers. Stepping from 190 bars to 200 bars similarly adds 10 bars, but this is only a 5.3% increase in the data available to the indicator. The indicator’s output changes far less in response to a 5% perturbation than a 100% perturbation, and this is true regardless of whether the strategy has captured a genuine signal or has merely fitted to noise.

The consequence is that parameter plateaux are mathematically expected at the upper end of any lookback range, even in the complete absence of a real edge. An indicator fitted to noise at a lookback of 190 bars will produce nearly identical output at 200 bars, because the two windows share approximately 95% of their data. The apparent stability is an artefact of the diminishing marginal information in each additional bar, and it says nothing about robustness. Conversely, the lower end of the range, where each step is a large proportional change, will naturally show greater variability, which may be misread as fragility even if the strategy does capture a genuine short-lookback effect.

This artefact has practical consequences for strategy selection. A researcher who scans a broad parameter range and selects the plateau region is systematically biased toward longer lookback values, which may correspond to slower, more smoothed versions of the strategy that appear robust in optimisation but are simply insensitive to parameter perturbation because each step changes so little. The correct approach is to evaluate parameter sensitivity on a proportional basis (for example, testing lookbacks of 10, 15, 22, 33, 50, 75, 112, 168, each approximately 50% larger than the last) so that each step is a comparable change in the information available to the indicator. Very few retail traders or backtesting tutorials use this technique, and the standard linear parameter sweep, with its built-in bias toward apparent stability at longer lookbacks, remains the default.