Detecting Overfitting in Momentum Strategies

Cross-sectional momentum is among the most carefully studied return patterns in equity markets. The economic idea is simple: stocks that outperformed their peers over the recent past tend, on average, to keep outperforming for a short subsequent window. That pattern survives many markets and decades, but the implementation details do not travel for free. Lookback length, holding horizon, and how tightly the book concentrates into winners all change turnover, crash exposure, and net Sharpe—and all of them are usually chosen by searching historical data.

Parameter search is where quiet overfitting enters. If a researcher evaluates a set of candidate rules $\Theta =\{{\theta }_1,\dots ,{\theta }_k\}$ on one historical path and reports only

$${\theta }_{\mathrm{IS}}^{\star }=\arg \underset{\theta \in \Theta }{\max }\widehat{\mathrm{SR}}(\theta ),$$

then the reported Sharpe is contaminated by selection luck. Under a null of no edge, the expected maximum of $k$ estimated Sharpes grows with $k$. Classical remedies—holdout samples, deflated performance metrics, combinatorial cross-validation, and stress probes—exist, yet they are often applied piecemeal rather than as a single filter.

This project constructs that filter for a concrete strategy. We study a long-only winners momentum book on point-in-time S&P 500 constituents. Each month we rank stocks by past J-month return, buy the top N names with equal weights, and hold the sleeve for K months. When $K>1$ we form overlapping books and average their returns, following the classical Jegadeesh–Titman overlapping-portfolio construction. Net returns subtract commission on turnover plus a square-root market-impact term.

Performance is scored with the Deflated Sharpe Ratio (DSR), which adjusts ordinary Sharpe for the number of trials and for return non-normality. Around that objective we stack three diagnostic layers: (1) train–test disparity, sensitivity, and block-bootstrap screens; (2) combinatorial CSCV/PBO plus walk-forward and windowed cross-validation; (3) stylised stress operators that reverse trends, inflate volatility, impose crash haircuts, and flip regimes.

The exhibits below accompany the prose. The first chart shows how many grid points survive each gate; later sections unpack every stage. The final selection ${\theta }^{\star }$ maximises a weighted combination of DSR and absolute maximum drawdown among survivors. That selection is a research diagnostic—evidence that a parameter region looked relatively robust historically—not a claim that the same sleeve will profit live.

Universe. Let ${\mathcal{U}}_t$ denote the investable set of S&P 500 constituents eligible at month-end $t$ under a point-in-time membership mask (so additions and deletions enter when they would have been known). Liquid names satisfying a minimum-price filter contribute monthly simple returns

$$r_{i,t}=\frac{P_{i,t}}{P_{i,t-1}}-1,$$

with extreme observations winsorised cross-sectionally. The working sample is monthly from the early 2000s through the end of the available history.

Momentum signal. For lookback $J$, formation-month momentum of name $i$ is the cumulative compound return over the prior $J$ months,

$$M_{i,t}^{(J)}=\prod _{j=1}^J(1+r_{i,t-j})-1.$$

The long-only book holds the equal-weighted top $N$ names,

$$w_{i,t}=\frac{1\{i\in \mathrm{TopN}(M_{\cdot ,t}^{(J)})\}}{N}.$$

Those weights are held for $K$ months. With monthly reformation, month-$t$ portfolio return averages the $K$ active overlapping sleeves formed at $t-K+1,\dots ,t$:

$$R_t^{\mathrm{gross}}=\frac{1}{K}\sum _{h=0}^{K-1}\sum _i{w}_{i,t-h}{r}_{i,t}.$$

Frictions. Let ${\tau }_t$ denote one-way turnover between successive formations. Commission at rate $c$ (here $c=10$ bps) and square-root impact yield a cost

$$C_t=c{\tau }_t+\alpha \sum _i{\sigma }_{i,t}\sqrt{q}|\Delta w_{i,t}|,$$

where ${\sigma }_{i,t}$ is trailing annualised volatility, $q$ is an assumed participation rate in average daily volume, and $\alpha$ is a market-impact coefficient. Net returns are $R_t^{\mathrm{net}}=R_t^{\mathrm{gross}}-C_t$ (with costs amortised across the holding horizon).

Sharpe and Deflated Sharpe. With monthly net returns and risk-free rate $r_f$,

$$\widehat{\mathrm{SR}}=\sqrt{12}\frac{\hat{\mu }-r_f}{\hat{\sigma }}.$$

Selecting the maximum of $k$ such Sharpes overstates edge. Bailey and López de Prado (2014) define a Deflated Sharpe that tests whether observed performance exceeds the expected maximum under a no-edge null, after correcting for skewness ${\hat{\gamma }}_3$ and kurtosis ${\hat{\gamma }}_4$. Writing ${\mathrm{SR}}^{\star }$ for that null benchmark,

$$\widehat{\mathrm{DSR}}=\Phi \left(\frac{(\widehat{\mathrm{SR}}-{\mathrm{SR}}^{\star })\sqrt{T-1}}{\sqrt{1-{\hat{\gamma }}_3\widehat{\mathrm{SR}}+\frac{{\hat{\gamma }}_4-1}{4}{\widehat{\mathrm{SR}}}^2}}\right),$$

where $\Phi$ is the standard normal cdf. We treat $\mathrm{DSR}(\theta )$ as the primary ranking statistic throughout the pipeline.

Search grid. The discrete space is $J\in \{3,6,9\}$, $K\in \{2,3,6\}$, $N\in \{1,2\}$—eighteen combinations. The chart below ranks every grid point by full-sample DSR before robustness filters.

Before filtering individual triples we ask how the length of the training sample shapes optimism. Split the monthly history at fraction $\pi$ so that the first $\lfloor \pi T\rfloor$ observations train and the remainder test. Define the mean-return gap

$$D(\theta ,\pi )={\bar{R}}_{\mathrm{train}}(\theta ,\pi )-{\bar{R}}_{\mathrm{test}}(\theta ,\pi ).$$

We evaluate $\pi \in \{0.6,0.7,0.8,0.9\}$. Across all $(\theta ,\pi )$ pairs compute sample mean ${\mu }_D$ and standard deviation ${\sigma }_D$, and define the overfit flag set

$${\Omega }_{\mathrm{overfit}}=\{(\theta ,\pi ):D(\theta ,\pi )>{\mu }_D+{\sigma }_D\}.$$

A Pearson correlation $\rho (\pi ,D(\theta ,\pi ))$ with associated $p$-value summarises whether longer training fractions systematically widen optimism.

Large positive $D$ means the strategy looked stronger where history was fitted and weaker where it was held out—the textbook fingerprint of overfit calibration. Later stages therefore lock the default split at **$\pi =0.8$**.

Read the exhibits in order: mean gaps versus $\pi$; the full $(\theta ,\pi )$ scatter with threshold flags; then per-parameter bars at a chosen $\pi$.

Even with a fixed strategy family, the number of alternatives searched changes the meaning of “best.” Write $S=|J|\times |K|\times |N|$. Under a null of zero edge, the expected maximum of $S$ estimated Sharpes rises with $S$—the data-snooping effect emphasised by Sullivan, Timmermann, and White (1999) and Harvey and Liu (2015).

We sample random subsets of size $s\le S$, compute the average train–test gap $\bar{D}(s)$ over many draws, and fit

$$\bar{D}(s)=as+b+\epsilon .$$

A positive slope $a$ would indicate that expanding the searchable space mechanically increases apparent optimism; a flat or negative slope in a particular sample still motivates using DSR’s explicit $k$-trial penalty.

The chart below shows mean gaps versus subset size $s$ with a fitted trend.

Stage 1 asks whether training and testing performance are consistent enough to survive a formal test. For each $\theta$ collect paired mean returns across several split ratios and apply a **paired $t$-test** of the differences

$$d_{\pi }={\bar{R}}_{\mathrm{train}}(\pi )-{\bar{R}}_{\mathrm{test}}(\pi ),H_0:\mathbb{E}[d_{\pi }]=0.$$

Rejecting $H_0$ at the 5% level ($p\le 0.05$) flags systematic disparity; those triples are removed. Parameter sets with $p>0.05$ continue. The gate does not prove that a strategy is good—it only discards combinations whose historical optimism is statistically hard to ignore.

The sorted $p$-value chart below colours passes in green and failures in red. Hover a bar for the mean disparity and $t$-statistic.

A second Stage 1 concern is brittleness. If a tiny move in $J$, $K$, or $N$ collapses DSR, the reported optimum sits on a knife-edge. With discrete neighbours we approximate elasticities by finite differences. For a performance metric $P$ (here DSR),

$${\hat{\epsilon }}_J=\frac{P(J+\Delta J,K,N)-P(J,K,N)}{P(J,K,N)}\cdot \frac{J}{\Delta J},$$

and likewise for ${\hat{\epsilon }}_K$ and ${\hat{\epsilon }}_N$. Axis thresholds are

$${\tau }_{\ell }=\mu (|{\hat{\epsilon }}_{\ell }|)+\sigma (|{\hat{\epsilon }}_{\ell }|),\ell \in \{J,K,N\}.$$

Any $\theta$ with $|{\hat{\epsilon }}_{\ell }|>{\tau }_{\ell }$ on any axis is marked sensitive and fails Stage 1. The stacked bars below sum absolute elasticities so brittle corners stand out visually; hover for baseline DSR.

Ordinary bootstrap treats months as i.i.d., which returns violate through volatility clustering. We use overlapping block resampling of length $\ell \approx 24$ months. Let $P(\theta )$ be the DSR of each grid member and $\bar{P}$ their mean. Residuals ${ϵ}_{\theta }=P(\theta )-\bar{P}$ are resampled in blocks to form a bootstrap distribution of the mean residual and a $(1-\alpha )$ confidence interval $[L,U]$.

Parameter sets with ${ϵ}_{\theta }\notin [L,U]$ are treated as outliers relative to the rest of the search and removed from the Stage 1 intersection

$${\Theta }_{\mathrm{robust}}={\Theta }_{\mathrm{disparity}}\cap {\Theta }_{\mathrm{nonsensitive}}\cap {\Theta }_{\mathrm{bootstrap}}.$$

The residual bars below highlight those outliers in red.

Stage 2 begins with Combinatorially Symmetric Cross-Validation (CSCV) (Bailey, Borwein, López de Prado, and Zhu, 2015). Form the performance matrix $M\in {\mathbb{R}}^{T\times S}$ whose columns are monthly returns of each grid trial. Partition the $T$ rows into an even number of contiguous blocks and, for every combinatorial assignment of half the blocks to the in-sample set $\mathcal{I}$ and the complement to the out-of-sample set $\mathcal{O}$, select

$${\theta }_{\mathrm{IS}}^{\star }(\mathcal{I})=\arg \underset{\theta }{\max }{\bar{R}}_{\theta }(\mathcal{I}).$$

Record the relative out-of-sample rank of that same trial among all columns on $\mathcal{O}$, and map it to a logit

$$\lambda =\log \left(\frac{p}{1-p}\right),$$

where $p$ is the OOS relative rank (higher is better). Large positive logits mean the IS winner remained competitive OOS; negative logits mean the champion faded—the signature of backtest overfitting.

Averaging the failure indicator across the $C$ combinatorial folds yields the probability of backtest overfitting,

$$\mathrm{PBO}=\frac{1}{C}\sum _{c=1}^{C}1\{{\lambda }_c<0\}.$$

We also label individual trials by their average OOS relative rank when selected. Trials below the median appear in red on the $J$–$K$ scatter (bubble size scales with $N$). The logit histogram summarises how often IS/OOS consistency held in this sample.

CSCV is powerful but abstract. We therefore add classical time-series protocols. Write ${\mathrm{DSR}}_{\mathrm{OOS}}(\theta ;\mathcal{W})$ for Deflated Sharpe of trial $\theta$ on a held-out window $\mathcal{W}$.

Walk-forward. Roll an in-sample window, choose ${\theta }_{\mathrm{IS}}^{\star }$ on that window, and score the next block. Expanding-window grows the training history while testing successive slices. Rolling-window recomputes DSR inside a fixed-length sliding window and averages those scores. A light nested scheme separates hyperparameter choice from final evaluation folds.

For each parameter set define the protocol average

$${\overline{\mathrm{DSR}}}_{\mathrm{CV}}(\theta )=\frac{1}{M}\sum _{m=1}^M{\mathrm{DSR}}_{\mathrm{OOS}}(\theta ;{\mathcal{W}}_m),$$

and retain only the top quartile of that distribution. CSCV survivors that miss the quartile are dropped before stress testing.

Use the toggle on the chart to compare walk-forward, expanding, and rolling DSR side by side, or collapse to mean CV DSR with top-quartile bars highlighted.

Parameters that look stable in calm holds can still fail when the return process is distorted. Stage 3 applies four stylised stress operators ${\mathcal{T}}_s$ to each surviving monthly path $(R_t)$ and recomputes Deflated Sharpe under each transform.

Return reversal: ${\mathcal{T}}_{\mathrm{rev}}(R_t)=-R_t$—the natural adversary of trend persistence. Volatility shocks: ${\mathcal{T}}_{\mathrm{vol}}(R_t)=R_t\cdot {\epsilon }_t$ with $\mathbb{E}[{\epsilon }_t]=1$ and log-normal shocks, raising turbulence without a directional bias. Market crashes: apply a haircut factor $\kappa \in (0,1)$ to a random subset of months. Regime shifts: invert returns over contiguous windows to mimic abrupt leadership changes.

For each $\theta$ report ${\mathrm{DSR}}_s(\theta )=\mathrm{DSR}({\mathcal{T}}_s(R(\theta )))$ and the average

$${\overline{\mathrm{DSR}}}_{\mathrm{stress}}(\theta )=\frac{1}{4}\sum _s{\mathrm{DSR}}_s(\theta ).$$

Combinations with average stressed DSR above the cross-grid median pass. The bar chart and table are sortable by any scenario so you can see which sleeves survive reversal versus which only survive calm volatility.

After robustness, CSCV, top-quartile cross-validation, and stress filtering, a candidate set ${\Theta }_{\mathrm{final}}$ remains. We choose a single ${\theta }^{\star }$ with a transparent objective that rewards risk-adjusted level and penalises path damage:

$${\theta }^{\star }=\arg \underset{\theta \in {\Theta }_{\mathrm{final}}}{\max }\{w_1\mathrm{DSR}(\theta )-w_2|\mathrm{MaxDD}(\theta )|\},$$

with default weights $w_1=1$ and $w_2=0.5$. Maximum drawdown is

$$\mathrm{MaxDD}=\underset{t}{\min }\left(\frac{W_t}{{\max }_{s\le t}{W}_s}-1\right),$$

where $W_t$ is cumulative net wealth. If a later stage returns an empty intersection, the funnel falls back to the previous non-empty survivor set so the page always reports a readable diagnostic.

The retention funnel shows how many of the eighteen grid points remain after each gate. Objective bars and the finalist table list surviving $\theta$ with DSR, Sharpe, MaxDD, and the objective score—click a row to switch the wealth path. Treat ${\theta }^{\star }$ as the region with the least historical evidence of overfitting under this pipeline, not as a production mandate.

For parallel vocabulary (White’s Reality Check, Hansen SPA, purged CV, Monte Carlo gates) see the site article on statistical analysis of trading strategies.

Several caveats bound what these diagnostics can say. The point-in-time membership mask is constructed from public index-change histories and remains an approximation to institutional reconstitution calendars. Corporate-action handling in public adjusted prices can differ from research-grade total-return panels. The eighteen-cell grid is deliberately small; expanding it would raise both compute and the DSR trial penalty $k$. Stress operators are parametric caricatures of crises, not historical replay of 2008 or 2020 path-by-path.

Most importantly, the framework assumes the strategy form is already well specified. It filters parameters of a fixed rule. It cannot repair a misspecified signal, missing risk factors, or capacity constraints that only appear at larger notionals. Passing Stages 1–3 is necessary hygiene for parameter choice, not sufficient evidence of an allocatable edge.

Bailey, D. H., & López de Prado, M. (2014). The Deflated Sharpe Ratio: Correcting for selection bias, backtest overfitting and non-normality. *Journal of Portfolio Management*, 40(5), 94–107.

Bailey, D. H., Borwein, J. M., López de Prado, M., & Zhu, Q. J. (2015). The probability of backtest overfitting. *Journal of Computational Finance*, 20(4), 39–69.

Harvey, C. R., & Liu, Y. (2015). Backtesting. *Journal of Portfolio Management*, 42(1), 13–28.

Jegadeesh, N., & Titman, S. (1993). Returns to buying winners and selling losers: Implications for stock market efficiency. *Journal of Finance*, 48(1), 65–91.

Sullivan, R., Timmermann, A., & White, H. (1999). Data-snooping, technical trading rule performance, and the bootstrap. *Journal of Finance*, 54(5), 1647–1691.

Almgren, R., Thum, C., Hauptmann, E., & Li, H. (2005). Direct estimation of equity market impact. *Risk*, 18(7), 58–62.