What is the purpose of this case study?
The case study is a worked example of the modern backtest audit methodology. It is not a new research finding; it is a reproducible demonstration of the validation framework developed by López de Prado and collaborators, applied to a well-known strategy (12-month price momentum) on a public dataset (Shiller's S&P). The purpose is to show, in concrete terms, how a research process distinguishes a genuine signal from a backtest artefact.
The strategy (12-month price momentum) was chosen because it is one of the most-cited anomalies in the academic literature and is well-documented in its in-sample and out-of-sample performance. The dataset (Shiller's S&P monthly) was chosen because it is public, free, and spans 1871-present, which is long enough to expose both the strategy's true performance and the overfit gap. The methodology (CPCV + deflated Sharpe) is the current state of the art in quantitative finance (López de Prado, 2018).
The case study produces four results, each of which corresponds to a standard diagnostic in the methodology. The four results are: the in-sample Sharpe ratio (the headline number from a naive backtest), the out-of-sample Sharpe ratio (the same strategy evaluated using purged k-fold cross-validation), the overfit gap (the difference between the two), and the deflated Sharpe ratio (the in-sample Sharpe corrected for the number of trials). The interpretation of each result is given below; the working Python code is given in the next section.
What is the dataset?
The dataset is the Shiller S&P Composite, maintained by Robert Shiller of Yale University and updated monthly. It contains monthly observations of the S&P Composite and its predecessors (the Standard Statistics Composite, the S&P 90, and the S&P 500) from 1871 to the present. The dataset also includes the consumer price index (CPI), the 10-year Treasury yield, and the earnings series, all of which are required for the standard real-return calculations.
The data is free for academic and personal use, with attribution. The canonical source is the Shiller data page at Yale (the URL is in the asset metadata). The data is also distributed through the FRED database (FRED code: Shiller S&P Composite) and through the multpl.com aggregator. The case study uses the raw Shiller file directly, which is the most defensible reference.
The standard caveats about the dataset apply. The S&P Composite is a price index, not a total return index, so the case study analyses price returns. The pre-1928 data is reconstructed by Shiller from historical stock price records, and is therefore less reliable than the post-1928 data. The case study uses the full sample but notes that the post-1928 sample is the standard for academic analysis of US equity returns.
What is the strategy?
The strategy is the 12-month price-momentum strategy of Jegadeesh & Titman (1993). The rule: rank the universe of stocks by their 12-month return excluding the most recent month, buy the top decile, sell the bottom decile, and rebalance monthly. The exclusion of the most recent month is the standard correction for the well-known short-term reversal effect (Jegadeesh, 1990).
In the original Jegadeesh-Titman implementation, the universe is NYSE/AMEX stocks and the strategy is implemented as a cross-sectional long-short portfolio. In the case study, the strategy is implemented as a long-only proxy on the S&P Composite, for two reasons. First, the S&P Composite is a single asset, so the cross-sectional decile sort is not directly applicable. Second, the case study's purpose is to demonstrate the audit methodology, not to reproduce the Jegadeesh-Titman result exactly. The long-only proxy captures the essential property of momentum (the tendency of recent winners to continue to outperform) without the complexity of a cross-sectional sort on a stock universe.
The expected performance of the strategy on the S&P Composite, applied as a long-only proxy, is in the range of 6-10% per annum above the market over the 1926-1989 period, with a Sharpe ratio of approximately 0.5-0.7, consistent with the in-sample results reported by Jegadeesh & Titman (1993) for the cross-sectional version. The out-of-sample performance, over the 1990-2020 period, is lower, consistent with the post-publication decay documented by McLean & Pontiff (2016).
What is the audit methodology?
The audit methodology is the CPCV (Combinatorial Purged Cross-Validation) framework of Bailey, Borwein, López de Prado & Zhu (2014), as implemented in López de Prado (2018) (Chapter 16). The procedure: divide the historical data into N groups; for each choice of test group (a single group), use the remaining N-1 groups as the training set; compute the strategy's Sharpe ratio on the test group; record the result. The collection of N results is the CPCV backtest path distribution. The probability of backtest overfitting (PBO) is the fraction of paths with negative logit performance.
The complementary diagnostic is the deflated Sharpe ratio (Bailey & López de Prado, 2014). The deflated Sharpe ratio corrects the reported (in-sample) Sharpe ratio for the number of trials, the non-normality of returns, and the correlation between trials. The deflated p-value is the probability that the observed Sharpe ratio would arise under the null hypothesis that the true Sharpe is zero, given the number of trials.
The combination of the two diagnostics gives a complete picture. The CPCV out-of-sample Sharpe is the honest performance estimate. The deflated Sharpe p-value is the statistical significance of the in-sample Sharpe, after accounting for the number of trials. A strategy with a high in-sample Sharpe, a high out-of-sample Sharpe, and a low deflated p-value is a genuine signal. A strategy with a high in-sample Sharpe, a low out-of-sample Sharpe, and a high deflated p-value is a backtest artefact.
What is the reproducible code?
The code below is a self-contained Python script that loads the Shiller data, computes the 12-month momentum signal, runs the in-sample backtest, runs the CPCV out-of-sample backtest, and computes the deflated Sharpe ratio. The script requires only pandas, numpy, scipy, and matplotlib. The expected output is a printed summary of the in-sample and out-of-sample Sharpe ratios and a plot of the cumulative return.
python# backtest_audit.py # Reproducible backtest audit of 12-month momentum on Shiller S&P. # López de Prado methodology (CPCV + deflated Sharpe). # Run: python backtest_audit.py import numpy as np import pandas as pd from scipy import stats # --- 1. Load Shiller S&P data --- # Download from: http://www.econ.yale.edu/~shiller/data/ie_data.xls # Columns include: Date, P (real price), D (real dividend), CPI # (We use price returns as the long-only proxy for momentum.) df = pd.read_excel("ie_data.xls", sheet_name="Data") df = df[["Date", "P"]].dropna() df["return"] = df["P"].pct_change() df = df.dropna().reset_index(drop=True) # --- 2. Compute the 12-month momentum signal (skip the most recent month) --- df["signal"] = df["P"].pct_change(periods=12).shift(1) # Long-only proxy: invest when signal > 0, cash otherwise. df["strategy"] = np.where(df["signal"] > 0, df["return"], 0) # --- 3. In-sample performance (the naive backtest) --- strategy_returns = df["strategy"].values market_returns = df["return"].values in_sample_sharpe = ( np.sqrt(12) * strategy_returns.mean() / strategy_returns.std() ) print(f"In-sample Sharpe (naive backtest): {in_sample_sharpe:.2f}") # --- 4. CPCV out-of-sample performance (the audit) --- # Divide the data into 6 groups; 6 backtest paths. n_groups = 6 group_size = len(strategy_returns) // n_groups oos_sharpes = [] for test_group in range(n_groups): test_start = test_group * group_size test_end = (test_group + 1) * group_size # Purge: skip the month just before and after the test window. test_returns = strategy_returns[test_start + 1 : test_end - 1] if len(test_returns) > 0: sharpe = np.sqrt(12) * test_returns.mean() / test_returns.std() oos_sharpes.append(sharpe) mean_oos_sharpe = np.mean(oos_sharpes) print(f"CPCV out-of-sample Sharpe (mean of {len(oos_sharpes)} paths): {mean_oos_sharpe:.2f}") print(f"Overfit gap: {in_sample_sharpe - mean_oos_sharpe:.2f}") # --- 5. Deflated Sharpe ratio (correcting for multiple testing) --- # Assume the strategy was found after N trials. n_trials = 100 # Skellam approximation: E[max SR] under the null. e_max_sr = ( (1 - np.euler_gamma) * stats.norm.ppf(1 - 1 / n_trials) + np.euler_gamma * stats.norm.ppf(1 - 1 / (n_trials * np.e)) ) se_max_sr = np.sqrt( 1 + e_max_sr**2 - (1 - np.euler_gamma) * stats.norm.ppf(1 - 1 / n_trials)**2 ) / np.sqrt(len(strategy_returns)) # Deflated Sharpe p-value: probability that max SR > observed SR. deflated_sharpe_z = (in_sample_sharpe - e_max_sr) / se_max_sr deflated_pvalue = 1 - stats.norm.cdf(deflated_sharpe_z) print(f"Deflated Sharpe (n_trials={n_trials}): p-value = {deflated_pvalue:.4f}") print(f" -> {'Statistically significant' if deflated_pvalue < 0.05 else 'Not statistically significant'}"
What are the expected findings?
The expected findings, in the case study, mirror the published empirical results. The in-sample Sharpe ratio (the naive backtest) is in the range of 0.5-0.8, depending on the sample period. The out-of-sample Sharpe ratio (CPCV) is in the range of 0.3-0.5, depending on the sample period and the number of groups. The overfit gap (in-sample minus out-of-sample) is in the range of 0.2-0.4, which is moderate but is materially larger for strategies found by searching across many candidates (Harvey, Liu & Zhu, 2016).
The deflated Sharpe ratio depends on the assumed number of trials. If the strategy was found by searching across a small number of candidates (say, N=10), the deflated p-value may still be significant, and the strategy is consistent with a genuine signal. If the strategy was found by searching across a large number of candidates (say, N=1000), the deflated p-value is large even when the in-sample Sharpe is reported as significant, and the strategy is consistent with a backtest artefact. The point of the deflated Sharpe ratio is precisely this: it forces the researcher to be honest about how many candidates were tested.
The interpretation of the case study is the same as the interpretation of any backtest audit. A strategy that survives the full methodology a high out-of-sample Sharpe, a low deflated p-value, a small overfit gap is a strong candidate for production. A strategy that fails any one of these a low out-of-sample Sharpe, a high deflated p-value, a large overfit gap is a candidate for rejection, regardless of the in-sample performance. The honest practitioner treats the methodology's output as the input to the researcher's decision, not as the decision itself.
How does this case study illustrate the methodology?
The case study illustrates the methodology in three ways. First, it shows the implementation of the CPCV and the deflated Sharpe ratio in concrete Python code, which can be run on the Shiller data and verified against the expected output. Second, it shows the interpretation of the four standard diagnostics (in-sample Sharpe, out-of-sample Sharpe, overfit gap, deflated p-value) in the context of a real strategy. Third, it shows the standard cross-check between the in-sample headline number and the out-of-sample robust number, which is the central discipline of the modern quant research process.
The case study is not a recommendation. The 12-month price-momentum strategy is a well-known anomaly, and the in-sample and out-of-sample performance are well-documented in the literature. The case study's purpose is to demonstrate the audit methodology, not to advocate for the strategy. The same methodology applies to any strategy, and the same diagnostics are applicable. A researcher's first task on any new strategy is to run the audit, not to deploy the strategy.
The case study is the first in a planned series. The next case study will be a long-short equity factor strategy (e.g., value or momentum) on the Fama-French research portfolio data, with the same methodology. The third will be a machine-learning strategy (e.g., gradient-boosted trees on cross-sectional features), with the same methodology and the additional complications of feature selection and hyperparameter tuning. The series is designed to build a reproducible knowledge base of methodology applications, each anchored in real data and real code.
What are the honest limits of this case study?
The honest limits of the case study are the same as the honest limits of the underlying methodology. The case study is a worked example, not a new research finding. The numbers cited in the case study are drawn from the published literature (Jegadeesh & Titman, 1993; McLean & Pontiff, 2016; Harvey, Liu & Zhu, 2016) and are not the result of a fresh backtest. A researcher who runs the case study code on the Shiller data will obtain a result that depends on the data, the code, and the specific sample period, and the result may differ from the numbers cited.
The case study uses a long-only proxy for the 12-month momentum strategy, applied to the S&P Composite as a single asset. The published literature on momentum is overwhelmingly about cross-sectional long-short strategies on stock universes, not about long-only proxies on broad indices. The cross-sectional long-short version is the more powerful implementation and the more relevant test of the methodology. The case study's use of the long-only proxy is for illustration only, and should not be interpreted as a representation of the standard implementation.
The case study is a single illustration, not a comprehensive audit. The full audit of a new strategy would require additional diagnostics: P&L attribution (market, alpha, execution), capacity analysis (the trade size at which the strategy is no longer profitable), transaction cost analysis (the impact of realistic costs on the Sharpe ratio), and live paper-trading validation (the strategy's performance in production with no real money for at least several months). The case study demonstrates the four standard diagnostics of the methodology, but the full audit of a production strategy requires the additional layers.