Why is standard k-fold wrong for backtests?
Standard k-fold shuffles observations, which leaks future information into training through serial correlation and overlapping labels. In finance, adjacent rows share information (momentum labels overlap, volatility clusters), so a random split lets the model train on the near-neighbours of its test points. The result is an optimistic bias that survives averaging — more folds do not fix leakage, they redistribute it.
Purging removes training observations whose labels overlap test-set times. Embargo extends the cut by a small buffer (e.g., 1–2% of the sample) to absorb residual serial dependence. Only after purging and embargo does a split deserve the name out-of-sample.
CPCV goes one step further: instead of one train/test split, it builds many combinatorial paths so the selection rule itself — not just one fitted model — is stress-tested across regimes.
How does CPCV work?
Partition history into N equal groups (N=6 is the textbook illustration; production uses 8–16). Form all train/test splits where the test set is k groups (typically k=2) and training is the rest: C(N,k) splits. For each split, rank all S candidate strategies in-sample (training groups) and record the rank of the winner out-of-sample (test groups).
Stack the out-of-sample performances of the selected winners across splits into backtest paths. The logit of each path's relative performance (selected vs median) is computed; PBO is the fraction of paths with negative logit — i.e., how often the in-sample winner underperforms the median out of sample.
Interpretation: PBO ≈ 0 means the selection rule generalizes; PBO ≈ 0.5 means it is noise; values above ~0.2–0.3 are the usual rejection region. Always report N, k, purge/embargo sizes, and the performance metric alongside PBO — the number alone is not auditable.
What is the reproducible code?
The skeleton below implements purged combinatorial splits and PBO on a strategies×time return matrix. Plug in your own candidate returns (rows = configs, columns = periods). It prints PBO and the distribution of selected-winner out-of-sample Sharpes.
python# cpcv.py # Combinatorial Purged CV + PBO skeleton (Bailey et al. 2014). # Run: python cpcv.py import itertools import numpy as np import pandas as pd def purged_split(n_periods: int, test_idx: np.ndarray, embargo: int = 2): """Boolean train mask with purge (test window) + embargo (buffer).""" test_mask = np.zeros(n_periods, dtype=bool) test_mask[test_idx] = True # Embargo: extend exclusion past the test window end. end = int(test_idx.max()) test_mask[end : end + embargo + 1] = True return ~test_mask def sharpe(returns: np.ndarray, annual_factor: float = 252.0) -> float: if len(returns) < 2 or np.std(returns) == 0: return 0.0 return float(np.sqrt(annual_factor) * np.mean(returns) / np.std(returns)) def cpcv_pbo( R: pd.DataFrame, n_groups: int = 6, k_test: int = 2, embargo_pct: float = 0.01 ) -> tuple[float, list[float]]: """R: strategies (rows) x time (cols) returns. Returns (PBO, selected OOS Sharpes).""" n_periods = R.shape[1] groups = np.array_split(np.arange(n_periods), n_groups) embargo = max(1, int(n_periods * embargo_pct)) logits: list[float] = [] selected_oos: list[float] = [] for test_groups in itertools.combinations(range(n_groups), k_test): test_idx = np.concatenate([groups[g] for g in test_groups]) train_mask = purged_split(n_periods, test_idx, embargo) test_mask = np.zeros(n_periods, dtype=bool) test_mask[test_idx] = True is_sharpes = R.values[:, train_mask].mean(axis=1) / ( R.values[:, train_mask].std(axis=1) + 1e-12 ) winner = int(np.nanargmax(is_sharpes)) oos = R.values[winner][test_mask] median_oos = float(np.median(R.values[:, test_mask].mean(axis=1))) w = float(oos.mean()) # Logit of selected vs median (signed relative performance). denom = abs(w) + abs(median_oos) + 1e-12 logits.append((w - median_oos) / denom) selected_oos.append(sharpe(oos)) pbo = float(np.mean([lg < 0 for lg in logits])) return pbo, selected_oos if __name__ == "__main__": rng = np.random.default_rng(7) # Demo: 50 pure-noise strategies x 1260 days -> PBO should be ~0.5. R = pd.DataFrame(rng.normal(0, 0.01, size=(50, 1260))) pbo, oos = cpcv_pbo(R, n_groups=6, k_test=2) print(f"PBO (noise universe): {pbo:.2f} (expect ~0.5)") print(f"Selected-winner mean OOS Sharpe: {np.mean(oos):.2f} (expect ~0)")
How do you choose N, k, purge and embargo?
N trades resolution against cost: C(N,k) splits grow fast (C(16,2)=120 is comfortable; C(16,8) is not). N=6–8 with k=2 suits research; N=12–16 with k=2–3 suits production audits. Purge width equals the maximum label overlap (e.g., a 20-day holding label purges 20 days around each test boundary). Embargo of ~1% of the sample absorbs autocorrelation; verify by checking that train/test residual correlation is near zero.
The metric must match the decision: Sharpe for risk-adjusted selection, Calmar for drawdown-sensitive mandates, turnover-adjusted return when costs dominate. Report the metric choice — PBO on Sharpe and PBO on net return can disagree, and the disagreement itself is informative.
Small samples (T < 500) make every split noisy; prefer fewer, larger groups and treat PBO as directional rather than precise. Document all four choices (N, k, purge, embargo) or the audit is not reproducible.
What are the honest limits?
CPCV audits the selection rule on history; it cannot insure against regime change. A rule with PBO ≈ 0 on 2010–2019 data can still fail in a 2020-style volatility regime. Pair CPCV with walk-forward (see the walk-forward asset) and live paper trading.
PBO is a relative measure (winner vs median), not an absolute performance forecast. A low PBO with a low absolute OOS Sharpe means the rule reliably picks the best of a bad bunch — useful to know, not a deployment signal.
Computational cost scales with S × C(N,k). For large ML grids, audit a representative subset of finalists rather than every discarded experiment, and disclose the subsetting. An honest subset audit beats an infeasible full audit that never runs.