Research Methodology

Research methodology, in the quant context, is the set of practices that distinguish a rigorous empirical process from a hopeful one. The defining problem is that financial data are small (by machine-learning standards), serially correlated, non-stationary, and subject to subtle leakage. The defining response is the set of practices backtest audit, multiple testing correction, purged and combinatorial cross-validation, walk-forward optimisation, the triple-barrier labelling method, the deflated Sharpe ratio that are designed to prevent the researcher from fooling themselves. The dominant practitioner reference is López de Prado (2018).

The defining feature of the methodology is the recognition that every empirical finding from financial data is subject to multiple sources of error: data errors, model errors, sampling errors, and most importantly, the error of having searched across many candidate hypotheses and reported only the best one. The cumulative effect of these errors is the overfit gap: the in-sample performance of a strategy is systematically higher than the out-of-sample performance, and the overfit gap is larger for strategies found by searching across many candidates. The standard response is to require that the in-sample performance survives multiple testing correction, and that the out-of-sample performance is validated using a procedure that is itself not subject to the same overfit.

The dominant framework for the methodology is the seven-stage research pipeline: data engineering, feature engineering, signal design, backtest, risk decomposition, execution modelling, and production deployment. The methodology is the discipline that governs the gates between stages: a strategy that fails the backtest gate (e.g., does not survive purged k-fold cross-validation) does not proceed to the risk decomposition gate, and a strategy that fails the risk decomposition gate (e.g., is entirely explained by a known factor) does not proceed to the execution modelling gate. The honest practitioner treats the methodology as a non-negotiable discipline, not as a research style preference (López de Prado, 2018).

Backtest audit is the discipline of validating that a backtest reflects what would have happened in production, not what the researcher wanted to have happened. The standard audit is the backtest exception test: at the end of each day, the realised P&L of the strategy is compared against the VaR forecast, and the number of exceptions (days when the realised loss exceeds the VaR) is tested against the expected number. The standard test is the Kupiec test (Kupiec, 1995), which is a likelihood-ratio test of the null hypothesis that the exception rate is equal to the VaR confidence level. The standard follow-up test is the Christoffersen test (Christoffersen, 1998), which tests the null hypothesis of independence of exceptions.

The second standard audit is the P&L attribution. The realised P&L of the strategy is decomposed into: market P&L (the P&L that would have been earned by holding the position passively), alpha P&L (the P&L due to the strategy's decisions, measured against the market P&L), and execution P&L (the P&L due to the execution algorithm, measured against a benchmark like arrival price or VWAP). The decomposition is done at multiple horizons: intraday, daily, weekly, monthly, and annual. The purpose is to identify which component of the strategy is contributing and which is detracting, and to confirm that the realised P&L is consistent with the strategy's design.

The third standard audit is the capacity analysis. The strategy's expected return is estimated as a function of the trade size, and the trade size is increased until the expected return is no longer profitable. The capacity constraint is the maximum trade size at which the strategy is still profitable, and the strategy's capacity is the product of the capacity constraint and the number of trading opportunities. The audit is done by backtesting the strategy at increasing trade sizes and observing the decay in the Sharpe ratio. The standard reference is Almgren & Chriss (2000), which provides the framework for the market impact and capacity analysis (López de Prado, 2018).

Multiple testing correction is the statistical discipline of accounting for the number of hypotheses tested when evaluating the significance of a single finding. The standard example: a researcher tests 100 candidate factors and finds that 5 are significant at the 5% level. Under the null hypothesis that none of the factors is genuinely significant, the expected number of false positives is 100 × 0.05 = 5, so the 5 significant factors are all expected to be false positives. The standard correction is the Bonferroni correction (multiply the p-value by the number of tests), the Holm-Bonferroni correction (a step-down variant), or the Benjamini-Hochberg correction (controls the false discovery rate rather than the family-wise error rate). The standard reference is Harvey, Liu & Zhu (2016), which applies the corrections to the published finance literature.

The most important application of multiple testing correction in quant research is the assessment of the published factor literature. The standard empirical result (Harvey, Liu & Zhu, 2016) is that the post-publication performance of academic anomalies is roughly half the in-sample performance, and that a substantial fraction of the published anomalies are not statistically significant after multiple testing correction. The standard response is to require a higher significance threshold (typically 3-5% after multiple testing correction, rather than 5% before) for any new factor, and to require that the out-of-sample performance is consistent with the in-sample performance.

The second important application is the deflated Sharpe ratio (Bailey & López de Prado, 2014). The deflated Sharpe ratio corrects the reported Sharpe ratio for the number of trials (every strategy that was tried and not reported is a trial), the non-normality of returns, and the correlation between trials. The deflated Sharpe ratio is the standard reporting metric for any new strategy, and the standard tool for the multiple testing correction in the strategy research context. The dominant reference is López de Prado (2018) (Section 16 on backtest overfitting).

Purged k-fold cross-validation is the standard time-series adaptation of the k-fold cross-validation. The standard k-fold cross-validation assumes that the data are independently and identically distributed, so the k folds can be drawn randomly from the data set. For time-series data, the random-fold assumption is wrong: observations are temporally correlated, and a random-fold draw will produce folds that overlap in time. The purged variant removes observations whose labels overlap the test set, and the embargo extends the removal by an additional percentage of the data to account for serial correlation. The dominant reference is López de Prado (2018) (Section 7 on cross-validation).

Combinatorial purged cross-validation (CPCV) extends the purged k-fold to multiple backtest paths. In the CPCV, the data is divided into N groups, and the test set is taken as a single group while the training set is the remaining N-1 groups. This produces N backtest paths (one for each choice of test group). The number of unique paths is C(N, k) where k is the number of groups. The CPCV estimates the probability of backtest overfitting (PBO) as the fraction of paths with negative logit performance. The standard reference is Bailey, Borwein, López de Prado & Zhu (2014).

The standard practitioner use case is the validation of a new strategy. The strategy is run through CPCV, and the PBO is computed. If the PBO is below (say) 5%, the strategy is considered validated. If the PBO is above 5%, the strategy is considered overfit, regardless of the in-sample performance. The dominant reference is López de Prado (2018) (Section 16 on backtest overfitting), with the modern treatment in the production ML literature.

Walk-forward optimisation is the standard multi-period validation method. The strategy is fit on a rolling training window, and evaluated on the subsequent out-of-sample period. The training window is then rolled forward, and the process is repeated. The result is a stitched out-of-sample equity curve that is the honest performance of the strategy over the entire evaluation period. The walk-forward is the standard method for simulating the production behaviour of a strategy over time, and is the most-cited practitioner use case in the modern quant research literature.

The standard walk-forward protocol is: at each re-fit date, the strategy is fit on the previous N days of data, the parameters that produce the best in-sample performance are selected, and the strategy is then evaluated on the next M days of data. The choice of N (the training window) and M (the out-of-sample period) is itself a hyperparameter. The standard practitioner choice is N = 3-5 years of daily data and M = 6-12 months. The ratio of N to M is the ratio of the training data to the test data, and is typically in the range 5:1 to 10:1.

The walk-forward output is a stitched out-of-sample equity curve. The cumulative return, the Sharpe ratio, the maximum drawdown, and the other performance metrics are computed on the stitched curve. The stitched curve is the honest out-of-sample performance, and is the standard input to the strategy's production decision. A strategy that has a strong in-sample performance but a weak stitched out-of-sample performance is overfit, regardless of the magnitude of the in-sample performance. The dominant reference is Pardo (2008), with the modern treatment in López de Prado (2018).

The triple-barrier method is the standard labelling method for financial machine learning. The standard problem is: given a feature vector at time t, what is the label for supervised learning? The fixed-horizon method uses the next-day return as the label, which has several problems: it ignores the duration of the trade, it does not account for the stopping-out of the trade, and it is sensitive to the choice of horizon. The triple-barrel method addresses these problems by labelling each observation by the first of three events touched: the upper take-profit barrier, the lower stop-loss barrier, or the vertical time barrier. The label is the sign of the return at the time the first barrier is touched.

The triple-barrier method is the standard labelling method for the supervised learning of return prediction. The three barriers the take-profit, the stop-loss, and the time are the standard parameters of any trade, and the method labels each observation by the trade outcome rather than by the next-day return. The method is robust to the choice of horizon: the same method works for a 1-day horizon and for a 20-day horizon. The method is also robust to the choice of meta-labels: a meta-label is a separate label for the sign of the position, which can be used to filter the strategy to trades where the model is confident.

The triple-barrier method is the standard reference for the modern quant research pipeline. The dominant reference is López de Prado (2018) (Section 3 on the triple-barrier method), with the modern extension in the production ML literature. The method is the basis for the deflated Sharpe ratio (Bailey & López de Prado, 2014), the combinatorial purged cross-validation (Bailey, Borwein, López de Prado & Zhu, 2014), and the meta-labelling approach (López de Prado, 2018, Section 10).

Reproducibility is the property that an independent team, given the same data and the same code, can reproduce the published result. It is the scientific method's basic contract; it is increasingly demanded in finance, both by journals and by the institutional investors who allocate capital to systematic strategies. The reproducibility standard in modern quant research is high: a published result should be accompanied by the data set, the code, the dependency versions, the random seeds, and the exact command-line invocation that produced the reported numbers.

The practical discipline of reproducibility has three layers. The first is code version control: every result is tied to a specific commit hash. The second is data version control: every result is tied to a specific snapshot of the data, with the snapshot's provenance documented. The third is environment reproducibility: the results should not depend on the specific version of the Python or R interpreter, the specific BLAS library, or the specific random number generator state. The most common tool for the third layer is a container image (Docker) or a pinned environment specification (Nix, conda-lock). The dominant reference is the academic reproducibility literature, with the financial application in López de Prado (2018).

The reproducibility standard is not just academic. A reproducible research process is also a more efficient research process: the same infrastructure that supports external reproducibility also supports internal handover (when a researcher leaves the team) and internal debugging (when a strategy that worked last year stops working this year). A team that cannot reproduce its own results cannot iterate on them. The most successful quant firms in the industry treat reproducibility as a first-class engineering discipline, not as a publication requirement.

The honest limits of the methodology are the same as the honest limits of any empirical discipline. The first limit is data: the methodology is only as good as the data it operates on. The standard data issues in finance (corporate actions, point-in-time, survivorship, look-ahead) require careful engineering, and the engineering is itself a source of model risk. The dominant reference is López de Prado (2018) (Section 2 on financial data structures).

The second limit is non-stationarity. The methodology is designed to detect overfitting on a given historical window, but it cannot detect regime change in the underlying market. A strategy that is validated on a single historical regime will not survive a regime change, regardless of the rigor of the validation. The standard response is continuous monitoring and recalibration, but the recalibration is itself a non-stationary process, and the practitioner literature is explicit that the recalibration is a necessary but not sufficient condition for robust performance.

The third limit is the human. The methodology is a discipline of judgement. The methodology's output is the input to the researcher's decision, not the decision itself. The most common single source of major losses in the quant research function is a known methodology violation that was not addressed, a known model that was not validated, or a known result that was not reproducible. The honest practitioner treats the methodology as a non-negotiable discipline, not as a research style preference, and uses the methodology to make better decisions, not to replace the decision (López de Prado, 2018).

Quant Score v1.0 is a 5–98 bounded composite (baseline 50) that aggregates five technical contributions derived from a single 60-to-120-day daily-bar window via TechnicalEngine. Trend contributes −25 to +25 (STRONG_BEAR to STRONG_BULL from the MA5/MA10/MA20 alignment and MA5 bias discipline), volume −14 to +12 (heavy/shrink regimes), MACD −12 to +12 (golden/death cross and histogram regime), RSI −10 to +10 (overbought/oversold/strong-buy), and volatility −2 to +5 (Bollinger squeeze and ATR14 regime). The sum is clamped to 5–98 and mapped to STRONG_BUY ≥80, BUY ≥68, HOLD ≥52, WAIT ≥40, SELL ≥25, else STRONG_SELL. This is an educational, data-driven read — not financial advice.

Data sources are Finnhub and Alpaca for price/volume (UnifiedMarketDataClient tries Alpaca then Finnhub), with FRED, CFTC, SEC EDGAR and Alpaca fundamentals referenced for the broader TQH data platform (see llms.txt). Each /quant-score/[ticker] page records methodologyVersion, calculatedAt, barCount, dataSources, sector (when resolvable via profile2), component table, signalReasons/riskFactors, and a unique interpretation that blends bias, squeeze, RSI and MACD. Pages with fewer than 20 bars or an empty interpretation are noindex by isQuantScoreIndexable (§4C) until enriched — we do not index thin programmatic shells.

Known limitations: (1) single-window, single-asset, no cross-asset risk adjustment (no beta/IC, no survivorship-adjusted universe); (2) no transaction-cost, capacity, or slippage modelling; (3) indicators are retrospective and silently lag regime change (see non-stationarity above); (4) adapter outages (ALPACA_API_KEY/FINNHUB_API_KEY) degrade coverage and surface as 5+ bar errors rather than stale scores. Quant Score is versioned (v1.0) and will be recalibrated only with an updated methodology note and backtest audit (see /methodology/backtest-audit).