Key Points

• Backtesting evaluates the feasibility of a trading strategy or pricing model by reviewing historical data.
• The rationale is that any strategy that performed well in the past is likely to perform well in the future and vice versa.
• When testing ideas on historical data, it is very beneficial to retain a period of historical data for testing purposes. If the test is successful, further validation of its potential feasibility on other time periods or out-of-sample data can increase confidence.

Understanding backtesting

Backtesting allows traders to simulate trading strategies using historical data to generate results and analyze risk and profitability before actually investing any money.

A well-conducted backtest, with positive results, will assure traders that the strategy is fundamentally sound and potentially profitable when implemented in reality. On the other hand, backtesting that yields suboptimal results will prompt traders to modify or abandon the strategy.

Important: Particularly complex trading strategies, such as those implemented by automated trading systems, often rely heavily on backtesting to prove their value because these strategies are too complex to evaluate in other ways.

As long as the trading idea can be quantified, it can be backtested. Some traders and investors may look for qualified programmers to develop ideas into a testable form. Typically, this involves programmers coding ideas into a proprietary language supported by the trading platform.

Programmers can integrate user-defined input variables, allowing traders to "tune" the system. For example, in a simple moving average (SMA) crossover system, traders can enter (or change) the lengths of the two moving averages used in the system. Traders can then backtest to determine which moving average length performs best in historical data.

Ideal backtesting scenario

An ideal backtest selects sample data for a relevant time period, with a duration that reflects various market conditions. This way, you can better judge whether the backtest results are a fluke or reasonable trading performance.

Historical data sets must contain a realistic representative sample of stocks, including those of companies that eventually went bankrupt, were sold, or were liquidated. Conversely, only including historical stock data that still exists will produce artificially high returns in backtests.

Backtesting should take into account all trading costs, no matter how trivial, as these costs can accumulate during the backtesting period, significantly affecting the appearance of a strategy's profitability. Traders should ensure that their backtesting software takes these costs into account.

Beyond-sample testing and forward-looking performance testing provide further confirmation regarding the effectiveness of the system and can demonstrate the true performance of the system before real money is involved. Strong correlations between backtest, out-of-sample, and forward-looking performance test results are critical to determining the viability of a trading system.

Backtesting and forward-looking performance testing

Forward performance testing, also known as paper trading, gives traders another set of out-of-sample data to evaluate the system. Forward-looking performance testing simulates actual trading and involves following the logic of the system in live markets. This is also known as paper trading because all trades are executed on paper only; that is, trades in and out and any profits and losses in the system are recorded, but there is no real trade execution.

An important aspect of prospective performance testing is to strictly follow the logic of the system; otherwise, it is difficult, if not impossible, to accurately evaluate this step of the process. Traders should honestly record the entry and exit of any trade and avoid behaviors such as selective recording of transactions or not recording a transaction on paper, rationalizing that "I will never enter into this transaction." If a transaction occurs following system logic, it should be recorded and evaluated.

Backtesting and Scenario Analysis

While backtesting uses actual historical data for fit or success testing, scenario analysis uses hypothetical data to simulate various possible outcomes. For example, scenario analysis simulates specific changes in portfolio securities or key factors, such as changes in interest rates.

Scenario analysis is often used to estimate the change in a portfolio's value if an adverse event occurs and can be used to examine theoretical worst-case scenarios.

Some pitfalls of backtesting

In order for backtesting to provide meaningful results, traders must develop strategies and test them with good faith and try to avoid bias. This means that the strategy should be developed independently of the data used for backtesting.

This may seem simple, but it's actually quite difficult. Traders often build strategies based on historical data. They must strictly test on a different data set than the one on which the model was trained. Otherwise, backtesting will produce inflated results that have no real meaning.

Likewise, traders must avoid data mining, i.e. testing numerous hypothetical strategies against the same data set, which can equally produce successes that fail in live markets, as there are many ineffective strategies that may outperform the market in a given period due to chance.

One way to compensate for the tendency to data mine or selectively record is to use a strategy that was successful during the relevant or in-sample time period and backtest it using data from a different or out-of-sample time period. If in-sample and out-of-sample backtest results are similar, their validity is more likely to be verified.

About TheQuantHackers

TheQuantHackers is a cutting-edge community composed of a group of people from the world's top universities and quantitative finance practitioners, dedicated to exploring the infinite possibilities in the fields of artificial intelligence (AI) and quantification (Quant). Our team members come from world-renowned universities such as the University of Cambridge, Oxford University, Harvard University, ETH Zurich, Peking University, and the University of Science and Technology of China. Our external consultants come from first-class companies such as Microsoft, HSBC, Citadel, Man Group, Citi, Jump Trading, and top domestic private equity companies.