Portfolio Optimization

Portfolio optimization is the construction of an asset allocation that best meets an investor's objectives typically the maximisation of expected risk-adjusted return subject to a set of constraints. The dominant framework, due to Markowitz (1952), is mean-variance optimisation: the investor specifies an expected return and a risk tolerance (or, equivalently, a risk aversion), and the optimiser produces the portfolio that maximises expected utility under the assumption of quadratic utility (or, equivalently, that minimises variance for a given expected return). The output of the optimisation is a vector of portfolio weights.

The modern framework extends mean-variance in three important ways. The first is factor-based portfolio construction: rather than optimising over individual assets, the optimiser constructs a portfolio as a set of factor exposures, and the assets are chosen to deliver the target factor exposures at minimum tracking error to the benchmark. The second is the integration of risk models: the covariance matrix is replaced by a factor risk model (Barra, Axioma, MSCI RiskMetrics), which decomposes risk into systematic factor risk and idiosyncratic asset-specific risk. The third is the integration of transaction costs and turnover constraints: the optimiser produces a portfolio that is optimal net of the costs of moving from the current portfolio to the target portfolio.

Portfolio optimization is the operational core of institutional asset management. Every major asset manager runs a portfolio optimiser continuously, and the optimiser's output is the basis for the portfolio manager's rebalancing decisions. The optimiser is also the basis for the construction of smart-beta and factor portfolios, for the construction of risk-parity portfolios, and for the construction of liability-driven investment (LDI) portfolios. The discipline is mature in its mathematical foundations and in the standard practice, but the practitioner literature emphasises that the output of the optimiser is highly sensitive to the inputs particularly the expected returns and the covariance matrix and that small changes in the inputs can produce large changes in the output (Michaud, 1989).

Mean-variance optimisation was introduced by Harry Markowitz in his 1952 paper “Portfolio Selection” (Markowitz, 1952), for which he shared the 1990 Nobel Memorial Prize in Economic Sciences. The contribution was to formulate the asset allocation problem as a quadratic optimisation problem with a closed-form solution, given the vector of expected returns and the covariance matrix of asset returns. The efficient frontier the set of portfolios that maximise expected return for a given level of variance is the central object of the theory.

The 1960s saw the development of the capital asset pricing model (Sharpe, 1964; Lintner, 1965; Mossin, 1966), which provided a theoretical foundation for the expected returns used in mean-variance optimisation. The CAPM says that the expected excess return of an asset is proportional to its beta to the market portfolio, with the constant of proportionality being the market risk premium. The CAPM reduces the problem of estimating N expected returns to the problem of estimating one number (the market risk premium) and N betas.

The 1990s and 2000s saw the development of multi-factor models, the most prominent being the Fama-French three-factor model (Fama & French, 1993) and the Carhart four-factor model (Carhart, 1997), which add size and value factors (and momentum in the four-factor model) to the CAPM. The Barra risk model, originally developed at BARRA Inc. in the 1970s and 1980s and now distributed by MSCI, became the industry standard for the risk-model input to portfolio optimisation. The Black-Litterman model (Black & Litterman, 1992) addressed the optimiser's extreme sensitivity to expected return inputs by combining the market-implied equilibrium returns (from the CAPM) with the investor's subjective views in a Bayesian framework (Michaud, 1989).

The standard portfolio optimisation problem is: maximise expected portfolio return minus λ times portfolio variance, subject to a set of linear constraints (e.g., full investment, no short positions, sector exposure limits, factor exposure targets). Mathematically: max wᵀμ − (λ/2) wᵀΣw, subject to A w = b and l ≤ w ≤ u, where w is the vector of portfolio weights, μ is the vector of expected returns, Σ is the covariance matrix, λ is the risk aversion, and the constraints are the standard institutional constraints.

In practice, the optimiser is solved as a quadratic programme. The inputs are: the vector of expected returns (from the investor's views, the CAPM, the Fama-French model, or a combination), the covariance matrix (from the Barra risk model or equivalent), the constraints (from the investment mandate), and the cost model (from the transaction cost model). The output is the vector of portfolio weights. The standard commercial optimiser is the Barra Optimizer (now part of MSCI), with alternatives including Axioma (now part of Qontigo, a Deutsche Börse subsidiary), the Northfield Optimizer, and various in-house optimisers at the major asset managers.

The most-cited limitation of mean-variance optimisation is the sensitivity of the output to the inputs. Small changes in the expected return vector can produce large changes in the portfolio weights, particularly for assets with high variance and high correlation. Michaud (1989) was the most prominent early critic; the Black-Litterman model (Black & Litterman, 1992) is the standard response, with the equilibrium-implied returns as the prior and the investor's views as the likelihood. The modern practice is to combine Black-Litterman with explicit transaction cost and turnover constraints, and to use the optimiser as a decision-support tool rather than as a black-box allocator.

The most important extension is risk parity, which constructs the portfolio so that each asset (or each asset class) contributes equally to the total portfolio risk. The simplest risk-parity portfolio is the inverse-volatility portfolio: each weight is proportional to 1/σᵢ. The more sophisticated version accounts for the correlation between assets and uses the marginal contribution to risk to set the weights (Qian, 2005). Risk parity has become the dominant framework for institutional multi-asset allocation, particularly for pension funds and endowments that want diversification without the leverage of a traditional 60/40 portfolio.

The second extension is the Black-Litterman model (Black & Litterman, 1992), which is a Bayesian framework for combining the market-implied equilibrium returns with the investor's subjective views. The prior is the CAPM equilibrium returns (or, in a more sophisticated version, the Fama-French equilibrium returns). The likelihood is the investor's views, expressed as expected returns on one or more assets or asset groups, with associated confidence. The posterior is a set of expected returns that combines the prior and the views in a Bayesian way, and that is much more stable to small changes in the inputs than the pure-mean-variance optimiser. Black-Litterman has become the standard framework for institutional asset allocation (Michaud, 1989).

The third extension is the integration of transaction costs. The basic mean-variance optimiser is a single-period optimiser: it does not consider the cost of moving from the current portfolio to the target portfolio. The multi-period extension (introduced by Davis & Norman, 1990, and popularised by the Almgren-Chriss framework for execution) explicitly models the cost of trading as a function of the trade size, the volatility, and the bid-ask spread, and the optimiser balances the expected improvement in the portfolio from rebalancing against the cost of the rebalancing. The modern practice in institutional asset management is to run a multi-period optimiser with explicit turnover and cost constraints (Ceria, Hubenschmid & Maddahi, 2019).

In institutional asset management, the portfolio optimiser is the operational core of the rebalancing decision. The cycle is: at each rebalance date (typically monthly or quarterly), the optimiser is run with the latest expected return inputs (from the firm's house view), the latest risk model (from Barra or equivalent), the latest constraint set (from the investment mandate), and the current portfolio (from the position master). The output is a target portfolio. The portfolio manager reviews the target portfolio, may override it for tactical reasons, and the resulting target is implemented via the trading desk, typically using an execution algorithm to minimise market impact.

In hedge funds and proprietary trading firms, the optimiser is the same in principle but different in practice. The asset universe is typically much larger (thousands of instruments across multiple asset classes), the constraints are more flexible (allowing short positions and leverage), and the cost of rebalancing is much higher (because the positions are larger and the markets are less liquid). The output of the optimiser is typically combined with a risk model and a capacity model: the optimiser produces a target portfolio, the risk model produces the marginal contribution to risk of each position, and the capacity model produces the maximum position size that can be traded without breaching the capacity constraint. The result is the tradable target portfolio (Qian, 2005).

In private wealth and retail, the optimiser is less common because the cost of rebalancing is proportionally larger and the institutional infrastructure (the Barra risk model, the Almgren-Chriss cost model, the Black-Litterman view-generation) is not available. The retail equivalent is the target-date fund, the risk-parity ETF, or the smart-beta ETF, which encodes the result of an institutional-grade optimisation in a single tradeable product. The trend in the institutional industry is toward more frequent rebalancing (monthly or even weekly) and toward more sophisticated multi-period optimisation, with the most advanced firms using machine-learning-based expected return models and risk models within the same optimisation framework (Ceria, Hubenschmid & Maddahi, 2019).

The most common failure mode of portfolio optimisation is the sensitivity of the output to the inputs. The mean-variance optimiser, with the standard institutional risk model, can produce portfolio weights that are highly concentrated in a small number of assets, particularly for assets with high expected returns and high correlations. The result is that the optimal portfolio looks like a long-short equity portfolio with a few large positions, not a diversified multi-asset portfolio. The standard responses are: the Black-Litterman model (which stabilises the output by anchoring to the market-implied returns), explicit position-size constraints, and the shrinkage of the covariance matrix (Ledoit & Wolf, 2004).

The second common failure mode is the underestimation of transaction costs. A multi-period optimisation that does not include realistic transaction costs will produce a target portfolio that is optimal in theory but unachievable in practice, because the cost of moving from the current portfolio to the target exceeds the expected improvement in risk-adjusted return. The standard responses are: explicit cost models (Almgren-Chriss or the Barra cost model), explicit turnover constraints (e.g., no more than 10% of the portfolio can be turned over in a single rebalance), and explicit participation-rate constraints (e.g., no trade can exceed 10% of the daily volume).

The third common failure mode is the failure to account for model risk. The optimiser is only as good as the inputs, and the inputs are model outputs. The most common source of input error is the expected return vector: even a small error in the expected return of a high-variance asset can produce a large change in the optimal portfolio weight. The standard responses are: shrinkage of the expected return vector towards the market-implied returns (Black-Litterman), explicit confidence intervals on the expected returns, and the use of robust optimisation (which explicitly optimises the worst-case portfolio across a set of possible input scenarios). The honest practitioner treats the optimiser's output as a starting point for the portfolio manager's judgement, not as the final answer (Michaud, 1989).

Factor models are the dominant framework for portfolio construction in modern institutional asset management. The standard factor model is the Barra risk model, which decomposes the total return of each asset into a sum of factor returns plus an idiosyncratic return. The factors include market (the equity market return), style (value, growth, size, momentum, volatility, quality), industry (the return of the asset's industry), and country (for international portfolios). The Barra risk model is the industry standard for the risk model input to portfolio optimisation (Fama & French, 1993; Carhart, 1997).

The factor-based portfolio is constructed in two stages. The first stage is the factor portfolio: for each factor, construct a portfolio that is long the assets with high exposure to the factor and short the assets with low exposure, neutralised with respect to the other factors. The second stage is the alpha overlay: the investor's house view (the source of expected alpha) is layered on top of the factor portfolio. The result is a portfolio that is exposed to the systematic factors (the source of the risk premia) and to the investor's house view (the source of the alpha). The dominant style of factor portfolio is the long-short equity portfolio that is neutral to the market, size, value, momentum, and other style factors, and that is long the names with the highest expected alpha.

The factor-based portfolio is the dominant alternative to the pure mean-variance portfolio. The reason is robustness: the factor portfolio is much less sensitive to the expected return vector than the mean-variance portfolio, because the factor exposure is much more stable than the alpha. The honest practitioner uses the factor-based portfolio as the baseline and the alpha overlay as the source of edge, rather than the other way around. The Fama-French and Carhart factor models are the academic standard for the factor set; the Barra and Axioma models are the industry standard for the asset-level factor exposures (Fama & French, 1993).

The honest limits of portfolio optimisation are the same as the honest limits of any model-based discipline. The optimiser is a model: it produces a number from a model, with explicit assumptions, and the assumptions are sometimes wrong. The principal assumption is the expected return vector, and the principal failure mode is the sensitivity of the output to the expected return vector. The standard responses Black-Litterman, shrinkage, robust optimisation reduce the sensitivity but do not eliminate it.

The second limit is the cost of rebalancing. The mean-variance optimiser is, in its single-period form, an instantaneous allocator; it does not consider the cost of moving from the current portfolio to the target. The multi-period extension addresses this, but the multi-period optimiser is computationally expensive and is sensitive to the cost-model parameters. The standard practice is to use the multi-period optimiser as a decision-support tool, not as a black-box allocator, and to override the optimiser's output when the cost of rebalancing is judged to exceed the expected improvement.

The third limit is the human. Portfolio optimisation is a discipline of judgement. The optimiser's output is the input to the portfolio manager's decision, not the decision itself. The most common single source of major losses in the portfolio management function is a known risk that was not managed, a known constraint that was violated, or a known model that was overridden. The honest practitioner treats the optimiser as a tool, not as an oracle, and uses the optimiser to make better decisions, not to replace the decision (Michaud, 1989).