Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Quadratic Link between Out-of-Sample $R^2$ and Directional Accuracy

Cheng Zhang

TL;DR

The paper tackles the persistent metric disconnect between out-of-sample R^2 and directional accuracy in financial forecasting. It derives an analytical link under a random walk baseline and a sign-magnitude decomposition, showing that the expected out-of-sample R^2 follows a quadratic function of DA: E[R^2_{OOS}] = kappa (2p-1)^2, where p is the directional accuracy and kappa is a shape parameter. The authors validate the link via simulations using historical price data, finding negative R^2_{OOS} values at modest DA and estimating kappa_hat values around 0.55 for the S&P 500 and 0.48 for the DJIA, consistent with heavy-tailed returns. The work provides a theoretical bridge between magnitude-based and direction-based forecast evaluation and offers a practical benchmark for assessing directional timing ability in financial markets.

Abstract

This study provides a novel perspective on the metric disconnect phenomenon in financial time series forecasting through an analytical link that reconciles the out-of-sample $R^2$ ($R^2_{OOS}$) and directional accuracy (DA). In particular, using the random walk model as a baseline and assuming that sign correctness is independent of realized magnitude, we show that these two metrics exhibit a quadratic relationship for MSE-optimal point forecasts. For point forecasts with modest DA, the theoretical value of $R^2_{OOS}$ is intrinsically negligible. Thus, a negative empirical $R^2_{OOS}$ is expected if the model is suboptimal or affected by finite sample noise.

A Quadratic Link between Out-of-Sample $R^2$ and Directional Accuracy

TL;DR

The paper tackles the persistent metric disconnect between out-of-sample R^2 and directional accuracy in financial forecasting. It derives an analytical link under a random walk baseline and a sign-magnitude decomposition, showing that the expected out-of-sample R^2 follows a quadratic function of DA: E[R^2_{OOS}] = kappa (2p-1)^2, where p is the directional accuracy and kappa is a shape parameter. The authors validate the link via simulations using historical price data, finding negative R^2_{OOS} values at modest DA and estimating kappa_hat values around 0.55 for the S&P 500 and 0.48 for the DJIA, consistent with heavy-tailed returns. The work provides a theoretical bridge between magnitude-based and direction-based forecast evaluation and offers a practical benchmark for assessing directional timing ability in financial markets.

Abstract

This study provides a novel perspective on the metric disconnect phenomenon in financial time series forecasting through an analytical link that reconciles the out-of-sample () and directional accuracy (DA). In particular, using the random walk model as a baseline and assuming that sign correctness is independent of realized magnitude, we show that these two metrics exhibit a quadratic relationship for MSE-optimal point forecasts. For point forecasts with modest DA, the theoretical value of is intrinsically negligible. Thus, a negative empirical is expected if the model is suboptimal or affected by finite sample noise.
Paper Structure (8 sections, 1 theorem, 17 equations, 2 figures)

This paper contains 8 sections, 1 theorem, 17 equations, 2 figures.

Table of Contents

  1. Introduction
  2. The Analytical Link
  3. Point Forecast Decomposition
  4. The Out-of-Sample $R^2$ Expression
  5. Numerical Illustration
  6. Experimental Setup
  7. Experimental Results
  8. Conclusion

Key Result

Proposition 1

Under Assumptions ass:independence and ass:magnitude, the expected out-of-sample $R^2$ of MSE-optimal point forecasts follows a quadratic function of the directional probability $p$: where $\kappa$ is a distributional shape parameter defined as:

Figures (2)

  • Figure 1: Out-of-sample $R^2$ versus directional accuracy: S&P 500.
  • Figure 2: Out-of-sample $R^2$ versus directional accuracy: DJIA.

Theorems & Definitions (1)

  • Proposition 1