Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Cross-validation for change-point regression: pitfalls and solutions

Florian Pein, Rajen D. Shah

TL;DR

Numerical experiments show that the absolute error approach in particular is competitive with common change-point methods using classical tuning parameter choices when error distributions are well-specified, but can substantially outperform these in misspecified models.

Abstract

Cross-validation is the standard approach for tuning parameter selection in many non-parametric regression problems. However its use is less common in change-point regression, perhaps as its prediction error-based criterion may appear to permit small spurious changes and hence be less well-suited to estimation of the number and location of change-points. We show that in fact the problems of cross-validation with squared error loss are more severe and can lead to systematic under- or over-estimation of the number of change-points, and highly suboptimal estimation of the mean function in simple settings where changes are easily detectable. We propose two simple approaches to remedy these issues, the first involving the use of absolute error rather than squared error loss, and the second involving modifying the holdout sets used. For the latter, we provide conditions that permit consistent estimation of the number of change-points for a general change-point estimation procedure. We show these conditions are satisfied for least squares estimation using new results on its performance when supplied with the incorrect number of change-points. Numerical experiments show that our new approaches are competitive with common change-point methods using classical tuning parameter choices when error distributions are well-specified, but can substantially outperform these in misspecified models. An implementation of our methodology is available in the R package crossvalidationCP on CRAN.

Cross-validation for change-point regression: pitfalls and solutions

TL;DR

Numerical experiments show that the absolute error approach in particular is competitive with common change-point methods using classical tuning parameter choices when error distributions are well-specified, but can substantially outperform these in misspecified models.

Abstract

Cross-validation is the standard approach for tuning parameter selection in many non-parametric regression problems. However its use is less common in change-point regression, perhaps as its prediction error-based criterion may appear to permit small spurious changes and hence be less well-suited to estimation of the number and location of change-points. We show that in fact the problems of cross-validation with squared error loss are more severe and can lead to systematic under- or over-estimation of the number of change-points, and highly suboptimal estimation of the mean function in simple settings where changes are easily detectable. We propose two simple approaches to remedy these issues, the first involving the use of absolute error rather than squared error loss, and the second involving modifying the holdout sets used. For the latter, we provide conditions that permit consistent estimation of the number of change-points for a general change-point estimation procedure. We show these conditions are satisfied for least squares estimation using new results on its performance when supplied with the incorrect number of change-points. Numerical experiments show that our new approaches are competitive with common change-point methods using classical tuning parameter choices when error distributions are well-specified, but can substantially outperform these in misspecified models. An implementation of our methodology is available in the R package crossvalidationCP on CRAN.
Paper Structure (29 sections, 22 theorems, 323 equations, 3 figures, 16 tables)

This paper contains 29 sections, 22 theorems, 323 equations, 3 figures, 16 tables.

Table of Contents

  1. Introduction
  2. Pitfalls of using cross-validation with squared error loss
  3. Setting
  4. Underestimation
  5. Overestimation
  6. New cross-validation criteria
  7. Cross-validation with absolute error loss
  8. Modified cross-validation with squared error loss
  9. General multivariate model
  10. Least Squares Estimation
  11. Data-driven choice of $K_{\max}$
  12. Numerical experiments
  13. Sensitivity to locations of large changes
  14. Small sample size
  15. Blocks and stairs signal
  16. ...and 14 more sections

Key Result

Theorem 1

Let $Y \in \mathbb{R}^n$ be as in Example example:underestmation. Suppose that the following hold: Then $\hat{K}$eq:estimatedKgaussian satisfies $\mathbb{P}(\hat{K} = 2) \to 0$. Moreover, if additionally $K_{\max} = K = 2$,

Figures (3)

  • Figure 1: Schematic of Example \ref{['example:underestmation']}. Expectations are visualized by grey dots and observations by coloured crosses, split into the two folds given by $Y^O$ (red) and $Y^E$ (blue). The predictions $(\overline{Y}^O_{\hat{\tau}^O_{L,l} :\hat{\tau}^O_{L,l + 1}})_{l=0}^L$ from the odd observations, are shown for $L = 2$ (orange dots) and $L = 1$ (brown dots). Distances $e_2$ and $e_1$ between $Y^E_{\tau^O_2}$ and the corresponding predictions are much larger than those corresponding to other observations, and can dominate the cross-validation criterion. The larger size of $e_2$ results in $\operatorname{CV}_{(2)}$ being minimised at $L=1$.
  • Figure 2: Signal with $n = 2048$ observations and $K = 11$ change-points.
  • Figure 3: Blocks signal with $K = 11$ change-points.

Theorems & Definitions (45)

  • Example 1: Underestimation
  • Theorem 1
  • Example 2: Overestimation
  • Theorem 2
  • Theorem 3
  • Theorem 4: Consistency
  • Theorem 5
  • Lemma 1: Lemma 4 on p. 33 in verzelen2020optimal
  • Lemma 2
  • proof
  • ...and 35 more