Optimal Hold-Out Size in Cross-Validation

TL;DR

This work addresses how to choose the cross-validation hold-out size $K$ by explicitly balancing predictive accuracy against evaluation uncertainty driven by irreducible noise $\sigma^{2}$. It develops finite-sample results, exact variance bounds under symmetric errors and general upper bounds otherwise, and embeds them in a mean–variance utility to derive a principled, context-specific rule for selecting $K$. The framework produces a robustness map (Pareto frontier) showing how the optimal hold-out size shifts with model class, data, and assumptions about $\sigma^{2}$, rather than advocating a single convention. Empirical demonstrations across linear regression, random forests, and high-dimensional genomics reveal that conventional $K$ can mislead inference and that the optimal hold-out size is highly data- and model-dependent, motivating a data-driven, assumption-aware approach to CV design.

Abstract

Cross-validation (CV) is routinely used across the sciences to select models and tune parameters, and the resulting choices are often interpreted as substantive scientific conclusions (e.g., which variables, mechanisms, or risk factors are ``supported by the data''). A key part of the CV procedure -- the hold-out size, or equivalently the fold count $K$ -- is typically set by convention (e.g., 80/20, $K=5$) rather than by a principled criterion. Central to the issue is the tradeoff between training and testing: increasing the training sample size improves model accuracy, while sacrificing certainty around the accuracy itself. We formalize the tradeoff by targeting predictive performance and explicitly penalizing evaluation uncertainty, which cannot be identified from the data without additional assumptions. We derive finite-sample expressions of this evaluation uncertainty under symmetric errors and general upper bounds under broader error conditions, yielding a transparent utility-based rule for selecting the hold-out size as a function of an irreducible-noise parameter. Empirical analyses with linear regression and random forests across multiple domains, and a high-dimensional genomics application, show that (i) the choice of $K$ is dependent on the data and model. (ii) the optimal $K$ varies based on the assumption on the irreducible error, and (iii) the implied inferential conclusions can change materially as the irreducible error, and thus $K$, varies. The resulting framework replaces a one-size-fits-all convention with a context-specific, assumption-explicit choice of $K$, enabling more reliable model comparisons and downstream scientific inference.

Figures (9)

• Figure 1: Abalone data: Estimated loss curve for the linear model and random forest. The y-axis is the estimated mean squared error under different hold-out data sizes ($m$: x-axis). $\sigma^2$ is the irreducible error, which is set to $\sigma^2=1$ for illustration.
• Figure 2: Abalone data: Estimated variance curve for the linear model and random forest. The y-axis is the calculated bound of the variance of the loss under different hold-out data sizes ($m$: x-axis). $\sigma^2$ is the irreducible error, which is set to $\sigma^2=1$ for illustration.
• Figure 3: Abalone data: Negative utility of the linear model (left) and random forest (right), for $\sigma^2=\{0.01,0.1,1\}$. Dashed lines are the minimal point under each $\sigma^2$.
• Figure 4: Abalone data: Pareto frontier for the linear model and random forest. Dashed lines denote sample size that equals $K=4,5,10,20$, with $m=N/K$.
• Figure 5: Empirical CV loss and fitted curves (power-law, PCHIP, spline). Circles denote points used for fitting; triangles denote withheld points.

Theorems & Definitions (5)

• Theorem 1
• Theorem 2
• proof
• Theorem 3
• proof