LCDB 1.1: A Database Illustrating Learning Curves Are More Ill-Behaved Than Previously Thought

TL;DR

LCDB 1.1 investigates sample-wise learning curves at high resolution across modern tabular learners to challenge the assumption that curves are typically monotone and convex. By introducing data-leakage variants and multiple feature-scaling options, it reveals that roughly 15% of curves exhibit significant ill-behavior, with certain learners (e.g., Sigmoid SVM, TabNet) contributing disproportionately. Ill-behavior degrades parametric curve fitting and complicates multi-fidelity model selection, and feature scaling rarely resolves these issues. The dataset and analyses position LCDB 1.1 as a challenging benchmark to drive robust learning-curve modeling and reliable downstream decision-making in model selection and data-efficiency studies.

Abstract

Sample-wise learning curves plot performance versus training set size. They are useful for studying scaling laws and speeding up hyperparameter tuning and model selection. Learning curves are often assumed to be well-behaved: monotone (i.e. improving with more data) and convex. By constructing the Learning Curves Database 1.1 (LCDB 1.1), a large-scale database with high-resolution learning curves including more modern learners (CatBoost, TabNet, RealMLP and TabPFN), we show that learning curves are less often well-behaved than previously thought. Using statistically rigorous methods, we observe significant ill-behavior in approximately 15% of the learning curves, almost twice as much as in previous estimates. We also identify which learners are to blame and show that specific learners are more ill-behaved than others. Additionally, we demonstrate that different feature scalings rarely resolve ill-behavior. We evaluate the impact of ill-behavior on downstream tasks, such as learning curve fitting and model selection, and find it poses significant challenges, underscoring the relevance and potential of LCDB 1.1 as a challenging benchmark for future research.

Figures (22)

• Figure 1: Motivation for new LCDB 1.1 features. (a) Feature scaling can mitigate an ill-behaved learning curve. (b) Low-resolution curves may omit certain phenomena or render them less apparent.
• Figure 2: Monotonicity Violation
• Figure 3: Convexity Violation
• Figure 4: Estimated probability ($\%$) of different ill-behaviors, (a) Monotonicity Violation, (b) Convexity Violation, (c) Dipping, (d) Peaking, for learners with different feature scalings. For all results see Appendix \ref{['appendix: detailed LCDB statistics']}. Observe that feature scaling for most learners does not lead to significant changes. Ridge and MLP improve significantly, while NC becomes more ill-behaved.
• Figure 5: Ill-behaved learning curves pose challenges for curve fitting. (a) The distribution of fitting MSE when applying a parametric model to monotone vs. non-monotone (left) and convex vs. non-convex (right) learning curves. The dashed lines represent mean of the log MSE. Ill-behavior leads to to significantly larger MSE. (b) Larger violation sizes (x-axis) coincide with larger MSE (y-axis).

Theorems & Definitions (5)

• Definition 1: Monotonicity Violation Error
• Definition 2: Convexity Violation Error
• Definition 3: Peaking Phenomenon
• Definition 4: Dipping Phenomenon
• proof