Learning Curves for Decision Making in Supervised Machine Learning: A Survey

TL;DR

This survey addresses learning curves in supervised ML, framing their use for data acquisition, early stopping, and model selection within a unified three-axis framework that separates decision contexts, query types, and data resources. It surveys a wide spectrum of methods, from model-free decisions to explicit curve models (point, range, and distribution estimates), including parametric forms like the inverse power law and Bayesian/extrapolative approaches such as Freeze-thaw BO and FABOLAS. Key concepts such as anchor points, limit and saturation performance, and utility-based stopping are formalized, and a taxonomy is provided to categorize methods by decision task, the questions they answer, and data resources used. The review identifies gaps, notably in data acquisition and early stopping, and calls for richer experimental databases, tighter AutoML integration, and broader, uncertainty-aware methods. Collectively, the work offers a comprehensive foundation for designing and evaluating learning-curve-based decision mechanisms in practical ML systems, from hyperparameter search to resource allocation.

Abstract

Learning curves are a concept from social sciences that has been adopted in the context of machine learning to assess the performance of a learning algorithm with respect to a certain resource, e.g., the number of training examples or the number of training iterations. Learning curves have important applications in several machine learning contexts, most notably in data acquisition, early stopping of model training, and model selection. For instance, learning curves can be used to model the performance of the combination of an algorithm and its hyperparameter configuration, providing insights into their potential suitability at an early stage and often expediting the algorithm selection process. Various learning curve models have been proposed to use learning curves for decision making. Some of these models answer the binary decision question of whether a given algorithm at a certain budget will outperform a certain reference performance, whereas more complex models predict the entire learning curve of an algorithm. We contribute a framework that categorises learning curve approaches using three criteria: the decision-making situation they address, the intrinsic learning curve question they answer and the type of resources they use. We survey papers from the literature and classify them into this framework.

Figures (14)

• Figure 1: The three types of decision-making situations in which learning curves are typically used. The $x$-axis of each figure represents the budget in the applicable unit, and the $y$-axis represents the performance.
• Figure 2: Left: (Standard) $\mathit{\text{sample-wise curve}}$ (green) for a single learner on a particular data source together with learning curves under sample optimisation (pink) and learning curves on streams (blue). Right: $\mathit{\text{iteration-wise curves}}$ of a single learner on a particular data source for two different dataset sizes $n_1 < n_2$.
• Figure 3: Empirical learning curves for the waveform dataset. Left: $\mathit{\text{sample-wise curves}}$ at different training set sizes up to 80% of the data. The remaining 20% are used to compute the error. Right: $\mathit{\text{iteration-wise curves}}$ with one entry for each forest size (RF), epoch (NN), or optimization iteration (SVM) when a fixed set of 80% of the available data is used in each iteration for training and the rest to compute the error.
• Figure 4: A utility curve with the corresponding learning curve.
• Figure 5: Concepts related to a learning curve. Red: Limit performance. Green: Saturation point $\mathit{\mathit{b}\xspace_{sat}}$ (vertical line) and saturation performance $\mathit{p_{sat}}$ (horizontal line). Orange: Pre-exponential point (vertical line) and pre-exponential performance (horizontal line). The curve plateaus at a performance of 0.2; the plateau is not visualized here to emphasise the difference between the pre-exponential point and saturation point.