Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Are Ensembles Getting Better all the Time?

Pierre-Alexandre Mattei, Damien Garreau

TL;DR

It is shown that ensembles are getting better all the time if, and only if, the considered loss function is convex, and that ensembles of good models keep getting better, and ensembles of bad models keep getting worse.

Abstract

Ensemble methods combine the predictions of several base models. We study whether or not including more models always improves their average performance. This question depends on the kind of ensemble considered, as well as the predictive metric chosen. We focus on situations where all members of the ensemble are a priori expected to perform equally well, which is the case of several popular methods such as random forests or deep ensembles. In this setting, we show that ensembles are getting better all the time if, and only if, the considered loss function is convex. More precisely, in that case, the loss of the ensemble is a decreasing function of the number of models. When the loss function is nonconvex, we show a series of results that can be summarised as: ensembles of good models keep getting better, and ensembles of bad models keep getting worse. To this end, we prove a new result on the monotonicity of tail probabilities that may be of independent interest. We illustrate our results on a medical problem (diagnosing melanomas using neural nets) and a "wisdom of crowds" experiment (guessing the ratings of upcoming movies).

Are Ensembles Getting Better all the Time?

TL;DR

It is shown that ensembles are getting better all the time if, and only if, the considered loss function is convex, and that ensembles of good models keep getting better, and ensembles of bad models keep getting worse.

Abstract

Ensemble methods combine the predictions of several base models. We study whether or not including more models always improves their average performance. This question depends on the kind of ensemble considered, as well as the predictive metric chosen. We focus on situations where all members of the ensemble are a priori expected to perform equally well, which is the case of several popular methods such as random forests or deep ensembles. In this setting, we show that ensembles are getting better all the time if, and only if, the considered loss function is convex. More precisely, in that case, the loss of the ensemble is a decreasing function of the number of models. When the loss function is nonconvex, we show a series of results that can be summarised as: ensembles of good models keep getting better, and ensembles of bad models keep getting worse. To this end, we prove a new result on the monotonicity of tail probabilities that may be of independent interest. We illustrate our results on a medical problem (diagnosing melanomas using neural nets) and a "wisdom of crowds" experiment (guessing the ratings of upcoming movies).
Paper Structure (50 sections, 13 theorems, 68 equations, 10 figures)

This paper contains 50 sections, 13 theorems, 68 equations, 10 figures.

Table of Contents

  1. Introduction
  2. A motivating example: neural nets for medical diagnosis
  3. Summary and organisation of the paper
  4. Related works: monotonicity and the wisdom of crowds
  5. Monotonicity beyond ensembles
  6. Background, notations, and examples
  7. A general predictive framework
  8. Why is $L$ only a function of $\hat{y}$?
  9. Several predictions, and how to average them
  10. Where do our predictions come from?
  11. How to aggregate predictions?
  12. What sort of results are we going to prove?
  13. Ensembles get better for convex losses
  14. Warm-up: An ensemble predicts better than a single model
  15. A first monotonicity lemma
  16. ...and 35 more sections

Key Result

Lemma 1

Assume that $L$ is a convex function. Then, for any $\hat{y}_1,\ldots,\hat{y}_K \in C$, we have

Figures (10)

  • Figure 1: Evolution of the accuracy of a dropout ensemble on the DermaMNIST dataset as the number of models grows (mean and standard deviation over $500$ repetitions). (Left) The accuracy averaged over the whole test set has no clear monotonic pattern. (Middle) Accuracy averaged over images for which the asymptotic prediction (i.e. the prediction of an ensemble of infinite size) $\overline{y}_\infty$ is correct. The accuracy is getting better when the ensemble size grows. (Right) Accuracy averaged over images for which $\overline{y}_\infty$ is incorrect. The accuracy is going down. The behaviours of these middle and right panels are related to the non-convexity of the classfication error, and are explained by our theory. Notice that the three $y$-axes have different scales.
  • Figure 2: Evolution of the cross-entropy (or negative log-likelihood) of a dropout ensemble on the dermatology data set as the number of models grows (mean and standard deviation over 500 repetitions). (Left) Cross-entropy averaged over the whole test set. (Middle) Cross-entropy averaged over images for which the asymptotic prediction $\overline{y}_\infty$ is correct. (Right) Cross-entropy averaged over images for which the asymptotic prediction $\overline{y}_\infty$ in incorrect. As predicted by our theory, all curves are decreasing because the cross-entropy loss is convex. Notice that the three $y$-axes have different scales.
  • Figure 3: Nonconvex loss functions for binary classification (left panel) and regression (right panel). In both case the true label/response is $y=0$. For smooth losses, purple dots denote inflexion points: the frontier between the concave part of the loss (that corresponds to "right" predictions) and the concave one (that corresponds to "wrong" ones).
  • Figure 4: Evolution of the sigmoid loss of a dropout ensemble on the dermatology data set as the number of models grows (mean and standard deviation over $500$ repetitions). (Left) The sigmoid loss averaged over the whole test appears to be decreasing. (Middle) The loss averaged over images for which the asymptotic prediction $\overline{y}_\infty$ is correct is getting better. (Right) The loss averaged over images for which the asymptotic prediction $\overline{y}_\infty$ in incorrect is getting worse. The behaviours of these middle and right panels are explained by our theory. Notice that the three $y$-axes have different scales.
  • Figure 5: Evolution of the accuracy of a dropout ensemble on the 3-class dermatology data set as the number of models grows (mean and standard deviation over $500$ repetitions). (Left) The accuracy averaged over the whole test set has a non-monotonic behaviour. (Centre-left) The accuracy averaged over images for which the prediction is asymptotically correct is improving. (Centre-Right) The accuracy averaged over images for which the prediction is asymptotically incorrect (yet the correct label is not the one with the smallest predicted probability) is degrading. (Right) The accuracy averaged over images for which the prediction is asymptotically completely incorrect (i.e. the correct label has the smallest predicted probability) is degrading. Theorem \ref{['th:class-error']} provides a theoretical explanation of the behaviour of centre-left and right panels.
  • ...and 5 more figures

Theorems & Definitions (19)

  • Lemma 1: monotonicity lemma
  • Theorem 2: monotonicity for convex losses, marshall1965inequality
  • Definition 3: strong convexity
  • Theorem 4: strict monotonicity for strongly convex losses
  • Theorem 5: Monotonicity of smooth nonconvex losses
  • Lemma 6: accuracy as margin tail probability
  • Theorem 7: monotonicity of tail probabilities, univariate
  • Theorem 8: Monotonicity of tails, multivariate
  • Remark 9: Monotonicity of tails under Gaussian assumption
  • Theorem 10: Monotonicity of the classification error
  • ...and 9 more