A Comparative Evaluation of Quantification Methods

TL;DR

The paper tackles the problem of estimating class prevalences under distribution shift by conducting a large-scale empirical comparison of 24 quantification methods across 40 datasets, covering binary and multiclass settings. It categorizes methods into adjusted count, distribution matching, and classifier-based approaches, and evaluates them under varied training/test shifts and sample sizes, including a LeQua challenge case study. Key findings show no single best method across all scenarios; binary quantification favors threshold-based and distribution-matching methods, while multiclass quantification benefits from distribution-matching approaches like GPAC, ED, FM, EM, readme, and especially HDx, with multiclass posing a substantially harder challenge. The results have practical implications for practitioners selecting quantification methods and for researchers pursuing robust multiclass quantification, while also suggesting that tuning base classifiers offers limited gains in many settings.

Abstract

Quantification represents the problem of estimating the distribution of class labels on unseen data. It also represents a growing research field in supervised machine learning, for which a large variety of different algorithms has been proposed in recent years. However, a comprehensive empirical comparison of quantification methods that supports algorithm selection is not available yet. In this work, we close this research gap by conducting a thorough empirical performance comparison of 24 different quantification methods on overall more than 40 data sets, considering binary as well as multiclass quantification settings. We observe that no single algorithm generally outperforms all competitors, but identify a group of methods including the threshold selection-based Median Sweep and TSMax methods, the DyS framework including the HDy method, Forman's mixture model, and Friedman's method that performs best in the binary setting. For the multiclass setting, we observe that a different, broad group of algorithms yields good performance, including the HDx method, the Generalized Probabilistic Adjusted Count, the readme method, the energy distance minimization method, the EM algorithm for quantification, and Friedman's method. We also find that tuning the underlying classifiers has in most cases only a limited impact on the quantification performance. More generally, we find that the performance on multiclass quantification is inferior to the results obtained in the binary setting. Our results can guide practitioners who intend to apply quantification algorithms and help researchers to identify opportunities for future research.

Figures (18)

• Figure 1: Visual representation of the main results for binary quantification. The top row shows results for the absolute error (AE), the bottom row for normalized Kullback-Leibler divergence (NKLD) scores. On the left, letter-value plots for the distribution of error score across all scenarios per algorithm are shown. Colors indicate the category of the algorithm, with count adaptation-based algorithms shown in blue, distribution matching methods in orange, and adaptations of traditional classification algorithms in green. Plots are scaled logarithmically above the dotted vertical threshold, and linearly below. On the right, we plot the distributions of rankings with a Nemenyi post-hoc test at 5% significance. For each algorithm, we depict the average performance rank over all algorithms. Horizontal bars indicate which average rankings do not differ to a degree that is statistically significant. The critical difference (CD) was $5.6973$. Overall, the HDy, MS, FMM, and DyS methods appear to work best in general.
• Figure 2: Impact of distribution shift. We show the distribution of error scores split by the amount of shift in the evaluation scenario. The left column shows results according to the absolute error, the right one according to NKLD scores. Colors indicate the category of the algorithm. Plots are scaled logarithmically above the dotted vertical threshold, and linearly below. GPAC appears to perform best under minor shifts, FMM under major shifts.
• Figure 3: Performance with small amounts of training data. Plots are scaled logarithmically above the dotted vertical threshold, and linearly below. Colors indicate the category of the algorithm. We observe similar trends compared to the general setting, with MS, HDy, and FMM being among the best-performing algorithms.
• Figure 4: Visual representation of the main results for multiclass quantification. The top row shows results for the absolute error (AE), the bottom row for normalized Kullback-Leibler divergence (NKLD) scores. On the left, letter-value plots for the distribution of error score across all scenarios per algorithm are shown, colors indicate the category of the algorithm. Plots are scaled logarithmically above the dotted vertical threshold, and linearly below. On the right, we plot the distributions of rankings with a Nemenyi post-hoc test at 5% significance. Horizontal bars indicate which average rankings do not differ to a degree that is statistically significant. The critical difference (CD) was $7.0045$. Overall, performance scores are much worse than in the binary setting. Best performances are generally achieved by distribution matching methods that naturally extend to the multiclass setting, with the HDx method standing out.
• Figure 5: Impact of distribution shift in the multiclass setting. We show the distribution of error scores split by the amount of shift in the evaluation scenario. The left column shows results according to the absolute error, the right one according to NKLD scores. Plots are scaled logarithmically above the dotted vertical threshold, and linearly below. Colors indicate the category of the algorithm. Performances generally deteriorate under major shift. Best performances under major shift are achieved by algorithms that also do work well in general. The GPAC and FM methods appear most robust toward major shifts.