Generalized test utilities for long-tail performance in extreme multi-label classification

TL;DR

This work tackles long-tail tail-label evaluation in extreme multi-label classification by proposing budgeted-$k$ predictions optimized under an expected test utility (ETU) framework. It derives Bayes-optimal rules, introduces a semi-empirical ETU surrogate with provable regret guarantees, and develops scalable block-coordinate ascent and efficient inference methods, including sparsity-aware and PLT-based approaches. The framework accommodates both instance-wise and macro-averaged, non-decomposable utilities and provides rigorous regret bounds under Lipschitz assumptions, showing robustness to misspecification. The results demonstrate improved tail-label performance without retraining and offer practical, scalable strategies compatible with state-of-the-art XMLC pipelines, with clear pathways for combining tail-focused objectives with head-label performance.

Abstract

Extreme multi-label classification (XMLC) is the task of selecting a small subset of relevant labels from a very large set of possible labels. As such, it is characterized by long-tail labels, i.e., most labels have very few positive instances. With standard performance measures such as precision@k, a classifier can ignore tail labels and still report good performance. However, it is often argued that correct predictions in the tail are more "interesting" or "rewarding," but the community has not yet settled on a metric capturing this intuitive concept. The existing propensity-scored metrics fall short on this goal by confounding the problems of long-tail and missing labels. In this paper, we analyze generalized metrics budgeted "at k" as an alternative solution. To tackle the challenging problem of optimizing these metrics, we formulate it in the expected test utility (ETU) framework, which aims to optimize the expected performance on a fixed test set. We derive optimal prediction rules and construct computationally efficient approximations with provable regret guarantees and robustness against model misspecification. Our algorithm, based on block coordinate ascent, scales effortlessly to XMLC problems and obtains promising results in terms of long-tail performance.

Figures (8)

• Figure 1: Results of an inference strategy with a mixed utility on and $k=3$. The green line shows the results for different interpolations between two measures.
• Figure 2: Comparison of the baseline algorithms with the BCA inference with mixed objectives with $k=3$. The green line shows the results for different interpolations between two measures.
• Figure 3: Comparison of the baseline algorithms with the BCA inference with mixed objectives with $k=5$. The green line shows the results for different interpolations between two measures.
• Figure : BCA$(\sinstancematrix, \empirical \marginals, k, \epsilon)$
• Figure : BCA for coverage$(\sinstancematrix, \empirical \marginals, k, \epsilon)$

Theorems & Definitions (25)

• Definition 5.0: $\condpos$-Lipschitz Dembczynski_etal_ICML2017
• Theorem 5.1
• Theorem 5.2
• Definition A.1: Invariant under instance reordering
• Lemma A.2
• proof
• Theorem A.3: Equivalence of order-invariance and confusion-matrix losses
• proof
• Lemma A.4
• proof