Optimizing Partial Area Under the Top-k Curve: Theory and Practice
Zitai Wang, Qianqian Xu, Zhiyong Yang, Yuan He, Xiaochun Cao, Qingming Huang
TL;DR
This work addresses the inadequacy of the top-$k$ error in large-label, ambiguous settings by introducing AUTKC, a discriminative metric that integrates area under the top-$k$ curve (and a partial variant) to better capture performance across plausible top-$K$ predictions. It establishes a Fisher-consistent surrogate-risk framework for optimizing AUTKC, including a Bayes-optimality characterization and sufficient conditions for surrogate-loss consistency (bounded, differentiable, strictly decreasing), while showing hinge loss is inconsistent. Generalization analyses yield bounds that are insensitive to the total number of classes under certain regularity, and the theory is extended to convolutional networks via chaining bounds and Lipschitz properties. Empirically, AUTKC-based optimization improves performance on datasets with significant label ambiguity (CIFAR-100, Tiny-ImageNet-200, Places-365) and remains robust to hyperparameters, validating both the metric and the learning framework for practical deployment. The work thus offers a principled, scalable path to discriminative top-$K$ predictions in large-scale, semantically overlapping label spaces.
Abstract
Top-k error has become a popular metric for large-scale classification benchmarks due to the inevitable semantic ambiguity among classes. Existing literature on top-k optimization generally focuses on the optimization method of the top-k objective, while ignoring the limitations of the metric itself. In this paper, we point out that the top-k objective lacks enough discrimination such that the induced predictions may give a totally irrelevant label a top rank. To fix this issue, we develop a novel metric named partial Area Under the top-k Curve (AUTKC). Theoretical analysis shows that AUTKC has a better discrimination ability, and its Bayes optimal score function could give a correct top-K ranking with respect to the conditional probability. This shows that AUTKC does not allow irrelevant labels to appear in the top list. Furthermore, we present an empirical surrogate risk minimization framework to optimize the proposed metric. Theoretically, we present (1) a sufficient condition for Fisher consistency of the Bayes optimal score function; (2) a generalization upper bound which is insensitive to the number of classes under a simple hyperparameter setting. Finally, the experimental results on four benchmark datasets validate the effectiveness of our proposed framework.