Top-K Pairwise Ranking: Bridging the Gap Among Ranking-Based Measures for Multi-Label Classification
Zitai Wang, Qianqian Xu, Zhiyong Yang, Peisong Wen, Yuan He, Xiaochun Cao, Qingming Huang
TL;DR
The paper tackles the challenge of inconsistent performance across ranking-based measures in multi-label classification by introducing Top-K Pairwise Ranking (TKPR), a unifying objective with three equivalent formulations that align with pointwise, pairwise, and listwise views. It develops an Empirical Risk Minimization (ERM) framework for TKPR supported by Fisher consistency, Bayes optimality under a top-K ranking-with-ties criterion, and a data-dependent contraction technique that yields sharp generalization bounds, including for missing-label settings. The authors show TKPR is compatible with existing measures such as precision@K, recall@K, AP@K, and NDCG@K, while providing a tight upper bound to the ranking loss and a bound relation to AP@K. Empirically, TKPR achieves consistent improvements across mAP@K, NDCG@K, and ranking loss on benchmarks like MS-COCO and Pascal VOC, with favorable computational properties (O(CK) per sample) compared to O(C^2) for the ranking loss. The work demonstrates that optimizing TKPR can serve as a practical, theoretically grounded surrogate that harmonizes multiple ranking-based metrics and scales to large label sets, offering strong potential for applications in visual recognition and related retrieval tasks.
Abstract
Multi-label ranking, which returns multiple top-ranked labels for each instance, has a wide range of applications for visual tasks. Due to its complicated setting, prior arts have proposed various measures to evaluate model performances. However, both theoretical analysis and empirical observations show that a model might perform inconsistently on different measures. To bridge this gap, this paper proposes a novel measure named Top-K Pairwise Ranking (TKPR), and a series of analyses show that TKPR is compatible with existing ranking-based measures. In light of this, we further establish an empirical surrogate risk minimization framework for TKPR. On one hand, the proposed framework enjoys convex surrogate losses with the theoretical support of Fisher consistency. On the other hand, we establish a sharp generalization bound for the proposed framework based on a novel technique named data-dependent contraction. Finally, empirical results on benchmark datasets validate the effectiveness of the proposed framework.