Ranking hierarchical multi-label classification results with mLPRs

TL;DR

This paper tackles the second-stage decision problem in hierarchical multi-label classification by introducing the multidimensional Local Precision Rate (mLPR) and the Conditional expected Area under the Curve of Hit (CATCH) as a principled objective. It proves that ranking objects by true $mLPR$ under the given hierarchy optimizes CATCH, and develops HierRank to maximize an empirical version of CATCH when $mLPR$ must be estimated. The authors propose full, independence, and neighborhood mLPR estimation, analyze convergence, and present a Chain-Merge based HierRank algorithm with a formal and toy-example exposition. They validate the approach on synthetic and real datasets (disease diagnosis and RCV1v2), showing improved early-precision decisions and competitive performance across metrics like precision-recall and F1, while guaranteeing hierarchical consistency. The framework provides a statistically interpretable, flexible solution for HMC decision-making with practical guidance on when to use each estimation variant and how to apply cutoff-based decision rules.

Abstract

Hierarchical multi-label classification (HMC) has gained considerable attention in recent decades. A seminal line of HMC research addresses the problem in two stages: first, training individual classifiers for each class, then integrating these classifiers to provide a unified set of classification results across classes while respecting the given hierarchy. In this article, we focus on the less attended second-stage question while adhering to the given class hierarchy. This involves addressing a key challenge: how to manage the hierarchical constraint and account for statistical differences in the first-stage classifier scores across different classes to make classification decisions that are optimal under a justifiable criterion. To address this challenge, we introduce a new objective function, called CATCH, to ensure reasonable classification performance. To optimize this function, we propose a decision strategy built on a novel metric, the multidimensional Local Precision Rate (mLPR), which reflects the membership chance of an object in a class given all classifier scores and the class hierarchy. Particularly, we demonstrate that, under certain conditions, transforming the classifier scores into mLPRs and comparing mLPR values for all objects against all classes can, in theory, ensure the class hierarchy and maximize CATCH. In practice, we propose an algorithm HierRank to rank estimated mLPRs under the hierarchical constraint, leading to a ranking that maximizes an empirical version of CATCH. Our approach was evaluated on a synthetic dataset and two real datasets, exhibiting superior performance compared to several state-of-the-art methods in terms of improved decision accuracy.

Figures (5)

• Figure 1: Example hierarchical graph $\mathop{\mathrm{\mathcal{G}}}\limits$ and its associated augmented graph $\mathop{\mathrm{\overline{\mathcal{G}}}}\limits$.
• Figure 2: Example of the merging process in Algorithm 1: Bold circles indicate the sub-chain with the highest average $\widehat{mLPR}$ values, while gray-filled circles represent the ranking produced by the merging procedure.
• Figure 3: Class trees of the synthetic dataset: White, gray, and black indicate classes with high, medium, and low classifier quality, respectively.
• Figure 4: Structure of the classes of the disease diagnosis dataset. The grayscale corresponds to the node quality: white indicates that a node's base classifier has an area under the curve (AUC) value of receiver operating characteristic curve (ROC) within the interval $(0.9, 1]$; light gray, $(0.7, 0.9]$; and dark gray, $(0, 0.7]$. The values inside the circles indicate the number of positive cases, and the values underneath indicate the maximum percentage of positive cases shared with a parent node.
• Figure 5: Precision--recall curve for several classifiers applied to the disease diagnosis dataset of huang2010.

Theorems & Definitions (8)

• Proposition 3.1
• proof
• Proposition 3.2
• proof
• Corollary 3.3
• proof
• Theorem 4.1
• Theorem 4.2