RankOOD -- Class Ranking-based Out-of-Distribution Detection

TL;DR

RankOOD tackles OOD detection by exploiting rank-based structure in classifier logits. It derives canonical per-class rankings from an RPM via ILP, then trains a model with a hybrid $L_{CE}$ and $L_{ListMLE}$ objective to enforce global rank consistency using the $Plackett-Luce$ formulation. At inference, RankOOD computes a score that penalizes deviations from the canonical ranks and from class-specific logit thresholds, achieving state-of-the-art near-OOD performance on TinyImageNet and strong results overall without requiring outlier data or architectural changes. The approach demonstrates that preserving structured logit hierarchies improves OOD discrimination and offers robust, interpretable signaling for deployment in safety-critical systems.

Abstract

We propose RankOOD, a rank-based Out-of-Distribution (OOD) detection approach based on training a model with the Placket-Luce loss, which is now extensively used for preference alignment tasks in foundational models. Our approach is based on the insight that with a deep learning model trained using the Cross Entropy Loss, in-distribution (ID) class prediction induces a ranking pattern for each ID class prediction. The RankOOD framework formalizes the insight by first extracting a rank list for each class using an initial classifier and then uses another round of training with the Plackett-Luce loss, where the class rank, a fixed permutation for each class, is the predicted variable. An OOD example may get assigned with high probability to an ID example, but the probability of it respecting the ranking classification is likely to be small. RankOOD, achieves SOTA performance on the near-ODD TinyImageNet evaluation benchmark, reducing FPR95 by 4.3%.

Figures (6)

• Figure 1: The performance comparison of average FPR95 on Far-OOD (x-axis) and Near-OOD (y-axis). RankOOD outperforms all post-hoc and training methods without outliers. Compared to training methods with outliers, its performance is only second to the Outlier Exposure (OE) method.
• Figure 2: Logit distributions at selected rank positions for predicted class $82$ on CIFAR100. Across training epochs, (b) ID samples show increasing separation across rank positions, while (b) OOD samples remain concentrated near zero with minimal separation. OOD samples are shown for comparison only.
• Figure 3: Left: Distributions of RankOOD and MSP scores for CIFAR-10 (ID) and SVHN (OOD) samples under models trained with CE and RankOOD losses. Right: Conditional probability matrix (CP) of rank position $i$ given that all prior ranks have been correctly predicted.
• Figure 4: OOD and ID detection performance on CIFAR-100 when training with different rank subsets. (left): AUROC (right):ID accuracy. Top Ranks uses the top$-N$ class ranks, Bottom Ranks uses the lowest $N-1$ ranks along with rank$-0$, and Top-Bottom samples half of the ranks from the top and half from the lowest ranks.
• Figure 5: OOD detection performance on CIFAR-100 with respect to $\alpha$ and logit threshold ($Ref$) hyper-parameters.