Harnessing Hierarchical Label Distribution Variations in Test Agnostic Long-tail Recognition

TL;DR

This work tackles test-agnostic long-tail recognition where test label distributions are unknown and arbitrarily imbalanced. It introduces DirMixE, a Dirichlet mixture of experts that models a meta-distribution $\mathcal{E}$ over label distributions as $\mathcal{E}=\sum_{i=1}^K p_i\,\mathsf{Dir}(\alpha^{(i)})$, with an expert per Dirichlet component and self-supervised aggregation at test time. Training optimizes the logit-adjusted loss $\ell_{LA}$ across distributions drawn from $\mathcal{E}$ and uses Monte Carlo estimation to compute the mean and a semi-variance $\mathbb{V}_+(\ell_{LA})$ to regularize the objective, yielding a sharper generalization bound. Theoretical results show improved generalization under distributional shifts and empirical findings demonstrate strong gains, especially in backward long-tail settings, with ablations validating the semi-variance benefit and the effectiveness of the hierarchical expert routing. Overall, DirMixE provides a principled, scalable approach to capture global and local distribution variations, enabling robust performance across diverse test-time imbalances.

Abstract

This paper explores test-agnostic long-tail recognition, a challenging long-tail task where the test label distributions are unknown and arbitrarily imbalanced. We argue that the variation in these distributions can be broken down hierarchically into global and local levels. The global ones reflect a broad range of diversity, while the local ones typically arise from milder changes, often focused on a particular neighbor. Traditional methods predominantly use a Mixture-of-Expert (MoE) approach, targeting a few fixed test label distributions that exhibit substantial global variations. However, the local variations are left unconsidered. To address this issue, we propose a new MoE strategy, $\mathsf{DirMixE}$, which assigns experts to different Dirichlet meta-distributions of the label distribution, each targeting a specific aspect of local variations. Additionally, the diversity among these Dirichlet meta-distributions inherently captures global variations. This dual-level approach also leads to a more stable objective function, allowing us to sample different test distributions better to quantify the mean and variance of performance outcomes. Theoretically, we show that our proposed objective benefits from enhanced generalization by virtue of the variance-based regularization. Comprehensive experiments across multiple benchmarks confirm the effectiveness of $\mathsf{DirMixE}$. The code is available at \url{https://github.com/scongl/DirMixE}.

Figures (11)

• Figure 1: Traditional methods can only capture global variations of label distribution. By contrast, our $\mathsf{DirMixE}$ learns from both global and local variations, covering more test distributions.
• Figure 2: Illustration of the training process, where a hierarchical sampling process is employed to esitmate the empirical risk.
• Figure 3: The Upper Bound for $\rho$ for different distributions. (a) shows the distribution density of the Gamma distribution with $\beta$ fixed as 0.5 and $\alpha$ varied. The corresponding distribution has a tailed shape when $\alpha<1$. (b) shows the upper bound $\rho$ for $\alpha <1$. One can observe that $\rho$ is roughly greater than $0.4$ in this range. (c) shows the distribution density of the Pareto distribution with $\ell_m$ fixed as $0.1$ and $\theta$ varied. (d) shows the $\rho$ value for $2<\theta \le 10$, we can see that $\rho$ is slightly greater $0.2$ for large $\theta$. In all these observed cases, the assumption $\mathop{\mathrm{\mathbb{V}}}\limits_{+} \asymp \mathop{\mathrm{\mathbb{V}}}\limits$ is admissible.
• Figure 4: Weight Assignment in the Self-supervised Aggregations on CIFAR-100.F,U,B represent the forward, uniform and backward distributions, see Appendix \ref{['app:exp']} for more.
• Figure 5: The Correlation between Expert Weights and Loss.

Theorems & Definitions (24)

• Theorem 1: Generalization Bound, Informal
• Theorem 2: Model Averaging Error
• Theorem 3
• Theorem 4
• proof
• Lemma 1
• proof
• proof
• Lemma 2
• proof