LPT++: Efficient Training on Mixture of Long-tailed Experts

TL;DR

LPT++ addresses long-tailed image classification by combining parameter-efficient fine-tuning with a model-ensemble strategy. It introduces a universal long-tailed adaptation module, a mixture of long-tailed experts (MoLEs) with a lightweight MoE scorer, and a three-phase training regime to separately optimize prompts, adapters, and the MoE scorer. The simpler LPT variant isolates the impact of long-tailed prompts on visual-only pretrained ViTs. Empirically, LPT++ achieves state-of-the-art or near-state-of-the-art results on Places-LT and iNaturalist 2018, with robust performance under domain shift and minimal additional trainable parameters (~1%), highlighting its practicality for deployment.

Abstract

We introduce LPT++, a comprehensive framework for long-tailed classification that combines parameter-efficient fine-tuning (PEFT) with a learnable model ensemble. LPT++ enhances frozen Vision Transformers (ViTs) through the integration of three core components. The first is a universal long-tailed adaptation module, which aggregates long-tailed prompts and visual adapters to adapt the pretrained model to the target domain, meanwhile improving its discriminative ability. The second is the mixture of long-tailed experts framework with a mixture-of-experts (MoE) scorer, which adaptively calculates reweighting coefficients for confidence scores from both visual-only and visual-language (VL) model experts to generate more accurate predictions. Finally, LPT++ employs a three-phase training framework, wherein each critical module is learned separately, resulting in a stable and effective long-tailed classification training paradigm. Besides, we also propose the simple version of LPT++ namely LPT, which only integrates visual-only pretrained ViT and long-tailed prompts to formulate a single model method. LPT can clearly illustrate how long-tailed prompts works meanwhile achieving comparable performance without VL pretrained models. Experiments show that, with only ~1% extra trainable parameters, LPT++ achieves comparable accuracy against all the counterparts.

Figures (5)

• Figure 1: Comparison among state-of-the-art long-tailed approaches on Places-LT and iNaturalist 2018, where the size of each spot means the model size of the whole model. LPT++ is our proposed visual-language pretrained long-tailed classification method with mixture of long-tailed experts framework. And LPT is the simple version of LPT++ which removes visual-language pretrained models, visual adapters and mixture-of-experts formulation. LPT and LPT++ only requires $\sim$1% extra trainable parameters while achieving higher accuracy on two highly long-tailed datasets.
• Figure 2: LDA visualization of VPT.
• Figure 3: Analysis of features from ViT-B and VPT.
• Figure 4: (a) Pipeline of LPT++, where snow means frozen parameters and fire means trainable parameters. LPT++ generates confidence scores via both visual-only and visual-language models, then utilizes MoE scorer to calculate reweighting coefficient $\mathbf{W}_{\text{moe}}$ for final prediction. (b) Structure of LPT++ block, where "S" means scale operation. (c) is the pipeline of MoE scorer, which leverages searching-based scorer to solve shared weight $\mathbf{W}_{\text{base}}$, then uses learning-based scorer to calculate offset $\mathbf{W}_{\text{offset}}$.
• Figure 5: Multi-phase training framework of LPT++. In phase 1, LPT++ optimizes both shared prompt and visual adapters simultaneously to adapt pretrained model to target domain and improve the discriminative ability. In phase 2, LPT++ freezes the learned shared prompts and visual adapters, and optimizes the group-specific prompts to further improve the discriminative ability. Both phases bring the visual-only and visual-language LPT++ model experts. And in phase 3, with confidence scores from both model experts, LPT++ optimizes searching-based and learning-based scorer to adaptively reweight confidence scores.