Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Mastering the Minority: An Uncertainty-guided Multi-Expert Framework for Challenging-tailed Sequence Learning

Ye Wang, Zixuan Wu, Lifeng Shen, Jiang Xie, Xiaoling Wang, Hong Yu, Guoyin Wang

Abstract

Imbalanced data distribution remains a critical challenge in sequential learning, leading models to easily recognize frequent categories while failing to detect minority classes adequately. The Mixture-of-Experts model offers a scalable solution, yet its application is often hindered by parameter inefficiency, poor expert specialization, and difficulty in resolving prediction conflicts. To Master the Minority classes effectively, we propose the Uncertainty-based Multi-Expert fusion network (UME) framework. UME is designed with three core innovations: First, we employ Ensemble LoRA for parameter-efficient modeling, significantly reducing the trainable parameter count. Second, we introduce Sequential Specialization guided by Dempster-Shafer Theory (DST), which ensures effective specialization on the challenging-tailed classes. Finally, an Uncertainty-Guided Fusion mechanism uses DST's certainty measures to dynamically weigh expert opinions, resolving conflicts by prioritizing the most confident expert for reliable final predictions. Extensive experiments across four public hierarchical text classification datasets demonstrate that UME achieves state-of-the-art performance. We achieve a performance gain of up to 17.97\% over the best baseline on individual categories, while reducing trainable parameters by up to 10.32\%. The findings highlight that uncertainty-guided expert coordination is a principled strategy for addressing challenging-tailed sequence learning. Our code is available at https://github.com/CQUPTWZX/Multi-experts.

Mastering the Minority: An Uncertainty-guided Multi-Expert Framework for Challenging-tailed Sequence Learning

Abstract

Imbalanced data distribution remains a critical challenge in sequential learning, leading models to easily recognize frequent categories while failing to detect minority classes adequately. The Mixture-of-Experts model offers a scalable solution, yet its application is often hindered by parameter inefficiency, poor expert specialization, and difficulty in resolving prediction conflicts. To Master the Minority classes effectively, we propose the Uncertainty-based Multi-Expert fusion network (UME) framework. UME is designed with three core innovations: First, we employ Ensemble LoRA for parameter-efficient modeling, significantly reducing the trainable parameter count. Second, we introduce Sequential Specialization guided by Dempster-Shafer Theory (DST), which ensures effective specialization on the challenging-tailed classes. Finally, an Uncertainty-Guided Fusion mechanism uses DST's certainty measures to dynamically weigh expert opinions, resolving conflicts by prioritizing the most confident expert for reliable final predictions. Extensive experiments across four public hierarchical text classification datasets demonstrate that UME achieves state-of-the-art performance. We achieve a performance gain of up to 17.97\% over the best baseline on individual categories, while reducing trainable parameters by up to 10.32\%. The findings highlight that uncertainty-guided expert coordination is a principled strategy for addressing challenging-tailed sequence learning. Our code is available at https://github.com/CQUPTWZX/Multi-experts.
Paper Structure (22 sections, 22 equations, 10 figures, 9 tables)

This paper contains 22 sections, 22 equations, 10 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Hierarchical Text Classification
  4. Uncertainty-based Text Classification
  5. Multi-Expert Text Classification
  6. Methodology
  7. Preliminaries
  8. Representations of Text and Label Hierarchy
  9. Low-rank Experts
  10. Uncertainty-based Multi-Expert Fusion Network
  11. Training Objectives
  12. Experiments
  13. Datasets
  14. Baseline and Parameter Settings
  15. Main Results on Hierarchical Text Classification (RQ1)
  16. ...and 7 more sections

Figures (10)

  • Figure 1: As the classification hierarchy becomes deeper, the number of samples per label decreases significantly, leading to a pronounced challenging-tailed problem and a sharp drop in the classification performance of minority classes.
  • Figure 2: The structure of our proposed model UME. The uncertainty of every single expert is attained for multi-expert learning. Multi-expert Joint Uncertainty is dynamically ensembled by the degree of uncertainty among experts.
  • Figure 3: Class distribution analysis across four benchmark datasets. The histograms use a logarithmic scale to visualize the severe long-tailed nature and extreme class imbalance ratios (IR) present in WOS, RCV1-V2, AAPD, and BGC.
  • Figure 4: The confusion matrix highlights the contrast between HGCLR and multi-experts in terms of prediction accuracy.
  • Figure 5: Two examples illustrate the effectiveness of uncertainty-based multi-expert fusion. Compared with baselines, our classification results are more reliable.
  • ...and 5 more figures

Theorems & Definitions (2)

  • Definition 1
  • Definition 2