Collaborative Multi-LoRA Experts with Achievement-based Multi-Tasks Loss for Unified Multimodal Information Extraction

TL;DR

The paper tackles gradient conflicts and computational cost in unified multimodal information extraction by introducing Collaborative Multi-LoRA Experts (C-LoRAE), a two-level LoRA system with a universal expert and task-specific experts. It couples mutual information maximization and an expert-gated router to balance sharing and specialization, and introduces an achievement-based multi-task loss to dynamically balance training across tasks with uneven data. Across seven benchmarks spanning MNER, MRE, and MEE, C-LoRAE achieves state-of-the-art performance on several datasets while using far fewer trainable parameters than full fine-tuning, demonstrating both efficiency and scalability. The approach highlights effective cross-task knowledge sharing in multimodal settings and lays groundwork for further scaling to larger vision-language models.

Abstract

Multimodal Information Extraction (MIE) has gained attention for extracting structured information from multimedia sources. Traditional methods tackle MIE tasks separately, missing opportunities to share knowledge across tasks. Recent approaches unify these tasks into a generation problem using instruction-based T5 models with visual adaptors, optimized through full-parameter fine-tuning. However, this method is computationally intensive, and multi-task fine-tuning often faces gradient conflicts, limiting performance. To address these challenges, we propose collaborative multi-LoRA experts with achievement-based multi-task loss (C-LoRAE) for MIE tasks. C-LoRAE extends the low-rank adaptation (LoRA) method by incorporating a universal expert to learn shared multimodal knowledge from cross-MIE tasks and task-specific experts to learn specialized instructional task features. This configuration enhances the model's generalization ability across multiple tasks while maintaining the independence of various instruction tasks and mitigating gradient conflicts. Additionally, we propose an achievement-based multi-task loss to balance training progress across tasks, addressing the imbalance caused by varying numbers of training samples in MIE tasks. Experimental results on seven benchmark datasets across three key MIE tasks demonstrate that C-LoRAE achieves superior overall performance compared to traditional fine-tuning methods and LoRA methods while utilizing a comparable number of training parameters to LoRA.

Figures (6)

• Figure 1: Three Key MIE Tasks, where O$_1$,O$_2$, and O$_3$ are visual objects.
• Figure 2: Different fine-tuning methods for MIE.
• Figure 3: Overall framework of our Collaborative Multi-LoRA Experts. Our model is based on the General Large Visual-Language Model. The input text is tokenized and embedded and then concatenated with the visual input to feed into the LLMs.
• Figure 4: Normalized performance across seven tasks for LoRA, MoSLoRA, LoRAMoE, and our proposed C-LoRA.
• Figure 5: Experimental results of our model with different C-LoRAE parameters. The datasets denoted by T15, T17, RE-V1, and RE-V2 represent Twitter-15, Twitter-17, MRE-V1, and MRE-V2, respectively.