Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

UnifiedMLLM: Enabling Unified Representation for Multi-modal Multi-tasks With Large Language Model

Zhaowei Li, Wei Wang, YiQing Cai, Xu Qi, Pengyu Wang, Dong Zhang, Hang Song, Botian Jiang, Zhida Huang, Tao Wang

TL;DR

<3-5 sentence high-level summary> UnifiedMLLM tackles the challenge of representing and executing a broad spectrum of multi-modal tasks within a single framework. It introduces a unified representation built from task tokens and grounding tokens, guided by a task router that activates specialized external experts; training proceeds in three stages to preserve base knowledge while expanding capabilities. The approach demonstrates strong performance across referring segmentation, reasoning editing, layout-based generation, and multi-modal generation tasks, and shows notable scalability and generalization. This work offers a practical path toward end-to-end, unified multi-modal instruction tuning with modular expert integration, enabling easier extension to new tasks and modalities.

Abstract

Significant advancements has recently been achieved in the field of multi-modal large language models (MLLMs), demonstrating their remarkable capabilities in understanding and reasoning across diverse tasks. However, these models are often trained for specific tasks and rely on task-specific input-output formats, limiting their applicability to a broader range of tasks. This raises a fundamental question: Can we develop a unified approach to represent and handle different multi-modal tasks to maximize the generalizability of MLLMs? In this paper, we propose UnifiedMLLM, a comprehensive model designed to represent various tasks using a unified representation. Our model exhibits strong capabilities in comprehending the implicit intent of user instructions and preforming reasoning. In addition to generating textual responses, our model also outputs task tokens and grounding tokens, serving as indicators of task types and task granularity. These outputs are subsequently routed through the task router and directed to specific expert models for task completion. To train our model, we construct a task-specific dataset and an 100k multi-task dataset encompassing complex scenarios. Employing a three-stage training strategy, we equip our model with robust reasoning and task processing capabilities while preserving its generalization capacity and knowledge reservoir. Extensive experiments showcase the impressive performance of our unified representation approach across various tasks, surpassing existing methodologies. Furthermore, our approach exhibits exceptional scalability and generality. Our code, model, and dataset will be available at \url{https://github.com/lzw-lzw/UnifiedMLLM}.

UnifiedMLLM: Enabling Unified Representation for Multi-modal Multi-tasks With Large Language Model

TL;DR

<3-5 sentence high-level summary> UnifiedMLLM tackles the challenge of representing and executing a broad spectrum of multi-modal tasks within a single framework. It introduces a unified representation built from task tokens and grounding tokens, guided by a task router that activates specialized external experts; training proceeds in three stages to preserve base knowledge while expanding capabilities. The approach demonstrates strong performance across referring segmentation, reasoning editing, layout-based generation, and multi-modal generation tasks, and shows notable scalability and generalization. This work offers a practical path toward end-to-end, unified multi-modal instruction tuning with modular expert integration, enabling easier extension to new tasks and modalities.

Abstract

Significant advancements has recently been achieved in the field of multi-modal large language models (MLLMs), demonstrating their remarkable capabilities in understanding and reasoning across diverse tasks. However, these models are often trained for specific tasks and rely on task-specific input-output formats, limiting their applicability to a broader range of tasks. This raises a fundamental question: Can we develop a unified approach to represent and handle different multi-modal tasks to maximize the generalizability of MLLMs? In this paper, we propose UnifiedMLLM, a comprehensive model designed to represent various tasks using a unified representation. Our model exhibits strong capabilities in comprehending the implicit intent of user instructions and preforming reasoning. In addition to generating textual responses, our model also outputs task tokens and grounding tokens, serving as indicators of task types and task granularity. These outputs are subsequently routed through the task router and directed to specific expert models for task completion. To train our model, we construct a task-specific dataset and an 100k multi-task dataset encompassing complex scenarios. Employing a three-stage training strategy, we equip our model with robust reasoning and task processing capabilities while preserving its generalization capacity and knowledge reservoir. Extensive experiments showcase the impressive performance of our unified representation approach across various tasks, surpassing existing methodologies. Furthermore, our approach exhibits exceptional scalability and generality. Our code, model, and dataset will be available at \url{https://github.com/lzw-lzw/UnifiedMLLM}.
Paper Structure (29 sections, 5 equations, 3 figures, 6 tables)

This paper contains 29 sections, 5 equations, 3 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Multi-modal Large Language Models (MLLMs)
  4. Multi-tasks MLLMs
  5. Method
  6. Architecture
  7. Encoder and Adapter
  8. Unified Representation
  9. Experts Integration
  10. LoRA Mixture of Experts
  11. Dataset
  12. Task-specific Dataset
  13. Multi-turn Multi-task Dataset
  14. Training
  15. Modality-perception Pretraining
  16. ...and 14 more sections

Figures (3)

  • Figure 1: We introduce UnifiedMLLM, a model that represents and handles different multi-modal tasks in a unified manner, enabling it to perform tasks involving multi-modal understanding, processing, and generation.
  • Figure 2: The model structure and three-stage training strategy of UnifiedMLLM.(a) Model structure and training strategy in the first two stages. (b) Training strategy in the third stage.
  • Figure 3: Examples of UnifiedMLLM Dataset in Unified Representation Format. We provide examples of task-specific datasets and multi-task multi-turn datasets.