Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Orchestrate Multimodal Data with Batch Post-Balancing to Accelerate Multimodal Large Language Model Training

Yijie Zheng, Bangjun Xiao, Lei Shi, Xiaoyang Li, Faming Wu, Tianyu Li, Xuefeng Xiao, Yang Zhang, Yuxuan Wang, Shouda Liu

TL;DR

OrchMLLM tackles the efficiency bottleneck in multimodal LLM training caused by Modality Composition Incoherence by introducing a post-balancing strategy. It pairs Batch Post-Balancing Dispatchers with a MLLM Global Orchestrator to rearrange mini-batches after sampling, achieving balanced GPU utilization across all training phases. The system employs a Node-wise All-to-All Communicator and ILP-based Node-wise Rearrangement to minimize inter-node communication, while the orchestrator ensures correct data assembly and overlapping of computation with communication. Experiments on a $2560$-GPU cluster show MFU of $41.6\%$ and up to $3.1\times$ throughput gains over Megatron-LM, with overhead kept below $2\%$ of forward time, demonstrating strong scalability and practical impact for large-scale MLLM training.

Abstract

Multimodal large language models (MLLMs), such as GPT-4o, are garnering significant attention. During the exploration of MLLM training, we identified Modality Composition Incoherence, a phenomenon that the proportion of a certain modality varies dramatically across different examples. It exacerbates the challenges of addressing mini-batch imbalances, which lead to uneven GPU utilization between Data Parallel (DP) instances and severely degrades the efficiency and scalability of MLLM training, ultimately affecting training speed and hindering further research on MLLMs. To address these challenges, we introduce OrchMLLM, a comprehensive framework designed to mitigate the inefficiencies in MLLM training caused by Modality Composition Incoherence. First, we propose Batch Post-Balancing Dispatcher, a technique that efficiently eliminates mini-batch imbalances in sequential data. Additionally, we integrate MLLM Global Orchestrator into the training framework to orchestrate multimodal data and tackle the issues arising from Modality Composition Incoherence. We evaluate OrchMLLM across various MLLM sizes, demonstrating its efficiency and scalability. Experimental results reveal that OrchMLLM achieves a Model FLOPs Utilization (MFU) of $41.6\%$ when training an 84B MLLM with three modalities on $2560$ H100 GPUs, outperforming Megatron-LM by up to $3.1\times$ in throughput.

Orchestrate Multimodal Data with Batch Post-Balancing to Accelerate Multimodal Large Language Model Training

TL;DR

OrchMLLM tackles the efficiency bottleneck in multimodal LLM training caused by Modality Composition Incoherence by introducing a post-balancing strategy. It pairs Batch Post-Balancing Dispatchers with a MLLM Global Orchestrator to rearrange mini-batches after sampling, achieving balanced GPU utilization across all training phases. The system employs a Node-wise All-to-All Communicator and ILP-based Node-wise Rearrangement to minimize inter-node communication, while the orchestrator ensures correct data assembly and overlapping of computation with communication. Experiments on a -GPU cluster show MFU of and up to throughput gains over Megatron-LM, with overhead kept below of forward time, demonstrating strong scalability and practical impact for large-scale MLLM training.

Abstract

Multimodal large language models (MLLMs), such as GPT-4o, are garnering significant attention. During the exploration of MLLM training, we identified Modality Composition Incoherence, a phenomenon that the proportion of a certain modality varies dramatically across different examples. It exacerbates the challenges of addressing mini-batch imbalances, which lead to uneven GPU utilization between Data Parallel (DP) instances and severely degrades the efficiency and scalability of MLLM training, ultimately affecting training speed and hindering further research on MLLMs. To address these challenges, we introduce OrchMLLM, a comprehensive framework designed to mitigate the inefficiencies in MLLM training caused by Modality Composition Incoherence. First, we propose Batch Post-Balancing Dispatcher, a technique that efficiently eliminates mini-batch imbalances in sequential data. Additionally, we integrate MLLM Global Orchestrator into the training framework to orchestrate multimodal data and tackle the issues arising from Modality Composition Incoherence. We evaluate OrchMLLM across various MLLM sizes, demonstrating its efficiency and scalability. Experimental results reveal that OrchMLLM achieves a Model FLOPs Utilization (MFU) of when training an 84B MLLM with three modalities on H100 GPUs, outperforming Megatron-LM by up to in throughput.

Paper Structure

This paper contains 24 sections, 13 equations, 13 figures, 2 tables, 5 algorithms.

Table of Contents

  1. Introduction
  2. Background
  3. Data Parallelism
  4. Imbalance in Mini-batches
  5. MLLM training
  6. Motivation
  7. Modality Composition Incoherence
  8. Limitation of Existing Methods
  9. Opportunity
  10. OrchMLLM Overview
  11. Batch Post-Balancing Dispatcher
  12. Batch Post-Balancing Algorithms
  13. Node-wise All-to-All Communicator
  14. All-to-All Batch Communicator.
  15. Node-wise Rearrangement Algorithm
  16. ...and 9 more sections

Figures (13)

  • Figure 1: A comparative illustration showcasing the effectiveness of OrchMLLM.
  • Figure 2: The diagram for the imbalance in mini-batches in the training of sequential data, using the padding batching method as an example.
  • Figure 3: Modality Composition Incoherence in MLLM training. Both of the ratios bear substantial variance, which exhibits the incoherence in composition of multimodal data.
  • Figure 4: The system overview of OrchMLLM.
  • Figure 5: An illustrative diagram exhibiting the rearrangement process using the All-to-All operation, where each number refers to the length of a sequence in batches $S_i$.
  • ...and 8 more figures