Fast-Slow Efficient Training for Multimodal Large Language Models via Visual Token Pruning

TL;DR

The paper tackles the high training cost of Multimodal Large Language Models arising from vast visual token counts. It introduces DualSpeed, a fast-slow training framework that uses Visual Token Pruning in fast-mode and full-sequence learning with self-distillation in slow-mode, bridged by a learnable mode isolator and a probabilistic mode-switch $r$. Empirically, DualSpeed delivers 2.1x–4.0x training speedups with roughly 99% retention across LLaVA-1.5 and LLaVA-NeXT while significantly reducing the training-inference mismatch to about $3.72\%$. This approach provides a practical, generalizable path for efficient, scalable pretraining of high-resolution Multimodal Large Language Models.

Abstract

Multimodal Large Language Models (MLLMs) suffer from severe training inefficiency issue, which is associated with their massive model sizes and visual token numbers. Existing efforts in efficient training focus on reducing model sizes or trainable parameters. Inspired by the success of Visual Token Pruning (VTP) in improving inference efficiency, we are exploring another substantial research direction for efficient training by reducing visual tokens. However, applying VTP at the training stage results in a training-inference mismatch: pruning-trained models perform poorly when inferring on non-pruned full visual token sequences. To close this gap, we propose DualSpeed, a fast-slow framework for efficient training of MLLMs. The fast-mode is the primary mode, which incorporates existing VTP methods as plugins to reduce visual tokens, along with a mode isolator to isolate the model's behaviors. The slow-mode is the auxiliary mode, where the model is trained on full visual sequences to retain training-inference consistency. To boost its training, it further leverages self-distillation to learn from the sufficiently trained fast-mode. Together, DualSpeed can achieve both training efficiency and non-degraded performance. Experiments show DualSpeed accelerates the training of LLaVA-1.5 by 2.1$\times$ and LLaVA-NeXT by 4.0$\times$, retaining over 99% performance. Code: https://github.com/dingkun-zhang/DualSpeed

Figures (6)

• Figure 1: DualSpeed significantly accelerates MLLM training while retaining performance. It achieves up to 4.0$\times$ training speedup, with nearly lossless performance.
• Figure 2: Illustration of the performance gaps caused by training-inference mismatch. The performance upper bound of pruning inference is theoretically lower than normal inference, as pruning leads to the loss of visual information. Therefore, the possible path to approaching 100% performance is to address the training-inference mismatch.
• Figure 3: Overview of DualSpeed. The framework comprises a fast-mode and a slow-mode. The two modes are randomly switched between different mini-batches, with fast-mode being the primary mode to maximize training efficiency, and slow-mode being the auxiliary mode to retain training-inference consistency. Different VTP methods are plugable for the DualSpeed framework.
• Figure 4: Comparison of training time and performance between DualSpeed and the baseline. We measure the performance of 10 checkpoints uniformly saved during the SFT phase. Performance in the pretraining phase is not measured since the model in this phase can not follow the common instructions of many benchmarks. The wall-clock time is recorded from the start of pretraining. Times are measured on a single NVIDIA L40. $\dagger$: The training follows the training recipe and data of LLaVA-1.5 due to the unavailability of a reproducible recipe of LLaVA-NeXT.
• Figure 5: Different speed-performance trade-offs under different $p$ and $r$. We evaluate different $p=\{50,60,70,80,90,95\}\%$ by fixing $r$ to 10$\%$, and conversely, evaluate different $r=\{50,40,30,20,10,5,0\}\%$ by fixing $p$ to 90$\%$. When $r=0\%$, it degenerates to NaivePrune. Speedups are measured on a single NVIDIA L40.