MergeMix: A Unified Augmentation Paradigm for Visual and Multi-Modal Understanding

TL;DR

MergeMix presents a unified data-augmentation framework that bridges supervised fine-tuning and reinforcement-learning style preference optimization for multi-modal large language models. It leverages token-merge based Mixup (ToMe-informed) to create contextually aligned mixed images and labels, and introduces a Gaussian re-sampling of mixing ratios together with a mixed SimPO objective to steer preference alignment. The approach yields consistent improvements in image classification and MLLM benchmarks, while enhancing calibration and inference efficiency via token merging. This work offers a scalable, stable paradigm for multi-modal alignment that neatly integrates augmentation, efficiency, and human-preference tuning, with clear paths for extending mixups to text and learning the token-merging policy.

Abstract

Vision-language alignment in multi-modal large language models (MLLMs) relies on supervised fine-tuning (SFT) or reinforcement learning (RL). To align multi-modal large language models (MLLMs) in the post-training stage, supervised fine-tuning (SFT) is a stable choice but requires human annotations and lacks task generalizations, while Reinforcement Learning (RL) searches for better answers from reward signals but suffers from computational overhead and instability. To achieve balance among scalability, efficiency, and alignment generalizations, we propose MergeMix, a unified paradigm that bridges SFT and RL with an efficient Token Merge based Mixup augmentation. As for the Mixup policy, we generate contextual aligned mixed images with the corresponding labels according to the merged attention maps with cluster regions. Then, we enhance the preference-driven paradigm for MLLMs by building preference pairs with raw images and MergeMix-generated ones and optimizing the soft preference margin with the mixed SimPO loss. Extensive experiments demonstrate that MergeMix not only achieves dominant classification accuracy as an augmentation method but also improves generalization abilities and alignment of MLLMs, providing a new learning paradigm for preference alignment with training efficiency and stability.

Figures (17)

• Figure 1: Efficiency and for MergeMix: (a) The training time vs. accuracy of mixup methods with the DeiT-Small model. (b) The image classification Top-1 accuracy vs. training epochs of different mixup methods on the CIFAR100 dataset with the DeiT-Tiny model. (c) The radar plot of the results on part VQA tasks by LLaVA-7B, LLaVA with SFT, and MergeMix.
• Figure 2: The overall of the two scenarios of MergeMix:(a) MergeMix for Image Classification: The image is processed by the ToMe encoder, with Attention Score Recovery and TopK sampling to generate the corresponding class prediction. (b) MergeMix for MLLM: Preference pairs are encoded by the vision model with token merging, and the LLM decoder generates response text for the loser and winner, optimized via a ranking loss.
• Figure 3: Overall illustration of MergeMix for MLLM.(a) MergeMix performs attention-based mask mixing guided by the ToMe Vision Encoder, recovering token attention scores and generating a mixed image through an augmenter. Specifically, Token Merging hierarchically merges visual tokens via Bipartite Soft Matching (BSM) to enhance efficiency, which is trained with both the SFT and ranking losses. (b) Case study of preference data generated by MergeMix with LLaVA-v1.5-7B.
• Figure 4: The confidence plots of mixup variants and MergeMix on the CIFAR100 dataset using DeiT-Tiny and ViT-Small. The red line indicates the expected prediction tendency.
• Figure 5: The calibration results of LLaVA-v1.5-7B on POPE, ScienceVQA$^I$, GQA & SEED$^I$. $\text{rl}$ denotes training with ranking loss.