Modality-Balancing Preference Optimization of Large Multimodal Models by Adversarial Negative Mining

TL;DR

MBPO addresses modality imbalance in large multimodal models by combining offline adversarial negative mining that enforces visual grounding with online, verifiable rewards optimized via Group Relative Policy Optimization. It introduces the Image Information Gain metric to quantify visual reliance and uses adversarial perturbations to create hard negatives, forcing the model to ground outputs in vision. The online component leverages closed-ended visual instruction data with verified rewards to adapt to distribution shifts during training, yielding robust alignment. Across diverse vision-language benchmarks, MBPO improves multimodal alignment and substantially reduces hallucinations compared with existing preference-learning approaches, demonstrating practical gains in grounding LMMs to visual evidence.

Abstract

The task adaptation and alignment of Large Multimodal Models (LMMs) have been significantly advanced by instruction tuning and further strengthened by recent preference optimization. Yet, most LMMs still suffer from severe modality imbalance during reasoning, i.e., outweighing language prior biases over visual inputs, which bottlenecks their generalization to downstream tasks and causes hallucinations. However, existing preference optimization approaches for LMMs do not focus on restraining the internal biases of their Large Language Model (LLM) backbones when curating the training data. Moreover, they heavily rely on offline data and lack the capacity to explore diverse responses adaptive to dynamic distributional shifts during training. Meanwhile, Group Relative Policy Optimization (GRPO), a recent method using online-generated data and verified rewards to improve reasoning capabilities, remains largely underexplored in LMM alignment. In this paper, we propose a novel preference learning framework, Modality-Balancing Preference Optimization (MBPO), to address the modality imbalance in LMMs. MBPO constructs a more effective offline preference dataset by generating hard negatives, i.e., rejected responses misled by LLM biases due to limited usage of visual information, through adversarial perturbation of input images. Moreover, MBPO leverages the easy-to-verify nature of close-ended tasks to generate online responses with verified rewards. GRPO is then employed to train the model with offline-online hybrid data. Extensive experiments demonstrate that MBPO can enhance LMM performance on challenging vision-language tasks and effectively reduce hallucinations.

Figures (8)

• Figure 1: Overview of MBPO framework. To construct the offline preference dataset, we generate adversarial perturbations for each input image to minimize the output probability of the chosen response. Rejected responses are then generated using these adversarially perturbed images. This process amplifies modality imbalance, causing the LMM to rely more heavily on the prior biases of its LLM backbone rather than the visual information. In parallel, MBPO incorporates an online dataset composed of closed-ended examples, where response correctness can be easily verified. During training, the LMM generates multiple responses, and verified rewards are assigned based on their correctness. Finally, the offline and online datasets are combined to optimize the LMM using the MBPO loss in a hybrid training paradigm.
• Figure 2: An example comparing model responses of the image with adversarial noise and random noise. The prior bias from LLM is marked in red.
• Figure 3: IIG of chosen and rejected responses change along with the training.
• Figure 4: Reward of the online closed-end data changes along with the training.
• Figure 5: An example comparing model responses of the image with adversarial noise and random noise. The prior bias from LLM is marked in red.