Empowering Reliable Visual-Centric Instruction Following in MLLMs

TL;DR

This work tackles multimodal instruction-following by introducing a vision-centric data and evaluation framework. It presents VC-IFEngine to generate visually grounded tasks and constraints, builds VC-IFInstruct for supervised fine-tuning and VC-IFDPO for preference optimization, and introduces VC-IFEval with a hybrid evaluation that isolates the contribution of visual inputs. Empirical results show that training on these vision-grounded datasets improves instruction adherence in both visual and textual contexts and maintains general visual understanding. The framework offers a scalable path to more reliable and visually grounded MLLMs, with implications for trustworthy deployment in real-world settings.

Abstract

Evaluating the instruction-following (IF) capabilities of Multimodal Large Language Models (MLLMs) is essential for rigorously assessing how faithfully model outputs adhere to user-specified intentions. Nevertheless, existing benchmarks for evaluating MLLMs' instruction-following capability primarily focus on verbal instructions in the textual modality. These limitations hinder a thorough analysis of instruction-following capabilities, as they overlook the implicit constraints embedded in the semantically rich visual modality. To address this gap, we introduce VC-IFEval, a new benchmark accompanied by a systematically constructed dataset that evaluates MLLMs' instruction-following ability under multimodal settings. Our benchmark systematically incorporates vision-dependent constraints into instruction design, enabling a more rigorous and fine-grained assessment of how well MLLMs align their outputs with both visual input and textual instructions. Furthermore, by fine-tuning MLLMs on our dataset, we achieve substantial gains in visual instruction-following accuracy and adherence. Through extensive evaluation across representative MLLMs, we provide new insights into the strengths and limitations of current models.

Figures (13)

• Figure 1: We remove the visual input and prompt the model separately with textual and visual instructions. The response generated under the textual instruction is visually inconsistent with the image, yet it can still pass the instruction-following judgment. In contrast, the response generated under the visual instruction is visually inconsistent with the image and is thus judged by the model as not following the instruction.
• Figure 2: Overall pipeline of VC-IFEngine. The framework operates in three stages: (1) Image Filter, which removes visually uninformative samples with low semantics or saliency; (2) Task Generation, where we refine existing annotations or sample new tasks from an external pool depending on whether images contain predefined tasks; and (3) Constraint Generation, which selects and customizes constraints from a predefined pool and ensures their coherence through a quality-judging step. The resulting visually grounded tasks and validated constraints form the foundation for constructing VC-IFInstruct, VC-IFDPO, and VC-IFEval, enabling fine-grained multimodal instruction generation and reliable evaluation.
• Figure 3: Illustration of SFT and DPO data settings
• Figure 4: Distribution of IC9600 information-density scores across sampled images from Allava, Visual Genome, and LLaVA-Instruct-150k.
• Figure 5: An example of MM-IFEval limitation, where the model follows textual constraints without visual grounding.