iGVLM: Dynamic Instruction-Guided Vision Encoding for Question-Aware Multimodal Understanding

TL;DR

iGVLM introduces a decoupled dual-branch architecture that enables a smooth transition from general-purpose perception to instruction-aware reasoning while maintaining the structural integrity and stability of pre-trained visual priors, offering a plug-and-play paradigm for bridging passive perception and active reasoning.

Abstract

Despite the success of Large Vision--Language Models (LVLMs), most existing architectures suffer from a representation bottleneck: they rely on static, instruction-agnostic vision encoders whose visual representations are utilized in an invariant manner across different textual tasks. This rigidity hinders fine-grained reasoning where task-specific visual cues are critical. To address this issue, we propose iGVLM, a general framework for instruction-guided visual modulation. iGVLM introduces a decoupled dual-branch architecture: a frozen representation branch that preserves task-agnostic visual representations learned during pre-training, and a dynamic conditioning branch that performs affine feature modulation via Adaptive Layer Normalization (AdaLN). This design enables a smooth transition from general-purpose perception to instruction-aware reasoning while maintaining the structural integrity and stability of pre-trained visual priors. Beyond standard benchmarks, we introduce MM4, a controlled diagnostic probe for quantifying logical consistency under multi-query, multi-instruction settings. Extensive results show that iGVLM consistently enhances instruction sensitivity across diverse language backbones, offering a plug-and-play paradigm for bridging passive perception and active reasoning.

Figures (10)

• Figure 1: Visualization of vision features. We employ Grad-CAM to visualize the vision encoders of the two branches in iGVLM, highlighting the regions most relevant to the correct answer. As shown in the figure, the instruction-guided branch distinctly focuses on areas that are more closely associated with the correct answer.
• Figure 2: (a): The proposed iGVLM architecture. The Text Encoder extracts features from the input instructions to guide the Vision Encoder, enabling dynamic modulation of visual representations. These instruction-conditioned features are then fused with static visual features. The fused representation is aligned with a Large Language Model (LLM) to generate responses. The illustrated example comes from a real-world VQA scenario rather than the MM4 benchmark. (b): AdaLN-Modified ViT. We leverage textual information to modulate the multi-head attention and MLP modules within the ViT the AdaLN adapter, enabling instruction-aware adjustment of visual attention.
• Figure 3: Representative Examples from MM4. Deceptively simple questions demand multiperspective reasoning. Correct answers (highlighted) are uniformly distributed across options to prevent positional bias. More examples can be found in \ref{['sec:cases']}.
• Figure 4: Different questions similarity heatmap.
• Figure :