AMPS: Adaptive Modality Preference Steering via Functional Entropy

TL;DR

AMPS presents Modality Contribution Score (MCS), a diagnostic of per-modality information contribution, and an instance-aware Adaptive Modality Preference Steering (AMPS) framework that scales steering strength based on sample sensitivity. The approach combines a Monte Carlo-based Modality Contribution Ratio (MCR) estimator with a learnable steering module to adaptively adjust cross-modal bias, reducing inference disruption while achieving effective modality control. Across MC^2 and multiple model scales, AMPS consistently outperforms prompt engineering, static steering, and prior adaptive controllers, with stronger modality shift and lower generation collapse. The work advances robust, context-aware modulation of multimodal reasoning, enabling safer and more reliable steering in real-world deployments.

Abstract

Multimodal Large Language Models (MLLMs) often exhibit significant modality preference, which is a tendency to favor one modality over another. Depending on the input, they may over-rely on linguistic priors relative to visual evidence, or conversely over-attend to visually salient but facts in textual contexts. Prior work has applied a uniform steering intensity to adjust the modality preference of MLLMs. However, strong steering can impair standard inference and increase error rates, whereas weak steering is often ineffective. In addition, because steering sensitivity varies substantially across multimodal instances, a single global strength is difficult to calibrate. To address this limitation with minimal disruption to inference, we introduce an instance-aware diagnostic metric that quantifies each modality's information contribution and reveals sample-specific susceptibility to steering. Building on these insights, we propose a scaling strategy that reduces steering for sensitive samples and a learnable module that infers scaling patterns, enabling instance-aware control of modality preference. Experimental results show that our instance-aware steering outperforms conventional steering in modulating modality preference, achieving effective adjustment while keeping generation error rates low.

Figures (7)

• Figure 1: AMPS handles failure cases in modality preference steering parekh2025learning with uniform steering intensity.
• Figure 2: Mean Steering with different steering intensity.
• Figure 3: MCR Measurement Pipeline, in this figure the $K_{T_i}$ and $K_{V_j}$ means the corresponding to the index of the i-th text or the index of the j-th visual token in K cache; $V_{T_i}$ and $V_{V_j}$ means the corresponding to the index of the i-th text or index of the j-th visual token in V cache.
• Figure 4: AMPS Training and Application Pipeline
• Figure 5: Sensitivity scores of different models across different tasks.