FiLoRA: Focus-and-Ignore LoRA for Controllable Feature Reliance

TL;DR

FiLoRA addresses how multimodal foundation models rely on internal feature groups and asks whether this reliance can be explicitly controlled without changing task semantics. It introduces a grouped LoRA framework with instruction-conditioned gating to steer internal computation along feature paths corresponding to core and spurious cues while keeping the base model fixed. Across text–image and audio–visual benchmarks, FiLoRA demonstrates consistent, causal shifts in feature reliance and improved robustness to spurious interventions, without altering labels or objectives. The work provides a principled mechanism for regulating reliance beyond correlation driven learning and suggests paths toward more interpretable and robust multimodal systems.

Abstract

Multimodal foundation models integrate heterogeneous signals across modalities, yet it remains poorly understood how their predictions depend on specific internal feature groups and whether such reliance can be deliberately controlled. Existing studies of shortcut and spurious behavior largely rely on post hoc analyses or feature removal, offering limited insight into whether reliance can be modulated without altering task semantics. We introduce FiLoRA (Focus-and-Ignore LoRA), an instruction-conditioned, parameter-efficient adaptation framework that enables explicit control over internal feature reliance while keeping the predictive objective fixed. FiLoRA decomposes adaptation into feature group-aligned LoRA modules and applies instruction-conditioned gating, allowing natural language instructions to act as computation-level control signals rather than task redefinitions. Across text--image and audio--visual benchmarks, we show that instruction-conditioned gating induces consistent and causal shifts in internal computation, selectively amplifying or suppressing core and spurious feature groups without modifying the label space or training objective. Further analyses demonstrate that FiLoRA yields improved robustness under spurious feature interventions, revealing a principled mechanism to regulate reliance beyond correlation-driven learning.

Figures (17)

• Figure 1: Feature reliance control mechanisms in multimodal models. (a) Standard instruction tuning aligns surface-level behavior without explicitly modulating internal computation. (b) Standard LoRA applies uniform parameter updates without path-level control. (c) FiLoRA (ours) enables instruction-conditioned control by gating feature group-aligned LoRA modules. In a movie genre classification example, FiLoRA suppresses reliance on spurious visual scene cues and grounds predictions in the textual plot, as specified by the instruction.
• Figure 2: Average gate activations aggregated over taxonomy-aligned feature families under different instruction conditions. Focus-core instructions amplify semantic and narrative features, while ignore-spurious instructions suppress demographic and appearance-related cues, demonstrating structured instruction-conditioned gate control.
• Figure 3: Prediction sensitivity to feature group gate perturbations under different instruction conditions. Reliance Sensitivity (RS) is measured as the average absolute change in the log-probability of the output.
• Figure 4: Decision stability under spurious feature removal (left) and performance degradation under increasing spurious feature suppression (right). Baselines are evaluated on dataset-appropriate backbones (Qwen2.5-VL for MM-IMDb and Qwen2.5-Omni for CREMA-D/RAVDESS). Full fine-tuning, LoRA, and P-only (prompt-only) baselines exhibit substantial instability and sharp degradation as spurious cues are removed, whereas FiLoRA preserves higher agreement and degrades more gradually, indicating improved robustness through reliance modulation. Full results are shown in Appendix \ref{['appendix:robustness_dataset']}.
• Figure 5: Dataset-wise reliance profiles measured by average RS. Each point reports prediction sensitivity to gate perturbations for core and spurious feature groups under neutral instructions.