Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

BiPrompt: Bilateral Prompt Optimization for Visual and Textual Debiasing in Vision-Language Models

Sunny Gupta, Shounak Das, Amit Sethi

TL;DR

BiPrompt tackles the vulnerability of vision-language models to spurious cues by jointly debiasing visual and textual modalities at test time. It introduces Balanced Prompt Normalization for text isotropy and Structured Spurious-Region Erasure guided by Grad-CAM to suppress background cues, all optimized under a joint objective without weight updates. Empirical results across real-world OOD and simulated bias benchmarks show consistent improvements in both average and worst-group accuracy, across multiple backbones. This work offers a lightweight, practical path toward trustworthy, causally grounded vision-language adaptation under distribution shifts.

Abstract

Vision language foundation models such as CLIP exhibit impressive zero-shot generalization yet remain vulnerable to spurious correlations across visual and textual modalities. Existing debiasing approaches often address a single modality either visual or textual leading to partial robustness and unstable adaptation under distribution shifts. We propose a bilateral prompt optimization framework (BiPrompt) that simultaneously mitigates non-causal feature reliance in both modalities during test-time adaptation. On the visual side, it employs structured attention-guided erasure to suppress background activations and enforce orthogonal prediction consistency between causal and spurious regions. On the textual side, it introduces balanced prompt normalization, a learnable re-centering mechanism that aligns class embeddings toward an isotropic semantic space. Together, these modules jointly minimize conditional mutual information between spurious cues and predictions, steering the model toward causal, domain invariant reasoning without retraining or domain supervision. Extensive evaluations on real-world and synthetic bias benchmarks demonstrate consistent improvements in both average and worst-group accuracies over prior test-time debiasing methods, establishing a lightweight yet effective path toward trustworthy and causally grounded vision-language adaptation.

BiPrompt: Bilateral Prompt Optimization for Visual and Textual Debiasing in Vision-Language Models

TL;DR

BiPrompt tackles the vulnerability of vision-language models to spurious cues by jointly debiasing visual and textual modalities at test time. It introduces Balanced Prompt Normalization for text isotropy and Structured Spurious-Region Erasure guided by Grad-CAM to suppress background cues, all optimized under a joint objective without weight updates. Empirical results across real-world OOD and simulated bias benchmarks show consistent improvements in both average and worst-group accuracy, across multiple backbones. This work offers a lightweight, practical path toward trustworthy, causally grounded vision-language adaptation under distribution shifts.

Abstract

Vision language foundation models such as CLIP exhibit impressive zero-shot generalization yet remain vulnerable to spurious correlations across visual and textual modalities. Existing debiasing approaches often address a single modality either visual or textual leading to partial robustness and unstable adaptation under distribution shifts. We propose a bilateral prompt optimization framework (BiPrompt) that simultaneously mitigates non-causal feature reliance in both modalities during test-time adaptation. On the visual side, it employs structured attention-guided erasure to suppress background activations and enforce orthogonal prediction consistency between causal and spurious regions. On the textual side, it introduces balanced prompt normalization, a learnable re-centering mechanism that aligns class embeddings toward an isotropic semantic space. Together, these modules jointly minimize conditional mutual information between spurious cues and predictions, steering the model toward causal, domain invariant reasoning without retraining or domain supervision. Extensive evaluations on real-world and synthetic bias benchmarks demonstrate consistent improvements in both average and worst-group accuracies over prior test-time debiasing methods, establishing a lightweight yet effective path toward trustworthy and causally grounded vision-language adaptation.
Paper Structure (15 sections, 9 equations, 1 figure, 3 tables)

This paper contains 15 sections, 9 equations, 1 figure, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. (a) Balanced Prompt Normalization.
  5. (b) Structured Spurious-Region Erasure.
  6. (c) Overall Objective.
  7. Optimization and Inference
  8. Results
  9. Experimental Setup
  10. Datasets.
  11. Baselines and Backbone.
  12. Results on Real-World OOD Scenarios
  13. Results on Simulated Spurious-Bias Scenarios
  14. Effectiveness Across Different Model Architectures
  15. Conclusion

Figures (1)

  • Figure 1: Framework of BiPrompt. Given a test sample, BiPrompt performs bilateral debiasing by jointly optimizing visual and textual representations. Structured attention-guided erasure suppresses spurious visual cues, while balanced prompt normalization aligns textual embeddings. The optimized bilateral prompt is then used for zero-shot inference, yielding causal and robust predictions under distribution shift.