Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Scaling Test-Time Robustness of Vision-Language Models via Self-Critical Inference Framework

Kaihua Tang, Jiaxin Qi, Jinli Ou, Yuhua Zheng, Jianqiang Huang

TL;DR

A novel Self-Critical Inference (SCI) framework is proposed that extends Visual Contrastive Decoding by conducting multi-round counterfactual reasoning through both textual and visual perturbations and shows that SCI consistently outperforms baseline methods on DRBench, and that increasing the number of inference rounds further boosts robustness beyond existing single-step counterfactual reasoning methods.

Abstract

The emergence of Large Language Models (LLMs) has driven rapid progress in multi-modal learning, particularly in the development of Large Vision-Language Models (LVLMs). However, existing LVLM training paradigms place excessive reliance on the LLM component, giving rise to two critical robustness challenges: language bias and language sensitivity. To address both issues simultaneously, we propose a novel Self-Critical Inference (SCI) framework that extends Visual Contrastive Decoding by conducting multi-round counterfactual reasoning through both textual and visual perturbations. This process further introduces a new strategy for improving robustness by scaling the number of counterfactual rounds. Moreover, we also observe that failure cases of LVLMs differ significantly across models, indicating that fixed robustness benchmarks may not be able to capture the true reliability of LVLMs. To this end, we propose the Dynamic Robustness Benchmark (DRBench), a model-specific evaluation framework targeting both language bias and sensitivity issues. Extensive experiments show that SCI consistently outperforms baseline methods on DRBench, and that increasing the number of inference rounds further boosts robustness beyond existing single-step counterfactual reasoning methods.

Scaling Test-Time Robustness of Vision-Language Models via Self-Critical Inference Framework

TL;DR

A novel Self-Critical Inference (SCI) framework is proposed that extends Visual Contrastive Decoding by conducting multi-round counterfactual reasoning through both textual and visual perturbations and shows that SCI consistently outperforms baseline methods on DRBench, and that increasing the number of inference rounds further boosts robustness beyond existing single-step counterfactual reasoning methods.

Abstract

The emergence of Large Language Models (LLMs) has driven rapid progress in multi-modal learning, particularly in the development of Large Vision-Language Models (LVLMs). However, existing LVLM training paradigms place excessive reliance on the LLM component, giving rise to two critical robustness challenges: language bias and language sensitivity. To address both issues simultaneously, we propose a novel Self-Critical Inference (SCI) framework that extends Visual Contrastive Decoding by conducting multi-round counterfactual reasoning through both textual and visual perturbations. This process further introduces a new strategy for improving robustness by scaling the number of counterfactual rounds. Moreover, we also observe that failure cases of LVLMs differ significantly across models, indicating that fixed robustness benchmarks may not be able to capture the true reliability of LVLMs. To this end, we propose the Dynamic Robustness Benchmark (DRBench), a model-specific evaluation framework targeting both language bias and sensitivity issues. Extensive experiments show that SCI consistently outperforms baseline methods on DRBench, and that increasing the number of inference rounds further boosts robustness beyond existing single-step counterfactual reasoning methods.
Paper Structure (16 sections, 8 equations, 5 figures, 10 tables)

This paper contains 16 sections, 8 equations, 5 figures, 10 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Preliminaries
  5. Self-Critical Inference Framework
  6. Dynamic Robustness Benchmark
  7. Experiments
  8. Benchmark Settings
  9. Implementation Details
  10. Experimental Results
  11. Discussions
  12. Conclusion
  13. Appendix
  14. Adaptive Plausibility Constraint
  15. Generation of Counterfactual Inputs
  16. ...and 1 more sections

Figures (5)

  • Figure 1: (a) and (b) are real DRBench examples suffering from language sensitivity and bias issues; (c) shows the overall proportion of different types of non-robust samples across all 6 datasets under two commonly used LVLMs; (d) demonstrates a novel test-time scaling strategy of robustness regarding the increased counterfactual rounds in the proposed SCI.
  • Figure 2: Investigating the test-time scaling effect on robustness with respect to the number of inference rounds on B/S/BS subsets across different question types and LVLMs.
  • Figure 3: The list of all TC-V1 prompts that add an additional system prompt instructing the model to focus on image details.
  • Figure 4: The list of all TC-V2 prompts that further modify the system prompt’s language from English to Chinese or vice versa.
  • Figure 5: The list of all TC-V3 prompts that inject identity information by prompting the model to respond as a clever student.