Sycophancy in Vision-Language Models: A Systematic Analysis and an Inference-Time Mitigation Framework

TL;DR

The paper tackles sycophancy in large vision-language models by introducing an inference-time, model-agnostic mitigation framework (ITSM). ITSM combines a query neutralizer, sycophancy-aware contrastive decoding, and adaptive logits refinement (adaptive plausibility filter and query sentiment scaler) to reduce prompt-induced bias while preserving factual multimodal grounding. Extensive experiments across POPE, AMBER, RealworldQA, ScienceQA, MM-Vet, and the MME benchmark show ITSM consistently mitigates sycophancy better than baseline approaches, often matching or exceeding neutral-prompt performance. The framework demonstrates robustness across multiple LVLMs and datasets, with a moderate computational overhead that can be mitigated via system design. Overall, ITSM offers a practical, transferable approach toward trustworthy multimodal reasoning without retraining models.

Abstract

Large Vision-Language Models (LVLMs) have shown significant capability in vision-language understanding. However, one critical issue that persists in these models is sycophancy, where models are unduly influenced by leading or deceptive prompts, resulting in biased outputs and hallucinations. Despite the rapid development of LVLMs, evaluating and mitigating sycophancy remains largely under-explored. In this work, we fill this gap by systematically analyzing sycophancy across multiple vision-language benchmarks and propose an inference-time mitigation framework. We curate leading queries and quantify the susceptibility of state-of-the-art LVLMs to prompt-induced bias, revealing consistent performance degradation and instability across models and tasks. Our analysis further uncovers model-specific behavioral traits, such as sentiment sensitivity and prediction polarity shifts under sycophancy. To mitigate these issues, we propose a training-free, model-agnostic framework that operates entirely at inference time. Our approach first employs a query neutralizer, leveraging an language model to suppress implicit sycophantic bias in user queries. We then introduce a sycophancy-aware contrastive decoding mechanism that dynamically recalibrates token-level output distributions by contrasting responses to neutralized and leading queries. Finally, an adaptive logits refinement module further modifies the contrasted logits by integrating both a adaptive plausibility filter and query sentiment scaler, ensuring coherent and robust generation. Extensive experiments demonstrate that this framework effectively mitigates sycophancy across all evaluated models, while maintaining performance on neutral prompts. Our results suggest that sycophancy in LVLMs is a general and urgent challenge, and that inference-time strategies offer a promising path toward trustworthy multimodal reasoning.

Figures (9)

• Figure 1: Illustration of sycophantic behavior in LVLMs. Compared to neutral questions (left), leading queries (right) introduce subtle biases that cause the model to hallucinate or flip its answers, prioritizing linguistic cues over visual grounding.
• Figure 2: Dataset Extending for evaluating sycophancy.
• Figure 3: Overall architecture of the proposed inference-time sycophancy mitigation framework for LVLMs. The framework consists of three key modules: (1) a Query Neutralizer that transforms leading user queries into neutral forms via an LLM backbone; (2) Contrastive Decoding, which computes the difference between token distributions from neutralized and leading queries, with adaptive coefficients dynamically tuned based on divergence and entropy; and (3) Adaptive Logits Refinement, where a Query Sentiment Scaler and Plausibility Filter jointly calibrate the contrasted logits to suppress sycophantic bias and ensure fluent, factual outputs. Key modules and data flow are illustrated; red highlights indicate sycophancy-affected paths, while green highlights indicate factual, corrected predictions.
• Figure 4: POPE evaluation results. For each model, the darker bars represent accuracy under original neutral queries, while the lighter bars correspond to results under leading queries. Solid lines indicate changes in F1 Score. A clear decline in both accuracy and F1 Score under leading queries highlights the presence of sycophancy across all evaluated models.
• Figure 5: AMBER evaluation results under attribute and relation hallucination. For each model, different colors represent attribute-based and relation-based queries, while color depth differentiates between neutral queries (darker bars) and leading queries (lighter bars). Solid lines show the change of F1 Score. The performance drop from neutral to leading queries reflects the impact of sycophancy under different hallucination types.