Flattery in Motion: Benchmarking and Analyzing Sycophancy in Video-LLMs

TL;DR

This work introduces ViSE, the first benchmark specifically designed to evaluate sycophancy in Video-LLMs by extending linguistic bias concepts to dynamic video reasoning. It provides a curated dataset of 367 videos and 6,367 MCQs (with 141 longer videos annotated for 8 visual tasks) to probe seven sycophancy types under diverse prompt structures, quantified by Misleading Susceptibility Score $\text{MSS}$ and Correction Receptiveness Score $\text{CRS}$. Through extensive evaluation of six Video-LLMs (across nine variants), the study reveals model-dependent susceptibility patterns, with strong biases and complex question types amplifying sycophancy, and demonstrates two training-free mitigations: key-frame selection to improve grounding and representation steering to suppress sycophantic activations. These findings advance understanding of trustworthiness in multimodal video reasoning and provide practical, deployment-friendly strategies to curb misleading user influence in Video-LLMs.

Abstract

As video large language models (Video-LLMs) become increasingly integrated into real-world applications that demand grounded multimodal reasoning, ensuring their factual consistency and reliability is of critical importance. However, sycophancy, the tendency of these models to align with user input even when it contradicts the visual evidence, undermines their trustworthiness in such contexts. Current sycophancy research has largely overlooked its specific manifestations in the video-language domain, resulting in a notable absence of systematic benchmarks and targeted evaluations to understand how Video-LLMs respond under misleading user input. To fill this gap, we propose VISE (Video-LLM Sycophancy Benchmarking and Evaluation), the first benchmark designed to evaluate sycophantic behavior in state-of-the-art Video-LLMs across diverse question formats, prompt biases, and visual reasoning tasks. Specifically, VISE pioneeringly brings linguistic perspectives on sycophancy into the video domain, enabling fine-grained analysis across multiple sycophancy types and interaction patterns. Furthermore, we propose two potential training-free mitigation strategies, revealing potential paths for reducing sycophantic bias: (i) enhancing visual grounding through interpretable key-frame selection and (ii) steering model behavior away from sycophancy via targeted, inference-time intervention on its internal neural representations. Our code is available at https://github.com/William030422/Video-Sycophancy.

Figures (6)

• Figure 1: Left: Video Pool Curation: We prioritize samples exhibiting high MSS and low CRS (annotated with red dots), which reflect strong sycophantic tendencies with limited self-correction. Right: Dataset Composition: ViSE comprises videos of varying lengths and topics, accompanied by a broad spectrum of annotated questions. These include temporal, descriptive, and reasoning-based formats to comprehensively evaluate sycophantic behavior under diverse visual and linguistic conditions.
• Figure 2: Overview of sycophancy types and question formats in ViSE . We define four main sycophancy categories, each with specific question templates to probe distinct behaviors.
• Figure 3: Left: Average attention score for 9-frame input. Middel: Average attention score for 3 key-frame extraction under the same conditions. Right: Comparison of average attention score shifts across 100 pairs of strong bias feedback sycophancy cases, averaged over frames.
• Figure 4: Example 1 in ViSE .
• Figure 5: Example 2 in ViSE .