VideoSTF: Stress-Testing Output Repetition in Video Large Language Models

TL;DR

VideoSTF formalizes a stability-focused framework for VideoLLMs by introducing three $n$-gram–based repetition metrics ($RR$, $RI$, $IE$), a standardized 10,000-video testbed, and a temporal stressor library to probe repetition under controlled temporal transformations. Its large-scale evaluation across 10 VideoLLMs shows that output repetition is pervasive and highly sensitive to temporal perturbations, with simple black-box temporal transforms able to induce degeneration in many cases. The results expose a practical reliability and security risk in video-language systems, highlighting the need for stability-aware evaluation and robust design beyond decoding strategies. Overall, VideoSTF provides a principled benchmark and methodology for diagnosing and mitigating generation instability in video-conditioned language models.

Abstract

Video Large Language Models (VideoLLMs) have recently achieved strong performance in video understanding tasks. However, we identify a previously underexplored generation failure: severe output repetition, where models degenerate into self-reinforcing loops of repeated phrases or sentences. This failure mode is not captured by existing VideoLLM benchmarks, which focus primarily on task accuracy and factual correctness. We introduce VideoSTF, the first framework for systematically measuring and stress-testing output repetition in VideoLLMs. VideoSTF formalizes repetition using three complementary n-gram-based metrics and provides a standardized testbed of 10,000 diverse videos together with a library of controlled temporal transformations. Using VideoSTF, we conduct pervasive testing, temporal stress testing, and adversarial exploitation across 10 advanced VideoLLMs. We find that output repetition is widespread and, critically, highly sensitive to temporal perturbations of video inputs. Moreover, we show that simple temporal transformations can efficiently induce repetitive degeneration in a black-box setting, exposing output repetition as an exploitable security vulnerability. Our results reveal output repetition as a fundamental stability issue in modern VideoLLMs and motivate stability-aware evaluation for video-language systems. Our evaluation code and scripts are available at: https://github.com/yuxincao22/VideoSTF_benchmark.

Figures (13)

• Figure 1: Examples of normal and repetitive outputs.
• Figure 2: An overview of VideoSTF. (a) Framework Components. VideoSTF comprises three n-gram-based repetition metrics, a standardized testbed of 10,000 videos with diverse durations and content categories, and a library of controlled temporal stressors for applying temporal transformations. (b) Evaluation Protocols. VideoSTF assesses output repetition through three tests. Pervasive Testing reveals widespread repetition across VideoLLMs under different frame sampling settings across all three metrics. Temporal Stress Testing shows that temporal transformations amplify repetition. Adversarial Exploitation demonstrates that, through temporal transformations, videos with normal outputs can be efficiently induced to become repetitive with high success rates and few queries.
• Figure 3: RI (\ref{['eq:ri']}) and IE (\ref{['eq:ie']}) distributions under original and temporally transformed inputs across VideoLLMs.
• Figure 4: Repetition results of different VideoLLMs under three metrics across varying $n$.
• Figure 5: Examples of repetitive outputs generated by LLaVA-Video-7B-Qwen2 and LLaVA-Video-7B-Qwen2-Video-Only, with repeated phrases or sentences highlighted in different colors.