Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MVTamperBench: Evaluating Robustness of Vision-Language Models

Amit Agarwal, Srikant Panda, Angeline Charles, Bhargava Kumar, Hitesh Patel, Priyaranjan Pattnayak, Taki Hasan Rafi, Tejaswini Kumar, Hansa Meghwani, Karan Gupta, Dong-Kyu Chae

TL;DR

To address the lack of robust evaluation under real-world video tampering, this work introduces MVTamperBench, a benchmark that challenges MLLMs with five tampering techniques applied to thousands of videos across 19 tasks. It employs a unified multiple-choice detection framework and evaluates 45 models from 15+ families, revealing substantial variability in tampering resilience that is not explained by model size. Key findings include that larger parameter counts do not guarantee robustness and that certain architectures achieve strong, consistent performance. The work provides open-source code and data to accelerate development of tamper-resilient video-language understanding for safety-critical applications like moderation and misinformation detection.

Abstract

Multimodal Large Language Models (MLLMs), are recent advancement of Vision-Language Models (VLMs) that have driven major advances in video understanding. However, their vulnerability to adversarial tampering and manipulations remains underexplored. To address this gap, we introduce \textbf{MVTamperBench}, a benchmark that systematically evaluates MLLM robustness against five prevalent tampering techniques: rotation, masking, substitution, repetition, and dropping; based on real-world visual tampering scenarios such as surveillance interference, social media content edits, and misinformation injection. MVTamperBench comprises ~3.4K original videos, expanded into over ~17K tampered clips covering 19 distinct video manipulation tasks. This benchmark challenges models to detect manipulations in spatial and temporal coherence. We evaluate 45 recent MLLMs from 15+ model families. We reveal substantial variability in resilience across tampering types and show that larger parameter counts do not necessarily guarantee robustness. MVTamperBench sets a new benchmark for developing tamper-resilient MLLM in safety-critical applications, including detecting clickbait, preventing harmful content distribution, and enforcing policies on media platforms. We release all code, data, and benchmark to foster open research in trustworthy video understanding. Code: https://amitbcp.github.io/MVTamperBench/ Data: https://huggingface.co/datasets/Srikant86/MVTamperBench

MVTamperBench: Evaluating Robustness of Vision-Language Models

TL;DR

To address the lack of robust evaluation under real-world video tampering, this work introduces MVTamperBench, a benchmark that challenges MLLMs with five tampering techniques applied to thousands of videos across 19 tasks. It employs a unified multiple-choice detection framework and evaluates 45 models from 15+ families, revealing substantial variability in tampering resilience that is not explained by model size. Key findings include that larger parameter counts do not guarantee robustness and that certain architectures achieve strong, consistent performance. The work provides open-source code and data to accelerate development of tamper-resilient video-language understanding for safety-critical applications like moderation and misinformation detection.

Abstract

Multimodal Large Language Models (MLLMs), are recent advancement of Vision-Language Models (VLMs) that have driven major advances in video understanding. However, their vulnerability to adversarial tampering and manipulations remains underexplored. To address this gap, we introduce \textbf{MVTamperBench}, a benchmark that systematically evaluates MLLM robustness against five prevalent tampering techniques: rotation, masking, substitution, repetition, and dropping; based on real-world visual tampering scenarios such as surveillance interference, social media content edits, and misinformation injection. MVTamperBench comprises ~3.4K original videos, expanded into over ~17K tampered clips covering 19 distinct video manipulation tasks. This benchmark challenges models to detect manipulations in spatial and temporal coherence. We evaluate 45 recent MLLMs from 15+ model families. We reveal substantial variability in resilience across tampering types and show that larger parameter counts do not necessarily guarantee robustness. MVTamperBench sets a new benchmark for developing tamper-resilient MLLM in safety-critical applications, including detecting clickbait, preventing harmful content distribution, and enforcing policies on media platforms. We release all code, data, and benchmark to foster open research in trustworthy video understanding. Code: https://amitbcp.github.io/MVTamperBench/ Data: https://huggingface.co/datasets/Srikant86/MVTamperBench
Paper Structure (63 sections, 16 figures, 7 tables)

This paper contains 63 sections, 16 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Image-based Understanding
  4. Video-based Understanding
  5. MVTamperBench
  6. Benchmark Construction
  7. Tampering Techniques
  8. Design & Implementation
  9. Tampering Duration (1 Second).
  10. Tampering Location (Middle).
  11. Substitution Source.
  12. Modular & Scalable Framework.
  13. Dataset Scope and Statistics
  14. Summary
  15. Experiments
  16. ...and 48 more sections

Figures (16)

  • Figure 1: Illustration of the five video frame tampering techniques. Each row shows a specific tamper type applied to a video snippet, describing the respective impact on temporal/spatial coherence.
  • Figure 2: Distribution of video durations. Our dataset spans a broad range of durations, which can reflect varied real-world conditions.
  • Figure 3: F1 scores across models and tampering types. High-performing models are robust across all types.
  • Figure 4: Distribution of F1 (overall) scores across models.
  • Figure 5: Scatter plot showing no correlation between model size and overall F1 (Pearson r=0.05).
  • ...and 11 more figures