VACoT: Rethinking Visual Data Augmentation with VLMs

TL;DR

VACoT introduces a post-hoc visual augmentation framework for visual language models, enabling dynamic image transformations during inference to boost robustness against adversarial and fine-grained perception challenges. The method combines a structured augmentation set with agentic reinforcement learning (GRPO) and a three-stage training pipeline (knowledge SFT, API-format SFT, RL) to learn when and which augmentations to apply. Extensive evaluation across 13 perception benchmarks demonstrates notable improvements over baselines and competitive results with larger models, with AdvOCR showcasing robustness to adversarial OCR scenarios. The work also contributes AdvOCR as a challenging real-world OCR benchmark and demonstrates that efficient, learned augmentation policies can yield substantial perceptual gains without prohibitive training costs.

Abstract

While visual data augmentation remains a cornerstone for training robust vision models, it has received limited attention in visual language models (VLMs), which predominantly rely on large-scale real data acquisition or synthetic diversity. Consequently, they may struggle with basic perception tasks that conventional models handle reliably. Given the substantial cost of pre-training and fine-tuning VLMs, continue training on augmented data yields limited and diminishing returns. In this paper, we present Visual Augmentation Chain-of-Thought (VACoT), a framework that dynamically invokes image augmentations during model inference. By incorporating post-hoc transformations such as denoising, VACoT substantially improves robustness on challenging and out-of-distribution inputs, especially in OCR-related adversarial scenarios. Distinct from prior approaches limited to local cropping, VACoT integrates a structured collection of general visual augmentations, broadening the query image views while reducing training complexity and computational overhead with efficient agentic reinforcement learning. We propose a conditional reward scheme that encourages necessary augmentation while penalizing verbose responses, ensuring concise and effective reasoning in perception tasks. We demonstrate the superiority of VACoT with extensive experiments on 13 perception benchmarks and further introduce AdvOCR to highlight the generalization benefits of post-hoc visual augmentations in adversarial scenarios.

Figures (10)

• Figure 1: VACoT inference example. We address adversarial text recognition through iterative post-hoc visual augmentations.
• Figure 2: The overall architecture of VACoT. We leverage stop-words to achieve iterative post-hoc visual augmentation, providing more diverse image perspectives and higher-quality visual interactions compared to cropping-only agentic models.
• Figure 3: Illustration of pass@k data filter. The difficulty score is aligned with the number of correct answers for pass@k inference.
• Figure 4: Construction of visual augmentation trajectory data.
• Figure 5: Example visualizations of AdvOCR. It poses greater demands on fine-grained and adversarial perception capabilities.