What Matters in Data Curation for Multimodal Reasoning? Insights from the DCVLR Challenge

TL;DR

The paper analyzes data curation for multimodal reasoning under the DCVLR challenge by fixing the model and training recipe to isolate dataset effects. It shows that difficulty-based filtering on a well-aligned base dataset is the dominant driver of performance gains, while simply increasing data size mainly stabilizes results, and common diversity or synthetic augmentation strategies offer little to no benefit or can harm. Starting from Walton Multimodal Cold Start and using a 1,000-example curated set, the study demonstrates a saturation-regime behavior where alignment and example difficulty trump data volume. The findings have implications for interpreting DCVLR benchmarks and for designing data curation practices in reasoning-focused multimodal systems, suggesting that careful, difficulty-aware filtering on aligned data may be more impactful than large-scale or diversity-driven approaches under fixed-training regimes.

Abstract

We study data curation for multimodal reasoning through the NeurIPS 2025 Data Curation for Vision-Language Reasoning (DCVLR) challenge, which isolates dataset selection by fixing the model and training protocol. Using a compact curated dataset derived primarily from Walton Multimodal Cold Start, our submission placed first in the challenge. Through post-competition ablations, we show that difficulty-based example selection on an aligned base dataset is the dominant driver of performance gains. Increasing dataset size does not reliably improve mean accuracy under the fixed training recipe, but mainly reduces run-to-run variance, while commonly used diversity and synthetic augmentation heuristics provide no additional benefit and often degrade performance. These results characterize DCVLR as a saturation-regime evaluation and highlight the central role of alignment and difficulty in data-efficient multimodal reasoning.

Figures (8)

• Figure 1: Overview of our dataset curation pipeline for the submission, from base datasets through filtering, augmentation, and final sampling.
• Figure 2: DCVLR baseline curation strategies grouped into synthesis-based and filtration-based approaches. The Walton Multi-modal Cold Start filtration baseline provides a strong bootstrap corpus and motivates our difficulty-driven refinement.
• Figure 3: Embedding projection via PCA for LiveXivTQA and three DCVLR baseline datasets (Walton, MM-Open-R1, MM-MathInstruct), using Qwen2.5-VL-7B-Instruct representations.
• Figure 4: Accuracy vs. dataset size for overall performance, the aligned LiveXivTQA benchmark, and the aggregate of non-LiveXivTQA benchmarks. Mean accuracy plateaus beyond 1k samples, while variance decreases with scale.
• Figure 5: Diversity-oriented ablations on the Walton base dataset. Across clustering, category balancing, and related heuristics, none improve upon difficulty-only filtering.