Perception Test 2025: Challenge Summary and a Unified VQA Extension

TL;DR

Perception Test 2025 advances multimodal video understanding by introducing a unified VQA interface that casts diverse perception tasks as multiple-choice video questions, enabling evaluation of unified models across long and short videos. The workshop introduces five tracks—unified video QA, joint object/point tracking, joint action/sound localisation, grounded video QA, and hour-long video QA—along with an interpretability track, and reports substantial gains for language-enabled tracks while revealing remaining gaps in non-language unified tasks. A key contribution is the Unified mc-VQA dataset extension (adding 1,842 five-option questions to the original set) and the use of in-painting and temporal cues to ground reasoning in the video stream. Overall, the results demonstrate rapid progress in video-language models, particularly for tasks requiring long-range reasoning and grounding, while underscoring the need for stronger unified approaches beyond language interfaces to close remaining performance gaps.

Abstract

The Third Perception Test challenge was organised as a full-day workshop alongside the IEEE/CVF International Conference on Computer Vision (ICCV) 2025. Its primary goal is to benchmark state-of-the-art video models and measure the progress in multimodal perception. This year, the workshop featured 2 guest tracks as well: KiVA (an image understanding challenge) and Physic-IQ (a video generation challenge). In this report, we summarise the results from the main Perception Test challenge, detailing both the existing tasks as well as novel additions to the benchmark. In this iteration, we placed an emphasis on task unification, as this poses a more challenging test for current SOTA multimodal models. The challenge included five consolidated tracks: unified video QA, unified object and point tracking, unified action and sound localisation, grounded video QA, and hour-long video QA, alongside an analysis and interpretability track that is still open for submissions. Notably, the unified video QA track introduced a novel subset that reformulates traditional perception tasks (such as point tracking and temporal action localisation) as multiple-choice video QA questions that video-language models can natively tackle. The unified object and point tracking merged the original object tracking and point tracking tasks, whereas the unified action and sound localisation merged the original temporal action localisation and temporal sound localisation tracks. Accordingly, we required competitors to use unified approaches rather than engineered pipelines with task-specific models. By proposing such a unified challenge, Perception Test 2025 highlights the significant difficulties existing models face when tackling diverse perception tasks through unified interfaces.

Figures (7)

• Figure 1: Top-1 accuracy of various VLMs at the moment of their first release compared to human baseline on the Perception Test multiple-choice video QA task. We include the results published by models' authors where available, otherwise we ran the models independently at the moment of their first release (GPT-4V, SeViLA, Flamingo).
• Figure 2: An example of each of the 4 new question types in the unified mc-VQA set (labeled a-d). We show sampled frames from the videos (some with visually in-painted annotations) that help define the task. The model must then choose the correct answer (highlighted in green) from the 5 options.
• Figure 3: Per-track performance improvement compared to baselines and compared to best models from previous years.
• Figure 4: Per-task performance improvement of top models during the 2025 test submission phase.
• Figure 5: Evolution over time of the best performing models competing in our challenges for the multiple-choice videoQA task compared to random and frequency-based baselines.