Perception Test 2024: Challenge Summary and a Novel Hour-Long VideoQA Benchmark

TL;DR

The paper reports the Second Perception Test challenge held as an ECCV 2024 workshop, expanding to seven tracks and introducing the novel hour-long 1h-walk VQA benchmark to probe long-context multimodal reasoning. It details benchmark design, evaluation protocols, and results across object tracking, point tracking, temporal action localisation, temporal sound localisation, mc-vQA, grounded vQA, and hour-long vQA, revealing substantial progress over the 2023 edition while highlighting remaining gaps to human performance. The 1h-walk VQA benchmark, built on Walking Tours videos with 70 five-way questions, enforces zero-shot evaluation and long-range temporal reasoning, offering a challenging testbed for future models. Collectively, the results demonstrate meaningful gains from multimodal feature integration and task-specific architectures, and they motivate future zero-shot emphasis and cross-track single-model evaluation strategies to advance robust, long-span video understanding.

Abstract

Following the successful 2023 edition, we organised the Second Perception Test challenge as a half-day workshop alongside the IEEE/CVF European Conference on Computer Vision (ECCV) 2024, with the goal of benchmarking state-of-the-art video models and measuring the progress since last year using the Perception Test benchmark. This year, the challenge had seven tracks (up from six last year) and covered low-level and high-level tasks, with language and non-language interfaces, across video, audio, and text modalities; the additional track covered hour-long video understanding and introduced a novel video QA benchmark 1h-walk VQA. Overall, the tasks in the different tracks were: object tracking, point tracking, temporal action localisation, temporal sound localisation, multiple-choice video question-answering, grounded video question-answering, and hour-long video question-answering. We summarise in this report the challenge tasks and results, and introduce in detail the novel hour-long video QA benchmark 1h-walk VQA.

Figures (12)

• Figure 1: Top-1 accuracy of recent VLMs vs human baseline on the Perception Test multiple-choice video QA task. We include the results published by models' authors where available, otherwise we ran the models independently (GPT-4V, SeViLA, Flamingo).
• Figure 2: Average video length in our newly-proposed 1h-walk VQA benchmark compared to existing benchmarks.
• Figure 3: Example of a counting question in 1h-walk VQA that spans more than 30 minutes. We show the relevant frames and their associated timestamps. The correct answer is marked in bold.
• Figure 4: Per-track performance improvement compared to baselines and compared to best models from 2023, respectively.
• Figure 5: Per-task performance improvement of top models during the 2024 test submission phase.