Understanding Complexity in VideoQA via Visual Program Generation

TL;DR

This paper tackles the problem of estimating question difficulty in VideoQA by showing that automatically generated visual programs can serve as a robust proxy for complexity. The authors introduce CodePlexity, which analyzes executable code from questions through Abstract Syntax Trees to identify subroutines that correlate with model error, training a logistic regression to predict failure across diverse models. They further demonstrate how this metric can automatically generate harder questions to form CodePlex-QA, a dataset shown to be substantially more challenging than NExT-QA. The work provides practical tools for measuring model-specific challenges, revealing common failure modes in temporal and fine-grained reasoning, and enabling scalable benchmark creation that adapts to evolving models.

Abstract

We propose a data-driven approach to analyzing query complexity in Video Question Answering (VideoQA). Previous efforts in benchmark design have relied on human expertise to design challenging questions, yet we experimentally show that humans struggle to predict which questions are difficult for machine learning models. Our automatic approach leverages recent advances in code generation for visual question answering, using the complexity of generated code as a proxy for question difficulty. We demonstrate that this measure correlates significantly better with model performance than human estimates. To operationalize this insight, we propose an algorithm for estimating question complexity from code. It identifies fine-grained primitives that correlate with the hardest questions for any given set of models, making it easy to scale to new approaches in the future. Finally, to further illustrate the utility of our method, we extend it to automatically generate complex questions, constructing a new benchmark that is 1.9 times harder than the popular NExT-QA.

Figures (19)

• Figure 1: Humans struggle to judge which questions present higher challenges for machine learning models. In our study, the question on the left is universally perceived as being easier than the one on the right, which is inversely correlated with the models' performance. We show that the complexity of the corresponding visual program can serve as a much more reliable predictor.
• Figure 2: Estimating question complexity via code. Our approach to estimating question complexity involves converting the question into code, decomposing the pseudo-code into abstract syntax subtrees ($S_i$), before correlating subtree presence with model performance.
• Figure 3: Correlation of various approaches for estimating question complexity with VideoQA models' success rate on these questions. We observe that humans struggle to accurately predict what is hard for machine learning models and that code can serve as a more reliable source of prediction than natural language questions.
• Figure 4: Detailed analysis of subtrees that correlate with challenging questions among several models. We find that, although each model has its own error-modes, 8 subroutines are shared among all 3 of them (right). One of the patterns we find then analyzing the shared code structures is reasoning about the order of events (left).
• Figure 5: Example questions in CodePlex-QA generated with our approach. It features many challenges that are under-represented in existing, manually-designed benchmarks, motivating development of new approaches with enhanced spatio-temporal modeling capacity.