WorldQA: Multimodal World Knowledge in Videos through Long-Chain Reasoning

TL;DR

WorldQA introduces a video QA benchmark that requires multimodal processing, world knowledge and long-chain reasoning, addressing the gap where prior datasets emphasize perception over cognitive integration. The authors propose WorldRetriever, a modular agent that combines a multimodal key info retriever, a world knowledge retriever, and an answer composer to generate reasoned responses. Empirical results across 13 models show WorldRetriever achieves the best open-ended ($$36.38$$) and multiple-choice ($$36.59$$) performance among peers but remains well below human level, underscoring the need for advances in multimodal fusion and world-knowledge reasoning. The work highlights key insights about frame usage, model consistency, and the potential of retrieval-augmented reasoning to push multimodal world understanding forward, while also revealing limitations in current video-long-frame processing capabilities.

Abstract

Multimodal information, together with our knowledge, help us to understand the complex and dynamic world. Large language models (LLM) and large multimodal models (LMM), however, still struggle to emulate this capability. In this paper, we present WorldQA, a video understanding dataset designed to push the boundaries of multimodal world models with three appealing properties: (1) Multimodal Inputs: The dataset comprises 1007 question-answer pairs and 303 videos, necessitating the analysis of both auditory and visual data for successful interpretation. (2) World Knowledge: We identify five essential types of world knowledge for question formulation. This approach challenges models to extend their capabilities beyond mere perception. (3) Long-Chain Reasoning: Our dataset introduces an average reasoning step of 4.45, notably surpassing other videoQA datasets. Furthermore, we introduce WorldRetriever, an agent designed to synthesize expert knowledge into a coherent reasoning chain, thereby facilitating accurate responses to WorldQA queries. Extensive evaluations of 13 prominent LLMs and LMMs reveal that WorldRetriever, although being the most effective model, achieved only 70% of humanlevel performance in multiple-choice questions. This finding highlights the necessity for further advancement in the reasoning and comprehension abilities of models. Our experiments also yield several key insights. For instance, while humans tend to perform better with increased frames, current LMMs, including WorldRetriever, show diminished performance under similar conditions. We hope that WorldQA,our methodology, and these insights could contribute to the future development of multimodal world models.

Figures (16)

• Figure 1: An example video from our WorldQA. To determine where the lady went when she was absent from the video, we rely on visual cues, auditory hints, and the application of world knowledge. This forms a reasoning chain to deduce the answer. WorldQA comprises 1007 question-answer pairs and 303 videos, spanning five types of world knowledge. On average, the reasoning chain consists of 4.45 steps.We recommand watch the video: https://www.youtube.com/watch?v=NXbJLLf9E_I
• Figure 2: (a) An example for reformating open-ended QA into multi-choice QA. (b) the distribution of different world-knowledge types. (c) the distribution of reasoning step counts.
• Figure 3: WorldRetriever, an agent designed to synthesize expert knowledge into a coherent reasoning chain for answering questions.
• Figure 4: Comparative performance of advanced LMM and our method across increasing reasoning steps.
• Figure 5: Examples of how does GPT-4 score in the open-ended question.