4D-Bench: Benchmarking Multi-modal Large Language Models for 4D Object Understanding

TL;DR

4D-Bench introduces the first public benchmark dedicated to 4D object understanding, combining 4D object QA and 4D object captioning to test multimodal LLMs on multi-view spatial-temporal reasoning. By rendering thousands of synthetic, multi-view 4D objects from Objaverse-XL and applying careful data curation and human/LLM annotations, the benchmark provides high-quality, counterfactual-rich evaluation. Experimental results show that state-of-the-art MLLMs lag far behind human performance, especially in counting and action/temporal understanding, and that closed-source models generally outperform open-source ones in temporal-centric tasks. The dataset and evaluation framework aim to drive progress in 4D object-language understanding, enabling more robust interactions with dynamic 3D assets for applications like digital twins, AR/VR, and robotics.

Abstract

Multimodal Large Language Models (MLLMs) have demonstrated impressive 2D image/video understanding capabilities. However, there are no publicly standardized benchmarks to assess the abilities of MLLMs in understanding the 4D objects (3D objects with temporal evolution over time). In this paper, we introduce 4D-Bench, the first benchmark to evaluate the capabilities of MLLMs in 4D object understanding, featuring tasks in 4D object Question Answering (4D object QA) and 4D object captioning. 4D-Bench provides 4D objects with diverse categories, high-quality annotations, and tasks necessitating multi-view spatial-temporal understanding, different from existing 2D image/video-based benchmarks. With 4D-Bench, we evaluate a wide range of open-source and closed-source MLLMs. The results from the 4D object captioning experiment indicate that MLLMs generally exhibit weaker temporal understanding compared to their appearance understanding, notably, while open-source models approach closed-source performance in appearance understanding, they show larger performance gaps in temporal understanding. 4D object QA yields surprising findings: even with simple single-object videos, MLLMs perform poorly, with state-of-the-art GPT-4o achieving only 63\% accuracy compared to the human baseline of 91\%. These findings highlight a substantial gap in 4D object understanding and the need for further advancements in MLLMs.

Figures (25)

• Figure 1: An example demonstrating the challenges of 4D object understanding involves multi-view spatial-temporal reasoning. Given the 4D object, the robot's right hand seems ambiguous in some views at first and eventually disappears over time. Hence, answering the question needs to (1) address multi-view ambiguity and choose proper views and time that the right hand is visible, (2) localize the right hand, (3) and track its evolutions along the time dimension.
• Figure 2: Illustration of the 4D-Bench. 4D-Bench consists of two critical tasks (a) 4D object QA and (b) 4D object captioning. 4D object QA provides one question and four choices per QA to evaluate MLLMs. 4D object captioning provides five human captions per 4D object.
• Figure 3: Pipeline for constructing the 4D-Bench dataset. The pipeline includes rendering multi-view videos for 4D objects from Objaverse-XL, motion filtering, visual quality filtering, and multi-stage annotations for QA pairs and captions. Captions are purely human-annotated, while QA pairs are generated through a hybrid approach using MLLMs and human validation.
• Figure 4: Subtask and category distributions in 4D object QA and captioning. Left: Distribution of five subtasks in the 4D object QA task, 751 question-answering pairs in total. Right: Distribution of 4D object categories in 4D object captioning task, 580 4D objects in total.
• Figure 5: An example from Object Counting subtask. Answering this question requires integrating multi-view information and capturing cross-view correspondences to count the presents, necessitating multi-view reasoning abilities. If relying solely on a single view (e.g. the middle row), it would lead to wrong answers (e.g. four), since some boxes are occluded and invisible in this view.