LLaVA-4D: Embedding SpatioTemporal Prompt into LMMs for 4D Scene Understanding

TL;DR

LLaVA-4D advances 4D scene understanding by introducing a dynamic-aware spatiotemporal prompt and a spatiotemporal-disentangled vision embedding to align visual and language representations across static backgrounds and dynamic objects. The approach encodes 4D coordinates from multi-view videos, disentangles visual features into spatial and temporal components, and fuses them with textual coordinates in a cross-attention framework, enabling robust 4D reasoning. A dedicated Chat4D dataset supports multi-stage instruction fine-tuning across 2D, 3D, and 4D data, with extensive experiments showing improvements over 3D LMMs on 4D tasks and validating the critical roles of coordinate encoding, feature disentanglement, and fusion strategy. This work provides a practical pathway to incorporate temporal dynamics into vision-language models, potentially impacting robotics, AR/VR, and dynamic scene understanding applications.

Abstract

Despite achieving significant progress in 2D image understanding, large multimodal models (LMMs) struggle in the physical world due to the lack of spatial representation. Typically, existing 3D LMMs mainly embed 3D positions as fixed spatial prompts within visual features to represent the scene. However, these methods are limited to understanding the static background and fail to capture temporally varying dynamic objects. In this paper, we propose LLaVA-4D, a general LMM framework with a novel spatiotemporal prompt for visual representation in 4D scene understanding. The spatiotemporal prompt is generated by encoding 3D position and 1D time into a dynamic-aware 4D coordinate embedding. Moreover, we demonstrate that spatial and temporal components disentangled from visual features are more effective in distinguishing the background from objects. This motivates embedding the 4D spatiotemporal prompt into these features to enhance the dynamic scene representation. By aligning visual spatiotemporal embeddings with language embeddings, LMMs gain the ability to understand both spatial and temporal characteristics of static background and dynamic objects in the physical world. Additionally, we construct a 4D vision-language dataset with spatiotemporal coordinate annotations for instruction fine-tuning LMMs. Extensive experiments have been conducted to demonstrate the effectiveness of our method across different tasks in 4D scene understanding.

Figures (7)

• Figure 1: Illustration of 3D and 4D LMM paradigms for physical world understanding. (a) Existing 3D-LMMs encode 3D positions as spatial prompts but overlook dynamic objects.(b) Our 4D LMM framework embeds 4D coordinates: position and time as spatiotemporal prompts to capture both background and dynamic objects.(c) Performance comparison of LMMs on benchmarks. (d) Comparison on 4D understanding tasks.
• Figure 2: Our LLaVA-4D consists of three stages: 1) 4D coordinate encoding. Encode 3D position and 1D time with optical flow. 2) Vision embedding. Disentangle visual features into spatiotemporal features and embed the encoded 4D coordinates via cross-attention fusion. 3) Language embedding. Align textual position and time with the fused vision embedding for 4D scene understanding.
• Figure 3: Feature distribution of static background and dynamic object in a 4D dynamic scene. Visual features of dynamic objects appear scattered while static backgrounds are clustered. In contrast, spatiotemporal features show clear discrimination between objects and background.
• Figure 4: Overview of our dataset and training pipeline. (a) Chat4D dataset includes 2D, 3D, and 4D vision-language training sets for dense captioning, QA, and visual grounding. (b) Three-stage training: stages 1-2 use 2D/3D data for initialization; stage 3 uses 4D data for instruction fine-tuning. (c) Spatiotemporal characteristics are extracted as local descriptions to generate 4D instructions.
• Figure 5: Visual comparison of LMMs on 4D scene understanding.