Advances in 4D Representation: Geometry, Motion, and Interaction

TL;DR

This survey addresses the challenge of modeling 4D content—geometry evolving over time with motion and interaction—by organizing representations around geometry, motion, and interaction, and by examining datasets, training strategies, and evaluation metrics. It analyzes both popular (e.g., NeRF, 3D Gaussian Splatting) and under-explored (e.g., structured models) representations, and discusses how LLMs and video foundation models can enable broad 4D applications. The paper highlights three emerging trends: a shift from per-scene optimization to fast feed-forward reconstruction, the rise of hybrid generation–reconstruction pipelines, and the integration of world knowledge and physics-based priors to enhance realism and controllability. It concludes with a call for unified benchmarks, scalable 4D datasets, and structure-aware, physics-informed representations to bridge vision, graphics, and robotics in 4D.

Abstract

We present a survey on 4D generation and reconstruction, a fast-evolving subfield of computer graphics whose developments have been propelled by recent advances in neural fields, geometric and motion deep learning, as well 3D generative artificial intelligence (GenAI). While our survey is not the first of its kind, we build our coverage of the domain from a unique and distinctive perspective of 4D representations\/}, to model 3D geometry evolving over time while exhibiting motion and interaction. Specifically, instead of offering an exhaustive enumeration of many works, we take a more selective approach by focusing on representative works to highlight both the desirable properties and ensuing challenges of each representation under different computation, application, and data scenarios. The main take-away message we aim to convey to the readers is on how to select and then customize the appropriate 4D representations for their tasks. Organizationally, we separate the 4D representations based on three key pillars: geometry, motion, and interaction. Our discourse will not only encompass the most popular representations of today, such as neural radiance fields (NeRFs) and 3D Gaussian Splatting (3DGS), but also bring attention to relatively under-explored representations in the 4D context, such as structured models and long-range motions. Throughout our survey, we will reprise the role of large language models (LLMs) and video foundational models (VFMs) in a variety of 4D applications, while steering our discussion towards their current limitations and how they can be addressed. We also provide a dedicated coverage on what 4D datasets are currently available, as well as what is lacking, in driving the subfield forward. Project page:https://mingrui-zhao.github.io/4DRep-GMI/

Figures (3)

• Figure 4: Interaction specific representations. (a) Parametrizing poses between interaction entities NeuralDescriptorFields; (b) Representing contacts POSA; (c) Representing affordances 3DAffordanceNet; (d) Representing physical properties FORCEIntuitivePhysics.
• Figure 5: Overview of key datasets in 4D representation research: (a) Objaverse-Dy yao2025sv4d provides a large-scale collection of synthetic animated 3D assets, serving as the primary source for current 4D generation works after filtering; (b) ActorsHQ icsik2023humanrf is a high-fidelity, multi-view human performance capture dataset with per-frame 3D mesh reconstructions for dynamic human motion (c) Stereo4D jin2024stereo4d represent emerging large-scale datasets capturing native 4D data from real-world scenarios.(d) PartNet-Mobility Xiang_2020_SAPIEN offers an articulated object database that facilitates part-based representation and articulated motion learning; (e) Ego-Exo4D grauman2024ego is a multi-camera synchronized video corpus capturing first-person (egocentric) and third-person (exocentric) views of skilled human activities. (f) Parahome kim2025parahome is a dataset capturing 3D human–object interaction in a natural home environment and (g) DeformingThings4D li20214dcomplete is a large-scale synthetic dataset of non-rigidly deforming 3D objects (humanoids and animals) with dense 4D annotations; Actors-HQ, Stereo4D and Ego-Exo4D represent emerging large-scale datasets capturing native 4D data from real-world scenarios.
• Figure 6: Overview of three main training paradigms for 4D generation: (a) Per-scene optimization treats each scene as an individual optimization problem, leveraging information distilled from large foundation models (e.g., image and video diffusion models via score distillation sampling) to optimize a 4D representation; (b) End-to-end training employs feed-forward models trained on large-scale native 4D assets to learn strong data priors; (c) Hybrid optimization combines both approaches by training a customized feed-forward multi-view video generator, then using the generated videos to optimize individual scenes.