A Three-Level Alignment Framework for Large-Scale 3D Retrieval and Controlled 4D Generation

TL;DR

Uni4D tackles open-vocabulary 3D retrieval and controllable 4D generation by leveraging a three-level alignment across text, 3D models, and images, built on Align3D-130 with $|Align3D-130|=1.3\text{M}$ samples. It introduces the ATMS model for text-to-3D retrieval and the CrOSSMA framework for cross-modal alignment, optimized with Monte Carlo Tree Search (MCTS) reinforcement learning. A three-level alignment feeds into a Gaussian-process-based 4D generator, conditioning on text, 3D, image, and video embeddings. Experiments show state-of-the-art text-to-3D retrieval on large-scale data and high-quality, temporally coherent 4D assets, highlighting Uni4D's potential for scalable dynamic multimodal understanding.

Abstract

We introduce Uni4D, a unified framework for large scale open vocabulary 3D retrieval and controlled 4D generation based on structured three level alignment across text, 3D models, and image modalities. Built upon the Align3D 130 dataset, Uni4D employs a 3D text multi head attention and search model to optimize text to 3D retrieval through improved semantic alignment. The framework further strengthens cross modal alignment through three components: precise text to 3D retrieval, multi view 3D to image alignment, and image to text alignment for generating temporally consistent 4D assets. Experimental results demonstrate that Uni4D achieves high quality 3D retrieval and controllable 4D generation, advancing dynamic multimodal understanding and practical applications.

Figures (7)

• Figure 1: Uni4D Freamwork:The Uni4D architecture, trained and inferred on the Align3D-130 dataset, processes a text prompt via a tokenizer and generates embeddings through CAMS for image and video generation. Using ATMS, it retrieves a 3D model, which is expanded into multiple views for 4D generation. The output includes 4D assets of dynamic objects, controllable with both images and videos
• Figure 2: Names and Sizes of Different Source Datasets Comprising Align3D-130.
• Figure 3: ATMS
• Figure 4: Comparison of 4D assets generated by different models for the same text: For the text 'pink teddy bear dancing,' the models generated by Dream-Gaussian4D and Stag4D exhibit somewhat blurred appearances, while 4D-fly does not match the 'dance' motion in terms of temporal consistency.
• Figure 5: The top-k retrieval accuracy of the three models mentioned above was evaluated on both the pre-clustered and randomly shuffled datasets.