Cog2Gen3D: Sculpturing 3D Semantic-Geometric Cognition for 3D Generation

TL;DR

This work argues that semantic information and absolute geometry empower 3D cognition, thereby enabling controllable 3D generation for the physical world and significantly outperforms existing methods in both semantic fidelity and geometric plausibility.

Abstract

Generative models have achieved success in producing semantically plausible 2D images, but it remains challenging in 3D generation due to the absence of spatial geometry constraints. Typically, existing methods utilize geometric features as conditions to enhance spatial awareness. However, these methods can only model relative relationships and are prone to scale inconsistency of absolute geometry. Thus, we argue that semantic information and absolute geometry empower 3D cognition, thereby enabling controllable 3D generation for the physical world. In this work, we propose Cog2Gen3D, a 3D cognition-guided diffusion framework for 3D generation. Our model is guided by three key designs: 1) Cognitive Feature Embeddings. We encode different modalities into semantic and geometric representations and further extract logical representations. 2) 3D Latent Cognition Graph. We structure different representations into dual-stream semantic-geometric graphs and fuse them via common-based cross-attention to obtain a 3D cognition graph. 3) Cognition-Guided Latent Diffusion. We leverage the fused 3D cognition graph as the condition to guide the latent diffusion process for 3D Gaussian generation. Under this unified framework, the 3D cognition graph ensures the physical plausibility and structural rationality of 3D generation. Moreover, we construct a validation subset based on the Marble World Labs. Extensive experiments demonstrate that our Cog2Gen3D significantly outperforms existing methods in both semantic fidelity and geometric plausibility.

Figures (10)

• Figure 1: Paradigm shift of 3D generation. Semantic-Guided Generation relies heavily on 2D semantic priors, resulting in physical violations such as object intersections. 2D Geometry-Guided 3D Generation introduces relative geometric constraints but frequently suffers from scale inconsistency due to the lack of absolute metric awareness. Our 3D Cognition-Guided paradigm leverages a 3D cognition graph as the condition, ensuring 3D generation with fidelity and plausibility.
• Figure 2: Overview of Cog2Gen3D. The model first extracts multiple cognitive tokens ($T_S, T_G, T_L$). These tokens are structured into a 3D Cognition Graph, where the logical tokens act as a bridge for semantic-geometric alignment. Finally, the awakened 3D cognition steers a latent diffusion process to generate 3D Gaussians.
• Figure 3: Cross-view feature correspondence analysis. The attention map and the t-SNE visualization demonstrate that the VGGT encoder has superior cross-view geometric consistency. This validates its capability of capturing absolute geometry information, which motivates us to introduce VGGT encoder as our geometric expert.
• Figure 4: Feature distribution of explicit vs. latent scene graphs under prompt perturbations. Explicit graphs diverge significantly when given missing or incorrect relations compared to the correct baseline. Conversely, our latent scene graph maintains a stable distribution, demonstrating superior robustness to unpromising prompts.
• Figure 5: Architectural details and interpretability of the 3D Latent Cognition Graph. (a) The pipeline for constructing cognition graph from cognitive tokens. (b) The correspondence visualization showing how graph components precisely align with 3D entities and spatial boundaries.