PhysX-3D: Physical-Grounded 3D Asset Generation

TL;DR

This work addresses the gap between geometry-centric 3D generation and real-world physics-based application by introducing PhysXNet, a physics-annotated 3D dataset with five fundamental properties, and PhysXGen, a dual-branch feed-forward framework that injects physical knowledge into pre-trained 3D priors. The dataset includes PhysXNet-XL, a 6M-strong procedurally generated extension, enabling robust learning of physical-property distributions. PhysXGen jointly learns physical and structural latents via a physical 3D VAE and a transformer-based diffusion model, achieving plausible physics predictions without sacrificing geometry quality. Extensive experiments on PhysXNet demonstrate improved physical property generation and generalization, and the authors provide code, data, and models to spur further research in embodied AI and robotics.

Abstract

3D modeling is moving from virtual to physical. Existing 3D generation primarily emphasizes geometries and textures while neglecting physical-grounded modeling. Consequently, despite the rapid development of 3D generative models, the synthesized 3D assets often overlook rich and important physical properties, hampering their real-world application in physical domains like simulation and embodied AI. As an initial attempt to address this challenge, we propose \textbf{PhysX-3D}, an end-to-end paradigm for physical-grounded 3D asset generation. 1) To bridge the critical gap in physics-annotated 3D datasets, we present PhysXNet - the first physics-grounded 3D dataset systematically annotated across five foundational dimensions: absolute scale, material, affordance, kinematics, and function description. In particular, we devise a scalable human-in-the-loop annotation pipeline based on vision-language models, which enables efficient creation of physics-first assets from raw 3D assets.2) Furthermore, we propose \textbf{PhysXGen}, a feed-forward framework for physics-grounded image-to-3D asset generation, injecting physical knowledge into the pre-trained 3D structural space. Specifically, PhysXGen employs a dual-branch architecture to explicitly model the latent correlations between 3D structures and physical properties, thereby producing 3D assets with plausible physical predictions while preserving the native geometry quality. Extensive experiments validate the superior performance and promising generalization capability of our framework. All the code, data, and models will be released to facilitate future research in generative physical AI.

Figures (11)

• Figure 1: Visualizations of our PhysXNet for phsycial 3D generation. 3D assets in our dataset have fine-grained physical property annotations, including 1) absolute scale, 2) material, 3) affordance, 4) kinematics, and 5) function descriptions (basic, functional, and kinematical descriptions).
• Figure 2: Top: Definition of properties in PhysXNet . By defining and annotating properties across three categories, common physical quantities can be systematically calculated to enable physical simulations. Bottom: Overview of our human-in-the-loop annotation pipeline. We utilize GPT-4o to gather foundational raw data, which is subsequently verified through human oversight. The kinematic parameters are then rigorously determined and finalized through human review.
• Figure 3: Statistics and distribution of PhysXNet and PhysXNet-XL. (a) Distribution histogram of part number in PhysXNet. (b) Dimensional distribution analysis in PhysXNet, showing physical measurements (length/width/height) frequency. (c) Proportional composition of kinematic states and material, including density, Young's modulus, and Poisson's ratio distribution in PhysXNet, visualized through sectoral ratios. (d) Tag frequency statistics for prevalent object labels in PhysXNet-XL. (e) Component-Category distribution of procedurally generated 3D objects in PhysXNet-XL.
• Figure 4: The architecture of PhysXGen framework. PhysXGen features a two-stage architecture comprising: a physical 3D VAE framework for latent space learning, and a physics-aware generative process for structured latent. The former focuses on establishing a compressed yet information-rich latent representation that encodes physical properties, while the latter specializes in generating physical latents.
• Figure 5: Visualization of the generated results. Given a single image as the prompt, our PhysXGen can generate the physical-grounded 3D assets.