PhysGen3D: Crafting a Miniature Interactive World from a Single Image

TL;DR

PhysGen3D tackles the challenge of generating physically plausible videos from a single image by constructing an amodal 3D world, inferring geometry, materials, and lighting, and then simulating dynamics with a material point method-based engine. The pipeline combines three modules—3D world creation, Taichi Elements-based dynamics, and physics-based rendering—without task-specific training, enabling user-specified initial conditions and diverse material behaviors. It demonstrates improved physical realism and user control over motion compared with state-of-the-art image-to-video models, while maintaining rendering quality. This approach enables interactive, physics-grounded video synthesis from static imagery, with broad potential for visual effects, design exploration, and digital twin applications.

Abstract

Envisioning physically plausible outcomes from a single image requires a deep understanding of the world's dynamics. To address this, we introduce PhysGen3D, a novel framework that transforms a single image into an amodal, camera-centric, interactive 3D scene. By combining advanced image-based geometric and semantic understanding with physics-based simulation, PhysGen3D creates an interactive 3D world from a static image, enabling us to "imagine" and simulate future scenarios based on user input. At its core, PhysGen3D estimates 3D shapes, poses, physical and lighting properties of objects, thereby capturing essential physical attributes that drive realistic object interactions. This framework allows users to specify precise initial conditions, such as object speed or material properties, for enhanced control over generated video outcomes. We evaluate PhysGen3D's performance against closed-source state-of-the-art (SOTA) image-to-video models, including Pika, Kling, and Gen-3, showing PhysGen3D's capacity to generate videos with realistic physics while offering greater flexibility and fine-grained control. Our results show that PhysGen3D achieves a unique balance of photorealism, physical plausibility, and user-driven interactivity, opening new possibilities for generating dynamic, physics-grounded video from an image.

Figures (18)

• Figure 1: PhysGen3D generates realistic, physically plausible motion from a single image and a text prompt by reasoning about geometry, semantics, and material properties. (a) An apple rolls under the influence of its initial velocity, friction, and shape, producing a natural progression over time. (b) Three animal figures interact dynamically, colliding after being propelled upwards and forwards. (c) A toy potato bounces back with soft-body dynamics in response to an initial downward force, capturing material-specific behaviors. PhysGen3D lets users quickly explore physics‑driven object interactions and behaviors in a compact virtual scene generated from a single input image.
• Figure 2: Method Overview. PhysGen3D's framework consists of three modules: a) 3D world creation, which infers geometry, semantics, rendering and physical parameters from the input image; b) dynamics simulation using Taichi-Elements for particle-based physics; and c) physics-based rendering with a two-pass shadow mapping technique.
• Figure 3: Video generation results. Left: Input initial frame. Right: Generated future frames. We apply an initial velocity to each movable object and use the physically grounded parameters outlined in \ref{['physicalpara']} to generate physically plausible results.
• Figure 4: Qualitative comparison. We compare videos generated from our framework with three state-of-the-art I2V models: Gen-3 Runway2024website, Pika Pika2024website and Kling klingai2024website. We carefully designed the prompt to describe the motion outcome, and uses motion brush to control Kling. Our framework employs initial velocity control. Results show that our method can follow text instructions while maintaining plausible physics.
• Figure 5: Dynamics Effects. We can generate various dynamics from the same input image. The left three columns share the same initial positions and velocities, but are in different materials. The right three columns have the same material, but defer in velocity directions. This showcases the potential of our method for generating diverse physical scenarios.