Physically Compatible 3D Object Modeling from a Single Image

TL;DR

This work tackles the gap between visual fidelity and physical plausibility in single-image to 3D reconstruction by introducing physical compatibility optimization. It decomposes object behavior into mechanical properties $\Theta$, external forces $\mathbf{f}_{ext}$, and rest-shape geometry $\mathcal{M}_{rest}$, and enforces static equilibrium $\mathbf{f}_{int}(\mathbf{x}_{static}, \mathbf{X}_{rest}; \Theta) = \mathbf{f}_{ext}(\mathbf{x}_{static})$ as a hard constraint. The rest-shape is parameterized through plastic deformation $\mathbf{F}_p$ applied to an initial configuration, with implicit differentiation used to compute gradients for efficient optimization, enabling integration with existing single-view pipelines. Evaluations on Objaverse show consistent improvements across multiple baselines in physical compatibility metrics, and the method supports dynamic simulation and 3D printing, highlighting strong potential for reliable, fabrication-ready designs derived from images.

Abstract

We present a computational framework that transforms single images into 3D physical objects. The visual geometry of a physical object in an image is determined by three orthogonal attributes: mechanical properties, external forces, and rest-shape geometry. Existing single-view 3D reconstruction methods often overlook this underlying composition, presuming rigidity or neglecting external forces. Consequently, the reconstructed objects fail to withstand real-world physical forces, resulting in instability or undesirable deformation -- diverging from their intended designs as depicted in the image. Our optimization framework addresses this by embedding physical compatibility into the reconstruction process. We explicitly decompose the three physical attributes and link them through static equilibrium, which serves as a hard constraint, ensuring that the optimized physical shapes exhibit desired physical behaviors. Evaluations on a dataset collected from Objaverse demonstrate that our framework consistently enhances the physical realism of 3D models over existing methods. The utility of our framework extends to practical applications in dynamic simulations and 3D printing, where adherence to physical compatibility is paramount.

Figures (9)

• Figure 1: Existing methods for single-view reconstruction often result in objects that, when subjected to real-world physical forces (such as gravity) and user-required mechanical materials, exhibit problematic behaviors such as toppling over (top left) and undesirable deformation (top right), diverging from their intended depiction in the input images. In contrast, our approach produces physical objects that maintain stability (bottom left) and mirror the objects' static equilibrium state captured in the input images (bottom right).
• Figure 2: Overall pipeline. Given predefined mechanical properties and external forces, our pipeline optimizes the rest-shape geometry to ensure that the shape, when in a state of static equilibrium, aligns with the target image and meets stability criteria. We visualize the stress distribution of the static geometry using a colored heat map, illustrating the spatially varying deformation of the physical object under static equilibrium.
• Figure 3: Quantitative results on fracture rate. We plot the relationship between the fracture rate and the maximum stress threshold across five single-image reconstruction methods. The shapes optimized with our framework exhibit a consistently lower fracture rate compared to those shapes obtained without our pipeline. MeshLRM and TripoSR feature prevalent thin structures in their reconstructed shapes, whereas our approach significantly reduces the fracture rate in both cases.
• Figure 4: Qualitative results on physical compatibility optimization. Left: Rest shapes optimized using our approach result in static shapes that closely match the input images when subjected to gravity. In contrast, shapes without the optimization fail to replicate the geometry in the input image. Right: our optimization process ensures that the optimized shapes are capable of supporting themselves, whereas the baseline methods fail to achieve this stability.
• Figure 5: Ablation study on Young's modulus. By changing the material properties, our method can produce various rest-shape geometries (top), which all result in the same static shapes that match the input image (middle). Although these static shapes appear identical under static equilibrium, they exhibit different deformation when subjected to the same compression forces exerted by the yellow block, attributable to the differences in their material properties (bottom).