Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SLAT-Phys: Fast Material Property Field Prediction from Structured 3D Latents

Rocktim Jyoti Das, Dinesh Manocha

Abstract

Estimating the material property field of 3D assets is critical for physics-based simulation, robotics, and digital twin generation. Existing vision-based approaches are either too expensive and slow or rely on 3D information. We present SLAT-Phys, an end-to-end method that predicts spatially varying material property fields of 3D assets directly from a single RGB image without explicit 3D reconstruction. Our approach leverages spatially organised latent features from a pretrained 3D asset generation model that encodes rich geometry and semantic prior, and trains a lightweight neural decoder to estimate Young's modulus, density, and Poisson's ratio. The coarse volumetric layout and semantic cues of the latent representation about object geometry and appearance enable accurate material estimation. Our experiments demonstrate that our method provides competitive accuracy in predicting continuous material parameters when compared against prior approaches, while significantly reducing computation time. In particular, SLAT-Phys requires only 9.9 seconds per object on an NVIDIA RTXA5000 GPU and avoids reconstruction and voxelization preprocessing. This results in 120x speedup compared to prior methods and enables faster material property estimation from a single image.

SLAT-Phys: Fast Material Property Field Prediction from Structured 3D Latents

Abstract

Estimating the material property field of 3D assets is critical for physics-based simulation, robotics, and digital twin generation. Existing vision-based approaches are either too expensive and slow or rely on 3D information. We present SLAT-Phys, an end-to-end method that predicts spatially varying material property fields of 3D assets directly from a single RGB image without explicit 3D reconstruction. Our approach leverages spatially organised latent features from a pretrained 3D asset generation model that encodes rich geometry and semantic prior, and trains a lightweight neural decoder to estimate Young's modulus, density, and Poisson's ratio. The coarse volumetric layout and semantic cues of the latent representation about object geometry and appearance enable accurate material estimation. Our experiments demonstrate that our method provides competitive accuracy in predicting continuous material parameters when compared against prior approaches, while significantly reducing computation time. In particular, SLAT-Phys requires only 9.9 seconds per object on an NVIDIA RTXA5000 GPU and avoids reconstruction and voxelization preprocessing. This results in 120x speedup compared to prior methods and enables faster material property estimation from a single image.
Paper Structure (27 sections, 9 equations, 4 figures, 2 tables)

This paper contains 27 sections, 9 equations, 4 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Vision-based Physics Estimation
  4. Learned 3D Representations
  5. Our Method: SLAT-Phys
  6. Method Overview
  7. Problem Formulation
  8. SLAT Feature Extraction
  9. Physics Decoder
  10. Training Data
  11. Training Objective
  12. Experiments
  13. Implementation Details
  14. SLAT Generation
  15. Dataset
  16. ...and 12 more sections

Figures (4)

  • Figure 1: Different material field for different parts of the flower vase. The flower leaves are deformable whereas the vase is rigid.
  • Figure 2: Overview of SLAT-Phys: Given a single image of a 3D asset, our framework first encodes the image using the Trellis encoder to obtain a structured 3D latent (SLAT) representation. From this latent, two decoders predict the corresponding 3D Gaussian Splatting (3DGS) representation and spatially varying material properties $(E, \rho, \nu)$ at each voxel. The resulting Gaussian representation together with the predicted physical parameters are then passed to an MPM solver to perform physics-based simulation.
  • Figure 3: Physics-to-SLAT voxel alignment. Physics annotations predicted by Pixie and the SLAT voxel grid produced by TRELLIS may exhibit a rigid rotational offset due to independent reconstruction pipelines. The figure illustrates the voxel grids before alignment and after applying the estimated rigid transformation using ICP.
  • Figure 4: Physics-based simulation results using the predicted physical parameters computed using SLAT-Phys. Given a single image, our method predicts spatially varying Young's modulus ($E$), Poisson's ratio ($\nu$), and density ($\rho$) from SLAT features. The same SLAT representation is also decoded to obtain 3D Gaussian splats for geometry and appearance reconstruction. The resulting geometry and predicted physical parameters are then used in an MPM simulator to generate physically plausible dynamics. We highlight the simulation results for three objects with different material behaviors: a snow man and a rubber duck falling under gravity and a flower under the influence of wind.