PhysGS: Bayesian-Inferred Gaussian Splatting for Physical Property Estimation

TL;DR

PhysGS presents a Bayesian extension to 3D Gaussian Splatting to infer dense per-point physical properties from RGB observations by leveraging vision–language priors. It combines a Dirichlet–Categorical material fusion with a Normal–Inverse–Gamma uncertainty model to produce calibrated aleatoric and epistemic uncertainty, yielding a spatially continuous property field and object-level mass estimates. The approach achieves significant gains in mass, friction, and hardness estimation across indoor and outdoor datasets (e.g., up to $22.8\%$ improvement in mass error and up to $61.2\%$ in hardness error) and provides uncertainty maps that reflect segmentation quality and evidence strength. This framework unifies 3D reconstruction, semantic/material reasoning, and physical-property estimation, supporting robust robotic interaction in diverse real-world environments.

Abstract

Understanding physical properties such as friction, stiffness, hardness, and material composition is essential for enabling robots to interact safely and effectively with their surroundings. However, existing 3D reconstruction methods focus on geometry and appearance and cannot infer these underlying physical properties. We present PhysGS, a Bayesian-inferred extension of 3D Gaussian Splatting that estimates dense, per-point physical properties from visual cues and vision--language priors. We formulate property estimation as Bayesian inference over Gaussian splats, where material and property beliefs are iteratively refined as new observations arrive. PhysGS also models aleatoric and epistemic uncertainties, enabling uncertainty-aware object and scene interpretation. Across object-scale (ABO-500), indoor, and outdoor real-world datasets, PhysGS improves accuracy of the mass estimation by up to 22.8%, reduces Shore hardness error by up to 61.2%, and lowers kinetic friction error by up to 18.1% compared to deterministic baselines. Our results demonstrate that PhysGS unifies 3D reconstruction, uncertainty modeling, and physical reasoning in a single, spatially continuous framework for dense physical property estimation. Additional results are available at https://samchopra2003.github.io/physgs.

Figures (11)

• Figure 1: Overview of PhysGS. Top: Our method estimates per-point and dense physical properties (e.g., friction, hardness, density, stiffness, and mass) by combining vision–language material priors with Bayesian updates over 3D Gaussian splats. Bottom: PhysGS can also be deployed in outdoor environments to infer scene-level properties such as friction and predictive uncertainty; we visualize the total uncertainty (aleatoric + epistemic).
• Figure 2: PhysGS architecture. Given multi-view images, SAM provides part-level segmentations that are used for 3D Gaussian Splatting (3DGS) reconstruction. For each segmented part, a VLM produces material labels, density estimates, and confidence scores across multiple views. These observations are fused using Bayesian inference with uncertainty quantification to obtain final per-material property distributions. By propagating the estimated densities over the reconstructed 3D Gaussian field, PhysGS predicts per-point density and full-object mass.
• Figure 3: Qualitative comparison on the ABO-500 dataset. For each object, we show the input RGB image, material segmentation and mass-density predictions from NeRF2Physics, and the corresponding results from our method. PhysGS produces cleaner material segmentation with fewer artifacts compared to NeRF2Physics and more consistent part boundaries, and yields sharper, more plausible mass-density fields across diverse object categories.
• Figure 4: Qualitative results for the friction and hardness dataset. Given a single RGB view, PhysGS predicts dense friction coefficients and Shore hardness values for a variety of household objects. The resulting property fields are spatially smooth, physically plausible, and consistent across diverse materials and geometries.
• Figure 5: Outdoor scene results on real environments. From a single RGB view, PhysGS predicts material segmentation, friction coefficients, Young’s modulus, and total uncertainty (aleatoric + epistemic). The method captures broad material variations across natural terrain and vegetation while producing pixel-wise physical property estimates with associated confidence. Higher total uncertainty in Rows 2 and 3 corresponds to scenes with dense clutter and visually ambiguous regions, where SAM provides less precise part-level masks (e.g., separating leaf litter from wood or mud from grass). Rows 1 and 4 exhibit lower uncertainty due to clearer material boundaries and more uniform regions.