Physically Guided Visual Mass Estimation from a Single RGB Image

TL;DR

This work tackles the challenge of estimating object mass from a single RGB image, which is ill-posed because mass depends on both geometry (volume) and material density. The authors propose a physically guided framework that decouples these factors: geometry-derived monocular depth yields an explicit object-centric point cloud for volume $\widehat{V}$, while coarse material semantics from a vision–language model informs density $\hat{\rho}$; these cues are fused by an instance-adaptive gate, and two separate regression heads predict $\widehat{V}$ and $\hat{\rho}$ with the final mass $\widehat{m} = \widehat{V} \hat{\rho}$ under mass supervision only. The method leverages appearance, geometry, and semantic density cues, with geometry typically dominating and semantics providing robust density priors, outperforming end-to-end RGB baselines and VLM-only approaches on image2mass and ABO-500, and approaching multi-view NeRF2Physics when restricted to a single view. This yields interpretable, robust mass estimation for safe robotic manipulation, broadening applicability where direct interaction is costly or hazardous. The work also highlights the value of explicit physical decomposition and shows the benefits of integrating monocular geometry and semantic priors in a learnable fusion framework.

Abstract

Estimating object mass from visual input is challenging because mass depends jointly on geometric volume and material-dependent density, neither of which is directly observable from RGB appearance. Consequently, mass prediction from pixels is ill-posed and therefore benefits from physically meaningful representations to constrain the space of plausible solutions. We propose a physically structured framework for single-image mass estimation that addresses this ambiguity by aligning visual cues with the physical factors governing mass. From a single RGB image, we recover object-centric three-dimensional geometry via monocular depth estimation to inform volume and extract coarse material semantics using a vision-language model to guide density-related reasoning. These geometry, semantic, and appearance representations are fused through an instance-adaptive gating mechanism, and two physically guided latent factors (volume- and density-related) are predicted through separate regression heads under mass-only supervision. Experiments on image2mass and ABO-500 show that the proposed method consistently outperforms state-of-the-art methods.

Figures (11)

• Figure 1: Overview of the proposed physically structured visual mass estimation framework. From a single RGB image, the model infers appearance features, object-centric geometry via monocular depth estimation, and a coarse semantic description of material composition. Geometry and semantic cues are explicitly aligned with volume- and density-related factor inference, respectively, while appearance provides complementary context. These representations are fused using an instance-adaptive mechanism to predict object mass. The framework is trained using mass supervision only and avoids the need for explicit density or volume labels.
• Figure 2: Qualitative comparison of our method with image2massSSC17 and the RGB+Depth approach CM25 on household-object images. Values are masses (kg).
• Figure 3: Examples of overlapping categories with visual differences between training (left) and test (right).
• Figure 4: Qualitative comparison of our method with image2massSSC17, an RGB+Depth baseline CM25, LLaVALLW23, and QwenYYZ24 on the image2mass test set. For LLaVA and Qwen, we evaluate direct and reasoning strategies. Values are masses (kg). Bold: best; underlined: second-best performance.
• Figure 5: Preprocessing workflow used for evaluation beyond segmented benchmarks. We segment the target object using a text prompt (e.g., "Binocular") and feed the resulting object-centric image to the mass predictor.