Accelerating Physical Property Reasoning for Augmented Visual Cognition

TL;DR

This work targets real-time vision-guided reasoning of object physical properties for augmented visual cognition. It introduces AURA, which combines a one-shot VGGT-based geometry module with efficient semantic fusion using DINO features, adaptive sampling, view filtering, and parallelized CLIP processing, plus GPT-4o-driven material grounding and feature-space densification for voxel- and object-level inferences. The approach achieves latency reductions of roughly 63x to over 280x compared with NeRF2Physics and PUGS, while maintaining competitive mass accuracy and delivering dense, per-point material and property maps. A real-world case study with smart glasses demonstrates robust performance in cluttered environments, enabling gaze-guided, near real-time reasoning and overlays for augmented reality applications with practical impact in retail, safety, and AR-driven training.

Abstract

This paper introduces \sysname, a system that accelerates vision-guided physical property reasoning to enable augmented visual cognition. \sysname minimizes the run-time latency of this reasoning pipeline through a combination of both algorithmic and systematic optimizations, including rapid geometric 3D reconstruction, efficient semantic feature fusion, and parallel view encoding. Through these simple yet effective optimizations, \sysname reduces the end-to-end latency of this reasoning pipeline from 10--20 minutes to less than 6 seconds. A head-to-head comparison on the ABO dataset shows that \sysname achieves this 62.9$\times$--287.2$\times$ speedup while not only reaching on-par (and sometimes slightly better) object-level physical property estimation accuracy(e.g. mass), but also demonstrating superior performance in material segmentation and voxel-level inference than two SOTA baselines. We further combine gaze-tracking with \sysname to localize the object of interest in cluttered, real-world environments, streamlining the physical property reasoning on smart glasses. The case study with Meta Aria Glasses conducted at an IKEA furniture store demonstrates that \sysname achives consistently high performance compared to controlled captures, providing robust property estimations even with fewer views in real-world scenarios.

Figures (10)

• Figure 1: End-to-end system latency comparison of AURA with two SoTA solutions NeRF2Physics (N2P) zhai_physical_2024 and PUGS shuai_pugs_2025. By optimizing each processing component, AURA achieves a significant latency reduction, demonstrating a performance gain of 62.9$\times$ over NeRF2Physics and 287.2$\times$ over PUGS.
• Figure 2: System workflow. AURA accelerates multi-view physical property reasoning by (1) rapidly reconstructing object geometry, (2) fusing semantic cues from multi-view images into a unified 3D representation, and (3) mapping them into physical property fields for real-time reasoning.
• Figure 3: Rapid geometric 3D reconstruction. Given a small set of multi-view images, VGGT predicts camera poses, depth maps, and a coarse point cloud in a single forward pass. Intermediate DINO tokens are extracted and reshaped into 2D feature maps, which are then fused into a 3D semantic representation. A lightweight Sim(3) alignment step refines the predicted poses against ground-truth or auxiliary stereo poses to recover a consistent metric scale.
• Figure 4: An illustration of view importance-based filtering. AURA excludes those bad views that contribute less to semantic fusion, thereby reducing the runtime latency.
• Figure 5: Material proposal pipeline.