Picasso: Holistic Scene Reconstruction with Physics-Constrained Sampling

TL;DR

Picasso addresses the challenge of physically plausible multi-object scene reconstruction under occlusions and noise by integrating geometry, non-penetration, and physics into a rejection-sampling framework guided by a contact scene graph. It decouples holistic scene reasoning into sequential, tractable subproblems and achieves fast inference by GPU-accelerated coarse-to-fine sampling. The Picasso dataset and the Scene Plausibility Score provide ground-truth, metrics, and benchmarks for physical plausibility, with extensive experiments showing improved pose accuracy and plausibility on YCB-V and Picasso datasets and alignment with human intuition. This work advances digital twin construction for contact-rich scenes, enabling more reliable simulation-based planning and control in robotics and related domains.

Abstract

In the presence of occlusions and measurement noise, geometrically accurate scene reconstructions -- which fit the sensor data -- can still be physically incorrect. For instance, when estimating the poses and shapes of objects in the scene and importing the resulting estimates into a simulator, small errors might translate to implausible configurations including object interpenetration or unstable equilibrium. This makes it difficult to predict the dynamic behavior of the scene using a digital twin, an important step in simulation-based planning and control of contact-rich behaviors. In this paper, we posit that object pose and shape estimation requires reasoning holistically over the scene (instead of reasoning about each object in isolation), accounting for object interactions and physical plausibility. Towards this goal, our first contribution is Picasso, a physics-constrained reconstruction pipeline that builds multi-object scene reconstructions by considering geometry, non-penetration, and physics. Picasso relies on a fast rejection sampling method that reasons over multi-object interactions, leveraging an inferred object contact graph to guide samples. Second, we propose the Picasso dataset, a collection of 10 contact-rich real-world scenes with ground truth annotations, as well as a metric to quantify physical plausibility, which we open-source as part of our benchmark. Finally, we provide an extensive evaluation of Picasso on our newly introduced dataset and on the YCB-V dataset, and show it largely outperforms the state of the art while providing reconstructions that are both physically plausible and more aligned with human intuition.

Figures (12)

• Figure 1: We propose Picasso, an approach to build multi-object scene reconstructions by accounting for object geometry, non-penetration, and physics (i.e., objects should be in a stable equilibrium for the scene to be static). We also release the Picasso dataset: a collection of 10 contact-rich real-world scenes we use to test physical plausibility of scene reconstructions. The figure shows the digital twins generated from the 10 real-world scenes using ground-truth pose annotations.
• Figure 2: An example illustrating that a 3D scene reconstruction from SAM3D chen25arxiv-sam can be physically implausible. Left: The original image. Right: The reconstruction exhibits multiple penetrations, highlighted in the red boxes.
• Figure 3: (a) Inter-object penetration. (b) Object-support penetration. (c) Object-free-space penetration. Free space from an RGB-D image is defined as the empty volume along each camera ray up to the observed depth surface.
• Figure 4: Sample image and corresponding contact scene graph (CSG) with a directed acyclic graph (DAG) approximation.
• Figure 5: Conceptual illustration of the loss landscape on the $\mathcal{M} = \mathop{\mathrm{SE}}\nolimits(3)$ pose manifold for the red bowl shown in the left image. The landscape exhibits a region of similar loss values with two global minima (yellow balls) arising from the ambiguity of partial point cloud observations (dark blue). However, only one of these minima satisfies physics constraints and represents the correct pose. Physics constraints prune the feasible set to identify the physically valid solution.

Theorems & Definitions (2)

• Remark 1: The Single-Object Case
• Definition 1: Contact Scene Graph (CSG) hahn88scg-realistic