Point-based Instance Completion with Scene Constraints

TL;DR

This work tackles the problem of recovering complete object geometry from partial scans within indoor scenes, addressing the limitations of canonical coordinate assumptions and the absence of scene context. It introduces a point-based instance completion model with a Seed Generator and cross-attention to sparse scene constraints, enabling robust reasoning at the scene level. A new ScanWCF dataset provides aligned, watertight ground-truth meshes and collision-free scenes to evaluate instance-level completions in realistic indoor environments. Experiments show improved fidelity to partial observations, higher completion quality, and fewer collisions compared with prior methods, demonstrating the practical viability of scene-aware point-based completion for robotics and scene understanding.

Abstract

Recent point-based object completion methods have demonstrated the ability to accurately recover the missing geometry of partially observed objects. However, these approaches are not well-suited for completing objects within a scene, as they do not consider known scene constraints (e.g., other observed surfaces) in their completions and further expect the partial input to be in a canonical coordinate system, which does not hold for objects within scenes. While instance scene completion methods have been proposed for completing objects within a scene, they lag behind point-based object completion methods in terms of object completion quality and still do not consider known scene constraints during completion. To overcome these limitations, we propose a point cloud-based instance completion model that can robustly complete objects at arbitrary scales and pose in the scene. To enable reasoning at the scene level, we introduce a sparse set of scene constraints represented as point clouds and integrate them into our completion model via a cross-attention mechanism. To evaluate the instance scene completion task on indoor scenes, we further build a new dataset called ScanWCF, which contains labeled partial scans as well as aligned ground truth scene completions that are watertight and collision-free. Through several experiments, we demonstrate that our method achieves improved fidelity to partial scans, higher completion quality, and greater plausibility over existing state-of-the-art methods.

Figures (18)

• Figure 1: Visual comparison of completion results. Our approach is better at recovering missing geometry, avoiding collisions, and preserving observed surfaces and known free space.
• Figure 2: Overview of our instance scene completion framework. Instance segmentation is first performed on the partial scan to decompose the scene into its individual objects. Each object is run through our proposed object completion model, which predicts both the complete shape and surface normals. Meshes of each object are then reconstructed to produce the completed scene.
• Figure 3: Overview of our proposed seed generator. Predicting Patch Seed coordinates as offsets from the shape's predicted object center is more robust than directly regressing seed coordinates as in zhou2022seedformer. Our object completions additionally consider other objects in the scene through cross-attention with our known free and occluded space constraints.
• Figure 4: Our proposed ScanWCF has aligned ground truth meshes and partials scans while being free of collisions, unlike previous datasets Scan2CAD and ScanARCW.
• Figure 5: Qualitative comparison on the instance scene completion task.