View2CAD: Reconstructing View-Centric CAD Models from Single RGB-D Scans

TL;DR

This work tackles the challenge of reconstructing high-fidelity CAD B-Reps from a single RGB-D scan by introducing a view-centric B-Rep (VB-Rep) that encodes visibility bounds and geometric uncertainty. The method combines panoptic segmentation, geometry optimization, and VB-Rep extraction to recover view-consistent CAD topology from partial observations, avoiding hallucination of unseen geometry. It demonstrates robust reconstruction on real and synthetic RGB-D data, with ablations showing the critical roles of intersection guidance and axis alignment. The approach enables practical CAD workflows for in-situ reverse engineering, fixture design, and manufacturing planning even when full point clouds are unavailable. It also lays groundwork for extending to richer surfaces, multi-view integration, and CAD-system pipelines that leverage uncertainty-aware boundary representations.

Abstract

Parametric CAD models, represented as Boundary Representations (B-reps), are foundational to modern design and manufacturing workflows, offering the precision and topological breakdown required for downstream tasks such as analysis, editing, and fabrication. However, B-Reps are often inaccessible due to conversion to more standardized, less expressive geometry formats. Existing methods to recover B-Reps from measured data require complete, noise-free 3D data, which are laborious to obtain. We alleviate this difficulty by enabling the precise reconstruction of CAD shapes from a single RGB-D image. We propose a method that addresses the challenge of reconstructing only the observed geometry from a single view. To allow for these partial observations, and to avoid hallucinating incorrect geometry, we introduce a novel view-centric B-rep (VB-Rep) representation, which incorporates structures to handle visibility limits and encode geometric uncertainty. We combine panoptic image segmentation with iterative geometric optimization to refine and improve the reconstruction process. Our results demonstrate high-quality reconstruction on synthetic and real RGB-D data, showing that our method can bridge the reality gap.

Figures (18)

• Figure 1: The visible surfaces subtend regions in the 2D image whose boundaries form a planar graph (left). The corresponding VB-Rep edges (right) are classified as intersection (a), occluded (b), and visibility (c). All others we label as silhouette edges (d). The blue vertex is a triple intersection, and the magenta vertex is an intersection-visibility vertex.
• Figure 1: Wireframe post-processing steps. Visibility optimization displaces vertices, leading to protruding corners at the endpoints of visibility edges and overlapping edges at visibility-intersection corners (a). We can minimize the displacement at the endpoints of visibility edges (b), followed by removing degenerate vertices (c) to obtain a clean wireframe without artifacts.
• Figure 2: An overview of our pipeline. In the segmentation stage, we train a Mask2Former model cheng2021mask2former to perform panoptic segmentation of the RGB image into primitive instances along with classifying them according to primitive type and axis alignment. In the geometry optimization stage, we use the predicted instance masks and types to fit primitives to the input depth points, and optimize them to enforce consistency with predicted orientation and intersection constraints. Furthermore, we iterate this step to update our knowledge of which edges to use as guidance in the optimization. Finally, in the VB-Rep extraction stage, we use the refined surface primitives and edges to build a coherent CAD representation consisting of bounded surfaces, curves, and points, labeled according to visibility information.
• Figure 2: Visibility mesh extraction. Given the 2D boundary of a particular surface $\mathbf{S}$ (the blue curve in (a)), we extract the submesh of $\mathbf{S}$ with vertices whose projection is contained in the 2D boundary (b). Mutually exclusive regions may be incorrectly joined in this mesh (see the red circle), so we label the submesh faces according to their view normal (c), and use connected components on the face adjacency graph and the orientation labeling to extract all bounded submeshes (d).
• Figure 3: Fitting primitives to a noisy point cloud. Fitting primitives individually (middle) fails to recover the correct orientations especially in the thin cylindrical borders where point supervision is sparse. Meanwhile, globally aligned fitting (right) allows the thin structures to be correctly oriented with the same level of noise in the points.