Know Your Neighbors: Improving Single-View Reconstruction via Spatial Vision-Language Reasoning

TL;DR

Single-view 3D reconstruction is inherently ill-posed due to occlusions. KYN introduces vision-language modulation to enrich per-point features with semantic cues and a language-guided 3D spatial attention to reason about semantic context when predicting point densities $σ_i$. The approach achieves state-of-the-art scene and object reconstruction on KITTI-360 and exhibits improved zero-shot generalization on DDAD, underscoring the value of explicit semantic reasoning for occluded geometry. Overall, KYN advances open-vocabulary 3D scene understanding by marrying vision-language priors with 3D spatial reasoning to produce more plausible reconstructions from a single image.

Abstract

Recovering the 3D scene geometry from a single view is a fundamental yet ill-posed problem in computer vision. While classical depth estimation methods infer only a 2.5D scene representation limited to the image plane, recent approaches based on radiance fields reconstruct a full 3D representation. However, these methods still struggle with occluded regions since inferring geometry without visual observation requires (i) semantic knowledge of the surroundings, and (ii) reasoning about spatial context. We propose KYN, a novel method for single-view scene reconstruction that reasons about semantic and spatial context to predict each point's density. We introduce a vision-language modulation module to enrich point features with fine-grained semantic information. We aggregate point representations across the scene through a language-guided spatial attention mechanism to yield per-point density predictions aware of the 3D semantic context. We show that KYN improves 3D shape recovery compared to predicting density for each 3D point in isolation. We achieve state-of-the-art results in scene and object reconstruction on KITTI-360, and show improved zero-shot generalization compared to prior work. Project page: https://ruili3.github.io/kyn.

Figures (4)

• Figure 1: Single-view scene reconstruction results. We present the predicted 3D occupancy grids given a single input image. The camera is at the bottom left and points to the top right along the $z$-aixs. Previous methods like BTS wimbauer2023behind struggle to recover accurate object shapes (green box) and exhibit trailing effects in unobserved areas (blue box). In contrast, KYN recovers more accurate boundaries and mitigates the trailing effects prevalent in prior art.
• Figure 2: Overview. Given an input image $\textbf{I}_{0}$, we use two image encoders to obtain features ($F_{\text{app}}$, $F_{\text{vis}}$), and fuse these into feature map $F_{\text{fused}}$. We further extract category-level text features and a segmentation map $S$. For a given 3D point set $\mathbf{X}$, we query the extracted features by projecting them onto the image plane yielding point-wise visual and text features. Next, the vl modulation layers endow the point representation with fine-grained semantic information. Finally, the vl spatial attention aggregates these point representations across the 3D scene, yielding density predictions aware of the 3D semantic context.
• Figure 3: Qualitative comparisons on KITTI-360 dataset. We illustrate the scene reconstructions as voxel grids, where the camera is on the left side and points to the right along the $z$-axis. A lighter voxel color indicates higher voxel positions. Compared to previous methods that struggle with corrupted and trailing shapes, our method produces faithful scene geometry, especially for occluded areas.
• Figure 4: Object reconstruction in the KITTI-360 dataset liao2022kitti. From left to right: Reconstructions of the fence, tree, and car. Our method produces more faithful object geometries, in particular in occluded areas and for various semantic categories.