DSPNet: Dual-vision Scene Perception for Robust 3D Question Answering

TL;DR

DSPNet tackles robust 3D question answering by integrating dual-vision cues from point clouds and multi-view images. It introduces Text-guided Multi-view Fusion to weight views by textual relevance, Adaptive Dual-vision Perception to fuse texture-rich back-projected features with geometric point features, and Multimodal Context-guided Reasoning to perform efficient cross-modal reasoning. The approach achieves state-of-the-art results on SQA3D, ScanQA, and 3DQA benchmarks, particularly excelling in questions requiring fine-grained texture understanding and spatial reasoning under occlusion. These results highlight the practical impact of dual-vision perception for embodied AI tasks and point to future avenues in dynamic environments and large-scale multi-modal pretraining.

Abstract

3D Question Answering (3D QA) requires the model to comprehensively understand its situated 3D scene described by the text, then reason about its surrounding environment and answer a question under that situation. However, existing methods usually rely on global scene perception from pure 3D point clouds and overlook the importance of rich local texture details from multi-view images. Moreover, due to the inherent noise in camera poses and complex occlusions, there exists significant feature degradation and reduced feature robustness problems when aligning 3D point cloud with multi-view images. In this paper, we propose a Dual-vision Scene Perception Network (DSPNet), to comprehensively integrate multi-view and point cloud features to improve robustness in 3D QA. Our Text-guided Multi-view Fusion (TGMF) module prioritizes image views that closely match the semantic content of the text. To adaptively fuse back-projected multi-view images with point cloud features, we design the Adaptive Dual-vision Perception (ADVP) module, enhancing 3D scene comprehension. Additionally, our Multimodal Context-guided Reasoning (MCGR) module facilitates robust reasoning by integrating contextual information across visual and linguistic modalities. Experimental results on SQA3D and ScanQA datasets demonstrate the superiority of our DSPNet. Codes will be available at https://github.com/LZ-CH/DSPNet.

Figures (7)

• Figure 1: Comprehensive scene perception with dual-vision (point clouds and multi-view images). It is difficult to accurately perceive some flat and small objects (e.g., TV, picture, carpet, phone, etc.) by relying solely on the visual information of point clouds, while multi-view images have richer local texture information and provide more comprehensive visual signals for 3D QA.
• Figure 2: Inherent limitations of back-projection illustrated with a sample from the ScanQA dataset. (a) When aggregating features for coordinates from n mapped multi-view images, each view’s weight remains constant at $\frac{1}{n}$, regardless of the question context. (b) Feature degradation occurs during back-projection from multi-view images to 3D point cloud space due to inherent noise in camera poses, absence of certain views, and complex occlusions, reducing reliability at the edges of the field of view and in occluded areas. Red color points indicate points missed during back-projection (i.e., invalid points), and red ellipses highlight areas with noticeable degradation compared to the original point cloud features.
• Figure 3: (a) The overall architecture of the DSPNet: it takes the 3D scene, multi-view images, and question as the inputs, ultimately output answers to questions. (b) The Text-guided Multi-view Fusion (TGMF) module aims to fuse the multi-view features. (c) The Adaptive Dual-vision Perception (ADVP) module aims to adaptively perceive the vision information derived from point cloud and multi-view images.
• Figure 4: The qualitative comparison of our method with ScanQA and SQA. Our method achieves higher answer accuracy for questions that directly or indirectly involve some challenging entities, such as those with flat shapes and rich local texture details.
• Figure 5: The TGMF module dynamically prioritizes different views according to question context within the same scene.