GVSynergy-Det: Synergistic Gaussian-Voxel Representations for Multi-View 3D Object Detection

TL;DR

GVSynergy-Det tackles image-based 3D object detection without depth or dense 3D supervision by fusing continuous Gaussian splatting with discrete voxel representations. The method introduces a cross-representation enhancement that adaptively integrates Gaussian-derived geometry with voxel features, enabling more accurate localization and robust detection in indoor scenes. Key contributions include a generalizable Gaussian Splatting module, a voxelized Gaussian feature encoder with occupancy guidance, and an adaptive fusion mechanism, achieving state-of-the-art performance on ScanNetV2 and ARKitScenes among box-supervised methods. The results demonstrate that leveraging complementary geometric cues from both representations yields practical, efficient indoor 3D perception without costly depth data.

Abstract

Image-based 3D object detection aims to identify and localize objects in 3D space using only RGB images, eliminating the need for expensive depth sensors required by point cloud-based methods. Existing image-based approaches face two critical challenges: methods achieving high accuracy typically require dense 3D supervision, while those operating without such supervision struggle to extract accurate geometry from images alone. In this paper, we present GVSynergy-Det, a novel framework that enhances 3D detection through synergistic Gaussian-Voxel representation learning. Our key insight is that continuous Gaussian and discrete voxel representations capture complementary geometric information: Gaussians excel at modeling fine-grained surface details while voxels provide structured spatial context. We introduce a dual-representation architecture that: 1) adapts generalizable Gaussian Splatting to extract complementary geometric features for detection tasks, and 2) develops a cross-representation enhancement mechanism that enriches voxel features with geometric details from Gaussian fields. Unlike previous methods that either rely on time-consuming per-scene optimization or utilize Gaussian representations solely for depth regularization, our synergistic strategy directly leverages features from both representations through learnable integration, enabling more accurate object localization. Extensive experiments demonstrate that GVSynergy-Det achieves state-of-the-art results on challenging indoor benchmarks, significantly outperforming existing methods on both ScanNetV2 and ARKitScenes datasets, all without requiring any depth or dense 3D geometry supervision (e.g., point clouds or TSDF).

Figures (5)

• Figure 1: Model Size vs. Detection Performance. We compare our GVSynergy-Det (red star) against state-of-the-art methods. The x-axis represents model size (Parameters in Millions), and the y-axis represents detection performance (mAP@0.25). Our method achieves a superior Pareto frontier, offering the highest accuracy with a compact model size, notably without requiring any dense 3D supervision (e.g., point clouds or depth).
• Figure 2: Overall framework of GVSynergy-Det. Given multi-view RGB images, the GVSynergy-Det first extracts features using shared transformer backbones and constructs both a 3D voxel representation through back-projection and a Gaussian representation through pixel-aligned Gaussian prediction. Then, the Cross-Representation Enhancement module is proposed to synergistically integrate the 3D voxel representation and Gaussian representation. Specifically, Gaussian Voxelization transforms irregular Gaussian features into a voxel-aligned grid with occupancy guidance, followed by Adaptive Cross-Enhancement that learns to dynamically weight contributions from both representations based on concatenated features, producing a Gaussian-enhanced 3D representation for improved detection.
• Figure 3: 3D Detection Performance on ScanNetV2 with varying test views. We plot mAP@0.25 (top) and mAP@0.50 (bottom) against the number of input views ranging from 5 to 70. GVSynergy-Det (red line) consistently outperforms competing methods, demonstrating superior robustness particularly in sparse-view scenarios.
• Figure 4: Qualitative comparison of 3D object detection on ScanNetV2. The rows correspond to (from top to bottom): Ground Truth, Ours, MVSDet, ImVoxelNet, and CN-RMA. The dashed red boxes highlight specific challenging regions. In the second column, baselines hallucinate an extra object next to the cabinet (blue box), whereas our method correctly predicts only the single cabinet. In the fourth column, our method correctly identifies exactly two windows, whereas baselines predict incorrect counts or misaligned boxes.
• Figure 5: Qualitative results on the ARKitScenes dataset. The left column shows the Ground Truth annotations, and the right column shows the predictions from our method. Our model demonstrates high detection fidelity, producing bounding boxes that accurately match the scale, orientation, and location of the Ground Truth objects.