MonoCLUE : Object-Aware Clustering Enhances Monocular 3D Object Detection

TL;DR

MonoCLUE tackles monocular 3D object detection by injecting robust visual priors through local clustering of object-region features and a generalized scene memory learned across images. The method augments object queries with both cluster-level and memory-derived cues, guided by SAM segmentation and cosine-based re-localization, to improve detection under occlusion and limited visibility. Ablations and KITTI experiments show gains from each component, with a favorable efficiency profile compared to previous DETR-based monocular approaches. Overall, MonoCLUE advances monocular 3D perception by embedding appearance priors into the decoding process, enabling more accurate and stable 3D localization in challenging driving scenarios.

Abstract

Monocular 3D object detection offers a cost-effective solution for autonomous driving but suffers from ill-posed depth and limited field of view. These constraints cause a lack of geometric cues and reduced accuracy in occluded or truncated scenes. While recent approaches incorporate additional depth information to address geometric ambiguity, they overlook the visual cues crucial for robust recognition. We propose MonoCLUE, which enhances monocular 3D detection by leveraging both local clustering and generalized scene memory of visual features. First, we perform K-means clustering on visual features to capture distinct object-level appearance parts (e.g., bonnet, car roof), improving detection of partially visible objects. The clustered features are propagated across regions to capture objects with similar appearances. Second, we construct a generalized scene memory by aggregating clustered features across images, providing consistent representations that generalize across scenes. This improves object-level feature consistency, enabling stable detection across varying environments. Lastly, we integrate both local cluster features and generalized scene memory into object queries, guiding attention toward informative regions. Exploiting a unified local clustering and generalized scene memory strategy, MonoCLUE enables robust monocular 3D detection under occlusion and limited visibility, achieving state-of-the-art performance on the KITTI benchmark.

Figures (5)

• Figure 1: (a) Comparison between predicted and ground-truth regions, showing prediction errors on occluded objects. (b) Local clustering results, revealing that clusters reflect object parts and orientation, even in incomplete segmentation. (c) Activation maps of cluster features propagated across the image, capturing visually similar regions beyond initial segmentation.
• Figure 2: Overall architecture of the proposed MonoCLUE. Our core components are local clustering, similarity based re-localization, and query initialization. We perform clustering on the visual encoder features to extract local cluster features from specific regions. The local cluster features are then used for re-localization, generalized scene memory, and query initialization.
• Figure 3: The process of local clustering and generalized scene memory. Each feature corresponds to a monocular image. (a) Local cluster features are obtained by independently extracting $N_l$ features from the masked object regions. (b) Generalized scene memory integrates $\textbf{L}_c$ from multiple images into a shared representation.
• Figure 4: The process of re-localization and query initialization. (a) Local cluster similarity is used to compute $\textbf{S}$, which re-localizes features to object-like regions. (b) Query initialization using $\textbf{L}_c$, $\textbf{G}_c$, and $\textbf{B}_c$.
• Figure 5: Qualitative comparison on the KITTI validation set. Ground-truth boxes (green) and predictions (red) are shown for both 3D bounding boxes and bird’s-eye view (BEV).