MonoMM: A Multi-scale Mamba-Enhanced Network for Real-time Monocular 3D Object Detection

TL;DR

The paper tackles monocular 3D object detection by addressing depth ambiguity and computational efficiency in transformer-heavy approaches. It introduces MonoMM, a lightweight architecture with two key modules: Focused Multi-Scale Fusion (FMF) to preserve and fuse multi-scale context, and Depth-Aware Feature Enhancement Mamba (DMB) to integrate depth information with visual features via a Mamba-based sequence strategy. The approach shows significant gains on KITTI, delivering real-time performance (≈40 FPS on a 4090) and state-of-the-art or competitive accuracy for moderate objects, with ablations demonstrating additive benefits of FMF and DMB. This work provides a scalable, plug-and-play solution for real-time monocular 3D detection, with potential impact on autonomous driving and related perception tasks.

Abstract

Recent advancements in transformer-based monocular 3D object detection techniques have exhibited exceptional performance in inferring 3D attributes from single 2D images. However, most existing methods rely on resource-intensive transformer architectures, which often lead to significant drops in computational efficiency and performance when handling long sequence data. To address these challenges and advance monocular 3D object detection technology, we propose an innovative network architecture, MonoMM, a Multi-scale \textbf{M}amba-Enhanced network for real-time Monocular 3D object detection. This well-designed architecture primarily includes the following two core modules: Focused Multi-Scale Fusion (FMF) Module, which focuses on effectively preserving and fusing image information from different scales with lower computational resource consumption. By precisely regulating the information flow, the FMF module enhances the model adaptability and robustness to scale variations while maintaining image details. Depth-Aware Feature Enhancement Mamba (DMB) Module: It utilizes the fused features from image characteristics as input and employs a novel adaptive strategy to globally integrate depth information and visual information. This depth fusion strategy not only improves the accuracy of depth estimation but also enhances the model performance under different viewing angles and environmental conditions. Moreover, the modular design of MonoMM provides high flexibility and scalability, facilitating adjustments and optimizations according to specific application needs. Extensive experiments conducted on the KITTI dataset show that our method outperforms previous monocular methods and achieves real-time detection.

Figures (5)

• Figure 1: The general framework of MonoMM is proposed in this paper. First, the input image is processed by the backbone to extract features. The Focused Multi-Scale Fusion (FMF) module preserves detailed information at each scale through specific feature focusing and diffusion mechanisms. The Depth-Assisted Perception (DAP) module learns depth-aware features through assisted supervised learning. The Depth-Aware Feature Enhancement Mamba (DMB) module fully integrates visual information using adaptive strategies.
• Figure 2: Feature Focusing Diffusion Model.The FMF module preserves more detailed information through two stages. (a) Multi-scale feature fusion generates feature F, (b) further refines the features and fuses them with F, finally generating F$_{out}$.
• Figure 3: Depth-Aware Feature Enhancement Mamba Model.We first divide the input image into small patches and then project them onto patch embeddings. We then utilize Deformable Convolutional Networks (DCN) and DSSM to effectively learn the key features and contextual information within the image. This information is fused with another branch to effectively reduce the risk of gradient vanishing or explosion, while simultaneously enhancing feature learning capability.
• Figure 4: M3D-RPN after adding our module.M3D-RPN (Monocular 3D Region Proposal Network) is a framework designed for 3D object detection using a single camera image.
• Figure 5: Qualitative examples on the KITTI validation set. We provide four images. In the images, the green boxes represent ground truth, and the blue boxes represent predictions from MonoMM.