MGNet: Monocular Geometric Scene Understanding for Autonomous Driving

TL;DR

MGNet addresses monocular geometric scene understanding by unifying panoptic segmentation with self-supervised monocular depth estimation in a single real-time model. It employs an encoder–decoder architecture with a shared backbone and three task-specific decoders to produce dense panoptic maps and depth, enabling 3D reconstruction from a single image. The approach uses multi-task learning with uncertainty-based weighting and video-sequence supervision, achieving competitive Cityscapes performance (PQ 55.7, RMSE 8.3) at 30 FPS and KITTI performance (RMSE 3.761) at 82 FPS, while maintaining real-time feasibility. The work demonstrates practical monocular 3D perception for autonomous driving and provides a release-ready framework for future exploration of monocular geometric scene understanding.

Abstract

We introduce MGNet, a multi-task framework for monocular geometric scene understanding. We define monocular geometric scene understanding as the combination of two known tasks: Panoptic segmentation and self-supervised monocular depth estimation. Panoptic segmentation captures the full scene not only semantically, but also on an instance basis. Self-supervised monocular depth estimation uses geometric constraints derived from the camera measurement model in order to measure depth from monocular video sequences only. To the best of our knowledge, we are the first to propose the combination of these two tasks in one single model. Our model is designed with focus on low latency to provide fast inference in real-time on a single consumer-grade GPU. During deployment, our model produces dense 3D point clouds with instance aware semantic labels from single high-resolution camera images. We evaluate our model on two popular autonomous driving benchmarks, i.e., Cityscapes and KITTI, and show competitive performance among other real-time capable methods. Source code is available at https://github.com/markusschoen/MGNet.

Figures (3)

• Figure 1: Example prediction of our model, (\ref{['fig:prediction:input']}) the input image which is fed to the network, (\ref{['fig:prediction:panoptic']}) the panoptic prediction on top of the input image, (\ref{['fig:prediction:depth']}) the monocular depth estimation and (\ref{['fig:prediction:pcl']}) the final 3D point cloud generated by the network in real-time.
• Figure 2: Overview of our MGNet framework. We encode image features using a lightweight backbone and employ a to maximize the receptive field. Three task-specific decoders are used to fuse backbone features of different scales using and . Task specific heads convert the fused features to logits. Object instances are represented as instance centers and pixel-wise offsets, i.e., 2D vectors, representing the x-y-offset to the instance center. The post-processing groups instances to the closest center given the offset prediction and assigns a semantic class based on majority voting. The depth prediction is first scaled using a dgc and then used to convert the panoptic prediction to the final 3D point cloud prediction. The pose network predicting 6 DOF poses between adjacent camera frames $\boldsymbol{I}_{t-1}, \boldsymbol{I}_{t}$ and $\boldsymbol{I}_{t+1}$, auxiliary heads for the multi-view photometric loss, and the uncertainty parameters for multi-task optimization are only used during training.
• Figure 3: Qualitative results on unseen images from the Cityscapes and KITTI dataset. The columns (from left to right) show the input image, the panoptic prediction, the monocular depth estimation, and the final 3D point cloud prediction, respectively. Instances are omitted in the 3D point clouds for better visualization. The first two rows show predictions on the Cityscapes dataset, while the last two rows show predictions on the KITTI dataset.