Joint stereo 3D object detection and implicit surface reconstruction

TL;DR

This work tackles estimating object orientation in $SO(3)$ and recovering implicit rigid shapes from stereo RGB images. It introduces S-3D-RCNN, a two-stage framework where a proposal detector provides 3D cuboid candidates and Ego-Net++ predicts egocentric orientation and, for rigid objects, a resolution-agnostic implicit shape via a surface-hallucination pipeline built on visible-surface representations and normalization. Central contributions include the Labor-free Intermediate Representations (IGRs), the multi-view PPC extension, the visible-surface and hallucination modules, and new KITTI metrics $AP_{MMD}$ and $MMDTP$ for evaluating surface-level quality. Empirically, the approach achieves state-of-the-art system-level performance on KITTI for joint 3D object detection and orientation estimation and delivers credible implicit-shape reconstructions, highlighting strong potential for outdoor AR and autonomous driving applications. The method leverages stereo geometry to surpass monocular baselines and delivers a mesh-based, resolution-agnostic description within object boxes, advancing outdoor 3D scene understanding.

Abstract

We present a new learning-based framework S-3D-RCNN that can recover accurate object orientation in SO(3) and simultaneously predict implicit rigid shapes from stereo RGB images. For orientation estimation, in contrast to previous studies that map local appearance to observation angles, we propose a progressive approach by extracting meaningful Intermediate Geometrical Representations (IGRs). This approach features a deep model that transforms perceived intensities from one or two views to object part coordinates to achieve direct egocentric object orientation estimation in the camera coordinate system. To further achieve finer description inside 3D bounding boxes, we investigate the implicit shape estimation problem from stereo images. We model visible object surfaces by designing a point-based representation, augmenting IGRs to explicitly address the unseen surface hallucination problem. Extensive experiments validate the effectiveness of the proposed IGRs, and S-3D-RCNN achieves superior 3D scene understanding performance. We also designed new metrics on the KITTI benchmark for our evaluation of implicit shape estimation.

Figures (21)

• Figure 1: Given a pair of stereo RGB images, S-3D-RCNN can detect 3D objects and predict implicit rigid shapes with one forward pass. (a) Alpha-blended image pair to show the disparities. (b) 3D object proposals shown as 3D bounding boxes. (c) Shape predictions for the detected objects. (d) Estimated surface normal of the nearby object where the red ray indicates incorrect reflection effects with only the 3D bounding box prediction. (e) The predicted implicit shape supports a spatially varying resolution.
• Figure 2: Diagram of $\mathcal{E}$ (Ego-Net++). $\mathcal{E}$ performs orientation and rigid shape estimation with intermediate geometric representations. A local cost volume is constructed from instance features to estimate disparities. The visible surface coordinates are computed from the predicted disparities and an estimated mask, and then normalized to a canonical coordinate system. An encoder-decoder component $Ha$ infers the missing surface of the object. The complete surface coordinates are passed to an encoder to extract an implicit shape vector, which can be used by a decoder for resolution-agnostic mesh extraction. For orientation estimation, a zoomed-in view is shown in Fig. \ref{['fig:egonet++_orientaion']}. FCN stands for a fully convolutional network module. The 2D part coordinates are lifted to 3D coordinates by $Li$ in Eq. \ref{['eq:comp_graph_ego_net++']}.
• Figure 3: The proposal model $\mathcal{D}$ of S-3D-RCNN. In this implementation, a volumetric 3D scene representation is built from semantic features and cost-volume-based geometric features similar to chen2020dsgn. An anchor-based object detector processes the Bird's Eye View feature maps to generate 3D object proposals. Note that Ego-Net++ $\mathcal{E}$ is agnostic to the design choice of $\mathcal{D}$ and can be used with other 3D object detectors.
• Figure 4: Local appearance cannot uniquely determine egocentric pose. Existing solutions first estimate an allocentric pose in the object coordinate system (blue) and convert it to an egocentric pose in the camera coordinate system (green) based on the object location.
• Figure 5: Model architecture of Ego-Net. A fully convolutional model $\mathcal{H}$ regresses part heatmaps from a 2D patch of a proposal. The heatmaps are mapped to local coordinates with several strided convolution layers. The local coordinates are transformed to screen coordinates $\phi_g(\mathbf{x}_i)$ and mapped to a point-based 3D representation $\psi(\mathbf{x}_i)$ of a cuboid, whose orientation directly represents egocentric pose in the camera coordinate system. k=33 when q=2 as in Sec. \ref{['sec:igrs']}.