SPORTS: Simultaneous Panoptic Odometry, Rendering, Tracking and Segmentation for Urban Scenes Understanding

TL;DR

SPORTS introduces a unified framework that simultaneously performs VPS, VO, and SR from monocular video to enable holistic urban scene understanding. It features an attention-based adaptive geometry fusion (AG Fusion) and a post-matching strategy to robustly align cross-frame features and track instances, complemented by a point-based neural rendering backend that yields high-fidelity novel views. Through iterative interaction between VPS and VO, and a dense rendering pipeline built on neural descriptors, SPORTS achieves state-of-the-art performance across odometry, segmentation, tracking, and view synthesis on VKITTI2, VIPER, and KITTI. The work demonstrates the practical value of tightly coupling multi-task perception and rendering for embodied-AI in dynamic urban environments.

Abstract

The scene perception, understanding, and simulation are fundamental techniques for embodied-AI agents, while existing solutions are still prone to segmentation deficiency, dynamic objects' interference, sensor data sparsity, and view-limitation problems. This paper proposes a novel framework, named SPORTS, for holistic scene understanding via tightly integrating Video Panoptic Segmentation (VPS), Visual Odometry (VO), and Scene Rendering (SR) tasks into an iterative and unified perspective. Firstly, VPS designs an adaptive attention-based geometric fusion mechanism to align cross-frame features via enrolling the pose, depth, and optical flow modality, which automatically adjust feature maps for different decoding stages. And a post-matching strategy is integrated to improve identities tracking. In VO, panoptic segmentation results from VPS are combined with the optical flow map to improve the confidence estimation of dynamic objects, which enhances the accuracy of the camera pose estimation and completeness of the depth map generation via the learning-based paradigm. Furthermore, the point-based rendering of SR is beneficial from VO, transforming sparse point clouds into neural fields to synthesize high-fidelity RGB views and twin panoptic views. Extensive experiments on three public datasets demonstrate that our attention-based feature fusion outperforms most existing state-of-the-art methods on the odometry, tracking, segmentation, and novel view synthesis tasks.

Figures (11)

• Figure 1: The Overview of Our SPORTS Framework. The framework solely receives monocular video as input, iteratively determines the temporal panoptic segmentation result, camera's poses and sparse point cloud maps, finally generates the photo-realistic and panoptic scene rendering result.
• Figure 2: The Diagram of Our Enhanced Video Panoptic Segmentation (VPS) Module. We warp the feature map $\mathbf{z}_{t-1}$ generated by the neck of the reference frame $\mathbf{I}_{t-1}$ via leveraging poses, depth, and optical flow. The warped feature map $\mathbf{z}^w_{t-1}$ is then fused with the feature map $\mathbf{z}_{t}$ of the key frame $\mathbf{I}_{t}$ through $i$ fusion modules, resulting in i fused feature maps $\{\mathbf{z}^{1*}_{t}, \mathbf{z}^{2*}_{t}, ...,\mathbf{z}^{i*}_{t}\}$. This approach adaptively matches the feature maps with the kernel update heads by paying attention to the geometric knowledge, renamed as AG Fusion (Attention-based Adaptive Geometry Fusion, highlighted in red lines). Totally, our VPS module is enhanced via four levels of information exchanges (abbreviated as EX), including AG Fusion (EX 1), Kernels Fusion (EX 2), Kernels Linking (EX 3), and Kernels Embedding (EX 4), more details about AG Fusion is elaborated in Section \ref{['sec::sec::AGFusion']} and Fig. \ref{['fig:FusionModule']}.
• Figure 3: An Illustration Diagram of Our Proposed AG Fusion Module.$W$, $H$, and $C$ represent the width, height, and number of channels of the input features, respectively. The feature map $\mathbf{z}_t$ and the warped feature map $\mathbf{z}^\mathrm{w}_{t-1}$ are concatenated and made up by multiple pyramid scales. For each scale, the feature vector $\mathrm{z}_i$ is processed in parallel by $i_{th}$ branches, where the channel dimension of $\mathrm{z}_i$ is 2$C$ as well. The $\mathrm{z}_i$ is re-weighted using channel-attention mechanism and then passed through a convolutional layer of kernel size $n \times n \times 2C$. The fused feature map $\textbf{z}^{i*}_t$ for a branch is obtained by processing all the pyramid scale of feature maps.
• Figure 4: An Illustrative Diagram of Our Post-Matching Process.
• Figure 5: The Diagram of Our Enhanced VO Module. Our method combines the video panoptic segmentation information, $S^{\text{pan}}_t$, with optical flow for the motion estimation, iteratively enhances the accuracy of camera pose estimation. The VO module outputs final pose, depth map and optical flow to facilitate the VPS in return.