A Unified 3D Object Perception Framework for Real-Time Outside-In Multi-Camera Systems

TL;DR

An adapted Sparse4D framework specifically optimized for large-scale infrastructure environments that leverages absolute world-coordinate geometric priors and introduces an occlusion-aware ReID embedding module to maintain identity stability across distributed sensor networks is presented.

Abstract

Accurate 3D object perception and multi-target multi-camera (MTMC) tracking are fundamental for the digital transformation of industrial infrastructure. However, transitioning "inside-out" autonomous driving models to "outside-in" static camera networks presents significant challenges due to heterogeneous camera placements and extreme occlusion. In this paper, we present an adapted Sparse4D framework specifically optimized for large-scale infrastructure environments. Our system leverages absolute world-coordinate geometric priors and introduces an occlusion-aware ReID embedding module to maintain identity stability across distributed sensor networks. To bridge the Sim2Real domain gap without manual labeling, we employ a generative data augmentation strategy using the NVIDIA COSMOS framework, creating diverse environmental styles that enhance the model's appearance-invariance. Evaluated on the AI City Challenge 2025 benchmark, our camera-only framework achieves a state-of-the-art HOTA of $45.22$. Furthermore, we address real-time deployment constraints by developing an optimized TensorRT plugin for Multi-Scale Deformable Aggregation (MSDA). Our hardware-accelerated implementation achieves a $2.15\times$ speedup on modern GPU architectures, enabling a single Blackwell-class GPU to support over 64 concurrent camera streams.

Figures (10)

• Figure 1: Comparison among three types of MTMC tracking methods: (a) conducts 2D detection separately and associates objects among different views by appearance-based ReID; (b) considers geometric constraints as well, besides appearance, for cross-view association; (c) achieves multi-view association in the early stage by feature-level aggregation. Figure is from the MCBLT paper Wang_2025_MCBLT.
• Figure 2: Overview of our 3D object perception model. The system takes multi-view images and camera parameters as input, processes them with an image encoder and camera encoder, and performs multi-view deformable aggregation, occlusion-aware feature fusion, and temporal query propagation to produce consistent 3D object predictions.
• Figure 3: Illustration of the occlusion-aware embedding (OAE) module. For each object anchor box, fixed and learned 3D keypoints are projected into each camera view. Motion compensation is applied using the estimated object velocity, producing temporally aligned keypoints. Multi-keypoint, multi-scale, and multi-view image features are aggregated to generate instance-level embeddings. Visibility-aware weighting ensures that occluded or low-quality views contribute less to the final feature representation.
• Figure 4: Visualization of the COSMOS-based data augmentation. By applying diverse style transfers to synthetic multi-camera sequences, we generate varied environmental conditions while preserving the exact 3D geometric ground truth.
• Figure 5: Embedding distance distribution for ID matches vs. mismatches