Model Optimization for Multi-Camera 3D Detection and Tracking

TL;DR

This work evaluates Sparse4D as a deployment-oriented baseline for outside-in indoor multi-target multi-camera perception, focusing on practical constraints such as reduced frame rates and low-precision inference. It introduces AvgTrackDur to quantify identity stability and studies four deployment tracks: low-FPS robustness, post-training quantization, WILDTRACK adaptation, and Transformer Engine mixed-precision fine-tuning. Key findings show that identity association is the main bottleneck at very low FPS, selective PTQ on backbone/neck offers favorable speed-accuracy trade-offs, FP8 generally preserves localization more robustly than INT8, and Transformer Engine provides large latency gains at the risk of unstable identity propagation. The results offer actionable guidance for deploying sparse-query, world-frame MTMC systems in indoor environments and highlight directions for future work in association-aware training and stability-focused evaluation.

Abstract

Outside-in multi-camera perception is increasingly important in indoor environments, where networks of static cameras must support multi-target tracking under occlusion and heterogeneous viewpoints. We evaluate Sparse4D, a query-based spatiotemporal 3D detection and tracking framework that fuses multi-view features in a shared world frame and propagates sparse object queries via instance memory. We study reduced input frame rates, post-training quantization (INT8 and FP8), transfer to the WILDTRACK benchmark, and Transformer Engine mixed-precision fine-tuning. To better capture identity stability, we report Average Track Duration (AvgTrackDur), which measures identity persistence in seconds. Sparse4D remains stable under moderate FPS reductions, but below 2 FPS, identity association collapses even when detections are stable. Selective quantization of the backbone and neck offers the best speed-accuracy trade-off, while attention-related modules are consistently sensitive to low precision. On WILDTRACK, low-FPS pretraining yields large zero-shot gains over the base checkpoint, while small-scale fine-tuning provides limited additional benefit. Transformer Engine mixed precision reduces latency and improves camera scalability, but can destabilize identity propagation, motivating stability-aware validation.

Figures (5)

• Figure 1: High-level overview of the Sparse4D-based model. Multi-view images and camera parameters are encoded into multi-scale feature maps. Sparse 3D queries aggregate multi-view evidence via deformable sampling and are updated by a spatiotemporal transformer with temporal memory. Prediction heads output per-object 3D state and an embedding used for association. This figure is originally from wang2026unified3dobjectperception.
• Figure 2: Overview of the INT8 and FP8 PTQ pipelines used to build TensorRT engines from the baseline ONNX and evaluate latency and tracking performance on AI City 2025 Warehouse 14.
• Figure 3: Examples of appearance styles generated using COSMOS Transfer 1 for synthetic MTMC training clips.
• Figure 4: Zero-shot detection visualization on WILDTRACK test frame 360 (confidence threshold 0.2), using the COSMOS low-FPS checkpoint.
• Figure 5: Qualitative visualization of the Mixed-Precision model on the NVIDIA H200.