Resource-Efficient Multiview Perception: Integrating Semantic Masking with Masked Autoencoders

TL;DR

This work tackles bandwidth and computational constraints in distributed multiview perception by introducing semantic-guided masking at edge cameras and MAE-based reconstruction at an edge server. By prioritizing informative image patches through a pre-trained segmentation model and a tunable power function, the method maintains strong detection and tracking performance while reducing transmitted data, with a patch-based transmission scheme described by $f(x) = x^\kappa$ and related probability sampling. Evaluations on MultiviewX and Wildtrack show competitive MODA/MODP/MOTA/MOTP scores against state-of-the-art, even at high masking ratios, and achieve up to a 13.33x reduction in data volume due to masking and downsampling. The approach also demonstrates resilience to camera dropout and is suitable for resource-limited platforms, with edge-server MAE reconstruction enabling robust BEV fusion and JDE-based tracking; future work may extend this framework to non-terrestrial networks.

Abstract

Multiview systems have become a key technology in modern computer vision, offering advanced capabilities in scene understanding and analysis. However, these systems face critical challenges in bandwidth limitations and computational constraints, particularly for resource-limited camera nodes like drones. This paper presents a novel approach for communication-efficient distributed multiview detection and tracking using masked autoencoders (MAEs). We introduce a semantic-guided masking strategy that leverages pre-trained segmentation models and a tunable power function to prioritize informative image regions. This approach, combined with an MAE, reduces communication overhead while preserving essential visual information. We evaluate our method on both virtual and real-world multiview datasets, demonstrating comparable performance in terms of detection and tracking performance metrics compared to state-of-the-art techniques, even at high masking ratios. Our selective masking algorithm outperforms random masking, maintaining higher accuracy and precision as the masking ratio increases. Furthermore, our approach achieves a significant reduction in transmission data volume compared to baseline methods, thereby balancing multiview tracking performance with communication efficiency.

Figures (7)

• Figure 1: System model of the proposed distributed multiview pedestrian detection and tracking framework. Multiple cameras capture images, which are resized and undergo semantic-guided masking before wireless transmission. The received masked images are processed by an MAE with an encoder-decoder architecture. The reconstructed images are then fed into a combined detection and tracking module hou2020multiviewteepe2023earlybird, producing both detection and tracking outputs for the observed targets.
• Figure 2: Visual comparison of random and semantically-guided masking techniques for image reconstruction using an MAE. The masking ratio is set at 0.8, and the $\kappa = 0.1$ for semantically-guided masking. (a) Original input image. (b) Randomly masked image. (c) Reconstruction from random masking. (d) Instance segmentation of the original image. (e) Heatmap of patch importance based on instance segmentation results. (f) semantically-guided masked image. (g) Reconstruction from semantically-guided masking. The semantically-guided masking technique (d-g) demonstrates improved reconstruction quality compared to random masking (b-c), particularly in areas of high semantic importance as indicated by the heatmap (e).
• Figure 3: A plot of the MODA as the masking ratio increases for the Wildtrack dataset. For the semantically-guided masking $\kappa = 0.15$.
• Figure 4: A plot of the MOTA as the masking ratio increases for the Wildtrack dataset. For the semantically-guided masking $\kappa = 0.15$.
• Figure 5: A plot of the MODA as the masking ratio increases for the MultiviewX dataset. For the semantically-guided masking $\kappa = 0.15$.