ViTMAlis: Towards Latency-Critical Mobile Video Analytics with Vision Transformers

TL;DR

This work tackles the latency challenges of edge-assisted mobile video analytics when using vision transformers (ViTs) for dense prediction. It introduces ViTMAlis, a ViT-native offloading framework that uses dynamic mixed-resolution inference to reduce both transmission and server-side inference delays, adapting to frame content and network conditions without retraining ViT backbones. The system combines on-device region analysis, content-aware performance estimation, and Pareto-frontier-based configuration selection to achieve substantial end-to-end latency reductions while maintaining or improving rendering accuracy. The authors implement a functional prototype on commodity hardware and demonstrate significant improvements over strong baselines across diverse mobile video scenarios, establishing a practical pathway for latency-sensitive mobile intelligence with ViTs.

Abstract

Edge-assisted mobile video analytics (MVA) applications are increasingly shifting from using vision models based on convolutional neural networks (CNNs) to those built on vision transformers (ViTs) to leverage their superior global context modeling and generalization capabilities. However, deploying these advanced models in latency-critical MVA scenarios presents significant challenges. Unlike traditional CNN-based offloading paradigms where network transmission is the primary bottleneck, ViT-based systems are constrained by substantial inference delays, particularly for dense prediction tasks where the need for high-resolution inputs exacerbates the inherent quadratic computational complexity of ViTs. To address these challenges, we propose a dynamic mixed-resolution inference strategy tailored for ViT-backboned dense prediction models, enabling flexible runtime trade-offs between speed and accuracy. Building on this, we introduce ViTMAlis, a ViT-native device-to-edge offloading framework that dynamically adapts to network conditions and video content to jointly reduce transmission and inference delays. We implement a fully functional prototype of ViTMAlis on commodity mobile and edge devices. Extensive experiments demonstrate that, compared to state-of-the-art accuracy-centric, content-aware, and latency-adaptive baselines, ViTMAlis significantly reduces end-to-end offloading latency while improving user-perceived rendering accuracy, providing a practical foundation for next-generation mobile intelligence.

Figures (12)

• Figure 1: Edge-assisted mobile video analytics.
• Figure 2: CNNs and ViTs employ different strategies to extract visual features from images.
• Figure 3: Inference-compatible region partitioning.
• Figure 4: An example of how dynamic mixed-resolution inference works for ViTDet.
• Figure 5: Impacts of RPs.