Lightweight Transformer Architectures for Edge Devices in Real-Time Applications

TL;DR

This work surveys lightweight transformer architectures and optimization strategies for edge devices, addressing the core challenge of enabling real-time AI under strict resource constraints by leveraging distillation, pruning, quantization, and hardware-aware design. It catalogues architecture families (e.g., MobileBERT, TinyBERT, DistilBERT, EfficientFormer, EdgeFormer, MobileViT) and efficient attention mechanisms, then several industry-standard tooling and platforms to realize on-device inference. The analysis presents strong empirical evidence that 75–96% of full-model accuracy is achievable with 4–10× model size reduction and 3–9× latency reductions, often on devices consuming 2–5 W, with memory bandwidth often dictating throughput. Key findings highlight memory–latency tradeoffs, quantization sweet spots, and the superior gains from two-stage distillation and hardware-aware NAS, providing a practical deployment pipeline that yields 8–12× size reduction with less than 2% accuracy loss. The results inform practical guidance for practitioners and outline future directions in long-context processing, multimodal edge AI, and automated compression pipelines.

Abstract

The deployment of transformer-based models on resource-constrained edge devices represents a critical challenge in enabling real-time artificial intelligence applications. This comprehensive survey examines lightweight transformer architectures specifically designed for edge deployment, analyzing recent advances in model compression, quantization, pruning, and knowledge distillation techniques. We systematically review prominent lightweight variants including MobileBERT, TinyBERT, DistilBERT, EfficientFormer, EdgeFormer, and MobileViT, providing detailed performance benchmarks on standard datasets such as GLUE, SQuAD, ImageNet-1K, and COCO. Our analysis encompasses current industry adoption patterns across major hardware platforms (NVIDIA Jetson, Qualcomm Snapdragon, Apple Neural Engine, ARM architectures), deployment frameworks (TensorFlow Lite, ONNX Runtime, PyTorch Mobile, CoreML), and optimization strategies. Experimental results demonstrate that modern lightweight transformers can achieve 75-96% of full-model accuracy while reducing model size by 4-10x and inference latency by 3-9x, enabling deployment on devices with as little as 2-5W power consumption. We identify sparse attention mechanisms, mixed-precision quantization (INT8/FP16), and hardware-aware neural architecture search as the most effective optimization strategies. Novel findings include memory-bandwidth bottleneck analysis revealing 15-40M parameter models achieve optimal hardware utilization (60-75% efficiency), quantization sweet spots for different model types, and comprehensive energy efficiency profiling across edge platforms. We establish real-time performance boundaries and provide a practical 6-step deployment pipeline achieving 8-12x size reduction with less than 2% accuracy degradation.

Figures (7)

• Figure 1: Edge AI Hardware Performance Comparison (TOPS). Platforms range from ultra-low-power IoT (2-5W, 4-35 TOPS) to high-performance robotics (15-60W, 67-275 TOPS), enabling diverse deployment scenarios.
• Figure 2: Edge Deployment Framework Comparison. TensorFlow Lite and CoreML excel at hardware-specific optimization, while ONNX Runtime provides superior cross-platform compatibility. PyTorch Mobile offers easiest development workflow for PyTorch users.
• Figure 3: Model Size vs. GLUE Score Tradeoff. Lightweight models achieve comparable accuracy (75.8-79.4) with 13-60% of BERT-base parameters (110M), demonstrating effective compression.
• Figure 4: NLP Model Inference Latency on Pixel 4 Mobile Device. Lightweight models achieve 7-14$\times$ speedup versus BERT-base, enabling real-time on-device inference.
• Figure 5: Vision Transformer Accuracy-Latency Tradeoff on iPhone 12. EfficientFormer models achieve superior Pareto frontier, matching MobileNet speed at higher accuracy or achieving 83%+ accuracy at <7ms latency.