Extreme Model Compression for Edge Vision-Language Models: Sparse Temporal Token Fusion and Adaptive Neural Compression

TL;DR

This work addresses the challenge of deploying capable vision-language models on edge devices by introducing two adaptive compression techniques: Sparse Temporal Token Fusion (STTF) and Adaptive Neural Compression (ANC). The authors propose a three-stage pipeline: compression-aware pre-training on compact datasets, followed by stage-wise spatio-temporal redundancy elimination and model specialization via a unified compression operator. STTF dynamically merges temporal tokens and reuses representations, dramatically reducing token counts and enabling real-time inference, while ANC selects encoder branches based on scene complexity to scale computation with activity. The resulting TinyGPT-STTF and TinyGPT-ANC achieve competitive COCO captioning metrics at a fraction of parameters and FLOPs, with substantial speedups and energy savings on edge SoCs, thereby enabling practical on-device multimodal reasoning. The work demonstrates a principled path to edge-ready vision-language models, though it notes overfitting risks in STTF and routing stability concerns in ANC that warrant further regularization and stabilization strategies.

Abstract

The demand for edge AI in vision-language tasks requires models that achieve real-time performance on resource-constrained devices with limited power and memory. This paper proposes two adaptive compression techniques -- Sparse Temporal Token Fusion (STTF) and Adaptive Neural Compression (ANC) -- that integrate algorithmic innovations with hardware-aware optimizations. Unlike previous approaches relying on static pruning or uniform scaling, STTF dynamically reuses visual tokens through event-driven change detection, while ANC conditionally activates encoder branches via a learned router, enabling fine-grained adaptation to scene complexity. Our 3B-parameter TinyGPT-STTF achieves CIDEr 131.2, BLEU-4 0.38, METEOR 0.31, and ROUGE-L 0.56 on the COCO 2017 test set, surpassing LLaVA-1.5 7B by 17.6 CIDEr points while using 2.3x fewer parameters and 62x fewer on-device FLOPs. TinyGPT-ANC reaches CIDEr 128.5. On event-based vision tasks, STTF reduces average token count by 84% (from 196 to 31 tokens) while preserving 95.6% accuracy on the DVS128 Gesture dataset, and ANC cuts FLOPs by up to 90% in low-motion scenes. Compared to strong baselines, our models improve accuracy by up to 4.4% and reduce latency by up to 13x. These results enable efficient deployment of capable vision-language models on real-world edge devices.

Figures (6)

• Figure 1: Architecture of this model
• Figure 2: Accuracy over epoch of the STTF Algorithm.
• Figure 3: Loss over epoch of the STTF Algorithm.
• Figure 4: Confusion matrix using DVS128 Gesture dataset
• Figure 5: Accuracy graph of ANC.