Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

EdgeFlex-Transformer: Transformer Inference for Edge Devices

Shoaib Mohammad, Guanqun Song, Ting Zhu

TL;DR

EdgeFlex-Transformer tackles the challenge of running large Vision Transformers on memory- and compute-constrained edge devices. It introduces a training-free, multi-stage pipeline that combines activation profiling, memory-aware pruning, mixed-precision execution, and activation-aware quantization to compress and accelerate ViTs. Applied to a ViT-Huge backbone, the method yields substantial memory and latency reductions on CIFAR-10 while maintaining or improving accuracy, enabling practical on-device inference. The approach offers a practical path toward efficient edge transformer deployment and motivates future work on dynamic sparsity, MoE integration, and real-time edge scheduling.

Abstract

Deploying large-scale transformer models on edge devices presents significant challenges due to strict constraints on memory, compute, and latency. In this work, we propose a lightweight yet effective multi-stage optimization pipeline designed to compress and accelerate Vision Transformers (ViTs) for deployment in resource-constrained environments. Our methodology combines activation profiling, memory-aware pruning, selective mixed-precision execution, and activation-aware quantization (AWQ) to reduce the model's memory footprint without requiring costly retraining or task-specific fine-tuning. Starting from a ViT-Huge backbone with 632 million parameters, we first identify low-importance channels using activation statistics collected via forward hooks, followed by structured pruning to shrink the MLP layers under a target memory budget. We further apply FP16 conversion to selected components and leverage AWQ to quantize the remaining model weights and activations to INT8 with minimal accuracy degradation. Our experiments on CIFAR-10 demonstrate that the fully optimized model achieves a 76% reduction in peak memory usage and over 6x lower latency, while retaining or even improving accuracy compared to the original FP32 baseline. This framework offers a practical path toward efficient transformer inference on edge platforms, and opens future avenues for integrating dynamic sparsity and Mixture-of-Experts (MoE) architectures to further scale performance across diverse tasks.

EdgeFlex-Transformer: Transformer Inference for Edge Devices

TL;DR

EdgeFlex-Transformer tackles the challenge of running large Vision Transformers on memory- and compute-constrained edge devices. It introduces a training-free, multi-stage pipeline that combines activation profiling, memory-aware pruning, mixed-precision execution, and activation-aware quantization to compress and accelerate ViTs. Applied to a ViT-Huge backbone, the method yields substantial memory and latency reductions on CIFAR-10 while maintaining or improving accuracy, enabling practical on-device inference. The approach offers a practical path toward efficient edge transformer deployment and motivates future work on dynamic sparsity, MoE integration, and real-time edge scheduling.

Abstract

Deploying large-scale transformer models on edge devices presents significant challenges due to strict constraints on memory, compute, and latency. In this work, we propose a lightweight yet effective multi-stage optimization pipeline designed to compress and accelerate Vision Transformers (ViTs) for deployment in resource-constrained environments. Our methodology combines activation profiling, memory-aware pruning, selective mixed-precision execution, and activation-aware quantization (AWQ) to reduce the model's memory footprint without requiring costly retraining or task-specific fine-tuning. Starting from a ViT-Huge backbone with 632 million parameters, we first identify low-importance channels using activation statistics collected via forward hooks, followed by structured pruning to shrink the MLP layers under a target memory budget. We further apply FP16 conversion to selected components and leverage AWQ to quantize the remaining model weights and activations to INT8 with minimal accuracy degradation. Our experiments on CIFAR-10 demonstrate that the fully optimized model achieves a 76% reduction in peak memory usage and over 6x lower latency, while retaining or even improving accuracy compared to the original FP32 baseline. This framework offers a practical path toward efficient transformer inference on edge platforms, and opens future avenues for integrating dynamic sparsity and Mixture-of-Experts (MoE) architectures to further scale performance across diverse tasks.
Paper Structure (23 sections, 3 figures)

This paper contains 23 sections, 3 figures.

Table of Contents

  1. Introduction
  2. Related Works
  3. IoT Infrastructure and Security Context
  4. Model Pruning
  5. Quantization
  6. Knowledge Distillation
  7. Neural Architecture Search (NAS)
  8. Inference Frameworks
  9. System Infrastructure for Edge Intelligence
  10. Motivation
  11. Challenges
  12. Problem definition
  13. Methodology
  14. Activation Profiling:
  15. Memory-Aware Pruning:
  16. ...and 8 more sections

Figures (3)

  • Figure 1: EdgeFlex: Framework pipeline
  • Figure 2: ViT Huge model architecture.
  • Figure 3: Experimental results on 10,000 test samples