Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Ask the Expert: Collaborative Inference for Vision Transformers with Near-Edge Accelerators

Hao Liu, Suhaib A. Fahmy

Abstract

Deploying Vision Transformers on edge devices is challenging due to their high computational complexity, while full offloading to cloud resources presents significant latency overheads. We propose a novel collaborative inference framework, which orchestrates a lightweight generalist ViT on an edge device and multiple medium-sized expert ViTs on a near-edge accelerator. A novel routing mechanism uses the edge model's Top-$\mathit{k}$ predictions to dynamically select the most relevant expert for samples with low confidence. We further design a progressive specialist training strategy to enhance expert accuracy on dataset subsets. Extensive experiments on the CIFAR-100 dataset using a real-world edge and near-edge testbed demonstrate the superiority of our framework. Specifically, the proposed training strategy improves expert specialization accuracy by 4.12% on target subsets and enhances overall accuracy by 2.76% over static experts. Moreover, our method reduces latency by up to 45% compared to edge execution, and energy consumption by up to 46% compared to just near-edge offload.

Ask the Expert: Collaborative Inference for Vision Transformers with Near-Edge Accelerators

Abstract

Deploying Vision Transformers on edge devices is challenging due to their high computational complexity, while full offloading to cloud resources presents significant latency overheads. We propose a novel collaborative inference framework, which orchestrates a lightweight generalist ViT on an edge device and multiple medium-sized expert ViTs on a near-edge accelerator. A novel routing mechanism uses the edge model's Top- predictions to dynamically select the most relevant expert for samples with low confidence. We further design a progressive specialist training strategy to enhance expert accuracy on dataset subsets. Extensive experiments on the CIFAR-100 dataset using a real-world edge and near-edge testbed demonstrate the superiority of our framework. Specifically, the proposed training strategy improves expert specialization accuracy by 4.12% on target subsets and enhances overall accuracy by 2.76% over static experts. Moreover, our method reduces latency by up to 45% compared to edge execution, and energy consumption by up to 46% compared to just near-edge offload.
Paper Structure (18 sections, 3 equations, 7 figures, 7 tables, 1 algorithm)

This paper contains 18 sections, 3 equations, 7 figures, 7 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Motivation
  4. Proposed Method
  5. Overview
  6. Problem Formulation
  7. Progressive Specialist Training Strategy
  8. Top-$\mathit{k}$ Routing Mechanism
  9. Evaluation
  10. Experimental Setting
  11. Effectiveness of Progressive Specialist Training
  12. Performance Analysis under Varying Confidence Thresholds
  13. Discussion
  14. Conclusions
  15. Appendix
  16. ...and 3 more sections

Figures (7)

  • Figure 1: Latency-accuracy trade-off comparison for varying model sizes (DeiT-3/4/6H) on edge (RPi5 or Orin Nano), cloud (V100), and our framework. Data points represent these models in increasing order of complexity and accuracy.
  • Figure 2: Top-$\mathit{k}$ accuracy of ViT models of various sizes on CIFAR-100 dataset.
  • Figure 3: Overview of the proposed framework.
  • Figure 4: Accuracy comparison of Edge/Near-Edge-Only, Generalist Co-inference, and our method for DeiT-4H ($\leftarrow$) and DeiT-6H ($\rightarrow$) across varying confidence thresholds.
  • Figure 5: Normalized latency-energy comparison of Edge-Only, Near-Edge-Only, and our method for DeiT-4H and DeiT-6H across varying confidence thresholds.
  • ...and 2 more figures