Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Integer-only Quantized Transformers for Embedded FPGA-based Time-series Forecasting in AIoT

Tianheng Ling, Chao Qian, Gregor Schiele

TL;DR

The paper tackles on-device time-series forecasting with Transformers by employing integer-only quantization and Quantization-Aware Training, implemented via a software-hardware co-design on an embedded FPGA. It introduces an FPGA-friendly Transformer with parameter simplifications and encoder enhancements, including scaling integration and Batch Normalization, and provides RTL VHDL templates for 8-, 6-, and 4-bit quantization. The approach demonstrates that a 4-bit quantized Transformer can deliver only a 0.63% increase in test loss while achieving up to 132.33x faster inference and 48.19x lower energy consumption compared with an 8-bit baseline on the Spartan-7 XC7S15, albeit with complex optimization trade-offs. This work presents a practical pathway for deploying accurate, resource-efficient Transformers in AIoT time-series forecasting, and offers a reusable hardware template set and quantization framework for future mixed-precision explorations.

Abstract

This paper presents the design of a hardware accelerator for Transformers, optimized for on-device time-series forecasting in AIoT systems. It integrates integer-only quantization and Quantization-Aware Training with optimized hardware designs to realize 6-bit and 4-bit quantized Transformer models, which achieved precision comparable to 8-bit quantized models from related research. Utilizing a complete implementation on an embedded FPGA (Xilinx Spartan-7 XC7S15), we examine the feasibility of deploying Transformer models on embedded IoT devices. This includes a thorough analysis of achievable precision, resource utilization, timing, power, and energy consumption for on-device inference. Our results indicate that while sufficient performance can be attained, the optimization process is not trivial. For instance, reducing the quantization bitwidth does not consistently result in decreased latency or energy consumption, underscoring the necessity of systematically exploring various optimization combinations. Compared to an 8-bit quantized Transformer model in related studies, our 4-bit quantized Transformer model increases test loss by only 0.63%, operates up to 132.33x faster, and consumes 48.19x less energy. Relevant source code is provided in the accompanying GitHub repository\footnote{https://github.com/tianheng-ling/TinyTransformer4TS}.

Integer-only Quantized Transformers for Embedded FPGA-based Time-series Forecasting in AIoT

TL;DR

The paper tackles on-device time-series forecasting with Transformers by employing integer-only quantization and Quantization-Aware Training, implemented via a software-hardware co-design on an embedded FPGA. It introduces an FPGA-friendly Transformer with parameter simplifications and encoder enhancements, including scaling integration and Batch Normalization, and provides RTL VHDL templates for 8-, 6-, and 4-bit quantization. The approach demonstrates that a 4-bit quantized Transformer can deliver only a 0.63% increase in test loss while achieving up to 132.33x faster inference and 48.19x lower energy consumption compared with an 8-bit baseline on the Spartan-7 XC7S15, albeit with complex optimization trade-offs. This work presents a practical pathway for deploying accurate, resource-efficient Transformers in AIoT time-series forecasting, and offers a reusable hardware template set and quantization framework for future mixed-precision explorations.

Abstract

This paper presents the design of a hardware accelerator for Transformers, optimized for on-device time-series forecasting in AIoT systems. It integrates integer-only quantization and Quantization-Aware Training with optimized hardware designs to realize 6-bit and 4-bit quantized Transformer models, which achieved precision comparable to 8-bit quantized models from related research. Utilizing a complete implementation on an embedded FPGA (Xilinx Spartan-7 XC7S15), we examine the feasibility of deploying Transformer models on embedded IoT devices. This includes a thorough analysis of achievable precision, resource utilization, timing, power, and energy consumption for on-device inference. Our results indicate that while sufficient performance can be attained, the optimization process is not trivial. For instance, reducing the quantization bitwidth does not consistently result in decreased latency or energy consumption, underscoring the necessity of systematically exploring various optimization combinations. Compared to an 8-bit quantized Transformer model in related studies, our 4-bit quantized Transformer model increases test loss by only 0.63%, operates up to 132.33x faster, and consumes 48.19x less energy. Relevant source code is provided in the accompanying GitHub repository\footnote{https://github.com/tianheng-ling/TinyTransformer4TS}.
Paper Structure (24 sections, 19 equations, 3 figures, 4 tables, 1 algorithm)

This paper contains 24 sections, 19 equations, 3 figures, 4 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. FPGA-friendly Transformer
  3. Parameters Simplification
  4. Encoder Enhancements
  5. Scaling Integration
  6. Batch Normalization
  7. Integer-only Quantization
  8. Software-Hardware Co-Design
  9. Addition Component
  10. Matrix Multiplication Component
  11. Softmax Component
  12. Batch Normalization Component
  13. Global Average Pooling Component
  14. Experiments and Results
  15. Datasets and Data Processing
  16. ...and 9 more sections

Figures (3)

  • Figure 1: The Architecture of the Transformer Model
  • Figure 2: PeMS Dataset: RMSE Variation
  • Figure 3: AirU Dataset: RMSE Variation