Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Torch2Chip: An End-to-end Customizable Deep Neural Network Compression and Deployment Toolkit for Prototype Hardware Accelerator Design

Jian Meng, Yuan Liao, Anupreetham Anupreetham, Ahmed Hasssan, Shixing Yu, Han-sok Suh, Xiaofeng Hu, Jae-sun Seo

TL;DR

Torch2Chip addresses the disconnect between DL frameworks, state-of-the-art quantization algorithms, and prototype hardware design by delivering an end-to-end, highly customizable toolkit for compression and deployment. It introduces a Dual-Path, hierarchical quantization framework that separates training and integer-only inference, enabling user-defined quantizers and automatic fusion/parameter extraction to deployment-ready formats. The approach supports CNNs and ViTs, includes automatic 8-bit and sub-8-bit fusion, LUT-based non-linear approximations, and self-supervised foundation model pre-training to bolster compression gains. The open-source toolkit demonstrates competitive performance on ImageNet-1K and CIFAR-10, enables sparse and SSL-powered transfer learning, and exports multiple hardware-friendly formats, substantially accelerating prototype ASIC/FPGA design workflows.

Abstract

The development of model compression is continuously motivated by the evolution of various neural network accelerators with ASIC or FPGA. On the algorithm side, the ultimate goal of quantization or pruning is accelerating the expensive DNN computations on low-power hardware. However, such a "design-and-deploy" workflow faces under-explored challenges in the current hardware-algorithm co-design community. First, although the state-of-the-art quantization algorithm can achieve low precision with negligible degradation of accuracy, the latest deep learning framework (e.g., PyTorch) can only support non-customizable 8-bit precision, data format, and parameter extraction. Secondly, the objective of quantization is to enable the computation with low-precision data. However, the current SoTA algorithm treats the quantized integer as an intermediate result, while the final output of the quantizer is the "discretized" floating-point values, ignoring the practical needs and adding additional workload to hardware designers for integer parameter extraction and layer fusion. Finally, the compression toolkits designed by the industry are constrained to their in-house product or a handful of algorithms. The limited degree of freedom in the current toolkit and the under-explored customization hinder the prototype ASIC or FPGA-based accelerator design. To resolve these challenges, we propose Torch2Chip, an open-sourced, fully customizable, and high-performance toolkit that supports user-designed compression followed by automatic model fusion and parameter extraction. Torch2Chip incorporates the hierarchical design workflow, and the user-customized compression algorithm will be directly packed into the deployment-ready format for prototype chip verification with either CNN or vision transformer (ViT). The code is available at https://github.com/SeoLabCornell/torch2chip.

Torch2Chip: An End-to-end Customizable Deep Neural Network Compression and Deployment Toolkit for Prototype Hardware Accelerator Design

TL;DR

Torch2Chip addresses the disconnect between DL frameworks, state-of-the-art quantization algorithms, and prototype hardware design by delivering an end-to-end, highly customizable toolkit for compression and deployment. It introduces a Dual-Path, hierarchical quantization framework that separates training and integer-only inference, enabling user-defined quantizers and automatic fusion/parameter extraction to deployment-ready formats. The approach supports CNNs and ViTs, includes automatic 8-bit and sub-8-bit fusion, LUT-based non-linear approximations, and self-supervised foundation model pre-training to bolster compression gains. The open-source toolkit demonstrates competitive performance on ImageNet-1K and CIFAR-10, enables sparse and SSL-powered transfer learning, and exports multiple hardware-friendly formats, substantially accelerating prototype ASIC/FPGA design workflows.

Abstract

The development of model compression is continuously motivated by the evolution of various neural network accelerators with ASIC or FPGA. On the algorithm side, the ultimate goal of quantization or pruning is accelerating the expensive DNN computations on low-power hardware. However, such a "design-and-deploy" workflow faces under-explored challenges in the current hardware-algorithm co-design community. First, although the state-of-the-art quantization algorithm can achieve low precision with negligible degradation of accuracy, the latest deep learning framework (e.g., PyTorch) can only support non-customizable 8-bit precision, data format, and parameter extraction. Secondly, the objective of quantization is to enable the computation with low-precision data. However, the current SoTA algorithm treats the quantized integer as an intermediate result, while the final output of the quantizer is the "discretized" floating-point values, ignoring the practical needs and adding additional workload to hardware designers for integer parameter extraction and layer fusion. Finally, the compression toolkits designed by the industry are constrained to their in-house product or a handful of algorithms. The limited degree of freedom in the current toolkit and the under-explored customization hinder the prototype ASIC or FPGA-based accelerator design. To resolve these challenges, we propose Torch2Chip, an open-sourced, fully customizable, and high-performance toolkit that supports user-designed compression followed by automatic model fusion and parameter extraction. Torch2Chip incorporates the hierarchical design workflow, and the user-customized compression algorithm will be directly packed into the deployment-ready format for prototype chip verification with either CNN or vision transformer (ViT). The code is available at https://github.com/SeoLabCornell/torch2chip.
Paper Structure (27 sections, 10 equations, 9 figures, 4 tables)

This paper contains 27 sections, 10 equations, 9 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Gap between ML Framework and Hardware Designer.
  3. Gap between SoTA Algorithm and Hardware Designer.
  4. Gap between Industry-level Toolkit and Prototyping.
  5. 1) Hierarchical customized quantization build-up.
  6. 2) Automatic fusion and parameter extraction for ultra-low precision.
  7. 3) Unsupervised foundation model training for powerful model compression and deployment.
  8. 4) User-defined output data format for fast deployment
  9. Related Work
  10. Quantization
  11. Weight Sparsification
  12. Self-supervised Pre-training
  13. Torch2Chip
  14. Hierarchical Design for Customized Quantization
  15. Fully Customizable Post-training Fusion
  16. ...and 12 more sections

Figures (9)

  • Figure 1: (a) Current algorithm-hardware development environment with disconnected workflow. (b) Proposed systematic toolkit Torch2Chip with high degree of customization versatility.
  • Figure 2: Hierarchical architecture of layer and module with the "Dual-Path"-based design.
  • Figure 3: Training and fusion workflow of Torch2Chip: (a) training mode with fully customizable quantizer and layer, (b) inference mode with integer-only computation, and (c) deploy mode with integer-only parameters stored by the vanilla layer.
  • Figure 4: Training and fusion workflow of Torch2Chip with multi-head attention under 8-bit precision: (a) training mode with fully customizable quantizer and layer, (b) inference mode with integer-only computation and Look-up-Table (LUT)-based Softmax function, and (c) deploy mode with integer-only parameters stored by the vanilla fully-connected layer.
  • Figure 5: Automated and versatile parameter extraction with various output formats for RTL verification, integer-only model file, and native Pytorch quantization layer
  • ...and 4 more figures