Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Efficient Neural Compression with Inference-time Decoding

C. Metz, O. Bichler, A. Dupret

TL;DR

The paper addresses edge deployment by reducing the memory footprint of neural networks through a mix of mixed-precision quantization and entropy coding. It introduces zero-point quantization and an inference-time ANS-based decoder to push beyond the 1-bit frontier while keeping accuracy loss under 1% on ImageNet. The authors propose an entropy-driven training objective that learns per-parameter precision and maps to an odd number of quantization bins, achieving sub-binary compression with minimal performance impact. Empirical results on ResNet-18/50 show substantial storage reductions and practical, low-latency decoding, suggesting strong potential for on-device deployment.

Abstract

This paper explores the combination of neural network quantization and entropy coding for memory footprint minimization. Edge deployment of quantized models is hampered by the harsh Pareto frontier of the accuracy-to-bitwidth tradeoff, causing dramatic accuracy loss below a certain bitwidth. This accuracy loss can be alleviated thanks to mixed precision quantization, allowing for more flexible bitwidth allocation. However, standard mixed precision benefits remain limited due to the 1-bit frontier, that forces each parameter to be encoded on at least 1 bit of data. This paper introduces an approach that combines mixed precision, zero-point quantization and entropy coding to push the compression boundary of Resnets beyond the 1-bit frontier with an accuracy drop below 1% on the ImageNet benchmark. From an implementation standpoint, a compact decoder architecture features reduced latency, thus allowing for inference-compatible decoding.

Efficient Neural Compression with Inference-time Decoding

TL;DR

The paper addresses edge deployment by reducing the memory footprint of neural networks through a mix of mixed-precision quantization and entropy coding. It introduces zero-point quantization and an inference-time ANS-based decoder to push beyond the 1-bit frontier while keeping accuracy loss under 1% on ImageNet. The authors propose an entropy-driven training objective that learns per-parameter precision and maps to an odd number of quantization bins, achieving sub-binary compression with minimal performance impact. Empirical results on ResNet-18/50 show substantial storage reductions and practical, low-latency decoding, suggesting strong potential for on-device deployment.

Abstract

This paper explores the combination of neural network quantization and entropy coding for memory footprint minimization. Edge deployment of quantized models is hampered by the harsh Pareto frontier of the accuracy-to-bitwidth tradeoff, causing dramatic accuracy loss below a certain bitwidth. This accuracy loss can be alleviated thanks to mixed precision quantization, allowing for more flexible bitwidth allocation. However, standard mixed precision benefits remain limited due to the 1-bit frontier, that forces each parameter to be encoded on at least 1 bit of data. This paper introduces an approach that combines mixed precision, zero-point quantization and entropy coding to push the compression boundary of Resnets beyond the 1-bit frontier with an accuracy drop below 1% on the ImageNet benchmark. From an implementation standpoint, a compact decoder architecture features reduced latency, thus allowing for inference-compatible decoding.
Paper Structure (10 sections, 4 equations, 5 figures, 3 tables)

This paper contains 10 sections, 4 equations, 5 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Methods
  3. Entropy bound and ANS decoder
  4. Zero-point quantization
  5. Training
  6. Experiments
  7. ImageNet results
  8. Table size and decoding parallelization
  9. Analysis
  10. Conclusion and future works

Figures (5)

  • Figure 1: State machine for encoding and decoding of ANS duda2014asymmetric ($x$: state number)
  • Figure 2: ANS decoder diagram.
  • Figure 3: Decoder parallelization scheme.
  • Figure 4: Distribution of quantization precision among layers, Resnet-50, number of bins.
  • Figure 5: Memory cost distribution (MB, vertical axis) among Resnet-50 layers. Quantized uncompressed network (up), and quantized ANS compressed (down). Axes scales are different.