Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Quantized Approximately Orthogonal Recurrent Neural Networks

Armand Foucault, Franck Mamalet, François Malgouyres

TL;DR

The paper tackles the challenge of quantizing Orthogonal Recurrent Neural Networks (ORNNs) to enable edge deployment while preserving long-term dependencies. It introduces Quantized Approximately Orthogonal RNNs (QORNNs) and develops two Quantization-Aware Training approaches, STE-projUNN and STE-Björck, along with post-training activation quantization, achieving competitive results with as few as 4 bits for weights on benchmarks like copy-task and pMNIST. The key theoretical contribution includes bounds on the approximate orthogonality of quantized weights, and the empirical results demonstrate that a fully quantized recurrence can solve very long sequences (e.g., $T_0=1000$) with small model footprints, outperforming many prior quantized RNNs on long-range tasks. These findings advance practical, energy-efficient RNNs for edge computing and lay groundwork for quantizing other long-range models such as SSSMs.

Abstract

In recent years, Orthogonal Recurrent Neural Networks (ORNNs) have gained popularity due to their ability to manage tasks involving long-term dependencies, such as the copy-task, and their linear complexity. However, existing ORNNs utilize full precision weights and activations, which prevents their deployment on compact devices.In this paper, we explore the quantization of the weight matrices in ORNNs, leading to Quantized approximately Orthogonal RNNs (QORNNs). The construction of such networks remained an open problem, acknowledged for its inherent instability. We propose and investigate two strategies to learn QORNN by combining quantization-aware training (QAT) and orthogonal projections. We also study post-training quantization of the activations for pure integer computation of the recurrent loop. The most efficient models achieve results similar to state-of-the-art full-precision ORNN, LSTM and FastRNN on a variety of standard benchmarks, even with 4-bits quantization.

Quantized Approximately Orthogonal Recurrent Neural Networks

TL;DR

The paper tackles the challenge of quantizing Orthogonal Recurrent Neural Networks (ORNNs) to enable edge deployment while preserving long-term dependencies. It introduces Quantized Approximately Orthogonal RNNs (QORNNs) and develops two Quantization-Aware Training approaches, STE-projUNN and STE-Björck, along with post-training activation quantization, achieving competitive results with as few as 4 bits for weights on benchmarks like copy-task and pMNIST. The key theoretical contribution includes bounds on the approximate orthogonality of quantized weights, and the empirical results demonstrate that a fully quantized recurrence can solve very long sequences (e.g., ) with small model footprints, outperforming many prior quantized RNNs on long-range tasks. These findings advance practical, energy-efficient RNNs for edge computing and lay groundwork for quantizing other long-range models such as SSSMs.

Abstract

In recent years, Orthogonal Recurrent Neural Networks (ORNNs) have gained popularity due to their ability to manage tasks involving long-term dependencies, such as the copy-task, and their linear complexity. However, existing ORNNs utilize full precision weights and activations, which prevents their deployment on compact devices.In this paper, we explore the quantization of the weight matrices in ORNNs, leading to Quantized approximately Orthogonal RNNs (QORNNs). The construction of such networks remained an open problem, acknowledged for its inherent instability. We propose and investigate two strategies to learn QORNN by combining quantization-aware training (QAT) and orthogonal projections. We also study post-training quantization of the activations for pure integer computation of the recurrent loop. The most efficient models achieve results similar to state-of-the-art full-precision ORNN, LSTM and FastRNN on a variety of standard benchmarks, even with 4-bits quantization.
Paper Structure (60 sections, 30 equations, 3 figures, 14 tables)

This paper contains 60 sections, 30 equations, 3 figures, 14 tables.

Table of Contents

  1. Introduction
  2. Motivation:
  3. Contribution:
  4. Organization of the paper:
  5. Related works
  6. On quantized RNNs:
  7. Conclusion on RNNs:
  8. On quantized Transformers:
  9. Preliminaries and notations
  10. Vanilla RNNs
  11. Orthogonality
  12. Quantization
  13. QORNN are hard to train
  14. The instability problem:
  15. Evaluating the approximate orthogonality of $q_k(W)$
  16. ...and 45 more sections

Figures (3)

  • Figure 1: Denote by $q_k$ the quantizer with bitwidth $k$ as defined in \ref{['quant-sec']}, $\sigma_{min}(q_k(W))$ and $\sigma_{max}(q_k(W))$ the smallest and largest singular values of the matrix $q_k(W)$ respectively for $W \in \mathbb R^{200 \times 200}$ a uniformly sampled orthogonal matrix. (Left) $\frac{\|W^T - (q_k(W))^T\|_F}{\|W^T\|_F}$ for various $k$ and powers $T$. (Right) Boxplots for $1000$ random orthogonal matrices $W$ of the ratio $\sigma_{min}(q_k(W)) / \sigma_{max}(q_k(W))$ for various $k$.
  • Figure 2: Evolution of Test Loss During Training for the copy-task with $n_h=256$. (Left) STE-projUNN ; (Right) STE-Bjorck.
  • Figure 3: Evolution of the accuracy on the training set during training for pMNIST with $n_h=170$. (Left) STE-projUNN ; (Right) STE-Bjorck.