Minion Gated Recurrent Unit for Continual Learning

TL;DR

Minion Recurrent Unit (MiRU) addresses continual learning for recurrent models on edge devices by replacing GRU gates with scaling coefficients, drastically reducing parameters and compute while preserving accuracy. Two variants, MiRU-1 and MiRU-2, use scaling factors $\beta$ and $\lambda$ to control reset and update dynamics, enabling slower or faster adaptation as needed. Empirical results on MNIST and IMDB show MiRU is competitive with GRU in sequence classification and delivers substantial improvements in training time, inference time, and energy, while enabling stable domain-incremental learning under replay and global inhibition with permuted MNIST. The work demonstrates strong potential for practical edge deployment of continual learning RNNs in streaming data scenarios.

Abstract

The increasing demand for continual learning in sequential data processing has led to progressively complex training methodologies and larger recurrent network architectures. Consequently, this has widened the knowledge gap between continual learning with recurrent neural networks (RNNs) and their ability to operate on devices with limited memory and compute. To address this challenge, we investigate the effectiveness of simplifying RNN architectures, particularly gated recurrent unit (GRU), and its impact on both single-task and multitask sequential learning. We propose a new variant of GRU, namely the minion recurrent unit (MiRU). MiRU replaces conventional gating mechanisms with scaling coefficients to regulate dynamic updates of hidden states and historical context, reducing computational costs and memory requirements. Despite its simplified architecture, MiRU maintains performance comparable to the standard GRU while achieving 2.90x faster training and reducing parameter usage by 2.88x, as demonstrated through evaluations on sequential image classification and natural language processing benchmarks. The impact of model simplification on its learning capacity is also investigated by performing continual learning tasks with a rehearsal-based strategy and global inhibition. We find that MiRU demonstrates stable performance in multitask learning even when using only rehearsal, unlike the standard GRU and its variants. These features position MiRU as a promising candidate for edge-device applications.

Figures (6)

• Figure 1: The data flow and operations of the standard GRU (Left) which comprises of a reset gate, an update gate, and an additional unit to compute the candidate hidden state. The proposed MiRU unit (right) which uses reset ($\beta$) and update ($\lambda$) coefficients rather than gates to control network dynamics and information updates, respectively.
• Figure 2: The test accuracy of classifying sequences generated from the IMDB dataset (top row) and MNIST dataset (bottom row) when using the LSTM, GRU, GRU variants (MGU, VGRU, and SGRU), and the proposed MiRU-1 and MiRU-2. The accuracy is averaged across 5 runs, with the standard deviation represented by shaded light blue and red regions.).
• Figure 3: The impact of tuning the reset and update coefficients on test accuracy when classifying sequences generated from (a) IMDB and (b) MNIST datasets. (c) The impact of the sparsity level of the hidden layer neuronal activation on the mean accuracy when learning multiple tasks constructed from permuted MNIST. Best viewed in color.
• Figure 4: The test accuracy (worst-case) and the mean (dotted line) across 5 tasks generated from permuted-MNIST for the MiRU-2, GRU, MGU, VGRU, and SGRU models under the following scenarios: (first-row) replay buffer of size 1875 per task, (second row) replay buffer of size 3750 per task, and (third-row) replay buffer of size 1875 per task with global inhibition. Best viewed in color.
• Figure 5: The distribution of the GRU forget gate activation: (a) at initialization, (b) after learning the first task without global inhibition, and (c) after learning the first task with global inhibition.