GhostRNN: Reducing State Redundancy in RNN with Cheap Operations

TL;DR

An efficient RNN architecture, GhostRNN, which reduces hidden state redundancy with cheap operations, and observes that partial dimensions of hidden states are similar to the others in trained RNN models, suggesting that redundancy exists in specific RNNs.

Abstract

Recurrent neural network (RNNs) that are capable of modeling long-distance dependencies are widely used in various speech tasks, eg., keyword spotting (KWS) and speech enhancement (SE). Due to the limitation of power and memory in low-resource devices, efficient RNN models are urgently required for real-world applications. In this paper, we propose an efficient RNN architecture, GhostRNN, which reduces hidden state redundancy with cheap operations. In particular, we observe that partial dimensions of hidden states are similar to the others in trained RNN models, suggesting that redundancy exists in specific RNNs. To reduce the redundancy and hence computational cost, we propose to first generate a few intrinsic states, and then apply cheap operations to produce ghost states based on the intrinsic states. Experiments on KWS and SE tasks demonstrate that the proposed GhostRNN significantly reduces the memory usage (~40%) and computation cost while keeping performance similar.

Figures (3)

• Figure 1: The cosine similarity matrix of RNN hidden states (first 12 dimensions) in a well-trained DCCRN_GRU with 128 hidden units.
• Figure 2: The value of hidden states at different indexes.
• Figure 3: Schematic diagram of the RNN layer and proposed Ghost RNN.