A Tensor Decomposition Perspective on Second-order RNNs
Maude Lizaire, Michael Rizvi-Martel, Marawan Gamal Abdel Hameed, Guillaume Rabusseau
TL;DR
This work analyzes Second-order RNNs through CP tensor decomposition, introducing CPRNNs as a rank-controlled, parameter-efficient alternative to 2RNNs. It formalizes how CPRNN expressivity depends on CP rank $R$ and hidden size $n$, proving inclusions, saturation at $R_{\max}$, and a typical-rank threshold $R_{\mathrm{typ-max}}$, while showing CPRNNs interpolate between RNNs and 2RNNs and include MIRNN as a special case. Theoretical results are complemented by Penn Treebank experiments demonstrating that, for a fixed parameter budget, CPRNNs can outperform RNNs, 2RNNs, and MIRNNs by choosing appropriate rank and hidden size. The findings illuminate how tensor decompositions control bias-variance tradeoffs in higher-order RNNs and motivate exploring other decompositions and extensions to state-space models. Overall, the rank-driven CPRNN framework provides a principled path to scalable, expressive sequence models that balance capacity and efficiency.
Abstract
Second-order Recurrent Neural Networks (2RNNs) extend RNNs by leveraging second-order interactions for sequence modelling. These models are provably more expressive than their first-order counterparts and have connections to well-studied models from formal language theory. However, their large parameter tensor makes computations intractable. To circumvent this issue, one approach known as MIRNN consists in limiting the type of interactions used by the model. Another is to leverage tensor decomposition to diminish the parameter count. In this work, we study the model resulting from parameterizing 2RNNs using the CP decomposition, which we call CPRNN. Intuitively, the rank of the decomposition should reduce expressivity. We analyze how rank and hidden size affect model capacity and show the relationships between RNNs, 2RNNs, MIRNNs, and CPRNNs based on these parameters. We support these results empirically with experiments on the Penn Treebank dataset which demonstrate that, with a fixed parameter budget, CPRNNs outperforms RNNs, 2RNNs, and MIRNNs with the right choice of rank and hidden size.