A Survey of Recursive and Recurrent Neural Networks

TL;DR

This survey categorizes RecursiveNNs and RecurrentNNs into General, Structured, and Other families, detailing architectures from basic RecursiveNNs and LSTMs to Grid LSTMs, Graph RNNs, Lattice RNNs, Tree LSTMs, and memory-augmented variants. It emphasizes how these models address complex sequence, spatial, and temporal problems across vision, language, and multimodal domains, while detailing training challenges such as gradient vanishing/explosion and proposing numerous architectural innovations (gating, depth, attention, diffusion on graphs, and lattice reasoning). Key contributions include a comprehensive taxonomy, principle and structure descriptions, training objectives, and future directions for scaling RNNs, improving generalization, and expanding application domains. The findings underscore that while RNN variants mitigate long-range dependency issues and enable rich hierarchical and multimodal modeling, they still face efficiency, parameterization, and evaluation challenges that motivate ongoing research. Overall, the paper positions RNNs as a diverse and rapidly evolving toolkit for sequential and structured data analysis with broad impact across AI applications.

Abstract

In this paper, the branches of recursive and recurrent neural networks are classified in detail according to the network structure, training objective function and learning algorithm implementation. They are roughly divided into three categories: The first category is General Recursive and Recurrent Neural Networks, including Basic Recursive and Recurrent Neural Networks, Long Short Term Memory Recursive and Recurrent Neural Networks, Convolutional Recursive and Recurrent Neural Networks, Differential Recursive and Recurrent Neural Networks, One-Layer Recursive and Recurrent Neural Networks, High-Order Recursive and Recurrent Neural Networks, Highway Networks, Multidimensional Recursive and Recurrent Neural Networks, Bidirectional Recursive and Recurrent Neural Networks; the second category is Structured Recursive and Recurrent Neural Networks, including Grid Recursive and Recurrent Neural Networks, Graph Recursive and Recurrent Neural Networks, Temporal Recursive and Recurrent Neural Networks, Lattice Recursive and Recurrent Neural Networks, Hierarchical Recursive and Recurrent Neural Networks, Tree Recursive and Recurrent Neural Networks; the third category is Other Recursive and Recurrent Neural Networks, including Array Long Short Term Memory, Nested and Stacked Recursive and Recurrent Neural Networks, Memory Recursive and Recurrent Neural Networks. Various networks cross each other and even rely on each other to form a complex network of relationships. In the context of the development and convergence of various networks, many complex sequence, speech and image problems are solved. After a detailed description of the principle and structure of the above model and model deformation, the research progress and application of each model are described, and finally the recursive and recurrent neural network models are prospected and summarized.

Figures (48)

• Figure 1: RNNs structure.
• Figure 2: RNNs classification.
• Figure 3: Recursive neural network with bottom-up propagation.
• Figure 4: Unfold along time.
• Figure 5: RecursiveNNs.