Accuracy, Memory Efficiency and Generalization: A Comparative Study on Liquid Neural Networks and Recurrent Neural Networks

TL;DR

The paper investigates how Liquid Neural Networks (LNNs), with their continuous-time dynamics, compare to traditional RNNs (and LSTMs/GRUs) across accuracy, memory efficiency, and generalization. It surveys architectural foundations, including LTC, CfC, NCP, Liquid-S4, and LRC, and synthesizes findings from benchmarks and a case study spanning trajectory prediction, synthetic time series, and ICU patient data. Key contributions include demonstrating competitive accuracy for LNNs on dynamic tasks, substantial memory- and energy-efficiency advantages (especially on neuromorphic hardware), and improved robustness and OOD generalization attributed to continuous-time adaptivity. The work highlights practical trade-offs, such as solver-based LTC's higher compute time versus CfC's speed, and argues for solver-free, hardware-conscious designs and hybrid architectures to scale LNNs to complex, real-world problems. Overall, the review positions LNNs as a promising, potentially paradigm-shifting alternative to discrete-time RNNs for continuous, dynamic environments.

Abstract

This review aims to conduct a comparative analysis of liquid neural networks (LNNs) and traditional recurrent neural networks (RNNs) and their variants, such as long short-term memory networks (LSTMs) and gated recurrent units (GRUs). The core dimensions of the analysis include model accuracy, memory efficiency, and generalization ability. By systematically reviewing existing research, this paper explores the basic principles, mathematical models, key characteristics, and inherent challenges of these neural network architectures in processing sequential data. Research findings reveal that LNN, as an emerging, biologically inspired, continuous-time dynamic neural network, demonstrates significant potential in handling noisy, non-stationary data, and achieving out-of-distribution (OOD) generalization. Additionally, some LNN variants outperform traditional RNN in terms of parameter efficiency and computational speed. However, RNN remains a cornerstone in sequence modeling due to its mature ecosystem and successful applications across various tasks. This review identifies the commonalities and differences between LNNs and RNNs, summarizes their respective shortcomings and challenges, and points out valuable directions for future research, particularly emphasizing the importance of improving the scalability of LNNs to promote their application in broader and more complex scenarios.

Figures (12)

• Figure 1: Recurrent Neural Network (RNN) unfolding representation illustrating the temporal expansion of recurrent connections. The left side shows the compact recurrent structure with feedback connections, while the right side demonstrates the unfolded network across multiple time steps. Each time step receives input $x_t$, updates hidden state $h_t$, and produces output $o_t$, with weight matrices (W, U, V) shared across all time steps.
• Figure 2: Conceptual architecture of a Liquid Neural Network (LNN). Input layer (left):$m$ input streams $I_1,\ldots,I_m$ provide event-like signals over time, forming the input vector $x(t)$. Liquid layer (middle): a recurrent, heterogeneous reservoir with continuous-time state $h(t)$; its dynamics follow an ODE $\dot h(t)=f\!(h(t),x(t);\theta)$, and the arrows depict recurrent couplings among units. Readout layer (right):$n$ readout units $R_1,\ldots,R_n$ compute task-specific outputs from the liquid state, typically $r(t)=W h(t)+b$; dashed lines indicate dense projections from the liquid layer to each readout. The far-right sketch illustrates example output trajectories/decisions produced by the readouts.
• Figure 3: The number of the parameters of the Long Short-Term Memory (LSTM) and Gated Recurrent Unit (GRU) models as a function of the number of neurons and the order of the dynamics determined by $n_A = n_P$.
• Figure 4: The number of parameters of the Long Short-Term Memory (LSTM) and Gated Recurrent Unit (GRU) models as a function of the number of neurons and the order of the dynamics determined by $n_B$; $n_A=0$.
• Figure 5: Comparison of (a) average loss, (b) training time per round, and (c) training memory usage between Liquid Time-Constant (LTC) (Liquid Neural Network (LNN)) and Long Short-Term Memory (LSTM) (Recurrent Neural Network (RNN)) in the Walker2d trajectory prediction task.