Reservoir Computing as a Language Model

TL;DR

This work addresses the high energy and latency costs of large transformer language models by evaluating reservoir computing (RC) as an energy-efficient alternative for language modeling. It compares traditional RC, attention-enhanced RC (AERC), and transformer baselines on a character-level Shakespeare corpus with matched parameter budgets, and introduces Layered LAERC to extend RC to token-level LMs. The study shows transformers achieve the best prediction quality, while RC variants offer substantially lower training and inference costs; LAERC demonstrates scalable performance with hardware-friendly dynamics and shows power-law scaling in minimum loss with model size. The findings provide practical guidelines for balancing resource constraints with performance and highlight reservoir-based approaches as viable options for edge/embedded NLP with potential photonic or analog substrates.

Abstract

Large Language Models (LLM) have dominated the science and media landscape duo to their impressive performance on processing large chunks of data and produce human-like levels of text. Nevertheless, their huge energy demand and slow processing are still a bottleneck to further increasing quality while also making the models accessible to everyone. To solve this bottleneck, we will investigate how reservoir computing performs on natural text processing, which could enable fast and energy efficient hardware implementations. Studies investigating the use of reservoir computing as a language model remain sparse. In this paper, we compare three distinct approaches for character-level language modeling, two different \emph{reservoir computing} approaches, where only an output layer is trainable, and the well-known \emph{transformer}-based architectures, which fully learn an attention-based sequence representation. We explore the performance, computational cost and prediction accuracy for both paradigms by equally varying the number of trainable parameters for all models. Using a consistent pipeline for all three approaches, we demonstrate that transformers excel in prediction quality, whereas reservoir computers remain highly efficient reducing the training and inference speed. Furthermore, we investigate two types of reservoir computing: a \emph{traditional reservoir} with a static linear readout, and an \emph{attention-enhanced reservoir} that dynamically adapts its output weights via an attention mechanism. Our findings underline how these paradigms scale and offer guidelines to balance resource constraints with performance.

Figures (7)

• Figure 1: Diagram illustrating all three ML agent models with vector embedding, processing (Reservoir, AERC, Transformer), output layer, and predicted probability vector for the next letter. The vector embedding is trained for the transformer case, while being randomly initialized for the reservoirs. The predicted probabilities (e.g., $p_A$ for letter a) are used to compute the cross entropy loss $H(y,\hat{y})=-\sum_i y_i\log\hat{y}_i$, which drives the error feedback.
• Figure 2: Training and testing loss over accumulated number of shared epochs for the Transformer, Reservoir, and Attention-Enhanced Reservoir Computer for the 5 different complexity models of Table \ref{['tab:parameter_counts']}. The thin light-grey dashed vertical lines indicate a shard change, while the black dashed lines show one complete epoch of all shards. Every shard was run for 5 shard epochs until the next one was processed.
• Figure 3: 7-gram and 8-gram overlap for the Transformer, Reservoir, and Attention-Enhanced Reservoir Computer over the number of trainable parameters.
• Figure 4: Training (solid) and inference (dashed) time for a longer text generation over the number of trainable parameters for the Transformer, Reservoir, and Attention-Enhanced Reservoir Computer. The numbers above the line are from a fit applied via $y=\alpha\log_{10}(x)$, where $x$ is the number of trainable parameters, $y$ is the training time and $\alpha$ the slope.
• Figure 5: Overview of the layered attention-enhanced reservoir computing (LAERC) architecture: (a) the full language model with $L$ stacked reservoir blocks and (b) the internal structure of a single block combining a fixed reservoir, gating, and ReZero feed-forward refinement. The dashed lines indicate residual connections, enabling the LAERC to backpropagate past the fixed unknown reservoir.