Reservoir Computing inspired Matrix Multiplication-free Language Model

TL;DR

The paper tackles the prohibitive computational costs of large language models by fusing reservoir computing with a matrix multiplication-free architecture. It introduces RC MatMul-free LM and GRC MatMul-free LM, where MLGRU is replaced by RC-based variants, weights are fixed or shared across layers, and kernel-level fusion reduces memory accesses. On SlimPajama and standard benchmarks, the approach yields up to $0.19$ parameter reduction and training/inference time reductions of $0.099$ and $0.08$ respectively while maintaining comparable accuracy. The work demonstrates a practical path toward scalable, energy-efficient LLM deployment and motivates hardware-aware RC integrations for future LLM design.

Abstract

Large language models (LLMs) have achieved state-of-the-art performance in natural language processing; however, their high computational cost remains a major bottleneck. In this study, we target computational efficiency by focusing on a matrix multiplication free language model (MatMul-free LM) and further reducing the training cost through an architecture inspired by reservoir computing. Specifically, we partially fix and share the weights of selected layers in the MatMul-free LM and insert reservoir layers to obtain rich dynamic representations without additional training overhead. Additionally, several operations are combined to reduce memory accesses. Experimental results show that the proposed architecture reduces the number of parameters by up to 19%, training time by 9.9%, and inference time by 8.0%, while maintaining comparable performance to the baseline model.

Figures (10)

• Figure 1: Left : architecture of general LLM right : architecture of transformer-based LLM
• Figure 2: MatMul-free LM
• Figure 3: Repeating module of MLGRU
• Figure 4: GLU
• Figure 5: Repeating module of ternary LI ESN