Memory-based Language Models: An Efficient, Explainable, and Eco-friendly Approach to Large Language Modeling

TL;DR

This work investigates memory-based language modeling as an efficient, explainable, and eco-friendly alternative to neural language models. It introduces OLIFANT, a CPU-based framework that uses a prefix trie to implement three $k$-NN–style variants (IB1-IG, TRIBL2, IGTree) for next-token prediction, enabling incremental learning and scalable data-driven performance. The study provides learning curves, latency and emission analyses, memorization and explainability assessments, and a distributional perplexity analysis, showing log-linear gains with more data and substantially lower CO$_2$ emissions than large Transformer training, though with limited context width and sparser output distributions. Although current performance lags the largest GPT-style models on very large data, OLIFANT offers a transparent, composable, and energy-efficient alternative for scenarios demanding CPU-based, interpretable, and scalable next-token prediction.

Abstract

We present memory-based language modeling as an efficient, eco-friendly alternative to deep neural network-based language modeling. It offers log-linearly scalable next-token prediction performance and strong memorization capabilities. Implementing fast approximations of k-nearest neighbor classification, memory-based language modeling leaves a relatively small ecological footprint both in training and in inference mode, as it relies fully on CPUs and attains low token latencies. Its internal workings are simple and fully transparent. We compare our implementation of memory-based language modeling, OLIFANT, with GPT-2 and GPT-Neo on next-token prediction accuracy, estimated emissions and speeds, and offer some deeper analyses of the model.

Figures (11)

• Figure 1: Schematic conversion of an instance base with four examples of two context words predicting a third word, into a prefix trie with next-token distribution information stored at all nodes. Grey nodes, representing subsets of the instance base of which the majority class does not conflict with the parent node, are not stored when the prefix trie is used for decision-tree classification (the IGTree mode of olifant).
• Figure 2: Schematic visualization of classification in the three memory-based language modeling variants. The larger triangle represents the entire prefix trie; grayed zones represent $k$-NN classification; nodes and edges represent downward-pass decision-tree traversal of the prefix trie.
• Figure 3: Schematic visualization of the three uncoupled scaling dimensions in training Transformer-based LMs, versus the single coupled scaling dimension involved when training memory-based LMs, where the model equals the data.
• Figure 4: Learning curves in terms of correctly predicted tokens of TRIBL2 with increasing context widths.
• Figure 5: Learning curves on next-token prediction accuracy of the three olifant variants and the two GPT systems (with fixed training data sets but increasing model sizes). Dashed and dotted lines are regression functions fitted to all TRIBL2 measurements (dotted) and measurements from 1 billion training tokens (dashed).