Gradual Forgetting: Logarithmic Compression for Extending Transformer Context Windows

TL;DR

This work tackles extending transformer context without architectural changes by introducing a scale-invariant input preprocessing step that compresses the distant past using a log-spaced memory bank (Scale-Invariant Temporal History, SITH). The compressed history, computed as $\tilde{f}(t,\overset{*}{\tau})=\sum_{t'=1}^{M}\Phi(t',\overset{*}{\tau}) f(t-t')$ with $\Phi(t,\overset{*}{\tau})=\frac{k^{k+1}}{k!}\left(\frac{t}{\overset{*}{\tau}}\right)^k e^{-k t/\overset{*}{\tau}}$, is concatenated with the most recent uncompressed tokens and processed by an unmodified transformer. Experiments on WikiText-103 and PG-19 show perplexity improvements that grow with the memory size, achieving competitive results for models of similar parameter counts and demonstrating the practical viability of cognitive-inspired short-term to long-range memory mappings for language modeling.

Abstract

Most approaches to long-context processing increase the complexity of the transformer's internal architecture by integrating mechanisms such as recurrence or auxiliary memory modules. In this work, we introduce an alternative approach that modifies the input representation itself, rather than the transformer architecture. Inspired by cognitive models of human memory, our method applies a scale-invariant logarithmic compression to the input tokens. The resulting compressed representation is processed by a standard, unmodified transformer, preserving architectural simplicity. We evaluate this approach on the WikiText-103 and PG-19 language modeling benchmarks, showing a reduction in perplexity compared to uncompressed baselines. Moreover, performance improves consistently with longer compressed temporal contexts, showing that input-level logarithmic compression is a simple and effective way to extend a transformer's long-range memory.

Figures (6)

• Figure 1: Log-compressed impulse responses from ten $\tilde{f}$ neurons with $k=50$ show log-spaced peak times and a constant coefficient of variation (broader responses at later peaks). B. The same responses plotted on a log-time axis are uniformly spaced with equal widths. C. At time 30, $\tilde{f}$ neuron activations form a log-compressed memory of the input, approximating past values with finer resolution for more recent events. Grayscale dots correspond to impulse responses in panels A–B. Additional visualizations for $k=10$ and $k=100$ are shown in Fig. \ref{['fig:imp_res_appendix']}.
• Figure 2: A. Illustration of compressed memory applied to the input sequence. A subset of the last $m$ tokens, with $m=4$ in this example, is used to compute four keys, queries, and values. The rest of the input sequence is used as an input to the compressed memory composed of $L$ filters ($L=4$ in this example), producing an additional four keys, queries, and values. B. Illustration of a standard transformer where each token in the sequence is used to generate keys, queries, and values.
• Figure 3: Architecture of the transformer based on scale-invariant compression. Compressed represents the portion of the input that is compressed with the scale-invariant filters, UC represents the uncompressed portion of the input. Norm depicts Layer Normalization, Add depicts residual connections, and Feed-Forward depicts a multi-layer perceptron at the end of each layer. At the Multi-Head Attention block in the first layer $N_0$, the output of the scale-invariant compression is concatenated with the uncompressed portion and projected to form queries, keys, and values. In subsequent layers $N_X$, where $X$ is the layer number, the input is passed through without additional compression.
• Figure 4: Comparison of long-context approaches. Transformer-XL, Compressive Transformer, and RMT handle long range by carrying state across segments (caching, compression, or recurrence). Our approach performs scale-invariant compression of the distant past to yield a fixed-size memory concatenated with recent tokens.
• Figure 5: Test set perplexity decreases as the number of filters $L$ increases. WikiText-103 (A) shows an approximately linear decline; PG-19 (B) shows a downward, step-like trend with a knee around mid-range $L$. The top axis indicates the corresponding filter window $\tau_{\max}$ (geometric spacing). The results are consistent for different values of $k$.