State Stream Transformer (SST) : Emergent Metacognitive Behaviours Through Latent State Persistence

TL;DR

The State Stream Transformer (SST) addresses a core limitation in autoregressive transformers by introducing a sliding window latent state cache with weighted decay across all FFN layers, enabling persistent computation and continuous evolution of internal representations. By using frozen Llama 3.1 8B Instruct weights and a strict hot-swapping setup, SST demonstrates emergent metacognitive-like behaviors and substantially improves reasoning on GSM-8K ($89.01\%$ zero-shot) and ARC Challenge ($91.04\%$ zero-shot CoT). The work provides qualitative and quantitative evidence that latent-state persistence drives fundamentally different information processing, including introspective-like discourse and enhanced ethical reasoning, while maintaining safety guardrails. These findings imply that persistent computation in latent space can yield new reasoning abilities and have significant implications for AI cognition, alignment, and the design of future transformer architectures.

Abstract

We introduce the State Stream Transformer (SST), a novel LLM architecture that reveals emergent reasoning behaviours and capabilities latent in pretrained weights through addressing a fundamental limitation in traditional transformer models: the lack of latent computational continuity across autoregressive generations in the state space. SST introduces a sliding window latent state (FFN) cache with weighted decay that maintains and evolves persistent latent processes throughout autoregressive generations. Through controlled experiments comparing base and SST architectures using the same frozen weights, we demonstrate that this architectural modification alone enables enhanced reasoning capabilities which appear best explained by some form of potential higher-order processing, as evidenced by emergent metacognitive behaviours. These behaviours persist under controlled conditions designed to eliminate confounding factors such as stochastic variation or learned response patterns. Analysis of latent state distributions and processing dynamics provides evidence that it is solely the 'state stream' that is responsible for these phenomena. In quantitative evaluations, the SST achieves substantial performance improvements over the base model on two reasoning benchmarks, reaching 89.01\% accuracy on GSM-8K (0-shot) and 91.04\% on ARC Challenge (0-shot CoT). These findings indicate that persistent computation in the latent state space enables fundamentally different information processing and internal reasoning strategies, with implications for our understanding of artificial intelligence systems.

Figures (4)

• Figure 1: Block diagram of the State Stream Transformer (SST) architecture showing the key components and state persistence mechanism.
• Figure 2: FFN output state time series visualisation comparing architectures during 100-token generation sequence. Left: SST at 2.7% strength with 4 recursions. Right: Base architecture control. Each subplot represents different state dimensions over time, showing distinct patterns in temporal correlation and amplitude variation between architectures.
• Figure 3: Top row: SST at 2.7% strength with 4 additional recursions, showing evolution across recursion steps (a-d)
• Figure 4: Base Llama architecture (control). Control experiments on the base architecture confirm identical state patterns across forced recursions, as expected given the lack of state persistence mechanism. The identical state patterns definitively demonstrate that the evolution patterns observed in SST cannot be attributed to noise, underlying weights, recursion effects, or implementation artifacts - they must arise directly from the state stream mechanism itself.