Beyond Dense States: Elevating Sparse Transcoders to Active Operators for Latent Reasoning

TL;DR

This work tackles the trade-off between inference efficiency and interpretability in latent reasoning by introducing LSTR, a sparse latent reasoning framework. Central to LSTR is the Latent Transition Transcoder (LTT), which orchestrates reasoning through a linear transport path plus a sparse semantic path under an explicit sparsity budget $k$, enabling semantic resolution control via $k$ and compressed latent trajectories via a ratio $r$. Through supervised trajectory imitation and carefully designed losses (including Fraction of Variance Unexplained and ghost-gradient updates), LSTR achieves competitive accuracy with significantly shorter latent reasoning chains and demonstrates causal, interpretable contributions of sparse features to the reasoning process; it also scales to larger models with stable efficiency gains. The results show that sparse latent reasoning can match or surpass dense latent baselines under compression while enhancing interpretability and controllability, suggesting practical benefits for efficient, transparent reasoning in large language models.

Abstract

Latent reasoning compresses the chain-of-thought (CoT) into continuous hidden states, yet existing methods rely on dense latent transitions that remain difficult to interpret and control. Meanwhile, sparse representation models uncover human-interpretable semantic features but remain largely confined to post-hoc analysis. We reconcile this tension by proposing LSTR (Latent Sparse Transcoder Reasoning), a latent reasoning framework that elevates functional sparse transcoders into active reasoning operators to perform multi-step computation through sparse semantic transitions. At its core, LSTR employs a Latent Transition Transcoder (LTT) with a residual skip architecture that decouples linear manifold transport from sparse semantic updates, enabling controllable semantic resolution via explicit sparsity constraints. Extensive experiments show that LSTR preserves reasoning accuracy and compression efficiency while substantially improving interpretability over dense latent baselines. Causal interventions and trajectory analyses further demonstrate that these sparse features act as both interpretable and causally effective operators in the reasoning process.

Figures (9)

• Figure 1: Comparison of reasoning paradigms. CoT relies on long explicit token chains, incurring high latency. Dense latent reasoning compresses computation but lacks interpretability due to fully active latent dimensions. In contrast, LSTR performs reasoning through sparse, interpretable latent features, achieving both efficiency and semantic control.
• Figure 2: Overview of LSTR. A contiguous sequence of explicit reasoning tokens is temporally aggregated into a compressed target embedding. At each latent step, the Latent Transition Transcoder (LTT) performs a bilateral decomposition of the transition dynamics: a linear Transport Path maintains manifold continuity, while a Sparse Innovation Path (via Top-$k$ selection) injects interpretable semantic updates. Under a fixed $(r, k)$ budget, LSTR enables Semantic Resolution Control, where different sparse activation patterns correspond to distinct reasoning primitives such as arithmetic operations or logic gates.
• Figure 3: Mechanistic case study of sparse latent reasoning in LSTR. Left: Feature activations along a correct trajectory. Right: Perturbing a single feature causally diverts the reasoning path and answer correctness.
• Figure 4: Causal Necessity Distribution ($r=2$). KDE-estimated probability density of essential logical steps across heterogeneous lengths ($L \in [4, 35]$). All balanced cohorts exhibit a consistent front-loaded profile.
• Figure 5: Accuracy and average latent reasoning length under varying inference-time sparsity budgets $k$. The model is trained with $r=2$ and $k=128$, and evaluated by adjusting $k$ at inference time, effectively controlling the capacity of sparse latent transitions.