Dynamic Large Concept Models: Latent Reasoning in an Adaptive Semantic Space

TL;DR

This work introduces Dynamic Large Concept Models (DLCM), a hierarchical framework that rethinks where computation occurs in autoregressive language models by learning variable-length semantic concepts and performing most reasoning in a compressed concept space. It presents a compression-aware scaling law that disentangles token-level capacity, concept-level reasoning, and compression ratio, enabling principled compute allocation under fixed FLOPs, and a decoupled μP scheme for stable training across heterogeneous widths. Empirically, DLCM reallocates about one-third of inference compute to a higher-capacity reasoning backbone at $R=4$, achieving an average gain of $+2.69\%$ across 12 zero-shot benchmarks, with the most pronounced improvements on reasoning-heavy tasks. The paper also demonstrates practical gains in efficiency through a concept-replication cross-attention technique and global-load-balancing to adapt segmentation to content density, outlining a path toward more efficient and capable language models that reason over learned concepts rather than raw tokens.

Abstract

Large Language Models (LLMs) apply uniform computation to all tokens, despite language exhibiting highly non-uniform information density. This token-uniform regime wastes capacity on locally predictable spans while under-allocating computation to semantically critical transitions. We propose $\textbf{Dynamic Large Concept Models (DLCM)}$, a hierarchical language modeling framework that learns semantic boundaries from latent representations and shifts computation from tokens to a compressed concept space where reasoning is more efficient. DLCM discovers variable-length concepts end-to-end without relying on predefined linguistic units. Hierarchical compression fundamentally changes scaling behavior. We introduce the first $\textbf{compression-aware scaling law}$, which disentangles token-level capacity, concept-level reasoning capacity, and compression ratio, enabling principled compute allocation under fixed FLOPs. To stably train this heterogeneous architecture, we further develop a $\textbf{decoupled $μ$P parametrization}$ that supports zero-shot hyperparameter transfer across widths and compression regimes. At a practical setting ($R=4$, corresponding to an average of four tokens per concept), DLCM reallocates roughly one-third of inference compute into a higher-capacity reasoning backbone, achieving a $\textbf{+2.69$\%$ average improvement}$ across 12 zero-shot benchmarks under matched inference FLOPs.

Figures (9)

• Figure 1: Overview Structure of DLCM.
• Figure 2: Cross-Attention Optimization via Concept Replication. Left: The decoder's cross-attention creates an irregular $L \times M$ mask due to variable token-to-concept mappings. Right: By replicating concepts via repeat_interleave to match token positions, we obtain a standard $L \times L$ causal mask, enabling optimized Flash Attention kernels.
• Figure 3: Hyperparameter tuning and transfer under $\mu$P. Left: We sweep $\eta^{\text{base}}_{\text{concept}}$ while fixing $\eta^{\text{base}}_{\text{token}}$ at its $\mu$P-predicted value on the 87M proxy model. The loss curve shows a well-defined minimum near $100\%$. Right: We jointly scale both base learning rates by the same factor and observe consistent minima across model sizes (87M–834M), validating the zero-shot transferability of $\mu$P-derived hyperparameters.
• Figure 4: Full training trajectory fit. Comparison between predicted loss (Equation \ref{['eq:scaling_law']}) and empirical loss across model sizes (274M--833M), compression factors $R \in \{2,4,8\}$, and training budgets. The joint fit achieves $R^2 > 0.98$.
• Figure 5: Decay-phase fit. Simplified fit on the final portion of training tokens, validating that our WSD scaling law accurately captures late-stage behaviour with $R^2 = 0.93$.