Physics of Language Models: Part 4.1, Architecture Design and the Magic of Canon Layers
Zeyuan Allen-Zhu
TL;DR
This work introduces Canon layers, lightweight horizontal mixing mechanisms that augment information flow across neighboring tokens in sequence models. Using a synthetic pretraining framework with five targeted tasks (Depo, Brevo, Capo, Mano, Lano), the study isolates core skills to compare architectural impacts under controlled conditions. Canon layers consistently boost reasoning depth and breadth, knowledge capacity, and hierarchical structure processing across Transformers and linear-model families, and they revive NoPE and reduce reliance on RoPE for long-context generalization. Real-world 1.3B-scale pretraining echoes the synthetic findings but reveals substantial noise and variance, underscoring the value of principled synthetic benchmarks for architecture design. The results advocate a shift toward horizontal mixing primitives, with Canon layers enabling deeper, more scalable reasoning across diverse architectures and future-generation models.
Abstract
Understanding architectural differences in language models is challenging, especially at academic-scale pretraining (e.g., 1.3B parameters, 100B tokens), where results are often dominated by noise and randomness. To overcome this, we introduce controlled synthetic pretraining tasks that isolate and evaluate core model capabilities. Within this framework, we discover CANON LAYERS: lightweight architectural components -- named after the musical term "canon" -- that promote horizontal information flow across neighboring tokens. Canon layers compute weighted sums of nearby token representations and integrate seamlessly into Transformers, linear attention, state-space models, or any sequence architecture. We present 12 key results. This includes how Canon layers enhance reasoning depth (e.g., by $2\times$), reasoning breadth, knowledge manipulation, etc. They lift weak architectures like NoPE to match RoPE, and linear attention to rival SOTA linear models like Mamba2/GDN -- validated both through synthetic tasks and real-world academic-scale pretraining. This synthetic playground offers an economical, principled path to isolate core model capabilities often obscured at academic scales. Equipped with infinite high-quality data, it may even PREDICT how future architectures will behave as training pipelines improve -- e.g., through better data curation or RL-based post-training -- unlocking deeper reasoning and hierarchical inference.