Language Modeling by Language Models
Junyan Cheng, Peter Clark, Kyle Richardson
TL;DR
Language Modeling by Language Models introduces Genesys, a fully autonomous framework that uses LLM-driven designer and verifier agents to discover novel LM block architectures beyond transformers. Central to the approach are the Language Model Architecture Discovery Environment (LMADE) and a Ladder of Scales verification scheme, enabling progressive, budget-aware pretraining and evaluation across model scales. Across large-scale experiments, the authors report thousands of discovered designs and show competitive performance relative to established baselines, with ablations underscoring the importance of verification, literature grounding, and unit-based code generation. Overall, the work demonstrates the feasibility of autonomous architectural discovery in ML and provides a structured methodology and empirical insights for future discovery systems.
Abstract
Can we leverage LLMs to model the process of discovering novel language model (LM) architectures? Inspired by real research, we propose a multi-agent LLM approach that simulates the conventional stages of research, from ideation and literature search (proposal stage) to design implementation (code generation), generative pre-training, and downstream evaluation (verification). Using ideas from scaling laws, our system, Genesys, employs a Ladder of Scales approach; new designs are proposed, adversarially reviewed, implemented, and selectively verified at increasingly larger model scales (14M$\sim$350M parameters) with a narrowing budget (the number of models we can train at each scale). To help make discovery efficient and factorizable, Genesys uses a novel genetic programming backbone, which we show has empirical advantages over commonly used direct prompt generation workflows (e.g., $\sim$86\% percentage point improvement in successful design generation, a key bottleneck). We report experiments involving 1,162 newly discovered designs (1,062 fully verified through pre-training) and find the best designs to be highly competitive with known architectures (e.g., outperform GPT2, Mamba2, etc., on 6/9 common benchmarks). We couple these results with comprehensive system-level ablations and formal results, which give broader insights into the design of effective autonomous discovery systems.