Beyond the Black Box: Theory and Mechanism of Large Language Models

TL;DR

This paper addresses the lack of theoretical grounding for Large Language Models (LLMs) by introducing a lifecycle-based taxonomy that partitions theory into Data Preparation, Model Preparation, Training, Alignment, Inference, and Evaluation. It provides a systematic review of foundational theories and mechanisms across these stages, including data-mixture theory, representability of Transformers, pre-training scaling laws, RLHF dynamics, prompt engineering, and inference-time reasoning. The work highlights frontier challenges such as synthetic-data self-improvement limits, formal safety guarantees, and the mechanistic origins of emergent intelligence, offering a roadmap to shift LLM development from engineering heuristics to principled science. By connecting empirical observations with rigorous theory, it aims to unify disparate strands of LLM research and guide future investigations toward safe, trustworthy, and scalable AI systems.

Abstract

The rapid emergence of Large Language Models (LLMs) has precipitated a profound paradigm shift in Artificial Intelligence, delivering monumental engineering successes that increasingly impact modern society. However, a critical paradox persists within the current field: despite the empirical efficacy, our theoretical understanding of LLMs remains disproportionately nascent, forcing these systems to be treated largely as ``black boxes''. To address this theoretical fragmentation, this survey proposes a unified lifecycle-based taxonomy that organizes the research landscape into six distinct stages: Data Preparation, Model Preparation, Training, Alignment, Inference, and Evaluation. Within this framework, we provide a systematic review of the foundational theories and internal mechanisms driving LLM performance. Specifically, we analyze core theoretical issues such as the mathematical justification for data mixtures, the representational limits of various architectures, and the optimization dynamics of alignment algorithms. Moving beyond current best practices, we identify critical frontier challenges, including the theoretical limits of synthetic data self-improvement, the mathematical bounds of safety guarantees, and the mechanistic origins of emergent intelligence. By connecting empirical observations with rigorous scientific inquiry, this work provides a structured roadmap for transitioning LLM development from engineering heuristics toward a principled scientific discipline.

Figures (7)

• Figure 1: The roadmap of LLM theory and mechanisms. We organize the fragmented theoretical landscape into a unified lifecycle consisting of six stages: Data Preparation, Model Preparation, Training, Alignment, Inference, and Evaluation. The figure visualizes the flow of theoretical inquiry, mapping key sub-topics and algorithmic mechanisms to their respective developmental phases.
• Figure 2: An overview of the theoretical landscape in the Data Preparation Stage. This stage is categorized into two dimensions: (a) Core Theories & Methods addresses foundational mechanisms including Data Mixture Efficacy (optimizing the proportions of heterogeneous data sources for generalization), Data Deduplication & Filtering (strategies to enhance training efficiency by dropping redundant data), and Memorization (analyzing the trade-off between verbatim recall and reasoning capabilities). (b) Advanced Topics & Open Questions highlights frontier challenges, specifically Synthetic Data Generation (investigating the theoretical limits of recursive self-improvement) and Data Contamination (addressing the impact of benchmark leakage on evaluation integrity).
• Figure 3: An overview of the theoretical landscape in the Model Preparation Stage. This stage is categorized into two dimensions: (a) Core Theories & Methods addresses foundational principles including Representability (analyzing expressive power and fundamental limits), Optimization Characteristics (investigating training dynamics and properties), and Theoretical Design (interpreting internal operations through formal frameworks). (b) Advanced Topics & Open Questions highlights frontier challenges, specifically Linear Models (addressing the efficiency-representation trade-off) and Recurrent Models (exploring weight-tied architectures for iterative reasoning).
• Figure 4: An overview of the theoretical landscape in the Training Stage. This stage is categorized into two dimensions: (a) Core Theories & Methods addresses mechanisms of knowledge acquisition, including Analysis on Pre-Training (foundations of knowledge acquisition and scaling laws) and Guarantees of Model Tuning (mechanisms and optimization of fine-tuning paradigms). (b) Advanced Topics & Open Questions highlights frontier challenges, specifically Hyperparameter Transfer (zero-shot transfer of configurations across scales) and Evolution of Optimizers (matrix-aware and adaptive methods for LLMs).
• Figure 5: An overview of the theoretical landscape in the Alignment Stage. This stage is categorized into two dimensions: (a) Core Theories & Methods addresses the foundations of steering behavior, including Foundations for AI Alignment (safety limits and weak-to-strong generalization) and Reinforcement Learning for Alignment (mechanisms of preference-based optimization). (b) Advanced Topics & Open Questions highlights emerging frontiers, specifically Relationship Between Training & Alignment (distinctions between SFT and RL mechanisms) and Frontier of RL (dynamic exploration-exploitation and agentic reasoning).