LLMs as High-Dimensional Nonlinear Autoregressive Models with Attention: Training, Alignment and Inference
Vikram Krishnamurthy
TL;DR
The paper reconceptualizes LLMs as high-dimensional nonlinear autoregressive models with attention, linking training, alignment, and inference in a unified mathematical framework. It formalizes token generation as a $\Delta$-order autoregressive process with a learned state $f_{\theta,t}$ produced by a causal transformer, where next-token probabilities are given by a softmax over $W_{out} f_{\theta,t} + b_{out}$ with temperature $\tau$. It surveys pretraining (next-token prediction) and multiple alignment strategies—RLHF, Direct Preference Optimization (DPO), Rejection Sampling Fine-Tuning (RSFT), and RLVR—showing them as KL-regularized policy optimization with different reward constructions, and discusses inference-time phenomena such as hallucination, sycophancy, and RAG grounding. The work also covers extensions like vision-language multimodality, efficient KV caching for autoregressive inference, and prompting strategies such as ICL and CoT, framing continual learning and safety concerns (including hallucination mitigation and ethical considerations) within the same mathematical scaffold. Overall, the paper provides a principled reference for interpreting LLMs, analyzing their behavior under alignment and prompting, and guiding theoretical and practical advances in grounding, reliability, and long-horizon reasoning. It emphasizes that emergent reasoning and task adaptation arise from the same autoregressive mechanism conditioned by prompts, not from explicit symbolic modules, offering a rigorous lens for future research and deployment.
Abstract
Large language models (LLMs) based on transformer architectures are typically described through collections of architectural components and training procedures, obscuring their underlying computational structure. This review article provides a concise mathematical reference for researchers seeking an explicit, equation-level description of LLM training, alignment, and generation. We formulate LLMs as high-dimensional nonlinear autoregressive models with attention-based dependencies. The framework encompasses pretraining via next-token prediction, alignment methods such as reinforcement learning from human feedback (RLHF), direct preference optimization (DPO), rejection sampling fine-tuning (RSFT), and reinforcement learning from verifiable rewards (RLVR), as well as autoregressive generation during inference. Self-attention emerges naturally as a repeated bilinear--softmax--linear composition, yielding highly expressive sequence models. This formulation enables principled analysis of alignment-induced behaviors (including sycophancy), inference-time phenomena (such as hallucination, in-context learning, chain-of-thought prompting, and retrieval-augmented generation), and extensions like continual learning, while serving as a concise reference for interpretation and further theoretical development.