LLMOrbit: A Circular Taxonomy of Large Language Models -From Scaling Walls to Agentic AI Systems

TL;DR

LLMOport surveys the journey from Transformer foundations to agentic AI, presenting a circular taxonomy that unites eight interdependent dimensions—scaling wall dynamics, model taxonomy, training methodologies, architectural innovations, eight paradigms breaking scaling limits, agentic frameworks, benchmarking, and economics. It argues that progress is increasingly driven by post-training optimization, test-time compute, and architectural efficiency rather than brute-force scaling, evidenced by open-source models rivaling closed systems and large MoE/MLA-based architectures delivering GPT-4-level performance at a fraction of the cost. The paper catalogs three crises—data scarcity, cost inflation, and energy growth—and presents eight paradigms (e.g., test-time compute, MoE sparsity, efficient training, model merging, small specialized models) that collectively form a viable post-scaling strategy. It also explores agentic AI as a natural extension of LLMs, detailing planning, tool use, memory, and multi-agent coordination, while addressing safety, oversight, and governance in autonomous systems. The synthesis points to a future where capability gains arise from data quality, verifiable reasoning, and efficient computation, enabling widespread, responsible deployment across domains while maintaining environmental sustainability.

Abstract

The field of artificial intelligence has undergone a revolution from foundational Transformer architectures to reasoning-capable systems approaching human-level performance. We present LLMOrbit, a comprehensive circular taxonomy navigating the landscape of large language models spanning 2019-2025. This survey examines over 50 models across 15 organizations through eight interconnected orbital dimensions, documenting architectural innovations, training methodologies, and efficiency patterns defining modern LLMs, generative AI, and agentic systems. We identify three critical crises: (1) data scarcity (9-27T tokens depleted by 2026-2028), (2) exponential cost growth ($3M to $300M+ in 5 years), and (3) unsustainable energy consumption (22x increase), establishing the scaling wall limiting brute-force approaches. Our analysis reveals six paradigms breaking this wall: (1) test-time compute (o1, DeepSeek-R1 achieve GPT-4 performance with 10x inference compute), (2) quantization (4-8x compression), (3) distributed edge computing (10x cost reduction), (4) model merging, (5) efficient training (ORPO reduces memory 50%), and (6) small specialized models (Phi-4 14B matches larger models). Three paradigm shifts emerge: (1) post-training gains (RLHF, GRPO, pure RL contribute substantially, DeepSeek-R1 achieving 79.8% MATH), (2) efficiency revolution (MoE routing 18x efficiency, Multi-head Latent Attention 8x KV cache compression enables GPT-4-level performance at <$0.30/M tokens), and (3) democratization (open-source Llama 3 88.6% MMLU surpasses GPT-4 86.4%). We provide insights into techniques (RLHF, PPO, DPO, GRPO, ORPO), trace evolution from passive generation to tool-using agents (ReAct, RAG, multi-agent systems), and analyze post-training innovations.

Figures (12)

• Figure 1: Evolution from LLM Foundation to Agentic AI. Three nested paradigms converge to a unified framework: LLM Foundation (blue) encompasses model evolution, scaling challenges, and architectural innovations; GenAI (purple) adds training methodologies (RLHF, PPO, DPO, GRPO, ORPO) and environments; Agentic AI (light blue) extends capabilities through reasoning (ReAct, CoT/ToT), tool use (RAG), and multi-agent systems. This nested architecture illustrates how foundation models serve as the base for generative capabilities, which in turn enable autonomous agentic systems.
• Figure 2: LLMOrbit: A Circular Taxonomy of Large Language Models (2019-2025). This circular orbital architecture presents eight interconnected dimensions navigating the complete LLM landscape: (1) Scaling Wall Analysis examining data scarcity, training costs, and energy consumption with quantitative projections; (2) Model Taxonomy covering 50+ foundation models including GPT-4, Gemini 1.5, Claude 3.5, Llama 3, DeepSeek-V3, Qwen 3, Mistral, Phi-4, and Nemotron; (3) Training Methodology encompassing RLHF, PPO, DPO, GRPO, ORPO, and pure reinforcement learning (DeepSeek-R1); (4) Architecture Evolution featuring FlashAttention, Grouped Query Attention (GQA), Mixture-of-Experts (MoE), Mamba state space models, RetNet, Linear Attention, and multimodal systems (CLIP, Gemini, GPT-4o, Sora); (5) Paradigms that Break the Scaling Wall including test-time compute scaling, MoE sparsity (18$\times$ efficiency), quantization scaling laws, distributed edge computing, model merging, efficient training algorithms, small specialized models, and post-training compression; (6) Agentic AI Frameworks with Chain-of-Thought (CoT), Tree-of-Thoughts (ToT), ReAct, Retrieval-Augmented Generation (RAG), tool use, and multi-agent systems; (7) Benchmarking Analysis across MMLU, MATH, GPQA, HumanEval, GSM8K, AIME, MT-Bench, AlpacaEval, and LiveCodeBench; and (8) Economic & Environmental considerations including hardware costs, amortization formulas, energy metrics (TDP$\times$PUE), and cloud versus on-premise deployment trade-offs. LLMOrbit synthesizes the complete landscape from foundational models to agentic AI systems, highlighting technical innovations, evaluation protocols, and practical deployment challenges.
• Figure 3: Training Costs Evolution: Exponential Growth Across Three Eras. Stacked bar chart showing hardware costs (blue: amortized chip depreciation + 23% networking overhead) and energy costs (orange: electricity consumption) for 8 frontier models spanning 2020-2025 besiroglu2024trainingepochai2024computetrends. Era annotations mark three periods: Early (2020-2021), Scaling (2022-2023), and Frontier (2024-2025). Key numbers demonstrate 100$\times$ cost explosion: GPT-3 ($3.3M total, 2020), GPT-4 ($84.5M total, 25$\times$ increase), and DeepSeek-V3 ($110.7M total, 2025). Selected models represent the most compute-intensive training runs for their time. Open circles indicate estimated Google TPU production costs (higher uncertainty). Cloud rental costs (not shown) are typically 2-4$\times$ higher than amortized ownership costs.
• Figure 4: Cloud vs. Amortized Costs: Ownership Economics for Large-Scale Training. Grouped bar chart comparing cloud rental pricing (darker bars) with amortized ownership costs (lighter bars: hardware + energy) for 8 frontier models besiroglu2024trainingepochai2024computetrends. Cost multipliers displayed above bar pairs show cloud costs are typically 2-4$\times$ higher than ownership due to provider margins (30-50%), maintenance overhead, and infrastructure costs. Color coding distinguishes reported costs (green/blue) from estimated values (red/orange), with open circles marking estimated cloud prices where official pricing was unavailable. Key insight: the cost gap widens for larger models as economies of scale increasingly favor ownership for sustained multi-month training runs, making cloud economical only for short-term or exploratory training.
• Figure 5: Data Exhaustion Projection: The Impending Data Scarcity Crisis. Scatter plot showing historical dataset sizes of actual models (GPT-3: 300B tokens, Llama 2: 2T tokens, Llama 3: 15T tokens, DeepSeek-V3: 14.8T tokens) with exponential growth trend line (2019-2025) and future projection assuming compute-optimal training epochai2022datawallepochai2024scaling. The shaded region (9-27 trillion tokens) indicates the estimated range of total available high-quality public text, including books, scientific papers, news articles, Wikipedia, and filtered web content. Under current scaling trends, the available data stock will be fully utilized between 2026 and 2028, creating a fundamental bottleneck for continued model scaling. Data sources: Epoch AI research and model technical reports.