Youtu-LLM: Unlocking the Native Agentic Potential for Lightweight Large Language Models

TL;DR

Youtu-LLM addresses the demand for affordable yet capable agents by pre-training a 1.96B parameter model from scratch with native agentic capabilities. It combines a dense Multi-Latent Attention architecture, a 128k context window, and a principled Commonsense-STEM-Agent curriculum, including a scalable Agentic Mid-training data plant that fuses Agentic-CoT, Math, Code, Deep Research, and Tool-use trajectories totaling hundreds of billions of tokens. Through four-stage pre-training and a two-stage supervised fine-tuning plus reinforcement learning regime, the model achieves state-of-the-art results among sub-2B LLMs on agentic benchmarks and competitive performance on general benchmarks. The work demonstrates that carefully engineered trajectory data and agent-centric training can unlock robust planning, reflection, and tool-use capabilities in lightweight LLMs, with significant practical implications for on-device and resource-constrained deployments.

Abstract

We introduce Youtu-LLM, a lightweight yet powerful language model that harmonizes high computational efficiency with native agentic intelligence. Unlike typical small models that rely on distillation, Youtu-LLM (1.96B) is pre-trained from scratch to systematically cultivate reasoning and planning capabilities. The key technical advancements are as follows: (1) Compact Architecture with Long-Context Support: Built on a dense Multi-Latent Attention (MLA) architecture with a novel STEM-oriented vocabulary, Youtu-LLM supports a 128k context window. This design enables robust long-context reasoning and state tracking within a minimal memory footprint, making it ideal for long-horizon agent and reasoning tasks. (2) Principled "Commonsense-STEM-Agent" Curriculum: We curated a massive corpus of approximately 11T tokens and implemented a multi-stage training strategy. By progressively shifting the pre-training data distribution from general commonsense to complex STEM and agentic tasks, we ensure the model acquires deep cognitive abilities rather than superficial alignment. (3) Scalable Agentic Mid-training: Specifically for the agentic mid-training, we employ diverse data construction schemes to synthesize rich and varied trajectories across math, coding, and tool-use domains. This high-quality data enables the model to internalize planning and reflection behaviors effectively. Extensive evaluations show that Youtu-LLM sets a new state-of-the-art for sub-2B LLMs. On general benchmarks, it achieves competitive performance against larger models, while on agent-specific tasks, it significantly surpasses existing SOTA baselines, demonstrating that lightweight models can possess strong intrinsic agentic capabilities.

Figures (26)

• Figure 1: Parameter–performance scaling of base and instruct models on agentic benchmarks. The trend line represents the desired agent performance with the smallest possible number of parameters, among which Youtu-LLM stands out as a lightweight yet strong performer.
• Figure 2: Multi-field general capability comparison of similarly sized models. Youtu-LLM shows a balanced and competitive profile, highlighting its general-purpose performance potential under limited parameter budgets.
• Figure 3: Above: Vanilla long CoT enables thorough response preparation, while it can also tend to overthinking and unnecessary repetition. Below: Our agentic thinking paradigm implements a defined reasoning architecture that guides models through sequential steps of analysis, plan, action, reflection and summary. This disciplined process fosters the development of agentic capabilities.
• Figure 4: The illustration of the designed math agent framework for math trajectory construction.
• Figure 5: The Synthesis pipeline of Code Trajectories.