INTELLECT-3: Technical Report

TL;DR

INTELLECT-3 tackles the challenge of achieving strong reasoning and agentic capabilities in a large language model while remaining open and reproducible. The authors build a 106B-parameter Mixture-of-Experts model on top of GLM-4.5-Air and train it end-to-end using an open, scalable RL stack (prime-rl) with verifiers and the Environments Hub, supported by Prime Sandboxes for secure code execution and frontier GPU infrastructure. Their open infrastructure enables SFT followed by large-scale RL, achieving state-of-the-art results for its size on math, code, science, and logic benchmarks and even rivaling larger frontier models. By releasing the full framework, environments, and evaluation pipelines, INTELLECT-3 aims to democratize advanced RL pipelines for open research and community collaboration, while outlining scalable paths for even longer-horizon agentic reasoning.

Abstract

We present INTELLECT-3, a 106B-parameter Mixture-of-Experts model (12B active) trained with large-scale reinforcement learning on our end-to-end RL infrastructure stack. INTELLECT-3 achieves state of the art performance for its size across math, code, science and reasoning benchmarks, outperforming many larger frontier models. We open-source the model together with the full infrastructure stack used to create it, including RL frameworks, complete recipe, and a wide collection of environments, built with the verifiers library, for training and evaluation from our Environments Hub community platform. Built for this effort, we introduce prime-rl, an open framework for large-scale asynchronous reinforcement learning, which scales seamlessly from a single node to thousands of GPUs, and is tailored for agentic RL with first-class support for multi-turn interactions and tool use. Using this stack, we run both SFT and RL training on top of the GLM-4.5-Air-Base model, scaling RL training up to 512 H200s with high training efficiency.

Figures (11)

• Figure 1: INTELLECT-3 Evaluation Results.
• Figure 2: Architecture. A RL training run involves the coordination of a trainer, orchestrator and an inference service. The FSDP trainer and vLLM inference run disaggregated, and can be individually deployed across multiple nodes.
• Figure 3: Asynchronous Off-Policy Training. We show the execution graph of one-step off-policy training in an idealized setting where the trainer step time equals the inference step time. At step $n$, the inference engine uses a policy no older than $\theta_{\min{(0,n-1)}}$.
• Figure 4: Continuous Batching & In-Flight Weight Updates. Continuous batching maintains a constant inference load because finished rollout are immediately replaced with new rollout requests. The policy used to generate rollouts is updated in-flight as soon as it becomes available, allowing rollouts to be generated by multiple policies.
• Figure 5: Execution time and TFLOPS of torch._grouped_mm with hidden dim $4096$ and MoE dim $1408$ on H200 SXM at different sequence lengths and number of experts. We assume that the input is perfectly balanced between the experts and thus an increase in experts leads to an inversely proportional decrease in the number of tokens and work per expert, eventually causing lower TFLOPS as the work per expert is no longer able to saturate the kernel. At sequence lengths (N) $32,768$ and $65,536$, the TFLOPS remains in the saturated regime up to $128$ experts. We thus do not gain significant throughput from using expert parallel given our training parameters.