KARL: Knowledge Agents via Reinforcement Learning

TL;DR

Results show that tailored synthetic data in combination with multi-task reinforcement learning enables cost-efficient and high-performing knowledge agents for grounded reasoning.

Abstract

We present a system for training enterprise search agents via reinforcement learning that achieves state-of-the-art performance across a diverse suite of hard-to-verify agentic search tasks. Our work makes four core contributions. First, we introduce KARLBench, a multi-capability evaluation suite spanning six distinct search regimes, including constraint-driven entity search, cross-document report synthesis, tabular numerical reasoning, exhaustive entity retrieval, procedural reasoning over technical documentation, and fact aggregation over internal enterprise notes. Second, we show that models trained across heterogeneous search behaviors generalize substantially better than those optimized for any single benchmark. Third, we develop an agentic synthesis pipeline that employs long-horizon reasoning and tool use to generate diverse, grounded, and high-quality training data, with iterative bootstrapping from increasingly capable models. Fourth, we propose a new post-training paradigm based on iterative large-batch off-policy RL that is sample efficient, robust to train-inference engine discrepancies, and naturally extends to multi-task training with out-of-distribution generalization. Compared to Claude 4.6 and GPT 5.2, KARL is Pareto-optimal on KARLBench across cost-quality and latency-quality trade-offs, including tasks that were out-of-distribution during training. With sufficient test-time compute, it surpasses the strongest closed models. These results show that tailored synthetic data in combination with multi-task reinforcement learning enables cost-efficient and high-performing knowledge agents for grounded reasoning.

Figures (43)

• Figure 1: Performance of KARL, with and without test-time compute, compared to state-of-the-art agentic models on KARLBench. The cost–quality and latency–quality Pareto frontiers show that KARL achieves favorable trade-offs over existing models. All while being more cost and latency effective, KARL exceeds the quality of Sonnet 4.6 with three parallel rollouts and matches the best model, Opus 4.6, with ten parallel rollouts. See experiment details in app:exp_details.
• Figure 2: Stage I: The synthesis pipeline takes as input few-shot examples and the corpus for a task. Then, the Question-Answer Generator Agent explores the corpus via a vector search tool before proposing a possible synthetic question-answer pair that is grounded in the retrieved documents. To ensure no test data leakage, the Deduplication Agent filters out any exact or near-duplicates.
• Figure 3: Stage II: Multiple instantiations of the Solver Agent independently generate solutions for the synthetic questions produced in Stage I. Generated data at either extremes of difficulty, those solved on nearly all or almost no attempts, are filtered out, retaining only question-answer pairs where learning signal is richest. The Quality Filter Agent screens the remaining data points for ambiguity and incorrect reference answers. Synthetic data that pass both filters serve as inputs to RL training.
• Figure 4: Parallel thinking method: We first generate $N$ responses in the generation phase and then aggregate the $N$ rollouts. The solver agent and the aggregator agent here are the same model $\pi$ on which we apply TTC.
• Figure 5: Value-Guided Search method: Performs tree search, using a value model at each step to score candidate continuations, selecting the highest scoring branch. The search process is repeated $N$ times followed by aggregation.