DR Tulu: Reinforcement Learning with Evolving Rubrics for Deep Research

TL;DR

DR Tulu introduces Reinforcement Learning with Evolving Rubrics (RLER) to train open, long-form deep research agents. By co-evolving instance-specific rubrics grounded in retrieved knowledge, the approach provides discriminative, on-policy feedback and reduces reward hacking, enabling an 8B open model to rival proprietary systems on four long-form DR benchmarks. The work also delivers a scalable MCP-based tooling backend (dr-agent-lib) and asynchronous tool calling to support complex search workflows, achieving strong performance with significantly lower per-query cost. Extensive analyses on training dynamics, rubric design, and domain adaptation demonstrate the robustness and generalizability of DR Tulu and RLER, with full data/code releases to foster future research.

Abstract

Deep research models perform multi-step research to produce long-form, well-attributed answers. However, most open deep research models are trained on easily verifiable short-form QA tasks via reinforcement learning with verifiable rewards (RLVR), which does not extend to realistic long-form tasks. We address this with Reinforcement Learning with Evolving Rubrics (RLER), in which we construct and maintain rubrics that co-evolve with the policy model during training; this allows the rubrics to incorporate information that the model has newly explored and to provide discriminative, on-policy feedback. Using RLER, we develop Deep Research Tulu (DR Tulu-8B), the first open model that is directly trained for open-ended, long-form deep research. Across four long-form deep research benchmarks in science, healthcare and general domains, DR Tulu substantially outperforms existing open deep research models, and matches or exceeds proprietary deep research systems, while being significantly smaller and cheaper per query. To facilitate future research, we release all data, models, and code, including our new MCP-based agent infrastructure for deep research systems.

Figures (39)

• Figure 1: Performance vs. cost of deep research models. We report average performance over 4 long-form DR benchmarks (ScholarQA-CSv2, HealthBench, ResearchQA, and DeepResearchBench) against inference cost (USD per query on ScholarQA-CSv2). DR Tulu-8B lies on the Pareto frontier, outperforming larger open models and matching more expensive proprietary models. Full results in Table \ref{['tab:main_results']}.
• Figure 1: The fraction of assertive and factual rubrics. Both the initial search-based rubrics as well as the evolving rubrics (which continue to use search, as they are generated based on the full rollouts including search traces) have a higher proportion of assertive claims compared to closed-book or general rubrics.
• Figure 2: Overview of training a deep research model with reinforcement learning with evolving rubrics (RLER). Left: An example of a question and a long-form response from DR Tulu with citations. Right: We train the policy model on a dynamic set of rubrics that (1) co-evolve with the policy update (details in Figure \ref{['fig:main_workflow']}) and (2) are grounded on real-world, searched knowledge from the environment. Compared to commonly-used closed-book rubrics generated purely from LM parametric knowledge (blue circle), our evolving rubrics incorporate newly searched information and are continuously tailored to the current policy model's behaviors, better capturing the nuances required for long-form DR tasks.
• Figure 3: Training with RLER. Given a training instance, the policy LM $\pi_{\theta_t}$ generates multiple rollouts via interacting with the environment. We then invoke another LM to create new rubrics based on those rollouts and the current rubrics in the rubric buffer. We score each generation against those rubrics and use that score to update model weights. We then add the new rubrics to the rubric buffer and filter it to retain only a fixed number of rubrics with the highest variance among rollouts.
• Figure 4: Effect of negative evolving rubrics. Over training, negative evolving rubrics emerge that penalize undesirable behavior such as responding in Python (right), resulting in a reduction in undesirable behaviors over the course of training compared to using a static closed-book rubric that does not specify such undesirable behavior (left).