Learning Query-Specific Rubrics from Human Preferences for DeepResearch Report Generation

TL;DR

The paper tackles the lack of verifiable reward signals in DeepResearch long-form report generation by learning query-specific rubrics aligned with human preferences. It introduces a GRPO-based rubric generator trained with a hybrid reward that combines human preference signals and LLM-based rubric evaluation, and integrates these rubrics into a Multi-Agent Markov-State (MaMs) workflow to address long-horizon reasoning. Through a large-scale preference dataset and extensive experiments, the approach yields more discriminative and human-aligned supervision than static or LLM-generated rubrics, and, when used within MaMs, surpasses open-source baselines and rivals closed-source models on the DeepResearch Bench. The work demonstrates that learning evaluative criteria themselves can provide principled, scalable reinforcement learning signals for complex, multi-turn information synthesis tasks, with potential broad impact on AI-assisted research reporting.

Abstract

Nowadays, training and evaluating DeepResearch-generated reports remain challenging due to the lack of verifiable reward signals. Accordingly, rubric-based evaluation has become a common practice. However, existing approaches either rely on coarse, pre-defined rubrics that lack sufficient granularity, or depend on manually constructed query-specific rubrics that are costly and difficult to scale. In this paper, we propose a pipeline to train human-preference-aligned query-specific rubric generators tailored for DeepResearch report generation. We first construct a dataset of DeepResearch-style queries annotated with human preferences over paired reports, and train rubric generators via reinforcement learning with a hybrid reward combining human preference supervision and LLM-based rubric evaluation. To better handle long-horizon reasoning, we further introduce a Multi-agent Markov-state (MaMs) workflow for report generation. We empirically show that our proposed rubric generators deliver more discriminative and better human-aligned supervision than existing rubric design strategies. Moreover, when integrated into the MaMs training framework, DeepResearch systems equipped with our rubric generators consistently outperform all open-source baselines on the DeepResearch Bench and achieve performance comparable to that of leading closed-source models.

Figures (4)

• Figure 1: Overview of our method. (a) We construct diverse reporting queries and generate candidate reports. Human experts provide pairwise preference annotations based on usefulness, coherence, completeness, and alignment. (b) Given a query and its preferred and rejected reports, we train a rubric generator via GRPO to produce weighted evaluation rubrics, with rewards based on preference consistency, LLM-as-a-judge scores, and format validity. (c) Multi-Agent Markov-state (MaMs) workflow. A shared policy executes a structured workflow with search, chunking, state update, and report generation, interacting with external tools. Query-specific rubrics are used to compute rewards of the rollout reports under an individual query.
• Figure 2: Topic distribution of our created human preference dataset for DeepResearch reports.
• Figure 3: Comparison between GSPO and GRPO during training rubric generators (Qwen3-30B-A3B). (a) Reward curves of generated rollouts under the two algorithms, showing nearly identical reward values. (b) Entropy of generated rollouts, where GSPO consistently exhibits higher entropy than GRPO.
• Figure 4: Speed-up achieved by overlapping multiple micro-batches using the asynchronous event loop. The concurrency-limited scheduling allows high-latency API calls to run in parallel, maximizing resource utilization and reducing the effective runtime of the stage from linear in the dataset size $|\mathcal{D}|$ to approximately $|\mathcal{D}|/C$, where $C$ is the concurrency limit.