Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

JADE: Expert-Grounded Dynamic Evaluation for Open-Ended Professional Tasks

Lanbo Lin, Jiayao Liu, Tianyuan Yang, Li Cai, Yuanwu Xu, Lei Wei, Sicong Xie, Guannan Zhang

TL;DR

JADE tackles the stability–adaptivity dilemma in evaluating open-ended professional tasks by introducing a two-layer, expert-grounded framework that first activates domain-specific skills to form a query rubric and then performs report-specific, claim-level verification with dependency gating. The approach combines deterministic skill activation with dynamic verification, yielding stable alignment with expert rubrics while adapting to diverse reasoning strategies and evidence patterns. Empirical validation on BizBench shows JADE improves evaluation robustness, reveals failure modes like citation hallucination and evidence–reasoning gaps, and transfers to HealthBench, confirming cross-domain applicability. The work provides a publicly available evaluation pipeline and demonstrates that expert-grounded, dynamic evaluation can better support reliable deployment of professional agents in real-world settings.

Abstract

Evaluating agentic AI on open-ended professional tasks faces a fundamental dilemma between rigor and flexibility. Static rubrics provide rigorous, reproducible assessment but fail to accommodate diverse valid response strategies, while LLM-as-a-judge approaches adapt to individual responses yet suffer from instability and bias. Human experts address this dilemma by combining domain-grounded principles with dynamic, claim-level assessment. Inspired by this process, we propose JADE, a two-layer evaluation framework. Layer 1 encodes expert knowledge as a predefined set of evaluation skills, providing stable evaluation criteria. Layer 2 performs report-specific, claim-level evaluation to flexibly assess diverse reasoning strategies, with evidence-dependency gating to invalidate conclusions built on refuted claims. Experiments on BizBench show that JADE improves evaluation stability and reveals critical agent failure modes missed by holistic LLM-based evaluators. We further demonstrate strong alignment with expert-authored rubrics and effective transfer to a medical-domain benchmark, validating JADE across professional domains. Our code is publicly available at https://github.com/smiling-world/JADE.

JADE: Expert-Grounded Dynamic Evaluation for Open-Ended Professional Tasks

TL;DR

JADE tackles the stability–adaptivity dilemma in evaluating open-ended professional tasks by introducing a two-layer, expert-grounded framework that first activates domain-specific skills to form a query rubric and then performs report-specific, claim-level verification with dependency gating. The approach combines deterministic skill activation with dynamic verification, yielding stable alignment with expert rubrics while adapting to diverse reasoning strategies and evidence patterns. Empirical validation on BizBench shows JADE improves evaluation robustness, reveals failure modes like citation hallucination and evidence–reasoning gaps, and transfers to HealthBench, confirming cross-domain applicability. The work provides a publicly available evaluation pipeline and demonstrates that expert-grounded, dynamic evaluation can better support reliable deployment of professional agents in real-world settings.

Abstract

Evaluating agentic AI on open-ended professional tasks faces a fundamental dilemma between rigor and flexibility. Static rubrics provide rigorous, reproducible assessment but fail to accommodate diverse valid response strategies, while LLM-as-a-judge approaches adapt to individual responses yet suffer from instability and bias. Human experts address this dilemma by combining domain-grounded principles with dynamic, claim-level assessment. Inspired by this process, we propose JADE, a two-layer evaluation framework. Layer 1 encodes expert knowledge as a predefined set of evaluation skills, providing stable evaluation criteria. Layer 2 performs report-specific, claim-level evaluation to flexibly assess diverse reasoning strategies, with evidence-dependency gating to invalidate conclusions built on refuted claims. Experiments on BizBench show that JADE improves evaluation stability and reveals critical agent failure modes missed by holistic LLM-based evaluators. We further demonstrate strong alignment with expert-authored rubrics and effective transfer to a medical-domain benchmark, validating JADE across professional domains. Our code is publicly available at https://github.com/smiling-world/JADE.
Paper Structure (32 sections, 18 equations, 5 figures, 5 tables)

This paper contains 32 sections, 18 equations, 5 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Deep Research Agents and Professional Benchmarks
  4. Benchmark Environments and Data Construction
  5. Evaluation of Open-Ended Professional Reports
  6. JADE: Judge Agents with Dynamic Evaluation
  7. Notation and Overview
  8. Layer 1: Query-Specific Checklist Generation
  9. Layer 2: Report-Specific Checklist and Claim Extraction
  10. Scoring: Verification Agent and LLM Judge
  11. Dependency Gating
  12. Final Scoring and Theoretical Properties
  13. BizBench: A Benchmark for Validating JADE
  14. Domain and Data Collection
  15. Hierarchical Task Taxonomy
  16. ...and 17 more sections

Figures (5)

  • Figure 1: Example of End-to-End Evaluation of JADE on BizBench
  • Figure 2: Overview of JADE. Given a query and an agent-generated response, JADE first activates appropriate expert-authored skills to guide the generation of query-specific checklists. It then derives report-specific checklists for verifiable factual claims and reasoning quality. Factual claims are validated via real-time web verification, while reasoning is assessed by LLMs conditioned on the query-specific checklists, with evidence-based gating to ensure that unsupported facts invalidate dependent judgments.
  • Figure 3: Overview of BizBench. Queries are collected from real B2B sourcing and market research scenarios, filtered and de-identified, and verified by domain experts. Each query is annotated using a hierarchical taxonomy that instantiates JADE’s dynamic evaluation skills.
  • Figure 4: Evidence-Reasoning Gap.
  • Figure 5: JADE stability across 3 runs. Error bars show $\pm 2\sigma$.