Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ProfBench: Multi-Domain Rubrics requiring Professional Knowledge to Answer and Judge

Zhilin Wang, Jaehun Jung, Ximing Lu, Shizhe Diao, Ellie Evans, Jiaqi Zeng, Pavlo Molchanov, Yejin Choi, Jan Kautz, Yi Dong

TL;DR

ProfBench introduces a first-of-its-kind, expert-annotated rubric benchmark spanning Physics PhD, Chemistry PhD, Finance MBA, and Consulting MBA to evaluate LLMs on professional tasks requiring document processing and synthesis. It couples human-created rubrics with low-cost, bias-mitigated LLM-Judges to assess both response quality and rubric-fulfillment, revealing substantial challenges even for state-of-the-art models (top GPT-5-high at $65.9\%$ overall) and notable disparities between closed- and open-weight models. The framework demonstrates that grounding documents and tailored inference setups can substantially influence evaluation outcomes, while dynamic allocation strategies can reduce cost without sacrificing stability. Collectively, ProfBench offers a scalable, accessible platform for rigorous, domain-specific evaluation with practical implications for deploying LLMs in professional settings.

Abstract

Evaluating progress in large language models (LLMs) is often constrained by the challenge of verifying responses, limiting assessments to tasks like mathematics, programming, and short-form question-answering. However, many real-world applications require evaluating LLMs in processing professional documents, synthesizing information, and generating comprehensive reports in response to user queries. We introduce ProfBench: a set of over 7000 response-criterion pairs as evaluated by human-experts with professional knowledge across Physics PhD, Chemistry PhD, Finance MBA and Consulting MBA. We build robust and affordable LLM-Judges to evaluate ProfBench rubrics, by mitigating self-enhancement bias and reducing the cost of evaluation by 2-3 orders of magnitude, to make it fair and accessible to the broader community. Our findings reveal that ProfBench poses significant challenges even for state-of-the-art LLMs, with top-performing models like GPT-5-high achieving only 65.9\% overall performance. Furthermore, we identify notable performance disparities between proprietary and open-weight models and provide insights into the role that extended thinking plays in addressing complex, professional-domain tasks. Data: https://huggingface.co/datasets/nvidia/ProfBench and Code: https://github.com/NVlabs/ProfBench

ProfBench: Multi-Domain Rubrics requiring Professional Knowledge to Answer and Judge

TL;DR

ProfBench introduces a first-of-its-kind, expert-annotated rubric benchmark spanning Physics PhD, Chemistry PhD, Finance MBA, and Consulting MBA to evaluate LLMs on professional tasks requiring document processing and synthesis. It couples human-created rubrics with low-cost, bias-mitigated LLM-Judges to assess both response quality and rubric-fulfillment, revealing substantial challenges even for state-of-the-art models (top GPT-5-high at overall) and notable disparities between closed- and open-weight models. The framework demonstrates that grounding documents and tailored inference setups can substantially influence evaluation outcomes, while dynamic allocation strategies can reduce cost without sacrificing stability. Collectively, ProfBench offers a scalable, accessible platform for rigorous, domain-specific evaluation with practical implications for deploying LLMs in professional settings.

Abstract

Evaluating progress in large language models (LLMs) is often constrained by the challenge of verifying responses, limiting assessments to tasks like mathematics, programming, and short-form question-answering. However, many real-world applications require evaluating LLMs in processing professional documents, synthesizing information, and generating comprehensive reports in response to user queries. We introduce ProfBench: a set of over 7000 response-criterion pairs as evaluated by human-experts with professional knowledge across Physics PhD, Chemistry PhD, Finance MBA and Consulting MBA. We build robust and affordable LLM-Judges to evaluate ProfBench rubrics, by mitigating self-enhancement bias and reducing the cost of evaluation by 2-3 orders of magnitude, to make it fair and accessible to the broader community. Our findings reveal that ProfBench poses significant challenges even for state-of-the-art LLMs, with top-performing models like GPT-5-high achieving only 65.9\% overall performance. Furthermore, we identify notable performance disparities between proprietary and open-weight models and provide insights into the role that extended thinking plays in addressing complex, professional-domain tasks. Data: https://huggingface.co/datasets/nvidia/ProfBench and Code: https://github.com/NVlabs/ProfBench
Paper Structure (54 sections, 3 equations, 5 figures, 5 tables)

This paper contains 54 sections, 3 equations, 5 figures, 5 tables.

Table of Contents

  1. Introduction
  2. ProfBench Overview
  3. Data Collection
  4. Annotator Recruitment
  5. Prompt Curation
  6. Rubric Creation
  7. Response Annotation
  8. Benchmarking Models as LLM-Judges
  9. Task Formulation
  10. Evaluation
  11. Agreement with Human Annotations
  12. Bias
  13. Cost
  14. Inference Setup
  15. Human Validation
  16. ...and 39 more sections

Figures (5)

  • Figure 1: Example from ProfBench (Finance MBA) is substantially more challenging and detailed compared with Deep Research-Bench Finance & Business du2025deepresearchbenchcomprehensivebenchmark. More examples in § \ref{['app:examples']}.
  • Figure 2: Distribution of rubrics by category. Reasoning dominates (62.9%), with most on logical validity and correctness. Extraction accounts for 34.1%, emphasizing accurate retrieval of information (w/o meaningful subcategories). Style is minor (3.0%), with a focus on formatting and clarity.
  • Figure 3: Score Distribution and Optimal Samples for various tasks. Each blue box represents 25th, 50th and 75th percentile and whiskers represent the worst and best score out of 16 samples for a task by Gemini-2.5-Flash (Thinking). Tasks with lower variance can be estimated with fewer samples (shown as height of orange bar), reducing inference cost w/o sacrificing estimation robustness.
  • Figure 4: Optimal ProfBench performance at each price-point. OpenAI models are on the Pareto Frontier at each price-point, likely because of first-mover advantages in rubric-style data such as HealthBench arora2025healthbenchevaluatinglargelanguage and PaperBench starace2025paperbenchevaluatingaisability. Gemini-2.5 and Qwen3-Instruct-2507 models are close to the Pareto Frontier.
  • Figure 5: Standard deviation of overall performance using multiple samples per task. Optimal allocation of samples consistently reduce the variance across all budget levels.