Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

An Expert Schema for Evaluating Large Language Model Errors in Scholarly Question-Answering Systems

Anna Martin-Boyle, William Humphreys, Martha Brown, Cara Leckey, Harmanpreet Kaur

TL;DR

A schema for evaluating LLM errors in scholarly question-answering systems that reflects the assessment strategies of practicing scientists and shows not only which errors experts naturally identify but also how structured evaluation schemas can help them detect previously overlooked issues.

Abstract

Large Language Models (LLMs) are transforming scholarly tasks like search and summarization, but their reliability remains uncertain. Current evaluation metrics for testing LLM reliability are primarily automated approaches that prioritize efficiency and scalability, but lack contextual nuance and fail to reflect how scientific domain experts assess LLM outputs in practice. We developed and validated a schema for evaluating LLM errors in scholarly question-answering systems that reflects the assessment strategies of practicing scientists. In collaboration with domain experts, we identified 20 error patterns across seven categories through thematic analysis of 68 question-answer pairs. We validated this schema through contextual inquiries with 10 additional scientists, which showed not only which errors experts naturally identify but also how structured evaluation schemas can help them detect previously overlooked issues. Domain experts use systematic assessment strategies, including technical precision testing, value-based evaluation, and meta-evaluation of their own practices. We discuss implications for supporting expert evaluation of LLM outputs, including opportunities for personalized, schema-driven tools that adapt to individual evaluation patterns and expertise levels.

An Expert Schema for Evaluating Large Language Model Errors in Scholarly Question-Answering Systems

TL;DR

A schema for evaluating LLM errors in scholarly question-answering systems that reflects the assessment strategies of practicing scientists and shows not only which errors experts naturally identify but also how structured evaluation schemas can help them detect previously overlooked issues.

Abstract

Large Language Models (LLMs) are transforming scholarly tasks like search and summarization, but their reliability remains uncertain. Current evaluation metrics for testing LLM reliability are primarily automated approaches that prioritize efficiency and scalability, but lack contextual nuance and fail to reflect how scientific domain experts assess LLM outputs in practice. We developed and validated a schema for evaluating LLM errors in scholarly question-answering systems that reflects the assessment strategies of practicing scientists. In collaboration with domain experts, we identified 20 error patterns across seven categories through thematic analysis of 68 question-answer pairs. We validated this schema through contextual inquiries with 10 additional scientists, which showed not only which errors experts naturally identify but also how structured evaluation schemas can help them detect previously overlooked issues. Domain experts use systematic assessment strategies, including technical precision testing, value-based evaluation, and meta-evaluation of their own practices. We discuss implications for supporting expert evaluation of LLM outputs, including opportunities for personalized, schema-driven tools that adapt to individual evaluation patterns and expertise levels.
Paper Structure (82 sections, 1 equation, 2 figures, 4 tables)

This paper contains 82 sections, 1 equation, 2 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Scholarly Question Answering
  4. Automated Evaluation of Question Answering
  5. Human Evaluation of Question Answering
  6. LLM Error Taxonomies
  7. Methods
  8. Scholarly Question-Answering Setup
  9. Task Description
  10. System Implementation
  11. Document Processing and Vectorization
  12. Retrieval
  13. Context Formatting and Answer Generation
  14. Phase 1: Schema Development Method
  15. Phase 2: Schema Validation Method
  16. ...and 67 more sections

Figures (2)

  • Figure 1: Errors identified by domain experts and model developers with entity tags for anonymity. The expert recognized a chronological errorabout test sequences that the developer missed, showing how domain expertise can yield more precise error analysis.
  • Figure 2: Distribution of error types across question categories. Each row is normalized to sum to 1, showing the proportion of errors within each question type. Column labels indicate the total occurrences of each error type across all questions ($n$). Question types are sorted by total error frequency (descending); error types are sorted by total occurrences (descending). Darker cells indicate that a given error type accounts for a larger share of errors for that question category.