Expert Preference-based Evaluation of Automated Related Work Generation

TL;DR

This paper introduces GREP, a granular, expert-preference–driven evaluation framework for automated related-work generation. It combines hard constraints (e.g., citation verification, coherence, proper positioning) and soft constraints (e.g., length, emphasis, positioning type/ratio) within a multi-turn, feedback-driven setup, including Precise GREP and Open GREP variants. A new dataset of 44 NLP papers with 644 cited works is built from open-access sources to ground evaluation in domain content. Experiments with expert judges and multiple LLMs show GREP aligns more closely with human expert judgments than standard LLM-as-a-judge approaches, while highlighting challenges faced by SoTA models in satisfying domain-specific constraints and adapting to dynamic writing preferences. The work demonstrates the potential to reduce human-in-the-loop costs and to facilitate robust, expert-aligned evaluation for scientific writing tasks, with future directions including broader-domain applications and enhanced search-era assistance.

Abstract

Expert domain writing, such as scientific writing, typically demands extensive domain knowledge. Although large language models (LLMs) show promising potential in this task, evaluating the quality of automatically generated scientific writing is a crucial open issue, as it requires knowledge of domain-specific criteria and the ability to discern expert preferences. Conventional task-agnostic automatic evaluation metrics and LLM-as-a-judge systems, primarily designed for mainstream NLP tasks, are insufficient to grasp expert preferences and domain-specific quality standards. To address this gap and support realistic human-AI collaborative writing, we focus on related work generation, one of the most challenging scientific tasks, as an exemplar. We propose GREP, a multi-turn evaluation framework that integrates classical related work evaluation criteria with expert-specific preferences. Our framework decomposes the evaluation into smaller fine-grained dimensions. This localized evaluation is further augmented with contrastive examples to provide detailed contextual guidance for the evaluation dimensions. Empirical investigation reveals that our framework is able to assess the quality of related work sections in a much more robust manner compared to standard LLM judges, reflects natural scenarios of scientific writing, and bears a strong correlation with the assessment of human experts. We also observe that generations from state-of-the-art (SoTA) LLMs struggle to satisfy validation constraints of a suitable related work section.

Figures (9)

• Figure 1: Illustrative description of GREP. Generated related work drafts are evaluated by dedicated modules that consider hard and soft constraints. Oracle with access to the gold RW section defines the preferences over soft constraints. Natural language feedback is generated based on the evaluation report to guide the generator LLM in producing the revised draft in the next iteration.
• Figure 2: Overall results on Precise GREP with four generator LLMs. Scores for each criterion are averaged across RW sections. Coherence is the hardest test to pass, while all models deliver perfect scores for Positioning Existence.
• Figure 3: Adaptability to new paper introduction evaluated by Precise GREP. Missing paper increases at the point of new paper introduction (3rd iteration), implying the inability to accommodate new information.
• Figure 4: Adaptability to style change evaluated by Precise GREP. Positioning type and ratio-based score drops at the point of change (3rd iteration), and models struggle to acquire original performance even after repeated feedback.
• Figure 5: Improvements per iteration in hard constraint passing rates and soft constraint passing, evaluated by Precise GREP.