Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Evaluating LLM-Generated Versus Human-Authored Responses in Role-Play Dialogues

Dongxu Lu, Johan Jeuring, Albert Gatt

TL;DR

Addressing evaluation challenges in long-form, knowledge-grounded role-play dialogues, the paper contrasts LLM-generated and human-authored responses across multi-turn simulations using both human judgments and an automated LLM-as-a-judge. It finds that LLM performance degrades as dialogues progress while human-authored responses improve, with broad human preference for humans, a result corroborated by a validated LLM-judge (Gemini 2.0 Flash) under carefully designed prompting. The study demonstrates cross-scenario generalisability to three additional domains and establishes a scalable hybrid evaluation framework for training simulations. Together, these findings highlight current limitations in sustaining long-context coherence in LLMs and offer practical benchmarks and methodologies to guide responsible integration of LLMs into knowledge-grounded role-play systems.

Abstract

Evaluating large language models (LLMs) in long-form, knowledge-grounded role-play dialogues remains challenging. This study compares LLM-generated and human-authored responses in multi-turn professional training simulations through human evaluation ($N=38$) and automated LLM-as-a-judge assessment. Human evaluation revealed significant degradation in LLM-generated response quality across turns, particularly in naturalness, context maintenance and overall quality, while human-authored responses progressively improved. In line with this finding, participants also indicated a consistent preference for human-authored dialogue. These human judgements were validated by our automated LLM-as-a-judge evaluation, where Gemini 2.0 Flash achieved strong alignment with human evaluators on both zero-shot pairwise preference and stochastic 6-shot construct ratings, confirming the widening quality gap between LLM and human responses over time. Our work contributes a multi-turn benchmark exposing LLM degradation in knowledge-grounded role-play dialogues and provides a validated hybrid evaluation framework to guide the reliable integration of LLMs in training simulations.

Evaluating LLM-Generated Versus Human-Authored Responses in Role-Play Dialogues

TL;DR

Addressing evaluation challenges in long-form, knowledge-grounded role-play dialogues, the paper contrasts LLM-generated and human-authored responses across multi-turn simulations using both human judgments and an automated LLM-as-a-judge. It finds that LLM performance degrades as dialogues progress while human-authored responses improve, with broad human preference for humans, a result corroborated by a validated LLM-judge (Gemini 2.0 Flash) under carefully designed prompting. The study demonstrates cross-scenario generalisability to three additional domains and establishes a scalable hybrid evaluation framework for training simulations. Together, these findings highlight current limitations in sustaining long-context coherence in LLMs and offer practical benchmarks and methodologies to guide responsible integration of LLMs into knowledge-grounded role-play systems.

Abstract

Evaluating large language models (LLMs) in long-form, knowledge-grounded role-play dialogues remains challenging. This study compares LLM-generated and human-authored responses in multi-turn professional training simulations through human evaluation () and automated LLM-as-a-judge assessment. Human evaluation revealed significant degradation in LLM-generated response quality across turns, particularly in naturalness, context maintenance and overall quality, while human-authored responses progressively improved. In line with this finding, participants also indicated a consistent preference for human-authored dialogue. These human judgements were validated by our automated LLM-as-a-judge evaluation, where Gemini 2.0 Flash achieved strong alignment with human evaluators on both zero-shot pairwise preference and stochastic 6-shot construct ratings, confirming the widening quality gap between LLM and human responses over time. Our work contributes a multi-turn benchmark exposing LLM degradation in knowledge-grounded role-play dialogues and provides a validated hybrid evaluation framework to guide the reliable integration of LLMs in training simulations.

Paper Structure

This paper contains 49 sections, 8 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Automatic Dialogue Evaluation
  4. LLM-as-a-Judge Paradigm
  5. Multi-Turn Dialogue Evaluation
  6. Experiment 1: Human Evaluation
  7. Participants
  8. Materials and Stimuli
  9. Measures
  10. Procedure
  11. In-simulation pairwise preference
  12. Post-simulation rating
  13. Focus Group
  14. Data Analysis
  15. Results
  16. ...and 34 more sections

Figures (8)

  • Figure 1: Overview of the training scenario structure. Each scenario includes contextual knowledge (e.g., player and character roles) and a dialogue tree. Agent statements are represented by red nodes and player choices by blue nodes. The highlighted sequence represents the 'best-practice path'.
  • Figure 2: Perceived quality ratings of LLM-generated (orange) and human-authored (blue) responses over turns. Dotted lines connect the mean rating at each turn for each response condition. Solid lines represent the overall linear regression (OLS) trend.
  • Figure 3: Proportion of participants preferring the LLM-generated response over turns. Data points represent the preference proportion at each turn. The dashed OLS trend line and its shaded 95% confidence interval illustrate that there was no statistically significant trend over time ($p>.5$).
  • Figure 4: Heatmap of Spearman correlation coefficients ($r_s$) between all rated quality constructs.
  • Figure 5: The prompt template for response generation.
  • ...and 3 more figures