CoReflect: Conversational Evaluation via Co-Evolutionary Simulation and Reflective Rubric Refinement

TL;DR

CoReflect tackles the challenge of evaluating conversational agents in multi-turn settings by eliminating static evaluation scripts. It couples a planner-driven user simulator with an LLM-based judge and a reflective analyzer to co-evolve test templates and rubrics over $T$ iterations, feeding insights back to improve both components. The approach yields higher rubric discriminability and stability, enabling finer-grained model stratification across diverse personas and scenarios. By reducing human intervention, CoReflect offers a scalable, self-refining framework that adapts evaluation to rapidly advancing dialogue models and supports robust personalization assessment.

Abstract

Evaluating conversational systems in multi-turn settings remains a fundamental challenge. Conventional pipelines typically rely on manually defined rubrics and fixed conversational context$-$a static approach that limits coverage and fails to capture the diverse, emergent behaviors of dialogue models. To address this, we introduce CoReflect (Conversational Evaluation via Co-Evolutionary Simulation and Reflective Rubric Refinement), which unifies dialogue simulation and evaluation into an adaptive, iterative process. CoReflect employs a conversation planner that generates structured templates to guide a user simulator through diverse, goal-directed dialogues. Subsequently, a reflective analyzer processes these dialogues to identify systematic behavioral patterns and automatically refine the evaluation rubrics. Crucially, the insights from the conversation analysis are fed back into the planner to update conversation templates for subsequent iterations. This co-evolution loop ensures that the complexity of test cases and the diagnostic precision of rubrics improve in tandem. By minimizing human intervention, CoReflect provides a scalable and self-refining methodology that allows evaluation protocols to adapt alongside the rapidly advancing capabilities of dialogue models.

Figures (4)

• Figure 1: Overview of the CoReflect framework. Initially, evaluation instances are generated from persona-scenario pairs and validated through a consistency check to ensure contextual coherence. Then the core co-evolution loop proceeds through two parts: (i) Conversation simulation, where a planner generates structured conversation templates to guide a user simulator through goal-directed dialogues across diverse persona-scenario pairs; and (ii) Reflective rubric refinement, where an LLM-as-a-judge evaluates simulated interactions and a reflective analyzer extracts insights from clustered behavioral patterns to refine evaluation rubrics. The insights are fed back to the planner to update conversation templates for subsequent iterations.
• Figure 2: Model ratings across Task Completeness and User-Centric Personalization, stratified by conversation length.
• Figure 3: Model ratings across iterations. Iterative reflection improves model differentiation, yielding more distinct stratification and relative ordering.
• Figure 4: Human evaluation form used to assess the quality of simulated user behavior.