Conversation for Non-verifiable Learning: Self-Evolving LLMs through Meta-Evaluation

TL;DR

CoNL (Conversation for Non-verifiable Learning) addresses training LLMs on non-verifiable tasks by coupling generation with meta-evaluation through multi-agent self-play. It introduces a diagnostic reward that rewards critiques which enable others to improve, using Bradley-Terry aggregation of pairwise comparisons to yield two timepoint quality scores $V^{\text{init}}$ and $V^{\text{final}}$. The approach blends generation, evaluation, and meta-evaluation into a single trainable policy, achieving consistent improvements across five benchmarks and approaching RL performance with ground-truth rewards, all without external labels. This framework supports scalable improvement in domains like creative writing, dialogue, and ethical reasoning, though it relies on peer consensus and thus warrants safeguards in high-stakes settings.

Abstract

Training large language models (LLMs) for non-verifiable tasks, such as creative writing, dialogue, and ethical reasoning, remains challenging due to the absence of ground-truth labels. While LLM-as-Judge approaches offer a scalable alternative to human feedback, they face a fundamental limitation: performance is constrained by the evaluator's own quality. If the judge cannot recognize good solutions, it cannot provide useful training signals, and evaluation biases (e.g., favoring verbosity over quality) remain unaddressed. This motivates meta-evaluation: the ability to evaluate and improve the evaluator itself. We introduce CoNL, a framework that unifies generation, evaluation, and meta-evaluation through multi-agent self-play. Our key insight: critique quality can be measured by whether it helps others improve their solutions. In CoNL, multiple agents sharing the same policy engage in structured conversations to propose, critique, and revise solutions. Critiques that enable solution improvements earn a diagnostic reward, creating explicit supervision for meta-evaluation and enabling joint optimization of generation and judging capabilities through self-play, without external judges or ground truth. Experiments on five benchmarks show that CoNL achieves consistent improvements over self-rewarding baselines while maintaining stable training.

Figures (3)

• Figure 1: Multi-Agent Conversational Paradigm with Meta-Evaluation. Agents engage in iterative rounds where they generate solutions ($A$) and evaluate previous turns ($E$). Moving beyond standard LLM-as-Judge, the process incorporates Meta-Evaluation to scrutinize both the generation and the judging quality through peer consensus.
• Figure 2: Overview of CoNL training protocol. Given a query, $N$ agents with diverse personas engage in a four-round conversation: (Round 0) each agent proposes an initial solution; (Round 1) agents provide initial rankings and critique peers; (Round 2) agents revise solutions based on received critiques; (Round 3) agents provide final rankings. We aggregate rankings via Bradley-Terry model to compute quality scores $V^{\text{init}}$ and $V^{\text{final}}$. Training rewards are computed from conversation dynamics: solution quality rewards high-scoring solutions, diagnostic rewards critiques that enable improvements ($V^{\text{final}} > V^{\text{init}}$), and consensus rewards rankings aligned with group majority.
• Figure 3: Training Dynamics: CoNL vs Self-Rewarding Training. We compare training dynamics on DeepMath across 10k training steps. Left: CoNL maintains stable entropy while SRT shows erratic fluctuations. Middle: CoNL produces consistent solution lengths while SRT exhibits high variance. Right: CoNL shows steady accuracy improvement matching ground-truth RL, while SRT's majority-voting signals lead to unstable convergence. Overall, CoNL's conversation-derived rewards provide more stable training than self-rewarding baselines.