Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Self-evolving expertise in complex non-verifiable subject domains: dialogue as implicit meta-RL

Richard M. Bailey

TL;DR

The paper tackles solving wicked, non-verifiable problems by using dialogue among LLM agents as an implicit meta-learning process. It formalizes a two-time-scale learning framework where inner in-context adaptation operates with fixed model weights while an outer loop updates the dialogue context via memory and reflections. Through the Dialectica system, agents evolve their prompts and beliefs across multiple rounds, guided by memory consolidation, context editing, and optional external information sources; there is no explicit reward. Empirical results across two model families (Qwen3:30b and o4-mini) and three tournaments show that dialogue-driven context evolution yields superior performance on Elo, BTD, and AlphaRank metrics, with qualitative traces showing reflections guiding subsequent statements and coherent trajectories. The work suggests practical routes to targeted expertise amplification in open domains and highlights potential applications for LLM alignment and the production of expert-concept catalogs.

Abstract

So-called `wicked problems', those involving complex multi-dimensional settings, non-verifiable outcomes, heterogeneous impacts and a lack of single objectively correct answers, have plagued humans throughout history. Modern examples include decisions over justice frameworks, solving environmental pollution, planning for pandemic resilience and food security. The use of state-of-the-art artificial intelligence systems (notably Large Language Model-based agents) collaborating with humans on solving such problems is being actively explored. While the abilities of LLMs can be improved by, for example, fine-tuning, hand-crafted system prompts and scaffolding with external tools, LLMs lack endogenous mechanisms to develop expertise through experience in such settings. This work address this gap with Dialectica, a framework where agents engage in structured dialogue on defined topics, augmented by memory, self-reflection, and policy-constrained context editing. Formally, discussion is viewed as an implicit meta-reinforcement learning process. The `dialogue-trained' agents are evaluated post-hoc using judged pairwise comparisons of elicited responses. Across two model architectures (locally run Qwen3:30b and OpenAI's o4-mini) results show that enabling reflection-based context editing during discussion produces agents which dominate their baseline counterparts on Elo scores, normalized Bradley-Terry-Davidson ability, and AlphaRank mass. The predicted signatures of learning are observed qualitatively in statement and reflection logs, where reflections identify weaknesses and reliably shape subsequent statements. Agreement between quantitative and qualitative evidence supports dialogue-driven context evolution as a practical path to targeted expertise amplification in open non-verifiable domains.

Self-evolving expertise in complex non-verifiable subject domains: dialogue as implicit meta-RL

TL;DR

The paper tackles solving wicked, non-verifiable problems by using dialogue among LLM agents as an implicit meta-learning process. It formalizes a two-time-scale learning framework where inner in-context adaptation operates with fixed model weights while an outer loop updates the dialogue context via memory and reflections. Through the Dialectica system, agents evolve their prompts and beliefs across multiple rounds, guided by memory consolidation, context editing, and optional external information sources; there is no explicit reward. Empirical results across two model families (Qwen3:30b and o4-mini) and three tournaments show that dialogue-driven context evolution yields superior performance on Elo, BTD, and AlphaRank metrics, with qualitative traces showing reflections guiding subsequent statements and coherent trajectories. The work suggests practical routes to targeted expertise amplification in open domains and highlights potential applications for LLM alignment and the production of expert-concept catalogs.

Abstract

So-called `wicked problems', those involving complex multi-dimensional settings, non-verifiable outcomes, heterogeneous impacts and a lack of single objectively correct answers, have plagued humans throughout history. Modern examples include decisions over justice frameworks, solving environmental pollution, planning for pandemic resilience and food security. The use of state-of-the-art artificial intelligence systems (notably Large Language Model-based agents) collaborating with humans on solving such problems is being actively explored. While the abilities of LLMs can be improved by, for example, fine-tuning, hand-crafted system prompts and scaffolding with external tools, LLMs lack endogenous mechanisms to develop expertise through experience in such settings. This work address this gap with Dialectica, a framework where agents engage in structured dialogue on defined topics, augmented by memory, self-reflection, and policy-constrained context editing. Formally, discussion is viewed as an implicit meta-reinforcement learning process. The `dialogue-trained' agents are evaluated post-hoc using judged pairwise comparisons of elicited responses. Across two model architectures (locally run Qwen3:30b and OpenAI's o4-mini) results show that enabling reflection-based context editing during discussion produces agents which dominate their baseline counterparts on Elo scores, normalized Bradley-Terry-Davidson ability, and AlphaRank mass. The predicted signatures of learning are observed qualitatively in statement and reflection logs, where reflections identify weaknesses and reliably shape subsequent statements. Agreement between quantitative and qualitative evidence supports dialogue-driven context evolution as a practical path to targeted expertise amplification in open non-verifiable domains.
Paper Structure (46 sections, 19 equations, 5 figures, 9 tables)

This paper contains 46 sections, 19 equations, 5 figures, 9 tables.

Table of Contents

  1. Introduction
  2. This contribution
  3. Dialogue as meta-learning
  4. Key findings
  5. System description
  6. Over-view
  7. Agent basics
  8. Prompt Construction
  9. Memory architecture
  10. Learning Dynamics
  11. Post-round consolidation
  12. Context evolution
  13. End-of-Topic consolidation
  14. External information souces
  15. Facilitator
  16. ...and 31 more sections

Figures (5)

  • Figure 1: In each discussion experiment, a set of Topics are discussed over several Rounds of Orchestrator-guided interactions between multiple Agents. During each round of dialogue, agents use memory and other tools to gather information and have the ability to edit the information used to generate their prompt context prior to their next conversation statement.
  • Figure 2: Tournament results summary.
  • Figure 3: Tournament results analysis.
  • Figure : The main components of the system: the Orchestrator is a deterministic component responsible for calling all actions and holding the list of discussion topics; the agents take turns to deliver statements, which are broadcast to all other agents. Agents have configurable use of tools for working memory, external information collection, storage and retrieval, which they use to update their prompt context over time. The (optional) Facilitator helps to guide the discussion.
  • Figure : The main components of the system: the Orchestrator is a deterministic component responsible for calling all actions and holding the list of discussion topics; the agents take turns to deliver statements, which are broadcast to all other agents. Agents have configurable use of tools for working memory, external information collection, storage and retrieval, which they use to update their prompt context over time. The (optional) Facilitator helps to guide the discussion.