SocraSynth: Multi-LLM Reasoning with Conditional Statistics
Edward Y. Chang
TL;DR
SocraSynth addresses biases, hallucinations, and insufficient reasoning in LLMs by deploying a two-stage, multi-agent framework that uses conditional statistics and Socratic reasoning. A human moderator guides two opposing LLM agents during a knowledge-generating generative stage and a reasoning-evaluating CRIT-based stage, with a tunable contentiousness parameter to shape debate dynamics. The approach is validated across three experiments, showing that debate-driven information quality surpasses QA and enabling more thorough medical diagnostics and policy deliberation, while mitigating biases through iterative context refinement and diverse evaluations. The work presents four core innovations—conditional statistics, contentiousness modulation, context refinement, and reasonableness evaluation—and outlines future directions toward integrating higher-order logic for validated decision-support in interdisciplinary domains.
Abstract
Large language models (LLMs), while promising, face criticisms for biases, hallucinations, and a lack of reasoning capability. This paper introduces SocraSynth, a multi-LLM agent reasoning platform developed to mitigate these issues. SocraSynth utilizes conditional statistics and systematic context enhancement through continuous arguments, alongside adjustable debate contentiousness levels. The platform typically involves a human moderator and two LLM agents representing opposing viewpoints on a given subject. SocraSynth operates in two main phases: knowledge generation and reasoning evaluation. In the knowledge generation phase, the moderator defines the debate topic and contentiousness level, prompting the agents to formulate supporting arguments for their respective stances. The reasoning evaluation phase then employs Socratic reasoning and formal logic principles to appraise the quality of the arguments presented. The dialogue concludes with the moderator adjusting the contentiousness from confrontational to collaborative, gathering final, conciliatory remarks to aid in human reasoning and decision-making. Through case studies in three distinct application domains, this paper showcases SocraSynth's effectiveness in fostering rigorous research, dynamic reasoning, comprehensive assessment, and enhanced collaboration. This underscores the value of multi-agent interactions in leveraging LLMs for advanced knowledge extraction and decision-making support.