Measuring Reasoning in LLMs: a New Dialectical Angle

TL;DR

This paper reframes LLM reasoning evaluation from static correctness to dynamic, dialectical reasoning using the SIEV framework. SIEV employs a thesis–antithesis–synthesis pipeline with two agents and yields three complementary metrics: $p_S$ (synthesis performance), Dialectic Score (DS), and $ abla$ (the change from thesis to synthesis, i.e., $ abla = p_S - p_T$) alongside Oppositional Compliance (OC). Applying SIEV to GSM8K and MMLU across 20+ models reveals substantial reasoning gaps that conventional accuracy metrics miss, including large drops for models like GPT-5-chat on GSM and topic-dependent strengths. Cross-model experiments show that introducing diverse antitheses can improve dialectical reasoning, but results are context-dependent, suggesting that genuine reasoning in LLMs may be more about structured argumentative dynamics than general cognitive ability. Overall, dialectical evaluation provides a more discriminative and robust assessment of LLM reasoning and has implications for multi-agent systems and principled evaluation frameworks.

Abstract

What does it truly mean for a language model to "reason"? Most current evaluations and benchmarks reward models' correct standalone answers--but correctness alone reveals little about the process that produced them. In this work, we explore a different perspective: reasoning is not a static chain of steps, but a dynamic trajectory where ideas interact, clash, and evolve into deeper insights. To capture this dynamic, we draw on a well-established philosophical tradition: \textit{dialectics}, where reasoning unfolds through thesis, antithesis, and synthesis. Building on this, we present SIEV, a structured framework that evaluates reasoning of LLMs through dialectics. Unlike conventional evaluations, SIEV assesses not only the conclusion a model reaches, but how it gets there: its ability to resolve tension, integrate distinct ideas, and synthesize higher-order reasoning. This lens uncovers significant reasoning gaps in state-of-the-art models even under saturated benchmarks like GSM and MMLU. For instance, GPT-5-chat, a recent model, loses over 40 points (out of 100) when evaluated with SIEV on GSM. Our findings highlight that adopting a process-oriented, philosophically grounded approach enables a deeper, more rigorous, and more discriminative assessment of LLM reasoning.

Figures (23)

• Figure 2: Overview of SIEV and its TAS pipeline.
• Figure 3: An illustration of dialectical reasoning evaluation in SIEV using two sample LLMs
• Figure 5: Distance correlation analysis of dialectical reasoning in all MMLU sub-tests.
• Figure 8: An illustration of dialectical reasoning evaluation in SIEV using the GPT-5 model and a question from the conceptual physics section of the MMLU benchmark. In this example, the antithesis provided by GPT-5 model correctly identifies the answer, but even after observing this different angle, the model produces a synthesis that closely mirrors the flawed reasoning of the incorrect thesis.
• Figure 10: Detailed results of different models in different MMLU exams. Each circle shows the overall result of a model in a sub-topic in MMLU benchmark.