DialDefer: A Framework for Detecting and Mitigating LLM Dialogic Deference

TL;DR

DialDefer reveals that LLMs shift judgments when identical content is framed as a speaker’s statement rather than a standalone assertion. The framework contrasts factual inquiries with conversational judgments and introduces the Dialogic Deference Score (DDS) to quantify directional shifts that accuracy alone obscures, with DDS values reaching up to $|DDS| = 87$pp across nine domains and four models. Findings show domain-dependent deference or skepticism, with effects amplified in naturalistic r/AIO conversations, and identify human-vs-LLM attribution as a key driver of the shifts. Mitigation via prompting and supervised fine-tuning can reduce deference but often risks over-correction into skepticism, indicating calibration remains a central challenge for deploying LLMs as third-party judges. The work highlights critical ethical and practical implications for alignment and arbitration, guiding future research toward domain-robust calibration and safer deployment of conversational judgment systems.

Abstract

LLMs are increasingly used as third-party judges, yet their reliability when evaluating speakers in dialogue remains poorly understood. We show that LLMs judge identical claims differently depending on framing: the same content elicits different verdicts when presented as a statement to verify ("Is this statement correct?") versus attributed to a speaker ("Is this speaker correct?"). We call this dialogic deference and introduce DialDefer, a framework for detecting and mitigating these framing-induced judgment shifts. Our Dialogic Deference Score (DDS) captures directional shifts that aggregate accuracy obscures. Across nine domains, 3k+ instances, and four models, conversational framing induces large shifts (|DDS| up to 87pp, p < .0001) while accuracy remains stable (<2pp), with effects amplifying 2-4x on naturalistic Reddit conversations. Models can shift toward agreement (deference) or disagreement (skepticism) depending on domain -- the same model ranges from DDS = -53 on graduate-level science to +58 on social judgment. Ablations reveal that human-vs-LLM attribution drives the largest shifts (17.7pp swing), suggesting models treat disagreement with humans as more costly than with AI. Mitigation attempts reduce deference but can over-correct into skepticism, framing this as a calibration problem beyond accuracy optimization.

Figures (7)

• Figure 1: Task framing flips judgment on identical content. Under factual inquiry ($C_1$; left), GPT-4o-mini correctly rejects the claim. Under conversational judgment ($C_2$; right), the model endorses the same claim attributed to a speaker. This directional shift toward agreement is dialogic deference.
• Figure 2: The DialDefer framework transforms datasets into paired experimental conditions. (a) Unified Benchmark: Question--answer pairs become Factual Inquiry prompts ($C_1$: "Is this statement correct?") vs. Conversational Judgment prompts ($C_2$: "Is Speaker 2 correct?"). (b) r/AIO: Reddit conversations are neutralized (first-person pronouns replaced with speaker labels). Speaker 1 is the other party in the conversation; Speaker 2 is the original poster (OP). We construct a judgment about Speaker 1 ("Speaker 1 is [NOT] overreacting") and test whether attributing it to Speaker 2 changes model evaluation. Ground truth is derived from community consensus (inverted, since the community judges the OP). Right: Paired conditions share identical content: $C_{1}^{\mathrm{T}} \leftrightarrow C_{2}^{\mathrm{C}}$ (correct content) and $C_{1}^{\mathrm{F}} \leftrightarrow C_{2}^{\mathrm{I}}$ (incorrect content). The Dialogic Deference Score captures framing-induced judgment shift: $\mathrm{DDS} = \Delta_{\mathrm{Correct}} - \Delta_{\mathrm{Incorrect}}$, where DDS${}>0$ indicates deference and DDS${}<0$ indicates skepticism.
• Figure 3: Average accuracy masks directional judgment shifts. (a) Aggregate accuracy changes $<$2pp. (b) Conversational framing increases agreement with speakers. This improves accuracy on correct speakers ($\Delta$Correct$\uparrow$), but when speakers are incorrect, agreeing with them is the wrong answer, so accuracy drops ($\Delta$Incorrect$\downarrow$). These opposite shifts cancel in the average. (c) DDS captures this asymmetry; $^*p < .0001$.
• Figure 4: DDS varies across models and domains. Rows sorted by mean DDS. See text for interpretation.
• Figure 5: Cross-model DDS distribution by domain. Dashed line indicates neutral (DDS=0).

Theorems & Definitions (1)

• Definition 1: Dialogic Deference Score (DDS)