D-SMART: Enhancing LLM Dialogue Consistency via Dynamic Structured Memory And Reasoning Tree

TL;DR

D-SMART addresses the challenge of maintaining consistency in long, multi-turn dialogues by coupling a Dynamic Structured Memory (DSM), which incrementally builds an OWL-compliant knowledge graph from dialogue, with a Reasoning Tree (RT) that performs explicit, multi-path, tree-based reasoning over this memory. It introduces NLI-based metrics—Consistency Score (CS) and Dialogue Entailment Rate (DER)—to better capture logical coherence than holistic scores. Empirical results on MT-Bench-101 show substantial improvements in both quality and consistency for both proprietary and open-source models, with open-source models benefiting notably (up to +10.1% in GPT Score and +48% in DER). The work highlights a trade-off between computational overhead and reliability, arguing that the structured, verifiable reasoning offered by DSM-RT yields more trustworthy long-horizon dialogue behavior, especially in systems where accuracy and accountability matter.

Abstract

Large Language Models (LLMs) often exhibit factual inconsistencies and logical decay in extended, multi-turn dialogues, a challenge stemming from their reliance on static, pre-trained knowledge and an inability to reason adaptively over the dialogue history. Prevailing mitigation strategies, such as Retrieval-Augmented Generation (RAG) and agentic working memories, improve information recall but still engage with fundamentally static knowledge sources and follow pre-defined single reasoning path. This hinders their ability to preserve factual and logical consistency of their responses in multi-turn dialogues while the context evolves over time. To address this issue, we propose D-SMART, a model-agnostic framework designed to maintain multi-turn dialogue consistency by enabling LLMs to build and reason over a dynamic, structured representation of the conversational context. This is achieved via two synergistic components: (1) a Dynamic Structured Memory (DSM), which incrementally constructs and maintains an authoritative, OWL-compliant knowledge graph of the conversation; and (2) a Reasoning Tree (RT), which executes inferences as an explicit and traceable multi-step search over the graph. As the popular-used quality score (judged by GPT-4) can overlook logical flaws, we introduce new NLI-based metrics to better measure multi-turn dialogue consistency. Comprehensive experiments on the MT-Bench-101 benchmark show that D-SMART significantly outperforms state-of-the-art baselines, elevating the dialogue consistency score by over 48\% for both proprietary and open-source models, and notably improves the quality score of the latter by up to 10.1\%.

Figures (14)

• Figure 1: A case of LLM exhibiting logical inconsistency in a multi-turn dialogue.
• Figure 2: An overview of the D-SMART framework.
• Figure 3: The reasoning process at turn $t$. The tree expands by executing actions, enriching its local knowledge from the DSM. The LLM evaluates each step, and the final path leads to a grounded answer, which will update the DSM to $\mathcal{G}_t$ for the next turn.
• Figure 4: GPT Score (lines) and Consistency Score (bars) across dialogue turns. Error bands represent the standard error of the mean.
• Figure 5: Quality (GPT Score) is plotted against Consistency (CS), with bubble size representing DER.