Enabling Conversational Behavior Reasoning Capabilities in Full-Duplex Speech

TL;DR

The paper reframes full-duplex spoken dialogue as a perception–reasoning–generation problem and introduces a Graph-of-Thoughts (GoT) framework that models the causal chain from high-level conversational intents to low-level speech acts. It combines a hierarchical conversational behavior detector with GoT-based reasoning to produce interpretable rationales in real time, trained on a hybrid synthetic-real corpus. Empirical results show robust behavior detection, plausible rationale generation, and transferable reasoning from simulated to real speech, with ablations highlighting the benefits of multimodal fusion and constrained causality. This work advances interpretable, reasoning-driven duplex dialogue systems and provides benchmarks for evaluating conversational reasoning in real-time speech.

Abstract

Human conversation is organized by an implicit chain of thoughts that manifests as timed speech acts. Capturing this causal pathway is key to building natural full-duplex interactive systems. We introduce a framework that enables reasoning over conversational behaviors by modeling this process as causal inference within a Graph-of-Thoughts (GoT). Our approach formalizes the intent-to-action pathway with a hierarchical labeling scheme, predicting high-level communicative intents and low-level speech acts to learn their causal and temporal dependencies. To train this system, we develop a hybrid corpus that pairs controllable, event-rich simulations with human-annotated rationales and real conversational speech. The GoT framework structures streaming predictions as an evolving graph, enabling a multimodal transformer to forecast the next speech act, generate concise justifications for its decisions, and dynamically refine its reasoning. Experiments on both synthetic and real duplex dialogues show that the framework delivers robust behavior detection, produces interpretable reasoning chains, and establishes a foundation for benchmarking conversational reasoning in full duplex spoken dialogue systems.

Figures (7)

• Figure 1: Comparison of dialogue paradigms. (Left) Traditional duplex systems frame conversation as a direct sequence prediction task. (Right) We propose a framework based on next-behavior perception and reasoning, where an agent perceives the speaker's act, reasons using a Graph-of-Thoughts, and then generates a response.
• Figure 2: Conversational Behavior Detection and Reasoning Framework. When an audio clip is fed into the Conversational Detection System, the model segments the entire clip into 1-second chunks. For example, for the chunk shown in the figure, the Conversational Detection System labels the high-level behavior as Directive and the low-level speech act as Continuation. These two nodes, together with the context nodes extracted via OpenIE, constitute the primary GoT. In the Graph-of-Thought (GoT) Behavior-Reasoning System, the primary GoT, the transcript, and the raw audio are processed by separate encoders and then combined via gated fusion, producing the rationale text for this chunk as well as an updated GoT graph.
• Figure 3: Window size $W$$\times$ look-ahead $L$ ablations under strict-causal streaming. Columns vary modality (A = Audio; A+T = Audio+Text; A+T+GAT = Audio+Text+Graph), and rows report BLEU-1, ROUGE-1, ROUGE-L, and cosine similarity (SIM). Curves show mean scores and shaded bands indicate 95% confidence intervals across seeds.
• Figure 4: An example primary GoT representation for a one-second audio segment (Appendix \ref{['gotdataproc']}). The left panel shows the adjacency matrix (zeros initially), and the right panel the corresponding visualized representation of encoder's attention weight. Nodes represent context and speech-act concepts.
• Figure 5: An example of rationale text ground truth and predicted rationale generated by our GoT model.For overall accuracy statistics, see Sec \ref{['gotstatistics']}