Dialogue Telemetry: Turn-Level Instrumentation for Autonomous Information Gathering

TL;DR

This work tackles the lack of turn-level instrumentation in schema-grounded information-gathering dialogues by introducing Dialogue Telemetry (DT), a lightweight, model-agnostic framework that outputs two signals after every question–answer turn: a Progress Estimator (PE) and a Stalling Index (SI). PE estimates residual information or expected gain per knowledge category, while SI detects throughput degradation via patterns of repetitive probing and semantic similarity. The authors validate DT in SAR-inspired, LLM-driven simulations, showing DT can distinguish efficient from stalled dialogues and can inform reinforcement learning strategies to avoid costly stalls. The results demonstrate DT as a practical instrumentation layer for online monitoring, diagnosis, and closed-loop policy design in autonomous information gathering.

Abstract

Autonomous systems conducting schema-grounded information-gathering dialogues face an instrumentation gap, lacking turn-level observables for monitoring acquisition efficiency and detecting when questioning becomes unproductive. We introduce Dialogue Telemetry (DT), a measurement framework that produces two model-agnostic signals after each question-answer exchange: (i) a Progress Estimator (PE) quantifying residual information potential per category (with a bits-based variant), and (ii) a Stalling Index (SI) detecting an observable failure signature characterized by repeated category probing with semantically similar, low-marginal-gain responses. SI flags this pattern without requiring causal diagnosis, supporting monitoring in settings where attributing degradation to specific causes may be impractical. We validate DT in controlled search-and-rescue (SAR)-inspired interviews using large language model (LLM)-based simulations, distinguishing efficient from stalled dialogue traces and illustrating downstream utility by integrating DT signals into a reinforcement learning (RL) policy. Across these settings, DT provides interpretable turn-level instrumentation that improves policy performance when stalling carries operational costs.

Figures (6)

• Figure 1: The Dialogue Telemetry (DT) measurement architecture. The system ingests turn-level dialogue exchanges ($q_t, y_t$) and computes two parallel telemetry signals. The Progress Estimator (PE, blue) tracks the knowledge state per category to quantify residual information potential (including a bits-based variant). The Stalling Index (SI, orange) flags revisitation by combining discrete repetition with semantic embedding similarity. These turn-level observables quantify conversational efficiency and detect stalling online, independent of the downstream control policy.
• Figure 2: DT Telemetry Monitoring of Efficient Dialogue. Turn-by-turn tracking of optimal 20-turn $\mathrm{DT}$-guided interrogation. (a) Systematic knowledge evolution achieving high completion rates; (b) Stall Index ($\mathrm{SI}$) remaining below detection threshold throughout; (c) EIG values decreasing systematically as gaps close; (d) Information gain following diminishing returns pattern; (e) Final state showing successful completion aligned with importance weights.
• Figure 3: DT Telemetry Detection of Conversational Stalling. Monitoring of a 20-turn dialogue exhibiting dual stalling incidents. (a) Knowledge evolution showing suboptimal progression; (b) $\mathrm{SI}$ detecting two vortex periods (red shaded) exceeding threshold; (c) EIG values remaining persistently high due to unresolved information gaps; (d) Information gain showing near-zero acquisition during stalling periods; (e) Final state revealing poor completion across categories.
• Figure 4: RL Integration and Ablation results under Conditions A and B. (a) Condition A (standard termination): Full-DT vs Baseline. (b) Condition B (stall-aware termination): Full-DT vs Baseline. (c) Condition A ablation: Full-DT vs DT-without-SI-penalty. (d) Condition B ablation: Full-DT vs DT-without-SI-penalty. Four metrics track agent performance over 50k timesteps: Episode Rewards (higher = better, indicating successful task completion), Total Knowledge (higher = better, measuring information acquisition efficiency), Complete Categories (higher = better, showing number of resolved knowledge categories out of 8 total), and Stall Index/SI (lower = better, indicating reduced conversational repetition). All curves represent Gaussian-smoothed averages across 25 independent training runs with $\pm 1\sigma$ confidence bands.
• Figure 5: DT Telemetry Monitoring of Efficient Dialogue Using Heuristic PE Variant. Turn-by-turn tracking of optimal 20-turn $\mathrm{DT}$-guided interrogation using $\mathrm{PE}^{\text{H}}$ (Eq. \ref{['eq4:heuristic_PE']}). (a) Systematic knowledge evolution achieving high completion rates; (b) Stall Index ($\mathrm{SI}$) remaining below detection threshold throughout; (c) Expected gain (heuristic) values decreasing systematically as gaps close; (d) Information gain following diminishing returns pattern; (e) Final state showing successful completion aligned with importance weights. Compare with Fig. \ref{['fig2:results_4-2-efficient']} which uses the formal entropy-based PE variant.