Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Interaction Dynamics as a Reward Signal for LLMs

Sian Gooding, Edward Grefenstette

TL;DR

The paper tackles the limitation of relying on textual content for reward signals in multi-turn LLM interactions by introducing TRACE, a reward framework built on the conversational geometry of dialogue. By embedding dialogues into a semantic space and extracting trajectory-based signals, TRACE defines $R_{TRACE}(c) = \mathcal{M}(\mathbf{S}_{TRACE}(c))$ and demonstrates that dynamics alone can predict user satisfaction with accuracy comparable to transcript-based LLM judgments. A Hybrid TRACE+LLM model yields the best performance, underscoring the complementary nature of interaction dynamics and textual analysis. The work also uncovers higher-order, nonlinear dynamics—such as recovery after early errors and mismatched user-model effort—that govern satisfaction, providing a privacy-preserving diagnostic tool for scalable agent alignment and interaction understanding $.$

Abstract

The alignment of Large Language Models (LLMs) for multi-turn conversations typically relies on reward signals derived from the content of the text. This approach, however, overlooks a rich, complementary source of signal: the dynamics of the interaction itself. This paper introduces TRACE (Trajectory-based Reward for Agent Collaboration Estimation), a novel reward signal derived from the geometric properties of a dialogue's embedding trajectory--a concept we term 'conversational geometry'. Our central finding is that a reward model trained only on these structural signals achieves a pairwise accuracy (68.20%) comparable to a powerful LLM baseline that analyzes the full transcript (70.04%). Furthermore, a hybrid model combining interaction dynamics with textual analysis achieves the highest performance (80.17%), demonstrating their complementary nature. This work provides strong evidence that for interactive settings, how an agent communicates is as powerful a predictor of success as what it says, offering a new, privacy-preserving framework that not only aligns agents but also serves as a diagnostic tool for understanding the distinct interaction patterns that drive successful collaboration.

Interaction Dynamics as a Reward Signal for LLMs

TL;DR

The paper tackles the limitation of relying on textual content for reward signals in multi-turn LLM interactions by introducing TRACE, a reward framework built on the conversational geometry of dialogue. By embedding dialogues into a semantic space and extracting trajectory-based signals, TRACE defines and demonstrates that dynamics alone can predict user satisfaction with accuracy comparable to transcript-based LLM judgments. A Hybrid TRACE+LLM model yields the best performance, underscoring the complementary nature of interaction dynamics and textual analysis. The work also uncovers higher-order, nonlinear dynamics—such as recovery after early errors and mismatched user-model effort—that govern satisfaction, providing a privacy-preserving diagnostic tool for scalable agent alignment and interaction understanding

Abstract

The alignment of Large Language Models (LLMs) for multi-turn conversations typically relies on reward signals derived from the content of the text. This approach, however, overlooks a rich, complementary source of signal: the dynamics of the interaction itself. This paper introduces TRACE (Trajectory-based Reward for Agent Collaboration Estimation), a novel reward signal derived from the geometric properties of a dialogue's embedding trajectory--a concept we term 'conversational geometry'. Our central finding is that a reward model trained only on these structural signals achieves a pairwise accuracy (68.20%) comparable to a powerful LLM baseline that analyzes the full transcript (70.04%). Furthermore, a hybrid model combining interaction dynamics with textual analysis achieves the highest performance (80.17%), demonstrating their complementary nature. This work provides strong evidence that for interactive settings, how an agent communicates is as powerful a predictor of success as what it says, offering a new, privacy-preserving framework that not only aligns agents but also serves as a diagnostic tool for understanding the distinct interaction patterns that drive successful collaboration.

Paper Structure

This paper contains 31 sections, 1 equation, 4 figures, 9 tables.

Table of Contents

  1. Introduction: The Limits of Explicit Feedback for Collaborative Agents
  2. Experimental Setup
  3. Dataset
  4. Modelling and Evaluation Methodology
  5. Validation of TRACE as a Reward Signal.
  6. Analysis of the TRACE Signal.
  7. Validating TRACE as a Reward Signal
  8. Formalising the Objective Function.
  9. Experimental Setup: Pairwise Preference Evaluation.
  10. Results and Discussion.
  11. Analysis of the TRACE Reward Signal
  12. Statistical Analysis of Interaction Dynamics
  13. Uncovering Higher-Order Dynamics in Conversational Quality
  14. Expectation Violation and The Value of Recovery.
  15. The Compounding Effect of Mismatched Effort.
  16. ...and 16 more sections

Figures (4)

  • Figure 1: Visualising a conversation as a trajectory through high-dimensional semantic space, shown here as a conceptual 2D projection. User (U) and Model (M) turns create a path relative to an Initial Goal. Our TRACE method is built on the principle that the geometric properties of this path---such as its deviation from the goal (Goal Drift), its local instability (Volatility), or its abrupt turn-by-turn topic changes (Semantic Shift)---are powerful, content-agnostic signals of interaction quality.
  • Figure 2: Anatomy of a Mixed-Satisfaction Conversation. The dialogue is presented in the upper panel, with numbered markers corresponding to the signals in the lower panel.
  • Figure 3: Interaction effects revealing principles of user psychology and collaboration. (a) A non-linear relationship governed by user expectations, where a stable recovery from a poor start is rated higher than a good start followed by volatility. (b) The most powerful interaction found, showing that user satisfaction drops most when a user's consistent effort is met with a degrading model.
  • Figure 4: Interaction effects revealing principles of conversational structure. (a) A breakdown in both temporal rhythm and semantic stability is highly predictive of dissatisfaction. (b) Semantic Cohesion acts as a context-dependent amplifier of the dialogue's quality trend.