Aligning Dialogue Agents with Global Feedback via Large Language Model Reward Decomposition

TL;DR

This work tackles the problem of aligning dialogue agents when only a session-level human rating is available. It introduces LLM-GELI, a framework that uses a frozen pretrained LLM to decompose a global reward $R_{GE}(\tau)$ into per-turn implicit rewards $r_{LI}(s_t,a_t)$, which are then distilled into a light text-based reward model for RLHF. A multimodal extension, Multimodal-LLM-GELI, augments this with natural language descriptors of listener cues such as facial affect and prosody to ground the decomposition in extralinguistic signals. Evaluations on the CANDOR and SODA datasets show that LLM-GELI and especially Multimodal-LLM-GELI outperform baselines in human and automatic metrics, indicating that LLMs can serve as effective reward decomposers and that multimodal grounding enhances socially grounded alignment without manual reward shaping. These results suggest scalable, interpretable, and low-friction pathways to align long-form dialogue agents with nuanced human preferences in real-world multimodal interactions.

Abstract

We propose a large language model based reward decomposition framework for aligning dialogue agents using only a single session-level feedback signal. We leverage the reasoning capabilities of a frozen, pretrained large language model (LLM) to infer fine-grained local implicit rewards by decomposing global, session-level feedback. Our first text-only variant prompts the LLM to perform reward decomposition using only the dialogue transcript. The second multimodal variant incorporates additional behavioral cues, such as pitch, gaze, and facial affect, expressed as natural language descriptions. These inferred turn-level rewards are distilled into a lightweight reward model, which we utilize for RL-based fine-tuning for dialogue generation. We evaluate both text-only and multimodal variants against state-of-the-art reward decomposition methods and demonstrate notable improvements in human evaluations of conversation quality, suggesting that LLMs are strong reward decomposers that obviate the need for manual reward shaping and granular human feedback.

Figures (7)

• Figure 1: Multimodal LLM-GELI framework. A global reward $R_{\text{GE}}(\tau)$ is decomposed into turn-level pseudo-rewards using an LLM prompted with dialogue and multimodal listener cues. These local rewards supervise a text-only reward model $r_\theta$, distilling crossmodal reasoning into a learnable function for alignment (Sec. \ref{['sec:method']}).
• Figure 2: Comparison of reward decomposition across models on a CANDOR dialogue segment. MM-LLM GELI (purple) assigns contextually appropriate, affect-sensitive local rewards, aligning closely with social-emotional cues, better grounding in multimodal feedback, and stronger alignment with human intuition. See Sec. \ref{['sec:decomposed']}.
• Figure 3: High level schematic of different Reward Modeling and Reward Decomposition Methods. Arrows (→) represent forward passes or assignments (- - -) indicate supervised training targets used to train the reward model ${r_\theta}(\cdot) = \hat{r} \approx r$. (a) Local Explicit Reward Modelling, used in conventional RLHF ouyang2022training, (b) Local Implicit Rewards where signals are derived from proxy rewards in the form of outputs from a trained task-relevant function which takes in multimodal or task-related signals and outputs a score, pang2023leveraging (c) Global Explicit Reward Decomposition, where there is a single trajectory level reward and is decomposed to allocated scores for each step. arjona2019rudderren2021learninggangwani2020learning (d) GELI lee2024global approach where the global explicit rewards and local implicit rewards are used in conjunction. (e) Our proposed LLM-GELI approach where the reward decomposition is offloaded to a pretrained LLM, which interprets the local listener reaction cues.
• Figure 4: Candor Demographics
• Figure 5: Mturk experiment for human evaluation for generated samples