Controlling Multimodal Conversational Agents with Coverage-Enhanced Latent Actions

TL;DR

This work tackles the challenge of RL fine-tuning for vision-language multimodal conversational agents by introducing a compact latent action space. It constructs a codebook of latent actions learned from observation, leveraging both paired image-text data and abundant text-only data through a cross-modal projector trained with cycle-consistency, thereby increasing coverage and reducing unimodal bias. The latent-action RL framework reduces the exploration space from $|\,\mathcal{V}\,|$ to $|\mathcal{C}|=128$ and shows robust improvements across two downstream tasks and multiple RL algorithms, with enhanced rollout diversity and notable gains on fine-grained conversational attributes. Although it incurs modest computational overhead, the approach provides scalable, generalizable improvements for RL-tuned MCAs and highlights practical trade-offs for deployment.

Abstract

Vision-language models are increasingly employed as multimodal conversational agents (MCAs) for diverse conversational tasks. Recently, reinforcement learning (RL) has been widely explored for adapting MCAs to various human-AI interaction scenarios. Despite showing great enhancement in generalization performance, fine-tuning MCAs via RL still faces challenges in handling the extremely large text token space. To address this, we learn a compact latent action space for RL fine-tuning instead. Specifically, we adopt the learning from observation mechanism to construct the codebook for the latent action space, where future observations are leveraged to estimate current latent actions that could further be used to reconstruct future observations. However, the scarcity of paired image-text data hinders learning a codebook with sufficient coverage. Thus, we leverage both paired image-text data and text-only data to construct the latent action space, using a cross-modal projector for transforming text embeddings into image-text embeddings. We initialize the cross-modal projector on paired image-text data, and further train it on massive text-only data with a novel cycle consistency loss to enhance its robustness. We show that our latent action based method outperforms competitive baselines on two conversation tasks across various RL algorithms.

Figures (7)

• Figure 1: Illustrations of integrating latent actions with vision-language models.
• Figure 2: Pipeline for constructing the latent action space. (a) Inverse dynamics learning: Given future observations, the inverse dynamics model infers a discrete latent action from a learnable codebook; the language world model then uses this latent action and current observations to reconstruct the next token $x^{T_{t+1}}$. The language world model, inverse dynamics model, and codebook are jointly trained. (b) Policy behavior cloning: A policy model is trained to predict the same latent actions as those inferred by the inverse dynamics model, using only current observations.
• Figure 3: Illustrations of latent action RL. The language world model is frozen, while the policy model is optimized to select latent actions from the codebook that steer the generated responses toward higher rewards.
• Figure 4: Fine-grained performance comparison on (a) MMRole and (b) PCogAlignBench. Results using latent actions are shown with dashed lines, while results using token-level RL are plotted with solid lines.
• Figure 5: Time cost per step during RL training, including rollout, policy update, and total time.