Reinforced Natural Language Interfaces via Entropy Decomposition

TL;DR

Reinforced Natural Language Interfaces via Entropy Decomposition tackles rapid adaptation of conversational agents to unseen tasks by splitting language uncertainty into a structural term $H(M|T)$ and a functional term $MI(M;A|T)$. The approach learns task-aware, succinct language through entropy minimization (with occasional human supervision) and mutual-information maximization within a reinforcement-learning framework, implemented with a three-module agent architecture (language encoder, language decoder, and policy). Experiments in a referential game and a 3D navigation task show that entropy-guided human input accelerates natural-language acquisition, while MI optimization improves the communicative utility and discriminative power of the language, particularly when combined with task rewards. The work offers a theoretically grounded, practical pathway toward interpretable, human-friendly spoken dialogue interfaces capable of coordinating complex, temporally extended tasks.

Abstract

In this paper, we study the technical problem of developing conversational agents that can quickly adapt to unseen tasks, learn task-specific communication tactics, and help listeners finish complex, temporally extended tasks. We find that the uncertainty of language learning can be decomposed to an entropy term and a mutual information term, corresponding to the structural and functional aspect of language, respectively. Combined with reinforcement learning, our method automatically requests human samples for training when adapting to new tasks and learns communication protocols that are succinct and helpful for task completion. Human and simulation test results on a referential game and a 3D navigation game prove the effectiveness of the proposed method.

Figures (6)

• Figure 1: Learning framework of our method.
• Figure 2: Learned language for the referential game, after supervised training using different numbers of human samples, and after optimizing the mutual information objective (Eq. \ref{['equ:functional_gradient']}).
• Figure 3: (a) Requesting human samples according to the value of entropy learns more natural language than random requests. (b) The influence of different entropy thresholds on learning performance. (c) Optimizing the mutual information loss accelerates learning.
• Figure 4: Up: the success rate of the referential game when we replace the listener with human. Bottom: BLEU scores of the learned language encoder. Results under different entropy thresholds are shown.
• Figure 5: Mutual information between each word and the listener's action selection in the referential game.