An Information Theoretic Approach to Interaction-Grounded Learning
Xiaoyan Hu, Farzan Farnia, Ho-fung Leung
TL;DR
This work tackles reinforcement learning where the agent must infer latent binary rewards from noisy feedback by enforcing conditional independence of the feedback from context-action given the latent reward. It introduces Variational Information-based IGL (VI-IGL), which optimizes a min-max objective that minimizes $I(Y;X,A|R_\psi)$ while regularizing with $\beta\,I(X,A;R_\psi)$, and solves it via a variational representation of mutual information. The framework extends naturally to general $f$-information (f-VI-IGL), enabling flexible divergence choices and robust estimation using function-approximation for $G$ and $T$. Empirical results on a number-guessing task with noisy feedback show VI-IGL improves robustness and performance over prior IGL methods, highlighting its potential for efficient, information-driven learning under uncertain rewards.
Abstract
Reinforcement learning (RL) problems where the learner attempts to infer an unobserved reward from some feedback variables have been studied in several recent papers. The setting of Interaction-Grounded Learning (IGL) is an example of such feedback-based RL tasks where the learner optimizes the return by inferring latent binary rewards from the interaction with the environment. In the IGL setting, a relevant assumption used in the RL literature is that the feedback variable $Y$ is conditionally independent of the context-action $(X,A)$ given the latent reward $R$. In this work, we propose Variational Information-based IGL (VI-IGL) as an information-theoretic method to enforce the conditional independence assumption in the IGL-based RL problem. The VI-IGL framework learns a reward decoder using an information-based objective based on the conditional mutual information (MI) between $(X,A)$ and $Y$. To estimate and optimize the information-based terms for the continuous random variables in the RL problem, VI-IGL leverages the variational representation of mutual information to obtain a min-max optimization problem. Also, we extend the VI-IGL framework to general $f$-Information measures leading to the generalized $f$-VI-IGL framework for the IGL-based RL problems. We present numerical results on several reinforcement learning settings indicating an improved performance compared to the existing IGL-based RL algorithm.