Contrastive Active Inference
Pietro Mazzaglia, Tim Verbelen, Bart Dhoedt
TL;DR
This paper presents Contrastive Active Inference, a self-supervised approach that replaces pixel-wise reconstructions with mutual-information-based contrastive losses to learn latent state representations and plan actions in high-dimensional, image-based environments. By defining the contrastive free energy of the past $\mathcal{F}_{\text{NCE}}$ and the contrastive free energy of the future $\mathcal{G}_{\text{NCE}}$, the method avoids expensive reconstructions while promoting alignment with preferred outcomes and discriminability from non-goal observations. The world and behavior models are trained with contrastive losses and amortized planning via a TD-like estimator, and the approach is shown to be computationally more efficient than likelihood-based AIF while delivering competitive or superior performance to reward-based RL on MiniGrid and Reacher benchmarks, with notable robustness to distractors and background variation. These results suggest that contrastive objectives can make active inference more scalable and practical for real-world robotics and high-dimensional control problems, bridging the gap between AIF and state-of-the-art model-based RL.
Abstract
Active inference is a unifying theory for perception and action resting upon the idea that the brain maintains an internal model of the world by minimizing free energy. From a behavioral perspective, active inference agents can be seen as self-evidencing beings that act to fulfill their optimistic predictions, namely preferred outcomes or goals. In contrast, reinforcement learning requires human-designed rewards to accomplish any desired outcome. Although active inference could provide a more natural self-supervised objective for control, its applicability has been limited because of the shortcomings in scaling the approach to complex environments. In this work, we propose a contrastive objective for active inference that strongly reduces the computational burden in learning the agent's generative model and planning future actions. Our method performs notably better than likelihood-based active inference in image-based tasks, while also being computationally cheaper and easier to train. We compare to reinforcement learning agents that have access to human-designed reward functions, showing that our approach closely matches their performance. Finally, we also show that contrastive methods perform significantly better in the case of distractors in the environment and that our method is able to generalize goals to variations in the background. Website and code: https://contrastive-aif.github.io/