Action and Perception as Divergence Minimization

TL;DR

This work introduces Action Perception Divergence (APD), a unified objective that treats perception and action as joint KL minimization against a shared target distribution. By employing expressive targets (e.g., world models), APD maximizes mutual information between inputs and latent representations while driving exploration through information gain, empowerment, and skill discovery. The framework provides a principled decomposition that recovers and relates many existing objectives—from variational inference to various RL paradigms—under a single mathematical lens. It argues that powerful world models can render task rewards optional, guiding adaptive behavior across large ecological niches. The paper also outlines a practical recipe for deriving new objectives from APD and discusses connections to, and differences from, active inference and control-as-inference, suggesting directions for future empirical investigation.

Abstract

To learn directed behaviors in complex environments, intelligent agents need to optimize objective functions. Various objectives are known for designing artificial agents, including task rewards and intrinsic motivation. However, it is unclear how the known objectives relate to each other, which objectives remain yet to be discovered, and which objectives better describe the behavior of humans. We introduce the Action Perception Divergence (APD), an approach for categorizing the space of possible objective functions for embodied agents. We show a spectrum that reaches from narrow to general objectives. While the narrow objectives correspond to domain-specific rewards as typical in reinforcement learning, the general objectives maximize information with the environment through latent variable models of input sequences. Intuitively, these agents use perception to align their beliefs with the world and use actions to align the world with their beliefs. They infer representations that are informative of past inputs, explore future inputs that are informative of their representations, and select actions or skills that maximally influence future inputs. This explains a wide range of unsupervised objectives from a single principle, including representation learning, information gain, empowerment, and skill discovery. Our findings suggest leveraging powerful world models for unsupervised exploration as a path toward highly adaptive agents that seek out large niches in their environments, rendering task rewards optional.

Figures (10)

• Figure 1: Overview of methods connected by the introduced framework of action and perception as divergence minimization. Each latent variable leads to a mutual information term between said variable and the data. The mutual information with past inputs explains representation learning. The mutual information with future inputs explains information gain, empowerment, and skill discovery. By leveraging multiple latent variables for the decision making process, agents can naturally combine multiple of the objectives. This figure shows the methods that drive from the well-established KL divergence and analogous method trees can be derived by choosing different divergence measures.
• Figure 2: Action and perception minimize the joint KL divergence to a unified target distribution that can be interpreted as a learning probabilistic model of the system. Given the target, perception aligns the agent's beliefs with past inputs while actions align future inputs with its beliefs. There are many ways to specify the target, for example as a latent variable model that explains past inputs and predicts future inputs and an optional reward factor that is shown as a filled square.
• Figure 3: Variational Inference
• Figure 4: Amortized Inference
• Figure 5: Future Inputs