Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Controllable Information Production

Tristan Shah, Stas Tiomkin

TL;DR

This work introduces Controllable Information Production (CIP), a principled intrinsic motivation derived from Optimal Control that measures the gap between open-loop and closed-loop Kolmogorov–Sinai entropies to identify controllable chaos. By decomposing the optimal policy into extrinsic and intrinsic components and connecting the value Hessian to open- and closed-loop entropy via auxiliary Riccati-based recursions, CIP provides a representation-free objective that encourages exploration of environments rich in chaotic dynamics while remaining controllable. The authors show CIP is nonnegative and emergent from OC, and demonstrate its effectiveness through a finite-horizon MPC implementation (iCEM) across pendulum benchmarks, where agents seek edge-of-chaos states and stabilize under control. The framework offers a theoretically grounded alternative to mutual-information IM objectives and suggests a path toward integrating intrinsic and extrinsic motivations within dynamical control systems. Practical impact lies in principled IM signals that can drive autonomous systems toward informative, controllable regimes without task-specific rewards.

Abstract

Intrinsic Motivation (IM) is a paradigm for generating intelligent behavior without external utilities. The existing information-theoretic methods for IM are predominantly based on information transmission, which explicitly depends on the designer's choice of which random variables engage in transmission. In this work, we introduce a novel IM principle, Controllable Information Production (CIP), that avoids both external utilities and designer-specified variables. We derive the CIP objective from Optimal Control, showing a connection between extrinsic and intrinsic behaviors. CIP appears as the gap between open-loop and closed-loop Kolmogorov-Sinai entropies, which simultaneously rewards the pursuit and regulation of chaos. We establish key theoretical properties of CIP and demonstrate its effectiveness on standard IM benchmarks.

Controllable Information Production

TL;DR

This work introduces Controllable Information Production (CIP), a principled intrinsic motivation derived from Optimal Control that measures the gap between open-loop and closed-loop Kolmogorov–Sinai entropies to identify controllable chaos. By decomposing the optimal policy into extrinsic and intrinsic components and connecting the value Hessian to open- and closed-loop entropy via auxiliary Riccati-based recursions, CIP provides a representation-free objective that encourages exploration of environments rich in chaotic dynamics while remaining controllable. The authors show CIP is nonnegative and emergent from OC, and demonstrate its effectiveness through a finite-horizon MPC implementation (iCEM) across pendulum benchmarks, where agents seek edge-of-chaos states and stabilize under control. The framework offers a theoretically grounded alternative to mutual-information IM objectives and suggests a path toward integrating intrinsic and extrinsic motivations within dynamical control systems. Practical impact lies in principled IM signals that can drive autonomous systems toward informative, controllable regimes without task-specific rewards.

Abstract

Intrinsic Motivation (IM) is a paradigm for generating intelligent behavior without external utilities. The existing information-theoretic methods for IM are predominantly based on information transmission, which explicitly depends on the designer's choice of which random variables engage in transmission. In this work, we introduce a novel IM principle, Controllable Information Production (CIP), that avoids both external utilities and designer-specified variables. We derive the CIP objective from Optimal Control, showing a connection between extrinsic and intrinsic behaviors. CIP appears as the gap between open-loop and closed-loop Kolmogorov-Sinai entropies, which simultaneously rewards the pursuit and regulation of chaos. We establish key theoretical properties of CIP and demonstrate its effectiveness on standard IM benchmarks.
Paper Structure (28 sections, 5 theorems, 43 equations, 2 figures, 1 algorithm)

This paper contains 28 sections, 5 theorems, 43 equations, 2 figures, 1 algorithm.

Table of Contents

  1. Introduction
  2. Existing Intrinsic Motivation Methods
  3. Overview of Kolmogorov-Sinai Entropy
  4. The CIP Principle
  5. Setting
  6. Policy Decomposition
  7. Value Hessian Decomposition
  8. Recovery of CIP
  9. Experiments
  10. Controller
  11. Behavior in Benchmark Environments
  12. Discussion
  13. Interpretation of CIP
  14. Emergence of CIP from Optimal Control
  15. Information Transmission and Production
  16. ...and 13 more sections

Key Result

Lemma 4.1

The optimal linear feedback policy is decomposed into a goal-directed extrinsic component $d_t$ and a goal-agnostic intrinsic component $\pi_{x_t}$: with the Hessian of the value function satisfying the Discrete Algebraic Riccati Equation (DARE): where $V_{xx_T}^\pi = c_{xx_T}$ is the terminal condition.

Figures (2)

  • Figure 1: Roadmap of the CIP derivation. The optimal policy decomposes into intrinsic (feedback) and extrinsic (drift) components in Lemma \ref{['lemma:policy_decomp']}. Perturbation analysis through the value Hessian separates open-loop and closed-loop entropy production rates in Lemma \ref{['lemma:value_decomp']}, whose difference defines CIP, Theorem \ref{['theorem:controllable_information_production']}.
  • Figure 2: Swing-up, stabilization, and intrinsic control across pendulum-based systems. Values of CIP are plotted for each environment.

Theorems & Definitions (13)

  • Lemma 4.1: Intrinsic-Extrinsic Policy Decomposition
  • proof
  • Definition 4.2: Auxiliary Recursions From the Value Hessian
  • Lemma 4.3: Value Hessian Decomposition
  • proof
  • Corollary 4.4: Closed-Loop Entropy Value Hessian Equivalence
  • proof
  • Theorem 4.5: Controllable Information Production
  • proof
  • proof
  • ...and 3 more