An Optimal Policy for Learning Controllable Dynamics by Exploration

TL;DR

This work addresses learning controllable dynamics in unknown environments by exploring over a finite horizon. It proposes a simple, non-stationary policy structure: greedily maximize an information measure $h(i,u,F)$ over a time-varying, parametric control set $U_k(i,r)$ that accounts for restrictive states such as absorbing, transient, and non-backtracking states. By combining Monte Carlo simulation, cross-entropy optimization, and rollout improvements, the approach demonstrates near-optimal exploration across six diverse controllable Markov chains, highlighting the necessity of planning and non-stationarity for finite-horizon learning. The results offer a practical framework for informative exploration with potential applications in robotics, AI, and modeling of exploratory behavior in animals and humans.

Abstract

Controllable Markov chains describe the dynamics of sequential decision making tasks and are the central component in optimal control and reinforcement learning. In this work, we give the general form of an optimal policy for learning controllable dynamics in an unknown environment by exploring over a limited time horizon. This policy is simple to implement and efficient to compute, and allows an agent to ``learn by exploring" as it maximizes its information gain in a greedy fashion by selecting controls from a constraint set that changes over time during exploration. We give a simple parameterization for the set of controls, and present an algorithm for finding an optimal policy. The reason for this policy is due to the existence of certain types of states that restrict control of the dynamics; such as transient states, absorbing states, and non-backtracking states. We show why the occurrence of these states makes a non-stationary policy essential for achieving optimal exploration. Six interesting examples of controllable dynamics are treated in detail. Policy optimality is demonstrated using counting arguments, comparing with suboptimal policies, and by making use of a sequential improvement property from dynamic programming.

Figures (12)

• Figure 1: A simple environment given by a $5\times 5$ maze with three trapping states (red squares). An agent can explore this environment to learn the maze but will likely encounter a trapping state and become trapped along the way. When this happens the agent can no longer continue to explore.
• Figure 2: Optimal policies for Example \ref{['e1']}. (Left Panel) Applying the parametric policy to the CMC in Example \ref{['e1']} for twenty time periods leads to the states (line) and controls (bars) shown for each time period. For example, during time period 1 the CMC is in state 1 and control 2 has been selected by the policy. (Center Panel) States and controls for the rollout policy. (Right Panel) The corresponding decrease in missing information at each time period is shown for each policy.
• Figure 3: Suboptimal policies for Example \ref{['e1']}. The decrease in missing information at each time period is shown for the uniform random policy over unrestricted controls (Random), and the greedy policy over unrestricted controls (Greedy U). The rollout policy with a Greedy U base policy is shown for comparison (Rollout).
• Figure 4: Policies for Example \ref{['e1']} with stochastic dynamics ($p=0.1$). The decrease in the Monte Carlo averaged missing information is shown for parametric and rollout policies.
• Figure 5: Optimal policies for Example \ref{['e2']}. (Left Panel) Parametric policy applied to the CMC in Example \ref{['e2']}. (Center Panel) Rollout policy. (Right Panel) Decrease in missing information for each policy.

Theorems & Definitions (1)

• Claim 1