Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Thermodynamics of Reinforcement Learning Curricula

Jacob Adamczyk, Juan Sebastian Rojas, Rahul V. Kulkarni

Abstract

Connections between statistical mechanics and machine learning have repeatedly proven fruitful, providing insight into optimization, generalization, and representation learning. In this work, we follow this tradition by leveraging results from non-equilibrium thermodynamics to formalize curriculum learning in reinforcement learning (RL). In particular, we propose a geometric framework for RL by interpreting reward parameters as coordinates on a task manifold. We show that, by minimizing the excess thermodynamic work, optimal curricula correspond to geodesics in this task space. As an application of this framework, we provide an algorithm, "MEW" (Minimum Excess Work), to derive a principled schedule for temperature annealing in maximum-entropy RL.

Thermodynamics of Reinforcement Learning Curricula

Abstract

Connections between statistical mechanics and machine learning have repeatedly proven fruitful, providing insight into optimization, generalization, and representation learning. In this work, we follow this tradition by leveraging results from non-equilibrium thermodynamics to formalize curriculum learning in reinforcement learning (RL). In particular, we propose a geometric framework for RL by interpreting reward parameters as coordinates on a task manifold. We show that, by minimizing the excess thermodynamic work, optimal curricula correspond to geodesics in this task space. As an application of this framework, we provide an algorithm, "MEW" (Minimum Excess Work), to derive a principled schedule for temperature annealing in maximum-entropy RL.
Paper Structure (13 sections, 11 equations, 4 figures, 1 algorithm)

This paper contains 13 sections, 11 equations, 4 figures, 1 algorithm.

Table of Contents

  1. Introduction
  2. Background
  3. Statistical Mechanics
  4. Maximum-Entropy Reinforcement Learning
  5. A Thermodynamic Framing of Curriculum Learning
  6. Case Study: Linear Reward Parameterizations
  7. Temperature Annealing
  8. Discussion
  9. Future Work
  10. Thermodynamic Correspondence
  11. Experiments
  12. Linear Features Experiment
  13. Related Work

Figures (4)

  • Figure 1: Visualization of the 7$\times$7 Grid World task (inset, left), regret under the linear and geodesic paths (left), and the resulting thermodynamic manifold with the optimal geodesic protocol, which navigates around the phase transition at $\lambda_1=\lambda_2$ (right). Further details can be found in Appendix \ref{['app:experiments']}.
  • Figure 2: (Humanoid-v5) Proposed method (MEW), SAC's automatic temperature adjustment based on minimum entropy constraint, and two constant temperatures. Base algorithm is ASAC adamczyk2025average.
  • Figure 3: Comparison of different "thermodynamic speeds" (speed of traversal along the path from $\alpha=0.2\to0$). Across three orders of magnitude, we obtain similar qualitative performance. The constant temperature, i.e. zero speed, $\alpha=0.2$ is included for comparison. The log-log plot of temperatures indicates that a non-trivial protocol is learned. (If variance were constant throughout, a power law, i.e., a straight line, would result.)
  • Figure 4: Comparison of performance for various "recency thresholds" (cf. $N$ in Algorithm \ref{['alg:mew']}). For an optimal policy, episodes are of length $1,000$; the recency thresholds used to calculate the friction therefore correspond to $1/2$, $5$ or $50$ episodes. The results shown here and in Figure \ref{['fig:speed-comparisons']} demonstrate that the proposed method is robust to the required hyperparameters.

Theorems & Definitions (1)

  • Definition 1