Toward Agents That Reason About Their Computation

TL;DR

The paper introduces Compute DQN, an agent that learns to reason about and control its own compute by augmenting the RL objective with a compute cost and arming the agent with temporally extended actions that adjust observation and action frequencies. With training on the Arcade Learning Environment, Compute DQN attains higher performance than a fixed-rate DQN in 75% of games while cutting compute usage by about 3x under the same budget, and it discovers per-game compute strategies that adapt over time. The work demonstrates that compute-aware control is learnable from experience and can tailor compute to game dynamics, suggesting broad implications for energy-efficient and scalable RL. It also discusses broader connections to cognition-like planning of computations and outlines future directions for expanding compute-aware capabilities in long-lived, adaptive agents.

Abstract

While reinforcement learning agents can achieve superhuman performance in many complex tasks, they typically do not become more computationally efficient as they improve. In contrast, humans gradually require less cognitive effort as they become more proficient at a task. If agents could reason about their compute as they learn, could they similarly reduce their computation footprint? If they could, we could have more energy efficient agents or free up compute cycles for other processes like planning. In this paper, we experiment with showing agents the cost of their computation and giving them the ability to control when they use compute. We conduct our experiments on the Arcade Learning Environment, and our results demonstrate that with the same training compute budget, agents that reason about their compute perform better on 75% of games. Furthermore, these agents use three times less compute on average. We analyze individual games and show where agents gain these efficiencies.

Figures (8)

• Figure 1: Improvement of Compute DQN over DQN in human normalized score (HNS) across 46 Atari games. Compute DQN achieves higher HNS in 75% of games. Amidar shows the largest gain of 487%, while the largest loss is 30% below DQN.
• Figure 2: Compute DQN learns game-specific decision rates across the Arcade suite. Each green dot shows the average decisions per second for a single game, measured over 10 independently trained agents. The dashed green line indicates the suite-wide mean of 3.6 Hz. For comparison, Compute DQN uses $3.4$ times less compute than DQN, which acts at a fixed 12 Hz (red dashed line). These results demonstrate that Compute DQN can adapt its compute use per game while maintaining performance.
• Figure 3: Population dynamics of learning: each point is a moment in time during training, reported across 10 seeds per game. Color encodes training progress.
• Figure 4: The decision rate within a trajectory differs by game. Pong and Asterix are uni modal, whereas Breakout is bimodal.
• Figure 5: Exponentially decayed average of decision rate for Compute DQN on segments of an episode in Pong. Coloured regions indicate when the opponent has just hit the ball, the ball is approaching the player and dotted line indicates when the ball is hit, the player has just hit the ball, and the ball is heading toward the opponent. There are typically rises in decision rate in the red region in anticipation of the ball reaching the player's paddle.