Contingency-Aware Exploration in Reinforcement Learning

TL;DR

This work introduces contingency-awareness through an attentive dynamics model (ADM) to identify controllable regions in observations, enabling approximate self-localization in 2D Atari environments. By forming a compact state representation that includes location, high-level context, and cumulative reward, and integrating it with count-based exploration, the approach significantly improves exploration in sparse-reward reinforcement learning. Empirical results across eight Atari games demonstrate robust gains over baselines, with a notable state-of-the-art performance on Montezuma's Revenge without supervision, and extending to PPO in sticky-action settings. This framework offers interpretable, contingency-driven representations that enhance exploration and suggests broad potential for extending to more complex, higher-dimensional environments.

Abstract

This paper investigates whether learning contingency-awareness and controllable aspects of an environment can lead to better exploration in reinforcement learning. To investigate this question, we consider an instantiation of this hypothesis evaluated on the Arcade Learning Element (ALE). In this study, we develop an attentive dynamics model (ADM) that discovers controllable elements of the observations, which are often associated with the location of the character in Atari games. The ADM is trained in a self-supervised fashion to predict the actions taken by the agent. The learned contingency information is used as a part of the state representation for exploration purposes. We demonstrate that combining actor-critic algorithm with count-based exploration using our representation achieves impressive results on a set of notoriously challenging Atari games due to sparse rewards. For example, we report a state-of-the-art score of >11,000 points on Montezuma's Revenge without using expert demonstrations, explicit high-level information (e.g., RAM states), or supervisory data. Our experiments confirm that contingency-awareness is indeed an extremely powerful concept for tackling exploration problems in reinforcement learning and opens up interesting research questions for further investigations.

Figures (10)

• Figure 1: Left: Contingent region in Freeway; an object in a red box denotes what is under the agent's control, whereas the rest is not. Right: A diagram for the proposed ADM architecture.
• Figure 2: Learning curves on several Atari games: A2C+CoEX and A2C. The x-axis represents total environment steps and the y-axis the mean episode reward averaged over 40 recent episodes. The mean curve is obtained by averaging over 3 random seeds, each shown in a light color.
• Figure 3: Performance plot of ADM trained using on-policy samples from the A2C+CoEX agent.
• Figure 4: Curves of ARI score during training of A2C+CoEX, averaged over 100 recent observations.
• Figure 5: The learning curve of PPO+CoEX on several Atari games with sticky actions setup. The x-axis represents the total number of environment steps and the y-axis the mean episode reward averaged over 40 recent episodes. The mean curve is obtained by averaging over 3 random seeds, each shown in a light color.