Using Forwards-Backwards Models to Approximate MDP Homomorphisms

TL;DR

This work proposes a novel approach to constructing homomorphisms in discrete action spaces, which uses a learnt model of environment dynamics to infer which state-action pairs lead to the same state -- which can reduce the size of the state-action space by a factor as large as the cardinality of the original action space.

Abstract

Reinforcement learning agents must painstakingly learn through trial and error what sets of state-action pairs are value equivalent -- requiring an often prohibitively large amount of environment experience. MDP homomorphisms have been proposed that reduce the MDP of an environment to an abstract MDP, enabling better sample efficiency. Consequently, impressive improvements have been achieved when a suitable homomorphism can be constructed a priori -- usually by exploiting a practitioner's knowledge of environment symmetries. We propose a novel approach to constructing homomorphisms in discrete action spaces, which uses a learnt model of environment dynamics to infer which state-action pairs lead to the same state -- which can reduce the size of the state-action space by a factor as large as the cardinality of the original action space. In MinAtar, we report an almost 4x improvement over a value-based off-policy baseline in the low sample limit, when averaging over all games and optimizers.

Figures (5)

• Figure 1: We predict equivalent state-action pairs with dynamics models. In Asterix, an agent needs to collect gold. To predict the value of state $S$ and action move up, we first predict forward in time to get $S'$. We then predict backwards assuming the previous action was the alternative move right, to obtain a hypothetical state-action pair for our Q-network ($S_{hyp}$, move right). Icons generated with Dall-e 3 betker2023improving.
• Figure 2: $\mathbf{(a),(b)}$ In a gridworld, equivalent effect abstraction improves sample efficiency. Equivalent effect abstraction improves both model-free Q-learning and model-based approaches Q-planning. 50 seeds are used for Q-learning while 10 seeds are used for Q-planning.
• Figure 3: $\mathbf{(a, b)}$ Equivalent effect abstraction learns more quickly than the DQN baseline. For Cartpole, dynamics models are learned in 3 initial episodes of experience that are shown on the x-axis of the plot. For predator prey, the experience used to train the dynamics models is not included in the plot. PRAE 10 refers to using PRAE with 10 episodes of data to construct an environment model, which it then plans in. For predator prey the equivalent effect abstraction results reuse one set of pre-trained backwards and forwards models for each RL run.
• Figure 4: EEA delivers an almost 4x improvement at 250k frames when averaging over all games and optimizers.
• Figure 5: Full results for all different configurations of equivalent effect abstraction and the other baselines. Equivalent effect abstraction with the Adam optimizer generally performs best in the low sample limit.

Theorems & Definitions (7)

• Definition 2.1
• Definition 2.5
• Definition 2.6
• Theorem 2.7
• proof
• Theorem 2.8
• proof