Preliminary Tests of the Anticipatory Classifier System with Hindsight Experience Replay

TL;DR

The paper addresses sparse-reward learning in Anticipatory Learning Classifier Systems by integrating Hindsight Experience Replay to ACS2, creating ACS2HER. It introduces a Replay Memory, hindsight factor $k$, learning repeats $m$, and a goal-selection strategy $S$, triggering hindsight updates on failure to densify learning signals. Across deterministic Maze 6 and stochastic FrozenLake benchmarks, ACS2ER and ACS2HER accelerate knowledge acquisition and expand classifier populations, but at substantial computational cost and with mixed effects on goal success in stochastic environments. This work provides the first analysis of combining anticipatory mechanisms with retrospective goal relabeling in Learning Classifier Systems, highlighting key trade-offs and guiding future optimizations for scalable, efficient learning in sparse-reward domains.

Abstract

This paper introduces ACS2HER, a novel integration of the Anticipatory Classifier System (ACS2) with the Hindsight Experience Replay (HER) mechanism. While ACS2 is highly effective at building cognitive maps through latent learning, its performance often stagnates in environments characterized by sparse rewards. We propose a specific architectural variant that triggers hindsight learning when the agent fails to reach its primary goal, re-labeling visited states as virtual goals to densify the learning signal. The proposed model was evaluated on two benchmarks: the deterministic \texttt{Maze 6} and the stochastic \texttt{FrozenLake}. The results demonstrate that ACS2HER significantly accelerates knowledge acquisition and environmental mastery compared to the standard ACS2. However, this efficiency gain is accompanied by increased computational overhead and a substantial expansion in classifier numerosity. This work provides the first analysis of combining anticipatory mechanisms with retrospective goal-relabeling in Learning Classifier Systems.

Figures (8)

• Figure 1: The $9 \times 9$Maze 6 environment layout. Dark gray cells denote non-traversable walls (obstacles), light blue cells denote the traversable path, and 'G' (light green) marks the goal state.
• Figure 2: The $4 \times 4$FrozenLake environment layout. S: Start, F: Frozen (safe), H: Hole (failure), G: Goal (success).
• Figure 3: Maze 6.The knowledge
• Figure 4: Maze 6. The knowledge up to trial 200
• Figure 5: Maze 6. Numerosity of population and the number of reliable classifiers.