Adapting the Behavior of Reinforcement Learning Agents to Changing Action Spaces and Reward Functions

TL;DR

The paper tackles non-stationary reinforcement learning where reward functions and action spaces change over time. It introduces MORPHIN, a self-adaptive Q-learning framework that couples online concept-drift detection via the Page-Hinkley test with dynamic adjustments of exploration and learning rates, while preserving a single $Q$-table to prevent forgetting and support new actions. Empirical results in Gridworld and traffic-signal control show MORPHIN achieves faster adaptation and improved learning efficiency (up to $1.7\times$) compared to standard Q-learning, though drift detector sensitivity can miss subtler drifts. This work contributes a lightweight, practical approach for continual learning in dynamic environments, with potential extensions to deep RL and edge deployments.

Abstract

Reinforcement Learning (RL) agents often struggle in real-world applications where environmental conditions are non-stationary, particularly when reward functions shift or the available action space expands. This paper introduces MORPHIN, a self-adaptive Q-learning framework that enables on-the-fly adaptation without full retraining. By integrating concept drift detection with dynamic adjustments to learning and exploration hyperparameters, MORPHIN adapts agents to changes in both the reward function and on-the-fly expansions of the agent's action space, while preserving prior policy knowledge to prevent catastrophic forgetting. We validate our approach using a Gridworld benchmark and a traffic signal control simulation. The results demonstrate that MORPHIN achieves superior convergence speed and continuous adaptation compared to a standard Q-learning baseline, improving learning efficiency by up to 1.7x.

Figures (3)

• Figure 1: Internal dynamics of morphin versus a standard Q-learning agent, illustrating their contrasting responses to concept drift (red dashed lines) in the Gridworld scenario, the y-axis for the scatter points represents the Q-value of the chosen state-action pair for that step, while their color indicates whether the action was exploratory (yellow) or exploitative (blue). (a) morphin's adaptive response: Upon detecting a drift, the PH-test triggers a reset of the exploration rate $\varepsilon^*$ (purple), forcing re-exploration. The resulting high TD-errors cause the dynamic learning rate $\alpha^*$ (green) to increase, accelerating the integration of new knowledge and enabling rapid policy recovery. (b) Standard Q-learning's static behavior: The baseline agent uses a single, exponentially decaying exploration schedule ($\varepsilon$) and a fixed $\alpha$ value. After the initial convergence, the low exploration rate prevents it from adapting to subsequent drifts, causing the agent to remain committed to an obsolete policy and resulting in a sustained performance collapse.
• Figure 2: Comparison of Q-value heatmaps demonstrating morphin's superior adaptation over standard Q-learning in two non-stationary scenarios. (a) In a goal-switching environment, morphin (left) preserves high Q-values for both initial and subsequent goals, showcasing effective knowledge retention and preventing catastrophic forgetting, unlike the baseline agent (right). (b) When the action space is expanded, morphin (left) successfully integrates a new "jump" action (indicated by double arrows) to discover a more optimal policy, while the baseline (right) fails to adapt, remaining committed to a suboptimal policy.
• Figure 3: Learning performance in the traffic scenario. After the first drift (episode 3,000), morphin detects the change (green line) and rapidly recovers. Traditional Q-learning suffers prolonged degradation. The second drift (episode 8,000) is not detected by the PH-tests because the new reward distribution is a subset of previously seen values and does not exceed the sensitivity threshold.