Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Getting By Goal Misgeneralization With a Little Help From a Mentor

Tu Trinh, Mohamad H. Danesh, Nguyen X. Khanh, Benjamin Plaut

TL;DR

This paper evaluates multiple methods for determining when the agent should request help and finds that asking for help consistently improves performance, but finds that methods based on the agent's internal state fail to proactively request help and suggest that the agent's internal state does not represent the coin at all.

Abstract

While reinforcement learning (RL) agents often perform well during training, they can struggle with distribution shift in real-world deployments. One particularly severe risk of distribution shift is goal misgeneralization, where the agent learns a proxy goal that coincides with the true goal during training but not during deployment. In this paper, we explore whether allowing an agent to ask for help from a supervisor in unfamiliar situations can mitigate this issue. We focus on agents trained with PPO in the CoinRun environment, a setting known to exhibit goal misgeneralization. We evaluate multiple methods for determining when the agent should request help and find that asking for help consistently improves performance. However, we also find that methods based on the agent's internal state fail to proactively request help, instead waiting until mistakes have already occurred. Further investigation suggests that the agent's internal state does not represent the coin at all, highlighting the importance of learning nuanced representations, the risks of ignoring everything not immediately relevant to reward, and the necessity of developing ask-for-help strategies tailored to the agent's training algorithm.

Getting By Goal Misgeneralization With a Little Help From a Mentor

TL;DR

This paper evaluates multiple methods for determining when the agent should request help and finds that asking for help consistently improves performance, but finds that methods based on the agent's internal state fail to proactively request help and suggest that the agent's internal state does not represent the coin at all.

Abstract

While reinforcement learning (RL) agents often perform well during training, they can struggle with distribution shift in real-world deployments. One particularly severe risk of distribution shift is goal misgeneralization, where the agent learns a proxy goal that coincides with the true goal during training but not during deployment. In this paper, we explore whether allowing an agent to ask for help from a supervisor in unfamiliar situations can mitigate this issue. We focus on agents trained with PPO in the CoinRun environment, a setting known to exhibit goal misgeneralization. We evaluate multiple methods for determining when the agent should request help and find that asking for help consistently improves performance. However, we also find that methods based on the agent's internal state fail to proactively request help, instead waiting until mistakes have already occurred. Further investigation suggests that the agent's internal state does not represent the coin at all, highlighting the importance of learning nuanced representations, the risks of ignoring everything not immediately relevant to reward, and the necessity of developing ask-for-help strategies tailored to the agent's training algorithm.

Paper Structure

This paper contains 13 sections, 7 figures.

Table of Contents

  1. Introduction
  2. Experimental Setup
  3. Training and Testing Environments
  4. The Cost of Asking For Help
  5. Experimentation Pipeline
  6. Using Agent's Action Distribution
  7. Using Anomalous Observations
  8. Ask-For-Help Skylines and the Importance of Learned Representations
  9. Conclusion
  10. Social Impact Statement
  11. Deep-SVDD Model
  12. Related Work
  13. Future Work

Figures (7)

  • Figure 1: Left (training environment): agent successfully gets the coin on the far right. Right (testing environment): agent jumps over the coin in the middle and still heads towards the right.
  • Figure 2: How different percentile thresholds result in different ask-for-help percentages
  • Figure 3: Performance on coinrun_aisc using action-based methods
  • Figure 4: Higher percentile thresholds correspond to being more "lenient" with observation abnormality, leading to lower AFHPs
  • Figure 5: Performance on coinrun_aisc using observation-based methods, compared to that of action-based methods
  • ...and 2 more figures