Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Affectively Framework: Towards Human-like Affect-Based Agents

Matthew Barthet, Roberto Gallotta, Ahmed Khalifa, Antonios Liapis, Georgios N. Yannakakis

TL;DR

Problem: Reinforcement learning for gameplay lacks incorporation of human affect in observations and rewards. Approach: the Affectively Framework extends OpenAI Gym with three diverse game environments (Pirates, Heist, Solid Rally) and a KNN-based affect model that produces arousal signals every 3 seconds; the total reward combines a behaviour-based component $R_B$ and an affect-based component $R_A$ via $R_t = (1 - \lambda) * n(R_B) + \lambda * R_A$. Key findings: PPO agents can raise arousal when optimizing $R_A$, but often fail to achieve strong in-game scores; blended rewards produce mixed performance and can be dominated by either $R_B$ or $R_A$ depending on the game; human demonstrations provide a strong benchmark. Significance: provides an open-source, extensible platform for affect-aware reinforcement learning research and a foundation for future persona-driven and exploration-aware training to produce more human-like gameplay agents.

Abstract

Game environments offer a unique opportunity for training virtual agents due to their interactive nature, which provides diverse play traces and affect labels. Despite their potential, no reinforcement learning framework incorporates human affect models as part of their observation space or reward mechanism. To address this, we present the \emph{Affectively Framework}, a set of Open-AI Gym environments that integrate affect as part of the observation space. This paper introduces the framework and its three game environments and provides baseline experiments to validate its effectiveness and potential.

Affectively Framework: Towards Human-like Affect-Based Agents

TL;DR

Problem: Reinforcement learning for gameplay lacks incorporation of human affect in observations and rewards. Approach: the Affectively Framework extends OpenAI Gym with three diverse game environments (Pirates, Heist, Solid Rally) and a KNN-based affect model that produces arousal signals every 3 seconds; the total reward combines a behaviour-based component and an affect-based component via . Key findings: PPO agents can raise arousal when optimizing , but often fail to achieve strong in-game scores; blended rewards produce mixed performance and can be dominated by either or depending on the game; human demonstrations provide a strong benchmark. Significance: provides an open-source, extensible platform for affect-aware reinforcement learning research and a foundation for future persona-driven and exploration-aware training to produce more human-like gameplay agents.

Abstract

Game environments offer a unique opportunity for training virtual agents due to their interactive nature, which provides diverse play traces and affect labels. Despite their potential, no reinforcement learning framework incorporates human affect models as part of their observation space or reward mechanism. To address this, we present the \emph{Affectively Framework}, a set of Open-AI Gym environments that integrate affect as part of the observation space. This paper introduces the framework and its three game environments and provides baseline experiments to validate its effectiveness and potential.
Paper Structure (11 sections, 4 equations, 2 figures, 2 tables)

This paper contains 11 sections, 4 equations, 2 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Affectively Framework
  3. Environments
  4. Affect Model
  5. Experimental Protocol
  6. Random Agent
  7. Humans
  8. PPO Agent
  9. Results
  10. Discussion
  11. Conclusion

Figures (2)

  • Figure 1: Overview of the Affectively Framework architecture for training affect-based RL agents. $A_t$ denotes the action taken by the agent, $S_t$ is the current observation, Aff$_t$ is the current affect value and $R_t$ is the reward assigned to the agent.
  • Figure 2: The Affectively Framework. From top to bottom: (a) Pirates platformer game, (b) Heist first-person shooter, and (c) Solid Rally racing game.