Closing the Affective Loop via Experience-Driven Reinforcement Learning Designers

TL;DR

This paper addresses closing the affective loop in content generation by introducing Experience-Driven Reinforcement Learning (EDRL) to autonomously create racetracks that elicit target continuous arousal patterns. The approach builds on EDPCG by incorporating a learned RL designer and a data-driven arousal model, optimizing content so that the generated affect trace A_L matches a target A_T with reward $R_L = - D(A_L, A_T)$ where $D$ is the area-between-curves distance. The authors compare EDRL to a genetic EDPCG baseline across multiple player clusters and three arousal patterns in the Solid Rally domain using the AGAIN corpus, showing that EDRL is more efficient and robust in many settings. The work demonstrates the potential of online, personalized affect-driven content generation and suggests broad applicability to any domain requiring affective adaptation through generated stimuli.

Abstract

Autonomously tailoring content to a set of predetermined affective patterns has long been considered the holy grail of affect-aware human-computer interaction at large. The experience-driven procedural content generation framework realises this vision by searching for content that elicits a certain experience pattern to a user. In this paper, we propose a novel reinforcement learning (RL) framework for generating affect-tailored content, and we test it in the domain of racing games. Specifically, the experience-driven RL (EDRL) framework is given a target arousal trace, and it then generates a racetrack that elicits the desired affective responses for a particular type of player. EDRL leverages a reward function that assesses the affective pattern of any generated racetrack from a corpus of arousal traces. Our findings suggest that EDRL can accurately generate affect-driven racing game levels according to a designer's style and outperforms search-based methods for personalised content generation. The method is not only directly applicable to game content generation tasks but also employable broadly to any domain that uses content for affective adaptation.

Figures (4)

• Figure 1: Two examples of maximally and minimally arousing tracks generated by EDRL for the Solid Rally racing game. Top image: EDRL offers content that constantly elicits higher levels of arousal, such as loops and sequences of alternating turns. Bottom image: EDRL generates simple and straight tracks, leaving limited room for highly arousing gameplay. Blue overlay elements denote regions with a strong decrease in arousal, while red denotes regions of a strong increase in arousal, and white denotes neutral arousal changes.
• Figure 2: High-level overview of our EDRL framework, with visual examples from our case study detailed in Section \ref{['sec:edrl_solid']}. L refers to the level generated by the designer, $R_L$ refers to the reward function (Eq. \ref{['eq:reward']}), $S_T$ and $A_T$ are the state and affect traces generated by the evaluator during testing, and $A_T$ is the target affect signal the agent is trying to imitate.
• Figure 3: Visualisation of the different possible track components found in Solid Rally.
• Figure 4: Mean arousal traces from all players and the three identified clusters of players within Solid Rally, which we call the Beginners, the Excited Experts and Unexcited Experts. Shaded areas denote a 95% confidence interval.