Personalizing Exposure Therapy via Reinforcement Learning

TL;DR

This study addresses the challenge of personalizing VR exposure therapy for arachnophobia by introducing EDPCGRL, an Experience-Driven Procedural Content Generation framework that uses tabular Q-learning to adapt a VR spider’s attributes in real time based on Skin Conductance Level. The spider environment comprises six adjustable attributes, a 486-state space, and a reward shaped to encourage reaching therapist-defined target anxiety levels within $[-1,1]$, with exploration via an $\\epsilon=0.05$ policy. In a human-subject study, the RL-based method outperformed a rules-based baseline in achieving precise and higher anxiety levels, as evidenced by SUDs, SCL, SCR features, and STAI-6 ratings, though it also elicited higher perceived stress. A follow-up spider-personalization analysis reveals diverse attribute sensitivities across participants, underscoring the need for individualized stimulus generation in VR-based exposure therapy and suggesting pathways toward clinically validated, personalized digital interventions.

Abstract

Personalized therapy, in which a therapeutic practice is adapted to an individual patient, can lead to improved health outcomes. Typically, this is accomplished by relying on a therapist's training and intuition along with feedback from a patient. However, this requires the therapist to become an expert on any technological components, such as in the case of Virtual Reality Exposure Therapy (VRET). While there exist approaches to automatically adapt therapeutic content to a patient, they generally rely on hand-authored, pre-defined rules, which may not generalize to all individuals. In this paper, we propose an approach to automatically adapt therapeutic content to patients based on physiological measures. We implement our approach in the context of virtual reality arachnophobia exposure therapy, and rely on experience-driven procedural content generation via reinforcement learning (EDPCGRL) to generate virtual spiders to match an individual patient. Through a human subject study, we demonstrate that our system significantly outperforms a more common rules-based method, highlighting its potential for enhancing personalized therapeutic interventions.

Figures (7)

• Figure 1: Proposed EDPCGRL framework
• Figure 3: Average SUDs (top) and SCL (bottom) of participants during the VR experience. The left plots display anxiety levels for participants first exposed to an anxious environment adapted by the RL method, followed by the rules-based method. The right plot depicts the opposite order. The shaded region represents the standard deviation. Participants were not asked to rate their SUDs during the relaxing environment.
• Figure 4: Comparison of three SCR features between the low-anxiety and high-anxiety segments when the adaptive methods were set to the rules-based method or RL method. The leftmost plot gives the number of peaks, the middle plot gives the mean of SCR amplitudes, and the rightmost gives the sum of SCR amplitudes. Across all three features, the RL method has a significant difference between the low and high anxiety values.
• Figure : Relaxing
• Figure : Relaxing