Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Coprocessor Actor Critic: A Model-Based Reinforcement Learning Approach For Adaptive Brain Stimulation

Michelle Pan, Mariah Schrum, Vivek Myers, Erdem Bıyık, Anca Dragan

TL;DR

Coprocessor Actor Critic is introduced, a novel, model-based reinforcement learning (MBRL) approach for learning neural coprocessor policies for brain stimulation that overcomes the limitations of traditional MFRL methods in terms of sample efficiency and task success and outperforms baseline MBRL approaches in a neurologically realistic model of an injured brain.

Abstract

Adaptive brain stimulation can treat neurological conditions such as Parkinson's disease and post-stroke motor deficits by influencing abnormal neural activity. Because of patient heterogeneity, each patient requires a unique stimulation policy to achieve optimal neural responses. Model-free reinforcement learning (MFRL) holds promise in learning effective policies for a variety of similar control tasks, but is limited in domains like brain stimulation by a need for numerous costly environment interactions. In this work we introduce Coprocessor Actor Critic, a novel, model-based reinforcement learning (MBRL) approach for learning neural coprocessor policies for brain stimulation. Our key insight is that coprocessor policy learning is a combination of learning how to act optimally in the world and learning how to induce optimal actions in the world through stimulation of an injured brain. We show that our approach overcomes the limitations of traditional MFRL methods in terms of sample efficiency and task success and outperforms baseline MBRL approaches in a neurologically realistic model of an injured brain.

Coprocessor Actor Critic: A Model-Based Reinforcement Learning Approach For Adaptive Brain Stimulation

TL;DR

Coprocessor Actor Critic is introduced, a novel, model-based reinforcement learning (MBRL) approach for learning neural coprocessor policies for brain stimulation that overcomes the limitations of traditional MFRL methods in terms of sample efficiency and task success and outperforms baseline MBRL approaches in a neurologically realistic model of an injured brain.

Abstract

Adaptive brain stimulation can treat neurological conditions such as Parkinson's disease and post-stroke motor deficits by influencing abnormal neural activity. Because of patient heterogeneity, each patient requires a unique stimulation policy to achieve optimal neural responses. Model-free reinforcement learning (MFRL) holds promise in learning effective policies for a variety of similar control tasks, but is limited in domains like brain stimulation by a need for numerous costly environment interactions. In this work we introduce Coprocessor Actor Critic, a novel, model-based reinforcement learning (MBRL) approach for learning neural coprocessor policies for brain stimulation. Our key insight is that coprocessor policy learning is a combination of learning how to act optimally in the world and learning how to induce optimal actions in the world through stimulation of an injured brain. We show that our approach overcomes the limitations of traditional MFRL methods in terms of sample efficiency and task success and outperforms baseline MBRL approaches in a neurologically realistic model of an injured brain.
Paper Structure (20 sections, 6 equations, 11 figures, 2 algorithms)

This paper contains 20 sections, 6 equations, 11 figures, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Reinforcement Learning for Brain Stimulation:
  4. Model Based Reinforcement Learning:
  5. Problem Setup
  6. Methodology
  7. Training world-action value model
  8. Training Brain Model
  9. Updating world-action value model
  10. Experiments
  11. In Silico Evaluation Environments
  12. Evaluation Reward
  13. Training Reward
  14. Discussion
  15. Conclusion
  16. ...and 5 more sections

Figures (11)

  • Figure 1: Overview of our framework. We first learn the Q-function, $Q^\psi$, for world actions, $a$, via a biomechanically realistic simulator. We then learn the mapping, $\widehat{F}_{\text{brain}}^{\,\phi}$, from coprocessor actions, $\bar{a}$, to world actions. Simultaneously, we update $Q^\psi$ to account for the altered MDP.
  • Figure 2: This figure shows the brain stimulation domain for the MyoSim Arm Reach task. We model the biomechanics of the reaching tasks using the MyoSuite physics simulator wang2022myosim. The brain of a stroke patient is modeled via the approach described by michaelsmodel and consists of the anterior intraparietal area (AIP), ventral premotor cortex (F5), and primary motor cortex (M1) modules. The coprocessor applies stimulation to the motor cortex (M1) which modifies the world action of the patient.
  • Figure 3: Evaluation results for CopAC compared to SAC, MBPO, and ablated CopAC. The dashed line represents the reward obtained by the healthy brain.
  • Figure 4: Evaluation results for CopAC compared to SAC and MBPO, and ablated CopAC. We display the evaluation reward after 25 episodes of training. Statistically significant differences between CopAC and other methods are marked with brackets.
  • Figure 5: Training results for CopAC compared to SAC, MBPO, and ablated CopAC. The dashed line represents the reward obtained by the healthy brain.
  • ...and 6 more figures