Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Show, Don't Tell: Learning Reward Machines from Demonstrations for Reinforcement Learning-Based Cardiac Pacemaker Synthesis

John Komp, Dananjay Srinivas, Maria Pacheco, Ashutosh Trivedi

TL;DR

It is posit that it is significantly easier for domain experts, such as electrophysiologists, to observe and identify abnormalities in electrocardiograms that correspond to patient-pacemaker interactions and train an RL agent to synthesize a cardiac pacemaker based on the resulting reward machine.

Abstract

An (artificial cardiac) pacemaker is an implantable electronic device that sends electrical impulses to the heart to regulate the heartbeat. As the number of pacemaker users continues to rise, so does the demand for features with additional sensors, adaptability, and improved battery performance. Reinforcement learning (RL) has recently been proposed as a performant algorithm for creative design space exploration, adaptation, and statistical verification of cardiac pacemakers. The design of correct reward functions, expressed as a reward machine, is a key programming activity in this process. In 2007, Boston Scientific published a detailed description of their pacemaker specifications. This document has since formed the basis for several formal characterizations of pacemaker specifications using real-time automata and logic. However, because these translations are done manually, they are challenging to verify. Moreover, capturing requirements in automata or logic is notoriously difficult. We posit that it is significantly easier for domain experts, such as electrophysiologists, to observe and identify abnormalities in electrocardiograms that correspond to patient-pacemaker interactions. Therefore, we explore the possibility of learning correctness specifications from such labeled demonstrations in the form of a reward machine and training an RL agent to synthesize a cardiac pacemaker based on the resulting reward machine. We leverage advances in machine learning to extract signals from labeled demonstrations as reward machines using recurrent neural networks and transformer architectures. These reward machines are then used to design a simple pacemaker with RL. Finally, we validate the resulting pacemaker using properties extracted from the Boston Scientific document.

Show, Don't Tell: Learning Reward Machines from Demonstrations for Reinforcement Learning-Based Cardiac Pacemaker Synthesis

TL;DR

It is posit that it is significantly easier for domain experts, such as electrophysiologists, to observe and identify abnormalities in electrocardiograms that correspond to patient-pacemaker interactions and train an RL agent to synthesize a cardiac pacemaker based on the resulting reward machine.

Abstract

An (artificial cardiac) pacemaker is an implantable electronic device that sends electrical impulses to the heart to regulate the heartbeat. As the number of pacemaker users continues to rise, so does the demand for features with additional sensors, adaptability, and improved battery performance. Reinforcement learning (RL) has recently been proposed as a performant algorithm for creative design space exploration, adaptation, and statistical verification of cardiac pacemakers. The design of correct reward functions, expressed as a reward machine, is a key programming activity in this process. In 2007, Boston Scientific published a detailed description of their pacemaker specifications. This document has since formed the basis for several formal characterizations of pacemaker specifications using real-time automata and logic. However, because these translations are done manually, they are challenging to verify. Moreover, capturing requirements in automata or logic is notoriously difficult. We posit that it is significantly easier for domain experts, such as electrophysiologists, to observe and identify abnormalities in electrocardiograms that correspond to patient-pacemaker interactions. Therefore, we explore the possibility of learning correctness specifications from such labeled demonstrations in the form of a reward machine and training an RL agent to synthesize a cardiac pacemaker based on the resulting reward machine. We leverage advances in machine learning to extract signals from labeled demonstrations as reward machines using recurrent neural networks and transformer architectures. These reward machines are then used to design a simple pacemaker with RL. Finally, we validate the resulting pacemaker using properties extracted from the Boston Scientific document.

Paper Structure

This paper contains 30 sections, 4 equations, 4 figures, 4 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Advances in Pacemaker Design.
  3. Design and Verification of Pacemakers.
  4. RL-based Synthesis.
  5. Reward Machines.
  6. Availability of Labelled Traces.
  7. RL-driven Pacemaker Synthesis.
  8. Contributions.
  9. Basics of Pacing Therapy
  10. Heart Disease.
  11. Bradycardia Pacing Therapy.
  12. Pacemaker Requirements.
  13. Learning Reward Machines
  14. Learning a Trace Classifier.
  15. Pacemaker--Traces Dataset.
  16. ...and 15 more sections

Figures (4)

  • Figure 1: Learning Process from requirements and exemplars to trained pacemaker agent
  • Figure 2: Segment of a pacemaker trace showing blanking periods (gray bars), refractory periods (white bars), intrinsic beats (blue text), and paced events (green text). Atrial periods are shown above the timeline; ventricular are below.
  • Figure 3: Box plots, showing distribution of $F1$ scores across different context sizes for the $2$ different architectures.
  • Figure 4: Learned State Machine - AS = Atrial Sense, VS = Ventricular Sense, AP = Atrial Pace, VP = Ventricular Pace