Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Distilling Reinforcement Learning Policies for Interpretable Robot Locomotion: Gradient Boosting Machines and Symbolic Regression

Fernando Acero, Zhibin Li

TL;DR

A novel approach to distill neural RL policies into more interpretable forms using Gradient Boosting Machines, Explainable Boosting Machines and Symbolic Regression, by leveraging the inherent interpretability of generalized additive models, decision trees, and analytical expressions.

Abstract

Recent advancements in reinforcement learning (RL) have led to remarkable achievements in robot locomotion capabilities. However, the complexity and ``black-box'' nature of neural network-based RL policies hinder their interpretability and broader acceptance, particularly in applications demanding high levels of safety and reliability. This paper introduces a novel approach to distill neural RL policies into more interpretable forms using Gradient Boosting Machines (GBMs), Explainable Boosting Machines (EBMs) and Symbolic Regression. By leveraging the inherent interpretability of generalized additive models, decision trees, and analytical expressions, we transform opaque neural network policies into more transparent ``glass-box'' models. We train expert neural network policies using RL and subsequently distill them into (i) GBMs, (ii) EBMs, and (iii) symbolic policies. To address the inherent distribution shift challenge of behavioral cloning, we propose to use the Dataset Aggregation (DAgger) algorithm with a curriculum of episode-dependent alternation of actions between expert and distilled policies, to enable efficient distillation of feedback control policies. We evaluate our approach on various robot locomotion gaits -- walking, trotting, bounding, and pacing -- and study the importance of different observations in joint actions for distilled policies using various methods. We train neural expert policies for 205 hours of simulated experience and distill interpretable policies with only 10 minutes of simulated interaction for each gait using the proposed method.

Distilling Reinforcement Learning Policies for Interpretable Robot Locomotion: Gradient Boosting Machines and Symbolic Regression

TL;DR

A novel approach to distill neural RL policies into more interpretable forms using Gradient Boosting Machines, Explainable Boosting Machines and Symbolic Regression, by leveraging the inherent interpretability of generalized additive models, decision trees, and analytical expressions.

Abstract

Recent advancements in reinforcement learning (RL) have led to remarkable achievements in robot locomotion capabilities. However, the complexity and ``black-box'' nature of neural network-based RL policies hinder their interpretability and broader acceptance, particularly in applications demanding high levels of safety and reliability. This paper introduces a novel approach to distill neural RL policies into more interpretable forms using Gradient Boosting Machines (GBMs), Explainable Boosting Machines (EBMs) and Symbolic Regression. By leveraging the inherent interpretability of generalized additive models, decision trees, and analytical expressions, we transform opaque neural network policies into more transparent ``glass-box'' models. We train expert neural network policies using RL and subsequently distill them into (i) GBMs, (ii) EBMs, and (iii) symbolic policies. To address the inherent distribution shift challenge of behavioral cloning, we propose to use the Dataset Aggregation (DAgger) algorithm with a curriculum of episode-dependent alternation of actions between expert and distilled policies, to enable efficient distillation of feedback control policies. We evaluate our approach on various robot locomotion gaits -- walking, trotting, bounding, and pacing -- and study the importance of different observations in joint actions for distilled policies using various methods. We train neural expert policies for 205 hours of simulated experience and distill interpretable policies with only 10 minutes of simulated interaction for each gait using the proposed method.
Paper Structure (11 sections, 4 equations, 11 figures, 2 tables, 1 algorithm)

This paper contains 11 sections, 4 equations, 11 figures, 2 tables, 1 algorithm.

Table of Contents

  1. INTRODUCTION
  2. Background
  3. Reinforcement Learning for Robot Locomotion
  4. Gradient Boosting Machines and Symbolic Regression
  5. Methodology
  6. Training Reinforcement Learning Expert Policies
  7. Distilling Interpretable Locomotion Policies
  8. RESULTS
  9. Performance of Distilled Policies
  10. Interpretability of Distilled Policies
  11. CONCLUSION

Figures (11)

  • Figure 1: From black-box to glass-box: summary of the proposed framework for distillation of neural network-based RL policies into interpretable policies consisting of GBMs, EBMs, and symbolic policies.
  • Figure 2: Average episodic performance of distilled policies for all gaits tested using various alternation ratios. Note that alternation ratio of 1 means only the neural RL expert is used, 0 means only the distilled policy is used.
  • Figure 3: Gait sequences for walk, trot, pace, and bound GBM policies.
  • Figure 4: Gait sequences for walk, trot, pace, and bound EBM policies.
  • Figure 5: Gait sequences for walk, trot, pace, and bound symbolic policies.
  • ...and 6 more figures