Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SMOSE: Sparse Mixture of Shallow Experts for Interpretable Reinforcement Learning in Continuous Control Tasks

Mátyás Vincze, Laura Ferrarotti, Leonardo Lucio Custode, Bruno Lepri, Giovanni Iacca

TL;DR

SMoSE presents a sparse top-1 Mixture-of-Experts architecture for interpretable reinforcement learning in continuous control. It combines multiple linear, interpretable experts with an interpretable router, trained end-to-end via SAC and augmented with load-balancing penalties, followed by distillation of the router into decision trees for transparency. The approach yields competitive performance against non-interpretable methods and outperforms existing interpretable baselines across six MuJoCo benchmarks, while providing detailed expert-level interpretations (including Reacher-v4) through both router weights and DT-based explanations. This work demonstrates that high-performance, interpretable policies can be realized in continuous control by leveraging a sparse MoE and post-hoc distillation, with practical implications for safety, auditability, and trust in automated systems.

Abstract

Continuous control tasks often involve high-dimensional, dynamic, and non-linear environments. State-of-the-art performance in these tasks is achieved through complex closed-box policies that are effective, but suffer from an inherent opacity. Interpretable policies, while generally underperforming compared to their closed-box counterparts, advantageously facilitate transparent decision-making within automated systems. Hence, their usage is often essential for diagnosing and mitigating errors, supporting ethical and legal accountability, and fostering trust among stakeholders. In this paper, we propose SMOSE, a novel method to train sparsely activated interpretable controllers, based on a top-1 Mixture-of-Experts architecture. SMOSE combines a set of interpretable decisionmakers, trained to be experts in different basic skills, and an interpretable router that assigns tasks among the experts. The training is carried out via state-of-the-art Reinforcement Learning algorithms, exploiting load-balancing techniques to ensure fair expert usage. We then distill decision trees from the weights of the router, significantly improving the ease of interpretation. We evaluate SMOSE on six benchmark environments from MuJoCo: our method outperforms recent interpretable baselines and narrows the gap with noninterpretable state-of-the-art algorithms

SMOSE: Sparse Mixture of Shallow Experts for Interpretable Reinforcement Learning in Continuous Control Tasks

TL;DR

SMoSE presents a sparse top-1 Mixture-of-Experts architecture for interpretable reinforcement learning in continuous control. It combines multiple linear, interpretable experts with an interpretable router, trained end-to-end via SAC and augmented with load-balancing penalties, followed by distillation of the router into decision trees for transparency. The approach yields competitive performance against non-interpretable methods and outperforms existing interpretable baselines across six MuJoCo benchmarks, while providing detailed expert-level interpretations (including Reacher-v4) through both router weights and DT-based explanations. This work demonstrates that high-performance, interpretable policies can be realized in continuous control by leveraging a sparse MoE and post-hoc distillation, with practical implications for safety, auditability, and trust in automated systems.

Abstract

Continuous control tasks often involve high-dimensional, dynamic, and non-linear environments. State-of-the-art performance in these tasks is achieved through complex closed-box policies that are effective, but suffer from an inherent opacity. Interpretable policies, while generally underperforming compared to their closed-box counterparts, advantageously facilitate transparent decision-making within automated systems. Hence, their usage is often essential for diagnosing and mitigating errors, supporting ethical and legal accountability, and fostering trust among stakeholders. In this paper, we propose SMOSE, a novel method to train sparsely activated interpretable controllers, based on a top-1 Mixture-of-Experts architecture. SMOSE combines a set of interpretable decisionmakers, trained to be experts in different basic skills, and an interpretable router that assigns tasks among the experts. The training is carried out via state-of-the-art Reinforcement Learning algorithms, exploiting load-balancing techniques to ensure fair expert usage. We then distill decision trees from the weights of the router, significantly improving the ease of interpretation. We evaluate SMOSE on six benchmark environments from MuJoCo: our method outperforms recent interpretable baselines and narrows the gap with noninterpretable state-of-the-art algorithms

Paper Structure

This paper contains 72 sections, 56 equations, 19 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Related work
  4. Method
  5. Results
  6. Training performance
  7. Policy evaluation
  8. Policy interpretation
  9. Expert 1 ($S_1 \, \approx \, 2.7 \, y_{\rm T} - 5.6\, \Delta_y$)
  10. Expert 2 ($S_2 \, \approx \, -3.5 \, y_{\rm T} + 8.3 \, \Delta_y$)
  11. Expert 3 ($S_3 \, \approx \, 3.9 \, y_{\rm T} - 5.8 \, \Delta_y$)
  12. Expert 4 ($S_4 \, \approx \, 4.1 \, y_{\rm T} + 2.1 \, \Delta_x - 6.0 \,\Delta_y$)
  13. Expert 5 ($S_5 \, \approx \, 3.5 \, \Delta_x$)
  14. Expert 6 ($S_6 \, \approx \, -2.6 \, \Delta_x - 4.0 \, \Delta_y$)
  15. Expert 7 ($S_7 \, \approx \, -2.5 \, y_{\rm T} + 6.4 \, \Delta_y$)
  16. ...and 57 more sections

Figures (19)

  • Figure 1: SMoSE. Schematic summary of the proposed architecture.
  • Figure 2: Performance in training.SMoSE compares to non-interpretable models of the same size, considering the number of overall (SAC-M) and active (SAC-S) parameters.
  • Figure 3: Reacher-v4. Visualization of the weights for each expert and the corresponding column of the router's weight matrix.
  • Figure 4: Visual representation of the Reacher-v4 and Walker2d-v4 environments.
  • Figure 5: Visual representation of the Hopper-v4 and Swimmer-v4 environments.
  • ...and 14 more figures

Theorems & Definitions (1)

  • Remark