Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SPOT: Scalable Policy Optimization with Trees for Markov Decision Processes

Xuyuan Xiong, Pedro Chumpitaz-Flores, Kaixun Hua, Cheng Hua

TL;DR

SPOT addresses the challenge of learning interpretable decision-tree policies for finite MDPs by formulating policy optimization as a MILP and solving it with a reduced-space branch-and-bound policy-iteration scheme. It decouples MDP dynamics from tree-structure constraints and exploits parallelism to scale to large state spaces, with theoretical convergence guarantees under a sufficiently expressive tree class and positive occupancy weights. Empirically, SPOT achieves orders of magnitude speedups over prior exact methods while producing compact, interpretable trees suitable for safety-critical deployment. The approach advances practical, verifiable RL by delivering high-quality policies that balance interpretability and scalability.

Abstract

Interpretable reinforcement learning policies are essential for high-stakes decision-making, yet optimizing decision tree policies in Markov Decision Processes (MDPs) remains challenging. We propose SPOT, a novel method for computing decision tree policies, which formulates the optimization problem as a mixed-integer linear program (MILP). To enhance efficiency, we employ a reduced-space branch-and-bound approach that decouples the MDP dynamics from tree-structure constraints, enabling efficient parallel search. This significantly improves runtime and scalability compared to previous methods. Our approach ensures that each iteration yields the optimal decision tree. Experimental results on standard benchmarks demonstrate that SPOT achieves substantial speedup and scales to larger MDPs with a significantly higher number of states. The resulting decision tree policies are interpretable and compact, maintaining transparency without compromising performance. These results demonstrate that our approach simultaneously achieves interpretability and scalability, delivering high-quality policies an order of magnitude faster than existing approaches.

SPOT: Scalable Policy Optimization with Trees for Markov Decision Processes

TL;DR

SPOT addresses the challenge of learning interpretable decision-tree policies for finite MDPs by formulating policy optimization as a MILP and solving it with a reduced-space branch-and-bound policy-iteration scheme. It decouples MDP dynamics from tree-structure constraints and exploits parallelism to scale to large state spaces, with theoretical convergence guarantees under a sufficiently expressive tree class and positive occupancy weights. Empirically, SPOT achieves orders of magnitude speedups over prior exact methods while producing compact, interpretable trees suitable for safety-critical deployment. The approach advances practical, verifiable RL by delivering high-quality policies that balance interpretability and scalability.

Abstract

Interpretable reinforcement learning policies are essential for high-stakes decision-making, yet optimizing decision tree policies in Markov Decision Processes (MDPs) remains challenging. We propose SPOT, a novel method for computing decision tree policies, which formulates the optimization problem as a mixed-integer linear program (MILP). To enhance efficiency, we employ a reduced-space branch-and-bound approach that decouples the MDP dynamics from tree-structure constraints, enabling efficient parallel search. This significantly improves runtime and scalability compared to previous methods. Our approach ensures that each iteration yields the optimal decision tree. Experimental results on standard benchmarks demonstrate that SPOT achieves substantial speedup and scales to larger MDPs with a significantly higher number of states. The resulting decision tree policies are interpretable and compact, maintaining transparency without compromising performance. These results demonstrate that our approach simultaneously achieves interpretability and scalability, delivering high-quality policies an order of magnitude faster than existing approaches.
Paper Structure (23 sections, 3 theorems, 22 equations, 4 figures, 3 tables, 2 algorithms)

This paper contains 23 sections, 3 theorems, 22 equations, 4 figures, 3 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Interpretable Policy Learning in RL
  3. Optimal Decision Tree Policies
  4. Preliminaries
  5. Markov Decision Processes.
  6. Solving MDPs via Linear Programming and its Dual Formulation.
  7. Scalable Policy Optimization with Trees
  8. Reduced-Space Branch-and-Bound Framework
  9. Two Stage Stochastic Program
  10. Closed-Form Upper Bounding Strategy
  11. Experiments
  12. Conclusion
  13. Notation
  14. Case study: tiger_vs_ant
  15. Additional Experiment Details
  16. ...and 8 more sections

Key Result

Proposition 1

Each iteration of SPOT corresponds to performing a one-step policy gradient ascent update starting from $\pi^{\text{old}}$. In particular, the update maximizes the same first-order objective that the policy gradient theorem uses, resulting in a new policy that is a greedy improvement on $\pi^{\text{

Figures (4)

  • Figure 1: Left: An interpretable policy learned by a decision tree of depth $D=2$. Right: Illustration of the Frozen Lake 4×4 (Stochastic) Markov Decision Process.
  • Figure 2: Branching Strategy Comparison between SPOT and OMDT. OMDT uses standard solvers (e.g., Gurobi), requiring branching on all integer variables, thus linearly increasing complexity with the number of states. In contrast, SPOT employs a two-stage reduced-space branch-and-bound approach, branching only on tree variables.
  • Figure 3: Learned interpretable decision tree policy with $D=3$ in environment tiger_vs_ant
  • Figure 4: Running time comparisons across different MDPs using Gurobi, serial RSBB, and parallel RSBB. Each bar represents the mean runtime, with error bars showing standard deviation.

Theorems & Definitions (5)

  • Proposition 1
  • Proposition 2
  • Theorem 1
  • proof
  • proof