OptimizedDP: An Efficient, User-friendly Library For Optimal Control and Dynamic Programming

TL;DR

OptimizedDP tackles the exponential bottleneck of grid-based dynamic programming in control and robotics by delivering a high-performance Python toolbox with a HeteroCL-backed solver core. It supports time-dependent and time-independent Hamilton-Jacobi PDEs as well as discretized continuous-space MDP value iteration, combining level-set methods and Lax-Friedrichs sweeping to reach high-dimensional problems (up to 6–8 dimensions) efficiently. The framework emphasizes modular problem description, cache-aware parallel execution, and flexible visualization, delivering substantial speedups (order-of-magnitude in many cases) over existing toolboxes while maintaining a user-friendly interface. While not removing the curse of dimensionality, OptimizedDP provides a practical platform for generating ground-truth value functions and a staging ground for dimension-reduction and learning-based extensions, with public availability at the project repository.

Abstract

This paper introduces OptimizedDP, a high-performance software library for several common grid-based dynamic programming (DP) algorithms used in control theory and robotics. Specifically, OptimizedDP provides functions to numerically solve a class of time-dependent (dynamic) Hamilton-Jacobi (HJ) partial differential equations (PDEs), time-independent (static) HJ PDEs, and additionally value iteration for continuous action-state space Markov Decision Processes (MDP). The computational complexity of grid-based DP is exponential with respect to the number of grid or state space dimensions, and thus can have bad execution runtimes and memory usage whenapplied to large state spaces. We leverage the user-friendliness of Python for different problem specifications without sacrificing the efficiency of the core computation. This is achieved by implementing the core part of the code which the user does not see in heterocl, a framework we use to abstract away details of how computation is parallelized. Compared to similar toolboxes for level set methods that are used to solve the HJ PDE, our toolbox makes solving the PDE at higher dimensions possible as well as achieving an order of magnitude improvements in execution times, while keeping the interface easy for specifying different problem descriptions. Because of that, the toolbox has been adopted to solve control and optimization problems that were considered intractable before. Our toolbox is available publicly at https://github.com/SFU-MARS/optimized_dp.

Figures (14)

• Figure 1: Obtaining Minimal Backward Reachable Tube is crucial for guaranteeing safety. The Tube contains all the states the system will inevitable arrive at target set despite applying optimal control to avoid.
• Figure 2: Illustration of stages in numerical process of solving time-dependent HJ PDE. In ToolboxLS LsetToolbox1, temporary variables are stored in multidimensional arrays as the same size of the grid. As we increase the number of dimensions, the DRAM memory required for these temporary array goes up linearly. If the depth of the computation is large, the total amount memory used for temporary variables will exceed system's DRAM capabilities, limiting computations to low-dimensional control problem only.
• Figure 3: OptimizedDP's implementation of algorithm \ref{['algo:Time-dependent HJ PDE']} does not buffer temporary variables into multidimensional arrays. Instead, within each grid iteration, a grid point value in $V_{\text{new}}$ is directly computed. Each thread is assigned a chunk of grid points for parallel computation.
• Figure 4: The overall structure of OptimizedDP consists of red blocks (Python with NumPy) for problem specification, grid initialization, and plotting, and solid green blocks (Python/HeteroCL) for core algorithms. User-specified system dynamics object containing problem parameters and subroutines for optimal controls are then plugged into core solvers.
• Figure 5: 3D visualization of sub-zero level set across different timestep. User can choose value function at certain timestep to visualize