Accelerated Point-wise Maximum Approach to Approximate Dynamic Programming

TL;DR

The paper tackles the challenge of obtaining reliable sub-optimality bounds for infinite-horizon stochastic optimal control via dynamic programming. It introduces a point-wise maximum formulation and a gradient-based ADP algorithm that jointly builds a family of lower-bounding value functions, with inner refinements and outer updates guided by Dirac-pulse sampling to approximate high-dimensional integrals. Convergence guarantees are provided for the gradient-ascent procedure, and the authors give an interpretation of the gradient steps as adaptive choices of state relevance weighting. Numerical results on 1D and 10D linear-quadratic problems show that the proposed method yields significantly tighter sub-optimality bounds with substantially less computation time compared to existing PWM-based and iterated Bellman approaches, implying practical value for offline bound certification and offline policy evaluation.

Abstract

We describe an approximate dynamic programming approach to compute lower bounds on the optimal value function for a discrete time, continuous space, infinite horizon setting. The approach iteratively constructs a family of lower bounding approximate value functions by using the so-called Bellman inequality. The novelty of our approach is that, at each iteration, we aim to compute an approximate value function that maximizes the point-wise maximum taken with the family of approximate value functions computed thus far. This leads to a non-convex objective, and we propose a gradient ascent algorithm to find stationary points by solving a sequence of convex optimization problems. We provide convergence guarantees for our algorithm and an interpretation for how the gradient computation relates to the state relevance weighting parameter appearing in related approximate dynamic programming approaches. We demonstrate through numerical examples that, when compared to existing approaches, the algorithm we propose computes tighter sub-optimality bounds with less computation time.

Figures (6)

• Figure 1: Providing visual insight for Algorithm \ref{['alg:inner_problem']} using the 1-dimensional example described in Section \ref{['sec:numerical:1d']}. Sub-figure (b) is a zoomed view of sub-figure (a). On the upper axes, the dotted black line is $\bar{V}_{\mathrm{obj}}$ and $\bar{V}_{\mathrm{con}}$, the blue dot and blue line are the $x_{c,i}$ and $\alpha^{(0)}$ generated on line \ref{['alg:outer_problem:generate']} of Algorithm \ref{['alg:outer_problem']}. The red lines (solid and dashed) are the approximate value functions from the refinement steps of Algorithm \ref{['alg:inner_problem']}, with the solid line corresponding to the terminal iteration. Algorithm \ref{['alg:inner_problem']} converged in four steps to a $0.1\%$ relative tolerance on the objective value increase. The lower axes show the ${N_c \!=\! 10^6}$ samples as a histogram, with grey bars showing all samples, blue bars showing where the blue line is greater than $\bar{V}_{\mathrm{obj}}$, and red bars showing where the solid red line is greater than $\bar{V}_{\mathrm{obj}}$.
• Figure 2: Details of the proposed algorithms for: (left) the 1-dimensional example described in Section \ref{['sec:numerical:1d']}; (right) the 10-dimensional example described in Section \ref{['sec:numerical:linquad_ND']}. The blue lines show the results from running Algorithm \ref{['alg:outer_problem']} without using Algorithm \ref{['alg:inner_problem']} to refine the solution at each iteration, while the red lines show the results with refinement. The top figures show the point-wise maximum objective integrated with respect to the $N_c$ samples. The bottom figures show number of iterations of Algorithm \ref{['alg:inner_problem']} until the $\epsilon$-convergence criterion is triggered, i.e., the number of refinement steps performed. To make the bottom figures readable, the results are grouped between the deciles of each order of magnitude, with the horizontal red line showing the average number of iterations, and the grey box spanning the minimum and maximum. The online performance of a representative policy is shown by the dotted black line on the top figures. For the 1-dimensional example (left) an LQR policy was used with the input clipped to the constraints, while for the 10-dimensional example (right) a Model Predictive Controller was used with a 10-time-step horizon length and the Riccati equation solution as the terminal cost.
• Figure 3: Comparison with existing methods for computing lower bounds using a point-wise maximum of approximate value functions for the 10-dimensional example of Section \ref{['sec:numerical:linquad_ND']}. The top figure is comparable with Figure \ref{['fig:linquad']} (top right). The iterated Bellman inequality method boyd_2013_iteratedApproxValueFunctions (solid green) is computed with $100$ Bellman inequality iterations, while hand-tuning method beuchat_2017_pwm_at_CDC (dotted green) uses a sequence of 20 zero-mean Gaussian distributions in the objective with the variance increasing from ${\Sigma_{\nu} \!=\! 0.1 I_{n_x}}$ to ${\Sigma_{\nu} \!=\! 18 I_{n_x}}$ and repeated 10 times, i.e., 200 iterations total. The bottom figure shows the cumulative computation time for solving line \ref{['alg:inner_problem:lp']} of Algorithm \ref{['alg:inner_problem']} and line \ref{['alg:outer_problem:generate']} of Algorithm \ref{['alg:outer_problem']}.
• Figure : Find points satisfying necessary optimality conditions of problem \ref{['eq:canonical_inner_problem']} with $c$ as a sum of Dirac pulses
• Figure : Maximise the value of $\int \bar{V}_{\mathrm{obj}} dc$

Theorems & Definitions (4)

• Lemma 2.1
• Theorem 3.3
• Theorem 3.4
• Theorem B.1