Kernel-based diffusion approximated Markov decision processes for autonomous navigation and control on unstructured terrains

TL;DR

The paper introduces a diffusion-approximation framework for continuous-state MDPs tailored to autonomous navigation on unstructured terrains. By applying a second-order Taylor expansion of the value function, the Bellman equations are approximated as a diffusion-type PDE that depends only on the first and second moments of the transition distribution, obviating the need for a fully specified transition model. A kernel-based representation of the value function is then used to transform the PDE into a solvable linear system over a finite set of supporting states, enabling efficient kernel Taylor-based policy evaluation and iteration. The approach is validated through plane and Martian terrain simulations, realistic physics-based simulations, and real-world indoor/outdoor experiments, demonstrating improved policy quality, computational efficiency, and robustness to action uncertainty. The results indicate practical applicability for real-time decision-making in challenging off-road environments and highlight avenues for improved support-state placement and high-dimensional scalability.

Abstract

We propose a diffusion approximation method to the continuous-state Markov Decision Processes (MDPs) that can be utilized to address autonomous navigation and control in unstructured off-road environments. In contrast to most decision-theoretic planning frameworks that assume fully known state transition models, we design a method that eliminates such a strong assumption that is often extremely difficult to engineer in reality. We first take the second-order Taylor expansion of the value function. The Bellman optimality equation is then approximated by a partial differential equation, which only relies on the first and second moments of the transition model. By combining the kernel representation of the value function, we design an efficient policy iteration algorithm whose policy evaluation step can be represented as a linear system of equations characterized by a finite set of supporting states. We first validate the proposed method through extensive simulations in 2D obstacle avoidance and 2.5D terrain navigation problems. The results show that the proposed approach leads to a much superior performance over several baselines. We then develop a system that integrates our decision-making framework with onboard perception and conduct real-world experiments in both cluttered indoor and unstructured outdoor environments. The results from the physical systems further demonstrate the applicability of our method in challenging real-world environments.

Figures (14)

• Figure 1: In unstructured environments, the robot needs to make motion decisions in the navigable space with spatially varying terrestrial characteristics (hills, ridges, valleys, slopes). This is different from the simplified and structured environments where there are only two types of representations, i.e., either obstacle-occupied or obstacle-free. Evenly tessellating the complex terrain to create a discretized state space cannot effectively characterize the underlying value function used for computing the MDP solution. (Picture credit: NASA)
• Figure 2: Illustration of the boundary condition in Eq. \ref{['boundary-condition-a']} using a 2D example. The blue and green regions indicate the state space and the goal region, respectively. The grey areas represent the infeasible space, e.g., obstacles, and the boundaries are shown as black curves. The normal vector $\hat{\mathbf{n}}$ and the gradient of value function $\nabla v^{\pi}(s)$ at three arbitrary boundary points are indicated by yellow and red arrows, respectively.
• Figure 3: Evaluation with a traditional simplified scenario where obstacles and goal are depicted as red and green blocks, respectively. We compare the final value function and the final policy obtained from (a) kernel Taylor-based PI, (b) direct kernel-based PI, (c) N-FVPI, and (d) grid-based PI. A brighter background color represents a higher state value. The policies are the arrows (vector fields), and each arrow points to some next waypoint. Orange dots denote the states, or the grid centers (in the case of grid-based PI), which are used to update the value functions.
• Figure 4: The comparison of the average return of the policies computed by our method (kernel Taylor-based PI) and four other baselines. The x-axis is the number of supporting states/grids used in computing the policy. In the case of PPO, it represents the number of samples used for value and policy updates. The y-axis shows the average return. The error bars represent the standard deviations.
• Figure 5: Computational time comparisons of the four value-based algorithms with changing number of states. (a) The computational time per iteration. (b) Number of iterations to convergence.