A hierarchical control framework for autonomous decision-making systems: Integrating HMDP and MPC

TL;DR

The paper tackles autonomous decision-making by unifying discrete high-level maneuver decisions with continuous low-level dynamics using a Hybrid MDP (HMDP) and an MPC-like planning scheme. It models the interaction between discrete states $s(k)$ and continuous states $x(k)$ with environment dynamics $\\Xi(k)$ via $s(k+1)=f(s(k,\\pi(s(k),x(k),\\Xi(k)))$, $x(k+1)=\\tilde{g}_{s(k)}(x(k))$, and $\\Xi(k+1)=g_e(\\Xi(k),x(k))$, enforcing safety through a constrained set $\\mathcal{S}_{f,x}$. The authors propose an MPC-based solution that ensures recursive feasibility and stability by solving a finite-horizon optimization with horizon $N_h$ and a terminal cost $\\bar{J}(s_{N_h})$ derived from a baseline policy, then applying the first action and re-solving at every $T_h$. The framework is demonstrated on autonomous lane changing, showing safer and more adaptable decisions in dynamic environments compared to rule-based baselines, and offering a path toward scalable, safer autonomous decision-making in real-world settings.

Abstract

This paper proposes a comprehensive hierarchical control framework for autonomous decision-making arising in robotics and autonomous systems. In a typical hierarchical control architecture, high-level decision making is often characterised by discrete state and decision/control sets. However, a rational decision is usually affected by not only the discrete states of the autonomous system, but also the underlying continuous dynamics even the evolution of its operational environment. This paper proposes a holistic and comprehensive design process and framework for this type of challenging problems, from new modelling and design problem formulation to control design and stability analysis. It addresses the intricate interplay between traditional continuous systems dynamics utilized at the low levels for control design and discrete Markov decision processes (MDP) for facilitating high-level decision making. We model the decision making system in complex environments as a hybrid system consisting of a controlled MDP and autonomous (i.e. uncontrolled) continuous dynamics. Consequently, the new formulation is called as hybrid Markov decision process (HMDP). The design problem is formulated with a focus on ensuring both safety and optimality while taking into account the influence of both the discrete and continuous state variables of different levels. With the help of the model predictive control (MPC) concept, a decision maker design scheme is proposed for the proposed hybrid decision making model. By carefully designing key ingredients involved in this scheme, it is shown that the recursive feasibility and stability of the proposed autonomous decision making scheme are guaranteed. The proposed framework is applied to develop an autonomous lane changing system for intelligent vehicles.

Figures (14)

• Figure 1: Hierarchical autonomous control architecture that may consist of several levels.
• Figure 2: Wait behind the parked cars to give way to oncoming vehicles (Blue one is the ego vehicle).
• Figure 3: Overtaking car collision with oncoming car (Blue one is the ego vehicle).
• Figure 4: Abandon overtaking to give way to the sudden-emerging vehicle (Blue one is the ego vehicle).
• Figure 5: Hybrid decision-making systems where $s^*$ is the high-level optimal operational mode transmitted to the low level.

Theorems & Definitions (8)

• Remark 1
• Remark 2
• Remark 3
• Definition 1
• Theorem 1
• Theorem 2
• Remark 4
• Remark 5