Hierarchical Control for Head-to-Head Autonomous Racing

TL;DR

The paper addresses the challenge of safe and fair head-to-head autonomous racing by encoding nuanced racing rules into a hierarchical planning framework. A high-level tactical planner creates long-horizon, discrete waypoint strategies via Monte Carlo Tree Search, while a low-level path planner tracks these waypoints using either a multi-agent reinforcement learning approach or a linear-quadratic Nash game, operating at high frequency. The proposed architecture outperforms baselines including end-to-end learning and fixed-trajectory controllers in head-to-head races and shows human-like strategies such as shielding and delayed overtakes, with MARL-based low-level control achieving the best overall performance (~90% wins). This work demonstrates that decoupling long-horizon game-theoretic reasoning from high-resolution control yields robust, rule-abiding, and competitive autonomous racing behavior with potential applications to other multi-agent systems with complex constraints.

Abstract

We develop a hierarchical controller for head-to-head autonomous racing. We first introduce a formulation of a racing game with realistic safety and fairness rules. A high-level planner approximates the original formulation as a discrete game with simplified state, control, and dynamics to easily encode the complex safety and fairness rules and calculates a series of target waypoints. The low-level controller takes the resulting waypoints as a reference trajectory and computes high-resolution control inputs by solving an alternative formulation approximation with simplified objectives and constraints. We consider two approaches for the low-level planner, constructing two hierarchical controllers. One approach uses multi-agent reinforcement learning (MARL), and the other solves a linear-quadratic Nash game (LQNG) to produce control inputs. The controllers are compared against three baselines: an end-to-end MARL controller, a MARL controller tracking a fixed racing line, and an LQNG controller tracking a fixed racing line. Quantitative results show that the proposed hierarchical methods outperform their respective baseline methods in terms of head-to-head race wins and abiding by the rules. The hierarchical controller using MARL for low-level control consistently outperformed all other methods by winning over 90% of head-to-head races and more consistently adhered to the complex racing rules. Qualitatively, we observe the proposed controllers mimicking actions performed by expert human drivers such as shielding/blocking, overtaking, and long-term planning for delayed advantages. We show that hierarchical planning for game-theoretic reasoning produces competitive behavior even when challenged with complex rules and constraints.

Figures (9)

• Figure 1: Two-level planning architecture of the proposed racing controller.
• Figure 2: The uncountably infinite trajectories of the general game (left) discretized by the high-level planner (middle). The sequence of target waypoints calculated by the high-level planner (in green) is tracked by the low-level planner (right) and converges to a continuous trajectory (in black).
• Figure 3: An example of a player's state in the original game (left) with a mix of discrete and continuous elements converted into our high-level tactical planner state approximation (right). The position is converted into a lane ID and checkpoint index. Velocity and tire wear are projected into ranges of some fixed size. The time step is reduced to a finite precision time state in the discrete game; in this example, it is tenths of a second. The recent lane changes state variable remains unchanged because it is inherently discrete.
• Figure 4: Three steps of an example trajectory for an individual player in the high-level tactical planning model. Components of the state space are discretized or rounded to finite precision. Players choose actions at each checkpoint. Using this model allows us to rapidly simulate long-term plans and evaluate strategies using MCTS.
• Figure 5: Kart racing environment from a racer's perspective (left), a bird's eye view of the oval track (right-top), and the complex track (right-bottom) in the Unity environment. The purple boxes visualize the lanes across checkpoints along the track, and the highlighted green boxes show planned waypoints determined by MCTS-RL's high-level tactical planner.