Learning Autonomous Race Driving with Action Mapping Reinforcement Learning

TL;DR

The paper tackles autonomous race driving under state-dependent tire-friction constraints and introduces a numerical action-mapping (AM) mechanism that converts unconstrained neural-policy actions into friction-safe real inputs. By integrating AM with TD3 (TD3-AM), the approach preserves the original optimization objective while guaranteeing constraint satisfaction, enabling the policy to exploit maximum tire grip. In simulations across two tracks, TD3-AM delivers up to a 22% reduction in best lap times and a 90% success rate, significantly outperforming conventional RL and safe-RL baselines, while demonstrating generalization to different friction conditions via adjustable friction bounds. This work provides a scalable method to enforce safety constraints in continuous-control RL for high-speed autonomous racing with potential real-world applicability under varying tire-road conditions.

Abstract

Autonomous race driving poses a complex control challenge as vehicles must be operated at the edge of their handling limits to reduce lap times while respecting physical and safety constraints. This paper presents a novel reinforcement learning (RL)-based approach, incorporating the action mapping (AM) mechanism to manage state-dependent input constraints arising from limited tire-road friction. A numerical approximation method is proposed to implement AM, addressing the complex dynamics associated with the friction constraints. The AM mechanism also allows the learned driving policy to be generalized to different friction conditions. Experimental results in our developed race simulator demonstrate that the proposed AM-RL approach achieves superior lap times and better success rates compared to the conventional RL-based approaches. The generalization capability of driving policy with AM is also validated in the experiments.

Figures (15)

• Figure 1: Single track vehicle model in earth-fixed frame and vehicle-fixed frame.
• Figure 2: Constraint of tire friction described by the friction circle
• Figure 3: Part of the states defined in our race driving MDP.
• Figure 4: Action mapping example at state: $v_x=15.4~\text{m/s}, \delta=7.9~\text{deg}$. The virtual action vector examples $a_{t1}$ and $a_{t2}$ and the boundary are shown on the left. The real control input examples $u_{t1}$ and $u_{t2}$ are shown on the right.
• Figure 5: Admissible control input space at $v_x=15.4~\text{m/s}$ with full range of steering angle.