Drive Fast, Learn Faster: On-Board RL for High Performance Autonomous Racing

TL;DR

The paper develops an on-board reinforcement learning framework for high-speed autonomous racing that avoids simulation pre-training by learning directly on the vehicle. It deploys a residual policy atop a baseline controller within a Soft Actor-Critic framework, enhanced with multi-step temporal-difference learning, asynchronous training, and a heuristic delayed reward adjustment mechanism. Empirical results on the F1TENTH platform show up to 11.5% improvement in minimum lap times (and substantial gains in mean performance) with roughly 20 minutes of training, and an End-to-End RL variant that also surpasses previous state-of-the-art baselines. The work demonstrates robust, efficient on-board learning suitable for real-time, dynamic autonomous systems and points to broader applicability beyond autonomous racing.

Abstract

Autonomous racing presents unique challenges due to its non-linear dynamics, the high speed involved, and the critical need for real-time decision-making under dynamic and unpredictable conditions. Most traditional Reinforcement Learning (RL) approaches rely on extensive simulation-based pre-training, which faces crucial challenges in transfer effectively to real-world environments. This paper introduces a robust on-board RL framework for autonomous racing, designed to eliminate the dependency on simulation-based pre-training enabling direct real-world adaptation. The proposed system introduces a refined Soft Actor-Critic (SAC) algorithm, leveraging a residual RL structure to enhance classical controllers in real-time by integrating multi-step Temporal-Difference (TD) learning, an asynchronous training pipeline, and Heuristic Delayed Reward Adjustment (HDRA) to improve sample efficiency and training stability. The framework is validated through extensive experiments on the F1TENTH racing platform, where the residual RL controller consistently outperforms the baseline controllers and achieves up to an 11.5 % reduction in lap times compared to the State-of-the-Art (SotA) with only 20 min of training. Additionally, an End-to-End (E2E) RL controller trained without a baseline controller surpasses the previous best results with sustained on-track learning. These findings position the framework as a robust solution for high-performance autonomous racing and a promising direction for other real-time, dynamic autonomous systems.

Figures (16)

• Figure 1: Overview of the proposed architecture, which processes observations $O_{car}$, providing proprioceptive information such as position and velocity $(v_x,\,v_y)$, and $O_{track}$, describing track boundaries and a given reference trajectory to generate a residual control command $u_{RL}$. Such a command is additively combined with the output of a conventional controller $u_{base}$. Since the agent starts untrained and may encounter unsafe states, a safety filter is employed to guide the robot back to a safe state. Training happens entirely on-board in an asynchronous fashion: the Acting node provides residual control commands at 10, while the training node updates the control policy at 32. The policy of the acting node is updated with the policy of the training node every second.
• Figure 2: Lap Time vs. Wall Time during training for different residual controllers, with respective reference for best lap time of the baseline controller and the controller with baseline. Only lap times without boundary violations are recorded.
• Figure 3: Comparison of the driven trajectories, speed profiles, and the time delta for (blue) vs. (orange) on the C-track with Turbo tires. Speed profiles and time deltas are represented along the positional advancement on the track. The zero progress position corresponds to the black arrow in the bottom left corner, and the track is driven in the sense of the arrow. Dark blue represents the fastest lap of .
• Figure 4: Stacked histogram of lap times achieved during zero-shot and few-shot transfer from the C-track to Y-track during 20 boundary-violation-free laps. Different shades represent the three different weight initialization and resulting policies learned on the C-track.
• Figure 5: Lap times during training averaged over 5 runs for ablation regarding multi-step over one (TD1) and three (TD3) steps, a synchronous (sync) and asynchronous (async) architecture as well as with and without. Lines only start when the first collision-free lap is completed. The area between the minimum and the maximum lap time is shaded.