Online Fine-Tuning of Pretrained Controllers for Autonomous Driving via Real-Time Recurrent RL

TL;DR

This work tackles the problem of deploying pretrained driving policies under distribution shifts by enabling online fine-tuning with Real-Time Recurrent Reinforcement Learning (RTRRL). It combines offline behavioral cloning with online RTRRL-based updates, leveraging bio-inspired Liquid-Resistance Liquid-Capacitance state-space models (LrcSSMs) to achieve stable, expressive recurrent dynamics. The approach is validated in a CarRacing simulator and on a real-world 1:10 RoboRacer using event-camera data, demonstrating that online adaptation can significantly improve performance and robustness, with LrcSSMs offering the strongest gains. This bio-inspired, high-frequency online learning pipeline on standard hardware broadens the practicality of learning-based controllers for autonomous driving under changing conditions and sensor modalities.

Abstract

Deploying pretrained policies in real-world applications presents substantial challenges that fundamentally limit the practical applicability of learning-based control systems. When autonomous systems encounter environmental changes in system dynamics, sensor drift, or task objectives, fixed policies rapidly degrade in performance. We show that employing Real-Time Recurrent Reinforcement Learning (RTRRL), a biologically plausible algorithm for online adaptation, can effectively fine-tune a pretrained policy to improve autonomous agents' performance on driving tasks. We further show that RTRRL synergizes with a recent biologically inspired recurrent network model, the Liquid-Resistance Liquid-Capacitance RNN. We demonstrate the effectiveness of this closed-loop approach in a simulated CarRacing environment and in a real-world line-following task with a RoboRacer car equipped with an event camera.

Figures (17)

• Figure 1: Overview of our proposed method and experiments. After collecting human control data in the environment, a policy is pretrained using behavioral cloning. The policy is then fine-tuned online using RTRRL. The gradients needed for optimization are computed with RTRL or RFLO for diagonalized or fully connected RNN models respectively.
• Figure 2: Model architecture. A: For pretraining, a CNN autoencoder is used to encode the image observation $o_t$; the encoded vector $x_t$ serves as input to the RNN policy predicting action distribution $\mathcal{N}(\hat{\mu}_t, \hat{\sigma}_t)$. B: The RNN policy is fine-tuned; to this end, a RNN value function is used that predicts the value estimate $\hat{v}_t$ from encoded vector $x_t$.
• Figure 3: RoboRacer car equipped with Sony/Prophesee IMX636 sensor for the real-world deployment of the proposed algorithm.
• Figure 4: RGB frame and the corresponding DVS event frame representation.
• Figure 5: Boxplots of evaluation reward on three different tracks for five different pretrained models, aggregated per type. Left shows rewards before fine-tuning – right after.