CRLLK: Constrained Reinforcement Learning for Lane Keeping in Autonomous Driving

TL;DR

The paper addresses lane-keeping in autonomous driving, where multiple objectives must be balanced and fixed-weight RL struggles to generalize across scenarios. It proposes a constrained reinforcement learning framework using Lagrangian relaxation to learn adaptive weight coefficients $\lambda_1$ and $\lambda_2$, forming a modified reward $\hat{r} = r - \lambda_1 c_{\mathrm{lane}} - \lambda_2 c_{\mathrm{coll}}$ and updating the policy parameters $\theta$ in a one-timescale scheme. The key contributions are the automatic weight learning that removes extensive scenario-specific tuning, and empirical results showing improved travel distance $J_R$ with reduced lane deviation $J_{c_{\mathrm{lane}}}$ and collisions $J_{c_{\mathrm{coll}}}$ in simulation, plus successful real-world demonstrations on Duckiebots. This work enables practical, real-time constrained lane keeping with sim-to-real transfer, offering a scalable approach for multi-objective autonomous driving control.

Abstract

Lane keeping in autonomous driving systems requires scenario-specific weight tuning for different objectives. We formulate lane-keeping as a constrained reinforcement learning problem, where weight coefficients are automatically learned along with the policy, eliminating the need for scenario-specific tuning. Empirically, our approach outperforms traditional RL in efficiency and reliability. Additionally, real-world demonstrations validate its practical value for real-world autonomous driving.

Figures (2)

• Figure 1: (a-c) Learning curves for different constraint threshold values ($\alpha_1$). (d) Action distribution for one test episode.
• Figure 2: Real-world evaluation in scenarios.