Reinforcement Learning Goal-Reaching Control with Guaranteed Lyapunov-Like Stabilizer for Mobile Robots

TL;DR

This work addresses the lack of formal guarantees in reinforcement learning for goal-reaching by integrating a benchmark RL policy with a Lyapunov-like stabilizer, enabling formal convergence to goal sets for large wheeled mobile robots in unstructured, slip-prone environments. It introduces a real-time, acceleration-based action space and a 12-term shaped reward (with potential-based shaping) to drive smooth, safe behavior, while the stabilizer enforces a monotone decrease constraint on a learned critic and allows fallback to the baseline policy to guarantee safety. A key theoretical contribution is Theorem 5.1, which proves that the stabilizer preserves the goal-reaching guarantee with probability at least $1-\eta$ if the baseline policy does so, and the approach is validated on a 6000 kg WMR both numerically and experimentally, showing substantial improvements in goal-reaching rate, timeout reduction, and efficiency. Practically, this framework enables safe, real-time deployment of RL-based goal-reaching on large, off-road robots without requiring a known Lyapunov function, offering a scalable alternative to conservative shields and barrier-based safety layers.

Abstract

Reinforcement learning (RL) can be highly effective at learning goal-reaching policies, but it typically does not provide formal guarantees that the goal will always be reached. A common approach to provide formal goal-reaching guarantees is to introduce a shielding mechanism that restricts the agent to actions that satisfy predefined safety constraints. The main challenge here is integrating this mechanism with RL so that learning and exploration remain effective without becoming overly conservative. Hence, this paper proposes an RL-based control framework that provides formal goal-reaching guarantees for wheeled mobile robots operating in unstructured environments. We first design a real-time RL policy with a set of 15 carefully defined reward terms. These rewards encourage the robot to reach both static and dynamic goals while generating sufficiently smooth command signals that comply with predefined safety specifications, which is critical in practice. Second, a Lyapunov-like stabilizer layer is integrated into the benchmark RL framework as a policy supervisor to formally strengthen the goal-reaching control while preserving meaningful exploration of the state action space. The proposed framework is suitable for real-time deployment in challenging environments, as it provides a formal guarantee of convergence to the intended goal states and compensates for uncertainties by generating real-time control signals based on the current state, while respecting real-world motion constraints. The experimental results show that the proposed Lyapunov-like stabilizer consistently improves the benchmark RL policies, boosting the goal-reaching rate from 84.6% to 99.0%, sharply reducing failures, and improving efficiency.

Figures (15)

• Figure 1: Architecture of the benchmark RL policy $\pi^0$.
• Figure 2: Lyapunov-Like Stabilizer Agent Framework as the additional layer of the benchmark RL designed in Section 3.
• Figure 3: The proposed RL policy enhanced by the stabilizer agent within the whole experimental framework.
• Figure 4: Visitation and cost-to-go over the $\left(d_t, e_{\text{t}}\right)$ grid for the most frequently visited $(v, \omega)$ slice $(v \approx 0.025, \omega \approx 0)$.
• Figure 5: Comparison between the benchmark RL cost proxy (left) and stabilizer critic $\min _a \hat{Q}(s, a)$ (right).