Equilibrium of Feasible Zone and Uncertain Model in Safe Exploration

TL;DR

Safe exploration is reframed as achieving an equilibrium between the feasible zone and an uncertain environment, formalized through the pair $(Z,\hat{f})$ with $Z=Z^*(\hat{f})$ at equilibrium. The authors introduce Safe Equilibrium Exploration (SEE), an iterative method that alternates between computing the maximum feasible zone via a risky Bellman-based controller and deriving the least uncertain model through graph-based removals under Lipschitz continuity. They prove monotonic improvement properties and convergence to equilibrium, and they validate SEE on the double integrator, pendulum, and unicycle, achieving zero constraint violations and rapid convergence to the equilibrium. The framework provides a principled link between safety, model uncertainty, and exploration, offering practical guidance for real-world safe RL and robotics with potential extensions to larger-scale or continuous domains.

Abstract

Ensuring the safety of environmental exploration is a critical problem in reinforcement learning (RL). While limiting exploration to a feasible zone has become widely accepted as a way to ensure safety, key questions remain unresolved: what is the maximum feasible zone achievable through exploration, and how can it be identified? This paper, for the first time, answers these questions by revealing that the goal of safe exploration is to find the equilibrium between the feasible zone and the environment model. This conclusion is based on the understanding that these two components are interdependent: a larger feasible zone leads to a more accurate environment model, and a more accurate model, in turn, enables exploring a larger zone. We propose the first equilibrium-oriented safe exploration framework called safe equilibrium exploration (SEE), which alternates between finding the maximum feasible zone and the least uncertain model. Using a graph formulation of the uncertain model, we prove that the uncertain model obtained by SEE is monotonically refined, the feasible zones monotonically expand, and both converge to the equilibrium of safe exploration. Experiments on classic control tasks show that our algorithm successfully expands the feasible zones with zero constraint violation, and achieves the equilibrium of safe exploration within a few iterations.

Figures (11)

• Figure 1: Mechanism of SEE. Exploration starts from a small feasible zone obtained from a prior uncertain model. As more data are collected inside the feasible zone, the model becomes more accurate and the zone is expanded. This process is repeated until the equilibrium between the maximum feasible zone and the least uncertain model is reached.
• Figure 2: Feasible regions at different iterations on double integrator. The black line is the boundary of the maximum feasible region under the true model. The dark blue region stands for the feasible region in the previous iteration. The light blue region stands for the expanded part of the feasible region in the current iteration.
• Figure 3: Model uncertainty degree at different iterations on double integrator. The black line is the boundary of the feasible region in the previous iteration. The colors of the grids stand for values of uncertainty degree.
• Figure 4: Feasible zone at convergence on double integrator.
• Figure 5: Feasible zone recall versus overall Lipschitz constant $L$.

Theorems & Definitions (38)

• Definition 2.1: Uncertain model
• Definition 2.2: Well-calibration
• Definition 2.3: Zone
• Definition 2.4: Feasible zone
• Definition 2.5: Maximum feasible zone
• Definition 3.1: Expandability
• Theorem 3.1
• proof
• Corollary 3.1.1
• proof