When to Localize? A Risk-Constrained Reinforcement Learning Approach

TL;DR

This work addresses when a robot should localize to balance safety and efficiency under uncertainty. It introduces RiskRL, a constrained reinforcement learning framework that uses a particle filter for belief maintenance and a recurrent Soft Actor-Critic policy to minimize localization actions while enforcing a probabilistic failure bound via a cumulative risk constraint. RiskRL outperforms baselines including CC-POMCP and a risk-unaware RL in unseen environments by at least 26% in success rate, and it achieves real-time inference suitable for deployment, while maintaining robustness to transition and observation noise and enabling zero-shot transfer. The approach advances active localization by handling partial observability and chance constraints, with practical implications for underwater robotics and other resource-constrained navigation tasks.

Abstract

In a standard navigation pipeline, a robot localizes at every time step to lower navigational errors. However, in some scenarios, a robot needs to selectively localize when it is expensive to obtain observations. For example, an underwater robot surfacing to localize too often hinders it from searching for critical items underwater, such as black boxes from crashed aircraft. On the other hand, if the robot never localizes, poor state estimates cause failure to find the items due to inadvertently leaving the search area or entering hazardous, restricted areas. Motivated by these scenarios, we investigate approaches to help a robot determine "when to localize?" We formulate this as a bi-criteria optimization problem: minimize the number of localization actions while ensuring the probability of failure (due to collision or not reaching a desired goal) remains bounded. In recent work, we showed how to formulate this active localization problem as a constrained Partially Observable Markov Decision Process (POMDP), which was solved using an online POMDP solver. However, this approach is too slow and requires full knowledge of the robot transition and observation models. In this paper, we present RiskRL, a constrained Reinforcement Learning (RL) framework that overcomes these limitations. RiskRL uses particle filtering and recurrent Soft Actor-Critic network to learn a policy that minimizes the number of localizations while ensuring the probability of failure constraint is met. Our numerical experiments show that RiskRL learns a robust policy that leads to at least a 26% increase in success rates when traversing unseen test environments.

Figures (11)

• Figure 1: Motivating example. Consider a robot that may want to seldom localize (e.g., due to resource constraints) while traveling along a path (black dashed line). Despite obstacles (rocks and water), the autonomous robot can execute a series of open-loop motions for some period. However, as the dead reckoning uncertainty (gold and red ellipses) grows, the probability of failure (such as collision) may become too large. As such the robot must localize at some point to avoid failures. Thus, our question is: When should the robot localize to reduce failure probabilities?
• Figure 2: This flowchart depicts a decision-making process for a robot interacting with an environment. It combines a high-level (Soft Actor-Critic) RL planner for decision-making (Section \ref{['sec:method:highlevel_planner']}) and a Particle Filter for state estimation.
• Figure 3: A diagram of the RL architecture used to train the models
• Figure 4: This figure shows our three evaluation environments in Minigrid. A red triangle denotes the agent, the orange squares denote the failure states (obstacles), and the sample paths are denoted in blue with the green squares representing the goal. We trained the RL agents using train.
• Figure 5: A qualitative example of robot navigation: blue) actual path, red) estimated path and localization positions.