Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Contextual Safety Reasoning and Grounding for Open-World Robots

Zachary Ravichadran, David Snyder, Alexander Robey, Hamed Hassani, Vijay Kumar, George J. Pappas

TL;DR

This work provides probabilistic safety guarantees for CORE that account for perceptual uncertainty, and demonstrates that CORE enforces contextually appropriate behavior in unseen environments, significantly outperforming prior semantic safety methods that lack online contextual reasoning.

Abstract

Robots are increasingly operating in open-world environments where safe behavior depends on context: the same hallway may require different navigation strategies when crowded versus empty, or during an emergency versus normal operations. Traditional safety approaches enforce fixed constraints in user-specified contexts, limiting their ability to handle the open-ended contextual variability of real-world deployment. We address this gap via CORE, a safety framework that enables online contextual reasoning, grounding, and enforcement without prior knowledge of the environment (e.g., maps or safety specifications). CORE uses a vision-language model (VLM) to continuously reason about context-dependent safety rules directly from visual observations, grounds these rules in the physical environment, and enforces the resulting spatially-defined safe sets via control barrier functions. We provide probabilistic safety guarantees for CORE that account for perceptual uncertainty, and we demonstrate through simulation and real-world experiments that CORE enforces contextually appropriate behavior in unseen environments, significantly outperforming prior semantic safety methods that lack online contextual reasoning. Ablation studies validate our theoretical guarantees and underscore the importance of both VLM-based reasoning and spatial grounding for enforcing contextual safety in novel settings. We provide additional resources at https://zacravichandran.github.io/CORE.

Contextual Safety Reasoning and Grounding for Open-World Robots

TL;DR

This work provides probabilistic safety guarantees for CORE that account for perceptual uncertainty, and demonstrates that CORE enforces contextually appropriate behavior in unseen environments, significantly outperforming prior semantic safety methods that lack online contextual reasoning.

Abstract

Robots are increasingly operating in open-world environments where safe behavior depends on context: the same hallway may require different navigation strategies when crowded versus empty, or during an emergency versus normal operations. Traditional safety approaches enforce fixed constraints in user-specified contexts, limiting their ability to handle the open-ended contextual variability of real-world deployment. We address this gap via CORE, a safety framework that enables online contextual reasoning, grounding, and enforcement without prior knowledge of the environment (e.g., maps or safety specifications). CORE uses a vision-language model (VLM) to continuously reason about context-dependent safety rules directly from visual observations, grounds these rules in the physical environment, and enforces the resulting spatially-defined safe sets via control barrier functions. We provide probabilistic safety guarantees for CORE that account for perceptual uncertainty, and we demonstrate through simulation and real-world experiments that CORE enforces contextually appropriate behavior in unseen environments, significantly outperforming prior semantic safety methods that lack online contextual reasoning. Ablation studies validate our theoretical guarantees and underscore the importance of both VLM-based reasoning and spatial grounding for enforcing contextual safety in novel settings. We provide additional resources at https://zacravichandran.github.io/CORE.
Paper Structure (31 sections, 1 theorem, 25 equations, 8 figures, 6 tables)

This paper contains 31 sections, 1 theorem, 25 equations, 8 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Robot Safety Filters
  4. Contextual Robot Safety
  5. VLMs for Robot Context Analysis
  6. Problem formulation
  7. The CORE Architecture
  8. Contextual Safety Reasoning
  9. Semantic Grounding
  10. Contextual Safety Enforcement
  11. Accounting for Perception Uncertainty
  12. Experimental Results
  13. Simulation Results
  14. VLM Safety Reasoning Comparison
  15. Structured Reasoning Ablation
  16. ...and 16 more sections

Key Result

Theorem 1

Assume a measurement function $\underline{m}(\cdot)$ satisfying the conditions given in Section the_perception_uncertainty. Further, assume an initial safe radius $R > 0$ and consider a safety-filtered system which gathers, at each time $t$, $k^*(t)$ observations such that: Then, $\mathbb{P} [\exists t : x_t \notin \mathcal{S} ] \leq \delta$.

Figures (8)

  • Figure 1: CORE enforces contextual safety via a three-stage process of contextual safety reasoning, semantic grounding, and safe control synthesis, enabling it to operate in open-world environments.
  • Figure 2: The CORE Framework enforces contextual safety via three modules. First, a VLM provides context-dependent safety constraints from visual observations. Constraints are grounded into spatially-defined safe sets, which are used for CBF-based control synthesis.
  • Figure 3: Illustration of CORE's contextual reasoning and grounding process. Given an observation (A), the VLM predicts contextual safety constraints (B). CORE then segments the image (C) and constructs a image space safe set (D), which is integrated into a barrier function (E).
  • Figure 4: Simulation Environments and platforms used for experiments.
  • Figure 5: Example safe and unsafe tasks. Top: the nominal controller attempts to guide the robot into an area prohibited by cones. Bottom: the nominal controller attempts to guide the robot through a space constrained aisle.
  • ...and 3 more figures

Theorems & Definitions (3)

  • Theorem 1: Probabilistic Trajectory-Length Safety
  • proof
  • proof : Proof of Theorem \ref{['the_probabilistic_cbf_theorem']}