Safety Control of Service Robots with LLMs and Embodied Knowledge Graphs

TL;DR

This work presents an integrated safety framework for service robots by coupling Embodied Robotic Control Prompts (ERCPs) with Embodied Knowledge Graphs (EKGs) and Large Language Models (LLMs). ERCPs refine human instructions into executable, safe task plans, while EKGs provide real-time grounding and safety validation through Graph Attention Networks and Hamiltonian Paths to enforce prioritization and sequencing. The approach is formalized within an ERCP–GK–MDP architecture, where a real-time loop updates the knowledge graph, revises plans, and validates actions against force, temperature, collision, stability, and temporal constraints. Experiments on two robotic platforms show substantial improvements in ambiguity resolution, safety compliance, and task success, with human evaluators rating high usability and safety. Overall, the framework offers a scalable, safety-focused pathway toward robust human–robot collaboration in dynamic, real-world environments.

Abstract

Safety limitations in service robotics across various industries have raised significant concerns about the need for robust mechanisms ensuring that robots adhere to safe practices, thereby preventing actions that might harm humans or cause property damage. Despite advances, including the integration of Knowledge Graphs (KGs) with Large Language Models (LLMs), challenges in ensuring consistent safety in autonomous robot actions persist. In this paper, we propose a novel integration of Large Language Models with Embodied Robotic Control Prompts (ERCPs) and Embodied Knowledge Graphs (EKGs) to enhance the safety framework for service robots. ERCPs are designed as predefined instructions that ensure LLMs generate safe and precise responses. These responses are subsequently validated by EKGs, which provide a comprehensive knowledge base ensuring that the actions of the robot are continuously aligned with safety protocols, thereby promoting safer operational practices in varied contexts. Our experimental setup involved diverse real-world tasks, where robots equipped with our framework demonstrated significantly higher compliance with safety standards compared to traditional methods. This integration fosters secure human-robot interactions and positions our methodology at the forefront of AI-driven safety innovations in service robotics.

Figures (10)

• Figure 1: Problem Statement: The diagram highlights challenges in service robot safety—ambiguities in natural language commands, conflicts between user commands and environmental constraints, and perception errors.
• Figure 2: Integrated Framework for Service Robot Safety and Control: The diagram illustrates the interaction between the Embodied Robotic Control Prompt (ERCP), Large Language Model (LLM), Embodied Knowledge Graph (EKG), Graph Attention Networks (GATs), and Hamiltonian Paths. The ERCP refines ambiguous user instructions, the LLM generates task plans, the EKG validates environmental and object constraints, GATs prioritize safety-critical nodes, and Hamiltonian Paths ensure the correct sequence of actions for safe and efficient task execution.
• Figure 3: ERCP workflow starts with a user command, followed by the LLM analyzing for ambiguities. If ambiguities exist, the LLM generates clarification prompts, leading to user responses. This loop continues until the instruction is clear, culminating in a detailed task plan executed by the robot.
• Figure 4: Embodied Knowledge Graph (EKG): A spatial and relational map of an indoor environment, showing entities (e.g., locations, objects, robot), their relationships (e.g., on_top_of, next_to), and real-time attributes (e.g., robot position, battery status). The EKG integrates sensor data (e.g., /scan, /camera/mage_raw) to dynamically represent the environment, enabling safe and efficient task execution.
• Figure 5: General Attention Network (GAT) for Task Pick up the glass cup