Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

From Prompt to Physical Action: Structured Backdoor Attacks on LLM-Mediated Robotic Control Systems

Mingyang Xie, Jin Wei-Kocsis

Abstract

The integration of large language models (LLMs) into robotic control pipelines enables natural language interfaces that translate user prompts into executable commands. However, this digital-to-physical interface introduces a critical and underexplored vulnerability: structured backdoor attacks embedded during fine-tuning. In this work, we experimentally investigate LoRA-based supply-chain backdoors in LLM-mediated ROS2 robotic control systems and evaluate their impact on physical robot execution. We construct two poisoned fine-tuning strategies targeting different stages of the command generation pipeline and reveal a key systems-level insight: back-doors embedded at the natural-language reasoning stage do not reliably propagate to executable control outputs, whereas backdoors aligned directly with structured JSON command formats successfully survive translation and trigger physical actions. In both simulation and real-world experiments, backdoored models achieve an average Attack Success Rate of 83% while maintaining over 93% Clean Performance Accuracy (CPA) and sub-second latency, demonstrating both reliability and stealth. We further implement an agentic verification defense using a secondary LLM for semantic consistency checking. Although this reduces the Attack Success Rate (ASR) to 20%, it increases end-to-end latency to 8-9 seconds, exposing a significant security-responsiveness trade-off in real-time robotic systems. These results highlight structural vulnerabilities in LLM-mediated robotic control architectures and underscore the need for robotics-aware defenses for embodied AI systems.

From Prompt to Physical Action: Structured Backdoor Attacks on LLM-Mediated Robotic Control Systems

Abstract

The integration of large language models (LLMs) into robotic control pipelines enables natural language interfaces that translate user prompts into executable commands. However, this digital-to-physical interface introduces a critical and underexplored vulnerability: structured backdoor attacks embedded during fine-tuning. In this work, we experimentally investigate LoRA-based supply-chain backdoors in LLM-mediated ROS2 robotic control systems and evaluate their impact on physical robot execution. We construct two poisoned fine-tuning strategies targeting different stages of the command generation pipeline and reveal a key systems-level insight: back-doors embedded at the natural-language reasoning stage do not reliably propagate to executable control outputs, whereas backdoors aligned directly with structured JSON command formats successfully survive translation and trigger physical actions. In both simulation and real-world experiments, backdoored models achieve an average Attack Success Rate of 83% while maintaining over 93% Clean Performance Accuracy (CPA) and sub-second latency, demonstrating both reliability and stealth. We further implement an agentic verification defense using a secondary LLM for semantic consistency checking. Although this reduces the Attack Success Rate (ASR) to 20%, it increases end-to-end latency to 8-9 seconds, exposing a significant security-responsiveness trade-off in real-time robotic systems. These results highlight structural vulnerabilities in LLM-mediated robotic control architectures and underscore the need for robotics-aware defenses for embodied AI systems.

Paper Structure

This paper contains 17 sections, 5 figures.

Table of Contents

  1. Introduction
  2. Literature Review
  3. LLM-Robot Integration and Structured Control Interfaces
  4. Backdoor and Jailbreak Attacks in Embodied Systems
  5. ROS Security and LLM-Mediated Attack Surfaces
  6. Defense Mechanisms and Fine-Tuning Risks
  7. Methodology
  8. System Architecture
  9. Threat Model
  10. Backdoor Attack Implementation Strategy
  11. Agentic Semantic Verification Defense
  12. Experimental Implementation and Results
  13. Experimental Setup
  14. Backdoor Propagation Analysis
  15. Defense Mechanism Evaluation
  16. ...and 2 more sections

Figures (5)

  • Figure C1: System Architecture: ROS 2-based LLM-to-control integration pipeline translating natural language inputs into structured JSON commands for robotic actuation.
  • Figure C2: Threat Model: Phase 1 illustrates supply-chain compromise via data poisoning and LoRA fine-tuning. Phase 2 depicts runtime exploitation, where the backdoored adapter generates malicious structured JSON commands that propagate through the ROS 2 control pipeline to induce unintended physical actions.
  • Figure C3: Representative samples from the two poisoned fine-tuning datasets.
  • Figure C4: Agentic Semantic Verification Defense Mechanism: A secondary LLM verifies semantic alignment between the user/operator instruction and the structured JSON command before ROS 2 execution, enabling conditional publication or blocking of control commands.
  • Figure D1: ASR and CPA across evaluated LLMs under structured-output poisoning in the undefended configuration.