Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

StruPhantom: Evolutionary Injection Attacks on Black-Box Tabular Agents Powered by Large Language Models

Yang Feng, Xudong Pan

TL;DR

StruPhantom addresses the security vulnerability of black-box LLM-driven tabular agents to indirect prompt injection by embedding malicious instructions in structured data. It introduces a constrained Monte Carlo Tree Search–based optimization pipeline, guided by a shadow ReAct agent and an off-topic evaluator, to evolve attack templates that maximize attack success rates. Across CSV, XLSX, XML, and JSON formats—and on real platforms like Doubao and Coze—the optimized templates achieve markedly higher ASR than manual baselines, illustrating concrete security risks for tabular data pipelines. The work underscores the need for robust defenses, including strict input validation, interpretable auditing, and decoupled processing to mitigate IPI threats in industrial LLM-powered tabular systems.

Abstract

The proliferation of autonomous agents powered by large language models (LLMs) has revolutionized popular business applications dealing with tabular data, i.e., tabular agents. Although LLMs are observed to be vulnerable against prompt injection attacks from external data sources, tabular agents impose strict data formats and predefined rules on the attacker's payload, which are ineffective unless the agent navigates multiple layers of structural data to incorporate the payload. To address the challenge, we present a novel attack termed StruPhantom which specifically targets black-box LLM-powered tabular agents. Our attack designs an evolutionary optimization procedure which continually refines attack payloads via the proposed constrained Monte Carlo Tree Search augmented by an off-topic evaluator. StruPhantom helps systematically explore and exploit the weaknesses of target applications to achieve goal hijacking. Our evaluation validates the effectiveness of StruPhantom across various LLM-based agents, including those on real-world platforms, and attack scenarios. Our attack achieves over 50% higher success rates than baselines in enforcing the application's response to contain phishing links or malicious codes.

StruPhantom: Evolutionary Injection Attacks on Black-Box Tabular Agents Powered by Large Language Models

TL;DR

StruPhantom addresses the security vulnerability of black-box LLM-driven tabular agents to indirect prompt injection by embedding malicious instructions in structured data. It introduces a constrained Monte Carlo Tree Search–based optimization pipeline, guided by a shadow ReAct agent and an off-topic evaluator, to evolve attack templates that maximize attack success rates. Across CSV, XLSX, XML, and JSON formats—and on real platforms like Doubao and Coze—the optimized templates achieve markedly higher ASR than manual baselines, illustrating concrete security risks for tabular data pipelines. The work underscores the need for robust defenses, including strict input validation, interpretable auditing, and decoupled processing to mitigate IPI threats in industrial LLM-powered tabular systems.

Abstract

The proliferation of autonomous agents powered by large language models (LLMs) has revolutionized popular business applications dealing with tabular data, i.e., tabular agents. Although LLMs are observed to be vulnerable against prompt injection attacks from external data sources, tabular agents impose strict data formats and predefined rules on the attacker's payload, which are ineffective unless the agent navigates multiple layers of structural data to incorporate the payload. To address the challenge, we present a novel attack termed StruPhantom which specifically targets black-box LLM-powered tabular agents. Our attack designs an evolutionary optimization procedure which continually refines attack payloads via the proposed constrained Monte Carlo Tree Search augmented by an off-topic evaluator. StruPhantom helps systematically explore and exploit the weaknesses of target applications to achieve goal hijacking. Our evaluation validates the effectiveness of StruPhantom across various LLM-based agents, including those on real-world platforms, and attack scenarios. Our attack achieves over 50% higher success rates than baselines in enforcing the application's response to contain phishing links or malicious codes.

Paper Structure

This paper contains 19 sections, 1 equation, 5 figures, 2 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Background
  3. LLM Agents with Structural Inputs
  4. Prompt Injection Attacks in LLM Agents
  5. Threat Model
  6. StruPhantom
  7. Methodology Overview
  8. Selection Strategies
  9. Optimization Tactics
  10. Evaluation and Updating
  11. Evaluation and Results
  12. Experiment Setup
  13. Experiment Results
  14. Discussion
  15. Conclusion
  16. ...and 4 more sections

Figures (5)

  • Figure 1: Indirect prompt injection attacks on LLM-based agents with structural inputs (i.e., tabular agents).
  • Figure 2: A schematic diagram of the StruPhantom workflow.
  • Figure 3: Improvements in the attack success rate of different schemes over the optimization iterations.
  • Figure 4: Snapshots on a successful attack with Website template on a tabular agent application on ByteDance's Doubao platform (The application is crafted by the authors for ethical reasons).
  • Figure 5: Snapshots on a successful attack on a tabular agent application on ByteDance's Coze platform (The application is crafted by the authors for ethical reasons).