Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Language Model Agents Under Attack: A Cross Model-Benchmark of Profit-Seeking Behaviors in Customer Service

Jingyu Zhang

TL;DR

This paper tackles the deployment-relevant risk of profit-seeking direct prompt injection in customer-service LLM agents by introducing a cross-model benchmark covering 10 service domains, 100 injected conversations, and five PI families. It evaluates five widely used models with an end-to-end pipeline that includes dual LLM evaluators and uncertainty reporting, revealing strong domain- and model-level variability in vulnerability. Payload splitting (PI3) emerges as the most consistently effective PI family, while airline-supply domains, particularly Airline Support, show the greatest susceptibility. The authors provide a complete dataset and tooling to enable auditing, defenses, and safer, human-centered agent interfaces, emphasizing practical guardrails, escalation, and transparency as core defenses.

Abstract

Customer-service LLM agents increasingly make policy-bound decisions (refunds, rebooking, billing disputes), but the same ``helpful'' interaction style can be exploited: a small fraction of users can induce unauthorized concessions, shifting costs to others and eroding trust in agentic workflows. We present a cross-domain benchmark of profit-seeking direct prompt injection in customer-service interactions, spanning 10 service domains and 100 realistic attack scripts grouped into five technique families. Across five widely used models under a unified rubric with uncertainty reporting, attacks are highly domain-dependent (airline support is most exploitable) and technique-dependent (payload splitting is most consistently effective). We release data and evaluation code to support reproducible auditing and to inform the design of oversight and recovery workflows for trustworthy, human centered agent interfaces.

Language Model Agents Under Attack: A Cross Model-Benchmark of Profit-Seeking Behaviors in Customer Service

TL;DR

This paper tackles the deployment-relevant risk of profit-seeking direct prompt injection in customer-service LLM agents by introducing a cross-model benchmark covering 10 service domains, 100 injected conversations, and five PI families. It evaluates five widely used models with an end-to-end pipeline that includes dual LLM evaluators and uncertainty reporting, revealing strong domain- and model-level variability in vulnerability. Payload splitting (PI3) emerges as the most consistently effective PI family, while airline-supply domains, particularly Airline Support, show the greatest susceptibility. The authors provide a complete dataset and tooling to enable auditing, defenses, and safer, human-centered agent interfaces, emphasizing practical guardrails, escalation, and transparency as core defenses.

Abstract

Customer-service LLM agents increasingly make policy-bound decisions (refunds, rebooking, billing disputes), but the same ``helpful'' interaction style can be exploited: a small fraction of users can induce unauthorized concessions, shifting costs to others and eroding trust in agentic workflows. We present a cross-domain benchmark of profit-seeking direct prompt injection in customer-service interactions, spanning 10 service domains and 100 realistic attack scripts grouped into five technique families. Across five widely used models under a unified rubric with uncertainty reporting, attacks are highly domain-dependent (airline support is most exploitable) and technique-dependent (payload splitting is most consistently effective). We release data and evaluation code to support reproducible auditing and to inform the design of oversight and recovery workflows for trustworthy, human centered agent interfaces.
Paper Structure (50 sections, 5 equations, 9 figures)

This paper contains 50 sections, 5 equations, 9 figures.

Table of Contents

  1. Introduction
  2. Literature Review
  3. Agentic AI in customer-service settings
  4. Prompt injection methods and evaluations
  5. Benchmarks for agent behavior in managerial settings
  6. Experimental Setting
  7. Benchmark construction
  8. Attack taxonomy and scope
  9. Target models and response generation
  10. LLM-as-evaluator protocol
  11. Data artifacts and canonical representation
  12. Outcome definitions and uncertainty reporting
  13. Methodology
  14. Task formulation
  15. Attack and prompting protocol
  16. ...and 35 more sections

Figures (9)

  • Figure 1: End-to-end benchmark pipeline for profit-seeking direct prompt injection in customer-service scenarios: construction, first-turn generation, dual-evaluator scoring, and aggregation into analysis-ready metrics.
  • Figure 2: Prompt injection taxonomy
  • Figure 3: Scenario ranking by prompt-injection success rate (score $\geq 4$), with 95% bootstrap confidence intervals.
  • Figure 4: Success-rate difference vs. Airline Support.
  • Figure 5: Adjusted prompt-injection success probability by model
  • ...and 4 more figures