Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Prompt Injection Detection and Mitigation via AI Multi-Agent NLP Frameworks

Diego Gosmar, Deborah A. Dahl, Dario Gosmar

TL;DR

This work tackles prompt injection in generative AI by introducing a layered, multi-agent NLP framework that uses OVON-based inter-agent communication. The architecture distributes responsibilities across a Front-End Generator, Guard/Sanitizer, Policy Enforcer, and KPI Evaluation Agent, all built on open-weight Meta Llama models and coordinated via structured JSON messages. It introduces four injection-specific KPIs—$ISR$, $POF$, $PSR$, and $CCS$—and aggregates them into the Total Injection Vulnerability Score ($TIVS$) to quantify resilience, demonstrated on 500 crafted prompts with notable reductions in vulnerability. Compared to prior hallucination-mitigation work, the results show robust improvements and illustrate how a layered, agentic approach can bolster AI safety across different adversarial challenges while highlighting areas for future enhancement such as broader model variety and extended metadata exchange.

Abstract

Prompt injection constitutes a significant challenge for generative AI systems by inducing unintended outputs. We introduce a multi-agent NLP framework specifically designed to address prompt injection vulnerabilities through layered detection and enforcement mechanisms. The framework orchestrates specialized agents for generating responses, sanitizing outputs, and enforcing policy compliance. Evaluation on 500 engineered injection prompts demonstrates a marked reduction in injection success and policy breaches. Novel metrics, including Injection Success Rate (ISR), Policy Override Frequency (POF), Prompt Sanitization Rate (PSR), and Compliance Consistency Score (CCS), are proposed to derive a composite Total Injection Vulnerability Score (TIVS). The system utilizes the OVON (Open Voice Network) framework for inter-agent communication via structured JSON messages, extending a previously established multi-agent architecture from hallucination mitigation to address the unique challenges of prompt injection.

Prompt Injection Detection and Mitigation via AI Multi-Agent NLP Frameworks

TL;DR

This work tackles prompt injection in generative AI by introducing a layered, multi-agent NLP framework that uses OVON-based inter-agent communication. The architecture distributes responsibilities across a Front-End Generator, Guard/Sanitizer, Policy Enforcer, and KPI Evaluation Agent, all built on open-weight Meta Llama models and coordinated via structured JSON messages. It introduces four injection-specific KPIs—, , , and —and aggregates them into the Total Injection Vulnerability Score () to quantify resilience, demonstrated on 500 crafted prompts with notable reductions in vulnerability. Compared to prior hallucination-mitigation work, the results show robust improvements and illustrate how a layered, agentic approach can bolster AI safety across different adversarial challenges while highlighting areas for future enhancement such as broader model variety and extended metadata exchange.

Abstract

Prompt injection constitutes a significant challenge for generative AI systems by inducing unintended outputs. We introduce a multi-agent NLP framework specifically designed to address prompt injection vulnerabilities through layered detection and enforcement mechanisms. The framework orchestrates specialized agents for generating responses, sanitizing outputs, and enforcing policy compliance. Evaluation on 500 engineered injection prompts demonstrates a marked reduction in injection success and policy breaches. Novel metrics, including Injection Success Rate (ISR), Policy Override Frequency (POF), Prompt Sanitization Rate (PSR), and Compliance Consistency Score (CCS), are proposed to derive a composite Total Injection Vulnerability Score (TIVS). The system utilizes the OVON (Open Voice Network) framework for inter-agent communication via structured JSON messages, extending a previously established multi-agent architecture from hallucination mitigation to address the unique challenges of prompt injection.

Paper Structure

This paper contains 29 sections, 5 equations, 9 figures, 3 tables.

Table of Contents

  1. Organization of the Paper
  2. Introduction
  3. Crafting Injection Prompts
  4. Defense strategies against prompt injection
  5. Multi-Agent Architecture and OVON Integration
  6. Role of OVON in Inter-Agent Communication
  7. Experimental Design and Methods
  8. Multi-Agent Framework Implementation
  9. System Prompts for Agent Customization
  10. Front-End Generator (Agent 1)
  11. Guard/Sanitizer (Agent 2)
  12. Policy Enforcer (Agent 3)
  13. KPI Evaluation Agent (Agent 4)
  14. Injection Vulnerability KPIs
  15. Empirical Results
  16. ...and 14 more sections

Figures (9)

  • Figure 1: Multi-Agent Architecture for evaluating Prompt Injection Detection and Mitigation
  • Figure 2: Grouped bar chart showing the average KPI and TIVS values by agent.
  • Figure 3: TIVS across 500 prompts processed by the 3-agent system
  • Figure 4: Total Injection Vulnerability Score (TIVS) transition across three agent levels
  • Figure 5: Delta TIVS showing injection mitigation per prompt
  • ...and 4 more figures