Zero-Shot Embedding Drift Detection: A Lightweight Defense Against Prompt Injections in LLMs

TL;DR

Prompt injection poses a critical risk to LLM safety, especially via indirect channels like email content. The authors introduce Zero-Shot Embedding Drift Detection (ZEDD), a lightweight, model-agnostic defense that detects injections by measuring semantic drift in embedding space between clean and suspect prompts, without requiring model access or retraining. The pipeline uses embedding extraction from multiple encoders, drift scoring with $Drift(x,x')=1-rac{f(x)\,f(x')}{\|f(x)\|\,\|f(x')\|}$, and ensemble flagging via Gaussian Mixture Modeling with KDE fallback, calibrated to maintain a low false-positive rate (~3%) while flagging about half of suspicious cases. Evaluated on the LLMail-Inject dataset across several architectures (e.g., Llama 3, Qwen 2, Mistral), ZEDD achieves over 93% accuracy and strong cross-model transfer, suggesting embedding drift is a robust, transferable signal for prompt-injection defense. Limitations include dependence on embedding quality and potential bypass strategies, with future work aimed at scalability, few-shot enhancements, and broader data coverage.

Abstract

Prompt injection attacks have become an increasing vulnerability for LLM applications, where adversarial prompts exploit indirect input channels such as emails or user-generated content to circumvent alignment safeguards and induce harmful or unintended outputs. Despite advances in alignment, even state-of-the-art LLMs remain broadly vulnerable to adversarial prompts, underscoring the urgent need for robust, productive, and generalizable detection mechanisms beyond inefficient, model-specific patches. In this work, we propose Zero-Shot Embedding Drift Detection (ZEDD), a lightweight, low-engineering-overhead framework that identifies both direct and indirect prompt injection attempts by quantifying semantic shifts in embedding space between benign and suspect inputs. ZEDD operates without requiring access to model internals, prior knowledge of attack types, or task-specific retraining, enabling efficient zero-shot deployment across diverse LLM architectures. Our method uses adversarial-clean prompt pairs and measures embedding drift via cosine similarity to capture subtle adversarial manipulations inherent to real-world injection attacks. To ensure robust evaluation, we assemble and re-annotate the comprehensive LLMail-Inject dataset spanning five injection categories derived from publicly available sources. Extensive experiments demonstrate that embedding drift is a robust and transferable signal, outperforming traditional methods in detection accuracy and operational efficiency. With greater than 93% accuracy in classifying prompt injections across model architectures like Llama 3, Qwen 2, and Mistral and a false positive rate of <3%, our approach offers a lightweight, scalable defense layer that integrates into existing LLM pipelines, addressing a critical gap in securing LLM-powered systems to withstand adaptive adversarial threats.

Figures (8)

• Figure 1: Overview of the ZEDD pipeline
• Figure 2: Result comparison from experiments proposed in ayub2024embedding and illustrates the effectiveness and strong performance of ZEDD in comparison with models from other experiments.
• Figure 3: The jailbreak attack was neutralized by removing the authority establishment markers (triple exclamation points) that attempted to create a false sense of legitimate administrative hierarchy and command structure. By stripping away these emphatic formatting elements that were designed to make the email request appear more authoritative and urgent, the cleaning process eliminated the attack's social engineering component that tried to pressure the model into compliance through artificial authority signals.
• Figure 4: The system leak injection attack was neutralized by removing fake context boundaries and session terminators designed to make the AI model believe it had exited user interaction mode, along with malicious XML commands attempting to trigger unauthorized email notifications. By removing these deceptive system-level instructions, the cleaning process prevented the attack from exploiting simulated administrative privileges to bypass security controls.
• Figure 5: The task override attack was neutralized by removing the deceptive newsletter format that attempted to disguise malicious email collection instructions as legitimate subscription management communications. By stripping away the fake unsubscribe mechanism that redirected users to send confirmation messages to unauthorized addresses, the cleaning process eliminated the attack's ability to hijack routine communication tasks for data harvesting purposes.