AegisAgent: An Autonomous Defense Agent Against Prompt Injection Attacks in LLM-HARs

TL;DR

This work addresses the vulnerability of LLM-based HAR systems to prompt injection by introducing AegisAgent, an autonomous defense agent that shifts from passive filtering to active protection. The system combines input sanitization, cross-modal consistency verification, and robust reasoning within a memory-driven, planning-execution loop to detect, correct, and verify defenses in real time. Across five state-of-the-art LLM-HAR pipelines and three public datasets, AegisAgent achieves high defense performance, strongly reducing attack success rates while incurring modest latency. The approach offers a flexible, training-free, and model-agnostic solution with practical implications for deploying secure and trustworthy LLM-HAR systems in safety-critical applications.

Abstract

The integration of Large Language Models (LLMs) into wearable sensing is creating a new class of mobile applications capable of nuanced human activity understanding. However, the reliability of these systems is critically undermined by their vulnerability to prompt injection attacks, where attackers deliberately input deceptive instructions into LLMs. Traditional defenses, based on static filters and rigid rules, are insufficient to address the semantic complexity of these new attacks. We argue that a paradigm shift is needed -- from passive filtering to active protection and autonomous reasoning. We introduce AegisAgent, an autonomous agent system designed to ensure the security of LLM-driven HAR systems. Instead of merely blocking threats, AegisAgent functions as a cognitive guardian. It autonomously perceives potential semantic inconsistencies, reasons about the user's true intent by consulting a dynamic memory of past interactions, and acts by generating and executing a multi-step verification and repair plan. We implement AegisAgent as a lightweight, full-stack prototype and conduct a systematic evaluation on 15 common attacks with five state-of-the-art LLM-based HAR systems on three public datasets. Results show it reduces attack success rate by 30\% on average while incurring only 78.6 ms of latency overhead on a GPU workstation. Our work makes the first step towards building secure and trustworthy LLM-driven HAR systems.

Figures (7)

• Figure 1: Overview of prompt injection threats in LLM-based HAR systems. Motion signals generated by wearable sensors are converted into text descriptions and integrated with user instruction prompts. Attackers inject adversarial manipulations in the sensor-to-prompt pathway, altering the generated descriptions and thereby misleading downstream LLM-based HAR models. This can lead to hazardous result in safety-critical applications, such as changing a fall result to be walking normally.
• Figure 2: Comparison of HAR classification accuracy before and after prompt injection attacks. Prompt injection attacks cause significant degradation, with accuracy dropping from 92.13%, 88.47%, and 85.26% to 52.67%, 47.92%, and 45.27%, respectively. Even with standard text defense measures (data cleaning, adversarial training, semantic filtering), accuracy is only partially recovered to 57.94%, 52.72%, and 49.79%, indicating traditional defenses remain insufficient against multimodal prompt injection threats.
• Figure 3: Impact of prompt injection on motion synthesis in the IMUGPT-2.0 pipeline. Adversarial prompts cause the motion generation module (T2M-GPT) within IMUGPT-2.0 to generate corrupted or misleading motion trajectories, which propagate into virtual IMU signals and degrade downstream classification accuracy.
• Figure 4: Multilayer attack paths within a LLM-HAR pipeline. The diagram highlights adversarial intervention points across signal path, text path, and prompt path. Such manipulations can divert the system from legitimate HAR objectives, degrade semantic fidelity, and ultimately compromise the accuracy of activity predictions.
• Figure 5: System overview of AegisAgent. Subfigure (a) illustrates the end-to-end process pipeline, while subfigure (b) expands the AegisAgent module, detailing how secure outputs are generated through Input Sanitization, Consistency Verifier, and Robust Reasoner.