Motion-to-Response Content Generation via Multi-Agent AI System with Real-Time Safety Verification

TL;DR

This work tackles turning audio-derived emotional signals into safe, age-appropriate, real-time content by introducing a four-agent pipeline—Emotion Recognition, Response Policy Decision, Content Parameter Generation, and Safety Verification—coupled with a safety loop that regenerates content until rules are satisfied. The system is designed for on-device deployment, achieving sub-100 ms latency while providing interpretability and modularity through explicit agent interfaces and a rule-based policy. Experimental results show competitive emotion-recognition accuracy (approximately 73% on standard benchmarks), high response-mode consistency (≈89%), and perfect safety compliance (100%), with robust performance across devices from smartphones to embedded boards. This architecture enables applications in child-oriented media, therapeutic settings, and emotionally responsive smart devices, offering a practical path toward trustworthy, privacy-preserving affect-aware content generation.

Abstract

This paper proposes a multi-agent artificial intelligence system that generates response-oriented media content in real time based on audio-derived emotional signals. Unlike conventional speech emotion recognition studies that focus primarily on classification accuracy, our approach emphasizes the transformation of inferred emotional states into safe, age-appropriate, and controllable response content through a structured pipeline of specialized AI agents. The proposed system comprises four cooperative agents: (1) an Emotion Recognition Agent with CNN-based acoustic feature extraction, (2) a Response Policy Decision Agent for mapping emotions to response modes, (3) a Content Parameter Generation Agent for producing media control parameters, and (4) a Safety Verification Agent enforcing age-appropriateness and stimulation constraints. We introduce an explicit safety verification loop that filters generated content before output, ensuring compliance with predefined rules. Experimental results on public datasets demonstrate that the system achieves 73.2% emotion recognition accuracy, 89.4% response mode consistency, and 100% safety compliance while maintaining sub-100ms inference latency suitable for on-device deployment. The modular architecture enables interpretability and extensibility, making it applicable to child-adjacent media, therapeutic applications, and emotionally responsive smart devices.

Figures (5)

• Figure 1: Overall architecture of the proposed multi-agent system for emotion-to-response content generation. The system comprises four specialized agents operating sequentially: Emotion Recognition Agent, Response Policy Decision Agent, Content Parameter Generation Agent, and Safety Verification Agent. Failed safety verification triggers content regeneration.
• Figure 2: Processing flowchart of the emotion-to-response content generation method. The system processes audio input through sequential stages (S10-S60) with a safety verification loop that triggers content regeneration upon failure.
• Figure 3: Response mode mapping policy. The system maps recognized emotion states to discrete response modes, which then determine content parameters across audio, visual, and interaction modalities.
• Figure 4: On-device deployment configurations. The multi-agent system can be embedded in smartphones, smart devices , and embedded systems with varying computational resources.
• Figure 5: Experimental results. (a) Emotion recognition accuracy across different datasets. (b) Inference latency on various hardware platforms, all meeting the 100ms real-time threshold. (c) Ablation study showing the contribution of policy and safety agents to response consistency and safety compliance.