Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

HoverAI: An Embodied Aerial Agent for Natural Human-Drone Interaction

Yuhua Jin, Nikita Kuzmin, Georgii Demianchuk, Mariya Lezina, Fawad Mehboob, Issatay Tokmurziyev, Miguel Altamirano Cabrera, Muhammad Ahsan Mustafa, Dzmitry Tsetserukou

TL;DR

HoverAI tackles the challenge of natural human-drone interaction by delivering a self-contained aerial agent that merges mobility, infrastructure-free visual projection, and real-time conversational AI. The system stacks onboard processing, an ultra-light MEMS projection module, and a multimodal pipeline combining VAD, ASR, LLM/RAG dialogue, and face-based avatar personalization to deliver lip-synced, demographic-adaptive avatars. Evaluation across 12 participants shows strong speech transcription (WER 0.181), command recognition (F1 0.90), and demographic estimation (gender F1 0.89; age MAE 5.14), with end-to-end latency around 950 ms. HoverAI enables spatially-aware guidance, assistance, and companionship in shared spaces, illustrating a new class of mobile, socially responsive embodied aerial agents. Future work will extend autonomy, knowledge bases, and multi-drone coordination to broaden applicability and robustness in real-world settings.

Abstract

Drones operating in human-occupied spaces suffer from insufficient communication mechanisms that create uncertainty about their intentions. We present HoverAI, an embodied aerial agent that integrates drone mobility, infrastructure-independent visual projection, and real-time conversational AI into a unified platform. Equipped with a MEMS laser projector, onboard semi-rigid screen, and RGB camera, HoverAI perceives users through vision and voice, responding via lip-synced avatars that adapt appearance to user demographics. The system employs a multimodal pipeline combining VAD, ASR (Whisper), LLM-based intent classification, RAG for dialogue, face analysis for personalization, and voice synthesis (XTTS v2). Evaluation demonstrates high accuracy in command recognition (F1: 0.90), demographic estimation (gender F1: 0.89, age MAE: 5.14 years), and speech transcription (WER: 0.181). By uniting aerial robotics with adaptive conversational AI and self-contained visual output, HoverAI introduces a new class of spatially-aware, socially responsive embodied agents for applications in guidance, assistance, and human-centered interaction.

HoverAI: An Embodied Aerial Agent for Natural Human-Drone Interaction

TL;DR

HoverAI tackles the challenge of natural human-drone interaction by delivering a self-contained aerial agent that merges mobility, infrastructure-free visual projection, and real-time conversational AI. The system stacks onboard processing, an ultra-light MEMS projection module, and a multimodal pipeline combining VAD, ASR, LLM/RAG dialogue, and face-based avatar personalization to deliver lip-synced, demographic-adaptive avatars. Evaluation across 12 participants shows strong speech transcription (WER 0.181), command recognition (F1 0.90), and demographic estimation (gender F1 0.89; age MAE 5.14), with end-to-end latency around 950 ms. HoverAI enables spatially-aware guidance, assistance, and companionship in shared spaces, illustrating a new class of mobile, socially responsive embodied aerial agents. Future work will extend autonomy, knowledge bases, and multi-drone coordination to broaden applicability and robustness in real-world settings.

Abstract

Drones operating in human-occupied spaces suffer from insufficient communication mechanisms that create uncertainty about their intentions. We present HoverAI, an embodied aerial agent that integrates drone mobility, infrastructure-independent visual projection, and real-time conversational AI into a unified platform. Equipped with a MEMS laser projector, onboard semi-rigid screen, and RGB camera, HoverAI perceives users through vision and voice, responding via lip-synced avatars that adapt appearance to user demographics. The system employs a multimodal pipeline combining VAD, ASR (Whisper), LLM-based intent classification, RAG for dialogue, face analysis for personalization, and voice synthesis (XTTS v2). Evaluation demonstrates high accuracy in command recognition (F1: 0.90), demographic estimation (gender F1: 0.89, age MAE: 5.14 years), and speech transcription (WER: 0.181). By uniting aerial robotics with adaptive conversational AI and self-contained visual output, HoverAI introduces a new class of spatially-aware, socially responsive embodied agents for applications in guidance, assistance, and human-centered interaction.
Paper Structure (12 sections, 3 figures)

This paper contains 12 sections, 3 figures.

Table of Contents

  1. Introduction
  2. System Architecture
  3. Hardware Architecture
  4. Interaction Pipeline
  5. Evaluation
  6. Experimental Setup
  7. Results
  8. Applications and Use Cases
  9. Discussion
  10. Limitations
  11. Future Directions
  12. Conclusion

Figures (3)

  • Figure 1: HoverAI quadcopter key hardware components.
  • Figure 2: HoverAI pipeline with audio analysis, video-based face processing, and TTS-driven avatar output.
  • Figure 3: Performance metrics averaged across 12 participants: speech transcription (WER: 0.181), command recognition (F1: 0.90), gender classification (F1: 0.89), and age estimation (MAE: 5.14 years).