HGIC: A Hand Gesture Based Interactive Control System for Efficient and Scalable Multi-UAV Operations

TL;DR

This work tackles the cognitive burden and scalability limits of coordinating large multi-UAV teams. It introduces HGIC, a camera-based hand-gesture interface that translates natural gestures into modular swarm commands via a mode-based library and a three-layer distributed control architecture, augmented by a real-time UI and UDP-based communication. The approach combines static and dynamic gesture classifiers, a JSON-configurable command converter, and a buffering decision fusion to ensure robust, low-latency control. In simulation and user studies, HGIC demonstrates fast gesture processing, high gesture recognition accuracy, reliable formation control, and favorable user acceptance, suggesting significant practical impact for scalable, human-in-the-loop mUAV operations.

Abstract

As technological advancements continue to expand the capabilities of multi unmanned-aerial-vehicle systems (mUAV), human operators face challenges in scalability and efficiency due to the complex cognitive load and operations associated with motion adjustments and team coordination. Such cognitive demands limit the feasible size of mUAV teams and necessitate extensive operator training, impeding broader adoption. This paper developed a Hand Gesture Based Interactive Control (HGIC), a novel interface system that utilize computer vision techniques to intuitively translate hand gestures into modular commands for robot teaming. Through learning control models, these commands enable efficient and scalable mUAV motion control and adjustments. HGIC eliminates the need for specialized hardware and offers two key benefits: 1) Minimal training requirements through natural gestures; and 2) Enhanced scalability and efficiency via adaptable commands. By reducing the cognitive burden on operators, HGIC opens the door for more effective large-scale mUAV applications in complex, dynamic, and uncertain scenarios. HGIC will be open-sourced after the paper being published online for the research community, aiming to drive forward innovations in human-mUAV interactions.

Figures (5)

• Figure 1: Architecture illustration for hand gesture interactive control (HGIC) system.
• Figure 2: HGIC graphical user interface. The GUI is divided into three main sections: 1) Current command feedback; 2) mUAV feedback; and 3) Real-time feedback for gesture recognition. Additionally, the frame rate is displayed at the top side.
• Figure 3: Gestures for swarm control. The upper part of the figure shows various formations/tasks adopted by the swarm, including evolving from circles to lines, culminating in a V-shape, expanding, merging, and splitting into three distinct groupings. The bottom part summarizes the common gestures for multi-UAVs control.
• Figure 4: mUAV system control architecture. The architecture comprises of three-levels, basic motion control at the low level, advanced task and formation management at the high level, and task allocation at the coordinated control level.
• Figure 5: Multi-UAV formation and coordination capabilities. (A) and (B): formation maintenance and generation during the Search mission. UAV Trajectories over Time (Top) and Velocity Change over Time (Bottom); (C): dynamic formation switching. (1) V to Circle, (2). Circle to V, and (3) Circle to the line. Trajectories over Time (Left) and Velocity change over time (Right). Note: The solid curve is the average of all UAV velocities, and the upper and lower distributions of the transparent part are the maximum and minimum values of velocities.