DVRP-MHSI: Dynamic Visualization Research Platform for Multimodal Human-Swarm Interaction
Pengming Zhu, Zhiwen Zeng, Weijia Yao, Wei Dai, Huimin Lu, Zongtan Zhou
TL;DR
The paper presents DVRP-MHSI, a Dynamic Visualization Research Platform for Multimodal Human-Swarm Interaction, designed to accelerate research by providing real-time visualization and support for multiple input modalities. It combines a multitouch-based test field, a low-cost omnidirectional robot, and a flexible ROS+MQTT software stack with TUIO and Unity to enable rapid scenario deployment and open interfacing. Validation is demonstrated through three case studies using BCI (SSVEP-CCA), EMG gesture control, and eye-tracking for trajectory guidance, illustrating real-time human-swarm collaboration and control scalability. Overall, DVRP-MHSI reduces hardware/software coupling and offers a versatile platform for researchers to develop and test diverse swarm algorithms and human-swarm interaction methods.
Abstract
In recent years, there has been a significant amount of research on algorithms and control methods for distributed collaborative robots. However, the emergence of collective behavior in a swarm is still difficult to predict and control. Nevertheless, human interaction with the swarm helps render the swarm more predictable and controllable, as human operators can utilize intuition or knowledge that is not always available to the swarm. Therefore, this paper designs the Dynamic Visualization Research Platform for Multimodal Human-Swarm Interaction (DVRP-MHSI), which is an innovative open system that can perform real-time dynamic visualization and is specifically designed to accommodate a multitude of interaction modalities (such as brain-computer, eye-tracking, electromyographic, and touch-based interfaces), thereby expediting progress in human-swarm interaction research. Specifically, the platform consists of custom-made low-cost omnidirectional wheeled mobile robots, multitouch screens and two workstations. In particular, the mutitouch screens can recognize human gestures and the shapes of objects placed on them, and they can also dynamically render diverse scenes. One of the workstations processes communication information within robots and the other one implements human-robot interaction methods. The development of DVRP-MHSI frees researchers from hardware or software details and allows them to focus on versatile swarm algorithms and human-swarm interaction methods without being limited to fixed scenarios, tasks, and interfaces. The effectiveness and potential of the platform for human-swarm interaction studies are validated by several demonstrative experiments.