E2H: A Two-Stage Non-Invasive Neural Signal Driven Humanoid Robotic Whole-Body Control Framework

TL;DR

This work presents E2H (EEG-to-Humanoid), an innovative framework that pioneers the control of humanoid robots using high-frequency non-invasive neural signals, and decomposes the E2H framework in an innovative two-stage formation.

Abstract

Recent advancements in humanoid robotics, including the integration of hierarchical reinforcement learning-based control and the utilization of LLM planning, have significantly enhanced the ability of robots to perform complex tasks. In contrast to the highly developed humanoid robots, the human factors involved remain relatively unexplored. Directly controlling humanoid robots with the brain has already appeared in many science fiction novels, such as Pacific Rim and Gundam. In this work, we present E2H (EEG-to-Humanoid), an innovative framework that pioneers the control of humanoid robots using high-frequency non-invasive neural signals. As the none-invasive signal quality remains low in decoding precise spatial trajectory, we decompose the E2H framework in an innovative two-stage formation: 1) decoding neural signals (EEG) into semantic motion keywords, 2) utilizing LLM facilitated motion generation with a precise motion imitation control policy to realize humanoid robotics control. The method of directly driving robots with brainwave commands offers a novel approach to human-machine collaboration, especially in situations where verbal commands are impractical, such as in cases of speech impairments, space exploration, or underwater exploration, unlocking significant potential. E2H offers an exciting glimpse into the future, holding immense potential for human-computer interaction.

Figures (7)

• Figure 1: The illustration of the E2H framework, which could be divided into two stages as described in a) and b). The conformed encoder $\&$ decoder structures shown in part a) first convert the human brain intention to discrete motion keywords. Then in part b) human intention to humanoid robotic control module converts the keyword motion intention to trajectory references. Based on the trajectory references. Given the generated motion reference, the controller model first learns to control policy in the simulation environment then the sim-to-real transfer is performed to get the control model for the physical robot.
• Figure 2: Illustration of the data collection process, where the left part illustrates the channel layout of the collection system. The EEG-to-motion synchronized data is collected by giving the human subjects a visual prompt as a cue and letting the subjects perform silent speech when actively thinking about the motion keywords. The lower part suggests the brain signal change during the data collection period.
• Figure 3: Illustration of the EEG classification model. In the classification task, we aim to predict the silently spoken class from the wavelet spectrogram of the multi-channel EEG signals using a Conformer encoder.
• Figure 4: Illustration of the physical Tien Kung robots E2H utilized.
• Figure 5: The inference flow of the proposed E2H framework. Subfigure a) denotes how the modules are connected during the inference stage. Subfigure b) denotes that during the interaction, the human subject observes the physical robot and actively adjusts its neural signal through visual feedback.