HARMONIOUS -- Human-like reactive motion control and multimodal perception for humanoid robots

TL;DR

This paper addresses safe real-time operation of a 17-DoF upper body humanoid in dynamic human environments by introducing HARMONIOUS, a quadratic-programming based reactive controller that fuses visual, proximity, and tactile data into unified, body-wide constraints to achieve human-like minimum-jerk motion. The method combines dual-arm coordination with a common torso, self-collision avoidance, and a flexible obstacle-processing pipeline that maps multimodal cues to peripersonal-space constraints on the robot surface. The approach is validated through extensive simulation and real-world experiments, including obstacle-rich interactions and a board-game demonstration, demonstrating improved reachability, smoother trajectories, and robust dynamic obstacle avoidance compared to baselines. The work demonstrates the feasibility and value of a dense, multimodal perception-to-control loop for safe, natural human-robot interaction in unstructured settings, while remaining modular and extensible for future enhancements such as gaze control and dynamics-based controllers.

Abstract

For safe and effective operation of humanoid robots in human-populated environments, the problem of commanding a large number of Degrees of Freedom (DoF) while simultaneously considering dynamic obstacles and human proximity has still not been solved. We present a new reactive motion controller that commands two arms of a humanoid robot and three torso joints (17 DoF in total). We formulate a quadratic program that seeks joint velocity commands respecting multiple constraints while minimizing the magnitude of the velocities. We introduce a new unified treatment of obstacles that dynamically maps visual and proximity (pre-collision) and tactile (post-collision) obstacles as additional constraints to the motion controller, in a distributed fashion over the surface of the upper body of the iCub robot (with 2000 pressure-sensitive receptors). This results in a bio-inspired controller that: (i) gives rise to a robot with whole-body visuo-tactile awareness, resembling peripersonal space representations, and (ii) produces human-like minimum jerk movement profiles. The controller was extensively experimentally validated, including a physical human-robot interaction scenario.

Figures (7)

• Figure 1: iCub robot in action during an interactive game demonstration.
• Figure 2: HARMONIOUS -- overview. The blue box represents the local trajectory sampling. The trajectory towards the target pose is sampled using a minimum-jerk generator (position) and a low-pass filter (orientation). The target pose is received from an external target generator. The green box represents the obstacle processing. It takes inputs from proximity sensors, skin parts (tactile stimuli), and RGB-D camera (visual stimuli) and computes linear inequality constraints to dodge obstacles or react to the collision that occurred. The obstacle exists virtually for a specific surviving time with a decreasing threat level, giving humans time to react. The red box represents the controller with a QP solver that solves the differential kinematics problem. It takes the next desired pose and current joint positions as input. If the problem is feasible, the computed joint velocities are integrated into new joint positions and sent to the robot. Otherwise, the position constraint is relaxed, and the QP problem is solved again.
• Figure 3: Processing of obstacles. Obstacles (red cubes or stars) are projected onto the robot body ($P_C$) and connected (yellow line) to the robot kinematic chain (green line). Each collision point is characterized by a normal direction (purple arrow) and threat level (color of the circle, red means highest threat).
• Figure 4: Simulation experiments results. LTS stands for local trajectory sampling.
• Figure 5: Visualizations from real-world collision avoidance experiments.