Context-aware collaborative pushing of heavy objects using skeleton-based intention prediction

TL;DR

This work addresses intent prediction for human-robot collaboration when force feedback is unavailable, focusing on pushing/pulling heavy objects on frictional surfaces. It introduces a context-aware approach that uses skeleton-based posture data and a Directed Graph Neural Network (DGNN) to predict the human’s intention (pull, idle, push) in real time and drives an assistive controller without relying on force sensing alone. The system couples intention prediction with a friction compensation term to generate target forces and employs a force-controller to realize position corrections, validated through real-world experiments. Results show that robot-assisted collaboration substantially reduces human effort and improves efficiency, demonstrating the value of posture-based context in guiding pHRI decisions and control.

Abstract

In physical human-robot interaction, force feedback has been the most common sensing modality to convey the human intention to the robot. It is widely used in admittance control to allow the human to direct the robot. However, it cannot be used in scenarios where direct force feedback is not available since manipulated objects are not always equipped with a force sensor. In this work, we study one such scenario: the collaborative pushing and pulling of heavy objects on frictional surfaces, a prevalent task in industrial settings. When humans do it, they communicate through verbal and non-verbal cues, where body poses, and movements often convey more than words. We propose a novel context-aware approach using Directed Graph Neural Networks to analyze spatio-temporal human posture data to predict human motion intention for non-verbal collaborative physical manipulation. Our experiments demonstrate that robot assistance significantly reduces human effort and improves task efficiency. The results indicate that incorporating posture-based context recognition, either together with or as an alternative to force sensing, enhances robot decision-making and control efficiency.

Figures (9)

• Figure 1: The task is split into three possible movements: when the human intends to push, pull, or stay idle. The collaborative robot predicts the intended motion from the human skeleton tracking data to act appropriately in the context.
• Figure 2: The information flow in our system. The independent bodies of the human, object, and robot interact by direct forces $\boldsymbol{f}_{h}, \boldsymbol{f}_{o}$ and $\boldsymbol{f}_{r}$. Human intention is predicted using human posture sensing, and object friction information is discovered using the robot-side F/T sensor. Intention vector $\boldsymbol{i}_h$ and compensation force ${f}_{com}$ are used by the assistance controller to generate the desired robot forces $\boldsymbol{f}_d$ that are aligned with the expected human forces.
• Figure 3: We formulate the intention prediction problem as computing the output class (orange: push/idle/pull) given a time window of input features (blue: 3D joint and bone data). In this work, we set the window size and prediction offset as $0.5$s and $0.25$s.
• Figure 4: Visualisation of the push (left) and pull (right) actions during runtime. The measured object velocity (green) and the intention prediction output (red) are also shown as arrows. The figure also shows examples of bone and joint data.
• Figure 5: a) The user wears the Xsens markers for skeleton tracking; b) An F/T sensor is attached between the human-side handle and the box; c) An Optitrack marker is attached to the object for tracking its pose $\boldsymbol{x}_b$; d) A rigid wooden box filled with heavy items; e) The robot grasps the other handle using an anthropomorphic robotic hand; f) A 6-DoF position-controlled robot arm with a wrist F/T sensor. We also indicate the $x$-axis of the robot's reference frame.