Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SENSORIMOTOR GRAPH: Action-Conditioned Graph Neural Network for Learning Robotic Soft Hand Dynamics

João Damião Almeida, Paul Schydlo, Atabak Dehban, José Santos-Victor

TL;DR

<sentence>Soft robotics are flexible and challenging to model due to continuous topology and time-varying parameters.</sentence> <sentence>The authors propose an action-conditioned Sensorimotor Graph, a graph neural network that learns the connectivity of a non-rigid hand from observations and uses it to predict future states under actuation.</sentence> <sentence>Leveraging Neural Relational Inference, the approach infers edges, supports variable node counts, and can be trained self-supervised via motor babbling, demonstrating robustness to configuration and sensing variations in simulation.</sentence> <sentence>Results show the Sensorimotor Graph outperforms non-structured baselines and offers a differentiable dynamics model suitable for model-based control of soft robots, with potential for broader applications in complex, interacting systems.</sentence>

Abstract

Soft robotics is a thriving branch of robotics which takes inspiration from nature and uses affordable flexible materials to design adaptable non-rigid robots. However, their flexible behavior makes these robots hard to model, which is essential for a precise actuation and for optimal control. For system modelling, learning-based approaches have demonstrated good results, yet they fail to consider the physical structure underlying the system as an inductive prior. In this work, we take inspiration from sensorimotor learning, and apply a Graph Neural Network to the problem of modelling a non-rigid kinematic chain (i.e. a robotic soft hand) taking advantage of two key properties: 1) the system is compositional, that is, it is composed of simple interacting parts connected by edges, 2) it is order invariant, i.e. only the structure of the system is relevant for predicting future trajectories. We denote our model as the 'Sensorimotor Graph' since it learns the system connectivity from observation and uses it for dynamics prediction. We validate our model in different scenarios and show that it outperforms the non-structured baselines in dynamics prediction while being more robust to configurational variations, tracking errors or node failures.

SENSORIMOTOR GRAPH: Action-Conditioned Graph Neural Network for Learning Robotic Soft Hand Dynamics

TL;DR

<sentence>Soft robotics are flexible and challenging to model due to continuous topology and time-varying parameters.</sentence> <sentence>The authors propose an action-conditioned Sensorimotor Graph, a graph neural network that learns the connectivity of a non-rigid hand from observations and uses it to predict future states under actuation.</sentence> <sentence>Leveraging Neural Relational Inference, the approach infers edges, supports variable node counts, and can be trained self-supervised via motor babbling, demonstrating robustness to configuration and sensing variations in simulation.</sentence> <sentence>Results show the Sensorimotor Graph outperforms non-structured baselines and offers a differentiable dynamics model suitable for model-based control of soft robots, with potential for broader applications in complex, interacting systems.</sentence>

Abstract

Soft robotics is a thriving branch of robotics which takes inspiration from nature and uses affordable flexible materials to design adaptable non-rigid robots. However, their flexible behavior makes these robots hard to model, which is essential for a precise actuation and for optimal control. For system modelling, learning-based approaches have demonstrated good results, yet they fail to consider the physical structure underlying the system as an inductive prior. In this work, we take inspiration from sensorimotor learning, and apply a Graph Neural Network to the problem of modelling a non-rigid kinematic chain (i.e. a robotic soft hand) taking advantage of two key properties: 1) the system is compositional, that is, it is composed of simple interacting parts connected by edges, 2) it is order invariant, i.e. only the structure of the system is relevant for predicting future trajectories. We denote our model as the 'Sensorimotor Graph' since it learns the system connectivity from observation and uses it for dynamics prediction. We validate our model in different scenarios and show that it outperforms the non-structured baselines in dynamics prediction while being more robust to configurational variations, tracking errors or node failures.

Paper Structure

This paper contains 25 sections, 7 equations, 3 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Motivation
  3. Related Work
  4. Forward Models
  5. Graph Neural Networks
  6. Our Approach
  7. Contributions
  8. Problem Statement
  9. Model
  10. Structure
  11. Encoder
  12. Decoder
  13. Loss
  14. Experimental Setup
  15. Data Collection
  16. ...and 10 more sections

Figures (3)

  • Figure 1: Model based control applications require accurate prediction models. a) In this work we consider a Robotic Soft hand gripper system Homberg b) simulate the gripper as soft material in a physics based simulator (SOFA) SOFA under different conditions c) then given keypoints on the soft hand we learn a robust graph structured model d) the resulting differentiable system dynamics outperform the baseline models in a set of experiments validating different model assumptions neurips2018.]
  • Figure 2: Sources of variability in the datasets: a) finger positions in a dodecagon arrangement with one configuration selected (green), configurations are generated by permutations of the selected fingers b) Sample of cable pull actuation signals used in the Trainset .
  • Figure 3: Cumulative mean square error (MSE) over 10 time-steps for different models.