Neural Associative Skill Memories for safer robotics and modelling human sensorimotor repertoires

TL;DR

The paper tackles the challenge of learning and safely executing multiple sensorimotor skills without a hand-crafted library of primitives. It proposes Neural Associative Skill Memories (ASMs) based on temporal predictive coding (tPC), a biologically plausible, energy-based framework that unifies skill learning, recall, fault detection, and reactive control under context-driven inference. The approach demonstrates fault detection and reactive correction in simulation, shows context-driven memory separation and expression, and predicts a speed-accuracy trade-off in memory recall that aligns with motor preparation concepts. This work advances neurorobotics by offering a unified, self-supervised model for safer robotics and provides a computational lens on biological sensorimotor learning.

Abstract

Modern robots face challenges shared by humans, where machines must learn multiple sensorimotor skills and express them adaptively. Equipping robots with a human-like memory of how it feels to do multiple stereotypical movements can make robots more aware of normal operational states and help develop self-preserving safer robots. Associative Skill Memories (ASMs) aim to address this by linking movement primitives to sensory feedback, but existing implementations rely on hard-coded libraries of individual skills. A key unresolved problem is how a single neural network can learn a repertoire of skills while enabling fault detection and context-aware execution. Here we introduce Neural Associative Skill Memories (ASMs), a framework that utilises self-supervised predictive coding for temporal prediction to unify skill learning and expression, using biologically plausible learning rules. Unlike traditional ASMs which require explicit skill selection, Neural ASMs implicitly recognize and express skills through contextual inference, enabling fault detection across learned behaviours without an explicit skill selection mechanism. Compared to recurrent neural networks trained via backpropagation through time, our model achieves comparable qualitative performance in skill memory expression while using local learning rules and predicts a biologically relevant speed-accuracy trade-off during skill memory expression. This work advances the field of neurorobotics by demonstrating how predictive coding principles can model adaptive robot control and human motor preparation. By unifying fault detection, reactive control, skill memorisation and expression into a single energy-based architecture, Neural ASMs contribute to safer robotics and provide a computational lens to study biological sensorimotor learning.

Figures (9)

• Figure 1: (A & B) Comparison of our Neural ASM model using temporal predictive coding (tPC) network with ASMs pastor2012towards which uses Dynamic Movement Primitives (DMPs). Our model replaces the hard-coded library of movement primitives in ASMs with a single tPC network responsible for learning multiple skill memories. The input to neural ASMs is multiple repetitions or demonstrations of multiple skills, where a skill demonstration is a time series of sensorimotor observations (C) Graphical model tPC is that of a Hidden Markov Model (HMM). $x$ represents sensorimotor observations, whereas $z$ are the hidden states which capture the dynamics and the inferred context. Here, we show offline recall or prediction of a sequence of future sensorimotor observations using only the first sensorimotor observation $x^\mu$ as input during cued inference. It is also possible to use observations from multiple time steps during cued inference, which is not shown here. In an HMM, the prediction step is equivalent to a forward pass (this may not always be the case and should be performed by iterative energy minimisation in such scenarios).
• Figure 2: (A) Demonstration of Neural ASMs learning two pick and place skills in simulation. The sensorimotor sequences in the dataset used for learning from demonstrations are generated using predefined end-effector goals and use inverse kinematics to get joint angles. This is intended to be a proxy for teleoperation in simulation. (B) A schematic of Franka Panda arm joint labels (adapted from rogel2022robogroove)
• Figure 3: (A-E) Minor fault example with fault detection using energies and correct fault isolation using absolute prediction errors along with the joint angle time series. (F-J) Major fault example with fault detection using energies and incorrect fault isolation using absolute prediction errors along with the joint angle time series. (K-M) A basic demonstration of systematic evaluation of fault detection.
• Figure 4: Demonstration of a fault resulting from a collision with a falling object. The fault is corrected reactively on the fly by having the low-level controller minimise proprioceptive prediction errors in joint configuration space using the proprioceptive predictions from Neural ASMs.
• Figure 5: (A) Schematic from sheahan2016motor describing their experiment which inspires our robotics experiment (B) Adaptation result from sheahan2016motor(C) Motor plan (end-effector positions) in the synthetic dataset, S: starting point, T: central target, ST: secondary target. Further details in the Appendix 2, Fig \ref{['sheahan_appendix']} (D) Simplified representations of our model and baseline RNNs. S-to-M RNN predicts the motor observations at the next discrete time step ($\mu +1$) using the sensory observations at discrete time step ($\mu$) as input. SM-to-SM predicts the sensorimotor observations at the next discrete time step ($\mu +1$) using sensorimotor observations at the current time step ($\mu$) (E) Results qualitatively replicated by our model, on par with baseline models. (F) Qualitative comparison of post-exposure (hand-path) trajectories (G) Speed accuracy trade-off predicted in-memory expression. Here, demonstrated in offline skill recall.