A Hetero-Associative Sequential Memory Model Utilizing Neuromorphic Signals: Validated on a Mobile Manipulator

TL;DR

The paper introduces a hetero-associative sequential memory (HASMS) that binds robot joint states and tactile observations into compact binary representations using population place coding and RoPE3D. Tactile data are converted into neuromorphic spike features via an Izhikevich neuron model, enabling geometry-aware, memory-efficient recall via a softmax-weighted memory. Validated on a Toyota HSR with robot skin, HASMS achieves a pseudo-compliance controller and tactile-guided multi-joint grasp sequences, demonstrating fast setup, low compute/memory demands, and generalization potential. The approach offers a neuromorphic, memory-centric alternative for motion control and planning, with prospects for imitation learning and multi-modal integration.

Abstract

This paper presents a hetero-associative sequential memory system for mobile manipulators that learns compact, neuromorphic bindings between robot joint states and tactile observations to produce step-wise action decisions with low compute and memory cost. The method encodes joint angles via population place coding and converts skin-measured forces into spike-rate features using an Izhikevich neuron model; both signals are transformed into bipolar binary vectors and bound element-wise to create associations stored in a large-capacity sequential memory. To improve separability in binary space and inject geometry from touch, we introduce 3D rotary positional embeddings that rotate subspaces as a function of sensed force direction, enabling fuzzy retrieval through a softmax weighted recall over temporally shifted action patterns. On a Toyota Human Support Robot covered by robot skin, the hetero-associative sequential memory system realizes a pseudocompliance controller that moves the link under touch in the direction and with speed correlating to the amplitude of applied force, and it retrieves multi-joint grasp sequences by continuing tactile input. The system sets up quickly, trains from synchronized streams of states and observations, and exhibits a degree of generalization while remaining economical. Results demonstrate single-joint and full-arm behaviors executed via associative recall, and suggest extensions to imitation learning, motion planning, and multi-modal integration.

Figures (6)

• Figure 1: Overview of the structure of hetero-associative sequential memory. The joint states and the observations are encoded and embedded into binary vectors in a high-dimensional space. For the training process, the memory matrix K stores associations between joint states and observations.
• Figure 2: Robot skin cell. (a) Sensor distribution on the cell. (b) Microcontroller, connectivity, and dimensions of the cell.
• Figure 3: Illustration of the HSR, whose hand, wrist, and part of the body are covered with robot skin cells.
• Figure 4: Demonstration of the pseudo-compliance controller: the robotic arm is guided by gentle touches on skin patches, enabling intuitive human interaction
• Figure 5: Trajectory, force amplitude, and joint angular velocity of arm_flex_joint of the HSR under the pseudo-compliance control. The forces on the skin patch wrist_upper and wrist_under are opposite forces. The joint angular velocity is positively correlated with the force applied to the skin patches.