Object Classification Utilizing Neuromorphic Proprioceptive Signals in Active Exploration: Validated on a Soft Anthropomorphic Hand

TL;DR

This work tackles proprioceptive object classification during active exploration using a soft anthropomorphic hand by integrating Bend Labs stretch sensors with the QB SoftHand and encoding proprioceptive signals via a muscle spindle model into spike trains. A four-layer hybrid SNN, trained with SLAYER, processes these neuromorphic signals to classify ten objects from the YCB benchmark, achieving superior accuracy, particularly in the early stages of exploration, compared with LSTM and kNN baselines. The study demonstrates a hardware-software pipeline that enables energy-efficient, real-time proprioceptive perception, with strong implications for haptic feedback and neural prosthetics. The approach combines neuromorphic encoding, active exploration, and real-time inference to advance proprioceptive sensing in robot perception and manipulation.

Abstract

Proprioception, a key sensory modality in haptic perception, plays a vital role in perceiving the 3D structure of objects by providing feedback on the position and movement of body parts. The restoration of proprioceptive sensation is crucial for enabling in-hand manipulation and natural control in the prosthetic hand. Despite its importance, proprioceptive sensation is relatively unexplored in an artificial system. In this work, we introduce a novel platform that integrates a soft anthropomorphic robot hand (QB SoftHand) with flexible proprioceptive sensors and a classifier that utilizes a hybrid spiking neural network with different types of spiking neurons to interpret neuromorphic proprioceptive signals encoded by a biological muscle spindle model. The encoding scheme and the classifier are implemented and tested on the datasets we collected in the active exploration of ten objects from the YCB benchmark. Our results indicate that the classifier achieves more accurate inferences than existing learning approaches, especially in the early stage of the exploration. This system holds the potential for development in the areas of haptic feedback and neural prosthetics.

Figures (7)

• Figure 1: The schematic diagram of the system. The proprioceptive sensor readings are encoded into spike trains with a biological muscle spindle model and fed to a hybrid spiking neural network for classification. The hybrid SNN includes a resonator layer and multiple integrator layers.
• Figure 2: The QB Softhand integrated with four flexible stretch sensors on the dorsal side of the thumb, index finger, middle finger, and ring finger, respectively.
• Figure 3: The schematic diagram of the muscle spindle model, which consists of 3 intrafusal fiber models.
• Figure 4: Encoding of proprioceptive signal with muscle spindle model while the hand is closing.
• Figure 5: Illustration of a whole active exploration process in data collection. (a) shows the initial setup. (b) and (d) compares smaller and larger objects in light and strong grasp. (c) shows the release after the light grasp. (e) and (f) illustrate the lift-and-weigh stage in the exploration. In (g) and (h), the object is put down for the next trail.