Neuromorphic BrailleNet: Accurate and Generalizable Braille Reading Beyond Single Characters through Event-Based Optical Tactile Sensing

TL;DR

This work tackles continuous Braille reading by leveraging a neuromorphic tactile sensor (Evetac) to capture dynamic contact events during fingertip-like sliding. The authors introduce BrailleNet, a ResNet-based classifier paired with a segmentation module and a NormAug training strategy to process sparse event streams into robust character and word recognition. They demonstrate near-perfect character accuracy (≈99.5%) across varied indentation depths and strong word-level performance (>90%) on boards containing daily-living vocabulary, including resilience to speeds up to 32 mm/s. The results advocate for neuromorphic tactile sensing as a low-latency, scalable solution for assistive Braille readers and broader tactile perception tasks in robotics, with potential extensions to more complex Braille layouts and real-world devices.

Abstract

Conventional robotic Braille readers typically rely on discrete, character-by-character scanning, limiting reading speed and disrupting natural flow. Vision-based alternatives often require substantial computation, introduce latency, and degrade in real-world conditions. In this work, we present a high accuracy, real-time pipeline for continuous Braille recognition using Evetac, an open-source neuromorphic event-based tactile sensor. Unlike frame-based vision systems, the neuromorphic tactile modality directly encodes dynamic contact events during continuous sliding, closely emulating human finger-scanning strategies. Our approach combines spatiotemporal segmentation with a lightweight ResNet-based classifier to process sparse event streams, enabling robust character recognition across varying indentation depths and scanning speeds. The proposed system achieves near-perfect accuracy (>=98%) at standard depths, generalizes across multiple Braille board layouts, and maintains strong performance under fast scanning. On a physical Braille board containing daily-living vocabulary, the system attains over 90% word-level accuracy, demonstrating robustness to temporal compression effects that challenge conventional methods. These results position neuromorphic tactile sensing as a scalable, low latency solution for robotic Braille reading, with broader implications for tactile perception in assistive and robotic applications.

Figures (6)

• Figure 1: Neuromorphic tactile-based Braille reading system: (a) A robotic arm mounted with an open-source event-based optical tactile sensor (Evetac) scanning 3D-printed Braille boards; (b) Braille alphabet layout, where black-filled circles indicate raised dots; (c) visualized spatiotemporal event streams captured by the sensor (input to the network); (d) schematic of the neural network architecture for processing the tactile input; (e) output classification results showing representative images after inference.
• Figure 2: Experimental setup for neuromorphic Braille data collection. (A) Exploded view of the customized Evetac optical tactile sensor. From top to bottom: DVXplorer Mini event-based camera, 3D-printed camera housing, internal LED strip for illumination, and a flat GelSight Mini elastomer. (B) Training Boards: 3D-printed Braille boards used for model training. Data were collected from two boards, one containing all 26 letters of the English alphabet and two containing all characters from the phrase "University of .......", arranged at equal spacing. (C) Evaluation Boards: 3D-printed Braille boards used to evaluate model performance. The evaluation set includes two boards containing ten commonly used words and all 26 letters of the English alphabet in random order arranged at equal distance. (D) Evetac sensor mounted on the end-effector of a DOBOT robotic arm, performing a scanning motion across the Braille boards for tactile data collection.
• Figure 3: Schematic overview of the proposed neuromorphic tactile processing and classification pipeline based on a deep ResNet-34 architecture for Braille character recognition. Raw asynchronous tactile events are separated into ON and OFF polarities and pre-processed via spatiotemporal event integration over a 200 ms window (20 event frames) to form normalized event representations. During training, online data augmentation is applied to improve robustness to variations in contact conditions and scanning dynamics. The resulting inputs are processed by Neuromorphic BrailleNet, comprising an initial convolution and max-pooling stage followed by four residual layers (ResLayer0–ResLayer3), adaptive average pooling, and three fully connected layers. The network predicts one of 26 Braille alphabet classes, which is compared with the corresponding ground-truth Braille pattern.
• Figure 4: Classification accuracy of the proposed ResNet-based Braille recognition model for random alphabet board. (99.54% accuracy)
• Figure 5: Classification accuracy of the proposed ResNet-based Braille recognition model across indentation depths from 0.2 to 1.5 mm.