SymbolSight: Minimizing Inter-Symbol Interference for Reading with Prosthetic Vision

TL;DR

Retinal prostheses impose a low-resolution, temporally persistent channel that degrades reading through inter-symbol interference. SymbolSight proposes a software-driven solution: a modular pipeline that simulates prosthetic vision, uses a neural proxy observer to estimate symbol confusability under distortion, and optimizes a language-aware symbol-to-letter mapping via the Hungarian algorithm. Across Arabic, Bulgarian, and English, heterogeneous symbol pools combined with optimization yield large reductions in predicted confusion (median ≈ $22\times$) compared with native alphabets or randomized baselines, highlighting the value of codebook design over hardware changes. The work provides a practical framework to narrow design choices for prosthetic reading and sets a path toward psychophysical validation and device-specific, word-level encodings.

Abstract

Retinal prostheses restore limited visual perception, but low spatial resolution and temporal persistence make reading difficult. In sequential letter presentation, the afterimage of one symbol can interfere with perception of the next, leading to systematic recognition errors. Rather than relying on future hardware improvements, we investigate whether optimizing the visual symbols themselves can mitigate this temporal interference. We present SymbolSight, a computational framework that selects symbol-to-letter mappings to minimize confusion among frequently adjacent letters. Using simulated prosthetic vision (SPV) and a neural proxy observer, we estimate pairwise symbol confusability and optimize assignments using language-specific bigram statistics. Across simulations in Arabic, Bulgarian, and English, the resulting heterogeneous symbol sets reduced predicted confusion by a median factor of 22 relative to native alphabets. These results suggest that standard typography is poorly matched to serial, low-bandwidth prosthetic vision and demonstrate how computational modeling can efficiently narrow the design space of visual encodings to generate high-potential candidates for future psychophysical and clinical evaluation.

Figures (6)

• Figure 1: SymbolSight pipeline: candidate symbols undergo phosphene simulation and recognition modeling, then are assigned to letters based on confusion probabilities and bigram statistics.
• Figure 2: Letter transition probabilities across three languages. Heatmaps show $P(L_{n+1} \mid L_n)$ for Arabic (left), Bulgarian (middle), and English (right). The vertical axis is the leading (current) letter; the horizontal axis is the following (next) letter. Darker cells indicate higher probability. Language-specific bigram probabilities were estimated from November 2023 Wikipedia database dumps wikimedia2023 for Arabic, Bulgarian, and English. For each language, we processed the first 100,000 Wikipedia documents. English text was case-folded and filtered to the 26 Latin letters using regex word extraction. Bulgarian text was Unicode NFC-normalized and case-folded across its 30-letter Cyrillic alphabet, with the accented variant (Cyrillic I with grave accent) mapped to plain Cyrillic I. Arabic text underwent NFKD decomposition with diacritical marks stripped, tatweel removed, alif maqsura normalized to yeh, ta marbuta mapped to ta, and standalone hamza treated as a word boundary, yielding a 28-letter alphabet. Documents with fewer than 500 valid letters were excluded to ensure reliable statistics. The resulting co-occurrence matrices were normalized to estimate conditional bigram probabilities $P(L_{n+1}\mid L_n)$ used in the assignment optimization. For visualization, each row of the heatmap is normalized to sum to $1$.
• Figure 3: Top row: low distortion; middle row: medium distortion; bottom row: high distortion. Middle column: 146 symbols with spatial distortion (Latin 0--25, Braille 26--51, Arabic 52--79, DCT 80--115, Cyrillic 116--145). Right column: example of temporal distortion showing perceptual residue when symbols are presented sequentially. Left column: symbol confusion probability heatmaps from fine-tuned neural networks evaluated on a held out validation dataset. Darker cells indicate higher confusion; white indicates near-zero. Our proxy observer networks were trained on a symbol pool of 146 glyphs comprising 26 Latin letters, 26 Braille characters, 28 Arabic letters, 36 DCT basis patterns, and 30 Cyrillic letters. Each symbol was rendered at $64 \times 64$ pixels and processed through four distortion conditions using the pulse2percept library beyeler2017pulse2percept: undistorted, and three prosthetic vision simulations on a $16 \times 16$ electrode grid with phosphene sizes $\rho \in \{100, 300, 500\}\,\mu\mathrm{m}$ and axon streak parameters $\lambda \in \{0, 1000, 5000\}\,\mu\mathrm{m}$. For each distortion level, a separate MobileNetV3Large howard2019searching classifier was trained with ImageNet-pretrained weights frozen and a custom head consisting of global average pooling, 20% dropout, and a 146-way softmax output layer with L1-L2 regularization ($\ell_1 = 0.001$, $\ell_2 = 0.02$). Training used the Adam optimizer with learning rate of $10^{-4}$, batch size of 64, and MixUp augmentation zhang2018mixup with $\alpha = \beta = 2.0$ to generate 500 augmented training samples per symbol (73,000 images) and 100 validation samples per symbol (14,600 images). Early stopping monitored validation loss with patience of 20 epochs, and learning rate was reduced by half after 3 epochs without improvement. To match MobileNetV3Large's input requirements, images were resized to $224 \times 224$ RGB. The train and validation sets contained all 146 symbol classes with independently augmented samples, ensuring no leakage between splits.
• Figure 4: Arabic symbol comparison. Top three rows: native Arabic at low, medium, and high distortion. Bottom three rows: optimized symbols at each distortion level in matching order. Note how the native characters blur into indistinguishable blobs at high distortion (Row 3), whereas the optimized glyphs maintain distinct structural footprints (Row 6).
• Figure 5: Bulgarian symbol comparison. Top three rows: native Cyrillic at low, medium, and high distortion. Bottom three rows: optimized symbols at each distortion level in matching order.