Evaluating Deep Human-in-the-Loop Optimization for Retinal Implants Using Sighted Participants

TL;DR

The study evaluates the viability of human-in-the-loop optimization (HILO) for personalizing deep stimulus encoders in simulated prosthetic vision using sighted participants. It shows that participants consistently prefer HILO-generated stimuli over naive and non-personalized encoders across standard, misspecification, and out-of-distribution scenarios, though human decision-making diverges from purely simulated rules, suggesting perceptual factors beyond pixel-wise similarity influence choices. The results support HILO as a robust approach for tailoring visual prostheses to individual users while highlighting the need for human validation and careful generalization considerations. Limitations include the use of static, sighted participants with simulated vision and the absence of temporal dynamics, pointing to future work with blind prosthesis users and dynamic, real-world testing to advance clinical translation.

Abstract

Human-in-the-loop optimization (HILO) is a promising approach for personalizing visual prostheses by iteratively refining stimulus parameters based on user feedback. Previous work demonstrated HILO's efficacy in simulation, but its performance with human participants remains untested. Here we evaluate HILO using sighted participants viewing simulated prosthetic vision to assess its ability to optimize stimulation strategies under realistic conditions. Participants selected between phosphenes generated by competing encoders to iteratively refine a deep stimulus encoder (DSE). We tested HILO in three conditions: standard optimization, threshold misspecifications, and out-of-distribution parameter sampling. Participants consistently preferred HILO-generated stimuli over both a naive encoder and the DSE alone, with log odds favoring HILO across all conditions. We also observed key differences between human and simulated decision-making, highlighting the importance of validating optimization strategies with human participants. These findings support HILO as a viable approach for adapting visual prostheses to individuals. Clinical relevance: Validating HILO with sighted participants viewing simulated prosthetic vision is an important step toward personalized calibration of future visual prostheses.

Figures (3)

• Figure 1: A) Deep stimulus encoder (DSE). A forward model ($f$) predicts the perceptual response to visual stimuli based on user-specific parameters ($\phi$), while an encoder ($f^{-1}$) learns to minimize the perceptual error between predicted and target percepts. B) Human-in-the-loop optimization (HILO). The parameters from the DSE are refined using user preferences, collected through 60 binary comparison trials per condition. New parameter pairs are adaptively selected to efficiently converge on the most preferred percept. The target percept changes each iteration. Adapted under CC-BY from https://doi.org/10.48550/arXiv.2306.13104. C) Example duels used to infer user preferences are presented to sighted participants on a computer monitor. Participants selected the preferred stimulus based on both shape and brightness.
• Figure 2: Example stimuli illustrating the range of phosphene brightness levels shown to participants. A value of 0 represents complete darkness, 2 is the ideal brightness for a retinal prosthesis user, 5 is overly bright, and 10 is extremely bright, with white filling most of the stimulus area.
• Figure 3: A) Number of participants who significantly preferred HILO over the naïve encoder (left) and the DSE without HILO (right), based on log odds less than 0 in a linear mixed-effects model. B) Example percepts generated by the HILO encoder, the DSE without HILO, and the naïve encoder for three participants across the three experimental conditions. C) Distribution of log odds for HILO across the three experiments. D) Distribution of the two main user-specific parameters, $\rho$ (phosphene size) and $\lambda$ (axon-aligned elongation), colored by the log odds indicating preference for HILO in the main experiment. E) Median MSE over the course of optimization for each experiment, with shaded regions denoting the interquartile range (IQR). F) Proportion of duels where participant decisions matched those of the simulated agent.