Semantic and structural image segmentation for prosthetic vision

TL;DR

The paper tackles the limited information bandwidth of prosthetic vision by combining convolutional neural networks to extract structural scene information and object silhouettes, which are then conveyed through a schematic phosphene representation for indoor environments. It proposes an end-to-end approach to generate informative phosphene-like images that emphasize edges and object silhouettes, improving scene understanding under simulated prosthetic vision. Experiments with normally sighted subjects using a simulated prosthetic vision setup show improved object recognition and room identification in indoor scenes compared with other image processing methods. This work demonstrates how learning-based scene representations can optimize the visual information delivered to prosthetic users, potentially aiding navigation and environment comprehension.

Abstract

Prosthetic vision is being applied to partially recover the retinal stimulation of visually impaired people. However, the phosphenic images produced by the implants have very limited information bandwidth due to the poor resolution and lack of color or contrast. The ability of object recognition and scene understanding in real environments is severely restricted for prosthetic users. Computer vision can play a key role to overcome the limitations and to optimize the visual information in the simulated prosthetic vision, improving the amount of information that is presented. We present a new approach to build a schematic representation of indoor environments for phosphene images. The proposed method combines a variety of convolutional neural networks for extracting and conveying relevant information about the scene such as structural informative edges of the environment and silhouettes of segmented objects. Experiments were conducted with normal sighted subjects with a Simulated Prosthetic Vision system. The results show good accuracy for object recognition and room identification tasks for indoor scenes using the proposed approach, compared to other image processing methods.

Figures (1)

• Figure 1: Vertical lines in indoor environments. (a) Example of indoor scene, (b) prevalence of vertical lines, (c) square pattern and (d) hexagonal pattern. As a result of the abundance of vertical lines in indoor environments, we opted to use the square grid.