Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Progress Towards Decoding Visual Imagery via fNIRS

Michel Adamic, Wellington Avelino, Anna Brandenberger, Bryan Chiang, Hunter Davis, Stephen Fay, Andrew Gregory, Aayush Gupta, Raphael Hotter, Grace Jiang, Fiona Leng, Stephen Polcyn, Thomas Ribeiro, Paul Scotti, Michelle Wang, Marley Xiong, Jonathan Xu

TL;DR

This work examines the feasibility of decoding visual imagery from fNIRS signals and moving toward a wearable TD-fNIRS device. By leveraging downsampled fMRI data, the authors show that 1-cm spatial resolution can support image retrieval and reconstruction, and they provide tomographic simulations indicating TD-fNIRS offers superior spatial resolution and depth penetration over CW-fNIRS. They also report progress toward a low-cost hardware prototype including a gain-switching laser approach, APD/SiPM detectors, and a time-to-digital converter, along with an open-source codebase for ML and tomography. While the findings are promising for portable brain imaging and imagined-image communication, the authors candidly discuss limitations (e.g., 2D simulations, lack of skull modeling, temperature sensitivity) and next steps required to validate a practical device. Overall, the work outlines a concrete path to high-density, high-timing-resolution fNIRS systems capable of reconstructing visual representations, with potential impact on wearable neuroimaging and brain-computer interfaces.

Abstract

We demonstrate the possibility of reconstructing images from fNIRS brain activity and start building a prototype to match the required specs. By training an image reconstruction model on downsampled fMRI data, we discovered that cm-scale spatial resolution is sufficient for image generation. We obtained 71% retrieval accuracy with 1-cm resolution, compared to 93% on the full-resolution fMRI, and 20% with 2-cm resolution. With simulations and high-density tomography, we found that time-domain fNIRS can achieve 1-cm resolution, compared to 2-cm resolution for continuous-wave fNIRS. Lastly, we share designs for a prototype time-domain fNIRS device, consisting of a laser driver, a single photon detector, and a time-to-digital converter system.

Progress Towards Decoding Visual Imagery via fNIRS

TL;DR

This work examines the feasibility of decoding visual imagery from fNIRS signals and moving toward a wearable TD-fNIRS device. By leveraging downsampled fMRI data, the authors show that 1-cm spatial resolution can support image retrieval and reconstruction, and they provide tomographic simulations indicating TD-fNIRS offers superior spatial resolution and depth penetration over CW-fNIRS. They also report progress toward a low-cost hardware prototype including a gain-switching laser approach, APD/SiPM detectors, and a time-to-digital converter, along with an open-source codebase for ML and tomography. While the findings are promising for portable brain imaging and imagined-image communication, the authors candidly discuss limitations (e.g., 2D simulations, lack of skull modeling, temperature sensitivity) and next steps required to validate a practical device. Overall, the work outlines a concrete path to high-density, high-timing-resolution fNIRS systems capable of reconstructing visual representations, with potential impact on wearable neuroimaging and brain-computer interfaces.

Abstract

We demonstrate the possibility of reconstructing images from fNIRS brain activity and start building a prototype to match the required specs. By training an image reconstruction model on downsampled fMRI data, we discovered that cm-scale spatial resolution is sufficient for image generation. We obtained 71% retrieval accuracy with 1-cm resolution, compared to 93% on the full-resolution fMRI, and 20% with 2-cm resolution. With simulations and high-density tomography, we found that time-domain fNIRS can achieve 1-cm resolution, compared to 2-cm resolution for continuous-wave fNIRS. Lastly, we share designs for a prototype time-domain fNIRS device, consisting of a laser driver, a single photon detector, and a time-to-digital converter system.
Paper Structure (30 sections, 2 equations, 22 figures, 1 table)

This paper contains 30 sections, 2 equations, 22 figures, 1 table.

Table of Contents

  1. Introduction
  2. Resolution needed for image reconstruction
  3. Dataset and model
  4. Results
  5. Limitations
  6. Tomographic spatial resolution simulation
  7. Diffuse optical tomography
  8. Simulation details
  9. Results and limitations
  10. Time-domain hardware prototype
  11. System overview and requirements
  12. Pulsed light source
  13. Single photon detectors
  14. Avalanche photodiode (APD)
  15. Silicon photomultiplier (SiPM) detector
  16. ...and 15 more sections

Figures (22)

  • Figure 1: Left: image shown to participant. Right: image reconstructed from fMRI via MindEye.
  • Figure 2: Example reconstructions from downsampled MindEye at various resolutions.
  • Figure 3: Spatial resolution of time-domain and continuous-wave fNIRS. The inclusions are placed 0.5 cm, 1 cm, and 2 cm apart. The width of each inclusion is set to be half of the separation.
  • Figure 4: Depth penetration of time-domain and continuous-wave fNIRS. The centers of the inclusions are placed at a depth of 2 cm, 3 cm, 4 cm, and 5 cm. The inclusions are placed 3 cm apart.
  • Figure 5: Left: Photons follow a banana-shaped path through the tissue and are picked up by the detector a couple centimeters away from the source (image from Strait_2014). Right: TD fNIRS emits a very short pulse of light in the tissue and measures the arrival times of photons that emerge (image from Scholkmann2014).
  • ...and 17 more figures