Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Calibration of Deep Learning Classification Models in fNIRS

Zhihao Cao, Zizhou Luo

TL;DR

This paper proposes integrating calibration into fNIRS field and assess the reliability of existing models and results indicate poor calibration performance in many proposed models.

Abstract

Functional near-infrared spectroscopy (fNIRS) is a valuable non-invasive tool for monitoring brain activity. The classification of fNIRS data in relation to conscious activity holds significance for advancing our understanding of the brain and facilitating the development of brain-computer interfaces (BCI). Many researchers have turned to deep learning to tackle the classification challenges inherent in fNIRS data due to its strong generalization and robustness. In the application of fNIRS, reliability is really important, and one mathematical formulation of the reliability of confidence is calibration. However, many researchers overlook the important issue of calibration. To address this gap, we propose integrating calibration into fNIRS field and assess the reliability of existing models. Surprisingly, our results indicate poor calibration performance in many proposed models. To advance calibration development in the fNIRS field, we summarize three practical tips. Through this letter, we hope to emphasize the critical role of calibration in fNIRS research and argue for enhancing the reliability of deep learning-based predictions in fNIRS classification tasks. All data from our experimental process are openly available on GitHub.

Calibration of Deep Learning Classification Models in fNIRS

TL;DR

This paper proposes integrating calibration into fNIRS field and assess the reliability of existing models and results indicate poor calibration performance in many proposed models.

Abstract

Functional near-infrared spectroscopy (fNIRS) is a valuable non-invasive tool for monitoring brain activity. The classification of fNIRS data in relation to conscious activity holds significance for advancing our understanding of the brain and facilitating the development of brain-computer interfaces (BCI). Many researchers have turned to deep learning to tackle the classification challenges inherent in fNIRS data due to its strong generalization and robustness. In the application of fNIRS, reliability is really important, and one mathematical formulation of the reliability of confidence is calibration. However, many researchers overlook the important issue of calibration. To address this gap, we propose integrating calibration into fNIRS field and assess the reliability of existing models. Surprisingly, our results indicate poor calibration performance in many proposed models. To advance calibration development in the fNIRS field, we summarize three practical tips. Through this letter, we hope to emphasize the critical role of calibration in fNIRS research and argue for enhancing the reliability of deep learning-based predictions in fNIRS classification tasks. All data from our experimental process are openly available on GitHub.
Paper Structure (21 sections, 7 equations, 6 figures, 5 tables)

This paper contains 21 sections, 7 equations, 6 figures, 5 tables.

Table of Contents

  1. INTRODUCTION
  2. FUNCTIONAL NEAR-INFRARED SPECTROSCOPY DATASET
  3. Mental Arithmetic (MA)
  4. Unilateral Finger and Foot Tapping (UFFT)
  5. CALIBRATION ERROR
  6. Expected Calibration Error (ECE)
  7. Maximum Calibration Error (MCE)
  8. Overconfidence Error (OE)
  9. Static Calibration Error (SCE)
  10. Adaptive Calibration Error (ACE)
  11. Adaptive Calibration Error (TACE)
  12. EXPERIMENT
  13. Signal Preprocessing
  14. Training Setting
  15. Evaluation Setting
  16. ...and 6 more sections

Figures (6)

  • Figure 2: Examining the results from the last epoch of each cross-validation, and observing nearly similar accuracy and confidence levels between (a) mughal2021fnirs and (b) wang2023rethinking. However, the calibration of (b) surpasses that of (a). Section \ref{['sec:experiment']} further validates these findings, emphasizing the important role of calibration
  • Figure 3: (a) Sensor location layout for MA shin2016open. (b) MA's trial consists of an introduction period, a task period, and a rest period.
  • Figure 4: (a) Sensor location layout for UFFT bak2019open. (b) UFFT's trial consists of an introduction period, a task period, and a rest period.
  • Figure 5: LSTM and fNIRSNet on MA Dataset
  • Figure 6: CNN+LSTM and fNIRSNet on UFFT Dataset
  • ...and 1 more figures