A Multi-decoder Neural Tracking Method for Accurately Predicting Speech Intelligibility

TL;DR

The multi-decoder method is introduced to predict speech reception thresholds (SRTs) from EEG recordings, enabling objective assessment for populations unable to perform behavioral tests; such as those with disorders of consciousness or during hearing aid fitting.

Abstract

Objective: EEG-based methods can predict speech intelligibility, but their accuracy and robustness lag behind behavioral tests, which typically show test-retest differences under 1 dB. We introduce the multi-decoder method to predict speech reception thresholds (SRTs) from EEG recordings, enabling objective assessment for populations unable to perform behavioral tests; such as those with disorders of consciousness or during hearing aid fitting. Approach: The method aggregates data from hundreds of decoders, each trained on different speech features and EEG preprocessing setups to quantify neural tracking (NT) of speech signals. Using data from 39 participants (ages 18-24), we recorded 29 minutes of EEG per person while they listened to speech at six signal-to-noise ratios and a quiet story. NT values were combined into a high-dimensional feature vector per subject, and a support vector regression model was trained to predict SRTs from these vectors. Main Result: Predictions correlated significantly with behavioral SRTs (r = 0.647, p < 0.001; NRMSE = 0.19), with all differences under 1 dB. SHAP analysis showed theta/delta bands and early lags had slightly greater influence. Using pretrained subject-independent decoders reduced required EEG data collection to 15 minutes (3 minutes of story, 12 minutes across six SNR conditions) without losing accuracy.

Figures (7)

• Figure 1: Overview of the experimental procedure. Each participant completed three main tasks: a behavioral task in which they listened to matrix sentences masked by speech-weighted noise and repeated them to determine their behavioral ; followed by two tasks involving passive listening—one to a narrated story, and the other to matrix sentences presented with varying levels of speech-weighted noise. Feature extraction focused on the envelope and acoustic onsets of the clean stimuli. Both the stimulus features and the recordings from the narrated story and matrix sentences were then used to train and test hundreds of different decoder configurations. These decoders were subsequently combined using our proposed multi-decoder method, which integrates information from all decoders to predict the , allowing comparison with the behavioral .
• Figure 2: Subject-independent and subject-specific decoders. (A) For the decoder, leave-one-subject-out cross-validation is applied; this panel depicts the process for a single subject, representing one fold of the cross-validation. Two decoders are shown: one trained on data from the story-listening task ( story decoder) and another trained on data from a specific condition of the matrix listening task ( matrix decoder). Each condition includes two matrix sentence lists, totaling 40 sentences. (B) For the decoder, leave-one-out cross-validation was performed on matrix sentences within a single condition. Similar to the decoders, two decoders were trained: one on story data ( story decoder) and one on data from 39 of the 40 matrix sentences ( matrix decoder). After reconstructing the features for all 40 sentences across the cross-validation folds, the reconstructed sentence features were concatenated into a single feature vector.
• Figure 3: ERF-adjusted neural tracking vectors. (A) Displays values for seven conditions using a specific decoder configuration (e.g., theta-band, story listening, stimulus envelope, lags up to 250 ms). Seven decoders—one per condition—were trained and tested using the same decoder configuration, resulting in seven neural tracking values corresponding to each condition. The value at –12.5 serves as the noise baseline, and the silence condition is excluded from further analysis. (B) Adjusted values are computed by subtracting the noise baseline (–12.5 ) from the values at the remaining conditions. (C) Each decoder configuration produces its own set of adjusted values across the . These values from all decoder configurations are concatenated into a single comprehensive adjusted vector. The alternating grey and white backgrounds highlight the adjusted values corresponding to the same decoder configuration across different . For clarity, the decoder type parameter is not shown in this illustration. (D) Each subject's adjusted vector is transformed into an -adjusted vector using the erf.
• Figure 4: Model performance results. (A) Correlation ($p < 0.001$) between the behavioral ($_{behavioral}$) and the predicted ($_{eeg}$). The blue points are subjects, the red line is the regression fit, and the dotted line is the identity line. (B) Histogram of the absolute differences between $_{behavioral}$ and $_{eeg}$.
• Figure 5: SHAP analysis results. Each point represents the value of a -adjusted value for a specific parameter used in the decoder configuration. values are grouped as follows: (A) values are organized by parameter group. If a SHAP value corresponds to multiple parameters (e.g., both and ), it appears in each relevant parameter group. (B) values are grouped according to the and decoders type.