Elementary, My Dear Watson: Non-Invasive Neural Keyword Spotting in the LibriBrain Dataset

TL;DR

This paper introduces Neural Keyword Spotting (KWS) for non-invasive brain-computer interfaces using LibriBrain MEG data, addressing the lack of standardized benchmarks for brain-based wake-word detection. It defines a fixed-lexicon KWS task, provides standardized train/validation/test splits, a robust AUPRC-based evaluation with FA/h, and releases a reference 1-D Conv/ResNet baseline along with PNPL library updates and Colab tutorials. Results show detectable neural signal for keyword detection (AUPRC ≈ 0.094, AUROC ≈ 0.80) with data-scale benefits and insights into frequency and onset timing. The work establishes a reproducible benchmark, discusses practical utility and current limitations, and outlines future directions toward multi-subject datasets, expanded vocabularies, and deployable, low-False Alarm systems.

Abstract

Non-invasive brain-computer interfaces (BCIs) are beginning to benefit from large, public benchmarks. However, current benchmarks target relatively simple, foundational tasks like Speech Detection and Phoneme Classification, while application-ready results on tasks like Brain-to-Text remain elusive. We propose Keyword Spotting (KWS) as a practically applicable, privacy-aware intermediate task. Using the deep 52-hour, within-subject LibriBrain corpus, we provide standardized train/validation/test splits for reproducible benchmarking, and adopt an evaluation protocol tailored to extreme class imbalance. Concretely, we use area under the precision-recall curve (AUPRC) as a robust evaluation metric, complemented by false alarms per hour (FA/h) at fixed recall to capture user-facing trade-offs. To simplify deployment and further experimentation within the research community, we are releasing an updated version of the pnpl library with word-level dataloaders and Colab-ready tutorials. As an initial reference model, we present a compact 1-D Conv/ResNet baseline with focal loss and top-k pooling that is trainable on a single consumer-class GPU. The reference model achieves approximately 13x the permutation baseline AUPRC on held-out sessions, demonstrating the viability of the task. Exploratory analyses reveal: (i) predictable within-subject scaling - performance improves log-linearly with more training hours - and (ii) the existence of word-level factors (frequency and duration) that systematically modulate detectability.

Figures (7)

• Figure 1: KWS task setup. The participant listens to a Sherlock Holmes audiobook while the model spots a chosen keyword (e.g., "Watson") from MEG signals.
• Figure 2: Reference model overview. A 306 $\times$T MEG window passes through a temporal convolutional trunk (with time downsampling) to produce a 128 $\times$T' representation. A projection stage feeds two temporal heads that emit per-time logits and attention scores. An attention-weighted pooling aggregates over time to a scalar logit, which is mapped to the keyword probability via a sigmoid.
• Figure 3: Keyword choice trade-offs. (Left) Relationship between word length in phonemes and token frequency across the full LibriBrain corpus (points are unique words; y-axis is log-scaled for readability; the line shows the median frequency per length). (Right) %$\Delta$AUPRC over the base rate for the shortlisted keywords as a function of their phoneme count.
• Figure 4: Left: %$\Delta$AUPRC over the empirical base rate for three similarly frequent keywords with 3, 5, and 8 phonemes, showing no significant difference. Right: Recall--FA/h operating curve (assistive, $\lambda=2$/h) with budget markers at 0.5 and 2.0 FA/h.
• Figure 5: Left: Keyword spotting scales with the amount of training data; AUPRC improves roughly log-linearly as a larger fraction of the 52-hour corpus is used. Right: Effect of temporal offsets around the keyword onset. The X marks where AUPRC peaks with a modest pre-onset context ($\sim\!0.1$s) and a slightly longer post-onset window ($\sim\!0.3$s).