Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Auditory Brain Passage Retrieval: Cross-Sensory EEG Training for Neural Information Retrieval

Niall McGuire, Yashar Moshfeghi

TL;DR

The paper tackles the challenge of translating internal information needs into retrieval actions by evaluating auditory EEG for Brain Passage Retrieval (BPR) and exploring cross-sensory training with visual data. It uses a dual-encoder setup with four pooling strategies and InfoNCE-based contrastive learning to map auditory EEG queries to passage embeddings, comparing auditory (Alice) and visual (Nieuwland) datasets and their combination. Key findings show that auditory EEG can achieve retrieval performance that rivals traditional text baselines, and cross-sensory training with CLS pooling yields substantial improvements over modality-specific models, addressing data scarcity and enabling accessible voice-based IR interfaces. These results highlight the viability of neural queries for IR and suggest architecture-aware strategies for multi-modal neural data in retrieval tasks, with implications for hearing-impaired users and audio-centric search scenarios.

Abstract

Query formulation from internal information needs remains fundamentally challenging across all Information Retrieval paradigms due to cognitive complexity and physical impairments. Brain Passage Retrieval (BPR) addresses this by directly mapping EEG signals to passage representations without intermediate text translation. However, existing BPR research exclusively uses visual stimuli, leaving critical questions unanswered: Can auditory EEG enable effective retrieval for voice-based interfaces and visually impaired users? Can training on combined EEG datasets from different sensory modalities improve performance despite severe data scarcity? We present the first systematic investigation of auditory EEG for BPR and evaluate cross-sensory training benefits. Using dual encoder architectures with four pooling strategies (CLS, mean, max, multi-vector), we conduct controlled experiments comparing auditory-only, visual-only, and combined training on the Alice (auditory) and Nieuwland (visual) datasets. Results demonstrate that auditory EEG consistently outperforms visual EEG, and cross-sensory training with CLS pooling achieves substantial improvements over individual training: 31% in MRR (0.474), 43% in Hit@1 (0.314), and 28% in Hit@10 (0.858). Critically, combined auditory EEG models surpass BM25 text baselines (MRR: 0.474 vs 0.428), establishing neural queries as competitive with traditional retrieval whilst enabling accessible interfaces. These findings validate auditory neural interfaces for IR tasks and demonstrate that cross-sensory training addresses data scarcity whilst outperforming single-modality approaches Code: https://github.com/NiallMcguire/Audio_BPR

Auditory Brain Passage Retrieval: Cross-Sensory EEG Training for Neural Information Retrieval

TL;DR

The paper tackles the challenge of translating internal information needs into retrieval actions by evaluating auditory EEG for Brain Passage Retrieval (BPR) and exploring cross-sensory training with visual data. It uses a dual-encoder setup with four pooling strategies and InfoNCE-based contrastive learning to map auditory EEG queries to passage embeddings, comparing auditory (Alice) and visual (Nieuwland) datasets and their combination. Key findings show that auditory EEG can achieve retrieval performance that rivals traditional text baselines, and cross-sensory training with CLS pooling yields substantial improvements over modality-specific models, addressing data scarcity and enabling accessible voice-based IR interfaces. These results highlight the viability of neural queries for IR and suggest architecture-aware strategies for multi-modal neural data in retrieval tasks, with implications for hearing-impaired users and audio-centric search scenarios.

Abstract

Query formulation from internal information needs remains fundamentally challenging across all Information Retrieval paradigms due to cognitive complexity and physical impairments. Brain Passage Retrieval (BPR) addresses this by directly mapping EEG signals to passage representations without intermediate text translation. However, existing BPR research exclusively uses visual stimuli, leaving critical questions unanswered: Can auditory EEG enable effective retrieval for voice-based interfaces and visually impaired users? Can training on combined EEG datasets from different sensory modalities improve performance despite severe data scarcity? We present the first systematic investigation of auditory EEG for BPR and evaluate cross-sensory training benefits. Using dual encoder architectures with four pooling strategies (CLS, mean, max, multi-vector), we conduct controlled experiments comparing auditory-only, visual-only, and combined training on the Alice (auditory) and Nieuwland (visual) datasets. Results demonstrate that auditory EEG consistently outperforms visual EEG, and cross-sensory training with CLS pooling achieves substantial improvements over individual training: 31% in MRR (0.474), 43% in Hit@1 (0.314), and 28% in Hit@10 (0.858). Critically, combined auditory EEG models surpass BM25 text baselines (MRR: 0.474 vs 0.428), establishing neural queries as competitive with traditional retrieval whilst enabling accessible interfaces. These findings validate auditory neural interfaces for IR tasks and demonstrate that cross-sensory training addresses data scarcity whilst outperforming single-modality approaches Code: https://github.com/NiallMcguire/Audio_BPR
Paper Structure (12 sections, 6 equations, 4 figures, 2 tables)

This paper contains 12 sections, 6 equations, 4 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Task Formulation
  5. Datasets and Materials
  6. Model Architecture
  7. Experimental Setup
  8. Evaluation Methodology
  9. Results & Discussion
  10. Auditory EEG Demonstrates Effective Brain Passage Retrieval Performance
  11. Cross-Sensory Training Benefits Manifest Through Architecture-Dependent Mechanisms
  12. Conclusion

Figures (4)

  • Figure 1: EEG signals recorded during visual or auditory stimulus presentation serve as brain queries (qe), which are encoded alongside text passages (p). Cosine similarity between encoded representations produces relevance scores for passage ranking and retrieval.
  • Figure 2: Inverse Cloze Task (ICT) overview: During stimulus presentation (audio or visual), EEG signals are recorded synchronously with text comprehension. Query spans $q_e$ comprising 30% of the total passage length are extracted along with their corresponding neural responses. Positive passages $P^+$ are created by masking the query span from the original passage with probability $p_{\text{mask}} = 0.9$.
  • Figure 3: cross-sensory training effects comparing individual versus combined training for Auditory (Alice, top row) and Visual(Nieuwland, bottom row) datasets across MRR, Hit@1, and Hit@10 metrics. Individual training (blue) uses only modality-specific data, whilst combined training (orange) leverages both auditory and visual datasets. Grey bars show document pool sizes at each masking level.
  • Figure 4: Architectural comparison of pooling strategies (CLS, MEAN, MAX, MULTI) on combined training data across three key metrics (MRR, Hit@1, Hit@10). Performance is evaluated at six masking levels (0-100%). Grey bars indicate document pool sizes.