Beyond Hearing: Learning Task-agnostic ExG Representations from Earphones via Physiology-informed Tokenization

TL;DR

This work tackles the lack of diverse real-world ExG data and the task-specific bottlenecks in ExG models by introducing NeuroBuds, an earphone-based sensing platform, and Physiology-informed Multi-band Tokenization (PiMT). PiMT decomposes ExG signals into $N_F=12$ physiology-aligned sub-bands to produce structured tokens processed by a Bidirectional-Mamba encoder, with self-supervised reconstruction-based pre-training on 50 hours of free-living data followed by task-specific fine-tuning. On the DailySense dataset and four public benchmarks, PiMT delivers strong cross-task generalization, outperforming state-of-the-art baselines and showing robust performance even in cross-subject settings, aided by multi-band tokenization and large-scale pre-training. These results suggest a scalable path toward real-world, foundation-model-like ExG sensing across multiple senses, with potential applications in health monitoring and cognitive interfaces.

Abstract

Electrophysiological (ExG) signals offer valuable insights into human physiology, yet building foundation models that generalize across everyday tasks remains challenging due to two key limitations: (i) insufficient data diversity, as most ExG recordings are collected in controlled labs with bulky, expensive devices; and (ii) task-specific model designs that require tailored processing (i.e., targeted frequency filters) and architectures, which limit generalization across tasks. To address these challenges, we introduce an approach for scalable, task-agnostic ExG monitoring in the wild. We collected 50 hours of unobtrusive free-living ExG data with an earphone-based hardware prototype to narrow the data diversity gap. At the core of our approach is Physiology-informed Multi-band Tokenization (PiMT), which decomposes ExG signals into 12 physiology-informed tokens, followed by a reconstruction task to learn robust representations. This enables adaptive feature recognition across the full frequency spectrum while capturing task-relevant information. Experiments on our new DailySense dataset-the first to enable ExG-based analysis across five human senses-together with four public ExG benchmarks, demonstrate that PiMT consistently outperforms state-of-the-art methods across diverse tasks.

Figures (9)

• Figure 1: Overview of PiMT. ExG signals are decomposed into 12 sub-bands via Physiology-informed Multi-band Tokenization (PiFT). A Bidirectional-Mamba encoder processes the tokens, and the model is pre-trained with reconstruction tasks before fine-tuning on downstream tasks.
• Figure 2: Overview of DailySense dataset. Using our earphone-based ExG analysis device, NeuroBuds, we collect free-living unlabeled data for task-agnostic pre-training and labeled data spanning five human senses, serving as benchmarks for downstream tasks.
• Figure 3: Comparison of different ExG tokenization strategies: 1-band (0.1$\sim$75 Hz), 2-band (0.1$\sim$15 Hz and 15$\sim$75 Hz), 4-band (0.1$\sim$5 Hz, 5$\sim$15 Hz, 15$\sim$35 Hz, and 35$\sim$75 Hz), and our 12-band filter bank (described in Section \ref{['sec:multi_band_tokenization']}).
• Figure 4: Saliency analysis demonstrating how the model dynamically captures task-relevant frequency bands via multi-band tokenization.
• Figure 5: Test loss across different pre-training data scales.