Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

WearVox: An Egocentric Multichannel Voice Assistant Benchmark for Wearables

Zhaojiang Lin, Yong Xu, Kai Sun, Jing Zheng, Yin Huang, Surya Teja Appini, Krish Narang, Renjie Tao, Ishan Kapil Jain, Siddhant Arora, Ruizhi Li, Yiteng Huang, Kaushik Patnaik, Wenfang Xu, Suwon Shon, Yue Liu, Ahmed A Aly, Anuj Kumar, Florian Metze, Xin Luna Dong

TL;DR

WearVox introduces the first egocentric, multichannel benchmark for wearable voice assistants, capturing realistic, noisy, and interactive wearable scenarios across five practical tasks. By evaluating a wide range of SLLMs and a multichannel belajar, it demonstrates substantial gaps between current models and wearable-era requirements, highlights the value of spatial audio cues, and shows that multichannel inputs improve robustness to ambient noise and task discrimination. The benchmark’s two-tier evaluation (turn-based QA/tool/side-talk and session-based translation) paired with LLM judge validation provides a rigorous framework for end-to-end wearable voice AI development. WearVox thus offers a realistic testbed to drive advances in context-aware, low-latency wearable assistants capable of handling side-talk, tool invocation, and cross-language translation in real-world environments.

Abstract

Wearable devices such as AI glasses are transforming voice assistants into always-available, hands-free collaborators that integrate seamlessly with daily life, but they also introduce challenges like egocentric audio affected by motion and noise, rapid micro-interactions, and the need to distinguish device-directed speech from background conversations. Existing benchmarks largely overlook these complexities, focusing instead on clean or generic conversational audio. To bridge this gap, we present WearVox, the first benchmark designed to rigorously evaluate voice assistants in realistic wearable scenarios. WearVox comprises 3,842 multi-channel, egocentric audio recordings collected via AI glasses across five diverse tasks including Search-Grounded QA, Closed-Book QA, Side-Talk Rejection, Tool Calling, and Speech Translation, spanning a wide range of indoor and outdoor environments and acoustic conditions. Each recording is accompanied by rich metadata, enabling nuanced analysis of model performance under real-world constraints. We benchmark leading proprietary and open-source speech Large Language Models (SLLMs) and find that most real-time SLLMs achieve accuracies on WearVox ranging from 29% to 59%, with substantial performance degradation on noisy outdoor audio, underscoring the difficulty and realism of the benchmark. Additionally, we conduct a case study with two new SLLMs that perform inference with single-channel and multi-channel audio, demonstrating that multi-channel audio inputs significantly enhance model robustness to environmental noise and improve discrimination between device-directed and background speech. Our results highlight the critical importance of spatial audio cues for context-aware voice assistants and establish WearVox as a comprehensive testbed for advancing wearable voice AI research.

WearVox: An Egocentric Multichannel Voice Assistant Benchmark for Wearables

TL;DR

WearVox introduces the first egocentric, multichannel benchmark for wearable voice assistants, capturing realistic, noisy, and interactive wearable scenarios across five practical tasks. By evaluating a wide range of SLLMs and a multichannel belajar, it demonstrates substantial gaps between current models and wearable-era requirements, highlights the value of spatial audio cues, and shows that multichannel inputs improve robustness to ambient noise and task discrimination. The benchmark’s two-tier evaluation (turn-based QA/tool/side-talk and session-based translation) paired with LLM judge validation provides a rigorous framework for end-to-end wearable voice AI development. WearVox thus offers a realistic testbed to drive advances in context-aware, low-latency wearable assistants capable of handling side-talk, tool invocation, and cross-language translation in real-world environments.

Abstract

Wearable devices such as AI glasses are transforming voice assistants into always-available, hands-free collaborators that integrate seamlessly with daily life, but they also introduce challenges like egocentric audio affected by motion and noise, rapid micro-interactions, and the need to distinguish device-directed speech from background conversations. Existing benchmarks largely overlook these complexities, focusing instead on clean or generic conversational audio. To bridge this gap, we present WearVox, the first benchmark designed to rigorously evaluate voice assistants in realistic wearable scenarios. WearVox comprises 3,842 multi-channel, egocentric audio recordings collected via AI glasses across five diverse tasks including Search-Grounded QA, Closed-Book QA, Side-Talk Rejection, Tool Calling, and Speech Translation, spanning a wide range of indoor and outdoor environments and acoustic conditions. Each recording is accompanied by rich metadata, enabling nuanced analysis of model performance under real-world constraints. We benchmark leading proprietary and open-source speech Large Language Models (SLLMs) and find that most real-time SLLMs achieve accuracies on WearVox ranging from 29% to 59%, with substantial performance degradation on noisy outdoor audio, underscoring the difficulty and realism of the benchmark. Additionally, we conduct a case study with two new SLLMs that perform inference with single-channel and multi-channel audio, demonstrating that multi-channel audio inputs significantly enhance model robustness to environmental noise and improve discrimination between device-directed and background speech. Our results highlight the critical importance of spatial audio cues for context-aware voice assistants and establish WearVox as a comprehensive testbed for advancing wearable voice AI research.
Paper Structure (40 sections, 3 equations, 7 figures, 6 tables)

This paper contains 40 sections, 3 equations, 7 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Voice Assistant Benchmarks.
  4. Speech LLMs.
  5. WearVox Dataset
  6. Problem Definition
  7. Tasks Formulation
  8. Speaker Roles
  9. Acoustic Conditions
  10. Indoor Environments:
  11. Outdoor and Mobile Scenarios:
  12. Noise Diversity and Signal-to-Noise Ratios:
  13. Data Collection
  14. Script Collection
  15. Egocentric Audio Recording
  16. ...and 25 more sections

Figures (7)

  • Figure 1: Examples of tasks from the WearVox dataset. The audio queries are recorded with AI glasses (transcribed in blue). The ground truth for each task is provided in text format.
  • Figure 2: Illustration of SC Wearllama and MC Wearllama inference. SC Wearllama encodes only the beamformed audio channel (c_x), whereas MC Wearllama processes both channel 0 (c_0),typically the channel with the highest SNR, and the beamformed channel in an interleaved manner.
  • Figure 3: Effect of acoustic environment on SLLM performance in turn-based tasks.
  • Figure 4: Ego-centric audio recordings, with conversational partners positioned between $-60^\circ$ and $60^\circ$. Bystanders may speak from any angle.
  • Figure 5: Audio Recording Distribution
  • ...and 2 more figures