Lip-Siri: Contactless Open-Sentence Silent Speech with Wi-Fi Backscatter

TL;DR

Lip-Siri introduces a contactless silent speech interface that uses frequency-shifted Wi-Fi backscatter to sense lip motions and decode open-ended sentences. It combines a robust signal-processing pipeline, segmentation and clustering of lip-motion units with cluster-based self-supervised pretraining, and a lexicon-guided Transformer decoder with beam search to map continuous lip dynamics to subword sequences. The system achieves 85.61% word accuracy and 36.87% WER on 340 sentences from 15 participants, approaching the performance of vision-based lip-reading systems while maintaining privacy and low power. This work expands silent interaction to open vocabulary, reduces user instrumentation, and demonstrates practical deployment in real indoor environments with low latency, enabling privacy-preserving silent input in everyday settings.

Abstract

Silent speech interfaces (SSIs) enable silent interaction in noise-sensitive or privacy-sensitive settings. However, existing SSIs face practical deployment trade-offs among privacy, user experience, and energy consumption, and most remain limited to closed-set recognition over small, pre-defined vocabularies of words or sentences, which restricts real-world expressiveness. In this paper, we present Lip-Siri, to the best of our knowledge, the first Wi-Fi backscatter--based SSI that supports open-vocabulary sentence recognition via lexicon-guided subword decoding. Lip-Siri designs a frequency-shifted backscatter tag to isolate tag-modulated reflections and suppress interference from non-target motions, enabling reliable extraction of lip-motion traces from ubiquitous Wi-Fi signals. We then segment continuous traces into lip-motion units, cluster them, learn robust unit representations via cluster-based self-supervision, and finally propose a lexicon-guided Transformer encoder--decoder with beam search to decode variable-length sentence sequences. We implement an end-to-end prototype and evaluate it with 15 participants on 340 sentences and 3,398 words across multiple scenarios. Lip-Siri achieves 85.61% accuracy on word prediction and a WER of 36.87% on continuous sentence recognition, approaching the performance of representative vision-based lip-reading systems.

Figures (13)

• Figure 1: The concept and use cases of Lip-Siri: to provide non-contact and privacy protection silent lip-reading services for people, especially patients with speech disorders. It is also suitable for quiet situations that require silence or are inconvenient to speak, such as libraries, offices, dormitories, etc.
• Figure 2: Comparison of normalized short-time energy curves for example vowel and consonant silent-articulation signals. Negative values indicate energy below the baseline due to normalization.
• Figure 3: Illustration and theoretical model of silent articulation for wireless sensing. (a) Key articulators and the main active region during silent articulation. (b) Representative mouth shapes that produce distinct lip-motion patterns. (c) Geometric motion model of lip movements and the resulting Doppler variation in the reflected signal.
• Figure 4: Phonemes /a:/ and //.
• Figure 5: System overview of Lip-Siri: Lip-Siri first extracts lip-motion traces from Wi-Fi backscatter signals and suppresses motion interference, then learns features of lip-motion units via segmentation, clustering, and self-supervised pretraining, and finally performs lexicon-based subword sequence decoding to enable open-sentence silent speech inference.