SonarWatch: Field sensing technique for smartwatches based on ultrasound and motion

TL;DR

SonarWatch introduces a field-sensing approach for smartwatches that combines an ultrasonic acoustic field with IMU data to recognize a diverse set of gestures in three interaction subspaces: opposite-side, same-side, and body/object interactions. Built entirely on existing smartwatch sensors, it uses a high-frequency chirp, feature extraction from time-frequency domains and motion data, and a LightGBM classifier, achieving 93.7% accuracy across 12 gestures and 97.6% for same-side gestures, with robust performance across noise environments and real-world use. The work provides a detailed interaction design space, a rigorous data collection and processing pipeline, and a comprehensive evaluation of recognition accuracy, energy consumption, and potential limitations, highlighting practical applications in accessibility and cross-device control. Overall, SonarWatch demonstrates practical, energy-efficient field-based sensing for over-screen smartwatch interaction, with clear pathways for extension and real-world deployment.

Abstract

A smartwatch worn continuously on the wrist has the potential to perceive rich interactive gestures and natural behaviors of the user. Unfortunately, the current interaction functionality of smartwatches is primarily limited by the small touch screen. This paper proposes SonarWatch, a novel sensing technique that uses the acoustic field generated by the transceiver on the opposite sides of the watch to detect the presence of nearby objects and their shapes. This enables a range of gesture interactions and natural behavior perception. We designed an algorithm combining IMU and acoustic fields to identify these actions and optimize power consumption. We tested the performance of SonarWatch in different noise environments, achieving an overall accuracy of 93.7%. Its power consumption is close to that of physiological sensors. SonarWatch can achieve the above capabilities by utilizing the existing built-in sensors, making it a technology with solid practical value.

Figures (12)

• Figure 1: The sensor distribution of the SonarWatch, consistent with the Mi watch, has a speaker on the left, a microphone on the right, and a nine-axis IMU inside. A microphone was added below the screen to explore more sensing possibilities
• Figure 2: The sensor distribution when the user wears the watch (left) and the complete SonarWatch hardware system (right)
• Figure 3: The signal when the user performs Wrist Up. The left image contains the overall acceleration of the three axes and Euler angle from the IMU, the amplitude and the short-term energy signal of the audio. The right image contains the STFT (2048 in each window) magnitude of the audio signal. The red line in the figure divides the signal segments for the user to relax the wrist and tilt the wrist.
• Figure 4: The signal when the user taps two microphones for five times. The image contains the overall acceleration of the three and Euler angle from IMU, the amplitude, short-term energy, and the STFT magnitude of the two audio chanels. The two red boxes mark the one of the signal fragments that user performs a single gesture.
• Figure 5: 15 gestures selected for SonarWatch. The gestures in the red box are the implicit input of the user, and the others are the user's explicit input. Dots mark dynamic gestures. Asterisks mark novel gestures.