TouchFusion: Multimodal Wristband Sensing for Ubiquitous Touch Interactions

TL;DR

It is shown that TouchFusion can enable several common touch interaction tasks, and is validated on a dataset of 100 participants, significantly exceeding the population size of typical wearable sensing studies to capture a wider variance of wrist anatomies, skin conductivities, and behavioral patterns.

Abstract

TouchFusion is a wristband that enables touch interactions on nearby surfaces without any additional instrumentation or computer vision. TouchFusion combines surface electromyography (sEMG), bioimpedance, inertial, and optical sensing to capture multiple facets of hand activity during touch interactions. Through a combination of early and late fusion, TouchFusion enables stateful touch detection on both environmental and body surfaces, simple surface gestures, and tracking functionality for contextually adaptive interfaces as well as basic trackpad-like interactions. We validate our approach on a dataset of 100 participants, significantly exceeding the population size of typical wearable sensing studies to capture a wider variance of wrist anatomies, skin conductivities, and behavioral patterns. We show that TouchFusion can enable several common touch interaction tasks. Using TouchFusion, a wearer can summon a trackpad on any surface, control contextually adaptive interfaces based on where they tap, or use their palm as an always-available touch surface. When paired with smart glasses or augmented reality devices, TouchFusion enables a ubiquitous, contextually adaptive interaction model.

Figures (21)

• Figure 1: (top) A rendering of the TouchFusion device indicating each of the major subsystems with the puck and two active straps and (bottom) a diagram of the device as seen from below. (1) Electrodes distributed around the device measure sEMG and bioimpedance. Six shared electrodes around the strap measure both sEMG and bioimpedance signals in a time-multiplexed fashion. Beneath the puck are two additional sEMG electrodes along with the sEMG reference electrode, transmit electrodes for the bioimpedance signal, and an active shield electrode to reduce interference. (2) Infrared proximity sensors around the strap measure deflection of the wrist and hand. (3) Time-of-flight sensors on the palmar side of the device track nearby potential surfaces beneath the hand. An IMU (4) and haptic actuator (5) are located in the puck with additional electronics.
• Figure 2: (top) Bioimpedance sensing principle for body touch: A constant-current differential transmitter (Tx) couples a low-current AC signal to the body. When touching the body, some of this signal is shunted off through this alternative current path. By placing differential sense electrodes (Rx) along the current path, we can measure the reduction in signal during a touch. (bottom) The bioimpedance module uses a differential transmitter operating at 1 MHz and two channels of differential sense electrodes that are processed with a synchronous detector.
• Figure 3: Example images captured by the 8x8 ToF sensor (left three images) and signals captured by infrared proximity sensors (far right). The ToF imager provides information about the hand pose relative to a nearby surface. Note that the sensor is offset from the middle of the wrist and positioned at an angle. Infrared proximity sensors around the top of the wristband provide information about wrist deflection.
• Figure 4: TouchFusion estimates touch state and tracks motion during touch events. We decompose this problem into smaller independent modules in a late-fusion approach. Each of the sensors in TouchFusion was chosen to contribute to one of these modules. This multimodal approach contributes to the breadth of TouchFusion's capabilities and improves resilience.
• Figure 5: Confusion matrix for stateful world touch (left) and body touch (right) models. Results are reported on a per-window basis.