Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

iFace: Hand-Over-Face Gesture Recognition Leveraging Impedance Sensing

Mengxi Liu, Hymalai Bello, Bo Zhou, Paul Lukowicz, Jakob Karolus

TL;DR

Problem addressed: non-visual communication often misses hand-over-face cues that convey affect and cognition. Approach: a wearable impedance-sensing device placed on the shoulders (iFace) detects hand-over-face gestures without sensing on the face or hands, using shoulder electrodes and impedance variation; a lightweight CNN with user-dependent learning performs classification. Findings: eight participants, six gestures, macro F1 of 82.58% achieved; null and nose-pinching are most confusable, while boredom is most robust due to large contact area. Significance: demonstrates privacy-preserving, unobtrusive gesture recognition with practical potential for implicit interaction, non-visual communication augmentation, and cognitive-state inference in everyday devices.

Abstract

Hand-over-face gestures can provide important implicit interactions during conversations, such as frustration or excitement. However, in situations where interlocutors are not visible, such as phone calls or textual communication, the potential meaning contained in the hand-over-face gestures is lost. In this work, we present iFace, an unobtrusive, wearable impedance-sensing solution for recognizing different hand-over-face gestures. In contrast to most existing works, iFace does not require the placement of sensors on the user's face or hands. Instead, we proposed a novel sensing configuration, the shoulders, which remains invisible to both the user and outside observers. The system can monitor the shoulder-to-shoulder impedance variation caused by gestures through electrodes attached to each shoulder. We evaluated iFace in a user study with eight participants, collecting six kinds of hand-over-face gestures with different meanings. Using a convolutional neural network and a user-dependent classification, iFace reaches 82.58 \% macro F1 score. We discuss potential application scenarios of iFace as an implicit interaction interface.

iFace: Hand-Over-Face Gesture Recognition Leveraging Impedance Sensing

TL;DR

Problem addressed: non-visual communication often misses hand-over-face cues that convey affect and cognition. Approach: a wearable impedance-sensing device placed on the shoulders (iFace) detects hand-over-face gestures without sensing on the face or hands, using shoulder electrodes and impedance variation; a lightweight CNN with user-dependent learning performs classification. Findings: eight participants, six gestures, macro F1 of 82.58% achieved; null and nose-pinching are most confusable, while boredom is most robust due to large contact area. Significance: demonstrates privacy-preserving, unobtrusive gesture recognition with practical potential for implicit interaction, non-visual communication augmentation, and cognitive-state inference in everyday devices.

Abstract

Hand-over-face gestures can provide important implicit interactions during conversations, such as frustration or excitement. However, in situations where interlocutors are not visible, such as phone calls or textual communication, the potential meaning contained in the hand-over-face gestures is lost. In this work, we present iFace, an unobtrusive, wearable impedance-sensing solution for recognizing different hand-over-face gestures. In contrast to most existing works, iFace does not require the placement of sensors on the user's face or hands. Instead, we proposed a novel sensing configuration, the shoulders, which remains invisible to both the user and outside observers. The system can monitor the shoulder-to-shoulder impedance variation caused by gestures through electrodes attached to each shoulder. We evaluated iFace in a user study with eight participants, collecting six kinds of hand-over-face gestures with different meanings. Using a convolutional neural network and a user-dependent classification, iFace reaches 82.58 \% macro F1 score. We discuss potential application scenarios of iFace as an implicit interaction interface.
Paper Structure (18 sections, 5 figures, 1 table)

This paper contains 18 sections, 5 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. Design Choices for In-Face Gestures
  4. Choice of Sensor Modality
  5. Choice of Sensor Location
  6. Choice of Gestures
  7. Implementation
  8. Sensing Principle
  9. Hardware Design
  10. System Evaluation
  11. Data Collection
  12. Data Processing
  13. Classification Result and Discussion
  14. Potential Applications for Impedance-Based Hand-over-Face Gesture Recognition
  15. Enriching Implicit Interaction Through iFace
  16. ...and 3 more sections

Figures (5)

  • Figure 1: Example of hand-over-face gestures
  • Figure 2: Sensing principle of iFace (Null class: the impedance of shoulder-to-shoulder includes only head and trunk part. Hand-over-Face gestures: as the new current path from shoulder to head via arm is built, the arm impedance will be added to the impedance between shoulder-to-shoulder). Electrodes are hidden under clothing and cannot be observed from the outside.
  • Figure 3: An example of raw signals from iFace (sampling rate is 20 Hz)
  • Figure 4: Data processing pipeline for hand-over-face gesture recognition. The impedance data includes two channels: magnitude and phase, which are preprocessed by channel-wise normalization method before being inputted to the neural network
  • Figure 5: Joint confusion matrices for hand-over-face gesture recognition from eight subjects together