Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

GAZEploit: Remote Keystroke Inference Attack by Gaze Estimation from Avatar Views in VR/MR Devices

Hanqiu Wang, Zihao Zhan, Haoqi Shan, Siqi Dai, Max Panoff, Shuo Wang

TL;DR

GAZEploit is unveiled, a novel eye-tracking based attack specifically designed to exploit these eye-tracking information by leveraging the common use of virtual appearances in VR applications, and takes advantage of this vulnerability to remotely extract gaze estimations and steal sensitive keystroke information across various typing scenarios.

Abstract

The advent and growing popularity of Virtual Reality (VR) and Mixed Reality (MR) solutions have revolutionized the way we interact with digital platforms. The cutting-edge gaze-controlled typing methods, now prevalent in high-end models of these devices, e.g., Apple Vision Pro, have not only improved user experience but also mitigated traditional keystroke inference attacks that relied on hand gestures, head movements and acoustic side-channels. However, this advancement has paradoxically given birth to a new, potentially more insidious cyber threat, GAZEploit. In this paper, we unveil GAZEploit, a novel eye-tracking based attack specifically designed to exploit these eye-tracking information by leveraging the common use of virtual appearances in VR applications. This widespread usage significantly enhances the practicality and feasibility of our attack compared to existing methods. GAZEploit takes advantage of this vulnerability to remotely extract gaze estimations and steal sensitive keystroke information across various typing scenarios-including messages, passwords, URLs, emails, and passcodes. Our research, involving 30 participants, achieved over 80% accuracy in keystroke inference. Alarmingly, our study also identified over 15 top-rated apps in the Apple Store as vulnerable to the GAZEploit attack, emphasizing the urgent need for bolstered security measures for this state-of-the-art VR/MR text entry method.

GAZEploit: Remote Keystroke Inference Attack by Gaze Estimation from Avatar Views in VR/MR Devices

TL;DR

GAZEploit is unveiled, a novel eye-tracking based attack specifically designed to exploit these eye-tracking information by leveraging the common use of virtual appearances in VR applications, and takes advantage of this vulnerability to remotely extract gaze estimations and steal sensitive keystroke information across various typing scenarios.

Abstract

The advent and growing popularity of Virtual Reality (VR) and Mixed Reality (MR) solutions have revolutionized the way we interact with digital platforms. The cutting-edge gaze-controlled typing methods, now prevalent in high-end models of these devices, e.g., Apple Vision Pro, have not only improved user experience but also mitigated traditional keystroke inference attacks that relied on hand gestures, head movements and acoustic side-channels. However, this advancement has paradoxically given birth to a new, potentially more insidious cyber threat, GAZEploit. In this paper, we unveil GAZEploit, a novel eye-tracking based attack specifically designed to exploit these eye-tracking information by leveraging the common use of virtual appearances in VR applications. This widespread usage significantly enhances the practicality and feasibility of our attack compared to existing methods. GAZEploit takes advantage of this vulnerability to remotely extract gaze estimations and steal sensitive keystroke information across various typing scenarios-including messages, passwords, URLs, emails, and passcodes. Our research, involving 30 participants, achieved over 80% accuracy in keystroke inference. Alarmingly, our study also identified over 15 top-rated apps in the Apple Store as vulnerable to the GAZEploit attack, emphasizing the urgent need for bolstered security measures for this state-of-the-art VR/MR text entry method.
Paper Structure (30 sections, 9 equations, 20 figures, 4 tables)

This paper contains 30 sections, 9 equations, 20 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Eye trackers for VR/MR and Text Entry
  4. Virtual Avatar and Virtual Camera
  5. Appearance-based Eye Gaze Estimation
  6. Threat Model
  7. Attack Scenarios
  8. Attack Vectors on Various Typing Scenarios
  9. Limitations on Attack Scenarios
  10. Preliminary Attack Analysis
  11. Eye-related Biometrics Extraction
  12. Typing Activity Identification
  13. Adaptive Virtual Keyboard Layout Mapping
  14. Top K Keystroke Candidate Accuracy
  15. Passcode Inference
  16. ...and 15 more sections

Figures (20)

  • Figure 1: (a) A victim is logging into a Google account while sharing the Persona. (b) An attacker viewing the victim's Persona from a live video on X.
  • Figure 2: Camera access indicators on different platforms (a) Apple Vision Pro's subtle green dot that can be covered by any other window. (b) iPhone's camera access indicator. (c) MacBook's system menu bar camera indicator.
  • Figure 3: Keyboards used on Apple Vision Pro (a) Passcode(PIN) keyboard (b) Default QWERTY keyboard (c) Number and special character keyboard
  • Figure 4: GAZEploit Attack Overview: The attacker starts by extracting facial biometrics from the victim's virtual representation, focusing on gaze estimation and eye aspect ratios (EARs). These biometrics are then used to distinguish typing sessions from other activities and to identify the timing of each keystroke. Finally, the attacker maps the gaze vectors to a virtual keyboard to infer the pressed keys.
  • Figure 5: Bidirectional RNN Architecture for classifying typing sessions
  • ...and 15 more figures