GazeSwipe: Enhancing Mobile Touchscreen Reachability through Seamless Gaze and Finger-Swipe Integration

TL;DR

GazeSwipe introduces a handheld gaze-plus-swipe interaction to mitigate one-handed reachability issues on large mobile screens without extra hardware. It combines front-facing camera-based gaze estimation with a user-unaware auto-calibration pipeline and a touch-drag-release refinement to improve targeting accuracy, snapping to nearest elements and enabling fine-tuning via swipes. Across two user studies, AC2 auto-calibration consistently improves gaze accuracy over explicit calibration, while GazeSwipe achieves higher success rates and user preference than Direct Touch, One-handed Mode, and Pure Cursor on both smartphones and tablets, with tablet benefits being most pronounced. The approach offers a practical, calibration-free path to more accessible one-handed mobile interaction and suggests directions for tighter integration of gaze with multimodal input in everyday devices.

Abstract

Smartphones with large screens provide users with increased display and interaction space but pose challenges in reaching certain areas with the thumb when using the device with one hand. To address this, we introduce GazeSwipe, a multimodal interaction technique that combines eye gaze with finger-swipe gestures, enabling intuitive and low-friction reach on mobile touchscreens. Specifically, we design a gaze estimation method that eliminates the need for explicit gaze calibration. Our approach also avoids the use of additional eye-tracking hardware by leveraging the smartphone's built-in front-facing camera. Considering the potential decrease in gaze accuracy without dedicated eye trackers, we use finger-swipe gestures to compensate for any inaccuracies in gaze estimation. Additionally, we introduce a user-unaware auto-calibration method that improves gaze accuracy during interaction. Through extensive experiments on smartphones and tablets, we compare our technique with various methods for touchscreen reachability and evaluate the performance of our auto-calibration strategy. The results demonstrate that our method achieves high success rates and is preferred by users. The findings also validate the effectiveness of the auto-calibration strategy.

Figures (9)

• Figure 1: Overview of the GazeSwipe method. Our approach is designed to tackle the reachability challenges encountered in one-handed mobile touchscreen usage scenarios. It involves feeding facial images captured by the front-facing camera into a lightweight neural network to estimate the gaze point and display a cursor accordingly. Additionally, the system performs user-unaware auto-calibration based on historical interaction samples, improving gaze accuracy over time. To complete interactions, users gaze at the target and fine-tune the gaze cursor position using a "touch-drag-release" approach.
• Figure 2: User interface during the studies. A 12×6 grid of square targets is randomly generated, comprising of two different sizes. Left: Smartphone interface (shown with GazeSwipe). Participants fixate their gaze on a target, causing the cursor to appear near it. They then swipe the cursor to the target, which turns green once reached. Right: Tablet interface (shown with Pure Cursor).
• Figure 3: Example of gaze scanpaths collected during interactions. (a) At system startup, gaze errors are relatively large, requiring longer finger swipes. (b-c) After a few interactions, the length of finger swipes progressively decreases. (d) With prolonged use, the gaze can directly snap to the target, and a simple tap is sufficient to confirm the interaction.
• Figure 4: Gaze error for the four calibration strategies. Results on the smartphone (left) and on the tablet (right). The four strategies include: No Calibration (NC), Explicit Calibration (EC) using a 9-point calibration process, Auto-calibration Strategy 1 (AC1) without head pose, and Auto-calibration Strategy 2 (AC2) incorporating head pose. Error bars represent standard deviations, with significant differences marked by ** ($p < 0.05$).
• Figure 5: Gaze error for each participant. The error varies across participants before calibration but stabilizes after calibration, particularly with the two auto-calibration strategies.