PinchCatcher: Enabling Multi-selection for Gaze+Pinch

TL;DR

PinchCatcher introduces a one-handed, gaze-driven multi-selection framework for XR by leveraging a semi-pinch quasi-mode to contextualize subsequent actions. It compares four subselection triggering methods (SemiDwell, SemiSwipe, SemiTilt, SemiNDH) against a bimanual FullDH baseline, finding that SemiSwipe and SemiDwell provide the best balance of accuracy, effort, and user preference in a serial selection task. The authors validate the approach through a within-subject user study and illustrate practical use in 2D file management and a 3D RTS game, offering design guidance for future eye–hand XR interfaces. The work highlights trade-offs between input effort, error rates, and fatigue, contributing actionable insights for advancing gaze–pinch interaction in XR contexts.

Abstract

This paper investigates multi-selection in XR interfaces based on eye and hand interaction. We propose enabling multi-selection using different variations of techniques that combine gaze with a semi-pinch gesture, allowing users to select multiple objects, while on the way to a full-pinch. While our exploration is based on the semi-pinch mode for activating a quasi-mode, we explore four methods for confirming subselections in multi-selection mode, varying in effort and complexity: dwell-time (SemiDwell), swipe (SemiSwipe), tilt (SemiTilt), and non-dominant hand input (SemiNDH), and compare them to a baseline technique. In the user study, we evaluate their effectiveness in reducing task completion time, errors, and effort. The results indicate the strengths and weaknesses of each technique, with SemiSwipe and SemiDwell as the most preferred methods by participants. We also demonstrate their utility in file managing and RTS gaming application scenarios. This study provides valuable insights to advance 3D input systems in XR.

Figures (13)

• Figure 1: A diagram of a multi-selection process in 2D desktop and touch screen environments.
• Figure 2: Illustration of the three hand-pinch states used to design the multi-selection method. Each state was detected based on the distance between the index and thumb fingertips. The indicator for each state is shown on the index fingertip. (A) Full pinch is detected when both fingertips are touching, and the indicator is highlighted in green. Users were able to interact with the selected objects in this state. (B) Semi-pinch is used to enable multi-selection mode and is activated when the fingertip distance is between 2 and 7 cm, as shown in the yellow indicator. (C) Full-release pinch is a state when the distance is over 10 cm and is indicated in red. It disables the grouping of all objects.
• Figure 3: Illustration of PinchCatcher, serial multi-selection techniques utilizing the semi-pinch gesture for mode switching. While maintaining the semi-pinch gesture, the user performs a pinch-clicking motion with their non-dominant hand (SemiNDH), maintains gaze for 500 ms (SemiDwell), swipes left (SemiSwipe), or tilts right (SemiTilt) to confirm the grouping of the gazed object. A typical interaction flow involves four steps: (1) maintaining a semi-pinch to retain multi-selection mode, (2) directing gaze at a target and issuing a new command to sub-select it, (3) repeating step 2 until all targets are grouped, and (4) concluding with a full-pinch, which can then be used to drag all targets. After the interaction, users can disable the group by making a full-release pinch.
• Figure 4: The multi-selection process with PinchCatcher.
• Figure 5: Illustration of FullDH technique process. We included this technique as a baseline, which resembles the mechanism of 2D desktop multi-selection. The FullDH uses a non-dominant hand (NDH) full pinch to activate a multi-selection state instead of a semi-pinch. While maintaining the full pinch gesture, users perform pinch-clicking with their dominant hand (DH) to subselect the gazed object.