STAR: Smartphone-analogous Typing in Augmented Reality

TL;DR

STAR addresses the challenge of efficient AR text entry by transferring smartphone two-thumb typing to a bare-hand AR setting. It introduces a knuckle-triggered, on-skin QWERTY keyboard, implemented with index-finger surface typing, stationary keyboard placement, capacitive thumb taps, and visual depth cues, plus a probabilistic word-suggestion decoder. In a controlled study with ten participants, STAR achieved a mean speed of $21.9$ WPM (about $56\%$ of smartphone speed) and a mean uncorrected error rate of $0.2\%$, highlighting the key role of improved hand tracking and sensing in narrowing the gap to smartphone typing. The work lays out a practical design space and future directions for real-world deployment, including richer sensing (rings, wearables) and expanded modes (one-handed STAR), to enable robust, socially acceptable AR text entry.

Abstract

While text entry is an essential and frequent task in Augmented Reality (AR) applications, devising an efficient and easy-to-use text entry method for AR remains an open challenge. This research presents STAR, a smartphone-analogous AR text entry technique that leverages a user's familiarity with smartphone two-thumb typing. With STAR, a user performs thumb typing on a virtual QWERTY keyboard that is overlain on the skin of their hands. During an evaluation study of STAR, participants achieved a mean typing speed of 21.9 WPM (i.e., 56% of their smartphone typing speed), and a mean error rate of 0.3% after 30 minutes of practice. We further analyze the major factors implicated in the performance gap between STAR and smartphone typing, and discuss ways this gap could be narrowed.

Figures (7)

• Figure 1: Hand postures during two-handed physical smartphone use that were observed during Hoober's studies (image from steven2022designing, with permission).
• Figure 2: Results of the elicitation study depicting the skin surfaces and hand postures that users would use for imaginary smartphone typing. These are (a) Single Palm Surface with one hand and tapping with a finger from the other hand (n = 2), (b) Nested Finger Surface with both hands folded and tapping with both thumbs (n = 20) and (c) Index Finger Surface with the hands in a symmetric contact and tapping with both thumbs (n = 6). One participant used both thumbs while separating their hands, which did not fit into any category.
• Figure 3: The three design parameters explored during the design phase for STAR: (a) the use of hand surfaces for two-thumb typing such as the Index Finger Surface and Nested Finger Surface, (b) different Positionings such as Hands-Following and Stationary, and (c) different Sizes such as Original (i.e., the iPhone XR's default keyboard layout) and Enlarged. Boldfaced options were chosen for the final STAR design.
• Figure 4: The observed (a) tracking latency and (b) tracking jitter of the thumb tip position from the Hololens 2 hand tracking. The tracking latency was measured while moving the hand approximately 200 mm along a single axis, whereas the measured position was approximated from the recorded video. (i.e., A ruler was put at a fixed position in the recorded video, and the tracked thumb was moved in line with the ruler.) The tracking jitter was measured for 5 seconds while the hand was held stationary on an armrest.
• Figure 5: A typical interaction sequence while using STAR. (a) A user triggers STAR typing by making a knuckle posture. (b) The user performs familiar two-thumb typing on their Index Finger Surface with visualized thumbs and projection arcs. (c) Finally, the user exits the typing mode by releasing the knuckle posture. (d) The clear view of the projection arc and visualized thumb joints.