Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Hovering Over the Key to Text Input in XR

Mar Gonzalez-Franco, Diar Abdlkarim, Arpit Bhatia, Stuart Macgregor, Jason Alexander Fotso-Puepi, Eric J Gonzalez, Hasti Seifi, Massimiliano Di Luca, Karan Ahuja

TL;DR

This work analyzes the challenges of efficient text input in XR and emphasizes keyboards—both physical and virtual—as central to enabling productive XR use. It surveys design constraints (display resolution, FOV, finger tracking, hand representations) and explores three keyboard paradigms (physical, mid-air virtual, and surface-anchored virtual), plus ML-enabled decoding to boost accuracy. Key contributions include a synthesis of design trends, identification of major hurdles (occlusion, feedback, fatigue), and proposed hybrid approaches that combine duplicated keyboards and ML-assisted decoding. The findings highlight the potential to significantly uplift XR productivity by tailoring keyboard modalities to context, device capabilities, and user embodiment, thereby accelerating the adoption of XR productivity tools and immersive experiences.

Abstract

Virtual, Mixed, and Augmented Reality (XR) technologies hold immense potential for transforming productivity beyond PC. Therefore there is a critical need for improved text input solutions for XR. However, achieving efficient text input in these environments remains a significant challenge. This paper examines the current landscape of XR text input techniques, focusing on the importance of keyboards (both physical and virtual) as essential tools. We discuss the unique challenges and opportunities presented by XR, synthesizing key trends from existing solutions.

Hovering Over the Key to Text Input in XR

TL;DR

This work analyzes the challenges of efficient text input in XR and emphasizes keyboards—both physical and virtual—as central to enabling productive XR use. It surveys design constraints (display resolution, FOV, finger tracking, hand representations) and explores three keyboard paradigms (physical, mid-air virtual, and surface-anchored virtual), plus ML-enabled decoding to boost accuracy. Key contributions include a synthesis of design trends, identification of major hurdles (occlusion, feedback, fatigue), and proposed hybrid approaches that combine duplicated keyboards and ML-assisted decoding. The findings highlight the potential to significantly uplift XR productivity by tailoring keyboard modalities to context, device capabilities, and user embodiment, thereby accelerating the adoption of XR productivity tools and immersive experiences.

Abstract

Virtual, Mixed, and Augmented Reality (XR) technologies hold immense potential for transforming productivity beyond PC. Therefore there is a critical need for improved text input solutions for XR. However, achieving efficient text input in these environments remains a significant challenge. This paper examines the current landscape of XR text input techniques, focusing on the importance of keyboards (both physical and virtual) as essential tools. We discuss the unique challenges and opportunities presented by XR, synthesizing key trends from existing solutions.
Paper Structure (15 sections, 1 figure)

This paper contains 15 sections, 1 figure.

Table of Contents

  1. Introduction
  2. Design Considerations
  3. Display Resolution
  4. Field of View
  5. Finger Tracking and Typing
  6. Hand Representations
  7. Flexible Keyboard Layouts
  8. Other considerations
  9. Opportunities for Keyboards in XR
  10. The Case for a Physical Keyboard
  11. The Need for a Virtual Keyboard
  12. Mid-air Virtual Keyboards
  13. Surface-anchored Virtual Keyboards
  14. ML-enabled Keyboards
  15. Discussion and Conclusions

Figures (1)

  • Figure 1: Ergonomic Issues of mid air touch include lack of precision and ergonomic issues with physical surface include field of view limitations. And additionally challenges on hand tracking from an egocentric view. As the finger tips are easily occluded by the rest of the hand.