Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

World Mouse: Exploring Interactions with a Cross-Reality Cursor

Esen K. Tütüncü, Mar Gonzalez-Franco, Khushman Patel, Eric J. Gonzalez

TL;DR

This work introduces World Mouse, a cross-reality cursor that reinterprets the familiar 2D desktop mouse for complex 3D scenes and illustrates how cross-reality cursors may enable seamless interactions across real and virtual environments.

Abstract

As Extended Reality (XR) systems increasingly map and understand the physical world, interacting with these blended representations remains challenging. The current push for "natural" inputs has its trade-offs: touch is limited by human reach and fatigue, while gaze often lacks the precision for fine interaction. To bridge this gap, we introduce World Mouse, a cross-reality cursor that reinterprets the familiar 2D desktop mouse for complex 3D scenes. The system is driven by two core mechanisms: within-object interaction, which uses surface normals for precise cursor placement, and between-object navigation, which leverages interpolation to traverse empty space. Unlike previous virtual-only approaches, World Mouse leverages semantic segmentation and mesh reconstruction to treat physical objects as interactive surfaces. Through a series of prototypes, including object manipulation and screen-to-world transitions, we illustrate how cross-reality cursors may enable seamless interactions across real and virtual environments.

World Mouse: Exploring Interactions with a Cross-Reality Cursor

TL;DR

This work introduces World Mouse, a cross-reality cursor that reinterprets the familiar 2D desktop mouse for complex 3D scenes and illustrates how cross-reality cursors may enable seamless interactions across real and virtual environments.

Abstract

As Extended Reality (XR) systems increasingly map and understand the physical world, interacting with these blended representations remains challenging. The current push for "natural" inputs has its trade-offs: touch is limited by human reach and fatigue, while gaze often lacks the precision for fine interaction. To bridge this gap, we introduce World Mouse, a cross-reality cursor that reinterprets the familiar 2D desktop mouse for complex 3D scenes. The system is driven by two core mechanisms: within-object interaction, which uses surface normals for precise cursor placement, and between-object navigation, which leverages interpolation to traverse empty space. Unlike previous virtual-only approaches, World Mouse leverages semantic segmentation and mesh reconstruction to treat physical objects as interactive surfaces. Through a series of prototypes, including object manipulation and screen-to-world transitions, we illustrate how cross-reality cursors may enable seamless interactions across real and virtual environments.
Paper Structure (19 sections, 3 figures)

This paper contains 19 sections, 3 figures.

Table of Contents

  1. Introduction
  2. Background & Related Work
  3. Precision in HCI
  4. XR Interaction Techniques
  5. Cursor-based Interactions in XR
  6. World Mouse Design and Implementation
  7. Bringing the 2D Mouse into 3D Space
  8. Within Object Navigation
  9. Between Objects Navigation
  10. 2D to 3D Cursor Transition
  11. Bringing the Mouse into the Real World
  12. Prototype Explorations
  13. The Desktop Metaphor in 3D Space
  14. 3D Authoring and Manipulation
  15. Cross-Reality and Ambient Interaction
  16. ...and 4 more sections

Figures (3)

  • Figure 1: Cursor behaviors explored through the World Mouse. Left (Upper): Between Objects Navigation leverages interpolation allowing the cursor to traverse through the space between objects. Left (Lower): Within Object Navigation allows the cursor to move along the surface of any object. Right: 2D to 3D Cursor Transition enables a smooth visual transition when using moving from 2D panels like browsers to a 3D objects.
  • Figure 2: World Mouse Application Scenarios. (Left) Spatial Authoring: Users can spawn virtual objects and manipulate them using high-precision 3D gizmos, including spline editing and vertex-snapping to real-world meshes. (Middle) Cross-Device & IoT Control: Digital content can be transitioned from physical screens into the environment, while IoT devices are controlled via interactive proxies and re-rendered reality filters. (Right) Social XR: The system supports collaborative workflows allowing multiple users to interact with shared virtual assets.
  • Figure 3: Controlling the World Mouse using the touchscreen of a smartphone (left) or smartwatch (right) via XDTK gonzalez2024xdtk.