Gazeify Then Voiceify: Physical Object Referencing Through Gaze and Voice Interaction with Displayless Smart Glasses

TL;DR

This work addresses how to reference real-world objects using displayless smart glasses that provide no visual feedback. It introduces Gazeify Then Voiceify, a multimodal pipeline that uses gaze to generate object masks and a vision-language model to produce audible descriptions, enabling user-initiated corrections via natural speech. The system integrates an open-vocabulary detection and segmentation stack with a retrieval-based language-driven refinement to update masks based on user commands, and validates the approach with a user study showing 53% gaze accuracy and 58% success in voice-driven disambiguation, along with favorable usability ratings. The findings suggest that gaze-based selection can support everyday interactions with ordinary-sized, simply structured objects in clean environments, while emphasizing the need for efficient, latency-aware models to handle more complex scenes and distant targets.

Abstract

Smart glasses enhance interactions with the environment by using head-mounted cameras to observe the user's viewpoint, but lack the visual feedback used for common interactions. We introduce Gazeify then Voiceify, a multimodal approach allowing object selection via gaze and voice using displayless smart glasses. Users can select a physical object with their gaze, and the system generates a digital mask and a voice description of the object's semantics. Users can further correct errors through free-form conversation. To demonstrate our approach, we develop an interactive system by integrating advanced object segmentation and detection with a vision-language model. User studies reveal that participants achieve correct gaze selection in 53% of the task trials and use voice disambiguation to correct 58% of the remaining errors. Participants also rated the system as likable, useful, and easy to use.

Figures (11)

• Figure 1: The interaction flow and system architecture illustration. (1) User targets object and activates selection with a pinch; (2) User's gaze unintentionally shifts off target, the red dot represents gaze; (3) Gazeify incorrectly segments an object; (4) Gazeify outputs an incorrect object mask; (5) Voiceify provides a description of the selection; (6) User detects the error and issues a voice command for correction; (7) Upon gaze selection activation, the system identifies comprehensive objects in the focal view; (8) VLM filters out noisy detections; (9) Candidate object masks are generated and sent to Voiceify; (10) VLM processes the user's command; (11) Heuristic filtering uses spatial relations identified by VLM; (12) VLM selects the most likely object from candidates based on the user command and notifies the user.
• Figure 2: The illustration of different disambiguation strategies that users can utilize. Users can specify object property, hierarchical, spatial (ordinal) relationships for instructing the system to correct selection.
• Figure 3: Images of the study environment, where participants were asked to select objects using Gazeify Then Voiceify on the table, storage shelf, desk surface, and bookshelf.
• Figure 4: Gaze referencing accuracy when different amounts of environmental clutter, perceived object sizes, and ambiguities are present.
• Figure 5: Distribution of gaze referencing error types by trial condition.