Intent at a Glance: Gaze-Guided Robotic Manipulation via Foundation Models

TL;DR

This work tackles the problem of intuitive, scalable human-robot interaction for assistive tabletop manipulation using gaze as the primary input. It introduces GAMMA, a pipeline that grounds egocentric gaze in the scene and uses vision-language foundation models to infer user intent $I_g$ and synthesize a task plan $P$ comprising atomic skills $s(o,a)$ for a robotic arm, enabling zero-shot manipulation without task-specific training. The system integrates Aria-based gaze sensing, SAM2-based object segmentation, Contact-GraspNet grasp generation, and VLM-based reasoning (intent and grasp selection) to provide context-aware actions. Experiments and a user study show GAMMA reduces task time and effort compared to a panel-based gaze baseline, but users often prefer direct control for agency. Limitations include grasp reliability, inference latency, and the potential of hybrid control modes for more natural interaction.

Abstract

Designing intuitive interfaces for robotic control remains a central challenge in enabling effective human-robot interaction, particularly in assistive care settings. Eye gaze offers a fast, non-intrusive, and intent-rich input modality, making it an attractive channel for conveying user goals. In this work, we present GAMMA (Gaze Assisted Manipulation for Modular Autonomy), a system that leverages ego-centric gaze tracking and a vision-language model to infer user intent and autonomously execute robotic manipulation tasks. By contextualizing gaze fixations within the scene, the system maps visual attention to high-level semantic understanding, enabling skill selection and parameterization without task-specific training. We evaluate GAMMA on a range of table-top manipulation tasks and compare it against baseline gaze-based control without reasoning. Results demonstrate that GAMMA provides robust, intuitive, and generalizable control, highlighting the potential of combining foundation models and gaze for natural and scalable robot autonomy. Project website: https://gamma0.vercel.app/

Figures (10)

• Figure 1: Overview of gamma. A user wearing smart glasses uses their gaze to specify a manipulation task. In this example, the user wants the robot to pick up a plant under the lamp and place it in a tray. gamma transforms the gaze fixations into robot's view and prompts the VLM to predict user intent. Given the predicted user intent, gamma calls corresponding functions for perception, planning, and execution. gamma prompts a VLM to select a proper grasping pose that takes the task context into consideration (e.g. not colliding with the lamp).
• Figure 1: Example scenarios used in the intent recognition task. (a) An example of easy tasks. (b) An example of medium-difficulty tasks. (c) An example of hard tasks.
• Figure 2: Functional Modules of gamma.gamma consists of various sensing & perception modules that leverages pretrained vision models, and VLM-based reasoning modules.
• Figure 2: Candidate Grasp Poses
• Figure 3: Gaze-based Intent Reasoning Tasks. (Top) We designed 30 tabletop manipulation scenarios in lab for intent reasoning with diverse difficulty levels. Easy scenes are relatively clean, medium difficulty-level scenes are cluttered or contain longer sequences of gaze points. The hard cases involves visual attacks. (Bottom) We also randomly sampled 45 scenes from the DROID khazatsky2024droid dataset containing in-the-wild manipulation tasks and annotated the scenes for intent reasoning.