Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SIGMA: An Open-Source Interactive System for Mixed-Reality Task Assistance Research

Dan Bohus, Sean Andrist, Nick Saw, Ann Paradiso, Ishani Chakraborty, Mahdi Rad

TL;DR

Sigma tackles the lack of open, end-to-end platforms for researching mixed-reality procedural task guidance with AI. It provides an open-source, end-to-end client-server system running on HoloLens 2 that offloads perception and AI inference to a server, enabling flexible integration of LLMs and large vision models. Core contributions include an extensible architecture, a task-recipe framework (predefined and auto-generated), modular perception modules (Detic/SEEM), and a development toolchain for data logging and visualization. The platform catalyzes reproducible, interactive evaluations of AI in MR settings and lowers entry barriers for researchers to build, extend, and compare new models in real-world tasks.

Abstract

We introduce an open-source system called SIGMA (short for "Situated Interactive Guidance, Monitoring, and Assistance") as a platform for conducting research on task-assistive agents in mixed-reality scenarios. The system leverages the sensing and rendering affordances of a head-mounted mixed-reality device in conjunction with large language and vision models to guide users step by step through procedural tasks. We present the system's core capabilities, discuss its overall design and implementation, and outline directions for future research enabled by the system. SIGMA is easily extensible and provides a useful basis for future research at the intersection of mixed reality and AI. By open-sourcing an end-to-end implementation, we aim to lower the barrier to entry, accelerate research in this space, and chart a path towards community-driven end-to-end evaluation of large language, vision, and multimodal models in the context of real-world interactive applications.

SIGMA: An Open-Source Interactive System for Mixed-Reality Task Assistance Research

TL;DR

Sigma tackles the lack of open, end-to-end platforms for researching mixed-reality procedural task guidance with AI. It provides an open-source, end-to-end client-server system running on HoloLens 2 that offloads perception and AI inference to a server, enabling flexible integration of LLMs and large vision models. Core contributions include an extensible architecture, a task-recipe framework (predefined and auto-generated), modular perception modules (Detic/SEEM), and a development toolchain for data logging and visualization. The platform catalyzes reproducible, interactive evaluations of AI in MR settings and lowers entry barriers for researchers to build, extend, and compare new models in real-world tasks.

Abstract

We introduce an open-source system called SIGMA (short for "Situated Interactive Guidance, Monitoring, and Assistance") as a platform for conducting research on task-assistive agents in mixed-reality scenarios. The system leverages the sensing and rendering affordances of a head-mounted mixed-reality device in conjunction with large language and vision models to guide users step by step through procedural tasks. We present the system's core capabilities, discuss its overall design and implementation, and outline directions for future research enabled by the system. SIGMA is easily extensible and provides a useful basis for future research at the intersection of mixed reality and AI. By open-sourcing an end-to-end implementation, we aim to lower the barrier to entry, accelerate research in this space, and chart a path towards community-driven end-to-end evaluation of large language, vision, and multimodal models in the context of real-world interactive applications.
Paper Structure (17 sections, 5 figures, 1 table)

This paper contains 17 sections, 5 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. System Functionality
  4. System Implementation
  5. Architectural Overview
  6. Client Application
  7. Sensor capture
  8. User Interface
  9. Server Application
  10. Speech Recognition
  11. Large Language Model Queries
  12. Object Tracking
  13. Interaction Controller
  14. Development Tools
  15. Discussion
  16. ...and 2 more sections

Figures (5)

  • Figure 1: Left: user performing a procedural task with a mixed-reality headset running Sigma. Middle: first-person view showing Sigma guidance panel and task-specific holograms. Right: visualization of system's scene understanding showing the egocentric camera view, depth map, detected objects, gaze, hand and head pose in 3D space. © 2024 IEEE
  • Figure 2: An illustrative example of an interaction with the Sigma system. (portions © 2024 IEEE)
  • Figure 3: First-person view during a cooking task showing a spatially-placed timer hologram.
  • Figure 4: Sigma uses a client-server architecture that bypasses computational limits on device and simplifies development efforts.
  • Figure 5: Sigma data visualization constructed using Platform for Situated Intelligence Studio. Left panel: composite 2D visualization containing image and object detection streams. Right panel: composite 3D visualization containing camera image view stream, head coordinate system, gaze direction (orange), UI placement (blue rectangles), depth point cloud, and sub-pointclouds for detected objects. Bottom: timeline visualizations for audio, speech synthesis and speech recognition result streams.