SigmaCollab: An Application-Driven Dataset for Physically Situated Collaboration
Dan Bohus, Sean Andrist, Ann Paradiso, Nick Saw, Tim Schoonbeek, Maia Stiber
TL;DR
SigmaCollab addresses the need for ecologically valid research on physically situated human–AI collaboration by providing an interactive, multimodal dataset captured as participants use a mixed-reality assistant to complete diverse procedures. The dataset is collected with the open-source Sigma platform on HoloLens 2, incorporating audio, egocentric video, depth, gaze, and pose data, plus post-hoc annotations. It enables benchmarks that test real-time coordination, grounding, and cognitive-state understanding in end-to-end tasks, moving beyond static, non-interactive datasets. The work demonstrates the feasibility of application-driven data collection and highlights opportunities to study proactive interventions, self-talk, and end-to-end system performance in realistic settings.
Abstract
We introduce SigmaCollab, a dataset enabling research on physically situated human-AI collaboration. The dataset consists of a set of 85 sessions in which untrained participants were guided by a mixed-reality assistive AI agent in performing procedural tasks in the physical world. SigmaCollab includes a set of rich, multimodal data streams, such as the participant and system audio, egocentric camera views from the head-mounted device, depth maps, head, hand and gaze tracking information, as well as additional annotations performed post-hoc. While the dataset is relatively small in size (~ 14 hours), its application-driven and interactive nature brings to the fore novel research challenges for human-AI collaboration, and provides more realistic testing grounds for various AI models operating in this space. In future work, we plan to use the dataset to construct a set of benchmarks for physically situated collaboration in mixed-reality task assistive scenarios. SigmaCollab is available at https://github.com/microsoft/SigmaCollab.