Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Aria Gen 2 Pilot Dataset

Chen Kong, James Fort, Aria Kang, Jonathan Wittmer, Simon Green, Tianwei Shen, Yipu Zhao, Cheng Peng, Gustavo Solaira, Andrew Berkovich, Nikhil Raina, Vijay Baiyya, Evgeniy Oleinik, Eric Huang, Fan Zhang, Julian Straub, Mark Schwesinger, Luis Pesqueira, Xiaqing Pan, Jakob Julian Engel, Carl Ren, Mingfei Yan, Richard Newcombe

TL;DR

The paper introduces the Aria Gen 2 Pilot Dataset (A2PD), an open, multimodal egocentric dataset built on the newer Aria Gen 2 glasses to advance perception, contextual AI, and robotics research. It documents an initial release collected over a weekend from four participants across five everyday scenarios, providing rich raw sensor streams, on-device perception outputs, and offline MPS and algorithmic results, including VIO, eye and hand tracking, diarization, depth, and 3D object detection. The work provides a detailed description of data modalities, collection methodology, processing pipelines, and tooling to download, load, and visualize the data, enabling researchers to benchmark and develop on-device perception and multimodal fusion. By releasing A2PD incrementally with accompanying tooling and open formats, the authors aim to foster reproducible research and accelerate progress in egocentric perception for real-world contexts.

Abstract

The Aria Gen 2 Pilot Dataset (A2PD) is an egocentric multimodal open dataset captured using the state-of-the-art Aria Gen 2 glasses. To facilitate timely access, A2PD is released incrementally with ongoing dataset enhancements. The initial release features Dia'ane, our primary subject, who records her daily activities alongside friends, each equipped with Aria Gen 2 glasses. It encompasses five primary scenarios: cleaning, cooking, eating, playing, and outdoor walking. In each of the scenarios, we provide comprehensive raw sensor data and output data from various machine perception algorithms. These data illustrate the device's ability to perceive the wearer, the surrounding environment, and interactions between the wearer and the environment, while maintaining robust performance across diverse users and conditions. The A2PD is publicly available at projectaria.com, with open-source tools and usage examples provided in Project Aria Tools.

Aria Gen 2 Pilot Dataset

TL;DR

The paper introduces the Aria Gen 2 Pilot Dataset (A2PD), an open, multimodal egocentric dataset built on the newer Aria Gen 2 glasses to advance perception, contextual AI, and robotics research. It documents an initial release collected over a weekend from four participants across five everyday scenarios, providing rich raw sensor streams, on-device perception outputs, and offline MPS and algorithmic results, including VIO, eye and hand tracking, diarization, depth, and 3D object detection. The work provides a detailed description of data modalities, collection methodology, processing pipelines, and tooling to download, load, and visualize the data, enabling researchers to benchmark and develop on-device perception and multimodal fusion. By releasing A2PD incrementally with accompanying tooling and open formats, the authors aim to foster reproducible research and accelerate progress in egocentric perception for real-world contexts.

Abstract

The Aria Gen 2 Pilot Dataset (A2PD) is an egocentric multimodal open dataset captured using the state-of-the-art Aria Gen 2 glasses. To facilitate timely access, A2PD is released incrementally with ongoing dataset enhancements. The initial release features Dia'ane, our primary subject, who records her daily activities alongside friends, each equipped with Aria Gen 2 glasses. It encompasses five primary scenarios: cleaning, cooking, eating, playing, and outdoor walking. In each of the scenarios, we provide comprehensive raw sensor data and output data from various machine perception algorithms. These data illustrate the device's ability to perceive the wearer, the surrounding environment, and interactions between the wearer and the environment, while maintaining robust performance across diverse users and conditions. The A2PD is publicly available at projectaria.com, with open-source tools and usage examples provided in Project Aria Tools.
Paper Structure (14 sections, 5 figures)

This paper contains 14 sections, 5 figures.

Table of Contents

  1. Introduction
  2. Dataset Description
  3. Dataset Content
  4. Raw Sensor Data
  5. On-Device Machine Perception Results
  6. Machine Perception Services Results
  7. Additional Perception Algorithms Results
  8. Diarization
  9. Heart rate estimation
  10. Hand Object Interaction
  11. 3D Object Detection
  12. Depth Estimation
  13. Dataset Tools
  14. Release Plan

Figures (5)

  • Figure 1: Plot of estimated heart rate together with IMU signals from the walking sequence. One can observe that the heart rate stays elevated during the walk, and then returns to the normal resting level afterward.
  • Figure 2: Example of detecting hand object interaction by segmenting hands (left hand in green, right hand in magenta) and the interacted objects (yellow).
  • Figure 3: Example of 3D bounding boxes estimated by EVL straub2024efm3d projected back on RGB images and visualized together with semi-dense point clouds from MPS.
  • Figure 4: An example of generated depth images generated by Foundation Stereo wen2025foundationstereo with corresponding rectified CV images and 3D point cloud overlaid on detected 3D bounding boxes.
  • Figure 5: Screenshots of the raw sensor visualizer and machine perception visualizer.