Hierarchical and Multimodal Data for Daily Activity Understanding

TL;DR

DARai introduces a large-scale, open-source multimodal dataset for daily activity understanding that is hierarchically annotated across three levels of granularity (L1: activities, L2: actions, L3: procedures). It combines 20 data modalities from 12+ devices—including RGB/depth cameras, IMUs, EMG, insole pressure, biomonitoring, and gaze—to study recognition, temporal localization, and action anticipation in realistic, unscripted settings with counterfactual variations. The paper provides a comprehensive benchmark suite covering unimodal and multimodal fusion, cross-view and cross-body robustness, and temporal dependencies, highlighting strengths and limitations of each modality and the benefits of hierarchical modeling. DARai demonstrates the importance of multimodal, hierarchical data for robust real-world activity understanding and offers datasets, benchmarks, and code to advance research in human-centered AI applications.

Abstract

Daily Activity Recordings for Artificial Intelligence (DARai, pronounced "Dahr-ree") is a multimodal, hierarchically annotated dataset constructed to understand human activities in real-world settings. DARai consists of continuous scripted and unscripted recordings of 50 participants in 10 different environments, totaling over 200 hours of data from 20 sensors including multiple camera views, depth and radar sensors, wearable inertial measurement units (IMUs), electromyography (EMG), insole pressure sensors, biomonitor sensors, and gaze tracker. To capture the complexity in human activities, DARai is annotated at three levels of hierarchy: (i) high-level activities (L1) that are independent tasks, (ii) lower-level actions (L2) that are patterns shared between activities, and (iii) fine-grained procedures (L3) that detail the exact execution steps for actions. The dataset annotations and recordings are designed so that 22.7% of L2 actions are shared between L1 activities and 14.2% of L3 procedures are shared between L2 actions. The overlap and unscripted nature of DARai allows counterfactual activities in the dataset. Experiments with various machine learning models showcase the value of DARai in uncovering important challenges in human-centered applications. Specifically, we conduct unimodal and multimodal sensor fusion experiments for recognition, temporal localization, and future action anticipation across all hierarchical annotation levels. To highlight the limitations of individual sensors, we also conduct domain-variant experiments that are enabled by DARai's multi-sensor and counterfactual activity design setup. The code, documentation, and dataset are available at the dedicated DARai website: https://alregib.ece.gatech.edu/software-and-datasets/darai-daily-activity-recordings-for-artificial-intelligence-and-machine-learning/

Figures (32)

• Figure 1: DARai multimodal data collected to capture and reflect counterfactual variations in activities across different modalities. Hierarchical structure of DARai data, demonstrated using an example activity: Moving an Object. Two variations are shown, moving a heavy box (blue) and a light box (green), along with their corresponding lower hierarchy levels. For instance, variations in insole pressure and forearm muscle interactions are observed when lifting or carrying a box.
• Figure 2: A sequence of frames depicting an example of an annotated activity sample: L1 activity 'Sleeping' with a sequence of L2 actions: 'Prepare for Activity' and 'Rest', and L3 procedures: 'Lay Down' and 'Use Blanket'. Boundaries indicate the start and end times of each annotation level within the corresponding upper-level video sample.
• Figure 4: Overview of machine learning applications using the multimodal hierarchical DARai dataset. This figure highlights key aspects explored in this work, including temporal modeling, multimodal modeling, and hierarchical modeling, as well as potential applications for the machine learning community as direct or combined downstream tasks of these approaches.
• Figure 5: Illustration of the unique information provided by various data modalities, highlighting the distinct insights each contributes to modeling human activity and state patterns.
• Figure 6: Comparison of same-view and cross-view inference performance for ResNet, MViT-s, and Swin-t using visual data. The results show top-1 accuracy for each model when tested on the same camera view and cross views.