MHAD: Multimodal Home Activity Dataset with Multi-Angle Videos and Synchronized Physiological Signals
Lei Yu, Jintao Fei, Xinyi Liu, Yang Yao, Jun Zhao, Guoxin Wang, Xin Li
TL;DR
The MHAD dataset tackles the lack of real-home, multi-angle data for video-based physiology by providing 1,440 30-second videos from three angles for 40 subjects performing six everyday activities, with synchronized five physiological signals captured by gold-standard BIOPAC sensors. The work benchmarks both unsupervised (ICA, POS, PBVP, optical flow for respiration) and supervised (EfficientPhys, TS-CANTSCAN, DeepPhys) methods across angles and activities, revealing limited generalization of supervised models to side angles and the relative strength of unsupervised respiration estimation. Key contributions include the first public multi-angle, real-home video-based physiology dataset with rich signals and a comprehensive evaluation across modalities, highlighting the importance of diverse angles and realistic tasks for passive home monitoring. MHAD thus enables better training and evaluation of video-based physiological methods in home-like conditions and supports the development of broader signal extraction beyond pulse and respiration, such as SpO2 and blood pressure.
Abstract
Video-based physiology, exemplified by remote photoplethysmography (rPPG), extracts physiological signals such as pulse and respiration by analyzing subtle changes in video recordings. This non-contact, real-time monitoring method holds great potential for home settings. Despite the valuable contributions of public benchmark datasets to this technology, there is currently no dataset specifically designed for passive home monitoring. Existing datasets are often limited to close-up, static, frontal recordings and typically include only 1-2 physiological signals. To advance video-based physiology in real home settings, we introduce the MHAD dataset. It comprises 1,440 videos from 40 subjects, capturing 6 typical activities from 3 angles in a real home environment. Additionally, 5 physiological signals were recorded, making it a comprehensive video-based physiology dataset. MHAD is compatible with the rPPG-toolbox and has been validated using several unsupervised and supervised methods. Our dataset is publicly available at https://github.com/jdh-algo/MHAD-Dataset.