WS-IMUBench: Can Weakly Supervised Methods from Audio, Image, and Video Be Adapted for IMU-based Temporal Action Localization?

TL;DR

This work investigates scalable weakly supervised temporal action localization for IMU data (WS-IMU-TAL) by reframing it as a Multiple Instance Learning problem and transferring representative weakly supervised localization paradigms from audio, image, and video to continuous IMU streams. It introduces WS-IMUBench, a standardized benchmarking framework evaluated on seven public IMU datasets using seven methods (plus three fully supervised baselines), with LOSO and in-domain settings and two inference modes. Key findings show modality-dependent transferability—temporal-domain methods from audio/video generally outperform image-derived WSOD approaches—while weak supervision is most effective on datasets with longer actions and richer sensing; major bottlenecks include short actions, temporal ambiguity, and proposal quality. The paper outlines concrete directions for improvement (IMU-specific proposals, boundary-aware objectives, stronger temporal reasoning) and provides a reproducible template to accelerate progress in WS-IMU-TAL.

Abstract

IMU-based Human Activity Recognition (HAR) has enabled a wide range of ubiquitous computing applications, yet its dominant clip classification paradigm cannot capture the rich temporal structure of real-world behaviors. This motivates a shift toward IMU Temporal Action Localization (IMU-TAL), which predicts both action categories and their start/end times in continuous streams. However, current progress is strongly bottlenecked by the need for dense, frame-level boundary annotations, which are costly and difficult to scale. To address this bottleneck, we introduce WS-IMUBench, a systematic benchmark study of weakly supervised IMU-TAL (WS-IMU-TAL) under only sequence-level labels. Rather than proposing a new localization algorithm, we evaluate how well established weakly supervised localization paradigms from audio, image, and video transfer to IMU-TAL under only sequence-level labels. We benchmark seven representative weakly supervised methods on seven public IMU datasets, resulting in over 3,540 model training runs and 7,080 inference evaluations. Guided by three research questions on transferability, effectiveness, and insights, our findings show that (i) transfer is modality-dependent, with temporal-domain methods generally more stable than image-derived proposal-based approaches; (ii) weak supervision can be competitive on favorable datasets (e.g., with longer actions and higher-dimensional sensing); and (iii) dominant failure modes arise from short actions, temporal ambiguity, and proposal quality. Finally, we outline concrete directions for advancing WS-IMU-TAL (e.g., IMU-specific proposal generation, boundary-aware objectives, and stronger temporal reasoning). Beyond individual results, WS-IMUBench establishes a reproducible benchmarking template, datasets, protocols, and analyses, to accelerate community-wide progress toward scalable WS-IMU-TAL.

Figures (9)

• Figure 1: In established domains like weakly supervised sound event detection from audio (WSSED), weakly supervised object detection from images (WSOD), and weakly supervised video-based action localization (WSVAL), Weakly Supervised Learning has proven highly effective. These methods all use only coarse, audio/image/video-level labels to infer fine-grained boundaries (e.g., bounding boxes or temporal onset/offset). Can this successful paradigm be analogously applied to IMU data to achieve Weakly Supervised Temporal Action Localization (WS-IMU-TAL)?
• Figure 2: An overview of the Multiple Instance Learning (MIL) pipeline. Block diagram of a MIL pipeline: an untrimmed sequence (bag) is decomposed into candidate segments (instances), scored by an instance classifier, aggregated into a bag-level prediction, and trained with a loss against the bag label.
• Figure 3: Network diagram of the CDur model: CNN feature extractor followed by a BiGRU for temporal modeling, with two output branches producing sequence-level predictions and upsampled slice-level predictions.
• Figure 4: Our WSDDN-adapted architecture for IMU action localization. After feature extraction, each temporal proposal is processed by two parallel branches: a classification stream for class scores ($\mathbf{c}_p$) and a detection stream for localization weights ($\mathbf{l}_p$). The final score for each proposal is their element-wise product, and the sum of all scores is supervised by the sequence-level bag label.
• Figure 5: Comparison of proposal generation strategies. (a) Selective Search for Images: This algorithm leverages rich visual cues (color, texture) to intelligently propose a sparse set of candidate object regions, avoiding an inefficient brute-force search. (b) Our Multi-Scale Temporal Sampling for IMU Data: Lacking visual cues, our method adapts this concept to the time domain. It systematically samples temporal segments across multiple predefined durations to generate a diverse set of candidate instances, serving as an efficient, content-agnostic alternative for 1D signals. (see Algorithm \ref{['alg:proposal_boxes']}.)