DETACH : Decomposed Spatio-Temporal Alignment for Exocentric Video and Ambient Sensors with Staged Learning

TL;DR

This work tackles the challenge of aligning exocentric video with ambient sensors, addressing the shortcomings of Global Alignment in exocentric-ambient settings. DETACH decomposes each modality into spatial and temporal components, first establishing spatial grounding via online sensor clustering and cross-modal supervision, then performing temporally fine-grained alignment with a spatial-temporal weighted contrastive loss that emphasizes hard negatives while mitigating false negatives. The method yields state-of-the-art results on Opportunity++ and HWU-USP, significantly outperforming adapted egocentric–wearable baselines and demonstrating robustness across sensor encoders. The approach offers a scalable, non-intrusive pathway for multimodal action understanding in ambient environments, with practical implications for privacy-preserving monitoring and large-scale deployment.

Abstract

Aligning egocentric video with wearable sensors have shown promise for human action recognition, but face practical limitations in user discomfort, privacy concerns, and scalability. We explore exocentric video with ambient sensors as a non-intrusive, scalable alternative. While prior egocentric-wearable works predominantly adopt Global Alignment by encoding entire sequences into unified representations, this approach fails in exocentric-ambient settings due to two problems: (P1) inability to capture local details such as subtle motions, and (P2) over-reliance on modality-invariant temporal patterns, causing misalignment between actions sharing similar temporal patterns with different spatio-semantic contexts. To resolve these problems, we propose DETACH, a decomposed spatio-temporal framework. This explicit decomposition preserves local details, while our novel sensor-spatial features discovered via online clustering provide semantic grounding for context-aware alignment. To align the decomposed features, our two-stage approach establishes spatial correspondence through mutual supervision, then performs temporal alignment via a spatial-temporal weighted contrastive loss that adaptively handles easy negatives, hard negatives, and false negatives. Comprehensive experiments with downstream tasks on Opportunity++ and HWU-USP datasets demonstrate substantial improvements over adapted egocentric-wearable baselines.

Figures (10)

• Figure 1: Limitations of Global Alignment-based approaches. (a) Inability to capture local details: subtle motions become indistinguishable in video feature space. (b) Over-reliance on modality-invariant features: actions incorrectly aligned due to shared temporal patterns, despite different spatio-semantic contexts.
• Figure 2: Negative sampling strategy. Hard negatives (same object, different actions) are prioritized, easy negatives (different objects, different actions) are down-weighted, and false negatives (same object, same action) are filtered.
• Figure 3: Overall Architecture of DETACH. Stage 1 decomposes and learns spatial representations. Stage 2 leverages these spatial features to guide temporal alignment by decomposing temporal dynamics. This strategy enables fine-grained cross-modal understanding by separately capturing spatial context and temporal patterns.
• Figure 4: Temporal evolution of negative sample discrimination. This figure represents cumulative distribution functions (CDFs) of final weights $W_{ij}$ across training epochs. Initially overlapped distributions of hard and false negatives gradually separate, while easy negatives maintain unit weights throughout training.
• Figure 5: Class-wise analysis of hard and false negative separation. Comparison of weight distributions between top-3 classes with the largest separation (close door2, open drawer3, open dishwasher) and bottom-3 classes with the smallest separation (open drawer1, close drawer1, open drawer2).