HiERO: understanding the hierarchy of human behavior enhances reasoning on egocentric videos

TL;DR

HiERO tackles the challenge of understanding long-horizon, hierarchical human behavior in egocentric videos by learning functionally organized activity threads from unscripted data. It builds a video graph of short segments and processes it with a Temporal Encoder for local temporal reasoning and a Function-Aware Decoder that uses spectral clustering (Cut & Match) to form functionally related groups, trained with a video-narrations alignment loss $\mathcal{L}_{vna}$ and a functional threads loss $\mathcal{L}_{ft}$ so that high-level patterns emerge without explicit supervision. The approach yields state-of-the-art results on EgoMCQ and EgoNLQ, strong zero-shot performance on EgoProceL (+4.5–+4.9% over prior SOTA depending on backbone) and Ego4D Goal-Step, and robustly supports zero-shot procedure learning, step grounding, and step localization. By leveraging hierarchical structure in video data, HiERO enables multi-task reasoning with minimal supervision and demonstrates that functional threads naturally emerge from unscripted activity, offering a scalable path to richer egocentric understanding.

Abstract

Human activities are particularly complex and variable, and this makes challenging for deep learning models to reason about them. However, we note that such variability does have an underlying structure, composed of a hierarchy of patterns of related actions. We argue that such structure can emerge naturally from unscripted videos of human activities, and can be leveraged to better reason about their content. We present HiERO, a weakly-supervised method to enrich video segments features with the corresponding hierarchical activity threads. By aligning video clips with their narrated descriptions, HiERO infers contextual, semantic and temporal reasoning with an hierarchical architecture. We prove the potential of our enriched features with multiple video-text alignment benchmarks (EgoMCQ, EgoNLQ) with minimal additional training, and in zero-shot for procedure learning tasks (EgoProceL and Ego4D Goal-Step). Notably, HiERO achieves state-of-the-art performance in all the benchmarks, and for procedure learning tasks it outperforms fully-supervised methods by a large margin (+12.5% F1 on EgoProceL) in zero shot. Our results prove the relevance of using knowledge of the hierarchy of human activities for multiple reasoning tasks in egocentric vision.

Figures (7)

• Figure 1: Zero-Shot procedure step localization with HiERO. Given a long egocentric video, HiERO computes segment-level features that encode the functional dependencies between the actions in the video at different scales. This enables the detection of procedure steps through a simple clustering in feature space.
• Figure 2: Emergence of step clusters in the features similarity matrix of a video from Ego4D ego4d. Colored rectangles indicate the ground truth steps. Ideally, we expect high similarity (brighter regions) if two segments represent the same or semantically similar steps, e.g.cut onion and cut carrot. On Omnivore features, this behavior is only partially visible. On EgoVLP features, we observe sharper clusters of temporal segments that are not necessarily close temporally, but represent similar high-level actions. Our approach makes this behavior even more visible.
• Figure 3: Architecture of HiERO. HiERO is designed as an encoder-decoder architecture to implement Function-Aware video-text alignment. The Temporal Encoder$\mathcal{E}$ performs temporal reasoning on graph representations of the input video at different scales, while the Function-Aware Decoder$\mathcal{D}$ recombines nodes in the video graph by matching segments that represent functional dependencies between the actions (Cut & Match module). HiERO is trained to align video segments with their corresponding textual narrations at the shallower layer, and to strengthen thread-aware clustering in deeper layers.
• Figure 4: Zero-Shot Localization results on Ego4D Goal-Step, showing some of the HiERO's success and failure cases. We observe that many failure cases of HiERO are related to the ambiguous granularity of the step annotations in the dataset. In Fig. \ref{['fig:zs_segm_a']}, HiERO confuses the step Cook or prepare the vegetables with the closely related Cut the pepper. In Fig. \ref{['fig:zs_segm_c']}, HiERO correctly identifies many steps but confuses Mix ingredients to cook with some of its possible sub-steps, e.g., Cook with or prepare milk.
• Figure 5: Features distribution of narrations and procedural steps in Goal-Step song2024ego4d. Dots and stars represent the textual embeddings of the narrations and key-step labels, respectively, while the colors indicate the step to which the narrations are assigned.