Hier-EgoPack: Hierarchical Egocentric Video Understanding with Diverse Task Perspectives

TL;DR

Hier-EgoPack extends EgoPack by introducing hierarchical temporal reasoning for egocentric video understanding, enabling multiple tasks with differing temporal granularities to share knowledge via a backpack of task prototypes. The core innovations include a unified graph-based temporal backbone using Temporal Distance Gated Convolution (TDGC) and a prototype-driven cross-task interaction mechanism that refines novel tasks without extensive retraining. Empirical results on Ego4D show notable improvements over single-task, multi-task, and task-translation baselines across AR, OSCC, PNR, LTA, and MQ, with strong qualitative evidence of effective knowledge reuse. This framework advances holistic video understanding by integrating short- and long-horizon reasoning into a single, scalable architecture suitable for real-world egocentric applications.

Abstract

Our comprehension of video streams depicting human activities is naturally multifaceted: in just a few moments, we can grasp what is happening, identify the relevance and interactions of objects in the scene, and forecast what will happen soon, everything all at once. To endow autonomous systems with such a holistic perception, learning how to correlate concepts, abstract knowledge across diverse tasks, and leverage tasks synergies when learning novel skills is essential. A significant step in this direction is EgoPack, a unified framework for understanding human activities across diverse tasks with minimal overhead. EgoPack promotes information sharing and collaboration among downstream tasks, essential for efficiently learning new skills. In this paper, we introduce Hier-EgoPack, which advances EgoPack by enabling reasoning also across diverse temporal granularities, which expands its applicability to a broader range of downstream tasks. To achieve this, we propose a novel hierarchical architecture for temporal reasoning equipped with a GNN layer specifically designed to tackle the challenges of multi-granularity reasoning effectively. We evaluate our approach on multiple Ego4d benchmarks involving both clip-level and frame-level reasoning, demonstrating how our hierarchical unified architecture effectively solves these diverse tasks simultaneously.

Figures (7)

• Figure 1: Overview of the Hier-EgoPack architecture. First, the video is converted into a graph representation $\mathcal{G}^{(0)}$ whose node embeddings are extracted using a frozen video features extractor. The graph is then processed by the hierarchical temporal backbone$\mathcal{M}_t$, shared by all the tasks, to progressively learn higher level representations of the input video $\{\mathcal{G}^{(1)}, \mathcal{G}^{(2)}, \dots, \mathcal{G}^{(L)}\}$. The node embeddings of these graphs are projected by the task-specific necks$\mathcal{N}_i$ in the features space of each task $\mathcal{T}_i$ and to the corresponding output space with the task-specific heads$\mathcal{H}_i$.
• Figure 2: Temporal Distance Gated Convolution layer (TDGC), specifically designed to integrate past and future events grounding ($s_{ij}$) and to reason about the temporal distance between nodes ($\mathbf{w}_{ij}$) in the aggregation step.
• Figure 3: Learning a novel task with a backpack. After the Multi-Task training phase, we extract a set of prototypes $\mathbf{P}^k$ that summarize what the network has learned from each support task$\mathcal{T}_k$, like a backpack of skills that we can carry over. In this Cross-Tasks Interaction phase, the network can peek at these different task-perspective to enrich the learning of the novel task.
• Figure 4: Activation frequency for the task-specific prototypes from different support tasks. We focus on the Top-20 most activated prototypes across the support tasks. LTA and OSCC have more uniform activations across different support tasks, i.e., they look at similar prototypes, while MQ exhibit more diverse activations.
• Figure 5: Activations consensus for different novel tasks. Activations consensus between two support tasks is defined as the percentage of their prototypes corresponding to the same label activated by the two tasks. Fine-grained tasks, i.e., AR, OSCC and LTA, have higher average consensus. On the contrary, MQ has lower average consensus.