Test-time Ego-Exo-centric Adaptation for Action Anticipation via Multi-Label Prototype Growing and Dual-Clue Consistency

TL;DR

A novel Dual-Clue enhanced Prototype Growing Network (DCPGN), which accumulates multi-label knowledge and integrates cross-modality clues for effective test-time Ego-Exo adaptation and action anticipation and outperforms related state-of-the-art methods by a large margin.

Abstract

Efficient adaptation between Egocentric (Ego) and Exocentric (Exo) views is crucial for applications such as human-robot cooperation. However, the success of most existing Ego-Exo adaptation methods relies heavily on target-view data for training, thereby increasing computational and data collection costs. In this paper, we make the first exploration of a Test-time Ego-Exo Adaptation for Action Anticipation (TE$^{2}$A$^{3}$) task, which aims to adjust the source-view-trained model online during test time to anticipate target-view actions. It is challenging for existing Test-Time Adaptation (TTA) methods to address this task due to the multi-action candidates and significant temporal-spatial inter-view gap. Hence, we propose a novel Dual-Clue enhanced Prototype Growing Network (DCPGN), which accumulates multi-label knowledge and integrates cross-modality clues for effective test-time Ego-Exo adaptation and action anticipation. Specifically, we propose a Multi-Label Prototype Growing Module (ML-PGM) to balance multiple positive classes via multi-label assignment and confidence-based reweighting for class-wise memory banks, which are updated by an entropy priority queue strategy. Then, the Dual-Clue Consistency Module (DCCM) introduces a lightweight narrator to generate textual clues indicating action progressions, which complement the visual clues containing various objects. Moreover, we constrain the inferred textual and visual logits to construct dual-clue consistency for temporally and spatially bridging Ego and Exo views. Extensive experiments on the newly proposed EgoMe-anti and the existing EgoExoLearn benchmarks show the effectiveness of our method, which outperforms related state-of-the-art methods by a large margin. Code is available at \href{https://github.com/ZhaofengSHI/DCPGN}{https://github.com/ZhaofengSHI/DCPGN}.

Figures (7)

• Figure 1: Schematic of the newly proposed TE$^{2}$A$^{3}$ task.
• Figure 2: Overview of our DCPGN. The model is trained using the source-view data. During testing, the ML-PGM assigns each representation to multiple labels, then performs confidence-based reweighting under an entropy priority queue strategy to balance multiple positive classes. Moreover, to complement the visual clue containing various objects, the DCCM incorporates a lightweight narrator to generate descriptions for action progressions with nouns (green) and verbs (purple). We integrate cross-modality clues by constraining their logits for temporally and spatially bridging the view gap. Finally, the prototype, visual, and textual logits are added for target-view anticipation.
• Figure 3: Visualization of representations in memory banks and the corresponding prototypes of several classes. Dots denote saved representations and stars denote prototypes (best viewed in color).
• Figure 4: Visualization of noun and verb anticipation candidates with or without DCCM under the Exo2Ego setting.
• Figure A1: t-SNE visualization of representations in the memory banks and the corresponding prototypes of five dominant classes under Exo2Ego and Ego2Exo settings. The left panel shows the results on the EgoExoLearn benchmark and the right panel shows the results on the EgoMe-anti benchmark. Dots denote class-wise representations and stars denote prototypes (best viewed in color).