DMC$^3$: Dual-Modal Counterfactual Contrastive Construction for Egocentric Video Question Answering

TL;DR

The paper tackles Egocentric VideoQA by addressing multi-event reasoning and hand-object interaction challenges. It introduces DMC3, a framework that combines a baseline dual-stream model with counterfactual sample construction (EDP for text, CIM for visuals) and counterfactual contrastive optimization to align original samples with positive variants while distancing from negatives. Through a two-stage training process and comprehensive ablations, DMC3 achieves state-of-the-art results on EgoTaskQA and QAEGO4D, validating the effectiveness of dual-modal counterfactual signals. The work highlights the value of targeted data augmentation paired with contrastive objectives for robust egocentric reasoning.

Abstract

Egocentric Video Question Answering (Egocentric VideoQA) plays an important role in egocentric video understanding, which refers to answering questions based on first-person videos. Although existing methods have made progress through the paradigm of pre-training and fine-tuning, they ignore the unique challenges posed by the first-person perspective, such as understanding multiple events and recognizing hand-object interactions. To deal with these challenges, we propose a Dual-Modal Counterfactual Contrastive Construction (DMC$^3$) framework, which contains an egocentric videoqa baseline, a counterfactual sample construction module and a counterfactual sample-involved contrastive optimization. Specifically, We first develop a counterfactual sample construction module to generate positive and negative samples for textual and visual modalities through event description paraphrasing and core interaction mining, respectively. Then, We feed these samples together with the original samples into the baseline. Finally, in the counterfactual sample-involved contrastive optimization module, we apply contrastive loss to minimize the distance between the original sample features and the positive sample features, while maximizing the distance from the negative samples. Experiments show that our method achieve 52.51\% and 46.04\% on the \textit{normal} and \textit{indirect} splits of EgoTaskQA, and 13.2\% on QAEGO4D, both reaching the state-of-the-art performance.

Figures (4)

• Figure 1: Two challenges of Egocentric VideoQA. (a) Egocentric video has a long time span and contains multiple fine-grained events, requiring the model to understand these events and the contextual information. (b) Egocentric video emphasizes hand-object interactions, requiring the model to identify and interpret relevant visual regions.
• Figure 2: The Dual-Modal Counterfactual Contrastive Construction framework comprises a egocentric videoqa baseline, a counterfactual sample construction module and a counterfactual sample-involved contrastive optimization module. CIM and EDP are utilized to generate factual and counterfactual samples for visual and textual modalities, respectively. Positive factual samples are integrated into the model through paths denoted by blue arrows, while the paths of negative counterfactual samples are represented by red arrows.
• Figure 3: Examples of positive factual and negative counterfactual samples derived from an original sample. The white part of the video frame is the mask area. The event in the text is replaced with a special token "[MASK]".
• Figure 4: Ablation studies of hyper-parameters on the indirect split of EgoTashQA. (a), (b), (c) and (d) are the performance changes of temperature parameter $\tau$ and weight parameter $\alpha$, $\beta$, $\lambda$, respectively.