Robust Egocentric Referring Video Object Segmentation via Dual-Modal Causal Intervention

TL;DR

This work tackles Ego-RVOS by reframing segmentation as a causal problem with two confounding sources: language bias (observable) and egocentric visual confounding (unobservable). It introduces CERES, a plug-in framework that uses back-door adjustment for text grounding and a vision-depth front-door mediator for robust visual processing, realized through depth-guided attention and memory-based context. By decomposing visual mediators into semantic and geometric components and integrating them with a memory bank, CERES achieves state-of-the-art performance on VISOR, VOST, and VSCOS, while showing strong generalization to novel concepts. The approach demonstrates that explicitly modeling causal pathways across modalities improves reliability and generalization for complex egocentric video understanding tasks, with practical implications for assistive tech and embodied AI.

Abstract

Egocentric Referring Video Object Segmentation (Ego-RVOS) aims to segment the specific object actively involved in a human action, as described by a language query, within first-person videos. This task is critical for understanding egocentric human behavior. However, achieving such segmentation robustly is challenging due to ambiguities inherent in egocentric videos and biases present in training data. Consequently, existing methods often struggle, learning spurious correlations from skewed object-action pairings in datasets and fundamental visual confounding factors of the egocentric perspective, such as rapid motion and frequent occlusions. To address these limitations, we introduce Causal Ego-REferring Segmentation (CERES), a plug-in causal framework that adapts strong, pre-trained RVOS backbones to the egocentric domain. CERES implements dual-modal causal intervention: applying backdoor adjustment principles to counteract language representation biases learned from dataset statistics, and leveraging front-door adjustment concepts to address visual confounding by intelligently integrating semantic visual features with geometric depth information guided by causal principles, creating representations more robust to egocentric distortions. Extensive experiments demonstrate that CERES achieves state-of-the-art performance on Ego-RVOS benchmarks, highlighting the potential of applying causal reasoning to build more reliable models for broader egocentric video understanding.

Figures (11)

• Figure 1: Motivation and overview of the CERES for addressing biases. (a) Ego-RVOS needs to segment the objects related to action (positive objects, colored) instead of objects unrelated to action (negative objects, gray). (b) Example failure cases of baseline ouyang2024actionvos-eccv because of visual occlusion and rare objects outside the training set. (c) Our CERES from text and visual modal performs causal intervention to achieve robust Ego-RVOS.
• Figure 2: The proposed SCM for Ego-RVOS. (Dashed lines indicate confounding paths; solid lines indicate causal paths.)
• Figure 3: Overview of the CERES framework. The Linguistic Back-door Deconfounder (LBD) de-biases the input text query $\mathbf{T}$ into features $\mathbf{f}'_{\mathcal{T}}(t)$. Concurrently, the Visual Front-door Deconfounder (VFD) processes the video frame $\mathbf{X}_t$; it forms a vision-depth mediator $\hat{\mathbf{M}}(x_t)$ using Depth-guided Attention (DAttn) and estimates temporal visual context $\hat{\mathbf{X}}_t$ via Memory Attention (MAttn), yielding de-biased visual features $\mathbf{f}'_{\mathcal{X}}(x_t)$. These de-biased multimodal features are then used by the RVOS model to predict the segmentation mask $\hat{\mathbf{Y}}_t$.
• Figure 4: Ablation study (%) of proposed modules on VISOR (ResNet101). ($\Diamond$ indicates MLP-based depth fusion)
• Figure 4: Qualitative analysis of ActionVOS and our CERES.