Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Ego4OOD: Rethinking Egocentric Video Domain Generalization via Covariate Shift Scoring

Zahra Vaseqi, James Clark

TL;DR

This work addresses domain generalization in egocentric video by distinguishing covariate shift from concept shift and introducing Ego4OOD, a covariate-diverse benchmark built from Ego4D moment-level annotations across eight domains. It provides a clustering-based covariate shift score to quantify domain difficulty and proposes a lightweight two-layer architecture (MLP-Lite) trained with a one-vs-all objective on pre-extracted SlowFast features, achieving competitive results with fewer parameters and no textual modality. Empirical results on Argo1M and Ego4OOD show that higher covariate shift scores correlate with lower recognition accuracy, underscoring the metric’s predictive value and the importance of controlled benchmarks. The work suggests that careful benchmark design and simple, covariate-focused training strategies can yield robust generalization in egocentric video, with Ego4OOD serving as a practical evaluation framework for future DG research.

Abstract

Egocentric video action recognition under domain shifts remains challenging due to large intra-class spatio-temporal variability, long-tailed feature distributions, and strong correlations between actions and environments. Existing benchmarks for egocentric domain generalization often conflate covariate shifts with concept shifts, making it difficult to reliably evaluate a model's ability to generalize across input distributions. To address this limitation, we introduce Ego4OOD, a domain generalization benchmark derived from Ego4D that emphasizes measurable covariate diversity while reducing concept shift through semantically coherent, moment-level action categories. Ego4OOD spans eight geographically distinct domains and is accompanied by a clustering-based covariate shift metric that provides a quantitative proxy for domain difficulty. We further leverage a one-vs-all binary training objective that decomposes multi-class action recognition into independent binary classification tasks. This formulation is particularly well-suited for covariate shift by reducing interference between visually similar classes under feature distribution shift. Using this formulation, we show that a lightweight two-layer fully connected network achieves performance competitive with state-of-the-art egocentric domain generalization methods on both Argo1M and Ego4OOD, despite using fewer parameters and no additional modalities. Our empirical analysis demonstrates a clear relationship between measured covariate shift and recognition performance, highlighting the importance of controlled benchmarks and quantitative domain characterization for studying out-of-distribution generalization in egocentric video.

Ego4OOD: Rethinking Egocentric Video Domain Generalization via Covariate Shift Scoring

TL;DR

This work addresses domain generalization in egocentric video by distinguishing covariate shift from concept shift and introducing Ego4OOD, a covariate-diverse benchmark built from Ego4D moment-level annotations across eight domains. It provides a clustering-based covariate shift score to quantify domain difficulty and proposes a lightweight two-layer architecture (MLP-Lite) trained with a one-vs-all objective on pre-extracted SlowFast features, achieving competitive results with fewer parameters and no textual modality. Empirical results on Argo1M and Ego4OOD show that higher covariate shift scores correlate with lower recognition accuracy, underscoring the metric’s predictive value and the importance of controlled benchmarks. The work suggests that careful benchmark design and simple, covariate-focused training strategies can yield robust generalization in egocentric video, with Ego4OOD serving as a practical evaluation framework for future DG research.

Abstract

Egocentric video action recognition under domain shifts remains challenging due to large intra-class spatio-temporal variability, long-tailed feature distributions, and strong correlations between actions and environments. Existing benchmarks for egocentric domain generalization often conflate covariate shifts with concept shifts, making it difficult to reliably evaluate a model's ability to generalize across input distributions. To address this limitation, we introduce Ego4OOD, a domain generalization benchmark derived from Ego4D that emphasizes measurable covariate diversity while reducing concept shift through semantically coherent, moment-level action categories. Ego4OOD spans eight geographically distinct domains and is accompanied by a clustering-based covariate shift metric that provides a quantitative proxy for domain difficulty. We further leverage a one-vs-all binary training objective that decomposes multi-class action recognition into independent binary classification tasks. This formulation is particularly well-suited for covariate shift by reducing interference between visually similar classes under feature distribution shift. Using this formulation, we show that a lightweight two-layer fully connected network achieves performance competitive with state-of-the-art egocentric domain generalization methods on both Argo1M and Ego4OOD, despite using fewer parameters and no additional modalities. Our empirical analysis demonstrates a clear relationship between measured covariate shift and recognition performance, highlighting the importance of controlled benchmarks and quantitative domain characterization for studying out-of-distribution generalization in egocentric video.
Paper Structure (16 sections, 7 equations, 3 figures, 7 tables)

This paper contains 16 sections, 7 equations, 3 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Egocentric Action Recognition
  4. Distribution Shift in Egocentric Video Datasets
  5. Ego4OOD Benchmark
  6. Ego4OOD Splits.
  7. Analyzing Covariate Shift.
  8. Approach
  9. Problem Formulation.
  10. Model Architecture.
  11. One-vs-all Objective.
  12. Experiments
  13. Implementation Details.
  14. Argo1M Results.
  15. Ego4OOD Results.
  16. ...and 1 more sections

Figures (3)

  • Figure 1: Example frames from five video clips, each row corresponding to a single clip as annotated in the Argo1M benchmark: (a) bring, (b) pat, (c) tear, (d) open, (e) bring. These examples illustrate annotation ambiguities and co-occurring actions that can lead to misclassification, highlighting key failure modes of the Argo1M benchmark.
  • Figure 2: Number of video clips in Ego4OOD aggregated by (left) category and (right) domain.
  • Figure 3: Illustration of three representative video clips per domain from the food prep category across five Ego4OOD domains: India, FRL, UK, Saudi, and Japan. Within each domain, clips share characteristic visual features. India kitchens differ from other domains in appearance of the walls, utensils, and overall layout. FRL consists of indoor office kitchens with distinctive background objects, and bright, consistent illumination. The UK, Saudi, and Japan domains, on the other hand, share similar kitchen utensils, layouts, and lighting. Across domains, differences in walls, objects, layouts, and lighting highlight how visual features shift while the food prep category remains the same. Horizontal lines separate domains for clarity, and each row shows frames sampled from a single video clip.