Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ALGO: Object-Grounded Visual Commonsense Reasoning for Open-World Egocentric Action Recognition

Sanjoy Kundu, Shubham Trehan, Sathyanarayanan N. Aakur

TL;DR

This work tackles open-world egocentric action recognition by grounding objects with evidence-based prompts and discovering actions through energy-based reasoning over a commonsense knowledge graph. The ALGO framework combines neural grounding (via CLIP) of object concepts with pattern-theory–driven symbolic inference to identify plausible action-object pairs and to ground unseen verbs in a visual context, using a config-energy formulation that combines probabilistic terms: $E(c)= \phi(p(\underline{g}^o_i | \bar{g}^o_j, I_t, K_{CS})) + \phi(p(g^a_k, \underline{g}^o_i | K_{CS})) + \phi(p(g^a_k | I_t))$. Contributions include (i) a neuro-symbolic grounding approach leveraging object compositionality and CLIP prompts, (ii) object-driven activity discovery that constrains search with action-object affinities, and (iii) victorious grounding of unseen verbs through visual-semantic mappings to ConceptNet Numberbatch, enabling open-world generalization. Evaluations on GTEA Gaze, GTEA Gaze Plus, and EPIC-Kitchens-100 show notable gains over prior open-world baselines, highlighting practical potential for autonomous, data-efficient egocentric understanding.

Abstract

Learning to infer labels in an open world, i.e., in an environment where the target "labels" are unknown, is an important characteristic for achieving autonomy. Foundation models pre-trained on enormous amounts of data have shown remarkable generalization skills through prompting, particularly in zero-shot inference. However, their performance is restricted to the correctness of the target label's search space. In an open world, this target search space can be unknown or exceptionally large, which severely restricts the performance of such models. To tackle this challenging problem, we propose a neuro-symbolic framework called ALGO - Action Learning with Grounded Object recognition that uses symbolic knowledge stored in large-scale knowledge bases to infer activities in egocentric videos with limited supervision using two steps. First, we propose a neuro-symbolic prompting approach that uses object-centric vision-language models as a noisy oracle to ground objects in the video through evidence-based reasoning. Second, driven by prior commonsense knowledge, we discover plausible activities through an energy-based symbolic pattern theory framework and learn to ground knowledge-based action (verb) concepts in the video. Extensive experiments on four publicly available datasets (EPIC-Kitchens, GTEA Gaze, GTEA Gaze Plus) demonstrate its performance on open-world activity inference.

ALGO: Object-Grounded Visual Commonsense Reasoning for Open-World Egocentric Action Recognition

TL;DR

This work tackles open-world egocentric action recognition by grounding objects with evidence-based prompts and discovering actions through energy-based reasoning over a commonsense knowledge graph. The ALGO framework combines neural grounding (via CLIP) of object concepts with pattern-theory–driven symbolic inference to identify plausible action-object pairs and to ground unseen verbs in a visual context, using a config-energy formulation that combines probabilistic terms: . Contributions include (i) a neuro-symbolic grounding approach leveraging object compositionality and CLIP prompts, (ii) object-driven activity discovery that constrains search with action-object affinities, and (iii) victorious grounding of unseen verbs through visual-semantic mappings to ConceptNet Numberbatch, enabling open-world generalization. Evaluations on GTEA Gaze, GTEA Gaze Plus, and EPIC-Kitchens-100 show notable gains over prior open-world baselines, highlighting practical potential for autonomous, data-efficient egocentric understanding.

Abstract

Learning to infer labels in an open world, i.e., in an environment where the target "labels" are unknown, is an important characteristic for achieving autonomy. Foundation models pre-trained on enormous amounts of data have shown remarkable generalization skills through prompting, particularly in zero-shot inference. However, their performance is restricted to the correctness of the target label's search space. In an open world, this target search space can be unknown or exceptionally large, which severely restricts the performance of such models. To tackle this challenging problem, we propose a neuro-symbolic framework called ALGO - Action Learning with Grounded Object recognition that uses symbolic knowledge stored in large-scale knowledge bases to infer activities in egocentric videos with limited supervision using two steps. First, we propose a neuro-symbolic prompting approach that uses object-centric vision-language models as a noisy oracle to ground objects in the video through evidence-based reasoning. Second, driven by prior commonsense knowledge, we discover plausible activities through an energy-based symbolic pattern theory framework and learn to ground knowledge-based action (verb) concepts in the video. Extensive experiments on four publicly available datasets (EPIC-Kitchens, GTEA Gaze, GTEA Gaze Plus) demonstrate its performance on open-world activity inference.
Paper Structure (8 sections, 1 equation, 1 figure, 2 tables)

This paper contains 8 sections, 1 equation, 1 figure, 2 tables.

Table of Contents

  1. Introduction
  2. Proposed Framework: ALGO
  3. Evidence-based Object Grounding
  4. Object-driven Activity Discovery
  5. Visual-Semantic Action Grounding
  6. Experimental Evaluation
  7. Open World Activity Recognition
  8. Discussion, Limitations, and Future Work

Figures (1)

  • Figure 1: Overall architecture of the proposed approach (ALGO) is illustrated here. Using a two-step process, we first ground the objects within a gaze-driven ROI using CLIP radford2021learning as a noisy oracle before reasoning over the plausible activities performed in the video.