Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Short-term Object Interaction Anticipation with Disentangled Object Detection @ Ego4D Short Term Object Interaction Anticipation Challenge

Hyunjin Cho, Dong Un Kang, Se Young Chun

TL;DR

The paper tackles short-term object interaction anticipation in egocentric video by disentangling active-object detection from interaction and time-to-contact prediction. It introduces SOIA-DOD, a cascaded pipeline that first detects potential active objects from the last frame using a fine-tuned YOLOv9, then uses a transformer to fuse these detections with CLIP visual features to predict the next object, its interaction, and the time-to-contact. On Ego4D, SOIA-DOD achieves state-of-the-art performance for noun and verb prediction and places third in top-5 mAP when time-to-contact is included, demonstrating the benefit of decoupling detection from predictive tasks. This modular approach improves localization of active objects and reduces learning complexity for near-future anticipation in egocentric settings, with potential applications in robotics and AR systems.

Abstract

Short-term object interaction anticipation is an important task in egocentric video analysis, including precise predictions of future interactions and their timings as well as the categories and positions of the involved active objects. To alleviate the complexity of this task, our proposed method, SOIA-DOD, effectively decompose it into 1) detecting active object and 2) classifying interaction and predicting their timing. Our method first detects all potential active objects in the last frame of egocentric video by fine-tuning a pre-trained YOLOv9. Then, we combine these potential active objects as query with transformer encoder, thereby identifying the most promising next active object and predicting its future interaction and time-to-contact. Experimental results demonstrate that our method outperforms state-of-the-art models on the challenge test set, achieving the best performance in predicting next active objects and their interactions. Finally, our proposed ranked the third overall top-5 mAP when including time-to-contact predictions. The source code is available at https://github.com/KeenyJin/SOIA-DOD.

Short-term Object Interaction Anticipation with Disentangled Object Detection @ Ego4D Short Term Object Interaction Anticipation Challenge

TL;DR

The paper tackles short-term object interaction anticipation in egocentric video by disentangling active-object detection from interaction and time-to-contact prediction. It introduces SOIA-DOD, a cascaded pipeline that first detects potential active objects from the last frame using a fine-tuned YOLOv9, then uses a transformer to fuse these detections with CLIP visual features to predict the next object, its interaction, and the time-to-contact. On Ego4D, SOIA-DOD achieves state-of-the-art performance for noun and verb prediction and places third in top-5 mAP when time-to-contact is included, demonstrating the benefit of decoupling detection from predictive tasks. This modular approach improves localization of active objects and reduces learning complexity for near-future anticipation in egocentric settings, with potential applications in robotics and AR systems.

Abstract

Short-term object interaction anticipation is an important task in egocentric video analysis, including precise predictions of future interactions and their timings as well as the categories and positions of the involved active objects. To alleviate the complexity of this task, our proposed method, SOIA-DOD, effectively decompose it into 1) detecting active object and 2) classifying interaction and predicting their timing. Our method first detects all potential active objects in the last frame of egocentric video by fine-tuning a pre-trained YOLOv9. Then, we combine these potential active objects as query with transformer encoder, thereby identifying the most promising next active object and predicting its future interaction and time-to-contact. Experimental results demonstrate that our method outperforms state-of-the-art models on the challenge test set, achieving the best performance in predicting next active objects and their interactions. Finally, our proposed ranked the third overall top-5 mAP when including time-to-contact predictions. The source code is available at https://github.com/KeenyJin/SOIA-DOD.
Paper Structure (10 sections, 1 equation, 2 figures, 2 tables)

This paper contains 10 sections, 1 equation, 2 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Method
  3. Potential Active Objects Detection
  4. Anticipating Short-term Object Interaction with Transformer
  5. Training Objective
  6. Experiments
  7. Implementation Details
  8. Experimental Results
  9. Limitations
  10. Conclusion

Figures (2)

  • Figure 1: The overall pipeline of our proposed SOIA-DOD. The SOIA-DOD consists of two cascaded stages. (a) Potential active object detection: A disentangled object detection model (fine-tuned YOLOv9) detects potential active objects from the last egocentric frame. Based on the detection confidence scores, it selects the top-k active objects and generates queries representing these objects, including their class labels and bounding boxes. (b) Anticipating short-term object interaction with transformer: Visual tokens and active object queries are concatenated and processed by the self-attention mechanism within a transformer layer. This allows the model to learn relationships between different parts of the image and the active objects. Finally, the model outputs final predictions, including active object probability, interaction class score and time-to-contact.
  • Figure 2: Qualitative success and failure cases of our SOIA-DOD. Our SOIA-DOD excels at detecting active objects and their interactions, but it often struggles with accurately estimating time-to-contact.