Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Foundation Model-Driven Framework for Human-Object Interaction Prediction with Segmentation Mask Integration

Juhan Park, Kyungjae Lee, Hyung Jin Chang, Jungchan Cho

TL;DR

The paper tackles HOI prediction by leveraging segmentation foundation models to produce HOI quadruplets that include segmentation masks for human-object pairs. It introduces Seg2HOI, a frozen foundation-model pipeline with a dedicated HOI decoder implementing implicit relation learning and multi-head predictions, plus pseudo-labeling for HOI masks to enable training without HOI ground-truth masks. The method supports open-vocabulary and interactive quadruplet segmentation via prompts, and demonstrates competitive performance on HICO-DET and V-COCO, including zero-shot settings. This approach offers a resource-efficient path to joint segmentation and HOI understanding, with potential applications in robotics and interactive systems.

Abstract

In this work, we introduce Segmentation to Human-Object Interaction (\textit{\textbf{Seg2HOI}}) approach, a novel framework that integrates segmentation-based vision foundation models with the human-object interaction task, distinguished from traditional detection-based Human-Object Interaction (HOI) methods. Our approach enhances HOI detection by not only predicting the standard triplets but also introducing quadruplets, which extend HOI triplets by including segmentation masks for human-object pairs. More specifically, Seg2HOI inherits the properties of the vision foundation model (e.g., promptable and interactive mechanisms) and incorporates a decoder that applies these attributes to HOI task. Despite training only for HOI, without additional training mechanisms for these properties, the framework demonstrates that such features still operate efficiently. Extensive experiments on two public benchmark datasets demonstrate that Seg2HOI achieves performance comparable to state-of-the-art methods, even in zero-shot scenarios. Lastly, we propose that Seg2HOI can generate HOI quadruplets and interactive HOI segmentation from novel text and visual prompts that were not used during training, making it versatile for a wide range of applications by leveraging this flexibility.

Foundation Model-Driven Framework for Human-Object Interaction Prediction with Segmentation Mask Integration

TL;DR

The paper tackles HOI prediction by leveraging segmentation foundation models to produce HOI quadruplets that include segmentation masks for human-object pairs. It introduces Seg2HOI, a frozen foundation-model pipeline with a dedicated HOI decoder implementing implicit relation learning and multi-head predictions, plus pseudo-labeling for HOI masks to enable training without HOI ground-truth masks. The method supports open-vocabulary and interactive quadruplet segmentation via prompts, and demonstrates competitive performance on HICO-DET and V-COCO, including zero-shot settings. This approach offers a resource-efficient path to joint segmentation and HOI understanding, with potential applications in robotics and interactive systems.

Abstract

In this work, we introduce Segmentation to Human-Object Interaction (\textit{\textbf{Seg2HOI}}) approach, a novel framework that integrates segmentation-based vision foundation models with the human-object interaction task, distinguished from traditional detection-based Human-Object Interaction (HOI) methods. Our approach enhances HOI detection by not only predicting the standard triplets but also introducing quadruplets, which extend HOI triplets by including segmentation masks for human-object pairs. More specifically, Seg2HOI inherits the properties of the vision foundation model (e.g., promptable and interactive mechanisms) and incorporates a decoder that applies these attributes to HOI task. Despite training only for HOI, without additional training mechanisms for these properties, the framework demonstrates that such features still operate efficiently. Extensive experiments on two public benchmark datasets demonstrate that Seg2HOI achieves performance comparable to state-of-the-art methods, even in zero-shot scenarios. Lastly, we propose that Seg2HOI can generate HOI quadruplets and interactive HOI segmentation from novel text and visual prompts that were not used during training, making it versatile for a wide range of applications by leveraging this flexibility.
Paper Structure (22 sections, 24 equations, 10 figures, 7 tables)

This paper contains 22 sections, 24 equations, 10 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Object detection and Segmentation
  4. Human-object interaction recognition
  5. Various application in computer vision tasks
  6. Backgrounds: Vision Foundation Model
  7. Transformer-Based Framework
  8. Hungarian matching and loss for fundamental tasks
  9. Problem Definition
  10. Proposed Seg2HOI Framework
  11. Implicit Human-Object Relation Learning
  12. Multi-Head HOI Prediction
  13. HOI Mask Pseudo Labeling
  14. Training and Inference
  15. Interactive Quadruplet Image Segmentation Extension
  16. ...and 7 more sections

Figures (10)

  • Figure 1: In (a), the prediction set from traditional HOI detection methods, while (b) illustrates the proposed method from this paper, which includes HOI segmentation in addition to standard prediction. In (c), examples of potential vision and robotic applications are presented.
  • Figure 2: Overview of the proposed Seg2HOI framework which consists of two steps: pretrained vision foundation model and HOI model. The black arrow represents the process of the pretrained segmentation model, the blue arrow represents the process added in this paper, and the purple arrow represents the process for pseudo-labeling.
  • Figure 3: Implicit pair learning in HOI deocoder.
  • Figure 4: HOI decoder output heads.
  • Figure 5: HOI mask pseudo labeling process
  • ...and 5 more figures