Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Learning Human-Object Interaction as Groups

Jiajun Hong, Jianan Wei, Wenguan Wang

TL;DR

GroupHOI reframes HOI-DET from pairwise reasoning to group-based reasoning by introducing two grouping principles: geometric proximity and semantic similarity. It builds geometric groups via a learnable proximity estimator and local self-attention within groups, and enhances the interaction decoder with semantic-group priors for local-global context. The approach yields state-of-the-art results on HOI-DET benchmarks and leading performance on NVI-DET, while maintaining efficiency comparable to existing DETR-based methods. This group-centric framework improves robustness to complex, multi-entity interactions and offers interpretable grouping patterns with minimal architectural overhead.

Abstract

Human-Object Interaction Detection (HOI-DET) aims to localize human-object pairs and identify their interactive relationships. To aggregate contextual cues, existing methods typically propagate information across all detected entities via self-attention mechanisms, or establish message passing between humans and objects with bipartite graphs. However, they primarily focus on pairwise relationships, overlooking that interactions in real-world scenarios often emerge from collective behaviors (multiple humans and objects engaging in joint activities). In light of this, we revisit relation modeling from a group view and propose GroupHOI, a framework that propagates contextual information in terms of geometric proximity and semantic similarity. To exploit the geometric proximity, humans and objects are grouped into distinct clusters using a learnable proximity estimator based on spatial features derived from bounding boxes. In each group, a soft correspondence is computed via self-attention to aggregate and dispatch contextual cues. To incorporate the semantic similarity, we enhance the vanilla transformer-based interaction decoder with local contextual cues from HO-pair features. Extensive experiments on HICO-DET and V-COCO benchmarks demonstrate the superiority of GroupHOI over the state-of-the-art methods. It also exhibits leading performance on the more challenging Nonverbal Interaction Detection (NVI-DET) task, which involves varied forms of higher-order interactions within groups.

Learning Human-Object Interaction as Groups

TL;DR

GroupHOI reframes HOI-DET from pairwise reasoning to group-based reasoning by introducing two grouping principles: geometric proximity and semantic similarity. It builds geometric groups via a learnable proximity estimator and local self-attention within groups, and enhances the interaction decoder with semantic-group priors for local-global context. The approach yields state-of-the-art results on HOI-DET benchmarks and leading performance on NVI-DET, while maintaining efficiency comparable to existing DETR-based methods. This group-centric framework improves robustness to complex, multi-entity interactions and offers interpretable grouping patterns with minimal architectural overhead.

Abstract

Human-Object Interaction Detection (HOI-DET) aims to localize human-object pairs and identify their interactive relationships. To aggregate contextual cues, existing methods typically propagate information across all detected entities via self-attention mechanisms, or establish message passing between humans and objects with bipartite graphs. However, they primarily focus on pairwise relationships, overlooking that interactions in real-world scenarios often emerge from collective behaviors (multiple humans and objects engaging in joint activities). In light of this, we revisit relation modeling from a group view and propose GroupHOI, a framework that propagates contextual information in terms of geometric proximity and semantic similarity. To exploit the geometric proximity, humans and objects are grouped into distinct clusters using a learnable proximity estimator based on spatial features derived from bounding boxes. In each group, a soft correspondence is computed via self-attention to aggregate and dispatch contextual cues. To incorporate the semantic similarity, we enhance the vanilla transformer-based interaction decoder with local contextual cues from HO-pair features. Extensive experiments on HICO-DET and V-COCO benchmarks demonstrate the superiority of GroupHOI over the state-of-the-art methods. It also exhibits leading performance on the more challenging Nonverbal Interaction Detection (NVI-DET) task, which involves varied forms of higher-order interactions within groups.
Paper Structure (21 sections, 9 equations, 5 figures, 13 tables, 2 algorithms)

This paper contains 21 sections, 9 equations, 5 figures, 13 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Preliminary
  5. Rethinking the Relation Modeling in HOI-DET
  6. Learning Entities as Geometric Groups
  7. Learning Interactions as Semantic Groups
  8. Implementation Details
  9. Experiment
  10. Experiments on HOI-DET
  11. Experiments on NVI-DET
  12. Diagnostic Experiments
  13. Qualitative Results
  14. Visualization of Groups
  15. Discussion
  16. ...and 6 more sections

Figures (5)

  • Figure 1: Relation modeling paradigms. While existing methods primarily model relations using complete (a) or bipartite (b) graphs, our approach introduces geometric (c) and semantic (d) graphs, enabling more structured and context-aware relational reasoning. , , and denote humans, objects, and interactions, respectively.
  • Figure 2: Overview of GroupHOI. GroupHOI comprises three key modules: i) a visual encoder extracts features, followed by an instance decoder to locate human-object pairs; ii) a geometric-aware local transformer aggregates and distributes contextual information within geometric groups; iii) a semantic-aware interaction decoder leverages local-global semantic dependencies for interaction prediction.
  • Figure 3: Visualization of GroupHOI results on V-COCO gupta2015visualtest. The first two rows represent samples where all interactions are successfully detected, while the third row corresponds to samples with missed interactions. Detected interactions are marked in Green, while missed interactions and objects are in Red.
  • Figure 4: Visualization of groups on V-COCO gupta2015visualtest. L1, L2 and L3 represent the first, second and last layer of instance or interaction decoder. The center of the geometric and semantic groups is mark in Red (§\ref{['sec:group']}).
  • Figure S1: Visualization of GroupHOI results on V-COCO gupta2015visualtest. Detected interactions are marked in Green, while missed interactions and objects are in Red.