Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Boosting Human-Object Interaction Detection with Text-to-Image Diffusion Model

Jie Yang, Bingliang Li, Fengyu Yang, Ailing Zeng, Lei Zhang, Ruimao Zhang

TL;DR

The paper tackles HOI detection challenges driven by long-tail distributions and ambiguous verb semantics. It introduces DiffHOI, which leverages a frozen text-to-image diffusion model to obtain verb-centered contextual representations through adapter-style tuning and CLIP alignment, and pairs this with SynHOI, a large synthetic dataset of 140K annotated HOI images. The approach yields state-of-the-art results on HICO-DET (41.50 mAP) and significant zero-shot and rare-class improvements, demonstrating the value of diffusion-based representations and scalable synthetic data for human-object interaction understanding. The work suggests a scalable pathway to boost HOI detectors across backbones and settings by blending generative priors with discriminative training.

Abstract

This paper investigates the problem of the current HOI detection methods and introduces DiffHOI, a novel HOI detection scheme grounded on a pre-trained text-image diffusion model, which enhances the detector's performance via improved data diversity and HOI representation. We demonstrate that the internal representation space of a frozen text-to-image diffusion model is highly relevant to verb concepts and their corresponding context. Accordingly, we propose an adapter-style tuning method to extract the various semantic associated representation from a frozen diffusion model and CLIP model to enhance the human and object representations from the pre-trained detector, further reducing the ambiguity in interaction prediction. Moreover, to fill in the gaps of HOI datasets, we propose SynHOI, a class-balance, large-scale, and high-diversity synthetic dataset containing over 140K HOI images with fully triplet annotations. It is built using an automatic and scalable pipeline designed to scale up the generation of diverse and high-precision HOI-annotated data. SynHOI could effectively relieve the long-tail issue in existing datasets and facilitate learning interaction representations. Extensive experiments demonstrate that DiffHOI significantly outperforms the state-of-the-art in regular detection (i.e., 41.50 mAP) and zero-shot detection. Furthermore, SynHOI can improve the performance of model-agnostic and backbone-agnostic HOI detection, particularly exhibiting an outstanding 11.55% mAP improvement in rare classes.

Boosting Human-Object Interaction Detection with Text-to-Image Diffusion Model

TL;DR

The paper tackles HOI detection challenges driven by long-tail distributions and ambiguous verb semantics. It introduces DiffHOI, which leverages a frozen text-to-image diffusion model to obtain verb-centered contextual representations through adapter-style tuning and CLIP alignment, and pairs this with SynHOI, a large synthetic dataset of 140K annotated HOI images. The approach yields state-of-the-art results on HICO-DET (41.50 mAP) and significant zero-shot and rare-class improvements, demonstrating the value of diffusion-based representations and scalable synthetic data for human-object interaction understanding. The work suggests a scalable pathway to boost HOI detectors across backbones and settings by blending generative priors with discriminative training.

Abstract

This paper investigates the problem of the current HOI detection methods and introduces DiffHOI, a novel HOI detection scheme grounded on a pre-trained text-image diffusion model, which enhances the detector's performance via improved data diversity and HOI representation. We demonstrate that the internal representation space of a frozen text-to-image diffusion model is highly relevant to verb concepts and their corresponding context. Accordingly, we propose an adapter-style tuning method to extract the various semantic associated representation from a frozen diffusion model and CLIP model to enhance the human and object representations from the pre-trained detector, further reducing the ambiguity in interaction prediction. Moreover, to fill in the gaps of HOI datasets, we propose SynHOI, a class-balance, large-scale, and high-diversity synthetic dataset containing over 140K HOI images with fully triplet annotations. It is built using an automatic and scalable pipeline designed to scale up the generation of diverse and high-precision HOI-annotated data. SynHOI could effectively relieve the long-tail issue in existing datasets and facilitate learning interaction representations. Extensive experiments demonstrate that DiffHOI significantly outperforms the state-of-the-art in regular detection (i.e., 41.50 mAP) and zero-shot detection. Furthermore, SynHOI can improve the performance of model-agnostic and backbone-agnostic HOI detection, particularly exhibiting an outstanding 11.55% mAP improvement in rare classes.
Paper Structure (3 sections, 1 figure)

This paper contains 3 sections, 1 figure.

Table of Contents

  1. Introduction
  2. Related Work
  3. Conclusion

Figures (1)

  • Figure 1: We show a) the long-tail distribution issue in HICO-DET and b) the high correlation between HOI text (i.e., nouns and verbs) and internal image features within the frozen stable diffusion.