SGC-Net: Stratified Granular Comparison Network for Open-Vocabulary HOI Detection

TL;DR

This work tackles open-vocabulary HOI detection by addressing two core problems: feature granularity deficiency and semantic similarity confusion. It introduces SGC-Net, which combines Granularity Sensing Alignment to fuse multi-granularity CLIP features with trainable Gaussian weighting and visual prompt tuning, and Hierarchical Group Comparison to recursively group and compare HOI categories via LLMs, generating discriminative, hierarchy-aware text embeddings. Through extensive experiments on SWIG-HOI and HICO-DET, SGC-Net achieves state-of-the-art results without pretrained detectors, notably improving performance on unseen and rare categories and demonstrating strong transferability and interpretability. The approach offers a practical, detector-free solution for robust open-vocabulary HOI detection with meaningful CLIP-VLM alignment and discriminative language-driven classification.

Abstract

Recent open-vocabulary human-object interaction (OV-HOI) detection methods primarily rely on large language model (LLM) for generating auxiliary descriptions and leverage knowledge distilled from CLIP to detect unseen interaction categories. Despite their effectiveness, these methods face two challenges: (1) feature granularity deficiency, due to reliance on last layer visual features for text alignment, leading to the neglect of crucial object-level details from intermediate layers; (2) semantic similarity confusion, resulting from CLIP's inherent biases toward certain classes, while LLM-generated descriptions based solely on labels fail to adequately capture inter-class similarities. To address these challenges, we propose a stratified granular comparison network. First, we introduce a granularity sensing alignment module that aggregates global semantic features with local details, refining interaction representations and ensuring robust alignment between intermediate visual features and text embeddings. Second, we develop a hierarchical group comparison module that recursively compares and groups classes using LLMs, generating fine-grained and discriminative descriptions for each interaction category. Experimental results on two widely-used benchmark datasets, SWIG-HOI and HICO-DET, demonstrate that our method achieves state-of-the-art results in OV-HOI detection. Codes will be released on https://github.com/Phil0212/SGC-Net.

Figures (4)

• Figure 1: (a) The last layer capture high-level global semantics but contain fewer low-level local details compared to intermediate layers. The red dot marks the selected patch. (b) Category-level and part-level descriptions overlook inter-class similarity, leading to difficulty in distinguishing semantically similar classes.
• Figure 2: The framework of SGC-Net. It eliminates the need for a pretrained object detector and includes two new modules for OV-HOI detection: (1) The GSA module partitions the CLIP visual encoder into blocks, aggregates features via DGW, and integrates an HOI decoder for fine-grained HOI representations. (2) The HGC module uses LLM to recursively construct class hierarchy, enabling HOI classification by traversing the hierarchy from top to bottom and comparing HOI representations with text embeddings at each level.
• Figure 3: Evaluation on the value of grouping threshold $N$.
• Figure 4: Qualitative results of SGC-Net. On the left-hand side, we visualize feature correspondence through cosine similarity calculations from both deep and shallow layers, and compare the predicted HOI categories. On the right-hand side, we show two inference examples and the absolute score gap between two images at each hierarchical level by querying the description with extracted HOI representations.