Locality-Aware Zero-Shot Human-Object Interaction Detection

TL;DR

This work tackles zero-shot Human-Object Interaction detection by addressing CLIP's limitation in capturing locality within object regions. It introduces Locality Adapter (LA) and Interaction Adapter (IA) to infuse locality and interaction awareness into CLIP representations, enabling fine-grained HOI reasoning and stronger zero-shot generalization. HO tokens generated from DETR detections are refined through LA and IA across CLIP layers, with interaction cues modeled via cross-attention between human and object contexts and scores computed by aligning updated tokens with text embeddings. Experiments on HICO-DET and V-COCO show state-of-the-art zero-shot performance with efficient overhead, validating locality and interaction cues as essential for robust HOI generalization to unseen categories.

Abstract

Recent methods for zero-shot Human-Object Interaction (HOI) detection typically leverage the generalization ability of large Vision-Language Model (VLM), i.e., CLIP, on unseen categories, showing impressive results on various zero-shot settings. However, existing methods struggle to adapt CLIP representations for human-object pairs, as CLIP tends to overlook fine-grained information necessary for distinguishing interactions. To address this issue, we devise, LAIN, a novel zero-shot HOI detection framework enhancing the locality and interaction awareness of CLIP representations. The locality awareness, which involves capturing fine-grained details and the spatial structure of individual objects, is achieved by aggregating the information and spatial priors of adjacent neighborhood patches. The interaction awareness, which involves identifying whether and how a human is interacting with an object, is achieved by capturing the interaction pattern between the human and the object. By infusing locality and interaction awareness into CLIP representation, LAIN captures detailed information about the human-object pairs. Our extensive experiments on existing benchmarks show that LAIN outperforms previous methods on various zero-shot settings, demonstrating the importance of locality and interaction awareness for effective zero-shot HOI detection.

Figures (4)

• Figure 1: (a)-(b): Since CLIP primarily encodes global information, it struggles to capture the fine-grained details required to accurately identify interactions within human-object pairs. (c): When existing methods adapt CLIP representations to zero-shot HOI detection, this limitation hinders CLIP's generalization, and results in degraded performance which is even lower than CLIP's original zero-shot performance in UC-RF and UV settings.
• Figure 2: The overall architecture of LAIN. All valid human-object pairs are constructed and embedded into HO tokens based on detection results from a pre-trained DETR detr. Image patch tokens are passed through the Locality Adapter (LA), which infuses locality awareness into each patch token. The updated patch tokens and HO tokens are then passed through the Interaction Adapter (IA), which enhances each HO token with interaction awareness. The HO, $\textrm{CLS}$, patch tokens are subsequently refined by the frozen $l$-th ViT layer of the CLIP CLIP visual encoder. After repeating this process for $L$ layers, HOI scores are computed by measuring the cosine similarity between the HO tokens and text embeddings extracted from CLIP text encoder.
• Figure 3: Qualitative results on HICO-DET under UV settings. We represent a human with a red box and an object with a blue box, along with HOI score.
• Figure 4: Qualitative comparison of non-interactive pairs between LAIN and the baseline, i.e., without LA and IA, on the HICO-DET under the UV setting. We represent a human with a red box and an object with a blue box, along with HOI scores.