Holi-DETR: Holistic Fashion Item Detection Leveraging Contextual Information

TL;DR

Fashion item detection suffers from ambiguities due to diverse appearances and subcategory similarities. Holi-DETR introduces a holistic, DETR-based detector that injects three contextual cues—item co-occurrence, inter-item spatial relations, and item–body keypoint relations—via a context encoder and a pose estimator to modulate attention. The approach yields consistent gains over DETR and Co-DETR on the Showniq-H dataset, with ablations clarifying the contribution of each contextual cue. This methodology enhances disambiguation among similar subcategories and offers a scalable path for more reliable fashion item detection in realistic outfits.

Abstract

Fashion item detection is challenging due to the ambiguities introduced by the highly diverse appearances of fashion items and the similarities among item subcategories. To address this challenge, we propose a novel Holistic Detection Transformer (Holi-DETR) that detects fashion items in outfit images holistically, by leveraging contextual information. Fashion items often have meaningful relationships as they are combined to create specific styles. Unlike conventional detectors that detect each item independently, Holi-DETR detects multiple items while reducing ambiguities by leveraging three distinct types of contextual information: (1) the co-occurrence relationship between fashion items, (2) the relative position and size based on inter-item spatial arrangements, and (3) the spatial relationships between items and human body key-points. %Holi-DETR explicitly incorporates three types of contextual information: (1) the co-occurrence probability between fashion items, (2) the relative position and size based on inter-item spatial arrangements, and (3) the spatial relationships between items and human body key-points. To this end, we propose a novel architecture that integrates these three types of heterogeneous contextual information into the Detection Transformer (DETR) and its subsequent models. In experiments, the proposed methods improved the performance of the vanilla DETR and the more recently developed Co-DETR by 3.6 percent points (pp) and 1.1 pp, respectively, in terms of average precision (AP).

Figures (6)

• Figure 1: Example of interclass similarity and intraclass diversity. The jacket in the (b) is more similar to the coat in the (c) than the jacket in the (a) in terms of collar, pockets, and shape of the garment
• Figure 2: The overall structure of Holi-DETR. The Transformer decoder generates output embeddings based on the object queries, the image features from the encoder, and three additional types of contextual information: item-item co-occurrence, item-item relational positions, and item-human relative positions
• Figure 3: Holi-DETR decoder layer. The context encoder extracts item-item co-occurrence matrices, item-item relative position matrices, and item-human relative position matrices, combines them, and encodes them into relation weights, which are then added to the self-attention weights. $N$ and $N_C$ denote the number of object queries and the number of classes, respectively, while $d_{model}$ refers to the feature dimension of the model
• Figure 4: Examples of fashion images in the Showniq-H dataset
• Figure 5: Examples of fashion item detection results. The left, center, and right columns represent the ground-truth labels, DETR results, and Holi-DETR results, respectively