LookBench: A Live and Holistic Open Benchmark for Fashion Image Retrieval

TL;DR

The paper introduces LookBench, a live, contamination-aware benchmark for fashion image retrieval that reflects real-world e-commerce intents by combining real and AI-generated images across four subsets with time-stamped samples. It presents a category–attribute taxonomy and a pre-annotated evaluation protocol, enabling fine-grained, attribute-aware retrieval evaluation and robust contamination control. The authors propose GensmoRetro (GR), including GR-Pro and GR-Lite, text-free visual encoders trained on large fashion corpora with ArcFace objectives, demonstrating state-of-the-art performance on LookBench and strong transfer to legacy datasets like Fashion200K. Through comprehensive experiments, LookBench reveals gaps in generic vision-language models for attribute-faithful fashion search and establishes a scalable, reusable framework for ongoing benchmarking and model development in fashion retrieval.

Abstract

In this paper, we present LookBench (We use the term "look" to reflect retrieval that mirrors how people shop -- finding the exact item, a close substitute, or a visually consistent alternative.), a live, holistic and challenging benchmark for fashion image retrieval in real e-commerce settings. LookBench includes both recent product images sourced from live websites and AI-generated fashion images, reflecting contemporary trends and use cases. Each test sample is time-stamped and we intend to update the benchmark periodically, enabling contamination-aware evaluation aligned with declared training cutoffs. Grounded in our fine-grained attribute taxonomy, LookBench covers single-item and outfit-level retrieval across. Our experiments reveal that LookBench poses a significant challenge on strong baselines, with many models achieving below $60\%$ Recall@1. Our proprietary model achieves the best performance on LookBench, and we release an open-source counterpart that ranks second, with both models attaining state-of-the-art results on legacy Fashion200K evaluations. LookBench is designed to be updated semi-annually with new test samples and progressively harder task variants, providing a durable measure of progress. We publicly release our leaderboard, dataset, evaluation code, and trained models.

Figures (12)

• Figure 1: Fine Recall@1 of all evaluated models on the two LookBench subsets, RealStudioFlat and RealStreetLook. Our GR-Pro and GR-Lite models achieve the highest scores on both tasks.
• Figure 2: Illustration of the StreetLook multi-item retrieval task in LookBench. Given a street-look outfit image (left), we detect the shirt, skirt, bag and shoes and retrieve ranked outfit candidates from the catalog. For every retrieved product image we attach its pre-annotated attributes, and the rank (Rank #1– #5 in the benchmark; only Rank #1–#4 are shown here for readability) is determined by how many attributes match the corresponding query item. Attributes shown in red indicate attribute-level mismatches. For simplicity, this example does not distinguish between the main and other attributes; all attributes are displayed uniformly.
• Figure 3: RealStreetLook retrieval sets construction pipeline in LookBench.
• Figure 4: LookBench dataset overview. (a) The benchmark spans a diverse set of apparel categories. (b) Each query is paired with rich metadata to support fine-grained fashion image retrieval.
• Figure 5: Outfit-level fine-grained retrieval performance on RealStreetLook subset of LookBench.