From Pixels to Purchase: Building and Evaluating a Taxonomy-Decoupled Visual Search Engine for Home Goods E-commerce

TL;DR

This paper tackles visual search for style-driven home goods by decoupling object localization from taxonomy-based classification, enabling flexible, appearance-based retrieval. It introduces a classification-free, taxonomy-decoupled pipeline with a YOLOX superclass detector and a unified OpenCLIP-based embedding space, plus a scalable online retrieval system built on Vertex AI Vector Search. To evaluate such a system without noisy catalog labels, the authors propose an LLM-as-a-Judge framework that assesses category relevance and visual similarity, validated against human judgments. Deployed at Wayfair, the approach delivers improved retrieval quality and measurable uplifts in live engagement and discovery metrics, with offline metrics closely correlating with real-world outcomes, demonstrating practical impact.

Abstract

Visual search is critical for e-commerce, especially in style-driven domains where user intent is subjective and open-ended. Existing industrial systems typically couple object detection with taxonomy-based classification and rely on catalog data for evaluation, which is prone to noise that limits robustness and scalability. We propose a taxonomy-decoupled architecture that uses classification-free region proposals and unified embeddings for similarity retrieval, enabling a more flexible and generalizable visual search. To overcome the evaluation bottleneck, we propose an LLM-as-a-Judge framework that assesses nuanced visual similarity and category relevance for query-result pairs in a zero-shot manner, removing dependence on human annotations or noise-prone catalog data. Deployed at scale on a global home goods platform, our system improves retrieval quality and yields a measurable uplift in customer engagement, while our offline evaluation metrics strongly correlate with real-world outcomes.

Figures (3)

• Figure 1: An overview of our taxonomy-decoupled visual search architecture. 1) Offline Indexing: Product images from the catalog are encoded into unified embeddings to build a large-scale vector index. 2) Online Serving: A taxonomy-decoupled object detector first localizes objects in the user's query image. Each detected object is then encoded and used to query the vector index. The retrieved candidates are subsequently processed through a multi-stage refinement pipeline to generate a shoppable gallery.
• Figure 2: LLM-as-a-Judge evaluation framework evaluates each query-result pair in three steps: 1) Category Relevance rating whether the retrieved item belongs to the same product category as the query on a 3-point scale; 2) Visual Similarity rating the aesthetic and functional similarity between query and result on a 5-point scale; and 3) Rating Consistency verifying logical consistency between the two ratings and refining judgments when predefined conflicts arise (e.g., low category relevance but high visual similarity).
• Figure 3: Qualitative comparison. Left: Our system detects multiple objects ("Sofas", "Coffee Tables"). For the primary object, it retrieves similar products from product categories Loveseats and Sofas, organized under a broad class "Sofas" (\ref{['sec:proxy']}). Center: Google Lens results. Right: Evaluation using granular product categories. Human validation of the LLM-as-a-Judge assessment confirms that retrieving Loveseats for a sofa query is correct due to their shared straight-line configuration, while penalizing Sectionals (from Google Lens) for their incompatible L-shaped structure.