Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Enhancing Product Search Interfaces with Sketch-Guided Diffusion and Language Agents

Edward Sun

TL;DR

This work tackles the limitations of single-modality sketch-based image search by introducing Sketch-Search Agent, a multimodal framework that couples a memory-enabled language agent with a T2I-adapted diffusion model to generate sketch- conditioned query images. The agent refines user sketches with natural language prompts, generates conditioned images, and uses CLIP-based embeddings to perform reverse image search over a product catalog, while updating user memory to personalize results. Key contributions include memory-aware interactions, prompt refinement, and diffusion-guided search that together deliver fast, accurate, and personalized product suggestions in an interactive interface. Experimental results on a real product index demonstrate improved success rates and reasonable latency, illustrating the potential of diffusion-guided, memory-enabled visual search for e-commerce and beyond.

Abstract

The rapid progress in diffusion models, transformers, and language agents has unlocked new possibilities, yet their potential in user interfaces and commercial applications remains underexplored. We present Sketch-Search Agent, a novel framework that transforms the image search experience by integrating a multimodal language agent with freehand sketches as control signals for diffusion models. Using the T2I-Adapter, Sketch-Search Agent combines sketches and text prompts to generate high-quality query images, encoded via a CLIP image encoder for efficient matching against an image corpus. Unlike existing methods, Sketch-Search Agent requires minimal setup, no additional training, and excels in sketch-based image retrieval and natural language interactions. The multimodal agent enhances user experience by dynamically retaining preferences, ranking results, and refining queries for personalized recommendations. This interactive design empowers users to create sketches and receive tailored product suggestions, showcasing the potential of diffusion models in user-centric image retrieval. Experiments confirm Sketch-Search Agent's high accuracy in delivering relevant product search results.

Enhancing Product Search Interfaces with Sketch-Guided Diffusion and Language Agents

TL;DR

This work tackles the limitations of single-modality sketch-based image search by introducing Sketch-Search Agent, a multimodal framework that couples a memory-enabled language agent with a T2I-adapted diffusion model to generate sketch- conditioned query images. The agent refines user sketches with natural language prompts, generates conditioned images, and uses CLIP-based embeddings to perform reverse image search over a product catalog, while updating user memory to personalize results. Key contributions include memory-aware interactions, prompt refinement, and diffusion-guided search that together deliver fast, accurate, and personalized product suggestions in an interactive interface. Experimental results on a real product index demonstrate improved success rates and reasonable latency, illustrating the potential of diffusion-guided, memory-enabled visual search for e-commerce and beyond.

Abstract

The rapid progress in diffusion models, transformers, and language agents has unlocked new possibilities, yet their potential in user interfaces and commercial applications remains underexplored. We present Sketch-Search Agent, a novel framework that transforms the image search experience by integrating a multimodal language agent with freehand sketches as control signals for diffusion models. Using the T2I-Adapter, Sketch-Search Agent combines sketches and text prompts to generate high-quality query images, encoded via a CLIP image encoder for efficient matching against an image corpus. Unlike existing methods, Sketch-Search Agent requires minimal setup, no additional training, and excels in sketch-based image retrieval and natural language interactions. The multimodal agent enhances user experience by dynamically retaining preferences, ranking results, and refining queries for personalized recommendations. This interactive design empowers users to create sketches and receive tailored product suggestions, showcasing the potential of diffusion models in user-centric image retrieval. Experiments confirm Sketch-Search Agent's high accuracy in delivering relevant product search results.

Paper Structure

This paper contains 8 sections, 4 equations, 4 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Works
  3. Methods
  4. Module Definitions
  5. User-Agent Interaction Steps
  6. Experiments
  7. Results
  8. Conclusion

Figures (4)

  • Figure 1: Sketch-Search Agent enables multimodal product search by integrating sketches with natural language refinement and personalized feedback from a language agent.
  • Figure 2: Sketch-Search Agent framework outline. Dashed lines represent possible information routes depending on the agent's choices and tool usage.
  • Figure 3: Averaged success rate ablations of Sketch-Search Agent
  • Figure 4: Averaged personalization score ablations of Sketch-Search Agent