Large Language Model Informed Patent Image Retrieval

TL;DR

The paper tackles the problem of image-based patent image retrieval, which has limited practical value when evaluated only on visual similarity. It introduces a language-informed, distribution-aware multimodal framework that enriches patent images with textual semantics generated via captions and large language models, and learns robust representations through a mix of instance- and coarse-grained losses modulo learnable uncertainty to address long-tail class distributions. On the DeepPatent2 dataset, the approach achieves state-of-the-art performance with substantial gains in mAP, Recall@K, and MRR@K, and is supported by qualitative analyses and a user study demonstrating real-world utility for patent professionals. The work highlights the importance of joint visual-textual representations and distribution-aware training to align retrieval with industrial workflows for novelty detection and prior-art search.

Abstract

In patent prosecution, image-based retrieval systems for identifying similarities between current patent images and prior art are pivotal to ensure the novelty and non-obviousness of patent applications. Despite their growing popularity in recent years, existing attempts, while effective at recognizing images within the same patent, fail to deliver practical value due to their limited generalizability in retrieving relevant prior art. Moreover, this task inherently involves the challenges posed by the abstract visual features of patent images, the skewed distribution of image classifications, and the semantic information of image descriptions. Therefore, we propose a language-informed, distribution-aware multimodal approach to patent image feature learning, which enriches the semantic understanding of patent image by integrating Large Language Models and improves the performance of underrepresented classes with our proposed distribution-aware contrastive losses. Extensive experiments on DeepPatent2 dataset show that our proposed method achieves state-of-the-art or comparable performance in image-based patent retrieval with mAP +53.3%, Recall@10 +41.8%, and MRR@10 +51.9%. Furthermore, through an in-depth user analysis, we explore our model in aiding patent professionals in their image retrieval efforts, highlighting the model's real-world applicability and effectiveness.

Figures (3)

• Figure 1: Model Architecture. (a) Generation of diverse, alias-containing, fine-grained descriptions for each patent image using a captioner and LLMs. (b) Text feature extraction from the enriched text via a frozen text encoder. (c) Visual feature extraction through a trainable visual encoder; a projector is employed when a mismatch between text and visual features. (d) Proposed distribution-aware contrastive losses. (e) Query patent images are converted into embeddings for retrieval based on cosine similarity.
• Figure 2: Qualitative Results of the Image-Based Patent Image Retrieval System. The leftmost image represents the query image, annotated with its object name and classification as either a head or tail class. The middle section displays the retrieval results from our method, where images framed in black indicate a match with the query image's class, and those framed in red indicate a mismatch. The right section shows the comparative results using the previous state-of-the-art method (SwinV2-B + ArcFace).
• Figure 3: t-SNE Visualization of Image Embeddings. This figure presents a two-dimensional t-SNE projection of randomly sampled 2,000 image embeddings, with each axis representing one dimension. Different colors and shapes in the plot indicate distinct classes. The left subplot illustrates the results from our model, while the right subplot displays the results using the previous state-of-the-art, SwinV2-B+ArcFace.