Patent Figure Classification using Large Vision-language Models

TL;DR

The paper addresses the challenge of patent figure understanding to enable faceted patent retrieval by leveraging large vision-language models (LVLMs). It introduces two new datasets, PatFigVQA and PatFigCLS, and proposes a novel tournament-style multiple-choice classification approach to handle large label spaces in zero-shot and few-shot regimes. Through extensive experiments, the study shows that LVLMs can outperform CNN baselines on certain aspects (e.g., Type and USPC) and that the SemEq metric provides a more nuanced evaluation of semantic alignment between predicted and ground-truth concepts. The work demonstrates practical potential for improving patent figure analysis and retrieval, with public datasets and code enabling broader adoption and future LVLM-driven improvements in patent analytics.

Abstract

Patent figure classification facilitates faceted search in patent retrieval systems, enabling efficient prior art search. Existing approaches have explored patent figure classification for only a single aspect and for aspects with a limited number of concepts. In recent years, large vision-language models (LVLMs) have shown tremendous performance across numerous computer vision downstream tasks, however, they remain unexplored for patent figure classification. Our work explores the efficacy of LVLMs in patent figure visual question answering (VQA) and classification, focusing on zero-shot and few-shot learning scenarios. For this purpose, we introduce new datasets, PatFigVQA and PatFigCLS, for fine-tuning and evaluation regarding multiple aspects of patent figures~(i.e., type, projection, patent class, and objects). For a computational-effective handling of a large number of classes using LVLM, we propose a novel tournament-style classification strategy that leverages a series of multiple-choice questions. Experimental results and comparisons of multiple classification approaches based on LVLMs and Convolutional Neural Networks (CNNs) in few-shot settings show the feasibility of the proposed approaches.

Figures (5)

• Figure 1: A figure from patent USD534354S1 showing a drawing of a modular tool storage drawer in cross-sectional projection, and three different questions asking about various aspects of the figure: 1) Binary question asking about projection 2) Multiple-choice question asking about figure type, and 3) Open-ended question asking about object depicted. The figure also shows the response (and token probability) generated from an LVLM and the corresponding concept assigned to the figure.
• Figure 2: Workflow of patent figure classification using different LVLM-based classification approaches. On the left, question templates for different question types used to create PatFigVQA dataset are shown. On the right, three different approaches to figure classification using a fine-tuned LVLM is shown, which include Binary Classification [BC], Multiple-choice Classification - Tournament-style Strategy [MC-TS], and Open-ended Classification [OC].
• Figure 3: Average exact string matching accuracy of InstructBLIP (left) across different question types, and (right) across different aspects on the PatFigVQA dataset with increasing number of samples per concept.
• Figure 4: Confusion matrix on results produced by fine-tuned InstructBLIP using MC-TS (5) approach for aspects Type (left) and Projection (right)
• Figure 5: Qualitative examples for aspects Object (rows 1 and 2) and USPC (rows 3 and 4) comparing classification results of MC-TS and ResNext101