PatentLMM: Large Multimodal Model for Generating Descriptions for Patent Figures

TL;DR

This work tackles the challenge of automatically generating descriptions for patent figures by introducing PatentDesc-355K, a large-scale patent figure dataset with brief and detailed descriptions, and PatentLMM, a two-component multimodal model. PatentLMM combines a patent-focused vision encoder (PatentMME) with a domain-adapted language model (PatentLLaMA), trained with specialized losses to capture patent-specific structural cues. Empirical results show substantial improvements over strong baselines in both brief and detailed description generation, validating the benefits of domain-specific pretraining and architecture. The research enables faster, more accurate patent document understanding and drafting, with public release of code and data guiding future work in multilingual patents and grounded, cross-figure reasoning.

Abstract

Writing comprehensive and accurate descriptions of technical drawings in patent documents is crucial to effective knowledge sharing and enabling the replication and protection of intellectual property. However, automation of this task has been largely overlooked by the research community. To this end, we introduce PatentDesc-355K, a novel large-scale dataset containing ~355K patent figures along with their brief and detailed textual descriptions extracted from more than 60K US patent documents. In addition, we propose PatentLMM - a novel multimodal large language model specifically tailored to generate high-quality descriptions of patent figures. Our proposed PatentLMM comprises two key components: (i) PatentMME, a specialized multimodal vision encoder that captures the unique structural elements of patent figures, and (ii) PatentLLaMA, a domain-adapted version of LLaMA fine-tuned on a large collection of patents. Extensive experiments demonstrate that training a vision encoder specifically designed for patent figures significantly boosts the performance, generating coherent descriptions compared to fine-tuning similar-sized off-the-shelf multimodal models. PatentDesc-355K and PatentLMM pave the way for automating the understanding of patent figures, enabling efficient knowledge sharing and faster drafting of patent documents. We make the code and data publicly available.

Figures (17)

• Figure 1: An example of generated and ground truth brief and detailed descriptions using our proposed PatentLMM.
• Figure 2: PatentMME Architecture. We jointly process OCR tokens and visual embeddings to produce multimodal context-aware embeddings. These contextual embeddings are optimized using our proposed MLM, LA-MIM and PC objectives.
• Figure 3: PatentLMM Architecture. Language Instruction is a fixed prompt guiding the model to generate either brief or detailed descriptions.
• Figure 4: PatentDesc-355K Analysis. (a) and (c): Word clouds of most common occurrences in brief and detailed descriptions respectively. (b) and (d): frequency distribution of description lengths for the brief and detailed descriptions respectively.
• Figure 5: Samples from our PatentDesc-355K dataset showing $\langle$patent figure, brief description, detailed description$\rangle$ triplets.