PatenTEB: A Comprehensive Benchmark and Model Family for Patent Text Embedding

TL;DR

Patents present long, structured texts that challenge generic embeddings. The authors propose PatenTEB, a 15-task, 2.06M-example benchmark with domain-stratified splits and hard negatives to reflect real-world patent matching, plus the patembed model family trained via multi-task learning across 13 training tasks and distillation to multiple sizes. External validation shows state-of-the-art results on BigPatentClustering.v2 (0.494 V-measure with patembed-base) and strong cross-domain retrieval on DAPFAM (0.377 NDCG@100 with patembed-large), while analyses reveal a 3–6× cross-domain retrieval gap and that domain pretraining yields robust gains. The work emphasizes that multi-task supervision and domain-aware design improve generalization beyond benchmark optimization, and it provides reproducible resources to advance patent text embedding research.

Abstract

Patent text embeddings enable prior art search, technology landscaping, and patent analysis, yet existing benchmarks inadequately capture patent-specific challenges. We introduce PatenTEB, a comprehensive benchmark comprising 15 tasks across retrieval, classification, paraphrase, and clustering, with 2.06 million examples. PatenTEB employs domain-stratified splits, domain specific hard negative mining, and systematic coverage of asymmetric fragment-to-document matching scenarios absent from general embedding benchmarks. We develop the patembed model family through multi-task training, spanning 67M to 344M parameters with context lengths up to 4096 tokens. External validation shows strong generalization: patembed-base achieves state-of-the-art on MTEB BigPatentClustering.v2 (0.494 V-measure vs. 0.445 previous best), while patembed-large achieves 0.377 NDCG@100 on DAPFAM. Systematic ablations reveal that multi-task training improves external generalization despite minor benchmark costs, and that domain-pretrained initialization provides consistent advantages across task families. All resources will be made available at https://github.com/iliass-y/patenteb. Keywords: patent retrieval, sentence embeddings, multi-task learning, asymmetric retrieval, benchmark evaluation, contrastive learning.

Figures (11)

• Figure 1: PatenTEB construction pipeline: data acquisition from Lens.org, family reconstruction, IPC3-level domain stratification (109 domains, $\ge$100 families each), 80/10/10 splitting, task construction (15 tasks across 4 families), and various quality controls.
• Figure 2: Per-task performance heatmap (test set): 15 tasks $\times$ 10 models. patembed-large achieves the highest score on 12 of the 15 tasks.
• Figure 3: Initial base models vs fine-tuned patembed variants : Multi-task fine-tuning yields substantial gains, with largest improvements on retrieval and paraphrase.
• Figure 4: PatenTEB task family performance (top-8 models). patembed-large achieves highest overall score (0.654) with balanced performance across retrieval, paraphrase, classification, and clustering.
• Figure 5: Domain difficulty variation for retrieval (top-8 models). IN-domain (shared IPC3, 0.29--0.51 NDCG) → MIXED-domain (partial overlap, 0.21--0.44) → OUT-domain (disjoint IPC3, 0.06--0.17) shows 3--6$\times$ performance degradation, highlighting cross-domain retrieval challenges.