Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

PhysBERT: A Text Embedding Model for Physics Scientific Literature

Thorsten Hellert, João Montenegro, Andrea Pollastro

TL;DR

Physics NLP faces language and concept gaps that limit general embeddings. PhysBERT addresses this by a physics-domain transformer-based sentence embedder trained on 1.2 million arXiv papers with a custom tokenizer, followed by SimCSE-based fine-tuning to obtain semantic representations. Across category clustering, information retrieval, and citation classification, PhysBERT surpasses general-purpose models and proves effective as a foundation for subdomain fine-tuning, with results maintained at a smaller parameter footprint. By releasing training data and weights, the work enables more accurate physics literature search and domain-adaptive NLP in physics.

Abstract

The specialized language and complex concepts in physics pose significant challenges for information extraction through Natural Language Processing (NLP). Central to effective NLP applications is the text embedding model, which converts text into dense vector representations for efficient information retrieval and semantic analysis. In this work, we introduce PhysBERT, the first physics-specific text embedding model. Pre-trained on a curated corpus of 1.2 million arXiv physics papers and fine-tuned with supervised data, PhysBERT outperforms leading general-purpose models on physics-specific tasks including the effectiveness in fine-tuning for specific physics subdomains.

PhysBERT: A Text Embedding Model for Physics Scientific Literature

TL;DR

Physics NLP faces language and concept gaps that limit general embeddings. PhysBERT addresses this by a physics-domain transformer-based sentence embedder trained on 1.2 million arXiv papers with a custom tokenizer, followed by SimCSE-based fine-tuning to obtain semantic representations. Across category clustering, information retrieval, and citation classification, PhysBERT surpasses general-purpose models and proves effective as a foundation for subdomain fine-tuning, with results maintained at a smaller parameter footprint. By releasing training data and weights, the work enables more accurate physics literature search and domain-adaptive NLP in physics.

Abstract

The specialized language and complex concepts in physics pose significant challenges for information extraction through Natural Language Processing (NLP). Central to effective NLP applications is the text embedding model, which converts text into dense vector representations for efficient information retrieval and semantic analysis. In this work, we introduce PhysBERT, the first physics-specific text embedding model. Pre-trained on a curated corpus of 1.2 million arXiv physics papers and fine-tuned with supervised data, PhysBERT outperforms leading general-purpose models on physics-specific tasks including the effectiveness in fine-tuning for specific physics subdomains.
Paper Structure (17 sections, 2 figures, 2 tables)

This paper contains 17 sections, 2 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Training
  3. Downstream tasks
  4. Category Clustering
  5. Information Retrieval
  6. Citation Classification
  7. Fine-tuning on physics subdomains
  8. Datasets
  9. Unsupervised pre-training
  10. Supervised fine-tuning
  11. Abstract pairs from categories:
  12. Citation pairs:
  13. Synthetic Query-Source Data:
  14. Model evaluation data
  15. results
  16. ...and 2 more sections

Figures (2)

  • Figure 1: Schematic overview of the steps involved in developing PhysBERT. The process begins with pre-training on a large corpus from arXiv, followed by supervised fine-tuning using SimCSE. Finally, the model is evaluated on downstream tasks such as citation classification, category clustering, information retrieval, and sub-domain fine-tuning.
  • Figure 2: Comparison of embedding space visualizations for PhysBERT (left) and bge-base-v1.5 bge_embedding (right, see also Table \ref{['tab:results']}), using PCA on text embeddings from 500 random abstracts per physics category. It is worth noting that no explicit clustering algorithm was applied; the observed patterns reflect the model’s internal organization of the data.