Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Encoder Fine-tuning with Stochastic Sampling Outperforms Open-weight GPT in Astronomy Knowledge Extraction

Shivam Rawat, Lucie Flek, Akbar Karimi

TL;DR

The paper tackles scalable knowledge extraction from astronomy literature by predicting telescope usage and instrumentation mentions. It advances an encoder-based approach using SciBERT with domain-specific fine-tuning on 512-token article segments, employing majority voting during inference and two segment-sampling regimes. On the TRACS dataset, the method achieves macro F1 scores around 0.72–0.73, markedly outperforming a 20B-parameter open-weight GPT baseline (~0.31) and the initial SciBERT baseline (~0.18). The work demonstrates that lightweight, domain-aligned NLP models can surpass large general-purpose LLMs for specialized scientific information extraction, enabling efficient linking of observational data to the scholarly record.

Abstract

Scientific literature in astronomy is rapidly expanding, making it increasingly important to automate the extraction of key entities and contextual information from research papers. In this paper, we present an encoder-based system for extracting knowledge from astronomy articles. Our objective is to develop models capable of classifying telescope references, detecting auxiliary semantic attributes, and recognizing instrument mentions from textual content. To this end, we implement a multi-task transformer-based system built upon the SciBERT model and fine-tuned for astronomy corpora classification. To carry out the fine-tuning, we stochastically sample segments from the training data and use majority voting over the test segments at inference time. Our system, despite its simplicity and low-cost implementation, significantly outperforms the open-weight GPT baseline.

Encoder Fine-tuning with Stochastic Sampling Outperforms Open-weight GPT in Astronomy Knowledge Extraction

TL;DR

The paper tackles scalable knowledge extraction from astronomy literature by predicting telescope usage and instrumentation mentions. It advances an encoder-based approach using SciBERT with domain-specific fine-tuning on 512-token article segments, employing majority voting during inference and two segment-sampling regimes. On the TRACS dataset, the method achieves macro F1 scores around 0.72–0.73, markedly outperforming a 20B-parameter open-weight GPT baseline (~0.31) and the initial SciBERT baseline (~0.18). The work demonstrates that lightweight, domain-aligned NLP models can surpass large general-purpose LLMs for specialized scientific information extraction, enabling efficient linking of observational data to the scholarly record.

Abstract

Scientific literature in astronomy is rapidly expanding, making it increasingly important to automate the extraction of key entities and contextual information from research papers. In this paper, we present an encoder-based system for extracting knowledge from astronomy articles. Our objective is to develop models capable of classifying telescope references, detecting auxiliary semantic attributes, and recognizing instrument mentions from textual content. To this end, we implement a multi-task transformer-based system built upon the SciBERT model and fine-tuned for astronomy corpora classification. To carry out the fine-tuning, we stochastically sample segments from the training data and use majority voting over the test segments at inference time. Our system, despite its simplicity and low-cost implementation, significantly outperforms the open-weight GPT baseline.

Paper Structure

This paper contains 11 sections, 1 equation, 2 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Task Description
  3. Objective
  4. Dataset
  5. Data Statistics and Preprocessing
  6. Experiments and Results
  7. Model Selection
  8. Experiments and Results
  9. Error Analysis
  10. Conclusion and Future Work
  11. Limitations

Figures (2)

  • Figure 1: System design for the shared task. Input documents are chunked into equal segments with 512 tokens. Each segment is given the same label as the original input and is used to fine-tune the model. At inference time, we use majority voting to assign the test labels.
  • Figure 2: Telescope predictions confusion matrix