Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

HyperPIE: Hyperparameter Information Extraction from Scientific Publications

Tarek Saier, Mayumi Ohta, Takuto Asakura, Michael Färber

TL;DR

HyperPIE introduces a novel information extraction task to obtain hyperparameter information from scientific publications, framing it as simultaneous entity recognition and relation extraction. It provides a manually labeled, cross-disciplinary ML dataset and develops two complementary approaches: fine-tuned BERT-based models with a dedicated relation extractor, and large language models guided by YAML-based structured outputs in zero- and few-shot settings. The results show strong gains for the supervised approach in both entity and relation extraction, while LLMs benefit from structured prompts and few-shot context, with overall ER improvements and robust handling of complex notation. The work enables scalable, reproducible extraction of hyperparameters, supporting reproducibility research, enhanced search/recommendation, and knowledge-graph construction from large publication corpora.

Abstract

Automatic extraction of information from publications is key to making scientific knowledge machine readable at a large scale. The extracted information can, for example, facilitate academic search, decision making, and knowledge graph construction. An important type of information not covered by existing approaches is hyperparameters. In this paper, we formalize and tackle hyperparameter information extraction (HyperPIE) as an entity recognition and relation extraction task. We create a labeled data set covering publications from a variety of computer science disciplines. Using this data set, we train and evaluate BERT-based fine-tuned models as well as five large language models: GPT-3.5, GALACTICA, Falcon, Vicuna, and WizardLM. For fine-tuned models, we develop a relation extraction approach that achieves an improvement of 29% F1 over a state-of-the-art baseline. For large language models, we develop an approach leveraging YAML output for structured data extraction, which achieves an average improvement of 5.5% F1 in entity recognition over using JSON. With our best performing model we extract hyperparameter information from a large number of unannotated papers, and analyze patterns across disciplines. All our data and source code is publicly available at https://github.com/IllDepence/hyperpie

HyperPIE: Hyperparameter Information Extraction from Scientific Publications

TL;DR

HyperPIE introduces a novel information extraction task to obtain hyperparameter information from scientific publications, framing it as simultaneous entity recognition and relation extraction. It provides a manually labeled, cross-disciplinary ML dataset and develops two complementary approaches: fine-tuned BERT-based models with a dedicated relation extractor, and large language models guided by YAML-based structured outputs in zero- and few-shot settings. The results show strong gains for the supervised approach in both entity and relation extraction, while LLMs benefit from structured prompts and few-shot context, with overall ER improvements and robust handling of complex notation. The work enables scalable, reproducible extraction of hyperparameters, supporting reproducibility research, enhanced search/recommendation, and knowledge-graph construction from large publication corpora.

Abstract

Automatic extraction of information from publications is key to making scientific knowledge machine readable at a large scale. The extracted information can, for example, facilitate academic search, decision making, and knowledge graph construction. An important type of information not covered by existing approaches is hyperparameters. In this paper, we formalize and tackle hyperparameter information extraction (HyperPIE) as an entity recognition and relation extraction task. We create a labeled data set covering publications from a variety of computer science disciplines. Using this data set, we train and evaluate BERT-based fine-tuned models as well as five large language models: GPT-3.5, GALACTICA, Falcon, Vicuna, and WizardLM. For fine-tuned models, we develop a relation extraction approach that achieves an improvement of 29% F1 over a state-of-the-art baseline. For large language models, we develop an approach leveraging YAML output for structured data extraction, which achieves an average improvement of 5.5% F1 in entity recognition over using JSON. With our best performing model we extract hyperparameter information from a large number of unannotated papers, and analyze patterns across disciplines. All our data and source code is publicly available at https://github.com/IllDepence/hyperpie
Paper Structure (24 sections, 6 figures, 3 tables)

This paper contains 24 sections, 6 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Fine-Tuned Models
  4. LLMs
  5. Hyperparameter Information Extraction
  6. Task Definition
  7. Data Set Construction
  8. Initial Annotation Round
  9. Main Annotation Round
  10. Methods
  11. Fine-Tuned Models
  12. LLM
  13. Zero-shot
  14. Few-shot
  15. Experiments
  16. ...and 9 more sections

Figures (6)

  • Figure 1: Illustration of hyperparameter information in a text example alongside the extracted entities and relations. Entity types are research artifact, parameter, value, and context. Relations are indicated by arrows.
  • Figure 2: Observations of initial annotation round
  • Figure 3: RE with emphasis on entity candidate pair types and distance.
  • Figure 4: Fine-tuned model evaluation (5-fold cross validation).
  • Figure 5: Mentioning position in papers.
  • ...and 1 more figures