Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Evaluating Large Language Models for Structured Science Summarization in the Open Research Knowledge Graph

Vladyslav Nechakhin, Jennifer D'Souza, Steffen Eger

TL;DR

The paper investigates whether Large Language Models (GPT-3.5, Llama 2, and Mistral) can automatically generate structured research dimensions for Open Research Knowledge Graph (ORKG) contributions. It builds a gold-standard, multidisciplinary ORKG dataset and evaluates LLM outputs against domain-expert properties using semantic alignment, fine-grained mappings, embedding-based similarity via SciNCL, and expert surveys. Results show moderate semantic alignment but notable deviation and low mapping overlap, with embedding-based similarity peaking for GPT-3.5; experts viewed LLM suggestions as useful but not a substitute for manual curation. The work demonstrates potential for LLMs as recommendation services to aid structured science summarization and highlights the need for domain-specific fine-tuning and user-centered prompts to improve readiness for practical deployment.

Abstract

Structured science summaries or research contributions using properties or dimensions beyond traditional keywords enhances science findability. Current methods, such as those used by the Open Research Knowledge Graph (ORKG), involve manually curating properties to describe research papers' contributions in a structured manner, but this is labor-intensive and inconsistent between the domain expert human curators. We propose using Large Language Models (LLMs) to automatically suggest these properties. However, it's essential to assess the readiness of LLMs like GPT-3.5, Llama 2, and Mistral for this task before application. Our study performs a comprehensive comparative analysis between ORKG's manually curated properties and those generated by the aforementioned state-of-the-art LLMs. We evaluate LLM performance through four unique perspectives: semantic alignment and deviation with ORKG properties, fine-grained properties mapping accuracy, SciNCL embeddings-based cosine similarity, and expert surveys comparing manual annotations with LLM outputs. These evaluations occur within a multidisciplinary science setting. Overall, LLMs show potential as recommendation systems for structuring science, but further finetuning is recommended to improve their alignment with scientific tasks and mimicry of human expertise.

Evaluating Large Language Models for Structured Science Summarization in the Open Research Knowledge Graph

TL;DR

The paper investigates whether Large Language Models (GPT-3.5, Llama 2, and Mistral) can automatically generate structured research dimensions for Open Research Knowledge Graph (ORKG) contributions. It builds a gold-standard, multidisciplinary ORKG dataset and evaluates LLM outputs against domain-expert properties using semantic alignment, fine-grained mappings, embedding-based similarity via SciNCL, and expert surveys. Results show moderate semantic alignment but notable deviation and low mapping overlap, with embedding-based similarity peaking for GPT-3.5; experts viewed LLM suggestions as useful but not a substitute for manual curation. The work demonstrates potential for LLMs as recommendation services to aid structured science summarization and highlights the need for domain-specific fine-tuning and user-centered prompts to improve readiness for practical deployment.

Abstract

Structured science summaries or research contributions using properties or dimensions beyond traditional keywords enhances science findability. Current methods, such as those used by the Open Research Knowledge Graph (ORKG), involve manually curating properties to describe research papers' contributions in a structured manner, but this is labor-intensive and inconsistent between the domain expert human curators. We propose using Large Language Models (LLMs) to automatically suggest these properties. However, it's essential to assess the readiness of LLMs like GPT-3.5, Llama 2, and Mistral for this task before application. Our study performs a comprehensive comparative analysis between ORKG's manually curated properties and those generated by the aforementioned state-of-the-art LLMs. We evaluate LLM performance through four unique perspectives: semantic alignment and deviation with ORKG properties, fine-grained properties mapping accuracy, SciNCL embeddings-based cosine similarity, and expert surveys comparing manual annotations with LLM outputs. These evaluations occur within a multidisciplinary science setting. Overall, LLMs show potential as recommendation systems for structuring science, but further finetuning is recommended to improve their alignment with scientific tasks and mimicry of human expertise.
Paper Structure (18 sections, 11 figures, 4 tables)

This paper contains 18 sections, 11 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Materials and Methods
  4. Material: Our Evaluation Dataset
  5. Method: Three Large Language Models Applied for Research Dimensions Generation
  6. The three LLMs
  7. Research dimensions generation prompt for the LLMs
  8. Method: Three Types of Similarity Evaluations between ORKG Properties vs. LLM-generated Research Dimensions
  9. Semantic alignment and deviation evaluations using GPT-3.5
  10. ORKG property and LLM-generated research dimension mappings by GPT-3.5
  11. Scientific embeddings-based semantic distance evaluations
  12. Human assessment survey comparing ORKG properties with LLM-generated research dimensions
  13. Results and Discussion
  14. Semantic alignment and deviation evaluations
  15. Property and research dimension mappings
  16. ...and 3 more sections

Figures (11)

  • Figure 1: ORKG Comparison - A Catalog of Transformer Models (https://orkg.org/comparison/R656113/)
  • Figure 2: ORKG Comparison - Survey of sequencing techniques (https://orkg.org/comparison/R44668/)
  • Figure 3: Research field distribution of the selected papers in our evaluation dataset containing domain-expert human-annotated properties that were applied to represent the paper's structured contribution descriptions in the Open Research Knowledge Graph (ORKG).
  • Figure 4: Example of ORKG properties shown to survey respondents
  • Figure 5: Example of GPT dimensions shown to survey respondents
  • ...and 6 more figures