Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Annotating Scientific Uncertainty: A comprehensive model using linguistic patterns and comparison with existing approaches

Panggih Kusuma Ningrum, Philipp Mayr, Nina Smirnova, Iana Atanassova

TL;DR

This paper tackles the detection of scientific uncertainty in scholarly articles by introducing UnScientify, a weakly supervised, rule-driven pipeline that combines span-based uncertainty patterns, complex sentence checks, and authorial-reference analysis. It demonstrates that UnScientify achieves 0.808 accuracy on a curated 975-sentence corpus from 59 journals, outperforming fine-tuned BERT variants and various zero-shot LLM baselines. By releasing a gold-standard corpus (AURORA-MESS) and emphasizing interpretability, the work provides a reusable, domain-robust tool for information retrieval, text mining, and scientific document processing. The study also discusses the trade-offs between traditional pattern-based methods and large language models, highlighting practical advantages in efficiency, transparency, and adaptability to domain-specific uncertainty expressions.

Abstract

UnScientify, a system designed to detect scientific uncertainty in scholarly full text. The system utilizes a weakly supervised technique to identify verbally expressed uncertainty in scientific texts and their authorial references. The core methodology of UnScientify is based on a multi-faceted pipeline that integrates span pattern matching, complex sentence analysis and author reference checking. This approach streamlines the labeling and annotation processes essential for identifying scientific uncertainty, covering a variety of uncertainty expression types to support diverse applications including information retrieval, text mining and scientific document processing. The evaluation results highlight the trade-offs between modern large language models (LLMs) and the UnScientify system. UnScientify, which employs more traditional techniques, achieved superior performance in the scientific uncertainty detection task, attaining an accuracy score of 0.808. This finding underscores the continued relevance and efficiency of UnScientify's simple rule-based and pattern matching strategy for this specific application. The results demonstrate that in scenarios where resource efficiency, interpretability, and domain-specific adaptability are critical, traditional methods can still offer significant advantages.

Annotating Scientific Uncertainty: A comprehensive model using linguistic patterns and comparison with existing approaches

TL;DR

This paper tackles the detection of scientific uncertainty in scholarly articles by introducing UnScientify, a weakly supervised, rule-driven pipeline that combines span-based uncertainty patterns, complex sentence checks, and authorial-reference analysis. It demonstrates that UnScientify achieves 0.808 accuracy on a curated 975-sentence corpus from 59 journals, outperforming fine-tuned BERT variants and various zero-shot LLM baselines. By releasing a gold-standard corpus (AURORA-MESS) and emphasizing interpretability, the work provides a reusable, domain-robust tool for information retrieval, text mining, and scientific document processing. The study also discusses the trade-offs between traditional pattern-based methods and large language models, highlighting practical advantages in efficiency, transparency, and adaptability to domain-specific uncertainty expressions.

Abstract

UnScientify, a system designed to detect scientific uncertainty in scholarly full text. The system utilizes a weakly supervised technique to identify verbally expressed uncertainty in scientific texts and their authorial references. The core methodology of UnScientify is based on a multi-faceted pipeline that integrates span pattern matching, complex sentence analysis and author reference checking. This approach streamlines the labeling and annotation processes essential for identifying scientific uncertainty, covering a variety of uncertainty expression types to support diverse applications including information retrieval, text mining and scientific document processing. The evaluation results highlight the trade-offs between modern large language models (LLMs) and the UnScientify system. UnScientify, which employs more traditional techniques, achieved superior performance in the scientific uncertainty detection task, attaining an accuracy score of 0.808. This finding underscores the continued relevance and efficiency of UnScientify's simple rule-based and pattern matching strategy for this specific application. The results demonstrate that in scenarios where resource efficiency, interpretability, and domain-specific adaptability are critical, traditional methods can still offer significant advantages.

Paper Structure

This paper contains 29 sections, 8 figures, 12 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Rule-Based Approaches
  4. Machine Learning Approaches
  5. Large Language Models and Transformer-Based Approaches
  6. Research Objectives
  7. Materials and Methods
  8. Conceptual Definition of Scientific Uncertainty
  9. Data
  10. Approach
  11. Fine-grained SU annotation scheme and patterns formulation
  12. Authorial Reference Checking
  13. UnScientify Workflow
  14. UnScientify Performance Evaluation
  15. Fine-tuning BERT-based language representation models
  16. ...and 14 more sections

Figures (8)

  • Figure 1: Two annotated sentences with SU expressions. Samples of output from the span annotation process are shown in different colours based on their SU Pattern Group.
  • Figure 2: Scientific Uncertainty (SU) expression identification workflow
  • Figure 3: P-values of Pairwise Wilcoxon Rank Sum Test for classifier comparisons. P-values below the threshold of 0.05 indicate a statistically significant difference in the performance of classifiers.
  • Figure 4: Examples of result inconsistency from LLM (using identical model, setting & prompt)
  • Figure 5: Distribution of labels (claim/uncertainty) across three different zero-shot approaches of the knowledgator/comprehend_it-base1 model (top 50 data points)
  • ...and 3 more figures