Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Efficient Scientific Full Text Classification: The Case of EICAT Impact Assessments

Marc Felix Brinner, Sina Zarrieß

TL;DR

The paper tackles efficient full-text classification of scientific literature by comparing small BERT-based encoders and local LLMs on a new EICAT-based invasion biology dataset. It introduces a sentence-selection pipeline to reduce input length while preserving signal, demonstrating that selecting informative sentences—especially using an evidence-based selector—can improve classification accuracy and efficiency, sometimes surpassing models that process full text. The results reveal that encoder models like PubMedBERT typically outperform the local Llama-3.1 8B LLM on this task, though randomized and entropy/importance-based strategies can boost performance further. The work presents a generalizable workflow for accelerating inference in scientific text classification and discusses practical implications, limitations, and directions for future work, including fine-tuning local LLMs and exploring more capable models.

Abstract

This study explores strategies for efficiently classifying scientific full texts using both small, BERT-based models and local large language models like Llama-3.1 8B. We focus on developing methods for selecting subsets of input sentences to reduce input size while simultaneously enhancing classification performance. To this end, we compile a novel dataset consisting of full-text scientific papers from the field of invasion biology, specifically addressing the impacts of invasive species. These papers are aligned with publicly available impact assessments created by researchers for the International Union for Conservation of Nature (IUCN). Through extensive experimentation, we demonstrate that various sources like human evidence annotations, LLM-generated annotations or explainability scores can be used to train sentence selection models that improve the performance of both encoder- and decoder-based language models while optimizing efficiency through the reduction in input length, leading to improved results even if compared to models like ModernBERT that are able to handle the complete text as input. Additionally, we find that repeated sampling of shorter inputs proves to be a very effective strategy that, at a slightly increased cost, can further improve classification performance.

Efficient Scientific Full Text Classification: The Case of EICAT Impact Assessments

TL;DR

The paper tackles efficient full-text classification of scientific literature by comparing small BERT-based encoders and local LLMs on a new EICAT-based invasion biology dataset. It introduces a sentence-selection pipeline to reduce input length while preserving signal, demonstrating that selecting informative sentences—especially using an evidence-based selector—can improve classification accuracy and efficiency, sometimes surpassing models that process full text. The results reveal that encoder models like PubMedBERT typically outperform the local Llama-3.1 8B LLM on this task, though randomized and entropy/importance-based strategies can boost performance further. The work presents a generalizable workflow for accelerating inference in scientific text classification and discusses practical implications, limitations, and directions for future work, including fine-tuning local LLMs and exploring more capable models.

Abstract

This study explores strategies for efficiently classifying scientific full texts using both small, BERT-based models and local large language models like Llama-3.1 8B. We focus on developing methods for selecting subsets of input sentences to reduce input size while simultaneously enhancing classification performance. To this end, we compile a novel dataset consisting of full-text scientific papers from the field of invasion biology, specifically addressing the impacts of invasive species. These papers are aligned with publicly available impact assessments created by researchers for the International Union for Conservation of Nature (IUCN). Through extensive experimentation, we demonstrate that various sources like human evidence annotations, LLM-generated annotations or explainability scores can be used to train sentence selection models that improve the performance of both encoder- and decoder-based language models while optimizing efficiency through the reduction in input length, leading to improved results even if compared to models like ModernBERT that are able to handle the complete text as input. Additionally, we find that repeated sampling of shorter inputs proves to be a very effective strategy that, at a slightly increased cost, can further improve classification performance.

Paper Structure

This paper contains 18 sections, 1 figure, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Language Models for Scientific Literature
  4. Language Models for Biodiversity Science
  5. The EICAT Dataset
  6. Baseline Classification Experiments
  7. Experimental Setup
  8. Results
  9. Evidence Sentence Selection
  10. Experimental Setup
  11. Results
  12. Sentence Selector Agreement
  13. BERT Classification Results
  14. Llama Classification Results
  15. Randomization
  16. ...and 3 more sections

Figures (1)

  • Figure 1: The Llama-3.1 8B prompt to classify scientific full texts.