SciLitLLM: How to Adapt LLMs for Scientific Literature Understanding

TL;DR

This work tackles the challenge of enabling large language models to understand scientific literature by coupling continual pre-training on high-quality scientific corpora with supervised fine-tuning on diverse, synthetic scientific instructions. It introduces SciLitLLM, with 7B and 14B variants, and SciLitIns, a large, curated instruction set generated to cover underrepresented scientific domains. Experimental results on SciRIFF and SciAssess show notable gains, with the 14B variant surpassing many open-source baselines and the 7B variant achieving strong results even against larger models. The proposed CPT+SFT pipeline, plus instruction synthesis and quality control, offers a reproducible path for domain adaptation of language models beyond science.

Abstract

Scientific literature understanding is crucial for extracting targeted information and garnering insights, thereby significantly advancing scientific discovery. Despite the remarkable success of Large Language Models (LLMs), they face challenges in scientific literature understanding, primarily due to (1) a lack of scientific knowledge and (2) unfamiliarity with specialized scientific tasks. To develop an LLM specialized in scientific literature understanding, we propose a hybrid strategy that integrates continual pre-training (CPT) and supervised fine-tuning (SFT), to simultaneously infuse scientific domain knowledge and enhance instruction-following capabilities for domain-specific tasks.cIn this process, we identify two key challenges: (1) constructing high-quality CPT corpora, and (2) generating diverse SFT instructions. We address these challenges through a meticulous pipeline, including PDF text extraction, parsing content error correction, quality filtering, and synthetic instruction creation. Applying this strategy, we present a suite of LLMs: SciLitLLM, specialized in scientific literature understanding. These models demonstrate promising performance on scientific literature understanding benchmarks. Our contributions are threefold: (1) We present an effective framework that integrates CPT and SFT to adapt LLMs to scientific literature understanding, which can also be easily adapted to other domains. (2) We propose an LLM-based synthesis method to generate diverse and high-quality scientific instructions, resulting in a new instruction set -- SciLitIns -- for supervised fine-tuning in less-represented scientific domains. (3) SciLitLLM achieves promising performance improvements on scientific literature understanding benchmarks.

Figures (7)

• Figure 1: Average scores on SciRIFF of models with varying parameter sizes.
• Figure 2: An example of scientific literature understanding in SciRIFF. It involves extracting accurate entities from a biomedicine paper. SciLitLLM-7B demonstrates sufficient scientific knowledge and instruction-following ability to accurately identify and extract these entities.
• Figure 3: Comparison of strategies to adapt LLMs to scientific tasks.
• Figure 4: Examples of high and low-quality CPT text and SFT instructions.
• Figure 5: The pipeline of SciLitLLM consists of two key stages: continual pre-training (CPT) for scientific knowledge injection and supervised fine-tuning (SFT) for scientific instruction following. Specifically, the CPT stage involves several modules: PDF parsing, format & grammar correction (cf. Section \ref{['sec:format_correct']}), and quality filtering (cf. Section \ref{['sec:quality_filter']}) modules. These modules ensure the model is equipped with high-quality scientific domain knowledge. The SFT stage includes LLM-based instruction generation (cf. Section \ref{['sec:instruction_synthsis']}) and instruction quality control (cf. Section \ref{['sec:instruction-quality']}) measures. These steps are designed to fine-tune the model's ability to follow scientific instructions accurately and effectively.