A Comprehensive Survey of Scientific Large Language Models and Their Applications in Scientific Discovery

TL;DR

This survey systematically maps the landscape of scientific large language models across multiple fields and modalities, revealing three core pre-training strategies (MLM encoder, encoder–decoder next-token with instruction tuning, and contrastive cross-modal mapping) and then unifies cross-field and cross-modal connections through a coherent taxonomy. It catalogs hundreds of models, datasets, architectures, and evaluation tasks, and it highlights how LLMs are deployed to accelerate scientific discovery—from hypothesis generation and problem solving to experimental design and literature-based reasoning. The work identifies common patterns, key datasets, and representative successes across general science, mathematics, physics, chemistry, biology, medicine, and geoscience, while also exposing gaps such as out-of-distribution robustness and trustworthiness. Overall, the article provides a structured foundation for building, comparing, and deploying domain-specific LLMs in science, with practical guidance and a public resource for practitioners and researchers.

Abstract

In many scientific fields, large language models (LLMs) have revolutionized the way text and other modalities of data (e.g., molecules and proteins) are handled, achieving superior performance in various applications and augmenting the scientific discovery process. Nevertheless, previous surveys on scientific LLMs often concentrate on one or two fields or a single modality. In this paper, we aim to provide a more holistic view of the research landscape by unveiling cross-field and cross-modal connections between scientific LLMs regarding their architectures and pre-training techniques. To this end, we comprehensively survey over 260 scientific LLMs, discuss their commonalities and differences, as well as summarize pre-training datasets and evaluation tasks for each field and modality. Moreover, we investigate how LLMs have been deployed to benefit scientific discovery. Resources related to this survey are available at https://github.com/yuzhimanhua/Awesome-Scientific-Language-Models.

Figures (1)

• Figure 1: Three major types of scientific LLM pre-training techniques. (Column 1): Pre-training encoder LLMs with sequentialized scientific data (e.g., text, academic graphs, molecules, biological sequences) via masked language modeling. (Column 2): Pre-training (encoder-)decoder LLMs with sequentialized scientific data (e.g., text, tables, crystals, images) via next token prediction (possibly with instruction tuning). (Column 3): Mapping text and relevant sequences/graphs/images closer in the latent space via contrastive learning.