Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

An Evaluation of Large Language Models in Bioinformatics Research

Hengchuang Yin, Zhonghui Gu, Fanhao Wang, Yiparemu Abuduhaibaier, Yanqiao Zhu, Xinming Tu, Xian-Sheng Hua, Xiao Luo, Yizhou Sun

TL;DR

This work systematically evaluates GPT-family large language models on six core bioinformatics tasks by converting DNA, peptide, and molecular data into text prompts. It finds that, with careful prompting and model selection, LLMs achieve competitive performance on several tasks (e.g., CDS identification, AMP/ACP detection, molecular optimization) but face notable limitations in complex reasoning tasks like gene/protein NER and certain educational problems. Through extensive comparisons against domain-specific models and traditional baselines, the study highlights both the potential and the reliability challenges of applying LLMs to bioinformatics. The findings offer practical guidance for prompt engineering and model choice in AI for science, and point to future directions in functionally rich biomolecule design, scientific knowledge mining, and education-oriented AI tools.

Abstract

Large language models (LLMs) such as ChatGPT have gained considerable interest across diverse research communities. Their notable ability for text completion and generation has inaugurated a novel paradigm for language-interfaced problem solving. However, the potential and efficacy of these models in bioinformatics remain incompletely explored. In this work, we study the performance LLMs on a wide spectrum of crucial bioinformatics tasks. These tasks include the identification of potential coding regions, extraction of named entities for genes and proteins, detection of antimicrobial and anti-cancer peptides, molecular optimization, and resolution of educational bioinformatics problems. Our findings indicate that, given appropriate prompts, LLMs like GPT variants can successfully handle most of these tasks. In addition, we provide a thorough analysis of their limitations in the context of complicated bioinformatics tasks. In conclusion, we believe that this work can provide new perspectives and motivate future research in the field of LLMs applications, AI for Science and bioinformatics.

An Evaluation of Large Language Models in Bioinformatics Research

TL;DR

This work systematically evaluates GPT-family large language models on six core bioinformatics tasks by converting DNA, peptide, and molecular data into text prompts. It finds that, with careful prompting and model selection, LLMs achieve competitive performance on several tasks (e.g., CDS identification, AMP/ACP detection, molecular optimization) but face notable limitations in complex reasoning tasks like gene/protein NER and certain educational problems. Through extensive comparisons against domain-specific models and traditional baselines, the study highlights both the potential and the reliability challenges of applying LLMs to bioinformatics. The findings offer practical guidance for prompt engineering and model choice in AI for science, and point to future directions in functionally rich biomolecule design, scientific knowledge mining, and education-oriented AI tools.

Abstract

Large language models (LLMs) such as ChatGPT have gained considerable interest across diverse research communities. Their notable ability for text completion and generation has inaugurated a novel paradigm for language-interfaced problem solving. However, the potential and efficacy of these models in bioinformatics remain incompletely explored. In this work, we study the performance LLMs on a wide spectrum of crucial bioinformatics tasks. These tasks include the identification of potential coding regions, extraction of named entities for genes and proteins, detection of antimicrobial and anti-cancer peptides, molecular optimization, and resolution of educational bioinformatics problems. Our findings indicate that, given appropriate prompts, LLMs like GPT variants can successfully handle most of these tasks. In addition, we provide a thorough analysis of their limitations in the context of complicated bioinformatics tasks. In conclusion, we believe that this work can provide new perspectives and motivate future research in the field of LLMs applications, AI for Science and bioinformatics.
Paper Structure (21 sections, 7 figures, 7 tables)

This paper contains 21 sections, 7 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Large Language Models (LLMs)
  4. Language Models for Bioinformatics
  5. Evaluated Tasks
  6. Identifying Potential Coding Regions from DNA Sequences
  7. Identifying Antimicrobial Peptide
  8. Identifying Anti-cancer Peptide
  9. Molecule Optimization
  10. Gene and Protein Named Entities Extraction
  11. Evaluation on Educational Bioinformatics Problem-Solving Set
  12. How our tasks are beneficial to bioinformatics research.
  13. Results and Discussions
  14. Performance on Identifying Potential Coding Regions
  15. Performance on Identifying Antimicrobial Peptide
  16. ...and 6 more sections

Figures (7)

  • Figure 1: Our works aim to validate how LLMs can benefit bioinformatics research.
  • Figure 2: Illustration of the identification of coding regions utilizing GPT-4. (The dialogue on the right depicts a comparison using chain of thought.)
  • Figure 3: Comparison of LLMs for identifying potential coding regions from DNA sequences.
  • Figure 4: Comparison of distributions for different metrics.
  • Figure 5: Illustration of gene and protein NER using GPT-3.5 (gpt-3.5-turbo-0613) and GPT-4.
  • ...and 2 more figures