An Empirical Analysis of Fine-Tuning Large Language Models on Bioinformatics Literature: PRSGPT and BioStarsGPT

TL;DR

The paper proposes a reproducible nine-step pipeline for domain-specific fine-tuning of large language models on bioinformatics data and demonstrates it with two use cases: PRSGPT, focused on polygenic risk score tools, and BioStarsGPT, trained on community forum discussions. It combines automated QA generation, NLI-based quality control, semantic deduplication, and LoRA-based parameter-efficient fine-tuning across multiple models, evaluated with an extensive set of lexical, semantic, and entailment metrics. Qwen2.5-7B emerges as the strongest performer, achieving substantial gains on both tasks, and the authors release large open datasets containing tens of thousands of QA pairs. Human evaluation shows competitive factual accuracy and richer methodological detail for PRSGPT, while BioStarsGPT demonstrates solid conceptual understanding with room for technical refinement. The work provides a scalable, privacy-preserving framework for deploying domain-tailored bioinformatics assistants and outlines practical challenges and mitigation strategies for real-world use.

Abstract

Large language models (LLMs) often lack specialized knowledge for complex bioinformatics applications. We present a reproducible pipeline for fine-tuning LLMs on specialized bioinformatics data, demonstrated through two use cases: PRSGPT, focused on polygenic risk score (PRS) tools, and BioStarsGPT, trained on community forum discussions. The nine-step pipeline integrates diverse data sources, structured preprocessing, prompt-based question-answer (QA) generation (via Google Gemini), natural language inference (NLI) for quality control, semantic deduplication, clustering-based data splitting, and parameter-efficient fine-tuning using LoRA. We fine-tuned three LLMs (LLaMA-3.2-3B, Qwen2.5-7B, Gemma) and benchmarked them on over 14 lexical and semantic metrics. Qwen2.5-7B emerged as the best performer, with BLEU-4 and ROUGE-1 improvements of 82\% and 70\% for PRSGPT and 6\% and 18\% for BioStarsGPT, respectively. The open-source datasets produced include over 28,000 QA pairs for PRSGPT and 154,282 for BioStarsGPT. Human evaluation of PRSGPT yielded 61.9\% accuracy on the PRS tools comparison task, comparable to Google Gemini (61.4\%), but with richer methodological detail and accurate citations. BioStarsGPT demonstrated 59\% conceptual accuracy across 142 curated bioinformatics questions. Our pipeline enables scalable, domain-specific fine-tuning of LLMs. It enables privacy-preserving, locally deployable bioinformatics assistants, explores their practical applications, and addresses the challenges, limitations, and mitigation strategies associated with their development and use.

Figures (3)

• Figure 1: Overview of the proposed supervised fine-tuning workflow for bioinformatics applications. The pipeline consists of data discovery, curation, preprocessing, QA generation, quality assessment, model training, and evaluation, supporting both tool documentation and forum-based data.
• Figure 2: Overview of the pipeline steps for data acquisition and QA generation for Bioinformatics tools.
• Figure 3: This t-SNE visualization clusters 192,382 BioStars answers into 15 distinct topical groups, revealing the semantic organization of bioinformatics discussions on the platform. The largest cluster (Cluster 0, 46,736 items) focuses on general data/user themes. In contrast, others correspond to specialized workflows such as RNA sequencing, genomic coordinates (BED), variant calling (VCF), and sequence alignment (BAM/SAMtools). The spatial separation of clusters reflects forum diversity.