Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Fine-Tuning BERT for Domain-Specific Question Answering: Toward Educational NLP Resources at University Scale

Aurélie Montfrond

TL;DR

The paper addresses extracting domain-specific QA for university course information by fine-tuning BERT variants on a 1,203-question SQuAD-formatted dataset from the University of Limerick. It systematically compares general, biomedical, and scientific pretraining variants, finding that larger, SQuAD-pretrained models achieve the best Exact Match and F1 scores, though the small dataset risks overfitting. The study demonstrates feasibility of domain-adapted educational QA and argues for a large-scale university dataset (~100k entries) to enable a first domain-specific QA resource for universities. This work lays the groundwork for scalable, autonomous educational knowledge systems tailored to institutional content.

Abstract

Prior work on scientific question answering has largely emphasized chatbot-style systems, with limited exploration of fine-tuning foundation models for domain-specific reasoning. In this study, we developed a chatbot for the University of Limerick's Department of Electronic and Computer Engineering to provide course information to students. A custom dataset of 1,203 question-answer pairs in SQuAD format was constructed using the university book of modules, supplemented with manually and synthetically generated entries. We fine-tuned BERT (Devlin et al., 2019) using PyTorch and evaluated performance with Exact Match and F1 scores. Results show that even modest fine-tuning improves hypothesis framing and knowledge extraction, demonstrating the feasibility of adapting foundation models to educational domains. While domain-specific BERT variants such as BioBERT and SciBERT exist for biomedical and scientific literature, no foundation model has yet been tailored to university course materials. Our work addresses this gap by showing that fine-tuning BERT with academic QA pairs yields effective results, highlighting the potential to scale towards the first domain-specific QA model for universities and enabling autonomous educational knowledge systems.

Fine-Tuning BERT for Domain-Specific Question Answering: Toward Educational NLP Resources at University Scale

TL;DR

The paper addresses extracting domain-specific QA for university course information by fine-tuning BERT variants on a 1,203-question SQuAD-formatted dataset from the University of Limerick. It systematically compares general, biomedical, and scientific pretraining variants, finding that larger, SQuAD-pretrained models achieve the best Exact Match and F1 scores, though the small dataset risks overfitting. The study demonstrates feasibility of domain-adapted educational QA and argues for a large-scale university dataset (~100k entries) to enable a first domain-specific QA resource for universities. This work lays the groundwork for scalable, autonomous educational knowledge systems tailored to institutional content.

Abstract

Prior work on scientific question answering has largely emphasized chatbot-style systems, with limited exploration of fine-tuning foundation models for domain-specific reasoning. In this study, we developed a chatbot for the University of Limerick's Department of Electronic and Computer Engineering to provide course information to students. A custom dataset of 1,203 question-answer pairs in SQuAD format was constructed using the university book of modules, supplemented with manually and synthetically generated entries. We fine-tuned BERT (Devlin et al., 2019) using PyTorch and evaluated performance with Exact Match and F1 scores. Results show that even modest fine-tuning improves hypothesis framing and knowledge extraction, demonstrating the feasibility of adapting foundation models to educational domains. While domain-specific BERT variants such as BioBERT and SciBERT exist for biomedical and scientific literature, no foundation model has yet been tailored to university course materials. Our work addresses this gap by showing that fine-tuning BERT with academic QA pairs yields effective results, highlighting the potential to scale towards the first domain-specific QA model for universities and enabling autonomous educational knowledge systems.

Paper Structure

This paper contains 13 sections, 2 figures.

Table of Contents

  1. Introduction
  2. Related work
  3. Methods
  4. Dataset preparation
  5. Model selection
  6. Preprocessing and tokenization
  7. Fine-Tuning Setup
  8. Evaluation
  9. Results
  10. Training results
  11. Performance results
  12. Discussion
  13. CONCLUSION

Figures (2)

  • Figure 1: Exact Match and F‑1 scores by extractive question answering models.
  • Figure 2: Chart comparing all of the extractive question answering models by Exact Match and F-1 scores. Abbreviations: BERT-L=BERT large uncased WWM SQuAD; BioBERT-L=BioBERT large v1.1 SQuAD; BioBERT-B=BioBERT base v1.1 SQuAD; RoBERTa-B=RoBERTa base SQuAD 2.0; XLM-R=XLM-RoBERTa base SQuAD 2.0; SciBERT-C=SciBERT cased; SciBERT-U=SciBERT uncased; Distil-U=DistilBERT base uncased; Distil-C=DistilBERT base cased; Distil-CS=DistilBERT cased distilled SQuAD.