Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

NukeBERT: A Pre-trained language model for Low Resource Nuclear Domain

Ayush Jain, N. M. Meenachi, B. Venkatraman

TL;DR

The Nuclear Bidirectional Encoder Representational Transformers (NukeBERT) is proposed, which incorporates a novel technique for building BERT vocabulary to make it suitable for tasks with less training data, and validating the adopted methodology.

Abstract

Significant advances have been made in recent years on Natural Language Processing with machines surpassing human performance in many tasks, including but not limited to Question Answering. The majority of deep learning methods for Question Answering targets domains with large datasets and highly matured literature. The area of Nuclear and Atomic energy has largely remained unexplored in exploiting non-annotated data for driving industry viable applications. Due to lack of dataset, a new dataset was created from the 7000 research papers on nuclear domain. This paper contributes to research in understanding nuclear domain knowledge which is then evaluated on Nuclear Question Answering Dataset (NQuAD) created by nuclear domain experts as part of this research. NQuAD contains 612 questions developed on 181 paragraphs randomly selected from the IGCAR research paper corpus. In this paper, the Nuclear Bidirectional Encoder Representational Transformers (NukeBERT) is proposed, which incorporates a novel technique for building BERT vocabulary to make it suitable for tasks with less training data. The experiments evaluated on NQuAD revealed that NukeBERT was able to outperform BERT significantly, thus validating the adopted methodology. Training NukeBERT is computationally expensive and hence we will be open-sourcing the NukeBERT pretrained weights and NQuAD for fostering further research work in the nuclear domain.

NukeBERT: A Pre-trained language model for Low Resource Nuclear Domain

TL;DR

The Nuclear Bidirectional Encoder Representational Transformers (NukeBERT) is proposed, which incorporates a novel technique for building BERT vocabulary to make it suitable for tasks with less training data, and validating the adopted methodology.

Abstract

Significant advances have been made in recent years on Natural Language Processing with machines surpassing human performance in many tasks, including but not limited to Question Answering. The majority of deep learning methods for Question Answering targets domains with large datasets and highly matured literature. The area of Nuclear and Atomic energy has largely remained unexplored in exploiting non-annotated data for driving industry viable applications. Due to lack of dataset, a new dataset was created from the 7000 research papers on nuclear domain. This paper contributes to research in understanding nuclear domain knowledge which is then evaluated on Nuclear Question Answering Dataset (NQuAD) created by nuclear domain experts as part of this research. NQuAD contains 612 questions developed on 181 paragraphs randomly selected from the IGCAR research paper corpus. In this paper, the Nuclear Bidirectional Encoder Representational Transformers (NukeBERT) is proposed, which incorporates a novel technique for building BERT vocabulary to make it suitable for tasks with less training data. The experiments evaluated on NQuAD revealed that NukeBERT was able to outperform BERT significantly, thus validating the adopted methodology. Training NukeBERT is computationally expensive and hence we will be open-sourcing the NukeBERT pretrained weights and NQuAD for fostering further research work in the nuclear domain.

Paper Structure

This paper contains 12 sections, 4 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. NText Preparation
  5. PREPROCESSING
  6. NukeVOCAB
  7. NQUAD: Nuclear Question Answering Dataset
  8. Experiment
  9. BERT PRETRAINING
  10. BERT Fine-Tuning
  11. Results
  12. Conclusion

Figures (4)

  • Figure 1: NukeBERT Pipeline: Using a vocabulary having nuclear jargons, the preprocessed data is pre-trained on a corpus of nuclear text, which is then further finetuned on SQuAD and NQuAD.
  • Figure 2: Flowchart of steps for preparation of preprocessed data from raw corpus of 7000 research papers.
  • Figure 3: Flowchart for preparation of NukeVocab. Notice we only concatenate “Bad Tokenization” with BERT Vocabulary.
  • Figure 4: Results of BERT and NukeBERT on NQuAD test-set