Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Quokka: An Open-source Large Language Model ChatBot for Material Science

Xianjun Yang, Stephen D. Wilson, Linda Petzold

TL;DR

Quokka addresses the need for open, domain-tailored large language capabilities in materials science by continuing pretraining of LLaMA-2 on a vast materials-specific corpus from S2ORC and applying instruction tuning with domain-relevant prompts. The approach yields four open checkpoints (Quokka-7B/13B and their Chat variants) and demonstrates decreasing pretraining and instruction-tuning losses, with 13B offering lower perplexity and robust zero-shot performance. Case studies show accurate, context-aware responses, safe handling of sensitive prompts, and effective summarization, underscoring practical utility for researchers, educators, and students. By releasing these resources, the work enables downstream materials-science NLP applications and paves the way for future multimodal extensions.

Abstract

This paper presents the development of a specialized chatbot for materials science, leveraging the Llama-2 language model, and continuing pre-training on the expansive research articles in the materials science domain from the S2ORC dataset. The methodology involves an initial pretraining phase on over one million domain-specific papers, followed by an instruction-tuning process to refine the chatbot's capabilities. The chatbot is designed to assist researchers, educators, and students by providing instant, context-aware responses to queries in the field of materials science. We make the four trained checkpoints (7B, 13B, with or without chat ability) freely available to the research community at https://github.com/Xianjun-Yang/Quokka.

Quokka: An Open-source Large Language Model ChatBot for Material Science

TL;DR

Quokka addresses the need for open, domain-tailored large language capabilities in materials science by continuing pretraining of LLaMA-2 on a vast materials-specific corpus from S2ORC and applying instruction tuning with domain-relevant prompts. The approach yields four open checkpoints (Quokka-7B/13B and their Chat variants) and demonstrates decreasing pretraining and instruction-tuning losses, with 13B offering lower perplexity and robust zero-shot performance. Case studies show accurate, context-aware responses, safe handling of sensitive prompts, and effective summarization, underscoring practical utility for researchers, educators, and students. By releasing these resources, the work enables downstream materials-science NLP applications and paves the way for future multimodal extensions.

Abstract

This paper presents the development of a specialized chatbot for materials science, leveraging the Llama-2 language model, and continuing pre-training on the expansive research articles in the materials science domain from the S2ORC dataset. The methodology involves an initial pretraining phase on over one million domain-specific papers, followed by an instruction-tuning process to refine the chatbot's capabilities. The chatbot is designed to assist researchers, educators, and students by providing instant, context-aware responses to queries in the field of materials science. We make the four trained checkpoints (7B, 13B, with or without chat ability) freely available to the research community at https://github.com/Xianjun-Yang/Quokka.
Paper Structure (8 sections, 3 figures)

This paper contains 8 sections, 3 figures.

Table of Contents

  1. Introduction
  2. Related Work
  3. Instruction tuning for LLMs
  4. Materials Science MLP Tasks
  5. Experiment
  6. Results
  7. Case Study
  8. Conclusion and Future Work

Figures (3)

  • Figure 1: Quokka Training Pipeline: We first perform pretraining on over 1 million materials science articles, then conduct instruction tuning on both LLaMa-2-7B and 13 models.
  • Figure 2: The continued pre-training loss on 7B and 13B foundation model. Each step represents 100 iterations. The final perplexity score (PPL) is calculated on the held-out validation set.
  • Figure 3: The instruction-tuning loss on Quokka-7B and Quokka-13B foundation model.