Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

FeynTune: Large Language Models for High-Energy Theory

Paul Richmond, Prarit Agarwal, Borun Chowdhury, Vasilis Niarchos, Constantinos Papageorgakis

TL;DR

Twenty fine-tuned variants of the 8 billion parameter Llama-3.1 model for theoretical high-energy physics are presented, obtained as 20 fine-tuned variants of the Llama-3.1 model using two distinct low-rank adaptation fine-tuning approaches and varying dataset sizes.

Abstract

We present specialized Large Language Models for theoretical High-Energy Physics, obtained as 20 fine-tuned variants of the 8-billion parameter Llama-3.1 model. Each variant was trained on arXiv abstracts (through August 2024) from different combinations of hep-th, hep-ph and gr-qc. For a comparative study, we also trained models on datasets that contained abstracts from disparate fields such as the q-bio and cs categories. All models were fine-tuned using two distinct Low-Rank Adaptation fine-tuning approaches and varying dataset sizes, and outperformed the base model on hep-th abstract completion tasks. We compare performance against leading commercial LLMs (ChatGPT, Claude, Gemini, DeepSeek) and derive insights for further developing specialized language models for High-Energy Theoretical Physics.

FeynTune: Large Language Models for High-Energy Theory

TL;DR

Twenty fine-tuned variants of the 8 billion parameter Llama-3.1 model for theoretical high-energy physics are presented, obtained as 20 fine-tuned variants of the Llama-3.1 model using two distinct low-rank adaptation fine-tuning approaches and varying dataset sizes.

Abstract

We present specialized Large Language Models for theoretical High-Energy Physics, obtained as 20 fine-tuned variants of the 8-billion parameter Llama-3.1 model. Each variant was trained on arXiv abstracts (through August 2024) from different combinations of hep-th, hep-ph and gr-qc. For a comparative study, we also trained models on datasets that contained abstracts from disparate fields such as the q-bio and cs categories. All models were fine-tuned using two distinct Low-Rank Adaptation fine-tuning approaches and varying dataset sizes, and outperformed the base model on hep-th abstract completion tasks. We compare performance against leading commercial LLMs (ChatGPT, Claude, Gemini, DeepSeek) and derive insights for further developing specialized language models for High-Energy Theoretical Physics.

Paper Structure

This paper contains 21 sections, 4 equations, 4 figures, 10 tables.

Table of Contents

  1. Introduction
  2. Datasets and Training
  3. Evaluation Metrics
  4. Perplexity
  5. Semantic Similarity
  6. Human evaluation
  7. Statistical Analysis:
  8. Inter-Rater Reliability:
  9. Base vs fine-tuned models:
  10. Use of technical language:
  11. Comparison against commercial LLMs:
  12. Creative completions:
  13. s10 models:
  14. Discussion of results and future directions
  15. Data Curation
  16. ...and 6 more sections

Figures (4)

  • Figure 3.1: Learning curves comparing the s1-LoRA-all model (left) showcasing a step-wise decrease in loss with the s1-LoRA-qkv model (right) showing a standard loss function behavior.
  • Figure 3.2: Smoothed mean batch perplexity evaluation curves on the s1 validation dataset, comparing the s1-LoRA-all model (left) and the s1-LoRA-QKV model (right).
  • Figure 3.3: Bootstrapping the perplexities of trained models on the s1 test dataset: standard grouping (left) and paired comparison of LoRA-all vs LoRA-QKV models (right). Note that the perplexity for the base model is without fine-tuning (LoRA-all or LoRA-QKV); we have included these values to facilitate the comparison to the baseline.
  • Figure 3.4: Exponentiated Shannon entropy of completion against the length of said completion for (i) the ground truth, (ii) the base Llama model and (iii) the fine-tuned s1 model.