OMPGPT: A Generative Pre-trained Transformer Model for OpenMP

TL;DR

This paper presents OMPGPT, a novel domain-specific model meticulously designed to harness the inherent strengths of language models for OpenMP pragma generation, and leverages prompt engineering techniques from the NLP domain to create Chain-of-OMP, an innovative strategy designed to enhance OMPGPT's effectiveness.

Abstract

Large language models (LLMs)such as ChatGPT have significantly advanced the field of Natural Language Processing (NLP). This trend led to the development of code-based large language models such as StarCoder, WizardCoder, and CodeLlama, which are trained extensively on vast repositories of code and programming languages. While the generic abilities of these code LLMs are useful for many programmers in tasks like code generation, the area of high-performance computing (HPC) has a narrower set of requirements that make a smaller and more domain-specific model a smarter choice. This paper presents OMPGPT, a novel domain-specific model meticulously designed to harness the inherent strengths of language models for OpenMP pragma generation. Furthermore, we leverage prompt engineering techniques from the NLP domain to create Chain-of-OMP, an innovative strategy designed to enhance OMPGPT's effectiveness. Our extensive evaluations demonstrate that OMPGPT outperforms existing large language models specialized in OpenMP tasks and maintains a notably smaller size, aligning it more closely with the typical hardware constraints of HPC environments. We consider our contribution as a pivotal bridge, connecting the advantage of language models with the specific demands of HPC tasks.

Figures (5)

• Figure 1: (black): Traditional LLMs require extensive NL and PL training data for generating OpenMP pragmas, leading to an increased complexity and larger model size. Users usually carefully craft prompts and interpret outputs to obtain accurate OpenMP pragmas. (blue): OMPGPT is tailored for HPC code, with nearly half of its training data being OpenMP code. It aligns OpenMP pragmas with their scope during training to match GPT model instincts. OMPGPT not only addresses the limitations of current LLMs but also benefits from the Chain-of-OMP prompt engineering technique.
• Figure 2: Distribution of OpenMP Pragmas in the HPCorpus OpenMP subset with the top 15 most frequently occurring pragmas.
• Figure 3: Basic prompt for OMPGPT. (a): prompt for code generation. (b): prompt for OpenMP pragma generation.
• Figure 4: Chain-of-OMP. Start phase: using the standard OpenMP pragma generation prompts for OMPGPT. Directive/clause generation phase: the first generated Directive/clause is augmented to the previous input as the input for the next chain. This phase is updated till it number of chains reaches parameter $N_{\text{chain}}$. The final Control variable generation phase generates the last component to complete the chain-of-OMP and generate the OpenMP pragma.
• Figure 5: Pragma Generation Accuracy. Blue: OMPGPT. Red: GPT-Neo.