Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

OMPILOT: Harnessing Transformer Models for Auto Parallelization to Shared Memory Computing Paradigms

Arijit Bhattacharjee, Ali TehraniJamsaz, Le Chen, Niranjan Hasabnis, Mihai Capota, Nesreen Ahmed, Ali Jannesari

TL;DR

OMPilot targets automatic translation of C++ to OpenMP by introducing a domain-specific encoder–decoder transformer and a suite of pretraining tasks, including Masked Language Modeling, Syntax Structure Annotation, and a Weighted Denoising Auto Encoding loss. It couples these with Back Translation and Progressive Fine-Tuning to improve robustness, and introduces OMPBLEU, a composite metric that captures OpenMP-specific correctness beyond traditional NLP-code metrics. Empirical results show OMPilot outperforms larger baselines in clause generation accuracy and semantic correctness, while delivering up to 28× faster inference by removing natural-language prompts. The work demonstrates strong practical impact for high-performance computing by enabling scalable, verifiable OpenMP code generation and providing a specialized evaluation framework aligned with parallel correctness and compilation viability.

Abstract

Recent advances in large language models (LLMs) have significantly accelerated progress in code translation, enabling more accurate and efficient transformation across programming languages. While originally developed for natural language processing, LLMs have shown strong capabilities in modeling programming language syntax and semantics, outperforming traditional rule-based systems in both accuracy and flexibility. These models have streamlined cross-language conversion, reduced development overhead, and accelerated legacy code migration. In this paper, we introduce OMPILOT, a novel domain-specific encoder-decoder transformer tailored for translating C++ code into OpenMP, enabling effective shared-memory parallelization. OMPILOT leverages custom pre-training objectives that incorporate the semantics of parallel constructs and combines both unsupervised and supervised learning strategies to improve code translation robustness. Unlike previous work that focused primarily on loop-level transformations, OMPILOT operates at the function level to capture a wider semantic context. To evaluate our approach, we propose OMPBLEU, a novel composite metric specifically crafted to assess the correctness and quality of OpenMP parallel constructs, addressing limitations in conventional translation metrics.

OMPILOT: Harnessing Transformer Models for Auto Parallelization to Shared Memory Computing Paradigms

TL;DR

OMPilot targets automatic translation of C++ to OpenMP by introducing a domain-specific encoder–decoder transformer and a suite of pretraining tasks, including Masked Language Modeling, Syntax Structure Annotation, and a Weighted Denoising Auto Encoding loss. It couples these with Back Translation and Progressive Fine-Tuning to improve robustness, and introduces OMPBLEU, a composite metric that captures OpenMP-specific correctness beyond traditional NLP-code metrics. Empirical results show OMPilot outperforms larger baselines in clause generation accuracy and semantic correctness, while delivering up to 28× faster inference by removing natural-language prompts. The work demonstrates strong practical impact for high-performance computing by enabling scalable, verifiable OpenMP code generation and providing a specialized evaluation framework aligned with parallel correctness and compilation viability.

Abstract

Recent advances in large language models (LLMs) have significantly accelerated progress in code translation, enabling more accurate and efficient transformation across programming languages. While originally developed for natural language processing, LLMs have shown strong capabilities in modeling programming language syntax and semantics, outperforming traditional rule-based systems in both accuracy and flexibility. These models have streamlined cross-language conversion, reduced development overhead, and accelerated legacy code migration. In this paper, we introduce OMPILOT, a novel domain-specific encoder-decoder transformer tailored for translating C++ code into OpenMP, enabling effective shared-memory parallelization. OMPILOT leverages custom pre-training objectives that incorporate the semantics of parallel constructs and combines both unsupervised and supervised learning strategies to improve code translation robustness. Unlike previous work that focused primarily on loop-level transformations, OMPILOT operates at the function level to capture a wider semantic context. To evaluate our approach, we propose OMPBLEU, a novel composite metric specifically crafted to assess the correctness and quality of OpenMP parallel constructs, addressing limitations in conventional translation metrics.

Paper Structure

This paper contains 45 sections, 17 equations, 11 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Background and Related Works
  3. Rule-based Auto Parallelization
  4. LLMs for Code
  5. LLMs for HPC
  6. OMPilot
  7. Masked Language Modeling
  8. Syntax Structure Annotation
  9. Denoising Auto Encoding with Weighted Token Cross-Entropy Loss Function
  10. Training
  11. Back Translation
  12. Progressive Fine Tuning
  13. Inference
  14. OMPBLEU
  15. Weighted Clause Importance Score (WC)
  16. ...and 30 more sections

Figures (11)

  • Figure 1: Comparison between BLEU/CodeBLEU and OMPBLEU on the ground‑truth and the generated OpenMP parallel code. Even though the generated code omits the reduction(+:sum) schedule(static) clauses, both BLEU/CodeBLEU scores are high, 90.82/98.78. OMPBLEU score for the pair is 57.49.
  • Figure 2: Birds eye view of OMPilot.
  • Figure 3: Masked Language Modeling
  • Figure 4: Syntax Structure Annotation
  • Figure 5: Denoising Auto-Encoding
  • ...and 6 more figures