LLM4VV: Developing LLM-Driven Testsuite for Compiler Validation

TL;DR

This work investigates using large language models to automatically generate a validation and verification testsuite for OpenACC compiler implementations. It benchmarks multiple LLMs (including Codellama, Deepseek-Coder, Phind variants, and GPT-3.5/4-Turbo) across prompting strategies (expressive prompts, one-shot, and Retrieval-Augmented Generation) and a dedicated fine-tuning dataset, organized into a three-stage generation and evaluation pipeline. Key findings show that Deepseek-Coder-33b-Instruct and GPT-4-Turbo provide the strongest performance in different stages, with expressive prompts and RAG generally improving test quality, though many failures arise from test quality and LLM hallucinations requiring human review. The results demonstrate a viable path to accelerating HPC compiler validation while underscoring the need for careful oversight, continuous spec updates, and potential extension to other HPC paradigms such as OpenMP in future work.

Abstract

Large language models (LLMs) are a new and powerful tool for a wide span of applications involving natural language and demonstrate impressive code generation abilities. The goal of this work is to automatically generate tests and use these tests to validate and verify compiler implementations of a directive-based parallel programming paradigm, OpenACC. To do so, in this paper, we explore the capabilities of state-of-the-art LLMs, including open-source LLMs -- Meta Codellama, Phind fine-tuned version of Codellama, Deepseek Deepseek Coder and closed-source LLMs -- OpenAI GPT-3.5-Turbo and GPT-4-Turbo. We further fine-tuned the open-source LLMs and GPT-3.5-Turbo using our own testsuite dataset along with using the OpenACC specification. We also explored these LLMs using various prompt engineering techniques that include code template, template with retrieval-augmented generation (RAG), one-shot example, one-shot with RAG, expressive prompt with code template and RAG. This paper highlights our findings from over 5000 tests generated via all the above mentioned methods. Our contributions include: (a) exploring the capabilities of the latest and relevant LLMs for code generation, (b) investigating fine-tuning and prompt methods, and (c) analyzing the outcome of LLMs generated tests including manually analysis of representative set of tests. We found the LLM Deepseek-Coder-33b-Instruct produced the most passing tests followed by GPT-4-Turbo.

Figures (6)

• Figure 1: Example prompt and generated test, with no augmentation through context retrieval, one-shot example, or a code template. This test is a false pass, because it does not test the correct feature i.e. acc delete routine, rather the exit data directive.
• Figure 2: Expressive prompt with RAG and Template. This prompt includes detailed instructions on the task, and provides context from the OpenACC specification with a code template mentioned below the context.
• Figure 3: Format of LLM fine-tuning datasets that shows a set of prompts and desired responses
• Figure 4: Generated test with Expressive + Template + RAG prompt. This is a passing test, targeting the correct directive, and using the desired format for the testsuite infrastructure and is generated based on context from the specification.
• Figure 5: Failed test for serial in Fortran generated using Deepseek-Coder-33b-Instruct. This test results in a compiler error because the serial construct requires a closing directive when used on a loop.