Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LLM4VV: Exploring LLM-as-a-Judge for Validation and Verification Testsuites

Zachariah Sollenberger, Jay Patel, Christian Munley, Aaron Jarmusch, Sunita Chandrasekaran

TL;DR

This paper probes into the black box of one such LLM that has generated the best compiler tests for the directive-based programming models OpenMP and OpenACC, and adopts an agent-based and a pipeline-based approach to develop a more reliable method for automatically validating LLM-generated compiler tests.

Abstract

Large Language Models (LLM) are evolving and have significantly revolutionized the landscape of software development. If used well, they can significantly accelerate the software development cycle. At the same time, the community is very cautious of the models being trained on biased or sensitive data, which can lead to biased outputs along with the inadvertent release of confidential information. Additionally, the carbon footprints and the un-explainability of these black box models continue to raise questions about the usability of LLMs. With the abundance of opportunities LLMs have to offer, this paper explores the idea of judging tests used to evaluate compiler implementations of directive-based programming models as well as probe into the black box of LLMs. Based on our results, utilizing an agent-based prompting approach and setting up a validation pipeline structure drastically increased the quality of DeepSeek Coder, the LLM chosen for the evaluation purposes.

LLM4VV: Exploring LLM-as-a-Judge for Validation and Verification Testsuites

TL;DR

This paper probes into the black box of one such LLM that has generated the best compiler tests for the directive-based programming models OpenMP and OpenACC, and adopts an agent-based and a pipeline-based approach to develop a more reliable method for automatically validating LLM-generated compiler tests.

Abstract

Large Language Models (LLM) are evolving and have significantly revolutionized the landscape of software development. If used well, they can significantly accelerate the software development cycle. At the same time, the community is very cautious of the models being trained on biased or sensitive data, which can lead to biased outputs along with the inadvertent release of confidential information. Additionally, the carbon footprints and the un-explainability of these black box models continue to raise questions about the usability of LLMs. With the abundance of opportunities LLMs have to offer, this paper explores the idea of judging tests used to evaluate compiler implementations of directive-based programming models as well as probe into the black box of LLMs. Based on our results, utilizing an agent-based prompting approach and setting up a validation pipeline structure drastically increased the quality of DeepSeek Coder, the LLM chosen for the evaluation purposes.
Paper Structure (12 sections, 6 figures, 9 tables)

This paper contains 12 sections, 6 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Negative Probing
  5. Agent-based Approach for LLM-as-a-Judge (LLMJ)
  6. Validation Pipeline utilizing LLMJ
  7. Experimental Setup
  8. Defining Metrics
  9. Analysis of deepseek-coder-33B-instruct as an LLMJ
  10. Results for Part One
  11. Results for Part Two
  12. Conclusion

Figures (6)

  • Figure 1: An Overview of Agent-based Approach for LLMJ
  • Figure 2: An Overview of the Validation Pipeline
  • Figure 3: A Radar Plot for Validation Pipeline Results for OpenACC
  • Figure 4: A Radar Plot for Validation Pipeline Results for OpenMP
  • Figure 5: A Radar Plot for LLMJ Results for OpenACC
  • ...and 1 more figures