Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LLM-aided explanations of EDA synthesis errors

Siyu Qiu, Benjamin Tan, Hammond Pearce

TL;DR

This work investigates if Large Language Models (LLMs), which have demonstrated text comprehension and question-answering capabilities, can be used to generate novice-friendly explanations of compile-time synthesis error messages from Quartus Prime and Vivado.

Abstract

Training new engineers in digital design is a challenge, particularly when it comes to teaching the complex electronic design automation (EDA) tooling used in this domain. Learners will typically deploy designs in the Verilog and VHDL hardware description languages to Field Programmable Gate Arrays (FPGAs) from Altera (Intel) and Xilinx (AMD) via proprietary closed-source toolchains (Quartus Prime and Vivado, respectively). These tools are complex and difficult to use -- yet, as they are the tools used in industry, they are an essential first step in this space. In this work, we examine how recent advances in artificial intelligence may be leveraged to address aspects of this challenge. Specifically, we investigate if Large Language Models (LLMs), which have demonstrated text comprehension and question-answering capabilities, can be used to generate novice-friendly explanations of compile-time synthesis error messages from Quartus Prime and Vivado. To perform this study we generate 936 error message explanations using three OpenAI LLMs over 21 different buggy code samples. These are then graded for relevance and correctness, and we find that in approximately 71% of cases the LLMs give correct & complete explanations suitable for novice learners.

LLM-aided explanations of EDA synthesis errors

TL;DR

This work investigates if Large Language Models (LLMs), which have demonstrated text comprehension and question-answering capabilities, can be used to generate novice-friendly explanations of compile-time synthesis error messages from Quartus Prime and Vivado.

Abstract

Training new engineers in digital design is a challenge, particularly when it comes to teaching the complex electronic design automation (EDA) tooling used in this domain. Learners will typically deploy designs in the Verilog and VHDL hardware description languages to Field Programmable Gate Arrays (FPGAs) from Altera (Intel) and Xilinx (AMD) via proprietary closed-source toolchains (Quartus Prime and Vivado, respectively). These tools are complex and difficult to use -- yet, as they are the tools used in industry, they are an essential first step in this space. In this work, we examine how recent advances in artificial intelligence may be leveraged to address aspects of this challenge. Specifically, we investigate if Large Language Models (LLMs), which have demonstrated text comprehension and question-answering capabilities, can be used to generate novice-friendly explanations of compile-time synthesis error messages from Quartus Prime and Vivado. To perform this study we generate 936 error message explanations using three OpenAI LLMs over 21 different buggy code samples. These are then graded for relevance and correctness, and we find that in approximately 71% of cases the LLMs give correct & complete explanations suitable for novice learners.
Paper Structure (14 sections, 6 figures, 2 tables)

This paper contains 14 sections, 6 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Background and Related Work
  3. Large Language Models for hardware design
  4. LLM for training and education
  5. Digital Design Assistance via LLM prompts
  6. Overview
  7. Defining a corpus of bugs
  8. Harvesting Vivado and Quartus error log files
  9. Prompting LLMs for error explanations
  10. Manual grading with pedagogically focused metrics
  11. Results
  12. Top-line results:
  13. Discussion
  14. Conclusion

Figures (6)

  • Figure 1: Example unhelpful error message. It does not describe the real problem (a missing semicolon), and it links to line 46, not the fault on line 45!
  • Figure 2: Overall experimentation methodology
  • Figure 3: Example of 'good' and 'bad' error explanations for Bug 1 (Figure \ref{['fig:bad-error']}) generated by gpt-3.5-turbo. Each bug is presented with graded metrics.
  • Figure 4: LLM prompts
  • Figure 5: A 'good' error explanation for Bug 1 (Figure \ref{['fig:bad-error']}) generated by gpt-4. However, this explanation is flagged 'Solution is provided' as it contains a direct copy/paste solution on Lines 12-15---i.e., the explanation helps 'too much' according to constructivism in pedagogy.
  • ...and 1 more figures