A Novel Interactive-Guided Differential Testing Approach for FPGA Simulation Debugger Tools
Shikai Guo, Xiaoyu Wang, Xiaochen Li, Zhihao Xu, He Jiang
TL;DR
DB-Hunter targets the challenging problem of debugging FPGA chip debuggers by combining RTL design transformations, debug action transformations, and interactive differential testing to reveal inconsistencies in Vivado's debugger traces. It generates equivalent RTL variants and transforms debugging actions to exhaustively exercise the debugger, then compares outputs to detect bugs. Across experiments on Vivado 2022.1/2023.1, it identified 18 previously unknown bugs, with 10 confirmed by Xilinx and 6 fixed, demonstrating practical impact for FPGA toolchain reliability. The approach advances debugger testing for Verilog/Vivado and offers a reproducible framework with datasets and code for further research.
Abstract
Field-Programmable Gate Array (FPGA) development tool chains are widely used in FPGA design, simulation, and verification in critical areas like communications, automotive electronics, and aerospace. Commercial FPGA tool chains such as Xilinx' Vivado aids developers in swiftly identifying and rectifying bugs and issues in FPGA designs through a robust built-in debugger, ensuring the correctness and development efficiency of the FPGA design. Hardening such FPGA chip debugger tools by testing is crucial since engineers might misinterpret code and introduce incorrect fixes, leading to security risks. However, FPGA chip debugger tools are challenging to test as they require assessing both RTL designs and a series of debugging actions, including setting breakpoints and stepping through the code. To address this issue, we propose a interactive differential testing approach called DB-Hunter to detect bugs in Vivado's FPGA chip debugger tools. Specifically, DB-Hunter consists of three components: RTL design transformation component, debug action transformation component, and interactive differential testing component. By performing RTL design and debug action transformations, DB-Hunter generates diverse and complex RTL designs and debug actions, to thoroughly test the Vivado debugger using interactive differential testing to detect bugs. In three months, DB-Hunter reported 18 issues, including 10 confirmed as bugs by Xilinx Support, 6 bugs had been fixed in last version.