Just TestIt! An SBST Approach To Automate System-Integration Testing

TL;DR

TestIt introduces SBST-powered, open-source full-system integration testing for open-source RTL designs, supporting both simulation and FPGA execution to accelerate hardware-software validation and CI/CD adoption. By dynamically generating randomized inputs and golden references, TestIt automates end-to-end testing from HAL drivers to applications, producing compilable C artifacts that directly verify the System-Under-Test. A case study on the X-HEEP MCU with a custom SPC accelerator demonstrates that hardware-focused FP-GAs-based testing can reveal faults missed by formal methods and yields an 11-fold speedup over RTL simulations. Overall, TestIt offers a practical, extensible framework to streamline system-level verification and foster agile, continuous integration practices in hardware design.

Abstract

This paper introduces TestIt, an open-source Python package designed to automate full-system integration testing using a Software-Based Self-Test (SBST) approach. By dynamically generating test vectors and golden references, TestIt significantly reduces development time and complexity while supporting both simulation and FPGA environments. Its flexible design positions TestIt as a key enabler for the widespread adoption of CI/CD methodologies in open-source RTL development. A case study on the X-HEEP RISC-V microcontroller (MCU), which integrates a custom accelerator, showcases TestIt's ability to detect hardware and software faults that traditional formal methods may overlook. Furthermore, the case study highlights how TestIt can be leveraged to characterize system performance with minimal effort. By automating testing on the PYNQ-Z2 FPGA development board, we achieved a 11x speed-up with respect to RTL simulations.

Figures (3)

• Figure 1: Structure of a TestIt environment
• Figure 2: Test duration in seconds
• Figure 3: Integration test results with the DMA HAL fault