ChatGPT vs SBST: A Comparative Assessment of Unit Test Suite Generation
Yutian Tang, Zhijie Liu, Zhichao Zhou, Xiapu Luo
TL;DR
The paper compares ChatGPT-generated unit tests with EvoSuite-generated tests (SBST) for Java, evaluating correctness, readability, code coverage, and bug-detection ability. It uses DynaMOSA-based Java benchmarks and Defects4J to assess multiple facets of test quality, including static analysis and coverage metrics. Findings show EvoSuite generally achieves higher code coverage and bug detection, while ChatGPT yields compilable, readable tests and can uncover unique test scenarios, suggesting potential benefits from hybrid approaches. The work highlights current capabilities and limitations of LLM-driven test generation and motivates integrating LLMs with traditional SBST to improve software testing effectiveness.
Abstract
Recent advancements in large language models (LLMs) have demonstrated exceptional success in a wide range of general domain tasks, such as question answering and following instructions. Moreover, LLMs have shown potential in various software engineering applications. In this study, we present a systematic comparison of test suites generated by the ChatGPT LLM and the state-of-the-art SBST tool EvoSuite. Our comparison is based on several critical factors, including correctness, readability, code coverage, and bug detection capability. By highlighting the strengths and weaknesses of LLMs (specifically ChatGPT) in generating unit test cases compared to EvoSuite, this work provides valuable insights into the performance of LLMs in solving software engineering problems. Overall, our findings underscore the potential of LLMs in software engineering and pave the way for further research in this area.