Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Harnessing the Power of Large Language Models for Software Testing Education: A Focus on ISTQB Syllabus

Tuan-Phong Ngo, Bao-Ngoc Duong, Tuan-Anh Hoang, Joshua Dwight, Ushik Shrestha Khwakhali

TL;DR

The paper tackles the challenge of integrating large language models into ISTQB-aligned software testing education. It introduces a comprehensive open-source benchmark of 28 ISTQB exams (1,145 questions) and a domain-optimized prompt design, then systematically evaluates state-of-the-art LLMs across multiple prompts and models. Key findings show that advanced, domain-specific prompts and reasoning-capable models substantially improve ISTQB question accuracy and explanation quality, with the o3 model achieving high Q-P@1 and, in some cases, perfect pass rates, while GPT-3.5 baseline performance remains limited. The work demonstrates the potential of LLMs as virtual tutors and instructional aids for educators and students, providing actionable guidance for curriculum design and future research on scalable AI-assisted software testing education.

Abstract

Software testing is a critical component in the software engineering field and is important for software engineering education. Thus, it is vital for academia to continuously improve and update educational methods to reflect the current state of the field. The International Software Testing Qualifications Board (ISTQB) certification framework is globally recognized and widely adopted in industry and academia. However, ISTQB-based learning has been rarely applied with recent generative artificial intelligence advances. Despite the growing capabilities of large language models (LLMs), ISTQB-based learning and instruction with LLMs have not been thoroughly explored. This paper explores and evaluates how LLMs can complement the ISTQB framework for higher education. The findings present four key contributions: (i) the creation of a comprehensive ISTQB-aligned dataset spanning over a decade, consisting of 28 sample exams and 1,145 questions; (ii) the development of a domain-optimized prompt that enhances LLM precision and explanation quality on ISTQB tasks; (iii) a systematic evaluation of state-of-the-art LLMs on this dataset; and (iv) actionable insights and recommendations for integrating LLMs into software testing education. These findings highlight the promise of LLMs in supporting ISTQB certification preparation and offer a foundation for their broader use in software engineering at higher education.

Harnessing the Power of Large Language Models for Software Testing Education: A Focus on ISTQB Syllabus

TL;DR

The paper tackles the challenge of integrating large language models into ISTQB-aligned software testing education. It introduces a comprehensive open-source benchmark of 28 ISTQB exams (1,145 questions) and a domain-optimized prompt design, then systematically evaluates state-of-the-art LLMs across multiple prompts and models. Key findings show that advanced, domain-specific prompts and reasoning-capable models substantially improve ISTQB question accuracy and explanation quality, with the o3 model achieving high Q-P@1 and, in some cases, perfect pass rates, while GPT-3.5 baseline performance remains limited. The work demonstrates the potential of LLMs as virtual tutors and instructional aids for educators and students, providing actionable guidance for curriculum design and future research on scalable AI-assisted software testing education.

Abstract

Software testing is a critical component in the software engineering field and is important for software engineering education. Thus, it is vital for academia to continuously improve and update educational methods to reflect the current state of the field. The International Software Testing Qualifications Board (ISTQB) certification framework is globally recognized and widely adopted in industry and academia. However, ISTQB-based learning has been rarely applied with recent generative artificial intelligence advances. Despite the growing capabilities of large language models (LLMs), ISTQB-based learning and instruction with LLMs have not been thoroughly explored. This paper explores and evaluates how LLMs can complement the ISTQB framework for higher education. The findings present four key contributions: (i) the creation of a comprehensive ISTQB-aligned dataset spanning over a decade, consisting of 28 sample exams and 1,145 questions; (ii) the development of a domain-optimized prompt that enhances LLM precision and explanation quality on ISTQB tasks; (iii) a systematic evaluation of state-of-the-art LLMs on this dataset; and (iv) actionable insights and recommendations for integrating LLMs into software testing education. These findings highlight the promise of LLMs in supporting ISTQB certification preparation and offer a foundation for their broader use in software engineering at higher education.
Paper Structure (22 sections, 3 figures, 3 tables)

This paper contains 22 sections, 3 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Background
  3. The ISTQB Certification Framework
  4. Integrating the ISTQB Syllabus into Undergraduate Curriculum
  5. The Emergence of Large Language Models in Software Testing Education
  6. Gaps in Literature
  7. Study Design
  8. Research Questions
  9. Open-source ISTQB Benchmark
  10. Prompt Engineering Design
  11. LLM Models
  12. Evaluation Metrics
  13. Evaluation Framework
  14. Results
  15. RQ1: Evaluating LLMs in ISTQB Exams
  16. ...and 7 more sections

Figures (3)

  • Figure 1: Selected LLM models: reasoning and non-reasoning with gpt-3.5-turbo as equivalent to an undergraduate baseline.
  • Figure 2: Evaluation framework: Generating random exams from the ISTQB dataset to assess LLM performance.
  • Figure 3: Q-P@1 across cognitive levels for core-level exams using the baseline model and basic prompt. '---' means the corresponding level is not applicable.