Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Complexity of adaptive testing in scenarios defined extensionally

Ismael Rodriguez, David Rubio, Fernando Rubio

TL;DR

The paper studies adaptive testing when the IUT is specified extentionally as a finite set of possible definitions with a corresponding correct subset, and analyzes four problem variants arising from determinism and singleton versus multi-element incorrect/correct sets. It formalizes the model with collections $C$, inputs $I$, outputs $O$, and specification $E$, and defines four decision/optimization problems ($D$-ATDP, $D$-ATOP, $N$-ATDP, $N$-ATOP) whose complexities range from NP-complete (deterministic singleton cases) to PSPACE-complete (non-deterministic cases). It also establishes Log-APX hardness/completeness for optimization variants in the deterministic setting and provides PSPACE-hardness proofs via TQBF reductions for the non-deterministic variants, confirming that adaptive testing can be computationally intractable even in memoryless models. The results guide expectations for exact methods and motivate heuristic approaches (minimax and genetic algorithms) in practice. Overall, the work highlights the intrinsic hardness of adaptive testing under extensional definitions and connects it to classic complexity classes through explicit reductions.

Abstract

In this paper we consider a testing setting where the set of possible definitions of the Implementation Under Test (IUT), as well as the behavior of each of these definitions in all possible interactions, are extensionally defined, i.e., on an element-by-element and case-by-case basis. Under this setting, the problem of finding the minimum testing strategy such that collected observations will necessarily let us decide whether the IUT is correct or not (i.e., whether it necessarily belongs to the set of possible correct definitions or not) is studied in four possible problem variants: with or without non-determinism; and with or without more than one possible definition in the sets of possible correct and incorrect definitions. The computational complexity of these variants is studied, and properties such as PSPACE-completeness and Log-APX-hardness are identified.

Complexity of adaptive testing in scenarios defined extensionally

TL;DR

The paper studies adaptive testing when the IUT is specified extentionally as a finite set of possible definitions with a corresponding correct subset, and analyzes four problem variants arising from determinism and singleton versus multi-element incorrect/correct sets. It formalizes the model with collections , inputs , outputs , and specification , and defines four decision/optimization problems (-ATDP, -ATOP, -ATDP, -ATOP) whose complexities range from NP-complete (deterministic singleton cases) to PSPACE-complete (non-deterministic cases). It also establishes Log-APX hardness/completeness for optimization variants in the deterministic setting and provides PSPACE-hardness proofs via TQBF reductions for the non-deterministic variants, confirming that adaptive testing can be computationally intractable even in memoryless models. The results guide expectations for exact methods and motivate heuristic approaches (minimax and genetic algorithms) in practice. Overall, the work highlights the intrinsic hardness of adaptive testing under extensional definitions and connects it to classic complexity classes through explicit reductions.

Abstract

In this paper we consider a testing setting where the set of possible definitions of the Implementation Under Test (IUT), as well as the behavior of each of these definitions in all possible interactions, are extensionally defined, i.e., on an element-by-element and case-by-case basis. Under this setting, the problem of finding the minimum testing strategy such that collected observations will necessarily let us decide whether the IUT is correct or not (i.e., whether it necessarily belongs to the set of possible correct definitions or not) is studied in four possible problem variants: with or without non-determinism; and with or without more than one possible definition in the sets of possible correct and incorrect definitions. The computational complexity of these variants is studied, and properties such as PSPACE-completeness and Log-APX-hardness are identified.
Paper Structure (6 sections, 7 equations, 2 figures)

This paper contains 6 sections, 7 equations, 2 figures.

Table of Contents

  1. Introduction
  2. Motivational example
  3. Formal model and complexity results
  4. Conclusions and future work
  5. B.2.1 $\texttt{TQBF} \Rightarrow \texttt{N-ATDP}_1$
  6. B.2.2 $\texttt{N-ATDP}_1 \Rightarrow \texttt{TQBF}$

Figures (2)

  • Figure 1: The possible definitions of the IUT. Definitions $f_1$-$f_3$ are considered correct, whereas $f_4$-$f_{13}$ are considered incorrect.
  • Figure 2: Adaptive testing strategy applying at most two test cases in all situations.