Small Test Suites for Active Automata Learning
Loes Kruger, Sebastian Junges, Jurriaan Rot
TL;DR
This work tackles the EQ bottleneck in active automata learning by introducing smaller, assumption-driven test suites that remain complete under natural structural constraints. It presents three subalphabet-based experts—Active Inputs, Future, and Components—that generate restricted infix sets and can be embedded into ASI-style conformance testing. A Mixture of Experts using EXP3 multi-armed bandits adaptively selects among these test suites to accelerate counterexample finding during EQs. Empirical results on industrial and standard benchmarks show notable gains for large-scale SULs, while gains are modest for small to medium models, demonstrating scalable conformance testing in practice. The approach thus enables learning of industrial-scale systems previously out of reach by reducing test-suite size and employing data-driven, adaptive suite selection.
Abstract
A bottleneck in modern active automata learning is to test whether a hypothesized Mealy machine correctly describes the system under learning. The search space for possible counterexamples is given by so-called test suites, consisting of input sequences that have to be checked to decide whether a counterexample exists. This paper shows that significantly smaller test suites suffice under reasonable assumptions on the structure of the black box. These smaller test suites help to refute false hypotheses during active automata learning, even when the assumptions do not hold. We combine multiple test suites using a multi-armed bandit setup that adaptively selects a test suite. An extensive empirical evaluation shows the efficacy of our approach. For small to medium-sized models, the performance gain is limited. However, the approach allows learning models from large, industrial case studies that were beyond the reach of known methods.