Tools at the Frontiers of Quantitative Verification

TL;DR

QComp 2023 surveys state-of-the-art tooling for advanced quantitative verification, covering ten categories that extend beyond basic reachability and rewards to problems like LTL, parametric models, POMDPs, and stochastic games. It combines descriptive overviews of methods with first systematic performance benchmarks across tools ranging from prototypes to established toolchains, and provides data artifacts to enable reproducibility. The compilation identifies core capabilities, gaps, and practical implications, guiding tool developers and domain researchers toward more scalable and robust verification approaches. By highlighting data-driven benchmarks and community-standard formats (e.g., Jani), the report aims to accelerate progress toward mature, widely-adopted tooling for the frontier problems in quantitative verification. The work underscores both the maturation of several frontiers and the remaining open questions in uncertainty, rare events, and partial observability that shape future Tool Olympics-style evaluations.

Abstract

The analysis of formal models that include quantitative aspects such as timing or probabilistic choices is performed by quantitative verification tools. Broad and mature tool support is available for computing basic properties such as expected rewards on basic models such as Markov chains. Previous editions of QComp, the comparison of tools for the analysis of quantitative formal models, focused on this setting. Many application scenarios, however, require more advanced property types such as LTL and parameter synthesis queries as well as advanced models like stochastic games and partially observable MDPs. For these, tool support is in its infancy today. This paper presents the outcomes of QComp 2023: a survey of the state of the art in quantitative verification tool support for advanced property types and models. With tools ranging from first research prototypes to well-supported integrations into established toolsets, this report highlights today's active areas and tomorrow's challenges in tool-focused research for quantitative verification.

Figures (11)

• Figure 1: Performance comparison results of tools for multi-objective verification
• Figure 2: Prism's over- and under-approximations for grid and computation times
• Figure 3: Storm's over-approximations for refuel and computation times
• Figure 4: Storm's under-approximations for refuel and computation times
• Figure 5: Max. prob. achieved and time to compute the FSC for grid-avoid with Paynt