Statistical Confidence in Functional Correctness: An Approach for AI Product Functional Correctness Evaluation

TL;DR

This paper tackles the challenge of evaluating AI systems under uncertainty by introducing Statistical Confidence in Functional Correctness (SCFC), a four-step methodology that ties business requirements to a statistical measure of performance. By defining specification limits, conducting stratified probabilistic sampling, using bootstrapping to obtain confidence intervals, and computing a capability index $C_{pk}$, SCFC moves evaluation from single-point metrics to statements about statistical confidence and deployment risk. The approach is validated through two real-world case studies (cargo deck space estimation and fraud detection) and complemented by semi-structured expert interviews, which indicate high perceived usefulness and intention to adopt. The work demonstrates how CI-based assessments can support deployment decisions and ongoing monitoring, offering a practical bridge between ISO/IEC 25059 guidance and industrial QA needs in AI systems.

Abstract

The quality assessment of Artificial Intelligence (AI) systems is a fundamental challenge due to their inherently probabilistic nature. Standards such as ISO/IEC 25059 provide a quality model, but they lack practical and statistically robust methods for assessing functional correctness. This paper proposes and evaluates the Statistical Confidence in Functional Correctness (SCFC) approach, which seeks to fill this gap by connecting business requirements to a measure of statistical confidence that considers both the model's average performance and its variability. The approach consists of four steps: defining quantitative specification limits, performing stratified and probabilistic sampling, applying bootstrapping to estimate a confidence interval for the performance metric, and calculating a capability index as a final indicator. The approach was evaluated through a case study on two real-world AI systems in industry involving interviews with AI experts. Valuable insights were collected from the experts regarding the utility, ease of use, and intention to adopt the methodology in practical scenarios. We conclude that the proposed approach is a feasible and valuable way to operationalize the assessment of functional correctness, moving the evaluation from a point estimate to a statement of statistical confidence.

Figures (5)

• Figure 1: Steps of the SCFC approach.
• Figure 2: Bootstrapping procedure.
• Figure 3: STEGO image segmentation of a real oil platform deck, with purple pixels representing the deck floor.
• Figure 4: Interview process overview.
• Figure 5: Miro board semi-structured interview template.