The Second International Verification of Neural Networks Competition (VNN-COMP 2021): Summary and Results

TL;DR

VNN-COMP 2021 provides a structured, fair comparison of neural network verification tools using standardized ONNX/VNNLIB formats and aerosolized AWS hardware. The paper details a comprehensive rule set, overhead correction, and benchmark scoring to assess scalability and speed across 12 teams. It documents diverse verification approaches—from MILP-based to bound-propagation and abstract-interpretation methods—and highlights α,β-CROWN as the leading tool, with VeriNet and others close behind. The study demonstrates meaningful progress since 2020, discusses scoring caveats, and outlines concrete recommendations for future competitions and broader adoption in safety-critical ML verification. The work offers an openly accessible benchmark suite and results pipeline to drive ongoing improvements in NN verification methodologies.

Abstract

This report summarizes the second International Verification of Neural Networks Competition (VNN-COMP 2021), held as a part of the 4th Workshop on Formal Methods for ML-Enabled Autonomous Systems that was collocated with the 33rd International Conference on Computer-Aided Verification (CAV). Twelve teams participated in this competition. The goal of the competition is to provide an objective comparison of the state-of-the-art methods in neural network verification, in terms of scalability and speed. Along this line, we used standard formats (ONNX for neural networks and VNNLIB for specifications), standard hardware (all tools are run by the organizers on AWS), and tool parameters provided by the tool authors. This report summarizes the rules, benchmarks, participating tools, results, and lessons learned from this competition.

Figures (1)

• Figure 1: Timeline for VNN-COMP2021.