Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

OODTE: A Differential Testing Engine for the ONNX Optimizer

Nikolaos Louloudakis, Ajitha Rajan

TL;DR

This paper introduces OODTE, a differential testing engine for the ONNX Optimizer, designed to automatically detect and localize faults arising from optimization passes. It operates by compiling a set of ONNX models, running both original and optimized versions over a validation dataset, and comparing outputs with per-task metrics, enabling per-pass fault localization when discrepancies arise. In a case study of 130 models from the ONNX Model Hub, OODTE found that 9.2% of model instances crashed or produced invalid optimized models and that 26.1% exhibited output differences, with 14 of 15 identified issues previously unknown. The approach is generalizable to other AI compilers and provides a practical framework for validating optimizers in CI pipelines.

Abstract

With over 760 stars on GitHub and being part of the official ONNX repository, the ONNX Optimizer is the default tool for applying graph-based optimizations to ONNX models. Despite its widespread use, its ability to maintain model accuracy during optimization has not been thoroughly investigated. In this work, we present OODTE, a utility designed to automatically and comprehensively evaluate the correctness of the ONNX Optimizer. OODTE adopts a straightforward yet powerful differential testing and evaluation methodology, which can be readily adapted for use with other compiler optimizers. Specifically, OODTE takes a collection of ONNX models, applies optimizations, and executes both the original and optimized versions across a user-defined input set, automatically capturing any issues encountered during optimization. When discrepancies in accuracy arise, OODTE iteratively isolates the responsible optimization pass by repeating the process at a finer granularity. We applied OODTE to 130 well-known models from the official ONNX Model Hub, spanning diverse tasks including classification, object detection, semantic segmentation, text summarization, question answering, and sentiment analysis. Our evaluation revealed that 9.2% of the model instances either caused the optimizer to crash or led to the generation of invalid models using default optimization strategies. Additionally, 30% of classification models and 16.6% of object detection and segmentation models exhibited differing outputs across original and optimized versions, whereas models focused on text-related tasks were generally robust to optimization. OODTE uncovered 15 issues-14 previously unknown-affecting 9 of 47 optimization passes and the optimizer overall. All issues were reported to the ONNX Optimizer team. OODTE offers a simple but effective framework for validating AI model optimizers, applicable beyond the ONNX ecosystem.

OODTE: A Differential Testing Engine for the ONNX Optimizer

TL;DR

This paper introduces OODTE, a differential testing engine for the ONNX Optimizer, designed to automatically detect and localize faults arising from optimization passes. It operates by compiling a set of ONNX models, running both original and optimized versions over a validation dataset, and comparing outputs with per-task metrics, enabling per-pass fault localization when discrepancies arise. In a case study of 130 models from the ONNX Model Hub, OODTE found that 9.2% of model instances crashed or produced invalid optimized models and that 26.1% exhibited output differences, with 14 of 15 identified issues previously unknown. The approach is generalizable to other AI compilers and provides a practical framework for validating optimizers in CI pipelines.

Abstract

With over 760 stars on GitHub and being part of the official ONNX repository, the ONNX Optimizer is the default tool for applying graph-based optimizations to ONNX models. Despite its widespread use, its ability to maintain model accuracy during optimization has not been thoroughly investigated. In this work, we present OODTE, a utility designed to automatically and comprehensively evaluate the correctness of the ONNX Optimizer. OODTE adopts a straightforward yet powerful differential testing and evaluation methodology, which can be readily adapted for use with other compiler optimizers. Specifically, OODTE takes a collection of ONNX models, applies optimizations, and executes both the original and optimized versions across a user-defined input set, automatically capturing any issues encountered during optimization. When discrepancies in accuracy arise, OODTE iteratively isolates the responsible optimization pass by repeating the process at a finer granularity. We applied OODTE to 130 well-known models from the official ONNX Model Hub, spanning diverse tasks including classification, object detection, semantic segmentation, text summarization, question answering, and sentiment analysis. Our evaluation revealed that 9.2% of the model instances either caused the optimizer to crash or led to the generation of invalid models using default optimization strategies. Additionally, 30% of classification models and 16.6% of object detection and segmentation models exhibited differing outputs across original and optimized versions, whereas models focused on text-related tasks were generally robust to optimization. OODTE uncovered 15 issues-14 previously unknown-affecting 9 of 47 optimization passes and the optimizer overall. All issues were reported to the ONNX Optimizer team. OODTE offers a simple but effective framework for validating AI model optimizers, applicable beyond the ONNX ecosystem.
Paper Structure (31 sections, 2 figures, 4 tables)

This paper contains 31 sections, 2 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Problem Being Solved
  3. Case Studies
  4. Background
  5. Deep Learning Models
  6. Image Classification
  7. Object Detection and Semantic Segmentation
  8. Text Comprehension and Generation
  9. Graph Optimizations
  10. The ONNX Optimizer
  11. Model Output Comparison Methods
  12. System Architecture & Methodology
  13. Model Orchestrator Module
  14. Optimizer Module
  15. Runner Module
  16. ...and 16 more sections

Figures (2)

  • Figure 1: Architecture of OODTE with the following primary modules: (1) Model Orchestrator, (2) Optimizer Module, (3) Runner Module, and (4) Metrics Comparison Module.
  • Figure 2: Incorrect model graph transformation caused by the fuse_bn_to_conv pass in EfficientNet-Lite-4. The nodes affected are highlighted. It should produce a Conv node with updated weights and biases, but the pass generates multiple nodes that represent the BatchNormalization formula.