Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Effective Random Test Generation for Deep Learning Compilers

Luyao Ren, ZiHeng Wang, Yingfei Xiong, Li Zhang, Guoyue Jiang, Tao Xie

TL;DR

This paper tackles the challenge of testing deep learning compilers by generating semantically valid computation-graph inputs. It introduces Isra, a domain-specific, instantiation-based constraint solver that operates in a backtrack-free, sound, and complete manner by enforcing graph-level directional consistency and operation-level global consistency. Isra demonstrably outperforms state-of-the-art baselines on coverage metrics and detects 33 unique bugs (including 24 previously unknown) across TVM, Glow, and SophGo, with many fixes reported by developers. The approach provides practical value for robust compiler testing and offers a scalable framework for generating diverse, valid inputs for complex semantic specifications.

Abstract

Deep learning compilers help address the difficulties of deploying deep learning models on diverse types of hardware. Testing deep learning compilers is highly crucial, because they are impacting countless AI applications that use them for model optimization and deployment. To test deep learning compilers, random testing, the testing method popularly used for compiler testing practices, faces the challenge of generating semantically valid test inputs, i.e., deep learning models that satisfy the semantic model specifications (in short as semantic specifications). To tackle this challenge, in this paper, we propose a novel approach named Isra, including a domain-specific constraint solver that resolves the constraints from the semantic specifications without backtracking. We implement and apply our approach to three popular real-world deep learning compilers including TVM, Glow, and a commercial compiler named SophGo. The evaluation results show that Isra is more effective than the state-of-the-art approaches and the baseline approaches on constructing valid test inputs for compiler-bug detection, and Isra successfully finds 24 previously unknown bugs in released versions of the three compilers. These results indicate Isra's effectiveness and practical value.

Effective Random Test Generation for Deep Learning Compilers

TL;DR

This paper tackles the challenge of testing deep learning compilers by generating semantically valid computation-graph inputs. It introduces Isra, a domain-specific, instantiation-based constraint solver that operates in a backtrack-free, sound, and complete manner by enforcing graph-level directional consistency and operation-level global consistency. Isra demonstrably outperforms state-of-the-art baselines on coverage metrics and detects 33 unique bugs (including 24 previously unknown) across TVM, Glow, and SophGo, with many fixes reported by developers. The approach provides practical value for robust compiler testing and offers a scalable framework for generating diverse, valid inputs for complex semantic specifications.

Abstract

Deep learning compilers help address the difficulties of deploying deep learning models on diverse types of hardware. Testing deep learning compilers is highly crucial, because they are impacting countless AI applications that use them for model optimization and deployment. To test deep learning compilers, random testing, the testing method popularly used for compiler testing practices, faces the challenge of generating semantically valid test inputs, i.e., deep learning models that satisfy the semantic model specifications (in short as semantic specifications). To tackle this challenge, in this paper, we propose a novel approach named Isra, including a domain-specific constraint solver that resolves the constraints from the semantic specifications without backtracking. We implement and apply our approach to three popular real-world deep learning compilers including TVM, Glow, and a commercial compiler named SophGo. The evaluation results show that Isra is more effective than the state-of-the-art approaches and the baseline approaches on constructing valid test inputs for compiler-bug detection, and Isra successfully finds 24 previously unknown bugs in released versions of the three compilers. These results indicate Isra's effectiveness and practical value.
Paper Structure (45 sections, 3 equations, 8 figures, 4 tables)

This paper contains 45 sections, 3 equations, 8 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Background and Overview
  3. Computational Graph for Deep Learning
  4. Challenges
  5. Instantiation-based Constraint Solving
  6. Isra in a Nutshell
  7. Graph-level constraint resolving
  8. Operation-level constraint resolving
  9. Properties of Isra
  10. Detailed Approach Description
  11. Notations and Definitions
  12. Concepts in the Computation Graph
  13. Constraint Satisfaction Problem
  14. Local Consistency and Global Consistency
  15. Graph-level constraint resolving
  16. ...and 30 more sections

Figures (8)

  • Figure 1: The Pipeline of Testing Deep Learning Compilers
  • Figure 2: An Example of a Computation Graph.
  • Figure 3: A running example of Isra
  • Figure 3: # unique bugs detected on different compilers.
  • Figure 4: Distribution of Operation Frequency of Isra , DeclGen, and Randoop-Gen (captioning only parts of operations in x-axis for a clear presentation, numbers at y-axis are normalized with the total number).
  • ...and 3 more figures