Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

DocTer: Documentation Guided Fuzzing for Testing Deep Learning API Functions

Danning Xie, Yitong Li, Mijung Kim, Hung Viet Pham, Lin Tan, Xiangyu Zhang, Michael W. Godfrey

TL;DR

DocTer features a novel algorithm that automatically constructs rules to extract API parameter constraints from syntactic patterns in the form of dependency parse trees of API descriptions that are applied to a large volume of API documents in popular DL libraries to extract their input parameter constraints.

Abstract

Input constraints are useful for many software development tasks. For example, input constraints of a function enable the generation of valid inputs, i.e., inputs that follow these constraints, to test the function deeper. API functions of deep learning (DL) libraries have DL specific input constraints, which are described informally in the free form API documentation. Existing constraint extraction techniques are ineffective for extracting DL specific input constraints. To fill this gap, we design and implement a new technique, DocTer, to analyze API documentation to extract DL specific input constraints for DL API functions. DocTer features a novel algorithm that automatically constructs rules to extract API parameter constraints from syntactic patterns in the form of dependency parse trees of API descriptions. These rules are then applied to a large volume of API documents in popular DL libraries to extract their input parameter constraints. To demonstrate the effectiveness of the extracted constraints, DocTer uses the constraints to enable the automatic generation of valid and invalid inputs to test DL API functions. Our evaluation on three popular DL libraries (TensorFlow, PyTorch, and MXNet) shows that the precision of DocTer in extracting input constraints is 85.4%. DocTer detects 94 bugs from 174 API functions, including one previously unknown security vulnerability that is now documented in the CVE database, while a baseline technique without input constraints detects only 59 bugs. Most (63) of the 94 bugs are previously unknown, 54 of which have been fixed or confirmed by developers after we report them. In addition, DocTer detects 43 inconsistencies in documents, 39 of which are fixed or confirmed.

DocTer: Documentation Guided Fuzzing for Testing Deep Learning API Functions

TL;DR

DocTer features a novel algorithm that automatically constructs rules to extract API parameter constraints from syntactic patterns in the form of dependency parse trees of API descriptions that are applied to a large volume of API documents in popular DL libraries to extract their input parameter constraints.

Abstract

Input constraints are useful for many software development tasks. For example, input constraints of a function enable the generation of valid inputs, i.e., inputs that follow these constraints, to test the function deeper. API functions of deep learning (DL) libraries have DL specific input constraints, which are described informally in the free form API documentation. Existing constraint extraction techniques are ineffective for extracting DL specific input constraints. To fill this gap, we design and implement a new technique, DocTer, to analyze API documentation to extract DL specific input constraints for DL API functions. DocTer features a novel algorithm that automatically constructs rules to extract API parameter constraints from syntactic patterns in the form of dependency parse trees of API descriptions. These rules are then applied to a large volume of API documents in popular DL libraries to extract their input parameter constraints. To demonstrate the effectiveness of the extracted constraints, DocTer uses the constraints to enable the automatic generation of valid and invalid inputs to test DL API functions. Our evaluation on three popular DL libraries (TensorFlow, PyTorch, and MXNet) shows that the precision of DocTer in extracting input constraints is 85.4%. DocTer detects 94 bugs from 174 API functions, including one previously unknown security vulnerability that is now documented in the CVE database, while a baseline technique without input constraints detects only 59 bugs. Most (63) of the 94 bugs are previously unknown, 54 of which have been fixed or confirmed by developers after we report them. In addition, DocTer detects 43 inconsistencies in documents, 39 of which are fixed or confirmed.

Paper Structure

This paper contains 18 sections, 2 equations, 3 figures, 4 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Our Approach
  3. Contributions
  4. Approach
  5. Overview
  6. Preprocessing
  7. Rule Construction
  8. Constraint Extraction
  9. Testing Phase
  10. Experimental Setup
  11. Evaluation and Results
  12. RQ1: Effectiveness of Constraint Extraction
  13. RQ2: Comparison with Existing Approaches
  14. RQ3: Bug Detection Results
  15. RQ4: Valid-Input Generation Results
  16. ...and 3 more sections

Figures (3)

  • Figure 1: TensorFlow document helps our tool detect a bug that was fixed after we reported it to TensorFlow developers.
  • Figure 2: Overview of DocTer. $p_T$, $p_S$, and $p_D$ are the abstract constraints representing the data type, data structure, and number of dimensions of parameter $p$, respectively. Detailed constraint formats are explained in Section \ref{['sec:pattern_miner']}.
  • Figure 3: Ratio of passing inputs