Enhancing Differential Testing With LLMs For Testing Deep Learning Libraries

TL;DR

DL is prone to functional bugs in libraries like PyTorch and TensorFlow, and differential testing relies on finding counterparts and generating diverse inputs, which existing approaches struggle to do. DLLens introduces an LLM-enhanced framework with two main components: (1) LLM-based counterpart synthesis that constructs cross-library counterparts by composing APIs from another DL library, and (2) static analysis that extracts path constraints to steer diverse test input generation. The evaluation on TensorFlow and PyTorch shows that DLLens yields 1.84× more API counterparts than state-of-the-art, covers 7.23% more execution branches, and detects 1.88× more bugs on 200 APIs, totaling 71 detected bugs with 59 developer-confirmed (including 46 previously unknown, 10 fixed in latest versions). This demonstrates the practical effectiveness of combining LLM-driven knowledge with static-path analysis to improve DL library testing.

Abstract

Differential testing offers a promising strategy to alleviate the test oracle problem by comparing the test results between alternative implementations. However, existing differential testing techniques for deep learning (DL) libraries are limited by the key challenges of finding alternative implementations (called counterparts) for a given API and subsequently generating diverse test inputs. To address the two challenges, this paper introduces DLLens, an LLM-enhanced differential testing technique for DL libraries. To address the first challenge, DLLens incorporates an LLM-based counterpart synthesis workflow, with the insight that the counterpart of a given DL library API's computation could be successfully synthesized through certain composition and adaptation of the APIs from another DL library. To address the second challenge, DLLens incorporates a static analysis technique that extracts the path constraints from the implementations of a given API and its counterpart to guide diverse test input generation. The extraction is facilitated by LLM's knowledge of the concerned DL library and its upstream libraries. We evaluate DLLens on two popular DL libraries, TensorFlow and PyTorch. Our evaluation shows that DLLens synthesizes counterparts for 1.84 times as many APIs as those found by state-of-the-art techniques on these libraries. Moreover, under the same time budget, DLLens covers 7.23% more branches and detects 1.88 times as many bugs as state-of-the-art techniques on 200 randomly sampled APIs. DLLens has successfully detected 71 bugs in recent TensorFlow and PyTorch libraries. Among them, 59 are confirmed by developers, including 46 confirmed as previously unknown bugs, and 10 of these previously unknown bugs have been fixed in the latest version of TensorFlow and PyTorch.