Scalpel: Automotive Deep Learning Framework Testing via Assembling Model Components
Yinglong Zou, Juan Zhai, Chunrong Fang, An Guo, Jiawei Liu, Zhenyu Chen
TL;DR
Scalpel addresses the challenging QA problem of testing automotive DL frameworks by generating test input models at the model-component level (head, neck, backbone) to ensure multi-input/output, multi-modal data processing, and multi-level feature extraction. It maintains a repository of real-world components, uses heuristic-guided selection and mutation, and assembles models for differential testing against cloud-end versus vehicle-end frameworks, updating the repository with successful mutations. Empirically, Scalpel detects significantly more crashes and NaN/inconsistency bugs than state-of-the-art baselines in Apollo and Autoware, while achieving high model-generation efficiency and test coverage. The approach generalizes well across ROS-based Autoware and open-source communities, offering a practical, scalable path to improving the reliability of automotive DL systems.
Abstract
Deep learning (DL) plays a key role in autonomous driving systems. DL models support perception modules, equipped with tasks such as object detection and sensor fusion. These DL models enable vehicles to process multi-sensor inputs to understand complex surroundings. Deploying DL models in autonomous driving systems faces stringent challenges, including real-time processing, limited computational resources, and strict power constraints. To address these challenges, automotive DL frameworks (e.g., PaddleInference) have emerged to optimize inference efficiency. However, these frameworks encounter unique quality issues due to their more complex deployment environments, such as crashes stemming from limited scheduled memory and incorrect memory allocation. Unfortunately, existing DL framework testing methods fail to detect these quality issues due to the failure in deploying generated test input models, as these models lack three essential capabilities: (1) multi-input/output tensor processing, (2) multi-modal data processing, and (3) multi-level data feature extraction. These capabilities necessitate specialized model components, which existing testing methods neglect during model generation. To bridge this gap, we propose Scalpel, an automotive DL frameworks testing method that generates test input models at the model component level. Scalpel generates models by assembling model components (heads, necks, backbones) to support capabilities required by autonomous driving systems. Specifically, Scalpel maintains and updates a repository of model components, generating test inputs by selecting, mutating, and assembling them. Successfully generated models are added back to enrich the repository. Newly generated models are then deployed within the autonomous driving system to test automotive DL frameworks via differential testing.