Easy over Hard: A Simple Baseline for Test Failures Causes Prediction

TL;DR

The paper tackles automatic diagnosis of test failure causes from large, noisy logs. It introduces NCChecker, a simple, three-stage, heuristic-based framework that abstracts logs into events, builds a predictive lookup table, and predicts failure causes for new failures. On a dataset of over 10K industrial logs, NCChecker outperforms several baselines, handles imbalanced classes well, and remains memory-efficient and fast at inference. The work provides a practical baseline for failure analysis and demonstrates that carefully designed log-event representations with targeted heuristics can capture predictive signals beyond traditional retrieval-based methods.

Abstract

The test failure causes analysis is critical since it determines the subsequent way of handling different types of bugs, which is the prerequisite to get the bugs properly analyzed and fixed. After a test case fails, software testers have to inspect the test execution logs line by line to identify its root cause. However, manual root cause determination is often tedious and time-consuming, which can cost 30-40% of the time needed to fix a problem. Therefore, there is a need for automatically predicting the test failure causes to lighten the burden of software testers. In this paper, we present a simple but hard-to-beat approach, named NCChecker to automatically identify the failure causes for failed test logs. Our approach can help developers efficiently identify the test failure causes, and flag the most probable log lines of indicating the root causes for investigation. Our approach has three main stages: log abstraction, lookup table construction, and failure causes prediction. We first perform log abstraction to parse the unstructured log messages into structured log events. NCChecker then automatically maintains and updates a lookup table via employing our heuristic rules, which record the matching score between different log events and test failure causes. When it comes to the failure cause prediction stage, for a newly generated failed test log, NCChecker can easily infer its failed reason by checking out the associated log events' scores from the lookup table. We have developed a prototype and evaluated our tool on a real-world industrial dataset with more than 10K test logs. The extensive experiments show the promising performance of our model over a set of benchmarks. Moreover, our approach is highly efficient and memory-saving, and can successfully handle the data imbalance problem.

Figures (3)

• Figure 1: Log abstraction of failed/passed logs
• Figure 2: Workflow of our approach
• Figure 3: Lookup Table Construction of NCChecker