Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Predicting Intermittent Job Failure Categories for Diagnosis Using Few-Shot Fine-Tuned Language Models

Henri Aïdasso, Francis Bordeleau, Ali Tizghadam

TL;DR

The paper tackles the challenge of diagnosing intermittent (flaky) CI/CD failures by predicting their failure categories from job logs. It introduces FlaXifyer, a few-shot learning pipeline that uses log pre-processing, SetFit-based fine-tuning, and a lightweight classifier to achieve 84.3% Macro F1 and 92.0% Top-2 accuracy with only 12 labeled examples per category, evaluated on 2,458 TELUS failures across 13 categories. Complemented by LogSift, an interpretability method that localizes influential log statements, the approach reduces log review effort by 74.4% and surfaces actionable failure cues in 87% of cases, with near real-time execution. Together, FlaXifyer and LogSift demonstrate effective automated triage and diagnostic acceleration for intermittent job failures, enabling scalable extension to new categories and potential automated repair workflows in CI/CD environments.

Abstract

In principle, Continuous Integration (CI) pipeline failures provide valuable feedback to developers on code-related errors. In practice, however, pipeline jobs often fail intermittently due to non-deterministic tests, network outages, infrastructure failures, resource exhaustion, and other reliability issues. These intermittent (flaky) job failures lead to substantial inefficiencies: wasted computational resources from repeated reruns and significant diagnosis time that distracts developers from core activities and often requires intervention from specialized teams. Prior work has proposed machine learning techniques to detect intermittent failures, but does not address the subsequent diagnosis challenge. To fill this gap, we introduce FlaXifyer, a few-shot learning approach for predicting intermittent job failure categories using pre-trained language models. FlaXifyer requires only job execution logs and achieves 84.3% Macro F1 and 92.0% Top-2 accuracy with just 12 labeled examples per category. We also propose LogSift, an interpretability technique that identifies influential log statements in under one second, reducing review effort by 74.4% while surfacing relevant failure information in 87% of cases. Evaluation on 2,458 job failures from TELUS demonstrates that FlaXifyer and LogSift enable effective automated triage, accelerate failure diagnosis, and pave the way towards the automated resolution of intermittent job failures.

Predicting Intermittent Job Failure Categories for Diagnosis Using Few-Shot Fine-Tuned Language Models

TL;DR

The paper tackles the challenge of diagnosing intermittent (flaky) CI/CD failures by predicting their failure categories from job logs. It introduces FlaXifyer, a few-shot learning pipeline that uses log pre-processing, SetFit-based fine-tuning, and a lightweight classifier to achieve 84.3% Macro F1 and 92.0% Top-2 accuracy with only 12 labeled examples per category, evaluated on 2,458 TELUS failures across 13 categories. Complemented by LogSift, an interpretability method that localizes influential log statements, the approach reduces log review effort by 74.4% and surfaces actionable failure cues in 87% of cases, with near real-time execution. Together, FlaXifyer and LogSift demonstrate effective automated triage and diagnostic acceleration for intermittent job failures, enabling scalable extension to new categories and potential automated repair workflows in CI/CD environments.

Abstract

In principle, Continuous Integration (CI) pipeline failures provide valuable feedback to developers on code-related errors. In practice, however, pipeline jobs often fail intermittently due to non-deterministic tests, network outages, infrastructure failures, resource exhaustion, and other reliability issues. These intermittent (flaky) job failures lead to substantial inefficiencies: wasted computational resources from repeated reruns and significant diagnosis time that distracts developers from core activities and often requires intervention from specialized teams. Prior work has proposed machine learning techniques to detect intermittent failures, but does not address the subsequent diagnosis challenge. To fill this gap, we introduce FlaXifyer, a few-shot learning approach for predicting intermittent job failure categories using pre-trained language models. FlaXifyer requires only job execution logs and achieves 84.3% Macro F1 and 92.0% Top-2 accuracy with just 12 labeled examples per category. We also propose LogSift, an interpretability technique that identifies influential log statements in under one second, reducing review effort by 74.4% while surfacing relevant failure information in 87% of cases. Evaluation on 2,458 job failures from TELUS demonstrates that FlaXifyer and LogSift enable effective automated triage, accelerate failure diagnosis, and pave the way towards the automated resolution of intermittent job failures.
Paper Structure (44 sections, 3 equations, 5 figures, 5 tables, 1 algorithm)

This paper contains 44 sections, 3 equations, 5 figures, 5 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Background and Related Work
  3. CI/CD and Intermittent Job Failures
  4. Industrial Context.
  5. Motivation for Automated Failure Categorization.
  6. Related Work
  7. Intermittent Job Failure Detection
  8. Intermittent Job Failure Categories
  9. FlaXifyer
  10. Overview
  11. Log Pre-processing
  12. Few-Shot Fine-Tuning
  13. Classification Head
  14. Interpretability: Identifying Influential Log Statements
  15. Motivation
  16. ...and 29 more sections

Figures (5)

  • Figure 1: Predicting intermittent job failure categories from job logs using a language model and classification head.
  • Figure 2: LogSift relevance assessment results.
  • Figure 3: LogSift reduces a 359-line log to 2 lines, directly identifying the missing environment variable (IMAGE_NAME).
  • Figure 4: LogSift output for a misclassified job (true: external_file_invalid_format). Despite the incorrect prediction, the highlighted segment reveals the actual cause: a zip file read failure (curl: (26)) during upload, enabling developers to override the prediction.
  • Figure 5: Example of LogSift producing a non-relevant segment. Despite high reduction, the identified statements contain no failure indicators or host resolution information.