Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Stack Trace Deduplication: Faster, More Accurately, and in More Realistic Scenarios

Egor Shibaev, Denis Sushentsev, Yaroslav Golubev, Aleksandr Khvorov

TL;DR

This work tackles stack-trace deduplication in large software systems by proposing a two-stage model: a fast embedding stage for retrieval and a cross-encoder reranker for precise scoring. It introduces SlowOps, an industrial dataset with substantially more per-category reports than public datasets, to enable realistic, practice-oriented evaluation. Across multiple datasets, the approach achieves strong Acc@1 and ROC-AUC scores while delivering favorable runtime, outperforming open benchmarks and approaching or matching LLM-based methods in accuracy with far lower latency. The authors release code and SlowOps data to promote further industry-aligned research and practical deployment in error triage workflows.

Abstract

In large-scale software systems, there are often no fully-fledged bug reports with human-written descriptions when an error occurs. In this case, developers rely on stack traces, i.e., series of function calls that led to the error. Since there can be tens and hundreds of thousands of them describing the same issue from different users, automatic deduplication into categories is necessary to allow for processing. Recent works have proposed powerful deep learning-based approaches for this, but they are evaluated and compared in isolation from real-life workflows, and it is not clear whether they will actually work well at scale. To overcome this gap, this work presents three main contributions: a novel model, an industry-based dataset, and a multi-faceted evaluation. Our model consists of two parts - (1) an embedding model with byte-pair encoding and approximate nearest neighbor search to quickly find the most relevant stack traces to the incoming one, and (2) a reranker that re-ranks the most fitting stack traces, taking into account the repeated frames between them. To complement the existing datasets collected from open-source projects, we share with the community SlowOps - a dataset of stack traces from IntelliJ-based products developed by JetBrains, which has an order of magnitude more stack traces per category. Finally, we carry out an evaluation that strives to be realistic: measuring not only the accuracy of categorization, but also the operation time and the ability to create new categories. The evaluation shows that our model strikes a good balance - it outperforms other models on both open-source datasets and SlowOps, while also being faster on time than most. We release all of our code and data, and hope that our work can pave the way to further practice-oriented research in the area.

Stack Trace Deduplication: Faster, More Accurately, and in More Realistic Scenarios

TL;DR

This work tackles stack-trace deduplication in large software systems by proposing a two-stage model: a fast embedding stage for retrieval and a cross-encoder reranker for precise scoring. It introduces SlowOps, an industrial dataset with substantially more per-category reports than public datasets, to enable realistic, practice-oriented evaluation. Across multiple datasets, the approach achieves strong Acc@1 and ROC-AUC scores while delivering favorable runtime, outperforming open benchmarks and approaching or matching LLM-based methods in accuracy with far lower latency. The authors release code and SlowOps data to promote further industry-aligned research and practical deployment in error triage workflows.

Abstract

In large-scale software systems, there are often no fully-fledged bug reports with human-written descriptions when an error occurs. In this case, developers rely on stack traces, i.e., series of function calls that led to the error. Since there can be tens and hundreds of thousands of them describing the same issue from different users, automatic deduplication into categories is necessary to allow for processing. Recent works have proposed powerful deep learning-based approaches for this, but they are evaluated and compared in isolation from real-life workflows, and it is not clear whether they will actually work well at scale. To overcome this gap, this work presents three main contributions: a novel model, an industry-based dataset, and a multi-faceted evaluation. Our model consists of two parts - (1) an embedding model with byte-pair encoding and approximate nearest neighbor search to quickly find the most relevant stack traces to the incoming one, and (2) a reranker that re-ranks the most fitting stack traces, taking into account the repeated frames between them. To complement the existing datasets collected from open-source projects, we share with the community SlowOps - a dataset of stack traces from IntelliJ-based products developed by JetBrains, which has an order of magnitude more stack traces per category. Finally, we carry out an evaluation that strives to be realistic: measuring not only the accuracy of categorization, but also the operation time and the ability to create new categories. The evaluation shows that our model strikes a good balance - it outperforms other models on both open-source datasets and SlowOps, while also being faster on time than most. We release all of our code and data, and hope that our work can pave the way to further practice-oriented research in the area.

Paper Structure

This paper contains 39 sections, 5 equations, 2 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Background & Related Work
  3. Similarity Models for Stack Trace Deduplication
  4. Evaluation of Similarity Models
  5. Approach
  6. Embedding Model
  7. Preprocessing and tokenization of stack frames
  8. Creating embeddings of stack frames
  9. Creating embeddings of stack traces
  10. Training the Embedding model
  11. Reranker
  12. Preprocessing and tokenization of stack frames
  13. Creating embeddings of stack frames
  14. Creating embeddings of stack traces with cross-encoder
  15. Training the cross-encoder
  16. ...and 24 more sections

Figures (2)

  • Figure 1: The general pipeline of the proposed approach.
  • Figure 2: The architecture of the cross-encoder with biLSTM encoder and searching for identical frames. The identical frames are connected by dotted lines.