Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LOG.io: Unified Rollback Recovery and Data Lineage Capture for Distributed Data Pipelines

Eric Simon, Renato B. Hoffmann, Lucas Alf, Dalvan Griebler

TL;DR

LOG.io introduces a log-based rollback recovery framework for distributed, serverless data pipelines that supports non-deterministic operators and external interactions while enabling dynamic scaling. It couples a pessimistic, event-centric logging protocol with fine-grain data lineage capture, providing non-blocking recovery and scalable deployment, implemented within SAP Data Intelligence. Empirical results show LOG.io can match or surpass ABS in recovery scenarios, with data lineage overhead remaining under 1.5%, and demonstrate how data parallelization mitigates normal-processing overhead. The work delivers a practical, unified approach with operator APIs and a concrete implementation pathway for enterprise data pipelines.

Abstract

This paper introduces LOG.io, a comprehensive solution designed for correct rollback recovery and fine-grain data lineage capture in distributed data pipelines. It is tailored for serverless scalable architectures and uses a log-based rollback recovery protocol. LOG.io supports a general programming model, accommodating non-deterministic operators, interactions with external systems, and arbitrary custom code. It is non-blocking, allowing failed operators to recover independently without interrupting other active operators, thereby leveraging data parallelization, and it facilitates dynamic scaling of operators during pipeline execution. Performance evaluations, conducted within the SAP Data Intelligence system, compare LOG.io with the Asynchronous Barrier Snapshotting (ABS) protocol, originally implemented in Flink. Our experiments show that when there are straggler operators in a data pipeline and the throughput of events is moderate (e.g., 1 event every 100 ms), LOG.io performs as well as ABS during normal processing and outperforms ABS during recovery. Otherwise, ABS performs better than LOG.io for both normal processing and recovery. However, we show that in these cases, data parallelization can largely reduce the overhead of LOG.io while ABS does not improve. Finally, we show that the overhead of data lineage capture, at the granularity of the event and between any two operators in a pipeline, is marginal, with less than 1.5% in all our experiments.

LOG.io: Unified Rollback Recovery and Data Lineage Capture for Distributed Data Pipelines

TL;DR

LOG.io introduces a log-based rollback recovery framework for distributed, serverless data pipelines that supports non-deterministic operators and external interactions while enabling dynamic scaling. It couples a pessimistic, event-centric logging protocol with fine-grain data lineage capture, providing non-blocking recovery and scalable deployment, implemented within SAP Data Intelligence. Empirical results show LOG.io can match or surpass ABS in recovery scenarios, with data lineage overhead remaining under 1.5%, and demonstrate how data parallelization mitigates normal-processing overhead. The work delivers a practical, unified approach with operator APIs and a concrete implementation pathway for enterprise data pipelines.

Abstract

This paper introduces LOG.io, a comprehensive solution designed for correct rollback recovery and fine-grain data lineage capture in distributed data pipelines. It is tailored for serverless scalable architectures and uses a log-based rollback recovery protocol. LOG.io supports a general programming model, accommodating non-deterministic operators, interactions with external systems, and arbitrary custom code. It is non-blocking, allowing failed operators to recover independently without interrupting other active operators, thereby leveraging data parallelization, and it facilitates dynamic scaling of operators during pipeline execution. Performance evaluations, conducted within the SAP Data Intelligence system, compare LOG.io with the Asynchronous Barrier Snapshotting (ABS) protocol, originally implemented in Flink. Our experiments show that when there are straggler operators in a data pipeline and the throughput of events is moderate (e.g., 1 event every 100 ms), LOG.io performs as well as ABS during normal processing and outperforms ABS during recovery. Otherwise, ABS performs better than LOG.io for both normal processing and recovery. However, we show that in these cases, data parallelization can largely reduce the overhead of LOG.io while ABS does not improve. Finally, we show that the overhead of data lineage capture, at the granularity of the event and between any two operators in a pipeline, is marginal, with less than 1.5% in all our experiments.

Paper Structure

This paper contains 91 sections, 2 theorems, 2 equations, 13 figures, 9 tables, 13 algorithms.

Table of Contents

  1. Introduction
  2. Data pipelines
  3. Correct rollback recovery
  4. Fine-grain data lineage capture
  5. Research contributions
  6. Paper outline
  7. Preliminary definitions and assumptions
  8. Connections and events
  9. Read and write actions
  10. Read and write actions
  11. Data pipeline operators
  12. Types of operators
  13. Local state
  14. Stateless and stateful operator
  15. Input and Output Sets
  16. ...and 76 more sections

Key Result

Proposition 1

Let R be a Source operator in a data pipeline DP, and $\Gamma^{rec}(DP, \sigma(R))$ be a complete recovered execution of DP with schedule $\sigma(R)$. Let $out$ be the output port of R. Let $O = \Gamma^{rec}(DP, \sigma(R)).out$ be the sequence of output events produced by R on port $out$, then there

Figures (13)

  • Figure 1: Sales data pipeline
  • Figure 2: Data lineage paths
  • Figure 3: Data pipeline with data lineage enabled
  • Figure 4: Use Case data pipelines
  • Figure 5: Use Case 1 - 100 events
  • ...and 8 more figures

Theorems & Definitions (14)

  • Example 1
  • Example 2
  • Example 4
  • Example 5
  • Example 6
  • Example 7
  • Example 8
  • Proposition 1
  • proof
  • Proposition 2
  • ...and 4 more