StreamShield: A Production-Proven Resiliency Solution for Apache Flink at ByteDance

TL;DR

StreamShield addresses resiliency and stability challenges in ByteDance's large Apache Flink deployment by introducing engine-level optimizations, a hybrid replication strategy, and a robust release pipeline. The approach combines adaptive runtime mechanisms (Adaptive Shuffle, Group-Rescale, WeakHash, autoscaling), fine-grained fault tolerance (region checkpointing, single-task recovery, State LazyLoad), and dependency-aware high-availability for external services. A comprehensive testing and release pipeline, including chaos testing and performance benchmarking, validates stability before production. Production experiments show substantial improvements in startup time, throughput under skew, elasticity under bursts, and resilience against external-dependency failures, enabling reliable SLO satisfaction across heterogeneous workloads.

Abstract

Distributed Stream Processing Systems (DSPSs) form the backbone of real-time processing and analytics at ByteDance, where Apache Flink powers one of the largest production clusters worldwide. Ensuring resiliency, the ability to withstand and rapidly recover from failures, together with operational stability, which provides consistent and predictable performance under normal conditions, is essential for meeting strict Service Level Objectives (SLOs). However, achieving resiliency and stability in large-scale production environments remains challenging due to the cluster scale, business diversity, and significant operational overhead. In this work, we present StreamShield, a production-proven resiliency solution deployed in ByteDance's Flink clusters. Designed along complementary perspectives of the engine and cluster, StreamShield introduces key techniques to enhance resiliency, covering runtime optimization, fine-grained fault-tolerance, hybrid replication strategy, and high availability under external systems. Furthermore, StreamShield proposes a robust testing and deployment pipeline that ensures reliability and robustness in production releases. Extensive evaluations on a production cluster demonstrate the efficiency and effectiveness of techniques proposed by StreamShield.

Figures (9)

• Figure 1: The Architecture of Apache Flink.
• Figure 2: Shuffle Strategies Optimization.
• Figure 3: Original v.s. Region Checkpointing.
• Figure 4: Testing and Release Pipeline of the Flink Engine.
• Figure 5: Comparison of Job Startup Overhead Across Different Numbers of TMs.

Theorems & Definitions (1)

• Definition : Resiliency Problem