Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

RetryGuard: Preventing Self-Inflicted Retry Storms in Cloud Microservices Applications

Jhonatan Tavori, Anat Bremler-Barr, Hanoch Levy, Ofek Lavi

TL;DR

This work identifies self-inflicted retry storms as a critical cost and performance risk in microservice clouds caused by miscoordination between independently auto-scaled services. It introduces RetryGuard, a distributed productive-retry controller that dynamically enables or disables retries at the service level to suppress counterproductive retries during miscoordination while preserving benefits in normal operation. An analytic model links retries, throughput, delays, and costs, yielding threshold-based activation and robustness insights; experimental evaluation on AWS Lambda/DynamoDB and Istio-based BookInfo shows substantial reductions in retry storms, latency, and resource costs. The study also discusses security, economic implications, and practical integration strategies, highlighting RetryGuard’s potential to mitigate Denial-of-Wallet scenarios and to protect against DDoS-like exploitation of retry mechanisms.

Abstract

Modern cloud applications are built on independent, diverse microservices, offering scalability, flexibility, and usage-based billing. However, the structural design of these varied services, along with their reliance on auto-scalers for dynamic internet traffic, introduces significant coordination challenges. As we demonstrate in this paper, common default retry patterns used between misaligned services can turn into retry storms which drive up resource usage and costs, leading to self-inflicted Denial-of-Wallet (DoW) scenarios. To overcome these problems we introduce RetryGuard, a distributed framework for productive control of retry patterns across interdependent microservices. By managing retry policy on a per-service basis and making parallel decisions, RetryGuard prevents retry storms, curbs resource contention, and mitigates escalating operational costs. RetryGuard makes its decisions based on an analytic model that captures the relationships among retries, throughput (rejections), delays, and costs. Experimental results show that RetryGuard significantly reduces resource usage and costs compared to AWS standard and advanced retry policies. We further demonstrate its scalability and superior performance in a more complex Kubernetes deployment with the Istio service mesh, where it achieves substantial improvements.

RetryGuard: Preventing Self-Inflicted Retry Storms in Cloud Microservices Applications

TL;DR

This work identifies self-inflicted retry storms as a critical cost and performance risk in microservice clouds caused by miscoordination between independently auto-scaled services. It introduces RetryGuard, a distributed productive-retry controller that dynamically enables or disables retries at the service level to suppress counterproductive retries during miscoordination while preserving benefits in normal operation. An analytic model links retries, throughput, delays, and costs, yielding threshold-based activation and robustness insights; experimental evaluation on AWS Lambda/DynamoDB and Istio-based BookInfo shows substantial reductions in retry storms, latency, and resource costs. The study also discusses security, economic implications, and practical integration strategies, highlighting RetryGuard’s potential to mitigate Denial-of-Wallet scenarios and to protect against DDoS-like exploitation of retry mechanisms.

Abstract

Modern cloud applications are built on independent, diverse microservices, offering scalability, flexibility, and usage-based billing. However, the structural design of these varied services, along with their reliance on auto-scalers for dynamic internet traffic, introduces significant coordination challenges. As we demonstrate in this paper, common default retry patterns used between misaligned services can turn into retry storms which drive up resource usage and costs, leading to self-inflicted Denial-of-Wallet (DoW) scenarios. To overcome these problems we introduce RetryGuard, a distributed framework for productive control of retry patterns across interdependent microservices. By managing retry policy on a per-service basis and making parallel decisions, RetryGuard prevents retry storms, curbs resource contention, and mitigates escalating operational costs. RetryGuard makes its decisions based on an analytic model that captures the relationships among retries, throughput (rejections), delays, and costs. Experimental results show that RetryGuard significantly reduces resource usage and costs compared to AWS standard and advanced retry policies. We further demonstrate its scalability and superior performance in a more complex Kubernetes deployment with the Istio service mesh, where it achieves substantial improvements.

Paper Structure

This paper contains 42 sections, 19 equations, 15 figures, 1 table, 1 algorithm.

Table of Contents

  1. Introduction
  2. Background and Related Work
  3. Background
  4. Related Work
  5. Case Studies: Self-Inflicted DoW
  6. AWS Lambda and DynamoDB
  7. Setup.
  8. Self-Inflicted DoW: Over-Scaling and Extensive Delays
  9. The Price Tag: Extensive Costs
  10. Service Mesh: Istio's Bookinfo
  11. Setup
  12. Self-Inflicted DoW
  13. RetryGuard Design
  14. A Dynamic Controller for Productive Retries
  15. RetryGuard Principles and Properties
  16. ...and 27 more sections

Figures (15)

  • Figure 1: Retry Operation under Miscoordination.
  • Figure 2: AWS Lambda (fast reaction) and DynamoDB data-base (slow reaction) working in tandem.
  • Figure 3: Lambda delay and cost (log scale): up to 2000% latency increase during retry storms.
  • Figure 4: DB over-scaling: Over-provisioned (red dots) during and after the 'storm', due to retry overwhelm.
  • Figure 5: BookInfo retry storm: Product Page repeatedly attempts to access Review while scaling is not aligned.
  • ...and 10 more figures

Theorems & Definitions (1)

  • Remark 1