Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

The workflow motif: a widely-useful performance diagnosis abstraction for distributed applications

Mania Abdi, Peter Desnoyers, Mark Crovella, Raja R. Sambasivan

TL;DR

The paper addresses the challenge of diagnosing problems in deployed distributed applications due to a mismatch between developer abstractions and available diagnosis tools. It introduces the workflow motif, a formal abstraction for frequent processing patterns in request workflows, augmented with performance characteristics, and defines a formal framework for mining them from workflow-centric traces using frequent-subgraph mining. The authors discuss candidate mining systems (Arabesque, Gaston) and present a first case study on Hadoop HDFS that identifies about 40 motifs at 75% support and highlights a slow-read motif as a potential optimization. If refined, workflow motifs could enable top-down, hierarchical diagnosis, improved anomaly detection, and targeted performance optimization across distributed storage and compute platforms.

Abstract

Diagnosing problems in deployed distributed applications continues to grow more challenging. A significant reason is the extreme mismatch between the powerful abstractions developers have available to build increasingly complex distributed applications versus the simple ones engineers have available to diagnose problems in them. To help, we present a novel abstraction, the workflow motif, instantiations of which represent characteristics of frequently-repeating patterns within and among request executions. We argue that workflow motifs will benefit many diagnosis tasks, formally define them, and use this definition to identify which frequent-subgraph-mining algorithms are good starting points for mining workflow motifs. We conclude by using an early version of workflow motifs to suggest performance-optimization points in HDFS.

The workflow motif: a widely-useful performance diagnosis abstraction for distributed applications

TL;DR

The paper addresses the challenge of diagnosing problems in deployed distributed applications due to a mismatch between developer abstractions and available diagnosis tools. It introduces the workflow motif, a formal abstraction for frequent processing patterns in request workflows, augmented with performance characteristics, and defines a formal framework for mining them from workflow-centric traces using frequent-subgraph mining. The authors discuss candidate mining systems (Arabesque, Gaston) and present a first case study on Hadoop HDFS that identifies about 40 motifs at 75% support and highlights a slow-read motif as a potential optimization. If refined, workflow motifs could enable top-down, hierarchical diagnosis, improved anomaly detection, and targeted performance optimization across distributed storage and compute platforms.

Abstract

Diagnosing problems in deployed distributed applications continues to grow more challenging. A significant reason is the extreme mismatch between the powerful abstractions developers have available to build increasingly complex distributed applications versus the simple ones engineers have available to diagnose problems in them. To help, we present a novel abstraction, the workflow motif, instantiations of which represent characteristics of frequently-repeating patterns within and among request executions. We argue that workflow motifs will benefit many diagnosis tasks, formally define them, and use this definition to identify which frequent-subgraph-mining algorithms are good starting points for mining workflow motifs. We conclude by using an early version of workflow motifs to suggest performance-optimization points in HDFS.

Paper Structure

This paper contains 7 sections, 2 equations, 2 figures, 1 table.

Table of Contents

  1. Introduction
  2. Diagnosis tasks
  3. Workflow motifs & mining them
  4. Defining workflow motifs
  5. Initial progress
  6. Open questions
  7. Summary

Figures (2)

  • Figure 1: Diagnosis tasks, workflow motifs, and request workflows. The top of the figure shows examples of diagnosis tasks that involve identifying commonalities within and among request workflows. The bottom shows the workflows of two READ requests in a simple distributed application comprised of an app server, a database, and a storage system. The first request (purple) hits in the database's client cache; the second request (green) requires a storage-node access.
  • Figure 2: Workflow-motif-mining example. The leftmost figure shows a set of hypothetical workflow-centric traces mined from Ceph, a distributed-storage application. The rightmost figure shows the workflow motifs that would be mined from these traces.