Suki: Choreographed Distributed Dataflow in Rust

TL;DR

This extended abstract presents Suki, an embedded Rust DSL that lets developers implement streaming dataflow with explicit placement of computation, and explores how this approach, combined with Rust's trait system, enables a type-safe API for mapping dataflow programs to cloud computing resources.

Abstract

Programming models for distributed dataflow have long focused on analytical workloads that allow the runtime to dynamically place and schedule compute logic. Meanwhile, models that enable fine-grained control over placement, such as actors, make global optimization difficult. In this extended abstract, we present Suki, an embedded Rust DSL that lets developers implement streaming dataflow with explicit placement of computation. Key to this choreographic programming approach is our use of staged programming, which lets us expose a high-level Rust API while compiling local compute units into individual binaries with zero-overhead. We also explore how this approach, combined with Rust's trait system, enables a type-safe API for mapping dataflow programs to cloud computing resources.