Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Streaming Democratized: Ease Across the Latency Spectrum with Delayed View Semantics and Snowflake Dynamic Tables

Daniel Sotolongo, Daniel Mills, Tyler Akidau, Anirudh Santhiar, Attila-Péter Tóth, Ilaria Battiston, Ankur Sharma, Botong Huang, Boyuan Zhang, Dzmitry Pauliukevich, Enrico Sartorello, Igor Belianski, Ivan Kalev, Lawrence Benson, Leon Papke, Ling Geng, Matt Uhlar, Nikhil Shah, Niklas Semmler, Olivia Zhou, Saras Nowak, Sasha Lionheart, Till Merker, Vlad Lifliand, Wendy Grus, Yi Huang, Yiwen Zhu

TL;DR

Delayed view semantics (DVS) and Dynamic Tables (DTs) provide a SQL-native framework to bridge streaming and databases, enabling end-to-end reasoning about delayed results with target lag bounds. The work extends transaction isolation with derivations to model asynchronous streaming computations and implements these ideas inside Snowflake via scheduling, query rewriting, and incremental maintenance. The authors report architectural design, implementation details, and production learnings, demonstrating broad adoption and practical performance across latency regimes, while acknowledging limitations and outlining future research on optimization and semantic extensions. This approach promises to democratize streaming by embedding it in an enterprise-grade analytical DBMS, enabling practitioners to express streaming pipelines with familiar SQL semantics and predictable behavior.

Abstract

Streaming data pipelines remain challenging and expensive to build and maintain, despite significant advancements in stronger consistency, event time semantics, and SQL support over the last decade. Persistent obstacles continue to hinder usability, such as the need for manual incrementalization, semantic discrepancies across SQL implementations, and the lack of enterprise-grade operational features. While the rise of incremental view maintenance (IVM) as a way to integrate streaming with databases has been a huge step forward, transaction isolation in the presence of IVM remains underspecified, leaving the maintenance of application-level invariants as a painful exercise for the user. Meanwhile, most streaming systems optimize for latencies of 100 ms to 3 sec, whereas many practical use cases are well-served by latencies ranging from seconds to tens of minutes. We present delayed view semantics (DVS), a conceptual foundation that bridges the semantic gap between streaming and databases, and introduce Dynamic Tables, Snowflake's declarative streaming transformation primitive designed to democratize analytical stream processing. DVS formalizes the intuition that stream processing is primarily a technique to eagerly compute derived results asynchronously, while also addressing the need to reason about the resulting system end to end. Dynamic Tables then offer two key advantages: ease of use through DVS, enterprise-grade features, and simplicity; as well as scalable cost efficiency via IVM with an architecture designed for diverse latency requirements. We first develop extensions to transaction isolation that permit the preservation of invariants in streaming applications. We then detail the implementation challenges of Dynamic Tables and our experience operating it at scale. Finally, we share insights into user adoption and discuss our vision for the future of stream processing.

Streaming Democratized: Ease Across the Latency Spectrum with Delayed View Semantics and Snowflake Dynamic Tables

TL;DR

Delayed view semantics (DVS) and Dynamic Tables (DTs) provide a SQL-native framework to bridge streaming and databases, enabling end-to-end reasoning about delayed results with target lag bounds. The work extends transaction isolation with derivations to model asynchronous streaming computations and implements these ideas inside Snowflake via scheduling, query rewriting, and incremental maintenance. The authors report architectural design, implementation details, and production learnings, demonstrating broad adoption and practical performance across latency regimes, while acknowledging limitations and outlining future research on optimization and semantic extensions. This approach promises to democratize streaming by embedding it in an enterprise-grade analytical DBMS, enabling practitioners to express streaming pipelines with familiar SQL semantics and predictable behavior.

Abstract

Streaming data pipelines remain challenging and expensive to build and maintain, despite significant advancements in stronger consistency, event time semantics, and SQL support over the last decade. Persistent obstacles continue to hinder usability, such as the need for manual incrementalization, semantic discrepancies across SQL implementations, and the lack of enterprise-grade operational features. While the rise of incremental view maintenance (IVM) as a way to integrate streaming with databases has been a huge step forward, transaction isolation in the presence of IVM remains underspecified, leaving the maintenance of application-level invariants as a painful exercise for the user. Meanwhile, most streaming systems optimize for latencies of 100 ms to 3 sec, whereas many practical use cases are well-served by latencies ranging from seconds to tens of minutes. We present delayed view semantics (DVS), a conceptual foundation that bridges the semantic gap between streaming and databases, and introduce Dynamic Tables, Snowflake's declarative streaming transformation primitive designed to democratize analytical stream processing. DVS formalizes the intuition that stream processing is primarily a technique to eagerly compute derived results asynchronously, while also addressing the need to reason about the resulting system end to end. Dynamic Tables then offer two key advantages: ease of use through DVS, enterprise-grade features, and simplicity; as well as scalable cost efficiency via IVM with an architecture designed for diverse latency requirements. We first develop extensions to transaction isolation that permit the preservation of invariants in streaming applications. We then detail the implementation challenges of Dynamic Tables and our experience operating it at scale. Finally, we share insights into user adoption and discuss our vision for the future of stream processing.

Paper Structure

This paper contains 28 sections, 2 theorems, 6 equations, 6 figures.

Table of Contents

  1. Introduction
  2. Related Work
  3. Concepts
  4. What is computed?
  5. Delayed View Semantics (DVS)
  6. Data Timestamp Selection
  7. When is it computed?
  8. How is it computed?
  9. Warehouses
  10. Refresh Modes
  11. Skips and Errors
  12. Miscellaneous Features
  13. Delayed View Semantics & Transaction Isolation
  14. Implementation
  15. Architecture
  16. ...and 13 more sections

Key Result

Theorem 1

Given any history $H$ containing a transaction $T_i$ and a derivation $r = d_i(x_i | \dots)$, define another history $H'$ which moves $r$ into another transaction $T_j$ to create $d_j(x_j | \dots)$ and replaces all reads from $x_i$ in $H$ with reads from $x_j$. $H$ has exactly the same dependencies

Figures (6)

  • Figure 1: A diagram of persisted table semantics. The DSG is serializable despite the clear presence of read skew because the refresh transactions mask the conflict.
  • Figure 2: The example in Figure \ref{['fig:pts']}, mapped to delayed view semantics. The history now contains derivations, and the DSG contains a cycle, indicating read skew.
  • Figure 3: An architecture diagram of Dynamic Tables.
  • Figure 4: An example of lag over time. Boxes $v_i$ are data timestamps. $s_i$ and $e_i$ denote the start and end times of refreshes. Dashed arrows are measurements.
  • Figure 5: Distribution of the target lags of active DTs.
  • ...and 1 more figures

Theorems & Definitions (2)

  • Theorem 1: Transaction Invariance
  • corollary 1: Encapsulation