Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Causify DataFlow: A Framework For High-performance Machine Learning Stream Computing

Giacinto Paolo Saggese, Paul Smith

TL;DR

DataFlow tackles the mismatch between finite-batch data science and unbounded streaming time-series processing by introducing a DAG-based framework with point-in-time idempotency and tiling. It unifies batch and streaming semantics, enforces causality through knowledge time and embargo mechanisms, and enables memory-bounded, vectorized, and highly parallel execution with incremental caching and native Python data-science stack support. The approach provides formal guarantees for batch-streaming equivalence, supports various evaluation designs, and offers robust debugging, backtesting, and reproducibility features, making it suitable for finance, IoT, fraud detection, and real-time analytics. Collectively, the work delivers a comprehensive engine, semantics, and MLOps capabilities that bridge research prototyping and production deployment in time-series ML.

Abstract

We present DataFlow, a computational framework for building, testing, and deploying high-performance machine learning systems on unbounded time-series data. Traditional data science workflows assume finite datasets and require substantial reimplementation when moving from batch prototypes to streaming production systems. This gap introduces causality violations, batch boundary artifacts, and poor reproducibility of real-time failures. DataFlow resolves these issues through a unified execution model based on directed acyclic graphs (DAGs) with point-in-time idempotency: outputs at any time t depend only on a fixed-length context window preceding t. This guarantee ensures that models developed in batch mode execute identically in streaming production without code changes. The framework enforces strict causality by automatically tracking knowledge time across all transformations, eliminating future-peeking bugs. DataFlow supports flexible tiling across temporal and feature dimensions, allowing the same model to operate at different frequencies and memory profiles via configuration alone. It integrates natively with the Python data science stack and provides fit/predict semantics for online learning, caching and incremental computation, and automatic parallelization through DAG-based scheduling. We demonstrate its effectiveness across domains including financial trading, IoT, fraud detection, and real-time analytics.

Causify DataFlow: A Framework For High-performance Machine Learning Stream Computing

TL;DR

DataFlow tackles the mismatch between finite-batch data science and unbounded streaming time-series processing by introducing a DAG-based framework with point-in-time idempotency and tiling. It unifies batch and streaming semantics, enforces causality through knowledge time and embargo mechanisms, and enables memory-bounded, vectorized, and highly parallel execution with incremental caching and native Python data-science stack support. The approach provides formal guarantees for batch-streaming equivalence, supports various evaluation designs, and offers robust debugging, backtesting, and reproducibility features, making it suitable for finance, IoT, fraud detection, and real-time analytics. Collectively, the work delivers a comprehensive engine, semantics, and MLOps capabilities that bridge research prototyping and production deployment in time-series ML.

Abstract

We present DataFlow, a computational framework for building, testing, and deploying high-performance machine learning systems on unbounded time-series data. Traditional data science workflows assume finite datasets and require substantial reimplementation when moving from batch prototypes to streaming production systems. This gap introduces causality violations, batch boundary artifacts, and poor reproducibility of real-time failures. DataFlow resolves these issues through a unified execution model based on directed acyclic graphs (DAGs) with point-in-time idempotency: outputs at any time t depend only on a fixed-length context window preceding t. This guarantee ensures that models developed in batch mode execute identically in streaming production without code changes. The framework enforces strict causality by automatically tracking knowledge time across all transformations, eliminating future-peeking bugs. DataFlow supports flexible tiling across temporal and feature dimensions, allowing the same model to operate at different frequencies and memory profiles via configuration alone. It integrates natively with the Python data science stack and provides fit/predict semantics for online learning, caching and incremental computation, and automatic parallelization through DAG-based scheduling. We demonstrate its effectiveness across domains including financial trading, IoT, fraud detection, and real-time analytics.
Paper Structure (93 sections, 19 theorems, 65 equations, 13 figures, 10 tables)

This paper contains 93 sections, 19 theorems, 65 equations, 13 figures, 10 tables.

Table of Contents

  1. Introduction
  2. The fallacy of finite data in data science
  3. Definition of stream computing
  4. Core principles of stream computing
  5. Applications of stream computing
  6. DataFlow at a Glance
  7. Core Abstractions
  8. Tiling and Point-in-Time Idempotency
  9. Architectural Benefits
  10. Context Windows and Tile Composition
  11. Challenges in time series machine learning
  12. Prototype vs Production
  13. Frequency of Model Operation
  14. Non-stationarity time series
  15. Non-causal Bugs
  16. ...and 78 more sections

Key Result

Lemma 4.17

Let $N$ be a point-in-time idempotent node with context window length $L$. Then for each output index $j \in \{1,\dots,n\}$ there exists a function such that for any time $t \in \mathbb{T}$ and any interval $[s,t]$ with $t-s+1 \geq L$, if applying $N$ on $[s,t]$ yields outputs $(Y^{(1)},\dots,Y^{(n)})$, then

Figures (13)

  • Figure 1: Future-peeking bug example. The incorrect code uses shift(-1), which accesses data from the future (time $t+1$ when making decision at time $t$). This violates causality because the future price is not available at decision time. The correct approach uses shift(1) to access historical data. In backtesting, both approaches may appear to work, but only the causal version is valid for production deployment.
  • Figure 2: Transformation from long format (database style) to wide multi-index format (DataFlow style). The wide format enables vectorized operations across assets.
  • Figure 3: Point-in-time idempotency demonstration. Despite different input window lengths (5 vs. 10 time steps), both computations produce identical output at $t_0$ because they depend only on the context window of length $L=3$ (shown in blue). The moving average uses only the last 3 values $\{11, 13, 14\}$ in both cases.
  • Figure 4: Two-tile execution mechanism. The first tile provides context for boundary computations in the second tile. Outputs in the second tile (red) are valid because they have access to sufficient context from both tiles. For nodes with context window $L$, outputs at times like $t_{-3}$ depend on inputs spanning both tiles.
  • Figure 5: Simple DAG with concurrent execution. Nodes $C_1$ and $C_2$ can execute in parallel since neither depends on the other.
  • ...and 8 more figures

Theorems & Definitions (86)

  • Definition 4.1: Time index
  • Definition 4.2: Time intervals
  • Definition 4.3: Dataframe as unit of computation
  • Remark 4.4
  • Definition 4.5: DataFlow data format
  • Definition 4.6: Stream dataframe
  • Definition 4.7: Computational Node
  • Remark 4.8: Examples of Computational Node in Financial Trading Systems
  • Definition 4.9: Causal Computation
  • Definition 4.10: Knowledge Time
  • ...and 76 more