Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

FluxSieve: Unifying Streaming and Analytical Data Planes for Scalable Cloud Observability

Adriano Vogel, Sören Henning, Otmar Ertl

TL;DR

FluxSieve is proposed, a unified architecture that reconciles traditional pull-based query processing with push-based stream processing by embedding a lightweight in-stream precomputation and filtering layer directly into the data ingestion path and avoids the complexity and operational burden of running queries in dedicated stream processing frameworks.

Abstract

Despite many advances in query optimization, indexing techniques, and data storage, modern data platforms still face difficulties in delivering robust query performance under high concurrency and computationally intensive queries. This challenge is particularly pronounced in large-scale observability platforms handling high-volume, high-velocity data records. For instance, recurrent, expensive filtering queries at query time impose substantial computational and storage overheads in the analytical data plane. In this paper, we propose FluxSieve, a unified architecture that reconciles traditional pull-based query processing with push-based stream processing by embedding a lightweight in-stream precomputation and filtering layer directly into the data ingestion path. This avoids the complexity and operational burden of running queries in dedicated stream processing frameworks. Concretely, this work (i) introduces a foundational architecture that unifies streaming and analytical data planes via in-stream filtering and records enrichment, (ii) designs a scalable multi-pattern matching mechanism that supports concurrent evaluation and on-the-fly updates of filtering rules with minimal per-record overhead, (iii) demonstrates how to integrate this ingestion-time processing with two open-source analytical systems -- Apache Pinot as a Real-Time Online Analytical Processing (RTOLAP) engine and DuckDB as an embedded analytical database, and (iv) performs comprehensive experimental evaluation of our approach. Our evaluation across different systems, query types, and performance metrics shows up to orders-of-magnitude improvements in query performance at the cost of negligible additional storage and very low computational overhead.

FluxSieve: Unifying Streaming and Analytical Data Planes for Scalable Cloud Observability

TL;DR

FluxSieve is proposed, a unified architecture that reconciles traditional pull-based query processing with push-based stream processing by embedding a lightweight in-stream precomputation and filtering layer directly into the data ingestion path and avoids the complexity and operational burden of running queries in dedicated stream processing frameworks.

Abstract

Despite many advances in query optimization, indexing techniques, and data storage, modern data platforms still face difficulties in delivering robust query performance under high concurrency and computationally intensive queries. This challenge is particularly pronounced in large-scale observability platforms handling high-volume, high-velocity data records. For instance, recurrent, expensive filtering queries at query time impose substantial computational and storage overheads in the analytical data plane. In this paper, we propose FluxSieve, a unified architecture that reconciles traditional pull-based query processing with push-based stream processing by embedding a lightweight in-stream precomputation and filtering layer directly into the data ingestion path. This avoids the complexity and operational burden of running queries in dedicated stream processing frameworks. Concretely, this work (i) introduces a foundational architecture that unifies streaming and analytical data planes via in-stream filtering and records enrichment, (ii) designs a scalable multi-pattern matching mechanism that supports concurrent evaluation and on-the-fly updates of filtering rules with minimal per-record overhead, (iii) demonstrates how to integrate this ingestion-time processing with two open-source analytical systems -- Apache Pinot as a Real-Time Online Analytical Processing (RTOLAP) engine and DuckDB as an embedded analytical database, and (iv) performs comprehensive experimental evaluation of our approach. Our evaluation across different systems, query types, and performance metrics shows up to orders-of-magnitude improvements in query performance at the cost of negligible additional storage and very low computational overhead.
Paper Structure (50 sections, 15 figures)

This paper contains 50 sections, 15 figures.

Table of Contents

  1. Introduction
  2. Background and Motivation
  3. Real-world Use-case and Motivation
  4. Status Quo and Challenges
  5. Limitations of Pull-Query Analytics
  6. Stream Processing Potentials—and Its Drawbacks
  7. The FluxSieve Approach
  8. Core Idea
  9. Architecture
  10. In-Stream Multi-Pattern Matching
  11. Dynamic Adaptation for On-the-Fly Filter Condition Updates
  12. Architecture Overview
  13. Pattern Matching Engine Update
  14. Implementation on Kafka Streams
  15. Communication Protocol
  16. ...and 35 more sections

Figures (15)

  • Figure 1: Query processing paradigms: (a) Traditional pull-based queries; (b) Stream Processing; (c) FluxSieve unifies by performing in-stream multi-pattern filtering and enrichment while preserving the analytical plane as the source of truth.
  • Figure 2: Architecture of our approach.
  • Figure 3: Pattern matching engine update: (1) detects changes, (2) updater compiles and uploads (3) notifications, (4) processors fetch the matching engine, (5) acknowledgments.
  • Figure 4: Comparison of deployments.
  • Figure 5: Overhead Analysis.
  • ...and 10 more figures