Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Algorithms for Efficient, Compact Online Data Stream Curation

Matthew Andres Moreno, Santiago Rodriguez Papa, Emily Dolson

TL;DR

The paper tackles maintaining a temporally representative, memory-efficient archive of streaming data (stream curation) under read-once, ordered input. It introduces five stateless online policies—Fixed Resolution (FR), Depth-Proportional Resolution (DPR), Recency-Proportional Resolution (RPR), Geometric Sequence nth Root (GSNR), and Curbed Recency-Proportional Resolution (CRPR)—that realize different size-growth and temporal-coverage trade-offs while requiring minimal storage, including $O(1)$ per-update indexing. The authors prove self-consistency for each policy and derive asymptotic space bounds, showing how to achieve both even historical coverage and recency-focused detail, making these methods suitable for low-resource devices. Beyond hereditary stratigraphy, these techniques generalize to sensor networks, data logging, and other memory-constrained data stream mining scenarios, offering practical improvements in efficiency and scalability for online data processing.

Abstract

Data stream algorithms tackle operations on high-volume sequences of read-once data items. Data stream scenarios include inherently real-time systems like sensor networks and financial markets. They also arise in purely-computational scenarios like ordered traversal of big data or long-running iterative simulations. In this work, we develop methods to maintain running archives of stream data that are temporally representative, a task we call "stream curation." Our approach contributes to rich existing literature on data stream binning, which we extend by providing stateless (i.e., non-iterative) curation schemes that enable key optimizations to trim archive storage overhead and streamline processing of incoming observations. We also broaden support to cover new trade-offs between curated archive size and temporal coverage. We present a suite of five stream curation algorithms that span $\mathcal{O}(n)$, $\mathcal{O}(\log n)$, and $\mathcal{O}(1)$ orders of growth for retained data items. Within each order of growth, algorithms are provided to maintain even coverage across history or bias coverage toward more recent time points. More broadly, memory-efficient stream curation can boost the data stream mining capabilities of low-grade hardware in roles such as sensor nodes and data logging devices.

Algorithms for Efficient, Compact Online Data Stream Curation

TL;DR

The paper tackles maintaining a temporally representative, memory-efficient archive of streaming data (stream curation) under read-once, ordered input. It introduces five stateless online policies—Fixed Resolution (FR), Depth-Proportional Resolution (DPR), Recency-Proportional Resolution (RPR), Geometric Sequence nth Root (GSNR), and Curbed Recency-Proportional Resolution (CRPR)—that realize different size-growth and temporal-coverage trade-offs while requiring minimal storage, including per-update indexing. The authors prove self-consistency for each policy and derive asymptotic space bounds, showing how to achieve both even historical coverage and recency-focused detail, making these methods suitable for low-resource devices. Beyond hereditary stratigraphy, these techniques generalize to sensor networks, data logging, and other memory-constrained data stream mining scenarios, offering practical improvements in efficiency and scalability for online data processing.

Abstract

Data stream algorithms tackle operations on high-volume sequences of read-once data items. Data stream scenarios include inherently real-time systems like sensor networks and financial markets. They also arise in purely-computational scenarios like ordered traversal of big data or long-running iterative simulations. In this work, we develop methods to maintain running archives of stream data that are temporally representative, a task we call "stream curation." Our approach contributes to rich existing literature on data stream binning, which we extend by providing stateless (i.e., non-iterative) curation schemes that enable key optimizations to trim archive storage overhead and streamline processing of incoming observations. We also broaden support to cover new trade-offs between curated archive size and temporal coverage. We present a suite of five stream curation algorithms that span , , and orders of growth for retained data items. Within each order of growth, algorithms are provided to maintain even coverage across history or bias coverage toward more recent time points. More broadly, memory-efficient stream curation can boost the data stream mining capabilities of low-grade hardware in roles such as sensor nodes and data logging devices.
Paper Structure (15 sections, 1 theorem, 13 equations, 1 figure, 4 algorithms)

This paper contains 15 sections, 1 theorem, 13 equations, 1 figure, 4 algorithms.

Table of Contents

  1. Introduction
  2. Hereditary Stratigraphy
  3. Stream Curation
  4. Stream Curation Policy Stipulation
  5. Stream Curation Policy Algorithms
  6. Existing Work Related to Stream Curation
  7. Applications of Stream Curation
  8. Stream Curation Algorithms
  9. Fixed Resolution (FR) Policy Algorithm
  10. Depth-Proportional Resolution (DPR) Policy Algorithm
  11. Recency-proportional Resolution (RPR) Policy Algorithm
  12. Geometric Sequence $n$th Root (GSNR) Policy Algorithm
  13. Curbed Recency-proportional Resolution (CRPR) Policy Algorithm
  14. Conclusion
  15. Extant Record Size Order of Growth of Retention Policy Algorithms

Key Result

Theorem 1

Recency-proportional Resolution Space Complexity The of the Recency-proportional Resolution Policy Algorithm grows with order $\mathcal{\theta}{(k \log n)}.$

Figures (1)

  • Figure 1: Comparison of stream curation policy algorithms. Policy visualizations show retained strata in black. Time progresses along the $y$-axis from top to bottom. New strata are introduced along the diagonal and then "drip" downward as a vertical line until eliminated. The set of retained strata present within a column at a particular generation $g$ can be read as intersections of retained vertical lines with a horizontal line with intercept $g$. Policy visualizations are provided for two contrastive parameterizations for each policy algorithm.

Theorems & Definitions (2)

  • Theorem 1
  • proof