AHA: Scalable Alternative History Analysis for Operational Timeseries Applications

TL;DR

The paper tackles the challenge of retrospective, high-dimensional time-series analytics in operational systems by introducing AHA, a two-stage, leaf-based replay system that delivers exact fidelity for decomposable statistics while dramatically reducing storage and compute costs. By exploiting decomposability, sparsity in active leaf cohorts, and efficient cube-based rollups, AHA enables accurate alternative history analysis without resorting to storage-heavy raw data or lossy sketches. The authors provide formal guarantees of predictive equivalence under decomposability, demonstrate substantial performance and cost benefits across multiple real-world datasets, and report strong deployment results in a production video analytics pipeline. Practically, AHA offers a scalable, cost-effective solution for CI/CD style regression testing, anomaly analysis, and algorithm evaluation on historical data, with potential for broad adoption in large-scale operational time-series platforms.

Abstract

Many operational systems collect high-dimensional timeseries data about users/systems on key performance metrics. For instance, ISPs, content distribution networks, and video delivery services collect quality of experience metrics for user sessions associated with metadata (e.g., location, device, ISP). Over such historical data, operators and data analysts often need to run retrospective analysis; e.g., analyze anomaly detection algorithms, experiment with different configurations for alerts, evaluate new algorithms, and so on. We refer to this class of workloads as alternative history analysis for operational datasets. We show that in such settings, traditional data processing solutions (e.g., data warehouses, sampling, sketching, big-data systems) either pose high operational costs or do not guarantee accurate replay. We design and implement a system, called AHA (Alternative History Analytics), that overcomes both challenges to provide cost efficiency and fidelity for high-dimensional data. The design of AHA is based on analytical and empirical insights about such workloads: 1) the decomposability of underlying statistics; 2) sparsity in terms of active number of subpopulations over attribute-value combinations; and 3) efficiency structure of aggregation operations in modern analytics databases. Using multiple real-world datasets and as well as case-studies on production pipelines at a large video analytics company, we show that AHA provides 100% accuracy for a broad range of downstream tasks and up to 85x lower total cost of ownership (i.e., compute + storage) compared to conventional methods.

Figures (13)

• Figure 1: Relative costs of baselines w.r.t AHA to reach given minimum accuracy for at least 90 percentile of cohorts. AHA is 5.6x better than the best 100% accurate baseline, 73x cheaper than the default baseline of storing the raw data, and can be over 100x cheaper than approximate solutions to attain a common goal of $> 95\%$ accuracy.
• Figure 2: Data setting for Alternative history analysis. users require aggregate statics of arbitrary user cohorts across any time-step in the past.
• Figure 3: Design space for alternative history analysis: The session data is ingested and stored at each epoch of time. The user queries from the stored data across time for various tasks. AHA system computes the required features from the stored summary for the specific application.
• Figure 4: Decomposability: Features of a group can be derived from child groups' features.
• Figure 5: Evidence for subgroup sparsity (Insights \ref{['insight:combinationsparsity']}) and CUBE efficiency (Insight \ref{['insight:cubevsgroupby']})

Theorems & Definitions (3)

• Definition 1: Self-decomposable statistic
• Definition 2: Decomposable statistic
• Theorem 1