8 Years of Optimizing Apache Otava: How disconnected open source developers took an algorithm from n3 to constant time

TL;DR

The paper chronicles an 8-year evolution of the e-divisive change point detection algorithm across a loosely connected developer ecosystem, culminating in Apache Otava (incubating) and real-time, on-demand change point computation. It analyzes the computational bottlenecks, notably the Monte Carlo significance test, and surveys prior optimizations (e.g., shift-row-diff, vectorization, native C) plus the shift to a Student's $t$-test, which yields large speedups. A central contribution is the incremental optimization that recomputes only the trailing $W$-point windows when new data arrive, reducing complexity toward $O(TW)$ and achieving constant-time behavior in the end-appended data scenario. The results demonstrate substantial practical impact for Continuous Performance Engineering, including speedups on the order of $1.8\times 10^4$ to $3.0\times 10^5$ over the naive approach and broad adoption within industry workflows via Apache Otava.

Abstract

As the project now known as Apache Otava (incubating) makes it first release, we look back over the past 8 years that the codebase was developed by a rather uncoordinated, loosely connected group of performance engineers at MongoDB, Datastax, Confluent, Nyrkio and others. Ever since the first publication (Daly 2020), developers of the code base now known as Apache Otava (incubating), have continuosly improved its performance. Even when a contributor's primary motivation was to add functionality, it seems like they couldn't help themselves but to also make some performance optimizations while at it. When developing the Nyrkio web service to provide change detection for performance testing, we have observed that Otava had become fast enough that it was almost feasible to compute change points synchronously, as the user is browsing test results in a web browser. Inspired by this, we have developed and contributed a new optimization for the common case where new data points are appended to the end of the series. This is now the 7th generation of performance optimizations in Otava. These improvements have been done over the past 8 years of development, by disconnected individuals at different employees. Taken together, the historical optimizations and those published in this paper, represent a 18 000 to 300 000 speedup over the original by the book implementation of (Matteson and James 2014). In the language of computational complexity, an evolution from O (n3) to O (1) (constant time). The ability to compute and recompute change points in real-time unlocks new opportunities in the user experience.

Figures (4)

• Figure 1: A short timeseries used for illustration purposes
• Figure 2: Pairwise differences
• Figure 3: The E-divisive algorithm, illustrated. (See EDIV for actual math.
• Figure 4: $\hat{q}$ computed for each gap between points in the series