Data accounting and error counting

TL;DR

The paper tackles error tracing in large-scale data analytics by proposing data accounting, a data-space algebra that generalizes relational algebra to enable complete, information-preserving summarizations. It defines a formal framework (data spaces $( abla D, abla ext{≤}_{ abla D}, abla orm{ullet}_{ abla D}, ( abla M, e, abla ullet))$) and introduces information fusion monoids, partitions, and information-preserving operations, including tensor products and per-row enrichment, to support bidirectional, reversible transformations. It also integrates concepts from bidirectional computing, event sourcing, and railroad-oriented programming to enable injective debugging and parallel error-path processing, with concrete translations from relational algebra to data-space operations and guidance for SQL implementation. The practical impact is a principled pathway to automated error tracing, robust risk reporting, and scalable debugging in data pipelines by maintaining error traces alongside correct results and enabling minimal, per-record impact analysis.

Abstract

Can we infer sources of errors from outputs of the complex data analytics software? Bidirectional programming promises that we can reverse flow of software, and translate corrections of output into corrections of either input or data analysis. This allows us to achieve holy grail of automated approaches to debugging, risk reporting and large scale distributed error tracking. Since processing of risk reports and data analysis pipelines can be frequently expressed using a sequence relational algebra operations, we propose a replacement of this traditional approach with a data summarization algebra that helps to determine an impact of errors. It works by defining data analysis of a necessarily complete summarization of a dataset, possibly in multiple ways along multiple dimensions. We also present a description to better communicate how the complete summarizations of the input data may facilitates easier debugging and more efficient development of analysis pipelines. This approach can also be described as an generalization of axiomatic theories of accounting into data analytics, thus dubbed data accounting. We also propose formal properties that allow for transparent assertions about impact of individual records on the aggregated data and ease debugging by allowing to find minimal changes that change behaviour of data analysis on per-record basis.