Enzyme: Incremental View Maintenance for Data Engineering

Abstract

Materialized views are a core construct in database systems, used to accelerate analytical queries and optimize batch pipelines for extract-transform-load (ETL) workflows. Maintaining view consistency as underlying data evolves is a fundamental challenge, especially in high-throughput and real-time settings. Incremental view maintenance (IVM) has been studied for decades and continues to attract significant investment from major database vendors. However, most industrial systems either offer limited SQL-operator coverage or require users to hand-tune refresh strategies. This paper presents Enzyme, an IVM engine developed at Databricks to power Spark Declarative Pipelines. It provides a built-in, end-to-end approach to incremental pipelines, utilizing materialized views as first-class building blocks. By automating refresh planning, Enzyme reduces total cost of ownership and lets users focus on business logic rather than MV mechanics. Validation across thousands of large-scale production pipelines spanning diverse application domains has demonstrated substantial computational efficiency gains, yielding a cumulative daily compute reduction of billions of CPU seconds. Built atop Apache Spark primitives, Enzyme adds a cost-based optimization layer that selects refresh strategies for collections of materialized views organized into pipelines. Enzyme's modular architecture is designed to generalize across data sources and query engines. We present key design decisions for incremental refresh planning and execution, including optimizations that exploit batching opportunities across materialized view sources. Experimental results on standard benchmarks demonstrate significant performance improvements at scale.

Figures (11)

• Figure 1: Medallion architecture organizing data into progressively refined layers for analytics.
• Figure 2: A standalone MV that computes the average order amount for three selected regions. The MV is configured to refresh every hour. We use this as a running example throughout the paper.
• Figure 3: Simplified query plan for the MV query in \ref{['fig:running_example_mv']}.
• Figure 4: Illustration of operator-level delta plan composition. The incremental plan is constructed by traversing the query plan from the leaf nodes upward and progressively applying delta rules. At each operator node $\psi$, three outputs are generated: $\psi$ (the previous output), $\psi'$ (the output over the updated input), and $\Delta \psi$ (the change up to this node). These intermediate deltas are then used by parent operators, following the algebraic formulations in \ref{['subsec:ivm_logic']}, to compose the final incremental plan $\Delta Q$.
• Figure 5: Enzyme transforms a query into an incremental execution in six phases: normalization, fingerprinting, decomposition, plan generation, costing, and execution.