Chipmink: Efficient Delta Identification for Massive Object Graph

TL;DR

Chipmink introduces a structure-aware delta object store that partitions evolving object graphs into pods to support partial persistence for large-scale data science workflows. The core ideas are podding (subgraph partitioning), a virtual global memo space for cross-pod references, and a learned Greedy Algorithm (LGA) to optimize pod granularity and minimize persistence cost; change detection and a synonym cache reduce I/O by reusing unchanged pods, while asynchronous saving preserves interactivity via active-variable locking and an allowlist-based static code checker. The system achieves up to 36.5x storage reduction and 12.4x faster persistence over strong baselines on real notebooks, and it supports Python libraries across memory, GPUs, and remote objects. Together, these mechanisms enable efficient, correct delta-based persistence for massive object graphs, improving resilience and exploratory workflows in data science environments. Chipmink’s modular design suggests broad portability to other runtimes and serialization schemes, with potential further gains from more lightweight volatility modeling and parallelized serialization.

Abstract

Ranging from batch scripts to computational notebooks, modern data science tools rely on massive and evolving object graphs that represent structured data, models, plots, and more. Persisting these objects is critical, not only to enhance system robustness against unexpected failures but also to support continuous, non-linear data exploration via versioning. Existing object persistence mechanisms (e.g., Pickle, Dill) rely on complete snapshotting, often redundantly storing unchanged objects during execution and exploration, resulting in significant inefficiency in both time and storage. Unlike DBMSs, data science systems lack centralized buffer managers that track dirty objects. Worse, object states span various locations such as memory heaps, shared memory, GPUs, and remote machines, making dirty object identification fundamentally more challenging. In this work, we propose a graph-based object store, named Chipmink, that acts like the centralized buffer manager. Unlike static pages in DBMSs, persistence units in Chipmink are dynamically induced by partitioning objects into appropriate subgroups (called pods), minimizing expected persistence costs based on object sizes and reference structure. These pods effectively isolate dirty objects, enabling efficient partial persistence. Our experiments show that Chipmink is general, supporting libraries that rely on shared memory, GPUs, and remote objects. Moreover, Chipmink achieves up to 36.5x smaller storage sizes and 12.4x faster persistence than the best baselines in real-world notebooks and scripts.

Figures (20)

• Figure 1: Cumulative numbers of buildats's objects to be saved by snapshot-based storage vs. delta storage.
• Figure 2: Chipmink saves and loads state for exploration.
• Figure 3: Chipmink's system architecture and interactions with computation notebook systems and the underlying storage. This data flow diagram illustrates Chipmink's partial saving and loading (\ref{['sec:podding']}, blue components), podding optimization (\ref{['sec:podding-optimization']}, orange components), and safe unblocking for uninterrupted exploration (\ref{['sec:async']}, green components).
• Figure 4: An instance of ObjectGraph. Namespace consists of objects ${\mathcal{U}}$ (circles) and dependencies ${\mathcal{E}}$ (arrows) between them. Each variable naming $\ell$ (blue dash line) connects a variable name (rectangle) to its object $v \in {\mathcal{V}}$ (blue circle).
• Figure 5: Different podding decision actions on the same current object (orange circle), parent pod (grey round box), and new pod (blue round box) if any.

Theorems & Definitions (8)

• theorem 1
• theorem 2
• theorem 3
• Lemma 4
• Lemma 5
• theorem 4
• Lemma 6
• theorem 5