Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Larger than memory image processing

Jon Sporring, David Stansby

TL;DR

The paper tackles the challenge of analyzing images that exceed RAM on petascale datasets by arguing that I/O is the primary bottleneck and advocating slice-wise streaming as a robust path. It presents a memory model built around sweeping, halos, and higher-order functionals, and it introduces stackProcessing, a DSL that automatically optimizes windowing, fusion, and I/O scheduling under bounded RAM. Key contributions include formalizing memory- and I/O-aware pipeline execution, defining stack-based primitives for streaming computations, and providing practical strategies (mid-write splits and window-size tuning) to sustain throughput. The approach enables near-linear I/O scans and predictable memory usage, facilitating large-scale analyses on commodity hardware and existing segmentation/morphology tools without requiring full-volume residency in memory.

Abstract

This report addresses larger-than-memory image analysis for petascale datasets such as 1.4 PB electron-microscopy volumes and 150 TB human-organ atlases. We argue that performance is fundamentally I/O-bound. We show that structuring analysis as streaming passes over data is crucial. For 3D volumes, two representations are popular: stacks of 2D slices (e.g., directories or multi-page TIFF) and 3D chunked layouts (e.g., Zarr/HDF5). While for a few algorithms, chunked layout on disk is crucial to keep disk I/O at a minimum, we show how the slice-based streaming architecture can be built on top of either image representation in a manner that minimizes disk I/O. This is in particular advantageous for algorithms relying on neighbouring values, since the slicing streaming architecture is 1D, which implies that there are only 2 possible sweeping orders, both of which are aligned with the order in which images are read from the disk. This is in contrast to 3D chunks, in which any sweep cannot be done without accessing each chunk at least 9 times. We formalize this with sweep-based execution (natural 2D/3D orders), windowed operations, and overlap-aware tiling to minimize redundant access. Building on these principles, we introduce a domain-specific language (DSL) that encodes algorithms with intrinsic knowledge of their optimal streaming and memory use; the DSL performs compile-time and run-time pipeline analyses to automatically select window sizes, fuse stages, tee and zip streams, and schedule passes for limited-RAM machines, yielding near-linear I/O scans and predictable memory footprints. The approach integrates with existing tooling for segmentation and morphology but reframes pre/post-processing as pipelines that privilege sequential read/write patterns, delivering substantial throughput gains for extremely large images without requiring full-volume residency in memory.

Larger than memory image processing

TL;DR

The paper tackles the challenge of analyzing images that exceed RAM on petascale datasets by arguing that I/O is the primary bottleneck and advocating slice-wise streaming as a robust path. It presents a memory model built around sweeping, halos, and higher-order functionals, and it introduces stackProcessing, a DSL that automatically optimizes windowing, fusion, and I/O scheduling under bounded RAM. Key contributions include formalizing memory- and I/O-aware pipeline execution, defining stack-based primitives for streaming computations, and providing practical strategies (mid-write splits and window-size tuning) to sustain throughput. The approach enables near-linear I/O scans and predictable memory usage, facilitating large-scale analyses on commodity hardware and existing segmentation/morphology tools without requiring full-volume residency in memory.

Abstract

This report addresses larger-than-memory image analysis for petascale datasets such as 1.4 PB electron-microscopy volumes and 150 TB human-organ atlases. We argue that performance is fundamentally I/O-bound. We show that structuring analysis as streaming passes over data is crucial. For 3D volumes, two representations are popular: stacks of 2D slices (e.g., directories or multi-page TIFF) and 3D chunked layouts (e.g., Zarr/HDF5). While for a few algorithms, chunked layout on disk is crucial to keep disk I/O at a minimum, we show how the slice-based streaming architecture can be built on top of either image representation in a manner that minimizes disk I/O. This is in particular advantageous for algorithms relying on neighbouring values, since the slicing streaming architecture is 1D, which implies that there are only 2 possible sweeping orders, both of which are aligned with the order in which images are read from the disk. This is in contrast to 3D chunks, in which any sweep cannot be done without accessing each chunk at least 9 times. We formalize this with sweep-based execution (natural 2D/3D orders), windowed operations, and overlap-aware tiling to minimize redundant access. Building on these principles, we introduce a domain-specific language (DSL) that encodes algorithms with intrinsic knowledge of their optimal streaming and memory use; the DSL performs compile-time and run-time pipeline analyses to automatically select window sizes, fuse stages, tee and zip streams, and schedule passes for limited-RAM machines, yielding near-linear I/O scans and predictable memory footprints. The approach integrates with existing tooling for segmentation and morphology but reframes pre/post-processing as pipelines that privilege sequential read/write patterns, delivering substantial throughput gains for extremely large images without requiring full-volume residency in memory.
Paper Structure (14 sections, 16 equations, 4 figures, 3 tables)

This paper contains 14 sections, 16 equations, 4 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Image from the Human Organ Atlas Hub
  3. Image pipeline
  4. Organising Images on Disk as Stacks versus Chunks
  5. Review of existing tools for parallel programming and streaming
  6. Memory model for larger-than-memory 3D operators
  7. Sweeping and halos
  8. Higher-order functionals on stacks
  9. Sequential and parallel composition
  10. Optimising pipelines
  11. Mid-write splits:
  12. Window-size optimization:
  13. stackProcessing (FSharp.StackProcessing).
  14. Conclusion

Figures (4)

  • Figure 1: The segmentation of several structures in a zoom in of the human colon image at the HOAHub.
  • Figure 2: A common image processing pipeline
  • Figure 3: An illustration of representing a 3-dimensional image as a stack of 2-dimensional slices (LEFT) and as a set of non-overlapping 3-dimensional sub-images also called chuncks (RIGHT).
  • Figure 4: An illustration of the halo process: Each chunk to be processed is extended with parts of its neighbours, the computation is performed, and the center is extracted. The result is recombined into a resulting image.