Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Chunked-Object Pattern for Multi-Region Large Payload Storage in Managed NoSQL Databases

Manideep Reddy Chinthareddy

TL;DR

The paper addresses the challenge of storing large payloads in managed NoSQL databases that impose strict per-item size limits, which creates cross-region replication hazards when paired with object storage. It introduces the chunked-object pattern, storing large entities as an ordered set of chunks with a metadata barrier to ensure atomic visibility within a single replication stream, and provides a DynamoDB-based reference implementation. Empirical evaluation on production telemetry shows the approach eliminates the dangling-pointer hazard and significantly improves p99 cross-region latency compared with the traditional S3-pointer pattern. The work generalizes to other KV stores and highlights the practical benefits for active-active, multi-region deployments by keeping data and metadata within one replication domain.

Abstract

Many managed key-value and NoSQL databases - such as Amazon DynamoDB, Azure Cosmos DB, and Google Cloud Firestore - enforce strict maximum item sizes (e.g., 400 KB in DynamoDB). This constraint imposes significant architectural challenges for applications requiring low-latency, multi-region access to objects that exceed these limits. The standard industry recommendation is to offload payloads to object storage (e.g., Amazon S3) while retaining a pointer in the database. While cost-efficient, this "pointer pattern" introduces network overhead and exposes applications to non-deterministic replication lag between the database and the object store, creating race conditions in active-active architectures. This paper presents a "chunked-object" pattern that persists large logical entities as sets of ordered chunks within the database itself. We precisely define the pattern and provide a reference implementation using Amazon DynamoDB Global Tables. The design generalizes to any key-value store with per-item size limits and multi-region replication. We evaluate the approach using telemetry from a production system processing over 200,000 transactions per hour. Results demonstrate that the chunked-object pattern eliminates cross-system replication lag hazards and reduces p99 cross-region time-to-consistency for 1 MB payloads by keeping data and metadata within a single consistency domain.

A Chunked-Object Pattern for Multi-Region Large Payload Storage in Managed NoSQL Databases

TL;DR

The paper addresses the challenge of storing large payloads in managed NoSQL databases that impose strict per-item size limits, which creates cross-region replication hazards when paired with object storage. It introduces the chunked-object pattern, storing large entities as an ordered set of chunks with a metadata barrier to ensure atomic visibility within a single replication stream, and provides a DynamoDB-based reference implementation. Empirical evaluation on production telemetry shows the approach eliminates the dangling-pointer hazard and significantly improves p99 cross-region latency compared with the traditional S3-pointer pattern. The work generalizes to other KV stores and highlights the practical benefits for active-active, multi-region deployments by keeping data and metadata within one replication domain.

Abstract

Many managed key-value and NoSQL databases - such as Amazon DynamoDB, Azure Cosmos DB, and Google Cloud Firestore - enforce strict maximum item sizes (e.g., 400 KB in DynamoDB). This constraint imposes significant architectural challenges for applications requiring low-latency, multi-region access to objects that exceed these limits. The standard industry recommendation is to offload payloads to object storage (e.g., Amazon S3) while retaining a pointer in the database. While cost-efficient, this "pointer pattern" introduces network overhead and exposes applications to non-deterministic replication lag between the database and the object store, creating race conditions in active-active architectures. This paper presents a "chunked-object" pattern that persists large logical entities as sets of ordered chunks within the database itself. We precisely define the pattern and provide a reference implementation using Amazon DynamoDB Global Tables. The design generalizes to any key-value store with per-item size limits and multi-region replication. We evaluate the approach using telemetry from a production system processing over 200,000 transactions per hour. Results demonstrate that the chunked-object pattern eliminates cross-system replication lag hazards and reduces p99 cross-region time-to-consistency for 1 MB payloads by keeping data and metadata within a single consistency domain.

Paper Structure

This paper contains 15 sections, 2 figures, 2 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Background and Motivation
  3. The Multi-Region Replication Gap
  4. Motivating Case Study
  5. The Chunked-Object Pattern
  6. Data Model
  7. Write Protocol
  8. Read Path and Reassembly
  9. Applicability to Cosmos DB and Firestore
  10. Multi-Region Behavior Analysis
  11. Empirical Evaluation
  12. Cross-Region Replication Latency
  13. The "Dangling Pointer" Race Condition
  14. Related Work
  15. Conclusion

Figures (2)

  • Figure 1: Comparison of replication latency. The Chunked-Object pattern (Blue) maintains tight consistency, whereas the S3-Pointer pattern (Red) exhibits significant tail latency ($p99 \approx 28.5s$) due to asynchronous object replication.
  • Figure 2: Sequence Diagram of the Dangling Pointer Hazard. The Reader in Region B successfully reads the metadata (Step 4) but fails to read the payload (Step 5) because S3 replication (Step 6) lags behind DynamoDB replication.