A Survey of JSON-compatible Binary Serialization Specifications

TL;DR

This survey compares JSON-compatible binary serialization specifications by dividing them into schema-driven and schema-less families, and analyzes encoding primitives such as LEB128 and ZigZag. It synthesizes historical context, practical use-cases, and schema-evolution challenges to guide decisions on when to use schema-driven versus schema-less formats. A reproducible framework and open-source artifacts are provided to enable verifiable comparisons and future benchmarking. The work highlights that no single spec universally optimizes all use-cases, but clarifies how specific design choices map to space efficiency, runtime performance, and evolvability in real-world JSON-like data scenarios.

Abstract

In this paper, we present the recent advances that highlight the characteristics of JSON-compatible binary serialization specifications. We motivate the discussion by covering the history and evolution of binary serialization specifications across the years starting from 1960s to 2000s and onwards. We analyze the use cases of the most popular serialization specifications across the industries. Drawing on the schema-driven (ASN.1, Apache Avro, Microsoft Bond, Cap'n Proto, FlatBuffers, Protocol Buffers, and Apache Thrift) and schema-less (BSON, CBOR, FlexBuffers, MessagePack, Smile, and UBJSON) JSON-compatible binary serialization specifications, we compare and contrast their inner workings through our analysis. We explore a set of non-standardized binary integer encoding techniques (ZigZag integer encoding and Little Endian Base 128 variable-length integer encoding) that are essential to understand the various JSON-compatible binary serialization specifications. We systematically discuss the history, the characteristics, and the serialization processes of the selection of schema-driven and schema-less binary serialization specifications and we identify the challenges associated with schema evolution in the context of binary serialization. Through reflective exercise, we explain our observations of the selection of JSON-compatible binary serialization specifications. This paper aims to guide the reader to make informed decisions on the choice between schema-driven or schema-less JSON-compatible binary serialization specifications.

Figures (75)

• Figure 1: The process of translating a data structure to a bit-string is called serialization. The process of translating a bit-string back to its original data structure is called deserialization.
• Figure 2: A timeline showcasing some of the most popular serialization specifications since the early 1980s.
• Figure 3: The most popular serialization specifications by their use case.
• Figure 4: Textual and binary representations of the decimal number $3.1415926535$. The textual representation encodes the decimal number as a 96-bits sequence of numeric characters ("3" followed by ".", followed by "1", and so forth) that we can easily inspect and understand using a text editor. On the other hand, the binary representation encodes the decimal number in terms of its sign, exponent, and mantissa. The resulting bit-string is only 32 bits long - three times smaller than the textual representation. However, we are unable to understand it using generally-available text-based tools.
• Figure 5: An associative array (also known as a map) that consists of two decimal number properties, "latitude" and "longitude", serialized with fictitious schema-less and schema-driven representations. The schema-less representation (top) is self-descriptive and each property is self-delimited. Alternatively, schema-driven representations (bottom) omit most self-descriptive information except for the length of the associative array as an integer prefix. A reader cannot understand how the schema-driven representation translates to the original data structure without additional information such as a schema definition.