Lance: Efficient Random Access in Columnar Storage through Adaptive Structural Encodings

TL;DR

The paper investigates random access performance of columnar storage formats on NVMe-backed cloud storage, identifying structural encoding as the key factor shaping I/O efficiency. It introduces Lance, an adaptive encoding that combines full-zip and miniblock strategies to maintain strong random access across data widths and nesting levels while preserving full-scan performance and enabling a small search cache. Empirical results show Parquet can be tuned to achieve substantial improvements in random access, and Lance often matches or exceeds Parquet in both random access and full scans, with notable advantages in memory usage and tunability. The findings highlight the practical impact of encoding choices on NVMe utilization and offer a blueprint for future configurable, multi-purpose columnar formats.

Abstract

The growing interest in artificial intelligence has created workloads that require both sequential and random access. At the same time, NVMe-backed storage solutions have emerged, providing caching capability for large columnar datasets in cloud storage. Current columnar storage libraries fall short of effectively utilizing an NVMe device's capabilities, especially when it comes to random access. Historically, this has been assumed an implicit weakness in columnar storage formats, but this has not been sufficiently explored. In this paper, we examine the effectiveness of popular columnar formats such as Apache Arrow, Apache Parquet, and Lance in both random access and full scan tasks against NVMe storage. We argue that effective encoding of a column's structure, such as the repetition and validity information, is the key to unlocking the disk's performance. We show that Parquet, when configured correctly, can achieve over 60x better random access performance than default settings. We also show that this high random access performance requires making minor trade-offs in scan performance and RAM utilization. We then describe the Lance structural encoding scheme, which alternates between two different structural encodings based on data width, and achieves better random access performance without making trade-offs in scan performance or RAM utilization.

Figures (18)

• Figure 1: NVMe storage introduces unique performance characteristics
• Figure 2: An overview of the major components of each of the file formats
• Figure 3: An interpretation of a column chunk containing binary data (delta-length binary packed encoding) which has a buffer of lengths ($b_0$) and data ($b_1$) as a series of pages with $p0, p1, ...$ values in each page
• Figure 4: An example showing the layout and accesses needed to fetch the 10th item in a List<String> array.
• Figure 5: The full-zip encoding transposes the buffers into a row-major order.