Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Implementing and Evaluating E2LSH on Storage

Yu Nakanishi, Kazuhiro Hiwada, Yosuke Bando, Tomoya Suzuki, Hirotsugu Kajihara, Shintaro Sano, Tatsuro Endo, Tatsuo Shiozawa

TL;DR

It is shown that E2LSH is regaining the advantage in query speed with the advent of modern flash storage devices such as solid-state drives (SSDs), and also that its query time scales sublinearly with the database size beyond the index size limit of in-memory E2 LSH.

Abstract

Locality sensitive hashing (LSH) is one of the widely-used approaches to approximate nearest neighbor search (ANNS) in high-dimensional spaces. The first work on LSH for the Euclidean distance, E2LSH, showed how ANNS can be solved efficiently at a sublinear query time in the database size with theoretically-guaranteed accuracy, although it required a large hash index size. Since then, several LSH variants having much smaller index sizes have been proposed. Their query time is linear or superlinear, but they have been shown to run effectively faster because they require fewer I/Os when the index is stored on hard disk drives and because they also permit in-memory execution with modern DRAM capacity. In this paper, we show that E2LSH is regaining the advantage in query speed with the advent of modern flash storage devices such as solid-state drives (SSDs). We evaluate E2LSH on a modern single-node computing environment and analyze its computational cost and I/O cost, from which we derive storage performance requirements for its external memory execution. Our analysis indicates that E2LSH on a single consumer-grade SSD can run faster than the state-of-the-art small-index methods executed in-memory. It also indicates that E2LSH with emerging high-performance storage devices and interfaces can approach in-memory E2LSH speeds. We implement a simple adaptation of E2LSH to external memory, E2LSH-on-Storage (E2LSHoS), and evaluate it for practical large datasets of up to one billion objects using different combinations of modern storage devices and interfaces. We demonstrate that our E2LSHoS implementation runs much faster than small-index methods and can approach in-memory E2LSH speeds, and also that its query time scales sublinearly with the database size beyond the index size limit of in-memory E2LSH.

Implementing and Evaluating E2LSH on Storage

TL;DR

It is shown that E2LSH is regaining the advantage in query speed with the advent of modern flash storage devices such as solid-state drives (SSDs), and also that its query time scales sublinearly with the database size beyond the index size limit of in-memory E2 LSH.

Abstract

Locality sensitive hashing (LSH) is one of the widely-used approaches to approximate nearest neighbor search (ANNS) in high-dimensional spaces. The first work on LSH for the Euclidean distance, E2LSH, showed how ANNS can be solved efficiently at a sublinear query time in the database size with theoretically-guaranteed accuracy, although it required a large hash index size. Since then, several LSH variants having much smaller index sizes have been proposed. Their query time is linear or superlinear, but they have been shown to run effectively faster because they require fewer I/Os when the index is stored on hard disk drives and because they also permit in-memory execution with modern DRAM capacity. In this paper, we show that E2LSH is regaining the advantage in query speed with the advent of modern flash storage devices such as solid-state drives (SSDs). We evaluate E2LSH on a modern single-node computing environment and analyze its computational cost and I/O cost, from which we derive storage performance requirements for its external memory execution. Our analysis indicates that E2LSH on a single consumer-grade SSD can run faster than the state-of-the-art small-index methods executed in-memory. It also indicates that E2LSH with emerging high-performance storage devices and interfaces can approach in-memory E2LSH speeds. We implement a simple adaptation of E2LSH to external memory, E2LSH-on-Storage (E2LSHoS), and evaluate it for practical large datasets of up to one billion objects using different combinations of modern storage devices and interfaces. We demonstrate that our E2LSHoS implementation runs much faster than small-index methods and can approach in-memory E2LSH speeds, and also that its query time scales sublinearly with the database size beyond the index size limit of in-memory E2LSH.
Paper Structure (34 sections, 12 equations, 16 figures, 6 tables)

This paper contains 34 sections, 12 equations, 16 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Approximate Nearest Neighbor Search
  4. Locality Sensitive Hashing
  5. E2LSH
  6. Other LSH methods
  7. Other ANNS Methods
  8. Experimental Setup
  9. LSH Methods
  10. Evaluation Metrics
  11. Parameter Settings
  12. Datasets
  13. Execution Environment
  14. Storage Devices
  15. Storage Access Interfaces
  16. ...and 19 more sections

Figures (16)

  • Figure 1: Query time models of E2LSHoS
  • Figure 2: Speedup gains of E2LSH over SRS and QALSH
  • Figure 3: Average number of I/Os required to answer a query in SIFT dataset for varying block size $B$
  • Figure 4: IOPS requirements for SRS speeds with varying block size $B$ for SIFT dataset
  • Figure 5: IOPS requirements for SRS speeds with a block size of $B = 512$ bytes
  • ...and 11 more figures