Encrypted Vector Similarity Computations Using Partially Homomorphic Encryption: Applications and Performance Analysis

TL;DR

This work tackles privacy-preserving vector similarity by enabling cosine similarity computations over encrypted data using partially homomorphic encryption (PHE) rather than fully homomorphic encryption (FHE). It introduces a normalization-based transformation so that cosine similarity can be computed via encrypted dot products, leveraging additively homomorphic schemes like Paillier, Damgård-Jurik, and Okamoto-Uchiyama in a two-tower architecture. Through experiments on the LFW dataset with FaceNet and VGG-Face embeddings, the authors demonstrate that PHE offers substantially lower encryption, decryption, and operation times and smaller ciphertexts/keys than FHE, albeit with trade-offs in precision (loss) and memory for some configurations. The results suggest PHE as a practical, scalable option for privacy-preserving encrypted similarity search in facial recognition, image retrieval, and related applications on edge devices, with future work extending to more schemes and similarity measures.

Abstract

This paper explores the use of partially homomorphic encryption (PHE) for encrypted vector similarity search, with a focus on facial recognition and broader applications like reverse image search, recommendation engines, and large language models (LLMs). While fully homomorphic encryption (FHE) exists, we demonstrate that encrypted cosine similarity can be computed using PHE, offering a more practical alternative. Since PHE does not directly support cosine similarity, we propose a method that normalizes vectors in advance, enabling dot product calculations as a proxy. We also apply min-max normalization to handle negative dimension values. Experiments on the Labeled Faces in the Wild (LFW) dataset use DeepFace's FaceNet128d, FaceNet512d, and VGG-Face (4096d) models in a two-tower setup. Pre-encrypted embeddings are stored in one tower, while an edge device captures images, computes embeddings, and performs encrypted-plaintext dot products via additively homomorphic encryption. We implement this with LightPHE, evaluating Paillier, Damgard-Jurik, and Okamoto-Uchiyama schemes, excluding others due to performance or decryption complexity. Tests at 80-bit and 112-bit security (NIST-secure until 2030) compare PHE against FHE (via TenSEAL), analyzing encryption, decryption, operation time, cosine similarity loss, key/ciphertext sizes. Results show PHE is less computationally intensive, faster, and produces smaller ciphertexts/keys, making it well-suited for memory-constrained environments and real-world privacy-preserving encrypted similarity search.

Figures (1)

• Figure 1: Facial Recognition Pipeline as a Two Tower Architecture