Quantum-inspired Embeddings Projection and Similarity Metrics for Representation Learning

TL;DR

This work tackles the challenge of embedding dimensionality in representation learning by introducing a quantum-inspired projection head and a fidelity-based similarity metric. It maps classical embeddings to quantum states and employs a quantum circuit-based projection to compress embeddings efficiently, evaluated by extending BERT for information retrieval tasks. Empirical results show competitive performance with a 32-fold reduction in parameters, with notable gains when training from scratch on smaller datasets, highlighting data-efficiency advantages. The study demonstrates the practicality of low-entanglement quantum-inspired simulations within neural networks and suggests broad applicability beyond NLP to other representation-learning problems.

Abstract

Over the last decade, representation learning, which embeds complex information extracted from large amounts of data into dense vector spaces, has emerged as a key technique in machine learning. Among other applications, it has been a key building block for large language models and advanced computer vision systems based on contrastive learning. A core component of representation learning systems is the projection head, which maps the original embeddings into different, often compressed spaces, while preserving the similarity relationship between vectors. In this paper, we propose a quantum-inspired projection head that includes a corresponding quantum-inspired similarity metric. Specifically, we map classical embeddings onto quantum states in Hilbert space and introduce a quantum circuit-based projection head to reduce embedding dimensionality. To evaluate the effectiveness of this approach, we extended the BERT language model by integrating our projection head for embedding compression. We compared the performance of embeddings, which were compressed using our quantum-inspired projection head, with those compressed using a classical projection head on information retrieval tasks using the TREC 2019 and TREC 2020 Deep Learning benchmarks. The results demonstrate that our quantum-inspired method achieves competitive performance relative to the classical method while utilizing 32 times fewer parameters. Furthermore, when trained from scratch, it notably excels, particularly on smaller datasets. This work not only highlights the effectiveness of the quantum-inspired approach but also emphasizes the utility of efficient, ad hoc low-entanglement circuit simulations within neural networks as a powerful quantum-inspired technique.

Figures (6)

• Figure 1: Approaches to embedding compression.
• Figure 2: BERT architecture at training.
• Figure 3: Classical embedding compression for the base BERT model. Only the red components are tunable.
• Figure 4: Quantum-inspired embedding compression of base BERT. Only the red components are tunable; green components correspond to the different encoding methods ($\ket{\psi_i}$ writes back the measured probability distribution of the i-th qubit, and $enc(\cdot)$ is the data encoding as defined in Equation \ref{['eq2.5']}), and yellow components are fixed operations or frozen layers.
• Figure 5: Performance comparison of the classical and quantum-inspired compression models ($BERT_{base}$) on the TREC19 and TREC20 benchmarks. The two charts on the left illustrate the performance variation as a function of the training sample count, with 100% corresponding to the use of 100K samples. The right two charts depict performance progression over training epochs using the full training sample set. The scores are scaled by 100.