Learning Complex Word Embeddings in Classical and Quantum Spaces
Carys Harvey, Stephen Clark, Douglas Brown, Konstantinos Meichanetzidis
TL;DR
The paper investigates learning word embeddings in complex-valued spaces and via quantum-inspired representations. It formalizes a Skip-gram framework using a fidelity-based overlap $F(v_f,v_c)$ and a scaled variant $F_D(v_f,v_c)$ to train complex vectors, and explores two routes: direct complex-valued embeddings and embeddings produced by parameterised quantum circuits (PQCs). A two-stage approach—learning complex embeddings first and then fitting a PQC per word to these embeddings—yields results on standard similarity datasets that are competitive with the classical real-valued Skip-gram while enabling vocabulary-scale scalability; direct PQC training tends to degrade performance. The work highlights a scalable path for complex-valued and quantum-inspired NLP representations and discusses hardware considerations, including barren plateaus and gradient computation, guiding future quantum NLP developments.
Abstract
We present a variety of methods for training complex-valued word embeddings, based on the classical Skip-gram model, with a straightforward adaptation simply replacing the real-valued vectors with arbitrary vectors of complex numbers. In a more "physically-inspired" approach, the vectors are produced by parameterised quantum circuits (PQCs), which are unitary transformations resulting in normalised vectors which have a probabilistic interpretation. We develop a complex-valued version of the highly optimised C code version of Skip-gram, which allows us to easily produce complex embeddings trained on a 3.8B-word corpus for a vocabulary size of over 400k, for which we are then able to train a separate PQC for each word. We evaluate the complex embeddings on a set of standard similarity and relatedness datasets, for some models obtaining results competitive with the classical baseline. We find that, while training the PQCs directly tends to harm performance, the quantum word embeddings from the two-stage process perform as well as the classical Skip-gram embeddings with comparable numbers of parameters. This enables a highly scalable route to learning embeddings in complex spaces which scales with the size of the vocabulary rather than the size of the training corpus. In summary, we demonstrate how to produce a large set of high-quality word embeddings for use in complex-valued and quantum-inspired NLP models, and for exploring potential advantage in quantum NLP models.