Quantum Large Language Models via Tensor Network Disentanglers

TL;DR

By incorporating more complex and deeper quantum circuits, along with increasing the bond dimensions of the MPOs, the method captures additional correlations within the quantum-enhanced LLM, leading to improved accuracy beyond classical models while maintaining low memory overhead.

Abstract

We propose a method to enhance the performance of Large Language Models (LLMs) by integrating quantum computing and quantum-inspired techniques. Specifically, our approach involves replacing the weight matrices in the Self-Attention and Multi-layer Perceptron layers with a combination of two variational quantum circuits and a quantum-inspired tensor network, such as a Matrix Product Operator (MPO). This substitution enables the reproduction of classical LLM functionality by decomposing weight matrices through the application of tensor network disentanglers and MPOs, leveraging well-established tensor network techniques. By incorporating more complex and deeper quantum circuits, along with increasing the bond dimensions of the MPOs, our method captures additional correlations within the quantum-enhanced LLM, leading to improved accuracy beyond classical models while maintaining low memory overhead.

Figures (2)

• Figure 1: [Color online] (a) Transformer architecture for an LLM, as discussed originally in Ref. attention; (b) Our architecture for a quantum LLM, where layers involving weight matrices have been replaced by variational quantum circuits combined with a tensor network.
• Figure 2: [Color online] The weight matrix $W$ can be decomposed as an MPO with a given bond dimension, allowing for the LLM compression discussed in Ref. compactifai. This MPO can be further disentangled by quantum circuits $U$ and $V^\dagger$, and the remaining operator can also be written in MPO format (see text). When implemented on a quantum computer (QLLM), the inputs to the quantum circuits must be computed via quantum state encoding, and the outputs must be estimated via sampling. Allowing for more layers in the quantum circuits and for larger bond dimensions in the remaining MPO enhances the model beyond the capabilities of the original one.