HQViT: Hybrid Quantum Vision Transformer for Image Classification

TL;DR

HQViT tackles the quadratic complexity of Vision Transformer self-attention by introducing a moderate quantum–classical hybrid that processes whole images with amplitude encoding and computes attention coefficients in the quantum domain. It minimizes quantum resources to $O(\log_2(Td))$ qubits and $O(\log_2 d)$ PQGs while achieving a classical complexity reduction of $O(T^2 d)$, making it feasible for NISQ devices. Across MNIST, MedMNIST2D, CIFAR-10, and Mini-ImageNet, HQViT attains competitive or superior accuracy compared with existing quantum and classical baselines, including up to roughly 11 percentage points gains on MNIST 10-class tasks. The work demonstrates the practical potential of hybrid quantum–classical architectures for vision tasks and paves the way for deeper stacking and explicit positional encoding as future directions.

Abstract

Transformer-based architectures have revolutionized the landscape of deep learning. In computer vision domain, Vision Transformer demonstrates remarkable performance on par with or even surpassing that of convolutional neural networks. However, the quadratic computational complexity of its self-attention mechanism poses challenges for classical computing, making model training with high-dimensional input data, e.g., images, particularly expensive. To address such limitations, we propose a Hybrid Quantum Vision Transformer (HQViT), that leverages the principles of quantum computing to accelerate model training while enhancing model performance. HQViT introduces whole-image processing with amplitude encoding to better preserve global image information without additional positional encoding. By leveraging quantum computation on the most critical steps and selectively handling other components in a classical way, we lower the cost of quantum resources for HQViT. The qubit requirement is minimized to $O(log_2N)$ and the number of parameterized quantum gates is only $O(log_2d)$, making it well-suited for Noisy Intermediate-Scale Quantum devices. By offloading the computationally intensive attention coefficient matrix calculation to the quantum framework, HQViT reduces the classical computational load by $O(T^2d)$. Extensive experiments across various computer vision datasets demonstrate that HQViT outperforms existing models, achieving a maximum improvement of up to $10.9\%$ (on the MNIST 10-classification task) over the state of the art. This work highlights the great potential to combine quantum and classical computing to cope with complex image classification tasks.

Figures (8)

• Figure 1: The trade-off between the amount of quantum resource and classical computational complexity for existing quantum self-attention and quantum transformer models. Our motivation is to achieve a balance between the two, ensuring that the quantum resources align with the limitations of NISQ devices while maintaining high-performance and scalability. The vertical axis on the left represents the quantum resource requirements, while the vertical axis on the right represents the classical computational complexity of the model. Both axes are dimensionless, since we do not conduct a quantitative comparison between the two, merely illustrating the relative relationships of the quantum resource requirements (or classical computational complexity) across different models.
• Figure 2: Overview of HQViT's design principles, advantages, and contributions.
• Figure 3: The swap test circuit.
• Figure 4: Overall framework of HQViT. The image is first segmented and flattened as a whole-image embedding (we use a 4x4 grayscale image as an example), which is processed by the quantum self-attention module. The output is then passed through classical feedforward layers (FFN) and further refined by using a classical fully connected layer (FCL) for final classification. The quantumized self-attention module is the core of the model, while other components are handled classically.
• Figure 5: Quantum self-attention mechanism for HQViT. The whole-image embedding is fed into registers $R_Q$, $R_K$, and $R_V$ with amplitude encoding (AE), followed by unitary transformations ($U_Q$, $U_K$, $U_V$). A swap test is performed on the $Q$ and $K$ quantum states to compute the attention matrix, which is then combined with the $V$ to produce Weighted Values.