Quantum Transfer Learning for Acceptability Judgements

TL;DR

This paper investigates quantum transfer learning for NLP by applying amplitude-encoded embeddings from pre-trained Italian LMs to the acceptability-judgment task ItaCoLA. It introduces two quantum pipelines, BERT-Quant and ELECTRA-Quant, and benchmarks them against classical transfer-learning baselines, using a variational quantum classifier on top of quantum-encoded embeddings. ELECTRA-Quant achieves mean accuracy around $0.92$ and MCC around $0.678$, closely matching ELECTRA-Classic, while SHAP-based explanations reveal improved handling of complex Italian syntactic phenomena. The work demonstrates the feasibility of quantum-classical hybrids for NLP tasks, highlights potential quantum advantages in modeling language structure, and motivates further exploration of quantum representations and linguistic theory mappings.

Abstract

Hybrid quantum-classical classifiers promise to positively impact critical aspects of natural language processing tasks, particularly classification-related ones. Among the possibilities currently investigated, quantum transfer learning, i.e., using a quantum circuit for fine-tuning pre-trained classical models for a specific task, is attracting significant attention as a potential platform for proving quantum advantage. This work shows potential advantages, both in terms of performance and expressiveness, of quantum transfer learning algorithms trained on embedding vectors extracted from a large language model to perform classification on a classical Linguistics task: acceptability judgments. Acceptability judgment is the ability to determine whether a sentence is considered natural and well-formed by a native speaker. The approach has been tested on sentences extracted from ItaCoLa, a corpus that collects Italian sentences labeled with their acceptability judgment. The evaluation phase shows results for the quantum transfer learning pipeline comparable to state-of-the-art classical transfer learning algorithms, proving current quantum computers' capabilities to tackle NLP tasks for ready-to-use applications. Furthermore, a qualitative linguistic analysis, aided by explainable AI methods, reveals the capabilities of quantum transfer learning algorithms to correctly classify complex and more structured sentences, compared to their classical counterpart. This finding sets the ground for a quantifiable quantum advantage in NLP in the near future.

Figures (9)

• Figure 1: ELECTRA overview with replaced token detection.
• Figure 2: Schematic of the quantum transfer learning scheme adopted. Sentences from ItaCola dataset are tokenized and then embeddings from BERT and ELECTRA (both pre-trained) are obtained for each data point. The embeddings are then encoded in the parametrized quantum circuit via amplitude encoding. The results of the measurement on the quantum states are then fed into a multi-layer perceptron (MLP) through which the classification is performed.
• Figure 3: Loss function for the traing of BERT-Quant and ELECTRA-Quant. The training phase of the two quantum transfer learning strategies constructed, namely BERT-Quant and ELECTRA-Quant, both with learning rate $10^-5$, batch size $16$ and categorical cross entropy as objective function. In both cases, the parametrized quantum circuit is made of $6$ Basic Entangling Layers. The training reveals that ELECTRA-Quant is a more effective for minimizing the loss, thus learns better than BERT-Quant to classify the acceptability of the sentences.
• Figure 4: Example of a dendrogram representation for a simple sentence in an interrogative form. Orange arcs represent the parts of the sentence that positively impact classification, classifying it as an unacceptable sentence.
• Figure 5: Example of a acceptable sentence presenting a cleft structure. Orange arcs, representing correctly classified words, coincide with the main clause, which is also the most readable part, while the subordinate is the part that negatively impacts the correct classification of the sentence as acceptable (blue arcs).