NSP-BERT: A Prompt-based Few-Shot Learner Through an Original Pre-training Task--Next Sentence Prediction

TL;DR

<3-5 sentence high-level summary> NSP-BERT reframes prompt-based learning by using Next Sentence Prediction as a sentence-level pre-training task, addressing variable-length label representations and enabling zero-shot and few-shot learning without fixed token mappings. It introduces NSP-tuning, two-stage prompts, and two answer-mapping strategies (candidates-contrast and samples-contrast) to handle diverse downstream tasks, including entity linking and Chinese NLP benchmarks. By continuing pre-training BERT on RoBERTa-like corpora, NSP-BERT achieves strong zero-shot performance and competitive few-shot results on English and Chinese datasets (FewCLUE, DuEL2.0), often with faster convergence than token-level baselines. The work highlights the importance of pre-training data and model size in few-shot settings and expands prompt-learning beyond token-level MLM toward a robust, sentence-level paradigm with broad applicability.

Abstract

Using prompts to utilize language models to perform various downstream tasks, also known as prompt-based learning or prompt-learning, has lately gained significant success in comparison to the pre-train and fine-tune paradigm. Nonetheless, virtually all prompt-based methods are token-level, meaning they all utilize GPT's left-to-right language model or BERT's masked language model to perform cloze-style tasks. In this paper, we attempt to accomplish several NLP tasks in the zero-shot scenario using a BERT original pre-training task abandoned by RoBERTa and other models--Next Sentence Prediction (NSP). Unlike token-level techniques, our sentence-level prompt-based method NSP-BERT does not need to fix the length of the prompt or the position to be predicted, allowing it to handle tasks such as entity linking with ease. Based on the characteristics of NSP-BERT, we offer several quick building templates for various downstream tasks. We suggest a two-stage prompt method for word sense disambiguation tasks in particular. Our strategies for mapping the labels significantly enhance the model's performance on sentence pair tasks. On the FewCLUE benchmark, our NSP-BERT outperforms other zero-shot methods on most of these tasks and comes close to the few-shot methods.

Figures (10)

• Figure 1: Prompts for various NLP tasks of NSP-BERT.
• Figure 2: (Left) MLM task for token-level prompt-learning. (Right) NSP task for sentence-level prompt-learning.
• Figure 3: NSP-tuning for single-sentence classification. " Manchester United lose 4:0 to Brighton" is the original input, the gold label is Sports. The negative instances are building with wrong label Politics, Bussiness, etc.
• Figure 4: Two-stage prompt, examples in coreference resolution and entity linking/typing tasks.
• Figure 5: The accuracy of the 4 methods for each epoch during few-shot training on Yahoo! and AGNews.