OpenPrompt: An Open-source Framework for Prompt-learning

TL;DR

Addresses the lack of a standard open-source framework for prompt-learning by introducing OpenPrompt, a modular toolkit that unifies templates, verbalizers, PLMs, and training strategies. By enabling flexible combinations across task formats, PLMs (MLM/LM/Seq2Seq), and prompting modules, OpenPrompt facilitates rapid experimentation and fair comparisons. It provides a novel template language and tokenization layer to simplify prompt construction and a model-agnostic forward interface for training and inference. Evaluations across diverse NLP tasks demonstrate the framework's generality, while ongoing development and tutorials aim to boost adoption.

Abstract

Prompt-learning has become a new paradigm in modern natural language processing, which directly adapts pre-trained language models (PLMs) to $cloze$-style prediction, autoregressive modeling, or sequence to sequence generation, resulting in promising performances on various tasks. However, no standard implementation framework of prompt-learning is proposed yet, and most existing prompt-learning codebases, often unregulated, only provide limited implementations for specific scenarios. Since there are many details such as templating strategy, initializing strategy, and verbalizing strategy, etc. need to be considered in prompt-learning, practitioners face impediments to quickly adapting the desired prompt learning methods to their applications. In this paper, we present {OpenPrompt}, a unified easy-to-use toolkit to conduct prompt-learning over PLMs. OpenPrompt is a research-friendly framework that is equipped with efficiency, modularity, and extendibility, and its combinability allows the freedom to combine different PLMs, task formats, and prompting modules in a unified paradigm. Users could expediently deploy prompt-learning frameworks and evaluate the generalization of them on different NLP tasks without constraints. OpenPrompt is publicly released at {\url{ https://github.com/thunlp/OpenPrompt}}.

Figures (5)

• Figure 1: The overall architecture of OpenPrompt. Note that according to the prompt-learning strategies, not all the modules are necessarily used. For example, in generation tasks, there are no verbalizers in the learning procedure. The PromptTrainer is a controller that controls the data flow and the training process with some unique attributes, users can also implement the training process in a conventional fashion.
• Figure 2: Some examples of our template language. In our template language, we can use the key "meta" to refer the original input text (Example B), parts of the original input (Example A, C, G), or other key information. We can also freely specify which tokens are hard and which are soft (and their initialization strategy). We could assign an id for a soft token to specify which tokens are sharing embeddings (Example F). OpenPrompt also supports the post processing (Example E) for each token, e.g., lambda expression or MLP.
• Figure 3: An example to define a Verbalizer, the number of the label words for each class is flexible.
• Figure 4: The illustration of the validation space of OpenPrompt. By driving different modules of the framework, we could implement and evaluate different methods on a broad set of NLP tasks. We show four examples in this illustration, the colored lines denote the implementation flow of the corresponding method.
• Figure 5: An example to define a PromptModel and conduct evaluation.