A Tutorial on the Pretrain-Finetune Paradigm for Natural Language Processing

TL;DR

The paper addresses how social scientists can leverage NLP for unstructured text analysis with limited labeled data by adopting the pretrain-finetune paradigm. It articulates the workflow from pretraining on unlabeled data (tokenization, encoding, and objectives) to finetuning pretrained models for downstream tasks like classification and regression, and provides concrete, open-source exercises comparing finetuned large language models against traditional baselines. Through practical tutorials and replication materials, it demonstrates superior performance and ease of use, promoting broader adoption in psychology and related fields. The work emphasizes accessibility, reproducibility, and real-world applicability across diverse psychological inquiry.

Abstract

Given that natural language serves as the primary conduit for expressing thoughts and emotions, text analysis has become a key technique in psychological research. It enables the extraction of valuable insights from natural language, facilitating endeavors like personality traits assessment, mental health monitoring, and sentiment analysis in interpersonal communications. In text analysis, existing studies often resort to either human coding, which is time-consuming, using pre-built dictionaries, which often fails to cover all possible scenarios, or training models from scratch, which requires large amounts of labeled data. In this tutorial, we introduce the pretrain-finetune paradigm. The pretrain-finetune paradigm represents a transformative approach in text analysis and natural language processing. This paradigm distinguishes itself through the use of large pretrained language models, demonstrating remarkable efficiency in finetuning tasks, even with limited training data. This efficiency is especially beneficial for research in social sciences, where the number of annotated samples is often quite limited. Our tutorial offers a comprehensive introduction to the pretrain-finetune paradigm. We first delve into the fundamental concepts of pretraining and finetuning, followed by practical exercises using real-world applications. We demonstrate the application of the paradigm across various tasks, including multi-class classification and regression. Emphasizing its efficacy and user-friendliness, the tutorial aims to encourage broader adoption of this paradigm. To this end, we have provided open access to all our code and datasets. The tutorial is highly beneficial across various psychology disciplines, providing a comprehensive guide to employing text analysis in diverse research settings.

Figures (6)

• Figure 1: High-level illustration of the pretraining (left) and finetuning (right) workflows. Pretraining happens once and is mostly done by large corporations such as Google, Meta, Apple and Microsoft. Finetuning happens whenever a researcher needs to use a pretrained model on a specific task, such as personality classification.
• Figure 2: Illustration of the BERT model's self-attention mechanism.
• Figure 3: Illustration of the BERT model's masked language modeling.
• Figure 4: Illustration of the finetuning process. Same as with pretraining, the model takes in a text input and generates an encoding for each token. What differentiates finetuning from pretraining is that it takes a particular token representation to make task-specific predictions.
• Figure 5: Given an unstructured text, we transform it into one particular psychological underpinning through text analysis, specifically finetuning a large language model. In the provided example, the text is classified under the fourth topic, reducing use.