Efficient Few-shot Learning for Multi-label Classification of Scientific Documents with Many Classes

TL;DR

This work proposes FusionSent (Fusion-based Sentence Embedding Fine-tuning), an efficient and prompt-free approach for few-shot classification of scientific documents with many classes, and introduces a new dataset for multi-label classification of scientific documents.

Abstract

Scientific document classification is a critical task and often involves many classes. However, collecting human-labeled data for many classes is expensive and usually leads to label-scarce scenarios. Moreover, recent work has shown that sentence embedding model fine-tuning for few-shot classification is efficient, robust, and effective. In this work, we propose FusionSent (Fusion-based Sentence Embedding Fine-tuning), an efficient and prompt-free approach for few-shot classification of scientific documents with many classes. FusionSent uses available training examples and their respective label texts to contrastively fine-tune two different sentence embedding models. Afterward, the parameters of both fine-tuned models are fused to combine the complementary knowledge from the separate fine-tuning steps into a single model. Finally, the resulting sentence embedding model is frozen to embed the training instances, which are then used as input features to train a classification head. Our experiments show that FusionSent significantly outperforms strong baselines by an average of $6.0$ $F_{1}$ points across multiple scientific document classification datasets. In addition, we introduce a new dataset for multi-label classification of scientific documents, which contains 203,961 scientific articles and 130 classes from the arXiv category taxonomy. Code and data are available at https://github.com/sebischair/FusionSent.

Figures (3)

• Figure 1: The training process of FusionSent comprises three steps: (1) Fine-tune two different sentence embedding models from the same Pre-trained Language Model (PLM), with parameters $\theta_{1}$, $\theta_{2}$ respectively. $\theta_{1}$ is fine-tuned on pairs of training sentences using cosine similarity loss and $\theta_{2}$ is fine-tuned on pairs of training sentences and their corresponding label texts, using contrastive loss. Label texts can consist of simple label/class names or of more extensive texts that semantically describe the meaning of a label/class. (2) Merge parameter sets $\theta_{1}$, $\theta_{2}$ into $\theta_{3}$ using Spherical Linear Interpolation (SLERP). (3) Freeze $\theta_{3}$ to embed the training sentences, which are then used as input features to train a classification head.
• Figure 2: Number of papers in each category of the arXiv dataset.
• Figure 3: FusionSent micro $F_{1}$ scores for few-shot classification on 8 different datasets using either extensive label descriptions or simple label names. We report the average score over the random training splits of each dataset using $|N|=8$ training examples per class.