Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

AF Adapter: Continual Pretraining for Building Chinese Biomedical Language Model

Yongyu Yan, Kui Xue, Xiaoming Shi, Qi Ye, Jingping Liu, Tong Ruan

TL;DR

This work tackles catastrophic forgetting in continual pretraining for domain-specific language models in Chinese biomedicine. It introduces AF Adapter, a layer-extending adapter that adds a small set of domain-specific attention heads and FFN units while freezing the original general-domain parameters, enabling continual pretraining without deepening the network. Applied to a Chinese biomedical RoBERTa, AF Adapter RoBERTa achieves state-of-the-art average performance on the CBLUE benchmark and reduces forgetting by a notable margin compared with fine-tuning and other baselines, using only about 17% of trainable parameters. The results demonstrate a memory-efficient pathway to domain adaptation with robust convergence and stability, suggesting broad applicability of layer-extending adapters for domain-specific pretraining.

Abstract

Continual pretraining is a popular way of building a domain-specific pretrained language model from a general-domain language model. In spite of its high efficiency, continual pretraining suffers from catastrophic forgetting, which may harm the model's performance in downstream tasks. To alleviate the issue, in this paper, we propose a continual pretraining method for the BERT-based model, named Attention-FFN Adapter. Its main idea is to introduce a small number of attention heads and hidden units inside each self-attention layer and feed-forward network. Furthermore, we train a domain-specific language model named AF Adapter based RoBERTa for the Chinese biomedical domain. In experiments, models are applied to downstream tasks for evaluation. The results demonstrate that with only about 17% of model parameters trained, AF Adapter achieves 0.6%, 2% gain in performance on average, compared to strong baselines. Further experimental results show that our method alleviates the catastrophic forgetting problem by 11% compared to the fine-tuning method.

AF Adapter: Continual Pretraining for Building Chinese Biomedical Language Model

TL;DR

This work tackles catastrophic forgetting in continual pretraining for domain-specific language models in Chinese biomedicine. It introduces AF Adapter, a layer-extending adapter that adds a small set of domain-specific attention heads and FFN units while freezing the original general-domain parameters, enabling continual pretraining without deepening the network. Applied to a Chinese biomedical RoBERTa, AF Adapter RoBERTa achieves state-of-the-art average performance on the CBLUE benchmark and reduces forgetting by a notable margin compared with fine-tuning and other baselines, using only about 17% of trainable parameters. The results demonstrate a memory-efficient pathway to domain adaptation with robust convergence and stability, suggesting broad applicability of layer-extending adapters for domain-specific pretraining.

Abstract

Continual pretraining is a popular way of building a domain-specific pretrained language model from a general-domain language model. In spite of its high efficiency, continual pretraining suffers from catastrophic forgetting, which may harm the model's performance in downstream tasks. To alleviate the issue, in this paper, we propose a continual pretraining method for the BERT-based model, named Attention-FFN Adapter. Its main idea is to introduce a small number of attention heads and hidden units inside each self-attention layer and feed-forward network. Furthermore, we train a domain-specific language model named AF Adapter based RoBERTa for the Chinese biomedical domain. In experiments, models are applied to downstream tasks for evaluation. The results demonstrate that with only about 17% of model parameters trained, AF Adapter achieves 0.6%, 2% gain in performance on average, compared to strong baselines. Further experimental results show that our method alleviates the catastrophic forgetting problem by 11% compared to the fine-tuning method.
Paper Structure (17 sections, 7 equations, 5 figures, 5 tables)

This paper contains 17 sections, 7 equations, 5 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Language Modeling
  4. Domain-Specific Pretraining
  5. Preliminaries
  6. BERT
  7. Methodology
  8. Model Architecture
  9. Pretraining Details
  10. Experiments
  11. Pretraining Data and Settings
  12. Evaluation Tasks
  13. Main Results
  14. Comparison of PLMs
  15. Comparison of Pretraining Techniques
  16. ...and 2 more sections

Figures (5)

  • Figure 1: Outputs of RoBERTa, fine-tuned RoBERTa, and AF Adapter based RoBERTa for two input items.
  • Figure 2: The illustration of AF Adapter. AF Adapter introduces a small number of additional heads and hidden units to each self-attention layer and feed-forward network of the BERT-based model. The added parameters of each layer are independent. The heads and hidden units with orange color are trainable, and all parameters with grey color from the original model are frozen.
  • Figure 3: The attention layer architecture of AF Adapter.
  • Figure 4: The feed-forward network architecture of AF Adapter.
  • Figure 5: Convergence comparison of Fine-Tuning, FL-Tuning, LoRA and AF Adapter. The curves in dark color are obtained by smoothing the loss curves (light color). The smoothing function is $\alpha * previous\_smoothed\_value +(1-\alpha)*current\_value$, where $\alpha=0.6$ is the smooth weight.