Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Improving Pre-trained Language Model Sensitivity via Mask Specific losses: A case study on Biomedical NER

Micheal Abaho, Danushka Bollegala, Gary Leeming, Dan Joyce, Iain E Buchan

TL;DR

To address insensitive fine-tuning, Mask Specific Language Modeling (MSLM) is proposed, an approach that efficiently acquires target domain knowledge by appropriately weighting the importance of domain-specific terms (DS-terms) during fine-tuning.

Abstract

Adapting language models (LMs) to novel domains is often achieved through fine-tuning a pre-trained LM (PLM) on domain-specific data. Fine-tuning introduces new knowledge into an LM, enabling it to comprehend and efficiently perform a target domain task. Fine-tuning can however be inadvertently insensitive if it ignores the wide array of disparities (e.g in word meaning) between source and target domains. For instance, words such as chronic and pressure may be treated lightly in social conversations, however, clinically, these words are usually an expression of concern. To address insensitive fine-tuning, we propose Mask Specific Language Modeling (MSLM), an approach that efficiently acquires target domain knowledge by appropriately weighting the importance of domain-specific terms (DS-terms) during fine-tuning. MSLM jointly masks DS-terms and generic words, then learns mask-specific losses by ensuring LMs incur larger penalties for inaccurately predicting DS-terms compared to generic words. Results of our analysis show that MSLM improves LMs sensitivity and detection of DS-terms. We empirically show that an optimal masking rate not only depends on the LM, but also on the dataset and the length of sequences. Our proposed masking strategy outperforms advanced masking strategies such as span- and PMI-based masking.

Improving Pre-trained Language Model Sensitivity via Mask Specific losses: A case study on Biomedical NER

TL;DR

To address insensitive fine-tuning, Mask Specific Language Modeling (MSLM) is proposed, an approach that efficiently acquires target domain knowledge by appropriately weighting the importance of domain-specific terms (DS-terms) during fine-tuning.

Abstract

Adapting language models (LMs) to novel domains is often achieved through fine-tuning a pre-trained LM (PLM) on domain-specific data. Fine-tuning introduces new knowledge into an LM, enabling it to comprehend and efficiently perform a target domain task. Fine-tuning can however be inadvertently insensitive if it ignores the wide array of disparities (e.g in word meaning) between source and target domains. For instance, words such as chronic and pressure may be treated lightly in social conversations, however, clinically, these words are usually an expression of concern. To address insensitive fine-tuning, we propose Mask Specific Language Modeling (MSLM), an approach that efficiently acquires target domain knowledge by appropriately weighting the importance of domain-specific terms (DS-terms) during fine-tuning. MSLM jointly masks DS-terms and generic words, then learns mask-specific losses by ensuring LMs incur larger penalties for inaccurately predicting DS-terms compared to generic words. Results of our analysis show that MSLM improves LMs sensitivity and detection of DS-terms. We empirically show that an optimal masking rate not only depends on the LM, but also on the dataset and the length of sequences. Our proposed masking strategy outperforms advanced masking strategies such as span- and PMI-based masking.
Paper Structure (34 sections, 10 equations, 7 figures, 14 tables, 2 algorithms)

This paper contains 34 sections, 10 equations, 7 figures, 14 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Mask-Specific Language Modeling
  3. Masking
  4. Mask specific losses
  5. Entity detection and Classification
  6. Task formulation:
  7. Entity Linking/Classification loss:
  8. Model loss:
  9. Experiments
  10. Datasets:
  11. Metrics:
  12. Setup:
  13. Implementation details:
  14. Sensitivity towards DS-terms
  15. Is DSFT destructive?
  16. ...and 19 more sections

Figures (7)

  • Figure 1: Joint ELM-BLM masking of tokens in an input sequence.
  • Figure 2: Visualization of the confidence score with which different DS-terms belonging to different outcome (within the EBM-NLP dataset) are predicted. The color intensity increases with the confidence score.
  • Figure 3: Downstream NER F1 performance of the vanilla and the MSLM-fine-tuned models (i.e. DSFT models). ELM and BLM rates used in §\ref{['sec:sensitivity_dst']} are maintained. More plots in Appendix \ref{['sec:dsft_destructive']}.
  • Figure 4: Test Exact match (EM) scores of varying ELM and BLM rates when two MSLM-fine-tuned models (MSLM_biobert and MSLM_PubMedBERT) are evaluated on the datasets.
  • Figure 5: Comparing performance of other masking strategies across various rates with the best performance of our proposed Joint ELM-BLM. Results of BioBERT (left) and PubMedBERT (right) evaluated on BC2GM and BC5CDR-chem and hatch the bars with best scores.
  • ...and 2 more figures