Named Entity Recognition Under Domain Shift via Metric Learning for Life Sciences

TL;DR

This work tackles the challenge of named entity recognition under domain shift from biomedical to chemical texts, where large models underperform due to domain-specific nuances. It introduces a two-stage framework that first pretrains on a biomedical source with event-derived auxiliary data to shape a grouped source feature space, then finetunes on a chemical target with pseudo-labeling and a refined multi-similarity loss to keep source and target representations separate. The approach achieves up to 5% absolute improvement over baselines across twelve source-target scenarios, with ablations confirming the value of event-based embeddings, auxiliary similarity, and pseudo-label discrimination. By enabling robust cross-domain NER in life sciences, the method enhances information extraction capabilities for chemical domains using biomedical resources, with evidence of generalization to other domain pairs.

Abstract

Named entity recognition is a key component of Information Extraction (IE), particularly in scientific domains such as biomedicine and chemistry, where large language models (LLMs), e.g., ChatGPT, fall short. We investigate the applicability of transfer learning for enhancing a named entity recognition model trained in the biomedical domain (the source domain) to be used in the chemical domain (the target domain). A common practice for training such a model in a few-shot learning setting is to pretrain the model on the labeled source data, and then, to finetune it on a hand-full of labeled target examples. In our experiments, we observed that such a model is prone to mislabeling the source entities, which can often appear in the text, as the target entities. To alleviate this problem, we propose a model to transfer the knowledge from the source domain to the target domain, but, at the same time, to project the source entities and target entities into separate regions of the feature space. This diminishes the risk of mislabeling the source entities as the target entities. Our model consists of two stages: 1) entity grouping in the source domain, which incorporates knowledge from annotated events to establish relations between entities, and 2) entity discrimination in the target domain, which relies on pseudo labeling and contrastive learning to enhance discrimination between the entities in the two domains. We conduct our extensive experiments across three source and three target datasets, demonstrating that our method outperforms the baselines by up to 5% absolute value.

Figures (5)

• Figure 1: A test example in the chemical domain. The words marked with blue indicators are chemical entities, and the words marked with red and orange indicators are biomedical entities. The entities in red are mislabeled by a few-shot model as chemical entities.
• Figure 2: Overview of proposed entity grouping and entity discrimination frameworks. Entity grouping on the source domain is shown in the upper part. Based on event annotations, a set of event embeddings is constructed under two paradigms. Afterward, pairwise auxiliary similarity scores are calculated according to argument embeddings. Extended multi-similarity loss concerning four types of similarities, combined with cross-entropy loss, are jointly learned during pretraining. Entity discrimination on the target domain is shown in the lower part. Pseudo labels are formed by the named entity recognition model pretrained in the source domain, and in contrast to annotated labels, a multi-similarity loss is injected into finetuning.
• Figure 3: Davies-Bouldin index criterion of clusters. For baseline and ED-concerned settings, pseudo entities are included and viewed in the same cluster as Disease.
• Figure 4: Components of concatenation based event embeddings. Arguments of events, along with event type, are encoded by an off-the-shelf model and concatenated afterwards. For nested events as arguments, we fill in compressed event embeddings recursively.
• Figure 5: t-SNE visualization of entities in the test corpus of BC5CDR. Pseudo is labeled by model pretrained on source task, Disease and Chemical are gold-standard annotations. BERT represents vanilla BERT model without pretraining or finetuning, and all the settings are same as Main Results.