Positional Attention for Efficient BERT-Based Named Entity Recognition

TL;DR

The paper tackles the high training cost of adapting BERT for domain-specific Named Entity Recognition (NER) by introducing a cost-efficient framework that integrates positional attention with pre-trained BERT parameters. By leveraging WordPiece embeddings, a [CLS] token, and a [PAD] scheme for out-of-vocabulary words, the approach fine-tunes BERT with lightweight modifications and a modest training budget, achieving an F1 score of 0.8143 on a Groningen Meaning Bank-derived Kaggle dataset. Compared to CRF, Transformer, and BiLSTM baselines, the proposed model demonstrates competitive accuracy with relatively few epochs (best at 12) and a small learning rate (3e-5), highlighting the practicality of pre-trained parameter customization for NER. The work suggests a practical pathway to deploy BERT-based NER in real-world settings and points toward future extensions to multi-modal and localized attention for scalable, efficient sequence labeling.

Abstract

This paper presents a framework for Named Entity Recognition (NER) leveraging the Bidirectional Encoder Representations from Transformers (BERT) model in natural language processing (NLP). NER is a fundamental task in NLP with broad applicability across downstream applications. While BERT has established itself as a state-of-the-art model for entity recognition, fine-tuning it from scratch for each new application is computationally expensive and time-consuming. To address this, we propose a cost-efficient approach that integrates positional attention mechanisms into the entity recognition process and enables effective customization using pre-trained parameters. The framework is evaluated on a Kaggle dataset derived from the Groningen Meaning Bank corpus and achieves strong performance with fewer training epochs. This work contributes to the field by offering a practical solution for reducing the training cost of BERT-based NER systems while maintaining high accuracy.

Figures (3)

• Figure 1: BERT’s 12 layers of two-part encoders
• Figure 2: Loss function of the training process
• Figure 3: Confidence Matrix for Transformer