Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Acquiring Bidirectionality via Large and Small Language Models

Takumi Goto, Hiroyoshi Nagao, Yuta Koreeda

TL;DR

This work tackles the lack of bidirectionality in large unidirectional language models for token-level classification. It introduces a lightweight backward LM trained separately and concatenates its token representations with a forward LM, producing pseudo bidirectional token features without modifying the large forward model. Across chunking, POS, and NER tasks, especially CoNLL2003-NER and rare-domain/few-shot settings, the approach yields significant performance gains (up to 10+ F1 points) and demonstrates practical utility for leveraging large UniLMs as bidirectional encoders. The method is simple, reusable across different backbones, and accompanied by open-source training code, offering a scalable path to improve token-level NLP with existing models while highlighting computation considerations and scope limitations.

Abstract

Using token representation from bidirectional language models (LMs) such as BERT is still a widely used approach for token-classification tasks. Even though there exist much larger unidirectional LMs such as Llama-2, they are rarely used to replace the token representation of bidirectional LMs. In this work, we hypothesize that their lack of bidirectionality is keeping them behind. To that end, we propose to newly train a small backward LM and concatenate its representations to those of existing LM for downstream tasks. Through experiments in named entity recognition, we demonstrate that introducing backward model improves the benchmark performance more than 10 points. Furthermore, we show that the proposed method is especially effective for rare domains and in few-shot learning settings.

Acquiring Bidirectionality via Large and Small Language Models

TL;DR

This work tackles the lack of bidirectionality in large unidirectional language models for token-level classification. It introduces a lightweight backward LM trained separately and concatenates its token representations with a forward LM, producing pseudo bidirectional token features without modifying the large forward model. Across chunking, POS, and NER tasks, especially CoNLL2003-NER and rare-domain/few-shot settings, the approach yields significant performance gains (up to 10+ F1 points) and demonstrates practical utility for leveraging large UniLMs as bidirectional encoders. The method is simple, reusable across different backbones, and accompanied by open-source training code, offering a scalable path to improve token-level NLP with existing models while highlighting computation considerations and scope limitations.

Abstract

Using token representation from bidirectional language models (LMs) such as BERT is still a widely used approach for token-classification tasks. Even though there exist much larger unidirectional LMs such as Llama-2, they are rarely used to replace the token representation of bidirectional LMs. In this work, we hypothesize that their lack of bidirectionality is keeping them behind. To that end, we propose to newly train a small backward LM and concatenate its representations to those of existing LM for downstream tasks. Through experiments in named entity recognition, we demonstrate that introducing backward model improves the benchmark performance more than 10 points. Furthermore, we show that the proposed method is especially effective for rare domains and in few-shot learning settings.
Paper Structure (19 sections, 3 equations, 2 figures, 2 tables)

This paper contains 19 sections, 3 equations, 2 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Proposed Method
  3. Prerequisite
  4. Utilization of the Bidirectional Language Model
  5. Experiments
  6. Training Backward LM
  7. Downstream Tasks
  8. Few-shot Setting
  9. Results
  10. Full Dataset Setting
  11. Few-shot Setting
  12. Case Study
  13. Related Work
  14. Conclusion
  15. Limited Scope of Tasks and Conditions
  16. ...and 4 more sections

Figures (2)

  • Figure 1: The performance of UniLMs ("Forward") and the proposed concatenated LMs ("Concat") against a BiLM (BERT)
  • Figure 2: Few-shot setting results on CoNLL-2003 test set. In the $x$-axis, the number of training examples is represented by multiplication of the number of entities ($=4$) and $K$.