Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Dependency Parsing with the Structuralized Prompt Template

Keunha Kim, Youngjoong Ko

TL;DR

This work introduces Structuralized Prompt Template Dependency Parsing (SPT-DP), a method that performs dependency parsing solely with a pre-trained encoder by encoding tree structure into text via a structured prompt template. By representing dependencies as text and using soft prompts for head indices and relation labels, the approach reframes parsing as a text-to-text task compatible with MLM-style or autoregressive objectives, minimizing architectural overhead. Empirical results show competitive PTB performance (UAS ~96.95, LAS ~95.89) and state-of-the-art cross-lingual results on UD 2.2 in several languages, with strong performance on Sejong Korean using lighter models and significant inference speed advantages. The method scales across encoder- and decoder-based models, supports cross-lingual transfer, and highlights the importance of prompts, especially the explicit index prompt, for effective dependency parsing.

Abstract

Dependency parsing is a fundamental task in natural language processing (NLP), aiming to identify syntactic dependencies and construct a syntactic tree for a given sentence. Traditional dependency parsing models typically construct embeddings and utilize additional layers for prediction. We propose a novel dependency parsing method that relies solely on an encoder model with a text-to-text training approach. To facilitate this, we introduce a structured prompt template that effectively captures the structural information of dependency trees. Our experimental results demonstrate that the proposed method achieves outstanding performance compared to traditional models, despite relying solely on a pre-trained model. Furthermore, this method is highly adaptable to various pre-trained models across different target languages and training environments, allowing easy integration of task-specific features.

Dependency Parsing with the Structuralized Prompt Template

TL;DR

This work introduces Structuralized Prompt Template Dependency Parsing (SPT-DP), a method that performs dependency parsing solely with a pre-trained encoder by encoding tree structure into text via a structured prompt template. By representing dependencies as text and using soft prompts for head indices and relation labels, the approach reframes parsing as a text-to-text task compatible with MLM-style or autoregressive objectives, minimizing architectural overhead. Empirical results show competitive PTB performance (UAS ~96.95, LAS ~95.89) and state-of-the-art cross-lingual results on UD 2.2 in several languages, with strong performance on Sejong Korean using lighter models and significant inference speed advantages. The method scales across encoder- and decoder-based models, supports cross-lingual transfer, and highlights the importance of prompts, especially the explicit index prompt, for effective dependency parsing.

Abstract

Dependency parsing is a fundamental task in natural language processing (NLP), aiming to identify syntactic dependencies and construct a syntactic tree for a given sentence. Traditional dependency parsing models typically construct embeddings and utilize additional layers for prediction. We propose a novel dependency parsing method that relies solely on an encoder model with a text-to-text training approach. To facilitate this, we introduce a structured prompt template that effectively captures the structural information of dependency trees. Our experimental results demonstrate that the proposed method achieves outstanding performance compared to traditional models, despite relying solely on a pre-trained model. Furthermore, this method is highly adaptable to various pre-trained models across different target languages and training environments, allowing easy integration of task-specific features.

Paper Structure

This paper contains 34 sections, 9 equations, 5 figures, 10 tables.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Dependency Parsing
  4. Prompt based tuning
  5. Pre-trained Language Model
  6. Structuralized Prompt Template
  7. Dependency Parsing with Text Representation
  8. Prediction Task using Soft Prompts $P_{MASK}$ : [HEAD] and [DEP]
  9. Encoder-only Models
  10. Decoder-based Models
  11. Prompt-based Training
  12. Experiments
  13. Datasets
  14. PTB
  15. UD 2.2
  16. ...and 19 more sections

Figures (5)

  • Figure 1: A figure shows that pipeline of traditional dependency parsing
  • Figure 2: Dependency Parsing Example
  • Figure 3: Overview of the Structuralized Prompt Template based Dependency Parsing (SPT-DP) method.
  • Figure 4: A figure of word level SPT example
  • Figure 5: A heatmap of Attention scores and cosine similarities in hidden layer