Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Dependency Parsing is More Parameter-Efficient with Normalization

Paolo Gajo, Domenic Rosati, Hassan Sajjad, Alberto Barrón-Cedeño

TL;DR

This work addresses overparameterization in biaffine-based dependency parsing caused by unnormalized outputs that inflate score variance. It proposes explicit score normalization, inspired by Transformer scaling, to stabilize pre-softmax scores (akin to applying $a=1/\sqrt{d_k}$) and shows that layer depth can provide a similar normalization effect via implicit regularization. Through extensive experiments on semantic and syntactic DP across multilingual and non-linguistic LGI datasets, normalization achieves state-of-the-art performance with far fewer parameters and samples, sometimes beating prior SOTA with a single BiLSTM layer. The findings suggest that many DP models are overparameterized and that normalization substantially improves parameter efficiency and convergence, with broad implications for graph inference tasks and practical NLP systems.

Abstract

Dependency parsing is the task of inferring natural language structure, often approached by modeling word interactions via attention through biaffine scoring. This mechanism works like self-attention in Transformers, where scores are calculated for every pair of words in a sentence. However, unlike Transformer attention, biaffine scoring does not use normalization prior to taking the softmax of the scores. In this paper, we provide theoretical evidence and empirical results revealing that a lack of normalization necessarily results in overparameterized parser models, where the extra parameters compensate for the sharp softmax outputs produced by high variance inputs to the biaffine scoring function. We argue that biaffine scoring can be made substantially more efficient by performing score normalization. We conduct experiments on semantic and syntactic dependency parsing in multiple languages, along with latent graph inference on non-linguistic data, using various settings of a $k$-hop parser. We train $N$-layer stacked BiLSTMs and evaluate the parser's performance with and without normalizing biaffine scores. Normalizing allows us to achieve state-of-the-art performance with fewer samples and trainable parameters. Code: https://github.com/paolo-gajo/EfficientSDP

Dependency Parsing is More Parameter-Efficient with Normalization

TL;DR

This work addresses overparameterization in biaffine-based dependency parsing caused by unnormalized outputs that inflate score variance. It proposes explicit score normalization, inspired by Transformer scaling, to stabilize pre-softmax scores (akin to applying ) and shows that layer depth can provide a similar normalization effect via implicit regularization. Through extensive experiments on semantic and syntactic DP across multilingual and non-linguistic LGI datasets, normalization achieves state-of-the-art performance with far fewer parameters and samples, sometimes beating prior SOTA with a single BiLSTM layer. The findings suggest that many DP models are overparameterized and that normalization substantially improves parameter efficiency and convergence, with broad implications for graph inference tasks and practical NLP systems.

Abstract

Dependency parsing is the task of inferring natural language structure, often approached by modeling word interactions via attention through biaffine scoring. This mechanism works like self-attention in Transformers, where scores are calculated for every pair of words in a sentence. However, unlike Transformer attention, biaffine scoring does not use normalization prior to taking the softmax of the scores. In this paper, we provide theoretical evidence and empirical results revealing that a lack of normalization necessarily results in overparameterized parser models, where the extra parameters compensate for the sharp softmax outputs produced by high variance inputs to the biaffine scoring function. We argue that biaffine scoring can be made substantially more efficient by performing score normalization. We conduct experiments on semantic and syntactic dependency parsing in multiple languages, along with latent graph inference on non-linguistic data, using various settings of a -hop parser. We train -layer stacked BiLSTMs and evaluate the parser's performance with and without normalizing biaffine scores. Normalizing allows us to achieve state-of-the-art performance with fewer samples and trainable parameters. Code: https://github.com/paolo-gajo/EfficientSDP

Paper Structure

This paper contains 22 sections, 17 equations, 8 figures, 13 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Layer depth can compensate for normalization
  4. Experimental setting
  5. Data
  6. Model
  7. Hyperparameters
  8. Evaluation
  9. Results and discussion
  10. Conclusions
  11. Non-English data results
  12. Non-linguistic results
  13. GAT results
  14. Ablations
  15. Tagger
  16. ...and 7 more sections

Figures (8)

  • Figure 1: Effective rank $\rho(W)$ reduces over training epochs as we increase $N$ of BiLSTM layers.
  • Figure 2: Dependency parsing diagram. Dashed components are the targets of ablation experiments.
  • Figure 3: Micro-F$_1$ (SemDP) and LAS (SynDP) vs $N \in \{0, 1, 2, 3\}$ at 2$k$ training steps. Red = norm; blue = raw. $^*$Performance increase with normalization is statistically significant ($p < 0.01$).
  • Figure 4: Test performance on SciERC (micro-averaged F$_1$-measure vs number of training steps). The $p$-values indicate greater performance with normalization (one-tailed Wilcoxon signed-rank test).
  • Figure 5: Extended parser architecture, featuring $L_\gamma \in \{0, 1, 2, 3\}$ pairs of biaffine and GAT layers.
  • ...and 3 more figures

Theorems & Definitions (2)

  • Claim 1: Monotonic Increase of Output Variance with Rank
  • proof