Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Binary Token-Level Classification with DeBERTa for All-Type MWE Identification: A Lightweight Approach with Linguistic Enhancement

Diego Rossini, Lonneke van der Plas

TL;DR

This work reframes all-type MWE identification as binary token-level START/END/INSIDE classification, implemented on a DeBERTa-v3-large backbone augmented with NP chunking and dependency-path features. By converting span annotations to token-level labels and reconstructing MWE spans with learned thresholds, the approach achieves a runtime-linear complexity $O(n)$ versus $O(n^2)$ for span enumeration, enabling efficient reasoning over long sequences. Across CoAM and STREUSLE, the method delivers state-of-the-art F1 scores (notably 69.8% on CoAM and 78.9% on STREUSLE) while using dramatically fewer parameters than large LLMs, and shows particular strength in discontinuous MWEs when combined with linguistic cues and data augmentation. The results highlight that carefully designed smaller models with targeted linguistic information can surpass massive LLMs on structured NLP tasks and offer practical benefits for resource-constrained deployments, albeit with limitations in dataset scale, language coverage, and typed MWE categorization.

Abstract

We present a comprehensive approach for multiword expression (MWE) identification that combines binary token-level classification, linguistic feature integration, and data augmentation. Our DeBERTa-v3-large model achieves 69.8% F1 on the CoAM dataset, surpassing the best results (Qwen-72B, 57.8% F1) on this dataset by 12 points while using 165x fewer parameters. We achieve this performance by (1) reformulating detection as binary token-level START/END/INSIDE classification rather than span-based prediction, (2) incorporating NP chunking and dependency features that help discontinuous and NOUN-type MWEs identification, and (3) applying oversampling that addresses severe class imbalance in the training data. We confirm the generalization of our method on the STREUSLE dataset, achieving 78.9% F1. These results demonstrate that carefully designed smaller models can substantially outperform LLMs on structured NLP tasks, with important implications for resource-constrained deployments.

Binary Token-Level Classification with DeBERTa for All-Type MWE Identification: A Lightweight Approach with Linguistic Enhancement

TL;DR

This work reframes all-type MWE identification as binary token-level START/END/INSIDE classification, implemented on a DeBERTa-v3-large backbone augmented with NP chunking and dependency-path features. By converting span annotations to token-level labels and reconstructing MWE spans with learned thresholds, the approach achieves a runtime-linear complexity versus for span enumeration, enabling efficient reasoning over long sequences. Across CoAM and STREUSLE, the method delivers state-of-the-art F1 scores (notably 69.8% on CoAM and 78.9% on STREUSLE) while using dramatically fewer parameters than large LLMs, and shows particular strength in discontinuous MWEs when combined with linguistic cues and data augmentation. The results highlight that carefully designed smaller models with targeted linguistic information can surpass massive LLMs on structured NLP tasks and offer practical benefits for resource-constrained deployments, albeit with limitations in dataset scale, language coverage, and typed MWE categorization.

Abstract

We present a comprehensive approach for multiword expression (MWE) identification that combines binary token-level classification, linguistic feature integration, and data augmentation. Our DeBERTa-v3-large model achieves 69.8% F1 on the CoAM dataset, surpassing the best results (Qwen-72B, 57.8% F1) on this dataset by 12 points while using 165x fewer parameters. We achieve this performance by (1) reformulating detection as binary token-level START/END/INSIDE classification rather than span-based prediction, (2) incorporating NP chunking and dependency features that help discontinuous and NOUN-type MWEs identification, and (3) applying oversampling that addresses severe class imbalance in the training data. We confirm the generalization of our method on the STREUSLE dataset, achieving 78.9% F1. These results demonstrate that carefully designed smaller models can substantially outperform LLMs on structured NLP tasks, with important implications for resource-constrained deployments.
Paper Structure (20 sections, 1 figure, 6 tables)

This paper contains 20 sections, 1 figure, 6 tables.

Table of Contents

  1. Introduction
  2. Method
  3. Datasets
  4. Binary Token-Level Classification and MWE Reconstruction
  5. Linguistic Feature Integration
  6. Data Augmentation
  7. Experiments
  8. Experimental Design
  9. Setup
  10. Main Results
  11. Analysis by MWE Type and Continuity
  12. Conclusion
  13. Implementation and Performance Details
  14. Threshold Values and Hyperparameters
  15. Training Dynamics and Model Performance
  16. ...and 5 more sections

Figures (1)

  • Figure 1: Learning curves for DLT+lo: training loss decreases while development F1 plateaus at epoch 8 (approximately 0.74), triggering early stopping. The initial fluctuation (epochs 2-3) reflects the model adapting to the challenging task of distinguishing genuine MWEs from compositional phrases.