Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

D-Nikud: Enhancing Hebrew Diacritization with LSTM and Pretrained Models

Adi Rosenthal, Nadav Shaked

TL;DR

Hebrew diacritization is critical for NLP and TTS but remains challenging due to morphology and data diversity. D-Nikud fuses TavBERT-based embeddings with Bi-LSTM sequence tagging to predict Nikud, Dagesh, and Sin, trained on Nakdimon and Dicta data. The paper introduces a three-class letter representation, full-script reconciliation, and a practical training-prediction pipeline. Experiments show state-of-the-art performance across multiple Hebrew genres and faster inference than competing systems, highlighting potential for real-time diacritization in modern Hebrew text including gender forms.

Abstract

D-Nikud, a novel approach to Hebrew diacritization that integrates the strengths of LSTM networks and BERT-based (transformer) pre-trained model. Inspired by the methodologies employed in Nakdimon, we integrate it with the TavBERT pre-trained model, our system incorporates advanced architectural choices and diverse training data. Our experiments showcase state-of-the-art results on several benchmark datasets, with a particular emphasis on modern texts and more specified diacritization like gender.

D-Nikud: Enhancing Hebrew Diacritization with LSTM and Pretrained Models

TL;DR

Hebrew diacritization is critical for NLP and TTS but remains challenging due to morphology and data diversity. D-Nikud fuses TavBERT-based embeddings with Bi-LSTM sequence tagging to predict Nikud, Dagesh, and Sin, trained on Nakdimon and Dicta data. The paper introduces a three-class letter representation, full-script reconciliation, and a practical training-prediction pipeline. Experiments show state-of-the-art performance across multiple Hebrew genres and faster inference than competing systems, highlighting potential for real-time diacritization in modern Hebrew text including gender forms.

Abstract

D-Nikud, a novel approach to Hebrew diacritization that integrates the strengths of LSTM networks and BERT-based (transformer) pre-trained model. Inspired by the methodologies employed in Nakdimon, we integrate it with the TavBERT pre-trained model, our system incorporates advanced architectural choices and diverse training data. Our experiments showcase state-of-the-art results on several benchmark datasets, with a particular emphasis on modern texts and more specified diacritization like gender.
Paper Structure (26 sections, 11 figures, 5 tables)

This paper contains 26 sections, 11 figures, 5 tables.

Table of Contents

  1. Abstract
  2. Introduction
  3. Data
  4. Data Sources
  5. Full Script Reconciliation
  6. Character Representations
  7. Data processing
  8. Read data and combine sentences
  9. Data Splitting
  10. Data Tokenization
  11. D-Nikud Model
  12. Training and Prediction Flow for D-Nikud Model
  13. Training:
  14. Prediction:
  15. Experiments
  16. ...and 11 more sections

Figures (11)

  • Figure 1: Snapshot of the file and token counts in modern texts by genre. Note that each letter is considered as a token
  • Figure 1: Representation of the D-Nikud Letter class.
  • Figure 2: D-Nikud model architecture.
  • Figure 2: D-Nikud test diacritization accuracy across different Hebrew genres (%).
  • Figure 3: Train set loss by steps of 100 mini-batches plot.
  • ...and 6 more figures