Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

VNLP: Turkish NLP Package

Meliksah Turker, Mehmet Erdi Ari, Aydin Han

Abstract

In this work, we present VNLP: the first dedicated, complete, open-source, well-documented, lightweight, production-ready, state-of-the-art Natural Language Processing (NLP) package for the Turkish language. It contains a wide variety of tools, ranging from the simplest tasks, such as sentence splitting and text normalization, to the more advanced ones, such as text and token classification models. Its token classification models are based on "Context Model", a novel architecture that is both an encoder and an auto-regressive model. NLP tasks solved by VNLP models include but are not limited to Sentiment Analysis, Named Entity Recognition, Morphological Analysis \& Disambiguation and Part-of-Speech Tagging. Moreover, it comes with pre-trained word embeddings and corresponding SentencePiece Unigram tokenizers. VNLP has an open-source GitHub repository, ReadtheDocs documentation, PyPi package for convenient installation, Python and command-line API and a demo page to test all the functionality. Consequently, our main contribution is a complete, compact, easy-to-install and easy-to-use NLP package for Turkish.

VNLP: Turkish NLP Package

Abstract

In this work, we present VNLP: the first dedicated, complete, open-source, well-documented, lightweight, production-ready, state-of-the-art Natural Language Processing (NLP) package for the Turkish language. It contains a wide variety of tools, ranging from the simplest tasks, such as sentence splitting and text normalization, to the more advanced ones, such as text and token classification models. Its token classification models are based on "Context Model", a novel architecture that is both an encoder and an auto-regressive model. NLP tasks solved by VNLP models include but are not limited to Sentiment Analysis, Named Entity Recognition, Morphological Analysis \& Disambiguation and Part-of-Speech Tagging. Moreover, it comes with pre-trained word embeddings and corresponding SentencePiece Unigram tokenizers. VNLP has an open-source GitHub repository, ReadtheDocs documentation, PyPi package for convenient installation, Python and command-line API and a demo page to test all the functionality. Consequently, our main contribution is a complete, compact, easy-to-install and easy-to-use NLP package for Turkish.
Paper Structure (24 sections, 2 figures, 5 tables)

This paper contains 24 sections, 2 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Functionality & Models
  4. Sentence Splitter
  5. Normalizer
  6. Spelling Correction
  7. Deasciifier
  8. Number to Word
  9. Stopword Remover
  10. Static Method
  11. Dynamic Method
  12. SentencePiece Unigram Tokenizer
  13. Pre-trained Word Embeddings
  14. Word2Vec
  15. FastText
  16. ...and 9 more sections

Figures (2)

  • Figure 1: Context Model consists of 4 components. Word Model that processes the subword tokens in a word, Left Context Model that processes words in the left context left-to-right, Right Context Model that processes words in the right context right-to-left and Left Context Tag Model that processes the classified tags in the left context.
  • Figure 2: Dependency Parser produces arcs and labels to indicate the relations between words. Part-of-speech tags below are produced by Part-of-Speech Tagger.