Recent Trends in Deep Learning Based Natural Language Processing

TL;DR

This comprehensive survey traces deep learning's impact on NLP from distributed word representations through contextual embeddings to Transformer-based models and memory-augmented systems. It covers CNNs, RNNs, and recursive architectures, highlighting how attention, pretraining, and generative approaches have shaped performance across POS tagging, parsing, NER, SRL, sentiment analysis, MT, QA, and dialogue. The authors emphasize unsupervised and semi-supervised learning, reinforcement learning for generation, and memory-augmented networks as pivotal future directions, with BERT-style contextual embeddings marking a performance milestone. The work underscores a trend toward parallelizable architectures, broader transfer learning, and integration of symbolic memory for deeper language understanding and multimodal applications.

Abstract

Deep learning methods employ multiple processing layers to learn hierarchical representations of data and have produced state-of-the-art results in many domains. Recently, a variety of model designs and methods have blossomed in the context of natural language processing (NLP). In this paper, we review significant deep learning related models and methods that have been employed for numerous NLP tasks and provide a walk-through of their evolution. We also summarize, compare and contrast the various models and put forward a detailed understanding of the past, present and future of deep learning in NLP.

Figures (21)

• Figure 1: Percentage of deep learning papers in ACL, EMNLP, EACL, NAACL over the last 6 years (long papers).
• Figure 2: Distributional vectors represented by a ${\bf D}$-dimensional vector where ${\bf D} << {\bf V}$, where ${\bf V}$ is size of Vocabulary. Figure Source: http://veredshwartz.blogspot.sg.
• Figure 3: Neural Language Model (Figure reproduced from bengio2003neural). $C(i)$ is the $i^{th}$ word embedding.
• Figure 4: Model for CBOW (Figure source: rong2014word2vec)
• Figure 5: CNN framework used to perform word wise class prediction (Figure source: collobert2008unified)