Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Next Word Suggestion using Graph Neural Network

Abisha Thapa Magar, Anup Shakya

TL;DR

The paper tackles next-word prediction by encoding global context with Graph Convolutional Networks and leveraging local context with LSTMs. It builds a word co-occurrence graph from a custom Wikipedia corpus and trains a GCNL to obtain global-context embeddings and a Many-to-One LSTM on 1–10 gram sequences derived from these embeddings. The graph convolution update is $h^{i+1} = \sigma(W^{i} h^{i} A)$, aggregating information from one-hop neighborhoods. Results indicate context embeddings improve over a random-embedding baseline but generalization remains limited due to small data and resource constraints, motivating future work with more GNN variants and embedding analyses.

Abstract

Language Modeling is a prevalent task in Natural Language Processing. The currently existing most recent and most successful language models often tend to build a massive model with billions of parameters, feed in a tremendous amount of text data, and train with enormous computation resources which require millions of dollars. In this project, we aim to address an important sub-task in language modeling, i.e., context embedding. We propose an approach to exploit the Graph Convolution operation in GNNs to encode the context and use it in coalition with LSTMs to predict the next word given a local context of preceding words. We test this on the custom Wikipedia text corpus using a very limited amount of resources and show that this approach works fairly well to predict the next word.

Next Word Suggestion using Graph Neural Network

TL;DR

The paper tackles next-word prediction by encoding global context with Graph Convolutional Networks and leveraging local context with LSTMs. It builds a word co-occurrence graph from a custom Wikipedia corpus and trains a GCNL to obtain global-context embeddings and a Many-to-One LSTM on 1–10 gram sequences derived from these embeddings. The graph convolution update is , aggregating information from one-hop neighborhoods. Results indicate context embeddings improve over a random-embedding baseline but generalization remains limited due to small data and resource constraints, motivating future work with more GNN variants and embedding analyses.

Abstract

Language Modeling is a prevalent task in Natural Language Processing. The currently existing most recent and most successful language models often tend to build a massive model with billions of parameters, feed in a tremendous amount of text data, and train with enormous computation resources which require millions of dollars. In this project, we aim to address an important sub-task in language modeling, i.e., context embedding. We propose an approach to exploit the Graph Convolution operation in GNNs to encode the context and use it in coalition with LSTMs to predict the next word given a local context of preceding words. We test this on the custom Wikipedia text corpus using a very limited amount of resources and show that this approach works fairly well to predict the next word.
Paper Structure (15 sections, 2 equations, 5 figures, 2 tables)

This paper contains 15 sections, 2 equations, 5 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Proposed Approach
  3. Context Embeddings
  4. Global Context
  5. Local Context
  6. Graph Convolution
  7. Recurrent Neural Networks(RNN)
  8. Long Short Term Memory (LSTM) Networks
  9. Many to One LSTM
  10. Experiments and Results
  11. Dataset
  12. Baseline Method
  13. Setup
  14. Results
  15. Discussion

Figures (5)

  • Figure 1: Illustrating the graph structure constructed from the toy example. Each node represents the unique words in the corpus and links represent the co-occurrence of the words.
  • Figure 2: Illustrating the convolution operation. Left image shows the 2D convolution in Convolutional Neural Networks. Right image shows the convolution in Graph Convolutional Networks [source: gnn_survey2019]
  • Figure 3: Working of Recurrent Neural Network [Source: rnn_article]
  • Figure 4: Illustrating the overall architecture of the proposed approach for predicting the next word.
  • Figure 5: Bar chart showing the accuracy of next word prediction.