Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Gated Word-Character Recurrent Language Model

Yasumasa Miyamoto, Kyunghyun Cho

TL;DR

The paper tackles fixed vocabulary limitations in word-based language models by introducing a gated word-character RNN-LM that adaptively fuses word-looked-up vectors with character-derived representations. The gating mechanism computes a word-specific gate to combine two embeddings, enabling better handling of rare and out-of-vocabulary words while feeding into a standard two-layer LSTM language model. Empirical results across PTB, BBC, and IMDB show competitive perplexities, with adaptive gating offering robust performance without manual gate tuning; pre-training influences gating behavior and improves results in several settings. This approach demonstrates a practical, flexible method to enrich word representations using subword information in recurrent language modeling.

Abstract

We introduce a recurrent neural network language model (RNN-LM) with long short-term memory (LSTM) units that utilizes both character-level and word-level inputs. Our model has a gate that adaptively finds the optimal mixture of the character-level and word-level inputs. The gate creates the final vector representation of a word by combining two distinct representations of the word. The character-level inputs are converted into vector representations of words using a bidirectional LSTM. The word-level inputs are projected into another high-dimensional space by a word lookup table. The final vector representations of words are used in the LSTM language model which predicts the next word given all the preceding words. Our model with the gating mechanism effectively utilizes the character-level inputs for rare and out-of-vocabulary words and outperforms word-level language models on several English corpora.

Gated Word-Character Recurrent Language Model

TL;DR

The paper tackles fixed vocabulary limitations in word-based language models by introducing a gated word-character RNN-LM that adaptively fuses word-looked-up vectors with character-derived representations. The gating mechanism computes a word-specific gate to combine two embeddings, enabling better handling of rare and out-of-vocabulary words while feeding into a standard two-layer LSTM language model. Empirical results across PTB, BBC, and IMDB show competitive perplexities, with adaptive gating offering robust performance without manual gate tuning; pre-training influences gating behavior and improves results in several settings. This approach demonstrates a practical, flexible method to enrich word representations using subword information in recurrent language modeling.

Abstract

We introduce a recurrent neural network language model (RNN-LM) with long short-term memory (LSTM) units that utilizes both character-level and word-level inputs. Our model has a gate that adaptively finds the optimal mixture of the character-level and word-level inputs. The gate creates the final vector representation of a word by combining two distinct representations of the word. The character-level inputs are converted into vector representations of words using a bidirectional LSTM. The word-level inputs are projected into another high-dimensional space by a word lookup table. The final vector representations of words are used in the LSTM language model which predicts the next word given all the preceding words. Our model with the gating mechanism effectively utilizes the character-level inputs for rare and out-of-vocabulary words and outperforms word-level language models on several English corpora.

Paper Structure

This paper contains 10 sections, 7 equations, 2 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Model Description
  3. Experimental Settings
  4. Model Variations
  5. Datasets
  6. Pre-training
  7. Results and Discussion
  8. Perplexity
  9. Values of Word--Character Gate
  10. Conclusion

Figures (2)

  • Figure 1: The model architecture of the gated word-character recurrent language model. $w_t$ is an input word at $t$. $\mathbf{x}^{\text{word}}_{w_t}$ is a word vector stored in the word lookup table. $\mathbf{x}^{\text{char}}_{w_t}$ is a word vector derived from the character-level input. $g_{w_t}$ is a gating value of a word $w_t$. $\hat{w}_{t+1}$ is a prediction made at $t$.
  • Figure 2: A log-log plot of frequency ranks and gating values trained in the gated word & character models with/without pre-training.