Contextual LSTM (CLSTM) models for Large scale NLP tasks

TL;DR

CLSTM introduces topic-conditioned LSTM units to capture hierarchical long-range context in text. By integrating segment-level topics into the LSTM gates, CLSTM improves word prediction, next-sentence selection, and sentence-topic prediction on Wikipedia and Google News, outperforming strong word-only baselines. The approach demonstrates significant gains (e.g., up to ~21% in next-sentence accuracy and notable perplexity reductions), indicating the practical value of contextual signals for large-scale NLP. The work also explores unsupervised thought vectors as context signals and outlines promising future directions for hierarchical and dialog-oriented applications.

Abstract

Documents exhibit sequential structure at multiple levels of abstraction (e.g., sentences, paragraphs, sections). These abstractions constitute a natural hierarchy for representing the context in which to infer the meaning of words and larger fragments of text. In this paper, we present CLSTM (Contextual LSTM), an extension of the recurrent neural network LSTM (Long-Short Term Memory) model, where we incorporate contextual features (e.g., topics) into the model. We evaluate CLSTM on three specific NLP tasks: word prediction, next sentence selection, and sentence topic prediction. Results from experiments run on two corpora, English documents in Wikipedia and a subset of articles from a recent snapshot of English Google News, indicate that using both words and topics as features improves performance of the CLSTM models over baseline LSTM models for these tasks. For example on the next sentence selection task, we get relative accuracy improvements of 21% for the Wikipedia dataset and 18% for the Google News dataset. This clearly demonstrates the significant benefit of using context appropriately in natural language (NL) tasks. This has implications for a wide variety of NL applications like question answering, sentence completion, paraphrase generation, and next utterance prediction in dialog systems.

Figures (9)

• Figure 1: CLSTM model ($<$Topic$>$ = topic input)
• Figure 2: Hierarchical Features used in CLSTM models
• Figure 3: Next Sentence Selection Example
• Figure 4: Next Sentence Scoring in CLSTM model
• Figure 5: Error Type A: CLSTM correct, LSTM incorrect