Transition-Based Dependency Parsing with Stack Long Short-Term Memory

TL;DR

We address the challenge of representing parser states in transition-based dependency parsing. We introduce stack LSTMs that support push and pop while maintaining embeddings for the buffer, the stack of partial trees, and the action history, enabling a global, compositional representation of parser state. The model uses three stack-LSTM encodings and recursive composition to build representations of tree fragments, trained by conditional log-likelihood to achieve state-of-the-art results on English and Chinese parsing. This work demonstrates the effectiveness of stack memory for learning rich, context-sensitive parser decisions and suggests future directions toward unsupervised or memory-augmented parsing.

Abstract

We propose a technique for learning representations of parser states in transition-based dependency parsers. Our primary innovation is a new control structure for sequence-to-sequence neural networks---the stack LSTM. Like the conventional stack data structures used in transition-based parsing, elements can be pushed to or popped from the top of the stack in constant time, but, in addition, an LSTM maintains a continuous space embedding of the stack contents. This lets us formulate an efficient parsing model that captures three facets of a parser's state: (i) unbounded look-ahead into the buffer of incoming words, (ii) the complete history of actions taken by the parser, and (iii) the complete contents of the stack of partially built tree fragments, including their internal structures. Standard backpropagation techniques are used for training and yield state-of-the-art parsing performance.

Figures (5)

• Figure 1: A stack LSTM extends a conventional left-to-right LSTM with the addition of a stack pointer (notated as TOP in the figure). This figure shows three configurations: a stack with a single element (left), the result of a pop operation to this (middle), and then the result of applying a push operation (right). The boxes in the lowest rows represent stack contents, which are the inputs to the LSTM, the upper rows are the outputs of the LSTM (in this paper, only the output pointed to by TOP is ever accessed), and the middle rows are the memory cells (the $\mathbf{c}_t$'s and $\mathbf{h}_t$'s) and gates. Arrows represent function applications (usually affine transformations followed by a nonlinearity), refer to §\ref{['subsec:lstms']} for specifics.
• Figure 2: Parser state computation encountered while parsing the sentence "an overhasty decision was made." Here $S$ designates the stack of partially constructed dependency subtrees and its LSTM encoding; $B$ is the buffer of words remaining to be processed and its LSTM encoding; and $A$ is the stack representing the history of actions taken by the parser. These are linearly transformed, passed through a ReLU nonlinearity to produce the parser state embedding $\mathbf{p}_t$. An affine transformation of this embedding is passed to a softmax layer to give a distribution over parsing decisions that can be taken.
• Figure 3: Parser transitions indicating the action applied to the stack and buffer and the resulting stack and buffer states. Bold symbols indicate (learned) embeddings of words and relations, script symbols indicate the corresponding words and relations.
• Figure 4: Token embedding of the words decision, which is present in both the parser's training data and the language model data, and overhasty, an adjective that is not present in the parser's training data but is present in the LM data.
• Figure 5: The representation of a dependency subtree (above) is computed by recursively applying composition functions to $\langle \textrm{head}, \textrm{modifier}, \textrm{relation} \rangle$ triples. In the case of multiple dependents of a single head, the recursive branching order is imposed by the order of the parser's reduce operations (below).