An Autoregressive Text-to-Graph Framework for Joint Entity and Relation Extraction

TL;DR

This work reframes joint entity and relation extraction as conditional sequence generation that yields a linearized information-graph, using an autoregressive transformer with a dynamic, span-based vocabulary. A pointing mechanism grounds outputs in the source text, and constrained decoding guarantees well-formed graphs, while sentence augmentation mitigates premature termination of generation. ATG achieves strong results across ACE 05, CoNLL 2004, and SciERC, notably excelling on SciERC and matching or surpassing baselines on other benchmarks, with detailed analyses of attention and structure embeddings to illuminate the model's interpretability. The approach offers a practical, grounded, and controllable alternative to plain-text generative IE methods, with potential for improved grounding and efficiency in real-world knowledge-graph construction tasks.

Abstract

In this paper, we propose a novel method for joint entity and relation extraction from unstructured text by framing it as a conditional sequence generation problem. In contrast to conventional generative information extraction models that are left-to-right token-level generators, our approach is \textit{span-based}. It generates a linearized graph where nodes represent text spans and edges represent relation triplets. Our method employs a transformer encoder-decoder architecture with pointing mechanism on a dynamic vocabulary of spans and relation types. Our model can capture the structural characteristics and boundaries of entities and relations through span representations while simultaneously grounding the generated output in the original text thanks to the pointing mechanism. Evaluation on benchmark datasets validates the effectiveness of our approach, demonstrating competitive results. Code is available at https://github.com/urchade/ATG.

Figures (11)

• Figure 1: Linearization for Information Graph Generation. The input text is mapped into an information extraction graph. The graph consists of entities and relation triplets, which are generated sequentially by first producing entity spans (represented by start word, end word, and entity type) followed by relation triplets (head entity, tail entity, and relation type).
• Figure 2: Illustration of the architecture of our model, ATG. (left) The Encoder takes in the input sequence $X$ and generates representations of the tokens ${\bm{H}}$ and spans ${\bm{S}}$. (middle) The Decoder then generates the next token conditioned on the previous tokens and the input representation ${\bm{H}}$. (right) The vocabulary matrix used for decoding consists of the concatenation of span embeddings ${\bm{S}}$, learned relation type embeddings, and special token embeddings.
• Figure 3: Input/ouptut of the decoder. The process starts with the special token <START> and continues until the <END> token is generated. To separate the generation of nodes and edges, a special token <SEP> is used. At each position, the decoder takes in the sum of the embedding of the current token, absolute position embedding, and structural embedding.
• Figure 4: State-Transition diagram for constrained decoding. This diagram illustrates the state-based decision process used during the inference phase, which ensures the generation of a correct graph. Each state is represented by a node, and directed edges indicate valid actions. We use the same color code as the structural embedding in Figure \ref{['fig:embed']}.
• Figure 5: Linearization for different models. In contrast to existing approaches (TANLpaolini2021structured, UIElu-etal-2022-unified, LasUIEfei2022lasuie), our proposed model, ATG, generates spans (along with relation/special tokens) instead of text tokens, which allows for a shorter output sequence, richer (span-level) representation and fully controlled decoding.