Strategies for Structuring Story Generation

TL;DR

The paper tackles the difficulty of long-range coherence in story generation by proposing a coarse-to-fine, three-stage decomposition: (1) generate a structured action sequence via semantic role labeling, (2) produce an anonymized narrative with placeholders for entities, and (3) instantiate those placeholders with coherent surface forms using specialized entity reference models. A verb-attention mechanism and a pointer-copy, subword-based entity generation pipeline enable diverse and coherent action sequences and consistent, context-appropriate entity naming. Extensive experiments on WritingPrompts data show that SRL-based action planning, combined with coreference-aware entity anonymization and subword entity realization, yields substantially more diverse verbs, more coherent coreference clusters, and higher human preference than baselines including a fully end-to-end Fusion model. These results demonstrate the practicality of multi-stage generation for improving long-form narrative quality, with clear implications for controllable and coherent story generation in large-scale language models.

Abstract

Writers generally rely on plans or sketches to write long stories, but most current language models generate word by word from left to right. We explore coarse-to-fine models for creating narrative texts of several hundred words, and introduce new models which decompose stories by abstracting over actions and entities. The model first generates the predicate-argument structure of the text, where different mentions of the same entity are marked with placeholder tokens. It then generates a surface realization of the predicate-argument structure, and finally replaces the entity placeholders with context-sensitive names and references. Human judges prefer the stories from our models to a wide range of previous approaches to hierarchical text generation. Extensive analysis shows that our methods can help improve the diversity and coherence of events and entities in generated stories.

Figures (9)

• Figure 1: Proposed Model. Conditioned upon the prompt, we generate sequences of predicates and arguments. Then, a story is generated with placeholder entities such as ent0. Finally we replace the placeholders with specific references.
• Figure 2: Verb-Attention. To improve the model's ability to condition upon past verbs, one head of the decoder's self-attention mechanism is specialized to only attend to previously generated verbs.
• Figure 3: Input for Coreferent entity reference generation. The model has a representation of the entity context in a bag of words form, all previous predicted values for the same anonymized entity token, and the full text story. The green circle represents the entity mention the model is attempting to fill.
• Figure 4: Human evaluations of different decomposed models for story generation. We find that using SRL action plans and coreference-resolution to build entity clusters generates stories that are preferred by human judges.
• Figure 5: Average Longest Common Subsequence of Generated Stories with human-written stories in the training set.