Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

DocSynthv2: A Practical Autoregressive Modeling for Document Generation

Sanket Biswas, Rajiv Jain, Vlad I. Morariu, Jiuxiang Gu, Puneet Mathur, Curtis Wigington, Tong Sun, Josep Lladós

TL;DR

DocSynthv2 tackles end-to-end document generation by jointly modeling layout structure and textual content within an autoregressive framework. It represents documents as a sequence of quantized elements and trains a transformer decoder to predict the next element given prior ones, optimizing a variational loss over discrete attributes and continuous positions. The approach is evaluated on Crello and PubGenNet, showing that text-aware layout generation yields improved alignment, coherence, and competitive results against existing layout transformers. The work advances the field by introducing PubGenNet, validating text-conditioned document generation, and outlining future directions in multimodal document design and evaluation frameworks.

Abstract

While the generation of document layouts has been extensively explored, comprehensive document generation encompassing both layout and content presents a more complex challenge. This paper delves into this advanced domain, proposing a novel approach called DocSynthv2 through the development of a simple yet effective autoregressive structured model. Our model, distinct in its integration of both layout and textual cues, marks a step beyond existing layout-generation approaches. By focusing on the relationship between the structural elements and the textual content within documents, we aim to generate cohesive and contextually relevant documents without any reliance on visual components. Through experimental studies on our curated benchmark for the new task, we demonstrate the ability of our model combining layout and textual information in enhancing the generation quality and relevance of documents, opening new pathways for research in document creation and automated design. Our findings emphasize the effectiveness of autoregressive models in handling complex document generation tasks.

DocSynthv2: A Practical Autoregressive Modeling for Document Generation

TL;DR

DocSynthv2 tackles end-to-end document generation by jointly modeling layout structure and textual content within an autoregressive framework. It represents documents as a sequence of quantized elements and trains a transformer decoder to predict the next element given prior ones, optimizing a variational loss over discrete attributes and continuous positions. The approach is evaluated on Crello and PubGenNet, showing that text-aware layout generation yields improved alignment, coherence, and competitive results against existing layout transformers. The work advances the field by introducing PubGenNet, validating text-conditioned document generation, and outlining future directions in multimodal document design and evaluation frameworks.

Abstract

While the generation of document layouts has been extensively explored, comprehensive document generation encompassing both layout and content presents a more complex challenge. This paper delves into this advanced domain, proposing a novel approach called DocSynthv2 through the development of a simple yet effective autoregressive structured model. Our model, distinct in its integration of both layout and textual cues, marks a step beyond existing layout-generation approaches. By focusing on the relationship between the structural elements and the textual content within documents, we aim to generate cohesive and contextually relevant documents without any reliance on visual components. Through experimental studies on our curated benchmark for the new task, we demonstrate the ability of our model combining layout and textual information in enhancing the generation quality and relevance of documents, opening new pathways for research in document creation and automated design. Our findings emphasize the effectiveness of autoregressive models in handling complex document generation tasks.
Paper Structure (11 sections, 2 equations, 2 figures, 2 tables)

This paper contains 11 sections, 2 equations, 2 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Method
  4. Overview
  5. Model Architecture
  6. Experiments
  7. Datasets
  8. Tasks
  9. Quantitative Evaluation
  10. Qualitative Evaluation
  11. Future Scope and Challenges

Figures (2)

  • Figure 1: Overall Architecture of DocSynthv2.
  • Figure 2: Qualitative results over the two tasks a) Text Prediction: Input (L) and Predicted text content (R) for a Crello sample b) Document Completion: Input (L) with partially filled document and Predicted output (R) for PubGenNet sample