Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Latent Diffusion for Guided Document Table Generation

Syed Jawwad Haider Hamdani, Saifullah Saifullah, Stefan Agne, Andreas Dengel, Sheraz Ahmed

TL;DR

This work addresses the challenge of scarce annotated data for complex document table structure by introducing a latent diffusion framework conditioned on row/column masks to synthesize realistic table images. An autoencoder compresses images to a latent space, where a diffusion model guided by a mask produces new samples, with a diffusion-transformer f_theta learning the denoising conditioned on the mask. The approach yields low FID scores and improves table-structure recognition when synthetic data augments real data, achieving competitive mAP on PubTables-1M and ICDAR2013 benchmarks. While promising, limitations in textual fidelity within generated tables are acknowledged, with future work aimed at longer training and richer conditioning to further close the gap to real data. Overall, the method offers a practical path to scalable, structure-aware synthetic data for document-layout understanding.

Abstract

Obtaining annotated table structure data for complex tables is a challenging task due to the inherent diversity and complexity of real-world document layouts. The scarcity of publicly available datasets with comprehensive annotations for intricate table structures hinders the development and evaluation of models designed for such scenarios. This research paper introduces a novel approach for generating annotated images for table structure by leveraging conditioned mask images of rows and columns through the application of latent diffusion models. The proposed method aims to enhance the quality of synthetic data used for training object detection models. Specifically, the study employs a conditioning mechanism to guide the generation of complex document table images, ensuring a realistic representation of table layouts. To evaluate the effectiveness of the generated data, we employ the popular YOLOv5 object detection model for training. The generated table images serve as valuable training samples, enriching the dataset with diverse table structures. The model is subsequently tested on the challenging pubtables-1m testset, a benchmark for table structure recognition in complex document layouts. Experimental results demonstrate that the introduced approach significantly improves the quality of synthetic data for training, leading to YOLOv5 models with enhanced performance. The mean Average Precision (mAP) values obtained on the pubtables-1m testset showcase results closely aligned with state-of-the-art methods. Furthermore, low FID results obtained on the synthetic data further validate the efficacy of the proposed methodology in generating annotated images for table structure.

Latent Diffusion for Guided Document Table Generation

TL;DR

This work addresses the challenge of scarce annotated data for complex document table structure by introducing a latent diffusion framework conditioned on row/column masks to synthesize realistic table images. An autoencoder compresses images to a latent space, where a diffusion model guided by a mask produces new samples, with a diffusion-transformer f_theta learning the denoising conditioned on the mask. The approach yields low FID scores and improves table-structure recognition when synthetic data augments real data, achieving competitive mAP on PubTables-1M and ICDAR2013 benchmarks. While promising, limitations in textual fidelity within generated tables are acknowledged, with future work aimed at longer training and richer conditioning to further close the gap to real data. Overall, the method offers a practical path to scalable, structure-aware synthetic data for document-layout understanding.

Abstract

Obtaining annotated table structure data for complex tables is a challenging task due to the inherent diversity and complexity of real-world document layouts. The scarcity of publicly available datasets with comprehensive annotations for intricate table structures hinders the development and evaluation of models designed for such scenarios. This research paper introduces a novel approach for generating annotated images for table structure by leveraging conditioned mask images of rows and columns through the application of latent diffusion models. The proposed method aims to enhance the quality of synthetic data used for training object detection models. Specifically, the study employs a conditioning mechanism to guide the generation of complex document table images, ensuring a realistic representation of table layouts. To evaluate the effectiveness of the generated data, we employ the popular YOLOv5 object detection model for training. The generated table images serve as valuable training samples, enriching the dataset with diverse table structures. The model is subsequently tested on the challenging pubtables-1m testset, a benchmark for table structure recognition in complex document layouts. Experimental results demonstrate that the introduced approach significantly improves the quality of synthetic data for training, leading to YOLOv5 models with enhanced performance. The mean Average Precision (mAP) values obtained on the pubtables-1m testset showcase results closely aligned with state-of-the-art methods. Furthermore, low FID results obtained on the synthetic data further validate the efficacy of the proposed methodology in generating annotated images for table structure.
Paper Structure (18 sections, 7 equations, 5 figures, 4 tables)

This paper contains 18 sections, 7 equations, 5 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Generative Models in Image Synthesis
  4. Diffusion Models in Document AI
  5. Diffusion for Guided Table Generation
  6. Autoencoder Definition
  7. Forward Diffusion
  8. Reverse Diffusion
  9. Diffusion Transformer
  10. Sampling
  11. Experimental Setup
  12. Datasets.
  13. Data Preparation.
  14. Training Setup.
  15. Results
  16. ...and 3 more sections

Figures (5)

  • Figure 1: The entire model depicting our methodology. $z_0$ is the latent representation of input document table image $x$, $z_t$ is the is the noisy latent after forward diffusion, Mask Condition is the conditioning mask image that depicts the row/column structure of the training table image, Sampling region depicts the newly generated synthetic image.
  • Figure 2: A few sample images from the PubTables-1M pubtables dataset (top), along with their corresponding mask images depicting row and column structures (bottom), are shown.
  • Figure 3: Multiple image generations from single mask. Leftmost column shows the input mask. Columns 2,3,4 and 5 show the synthetic images generated using a single mask shown in the leftmost column.
  • Figure 4: Results of mask conditioned table image generations. Top row shows the input masks to the model. Second row shows the generated images. Third row shows the overlap of the mask on the generated image.
  • Figure 5: Sample results of unconditional table image generations of size 512x512.