DocEdit-v2: Document Structure Editing Via Multimodal LLM Grounding

TL;DR

The DocEditAgent is introduced, a novel framework that performs end-to-end document editing by leveraging Large Multimodal Models (LMMs) and significantly outperforms strong baselines on edit command generation, RoI bounding box detection, and overall document editing tasks.

Abstract

Document structure editing involves manipulating localized textual, visual, and layout components in document images based on the user's requests. Past works have shown that multimodal grounding of user requests in the document image and identifying the accurate structural components and their associated attributes remain key challenges for this task. To address these, we introduce the DocEdit-v2, a novel framework that performs end-to-end document editing by leveraging Large Multimodal Models (LMMs). It consists of three novel components: (1) Doc2Command, which simultaneously localizes edit regions of interest (RoI) and disambiguates user edit requests into edit commands; (2) LLM-based Command Reformulation prompting to tailor edit commands originally intended for specialized software into edit instructions suitable for generalist LMMs. (3) Moreover, DocEdit-v2 processes these outputs via Large Multimodal Models like GPT-4V and Gemini, to parse the document layout, execute edits on grounded Region of Interest (RoI), and generate the edited document image. Extensive experiments on the DocEdit dataset show that DocEdit-v2 significantly outperforms strong baselines on edit command generation (2-33%), RoI bounding box detection (12-31%), and overall document editing (1-12\%) tasks.

Figures (18)

• Figure 1: DocEdit-v2 framework performs multimodal grounding and edit command generation via Doc2Command, utilizes LLM-based Command Reformulation prompting to refine the command into LMM instruction format (<Action><Component>, <Initial State>, <Final State>), and employs LMMs to edit the HTML structure using multimodal (edit instruction and grounded RoI) prompt.
• Figure 2: Doc2Command: Given a document image and a user request, the user request is rendered onto the document, and passed as a singular visual modality to an image encoder. The image encoder feeds into a text decoder and a mask transformer to generate the command text and segmentation maps, respectively.
• Figure 3: Examples showing commands generated post-Doc2Command and Command Reformulation prompting.
• Figure 4: Correlation Heatmaps for Tree Edit Distance and CSS IoU with Human and Automated Evaluation Metrics.
• Figure 5: Examples of segmentation outputs and bounding boxes. The bright white areas represent segmentation outputs. Green boxes represent ground truth bounding boxes, and red boxes represent the inferred bounding boxes.