Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Hierarchical Multimodal Pre-training for Visually Rich Webpage Understanding

Hongshen Xu, Lu Chen, Zihan Zhao, Da Ma, Ruisheng Cao, Zichen Zhu, Kai Yu

TL;DR

WebLM addresses Visually Rich Webpage Understanding by integrating HTML structure, text content, and page visuals in a unified Transformer. It introduces hierarchical visual feature extraction guided by HTML, plus Tree Structure Prediction and Visual Misalignment Detection as cross-modal objectives. The approach, trained on 6 million webpages and evaluated on WebSRC and SWDE, outperforms state-of-the-art HTML-only and multimodal baselines, with ablations underscoring the value of visual features and hierarchical HTML structure. This work highlights rendering-aware, cross-modal pre-training as a practical pathway for robust, real-world webpage understanding and provides resources for replication and further research.

Abstract

The growing prevalence of visually rich documents, such as webpages and scanned/digital-born documents (images, PDFs, etc.), has led to increased interest in automatic document understanding and information extraction across academia and industry. Although various document modalities, including image, text, layout, and structure, facilitate human information retrieval, the interconnected nature of these modalities presents challenges for neural networks. In this paper, we introduce WebLM, a multimodal pre-training network designed to address the limitations of solely modeling text and structure modalities of HTML in webpages. Instead of processing document images as unified natural images, WebLM integrates the hierarchical structure of document images to enhance the understanding of markup-language-based documents. Additionally, we propose several pre-training tasks to model the interaction among text, structure, and image modalities effectively. Empirical results demonstrate that the pre-trained WebLM significantly surpasses previous state-of-the-art pre-trained models across several webpage understanding tasks. The pre-trained models and code are available at https://github.com/X-LANCE/weblm.

Hierarchical Multimodal Pre-training for Visually Rich Webpage Understanding

TL;DR

WebLM addresses Visually Rich Webpage Understanding by integrating HTML structure, text content, and page visuals in a unified Transformer. It introduces hierarchical visual feature extraction guided by HTML, plus Tree Structure Prediction and Visual Misalignment Detection as cross-modal objectives. The approach, trained on 6 million webpages and evaluated on WebSRC and SWDE, outperforms state-of-the-art HTML-only and multimodal baselines, with ablations underscoring the value of visual features and hierarchical HTML structure. This work highlights rendering-aware, cross-modal pre-training as a practical pathway for robust, real-world webpage understanding and provides resources for replication and further research.

Abstract

The growing prevalence of visually rich documents, such as webpages and scanned/digital-born documents (images, PDFs, etc.), has led to increased interest in automatic document understanding and information extraction across academia and industry. Although various document modalities, including image, text, layout, and structure, facilitate human information retrieval, the interconnected nature of these modalities presents challenges for neural networks. In this paper, we introduce WebLM, a multimodal pre-training network designed to address the limitations of solely modeling text and structure modalities of HTML in webpages. Instead of processing document images as unified natural images, WebLM integrates the hierarchical structure of document images to enhance the understanding of markup-language-based documents. Additionally, we propose several pre-training tasks to model the interaction among text, structure, and image modalities effectively. Empirical results demonstrate that the pre-trained WebLM significantly surpasses previous state-of-the-art pre-trained models across several webpage understanding tasks. The pre-trained models and code are available at https://github.com/X-LANCE/weblm.
Paper Structure (32 sections, 5 equations, 6 figures, 5 tables)

This paper contains 32 sections, 5 equations, 6 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Multi-modal Document Pre-training
  4. Webpages Understanding
  5. WebLM
  6. Motivation of WebLM
  7. Model Architecture
  8. Token Embedding
  9. Tag Embedding
  10. Image Embedding
  11. 2D Position Embedding
  12. Pre-training Objectives
  13. Experiments
  14. Data
  15. Pre-training Data
  16. ...and 17 more sections

Figures (6)

  • Figure 1: Comparison among different multimodal document pre-training methods.
  • Figure 2: The rendering process from HTML to a webpage.
  • Figure 3: The architecture and pre-training objectives of WebLM. The blue and green parts represent structure and content inputs from HTML code respectively, while the yellow part corresponds to visual inputs from webpage screenshots.
  • Figure 4: The performance comparison on different types of websites of WebSRC development set.
  • Figure 5: Relative performance on WebSRC Dev set compared to full-structured methods after truncating HTML structures.
  • ...and 1 more figures