Multi-Record Web Page Information Extraction From News Websites

TL;DR

This work tackles information extraction from multi-record web pages, focusing on Russian-language news lists and the absence of visual cues. It introduces a large-scale dataset of 13,120 pages, with manual annotations for multiple attributes, and proposes a multi-stage extraction framework built around MarkupLM to operate directly on HTML. The study demonstrates that a sequential pipeline with record-context classification and MarkupLM yields robust improvements over parallel architectures, particularly in recall and overall F1 scores, and validates the approach on segmentation, classification, and matching tasks. Release of the dataset and the methodology provides a strong, language-specific foundation for advancing multi-record information extraction on semi-structured web pages with practical impact for SEO and data harvesting.

Abstract

In this paper, we focused on the problem of extracting information from web pages containing many records, a task of growing importance in the era of massive web data. Recently, the development of neural network methods has improved the quality of information extraction from web pages. Nevertheless, most of the research and datasets are aimed at studying detailed pages. This has left multi-record "list pages" relatively understudied, despite their widespread presence and practical significance. To address this gap, we created a large-scale, open-access dataset specifically designed for list pages. This is the first dataset for this task in the Russian language. Our dataset contains 13,120 web pages with news lists, significantly exceeding existing datasets in both scale and complexity. Our dataset contains attributes of various types, including optional and multi-valued, providing a realistic representation of real-world list pages. These features make our dataset a valuable resource for studying information extraction from pages containing many records. Furthermore, we proposed our own multi-stage information extraction methods. In this work, we explore and demonstrate several strategies for applying MarkupLM to the specific challenges of multi-record web pages. Our experiments validate the advantages of our methods. By releasing our dataset to the public, we aim to advance the field of information extraction from multi-record pages.

Figures (7)

• Figure 1: MarkupLM architecture*
• Figure 2: Parallel pipeline scheme
• Figure 3: Sequential pipeline scheme
• Figure 4: Distribution between pages and amount of records at page at train and test parts
• Figure 5: Attribute distribution between train and test