Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Web Page Classification using LLMs for Crawling Support

Yuichi Sasazawa, Yasuhiro Sogawa

TL;DR

The paper tackles efficient web crawling when sites lack universal update signals by using LLM-based binary page-type classification to distinguish Index Pages (link hubs) from Content Pages (articles) and by starting crawl from identified index pages to reach new pages. It introduces a two-stage framework with automatically annotated datasets and prompts evaluated on GPT-4o variants, comparing title-only versus title+body inputs. Key findings show GPT-4o with Title+Body achieves the best F1 in page-type classification, and that LLM-based approaches generally improve new-page coverage relative to baselines, with some caveats regarding noise from body text and performance at scale. The work demonstrates practical gains in crawling efficiency and provides a dataset and methodology for further refinement of page-type taxonomy and cost-aware deployment in real-world crawlers.

Abstract

A web crawler is a system designed to collect web pages, and efficient crawling of new pages requires appropriate algorithms. While website features such as XML sitemaps and the frequency of past page updates provide important clues for accessing new pages, their universal application across diverse conditions is challenging. In this study, we propose a method to efficiently collect new pages by classifying web pages into two types, "Index Pages" and "Content Pages," using a large language model (LLM), and leveraging the classification results to select index pages as starting points for accessing new pages. We construct a dataset with automatically annotated web page types and evaluate our approach from two perspectives: the page type classification performance and coverage of new pages. Experimental results demonstrate that the LLM-based method outperformed baseline methods in both evaluation metrics.

Web Page Classification using LLMs for Crawling Support

TL;DR

The paper tackles efficient web crawling when sites lack universal update signals by using LLM-based binary page-type classification to distinguish Index Pages (link hubs) from Content Pages (articles) and by starting crawl from identified index pages to reach new pages. It introduces a two-stage framework with automatically annotated datasets and prompts evaluated on GPT-4o variants, comparing title-only versus title+body inputs. Key findings show GPT-4o with Title+Body achieves the best F1 in page-type classification, and that LLM-based approaches generally improve new-page coverage relative to baselines, with some caveats regarding noise from body text and performance at scale. The work demonstrates practical gains in crawling efficiency and provides a dataset and methodology for further refinement of page-type taxonomy and cost-aware deployment in real-world crawlers.

Abstract

A web crawler is a system designed to collect web pages, and efficient crawling of new pages requires appropriate algorithms. While website features such as XML sitemaps and the frequency of past page updates provide important clues for accessing new pages, their universal application across diverse conditions is challenging. In this study, we propose a method to efficiently collect new pages by classifying web pages into two types, "Index Pages" and "Content Pages," using a large language model (LLM), and leveraging the classification results to select index pages as starting points for accessing new pages. We construct a dataset with automatically annotated web page types and evaluate our approach from two perspectives: the page type classification performance and coverage of new pages. Experimental results demonstrate that the LLM-based method outperformed baseline methods in both evaluation metrics.
Paper Structure (10 sections, 2 figures, 8 tables)

This paper contains 10 sections, 2 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Method
  3. Dataset Construction
  4. Page Type Classification using LLMs
  5. Experiments
  6. Evaluation Methods
  7. Comparison Methods
  8. Results
  9. Conclusion
  10. Experiments with the reconstructed dataset

Figures (2)

  • Figure 1: Examples of "Index Pages" space-com-index, which aim to contain hyperlinks to other pages within the website, and "Content Pages," space-com-contents which contain content such as news articles and columns.
  • Figure 2: Overview of the proposed method and evaluation approach. Each web page is classified as either an index page or a content page, and new pages are efficiently retrieved starting from index pages. The experiments evaluate two aspects: the page type classification performance and the coverage of new pages.