Tree-based Focused Web Crawling with Reinforcement Learning

TL;DR

This work addresses efficient, topic-focused web crawling by formulating the problem as a Markov Decision Process and introducing TRES, a framework that combines reinforcement learning with a novel Tree-Frontier sampling strategy. Tree-Frontier enables scalable action selection in large frontiers by online discretization of states and actions and sampling a small, representative subset for Q-value estimation. The authors demonstrate that TRES Pareto-dominates state-of-the-art focused crawlers in harvest rate and retrieved domains while significantly reducing the number of URLs evaluated per step, thanks to the tree-based frontier sampling and informative state-action representations. The approach is validated on real-world web data across multiple topics, showing robust performance and providing theoretical bounds for the sampling method’s efficiency and suboptimality under plausible assumptions. Overall, the work offers a practical, scalable RL-based solution for focused crawling without external paid APIs, with clear implications for search, data collection, and topic-specific web mining.

Abstract

A focused crawler aims at discovering as many web pages and web sites relevant to a target topic as possible, while avoiding irrelevant ones. Reinforcement Learning (RL) has been a promising direction for optimizing focused crawling, because RL can naturally optimize the long-term profit of discovering relevant web locations within the context of a reward. In this paper, we propose TRES, a novel RL-empowered framework for focused crawling that aims at maximizing both the number of relevant web pages (aka \textit{harvest rate}) and the number of relevant web sites (\textit{domains}). We model the focused crawling problem as a novel Markov Decision Process (MDP), which the RL agent aims to solve by determining an optimal crawling strategy. To overcome the computational infeasibility of exhaustively searching for the best action at each time step, we propose Tree-Frontier, a provably efficient tree-based sampling algorithm that adaptively discretizes the large state and action spaces and evaluates only a few representative actions. Experimentally, utilizing online real-world data, we show that TRES significantly outperforms and Pareto-dominates state-of-the-art methods in terms of harvest rate and the number of retrieved relevant domains, while it provably reduces by orders of magnitude the number of URLs needed to be evaluated at each crawling step.

Figures (4)

• Figure 1: A simple state and action example: The illustrated state contains the nodes Seed, A, B and C, while the edge (C, D) is an action for this state.
• Figure 2: An example of Tree-Frontier sampling
• Figure 3: The Pareto front of the examined methods in the Hardware setting. An ideal focused crawler would correspond to a point at top right, that is a point maximizing both objectives.
• Figure 4: Frontier Sampling: (a) Number of Tree-Frontier leaves over time and (b) Frontier size over time. The figures illustrate that with Tree-Frontier sampling (corresponding to left figure) the RL crawler ought to examine approximately 320 times less URLs than with the synchronous method (corresponding to right figure), which makes the latter computationally infeasible to run for long crawling sessions in practice

Theorems & Definitions (18)

• Definition 1: Web Path
• Definition 2: Frontier
• Theorem 1: Time Complexity of the Synchronous Method 10.1007/978-3-319-91662-0_20
• proof
• Theorem 2: Time Complexity of Tree-Frontier sampling
• proof
• Proposition 1
• Proposition 2
• Theorem 3.a
• Theorem 3.b