Dr Web: a modern, query-based web data retrieval engine
Ylli Prifti, Alessandro Provetti, Pasquale de Meo
TL;DR
Doctor Web introduces a modern, query-based web data retrieval engine that uses a JSON5/YAML-based query language to extract structured data from web pages. It positions JSON queryability as a practical alternative to XML/XPath and OXPath, leveraging Playwright for dynamic page rendering and offering a modular, open-source architecture that supports future engine integrations. Benchmark results show Doctor Web achieving about 2x faster execution times and substantial reductions in CPU and memory usage compared to OXPath, though docker-based testing and feature gaps are acknowledged. The work aims to broaden access to web data by simplifying query authoring, enabling community contributions, and providing readily deployable releases via Docker and pip, with ongoing plans to expand efficiency and reliability.
Abstract
This article introduces the Data Retrieval Web Engine (also referred to as doctor web), a flexible and modular tool for extracting structured data from web pages using a simple query language. We discuss the engineering challenges addressed during its development, such as dynamic content handling and messy data extraction. Furthermore, we cover the steps for making the DR Web Engine public, highlighting its open source potential.