Smart Bilingual Focused Crawling of Parallel Documents

TL;DR

The paper tackles the inefficiency of unguided web crawling for parallel corpora by introducing two URL-derived models: one for inferring document language from URLs and another for predicting whether two URLs point to parallel documents. Built on XLM-RoBERTa, these models are integrated into a smart bilingual focused crawler that prioritizes downloads likely to yield parallel content, reducing wasted bandwidth. Extensive experiments across language pairs and datasets, including synthetic negatives, show improved parallel-data yield and evidence of zero-shot generalization, with four low-resource pairs demonstrating practical benefits. The work provides a scalable approach to parallel corpora harvesting and offers data and code to enable replication and extension.

Abstract

Crawling parallel texts $\unicode{x2014}$texts that are mutual translations$\unicode{x2014}$ from the Internet is usually done following a brute-force approach: documents are massively downloaded in an unguided process, and only a fraction of them end up leading to actual parallel content. In this work we propose a smart crawling method that guides the crawl towards finding parallel content more rapidly. Our approach builds on two different models: one that infers the language of a document from its URL, and another that infers whether a pair of URLs link to parallel documents. We evaluate both models in isolation and their integration into a crawling tool. The results demonstrate the individual effectiveness of both models and highlight that their combination enables the early discovery of parallel content during crawling, leading to a reduction in the amount of downloaded documents deemed useless, and yielding a greater quantity of parallel documents compared to conventional crawling approaches.

Figures (7)

• Figure 1: Architecture of the model used for language identification from URLs.
• Figure 2: Language distribution for the training, development and test sets used in the experiments for language indentification.
• Figure 3: Language identification results, comparing the baseline and our model on a per-language basis. Only languages with a minimum of 100 URLs and 10 different web domains in the test set are included.
• Figure 4: Architecture of the model used for inferring parallelness from URL pairs.
• Figure 5: Macro F1 scores per language (paired with English) for the parallelness identifier from URLs on the dataset described in Sec. \ref{['se:classifier-dataset']}.