Improved methodology for longitudinal Web analytics using Common Crawl

TL;DR

This work tackles the high resource burden of longitudinal analyses on the Common Crawl corpus by proposing two index-based strategies: (i) augmenting each archive's index with Last-Modified timestamps to enable time-evolution studies from a single archive, and (ii) identifying representative archive segments via segment-level distributions to proxy full-archive properties. The authors develop a methodology that uses mime and mime-detected distributions to quantify segment representativeness with Spearman rank correlations and demonstrate cross-property proxying through languages and lengths, showing high fidelity for selected segments across multiple archives. They further extend the analysis by exploiting Last-Modified headers to infer URI-length trends, revealing slow growth driven by path length and just-in-time page dynamics, while also uncovering and correcting an anomaly in 2005. Collectively, the approaches reduce compute, storage, and bandwidth needs for longitudinal web analytics and enable reliable inferences using small, representative samples of archives, with practical implications for large-scale web evolution studies.

Abstract

Common Crawl is a multi-petabyte longitudinal dataset containing over 100 billion web pages which is widely used as a source of language data for sequence model training and in web science research. Each of its constituent archives is on the order of 75TB in size. Using it for research, particularly longitudinal studies, which necessarily involve multiple archives, is therefore very expensive in terms of compute time and storage space and/or web bandwidth. Two new methods for mitigating this problem are presented here, based on exploiting and extending the much smaller (<200 gigabytes (GB) compressed) _index_ which is available for each archive. By adding Last-Modified timestamps to the index we enable longitudinal exploration using only a single archive. By comparing the distribution of index features for each of the 100 segments into which archive is divided with their distribution over the whole archive, we have identified the least and most representative segments for a number of recent archives. Using this allows the segment(s) that are most representative of an archive to be used as proxies for the whole. We illustrate this approach in an analysis of changes in URI length over time, leading to an unanticipated insight into the how the creation of Web pages has changed over time.

Figures (14)

• Figure 1: QQ plot for segment-vs-whole correlations, 2019-35. Based on the distribution of the mime property
• Figure 2: Histogram for segment-vs-whole correlations, 2019-35. Based on the distribution of the mime property. Blue vertical lines show standard deviations from the mean.
• Figure 3: Segment-vs-whole correlations, ordered by segment, 2019-35. Based on the distribution of the mime property
• Figure 4: Segment-vs-whole correlations, ordered by correlation, with 95% confidence intervals, 2019-35. Based on the distribution of the mime property. Calculated using the atanh approach as described in atanh. The worst error bar is just disjoint from the best.
• Figure 5: Heat map for prediction percentiles, 2019-35. The first column lists what property is being predicted by what: length ("len"), mime and language ("nl1"). For example the first row gives the results for predicting what segment(s) should be used to model the distribution of the length property (the target) based on the best N segments from the distribution of the mime property (the basis). The figure in a given cell is the percentile rank of the average of the segment-to-whole correlations from those segments in the target identified as the best N segments in the basis. The heat map colouring is per row, that is, across all the predictions named in the first column as we vary the number used (N). The black-margin cells highlight the best combination of basis and N for each target. The final two columns give mean and standard deviation over all the predictions for a given basis, i.e. the based on the twenty cells to the left.