MajinBook: An open catalogue of digital world literature with likes

TL;DR

MajinBook tackles the challenge of building a high-quality, diachronic corpus of culturally representative literature by linking shadow-library metadata with Goodreads work-edition mappings. It introduces a scalable, privacy-conscious pipeline using MinHash and Locality-Sensitive Hashing to create a high-precision English-language catalogue (539,530 items) and companion datasets in French, German, and Spanish. The approach prioritizes natively digital EPUB content to avoid OCR noise and demonstrates a rigorous matching workflow validated by human evaluation, culminating in openly released data, code, and provenance. The work offers a practical resource for computational social science and cultural analytics, while engaging with contemporary legal and ethical considerations surrounding shadow libraries and text mining.

Abstract

This data paper introduces MajinBook, an open catalogue designed to facilitate the use of shadow libraries--such as Library Genesis and Z-Library--for computational social science and cultural analytics. By linking metadata from these vast, crowd-sourced archives with structured bibliographic data from Goodreads, we create a high-precision corpus of over 539,000 references to English-language books spanning three centuries, enriched with first publication dates, genres, and popularity metrics like ratings and reviews. Our methodology prioritizes natively digital EPUB files to ensure machine-readable quality, while addressing biases in traditional corpora like HathiTrust, and includes secondary datasets for French, German, and Spanish. We evaluate the linkage strategy for accuracy, release all underlying data openly, and discuss the project's legal permissibility under EU and US frameworks for text and data mining in research.

Figures (4)

• Figure 1: Temporal distributions and biases of key corpora. The figure illustrates the distinct temporal biases of the key corpora, justifying our methodological focus on natively digital content. All three plots are semi-logarithmic (log y-axis), displaying item counts binned by publication decade. (a) Compares the epub and pdf subsets of shadow libraries. (b) Contrasts the scanned HathiTrust corpus with all Goodreads editions. (c) Compares our final MajinBook primary corpus (English) to its Goodreads works scaffold. The plots reveal a fundamental difference in corpus structure. The scanned corpora (pdf, HathiTrust) show relatively stable exponential growth (a linear shape), while the social and natively digital corpora (epub, Goodreads, MajinBook) exhibit super-exponential growth (a convex shape) accelerating towards the present. This validates our decision to discard pdfs and confirms that MajinBook (c) is a representative temporal sample of its source.
• Figure 2: The crawl of Goodreads: Item acquisition and recommendation decay. The figure illustrates the efficiency of our crawl methodology. The bars show the cumulative counts of Editions, Works, and Authors (left axis, in millions) gathered at each stage. The line plot tracks the number of new Recommendations (right axis, in thousands) discovered at each depth. The plot reveals a power-law distribution: the initial depths rapidly capture the most prominent items, while subsequent depths explore a long tail of less-connected content.
• Figure 3: Precision-recall trade-off for book matching based on the title score threshold. The plot shows the point estimates for precision (dashed line) and recall (dotted line), along with their 95% confidence intervals (shaded areas), derived from bootstrap resampling of 143 human evaluations. The solid black line indicates the percentage of the dataset retained at each threshold. A vertical line marks our chosen operational threshold of 80, which prioritises high precision for the final catalogue.
• Figure 4: Temporal distribution of primary (English) v. secondary datasets.