MagnetDB: A Longitudinal Torrent Discovery Dataset with IMDb-Matched Movies and TV Shows

TL;DR

MagnetDB addresses the need for a long-running, content-rich view of BitTorrent supply-side dynamics by continuously crawling the BitTorrent DHT over 2018–2024 to compile 28.6M torrents and 950M files, later enriching a subset of video files with IMDb metadata via an Elasticsearch-based matching pipeline using a $2\sigma$ threshold. The approach blends large-scale torrent discovery with detailed file-level parsing and metadata augmentation, yielding a 1.56M IMDb-matched video file subset across roughly 751K movies and 811K TV episodes. Key contributions include a longitudinal, content-centric dataset that reveals supply-side practices, The Scene’s influence, and selective coverage patterns, alongside a transparent processing pipeline and FAIR data principles to enable reproducibility and broad reuse. The work’s significance lies in enabling nuanced analyses of piracy ecosystems, distribution trajectories, and cross-genre platform targeting, with practical implications for cultural analytics, policy, and anti-piracy strategies. MagnetDB thus provides a foundational, open resource for studying the socio-technical dynamics of digital piracy at scale.

Abstract

BitTorrent remains a prominent channel for illicit distribution of copyrighted material, yet the supply side of such content remains understudied. We introduce MagnetDB, a longitudinal dataset of torrents discovered through the BitTorrent DHT between 2018 and 2024, containing more than 28.6 million torrents and metadata of more than 950 million files. While our primary focus is on enabling research based on the supply of pirated movies and TV shows, the dataset also encompasses other legitimate and illegitimate torrents. By applying IMDb-matching and annotation to movie and TV show torrents, MagnetDB facilitates detailed analyses of pirated content evolution in the BitTorrent network. Researchers can leverage MagnetDB to examine distribution trends, subcultural practices, and the gift economy within piracy ecosystems. Through its scale and temporal scope, MagnetDB presents a unique opportunity for investigating the broader dynamics of BitTorrent and advancing empirical knowledge on digital piracy.

Figures (7)

• Figure 1: Histogram of discovered torrents with crawler downtime (red) and degraded periods (orange). Degraded is defined as weeks where less than 10,000 torrents were discovered. Uptime over 5 years and 9 months was 93.7%.
• Figure 2: Sankey diagram of the data processing pipeline going from torrents to the final matched dataset. MagnetDB provides all data at each processing step so that other researchers can tailor their own data pipeline to fit their use case.
• Figure 3: Density plot with overlaid CDF of file sizes in bytes.
• Figure 4: Distribution of BM25 Match Scores to the IMDb Non-Commercial Database. The Red-dashed Line indicates the $2\sigma$ (138) threshold past which we retain rows for the IMDb-matched subset.
• Figure 5: Select breakdowns of the matched video files: Quality labels, language tags, distribution "sites," and encoders (creators of the torrents themselves).