Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Flexible and Scalable Approach for Collecting Wildlife Advertisements on the Web

Juliana Barbosa, Sunandan Chakraborty, Juliana Freire

TL;DR

This work tackles the problem of obtaining large-scale, cross-market data on online wildlife advertising to understand trafficking dynamics and enable disruption. It presents a flexible, scalable data-collection pipeline that combines scoped web crawling, extraction, and zero-shot ML filtering, implemented with open-source tools and deployed in a Kubernetes environment with cloud-like storage. The authors build a substantial dataset—nearly 1 million ads from 41 marketplaces, spanning 235 species and 20 languages—by generating seeds from endangered-species lists and wide keyword queries. The pipeline demonstrates substantial throughput, broad language/domain coverage, and an open-source framework that can be extended with finer-tuned models and multi-modal extraction for improved accuracy.

Abstract

Wildlife traffickers are increasingly carrying out their activities in cyberspace. As they advertise and sell wildlife products in online marketplaces, they leave digital traces of their activity. This creates a new opportunity: by analyzing these traces, we can obtain insights into how trafficking networks work as well as how they can be disrupted. However, collecting such information is difficult. Online marketplaces sell a very large number of products and identifying ads that actually involve wildlife is a complex task that is hard to automate. Furthermore, given that the volume of data is staggering, we need scalable mechanisms to acquire, filter, and store the ads, as well as to make them available for analysis. In this paper, we present a new approach to collect wildlife trafficking data at scale. We propose a data collection pipeline that combines scoped crawlers for data discovery and acquisition with foundational models and machine learning classifiers to identify relevant ads. We describe a dataset we created using this pipeline which is, to the best of our knowledge, the largest of its kind: it contains almost a million ads obtained from 41 marketplaces, covering 235 species and 20 languages. The source code is publicly available at \url{https://github.com/VIDA-NYU/wildlife_pipeline}.

A Flexible and Scalable Approach for Collecting Wildlife Advertisements on the Web

TL;DR

This work tackles the problem of obtaining large-scale, cross-market data on online wildlife advertising to understand trafficking dynamics and enable disruption. It presents a flexible, scalable data-collection pipeline that combines scoped web crawling, extraction, and zero-shot ML filtering, implemented with open-source tools and deployed in a Kubernetes environment with cloud-like storage. The authors build a substantial dataset—nearly 1 million ads from 41 marketplaces, spanning 235 species and 20 languages—by generating seeds from endangered-species lists and wide keyword queries. The pipeline demonstrates substantial throughput, broad language/domain coverage, and an open-source framework that can be extended with finer-tuned models and multi-modal extraction for improved accuracy.

Abstract

Wildlife traffickers are increasingly carrying out their activities in cyberspace. As they advertise and sell wildlife products in online marketplaces, they leave digital traces of their activity. This creates a new opportunity: by analyzing these traces, we can obtain insights into how trafficking networks work as well as how they can be disrupted. However, collecting such information is difficult. Online marketplaces sell a very large number of products and identifying ads that actually involve wildlife is a complex task that is hard to automate. Furthermore, given that the volume of data is staggering, we need scalable mechanisms to acquire, filter, and store the ads, as well as to make them available for analysis. In this paper, we present a new approach to collect wildlife trafficking data at scale. We propose a data collection pipeline that combines scoped crawlers for data discovery and acquisition with foundational models and machine learning classifiers to identify relevant ads. We describe a dataset we created using this pipeline which is, to the best of our knowledge, the largest of its kind: it contains almost a million ads obtained from 41 marketplaces, covering 235 species and 20 languages. The source code is publicly available at \url{https://github.com/VIDA-NYU/wildlife_pipeline}.
Paper Structure (4 sections, 3 figures, 1 table)

This paper contains 4 sections, 3 figures, 1 table.

Table of Contents

  1. Introduction
  2. Data collection Pipeline
  3. The WildLife Ad Data Collection
  4. Discussion

Figures (3)

  • Figure 1: We describe a pipeline that automatically collects a dataset of wildlife advertisements (ads) from e-commerce websites. Issuing a query (A) "Brazilian blue parrot", we can find and follow links to several product pages (E). For each product (D), we extract product attributes such as (B) product title and (C) price. A challenge in constructing this dataset is how to distinguish ads for wildlife products from other types of products, such as postcards and toys.
  • Figure 2: Data Collection Pipeline.
  • Figure 3: Distribution of zero-shot classes and domains.