Flexible metadata harvesting for ecology using large language models

TL;DR

Ecology faces a challenge in discovering and reusing datasets across heterogeneous data providers. This work presents an LLM-based metadata harvester that scrapes dataset landing pages and converts metadata into user-defined unified schemas, enabling cross-provider data discovery. By comparing two metadata standards (LTER-LIFE and Croissant) and applying a post-processing step, the tool retrieves both structured and unstructured metadata with comparable accuracy and demonstrates methods to link datasets via embedding similarity and temporal-coverage unification for graph-based data access. The approach advances FAIR data practices and can be integrated into virtual research environments to support ontology creation, knowledge bases, and efficient dataset discovery.

Abstract

Large, open datasets can accelerate ecological research, particularly by enabling researchers to develop new insights by reusing datasets from multiple sources. However, to find the most suitable datasets to combine and integrate, researchers must navigate diverse ecological and environmental data provider platforms with varying metadata availability and standards. To overcome this obstacle, we have developed a large language model (LLM)-based metadata harvester that flexibly extracts metadata from any dataset's landing page, and converts these to a user-defined, unified format using existing metadata standards. We validate that our tool is able to extract both structured and unstructured metadata with equal accuracy, aided by our LLM post-processing protocol. Furthermore, we utilise LLMs to identify links between datasets, both by calculating embedding similarity and by unifying the formats of extracted metadata to enable rule-based processing. Our tool, which flexibly links the metadata of different datasets, can therefore be used for ontology creation or graph-based queries, for example, to find relevant ecological and environmental datasets in a virtual research environment.

Figures (7)

• Figure 1: Diagram of our LLM-based metadata harvester that retrieves and converts metadata from any data provider to any metadata format.
• Figure 2: Overview of the manually annotated metadata for $N=16$ datasets (Table \ref{['tab:metadatafield_overview']}), which was either unavailable, available and structured, or available but unstructured.
• Figure 3: Retrieval accuracy of post-processed, structured, present metadata per provider and metadata format (averaged across LLMs). Note that because the Croissant metadata format contains fewer fields, its uncertainty estimates are generally higher, especially for LP DAAC where N=2 for Croissant.
• Figure 4: Retrieval accuracy of present, structured metadata, split by processing stage (a), LLM (b) and metadata format (c).
• Figure 5: Retrieval accuracy of structured vs unstructured metadata using Gemini 2.5 Flash. Please note that for some metadata fields all metadata (across datasets) was either structured or unstructured (see Fig \ref{['fig:metadata_availability']}).