Making Metadata More FAIR Using Large Language Models

TL;DR

This paper tackles the problem of poor metadata quality hindering findability and interoperability of scientific data. It presents FAIRMetaText, an NLP-driven framework that converts metadata descriptions into embeddings from large language models and uses a cosine similarity $cos(\vec{w},\vec{d})$ to compare terms. The approach enables two tasks—retrieval for metadata compliance and clustering for metadata unification—and is evaluated on simulated and real datasets across multiple LLMs, with GPT embeddings often delivering the strongest performance. The work suggests substantial potential for automatically cleaning and harmonizing metadata to satisfy the FAIR principles and reduce manual curation, with future work including fine-tuning and open-source deployment.

Abstract

With the global increase in experimental data artifacts, harnessing them in a unified fashion leads to a major stumbling block - bad metadata. To bridge this gap, this work presents a Natural Language Processing (NLP) informed application, called FAIRMetaText, that compares metadata. Specifically, FAIRMetaText analyzes the natural language descriptions of metadata and provides a mathematical similarity measure between two terms. This measure can then be utilized for analyzing varied metadata, by suggesting terms for compliance or grouping similar terms for identification of replaceable terms. The efficacy of the algorithm is presented qualitatively and quantitatively on publicly available research artifacts and demonstrates large gains across metadata related tasks through an in-depth study of a wide variety of Large Language Models (LLMs). This software can drastically reduce the human effort in sifting through various natural language metadata while employing several experimental datasets on the same topic.

Figures (5)

• Figure 1: Real World Metadata - The red boxes bring attention to messy metadata in real records denoted by black boxes.
• Figure 2: Visualising Semantic Similarity of Embeddings: The vectorial representations of seven terms have been projected onto two dimensions and presented here. Given two types of terms - domain specific terms and a set of Boolean terms, the clustering algorithm correctly identifies the semantic categories. Furthermore, the full form 'optimal cutting temperature' is close to the phrase 'OCT embedded'.
• Figure 3: Clustering of Personal vs Diagnostic Entities - Two dimensional projection of embedding vectors of metadata fields and values. Red terms correctly clustered diagnostic metadata and purple terms clustered patient metadata. The axes are a projection of large vector dimensions on to a 2D space and hence are unlabelled.
• Figure 4: Clustering of 'Age' Related Terms - Two dimensional projection of embedding vectors of metadata fields and values relating to 'age'. Different configurations of the term 'age' are clustered into the same bin - whether they are 'age = [X]y' or 'age in years: [X]'. The axes are a projection of large vector dimensions on to a 2D space and hence are unlabelled.
• Figure 5: ChatGPT and Metadata Compliance