Identity resolution of software metadata using Large Language Models

TL;DR

This work evaluates instruction-tuned large language models for resolving identity across heterogeneous software metadata, a key step in assembling a cohesive software observatory for FAIR assessment. By framing the task as three-way classification and benchmarking against a human-annotated gold standard, the study reveals that several open, instruction-tuned LLMs reach high accuracy and can drastically reduce annotation time. The authors introduce an agreement-based proxy that combines model consensus to achieve high precision with substantial coverage, enabling scalable, selective automation in metadata workflows. The findings support integrating LLMs into FAIR-aligned software metadata pipelines while underscoring limits in recognizing ambiguous cases and the need for human review in deferral scenarios.

Abstract

Software is an essential component of research. However, little attention has been paid to it compared with that paid to research data. Recently, there has been an increase in efforts to acknowledge and highlight the importance of software in research activities. Structured metadata from platforms like bio.tools, Bioconductor, and Galaxy ToolShed offers valuable insights into research software in the Life Sciences. Although originally intended to support discovery and integration, this metadata can be repurposed for large-scale analysis of software practices. However, its quality and completeness vary across platforms, reflecting diverse documentation practices. To gain a comprehensive view of software development and sustainability, consolidating this metadata is necessary, but requires robust mechanisms to address its heterogeneity and scale. This article presents an evaluation of instruction-tuned large language models for the task of software metadata identity resolution, a critical step in assembling a cohesive collection of research software. Such a collection is the reference component for the Software Observatory at OpenEBench, a platform that aggregates metadata to monitor the FAIRness of research software in the Life Sciences. We benchmarked multiple models against a human-annotated gold standard, examined their behavior on ambiguous cases, and introduced an agreement-based proxy for high-confidence automated decisions. The proxy achieved high precision and statistical robustness, while also highlighting the limitations of current models and the broader challenges of automating semantic judgment in FAIR-aligned software metadata across registries and repositories.

Figures (5)

• Figure 1: Distributions of human annotations in the gold standard. Left: class distribution across the three possible verdicts (same, different, unclear). Center: distribution of annotator confidence levels across all annotated cases. Right: joint distribution of verdict and confidence, restricted to entries labeled as same or different. Unclear cases were excluded from confidence scoring.
• Figure 2: Multiclass evaluation metrics by model. Each bar shows accuracy and macro-F1 for a given language model.
• Figure 3: Comparison of model performance on “hard” vs. “easy” disambiguation cases. The left panel shows error rates with 95% bootstrapped confidence intervals; each dot represents the mean error rate for a given case type (easy or hard). The right panel displays the corresponding number of errors on each case type. Asterisks (*) indicate LLMs for which the error rate on hard cases was significantly higher than on easy cases ($p < 0.05$, bootstrap test).
• Figure 4: Performance and coverage of agreement-based proxies. Each point shows the accuracy and macro F1-score of a proxy with 95% bootstrap confidence intervals. The rightmost panel shows the percentage of cases deferred because there was a disagreement between the LLMs. All metrics are computed only over non-deferred case
• Figure 5: Extrapolated cumulative annotation time as a function of dataset size for humans, Llama 4 Scout, and proxy methods. All time estimates are extrapolated from a shared gold standard of 100 annotated records. The human curve is based on measured total annotation time across those 100 cases, with a 95% confidence interval shown as a shaded area. Llama 4 Scout time is derived from average end-to-end latency per record across the same set. Proxy I and Proxy V simulate agreement-based workflows, combining Llama 4 Scout with Mixtral 8x22B and Mixtral 8x7B, respectively, with fallback to human annotation in 15% (Proxy I) and 6% (Proxy V) of cases. No full-scale annotation was performed beyond the initial 100; curves represent linear projections.