LLM-based Evaluation Policy Extraction for Ecological Modeling

TL;DR

The paper tackles the challenge of evaluating ecological time-series predictions beyond traditional metrics by introducing the Adaptive Policy Evaluation Framework (APEF). APEF combines a learnable base metric that captures temporal segmentation, peak behavior, and derivatives with an LLM-driven policy extractor to produce interpretable evaluation rules derived from expert pairwise judgments. It iterates weight optimization and policy generation, and extends to multi-variate series to account for interdependencies, validated across synthetic data, human expert alignment, and ILAMB benchmarking. The framework yields policies that align with expert priorities, enhance interpretability, and adapt to task-specific evaluation demands, offering a scalable approach for ecological model benchmarking with practical decision-making impact.

Abstract

Evaluating ecological time series is critical for benchmarking model performance in many important applications, including predicting greenhouse gas fluxes, capturing carbon-nitrogen dynamics, and monitoring hydrological cycles. Traditional numerical metrics (e.g., R-squared, root mean square error) have been widely used to quantify the similarity between modeled and observed ecosystem variables, but they often fail to capture domain-specific temporal patterns critical to ecological processes. As a result, these methods are often accompanied by expert visual inspection, which requires substantial human labor and limits the applicability to large-scale evaluation. To address these challenges, we propose a novel framework that integrates metric learning with large language model (LLM)-based natural language policy extraction to develop interpretable evaluation criteria. The proposed method processes pairwise annotations and implements a policy optimization mechanism to generate and combine different assessment metrics. The results obtained on multiple datasets for evaluating the predictions of crop gross primary production and carbon dioxide flux have confirmed the effectiveness of the proposed method in capturing target assessment preferences, including both synthetically generated and expert-annotated model comparisons. The proposed framework bridges the gap between numerical metrics and expert knowledge while providing interpretable evaluation policies that accommodate the diverse needs of different ecosystem modeling studies.

Figures (9)

• Figure 1: Illustrative example showing the inconsistency in model ranking by traditional metric (e.g., RMSE) and human expert (e.g., emphasizing peak period alignment).
• Figure 2: Overall flow of APEF. LLM is first used to adjust weights in the base metric to align with expert annotation. The weight adjustment records are then fed to the LLM to extract interpretable evaluation policies.
• Figure 3: An example of policy extracted by APEF.
• Figure 4: Correlation performance on the synthetic data using preset weights.
• Figure 5: Examples of ranking mismatch in the Peak period weight setting. In these examples, prediction (a) is better than prediction (b) according to target annotation and APEF, but standard metric $R^2$ produces opposite comparison outcome.