UniCrop: A Universal, Multi-Source Data Engineering Pipeline for Scalable Crop Yield Prediction

TL;DR

UniCrop tackles the data-engineering bottleneck in crop yield prediction by delivering a universal, configuration-driven pipeline that automates acquisition, harmonisation, and feature engineering of multi-source environmental data. By decoupling feature specification from implementation, UniCrop can adapt to new crops, regions, and time windows, and it reduces a large pool of variables (>200) to a compact, interpretable subset of 15 features via minimum redundancy maximum relevance (mRMR). In a rice yield case study with 557 field observations, UniCrop achieved competitive predictive performance using standard models (e.g., RMSE ≈ 465 kg/ha, $R^2$ ≈ 0.66 with a single model; ensemble ≈ 463 kg/ha, $R^2$ ≈ 0.66) and demonstrated interpretability through SHAP analyses that aligned with agronomic knowledge. The work provides a transparent, reusable data foundation for scalable agricultural analytics, with open-source code and documentation to enable reproducibility, transferability, and broader deployment across crops and regions.

Abstract

Accurate crop yield prediction relies on diverse data streams, including satellite, meteorological, soil, and topographic information. However, despite rapid advances in machine learning, existing approaches remain crop- or region-specific and require data engineering efforts. This limits scalability, reproducibility, and operational deployment. This study introduces UniCrop, a universal and reusable data pipeline designed to automate the acquisition, cleaning, harmonisation, and engineering of multi-source environmental data for crop yield prediction. For any given location, crop type, and temporal window, UniCrop automatically retrieves, harmonises, and engineers over 200 environmental variables (Sentinel-1/2, MODIS, ERA5-Land, NASA POWER, SoilGrids, and SRTM), reducing them to a compact, analysis-ready feature set utilising a structured feature reduction workflow with minimum redundancy maximum relevance (mRMR). To validate, UniCrop was applied to a rice yield dataset comprising 557 field observations. Using only the selected 15 features, four baseline machine learning models (LightGBM, Random Forest, Support Vector Regression, and Elastic Net) were trained. LightGBM achieved the best single-model performance (RMSE = 465.1 kg/ha, $R^2 = 0.6576$), while a constrained ensemble of all baselines further improved accuracy (RMSE = 463.2 kg/ha, $R^2 = 0.6604$). UniCrop contributes a scalable and transparent data-engineering framework that addresses the primary bottleneck in operational crop yield modelling: the preparation of consistent and harmonised multi-source data. By decoupling data specification from implementation and supporting any crop, region, and time frame through simple configuration updates, UniCrop provides a practical foundation for scalable agricultural analytics. The code and implementation documentation are shared in https://github.com/CoDIS-Lab/UniCrop.

Figures (9)

• Figure 1: Hybrid schematic of the UniCrop pipeline. Multiple environmental data sources are ingested in parallel, harmonised into a unified structure, enriched with agronomic features, reduced via mRMR into a compact predictor subset, and finally used for baseline modelling and ensemble evaluation.
• Figure 2: Schematic representation of UniCrop’s feature mapping system, illustrated with real examples from the mapping file. The mapping table defines variables, API parameters, datasets, and derivations used to build the automated fetch plan.
• Figure 3: Spatial distribution of field parcels included in the case study, colour-coded by growing season.
• Figure 4: Distribution of rice yield (kg/ha).
• Figure 5: Seasonal missingness patterns for vegetation indices.