CienaLLM: Generative Climate-Impact Extraction from News Articles with Autoregressive LLMs

TL;DR

This paper presents CienaLLM, a modular GenIE framework that leverages open-weight LLMs to extract structured drought-impact information from Spanish news without task-specific retraining. Through a large factorial study across model families, sizes, precision regimes, and prompting strategies, it demonstrates that larger models and certain prompts improve accuracy and reliability, while quantization dramatically enhances efficiency with modest accuracy trade-offs. The approach delivers competitive drought-impact extraction and relevance classification versus a supervised baseline, and offers flexible, schema-driven extraction including location data, with on-premise inference enabling privacy and control. The work emphasizes parsing reliability, model recency, and prompt strategy as key levers, and highlights broader applicability to other hazards and languages through prompt and schema adjustments rather than retraining.

Abstract

Understanding and monitoring the socio-economic impacts of climate hazards requires extracting structured information from heterogeneous news articles on a large scale. To that end, we have developed CienaLLM, a modular framework based on schema-guided Generative Information Extraction. CienaLLM uses open-weight Large Language Models for zero-shot information extraction from news articles, and supports configurable prompts and output schemas, multi-step pipelines, and cloud or on-premise inference. To systematically assess how the choice of LLM family, size, precision regime, and prompting strategy affect performance, we run a large factorial study in models, precisions, and prompt engineering techniques. An additional response parsing step nearly eliminates format errors while preserving accuracy; larger models deliver the strongest and most stable performance, while quantization offers substantial efficiency gains with modest accuracy trade-offs; and prompt strategies show heterogeneous, model-specific effects. CienaLLM matches or outperforms the supervised baseline in accuracy for extracting drought impacts from Spanish news, although at a higher inference cost. While evaluated in droughts, the schema-driven and model-agnostic design is suitable for adapting to related information extraction tasks (e.g., other hazards, sectors, or languages) by editing prompts and schemas rather than retraining. We release code, configurations, and schemas to support reproducible use.

Figures (23)

• Figure 1: Main components of the CienaLLM pipeline.
• Figure 2: Parsing error rates by model (LLM) with response parsing (RPARSE) enabled (left) and disabled (right).Without response parsing, several models, particularly the smaller Llama variants, exhibit high and variable error rates. In contrast, response parsing eliminates such failures across nearly all models.
• Figure 3: Distribution of F1 scores across prompt configurations for each mode (LLM). Larger models generally achieve higher and more stable performance.
• Figure 4: Effect of prompt strategies on F1 scores by model (LLM). Bars show the change in mean F1 ($\Delta$) relative to the base prompt for Summarization (SUM), Self-Criticism (SC), Chain of Thought (CoT), and Impact Descriptions (DESC), showing only statistically significant changes.
• Figure 5: Multi-objective Pareto-front analysis of all evaluated configurations, showing the trade-off between F1 score and execution time per article. Pareto-optimal configurations are marked with red crosses, and three representative setups, Best-F1, Efficient, and Fastest, are highlighted.