ChEmREF: Evaluating Language Model Readiness for Chemical Emergency Response

TL;DR

This paper introduces ChEmREF, a benchmark to assess whether large language models are ready to assist chemical emergency responders. It defines three tasks—chemical representation translation, incident-response generation, and HazMat exam question answering—applied to 1,035 HazMat chemicals, eight representation types, six ERG guidance dimensions, and six examination categories. Results show strong performance on structured translation and some incident-response aspects, yet consistent gaps in unstructured representations, numerical risk guidance, and domain-specific HazMat knowledge, underscoring the need for cautious human oversight. The findings highlight both the promise and limitations of current LLMs for high-stakes HazMat decision support and point to directions such as retrieval augmentation, long-context reasoning, and cross-modal integration for safer, more effective deployment.

Abstract

Emergency responders managing hazardous material HAZMAT incidents face critical, time-sensitive decisions, manually navigating extensive chemical guidelines. We investigate whether today's language models can assist responders by rapidly and reliably understanding critical information, identifying hazards, and providing recommendations. We introduce the Chemical Emergency Response Evaluation Framework (ChEmREF), a new benchmark comprising questions on 1,035 HAZMAT chemicals from the Emergency Response Guidebook and the PubChem Database. ChEmREF is organized into three tasks: (1) translation of chemical representation between structured and unstructured forms (e.g., converting C2H6O to ethanol), (2) emergency response generation (e.g., recommending appropriate evacuation distances) and (3) domain knowledge question answering from chemical safety and certification exams. Our best evaluated models received an exact match of 68.0% on unstructured HAZMAT chemical representation translation, a LLM Judge score of 52.7% on incident response recommendations, and a multiple-choice accuracy of 63.9% on HAMZAT examinations. These findings suggest that while language models show potential to assist emergency responders in various tasks, they require careful human oversight due to their current limitations.

Figures (13)

• Figure 1: LLM Assistants in Emergency Response. In HazMat emergencies, timely decisions during scene size-up are critical scenesizeup. This figure compares the traditional chemical response timeline with one aided by an LLM. While LLMs may accelerate decision-making, they can also produce incorrect guidance. These dynamics informed the design of ChEmREF.
• Figure 2: ChEmREF Evaluation Results With Per-Metric Normalization. To facilitate clearer model comparison, we normalize each column in Table \ref{['tab:overall_model_summary']} to the [0,1] range. This visualization reveals that models exhibit distinct strengths and weaknesses across tasks.
• Figure 3: Task I Performance Breakdown. Left/Right: Structured/Unstructured Translation. For brevity, we use "Mole." to denote Molecular Formula. We report EM scores for each source-target pair, averaged across all evaluated models.
• Figure 4: Translation Task Model Performance Across Prompt Types: Performance heatmap using HAZMAT data.
• Figure 5: Translation Task Model Performance Across Prompt Types: Performance heatmap using Non-Hazmat chemical data.