CURIE: Evaluating LLMs On Multitask Scientific Long Context Understanding and Reasoning

TL;DR

CURIE introduces a scientific long-context benchmark to probe LLMs' ability to read, extract, and reason over full-length research papers across six domains. It features 10 domain-relevant tasks, totaling 580 input-solution pairs, and evaluates eight models with 32k+ context windows using a mix of standard and model-based metrics, plus human judgments. The results show only modest performance gains, with Claude-3 and Gemini achieving the best overall scores while protein sequencing poses major challenges for several models, highlighting significant room for improvement. By providing rich annotations, a reproducible evaluation pipeline, and ground-truth protocols, CURIE aims to guide future development of LLMs for scientific workflows.

Abstract

Scientific problem-solving involves synthesizing information while applying expert knowledge. We introduce CURIE, a scientific long-Context Understanding,Reasoning and Information Extraction benchmark to measure the potential of Large Language Models (LLMs) in scientific problem-solving and assisting scientists in realistic workflows. This benchmark introduces ten challenging tasks with a total of 580 problems and solution pairs curated by experts in six disciplines - materials science, condensed matter physics, quantum computing, geospatial analysis, biodiversity, and proteins - covering both experimental and theoretical work-flows in science. We evaluate a range of closed and open LLMs on tasks in CURIE which requires domain expertise, comprehension of long in-context information,and multi-step reasoning. While Gemini Flash 2.0 and Claude-3 show consistent high comprehension across domains, the popular GPT-4o and command-R+ fail dramatically on protein sequencing tasks. With the best performance at 32% there is much room for improvement for all models. We hope that insights gained from CURIE can guide the future development of LLMs in sciences. Evaluation code and data are in https://github.com/google/curie

Figures (35)

• Figure 1: (a) Average normalized performance of state-of-the-art LLMs across 10 tasks from six scientific domains in CURIE. (b) Comparing performance of different model versions supporting long-context windows on previous benchmarks testing Knowledge (MMLU), Linguistic (DROP), and Science expertise (GPQA), along with our new scientific long-context understanding CURIE benchmark, highlighting the difficulty of the tasks in the benchmark.
• Figure 2: CURIE dataset. (a) CURIE introduces diverse long context tasks on scientific literature, (b) Distribution of 10 tasks and 429 research documents in six disciplines from which 580 examples were curated, and (c) Length of input context in each domain (log scale), avg. is about 15k.
• Figure 3: Examples of tasks in the CURIE benchmark. The DFT, HFD, QECC, and GEO tasks require the LLM to perform tasks on scientific papers (top blocks), as described in the task snippets (in orange), to extract, calculate, or aggregate information. Expected output (ground truth) snippets are shown in the blue blocks. (Only snippets of the query /outputs are shown for illustrative purposes.)
• Figure 4: (a) A map input from the BIOGR Task. (b) Example PDB task of reconstructing a protein's amino acid sequence from the 3D structure. (c) Length of ground truth per domain (avg. 954 words).
• Figure 5: Per task normalized scores of various LLMs on the CURIE benchmark that measures performance of LLMs on 10 long-context tasks requiring expertise across six scientific disciplines. DFT-S, DFT-P, and MPV are scored using LLMSim, while others use programmatic metrics.