Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

FormationEval, an open multiple-choice benchmark for petroleum geoscience

Almaz Ermilov

TL;DR

FormationEval presents an open, concept-based MCQ benchmark with 505 questions across seven petroleum geoscience domains to systematically evaluate 72 language models. The pipeline emphasizes copyright-respecting question generation, provenance-tracked metadata, and bias mitigation, enabling transparent cross-model comparisons. Front-tier models reach near-perfect accuracies (e.g., 99.8% by Gemini 3 Pro Preview), while open-weight models demonstrate strong domain knowledge and cost-effectiveness, particularly in Reservoir Engineering and Geophysics. The work highlights domain-specific performance patterns, persistent Petrophysics challenges, and a commitment to reproducibility and ongoing expansion of the benchmark as a living resource.

Abstract

This paper presents FormationEval, an open multiple-choice question benchmark for evaluating language models on petroleum geoscience and subsurface disciplines. The dataset contains 505 questions across seven domains including petrophysics, petroleum geology and reservoir engineering, derived from three authoritative sources using a reasoning model with detailed instructions and a concept-based approach that avoids verbatim copying of copyrighted text. Each question includes source metadata to support traceability and audit. The evaluation covers 72 models from major providers including OpenAI, Anthropic, Google, Meta and open-weight alternatives. The top performers achieve over 97\% accuracy, with Gemini 3 Pro Preview reaching 99.8\%, while tier and domain gaps persist. Among open-weight models, GLM-4.7 leads at 98.6\%, with several DeepSeek, Llama, Qwen and Mistral models also exceeding 93\%. The performance gap between open-weight and closed models is narrower than expected, with several lower-cost open-weight models exceeding 90\% accuracy. Petrophysics emerges as the most challenging domain across all models, while smaller models show wider performance variance. Residual length bias in the dataset (correct answers tend to be longer) is documented along with bias mitigation strategies applied during construction. The benchmark, evaluation code and results are publicly available.

FormationEval, an open multiple-choice benchmark for petroleum geoscience

TL;DR

FormationEval presents an open, concept-based MCQ benchmark with 505 questions across seven petroleum geoscience domains to systematically evaluate 72 language models. The pipeline emphasizes copyright-respecting question generation, provenance-tracked metadata, and bias mitigation, enabling transparent cross-model comparisons. Front-tier models reach near-perfect accuracies (e.g., 99.8% by Gemini 3 Pro Preview), while open-weight models demonstrate strong domain knowledge and cost-effectiveness, particularly in Reservoir Engineering and Geophysics. The work highlights domain-specific performance patterns, persistent Petrophysics challenges, and a commitment to reproducibility and ongoing expansion of the benchmark as a living resource.

Abstract

This paper presents FormationEval, an open multiple-choice question benchmark for evaluating language models on petroleum geoscience and subsurface disciplines. The dataset contains 505 questions across seven domains including petrophysics, petroleum geology and reservoir engineering, derived from three authoritative sources using a reasoning model with detailed instructions and a concept-based approach that avoids verbatim copying of copyrighted text. Each question includes source metadata to support traceability and audit. The evaluation covers 72 models from major providers including OpenAI, Anthropic, Google, Meta and open-weight alternatives. The top performers achieve over 97\% accuracy, with Gemini 3 Pro Preview reaching 99.8\%, while tier and domain gaps persist. Among open-weight models, GLM-4.7 leads at 98.6\%, with several DeepSeek, Llama, Qwen and Mistral models also exceeding 93\%. The performance gap between open-weight and closed models is narrower than expected, with several lower-cost open-weight models exceeding 90\% accuracy. Petrophysics emerges as the most challenging domain across all models, while smaller models show wider performance variance. Residual length bias in the dataset (correct answers tend to be longer) is documented along with bias mitigation strategies applied during construction. The benchmark, evaluation code and results are publicly available.
Paper Structure (36 sections, 8 figures, 6 tables)

This paper contains 36 sections, 8 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Benchmark design and construction
  4. Task definition and scope
  5. Source selection and licensing policy
  6. Schema and metadata
  7. Multiple-choice question generation pipeline
  8. Quality assurance and audit
  9. Bias analysis and mitigation
  10. PDF export for review
  11. Dataset summary
  12. Evaluation setup
  13. Models and providers
  14. Prompting and answer extraction
  15. Metrics
  16. ...and 21 more sections

Figures (8)

  • Figure 1: MCQ generation pipeline. Source PDFs are converted to Markdown via OCR, split into chapter chunks, processed by GPT-5.2 to generate candidate questions and verified for schema compliance and source evidence.
  • Figure 2: Dataset composition. (a) Questions by domain, with Petrophysics dominating due to source coverage; (b) difficulty distribution close to 30/50/20 targets; (c) answer position distribution near the expected 25% baseline.
  • Figure 3: Evaluation pipeline. Models are configured via YAML, questions sent to Azure OpenAI or OpenRouter APIs, responses cached per model/question and analyzed to generate leaderboard and analysis reports.
  • Figure 4: Top 30 models on FormationEval v0.1 by accuracy. Blue bars indicate open-weight models. GLM-4.7 (98.6%) leads among open-weight models, ranking second overall.
  • Figure 5: Cost-effectiveness analysis. Accuracy versus average token price (mean of input and output prices). Several high-accuracy models (Grok-4.1-fast, DeepSeek-R1) offer strong performance at lower cost. Open-weight models (blue) provide lower-cost alternatives to closed models (orange).
  • ...and 3 more figures