Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Omanic: Towards Step-wise Evaluation of Multi-hop Reasoning in Large Language Models

Xiaojie Gu, Sherry T. Tong, Aosong Feng, Sophia Simeng Han, Jinghui Lu, Yingjian Chen, Yusuke Iwasawa, Yutaka Matsuo, Chanjun Park, Rex Ying, Irene Li

Abstract

Reasoning-focused large language models (LLMs) have advanced in many NLP tasks, yet their evaluation remains challenging: final answers alone do not expose the intermediate reasoning steps, making it difficult to determine whether a model truly reasons correctly and where failures occur, while existing multi-hop QA benchmarks lack step-level annotations for diagnosing reasoning failures. To address this gap, we propose Omanic, an open-domain multi-hop QA resource that provides decomposed sub-questions and intermediate answers as structural annotations for analyzing reasoning processes. It contains 10,296 machine-generated training examples (OmanicSynth) and 967 expert-reviewed human-annotated evaluation examples (OmanicBench). Systematic evaluations show that state-of-the-art LLMs achieve only 73.11% multiple-choice accuracy on OmanicBench, confirming its high difficulty. Stepwise analysis reveals that CoT's performance hinges on factual completeness, with its gains diminishing under knowledge gaps and errors amplifying in later hops. Additionally, supervised fine-tuning on OmanicSynth brings substantial transfer gains (7.41 average points) across six reasoning and math benchmarks, validating the dataset's quality and further supporting the effectiveness of OmanicSynth as supervision for reasoning-capability transfer. We release the data at https://huggingface.co/datasets/li-lab/Omanic and the code at https://github.com/XiaojieGu/Omanic.

Omanic: Towards Step-wise Evaluation of Multi-hop Reasoning in Large Language Models

Abstract

Reasoning-focused large language models (LLMs) have advanced in many NLP tasks, yet their evaluation remains challenging: final answers alone do not expose the intermediate reasoning steps, making it difficult to determine whether a model truly reasons correctly and where failures occur, while existing multi-hop QA benchmarks lack step-level annotations for diagnosing reasoning failures. To address this gap, we propose Omanic, an open-domain multi-hop QA resource that provides decomposed sub-questions and intermediate answers as structural annotations for analyzing reasoning processes. It contains 10,296 machine-generated training examples (OmanicSynth) and 967 expert-reviewed human-annotated evaluation examples (OmanicBench). Systematic evaluations show that state-of-the-art LLMs achieve only 73.11% multiple-choice accuracy on OmanicBench, confirming its high difficulty. Stepwise analysis reveals that CoT's performance hinges on factual completeness, with its gains diminishing under knowledge gaps and errors amplifying in later hops. Additionally, supervised fine-tuning on OmanicSynth brings substantial transfer gains (7.41 average points) across six reasoning and math benchmarks, validating the dataset's quality and further supporting the effectiveness of OmanicSynth as supervision for reasoning-capability transfer. We release the data at https://huggingface.co/datasets/li-lab/Omanic and the code at https://github.com/XiaojieGu/Omanic.
Paper Structure (16 sections, 38 figures, 12 tables)

This paper contains 16 sections, 38 figures, 12 tables.

Table of Contents

  1. Introduction
  2. Omanic Construction Pipeline
  3. Triplets Retrieval
  4. Constrained Synthesis
  5. Automated Filtering
  6. Expert Review
  7. Experiments and Analysis
  8. Models and Setup
  9. Main Results
  10. Key Observations
  11. Conclusion
  12. Human Annotation Guidance & Dataset Statistics
  13. Full Results
  14. Implementation details
  15. Discussion
  16. ...and 1 more sections

Figures (38)

  • Figure 1: Overview of Omanic construction pipeline, where rectangles denote entities and circles denote single-hop questions.
  • Figure 2: Comparison of accuracy across benchmarks between the vanilla model and its counterparts fine-tuned on OmanicSynth.
  • Figure 3: Results averaged across LLMs under Direct and CoT prompting. Left: Multi-hop accuracy by number of single-hop errors. Right: Step-wise error rates under independent and chain evaluation.
  • Figure 4: Annotation interface for single-hop question.
  • Figure 5: Annotation interface for 4-hop question.
  • ...and 33 more figures