Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

STELLA: Self-Reflective Terminology-Aware Framework for Building an Aerospace Information Retrieval Benchmark

Bongmin Kim

TL;DR

STELLA introduces a domain-specific aerospace IR benchmark construction pipeline that leverages NASA NTRS documents to create passage-level retrieval units and a terminology dictionary, enabling dual-type synthetic queries (TCQ for lexical matching and TAQ for semantic matching). The framework integrates document parsing with robust terminology extraction, intent-driven candidate passage selection, and a CoD plus Self-Reflection based query generation process, followed by a cross-lingual extension that mirrors real-world querying practices. Empirical evaluation shows that large decoder-based embeddings achieve strong semantic understanding while lexical methods like BM25 remain competitive in terminology-heavy aerospace tasks, and that cross-lingual TCQ significantly outperforms TAQ, underscoring the value of terminology-preserving queries in multilingual settings. Overall, STELLA provides a reproducible, domain-grounded platform for assessing and improving aerospace IR models, highlighting the continued importance of hybrid retrieval and domain-specific evaluation.

Abstract

Tasks in the aerospace industry heavily rely on searching and reusing large volumes of technical documents, yet there is no public information retrieval (IR) benchmark that reflects the terminology- and query-intent characteristics of this domain. To address this gap, this paper proposes the STELLA (Self-Reflective TErminoLogy-Aware Framework for BuiLding an Aerospace Information Retrieval Benchmark) framework. Using this framework, we introduce the STELLA benchmark, an aerospace-specific IR evaluation set constructed from NASA Technical Reports Server (NTRS) documents via a systematic pipeline that comprises document layout detection, passage chunking, terminology dictionary construction, synthetic query generation, and cross-lingual extension. The framework generates two types of queries: the Terminology Concordant Query (TCQ), which includes the terminology verbatim to evaluate lexical matching, and the Terminology Agnostic Query (TAQ), which utilizes the terminology's description to assess semantic matching. This enables a disentangled evaluation of the lexical and semantic matching capabilities of embedding models. In addition, we combine Chain-of-Density (CoD) and the Self-Reflection method with query generation to improve quality and implement a hybrid cross-lingual extension that reflects real user querying practices. Evaluation of seven embedding models on the STELLA benchmark shows that large decoder-based embedding models exhibit the strongest semantic understanding, while lexical matching methods such as BM25 remain highly competitive in domains where exact lexical matching technical term is crucial. The STELLA benchmark provides a reproducible foundation for reliable performance evaluation and improvement of embedding models in aerospace-domain IR tasks. The STELLA benchmark can be found in https://huggingface.co/datasets/telepix/STELLA.

STELLA: Self-Reflective Terminology-Aware Framework for Building an Aerospace Information Retrieval Benchmark

TL;DR

STELLA introduces a domain-specific aerospace IR benchmark construction pipeline that leverages NASA NTRS documents to create passage-level retrieval units and a terminology dictionary, enabling dual-type synthetic queries (TCQ for lexical matching and TAQ for semantic matching). The framework integrates document parsing with robust terminology extraction, intent-driven candidate passage selection, and a CoD plus Self-Reflection based query generation process, followed by a cross-lingual extension that mirrors real-world querying practices. Empirical evaluation shows that large decoder-based embeddings achieve strong semantic understanding while lexical methods like BM25 remain competitive in terminology-heavy aerospace tasks, and that cross-lingual TCQ significantly outperforms TAQ, underscoring the value of terminology-preserving queries in multilingual settings. Overall, STELLA provides a reproducible, domain-grounded platform for assessing and improving aerospace IR models, highlighting the continued importance of hybrid retrieval and domain-specific evaluation.

Abstract

Tasks in the aerospace industry heavily rely on searching and reusing large volumes of technical documents, yet there is no public information retrieval (IR) benchmark that reflects the terminology- and query-intent characteristics of this domain. To address this gap, this paper proposes the STELLA (Self-Reflective TErminoLogy-Aware Framework for BuiLding an Aerospace Information Retrieval Benchmark) framework. Using this framework, we introduce the STELLA benchmark, an aerospace-specific IR evaluation set constructed from NASA Technical Reports Server (NTRS) documents via a systematic pipeline that comprises document layout detection, passage chunking, terminology dictionary construction, synthetic query generation, and cross-lingual extension. The framework generates two types of queries: the Terminology Concordant Query (TCQ), which includes the terminology verbatim to evaluate lexical matching, and the Terminology Agnostic Query (TAQ), which utilizes the terminology's description to assess semantic matching. This enables a disentangled evaluation of the lexical and semantic matching capabilities of embedding models. In addition, we combine Chain-of-Density (CoD) and the Self-Reflection method with query generation to improve quality and implement a hybrid cross-lingual extension that reflects real user querying practices. Evaluation of seven embedding models on the STELLA benchmark shows that large decoder-based embedding models exhibit the strongest semantic understanding, while lexical matching methods such as BM25 remain highly competitive in domains where exact lexical matching technical term is crucial. The STELLA benchmark provides a reproducible foundation for reliable performance evaluation and improvement of embedding models in aerospace-domain IR tasks. The STELLA benchmark can be found in https://huggingface.co/datasets/telepix/STELLA.
Paper Structure (42 sections, 2 figures, 12 tables)

This paper contains 42 sections, 2 figures, 12 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. General IR Benchmarks
  4. Domain-Specific IR Benchmarks
  5. Multilingual IR Benchmarks
  6. Synthetic Query Generation for IR
  7. The STELLA Benchmark: Construction Pipeline
  8. Data Collection
  9. Document Parsing
  10. Terminology Extraction
  11. Candidate Extraction
  12. Multi-stage Filtering
  13. Candidate Passage Selection
  14. Terminology-based Filtering
  15. Query Intent Classification
  16. ...and 27 more sections

Figures (2)

  • Figure 1: Overall pipeline for constructing the STELLA benchmark. The process comprises six systematic stages: (1) data collection from the NASA Technical Reports Server (NTRS), (2) document layout detection and passage chunking, (3) construction of a domain-specific terminology dictionary, (4) candidate passage selection based on query intents, (5) generation of dual-type synthetic queries (TCQ and TAQ) using Chain-of-Density and Self-Reflection, and (6) cross-lingual extension via multilingual query translation.
  • Figure 2: Quality Validation of Synthetic Queries via G-Eval. Performance comparison of synthetic query generation with (red solid line) and without (grey dashed line) Self-Reflection across five core metrics: Answerability, No External Knowledge, Intent Adherence, Format Compliance, and Style & Length. The top row presents large-scale models, showing significant improvement in Answerability and constraint adherence. The bottom row presents small-scale models, where the benefits of Self-Reflection are limited or negative due to reasoning limitations.