Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Agri-Query: A Case Study on RAG vs. Long-Context LLMs for Cross-Lingual Technical Question Answering

Julius Gun, Timo Oksanen

TL;DR

A detailed analysis of LLM performance in a specialized industrial domain and an open framework for similar evaluations, highlighting practical trade-offs and challenges is contributed.

Abstract

We present a case study evaluating large language models (LLMs) with 128K-token context windows on a technical question answering (QA) task. Our benchmark is built on a user manual for an agricultural machine, available in English, French, and German. It simulates a cross-lingual information retrieval scenario where questions are posed in English against all three language versions of the manual. The evaluation focuses on realistic "needle-in-a-haystack" challenges and includes unanswerable questions to test for hallucinations. We compare nine long-context LLMs using direct prompting against three Retrieval-Augmented Generation (RAG) strategies (keyword, semantic, hybrid), with an LLM-as-a-judge for evaluation. Our findings for this specific manual show that Hybrid RAG consistently outperforms direct long-context prompting. Models like Gemini 2.5 Flash and the smaller Qwen 2.5 7B achieve high accuracy (over 85%) across all languages with RAG. This paper contributes a detailed analysis of LLM performance in a specialized industrial domain and an open framework for similar evaluations, highlighting practical trade-offs and challenges.

Agri-Query: A Case Study on RAG vs. Long-Context LLMs for Cross-Lingual Technical Question Answering

TL;DR

A detailed analysis of LLM performance in a specialized industrial domain and an open framework for similar evaluations, highlighting practical trade-offs and challenges is contributed.

Abstract

We present a case study evaluating large language models (LLMs) with 128K-token context windows on a technical question answering (QA) task. Our benchmark is built on a user manual for an agricultural machine, available in English, French, and German. It simulates a cross-lingual information retrieval scenario where questions are posed in English against all three language versions of the manual. The evaluation focuses on realistic "needle-in-a-haystack" challenges and includes unanswerable questions to test for hallucinations. We compare nine long-context LLMs using direct prompting against three Retrieval-Augmented Generation (RAG) strategies (keyword, semantic, hybrid), with an LLM-as-a-judge for evaluation. Our findings for this specific manual show that Hybrid RAG consistently outperforms direct long-context prompting. Models like Gemini 2.5 Flash and the smaller Qwen 2.5 7B achieve high accuracy (over 85%) across all languages with RAG. This paper contributes a detailed analysis of LLM performance in a specialized industrial domain and an open framework for similar evaluations, highlighting practical trade-offs and challenges.

Paper Structure

This paper contains 35 sections, 1 equation, 31 figures, 14 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Materials
  4. Models / LLM
  5. User Manual
  6. Methods
  7. RAG system
  8. Keyword-based Retrieval
  9. Semantic Retrieval
  10. Hybrid Retrieval
  11. Long-Context Testing
  12. Results
  13. Long-Context QA Performance
  14. RAG Performance
  15. Cross-lingual Performance using Hybrid RAG
  16. ...and 20 more sections

Figures (31)

  • Figure 1: Process of data acquisition (Illustrations created by the authors using Microsoft Bing Image Creator powered by DALL-E 3).
  • Figure 2: Long-Context QA: F1 score vs. noise: English.
  • Figure 3: F1 comparison for English language across RAG retrieval and Full Manual (59k tokens).
  • Figure 4: Accuracy comparison across different languages.
  • Figure 5: F1 score comparison across different languages.
  • ...and 26 more figures