Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

AILS-NTUA at SemEval-2026 Task 8: Evaluating Multi-Turn RAG Conversations

Dimosthenis Athanasiou, Maria Lymperaiou, Giorgos Filandrianos, Athanasios Voulodimos, Giorgos Stamou

TL;DR

Empirical analysis shows that query diversity over a well-aligned retriever outperforms heterogeneous retriever ensembling, and that answerability calibration-rather than retrieval coverage-is the primary bottleneck in end-to-end performance.

Abstract

We present the AILS-NTUA system for SemEval-2026 Task 8 (MTRAGEval), addressing all three subtasks of multi-turn retrieval-augmented generation: passage retrieval (A), reference-grounded response generation (B), and end-to-end RAG (C). Our unified architecture is built on two principles: (i) a query-diversity-over-retriever-diversity strategy, where five complementary LLM-based query reformulations are issued to a single corpus-aligned sparse retriever and fused via variance-aware nested Reciprocal Rank Fusion; and (ii) a multistage generation pipeline that decomposes grounded generation into evidence span extraction, dual-candidate drafting, and calibrated multi-judge selection. Our system ranks 1st in Task A (nDCG@5: 0.5776, +20.5% over the strongest baseline) and 2nd in Task B (HM: 0.7698). Empirical analysis shows that query diversity over a well-aligned retriever outperforms heterogeneous retriever ensembling, and that answerability calibration-rather than retrieval coverage-is the primary bottleneck in end-to-end performance.

AILS-NTUA at SemEval-2026 Task 8: Evaluating Multi-Turn RAG Conversations

TL;DR

Empirical analysis shows that query diversity over a well-aligned retriever outperforms heterogeneous retriever ensembling, and that answerability calibration-rather than retrieval coverage-is the primary bottleneck in end-to-end performance.

Abstract

We present the AILS-NTUA system for SemEval-2026 Task 8 (MTRAGEval), addressing all three subtasks of multi-turn retrieval-augmented generation: passage retrieval (A), reference-grounded response generation (B), and end-to-end RAG (C). Our unified architecture is built on two principles: (i) a query-diversity-over-retriever-diversity strategy, where five complementary LLM-based query reformulations are issued to a single corpus-aligned sparse retriever and fused via variance-aware nested Reciprocal Rank Fusion; and (ii) a multistage generation pipeline that decomposes grounded generation into evidence span extraction, dual-candidate drafting, and calibrated multi-judge selection. Our system ranks 1st in Task A (nDCG@5: 0.5776, +20.5% over the strongest baseline) and 2nd in Task B (HM: 0.7698). Empirical analysis shows that query diversity over a well-aligned retriever outperforms heterogeneous retriever ensembling, and that answerability calibration-rather than retrieval coverage-is the primary bottleneck in end-to-end performance.
Paper Structure (91 sections, 6 equations, 12 figures, 39 tables)

This paper contains 91 sections, 6 equations, 12 figures, 39 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Task description
  4. Related work
  5. System Overview
  6. Task A: Multi-Strategy Retrieval
  7. Task B: Agentic Generation Pipeline
  8. Task C: End-to-End RAG
  9. Experimental setup
  10. Dataset
  11. Task A Configuration.
  12. Task B Configuration.
  13. Task C Configuration.
  14. Evaluation Metrics
  15. Results and Analysis
  16. ...and 76 more sections

Figures (12)

  • Figure 1: System architecture: (a) Task A retrieval pipeline, (b) Task B generation pipeline, (c) Task C end-to-end RAG with answerability gate.
  • Figure 2: Question-type distribution per domain (dev set, % of domain turns). FiQA's Opinion-heavy profile and Cloud's even distribution across types stand in contrast to the Factoid/Summarization-dominated ClapNQ and Govt corpora.
  • Figure 3: Reference answer length distribution by question type (dev set). Red diamonds indicate means; Summarization answers are significantly longer, motivating type-specific generation targets.
  • Figure 4: Non-standalone phenomena in dev set non-first turns ($N=732$, multi-label). Pronoun coreference and implicit topic carryover are the most frequent sources of query underspecification, motivating history-aware query rewriting.
  • Figure 5: Turn position distribution for evaluated turns in the development (blue) and test (orange) sets. The dev set spans positions 1--10+, while all 507 test turns are non-first turns---eliminating the easy first-turn queries that inflate dev-set averages and ensuring every test turn requires contextual understanding.
  • ...and 7 more figures