Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

From Questions to Trust Reports: A LLM-IR Framework for the TREC 2025 DRAGUN Track

Ignacy Alwasiak, Kene Nnolim, Jaclyn Thi, Samy Ateia, Markus Bink, Gregor Donabauer, David Elsweiler, Udo Kruschwitz

Abstract

The DRAGUN Track at TREC 2025 targets the growing need for effective support tools that help users evaluate the trustworthiness of online news. We describe the UR_Trecking system submitted for both Task 1 (critical question generation) and Task 2 (retrieval-augmented trustworthiness reporting). Our approach combines LLM-based question generation with semantic filtering, diversity enforcement using clustering, and several query expansion strategies (including reasoning-based Chain-of-Thought expansion) to retrieve relevant evidence from the MS MARCO V2.1 segmented corpus. Retrieved documents are re-ranked using a monoT5 model and filtered using an LLM relevance judge together with a domain-level trustworthiness dataset. For Task 2, selected evidence is synthesized by an LLM into concise trustworthiness reports with citations. Results from the official evaluation indicate that Chain-of-Thought query expansion and re-ranking substantially improve both relevance and domain trust compared to baseline retrieval, while question-generation performance shows moderate quality with room for improvement. We conclude by outlining key challenges encountered and suggesting directions for enhancing robustness and trustworthiness assessment in future iterations of the system.

From Questions to Trust Reports: A LLM-IR Framework for the TREC 2025 DRAGUN Track

Abstract

The DRAGUN Track at TREC 2025 targets the growing need for effective support tools that help users evaluate the trustworthiness of online news. We describe the UR_Trecking system submitted for both Task 1 (critical question generation) and Task 2 (retrieval-augmented trustworthiness reporting). Our approach combines LLM-based question generation with semantic filtering, diversity enforcement using clustering, and several query expansion strategies (including reasoning-based Chain-of-Thought expansion) to retrieve relevant evidence from the MS MARCO V2.1 segmented corpus. Retrieved documents are re-ranked using a monoT5 model and filtered using an LLM relevance judge together with a domain-level trustworthiness dataset. For Task 2, selected evidence is synthesized by an LLM into concise trustworthiness reports with citations. Results from the official evaluation indicate that Chain-of-Thought query expansion and re-ranking substantially improve both relevance and domain trust compared to baseline retrieval, while question-generation performance shows moderate quality with room for improvement. We conclude by outlining key challenges encountered and suggesting directions for enhancing robustness and trustworthiness assessment in future iterations of the system.
Paper Structure (14 sections, 5 figures, 1 table)

This paper contains 14 sections, 5 figures, 1 table.

Table of Contents

  1. Introduction
  2. Methods
  3. Data and Resources
  4. Question Generation and Filtering
  5. Retrieval
  6. Re-ranking and Filtering
  7. Answer and Report Generation
  8. Evaluation
  9. Results
  10. Runs Submitted
  11. Question Generation (Task 1)
  12. Retrieval and Report Generation (Task 2)
  13. Discussion
  14. Conclusion

Figures (5)

  • Figure 1: Flowchart of our pipeline for Tasks 1 and 2.
  • Figure 2: Relevance scores for the first 20 example TREC questions before and after re-ranking across all four query expansion methods.
  • Figure 3: Domain trustworthiness scores for the first 20 example TREC questions before and after re-ranking across all four query expansion methods.
  • Figure 4: Change in relevance scores after re-ranking for all four query expansion methods.
  • Figure 5: Change in domain trustworthiness scores after re-ranking for all four query expansion methods.