Retrieval Augmented Decision-Making: A Requirements-Driven, Multi-Criteria Framework for Structured Decision Support

TL;DR

This work tackles the challenge of deriving actionable, transparent decisions from structurally complex industry documents. It introduces Retrieval Augmented Decision-Making (RAD), a framework that combines multi-criteria decision making with LLM-based semantic understanding to automatically build a weighted hierarchical decision model and generate structured, evidence-based decision reports. Key contributions include a hierarchical criteria modeling method, a multi-agent weighting mechanism for interpretability, and an end-to-end decision generation process that yields traceable reasoning. Empirical results show RAD outperforms retrieval-based baselines in detail, rationality, and structure across real-world datasets, underscoring its potential for scalable, transparent decision support in regulated and complex domains.

Abstract

Various industries have produced a large number of documents such as industrial plans, technical guidelines, and regulations that are structurally complex and content-wise fragmented. This poses significant challenges for experts and decision-makers in terms of retrieval and understanding. Although existing LLM-based Retrieval-Augmented Generation methods can provide context-related suggestions, they lack quantitative weighting and traceable reasoning paths, making it difficult to offer multi-level and transparent decision support. To address this issue, this paper proposes the RAD method, which integrates Multi-Criteria Decision Making with the semantic understanding capabilities of LLMs. The method automatically extracts key criteria from industry documents, builds a weighted hierarchical decision model, and generates structured reports under model guidance. The RAD framework introduces explicit weight assignment and reasoning chains in decision generation to ensure accuracy, completeness, and traceability. Experiments show that in various decision-making tasks, the decision reports generated by RAD significantly outperform existing methods in terms of detail, rationality, and structure, demonstrating its application value and potential in complex decision support scenarios.

Figures (2)

• Figure 1: The pipeline of the method.
• Figure 2: The figure displays the direct win rate percentages of the row condition over the column condition across two datasets and four evaluation metrics, based on 50 questions per comparison. Self-comparison win rates (diagonal) are shown as the expected 50% reference value.