PD$^3$: A Project Duplication Detection Framework via Adapted Multi-Agent Debate

TL;DR

PD$^3$ addresses the challenge of detecting project duplication in large-scale domains by combining an adapted multi-agent debate with LLM-based feedback. Its round-robin MAD retrieval balances global information with context length while producing a top-five set of reference projects, complemented by both quantitative duplication scores and qualitative similarity analyses. Validation on 833 SGCC power projects and deployment via Review Dingdang demonstrate substantial improvements over baselines and tangible real-world impact through millions of dollars saved. The work advances human-centered evaluation in technical review by delivering actionable insights alongside robust retrieval, with potential for broader domain adoption.

Abstract

Project duplication detection is critical for project quality assessment, as it improves resource utilization efficiency by preventing investing in newly proposed project that have already been studied. It requires the ability to understand high-level semantics and generate constructive and valuable feedback. Existing detection methods rely on basic word- or sentence-level comparison or solely apply large language models, lacking valuable insights for experts and in-depth comprehension of project content and review criteria. To tackle this issue, we propose PD$^3$, a Project Duplication Detection framework via adapted multi-agent Debate. Inspired by real-world expert debates, it employs a fair competition format to guide multi-agent debate to retrieve relevant projects. For feedback, it incorporates both qualitative and quantitative analysis to improve its practicality. Over 800 real-world power project data spanning more than 20 specialized fields are used to evaluate the framework, demonstrating that our method outperforms existing approaches by 7.43% and 8.00% in two downstream tasks. Furthermore, we establish an online platform, Review Dingdang, to assist power experts, saving 5.73 million USD in initial detection on more than 100 newly proposed projects.

Figures (9)

• Figure 1: (a) Case on the lack of strict partial order. This case shows exhibiting ranking intransitivity: Reference projects A, B, and C form a pair-wise comparison cycle because each demonstrates closer relevance than the other in either research content, key technology, or application results comparing to the target project. (b) Top-5 coverage ratio in number of retrieved reference projects.
• Figure 2: Review Dingdang, the power project duplication platform based on PD$^3$.
• Figure 3: Overview of PD$^3$ framework. According to the I/O sequence of the project under detection, the framework consists of four parts: data pre-processing, database and preliminary retrieval, MAD-based round-robin retrieval and LLM-as-a-Judge-based Feedback.
• Figure 4: (a) Performance comparison across different group size. (b) Performance comparison across different number of debate rounds. (c) Performance comparison across different number of debate agents.
• Figure 5: A case study from the Review DingDang detection process conducted by PD$^3$. In one sub-competition phase, three experts first independently select their top-5 choices. After debating their differing perspectives, Expert A ultimately accepts the views of Experts B and C. A senior judge then makes the final decision. With the voted top-5 reference projects proceeding to the feedback stage for quantitative scoring and qualitative feedback.