MARVEL: A Multi Agent-based Research Validator and Enabler using Large Language Models

TL;DR

MARVEL introduces a locally deployable, open-source framework for domain-aware scientific assistance that combines hybrid retrieval, open-weight LLM reasoning, and a DeepSearch module based on Monte Carlo Tree Search to perform multi-hop, citation-grounded research. By leveraging a global evidence ledger, answer-aware chunk filtering, and hierarchical LangGraph-based synthesis, MARVEL delivers transparent, source-backed responses while remaining compute-efficient for community-scale hardware. Evaluations on ArXivData and LogbookData demonstrate that MARVEL-Standard can match or exceed a GPT-4o mini baseline on literature-style queries and significantly outperform it on detector-operations content, with DeepSearch further enhancing recall, correctness, and faithfulness. The open-source release and built-in reproducibility provide a practical foundation for developing domain-specific scientific assistants without reliance on proprietary models, enabling broader participation and customization across research communities.

Abstract

We present MARVEL (https://ligogpt.mit.edu/marvel), a locally deployable, open-source framework for domain-aware question answering and assisted scientific research. It is designed to address the increasing demands of a digital assistant for scientific groups that can read highly technical data, cite precisely, and operate within authenticated networks. MARVEL combines a fast path for straightforward queries with a more deliberate DeepSearch mode that integrates retrieval-augmented generation and Monte Carlo Tree Search. It explores complementary subqueries, allocates more compute to promising branches, and maintains a global evidence ledger that preserves sources during drafting. We applied this framework in the context of gravitational-wave research related to the Laser Interferometer Gravitational-wave Observatory. Answers are grounded in a curated semantic index of research literature, doctoral theses, LIGO documents, and long-running detector electronic logbooks, with targeted web searches when appropriate. Because direct benchmarking against commercial LLMs cannot be performed on private data, we evaluated MARVEL on two publicly available surrogate datasets that capture comparable semantic and technical characteristics. On these benchmarks, MARVEL matches a GPT-4o mini baseline on literature-centric queries and substantially outperforms it on detector-operations content, where domain retrieval and guided reasoning are decisive. By making the complete framework and evaluation datasets openly available, we aim to provide a reproducible foundation for developing domain-specific scientific assistants.

Figures (7)

• Figure 1: UMAP projections of dense text embeddings, using 500 documents per category from the ten most frequent classes in each text corpus. (a) Logbook entries contain topics related to routine operational activity at the detector sites. (b) Technical documents cover topics related to LIGO and broader gravitational-wave related fields.
• Figure 2: Overview of the MARVEL architecture and the status of the current implementation. The central agent manages the helper modules, the available tools, and the retrieval system built from the document embeddings. The green box shows the parts implemented in this work. Yellow box indicates the research tasks that MARVEL aims to support. Future work is marked in purple.
• Figure 3: MARVEL workflow. A user query is first processed by the primary agent for task planning, domain detection, and initial reasoning (System-1). If agreement between fast-path answers is low, the query escalates to the System-2 orchestrator for retrieval-augmented generation (RAG) using hybrid semantic, lexical, and web-based searches. Relevant contexts are re-ranked, verified, and merged before synthesis into a citation-grounded draft. When necessary, the DeepSearch controller invokes a Monte Carlo Tree Search (MCTS) based exploration to collect complementary information, allowing the system to generate a cited report even when the evidence is spread across multiple sources.
• Figure 4: Blind A/B judging (GPT-4o mini as judge) on two publicly available surrogate datasets. Violin plots compare GPT-4o mini vs. MARVEL-Standard. On ArXivData, the judge slightly favors GPT-4o mini over MARVEL-Standard. On LogbookData, MARVEL-Standard is preferred, reflecting the benefit of domain retrieval on operational records.
• Figure 5: Evaluation of answer quality for GPT-4o mini baseline and MARVEL-Standard on different datasets. Answers are analyzed and evaluated using open-weight LLM, meta-llama/llama-4-scout-17b-16e-instruct.