CollEX -- A Multimodal Agentic RAG System Enabling Interactive Exploration of Scientific Collections

TL;DR

CollEX presents a multimodal agentic Retrieval-Augmented Generation system to enable interactive exploration of large scientific collections. By integrating cross-modal embeddings, LVLMs, and a tool-augmented agent loop, it supports curiosity-driven search across text and imagery, with a modular architecture that can be deployed on Weaviate-backed data stores. The authors validate the approach via a proof-of-concept containing 64,469 records across 32 collections, illustrating educational scenarios and interdisciplinary connections. This work delivers a practical blueprint for scalable, interactive access to scientific knowledge, with potential to enhance learning, discovery, and research across diverse collections.

Abstract

In this paper, we introduce CollEx, an innovative multimodal agentic Retrieval-Augmented Generation (RAG) system designed to enhance interactive exploration of extensive scientific collections. Given the overwhelming volume and inherent complexity of scientific collections, conventional search systems often lack necessary intuitiveness and interactivity, presenting substantial barriers for learners, educators, and researchers. CollEx addresses these limitations by employing state-of-the-art Large Vision-Language Models (LVLMs) as multimodal agents accessible through an intuitive chat interface. By abstracting complex interactions via specialized agents equipped with advanced tools, CollEx facilitates curiosity-driven exploration, significantly simplifying access to diverse scientific collections and records therein. Our system integrates textual and visual modalities, supporting educational scenarios that are helpful for teachers, pupils, students, and researchers by fostering independent exploration as well as scientific excitement and curiosity. Furthermore, CollEx serves the research community by discovering interdisciplinary connections and complementing visual data. We illustrate the effectiveness of our system through a proof-of-concept application containing over 64,000 unique records across 32 collections from a local scientific collection from a public university.

Figures (14)

• Figure 1: An overview of the CollEX Agentic System.
• Figure 2: The CollEX Data Schema
• Figure 3: Examples records contained in the CollEX database.
• Figure 4: Overview of the CollEX system architecture.
• Figure 5: Show-casing CollEX general functionality.