Towards Unlocking Insights from Logbooks Using AI
Antonin Sulc, Alex Bien, Annika Eichler, Daniel Ratner, Florian Rehm, Frank Mayet, Gregor Hartmann, Hayden Hoschouer, Henrik Tuennermann, Jan Kaiser, Jason St. John, Jennefer Maldonado, Kyle Hazelwood, Raimund Kammering, Thorsten Hellert, Tim Wilksen, Verena Kain, Wan-Lin Hu
TL;DR
This work tackles the underutilization of particle accelerator eLogs due to their technical language and privacy concerns by proposing a tailored Retrieval Augmented Generation (RAG) pipeline. It demonstrates cross-facility efforts (ALS, BNL, CERN, DESY/BESSY, Fermilab, SLAC) to tailor embeddings, vector stores, and re-ranking, while integrating multimodal data and metadata to ground LLM-generated insights in actual log content. The contributions include facility-specific RAG implementations, embedding strategies, and metadata-enabled retrieval, all aimed at improving findability, accessibility, interoperability, and reusability of eLogs and moving toward automation. While promising improvements in semantic search and grounded Q&A are shown, challenges remain in long-text embedding, data quality, and effective recall, motivating ongoing cross-institution benchmarking and methodological refinements.
Abstract
Electronic logbooks contain valuable information about activities and events concerning their associated particle accelerator facilities. However, the highly technical nature of logbook entries can hinder their usability and automation. As natural language processing (NLP) continues advancing, it offers opportunities to address various challenges that logbooks present. This work explores jointly testing a tailored Retrieval Augmented Generation (RAG) model for enhancing the usability of particle accelerator logbooks at institutes like DESY, BESSY, Fermilab, BNL, SLAC, LBNL, and CERN. The RAG model uses a corpus built on logbook contributions and aims to unlock insights from these logbooks by leveraging retrieval over facility datasets, including discussion about potential multimodal sources. Our goals are to increase the FAIR-ness (findability, accessibility, interoperability, and reusability) of logbooks by exploiting their information content to streamline everyday use, enable macro-analysis for root cause analysis, and facilitate problem-solving automation.