Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LLM as HPC Expert: Extending RAG Architecture for HPC Data

Yusuke Miyashita, Patrick Kin Man Tung, Johan Barthélemy

TL;DR

The paper tackles the accessibility and safety challenges of using large language models in HPC environments by introducing HyCE, a Hypothetical Command Embeddings extension to Retrieval-Augmented Generation that ingests cluster documentation and real-time command outputs. It presents an automated evaluation framework where the LLM generates synthetic HPC data and serves as a judge to assess RAG performance with HPC-specific metrics, while addressing data privacy and command execution security. Empirical results show HyCE improves context relevance and user-specific accuracy, and the approach remains compatible with other RAG enhancements, offering a scalable path for deploying LLMs as HPC experts. The work also emphasizes secure deployment practices and open-sourcing the extended RAG tooling to facilitate adoption across HPC sites.

Abstract

High-Performance Computing (HPC) is crucial for performing advanced computational tasks, yet their complexity often challenges users, particularly those unfamiliar with HPC-specific commands and workflows. This paper introduces Hypothetical Command Embeddings (HyCE), a novel method that extends Retrieval-Augmented Generation (RAG) by integrating real-time, user-specific HPC data, enhancing accessibility to these systems. HyCE enriches large language models (LLM) with real-time, user-specific HPC information, addressing the limitations of fine-tuned models on such data. We evaluate HyCE using an automated RAG evaluation framework, where the LLM itself creates synthetic questions from the HPC data and serves as a judge, assessing the efficacy of the extended RAG with the evaluation metrics relevant for HPC tasks. Additionally, we tackle essential security concerns, including data privacy and command execution risks, associated with deploying LLMs in HPC environments. This solution provides a scalable and adaptable approach for HPC clusters to leverage LLMs as HPC expert, bridging the gap between users and the complex systems of HPC.

LLM as HPC Expert: Extending RAG Architecture for HPC Data

TL;DR

The paper tackles the accessibility and safety challenges of using large language models in HPC environments by introducing HyCE, a Hypothetical Command Embeddings extension to Retrieval-Augmented Generation that ingests cluster documentation and real-time command outputs. It presents an automated evaluation framework where the LLM generates synthetic HPC data and serves as a judge to assess RAG performance with HPC-specific metrics, while addressing data privacy and command execution security. Empirical results show HyCE improves context relevance and user-specific accuracy, and the approach remains compatible with other RAG enhancements, offering a scalable path for deploying LLMs as HPC experts. The work also emphasizes secure deployment practices and open-sourcing the extended RAG tooling to facilitate adoption across HPC sites.

Abstract

High-Performance Computing (HPC) is crucial for performing advanced computational tasks, yet their complexity often challenges users, particularly those unfamiliar with HPC-specific commands and workflows. This paper introduces Hypothetical Command Embeddings (HyCE), a novel method that extends Retrieval-Augmented Generation (RAG) by integrating real-time, user-specific HPC data, enhancing accessibility to these systems. HyCE enriches large language models (LLM) with real-time, user-specific HPC information, addressing the limitations of fine-tuned models on such data. We evaluate HyCE using an automated RAG evaluation framework, where the LLM itself creates synthetic questions from the HPC data and serves as a judge, assessing the efficacy of the extended RAG with the evaluation metrics relevant for HPC tasks. Additionally, we tackle essential security concerns, including data privacy and command execution risks, associated with deploying LLMs in HPC environments. This solution provides a scalable and adaptable approach for HPC clusters to leverage LLMs as HPC expert, bridging the gap between users and the complex systems of HPC.
Paper Structure (23 sections, 5 figures, 2 tables, 1 algorithm)

This paper contains 23 sections, 5 figures, 2 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. LLMs in HPC Environments
  4. Retrieval-Augmented Generation (RAG)
  5. RAG with HPC Data
  6. Approach
  7. HPC data
  8. Hypothetical Command Embeddings (HyCE)
  9. Automatic RAG evaluation with HPC data
  10. Question & Answer Generation and Filtering
  11. Evaluation of RAG via LLM as a Judge
  12. Experiment and Discussion
  13. Qualitative Analysis of RAG with HPC Data
  14. Orthogonality of HyCE to Other RAG Improvement Methods
  15. Limitations of the Automatic RAG Evaluation
  16. ...and 8 more sections

Figures (5)

  • Figure 1: Example of comparing of two RAG responses. Unlike the ambiguous response provided by RAG with just HPC documents, RAG combined with HyCE can address user specific HPC questions more precisely.
  • Figure 2: The prompts for RAG Q&A generation
  • Figure 3: The prompts for RAG Q&A filtering
  • Figure 4: The prompts for RAG answering
  • Figure 5: The prompts for RAG evaluation