Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

FIRST: Federated Inference Resource Scheduling Toolkit for Scientific AI Model Access

Aditya Tanikanti, Benoit Côté, Yanfei Guo, Le Chen, Nickolaus Saint, Ryan Chard, Ken Raffenetti, Rajeev Thakur, Thomas Uram, Ian Foster, Michael E. Papka, Venkatram Vishwanath

TL;DR

FIRST tackles the need for secure, scalable on-prem AI inference in scientific workflows by enabling Inference-as-a-Service across federated HPC clusters with an OpenAI-compatible API. The system combines a dedicated Inference Gateway API, Globus Compute for distributed execution, and backend model serving tools (primarily vLLM) to achieve cloud-like access on existing infrastructure, including auto-scaling and hot nodes for low latency. The authors demonstrate a federation-capable architecture, a web UI, and extensive performance evaluation showing strong throughput under load, effective batch processing, and meaningful comparisons to cloud offerings. This work highlights the practical viability and impact of integrating AI inference into HPC centers, preserving data governance while enabling large-scale token generation on premises.

Abstract

We present the Federated Inference Resource Scheduling Toolkit (FIRST), a framework enabling Inference-as-a-Service across distributed High-Performance Computing (HPC) clusters. FIRST provides cloud-like access to diverse AI models, like Large Language Models (LLMs), on existing HPC infrastructure. Leveraging Globus Auth and Globus Compute, the system allows researchers to run parallel inference workloads via an OpenAI-compliant API on private, secure environments. This cluster-agnostic API allows requests to be distributed across federated clusters, targeting numerous hosted models. FIRST supports multiple inference backends (e.g., vLLM), auto-scales resources, maintains "hot" nodes for low-latency execution, and offers both high-throughput batch and interactive modes. The framework addresses the growing demand for private, secure, and scalable AI inference in scientific workflows, allowing researchers to generate billions of tokens daily on-premises without relying on commercial cloud infrastructure.

FIRST: Federated Inference Resource Scheduling Toolkit for Scientific AI Model Access

TL;DR

FIRST tackles the need for secure, scalable on-prem AI inference in scientific workflows by enabling Inference-as-a-Service across federated HPC clusters with an OpenAI-compatible API. The system combines a dedicated Inference Gateway API, Globus Compute for distributed execution, and backend model serving tools (primarily vLLM) to achieve cloud-like access on existing infrastructure, including auto-scaling and hot nodes for low latency. The authors demonstrate a federation-capable architecture, a web UI, and extensive performance evaluation showing strong throughput under load, effective batch processing, and meaningful comparisons to cloud offerings. This work highlights the practical viability and impact of integrating AI inference into HPC centers, preserving data governance while enabling large-scale token generation on premises.

Abstract

We present the Federated Inference Resource Scheduling Toolkit (FIRST), a framework enabling Inference-as-a-Service across distributed High-Performance Computing (HPC) clusters. FIRST provides cloud-like access to diverse AI models, like Large Language Models (LLMs), on existing HPC infrastructure. Leveraging Globus Auth and Globus Compute, the system allows researchers to run parallel inference workloads via an OpenAI-compliant API on private, secure environments. This cluster-agnostic API allows requests to be distributed across federated clusters, targeting numerous hosted models. FIRST supports multiple inference backends (e.g., vLLM), auto-scales resources, maintains "hot" nodes for low-latency execution, and offers both high-throughput batch and interactive modes. The framework addresses the growing demand for private, secure, and scalable AI inference in scientific workflows, allowing researchers to generate billions of tokens daily on-premises without relying on commercial cloud infrastructure.
Paper Structure (39 sections, 5 figures, 1 table)

This paper contains 39 sections, 5 figures, 1 table.

Table of Contents

  1. Introduction
  2. Background and Related Work
  3. Commercial Cloud Solutions
  4. Inference Optimization Frameworks
  5. Inference Services at HPC Facilities
  6. Scientific Computing with Globus
  7. FIRST Architecture
  8. Inference Gateway API
  9. Role and Responsibilities
  10. Authentication and Authorization
  11. Globus Compute
  12. Role and Responsibilities
  13. Endpoint Configuration and Safety Features
  14. Access and Permissions
  15. Model Serving Tools
  16. ...and 24 more sections

Figures (5)

  • Figure 1: High-level overview of the FIRST framework integration with ALCF infrastructure and Globus services.
  • Figure 2: Inference service user chat interface.
  • Figure 3: Performance vs. request rate for FIRST vs. vLLM Direct for Llama 3.3 70B on a single Sophia node with 8 A100 GPUs, showing request throughput, output token throughput, median end-to-end latency, and duration vs. offered request rate.
  • Figure 4: Auto-scaling performance: Single vs. two, three, and four instances for Llama 3.3 70B on Sophia's A100 GPUs under maximum load.
  • Figure 5: Benchmark comparison of FIRST (Llama 3.1 8B) vs. OpenAI API (the $\sim$8B-parameter GPT-4o-mini). Note: OpenAI API results may be influenced by service-side rate limiting.