Scalable and Secure AI Inference in Healthcare: A Comparative Benchmarking of FastAPI and Triton Inference Server on Kubernetes

TL;DR

This paper tackles the challenge of scalable and secure AI inference in regulated healthcare environments by empirically comparing a lightweight FastAPI REST gateway with the high-performance NVIDIA Triton Inference Server deployed on Kubernetes. Using a DistilBERT sentiment analysis model, it measures latency (p50/p95) and throughput under controlled conditions and demonstrates a clear trade-off: FastAPI yields lower latency for single requests, while Triton with dynamic batching delivers substantially higher throughput. The authors advocate a hybrid architecture that leverages FastAPI for secure input handling and preprocessing, coupled with Triton for GPU-backed inference, as an optimal configuration for enterprise clinical AI. The work provides a practical blueprint for secure, high-availability deployments in healthcare, balancing latency, throughput, and regulatory compliance.

Abstract

Efficient and scalable deployment of machine learning (ML) models is a prerequisite for modern production environments, particularly within regulated domains such as healthcare and pharmaceuticals. In these settings, systems must balance competing requirements, including minimizing inference latency for real-time clinical decision support, maximizing throughput for batch processing of medical records, and ensuring strict adherence to data privacy standards such as HIPAA. This paper presents a rigorous benchmarking analysis comparing two prominent deployment paradigms: a lightweight, Python-based REST service using FastAPI, and a specialized, high-performance serving engine, NVIDIA Triton Inference Server. Leveraging a reference architecture for healthcare AI, we deployed a DistilBERT sentiment analysis model on Kubernetes to measure median (p50) and tail (p95) latency, as well as throughput, under controlled experimental conditions. Our results indicate a distinct trade-off. While FastAPI provides lower overhead for single-request workloads with a p50 latency of 22 ms, Triton achieves superior scalability through dynamic batching, delivering a throughput of 780 requests per second on a single NVIDIA T4 GPU, nearly double that of the baseline. Furthermore, we evaluate a hybrid architectural approach that utilizes FastAPI as a secure gateway for protected health information de-identification and Triton for backend inference. This study validates the hybrid model as a best practice for enterprise clinical AI and offers a blueprint for secure, high-availability deployments.

Figures (1)

• Figure 2: Latency Distribution Comparison