Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Dynamic Fog Computing for Enhanced LLM Execution in Medical Applications

Philipp Zagar, Vishnu Ravi, Lauren Aalami, Stephan Krusche, Oliver Aalami, Paul Schmiedmayer

TL;DR

The paper addresses privacy, trust, and cost barriers to deploying large language models in healthcare by shifting LLM execution from cloud-centric to a decentralized fog computing architecture that brings inference closer to patients. It introduces SpeziLLM, an open-source, provider-agnostic Swift framework that orchestrates LLM tasks across edge, fog, and cloud, supported by a uniform mental model (Schema, Runner, Platform, Session) and retrieval-augmented generation via function calling. The authors demonstrate six mobile digital-health applications and report a positive usability survey from CS342 students, highlighting the framework’s practicality, ease of integration, and cross-layer capabilities. This approach holds promise for privacy-preserving, cost-efficient, and scalable LLM-enabled health tools, while outlining challenges related to context size, local-resource limits, and the need for continued improvements in mobile inference and interoperability.

Abstract

The ability of large language models (LLMs) to transform, interpret, and comprehend vast quantities of heterogeneous data presents a significant opportunity to enhance data-driven care delivery. However, the sensitive nature of protected health information (PHI) raises valid concerns about data privacy and trust in remote LLM platforms. In addition, the cost associated with cloud-based artificial intelligence (AI) services continues to impede widespread adoption. To address these challenges, we propose a shift in the LLM execution environment from opaque, centralized cloud providers to a decentralized and dynamic fog computing architecture. By executing open-weight LLMs in more trusted environments, such as the user's edge device or a fog layer within a local network, we aim to mitigate the privacy, trust, and financial challenges associated with cloud-based LLMs. We further present SpeziLLM, an open-source framework designed to facilitate rapid and seamless leveraging of different LLM execution layers and lowering barriers to LLM integration in digital health applications. We demonstrate SpeziLLM's broad applicability across six digital health applications, showcasing its versatility in various healthcare settings.

Dynamic Fog Computing for Enhanced LLM Execution in Medical Applications

TL;DR

The paper addresses privacy, trust, and cost barriers to deploying large language models in healthcare by shifting LLM execution from cloud-centric to a decentralized fog computing architecture that brings inference closer to patients. It introduces SpeziLLM, an open-source, provider-agnostic Swift framework that orchestrates LLM tasks across edge, fog, and cloud, supported by a uniform mental model (Schema, Runner, Platform, Session) and retrieval-augmented generation via function calling. The authors demonstrate six mobile digital-health applications and report a positive usability survey from CS342 students, highlighting the framework’s practicality, ease of integration, and cross-layer capabilities. This approach holds promise for privacy-preserving, cost-efficient, and scalable LLM-enabled health tools, while outlining challenges related to context size, local-resource limits, and the need for continued improvements in mobile inference and interoperability.

Abstract

The ability of large language models (LLMs) to transform, interpret, and comprehend vast quantities of heterogeneous data presents a significant opportunity to enhance data-driven care delivery. However, the sensitive nature of protected health information (PHI) raises valid concerns about data privacy and trust in remote LLM platforms. In addition, the cost associated with cloud-based artificial intelligence (AI) services continues to impede widespread adoption. To address these challenges, we propose a shift in the LLM execution environment from opaque, centralized cloud providers to a decentralized and dynamic fog computing architecture. By executing open-weight LLMs in more trusted environments, such as the user's edge device or a fog layer within a local network, we aim to mitigate the privacy, trust, and financial challenges associated with cloud-based LLMs. We further present SpeziLLM, an open-source framework designed to facilitate rapid and seamless leveraging of different LLM execution layers and lowering barriers to LLM integration in digital health applications. We demonstrate SpeziLLM's broad applicability across six digital health applications, showcasing its versatility in various healthcare settings.
Paper Structure (13 sections, 9 figures, 3 tables)

This paper contains 13 sections, 9 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Architecture
  4. Retrieval-Augmented Generation Capabilities
  5. Dynamic Task Dispatching
  6. SpeziLLM
  7. Cloud or remote layer
  8. Fog layer
  9. Edge or local layer
  10. Methods
  11. Results
  12. Discussion
  13. Conclusion

Figures (9)

  • Figure 1: The proposed transition of the inference environment from cloud-based platforms to the immediate proximity of the user's device.
  • Figure 2: Mental model of all interactions as a class diagram.
  • Figure 3: sequence diagram of the typical function calling mechanism integrated into our proposed mental model.
  • Figure 4: Typical procedure for inference job execution within the fog layer and our established mental model as a sequence diagram.
  • Figure 5: Swift code showcasing the usage of SpeziLLM's declarative function calling . The function fetches the health data from a patient record based on requested health data categories and returns the data to the .
  • ...and 4 more figures