Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Modular Foundation Model Inference at the Edge: Network-Aware Microservice Optimization

Juan Zhu, Zixin Wang, Shenghui Song, Jun Zhang, Khaled Ben Letaief

TL;DR

The paper tackles the challenge of running foundation model inference at the network edge under resource contention and uncertain network dynamics. It introduces a two-tier microservice framework that statically deploys heavyweight core MSs to form a fault-tolerant backbone and dynamically orchestrates lightweight light MSs to adapt to real-time workload fluctuations. Core placement is achieved via a sparsity-constrained integer program that balances cost, statistical QoS, and deployment diversity, while light-service allocation uses an online controller that combines effective capacity theory with Lyapunov optimization to provide probabilistic latency guarantees. Simulations demonstrate that the framework attains high on-time task completion (>84%) and robust performance as load scales, indicating a practical path to privacy-preserving, low-latency FM inference at the edge.

Abstract

Foundation models (FMs) unlock unprecedented multimodal and multitask intelligence, yet their cloud-centric deployment precludes real-time responsiveness and compromises user privacy. Meanwhile, monolithic execution at the edge remains infeasible under stringent resource limits and uncertain network dynamics. To bridge this gap, we propose a microservice-based FM inference framework that exploits the intrinsic functional asymmetry between heavyweight core services and agile light services. Our two-tier deployment strategy ensures robust Quality of Service (QoS) under resource contention. Specifically, core services are placed statically via a long-term network-aware integer program with sparsity constraints to form a fault-tolerant backbone. On the other hand, light services are orchestrated dynamically by a low-complexity online controller that integrates effective capacity theory with Lyapunov optimization, providing probabilistic latency guarantees under real-time workload fluctuations. Simulations demonstrate that our framework achieves over 84% average on-time task completion with moderate deployment costs and maintains strong robustness as the system load scales.

Modular Foundation Model Inference at the Edge: Network-Aware Microservice Optimization

TL;DR

The paper tackles the challenge of running foundation model inference at the network edge under resource contention and uncertain network dynamics. It introduces a two-tier microservice framework that statically deploys heavyweight core MSs to form a fault-tolerant backbone and dynamically orchestrates lightweight light MSs to adapt to real-time workload fluctuations. Core placement is achieved via a sparsity-constrained integer program that balances cost, statistical QoS, and deployment diversity, while light-service allocation uses an online controller that combines effective capacity theory with Lyapunov optimization to provide probabilistic latency guarantees. Simulations demonstrate that the framework attains high on-time task completion (>84%) and robust performance as load scales, indicating a practical path to privacy-preserving, low-latency FM inference at the edge.

Abstract

Foundation models (FMs) unlock unprecedented multimodal and multitask intelligence, yet their cloud-centric deployment precludes real-time responsiveness and compromises user privacy. Meanwhile, monolithic execution at the edge remains infeasible under stringent resource limits and uncertain network dynamics. To bridge this gap, we propose a microservice-based FM inference framework that exploits the intrinsic functional asymmetry between heavyweight core services and agile light services. Our two-tier deployment strategy ensures robust Quality of Service (QoS) under resource contention. Specifically, core services are placed statically via a long-term network-aware integer program with sparsity constraints to form a fault-tolerant backbone. On the other hand, light services are orchestrated dynamically by a low-complexity online controller that integrates effective capacity theory with Lyapunov optimization, providing probabilistic latency guarantees under real-time workload fluctuations. Simulations demonstrate that our framework achieves over 84% average on-time task completion with moderate deployment costs and maintains strong robustness as the system load scales.
Paper Structure (10 sections, 23 equations, 4 figures, 1 table, 1 algorithm)

This paper contains 10 sections, 23 equations, 4 figures, 1 table, 1 algorithm.

Table of Contents

  1. Introduction
  2. System Model and Problem Formulation
  3. Microservice Specification for FM Inference
  4. Latency Formulation under Network Uncertainty
  5. Problem Formulation
  6. Network-Aware Microservice Deployment for FM Edge Inference
  7. Reliable Core MS Deployment
  8. Dynamic Light MS Deployment
  9. Performance Evaluation
  10. Conclusions

Figures (4)

  • Figure 1: Illustration of FM-based inference application with the MS architecture. Squares denote core MSs, circles denote light MSs, and different line styles represent different task types of inter-service dependencies.
  • Figure 2: MS-based FM inference on a heterogeneous edge network.
  • Figure 3: Violin-plot comparison of on-time task completion rate and total system cost across four deployment strategies.
  • Figure 4: Comparison of the proposed framework and the PropAvg ablation under escalating system loads.