HybridFlow: Adaptive Task Scheduling for Fast and Token-Efficient LLM Inference in Edge-Cloud Collaboration

TL;DR

HybridFlow tackles the latency and token-cost challenges of real-time LLM inference on resource-constrained edge devices by enabling fine-grained, dependency-aware collaboration with a cloud model. It combines a DAG-based task planner with a resource-aware router that uses offline utility estimation and online budget management, grounded in a 0–1 knapsack formulation and a Lagrangian dual thresholding framework. The approach is augmented with online calibration via contextual bandits to adapt to shifts in latency and pricing, yielding an adaptive, parallelizable inference pipeline. Empirical results on GPQA, AIME, LiveBench-Reasoning, and MMLU-Pro show that HybridFlow reduces end-to-end latency and cloud token usage while maintaining competitive accuracy, offering a practical path to fast, token-efficient edge–cloud reasoning. Key mathematical ideas include modeling per-subtask offloading as a knapsack problem with normalized cost $c_i$ and value $\Delta q_i$, solved approximately via an online threshold $\tau_t$ linked to a dual price $\lambda_t$, and a learned utility predictor $\hat{u}_i$ that guides subtask routing under budget constraints. Offline profiling yields embeddings $z_i$ and a predictor $\hat{u}_i=\sigma(f_{\theta}(z_i))$ trained by MSE to the target $u_i = \frac{\Delta q_i}{c_i + \varepsilon}$. The combination of DAG planning, parallel execution, and budget-aware routing enables improved latency–cost–accuracy trade-offs in edge–cloud LLM inference.

Abstract

Large language models (LLMs) exhibit impressive reasoning and problem-solving abilities, yet their substantial inference latency and token consumption pose major challenges for real-time deployment on resource-limited edge devices. Recent efforts toward edge-cloud collaboration have attempted to mitigate this issue, but most existing methods adopt coarse-grained task allocation strategies-assigning entire queries either to the edge or the cloud. Such rigid partitioning fails to exploit fine-grained reasoning parallelism and often leads to redundant computation and inefficient resource utilization. To this end, we propose HybridFlow, a resource-adaptive inference framework that enables fast and token-efficient collaborative reasoning between edge and cloud LLMs. HybridFlow operates in two stages: (1) task decomposition and parallel execution, which dynamically splits a complex query into interdependent subtasks that can execute as soon as their dependencies are resolved; and (2) resource-aware subtask routing, where a learned router adaptively assigns each subtask to the edge or cloud model according to predicted utility gains and real-time budget states. Comprehensive evaluations on GPQA, MMLU-Pro, AIME, and LiveBench-Reasoning demonstrate that HybridFlow effectively reduces end-to-end inference time and overall token usage while maintaining competitive accuracy.

Figures (7)

• Figure 1: Overview of the HybridFlow framework. (I) Task Decomposition: The planner decomposes a complex query into a directed acyclic graph of subtasks with explicit dependencies. (II) Subtask Allocation: The router encodes each subtask with semantic and resource features, predicts its utility score considering quality, latency, and API cost, and adaptively allocates it to either the edge SLM or the cloud LLM for efficient collaboration.
• Figure 2: Overview of comparative LLM inference pipeline. HybridFlow uniquely integrates dependency-aware planning concurrently with parallel execution. This achieves an optimal balance between speed and reasoning quality by actively exploiting concurrent opportunities within a logically constrained workflow.
• Figure 3: Distribution of executed subtasks between the edge and cloud models across subtask positions on GPQA. Bars show the number of subtasks executed on the edge (purple) and on the cloud (blue) at each subtask index, and the line shows the average adaptive threshold at that position.
• Figure 4: Performance–cost trends under different fixed offload thresholds $\tau_0$ on the GPQA benchmark. Increasing $\tau_0$ makes the router more conservative, leading to lower offload rate and cost but gradually reducing accuracy.
• Figure 5: Results of our planner evaluation, assessing models on five key dimensions of task decomposition quality. We compare our two models (4B and 1.7B) against several leading standalone models, including Qwen3-235B-a22b-thinking-2507, Gemini-2.5-Pro, and Llama-3-8B-Instruct.

Theorems & Definitions (4)

• Definition 3.1: Normalized Cost
• Definition 3.2: Utility
• Definition 3.1: Subtask
• Definition 3.2: Valid decomposition