Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Task-oriented Age of Information for Remote Inference with Hybrid Language Models

Shuying Gan, Xijun Wang, Chenyuan Feng, Chao Xu, Howard H. Yang, Xiang Chen, Tony Q. S. Quek

TL;DR

This work tackles timely, accurate remote inference by integrating both small and large language models and dynamically selecting image resolution. The authors model the system as a finite Semi-Markov Decision Process with state $s_t$ representing TAoI and actions $oldsymbol{a}_t=(a_t^u,a_t^c)$, where the decision epoch length is $L(oldsymbol{a}_t)$, and they convert it to an equivalent discrete-time MDP with per-step cost $ar{R}(oldsymbol{ riangle_t},oldsymbol{a}_t)= rac{R(oldsymbol{ riangle_t},oldsymbol{a}_t)}{L(oldsymbol{a}_t)}$. They prove the optimal policy has a threshold structure and propose a Relative Policy Iteration (RPI) algorithm that exploits this property to compute the policy efficiently. Simulations show the proposed policy achieves lower long-run TAoI than random or greedy baselines and reveal insights such as TAoI plateaus under high transmission latency and favorable alignment with greedy decisions when latency dominates, demonstrating practical benefits for edge remote inference with hybrid models.

Abstract

Large Language Models (LLMs) have revolutionized the field of artificial intelligence (AI) through their advanced reasoning capabilities, but their extensive parameter sets introduce significant inference latency, posing a challenge to ensure the timeliness of inference results. While Small Language Models (SLMs) offer faster inference speeds with fewer parameters, they often compromise accuracy on complex tasks. This study proposes a novel remote inference system comprising a user, a sensor, and an edge server that integrates both model types alongside a decision maker. The system dynamically determines the resolution of images transmitted by the sensor and routes inference tasks to either an SLM or LLM to optimize performance. The key objective is to minimize the Task-oriented Age of Information (TAoI) by jointly considering the accuracy and timeliness of the inference task. Due to the non-uniform transmission time and inference time, we formulate this problem as a Semi-Markov Decision Process (SMDP). By converting the SMDP to an equivalent Markov decision process, we prove that the optimal control policy follows a threshold-based structure. We further develop a relative policy iteration algorithm leveraging this threshold property. Simulation results demonstrate that our proposed optimal policy significantly outperforms baseline approaches in managing the accuracy-timeliness trade-off.

Task-oriented Age of Information for Remote Inference with Hybrid Language Models

TL;DR

This work tackles timely, accurate remote inference by integrating both small and large language models and dynamically selecting image resolution. The authors model the system as a finite Semi-Markov Decision Process with state representing TAoI and actions , where the decision epoch length is , and they convert it to an equivalent discrete-time MDP with per-step cost . They prove the optimal policy has a threshold structure and propose a Relative Policy Iteration (RPI) algorithm that exploits this property to compute the policy efficiently. Simulations show the proposed policy achieves lower long-run TAoI than random or greedy baselines and reveal insights such as TAoI plateaus under high transmission latency and favorable alignment with greedy decisions when latency dominates, demonstrating practical benefits for edge remote inference with hybrid models.

Abstract

Large Language Models (LLMs) have revolutionized the field of artificial intelligence (AI) through their advanced reasoning capabilities, but their extensive parameter sets introduce significant inference latency, posing a challenge to ensure the timeliness of inference results. While Small Language Models (SLMs) offer faster inference speeds with fewer parameters, they often compromise accuracy on complex tasks. This study proposes a novel remote inference system comprising a user, a sensor, and an edge server that integrates both model types alongside a decision maker. The system dynamically determines the resolution of images transmitted by the sensor and routes inference tasks to either an SLM or LLM to optimize performance. The key objective is to minimize the Task-oriented Age of Information (TAoI) by jointly considering the accuracy and timeliness of the inference task. Due to the non-uniform transmission time and inference time, we formulate this problem as a Semi-Markov Decision Process (SMDP). By converting the SMDP to an equivalent Markov decision process, we prove that the optimal control policy follows a threshold-based structure. We further develop a relative policy iteration algorithm leveraging this threshold property. Simulation results demonstrate that our proposed optimal policy significantly outperforms baseline approaches in managing the accuracy-timeliness trade-off.

Paper Structure

This paper contains 8 sections, 4 theorems, 16 equations, 4 figures, 1 table, 1 algorithm.

Table of Contents

  1. Introduction
  2. System Model
  3. SMDP Formulation and Solution
  4. SMDP Formulation
  5. Structural Analysis and Optimal Policy
  6. Simulation Results
  7. Conclusions
  8. Proof of Theorem 1

Key Result

Lemma 1

The value function $V(\Delta)$ is non-decreasing with $\Delta$.

Figures (4)

  • Figure 1: An illustration of the remote inference system with hybrid SLM and LLM.
  • Figure 2: Structure of the optimal policy for different $T_{2}^{u}$ ($T_{1}^{u}=4$, $T_{1}^{c}=3$, $T_{2}^{c}=4$, $p_{s}=0.3$, $q_{s}=0.7$, $p_{l}=0.5$, $q_{l}=0.8$).
  • Figure 3: Average TAoI versus $T_{2}^{u}$ or $T_{1}^{c}$ ($p_{s}=0.4$, $q_{s}=0.5$, $p_{l}=0.6$, $q_{l}=0.8$).
  • Figure 4: Average TAoI versus $p_{s}$ or $q_{l}$ ($T_{1}^{u}=3$, $T_{2}^{u}=4$, $T_{1}^{c}=8$, $T_{2}^{c}=10$).

Theorems & Definitions (4)

  • Lemma 1
  • Lemma 2
  • Lemma 3
  • Theorem 4