ORXE: Orchestrating Experts for Dynamically Configurable Efficiency

TL;DR

ORXE tackles the problem of variable real-world resource constraints by orchestrating multiple pre-trained experts with increasing cost and accuracy. It employs a lightweight, confidence-based pre- and post-expert gating mechanism and a runtime trade-off parameter $\lambda \in [0,1]$ to achieve training-free, on-device configurable efficiency for image classification. Key contributions include a modular metamodel decoupled from training complexity, a data-driven configuration search with regularization and monotonicity safeguards, and extensive experiments showing efficiency gains over individual experts and many dynamic baselines across devices. The approach enables scalable deployment in diverse environments and can be extended to other tasks with suitable gate and expert designs.

Abstract

This paper presents ORXE, a modular and adaptable framework for achieving real-time configurable efficiency in AI models. By leveraging a collection of pre-trained experts with diverse computational costs and performance levels, ORXE dynamically adjusts inference pathways based on the complexity of input samples. Unlike conventional approaches that require complex metamodel training, ORXE achieves high efficiency and flexibility without complicating the development process. The proposed system utilizes a confidence-based gating mechanism to allocate appropriate computational resources for each input. ORXE also supports adjustments to the preference between inference cost and prediction performance across a wide range during runtime. We implemented a training-free ORXE system for image classification tasks, evaluating its efficiency and accuracy across various devices. The results demonstrate that ORXE achieves superior performance compared to individual experts and other dynamic models in most cases. This approach can be extended to other applications, providing a scalable solution for diverse real-world deployment scenarios.

Figures (9)

• Figure 1: The overview of proposed method. The system employs multiple pre-trained experts with incremental computational costs and performance levels. The input samples are dynamically routed through the experts based on its difficulty to experts. The overall system performance is adjustable by changing the threshold of each gate. $\mathit{Expert}^{(i)}$ and $\mathit{Gate}^{(i)}$ are the expert and the post-expert gate for the $i$-th node. $N_{\mathit{exp}}$ is the number of experts. The procedure starts with the first enabled node which produce the initial prediction and confidence. If the confidence is greater than the threshold $t_{2}^{(1)}$, the computation exits. Otherwise, it goes to the next enabled node. At the entry of an intermediate node, this node is skipped without any computation if the confidence from the previous node is too low.
• Figure 2: Comparison of ORXE metamodels with related work on ImageNet Val
• Figure 3: The speed test of ORXE and other models on different GPU and CPU with ONNX Runtimeonnxruntime
• Figure 4: Evaluation results on independent datasets
• Figure C1: The speed test of ORXE and other models on Raspberry Pi and GPU