The M-factor: A Novel Metric for Evaluating Neural Architecture Search in Resource-Constrained Environments

TL;DR

The paper tackles the NAS efficiency problem by introducing M-Factor, a metric defined as $M = \frac{2 \times A \times S'}{A + S'}$ with $S' = \frac{P_{min}}{P}$, balancing accuracy $A$ and model size $P$ in resource-limited deployments. It evaluates four NAS strategies—Policy-based Reinforcement Learning, Regularized Evolution, Tree-structured Parzen Estimator, and Multi-trial Random Search—on a ResNet CIFAR-10 search space of $3^9=19{,}683$ configurations, demonstrating that RL and Regularized Evolution achieve the top M-Factor values ($0.84$ and $0.82$) while TPE lags ($0.67$). The results reveal distinct optimization dynamics, with RL improving late (after trial $39$) and Regularized Evolution showing rapid early gains (by trial $20$); Random search remains a strong baseline in some cases. Overall, the M-Factor metric addresses the accuracy-vs-size limitation of existing NAS evaluations and offers practical guidance for selecting NAS strategies in constrained environments, with future work aimed at expanding search spaces, datasets, and incorporating additional efficiency metrics.

Abstract

Neural Architecture Search (NAS) aims to automate the design of deep neural networks. However, existing NAS techniques often focus on maximising accuracy, neglecting model efficiency. This limitation restricts their use in resource-constrained environments like mobile devices and edge computing systems. Moreover, current evaluation metrics prioritise performance over efficiency, lacking a balanced approach for assessing architectures suitable for constrained scenarios. To address these challenges, this paper introduces the M-factor, a novel metric combining model accuracy and size. Four diverse NAS techniques are compared: Policy-Based Reinforcement Learning, Regularised Evolution, Tree-structured Parzen Estimator (TPE), and Multi-trial Random Search. These techniques represent different NAS paradigms, providing a comprehensive evaluation of the M-factor. The study analyses ResNet configurations on the CIFAR-10 dataset, with a search space of 19,683 configurations. Experiments reveal that Policy-Based Reinforcement Learning and Regularised Evolution achieved M-factor values of 0.84 and 0.82, respectively, while Multi-trial Random Search attained 0.75, and TPE reached 0.67. Policy-Based Reinforcement Learning exhibited performance changes after 39 trials, while Regularised Evolution optimised within 20 trials. The research investigates the optimisation dynamics and trade-offs between accuracy and model size for each strategy. Findings indicate that, in some cases, random search performed comparably to more complex algorithms when assessed using the M-factor. These results highlight how the M-factor addresses the limitations of existing metrics by guiding NAS towards balanced architectures, offering valuable insights for selecting strategies in scenarios requiring both performance and efficiency.

Figures (6)

• Figure 1: The M-factor values of Policy-based RL and Regularized Evolution techniques over 50 trials. Red lines indicate clear improvement trends.
• Figure 2: The M-factor values of TPE and Multi-trial Random. There is no clear improvement trend for these two techniques.
• Figure 3: Performance of top 20% trials of Policy-based RL.
• Figure 4: Performance of top 20% trials of Regularized Evolution.
• Figure 5: Performance of top 20% trials of TPE.