MetaLLMix : An XAI Aided LLM-Meta-learning Based Approach for Hyper-parameters Optimization
Mohamed Bal-Ghaoui, Mohammed Tiouti
TL;DR
This work tackles the costly process of hyperparameter and model selection in deep learning by introducing MetaLLMix, a zero-shot optimization framework that fuses meta-learning, explainable AI, and lightweight LLM reasoning. It builds a meta-dataset from eight medical-imaging tasks, trains an XGBoost meta-learner, and uses SHAP explanations to guide an LLM-based recommendation system powered by Retrieval-Augmented Generation over FAISS. The approach yields competitive accuracy while delivering substantial reductions in response and training times, including improvements of up to $2.4\times$ to $15.7\times$ in training speed and over $99.6\%$ to $99.9\%$ faster response times on many tasks, with best results on several datasets using edge-deployable, open-source models. SHAP-driven explanations provide transparent, evidence-based justifications for hyperparameter and model choices, supporting trustworthy AutoML in resource-constrained environments and highlighting avenues for broader domain generalization and multi-objective optimization in future work.
Abstract
Effective model and hyperparameter selection remains a major challenge in deep learning, often requiring extensive expertise and computation. While AutoML and large language models (LLMs) promise automation, current LLM-based approaches rely on trial and error and expensive APIs, which provide limited interpretability and generalizability. We propose MetaLLMiX, a zero-shot hyperparameter optimization framework combining meta-learning, explainable AI, and efficient LLM reasoning. By leveraging historical experiment outcomes with SHAP explanations, MetaLLMiX recommends optimal hyperparameters and pretrained models without additional trials. We further employ an LLM-as-judge evaluation to control output format, accuracy, and completeness. Experiments on eight medical imaging datasets using nine open-source lightweight LLMs show that MetaLLMiX achieves competitive or superior performance to traditional HPO methods while drastically reducing computational cost. Our local deployment outperforms prior API-based approaches, achieving optimal results on 5 of 8 tasks, response time reductions of 99.6-99.9%, and the fastest training times on 6 datasets (2.4-15.7x faster), maintaining accuracy within 1-5% of best-performing baselines.