Tuning LLM-based Code Optimization via Meta-Prompting: An Industrial Perspective

TL;DR

The paper tackles the cross-model prompt engineering bottleneck in industrial, multi-LLM code optimization by introducing Meta-Prompted Code Optimization (Mpco), which automatically generates model-adaptive prompts via a meta-prompting LLM using project, task, and LLM contexts integrated with the ARTEMIS platform. Through a four-stage workflow—profiling bottlenecks, context collection, multi-LLM optimization, and validated evaluation—Mpco demonstrates up to 19.06% runtime improvements across five real-world codebases and three LLM providers, with 366 hours of benchmarking. Comprehensive ablation studies show that incorporating comprehensive contextual information is essential for effectiveness, while sensitivity analysis confirms that multiple meta-prompters can yield strong results, offering practical flexibility for industrial deployment. The findings imply that automated, context-aware meta-prompting can substantially reduce maintenance costs and accelerate scalable, cross-model code optimization in production environments, guiding practitioners in context provisioning and LLM selection.

Abstract

There is a growing interest in leveraging multiple large language models (LLMs) for automated code optimization. However, industrial platforms deploying multiple LLMs face a critical challenge: prompts optimized for one LLM often fail with others, requiring expensive model-specific prompt engineering. This cross-model prompt engineering bottleneck severely limits the practical deployment of multi-LLM systems in production environments. We introduce Meta-Prompted Code Optimization (MPCO), a framework that automatically generates high-quality, task-specific prompts across diverse LLMs while maintaining industrial efficiency requirements. MPCO leverages metaprompting to dynamically synthesize context-aware optimization prompts by integrating project metadata, task requirements, and LLM-specific contexts. It is an essential part of the ARTEMIS code optimization platform for automated validation and scaling. Our comprehensive evaluation on five real-world codebases with 366 hours of runtime benchmarking demonstrates MPCO's effectiveness: it achieves overall performance improvements up to 19.06% with the best statistical rank across all systems compared to baseline methods. Analysis shows that 96% of the top-performing optimizations stem from meaningful edits. Through systematic ablation studies and meta-prompter sensitivity analysis, we identify that comprehensive context integration is essential for effective meta-prompting and that major LLMs can serve effectively as meta-prompters, providing actionable insights for industrial practitioners.

Figures (12)

• Figure 1: Artemis' multi-LLM code optimization workflow.
• Figure 2: Cross-model optimization results on Llama.cpp.
• Figure 3: The end-to-end workflow of Mpco.
• Figure 4: The Mpco meta-prompting template.
• Figure 5: Distribution of optimization types among the top 10% performing optimizations generated by Mpco.