Finding the Sweet Spot: Trading Quality, Cost, and Speed During Inference-Time LLM Reflection
Jack Butler, Nikita Kozodoi, Zainab Afolabi, Brian Tyacke, Gaiar Baimuratov
TL;DR
This work systematically compares inference-time optimization techniques—self-reflection and budget tuning—across mathematical reasoning, translation, text-to-SQL, and sentiment analysis tasks using a diverse set of LLMs. It constructs Pareto frontiers in accuracy-cost-latency space, analyzes how reflection depth and feedback quality affect performance, and validates findings in a real-world deployment for marketing localisation. Key results show strong domain dependence (e.g., up to 220% gains in math) and that a single reflection round often captures most benefits, while translation and SQL tasks exhibit more mixed outcomes. The study provides practical, domain-specific guidance for selecting inference-time strategies under resource constraints and shares an open-source reflection implementation for reproducibility.
Abstract
As Large Language Models (LLMs) continue to evolve, practitioners face increasing options for enhancing inference-time performance without model retraining, including budget tuning and multi-step techniques like self-reflection. While these methods improve output quality, they create complex trade-offs among accuracy, cost, and latency that remain poorly understood across different domains. This paper systematically compares self-reflection and budget tuning across mathematical reasoning and translation tasks. We evaluate prominent LLMs, including Anthropic Claude, Amazon Nova, and Mistral families, along with other models under varying reflection depths and compute budgets to derive Pareto optimal performance frontiers. Our analysis reveals substantial domain dependent variation in self-reflection effectiveness, with performance gains up to 220\% in mathematical reasoning. We further investigate how reflection round depth and feedback mechanism quality influence performance across model families. To validate our findings in a real-world setting, we deploy a self-reflection enhanced marketing content localisation system at Lounge by Zalando, where it shows market-dependent effectiveness, reinforcing the importance of domain specific evaluation when deploying these techniques. Our results provide actionable guidance for selecting optimal inference strategies given specific domains and resource constraints. We open source our self-reflection implementation for reproducibility at https://github.com/aws-samples/sample-genai-reflection-for-bedrock.