Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Trust by Design: Skill Profiles for Transparent, Cost-Aware LLM Routing

Mika Okamoto, Ansel Kaplan Erol, Glenn Matlin

TL;DR

This paper tackles the problem of deploying LLMs under real-world budget constraints by introducing BELLA, a framework that grounds model selection in interpretable, multi-skill profiles derived from critic-based analyses of model outputs. It decomposes task performance into granular skills, clusters them into a structured capability matrix, and solves a constraint-aware optimization to maximize expected performance within a budget, while providing transparent, natural-language rationales for recommendations. The key contributions include critic-based skill profiling without predefined taxonomies, a matrix-based representation (C, R, c) that supports various retrieval and prediction strategies, and a leave-one-out evaluation framework to assess generalizability and cost-performance trade-offs on financial reasoning benchmarks. BELLA’s emphasis on interpretability and principled cost-aware decision-making offers a practical path to trustworthy, resource-efficient LLM deployment across domains with diverse skill requirements and constraints.

Abstract

How should Large Language Model (LLM) practitioners select the right model for a task without wasting money? We introduce BELLA (Budget-Efficient LLM Selection via Automated skill-profiling), a framework that recommends optimal LLM selection for tasks through interpretable skill-based model selection. Standard benchmarks report aggregate metrics that obscure which specific capabilities a task requires and whether a cheaper model could suffice. BELLA addresses this gap through three stages: (1) decomposing LLM outputs and extract the granular skills required by using critic-based profiling, (2) clustering skills into structured capability matrices, and (3) multi-objective optimization to select the right models to maximize performance while respecting budget constraints. BELLA provides natural-language rationale for recommendations, providing transparency that current black-box routing systems lack. We describe the framework architecture, situate it within the landscape of LLM routing and evaluation, and discuss its application to financial reasoning as a representative domain exhibiting diverse skill requirements and cost-variation across models. Our framework enables practitioners to make principled and cost-performance trade-offs for deploying LLMs.

Trust by Design: Skill Profiles for Transparent, Cost-Aware LLM Routing

TL;DR

This paper tackles the problem of deploying LLMs under real-world budget constraints by introducing BELLA, a framework that grounds model selection in interpretable, multi-skill profiles derived from critic-based analyses of model outputs. It decomposes task performance into granular skills, clusters them into a structured capability matrix, and solves a constraint-aware optimization to maximize expected performance within a budget, while providing transparent, natural-language rationales for recommendations. The key contributions include critic-based skill profiling without predefined taxonomies, a matrix-based representation (C, R, c) that supports various retrieval and prediction strategies, and a leave-one-out evaluation framework to assess generalizability and cost-performance trade-offs on financial reasoning benchmarks. BELLA’s emphasis on interpretability and principled cost-aware decision-making offers a practical path to trustworthy, resource-efficient LLM deployment across domains with diverse skill requirements and constraints.

Abstract

How should Large Language Model (LLM) practitioners select the right model for a task without wasting money? We introduce BELLA (Budget-Efficient LLM Selection via Automated skill-profiling), a framework that recommends optimal LLM selection for tasks through interpretable skill-based model selection. Standard benchmarks report aggregate metrics that obscure which specific capabilities a task requires and whether a cheaper model could suffice. BELLA addresses this gap through three stages: (1) decomposing LLM outputs and extract the granular skills required by using critic-based profiling, (2) clustering skills into structured capability matrices, and (3) multi-objective optimization to select the right models to maximize performance while respecting budget constraints. BELLA provides natural-language rationale for recommendations, providing transparency that current black-box routing systems lack. We describe the framework architecture, situate it within the landscape of LLM routing and evaluation, and discuss its application to financial reasoning as a representative domain exhibiting diverse skill requirements and cost-variation across models. Our framework enables practitioners to make principled and cost-performance trade-offs for deploying LLMs.
Paper Structure (27 sections, 4 figures, 1 table)

This paper contains 27 sections, 4 figures, 1 table.

Table of Contents

  1. Introduction
  2. Background and Problem Setting
  3. The Model Selection Challenge
  4. Desiderata for Model Selection
  5. Related Work
  6. Fine-grained LLM evaluation
  7. Model Routing and Selection
  8. Positioning of Bella.
  9. The Bella Framework
  10. Benchmarking
  11. Skill Profiling via Critic LLM
  12. Skill Clustering
  13. Capability Matrices.
  14. Generalizability of the Matrix Framework.
  15. Skill-aware Model Selection
  16. ...and 12 more sections

Figures (4)

  • Figure 1: Bella's architecture. A critic model analyzes LLM outputs on benchmark tasks to extract skill profiles. Given a new task and budget constraint, Bella transparently selects the cost-optimal choice.
  • Figure 2: Cost-performance trade-offs across LLMs on financial reasoning benchmarks. More expensive models achieve higher average scores, but marginal gains diminish and capability profiles vary substantially at similar price points.
  • Figure 3: Skill extraction via automated critique. Given a task, reference solution, and model output, the critic identifies demonstrated and missing skills.
  • Figure 4: Skill taxonomy construction. Raw skill descriptions are embedded, clustered, and assigned canonical labels to create a consistent vocabulary for capability representation.