Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

The Agent Capability Problem: Predicting Solvability Through Information-Theoretic Bounds

Shahar Lutati

TL;DR

ACP reframes solvability under resource constraints as an information-theoretic problem. It defines the effective cost $C_{ ext{effective}} = (I_{ ext{total}}/ar{I}_s)ar{C}_s$ and proves a two-sided bound and a high-probability bound on expected search cost, enabling upfront solvability estimates and risk-aware budgeting. Empirical validation shows ACP bounds closely track actual performance and improve search efficiency over baselines across LLM-based and agentic tasks, including NP-hard graph-coloring problems. The framework also extends to approximation settings, offering a principled approach to resource allocation for autonomous problem solving.

Abstract

When should an autonomous agent commit resources to a task? We introduce the Agent Capability Problem (ACP), a framework for predicting whether an agent can solve a problem under resource constraints. Rather than relying on empirical heuristics, ACP frames problem-solving as information acquisition: an agent requires $\Itotal$ bits to identify a solution and gains $\Istep$ bits per action at cost $\Cstep$, yielding an effective cost $\Ceff = (\Itotal/\Istep), \Cstep$ that predicts resource requirements before search. We prove that $\Ceff$ lower-bounds expected cost and provide tight probabilistic upper bounds. Experimental validation shows that ACP predictions closely track actual agent performance, consistently bounding search effort while improving efficiency over greedy and random strategies. The framework generalizes across LLM-based and agentic workflows, linking principles from active learning, Bayesian optimization, and reinforcement learning through a unified information-theoretic lens. \

The Agent Capability Problem: Predicting Solvability Through Information-Theoretic Bounds

TL;DR

ACP reframes solvability under resource constraints as an information-theoretic problem. It defines the effective cost and proves a two-sided bound and a high-probability bound on expected search cost, enabling upfront solvability estimates and risk-aware budgeting. Empirical validation shows ACP bounds closely track actual performance and improve search efficiency over baselines across LLM-based and agentic tasks, including NP-hard graph-coloring problems. The framework also extends to approximation settings, offering a principled approach to resource allocation for autonomous problem solving.

Abstract

When should an autonomous agent commit resources to a task? We introduce the Agent Capability Problem (ACP), a framework for predicting whether an agent can solve a problem under resource constraints. Rather than relying on empirical heuristics, ACP frames problem-solving as information acquisition: an agent requires bits to identify a solution and gains bits per action at cost , yielding an effective cost that predicts resource requirements before search. We prove that lower-bounds expected cost and provide tight probabilistic upper bounds. Experimental validation shows that ACP predictions closely track actual agent performance, consistently bounding search effort while improving efficiency over greedy and random strategies. The framework generalizes across LLM-based and agentic workflows, linking principles from active learning, Bayesian optimization, and reinforcement learning through a unified information-theoretic lens. \

Paper Structure

This paper contains 19 sections, 3 theorems, 20 equations, 1 figure, 1 table, 1 algorithm.

Table of Contents

  1. Introduction
  2. Formal Framework
  3. Theoretical Bounds
  4. Stochastic Model
  5. Main Result
  6. High-Probability Bound
  7. Connections to Information-Theoretic Search
  8. Active Learning
  9. Bayesian Optimization
  10. Intrinsic Motivation in Reinforcement Learning
  11. Estimation Procedure
  12. Experimental Validation
  13. Experimental Validation on Graph Coloring
  14. Setup.
  15. ACP Bound.
  16. ...and 4 more sections

Key Result

Theorem 3.1

Under the assumptions above, where $M_2 = \sup_i \mathbb{E}[X_i^2]$ and $\mu_{\inf} = \inf_i \mu_i$.

Figures (1)

  • Figure 1: ACP predictions versus actual LLM agent performance for noisy slope identification across noise levels. Predictions serve as consistent lower bounds, with gaps increasing with problem difficulty.

Theorems & Definitions (12)

  • Definition 2.1: Solvability Criterion
  • Definition 2.2: Total Information Requirement
  • Definition 2.3: Information Gain per Action
  • Definition 2.4: Optimal Action Selection
  • Definition 2.5: Effective Cost
  • Theorem 3.1: Two-Sided Cost Bound
  • proof
  • Proposition 4.1: Monte Carlo Error
  • proof
  • Theorem 4.1: Surrogate Error Under RKHS
  • ...and 2 more