Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

AWARE-US: Benchmark for Preference-Aware Resolution in Tool-Calling Agents

Mehmet Kurmaz

TL;DR

The paper tackles how tool-calling agents handle underspecification and infeasibility in querying structured data, introducing Preference-Aware Resolution. It proposes AWARE-US, a persona-grounded dataset in the car domain, and three LLM-based methods (local weighting, global one-shot weighting, and pairwise ranking) to infer constraint importance from dialogue for repair and ranking. Through experiments, weighting-based approaches—especially local weighting—achieve stronger alignment with oracle preferences and better downstream recommendations, with SFT+Direct Preference Optimization further improving performance. The work provides a benchmark and a principled framework for aligning constraint-relaxation decisions with user preferences in interactive, tool-using agents.

Abstract

Tool-calling conversational agents querying structured databases often face two linked failures: underspecification (missing constraints needed to run a precise query) and infeasibility (the fully specified query returns an empty set because no item satisfies all constraints). Existing work often responds with "no results" or relaxes constraints using ad hoc rules, which can violate user intent by discarding requirements the user cares about most. We frame infeasibility handling as a preference-aware query repair problem: when a query is unsatisfiable, the agent should relax the least important constraints to the user. We propose three LLM-based methods for inferring relative constraint importance from dialogue: (1) local weighting, (2) global one-shot weighting, and (3) pairwise ranking. Experiments show local weighting achieves the best preference alignment, while global weighting performs best on correct constraint relaxation. We also introduce AWARE-US, a benchmark of persona-grounded queries requiring agents to disambiguate requests via conversation and resolve infeasibility in a way consistent with persona-implied preferences.

AWARE-US: Benchmark for Preference-Aware Resolution in Tool-Calling Agents

TL;DR

The paper tackles how tool-calling agents handle underspecification and infeasibility in querying structured data, introducing Preference-Aware Resolution. It proposes AWARE-US, a persona-grounded dataset in the car domain, and three LLM-based methods (local weighting, global one-shot weighting, and pairwise ranking) to infer constraint importance from dialogue for repair and ranking. Through experiments, weighting-based approaches—especially local weighting—achieve stronger alignment with oracle preferences and better downstream recommendations, with SFT+Direct Preference Optimization further improving performance. The work provides a benchmark and a principled framework for aligning constraint-relaxation decisions with user preferences in interactive, tool-using agents.

Abstract

Tool-calling conversational agents querying structured databases often face two linked failures: underspecification (missing constraints needed to run a precise query) and infeasibility (the fully specified query returns an empty set because no item satisfies all constraints). Existing work often responds with "no results" or relaxes constraints using ad hoc rules, which can violate user intent by discarding requirements the user cares about most. We frame infeasibility handling as a preference-aware query repair problem: when a query is unsatisfiable, the agent should relax the least important constraints to the user. We propose three LLM-based methods for inferring relative constraint importance from dialogue: (1) local weighting, (2) global one-shot weighting, and (3) pairwise ranking. Experiments show local weighting achieves the best preference alignment, while global weighting performs best on correct constraint relaxation. We also introduce AWARE-US, a benchmark of persona-grounded queries requiring agents to disambiguate requests via conversation and resolve infeasibility in a way consistent with persona-implied preferences.
Paper Structure (40 sections, 11 equations, 2 figures, 6 tables, 2 algorithms)

This paper contains 40 sections, 11 equations, 2 figures, 6 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. AWARE-US: Dataset Construction
  4. Data source and query factorization
  5. Minimal infeasibility (MUS-4) as the core hardness principle
  6. Predicate generation with quality-preserving numeric thresholds
  7. Oracle weights, relaxation target, and gold recommendation
  8. Personas and dialogue supervision
  9. Benchmark settings
  10. Setting S1: MUS-4 (unique repair).
  11. Setting S2: MUS-4 (any-one repair).
  12. Setting S3: MUS-2 (any-one repair).
  13. Infeasibility Task
  14. Baseline: solver-style satisfiability + repair.
  15. Preference-aware repair: three variants.
  16. ...and 25 more sections

Figures (2)

  • Figure 1: Infeasible query handling. (a) Incorrect relaxation (wasted tokens) leads to an irrelevant recommendation, reduced user satisfaction, and poorer computational efficiency. (b) Correct relaxation (relevant result) yields a relevant recommendation, improving both user satisfaction and computational efficiency..
  • Figure 2: Overview of our Infeasibility-Aware Resolution Framework: Given a user query, specifying desired attributes and soft preferences, our framework elicits missing constraints via dialogue, extracts the resulting constraint set, and estimates preference weights (local weighting, global weighting, or pairwise ranking). The weighted constraints are used to query the constraint database; if the query is infeasible, the agent relaxes the lowest-importance constraint(s) to recover a feasible candidate and recommends optimized result based on user preferences.