Balancing Act: Prioritization Strategies for LLM-Designed Restless Bandit Rewards
Shresth Verma, Niclas Boehmer, Lingkai Kong, Milind Tambe
TL;DR
This work tackles the problem of designing reward functions for Restless Multi-Armed Bandits (RMABs) that reflect multi-objective human preferences. It introduces the Social Choice Language Model (SCLM), a two-stage framework: an LLM-based generator creates a pool of candidate reward functions, and an adjudicator selects one using social welfare aggregation over per-clause alignment scores. Alignment is computed via simulation-based (SCLM-SIM) and LLM-based (SCLM-LLM) scorers, with safeguards for unintended utility shifts and potential utility drops, and theoretical guarantees on selection quality under scoring noise. Empirical results on synthetic and real-world RMAB problems demonstrate that SCLM reliably yields reward functions that are more aligned, balanced, and less prone to bias than purely LLM-driven approaches, while enabling explicit control over the trade-offs via welfare function choices. The framework thus provides a transparent, human-centric method for multi-objective reward design in multiagent planning with RMABs, with potential applicability to other domains requiring balanced preference integration.
Abstract
LLMs are increasingly used to design reward functions based on human preferences in Reinforcement Learning (RL). We focus on LLM-designed rewards for Restless Multi-Armed Bandits, a framework for allocating limited resources among agents. In applications such as public health, this approach empowers grassroots health workers to tailor automated allocation decisions to community needs. In the presence of multiple agents, altering the reward function based on human preferences can impact subpopulations very differently, leading to complex tradeoffs and a multi-objective resource allocation problem. We are the first to present a principled method termed Social Choice Language Model for dealing with these tradeoffs for LLM-designed rewards for multiagent planners in general and restless bandits in particular. The novel part of our model is a transparent and configurable selection component, called an adjudicator, external to the LLM that controls complex tradeoffs via a user-selected social welfare function. Our experiments demonstrate that our model reliably selects more effective, aligned, and balanced reward functions compared to purely LLM-based approaches.