SelfGoal: Your Language Agents Already Know How to Achieve High-level Goals

TL;DR

SelfGoal tackles the problem of achieving high-level goals with limited human feedback by building a dynamic GoalTree that adaptively decomposes goals into subgoals during interaction. It introduces non-parametric modules—Search, Decompose, and Act—within a loop that grounds guidance in environment states, enabling language agents to plan and act with delayed rewards. Across four competitive and cooperative tasks, SelfGoal outperforms baselines, with larger LLMs amplifying gains and GPT-4-derived GoalTrees providing particularly strong improvements ($+2.87$ and $+3.10$ in Auction and Bargaining, respectively). The approach offers a practical path toward robust, goal-consistent behavior in dynamic environments without continual model retraining, though it benefits from stronger reasoning capabilities in smaller models and could be extended with improved summarization and pruning strategies. Overall, SelfGoal demonstrates that a grounded, adaptive hierarchical guidance structure can significantly enhance long-horizon decision-making for language agents in diverse settings.

Abstract

Language agents powered by large language models (LLMs) are increasingly valuable as decision-making tools in domains such as gaming and programming. However, these agents often face challenges in achieving high-level goals without detailed instructions and in adapting to environments where feedback is delayed. In this paper, we present SelfGoal, a novel automatic approach designed to enhance agents' capabilities to achieve high-level goals with limited human prior and environmental feedback. The core concept of SelfGoal involves adaptively breaking down a high-level goal into a tree structure of more practical subgoals during the interaction with environments while identifying the most useful subgoals and progressively updating this structure. Experimental results demonstrate that SelfGoal significantly enhances the performance of language agents across various tasks, including competitive, cooperative, and deferred feedback environments. Project page: https://selfgoal-agent.github.io.

Figures (7)

• Figure 1: An overview of SelfGoal, illustrated with a bargaining example. The agent interacts with environments, and make actions based on environmental feedback and the GoalTree dynamically constructs, utilizes and updates with Search and Decompose Modules.
• Figure 2: Granularity control of the threshold $\xi$ in SelfGoal's stopping mechanism.
• Figure 3: Ablation study of different search modules.
• Figure 4: Ablation study of the model that generates GoalTree, either by a stronger (GPT-4) or weaker (GPT-3.5) model. The rest of the agent framework is driven by GPT-3.5.
• Figure 5: Patterns of model behavior in repeated games. (a): Fluctuations in contributions within the Public Goods game. The agent equipped with SelfGoal displays more rational behavior (i.e., achieving a Nash equilibrium) by consistently contributing fewer tokens than other methods. (b): Adjustments in number predictions within the Guessing Game. Our SelfGoal shows enhanced ToM abilities by converging to a guess of zero more quickly in each round.