Removing Planner Bias in Goal Recognition Through Multi-Plan Dataset Generation
Mustafa F. Abdelwahed, Felipe Meneguzzi Kin Max Piamolini Gusmao, Joan Espasa
TL;DR
This work identifies a critical bias in goal-recognition benchmarks arising from the planners used to generate datasets. It introduces a resilient dataset generation approach based on top-$k$ planning to produce multiple plans for the same goal and a Version Coverage Score ($VCS$) to quantify recogniser resilience across these plan variants. The authors demonstrate that state-of-the-art recognisers degrade under low observability on the new, planner-diverse dataset, highlighting the limitations of traditional benchmarks. This approach provides a more robust framework for evaluating goal recognisers and guides future work toward planner-agnostic robustness and diverse planning strategies.
Abstract
Autonomous agents require some form of goal and plan recognition to interact in multiagent settings. Unfortunately, all existing goal recognition datasets suffer from a systematical bias induced by the planning systems that generated them, namely heuristic-based forward search. This means that existing datasets lack enough challenge for more realistic scenarios (e.g., agents using different planners), which impacts the evaluation of goal recognisers with respect to using different planners for the same goal. In this paper, we propose a new method that uses top-k planning to generate multiple, different, plans for the same goal hypothesis, yielding benchmarks that mitigate the bias found in the current dataset. This allows us to introduce a new metric called Version Coverage Score (VCS) to measure the resilience of the goal recogniser when inferring a goal based on different sets of plans. Our results show that the resilience of the current state-of-the-art goal recogniser degrades substantially under low observability settings.