PARADISE: Evaluating Implicit Planning Skills of Language Models with Procedural Warnings and Tips Dataset

TL;DR

PARADISE introduces a large-scale abductive reasoning dataset to evaluate implicit planning in language models by requiring them to infer warnings and tips from goals in wikiHow procedural text. The authors deploy two evaluation pipelines—finetuning encoder PLMs and zero-shot prompting of diverse LLMs—to test implicit planning capabilities, using robust data construction, semantic negative sampling, and expert test-set curation. Key findings show that task-tuned small models often outperform LLMs, though all models lag behind human performance, with tip inference generally easier than warning inference and substantial model-specific failure modes. The dataset enables cross-domain and out-of-domain transfer studies, demonstrating generalization across related procedural tasks and providing a valuable resource for advancing unseen procedural understanding in AI systems.

Abstract

Recently, there has been growing interest within the community regarding whether large language models are capable of planning or executing plans. However, most prior studies use LLMs to generate high-level plans for simplified scenarios lacking linguistic complexity and domain diversity, limiting analysis of their planning abilities. These setups constrain evaluation methods (e.g., predefined action space), architectural choices (e.g., only generative models), and overlook the linguistic nuances essential for realistic analysis. To tackle this, we present PARADISE, an abductive reasoning task using Q\&A format on practical procedural text sourced from wikiHow. It involves warning and tip inference tasks directly associated with goals, excluding intermediary steps, with the aim of testing the ability of the models to infer implicit knowledge of the plan solely from the given goal. Our experiments, utilizing fine-tuned language models and zero-shot prompting, reveal the effectiveness of task-specific small models over large language models in most scenarios. Despite advancements, all models fall short of human performance. Notably, our analysis uncovers intriguing insights, such as variations in model behavior with dropped keywords, struggles of BERT-family and GPT-4 with physical and abstract goals, and the proposed tasks offering valuable prior knowledge for other unseen procedural tasks. The PARADISE dataset and associated resources are publicly available for further research exploration with https://github.com/GGLAB-KU/paradise.

Figures (7)

• Figure 1: A procedural tutorial on "Removing Ink Stains from Fabric". Here, one can damage the fabric if they ignore the warning "Always blot, never rub, when dealing with ink stains".
• Figure 2: Example questions for warning (1) and tip (2) inference tasks. Correct choices are bold.
• Figure 3: Model performances tested on manipulated test data.
• Figure 4: Correlation matrices of incorrect predictions of each model for tip (left) and warning (right) inference.
• Figure 5: Accuracy of the three BERT variations plotted over the training epoch.