Structured Self-Consistency:A Multi-Task Evaluation of LLMs on VirtualHome

TL;DR

The paper addresses the challenge of using Large Language Models for embodied AI in realistic simulations, where outputs must be structurally valid and causally executable. It introduces Structured Self-Consistency (SSC), an inference-time framework that combines Diversity Sampling, Semantic Canonicalization, and Structure-Aware Voting to produce semantically robust structured outputs, aided by task-adaptive prompts. Through a unified multi-task evaluation on VirtualHome via the EAI framework, the authors show SSC improves performance acrossGoal Interpretation, Action Sequencing, Subgoal Decomposition, and Transition Modeling for two open-source 7B LLMs, with OpenPangu-7B showing large gains in hierarchical planning and Qwen2.5-7B-Instruct showing strong action-level execution improvements; some models exhibit an awakening effect, revealing latent planning capabilities when structural constraints are enforced. The results demonstrate that inference-time structure enforcement is a practical, low-cost approach to boosting embodied AI capabilities without additional fine-tuning, with significant implications for robust, real-world planning in simulated worlds.$6.18\%$ to $42.10\%$ transitions in one case illustrate the potential magnitude of improvements, and the overall gains of up to $+7.3\%$ task success and $+12.1\%$ execution success highlight SSC’s broad applicability.

Abstract

Embodied AI requires agents to understand goals, plan actions, and execute tasks in simulated environments.We present a comprehensive evaluation of Large Language Models (LLMs) on the VirtualHome benchmark using the Embodied Agent Interface (EAI) framework.We compare two representative 7B-parameter models OPENPANGU-7B and QWEN2.5-7B across four fundamental tasks: Goal Interpretation, Action Sequencing, Subgoal Decomposition, and Transition Modeling.We propose Structured Self-Consistency (SSC), an enhanced decoding strategy that leverages multiple sampling with domain-specific voting mechanisms to improve output quality for structured generation tasks. Experimental results demonstrate that SSC significantly enhances performance, with OPENPANGU-7B excelling at hierarchical planning while QWEN2.5-7B show advantages in action-level tasks. Our analysis reveals complementary strengths across model types, providing insights for future embodied AI system development.

Figures (3)

• Figure 1: Overview of the four tasks in VirtualHome. Each task targets a distinct aspect of embodied intelligence: Goal Interpretation extracts structured goals from natural language instructions, Action Sequencing generates executable action programs, Subgoal Decomposition decomposes complex tasks into intermediate states, and Transition Modeling predicts action preconditions and effects.
• Figure 2: Overview of the Structured Self-Consistency (SSC) Methodology, illustrated with $N=4$ for simplicity (experiments use $N=5$). The framework handles two types of noise: (1) Syntactic Errors (e.g., broken JSON) are intercepted by the Schema Validator ($\Phi(y)=\bot$). (2) Semantic Instability (e.g., $y_3$ generating "OFF" instead of "ON") is resolved via structure-aware voting, where the minority hallucination ($Z_{other}$) is outvoted by the consistent consensus ($Z^*$).
• Figure 3: Relative Performance Gain via SSC.OpenPangu-7B exhibits a prominent "awakening" effect, particularly in Transition Modeling (+581%), demonstrating that SSC effectively bridges the performance gap for models with limited zero-shot formatting alignment by enforcing structural validity during inference.