Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

U2F: Encouraging SWE-Agent to Seize Novelty without Losing Feasibility

Wencheng Ye, Yan Liu

TL;DR

This paper tackles the limited novelty of LLM-based SWE-Agents in open-world contexts by introducing U2F, a cognitive-inspired multi-agent framework that systematically surfaces Unknown Unknowns through Discovery, Exploration, and Integration. It operationalizes UU discovery with cross-domain analogies, backward reasoning, and external validation, and establishes a rich experimental pipeline built on Enabler Stories and the Double Diamond model. Empirical results on 218 real-world tasks show meaningful boosts in novelty (∼14%) and semantic novelty (∼51%) while preserving feasibility (4.02/5), validated by both human experts and an LLM evaluator; ablation studies confirm the critical roles of exploration and integration. The work demonstrates that embracing uncertainty, grounded in structured cognitive scaffolding and human-in-the-loop collaboration, can meaningfully expand solution spaces in software engineering, with practical gains in speed and perspective for industry teams.

Abstract

Large language models (LLMs) have shown strong capabilities in software engineering tasks, yet most existing LLM-based SWE-Agents mainly tackle well-defined problems using conventional methods, often overlooking alternative or innovative solutions beyond their predefined frameworks. This limitation is evident in open-world software environments, where emerging challenges transcend established paradigms. We propose U2F (Unknown Unknowns to Functional solutions), a cognitive-inspired, uncertainty-embracing multi-agent framework that systematically surfaces "Unknown Unknowns" - novel solution pathways absent from initial formulations but holding innovative potential. U2F consists of two key components: (1) a Discovery-Exploration-Integration agent system for uncovering and synthesizing potential solutions, and (2) cognitive enhancement mechanisms across three dimensions: cross-domain analogical reasoning, reverse thinking, and external validation, which strategically reframe and extend conventional solution boundaries. Applied to 218 real-world software enabler stories curated from authentic engineering tasks, U2F achieved notable improvements: human experts reported a 14 percent increase in overall novelty, 51 percent improvement in semantic novelty, and stable feasibility (4.02/5.0), corroborated by an LLM-based evaluator. These results highlight the potential of embracing uncertainty as a catalyst for innovation in software engineering.

U2F: Encouraging SWE-Agent to Seize Novelty without Losing Feasibility

TL;DR

This paper tackles the limited novelty of LLM-based SWE-Agents in open-world contexts by introducing U2F, a cognitive-inspired multi-agent framework that systematically surfaces Unknown Unknowns through Discovery, Exploration, and Integration. It operationalizes UU discovery with cross-domain analogies, backward reasoning, and external validation, and establishes a rich experimental pipeline built on Enabler Stories and the Double Diamond model. Empirical results on 218 real-world tasks show meaningful boosts in novelty (∼14%) and semantic novelty (∼51%) while preserving feasibility (4.02/5), validated by both human experts and an LLM evaluator; ablation studies confirm the critical roles of exploration and integration. The work demonstrates that embracing uncertainty, grounded in structured cognitive scaffolding and human-in-the-loop collaboration, can meaningfully expand solution spaces in software engineering, with practical gains in speed and perspective for industry teams.

Abstract

Large language models (LLMs) have shown strong capabilities in software engineering tasks, yet most existing LLM-based SWE-Agents mainly tackle well-defined problems using conventional methods, often overlooking alternative or innovative solutions beyond their predefined frameworks. This limitation is evident in open-world software environments, where emerging challenges transcend established paradigms. We propose U2F (Unknown Unknowns to Functional solutions), a cognitive-inspired, uncertainty-embracing multi-agent framework that systematically surfaces "Unknown Unknowns" - novel solution pathways absent from initial formulations but holding innovative potential. U2F consists of two key components: (1) a Discovery-Exploration-Integration agent system for uncovering and synthesizing potential solutions, and (2) cognitive enhancement mechanisms across three dimensions: cross-domain analogical reasoning, reverse thinking, and external validation, which strategically reframe and extend conventional solution boundaries. Applied to 218 real-world software enabler stories curated from authentic engineering tasks, U2F achieved notable improvements: human experts reported a 14 percent increase in overall novelty, 51 percent improvement in semantic novelty, and stable feasibility (4.02/5.0), corroborated by an LLM-based evaluator. These results highlight the potential of embracing uncertainty as a catalyst for innovation in software engineering.

Paper Structure

This paper contains 24 sections, 1 equation, 4 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Unknowns Unknowns in Software Engineering Contexts
  4. Enabler Stories and Double Diamond Model as Vehicles for Innovation Exploration
  5. LLM-Based Agents in Requirements Engineering: Capabilities and Limitations
  6. The U2F Framework: Design, Implementation, and Methodology
  7. Operationalizing "Unknown Unknowns": Definition and Criteria
  8. The U2F Cognitive Architecture: Design Principles and Collaborative Model
  9. Core Components: From Cognitive Agents to Human-AI Collaboration
  10. Building the Experimental Dataset: Generation and Curation of Enabler Stories
  11. Experimental Design
  12. Research Questions
  13. Baselines & Experimental Setup
  14. Evaluation Metrics
  15. Results
  16. ...and 9 more sections

Figures (4)

  • Figure 1: Progressive discovery of Unknown Unknowns. The dialogue demonstrates how systematic exploration reveals solution pathways absent from the initial requirement.
  • Figure 2: Overall architecture of U2F.
  • Figure 3: This figure shows a real-world interaction case within our framework, using the context of developing silent photography in quiet environments. The serial number indicates the processing order of the agent, and the magnifying glass represents the invocation of search API.
  • Figure 4: Construction process of the Enabler Story dataset.