AppForge: From Assistant to Independent Developer -- Are GPTs Ready for Software Development?

TL;DR

AppForge addresses the gap in evaluating LLMs for end-to-end Android development by constructing a 101-task benchmark sourced from real-world apps and paired with an automated APK-based evaluation pipeline. It combines automated specification extraction, test synthesis, and expert validation to enable scalable, reproducible assessment of full-stack Android development from scratch. Across 12 flagship LLMs, results show a large gap between function-level code generation and reliable, maintainable apps, with the best model achieving only about 15% functional success even after error-correction attempts. The study underscores the need for new approaches that integrate multi-file coordination, platform-specific constraints, and robust testing to advance autonomous software engineering.

Abstract

Large language models (LLMs) have demonstrated remarkable capability in function-level code generation tasks. Unlike isolated functions, real-world applications demand reasoning over the entire software system: developers must orchestrate how different components interact, maintain consistency across states over time, and ensure the application behaves correctly within the lifecycle and framework constraints. Yet, no existing benchmark adequately evaluates whether LLMs can bridge this gap and construct entire software systems from scratch. To address this gap, we propose APPFORGE, a benchmark consisting of 101 software development problems drawn from real-world Android apps. Given a natural language specification detailing the app functionality, a language model is tasked with implementing the functionality into an Android app from scratch. Developing an Android app from scratch requires understanding and coordinating app states, lifecycle management, and asynchronous operations, calling for LLMs to generate context-aware, robust, and maintainable code. To construct APPFORGE, we design a multi-agent system to automatically summarize the main functionalities from app documents and navigate the app to synthesize test cases validating the functional correctness of app implementation. Following rigorous manual verification by Android development experts, APPFORGE incorporates the test cases within an automated evaluation framework that enables reproducible assessment without human intervention, making it easily adoptable for future research. Our evaluation on 12 flagship LLMs show that all evaluated models achieve low effectiveness, with the best-performing model (GPT-5) developing only 18.8% functionally correct applications, highlighting fundamental limitations in current models' ability to handle complex, multi-component software engineering challenges.

Figures (11)

• Figure 1: Our Work Compared with Existing Code Generation Benchmarks.
• Figure 2: Workflow of AppForge.
• Figure 3: Construction of AppForge.
• Figure 4: Distribution of Category.
• Figure 5: Performance evolution with compilation feedback.