Comparative Analysis of Large Language Models for the Machine-Assisted Resolution of User Intentions
Justus Flerlage, Alexander Acker, Odej Kao
TL;DR
The paper investigates the feasibility of locally deployable open-source and open-access LLMs for machine-assisted resolution of user intentions in a GUI-less, intent-based operating-system setting. It introduces a Controller-based architecture that translates user intents into executable workflows modeled as deterministic state machines, guided by a Function Table and executed in a sandboxed local environment. Through a comparative evaluation against GPT-4 family models, open-source models such as Falcon-3-10b-instruct, Phi-4, and Qwen-2.5-14b-instruct achieve seven of nine intention resolutions, approaching proprietary performance while offering privacy and autonomy benefits. The findings support the viability of self-hosted AI middleware for future on-device intent systems, while highlighting security, optimization, and API design considerations for real-world deployment.
Abstract
Large Language Models (LLMs) have emerged as transformative tools for natural language understanding and user intent resolution, enabling tasks such as translation, summarization, and, increasingly, the orchestration of complex workflows. This development signifies a paradigm shift from conventional, GUI-driven user interfaces toward intuitive, language-first interaction paradigms. Rather than manually navigating applications, users can articulate their objectives in natural language, enabling LLMs to orchestrate actions across multiple applications in a dynamic and contextual manner. However, extant implementations frequently rely on cloud-based proprietary models, which introduce limitations in terms of privacy, autonomy, and scalability. For language-first interaction to become a truly robust and trusted interface paradigm, local deployment is not merely a convenience; it is an imperative. This limitation underscores the importance of evaluating the feasibility of locally deployable, open-source, and open-access LLMs as foundational components for future intent-based operating systems. In this study, we examine the capabilities of several open-source and open-access models in facilitating user intention resolution through machine assistance. A comparative analysis is conducted against OpenAI's proprietary GPT-4-based systems to assess performance in generating workflows for various user intentions. The present study offers empirical insights into the practical viability, performance trade-offs, and potential of open LLMs as autonomous, locally operable components in next-generation operating systems. The results of this study inform the broader discussion on the decentralization and democratization of AI infrastructure and point toward a future where user-device interaction becomes more seamless, adaptive, and privacy-conscious through locally embedded intelligence.