Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LLM-for-X: Application-agnostic Integration of Large Language Models to Support Personal Writing Workflows

Lukas Teufelberger, Xintong Liu, Zhipeng Li, Max Moebus, Christian Holz

TL;DR

The paper addresses the fragmentation of LLM-enabled capabilities across desktop and web applications by introducing LLM-for-X, an OS-level, cross-application bridge that enables LLM interactions inside any app via a lightweight popup. The approach supports either chat-based backends through emulated input or direct API access, coordinated by an OS-level background service and a browser extension to handle native and web contexts. The authors validate the system with a within-subject user study (n=14) comparing LLM-for-X against ChatGPT, reporting faster text editing and higher usability scores for LLM-for-X, along with qualitative evidence that integration reduces context switching. The work demonstrates that a system-wide shortcut layer can deliver seamless, context-preserving LLM assistance across diverse writing, reading, and coding tasks, potentially reducing subscription fragmentation and improving practical productivity for end users.

Abstract

To enhance productivity and to streamline workflows, there is a growing trend to embed large language model (LLM) functionality into applications, from browser-based web apps to native apps that run on personal computers. Here, we introduce LLM-for-X, a system-wide shortcut layer that seamlessly augments any application with LLM services through a lightweight popup dialog. Our native layer seamlessly connects front-end applications to popular LLM backends, such as ChatGPT and Gemini, using their uniform chat front-ends as the programming interface or their custom API calls. We demonstrate the benefits of LLM-for-X across a wide variety of applications, including Microsoft Office, VSCode, and Adobe Acrobat as well as popular web apps such as Overleaf. In our evaluation, we compared LLM-for-X with ChatGPT's web interface in a series of tasks, showing that our approach can provide users with quick, efficient, and easy-to-use LLM assistance without context switching to support writing and reading tasks that is agnostic of the specific application.

LLM-for-X: Application-agnostic Integration of Large Language Models to Support Personal Writing Workflows

TL;DR

The paper addresses the fragmentation of LLM-enabled capabilities across desktop and web applications by introducing LLM-for-X, an OS-level, cross-application bridge that enables LLM interactions inside any app via a lightweight popup. The approach supports either chat-based backends through emulated input or direct API access, coordinated by an OS-level background service and a browser extension to handle native and web contexts. The authors validate the system with a within-subject user study (n=14) comparing LLM-for-X against ChatGPT, reporting faster text editing and higher usability scores for LLM-for-X, along with qualitative evidence that integration reduces context switching. The work demonstrates that a system-wide shortcut layer can deliver seamless, context-preserving LLM assistance across diverse writing, reading, and coding tasks, potentially reducing subscription fragmentation and improving practical productivity for end users.

Abstract

To enhance productivity and to streamline workflows, there is a growing trend to embed large language model (LLM) functionality into applications, from browser-based web apps to native apps that run on personal computers. Here, we introduce LLM-for-X, a system-wide shortcut layer that seamlessly augments any application with LLM services through a lightweight popup dialog. Our native layer seamlessly connects front-end applications to popular LLM backends, such as ChatGPT and Gemini, using their uniform chat front-ends as the programming interface or their custom API calls. We demonstrate the benefits of LLM-for-X across a wide variety of applications, including Microsoft Office, VSCode, and Adobe Acrobat as well as popular web apps such as Overleaf. In our evaluation, we compared LLM-for-X with ChatGPT's web interface in a series of tasks, showing that our approach can provide users with quick, efficient, and easy-to-use LLM assistance without context switching to support writing and reading tasks that is agnostic of the specific application.
Paper Structure (40 sections, 5 figures)

This paper contains 40 sections, 5 figures.

Table of Contents

  1. Introduction
  2. Walk-through and interaction design
  3. Design objectives
  4. Contributions
  5. Related work
  6. Virtual Assistants
  7. LLM-based Tools
  8. Efficient interaction and shortcuts
  9. LLM-for-X Implementation
  10. OS-level background service
  11. Native app interface (via accessibility API)
  12. Fallback: if the API fails
  13. Web app interface (via browser extension)
  14. Native service-to-extension interface
  15. LLM interface (via browser extension)
  16. ...and 25 more sections

Figures (5)

  • Figure 1: (a) LLM-for-X allows users to select text inside native and web apps and (b) execute predefined LLM commands or enter a custom query to (c) directly insert the response into the app---without the need for context switching or invoking copy & paste to transfer content between apps. Video figure: https://youtube.com/watch?v=fDDMaWobjVY
  • Figure 2: LLM-for-X walk-through. (a) Iterating on LLM responses, (b) pasting responses as 'insert below' vs. 'replacing' with diff view, (c) direct in-place pasting without preview, and (d) selecting and querying for information retrieval.
  • Figure 3: LLM-for-X offers a system-wide shortcut from any web or native application to various text-based LLM backend.
  • Figure 4: Effect of Interface on task completion time [sec].
  • Figure 5: Effect of Interface on SUS and NASA TLX scores.