LLMartini: Seamless and Interactive Leveraging of Multiple LLMs through Comparison and Composition

TL;DR

The paper tackles the inefficiencies of comparing and composing outputs from multiple LLMs, which impose cognitive burden on users navigating diverse model capabilities. It proposes LLMartini, a task-aware platform that segments model outputs semantically, highlights differences with color coding, and supports both side-by-side comparison and flexible result fusion. Through two user studies, it demonstrates that LLMartini reduces task time and cognitive load while increasing usability and user satisfaction compared with traditional multi-tab workflows, illustrating the value of human-centered design in multi-LLM collaboration. The work suggests practical implications for enabling dynamic model dispatch, transparent fusion strategies, and adaptive presentation, with future directions including multimodal outputs and personalized, long-term model collaboration.

Abstract

The growing diversity of large language models (LLMs) means users often need to compare and combine outputs from different models to obtain higher-quality or more comprehensive responses. However, switching between separate interfaces and manually integrating outputs is inherently inefficient, leading to a high cognitive burden and fragmented workflows. To address this, we present LLMartini, a novel interactive system that supports seamless comparison, selection, and intuitive cross-model composition tools. The system decomposes responses into semantically aligned segments based on task-specific criteria, automatically merges consensus content, and highlights model differences through color coding while preserving unique contributions. In a user study (N=18), LLMartini significantly outperformed conventional manual methods across all measured metrics, including task completion time, cognitive load, and user satisfaction. Our work highlights the importance of human-centered design in enhancing the efficiency and creativity of multi-LLM interactions and offers practical implications for leveraging the complementary strengths of various language models.

Figures (13)

• Figure 1: The frequency distribution of users' interactions with LLM and the number of models used simultaneously by users in daily interactions.
• Figure 2: Website prototype, with the history bar on the left, user interaction input and multi-model cards in the middle, and model output details on the right. Zoomed highlights show: (a) cited model name, (b) four action buttons ("Cite", "Like", "Dislike", and "Regenerate") on summary card, (c) dropdown model selector, (d) "Copy" and "Comment" buttons on expanded panel.
• Figure 3: Based on the statistical results of the task topic, the left figure shows the average total number of tasks for each topic, and the right figure shows the average number of user request rounds required to complete a task for the topic.
• Figure 4: The statistical results based on the task purpose are as follows: the left side shows the average number of tasks for each topic, and the right side shows the average number of user request rounds required to complete a task for that topic. The capital letters after the task classification represent the major categories to which it belongs.
• Figure 5: The heat map of model interaction (Darker blue indicates more usage, Lighter yellow indicates less usage). The results are sorted from top to bottom according to the total number of user interactions. The number in each block represents the number of interaction times.