Router-Suggest: Dynamic Routing for Multimodal Auto-Completion in Visually-Grounded Dialogs

TL;DR

This work defines Multimodal Auto-Completion (MAC) to predict inline user input in visually grounded dialogs by grounding prefixes in both text and imagery. It introduces Router-Suggest to dynamically select between lightweight textual models and high-capacity vision-language models, optimizing for latency and accuracy with a cost-aware objective. Standardized MAC benchmarks are built by adapting MMDialog and ImageChat, and a comprehensive evaluation shows VLMs surmount textual baselines on unseen multimodal prefixes, while routing achieves substantial speedups (2.3×–10×) with competitive quality. A user study confirms that multimodal predictions reduce typing effort and enhance user satisfaction, highlighting practical benefits for visually grounded assistants and copilots. The work also acknowledges biases from data curation, domain limitations to single-image contexts, and ethical considerations around fairness and compute costs in routing decisions.

Abstract

Real-time multimodal auto-completion is essential for digital assistants, chatbots, design tools, and healthcare consultations, where user inputs rely on shared visual context. We introduce Multimodal Auto-Completion (MAC), a task that predicts upcoming characters in live chats using partially typed text and visual cues. Unlike traditional text-only auto-completion (TAC), MAC grounds predictions in multimodal context to better capture user intent. To enable this task, we adapt MMDialog and ImageChat to create benchmark datasets. We evaluate leading vision-language models (VLMs) against strong textual baselines, highlighting trade-offs in accuracy and efficiency. We present Router-Suggest, a router framework that dynamically selects between textual models and VLMs based on dialog context, along with a lightweight variant for resource-constrained environments. Router-Suggest achieves a 2.3x to 10x speedup over the best-performing VLM. A user study shows that VLMs significantly excel over textual models on user satisfaction, notably saving user typing effort and improving the quality of completions in multi-turn conversations. These findings underscore the need for multimodal context in auto-completions, leading to smarter, user-aware assistants.

Figures (6)

• Figure 1: Example of multimodal auto-completion. Given the image context (a man walking a golden retriever in a sunlit park) and the partial user input "That's why I love bringing my ", the MAC model predicts "dog out for walks here!", while a text-based TAC model incorrectly predicts "children for playing here!". The MAC model prediction leverages both the textual prefix and visual context for a grounded completion.
• Figure 2: During router training, VLMs receive the entire input context, while the textual QB model only uses the prefix. We calculate partial-F1 scores of predictions to determine the gold label. Further, we generate a feature vector for the input prefix of the training sample using EmbeddingGemma-300m for training the neural classifier.
• Figure 3: Different router configurations for Router-4 at different $\lambda$ and their latency vs PR-F1 score tradeoff for (a) MMDD and (b) ImageChat.
• Figure 4: Comparison of mean TES and user ratings (normalized) for various models. TES is calculated relative to the final text approved by the user at the moment the rating is submitted.
• Figure 5: Prompt template for relevance filtering using GPT-4V.