A Multimedia Analytics Model for the Foundation Model Era

TL;DR

Existing multimedia analytics frameworks fail to fully account for the capabilities and challenges of foundation models and agentic AI. The authors propose a comprehensive multimedia analytics framework that couples foundation-model-driven analysis with visual analytics agents, a human-AI teaming layer, and an explicit interaction channel between expert users and semi-autonomous analytics. The model extends knowledge-generation concepts, task definitions (Analyze/Search/Query/Generate) with prompt templates, expert modules, and a guided strategy loop to manage complex multimodal workflows, demonstrated through case studies in investigative intelligence, video search, and generative prompting. Through these contributions, the work provides a blueprint for designing, evaluating, and advancing trustworthy, scalable multimedia analytics in the foundation-model era, with explicit attention to explainability, trust, and human oversight.

Abstract

The rapid advances in Foundation Models and agentic Artificial Intelligence are transforming multimedia analytics by enabling richer, more sophisticated interactions between humans and analytical systems. Existing conceptual models for visual and multimedia analytics, however, do not adequately capture the complexity introduced by these powerful AI paradigms. To bridge this gap, we propose a comprehensive multimedia analytics model specifically designed for the foundation model era. Building upon established frameworks from visual analytics, multimedia analytics, knowledge generation, analytic task definition, mixed-initiative guidance, and human-in-the-loop reinforcement learning, our model emphasizes integrated human-AI teaming based on visual analytics agents from both technical and conceptual perspectives. Central to the model is a seamless, yet explicitly separable, interaction channel between expert users and semi-autonomous analytical processes, ensuring continuous alignment between user intent and AI behavior. The model addresses practical challenges in sensitive domains such as intelligence analysis, investigative journalism, and other fields handling complex, high-stakes data. We illustrate through detailed case studies how our model facilitates deeper understanding and targeted improvement of multimedia analytics solutions. By explicitly capturing how expert users can optimally interact with and guide AI-powered multimedia analytics systems, our conceptual framework sets a clear direction for system design, comparison, and future research.

Figures (3)

• Figure 1: Overview of our proposed multimedia analytics model. It extends upon and connects several existing models from visual analytics and multimedia analytics in a coherent, modern framework. Its focus lies in connecting emerging AI models with human understanding through a human-AI teaming component based on visual analytics agents and specialized visualizations, which, based on its internal strategy, deploys the AI model and the rationale it provides to assist users to learn and align with the system. The core concept of the human-AI teaming is further schematized in detail in \ref{['fig:human-in-the-loop']}.
• Figure 2: Human-AI teaming model, a zoomed-in version of the center part in \ref{['fig:teaser']}, conceptualizes how visual analytics agents let users communicate in an effective way via a targeted user interface with the foundation models and AI agents. Results and rationale are displayed through various UI components, while the VA agents act as coordinator and execution controller towards the foundation models for specific actions, informed through system state and human feedback.
• Figure 3: Schematic VA agent leveraging a goal-reward model to optimize an internal VA strategy, incorporating feedback and model responses for prompt refinement and outputting structure, rationale, and results.