Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MULTI-CASE: A Transformer-based Ethics-aware Multimodal Investigative Intelligence Framework

Maximilian T. Fischer, Yannick Metz, Lucas Joos, Matthias Miller, Daniel A. Keim

TL;DR

MULTI-CASE tackles ethics and privacy challenges in multimodal intelligence analytics by delivering a holistic, visual-analytics framework that couples a fully integrated graph data model with modular, plug-in analytics and a GPU-accelerated knowledge graph UI. It combines transformer-based NER for domain-specific entity extraction with an ontology-driven search to enable cross-modal linking, explanation, and provenance, supporting equal human–AI agency. The paper presents a war-crimes case study and a formative expert evaluation showing improved human agency, transparency, and accountability, while acknowledging limitations in language generalization and potential over-reliance on automation. Overall, it offers a principled blueprint for accountable, multimodal intelligence exploration that can be extended with LLMs and multilingual capabilities while maintaining oversight and privacy safeguards.

Abstract

AI-driven models are increasingly deployed in operational analytics solutions, for instance, in investigative journalism or the intelligence community. Current approaches face two primary challenges: ethical and privacy concerns, as well as difficulties in efficiently combining heterogeneous data sources for multimodal analytics. To tackle the challenge of multimodal analytics, we present MULTI-CASE, a holistic visual analytics framework tailored towards ethics-aware and multimodal intelligence exploration, designed in collaboration with domain experts. It leverages an equal joint agency between human and AI to explore and assess heterogeneous information spaces, checking and balancing automation through Visual Analytics. MULTI-CASE operates on a fully-integrated data model and features type-specific analysis with multiple linked components, including a combined search, annotated text view, and graph-based analysis. Parts of the underlying entity detection are based on a RoBERTa-based language model, which we tailored towards user requirements through fine-tuning. An overarching knowledge exploration graph combines all information streams, provides in-situ explanations, transparent source attribution, and facilitates effective exploration. To assess our approach, we conducted a comprehensive set of evaluations: We benchmarked the underlying language model on relevant NER tasks, achieving state-of-the-art performance. The demonstrator was assessed according to intelligence capability assessments, while the methodology was evaluated according to ethics design guidelines. As a case study, we present our framework in an investigative journalism setting, supporting war crime investigations. Finally, we conduct a formative user evaluation with domain experts in law enforcement. Our evaluations confirm that our framework facilitates human agency and steering in security-sensitive applications.

MULTI-CASE: A Transformer-based Ethics-aware Multimodal Investigative Intelligence Framework

TL;DR

MULTI-CASE tackles ethics and privacy challenges in multimodal intelligence analytics by delivering a holistic, visual-analytics framework that couples a fully integrated graph data model with modular, plug-in analytics and a GPU-accelerated knowledge graph UI. It combines transformer-based NER for domain-specific entity extraction with an ontology-driven search to enable cross-modal linking, explanation, and provenance, supporting equal human–AI agency. The paper presents a war-crimes case study and a formative expert evaluation showing improved human agency, transparency, and accountability, while acknowledging limitations in language generalization and potential over-reliance on automation. Overall, it offers a principled blueprint for accountable, multimodal intelligence exploration that can be extended with LLMs and multilingual capabilities while maintaining oversight and privacy safeguards.

Abstract

AI-driven models are increasingly deployed in operational analytics solutions, for instance, in investigative journalism or the intelligence community. Current approaches face two primary challenges: ethical and privacy concerns, as well as difficulties in efficiently combining heterogeneous data sources for multimodal analytics. To tackle the challenge of multimodal analytics, we present MULTI-CASE, a holistic visual analytics framework tailored towards ethics-aware and multimodal intelligence exploration, designed in collaboration with domain experts. It leverages an equal joint agency between human and AI to explore and assess heterogeneous information spaces, checking and balancing automation through Visual Analytics. MULTI-CASE operates on a fully-integrated data model and features type-specific analysis with multiple linked components, including a combined search, annotated text view, and graph-based analysis. Parts of the underlying entity detection are based on a RoBERTa-based language model, which we tailored towards user requirements through fine-tuning. An overarching knowledge exploration graph combines all information streams, provides in-situ explanations, transparent source attribution, and facilitates effective exploration. To assess our approach, we conducted a comprehensive set of evaluations: We benchmarked the underlying language model on relevant NER tasks, achieving state-of-the-art performance. The demonstrator was assessed according to intelligence capability assessments, while the methodology was evaluated according to ethics design guidelines. As a case study, we present our framework in an investigative journalism setting, supporting war crime investigations. Finally, we conduct a formative user evaluation with domain experts in law enforcement. Our evaluations confirm that our framework facilitates human agency and steering in security-sensitive applications.
Paper Structure (18 sections, 3 figures, 1 table)

This paper contains 18 sections, 3 figures, 1 table.

Table of Contents

  1. Introduction
  2. Introduction
  3. Related Work
  4. Methodology: Model Development
  5. System Design
  6. Data Model
  7. Component Integration
  8. Interfaces and Interaction Principles
  9. Evaluation
  10. Case Study
  11. Ethics Design Guidelines
  12. Intelligence Capability Assessment
  13. Model Evaluation
  14. Domain Expert Evaluation
  15. Discussion and Future Work
  16. ...and 3 more sections

Figures (3)

  • Figure 1: MULTI-CASE: A holistic visual analytics framework tailored towards ethics-aware and multimodal intelligence exploration. Built upon a fully-integrated data model, it features type-specific, graph-based analysis through individual models with multiple, linked components: a combined ontological search and result interface, an interactive textual view, and a knowledge graph interface. Additional components (not shown) include more specialized modules like video or audio analysis. The interface facilitates in-situ contextualization across modalities, graph neighborhood explorations, relevance scoring for accountability and oversight, transparent source explanations, integrated navigation, and collaborative user participation.
  • Figure 2: High-level architecture of MULTI-CASE, highlighting the main components - , like the analysis modules or the graph UI with visualization modules, the fully-integrated graph data model, as well as different data paths and user handling, like the --- analytics workflow. A detailed description is provided in Section \ref{['sec:teamcase_system']}.
  • Figure 3: Neighborhood exploration, acting as a magnifying spotlight to show a manageable local context for a seamless exploration.