Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Talk, Snap, Complain: Validation-Aware Multimodal Expert Framework for Fine-Grained Customer Grievances

Rishu Kumar Singh, Navneet Shreya, Sarmistha Das, Apoorva Singh, Sriparna Saha

TL;DR

The paper introduces CIViL, a benchmark multimodal dialogue dataset of customer-support conversations paired with images, and VALOR, a Validation-Aware Learner with Expert Routing for fine-grained aspect and severity classification. VALOR uses a two-stage Mixture-of-Experts framework with Chain-of-Thought reasoning and a validation MoE, incorporating a semantic alignment score and meta-fusion to integrate multimodal signals and cross-view evidence. Experiments on CIViL show VALOR outperforming strong baselines, with ablations demonstrating the value of CoT experts, SAS-based alignment, and validation modules; human evaluation corroborates the model’s interpretability and reliability. The work advances practical multimodal grievance understanding in dialogue systems and lays groundwork for scalable, context-aware service analytics, with future directions including multilingual support and richer modality integration.

Abstract

Existing approaches to complaint analysis largely rely on unimodal, short-form content such as tweets or product reviews. This work advances the field by leveraging multimodal, multi-turn customer support dialogues, where users often share both textual complaints and visual evidence (e.g., screenshots, product photos) to enable fine-grained classification of complaint aspects and severity. We introduce VALOR, a Validation-Aware Learner with Expert Routing, tailored for this multimodal setting. It employs a multi-expert reasoning setup using large-scale generative models with Chain-of-Thought (CoT) prompting for nuanced decision-making. To ensure coherence between modalities, a semantic alignment score is computed and integrated into the final classification through a meta-fusion strategy. In alignment with the United Nations Sustainable Development Goals (UN SDGs), the proposed framework supports SDG 9 (Industry, Innovation and Infrastructure) by advancing AI-driven tools for robust, scalable, and context-aware service infrastructure. Further, by enabling structured analysis of complaint narratives and visual context, it contributes to SDG 12 (Responsible Consumption and Production) by promoting more responsive product design and improved accountability in consumer services. We evaluate VALOR on a curated multimodal complaint dataset annotated with fine-grained aspect and severity labels, showing that it consistently outperforms baseline models, especially in complex complaint scenarios where information is distributed across text and images. This study underscores the value of multimodal interaction and expert validation in practical complaint understanding systems. Resources related to data and codes are available here: https://github.com/sarmistha-D/VALOR

Talk, Snap, Complain: Validation-Aware Multimodal Expert Framework for Fine-Grained Customer Grievances

TL;DR

The paper introduces CIViL, a benchmark multimodal dialogue dataset of customer-support conversations paired with images, and VALOR, a Validation-Aware Learner with Expert Routing for fine-grained aspect and severity classification. VALOR uses a two-stage Mixture-of-Experts framework with Chain-of-Thought reasoning and a validation MoE, incorporating a semantic alignment score and meta-fusion to integrate multimodal signals and cross-view evidence. Experiments on CIViL show VALOR outperforming strong baselines, with ablations demonstrating the value of CoT experts, SAS-based alignment, and validation modules; human evaluation corroborates the model’s interpretability and reliability. The work advances practical multimodal grievance understanding in dialogue systems and lays groundwork for scalable, context-aware service analytics, with future directions including multilingual support and richer modality integration.

Abstract

Existing approaches to complaint analysis largely rely on unimodal, short-form content such as tweets or product reviews. This work advances the field by leveraging multimodal, multi-turn customer support dialogues, where users often share both textual complaints and visual evidence (e.g., screenshots, product photos) to enable fine-grained classification of complaint aspects and severity. We introduce VALOR, a Validation-Aware Learner with Expert Routing, tailored for this multimodal setting. It employs a multi-expert reasoning setup using large-scale generative models with Chain-of-Thought (CoT) prompting for nuanced decision-making. To ensure coherence between modalities, a semantic alignment score is computed and integrated into the final classification through a meta-fusion strategy. In alignment with the United Nations Sustainable Development Goals (UN SDGs), the proposed framework supports SDG 9 (Industry, Innovation and Infrastructure) by advancing AI-driven tools for robust, scalable, and context-aware service infrastructure. Further, by enabling structured analysis of complaint narratives and visual context, it contributes to SDG 12 (Responsible Consumption and Production) by promoting more responsive product design and improved accountability in consumer services. We evaluate VALOR on a curated multimodal complaint dataset annotated with fine-grained aspect and severity labels, showing that it consistently outperforms baseline models, especially in complex complaint scenarios where information is distributed across text and images. This study underscores the value of multimodal interaction and expert validation in practical complaint understanding systems. Resources related to data and codes are available here: https://github.com/sarmistha-D/VALOR

Paper Structure

This paper contains 65 sections, 6 equations, 4 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Complaint Detection
  4. Fine-grained Complaint Analysis
  5. Mixture-of-Experts
  6. Research Gap
  7. Customer Interactions with Visual and Linguistic Signals Dataset (CIViL)
  8. Annotator Details
  9. Annotation Phase & Dataset Analysis
  10. Proposed Approach
  11. Experiments and Results
  12. Evaluation Protocol
  13. Baseline Comparison
  14. Ablation Study
  15. Results Analysis
  16. ...and 50 more sections

Figures (4)

  • Figure 1: A conversation snippet from the CIViL dataset. Labels indicate the aspect-severity pairs identified from the conversation.
  • Figure 2: Architectural view of proposed VALOR framework
  • Figure 3: Human Evaluation on win-loss-draw performance criteria between popular baselines against CIViL; Here A-Stands for Aspect and S-stands for Severity
  • Figure 4: Expert weight matrix similarity heatmaps for different model architectures. (a) Mixtral 7B shows 8 experts with moderate diversity (similarity range: 0.08-0.25). (b) DeepSeek demonstrates 57 experts with high diversity and group-based organization (similarity range: 0.04-0.38). (c) Mixtral 22 displays 8 experts with good diversity (similarity range: 0.08-0.22). High diagonal similarity (yellow) indicates self-similarity, while low off-diagonal similarity (dark blue) indicates diverse, specialized experts.