Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

GRCF: Two-Stage Groupwise Ranking and Calibration Framework for Multimodal Sentiment Analysis

Manning Gao, Leheng Zhang, Shiqin Han, Haifeng Hu, Yuncheng Jiang, Sijie Mai

TL;DR

GRCF tackles the ordinal and noisy nature of multimodal sentiment analysis by introducing a two-stage framework that first builds a semantically aware ordinal manifold using an Advantage-Weighted Dynamic Margin Ranking Loss with overlapping groups, and then calibrates absolute scores via MAE-driven fine-tuning. The approach, inspired by GRPO, adaptively focuses on hard-to-rank sample pairs and scales margins with semantic distance, yielding state-of-the-art results on MOSI, MOSEI, and CH-SIMS v2, while also generalizing to sarcasm and humor detection. Extending to classification requires a separative-compactness adaptation, and extensive ablations confirm the critical roles of Stage 1 structure and Stage 2 calibration in achieving robust, well-calibrated predictions. Overall, GRCF advances reliable ordinal modeling and absolute calibration in multimodal sentiment tasks, with strong generalization and robustness to input noise.

Abstract

Most Multimodal Sentiment Analysis research has focused on point-wise regression. While straightforward, this approach is sensitive to label noise and neglects whether one sample is more positive than another, resulting in unstable predictions and poor correlation alignment. Pairwise ordinal learning frameworks emerged to address this gap, capturing relative order by learning from comparisons. Yet, they introduce two new trade-offs: First, they assign uniform importance to all comparisons, failing to adaptively focus on hard-to-rank samples. Second, they employ static ranking margins, which fail to reflect the varying semantic distances between sentiment groups. To address this, we propose a Two-Stage Group-wise Ranking and Calibration Framework (GRCF) that adapts the philosophy of Group Relative Policy Optimization (GRPO). Our framework resolves these trade-offs by simultaneously preserving relative ordinal structure, ensuring absolute score calibration, and adaptively focusing on difficult samples. Specifically, Stage 1 introduces a GRPO-inspired Advantage-Weighted Dynamic Margin Ranking Loss to build a fine-grained ordinal structure. Stage 2 then employs an MAE-driven objective to align prediction magnitudes. To validate its generalizability, we extend GRCF to classification tasks, including multimodal humor detection and sarcasm detection. GRCF achieves state-of-the-art performance on core regression benchmarks, while also showing strong generalizability in classification tasks.

GRCF: Two-Stage Groupwise Ranking and Calibration Framework for Multimodal Sentiment Analysis

TL;DR

GRCF tackles the ordinal and noisy nature of multimodal sentiment analysis by introducing a two-stage framework that first builds a semantically aware ordinal manifold using an Advantage-Weighted Dynamic Margin Ranking Loss with overlapping groups, and then calibrates absolute scores via MAE-driven fine-tuning. The approach, inspired by GRPO, adaptively focuses on hard-to-rank sample pairs and scales margins with semantic distance, yielding state-of-the-art results on MOSI, MOSEI, and CH-SIMS v2, while also generalizing to sarcasm and humor detection. Extending to classification requires a separative-compactness adaptation, and extensive ablations confirm the critical roles of Stage 1 structure and Stage 2 calibration in achieving robust, well-calibrated predictions. Overall, GRCF advances reliable ordinal modeling and absolute calibration in multimodal sentiment tasks, with strong generalization and robustness to input noise.

Abstract

Most Multimodal Sentiment Analysis research has focused on point-wise regression. While straightforward, this approach is sensitive to label noise and neglects whether one sample is more positive than another, resulting in unstable predictions and poor correlation alignment. Pairwise ordinal learning frameworks emerged to address this gap, capturing relative order by learning from comparisons. Yet, they introduce two new trade-offs: First, they assign uniform importance to all comparisons, failing to adaptively focus on hard-to-rank samples. Second, they employ static ranking margins, which fail to reflect the varying semantic distances between sentiment groups. To address this, we propose a Two-Stage Group-wise Ranking and Calibration Framework (GRCF) that adapts the philosophy of Group Relative Policy Optimization (GRPO). Our framework resolves these trade-offs by simultaneously preserving relative ordinal structure, ensuring absolute score calibration, and adaptively focusing on difficult samples. Specifically, Stage 1 introduces a GRPO-inspired Advantage-Weighted Dynamic Margin Ranking Loss to build a fine-grained ordinal structure. Stage 2 then employs an MAE-driven objective to align prediction magnitudes. To validate its generalizability, we extend GRCF to classification tasks, including multimodal humor detection and sarcasm detection. GRCF achieves state-of-the-art performance on core regression benchmarks, while also showing strong generalizability in classification tasks.
Paper Structure (30 sections, 20 equations, 10 figures, 13 tables)

This paper contains 30 sections, 20 equations, 10 figures, 13 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Point-wise MSA methods
  4. Ordinal and Pair-wise Ranking Learning
  5. Advantage-Weighted Optimization
  6. Methodology
  7. Multimodal Encoder Architecture
  8. Unimodal Encoders.
  9. Multimodal Fusion.
  10. Stage 1: Structural Foundation
  11. Group-Aware Ranking ($L_{\text{group}}$)
  12. Distribution Regularization ($L_{\text{reg}}$)
  13. Boundary Loss ($L_{\text{bound}}$)
  14. Stage 2: Fine-Tuning
  15. Extending to Classification Tasks
  16. ...and 15 more sections

Figures (10)

  • Figure 1: Comparison of optimization strategies in MSA.
  • Figure 2: The forward pass model and core loss calculation which decouples the ranking "standard" from the "strength". (a) The model processes inputs through multimodal encoders to output a predicted score. (b) For the "Punishment Standard", ground-truth scores are mapped to overlapping groups to define the dynamic margin. For the "Punishment Strength", predicted scores are compared against batch statistics to calculate the advantage.
  • Figure 3: Training loss curves on CMU-MOSI dataset. (Top) Stage 1 loss. (Bottom) Stage 2 loss.
  • Figure 4: Main results (Acc2) on the classification tasks.
  • Figure 5: t-SNE visualization of ablations on CMU-MOSI. (a) Full model. Component ablations of Group-Aware Ranking Loss: (b) Ablating GRPO advantage-weighting. (c) Ablating dynamic margin. Strategy ablations: (d) Direct regression. (e) Stage 1 alone.
  • ...and 5 more figures