Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

CSS: Contrastive Semantic Similarity for Uncertainty Quantification of LLMs

Shuang Ao, Stefan Rueger, Advaith Siddharthan

TL;DR

This work tackles uncertainty quantification for large language models by addressing the limitations of NLI-logit–based semantic clustering. It introduces Contrastive Semantic Similarity (CSS), which leverages a CLIP text encoder to produce text-pair features and uses graph Laplacian-based spectral clustering to quantify uncertainty in LLM outputs. Through extensive experiments on open- and closed-book QA datasets with LLaMA, OPT, and GPT-3.5-turbo, CSS-based methods outperform state-of-the-art baselines, including NLI-based approaches, in selective NLG and overall reliability. The findings indicate that CLIP-derived contrastive features capture richer semantic relationships between responses, and dimensionality reduction via PCA enhances clustering, with METEOR offering a more semantic evaluation than Rouge-L. The work provides a practical, faster post-hoc UQ method and contributes to safer, more trustworthy LLM applications in real-world QA tasks.

Abstract

Despite the impressive capability of large language models (LLMs), knowing when to trust their generations remains an open challenge. The recent literature on uncertainty quantification of natural language generation (NLG) utilises a conventional natural language inference (NLI) classifier to measure the semantic dispersion of LLMs responses. These studies employ logits of NLI classifier for semantic clustering to estimate uncertainty. However, logits represent the probability of the predicted class and barely contain feature information for potential clustering. Alternatively, CLIP (Contrastive Language-Image Pre-training) performs impressively in extracting image-text pair features and measuring their similarity. To extend its usability, we propose Contrastive Semantic Similarity, the CLIP-based feature extraction module to obtain similarity features for measuring uncertainty for text pairs. We apply this method to selective NLG, which detects and rejects unreliable generations for better trustworthiness of LLMs. We conduct extensive experiments with three LLMs on several benchmark question-answering datasets with comprehensive evaluation metrics. Results show that our proposed method performs better in estimating reliable responses of LLMs than comparable baselines. Results show that our proposed method performs better in estimating reliable responses of LLMs than comparable baselines. The code are available at \url{https://github.com/AoShuang92/css_uq_llms}.

CSS: Contrastive Semantic Similarity for Uncertainty Quantification of LLMs

TL;DR

This work tackles uncertainty quantification for large language models by addressing the limitations of NLI-logit–based semantic clustering. It introduces Contrastive Semantic Similarity (CSS), which leverages a CLIP text encoder to produce text-pair features and uses graph Laplacian-based spectral clustering to quantify uncertainty in LLM outputs. Through extensive experiments on open- and closed-book QA datasets with LLaMA, OPT, and GPT-3.5-turbo, CSS-based methods outperform state-of-the-art baselines, including NLI-based approaches, in selective NLG and overall reliability. The findings indicate that CLIP-derived contrastive features capture richer semantic relationships between responses, and dimensionality reduction via PCA enhances clustering, with METEOR offering a more semantic evaluation than Rouge-L. The work provides a practical, faster post-hoc UQ method and contributes to safer, more trustworthy LLM applications in real-world QA tasks.

Abstract

Despite the impressive capability of large language models (LLMs), knowing when to trust their generations remains an open challenge. The recent literature on uncertainty quantification of natural language generation (NLG) utilises a conventional natural language inference (NLI) classifier to measure the semantic dispersion of LLMs responses. These studies employ logits of NLI classifier for semantic clustering to estimate uncertainty. However, logits represent the probability of the predicted class and barely contain feature information for potential clustering. Alternatively, CLIP (Contrastive Language-Image Pre-training) performs impressively in extracting image-text pair features and measuring their similarity. To extend its usability, we propose Contrastive Semantic Similarity, the CLIP-based feature extraction module to obtain similarity features for measuring uncertainty for text pairs. We apply this method to selective NLG, which detects and rejects unreliable generations for better trustworthiness of LLMs. We conduct extensive experiments with three LLMs on several benchmark question-answering datasets with comprehensive evaluation metrics. Results show that our proposed method performs better in estimating reliable responses of LLMs than comparable baselines. Results show that our proposed method performs better in estimating reliable responses of LLMs than comparable baselines. The code are available at \url{https://github.com/AoShuang92/css_uq_llms}.
Paper Structure (19 sections, 6 equations, 2 figures, 7 tables)

This paper contains 19 sections, 6 equations, 2 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Background
  5. NLI Classifier.
  6. Semantic Entropy with NLI Classifier.
  7. Graph Laplacian with NLI Classifier.
  8. Contrastive Feature Extraction.
  9. UQ with Contrastive Semantic Similarities
  10. Contrastive Semantic Similarities
  11. Graph Laplacian with Contrastive Semantic Similarities
  12. Experiments
  13. Dataset and Baselines
  14. Implementation Details
  15. Evaluation Metrics
  16. ...and 4 more sections

Figures (2)

  • Figure 1: The demonstration of obtaining our proposed contrastive semantic similarities for text pairs. After passing each generation to the CLIP text encoder, we conduct point-wise product on the corresponding embeddings to obtain the similarity features.
  • Figure 2: The accuracy-rejection curve for OPT sampled generations for CoQA, with Rouge-L>0.3 as the correctness criterion to obtain the base accuracy. After ranking samples based on their uncertainty scores obtained by the listed methods, we reject samples with higher uncertainty and calculate the accuracy for the remaining data. Oracle represents the highest performance of the model, where the model is perfectly calibrated and all rejected samples are wrongly predicted. We compare the Eccentricity of our method (ECC (ours)) with other baselines, namely p(true), semantic entropy (SE), Eccentricity in L-GL (Ecc (L-GL)), and number of Semantically distinct answers (NumSem).