Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

TRUSTVIS: A Multi-Dimensional Trustworthiness Evaluation Framework for Large Language Models

Ruoyu Sun, Da Song, Jiayang Song, Yuheng Huang, Lei Ma

TL;DR

TRUSTVIS addresses the fragmented evaluation of LLM trustworthiness by unifying safety and robustness assessment within an automated, interactive framework. It combines a back-end pipeline that labels outputs using the MLCommons Safety Taxonomy and ensembles multiple safeguards with a front-end dashboard that visualizes results and drives targeted improvements. The framework uses adversarial prompt perturbations (AutoDAN with GA/HGA) to probe robustness and repurposes those tests as stress checks for safety, yielding detailed taxonomy-level insights. Open-sourced and demonstrated on Vicuna-7b, LLaMA-2-7B, and GPT-3.5, TRUSTVIS offers a practical, user-friendly tool for researchers and industry practitioners to diagnose vulnerabilities and guide model enhancements.

Abstract

As Large Language Models (LLMs) continue to revolutionize Natural Language Processing (NLP) applications, critical concerns about their trustworthiness persist, particularly in safety and robustness. To address these challenges, we introduce TRUSTVIS, an automated evaluation framework that provides a comprehensive assessment of LLM trustworthiness. A key feature of our framework is its interactive user interface, designed to offer intuitive visualizations of trustworthiness metrics. By integrating well-known perturbation methods like AutoDAN and employing majority voting across various evaluation methods, TRUSTVIS not only provides reliable results but also makes complex evaluation processes accessible to users. Preliminary case studies on models like Vicuna-7b, Llama2-7b, and GPT-3.5 demonstrate the effectiveness of our framework in identifying safety and robustness vulnerabilities, while the interactive interface allows users to explore results in detail, empowering targeted model improvements. Video Link: https://youtu.be/k1TrBqNVg8g

TRUSTVIS: A Multi-Dimensional Trustworthiness Evaluation Framework for Large Language Models

TL;DR

TRUSTVIS addresses the fragmented evaluation of LLM trustworthiness by unifying safety and robustness assessment within an automated, interactive framework. It combines a back-end pipeline that labels outputs using the MLCommons Safety Taxonomy and ensembles multiple safeguards with a front-end dashboard that visualizes results and drives targeted improvements. The framework uses adversarial prompt perturbations (AutoDAN with GA/HGA) to probe robustness and repurposes those tests as stress checks for safety, yielding detailed taxonomy-level insights. Open-sourced and demonstrated on Vicuna-7b, LLaMA-2-7B, and GPT-3.5, TRUSTVIS offers a practical, user-friendly tool for researchers and industry practitioners to diagnose vulnerabilities and guide model enhancements.

Abstract

As Large Language Models (LLMs) continue to revolutionize Natural Language Processing (NLP) applications, critical concerns about their trustworthiness persist, particularly in safety and robustness. To address these challenges, we introduce TRUSTVIS, an automated evaluation framework that provides a comprehensive assessment of LLM trustworthiness. A key feature of our framework is its interactive user interface, designed to offer intuitive visualizations of trustworthiness metrics. By integrating well-known perturbation methods like AutoDAN and employing majority voting across various evaluation methods, TRUSTVIS not only provides reliable results but also makes complex evaluation processes accessible to users. Preliminary case studies on models like Vicuna-7b, Llama2-7b, and GPT-3.5 demonstrate the effectiveness of our framework in identifying safety and robustness vulnerabilities, while the interactive interface allows users to explore results in detail, empowering targeted model improvements. Video Link: https://youtu.be/k1TrBqNVg8g
Paper Structure (10 sections, 2 figures, 3 tables)

This paper contains 10 sections, 2 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. Back-End Design
  4. Front-End Design
  5. Evaluation
  6. Safety Evaluation
  7. Robustness Evaluation
  8. Usability Evaluation
  9. Related Work
  10. Conclusion

Figures (2)

  • Figure 1: Overview of TrustVis
  • Figure 2: Overview of the interactive evaluation interface. The dashboard includes sections for summary statistics, detailed taxonomy breakdowns, and example problematic responses.