Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

FINEST: Improving LLM Responses to Sensitive Topics Through Fine-Grained Evaluation

Juhyun Oh, Nayeon Lee, Chani Jung, Jiho Jin, Junho Myung, Jongwon Lee, Taeui Song, Alice Oh

TL;DR

FinEST, a FINE-grained response evaluation taxonomy for Sensitive Topics, which breaks down helpfulness and harmlessness into errors across three main categories: Content, Logic, and Appropriateness, is introduced, laying the foundation for a more explainable and comprehensive evaluation and improvement of LLM responses to sensitive questions.

Abstract

Large Language Models (LLMs) often generate overly cautious and vague responses on sensitive topics, sacrificing helpfulness for safety. Existing evaluation frameworks lack systematic methods to identify and address specific weaknesses in responses to sensitive topics, making it difficult to improve both safety and helpfulness simultaneously. To address this, we introduce FINEST, a FINE-grained response evaluation taxonomy for Sensitive Topics, which breaks down helpfulness and harmlessness into errors across three main categories: Content, Logic, and Appropriateness. Experiments on a Korean-sensitive question dataset demonstrate that our score- and error-based improvement pipeline, guided by FINEST, significantly improves the model responses across all three categories, outperforming refinement without guidance. Notably, score-based improvement -- providing category-specific scores and justifications -- yields the most significant gains, reducing the error sentence ratio for Appropriateness by up to 33.09%. This work lays the foundation for a more explainable and comprehensive evaluation and improvement of LLM responses to sensitive questions.

FINEST: Improving LLM Responses to Sensitive Topics Through Fine-Grained Evaluation

TL;DR

FinEST, a FINE-grained response evaluation taxonomy for Sensitive Topics, which breaks down helpfulness and harmlessness into errors across three main categories: Content, Logic, and Appropriateness, is introduced, laying the foundation for a more explainable and comprehensive evaluation and improvement of LLM responses to sensitive questions.

Abstract

Large Language Models (LLMs) often generate overly cautious and vague responses on sensitive topics, sacrificing helpfulness for safety. Existing evaluation frameworks lack systematic methods to identify and address specific weaknesses in responses to sensitive topics, making it difficult to improve both safety and helpfulness simultaneously. To address this, we introduce FINEST, a FINE-grained response evaluation taxonomy for Sensitive Topics, which breaks down helpfulness and harmlessness into errors across three main categories: Content, Logic, and Appropriateness. Experiments on a Korean-sensitive question dataset demonstrate that our score- and error-based improvement pipeline, guided by FINEST, significantly improves the model responses across all three categories, outperforming refinement without guidance. Notably, score-based improvement -- providing category-specific scores and justifications -- yields the most significant gains, reducing the error sentence ratio for Appropriateness by up to 33.09%. This work lays the foundation for a more explainable and comprehensive evaluation and improvement of LLM responses to sensitive questions.
Paper Structure (48 sections, 3 figures, 9 tables)

This paper contains 48 sections, 3 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Fine-grained Evaluation.
  4. Feedback for LLM Response Improvement.
  5. FINEST: Fine-grained Evaluation Taxonomy for Sensitive Topics
  6. Taxonomy Design.
  7. Content
  8. Logic
  9. Appropriateness
  10. Response Improvement Pipeline
  11. Pipeline Overview
  12. Evaluation
  13. Evaluation Schemes.
  14. Evaluation Generation.
  15. Improvement
  16. ...and 33 more sections

Figures (3)

  • Figure 1: Overview of response evaluation and improvement using FINEST. The figure illustrates how FINEST identifies fine-grained errors in LLM responses to sensitive questions, which are then used to enhance the helpfulness and harmlessness of the responses.
  • Figure 2: Win count across response improvement methods. Win count indicates the number of metrics (out of six: error sentence ratio and score for Content, Logic, and Appropriateness) in which a method achieves the best performance. These wins are computed directly from the quantitative results in Table \ref{['tab:diff_percentage']}.
  • Figure 3: Ratio of appropriate, excessive, and insufficient feedback provided by models across three categories: Content, Logic, and Appropriateness (App.), using both (a) score-based and (b) error-based evaluation methods. 80.2% of the evaluations, on average, are considered acceptable (appropriate and excessive), as insufficient evaluations hinder improving responses in terms of not pointing out errors.