Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

The Dual-use Dilemma in LLMs: Do Empowering Ethical Capacities Make a Degraded Utility?

Yiyi Zhang, Xingyu Chen, Kexin Chen, Yuyang Du, Xilin Dang, Pheng-Ann Heng

TL;DR

The paper tackles the dual-use dilemma in chemistry-focused LLMs by balancing safety and utility through a Direct Preference Optimization (DPO) based LibraAlign framework. It introduces LibraChemQA, a large chemistry ethics dataset built via a GPT-assisted three-phase data generation scheme and balanced seed concept, enabling robust learning of safe yet useful responses. A hybrid evaluation framework combining rule-based and GPT-based judges provides precise safety-utility assessment, and LibraChem demonstrates superior overall performance against multiple baselines on both text and SMILES formats. The study highlights ethical risks in current models (notably DeepSeek-R1 CoT outputs) and offers a blueprint for deploying ethical, high-utility domain-specific LLMs with broader applicability beyond chemistry.

Abstract

Recent years have witnessed extensive efforts to enhance Large Language Models (LLMs) across various domains, alongside growing attention to their ethical implications. However, a critical challenge remains largely overlooked: LLMs must balance between rejecting harmful requests for safety and accommodating legitimate ones for utility. This paper presents a Direct Preference Optimization (DPO) based alignment framework that achieves better overall performance by addressing this ethical-utility trade-off, using chemical domain applications as a proof-of-concept. Our alignment pipeline starts with a GPT-assisted three-phase data generation scheme, in which we create LibraChemQA, a chemical question-answering dataset comprising 31.6k triplet instances. By incorporating an innovative balanced seed in the data generation process, our framework systematically considers both legitimate and illegitimate requests. The framework also introduces a rephrasing mechanism for efficient data augmentation that enhances the model's chemical comprehension. We further develop a novel hybrid evaluation scheme with LLM judges for precise assessment of both safety and utility. Experimental results demonstrate our model's substantial improvements in overall performance where both safety and utility are considered - the resulting model outperforms leading LLMs including Claude-3, GPT-4o, and LLaMA-3 by margins of 13.44%, 7.16%, and 7.10% respectively on our released benchmark. At the end of this paper, we analyze experimental results obtained from testing DeepSeek-R1 on our benchmark and reveal the critical ethical concerns raised by this highly acclaimed model. We highlight that the long Chain-of-Thought (CoT) reasoning process employed by DeepSeek-R1, as well as other LLMs distilled from it, introduces significant ethical vulnerabilities when exposed to users.

The Dual-use Dilemma in LLMs: Do Empowering Ethical Capacities Make a Degraded Utility?

TL;DR

The paper tackles the dual-use dilemma in chemistry-focused LLMs by balancing safety and utility through a Direct Preference Optimization (DPO) based LibraAlign framework. It introduces LibraChemQA, a large chemistry ethics dataset built via a GPT-assisted three-phase data generation scheme and balanced seed concept, enabling robust learning of safe yet useful responses. A hybrid evaluation framework combining rule-based and GPT-based judges provides precise safety-utility assessment, and LibraChem demonstrates superior overall performance against multiple baselines on both text and SMILES formats. The study highlights ethical risks in current models (notably DeepSeek-R1 CoT outputs) and offers a blueprint for deploying ethical, high-utility domain-specific LLMs with broader applicability beyond chemistry.

Abstract

Recent years have witnessed extensive efforts to enhance Large Language Models (LLMs) across various domains, alongside growing attention to their ethical implications. However, a critical challenge remains largely overlooked: LLMs must balance between rejecting harmful requests for safety and accommodating legitimate ones for utility. This paper presents a Direct Preference Optimization (DPO) based alignment framework that achieves better overall performance by addressing this ethical-utility trade-off, using chemical domain applications as a proof-of-concept. Our alignment pipeline starts with a GPT-assisted three-phase data generation scheme, in which we create LibraChemQA, a chemical question-answering dataset comprising 31.6k triplet instances. By incorporating an innovative balanced seed in the data generation process, our framework systematically considers both legitimate and illegitimate requests. The framework also introduces a rephrasing mechanism for efficient data augmentation that enhances the model's chemical comprehension. We further develop a novel hybrid evaluation scheme with LLM judges for precise assessment of both safety and utility. Experimental results demonstrate our model's substantial improvements in overall performance where both safety and utility are considered - the resulting model outperforms leading LLMs including Claude-3, GPT-4o, and LLaMA-3 by margins of 13.44%, 7.16%, and 7.10% respectively on our released benchmark. At the end of this paper, we analyze experimental results obtained from testing DeepSeek-R1 on our benchmark and reveal the critical ethical concerns raised by this highly acclaimed model. We highlight that the long Chain-of-Thought (CoT) reasoning process employed by DeepSeek-R1, as well as other LLMs distilled from it, introduces significant ethical vulnerabilities when exposed to users.
Paper Structure (25 sections, 6 equations, 4 figures, 4 tables)

This paper contains 25 sections, 6 equations, 4 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Preliminary
  3. RLHF
  4. DPO
  5. Method
  6. GPT-assisted Three-phase Data Generation
  7. Raw Data Mining and Crafting
  8. GPT-Assisted Data Rephrasing
  9. Data Combination
  10. Hybrid Evaluation Framework
  11. Dataset and Experiments
  12. Custom Dataset
  13. LibraChemQA_TEXT
  14. LibraChemQA_SMILES
  15. Experimental Results
  16. ...and 10 more sections

Figures (4)

  • Figure 1: Overview of the LibraChemQA dataset. LibraChemQA is generated through a GPT-assisted data generation scheme, which consists of the following phases: 1) raw data mining and crafting, 2) GPT-assisted data rephrasing, and 3) data combination.
  • Figure 2: Ablation results of safety, utility, and overall performance along with different settings of the balanced seed.
  • Figure 3: Ablation results of safety, utility and overall performance along with different combination methods.
  • Figure 4: An example illustrating DeepSeek-R1’s CoT reasoning process in response to a query about synthesizing MDPV. While the model ultimately refuses to provide explicit assistance, its step-by-step reasoning inadvertently reveals a detailed chemical synthesis pathway, highlighting a critical ethical vulnerability. Note that we have redacted specific chemical reaction details in this example for ethical considerations in our paper, but in the actual DeepSeek response, these details were clearly presented without any omission.