NurValues: Real-World Nursing Values Evaluation for Large Language Models in Clinical Context

TL;DR

NurValues presents the first real-world benchmark for nursing-value alignment in LLMs, derived from a five-month field study across three hospitals and annotated by clinical nurses. It defines five core nursing values and creates Easy- and Hard-Level datasets via counterfactual augmentation and adversarial dialogues, totaling 4,400 labeled samples. Evaluations of 23 SoTA LLMs reveal that Justice is the most challenging dimension and that general LLMs outperform medical LLMs, with in-context learning substantially boosting alignment. The work demonstrates the benchmark’s discriminative power and supports using NurValues to guide value-sensitive development of clinical LLMs, while providing the dataset and code for broader adoption.

Abstract

This work introduces the first benchmark for nursing value alignment, consisting of five core value dimensions distilled from international nursing codes: Altruism, Human Dignity, Integrity, Justice, and Professionalism. The benchmark comprises 1,100 real-world nursing behavior instances collected through a five-month longitudinal field study across three hospitals of varying tiers. These instances are annotated by five clinical nurses and then augmented with LLM-generated counterfactuals with reversed ethic polarity. Each original case is paired with a value-aligned and a value-violating version, resulting in 2,200 labeled instances that constitute the Easy-Level dataset. To increase adversarial complexity, each instance is further transformed into a dialogue-based format that embeds contextual cues and subtle misleading signals, yielding a Hard-Level dataset. We evaluate 23 state-of-the-art (SoTA) LLMs on their alignment with nursing values. Our findings reveal three key insights: (1) DeepSeek-V3 achieves the highest performance on the Easy-Level dataset (94.55), where Claude 3.5 Sonnet outperforms other models on the Hard-Level dataset (89.43), significantly surpassing the medical LLMs; (2) Justice is consistently the most difficult nursing value dimension to evaluate; and (3) in-context learning significantly improves alignment. This work aims to provide a foundation for value-sensitive LLMs development in clinical settings. The dataset and the code are available at https://huggingface.co/datasets/Ben012345/NurValues.

Figures (10)

• Figure 1: The pipeline for dataset construction.
• Figure 2: A:The topic consistency between the simple instances and the complicated dialogues across three LLMs. B: The distribution of nursing behavior types.
• Figure 3: The comparison between the five medical LLMs and their base models.
• Figure 4: Examples from the NurValues Easy-Level dataset illustrating the five core nursing value dimensions (in both English and Chinese).
• Figure 5: A: The nurse station in the orthopedic ward. B: The nurse station in the respiratory and critical care Medicine ward. C: The nurse is measuring the temperature of the patient. D: The Hall and Vaccination Center. All the pictures were taken with the permission of the nurses and patients.