RAIL in the Wild: Operationalizing Responsible AI Evaluation Using Anthropic's Value Dataset
Sumit Verma, Pritam Prasun, Arpit Jaiswal, Pritish Kumar
TL;DR
This paper tackles the challenge of turning abstract AI ethics into actionable evaluation by applying the eight-dimension RAIL framework to real-world conversations. Using Anthropic's Values in the Wild dataset (308,210 anonymized Claude conversations and 3,307 value expressions), it maps AI-expressed values to RAIL dimensions and computes a normalized score for each dimension via $Score_D = \\sum_{v \\in V_D} \\mathrm{pct\\_convos}(v)$ and $\\text{Normalized Score}_D = 10 \\times \\frac{\\text{Score}_D}{\\max(S)}$. It reports that User Impact, Inclusivity, and Reliability are strong, while Privacy, Accountability, and some aspects of Fairness are underexpressed, highlighting measurable gaps. The work offers practical recommendations and deployment guidance for developers and demonstrates how domain-specific value monitoring can support governance and auditability of LLMs.
Abstract
As AI systems become embedded in real-world applications, ensuring they meet ethical standards is crucial. While existing AI ethics frameworks emphasize fairness, transparency, and accountability, they often lack actionable evaluation methods. This paper introduces a systematic approach using the Responsible AI Labs (RAIL) framework, which includes eight measurable dimensions to assess the normative behavior of large language models (LLMs). We apply this framework to Anthropic's "Values in the Wild" dataset, containing over 308,000 anonymized conversations with Claude and more than 3,000 annotated value expressions. Our study maps these values to RAIL dimensions, computes synthetic scores, and provides insights into the ethical behavior of LLMs in real-world use.