Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Evaluating LLM Metrics Through Real-World Capabilities

Justin K Miller, Wenjia Tang

TL;DR

Evaluates LLMs through real-world utility by identifying six capabilities—Summarization, Technical Assistance, Reviewing Work, Data Structuring, Generation, and Information Retrieval—and five human-centered criteria (coherence, accuracy, clarity, relevance, efficiency). It triangulates large-scale Danish labor data and 4 million Anthropic Claude prompts to map real-world usage to these capabilities, then assesses how well current benchmarks cover them, revealing significant gaps and a recency-driven leaderboard dynamic. Key findings show that only a subset of capabilities is well represented by existing benchmarks, with notable underrepresentation of Reviewing Work and Data Structuring, and that efficiency and interpretability are often neglected. The authors advocate capability-aligned, human-centered evaluation suites, emphasizing multi-turn, transparent, and user-centric assessments to better reflect everyday workflows and guide responsible AI deployment.

Abstract

As generative AI becomes increasingly embedded in everyday workflows, it is important to evaluate its performance in ways that reflect real-world usage rather than abstract notions of intelligence. Unlike many existing benchmarks that assess general intelligence, our approach focuses on real-world utility, evaluating how well models support users in everyday tasks. While current benchmarks emphasize code generation or factual recall, users rely on AI for a much broader range of activities-from writing assistance and summarization to citation formatting and stylistic feedback. In this paper, we analyze large-scale survey data and usage logs to identify six core capabilities that represent how people commonly use Large Language Models (LLMs): Summarization, Technical Assistance, Reviewing Work, Data Structuring, Generation, and Information Retrieval. We then assess the extent to which existing benchmarks cover these capabilities, revealing significant gaps in coverage, efficiency measurement, and interpretability. Drawing on this analysis, we use human-centered criteria to identify gaps in how well current benchmarks reflect common usage that is grounded in five practical criteria: coherence, accuracy, clarity, relevance, and efficiency. For four of the six capabilities, we identify the benchmarks that best align with real-world tasks and use them to compare leading models. We find that Google Gemini outperforms other models-including OpenAI's GPT, xAI's Grok, Meta's LLaMA, Anthropic's Claude, DeepSeek, and Qwen from Alibaba-on these utility-focused metrics.

Evaluating LLM Metrics Through Real-World Capabilities

TL;DR

Evaluates LLMs through real-world utility by identifying six capabilities—Summarization, Technical Assistance, Reviewing Work, Data Structuring, Generation, and Information Retrieval—and five human-centered criteria (coherence, accuracy, clarity, relevance, efficiency). It triangulates large-scale Danish labor data and 4 million Anthropic Claude prompts to map real-world usage to these capabilities, then assesses how well current benchmarks cover them, revealing significant gaps and a recency-driven leaderboard dynamic. Key findings show that only a subset of capabilities is well represented by existing benchmarks, with notable underrepresentation of Reviewing Work and Data Structuring, and that efficiency and interpretability are often neglected. The authors advocate capability-aligned, human-centered evaluation suites, emphasizing multi-turn, transparent, and user-centric assessments to better reflect everyday workflows and guide responsible AI deployment.

Abstract

As generative AI becomes increasingly embedded in everyday workflows, it is important to evaluate its performance in ways that reflect real-world usage rather than abstract notions of intelligence. Unlike many existing benchmarks that assess general intelligence, our approach focuses on real-world utility, evaluating how well models support users in everyday tasks. While current benchmarks emphasize code generation or factual recall, users rely on AI for a much broader range of activities-from writing assistance and summarization to citation formatting and stylistic feedback. In this paper, we analyze large-scale survey data and usage logs to identify six core capabilities that represent how people commonly use Large Language Models (LLMs): Summarization, Technical Assistance, Reviewing Work, Data Structuring, Generation, and Information Retrieval. We then assess the extent to which existing benchmarks cover these capabilities, revealing significant gaps in coverage, efficiency measurement, and interpretability. Drawing on this analysis, we use human-centered criteria to identify gaps in how well current benchmarks reflect common usage that is grounded in five practical criteria: coherence, accuracy, clarity, relevance, and efficiency. For four of the six capabilities, we identify the benchmarks that best align with real-world tasks and use them to compare leading models. We find that Google Gemini outperforms other models-including OpenAI's GPT, xAI's Grok, Meta's LLaMA, Anthropic's Claude, DeepSeek, and Qwen from Alibaba-on these utility-focused metrics.
Paper Structure (31 sections, 3 figures, 1 table)

This paper contains 31 sections, 3 figures, 1 table.

Table of Contents

  1. Introduction
  2. Generative AI Limitations
  3. Patterns of AI Use and Implications for Benchmarking
  4. AI use as Conversation
  5. What are "Good" AI Conversations?
  6. Methods
  7. Identifying AI-Capable Tasks
  8. Validating Capabilities with Real-World Usage
  9. Constructing Prompts and Usage Mapping
  10. Evaluation of Metrics
  11. Results
  12. AI Capabilities
  13. Metrics Used by AI Companies
  14. Technical Assistance
  15. Information Retrieval
  16. ...and 16 more sections

Figures (3)

  • Figure 1: Cumulative distribution of AI usage across 35,014 occupational tasks. Each task is ranked by the percentage of total Claude.ai prompts associated with it, using data from Handa et al. (2025) handa2025economic. The x-axis shows task rank in descending order of frequency, and the y-axis represents the cumulative percentage of total prompts. A small number of tasks dominate usage: the top 100 tasks account for just over 50% of all prompts.
  • Figure 2: Percentage of the top 100 occupational tasks for which each AI capability is relevant. A task can be associated with multiple capabilities. For example, the task “modify existing software to correct errors, to adapt it to new hardware, or to upgrade interfaces and improve performance” involves both Technical Assistance and Reviewing Work. The percentages reflect the sum of the percentage scores (derived from over 4 million prompts) associated with each task–capability pair, divided by the maximum cumulative importance across the top 100 tasks. This highlights how commonly each capability appears among the tasks people are most likely to use AI for.
  • Figure 3: Evaluation results for highest ranking models, we only take the highest scoring model from one company, so they may have models that score higher than other companies. Scores are correct as of 12th May 2025. Across four of the six different AI capabilities outlined in this paper. (a) Technical Assistance (WebDev Arena): Elo-style Arena scores, derived from head-to-head matchups in web development tasks, reflect relative performance. Higher values indicate better comparative results: error bars show 95% confidence intervals. (b) Information Retrieval (SimpleQA benchmark): Accuracy percentage on a benchmark testing factual correctness in short-answer question responses. The y-axis shows the proportion of correct answers; higher values indicate better factual precision. (c) Summarization (Facts-grounding benchmark): Percentage of grounded content in model-generated summaries, based on human judgments of factual consistency with source documents. Error bars indicate 95% confidence intervals. (d) Generation (Creative Writing Arena): Elo scores from subjective human preferences in open-ended Generation tasks such as storytelling and creative writing. Higher scores denote consistent wins in pairwise comparisons; error bars show 95% confidence intervals.