Measuring Human Contribution in AI-Assisted Content Generation

TL;DR

This study raises the research question of measuring human contribution in AI-assisted content generation and introduces a framework to address this question that is grounded in information theory and demonstrates that the proposed measure effectively discriminates between varying degrees of human contribution across multiple creative domains.

Abstract

With the growing prevalence of generative artificial intelligence (AI), an increasing amount of content is no longer exclusively generated by humans but by generative AI models with human guidance. This shift presents notable challenges for the delineation of originality due to the varying degrees of human contribution in AI-assisted works. This study raises the research question of measuring human contribution in AI-assisted content generation and introduces a framework to address this question that is grounded in information theory. By calculating mutual information between human input and AI-assisted output relative to self-information of AI-assisted output, we quantify the proportional information contribution of humans in content generation. Our experimental results demonstrate that the proposed measure effectively discriminates between varying degrees of human contribution across multiple creative domains. We hope that this work lays a foundation for measuring human contributions in AI-assisted content generation in the era of generative AI.

Figures (9)

• Figure 1: a. Illustration of AI-assisted content generation, where an AI model is prompted with human input and generates output. b. Overview of the proposed method for measuring human contribution, quantified by the ratio of mutual information between human input and AI-assisted output to the total self-information of the AI-assisted output. c. Outcomes of our proposed measure across various poem generation scenarios using Llama-3, involving varying degrees of human contribution (polishing a human poem, generation with the summary, in other words, key ideas, of a human poem, and generation with a poem title). The center line represents the median, the box limits indicate the upper and lower quartiles, the whiskers extend to 1.5x the interquartile range, and the points are outliers. Our measure effectively differentiates varying degrees of human contribution across the scenarios.
• Figure 2: The distribution of the outcomes of the proposed measure for the constructed dataset. The center line represents the median, the box limits indicate the upper and lower quartiles, the whiskers extend to 1.5x the interquartile range, and the points are outliers. Overall, the proposed measure exhibits the expected trend that lower values are obtained for the generated content with less human contribution.
• Figure 3: The distribution of the outcomes of the proposed measure for academic paper and patent abstracts of different lengths, generated with titles using Llama-3. The center line represents the median, the box limits indicate the upper and lower quartiles, the whiskers extend to 1.5x the interquartile range, and the points are outliers. Overall, the results align with our expectation that with the same human input information the longer the AI-assisted output, the smaller the measured human contribution.
• Figure 4: The distribution of the proposed measurement outcomes for academic paper and patent abstracts, generated using different temperature settings with Llama-3. The center line represents the median, the box limits indicate the upper and lower quartiles, the whiskers extend to 1.5x the interquartile range, and the points are outliers. Overall, the results align with our expectation that larger temperature leads to smaller measured human contribution.
• Figure 5: The distribution of the outcomes of the proposed measure for the constructed dataset of news with and without adaptive attacks using Llama-3. The center line represents the median, the box limits indicate the upper and lower quartiles, the whiskers extend to 1.5x the interquartile range, and the points are outliers. Overall, the adaptive attacks have little to no influence on the measurement outcomes.

Theorems & Definitions (2)

• Definition 1: Human contribution in AI-assisted generation
• Definition 2: Minimal Human Contribution in AI-Assisted Generation