The 2024 Foundation Model Transparency Index

TL;DR

The paper presents the Foundation Model Transparency Index (FMTI) v1.1, a follow-up measurement of transparency among leading foundation-model developers using developer-submitted reports for 100 indicators across upstream, model, and downstream domains. It reports a substantial improvement, with an average score of $58$/100 from $37$ in v1.0, driven by $16.6$ new disclosures per developer, yet identifies persistent opacity in upstream data, labor, and data access. The authors argue that transparency can be meaningfully advanced through iterative reporting, open model-card practices, and targeted policy interventions for developers, customers, transparency advocates, and policymakers. They also acknowledge limitations of the indexing approach and propose next steps, including evolving indicators, standard-setting via procurement, and ongoing governance to translate transparency into tangible societal outcomes.

Abstract

Foundation models are increasingly consequential yet extremely opaque. To characterize the status quo, the Foundation Model Transparency Index (FMTI) was launched in October 2023 to measure the transparency of leading foundation model developers. FMTI 2023 assessed 10 major foundation model developers (e.g. OpenAI, Google) on 100 transparency indicators (e.g. does the developer disclose the wages it pays for data labor?). At the time, developers publicly disclosed very limited information with the average score being 37 out of 100. To understand how the status quo has changed, we conduct a follow-up study after 6 months: we score 14 developers against the same 100 indicators. While in FMTI 2023 we searched for publicly available information, in FMTI 2024 developers submit reports on the 100 transparency indicators, potentially including information that was not previously public. We find that developers now score 58 out of 100 on average, a 21 point improvement over FMTI 2023. Much of this increase is driven by developers disclosing information during the FMTI 2024 process: on average, developers disclosed information related to 16.6 indicators that was not previously public. We observe regions of sustained (i.e. across 2023 and 2024) and systemic (i.e. across most or all developers) opacity such as on copyright status, data access, data labor, and downstream impact. We publish transparency reports for each developer that consolidate information disclosures: these reports are based on the information disclosed to us via developers. Our findings demonstrate that transparency can be improved in this nascent ecosystem, the Foundation Model Transparency Index likely contributes to these improvements, and policymakers should consider interventions in areas where transparency has not improved.

Figures (15)

• Figure 1: Indicators. The 100 indicators we use across 3 domains (upstream, model, and downstream) that are the same as in the October 2023 Foundation Model Transparency Index.
• Figure 2: Scores by Domain. The overall scores disaggregated into the three domains: upstream, model, and downstream.
• Figure 3: Scores by Major Dimensions of Transparency. The fraction of achieved indicators in each of the 13 major dimension of transparency. Major dimension of transparency are large subdomains within the 23 subdomains.
• Figure 4: Overall Scores by Release Strategy. The overall scores for the 6 open developers (Adept, BigCode/Hugging Face/ServiceNow, Meta, Microsoft, Mistral, Stability AI) and the 8 closed developers (AI21 Labs, Aleph Alpha, Amazon, Anthropic, Google, IBM, OpenAI, Writer).
• Figure 5: Change in Overall Scores. The FMTI v1.0 and v1.1 overall scores for the eight developers assessed in both versions.