Human-like Content Analysis for Generative AI with Language-Grounded Sparse Encoders

TL;DR

A content analysis tool, Language-Grounded Sparse Encoders (LanSE), which decompose images into interpretable visual patterns with natural language descriptions and provides decomposed evaluation outperforming existing methods, establishes the first systematic evaluation of physical plausibility, and extends to medical imaging settings.

Abstract

The rapid development of generative AI has transformed content creation, communication, and human development. However, this technology raises profound concerns in high-stakes domains, demanding rigorous methods to analyze and evaluate AI-generated content. While existing analytic methods often treat images as indivisible wholes, real-world AI failures generally manifest as specific visual patterns that can evade holistic detection and suit more granular and decomposed analysis. Here we introduce a content analysis tool, Language-Grounded Sparse Encoders (LanSE), which decompose images into interpretable visual patterns with natural language descriptions. Utilizing interpretability modules and large multimodal models, LanSE can automatically identify visual patterns within data modalities. Our method discovers more than 5,000 visual patterns with 93\% human agreement, provides decomposed evaluation outperforming existing methods, establishes the first systematic evaluation of physical plausibility, and extends to medical imaging settings. Our method's capability to extract language-grounded patterns can be naturally adapted to numerous fields, including biology and geography, as well as other data modalities such as protein structures and time series, thereby advancing content analysis for generative AI.

Figures (32)

• Figure 1: Content Analysis with Language-Grounded Sparse Encoders. LanSE systematically identifies visual patterns in AI-generated content and describes them in natural language (e.g., "dogs in outdoor environments" in natural images and "evidence of bibasilar atelectasis" in medical images). These patterns enable systematic content analysis across multiple applications: quality evaluation, multi-label annotation, and inter-model analysis of generative models across various domains.
• Figure 2: Visual patterns identified by LanSE for natural and medical images. A total of 5,309 different visual patterns in natural images and 899 visual patterns in medical images are automatically identified by LanSE. More than 11,000 human annotations from in total six independent human annotators, along with 18,000 annotations from two LMMs are collected to determine the accuracy of these visual patterns.
• Figure 3: Multi-label pattern detection enables detailed image analysis across domains. LanSE automatically identifies all activated visual patterns in generated images (top) and medical images (bottom), providing comprehensive semantic fingerprints that characterize both content and potential errors. This granular annotation supports applications from dataset curation to targeted model debugging.
• Figure 4: Image diagnostic metrics derived from LanSE visual patterns. Utilizing visual patterns obtained using LanSE, we define four diagnostic metrics for AI-generated natural images, each measuring a distinct aspect of the generative model.
• Figure 5: Validation of LanSE metrics against human judgments. Comparison of average metric values between positive (error-present) and negative (error-free) image sets as classified by human annotators and LMMs. Shown here for visual realism metric across natural images, generated images, and subsets classified by error type. Significant separation between positive and negative sets validates metric effectiveness.