Understanding Generative AI Content with Embedding Models

TL;DR

Of the many applications for this framework, it is found empirical evidence that there is intrinsic separability between real samples and those generated by artificial intelligence (AI).

Abstract

Constructing high-quality features is critical to any quantitative data analysis. While feature engineering was historically addressed by carefully hand-crafting data representations based on domain expertise, deep neural networks (DNNs) now offer a radically different approach. DNNs implicitly engineer features by transforming their input data into hidden feature vectors called embeddings. For embedding vectors produced by foundation models -- which are trained to be useful across many contexts -- we demonstrate that simple and well-studied dimensionality-reduction techniques such as Principal Component Analysis uncover inherent heterogeneity in input data concordant with human-understandable explanations. Of the many applications for this framework, we find empirical evidence that there is intrinsic separability between real samples and those generated by artificial intelligence (AI).

Figures (14)

• Figure 1: (A) Framework of our techniques. The full neural network begins by engineering a vector representation of the data through its feature embedder (FE). The embedded data is then passed through an interchangeable machine learning layer, such as Principal Component Analysis (PCA). (B) Sample images of cat images from the LSUN dataset and Cats&Dogs dataset, embedded by Apple's Data Filtering Network. (C) Top three principal components (PCs) on Spanish and Russian news articles, including Russian-to-Spanish machine translations, data embedded by Microsoft's multilingual-e5-large. In order, PCs correspond to language, topic, and natural or translated text.
• Figure 2: Embedded data to distinguish real and fake data. DNN embeddings of AI-generated data alongside real data, after dimensionality reductions. (A) User-posted Stack Exchange answers and responses by three language models (Mistral 8x7B, Llama-2 70B, and Falcon 40B). Embeddings by Mistral 7B. (B) Real cat images from the LSUN dataset and images produced by three image generation models (DDPM, Stable Diffusion, and Open-Dalle). Embeddings by Apple’s Data Filtering Network. (C) Real economics abstracts and two synthetic samples created by Llama-2 70B using different prompting methods. Embeddings by Mistral 7B. (D) Real and AI-generated images from the GenImage dataset containing data produced by eight generative models and ImageNet. Embeddings by Apple’s Data Filtering network. (E) Principal components of embedded Stack Exchange responses, contaminated with AI-generated answers by Llama-2 70B. Embeddings by Mistral 7B. AI-generated answers are revealed as outliers when projecting to PCs 11 and 20.
• Figure 3: Organizing data through embeddings.(A) We can deduce explanations for the differences in real and AI-generated abstracts using regressions on the PCs; we find the high occurrence of the words 'important,' 'significant,' or 'valuable' highly correlate with the shift in PC4 (see Table \ref{['tab:regressions']} for details). (B) Top three principal components on embedded abstracts yield clustering of physics, biology, and computer science subjects. (C) Supervised learning through LDA easily distinguished all ten abstract sources: five subject areas by real and AI. Test acc: $99.0\% \pm 0.47\%$.
• Figure S1: Embedding models comparison. Comparison of test accuracy across all classes of the listed experiment of LDA applied to embeddings from respective models is provided. Boldface indicates that that model was used for the results presented in this work, unless otherwise indicated.
• Figure S2: Top three components of multilingual news data for various language pairs.