ConSCompF: Consistency-focused Similarity Comparison Framework for Generative Large Language Models
Alexey Karev, Dong Xu
TL;DR
ConSCompF tackles the challenge of comparing generative LLM outputs using minimal labeled data by introducing a consistency-aware similarity framework. It encodes multiple responses per instruction, derives a general representation per instruction, and adjusts inter-model similarity by instruction consistency, enabling robust few-shot comparisons. Through experiments on quantized TinyLlama and a diverse set of open LLMs, the approach reveals degradation patterns, clustering by data sources, and the impact of prompts, with PCA visualizations. The results show that ConSCompF correlates with traditional benchmarks (e.g., ROUGE-L) and remains effective in few-shot settings, albeit sensitive to instruction consistency and encoder choice. This framework offers a fast, inexpensive tool for LLM categorization, similarity mapping, and potential fraud-detection signals, complementing existing benchmarks.
Abstract
Large language models (LLMs) have been one of the most important discoveries in machine learning in recent years. LLM-based artificial intelligence (AI) assistants, such as ChatGPT, have consistently attracted the attention from researchers, investors, and the general public, driving the rapid growth of this industry. With the frequent introduction of new LLMs to the market, it becomes increasingly difficult to differentiate between them, creating a demand for new LLM comparison methods. In this research, the Consistency-focused Similarity Comparison Framework (ConSCompF) for generative large language models is proposed. It compares texts generated by two LLMs and produces a similarity score, indicating the overall degree of similarity between their responses. The main advantage of this framework is that it can operate on a small number of unlabeled data, such as chatbot instruction prompts, and does not require LLM developers to disclose any information about their product. To evaluate the efficacy of ConSCompF, two experiments aimed at identifying similarities between multiple LLMs are conducted. Additionally, these experiments examine the correlation between the similarity scores generated by ConSCompF and the differences in the outputs produced by other benchmarking techniques, such as ROUGE-L. Finally, a series of few-shot LLM comparison experiments is conducted to evaluate the performance of ConSCompF in a few-shot LLM comparison scenario. The proposed framework can be used for calculating similarity matrices of multiple LLMs, which can be effectively visualized using principal component analysis (PCA). The ConSCompF output may provide useful insights into data that might have been used during LLM training and help detect possible investment fraud attempts.