Interactive Visual Assessment for Text-to-Image Generation Models

TL;DR

DyEval is proposed, an LLM-powered dynamic interactive visual assessment framework that facilitates collaborative evaluation between humans and generative models for text-to-image systems and provides valuable insights for improving generative models.

Abstract

Visual generation models have achieved remarkable progress in computer graphics applications but still face significant challenges in real-world deployment. Current assessment approaches for visual generation tasks typically follow an isolated three-phase framework: test input collection, model output generation, and user assessment. These fashions suffer from fixed coverage, evolving difficulty, and data leakage risks, limiting their effectiveness in comprehensively evaluating increasingly complex generation models. To address these limitations, we propose DyEval, an LLM-powered dynamic interactive visual assessment framework that facilitates collaborative evaluation between humans and generative models for text-to-image systems. DyEval features an intuitive visual interface that enables users to interactively explore and analyze model behaviors, while adaptively generating hierarchical, fine-grained, and diverse textual inputs to continuously probe the capability boundaries of the models based on their feedback. Additionally, to provide interpretable analysis for users to further improve tested models, we develop a contextual reflection module that mines failure triggers of test inputs and reflects model potential failure patterns supporting in-depth analysis using the logical reasoning ability of LLM. Qualitative and quantitative experiments demonstrate that DyEval can effectively help users identify max up to 2.56 times generation failures than conventional methods, and uncover complex and rare failure patterns, such as issues with pronoun generation and specific cultural context generation. Our framework provides valuable insights for improving generative models and has broad implications for advancing the reliability and capabilities of visual generation systems across various domains.

Figures (15)

• Figure 1: Diagram of the DyEval, a dynamic interactive testing framework for Text-to-Image (T2I) models. DyEval leverages a Large Language Model (LLM) to generate test prompts based on model feedback dynamically. The T2I model outputs are assessed, and results are used to update a test tree. Nodes with high pass rates (green) continue to explore deeper layers, while nodes with high fail rates (brown) are analyzed by the LLM to analyze failure reasons.
• Figure 2: Overview of DyEval. The DyEval process begins with setting initial topics and model parameters (Step A). In the test input generation phase, the LLM creates initial test inputs for the selected topic (Step B). Evaluators then review and annotate the generated images (Step C) and update the test tree accordingly (Step D). If a test node shows low average pass rates (Step E), the LLM refines these inputs to identify potential failure triggers in the dynamic failure location module (Step F). It then reflects on all available information related to the current topic (Step G) and updates the test tree (Step H). If the pass rates are satisfactory, the LLM proposes new topics based on the test context (history testing records in the testing tree) (Step I). Evaluators can select these new topics (Step J) to generate additional test inputs, continuing the evaluation cycle. This iterative process continues until the maximum exploration depth is reached.
• Figure 3: Average number of bugs accumulated within an initial topic throughout thirteen test nodes (1 + 1$\times$3 + 3$\times$3) during the testing process of DyEval of SD1-5, SD2-1, SDXL, SD3. The standard error is over nine initial topics. The index of the test node of the test tree is obtained according to the breadth-first search. DyEval can constantly find bugs in the model under test, and weaker models are likelier to find bugs (SD1-5, SD2-1 consistently higher than SDXL, SD3). The shading represents the variance.
• Figure 4: Comparison experiments between DyEval and non-adaptive testing (LLM directly generates the same amount of text input). The horizontal line represents the variance. Compared to the non-adaptive fashions, DyEval can find more generation failures in assessing the same number of text-image pairs, with greater distinguishability across models and better stability across test aspects.
• Figure 5: A test process case of DyEval. The initial test topic is "Spacial relationships" and the APR of this test node is 0.60. Based on the test records of the current topic, LLM continuously generates new test topics to analyze the model capability boundaries further. We also provide specific test inputs in the path of the test tree. Green represents nodes with test pass rate greater than or equal to 0.6, light orange represents pass rate (0.6,0.3], and less than 0.3 is dark orange.