AutoTestForge: A Multidimensional Automated Testing Framework for Natural Language Processing Models

TL;DR

AutoTestForge presents an automated, multidimensional framework for evaluating NLP models by leveraging LLMs to generate rich test templates and instantiate large test suites. It integrates differential testing with a multi-model voting mechanism and expands coverage across taxonomy, fairness, and robustness, enabling thorough identification of model weaknesses. Across sentiment analysis and semantic textual similarity, AutoTestForge achieves higher error-detection rates than traditional datasets and state-of-the-art tooling, with consistent performance across generation modes and LLMs. The work demonstrates significant efficiency gains over manual template design and proposes a practical, scalable approach for comprehensive NLP model testing with potential for broad adoption and extension.

Abstract

In recent years, the application of behavioral testing in Natural Language Processing (NLP) model evaluation has experienced a remarkable and substantial growth. However, the existing methods continue to be restricted by the requirements for manual labor and the limited scope of capability assessment. To address these limitations, we introduce AutoTestForge, an automated and multidimensional testing framework for NLP models in this paper. Within AutoTestForge, through the utilization of Large Language Models (LLMs) to automatically generate test templates and instantiate them, manual involvement is significantly reduced. Additionally, a mechanism for the validation of test case labels based on differential testing is implemented which makes use of a multi-model voting system to guarantee the quality of test cases. The framework also extends the test suite across three dimensions, taxonomy, fairness, and robustness, offering a comprehensive evaluation of the capabilities of NLP models. This expansion enables a more in-depth and thorough assessment of the models, providing valuable insights into their strengths and weaknesses. A comprehensive evaluation across sentiment analysis (SA) and semantic textual similarity (STS) tasks demonstrates that AutoTestForge consistently outperforms existing datasets and testing tools, achieving higher error detection rates (an average of $30.89\%$ for SA and $34.58\%$ for STS). Moreover, different generation strategies exhibit stable effectiveness, with error detection rates ranging from $29.03\% - 36.82\%$.

Figures (4)

• Figure 1: Overview of AutoTestForge
• Figure 2: An example of test template
• Figure 3: Overview of constructing the final test suite
• Figure 4: Performance of various capability test suites generated by AutoTestForge using LLMs on Distilbert-base-cased-SST-2. The Original and Expand test suites correspond to the transformed test suites in Section \ref{['sec:framework-A']} by $\mathcal{L}(D_i)$ and $\mathcal{E}(\mathcal{L}(D_i))$, respectively. The Fairness, Taxonomy, and Pre-rob test suites refer to the identically named test suites $\mathcal{T}_{fair}$,$\mathcal{T}_{tax}$, and $\mathcal{T}_{pre-rob}$ in Section \ref{['sec:framework-C']}, respectively. The Adv-rob test suite refers to the suite $\mathcal{T}_{adv-rob}$ from Section \ref{['sec:framework-C']}.