PPTC-R benchmark: Towards Evaluating the Robustness of Large Language Models for PowerPoint Task Completion

TL;DR

The PowerPoint Task Completion Robustness benchmark (PPTC-R) is proposed to measure LLMs' robustness to the user PPT task instruction and software version, and constructs adversarial user instructions by attacking user instructions at sentence, semantic, and multi-language levels.

Abstract

The growing dependence on Large Language Models (LLMs) for finishing user instructions necessitates a comprehensive understanding of their robustness to complex task completion in real-world situations. To address this critical need, we propose the PowerPoint Task Completion Robustness benchmark (PPTC-R) to measure LLMs' robustness to the user PPT task instruction and software version. Specifically, we construct adversarial user instructions by attacking user instructions at sentence, semantic, and multi-language levels. To assess the robustness of Language Models to software versions, we vary the number of provided APIs to simulate both the newest version and earlier version settings. Subsequently, we test 3 closed-source and 4 open-source LLMs using a benchmark that incorporates these robustness settings, aiming to evaluate how deviations impact LLMs' API calls for task completion. We find that GPT-4 exhibits the highest performance and strong robustness in our benchmark, particularly in the version update and the multilingual settings. However, we find that all LLMs lose their robustness when confronted with multiple challenges (e.g., multi-turn) simultaneously, leading to significant performance drops. We further analyze the robustness behavior and error reasons of LLMs in our benchmark, which provide valuable insights for researchers to understand the LLM's robustness in task completion and develop more robust LLMs and agents. We release the code and data at \url{https://github.com/ZekaiGalaxy/PPTCR}.

Figures (11)

• Figure 1: We illustrate the turn-base multilingual results of closed-source LLMs.
• Figure 2: We illustrate two examples for constructing our robustness benchmark. The perturbations correctly distract the LLM from completing the user instruction (the left) and mislead the LLM into generating the wrong API sequence (the right), which underscores the importance of evaluating and analyzing LLMs' task completion robustness.
• Figure 3: The prompts we used to create the sentence and semantic level perturbations. '<number>' is the number of paraphrased Instructions.
• Figure 4: We illustrate the turn-based results of closed-source LLMs in the creating new slides task, where the instructions are translated into 14 non-English languages. The bar for each language represents the LLM's accuracy in the corresponding language setting. The dotted line is the LLM's accuracy when tested in the English setting.
• Figure 5: We report the results of three LLMs with different numbers of new APIs in sub-figures (a), (b), and (c). The session-based accuracy of all LLMs' editing template task performance is pretty low (<4).