Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

CIBench: Evaluating Your LLMs with a Code Interpreter Plugin

Chuyu Zhang, Songyang Zhang, Yingfan Hu, Haowen Shen, Kuikun Liu, Zerun Ma, Fengzhe Zhou, Wenwei Zhang, Xuming He, Dahua Lin, Kai Chen

TL;DR

An interactive evaluation framework, named CIBench, is proposed to comprehensively assess LLMs' ability to utilize code interpreters for data science tasks and provides valuable insights for future LLMs in code interpreter utilization.

Abstract

While LLM-Based agents, which use external tools to solve complex problems, have made significant progress, benchmarking their ability is challenging, thereby hindering a clear understanding of their limitations. In this paper, we propose an interactive evaluation framework, named CIBench, to comprehensively assess LLMs' ability to utilize code interpreters for data science tasks. Our evaluation framework includes an evaluation dataset and two evaluation modes. The evaluation dataset is constructed using an LLM-human cooperative approach and simulates an authentic workflow by leveraging consecutive and interactive IPython sessions. The two evaluation modes assess LLMs' ability with and without human assistance. We conduct extensive experiments to analyze the ability of 24 LLMs on CIBench and provide valuable insights for future LLMs in code interpreter utilization.

CIBench: Evaluating Your LLMs with a Code Interpreter Plugin

TL;DR

An interactive evaluation framework, named CIBench, is proposed to comprehensively assess LLMs' ability to utilize code interpreters for data science tasks and provides valuable insights for future LLMs in code interpreter utilization.

Abstract

While LLM-Based agents, which use external tools to solve complex problems, have made significant progress, benchmarking their ability is challenging, thereby hindering a clear understanding of their limitations. In this paper, we propose an interactive evaluation framework, named CIBench, to comprehensively assess LLMs' ability to utilize code interpreters for data science tasks. Our evaluation framework includes an evaluation dataset and two evaluation modes. The evaluation dataset is constructed using an LLM-human cooperative approach and simulates an authentic workflow by leveraging consecutive and interactive IPython sessions. The two evaluation modes assess LLMs' ability with and without human assistance. We conduct extensive experiments to analyze the ability of 24 LLMs on CIBench and provide valuable insights for future LLMs in code interpreter utilization.
Paper Structure (37 sections, 21 figures, 5 tables)

This paper contains 37 sections, 21 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Model with Plugins
  4. Related Benchmarks
  5. CIBench
  6. Evaluation Dataset
  7. Python Modules Selection
  8. Topic Candidates Generation
  9. Tasks Generation and Refinement
  10. Human Refinement
  11. Evaluation Modes and Metrics
  12. Evaluation Modes
  13. Evaluation Metrics
  14. Experiments
  15. Experiments Setup
  16. ...and 22 more sections

Figures (21)

  • Figure 1: Features of our benchmark. Our benchmark consists of interactive sessions, diverse tasks covering various Python modules, and comprehensive evaluations (The tool-call rate is not displayed).
  • Figure 2: Overview of CIBench. CIBench first selects Python modules to generate candidate topics and then generates tasks based on these modules and the selected topic. Additionally, humans are engaged to generate new tasks to ensure diversity and filter out incorrect questions to enhance quality.
  • Figure 3: An example prompt of task generation.
  • Figure 4: Evaluation modes: In end-to-end mode, the LLM addresses the user's question (bottom) within the context of its response, while in oracle mode, it answers the user's question (bottom) within the context of ground truth.
  • Figure 5: Correlation of CIBench with other benchmarks. The small p-value (top-left) and high Pearson correlation coefficients (title) indicate a strong correlation between CIBench and IFEval, BBH, GSM8K, MATH, HumanEval, and MBPP. These benchmarks evaluate the instruction-following, reasoning, and coding abilities of LLMs, respectively.
  • ...and 16 more figures