Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

TaskEval: Synthesised Evaluation for Foundation-Model Tasks

Dilani Widanapathiranage, Scott Barnett, Stefanus Kurniawan, Wannita Takerngsaksiri

TL;DR

The paper tackles the challenge of evaluating foundation-model tasks by proposing TaskEval, a framework that synthesises task-specific evaluators via a task-agnostic meta-model, a structured interaction protocol, and an eval synthesiser. It implements a runnable Evaluator API and UI to automate evaluation while enabling human-in-the-loop feedback, aiming to support continuous evaluation without ground-truth data. Demonstrations on chart data extraction and document question answering show promising preliminary accuracy (93% and 90%, respectively) in generating and validating evals. The work offers a path for engineering teams to bootstrap task-specific evaluation tooling for FM-driven applications, bridging the gap between automated evaluation and human oversight.

Abstract

Hallucinations are a key concern when creating applications that rely on Foundation models (FMs). Understanding where and how these subtle failures occur in an application relies on evaluation methods known as \textit{evals}. Prior work focuses on defining new eval methods or benchmark datasets for specific tasks. However, neither helps a software team with a task-specific FM application when there is no metric or dataset. The demand for both automated approaches and deep integration of human insight makes this a challenging problem. We address this gap by proposing an approach to synthesise a FM task-specific evaluator program that provides automation and a custom UI for capturing feedback. The core novelty of our approach lies in: (1) a task-agnostic meta-model that captures properties of any FM task, (2) an interaction protocol for efficient use of human feedback, and (3) an eval synthesiser that selects or generates an appropriate set of evals. We implement our approach in \toolname and demonstrate the concept on two diverse FM tasks: chart data extraction and document question answering. A preliminary evaluation on the quality of our selected evals shows 93\% and 90\% accuracy respectively. Our research tackles a growing problem facing engineering teams, how to evaluate and review outputs from FM tasks.

TaskEval: Synthesised Evaluation for Foundation-Model Tasks

TL;DR

The paper tackles the challenge of evaluating foundation-model tasks by proposing TaskEval, a framework that synthesises task-specific evaluators via a task-agnostic meta-model, a structured interaction protocol, and an eval synthesiser. It implements a runnable Evaluator API and UI to automate evaluation while enabling human-in-the-loop feedback, aiming to support continuous evaluation without ground-truth data. Demonstrations on chart data extraction and document question answering show promising preliminary accuracy (93% and 90%, respectively) in generating and validating evals. The work offers a path for engineering teams to bootstrap task-specific evaluation tooling for FM-driven applications, bridging the gap between automated evaluation and human oversight.

Abstract

Hallucinations are a key concern when creating applications that rely on Foundation models (FMs). Understanding where and how these subtle failures occur in an application relies on evaluation methods known as \textit{evals}. Prior work focuses on defining new eval methods or benchmark datasets for specific tasks. However, neither helps a software team with a task-specific FM application when there is no metric or dataset. The demand for both automated approaches and deep integration of human insight makes this a challenging problem. We address this gap by proposing an approach to synthesise a FM task-specific evaluator program that provides automation and a custom UI for capturing feedback. The core novelty of our approach lies in: (1) a task-agnostic meta-model that captures properties of any FM task, (2) an interaction protocol for efficient use of human feedback, and (3) an eval synthesiser that selects or generates an appropriate set of evals. We implement our approach in \toolname and demonstrate the concept on two diverse FM tasks: chart data extraction and document question answering. A preliminary evaluation on the quality of our selected evals shows 93\% and 90\% accuracy respectively. Our research tackles a growing problem facing engineering teams, how to evaluate and review outputs from FM tasks.

Paper Structure

This paper contains 10 sections, 3 figures.

Table of Contents

  1. Introduction
  2. Motivation
  3. Vision
  4. Task-agnostic Meta-model
  5. Task Interaction Protocol
  6. Eval Synthesiser
  7. Why is it new?
  8. TaskEval Demonstration
  9. Future plans
  10. Risk

Figures (3)

  • Figure 1: Chart-to-dataframe example polak2025leveraging with GPT-5. Human review required to identify: incorrect values (red), spurious values (yellow), and missing values (purple).
  • Figure 2: TASKEVAL system overview. Blue boxes show the novel elements of our solution: a) Task Interaction protocol, b) task-agnostic meta-model, and c) Eval Synthesiser.
  • Figure 3: Output from TaskEval for two tasks: chart data extraction (left) and document QA (right). Depending on task description, availability of labels and input data, TaskEval selects task-specific a) evals (custom visualisation vs LLM-as-a-judge), b) human labels, and c) UI components and layout.