Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ReFeR: Improving Evaluation and Reasoning through Hierarchy of Models

Yaswanth Narsupalli, Abhranil Chandra, Sreevatsa Muppirala, Manish Gupta, Pawan Goyal

TL;DR

This study introduces a tuning-free framework, ReFeR, designed to evaluate generative outputs, including both text and images, by leveraging a 2-level hierarchy of LLMs and VLMs themselves, and presents two variants of the framework: ReFeR-Turbo, optimized for accelerated performance, and ReFeR-Lite, offering a more cost-effective solution.

Abstract

Assessing the quality of outputs generated by generative models, such as large language models and vision language models, presents notable challenges. Traditional methods for evaluation typically rely on either human assessments, which are resource-intensive, or automatic metrics that often show a low correlation with human judgment. Another common approach is to use deep learning systems, which not only consume a substantial amount of compute and time but also require extensive training data. In this study, we introduce a tuning-free framework called ReFeR, designed to evaluate generative outputs, including both text and images, by leveraging a 2-level hierarchy of LLMs and VLMs themselves. We rigorously evaluate our framework, ReFeR, across four diverse evaluation tasks. The framework not only improves the accuracy of these evaluations, surpassing previous benchmarks but also generates constructive feedback. Interestingly, the framework is also applicable to reasoning tasks. Experiments on four reasoning tasks demonstrate superior collective reasoning abilities of the framework. We present two variants of the framework: ReFeR-Turbo, optimized for accelerated performance, and ReFeR-Lite, offering a more cost-effective solution. ReFeR-Lite is $\sim7.7\times$ more efficient while being comparably accurate to ReFeR-Turbo. We make code, data and PIP package publicly available. See this PIP URL https://pypi.org/project/refer-agents/ and this Git URL https://github.com/yaswanth-iitkgp/ReFeR_Code .

ReFeR: Improving Evaluation and Reasoning through Hierarchy of Models

TL;DR

This study introduces a tuning-free framework, ReFeR, designed to evaluate generative outputs, including both text and images, by leveraging a 2-level hierarchy of LLMs and VLMs themselves, and presents two variants of the framework: ReFeR-Turbo, optimized for accelerated performance, and ReFeR-Lite, offering a more cost-effective solution.

Abstract

Assessing the quality of outputs generated by generative models, such as large language models and vision language models, presents notable challenges. Traditional methods for evaluation typically rely on either human assessments, which are resource-intensive, or automatic metrics that often show a low correlation with human judgment. Another common approach is to use deep learning systems, which not only consume a substantial amount of compute and time but also require extensive training data. In this study, we introduce a tuning-free framework called ReFeR, designed to evaluate generative outputs, including both text and images, by leveraging a 2-level hierarchy of LLMs and VLMs themselves. We rigorously evaluate our framework, ReFeR, across four diverse evaluation tasks. The framework not only improves the accuracy of these evaluations, surpassing previous benchmarks but also generates constructive feedback. Interestingly, the framework is also applicable to reasoning tasks. Experiments on four reasoning tasks demonstrate superior collective reasoning abilities of the framework. We present two variants of the framework: ReFeR-Turbo, optimized for accelerated performance, and ReFeR-Lite, offering a more cost-effective solution. ReFeR-Lite is more efficient while being comparably accurate to ReFeR-Turbo. We make code, data and PIP package publicly available. See this PIP URL https://pypi.org/project/refer-agents/ and this Git URL https://github.com/yaswanth-iitkgp/ReFeR_Code .
Paper Structure (48 sections, 3 equations, 7 figures, 12 tables, 1 algorithm)

This paper contains 48 sections, 3 equations, 7 figures, 12 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Methodology
  3. ReFeR Framework
  4. Prompting Schema
  5. ReFeR Variants
  6. ReFeR-Turbo
  7. ReFeR-Lite
  8. Experiments
  9. Datasets
  10. Baselines
  11. Implementation Details
  12. Results and Discussions
  13. NLG Evaluation
  14. Multimodal Evaluation
  15. Reasoning
  16. ...and 33 more sections

Figures (7)

  • Figure 1: Illustration of the ReFeR Framework on the TopicalChat dataset. Refer to Fig. \ref{['fig:refer-multimodal']} (in the appendix) for illustration of ReFeR for multimodality and Algorithm \ref{['algo:refer']} showing the framework's working. We use the predictions from AC to create an Instruction tuning dataset which can be used to improve the performance of smaller models as evaluators, shown in Appendix \ref{['appendix: finetuning']}
  • Figure 2: Framework Ablation. Results obtained on ReFeR-Turbo by progressively adding different peers for the TopicalChat Dataset. The points in the figure indicate the performance of ReFeR when specific labelled peers were used in conjunction with the area chair (GPT-4o-mini). "3 Peers" refers to the Llama, Nemo, and Gemma models being used as peers. "4 Peers" includes the same 3 peers along with the Mixtral model added as the fourth peer. Detailed results are presented in Table \ref{['tab:peer-ablation']}.
  • Figure 3: Performance analysis wrt framework scale. Pie-charts showing Peer and AC performance on evaluation and reasoning tasks. (P- Peer model, AC- area chair Model)
  • Figure 4: Prompting Schema
  • Figure 5: Illustration of ReFeR for Multimodal evaluation shown on AGIQA dataset. A similar version of ReFeR working on textual TopicalChat dataset is shown in \ref{['fig:ReFeR']}.
  • ...and 2 more figures