Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

GEMS: Generative Expert Metric System through Iterative Prompt Priming

Ti-Chung Cheng, Carmen Badea, Christian Bird, Thomas Zimmermann, Robert DeLine, Nicole Forsgren, Denae Ford

TL;DR

A prompt-engineering framework inspired by neural activities is proposed, demonstrating that generative models can extract and summarize theories and perform basic reasoning, thereby transforming concepts into context-aware metrics to support software communities given software repository data.

Abstract

Across domains, metrics and measurements are fundamental to identifying challenges, informing decisions, and resolving conflicts. Despite the abundance of data available in this information age, not only can it be challenging for a single expert to work across multi-disciplinary data, but non-experts can also find it unintuitive to create effective measures or transform theories into context-specific metrics that are chosen appropriately. This technical report addresses this challenge by examining software communities within large software corporations, where different measures are used as proxies to locate counterparts within the organization to transfer tacit knowledge. We propose a prompt-engineering framework inspired by neural activities, demonstrating that generative models can extract and summarize theories and perform basic reasoning, thereby transforming concepts into context-aware metrics to support software communities given software repository data. While this research zoomed in on software communities, we believe the framework's applicability extends across various fields, showcasing expert-theory-inspired metrics that aid in triaging complex challenges.

GEMS: Generative Expert Metric System through Iterative Prompt Priming

TL;DR

A prompt-engineering framework inspired by neural activities is proposed, demonstrating that generative models can extract and summarize theories and perform basic reasoning, thereby transforming concepts into context-aware metrics to support software communities given software repository data.

Abstract

Across domains, metrics and measurements are fundamental to identifying challenges, informing decisions, and resolving conflicts. Despite the abundance of data available in this information age, not only can it be challenging for a single expert to work across multi-disciplinary data, but non-experts can also find it unintuitive to create effective measures or transform theories into context-specific metrics that are chosen appropriately. This technical report addresses this challenge by examining software communities within large software corporations, where different measures are used as proxies to locate counterparts within the organization to transfer tacit knowledge. We propose a prompt-engineering framework inspired by neural activities, demonstrating that generative models can extract and summarize theories and perform basic reasoning, thereby transforming concepts into context-aware metrics to support software communities given software repository data. While this research zoomed in on software communities, we believe the framework's applicability extends across various fields, showcasing expert-theory-inspired metrics that aid in triaging complex challenges.
Paper Structure (28 sections, 3 figures, 33 tables)

This paper contains 28 sections, 3 figures, 33 tables.

Table of Contents

  1. Introduction
  2. Problem Definition
  3. Approach and System Design
  4. System Overview
  5. Implementation Details
  6. Expert Agent and Iterative Prompt Priming
  7. Stage 1
  8. Stage 2
  9. Stage 3
  10. Multiple Expert Agents
  11. Judges
  12. Methods
  13. Goals
  14. Evaluation Process
  15. Results
  16. ...and 13 more sections

Figures (3)

  • Figure 1: System Overview. The system begins with the Lead Orchestrator defining a naive metric to identify Team $T_x$. For simplicity, we omit the details of how this metric was generated. The Lead Orchestrator subsequently forms a panel of experts augmented through perspectives relevant to the given goal. The experts generate teams' results that pass through to a judge agent to form a decision. A final result is then aggregated.
  • Figure 2: Expert Agents. The agents are 'primed' by the given expert's research contributions to the field. This model is inspired by the generated knowledge prompting technique proposed by liuGeneratedKnowledgePrompting2022.
  • Figure 3: System Architecture Diagram of GEMS