Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Improving Examples in Web API Specifications using Iterated-Calls In-Context Learning

Kush Jain, Kiran Kate, Jason Tsay, Claire Le Goues, Martin Hirzel

TL;DR

The paper tackles the scarcity of human-written examples in OpenAPI specifications by introducing IcIcl, an iterated-context, retrieval-driven prompting framework for generating API parameter examples. IcIcl builds a parameter bank from mined specs, retrieves relevant context via BM25, and uses multiple prompt contexts with greedy and diverse sampling, followed by postprocessing to ensure type-correct, semantically aligned examples. Intrinsic evaluation shows high type correctness and increased diversity, while extrinsic tests demonstrate substantial gains in fuzz testing coverage, improved dialog task performance, and enhanced human API understanding. The approach offers a scalable, design-robust method to enrich API specifications, with clear benefits for testing, chatbots, and developer comprehension across evolving APIs.

Abstract

Examples in web API specifications can be essential for API testing, API understanding, and even building chat-bots for APIs. Unfortunately, most API specifications lack human-written examples. This paper introduces a novel technique for generating examples for web API specifications. We start from in-context learning (ICL): given an API parameter, use a prompt context containing a few examples from other similar API parameters to call a model to generate new examples. However, while ICL tends to generate correct examples, those lack diversity, which is also important for most downstream tasks. Therefore, we extend the technique to iterated-calls ICL (ICICL): use a few different prompt contexts, each containing a few examples,to iteratively call the model with each context. Our intrinsic evaluation demonstrates that ICICL improves both correctness and diversity of generated examples. More importantly, our extrinsic evaluation demonstrates that those generated examples significantly improve the performance of downstream tasks of testing, understanding, and chat-bots for APIs.

Improving Examples in Web API Specifications using Iterated-Calls In-Context Learning

TL;DR

The paper tackles the scarcity of human-written examples in OpenAPI specifications by introducing IcIcl, an iterated-context, retrieval-driven prompting framework for generating API parameter examples. IcIcl builds a parameter bank from mined specs, retrieves relevant context via BM25, and uses multiple prompt contexts with greedy and diverse sampling, followed by postprocessing to ensure type-correct, semantically aligned examples. Intrinsic evaluation shows high type correctness and increased diversity, while extrinsic tests demonstrate substantial gains in fuzz testing coverage, improved dialog task performance, and enhanced human API understanding. The approach offers a scalable, design-robust method to enrich API specifications, with clear benefits for testing, chatbots, and developer comprehension across evolving APIs.

Abstract

Examples in web API specifications can be essential for API testing, API understanding, and even building chat-bots for APIs. Unfortunately, most API specifications lack human-written examples. This paper introduces a novel technique for generating examples for web API specifications. We start from in-context learning (ICL): given an API parameter, use a prompt context containing a few examples from other similar API parameters to call a model to generate new examples. However, while ICL tends to generate correct examples, those lack diversity, which is also important for most downstream tasks. Therefore, we extend the technique to iterated-calls ICL (ICICL): use a few different prompt contexts, each containing a few examples,to iteratively call the model with each context. Our intrinsic evaluation demonstrates that ICICL improves both correctness and diversity of generated examples. More importantly, our extrinsic evaluation demonstrates that those generated examples significantly improve the performance of downstream tasks of testing, understanding, and chat-bots for APIs.

Paper Structure

This paper contains 33 sections, 1 figure, 5 tables.

Table of Contents

  1. Introduction
  2. Motivating Example and Overview
  3. IcIcl
  4. Offline: Mining Examples, Model Selection
  5. Retrieving Relevant Parameters
  6. Prompt Context Generation
  7. Postprocessing
  8. Intrinsic Evaluation
  9. Experimental Setup
  10. Dataset
  11. Approach
  12. Metrics
  13. Intrinsic Evaluation Results
  14. Extrinsic Evaluation
  15. Software Testing
  16. ...and 18 more sections

Figures (1)

  • Figure 1: Overview and running example of our approach. Circled numbers correspond to different steps in our approach.