Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Solving Probability and Statistics Problems by Program Synthesis

Leonard Tang, Elizabeth Ke, Nikhil Singh, Nakul Verma, Iddo Drori

TL;DR

This paper tackles solving university-level probability and statistics questions by translating questions into programming tasks and solving them with Codex-generated probabilistic programs. It introduces two undergraduate datasets (MIT 18.05 and Harvard STAT110) and a concept-grounded prompting framework comprising program-task, probabilistic-simulation, and context-driven prompts to achieve correct solutions via large-scale simulation. The authors evaluate not only numerical accuracy but also the logical correctness of the generated code, reporting impressive results with a 1% tolerance and measuring the transformation effort using semantic similarity. They also provide implementation details and discuss scalability and future directions for automatic translation of questions into programming tasks.

Abstract

We solve university level probability and statistics questions by program synthesis using OpenAI's Codex, a Transformer trained on text and fine-tuned on code. We transform course problems from MIT's 18.05 Introduction to Probability and Statistics and Harvard's STAT110 Probability into programming tasks. We then execute the generated code to get a solution. Since these course questions are grounded in probability, we often aim to have Codex generate probabilistic programs that simulate a large number of probabilistic dependencies to compute its solution. Our approach requires prompt engineering to transform the question from its original form to an explicit, tractable form that results in a correct program and solution. To estimate the amount of work needed to translate an original question into its tractable form, we measure the similarity between original and transformed questions. Our work is the first to introduce a new dataset of university-level probability and statistics problems and solve these problems in a scalable fashion using the program synthesis capabilities of large language models.

Solving Probability and Statistics Problems by Program Synthesis

TL;DR

This paper tackles solving university-level probability and statistics questions by translating questions into programming tasks and solving them with Codex-generated probabilistic programs. It introduces two undergraduate datasets (MIT 18.05 and Harvard STAT110) and a concept-grounded prompting framework comprising program-task, probabilistic-simulation, and context-driven prompts to achieve correct solutions via large-scale simulation. The authors evaluate not only numerical accuracy but also the logical correctness of the generated code, reporting impressive results with a 1% tolerance and measuring the transformation effort using semantic similarity. They also provide implementation details and discuss scalability and future directions for automatic translation of questions into programming tasks.

Abstract

We solve university level probability and statistics questions by program synthesis using OpenAI's Codex, a Transformer trained on text and fine-tuned on code. We transform course problems from MIT's 18.05 Introduction to Probability and Statistics and Harvard's STAT110 Probability into programming tasks. We then execute the generated code to get a solution. Since these course questions are grounded in probability, we often aim to have Codex generate probabilistic programs that simulate a large number of probabilistic dependencies to compute its solution. Our approach requires prompt engineering to transform the question from its original form to an explicit, tractable form that results in a correct program and solution. To estimate the amount of work needed to translate an original question into its tractable form, we measure the similarity between original and transformed questions. Our work is the first to introduce a new dataset of university-level probability and statistics problems and solve these problems in a scalable fashion using the program synthesis capabilities of large language models.
Paper Structure (15 sections, 4 figures, 40 tables)

This paper contains 15 sections, 4 figures, 40 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Foundation models.
  4. Probability benchmarks.
  5. Dataset
  6. Methods
  7. Models and Evaluation
  8. Prompt Engineering
  9. Results
  10. Evaluating Program Output
  11. Achieving Perfect Results
  12. Implementation Details
  13. Conclusion
  14. MIT 18.05: Introduction to Probability and Statistics
  15. Harvard STAT110: Probability

Figures (4)

  • Figure 1: Probabilistic Simulation Program Workflow Example: (i) The original problem is translated into a programming task that asks Codex to simulate a large number of probabilistic scenarios, (ii) Codex generates such a program, and (iii) the program is executed to yield an answer.
  • Figure 2: Concept-Grounded Workflow Example: (i) The original problem is translated into a programming task that includes Bayes' Theorem within its context, (ii) Codex generates a program, and (iii) the program is executed to yield an answer.
  • Figure 3: Evaluating Codex requires more than just checking the numerical answer. It requires evaluating the logic of the program, and seeing that it actually answers the program. Though the computed answer matches the ground truth solution of 0.5, the Codex program is written with the intention of calculating the unconditional probability of having fraternal twins (assuming fraternal and identical twins have equal probabilities), and completely ignores the conditioning information in the problem.
  • Figure 4: Sentence-BERT Similarity between original questions and programming tasks by course. Each course's transformation similarities are contextualized by baseline question similarities, i.e. the average pairwise similarity between the original questions.