Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Solving Linear Algebra by Program Synthesis

Iddo Drori, Nakul Verma

TL;DR

The paper presents a pipeline that converts linear algebra questions into program synthesis tasks solved by Codex, achieving perfect accuracy on two university-level datasets. It demonstrates an interactive workflow where prompts are refined and, when needed, questions are augmented to reveal missing components like projections, enabling both numerical solutions and plotting outputs. Key contributions include perfect dataset performance, evidence of close text-prompt similarity to original questions, and the ability to generate new course questions automatically. The work suggests a scalable direction for automated problem solving and content generation across STEM education, with notable implications for grading and curriculum development.

Abstract

We solve MIT's Linear Algebra 18.06 course and Columbia University's Computational Linear Algebra COMS3251 courses with perfect accuracy by interactive program synthesis. This surprisingly strong result is achieved by turning the course questions into programming tasks and then running the programs to produce the correct answers. We use OpenAI Codex with zero-shot learning, without providing any examples in the prompts, to synthesize code from questions. We quantify the difference between the original question text and the transformed question text that yields a correct answer. Since all COMS3251 questions are not available online the model is not overfitting. We go beyond just generating code for questions with numerical answers by interactively generating code that also results visually pleasing plots as output. Finally, we automatically generate new questions given a few sample questions which may be used as new course content. This work is a significant step forward in solving quantitative math problems and opens the door for solving many university level STEM courses by machine.

Solving Linear Algebra by Program Synthesis

TL;DR

The paper presents a pipeline that converts linear algebra questions into program synthesis tasks solved by Codex, achieving perfect accuracy on two university-level datasets. It demonstrates an interactive workflow where prompts are refined and, when needed, questions are augmented to reveal missing components like projections, enabling both numerical solutions and plotting outputs. Key contributions include perfect dataset performance, evidence of close text-prompt similarity to original questions, and the ability to generate new course questions automatically. The work suggests a scalable direction for automated problem solving and content generation across STEM education, with notable implications for grading and curriculum development.

Abstract

We solve MIT's Linear Algebra 18.06 course and Columbia University's Computational Linear Algebra COMS3251 courses with perfect accuracy by interactive program synthesis. This surprisingly strong result is achieved by turning the course questions into programming tasks and then running the programs to produce the correct answers. We use OpenAI Codex with zero-shot learning, without providing any examples in the prompts, to synthesize code from questions. We quantify the difference between the original question text and the transformed question text that yields a correct answer. Since all COMS3251 questions are not available online the model is not overfitting. We go beyond just generating code for questions with numerical answers by interactively generating code that also results visually pleasing plots as output. Finally, we automatically generate new questions given a few sample questions which may be used as new course content. This work is a significant step forward in solving quantitative math problems and opens the door for solving many university level STEM courses by machine.
Paper Structure (12 sections, 3 figures, 61 tables)

This paper contains 12 sections, 3 figures, 61 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methods
  4. Datasets
  5. Interactive Workflow
  6. Results
  7. Performance Evaluation
  8. Generating New Questions
  9. Limitations
  10. Conclusion
  11. Computational Linear Algebra: Columbia University COMS3251
  12. Introduction to Linear Algebra: MIT 18.06

Figures (3)

  • Figure 1: Workflow for solving Linear Algebra questions: (i) Given a question in text, the example shown is Q1 in Ch. 7.3 of strang_lin_alg, (ii) we run the question through Codex to generate a program, (iii) we execute the program to generate the solution. We transform the question repeating steps (ii) and (iii) until we get it correct.
  • Figure 2: Interactive workflow: (A) We begin with the original question. Codex generates a program which is executed. The result is missing the projection. (B) We transform the question and Codex generates a program again to get the correct answer, though the zero projection vector does not appear on the plot, (C) An additional task to plot the projection vector with a marker so that it is visible results in Codex generating modified code which is executed to yield a correct answer and visually pleasing result.
  • Figure 3: High similarity between the original questions and programming prompts for both COMS3251 and MIT 18.06. Baseline mean pairwise similarity among original questions for both courses shown as a solid horizontal line.