From Prompts to Propositions: A Logic-Based Lens on Student-LLM Interactions

TL;DR

This paper introduces Prompt2Constraints, a propositional-logic based framework that translates student prompts for programming tasks into structured logical constraints, enabling scalable, semantic analysis of prompt evolution. By applying few-shot learning to map prompts to constraints, the authors analyze 1,872 prompts from 203 CS1 students across three Python tasks, revealing patterns in successful versus unsuccessful prompting and moments where interventions can help. Key contributions include evidence of robustness in constraint extraction, insights into how prompts evolve, and a practical algorithm for measuring student progress toward solutions. The work has notable implications for scalable educational analytics and real-time hinting in AI-assisted programming tools.

Abstract

Background and Context. The increasing integration of large language models (LLMs) in computing education presents an emerging challenge in understanding how students use LLMs and craft prompts to solve computational tasks. Prior research has used both qualitative and quantitative methods to analyze prompting behavior, but these approaches lack scalability or fail to effectively capture the semantic evolution of prompts. Objective. In this paper, we investigate whether students prompts can be systematically analyzed using propositional logic constraints. We examine whether this approach can identify patterns in prompt evolution, detect struggling students, and provide insights into effective and ineffective strategies. Method. We introduce Prompt2Constraints, a novel method that translates students prompts into logical constraints. The constraints are able to represent the intent of the prompts in succinct and quantifiable ways. We used this approach to analyze a dataset of 1,872 prompts from 203 students solving introductory programming tasks. Findings. We find that while successful and unsuccessful attempts tend to use a similar number of constraints overall, when students fail, they often modify their prompts more significantly, shifting problem-solving strategies midway. We also identify points where specific interventions could be most helpful to students for refining their prompts. Implications. This work offers a new and scalable way to detect students who struggle in solving natural language programming tasks. This work could be extended to investigate more complex tasks and integrated into programming tools to provide real-time support.

Figures (8)

• Figure 1: Workflow in Prompt2Constraints
• Figure 2: Few-shot learning examples used in this paper for analysis of the denny2024prompt's dataset.
• Figure 3: Distribution of Constraints per User
• Figure 4: Changes in the constraints for each prompting task for students who finished all three tasks
• Figure 5: Changes in constraints