Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Logical Modelling in CS Education: Bridging the Natural Language Gap

Tristan Kneisel, Fabian Vehlken, Thomas Zeume

TL;DR

This work tackles the problem of bridging the natural language gap in vocabulary design for formal modelling in CS education by proposing a two-phase framework that maps natural-language symbol descriptions to a formal vocabulary and then verifies the resulting solution space. It implements vocabulary design tasks for propositional and first-order logic within the Iltis educational system and evaluates them on German-language assignments using a dataset of over $25{,}000$ data points. The evaluation demonstrates that small fine-tuned NLP models achieve high binary classification accuracy (often >90%) and competitive multi-class performance, with first-order vocabularies generally performing better, and discusses the trade-offs with larger LLMs and data-sovereignty considerations. The findings support the viability of NLP-based, grammar-generated feedback for vocabulary design in CS education and point to future work extending the approach to other formal languages and natural-language grounding tasks.

Abstract

An important learning objective for computer science students is to learn how to formalize descriptions of real world scenarios in order to subsequently solve real world challenges using methods and algorithms from formal foundations of computer science. Two key steps when formalizing with logical formalisms are to (a) choose a suitable vocabulary, that is, e.g., which propositional variables or first-order symbols to use, and with which intended meaning, and then to (b) construct actual formal descriptions, i.e. logical formulas over the chosen vocabulary. While (b) is addressed by several educational support systems for formal foundations of computer science, (a) is so far not addressed at all -- likely because it involves specifying the intended meaning of symbols in natural language. We propose a conceptual framework for educational tasks where students choose a vocabulary, including an enriched language for describing solution spaces as well as an NLP-approach for checking student attempts and providing feedback. We implement educational tasks for designing propositional and first-order vocabularies within the Iltis educational system, and report on experiments with data from introductory logic courses for computer science students with > 25.000 data points.

Logical Modelling in CS Education: Bridging the Natural Language Gap

TL;DR

This work tackles the problem of bridging the natural language gap in vocabulary design for formal modelling in CS education by proposing a two-phase framework that maps natural-language symbol descriptions to a formal vocabulary and then verifies the resulting solution space. It implements vocabulary design tasks for propositional and first-order logic within the Iltis educational system and evaluates them on German-language assignments using a dataset of over data points. The evaluation demonstrates that small fine-tuned NLP models achieve high binary classification accuracy (often >90%) and competitive multi-class performance, with first-order vocabularies generally performing better, and discusses the trade-offs with larger LLMs and data-sovereignty considerations. The findings support the viability of NLP-based, grammar-generated feedback for vocabulary design in CS education and point to future work extending the approach to other formal languages and natural-language grounding tasks.

Abstract

An important learning objective for computer science students is to learn how to formalize descriptions of real world scenarios in order to subsequently solve real world challenges using methods and algorithms from formal foundations of computer science. Two key steps when formalizing with logical formalisms are to (a) choose a suitable vocabulary, that is, e.g., which propositional variables or first-order symbols to use, and with which intended meaning, and then to (b) construct actual formal descriptions, i.e. logical formulas over the chosen vocabulary. While (b) is addressed by several educational support systems for formal foundations of computer science, (a) is so far not addressed at all -- likely because it involves specifying the intended meaning of symbols in natural language. We propose a conceptual framework for educational tasks where students choose a vocabulary, including an enriched language for describing solution spaces as well as an NLP-approach for checking student attempts and providing feedback. We implement educational tasks for designing propositional and first-order vocabularies within the Iltis educational system, and report on experiments with data from introductory logic courses for computer science students with > 25.000 data points.
Paper Structure (4 sections, 2 figures)

This paper contains 4 sections, 2 figures.

Table of Contents

  1. Introduction and Motivation
  2. Related work.
  3. Outline.
  4. A Framework for Vocabulary Design Tasks

Figures (2)

  • Figure 1: Typical (simplified) assignments from an introductory course on logic which ask students to formally model real-world scenarios by designing a suitable vocabulary, constructing and manipulating formulas, and inferring conclusions.
  • Figure 2: An assignment in the Iltis educational support system where students (a) design a first-order vocabulary (left), and then (b) write first-order formulas using their vocabulary (right).