A Justice Lens on Fairness and Ethics Courses in Computing Education: LLM-Assisted Multi-Perspective and Thematic Evaluation

TL;DR

This paper tackles the challenge of assessing justice-oriented content in AI/ML and related computing courses by combining a structured, 20-item justice rubric with LLM-based multi-perspective evaluations across 24 syllabi from diverse U.S. institutions. It demonstrates that evaluator role strongly shapes judgments and reveals gap areas, such as explicit learning objectives and instructor status, while highlighting consistent practices like neutral language and bias-mitigation discussions. The study provides a scalable methodology that surfaces cross-institutional trends in representation, ethics, and social impact, with graduate courses more often embedding justice-oriented content than undergraduate ones. The results offer actionable guidance for curriculum designers to enhance inclusion, transparency, and critical engagement in fairness, ethics, and justice topics across AI/ML education.

Abstract

Course syllabi set the tone and expectations for courses, shaping the learning experience for both students and instructors. In computing courses, especially those addressing fairness and ethics in artificial intelligence (AI), machine learning (ML), and algorithmic design, it is imperative that we understand how approaches to navigating barriers to fair outcomes are being addressed.These expectations should be inclusive, transparent, and grounded in promoting critical thinking. Syllabus analysis offers a way to evaluate the coverage, depth, practices, and expectations within a course. Manual syllabus evaluation, however, is time-consuming and prone to inconsistency. To address this, we developed a justice-oriented scoring rubric and asked a large language model (LLM) to review syllabi through a multi-perspective role simulation. Using this rubric, we evaluated 24 syllabi from four perspectives: instructor, departmental chair, institutional reviewer, and external evaluator. We also prompted the LLM to identify thematic trends across the courses. Findings show that multiperspective evaluation aids us in noting nuanced, role-specific priorities, leveraging them to fill hidden gaps in curricula design of AI/ML and related computing courses focused on fairness and ethics. These insights offer concrete directions for improving the design and delivery of fairness, ethics, and justice content in such courses.

Figures (9)

• Figure 1: LLM Thematic Prompt: Quantitative Findings
• Figure 2: LLM Thematic Prompt: Quantitative Findings
• Figure 3: Average scores for justice-linked criteria across evaluators.
• Figure 4: Mean syllabus scores by criterion (0–1 scale) across evaluators and courses.
• Figure 5: Graduate vs. Undergraduate syllabi mean scores (0–3) per criterion, across evaluators.