Supporting Co-Adaptive Machine Teaching through Human Concept Learning and Cognitive Theories

TL;DR

MOCHA introduces a co-adaptive teaching framework that blends Variation Theory-based counterfactual generation with Structural Alignment Theory-based rendering to improve human–AI alignment in subjective labeling tasks. It leverages neuro-symbolic pattern rules to generate counterfactuals that challenge existing label boundaries and renders them in a way that highlights alignable differences, fostering sensemaking and reflection. A lab study with 18 participants demonstrates improved annotation efficiency and downstream learning gains, supporting the viability of bi-directional human–AI adaptation in interactive ML pipelines. The work contributes VT-guided counterfactual generation, SAT-based rendering, and empirical evidence for cognition-informed design in co-adaptive machine teaching.

Abstract

An important challenge in interactive machine learning, particularly in subjective or ambiguous domains, is fostering bi-directional alignment between humans and models. Users teach models their concept definition through data labeling, while refining their own understandings throughout the process. To facilitate this, we introduce MOCHA, an interactive machine learning tool informed by two theories of human concept learning and cognition. First, it utilizes a neuro-symbolic pipeline to support Variation Theory-based counterfactual data generation. By asking users to annotate counterexamples that are syntactically and semantically similar to already-annotated data but predicted to have different labels, the system can learn more effectively while helping users understand the model and reflect on their own label definitions. Second, MOCHA uses Structural Alignment Theory to present groups of counterexamples, helping users comprehend alignable differences between data items and annotate them in batch. We validated MOCHA's effectiveness and usability through a lab study with 18 participants.

Figures (6)

• Figure 1: Mocha uses neuro-symbolic pattern rules (A) to generate counterfactuals. For each example labeled by the rules (B), Mocha generates counterfactual examples that match the original patterns (D) but belong to a other than the original label (C). The generated counterfactuals are then rendered below the original example with highlighting of what has changed and what has stayed the same (E) for each alternative label.
• Figure 2: Mocha facilitates analogical reasoning using visual cues. For each model-labeled example (A) and its corresponding learned neuro-symbolic rule (B), counterfactual examples are generated for a set of target labels (C). Phrases consistent with the original example are displayed in gray text (E), while varying phrases are displayed in black text for visual salience (F). Additionally, the text of the counterfactual that would mislead the neuro-symbolic model into classifying it as the original label (by matching the original label's rule) are highlighted in the theme color (D), helping users understand how their annotations contribute to model updates.
• Figure 3: Comparison of average annotation times under the three conditions. The table shows that while VT-based counterfactuals increase the time for the first annotation, SAT-based rendering significantly reduces the time for annotating each data point in the batch.
• Figure 4: Participants' response to post study questionnaire comparing the three conditions.
• Figure 5: Our study finds a bilateral relationship between Variation Theory and Structural Alignment Theory. The Variation Theory-based counterfactual generation method enabled the rendering of structurally alignable differences. In turn, the rendering supported the users' sensemaking of the variation in the generated counterfactuals