Human in the Latent Loop (HILL): Interactively Guiding Model Training Through Human Intuition

TL;DR

This paper Tackles latent-space opacity by introducing Human In the Latent Loop (HILL), an interactive framework that lets humans reshape latent representations during training using a knowledge-distillation-inspired mechanism where human input acts as a teacher. Latent-space guidance is operationalized via a loss function that combines $L_{CE}$ with a human-focused term $L_{human}$ and a scale-regularization term, balanced by $\alpha$ (set to 0.5) and $\lambda$, while human adjustments are expressed through center movement, spread, and separation across $K$ comparisons. The authors validate HILL through a user study on CIFAR-10 and PAMAP2, showing improvements in validation accuracy (e.g., CIFAR-10 to 87.3% and PAMAP2 to 75%), faster convergence, and rich qualitative insights into strategies and interactions, alongside usability and workload assessments. They also discuss risks of human biases and the need for transparent feedback mechanisms, arguing that HILL enables a productive human-AI symbiosis that can uncover meaningful decision boundaries beyond standard optimization.

Abstract

Latent space representations are critical for understanding and improving the behavior of machine learning models, yet they often remain obscure and intricate. Understanding and exploring the latent space has the potential to contribute valuable human intuition and expertise about respective domains. In this work, we present HILL, an interactive framework allowing users to incorporate human intuition into the model training by interactively reshaping latent space representations. The modifications are infused into the model training loop via a novel approach inspired by knowledge distillation, treating the user's modifications as a teacher to guide the model in reshaping its intrinsic latent representation. The process allows the model to converge more effectively and overcome inefficiencies, as well as provide beneficial insights to the user. We evaluated HILL in a user study tasking participants to train an optimal model, closely observing the employed strategies. The results demonstrated that human-guided latent space modifications enhance model performance while maintaining generalization, yet also revealing the risks of including user biases. Our work introduces a novel human-AI interaction paradigm that infuses human intuition into model training and critically examines the impact of human intervention on training strategies and potential biases.

Figures (7)

• Figure 1: Comparison of traditional iterative training, commonly resulting in a unsatisfying black-box training, with the approach of adding the Human In the Latent Loop (HILL) to insert the knowledge of human experts towards explainability, optimization and efficiency.
• Figure 2: Utilizing a weighted loss function through $\alpha$ to balance the classic cross-entropy with the human as the teacher. The human input is gathered from center movement, spread of classes and separation of clusters, which are finally normalized over the total number of pairwise comparisons $K$. Additionally the scale of the model is added in the global loss to regulate the loss function further.
• Figure 3: The user interface of HILL; sidebar on the left to control the model training through the tool; main window obtaining the interactive scatter plots with relevant controls; a legend on the right to reference class labels with colors.
• Figure 4: The extracted latent space of an exemplary training iteration. The model's latent space is initially unstructured and the model struggles to separate the classes properly, whereas after the insertion of human guidance, the model adapts its internal representation and classes can be distinguished with greater accuracy.
• Figure 5: The model performance evaluation of HILL compared to the traditional training as baseline; Dark grey represents the pretraining phase; red dashed lines human interaction points; the light grey area represents the range of best (green) and worst (orange) participant utilizing HILL during the user study.