Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Semantic Interactivity: leveraging NLP to enable a shared interaction approach for joint activities

Olaf V. Adan, Dimitra Dritsa, Steven Houben

TL;DR

This paper addresses the mismatch between collocated collaboration needs and systems designed around individual tasks by introducing semantic interactivity, a shared interaction mechanism realized through two interleaved loops: an implicit NLP-driven content stream and an explicit tangible-annotation workflow. Implemented in CollEagle, a tangible tabletop system, the approach enables continuous transcription, extraction, and curation of discussion content, supporting joint activity without heavy configuration. The study provides empirical insights into how semantic interactivity mediates collaboration, including content persistence, turn-taking, and shared sense-making, and offers design implications for reducing configuration work while enhancing group engagement. The work demonstrates the potential to transform joint activities by merging NLP-driven automation with direct manipulation on shared surfaces, with practical implications for future collaborative interfaces across devices and spaces.

Abstract

Collocated collaboration, where individuals work together in the same physical space and time, remains a cornerstone of effective teamwork. However, most collaborative systems are designed to support individual tasks rather than joint activities; they enable interactions for users to complete tasks rather than interactivity to engage in shared experiences. In this work, we introduce an NLP-driven mechanism that enables semantic interactivity through a shared interaction mechanism. This mechanism was developed as part of CollEagle, an interactive tabletop system that supports shared externalisation practices by offering a low-effort way for users to create, curate, organise, and structure information to capture the essence of collaborative discussions. Our preliminary study highlights the potential for semantic interactivity to mediate group interactions, suggesting that the interaction approach paves the way for designing novel collaborative interfaces. We contribute our implementation and offer insights for future research to enable semantic interactivity in systems that support joint activities.

Semantic Interactivity: leveraging NLP to enable a shared interaction approach for joint activities

TL;DR

This paper addresses the mismatch between collocated collaboration needs and systems designed around individual tasks by introducing semantic interactivity, a shared interaction mechanism realized through two interleaved loops: an implicit NLP-driven content stream and an explicit tangible-annotation workflow. Implemented in CollEagle, a tangible tabletop system, the approach enables continuous transcription, extraction, and curation of discussion content, supporting joint activity without heavy configuration. The study provides empirical insights into how semantic interactivity mediates collaboration, including content persistence, turn-taking, and shared sense-making, and offers design implications for reducing configuration work while enhancing group engagement. The work demonstrates the potential to transform joint activities by merging NLP-driven automation with direct manipulation on shared surfaces, with practical implications for future collaborative interfaces across devices and spaces.

Abstract

Collocated collaboration, where individuals work together in the same physical space and time, remains a cornerstone of effective teamwork. However, most collaborative systems are designed to support individual tasks rather than joint activities; they enable interactions for users to complete tasks rather than interactivity to engage in shared experiences. In this work, we introduce an NLP-driven mechanism that enables semantic interactivity through a shared interaction mechanism. This mechanism was developed as part of CollEagle, an interactive tabletop system that supports shared externalisation practices by offering a low-effort way for users to create, curate, organise, and structure information to capture the essence of collaborative discussions. Our preliminary study highlights the potential for semantic interactivity to mediate group interactions, suggesting that the interaction approach paves the way for designing novel collaborative interfaces. We contribute our implementation and offer insights for future research to enable semantic interactivity in systems that support joint activities.

Paper Structure

This paper contains 48 sections, 13 figures.

Table of Contents

  1. Introduction
  2. Background
  3. Collocated Collaboration
  4. Collaborative Computing
  5. Related Work
  6. Externalisation techniques
  7. Intent in Externalisation
  8. Automating Externalisation
  9. Summary
  10. Designing Collocated Interfaces
  11. Guiding principles
  12. CollEagle
  13. Interface Design
  14. Interaction Techniques
  15. Leveraging Modalities
  16. ...and 33 more sections

Figures (13)

  • Figure 1: The CollEagle system supports the use of tangibles (orange) to interact with automatically produced post-its.
  • Figure 2: Digital components to tangible annotations for each interaction technique supported by CollEagle
  • Figure 3: CollEagle System Prototype: (1) Azure Kinect Depth Sensor (2) Laptop (3) LCD-display (4) Tangibles
  • Figure 4: Software architecture.
  • Figure 5: Schematic overview of interface design and content extraction, detailing how an (1) identified utterance is filtered to identify content, (2) stream paths alternate, and (3) content is displayed.
  • ...and 8 more figures