Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Enhancing Augmentative and Alternative Communication with Card Prediction and Colourful Semantics

Jayr Pereira, Francisco Rodrigues, Jaylton Pereira, Cleber Zanchettin, Robson Fidalgo

TL;DR

The paper addresses improving AAC card prediction for users with complex communication needs by integrating Colourful Semantics into transformer-based models tailored to Brazilian Portuguese. It develops BERTptCS through the PrAACT pipeline, embedding Colourful Semantics roles into the vocabulary and decoder to guide card predictions. Empirical results show that BERTptCS achieves higher ACC@K and MRR and lower Entropy@K than a CS-free baseline, indicating more accurate and confident predictions and a more intuitive user experience. The work suggests practical benefits for AAC interfaces and points to future directions in UI integration and cross-language generalization of CS-enabled card prediction.

Abstract

This paper presents an approach to enhancing Augmentative and Alternative Communication (AAC) systems by integrating Colourful Semantics (CS) with transformer-based language models specifically tailored for Brazilian Portuguese. We introduce an adapted BERT model, BERTptCS, which incorporates the CS framework for improved prediction of communication cards. The primary aim is to enhance the accuracy and contextual relevance of communication card predictions, which are essential in AAC systems for individuals with complex communication needs (CCN). We compared BERTptCS with a baseline model, BERTptAAC, which lacks CS integration. Our results demonstrate that BERTptCS significantly outperforms BERTptAAC in various metrics, including top-k accuracy, Mean Reciprocal Rank (MRR), and Entropy@K. Integrating CS into the language model improves prediction accuracy and offers a more intuitive and contextual understanding of user inputs, facilitating more effective communication.

Enhancing Augmentative and Alternative Communication with Card Prediction and Colourful Semantics

TL;DR

The paper addresses improving AAC card prediction for users with complex communication needs by integrating Colourful Semantics into transformer-based models tailored to Brazilian Portuguese. It develops BERTptCS through the PrAACT pipeline, embedding Colourful Semantics roles into the vocabulary and decoder to guide card predictions. Empirical results show that BERTptCS achieves higher ACC@K and MRR and lower Entropy@K than a CS-free baseline, indicating more accurate and confident predictions and a more intuitive user experience. The work suggests practical benefits for AAC interfaces and points to future directions in UI integration and cross-language generalization of CS-enabled card prediction.

Abstract

This paper presents an approach to enhancing Augmentative and Alternative Communication (AAC) systems by integrating Colourful Semantics (CS) with transformer-based language models specifically tailored for Brazilian Portuguese. We introduce an adapted BERT model, BERTptCS, which incorporates the CS framework for improved prediction of communication cards. The primary aim is to enhance the accuracy and contextual relevance of communication card predictions, which are essential in AAC systems for individuals with complex communication needs (CCN). We compared BERTptCS with a baseline model, BERTptAAC, which lacks CS integration. Our results demonstrate that BERTptCS significantly outperforms BERTptAAC in various metrics, including top-k accuracy, Mean Reciprocal Rank (MRR), and Entropy@K. Integrating CS into the language model improves prediction accuracy and offers a more intuitive and contextual understanding of user inputs, facilitating more effective communication.
Paper Structure (12 sections, 4 figures, 2 tables)

This paper contains 12 sections, 4 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Message Authoring in AAC
  3. Color Coding Systems
  4. Card Prediction
  5. Method
  6. Corpus Annotation
  7. Model Fine-tuning
  8. Vocabulary Encoding
  9. Experiments
  10. Results
  11. Models' Predictions Analysis
  12. Conclusions

Figures (4)

  • Figure 1: Color coding systems. (a) the Fitzgerald Keys system, which color cards according to their grammatical class (e.g., noun). (b) Colourful Semantics, which color cards according to their role in a sentence.
  • Figure 2: PrAACT method adapted for adjusting BERTimbau to perform card prediction using Colourful Semantics.
  • Figure 3: Models' predictions for the beginning of sentence construction.
  • Figure 4: Models' predictions for verb completion.