Language Model Knowledge Distillation for Efficient Question Answering in Spanish

TL;DR

This work tackles the need for efficient Spanish QA models by distilling knowledge from a Spanish RoBERTa-large into a compact SpanishTinyRoBERTa. It adopts a TinyBERT-inspired distillation with a layer-mapping strategy to transfer knowledge across hierarchical transformer layers, optimized with a combined task- and layer-alignment objective. On the SQuAD-es QA task, the distilled 6-layer model achieves near-teacher performance (F1 and EM) while reducing parameters by about 6.9x and increasing inference speed by about 4.2x, demonstrating practical gains for resource-constrained deployment. The results suggest a viable path for broader compression of Spanish NLP models and motivate extending this approach to other tasks.

Abstract

Recent advances in the development of pre-trained Spanish language models has led to significant progress in many Natural Language Processing (NLP) tasks, such as question answering. However, the lack of efficient models imposes a barrier for the adoption of such models in resource-constrained environments. Therefore, smaller distilled models for the Spanish language could be proven to be highly scalable and facilitate their further adoption on a variety of tasks and scenarios. In this work, we take one step in this direction by developing SpanishTinyRoBERTa, a compressed language model based on RoBERTa for efficient question answering in Spanish. To achieve this, we employ knowledge distillation from a large model onto a lighter model that allows for a wider implementation, even in areas with limited computational resources, whilst attaining negligible performance sacrifice. Our experiments show that the dense distilled model can still preserve the performance of its larger counterpart, while significantly increasing inference speedup. This work serves as a starting point for further research and investigation of model compression efforts for Spanish language models across various NLP tasks.