Comparison of Autoencoders for tokenization of ASL datasets

TL;DR

The study evaluates three encoder–decoder architectures (Feedforward Autoencoder, Convolutional Autoencoder, Diffusion Autoencoder) on an ASL image dataset containing 87,000 images across 29 classes. Through objective reconstruction metrics and subjective human MOS assessments, the Diffusion Autoencoder consistently delivers the highest fidelity reconstructions, outperforming the convolutional and feedforward baselines. Convolutional models effectively capture spatial structure but lack the robustness provided by probabilistic diffusion, while the feedforward approach struggles with image data due to missing spatial feature extraction. The results highlight diffusion‑based tokenization and reconstruction as a promising strategy for robust multimodal AI applications, including sign language recognition and generation.

Abstract

Generative AI, powered by large language models (LLMs), has revolutionized applications across text, audio, images, and video. This study focuses on developing and evaluating encoder-decoder architectures for the American Sign Language (ASL) image dataset, consisting of 87,000 images across 29 hand sign classes. Three approaches were compared: Feedforward Autoencoders, Convolutional Autoencoders, and Diffusion Autoencoders. The Diffusion Autoencoder outperformed the others, achieving the lowest mean squared error (MSE) and highest Mean Opinion Score (MOS) due to its probabilistic noise modeling and iterative denoising capabilities. The Convolutional Autoencoder demonstrated effective spatial feature extraction but lacked the robustness of the diffusion process, while the Feedforward Autoencoder served as a baseline with limitations in handling complex image data. Objective and subjective evaluations confirmed the superiority of the Diffusion Autoencoder for high-fidelity image reconstruction, emphasizing its potential in multimodal AI applications such as sign language recognition and generation. This work provides critical insights into designing robust encoder-decoder systems to advance multimodal AI capabilities.

Figures (4)

• Figure 1: Visual Depiction of Feedforward Autoencoder architecture
• Figure 2: Visual Depiction of Convolutional Autoencoder architecture
• Figure 3: Visual Depiction of noise schedule and de-noising model
• Figure 4: Sample Reconstructions of Each Architecture and their Average Mean Opinion Scores (MOS)