Braille-to-Speech Generator: Audio Generation Based on Joint Fine-Tuning of CLIP and Fastspeech2

TL;DR

The paper tackles Braille-to-speech for Chinese visually impaired users under data-scarce conditions by proposing a two-stage CLIP-KNN-Fastspeech2 system. It pre-trains Chinese CLIP on MUGE and Fastspeech2 on Baker, then fine-tunes on a Braille-pinyin dataset BIT, using a CLIP-based image-text stage with KNN retrieval and a Fastspeech2-based text-to-audio stage with a variance adaptor and HiFi-GAN vocoder; the overall objective combines $Loss_{i-t}$ and $Loss_{t-a}$ as $Loss_{total}=\lambda_1 Loss_{i-t}+\lambda_2 Loss_{t-a}$ with $\lambda_1=\lambda_2=0.5$. Across Flickr8k, VGGSound, ImageHear, and the self-built BIT dataset, the model achieves favorable objective and perceptual metrics (e.g., BLEU/METEOR/CS/FAD and WER/MOS), demonstrating data-efficient and robust cross-modal translation in Chinese. The approach advances Braille-to-audio generation for visually impaired users, offering practical benefits in low-resource language contexts and informing cross-modal design with a two-stage image-text-audio framework.

Abstract

An increasing number of Chinese people are troubled by different degrees of visual impairment, which has made the modal conversion between a single image or video frame in the visual field and the audio expressing the same information a research hotspot. Deep learning technologies such as OCR+Vocoder and Im2Wav enable English audio synthesis or image-to-sound matching in a self-supervised manner. However, the audio data used for training is limited and English is not universal for visually impaired people with different educational levels. Therefore, for the sake of solving the problems of data volume and language applicability to improve the reading efficiency of visually impaired people, a set of image-to-speech framework CLIP-KNN-Fastspeech2 based on the Chinese context was constructed. The framework integrates multiple basic models and adopts the strategy of independent pre-training and joint fine-tuning. First, the Chinese CLIP and Fastspeech2 text-to-speech models were pre-trained on two public datasets, MUGE and Baker, respectively, and their convergence was verified. Subsequently, joint fine-tuning was performed using a self-built Braille image dataset. Experimental results on multiple public datasets such as VGGSound, Flickr8k, ImageHear, and the self-built Braille dataset BIT-DP show that the model has improved objective indicators such as BLEU4,FAD(Fréchet Audio Distance), WER(Word Error Ratio), and even inference speed. This verifies that the constructed model still has the ability to synthesize high-quality speech under limited data, and also proves the effectiveness of the joint training strategy that integrates multiple basic models.

Figures (12)

• Figure 1: Braille "Hao3" Example ( The first square in red represents the initial consonant "h", the second square in blue in the middle represents the compound vowel "ao", and the third square in green indicates the third tone, i.e., the rising tone.).
• Figure 2: Architecture of the Two-Stage Image-to-Speech Model (Left: In the I2T stage, the pre-trained Chinese CLIP model learns features of image-text pairs and retrieves text through the KNN network; in the T2A stage, the pre-trained Fastspeech2 vocoder demonstrates its audio-text conversion capability. Right: Fine-tuning on the braille dataset is conducted using the trained CLIP-KNN-Fastspeech2 model.).
• Figure 3: Types, Cleaning, and Enhancement Methods of the BIT Dataset. Left Figure: Illustrates the three categories of Braille correspondence contained in the dataset, where Type 1 corresponds to the numeral "24", Type 2 corresponds to the Pinyin "hao4", and Type 3 corresponds to the punctuation mark "!". Middle Figure: Presents two cases of "dirty data", namely, Excessive cropping and Incomplete cropping.Right Figure: Demonstrates two methods for data augmentation, namely, Flipping the image by 180 degrees and Adding background colors.).
• Figure 4: The performance metrics of CLIP on the MUGE dataset are presented as follows: (a) The trend of loss and accuracy during training is depicted in the chart. (b) The chart illustrates the recall rates and average recall rates of CLIP models at $k$=1 and $k$=5.
• Figure 5: The performance on the Baker dataset is documented as follows:(a) The trend of loss variation during model training is recorded, with significant fluctuations observed in the first 5 epochs followed by minimal changes in subsequent epochs.(b) The distribution of scores from 50 participants, who rated the naturalness of the synthesized speech generated by Fastspeech2, is presented across five rating levels.