LLM-ForcedAligner: A Non-Autoregressive and Accurate LLM-Based Forced Aligner for Multilingual and Long-Form Speech

TL;DR

The paper tackles the problem of forced alignment across multilingual and long-form speech by reframing FA as a slot-filling task with discrete timestamp slots inserted into transcripts. It combines an Audio Transformer encoder with a multilingual LLM and a timestamp head to predict slot indices in a non-autoregressive, globally coherent manner, using causal training and dynamic slot insertion. Training on MFA-generated pseudo-labels enables strong multilingual performance and robustness to long-form data, achieving substantial reductions in accumulated averaging shift compared with prior FA methods, while incurring modest increases in inference time. The approach demonstrates practical benefits for multilingual FA pipelines and long-form analysis, with future work aimed at expanding language coverage and real-world content types.

Abstract

Forced alignment (FA) predicts start and end timestamps for words or characters in speech, but existing methods are language-specific and prone to cumulative temporal shifts. The multilingual speech understanding and long-sequence processing abilities of speech large language models (SLLMs) make them promising for FA in multilingual, crosslingual, and long-form speech settings. However, directly applying the next-token prediction paradigm of SLLMs to FA results in hallucinations and slow inference. To bridge the gap, we propose LLM-ForcedAligner, reformulating FA as a slot-filling paradigm: timestamps are treated as discrete indices, and special timestamp tokens are inserted as slots into the transcript. Conditioned on the speech embeddings and the transcript with slots, the SLLM directly predicts the time indices at slots. During training, causal attention masking with non-shifted input and label sequences allows each slot to predict its own timestamp index based on itself and preceding context, with loss computed only at slot positions. Dynamic slot insertion enables FA at arbitrary positions. Moreover, non-autoregressive inference is supported, avoiding hallucinations and improving speed. Experiments across multilingual, crosslingual, and long-form speech scenarios show that LLM-ForcedAligner achieves a 69%~78% relative reduction in accumulated averaging shift compared with prior methods. The checkpoint and inference code will be released later.

Figures (4)

• Figure 1: Highlights of LLM-ForceAligner.
• Figure 2: Overview of LLM-ForcedAligner. Left: During training, we replace the word-level or character-level start and end timestamps in the transcript with a special token [time] to serve as slots, and use dynamic slot insertion to randomly determine which word or character slots to ignore. The speech embeddings produced by the AuT encoder are then concatenated and fed into the LLM for training with causal attention masking. When computing the cross-entropy loss, the output IDs and the labels are non-shifted, and the loss is computed only at the positions of the retained slots. Right: During inference, users can insert the special token [time] at arbitrary positions in the transcript and rapidly obtain the corresponding start and end timestamps through non-autoregressive decoding.
• Figure 3: AAS (ms) of LLM-ForcedAligner on the human-labeled test datasets for different dynamic slot insertion percentages.
• Figure 4: AAS (ms) on the MFA-labeled and human-labeled test datasets for LLM-ForcedAligner with different parameter settings.