Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Fixing ill-formed UTF-16 strings with SIMD instructions

Robert Clausecker, Daniel Lemire

TL;DR

This work tackles ill-formed UTF-16 strings caused by surrogate mismatches and proposes a SIMD-based algorithm to correct these strings by replacing invalid code units with U+FFFD. It provides architecture-independent and architecture-specific (AVX-512 and NEON) implementations, including an in-depth discussion of data layouts, overlap strategies, and mask-based decision mechanisms. Experimental results demonstrate up to eightfold speedups over scalar methods and sustained high throughputs (e.g., 18.9 GB/s on Apple M4 and 7.5 GB/s on Ice Lake), confirming the practicality of SIMD-accelerated UTF-16 correction for high-throughput environments like web engines. The approach is incorporated into the simdutf library and used within the Google V8 JavaScript engine, enabling faster, safer text processing in major browsers.

Abstract

UTF-16 is a widely used Unicode encoding representing characters with one or two 16-bit code units. The format relies on surrogate pairs to encode characters beyond the Basic Multilingual Plane, requiring a high surrogate followed by a low surrogate. Ill-formed UTF-16 strings -- where surrogates are mismatched -- can arise from data corruption or improper encoding, posing security and reliability risks. Consequently, programming languages such as JavaScript include functions to fix ill-formed UTF-16 strings by replacing mismatched surrogates with the Unicode replacement character (U+FFFD). We propose using Single Instruction, Multiple Data (SIMD) instructions to handle multiple code units in parallel, enabling faster and more efficient execution. Our software is part of the Google JavaScript engine (V8) and thus part of several major Web browsers.

Fixing ill-formed UTF-16 strings with SIMD instructions

TL;DR

This work tackles ill-formed UTF-16 strings caused by surrogate mismatches and proposes a SIMD-based algorithm to correct these strings by replacing invalid code units with U+FFFD. It provides architecture-independent and architecture-specific (AVX-512 and NEON) implementations, including an in-depth discussion of data layouts, overlap strategies, and mask-based decision mechanisms. Experimental results demonstrate up to eightfold speedups over scalar methods and sustained high throughputs (e.g., 18.9 GB/s on Apple M4 and 7.5 GB/s on Ice Lake), confirming the practicality of SIMD-accelerated UTF-16 correction for high-throughput environments like web engines. The approach is incorporated into the simdutf library and used within the Google V8 JavaScript engine, enabling faster, safer text processing in major browsers.

Abstract

UTF-16 is a widely used Unicode encoding representing characters with one or two 16-bit code units. The format relies on surrogate pairs to encode characters beyond the Basic Multilingual Plane, requiring a high surrogate followed by a low surrogate. Ill-formed UTF-16 strings -- where surrogates are mismatched -- can arise from data corruption or improper encoding, posing security and reliability risks. Consequently, programming languages such as JavaScript include functions to fix ill-formed UTF-16 strings by replacing mismatched surrogates with the Unicode replacement character (U+FFFD). We propose using Single Instruction, Multiple Data (SIMD) instructions to handle multiple code units in parallel, enabling faster and more efficient execution. Our software is part of the Google JavaScript engine (V8) and thus part of several major Web browsers.
Paper Structure (13 sections, 2 equations, 6 figures, 5 tables)

This paper contains 13 sections, 2 equations, 6 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. SIMD algorithm
  4. Architecture-independent algorithm
  5. Example
  6. Discussion
  7. AVX-512 (x64) implementation
  8. Architecture-specific adjustments
  9. NEON (Aarch64) implementation
  10. Architecture-specific adjustments
  11. Experiments
  12. Conclusion
  13. Funding

Figures (6)

  • Figure 1: Scalar C function to replace invalid UTF-16 surrogates with the replacement character
  • Figure 2: Generic SIMD function to replace invalid UTF-16 surrogates with the replacement character
  • Figure 3: UTF-16 AVX-512 processing diagram
  • Figure 4: AVX-512 function to replace invalid UTF-16 surrogates with the replacement character within a 64-byte block
  • Figure 5: ARM NEON implementation for UTF-16 correction
  • ...and 1 more figures