Structured Document Translation via Format Reinforcement Learning

TL;DR

<FormatRL> reframes structured document translation as a format-aware RL problem, introducing TreeSim and Node-chrF rewards and optimizing via Group Relative Policy Optimization to improve structural fidelity while preserving translation quality. It also proposes StrucAUC, a fine-grained document-level metric that blends node-level translation with structural alignment under controlled edit tolerances. Empirical results on the SAP software documentation dataset show meaningful gains in XML-Match and StrucAUC, with competitive translation quality, and analyses illuminate reward-metric alignment and synthetic data effects. The work demonstrates the practical viability of end-to-end, structure-aware document translation and provides a rigorous evaluation framework for future research in this area.

Abstract

Recent works on structured text translation remain limited to the sentence level, as they struggle to effectively handle the complex document-level XML or HTML structures. To address this, we propose \textbf{Format Reinforcement Learning (FormatRL)}, which employs Group Relative Policy Optimization on top of a supervised fine-tuning model to directly optimize novel structure-aware rewards: 1) TreeSim, which measures structural similarity between predicted and reference XML trees and 2) Node-chrF, which measures translation quality at the level of XML nodes. Additionally, we apply StrucAUC, a fine-grained metric distinguishing between minor errors and major structural failures. Experiments on the SAP software-documentation benchmark demonstrate improvements across six metrics and an analysis further shows how different reward functions contribute to improvements in both structural and translation quality.

Figures (12)

• Figure 1: A structured document translation example (English$\rightarrow$Japanese), with markup highlighted in color. The lower part shows the translation of XML tree structure with node $\phi(\mathord{\cdot})$, $\text{tag}(\mathord{\cdot})$, and $\text{text}(\mathord{\cdot})$ mappings.
• Figure 2: Our *[-20]FormatRL pipeline consists of two stages. First, we fine-tune a pre-trained LLM (e.g., Llama-3.1-8B-Instruct) using real and synthetic structured document pairs. Second, we reinforce the format handling ability through GRPO with our proposed format reward functions.
• Figure 3: Inline markup version of the example in Fig.\ref{['fig:task']}.
• Figure 4: Comparison with GPT-4.1-nano (2025-04-14), GPT-4o-mini (2024-07-18), and GPT-4o (2024-08-06).
• Figure 5: Comparison with parse-and-assemble baselines, in which the LLM acts as sentence-level MT model with or without document context.