Key Information Retrieval to Classify the Unstructured Data Content of Preferential Trade Agreements

TL;DR

The paper tackles the challenge of classifying provisions in ultra-long Preferential Trade Agreements by proposing a pipeline that compresses long texts through context-aware segmentation and top-k key paragraph extraction using BERT embeddings, followed by a classification task on the condensed input. It introduces Context-Aware Text Partitioning with dynamic window size $w=f(p, D)$ and overlap $o=g(w, D)$ to preserve semantic coherence, and relies on cosine similarity between question and paragraph embeddings to select top-k candidates (commonly $k=10$). Empirical results show that BERT-based retrieval yields over 50% gains in similarity scores compared with TF-IDF baselines and enables a reduction of input length to under 600 tokens for downstream classification, improving both runtime and accuracy. The work offers a scalable approach for long-text prediction with practical implications for automating provision mappings in PTAs and similar long-document classification tasks, with future directions including GPU acceleration, LLM-assisted retrieval, and privacy-preserving designs.

Abstract

With the rapid proliferation of textual data, predicting long texts has emerged as a significant challenge in the domain of natural language processing. Traditional text prediction methods encounter substantial difficulties when grappling with long texts, primarily due to the presence of redundant and irrelevant information, which impedes the model's capacity to capture pivotal insights from the text. To address this issue, we introduce a novel approach to long-text classification and prediction. Initially, we employ embedding techniques to condense the long texts, aiming to diminish the redundancy therein. Subsequently,the Bidirectional Encoder Representations from Transformers (BERT) embedding method is utilized for text classification training. Experimental outcomes indicate that our method realizes considerable performance enhancements in classifying long texts of Preferential Trade Agreements. Furthermore, the condensation of text through embedding methods not only augments prediction accuracy but also substantially reduces computational complexity. Overall, this paper presents a strategy for long-text prediction, offering a valuable reference for researchers and engineers in the natural language processing sphere.

Figures (3)

• Figure 1: Overview of the Long Document Classification Pipeline. The process begins with an original document pool combined with DESTADATA to initiate the text retriever process, which involves truncation, chunking, and selection stages, using models like BGE-base, GTE-base, and BERT embeddings. The refined data then passes through a Ground Truth Finding Algorithm and is further reduced to under 600 tokens. This optimized content, alongside question explanations, feeds into a fine-tuning stage on a classification task, outputting a label that determines if the long document contains the answer to the input question.
• Figure 2: Workflow of the Context-Aware Text Partitioning Method for dataset creation. The process compares a query question with long articles using two techniques: TF-IDF and BERT Embedding Methods. Each method selects the top k(10) relevant paragraphs, which are then assessed through a sliding window to establish ground truth data. The resulting dataset comprises paragraphs ranked by their similarity to the query.
• Figure 3: Sampled similiarity score improvement of BERT embedding method in comparison with the baseline(TF-IDF) model