LaMSUM: Amplifying Voices Against Harassment through LLM Guided Extractive Summarization of User Incident Reports

TL;DR

LaMSUM addresses the challenge of extracting representative, extractive summaries from large, multilingual Safe City posts that exceed standard LLM context windows. It introduces a multi-level framework that chunkifies data, generates per-chunk summaries, and aggregates them via voting-based methods (Pluralsity, Proportional Approval Voting, and Borda) to produce a robust final selection of $k$ posts. The approach mitigates positional bias through input shuffling, uses zero-shot prompting, and calibrates outputs to ensure fidelity to the input text, achieving superior ROUGE scores across five city datasets and three LLMs, with GPT-4o-mini + Proportional Approval Voting often performing best. This work advances practical, policy-relevant processing of long, code-mixed user-generated content and provides a companion website for rapid, area-specific incident overviews, while acknowledging ethical considerations and limitations inherent to LLM-based summarization.

Abstract

Citizen reporting platforms like Safe City in India help the public and authorities stay informed about sexual harassment incidents. However, the high volume of data shared on these platforms makes reviewing each individual case challenging. Therefore, a summarization algorithm capable of processing and understanding various Indian code-mixed languages is essential. In recent years, Large Language Models (LLMs) have shown exceptional performance in NLP tasks, including summarization. LLMs inherently produce abstractive summaries by paraphrasing the original text, while the generation of extractive summaries - selecting specific subsets from the original text - through LLMs remains largely unexplored. Moreover, LLMs have a limited context window size, restricting the amount of data that can be processed at once. We tackle these challenge by introducing LaMSUM, a novel multi-level framework designed to generate extractive summaries for large collections of Safe City posts using LLMs. LaMSUM integrates summarization with different voting methods to achieve robust summaries. Extensive evaluation using three popular LLMs (Llama, Mistral and GPT-4o) demonstrates that LaMSUM outperforms state-of-the-art extractive summarization methods for Safe City posts. Overall, this work represents one of the first attempts to achieve extractive summarization through LLMs, and is likely to support stakeholders by offering a comprehensive overview and enabling them to develop effective policies to minimize incidents of unwarranted harassment.

Figures (7)

• Figure 1: Current LLMs, by default, produce abstractive summaries. Llama-3.3-70b, despite specifically prompted for extractive summarization, generates abstractive summaries. This behavior underscores the need for a targeted approach to enable LLMs to effectively generate extractive summaries.
• Figure 2: LaMSUM: Multi-level framework for extractive summarization of large user-generated text. Input set $\mathcal{T}\xspace$ (level 0) is divided into $\lceil{\frac{|\mathcal{T}\xspace|}{s}}\rceil$ chunks each of size s. From each chunk a summary is produced of size $q$ (refer Figure \ref{['fig:shuffle']}), $q$ length summaries from $\lceil{\frac{|\mathcal{T}\xspace|}{s}}\rceil$ chunks are merged to form the input for the next level i.e., level 1. Iteratively the same procedure is repeated till we obtain a summary of size $k$. We set $q=k$ to ensure our algorithm can effectively handle the worst-case scenario where all the textual units in the final summary may come from the same input chunk.
• Figure 3: Textual units (e.g., posts) in the input chunk are shuffled to account for the positional bias. $m$ different chunk variations are obtained through shuffling, which are subsequently summarized using LLMs. $m$ summaries are then aggregated by voting algorithms to get the final summary.
• Figure 4: Metric scores obtained through four different LLM setups. (i) Vanilla LLM without shuffling and voting method (ii) LaMSUM with Plurality Approval Voting (iii) LaMSUM with Proportional Approval Voting and (iv) LaMSUM with Borda Count Ranked Voting. Results demonstrate that gpt-4o-mini with Proportional Approval Voting performs the best across all the cases. Here, Llama, Mixtral and GPT refers to llama-3.1-8B, open-mistral-nemo, and gpt-4o-mini respectively.
• Figure 5: Posts chosen by LaMSUM tend to be detailed and descriptive, offering a deeper level of information. Number of words in LaMSUM selected posts is often highest across various datasets, ensuring extensive and comprehensive summarization.