Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MeetingBank: A Benchmark Dataset for Meeting Summarization

Yebowen Hu, Tim Ganter, Hanieh Deilamsalehy, Franck Dernoncourt, Hassan Foroosh, Fei Liu

TL;DR

MeetingBank presents a large, publicly accessible benchmark of city council meetings to advance meeting summarization. It uses a divide-and-conquer approach to create 6,892 segment-level training instances from 1,366 meetings across six cities, aligning transcript segments with minutes-derived references. The study evaluates extractive, abstractive, and GPT-3 prompting methods, showing strong extractive baselines but superior performance for fine-tuned abstractive models and competitive human judgments for GPT-3, highlighting city-specific effects. By releasing videos, transcripts, minutes, agendas, and metadata, MeetingBank aims to spur robust, domain-specific summarization and improve public engagement with local government.

Abstract

As the number of recorded meetings increases, it becomes increasingly important to utilize summarization technology to create useful summaries of these recordings. However, there is a crucial lack of annotated meeting corpora for developing this technology, as it can be hard to collect meetings, especially when the topics discussed are confidential. Furthermore, meeting summaries written by experienced writers are scarce, making it hard for abstractive summarizers to produce sensible output without a reliable reference. This lack of annotated corpora has hindered the development of meeting summarization technology. In this paper, we present MeetingBank, a new benchmark dataset of city council meetings over the past decade. MeetingBank is unique among other meeting corpora due to its divide-and-conquer approach, which involves dividing professionally written meeting minutes into shorter passages and aligning them with specific segments of the meeting. This breaks down the process of summarizing a lengthy meeting into smaller, more manageable tasks. The dataset provides a new testbed of various meeting summarization systems and also allows the public to gain insight into how council decisions are made. We make the collection, including meeting video links, transcripts, reference summaries, agenda, and other metadata, publicly available to facilitate the development of better meeting summarization techniques. Our dataset can be accessed at: https://meetingbank.github.io

MeetingBank: A Benchmark Dataset for Meeting Summarization

TL;DR

MeetingBank presents a large, publicly accessible benchmark of city council meetings to advance meeting summarization. It uses a divide-and-conquer approach to create 6,892 segment-level training instances from 1,366 meetings across six cities, aligning transcript segments with minutes-derived references. The study evaluates extractive, abstractive, and GPT-3 prompting methods, showing strong extractive baselines but superior performance for fine-tuned abstractive models and competitive human judgments for GPT-3, highlighting city-specific effects. By releasing videos, transcripts, minutes, agendas, and metadata, MeetingBank aims to spur robust, domain-specific summarization and improve public engagement with local government.

Abstract

As the number of recorded meetings increases, it becomes increasingly important to utilize summarization technology to create useful summaries of these recordings. However, there is a crucial lack of annotated meeting corpora for developing this technology, as it can be hard to collect meetings, especially when the topics discussed are confidential. Furthermore, meeting summaries written by experienced writers are scarce, making it hard for abstractive summarizers to produce sensible output without a reliable reference. This lack of annotated corpora has hindered the development of meeting summarization technology. In this paper, we present MeetingBank, a new benchmark dataset of city council meetings over the past decade. MeetingBank is unique among other meeting corpora due to its divide-and-conquer approach, which involves dividing professionally written meeting minutes into shorter passages and aligning them with specific segments of the meeting. This breaks down the process of summarizing a lengthy meeting into smaller, more manageable tasks. The dataset provides a new testbed of various meeting summarization systems and also allows the public to gain insight into how council decisions are made. We make the collection, including meeting video links, transcripts, reference summaries, agenda, and other metadata, publicly available to facilitate the development of better meeting summarization techniques. Our dataset can be accessed at: https://meetingbank.github.io
Paper Structure (12 sections, 3 figures, 6 tables)

This paper contains 12 sections, 3 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Existing Datasets
  3. Creation of MeetingBank
  4. Data Analysis
  5. Performance of Existing Systems
  6. City-by-City Analysis
  7. Human Evaluation
  8. Conclusion
  9. Limitations
  10. Ethical Considerations
  11. Experimental Settings
  12. Comparison to ICSI/AMI

Figures (3)

  • Figure 1: A screenshot of a city council meeting of the City of Boston held on May 4, 2022. The meeting video is shown on the left, its corresponding minutes document on the right. The meeting includes discussions of multiple ordinances and resolutions. A summary of the discussion on item 2022-0578 is highlighted in red.
  • Figure 2: An example of a transcript snippet for a meeting segment, which serves as the source text for our summarizer. Similar to BillSum kornilova-eidelman-2019-billsum, a short description of the discussed bill serves as the segment-level reference summary. Source: Long Beach, 6/23/2022.
  • Figure 3: $\mathcal{C}$overage and $\mathcal{D}$ensity scores for segment-level summarization instances, plotted for individual cities. Seattle and Boston have the highest density scores among the cities studied, while Denver has the lowest, indicating that the minutes for this city have undergone a high degree of editing.