Towards More Effective Table-to-Text Generation: Assessing In-Context Learning and Self-Evaluation with Open-Source Models

TL;DR

This study explores the effectiveness of various in-context learning strategies in LMs across benchmark datasets, focusing on the impact of providing examples to the model, and employs a large language model (LLM) self-evaluation approach using chain-of-thought reasoning and assess its correlation with human-aligned metrics like BERTScore.

Abstract

Table processing, a key task in natural language processing, has significantly benefited from recent advancements in language models (LMs). However, the capabilities of LMs in table-to-text generation, which transforms structured data into coherent narrative text, require an in-depth investigation, especially with current open-source models. This study explores the effectiveness of various in-context learning strategies in LMs across benchmark datasets, focusing on the impact of providing examples to the model. More importantly, we examine a real-world use case, offering valuable insights into practical applications. To complement traditional evaluation metrics, we employ a large language model (LLM) self-evaluation approach using chain-of-thought reasoning and assess its correlation with human-aligned metrics like BERTScore. Our findings highlight the significant impact of examples in improving table-to-text generation and suggest that, while LLM self-evaluation has potential, its current alignment with human judgment could be enhanced. This points to the need for more reliable evaluation methods.

Figures (9)

• Figure 1: An example of in-context learning for table-to-text generation.
• Figure 2: The real-world table-to-text generation scenario investigated in this paper.
• Figure 3: An example in the Wikibio dataset: the pairs of infobox and biographical introductions. The goal of the task is given the infobox, generate a biography style description. The biography generated by our pipeline is: “Curtis Wright (born June 6, 1955, in Huntingdon, Pennsylvania, United States) is a country singer-songwriter. He has been active in the music industry since 1989 and is known for his solo performances as well as his work with various associated acts such as Vern Gosdin, Robert Ellis Orrall, Shenandoah, and others. Wright's music career has spanned multiple decades, during which he has released albums under labels including MCA/Airborne and has been part of groups like Orrall & Wright, Free Falls, Cumberland Road, Liberty, and Giant.”lebret2016neural.
• Figure 4: An example in the Mardi dataset. The biography generated by our pipeline is: “Douglas Bates is a researcher who holds degrees from two esteemed institutions: University of California, Los Angeles and Queen's University. Throughout his career, Douglas has been affiliated with several prestigious organizations, including the University of Wisconsin–Madison and the University of Alberta. As an accomplished researcher, he has guided numerous students in their doctoral studies, including Christian Ritter, William Whipple Neely, Mary Judith Lindstrom, Saikat Debroy, Sandra Jane Martin, José C. Pinheiro, Yuhwen Soo, Dennis A. Wolf, Gunseog Kang, and Andrzej P. Jaworski. Notably, Douglas has been recognized as a Fellow of the American Statistical Association and is an active member of the organization.”
• Figure 5: Example in the ToTTo dataset: a pair of table with highlighted cells and hand-crafted reference descriptions. The goal of the task is given the table, table metadata (such as the title), and set of highlighted cells, to produce a description. parikh2020totto. The description generated by our pipeline is: “In the 2015 TV series “The Returned”, Renn Hawkey played the role of Paul Koretsky.”