SDUs DAISY: A Benchmark for Danish Culture

TL;DR

SDU's Daisy introduces a culturally grounded Danish benchmark derived from the Kulturkanon to evaluate Danish heritage knowledge in LLMs. The dataset comprises 741 close-ended QA pairs generated from canon-related Wikipedia pages and validated by humans, with evaluation using F1 and BLEU metrics. Baseline results across five models reveal substantial gaps in cultural knowledge, highlighting the challenge of encoding national heritage in low-resource Danish contexts. Daisy provides a starting point for Danish foundation models and digital humanities research, with plans to expand coverage and invite community contributions to improve cultural competence.

Abstract

We introduce a new benchmark for Danish culture via cultural heritage, Daisy, based on the curated topics from the Danish Culture Canon 2006. For each artifact in the culture canon, we query the corresponding Wikipedia page and have a language model generate random questions. This yields a sampling strategy within each work, with a mix of central of peripheral questions for each work, not only knowledge of mainstream information, but also in-depth cornerstones defining the heritage of Danish Culture, defined by the Canon committee. Each question-answer pair is humanly approved or corrected in the final dataset consisting of 741 close-ended question answer pairs covering topics, from 1300 BC. archaeological findings, 1700 century poems and musicals pieces to contemporary pop music and Danish design and architecture.

Figures (1)

• Figure 1: Overview of the stages in Daisy's Generation Pipeline.