Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Exploring Generative AI Techniques in Government: A Case Study

Sunyi Liu, Mengzhe Geng, Rebecca Hart

TL;DR

This paper examines how Generative AI and large language models can be integrated into government operations through NRC's GenAI pilot, focusing on the Pubbie intelligent agent as a case study for automating performance measurement, data management, and reporting. It presents a three-fold architecture that combines a fine-tuned RoBERTa classifier for program prediction, LLM orchestration via the Semantic Kernel, and a lightweight SQLite backend to enable context-aware, scalable responses to open-ended queries while meeting resource constraints. The study includes quantitative evaluations, showing RoBERTa achieving 84.8% accuracy on challenge-program classification and text-to-SQL conversion accuracy of 96.15%, along with a cost analysis of roughly $18k and a user-friendly Streamlit interface. The findings demonstrate the practical viability and adaptability of GenAI-based workflows for general government use, with insights on capacity-building and collective problem-solving across multiple GenAI projects, and outline future work to improve stability and broaden applicability.

Abstract

The swift progress of Generative Artificial intelligence (GenAI), notably Large Language Models (LLMs), is reshaping the digital landscape. Recognizing this transformative potential, the National Research Council of Canada (NRC) launched a pilot initiative to explore the integration of GenAI techniques into its daily operation for performance excellence, where 22 projects were launched in May 2024. Within these projects, this paper presents the development of the intelligent agent Pubbie as a case study, targeting the automation of performance measurement, data management and insight reporting at the NRC. Cutting-edge techniques are explored, including LLM orchestration and semantic embedding via RoBERTa, while strategic fine-tuning and few-shot learning approaches are incorporated to infuse domain knowledge at an affordable cost. The user-friendly interface of Pubbie allows general government users to input queries in natural language and easily upload or download files with a simple button click, greatly reducing manual efforts and accessibility barriers.

Exploring Generative AI Techniques in Government: A Case Study

TL;DR

This paper examines how Generative AI and large language models can be integrated into government operations through NRC's GenAI pilot, focusing on the Pubbie intelligent agent as a case study for automating performance measurement, data management, and reporting. It presents a three-fold architecture that combines a fine-tuned RoBERTa classifier for program prediction, LLM orchestration via the Semantic Kernel, and a lightweight SQLite backend to enable context-aware, scalable responses to open-ended queries while meeting resource constraints. The study includes quantitative evaluations, showing RoBERTa achieving 84.8% accuracy on challenge-program classification and text-to-SQL conversion accuracy of 96.15%, along with a cost analysis of roughly $18k and a user-friendly Streamlit interface. The findings demonstrate the practical viability and adaptability of GenAI-based workflows for general government use, with insights on capacity-building and collective problem-solving across multiple GenAI projects, and outline future work to improve stability and broaden applicability.

Abstract

The swift progress of Generative Artificial intelligence (GenAI), notably Large Language Models (LLMs), is reshaping the digital landscape. Recognizing this transformative potential, the National Research Council of Canada (NRC) launched a pilot initiative to explore the integration of GenAI techniques into its daily operation for performance excellence, where 22 projects were launched in May 2024. Within these projects, this paper presents the development of the intelligent agent Pubbie as a case study, targeting the automation of performance measurement, data management and insight reporting at the NRC. Cutting-edge techniques are explored, including LLM orchestration and semantic embedding via RoBERTa, while strategic fine-tuning and few-shot learning approaches are incorporated to infuse domain knowledge at an affordable cost. The user-friendly interface of Pubbie allows general government users to input queries in natural language and easily upload or download files with a simple button click, greatly reducing manual efforts and accessibility barriers.

Paper Structure

This paper contains 24 sections, 2 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. LLM Orchestration
  4. BERT and RoBERTa
  5. SQLite Database
  6. The Intelligent Agent Pubbie: A Case Study
  7. Background
  8. Tasks and Challenges
  9. Overview of the Workflows
  10. Implementation Details
  11. Semantic embedding and classification
  12. LLM with few-shot learning
  13. Integration of the SQLite database
  14. User-friendly interface design
  15. System Evaluation
  16. ...and 9 more sections

Figures (2)

  • Figure 1: The interface of our intelligent agent, Pubbie, including (a) a button for uploading the CSV file; (b) a button for downloading the CSV file; and (c) a chat box for inquiries in natural language.
  • Figure 2: The three workflows of our intelligent agent, Pubbie. The parenthesis after "LLM" indicates the type of prompt template utilized for few-shot learning.