Generative AI for Analyzing Participatory Rural Appraisal Data: An Exploratory Case Study in Gender Research

TL;DR

The study investigates applying Generative AI to unstructured visual PRA data used in gender research, focusing on the Ideal Village activity. It benchmarks three state-of-the-art LLMs (GPT-4o, Claude 3.5 Sonnet, Gemini 1.5 Pro) on visual interpretation, multilingual translation, and data classification of hand-drawn artifacts. Findings show substantial challenges in translation, cultural-context understanding, and reliable categorization, with hallucinations and misclassifications undermining reliability. The work highlights the need for human-in-the-loop oversight and calls for developing inclusive, multi-modal AI models that align with participatory research principles to better support community-driven empowerment analyses.

Abstract

This study explores the novel application of Generative Artificial Intelligence (GenAI) in analyzing unstructured visual data generated through Participatory Rural Appraisal (PRA), specifically focusing on women's empowerment research in rural communities. Using the "Ideal Village" PRA activity as a case study, we evaluate three state-of-the-art Large Language Models (LLMs) - GPT-4o, Claude 3.5 Sonnet, and Gemini 1.5 Pro - in their ability to interpret hand-drawn artifacts containing multilingual content from various Indian states. Through comparative analysis, we assess the models' performance across critical dimensions including visual interpretation, language translation, and data classification. Our findings reveal significant challenges in AI's current capabilities to process such unstructured data, particularly in handling multilingual content, maintaining contextual accuracy, and avoiding hallucinations. While the models showed promise in basic visual interpretation, they struggled with nuanced cultural contexts and consistent classification of empowerment-related elements. This study contributes to both AI and gender research by highlighting the potential and limitations of AI in analyzing participatory research data, while emphasizing the need for human oversight and improved contextual understanding. Our findings suggest future directions for developing more inclusive AI models that can better serve community-based participatory research, particularly in gender studies and rural development contexts.

Figures (2)

• Figure 1: Sample output demonstrating GPT-4o's analysis of an Ideal Village participatory drawing from a rural Indian community. The analysis was generated using a structured prompt designed to identify and categorize village elements. The output table shows element classification across different dimensions, with green-highlighted rows indicating accurate interpretations and red-highlighted rows showing misclassifications or potential hallucinations by the model.
• Figure 2: Sample output illustrating GPT-4o’s analysis of a Circle of Control participatory drawing from a rural Indian community. The model identified elements and their locations (inner, middle, or outer circles) while categorizing them across different dimensions. In the output table, green-highlighted rows indicate accurate interpretations across all columns, orange-highlighted rows show partially correct classifications, and red-highlighted rows indicate misclassifications.