RoofNet: A Global Multimodal Dataset for Roof Material Classification

TL;DR

RoofNet tackles the lack of global, fine-grained roofing material data crucial for hazard risk modeling by combining high-resolution Earth Observation imagery with curated text descriptions and OpenStreetMap metadata. The approach fine-tunes a vision-language model (RemoteCLIP ViT-L/14) on 6,000 expert-labeled samples (within a 77-token prompting framework) to classify 14 roof-material types, while leveraging a rule-based validator and human-in-the-loop for robustness. The dataset comprises 51,503 annotated tiles from 184 sites and includes rich metadata (roof shape, footprint area, solar panels, multi-material indicators), enabling scalable, regionally adaptive exposure modeling and cross-region generalization. RoofNet also bridges high- and low-resolution imagery by incorporating disaster-domain data (xBD) to improve robustness in real-world hazard scenarios, supporting applications in insurance, disaster preparedness, and infrastructure planning. The work provides open-access resources and a practical pipeline for improving material-specific vulnerability assessments across diverse geographic contexts.

Abstract

Natural disasters are increasing in frequency and severity, causing hundreds of billions of dollars in damage annually and posing growing threats to infrastructure and human livelihoods. Accurate data on roofing materials is critical for modeling building vulnerability to natural hazards such as earthquakes, floods, wildfires, and hurricanes, yet such data remain unavailable. To address this gap, we introduce RoofNet, the largest and most geographically diverse novel multimodal dataset to date, comprising over 51,500 samples from 184 geographically diverse sites pairing high-resolution Earth Observation (EO) imagery with curated text annotations for global roof material classification. RoofNet includes geographically diverse satellite imagery labeled with 14 key roofing types -- such as asphalt shingles, clay tiles, and metal sheets -- and is designed to enhance the fidelity of global exposure datasets through vision-language modeling (VLM). We sample EO tiles from climatically and architecturally distinct regions to construct a representative dataset. A subset of 6,000 images was annotated in collaboration with domain experts to fine-tune a VLM. We used geographic- and material-aware prompt tuning to enhance class separability. The fine-tuned model was then applied to the remaining EO tiles, with predictions refined through rule-based and human-in-the-loop verification. In addition to material labels, RoofNet provides rich metadata including roof shape, footprint area, solar panel presence, and indicators of mixed roofing materials (e.g., HVAC systems). RoofNet supports scalable, AI-driven risk assessment and serves as a downstream benchmark for evaluating model generalization across regions -- offering actionable insights for insurance underwriting, disaster preparedness, and infrastructure policy planning.

Figures (6)

• Figure 1: Overview of RoofNet and downstream applications. (1) High-resolution EO imagery is annotated with prompts describing roof material and location. Materials are labeled using expert-validated and VLM-assisted classification (2) A VLM is trained using RoofNet to label a subset of the xBD gupta_xbd_2019 dataset. (3) RoofNet is hosted online to allow for open-access to advance risk modeling frameworks and other downstream applications, and (4) xBD gupta_xbd_2019 imagery and linguistic descriptions (demographic data, roofing materials, etc.) are, incorporated with infrastructure damage classification models to verify utility.
• Figure 2: RoofNet dataset construction and classification pipeline. Architecturally and geographically diverse cities are selected to collect geocoded metadata and representative satellite imagery. Roofs are centered using GroundingDINO liu_grounding_2024, with a 6k-sample subset manually verified for fine-tuning RemoteCLIP ViT-L/14 liu_remoteclip_2024. Remaining samples are classified using the model, followed by rule-based and human-in-the-loop validation.
• Figure 3: RoofNet material classes. RoofNet includes 14 distinct roof material classes to support downstream modeling of vulnerability to natural hazards.
• Figure 4: Geographic and categorical distribution of roof material classes in the RoofNet dataset. The four world maps (left) visualize spatial coverage for each major roof material group (i.e., Manufactured Tiles, Sheet Materials, Synthetic/Flat Roofs, and Traditional/Natural) colored by material class and spanning 184 geographically diverse sites across 112 countries. These maps illustrate RoofNet’s architectural, cultural, and climatic diversity across North America, Europe, Asia, Africa, South America, the Middle East, and Oceania. The horizontal stacked bar chart (right) summarizes class prevalence by continent on a log scale, highlighting significant class imbalance. Materials like Metal Sheet Materials, Amorphous Concrete, and Asphalt Tiles dominate the dataset, while rarer types such as Thatch, Green Vegetative, and Polycarbonate Sheet Materials are underrepresented—posing challenges for model generalization and underscoring the need for balancing or augmentation during training.
• Figure 5: Prompts were selected to maximize distance in material classifications while remaining in the 77 token limit. The image above shows how class names cause confusion and similar confidence levels in the out-of-the-box remoteCLIP, while carefully selecting prompts that use simple, common language with diverse descriptions allows for a greater separation between classes. The EO image uploaded is a Concrete Tile roof and the table demonstrates the difference in prompting strategies and model confidence.