GEO-Bench-2:#
The emergence of Geospatial Foundation Models (GeoFMs) holds great promise for advancing Earth Observation (EO), enabling more general and scalable solutions for a variety of tasks. However, the rapidly evolving nature of this field has meant that evaluation protocols have been difficult to standardize.
GEO-Bench-2 is an effort to address this challenge by providing a comprehensive and community-focused benchmarking framework tailored to various EO applications. Expanding upon its predecessor, this framework is designed to facilitate consistent, insightful, and fair comparison of GeoFMs.
What GEO-Bench-2 Offers#
We aim to simplify and standardize the evaluation process through several key features:
Diverse and Permissively Licensed Data: We include a curated selection of 19 datasets covering core EO tasks, including classification, segmentation, regression, object detection, and instance segmentation, ensuring broad usability.
Targeted Evaluation via “Capabilities”: Datasets are grouped into “capabilities” based on shared characteristics (e.g., resolution, band usage, temporality). This feature supports flexible benchmarking, allowing users to assess a model’s strengths on specific types of EO data.
Robust Metrics and Efficiency: We utilize the normalized interquartile mean (IQM) for more robust model comparison and incorporate subsampling strategies to help make large-scale evaluation more efficient.
Ease of Use with TerraTorch: Integration with the TerraTorch open-source toolkit. However, all datasets and datamodules can also be used independently of TerraTorch
Installation#
pip install geobenchv2
Dataset Overview#
Dataset |
Task |
Modalities |
Train/Val/Test Samples |
# Classes |
License |
Citation |
|---|---|---|---|---|---|---|
BigEarthNetV2 |
Multi-label land cover classification |
Sentinel-1 SAR (VV,VH) + Sentinel-2 Optical |
20000 / 4000 / 4000 |
19 (multi-label) |
CDLA-Permissive-1.0 |
|
TreeSatAI |
Tree species classification |
Sentinel-2 Time Series |
20000 / 4000 / 4000 |
13 |
CC-BY-4.0 |
|
m-so2sat |
Climate zone classification |
Sentinel-2 Optical |
19992 / 986 / 986 |
17 |
CC-BY-4.0 |
|
m-forestnet |
Tree species classification |
Landsat 8 |
6464 / 989 / 989 |
12 |
CC-BY-4.0 |
|
BioMassters |
Biomass regression |
Sentinel-1 SAR (VV,VH) + Sentinel-2 Optical |
4011 / 1739 / 2776 |
Continuous |
CC-BY-4.0 |
|
CaFFe |
Glacier zone segmentation |
Sentinel-1 SAR |
4000 / 1000 / 2000 |
4 |
CC-BY-4.0 |
|
CloudSEN12 |
Cloud/shadow segmentation |
Sentinel-1 SAR + Sentinel-2 Optical |
4000 / 535 / 975 |
4 |
CC0 |
|
Burn Scars |
Burned area segmentation |
HLS (Landsat 8/9 + Sentinel-2) |
524 / 160 / 120 |
2 (binary) |
CC-BY-4.0 |
|
Dynamic EarthNet |
Land cover semantic segmentation |
Sentinel-2 (10 bands) + Planet (4 bands) |
700 / 100 / 200 |
7 |
CC-BY-4.0 |
|
FLAIR2 |
Land cover semantic segmentation |
Aerial RGB+NIR + DEM |
4049 / 1022 / 3022 |
13 |
Open License 2.0 |
|
m-FoTW |
Field boundary segmentation |
Sentinel-2 Optical |
4000 / 1000 / 2000 |
2 (binary) |
CC-BY-SA |
|
KuroSiwo |
Flood segmentation |
Sentinel-1 SAR + DEM + Slope |
4000 / 1000 / 2000 |
4 |
MIT |
|
PASTIS (R) |
Crop type + parcel segmentation |
Sentinel-1 (asc/desc) + Sentinel-2 time series |
1455 / 482 / 496 |
19 (18 crops + bg) |
CC BY 4.0 |
|
SpaceNet2 |
Building footprint segmentation |
WorldView VHR Optical RGB |
5186 / 1461 / 2961 |
2 (binary) |
CC-BY-SA-4.0 |
|
SpaceNet7 |
Building segmentation/tracking |
Planet RGB time series |
3500 / 652 / 1152 |
2 (binary) |
CC-BY-SA-4.0 |
|
EverWatch |
Bird object detection |
Aerial RGB |
4429 / 500 / 196 |
9 |
CC0-1.0 |
|
m-nzcattle |
Cattle object detection |
Aerial RGB |
524 / 66 / 65 |
2 |
CC-BY-4.0 |
|
Substation |
Power substation segmentation |
Sentinel-2 Optical + OSM |
4000 / 500 / 500 |
2 (binary) |
CC-BY-4.0 / ODbL 1.0 |
|
So2Sat |
Local Climate Zones |
Sentinel-1 SAR (VV,VH) + Sentinel-2 Optical |
19992/986/986 |
17 |
CC-BY-4.0 |
Geographical Distribution of Datasets#
Global Sample Distribution#
Geographical Distribution across Continents#
Global Coverage Distribution Continets#