EuroCrops V2: the editions a national dataset no longer offers

CHANGELOG.md

@@ -32,6 +32,7 @@ and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.
 - ES: Added Spain-wide converter based on FEGA 2025+.
 - IE: Added `ie_lpis` with DAFM's Anonymous LPIS parcels for 2017-2022 and 2025
 - IT-1: Added Tuscany converter based on EuroCrops v2.
+- Added `EuroCropsV2Mixin` and the converters `ec_at`, `ec_be_wal`, `ec_de_bb`, `ec_de_nrw` and `ec_es` for the editions of a national dataset that only the JRC's EuroCrops V2 still offers. They take the national converter's crop table and admin codes, so their output fits the national collection.
 - IT-BZ: Added South Tyrol converter.
 - LT-KZS: Added Lithuania KŽS reference parcels converter.
 - PL:
@@ -41,6 +42,9 @@ and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.
 - ZA-FUSION: Added South Africa converter based on the ESA Fusion Competition dataset.
 
 ### Changed
+- EC-FR: Reads 2015, 2016 and 2018 from EuroCrops V2 (CC-BY-4.0) instead of only 2018 from EuroCrops V1.
+- EC-EE: Reads 2015 from EuroCrops V2 instead of 2021 from EuroCrops V1, which the national `ee` covers.
+- EC-BE-VLG: Reads the Flemish editions 2008-2017 from EuroCrops V2 instead of the 2021 EuroCrops V1 file, which the national `be_vlg` covers; buildings are left out.
 - `fiboa publish` no longer uploads to S3 or generates README/LICENSE files. It now creates GeoParquet, PMTiles and a STAC Collection with relative links, checksums and web-map-links.
 - Updated `aiohttp` to support Zenodo responses that include both returned `Content-Type` headers.
 - Improved geometry axis handling so generated tiles and bounding boxes keep x/y order consistent in output.
@@ -116,8 +120,10 @@ and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.
   - Excluded NBF ineligible patches with empty crop code.
   - Fixed source encoding and FLIK handling.
 - DE-NDS: Added stable collection IDs where missing.
+- DE-NRW: The INSPIRE id is now read from the source, as the declared INSPIRE extension requires.
 - DE-SAX: Updated to current available archive campaign.
 - DE-SAX: The title says it holds field blocks.
+- EE: Parcels that declare no crop are left out.
 - DE-SL:
   - Fixed parser issues with scientific notation in area values.
   - Restored full paging coverage and area derivation for complete output.

fiboa_cli/datasets/be_wal.py

@@ -37,6 +37,8 @@ class Converter(AdminConverterMixin, AddHCATMixin, FiboaBaseConverter):
         "determination:datetime": "determination:datetime",
     }
     hcat_mapping_csv = "be_wal_all_years.csv"
+    # codes after 2017, cut from the JRC's EuroCrops V2 table
+    hcat_mapping_supplements = ["https://fiboa.org/code/be/wal_supplement.csv"]
     # 208 of that table's 298 rows carry no code, only the crop name, so every
     # crop whose coded row is missing went unmapped: 23,216 fields over 59 codes.
     hcat_mapping_name_fallback = True

fiboa_cli/datasets/commons/ec.py

@@ -1,3 +1,5 @@
+from typing import Sequence
+
 from fiboa_cli.datasets.commons.hcat import AddHCATMixin
 
 
@@ -29,3 +31,76 @@ def __init__(self, *args, **kwargs):
         provider = "EuroCrops <https://github.com/maja601/EuroCrops>"
         self.provider = (f"{self.provider}, {provider}") if self.provider else provider
         self.license = "CC-BY-SA-4.0"
+
+
+JRC_V2_BASE = "https://jeodpp.jrc.ec.europa.eu/ftp/jrc-opendata/DRLL/EuroCropsV2/gpqtv202"
+
+
+class EuroCropsV2Mixin:
+    """
+    Converts the editions of a national dataset that only the JRC's EuroCrops V2 still offers, so
+    that they fit the national collection: they take its crop table, admin codes and crop columns.
+    A subclass names the national converter, the NUTS code of the files and the years it supplies.
+    """
+
+    national = None
+    nuts: str = ""
+    years: Sequence[int] = ()
+    # Codes of rows that are no field, e.g. buildings or landscape elements
+    drop_codes: set[str] = set()
+    # The national table is keyed on crop names, which the source codes otherwise (ee)
+    code_from_name = False
+
+    description = """
+The editions the national source no longer offers, from EuroCrops V2: the Joint Research Centre's
+harmonised archive of the Geo-Spatial Aid Application declarations, one GeoParquet per NUTS region
+and year in EPSG:3035. The crop codes are the national ones, mapped to HCAT with the national table.
+    """
+    provider = (
+        "Joint Research Centre, European Commission "
+        "<https://data.jrc.ec.europa.eu/dataset/b9fb9e67-78a9-4327-9d59-39a928d812d3>"
+    )
+    attribution = "European Commission, Joint Research Centre — EuroCropsV2"
+    license = "CC-BY-4.0"
+
+    def __init_subclass__(cls, **kwargs):
+        super().__init_subclass__(**kwargs)
+        national = cls.national
+        cls.id = f"ec_{national.id}"
+        cls.short_name = national.short_name
+        cls.title = f"{national.title.strip()} — EuroCrops V2"
+        cls.variants = {
+            str(y): f"{JRC_V2_BASE}/{cls.nuts}_{y}.parquet" for y in sorted(cls.years, reverse=True)
+        }
+        for attr in (
+            "hcat_mapping_csv",
+            "hcat_mapping_supplements",
+            "hcat_mapping_name_fallback",
+            "admin_subdivision_code",
+        ):
+            if hasattr(national, attr):
+                setattr(cls, attr, getattr(national, attr))
+        # AdminConverterMixin would otherwise read the country from this class's id, "ec_..."
+        cls.admin_country_code = (
+            getattr(national, "admin_country_code", None) or national.id.split("_")[0].upper()
+        )
+
+        cls.columns = {
+            "geometry": "geometry",
+            "cropfield": "id",
+            "area_ha": "metrics:area",
+            "original_code": "crop:code",
+        }
+        if "crop:name" in national.columns.values():
+            cls.columns["original_name"] = "crop:name"
+        if cls.drop_codes:
+            cls.column_filters = {"original_code": lambda col: ~col.isin(cls.drop_codes)}
+
+    def migrate(self, gdf):
+        # es 2023 stores its codes as floats
+        codes = gdf["original_code"].astype("string").str.replace(r"\.0$", "", regex=True)
+        if self.code_from_name:
+            names = self.hcat_lookup("original_name", "original_code", strip=True)
+            codes = gdf["original_name"].str.strip().map(names)
+        gdf["original_code"] = codes
+        return super().migrate(gdf)

fiboa_cli/datasets/de_bb.py

@@ -14,7 +14,11 @@ class Converter(AdminConverterMixin, AddHCATMixin, FiboaBaseConverter):
     license = "DL-DE-BY-2.0"
     provider = "Land Brandenburg <https://geobroker.geobasis-bb.de/gbss.php?MODE=GetProductInformation&PRODUCTID=9e95f21f-4ecf-4682-9a44-e5f7609f6fa0>"
     hcat_mapping_csv = "de.csv"
-    hcat_mapping_supplements = ["https://fiboa.org/code/de/de_supplement.csv"]
+    hcat_mapping_supplements = [
+        "https://fiboa.org/code/de/de_supplement.csv",
+        # Brandenburg's own codes, cut from the JRC's EuroCrops V2 table
+        "https://fiboa.org/code/de/bb_supplement.csv",
+    ]
     # The .cpg claims UTF-8 but the DBF is cp1252 (June 2026 download)
     open_options = dict(encoding="cp1252")
 

fiboa_cli/datasets/de_nrw.py

@@ -18,9 +18,12 @@ class Converter(AdminConverterMixin, AddHCATMixin, FiboaBaseConverter):
         "https://fiboa.org/flik-extension/v0.2.0/schema.yaml",
     }
     hcat_mapping_csv = "de_nrw_2021.csv"
+    # codes after 2021, cut from the JRC's EuroCrops V2 table
+    hcat_mapping_supplements = ["https://fiboa.org/code/de/nrw_supplement.csv"]
     columns = {
         "geometry": "geometry",
         "ID": "id",
+        "INSPIRE_ID": "inspire:id",
         "FLIK": "flik",
         "BEGINLIFES": "determination:datetime",
         "CODE": "crop:code",

fiboa_cli/datasets/ec_at.py

@@ -0,0 +1,12 @@
+from vecorel_cli.conversion.admin import AdminConverterMixin
+
+from ..conversion.fiboa_converter import FiboaBaseConverter
+from .at import Converter as ATConverter
+from .commons.ec import EuroCropsV2Mixin
+from .commons.hcat import AddHCATMixin
+
+
+class Converter(EuroCropsV2Mixin, AdminConverterMixin, AddHCATMixin, FiboaBaseConverter):
+    national = ATConverter
+    nuts = "at"
+    years = range(2015, 2018)

fiboa_cli/datasets/ec_be_vlg.py

@@ -1,17 +1,14 @@
-from .be_vlg import Converter as BEVLGBaseConverter
-from .commons.ec import EuroCropsConverterMixin
+from vecorel_cli.conversion.admin import AdminConverterMixin
 
+from ..conversion.fiboa_converter import FiboaBaseConverter
+from .be_vlg import Converter as BEVLGConverter
+from .commons.ec import EuroCropsV2Mixin
+from .commons.hcat import AddHCATMixin
 
-class ECConverter(EuroCropsConverterMixin, BEVLGBaseConverter):
-    id = "ec_be_vlg"
-    sources = {
-        "https://zenodo.org/records/10118572/files/BE_VLG_2021.zip?download=1": [
-            "BE_VLG_2021/BE_VLG_2021_EC21.shp"
-        ]
-    }
-    # the single EuroCrops 2021 release, not the parent's yearly editions
-    variants = {}
 
-    def __init__(self, *args, **kwargs):
-        super().__init__(*args, **kwargs)
-        del self.columns["BT_OMSCH"]
+class Converter(EuroCropsV2Mixin, AdminConverterMixin, AddHCATMixin, FiboaBaseConverter):
+    national = BEVLGConverter
+    nuts = "be2"
+    years = range(2008, 2018)
+    # buildings: "Hoofdgebouwen", "Andere gebouwen"
+    drop_codes = {"1", "2"}

fiboa_cli/datasets/ec_be_wal.py

@@ -0,0 +1,12 @@
+from vecorel_cli.conversion.admin import AdminConverterMixin
+
+from ..conversion.fiboa_converter import FiboaBaseConverter
+from .be_wal import Converter as BEWALConverter
+from .commons.ec import EuroCropsV2Mixin
+from .commons.hcat import AddHCATMixin
+
+
+class Converter(EuroCropsV2Mixin, AdminConverterMixin, AddHCATMixin, FiboaBaseConverter):
+    national = BEWALConverter
+    nuts = "be3"
+    years = [*range(2015, 2022), 2023]

fiboa_cli/datasets/ec_de_bb.py

@@ -0,0 +1,15 @@
+from vecorel_cli.conversion.admin import AdminConverterMixin
+
+from ..conversion.fiboa_converter import FiboaBaseConverter
+from .commons.ec import EuroCropsV2Mixin
+from .commons.hcat import AddHCATMixin
+from .de_bb import Converter as DEBBConverter
+
+
+class Converter(EuroCropsV2Mixin, AdminConverterMixin, AddHCATMixin, FiboaBaseConverter):
+    national = DEBBConverter
+    nuts = "de4"
+    years = range(2010, 2024)
+    # landscape elements (hedges, tree rows, copses, wetlands, single trees, stone walls, rock
+    # ridges) mapped as their own polygons from 2018
+    drop_codes = {"1", "2", "3", "4", "5", "10011", "10012"}

fiboa_cli/datasets/ec_de_nrw.py

@@ -0,0 +1,12 @@
+from vecorel_cli.conversion.admin import AdminConverterMixin
+
+from ..conversion.fiboa_converter import FiboaBaseConverter
+from .commons.ec import EuroCropsV2Mixin
+from .commons.hcat import AddHCATMixin
+from .de_nrw import Converter as DENRWConverter
+
+
+class Converter(EuroCropsV2Mixin, AdminConverterMixin, AddHCATMixin, FiboaBaseConverter):
+    national = DENRWConverter
+    nuts = "dea"
+    years = range(2019, 2024)

fiboa_cli/datasets/ec_ee.py

@@ -1,61 +1,12 @@
-import pandas as pd
-
 from ..conversion.fiboa_converter import FiboaBaseConverter
-from .commons.ec import EuroCropsConverterMixin
-
-# todo: The dataset doesn't validate due to a self intersecting polygon
-# How do we want to handle this?
-
+from .commons.ec import EuroCropsV2Mixin
+from .commons.hcat import AddHCATMixin
+from .ee import Convert as EEConverter
 
-class Convert(EuroCropsConverterMixin, FiboaBaseConverter):
-    hcat_mapping_csv = "ee_2021.csv"
-    ec_year = 2021
-    sources = {
-        "https://zenodo.org/records/14094196/files/EE_2021.zip?download=1": [
-            "EE_2021/EE_2021_EC21.shp"
-        ]
-    }
-    id = "ec_ee"
-    short_name = "Estonia"
-    title = "Field boundaries for Estonia"
-    description = """
-Geospatial Aid Application Estonia Agricultural parcels.
-The original dataset is provided by ARIB and obtained from the INSPIRE theme GSAA (specifically Geospaial Aid Application Estonia Agricultural parcels) through which the data layer Fields and Eco Areas (GSAA) is made available.
-The data comes from ARIB's database of agricultural parcels.
-    """
-    provider = "Põllumajanduse Registrite ja Informatsiooni Amet <http://data.europa.eu/88u/dataset/pria-pollud>"
-    attribution = "© Põllumajanduse Registrite ja Informatsiooni Amet"
 
-    columns = {
-        "geometry": "geometry",
-        "pollu_id": "id",
-        "taotlusaas": "determination:datetime",  # year
-        "pindala_ha": "metrics:area",  # area (in ha)
-        "taotletud_": "crop:code",  # requested crop culture
-        "taotletu_1": "taotletud_maakasutus",  # requested land use
-        "taotletu_2": "taotletud_toetus",  # requested support
-        "niitmise_t": "niitmise_tuvastamise_staatus",  # mowing detection status
-        "niitmise_1": "niitmise_tuvast_ajavahemik",  # mowing detection period
-        "viimase_mu": "viimase_muutmise_aeg",  # Last edit time (date-date)
-        "taotleja_n": "taotleja_nimi",  # name of applicant
-        "taotleja_r": "taotleja_registrikood",  # applicant's registration code
-    }
-    # The shapefile truncates taotlusaasta ("application year") to ten characters;
-    # it holds a bare year, which has to become a datetime for the STAC extent.
-    column_migrations = {"taotlusaas": lambda col: pd.to_datetime(col, format="%Y")}
-    missing_schemas = {
-        "required": [
-            "taotletud_maakasutus",
-            "viimase_muutmise_aeg",
-            "taotleja_nimi",
-        ],
-        "properties": {
-            "taotletud_maakasutus": {"type": "string"},
-            "niitmise_tuvastamise_staatus": {"type": "string"},
-            "niitmise_tuvast_ajavahemik": {"type": "string"},
-            "viimase_muutmise_aeg": {"type": "string"},
-            "taotletud_toetus": {"type": "string"},
-            "taotleja_nimi": {"type": "string"},
-            "taotleja_registrikood": {"type": "string"},
-        },
-    }
+class Converter(EuroCropsV2Mixin, AddHCATMixin, FiboaBaseConverter):
+    national = EEConverter
+    nuts = "ee"
+    # PRIA's WFS layers before 2016 are empty
+    years = (2015,)
+    code_from_name = True

fiboa_cli/datasets/ec_es.py

@@ -0,0 +1,12 @@
+from vecorel_cli.conversion.admin import AdminConverterMixin
+
+from ..conversion.fiboa_converter import FiboaBaseConverter
+from .commons.ec import EuroCropsV2Mixin
+from .commons.hcat import AddHCATMixin
+from .es import Converter as ESConverter
+
+
+class Converter(EuroCropsV2Mixin, AdminConverterMixin, AddHCATMixin, FiboaBaseConverter):
+    national = ESConverter
+    nuts = "es"
+    years = range(2021, 2024)

fiboa_cli/datasets/ec_fr.py

@@ -1,36 +1,13 @@
-from ..conversion.fiboa_converter import FiboaBaseConverter
-from .commons.ec import EuroCropsConverterMixin
-
+from vecorel_cli.conversion.admin import AdminConverterMixin
 
-class Converter(EuroCropsConverterMixin, FiboaBaseConverter):
-    hcat_mapping_csv = "fr_2018.csv"
-    hcat_mapping_supplements = ["https://fiboa.org/code/fr/fr_2018_supplement.csv"]
-    ec_year = 2018
-    sources = {
-        "https://zenodo.org/records/14094196/files/FR_2018.zip": ["FR_2018/FR_2018_EC21.shp"]
-    }
+from ..conversion.fiboa_converter import FiboaBaseConverter
+from .commons.ec import EuroCropsV2Mixin
+from .commons.hcat import AddHCATMixin
+from .fr import FRConverter
 
-    id = "ec_fr"
-    short_name = "France"
-    title = "Field boundaries for France"
-    description = """
-The 2018 campaign of the Registre Parcellaire Graphique, which IGN never released as an
-archive of its own: the RPG downloads run 2017 and 2019 onwards. EuroCrops publishes it
-with its HCAT columns already resolved.
-    """
-    provider = "Institut National de l'Information Géographique et Forestière <https://geoservices.ign.fr/rpg>"
-    attribution = "IGN - Original data from https://geoservices.ign.fr/rpg"
 
-    columns = {
-        "geometry": "geometry",
-        "ID_PARCEL": "id",
-        "SURF_PARC": "metrics:area",
-        "CODE_CULTU": "crop:code",
-        "CODE_GROUP": "group_code",
-    }
-    missing_schemas = {"properties": {"group_code": {"type": "string"}}}
-    column_migrations = {
-        "CODE_GROUP": lambda col: col.astype("string").str.strip(),
-    }
-    # SURF_PARC is rounded to 0.01 ha, so a parcel under 50 m2 reads as zero and is
-    # measured from the geometry (area_calculate_missing)
+class Converter(EuroCropsV2Mixin, AdminConverterMixin, AddHCATMixin, FiboaBaseConverter):
+    national = FRConverter
+    nuts = "fr"
+    # IGN's RPG downloads run 2017 and 2019 onwards
+    years = (2015, 2016, 2018)

fiboa_cli/datasets/ee.py

@@ -38,6 +38,8 @@ class Convert(AddHCATMixin, FiboaBaseConverter):
     attribution = "© Põllumajanduse Registrite ja Informatsiooni Amet"
     license = "CC-BY-SA-3.0"
     columns = COLUMNS
+    # a few parcels declare no crop
+    column_filters = {"taotletud_kultuur": lambda col: col.str.strip().fillna("") != ""}
     column_migrations = {"taotlusaasta": lambda col: pd.to_datetime(col, format="%Y")}
     missing_schemas = {
         "properties": {

tests/data-files/convert/be_wal/wal_supplement.csv

@@ -0,0 +1,18 @@
+original_code,original_name,translated_name,HCAT3_name,HCAT3_code
+19,Culture hydroponique ou aquaponique,Hydroponic or aquaponic cultivation,not_known_and_other,3399000000
+48,Cameline,Camelina,camelina,3301061500
+59,Fenugrec,Fenugreek,fenugreek,3301020400
+89,Terre en jachère à activité minimale avec EAG,Fallow land with minimal activity with EAG,fallow_land_not_crop,3301110000
+383,Millet,Millet,millet_sorghum,3301010900
+397,"Mélange froment ou épeautre ( plus de 50%) et de  pois ou féverole (plus de 20%), commercialisé à l'état sec","Mixture of wheat or spelt (more than 50%) and peas or fava beans (more than 20%), marketed in a dry state.",cereal,3301010000
+614,Prairie avec moins de 90 % couvert herbacé (pratiques locales établies),Pasture with less than 90% herbaceous cover (established local practices).,pasture_meadow_grassland_grass,3302000000
+700,Mélange graminées/trèfle,Grass/clover mix,clover,3301090303
+722,Trèfle pérenne,Perennial clover,clover,3301090303
+723,Trèfle violet,Red clover,clover,3301090303
+747,Vesce,Vetch,vetches,3301090305
+812,Jachère non herbacée,non-herbaceous fallow,fallow_land_not_crop,3301110000
+923,Lin d'hiver (fibre),Winter flax (fiber),flax_linseed,3301060700
+933,Pois chiche,Chickpea,chickpeas,3301020200
+934,lentille,lentil,lentils,3301020500
+967,Maraichage diversifié,Diversified vegetables,fresh_vegetables,3301070000
+9732,Cultures fruitières pluriannuelles (prunes) - hautes tiges,multi-annual fruit crops (plums) - tall stalks,plums,3303011300