diff --git a/_freeze/docs/force/guide/level2/execute-results/html.json b/_freeze/docs/force/guide/level2/execute-results/html.json index 8438206..fa6b5d1 100644 --- a/_freeze/docs/force/guide/level2/execute-results/html.json +++ b/_freeze/docs/force/guide/level2/execute-results/html.json @@ -1,8 +1,8 @@ { - "hash": "d7fc8d38e1e1a8c9b093993fa7f2fd53", + "hash": "785793aa7ac3061e9e1071a4dced2896", "result": { "engine": "jupyter", - "markdown": "---\nformat:\n html:\n page-navigation: true\nexecute:\n freeze: true\n---\n\n# FORCE level 2\n\n## Recap\n\n:::{.callout-tip collapse=\"true\" title=\"Selecting input products\"}\n\n::: {#748a2267 .cell execution_count=1}\n``` {.python .cell-code}\nL1C_COLLECTION_URL = \"https://stac.dataspace.copernicus.eu/v1/collections/sentinel-2-l1c\"\nSTAC_ROOT_URL = \"https://stac.dataspace.copernicus.eu/v1\"\n#w,s,e,n = 10.386, 44.437, 11.423, 44.973\nw, s, e, n = 11.0, 44.5, 11.1, 44.6\nspatial_extent = { \"west\": w, \"south\": s, \"east\": e, \"north\": n}\n# ATTENTION: Inclusion of the second date in the search depends on the query method.\ntemporal_extent=[\"2026-04-17\", \"2026-04-27\"]\n```\n:::\n\n\n:::{.panel-tabset group=\"query_engine\"}\n\n## openEO\n\n::: {#9026a05b .cell execution_count=2}\n``` {.python .cell-code}\nimport openeo\nconnection = openeo.connect(\"openeo.dataspace.copernicus.eu\").authenticate_oidc()\n\nquery_pg = openeo.processes.process(\n \"query_stac\",\n arguments={\n \"url\": L1C_COLLECTION_URL,\n \"temporal_extent\": temporal_extent,\n \"spatial_extent\": spatial_extent, }\n)\n```\n\n::: {.cell-output .cell-output-stdout}\n```\nAuthenticated using refresh token.\n```\n:::\n:::\n\n\n## Pystac client {.active}\n\n::: {#a6cae6cc .cell execution_count=3}\n``` {.python .cell-code}\nimport pystac_client\n\nclient = pystac_client.Client.open(STAC_ROOT_URL)\nsearch = client.search(\n datetime=temporal_extent,\n collections=[\"sentinel-2-l1c\"],\n bbox=[w, s, e, n]\n)\nitem_collection = search.item_collection()\n```\n:::\n\n\n \n:::\n \n:::\n\n## Level 2 processing\n\n\nThe [openeo Process](processes.qmd) is straightforwardly named `force_level2`. It takes most of the parameters that can be specified in the FORCE level 2 parameter file. See the article on [parametrization](parametrization.qmd) for more details which parameters are (not) exposed by the process.\n\n:::{.callout-tip}\nThe FORCE parameter file format uses [`SCREAMING_SNAKE_CASE`](https://en.wikipedia.org/wiki/screaming_snake_case) to name its parameters.\nThe integrated FORCE uses (lowercase) [`snake_case`](https://en.wikipedia.org/wiki/Snake_case) instead, to be consistent with openEO conventions. Otherwise, the parameter names are unchanged, so it is straightforward to determine the openEO name if you are familiar with the FORCE parameters.\n:::\n\nThe parameters are documented in the process description. We can inspect it using the Python client's [`describe_processes()`](https://open-eo.github.io/openeo-python-client/api.html#openeo.rest.connection.Connection.describe_process).\n\n\n\n::: {#d3c0a21b .cell execution_count=4}\n``` {.python .cell-code}\nconnection = openeo.connect(\"openeo.dataspace.copernicus.eu\").authenticate_oidc()\n```\n\n::: {.cell-output .cell-output-stdout}\n```\nAuthenticated using refresh token.\n```\n:::\n:::\n\n\n\n\n::: {#5b560fce .cell execution_count=5}\n\n::: {.cell-output .cell-output-stdout}\n```\nAuthenticated using refresh token.\n```\n:::\n:::\n\n\n:::{.callout-note collapse=\"true\" title=\"FORCE level 2 parameters\"}\n\n::: {#6479554f .cell execution_count=6}\n``` {.python .cell-code}\nconnection.describe_process(\"force_level2\")\n```\n\n::: {.cell-output .cell-output-display execution_count=24}\n```{=html}\n\n \n \n \n \n \n```\n:::\n:::\n\n\n:::\n\n### Building the Process Graph\n\nThe FORCE process graph is based on the `force_level2` process, as described in [openEO processes](processes.qmd). Because FORCE does not use the standard openEO version of the \"data cube\" concept, but\nuses its own data cube representation, we need to wrap the process so that we can interact with the results in a convenient way as a StacResource and export the processing results to a workspace.\n\n\nConcept | Qualified name (Python client) | Purpose\n--------|--------------------------------|-------------\n`StacResource` | [`openeo.rest.stac_resource.StacResource`](https://force-eo.readthedocs.io/en/latest/components/higher-level/tsa/) | A handle to the process graph results, makes it possible to inspect the results as usual for openEO datacubes.\n`export_workspace` | [`openeo.rest.result.SaveResult.export_workspace`](https://open-eo.github.io/openeo-python-client/api.html#openeo.rest.result.SaveResult.export_workspace) | A process to publish the processing results (e.g., a FORCE ARD data cube) to a shared cloud storage location. There is a special workspace configured for FORCE (`apex-force-force-results`) accessible to all users of the CDSE openEO backend. The workspace serves as a storage location to keep data cubes for later processing (e.g., with the [TSA](tsa.qmd) module).\n`PGNode` | [`openeo.internal.graph_building.PGNode`](https://open-eo.github.io/openeo-python-client/api.html#openeo.internal.graph_building.PGNode) | A single node in a process graph. `PGNode` is an internal class but may be used for advanced graph building purposes [as described here](https://open-eo.github.io/openeo-python-client/processes.html#callback-as-pgnode). We use it for the `graph` parameter of `StacResource`.\n\n\n#### Workspaces\n\nOpenEO uses the concept of [Workspaces](https://docs.openeo.cloud/workspaces/) to interface with object storage. For processing with FORCE, we can use a workspace dedicated to FORCE to store and exchange processing results between level 2 and higher level processing, and with colleagues.\n\nWe can use either the dedicated workspace `apex-force-results-workspace`, which is accessible to all CDSE openeo users, or another registered workspace. See the [openEO platform documentation on workspaces](https://docs.openeo.cloud/workspaces/#registering-a-workspace) on how workspaces can be accessed and registered.\n\nWhen running FORCE level2 on its own, we do not need to use a workspace. Simply download the results when you are done with processing. However, to reference the level 2 processing results in [further processing](tsa.qmd), we need to use a workspace.\n\nWe can instruct openEO to store our results in a workspace using the [`export_workspace`](https://openeo.org/documentation/1.0/processes.html#export_workspace) openEO process.\n\nWhen using `export_workspace`, we need to pass a `merge` string to identify your result. The merge will be a key prefix on object storage. The `merge` is something similar to a path in a file system. \nIt is very important to set a good `merge` parameter and remember the result, because results can only be accessed when the `merge` is known. Furthermore, all users of the workspace (in the case of the `apex-force-results-workspace`, anyone) may write data, so make sure to use a unique `merge`, to reduce the likelihood of accidentally modifying others' data and have others overwrite your data by accident.\n\n\n:::{.callout-tip}\n# FORCE workspace\n\nThe dedicated workspace for FORCE results is accessible to all users:\n`apex-force-results-workspace` ()\n:::\n\n\nWe generate the `force_level2` process graph using a Process Graph Node ([`PGNode`](https://open-eo.github.io/openeo-python-client/api.html#openeo.internal.graph_building.PGNode)). In order to interpret the backend response, we wrap the resulting process graph as a [`StacResource`](https://open-eo.github.io/openeo-python-client/api.html#openeo.rest.stac_resource.StacResource).\nThis makes it possible to render the backend response nicely in Jupyter environments and gives us convenient methods to access our results when the process is done.\n\nPass [parameters](parametrization.qmd) to the level 2 processing system in the `arguments` dictionary.\nYou may give the processing a name which will determine the name of the STAC item generated for the results using the parameter `name`.\n\n\n#### Area of extent\n\nAnother parameter which is handled in a special way (i.e., not passed directly to FORCE) is `aoi`. In FORCE level 2, you can specify the `FILE_AOI` parameter to clip the output to a certain extent.\nInstead of a file, `aoi` takes a geojson object to specify area of interest.\n\n- The `aoi` parameter controls the **area of the output produced by FORCE**. The `spatial_extent` parameter passed to the query limits the **input files**\n- `aoi` is applied after radiometric correction (so it will not reduce the computational effort) and before tiling [as per the FORCE documentation](https://force-eo.readthedocs.io/en/latest/components/lower-level/level2/param.html) (description of parameter `FILE_AOI`)\n- By default, the output will cover all of the inputs, without clipping.\n\n::: {#be232338 .cell execution_count=7}\n``` {.python .cell-code}\nw, s, e, n = spatial_extent[\"west\"], spatial_extent[\"south\"], spatial_extent[\"east\"], spatial_extent[\"north\"]\naoi = f'{{ \"type\": \"Feature\", \"geometry\": {{ \"type\": \"Polygon\", \"coordinates\": [[[{w},{s}],[{w},{n}],[{e},{n}],[{e},{s}],[{w},{s}]]] }}, \"properties\": {{ \"name\": \"FORCE test\" }} }}'\n```\n:::\n\n\n:::{.panel-tabset group=\"query_engine\"}\n\n#### openEO\n\nWhen using the `query_stac` process, we pass the **contents** of a STAC document to the process (not a reference). To do this, pass the `process_graph` directly as `stac_document`\n\n:::{.callout-warning}\nDue to backend limitations, the `stac_document` parameter can only handle a document of limited length. The `query_stac` process generates an item collection which quickly grows in size (the entire content of each STAC item is contained in it). Therefore, use `query_stac` only for small processing tasks with only a handful of items. Otherwise use a different query method to pass an URL or a catalog instead.\n\n**It is recommended to rely on Pystac client for catalog queries for now.**\n:::\n\n::: {#5eed5b0f .cell execution_count=8}\n``` {.python .cell-code}\nfrom openeo.rest.stac_resource import StacResource\nfrom openeo.internal.graph_building import PGNode\n\nprocessing_name = \"Modena\"\nforce_l2_stac_resource = StacResource(\n graph=PGNode(\n process_id=\"force_level2\",\n arguments={\n \"stac_document\": query_pg,\n \"name\": processing_name,\n \"aoi\": aoi,\n \"do_brdf\": True,\n }\n ),\n connection=connection,\n)\n\n# Optional when only processing level 2\n# If you intend to continue processing, or share the results with others,\n# add the `export_workspace` call:\nimport getpass\n\n# Be careful: The merge determines the storage location of your results.\n# You may overwrite your results or those of others, if the `merge` parameter is not\n# Set to a unique value.\nmerge = f\"{getpass.getuser()}/experiments/{processing_name}\"\nprint(\"merge: \", merge)\nforce_l2_stac_resource = force_l2_stac_resource.export_workspace(\n workspace=\"apex-force-results-workspace\",\n merge=merge,\n)\n```\n\n::: {.cell-output .cell-output-stdout}\n```\nmerge: hannes/experiments/Modena\n```\n:::\n:::\n\n\n\n\n#### Pystac client {.active}\n\n:::{.callout-warning}\n\nDue to backend limitations the `stac_document` parameter can only handle a document of limited length. This means it can only handle collections of very few items.\nIt is therefore advisable to either pass an URL (if possible) or to convert the item_collection returned by [`pystac_client.Client.search`](https://pystac-client.readthedocs.io/en/latest/api.html#pystac_client.Client.search) to a catalog. The item collection contains the entire content of each STAC item, whereas the catalog only contains a reference to the items, making a much shorter document.\n\n:::\n\nIn case the backend limitation will be fixed, the transformation step will no longer be necessary.\nIt is already possible to pass `item_collection.to_dict()` directly as `stac_document`, if the item collection is small.\n\n##### Convert Item Collection to Catalog\n\n::: {#25094442 .cell execution_count=10}\n``` {.python .cell-code}\nimport pystac\n\ndef transform_item_collection_to_catalog_with_links(item_collection):\n catalog = pystac.Catalog(\n id=\"item-collection-catalog\",\n description=\"Catalog from item collection\",\n )\n for item in item_collection.items:\n item_link = pystac.Link(\n rel=pystac.RelType.ITEM,\n target=item.self_href,\n media_type=pystac.MediaType.GEOJSON,\n title=f\"Item {item.id}\",\n )\n catalog.add_link(item_link)\n return catalog\n\nquery_catalog = transform_item_collection_to_catalog_with_links(item_collection)\n```\n:::\n\n\n##### Process Graph\n\n::: {#26d74c23 .cell execution_count=11}\n``` {.python .cell-code}\nfrom openeo.rest.stac_resource import StacResource\nfrom openeo.internal.graph_building import PGNode\n\nforce_l2_stac_resource = StacResource(\n graph=PGNode(\n process_id=\"force_level2\",\n arguments={\n \"stac_document\": query_catalog.to_dict(),\n \"name\": processing_name,\n \"do_brdf\": True,\n }\n ),\n connection=connection,\n)\n\n# Temporary fix\nforce_l2_stac_resource = StacResource(\n graph=PGNode(\n process_id=\"run_cwl_to_stac\",\n arguments=dict(\n cwl=\"https://github.com/bcdev/apex-force-openeo/releases/latest/download/force-level2.cwl\",\n context={\n \"stac_document\": query_catalog.to_dict(),\n \"name\": processing_name,\n \"do_brdf\": True,\n }\n ),\n ),\n connection=connection,\n)\n# Optional when only processing level 2\n# If you intend to continue processing, or share the results with others,\n# add the `export_workspace` call:\nimport getpass\n\n# Be careful: The merge determines the storage location of your results.\n# You may overwrite your results or those of others, if the `merge` parameter is not\n# Set to a unique value.\nmerge = f\"{getpass.getuser()}/experiments/{processing_name}\"\nprint(\"merge: \", merge)\nforce_l2_stac_resource = force_l2_stac_resource.export_workspace(\n workspace=\"apex-force-results-workspace\",\n merge=merge,\n)\n```\n\n::: {.cell-output .cell-output-stdout}\n```\nmerge: hannes/experiments/Modena\n```\n:::\n:::\n\n\n\n\n:::\n\n\nYou can (optionally) inspect the process graph to visualize it.\n\n:::{.callout-note title=\"Process Graph JSON\" collapse=\"true\"}\n\n:::{.panel-tabset group=\"query_engine\"}\n\n### openEO\n\n\n\n::: {#1da7be58 .cell execution_count=14}\n``` {.python .cell-code}\nprint(force_l2_stac_resource.to_json())\n```\n\n::: {.cell-output .cell-output-stdout}\n```\n{\n \"process_graph\": {\n \"querystac1\": {\n \"process_id\": \"query_stac\",\n \"arguments\": {\n \"spatial_extent\": {\n \"west\": 11.0,\n \"south\": 44.5,\n \"east\": 11.1,\n \"north\": 44.6\n },\n \"temporal_extent\": [\n \"2026-04-17\",\n \"2026-04-27\"\n ],\n \"url\": \"https://stac.dataspace.copernicus.eu/v1/collections/sentinel-2-l1c\"\n }\n },\n \"forcelevel21\": {\n \"process_id\": \"force_level2\",\n \"arguments\": {\n \"aoi\": \"{ \\\"type\\\": \\\"Feature\\\", \\\"geometry\\\": { \\\"type\\\": \\\"Polygon\\\", \\\"coordinates\\\": [[[11.0,44.5],[11.0,44.6],[11.1,44.6],[11.1,44.5],[11.0,44.5]]] }, \\\"properties\\\": { \\\"name\\\": \\\"FORCE test\\\" } }\",\n \"do_brdf\": true,\n \"name\": \"Modena\",\n \"stac_document\": {\n \"from_node\": \"querystac1\"\n }\n }\n },\n \"exportworkspace1\": {\n \"process_id\": \"export_workspace\",\n \"arguments\": {\n \"data\": {\n \"from_node\": \"forcelevel21\"\n },\n \"merge\": \"hannes/experiments/Modena\",\n \"workspace\": \"apex-force-results-workspace\"\n },\n \"result\": true\n }\n }\n}\n```\n:::\n:::\n\n\n### Pystac client {.active}\n\n\n\n::: {#257249b8 .cell execution_count=16}\n``` {.python .cell-code}\nprint(force_l2_stac_resource.to_json())\n```\n\n::: {.cell-output .cell-output-stdout}\n```\n{\n \"process_graph\": {\n \"runcwltostac1\": {\n \"process_id\": \"run_cwl_to_stac\",\n \"arguments\": {\n \"context\": {\n \"stac_document\": {\n \"type\": \"Catalog\",\n \"id\": \"item-collection-catalog\",\n \"stac_version\": \"1.1.0\",\n \"description\": \"Catalog from item collection\",\n \"links\": [\n {\n \"rel\": \"item\",\n \"href\": \"https://stac.dataspace.copernicus.eu/v1/collections/sentinel-2-l1c/items/S2C_MSIL1C_20260427T101021_N0512_R022_T32TPQ_20260427T142022\",\n \"type\": \"application/geo+json\",\n \"title\": \"Item S2C_MSIL1C_20260427T101021_N0512_R022_T32TPQ_20260427T142022\"\n },\n {\n \"rel\": \"item\",\n \"href\": \"https://stac.dataspace.copernicus.eu/v1/collections/sentinel-2-l1c/items/S2B_MSIL1C_20260422T101019_N0512_R022_T32TPQ_20260422T134442\",\n \"type\": \"application/geo+json\",\n \"title\": \"Item S2B_MSIL1C_20260422T101019_N0512_R022_T32TPQ_20260422T134442\"\n },\n {\n \"rel\": \"item\",\n \"href\": \"https://stac.dataspace.copernicus.eu/v1/collections/sentinel-2-l1c/items/S2A_MSIL1C_20260419T100711_N0512_R022_T32TPQ_20260419T152521\",\n \"type\": \"application/geo+json\",\n \"title\": \"Item S2A_MSIL1C_20260419T100711_N0512_R022_T32TPQ_20260419T152521\"\n },\n {\n \"rel\": \"item\",\n \"href\": \"https://stac.dataspace.copernicus.eu/v1/collections/sentinel-2-l1c/items/S2C_MSIL1C_20260417T101021_N0512_R022_T32TPQ_20260417T134842\",\n \"type\": \"application/geo+json\",\n \"title\": \"Item S2C_MSIL1C_20260417T101021_N0512_R022_T32TPQ_20260417T134842\"\n }\n ]\n },\n \"name\": \"Modena\",\n \"do_brdf\": true\n },\n \"cwl\": \"https://github.com/bcdev/apex-force-openeo/releases/latest/download/force-level2.cwl\"\n }\n },\n \"exportworkspace1\": {\n \"process_id\": \"export_workspace\",\n \"arguments\": {\n \"data\": {\n \"from_node\": \"runcwltostac1\"\n },\n \"merge\": \"hannes/experiments/Modena\",\n \"workspace\": \"apex-force-results-workspace\"\n },\n \"result\": true\n }\n }\n}\n```\n:::\n:::\n\n\n\n:::\n\n:::\n\n\n### Executing the Process Graph\n\nWith the process graph in place, we can submit it and wait for it to complete.\nYou can [monitor the progress](processes.qmd#monitoring-progress) through the client library or the web interface.\n\n\n```{.python}\nl2_job = force_l2_stac_resource.create_job(title=f\"FORCE level 2 {processing_name}\")\nl2_job.start() # use start_and_wait instead to block until the job is done\n```\n\n## Accessing results\n\nWhen the job has completed, we can inspect the generated STAC metadata of the FORCE data cube and download it.\n\n**Downloading is optional**, if you wish to proceed with [higher level processing](tsa.qmd). If your processing chain is complete after the level 2 module, you may download the results to continue your analysis.\n\nThe data cube is persistent on cloud storage, so you may download it at a later time or run multiple higher level processes with the same level 2 cube.\nThe data cube is referenced by the URL to its STAC catalog (or item) when passing it to other processes.\n\n\n\n:::{.callout-warning}\nWhile the cube will stay in place for multiple days, eventually the storage will be cleared. So please **make sure to download your processing results for long term storage**.\n:::\n\n\n\n\n\n#### Inspecting STAC metadata {#sec-inspecting-stac}\n\n:::{.callout-tip}\nThis step is optional\n:::\n\nThe basic URL of the FORCE workspace `apex-force-results-workspace` is \n. Together with the `merge` path defined for our processing, we can determine the path to the `catalog.json` for our FORCE level results:\n\n```{.python}\nfrom pystac.catalog import Catalog\n\nWORKSPACE_URL = \"https://s3.waw4-1.cloudferro.com/apex-force-results-waw4-1-exotc5yuexi2c5tvwqhoivj62fz8v0uupy0me\"\n\nl2_catalog_url = f\"{WORKSPACE_URL}/{merge}/catalog.json\"\nl2_catalog = Catalog.from_file(l2_catalog_url)\n\nl2_catalog.set_self_href(l2_catalog_url)\nl2_catalog\n```\n\n\n\n\n\n#### Downloading files\n\n:::{.callout-tip}\nThis step is optional if you plan to continue processing with FORCE on CDSE\n:::\n\nAssets can be downloaded using the standard openEO mechanism to access result assets.\nThe STAC metadata specifies the directory structure of the resulting cube. This structure is respected by openEO's [`download_files`](https://open-eo.github.io/openeo-python-client/api.html#openeo.rest.job.JobResults.download_files). \n\nTo inspect a particular asset, we may use [`download_file`](https://open-eo.github.io/openeo-python-client/api.html#openeo.rest.job.JobResults.download_file) to download singular files instead.\n\n```{.python}\nl2_results = l2_job.get_results()\nl2_results\n```\n\n```{.python}\nl2_results.download_files(\"force-level2-results\")\n```\n\n\n\n", + "markdown": "---\ntitle: FORCE level 2\norder: 4\n---\n\n## Recap\n\n:::{.callout-tip collapse=\"true\" title=\"Selecting input products\"}\n\n::: {#f1d1ba5d .cell execution_count=1}\n``` {.python .cell-code}\nL1C_COLLECTION_URL = \"https://stac.dataspace.copernicus.eu/v1/collections/sentinel-2-l1c\"\nSTAC_ROOT_URL = \"https://stac.dataspace.copernicus.eu/v1\"\nw, s, e, n = 11.0, 44.5, 11.1, 44.6\nspatial_extent = { \"west\": w, \"south\": s, \"east\": e, \"north\": n}\n# ATTENTION: Inclusion of the second date in the search depends on the query method.\ntemporal_extent=[\"2026-04-17\", \"2026-04-27\"]\n```\n:::\n\n\n:::{.panel-tabset group=\"query_engine\"}\n\n## openEO\n\n::: {#aebefb80 .cell execution_count=2}\n``` {.python .cell-code}\nimport openeo\nconnection = openeo.connect(\"openeo.dataspace.copernicus.eu\").authenticate_oidc()\n\nquery_pg = openeo.processes.process(\n \"query_stac\",\n arguments={\n \"url\": L1C_COLLECTION_URL,\n \"temporal_extent\": temporal_extent,\n \"spatial_extent\": spatial_extent, }\n)\n```\n\n::: {.cell-output .cell-output-stdout}\n```\nAuthenticated using refresh token.\n```\n:::\n:::\n\n\n## Pystac client {.active}\n\n::: {#857e85ab .cell execution_count=3}\n``` {.python .cell-code}\nimport pystac_client\n\nclient = pystac_client.Client.open(STAC_ROOT_URL)\nsearch = client.search(\n datetime=temporal_extent,\n collections=[\"sentinel-2-l1c\"],\n bbox=[w, s, e, n]\n)\nitem_collection = search.item_collection()\n```\n:::\n\n\n \n:::\n \n:::\n\n## Level 2 processing\n\n\nThe [openeo Process](processes.qmd) is straightforwardly named `force_level2`. It takes most of the parameters that can be specified in the FORCE level 2 parameter file. See the article on [parametrization](parametrization.qmd) for more details which parameters are (not) exposed by the process.\n\n:::{.callout-tip}\nThe FORCE parameter file format uses [`SCREAMING_SNAKE_CASE`](https://en.wikipedia.org/wiki/screaming_snake_case) to name its parameters.\nThe integrated FORCE uses (lowercase) [`snake_case`](https://en.wikipedia.org/wiki/Snake_case) instead, to be consistent with openEO conventions. Otherwise, the parameter names are unchanged, so it is straightforward to determine the openEO name if you are familiar with the FORCE parameters.\n:::\n\nThe parameters are documented in the process description. We can inspect it using the Python client's [`describe_processes()`](https://open-eo.github.io/openeo-python-client/api.html#openeo.rest.connection.Connection.describe_process).\n\n\n\n::: {#edd19cd5 .cell execution_count=4}\n``` {.python .cell-code}\nconnection = openeo.connect(\"openeo.dataspace.copernicus.eu\").authenticate_oidc()\n```\n\n::: {.cell-output .cell-output-stdout}\n```\nAuthenticated using refresh token.\n```\n:::\n:::\n\n\n\n\n::: {#fe9fda67 .cell execution_count=5}\n\n::: {.cell-output .cell-output-stdout}\n```\nAuthenticated using refresh token.\n```\n:::\n:::\n\n\n:::{.callout-note collapse=\"true\" title=\"FORCE level 2 parameters\"}\n\n::: {#10af29da .cell execution_count=6}\n``` {.python .cell-code}\nconnection.describe_process(\"force_level2\")\n```\n\n::: {.cell-output .cell-output-display execution_count=48}\n```{=html}\n\n \n \n \n \n \n```\n:::\n:::\n\n\n:::\n\n::: {#1b1b2332 .cell execution_count=7}\n\n::: {.cell-output .cell-output-stdout}\n```\nAuthenticated using refresh token.\n```\n:::\n:::\n\n\n### Building the Process Graph\n\nThe FORCE process graph is based on the `force_level2` process, as described in [openEO processes](processes.qmd). Because FORCE does not use the standard openEO version of the \"data cube\" concept, but\nuses its own data cube representation, we need to wrap the process so that we can interact with the results in a convenient way as a `StacResource` and export the processing results to a workspace.\n\n\nConcept | Qualified name (Python client) | Purpose\n--------|--------------------------------|-------------\n`StacResource` | [`openeo.rest.stac_resource.StacResource`](https://force-eo.readthedocs.io/en/latest/components/higher-level/tsa/) | A handle to the process graph results, makes it possible to inspect the results as usual for openEO datacubes.\n`export_workspace` | [`openeo.rest.result.SaveResult.export_workspace`](https://open-eo.github.io/openeo-python-client/api.html#openeo.rest.result.SaveResult.export_workspace) | A process to publish the processing results (e.g., a FORCE ARD data cube) to a shared cloud storage location. There is a special workspace configured for FORCE (`apex-force-force-results`) accessible to all users of the CDSE openEO backend. The workspace serves as a storage location to keep data cubes for later processing (e.g., with the [TSA](tsa.qmd) module).\n`PGNode` | [`openeo.internal.graph_building.PGNode`](https://open-eo.github.io/openeo-python-client/api.html#openeo.internal.graph_building.PGNode) | A single node in a process graph. `PGNode` is an internal class but may be used for advanced graph building purposes [as described here](https://open-eo.github.io/openeo-python-client/processes.html#callback-as-pgnode). We use it for the `graph` parameter of `StacResource`.\n\n\n#### Workspaces\n\nOpenEO uses the concept of [Workspaces](https://docs.openeo.cloud/workspaces/) to interface with object storage. For processing with FORCE, we can use a workspace dedicated to FORCE to store and exchange processing results between level 2 and higher level processing, and with colleagues.\n\nWe can use either the dedicated workspace `apex-force-results-workspace`, which is accessible to all CDSE openeo users, or another registered workspace. See the [openEO platform documentation on workspaces](https://docs.openeo.cloud/workspaces/#registering-a-workspace) on how workspaces can be accessed and registered.\n\nWhen running FORCE level2 on its own, we do not need to use a workspace. Simply download the results when you are done with processing. However, to reference the level 2 processing results in [further processing](tsa.qmd), we need to use a workspace.\n\nWe can instruct openEO to store our results in a workspace using the [`export_workspace`](https://openeo.org/documentation/1.0/processes.html#export_workspace) openEO process.\n\nWhen using `export_workspace`, we need to pass a `merge` string to identify your result. The merge will be a key prefix on object storage. The `merge` is something similar to a path in a file system.\nIt is very important to set a good `merge` parameter and remember the result, because results can only be accessed when the `merge` is known. Furthermore, all users of the workspace (in the case of the `apex-force-results-workspace`, anyone) may write data, so make sure to use a unique `merge`, to reduce the likelihood of accidentally modifying others' data and have others overwrite your data by accident.\n\n\n:::{.callout-tip}\n# FORCE workspace\n\nThe dedicated workspace for FORCE results is accessible to all users:\n\n- id: `apex-force-results-workspace`\n- url: https://s3.waw4-1.cloudferro.com/apex-force-results-waw4-1-exotc5yuexi2c5tvwqhoivj62fz8v0uupy0me\n:::\n\n\nWe generate the `force_level2` process graph using a Process Graph Node ([`PGNode`](https://open-eo.github.io/openeo-python-client/api.html#openeo.internal.graph_building.PGNode)). In order to interpret the backend response, we wrap the resulting process graph as a [`StacResource`](https://open-eo.github.io/openeo-python-client/api.html#openeo.rest.stac_resource.StacResource).\nThis makes it possible to render the backend response nicely in Jupyter environments and gives us convenient methods to access our results when the process is done.\n\nPass [parameters](parametrization.qmd) to the level 2 processing system in the `arguments` dictionary.\nYou may give the processing a name which will determine the name of the STAC item generated for the results using the parameter `name`.\n\n\n#### Area of extent\n\nAnother parameter which is handled in a special way (i.e., not passed directly to FORCE) is `aoi`. In FORCE level 2, you can specify the `FILE_AOI` parameter to clip the output to a certain extent.\nInstead of a file, `aoi` takes a geojson object to specify area of interest.\n\n- The `aoi` parameter controls the **area of the output produced by FORCE**. The `spatial_extent` parameter passed to the query limits the **input files**\n- `aoi` is applied after radiometric correction (so it will not reduce the computational effort) and before tiling [as per the FORCE documentation](https://force-eo.readthedocs.io/en/latest/components/lower-level/level2/param.html) (description of parameter `FILE_AOI`)\n- By default, the output will cover all of the inputs, without clipping.\n\n::: {#eb662174 .cell execution_count=8}\n``` {.python .cell-code}\nw, s, e, n = spatial_extent[\"west\"], spatial_extent[\"south\"], spatial_extent[\"east\"], spatial_extent[\"north\"]\naoi = f'{{ \"type\": \"Feature\", \"geometry\": {{ \"type\": \"Polygon\", \"coordinates\": [[[{w},{s}],[{w},{n}],[{e},{n}],[{e},{s}],[{w},{s}]]] }}, \"properties\": {{ \"name\": \"FORCE test\" }} }}'\n```\n:::\n\n\n:::{.panel-tabset group=\"query_engine\"}\n\n#### openEO\n\nWhen using the `query_stac` process, we pass the **contents** of a STAC document to the process (not a reference). To do this, pass the `process_graph` directly as `stac_document`\n\n:::{.callout-warning}\nDue to backend limitations, the `stac_document` parameter can only handle a document of limited length. The `query_stac` process generates an item collection which quickly grows in size (the entire content of each STAC item is contained in it). Therefore, use `query_stac` only for small processing tasks with only a handful of items. Otherwise use a different query method to pass an URL or a catalog instead.\n\n**It is recommended to rely on Pystac client for catalog queries for now.**\n:::\n\n::: {#71ff8990 .cell execution_count=9}\n``` {.python .cell-code}\nfrom openeo.rest.stac_resource import StacResource\nfrom openeo.internal.graph_building import PGNode\n\nprocessing_name = \"Modena\"\nforce_l2_stac_resource = StacResource(\n graph=PGNode(\n process_id=\"force_level2\",\n arguments={\n \"stac_document\": query_pg,\n \"name\": processing_name,\n \"aoi\": aoi,\n \"do_brdf\": True,\n }\n ),\n connection=connection,\n)\n\n# Optional when only processing level 2\n# If you intend to continue processing, or share the results with others,\n# add the `export_workspace` call:\nimport getpass\n\n# Be careful: The merge determines the storage location of your results.\n# You may overwrite your results or those of others, if the `merge` parameter is not\n# Set to a unique value.\nmerge = f\"{getpass.getuser()}/experiments/{processing_name}\"\nprint(\"merge: \", merge)\nforce_l2_stac_resource = force_l2_stac_resource.export_workspace(\n workspace=\"apex-force-results-workspace\",\n merge=merge,\n)\n```\n\n::: {.cell-output .cell-output-stdout}\n```\nmerge: hannes/experiments/Modena\n```\n:::\n:::\n\n\n\n\n#### Pystac client {.active}\n\n:::{.callout-warning}\n\nDue to backend limitations the `stac_document` parameter can only handle a document of limited length. This means it can only handle collections of very few items.\nIt is therefore advisable to either pass an URL (if possible) or to convert the item_collection returned by [`pystac_client.Client.search`](https://pystac-client.readthedocs.io/en/latest/api.html#pystac_client.Client.search) to a catalog. The item collection contains the entire content of each STAC item, whereas the catalog only contains a reference to the items, making a much shorter document.\n\n:::\n\nIn case the backend limitation will be fixed, the transformation step will no longer be necessary.\nIt is already possible to pass `item_collection.to_dict()` directly as `stac_document`, if the item collection is small.\n\n##### Convert Item Collection to Catalog\n\n::: {#9388c615 .cell execution_count=11}\n``` {.python .cell-code}\nimport pystac\n\ndef transform_item_collection_to_catalog_with_links(item_collection):\n catalog = pystac.Catalog(\n id=\"item-collection-catalog\",\n description=\"Catalog from item collection\",\n )\n for item in item_collection.items:\n item_link = pystac.Link(\n rel=pystac.RelType.ITEM,\n target=item.self_href,\n media_type=pystac.MediaType.GEOJSON,\n title=f\"Item {item.id}\",\n )\n catalog.add_link(item_link)\n return catalog\n\nquery_catalog = transform_item_collection_to_catalog_with_links(item_collection)\n```\n:::\n\n\n##### Process Graph\n\n::: {#1fa26431 .cell execution_count=12}\n``` {.python .cell-code}\nfrom openeo.rest.stac_resource import StacResource\nfrom openeo.internal.graph_building import PGNode\n\nforce_l2_stac_resource = StacResource(\n graph=PGNode(\n process_id=\"force_level2\",\n arguments={\n \"stac_document\": query_catalog.to_dict(),\n \"name\": processing_name,\n \"do_brdf\": True,\n }\n ),\n connection=connection,\n)\n\n# Temporary fix\nforce_l2_stac_resource = StacResource(\n graph=PGNode(\n process_id=\"run_cwl_to_stac\",\n arguments=dict(\n cwl=\"https://github.com/bcdev/apex-force-openeo/releases/latest/download/force-level2.cwl\",\n context={\n \"stac_document\": query_catalog.to_dict(),\n \"name\": processing_name,\n \"do_brdf\": True,\n }\n ),\n ),\n connection=connection,\n)\n# Optional when only processing level 2\n# If you intend to continue processing, or share the results with others,\n# add the `export_workspace` call:\nimport getpass\n\n# Be careful: The merge determines the storage location of your results.\n# You may overwrite your results or those of others, if the `merge` parameter is not\n# Set to a unique value.\nmerge = f\"{getpass.getuser()}/experiments/{processing_name}\"\nprint(\"merge: \", merge)\nforce_l2_stac_resource = force_l2_stac_resource.export_workspace(\n workspace=\"apex-force-results-workspace\",\n merge=merge,\n)\n```\n\n::: {.cell-output .cell-output-stdout}\n```\nmerge: hannes/experiments/Modena\n```\n:::\n:::\n\n\n\n\n:::\n\n\nYou can (optionally) inspect the process graph to visualize it.\n\n:::{.callout-note title=\"Process Graph JSON\" collapse=\"true\"}\n\n:::{.panel-tabset group=\"query_engine\"}\n\n### openEO\n\n\n\n::: {#b29ec56b .cell execution_count=15}\n``` {.python .cell-code}\nprint(force_l2_stac_resource.to_json())\n```\n\n::: {.cell-output .cell-output-stdout}\n```\n{\n \"process_graph\": {\n \"querystac1\": {\n \"process_id\": \"query_stac\",\n \"arguments\": {\n \"spatial_extent\": {\n \"west\": 11.0,\n \"south\": 44.5,\n \"east\": 11.1,\n \"north\": 44.6\n },\n \"temporal_extent\": [\n \"2026-04-17\",\n \"2026-04-27\"\n ],\n \"url\": \"https://stac.dataspace.copernicus.eu/v1/collections/sentinel-2-l1c\"\n }\n },\n \"forcelevel21\": {\n \"process_id\": \"force_level2\",\n \"arguments\": {\n \"aoi\": \"{ \\\"type\\\": \\\"Feature\\\", \\\"geometry\\\": { \\\"type\\\": \\\"Polygon\\\", \\\"coordinates\\\": [[[11.0,44.5],[11.0,44.6],[11.1,44.6],[11.1,44.5],[11.0,44.5]]] }, \\\"properties\\\": { \\\"name\\\": \\\"FORCE test\\\" } }\",\n \"do_brdf\": true,\n \"name\": \"Modena\",\n \"stac_document\": {\n \"from_node\": \"querystac1\"\n }\n }\n },\n \"exportworkspace1\": {\n \"process_id\": \"export_workspace\",\n \"arguments\": {\n \"data\": {\n \"from_node\": \"forcelevel21\"\n },\n \"merge\": \"hannes/experiments/Modena\",\n \"workspace\": \"apex-force-results-workspace\"\n },\n \"result\": true\n }\n }\n}\n```\n:::\n:::\n\n\n### Pystac client {.active}\n\n\n\n::: {#9af5ca11 .cell execution_count=17}\n``` {.python .cell-code}\nprint(force_l2_stac_resource.to_json())\n```\n\n::: {.cell-output .cell-output-stdout}\n```\n{\n \"process_graph\": {\n \"runcwltostac1\": {\n \"process_id\": \"run_cwl_to_stac\",\n \"arguments\": {\n \"context\": {\n \"stac_document\": {\n \"type\": \"Catalog\",\n \"id\": \"item-collection-catalog\",\n \"stac_version\": \"1.1.0\",\n \"description\": \"Catalog from item collection\",\n \"links\": [\n {\n \"rel\": \"item\",\n \"href\": \"https://stac.dataspace.copernicus.eu/v1/collections/sentinel-2-l1c/items/S2C_MSIL1C_20260427T101021_N0512_R022_T32TPQ_20260427T142022\",\n \"type\": \"application/geo+json\",\n \"title\": \"Item S2C_MSIL1C_20260427T101021_N0512_R022_T32TPQ_20260427T142022\"\n },\n {\n \"rel\": \"item\",\n \"href\": \"https://stac.dataspace.copernicus.eu/v1/collections/sentinel-2-l1c/items/S2B_MSIL1C_20260422T101019_N0512_R022_T32TPQ_20260422T134442\",\n \"type\": \"application/geo+json\",\n \"title\": \"Item S2B_MSIL1C_20260422T101019_N0512_R022_T32TPQ_20260422T134442\"\n },\n {\n \"rel\": \"item\",\n \"href\": \"https://stac.dataspace.copernicus.eu/v1/collections/sentinel-2-l1c/items/S2A_MSIL1C_20260419T100711_N0512_R022_T32TPQ_20260419T152521\",\n \"type\": \"application/geo+json\",\n \"title\": \"Item S2A_MSIL1C_20260419T100711_N0512_R022_T32TPQ_20260419T152521\"\n },\n {\n \"rel\": \"item\",\n \"href\": \"https://stac.dataspace.copernicus.eu/v1/collections/sentinel-2-l1c/items/S2C_MSIL1C_20260417T101021_N0512_R022_T32TPQ_20260417T134842\",\n \"type\": \"application/geo+json\",\n \"title\": \"Item S2C_MSIL1C_20260417T101021_N0512_R022_T32TPQ_20260417T134842\"\n }\n ]\n },\n \"name\": \"Modena\",\n \"do_brdf\": true\n },\n \"cwl\": \"https://github.com/bcdev/apex-force-openeo/releases/latest/download/force-level2.cwl\"\n }\n },\n \"exportworkspace1\": {\n \"process_id\": \"export_workspace\",\n \"arguments\": {\n \"data\": {\n \"from_node\": \"runcwltostac1\"\n },\n \"merge\": \"hannes/experiments/Modena\",\n \"workspace\": \"apex-force-results-workspace\"\n },\n \"result\": true\n }\n }\n}\n```\n:::\n:::\n\n\n\n:::\n\n:::\n\n\n### Executing the Process Graph\n\nWith the process graph in place, we can submit it and wait for it to complete.\nYou can [monitor the progress](processes.qmd#monitoring-progress) through the client library or the web interface.\n\n\n```{.python}\nl2_job = force_l2_stac_resource.create_job(title=f\"FORCE level 2 {processing_name}\")\nl2_job.start() # use start_and_wait instead to block until the job is done\n```\n\n## Accessing results\n\nWhen the job has completed, we can inspect the generated STAC metadata of the FORCE data cube and download it.\n\n**Downloading is optional**, if you wish to proceed with [higher level processing](tsa.qmd). If your processing chain is complete after the level 2 module, you may download the results to continue your analysis.\n\nThe data cube is persistent on cloud storage, so you may download it at a later time or run multiple higher level processes with the same level 2 cube.\nThe data cube is referenced by the URL to its STAC catalog (or item) when passing it to other processes.\n\n\n\n:::{.callout-warning}\nWhile the cube will stay in place for multiple days, eventually the storage will be cleared. So please **make sure to download your processing results for long term storage**.\n:::\n\n\n\n\n\n#### Inspecting STAC metadata {#sec-inspecting-stac}\n\n:::{.callout-tip}\nThis step is optional\n:::\n\nThe basic URL of the FORCE workspace `apex-force-results-workspace` is\n.\nTogether with the `merge` path defined for our processing, we can determine the path to the `catalog.json` for our FORCE level results:\n\n::: {#b8d19dab .cell execution_count=19}\n``` {.python .cell-code}\nfrom pystac.catalog import Catalog\n\nWORKSPACE_URL = \"https://s3.waw4-1.cloudferro.com/apex-force-results-waw4-1-exotc5yuexi2c5tvwqhoivj62fz8v0uupy0me\"\n\nl2_catalog_url = f\"{WORKSPACE_URL}/{merge}/catalog.json\"\nl2_catalog = Catalog.from_file(l2_catalog_url)\n\nl2_catalog.set_self_href(l2_catalog_url)\nl2_catalog\n```\n\n::: {.cell-output .cell-output-display execution_count=61}\n```{=html}\n\n\n\n
\n
\n <Catalog id=catalog>\n
\n\n
\n```\n:::\n:::\n\n\n#### Downloading files\n\n:::{.callout-tip}\nThis step is optional if you plan to continue processing with FORCE on CDSE\n:::\n\nAssets can be downloaded using the standard openEO mechanism to access result assets.\nThe STAC metadata specifies the directory structure of the resulting cube. This structure is respected by openEO's [`download_files`](https://open-eo.github.io/openeo-python-client/api.html#openeo.rest.job.JobResults.download_files).\n\nTo inspect a particular asset, we may use [`download_file`](https://open-eo.github.io/openeo-python-client/api.html#openeo.rest.job.JobResults.download_file) to download singular files instead.\n\n::: {#81b10f7e .cell execution_count=20}\n``` {.python .cell-code}\nl2_results = l2_job.get_results()\nl2_results\n```\n\n::: {.cell-output .cell-output-display execution_count=62}\n```{=html}\n\n \n \n \n \n \n```\n:::\n:::\n\n\n::: {#5cc02746 .cell execution_count=21}\n``` {.python .cell-code}\nl2_results.download_files(\"force-level2-results\")\n```\n\n::: {.cell-output .cell-output-display execution_count=63}\n```\n[PosixPath('force-level2-results/CITEME_0x6e.txt'),\n PosixPath('force-level2-results/X0031_Y0029/20260417_LEVEL2_SEN2C_BOA.tif'),\n PosixPath('force-level2-results/X0031_Y0029/20260417_LEVEL2_SEN2C_OVV.jpg'),\n PosixPath('force-level2-results/X0031_Y0029/20260417_LEVEL2_SEN2C_QAI.tif'),\n PosixPath('force-level2-results/X0031_Y0029/20260419_LEVEL2_SEN2A_BOA.tif'),\n PosixPath('force-level2-results/X0031_Y0029/20260419_LEVEL2_SEN2A_OVV.jpg'),\n PosixPath('force-level2-results/X0031_Y0029/20260419_LEVEL2_SEN2A_QAI.tif'),\n PosixPath('force-level2-results/X0031_Y0029/20260422_LEVEL2_SEN2B_BOA.tif'),\n PosixPath('force-level2-results/X0031_Y0029/20260422_LEVEL2_SEN2B_OVV.jpg'),\n PosixPath('force-level2-results/X0031_Y0029/20260422_LEVEL2_SEN2B_QAI.tif'),\n PosixPath('force-level2-results/X0031_Y0029/20260427_LEVEL2_SEN2C_BOA.tif'),\n PosixPath('force-level2-results/X0031_Y0029/20260427_LEVEL2_SEN2C_OVV.jpg'),\n PosixPath('force-level2-results/X0031_Y0029/20260427_LEVEL2_SEN2C_QAI.tif'),\n PosixPath('force-level2-results/X0031_Y0030/20260417_LEVEL2_SEN2C_BOA.tif'),\n PosixPath('force-level2-results/X0031_Y0030/20260417_LEVEL2_SEN2C_OVV.jpg'),\n PosixPath('force-level2-results/X0031_Y0030/20260417_LEVEL2_SEN2C_QAI.tif'),\n PosixPath('force-level2-results/X0031_Y0030/20260419_LEVEL2_SEN2A_BOA.tif'),\n PosixPath('force-level2-results/X0031_Y0030/20260419_LEVEL2_SEN2A_OVV.jpg'),\n PosixPath('force-level2-results/X0031_Y0030/20260419_LEVEL2_SEN2A_QAI.tif'),\n PosixPath('force-level2-results/X0031_Y0030/20260427_LEVEL2_SEN2C_BOA.tif'),\n PosixPath('force-level2-results/X0031_Y0030/20260427_LEVEL2_SEN2C_OVV.jpg'),\n PosixPath('force-level2-results/X0031_Y0030/20260427_LEVEL2_SEN2C_QAI.tif'),\n PosixPath('force-level2-results/X0032_Y0029/20260417_LEVEL2_SEN2C_BOA.tif'),\n PosixPath('force-level2-results/X0032_Y0029/20260417_LEVEL2_SEN2C_OVV.jpg'),\n PosixPath('force-level2-results/X0032_Y0029/20260417_LEVEL2_SEN2C_QAI.tif'),\n PosixPath('force-level2-results/X0032_Y0029/20260419_LEVEL2_SEN2A_BOA.tif'),\n PosixPath('force-level2-results/X0032_Y0029/20260419_LEVEL2_SEN2A_OVV.jpg'),\n PosixPath('force-level2-results/X0032_Y0029/20260419_LEVEL2_SEN2A_QAI.tif'),\n PosixPath('force-level2-results/X0032_Y0029/20260422_LEVEL2_SEN2B_BOA.tif'),\n PosixPath('force-level2-results/X0032_Y0029/20260422_LEVEL2_SEN2B_OVV.jpg'),\n PosixPath('force-level2-results/X0032_Y0029/20260422_LEVEL2_SEN2B_QAI.tif'),\n PosixPath('force-level2-results/X0032_Y0029/20260427_LEVEL2_SEN2C_BOA.tif'),\n PosixPath('force-level2-results/X0032_Y0029/20260427_LEVEL2_SEN2C_OVV.jpg'),\n PosixPath('force-level2-results/X0032_Y0029/20260427_LEVEL2_SEN2C_QAI.tif'),\n PosixPath('force-level2-results/X0032_Y0030/20260417_LEVEL2_SEN2C_BOA.tif'),\n PosixPath('force-level2-results/X0032_Y0030/20260417_LEVEL2_SEN2C_OVV.jpg'),\n PosixPath('force-level2-results/X0032_Y0030/20260417_LEVEL2_SEN2C_QAI.tif'),\n PosixPath('force-level2-results/X0032_Y0030/20260419_LEVEL2_SEN2A_BOA.tif'),\n PosixPath('force-level2-results/X0032_Y0030/20260419_LEVEL2_SEN2A_OVV.jpg'),\n PosixPath('force-level2-results/X0032_Y0030/20260419_LEVEL2_SEN2A_QAI.tif'),\n PosixPath('force-level2-results/X0032_Y0030/20260422_LEVEL2_SEN2B_BOA.tif'),\n PosixPath('force-level2-results/X0032_Y0030/20260422_LEVEL2_SEN2B_OVV.jpg'),\n PosixPath('force-level2-results/X0032_Y0030/20260422_LEVEL2_SEN2B_QAI.tif'),\n PosixPath('force-level2-results/X0032_Y0030/20260427_LEVEL2_SEN2C_BOA.tif'),\n PosixPath('force-level2-results/X0032_Y0030/20260427_LEVEL2_SEN2C_OVV.jpg'),\n PosixPath('force-level2-results/X0032_Y0030/20260427_LEVEL2_SEN2C_QAI.tif'),\n PosixPath('force-level2-results/datacube-definition.prj'),\n PosixPath('force-level2-results/job-results.json')]\n```\n:::\n:::\n\n\n", "supporting": [ "level2_files" ], diff --git a/_freeze/docs/force/guide/processes/execute-results/html.json b/_freeze/docs/force/guide/processes/execute-results/html.json index 8acd317..1bdb2e7 100644 --- a/_freeze/docs/force/guide/processes/execute-results/html.json +++ b/_freeze/docs/force/guide/processes/execute-results/html.json @@ -1,8 +1,8 @@ { - "hash": "684b200a45fcc2c82940d1005b0ab3c3", + "hash": "528c1c9633d2511c80fda28df3ebb688", "result": { "engine": "jupyter", - "markdown": "---\nformat:\n html:\n code-fold: false\n page-navigation: true\nexecute:\n # Cache is necessary to avoid having to authenticate to build the documentation\n freeze: true\n---\n\n# openEO processes\n\nUsing an openEO client, we can control FORCE through a number of [openEO processes](https://openeo.org/documentation/1.0/datacubes.html). The FORCE openEO integration is specific to the backend deployed on CDSE, so the processes described here are not listed among the [standard processes](https://openeo.org/documentation/1.0/processes.html) and are generally experimental.\n\nThe processes wrapping FORCE functionality support most of the options that can be set in the FORCE parameter file ([level2](https://force-eo.readthedocs.io/en/latest/components/lower-level/level2/param.html), [TSA](https://force-eo.readthedocs.io/en/latest/components/higher-level/tsa/param.html)). See the article on [parametrization](parametrization.qmd) for more details on the supported options and defaults.\n\n\nTo discover the processes, we must first connect to the CDSE openEO backend:\n\n::: {#6c51ce37 .cell execution_count=1}\n``` {.python .cell-code}\nimport openeo\nconnection = openeo.connect(\"https://openeo.dataspace.copernicus.eu\")\nconnection.authenticate_oidc()\n```\n\n::: {.cell-output .cell-output-stdout}\n```\nAuthenticated using refresh token.\n```\n:::\n\n::: {.cell-output .cell-output-display execution_count=1}\n```\n\n```\n:::\n:::\n\n\n\n\n::: {#e7080f59 .cell execution_count=2}\n\n::: {.cell-output .cell-output-stdout}\n```\nAuthenticated using refresh token.\n```\n:::\n:::\n\n\n\n\nIf you don't have your [credentials set as environment variables](https://open-eo.github.io/openeo-python-client/auth.html#oidc-client-credentials-using-environment-variables), you will be prompted to log in in a browser window by default. See also the openEO documentation on authentication using Open ID connect ([overview](https://openeo.org/documentation/1.0/authentication.html#device-flow), [Python client](https://openeo.org/documentation/1.0/authentication.html))\n\n\n## Query\n\n\nWe can use either the openEO process `query_stac` or another method to select input products for processing with FORCE. See the [next section](query.qmd) for an in depth discussion on the different methods how inputs may be specified.\n\n\n:::{.callout-note collapse=\"true\" title=\"query_stac process definition\"}\n\n::: {#fc6319b2 .cell execution_count=3}\n``` {.python .cell-code}\nconnection.describe_process(\"query_stac\")\n```\n\n::: {.cell-output .cell-output-display execution_count=3}\n```{=html}\n\n \n \n \n \n \n```\n:::\n:::\n\n\n:::\n\n\n\nWe can use openEO's `describe_process` API call to discover the parameters of an openEO process.\n\n\n## Level 2\n\n\nThe openEO integration of FORCE provides a `force_level2` process wrapping the [FORCE level 2 processing system](https://force-eo.readthedocs.io/en/latest/components/lower-level/level2/).\n\n:::{.callout-note collapse=\"true\" title=\"force_level2 process definition\"}\n\n::: {#8395daba .cell execution_count=4}\n``` {.python .cell-code}\nconnection.describe_process(\"force_level2\")\n```\n\n::: {.cell-output .cell-output-display execution_count=4}\n```{=html}\n\n \n \n \n \n \n```\n:::\n:::\n\n\n:::\n\nThe openEO processes exposes most of the options that can be set in the FORCE parameter file.\n\n\n\n## Time Series Analysis\n\nThe `force_tsa` process wraps the [Time Series Analysis](https://force-eo.readthedocs.io/en/latest/components/higher-level/tsa/) module.\n\n:::{.callout-note collapse=\"true\" title=\"force_tsa process definition\"}\n\n::: {#a5306e02 .cell execution_count=5}\n``` {.python .cell-code}\nconnection.describe_process(\"force_tsa\")\n```\n\n::: {.cell-output .cell-output-display execution_count=5}\n```{=html}\n\n \n \n \n \n \n```\n:::\n:::\n\n\n:::\n\n# openEO job management\n\nBesides the openEO processes which make up the process graph, the openEO client provides functionality to submit, schedule and control jobs.\nPlease refer to the [Python client documentation](https://open-eo.github.io/openeo-python-client/batch_jobs.html) (or the documentation of your preferred openEO client) for a detailed explanation of job management using openEO.\n\n\nName | Qualified name (Python client) | Purpose\n-----|----------------|--------\n`create_job` | [`openeo.rest.connection.Connection.create_job`](https://open-eo.github.io/openeo-python-client/api.html#openeo.rest.connection.Connection.create_job) | Submit a Process Graph to the backend \n`start_and_wait` | [`openeo.rest.job.BatchJob.start_and_wait`](https://open-eo.github.io/openeo-python-client/api.html#openeo.rest.job.BatchJob.start_and_wait) | Start a submitted job and block until completion\n`start` | [`openeo.rest.job.BatchJob.start`](https://open-eo.github.io/openeo-python-client/api.html#openeo.rest.job.BatchJob.start) | Same as `start_and_wait` but do not block execution\n`get_results` | [`openeo.rest.job.BatchJob.get_results`](https://open-eo.github.io/openeo-python-client/api.html#openeo.rest.job.BatchJob.get_results) | Get a `JobResults` object from a completed Job (contains references to output files)\n`download_files` | [`openeo.rest.job.JobResults.download_files`](https://open-eo.github.io/openeo-python-client/api.html#openeo.rest.job.JobResults.download_files) | Download all assets (files) produced by a job\n`download_file` | [`openeo.rest.job.JobResults.download_file`](https://open-eo.github.io/openeo-python-client/api.html#openeo.rest.job.JobResults.download_file) | Download a singular asset (file) produced by a job\n\n\n## Monitoring progress\n\nWhen using `start_and_wait` to start a job, status updates will automatically be printed to the command line or jupyter output cell.\n\nAlternatively, progress and logs can be monitored using the openEO web interface at [https://openeo.dataspace.copernicus.eu](https://open-eo.github.io/openeo-python-client/api.html#openeo.rest.job.JobResults.download_files) or use the client as explained e.g., in the [Python client documentation](https://open-eo.github.io/openeo-python-client/batch_jobs.html#create-a-batch-job).\n\n", + "markdown": "---\ntitle: \"openEO processes\"\norder: 2\n---\n\nUsing an openEO client, we can control FORCE through a number of [openEO processes](https://openeo.org/documentation/1.0/datacubes.html).\nThe FORCE openEO integration is specific to the backend deployed on CDSE, so the processes described here are\nnot listed among the [standard processes](https://openeo.org/documentation/1.0/processes.html) and are generally experimental.\n\nThis page gives an overview of the openEO processes specific to FORCE and client operations needed to interact\nwith them.\n\nThe processes wrapping FORCE functionality support most of the options that can be set in the FORCE parameter file\n([level2](https://force-eo.readthedocs.io/en/latest/components/lower-level/level2/param.html),\n[TSA](https://force-eo.readthedocs.io/en/latest/components/higher-level/tsa/param.html)).\nSee the article on [parametrization](parametrization.qmd) for more details on the supported options and defaults.\n\n\nTo discover the processes, we must first connect to the CDSE openEO backend:\n\n::: {#a6a2084e .cell execution_count=1}\n``` {.python .cell-code}\nimport openeo\nconnection = openeo.connect(\"https://openeo.dataspace.copernicus.eu\")\nconnection.authenticate_oidc()\n```\n\n::: {.cell-output .cell-output-stdout}\n```\nAuthenticated using refresh token.\n```\n:::\n\n::: {.cell-output .cell-output-display execution_count=1}\n```\n\n```\n:::\n:::\n\n\n\n\n::: {#3795abdb .cell execution_count=2}\n\n::: {.cell-output .cell-output-stdout}\n```\nAuthenticated using refresh token.\n```\n:::\n:::\n\n\nIf you don't have your\n[credentials set as environment variables](https://open-eo.github.io/openeo-python-client/auth.html#oidc-client-credentials-using-environment-variables),\nyou will be prompted to log in in a browser window by default. See also the openEO documentation on authentication using\nOpen ID connect ([overview](https://openeo.org/documentation/1.0/authentication.html),\n[Python client](https://open-eo.github.io/openeo-python-client/auth.html#oidc-client-credentials-using-environment-variables))\n\nWe can use openEO's\n[`describe_process`](https://open-eo.github.io/openeo-python-client/api.html#openeo.rest.connection.Connection.describe_process)\nAPI call to discover the parameters of an openEO process.\n\n## Query\n\nAs a first step, we must determine the input products for FORCE to process. This is best achieved by an external\nquerying method, such as [`pystac_client`](https://pystac-client.readthedocs.io/en/stable/).\nAlternatively, CDSE openEO provides a process `query_stac` to determine input products from `spatial_extent` and\n`temporal_extent`. Using this process in combination with `force_level2` is currently limited to very few input\nproducts, therefore we recommend using it only for small tests.\n\nSee the [next section](query.qmd) for an in-depth discussion on the different methods how inputs may be specified.\n\n\n:::{.callout-note collapse=\"true\" title=\"query_stac process definition\"}\n\n::: {#f18f7d35 .cell execution_count=3}\n``` {.python .cell-code}\nconnection.describe_process(\"query_stac\")\n```\n\n::: {.cell-output .cell-output-display execution_count=3}\n```{=html}\n\n \n \n \n \n \n```\n:::\n:::\n\n\n:::\n\n## Level 2\n\nThe openEO integration of FORCE provides a `force_level2` process wrapping the [FORCE level 2 processing system](https://force-eo.readthedocs.io/en/latest/components/lower-level/level2/).\n\nThe openEO processes exposes most of the\n[options that can be set in the FORCE level 2 parameter file](https://force-eo.readthedocs.io/en/latest/components/lower-level/level2/param.html#l2-param).\nExceptions and additional parameters are described in the [parametrization article](parametrization.qmd).\n\n:::{.callout-note collapse=\"true\" title=\"force_level2 process definition\"}\n\n::: {#bf039539 .cell execution_count=4}\n``` {.python .cell-code}\nconnection.describe_process(\"force_level2\")\n```\n\n::: {.cell-output .cell-output-display execution_count=4}\n```{=html}\n\n \n \n \n \n \n```\n:::\n:::\n\n\n:::\n\n## Time Series Analysis\n\nThe `force_tsa` process wraps the [Time Series Analysis](https://force-eo.readthedocs.io/en/latest/components/higher-level/tsa/) module.\n\nThe openEO processes exposes most of the\n[options that can be set in the FORCE TSA parameter file](https://force-eo.readthedocs.io/en/latest/components/higher-level/tsa/param.html).\nExceptions and additional parameters are described in the [parametrization article](parametrization.qmd).\n\n:::{.callout-note collapse=\"true\" title=\"force_tsa process definition\"}\n\n::: {#97ab98d0 .cell execution_count=5}\n``` {.python .cell-code}\nconnection.describe_process(\"force_tsa\")\n```\n\n::: {.cell-output .cell-output-display execution_count=5}\n```{=html}\n\n \n \n \n \n \n```\n:::\n:::\n\n\n:::\n\n# openEO job management\n\nBesides the openEO processes which make up the process graph, the openEO client provides functionality to submit, schedule and control jobs.\nPlease refer to the [Python client documentation](https://open-eo.github.io/openeo-python-client/batch_jobs.html) (or the documentation of your preferred openEO client) for a detailed explanation of job management using openEO.\n\n\nName | Qualified name (Python client) | Purpose\n-----|----------------|--------\n`create_job` | [`openeo.rest.connection.Connection.create_job`](https://open-eo.github.io/openeo-python-client/api.html#openeo.rest.connection.Connection.create_job) | Submit a Process Graph to the backend\n`start_and_wait` | [`openeo.rest.job.BatchJob.start_and_wait`](https://open-eo.github.io/openeo-python-client/api.html#openeo.rest.job.BatchJob.start_and_wait) | Start a submitted job and block until completion\n`start` | [`openeo.rest.job.BatchJob.start`](https://open-eo.github.io/openeo-python-client/api.html#openeo.rest.job.BatchJob.start) | Same as `start_and_wait` but do not block execution\n`get_results` | [`openeo.rest.job.BatchJob.get_results`](https://open-eo.github.io/openeo-python-client/api.html#openeo.rest.job.BatchJob.get_results) | Get a `JobResults` object from a completed Job (contains references to output files)\n`download_files` | [`openeo.rest.job.JobResults.download_files`](https://open-eo.github.io/openeo-python-client/api.html#openeo.rest.job.JobResults.download_files) | Download all assets (files) produced by a job\n`download_file` | [`openeo.rest.job.JobResults.download_file`](https://open-eo.github.io/openeo-python-client/api.html#openeo.rest.job.JobResults.download_file) | Download a singular asset (file) produced by a job\n`export_workspace` | [`openeo.rest.stac_resource.StacResource.export_workspace`](https://open-eo.github.io/openeo-python-client/api.html#openeo.rest.stac_resource.StacResource.export_workspace) | Export raw files (FORCE data cube) and STAC to remote cloud storage. Used for asynchronous further processing (TSA) without download of the intermediate result.\n\n\n## Monitoring progress\n\nWhen using `start_and_wait` to start a job, status updates will automatically be printed to the command line or jupyter output cell.\n\nAlternatively, progress and logs can be monitored using the openEO web interface at [https://openeo.dataspace.copernicus.eu](https://open-eo.github.io/openeo-python-client/api.html#openeo.rest.job.JobResults.download_files) or use the client as explained in the [Python client documentation](https://open-eo.github.io/openeo-python-client/batch_jobs.html#create-a-batch-job).\n\n", "supporting": [ "processes_files" ], diff --git a/_freeze/docs/force/guide/query/execute-results/html.json b/_freeze/docs/force/guide/query/execute-results/html.json index c0aba8b..269c85a 100644 --- a/_freeze/docs/force/guide/query/execute-results/html.json +++ b/_freeze/docs/force/guide/query/execute-results/html.json @@ -1,8 +1,8 @@ { - "hash": "90ad4503d338cebe5ed8c4be181c03c8", + "hash": "3781c4790e420db55f6a57205026372c", "result": { "engine": "jupyter", - "markdown": "---\nformat:\n html:\n page-navigation: true\nexecute:\n freeze: true\n---\n\n# Selecting input data\n\nWhen running FORCE level 2 locally, we are responsible to manually provide the necessary input data and reference their full path in the `FILE_QUEUE` parameter of the [FORCE level 2 parameter file](https://force-eo.readthedocs.io/en/latest/components/lower-level/level2/param.html).\n\nWhen running on CDSE, we do not have access to the excuting environment and its filesystem directly.\nDownloads, file management and creation of the file queue are managed by the integration. \nTo specify which files should be processed, we provide a [STAC](https://stacspec.org/en) document ([Catalog](https://github.com/radiantearth/stac-spec/blob/master/catalog-spec/catalog-spec.md), [Collection](https://github.com/radiantearth/stac-spec/blob/master/collection-spec/collection-spec.md), [Item](https://github.com/radiantearth/stac-spec/blob/master/item-spec/item-spec.md) or [ItemCollection](https://github.com/radiantearth/stac-api-spec/blob/main/fragments/itemcollection/README.md)) to the process. \n\n:::{.callout-warning}\nMake sure that the STAC document provided to `force_level2` only references the files that should actually be processed. The FORCE integration will attempt to download all products referenced in the document, so it is inadvisable to provide a reference to the entire Sentinel-2 L1C collection on CDSE for example\n:::\n\n## STAC\n\nThe integrated FORCE and openEO more generally uses the [SpatioTemporal Asset Catalog (STAC) specification](https://stacspec.org/en) to identify (groups of) EO products and annotate them with metadata. \n\nIf you are unfamiliar with STAC, we recommend that you familiarize yourself with the basic components of the specification (What are catalogs, items and feature collections?) [on the official STAC specification website](https://stacspec.org/en) before proceeding.\n\n\n**The CDSE Sentinel-2 L1C STAC collection which we use as input for FORCE level 2 processing, is hosted at [https://stac.dataspace.copernicus.eu/v1/collections/sentinel-2-l1c](https://stac.dataspace.copernicus.eu/v1/collections/sentinel-2-l1c)**\n\nIn the following, we show how this collection may be queried to to obtain inputs for FORCE level 2 processing.\n\n\n## Querying STAC catalogs/collections\n\nFORCE level 2 can work with any STAC URL or document referencing input products of the correct type (Sentinel-2 L1C). The following describes how to make a query to the CDSE STAC catalog.\nBe advised to query the CDSE STAC collection of L1C, as data access will be faster than external archives and the integration makes some assumptions about the metadata provided in the STAC items describing\nthe products.\n\nIt is recommended to make the query using `pystac_client` for now, as using the `query_stac` process limits us to very small data sets See below for more details.\n\n### Area of interest\n\nAs an example, we selected an area in northern Italy, around the city of Modena.\n\n::: {#cc4cc097 .cell execution_count=1}\n``` {.python .cell-code}\nw,s,e,n = 10.386, 44.437, 11.423, 44.973\nspatial_extent = { \"west\": w, \"south\": s, \"east\": e, \"north\": n}\n# ATTENTION: Inclusion of the second date in the search depends on the query method (see below).\ntemporal_extent=[\"2026-04-17\", \"2026-05-01\"]\n```\n:::\n\n\n:::{.callout-warning}\nWhen defining the temporal extent as shown above, be aware that different methods may interpret time ranges differently. OpenEO's `query_stac` behaves just like [`load_collection`](https://openeo.org/documentation/1.0/processes.html#load_collection): The second instant is **excluded** from the search.\n\nContrarily, `pystac_client`'s [`search`](https://pystac-client.readthedocs.io/en/stable/api.html#pystac_client.Client.search) method **includes** the second date when given as a list. Note that `pystac_client`'s search has many more options for specifying time ranges.\n:::\n\n::: {#efd5e51c .cell execution_count=2}\n``` {.python .cell-code code-fold=\"true\" code-summary=\"AOI plot\"}\nimport matplotlib.pyplot as plt\nimport contextily as ctx\nimport geopandas as gpd\nfrom shapely import Polygon\n\ndef plot_area_of_interest(\n w, s, e, n, *, \n context=5,\n color=\"red\",\n linewidth=2,\n title=\"Spatial Extent\",\n figsize=(12, 8),\n show_tick_labels=True,\n ):\n polygon = Polygon([(w, s), (e, s), (e, n), (w, n)])\n gdf = gpd.GeoDataFrame(geometry=[polygon], crs=\"EPSG:4326\")\n fig, ax = plt.subplots(1, 1, figsize=figsize)\n gdf.plot(edgecolor=color, facecolor=\"none\", linewidth=linewidth, ax=ax)\n gdf.plot(edgecolor=color, facecolor=\"none\", linewidth=linewidth, ax=ax)\n ax.set_xlim(w - context, e + context)\n ax.set_ylim(s - context, n + context)\n if not show_tick_labels:\n ax.set_xticks([])\n ax.set_yticks([])\n ctx.add_basemap(ax, source=ctx.providers.OpenStreetMap.Mapnik, crs=gdf.crs)\n fig.suptitle(title)\n fig.tight_layout()\n\nplot_area_of_interest(\n w=w,\n s=s,\n e=e,\n n=n,\n context=3,\n figsize=(10, 6),\n title=f\"Spatial Extent: Modena\",\n show_tick_labels=False\n)\n```\n\n::: {.cell-output .cell-output-display}\n![](query_files/figure-html/cell-3-output-1.png){width=414 height=570}\n:::\n:::\n\n\n### Querying a catalog\n\n:::{.panel-tabset group=\"query_engine\"}\n\n#### openEO\n\n::: {#0aa7078d .cell execution_count=3}\n``` {.python .cell-code}\nimport openeo\n\nL1C_COLLECTION_URL = \"https://stac.dataspace.copernicus.eu/v1/collections/sentinel-2-l1c\"\nSTAC_ROOT_URL = \"https://stac.dataspace.copernicus.eu/v1\"\n\nconnection = openeo.connect(\"openeo.dataspace.copernicus.eu\").authenticate_oidc()\nquery_pg = openeo.processes.process(\n \"query_stac\",\n arguments={\n \"url\": L1C_COLLECTION_URL,\n \"temporal_extent\": temporal_extent,\n \"spatial_extent\": spatial_extent,\n }\n)\n```\n\n::: {.cell-output .cell-output-stdout}\n```\nAuthenticated using refresh token.\n```\n:::\n:::\n\n\n\n\n::: {#06d59308 .cell execution_count=4}\n\n::: {.cell-output .cell-output-stdout}\n```\nAuthenticated using refresh token.\n```\n:::\n:::\n\n\nWe can directly use `query_pg` as a parameter for FORCE level 2, using openEO's process chaining.\nTo inspect the query results ahead of time, we may want to execute it ahead of time.\nIf we do not do this, execution of the query will be delayed until we submit the entire process graph (including processing with FORCE).\n\n::: {#3a0c3777 .cell execution_count=5}\n``` {.python .cell-code}\nquery_res = connection.execute(query_pg)\n\nprint(f\"Found products ({len(query_res['features'])}):\")\nprint(\"\\n\".join([f[\"id\"] for f in query_res[\"features\"]]), \"\\n\")\n```\n\n::: {.cell-output .cell-output-stdout}\n```\nFound products (18):\nS2C_MSIL1C_20260430T102021_N0512_R065_T32TPQ_20260430T135954\nS2C_MSIL1C_20260430T102021_N0512_R065_T32TNQ_20260430T135954\nS2A_MSIL1C_20260429T101041_N0512_R022_T32TPQ_20260429T152547\nS2A_MSIL1C_20260429T101041_N0512_R022_T32TNQ_20260429T152547\nS2C_MSIL1C_20260427T101021_N0512_R022_T32TPQ_20260427T142022\nS2C_MSIL1C_20260427T101021_N0512_R022_T32TNQ_20260427T142022\nS2B_MSIL1C_20260425T101559_N0512_R065_T32TPQ_20260425T142712\nS2B_MSIL1C_20260425T101559_N0512_R065_T32TNQ_20260425T142712\nS2A_MSIL1C_20260422T102041_N0512_R065_T32TPQ_20260422T153540\nS2A_MSIL1C_20260422T102041_N0512_R065_T32TNQ_20260422T153540\nS2B_MSIL1C_20260422T101019_N0512_R022_T32TPQ_20260422T134442\nS2B_MSIL1C_20260422T101019_N0512_R022_T32TNQ_20260422T134442\nS2C_MSIL1C_20260420T102021_N0512_R065_T32TPQ_20260420T135843\nS2C_MSIL1C_20260420T102021_N0512_R065_T32TNQ_20260420T135843\nS2A_MSIL1C_20260419T100711_N0512_R022_T32TPQ_20260419T152521\nS2A_MSIL1C_20260419T100711_N0512_R022_T32TNQ_20260419T152521\nS2C_MSIL1C_20260417T101021_N0512_R022_T32TPQ_20260417T134842\nS2C_MSIL1C_20260417T101021_N0512_R022_T32TNQ_20260417T134842 \n\n```\n:::\n:::\n\n\n#### Pystac client {.active}\n\n::: {#971ae466 .cell execution_count=6}\n``` {.python .cell-code}\nimport pystac_client\n\nL1C_COLLECTION_URL = \"https://stac.dataspace.copernicus.eu/v1/collections/sentinel-2-l1c\"\nSTAC_ROOT_URL = \"https://stac.dataspace.copernicus.eu/v1\"\n\nclient = pystac_client.Client.open(\"https://stac.dataspace.copernicus.eu/v1\")\nsearch = client.search(\n datetime=temporal_extent,\n collections=[\"sentinel-2-l1c\"],\n bbox=[w, s, e, n]\n)\nitem_collection = search.item_collection()\n\nprint(\"Found items:\")\nfor item in search.items():\n print(item.id)\n```\n\n::: {.cell-output .cell-output-stdout}\n```\nFound items:\nS2C_MSIL1C_20260430T102021_N0512_R065_T32TPQ_20260430T135954\nS2C_MSIL1C_20260430T102021_N0512_R065_T32TNQ_20260430T135954\nS2A_MSIL1C_20260429T101041_N0512_R022_T32TPQ_20260429T152547\nS2A_MSIL1C_20260429T101041_N0512_R022_T32TNQ_20260429T152547\nS2C_MSIL1C_20260427T101021_N0512_R022_T32TPQ_20260427T142022\nS2C_MSIL1C_20260427T101021_N0512_R022_T32TNQ_20260427T142022\nS2B_MSIL1C_20260425T101559_N0512_R065_T32TPQ_20260425T142712\nS2B_MSIL1C_20260425T101559_N0512_R065_T32TNQ_20260425T142712\nS2A_MSIL1C_20260422T102041_N0512_R065_T32TPQ_20260422T153540\nS2A_MSIL1C_20260422T102041_N0512_R065_T32TNQ_20260422T153540\nS2B_MSIL1C_20260422T101019_N0512_R022_T32TPQ_20260422T134442\nS2B_MSIL1C_20260422T101019_N0512_R022_T32TNQ_20260422T134442\nS2C_MSIL1C_20260420T102021_N0512_R065_T32TPQ_20260420T135843\nS2C_MSIL1C_20260420T102021_N0512_R065_T32TNQ_20260420T135843\nS2A_MSIL1C_20260419T100711_N0512_R022_T32TPQ_20260419T152521\nS2A_MSIL1C_20260419T100711_N0512_R022_T32TNQ_20260419T152521\nS2C_MSIL1C_20260417T101021_N0512_R022_T32TPQ_20260417T134842\nS2C_MSIL1C_20260417T101021_N0512_R022_T32TNQ_20260417T134842\n```\n:::\n:::\n\n\n:::\n\n:::{.content-visible when-format=\"html\"}\n\n### Inspecting query results\n\nWe may use pystac to inspect or process the feature collection further. In this example, we inspect the first of the returned items:\n\n\n:::{.panel-tabset group=\"query_engine\"}\n\n#### openEO\n\n::: {#d3ca7a76 .cell execution_count=7}\n``` {.python .cell-code}\nimport pystac\n\nitem = pystac.Item.from_dict(query_res[\"features\"][0])\nitem\n```\n\n::: {.cell-output .cell-output-display execution_count=15}\n```{=html}\n\n\n\n
\n
\n <Item id=S2C_MSIL1C_20260430T102021_N0512_R065_T32TPQ_20260430T135954>\n
\n\n
\n```\n:::\n:::\n\n\n#### Pystac client {.active}\n\n::: {#7c6ee016 .cell execution_count=8}\n``` {.python .cell-code}\nnext(iter(search.items()))\n```\n\n::: {.cell-output .cell-output-display execution_count=16}\n```{=html}\n\n\n\n
\n
\n <Item id=S2C_MSIL1C_20260430T102021_N0512_R065_T32TPQ_20260430T135954>\n
\n\n
\n```\n:::\n:::\n\n\n:::\n:::\n\n\n## Other methods\n\nCDSE provides a [graphical STAC browser](https://browser.stac.dataspace.copernicus.eu/collections/sentinel-2-l1c) that can be used to search for and select singular items.\n\n", + "markdown": "---\ntitle: Selecting input data\norder: 3\n---\n\nWhen running FORCE level 2 locally, we are responsible to manually provide the necessary input data and reference their full path in the `FILE_QUEUE` parameter of the [FORCE level 2 parameter file](https://force-eo.readthedocs.io/en/latest/components/lower-level/level2/param.html).\n\nWhen running on CDSE, we do not have access to the excuting environment and its filesystem directly.\nDownloads, file management and creation of the file queue are managed by the integration.\nTo specify which files should be processed, we provide a [STAC](https://stacspec.org/en) document ([Catalog](https://github.com/radiantearth/stac-spec/blob/master/catalog-spec/catalog-spec.md), [Collection](https://github.com/radiantearth/stac-spec/blob/master/collection-spec/collection-spec.md), [Item](https://github.com/radiantearth/stac-spec/blob/master/item-spec/item-spec.md) or [ItemCollection](https://github.com/radiantearth/stac-api-spec/blob/main/fragments/itemcollection/README.md)) to the process.\n\n:::{.callout-warning}\nMake sure that the STAC document provided to `force_level2` only references the files that should actually be processed. The FORCE integration will attempt to download all products referenced in the document, so it is inadvisable to provide a reference to something like the entire Sentinel-2 L1C collection on CDSE.\n:::\n\n## STAC\n\nThe integrated FORCE and openEO more generally uses the [SpatioTemporal Asset Catalog (STAC) specification](https://stacspec.org/en) to identify (groups of) EO products and annotate them with metadata.\n\nIf you are unfamiliar with STAC, we recommend that you familiarize yourself with the basic components of the specification (What are catalogs, items and feature collections?) [on the official STAC specification website](https://stacspec.org/en) before proceeding.\n\n\n**The CDSE Sentinel-2 L1C STAC collection which we use as input for FORCE level 2 processing, is hosted at [https://stac.dataspace.copernicus.eu/v1/collections/sentinel-2-l1c](https://stac.dataspace.copernicus.eu/v1/collections/sentinel-2-l1c)**\n\nIn the following, we show how this collection may be queried to to obtain inputs for FORCE level 2 processing.\n\n\n## Querying STAC catalogs/collections\n\nFORCE level 2 can work with any STAC URL or document referencing input products of the correct type (Sentinel-2 L1C). The following describes how to make a query to the CDSE STAC catalog.\nBe advised to query the CDSE STAC collection of L1C, as data access will be faster than external archives and the integration makes some assumptions about the metadata provided in the STAC items describing\nthe products.\n\nIt is recommended to make the query using `pystac_client` for now, as using the `query_stac` process limits us to very small data sets See below for more details.\n\n### Area of interest\n\nAs an example, we selected an area in northern Italy, around the city of Modena.\n\n::: {#a2cb0c85 .cell execution_count=1}\n``` {.python .cell-code}\nw,s,e,n = 10.386, 44.437, 11.423, 44.973\nspatial_extent = { \"west\": w, \"south\": s, \"east\": e, \"north\": n}\n# ATTENTION: Inclusion of the second date in the search depends on the query method (see below).\ntemporal_extent=[\"2026-04-17\", \"2026-05-01\"]\n```\n:::\n\n\n:::{.callout-warning}\nWhen defining the temporal extent as shown above, be aware that different methods may interpret time ranges differently. OpenEO's `query_stac` behaves just like [`load_collection`](https://openeo.org/documentation/1.0/processes.html#load_collection): The second instant is **excluded** from the search.\n\nContrarily, `pystac_client`'s [`search`](https://pystac-client.readthedocs.io/en/stable/api.html#pystac_client.Client.search) method **includes** the second date when given as a list. Note that `pystac_client`'s search has many more options for specifying time ranges.\n:::\n\n::: {#66de52d6 .cell execution_count=2}\n``` {.python .cell-code code-fold=\"true\" code-summary=\"AOI plot\"}\nimport matplotlib.pyplot as plt\nimport contextily as ctx\nimport geopandas as gpd\nfrom shapely import Polygon\n\ndef plot_area_of_interest(\n w, s, e, n, *,\n context=5,\n color=\"red\",\n linewidth=2,\n title=\"Spatial Extent\",\n figsize=(12, 8),\n show_tick_labels=True,\n ):\n polygon = Polygon([(w, s), (e, s), (e, n), (w, n)])\n gdf = gpd.GeoDataFrame(geometry=[polygon], crs=\"EPSG:4326\")\n fig, ax = plt.subplots(1, 1, figsize=figsize)\n gdf.plot(edgecolor=color, facecolor=\"none\", linewidth=linewidth, ax=ax)\n gdf.plot(edgecolor=color, facecolor=\"none\", linewidth=linewidth, ax=ax)\n ax.set_xlim(w - context, e + context)\n ax.set_ylim(s - context, n + context)\n if not show_tick_labels:\n ax.set_xticks([])\n ax.set_yticks([])\n ctx.add_basemap(ax, source=ctx.providers.OpenTopoMap, crs=gdf.crs)\n fig.suptitle(title)\n fig.tight_layout()\n\nplot_area_of_interest(\n w=w,\n s=s,\n e=e,\n n=n,\n context=3,\n figsize=(10, 6),\n title=f\"Spatial Extent: Modena\",\n show_tick_labels=False\n)\n```\n\n::: {.cell-output .cell-output-display}\n![](query_files/figure-html/cell-3-output-1.png){width=414 height=569}\n:::\n:::\n\n\n### Querying a catalog\n\n:::{.panel-tabset group=\"query_engine\"}\n\n#### openEO\n\n::: {#680621ef .cell execution_count=3}\n``` {.python .cell-code}\nimport openeo\n\nL1C_COLLECTION_URL = \"https://stac.dataspace.copernicus.eu/v1/collections/sentinel-2-l1c\"\nSTAC_ROOT_URL = \"https://stac.dataspace.copernicus.eu/v1\"\n\nconnection = openeo.connect(\"openeo.dataspace.copernicus.eu\").authenticate_oidc()\nquery_pg = openeo.processes.process(\n \"query_stac\",\n arguments={\n \"url\": L1C_COLLECTION_URL,\n \"temporal_extent\": temporal_extent,\n \"spatial_extent\": spatial_extent,\n }\n)\n```\n\n::: {.cell-output .cell-output-stdout}\n```\nAuthenticated using refresh token.\n```\n:::\n:::\n\n\n\n\n::: {#ac5b151a .cell execution_count=4}\n\n::: {.cell-output .cell-output-stdout}\n```\nAuthenticated using refresh token.\n```\n:::\n:::\n\n\nWe can directly use `query_pg` as a parameter for FORCE level 2, using openEO's process chaining.\nTo inspect the query results ahead of time, we may want to execute it ahead of time.\nIf we do not do this, execution of the query will be delayed until we submit the entire process graph (including processing with FORCE).\n\n::: {#3f54c523 .cell execution_count=5}\n``` {.python .cell-code}\nquery_res = connection.execute(query_pg)\n\nprint(f\"Found products ({len(query_res['features'])}):\")\nprint(\"\\n\".join([f[\"id\"] for f in query_res[\"features\"]]), \"\\n\")\n```\n\n::: {.cell-output .cell-output-stdout}\n```\nFound products (18):\nS2C_MSIL1C_20260430T102021_N0512_R065_T32TPQ_20260430T135954\nS2C_MSIL1C_20260430T102021_N0512_R065_T32TNQ_20260430T135954\nS2A_MSIL1C_20260429T101041_N0512_R022_T32TPQ_20260429T152547\nS2A_MSIL1C_20260429T101041_N0512_R022_T32TNQ_20260429T152547\nS2C_MSIL1C_20260427T101021_N0512_R022_T32TPQ_20260427T142022\nS2C_MSIL1C_20260427T101021_N0512_R022_T32TNQ_20260427T142022\nS2B_MSIL1C_20260425T101559_N0512_R065_T32TPQ_20260425T142712\nS2B_MSIL1C_20260425T101559_N0512_R065_T32TNQ_20260425T142712\nS2A_MSIL1C_20260422T102041_N0512_R065_T32TPQ_20260422T153540\nS2A_MSIL1C_20260422T102041_N0512_R065_T32TNQ_20260422T153540\nS2B_MSIL1C_20260422T101019_N0512_R022_T32TPQ_20260422T134442\nS2B_MSIL1C_20260422T101019_N0512_R022_T32TNQ_20260422T134442\nS2C_MSIL1C_20260420T102021_N0512_R065_T32TPQ_20260420T135843\nS2C_MSIL1C_20260420T102021_N0512_R065_T32TNQ_20260420T135843\nS2A_MSIL1C_20260419T100711_N0512_R022_T32TPQ_20260419T152521\nS2A_MSIL1C_20260419T100711_N0512_R022_T32TNQ_20260419T152521\nS2C_MSIL1C_20260417T101021_N0512_R022_T32TPQ_20260417T134842\nS2C_MSIL1C_20260417T101021_N0512_R022_T32TNQ_20260417T134842 \n\n```\n:::\n:::\n\n\n#### Pystac client {.active}\n\n::: {#af4330ee .cell execution_count=6}\n``` {.python .cell-code}\nimport pystac_client\nimport time\n\nL1C_COLLECTION_URL = \"https://stac.dataspace.copernicus.eu/v1/collections/sentinel-2-l1c\"\nSTAC_ROOT_URL = \"https://stac.dataspace.copernicus.eu/v1\"\n\nclient = pystac_client.Client.open(\"https://stac.dataspace.copernicus.eu/v1\")\nsearch = client.search(\n datetime=temporal_extent,\n collections=[\"sentinel-2-l1c\"],\n bbox=[w, s, e, n]\n)\nitem_collection = search.item_collection()\n\nprint(\"Found items:\")\nfor item in search.items():\n time.sleep(1) # CDSE Stac API has a rate limit of 5 requests per second\n print(item.id)\n```\n\n::: {.cell-output .cell-output-stdout}\n```\nFound items:\nS2C_MSIL1C_20260430T102021_N0512_R065_T32TPQ_20260430T135954\nS2C_MSIL1C_20260430T102021_N0512_R065_T32TNQ_20260430T135954\nS2A_MSIL1C_20260429T101041_N0512_R022_T32TPQ_20260429T152547\nS2A_MSIL1C_20260429T101041_N0512_R022_T32TNQ_20260429T152547\nS2C_MSIL1C_20260427T101021_N0512_R022_T32TPQ_20260427T142022\nS2C_MSIL1C_20260427T101021_N0512_R022_T32TNQ_20260427T142022\nS2B_MSIL1C_20260425T101559_N0512_R065_T32TPQ_20260425T142712\nS2B_MSIL1C_20260425T101559_N0512_R065_T32TNQ_20260425T142712\nS2A_MSIL1C_20260422T102041_N0512_R065_T32TPQ_20260422T153540\nS2A_MSIL1C_20260422T102041_N0512_R065_T32TNQ_20260422T153540\nS2B_MSIL1C_20260422T101019_N0512_R022_T32TPQ_20260422T134442\nS2B_MSIL1C_20260422T101019_N0512_R022_T32TNQ_20260422T134442\nS2C_MSIL1C_20260420T102021_N0512_R065_T32TPQ_20260420T135843\nS2C_MSIL1C_20260420T102021_N0512_R065_T32TNQ_20260420T135843\nS2A_MSIL1C_20260419T100711_N0512_R022_T32TPQ_20260419T152521\nS2A_MSIL1C_20260419T100711_N0512_R022_T32TNQ_20260419T152521\nS2C_MSIL1C_20260417T101021_N0512_R022_T32TPQ_20260417T134842\nS2C_MSIL1C_20260417T101021_N0512_R022_T32TNQ_20260417T134842\n```\n:::\n:::\n\n\n:::\n\n:::{.content-visible when-format=\"html\"}\n\n### Inspecting query results\n\nWe may use pystac to inspect or process the feature collection further. In this example, we inspect the first of the returned items:\n\n\n:::{.panel-tabset group=\"query_engine\"}\n\n#### openEO\n\n::: {#f598e54d .cell execution_count=7}\n``` {.python .cell-code}\nimport pystac\n\nitem = pystac.Item.from_dict(query_res[\"features\"][0])\nitem\n```\n\n::: {.cell-output .cell-output-display execution_count=7}\n```{=html}\n\n\n\n
\n
\n <Item id=S2C_MSIL1C_20260430T102021_N0512_R065_T32TPQ_20260430T135954>\n
\n\n
\n```\n:::\n:::\n\n\n#### Pystac client {.active}\n\n::: {#d89d0651 .cell execution_count=8}\n``` {.python .cell-code}\nnext(iter(search.items()))\n```\n\n::: {.cell-output .cell-output-display execution_count=8}\n```{=html}\n\n\n\n
\n
\n <Item id=S2C_MSIL1C_20260430T102021_N0512_R065_T32TPQ_20260430T135954>\n
\n\n
\n```\n:::\n:::\n\n\n:::\n:::\n\n\n## Other methods\n\nCDSE provides a [graphical STAC browser](https://browser.stac.dataspace.copernicus.eu/collections/sentinel-2-l1c) that can be used to search for and select singular items.\n\n", "supporting": [ "query_files" ], diff --git a/_freeze/docs/force/guide/query/figure-html/cell-3-output-1.png b/_freeze/docs/force/guide/query/figure-html/cell-3-output-1.png index 4caba72..4989be4 100644 Binary files a/_freeze/docs/force/guide/query/figure-html/cell-3-output-1.png and b/_freeze/docs/force/guide/query/figure-html/cell-3-output-1.png differ diff --git a/_freeze/docs/force/guide/tsa/execute-results/html.json b/_freeze/docs/force/guide/tsa/execute-results/html.json index 2e374ba..ca01c57 100644 --- a/_freeze/docs/force/guide/tsa/execute-results/html.json +++ b/_freeze/docs/force/guide/tsa/execute-results/html.json @@ -1,8 +1,8 @@ { - "hash": "fd24067de41fe2ca504c8761cc78a40d", + "hash": "4013afe951748106e9e3a48dd393f471", "result": { "engine": "jupyter", - "markdown": "---\nformat:\n html:\n page-navigation: true\nexecute:\n freeze: true\ntitle: FORCE Time Series Analysis\n---\n\n\n\n\n\n\n\n## Recap\n\n:::{.callout-warning}\nYou should use `pystac_client` based querying if your inputs encompass more than just a handful of items\n:::\n\n:::{.callout-tip collapse=\"true\" title=\"Selecting input products\"}\n\n\n::: {#37b34abb .cell execution_count=1}\n``` {.python .cell-code}\nL1C_COLLECTION_URL = \"https://stac.dataspace.copernicus.eu/v1/collections/sentinel-2-l1c\"\nSTAC_ROOT_URL = \"https://stac.dataspace.copernicus.eu/v1\"\n\nw,s,e,n = 10.386, 44.437, 11.423, 44.973\nspatial_extent = { \"west\": w, \"south\": s, \"east\": e, \"north\": n}\n# ATTENTION: Inclusion of the second date in the search depends on the query method (see below).\ntemporal_extent=[\"2026-04-17\", \"2026-05-01\"]\n```\n:::\n\n\n:::{.panel-tabset group=\"query_engine\"}\n\n\n## openEO\n\n::: {#b9f2ed7d .cell execution_count=2}\n``` {.python .cell-code}\nimport openeo\nconnection = openeo.connect(\"openeo-staging.dataspace.copernicus.eu\").authenticate_oidc()\n\nquery_pg = openeo.processes.process(\n \"query_stac\",\n arguments={\n \"url\": L1C_COLLECTION_URL,\n \"temporal_extent\": temporal_extent,\n \"spatial_extent\": spatial_extent,\n }\n)\n```\n\n::: {.cell-output .cell-output-stdout}\n```\nAuthenticated using refresh token.\n```\n:::\n:::\n\n\n## Pystac client {.active}\n\n::: {#3f754c40 .cell execution_count=3}\n``` {.python .cell-code}\nimport pystac_client\n\nclient = pystac_client.Client.open(STAC_ROOT_URL)\nsearch = client.search(\n datetime=temporal_extent,\n collections=[\"sentinel-2-l1c\"],\n bbox=[w, s, e, n]\n)\nitem_collection = search.item_collection()\n```\n:::\n\n\n \n:::\n \n:::\n\n:::{.callout-tip collapse=\"true\" title=\"level2 processing\"}\n:::{.panel-tabset group=\"query_engine\"}\n\n#### openEO\n\n::: {#f0ea342d .cell execution_count=4}\n``` {.python .cell-code}\nimport getpass\nfrom openeo.rest.stac_resource import StacResource\nfrom openeo.internal.graph_building import PGNode\n\n# Make sure to choose a unique name \nprocessing_name = \"Modena\"\n\nforce_l2_stac_resource = StacResource(\n graph=PGNode(\n process_id=\"force_level2\",\n arguments={\n \"stac_document\": query_pg,\n \"name\": processing_name,\n \"do_brdf\": True,\n }\n ),\n connection=connection,\n)\n\nmerge = f\"{getpass.getuser()}/experiments/{processing_name}\"\nforce_l2_stac_resource.export_workspace(\n workspace=\"apex-force-results-workspace\",\n merge=merge,\n);\n```\n:::\n\n\n\n\n#### Pystac client {.active}\n\n\n\n::: {#b963273a .cell execution_count=7}\n``` {.python .cell-code}\nfrom openeo.internal.graph_building import PGNode\nquery_catalog = transform_item_collection_to_catalog_with_links(item_collection)\n\nprocessing_name = \"Modena\"\n\nforce_l2_stac_resource = StacResource(\n graph=PGNode(\n process_id=\"force_level2\",\n arguments={\n \"stac_document\": query_catalog.to_dict(),\n \"name\": \"Modena\",\n \"do_brdf\": True,\n }\n ),\n connection=connection,\n)\n\nmerge = f\"{getpass.getuser()}/experiments/{processing_name}\"\nprint(merge)\nforce_l2_stac_resource.export_workspace(\n workspace=\"apex-force-results-workspace\",\n merge=merge,\n)\n```\n\n::: {.cell-output .cell-output-stdout}\n```\nhannes/experiments/Modena\n```\n:::\n\n::: {.cell-output .cell-output-display execution_count=7}\n```{=html}\n\n \n \n \n \n \n```\n:::\n:::\n\n\n \n:::\n\n```{.python}\nl2_job = force_l2_stac_resource.create_job()\nl2_job.start()\n```\n\n\n\n \n:::\n\n## Time Series Analysis\n\nThe openEO process for FORCE Time Series Analysis (TSA) is called `force_tsa`. Like the [level 2](level2.qmd) process, it takes most of the parameters usually passed to the [TSA module of FORCE](https://force-eo.readthedocs.io/en/latest/components/higher-level/tsa/) using the [parameter file](https://force-eo.readthedocs.io/en/latest/components/higher-level/tsa/param.html). Parameters use `snake_case` format instead of `SCREAMING_SNAKE_CASE` (used in FORCE parameter files).\n\nDetails of parametrization are discussed in the [dedicated article](parametrization.qmd).\n\n:::{.callout-note collapse=\"true\" title=\"FORCE TSA parameters\"}\n\n::: {#39173dcf .cell execution_count=9}\n``` {.python .cell-code}\nconnection.describe_process(\"force_tsa\")\n```\n\n::: {.cell-output .cell-output-display execution_count=9}\n```{=html}\n\n \n \n \n \n \n```\n:::\n:::\n\n\n:::\n\n### Using FORCE level 2 results as an input\n\nThe process graph for FORCE TSA built in the same way as the level 2 process graph is built.\nThe main input for FORCE TSA is the STAC catalog (or item) from the level 2 processing.\n\nWe can access the STAC catalog produced by our level 2 job in the workspace,\ndescribed in [level 2 processing article](level2.qmd#Accessing results).\n\nIt is important to remember the `merge` paramter passed to `export_workspace`, this is the path under which we can find the catalog.\n\n::: {#0a39712c .cell execution_count=10}\n``` {.python .cell-code}\nprocessing_name = \"Modena\"\nmerge = f\"{getpass.getuser()}/experiments/{processing_name}\"\nprint(merge)\n```\n\n::: {.cell-output .cell-output-stdout}\n```\nhannes/experiments/Modena\n```\n:::\n:::\n\n\nRemember, the dedicated workspace for FORCE has the URL \n\n\n\n::: {#336d5bc8 .cell execution_count=11}\n``` {.python .cell-code}\nWORKSPACE_URL = \"https://s3.waw4-1.cloudferro.com/apex-force-results-waw4-1-exotc5yuexi2c5tvwqhoivj62fz8v0uupy0me\"\n```\n:::\n\n\n### TSA Parameters\n\n\nSome parameters can be set a-priori or from prior knowledge of our level 2 processing parameters\n\n::: {#e0106455 .cell execution_count=12}\n``` {.python .cell-code}\ntemporal_extent=[\"2026-04-17\", \"2026-04-27\"]\n```\n:::\n\n\nFor other parameters, we may want to inspect the metadata of the level 2 processing.\nFor example, we can extract the tiles from the asset names of the STAC item describing the level 2 datacube. \n\n::: {#54dbfd3c .cell execution_count=13}\n``` {.python .cell-code}\nmerge = f\"{getpass.getuser()}/experiments/{processing_name}\"\nl2_catalog_url = f\"{WORKSPACE_URL}/{merge}/catalog.json\"\nprint(l2_catalog_url)\nl2_catalog = pystac.Catalog.from_file(l2_catalog_url)\nl2_item = next(l2_catalog.get_items()) # there is just one item\nl2_item\n```\n\n::: {.cell-output .cell-output-stdout}\n```\nhttps://s3.waw4-1.cloudferro.com/apex-force-results-waw4-1-exotc5yuexi2c5tvwqhoivj62fz8v0uupy0me/hannes/experiments/Modena/catalog.json\n```\n:::\n\n::: {.cell-output .cell-output-display execution_count=13}\n```{=html}\n\n\n\n
\n
\n <Item id=Modena-level2>\n
\n\n
\n```\n:::\n:::\n\n\nWe can extract the tiles manually or programmatically\n\n::: {#24b6c076 .cell execution_count=14}\n``` {.python .cell-code}\ntiles = set([k.split(\".\")[0] for k in l2_item.assets.keys() if k.endswith(\"BOA\")])\ntile_x = set(int(tile[1:5]) for tile in tiles)\ntile_y = set(int(tile[7:]) for tile in tiles)\ntile_x, tile_y\n```\n\n::: {.cell-output .cell-output-display execution_count=14}\n```\n({31, 32}, {29, 30})\n```\n:::\n:::\n\n\n### Process Graph\n\nThe main process is `force_tsa`.\n\n::: {#5851e5f5 .cell execution_count=15}\n``` {.python .cell-code}\nfrom openeo.rest.stac_resource import StacResource\nfrom openeo.internal.graph_building import PGNode\n\n\nforce_tsa_stac_resource = StacResource(\n graph=PGNode(\n process_id=\"force_tsa\",\n arguments={\n \"stac_url\": l2_catalog_url,\n \"name\": \"Modena_TSA\",\n \"date_range\": temporal_extent,\n \"x_tile_range\": [min(tile_x), min(tile_x)],\n \"y_tile_range\": [min(tile_y), max(tile_y)],\n \"stm\": [\"AVG\"],\n \"output_stm\": True,\n # ... more TSA parameters (optional)\n }\n ),\n connection=connection,\n)\n```\n:::\n\n\nWe can then create and start the job as before:\n\n```{.python}\ntsa_job = force_tsa_stac_resource.create_job(title=f\"FORCE TSA {processing_name}\")\ntsa_job.start() # use start_and_wait to block until finished\n```\n\nInspecting STAC metadata and downloading results works in the same way as for level 2 processing:\n\n\n\n```{.python}\ntsa_catalog_url = extract_catalog_url_from_job_logs(tsa_job.logs())\ntsa_catalog = pystac.Catalog.from_file(tsa_catalog_url)\ntsa_item = next(tsa_catalog.get_items())\ntsa_item\n```\n\nDownloading assets as well:\n\n```{.python}\ntsa_results = l2_job.get_results()\ntsa_results\n```\n\n```{.python}\ntsa_results.download_files(\"force-tsa-results\")\n```\n\n\n\n### Further processing\n\nBecause level 2 results are provided by a STAC URL to the TSA process, it is possible run multiple TSA jobs with the same level 2 input, without expensively reprocessing the level 2 data.\nWe just create a new TSA process graph and submit it, passing the same level 2 data cube URL.\n\nRemember that the job assets will be stored for some time on the backend which allows you to do processing over a span of a few days, but will not be stored permanently. Please download and archive results you wish to keep on your own infrastructure.\n\n", + "markdown": "---\ntitle: FORCE Time Series Analysis\norder: 5\n---\n\n## Recap\n\n:::{.callout-warning}\nYou should use `pystac_client` based querying if your inputs encompass more than just a handful of items\n:::\n\n:::{.callout-tip collapse=\"true\" title=\"Selecting input products\"}\n\n::: {#77d92ba2 .cell execution_count=1}\n``` {.python .cell-code}\nL1C_COLLECTION_URL = \"https://stac.dataspace.copernicus.eu/v1/collections/sentinel-2-l1c\"\nSTAC_ROOT_URL = \"https://stac.dataspace.copernicus.eu/v1\"\n\nw,s,e,n = 10.386, 44.437, 11.423, 44.973\nspatial_extent = { \"west\": w, \"south\": s, \"east\": e, \"north\": n}\n# ATTENTION: Inclusion of the second date in the search depends on the query method (see below).\ntemporal_extent=[\"2026-04-17\", \"2026-05-01\"]\n```\n:::\n\n\n:::{.panel-tabset group=\"query_engine\"}\n\n\n## openEO\n\n::: {#f6630c0f .cell execution_count=2}\n``` {.python .cell-code}\nimport openeo\nconnection = openeo.connect(\"openeo.dataspace.copernicus.eu\").authenticate_oidc()\n\nquery_pg = openeo.processes.process(\n \"query_stac\",\n arguments={\n \"url\": L1C_COLLECTION_URL,\n \"temporal_extent\": temporal_extent,\n \"spatial_extent\": spatial_extent,\n }\n)\n```\n\n::: {.cell-output .cell-output-stdout}\n```\nAuthenticated using refresh token.\n```\n:::\n:::\n\n\n## Pystac client {.active}\n\n::: {#babb0f2e .cell execution_count=3}\n``` {.python .cell-code}\nimport pystac_client\n\nclient = pystac_client.Client.open(STAC_ROOT_URL)\nsearch = client.search(\n datetime=temporal_extent,\n collections=[\"sentinel-2-l1c\"],\n bbox=[w, s, e, n]\n)\nitem_collection = search.item_collection()\n```\n:::\n\n\n \n:::\n \n:::\n\n:::{.callout-tip collapse=\"true\" title=\"level2 processing\"}\n:::{.panel-tabset group=\"query_engine\"}\n\n#### openEO\n\n::: {#6f503038 .cell execution_count=4}\n``` {.python .cell-code}\nimport getpass\nfrom openeo.rest.stac_resource import StacResource\nfrom openeo.internal.graph_building import PGNode\n\n# Make sure to choose a unique name\nprocessing_name = \"Modena\"\n\nforce_l2_stac_resource = StacResource(\n graph=PGNode(\n process_id=\"force_level2\",\n arguments={\n \"stac_document\": query_pg,\n \"name\": processing_name,\n \"do_brdf\": True,\n }\n ),\n connection=connection,\n)\n\nmerge = f\"{getpass.getuser()}/experiments/{processing_name}\"\nforce_l2_stac_resource.export_workspace(\n workspace=\"apex-force-results-workspace\",\n merge=merge,\n);\n```\n:::\n\n\n\n\n#### Pystac client {.active}\n\n\n\n::: {#4467d67d .cell execution_count=7}\n``` {.python .cell-code}\nfrom openeo.internal.graph_building import PGNode\nfrom openeo.rest.stac_resource import StacResource\n\nquery_catalog = transform_item_collection_to_catalog_with_links(item_collection)\n\nprocessing_name = \"Modena\"\n\nforce_l2_stac_resource = StacResource(\n graph=PGNode(\n process_id=\"force_level2\",\n arguments={\n \"stac_document\": query_catalog.to_dict(),\n \"name\": processing_name,\n \"do_brdf\": True,\n # level 2 parameters\n }\n ),\n connection=connection,\n)\n\nmerge = f\"{getpass.getuser()}/experiments/{processing_name}\"\nprint(merge)\nforce_l2_stac_resource.export_workspace(\n workspace=\"apex-force-results-workspace\",\n merge=merge,\n)\n```\n\n::: {.cell-output .cell-output-stdout}\n```\nhannes/experiments/Modena\n```\n:::\n\n::: {.cell-output .cell-output-display execution_count=7}\n```{=html}\n\n \n \n \n \n \n```\n:::\n:::\n\n\n \n:::\n\n```{.python}\nl2_job = force_l2_stac_resource.create_job()\nl2_job.start()\n```\n\n\n\n \n:::\n\n## Time Series Analysis\n\nThe openEO process for FORCE Time Series Analysis (TSA) is called `force_tsa`. Like the [level 2](level2.qmd) process,\nit takes most of the parameters usually passed to the\n[TSA module of FORCE](https://force-eo.readthedocs.io/en/latest/components/higher-level/tsa/) using the\n[parameter file](https://force-eo.readthedocs.io/en/latest/components/higher-level/tsa/param.html).\nParameters use `snake_case` format instead of `SCREAMING_SNAKE_CASE` (used in FORCE parameter files).\n\nDetails of parametrization are discussed in the [dedicated article](parametrization.qmd).\n\n```{.python}\nconnection = openeo.connect(\"openeo.dataspace.copernicus.eu\").authenticate_oidc()\n```\n\n\n\n::: {#a33d1a41 .cell execution_count=9}\n\n::: {.cell-output .cell-output-stdout}\n```\nAuthenticated using refresh token.\n```\n:::\n:::\n\n\n:::{.callout-note collapse=\"true\" title=\"FORCE TSA parameters\"}\n\n::: {#b89da13c .cell execution_count=10}\n``` {.python .cell-code}\nconnection.describe_process(\"force_tsa\")\n```\n\n::: {.cell-output .cell-output-display execution_count=10}\n```{=html}\n\n \n \n \n \n \n```\n:::\n:::\n\n\n:::\n\n\n\n::: {#72f27c4f .cell execution_count=11}\n\n::: {.cell-output .cell-output-stdout}\n```\nAuthenticated using refresh token.\n```\n:::\n:::\n\n\n### Using FORCE level 2 results as an input\n\nThe process graph for FORCE TSA built in the same way as the level 2 process graph is built.\nThe main input for FORCE TSA is the STAC catalog (or item) from the level 2 processing.\n\nWe can access the STAC catalog produced by our level 2 job in the workspace,\ndescribed in [level 2 processing article](level2.qmd#Accessing results).\n\n:::{.callout-important}\nIt is important to remember the `merge` parameter passed to `export_workspace`, this is the path under which we can find\nthe catalog later.\n:::\n\n::: {#553fb766 .cell execution_count=12}\n``` {.python .cell-code}\nprocessing_name = \"Modena\"\nmerge = f\"{getpass.getuser()}/experiments/{processing_name}\"\nprint(merge)\n```\n\n::: {.cell-output .cell-output-stdout}\n```\nhannes/experiments/Modena\n```\n:::\n:::\n\n\nRemember, the dedicated workspace for FORCE has the URL\n\n\n\n::: {#252c95ee .cell execution_count=13}\n``` {.python .cell-code}\nWORKSPACE_URL = \"https://s3.waw4-1.cloudferro.com/apex-force-results-waw4-1-exotc5yuexi2c5tvwqhoivj62fz8v0uupy0me\"\n```\n:::\n\n\n### TSA Parameters\n\n\nSome parameters can be set a-priori or from prior knowledge of our level 2 processing parameters\n\n::: {#1173c5ad .cell execution_count=14}\n``` {.python .cell-code}\ntemporal_extent=[\"2026-04-17\", \"2026-04-27\"]\n```\n:::\n\n\nFor other parameters, we may want to inspect the metadata of the level 2 processing.\nFor example, we can extract the tiles from the asset names of the STAC item describing the level 2 datacube.\n\n::: {#64561a18 .cell execution_count=15}\n``` {.python .cell-code}\nmerge = f\"{getpass.getuser()}/experiments/{processing_name}\"\nl2_catalog_url = f\"{WORKSPACE_URL}/{merge}/catalog.json\"\nprint(l2_catalog_url)\nl2_catalog = pystac.Catalog.from_file(l2_catalog_url)\nl2_item = next(l2_catalog.get_items()) # there is just one item\nl2_item\n```\n\n::: {.cell-output .cell-output-stdout}\n```\nhttps://s3.waw4-1.cloudferro.com/apex-force-results-waw4-1-exotc5yuexi2c5tvwqhoivj62fz8v0uupy0me/hannes/experiments/Modena/catalog.json\n```\n:::\n\n::: {.cell-output .cell-output-display execution_count=15}\n```{=html}\n\n\n\n
\n
\n <Item id=Modena-level2>\n
\n\n
\n```\n:::\n:::\n\n\nWe can extract the tiles manually or programmatically\n\n::: {#475747a4 .cell execution_count=16}\n``` {.python .cell-code}\ntiles = set([k.split(\".\")[0] for k in l2_item.assets.keys() if k.endswith(\"BOA\")])\ntile_x = set(int(tile[1:5]) for tile in tiles)\ntile_y = set(int(tile[7:]) for tile in tiles)\ntile_x, tile_y\n```\n\n::: {.cell-output .cell-output-display execution_count=16}\n```\n({31, 32}, {29, 30})\n```\n:::\n:::\n\n\n### Process Graph\n\nThe main process is `force_tsa`.\n\n::: {#556cd5d9 .cell execution_count=17}\n``` {.python .cell-code}\nfrom openeo.rest.stac_resource import StacResource\nfrom openeo.internal.graph_building import PGNode\n\ntsa_processing_name = f\"{processing_name}_TSA\"\n\nforce_tsa_stac_resource = StacResource(\n graph=PGNode(\n process_id=\"force_tsa\",\n arguments={\n \"stac_url\": l2_catalog_url,\n \"name\": tsa_processing_name,\n \"date_range\": temporal_extent,\n \"x_tile_range\": [min(tile_x), min(tile_x)],\n \"y_tile_range\": [min(tile_y), max(tile_y)],\n \"stm\": [\"AVG\"],\n \"output_stm\": True,\n # ... more TSA parameters (optional)\n }\n ),\n connection=connection,\n)\n```\n:::\n\n\nWe can then create and start the job as before:\n\n```{.python}\ntsa_job = force_tsa_stac_resource.create_job(title=tsa_processing_name)\ntsa_job.start() # use start_and_wait to block until finished\n```\n\n\n\nDownloading assets works the same way as for level2:\n\n::: {#3847115b .cell execution_count=19}\n``` {.python .cell-code}\ntsa_results = tsa_job.get_results()\ntsa_results\n```\n\n::: {.cell-output .cell-output-display execution_count=19}\n```{=html}\n\n \n \n \n \n \n```\n:::\n:::\n\n\n::: {#491217bf .cell execution_count=20}\n``` {.python .cell-code}\ntsa_results.download_files(\"force-tsa-results\")\n```\n\n::: {.cell-output .cell-output-display execution_count=20}\n```\n[PosixPath('force-tsa-results/CITEME_0x31.txt'),\n PosixPath('force-tsa-results/X0031_Y0029/20260417-20260427_001-365_HL_TSA_SEN2L_EVI_STM.tif'),\n PosixPath('force-tsa-results/X0031_Y0029/20260417-20260427_001-365_HL_TSA_SEN2L_NBR_STM.tif'),\n PosixPath('force-tsa-results/X0031_Y0029/20260417-20260427_001-365_HL_TSA_SEN2L_NDVI_STM.tif'),\n PosixPath('force-tsa-results/X0031_Y0030/20260417-20260427_001-365_HL_TSA_SEN2L_EVI_STM.tif'),\n PosixPath('force-tsa-results/X0031_Y0030/20260417-20260427_001-365_HL_TSA_SEN2L_NBR_STM.tif'),\n PosixPath('force-tsa-results/X0031_Y0030/20260417-20260427_001-365_HL_TSA_SEN2L_NDVI_STM.tif'),\n PosixPath('force-tsa-results/datacube-definition.prj'),\n PosixPath('force-tsa-results/job-results.json')]\n```\n:::\n:::\n\n\n### Further processing\n\nBecause level 2 results are provided by a STAC URL to the TSA process, it is possible run multiple TSA jobs with the same level 2 input, without expensively reprocessing the level 2 data.\nWe just create a new TSA process graph and submit it, passing the same level 2 data cube URL.\n\nRemember that the job assets will be stored for some time on the backend which allows you to do processing over a span of a few days, but will not be stored permanently. Please download and archive results you wish to keep on your own infrastructure.\n\n", "supporting": [ "tsa_files" ], diff --git a/docs/force/.gitignore b/docs/force/.gitignore deleted file mode 100644 index ad29309..0000000 --- a/docs/force/.gitignore +++ /dev/null @@ -1,2 +0,0 @@ -/.quarto/ -**/*.quarto_ipynb diff --git a/docs/force/SOURCE.yml b/docs/force/SOURCE.yml index 80994ed..77056b6 100644 --- a/docs/force/SOURCE.yml +++ b/docs/force/SOURCE.yml @@ -1,4 +1,4 @@ repository: bcdev/apex-force-openeo release: manual -commit: e0a10dd5385cd1c0798b1847e3e738aa961e5974 -Generated: Fri Jul 24 14:19:48 UTC 2026 +commit: d6917a4828a21ac94a563f6efd603949b7b83fc7 +Generated: Thu Aug 20 08:30:08 UTC 2026 diff --git a/docs/force/feature_overview.qmd b/docs/force/feature_overview.qmd index bd71d64..3abc406 100644 --- a/docs/force/feature_overview.qmd +++ b/docs/force/feature_overview.qmd @@ -1,10 +1,9 @@ --- title: Features and Limitations - --- -The FORCE integration into to CDSE is in a proof-of-concept stage. It is possible to use the FORCE level 2 and Time Series Analysis (TSA) modules on a small scale. Most parameters supported by FORCE are available through the openEO interface. +The FORCE integration into to CDSE is in a proof-of-concept stage. It is possible to use the FORCE level 2 and Time Series Analysis (TSA) modules on a small scale. Most parameters supported by FORCE are available through the openEO interface. Unsupported parameters are usually either managed by the implementation (parallelization options, input/output directores) or reuqire supplementary files which are not avaiable on the backend. The `aoi` (openEO interface) / `FILE_AOI` (FORCE parameter file) parameter is good example of a parameter that is exposed in a different way in openEO to avoid the file interface. Instead of a shape file, a GEOJSON string is passed. @@ -37,6 +36,6 @@ If you encounter limitations not mentioned here, please raise an issue in the [G ### Parallelizing large Time Series Analysis jobs Server-side parallelization of the TSA module is not currently enabled. It is possible to parallelize TSA by creating one openEO job per tile by setting the `x_tile_range` and `y_tile_range` arguments to a single tile per job. -Furthermore, consider computing spatio-temporal metrics (STM) in separate jobs to reduce memory usage. +Furthermore, consider computing spatio-temporal metrics (STM) in separate jobs to reduce memory usage. Finally, you may reduce the `chunk_size` which will reduce memory usage at the cost of a longer processing time. diff --git a/docs/force/guide/intro.qmd b/docs/force/guide/intro.qmd index 69f9e63..3ffce97 100644 --- a/docs/force/guide/intro.qmd +++ b/docs/force/guide/intro.qmd @@ -10,26 +10,34 @@ Instead of installing FORCE locally, manually downloading data and annotating yo ## Prerequisites -This guide assumes basic familiarity with the FORCE processing engine. +This guide assumes basic familiarity with the FORCE processing engine and Python. You do not need to be an expert FORCE user to make use of the CDSE deployment. +All examples use the [openEO Python client](https://open-eo.github.io/openeo-python-client/). Due to openEO's +language-agnostic architecture, you should be able to replicate all behavior with other client libraries as well +(e.g., [R](https://openeo.org/documentation/1.0/r/), [Javascript](https://openeo.org/documentation/1.0/javascript/)), +although this has not been tested. + ::: {.callout-tip title="What to read if you are unfamiliar with FORCE"} -If you are an openEO user who wants to get familiar with FORCE, we recommend to have a look at the FORCE documentation pages for +If you are an openEO user who wants to get familiar with FORCE, we recommend to first have a look at the FORCE documentation pages for: - **The [data cube](https://force-eo.readthedocs.io/en/latest/howto/datacube.html) concept and organization**: This is the file structure you will receive when running FORCE processing modules -- **The [level 2 processing system](https://force-eo.readthedocs.io/en/latest/howto/l2-ard.html#level-2-ard)**: the basic FORCE module to generate analysis ready data cubes from raw Satellite input products +- **The [level 2 processing system](https://force-eo.readthedocs.io/en/latest/howto/l2-ard.html#level-2-ard)**: the basic FORCE module to generate Analysis Ready Data Cubes from raw Satellite input products - (Optional) **The [Time Series Analysis (TSA)](https://force-eo.readthedocs.io/en/latest/components/higher-level/tsa/index.html#time-series-analysis) module**: This is the higher level processing module made available through openEO ::: ::: {.callout-tip title="What to read if you are unfamiliar with openEO"} -If you are a FORCE user curious how to process in the cloud and the openEO interface, it will be helpful to be familiar with +If you are a FORCE user curious how to process in the cloud and the openEO interface, it will be helpful to be familiar with: - The [openEO vocabulary](https://openeo.org/documentation/1.0/glossary.html#processes) -- **The [openEO Python client](https://open-eo.github.io/openeo-python-client/api.html)**: The main interface used to interact with the openEO backend. You may want to use the [R client](https://open-eo.github.io/openeo-r-client/) or [Javascript client](https://open-eo.github.io/openeo-js-client/latest/) instead, if you prefer. However, in this guide, examples will be given with the Python client. +- **The [openEO Python client](https://open-eo.github.io/openeo-python-client/api.html)**: The main interface used to interact with the openEO backend. + You may want to use the [R client](https://open-eo.github.io/openeo-r-client/) or [Javascript client](https://open-eo.github.io/openeo-js-client/latest/) instead, if you prefer. + However, in this guide, examples will be given exclusively with the Python client. ::: ::: {.callout-important title="Data cube representation"} -Please note that the FORCE integration does not make use of [openEO's data cube concept](https://openeo.org/documentation/1.0/datacubes.html), processing is done on FORCE's native data cubes. +Please note that the FORCE integration does not make use of [openEO's data cube concept](https://openeo.org/documentation/1.0/datacubes.html), +processing is done on FORCE's native data cubes. Therefore, other openEO processes cannot be applied to FORCE data cubes. ::: ## Scope @@ -37,10 +45,10 @@ Please note that the FORCE integration does not make use of [openEO's data cube In this guide, you will learn how to create a full processing pipeline using FORCE and openEO: 1. Discover inputs: Query a STAC catalog to determine input products -2. Generate an Analysis Ready (ARD) data cube with the FORCE level 2 processing system +2. Generate an Analysis Ready data cube with the FORCE level 2 processing system 3. Analyze time series with FORCE's higher level processing system -On the way, we will cover how to +On the way, we will cover how to: - Download data cubes for local processing / visualization / permanent storage - Apply higher level processing without having to download intermediate results diff --git a/docs/force/guide/level2.qmd b/docs/force/guide/level2.qmd index a19fc16..91317f3 100644 --- a/docs/force/guide/level2.qmd +++ b/docs/force/guide/level2.qmd @@ -1,11 +1,8 @@ --- -execute: - freeze: true +title: FORCE level 2 order: 4 --- -# FORCE level 2 - ## Recap :::{.callout-tip collapse="true" title="Selecting input products"} @@ -14,7 +11,6 @@ order: 4 L1C_COLLECTION_URL = "https://stac.dataspace.copernicus.eu/v1/collections/sentinel-2-l1c" STAC_ROOT_URL = "https://stac.dataspace.copernicus.eu/v1" -#w,s,e,n = 10.386, 44.437, 11.423, 44.973 w, s, e, n = 11.0, 44.5, 11.1, 44.6 spatial_extent = { "west": w, "south": s, "east": e, "north": n} # ATTENTION: Inclusion of the second date in the search depends on the query method. @@ -70,7 +66,7 @@ The integrated FORCE uses (lowercase) [`snake_case`](https://en.wikipedia.org/wi The parameters are documented in the process description. We can inspect it using the Python client's [`describe_processes()`](https://open-eo.github.io/openeo-python-client/api.html#openeo.rest.connection.Connection.describe_process). - ```{python} @@ -100,10 +97,15 @@ connection.describe_process("force_level2") ``` ::: +```{python} +#| echo: false +connection = openeo.connect("openeo.dataspace.copernicus.eu").authenticate_oidc() +``` + ### Building the Process Graph The FORCE process graph is based on the `force_level2` process, as described in [openEO processes](processes.qmd). Because FORCE does not use the standard openEO version of the "data cube" concept, but -uses its own data cube representation, we need to wrap the process so that we can interact with the results in a convenient way as a StacResource and export the processing results to a workspace. +uses its own data cube representation, we need to wrap the process so that we can interact with the results in a convenient way as a `StacResource` and export the processing results to a workspace. Concept | Qualified name (Python client) | Purpose @@ -123,7 +125,7 @@ When running FORCE level2 on its own, we do not need to use a workspace. Simply We can instruct openEO to store our results in a workspace using the [`export_workspace`](https://openeo.org/documentation/1.0/processes.html#export_workspace) openEO process. -When using `export_workspace`, we need to pass a `merge` string to identify your result. The merge will be a key prefix on object storage. The `merge` is something similar to a path in a file system. +When using `export_workspace`, we need to pass a `merge` string to identify your result. The merge will be a key prefix on object storage. The `merge` is something similar to a path in a file system. It is very important to set a good `merge` parameter and remember the result, because results can only be accessed when the `merge` is known. Furthermore, all users of the workspace (in the case of the `apex-force-results-workspace`, anyone) may write data, so make sure to use a unique `merge`, to reduce the likelihood of accidentally modifying others' data and have others overwrite your data by accident. @@ -131,7 +133,9 @@ It is very important to set a good `merge` parameter and remember the result, be # FORCE workspace The dedicated workspace for FORCE results is accessible to all users: -`apex-force-results-workspace` () + +- id: `apex-force-results-workspace` +- url: https://s3.waw4-1.cloudferro.com/apex-force-results-waw4-1-exotc5yuexi2c5tvwqhoivj62fz8v0uupy0me ::: @@ -367,7 +371,7 @@ While the cube will stay in place for multiple days, eventually the storage will -```{python} -#| echo: false -WORKSPACE_URL = "https://s3.waw4-1.cloudferro.com/apex-force-results-waw4-1-exotc5yuexi2c5tvwqhoivj62fz8v0uupy0me" -``` - - #### Downloading files :::{.callout-tip} @@ -424,21 +421,15 @@ This step is optional if you plan to continue processing with FORCE on CDSE ::: Assets can be downloaded using the standard openEO mechanism to access result assets. -The STAC metadata specifies the directory structure of the resulting cube. This structure is respected by openEO's [`download_files`](https://open-eo.github.io/openeo-python-client/api.html#openeo.rest.job.JobResults.download_files). +The STAC metadata specifies the directory structure of the resulting cube. This structure is respected by openEO's [`download_files`](https://open-eo.github.io/openeo-python-client/api.html#openeo.rest.job.JobResults.download_files). To inspect a particular asset, we may use [`download_file`](https://open-eo.github.io/openeo-python-client/api.html#openeo.rest.job.JobResults.download_file) to download singular files instead. -```{.python} +```{python} l2_results = l2_job.get_results() l2_results ``` -```{.python} +```{python} l2_results.download_files("force-level2-results") -``` - - +``` \ No newline at end of file diff --git a/docs/force/guide/parametrization.qmd b/docs/force/guide/parametrization.qmd index 12ecba1..174789c 100644 --- a/docs/force/guide/parametrization.qmd +++ b/docs/force/guide/parametrization.qmd @@ -1,17 +1,13 @@ --- -execute: - freeze: true +title: Parametrization order: 6 --- - -# Parametrization - The [FORCE openEO processes](processes.qmd) (`force_level2` and `force_tsa`) accept most of the parameters that can be passed using FORCE's native parametrization files ([level 2](https://force-eo.readthedocs.io/en/latest/components/lower-level/level2/param.html), [TSA](https://force-eo.readthedocs.io/en/latest/components/higher-level/tsa/param.html)). The exceptions to this rule are explained below. -There are 4 types of parameters that do not correspond exactly to the FORCE parameter files +There are 4 types of parameters that do not correspond exactly to the FORCE parameter files 1. Special parameters that require translation (e.g., the area of interest) -1. Additional parameters of the openEO integration (e.g., the processing name) +1. Additional parameters of the openEO integration (e.g., the processing `name`) 1. FORCE parameters that are not exposed through the openEO API (e.g., the number of processes used for processing) 1. Parameters that support only a subset of the possible values @@ -20,22 +16,27 @@ It is possible to inspect the processes (and see their parameters) using openEO' Parameter values may be strings (the same as in FORCE parameter files) or native python values which can be serialized to JSON. Values are converted to UPPERCASE before being passed into the parameter file. -Parameter values are passed as a dictionary to the `arguments` parameter of `openeo.processes.process`: +Parameter values are passed as a dictionary to the `arguments` parameter of `openeo.internal.graph_building.PGNode`: ```{.python} -graph = openeo.processes.process( +from openeo.internal.graph_building import PGNode + +graph = PGNode( process_id="force_level2", - arguments=dict( - stac_document=stac_catalog, - name="Modena", - do_brdf=True, - # ... <--- more parameters + arguments={ + "stac_document": stac_catalog, + "name": "Modena", + # FORCE parameters + "do_brdf": True, + # ... <--- more FORCE parameters } ) ``` +This process must be wrapped as a `StacResource` as described in [FORCE level 2](level2.qmd). + ## Examples | FORCE parameter file | openEO | @@ -77,7 +78,7 @@ Input/Output directories (managed by the integration) Parallel processing parameters (managed by the integration) -- `NPROC` +- `NPROC` - `NTHREAD` - `PARALLEL_READS` - `DELAY` diff --git a/docs/force/guide/processes.qmd b/docs/force/guide/processes.qmd index 8494e6b..30f0dcb 100644 --- a/docs/force/guide/processes.qmd +++ b/docs/force/guide/processes.qmd @@ -1,16 +1,19 @@ --- -execute: - # Cache is necessary to avoid having to authenticate to build the documentation - freeze: true - +title: "openEO processes" order: 2 --- -# openEO processes +Using an openEO client, we can control FORCE through a number of [openEO processes](https://openeo.org/documentation/1.0/datacubes.html). +The FORCE openEO integration is specific to the backend deployed on CDSE, so the processes described here are +not listed among the [standard processes](https://openeo.org/documentation/1.0/processes.html) and are generally experimental. -Using an openEO client, we can control FORCE through a number of [openEO processes](https://openeo.org/documentation/1.0/datacubes.html). The FORCE openEO integration is specific to the backend deployed on CDSE, so the processes described here are not listed among the [standard processes](https://openeo.org/documentation/1.0/processes.html) and are generally experimental. +This page gives an overview of the openEO processes specific to FORCE and client operations needed to interact +with them. -The processes wrapping FORCE functionality support most of the options that can be set in the FORCE parameter file ([level2](https://force-eo.readthedocs.io/en/latest/components/lower-level/level2/param.html), [TSA](https://force-eo.readthedocs.io/en/latest/components/higher-level/tsa/param.html)). See the article on [parametrization](parametrization.qmd) for more details on the supported options and defaults. +The processes wrapping FORCE functionality support most of the options that can be set in the FORCE parameter file +([level2](https://force-eo.readthedocs.io/en/latest/components/lower-level/level2/param.html), +[TSA](https://force-eo.readthedocs.io/en/latest/components/higher-level/tsa/param.html)). +See the article on [parametrization](parametrization.qmd) for more details on the supported options and defaults. To discover the processes, we must first connect to the CDSE openEO backend: @@ -20,12 +23,12 @@ import openeo connection = openeo.connect("https://openeo.dataspace.copernicus.eu") connection.authenticate_oidc() ``` - @@ -35,15 +38,25 @@ connection.authenticate_oidc() connection = openeo.connect("openeo-staging.dataspace.copernicus.eu").authenticate_oidc() ``` - - -If you don't have your [credentials set as environment variables](https://open-eo.github.io/openeo-python-client/auth.html#oidc-client-credentials-using-environment-variables), you will be prompted to log in in a browser window by default. See also the openEO documentation on authentication using Open ID connect ([overview](https://openeo.org/documentation/1.0/authentication.html#device-flow), [Python client](https://openeo.org/documentation/1.0/authentication.html)) +If you don't have your +[credentials set as environment variables](https://open-eo.github.io/openeo-python-client/auth.html#oidc-client-credentials-using-environment-variables), +you will be prompted to log in in a browser window by default. See also the openEO documentation on authentication using +Open ID connect ([overview](https://openeo.org/documentation/1.0/authentication.html), +[Python client](https://open-eo.github.io/openeo-python-client/auth.html#oidc-client-credentials-using-environment-variables)) +We can use openEO's +[`describe_process`](https://open-eo.github.io/openeo-python-client/api.html#openeo.rest.connection.Connection.describe_process) +API call to discover the parameters of an openEO process. ## Query +As a first step, we must determine the input products for FORCE to process. This is best achieved by an external +querying method, such as [`pystac_client`](https://pystac-client.readthedocs.io/en/stable/). +Alternatively, CDSE openEO provides a process `query_stac` to determine input products from `spatial_extent` and +`temporal_extent`. Using this process in combination with `force_level2` is currently limited to very few input +products, therefore we recommend using it only for small tests. -We can use either the openEO process `query_stac` or another method to select input products for processing with FORCE. See the [next section](query.qmd) for an in depth discussion on the different methods how inputs may be specified. +See the [next section](query.qmd) for an in-depth discussion on the different methods how inputs may be specified. :::{.callout-note collapse="true" title="query_stac process definition"} @@ -52,30 +65,28 @@ connection.describe_process("query_stac") ``` ::: - - -We can use openEO's `describe_process` API call to discover the parameters of an openEO process. - - ## Level 2 - The openEO integration of FORCE provides a `force_level2` process wrapping the [FORCE level 2 processing system](https://force-eo.readthedocs.io/en/latest/components/lower-level/level2/). +The openEO processes exposes most of the +[options that can be set in the FORCE level 2 parameter file](https://force-eo.readthedocs.io/en/latest/components/lower-level/level2/param.html#l2-param). +Exceptions and additional parameters are described in the [parametrization article](parametrization.qmd). + :::{.callout-note collapse="true" title="force_level2 process definition"} ```{python} connection.describe_process("force_level2") ``` ::: -The openEO processes exposes most of the options that can be set in the FORCE parameter file. - - - ## Time Series Analysis The `force_tsa` process wraps the [Time Series Analysis](https://force-eo.readthedocs.io/en/latest/components/higher-level/tsa/) module. +The openEO processes exposes most of the +[options that can be set in the FORCE TSA parameter file](https://force-eo.readthedocs.io/en/latest/components/higher-level/tsa/param.html). +Exceptions and additional parameters are described in the [parametrization article](parametrization.qmd). + :::{.callout-note collapse="true" title="force_tsa process definition"} ```{python} connection.describe_process("force_tsa") @@ -90,16 +101,17 @@ Please refer to the [Python client documentation](https://open-eo.github.io/open Name | Qualified name (Python client) | Purpose -----|----------------|-------- -`create_job` | [`openeo.rest.connection.Connection.create_job`](https://open-eo.github.io/openeo-python-client/api.html#openeo.rest.connection.Connection.create_job) | Submit a Process Graph to the backend +`create_job` | [`openeo.rest.connection.Connection.create_job`](https://open-eo.github.io/openeo-python-client/api.html#openeo.rest.connection.Connection.create_job) | Submit a Process Graph to the backend `start_and_wait` | [`openeo.rest.job.BatchJob.start_and_wait`](https://open-eo.github.io/openeo-python-client/api.html#openeo.rest.job.BatchJob.start_and_wait) | Start a submitted job and block until completion `start` | [`openeo.rest.job.BatchJob.start`](https://open-eo.github.io/openeo-python-client/api.html#openeo.rest.job.BatchJob.start) | Same as `start_and_wait` but do not block execution `get_results` | [`openeo.rest.job.BatchJob.get_results`](https://open-eo.github.io/openeo-python-client/api.html#openeo.rest.job.BatchJob.get_results) | Get a `JobResults` object from a completed Job (contains references to output files) `download_files` | [`openeo.rest.job.JobResults.download_files`](https://open-eo.github.io/openeo-python-client/api.html#openeo.rest.job.JobResults.download_files) | Download all assets (files) produced by a job `download_file` | [`openeo.rest.job.JobResults.download_file`](https://open-eo.github.io/openeo-python-client/api.html#openeo.rest.job.JobResults.download_file) | Download a singular asset (file) produced by a job +`export_workspace` | [`openeo.rest.stac_resource.StacResource.export_workspace`](https://open-eo.github.io/openeo-python-client/api.html#openeo.rest.stac_resource.StacResource.export_workspace) | Export raw files (FORCE data cube) and STAC to remote cloud storage. Used for asynchronous further processing (TSA) without download of the intermediate result. ## Monitoring progress When using `start_and_wait` to start a job, status updates will automatically be printed to the command line or jupyter output cell. -Alternatively, progress and logs can be monitored using the openEO web interface at [https://openeo.dataspace.copernicus.eu](https://open-eo.github.io/openeo-python-client/api.html#openeo.rest.job.JobResults.download_files) or use the client as explained e.g., in the [Python client documentation](https://open-eo.github.io/openeo-python-client/batch_jobs.html#create-a-batch-job). +Alternatively, progress and logs can be monitored using the openEO web interface at [https://openeo.dataspace.copernicus.eu](https://open-eo.github.io/openeo-python-client/api.html#openeo.rest.job.JobResults.download_files) or use the client as explained in the [Python client documentation](https://open-eo.github.io/openeo-python-client/batch_jobs.html#create-a-batch-job). diff --git a/docs/force/guide/query.qmd b/docs/force/guide/query.qmd index 3fd9313..97a8e8a 100644 --- a/docs/force/guide/query.qmd +++ b/docs/force/guide/query.qmd @@ -1,24 +1,21 @@ --- -execute: - freeze: true +title: Selecting input data order: 3 --- -# Selecting input data - When running FORCE level 2 locally, we are responsible to manually provide the necessary input data and reference their full path in the `FILE_QUEUE` parameter of the [FORCE level 2 parameter file](https://force-eo.readthedocs.io/en/latest/components/lower-level/level2/param.html). When running on CDSE, we do not have access to the excuting environment and its filesystem directly. -Downloads, file management and creation of the file queue are managed by the integration. -To specify which files should be processed, we provide a [STAC](https://stacspec.org/en) document ([Catalog](https://github.com/radiantearth/stac-spec/blob/master/catalog-spec/catalog-spec.md), [Collection](https://github.com/radiantearth/stac-spec/blob/master/collection-spec/collection-spec.md), [Item](https://github.com/radiantearth/stac-spec/blob/master/item-spec/item-spec.md) or [ItemCollection](https://github.com/radiantearth/stac-api-spec/blob/main/fragments/itemcollection/README.md)) to the process. +Downloads, file management and creation of the file queue are managed by the integration. +To specify which files should be processed, we provide a [STAC](https://stacspec.org/en) document ([Catalog](https://github.com/radiantearth/stac-spec/blob/master/catalog-spec/catalog-spec.md), [Collection](https://github.com/radiantearth/stac-spec/blob/master/collection-spec/collection-spec.md), [Item](https://github.com/radiantearth/stac-spec/blob/master/item-spec/item-spec.md) or [ItemCollection](https://github.com/radiantearth/stac-api-spec/blob/main/fragments/itemcollection/README.md)) to the process. :::{.callout-warning} -Make sure that the STAC document provided to `force_level2` only references the files that should actually be processed. The FORCE integration will attempt to download all products referenced in the document, so it is inadvisable to provide a reference to the entire Sentinel-2 L1C collection on CDSE for example +Make sure that the STAC document provided to `force_level2` only references the files that should actually be processed. The FORCE integration will attempt to download all products referenced in the document, so it is inadvisable to provide a reference to something like the entire Sentinel-2 L1C collection on CDSE. ::: ## STAC -The integrated FORCE and openEO more generally uses the [SpatioTemporal Asset Catalog (STAC) specification](https://stacspec.org/en) to identify (groups of) EO products and annotate them with metadata. +The integrated FORCE and openEO more generally uses the [SpatioTemporal Asset Catalog (STAC) specification](https://stacspec.org/en) to identify (groups of) EO products and annotate them with metadata. If you are unfamiliar with STAC, we recommend that you familiarize yourself with the basic components of the specification (What are catalogs, items and feature collections?) [on the official STAC specification website](https://stacspec.org/en) before proceeding. @@ -63,7 +60,7 @@ import geopandas as gpd from shapely import Polygon def plot_area_of_interest( - w, s, e, n, *, + w, s, e, n, *, context=5, color="red", linewidth=2, @@ -81,7 +78,7 @@ def plot_area_of_interest( if not show_tick_labels: ax.set_xticks([]) ax.set_yticks([]) - ctx.add_basemap(ax, source=ctx.providers.OpenStreetMap.Mapnik, crs=gdf.crs) + ctx.add_basemap(ax, source=ctx.providers.OpenTopoMap, crs=gdf.crs) fig.suptitle(title) fig.tight_layout() @@ -99,6 +96,9 @@ plot_area_of_interest( ### Querying a catalog +It is strongly recommended to use an external query method (such as `pystac_client`), because using the `query_stac` +process in combination with `force_level2` is limited to small numbers of items. + :::{.panel-tabset group="query_engine"} #### openEO @@ -120,7 +120,7 @@ query_pg = openeo.processes.process( ) ``` - ## Recap @@ -38,7 +30,7 @@ temporal_extent=["2026-04-17", "2026-05-01"] ```{python} import openeo -connection = openeo.connect("openeo-staging.dataspace.copernicus.eu").authenticate_oidc() +connection = openeo.connect("openeo.dataspace.copernicus.eu").authenticate_oidc() query_pg = openeo.processes.process( "query_stac", @@ -78,7 +70,7 @@ import getpass from openeo.rest.stac_resource import StacResource from openeo.internal.graph_building import PGNode -# Make sure to choose a unique name +# Make sure to choose a unique name processing_name = "Modena" force_l2_stac_resource = StacResource( @@ -132,6 +124,8 @@ query_catalog = transform_item_collection_to_catalog_with_links(item_collection) ```{python} from openeo.internal.graph_building import PGNode +from openeo.rest.stac_resource import StacResource + query_catalog = transform_item_collection_to_catalog_with_links(item_collection) processing_name = "Modena" @@ -141,8 +135,9 @@ force_l2_stac_resource = StacResource( process_id="force_level2", arguments={ "stac_document": query_catalog.to_dict(), - "name": "Modena", + "name": processing_name, "do_brdf": True, + # level 2 parameters } ), connection=connection, @@ -174,16 +169,42 @@ l2_job = connection.job("j-260527114522498d959f20eff3fc9b78") ## Time Series Analysis -The openEO process for FORCE Time Series Analysis (TSA) is called `force_tsa`. Like the [level 2](level2.qmd) process, it takes most of the parameters usually passed to the [TSA module of FORCE](https://force-eo.readthedocs.io/en/latest/components/higher-level/tsa/) using the [parameter file](https://force-eo.readthedocs.io/en/latest/components/higher-level/tsa/param.html). Parameters use `snake_case` format instead of `SCREAMING_SNAKE_CASE` (used in FORCE parameter files). +The openEO process for FORCE Time Series Analysis (TSA) is called `force_tsa`. Like the [level 2](level2.qmd) process, +it takes most of the parameters usually passed to the +[TSA module of FORCE](https://force-eo.readthedocs.io/en/latest/components/higher-level/tsa/) using the +[parameter file](https://force-eo.readthedocs.io/en/latest/components/higher-level/tsa/param.html). +Parameters use `snake_case` format instead of `SCREAMING_SNAKE_CASE` (used in FORCE parameter files). Details of parametrization are discussed in the [dedicated article](parametrization.qmd). +```{.python} +connection = openeo.connect("openeo.dataspace.copernicus.eu").authenticate_oidc() +``` + + + +```{python} +#| echo: false +connection = openeo.connect("openeo-staging.dataspace.copernicus.eu").authenticate_oidc() +``` + :::{.callout-note collapse="true" title="FORCE TSA parameters"} ```{python} connection.describe_process("force_tsa") ``` ::: + +```{python} +#| echo: false +connection = openeo.connect("openeo.dataspace.copernicus.eu").authenticate_oidc() +``` + ### Using FORCE level 2 results as an input The process graph for FORCE TSA built in the same way as the level 2 process graph is built. @@ -192,7 +213,10 @@ The main input for FORCE TSA is the STAC catalog (or item) from the level 2 proc We can access the STAC catalog produced by our level 2 job in the workspace, described in [level 2 processing article](level2.qmd#Accessing results). -It is important to remember the `merge` paramter passed to `export_workspace`, this is the path under which we can find the catalog. +:::{.callout-important} +It is important to remember the `merge` parameter passed to `export_workspace`, this is the path under which we can find +the catalog later. +::: ```{python} @@ -201,7 +225,7 @@ merge = f"{getpass.getuser()}/experiments/{processing_name}" print(merge) ``` -Remember, the dedicated workspace for FORCE has the URL +Remember, the dedicated workspace for FORCE has the URL ```{python} @@ -218,7 +242,7 @@ temporal_extent=["2026-04-17", "2026-04-27"] ``` For other parameters, we may want to inspect the metadata of the level 2 processing. -For example, we can extract the tiles from the asset names of the STAC item describing the level 2 datacube. +For example, we can extract the tiles from the asset names of the STAC item describing the level 2 datacube. ```{python} @@ -246,13 +270,14 @@ The main process is `force_tsa`. from openeo.rest.stac_resource import StacResource from openeo.internal.graph_building import PGNode +tsa_processing_name = f"{processing_name}_TSA" force_tsa_stac_resource = StacResource( graph=PGNode( process_id="force_tsa", arguments={ "stac_url": l2_catalog_url, - "name": "Modena_TSA", + "name": tsa_processing_name, "date_range": temporal_extent, "x_tile_range": [min(tile_x), min(tile_x)], "y_tile_range": [min(tile_y), max(tile_y)], @@ -268,41 +293,26 @@ force_tsa_stac_resource = StacResource( We can then create and start the job as before: ```{.python} -tsa_job = force_tsa_stac_resource.create_job(title=f"FORCE TSA {processing_name}") +tsa_job = force_tsa_stac_resource.create_job(title=tsa_processing_name) tsa_job.start() # use start_and_wait to block until finished ``` -Inspecting STAC metadata and downloading results works in the same way as for level 2 processing: - ```{python} #| echo: false -tsa_job = connection.job("j-260515103226413e919ee5d9934c9693") +tsa_job = connection.job("j-2608191330204189973ebc9251875552") ``` -```{.python} -tsa_catalog_url = extract_catalog_url_from_job_logs(tsa_job.logs()) -tsa_catalog = pystac.Catalog.from_file(tsa_catalog_url) -tsa_item = next(tsa_catalog.get_items()) -tsa_item -``` +Downloading assets works the same way as for level2: -Downloading assets as well: - -```{.python} -tsa_results = l2_job.get_results() +```{python} +tsa_results = tsa_job.get_results() tsa_results ``` -```{.python} +```{python} tsa_results.download_files("force-tsa-results") ``` - - ### Further processing Because level 2 results are provided by a STAC URL to the TSA process, it is possible run multiple TSA jobs with the same level 2 input, without expensively reprocessing the level 2 data. diff --git a/docs/force/img/src/force-eoap.d2 b/docs/force/img/src/force-eoap.d2 index 4b60841..2c83933 100644 --- a/docs/force/img/src/force-eoap.d2 +++ b/docs/force/img/src/force-eoap.d2 @@ -70,7 +70,7 @@ classes: { eoap { - label: Force EOAP + label: Force EOAP class: main Docker { class: group @@ -99,7 +99,7 @@ eoap { class: workflow } l2-clt { - label: FORCE Level 2 CommandLineTool + label: FORCE Level 2 CommandLineTool class: CLT } staging-clt { diff --git a/docs/force/index.qmd b/docs/force/index.qmd index a9b57e2..d67d49d 100644 --- a/docs/force/index.qmd +++ b/docs/force/index.qmd @@ -2,21 +2,23 @@ title: "FORCE with openEO on CDSE" --- -The integration of the [FORCE](https://force-eo.readthedocs.io/en/latest/) processing engine in CDSE provides access to [a subset of FORCE's processing functionality](feature_overview.qmd) via the [openEO API on CDSE](https://eoap.github.io/). +The integration of the [FORCE](https://force-eo.readthedocs.io/en/latest/) processing engine into the [Copernicus Data Space Ecosystem](https://dataspace.copernicus.eu/) (CDSE) provides access to [a subset of FORCE's processing functionality](feature_overview.qmd) via the [openEO API on CDSE](https://dataspace.copernicus.eu/). The integrated FORCE can be used to process on cloud infrastructure using an openEO client ([Python](https://open-eo.github.io/openeo-python-client/), [R](https://open-eo.github.io/openeo-r-client/) or [Javascript](https://open-eo.github.io/openeo-js-client/latest/)) and the openEO web-interface to create and monitor jobs. -Input Staging and [SpatioTemporal Asset Catalog (STAC)](https://stacspec.org/en) generation are handled automatically. +Input staging on the CDSE infrastructure and [SpatioTemporal Asset Catalog (STAC)](https://stacspec.org/en) generation are handled automatically. -The integration has been performed by [Brockmann Consult GmbH](https://www.brockmann-consult.de/) in the context of the European Space Agency's (ESA) [*Application Propagation Environments* (APEx)](https://apex.esa.int/) initiative. +The integration has been performed in the context of the [European Space Agency](https://www.esa.int/)'s (ESA) [*Application Propagation Environments* (APEx)](https://apex.esa.int/) initiative. +The source code is available [on Github](https://github.com/bcdev/apex-force-openeo). ## FORCE ![Official FORCE logo, [https://force-eo.readthedocs.io/en/latest/_images/force.png](https://force-eo.readthedocs.io/en/latest/_images/force.png)](https://force-eo.readthedocs.io/en/latest/_images/force.png) FORCE (Framework for Operational Radiometric Correction for Environmental Monitoring) is an EO processing engine developed by [Prof. Dr. David Frantz](https://www.uni-trier.de/universitaet/fachbereiche-faecher/fachbereich-vi/faecher/geoinformatik/team/prof-dr-d-frantz) -(*Geoinformatics - Spatial Data Science*, Trier University). +(*Geoinformatics - Spatial Data Science*, Trier University) and the Open Source FORCE community. -FORCE includes a large number of processing features, including the generation of Analysis Ready Data (ARD) Data Cubes and higher level data analysis processes. See the FORCE ['About' page](https://force-eo.readthedocs.io/en/latest/about.html) page for a full overview of FORCE's features. +FORCE includes a large number of processing features, including the generation of Analysis Ready Data (ARD) Data Cubes and higher level data analysis processes. See the FORCE ['About' page](https://force-eo.readthedocs.io/en/latest/about.html) page for a full overview of FORCE's features. The [feature overview page](feature_overview.qmd) shows which features +of FORCE are exposed via openEO on CDSE. ::: {.callout-note} @@ -30,67 +32,100 @@ FORCE includes a large number of processing features, including the generation o Details of FORCE functionality and validation are described in a number of [scientific publications](https://force-eo.readthedocs.io/en/latest/refs.html). +::: {.callout-important} +When processing with FORCE, please make sure to properly acknowledge FORCE and its authors when processing with FORCE and using the results +as described in the [Citation and Acknowledgements guidelines](https://force-eo.readthedocs.io/en/latest/policy/citation.html) +::: + ## ...with openEO... This project makes it possible to use (the) FORCE through the [Copernicus Data Space Ecosystem (CDSE)](https://dataspace.copernicus.eu/) [openEO Service](https://dataspace.copernicus.eu/analyse/openeo). Using, for example, the [openEO Python client](https://openeo.org/documentation/1.0/python/) to interact with the backend, FORCE can be parametrized and executed using the familiar openEO machinery. -::: {.callout-warning} +::: {.callout-important} The integrated FORCE does not use the openEO data cube structure. -FORCE has its own file-system-based [data cube](https://force-eo.readthedocs.io/en/latest/howto/datacube.html) model. The openEO integration makes it possible to control FORCE through the openEO API. It does not merge the datacube models of FORCE and openEO, so foreign openEO processes cannot work directly on FORCE data cubes. +FORCE has its own file-system-based [data cube](https://force-eo.readthedocs.io/en/latest/howto/datacube.html) model. The openEO integration makes it possible to control FORCE through the openEO API. It does not merge the datacube models of FORCE and openEO, so other openEO processes cannot work directly on FORCE data cubes. ::: ```{.python} import openeo +from openeo.rest.stac_resource import StacResource +from openeo.internal.graph_building import PGNode + -connection = openeo.connect("https://openeo.dataspace.copernicus.eu").authenticate_oidc() +connection = openeo.connect("openeo.dataspace.copernicus.eu").authenticate_oidc() +# Select a STAC item to process (collections, catalogs and item collections are also supported!) stac_item_url = "https://stac.dataspace.copernicus.eu/v1/collections/sentinel-2-l1c/items/S2A_MSIL1C_20260419T100711_N0512_R022_T32TPQ_20260419T152521" -stac_resource = openeo.rest.stac_resource.StacResource( - graph=openeo.internal.graph_building.PGNode( + +# Create the process graph +processing_name = "FORCE_level2" +force_l2_stac_resource = StacResource( + graph=PGNode( process_id="force_level2", - arguments=dict( - stac_url=stac_item_url, - do_brdf=True, - output_ovv=True, - # ... other parameters supported by FORCE level 2 - ) + arguments={ + "stac_url": stac_item_url, + "name": processing_name, + "do_brdf": True, + # other FORCE level 2 parameters + }, ), connection=connection, ) -job = stac_resource.create_job(title="FORCE level 2") -job.start_and_wait() -job.get_results() -l2_results.download_files("force-ARD") + +# Run processing +l2_job = force_l2_stac_resource.create_job(title=processing_name) +l2_job.start_and_wait() + +# Download results (alternatively: Continue processing without download with Time Series Analysis. Check out the guide!) + +l2_results = l2_job.get_results() +l2_results.download_files("force-level2-results") ``` For a full example, see [the guide](guide/intro.qmd). -With the integrated FORCE, you can +With the integrated FORCE, you can - Run the force level 2 processing system to generate Analysis Ready Data (example above) - Run the Time Series Analysis module of the FORCE higher level processing system (hlps) - Download data cubes in FORCE's native datacube format -For an overview of the FORCE functionality made available via openEO, see [features](feature_overview.qmd). +For an overview of the FORCE functionality made available via openEO, see the [feature overview](feature_overview.qmd). ## ...on CDSE -FORCE runs on the CDSE infrastructure using the same data archive available to other CDSE services. +FORCE runs on the CDSE cloud processing infrastructure using the same data archive available to other CDSE services. +The integrated FORCE exclusively supports the CDSE +[Sentinel-2 L1C collection](https://browser.stac.dataspace.copernicus.eu/collections/sentinel-2-l1c) available via the STAC API. + +## Getting Started + +To get started, have a look at our [user guide](guide/intro.qmd). It explains prerequisites for executing FORCE on CDSE +and leads though detailed examples for level 2 and Time Series Analysis processing. + +Alternatively, there are [example Jupyter notebooks](https://github.com/bcdev/apex-force-openeo/tree/main/examples) +available in the repository. ## EO Application Package :::{.callout-note} -This section contains information on implementation details, you can -safely ignore it. +This section contains information on implementation details ::: +FORCE is made available through the openEO API by using the openEO backend's support for EO Application Packages (EOAP) +according to the [OGC Best Practice for Earth Observation Application Package](http://www.opengis.net/doc/BP/eoap/1.0). +The Common Workflow Language (CWL) documents of the EOAP for FORCE level 2 and Time Series Analysis are available in the +[Git repository](https://github.com/bcdev/apex-force-openeo/tree/main/cwl). + +More information on EOAP can be found on the +[Earth Observation Application Package resources website](https://eoap.github.io/) -FORCE is integrated into CDSE openEO using the built-in support for EO Application Packages following the [OGC Best Practice for Earth Observation Application Packages](https://docs.ogc.org/bp/20-089r1.html) (EOAP). -EOAPs are implemented as a combination of a Docker Image and a [Common Workflow Language](https://www.commonwl.org/) (CWL) document. +This diagram shows the integration layers: ![integration onion](img/onion-concrete.svg) +The integration work has been performed by [Brockmann Consult GmbH](https://www.brockmann-consult.de/)