diff --git a/apps/docs/docs/introduction/key-concepts/datastreams.md b/apps/docs/docs/introduction/key-concepts/datastreams.md index 4fdded0cc..89250dc41 100644 --- a/apps/docs/docs/introduction/key-concepts/datastreams.md +++ b/apps/docs/docs/introduction/key-concepts/datastreams.md @@ -11,11 +11,11 @@ For example, you could have a datastream of: - a Sampled Medium of Surface Water - measured using A YSI EXO2 Multiparameter Water Quality Sonde sensor -A datastream acts as the hub for understanding what's being measured at a given location. For example, if you want to know what's being measured, you'll ask the API for `datastream.observed_property` which will return 'Water temperature' along with metadata describing exactly what 'Water temperature' means to your organization. If you want to know how the observations are produced, you'd ask for `datastream.method`, which may include instrument details when applicable. If you want the actual data, you'll call `datastream.observations`. +A datastream acts as the hub for understanding what's being measured at a given location. For example, if you want to know what's being measured, you'll ask the API for `datastream.observedProperty` which will return 'Water temperature' along with metadata describing exactly what 'Water temperature' means to your organization. If you want to know how the observations are produced, you'd ask for `datastream.method`, which may include instrument details when applicable. If you want the actual data, you'll call `datastream.observations`. ## Naming Datastreams -The `datastream.name` must be unique and unambiguous within its monitoring site. Use a short, data-focused name and do not append the monitoring site name. HydroServer applications provide the monitoring-site context when it is needed. +The `datastream.name` must be unique and unambiguous within its monitoring site. Use a short, data-focused name and do not append the monitoring site name. HydroServer applications provide the monitoring site context when it is needed. Use this recommended convention: diff --git a/apps/docs/docs/introduction/key-concepts/monitoring-sites.md b/apps/docs/docs/introduction/key-concepts/monitoring-sites.md index 1c56704ef..6323a838f 100644 --- a/apps/docs/docs/introduction/key-concepts/monitoring-sites.md +++ b/apps/docs/docs/introduction/key-concepts/monitoring-sites.md @@ -12,4 +12,4 @@ HydroServer's data model and Data Management API use `MonitoringSite`. The Senso ## key:value tagging system -Most of the metadata you will need is already supported by HydroServer’s API. However, for organization-specific or custom information, tags offer a flexible way to store additional metadata that doesn’t fit into the standard fields. Tags are extra, customizable, key:value pairs that provide more context or categorization to the data. For example, you might use tags to link to your sites to 3rd party websites: `website:https://my-website.com`. You can use tags to specify values for additional attributes of a site, or you can point to external websites or URLs where more detailed information about that site might be maintained. +Most of the metadata you will need is already supported by HydroServer’s API. However, for organization-specific or custom information, tags offer a flexible way to store additional metadata that doesn’t fit into the standard fields. Tags are extra, customizable, key:value pairs that provide more context or categorization to the data. For example, you might use tags to store the USGS Hydrologic Unit your site is located within: `huc_12:160102030307`. diff --git a/apps/docs/docs/introduction/key-concepts/workspaces-and-access-control.md b/apps/docs/docs/introduction/key-concepts/workspaces-and-access-control.md index 6093795ba..1d4c9a450 100644 --- a/apps/docs/docs/introduction/key-concepts/workspaces-and-access-control.md +++ b/apps/docs/docs/introduction/key-concepts/workspaces-and-access-control.md @@ -29,11 +29,11 @@ The following table is a comparison of permissions for the owners, editors, and Remote systems or applications may need to authenticate with HydroServer in data-loading, retrieval, or analysis workflows. Giving these systems your username and password is risky because they receive all of your permissions. Instead, create a service account, assign it the minimum workspace role required for its job, and authenticate it with the generated API key. Service accounts can use roles such as Data Loader, Editor, or Viewer just like user collaborators. -## Data Visibility +## Data Privacy and Visibility -It's assumed in a HydroServer setup that you want your data to be publicly viewable. By default, when you load data to a datastream, any guest of your site will be able to view that data and all related metadata. These visibility setting can be changed at the workspace, site, and datastream levels. +It's assumed in a HydroServer setup that you want your data to be publicly viewable. By default, when you load data to a datastream, any guest of your site will be able to view that data and all related metadata. These privacy settings can be changed at the workspace, site, and datastream levels. -1. **Workspace Privacy**: This setting determines whether a workspace is private or public. If your workspace is private, all sites, datastreams, and associated metadata will only be accesible to the workspace owner and collaborators. +1. **Workspace Privacy**: This setting determines whether a workspace is private or public. If your workspace is private, all sites, datastreams, and associated metadata will only be accessible to the workspace owner and collaborators. 2. **Site Privacy**: This setting determines whether your site is private or public. If you set your site to private, it means that only the workspace owners and collaborators can view the site and all associated datastreams through the website or API. This is like having a closed folder that only selected people can open. This allows users to set some sites as public and others as private within a public workspace. @@ -43,4 +43,4 @@ It's assumed in a HydroServer setup that you want your data to be publicly viewa These privacy settings work in a hierarchical, conditional manner: -If you set the **Site Privacy** to private (`monitoring_site.is_private`), then both **Datastream Visibility** (`datastream.is_visible`) and **Datastream Data Visibility** (`datastream.is_data_visible`) will automatically be set to private as well. However, if the site is public, you can still control the visibility of each datastream and its data individually. +If you set the **Site Privacy** to private (`monitoringSite.isPrivate`), then both **Datastream Provacy** (`datastream.isPrivate`) and **Datastream Data Visibility** (`datastream.isVisible`) will automatically be set to private as well. However, if the site is public, you can still control the visibility of each datastream and its data individually. diff --git "a/apps/docs/docs/public/hydroserver-101/Screenshot 2026-06-04 at 10.07.01\342\200\257AM.png" "b/apps/docs/docs/public/hydroserver-101/Screenshot 2026-06-04 at 10.07.01\342\200\257AM.png" deleted file mode 100644 index 241bab12d..000000000 Binary files "a/apps/docs/docs/public/hydroserver-101/Screenshot 2026-06-04 at 10.07.01\342\200\257AM.png" and /dev/null differ diff --git a/apps/docs/docs/public/hydroserver-101/access-control.png b/apps/docs/docs/public/hydroserver-101/access-control.png index d6064873f..4a52e4c0d 100644 Binary files a/apps/docs/docs/public/hydroserver-101/access-control.png and b/apps/docs/docs/public/hydroserver-101/access-control.png differ diff --git a/apps/docs/docs/public/hydroserver-101/add-workspace.png b/apps/docs/docs/public/hydroserver-101/add-workspace.png index aae6e5bf9..dee32fffa 100644 Binary files a/apps/docs/docs/public/hydroserver-101/add-workspace.png and b/apps/docs/docs/public/hydroserver-101/add-workspace.png differ diff --git a/apps/docs/docs/public/hydroserver-101/api-key.png b/apps/docs/docs/public/hydroserver-101/api-key.png deleted file mode 100644 index 02c650381..000000000 Binary files a/apps/docs/docs/public/hydroserver-101/api-key.png and /dev/null differ diff --git a/apps/docs/docs/public/hydroserver-101/data-visualization.png b/apps/docs/docs/public/hydroserver-101/data-visualization.png index 3828350c1..b13da5539 100644 Binary files a/apps/docs/docs/public/hydroserver-101/data-visualization.png and b/apps/docs/docs/public/hydroserver-101/data-visualization.png differ diff --git a/apps/docs/docs/public/hydroserver-101/datastream-form.png b/apps/docs/docs/public/hydroserver-101/datastream-form.png deleted file mode 100644 index f622a8e92..000000000 Binary files a/apps/docs/docs/public/hydroserver-101/datastream-form.png and /dev/null differ diff --git a/apps/docs/docs/public/hydroserver-101/datastream-table.png b/apps/docs/docs/public/hydroserver-101/datastream-table.png index 0ddb50e7d..e1b51b62b 100644 Binary files a/apps/docs/docs/public/hydroserver-101/datastream-table.png and b/apps/docs/docs/public/hydroserver-101/datastream-table.png differ diff --git a/apps/docs/docs/public/hydroserver-101/fresh-user.png b/apps/docs/docs/public/hydroserver-101/fresh-user.png deleted file mode 100644 index e44df87c7..000000000 Binary files a/apps/docs/docs/public/hydroserver-101/fresh-user.png and /dev/null differ diff --git a/apps/docs/docs/public/hydroserver-101/fresh-workspace.png b/apps/docs/docs/public/hydroserver-101/fresh-workspace.png deleted file mode 100644 index f050a05ff..000000000 Binary files a/apps/docs/docs/public/hydroserver-101/fresh-workspace.png and /dev/null differ diff --git a/apps/docs/docs/public/hydroserver-101/new-datastream.png b/apps/docs/docs/public/hydroserver-101/new-datastream.png index e75a73427..5ee5407d0 100644 Binary files a/apps/docs/docs/public/hydroserver-101/new-datastream.png and b/apps/docs/docs/public/hydroserver-101/new-datastream.png differ diff --git a/apps/docs/docs/public/hydroserver-101/observed-property-form.png b/apps/docs/docs/public/hydroserver-101/observed-property-form.png new file mode 100644 index 000000000..7c62c9c74 Binary files /dev/null and b/apps/docs/docs/public/hydroserver-101/observed-property-form.png differ diff --git a/apps/docs/docs/public/hydroserver-101/sensor-form.png b/apps/docs/docs/public/hydroserver-101/sensor-form.png deleted file mode 100644 index 0d5a60d16..000000000 Binary files a/apps/docs/docs/public/hydroserver-101/sensor-form.png and /dev/null differ diff --git a/apps/docs/docs/public/hydroserver-101/service-account.png b/apps/docs/docs/public/hydroserver-101/service-account.png new file mode 100644 index 000000000..053ebca33 Binary files /dev/null and b/apps/docs/docs/public/hydroserver-101/service-account.png differ diff --git a/apps/docs/docs/public/hydroserver-101/site-details.png b/apps/docs/docs/public/hydroserver-101/site-details.png index 58e199905..e1b51b62b 100644 Binary files a/apps/docs/docs/public/hydroserver-101/site-details.png and b/apps/docs/docs/public/hydroserver-101/site-details.png differ diff --git a/apps/docs/docs/public/hydroserver-101/site-form.png b/apps/docs/docs/public/hydroserver-101/site-form.png index ae9eaeb16..86b71124e 100644 Binary files a/apps/docs/docs/public/hydroserver-101/site-form.png and b/apps/docs/docs/public/hydroserver-101/site-form.png differ diff --git a/apps/docs/docs/references/data-dictionary.md b/apps/docs/docs/references/data-dictionary.md index 512ffcac2..05ef7e436 100644 --- a/apps/docs/docs/references/data-dictionary.md +++ b/apps/docs/docs/references/data-dictionary.md @@ -410,7 +410,7 @@ A Payload describes how the response body returned by the extractor should be pa ## DataConnectionNotification -A DataConnectionNotification records the configuration for the periodic data freshness notification associated with a DataConnection. +A DataConnectionNotification records the configuration for the periodic task run summary email associated with a DataConnection. | Required | Attribute | Definition | Data Type | | -------- | ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------- | diff --git a/apps/docs/docs/user-guides/how-to/configuring-python-etl-pipelines.md b/apps/docs/docs/user-guides/how-to/configuring-python-etl-pipelines.md index 5dbe889c7..6b22e9682 100644 --- a/apps/docs/docs/user-guides/how-to/configuring-python-etl-pipelines.md +++ b/apps/docs/docs/user-guides/how-to/configuring-python-etl-pipelines.md @@ -12,11 +12,15 @@ A pipeline is made up of three components: These are assembled into an `ETLPipeline` and executed with a set of data mappings that describe which source columns map to which HydroServer datastream targets. +### Local pipelines vs. scheduled ETL tasks + +HydroServer's scheduled ETL — the data connections and ETL tasks you create in the web app or with [`hs_api.dataconnections` and `hs_api.etltasks`](/user-guides/how-to/using-the-python-client#etl) — runs on this same engine on the server. Use a scheduled ETL task when HydroServer should fetch the data on a schedule. Build a pipeline yourself, as described on this page, when the data is only reachable from your own machine (for example a local file), when you need custom logic around the run, or when you want to test a configuration before scheduling it. + ## Configuring a Pipeline ### Extractors -Extractors retrieve data from a source and return a file-like object. The `source_uri` field supports template variables, which are resolved at runtime using keyword arguments passed to `run()`. +Extractors retrieve data from a source and return a file-like object. The `source_uri` field (`filepath` for FTP) supports template variables in `{braces}`, which are resolved at runtime using keyword arguments passed to `run()`. **HTTP:** ```python @@ -27,6 +31,16 @@ extractor = HTTPExtractor( ) ``` +If the source requires an API key or token, pass it as a request header with `auth_header_name` and `auth_header_value`: + +```python +extractor = HTTPExtractor( + source_uri="https://api.example.com/data/{station_id}/export.csv", + auth_header_name="Authorization", + auth_header_value="Bearer ", +) +``` + **FTP:** ```python from hydroserverpy.etl.extractors import FTPExtractor @@ -51,7 +65,7 @@ extractor = LocalFileExtractor( ### Transformers -Transformers parse the extracted payload into a DataFrame with columns `timestamp`, `value`, and `target_id`. +Transformers parse the extracted payload into observations for each mapped target datastream. **CSV:** ```python @@ -86,7 +100,7 @@ transformer = JSONTransformer( ) ``` -The `jmespath` expression selects the array of records from the JSON payload. Since JMESPath can rename fields natively, the timestamp column in the selected records must be named to match `timestamp_key`, or renamed in the expression itself. +The `jmespath` expression selects the array of records from the JSON payload. Each selected record must contain a field named `timestamp_key`, and the [data mappings](#data-mappings) refer to the other fields by name. If the source uses different names, rename them in the expression, for example `data[].{timestamp: time, discharge: q}`. The JSON transformer has no `identifier_type` option; fields are always referenced by name. ### Timestamp Configuration @@ -192,7 +206,7 @@ ETLTargetPath( ) ``` -**Rating curve** — maps input values to output values using linear interpolation against a two-column CSV lookup table (input, output), retrieved from a URL. +**Rating curve** — maps input values to output values using linear interpolation against a CSV lookup table retrieved from a URL. The first two columns of the CSV are used as input and output values (the file should have a header row); rows that aren't numeric are ignored, and the table needs at least two numeric rows. Input values outside the table's range are set to `no_data_value` if you provide one, and left empty otherwise, rather than being clamped to the table's ends. ```python from hydroserverpy.etl.transformers import ETLTargetPath @@ -203,6 +217,7 @@ ETLTargetPath( data_operations=[ RatingCurveDataOperation( rating_curve_url="https://example.com/curves/stage-discharge.csv", + no_data_value=-9999, target_identifier="", ) ], @@ -240,7 +255,7 @@ Because temporal aggregation is a per-target operation, different targets fed by #### Aggregation interval -`aggregation_interval` (integer, default `1`) and `aggregation_interval_unit` (currently `"day"`) together define the window width. An `aggregation_interval` of `3` with unit `"day"` produces 3-day windows. +`aggregation_interval` (integer, default `1`) and `aggregation_interval_unit` together define the window width. An `aggregation_interval` of `3` with unit `"day"` produces 3-day windows. `"day"` is currently the only supported unit; any other value is rejected when the operation is created. #### Timezone @@ -287,7 +302,7 @@ Days with no observations are omitted from the output rather than filled with nu from hydroserverpy import HydroServer from hydroserverpy.etl.loaders import HydroServerLoader -hs = HydroServer(host="https://playground.hydroserver.org", email="user@example.com", password="pass") +hs = HydroServer(host="https://playground.hydroserver.org", email="your@email.com", password="yourpassword") loader = HydroServerLoader( client=hs, @@ -295,6 +310,8 @@ loader = HydroServerLoader( ) ``` +The loader only writes observations that are newer than the most recent observation already in each target datastream. Running the same pipeline twice therefore doesn't duplicate data — on the second run, each target is reported as `skipped`. To reload or correct data you've already loaded, use a [data ingestion window](#reloading-a-time-window). + ### Assembling the Pipeline ```python @@ -328,19 +345,44 @@ context = pipeline.run( ) ``` +### Reloading a time window + +Pass `data_ingestion_window_start` and/or `data_ingestion_window_end` (timezone-aware datetimes) to load only observations within that window. With a window set, the loader replaces each target's existing observations inside the window with the newly transformed ones, instead of skipping data that's already loaded — useful for backfilling or correcting a period. + +```python +from datetime import datetime, timezone + +context = pipeline.run( + data_mappings=data_mappings, + station_id="station-42", + data_ingestion_window_start=datetime(2026, 5, 1, tzinfo=timezone.utc), + data_ingestion_window_end=datetime(2026, 5, 8, tzinfo=timezone.utc), +) +``` + +The loader uploads the window in chunks of `chunk_size` observations, and each chunk replaces its own slice of the window in a single transaction. If a chunk fails, the run is `INCOMPLETE`: slices before it hold the new data, and the failed slice and everything after it keep their existing observations. Nothing is deleted without being replaced, so you can rerun the same window to finish the reload. + +Scheduled data connections configure their data ingestion window in the web app. + ## Viewing Results `run()` returns an `ETLContext` object describing the outcome of the run. ```python -print(context.status) # ETLStatus.SUCCESS, FAILED, INCOMPLETE, etc. -print(context.stage) # ETLStage where the run ended: EXTRACT, TRANSFORM, LOAD +from hydroserverpy.etl import ETLStatus, ETLStage + +print(context.status) # ETLStatus.SUCCESS, INCOMPLETE, or FAILED +print(context.stage) # ETLStage where the run ended ``` -The `results` field holds an `ETLLoaderResult` with summary counts and per-target detail: +`context.stage` is `CLEANUP` after a run that completed, or the stage that failed: `EXTRACT` (including missing template variables), `TRANSFORM`, or `LOAD`. + +The `results` field holds an `ETLLoaderResult` with summary counts and per-target detail. It is `None` if the run failed before or during the load stage, so check it first: ```python results = context.results +if results is None: + raise SystemExit(f"Run failed during {context.stage}: {context.exception}") print(results.success_count) # number of datastreams successfully loaded print(results.failure_count) # number of datastreams that failed @@ -361,23 +403,31 @@ for target_id, target in results.target_results.items(): | `ETLStatus` | Meaning | |---|---| -| `SUCCESS` | All datastreams loaded without error | -| `INCOMPLETE` | At least one datastream failed or was skipped | -| `FAILED` | The pipeline did not reach the load stage | +| `SUCCESS` | The run completed and no target failed. This includes runs where every target was skipped because there was no new data. | +| `INCOMPLETE` | At least one target failed while uploading; others may have loaded successfully. | +| `FAILED` | A stage raised an error. This includes the loader's check, before anything is uploaded, that every target datastream exists — a missing or invalid target fails the whole run. | | `RUNNING` | The pipeline is currently executing | | `PENDING` | The pipeline has not yet started | ## Debugging Errors -When `raise_on_error=False`, failure details are available on the context: +When `raise_on_error=False`, the exception that ended the run is available on the context: ```python +import traceback +from hydroserverpy.etl import ETLStatus + if context.status == ETLStatus.FAILED: - print(context.error) # exception message - print(context.traceback) # full traceback string - print(context.stage) # which stage failed: EXTRACT, TRANSFORM, or LOAD + print(context.stage) # which stage failed + print(context.exception) # the error message + traceback.print_exception(context.exception) # full traceback ``` +The context also records what happened during the run: + +- `context.log_entries` — the run's log messages, each a dict with `timestamp`, `level`, `logger`, `stage`, and `message`. +- `context.runtime_variables` — the extractor, transformer, and loader settings after template variables were filled in, such as the resolved `source_uri`. + For partial failures where some datastreams loaded and others did not (`ETLStatus.INCOMPLETE`), inspect the per-target results: ```python @@ -389,30 +439,44 @@ for target_id, target in context.results.target_results.items(): ### Common errors and causes +Errors raised during a run are translated into the messages below. The message is what you see in `context.exception` (or in the raised exception) and in HydroServer's task run history. + **Extract stage** -| Error | Likely cause | +| Error message begins with | Likely cause | |---|---| -| `... not found` | The source URI resolved to a path or URL that doesn't exist | -| `... timed out` | The source system did not respond in time | -| `Authentication ... failed` | Credentials are incorrect or expired | -| `... is empty` | The source returned a file with no content | -| `Missing required runtime variables` | A template variable in `source_uri` or `filepath` was not passed to `run()` | +| `The extractor URL includes a placeholder '...', but no value was supplied` | A `{variable}` in `source_uri` or `filepath` wasn't passed to `run()` | +| `Failed to connect to the source system` | The source is offline or unreachable, or the request timed out | +| `The requested data could not be found on the source system` | The URL or file path doesn't exist | +| `Authentication with the source system failed` | Credentials or auth headers are incorrect or expired | +| `No observations were returned from the source system` | The source returned an empty file or response | **Transform stage** -| Error | Likely cause | +| Error message begins with | Likely cause | |---|---| -| `Column ... not found` | `timestamp_key` or a source identifier doesn't match any column in the payload | -| `... empty CSV file` | The extracted file contained no parseable rows | -| `One or more configured CSV columns were not found` | A named source identifier isn't present in the CSV header | -| `... not valid JSON` | The extracted payload is not valid JSON | -| `... could not be found with the specified query` | The JMESPath expression returned no results | -| `Failed to compile arithmetic expression` | An expression contains unsupported syntax or variables other than `x` | +| `The column mapped as the timestamp does not exist in the file` | `timestamp_key` doesn't match any column in the CSV | +| `The timestamp column is set incorrectly. Index mode expects a 1-based column number` | `identifier_type="index"` with a `timestamp_key` that isn't a valid column number | +| `The CSV header or data start row is set incorrectly` | `header_row` or `data_start_row` doesn't match the file | +| `A required column was not found in the file header` | A mapped source column isn't present in the CSV header | +| `A mapping source index (...) is out of range for the extracted data` | An index-mode source identifier is larger than the number of columns | +| `A mapped field named '...' was not found in the extracted data` | A source identifier doesn't match any column or JSON field | +| `The source did not return valid JSON` | The extracted payload isn't JSON | +| `The JSON query used to extract timestamps or values is invalid` | The `jmespath` expression has a syntax error | +| `Failed to find the timestamp or value using the current JSON query` / `The job could not find the expected timestamp or value fields` | The `jmespath` expression doesn't match the payload, or the selected records lack `timestamp_key` | +| `A custom timestamp format is required when the timestamp type is set to custom` | `timestamp_type="custom"` without `timestamp_format` | +| `The custom timestamp format is not valid` | `timestamp_format` isn't a valid strftime string | +| `The timestamp UTC offset is not valid` | `timezone` isn't in `±HHMM` or `±HH:MM` format | +| `The configured timezone is not recognized` | `timezone` isn't a valid IANA name | +| `Timezone information should not be provided when the timestamps already include their own timezone values` | A `timezone` was set for timestamps that already include an offset | +| `Failed to compile arithmetic expression for data target` | An expression contains unsupported syntax or variables other than `x` | + +A `jmespath` expression that selects an empty list isn't an error: the run completes with `SUCCESS`, nothing is loaded, and `results.target_results` is empty. **Load stage** -| Error | Likely cause | +| Error message begins with | Likely cause | |---|---| -| `Missing datastream IDs: ...` | One or more target datastream UUIDs don't exist on the HydroServer instance | -| `HydroServer loader failed to retrieve datastream` | A network or authentication error occurred while looking up a datastream | \ No newline at end of file +| `One or more destination datastream identifiers could not be found in HydroServer` | A `target_identifier` isn't the UUID of an existing datastream, or your account can't access it | +| `HydroServer rejected the load due to authorization` | Your account doesn't have permission to write to the target datastream | +| `HydroServer rejected some or all of the data` | Invalid timestamps or values were sent to a datastream | diff --git a/apps/docs/docs/user-guides/how-to/exporting-and-downloading-data.md b/apps/docs/docs/user-guides/how-to/exporting-and-downloading-data.md index cc7423317..253f9a3aa 100644 --- a/apps/docs/docs/user-guides/how-to/exporting-and-downloading-data.md +++ b/apps/docs/docs/user-guides/how-to/exporting-and-downloading-data.md @@ -28,4 +28,4 @@ Observation values are exported as a simple CSV table with columns for the times ## Automated or Scripted Data Download -If you want to automate data retrieval or code data download, you should consult the sections in the documentation on using HydroServer's Python client hydroserverpy or using HydroServer's API (for other programming languages). +If you want to automate data retrieval or code data download, use HydroServer's Python client, hydroserverpy — see [retrieving observations](/user-guides/how-to/using-the-python-client.md#observation-methods) in the client reference — or call [HydroServer's APIs](/references/hydroserver-apis.md) directly from other programming languages. diff --git a/apps/docs/docs/user-guides/how-to/index.md b/apps/docs/docs/user-guides/how-to/index.md index a427a7112..7fa6aebda 100644 --- a/apps/docs/docs/user-guides/how-to/index.md +++ b/apps/docs/docs/user-guides/how-to/index.md @@ -26,7 +26,7 @@ In this section, we provide detailed documentation for all HydroServer functiona * **[Archiving Data to HydroShare](/user-guides/how-to/archiving-data-to-hydroshare.md)**: Learn how to archive data from HydroServer into HydroShare. -* **[Using the Python Client](/user-guides/how-to/using-the-python-client.md)**: Learn how to use the HydroServer Python client package – hydroserverpy – to load and retrieve metadata and data. +* **[Using the Python Client](/user-guides/how-to/using-the-python-client.md)**: Reference for hydroserverpy, HydroServer's Python client package, covering the classes and methods for managing workspaces, metadata, observations, ETL, data products, and monitoring from Python. * **[Configuring Python ETL Pipelines](/user-guides/how-to/configuring-python-etl-pipelines.md)**: Learn how to configure Python ETL pipelines to load data into HydroServer. diff --git a/apps/docs/docs/user-guides/how-to/job-orchestration-ingestion.md b/apps/docs/docs/user-guides/how-to/job-orchestration-ingestion.md index 315eb91e1..bac891b08 100644 --- a/apps/docs/docs/user-guides/how-to/job-orchestration-ingestion.md +++ b/apps/docs/docs/user-guides/how-to/job-orchestration-ingestion.md @@ -94,7 +94,7 @@ Select `ISO 8601` if the timestamps use the standard ISO 8601 date/time format. ### Advanced Features -If you toggle the "Use Advanced features" option, you can add a description to your data connection. You can also add an authentication header if you are connecting to an API that requires authentication. Finally, you can specify a list of email addresses for people who should be notified if tasks associated with the data connection fail to execute. You can add any number of email addresses for notification recipients. +If you toggle the "Use Advanced features" option, you can add a description to your data connection. You can also add an authentication header if you are connecting to an API that requires authentication. Finally, you can specify a list of email addresses for people who should receive a summary of the data connection's task runs, including any failures, on a notification schedule. Each summary covers the task runs since the previous scheduled summary. You can add any number of email addresses for notification recipients. After all settings are filled in, click the **Save** button at the bottom of the form. The data connection you just created will appear in the left-side **Connections** list. diff --git a/apps/docs/docs/user-guides/how-to/using-the-python-client.md b/apps/docs/docs/user-guides/how-to/using-the-python-client.md index 211712ec2..37632eb1c 100644 --- a/apps/docs/docs/user-guides/how-to/using-the-python-client.md +++ b/apps/docs/docs/user-guides/how-to/using-the-python-client.md @@ -4,31 +4,39 @@ --- +## Installation + +hydroserverpy 2.x works with HydroServer 2.x and requires Python 3.9 or higher. + +```bash +pip install "hydroserverpy>=2.0,<3" +``` + +--- + ## Connection ```python from hydroserverpy import HydroServer # Email/password authentication -hs_api = HydroServer(host, email, password) +hs_api = HydroServer(host, email=email, password=password) -# API key authentication +# API key authentication (a service account key) hs_api = HydroServer(host, apikey=apikey) ``` | Parameter | Type | Default | Description | |---|---|---|---| | `host` | `str` | — | Base URL of your HydroServer instance | -| `email` | `str` | `None` | Account email | +| `base_route` | `str` | `"/api/ogc"` | API path prefix; only change this for non-standard deployments | +| `email` | `str` | `None` | Account email; must be given together with `password` | | `password` | `str` | `None` | Account password | -| `apikey` | `str` | `None` | API key (alternative to email/password) | +| `apikey` | `str` | `None` | Service account API key (alternative to email/password) | -**Methods** +API keys belong to [service accounts](#service-accounts). They can read and write data the service account has access to, but cannot manage workspace access (collaborators, service accounts, ownership) — use email/password for those operations. -| Method | Description | -|---|---| -| `login(email, password)` | Re-authenticate with new credentials without creating a new instance | -| `logout()` | End the current session | +`hs_api.request(method, path, **kwargs)` sends a raw request through the authenticated session, for endpoints the client doesn't wrap. --- @@ -41,43 +49,69 @@ hs_api = HydroServer(host, apikey=apikey) | Property | Type | Description | |---|---|---| | `items` | `List[T]` | The resources on the current page | -| `page` | `int` | Current page number (1-indexed) | -| `page_size` | `int` | Number of items per page | -| `total_pages` | `int` | Total number of pages | -| `total_count` | `int` | Total number of matching resources | +| `offset` | `int \| None` | Offset of the current page | +| `limit` | `int \| None` | Page size of the current page | +| `total_count` | `int \| None` | Total number of matching resources | +| `filters` | `dict \| None` | Filters used for this query (reused by the paging methods) | +| `sortby` | `List[str] \| None` | Sort order used for this query | | Method | Returns | Description | |---|---|---| | `next_page()` | `HydroServerCollection` | Fetch the next page | -| `previous_page()` | `HydroServerCollection \| None` | Fetch the previous page | -| `fetch_all()` | `HydroServerCollection` | Merge all pages into a single collection | +| `previous_page()` | `HydroServerCollection \| None` | Fetch the previous page (`None` on the first page) | +| `fetch_all()` | `HydroServerCollection` | Fetch every remaining page and merge them into one collection | ### Common list() parameters -All `list()` methods accept these parameters in addition to any entity-specific filters: +All `list()` methods accept these parameters in addition to any resource-specific filters: | Parameter | Type | Default | Description | |---|---|---|---| -| `page` | `int` | `1` | Page number to fetch | -| `page_size` | `int` | `100` | Results per page | -| `sortby` | `List[str]` | `[]` | Fields to sort by; prefix with `-` for descending | +| `offset` | `int` | server default (`0`) | Number of results to skip | +| `limit` | `int` | server default (`100`) | Results per page | +| `sortby` | `List[str]` | server default | Fields to sort by; prefix with `-` for descending, e.g. `["-name"]` | | `fetch_all` | `bool` | `False` | If `True`, fetches and merges all pages automatically | +Filters that take a resource (for example `workspace=` or `monitoring_site=`) accept either the object or its UUID. + ### Model methods -All resource objects share these methods inherited from `HydroServerBaseModel`: +Resource objects share these methods inherited from `HydroServerBaseModel`: | Method | Description | |---|---| -| `save()` | Persist any modified editable fields back to HydroServer | +| `save()` | Send modified **editable** fields back to HydroServer | | `refresh()` | Re-fetch this resource's data from HydroServer | | `delete()` | Delete this resource from HydroServer and set `uid` to `None` | -`save()` only sends fields listed in the **editable** column of each entity's property table. Read-only fields are never sent. Fields marked **computed** are lazy-loaded properties — they trigger an API call on first access and are then cached on the object. +By default, `save()` sends only the editable fields you have changed since the object was loaded. A few models override this and send all of their fields on every save: `DataConnection`, `EtlTask`, and `RatingCurve` (noted in their sections below). + +Properties marked **computed** in the tables below are lazy-loaded — they make an API call on first access and are then cached on the object. + +### Service update() methods + +Every service also has `update(uid, ...)` and `delete(uid)` methods. For most resources, any argument you leave out of `update()` is left unchanged on the server. The exceptions — methods that replace a value when you omit it — are called out in their sections: [`etltasks.update`](#etl-tasks), [`dataconnections.update`](#data-connections), [`ratingcurves.update`](#rating-curves), and the transformation `update_*` methods. + +All service methods that take a `uid` accept a `UUID` or a string. `create()` methods also accept an optional `uid=` if you want to choose the new resource's ID yourself; otherwise the server assigns one. + +### Schedules + +ETL tasks, data product tasks, and monitoring tasks share the same schedule fields: + +| Field | Type | Description | +|---|---|---| +| `enabled` | `bool \| None` | Whether the schedule is active | +| `crontab` | `str \| None` | Cron expression, e.g. `"0 6 * * *"`; use instead of `interval` | +| `interval` | `int \| None` | Repeat every `interval` × `interval_period` | +| `interval_period` | `str \| None` | `"minutes"`, `"hours"`, or `"days"` | +| `start_time` | `datetime \| None` | When the schedule starts | +| `next_run_at` | `datetime \| None` | Next scheduled run (read-only, set by the server) | + +When updating a task, schedule fields you leave out keep their current values. Passing both `crontab=None` and `interval=None` removes the schedule. ### TaskRun -Returned by `trigger()` and `list_runs()` across ETL tasks, data product tasks, and monitoring tasks. +Returned by `trigger()`, `list_runs()`, and `get_run()` across ETL tasks, data product tasks, and monitoring tasks, and in each task's `latest_run`. | Property | Type | Description | |---|---|---| @@ -88,6 +122,32 @@ Returned by `trigger()` and `list_runs()` across ETL tasks, data product tasks, | `started_at` | `datetime \| None` | When the run started | | `finished_at` | `datetime \| None` | When the run finished | +A run stays `PENDING` until an orchestration worker picks it up; if it never leaves `PENDING`, check that your instance's workers are running. + +### Error handling + +Failed requests raise `requests.HTTPError`. For 4xx responses, the message includes HydroServer's explanation, and the original response is available on the exception: + +```python +import requests + +try: + datastream.load_observations(df) +except requests.HTTPError as e: + print(e) # "409 Client Error: Duplicate phenomenonTime found on this datastream." + print(e.response.status_code) # 409 +``` + +| Status | Typical cause | +|---|---| +| `400` | Invalid input (the message says which field) | +| `401` | Missing or invalid credentials, or an API key used for an operation that requires email/password | +| `403` | Your account or service account lacks permission for this action | +| `404` | The resource doesn't exist, or you don't have permission to see it | +| `409` | Conflict with existing data, e.g. duplicate observation timestamps or a duplicate unique value | + +The client retries a request once, with a fresh session, after a connection error or a `401`/`403` response. + --- ## Workspaces @@ -101,80 +161,138 @@ Returned by `trigger()` and `list_runs()` across ETL tasks, data product tasks, | `uid` | `UUID` | No | | | `name` | `str` | Yes | | | `is_private` | `bool` | Yes | | -| `owner` | `Account` | No | See Account fields below | +| `owner_email` | `str` | No | Email of the workspace owner | +| `pending_transfer_to_email` | `str \| None` | No | Recipient of a pending ownership transfer | +| `collaborator_role_id` | `UUID \| None` | No | Your role in this workspace, if you're a collaborator | +| `collaborator_role` | `Role \| None` | No | Computed | +| `roles` | `List[Role]` | No | Computed; roles you can assign in this workspace, including system roles | | `collaborators` | `List[Collaborator]` | No | Computed | -| `apikeys` | `List[APIKey]` | No | Computed | -| `roles` | `List[Role]` | No | Computed | +| `serviceaccounts` | `List[ServiceAccount]` | No | Computed | | `monitoring_sites` | `List[MonitoringSite]` | No | Computed | +| `datastreams` | `List[Datastream]` | No | Computed | | `observedproperties` | `List[ObservedProperty]` | No | Computed | | `units` | `List[Unit]` | No | Computed | | `processinglevels` | `List[ProcessingLevel]` | No | Computed | +| `resultqualifiers` | `List[ResultQualifier]` | No | Computed | | `methods` | `List[Method]` | No | Computed | | `dataconnections` | `List[DataConnection]` | No | Computed | | `tasks` | `List[EtlTask]` | No | Computed; ETL tasks only | -**Account fields** (accessible via `workspace.owner`): +### Service methods + +```python +hs_api.workspaces.list(is_associated=..., is_private=...) -> HydroServerCollection[Workspace] +hs_api.workspaces.get(uid) -> Workspace +hs_api.workspaces.create(name, is_private, uid=None) -> Workspace +hs_api.workspaces.update(uid, name=..., is_private=...) -> Workspace +hs_api.workspaces.delete(uid) -> None +``` -| Field | Type | +| Filter | Type | Description | +|---|---|---| +| `is_associated` | `bool` | Only return workspaces you own or collaborate on | +| `is_private` | `bool` | Filter by privacy setting | + +### Collaborators + +| Method | Description | |---|---| -| `name` | `str` | -| `email` | `str` | -| `organization_name` | `str \| None` | -| `phone` | `str \| None` | -| `address` | `str \| None` | -| `link` | `str \| None` | -| `user_type` | `str` | +| `workspace.add_collaborator(email, role)` | Add a user (by email) or a `ServiceAccount` as a collaborator; `role` accepts a `Role` or role UUID | +| `workspace.edit_collaborator_role(email, role)` | Change an existing collaborator's role | +| `workspace.remove_collaborator(email)` | Remove a collaborator (user email or `ServiceAccount`) | + +Use `workspace.roles` to find the role to assign. The same methods are available on the service as `hs_api.workspaces.add_collaborator(uid, email, role)` and so on, plus `list_collaborators(uid)`. **Collaborator fields** (items in `workspace.collaborators`): | Field | Type | Notes | |---|---|---| -| `uid` | `UUID` | | -| `user` | `Account` | The collaborator's account info | -| `role_id` | `UUID` | | +| `user_email` | `str \| None` | Set when the collaborator is a user | +| `service_account_email` | `str \| None` | Set when the collaborator is a service account | +| `email` | `str` | Whichever of the two is set | +| `is_service_account` | `bool` | | +| `role_id` | `UUID` | Editable | | `workspace_id` | `UUID` | | | `role` | `Role` | Computed; also settable via assignment | | `workspace` | `Workspace` | Computed | -**APIKey fields** (items in `workspace.apikeys`): +`collaborator.save()` applies a changed role; `collaborator.delete()` removes the collaborator from the workspace. -| Field | Type | Editable | -|---|---|---| -| `uid` | `UUID` | No | -| `name` | `str` | Yes | -| `description` | `str \| None` | Yes | -| `role_id` | `UUID` | Yes | -| `workspace_id` | `UUID` | No | -| `is_active` | `bool` | Yes | -| `expires_at` | `datetime \| None` | Yes | +### Ownership transfer -`APIKey` objects support `save()`, `refresh()`, `delete()`, and `regenerate()` (invalidates the old key and returns a new key string). +| Method | Description | +|---|---| +| `workspace.transfer_ownership(email)` | Start an ownership transfer to another HydroServer user | +| `workspace.accept_ownership_transfer()` | Accept a pending transfer (called by the recipient) | +| `workspace.cancel_ownership_transfer()` | Cancel a pending transfer | -### Service methods +Each method refreshes the workspace afterward, so `owner_email` and `pending_transfer_to_email` reflect the new state. + +### Roles + +**Property:** `hs_api.roles` + +Roles define what a collaborator can do in a workspace. System roles are available in every workspace; a workspace can also have its own roles. Roles are read-only through the API — calling `save()` or `delete()` on a `Role` raises `NotImplementedError`. + +| Property | Type | Notes | +|---|---|---| +| `uid` | `UUID` | | +| `name` | `str` | | +| `description` | `str \| None` | | +| `workspace_id` | `UUID \| None` | `None` for system roles | +| `permissions` | `List[RolePermission]` | Each has `resource` (e.g. `"Datastream"`) and `action` (e.g. `"edit"`) | +| `workspace` | `Workspace \| None` | Computed | ```python -hs_api.workspaces.list(is_associated=None, is_private=None) -> HydroServerCollection[Workspace] -hs_api.workspaces.get(uid) -> Workspace -hs_api.workspaces.create(name, is_private=False) -> Workspace +hs_api.roles.list(workspace=...) -> HydroServerCollection[Role] +hs_api.roles.get(uid) -> Role ``` -| Filter | Type | Description | -|---|---|---| -| `is_associated` | `bool` | Only return workspaces you're a member of | -| `is_private` | `bool` | Filter by privacy setting | +`workspace` accepts a single workspace or a list; include `None` in the list to also match system roles. `workspace.roles` is shorthand for `hs_api.roles.list(workspace=[workspace, None], fetch_all=True).items`. + +--- + +## Service Accounts + +Service accounts let scripts and automated jobs access HydroServer with an API key instead of a user's email and password. A service account belongs to a workspace; give it access by assigning it a role (at creation, or later with `add_collaborator`). + +Creating, updating, deleting, and regenerating service accounts requires email/password authentication. + +### ServiceAccount properties + +| Property | Type | Editable | Notes | +|---|---|---|---| +| `uid` | `UUID` | No | | +| `name` | `str` | Yes | | +| `description` | `str \| None` | Yes | | +| `is_active` | `bool` | Yes | Inactive accounts can't authenticate | +| `key_expires_at` | `datetime \| None` | Yes | | +| `email` | `str` | No | Generated identifier used when adding the account as a collaborator | +| `workspace_id` | `UUID` | No | | +| `created_at` | `datetime` | No | | +| `last_used_at` | `datetime \| None` | No | | +| `workspace` | `Workspace` | No | Computed | + +`ServiceAccount` objects support `save()`, `refresh()`, `delete()`, and `regenerate()`, which invalidates the old key and returns the new key string. ### Workspace methods | Method | Description | |---|---| -| `add_collaborator(email, role)` | Add a user as a collaborator; `role` accepts a UUID string or Role object | -| `edit_collaborator_role(email, role)` | Change an existing collaborator's role | -| `remove_collaborator(email)` | Remove a collaborator | -| `transfer_ownership(email)` | Initiate an ownership transfer to another user | -| `accept_ownership_transfer()` | Accept a pending incoming ownership transfer | -| `cancel_ownership_transfer()` | Cancel a pending outgoing ownership transfer | -| `create_api_key(name, role, description=None, is_active=True, expires_at=None)` | Create a new API key; returns `(APIKey, key_string)` where `key_string` is the raw key value — store it, it won't be shown again | -| `delete_api_key(api_key_id)` | Delete an API key by its UUID | +| `create_service_account(name, description=None, is_active=True, key_expires_at=None, role=None)` | Create a service account; returns `(ServiceAccount, key)`. If `role` is given, the account is also added as a collaborator with that role. | +| `update_service_account(service_account_id, name=..., description=..., is_active=..., key_expires_at=...)` | Update a service account | +| `delete_service_account(service_account_id)` | Delete a service account | +| `regenerate_service_account_key(service_account_id)` | Returns `(ServiceAccount, key)` with a new key | + +The service exposes the same operations with the workspace UID as the first argument (`hs_api.workspaces.create_service_account(uid, name, ...)`), plus `list_service_accounts(uid)` and `get_service_account(uid, service_account_id)`. + +The key is only returned when it is created or regenerated — store it securely. To use it: + +```python +service_account, key = workspace.create_service_account(name="ETL job", role=editor_role) + +hs_job = HydroServer(host, apikey=key) +``` --- @@ -201,132 +319,156 @@ hs_api.workspaces.create(name, is_private=False) -> Workspace | `admin_area_2` | `str \| None` | Yes | County/district | | `country` | `str \| None` | Yes | ISO 3166-1 alpha-2 | | `workspace_id` | `UUID` | No | | -| `tags` | `Dict[str, str]` | No | Use tag methods to modify | -| `linked_resources` | `Dict[str, dict]` | No | Use linked resource methods to modify | +| `tags` | `Dict[str, str]` | No | Use the tag methods to modify | +| `linked_resources` | `Dict[str, dict]` | No | Keyed by name; each has `id`, `link`, and `type`. Use the linked resource methods to modify | | `workspace` | `Workspace` | No | Computed | | `datastreams` | `List[Datastream]` | No | Computed | ### Service methods ```python -hs_api.monitoring_sites.list(workspace=None, bbox=None, type=None, - tag=None, is_private=None) -> HydroServerCollection[MonitoringSite] +hs_api.monitoring_sites.list(workspace=..., bbox=..., admin_area_1=..., admin_area_2=..., + country=..., type=..., tag=..., is_private=...) -> HydroServerCollection[MonitoringSite] hs_api.monitoring_sites.get(uid) -> MonitoringSite hs_api.monitoring_sites.create(workspace, name, description, code, type, is_private, latitude, longitude, elevation_m=None, elevation_datum=None, admin_area_1=None, admin_area_2=None, - country=None, data_disclaimer=None, uid=None) -> MonitoringSite + country=None, data_disclaimer=None, tags=None, uid=None) -> MonitoringSite +hs_api.monitoring_sites.update(uid, name=..., description=..., code=..., type=..., is_private=..., + latitude=..., longitude=..., elevation_m=..., elevation_datum=..., + admin_area_1=..., admin_area_2=..., country=..., + data_disclaimer=..., tags=...) -> MonitoringSite +hs_api.monitoring_sites.delete(uid) -> None ``` | Filter | Type | Description | |---|---|---| -| `workspace` | `UUID \| str` | Filter by workspace | +| `workspace` | `Workspace \| UUID \| str` | Filter by workspace | | `bbox` | `tuple` | `(min_lon, min_lat, max_lon, max_lat)` bounding box | +| `admin_area_1` | `str` | Filter by state/province | +| `admin_area_2` | `str` | Filter by county/district | +| `country` | `str` | Filter by country code | | `type` | `str` | Filter by site type | | `tag` | `tuple` | `(key, value)` filter by tag | | `is_private` | `bool` | Filter by privacy setting | -### Tag and linked resource methods +### Tags + +| Method | Description | +|---|---| +| `site.set_tag(key, value)` | Add or update a tag | +| `site.delete_tag(key)` | Remove a tag | + +`update(uid, tags={...})` merges the given tags into the existing ones; a value of `None` removes that key. Tags you don't mention are kept. -| Method | Description | -|---|----------------------------------| -| `set_tag(key, value)` | Add or update a tag on this monitoring site | -| `delete_tag(key)` | Remove a tag | -| `add_linked_resource(name, type, file=None, url=None, description=None)` | Add a linked resource | -| `delete_linked_resource(name)` | Remove a linked resource by name | +### Linked resources + +| Method | Description | +|---|---| +| `site.add_linked_resource(name, type, file=None, url=None, description=None)` | Attach a file (`file` opened in binary mode) or a link (`url`) | +| `site.delete_linked_resource(name)` | Remove a linked resource by name | + +The service also provides `get_linked_resources(uid, offset=..., limit=..., type=...)`, `get_linked_resource(uid, linked_resource_id)`, `update_linked_resource(uid, linked_resource_id, name=None, type=None, file=None, url=None, description=None)`, and `delete_linked_resource(uid, linked_resource_id)`. --- ## Metadata -Methods, observed properties, units, processing levels, and result qualifiers all follow the same shape: a small set of editable fields, a read-only `workspace_id`, and a computed `workspace` property. They all support the standard `save()`, `refresh()`, and `delete()` model methods. +Methods, observed properties, units, processing levels, and result qualifiers all follow the same shape: a small set of editable fields, a read-only `workspace_id`, and a computed `workspace` property. They support the standard `save()`, `refresh()`, and `delete()` model methods. ### Methods **Property:** `hs_api.methods` -| Property | Type | Editable | -|---|---|---| -| `uid` | `UUID` | No | -| `name` | `str` | Yes | -| `description` | `str` | Yes | -| `code` | `str \| None` | Yes | -| `type` | `str` | Yes | -| `sensor_model` | `str \| None` | Yes | -| `sensor_model_manufacturer` | `str \| None` | Yes | -| `sensor_model_definition` | `str \| None` | Yes | -| `definition` | `str \| None` | Yes | -| `workspace_id` | `UUID \| None` | No | +| Property | Type | Editable | Notes | +|---|---|---|---| +| `uid` | `UUID` | No | | +| `name` | `str` | Yes | | +| `description` | `str` | Yes | | +| `code` | `str \| None` | Yes | | +| `type` | `str` | Yes | | +| `sensor_model` | `str \| None` | Yes | | +| `sensor_model_manufacturer` | `str \| None` | Yes | | +| `sensor_model_definition` | `str \| None` | Yes | | +| `definition` | `str \| None` | Yes | | +| `workspace_id` | `UUID \| None` | No | | | `workspace` | `Workspace \| None` | No | Computed | ```python -hs_api.methods.list(workspace=None) -> HydroServerCollection[Method] +hs_api.methods.list(workspace=..., monitoring_site=..., datastream=..., type=..., + sensor_model=..., sensor_model_manufacturer=...) -> HydroServerCollection[Method] hs_api.methods.get(uid) -> Method hs_api.methods.create(name, description, type, code=None, definition=None, sensor_model=None, sensor_model_manufacturer=None, sensor_model_definition=None, workspace=None, uid=None) -> Method +hs_api.methods.update(uid, name=..., description=..., type=..., code=..., definition=..., + sensor_model=..., sensor_model_manufacturer=..., + sensor_model_definition=...) -> Method ``` ### Observed Properties **Property:** `hs_api.observedproperties` -| Property | Type | Editable | -|---|---|---| -| `uid` | `UUID` | No | -| `name` | `str` | Yes | -| `definition` | `str \| None` | Yes | -| `description` | `str` | Yes | -| `type` | `str` | Yes | -| `code` | `str \| None` | Yes | -| `workspace_id` | `UUID \| None` | No | +| Property | Type | Editable | Notes | +|---|---|---|---| +| `uid` | `UUID` | No | | +| `name` | `str` | Yes | | +| `definition` | `str \| None` | Yes | | +| `description` | `str` | Yes | | +| `type` | `str` | Yes | | +| `code` | `str \| None` | Yes | | +| `workspace_id` | `UUID \| None` | No | | | `workspace` | `Workspace \| None` | No | Computed | ```python -hs_api.observedproperties.list(workspace=None) -> HydroServerCollection[ObservedProperty] +hs_api.observedproperties.list(workspace=..., monitoring_site=..., datastream=..., type=...) -> HydroServerCollection[ObservedProperty] hs_api.observedproperties.get(uid) -> ObservedProperty hs_api.observedproperties.create(name, description, type, code=None, definition=None, workspace=None, uid=None) -> ObservedProperty +hs_api.observedproperties.update(uid, name=..., description=..., type=..., code=..., definition=...) -> ObservedProperty ``` ### Units **Property:** `hs_api.units` -| Property | Type | Editable | -|---|---|---| -| `uid` | `UUID` | No | -| `name` | `str` | Yes | -| `symbol` | `str` | Yes | -| `definition` | `str \| None` | Yes | -| `type` | `str` | Yes | -| `workspace_id` | `UUID \| None` | No | +| Property | Type | Editable | Notes | +|---|---|---|---| +| `uid` | `UUID` | No | | +| `name` | `str` | Yes | | +| `symbol` | `str` | Yes | | +| `definition` | `str \| None` | Yes | | +| `type` | `str` | Yes | | +| `workspace_id` | `UUID \| None` | No | | | `workspace` | `Workspace \| None` | No | Computed | ```python -hs_api.units.list(workspace=None) -> HydroServerCollection[Unit] +hs_api.units.list(workspace=..., monitoring_site=..., datastream=..., type=...) -> HydroServerCollection[Unit] hs_api.units.get(uid) -> Unit hs_api.units.create(name, symbol, type, definition=None, workspace=None, uid=None) -> Unit +hs_api.units.update(uid, name=..., symbol=..., type=..., definition=...) -> Unit ``` ### Processing Levels **Property:** `hs_api.processinglevels` -| Property | Type | Editable | -|---|---|---| -| `uid` | `UUID` | No | -| `code` | `str \| None` | Yes | -| `name` | `str` | Yes | -| `description` | `str` | Yes | -| `definition` | `str \| None` | Yes | -| `workspace_id` | `UUID \| None` | No | +| Property | Type | Editable | Notes | +|---|---|---|---| +| `uid` | `UUID` | No | | +| `code` | `str \| None` | Yes | | +| `name` | `str` | Yes | | +| `description` | `str` | Yes | | +| `definition` | `str \| None` | Yes | | +| `workspace_id` | `UUID \| None` | No | | | `workspace` | `Workspace \| None` | No | Computed | ```python -hs_api.processinglevels.list(workspace=None) -> HydroServerCollection[ProcessingLevel] +hs_api.processinglevels.list(workspace=..., monitoring_site=..., datastream=...) -> HydroServerCollection[ProcessingLevel] hs_api.processinglevels.get(uid) -> ProcessingLevel hs_api.processinglevels.create(name, description, *, code=None, definition=None, workspace=None, uid=None) -> ProcessingLevel +hs_api.processinglevels.update(uid, code=..., name=..., description=..., definition=...) -> ProcessingLevel ``` Processing-level creation accepts `name` and `description` by position or by keyword. Optional arguments must be named, for example `create("Raw", "Unprocessed data", code="0")`. The old `create("0", "Raw", "Unprocessed data")` call raises Python's built-in `TypeError`; move the old first argument to `code=`. Processing-level updates retain their existing argument order: `update(uid, code, name, description, definition)`. @@ -335,21 +477,22 @@ Processing-level creation accepts `name` and `description` by position or by key **Property:** `hs_api.resultqualifiers` -| Property | Type | Editable | -|---|---|---| -| `uid` | `UUID` | No | -| `name` | `str` | Yes | -| `description` | `str` | Yes | -| `workspace_id` | `UUID \| None` | No | +| Property | Type | Editable | Notes | +|---|---|---|---| +| `uid` | `UUID` | No | | +| `name` | `str` | Yes | | +| `description` | `str \| None` | Yes | | +| `workspace_id` | `UUID \| None` | No | | | `workspace` | `Workspace \| None` | No | Computed | ```python -hs_api.resultqualifiers.list(workspace=None) -> HydroServerCollection[ResultQualifier] +hs_api.resultqualifiers.list(workspace=...) -> HydroServerCollection[ResultQualifier] hs_api.resultqualifiers.get(uid) -> ResultQualifier hs_api.resultqualifiers.create(name, description=None, workspace=None, uid=None) -> ResultQualifier +hs_api.resultqualifiers.update(uid, name=..., description=...) -> ResultQualifier ``` -Result-qualifier creation accepts `name` and `description` by position or by keyword; both are optional after `name`. Updates follow `update(uid, name, description)`, with all fields after `uid` optional. +Result-qualifier creation accepts `name` and `description` by position or by keyword; `description` is optional. --- @@ -387,8 +530,8 @@ Result-qualifier creation accepts `name` and `description` by position or by key | `processing_level_id` | `UUID` | Yes | | | `unit_id` | `UUID` | Yes | | | `workspace_id` | `UUID` | No | | -| `tags` | `Dict[str, str]` | No | Use tag methods to modify | -| `linked_resources` | `Dict[str, dict]` | No | Use linked resource methods to modify | +| `tags` | `Dict[str, str]` | No | Use the tag methods to modify | +| `linked_resources` | `Dict[str, dict]` | No | Use the linked resource methods to modify | | `workspace` | `Workspace` | No | Computed | | `monitoring_site` | `MonitoringSite` | No | Computed; also settable via assignment | | `method` | `Method` | No | Computed; also settable via assignment | @@ -406,39 +549,57 @@ datastream.save() ### Service methods ```python -hs_api.datastreams.list(workspace=None, monitoring_site=None) -> HydroServerCollection[Datastream] +hs_api.datastreams.list(workspace=..., monitoring_site=..., method=..., observed_property=..., + processing_level=..., unit=..., observation_type=..., sampled_medium=..., + status=..., result_type=..., tag=..., is_private=..., + value_count_min=..., value_count_max=..., + phenomenon_time_min=..., phenomenon_time_max=...) -> HydroServerCollection[Datastream] hs_api.datastreams.get(uid) -> Datastream hs_api.datastreams.create(name, description, monitoring_site, method, observed_property, processing_level, unit, observation_type, result_type, sampled_medium, no_data_value, aggregation_statistic, time_aggregation_interval, time_aggregation_interval_unit, intended_time_spacing=None, intended_time_spacing_unit=None, status=None, - is_private=False, is_visible=True, uid=None) -> Datastream + is_private=False, is_visible=True, tags=None, uid=None) -> Datastream +hs_api.datastreams.update(uid, name=..., description=..., monitoring_site=..., method=..., + observed_property=..., processing_level=..., unit=..., ...) -> Datastream +hs_api.datastreams.delete(uid) -> None ``` +`update()` accepts every editable field above. Relationships can be given as objects or UUIDs (`method=` or `method_id=`), and `tags=` merges the same way as for monitoring sites. + ### Observation methods ```python datastream.get_observations( - phenomenon_time_min=None, - phenomenon_time_max=None, - result_qualifiers=None, - offset=0, + phenomenon_time_min=..., + phenomenon_time_max=..., + result_qualifiers=..., + offset=..., limit=100000, - sortby=None, + sortby=..., fetch_all=False, ) -> ObservationCollection ``` -Returns an `ObservationCollection` with a `dataframe` property containing a pandas DataFrame with `phenomenon_time` (timezone-aware datetime), `result` (float), and `result_qualifiers` (list of qualifier names) columns. +Returns an `ObservationCollection`. Its `dataframe` property is a pandas DataFrame with `phenomenon_time` (timezone-aware datetime), `result` (float), and `result_qualifiers` (list of qualifier names) columns. Like `HydroServerCollection`, it has `offset`, `limit`, and `total_count` properties and `next_page()`, `previous_page()`, and `fetch_all()` methods. Pass `result_qualifiers="PF"` to return only observations with that qualifier. ```python -datastream.load_observations(observations, mode='insert') -> None +datastream.load_observations(observations, mode='insert', phenomenon_time_start=None, phenomenon_time_end=None) -> None ``` -`observations` must be a pandas DataFrame with `phenomenon_time` and `result` columns. `phenomenon_time` must be timezone-aware. `mode` is `"insert"` (skip existing timestamps) or `"replace"` (overwrite all observations in the datastream). +`observations` must be a pandas DataFrame with `phenomenon_time` and `result` columns; `phenomenon_time` must be timezone-aware. + +| `mode` | Behavior | +|---|---| +| `"insert"` (default) | Adds the observations. If any timestamp already exists in the datastream, the whole request is rejected with a `409` and nothing is loaded. | +| `"append"` | Adds the observations only if all of them are after the datastream's latest observation; otherwise the request is rejected with a `400`. | +| `"backfill"` | Adds the observations only if all of them are before the datastream's earliest observation; otherwise the request is rejected with a `400`. | +| `"replace"` | Replaces the datastream's observations between the earliest and latest timestamp in your DataFrame with the ones you provide. Observations outside that range are untouched. | + +In `replace` mode, pass both `phenomenon_time_start` and `phenomenon_time_end` (timezone-aware, inclusive) to replace that range instead of the DataFrame's span — for example, to also clear observations in the range that your DataFrame no longer contains. The delete and insert happen in a single transaction: if the load fails, the existing observations are kept. -To load observations with result qualifiers, include a `result_qualifiers` column containing a list of qualifier name strings per row: +To load observations with result qualifiers, include a `result_qualifiers` column containing a list of qualifier names per row. The names must match result qualifiers that exist in the datastream's workspace: ```python df = pd.DataFrame({ @@ -455,9 +616,15 @@ datastream.delete_observations(phenomenon_time_start=None, phenomenon_time_end=N Deletes observations within the given time range. If both parameters are omitted, all observations are deleted. +```python +datastream.sync_phenomenon_end_time() -> None +``` + +Reloads `phenomenon_end_time` and the datastream's other observation statistics, which HydroServer updates whenever observations are written. + ### Tag and linked resource methods -Same as monitoring sites — `set_tag`, `delete_tag`, `add_linked_resource`, `delete_linked_resource`. +Same as monitoring sites — `set_tag`, `delete_tag`, `add_linked_resource`, `delete_linked_resource` on the object, and `get_linked_resources`, `get_linked_resource`, `update_linked_resource`, `delete_linked_resource` on the service. --- @@ -476,17 +643,19 @@ Same as monitoring sites — `set_tag`, `delete_tag`, `add_linked_resource`, `de | `description` | `str \| None` | Yes | | | `source_url` | `str` | Yes | URL template; use `{variable_name}` for placeholder substitution | | `timezone_type` | `str \| None` | Yes | `"offset"` or `"iana"` | -| `timezone` | `str \| None` | Yes | e.g. `"+0000"` or `"America/Denver"` | +| `timezone` | `str \| None` | Yes | e.g. `"-0700"` or `"America/Denver"` | | `auth_header_name` | `str \| None` | Yes | | | `auth_header_value` | `str \| None` | Yes | | -| `payload` | `CSVPayload \| JSONPayload` | No | See below | -| `placeholder_variables` | `List[PlaceholderVariable]` | No | See below | -| `notification` | `Notification \| None` | No | | +| `payload` | `CSVPayload \| JSONPayload` | Yes | See below | +| `placeholder_variables` | `List[PlaceholderVariable]` | Yes | See below | +| `notification` | `Notification \| None` | Yes | See below | | `workspace_id` | `UUID` | No | | -| `workspace_name` | `str` | No | | +| `workspace` | `Workspace` | No | Computed | | `task_count` | `int` | No | Number of ETL tasks using this connection | | `task_attention_count` | `int` | No | Tasks with a recent failure or overdue schedule | +`DataConnection.save()` sends all of the connection's fields, including the full payload and placeholder variable list. + **CSVPayload fields:** `payload_type` (`"CSV"`), `timestamp_key`, `timestamp_format`, `header_row`, `data_start_row`, `delimiter` **JSONPayload fields:** `payload_type` (`"JSON"`), `timestamp_key`, `timestamp_format`, `jmespath` @@ -496,24 +665,41 @@ Same as monitoring sites — `set_tag`, `delete_tag`, `add_linked_resource`, `de | Field | Type | Description | |---|---|---| | `name` | `str` | Variable name as it appears in `{braces}` in `source_url` | -| `variable_type` | `str` | `"per_task"`, `"run_time"`, or `"latest_observation_timestamp"` | -| `timestamp_format` | `str \| None` | Custom strftime format; `None` uses ISO 8601 | +| `variable_type` | `str` | `"per_task"`, `"run_time"`, `"latest_observation_timestamp"`, `"window_start"`, or `"window_end"` | +| `timestamp_format` | `str \| None` | Custom strftime format for timestamp variables; `None` uses ISO 8601 | + +`per_task` values are supplied by each ETL task's `task_variables`. `run_time` is the time the task runs, and `latest_observation_timestamp` is the most recent observation already loaded (1970-01-01 for an empty datastream). + +**Notification fields:** `recipient_emails` (`List[str]`) and `schedule` (a [schedule](#schedules), or `None`). On each scheduled notification, HydroServer emails the recipients a summary of the connection's ETL task runs since the previous one. #### Service methods ```python -hs_api.dataconnections.list(workspace=None, payload_type=None) -> HydroServerCollection[DataConnection] +hs_api.dataconnections.list(workspace=..., payload_type=...) -> HydroServerCollection[DataConnection] hs_api.dataconnections.get(uid) -> DataConnection hs_api.dataconnections.create( - name, workspace, source_url, payload_type, timestamp_key, + name, workspace, source_url, timestamp_key, payload_type, description=None, timestamp_format=None, timezone_type=None, timezone=None, auth_header_name=None, auth_header_value=None, header_row=None, data_start_row=None, delimiter=None, jmespath=None, placeholder_variables=None, notification=None, uid=None ) -> DataConnection -hs_api.dataconnections.update(uid, name, source_url, payload_type, timestamp_key, ...) -> DataConnection +hs_api.dataconnections.update( + uid, name, source_url, timestamp_key, payload_type, + description=..., timestamp_format=..., timezone_type=..., timezone=..., + auth_header_name=..., auth_header_value=..., + header_row=None, data_start_row=None, delimiter=None, jmespath=None, + placeholder_variables=None, notification=... +) -> DataConnection +hs_api.dataconnections.delete(uid) -> None ``` +`placeholder_variables` is a list of dicts with `name`, `variable_type`, and optionally `timestamp_format`. + +::: warning +`dataconnections.update()` replaces the payload settings and the placeholder variable list with what you pass. If you leave out `placeholder_variables`, the connection's placeholder variables are **removed**; pass the full list on every update. Updating the object with `save()` avoids this, since it sends the current values. +::: + --- ### ETL Tasks @@ -529,44 +715,81 @@ hs_api.dataconnections.update(uid, name, source_url, payload_type, timestamp_key | `description` | `str \| None` | Yes | | | `task_variables` | `Dict[str, Any]` | Yes | Values for `per_task` placeholder variables | | `data_connection_id` | `UUID` | No | | -| `enabled` | `bool \| None` | Yes | | -| `start_time` | `datetime \| None` | Yes | | -| `crontab` | `str \| None` | Yes | | -| `interval` | `int \| None` | Yes | | -| `interval_period` | `str \| None` | Yes | `"minutes"`, `"hours"`, `"days"` | +| `workspace_id` | `UUID` | No | | +| `enabled`, `crontab`, `interval`, `interval_period`, `start_time` | | Yes | See [Schedules](#schedules) | | `next_run_at` | `datetime \| None` | No | | | `latest_run` | `TaskRun \| None` | No | | -| `mappings` | `List[EtlMapping]` | Yes | | +| `mapping_count` | `int` | No | Number of mappings on this task | +| `mappings` | `List[EtlMapping]` | No | Computed; manage with `hs_api.etlmappings` | | `data_connection` | `DataConnection` | No | Computed | -**EtlMapping fields:** `source_identifier` (`str`), `target_datastream` (`DatastreamSummary` with `id` and `name`) +`EtlTask.save()` sends all of the task's editable fields, including the full `task_variables` dict. #### Service methods ```python hs_api.etltasks.list( - workspace=None, data_connection=None, latest_run_status=None, - latest_run_started_at_min=None, latest_run_started_at_max=None, - latest_run_finished_at_min=None, latest_run_finished_at_max=None, + workspace=..., data_connection=..., latest_run_status=..., + latest_run_started_at_min=..., latest_run_started_at_max=..., + latest_run_finished_at_min=..., latest_run_finished_at_max=..., ) -> HydroServerCollection[EtlTask] hs_api.etltasks.get(uid) -> EtlTask hs_api.etltasks.create( name, data_connection, description=None, task_variables=None, - mappings=None, crontab=None, interval=None, interval_period=None, + crontab=None, interval=None, interval_period=None, start_time=None, enabled=True, uid=None ) -> EtlTask +hs_api.etltasks.update( + uid, name, description=..., task_variables=None, + crontab=..., interval=..., interval_period=..., start_time=..., enabled=... +) -> EtlTask +hs_api.etltasks.delete(uid) -> None ``` -Mappings are a list of dicts with `source_identifier` and `target_datastream_id` keys. +::: warning +`etltasks.update()` requires `name` and replaces `task_variables` with what you pass. If you leave out `task_variables`, the task's variables are **cleared**; pass the full dict on every update, or change the object and call `save()`. +::: #### Run methods | Method | Returns | Description | |---|---|---| | `trigger()` | `TaskRun` | Dispatch an immediate run | -| `list_runs(status=None, started_at_min=None, started_at_max=None, finished_at_min=None, finished_at_max=None, page=1, page_size=100, sortby=None)` | `List[TaskRun]` | Fetch run history | +| `list_runs(status=..., started_at_min=..., started_at_max=..., finished_at_min=..., finished_at_max=..., offset=..., limit=..., sortby=...)` | `List[TaskRun]` | Fetch run history, newest first | | `get_run(run_id)` | `TaskRun` | Fetch a single run by ID | +The service has the same methods with the task UID as the first argument, e.g. `hs_api.etltasks.trigger(uid)`. + +--- + +### ETL Mappings + +**Property:** `hs_api.etlmappings` + +A mapping tells an ETL task which field (or column) in the extracted data to load into which datastream. A task needs at least one mapping to load anything. + +| Property | Type | Editable | Notes | +|---|---|---|---| +| `uid` | `UUID` | No | | +| `task_id` | `UUID` | No | | +| `source_identifier` | `str` | Yes | Column name (CSV) or record field (JSON) holding the values | +| `target_datastream_id` | `UUID` | Yes | Datastream the values are loaded into | + +```python +hs_api.etlmappings.list(etl_task=..., workspace=..., source_identifier=...) -> HydroServerCollection[EtlMapping] +hs_api.etlmappings.get(uid) -> EtlMapping +hs_api.etlmappings.create(task_id, source_identifier, target_datastream, uid=None) -> EtlMapping +hs_api.etlmappings.update(uid, source_identifier=..., target_datastream=...) -> EtlMapping +hs_api.etlmappings.delete(uid) -> None +``` + +```python +task = hs_api.etltasks.create(name="Gage height", data_connection=connection, + task_variables={"site_code": "10109000"}) +hs_api.etlmappings.create(task_id=task.uid, source_identifier="value", + target_datastream=datastream) +``` + --- ## Data Products @@ -583,18 +806,24 @@ Mappings are a list of dicts with `source_identifier` and `target_datastream_id` | `name` | `str` | Yes | | | `description` | `str \| None` | Yes | | | `fitting_method` | `str` | Yes | `"linear"` or `"power_law"` | +| `points` | `List[Tuple[float, float]]` | Yes | List of `(input, output)` pairs | | `monitoring_site_id` | `UUID` | No | | -| `thing_name` | `str` | No | | -| `points` | `List[Tuple[float, float]]` | Yes | List of `(input, output)` coordinate pairs | +| `monitoring_site` | `MonitoringSite` | No | Computed | + +`RatingCurve.save()` sends all of the curve's editable fields. #### Service methods ```python -hs_api.ratingcurves.list(workspace=None, monitoring_site=None) -> HydroServerCollection[RatingCurve] +hs_api.ratingcurves.list(workspace=..., monitoring_site=...) -> HydroServerCollection[RatingCurve] hs_api.ratingcurves.get(uid) -> RatingCurve hs_api.ratingcurves.create(name, monitoring_site, fitting_method, description=None, points=None, uid=None) -> RatingCurve +hs_api.ratingcurves.update(uid, name, fitting_method, points, description=...) -> RatingCurve +hs_api.ratingcurves.delete(uid) -> None ``` +`update()` requires `name`, `fitting_method`, and the full `points` list, which replaces the existing points. + --- ### Data Product Tasks @@ -609,30 +838,32 @@ hs_api.ratingcurves.create(name, monitoring_site, fitting_method, description=No | `name` | `str` | Yes | | | `description` | `str \| None` | Yes | | | `monitoring_site_id` | `UUID` | No | | -| `thing_name` | `str` | No | | -| `enabled` | `bool \| None` | Yes | | -| `start_time` | `datetime \| None` | Yes | | -| `crontab` | `str \| None` | Yes | | -| `interval` | `int \| None` | Yes | | -| `interval_period` | `str \| None` | Yes | `"minutes"`, `"hours"`, `"days"` | +| `workspace_id` | `UUID` | No | | +| `enabled`, `crontab`, `interval`, `interval_period`, `start_time` | | Yes | See [Schedules](#schedules) | | `next_run_at` | `datetime \| None` | No | | | `latest_run` | `TaskRun \| None` | No | | -| `rating_curve_transformations` | `List[RatingCurveTransformation]` | No | Manage via `hs_api.dataproducttransformations` | -| `derivation_transformations` | `List[DerivationTransformation]` | No | Manage via `hs_api.dataproducttransformations` | -| `aggregation_transformations` | `List[AggregationTransformation]` | No | Manage via `hs_api.dataproducttransformations` | +| `transformation_types` | `List[str]` | No | Types of transformations attached to the task | +| `rating_curve_transformations` | `List[RatingCurveTransformation]` | No | Computed; manage via `hs_api.dataproducttransformations` | +| `derivation_transformations` | `List[DerivationTransformation]` | No | Computed; manage via `hs_api.dataproducttransformations` | +| `aggregation_transformations` | `List[AggregationTransformation]` | No | Computed; manage via `hs_api.dataproducttransformations` | #### Service methods ```python hs_api.dataproducttasks.list( - workspace=None, monitoring_site=None, latest_run_status=None, transformation_type=None, - output_datastream=None, input_datastream=None, rating_curve=None, + workspace=..., monitoring_site=..., latest_run_status=..., transformation_type=..., + output_datastream=..., input_datastream=..., rating_curve=..., ) -> HydroServerCollection[DataProductTask] hs_api.dataproducttasks.get(uid) -> DataProductTask hs_api.dataproducttasks.create( name, monitoring_site, description=None, crontab=None, interval=None, interval_period=None, start_time=None, enabled=True, uid=None ) -> DataProductTask +hs_api.dataproducttasks.update( + uid, name=..., description=..., crontab=..., interval=..., interval_period=..., + start_time=..., enabled=... +) -> DataProductTask +hs_api.dataproducttasks.delete(uid) -> None ``` #### Run methods @@ -645,23 +876,33 @@ Same as ETL Tasks: `trigger()`, `list_runs(...)`, `get_run(run_id)`. **Property:** `hs_api.dataproducttransformations` -Transformations are always scoped to a specific data product task via `task_id`. All three types share a consistent set of CRUD methods. +Transformations are attached to a data product task when created (`task_id`), and identified by their own `uid` after that. All three types share these fields: + +| Field | Type | Description | +|---|---|---| +| `id` | `UUID` | | +| `task_id` | `UUID` | | +| `output_datastream_id` | `UUID` | | +| `input_datastreams` | `List[TransformationInput]` | Each has `datastream_id` and `variable_name` | +| `transformation_type` | `str` | `"rating_curve"`, `"derivation"`, or `"aggregation"` | + +The `list_*` methods take optional filters: `task_id`, `workspace_id`, `output_datastream`, and `input_datastream`. #### Rating Curve Transformations +Converts each input value using a [rating curve](#rating-curves). + | Field | Type | Description | |---|---|---| -| `id` | `UUID` | | -| `output_datastream` | `DatastreamSummary` | `id` and `name` | -| `input_datastream` | `DatastreamSummary` | `id` and `name` | -| `rating_curve` | `RatingCurveSummary` | `id`, `name`, and `fitting_method` | +| `rating_curve_id` | `UUID` | | +| `input_datastream_id` | `UUID` | Convenience property for the single input | ```python -hs_api.dataproducttransformations.list_rating_curve(task_id, output_datastream=None, input_datastream=None) -hs_api.dataproducttransformations.get_rating_curve(task_id, uid) +hs_api.dataproducttransformations.list_rating_curve(task_id=None, workspace_id=None, output_datastream=None, input_datastream=None) +hs_api.dataproducttransformations.get_rating_curve(uid) hs_api.dataproducttransformations.create_rating_curve(task_id, output_datastream, input_datastream, rating_curve, uid=None) -hs_api.dataproducttransformations.update_rating_curve(task_id, uid, input_datastream, rating_curve) -hs_api.dataproducttransformations.delete_rating_curve(task_id, uid) +hs_api.dataproducttransformations.update_rating_curve(uid, input_datastream, rating_curve) +hs_api.dataproducttransformations.delete_rating_curve(uid) ``` #### Derivation Transformations @@ -670,37 +911,31 @@ Combines one or more input datastreams using a formula. Each input datastream is | Field | Type | Description | |---|---|---| -| `id` | `UUID` | | -| `output_datastream_id` | `UUID` | | -| `input_datastream_ids` | `List[UUID]` | | | `formula` | `str` | Python expression referencing the input variable name(s) | | `stop_on_no_data` | `bool` | If `True`, stop the run when an input holds its no-data value; if `False`, fill the output with its no-data value and continue | | `stop_on_error` | `bool` | If `True`, stop the run when the formula produces a non-finite result; if `False`, fill the output with its no-data value and continue | ```python -hs_api.dataproducttransformations.list_derivation(task_id, output_datastream=None, input_datastream=None) -hs_api.dataproducttransformations.get_derivation(task_id, uid) +hs_api.dataproducttransformations.list_derivation(task_id=None, workspace_id=None, output_datastream=None, input_datastream=None) +hs_api.dataproducttransformations.get_derivation(uid) hs_api.dataproducttransformations.create_derivation( task_id, output_datastream, input_datastreams, formula, stop_on_no_data=True, stop_on_error=True, uid=None ) hs_api.dataproducttransformations.update_derivation( - task_id, uid, input_datastreams=..., formula=..., - stop_on_no_data=..., stop_on_error=... + uid, input_datastreams=..., formula=..., stop_on_no_data=..., stop_on_error=... ) -hs_api.dataproducttransformations.delete_derivation(task_id, uid) +hs_api.dataproducttransformations.delete_derivation(uid) ``` -`input_datastreams` is a list of dicts with `datastream_id` and `variable_name`. On `update_derivation`, any argument left at its default (`...`) is omitted from the request, leaving the existing value in place. +`input_datastreams` is a list of dicts with `datastream_id` and `variable_name`, for example `[{'datastream_id': stage.uid, 'variable_name': 'x'}]`. On `update_derivation`, any argument you leave out is left unchanged. #### Aggregation Transformations | Field | Type | Description | |---|---|---| -| `id` | `UUID` | | -| `output_datastream` | `DatastreamSummary` | | -| `input_datastream` | `DatastreamSummary` | | -| `aggregation_method` | `str` | `"mean"`, `"sum"`, `"min"`, `"max"`, `"first"`, `"last"` | +| `input_datastream_id` | `UUID` | Convenience property for the single input | +| `aggregation_method` | `str` | `"mean"`, `"sum"`, `"min"`, `"max"`, `"first"`, `"last"`, or `"time_weighted_mean"` | | `output_interval` | `int` | | | `output_interval_units` | `str` | `"minutes"`, `"hours"`, `"days"`, `"weeks"`, `"months"` | | `timezone_type` | `str \| None` | `"utc"`, `"offset"`, or `"iana"` | @@ -708,21 +943,23 @@ hs_api.dataproducttransformations.delete_derivation(task_id, uid) | `min_values` | `int \| None` | Minimum number of input observations required to produce an output | ```python -hs_api.dataproducttransformations.list_aggregation(task_id, output_datastream=None, input_datastream=None) -hs_api.dataproducttransformations.get_aggregation(task_id, uid) +hs_api.dataproducttransformations.list_aggregation(task_id=None, workspace_id=None, output_datastream=None, input_datastream=None) +hs_api.dataproducttransformations.get_aggregation(uid) hs_api.dataproducttransformations.create_aggregation( task_id, output_datastream, input_datastream, aggregation_method, output_interval, output_interval_units, timezone_type=None, timezone=None, min_values=None, uid=None ) hs_api.dataproducttransformations.update_aggregation( - task_id, uid, input_datastream, aggregation_method, + uid, input_datastream, aggregation_method, output_interval, output_interval_units, timezone_type=None, timezone=None, min_values=None ) -hs_api.dataproducttransformations.delete_aggregation(task_id, uid) +hs_api.dataproducttransformations.delete_aggregation(uid) ``` +`update_rating_curve` and `update_aggregation` replace the whole transformation: pass every field, since optional fields you leave out are reset to `None`. + --- ## Monitoring @@ -739,26 +976,20 @@ hs_api.dataproducttransformations.delete_aggregation(task_id, uid) | `name` | `str` | Yes | | | `description` | `str \| None` | Yes | | | `monitoring_site_id` | `UUID` | No | | -| `thing_name` | `str` | No | | +| `workspace_id` | `UUID` | No | | | `recipients` | `List[str]` | Yes | Email addresses to alert | -| `enabled` | `bool \| None` | Yes | | -| `start_time` | `datetime \| None` | Yes | | -| `crontab` | `str \| None` | Yes | | -| `interval` | `int \| None` | Yes | | -| `interval_period` | `str \| None` | Yes | `"minutes"`, `"hours"`, `"days"` | +| `enabled`, `crontab`, `interval`, `interval_period`, `start_time` | | Yes | See [Schedules](#schedules) | | `next_run_at` | `datetime \| None` | No | | | `latest_run` | `TaskRun \| None` | No | | -| `monitored_datastreams` | `List[MonitoredDatastream]` | No | Inline summary; manage rules via `hs_api.monitoringrules` | +| `rule_type_counts` | `Dict[str, int]` | No | Number of rules of each type, e.g. `{"range": 1, "missing_data": 1}` | | `rules` | `List[MonitoringRule]` | No | Computed; full rule objects for this task | -**MonitoredDatastream fields:** `datastream_id`, `datastream_name`, `rules` (list of inline rule summaries with `id`, `rule_type`, `last_checked_at`, `min_value`, `max_value`, `window_interval`, `window_interval_units`) - #### Service methods ```python hs_api.monitoringtasks.list( - workspace=None, monitoring_site=None, latest_run_status=None, - datastream=None, rule_type=None, + workspace=..., monitoring_site=..., latest_run_status=..., + datastream=..., rule_type=..., ) -> HydroServerCollection[MonitoringTask] hs_api.monitoringtasks.get(uid) -> MonitoringTask hs_api.monitoringtasks.create( @@ -766,6 +997,11 @@ hs_api.monitoringtasks.create( crontab=None, interval=None, interval_period=None, start_time=None, enabled=True, uid=None ) -> MonitoringTask +hs_api.monitoringtasks.update( + uid, name=..., description=..., recipients=..., crontab=..., interval=..., + interval_period=..., start_time=..., enabled=... +) -> MonitoringTask +hs_api.monitoringtasks.delete(uid) -> None ``` #### Run methods @@ -778,7 +1014,7 @@ Same as ETL Tasks: `trigger()`, `list_runs(...)`, `get_run(run_id)`. **Property:** `hs_api.monitoringrules` -Rules are always scoped to a monitoring task via `task_id`. +Each rule belongs to a monitoring task and checks one datastream. #### MonitoringRule properties @@ -787,7 +1023,6 @@ Rules are always scoped to a monitoring task via `task_id`. | `uid` | `UUID` | No | | | `task_id` | `UUID` | No | | | `datastream_id` | `UUID` | No | | -| `datastream_name` | `str` | No | | | `rule_type` | `str` | No | `"range"`, `"rate_of_change"`, `"persistence"`, or `"missing_data"` | | `last_checked_at` | `datetime \| None` | No | | | `min_value` | `float \| None` | Yes | Used by `range` and `rate_of_change` | @@ -807,12 +1042,49 @@ Rules are always scoped to a monitoring task via `task_id`. #### Service methods ```python -hs_api.monitoringrules.list(task_id, datastream=None, rule_type=None) -> HydroServerCollection[MonitoringRule] -hs_api.monitoringrules.get(task_id, uid) -> MonitoringRule +hs_api.monitoringrules.list(task=..., workspace=..., datastream=..., rule_type=...) -> HydroServerCollection[MonitoringRule] +hs_api.monitoringrules.get(uid) -> MonitoringRule hs_api.monitoringrules.create( task_id, datastream, rule_type, min_value=None, max_value=None, window_interval=None, window_interval_units=None, uid=None ) -> MonitoringRule +hs_api.monitoringrules.update(uid, min_value=..., max_value=..., window_interval=..., window_interval_units=...) -> MonitoringRule +hs_api.monitoringrules.delete(uid) -> None ``` + +To get the datastream a rule checks, use `hs_api.datastreams.get(rule.datastream_id)`. + +--- + +## Quality Control + +`HydroServerQualityControl` provides quality-control operations on a pandas DataFrame of observations. It works entirely locally — it doesn't read from or write to HydroServer. Fetch observations first, edit them, then load the result back with `load_observations`. + +```python +from hydroserverpy import HydroServerQualityControl + +df = datastream.get_observations(fetch_all=True).dataframe +qc_df = df.rename(columns={"phenomenon_time": "timestamp", "result": "value"})[["timestamp", "value"]] + +qc = HydroServerQualityControl(datastream_id=datastream.uid, observations=qc_df) +qc.change_values(index_list=[0, 1], operator="MULT", value=0.3048) +corrected = qc.observations +``` + +The DataFrame must have a datetime `timestamp` column and a float `value` column. Edits are applied to the object's copy; read the result from `qc.observations`. + +| Method | Description | +|---|---| +| `filter(data_filter)` | Restrict `observations` to rows matching filters, e.g. `{"GT": 10.0, "LTE": 20.0}` (operations `LT`, `LTE`, `GT`, `GTE`, `E`) | +| `find_gaps(time_value, time_unit)` | Return rows that follow a gap longer than the given duration | +| `fill_gap(gap, fill, interpolate_values)` | Fill gaps longer than `gap` with points every `fill` (both `(value, unit)` tuples), optionally interpolating their values | +| `add_points(points, index=None)` | Insert `[timestamp, value]` points | +| `change_values(index_list, operator, value)` | Apply `MULT`, `DIV`, `ADD`, `SUB`, or `ASSIGN` to the values at the given row indices | +| `delete_points(index_list)` | Delete the rows at the given indices | +| `shift_points(index_list, time_value, time_unit)` | Shift the timestamps of the given rows | +| `interpolate(index_list)` | Replace the given rows' values by linear interpolation | +| `drift_correction(start, end, gap_width)` | Apply a linear drift correction across a row index range | + +Time units are `"s"`, `"m"`, `"h"`, `"D"`, `"W"`, `"M"`, and `"Y"`. diff --git a/apps/docs/docs/user-guides/tutorials/getting-started-with-hydroserverpy/index.md b/apps/docs/docs/user-guides/tutorials/getting-started-with-hydroserverpy/index.md index 8baf9d0ab..60b9d9c8e 100644 --- a/apps/docs/docs/user-guides/tutorials/getting-started-with-hydroserverpy/index.md +++ b/apps/docs/docs/user-guides/tutorials/getting-started-with-hydroserverpy/index.md @@ -7,7 +7,7 @@ In this tutorial series, you'll use `hydroserverpy` to build a complete end-to-e 1. **Setting Up Your Site and Datastreams** — Connect to HydroServer and programmatically create your monitoring site, linked metadata, and datastreams. 2. **Loading and Visualizing Observations** — Load a set of historical stage observations into your datastream and plot them. 3. **Automating Data Ingestion** — Set up a data connection and ETL task to automatically pull new stage readings from a web API on a schedule. -4. **Creating Data Products** — Define a rating curve to convert stage measurements into discharge, and set up a data product task to compute and store discharge values automatically. +4. **Creating Data Products** — Set up data product tasks that automatically compute daily mean gage height and convert the readings from feet to meters. 5. **Setting Up Monitoring** — Configure a monitoring task that checks your datastreams on a schedule and sends email alerts when values fall outside acceptable bounds. Each part builds directly on the last, so work through them in order. By the end, you'll have a fully automated monitoring pipeline running in HydroServer — set up entirely through Python. @@ -18,7 +18,7 @@ You'll need a HydroServer account. If you don't have one, you can create a free ### Python environment -This tutorial requires **Python 3.9 or higher** and uses **hydroserverpy 1.10.x**. We recommend installing into an isolated environment so it doesn't interfere with other projects on your machine. If you're using `venv`: +This tutorial requires **Python 3.9 or higher** and uses **hydroserverpy 2.x**. We recommend installing into an isolated environment so it doesn't interfere with other projects on your machine. If you're using `venv`: ```bash python -m venv hydroserver-env @@ -35,7 +35,7 @@ conda activate hydroserver-env Once your environment is active, install `hydroserverpy`: ```bash -pip install "hydroserverpy>=1.10,<1.11" +pip install "hydroserverpy>=2.0,<3" ``` That's it — let's get started! diff --git a/apps/docs/docs/user-guides/tutorials/getting-started-with-hydroserverpy/part-1-setting-up-your-site-and-datastreams.md b/apps/docs/docs/user-guides/tutorials/getting-started-with-hydroserverpy/part-1-setting-up-your-site-and-datastreams.md index 259c8308d..f20bf1e26 100644 --- a/apps/docs/docs/user-guides/tutorials/getting-started-with-hydroserverpy/part-1-setting-up-your-site-and-datastreams.md +++ b/apps/docs/docs/user-guides/tutorials/getting-started-with-hydroserverpy/part-1-setting-up-your-site-and-datastreams.md @@ -2,7 +2,7 @@ In this first part, we'll connect to HydroServer and create everything needed to represent a monitoring site: a workspace, a site, linked metadata (method, observed property, unit, and processing level), and a datastream. By the end you'll have a fully described datastream ready to receive observations in the next part. -The site we'll be building throughout this tutorial is a fictional stream gauge station on the Cache River in northern Utah. We'll use it to measure water surface elevation (stage) — a common measurement at stream monitoring stations. +The site we'll be building in this part is a stream gauge station on the Blacksmith Fork River in northern Utah, modeled on a Logan River Observatory site. We'll use it to record discharge — the volume of water flowing past the station per unit time. ## Connect to HydroServer @@ -108,7 +108,7 @@ discharge_observed_property = hs_api.observedproperties.create( discharge_unit = hs_api.units.create( workspace=workspace, name='Cubic meters per second', - symbol='m3^s', + symbol='m^3/s', definition='https://qudt.org/vocab/unit/M3-PER-SEC', type='Flow rate' ) @@ -165,4 +165,4 @@ We'll use `discharge_datastream` directly in the next part to load observations. In a few dozen lines of Python, we've created a fully described monitoring site in HydroServer with a datastream ready to receive data. If you open the web interface and navigate to your workspace, you should see the Blacksmith Fork River above confluence with Logan River on the map and the discharge datastream listed in its detail page. -In Part 2, we'll load some sample stage observations into that datastream and visualize them. +In Part 2, we'll load some sample discharge observations into that datastream and visualize them. diff --git a/apps/docs/docs/user-guides/tutorials/getting-started-with-hydroserverpy/part-2-loading-and-visualizing-observations.md b/apps/docs/docs/user-guides/tutorials/getting-started-with-hydroserverpy/part-2-loading-and-visualizing-observations.md index d459afb08..201805b19 100644 --- a/apps/docs/docs/user-guides/tutorials/getting-started-with-hydroserverpy/part-2-loading-and-visualizing-observations.md +++ b/apps/docs/docs/user-guides/tutorials/getting-started-with-hydroserverpy/part-2-loading-and-visualizing-observations.md @@ -10,7 +10,7 @@ pip install pandas numpy matplotlib ## Generate sample observations -Since we're working with a fictional monitoring station, we'll generate some synthetic discharge data rather than using a real dataset. The code below creates 30 days of 15-minute readings starting April 1, 2026, with a gentle rise-and-fall pattern typical of spring snowmelt. +To keep this part self-contained, we'll generate some synthetic discharge data rather than using a real dataset. The code below creates 30 days of 15-minute readings starting April 1, 2026. ```python import pandas as pd @@ -52,7 +52,7 @@ discharge_datastream.load_observations(observations) print(f"Loaded {len(observations)} observations.") ``` -By default, `load_observations` runs in `insert` mode, which adds new observations to the datastream if observations at those timestamps don't already exist. If you want to overwrite everything in the provided range, you can pass `mode='replace'` instead. +By default, `load_observations` runs in `insert` mode, which adds the observations to the datastream. If any of the timestamps already exist, the whole request is rejected with a `409` error and nothing is loaded — so running this cell twice will fail the second time. To overwrite existing observations instead, pass `mode='replace'`, which replaces the datastream's observations within the time range covered by your DataFrame. ## Fetch observations back @@ -70,7 +70,7 @@ print(f"Retrieved {len(df)} observations.") print(df.describe()) ``` -`get_observations` returns an `ObservationCollection` — a paginated result object. Calling `.dataframe` on it gives you a standard pandas DataFrame. We passed `fetch_all=True` here so that all pages are merged automatically; for large datasets with millions of rows you may want to work page by page instead. +`get_observations` returns an `ObservationCollection` — a paginated result object. Its `.dataframe` property gives you a standard pandas DataFrame. We passed `fetch_all=True` here so that all pages are merged automatically; for large datasets with millions of rows you may want to work page by page instead. ## Plot the data @@ -84,7 +84,7 @@ ax.plot(df['phenomenon_time'], df['result'], linewidth=0.8, color='steelblue') ax.set_xlabel('Date') ax.set_ylabel('Discharge (m3/s)') -ax.set_title('Blacksmith Fork River Discharge — April 2024') +ax.set_title('Blacksmith Fork River Discharge — April 2026') ax.xaxis.set_major_formatter(mdates.DateFormatter('%b %d')) ax.xaxis.set_major_locator(mdates.WeekdayLocator(byweekday=mdates.MO)) fig.autofmt_xdate() @@ -92,7 +92,7 @@ plt.tight_layout() plt.show() ``` -You should see a smooth curve that rises through mid-April and falls off toward the end of the month — our simulated snowmelt hydrograph. +You should see a month of 15-minute discharge readings plotted across April. ## What we've built diff --git a/apps/docs/docs/user-guides/tutorials/getting-started-with-hydroserverpy/part-3-automating-data-ingestion.md b/apps/docs/docs/user-guides/tutorials/getting-started-with-hydroserverpy/part-3-automating-data-ingestion.md index 76f2bca00..c3b495385 100644 --- a/apps/docs/docs/user-guides/tutorials/getting-started-with-hydroserverpy/part-3-automating-data-ingestion.md +++ b/apps/docs/docs/user-guides/tutorials/getting-started-with-hydroserverpy/part-3-automating-data-ingestion.md @@ -10,14 +10,14 @@ The real station we're mirroring is [USGS 10109000 — Logan River Above State D This should feel familiar from Part 1. We're creating a new site and stage datastream — this time for the USGS Logan River Above State Dam monitoring location. -Note that USGS reports gage height in feet, so we'll use feet as the unit here rather than meters. +Note that USGS reports gage height in feet, so we'll use feet as the unit for this datastream. ```python # Site logan_river_site = hs_api.monitoring_sites.create( workspace=workspace, - name='Logan River Above First Dam', - description='USGS gauging station 101090000 on the Logan River near Logan, Utah.', + name='Logan River Above State Dam', + description='USGS gauging station 10109000 on the Logan River near Logan, Utah.', code='USGS-10109000', type='Stream', is_private=False, @@ -79,7 +79,7 @@ stage_datastream = hs_api.datastreams.create( unit=feet_unit, observation_type='Field Observation', result_type='Timeseries', - sampled_medium='Surface Water', + sampled_medium='Surface water', no_data_value=-9999.0, aggregation_statistic='Instantaneous', time_aggregation_interval=1, @@ -95,7 +95,7 @@ print(f"{stage_datastream.name}: {stage_datastream.uid}") ## Create a data connection -A data connection describes where your data comes from and how to read it. Our source is the USGS Instantaneous Values API, which returns JSON. We'll define two run-time placeholder variables — `start_date` and `end_date` — that get substituted into the URL each time the task runs. HydroServer automatically sets `start_date` to the timestamp of the most recent observation already in the datastream, so you never re-fetch data you already have. Our datastream is new and has no observations yet, so the first run starts from 1970-01-01 and pulls in all the history USGS has for this gage; later runs pick up from the most recent observation. +A data connection describes where your data comes from and how to read it. Our source is the USGS Instantaneous Values API, which returns JSON. Each run asks USGS for the last seven days of readings (`period=P7D`). That window overlaps with data we've already loaded, but HydroServer only loads observations newer than the most recent one already in the datastream, so each run adds just the new readings. The `{site_code}` and `{param_code}` placeholders in the URL are filled in by each task that uses this connection. ```python usgs_data_connection = hs_api.dataconnections.create( @@ -106,8 +106,7 @@ usgs_data_connection = hs_api.dataconnections.create( '?format=json' '&sites={site_code}' '¶meterCd={param_code}' - '&startDT={start_date}' - '&endDT={end_date}' + '&period=P7D' ), payload_type='JSON', timestamp_key='dateTime', @@ -115,8 +114,6 @@ usgs_data_connection = hs_api.dataconnections.create( placeholder_variables=[ {'name': 'site_code', 'variable_type': 'per_task'}, {'name': 'param_code', 'variable_type': 'per_task'}, - {'name': 'start_date', 'variable_type': 'latest_observation_timestamp', 'timestamp_format': '%Y-%m-%dT%H:%M:%SZ'}, - {'name': 'end_date', 'variable_type': 'run_time', 'timestamp_format': '%Y-%m-%dT%H:%M:%SZ'}, ], ) @@ -127,7 +124,7 @@ The `site_code` and `param_code` variables are marked `per_task`, which means ea ## Create an ETL task -Now we tie everything together. The task supplies values for the `per_task` placeholder variables and defines the mapping from the source data to the target datastream. +Now we tie everything together. The task supplies values for the `per_task` placeholder variables and the schedule it runs on. A mapping, created right after the task, tells it which field in the source data goes to which datastream. ```python usgs_etl_task = hs_api.etltasks.create( @@ -137,19 +134,21 @@ usgs_etl_task = hs_api.etltasks.create( 'site_code': '10109000', 'param_code': '00065', }, - mappings=[{ - 'source_identifier': 'value', - 'target_datastream_id': str(stage_datastream.uid), - }], interval=15, interval_period='minutes', enabled=True, ) +hs_api.etlmappings.create( + task_id=usgs_etl_task.uid, + source_identifier='value', + target_datastream=stage_datastream, +) + print(f"{usgs_etl_task.name}: {usgs_etl_task.uid}") ``` -The `source_identifier` tells the pipeline which series in the USGS response maps to this datastream. USGS identifies each time series by its parameter code, so `'00065'` corresponds to the gage height series we requested. +The mapping's `source_identifier` names the field in each extracted record that holds the value. The JMESPath expression on the data connection turns the USGS response into records like `{"value": "2.57", "qualifiers": ["P"], "dateTime": "..."}`, so the identifier is `'value'`. Which USGS parameter you get — here gage height, `00065` — is chosen by the task's `param_code` variable. ## Trigger the task manually @@ -170,11 +169,9 @@ print(f"Status: {latest.status}") print(f"Message: {latest.message}") ``` -If the status comes back as `SUCCESS`, your stage datastream should now have real observations in it from USGS. You can verify by fetching them the same way we did in Part 2: +If the status comes back as `SUCCESS`, your stage datastream should now have real observations in it from USGS. If it stays `PENDING`, no orchestration worker has picked up the run yet — check that your instance's workers are running. You can verify the data by fetching it the same way we did in Part 2: ```python -from datetime import datetime, timezone - result = stage_datastream.get_observations(fetch_all=True) print(f"Observations in datastream: {len(result.dataframe)}") print(result.dataframe.tail()) @@ -184,6 +181,6 @@ If the status is `FAILURE`, the message field will tell you what went wrong — ## Scheduling -With `interval=15` and `interval_period='minutes'`, the orchestration system will keep re-running this task every 15 minutes automatically, each time picking up exactly from where the last successful run left off. You don't need to do anything else to keep the data flowing. +With `interval=15` and `interval_period='minutes'`, the orchestration system will keep re-running this task every 15 minutes automatically. Each run fetches the last seven days from USGS and loads only the readings newer than what's already in the datastream, so you don't need to do anything else to keep the data flowing. In Part 4, we'll take these raw stage readings and use a data product task to automatically convert them into discharge values using a rating curve. diff --git a/apps/docs/docs/user-guides/tutorials/getting-started-with-hydroserverpy/part-4-creating-data-products.md b/apps/docs/docs/user-guides/tutorials/getting-started-with-hydroserverpy/part-4-creating-data-products.md index 61f60408d..5fc613a12 100644 --- a/apps/docs/docs/user-guides/tutorials/getting-started-with-hydroserverpy/part-4-creating-data-products.md +++ b/apps/docs/docs/user-guides/tutorials/getting-started-with-hydroserverpy/part-4-creating-data-products.md @@ -42,7 +42,7 @@ daily_stage_datastream = hs_api.datastreams.create( unit=feet_unit, observation_type='Field Observation', result_type='Timeseries', - sampled_medium='Surface Water', + sampled_medium='Surface water', no_data_value=-9999.0, aggregation_statistic='Mean', time_aggregation_interval=1, @@ -62,7 +62,7 @@ stage_meters_datastream = hs_api.datastreams.create( unit=meters_unit, observation_type='Field Observation', result_type='Timeseries', - sampled_medium='Surface Water', + sampled_medium='Surface water', no_data_value=-9999.0, aggregation_statistic='Instantaneous', time_aggregation_interval=1, @@ -121,7 +121,7 @@ print(f"Status: {latest.status}") print(f"Message: {latest.message}") ``` -If it succeeds, the `daily_stage_datastream` will have one observation per calendar day. +If it succeeds, the `daily_stage_datastream` will have one observation per calendar day. As in Part 3, a status that stays `PENDING` means no orchestration worker has picked up the run yet. ## Derivation transformation task @@ -138,15 +138,14 @@ meters_conversion_task = hs_api.dataproducttasks.create( ) ``` -Attach a derivation transformation. The formula is evaluated for each input observation, where `x` is the raw value. One foot equals 0.3048 meters. +Attach a derivation transformation. Each input datastream is given a variable name to use in the formula — here `x` — and the formula is evaluated for each input observation. One foot equals 0.3048 meters. ```python hs_api.dataproducttransformations.create_derivation( task_id=meters_conversion_task.uid, output_datastream=stage_meters_datastream, - input_datastream=stage_datastream, + input_datastreams=[{'datastream_id': stage_datastream.uid, 'variable_name': 'x'}], formula='x * 0.3048', - variable_name='x', ) ``` diff --git a/apps/docs/docs/user-guides/tutorials/getting-started-with-hydroserverpy/part-5-setting-up-monitoring.md b/apps/docs/docs/user-guides/tutorials/getting-started-with-hydroserverpy/part-5-setting-up-monitoring.md index bc9a2b2a4..75777927b 100644 --- a/apps/docs/docs/user-guides/tutorials/getting-started-with-hydroserverpy/part-5-setting-up-monitoring.md +++ b/apps/docs/docs/user-guides/tutorials/getting-started-with-hydroserverpy/part-5-setting-up-monitoring.md @@ -79,11 +79,11 @@ print(f"Status: {latest.status}") print(f"Message: {latest.message}") ``` -A `SUCCESS` status means the task ran and evaluated your rules — not necessarily that everything is fine with the data. If a rule was violated, the recipients will have received an email. You can verify which rules are attached to the task with: +A `SUCCESS` status means the task ran and evaluated your rules — not necessarily that everything is fine with the data. (A status that stays `PENDING` means no orchestration worker has picked up the run yet.) If a rule was violated, the recipients will have received an email. You can verify which rules are attached to the task with: ```python for rule in stage_monitor.rules: - print(rule.rule_type, rule.datastream_name) + print(rule.rule_type, rule.datastream_id) ``` ## What we've built diff --git a/apps/docs/docs/user-guides/tutorials/hydroserver-101/part-1-creating-your-first-site.md b/apps/docs/docs/user-guides/tutorials/hydroserver-101/part-1-creating-your-first-site.md index 9dd02d8b1..8b12aeab3 100644 --- a/apps/docs/docs/user-guides/tutorials/hydroserver-101/part-1-creating-your-first-site.md +++ b/apps/docs/docs/user-guides/tutorials/hydroserver-101/part-1-creating-your-first-site.md @@ -18,7 +18,7 @@ Click 'Add workspace' and give your workspace a name. Make it unique since every ## Create your site -After the workspace is successfully created, open **Browse monitoring sites** from the main navigation. Expand the monitoring-sites panel if it is collapsed, then click the **Register a monitoring site** (+) button. Choose the workspace where the site should be registered. +After the workspace is successfully created, open **Browse monitoring sites** from the main navigation. Expand the monitoring sites panel if it is collapsed, then click the **Create Site** button. Choose the workspace where the site should be registered. The button opens the Register/Edit a site modal window. Click anywhere on the map and it will create a new map marker and automatically populate the location form fields. You can also type values into the 'Site Location' fields if you know them already. Fill out the rest of the form with the metadata for your site and click save. diff --git a/apps/docs/docs/user-guides/tutorials/hydroserver-101/part-2-creating-your-first-datastream.md b/apps/docs/docs/user-guides/tutorials/hydroserver-101/part-2-creating-your-first-datastream.md index 00498bd96..99b74f1f6 100644 --- a/apps/docs/docs/user-guides/tutorials/hydroserver-101/part-2-creating-your-first-datastream.md +++ b/apps/docs/docs/user-guides/tutorials/hydroserver-101/part-2-creating-your-first-datastream.md @@ -2,7 +2,7 @@ Now that your site is created, we'll next define a new datastream to specify what kind of data we'll be loading in the next step. If you're not already on the site details page of your new site, open **Browse monitoring sites**, select your site in the panel, and click **View details**. -Near the bottom of the site details page, there will be an empty datastreams table which will work the same as the sites table you saw before. Click 'Add new datastream' to open the datastream creation modal. +Near the bottom of the site details page, there will be an empty datastreams table. Click 'Add new datastream' to open the datastream creation modal. datastream table @@ -39,11 +39,11 @@ If you'd like to follow along exactly, we'll be creating a datastream that repre - type: Hydrology - code: temp -sensor form +sensor form ## Specify datastream metadata -Next, for 'Unit' and 'Processing Level', we'll use variables that were defined at the system level by an admin of playground.hydroserver.org. These values are available for all workspaces within the Playground instance because they were defined by an administrator as system-level metadata. For unit, type in 'degree celsius' and find it in the list. For processing level, select level-0, Raw Data. +Next, for 'Unit' and 'Processing Level', we'll use variables that were defined at the system level by an admin of playground.hydroserver.org. These values are available for all workspaces within the Playground instance because they were defined by an administrator as system-level metadata. For unit, type in 'Degree Celsius' and find it in the list. For processing level, select 'Raw Data'. For the rest of the fields, we'll do the following: @@ -54,7 +54,7 @@ For the rest of the fields, we'll do the following: - Time Aggregation Interval: 1 day - Intended Time Spacing: 1 day -Enter `Air Temperature - Daily Max, RAW` as the datastream name. See [Naming Datastreams](/user-guides/how-to/managing-datastream-metadata#naming-datastreams) for more guidance. After filling out the other fields, click 'Auto-Fill from Form' below the description to generate a description. You can edit this text before clicking 'Create Datastream'. +Enter `Air Temperature - Daily Max, RAW` as the datastream name. See [Naming Datastreams](/user-guides/how-to/managing-datastream-metadata#naming-datastreams) for more guidance. After filling out the other fields, click 'Auto-Fill from Form' below the description to generate a description. You can edit this text before clicking 'Create datastream'. new datastream diff --git a/apps/docs/docs/user-guides/tutorials/hydroserver-101/part-3-sdl-setup.md b/apps/docs/docs/user-guides/tutorials/hydroserver-101/part-3-sdl-setup.md index e9e7d4ff5..d31a249e0 100644 --- a/apps/docs/docs/user-guides/tutorials/hydroserver-101/part-3-sdl-setup.md +++ b/apps/docs/docs/user-guides/tutorials/hydroserver-101/part-3-sdl-setup.md @@ -15,18 +15,18 @@ If you're going with user name and password, click 'confirm' on the form and fee streaming data loader -## Create an API key +## Create a Service Account -From the main navigation bar on playground.hydroserver.org, go back to 'Your sites'. Right below the map, click the 'Manage workspaces' button to display the workspace table if it's not already present. Find the table row with your workspace name and click the blue padlock icon in the 'actions' section. This will open the 'access control' modal. Click on 'API keys' on the left menu. +From the main navigation bar on playground.hydroserver.org, go back to 'Manage workspaces'. Select your workspace name and click the 'Service accounts' tab. access control -On the top right of the access control modal, click the 'create API key' button and fill out the form. +Click the 'Create service account' button and fill out the form. -new API key +new service account -Notice I've selected 'Data Loader' as the role for this API key. This role grants the holder of the key permission to push data into the datastreams of this workspace and nothing else. Handy if you eventually need to share keys with others. +Notice I've selected 'Data Loader' as the role for this service account. This role grants the service account permission to push data into the datastreams of this workspace and nothing else. Handy if you eventually need to share a service account's API key with others. -As the form says, once you close the modal, the website will never show you this API key again. So store it somewhere safe. If you lose it, you can always come back to this modal and generate a new key to replace the lost one. +As the form says, once you close the modal, the website will never show you this service account's API key again. So store it somewhere safe. If you lose it, you can always come back to this modal and generate a new key to replace the lost one. Copy the API key and paste it in the HydroServer API key field of the Streaming Data Loader form to authenticate. If authentication is successful, you'll be prompted to select the CSV file you wish to pull data from and the datastream you wish to push those data into. We'll handle that in the next step. diff --git a/apps/docs/package-lock.json b/apps/docs/package-lock.json index 7a4a07ccd..01d220dd5 100644 --- a/apps/docs/package-lock.json +++ b/apps/docs/package-lock.json @@ -4,7 +4,6 @@ "requires": true, "packages": { "": { - "name": "docs", "devDependencies": { "vitepress": "^1.3.4" } diff --git a/django/contracts/openapi/ogc.openapi.json b/django/contracts/openapi/ogc.openapi.json index 6c6a642f4..dd9cc489f 100644 --- a/django/contracts/openapi/ogc.openapi.json +++ b/django/contracts/openapi/ogc.openapi.json @@ -16836,6 +16836,17 @@ } }, "description": "Forbidden" + }, + "409": { + "content": { + "application/json": { + "schema": { + "title": "Response", + "type": "string" + } + } + }, + "description": "Conflict" } }, "security": [ @@ -17088,6 +17099,17 @@ } }, "description": "Forbidden" + }, + "409": { + "content": { + "application/json": { + "schema": { + "title": "Response", + "type": "string" + } + } + }, + "description": "Conflict" } }, "security": [ @@ -17443,6 +17465,17 @@ } }, "description": "Not Found" + }, + "409": { + "content": { + "application/json": { + "schema": { + "title": "Response", + "type": "string" + } + } + }, + "description": "Conflict" } }, "security": [ @@ -17750,6 +17783,17 @@ } }, "description": "Not Found" + }, + "409": { + "content": { + "application/json": { + "schema": { + "title": "Response", + "type": "string" + } + } + }, + "description": "Conflict" } }, "security": [ @@ -19389,6 +19433,17 @@ } }, "description": "Not Found" + }, + "409": { + "content": { + "application/json": { + "schema": { + "title": "Response", + "type": "string" + } + } + }, + "description": "Conflict" } }, "security": [ @@ -19706,6 +19761,17 @@ } }, "description": "Not Found" + }, + "409": { + "content": { + "application/json": { + "schema": { + "title": "Response", + "type": "string" + } + } + }, + "description": "Conflict" } }, "security": [ @@ -19996,6 +20062,17 @@ } }, "description": "Forbidden" + }, + "409": { + "content": { + "application/json": { + "schema": { + "title": "Response", + "type": "string" + } + } + }, + "description": "Conflict" } }, "security": [ @@ -20248,6 +20325,17 @@ } }, "description": "Forbidden" + }, + "409": { + "content": { + "application/json": { + "schema": { + "title": "Response", + "type": "string" + } + } + }, + "description": "Conflict" } }, "security": [ @@ -20863,6 +20951,17 @@ } }, "description": "Forbidden" + }, + "409": { + "content": { + "application/json": { + "schema": { + "title": "Response", + "type": "string" + } + } + }, + "description": "Conflict" } }, "security": [ @@ -21514,6 +21613,17 @@ }, "description": "Forbidden" }, + "409": { + "content": { + "application/json": { + "schema": { + "title": "Response", + "type": "string" + } + } + }, + "description": "Conflict" + }, "413": { "content": { "application/json": { @@ -21860,6 +21970,17 @@ }, "description": "Not Found" }, + "409": { + "content": { + "application/json": { + "schema": { + "title": "Response", + "type": "string" + } + } + }, + "description": "Conflict" + }, "413": { "content": { "application/json": { @@ -22694,6 +22815,17 @@ } }, "description": "Forbidden" + }, + "409": { + "content": { + "application/json": { + "schema": { + "title": "Response", + "type": "string" + } + } + }, + "description": "Conflict" } }, "security": [ @@ -23070,6 +23202,17 @@ } }, "description": "Not Found" + }, + "409": { + "content": { + "application/json": { + "schema": { + "title": "Response", + "type": "string" + } + } + }, + "description": "Conflict" } }, "security": [ @@ -23375,6 +23518,17 @@ } }, "description": "Not Found" + }, + "409": { + "content": { + "application/json": { + "schema": { + "title": "Response", + "type": "string" + } + } + }, + "description": "Conflict" } }, "security": [ @@ -24935,6 +25089,17 @@ } }, "description": "Forbidden" + }, + "409": { + "content": { + "application/json": { + "schema": { + "title": "Response", + "type": "string" + } + } + }, + "description": "Conflict" } }, "security": [ @@ -25187,6 +25352,17 @@ } }, "description": "Forbidden" + }, + "409": { + "content": { + "application/json": { + "schema": { + "title": "Response", + "type": "string" + } + } + }, + "description": "Conflict" } }, "security": [ @@ -25477,6 +25653,17 @@ } }, "description": "Forbidden" + }, + "409": { + "content": { + "application/json": { + "schema": { + "title": "Response", + "type": "string" + } + } + }, + "description": "Conflict" } }, "security": [ @@ -25729,6 +25916,17 @@ } }, "description": "Forbidden" + }, + "409": { + "content": { + "application/json": { + "schema": { + "title": "Response", + "type": "string" + } + } + }, + "description": "Conflict" } }, "security": [ @@ -26190,6 +26388,17 @@ } }, "description": "Forbidden" + }, + "409": { + "content": { + "application/json": { + "schema": { + "title": "Response", + "type": "string" + } + } + }, + "description": "Conflict" } }, "security": [ @@ -26875,6 +27084,17 @@ } }, "description": "Not Found" + }, + "409": { + "content": { + "application/json": { + "schema": { + "title": "Response", + "type": "string" + } + } + }, + "description": "Conflict" } }, "security": [ @@ -27185,6 +27405,17 @@ } }, "description": "Not Found" + }, + "409": { + "content": { + "application/json": { + "schema": { + "title": "Response", + "type": "string" + } + } + }, + "description": "Conflict" } }, "security": [ @@ -27475,6 +27706,17 @@ } }, "description": "Forbidden" + }, + "409": { + "content": { + "application/json": { + "schema": { + "title": "Response", + "type": "string" + } + } + }, + "description": "Conflict" } }, "security": [ @@ -27727,6 +27969,17 @@ } }, "description": "Forbidden" + }, + "409": { + "content": { + "application/json": { + "schema": { + "title": "Response", + "type": "string" + } + } + }, + "description": "Conflict" } }, "security": [ @@ -28191,9 +28444,20 @@ } }, "description": "Unauthorized" - } - }, - "security": [ + }, + "409": { + "content": { + "application/json": { + "schema": { + "title": "Response", + "type": "string" + } + } + }, + "description": "Conflict" + } + }, + "security": [ { "SessionAuth": [] }, @@ -28761,6 +29025,17 @@ }, "description": "Forbidden" }, + "409": { + "content": { + "application/json": { + "schema": { + "title": "Response", + "type": "string" + } + } + }, + "description": "Conflict" + }, "413": { "content": { "application/json": { @@ -29107,6 +29382,17 @@ }, "description": "Not Found" }, + "409": { + "content": { + "application/json": { + "schema": { + "title": "Response", + "type": "string" + } + } + }, + "description": "Conflict" + }, "413": { "content": { "application/json": { @@ -30858,6 +31144,17 @@ } }, "description": "Not Found" + }, + "409": { + "content": { + "application/json": { + "schema": { + "title": "Response", + "type": "string" + } + } + }, + "description": "Conflict" } }, "security": [ @@ -31289,6 +31586,17 @@ } }, "description": "Forbidden" + }, + "409": { + "content": { + "application/json": { + "schema": { + "title": "Response", + "type": "string" + } + } + }, + "description": "Conflict" } }, "security": [ @@ -31895,6 +32203,17 @@ } }, "description": "Forbidden" + }, + "409": { + "content": { + "application/json": { + "schema": { + "title": "Response", + "type": "string" + } + } + }, + "description": "Conflict" } }, "security": [ @@ -32483,6 +32802,17 @@ } }, "description": "Forbidden" + }, + "409": { + "content": { + "application/json": { + "schema": { + "title": "Response", + "type": "string" + } + } + }, + "description": "Conflict" } }, "security": [ @@ -32735,6 +33065,17 @@ } }, "description": "Forbidden" + }, + "409": { + "content": { + "application/json": { + "schema": { + "title": "Response", + "type": "string" + } + } + }, + "description": "Conflict" } }, "security": [ @@ -33115,6 +33456,17 @@ } }, "description": "Forbidden" + }, + "409": { + "content": { + "application/json": { + "schema": { + "title": "Response", + "type": "string" + } + } + }, + "description": "Conflict" } }, "security": [ @@ -33761,6 +34113,17 @@ } }, "description": "Not Found" + }, + "409": { + "content": { + "application/json": { + "schema": { + "title": "Response", + "type": "string" + } + } + }, + "description": "Conflict" } }, "security": [ @@ -34368,6 +34731,17 @@ } }, "description": "Not Found" + }, + "409": { + "content": { + "application/json": { + "schema": { + "title": "Response", + "type": "string" + } + } + }, + "description": "Conflict" } }, "security": [ @@ -35102,6 +35476,17 @@ } }, "description": "Not Found" + }, + "409": { + "content": { + "application/json": { + "schema": { + "title": "Response", + "type": "string" + } + } + }, + "description": "Conflict" } }, "security": [ @@ -35438,6 +35823,17 @@ } }, "description": "Not Found" + }, + "409": { + "content": { + "application/json": { + "schema": { + "title": "Response", + "type": "string" + } + } + }, + "description": "Conflict" } }, "security": [ @@ -35754,6 +36150,17 @@ } }, "description": "Forbidden" + }, + "409": { + "content": { + "application/json": { + "schema": { + "title": "Response", + "type": "string" + } + } + }, + "description": "Conflict" } }, "security": [ @@ -36026,6 +36433,17 @@ } }, "description": "Forbidden" + }, + "409": { + "content": { + "application/json": { + "schema": { + "title": "Response", + "type": "string" + } + } + }, + "description": "Conflict" } }, "security": [ @@ -36642,6 +37060,17 @@ } }, "description": "Forbidden" + }, + "409": { + "content": { + "application/json": { + "schema": { + "title": "Response", + "type": "string" + } + } + }, + "description": "Conflict" } }, "security": [ @@ -36894,6 +37323,17 @@ } }, "description": "Forbidden" + }, + "409": { + "content": { + "application/json": { + "schema": { + "title": "Response", + "type": "string" + } + } + }, + "description": "Conflict" } }, "security": [ @@ -37184,6 +37624,17 @@ } }, "description": "Forbidden" + }, + "409": { + "content": { + "application/json": { + "schema": { + "title": "Response", + "type": "string" + } + } + }, + "description": "Conflict" } }, "security": [ @@ -37436,6 +37887,17 @@ } }, "description": "Forbidden" + }, + "409": { + "content": { + "application/json": { + "schema": { + "title": "Response", + "type": "string" + } + } + }, + "description": "Conflict" } }, "security": [ @@ -37824,6 +38286,17 @@ } }, "description": "Unauthorized" + }, + "409": { + "content": { + "application/json": { + "schema": { + "title": "Response", + "type": "string" + } + } + }, + "description": "Conflict" } }, "security": [ @@ -38462,6 +38935,17 @@ } }, "description": "Unauthorized" + }, + "409": { + "content": { + "application/json": { + "schema": { + "title": "Response", + "type": "string" + } + } + }, + "description": "Conflict" } }, "security": [ @@ -38752,6 +39236,17 @@ } }, "description": "Forbidden" + }, + "409": { + "content": { + "application/json": { + "schema": { + "title": "Response", + "type": "string" + } + } + }, + "description": "Conflict" } }, "security": [ @@ -39104,6 +39599,17 @@ } }, "description": "Forbidden" + }, + "409": { + "content": { + "application/json": { + "schema": { + "title": "Response", + "type": "string" + } + } + }, + "description": "Conflict" } }, "security": [ @@ -39420,6 +39926,17 @@ } }, "description": "Unauthorized" + }, + "409": { + "content": { + "application/json": { + "schema": { + "title": "Response", + "type": "string" + } + } + }, + "description": "Conflict" } }, "security": [ @@ -39429,9 +39946,6 @@ { "OIDCAuth": [] }, - { - "APIKeyAuth": [] - }, { "BasicAuth": [] } @@ -39501,6 +40015,9 @@ }, { "OIDCAuth": [] + }, + { + "BasicAuth": [] } ], "summary": "Delete Service Account", @@ -39619,6 +40136,12 @@ }, { "OIDCAuth": [] + }, + { + "APIKeyAuth": [] + }, + { + "BasicAuth": [] } ], "summary": "Get Service Account", @@ -39699,6 +40222,9 @@ }, { "OIDCAuth": [] + }, + { + "BasicAuth": [] } ], "summary": "Update Service Account", @@ -39784,6 +40310,9 @@ }, { "OIDCAuth": [] + }, + { + "BasicAuth": [] } ], "summary": "Regenerate Service Account Key", diff --git a/django/core/iam/migrations/0006_v2_0_0.py b/django/core/iam/migrations/0006_v2_0_0.py index 3b7325204..a876d0c1d 100644 --- a/django/core/iam/migrations/0006_v2_0_0.py +++ b/django/core/iam/migrations/0006_v2_0_0.py @@ -563,6 +563,7 @@ class Migration(migrations.Migration): fields=("workspace", "user"), name="unique_workspace_user_collaborator", violation_error_message="This account already collaborates on the workspace", + violation_error_code="unique", ), ), migrations.AddConstraint( @@ -572,6 +573,7 @@ class Migration(migrations.Migration): fields=("workspace", "service_account"), name="unique_workspace_service_account_collaborator", violation_error_message="This account already collaborates on the workspace", + violation_error_code="unique", ), ), diff --git a/django/core/iam/models/collaborator.py b/django/core/iam/models/collaborator.py index 203ec9945..9fa6a3e11 100644 --- a/django/core/iam/models/collaborator.py +++ b/django/core/iam/models/collaborator.py @@ -49,12 +49,14 @@ class Meta: condition=Q(user__isnull=False), name="unique_workspace_user_collaborator", violation_error_message="This account already collaborates on the workspace", + violation_error_code="unique", ), models.UniqueConstraint( fields=["workspace", "service_account"], condition=Q(service_account__isnull=False), name="unique_workspace_service_account_collaborator", violation_error_message="This account already collaborates on the workspace", + violation_error_code="unique", ), ] diff --git a/django/core/iam/models/workspace.py b/django/core/iam/models/workspace.py index 8aa487062..3e26f0c25 100644 --- a/django/core/iam/models/workspace.py +++ b/django/core/iam/models/workspace.py @@ -66,7 +66,7 @@ def clean(self): if Workspace.objects.filter(name=self.name, owner=self.owner).exclude( pk=self.pk ).exists(): - raise ValidationError("Workspace name conflicts with an owned workspace") + raise ValidationError("Workspace name conflicts with an owned workspace", code="unique") if not self._state.adding: original_workspace = Workspace.objects.get(pk=self.pk) diff --git a/django/interfaces/api/http/handlers.py b/django/interfaces/api/http/handlers.py index e7ef4f7a4..9c466dacc 100644 --- a/django/interfaces/api/http/handlers.py +++ b/django/interfaces/api/http/handlers.py @@ -8,6 +8,8 @@ logger = logging.getLogger(__name__) +UNIQUE_ERROR_CODES = {"unique", "unique_together"} + def http_error_handler(request, exc, api): return api.create_response( @@ -18,10 +20,14 @@ def http_error_handler(request, exc, api): def validation_error_handler(request, exc, api): + errors = ( + [e for errs in exc.error_dict.values() for e in errs] + if hasattr(exc, "error_dict") else exc.error_list + ) return api.create_response( request, {"message": "; ".join(exc.messages)}, - status=400, + status=409 if all(e.code in UNIQUE_ERROR_CODES for e in errors) else 400, ) diff --git a/django/interfaces/api/services/etl/task.py b/django/interfaces/api/services/etl/task.py index 7d15c5c28..00c6ccbf2 100644 --- a/django/interfaces/api/services/etl/task.py +++ b/django/interfaces/api/services/etl/task.py @@ -24,6 +24,8 @@ User = get_user_model() +CELERY_TASK_NAME = "processing.etl.tasks.run_etl_task" + class EtlTaskAPIService(TaskService[EtlTask], APIService): @@ -236,7 +238,7 @@ def create( interval_period=schedule.interval_period if schedule else None, start_time=schedule.start_time if schedule else None, enabled=schedule.enabled if schedule else True, - celery_task_name="processing.etl.tasks.run_etl_task", + celery_task_name=CELERY_TASK_NAME, ) return {"id": task.pk} @@ -270,5 +272,5 @@ def update( interval_period=schedule.interval_period if schedule else None, start_time=schedule.start_time if schedule else None, enabled=schedule.enabled if schedule else True, - celery_task_name="processing.etl.tasks.run_etl_task", + celery_task_name=CELERY_TASK_NAME, ) diff --git a/django/interfaces/api/views/etl/data_connection.py b/django/interfaces/api/views/etl/data_connection.py index 2e283661d..554c2ff37 100644 --- a/django/interfaces/api/views/etl/data_connection.py +++ b/django/interfaces/api/views/etl/data_connection.py @@ -121,6 +121,7 @@ def get_data_connection( 400: str, 401: str, 403: str, + 409: str, }, by_alias=True, ) diff --git a/django/interfaces/api/views/etl/mapping.py b/django/interfaces/api/views/etl/mapping.py index 08917f1dc..82f749c35 100644 --- a/django/interfaces/api/views/etl/mapping.py +++ b/django/interfaces/api/views/etl/mapping.py @@ -54,6 +54,7 @@ def get_etl_mappings( 401: str, 403: str, 404: str, + 409: str, }, by_alias=True, ) @@ -103,6 +104,7 @@ def get_etl_mapping( 401: str, 403: str, 404: str, + 409: str, }, by_alias=True, ) diff --git a/django/interfaces/api/views/iam/collaborator.py b/django/interfaces/api/views/iam/collaborator.py index 4547fe519..73321ee52 100644 --- a/django/interfaces/api/views/iam/collaborator.py +++ b/django/interfaces/api/views/iam/collaborator.py @@ -57,6 +57,7 @@ def get_collaborators( 401: str, 403: str, 400: str, + 409: str, }, by_alias=True, ) diff --git a/django/interfaces/api/views/iam/service_account.py b/django/interfaces/api/views/iam/service_account.py index 300b085f8..456e21fa9 100644 --- a/django/interfaces/api/views/iam/service_account.py +++ b/django/interfaces/api/views/iam/service_account.py @@ -53,10 +53,11 @@ def get_service_accounts( @service_account_router.post( "", - auth=[session_auth, oidc_iam_write_auth, apikey_auth, basic_auth], + auth=[session_auth, oidc_iam_write_auth, basic_auth], response={ 201: ServiceAccountCreatedResponse, 401: str, + 409: str, }, by_alias=True, ) @@ -79,7 +80,7 @@ def create_service_account( @service_account_router.get( "/{service_account_id}", - auth=[session_auth, oidc_iam_read_auth], + auth=[session_auth, oidc_iam_read_auth, apikey_auth, basic_auth], response={ 200: ItemResponse[ServiceAccountResponse], 401: str, @@ -107,7 +108,7 @@ def get_service_account( @service_account_router.patch( "/{service_account_id}", - auth=[session_auth, oidc_iam_write_auth], + auth=[session_auth, oidc_iam_write_auth, basic_auth], response={ 204: None, 401: str, @@ -138,7 +139,7 @@ def update_service_account( @service_account_router.delete( "/{service_account_id}", - auth=[session_auth, oidc_iam_write_auth], + auth=[session_auth, oidc_iam_write_auth, basic_auth], response={ 204: None, 401: str, @@ -163,7 +164,7 @@ def delete_service_account( @service_account_router.put( "/{service_account_id}/regenerate", - auth=[session_auth, oidc_iam_write_auth], + auth=[session_auth, oidc_iam_write_auth, basic_auth], response={ 201: ServiceAccountKeyResponse, 400: str, diff --git a/django/interfaces/api/views/iam/workspace.py b/django/interfaces/api/views/iam/workspace.py index 0a87d0063..7d1d31430 100644 --- a/django/interfaces/api/views/iam/workspace.py +++ b/django/interfaces/api/views/iam/workspace.py @@ -61,6 +61,7 @@ def get_workspaces( 201: CreatedResponse, 400: str, 401: str, + 409: str, }, by_alias=True, ) @@ -110,6 +111,7 @@ def get_workspace( 400: str, 401: str, 403: str, + 409: str, }, by_alias=True, ) diff --git a/django/interfaces/api/views/monitoring/rule.py b/django/interfaces/api/views/monitoring/rule.py index 44f98c8fd..e7867cb87 100644 --- a/django/interfaces/api/views/monitoring/rule.py +++ b/django/interfaces/api/views/monitoring/rule.py @@ -56,6 +56,7 @@ def get_monitoring_rules( 401: str, 403: str, 404: str, + 409: str, }, by_alias=True, ) @@ -109,6 +110,7 @@ def get_monitoring_rule( 401: str, 403: str, 404: str, + 409: str, }, by_alias=True, ) diff --git a/django/interfaces/api/views/products/rating_curve.py b/django/interfaces/api/views/products/rating_curve.py index af0ff1da1..eb4a49df9 100644 --- a/django/interfaces/api/views/products/rating_curve.py +++ b/django/interfaces/api/views/products/rating_curve.py @@ -54,6 +54,7 @@ def get_rating_curves( 401: str, 403: str, 404: str, + 409: str, }, by_alias=True, ) @@ -107,6 +108,7 @@ def get_rating_curve( 401: str, 403: str, 404: str, + 409: str, }, by_alias=True, ) diff --git a/django/interfaces/api/views/products/transformation.py b/django/interfaces/api/views/products/transformation.py index 9fd02cead..c719e0a13 100644 --- a/django/interfaces/api/views/products/transformation.py +++ b/django/interfaces/api/views/products/transformation.py @@ -54,6 +54,7 @@ def get_data_product_transformations( 401: str, 403: str, 404: str, + 409: str, }, by_alias=True, ) @@ -103,6 +104,7 @@ def get_data_product_transformation( 401: str, 403: str, 404: str, + 409: str, }, by_alias=True, ) diff --git a/django/interfaces/api/views/quality/history.py b/django/interfaces/api/views/quality/history.py index b99c982a0..69247eb9e 100644 --- a/django/interfaces/api/views/quality/history.py +++ b/django/interfaces/api/views/quality/history.py @@ -47,7 +47,7 @@ def get_qc_histories( @qc_history_router.post( "/items", auth=[session_auth, oidc_data_write_auth, apikey_auth, basic_auth], - response={201: CreatedResponse, 400: str, 401: str, 403: str, 404: str}, + response={201: CreatedResponse, 400: str, 401: str, 403: str, 404: str, 409: str}, by_alias=True, ) def create_qc_history( diff --git a/django/interfaces/api/views/quality/operation.py b/django/interfaces/api/views/quality/operation.py index 31f7bab97..0f2423222 100644 --- a/django/interfaces/api/views/quality/operation.py +++ b/django/interfaces/api/views/quality/operation.py @@ -45,7 +45,7 @@ def get_qc_operations( @qc_operation_router.post( "", auth=[session_auth, oidc_data_write_auth, apikey_auth, basic_auth], - response={201: list[CreatedResponse], 400: str, 401: str, 403: str, 404: str}, + response={201: list[CreatedResponse], 400: str, 401: str, 403: str, 404: str, 409: str}, by_alias=True, ) def create_qc_operations( @@ -90,7 +90,7 @@ def get_qc_operation( @qc_operation_router.patch( "/{operation_id}", auth=[session_auth, oidc_data_write_auth, apikey_auth, basic_auth], - response={204: None, 400: str, 401: str, 403: str, 404: str}, + response={204: None, 400: str, 401: str, 403: str, 404: str, 409: str}, by_alias=True, ) def update_qc_operation( diff --git a/django/interfaces/api/views/quality/session.py b/django/interfaces/api/views/quality/session.py index 190714303..afe11c152 100644 --- a/django/interfaces/api/views/quality/session.py +++ b/django/interfaces/api/views/quality/session.py @@ -49,7 +49,7 @@ def get_qc_sessions( @qc_session_router.post( "", auth=[session_auth, oidc_data_write_auth, apikey_auth, basic_auth], - response={201: CreatedResponse, 400: str, 401: str, 403: str, 404: str}, + response={201: CreatedResponse, 400: str, 401: str, 403: str, 404: str, 409: str}, by_alias=True, ) def create_qc_session( diff --git a/django/interfaces/api/views/sta/aggregation_statistic.py b/django/interfaces/api/views/sta/aggregation_statistic.py index b541d64af..b3fd5b8ca 100644 --- a/django/interfaces/api/views/sta/aggregation_statistic.py +++ b/django/interfaces/api/views/sta/aggregation_statistic.py @@ -55,6 +55,7 @@ def get_aggregation_statistics( 201: CreatedResponse, 401: str, 403: str, + 409: str, }, by_alias=True, ) @@ -107,6 +108,7 @@ def get_aggregation_statistic( 204: None, 401: str, 403: str, + 409: str, }, by_alias=True, ) diff --git a/django/interfaces/api/views/sta/datastream.py b/django/interfaces/api/views/sta/datastream.py index 5f87bb7e4..bd6dd66c9 100644 --- a/django/interfaces/api/views/sta/datastream.py +++ b/django/interfaces/api/views/sta/datastream.py @@ -88,6 +88,7 @@ def get_datastream_visualization_bootstrap( 400: str, 401: str, 403: str, + 409: str, }, by_alias=True, ) @@ -238,6 +239,7 @@ def get_datastream_linked_resources( 400: str, 401: str, 403: str, + 409: str, 413: str, }, by_alias=True, @@ -306,6 +308,7 @@ def get_datastream_linked_resource( 401: str, 403: str, 404: str, + 409: str, 413: str, }, by_alias=True, diff --git a/django/interfaces/api/views/sta/datastream_status.py b/django/interfaces/api/views/sta/datastream_status.py index 68284a5a6..fdf922df9 100644 --- a/django/interfaces/api/views/sta/datastream_status.py +++ b/django/interfaces/api/views/sta/datastream_status.py @@ -55,6 +55,7 @@ def get_datastream_statuses( 201: CreatedResponse, 401: str, 403: str, + 409: str, }, by_alias=True, ) @@ -107,6 +108,7 @@ def get_datastream_status( 204: None, 401: str, 403: str, + 409: str, }, by_alias=True, ) diff --git a/django/interfaces/api/views/sta/linked_resource_type.py b/django/interfaces/api/views/sta/linked_resource_type.py index c11d3d2ff..2063b1ddd 100644 --- a/django/interfaces/api/views/sta/linked_resource_type.py +++ b/django/interfaces/api/views/sta/linked_resource_type.py @@ -55,6 +55,7 @@ def get_linked_resource_types( 201: CreatedResponse, 401: str, 403: str, + 409: str, }, by_alias=True, ) @@ -107,6 +108,7 @@ def get_linked_resource_type( 204: None, 401: str, 403: str, + 409: str, }, by_alias=True, ) diff --git a/django/interfaces/api/views/sta/method.py b/django/interfaces/api/views/sta/method.py index 3f190b7e8..129760994 100644 --- a/django/interfaces/api/views/sta/method.py +++ b/django/interfaces/api/views/sta/method.py @@ -57,6 +57,7 @@ def get_methods( 400: str, 401: str, 403: str, + 409: str, }, by_alias=True, ) diff --git a/django/interfaces/api/views/sta/method_type.py b/django/interfaces/api/views/sta/method_type.py index 49af9af1d..b82837729 100644 --- a/django/interfaces/api/views/sta/method_type.py +++ b/django/interfaces/api/views/sta/method_type.py @@ -55,6 +55,7 @@ def get_method_types( 201: CreatedResponse, 401: str, 403: str, + 409: str, }, by_alias=True, ) @@ -107,6 +108,7 @@ def get_method_type( 204: None, 401: str, 403: str, + 409: str, }, by_alias=True, ) diff --git a/django/interfaces/api/views/sta/monitoring_site.py b/django/interfaces/api/views/sta/monitoring_site.py index 10a80bf0f..e897ddaf7 100644 --- a/django/interfaces/api/views/sta/monitoring_site.py +++ b/django/interfaces/api/views/sta/monitoring_site.py @@ -141,6 +141,7 @@ def get_monitoring_site_task_summaries( 201: CreatedResponse, 400: str, 401: str, + 409: str, }, by_alias=True, ) @@ -302,6 +303,7 @@ def get_monitoring_site_linked_resources( 400: str, 401: str, 403: str, + 409: str, 413: str, }, by_alias=True, @@ -370,6 +372,7 @@ def get_monitoring_site_linked_resource( 401: str, 403: str, 404: str, + 409: str, 413: str, }, by_alias=True, diff --git a/django/interfaces/api/views/sta/monitoring_site_type.py b/django/interfaces/api/views/sta/monitoring_site_type.py index c2be54f3c..8bb25e380 100644 --- a/django/interfaces/api/views/sta/monitoring_site_type.py +++ b/django/interfaces/api/views/sta/monitoring_site_type.py @@ -55,6 +55,7 @@ def get_monitoring_site_types( 201: CreatedResponse, 401: str, 403: str, + 409: str, }, by_alias=True, ) @@ -107,6 +108,7 @@ def get_monitoring_site_type( 204: None, 401: str, 403: str, + 409: str, }, by_alias=True, ) diff --git a/django/interfaces/api/views/sta/observation.py b/django/interfaces/api/views/sta/observation.py index 58086f3bf..ebe64b494 100644 --- a/django/interfaces/api/views/sta/observation.py +++ b/django/interfaces/api/views/sta/observation.py @@ -74,6 +74,7 @@ def get_observations( 400: str, 401: str, 403: str, + 409: str, }, by_alias=True, ) @@ -96,7 +97,7 @@ def create_observation( @observation_router.post( "/bulk-create", auth=[session_auth, oidc_data_write_auth, apikey_auth, basic_auth], - response={201: None, 400: str, 403: str, 404: str}, + response={201: None, 400: str, 403: str, 404: str, 409: str}, ) @transaction.atomic def insert_observations( diff --git a/django/interfaces/api/views/sta/observed_property.py b/django/interfaces/api/views/sta/observed_property.py index 82fdece43..616c3bf67 100644 --- a/django/interfaces/api/views/sta/observed_property.py +++ b/django/interfaces/api/views/sta/observed_property.py @@ -57,6 +57,7 @@ def get_observed_properties( 400: str, 401: str, 403: str, + 409: str, }, by_alias=True, ) diff --git a/django/interfaces/api/views/sta/observed_property_type.py b/django/interfaces/api/views/sta/observed_property_type.py index 5881193a5..3a89d8499 100644 --- a/django/interfaces/api/views/sta/observed_property_type.py +++ b/django/interfaces/api/views/sta/observed_property_type.py @@ -55,6 +55,7 @@ def get_observed_property_types( 201: CreatedResponse, 401: str, 403: str, + 409: str, }, by_alias=True, ) @@ -107,6 +108,7 @@ def get_observed_property_type( 204: None, 401: str, 403: str, + 409: str, }, by_alias=True, ) diff --git a/django/interfaces/api/views/sta/processing_level.py b/django/interfaces/api/views/sta/processing_level.py index b2dd468be..3a18a21d0 100644 --- a/django/interfaces/api/views/sta/processing_level.py +++ b/django/interfaces/api/views/sta/processing_level.py @@ -57,6 +57,7 @@ def get_processing_levels( 400: str, 401: str, 403: str, + 409: str, }, by_alias=True, ) diff --git a/django/interfaces/api/views/sta/result_qualifier.py b/django/interfaces/api/views/sta/result_qualifier.py index 8b5ad264f..8c90293a8 100644 --- a/django/interfaces/api/views/sta/result_qualifier.py +++ b/django/interfaces/api/views/sta/result_qualifier.py @@ -58,6 +58,7 @@ def get_result_qualifiers( 201: CreatedResponse, 401: str, 403: str, + 409: str, }, by_alias=True, ) @@ -111,6 +112,7 @@ def get_result_qualifier( 204: None, 401: str, 403: str, + 409: str, }, by_alias=True, ) diff --git a/django/interfaces/api/views/sta/sampled_medium.py b/django/interfaces/api/views/sta/sampled_medium.py index e28334b09..d6aa9aa62 100644 --- a/django/interfaces/api/views/sta/sampled_medium.py +++ b/django/interfaces/api/views/sta/sampled_medium.py @@ -55,6 +55,7 @@ def get_sampled_mediums( 201: CreatedResponse, 401: str, 403: str, + 409: str, }, by_alias=True, ) @@ -107,6 +108,7 @@ def get_sampled_medium( 204: None, 401: str, 403: str, + 409: str, }, by_alias=True, ) diff --git a/django/interfaces/api/views/sta/unit.py b/django/interfaces/api/views/sta/unit.py index d1a9ca7fa..af144f049 100644 --- a/django/interfaces/api/views/sta/unit.py +++ b/django/interfaces/api/views/sta/unit.py @@ -56,6 +56,7 @@ def get_units( 201: CreatedResponse, 400: str, 401: str, + 409: str, }, by_alias=True, ) diff --git a/django/interfaces/api/views/sta/unit_type.py b/django/interfaces/api/views/sta/unit_type.py index 3137b72bf..39cc85fc4 100644 --- a/django/interfaces/api/views/sta/unit_type.py +++ b/django/interfaces/api/views/sta/unit_type.py @@ -55,6 +55,7 @@ def get_unit_types( 201: CreatedResponse, 401: str, 403: str, + 409: str, }, by_alias=True, ) @@ -107,6 +108,7 @@ def get_unit_type( 204: None, 401: str, 403: str, + 409: str, }, by_alias=True, ) diff --git a/django/processing/etl/migrations/0010_v2_0_0.py b/django/processing/etl/migrations/0010_v2_0_0.py index 054db5099..5673e7cf2 100644 --- a/django/processing/etl/migrations/0010_v2_0_0.py +++ b/django/processing/etl/migrations/0010_v2_0_0.py @@ -84,6 +84,7 @@ class Migration(migrations.Migration): fields=("target_datastream",), name="unique_etl_mapping_target_datastream", violation_error_message="This datastream is already mapped to by another task.", + violation_error_code="unique", ), ), migrations.AddConstraint( @@ -92,6 +93,7 @@ class Migration(migrations.Migration): fields=("data_connection", "name", "variable_type"), name="unique_placeholder_variable_name_and_type_per_data_connection", violation_error_message="A placeholder variable with this name and type already exists on this data connection.", + violation_error_code="unique", ), ), migrations.AddField( diff --git a/django/processing/etl/models/data_connection.py b/django/processing/etl/models/data_connection.py index a3a1d5dfc..dcdc23628 100644 --- a/django/processing/etl/models/data_connection.py +++ b/django/processing/etl/models/data_connection.py @@ -139,6 +139,7 @@ class Meta: violation_error_message=( "A placeholder variable with this name and type already exists on this data connection." ), + violation_error_code="unique", ) ] diff --git a/django/processing/etl/models/task.py b/django/processing/etl/models/task.py index dd2e25f07..51a0f4dc9 100644 --- a/django/processing/etl/models/task.py +++ b/django/processing/etl/models/task.py @@ -45,6 +45,7 @@ class Meta: fields=["target_datastream"], name="unique_etl_mapping_target_datastream", violation_error_message="This datastream is already mapped to by another task.", + violation_error_code="unique", ) ] diff --git a/django/processing/etl/tasks.py b/django/processing/etl/tasks.py index 9cabf431c..080d2b859 100644 --- a/django/processing/etl/tasks.py +++ b/django/processing/etl/tasks.py @@ -1,13 +1,23 @@ +import logging + from uuid import UUID from datetime import datetime, timedelta, timezone from celery import shared_task +from django.conf import settings +from django.core.mail import EmailMultiAlternatives +from django.db.models import Prefetch +from django.template.loader import render_to_string +from django.utils.html import strip_tags from pydantic import ValidationError from pydantic.alias_generators import to_camel from interfaces.api.services.sta.datastream import DatastreamAPIService -from processing.etl.models import EtlTask +from processing.etl.models import EtlTask, DataConnectionNotification from processing.etl.loader import HydroServerInternalLoader +from processing.orchestration.models import TaskRun +from processing.orchestration.models.run import TaskRunStatus +from processing.orchestration.services.scheduling import SchedulingService from hydroserverpy.etl import ETLPipeline from hydroserverpy.etl.extractors import HTTPExtractor @@ -17,6 +27,8 @@ from hydroserverpy.etl.user_facing_errors import coerce_known_etl_error +logger = logging.getLogger(__name__) + datastream_service = DatastreamAPIService() @@ -182,3 +194,97 @@ def run_etl_task(self, task_id: str, run_id: str | None = None): raise Exception("Encountered an unexpected ETL error.") from e return result + + +@shared_task(bind=True, name="processing.etl.tasks.send_etl_notification_email", max_retries=3) +def send_etl_notification_email(self, data_connection_id: str, recipients: list[str] | None = None): + """ + Emails a summary of a data connection's ETL task runs since the previous scheduled notification. + Only the given recipients are emailed when retrying failed sends. + """ + + try: + notification = ( + DataConnectionNotification.objects + .select_related("data_connection", "periodic_task__crontab", "periodic_task__interval") + .get(pk=UUID(data_connection_id)) + ) + except DataConnectionNotification.DoesNotExist: + logger.info("Skipping email for DataConnection %s because it has no notification.", data_connection_id) + return {"message": "Data connection has no notification."} + + data_connection = notification.data_connection + window_start = ( + SchedulingService.compute_previous_run_at(notification.periodic_task) + or datetime.now(timezone.utc) - timedelta(days=1) + ) + + tasks = data_connection.etl_tasks.prefetch_related( + Prefetch( + "runs", + queryset=TaskRun.objects.filter(started_at__gte=window_start).order_by("-started_at"), + to_attr="period_runs", + ) + ).order_by("name") + + if recipients is None: + recipients = list(notification.recipients.values_list("email", flat=True)) + + task_summaries = [] + + for task in tasks: + failed_runs = [run for run in task.period_runs if run.status == TaskRunStatus.FAILURE] + task_summaries.append({ + "id": task.id, + "name": task.name, + "run_count": len(task.period_runs), + "failure_count": len(failed_runs), + "last_run_message": (task.period_runs[0].message or "") if task.period_runs else "", + "last_failure_message": (failed_runs[0].message or "") if failed_runs else "", + }) + + if not task_summaries: + logger.info("Skipping email for DataConnection %s because it has no tasks.", data_connection.name) + return {"message": "Data connection has no tasks."} + + if not recipients: + logger.info("Skipping email for DataConnection %s because it has no recipients.", data_connection.name) + return {"message": "Data connection has no notification recipients."} + + context = { + "data_connection_name": data_connection.name, + "data_connection_link": ( + f"{settings.WEB_CLIENT_URL.rstrip('/')}/orchestration/ingestion" + f"?workspace_id={data_connection.workspace_id}&data_connection_id={data_connection.id}" + ), + "tasks": task_summaries, + } + + html_content = render_to_string("orchestration/email/notification_email.html", context) + text_content = strip_tags(html_content) + + failed_recipients = [] + + for recipient in recipients: + try: + msg = EmailMultiAlternatives( + subject=f"Job Orchestration Status: {data_connection.name}", + body=text_content, + from_email=None, + to=[recipient], + ) + msg.attach_alternative(html_content, "text/html") + msg.send(fail_silently=False) + except Exception: + logger.error( + "Failed to send email for DataConnection %s to %s.", data_connection.name, recipient, exc_info=True + ) + failed_recipients.append(recipient) + + if failed_recipients: + raise self.retry( + kwargs={"data_connection_id": data_connection_id, "recipients": failed_recipients}, + countdown=60 * 2 ** self.request.retries, + ) + + return {"message": f"Sent ETL task summary to {len(recipients)} recipient(s)."} diff --git a/django/processing/orchestration/services/scheduling.py b/django/processing/orchestration/services/scheduling.py index e916308a3..bfba60172 100644 --- a/django/processing/orchestration/services/scheduling.py +++ b/django/processing/orchestration/services/scheduling.py @@ -258,3 +258,25 @@ def compute_next_run_at(periodic_task: Optional[PeriodicTask]) -> Optional[datet return now + delta return None + + @staticmethod + def compute_previous_run_at(periodic_task: Optional[PeriodicTask]) -> Optional[datetime]: + """ + Return the scheduled run time preceding the current run of a PeriodicTask, or None if unscheduled. + """ + + if not periodic_task: + return None + + now = timezone.now() + + if periodic_task.crontab: + cron_string = SchedulingService.get_crontab_string(periodic_task) + cron = croniter(cron_string, timezone.make_naive(now, dt_timezone.utc)) + cron.get_prev(datetime) # The run currently executing + return timezone.make_aware(cron.get_prev(datetime), dt_timezone.utc) + + if periodic_task.interval: + return now - timedelta(**{periodic_task.interval.period: periodic_task.interval.every}) + + return None diff --git a/django/processing/products/migrations/0004_v2_0_0.py b/django/processing/products/migrations/0004_v2_0_0.py index f8b6ca8c7..acee89d58 100644 --- a/django/processing/products/migrations/0004_v2_0_0.py +++ b/django/processing/products/migrations/0004_v2_0_0.py @@ -114,6 +114,16 @@ class Migration(migrations.Migration): violation_error_message="A point with this input_value already exists on this rating curve.", ), ), + migrations.AlterConstraint( + model_name="dataproducttransformationinput", + name="unique_data_product_transformation_input_variable", + constraint=models.UniqueConstraint( + condition=models.Q(("variable_name__isnull", False)), + fields=("transformation", "variable_name"), + name="unique_data_product_transformation_input_variable", + violation_error_code="unique", + ), + ), migrations.AddField( model_name="ratingcurve", name="search_vector", diff --git a/django/processing/products/models/transformation.py b/django/processing/products/models/transformation.py index e8bdef80f..8fb5c12f0 100644 --- a/django/processing/products/models/transformation.py +++ b/django/processing/products/models/transformation.py @@ -313,6 +313,7 @@ class Meta: fields=["transformation", "variable_name"], condition=models.Q(variable_name__isnull=False), name="unique_data_product_transformation_input_variable", + violation_error_code="unique", ), ] diff --git a/django/processing/quality/migrations/0003_v2_0_0.py b/django/processing/quality/migrations/0003_v2_0_0.py new file mode 100644 index 000000000..5a304534d --- /dev/null +++ b/django/processing/quality/migrations/0003_v2_0_0.py @@ -0,0 +1,28 @@ +""" +HydroServer v1.12 -> v2.0.0 schema migration for the quality app. + +This migration is forward-only. Back up the database before applying it. See the +"Migrating from HydroServer v1 to v2" deployment guide. +""" + +from django.db import migrations, models + + +class Migration(migrations.Migration): + + dependencies = [ + ("quality", "0002_qcoperation_created_by"), + ] + + operations = [ + migrations.AlterConstraint( + model_name="qcsession", + name="unique_in_progress_session_per_history", + constraint=models.UniqueConstraint( + condition=models.Q(("status", "in_progress")), + fields=("history",), + name="unique_in_progress_session_per_history", + violation_error_code="unique", + ), + ), + ] diff --git a/django/processing/quality/models/session.py b/django/processing/quality/models/session.py index ee3da7d9d..501a15534 100644 --- a/django/processing/quality/models/session.py +++ b/django/processing/quality/models/session.py @@ -36,6 +36,7 @@ class Meta: fields=["history"], condition=Q(status=SessionStatus.IN_PROGRESS), name="unique_in_progress_session_per_history", + violation_error_code="unique", ) ] diff --git a/django/templates/orchestration/email/notification_email.html b/django/templates/orchestration/email/notification_email.html index 841416929..5538d6289 100644 --- a/django/templates/orchestration/email/notification_email.html +++ b/django/templates/orchestration/email/notification_email.html @@ -1,7 +1,7 @@ - Task Status Notification: {{ notification_name }} + ETL Task Summary: {{ data_connection_name }}

Hello,

-

Here is a summary of tasks for the {{ notification_name }} notification:

+

Here is a summary of ETL task runs since the last notification for the {{ data_connection_name }} data connection:

+ +

+ View in HydroServer +

- + + + - {% for task in task_summaries %} + {% for task in tasks %} + + {% endfor %}
TaskTask IDTask Name Run Count Failure Count Last Run MessageLast Failure Message
{{ task.id }} {{ task.name }} {{ task.run_count }} {{ task.failure_count }} {{ task.last_run_message }}{{ task.last_failure_message }}
-

This is an automated message. Do not reply to this email.

- + \ No newline at end of file diff --git a/django/tests/interfaces/api/test_aggregation_statistic_views.py b/django/tests/interfaces/api/test_aggregation_statistic_views.py index 52cf58f53..bed0c02a6 100644 --- a/django/tests/interfaces/api/test_aggregation_statistic_views.py +++ b/django/tests/interfaces/api/test_aggregation_statistic_views.py @@ -138,7 +138,7 @@ def test_create_aggregation_statistic_returns_403_for_non_superuser(client): assert response.status_code == 403 -def test_create_aggregation_statistic_returns_400_for_duplicate_name(client): +def test_create_aggregation_statistic_returns_409_for_duplicate_name(client): AggregationStatisticFactory(name="Duplicate") superuser = UserFactory(is_superuser=True) client.force_login(superuser) @@ -149,7 +149,7 @@ def test_create_aggregation_statistic_returns_400_for_duplicate_name(client): content_type="application/json", ) - assert response.status_code == 400 + assert response.status_code == 409 # --- get_aggregation_statistic ---------------------------------------------------------------- diff --git a/django/tests/interfaces/api/test_collaborator_views.py b/django/tests/interfaces/api/test_collaborator_views.py index 7c38144c4..276ff16b1 100644 --- a/django/tests/interfaces/api/test_collaborator_views.py +++ b/django/tests/interfaces/api/test_collaborator_views.py @@ -247,7 +247,7 @@ def test_add_collaborator_succeeds_for_service_account_on_external_workspace(cli assert service_account.email in [c["serviceAccountEmail"] for c in collaborators] -def test_add_collaborator_returns_400_when_service_account_already_collaborates(client): +def test_add_collaborator_returns_409_when_service_account_already_collaborates(client): owner = UserFactory() workspace = WorkspaceFactory(owner=owner) role = RoleFactory(workspace=workspace) @@ -262,7 +262,7 @@ def test_add_collaborator_returns_400_when_service_account_already_collaborates( content_type="application/json", ) - assert response.status_code == 400 + assert response.status_code == 409 # --- edit_collaborator_role --------------------------------------------------- diff --git a/django/tests/interfaces/api/test_datastream_status_views.py b/django/tests/interfaces/api/test_datastream_status_views.py index 8b4ce8cd9..4b91eaceb 100644 --- a/django/tests/interfaces/api/test_datastream_status_views.py +++ b/django/tests/interfaces/api/test_datastream_status_views.py @@ -138,7 +138,7 @@ def test_create_datastream_status_returns_403_for_non_superuser(client): assert response.status_code == 403 -def test_create_datastream_status_returns_400_for_duplicate_name(client): +def test_create_datastream_status_returns_409_for_duplicate_name(client): DatastreamStatusFactory(name="Duplicate") superuser = UserFactory(is_superuser=True) client.force_login(superuser) @@ -149,7 +149,7 @@ def test_create_datastream_status_returns_400_for_duplicate_name(client): content_type="application/json", ) - assert response.status_code == 400 + assert response.status_code == 409 # --- get_datastream_status ---------------------------------------------------------------- diff --git a/django/tests/interfaces/api/test_datastream_views.py b/django/tests/interfaces/api/test_datastream_views.py index df3bddd59..548c34cd1 100644 --- a/django/tests/interfaces/api/test_datastream_views.py +++ b/django/tests/interfaces/api/test_datastream_views.py @@ -862,7 +862,7 @@ def test_add_datastream_linked_resource_succeeds_with_link(client): assert linked_resources[0]["name"] == "Datastream Report" -def test_add_datastream_linked_resource_returns_400_for_duplicate_name(client): +def test_add_datastream_linked_resource_returns_409_for_duplicate_name(client): owner = UserFactory() workspace = WorkspaceFactory(owner=owner) datastream = _make_datastream(workspace) @@ -877,7 +877,7 @@ def test_add_datastream_linked_resource_returns_400_for_duplicate_name(client): data={"name": "Datastream Report", "type": "Report", "link": "https://example.com/b.pdf"}, ) - assert response.status_code == 400 + assert response.status_code == 409 # --- include / properties --------------------------------------------------------- diff --git a/django/tests/interfaces/api/test_etl_mapping_views.py b/django/tests/interfaces/api/test_etl_mapping_views.py index c812cac06..e6e961929 100644 --- a/django/tests/interfaces/api/test_etl_mapping_views.py +++ b/django/tests/interfaces/api/test_etl_mapping_views.py @@ -194,7 +194,7 @@ def test_create_etl_mapping_returns_404_for_nonexistent_task(client): assert response.status_code == 404 -def test_create_etl_mapping_returns_400_for_duplicate_target_datastream(client): +def test_create_etl_mapping_returns_409_for_duplicate_target_datastream(client): owner = UserFactory() workspace = WorkspaceFactory(owner=owner) task = _make_task(workspace) @@ -211,7 +211,7 @@ def test_create_etl_mapping_returns_400_for_duplicate_target_datastream(client): content_type="application/json", ) - assert response.status_code == 400 + assert response.status_code == 409 # --- get_etl_mapping -------------------------------------------------------------- diff --git a/django/tests/interfaces/api/test_handlers.py b/django/tests/interfaces/api/test_handlers.py index cd5c4fbbc..acd917241 100644 --- a/django/tests/interfaces/api/test_handlers.py +++ b/django/tests/interfaces/api/test_handlers.py @@ -1,8 +1,9 @@ import pytest +from django.core.exceptions import NON_FIELD_ERRORS, ValidationError as DjangoValidationError from ninja.errors import ValidationError as NinjaValidationError from pydantic import validate_call -from interfaces.api.http.handlers import ninja_validation_error_handler +from interfaces.api.http.handlers import ninja_validation_error_handler, validation_error_handler class _FakeApi: @@ -32,3 +33,25 @@ def test_validation_error_handler_returns_400_for_ninja_validation_error(): assert response["status"] == 400 assert response["data"]["message"] == "field required" + + +def test_validation_error_handler_returns_409_for_unique_errors(): + exc = DjangoValidationError({ + "id": [DjangoValidationError("Unit with this Id already exists.", code="unique")], + NON_FIELD_ERRORS: [DjangoValidationError("Duplicate name.", code="unique_together")], + }) + + response = validation_error_handler(request=None, exc=exc, api=_FakeApi()) + + assert response["status"] == 409 + + +def test_validation_error_handler_returns_400_when_unique_errors_mixed_with_others(): + exc = DjangoValidationError({ + "id": [DjangoValidationError("Unit with this Id already exists.", code="unique")], + "name": [DjangoValidationError("This field cannot be blank.", code="blank")], + }) + + response = validation_error_handler(request=None, exc=exc, api=_FakeApi()) + + assert response["status"] == 400 diff --git a/django/tests/interfaces/api/test_linked_resource_type_views.py b/django/tests/interfaces/api/test_linked_resource_type_views.py index ac1aa864d..feea0f911 100644 --- a/django/tests/interfaces/api/test_linked_resource_type_views.py +++ b/django/tests/interfaces/api/test_linked_resource_type_views.py @@ -138,7 +138,7 @@ def test_create_linked_resource_type_returns_403_for_non_superuser(client): assert response.status_code == 403 -def test_create_linked_resource_type_returns_400_for_duplicate_name(client): +def test_create_linked_resource_type_returns_409_for_duplicate_name(client): LinkedResourceTypeFactory(name="Duplicate") superuser = UserFactory(is_superuser=True) client.force_login(superuser) @@ -149,7 +149,7 @@ def test_create_linked_resource_type_returns_400_for_duplicate_name(client): content_type="application/json", ) - assert response.status_code == 400 + assert response.status_code == 409 # --- get_linked_resource_type ---------------------------------------------------------------- diff --git a/django/tests/interfaces/api/test_method_type_views.py b/django/tests/interfaces/api/test_method_type_views.py index 1afb73d5a..ad117e4bc 100644 --- a/django/tests/interfaces/api/test_method_type_views.py +++ b/django/tests/interfaces/api/test_method_type_views.py @@ -138,7 +138,7 @@ def test_create_method_type_returns_403_for_non_superuser(client): assert response.status_code == 403 -def test_create_method_type_returns_400_for_duplicate_name(client): +def test_create_method_type_returns_409_for_duplicate_name(client): MethodTypeFactory(name="Duplicate") superuser = UserFactory(is_superuser=True) client.force_login(superuser) @@ -149,7 +149,7 @@ def test_create_method_type_returns_400_for_duplicate_name(client): content_type="application/json", ) - assert response.status_code == 400 + assert response.status_code == 409 # --- get_method_type ---------------------------------------------------------------- diff --git a/django/tests/interfaces/api/test_monitoring_rule_views.py b/django/tests/interfaces/api/test_monitoring_rule_views.py index 72d66e36f..77ead679c 100644 --- a/django/tests/interfaces/api/test_monitoring_rule_views.py +++ b/django/tests/interfaces/api/test_monitoring_rule_views.py @@ -471,7 +471,7 @@ def test_create_monitoring_rule_returns_400_when_range_rule_missing_bounds(clien assert response.status_code == 400 -def test_create_monitoring_rule_returns_400_for_duplicate_rule_type_on_datastream(client): +def test_create_monitoring_rule_returns_409_for_duplicate_rule_type_on_datastream(client): owner = UserFactory() workspace = WorkspaceFactory(owner=owner) task, datastream = _make_task_with_datastream(workspace) @@ -484,7 +484,7 @@ def test_create_monitoring_rule_returns_400_for_duplicate_rule_type_on_datastrea content_type="application/json", ) - assert response.status_code == 400 + assert response.status_code == 409 # --- get_monitoring_rule ----------------------------------------------------------------- diff --git a/django/tests/interfaces/api/test_monitoring_site_type_views.py b/django/tests/interfaces/api/test_monitoring_site_type_views.py index dffbbd4fb..fdd201c28 100644 --- a/django/tests/interfaces/api/test_monitoring_site_type_views.py +++ b/django/tests/interfaces/api/test_monitoring_site_type_views.py @@ -138,7 +138,7 @@ def test_create_monitoring_site_type_returns_403_for_non_superuser(client): assert response.status_code == 403 -def test_create_monitoring_site_type_returns_400_for_duplicate_name(client): +def test_create_monitoring_site_type_returns_409_for_duplicate_name(client): MonitoringSiteTypeFactory(name="Duplicate") superuser = UserFactory(is_superuser=True) client.force_login(superuser) @@ -149,7 +149,7 @@ def test_create_monitoring_site_type_returns_400_for_duplicate_name(client): content_type="application/json", ) - assert response.status_code == 400 + assert response.status_code == 409 # --- get_monitoring_site_type ---------------------------------------------------------------- diff --git a/django/tests/interfaces/api/test_monitoring_site_views.py b/django/tests/interfaces/api/test_monitoring_site_views.py index d56bbccf1..91ffbe1b6 100644 --- a/django/tests/interfaces/api/test_monitoring_site_views.py +++ b/django/tests/interfaces/api/test_monitoring_site_views.py @@ -1071,7 +1071,7 @@ def test_add_monitoring_site_linked_resource_succeeds_with_link(client): assert linked_resources[0]["name"] == "Site Report" -def test_add_monitoring_site_linked_resource_returns_400_for_duplicate_name(client): +def test_add_monitoring_site_linked_resource_returns_409_for_duplicate_name(client): owner = UserFactory() workspace = WorkspaceFactory(owner=owner) monitoring_site = MonitoringSiteFactory(workspace=workspace) @@ -1086,7 +1086,7 @@ def test_add_monitoring_site_linked_resource_returns_400_for_duplicate_name(clie data={"name": "Site Report", "type": "Report", "link": "https://example.com/b.pdf"}, ) - assert response.status_code == 400 + assert response.status_code == 409 def test_add_monitoring_site_linked_resource_returns_400_without_file_or_link(client): diff --git a/django/tests/interfaces/api/test_observation_views.py b/django/tests/interfaces/api/test_observation_views.py index 1f20d950a..c2c26be20 100644 --- a/django/tests/interfaces/api/test_observation_views.py +++ b/django/tests/interfaces/api/test_observation_views.py @@ -532,7 +532,7 @@ def test_create_observation_succeeds_with_datastream_view_only_and_observation_c assert response.status_code == 201 -def test_create_observation_returns_400_for_duplicate_phenomenon_time(client): +def test_create_observation_returns_409_for_duplicate_phenomenon_time(client): owner = UserFactory() workspace = WorkspaceFactory(owner=owner) datastream = _make_datastream(workspace) @@ -548,7 +548,7 @@ def test_create_observation_returns_400_for_duplicate_phenomenon_time(client): content_type="application/json", ) - assert response.status_code == 400 + assert response.status_code == 409 def test_create_observation_returns_400_for_invalid_result_qualifier_code(client): diff --git a/django/tests/interfaces/api/test_observed_property_type_views.py b/django/tests/interfaces/api/test_observed_property_type_views.py index fb5f6ba4a..229a24aa0 100644 --- a/django/tests/interfaces/api/test_observed_property_type_views.py +++ b/django/tests/interfaces/api/test_observed_property_type_views.py @@ -138,7 +138,7 @@ def test_create_observed_property_type_returns_403_for_non_superuser(client): assert response.status_code == 403 -def test_create_observed_property_type_returns_400_for_duplicate_name(client): +def test_create_observed_property_type_returns_409_for_duplicate_name(client): ObservedPropertyTypeFactory(name="Duplicate") superuser = UserFactory(is_superuser=True) client.force_login(superuser) @@ -149,7 +149,7 @@ def test_create_observed_property_type_returns_400_for_duplicate_name(client): content_type="application/json", ) - assert response.status_code == 400 + assert response.status_code == 409 # --- get_observed_property_type ---------------------------------------------------------------- diff --git a/django/tests/interfaces/api/test_qc_history_views.py b/django/tests/interfaces/api/test_qc_history_views.py index 50e64ebf0..ac856b310 100644 --- a/django/tests/interfaces/api/test_qc_history_views.py +++ b/django/tests/interfaces/api/test_qc_history_views.py @@ -155,7 +155,7 @@ def test_create_qc_history_returns_400_when_processing_levels_match(client): assert response.status_code == 400 -def test_create_qc_history_returns_400_when_managed_datastream_already_has_history(client): +def test_create_qc_history_returns_409_when_managed_datastream_already_has_history(client): owner = UserFactory() workspace = WorkspaceFactory(owner=owner) history = _make_history(workspace) @@ -171,7 +171,7 @@ def test_create_qc_history_returns_400_when_managed_datastream_already_has_histo content_type="application/json", ) - assert response.status_code == 400 + assert response.status_code == 409 # --- get_qc_history -------------------------------------------------------------------- diff --git a/django/tests/interfaces/api/test_qc_session_views.py b/django/tests/interfaces/api/test_qc_session_views.py index baa2979c0..238f0fc3e 100644 --- a/django/tests/interfaces/api/test_qc_session_views.py +++ b/django/tests/interfaces/api/test_qc_session_views.py @@ -155,7 +155,7 @@ def test_create_qc_session_returns_400_when_end_before_start(client): assert response.status_code == 400 -def test_create_qc_session_returns_400_when_in_progress_session_already_exists(client): +def test_create_qc_session_returns_409_when_in_progress_session_already_exists(client): owner = UserFactory() workspace = WorkspaceFactory(owner=owner) history = _make_history(workspace) @@ -168,7 +168,7 @@ def test_create_qc_session_returns_400_when_in_progress_session_already_exists(c content_type="application/json", ) - assert response.status_code == 400 + assert response.status_code == 409 # --- get_qc_session -------------------------------------------------------------------- diff --git a/django/tests/interfaces/api/test_result_qualifier_views.py b/django/tests/interfaces/api/test_result_qualifier_views.py index 8b5684480..30f0f922d 100644 --- a/django/tests/interfaces/api/test_result_qualifier_views.py +++ b/django/tests/interfaces/api/test_result_qualifier_views.py @@ -220,7 +220,7 @@ def test_create_result_qualifier_returns_403_without_create_permission(client): assert response.status_code == 403 -def test_create_result_qualifier_returns_400_for_duplicate_name_in_workspace(client): +def test_create_result_qualifier_returns_409_for_duplicate_name_in_workspace(client): owner = UserFactory() workspace = WorkspaceFactory(owner=owner) ResultQualifierFactory(workspace=workspace, name="Duplicate") @@ -232,7 +232,7 @@ def test_create_result_qualifier_returns_400_for_duplicate_name_in_workspace(cli content_type="application/json", ) - assert response.status_code == 400 + assert response.status_code == 409 # --- get_result_qualifier ---------------------------------------------------------------- diff --git a/django/tests/interfaces/api/test_sampled_medium_views.py b/django/tests/interfaces/api/test_sampled_medium_views.py index 7c0bbe580..153796b64 100644 --- a/django/tests/interfaces/api/test_sampled_medium_views.py +++ b/django/tests/interfaces/api/test_sampled_medium_views.py @@ -138,7 +138,7 @@ def test_create_sampled_medium_returns_403_for_non_superuser(client): assert response.status_code == 403 -def test_create_sampled_medium_returns_400_for_duplicate_name(client): +def test_create_sampled_medium_returns_409_for_duplicate_name(client): SampledMediumFactory(name="Duplicate") superuser = UserFactory(is_superuser=True) client.force_login(superuser) @@ -149,7 +149,7 @@ def test_create_sampled_medium_returns_400_for_duplicate_name(client): content_type="application/json", ) - assert response.status_code == 400 + assert response.status_code == 409 # --- get_sampled_medium ---------------------------------------------------------------- diff --git a/django/tests/interfaces/api/test_unit_type_views.py b/django/tests/interfaces/api/test_unit_type_views.py index b668f96f1..8a5325049 100644 --- a/django/tests/interfaces/api/test_unit_type_views.py +++ b/django/tests/interfaces/api/test_unit_type_views.py @@ -138,7 +138,7 @@ def test_create_unit_type_returns_403_for_non_superuser(client): assert response.status_code == 403 -def test_create_unit_type_returns_400_for_duplicate_name(client): +def test_create_unit_type_returns_409_for_duplicate_name(client): UnitTypeFactory(name="Duplicate") superuser = UserFactory(is_superuser=True) client.force_login(superuser) @@ -149,7 +149,7 @@ def test_create_unit_type_returns_400_for_duplicate_name(client): content_type="application/json", ) - assert response.status_code == 400 + assert response.status_code == 409 # --- get_unit_type ---------------------------------------------------------------- diff --git a/django/tests/interfaces/api/test_workspace_views.py b/django/tests/interfaces/api/test_workspace_views.py index 6b33f8efa..e225f0d26 100644 --- a/django/tests/interfaces/api/test_workspace_views.py +++ b/django/tests/interfaces/api/test_workspace_views.py @@ -224,7 +224,7 @@ def test_create_workspace_returns_401_when_unauthenticated(client): assert response.status_code == 401 -def test_create_workspace_returns_400_for_duplicate_name_owned_by_user(client): +def test_create_workspace_returns_409_for_duplicate_name_owned_by_user(client): owner = UserFactory() WorkspaceFactory(owner=owner, name="Existing Workspace") client.force_login(owner) @@ -235,7 +235,7 @@ def test_create_workspace_returns_400_for_duplicate_name_owned_by_user(client): content_type="application/json", ) - assert response.status_code == 400 + assert response.status_code == 409 def test_create_workspace_returns_400_when_owner_at_workspace_limit(client): diff --git a/django/tests/processing/etl/test_tasks.py b/django/tests/processing/etl/test_tasks.py index 6c16b5474..d0e8d71d4 100644 --- a/django/tests/processing/etl/test_tasks.py +++ b/django/tests/processing/etl/test_tasks.py @@ -1,8 +1,21 @@ +import json import uuid +from datetime import timedelta +from smtplib import SMTPException +from unittest import mock + import pytest -from processing.etl.tasks import run_etl_task +from celery.exceptions import Retry +from django.core.mail import EmailMultiAlternatives +from django.utils import timezone as django_tz + +from hydroserver.celery import app as celery_app +from interfaces.api.services.etl.data_connection import DataConnectionAPIService +from processing.etl.tasks import run_etl_task, send_etl_notification_email +from processing.orchestration.models import TaskRun +from tests.processing.etl.factories import DataConnectionFactory, EtlTaskFactory pytestmark = pytest.mark.django_db @@ -10,3 +23,65 @@ def test_run_etl_task_raises_for_nonexistent_task(): with pytest.raises(Exception, match="Encountered an unexpected ETL error."): run_etl_task(str(uuid.uuid4())) + + +def _make_notification(): + data_connection = DataConnectionFactory() + task = EtlTaskFactory(data_connection=data_connection) + notification = DataConnectionAPIService().apply_notification( + data_connection=data_connection, + recipient_emails=["a@example.com", "b@example.com"], + crontab="0 9 * * *", + ) + return notification, task + + +def test_notification_schedule_runs_on_worker_and_emails_each_recipient(mailoutbox): + notification, task = _make_notification() + TaskRun.objects.create(task=task, status="FAILURE", message="Source returned 500.") + TaskRun.objects.create(task=task, status="SUCCESS", message="Loaded 10 observations.") + TaskRun.objects.create( + task=task, status="FAILURE", message="Old failure.", started_at=django_tz.now() - timedelta(days=3) + ) + + # Dispatch the stored schedule the way a worker would. + periodic_task = notification.periodic_task + celery_app.loader.import_default_modules() + result = celery_app.tasks[periodic_task.task].apply(kwargs=json.loads(periodic_task.kwargs)) + + assert result.successful() + assert sorted(m.to for m in mailoutbox) == [["a@example.com"], ["b@example.com"]] + html = mailoutbox[0].alternatives[0][0] + assert task.name in html + assert "Loaded 10 observations." in html + assert "Source returned 500." in html + assert "Old failure." not in html + data_connection = task.data_connection + link = ( + f"/orchestration/ingestion?workspace_id={data_connection.workspace_id}" + f"&data_connection_id={data_connection.id}" + ) + assert html.count(link) == 1 + + +def test_notification_retries_only_failed_recipients(mailoutbox): + notification, _ = _make_notification() + real_send = EmailMultiAlternatives.send + + def send(msg, *args, **kwargs): + if msg.to == ["b@example.com"]: + raise SMTPException("Mailbox unavailable.") + return real_send(msg, *args, **kwargs) + + with ( + mock.patch.object(EmailMultiAlternatives, "send", send), + mock.patch.object(send_etl_notification_email, "retry", return_value=Retry()) as retry, + pytest.raises(Retry), + ): + send_etl_notification_email(str(notification.pk)) + + assert [m.to for m in mailoutbox] == [["a@example.com"]] + assert retry.call_args.kwargs["kwargs"] == { + "data_connection_id": str(notification.pk), + "recipients": ["b@example.com"], + } diff --git a/django/tests/processing/orchestration/services/test_scheduling.py b/django/tests/processing/orchestration/services/test_scheduling.py index 8bd81bad2..10b408c8e 100644 --- a/django/tests/processing/orchestration/services/test_scheduling.py +++ b/django/tests/processing/orchestration/services/test_scheduling.py @@ -1,3 +1,6 @@ +from datetime import datetime, timedelta, timezone as dt_timezone +from unittest import mock + import pytest from django.core.exceptions import ValidationError from django_celery_beat.models import PeriodicTask @@ -121,3 +124,33 @@ def test_apply_schedule_switches_between_crontab_and_interval(initial, patch, cr updated.refresh_from_db() assert service.get_crontab_string(updated) == crontab assert (updated.interval.every if updated.interval else None) == every + + +def test_compute_previous_run_at_returns_run_before_current_crontab_run(): + periodic_task = SchedulingService().apply_schedule( + periodic_task=None, + crontab="0 9 * * *", + celery_task_name="processing.etl.tasks.send_etl_notification_email", + periodic_task_name="test-previous-crontab", + ) + + with mock.patch("django.utils.timezone.now", return_value=datetime(2026, 1, 2, 9, 0, 5, tzinfo=dt_timezone.utc)): + previous = SchedulingService.compute_previous_run_at(periodic_task) + + assert previous == datetime(2026, 1, 1, 9, 0, tzinfo=dt_timezone.utc) + + +def test_compute_previous_run_at_subtracts_interval(): + periodic_task = SchedulingService().apply_schedule( + periodic_task=None, + interval=6, + interval_period="hours", + celery_task_name="processing.etl.tasks.send_etl_notification_email", + periodic_task_name="test-previous-interval", + ) + now = datetime(2026, 1, 2, 9, 0, tzinfo=dt_timezone.utc) + + with mock.patch("django.utils.timezone.now", return_value=now): + previous = SchedulingService.compute_previous_run_at(periodic_task) + + assert previous == now - timedelta(hours=6) diff --git a/django/tests/processing/orchestration/test_tasks.py b/django/tests/processing/orchestration/test_tasks.py new file mode 100644 index 000000000..5b2119554 --- /dev/null +++ b/django/tests/processing/orchestration/test_tasks.py @@ -0,0 +1,15 @@ +from hydroserver.celery import app as celery_app +from interfaces.api.services.etl import data_connection, task as etl_task +from interfaces.api.services.monitoring import task as monitoring_task +from interfaces.api.services.products import task as products_task + + +def test_scheduled_celery_task_names_are_registered(): + celery_app.loader.import_default_modules() + + assert { + etl_task.CELERY_TASK_NAME, + data_connection.ETL_NOTIFICATION_CELERY_TASK, + monitoring_task.CELERY_TASK_NAME, + products_task.CELERY_TASK_NAME, + } <= set(celery_app.tasks) diff --git a/packages/hydroserver-ts/src/generated/ogc.types.d.ts b/packages/hydroserver-ts/src/generated/ogc.types.d.ts index a34ea8895..b81d50e52 100644 --- a/packages/hydroserver-ts/src/generated/ogc.types.d.ts +++ b/packages/hydroserver-ts/src/generated/ogc.types.d.ts @@ -8156,6 +8156,15 @@ export interface operations { "application/json": string; }; }; + /** @description Conflict */ + 409: { + headers: { + [name: string]: unknown; + }; + content: { + "application/json": string; + }; + }; }; }; interfaces_api_views_sta_aggregation_statistic_get_aggregation_statistic: { @@ -8291,6 +8300,15 @@ export interface operations { "application/json": string; }; }; + /** @description Conflict */ + 409: { + headers: { + [name: string]: unknown; + }; + content: { + "application/json": string; + }; + }; }; }; interfaces_api_views_products_rating_curve_get_rating_curves: { @@ -8403,6 +8421,15 @@ export interface operations { "application/json": string; }; }; + /** @description Conflict */ + 409: { + headers: { + [name: string]: unknown; + }; + content: { + "application/json": string; + }; + }; }; }; interfaces_api_views_products_rating_curve_get_rating_curve: { @@ -8567,6 +8594,15 @@ export interface operations { "application/json": string; }; }; + /** @description Conflict */ + 409: { + headers: { + [name: string]: unknown; + }; + content: { + "application/json": string; + }; + }; }; }; interfaces_api_views_products_task_get_data_product_tasks: { @@ -9156,6 +9192,15 @@ export interface operations { "application/json": string; }; }; + /** @description Conflict */ + 409: { + headers: { + [name: string]: unknown; + }; + content: { + "application/json": string; + }; + }; }; }; interfaces_api_views_products_transformation_get_data_product_transformation: { @@ -9320,6 +9365,15 @@ export interface operations { "application/json": string; }; }; + /** @description Conflict */ + 409: { + headers: { + [name: string]: unknown; + }; + content: { + "application/json": string; + }; + }; }; }; interfaces_api_views_sta_datastream_status_get_datastream_statuses: { @@ -9419,6 +9473,15 @@ export interface operations { "application/json": string; }; }; + /** @description Conflict */ + 409: { + headers: { + [name: string]: unknown; + }; + content: { + "application/json": string; + }; + }; }; }; interfaces_api_views_sta_datastream_status_get_datastream_status: { @@ -9554,6 +9617,15 @@ export interface operations { "application/json": string; }; }; + /** @description Conflict */ + 409: { + headers: { + [name: string]: unknown; + }; + content: { + "application/json": string; + }; + }; }; }; interfaces_api_views_sta_datastream_get_datastreams: { @@ -9684,6 +9756,15 @@ export interface operations { "application/json": string; }; }; + /** @description Conflict */ + 409: { + headers: { + [name: string]: unknown; + }; + content: { + "application/json": string; + }; + }; }; }; interfaces_api_views_sta_datastream_get_datastream: { @@ -9973,6 +10054,15 @@ export interface operations { "application/json": string; }; }; + /** @description Conflict */ + 409: { + headers: { + [name: string]: unknown; + }; + content: { + "application/json": string; + }; + }; /** @description Content Too Large */ 413: { headers: { @@ -10152,6 +10242,15 @@ export interface operations { "application/json": string; }; }; + /** @description Conflict */ + 409: { + headers: { + [name: string]: unknown; + }; + content: { + "application/json": string; + }; + }; /** @description Content Too Large */ 413: { headers: { @@ -10494,6 +10593,15 @@ export interface operations { "application/json": string; }; }; + /** @description Conflict */ + 409: { + headers: { + [name: string]: unknown; + }; + content: { + "application/json": string; + }; + }; }; }; interfaces_api_views_etl_mapping_get_etl_mappings: { @@ -10626,6 +10734,15 @@ export interface operations { "application/json": string; }; }; + /** @description Conflict */ + 409: { + headers: { + [name: string]: unknown; + }; + content: { + "application/json": string; + }; + }; }; }; interfaces_api_views_etl_mapping_get_etl_mapping: { @@ -10790,6 +10907,15 @@ export interface operations { "application/json": string; }; }; + /** @description Conflict */ + 409: { + headers: { + [name: string]: unknown; + }; + content: { + "application/json": string; + }; + }; }; }; interfaces_api_views_etl_task_get_etl_tasks: { @@ -11346,6 +11472,15 @@ export interface operations { "application/json": string; }; }; + /** @description Conflict */ + 409: { + headers: { + [name: string]: unknown; + }; + content: { + "application/json": string; + }; + }; }; }; interfaces_api_views_sta_linked_resource_type_get_linked_resource_type: { @@ -11481,6 +11616,15 @@ export interface operations { "application/json": string; }; }; + /** @description Conflict */ + 409: { + headers: { + [name: string]: unknown; + }; + content: { + "application/json": string; + }; + }; }; }; interfaces_api_views_sta_method_type_get_method_types: { @@ -11580,6 +11724,15 @@ export interface operations { "application/json": string; }; }; + /** @description Conflict */ + 409: { + headers: { + [name: string]: unknown; + }; + content: { + "application/json": string; + }; + }; }; }; interfaces_api_views_sta_method_type_get_method_type: { @@ -11715,6 +11868,15 @@ export interface operations { "application/json": string; }; }; + /** @description Conflict */ + 409: { + headers: { + [name: string]: unknown; + }; + content: { + "application/json": string; + }; + }; }; }; interfaces_api_views_sta_method_get_methods: { @@ -11835,6 +11997,15 @@ export interface operations { "application/json": string; }; }; + /** @description Conflict */ + 409: { + headers: { + [name: string]: unknown; + }; + content: { + "application/json": string; + }; + }; }; }; interfaces_api_views_sta_method_get_method: { @@ -12122,6 +12293,15 @@ export interface operations { "application/json": string; }; }; + /** @description Conflict */ + 409: { + headers: { + [name: string]: unknown; + }; + content: { + "application/json": string; + }; + }; }; }; interfaces_api_views_monitoring_rule_get_monitoring_rule: { @@ -12286,6 +12466,15 @@ export interface operations { "application/json": string; }; }; + /** @description Conflict */ + 409: { + headers: { + [name: string]: unknown; + }; + content: { + "application/json": string; + }; + }; }; }; interfaces_api_views_sta_monitoring_site_type_get_monitoring_site_types: { @@ -12385,6 +12574,15 @@ export interface operations { "application/json": string; }; }; + /** @description Conflict */ + 409: { + headers: { + [name: string]: unknown; + }; + content: { + "application/json": string; + }; + }; }; }; interfaces_api_views_sta_monitoring_site_type_get_monitoring_site_type: { @@ -12520,6 +12718,15 @@ export interface operations { "application/json": string; }; }; + /** @description Conflict */ + 409: { + headers: { + [name: string]: unknown; + }; + content: { + "application/json": string; + }; + }; }; }; interfaces_api_views_sta_monitoring_site_get_monitoring_sites: { @@ -12625,6 +12832,15 @@ export interface operations { "application/json": string; }; }; + /** @description Conflict */ + 409: { + headers: { + [name: string]: unknown; + }; + content: { + "application/json": string; + }; + }; }; }; interfaces_api_views_sta_monitoring_site_get_monitoring_site: { @@ -12876,6 +13092,15 @@ export interface operations { "application/json": string; }; }; + /** @description Conflict */ + 409: { + headers: { + [name: string]: unknown; + }; + content: { + "application/json": string; + }; + }; /** @description Content Too Large */ 413: { headers: { @@ -13055,6 +13280,15 @@ export interface operations { "application/json": string; }; }; + /** @description Conflict */ + 409: { + headers: { + [name: string]: unknown; + }; + content: { + "application/json": string; + }; + }; /** @description Content Too Large */ 413: { headers: { @@ -13729,6 +13963,15 @@ export interface operations { "application/json": string; }; }; + /** @description Conflict */ + 409: { + headers: { + [name: string]: unknown; + }; + content: { + "application/json": string; + }; + }; }; }; interfaces_api_views_sta_observation_delete_observations: { @@ -13891,6 +14134,15 @@ export interface operations { "application/json": string; }; }; + /** @description Conflict */ + 409: { + headers: { + [name: string]: unknown; + }; + content: { + "application/json": string; + }; + }; }; }; interfaces_api_views_sta_observation_get_observation: { @@ -14102,6 +14354,15 @@ export interface operations { "application/json": string; }; }; + /** @description Conflict */ + 409: { + headers: { + [name: string]: unknown; + }; + content: { + "application/json": string; + }; + }; }; }; interfaces_api_views_sta_observed_property_get_observed_property: { @@ -14356,6 +14617,15 @@ export interface operations { "application/json": string; }; }; + /** @description Conflict */ + 409: { + headers: { + [name: string]: unknown; + }; + content: { + "application/json": string; + }; + }; }; }; interfaces_api_views_sta_observed_property_type_get_observed_property_type: { @@ -14491,6 +14761,15 @@ export interface operations { "application/json": string; }; }; + /** @description Conflict */ + 409: { + headers: { + [name: string]: unknown; + }; + content: { + "application/json": string; + }; + }; }; }; interfaces_api_views_sta_processing_level_get_processing_levels: { @@ -14605,6 +14884,15 @@ export interface operations { "application/json": string; }; }; + /** @description Conflict */ + 409: { + headers: { + [name: string]: unknown; + }; + content: { + "application/json": string; + }; + }; }; }; interfaces_api_views_sta_processing_level_get_processing_level: { @@ -14879,6 +15167,15 @@ export interface operations { "application/json": string; }; }; + /** @description Conflict */ + 409: { + headers: { + [name: string]: unknown; + }; + content: { + "application/json": string; + }; + }; }; }; interfaces_api_views_quality_history_get_qc_history: { @@ -15113,6 +15410,15 @@ export interface operations { "application/json": string; }; }; + /** @description Conflict */ + 409: { + headers: { + [name: string]: unknown; + }; + content: { + "application/json": string; + }; + }; }; }; interfaces_api_views_quality_session_get_qc_session: { @@ -15466,6 +15772,15 @@ export interface operations { "application/json": string; }; }; + /** @description Conflict */ + 409: { + headers: { + [name: string]: unknown; + }; + content: { + "application/json": string; + }; + }; }; }; interfaces_api_views_quality_operation_get_qc_operation: { @@ -15632,6 +15947,15 @@ export interface operations { "application/json": string; }; }; + /** @description Conflict */ + 409: { + headers: { + [name: string]: unknown; + }; + content: { + "application/json": string; + }; + }; }; }; interfaces_api_views_sta_result_qualifier_get_result_qualifiers: { @@ -15733,6 +16057,15 @@ export interface operations { "application/json": string; }; }; + /** @description Conflict */ + 409: { + headers: { + [name: string]: unknown; + }; + content: { + "application/json": string; + }; + }; }; }; interfaces_api_views_sta_result_qualifier_get_result_qualifier: { @@ -15870,6 +16203,15 @@ export interface operations { "application/json": string; }; }; + /** @description Conflict */ + 409: { + headers: { + [name: string]: unknown; + }; + content: { + "application/json": string; + }; + }; }; }; interfaces_api_views_iam_role_get_roles: { @@ -16062,6 +16404,15 @@ export interface operations { "application/json": string; }; }; + /** @description Conflict */ + 409: { + headers: { + [name: string]: unknown; + }; + content: { + "application/json": string; + }; + }; }; }; interfaces_api_views_sta_sampled_medium_get_sampled_medium: { @@ -16197,6 +16548,15 @@ export interface operations { "application/json": string; }; }; + /** @description Conflict */ + 409: { + headers: { + [name: string]: unknown; + }; + content: { + "application/json": string; + }; + }; }; }; interfaces_api_views_sta_unit_type_get_unit_types: { @@ -16296,6 +16656,15 @@ export interface operations { "application/json": string; }; }; + /** @description Conflict */ + 409: { + headers: { + [name: string]: unknown; + }; + content: { + "application/json": string; + }; + }; }; }; interfaces_api_views_sta_unit_type_get_unit_type: { @@ -16431,6 +16800,15 @@ export interface operations { "application/json": string; }; }; + /** @description Conflict */ + 409: { + headers: { + [name: string]: unknown; + }; + content: { + "application/json": string; + }; + }; }; }; interfaces_api_views_sta_unit_get_units: { @@ -16538,6 +16916,15 @@ export interface operations { "application/json": string; }; }; + /** @description Conflict */ + 409: { + headers: { + [name: string]: unknown; + }; + content: { + "application/json": string; + }; + }; }; }; interfaces_api_views_sta_unit_get_unit: { @@ -16797,6 +17184,15 @@ export interface operations { "application/json": string; }; }; + /** @description Conflict */ + 409: { + headers: { + [name: string]: unknown; + }; + content: { + "application/json": string; + }; + }; }; }; interfaces_api_views_iam_workspace_get_workspace: { @@ -16944,6 +17340,15 @@ export interface operations { "application/json": string; }; }; + /** @description Conflict */ + 409: { + headers: { + [name: string]: unknown; + }; + content: { + "application/json": string; + }; + }; }; }; interfaces_api_views_iam_collaborator_get_collaborators: { @@ -17101,6 +17506,15 @@ export interface operations { "application/json": string; }; }; + /** @description Conflict */ + 409: { + headers: { + [name: string]: unknown; + }; + content: { + "application/json": string; + }; + }; }; }; interfaces_api_views_iam_collaborator_remove_collaborator: { @@ -17229,6 +17643,15 @@ export interface operations { "application/json": string; }; }; + /** @description Conflict */ + 409: { + headers: { + [name: string]: unknown; + }; + content: { + "application/json": string; + }; + }; }; }; interfaces_api_views_iam_service_account_get_service_account: { diff --git a/packages/hydroserverpy/src/hydroserverpy/api/client.py b/packages/hydroserverpy/src/hydroserverpy/api/client.py index 16ceff2c6..1fac60637 100644 --- a/packages/hydroserverpy/src/hydroserverpy/api/client.py +++ b/packages/hydroserverpy/src/hydroserverpy/api/client.py @@ -112,13 +112,13 @@ def _raise_for_hs_status(response): response.raise_for_status() except requests.HTTPError as e: try: - http_error_msg = ( - f"{response.status_code} Client Error: " - f"{str(json.loads(response.content).get('detail'))}" - ) + body = json.loads(response.content) + error_detail = body.get("message") or body.get("detail") or response.text + http_error_msg = f"{response.status_code} Client Error: {error_detail}" except ( ValueError, TypeError, + AttributeError, ): http_error_msg = e if 400 <= response.status_code < 500: diff --git a/packages/hydroserverpy/src/hydroserverpy/api/models/etl/data_connection.py b/packages/hydroserverpy/src/hydroserverpy/api/models/etl/data_connection.py index bee71f37b..ea81e3d6c 100644 --- a/packages/hydroserverpy/src/hydroserverpy/api/models/etl/data_connection.py +++ b/packages/hydroserverpy/src/hydroserverpy/api/models/etl/data_connection.py @@ -1,12 +1,13 @@ import uuid from datetime import datetime from typing import ClassVar, List, Literal, Optional, Union, TYPE_CHECKING -from pydantic import BaseModel, ConfigDict, Field, AliasPath +from pydantic import BaseModel, ConfigDict, Field from pydantic.alias_generators import to_camel from ..base import HydroServerBaseModel if TYPE_CHECKING: from hydroserverpy import HydroServer + from hydroserverpy.api.models import Workspace class CSVPayload(BaseModel): @@ -61,8 +62,7 @@ class DataConnection(HydroServerBaseModel): name: str description: Optional[str] = None source_url: str - workspace_id: uuid.UUID = Field(..., validation_alias=AliasPath("workspace", "id")) - workspace_name: str = Field(..., validation_alias=AliasPath("workspace", "name")) + workspace_id: uuid.UUID timezone_type: Optional[Literal["offset", "iana"]] = None timezone: Optional[str] = None auth_header_name: Optional[str] = None @@ -78,6 +78,17 @@ class DataConnection(HydroServerBaseModel): def __init__(self, client: "HydroServer", **data): super().__init__(client=client, service=client.dataconnections, **data) + self._workspace = None + + @property + def workspace(self) -> "Workspace": + """The workspace this data connection belongs to.""" + + if self._workspace is None: + self._workspace = self.client.workspaces.get(uid=self.workspace_id) + + return self._workspace + def save(self): """Saves changes to this resource to HydroServer.""" diff --git a/packages/hydroserverpy/src/hydroserverpy/api/models/etl/task.py b/packages/hydroserverpy/src/hydroserverpy/api/models/etl/task.py index 6ab4c46c1..62a53acd0 100644 --- a/packages/hydroserverpy/src/hydroserverpy/api/models/etl/task.py +++ b/packages/hydroserverpy/src/hydroserverpy/api/models/etl/task.py @@ -17,6 +17,7 @@ class EtlTask(HydroServerBaseModel): name: str description: Optional[str] = None task_variables: Dict[str, Any] = Field(default_factory=dict) + workspace_id: uuid.UUID data_connection_id: uuid.UUID = Field( validation_alias=AliasChoices( "dataConnectionId", AliasPath("dataConnection", "id") diff --git a/packages/hydroserverpy/src/hydroserverpy/api/models/iam/role.py b/packages/hydroserverpy/src/hydroserverpy/api/models/iam/role.py index 891d2bc46..cae1191de 100644 --- a/packages/hydroserverpy/src/hydroserverpy/api/models/iam/role.py +++ b/packages/hydroserverpy/src/hydroserverpy/api/models/iam/role.py @@ -1,6 +1,6 @@ -from typing import Optional, Union, TYPE_CHECKING +from typing import List, Optional, Union, TYPE_CHECKING from uuid import UUID -from pydantic import Field +from pydantic import BaseModel, ConfigDict, Field from ..base import HydroServerBaseModel if TYPE_CHECKING: @@ -8,10 +8,18 @@ from hydroserverpy.api.models import Workspace +class RolePermission(BaseModel): + resource: str + action: str + + model_config = ConfigDict(populate_by_name=True) + + class Role(HydroServerBaseModel): name: str = Field(..., max_length=255) description: Optional[str] = None workspace_id: Optional[Union[UUID, str]] = None + permissions: List[RolePermission] = [] def __init__(self, client: "HydroServer", **data): super().__init__(client=client, service=client.workspaces, **data) diff --git a/packages/hydroserverpy/src/hydroserverpy/api/models/iam/workspace.py b/packages/hydroserverpy/src/hydroserverpy/api/models/iam/workspace.py index 22a30b2c6..e22a423dd 100644 --- a/packages/hydroserverpy/src/hydroserverpy/api/models/iam/workspace.py +++ b/packages/hydroserverpy/src/hydroserverpy/api/models/iam/workspace.py @@ -1,7 +1,7 @@ from typing import List, Union, Optional, ClassVar, TYPE_CHECKING from uuid import UUID from datetime import datetime -from pydantic import Field, EmailStr, AliasPath +from pydantic import Field, EmailStr, AliasPath, AliasChoices from ..base import HydroServerBaseModel if TYPE_CHECKING: @@ -10,7 +10,6 @@ Role, Collaborator, ServiceAccount, - Account, MonitoringSite, ObservedProperty, Method, @@ -18,7 +17,6 @@ ProcessingLevel, ResultQualifier, Datastream, - OrchestrationSystem, Task, DataConnection, ) @@ -27,11 +25,11 @@ class Workspace(HydroServerBaseModel): name: str = Field(..., max_length=255) is_private: bool - owner: "Account" + owner_email: EmailStr collaborator_role_id: Optional[Union[UUID, str]] = Field( - None, validation_alias=AliasPath("collaboratorRole", "id") + None, validation_alias=AliasChoices("collaboratorRoleId", AliasPath("collaboratorRole", "id")) ) - pending_transfer_to: Optional["Account"] = None + pending_transfer_to_email: Optional[EmailStr] = None _editable_fields: ClassVar[set[str]] = {"name", "is_private"} @@ -49,7 +47,6 @@ def __init__(self, client: "HydroServer", **data): self._units = None self._methods = None self._datastreams = None - self._orchestrationsystems = None self._dataconnections = None self._tasks = None @@ -59,10 +56,10 @@ def get_route(cls): @property def roles(self) -> List["Role"]: - """The roles that can be assigned for this workspace.""" + """The roles that can be assigned for this workspace, including system roles.""" if self._roles is None: - self._roles = self.client.roles.list(workspace=self.uid, fetch_all=True).items + self._roles = self.client.roles.list(workspace=[self.uid, None], fetch_all=True).items return self._roles @@ -156,15 +153,6 @@ def datastreams(self) -> List["Datastream"]: return self._datastreams - @property - def orchestrationsystems(self) -> List["OrchestrationSystem"]: - """The orchestration systems associated with this workspace.""" - - if self._orchestrationsystems is None: - self._orchestrationsystems = self.client.orchestrationsystems.list(workspace=self.uid, fetch_all=True).items - - return self._orchestrationsystems - @property def dataconnections(self) -> List["DataConnection"]: """The ETL data connections associated with this workspace.""" diff --git a/packages/hydroserverpy/src/hydroserverpy/api/models/monitoring/task.py b/packages/hydroserverpy/src/hydroserverpy/api/models/monitoring/task.py index df20928ae..d9b9f188c 100644 --- a/packages/hydroserverpy/src/hydroserverpy/api/models/monitoring/task.py +++ b/packages/hydroserverpy/src/hydroserverpy/api/models/monitoring/task.py @@ -1,9 +1,8 @@ import uuid from datetime import datetime from functools import cached_property -from typing import ClassVar, List, Literal, Optional, TYPE_CHECKING -from pydantic import BaseModel, ConfigDict, Field, AliasPath -from pydantic.alias_generators import to_camel +from typing import ClassVar, Dict, List, Literal, Optional, TYPE_CHECKING +from pydantic import Field, AliasPath from ..base import HydroServerBaseModel from ..orchestration.run import TaskRun @@ -11,28 +10,10 @@ from hydroserverpy import HydroServer -class MonitoringRuleInline(BaseModel): - id: uuid.UUID - rule_type: str - last_checked_at: Optional[datetime] = None - min_value: Optional[float] = None - max_value: Optional[float] = None - window_interval: Optional[int] = None - window_interval_units: Optional[Literal["minutes", "hours", "days"]] = None - - model_config = ConfigDict(alias_generator=to_camel, populate_by_name=True) - - -class MonitoredDatastream(BaseModel): - datastream_id: uuid.UUID - rules: List[MonitoringRuleInline] = [] - - model_config = ConfigDict(alias_generator=to_camel, populate_by_name=True) - - class MonitoringTask(HydroServerBaseModel): name: str description: Optional[str] = None + workspace_id: uuid.UUID monitoring_site_id: uuid.UUID enabled: Optional[bool] = Field(None, validation_alias=AliasPath("schedule", "enabled")) start_time: Optional[datetime] = Field(None, validation_alias=AliasPath("schedule", "startTime")) @@ -43,7 +24,7 @@ class MonitoringTask(HydroServerBaseModel): ) next_run_at: Optional[datetime] = Field(None, validation_alias=AliasPath("schedule", "nextRunAt")) latest_run: Optional[TaskRun] = None - monitored_datastreams: List[MonitoredDatastream] = [] + rule_type_counts: Dict[str, int] = {} recipients: List[str] = [] _editable_fields: ClassVar[set[str]] = { @@ -55,7 +36,6 @@ class MonitoringTask(HydroServerBaseModel): "interval", "interval_period", "start_time", - "next_run_at", } def __init__(self, client: "HydroServer", **data): diff --git a/packages/hydroserverpy/src/hydroserverpy/api/models/products/rating_curve.py b/packages/hydroserverpy/src/hydroserverpy/api/models/products/rating_curve.py index d92f45d3e..a58972184 100644 --- a/packages/hydroserverpy/src/hydroserverpy/api/models/products/rating_curve.py +++ b/packages/hydroserverpy/src/hydroserverpy/api/models/products/rating_curve.py @@ -1,18 +1,17 @@ import uuid from typing import ClassVar, List, Literal, Optional, Tuple, TYPE_CHECKING -from pydantic import Field, AliasPath from ..base import HydroServerBaseModel if TYPE_CHECKING: from hydroserverpy import HydroServer + from hydroserverpy.api.models import MonitoringSite class RatingCurve(HydroServerBaseModel): name: str description: Optional[str] = None fitting_method: Literal["linear", "power_law"] - monitoring_site_id: uuid.UUID = Field(..., validation_alias=AliasPath("monitoring_site", "id")) - monitoring_site_name: str = Field(..., validation_alias=AliasPath("monitoring_site", "name")) + monitoring_site_id: uuid.UUID points: List[Tuple[float, float]] = [] _editable_fields: ClassVar[set[str]] = set() @@ -20,6 +19,17 @@ class RatingCurve(HydroServerBaseModel): def __init__(self, client: "HydroServer", **data): super().__init__(client=client, service=client.ratingcurves, **data) + self._monitoring_site = None + + @property + def monitoring_site(self) -> "MonitoringSite": + """The monitoring site this rating curve belongs to.""" + + if self._monitoring_site is None: + self._monitoring_site = self.client.monitoring_sites.get(uid=self.monitoring_site_id) + + return self._monitoring_site + def save(self): """Saves changes to this resource to HydroServer.""" diff --git a/packages/hydroserverpy/src/hydroserverpy/api/models/products/task.py b/packages/hydroserverpy/src/hydroserverpy/api/models/products/task.py index ba3929c80..75ddc4206 100644 --- a/packages/hydroserverpy/src/hydroserverpy/api/models/products/task.py +++ b/packages/hydroserverpy/src/hydroserverpy/api/models/products/task.py @@ -1,7 +1,7 @@ import uuid from datetime import datetime from functools import cached_property -from typing import ClassVar, List, Optional, Union, TYPE_CHECKING +from typing import ClassVar, List, Literal, Optional, Union, TYPE_CHECKING from pydantic import Field, AliasPath from ..base import HydroServerBaseModel from ..orchestration.run import TaskRun @@ -9,7 +9,6 @@ RatingCurveTransformation, DerivationTransformation, AggregationTransformation, - Period, ) if TYPE_CHECKING: @@ -19,19 +18,28 @@ class DataProductTask(HydroServerBaseModel): name: str description: Optional[str] = None + workspace_id: uuid.UUID monitoring_site_id: uuid.UUID enabled: Optional[bool] = Field(None, validation_alias=AliasPath("schedule", "enabled")) start_time: Optional[datetime] = Field(None, validation_alias=AliasPath("schedule", "startTime")) crontab: Optional[str] = Field(None, validation_alias=AliasPath("schedule", "crontab")) interval: Optional[int] = Field(None, validation_alias=AliasPath("schedule", "interval")) - interval_period: Optional[Period] = Field( + interval_period: Optional[Literal["minutes", "hours", "days"]] = Field( None, validation_alias=AliasPath("schedule", "intervalPeriod") ) next_run_at: Optional[datetime] = Field(None, validation_alias=AliasPath("schedule", "nextRunAt")) latest_run: Optional[TaskRun] = None transformation_types: List[str] = [] - _editable_fields: ClassVar[set[str]] = {"name", "description"} + _editable_fields: ClassVar[set[str]] = { + "name", + "description", + "enabled", + "crontab", + "interval", + "interval_period", + "start_time", + } def __init__(self, client: "HydroServer", **data): super().__init__(client=client, service=client.dataproducttasks, **data) diff --git a/packages/hydroserverpy/src/hydroserverpy/api/models/sta/datastream.py b/packages/hydroserverpy/src/hydroserverpy/api/models/sta/datastream.py index 02dec4047..ec009c434 100644 --- a/packages/hydroserverpy/src/hydroserverpy/api/models/sta/datastream.py +++ b/packages/hydroserverpy/src/hydroserverpy/api/models/sta/datastream.py @@ -16,6 +16,7 @@ ObservedProperty, Unit, ProcessingLevel, + ObservationCollection, ) @@ -174,7 +175,7 @@ def get_observations( phenomenon_time_min: datetime = ..., result_qualifiers: str = ..., fetch_all: bool = False, - ) -> pd.DataFrame: + ) -> "ObservationCollection": """Retrieve the observations for this datastream.""" return self.client.datastreams.get_observations( @@ -191,14 +192,18 @@ def get_observations( def load_observations( self, observations: pd.DataFrame, - mode: str = "insert" + mode: str = "insert", + phenomenon_time_start: Optional[datetime] = None, + phenomenon_time_end: Optional[datetime] = None, ) -> None: """Load a DataFrame of observations to the datastream.""" return self.client.datastreams.load_observations( uid=self.uid, observations=observations, - mode=mode + mode=mode, + phenomenon_time_start=phenomenon_time_start, + phenomenon_time_end=phenomenon_time_end, ) def delete_observations( diff --git a/packages/hydroserverpy/src/hydroserverpy/api/services/iam/role.py b/packages/hydroserverpy/src/hydroserverpy/api/services/iam/role.py index 023f22e44..9d9fe0095 100644 --- a/packages/hydroserverpy/src/hydroserverpy/api/services/iam/role.py +++ b/packages/hydroserverpy/src/hydroserverpy/api/services/iam/role.py @@ -20,15 +20,23 @@ def list( offset: int = ..., limit: int = ..., sortby: List[str] = ..., - workspace: Optional[Union["Workspace", UUID, str]] = ..., + workspace: Optional[Union["Workspace", UUID, str, List[Optional[Union["Workspace", UUID, str]]]]] = ..., fetch_all: bool = False, ) -> List["Role"]: - """Fetch a collection of HydroServer roles.""" + """ + Fetch a collection of HydroServer roles. Pass a list to match several workspaces; + None in the list matches system roles that belong to no workspace. + """ + + if isinstance(workspace, (list, tuple)): + workspace_id = ["null" if w is None else normalize_uuid(w) for w in workspace] + else: + workspace_id = normalize_uuid(workspace) return super().list( offset=offset, limit=limit, sortby=sortby, fetch_all=fetch_all, - workspace_id=normalize_uuid(workspace), + workspace_id=workspace_id, ) diff --git a/packages/hydroserverpy/src/hydroserverpy/api/services/sta/datastream.py b/packages/hydroserverpy/src/hydroserverpy/api/services/sta/datastream.py index 0ce2627d9..f9ad6cc7e 100644 --- a/packages/hydroserverpy/src/hydroserverpy/api/services/sta/datastream.py +++ b/packages/hydroserverpy/src/hydroserverpy/api/services/sta/datastream.py @@ -6,6 +6,7 @@ from pydantic.alias_generators import to_camel from hydroserverpy.api.models import Datastream, ObservationCollection from hydroserverpy.api.utils import normalize_uuid, build_datetime_interval, sortby_to_camel +from hydroserverpy.api.models.base import HydroServerCollection from ..base import HydroServerBaseService if TYPE_CHECKING: @@ -51,7 +52,7 @@ def list( phenomenon_time_max: datetime = ..., phenomenon_time_min: datetime = ..., fetch_all: bool = False, - ) -> List["Workspace"]: + ) -> HydroServerCollection: """ Fetch a collection of HydroServer datastreams. """ @@ -277,9 +278,14 @@ def load_observations( self, uid: Union[UUID, str], observations: pd.DataFrame, - mode: str = "insert" + mode: str = "insert", + phenomenon_time_start: Optional[datetime] = None, + phenomenon_time_end: Optional[datetime] = None, ) -> None: - """Load observations to a datastream.""" + """ + Load observations to a datastream. In replace mode, the optional phenomenon time range sets + what the server deletes in the same transaction as the insert. + """ observations = observations.drop( columns=[col for col in observations.columns if to_camel(col) in READ_ONLY_OBSERVATION_PROPERTIES] @@ -288,6 +294,10 @@ def load_observations( path = f"/{self.client.base_route}/collections/observations/bulk-create" headers = {"Content-type": "application/json"} params = {"mode": mode} + if phenomenon_time_start is not None: + params["phenomenonTimeStart"] = phenomenon_time_start.isoformat() + if phenomenon_time_end is not None: + params["phenomenonTimeEnd"] = phenomenon_time_end.isoformat() body = { "datastreamId": str(uid), "fields": [to_camel(col) for col in observations.columns.tolist()], diff --git a/packages/hydroserverpy/src/hydroserverpy/etl/__init__.py b/packages/hydroserverpy/src/hydroserverpy/etl/__init__.py index b883a10ff..eed798faa 100644 --- a/packages/hydroserverpy/src/hydroserverpy/etl/__init__.py +++ b/packages/hydroserverpy/src/hydroserverpy/etl/__init__.py @@ -1,4 +1,4 @@ -from .models.pipeline import ETLPipeline +from .models.pipeline import ETLPipeline, ETLContext, ETLStatus, ETLStage -__all__ = ["ETLPipeline"] +__all__ = ["ETLPipeline", "ETLContext", "ETLStatus", "ETLStage"] diff --git a/packages/hydroserverpy/src/hydroserverpy/etl/loaders/hydroserver.py b/packages/hydroserverpy/src/hydroserverpy/etl/loaders/hydroserver.py index 11b1fca2a..5df40fd62 100644 --- a/packages/hydroserverpy/src/hydroserverpy/etl/loaders/hydroserver.py +++ b/packages/hydroserverpy/src/hydroserverpy/etl/loaders/hydroserver.py @@ -114,47 +114,55 @@ def load( self.chunk_size, ) - if data_ingestion_window_start is not None: - try: - self.client.datastreams.delete_observations( - uid=datastream.uid, - phenomenon_time_start=target_df["timestamp"].min(), - phenomenon_time_end=target_df["timestamp"].max(), - ) - except Exception as e: - etl_results.target_results[target_id].status = "failed" - etl_results.target_results[target_id].error = str(e) - etl_results.target_results[target_id].traceback = traceback.format_exc() - logger.info( - "Load result: loaded=0 available=%s cutoff=%s", - observations_to_load, - self._format_cutoff(datastream.phenomenon_end_time), - ) - continue - - upload_df = target_df.rename(columns={"timestamp": "phenomenon_time", "value": "result"}) + upload_df = ( + target_df.sort_values("timestamp") + .rename(columns={"timestamp": "phenomenon_time", "value": "result"}) + ) + replace = data_ingestion_window_start is not None for start_idx in range(0, observations_to_load, self.chunk_size): end_idx = min(start_idx + self.chunk_size, observations_to_load) chunk = upload_df.iloc[start_idx:end_idx] + chunk_start, chunk_end = chunk["phenomenon_time"].min(), chunk["phenomenon_time"].max() + + # Each chunk replaces a contiguous slice of the reload range in one atomic request, + # so a failed chunk leaves its slice (and every later one) untouched. + replace_range = {} + if replace: + replace_range = { + "phenomenon_time_start": chunk_start, + "phenomenon_time_end": ( + upload_df["phenomenon_time"].iloc[end_idx] - pd.Timedelta(microseconds=1) + if end_idx < observations_to_load else chunk_end + ), + } try: self.client.datastreams.load_observations( uid=datastream.uid, observations=chunk, - mode="insert" if data_ingestion_window_start is not None else "append", + mode="replace" if replace else "append", + **replace_range, + ) + target_result = etl_results.target_results[target_id] + target_result.values_loaded += len(chunk) + target_result.earliest_timestamp = ( + chunk_start if target_result.earliest_timestamp is None + else min(target_result.earliest_timestamp, chunk_start) + ) + target_result.latest_timestamp = ( + chunk_end if target_result.latest_timestamp is None + else max(target_result.latest_timestamp, chunk_end) ) - etl_results.target_results[target_id].values_loaded += len(chunk) except Exception as e: etl_results.target_results[target_id].status = "failed" etl_results.target_results[target_id].error = str(e) etl_results.target_results[target_id].traceback = traceback.format_exc() break - if not etl_results.target_results[target_id].values_loaded > 0: - etl_results.target_results[target_id].status = "skipped" - - if etl_results.target_results[target_id].status not in ["skipped", "failed"]: - etl_results.target_results[target_id].status = "success" + if etl_results.target_results[target_id].status is None: + etl_results.target_results[target_id].status = ( + "success" if etl_results.target_results[target_id].values_loaded > 0 else "skipped" + ) logger.info( "Load result: loaded=%s available=%s cutoff=%s", diff --git a/packages/hydroserverpy/src/hydroserverpy/etl/models/pipeline.py b/packages/hydroserverpy/src/hydroserverpy/etl/models/pipeline.py index ffd2cfbae..c7a30918e 100644 --- a/packages/hydroserverpy/src/hydroserverpy/etl/models/pipeline.py +++ b/packages/hydroserverpy/src/hydroserverpy/etl/models/pipeline.py @@ -182,6 +182,11 @@ def run( If raise_on_error is True, the first exception raised by any stage will be propagated to the caller. If raise_on_error is False, the ETLContext will be marked as failed and returned. + + Resulting status: FAILED if any stage raised, including the loader's + up-front check that every target exists; INCOMPLETE if at least one + target failed while uploading; otherwise SUCCESS, even if every target + was skipped because it had no new observations. """ context = ETLContext() diff --git a/packages/hydroserverpy/src/hydroserverpy/etl/models/timestamp.py b/packages/hydroserverpy/src/hydroserverpy/etl/models/timestamp.py index fa123f727..63bca0ae9 100644 --- a/packages/hydroserverpy/src/hydroserverpy/etl/models/timestamp.py +++ b/packages/hydroserverpy/src/hydroserverpy/etl/models/timestamp.py @@ -12,6 +12,10 @@ _OFFSET_RE = re.compile(r"^[+-](\d{4}|\d{2}:\d{2})$") +# A trailing ±HH[:MM] is only an offset when it follows a time component; otherwise it is the day +# or month of a date-only value such as '2026-01-01'. Whitespace may separate the time and offset. +_EMBEDDED_TZ_RE = r"[Zz]$|[T\s]\d{2}(?::?\d{2}){0,2}(?:[.,]\d+)?\s*[+-]\d{2}(?::?\d{2})?$" + class Timezone(BaseModel): timezone_type: Optional[TimezoneType] = None @@ -188,7 +192,7 @@ def parse_series_to_utc( # ISO strings: regex detects embedded tz per element to support mixed-offset series # (pandas cannot parse mixed tz/naive strings without coercing to NaT) - has_tz = series.str.contains(r"[Zz]$|[+-]\d{2}(?::?\d{2})?$", regex=True, na=False) + has_tz = series.str.contains(_EMBEDDED_TZ_RE, regex=True, na=False) tz_aware = pd.to_datetime(series[has_tz], utc=True, errors="raise") tz_naive_raw_series = pd.to_datetime(series[~has_tz], utc=False, errors="raise") diff --git a/packages/hydroserverpy/tests/api/test_merge_patch.py b/packages/hydroserverpy/tests/api/test_merge_patch.py index f73821702..1a79a6ea3 100644 --- a/packages/hydroserverpy/tests/api/test_merge_patch.py +++ b/packages/hydroserverpy/tests/api/test_merge_patch.py @@ -33,7 +33,12 @@ def test_task_save_generates_merge_patch_from_server_snapshot(case): client = MagicMock() client.base_route = "api/ogc" client.etltasks = TaskService(client) - fields = {"id": str(uuid4()), "name": "Task", "dataConnectionId": str(uuid4())} + fields = { + "id": str(uuid4()), + "name": "Task", + "workspaceId": str(uuid4()), + "dataConnectionId": str(uuid4()), + } task = EtlTask(client=client, **fields, **case["original"]) task.task_variables = case["updated"]["taskVariables"] client.request.return_value.json.return_value = {"data": {**fields, **case["updated"]}} diff --git a/packages/hydroserverpy/tests/api/test_response_models.py b/packages/hydroserverpy/tests/api/test_response_models.py new file mode 100644 index 000000000..af382ef7f --- /dev/null +++ b/packages/hydroserverpy/tests/api/test_response_models.py @@ -0,0 +1,315 @@ +import json +import uuid +from unittest.mock import MagicMock + +import pytest + +from hydroserverpy.api.services.etl.data_connection import DataConnectionService +from hydroserverpy.api.services.etl.task import TaskService +from hydroserverpy.api.services.iam.role import RoleService +from hydroserverpy.api.services.iam.workspace import WorkspaceService +from hydroserverpy.api.services.monitoring.task import MonitoringTaskService +from hydroserverpy.api.services.products.rating_curve import RatingCurveService +from hydroserverpy.api.services.products.task import DataProductTaskService + + +class FakeResponse: + def __init__(self, payload=None, status_code=200): + self._payload = payload + self.status_code = status_code + + def json(self): + if self._payload is None: + raise ValueError("No JSON content (empty response body)") + return self._payload + + +def make_client(): + client = MagicMock() + client.base_route = "/api/ogc" + return client + + +# Payloads mirror the server's *Response serialization, which returns related resources as flat +# IDs/emails rather than nested objects. + +SCHEDULE = { + "enabled": True, + "startTime": "2026-01-01T00:00:00+00:00", + "crontab": None, + "interval": 15, + "intervalPeriod": "minutes", + "nextRunAt": "2026-01-01T00:15:00+00:00", +} + +def workspace_payload(item_id): + return { + "id": item_id, + "name": "My Workspace", + "isPrivate": False, + "ownerEmail": "owner@example.com", + "pendingTransferToEmail": None, + "collaboratorRoleId": "00000000-0000-0000-0000-000000000001", + } + + +def data_connection_payload(item_id): + return { + "id": item_id, + "name": "My Connection", + "description": None, + "sourceUrl": "https://example.com/data.csv", + "authHeaderName": None, + "authHeaderValue": None, + "timezoneType": "offset", + "timezone": "-0700", + "workspaceId": "00000000-0000-0000-0000-000000000002", + "payload": { + "type": "CSV", + "timestampKey": "timestamp", + "timestampFormat": "ISO8601", + "headerRow": 1, + "dataStartRow": 2, + "delimiter": ",", + "dataIngestionWindow": None, + }, + "placeholderVariables": [], + "notification": None, + "taskCount": 0, + "taskAttentionCount": 0, + } + + +def rating_curve_payload(item_id): + return { + "id": item_id, + "name": "My Rating Curve", + "description": None, + "fittingMethod": "linear", + "points": [[0.0, 0.0], [1.0, 2.5]], + "monitoringSiteId": "00000000-0000-0000-0000-000000000003", + } + + +def etl_task_payload(item_id): + return { + "id": item_id, + "name": "My ETL Task", + "description": None, + "workspaceId": "00000000-0000-0000-0000-000000000002", + "dataConnectionId": "00000000-0000-0000-0000-000000000004", + "taskVariables": {"site_code": "10109000"}, + "schedule": SCHEDULE, + "latestRun": None, + "mappingCount": 1, + } + + +def monitoring_task_payload(item_id): + return { + "id": item_id, + "name": "My Monitoring Task", + "description": None, + "workspaceId": "00000000-0000-0000-0000-000000000002", + "monitoringSiteId": "00000000-0000-0000-0000-000000000003", + "schedule": SCHEDULE, + "latestRun": None, + "ruleTypeCounts": {"range": 1, "missing_data": 1}, + "recipients": ["owner@example.com"], + } + + +def data_product_task_payload(item_id): + return { + "id": item_id, + "name": "My Data Product Task", + "description": None, + "workspaceId": "00000000-0000-0000-0000-000000000002", + "monitoringSiteId": "00000000-0000-0000-0000-000000000003", + "schedule": SCHEDULE, + "latestRun": None, + "transformationTypes": ["derivation"], + } + + +@pytest.fixture( + params=[ + (WorkspaceService, workspace_payload, {"name": "My Workspace", "is_private": False}), + ( + DataConnectionService, + data_connection_payload, + { + "name": "My Connection", + "workspace": "00000000-0000-0000-0000-000000000002", + "source_url": "https://example.com/data.csv", + "timestamp_key": "timestamp", + "payload_type": "CSV", + }, + ), + ( + RatingCurveService, + rating_curve_payload, + { + "name": "My Rating Curve", + "monitoring_site": "00000000-0000-0000-0000-000000000003", + "fitting_method": "linear", + }, + ), + ( + TaskService, + etl_task_payload, + {"name": "My ETL Task", "data_connection": "00000000-0000-0000-0000-000000000004"}, + ), + ( + MonitoringTaskService, + monitoring_task_payload, + {"name": "My Monitoring Task", "monitoring_site": "00000000-0000-0000-0000-000000000003"}, + ), + ( + DataProductTaskService, + data_product_task_payload, + {"name": "My Data Product Task", "monitoring_site": "00000000-0000-0000-0000-000000000003"}, + ), + ], + ids=["workspace", "data_connection", "rating_curve", "etl_task", "monitoring_task", "data_product_task"], +) +def resource(request): + service_cls, payload_fn, create_kwargs = request.param + return service_cls(make_client()), payload_fn, create_kwargs + + +def test_get_parses_server_response(resource): + service, payload_fn, _ = resource + item_id = str(uuid.uuid4()) + service.client.request.return_value = FakeResponse({"data": payload_fn(item_id)}) + + result = service.get(item_id) + + assert str(result.uid) == item_id + + +def test_list_parses_server_response(resource): + service, payload_fn, _ = resource + item_ids = [str(uuid.uuid4()), str(uuid.uuid4())] + service.client.request.return_value = FakeResponse( + {"data": [payload_fn(i) for i in item_ids], "meta": {"numberMatched": 2}} + ) + + result = service.list() + + assert [str(item.uid) for item in result.items] == item_ids + + +def test_create_and_update_parse_refetched_server_response(resource): + service, payload_fn, create_kwargs = resource + item_id = str(uuid.uuid4()) + service.client.request.side_effect = [ + FakeResponse({"id": item_id}, status_code=201), + FakeResponse({"data": payload_fn(item_id)}), + FakeResponse(status_code=204), + FakeResponse({"data": payload_fn(item_id)}), + ] + + created = service.create(**create_kwargs) + update_kwargs = { + k: v for k, v in create_kwargs.items() if k not in ("workspace", "monitoring_site", "data_connection") + } + if isinstance(service, RatingCurveService): + update_kwargs["points"] = [] + updated = service.update(item_id, **update_kwargs) + + assert str(created.uid) == item_id + assert str(updated.uid) == item_id + + +def test_workspace_maps_flat_related_fields(): + service = WorkspaceService(make_client()) + item_id = str(uuid.uuid4()) + service.client.request.return_value = FakeResponse({"data": workspace_payload(item_id)}) + + workspace = service.get(item_id) + + assert workspace.owner_email == "owner@example.com" + assert workspace.pending_transfer_to_email is None + assert str(workspace.collaborator_role_id) == "00000000-0000-0000-0000-000000000001" + + +def test_related_resources_resolve_from_flat_ids(): + client = make_client() + dc_service = DataConnectionService(client) + rc_service = RatingCurveService(client) + client.request.side_effect = [ + FakeResponse({"data": data_connection_payload(str(uuid.uuid4()))}), + FakeResponse({"data": rating_curve_payload(str(uuid.uuid4()))}), + ] + + data_connection = dc_service.get(uuid.uuid4()) + rating_curve = rc_service.get(uuid.uuid4()) + + assert data_connection.workspace is client.workspaces.get.return_value + client.workspaces.get.assert_called_once_with(uid=uuid.UUID("00000000-0000-0000-0000-000000000002")) + assert rating_curve.monitoring_site is client.monitoring_sites.get.return_value + client.monitoring_sites.get.assert_called_once_with(uid=uuid.UUID("00000000-0000-0000-0000-000000000003")) + + +def test_task_models_map_workspace_and_rule_counts(): + client = make_client() + client.request.side_effect = [ + FakeResponse({"data": monitoring_task_payload(str(uuid.uuid4()))}), + FakeResponse({"data": etl_task_payload(str(uuid.uuid4()))}), + FakeResponse({"data": data_product_task_payload(str(uuid.uuid4()))}), + ] + workspace_id = uuid.UUID("00000000-0000-0000-0000-000000000002") + + monitoring_task = MonitoringTaskService(client).get(uuid.uuid4()) + etl_task = TaskService(client).get(uuid.uuid4()) + data_product_task = DataProductTaskService(client).get(uuid.uuid4()) + + assert monitoring_task.rule_type_counts == {"range": 1, "missing_data": 1} + assert {monitoring_task.workspace_id, etl_task.workspace_id, data_product_task.workspace_id} == {workspace_id} + + +def test_role_maps_permissions(): + client = make_client() + role_id = str(uuid.uuid4()) + client.request.return_value = FakeResponse({"data": { + "id": role_id, + "name": "Editor", + "description": None, + "workspaceId": None, + "permissions": [{"resource": "Datastream", "action": "edit"}], + }}) + + role = RoleService(client).get(role_id) + + assert [(p.resource, p.action) for p in role.permissions] == [("Datastream", "edit")] + + +def test_data_product_task_save_sends_schedule_changes(): + client = make_client() + item_id = str(uuid.uuid4()) + client.request.side_effect = [ + FakeResponse({"data": data_product_task_payload(item_id)}), + FakeResponse(status_code=204), + FakeResponse({"data": {**data_product_task_payload(item_id), "schedule": {**SCHEDULE, "interval": 30}}}), + ] + client.dataproducttasks = DataProductTaskService(client) + task = client.dataproducttasks.get(item_id) + + task.interval = 30 + task.save() + + patch_body = json.loads(client.request.call_args_list[1].kwargs["data"]) + assert patch_body == {"schedule": {"interval": 30}} + assert task.interval == 30 + + +def test_monitoring_task_does_not_send_server_computed_next_run_at(): + client = make_client() + item_id = str(uuid.uuid4()) + client.request.return_value = FakeResponse({"data": monitoring_task_payload(item_id)}) + task = MonitoringTaskService(client).get(item_id) + + task.next_run_at = None + + assert task.unsaved_changes == {} diff --git a/packages/hydroserverpy/tests/etl/test_hydroserver_loader.py b/packages/hydroserverpy/tests/etl/test_hydroserver_loader.py index 81fd833b3..d493d2938 100644 --- a/packages/hydroserverpy/tests/etl/test_hydroserver_loader.py +++ b/packages/hydroserverpy/tests/etl/test_hydroserver_loader.py @@ -133,11 +133,11 @@ def test_window_start_overrides_phenomenon_end_time_cutoff(self): _dt("2026-01-01T00:00:00"), _dt("2026-01-02T00:00:00"), _dt("2026-01-03T00:00:00"), ] - def test_mode_is_insert_when_window_start_is_set(self): + def test_mode_is_replace_when_window_start_is_set(self): loader, client = _make_loader({"target-1": _make_datastream()}) loader.load(_make_payload(), data_ingestion_window_start=_dt("2026-01-01T00:00:00")) - assert client.datastreams.load_observations.call_args.kwargs["mode"] == "insert" + assert client.datastreams.load_observations.call_args.kwargs["mode"] == "replace" def test_mode_is_append_when_window_start_is_not_set(self): loader, client = _make_loader({"target-1": _make_datastream()}) @@ -145,28 +145,19 @@ def test_mode_is_append_when_window_start_is_not_set(self): assert client.datastreams.load_observations.call_args.kwargs["mode"] == "append" - def test_replace_mode_issues_single_delete_over_full_range_before_inserting(self): - loader, client = _make_loader({"target-1": _make_datastream()}, chunk_size=1) + def test_replace_mode_gives_each_chunk_a_contiguous_range_without_separate_delete(self): + loader, client = _make_loader({"target-1": _make_datastream()}, chunk_size=2) loader.load(_make_payload(), data_ingestion_window_start=_dt("2026-01-01T00:00:00")) - client.datastreams.delete_observations.assert_called_once_with( - uid="target-1", - phenomenon_time_start=_dt("2026-01-01T00:00:00"), - phenomenon_time_end=_dt("2026-01-03T00:00:00"), - ) - assert client.datastreams.load_observations.call_count == 3 - assert all( - call.kwargs["mode"] == "insert" for call in client.datastreams.load_observations.call_args_list - ) - - def test_replace_mode_skips_inserts_when_delete_fails(self): - loader, client = _make_loader({"target-1": _make_datastream()}) - client.datastreams.delete_observations.side_effect = Exception("boom") - - result = loader.load(_make_payload(), data_ingestion_window_start=_dt("2026-01-01T00:00:00")) - - assert result.target_results["target-1"].status == "failed" - client.datastreams.load_observations.assert_not_called() + ranges = [ + (call.kwargs["phenomenon_time_start"], call.kwargs["phenomenon_time_end"]) + for call in client.datastreams.load_observations.call_args_list + ] + assert ranges == [ + (_dt("2026-01-01T00:00:00"), _dt("2026-01-02T23:59:59.999999")), + (_dt("2026-01-03T00:00:00"), _dt("2026-01-03T00:00:00")), + ] + client.datastreams.delete_observations.assert_not_called() # --------------------------------------------------------------------------- @@ -201,6 +192,45 @@ def test_uploads_in_multiple_chunks_when_over_chunk_size(self): assert client.datastreams.load_observations.call_count == 2 assert result.target_results["target-1"].values_loaded == 3 + @pytest.mark.parametrize("window_start", [None, _dt("2026-01-01T00:00:00")]) + def test_first_chunk_failure_is_reported_as_failed(self, window_start): + loader, client = _make_loader({"target-1": _make_datastream()}, chunk_size=2) + client.datastreams.load_observations.side_effect = Exception("503 Service Unavailable") + + result = loader.load(_make_payload(), data_ingestion_window_start=window_start) + + assert result.target_results["target-1"].status == "failed" + assert (result.failure_count, result.skipped_count) == (1, 0) + + def test_later_chunk_failure_is_reported_as_failed_with_partial_count(self): + loader, client = _make_loader({"target-1": _make_datastream()}, chunk_size=2) + client.datastreams.load_observations.side_effect = [None, Exception("503 Service Unavailable")] + + result = loader.load(_make_payload()) + + target = result.target_results["target-1"] + assert (target.status, target.values_loaded) == ("failed", 2) + assert result.failure_count == 1 + + def test_records_loaded_timestamp_range_across_chunks(self): + loader, client = _make_loader( + {"target-1": _make_datastream(), "target-2": _make_datastream(uid="target-2")}, chunk_size=2 + ) + payload = pd.concat([ + _make_payload(), + _make_payload(target_id="target-2", timestamps=["2026-01-05T00:00:00Z"], values=[5.0]), + ]) + + result = loader.load(payload) + + target = result.target_results["target-1"] + assert (target.earliest_timestamp, target.latest_timestamp) == ( + _dt("2026-01-01T00:00:00"), _dt("2026-01-03T00:00:00") + ) + assert (result.earliest_timestamp, result.latest_timestamp) == ( + _dt("2026-01-01T00:00:00"), _dt("2026-01-05T00:00:00") + ) + # --------------------------------------------------------------------------- # target_loaded_through diff --git a/packages/hydroserverpy/tests/etl/test_timestamp_model.py b/packages/hydroserverpy/tests/etl/test_timestamp_model.py index ef0d1cacd..0e7aa9ca0 100644 --- a/packages/hydroserverpy/tests/etl/test_timestamp_model.py +++ b/packages/hydroserverpy/tests/etl/test_timestamp_model.py @@ -282,6 +282,33 @@ def test_mixed_hour_only_and_full_offsets(self): assert result.iloc[1] == pd.Timestamp("2024-01-01 06:30:00", tz="UTC") +# --------------------------------------------------------------------------- +# parse_series_to_utc – date-only ISO strings +# --------------------------------------------------------------------------- + +class TestParseSeriesDateOnly: + + @pytest.mark.parametrize("value", ["2024-01-01", "2024-01"]) + @pytest.mark.parametrize("timestamp_fixture", ["iana_timestamp", "offset_timestamp"]) + def test_date_only_strings_are_localized_to_configured_timezone(self, value, timestamp_fixture, request): + ts = request.getfixturevalue(timestamp_fixture) + expected_hour = 5 if timestamp_fixture == "iana_timestamp" else 7 + result = ts.parse_series_to_utc(pd.Series([value])) + assert result.iloc[0] == pd.Timestamp(f"2024-01-01 {expected_hour:02d}:00:00", tz="UTC") + + @pytest.mark.parametrize("value", [ + "2024-01-01T00-07", + "2024-01-01 00:00:00-07:00", + "2024-01-01T000000-0700", + "2024-01-01 00:00:00 -07:00", + "2024-01-01T00:00:00 -0700", + "2024-01-01T00:00:00.000 -07:00", + ]) + def test_offsets_after_time_component_are_preserved(self, value, iana_timestamp): + result = iana_timestamp.parse_series_to_utc(pd.Series([value])) + assert result.iloc[0] == pd.Timestamp("2024-01-01 07:00:00", tz="UTC") + + # --------------------------------------------------------------------------- # parse_series_to_utc – IANA timezone type # --------------------------------------------------------------------------- diff --git a/packages/hydroserverpy/tests/test_api_client.py b/packages/hydroserverpy/tests/test_api_client.py index 500fcfade..3450045bb 100644 --- a/packages/hydroserverpy/tests/test_api_client.py +++ b/packages/hydroserverpy/tests/test_api_client.py @@ -10,6 +10,7 @@ class FakeResponse: def __init__(self, status_code=200, content=b"{}"): self.status_code = status_code self.content = content + self.text = content.decode() def json(self): return json.loads(self.content) @@ -199,3 +200,41 @@ def test_list_sends_camel_case_params_and_keeps_public_filter_names(fake_session assert not any("_" in key for key in params) assert collection.filters["workspace"] == "00000000-0000-0000-0000-000000000001" assert collection.filters["tag"] == ("river", "green") + + + +def test_workspace_roles_include_system_roles(fake_session_factory): + hs = client_module.HydroServer(host="https://example.com", apikey="hs_test_api_key") + workspace_id = "00000000-0000-0000-0000-000000000001" + fake_session_factory[0].queue( + "get", + FakeResponse(content=json.dumps({"data": { + "id": workspace_id, "name": "My Workspace", "isPrivate": False, "ownerEmail": "owner@example.com", + }}).encode()), + FakeResponse(content=json.dumps({"data": [ + {"id": "00000000-0000-0000-0000-000000000002", "name": "Editor", "workspaceId": None, "permissions": []}, + {"id": "00000000-0000-0000-0000-000000000003", "name": "Custom", "workspaceId": workspace_id, + "permissions": []}, + ]}).encode()), + ) + + roles = hs.workspaces.get(workspace_id).roles + + params = fake_session_factory[0].last_kwargs["params"] + assert params["workspaceId"] == [workspace_id, "null"] + assert [role.name for role in roles] == ["Editor", "Custom"] + +@pytest.mark.parametrize( + "content, expected", + [ + (b'{"message": "Duplicate phenomenonTime found on this datastream."}', + "409 Client Error: Duplicate phenomenonTime found on this datastream."), + (b'{"detail": "Unauthorized"}', "409 Client Error: Unauthorized"), + (b'{"other": "value"}', '409 Client Error: {"other": "value"}'), + ], +) +def test_client_errors_surface_the_server_error_message(content, expected): + with pytest.raises(requests.HTTPError) as exc_info: + client_module.HydroServer._raise_for_hs_status(FakeResponse(409, content)) + + assert str(exc_info.value) == expected