From f9bc431ca8d2ac5e9705b53b50c35cf58f5d6a61 Mon Sep 17 00:00:00 2001 From: JeffreyChen Date: Fri, 25 Sep 2026 17:49:40 +0800 Subject: [PATCH 01/59] docs: point progress.md at the workspace items still open --- progress.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/progress.md b/progress.md index 77dfd04..e45ec4f 100644 --- a/progress.md +++ b/progress.md @@ -2,7 +2,7 @@ Outstanding work only. When an item is done, delete it in the same commit and add a `#done` entry to `docs/updates/` (format and query commands: `docs/updates/README.md`). No finished items, no history, no rules (rules live in `CLAUDE.md`). Item numbers (`#n`) are never reused. Tags: [DECIDE] needs the owner's decision, [BLOCKED] waits on something else, [UNVERIFIED] observed but not confirmed. -Cross-repo and workspace items live in `D:\Codes\progress.md` (relevant here: X-6, X-16). +Cross-repo and workspace items live in `D:\Codes\progress.md` (relevant here: X-12, X-13). ## Open From d8f4a14042602cad796205decac9b5e31b53972a Mon Sep 17 00:00:00 2001 From: JeffreyChen Date: Fri, 25 Sep 2026 18:09:24 +0800 Subject: [PATCH 02/59] Wait 7 days before Dependabot proposes a new release --- .github/dependabot.yml | 7 +++++++ docs/updates/2026-09.md | 9 +++++++++ docs/updates/README.md | 3 ++- tests/test_workflow_actions.py | 10 ++++++++++ 4 files changed, 28 insertions(+), 1 deletion(-) diff --git a/.github/dependabot.yml b/.github/dependabot.yml index 8058307..201868a 100644 --- a/.github/dependabot.yml +++ b/.github/dependabot.yml @@ -12,6 +12,11 @@ updates: target-branch: "dev" schedule: interval: "daily" + # A new release waits 7 days before Dependabot proposes it, so a + # compromised version is more likely to be found and yanked first. + # Dependabot's own default is 3 days. Security updates are not delayed. + cooldown: + default-days: 7 # Workflow actions are pinned to commit SHAs with the version as a comment # (test_workflow_actions.py); Dependabot moves both together. - package-ecosystem: "github-actions" @@ -19,3 +24,5 @@ updates: target-branch: "dev" schedule: interval: "weekly" + cooldown: + default-days: 7 diff --git a/docs/updates/2026-09.md b/docs/updates/2026-09.md index 68e06c7..c228019 100644 --- a/docs/updates/2026-09.md +++ b/docs/updates/2026-09.md @@ -152,3 +152,12 @@ Index and query commands: [README.md](README.md). New entries go at the end. - **Tests**: `tests/test_workflow_actions.py` gains a check that every checkout step sets `persist-credentials` explicitly. The previous workflows fail it. - **Result / numbers**: zizmor reports no warnings or errors for these workflows any more; the test file passes. - **Files**: `.github/workflows/ci-dev.yml`, `.github/workflows/ci-stable.yml`, `.github/workflows/publish.yml`, `tests/test_workflow_actions.py`. + +## U-20260925-01 · 2026-09-25 · Dependabot waits 7 days before proposing a new release · #ci #security #deps + +- **What**: `.github/dependabot.yml` sets `cooldown: default-days: 7` on all 2 update blocks. Dependabot now proposes a version only after it has been public for 7 days, so a compromised release is more likely to be found and yanked before a PR exists. Without the setting Dependabot waits 3 days (its default since 2026-07-14). Security updates ignore the cooldown and still open at once. `github-actions` accepts only `default-days`, so no per-semver values are set. +- **Why now**: zizmor 1.30.1 (`zizmor --offline .github`) reported `dependabot-cooldown` (medium) on every block; the workflow scan earlier this week only covered `.github/workflows`. +- **Tests**: `tests/test_workflow_actions.py` checks that every update block waits at least 7 days. The previous config fails it. +- **Result / numbers**: zizmor medium findings 0. The workflow-actions test file passes. +- **Sources**: https://docs.zizmor.sh/audits/#dependabot-cooldown ; https://docs.github.com/en/code-security/dependabot/working-with-dependabot/dependabot-options-reference#cooldown- ; https://github.blog/changelog/2026-07-14-dependabot-version-updates-introduce-default-package-cooldown/ +- **Files**: `.github/dependabot.yml`, `tests/test_workflow_actions.py`. diff --git a/docs/updates/README.md b/docs/updates/README.md index 1c49f1f..3b9f8a0 100644 --- a/docs/updates/README.md +++ b/docs/updates/README.md @@ -58,6 +58,7 @@ In the same commit: delete the item from `progress.md`, add a `#done` entry here | ID | Date | Title | Tags | Batch | |---|---|---|---|---| +| U-20260925-01 | 2026-09-25 | Dependabot waits 7 days before proposing a new release | #ci #security #deps | [2026-09](2026-09.md) | | U-20260924-02 | 2026-09-24 | Keep checkout credentials only in the job that pushes | #ci #security | [2026-09](2026-09.md) | | U-20260924-01 | 2026-09-24 | Move CI to Node 24 actions pinned by commit | #ci #security #deps | [2026-09](2026-09.md) | | U-20260923-11 | 2026-09-23 | cryptography and sphinx floors from Dependabot | #done #deps | [2026-09](2026-09.md) | @@ -81,4 +82,4 @@ In the same commit: delete the item from `progress.md`, add a `#done` entry here | File | Period | Entries | |---|---|---:| -| [2026-09.md](2026-09.md) | 2026-09 | 18 | +| [2026-09.md](2026-09.md) | 2026-09 | 19 | diff --git a/tests/test_workflow_actions.py b/tests/test_workflow_actions.py index d8fdba8..e9ab412 100644 --- a/tests/test_workflow_actions.py +++ b/tests/test_workflow_actions.py @@ -67,6 +67,16 @@ def test_dependabot_keeps_pins_current_on_dev(): assert all(re.search(r"^\s*target-branch:\s*\"dev\"", block, re.MULTILINE) for block in blocks) +def test_dependabot_waits_a_week_before_proposing_a_release(): + # A compromised release is usually found and yanked within days. Dependabot's + # own default wait is 3 days, and zizmor's dependabot-cooldown audit asks + # for 7. The wait never delays security updates. + text = (_ROOT / ".github" / "dependabot.yml").read_text(encoding="utf-8") + blocks = re.split(r"^\s*-\s*package-ecosystem:", text, flags=re.MULTILINE)[1:] + days = [re.search(r"^\s*default-days:\s*(\d+)", block, re.MULTILINE) for block in blocks] + assert blocks and all(match and int(match.group(1)) >= 7 for match in days) + + def _checkout_steps(path: Path) -> list[tuple[int, str]]: """Return ``(line number, step text)`` for each ``actions/checkout`` step.""" lines = path.read_text(encoding="utf-8").splitlines() From 0450d50a0d1cfbaad470556204514d9526cee218 Mon Sep 17 00:00:00 2001 From: JeffreyChen Date: Fri, 25 Sep 2026 18:29:23 +0800 Subject: [PATCH 03/59] docs: CLAUDE.md: look up SonarCloud/Codacy findings through their APIs with the env keys; never reveal a key --- CLAUDE.md | 1 + 1 file changed, 1 insertion(+) diff --git a/CLAUDE.md b/CLAUDE.md index ee8866f..090e4de 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -253,6 +253,7 @@ Workspace rule shared by every repository under `D:\Codes` (full text: `D:\Codes - Stage only the files that stage touched (`git add `, never `git add -A`), follow this file's commit-message rules, and never add AI attribution. - Committing is not pushing: push or open a PR only as this project's branch flow says or when asked. - **Commit and push frequently.** After each big feature — a self-contained stage that passes this project's checks — commit and push to the remote; do not pile up a large batch of work before committing or pushing. Smaller batches collide less with other sessions, let CI catch problems earlier, and are easier to revert. Follow this project's normal branch flow (usually `dev`). + - **SonarCloud / Codacy findings.** When a PR or commit fails a SonarCloud or Codacy check, look the findings up through their APIs instead of guessing. The keys are in environment variables: `SonarCloudToken` (SonarCloud, e.g. `curl -s -u "$SonarCloudToken:" "https://sonarcloud.io/api/issues/search?componentKeys=&pullRequest=&resolved=false"`) and `CODACY_PROJECT_TOKEN` (Codacy; for a public repository `https://app.codacy.com/api/v3/analysis/organizations/gh//repositories//pull-requests//issues?status=new` also answers without a key). **Never reveal a key or any personal credential while doing so**: refer to the variables by name only, never echo or print their values, and never put them in files, commit messages, PR or issue text, logs, or any output that leaves the machine. - **`progress.md`** (repository root, tracked) holds outstanding work only: no finished items, no history, no rules. - **`docs/updates/`** records finished work: one batch file per month (`YYYY-MM.md`), one entry per piece of work headed `## U-YYYYMMDD-NN · date · title · #tags`, and an index with query commands in `docs/updates/README.md`. When a `progress.md` item is done, delete it and add a `#done` entry plus its index row in the same commit. - **`architecture.md`** (repository root) is the short architecture overview: layers, entry points, main flows, extension points, cross-project boundaries. Update it in the same commit whenever a change alters any of those. From 86aecdad7086fe0de40aa01c7f8eeb59df874583 Mon Sep 17 00:00:00 2001 From: JeffreyChen Date: Fri, 25 Sep 2026 18:32:37 +0800 Subject: [PATCH 04/59] docs: CLAUDE.md: the Codacy project token only works for its own project; query public repos without it --- CLAUDE.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CLAUDE.md b/CLAUDE.md index 090e4de..ca1871a 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -253,7 +253,7 @@ Workspace rule shared by every repository under `D:\Codes` (full text: `D:\Codes - Stage only the files that stage touched (`git add `, never `git add -A`), follow this file's commit-message rules, and never add AI attribution. - Committing is not pushing: push or open a PR only as this project's branch flow says or when asked. - **Commit and push frequently.** After each big feature — a self-contained stage that passes this project's checks — commit and push to the remote; do not pile up a large batch of work before committing or pushing. Smaller batches collide less with other sessions, let CI catch problems earlier, and are easier to revert. Follow this project's normal branch flow (usually `dev`). - - **SonarCloud / Codacy findings.** When a PR or commit fails a SonarCloud or Codacy check, look the findings up through their APIs instead of guessing. The keys are in environment variables: `SonarCloudToken` (SonarCloud, e.g. `curl -s -u "$SonarCloudToken:" "https://sonarcloud.io/api/issues/search?componentKeys=&pullRequest=&resolved=false"`) and `CODACY_PROJECT_TOKEN` (Codacy; for a public repository `https://app.codacy.com/api/v3/analysis/organizations/gh//repositories//pull-requests//issues?status=new` also answers without a key). **Never reveal a key or any personal credential while doing so**: refer to the variables by name only, never echo or print their values, and never put them in files, commit messages, PR or issue text, logs, or any output that leaves the machine. + - **SonarCloud / Codacy findings.** When a PR or commit fails a SonarCloud or Codacy check, look the findings up through their APIs instead of guessing. The keys are in environment variables: `SonarCloudToken` (SonarCloud, e.g. `curl -s -u "$SonarCloudToken:" "https://sonarcloud.io/api/issues/search?componentKeys=&pullRequest=&resolved=false"`) and `CODACY_PROJECT_TOKEN` (a Codacy project token, valid only for its own project: any other repository answers "Bad credentials", so for a public repository query `https://app.codacy.com/api/v3/analysis/organizations/gh//repositories//pull-requests//issues?status=new` without a key). **Never reveal a key or any personal credential while doing so**: refer to the variables by name only, never echo or print their values, and never put them in files, commit messages, PR or issue text, logs, or any output that leaves the machine. - **`progress.md`** (repository root, tracked) holds outstanding work only: no finished items, no history, no rules. - **`docs/updates/`** records finished work: one batch file per month (`YYYY-MM.md`), one entry per piece of work headed `## U-YYYYMMDD-NN · date · title · #tags`, and an index with query commands in `docs/updates/README.md`. When a `progress.md` item is done, delete it and add a `#done` entry plus its index row in the same commit. - **`architecture.md`** (repository root) is the short architecture overview: layers, entry points, main flows, extension points, cross-project boundaries. Update it in the same commit whenever a change alters any of those. From b68e88d3f243e953ff6183165afaff35953f4ef0 Mon Sep 17 00:00:00 2001 From: JeffreyChen Date: Fri, 25 Sep 2026 19:19:50 +0800 Subject: [PATCH 05/59] Declare the MIT license as an SPDX expression in both channels --- dev.toml | 5 +++-- docs/updates/2026-09.md | 7 +++++++ docs/updates/README.md | 3 ++- stable.toml | 3 ++- 4 files changed, 14 insertions(+), 4 deletions(-) diff --git a/dev.toml b/dev.toml index 8e80e96..8d3f4a8 100644 --- a/dev.toml +++ b/dev.toml @@ -1,6 +1,6 @@ # Dev release metadata — copied to pyproject.toml by the dev publish workflow. [build-system] -requires = ["setuptools>=61.0"] +requires = ["setuptools>=77"] build-backend = "setuptools.build_meta" [project] @@ -12,7 +12,8 @@ authors = [ description = "JSON-driven file, Drive, and cloud automation framework (dev channel)." readme = { file = "README.md", content-type = "text/markdown" } requires-python = ">=3.10" -license = { text = "MIT" } +license = "MIT" +license-files = ["LICENSE"] dependencies = [ "google-api-python-client>=2.100.0", "google-auth-httplib2>=0.2.0", diff --git a/docs/updates/2026-09.md b/docs/updates/2026-09.md index c228019..3b9830d 100644 --- a/docs/updates/2026-09.md +++ b/docs/updates/2026-09.md @@ -161,3 +161,10 @@ Index and query commands: [README.md](README.md). New entries go at the end. - **Result / numbers**: zizmor medium findings 0. The workflow-actions test file passes. - **Sources**: https://docs.zizmor.sh/audits/#dependabot-cooldown ; https://docs.github.com/en/code-security/dependabot/working-with-dependabot/dependabot-options-reference#cooldown- ; https://github.blog/changelog/2026-07-14-dependabot-version-updates-introduce-default-package-cooldown/ - **Files**: `.github/dependabot.yml`, `tests/test_workflow_actions.py`. + +## U-20260925-02 · 2026-09-25 · License metadata uses the SPDX expression in both channels · #packaging + +- **What**: `dev.toml` declared `license = { text = "MIT" }`, a form setuptools deprecates (builds that still use it stop being supported after 2027-02-18), and both `stable.toml` and `dev.toml` required only `setuptools>=61.0` although `license-files` in `[project]` needs setuptools 77. Both files now declare `license = "MIT"` plus `license-files = ["LICENSE"]` and require `setuptools>=77`, so the stable and dev metadata match again apart from name, version and description. +- **Result / numbers**: wheels built from a clean export with each file copied to `pyproject.toml` (what the publish workflows do) carry `License-Expression: MIT` and `License-File: LICENSE`, with no deprecation warning. `python -m pytest tests/` 769 passed, 8 skipped. +- **Sources**: https://packaging.python.org/en/latest/guides/writing-pyproject-toml/#license ; setuptools build output. +- **Files**: `stable.toml`, `dev.toml`. diff --git a/docs/updates/README.md b/docs/updates/README.md index 3b9f8a0..9f92967 100644 --- a/docs/updates/README.md +++ b/docs/updates/README.md @@ -58,6 +58,7 @@ In the same commit: delete the item from `progress.md`, add a `#done` entry here | ID | Date | Title | Tags | Batch | |---|---|---|---|---| +| U-20260925-02 | 2026-09-25 | License metadata uses the SPDX expression in both channels | #packaging | [2026-09](2026-09.md) | | U-20260925-01 | 2026-09-25 | Dependabot waits 7 days before proposing a new release | #ci #security #deps | [2026-09](2026-09.md) | | U-20260924-02 | 2026-09-24 | Keep checkout credentials only in the job that pushes | #ci #security | [2026-09](2026-09.md) | | U-20260924-01 | 2026-09-24 | Move CI to Node 24 actions pinned by commit | #ci #security #deps | [2026-09](2026-09.md) | @@ -82,4 +83,4 @@ In the same commit: delete the item from `progress.md`, add a `#done` entry here | File | Period | Entries | |---|---|---:| -| [2026-09.md](2026-09.md) | 2026-09 | 19 | +| [2026-09.md](2026-09.md) | 2026-09 | 20 | diff --git a/stable.toml b/stable.toml index 02f55a0..658453f 100644 --- a/stable.toml +++ b/stable.toml @@ -1,6 +1,6 @@ # Stable release metadata — copied to pyproject.toml by the publish workflow. [build-system] -requires = ["setuptools>=61.0"] +requires = ["setuptools>=77"] build-backend = "setuptools.build_meta" [project] @@ -12,6 +12,7 @@ authors = [ description = "JSON-driven file, Drive, and cloud automation framework." readme = { file = "README.md", content-type = "text/markdown" } requires-python = ">=3.10" +license = "MIT" license-files = ["LICENSE"] dependencies = [ "google-api-python-client>=2.100.0", From 8a9ac1ebb3d7da0ed80ab44305899b4de2689d95 Mon Sep 17 00:00:00 2001 From: JeffreyChen Date: Fri, 25 Sep 2026 19:46:13 +0800 Subject: [PATCH 06/59] List Python 3.11 to 3.14 in the classifiers, as CI tests them --- dev.toml | 4 ++++ docs/updates/2026-09.md | 6 ++++++ docs/updates/README.md | 3 ++- stable.toml | 4 ++++ tests/test_python_classifiers.py | 35 ++++++++++++++++++++++++++++++++ 5 files changed, 51 insertions(+), 1 deletion(-) create mode 100644 tests/test_python_classifiers.py diff --git a/dev.toml b/dev.toml index 8d3f4a8..191763a 100644 --- a/dev.toml +++ b/dev.toml @@ -39,6 +39,10 @@ dependencies = [ ] classifiers = [ "Programming Language :: Python :: 3.10", + "Programming Language :: Python :: 3.11", + "Programming Language :: Python :: 3.12", + "Programming Language :: Python :: 3.13", + "Programming Language :: Python :: 3.14", "Development Status :: 2 - Pre-Alpha", "Environment :: Win32 (MS Windows)", "Environment :: MacOS X", diff --git a/docs/updates/2026-09.md b/docs/updates/2026-09.md index 3b9830d..51ae533 100644 --- a/docs/updates/2026-09.md +++ b/docs/updates/2026-09.md @@ -168,3 +168,9 @@ Index and query commands: [README.md](README.md). New entries go at the end. - **Result / numbers**: wheels built from a clean export with each file copied to `pyproject.toml` (what the publish workflows do) carry `License-Expression: MIT` and `License-File: LICENSE`, with no deprecation warning. `python -m pytest tests/` 769 passed, 8 skipped. - **Sources**: https://packaging.python.org/en/latest/guides/writing-pyproject-toml/#license ; setuptools build output. - **Files**: `stable.toml`, `dev.toml`. + +## U-20260925-03 · 2026-09-25 · Python classifiers list every version CI tests · #packaging #tests + +- **What**: the package metadata (`stable.toml`, `dev.toml`) listed only `Programming Language :: Python :: 3.10`, while CI runs every test on 3.10, 3.11, 3.12, 3.13 and 3.14, so PyPI showed the package as 3.10-only. The classifiers for 3.11 to 3.14 are added to both files. +- **Tests**: new `tests/test_python_classifiers.py` checks that each metadata file's Python classifiers are exactly the versions in the workflows' `python-version` matrices, so adding a version to CI without the classifier (or the other way round) fails. The old metadata fails it. +- **Files**: `stable.toml`, `dev.toml`, `tests/test_python_classifiers.py` (new). diff --git a/docs/updates/README.md b/docs/updates/README.md index 9f92967..f183649 100644 --- a/docs/updates/README.md +++ b/docs/updates/README.md @@ -58,6 +58,7 @@ In the same commit: delete the item from `progress.md`, add a `#done` entry here | ID | Date | Title | Tags | Batch | |---|---|---|---|---| +| U-20260925-03 | 2026-09-25 | Python classifiers list every version CI tests | #packaging #tests | [2026-09](2026-09.md) | | U-20260925-02 | 2026-09-25 | License metadata uses the SPDX expression in both channels | #packaging | [2026-09](2026-09.md) | | U-20260925-01 | 2026-09-25 | Dependabot waits 7 days before proposing a new release | #ci #security #deps | [2026-09](2026-09.md) | | U-20260924-02 | 2026-09-24 | Keep checkout credentials only in the job that pushes | #ci #security | [2026-09](2026-09.md) | @@ -83,4 +84,4 @@ In the same commit: delete the item from `progress.md`, add a `#done` entry here | File | Period | Entries | |---|---|---:| -| [2026-09.md](2026-09.md) | 2026-09 | 20 | +| [2026-09.md](2026-09.md) | 2026-09 | 21 | diff --git a/stable.toml b/stable.toml index 658453f..c2d1456 100644 --- a/stable.toml +++ b/stable.toml @@ -39,6 +39,10 @@ dependencies = [ ] classifiers = [ "Programming Language :: Python :: 3.10", + "Programming Language :: Python :: 3.11", + "Programming Language :: Python :: 3.12", + "Programming Language :: Python :: 3.13", + "Programming Language :: Python :: 3.14", "Development Status :: 2 - Pre-Alpha", "Environment :: Win32 (MS Windows)", "Environment :: MacOS X", diff --git a/tests/test_python_classifiers.py b/tests/test_python_classifiers.py new file mode 100644 index 0000000..b630e1e --- /dev/null +++ b/tests/test_python_classifiers.py @@ -0,0 +1,35 @@ +"""The PyPI classifiers name exactly the Python versions CI tests. + +The package metadata listed only 3.10 while CI ran 3.10 to 3.14, so PyPI showed +the package as 3.10-only. A workflow matrix ``python-version: ["3.10", "3.11"]`` +needs ``Programming Language :: Python :: 3.10`` and ``:: 3.11``, nothing more. +""" +from __future__ import annotations + +import re +from pathlib import Path + +import pytest + +REPO_ROOT = Path(__file__).resolve().parents[1] +_MATRIX = re.compile(r"python-version:\s*\[([^\]]*)\]") +_CLASSIFIER = re.compile(r"Programming Language :: Python :: (3\.\d+)") + + +def _tested_versions() -> set[str]: + """Return every Python version named in a workflow's ``python-version`` matrix.""" + versions: set[str] = set() + for workflow in (REPO_ROOT / ".github" / "workflows").glob("*.yml"): + for match in _MATRIX.finditer(workflow.read_text(encoding="utf-8")): + versions.update(re.findall(r"3\.\d+", match.group(1))) + return versions + + +def test_ci_has_a_python_matrix(): + assert _tested_versions() + + +@pytest.mark.parametrize("metadata", ["stable.toml", "dev.toml"]) +def test_classifiers_match_the_ci_matrix(metadata): + declared = set(_CLASSIFIER.findall((REPO_ROOT / metadata).read_text(encoding="utf-8"))) + assert declared == _tested_versions() From fec7fdfff6dd7b3c393ed7b00ba4b42ee8f39599 Mon Sep 17 00:00:00 2001 From: JeffreyChen Date: Fri, 25 Sep 2026 19:54:43 +0800 Subject: [PATCH 07/59] Format the classifier test with ruff --- tests/test_python_classifiers.py | 1 + 1 file changed, 1 insertion(+) diff --git a/tests/test_python_classifiers.py b/tests/test_python_classifiers.py index b630e1e..7343d04 100644 --- a/tests/test_python_classifiers.py +++ b/tests/test_python_classifiers.py @@ -4,6 +4,7 @@ the package as 3.10-only. A workflow matrix ``python-version: ["3.10", "3.11"]`` needs ``Programming Language :: Python :: 3.10`` and ``:: 3.11``, nothing more. """ + from __future__ import annotations import re From fe50703530d951c72a99b4b36d27a697be300089 Mon Sep 17 00:00:00 2001 From: JeffreyChen Date: Thu, 1 Oct 2026 08:28:26 +0800 Subject: [PATCH 08/59] Give every workflow job a timeout --- .github/workflows/ci-dev.yml | 2 ++ .github/workflows/ci-stable.yml | 2 ++ .github/workflows/publish.yml | 1 + docs/updates/2026-10.md | 11 +++++++++++ docs/updates/README.md | 2 ++ tests/test_workflow_actions.py | 21 +++++++++++++++++++++ 6 files changed, 39 insertions(+) create mode 100644 docs/updates/2026-10.md diff --git a/.github/workflows/ci-dev.yml b/.github/workflows/ci-dev.yml index e1a005d..b2d2397 100644 --- a/.github/workflows/ci-dev.yml +++ b/.github/workflows/ci-dev.yml @@ -14,6 +14,7 @@ permissions: jobs: lint: runs-on: ubuntu-latest + timeout-minutes: 15 # about 3x the slowest recent run, at least 15 steps: - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: @@ -36,6 +37,7 @@ jobs: pytest: needs: lint runs-on: windows-latest + timeout-minutes: 15 # about 3x the slowest recent run, at least 15 strategy: fail-fast: false matrix: diff --git a/.github/workflows/ci-stable.yml b/.github/workflows/ci-stable.yml index 56840d4..c705b0a 100644 --- a/.github/workflows/ci-stable.yml +++ b/.github/workflows/ci-stable.yml @@ -14,6 +14,7 @@ permissions: jobs: lint: runs-on: ubuntu-latest + timeout-minutes: 15 # about 3x the slowest recent run, at least 15 steps: - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: @@ -36,6 +37,7 @@ jobs: pytest: needs: lint runs-on: windows-latest + timeout-minutes: 15 # about 3x the slowest recent run, at least 15 strategy: fail-fast: false matrix: diff --git a/.github/workflows/publish.yml b/.github/workflows/publish.yml index d4e47f7..1b658e1 100644 --- a/.github/workflows/publish.yml +++ b/.github/workflows/publish.yml @@ -14,6 +14,7 @@ concurrency: jobs: publish: runs-on: ubuntu-latest + timeout-minutes: 30 # about 3x the slowest recent run, at least 15 if: "!contains(github.event.head_commit.message, 'chore: bump version')" steps: - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 diff --git a/docs/updates/2026-10.md b/docs/updates/2026-10.md new file mode 100644 index 0000000..630b2da --- /dev/null +++ b/docs/updates/2026-10.md @@ -0,0 +1,11 @@ +# 2026-10 update log + +Index and query commands: [README.md](README.md). New entries go at the end. + +--- + +## U-20261001-01 · 2026-10-01 · Every workflow job has a timeout · #ci #tests + +- **What**: every job that runs on a runner sets `timeout-minutes`, about three times its slowest recent run and at least 15 minutes. Without it a hung job runs for GitHub's default 360 minutes and holds the runner. Timeouts: `ci-dev.yml`: `lint` 15, `pytest` 15; `ci-stable.yml`: `lint` 15, `pytest` 15; `publish.yml`: `publish` 30. +- **Tests**: `tests/test_workflow_actions.py` `test_every_job_has_a_timeout` fails for any job with `runs-on` and no `timeout-minutes`; the previous workflows fail it. +- **Files**: `.github/workflows/ci-dev.yml`, `.github/workflows/ci-stable.yml`, `.github/workflows/publish.yml`, `tests/test_workflow_actions.py`. diff --git a/docs/updates/README.md b/docs/updates/README.md index f183649..1376e3f 100644 --- a/docs/updates/README.md +++ b/docs/updates/README.md @@ -58,6 +58,7 @@ In the same commit: delete the item from `progress.md`, add a `#done` entry here | ID | Date | Title | Tags | Batch | |---|---|---|---|---| +| U-20261001-01 | 2026-10-01 | Every workflow job has a timeout | #ci #tests | [2026-10](2026-10.md) | | U-20260925-03 | 2026-09-25 | Python classifiers list every version CI tests | #packaging #tests | [2026-09](2026-09.md) | | U-20260925-02 | 2026-09-25 | License metadata uses the SPDX expression in both channels | #packaging | [2026-09](2026-09.md) | | U-20260925-01 | 2026-09-25 | Dependabot waits 7 days before proposing a new release | #ci #security #deps | [2026-09](2026-09.md) | @@ -84,4 +85,5 @@ In the same commit: delete the item from `progress.md`, add a `#done` entry here | File | Period | Entries | |---|---|---:| +| [2026-10.md](2026-10.md) | 2026-10 | 1 | | [2026-09.md](2026-09.md) | 2026-09 | 21 | diff --git a/tests/test_workflow_actions.py b/tests/test_workflow_actions.py index e9ab412..d933ea9 100644 --- a/tests/test_workflow_actions.py +++ b/tests/test_workflow_actions.py @@ -106,3 +106,24 @@ def test_every_checkout_decides_on_persisted_credentials(workflow): if not re.search(r"^\s*persist-credentials:\s*(true|false)\b", step, re.MULTILINE) ] assert bad == [] + + +_JOB_HEAD = re.compile(r"^ [A-Za-z0-9_-]+:\s*(#.*)?$") + + +def _jobs(path: Path) -> list[tuple[str, str]]: + """Return ``(job id, job text)`` for each job under ``jobs:`` in a workflow.""" + lines = path.read_text(encoding="utf-8").splitlines() + start = next(i for i, line in enumerate(lines) if re.match(r"^jobs:\s*(#.*)?$", line)) + heads = [i for i in range(start + 1, len(lines)) if _JOB_HEAD.match(lines[i])] + ends = [*heads[1:], len(lines)] + return [(lines[i].strip().rstrip(":"), "\n".join(lines[i:end])) for i, end in zip(heads, ends)] + + +@pytest.mark.parametrize("workflow", _WORKFLOWS, ids=lambda p: p.name) +def test_every_job_has_a_timeout(workflow): + # Without timeout-minutes a hung job runs for GitHub's default six hours. + # Each job sets about three times its slowest recent run, at least 15 minutes. + bad = [name for name, body in _jobs(workflow) + if "runs-on:" in body and not re.search(r"^\s*timeout-minutes:", body, re.MULTILINE)] + assert bad == [] From e35e1009ba4c6df49d91d91850c722f9acb9c7e8 Mon Sep 17 00:00:00 2001 From: JeffreyChen Date: Thu, 1 Oct 2026 08:43:55 +0800 Subject: [PATCH 09/59] Pass ruff B905 in the workflow-timeout test --- docs/updates/2026-10.md | 6 ++++++ docs/updates/README.md | 3 ++- tests/test_workflow_actions.py | 12 +++++++++--- 3 files changed, 17 insertions(+), 4 deletions(-) diff --git a/docs/updates/2026-10.md b/docs/updates/2026-10.md index 630b2da..22e654a 100644 --- a/docs/updates/2026-10.md +++ b/docs/updates/2026-10.md @@ -9,3 +9,9 @@ Index and query commands: [README.md](README.md). New entries go at the end. - **What**: every job that runs on a runner sets `timeout-minutes`, about three times its slowest recent run and at least 15 minutes. Without it a hung job runs for GitHub's default 360 minutes and holds the runner. Timeouts: `ci-dev.yml`: `lint` 15, `pytest` 15; `ci-stable.yml`: `lint` 15, `pytest` 15; `publish.yml`: `publish` 30. - **Tests**: `tests/test_workflow_actions.py` `test_every_job_has_a_timeout` fails for any job with `runs-on` and no `timeout-minutes`; the previous workflows fail it. - **Files**: `.github/workflows/ci-dev.yml`, `.github/workflows/ci-stable.yml`, `.github/workflows/publish.yml`, `tests/test_workflow_actions.py`. + +## U-20261001-02 · 2026-10-01 · Workflow-timeout test failed CI lint (ruff B905) · #incident #ci + +- **What happened**: the test added in U-20261001-01 put `zip()` without `strict=` in its `_jobs` helper, so CI run 36796361197 failed in the `lint` job (`ruff check`) with ruff B905. +- **Fix**: `zip(heads, ends, strict=True)`, and the file is formatted with `ruff format`, which `ci-dev.yml` / `ci-stable.yml` also check. The workflow test still passes, and the repository's CI lint commands pass locally. +- **Files**: `tests/test_workflow_actions.py`. diff --git a/docs/updates/README.md b/docs/updates/README.md index 1376e3f..83035bc 100644 --- a/docs/updates/README.md +++ b/docs/updates/README.md @@ -58,6 +58,7 @@ In the same commit: delete the item from `progress.md`, add a `#done` entry here | ID | Date | Title | Tags | Batch | |---|---|---|---|---| +| U-20261001-02 | 2026-10-01 | Workflow-timeout test failed CI lint (ruff B905) | #incident #ci | [2026-10](2026-10.md) | | U-20261001-01 | 2026-10-01 | Every workflow job has a timeout | #ci #tests | [2026-10](2026-10.md) | | U-20260925-03 | 2026-09-25 | Python classifiers list every version CI tests | #packaging #tests | [2026-09](2026-09.md) | | U-20260925-02 | 2026-09-25 | License metadata uses the SPDX expression in both channels | #packaging | [2026-09](2026-09.md) | @@ -85,5 +86,5 @@ In the same commit: delete the item from `progress.md`, add a `#done` entry here | File | Period | Entries | |---|---|---:| -| [2026-10.md](2026-10.md) | 2026-10 | 1 | +| [2026-10.md](2026-10.md) | 2026-10 | 2 | | [2026-09.md](2026-09.md) | 2026-09 | 21 | diff --git a/tests/test_workflow_actions.py b/tests/test_workflow_actions.py index d933ea9..992cdc7 100644 --- a/tests/test_workflow_actions.py +++ b/tests/test_workflow_actions.py @@ -117,13 +117,19 @@ def _jobs(path: Path) -> list[tuple[str, str]]: start = next(i for i, line in enumerate(lines) if re.match(r"^jobs:\s*(#.*)?$", line)) heads = [i for i in range(start + 1, len(lines)) if _JOB_HEAD.match(lines[i])] ends = [*heads[1:], len(lines)] - return [(lines[i].strip().rstrip(":"), "\n".join(lines[i:end])) for i, end in zip(heads, ends)] + return [ + (lines[i].strip().rstrip(":"), "\n".join(lines[i:end])) + for i, end in zip(heads, ends, strict=True) + ] @pytest.mark.parametrize("workflow", _WORKFLOWS, ids=lambda p: p.name) def test_every_job_has_a_timeout(workflow): # Without timeout-minutes a hung job runs for GitHub's default six hours. # Each job sets about three times its slowest recent run, at least 15 minutes. - bad = [name for name, body in _jobs(workflow) - if "runs-on:" in body and not re.search(r"^\s*timeout-minutes:", body, re.MULTILINE)] + bad = [ + name + for name, body in _jobs(workflow) + if "runs-on:" in body and not re.search(r"^\s*timeout-minutes:", body, re.MULTILINE) + ] assert bad == [] From 5eefb5342dc29db6b88c83d5080b47ba13c22ac1 Mon Sep 17 00:00:00 2001 From: JeffreyChen Date: Thu, 1 Oct 2026 12:51:27 +0800 Subject: [PATCH 10/59] refactor: build the registry, executor pipeline and helpers on je_action_core --- .github/workflows/ci-dev.yml | 6 ++ .github/workflows/ci-stable.yml | 6 ++ README.md | 3 +- README.zh-CN.md | 3 +- README.zh-TW.md | 3 +- architecture.md | 21 ++++- automation_file/core/action_executor.py | 93 +++++++++++------------ automation_file/core/action_registry.py | 53 +++---------- automation_file/core/callback_executor.py | 62 +++++---------- automation_file/core/json_store.py | 47 ++++++------ automation_file/core/package_loader.py | 51 +++++-------- dev.toml | 1 + docs/updates/2026-10.md | 36 +++++++++ docs/updates/README.md | 3 +- progress.md | 1 + stable.toml | 1 + 16 files changed, 186 insertions(+), 204 deletions(-) diff --git a/.github/workflows/ci-dev.yml b/.github/workflows/ci-dev.yml index b2d2397..ccbef74 100644 --- a/.github/workflows/ci-dev.yml +++ b/.github/workflows/ci-dev.yml @@ -26,6 +26,9 @@ jobs: - name: Install tooling run: | python -m pip install --upgrade pip + # je_action_core is not on PyPI yet (ActionCore progress.md #1): install it from GitHub at a + # fixed commit first, so mypy sees its types and `pip install -e .` finds it installed. + pip install --no-deps "je_action_core @ git+https://github.com/Integration-Automation/ActionCore@19bfe0a7c907af00c6a59af3de04448999b42e44" pip install ruff mypy - name: Ruff check run: ruff check automation_file tests @@ -54,6 +57,9 @@ jobs: - name: Install dependencies run: | python -m pip install --upgrade pip wheel + # je_action_core is not on PyPI yet (ActionCore progress.md #1): install it from GitHub at a + # fixed commit first, so mypy sees its types and `pip install -e .` finds it installed. + pip install --no-deps "je_action_core @ git+https://github.com/Integration-Automation/ActionCore@19bfe0a7c907af00c6a59af3de04448999b42e44" Copy-Item dev.toml pyproject.toml -Force pip install -e . pip install pytest pytest-cov diff --git a/.github/workflows/ci-stable.yml b/.github/workflows/ci-stable.yml index c705b0a..e29fa90 100644 --- a/.github/workflows/ci-stable.yml +++ b/.github/workflows/ci-stable.yml @@ -26,6 +26,9 @@ jobs: - name: Install tooling run: | python -m pip install --upgrade pip + # je_action_core is not on PyPI yet (ActionCore progress.md #1): install it from GitHub at a + # fixed commit first, so mypy sees its types and `pip install -e .` finds it installed. + pip install --no-deps "je_action_core @ git+https://github.com/Integration-Automation/ActionCore@19bfe0a7c907af00c6a59af3de04448999b42e44" pip install ruff mypy - name: Ruff check run: ruff check automation_file tests @@ -54,6 +57,9 @@ jobs: - name: Install dependencies run: | python -m pip install --upgrade pip wheel + # je_action_core is not on PyPI yet (ActionCore progress.md #1): install it from GitHub at a + # fixed commit first, so mypy sees its types and `pip install -e .` finds it installed. + pip install --no-deps "je_action_core @ git+https://github.com/Integration-Automation/ActionCore@19bfe0a7c907af00c6a59af3de04448999b42e44" Copy-Item stable.toml pyproject.toml -Force pip install -e . pip install pytest pytest-cov diff --git a/README.md b/README.md index 2e6497c..ed85aee 100644 --- a/README.md +++ b/README.md @@ -310,7 +310,8 @@ Requirements: `tqdm`, `boto3`, `azure-storage-blob`, `dropbox`, `paramiko`, `msal`, `boxsdk`, `PySide6`, `watchdog`, `cryptography`, `prometheus_client`, `defusedxml`, - `PyYAML`, `pyarrow`, `opentelemetry-api`, `opentelemetry-sdk` + `PyYAML`, `pyarrow`, `opentelemetry-api`, `opentelemetry-sdk`, `je_action_core` (the action executor + shared with APITestka, LoadDensity and MailThunder) ## Usage diff --git a/README.zh-CN.md b/README.zh-CN.md index 9ff5730..ce4e224 100644 --- a/README.zh-CN.md +++ b/README.zh-CN.md @@ -308,7 +308,8 @@ pip install "automation_file[dev]" # ruff, mypy, pre-commit, pytest-cov, b `tqdm`、`boto3`、`azure-storage-blob`、`dropbox`、 `paramiko`、`msal`、`boxsdk`、`PySide6`、 `watchdog`、`cryptography`、`prometheus_client`、`defusedxml`、 - `PyYAML`、`pyarrow`、`opentelemetry-api`、`opentelemetry-sdk` + `PyYAML`、`pyarrow`、`opentelemetry-api`、`opentelemetry-sdk`、`je_action_core`(与 APITestka、 + LoadDensity、MailThunder 共用的 action 执行器) ## 使用方式 diff --git a/README.zh-TW.md b/README.zh-TW.md index 965dc3a..49352f4 100644 --- a/README.zh-TW.md +++ b/README.zh-TW.md @@ -308,7 +308,8 @@ pip install "automation_file[dev]" # ruff, mypy, pre-commit, pytest-cov, b `tqdm`、`boto3`、`azure-storage-blob`、`dropbox`、 `paramiko`、`msal`、`boxsdk`、`PySide6`、 `watchdog`、`cryptography`、`prometheus_client`、`defusedxml`、 - `PyYAML`、`pyarrow`、`opentelemetry-api`、`opentelemetry-sdk` + `PyYAML`、`pyarrow`、`opentelemetry-api`、`opentelemetry-sdk`、`je_action_core`(與 APITestka、 + LoadDensity、MailThunder 共用的 action 執行器) ## 使用方式 diff --git a/architecture.md b/architecture.md index 03ef11e..fae8037 100644 --- a/architecture.md +++ b/architecture.md @@ -17,7 +17,7 @@ the CLI, over loopback TCP or HTTP servers, as MCP tools, or from the PySide6 GU | --- | --- | | `automation_file/__init__.py` | Public facade (`__all__`). Wires the shared `executor`, `callback_executor` and `package_manager` over one registry. `launch_ui` is loaded lazily through `__getattr__` | | `automation_file/__main__.py` | CLI: legacy flags plus subcommands | -| `automation_file/core/` | Engine: `action_registry.py` (`ActionRegistry`, `build_default_registry`), `action_executor.py` (`ActionExecutor`, shared `executor`), `callback_executor.py`, `package_loader.py`, `plugins.py`, `dag_executor.py`, `action_queue.py`, `json_store.py`, `substitution.py`. Also cross-cutting helpers: `retry`, `quota`, `rate_limit`, `circuit_breaker`, `file_lock`, `sqlite_lock`, `checksum`, `manifest`, `crypto`, `secrets`, `config`, `config_watcher`, `audit`, `metrics`, `tracing`, `progress`, `fim`, `content_store` | +| `automation_file/core/` | Engine, on je_action_core: `action_registry.py` (`ActionRegistry`, a `CommandRegistry`; `build_default_registry`), `action_executor.py` (`ActionExecutor`, an `ActionExecutor` with strict actions, indexed records and the dry-run, validate, substitute and parallel extras; shared `executor`), `callback_executor.py`, `package_loader.py`, `plugins.py`, `dag_executor.py`, `action_queue.py`, `json_store.py`, `substitution.py`. Also cross-cutting helpers: `retry`, `quota`, `rate_limit`, `circuit_breaker`, `file_lock`, `sqlite_lock`, `checksum`, `manifest`, `crypto`, `secrets`, `config`, `config_watcher`, `audit`, `metrics`, `tracing`, `progress`, `fim`, `content_store` | | `automation_file/local/` | Local strategy modules: file, dir, zip, tar and archive ops, sync, diff, text/JSON/data edits, templates, versioning, trash, `shell_ops` (argv-only subprocess), conditional branches. `safe_paths.py` guards against path traversal | | `automation_file/remote/` | `url_validator.py` (SSRF guard), `http_download.py`, `cross_backend.py`, `fsspec_bridge.py`. One subpackage per backend: `google_drive/`, `s3/`, `azure_blob/`, `dropbox_api/`, `sftp/`, `ftp/`, `onedrive/`, `box/`, each with `client.py`, `*_ops.py` and `register__ops`. `smb/` and `webdav/` have a client only | | `automation_file/server/` | `tcp_server.py`, `http_server.py`, `mcp_server.py`, `web_ui.py`, `metrics_server.py`, `action_acl.py` (`ActionACL`), `network_guards.py` (`ensure_loopback`) | @@ -66,7 +66,7 @@ the CLI, over loopback TCP or HTTP servers, as MCP tools, or from the PySide6 GU JSON file / --execute_str / Python → ActionExecutor.execute_action(list|dict, validate_first, dry_run, substitute) → _coerce (list or {"auto_control": [...]}) → _execute_event → registry.resolve(name) → FA_* callable in local/ | remote/ | utils/ | core/ (inside tracing.action_span) - → {"execute: ": return value | repr(error)} (one failure never aborts the batch) + → {"execute[]: ": return value | repr(error)} (one failure never aborts the batch) ``` **Remote transports** @@ -124,8 +124,21 @@ ActionExecutor() → build_default_registry(): local + http + utils + drive comm metrics 9945) on their defaults. - **Wire format**: TCP replies end with the same `Return_Data_Over_JE` terminator as the sibling servers. - **Builtins policy**: the default registry contains no Python builtins; only `PackageLoader` can add - them. In the siblings, APITestka uses an explicit allowlist, LoadDensity a `_UNSAFE_BUILTINS` - blacklist, and MailThunder registers every builtin (known gap). + them. In the siblings, APITestka registers none, and LoadDensity, MailThunder and WebRunner register the + same `SAFE_BUILTINS` allowlist. +- **ActionCore (this repo depends on it)**: `je_action_core` (Integration-Automation/ActionCore) holds the registry, + executor pipeline, package loader, callback executor and JSON files. FileAutomation configures them as follows: + - **registry**: accepts any callable; a refused one raises `AddCommandException(" is not callable")`; + - **executor**: `StrictActionParser` (its messages are this repo's), `execute[]: ` record keys, + document key `auto_control`, its list messages, and the tracing span through `invoke`; + - **package loader**: `_` names, import errors logged, gate off, the count returned through + `check_and_add`; + - **callback executor**: strict checks, errors raised; + - **JSON files**: `JSONDecodeError` / `OSError` wrapped on read, `OSError` / `TypeError` on write. + + The extras (dry run, validate, substitute, parallel, metrics) and the TCP / HTTP servers stay here. Until the + package is on PyPI, the CI installs it from GitHub at a fixed commit (`progress.md` #7). ActionCore lists + FileAutomation in its own §6. ## 7. Design constraints diff --git a/automation_file/core/action_executor.py b/automation_file/core/action_executor.py index 446882f..460be8a 100644 --- a/automation_file/core/action_executor.py +++ b/automation_file/core/action_executor.py @@ -20,6 +20,18 @@ from concurrent.futures import ThreadPoolExecutor from typing import Any +from je_action_core import ( + ActionExecutor as _CoreActionExecutor, +) +from je_action_core import ( + ActionListRules, + BoundAction, + ExecutorSettings, + ParsedAction, + StrictActionParser, + indexed_record_key, +) + from automation_file.core.action_registry import ActionRegistry, build_default_registry from automation_file.core.json_store import read_action_json from automation_file.core.metrics import record_action @@ -27,12 +39,29 @@ from automation_file.exceptions import ExecuteActionException, ValidationException from automation_file.logging_config import file_automation_logger +_PARSER = StrictActionParser(error=ExecuteActionException) +# The document key FileAutomation inherited from AutoControl, and its list errors. +_SETTINGS = ExecutorSettings( + rules=ActionListRules( + "auto_control", + error=ExecuteActionException, + missing_message="dict action list missing {key!r}", + not_list_message="action_list must be list, got {type}", + empty_message="action_list is empty", + ), + parser=_PARSER, + read_json=read_action_json, + record_key=indexed_record_key, +) + + +class ActionExecutor(_CoreActionExecutor): + """Execute named actions resolved through an :class:`ActionRegistry` (je_action_core's executor).""" -class ActionExecutor: - """Execute named actions resolved through an :class:`ActionRegistry`.""" + registry: ActionRegistry def __init__(self, registry: ActionRegistry | None = None) -> None: - self.registry: ActionRegistry = registry or build_default_registry() + super().__init__(_SETTINGS, registry or build_default_registry()) self.registry.register_many( { "FA_execute_action": self.execute_action, @@ -43,39 +72,16 @@ def __init__(self, registry: ActionRegistry | None = None) -> None: ) # Template-method: single action ------------------------------------ - def _execute_event(self, action: list) -> Any: + def invoke(self, bound: BoundAction) -> Any: + """Call the bound command inside a tracing span named after the action.""" from automation_file.core.tracing import action_span - name, payload_kind, payload = self._parse_action(action) - command = self.registry.resolve(name) - if command is None: - raise ExecuteActionException(f"unknown action: {name!r}") - with action_span(name): - if payload_kind == "none": - return command() - if payload_kind == "kwargs": - return command(**payload) - return command(*payload) + with action_span(bound.name): + return super().invoke(bound) @staticmethod - def _parse_action(action: list) -> tuple[str, str, Any]: - if not isinstance(action, list) or not action: - raise ExecuteActionException(f"malformed action: {action!r}") - name = action[0] - if not isinstance(name, str): - raise ExecuteActionException(f"action name must be str: {action!r}") - if len(action) == 1: - return name, "none", None - if len(action) == 2: - payload = action[1] - if isinstance(payload, dict): - return name, "kwargs", payload - if isinstance(payload, list): - return name, "args", payload - raise ExecuteActionException( - f"action {name!r} payload must be dict or list, got {type(payload).__name__}" - ) - raise ExecuteActionException(f"action has too many elements: {action!r}") + def _parse_action(action: list) -> ParsedAction: + return _PARSER.parse(action) # Public API -------------------------------------------------------- def validate(self, action_list: list | Mapping[str, Any]) -> list[str]: @@ -147,14 +153,10 @@ def execute_action_parallel( results[f"execute[{index}]: {action}"] = future.result() return results - def execute_files(self, execute_files_list: list[str]) -> list[dict[str, Any]]: - """Execute every JSON file's action list and return their results.""" - return [self.execute_action(read_action_json(path)) for path in execute_files_list] - def add_command_to_executor(self, command_dict: Mapping[str, Any]) -> None: """Register every ``name -> callable`` pair (Registry facade).""" file_automation_logger.info("add_command_to_executor: %s", list(command_dict.keys())) - self.registry.register_many(command_dict) + super().add_command_to_executor(command_dict) # Internals --------------------------------------------------------- def _run_one(self, action: list, dry_run: bool, display: list | None = None) -> Any: @@ -196,20 +198,11 @@ def _run_dry(self, action: list, display: list | None = None) -> Any: ) return f"dry_run:{name}" - @staticmethod - def _coerce(action_list: list | Mapping[str, Any]) -> list: - if isinstance(action_list, Mapping): - nested = action_list.get("auto_control") - if nested is None: - raise ExecuteActionException("dict action list missing 'auto_control'") - action_list = nested - if not isinstance(action_list, list): - raise ExecuteActionException( - f"action_list must be list, got {type(action_list).__name__}" - ) - if not action_list: + def _coerce(self, action_list: list | Mapping[str, Any]) -> list: + actions = self.settings.rules.extract(action_list) + if actions is None: # EmptyListPolicy.RAISE: extract raises instead raise ExecuteActionException("action_list is empty") - return action_list + return actions def _safe_action_name(action: Any) -> str: diff --git a/automation_file/core/action_registry.py b/automation_file/core/action_registry.py index c474890..80fd705 100644 --- a/automation_file/core/action_registry.py +++ b/automation_file/core/action_registry.py @@ -8,61 +8,26 @@ from __future__ import annotations -from collections.abc import Callable, Iterable, Iterator, Mapping +from collections.abc import Callable, Mapping from typing import Any +from je_action_core import CommandPolicy, CommandRegistry + from automation_file.exceptions import AddCommandException from automation_file.logging_config import file_automation_logger Command = Callable[..., Any] -class ActionRegistry: - """Mapping of action name -> callable.""" - - def __init__(self, initial: Mapping[str, Command] | None = None) -> None: - self._commands: dict[str, Command] = {} - if initial: - for name, command in initial.items(): - self.register(name, command) - - def register(self, name: str, command: Command) -> None: - """Add or overwrite a command. Raises if ``command`` is not callable.""" - if not callable(command): - raise AddCommandException(f"{name!r} is not callable") - self._commands[name] = command - - def register_many(self, mapping: Mapping[str, Command]) -> None: - """Register every ``name -> command`` pair in ``mapping``.""" - for name, command in mapping.items(): - self.register(name, command) - - def update(self, mapping: Mapping[str, Command]) -> None: - """Alias for :meth:`register_many` (dict-compatible).""" - self.register_many(mapping) +def _not_callable(name: str) -> AddCommandException: + return AddCommandException(f"{name!r} is not callable") - def unregister(self, name: str) -> None: - self._commands.pop(name, None) - def resolve(self, name: str) -> Command | None: - return self._commands.get(name) +class ActionRegistry(CommandRegistry): + """Mapping of action name -> callable: je_action_core's registry, accepting any callable.""" - def __contains__(self, name: object) -> bool: - return isinstance(name, str) and name in self._commands - - def __len__(self) -> int: - return len(self._commands) - - def __iter__(self) -> Iterator[str]: - return iter(self._commands) - - def names(self) -> Iterable[str]: - return self._commands.keys() - - @property - def event_dict(self) -> dict[str, Command]: - """Backwards-compatible view used by older ``package_manager`` style code.""" - return self._commands + def __init__(self, initial: Mapping[str, Command] | None = None) -> None: + super().__init__(initial, policy=CommandPolicy.ANY_CALLABLE, rejection=_not_callable) def _local_commands() -> dict[str, Command]: diff --git a/automation_file/core/callback_executor.py b/automation_file/core/callback_executor.py index 459a6a6..414e835 100644 --- a/automation_file/core/callback_executor.py +++ b/automation_file/core/callback_executor.py @@ -2,62 +2,34 @@ Implements the "do X then do Y" flow many automation JSON files want. The registry is shared with :class:`ActionExecutor`, so adding a command to one -adds it to the other. +adds it to the other (je_action_core's callback executor, strict checks). """ from __future__ import annotations -from collections.abc import Callable, Mapping -from typing import Any +from je_action_core import ( + CallbackErrorPolicy, + CallbackFunctionExecutor, + CallbackSettings, + CallbackStyle, +) from automation_file.core.action_registry import ActionRegistry from automation_file.exceptions import CallbackExecutorException from automation_file.logging_config import file_automation_logger -_VALID_METHODS = frozenset({"kwargs", "args"}) +_SETTINGS = CallbackSettings( + error=CallbackExecutorException, + style=CallbackStyle.STRICT, + on_error=CallbackErrorPolicy.RAISE, + log_info=file_automation_logger.info, +) -class CallbackExecutor: +class CallbackExecutor(CallbackFunctionExecutor): """Invoke ``trigger(**kwargs)`` then ``callback(*args | **kwargs)``.""" + registry: ActionRegistry + def __init__(self, registry: ActionRegistry) -> None: - self.registry: ActionRegistry = registry - - def callback_function( - self, - trigger_function_name: str, - callback_function: Callable[..., Any], - callback_function_param: Mapping[str, Any] | list[Any] | None = None, - callback_param_method: str = "kwargs", - **kwargs: Any, - ) -> Any: - trigger = self.registry.resolve(trigger_function_name) - if trigger is None: - raise CallbackExecutorException(f"unknown trigger: {trigger_function_name!r}") - if callback_param_method not in _VALID_METHODS: - raise CallbackExecutorException( - f"callback_param_method must be 'kwargs' or 'args', got {callback_param_method!r}" - ) - - file_automation_logger.info("callback: trigger=%s kwargs=%s", trigger_function_name, kwargs) - return_value = trigger(**kwargs) - - if callback_function_param is None: - callback_function() - elif callback_param_method == "kwargs": - if not isinstance(callback_function_param, Mapping): - raise CallbackExecutorException( - "callback_param_method='kwargs' requires a mapping payload" - ) - callback_function(**callback_function_param) - else: - if not isinstance(callback_function_param, (list, tuple)): - raise CallbackExecutorException( - "callback_param_method='args' requires a list/tuple payload" - ) - callback_function(*callback_function_param) - - file_automation_logger.info( - "callback: done trigger=%s callback=%r", trigger_function_name, callback_function - ) - return return_value + super().__init__(registry, _SETTINGS) diff --git a/automation_file/core/json_store.py b/automation_file/core/json_store.py index ac887df..191b6d8 100644 --- a/automation_file/core/json_store.py +++ b/automation_file/core/json_store.py @@ -1,43 +1,42 @@ """JSON persistence for action lists. -Reads/writes are serialised through a module-level lock so concurrent callers -cannot interleave writes against the same file. +Reads/writes are serialised through one lock so concurrent callers cannot +interleave writes against the same file (je_action_core's ``ActionJsonFile``). """ from __future__ import annotations import json -from pathlib import Path -from threading import Lock from typing import Any +from je_action_core import ActionJsonFile, JsonFileMessages, JsonFileSettings + from automation_file.exceptions import JsonActionException from automation_file.logging_config import file_automation_logger -_lock = Lock() +_json_file = ActionJsonFile( + JsonFileSettings( + error=JsonActionException, + messages=JsonFileMessages( + missing="can't read JSON file: {path}", + unreadable="can't read JSON file: {path}", + unwritable="can't write JSON file: {path}", + ), + read_errors=(OSError, json.JSONDecodeError), + write_errors=(OSError, TypeError), + log_info=file_automation_logger.info, + ) +) def read_action_json(json_file_path: str) -> Any: """Return the parsed JSON content at ``json_file_path``.""" - with _lock: - path = Path(json_file_path) - if not path.is_file(): - raise JsonActionException(f"can't read JSON file: {json_file_path}") - try: - with path.open(encoding="utf-8") as read_file: - data = json.load(read_file) - except (OSError, json.JSONDecodeError) as error: - raise JsonActionException(f"can't read JSON file: {json_file_path}") from error - file_automation_logger.info("read_action_json: %s", json_file_path) - return data + return _json_file.read(json_file_path) def write_action_json(json_save_path: str, action_json: Any) -> None: - """Write ``action_json`` to ``json_save_path`` as pretty UTF-8 JSON.""" - with _lock: - try: - with open(json_save_path, "w", encoding="utf-8") as file_to_write: - json.dump(action_json, file_to_write, indent=4, ensure_ascii=False) - except (OSError, TypeError) as error: - raise JsonActionException(f"can't write JSON file: {json_save_path}") from error - file_automation_logger.info("write_action_json: %s", json_save_path) + """Write ``action_json`` to ``json_save_path`` as pretty UTF-8 JSON. + + Data that cannot be serialised leaves the file as it was. + """ + _json_file.write(json_save_path, action_json) diff --git a/automation_file/core/package_loader.py b/automation_file/core/package_loader.py index 6f4a23e..456062a 100644 --- a/automation_file/core/package_loader.py +++ b/automation_file/core/package_loader.py @@ -1,59 +1,44 @@ """Dynamic plugin registration into an :class:`ActionRegistry`. ``PackageLoader`` imports an external package by name and registers every -top-level function / class / builtin under the key ``"_"``. +top-level function / class / builtin under the key ``"_"`` +(je_action_core's package manager with FileAutomation's settings). """ from __future__ import annotations -from importlib import import_module -from importlib.util import find_spec -from inspect import getmembers, isbuiltin, isclass, isfunction from types import ModuleType +from je_action_core import PackageGate, PackageManager, PackageManagerSettings + from automation_file.core.action_registry import ActionRegistry from automation_file.logging_config import file_automation_logger +_SETTINGS = PackageManagerSettings( + import_errors=(ImportError,), + # The package gate (workspace X-12) is not switched on here yet. + gate=PackageGate.OFF, + log_error=file_automation_logger.error, +) + -class PackageLoader: +class PackageLoader(PackageManager): """Load packages lazily and register their public callables.""" def __init__(self, registry: ActionRegistry) -> None: + super().__init__(_SETTINGS) self.registry: ActionRegistry = registry - self._cache: dict[str, ModuleType] = {} + self.executor = registry def load(self, package: str) -> ModuleType | None: """Import ``package`` once and return the module (cached).""" - cached = self._cache.get(package) - if cached is not None: - return cached - spec = find_spec(package) - if spec is None: - file_automation_logger.error("PackageLoader: cannot find %s", package) - return None - try: - # `package` is a trusted caller-supplied name (see PackageLoader docstring and - # the CLAUDE.md security note on plugin loading); it is not untrusted input. - name = spec.name - module = import_module(name) # nosemgrep - except ImportError as error: - file_automation_logger.error("PackageLoader import error: %r", error) - return None - self._cache[package] = module - return module - - def add_package_to_executor(self, package: str) -> int: + return self.check_package(package) + + def add_package_to_executor(self, package: str) -> int: # type: ignore[override] """Register every function / class / builtin from ``package``. Returns the number of commands that were registered. """ - module = self.load(package) - if module is None: - return 0 - count = 0 - for predicate in (isfunction, isbuiltin, isclass): - for member_name, member in getmembers(module, predicate): - self.registry.register(f"{package}_{member_name}", member) - count += 1 + count = self.check_and_add(package, self.registry) file_automation_logger.info("PackageLoader: registered %d members from %s", count, package) return count diff --git a/dev.toml b/dev.toml index 191763a..39d2a13 100644 --- a/dev.toml +++ b/dev.toml @@ -29,6 +29,7 @@ dependencies = [ "cryptography>=50.0.0", "prometheus_client>=0.26.0", "defusedxml>=0.7.1", + "je_action_core", "PyYAML>=6.0.3", "pyarrow>=25.0.1", "opentelemetry-api>=1.44.0", diff --git a/docs/updates/2026-10.md b/docs/updates/2026-10.md index 22e654a..d45b95c 100644 --- a/docs/updates/2026-10.md +++ b/docs/updates/2026-10.md @@ -15,3 +15,39 @@ Index and query commands: [README.md](README.md). New entries go at the end. - **What happened**: the test added in U-20261001-01 put `zip()` without `strict=` in its `_jobs` helper, so CI run 36796361197 failed in the `lint` job (`ruff check`) with ruff B905. - **Fix**: `zip(heads, ends, strict=True)`, and the file is formatted with `ruff format`, which `ci-dev.yml` / `ci-stable.yml` also check. The workflow test still passes, and the repository's CI lint commands pass locally. - **Files**: `tests/test_workflow_actions.py`. + +## U-20261001-03 · 2026-10-01 · Registry, executor pipeline and helpers move to je_action_core · #migration #executor #L-6 + +- **What**: workspace L-6. FileAutomation now builds on `je_action_core` (Integration-Automation/ActionCore), the executor core it shares with APITestka, LoadDensity and MailThunder. These now come from it: + - `ActionRegistry` is a `CommandRegistry` that accepts any callable; + - `ActionExecutor` subclasses the core's `ActionExecutor`; + - `PackageLoader` (gate off) and `CallbackExecutor` (strict checks, errors raised); + - `read_action_json` / `write_action_json`. +- **How `ActionExecutor` maps onto the core**: + - Action checking is the core's `StrictActionParser`, which carries this repo's messages; `_parse_action` is now a thin wrapper. + - Records are keyed `execute[]: `. + - `_coerce` is the core's `ActionListRules` with the `auto_control` key and its three messages. + - The tracing span moved into an `invoke` override. + - `execute_files` is the core's. + - Dry run, `validate`, substitution, `execute_action_parallel` and metrics stay here. +- **Unchanged**: + - every public name and signature, including `PackageLoader.load` and the member count returned by `add_package_to_executor`; + - every exception class and message; + - the shared registry between executor and callback executor; + - the TCP / HTTP servers. +- **Small differences, by design**: + - log wording comes from the core; + - `write_action_json` builds the JSON before opening the file, so unserialisable data no longer leaves a truncated file. +- **Dependency**: + - `stable.toml` and `dev.toml` list `je_action_core`, and the README "Bundled dependencies" (three languages) says so. + - It is not on PyPI yet, so both jobs of `ci-dev.yml` and `ci-stable.yml` install it from GitHub at commit `19bfe0a` before anything else. The lint job needs it too, so mypy reads its types: the core now ships `py.typed`. + - `progress.md` #7 switches to PyPI; do not release `main` before that. +- **Docs**: `architecture.md` §2, §4 and §6 updated: + - §4 now shows the indexed record key (it still said `execute: `); + - §6 corrects the stale builtins text (the siblings share `SAFE_BUILTINS`) and adds the ActionCore bullet. +- **Result / numbers**: 775 passed, 8 skipped, before and after. `ruff check` and `ruff format --check` pass on `automation_file tests`. `mypy automation_file`, with the core installed normally, reports no issues. zizmor reports informational findings only. +- **Files**: + - `automation_file/core/{action_registry,action_executor,package_loader,callback_executor,json_store}.py`; + - `stable.toml`, `dev.toml`, `.github/workflows/ci-{dev,stable}.yml`; + - `README.md`, `README.zh-TW.md`, `README.zh-CN.md`; + - `architecture.md`, `progress.md`. diff --git a/docs/updates/README.md b/docs/updates/README.md index 83035bc..8af4de1 100644 --- a/docs/updates/README.md +++ b/docs/updates/README.md @@ -58,6 +58,7 @@ In the same commit: delete the item from `progress.md`, add a `#done` entry here | ID | Date | Title | Tags | Batch | |---|---|---|---|---| +| U-20261001-03 | 2026-10-01 | Registry, executor pipeline and helpers move to je_action_core | #migration #executor #L-6 | [2026-10](2026-10.md) | | U-20261001-02 | 2026-10-01 | Workflow-timeout test failed CI lint (ruff B905) | #incident #ci | [2026-10](2026-10.md) | | U-20261001-01 | 2026-10-01 | Every workflow job has a timeout | #ci #tests | [2026-10](2026-10.md) | | U-20260925-03 | 2026-09-25 | Python classifiers list every version CI tests | #packaging #tests | [2026-09](2026-09.md) | @@ -86,5 +87,5 @@ In the same commit: delete the item from `progress.md`, add a `#done` entry here | File | Period | Entries | |---|---|---:| -| [2026-10.md](2026-10.md) | 2026-10 | 2 | +| [2026-10.md](2026-10.md) | 2026-10 | 3 | | [2026-09.md](2026-09.md) | 2026-09 | 21 | diff --git a/progress.md b/progress.md index e45ec4f..ea3518e 100644 --- a/progress.md +++ b/progress.md @@ -6,3 +6,4 @@ Cross-repo and workspace items live in `D:\Codes\progress.md` (relevant here: X- ## Open +- **#7** [BLOCKED: je_action_core on PyPI, ActionCore `progress.md` #1] Install `je_action_core` from PyPI: drop the `pip install --no-deps "je_action_core @ git+..."` lines from both jobs of `ci-dev.yml` and `ci-stable.yml` (`pip install -e .` then resolves it). Until then, do not release `main`: the published metadata requires `je_action_core`, which PyPI does not have yet. diff --git a/stable.toml b/stable.toml index c2d1456..3b67097 100644 --- a/stable.toml +++ b/stable.toml @@ -29,6 +29,7 @@ dependencies = [ "cryptography>=50.0.0", "prometheus_client>=0.26.0", "defusedxml>=0.7.1", + "je_action_core", "PyYAML>=6.0.3", "pyarrow>=25.0.1", "opentelemetry-api>=1.44.0", From 4fa2e39469db75179e124514df1782c56cd4e0ff Mon Sep 17 00:00:00 2001 From: JeffreyChen Date: Thu, 1 Oct 2026 13:38:48 +0800 Subject: [PATCH 11/59] build: install je_action_core from PyPI instead of the GitHub pin --- .github/workflows/ci-dev.yml | 9 ++------- .github/workflows/ci-stable.yml | 9 ++------- architecture.md | 4 ++-- dev.toml | 2 +- docs/updates/2026-10.md | 9 +++++++++ docs/updates/README.md | 3 ++- progress.md | 2 -- stable.toml | 2 +- 8 files changed, 19 insertions(+), 21 deletions(-) diff --git a/.github/workflows/ci-dev.yml b/.github/workflows/ci-dev.yml index ccbef74..c6f4c2c 100644 --- a/.github/workflows/ci-dev.yml +++ b/.github/workflows/ci-dev.yml @@ -26,10 +26,8 @@ jobs: - name: Install tooling run: | python -m pip install --upgrade pip - # je_action_core is not on PyPI yet (ActionCore progress.md #1): install it from GitHub at a - # fixed commit first, so mypy sees its types and `pip install -e .` finds it installed. - pip install --no-deps "je_action_core @ git+https://github.com/Integration-Automation/ActionCore@19bfe0a7c907af00c6a59af3de04448999b42e44" - pip install ruff mypy + # mypy reads je_action_core's types (py.typed) for the classes built on it + pip install ruff mypy "je_action_core>=0.0.1" - name: Ruff check run: ruff check automation_file tests - name: Ruff format check @@ -57,9 +55,6 @@ jobs: - name: Install dependencies run: | python -m pip install --upgrade pip wheel - # je_action_core is not on PyPI yet (ActionCore progress.md #1): install it from GitHub at a - # fixed commit first, so mypy sees its types and `pip install -e .` finds it installed. - pip install --no-deps "je_action_core @ git+https://github.com/Integration-Automation/ActionCore@19bfe0a7c907af00c6a59af3de04448999b42e44" Copy-Item dev.toml pyproject.toml -Force pip install -e . pip install pytest pytest-cov diff --git a/.github/workflows/ci-stable.yml b/.github/workflows/ci-stable.yml index e29fa90..ecf4ec4 100644 --- a/.github/workflows/ci-stable.yml +++ b/.github/workflows/ci-stable.yml @@ -26,10 +26,8 @@ jobs: - name: Install tooling run: | python -m pip install --upgrade pip - # je_action_core is not on PyPI yet (ActionCore progress.md #1): install it from GitHub at a - # fixed commit first, so mypy sees its types and `pip install -e .` finds it installed. - pip install --no-deps "je_action_core @ git+https://github.com/Integration-Automation/ActionCore@19bfe0a7c907af00c6a59af3de04448999b42e44" - pip install ruff mypy + # mypy reads je_action_core's types (py.typed) for the classes built on it + pip install ruff mypy "je_action_core>=0.0.1" - name: Ruff check run: ruff check automation_file tests - name: Ruff format check @@ -57,9 +55,6 @@ jobs: - name: Install dependencies run: | python -m pip install --upgrade pip wheel - # je_action_core is not on PyPI yet (ActionCore progress.md #1): install it from GitHub at a - # fixed commit first, so mypy sees its types and `pip install -e .` finds it installed. - pip install --no-deps "je_action_core @ git+https://github.com/Integration-Automation/ActionCore@19bfe0a7c907af00c6a59af3de04448999b42e44" Copy-Item stable.toml pyproject.toml -Force pip install -e . pip install pytest pytest-cov diff --git a/architecture.md b/architecture.md index fae8037..f79257a 100644 --- a/architecture.md +++ b/architecture.md @@ -136,8 +136,8 @@ ActionExecutor() → build_default_registry(): local + http + utils + drive comm - **callback executor**: strict checks, errors raised; - **JSON files**: `JSONDecodeError` / `OSError` wrapped on read, `OSError` / `TypeError` on write. - The extras (dry run, validate, substitute, parallel, metrics) and the TCP / HTTP servers stay here. Until the - package is on PyPI, the CI installs it from GitHub at a fixed commit (`progress.md` #7). ActionCore lists + The extras (dry run, validate, substitute, parallel, metrics) and the TCP / HTTP servers stay here. It is a PyPI + dependency (`je_action_core>=0.0.1`); the CI lint job installs it too, so mypy reads its types. ActionCore lists FileAutomation in its own §6. ## 7. Design constraints diff --git a/dev.toml b/dev.toml index 39d2a13..a39096c 100644 --- a/dev.toml +++ b/dev.toml @@ -29,7 +29,7 @@ dependencies = [ "cryptography>=50.0.0", "prometheus_client>=0.26.0", "defusedxml>=0.7.1", - "je_action_core", + "je_action_core>=0.0.1", "PyYAML>=6.0.3", "pyarrow>=25.0.1", "opentelemetry-api>=1.44.0", diff --git a/docs/updates/2026-10.md b/docs/updates/2026-10.md index d45b95c..15ce762 100644 --- a/docs/updates/2026-10.md +++ b/docs/updates/2026-10.md @@ -51,3 +51,12 @@ Index and query commands: [README.md](README.md). New entries go at the end. - `stable.toml`, `dev.toml`, `.github/workflows/ci-{dev,stable}.yml`; - `README.md`, `README.zh-TW.md`, `README.zh-CN.md`; - `architecture.md`, `progress.md`. + +## U-20261001-04 · 2026-10-01 · je_action_core comes from PyPI · #done #build #L-6 + +- **What**: `progress.md` #7, unblocked once je_action_core 0.0.1 was published. + - `stable.toml` and `dev.toml` require `je_action_core>=0.0.1`. + - Both CI workflows drop the GitHub install line: the pytest job gets the package from `pip install -e .`, and the lint job installs `je_action_core>=0.0.1` next to ruff and mypy so mypy reads its types. + - `main` can be released again. +- **Result / numbers**: zizmor reports informational findings only. +- **Files**: `stable.toml`, `dev.toml`, `.github/workflows/ci-{dev,stable}.yml`, `architecture.md`, `progress.md`. diff --git a/docs/updates/README.md b/docs/updates/README.md index 8af4de1..0ed6bc6 100644 --- a/docs/updates/README.md +++ b/docs/updates/README.md @@ -58,6 +58,7 @@ In the same commit: delete the item from `progress.md`, add a `#done` entry here | ID | Date | Title | Tags | Batch | |---|---|---|---|---| +| U-20261001-04 | 2026-10-01 | je_action_core comes from PyPI | #done #build #L-6 | [2026-10](2026-10.md) | | U-20261001-03 | 2026-10-01 | Registry, executor pipeline and helpers move to je_action_core | #migration #executor #L-6 | [2026-10](2026-10.md) | | U-20261001-02 | 2026-10-01 | Workflow-timeout test failed CI lint (ruff B905) | #incident #ci | [2026-10](2026-10.md) | | U-20261001-01 | 2026-10-01 | Every workflow job has a timeout | #ci #tests | [2026-10](2026-10.md) | @@ -87,5 +88,5 @@ In the same commit: delete the item from `progress.md`, add a `#done` entry here | File | Period | Entries | |---|---|---:| -| [2026-10.md](2026-10.md) | 2026-10 | 3 | +| [2026-10.md](2026-10.md) | 2026-10 | 4 | | [2026-09.md](2026-09.md) | 2026-09 | 21 | diff --git a/progress.md b/progress.md index ea3518e..d683eef 100644 --- a/progress.md +++ b/progress.md @@ -5,5 +5,3 @@ Item numbers (`#n`) are never reused. Tags: [DECIDE] needs the owner's decision, Cross-repo and workspace items live in `D:\Codes\progress.md` (relevant here: X-12, X-13). ## Open - -- **#7** [BLOCKED: je_action_core on PyPI, ActionCore `progress.md` #1] Install `je_action_core` from PyPI: drop the `pip install --no-deps "je_action_core @ git+..."` lines from both jobs of `ci-dev.yml` and `ci-stable.yml` (`pip install -e .` then resolves it). Until then, do not release `main`: the published metadata requires `je_action_core`, which PyPI does not have yet. diff --git a/stable.toml b/stable.toml index 3b67097..e254212 100644 --- a/stable.toml +++ b/stable.toml @@ -29,7 +29,7 @@ dependencies = [ "cryptography>=50.0.0", "prometheus_client>=0.26.0", "defusedxml>=0.7.1", - "je_action_core", + "je_action_core>=0.0.1", "PyYAML>=6.0.3", "pyarrow>=25.0.1", "opentelemetry-api>=1.44.0", From 1a6a468f42f99f8d9a5ad2da75153ed6b3e15ff3 Mon Sep 17 00:00:00 2001 From: JeffreyChen Date: Thu, 1 Oct 2026 13:40:44 +0800 Subject: [PATCH 12/59] test: guard that no action command loads packages, which keeps the package gate off --- architecture.md | 4 +++- automation_file/core/package_loader.py | 4 +++- docs/updates/2026-10.md | 10 ++++++++++ docs/updates/README.md | 3 ++- tests/test_package_loader.py | 14 ++++++++++++++ 5 files changed, 32 insertions(+), 3 deletions(-) diff --git a/architecture.md b/architecture.md index f79257a..212f8be 100644 --- a/architecture.md +++ b/architecture.md @@ -153,7 +153,9 @@ ActionExecutor() → build_default_registry(): local + http + utils + drive comm - Resolve user paths through `safe_join` / `is_within` (§ Security › Path traversal). SFTP keeps `paramiko.RejectPolicy()` (§ Security › SFTP host verification). - `retry_on_transient` retries only the listed exception types (§ Security › Reliability (retry / quota)). - `PackageLoader` is eval-grade; never expose it remotely (§ Security › Plugin / package loading). + `PackageLoader` is eval-grade; never expose it remotely (§ Security › Plugin / package loading). No `FA_*` + command reaches it, so je_action_core's package gate is off here (`tests/test_package_loader.py` fails if one + is added; workspace X-12). - No `shell=True`; subprocesses use argument lists and a timeout (§ Security › General rules; › Subprocess execution). - Backends and PySide6 are first-class runtime dependencies. Keep `stable.toml` and `dev.toml` dependencies in sync, and let the publish workflow bump versions (§ Branching & CI). diff --git a/automation_file/core/package_loader.py b/automation_file/core/package_loader.py index 456062a..137fd0b 100644 --- a/automation_file/core/package_loader.py +++ b/automation_file/core/package_loader.py @@ -16,7 +16,9 @@ _SETTINGS = PackageManagerSettings( import_errors=(ImportError,), - # The package gate (workspace X-12) is not switched on here yet. + # No FA_* command reaches the loader (it is Python-only, see CLAUDE.md, Plugin / package loading), + # so je_action_core's package gate would only warn the host itself. tests/test_package_loader.py + # fails if a command that loads packages is added; switch the gate on then (workspace X-12). gate=PackageGate.OFF, log_error=file_automation_logger.error, ) diff --git a/docs/updates/2026-10.md b/docs/updates/2026-10.md index 15ce762..454b23d 100644 --- a/docs/updates/2026-10.md +++ b/docs/updates/2026-10.md @@ -60,3 +60,13 @@ Index and query commands: [README.md](README.md). New entries go at the end. - `main` can be released again. - **Result / numbers**: zizmor reports informational findings only. - **Files**: `stable.toml`, `dev.toml`, `.github/workflows/ci-{dev,stable}.yml`, `architecture.md`, `progress.md`. + +## U-20261001-05 · 2026-10-01 · X-12: no action command loads packages, so the gate stays off · #decision #security #X-12 + +- **What**: workspace X-12 put a package gate in front of each framework's `*_add_package_to_executor`. That command lets an action list import `os` or `subprocess`. + - FileAutomation has no such command. `PackageLoader` is reachable only from Python (`automation_file.package_manager`), never from an action list, the CLI, the MCP server or the TCP/HTTP servers. + - The 139 registered commands include none with "package" in the name, and nothing outside `core/package_loader.py` calls `import_module`. + - Switching je_action_core's gate on would therefore only warn the host program about its own calls. It stays off. + - `tests/test_package_loader.py::test_no_action_command_loads_packages` now fails if a command that loads packages is added, so the gate has to be switched on first. The `package_loader.py` comment and `architecture.md` §7 say so. +- **Result / numbers**: `tests/test_package_loader.py` 5 passed. `ruff check` and `ruff format --check` pass. +- **Files**: `automation_file/core/package_loader.py`, `tests/test_package_loader.py`, `architecture.md`. diff --git a/docs/updates/README.md b/docs/updates/README.md index 0ed6bc6..a47295a 100644 --- a/docs/updates/README.md +++ b/docs/updates/README.md @@ -58,6 +58,7 @@ In the same commit: delete the item from `progress.md`, add a `#done` entry here | ID | Date | Title | Tags | Batch | |---|---|---|---|---| +| U-20261001-05 | 2026-10-01 | X-12: no action command loads packages, so the gate stays off | #decision #security #X-12 | [2026-10](2026-10.md) | | U-20261001-04 | 2026-10-01 | je_action_core comes from PyPI | #done #build #L-6 | [2026-10](2026-10.md) | | U-20261001-03 | 2026-10-01 | Registry, executor pipeline and helpers move to je_action_core | #migration #executor #L-6 | [2026-10](2026-10.md) | | U-20261001-02 | 2026-10-01 | Workflow-timeout test failed CI lint (ruff B905) | #incident #ci | [2026-10](2026-10.md) | @@ -88,5 +89,5 @@ In the same commit: delete the item from `progress.md`, add a `#done` entry here | File | Period | Entries | |---|---|---:| -| [2026-10.md](2026-10.md) | 2026-10 | 4 | +| [2026-10.md](2026-10.md) | 2026-10 | 5 | | [2026-09.md](2026-09.md) | 2026-09 | 21 | diff --git a/tests/test_package_loader.py b/tests/test_package_loader.py index 1a1f19e..2882293 100644 --- a/tests/test_package_loader.py +++ b/tests/test_package_loader.py @@ -2,6 +2,8 @@ from __future__ import annotations +from je_action_core import PackageGate + from automation_file.core.action_registry import ActionRegistry from automation_file.core.package_loader import PackageLoader @@ -31,3 +33,15 @@ def test_add_missing_package_returns_zero() -> None: registry = ActionRegistry() loader = PackageLoader(registry) assert loader.add_package_to_executor("not_a_real_package_xyz_123") == 0 + + +def test_no_action_command_loads_packages() -> None: + """The loader is Python-only, so the package gate stays off (workspace X-12). + + A command that loads packages would let an action list import ``os`` or ``subprocess``; + adding one means switching the gate on in ``core/package_loader.py`` first. + """ + from automation_file import executor, package_manager + + assert not [name for name in executor.registry if "package" in name.lower()] + assert package_manager.settings.gate is PackageGate.OFF From 3057b341d5a375f752429bc7002ee45330e0302b Mon Sep 17 00:00:00 2001 From: JeffreyChen Date: Thu, 1 Oct 2026 14:02:40 +0800 Subject: [PATCH 13/59] test: pin the TCP server's wire replies before it moves to je_action_core --- tests/test_tcp_wire.py | 111 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 111 insertions(+) create mode 100644 tests/test_tcp_wire.py diff --git a/tests/test_tcp_wire.py b/tests/test_tcp_wire.py new file mode 100644 index 0000000..0f364fb --- /dev/null +++ b/tests/test_tcp_wire.py @@ -0,0 +1,111 @@ +"""The TCP server's wire replies, byte for byte (kept when the server moves onto je_action_core).""" + +from __future__ import annotations + +import json +import socket +import time + +import pytest + +from automation_file.core.action_executor import executor +from automation_file.server.action_acl import ActionACL +from automation_file.server.tcp_server import start_autocontrol_socket_server + +END = b"Return_Data_Over_JE\n" +SECRET = "s3cr3t" + + +def _wire_echo(value: str) -> str: + return value + + +executor.registry.register("test_tcp_wire_echo", _wire_echo) + + +def _ask(port: int, payload: bytes) -> bytes: + with socket.create_connection(("127.0.0.1", port), timeout=5) as sock: + sock.sendall(payload) + sock.shutdown(socket.SHUT_WR) # end of request, so an empty one is seen as such + chunks = [] + chunk = sock.recv(4096) + while chunk: # the server closes the connection after its reply + chunks.append(chunk) + chunk = sock.recv(4096) + return b"".join(chunks) + + +@pytest.fixture(name="serve") +def _serve(): + servers = [] + + def start(**kwargs): + server = start_autocontrol_socket_server(host="127.0.0.1", port=0, **kwargs) + servers.append(server) + return server, server.server_address[1] + + yield start + for server in servers: + server.shutdown() + server.server_close() + + +def _echo(value: str) -> bytes: + return json.dumps([["test_tcp_wire_echo", {"value": value}]]).encode() + + +def test_records_are_key_arrow_value_lines(serve) -> None: + _, port = serve() + expected = b"execute[0]: ['test_tcp_wire_echo', {'value': 'hi'}] -> hi\n" + END + assert _ask(port, _echo("hi")) == expected + + +def test_bad_json(serve) -> None: + _, port = serve() + reply = _ask(port, b"not json") + assert ( + reply == b"json error: JSONDecodeError('Expecting value: line 1 column 1 (char 0)')\n" + END + ) + + +def test_undecodable_bytes(serve) -> None: + _, port = serve() + reply = _ask(port, b"\xff\xfe") + assert reply.startswith(b"decode error: UnicodeDecodeError(") + assert reply.endswith(b"\n" + END) + + +def test_empty_request_gets_no_reply(serve) -> None: + _, port = serve() + assert _ask(port, b"") == b"" + + +def test_quit_replies_and_stops(serve) -> None: + server, port = serve() + assert _ask(port, b"quit_server") == b"server shutting down\n" + deadline = time.monotonic() + 5 + while not server.close_flag and time.monotonic() < deadline: + time.sleep(0.05) + assert server.close_flag + + +@pytest.mark.parametrize( + "payload", + [b'[["test_tcp_wire_echo", ["x"]]]', b"AUTH wrong\n[]", b"AUTH " + SECRET.encode()], +) +def test_auth_failures(serve, payload: bytes) -> None: + _, port = serve(shared_secret=SECRET) + assert _ask(port, payload) == b"auth error\n" + END + + +def test_auth_success_runs_the_payload(serve) -> None: + _, port = serve(shared_secret=SECRET) + reply = _ask(port, b"AUTH " + SECRET.encode() + b"\n" + _echo("ok")) + assert reply == b"execute[0]: ['test_tcp_wire_echo', {'value': 'ok'}] -> ok\n" + END + + +def test_acl_refusal(serve) -> None: + _, port = serve(action_acl=ActionACL.build(denied=["test_tcp_wire_echo"])) + reply = _ask(port, _echo("no")) + assert reply.startswith(b"forbidden: ") + assert reply.endswith(b"\n" + END) From 8269caaa7650f7d8da0fb072ff3958e6bb53ff4e Mon Sep 17 00:00:00 2001 From: JeffreyChen Date: Thu, 1 Oct 2026 14:23:07 +0800 Subject: [PATCH 14/59] refactor: run the TCP server on je_action_core's secret-header handler --- .github/workflows/ci-dev.yml | 2 +- .github/workflows/ci-stable.yml | 2 +- architecture.md | 9 +- automation_file/server/tcp_server.py | 151 ++++++++------------------- dev.toml | 2 +- docs/updates/2026-10.md | 21 ++++ docs/updates/README.md | 3 +- stable.toml | 2 +- tests/test_tcp_server.py | 6 +- 9 files changed, 82 insertions(+), 116 deletions(-) diff --git a/.github/workflows/ci-dev.yml b/.github/workflows/ci-dev.yml index c6f4c2c..75f9420 100644 --- a/.github/workflows/ci-dev.yml +++ b/.github/workflows/ci-dev.yml @@ -27,7 +27,7 @@ jobs: run: | python -m pip install --upgrade pip # mypy reads je_action_core's types (py.typed) for the classes built on it - pip install ruff mypy "je_action_core>=0.0.1" + pip install ruff mypy "je_action_core>=0.0.2" - name: Ruff check run: ruff check automation_file tests - name: Ruff format check diff --git a/.github/workflows/ci-stable.yml b/.github/workflows/ci-stable.yml index ecf4ec4..56855bf 100644 --- a/.github/workflows/ci-stable.yml +++ b/.github/workflows/ci-stable.yml @@ -27,7 +27,7 @@ jobs: run: | python -m pip install --upgrade pip # mypy reads je_action_core's types (py.typed) for the classes built on it - pip install ruff mypy "je_action_core>=0.0.1" + pip install ruff mypy "je_action_core>=0.0.2" - name: Ruff check run: ruff check automation_file tests - name: Ruff format check diff --git a/architecture.md b/architecture.md index 212f8be..c9ed6ed 100644 --- a/architecture.md +++ b/architecture.md @@ -136,8 +136,13 @@ ActionExecutor() → build_default_registry(): local + http + utils + drive comm - **callback executor**: strict checks, errors raised; - **JSON files**: `JSONDecodeError` / `OSError` wrapped on read, `OSError` / `TypeError` on write. - The extras (dry run, validate, substitute, parallel, metrics) and the TCP / HTTP servers stay here. It is a PyPI - dependency (`je_action_core>=0.0.1`); the CI lint job installs it too, so mypy reads its types. ActionCore lists + - **TCP server**: `SecretHeaderRequestHandler` (`AUTH ` first line) on a `TCPActionServer` + (`ActionTCPServer` with daemon threads and address reuse), the ACL as its `validate` hook, and + `ReplyMessages` for ` -> ` lines and the `json error` / `forbidden` / `execution error` / + `decode error` / `auth error` replies; `ensure_loopback` runs before it starts. + + The extras (dry run, validate, substitute, parallel, metrics) and the HTTP server stay here. It is a PyPI + dependency (`je_action_core>=0.0.2`); the CI lint job installs it too, so mypy reads its types. ActionCore lists FileAutomation in its own §6. ## 7. Design constraints diff --git a/automation_file/server/tcp_server.py b/automation_file/server/tcp_server.py index e46420b..0fde8a3 100644 --- a/automation_file/server/tcp_server.py +++ b/automation_file/server/tcp_server.py @@ -8,120 +8,62 @@ begin with ``AUTH \\n`` before the JSON payload. This is the minimum bar for exposing the server beyond loopback; use a TLS-terminating proxy for anything resembling production. + +The server is je_action_core's action server with FileAutomation's dialect: +records are `` -> `` lines, failures say where they happened +(``json error``, ``forbidden``, ``execution error``, ``decode error``, +``auth error``), and every reply but the quit acknowledgement ends with +``Return_Data_Over_JE``. """ from __future__ import annotations -import hmac -import json -import socketserver import sys -import threading -from typing import Any +from typing import Any, cast + +from je_action_core import ( + ActionTCPServer, + ReplyMessages, + SecretHeaderRequestHandler, + SocketServerSettings, + start_action_socket_server, +) from automation_file.core.action_executor import execute_action -from automation_file.exceptions import TCPAuthException from automation_file.logging_config import file_automation_logger -from automation_file.server.action_acl import ActionACL, ActionNotPermittedException +from automation_file.server.action_acl import ActionACL from automation_file.server.network_guards import ensure_loopback _DEFAULT_HOST = "localhost" _DEFAULT_PORT = 9943 -_RECV_BYTES = 8192 -_END_MARKER = b"Return_Data_Over_JE\n" -_QUIT_COMMAND = "quit_server" -_AUTH_PREFIX = "AUTH " - - -class _TCPServerHandler(socketserver.StreamRequestHandler): - """One instance per connection; dispatches a single JSON payload.""" - - def handle(self) -> None: - raw = self.request.recv(_RECV_BYTES) - if not raw: - return - try: - command_string = raw.strip().decode("utf-8") - except UnicodeDecodeError as error: - self._send_line(f"decode error: {error!r}") - self._send_bytes(_END_MARKER) - return - - try: - command_string = self._enforce_auth(command_string) - except TCPAuthException as error: - file_automation_logger.warning("tcp_server auth: %r", error) - self._send_line("auth error") - self._send_bytes(_END_MARKER) - return - - file_automation_logger.info("tcp_server: recv %s", command_string) - if command_string == _QUIT_COMMAND: - self.server.close_flag = True # type: ignore[attr-defined] - threading.Thread(target=self.server.shutdown, daemon=True).start() - self._send_line("server shutting down") - return - - try: - payload = json.loads(command_string) - acl: ActionACL | None = getattr(self.server, "action_acl", None) - if acl is not None: - acl.enforce(payload) - results = execute_action(payload) - for key, value in results.items(): - self._send_line(f"{key} -> {value}") - except json.JSONDecodeError as error: - self._send_line(f"json error: {error!r}") - except ActionNotPermittedException as error: - file_automation_logger.warning("tcp_server acl: %r", error) - self._send_line(f"forbidden: {error}") - except Exception as error: # pylint: disable=broad-except - file_automation_logger.error("tcp_server handler: %r", error) - self._send_line(f"execution error: {error!r}") - finally: - self._send_bytes(_END_MARKER) - - def _enforce_auth(self, command_string: str) -> str: - secret: str | None = getattr(self.server, "shared_secret", None) - if not secret: - return command_string - head, _, rest = command_string.partition("\n") - if not head.startswith(_AUTH_PREFIX): - raise TCPAuthException("missing AUTH header") - supplied = head[len(_AUTH_PREFIX) :].strip() - if not hmac.compare_digest(supplied, secret): - raise TCPAuthException("bad shared secret") - if not rest: - raise TCPAuthException("empty payload after AUTH") - return rest - - def _send_line(self, text: str) -> None: - self._send_bytes(text.encode("utf-8") + b"\n") - - def _send_bytes(self, data: bytes) -> None: - try: - self.request.sendall(data) - except OSError as error: - file_automation_logger.error("tcp_server sendall: %r", error) - - -class TCPActionServer(socketserver.ThreadingMixIn, socketserver.TCPServer): +_MESSAGES = ReplyMessages( + record="{key} -> {value}", + error="execution error: {error!r}", + json_error="json error: {error!r}", + refused="forbidden: {error}", + decode_error="decode error: {error!r}", + quit="server shutting down", + auth_refused="auth error", + log_command="tcp_server: recv {text}", +) + + +class TCPActionServer(ActionTCPServer): """Threaded TCP server with an explicit close flag.""" daemon_threads = True allow_reuse_address = True - def __init__( - self, - server_address: tuple[str, int], - request_handler_class: type, - shared_secret: str | None = None, - action_acl: ActionACL | None = None, - ) -> None: - super().__init__(server_address, request_handler_class) - self.close_flag: bool = False - self.shared_secret: str | None = shared_secret - self.action_acl: ActionACL | None = action_acl + +def _settings(shared_secret: str | None, action_acl: ActionACL | None) -> SocketServerSettings: + return SocketServerSettings( + execute=execute_action, + validate=action_acl.enforce if action_acl is not None else None, + messages=_MESSAGES, + secret=shared_secret, + log_info=file_automation_logger.info, + log_error=file_automation_logger.error, + ) def start_autocontrol_socket_server( @@ -145,21 +87,20 @@ def start_autocontrol_socket_server( file_automation_logger.warning( "tcp_server: non-loopback bind without shared_secret is insecure", ) - server = TCPActionServer( - (host, port), - _TCPServerHandler, - shared_secret=shared_secret, - action_acl=action_acl, + server = start_action_socket_server( + host, + port, + _settings(shared_secret, action_acl), + SecretHeaderRequestHandler, + TCPActionServer, ) - thread = threading.Thread(target=server.serve_forever, daemon=True) - thread.start() file_automation_logger.info( "tcp_server: listening on %s:%d (auth=%s)", host, port, "on" if shared_secret else "off", ) - return server + return cast(TCPActionServer, server) def main(argv: list[str] | None = None) -> Any: diff --git a/dev.toml b/dev.toml index a39096c..16de537 100644 --- a/dev.toml +++ b/dev.toml @@ -29,7 +29,7 @@ dependencies = [ "cryptography>=50.0.0", "prometheus_client>=0.26.0", "defusedxml>=0.7.1", - "je_action_core>=0.0.1", + "je_action_core>=0.0.2", "PyYAML>=6.0.3", "pyarrow>=25.0.1", "opentelemetry-api>=1.44.0", diff --git a/docs/updates/2026-10.md b/docs/updates/2026-10.md index 454b23d..0fd61e6 100644 --- a/docs/updates/2026-10.md +++ b/docs/updates/2026-10.md @@ -70,3 +70,24 @@ Index and query commands: [README.md](README.md). New entries go at the end. - `tests/test_package_loader.py::test_no_action_command_loads_packages` now fails if a command that loads packages is added, so the gate has to be switched on first. The `package_loader.py` comment and `architecture.md` §7 say so. - **Result / numbers**: `tests/test_package_loader.py` 5 passed. `ruff check` and `ruff format --check` pass. - **Files**: `automation_file/core/package_loader.py`, `tests/test_package_loader.py`, `architecture.md`. + +## U-20261001-06 · 2026-10-01 · The TCP server moves to je_action_core · #migration #socket-server #L-6 + +- **What**: ActionCore's `progress.md` #2. `server/tcp_server.py` drops its own handler and builds on je_action_core 0.0.2's `SecretHeaderRequestHandler`: + - `TCPActionServer` is now an `ActionTCPServer`, still with daemon threads and address reuse; + - the ACL is the settings' `validate` hook; + - `ReplyMessages` carries this server's texts. + + `start_autocontrol_socket_server` keeps its signature, the loopback guard and the insecure-bind warning. +- **Behaviour kept**: `tests/test_tcp_wire.py` (10 byte-for-byte cases, committed in `3057b34` before the move) passes unchanged. It covers: + - ` -> ` records; + - `json error` / `decode error` replies; + - no reply to an empty request, and quit's `server shutting down` without the marker; + - the `AUTH` header refusals and success; + - ACL `forbidden`. + + The existing TCP, auth and ACL tests pass too. `tests/test_tcp_server.py` now defines the end marker itself instead of importing the module's private `_END_MARKER`. +- **Small differences, by design**: an authentication failure is logged as an error, where it used to be a warning. +- **Also**: the minimum becomes `je_action_core>=0.0.2` (`stable.toml`, `dev.toml`, and the CI lint job's install). `architecture.md` §6 describes the server's settings. +- **Result / numbers**: 786 passed, 8 skipped. `ruff check` and `ruff format --check` pass. `mypy automation_file` reports no issues with the core installed normally. +- **Files**: `automation_file/server/tcp_server.py`, `tests/test_tcp_server.py`, `stable.toml`, `dev.toml`, `.github/workflows/ci-{dev,stable}.yml`, `architecture.md`. diff --git a/docs/updates/README.md b/docs/updates/README.md index a47295a..3c9dd74 100644 --- a/docs/updates/README.md +++ b/docs/updates/README.md @@ -58,6 +58,7 @@ In the same commit: delete the item from `progress.md`, add a `#done` entry here | ID | Date | Title | Tags | Batch | |---|---|---|---|---| +| U-20261001-06 | 2026-10-01 | The TCP server moves to je_action_core | #migration #socket-server #L-6 | [2026-10](2026-10.md) | | U-20261001-05 | 2026-10-01 | X-12: no action command loads packages, so the gate stays off | #decision #security #X-12 | [2026-10](2026-10.md) | | U-20261001-04 | 2026-10-01 | je_action_core comes from PyPI | #done #build #L-6 | [2026-10](2026-10.md) | | U-20261001-03 | 2026-10-01 | Registry, executor pipeline and helpers move to je_action_core | #migration #executor #L-6 | [2026-10](2026-10.md) | @@ -89,5 +90,5 @@ In the same commit: delete the item from `progress.md`, add a `#done` entry here | File | Period | Entries | |---|---|---:| -| [2026-10.md](2026-10.md) | 2026-10 | 5 | +| [2026-10.md](2026-10.md) | 2026-10 | 6 | | [2026-09.md](2026-09.md) | 2026-09 | 21 | diff --git a/stable.toml b/stable.toml index e254212..02f14e6 100644 --- a/stable.toml +++ b/stable.toml @@ -29,7 +29,7 @@ dependencies = [ "cryptography>=50.0.0", "prometheus_client>=0.26.0", "defusedxml>=0.7.1", - "je_action_core>=0.0.1", + "je_action_core>=0.0.2", "PyYAML>=6.0.3", "pyarrow>=25.0.1", "opentelemetry-api>=1.44.0", diff --git a/tests/test_tcp_server.py b/tests/test_tcp_server.py index c8075d7..edffc45 100644 --- a/tests/test_tcp_server.py +++ b/tests/test_tcp_server.py @@ -7,13 +7,11 @@ import pytest -from automation_file.server.tcp_server import ( - _END_MARKER, - start_autocontrol_socket_server, -) +from automation_file.server.tcp_server import start_autocontrol_socket_server from tests._insecure_fixtures import ipv4 _HOST = "127.0.0.1" +_END_MARKER = b"Return_Data_Over_JE\n" def _free_port() -> int: From 15e481f0c788e3ff7a1a1b0aeafc8d6ea9b74c95 Mon Sep 17 00:00:00 2001 From: JeffreyChen Date: Thu, 1 Oct 2026 16:51:52 +0800 Subject: [PATCH 15/59] ci: publish automation_file_dev from a tested push to dev that changes the package --- .github/workflows/ci-dev.yml | 47 ++++++++++ CLAUDE.md | 9 +- architecture.md | 15 ++- dev.toml | 4 +- docs/updates/2026-10.md | 14 +++ docs/updates/README.md | 3 +- progress.md | 2 + scripts/dev_release.py | 134 +++++++++++++++++++++++++++ tests/test_dev_release.py | 168 ++++++++++++++++++++++++++++++++++ tests/test_dev_toml_parity.py | 62 +++++++++++++ 10 files changed, 450 insertions(+), 8 deletions(-) create mode 100644 scripts/dev_release.py create mode 100644 tests/test_dev_release.py create mode 100644 tests/test_dev_toml_parity.py diff --git a/.github/workflows/ci-dev.yml b/.github/workflows/ci-dev.yml index 75f9420..728c5d5 100644 --- a/.github/workflows/ci-dev.yml +++ b/.github/workflows/ci-dev.yml @@ -66,3 +66,50 @@ jobs: with: name: coverage-xml path: coverage.xml + + publish-dev: + # The dev channel. A push to dev that passes lint and the tests is built from dev.toml and uploaded + # when it is still the tip of dev and ships something the newest automation_file_dev does not. + # scripts/dev_release.py picks the version from PyPI, so nothing is committed back. + name: Publish automation_file_dev to PyPI + needs: [lint, pytest] + if: github.event_name == 'push' && github.ref == 'refs/heads/dev' + runs-on: ubuntu-latest + timeout-minutes: 15 # about 3x the slowest recent run, at least 15 + concurrency: + group: publish-dev + cancel-in-progress: false + steps: + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + persist-credentials: false + - name: Set up Python + uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0 + with: + python-version: "3.12" + - name: Install build tools + run: | + python -m pip install --upgrade pip + pip install build twine + - name: Write pyproject.toml from dev.toml with the next version + run: python scripts/dev_release.py prepare + - name: Build distribution + run: python -m build + - name: Verify distribution metadata + run: python -m twine check dist/* + - name: Compare with the newest published wheel + id: compare + run: python scripts/dev_release.py changed dist + # A run that finishes after a newer push would otherwise publish older code as the newest release. + - name: Check that this commit is still the tip of dev + id: tip + run: | + tip="$(git ls-remote origin refs/heads/dev | cut -f1)" + if [ "$tip" = "$GITHUB_SHA" ]; then current=true; else current=false; fi + echo "current=$current" >> "$GITHUB_OUTPUT" + - name: Publish to PyPI + if: steps.compare.outputs.changed == 'true' && steps.tip.outputs.current == 'true' + env: + TWINE_USERNAME: __token__ + TWINE_PASSWORD: ${{ secrets.PYPI_API_TOKEN }} + run: python -m twine upload --non-interactive dist/* diff --git a/CLAUDE.md b/CLAUDE.md index ca1871a..15f161f 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -65,12 +65,13 @@ automation_file/ ## Branching & CI - `main` branch: stable releases, publishes `automation_file` to PyPI (version in `stable.toml`). -- `dev` branch: development, publishes `automation_file_dev` to PyPI (version in `dev.toml`). -- Keep `dependencies` and `[project.optional-dependencies]` (`dev`) in sync across both TOMLs. Backends (`boto3`, `azure-storage-blob`, `dropbox`, `paramiko`) and `PySide6` are first-class runtime deps — do not move them back under extras. -- **Version bumping is automatic.** A dedicated publish workflow bumps the patch in both `stable.toml` and `dev.toml`, builds, uploads to PyPI, then commits the bump back to `main` tagged as `vX.Y.Z`. Do not hand-bump before merging to `main`. The next publish run is skipped via a commit-message guard (`chore: bump version`), so the bump itself never re-triggers publishing. +- `dev` branch: development, publishes `automation_file_dev` to PyPI from CI. The version in `dev.toml` is only a floor. +- Keep `dependencies` and `[project.optional-dependencies]` (`dev`) in sync across both TOMLs; `tests/test_dev_toml_parity.py` fails when those, the entry points, `requires-python`, `[build-system]` or `[tool.setuptools]` differ. Backends (`boto3`, `azure-storage-blob`, `dropbox`, `paramiko`) and `PySide6` are first-class runtime deps — do not move them back under extras. +- **Version bumping is automatic.** A dedicated publish workflow bumps the patch in both `stable.toml` and `dev.toml`, builds, uploads to PyPI, then commits the bump back to `main` tagged as `vX.Y.Z`. Do not hand-bump before merging to `main`. The next publish run is skipped via a commit-message guard (`chore: bump version`), so the bump itself never re-triggers publishing. The dev channel takes its number from PyPI, so never hand-bump `dev.toml` either. - CI: GitHub Actions — a `lint` job on Ubuntu (Python 3.12), then `pytest` on Windows across Python 3.10 / 3.11 / 3.12 / 3.13 / 3.14. One workflow per branch: `.github/workflows/ci-dev.yml`, `.github/workflows/ci-stable.yml`. - CI steps: `lint` (ruff check + ruff format --check + mypy) → `pytest` with coverage → uploads `coverage.xml` as an artifact. -- Publishing lives in a separate workflow (`.github/workflows/publish.yml`) that runs on push to `main`: bumps both TOMLs, copies `stable.toml` to `pyproject.toml`, builds the sdist + wheel, `twine upload` via `PYPI_API_TOKEN`, then commits + tags + pushes and creates `gh release create v --generate-notes`. +- Stable publishing lives in a separate workflow (`.github/workflows/publish.yml`) that runs on push to `main`: bumps both TOMLs, copies `stable.toml` to `pyproject.toml`, builds the sdist + wheel, `twine upload` via `PYPI_API_TOKEN`, then commits + tags + pushes and creates `gh release create v --generate-notes`. +- Dev publishing is the `publish-dev` job at the end of `ci-dev.yml`. It runs only on a push to `dev`, after `lint` and `pytest` pass: `scripts/dev_release.py prepare` writes `pyproject.toml` from `dev.toml` with one patch above the newest `automation_file_dev` on PyPI, the job builds and runs `twine check`, and it uploads (same `PYPI_API_TOKEN`) only when the commit is still the tip of `dev` and the wheel differs from the newest published one. Nothing is committed back. - `pre-commit` is configured (`.pre-commit-config.yaml`): trailing-whitespace, eof-fixer, check-yaml, check-toml, check-added-large-files, ruff, ruff-format, mypy. Install with `pre-commit install` after cloning. ## Development diff --git a/architecture.md b/architecture.md index c9ed6ed..fad46f7 100644 --- a/architecture.md +++ b/architecture.md @@ -27,7 +27,8 @@ the CLI, over loopback TCP or HTTP servers, as MCP tools, or from the PySide6 GU | `automation_file/ui/` | PySide6 GUI: `launcher.launch_ui`, `main_window.MainWindow`, `worker.ActionWorker`, `log_widget.LogPanel`, `tabs/` (backend panels are grouped under `TransferTab`) | | `automation_file/utils/` | File discovery, fast find, grep, duplicate finder, backup rotation | | `automation_file/exceptions.py`, `logging_config.py` | `FileAutomationException` hierarchy; `file_automation_logger` (INFO+ to stderr, DEBUG+ to `$FILE_AUTOMATION_LOG_FILE` or `~/.automation_file/logs/FileAutomation.log`, opened on first use) | -| `stable.toml`, `dev.toml` | Packaging for `automation_file` and `automation_file_dev`. No `pyproject.toml` is committed; CI and publish copy one of these TOMLs into place | +| `stable.toml`, `dev.toml` | Packaging for `automation_file` and `automation_file_dev`. No `pyproject.toml` is committed; CI and the publish jobs write one of these TOMLs into place. Apart from the name, version and description they say the same thing (`tests/test_dev_toml_parity.py`) | +| `scripts/dev_release.py` | Release helper for the dev channel (standard library only): picks the next `automation_file_dev` version from PyPI and tells whether the built wheel differs from the newest published one | | `main_ui.py` | Development shortcut for `launch_ui()` | | `tests/`, `docs/`, `examples/mcp/` | pytest suite (fixtures in `tests/conftest.py`); Sphinx docs; MCP host configuration example | @@ -57,6 +58,14 @@ the CLI, over loopback TCP or HTTP servers, as MCP tools, or from the PySide6 GU default port 9945). - **GUI**: `launch_ui()`, `python -m automation_file ui` or `python main_ui.py`. - **Plugins**: third-party packages register actions through the entry-point group `automation_file.actions`. +- **PyPI packages**: `automation_file` (stable) and `automation_file_dev` (dev channel), both the same + import package. + - Stable: a push to `main` runs `publish.yml`, which bumps both TOMLs, builds from `stable.toml`, + uploads, and commits and tags the bump. + - Dev: the `publish-dev` job of `ci-dev.yml` runs after `lint` and `pytest` on a push to `dev`. It + builds from `dev.toml` and uploads when the commit is still the tip of `dev` and the wheel differs + from the newest published one. `scripts/dev_release.py` takes the version from PyPI (newest release + plus one patch, never below the version in `dev.toml`), so nothing is committed back. ## 4. Main flows @@ -163,7 +172,8 @@ ActionExecutor() → build_default_registry(): local + http + utils + drive comm is added; workspace X-12). - No `shell=True`; subprocesses use argument lists and a timeout (§ Security › General rules; › Subprocess execution). - Backends and PySide6 are first-class runtime dependencies. Keep `stable.toml` and `dev.toml` - dependencies in sync, and let the publish workflow bump versions (§ Branching & CI). + in sync (`tests/test_dev_toml_parity.py`), and let CI number both channels: never bump a version by + hand (§ Branching & CI). - Limits: cyclomatic complexity ≤ 15 (hard cap 20), cognitive complexity ≤ 15, functions ≤ 75 lines, ≤ 7 parameters, nesting ≤ 4, files ≤ 1000 lines (§ Code quality › Complexity & size). - Run `ruff check`, `ruff format --check`, `mypy` and `pytest` before committing (§ Development). @@ -173,6 +183,7 @@ ActionExecutor() → build_default_registry(): local + http + utils + drive comm - A top-level subpackage, backend or server module is added, removed or renamed. - CLI flags, subcommands, `[project.scripts]` in the TOMLs, or the entry-point group change. +- How either PyPI package is built or published changes. - The action format, the `auto_control` key, the registry build order, or plugin override semantics change. - Server defaults (host, port, auth, ACL, terminator) or HTTP routes change. - A §6 contract changes: PyBreeze invocation, the Windows double decode, the facade names TestPioneer uses. diff --git a/dev.toml b/dev.toml index 16de537..3fb5e0a 100644 --- a/dev.toml +++ b/dev.toml @@ -1,4 +1,6 @@ -# Dev release metadata — copied to pyproject.toml by the dev publish workflow. +# The dev channel, automation_file_dev. The publish-dev job of .github/workflows/ci-dev.yml builds it by +# writing this file to pyproject.toml (scripts/dev_release.py). The version below is a floor: CI publishes +# one patch above the newest release on PyPI and commits nothing back. [build-system] requires = ["setuptools>=77"] build-backend = "setuptools.build_meta" diff --git a/docs/updates/2026-10.md b/docs/updates/2026-10.md index 0fd61e6..468c8e2 100644 --- a/docs/updates/2026-10.md +++ b/docs/updates/2026-10.md @@ -91,3 +91,17 @@ Index and query commands: [README.md](README.md). New entries go at the end. - **Also**: the minimum becomes `je_action_core>=0.0.2` (`stable.toml`, `dev.toml`, and the CI lint job's install). `architecture.md` §6 describes the server's settings. - **Result / numbers**: 786 passed, 8 skipped. `ruff check` and `ruff format --check` pass. `mypy automation_file` reports no issues with the core installed normally. - **Files**: `automation_file/server/tcp_server.py`, `tests/test_tcp_server.py`, `stable.toml`, `dev.toml`, `.github/workflows/ci-{dev,stable}.yml`, `architecture.md`. + +## U-20261001-07 · 2026-10-01 · CI publishes automation_file_dev from the dev branch · #release #ci #X-13 + +- **What**: workspace X-13, decided by the owner: the dev channel is published by CI. `automation_file_dev` had last been uploaded by hand on 2025-11-04 (0.0.31). + - `ci-dev.yml` gets a `publish-dev` job. It runs on a push to `dev` after `lint` and `pytest` pass, never on a pull request or the schedule, and it keeps no checkout credentials. It installs `build` and `twine` the way `publish.yml` does. + - `scripts/dev_release.py prepare` writes `pyproject.toml` from `dev.toml` with the next version: one patch above the newest `X.Y.Z` release on PyPI, or above the version in `dev.toml` when that is higher. Nothing is committed back, so `dev` does not move under the sessions pushing to it; the version in `dev.toml` is a floor. + - `scripts/dev_release.py changed dist` compares the built wheel with the newest published one, ignoring what only a new version number changes (`RECORD`, `WHEEL`, the `Version:` line). A push that ships nothing new uploads nothing. + - The job uploads only while its commit is still the tip of `dev`, so a run that finishes after a newer push cannot publish older code as the newest release. + - The job writes `pyproject.toml` in its own checkout, after the tests. `pytest` already copies `dev.toml` to `pyproject.toml` before `pip install -e .`; ruff, mypy and pytest read `ruff.toml`, `mypy.ini` and `pytest.ini`, so neither file changes how they run. +- **`dev.toml` against `stable.toml`**: no drift. Dependencies, the `dev` extra, `[project.scripts]`, `requires-python`, `[build-system]` and `[tool.setuptools]` are identical; only the name, version and description differ. `tests/test_dev_toml_parity.py` is new and fails when they stop matching. The header of `dev.toml` now says what builds it. +- **Result / numbers**: a local build in a throwaway worktree gives a wheel of about 337 kB and an sdist of about 260 kB as 0.0.34; `twine check` passes and `changed` reports `true` against 0.0.31. The wheel's top level is `automation_file` and `tests`, the same as the stable 0.0.51 wheel. 813 passed, 8 skipped; `ruff check` and `ruff format --check` pass. +- **Docs**: `architecture.md` §2 and §3 describe both PyPI channels; `CLAUDE.md` › Branching & CI says how the dev channel is published. The READMEs and the Sphinx pages do not mention the dev package, so they are unchanged. +- **Files**: `.github/workflows/ci-dev.yml`, `scripts/dev_release.py`, `dev.toml`, `tests/test_dev_release.py`, `tests/test_dev_toml_parity.py`, `architecture.md`, `CLAUDE.md`, `progress.md`. +- **Open items**: `progress.md` #8 (both wheels ship `tests` as a top-level package). The other repositories with a `dev.toml` are workspace X-13. diff --git a/docs/updates/README.md b/docs/updates/README.md index 3c9dd74..0f51773 100644 --- a/docs/updates/README.md +++ b/docs/updates/README.md @@ -58,6 +58,7 @@ In the same commit: delete the item from `progress.md`, add a `#done` entry here | ID | Date | Title | Tags | Batch | |---|---|---|---|---| +| U-20261001-07 | 2026-10-01 | CI publishes automation_file_dev from the dev branch | #release #ci #X-13 | [2026-10](2026-10.md) | | U-20261001-06 | 2026-10-01 | The TCP server moves to je_action_core | #migration #socket-server #L-6 | [2026-10](2026-10.md) | | U-20261001-05 | 2026-10-01 | X-12: no action command loads packages, so the gate stays off | #decision #security #X-12 | [2026-10](2026-10.md) | | U-20261001-04 | 2026-10-01 | je_action_core comes from PyPI | #done #build #L-6 | [2026-10](2026-10.md) | @@ -90,5 +91,5 @@ In the same commit: delete the item from `progress.md`, add a `#done` entry here | File | Period | Entries | |---|---|---:| -| [2026-10.md](2026-10.md) | 2026-10 | 6 | +| [2026-10.md](2026-10.md) | 2026-10 | 7 | | [2026-09.md](2026-09.md) | 2026-09 | 21 | diff --git a/progress.md b/progress.md index d683eef..f6f1feb 100644 --- a/progress.md +++ b/progress.md @@ -5,3 +5,5 @@ Item numbers (`#n`) are never reused. Tags: [DECIDE] needs the owner's decision, Cross-repo and workspace items live in `D:\Codes\progress.md` (relevant here: X-12, X-13). ## Open + +- **#8** Both wheels install the test suite as a top-level `tests` package next to `automation_file`: `find = { namespaces = false }` (`stable.toml:71`, `dev.toml:73`) has no `include`, and `tests/__init__.py` makes `tests` a package (the `automation_file` 0.0.51 wheel on PyPI lists both in `top_level.txt`). Limit discovery to `automation_file*` in both TOMLs together (`tests/test_dev_toml_parity.py` compares the table) and check the built wheel. diff --git a/scripts/dev_release.py b/scripts/dev_release.py new file mode 100644 index 0000000..2cdde02 --- /dev/null +++ b/scripts/dev_release.py @@ -0,0 +1,134 @@ +"""Release helper for the dev channel, ``automation_file_dev`` on PyPI. + +The ``publish-dev`` job of ``.github/workflows/ci-dev.yml`` runs it once lint and the tests pass: + +* ``python scripts/dev_release.py prepare`` writes ``pyproject.toml`` from ``dev.toml`` with the + next version: one patch above the newest ``X.Y.Z`` release on PyPI, or above the version in + ``dev.toml`` when that is higher. Nothing is committed back, so the version in ``dev.toml`` is a + floor: ``publish.yml`` raises it on ``main`` with each stable release, and it is raised by hand + only when PyPI refuses a number (a deleted release keeps its number). +* ``python scripts/dev_release.py changed dist`` compares the wheel in ``dist`` with the newest + published one and writes ``changed=true`` or ``changed=false`` to ``$GITHUB_OUTPUT``, so a push + that ships nothing new publishes nothing. +""" + +from __future__ import annotations + +import hashlib +import io +import json +import os +import re +import sys +import zipfile +from pathlib import Path +from urllib.error import HTTPError +from urllib.request import urlopen + +Version = tuple[int, int, int] + +PYPI_JSON = "https://pypi.org/pypi/{name}/json" +TRUSTED_PREFIXES = ("https://pypi.org/", "https://files.pythonhosted.org/") +NAME_LINE = re.compile(r'^name\s*=\s*"([^"]+)"', re.MULTILINE) +VERSION_LINE = re.compile(r'^(version\s*=\s*)"(\d+)\.(\d+)\.(\d+)"', re.MULTILINE) +RELEASE = re.compile(r"^(\d+)\.(\d+)\.(\d+)$") +DIST_INFO = re.compile(r"^[^/]+\.dist-info/") +METADATA_VERSION = re.compile(rb"^Version: .*\r?\n", re.MULTILINE) +# Members that differ between two builds of the same sources. +VOLATILE = ("dist-info/RECORD", "dist-info/WHEEL") + + +def fetch(url: str) -> bytes: + """Return the body of a PyPI URL; any other host is refused.""" + if not url.startswith(TRUSTED_PREFIXES): + raise ValueError(f"refusing to fetch {url}") + # The prefix check above leaves only https URLs on PyPI's two hosts. + with urlopen(url, timeout=60) as response: # nosec B310 + return response.read() + + +def as_version(parts: tuple[str, ...]) -> Version: + """Turn the three captured number strings of a version into a comparable tuple.""" + major, minor, patch = (int(part) for part in parts) + return major, minor, patch + + +def published(name: str) -> dict[Version, str | None]: + """Map each ``X.Y.Z`` release of ``name`` on PyPI to its wheel URL, ``None`` without one.""" + try: + releases = json.loads(fetch(PYPI_JSON.format(name=name)))["releases"] + except HTTPError as error: + if error.code == 404: # not on PyPI yet: the first release + return {} + raise + found: dict[Version, str | None] = {} + for version, files in releases.items(): + match = RELEASE.match(version) + if match and files: + wheels = [item["url"] for item in files if item["packagetype"] == "bdist_wheel"] + found[as_version(match.groups())] = wheels[0] if wheels else None + return found + + +def next_version(floor: Version, released: dict[Version, str | None]) -> str: + """Return one patch above the highest of ``floor`` and the released versions.""" + major, minor, patch = max([floor, *released]) + return f"{major}.{minor}.{patch + 1}" + + +def prepare(root: Path) -> str: + """Write ``pyproject.toml`` from ``dev.toml`` with the next version and return that version.""" + text = (root / "dev.toml").read_text(encoding="utf-8") + name, floor = NAME_LINE.search(text), VERSION_LINE.search(text) + if name is None or floor is None: + raise SystemExit("dev.toml needs a name and an X.Y.Z version") + version = next_version(as_version(floor.groups()[1:]), published(name.group(1))) + project = VERSION_LINE.sub(rf'\g<1>"{version}"', text, count=1) + (root / "pyproject.toml").write_text(project, encoding="utf-8", newline="\n") + return version + + +def fingerprint(wheel: bytes) -> dict[str, str]: + """Map each wheel member to its SHA-256, leaving out what only a new version number changes.""" + members: dict[str, str] = {} + with zipfile.ZipFile(io.BytesIO(wheel)) as archive: + for member in archive.namelist(): + name = DIST_INFO.sub("dist-info/", member) + if name in VOLATILE: + continue + data = archive.read(member) + if name == "dist-info/METADATA": + data = METADATA_VERSION.sub(b"", data, count=1) + members[name] = hashlib.sha256(data).hexdigest() + return members + + +def changed(dist: Path) -> bool: + """Tell whether the wheel in ``dist`` ships anything the newest published wheel does not.""" + built = next(dist.glob("*.whl")) + released = published(built.name.split("-")[0]) + latest = released[max(released)] if released else None + if latest is None: + return True + return fingerprint(built.read_bytes()) != fingerprint(fetch(latest)) + + +def main(argv: list[str]) -> int: + """Run ``prepare`` or ``changed ``; return the process exit code.""" + if argv == ["prepare"]: + print(f"version={prepare(Path.cwd())}") + return 0 + if len(argv) == 2 and argv[0] == "changed": + line = f"changed={str(changed(Path(argv[1]))).lower()}" + print(line) + output = os.environ.get("GITHUB_OUTPUT") + if output: + with open(output, "a", encoding="utf-8") as handle: + handle.write(line + "\n") + return 0 + print(__doc__, file=sys.stderr) + return 2 + + +if __name__ == "__main__": + sys.exit(main(sys.argv[1:])) diff --git a/tests/test_dev_release.py b/tests/test_dev_release.py new file mode 100644 index 0000000..35bd189 --- /dev/null +++ b/tests/test_dev_release.py @@ -0,0 +1,168 @@ +"""``scripts/dev_release.py`` numbers and gates the ``automation_file_dev`` releases CI publishes. + +A wrong version is refused by PyPI (a number is never reused) and a wrong comparison either +publishes on every push or never again, so both are pinned here without touching the network. +""" + +from __future__ import annotations + +import importlib.util +import io +import re +import zipfile +from pathlib import Path + +import pytest + +REPO_ROOT = Path(__file__).resolve().parents[1] +SCRIPT = REPO_ROOT / "scripts" / "dev_release.py" +WORKFLOW = REPO_ROOT / ".github" / "workflows" / "ci-dev.yml" +FILES = "https://files.pythonhosted.org/" + + +def _load_script(): + spec = importlib.util.spec_from_file_location("dev_release", SCRIPT) + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + return module + + +dev_release = _load_script() + + +def _wheel(version: str, source: str = "VALUE = 1\n", requires: str = "requests>=2.31.0") -> bytes: + info = f"automation_file_dev-{version}.dist-info" + metadata = f"Name: automation_file_dev\nVersion: {version}\nRequires-Dist: {requires}\n" + buffer = io.BytesIO() + with zipfile.ZipFile(buffer, "w") as archive: + archive.writestr("automation_file/__init__.py", source) + archive.writestr(f"{info}/METADATA", metadata) + archive.writestr(f"{info}/RECORD", f"automation_file/__init__.py,sha256={version}\n") + archive.writestr(f"{info}/WHEEL", f"Generator: setuptools ({version})\n") + archive.writestr(f"{info}/licenses/LICENSE", "MIT\n") + return buffer.getvalue() + + +@pytest.mark.parametrize( + "floor, released, expected", + [ + ((0, 0, 33), {(0, 0, 33): None, (0, 0, 32): None}, "0.0.34"), + ((0, 0, 33), {(0, 0, 31): None}, "0.0.34"), + ((0, 0, 33), {(0, 0, 53): None}, "0.0.54"), + ((0, 1, 0), {(0, 0, 53): None}, "0.1.1"), + ((0, 0, 33), {}, "0.0.34"), + ], +) +def test_next_version_is_one_patch_above_the_floor_and_every_release(floor, released, expected): + assert dev_release.next_version(floor, released) == expected + + +def test_published_keeps_plain_releases_and_their_wheels(monkeypatch): + payload = ( + b'{"releases": {' + b'"0.0.30": [{"packagetype": "sdist", "url": "https://files.pythonhosted.org/a.tar.gz"}],' + b'"0.0.31": [{"packagetype": "sdist", "url": "https://files.pythonhosted.org/b.tar.gz"},' + b' {"packagetype": "bdist_wheel", "url": "https://files.pythonhosted.org/b.whl"}],' + b'"0.0.32.dev1": [{"packagetype": "bdist_wheel",' + b' "url": "https://files.pythonhosted.org/c.whl"}],' + b'"0.0.29": []}}' + ) + monkeypatch.setattr(dev_release, "fetch", lambda url: payload) + assert dev_release.published("automation_file_dev") == { + (0, 0, 30): None, + (0, 0, 31): "https://files.pythonhosted.org/b.whl", + } + + +def test_fetch_refuses_a_host_that_is_not_pypi(): + with pytest.raises(ValueError): + dev_release.fetch("https://example.com/automation_file_dev.whl") + + +def test_prepare_writes_pyproject_from_dev_toml_with_the_next_version(tmp_path, monkeypatch): + dev_toml = (REPO_ROOT / "dev.toml").read_text(encoding="utf-8") + (tmp_path / "dev.toml").write_text(dev_toml, encoding="utf-8") + asked = [] + monkeypatch.setattr( + dev_release, "published", lambda name: asked.append(name) or {(9, 9, 9): None} + ) + + assert dev_release.prepare(tmp_path) == "9.9.10" + + written = (tmp_path / "pyproject.toml").read_text(encoding="utf-8") + assert asked == ["automation_file_dev"] + assert written == dev_release.VERSION_LINE.sub(r'\g<1>"9.9.10"', dev_toml, count=1) + assert written.count('version = "9.9.10"') == 1 + + +def test_fingerprint_ignores_what_only_the_version_number_changes(): + assert dev_release.fingerprint(_wheel("0.0.33")) == dev_release.fingerprint(_wheel("0.0.34")) + + +@pytest.mark.parametrize( + "difference", [{"source": "VALUE = 2\n"}, {"requires": "requests>=2.32.0"}] +) +def test_fingerprint_sees_changed_code_and_changed_metadata(difference): + changed = dev_release.fingerprint(_wheel("0.0.34", **difference)) + assert dev_release.fingerprint(_wheel("0.0.33")) != changed + + +@pytest.mark.parametrize( + "latest, expected", + [ + ({}, True), + ({"source": "VALUE = 0\n"}, True), + ({"source": "VALUE = 1\n"}, False), + ], +) +def test_changed_compares_the_built_wheel_with_the_newest_published_one( + tmp_path, monkeypatch, latest, expected +): + (tmp_path / "automation_file_dev-0.0.34-py3-none-any.whl").write_bytes(_wheel("0.0.34")) + url = f"{FILES}automation_file_dev-0.0.33-py3-none-any.whl" + released = {(0, 0, 32): f"{FILES}old.whl", (0, 0, 33): url} if latest else {} + monkeypatch.setattr(dev_release, "published", lambda name: released) + monkeypatch.setattr( + dev_release, "fetch", lambda asked: _wheel("0.0.33", **latest) if asked == url else b"" + ) + + assert dev_release.changed(tmp_path) is expected + + +def test_changed_publishes_when_the_newest_release_has_no_wheel(tmp_path, monkeypatch): + (tmp_path / "automation_file_dev-0.0.34-py3-none-any.whl").write_bytes(_wheel("0.0.34")) + monkeypatch.setattr(dev_release, "published", lambda name: {(0, 0, 33): None}) + + assert dev_release.changed(tmp_path) is True + + +def test_main_writes_the_result_where_the_workflow_reads_it(tmp_path, monkeypatch): + output = tmp_path / "github_output" + monkeypatch.setenv("GITHUB_OUTPUT", str(output)) + monkeypatch.setattr(dev_release, "changed", lambda dist: False) + + assert dev_release.main(["changed", str(tmp_path)]) == 0 + assert output.read_text(encoding="utf-8") == "changed=false\n" + assert dev_release.main(["publish"]) == 2 + + +def _publish_job() -> str: + text = WORKFLOW.read_text(encoding="utf-8") + return re.split(r"^ publish-dev:\s*$", text, maxsplit=1, flags=re.MULTILINE)[1] + + +def test_the_workflow_publishes_only_a_tested_push_to_dev(): + job = _publish_job() + assert "needs: [lint, pytest]" in job + assert "if: github.event_name == 'push' && github.ref == 'refs/heads/dev'" in job + + +def test_the_workflow_uploads_only_a_changed_build_and_keeps_no_credentials(): + job = _publish_job() + upload = job.index("twine upload") + assert job.index("dev_release.py prepare") < job.index("python -m build") < upload + assert job.index("dev_release.py changed dist") < upload + assert job.index("git ls-remote origin refs/heads/dev") < upload + guard = "if: steps.compare.outputs.changed == 'true' && steps.tip.outputs.current == 'true'" + assert guard in job + assert "persist-credentials: false" in job diff --git a/tests/test_dev_toml_parity.py b/tests/test_dev_toml_parity.py new file mode 100644 index 0000000..acbfb14 --- /dev/null +++ b/tests/test_dev_toml_parity.py @@ -0,0 +1,62 @@ +"""``dev.toml`` describes the same package as ``stable.toml`` under another name. + +CI installs and tests ``dev.toml``, and the dev channel (``automation_file_dev``) is built from it, +while the stable release is built from ``stable.toml``. A dependency, an entry point or a packaging +rule on one side only ships a package that differs from the one the tests ran against. +""" + +from __future__ import annotations + +import sys +from pathlib import Path + +import pytest + +if sys.version_info >= (3, 11): + import tomllib +else: + import tomli as tomllib # declared in *.toml for Python<3.11 + +REPO_ROOT = Path(__file__).resolve().parents[1] + + +def _load(name: str) -> dict: + with (REPO_ROOT / name).open("rb") as handle: + return tomllib.load(handle) + + +STABLE_FILE = _load("stable.toml") +DEV_FILE = _load("dev.toml") +STABLE = STABLE_FILE["project"] +DEV = DEV_FILE["project"] + + +def test_package_names_differ(): + assert STABLE["name"] == "automation_file" + assert DEV["name"] == "automation_file_dev" + + +def test_runtime_dependencies_match(): + assert sorted(DEV["dependencies"]) == sorted(STABLE["dependencies"]) + + +def test_python_floor_matches(): + assert DEV["requires-python"] == STABLE["requires-python"] + + +def test_optional_dependency_groups_match(): + assert DEV.get("optional-dependencies", {}) == STABLE.get("optional-dependencies", {}) + + +@pytest.mark.parametrize("table", ["scripts", "gui-scripts", "entry-points"]) +def test_entry_points_match(table): + assert DEV.get(table, {}) == STABLE.get(table, {}) + + +def test_shipped_files_match(): + # Package discovery and package data decide which files reach the wheel. + assert DEV_FILE["tool"]["setuptools"] == STABLE_FILE["tool"]["setuptools"] + + +def test_build_backend_matches(): + assert DEV_FILE["build-system"] == STABLE_FILE["build-system"] From 8c565ddff64708d43d1e4c03135163a5537dacfb Mon Sep 17 00:00:00 2001 From: JeffreyChen Date: Thu, 1 Oct 2026 17:41:57 +0800 Subject: [PATCH 16/59] build: stop the wheels installing the test suite as a top-level package --- dev.toml | 3 ++- docs/updates/2026-10.md | 24 ++++++++++++++++++++++++ docs/updates/README.md | 3 ++- progress.md | 2 -- stable.toml | 3 ++- tests/test_dev_toml_parity.py | 9 +++++++++ 6 files changed, 39 insertions(+), 5 deletions(-) diff --git a/dev.toml b/dev.toml index 3fb5e0a..f21b8fb 100644 --- a/dev.toml +++ b/dev.toml @@ -71,4 +71,5 @@ automation_file_mcp = "automation_file.server.mcp_server:_cli" "Homepage" = "https://github.com/Integration-Automation/FileAutomation" [tool.setuptools.packages] -find = { namespaces = false } +# Only the library is installed. Without `include`, `tests` (it has an `__init__.py`) ships as a top-level package. +find = { namespaces = false, include = ["automation_file", "automation_file.*"] } diff --git a/docs/updates/2026-10.md b/docs/updates/2026-10.md index 468c8e2..4bb6fb8 100644 --- a/docs/updates/2026-10.md +++ b/docs/updates/2026-10.md @@ -105,3 +105,27 @@ Index and query commands: [README.md](README.md). New entries go at the end. - **Docs**: `architecture.md` §2 and §3 describe both PyPI channels; `CLAUDE.md` › Branching & CI says how the dev channel is published. The READMEs and the Sphinx pages do not mention the dev package, so they are unchanged. - **Files**: `.github/workflows/ci-dev.yml`, `scripts/dev_release.py`, `dev.toml`, `tests/test_dev_release.py`, `tests/test_dev_toml_parity.py`, `architecture.md`, `CLAUDE.md`, `progress.md`. - **Open items**: `progress.md` #8 (both wheels ship `tests` as a top-level package). The other repositories with a `dev.toml` are workspace X-13. + +## U-20261001-08 · 2026-10-01 · The wheels stop installing the test suite · #done #packaging #tests + +- **What**: `progress.md` #8, decided by the owner. Both wheels installed the test suite as a top-level `tests` package next to `automation_file`: package discovery had no `include`, and `tests/__init__.py` makes `tests` a package. `[tool.setuptools.packages]` in `stable.toml` and `dev.toml` now reads `find = { namespaces = false, include = ["automation_file", "automation_file.*"] }`. +- **Tests**: `tests/test_dev_toml_parity.py` `test_only_the_library_is_packaged` reads both TOMLs and fails when `include` is anything else; the previous table (`{ namespaces = false }`) fails it. It needs no setuptools. +- **Result / numbers**: wheels built in a throwaway worktree, the stable one from `stable.toml` copied to `pyproject.toml`, the dev one from `scripts/dev_release.py prepare`. + + | Wheel | Top level | Members | Under `tests/` | + |---|---|---:|---:| + | `automation_file` 0.0.51 (PyPI) | `automation_file`, `tests` | 250 | 83 | + | `automation_file` built from `stable.toml` | `automation_file` | 167 | 0 | + | `automation_file_dev` 0.0.34 (PyPI) | `automation_file`, `tests` | 254 | 87 | + | `automation_file_dev` built as 0.0.35 | `automation_file` | 167 | 0 | + + - Every member that went away is under `tests/`; no member was added, and the 167 that remain have the same names as in the published wheels. + - `entry_points.txt` is unchanged (`automation_file_mcp = automation_file.server.mcp_server:_cli`). Neither wheel carries package data, before or after. + - The dev wheel's `METADATA` headers match 0.0.34 apart from the version. + - No module in the wheels imports `tests`, and all 159 modules import from an extracted wheel with no `tests` package beside it. + - `twine check` passes for both wheels and both sdists; `dev_release.py changed` reports `true` against 0.0.34, so the next push to `dev` publishes. + - The sdists still carry `tests/`, which setuptools adds to source distributions by default. + - 815 passed, 8 skipped. `ruff check` and `ruff format --check` pass; `mypy automation_file` reports no issues with je_action_core installed normally. +- **Docs**: none of `architecture.md`, `CLAUDE.md`, the READMEs or the Sphinx pages says what the wheel contains, so they are unchanged. +- **Files**: `stable.toml`, `dev.toml`, `tests/test_dev_toml_parity.py`, `progress.md`. +- **Open items**: none. The stable channel gets the change with the next release from `main`. diff --git a/docs/updates/README.md b/docs/updates/README.md index 0f51773..138c665 100644 --- a/docs/updates/README.md +++ b/docs/updates/README.md @@ -58,6 +58,7 @@ In the same commit: delete the item from `progress.md`, add a `#done` entry here | ID | Date | Title | Tags | Batch | |---|---|---|---|---| +| U-20261001-08 | 2026-10-01 | The wheels stop installing the test suite | #done #packaging #tests | [2026-10](2026-10.md) | | U-20261001-07 | 2026-10-01 | CI publishes automation_file_dev from the dev branch | #release #ci #X-13 | [2026-10](2026-10.md) | | U-20261001-06 | 2026-10-01 | The TCP server moves to je_action_core | #migration #socket-server #L-6 | [2026-10](2026-10.md) | | U-20261001-05 | 2026-10-01 | X-12: no action command loads packages, so the gate stays off | #decision #security #X-12 | [2026-10](2026-10.md) | @@ -91,5 +92,5 @@ In the same commit: delete the item from `progress.md`, add a `#done` entry here | File | Period | Entries | |---|---|---:| -| [2026-10.md](2026-10.md) | 2026-10 | 7 | +| [2026-10.md](2026-10.md) | 2026-10 | 8 | | [2026-09.md](2026-09.md) | 2026-09 | 21 | diff --git a/progress.md b/progress.md index f6f1feb..d683eef 100644 --- a/progress.md +++ b/progress.md @@ -5,5 +5,3 @@ Item numbers (`#n`) are never reused. Tags: [DECIDE] needs the owner's decision, Cross-repo and workspace items live in `D:\Codes\progress.md` (relevant here: X-12, X-13). ## Open - -- **#8** Both wheels install the test suite as a top-level `tests` package next to `automation_file`: `find = { namespaces = false }` (`stable.toml:71`, `dev.toml:73`) has no `include`, and `tests/__init__.py` makes `tests` a package (the `automation_file` 0.0.51 wheel on PyPI lists both in `top_level.txt`). Limit discovery to `automation_file*` in both TOMLs together (`tests/test_dev_toml_parity.py` compares the table) and check the built wheel. diff --git a/stable.toml b/stable.toml index 02f14e6..5e45eab 100644 --- a/stable.toml +++ b/stable.toml @@ -69,4 +69,5 @@ automation_file_mcp = "automation_file.server.mcp_server:_cli" "Homepage" = "https://github.com/Integration-Automation/FileAutomation" [tool.setuptools.packages] -find = { namespaces = false } +# Only the library is installed. Without `include`, `tests` (it has an `__init__.py`) ships as a top-level package. +find = { namespaces = false, include = ["automation_file", "automation_file.*"] } diff --git a/tests/test_dev_toml_parity.py b/tests/test_dev_toml_parity.py index acbfb14..e8b1121 100644 --- a/tests/test_dev_toml_parity.py +++ b/tests/test_dev_toml_parity.py @@ -58,5 +58,14 @@ def test_shipped_files_match(): assert DEV_FILE["tool"]["setuptools"] == STABLE_FILE["tool"]["setuptools"] +@pytest.mark.parametrize("metadata", [STABLE_FILE, DEV_FILE], ids=["stable.toml", "dev.toml"]) +def test_only_the_library_is_packaged(metadata): + # ``tests`` has an ``__init__.py``, so discovery without ``include`` installs the test suite as a + # top-level package next to ``automation_file``. + find = metadata["tool"]["setuptools"]["packages"]["find"] + assert find["include"] == ["automation_file", "automation_file.*"] + assert find["namespaces"] is False + + def test_build_backend_matches(): assert DEV_FILE["build-system"] == STABLE_FILE["build-system"] From 03fd4d7d0fddfcff35cca432c559c58962abe71e Mon Sep 17 00:00:00 2001 From: JeffreyChen Date: Thu, 1 Oct 2026 20:20:16 +0800 Subject: [PATCH 17/59] build: keep the test suite out of the source distributions --- MANIFEST.in | 3 +++ architecture.md | 1 + docs/updates/2026-10.md | 23 +++++++++++++++++++++++ docs/updates/README.md | 3 ++- tests/test_sdist_manifest.py | 32 ++++++++++++++++++++++++++++++++ 5 files changed, 61 insertions(+), 1 deletion(-) create mode 100644 MANIFEST.in create mode 100644 tests/test_sdist_manifest.py diff --git a/MANIFEST.in b/MANIFEST.in new file mode 100644 index 0000000..6e1603c --- /dev/null +++ b/MANIFEST.in @@ -0,0 +1,3 @@ +# The source distributions carry no tests. setuptools adds tests/test*.py to an sdist by default, +# whatever package discovery in stable.toml and dev.toml says (tests/test_sdist_manifest.py). +prune tests diff --git a/architecture.md b/architecture.md index fad46f7..01dcd76 100644 --- a/architecture.md +++ b/architecture.md @@ -28,6 +28,7 @@ the CLI, over loopback TCP or HTTP servers, as MCP tools, or from the PySide6 GU | `automation_file/utils/` | File discovery, fast find, grep, duplicate finder, backup rotation | | `automation_file/exceptions.py`, `logging_config.py` | `FileAutomationException` hierarchy; `file_automation_logger` (INFO+ to stderr, DEBUG+ to `$FILE_AUTOMATION_LOG_FILE` or `~/.automation_file/logs/FileAutomation.log`, opened on first use) | | `stable.toml`, `dev.toml` | Packaging for `automation_file` and `automation_file_dev`. No `pyproject.toml` is committed; CI and the publish jobs write one of these TOMLs into place. Apart from the name, version and description they say the same thing (`tests/test_dev_toml_parity.py`) | +| `MANIFEST.in` | Keeps `tests/` out of both source distributions (`tests/test_sdist_manifest.py`); package discovery in the TOMLs already keeps it out of the wheels | | `scripts/dev_release.py` | Release helper for the dev channel (standard library only): picks the next `automation_file_dev` version from PyPI and tells whether the built wheel differs from the newest published one | | `main_ui.py` | Development shortcut for `launch_ui()` | | `tests/`, `docs/`, `examples/mcp/` | pytest suite (fixtures in `tests/conftest.py`); Sphinx docs; MCP host configuration example | diff --git a/docs/updates/2026-10.md b/docs/updates/2026-10.md index 4bb6fb8..8046ae6 100644 --- a/docs/updates/2026-10.md +++ b/docs/updates/2026-10.md @@ -129,3 +129,26 @@ Index and query commands: [README.md](README.md). New entries go at the end. - **Docs**: none of `architecture.md`, `CLAUDE.md`, the READMEs or the Sphinx pages says what the wheel contains, so they are unchanged. - **Files**: `stable.toml`, `dev.toml`, `tests/test_dev_toml_parity.py`, `progress.md`. - **Open items**: none. The stable channel gets the change with the next release from `main`. + +## U-20261001-09 · 2026-10-01 · The source distributions stop carrying the tests · #done #packaging #tests + +- **What**: decided by the owner after U-20261001-08 (workspace X-13): the sdists carry no tests either. setuptools adds `tests/test*.py` to a source distribution by default, whatever package discovery says, so `automation_file_dev` 0.0.35 still shipped 84 test modules in its sdist. A new `MANIFEST.in` holds one command, `prune tests`; it applies to both channels because both are built from the same tree. +- **Tests**: `tests/test_sdist_manifest.py` reads `MANIFEST.in` as text (nothing is built at test time). It fails when `prune tests` is missing, and when a command after it (`include`, `recursive-include`, `global-include`, `graft`) could put files back. +- **Result / numbers**: sdist and wheel built with `python -m build` in a throwaway worktree, before and after the change, the stable pair from `stable.toml` copied to `pyproject.toml`, the dev pair from `scripts/dev_release.py prepare` (0.0.36). + + | Sdist | Members | Root files | `automation_file/` | `*.egg-info/` | `tests/` | + |---|---:|---:|---:|---:|---:| + | `automation_file_dev` 0.0.35 (PyPI) | 256 | 5 | 161 | 6 | 84 | + | `automation_file_dev` built before the change | 256 | 5 | 161 | 6 | 84 | + | `automation_file_dev` built after | 173 | 6 | 161 | 6 | 0 | + | `automation_file` built before the change | 256 | 5 | 161 | 6 | 84 | + | `automation_file` built after | 173 | 6 | 161 | 6 | 0 | + + - All 84 members that went away are under `tests/`. The one member added is `MANIFEST.in` itself; the root files are now `LICENSE`, `MANIFEST.in`, `PKG-INFO`, `README.md`, `pyproject.toml`, `setup.cfg`. + - Before and after, the only member whose bytes differ is `*.egg-info/SOURCES.txt`, which lists the members. The sdists shrink from about 261 kB to about 191 kB. + - Both wheels are unchanged: 167 members each, every member byte-identical before and after, `RECORD` included. The dev wheel differs from the published 0.0.35 wheel only in its version, so the `publish-dev` job uploads nothing for this commit; the new sdist goes out with the next release. + - `python -m build` builds the wheel from the sdist, so the pruned sdists still build. `twine check` passes for all four files. + - 817 passed, 8 skipped. `ruff check` and `ruff format --check` pass. +- **Docs**: `architecture.md` §2 lists `MANIFEST.in`. `CLAUDE.md`, the READMEs and the Sphinx pages do not say what an sdist contains, so they are unchanged. +- **Files**: `MANIFEST.in`, `tests/test_sdist_manifest.py`, `architecture.md`. +- **Open items**: none. The stable channel gets the change with the next release from `main`. diff --git a/docs/updates/README.md b/docs/updates/README.md index 138c665..9150853 100644 --- a/docs/updates/README.md +++ b/docs/updates/README.md @@ -58,6 +58,7 @@ In the same commit: delete the item from `progress.md`, add a `#done` entry here | ID | Date | Title | Tags | Batch | |---|---|---|---|---| +| U-20261001-09 | 2026-10-01 | The source distributions stop carrying the tests | #done #packaging #tests | [2026-10](2026-10.md) | | U-20261001-08 | 2026-10-01 | The wheels stop installing the test suite | #done #packaging #tests | [2026-10](2026-10.md) | | U-20261001-07 | 2026-10-01 | CI publishes automation_file_dev from the dev branch | #release #ci #X-13 | [2026-10](2026-10.md) | | U-20261001-06 | 2026-10-01 | The TCP server moves to je_action_core | #migration #socket-server #L-6 | [2026-10](2026-10.md) | @@ -92,5 +93,5 @@ In the same commit: delete the item from `progress.md`, add a `#done` entry here | File | Period | Entries | |---|---|---:| -| [2026-10.md](2026-10.md) | 2026-10 | 8 | +| [2026-10.md](2026-10.md) | 2026-10 | 9 | | [2026-09.md](2026-09.md) | 2026-09 | 21 | diff --git a/tests/test_sdist_manifest.py b/tests/test_sdist_manifest.py new file mode 100644 index 0000000..74d258f --- /dev/null +++ b/tests/test_sdist_manifest.py @@ -0,0 +1,32 @@ +"""The source distributions carry no test suite. + +Package discovery keeps ``tests`` out of the wheels (``test_dev_toml_parity.py``), but setuptools +adds ``tests/test*.py`` to an sdist by default whatever discovery says. ``MANIFEST.in`` prunes the +directory for both channels. The file is read as text: nothing is built at test time. +""" + +from __future__ import annotations + +from pathlib import Path + +REPO_ROOT = Path(__file__).resolve().parents[1] +TEST_DIRECTORY = Path(__file__).resolve().parent.name +# MANIFEST.in commands that put files into the sdist. +ADDING = {"include", "recursive-include", "global-include", "graft"} + + +def _commands() -> list[list[str]]: + """Return each ``MANIFEST.in`` command as its words, without comments and blank lines.""" + lines = (REPO_ROOT / "MANIFEST.in").read_text(encoding="utf-8").splitlines() + return [line.split() for line in lines if line.strip() and not line.lstrip().startswith("#")] + + +def test_sdist_prunes_the_test_directory(): + assert ["prune", TEST_DIRECTORY] in _commands() + + +def test_nothing_puts_the_tests_back(): + # Commands apply in order, so only one that adds files after the prune could bring tests back. + commands = _commands() + after_prune = commands[commands.index(["prune", TEST_DIRECTORY]) + 1 :] + assert [command for command in after_prune if command[0] in ADDING] == [] From 78cfc1d55ae6fc18027d72027bc4196c8d2ec56b Mon Sep 17 00:00:00 2001 From: JeffreyChen Date: Thu, 1 Oct 2026 20:30:04 +0800 Subject: [PATCH 18/59] ci: install hash-locked build tools in the publish jobs --- .github/dependabot.yml | 6 +- .github/requirements/publish.in | 7 + .github/requirements/publish.txt | 468 +++++++++++++++++++++++++++++++ .github/workflows/ci-dev.yml | 5 +- .github/workflows/publish.yml | 5 +- CLAUDE.md | 1 + architecture.md | 4 + docs/updates/2026-10.md | 25 ++ docs/updates/README.md | 3 +- progress.md | 2 + tests/test_workflow_actions.py | 80 +++++- 11 files changed, 598 insertions(+), 8 deletions(-) create mode 100644 .github/requirements/publish.in create mode 100644 .github/requirements/publish.txt diff --git a/.github/dependabot.yml b/.github/dependabot.yml index 201868a..1af5268 100644 --- a/.github/dependabot.yml +++ b/.github/dependabot.yml @@ -6,7 +6,11 @@ version: 2 updates: - package-ecosystem: "pip" # See documentation for possible values - directory: "/" # Location of package manifests + # "/" holds requirements.txt and dev_requirements.txt. .github/requirements + # holds the hash-locked tools of the publish jobs (publish.in, publish.txt). + directories: # Locations of package manifests + - "/" + - "/.github/requirements" # Updates belong on dev: main is the released branch and merging into it # publishes, so a PR opened against main just sits there. target-branch: "dev" diff --git a/.github/requirements/publish.in b/.github/requirements/publish.in new file mode 100644 index 0000000..205044e --- /dev/null +++ b/.github/requirements/publish.in @@ -0,0 +1,7 @@ +# Tools the two publish jobs run: publish-dev in ci-dev.yml and publish in publish.yml. Their version +# scripts use the standard library only. publish.txt is generated from this file, for the Python 3.12 +# on Linux those jobs set up: +# uv pip compile .github/requirements/publish.in --generate-hashes --python-version 3.12 --python-platform x86_64-manylinux_2_28 --only-binary :all: --exclude-newer -o .github/requirements/publish.txt +# --exclude-newer leaves out releases under a week old, the wait dependabot.yml sets. +build +twine diff --git a/.github/requirements/publish.txt b/.github/requirements/publish.txt new file mode 100644 index 0000000..5eec725 --- /dev/null +++ b/.github/requirements/publish.txt @@ -0,0 +1,468 @@ +# This file was autogenerated by uv via the following command: +# uv pip compile .github/requirements/publish.in --generate-hashes --python-version 3.12 --python-platform x86_64-manylinux_2_28 --only-binary :all: --exclude-newer 2026-09-24T00:00:00Z -o .github/requirements/publish.txt +build==1.6.1 \ + --hash=sha256:51cc11666391ab6f092070437ac747002ff46f3e4113a3622177ee6b488bfc53 \ + --hash=sha256:ecd351a4be9d35a9eaaba244a7687143c9c7d4aea6ac964e7e7ddab20cbcf4e7 + # via -r .github/requirements/publish.in +certifi==2026.7.22 \ + --hash=sha256:62f22742b58a1a33014a2b6b706588a8d7e2a88ae7bd1a6ebe8c992928483775 \ + --hash=sha256:741e2c3b351ddf169a738da9f2c048608ff7f2c5cc02f1ebc6b118bb090d5d55 + # via requests +cffi==2.1.1 \ + --hash=sha256:046bfc24911b37851ee1b51aab8bffe713d89c68c6a057b09484ce9fd5f69b4e \ + --hash=sha256:06c72bb76605a4b0cd0aad6930b69d4baf7dd5d806cfc409b824191099700e66 \ + --hash=sha256:0beceaabe56af686895136a2de78db54ecd8e4046b236b8fd6d6cb61389e9bf2 \ + --hash=sha256:154852545011f779917b11c78db2358d095da62a9a172b78ad0a583ee5adc0d0 \ + --hash=sha256:194cffa889098ced9976c3fc6340305e43f6303657d298da55366907c05c22d6 \ + --hash=sha256:19ee6127ee34de7d83ce3d371ebc5ed91addbdcc39f9ab15ce4eb35a4e534971 \ + --hash=sha256:1a18a57b58cfb21fc28d72e876acf10eaed67a1ed96226f92af4df681d571c4c \ + --hash=sha256:1aa5645c30469b09530c4ebca77ebf8f17618293c58f8549cb1a543a50236e7d \ + --hash=sha256:1dea0e4d7d4f11f619fe8c1d76caf49e24405b4b5743c0e3be16a500ecd930c9 \ + --hash=sha256:208f941bb9d18e768138677f0a6d2ce01f590df56043dda1df1535ac57c88517 \ + --hash=sha256:210019b6c7cf07f081b4c54635c8cf744377001350e29cc0f81c4377b4797735 \ + --hash=sha256:246fa40ce8645a614ff682e0b70f37134e460eaf93a775e0cbe3cca585a67a80 \ + --hash=sha256:25792eac27877609e7bb06d42ff88278a6624fff2ba9bbb523c09616b117e80f \ + --hash=sha256:27350daa11d4f10c540e6e89dada4c54feb7256ad03e9a4dc075ebad7ba360d1 \ + --hash=sha256:28907ab9bfb6aa13184cfc17c6b8e1023c5ab6fd7076d8c20a35e59fe04f8f29 \ + --hash=sha256:2ae64be792b8966f2c69538199728b290e34726562896df1e5dc8ffd8d8188e8 \ + --hash=sha256:31348097ff5bbe827ccc41795d4dd099d9f0625e7def00ee653c137a490c2a6c \ + --hash=sha256:3143d81e29e1e20a9ce10901ec369012947876596f75a222235965f2b7ae832e \ + --hash=sha256:3222ba5d678f80a030e6afbcc33dc1ae5cb45facabb61cee2c7016b8432fde48 \ + --hash=sha256:3311ed60d36f83378794e1009ac6258bafbf81f7888b4caa7b35a521e3f95813 \ + --hash=sha256:334644fbac4eff73d985a17a91226df55d0f394160c4cfb880e084c8f7161cac \ + --hash=sha256:34e261f78cb6ceaaa36f42f2613f4380d94d9c759a9c73c769ee6e0247364632 \ + --hash=sha256:363e05fa78e15116c3c32c210ee36884fd6b9afa6d440e47112c3bd511d64cb6 \ + --hash=sha256:398aff33cee2767e3e781d2554c54bd0dff386bb437581e0d8011fde1a942ec1 \ + --hash=sha256:3d22a20b1fb1632cc72c22f95f7b0d2961c3e1c235f245ba4c606c4771035659 \ + --hash=sha256:42a494cee34437f05546455144f2b5d9ac09b1face62bcfce597d2e521066688 \ + --hash=sha256:42e2f76b9455f5a9a844f770bf3e200ed3da0e15f5df3db9c31fe80b04b3d004 \ + --hash=sha256:42f6930c31dc7f50732c9ae793c2786c7b6b044195967bbdde40bb9be81c4cc0 \ + --hash=sha256:456a61fa52d579ebf9df2e9552ead5129855dbaff6c1e5a9b1bc408809bdc062 \ + --hash=sha256:471cee653ae88de62096552e6d24ccb4a5adb8c8c9f10b5054d0122c15bf2779 \ + --hash=sha256:49cbc70e6542d4ccccb936558d1064a8012541e78f821f955cff24e357776c94 \ + --hash=sha256:4a7c934f7360e8cd64fe9efadcbd10c7c6364f531e432b9a4bf5ccbc9e0e8b50 \ + --hash=sha256:4be96343e422f2dfcd12ab5c9f5aebe03f82f737c6bffeca6830b3875cb44aab \ + --hash=sha256:4f42141fc14250de6dde5ee7ea4432be017252d91f19c5ad043c084cea629cac \ + --hash=sha256:507a24c282e0f42f8ed737cf048572cbf580468da5555764a8331735e9c736b6 \ + --hash=sha256:51b31d1c98274844cfd7838ce00bfc27c7423a4dc00fc0772fc3331c2cc90676 \ + --hash=sha256:58acb8ab8e295e6c5ea12f888cbb13cf21511ef2a3303a23f4325c29d17fe5c1 \ + --hash=sha256:5a59cc1c4442bc3d5c703bf720b51138d0bfc173618807c9ee2490a7541dd3d9 \ + --hash=sha256:5bb4e7ea95dcd6a014a6fef62e62467d67d8e582326443f3d68e71d6320a9fcf \ + --hash=sha256:5c58fe613dc5e5336357eff555824a314d8e43282600435c8d1cb6a7a2fedd13 \ + --hash=sha256:5e7cecbaadb83884793e05828cee59b210b24583b9c7425d0ba6a754fe22eb4e \ + --hash=sha256:616f097f2fe415bc92a247f02e11f634e1f9e9a83d327e3c915c15089c87869e \ + --hash=sha256:63bbfd5ded17c4840ac07cd8f1c21ba9d9708141f840b324f422f41b207e3973 \ + --hash=sha256:64faea20f4e2613363a1a9b9c7dd73058f3ecd00133a511e72ad7c511658f527 \ + --hash=sha256:661c298b4821edebead0c91edd2b00374d67ad7c5a1f7a91d4442633b79d6a72 \ + --hash=sha256:68e62fe11f30d5ca8289242866f0a5291402d8529ca2178ab8afc5c9694ae890 \ + --hash=sha256:6a8dddef476fab96d066d578fc88526767b836ab5ab21754e1d5bf3879c31c7c \ + --hash=sha256:6e192623c49c94421616a5778fba35cf0d5a8d000650c1967ef4448ee5cdd990 \ + --hash=sha256:7225e4514edb64eb6740324353e0da0711954fd8d7da4576755b1c6e09b697cd \ + --hash=sha256:75f80557d1389eddbd0de2681f6a390a0c5338c31ddaa821381c203fc3fd50d9 \ + --hash=sha256:770de9db11e84213beec501cfcaa013b019820ca881e03344dea5844f7876d94 \ + --hash=sha256:7750c6449dff7864bb9bb27ddfb0267756189201a3afc911d82b3caacd70dfc3 \ + --hash=sha256:7bde5e4cc5c10140859842b9d383af292b22639a4dffb725314baf45968cef80 \ + --hash=sha256:7ce713ace7c0e4520535b42b77eaa742c16dab813978064913e5a3cf82973b41 \ + --hash=sha256:7da0c5eff80f0197f3b3d1232ec5a682a9325f4ae9016a78f5f5ca35f9ced1f5 \ + --hash=sha256:7dbb61fe3a7699468030f71bbe5f8a0e326a151daa91beb11a6fc1f980c55e1c \ + --hash=sha256:811bd1e21d32de12efca32393a0ab3f5133b54fce9bd44b8bd77ab07da14bf6a \ + --hash=sha256:8ef53b2de9bcb9197d31854256575d59dbac0cba72ac627bb291ef5eceb74be4 \ + --hash=sha256:937c0052c05a31ca1daf18de3158eed4dbfcb9cc107adbea227728d647be701e \ + --hash=sha256:9d2055050ea716bd38b7f7f1579c275386646b4894c155a3e2f3cd62ed41b7c6 \ + --hash=sha256:9f8d177621de5cb38ee3e731eda45d421db093ec0739f46a5594babda7987a98 \ + --hash=sha256:a2d7755bef5a12ed488f4ef1f1b69ee9191d7396083b755a5d2295f6edb4768b \ + --hash=sha256:a48d62ab9d6f4f98c983223a547af44be6ca3691074c31cecced6facd3ba2dc1 \ + --hash=sha256:a4f00aa42f75d6e4595e8866e748cc1705adc0cddfeb2ca86d0d03993d63ba03 \ + --hash=sha256:a6e721d4b0e45d5b65e87534470e67b18dcd092c83f68fba09f152b9cbc061af \ + --hash=sha256:a730a083190634c65cca36ba5f489531576ebd79bcd5c8e172130f6453127231 \ + --hash=sha256:a931079504ecc49efed7744c476a5c343a92fabf66dec2db95edb1b2fdc770e2 \ + --hash=sha256:aa9511c62d14da7aacc9b4bf51f3f697a621e83b2d6919008243c3aad168eea3 \ + --hash=sha256:ab36d55f9ed2d067327667c2fea18dda018eb628dd6347aa01dda6cf1f5d3836 \ + --hash=sha256:ad2c86c495b899d862ea0f4b42891b8713a3bd45dd4105c7fd51c2a72f39f3a5 \ + --hash=sha256:aeae0e330c9f6acd681f647d46cefd30c29f93e3392882e792e82080c9691399 \ + --hash=sha256:b0431303acaea1089ad4b3e9ce4e6518193def1118d4073ca848635ee4ea2e96 \ + --hash=sha256:b5bdfd1c873d4e093aabc0ca84c4ca6dbc4f752afb5c86f146d9742580c9da2e \ + --hash=sha256:baed1e86cc735622097354b9d1281406caf42ff42a886d29faa8e8d1630333be \ + --hash=sha256:c1453022f490d2459a11819d83ad1d586e9ff65a12ac3e705ffebd46d3685dcf \ + --hash=sha256:c26608d2222fb1e94487e4a387d85f13eb55d5ed725cb25a0c589ac4ee60e7bc \ + --hash=sha256:c7659f22557c5a0bc4855cd635f55edec690cc008a40768527762cb9fb263455 \ + --hash=sha256:c8c69575568085ba0b1b10c0249d779a214aea6f6522e949a0fc9fb0fcb449d0 \ + --hash=sha256:c8d2c9fd1f2d16f780d15127abb050d13d1a76c03a4bd87d7e4980e45e511e12 \ + --hash=sha256:ca82be1a1d406ecfe1d25dc16cb33488e5a16bf4438c9fb590484ea29d92478b \ + --hash=sha256:cc572dace3f60ef98d7b12ff411d20f5362feb31a0439eab0085bbfd349982d7 \ + --hash=sha256:d18e5ac0f2f03f4f518d3e23db0f0cad7faa1da8620e9c09461d443bbf6e6692 \ + --hash=sha256:d28630f5854ab07ab1fd4aba756de52326c82e6be15d414b12793f1975048b54 \ + --hash=sha256:d9c275eaacd24aa73f94ffd6de08fc3f932424d8b6c376f4bed7cde376fe7bc3 \ + --hash=sha256:da0e573f9f97159390c89d9f1a9e41908b66d408cc5b58d08cf3847d844c531b \ + --hash=sha256:dd31f52ea1086513bb9df30f8fcee9b8918323ae067a3d5b78bc826a000712be \ + --hash=sha256:dddad92b554513a31f272570678ba307fb9f618f05e3d4a5eacafff9eae03e1d \ + --hash=sha256:df423d40ee8654634421812bc3b196da3f9bd7d32929da813f8394c4348a5358 \ + --hash=sha256:df913725b79db7bcf03448f36b7bf8815363417d5b58deecf9305e3e30f0f21a \ + --hash=sha256:e0bcb7e0f677f543555d2adff3bf19c05f66cdb4796e5ff602442ab2fe3c4ef7 \ + --hash=sha256:e2d65b31f36619cda3999b78b2aa9632e76b78448e7a56fc4240824200e7c4fc \ + --hash=sha256:e6e8cff14d6fb0be70a09c0bdc58096f501952d04624ebf867e0e56da2df8960 \ + --hash=sha256:f16c709686a78c727bbbf059f92b0bf41c6fc60deec706d2dc19f529175a6125 \ + --hash=sha256:f24fb43132a4c6b4cb4eb029492919b2db645be6808d738f244fd146c03c32cb \ + --hash=sha256:f53e442b08449d42821fa4a4fba000095af9f62742a500f978a9f557ec44339a \ + --hash=sha256:f5cfbc5fe74540d335175b656c725d74d90e3730c626d92575eea35029d9afaa \ + --hash=sha256:f81b3b8f3d4e343550fa4baa0e479bba9f2d29ce9c2e9b51d1ce1718d7442fcf \ + --hash=sha256:f8ec5e643a9a937f64e1999eb9f75d072263751912dc5cd06d3c85f8f44be7c3 \ + --hash=sha256:fb92203a88b3d3053034db775110081c49d28be6551923805e039924093761e4 \ + --hash=sha256:fcd22650c908d7b7da162bbfaab594a1227a15d1643a98c68b122ac642fa2264 + # via cryptography +charset-normalizer==3.5.1 \ + --hash=sha256:00668ebb0609751758682eb0b5857e7c35b9f00e84dfdef062e103244ec94d45 \ + --hash=sha256:012a22b88a77ca2e59b98ac5889b0deb604147666032f45e6d6e217634d2550d \ + --hash=sha256:01e93745f7f219b703b60ba7afead36cfc4242782be5af484673fc500df12da5 \ + --hash=sha256:04368edf83514385ffc3e1cfd4546e595f4f1272dd23ba437a93a9cc3741d47b \ + --hash=sha256:0722590aabf9dc6a6c0343d523c05458fa2b5047dbe6302fd526bb570600753f \ + --hash=sha256:07ffd07412fc5d5e84cd8952acf9ff7e4ed7a708e69d1bada19d8ba91711353f \ + --hash=sha256:09a7bba9f739468c8e78c36a75c33768e53cb1959fc638f510454c14683f00d5 \ + --hash=sha256:0b2b1b3fa5670c127b246df1d0c059defd41f689a868a3b9d79df9b1cac42d22 \ + --hash=sha256:0c6dfb5ca6723eeed15aa8e564a014d69fcb8812f94eef11fe3631e0508199f5 \ + --hash=sha256:0d929fc574b4d6fd9e7c0f5c2ede8716a41911923aa7fa5fce38e0818aa4a1ac \ + --hash=sha256:13e3afe97712e8887cd516e960c63f0b93122971e5b5e4b2622fe7701771e838 \ + --hash=sha256:15f024313246a4ed976c60f440bb8d257815513a681d212ff74fd46f7d715a90 \ + --hash=sha256:195ce897c6153c0700078142cf8efe3e6454ca4cf4357499e4078dfd83396626 \ + --hash=sha256:19a3dd5aa73cef1c99687c4fc57db016a9c17104ae1185da88ba566a5d3bebe4 \ + --hash=sha256:1d1c7a53a6c2103925cdd6d7229f8c567379f211c869793df679f2e9f738c369 \ + --hash=sha256:1f5883d77fd409a261abb5dc8ccbe335720d798b1de4abb3b1d47ccbbc76b53b \ + --hash=sha256:21b82d8082f6f5e7f456ef0bd16323d08de1266efbfeb476e64b2a91d1471a4e \ + --hash=sha256:252d099029bcbea642f2a06c4ed5046bdf8b5a8150b64afa5e027e88b106e5ee \ + --hash=sha256:256dd4d85d9e4dc595e2bc983c980e73f62ddeb3165c58b4c3dfe78c5c8548c1 \ + --hash=sha256:26422d45fd13551cf564c58932f7d72b4f58b93b0fcf18c35ba6be12b46bb102 \ + --hash=sha256:2679de311c7946dde5d3b6f44941844133ff5c7cb86099c0061ab1e8901c20a8 \ + --hash=sha256:29880d17a8eb0b5cfdfd8944b468322928059aa35f1f5fa8ff22b149ec0b42f8 \ + --hash=sha256:2bced4061f000f7187254a02ad3433ae17eaf991747ceea2f478422590a5bba9 \ + --hash=sha256:2e9cf9253119d8e5d111f05d71626786fd3d6193817316eab1ca088cdb8593cf \ + --hash=sha256:2f06b7eae9dbe77fe1d644ca244dad508de8d302870a43f3c559b521270938a0 \ + --hash=sha256:2f293479cce755c75f1697e87c409b7ae4c555c7dfecb6e988ad13abba943031 \ + --hash=sha256:329fc3ccb63ad22d867d84c2adea759a64079a37ba4a343433b02c7a2816871e \ + --hash=sha256:343fb4f2821043bd87095f7b08a1a181febc8e36ac64212143bbfd0a0e1bc235 \ + --hash=sha256:3588e376b3ea2eea84976f67273d679f229e24c66dce7b82ae45aef04ff6e072 \ + --hash=sha256:35aea775dc2bd5f54cd84a1cd2696cc3207c479cb9cf0bd346f0d343e4300ddb \ + --hash=sha256:35fe081843b35aad20ffeccec3eeffbe637b15d14f3fb22cc1b59cd8ec17e93c \ + --hash=sha256:36047af20e17097c3bb9476c2b7655f2f7aa51322c0ba58c07695bedf755a950 \ + --hash=sha256:3617ac3cfd8b9888f145ad89dd6e692285834b0201c6074a5eeaad3fd4d668c2 \ + --hash=sha256:366ec70f5547c640d3ce1985722490f23faf4eb5216a7eeba78277490e78dacb \ + --hash=sha256:394fea06235c8543390050ed5f529187074b029fb027213f6c46ac11ab5d950e \ + --hash=sha256:3d27167433c0d5f18dc850f07d0b3816221984fecdc405d6c157a6f0b8f8e9e6 \ + --hash=sha256:3e5e1224c0a6a90e05843e07adfec669edebec17801c67072f51e59561d63c0b \ + --hash=sha256:41876ee62a3dddf48ff1121ad8f0798032aa03f2fd35f21f34a4cab14f18d8d2 \ + --hash=sha256:433c5a81eade63b47e522303bad236f59dba55ea6951746f5558355eeed8c75d \ + --hash=sha256:4582c27e8c889d64811987b5967fbd3ae0c823fe1fd933b543d55ac20bb475fa \ + --hash=sha256:485a0d363cafefcd2538a73c7c838daa2035f09b2c9f9b5e3133f80c6aeb84c2 \ + --hash=sha256:494b70049a4d69aec6e8137c13af4cf8db8c9f9820a1392ac293b0dd2987a818 \ + --hash=sha256:496846868fea80e479324862fa877f02411f2fd0f83b79ccee2607aa68b2a032 \ + --hash=sha256:4abdc5f9ad448c1ecbfae2974b820535d6bc6e7eef63babbab3d81cf46968c71 \ + --hash=sha256:4b599739b93b2cbeded49645ae3c8d1405c29ddfbceac1545c87a3f9580a9e96 \ + --hash=sha256:4bea7f8ebe90bbd7f0e4a2de42ca6924ba23e3e76418c408ff82f1d46fabd687 \ + --hash=sha256:4c4fb141a727957c93edfe5c32a26ceb6b5f6461d67146e2d39f51e16170bea8 \ + --hash=sha256:4c9548dc78002099910abaebc0a72ac58b7d30931869e0351c09b507dff4ece3 \ + --hash=sha256:4d26f14f041e83dd8edfd61f4cd4fa7285d31798b5bf1f28e70c367ba6c41d61 \ + --hash=sha256:4f298bdadb8f0b9e5672877f647d1be9373ef5320c9e2f049795e26cad28b6a9 \ + --hash=sha256:52ec005752a56ae79547a05c0139ca2501a0c866390b6115008456b9f0e7cde1 \ + --hash=sha256:55261ac0d2941c42f196dd576f543d87a8ee03cd6f5e30dfb4d807b2e3b9121a \ + --hash=sha256:56490c595a28b1bb27dfc583e816152a9767721ef58b2c03b13f954d2f707420 \ + --hash=sha256:58d3e12c88e0950bca850ae1f7c256055c097639c2edb9eb123af9807d8b15e4 \ + --hash=sha256:58d4aa13a59c969dbfdf9e6a9560e242cbfd9e8a8f50c2747714df1a423adf65 \ + --hash=sha256:59171c6e45bf07d0d5cab3b0bf81d945035530f6873398b3b531c31184d46663 \ + --hash=sha256:5b6d1386bf0096d26d3a863dc0a487a5b4eb9aa93cf5ba69683d29dde6b9d60f \ + --hash=sha256:5c0ea61a470e070686aa30892fed79e297d2c8d0ab46b8bcdf027d38c51da591 \ + --hash=sha256:5c84bec0ab5ae0c64bfe73a7d2adcb5ce73b467523fc27fd6a28ab2aa6cbe35a \ + --hash=sha256:5ca0555312ae2fe82715cada7fac375530c2f3349e1eaa1bcb33d0283ac79a18 \ + --hash=sha256:5d8531a6569d025f68e2321e7638fb7978f23db58e5f69f56913837aae03816e \ + --hash=sha256:5e2d0e146dcb57034f8b97dc58d2d512cb90aba253960ce449f695fec6a82c6f \ + --hash=sha256:5fc45d653ea8c9a20479167e11d4a0f8cb2fa3470737ab6f9c827532313187b7 \ + --hash=sha256:6117b84ea48435e5356dc737f5121485c30920ba43375fa7b434fd753df0eac3 \ + --hash=sha256:6199d5606e2bbf2b096cf64d03f8b6790c91081d5ac866b8e7bb6422738cc60c \ + --hash=sha256:62b55f6722735a6c472f88361cde6640608773d9443cebdbb51abf436a1fcdd3 \ + --hash=sha256:687c9ca3035544b113bea2055e180af96fb63c0c476e22a9180f51925186e7b7 \ + --hash=sha256:6b7430cf5728e68f6c462254009a6ef4086e1bea43cf2f57aa9c55fb4f50ff96 \ + --hash=sha256:6ba32c4d2abf1d2fe7cf27d280f4cca5664233b0f885549c7761719eb977f486 \ + --hash=sha256:6c9cdde8becb25a7fde49924511aa2644d6f8081cc8df8e9452724303348d8e3 \ + --hash=sha256:6df0ec430f9a831772c23ca5a224cba36517a58a84bb32c32bb59a9fa67c47f6 \ + --hash=sha256:6e2912d4babbc65196ac13c2f53468dc57fb8b9c25ef913e8c59ddf7c6dc0e1b \ + --hash=sha256:6e5e4d73d588ca5ed09df1b7dcd1b203d1df3c542e3f50d126c947d432b10731 \ + --hash=sha256:70055ff39b97c99e7ae40ea3e393fb62aa2e44dbd9b29f8d14f42fb0025c3959 \ + --hash=sha256:706bfd38730a5ac7a365793269a00f4e988178cec121391f4248d84ad8c972e9 \ + --hash=sha256:7235dc28fc6dd9d832ac7c7bce95367dedb85929f17368a0c2bee1e080b9acbf \ + --hash=sha256:774d157f112367ff4abd29019f38f023c24e00e56edc7829c20e358a5a913ad8 \ + --hash=sha256:77efcff2b23071c349402ac1066667a3d011f62398d81408c9b88ad991747c9e \ + --hash=sha256:789b8982559ae28dad2356519f841655756cdcd96616410590ae0b17454ee64f \ + --hash=sha256:7ac76cf9afd34929d76eb7fcb63be476a4853d8a96f0dcf2d0db68a0cbdf9885 \ + --hash=sha256:7c0c10730342b0c9b35dd1d619beb8214e520bd96a1f870f452680b238aab3e0 \ + --hash=sha256:823f82903d189af463d7df250ef1f7f696f3cee08cc8d91deb565e8d425f6506 \ + --hash=sha256:838648accb3a7fd9803fd45c87bce8509648eb0c11bc34e216141300977244f2 \ + --hash=sha256:854066be00447fa8de2ccbbe893e2ffc4b123ef16d897af794c1e18bd4a714b0 \ + --hash=sha256:85d5855daafc240cc045c026d7a15fd198a09b0fc8ff6f5ecbb5297b509cb11e \ + --hash=sha256:85de3134b5379856e323ba37c19c9256d39425f7b76a63af52b09fb4664c2e8f \ + --hash=sha256:87e4f41d375c0b9be2fb5251aee4b8a689169e134535aed81bf085c3b647451e \ + --hash=sha256:88ca277405c2d3b71c4e1c2ee0e7966e807bcba86a69d11e19ba199d18ae4491 \ + --hash=sha256:88e85ab89cb822c1e635f51d6d32e488f94e002e70e2f492bdb8b945543f345a \ + --hash=sha256:8ac8c94b6539074e0f40899301273ac8402b9b3e01c7b7ba269ff30340aaaf20 \ + --hash=sha256:8fe532b3c966d1fb794e0698e4589d0444017ae77fc0b31edea13c0e35bcc449 \ + --hash=sha256:9085f87b0e38a2b92b8923059b4e8789fe40d9279712d15dcc670048d77079af \ + --hash=sha256:90b7481fb62fbe172c558bc6fd1c4c98d82004a54a7551f20e11ac9bf0b8708c \ + --hash=sha256:92caef967d287a407085d61176fce4012b1dd62daed4eb6d5ceb26d3d2538712 \ + --hash=sha256:9362dd90aa7dab48c0054a21187791ccf05473f7dba5d92b8033ae62164675e7 \ + --hash=sha256:94d78ecec2605a8d0398b0f365d5f12a63248438516f5dac536a5eff7337df4a \ + --hash=sha256:94fbf1c0c6cc0d3d5e50f9a9313a8cdca90dd696d34b381cd1704f8c9e939f20 \ + --hash=sha256:950f23cb393f85543777b0433f082cddd25b51ab398eac7971146495679efe5f \ + --hash=sha256:96eefc178f8636b9c760c5829345307fd81cfae9ab1e80997dbddeb0f54ee9a3 \ + --hash=sha256:96fef3e886d6a9874b14f27fc193fbdc69d5d8035783d86aa4e1cea594e695f9 \ + --hash=sha256:977cdbd483a9cff38179bea4fd754289a6f2195c7abd414aba85410b3e66cc5e \ + --hash=sha256:978eab16f55b4ab2c2a745be9a0a840bf8f09a7f227d9c76eb30214d078865a5 \ + --hash=sha256:994e883d17c559cdfd38c84003c8b27d25424a1077272a17e7cd27bfe0bf57b2 \ + --hash=sha256:9ac4444d8d4fd4c4bd08bf451ed3167aa9e7ec6cdb41b648794f1d1103652e36 \ + --hash=sha256:9b5db6052055d34d41230fb78d7c439c23dc536a9896f6cb039e8dd92cfc1263 \ + --hash=sha256:9d9a0dc7cbe9bec24c3f767c9122c41fe5a1bc43f47cd099d00d393e09769de4 \ + --hash=sha256:9dbdd9205662134957cf0c324f639bdc5031c0ca056e2369e238db75187c0f11 \ + --hash=sha256:9eea3ab2597a5e65fe65296e2d6a84570845a6b55532d90333d740d48bbc850a \ + --hash=sha256:a2028475ba855475b8b4d3cfeb4994269c967aea8b9892dfba907f4263a863a3 \ + --hash=sha256:a3a370082ce34d0612f421e15fe011c53bb1feff21a26d06ad4fb244dab5a375 \ + --hash=sha256:a545775cfe815855ea32d7c27731d79da358ef2055b4a25830231b1622dd18aa \ + --hash=sha256:a5cbd90ecf0fc62e64726917ad083b73001f0563657a87ec3c0b504e277dc90d \ + --hash=sha256:a6d095662e73e74f0a49988e0593373e243e3a52e27bfeea0a859e88acf4a0f5 \ + --hash=sha256:a6dac12ff6b846103483683f60c5f8fee205121adc58ffd87e90a90a3af69e99 \ + --hash=sha256:a951ad59cad9145664a730d3036b40b844e74d2d3683da40111463cd3a83845d \ + --hash=sha256:aa1099b956fb795e686d073568f6dc002a0bb89765ea6d5b055dd7d9bf1b116c \ + --hash=sha256:aa2bb0b37202dca27175591f761108b5d34096ade1191ffe4808bdf6b1571488 \ + --hash=sha256:aae2ee51122d3ae968a3837d97dc24a0aeebb0dea23694422cd172bd30017cd6 \ + --hash=sha256:ab743e9bc90c1f73552ec33e10e3331315acd2c397b36065b591b0181de533cc \ + --hash=sha256:ac00177c4831ffa650f8609e4bdddd5fe09c03b1c0c47acece7e6ea20421598b \ + --hash=sha256:ac13b004224fb341e1e25a1ed5e19d32f57cdb2a403e01f003b46f051a550f6f \ + --hash=sha256:acaf604462bf330b0d07e7a07c1d6e4adac79e5fb13e9c5140590542cafacc00 \ + --hash=sha256:ae31a1a1db2ee6cc2942fccaf695c934bc7f3db9f2133a3fef1f367cf1a4ab10 \ + --hash=sha256:ae4a097991662cd4fff0ddc74e0fe7874f82e00042fa0ea00855645ed0c79598 \ + --hash=sha256:aea996a6aba25260827c9ea511d1addfde2da9eb686ac961838509086188b7e6 \ + --hash=sha256:b39b69b347e5e47a3b5b8cfc005c68c1ba347474e3960236c4944a8ecd174962 \ + --hash=sha256:b54e7e13267d49ffbfe68e25b3cbd774dab38fa37238f71265e91b36146eb21c \ + --hash=sha256:b9af956078716df40d985fb0dfeb2c2120c5ca92ba4ff4b388acfd01cdc14d08 \ + --hash=sha256:ba2f37ee79e6338845261a3c5b1784e5d1acdff2c0785b284f1b633033d136ab \ + --hash=sha256:ba501e667c17d8411f98e67a022d9604ef179aff0e459b7e292c796837c13573 \ + --hash=sha256:baf3775a2635e5a11fbd5e4e64ee69c7e86875d224a5c72aca4c141064589a90 \ + --hash=sha256:bb57753e36e4855b8ca375069482250a6246372331a3e4f3407eaebb007443f5 \ + --hash=sha256:bd6c173f04743d483881bffa1478d5a4624475b8cd1d2194956a75548e191c18 \ + --hash=sha256:be47f99644b208bff7766314013f9acf57b056b04191d570d68ad14022cf5b1d \ + --hash=sha256:c010f5581d9c612804cc59fcf7b524b707fbcb72828551237ab545bb5c7034af \ + --hash=sha256:c1dcc36dcb96abc02236e182d17e0f71430152a6c2c7447421da2d2dc144edea \ + --hash=sha256:c428c6c31eb5f4277d7f8eccaf767fbd548ddd5ce3c8b4f4cbbfab3d96b5904c \ + --hash=sha256:c658c50ac0c98cd755a2dd50b7977d3bca7df401dcc47fbdfa87db53ef7d4e8b \ + --hash=sha256:c71fb0d56c920c269cd3e2e3fe7c610e3f1fdb21a6ce60efa6430ff63676cea6 \ + --hash=sha256:c7b742bf31c88566b4bb6335a7f393bb322e580b6bb98df7bd0c25e6e3519ce8 \ + --hash=sha256:cc0329df4caaceb950d2f580b5ac716a377f7059624a0bafaeaf8a218c6ed774 \ + --hash=sha256:cc5d36d96478aa9c60654bd932525bf32964c62a7281eafdf16d85003a8d6004 \ + --hash=sha256:ce854f5f478050ade5a238731c4ca985a7d3b3cb53ff600a9b5c3b689b5f0a7a \ + --hash=sha256:ced3fdd71aaa83ce593746c2edb42b7a59cb4c19c8b5c407781c72e493aae55a \ + --hash=sha256:cee5dd7c6fb5dd52a0fe2a740f9bc6e3593f5f8b1788bde49de02086f30182b2 \ + --hash=sha256:cfa1c0cc3a8f9f53f1243a5a99ac36fd003880199383b37672e86ddda9cb07e2 \ + --hash=sha256:d1ee1e296209fdce05b81b663250eefa02213a2da7b41bf26f7829b8ba3545aa \ + --hash=sha256:d59b75732e9b6f27388e10c14b0259cc5f2e48c78627d185e6a177b58ad3cffe \ + --hash=sha256:d63600d620ad0064c3a748b950ac5ea38a80190e5498532efefa4b7b3f1da1f3 \ + --hash=sha256:dd732602a7009217f658d5863d12d79d373a4de0eebc111094bcdd3bb8e0a6cc \ + --hash=sha256:e06efa066f7dbadbc84ebc126a97c452a6451dfcf589d89d788484949e1cf795 \ + --hash=sha256:e199fb99720074809a7720f1c0b4d919eea8b87e88713e0f8f602f7bef543d9d \ + --hash=sha256:e4b018dc5a0eee4676e38fe84a47a427816c590b93b55d9025274ec4d6ffc2dc \ + --hash=sha256:e6621fb2a4988d6e53eedc455e5903e2679f3967b8acb3d639f1b63c14a2e893 \ + --hash=sha256:e71c909f353863b2b89c83de2ebed71ea6d0df8a6ef65a128193c5e650766bef \ + --hash=sha256:e90251c0c7bdd54a100a0dce3c07b7e637278c93af29dbf78ebb89a58c4bac7d \ + --hash=sha256:e9fbdce1e47394b09bc9f26ab117dfc8d6491977a11d86f592bb42c779db2fda \ + --hash=sha256:eb12fb2ba69ffa05f8695f61c69e591dc4b4a12ac3757ac8af8adb259bf56d17 \ + --hash=sha256:eda059b6bc8bc0812d626fd91a7ce01bf583df0a61296eff390fd94141a34e30 \ + --hash=sha256:f03ac127268b43ef4fe9e6ab6794a6794b49485a0cc0c1db79876d2f33f75bc7 \ + --hash=sha256:f298e218441525d3794428b4c8b8fb8662c6d3ea79925d4807ee6b9a96a3bca5 \ + --hash=sha256:f5542f9b941279d82d41eb0aa9f98eba36fe4df5c7086c651df7944935b37182 \ + --hash=sha256:f6f7deae3feb4edfa2efaf7c574fe88cbf055038a6abdb40188e4fff66d5699f \ + --hash=sha256:f9b1e28d0e8dbfa858abdba91d6b547beaf2df1a59bec6da6faae7b96a4991a9 \ + --hash=sha256:f9f8405c2c758532c74fed975dbee57be1f31a6e865c031870c79a6ed3212ada \ + --hash=sha256:fa48b1b63d639f9483e0633e092f5851e2348c352f1f9bb6c8182f87884ef876 \ + --hash=sha256:fb78f6e7fcd8ad785d28cd577168bc1aaee827b25bb8755638f694794ea98f0a \ + --hash=sha256:fbc597639158fd7c14d55e808718848319540f51b0e6746e3eefa59723a4a348 \ + --hash=sha256:fce8cbd4997efeb450bd298b54f755dcdff18d496f7a5ddbb4867c6d7c88fdc3 \ + --hash=sha256:fd0350afdc3aabd5576f60ea109228bd5538139713c7b094c5cd27c73a98bc6f \ + --hash=sha256:fd0a274c0e5f9a21565cd9d3dd749b61f96b7aa1e20a93aa1ba4029518f2e5c0 \ + --hash=sha256:fdb8a068947befafba9952162645dc2fecaeb400e64584829ed5e9b2fbe21a7f + # via requests +cryptography==50.0.1 \ + --hash=sha256:01f41478cf33fc605a6a089cd56d28b45c6c0b45a1928b61797f2621a04bac71 \ + --hash=sha256:05ba322c4da95b262a212c345af888ef2c37c88c0509756ea00a0e6d68850f23 \ + --hash=sha256:16c5ecd954b3330ebfb6605eca4fd952da8bef376551d5cc264534e3770a9ee6 \ + --hash=sha256:2a93d05e34d5f67fba6f891fe85d929999baa7195e853923ea6d7576c9e68c5e \ + --hash=sha256:2b34d76a652ea2b6faf777c35df230c5637842cd904e04f16230c3f9f03e4361 \ + --hash=sha256:2ebbfb0f1fed745e91796e3e1080a1440423fdae8ece1b995a1d80883a409054 \ + --hash=sha256:30a125032e5642a21ff816e021152bd4e7e94f03eff3f4b7fca41cd22bc3110f \ + --hash=sha256:330fbb252391c596f1ae42c5754449dc924e6ad012dca8efe0d703f9f2d12ec6 \ + --hash=sha256:359e62deae718bce96170e223fdcb6357e4fbd3bb7a3a75f4430763532560e49 \ + --hash=sha256:407fe2b6db00939c05c0e945e9914238f2f0a430974839429dafc82b1ee6bee5 \ + --hash=sha256:42be3bb70596b3abe4ac097b75be223e8b3ab614a0e5de068e3dcc54d71d6149 \ + --hash=sha256:4c4188f7c0cf655be5c06342b817ed0f9595b69ffa2b12026e5353eed29dea88 \ + --hash=sha256:51593d180cf6d179bde5c5d065bed81386b1f381656ae7d042b7ffc87a9895ad \ + --hash=sha256:51afcfceb15597cf2635068e4ac9a56b2abde622edde17f37d85fd7b5306497a \ + --hash=sha256:53e279950892dc102c6b4e52af03ae5ea92fac572a1ddab78ca73a997f62b69f \ + --hash=sha256:55d16b1ef3ee0958d893a977b19777887e546c9954ea81b200c3301a864013f2 \ + --hash=sha256:5dd9bda1c12b4162f6ff568eeb5e0ff956c28d14406e875cfe8a63a2d414ff20 \ + --hash=sha256:5fe002589592ed749ce77fe0695fcbd3500dd61d7d6db5858a7544c612fa8e45 \ + --hash=sha256:5fe939deeb161024a6be98229c953b6591fef1f41214497a78fe793a244c017f \ + --hash=sha256:693c99b49bd37d0d096e4334c10232c77248c415b98d35236094cdf96d57258b \ + --hash=sha256:76de83fbd91ac49c0feaaa983d0748fd7a53176afac5fb3bf7478d244f0eb527 \ + --hash=sha256:79bf008d1f9af6071c797ad133e39915dfee7614f18f18f4db9072eb715064a3 \ + --hash=sha256:804728ce710890870f3aaa344b2e161172d258d768ac139d02cfd9092d0d94e6 \ + --hash=sha256:8921d58f426793c5f1b47f0b59575780de9a095214958d0eb37d909593db8367 \ + --hash=sha256:8df2de9102026855887e4587084f6eabd80ed0f345b8ad8a7ac27ab9bf4723e0 \ + --hash=sha256:9cb3cb952cf5a8abd50c782a98a89d71699715e802fe349704b47f2425b42a94 \ + --hash=sha256:9dde0a357190eb3b1da1bb9ab750e9c85cba82ca5977aa0836cbb94e92611239 \ + --hash=sha256:9ebcdd5519be9b652a46f507817a74591774fc3d6923ac364e4dfa64e36b291b \ + --hash=sha256:a0b1a59e3a089064a0ec309e9428c8e3ae4e161419d20ac33600767e83fc658a \ + --hash=sha256:a255449073358275b64b67d3f595f268bbef70e72b6edb65e0c70c735bf739c9 \ + --hash=sha256:a8f40ea47330e71b594a7e246898f93177c259490c63183dbaf9e571d71ed9a5 \ + --hash=sha256:ac02b07824d4d1001bd4367599f839c19cb171924c796e52c23508ac14c2c0cc \ + --hash=sha256:aed8db4f6d71c51efb89530e12d9464e7bf2923d46c3205dc794a2a93f8c0648 \ + --hash=sha256:b8f852c65863251b9e3a1b8c150ce21e59b522dbb6a7d4bc80e680d38388e986 \ + --hash=sha256:be224a65493ec5b74a158ff22a5522ce4a5ca1e543c647a3a4730d4a09e5f959 \ + --hash=sha256:ca83d00d9e69cd5eb63f2e69c3a5a59e0cecae5ae14c6ae0b35830fe3b37bad0 \ + --hash=sha256:cbf74a81765ee67413503ca6e26dcc4f6f5a519822436cc0a1b97aab6c1b8a17 \ + --hash=sha256:d63ae8f6481fec907ac0f588eee8a90aefde112c633131fe540e5711ddbb5a4e \ + --hash=sha256:e22dfed744bd4002e909464cb23d2f0b05c6f3113a79ef2e9864a53db737c733 \ + --hash=sha256:e2ca8fd1b6b4b82a1c4cb02841d0837e3c12336c2e24b520ab8ab3b969733d8f \ + --hash=sha256:e74591e283fe6eb956416c929eb58262a719fe0311fd9054c62c3350ed8760d8 \ + --hash=sha256:f74455bb086a85d5e81246412602aaa97ed095e504cd40dd261ef50be42205bf \ + --hash=sha256:fb4b9672d389c738b175c4166e78310f8a70358886aacd9173ee03a85ffdc671 \ + --hash=sha256:fc3ed7ebd2a8c96f5b166de0ab9b624996bef3b07bbeb19364dfb78222c22c80 \ + --hash=sha256:fd3718b960d0b5dd213cdf03f3bcb7000e69dda0de8b956061947ff6bcff5558 \ + --hash=sha256:ff838d62ec1bfce4f9ba7fa16f4a7b554cd8d0c299e6be37502161a660c84eef + # via secretstorage +docutils==0.23 \ + --hash=sha256:25d013af9bf23bc1c7b2b093dff4208166c53a94786c9e447808335ef1185fea \ + --hash=sha256:746f5060322511280a1e50eb76846ed6bf2342984b2ac04dc42caa1a8d78799e + # via readme-renderer +id==1.6.1 \ + --hash=sha256:d0732d624fb46fd4e7bc4e5152f00214450953b9e772c182c1c22964def1a069 \ + --hash=sha256:f5ec41ed2629a508f5d0988eda142e190c9c6da971100612c4de9ad9f9b237ca + # via twine +idna==3.20 \ + --hash=sha256:a7db850025b95ded1eae8a46181a1a6c56c92c96f0e2b005d9ff8dc0210cab44 \ + --hash=sha256:ab7ae7122974553370f0bdb919e1a960b2cd1bc1ef0276416d896db81c14582c + # via requests +jaraco-classes==3.4.0 \ + --hash=sha256:47a024b51d0239c0dd8c8540c6c7f484be3b8fcf0b2d85c13825780d3b3f3acd \ + --hash=sha256:f662826b6bed8cace05e7ff873ce0f9283b5c924470fe664fff1c2f00f581790 + # via keyring +jaraco-context==6.1.2 \ + --hash=sha256:bf8150b79a2d5d91ae48629d8b427a8f7ba0e1097dd6202a9059f29a36379535 \ + --hash=sha256:f1a6c9d391e661cc5b8d39861ff077a7dc24dc23833ccee564b234b81c82dfe3 + # via keyring +jaraco-functools==4.6.0 \ + --hash=sha256:880c577ec9720b3a052d5bc611fb9f2269b3d87902ef42440df443b88e443280 \ + --hash=sha256:99e3dc0060c5cbe8fcd1cdb36258e2a65ca40f1566b2033b12abb1bb44dd3c30 + # via keyring +jeepney==0.9.0 \ + --hash=sha256:97e5714520c16fc0a45695e5365a2e11b81ea79bba796e26f9f1d178cb182683 \ + --hash=sha256:cf0e9e845622b81e4a28df94c40345400256ec608d0e55bb8a3feaa9163f5732 + # via + # keyring + # secretstorage +keyring==25.7.0 \ + --hash=sha256:be4a0b195f149690c166e850609a477c532ddbfbaed96a404d4e43f8d5e2689f \ + --hash=sha256:fe01bd85eb3f8fb3dd0405defdeac9a5b4f6f0439edbb3149577f244a2e8245b + # via twine +markdown-it-py==4.2.0 \ + --hash=sha256:04a21681d6fbb623de53f6f364d352309d4094dd4194040a10fd51833e418d49 \ + --hash=sha256:9f7ebbcd14fe59494226453aed97c1070d83f8d24b6fc3a3bcf9a38092641c4a + # via rich +mdurl==0.1.2 \ + --hash=sha256:84008a41e51615a49fc9966191ff91509e3c40b939176e643fd50a5c2196b8f8 \ + --hash=sha256:bb413d29f5eea38f31dd4754dd7377d4465116fb207585f97bf925588687c1ba + # via markdown-it-py +more-itertools==11.1.0 \ + --hash=sha256:48e8f4d9e7e5878571ecf6f2b4e57634f93cd474cc8cfbd2376f2d11b396e30d \ + --hash=sha256:4b65538ae22f6fed0ce4874efd317463a7489796a0939fa66824dd542125a192 + # via + # jaraco-classes + # jaraco-functools +nh3==0.3.7 \ + --hash=sha256:157ec1eb7a62f3d9a7badb8d82d89aa810e3e24e097eedfa481a25d0c8a99877 \ + --hash=sha256:15f5fbf090f5c88d61c820e1fc1fceecb6520cca9fe85649c06b57ef9dc9ff62 \ + --hash=sha256:18f4278ecd157d43cb35acd5aae9f35cfa79f546b4922bd86536adc0f6312102 \ + --hash=sha256:19f288c938ec6eef1f5d2c6cab47838e71fef8097e1c1233802be5a6230ba086 \ + --hash=sha256:4968fe8d2db97c6f047659bf46a449fd8ec377f44ebf3e0a1b96c0d3a333ae32 \ + --hash=sha256:5ffdfcb9a686ffb12765376bcfb6b5b55728516d3c0ee317d29982381ded3df8 \ + --hash=sha256:614dac4a4c36ad084e78447d16fe898dedd762e354a7ab9cda2984e82f67883d \ + --hash=sha256:618e3059caf41ccdf5dcccb3fa9df4cf6e4efe23d1382a8bbfca272a8a4f8bfc \ + --hash=sha256:6698a822132beedab80f131c08d8d0ac5a178ddeb488d02ca4b67716ecfac7af \ + --hash=sha256:6c3aa50eb26e9228238271db9f983cbc3b006dfbfeca2d4dc34c33ddc6ac5ea5 \ + --hash=sha256:6e4280115d44c3b278eef712a86748c1a723105cd79feec46952383117ab4e59 \ + --hash=sha256:70f5ac8626e899a4bab0ef74ca2f5bd602f49c7b739e6e5026b4afc6d63dac42 \ + --hash=sha256:71860d01c16f4d8c72e334e0674beb2b0899dbd0bf760de18932ef4390303848 \ + --hash=sha256:808def0c8c07843e6e50dc84f532457bfa2cfd17417b219a5d9e7c773709331a \ + --hash=sha256:874b7d67a067bd29a59223f6270fc30da4edd8e6d87fd219fc93bcbaa662c946 \ + --hash=sha256:91a4dab4e94d9fc54b9f67b1adfb23e81fab7ab43f33c3b8c97be9aa38f789ba \ + --hash=sha256:94fd6e59553fbb9ffd8ba71bbd5a54e3126ba01799a097ae30d5341d750bc6ac \ + --hash=sha256:9b7279d43323a25225df23576af6594a16693f61431170848b8b2ac21ad4f174 \ + --hash=sha256:bc42bb1193c1e28a1e74c2cabaca178e118a7103e8832699fef8a2b3e2496493 \ + --hash=sha256:be53a4825585f701955cb9baf49f478f56eb81e20294329fe4bc689dd5dd81fa \ + --hash=sha256:d56e76bd3cadb09b6b0cef364850811663734b348a25f5f587a2819c495367bd \ + --hash=sha256:de2b2aab32ea303405debefdcfc58043d3e635fa3f67b9eb140d2b0e0c0d2563 \ + --hash=sha256:e8fd1ab205258b29254f72db377d99e2c96aa7653ef3b015ccab0420b094b506 \ + --hash=sha256:eae64328e46a25785535afcb6885b6f182ecaf5ee8c88f8c075422db8aacc65b \ + --hash=sha256:f04b7d333b27f13ca439da3cf1c75c2fba34f104969f6ce4ac8e7079699c2f4a \ + --hash=sha256:f266d3f1b3647449923a8e406524632220dd5d8b647078dfe45b885d33d10479 \ + --hash=sha256:fd4a70efb45d5372174f718878eb7a35c12677626a63b2f103b23b833457dcac + # via readme-renderer +packaging==26.3 \ + --hash=sha256:94edc256424af38762eb31306eed28beb9f0efc50a8837492c9d6fd6004aed79 \ + --hash=sha256:d7193f7c8e4e93f444fde0262bf90af30e16fa0ad0ad44cb553c87339b23cd1c + # via + # build + # twine +pycparser==3.0 \ + --hash=sha256:600f49d217304a5902ac3c37e1281c9fe94e4d0489de643a9504c5cdfdfc6b29 \ + --hash=sha256:b727414169a36b7d524c1c3e31839a521725078d7b2ff038656844266160a992 + # via cffi +pygments==2.21.0 \ + --hash=sha256:2363c69b61c4a97c838da3b130dcd6468f4848992b21a82f2a63ec34377137d9 \ + --hash=sha256:610ca751c9bc2492b38eb9a38a7fbc93edbbb2d7182edaf34e66ae493dee5c8c + # via + # readme-renderer + # rich +pyproject-hooks==1.3.3 \ + --hash=sha256:5fc53fdac9f7bd63fbcdc868fb5f90b4784d78a53a3d3388cd738b807441a20b \ + --hash=sha256:defda19b854fa0d3bd4f76ea4ddcba8abd7dcfcdd585a6690ade050744fc5f43 + # via build +readme-renderer==46.0 \ + --hash=sha256:af3e964914f6310a33ff67b72a4bdd940bed8d7c3bdecd2d14f40edf284bfe90 \ + --hash=sha256:d0dae1f74bb273b534770cb4cccb6bb78735540afdb03c2146f4e19dcd412560 + # via twine +requests==2.34.2 \ + --hash=sha256:2a0d60c172f83ac6ab31e4554906c0f3b3588d37b5cb939b1c061f4907e278e0 \ + --hash=sha256:f288924cae4e29463698d6d60bc6a4da69c89185ad1e0bcc4104f584e960b9ed + # via + # requests-toolbelt + # twine +requests-toolbelt==1.0.0 \ + --hash=sha256:7681a0a3d047012b5bdc0ee37d7f8f07ebe76ab08caeccfc3921ce23c88d5bc6 \ + --hash=sha256:cccfdd665f0a24fcf4726e690f65639d272bb0637b9b92dfd91a5568ccf6bd06 + # via twine +rfc3986==2.0.0 \ + --hash=sha256:50b1502b60e289cb37883f3dfd34532b8873c7de9f49bb546641ce9cbd256ebd \ + --hash=sha256:97aacf9dbd4bfd829baad6e6309fa6573aaf1be3f6fa735c8ab05e46cecb261c + # via twine +rich==15.0.0 \ + --hash=sha256:33bd4ef74232fb73fe9279a257718407f169c09b78a87ad3d296f548e27de0bb \ + --hash=sha256:edd07a4824c6b40189fb7ac9bc4c52536e9780fbbfbddf6f1e2502c31b068c36 + # via twine +secretstorage==3.5.0 \ + --hash=sha256:0ce65888c0725fcb2c5bc0fdb8e5438eece02c523557ea40ce0703c266248137 \ + --hash=sha256:f04b8e4689cbce351744d5537bf6b1329c6fc68f91fa666f60a380edddcd11be + # via keyring +twine==7.0.0 \ + --hash=sha256:85cdb29c518efef867360ae4acd4b0dfd61c8654a22fca08e6f8539f05022177 \ + --hash=sha256:b854164df26db268af05f49aa5c0344b10e27a494343ff05b1e0bad3b135f5a7 + # via -r .github/requirements/publish.in +urllib3==2.8.0 \ + --hash=sha256:0cf3cae568d36aa9576b28dfb35f11328f1cb974ca7647d9475ebb86c75ac6e3 \ + --hash=sha256:63bf2ead4c879426ebf22ef2a781eeb4aa3b4ae798a0435506f8687fd5bb9b63 + # via + # id + # requests + # twine diff --git a/.github/workflows/ci-dev.yml b/.github/workflows/ci-dev.yml index 728c5d5..5ea6465 100644 --- a/.github/workflows/ci-dev.yml +++ b/.github/workflows/ci-dev.yml @@ -87,10 +87,11 @@ jobs: uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0 with: python-version: "3.12" + # This job holds the PyPI token, so its tools are hash-locked and wheels only. + # .github/requirements/publish.in says how publish.txt is generated. - name: Install build tools run: | - python -m pip install --upgrade pip - pip install build twine + python -m pip install --require-hashes --only-binary :all: -r .github/requirements/publish.txt - name: Write pyproject.toml from dev.toml with the next version run: python scripts/dev_release.py prepare - name: Build distribution diff --git a/.github/workflows/publish.yml b/.github/workflows/publish.yml index 1b658e1..3a4c7e1 100644 --- a/.github/workflows/publish.yml +++ b/.github/workflows/publish.yml @@ -27,10 +27,11 @@ jobs: with: python-version: "3.12" + # This job holds the PyPI token, so its tools are hash-locked and wheels only. + # .github/requirements/publish.in says how publish.txt is generated. - name: Install build tools run: | - python -m pip install --upgrade pip - pip install build twine + python -m pip install --require-hashes --only-binary :all: -r .github/requirements/publish.txt - name: Bump patch version in stable.toml and dev.toml id: bump diff --git a/CLAUDE.md b/CLAUDE.md index 15f161f..1c6b35a 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -72,6 +72,7 @@ automation_file/ - CI steps: `lint` (ruff check + ruff format --check + mypy) → `pytest` with coverage → uploads `coverage.xml` as an artifact. - Stable publishing lives in a separate workflow (`.github/workflows/publish.yml`) that runs on push to `main`: bumps both TOMLs, copies `stable.toml` to `pyproject.toml`, builds the sdist + wheel, `twine upload` via `PYPI_API_TOKEN`, then commits + tags + pushes and creates `gh release create v --generate-notes`. - Dev publishing is the `publish-dev` job at the end of `ci-dev.yml`. It runs only on a push to `dev`, after `lint` and `pytest` pass: `scripts/dev_release.py prepare` writes `pyproject.toml` from `dev.toml` with one patch above the newest `automation_file_dev` on PyPI, the job builds and runs `twine check`, and it uploads (same `PYPI_API_TOKEN`) only when the commit is still the tip of `dev` and the wheel differs from the newest published one. Nothing is committed back. +- Both publish jobs hold `PYPI_API_TOKEN`, so they install their tools (`build`, `twine`) with one command and nothing else: `python -m pip install --require-hashes --only-binary :all: -r .github/requirements/publish.txt`. No `pip install --upgrade pip`, no unpinned install; `tests/test_workflow_actions.py` fails on any other `pip install` in a job that is given the token. To add or raise a tool, edit `.github/requirements/publish.in` and regenerate `publish.txt` with the `uv pip compile` command written in that file. Dependabot reads the directory and proposes updates on `dev`. - `pre-commit` is configured (`.pre-commit-config.yaml`): trailing-whitespace, eof-fixer, check-yaml, check-toml, check-added-large-files, ruff, ruff-format, mypy. Install with `pre-commit install` after cloning. ## Development diff --git a/architecture.md b/architecture.md index 01dcd76..3ec0050 100644 --- a/architecture.md +++ b/architecture.md @@ -30,6 +30,7 @@ the CLI, over loopback TCP or HTTP servers, as MCP tools, or from the PySide6 GU | `stable.toml`, `dev.toml` | Packaging for `automation_file` and `automation_file_dev`. No `pyproject.toml` is committed; CI and the publish jobs write one of these TOMLs into place. Apart from the name, version and description they say the same thing (`tests/test_dev_toml_parity.py`) | | `MANIFEST.in` | Keeps `tests/` out of both source distributions (`tests/test_sdist_manifest.py`); package discovery in the TOMLs already keeps it out of the wheels | | `scripts/dev_release.py` | Release helper for the dev channel (standard library only): picks the next `automation_file_dev` version from PyPI and tells whether the built wheel differs from the newest published one | +| `.github/requirements/publish.in`, `publish.txt` | The tools of the two publish jobs (`build`, `twine`) and their hash-locked resolution for Python 3.12 on Linux. `publish.in` holds the `uv pip compile` command that regenerates `publish.txt`; Dependabot reads the directory | | `main_ui.py` | Development shortcut for `launch_ui()` | | `tests/`, `docs/`, `examples/mcp/` | pytest suite (fixtures in `tests/conftest.py`); Sphinx docs; MCP host configuration example | @@ -67,6 +68,9 @@ the CLI, over loopback TCP or HTTP servers, as MCP tools, or from the PySide6 GU builds from `dev.toml` and uploads when the commit is still the tip of `dev` and the wheel differs from the newest published one. `scripts/dev_release.py` takes the version from PyPI (newest release plus one patch, never below the version in `dev.toml`), so nothing is committed back. + - Both jobs hold the PyPI token and install only the wheels pinned by hash in + `.github/requirements/publish.txt` (`pip install --require-hashes --only-binary :all:`); + `tests/test_workflow_actions.py` fails on any other `pip install` in them. ## 4. Main flows diff --git a/docs/updates/2026-10.md b/docs/updates/2026-10.md index 8046ae6..5ddcc42 100644 --- a/docs/updates/2026-10.md +++ b/docs/updates/2026-10.md @@ -152,3 +152,28 @@ Index and query commands: [README.md](README.md). New entries go at the end. - **Docs**: `architecture.md` §2 lists `MANIFEST.in`. `CLAUDE.md`, the READMEs and the Sphinx pages do not say what an sdist contains, so they are unchanged. - **Files**: `MANIFEST.in`, `tests/test_sdist_manifest.py`, `architecture.md`. - **Open items**: none. The stable channel gets the change with the next release from `main`. + +## U-20261001-10 · 2026-10-01 · The publish jobs install hash-locked build tools · #done #ci #security #deps + +- **What**: decided by the owner (workspace X-13). The two jobs that are given `PYPI_API_TOKEN`, `publish-dev` in `ci-dev.yml` and `publish` in `publish.yml`, installed their tools with `pip install --upgrade pip` and an unpinned `pip install build twine`, so whatever was newest on PyPI at that minute ran beside the token. The "Install build tools" step of both is now one command and nothing else: + + `python -m pip install --require-hashes --only-binary :all: -r .github/requirements/publish.txt` + + Triggers, the other steps and the secret are unchanged. The version scripts (`scripts/dev_release.py`, the inline bump in `publish.yml`) use the standard library only, so the lock holds no TOML package. +- **The lock**: `.github/requirements/publish.in` names `build` and `twine` and carries the command that generates `publish.txt`: + + `uv pip compile .github/requirements/publish.in --generate-hashes --python-version 3.12 --python-platform x86_64-manylinux_2_28 --only-binary :all: --exclude-newer 2026-09-24T00:00:00Z -o .github/requirements/publish.txt` + + - 29 packages, 395 hashes: `build` 1.6.1, `twine` 7.0.0 and their 27 dependencies, for the Python 3.12 on Linux both jobs set up. + - `--exclude-newer` leaves out releases under a week old, the wait `dependabot.yml` sets. Without it the resolution took `charset-normalizer` 3.5.2 and `cryptography` 50.0.2, both published on 2026-09-30; the lock has 3.5.1 and 50.0.1. The unpinned step of the previous run on `dev` had installed those two day-old releases. +- **Dependabot**: the `pip` entry now lists `directories: "/"` and `"/.github/requirements"`, still with `target-branch: "dev"` and the 7-day cooldown. +- **Tests**: `tests/test_workflow_actions.py` gains five guards. + - Every workflow parses as YAML. The other guards read the files as text and would miss a file GitHub cannot parse; the locked command written on one line (`run: python -m pip ... :all: -r ...`) is such a file, because of the `: ` inside it, so both steps use a block (`run: |`). + - The jobs given the token are exactly `ci-dev.yml:publish-dev` and `publish.yml:publish`. + - Every `pip install` in such a job is the locked command, and there is exactly one. The previous steps fail it, as do an extra `pip install`, `--upgrade pip` first, a missing `--require-hashes`, a missing `--only-binary :all:`, `uv pip install`, and no install at all. + - Every tool in `publish.in` is pinned in `publish.txt`. + - The `pip` entry of `dependabot.yml` lists `/.github/requirements`. +- **Result / numbers**: `pip download --no-deps --require-hashes --only-binary :all:` for CPython 3.12 on manylinux x86_64 fetched all 29 wheels with matching hashes. 825 passed, 8 skipped; `ruff check` and `ruff format --check` pass. `publish.yml` takes effect once `dev` is merged into `main`. +- **Docs**: `CLAUDE.md` › Branching & CI and `architecture.md` §2 and §3 describe the lock. The READMEs and the Sphinx pages do not describe the publish jobs, so they are unchanged. +- **Files**: `.github/requirements/publish.in`, `.github/requirements/publish.txt`, `.github/workflows/ci-dev.yml`, `.github/workflows/publish.yml`, `.github/dependabot.yml`, `tests/test_workflow_actions.py`, `CLAUDE.md`, `architecture.md`, `progress.md`. +- **Open items**: `progress.md` #9 (the build backend is still fetched unpinned by `python -m build`). diff --git a/docs/updates/README.md b/docs/updates/README.md index 9150853..ab9fe3c 100644 --- a/docs/updates/README.md +++ b/docs/updates/README.md @@ -58,6 +58,7 @@ In the same commit: delete the item from `progress.md`, add a `#done` entry here | ID | Date | Title | Tags | Batch | |---|---|---|---|---| +| U-20261001-10 | 2026-10-01 | The publish jobs install hash-locked build tools | #done #ci #security #deps | [2026-10](2026-10.md) | | U-20261001-09 | 2026-10-01 | The source distributions stop carrying the tests | #done #packaging #tests | [2026-10](2026-10.md) | | U-20261001-08 | 2026-10-01 | The wheels stop installing the test suite | #done #packaging #tests | [2026-10](2026-10.md) | | U-20261001-07 | 2026-10-01 | CI publishes automation_file_dev from the dev branch | #release #ci #X-13 | [2026-10](2026-10.md) | @@ -93,5 +94,5 @@ In the same commit: delete the item from `progress.md`, add a `#done` entry here | File | Period | Entries | |---|---|---:| -| [2026-10.md](2026-10.md) | 2026-10 | 9 | +| [2026-10.md](2026-10.md) | 2026-10 | 10 | | [2026-09.md](2026-09.md) | 2026-09 | 21 | diff --git a/progress.md b/progress.md index d683eef..f93d1b4 100644 --- a/progress.md +++ b/progress.md @@ -5,3 +5,5 @@ Item numbers (`#n`) are never reused. Tags: [DECIDE] needs the owner's decision, Cross-repo and workspace items live in `D:\Codes\progress.md` (relevant here: X-12, X-13). ## Open + +- **#9** [DECIDE] The publish jobs still fetch the build backend unpinned. `python -m build` (`.github/workflows/ci-dev.yml:98`, `.github/workflows/publish.yml:74`) creates an isolated environment and installs `setuptools>=77` from `[build-system] requires` (`stable.toml:3`, `dev.toml:5`) at whatever version is newest, outside `.github/requirements/publish.txt`. Closing it means adding `setuptools` to `publish.in` and building with `python -m build --no-isolation`, or pointing the isolated install at a hash-locked constraints file. The other repositories locked under workspace X-13 build the same way. diff --git a/tests/test_workflow_actions.py b/tests/test_workflow_actions.py index 992cdc7..fb6ab27 100644 --- a/tests/test_workflow_actions.py +++ b/tests/test_workflow_actions.py @@ -13,6 +13,7 @@ from pathlib import Path import pytest +import yaml _ROOT = next(p for p in Path(__file__).resolve().parents if (p / ".github" / "workflows").is_dir()) _WORKFLOWS = sorted((_ROOT / ".github" / "workflows").glob("*.yml")) @@ -36,6 +37,14 @@ def test_workflows_exist(): assert _WORKFLOWS +@pytest.mark.parametrize("workflow", _WORKFLOWS, ids=lambda p: p.name) +def test_every_workflow_is_valid_yaml(workflow): + # GitHub runs nothing from a workflow it cannot parse, and the checks below read the files as + # text, so they would not notice. One way to get there: a one-line `run:` holding ": ", as in + # pip's `--only-binary :all: -r`. Such a command needs a block (`run: |`). + assert yaml.safe_load(workflow.read_text(encoding="utf-8"))["jobs"] + + @pytest.mark.parametrize("workflow", _WORKFLOWS, ids=lambda p: p.name) def test_every_action_is_pinned_to_a_commit_with_its_version(workflow): bad = [ @@ -58,8 +67,7 @@ def test_one_version_per_action(): def test_dependabot_keeps_pins_current_on_dev(): # Pinned SHAs only stay current if something bumps them; every update - # goes to dev because main is the release branch. Parsed as text: PyYAML - # is not a test dependency. + # goes to dev because main is the release branch. text = (_ROOT / ".github" / "dependabot.yml").read_text(encoding="utf-8") blocks = re.split(r"^\s*-\s*package-ecosystem:", text, flags=re.MULTILINE)[1:] ecosystems = {block.split()[0].strip("\"'") for block in blocks} @@ -133,3 +141,71 @@ def test_every_job_has_a_timeout(workflow): if "runs-on:" in body and not re.search(r"^\s*timeout-minutes:", body, re.MULTILINE) ] assert bad == [] + + +_REQUIREMENTS = _ROOT / ".github" / "requirements" +_LOCKED_INSTALL = ( + "python -m pip install --require-hashes --only-binary :all: -r .github/requirements/publish.txt" +) +_PIP_INSTALL = re.compile(r"\bpip\d*\s+install\b") +_STEP_PREFIX = re.compile(r"^\s*(?:-\s*)?(?:run:\s*)?") +_TOOL = re.compile(r"^([A-Za-z0-9][A-Za-z0-9_.-]*)", re.MULTILINE) +_PIN = re.compile(r"^([A-Za-z0-9][A-Za-z0-9_.-]*)==", re.MULTILINE) + + +def _publish_jobs() -> list[tuple[str, str]]: + """Return ``(workflow:job, job text)`` for each job that is given the PyPI token.""" + return [ + (f"{workflow.name}:{name}", body) + for workflow in _WORKFLOWS + for name, body in _jobs(workflow) + if "secrets.PYPI_API_TOKEN" in body + ] + + +def _pip_installs(job: str) -> list[str]: + """Return each ``pip install`` command of a job, without its YAML key and any comment.""" + commands = [] + for line in job.splitlines(): + code = line.split("#", 1)[0] + if _PIP_INSTALL.search(code): + commands.append(_STEP_PREFIX.sub("", code).strip()) + return commands + + +def _normalised(names: list[str]) -> set[str]: + """Return project names in the form PyPI compares them: lower case, runs of ``-_.`` as ``-``.""" + return {re.sub(r"[-_.]+", "-", name).lower() for name in names} + + +def test_the_publish_jobs_are_the_two_known_ones(): + # A new job that is given the token has to be looked at against the rule below. + assert [name for name, _body in _publish_jobs()] == [ + "ci-dev.yml:publish-dev", + "publish.yml:publish", + ] + + +@pytest.mark.parametrize("job", _publish_jobs(), ids=lambda job: job[0]) +def test_publish_jobs_install_only_hash_locked_tools(job): + # What such a job installs runs beside the PyPI token and builds the files it uploads. Its one + # install takes wheels whose hashes are in publish.txt: no `pip install --upgrade pip` and no + # unpinned package, so a release published a minute ago cannot reach the job. + _name, body = job + assert _pip_installs(body) == [_LOCKED_INSTALL] + + +def test_publish_lock_pins_every_tool_in_publish_in(): + # A tool named in publish.in but not pinned in publish.txt means the lock was not regenerated. + tools = _normalised(_TOOL.findall((_REQUIREMENTS / "publish.in").read_text(encoding="utf-8"))) + pinned = _normalised(_PIN.findall((_REQUIREMENTS / "publish.txt").read_text(encoding="utf-8"))) + assert tools and tools <= pinned + + +def test_dependabot_reads_the_publish_lock(): + # Locked versions only move when Dependabot is told which directory holds the lock. + text = (_ROOT / ".github" / "dependabot.yml").read_text(encoding="utf-8") + blocks = re.split(r"^\s*-\s*package-ecosystem:", text, flags=re.MULTILINE)[1:] + pip = [block for block in blocks if block.split()[0].strip("\"'") == "pip"] + listed = re.compile(r"^\s*(?:-|directory:)\s*\"/\.github/requirements\"", re.MULTILINE) + assert any(listed.search(block) for block in pip) From a0dd11f7305d6f37fb2b3540e56d31db48dd4fb0 Mon Sep 17 00:00:00 2001 From: JeffreyChen Date: Thu, 1 Oct 2026 21:54:16 +0800 Subject: [PATCH 19/59] ci: build with the locked setuptools in the publish jobs --- .github/requirements/publish.in | 4 ++ .github/requirements/publish.txt | 4 ++ .github/workflows/ci-dev.yml | 3 +- .github/workflows/publish.yml | 3 +- CLAUDE.md | 3 +- architecture.md | 5 ++- docs/updates/2026-10.md | 15 ++++++++ docs/updates/README.md | 3 +- progress.md | 2 - tests/test_dev_release.py | 3 +- tests/test_workflow_actions.py | 65 ++++++++++++++++++++++++++------ 11 files changed, 91 insertions(+), 19 deletions(-) diff --git a/.github/requirements/publish.in b/.github/requirements/publish.in index 205044e..2826993 100644 --- a/.github/requirements/publish.in +++ b/.github/requirements/publish.in @@ -5,3 +5,7 @@ # --exclude-newer leaves out releases under a week old, the wait dependabot.yml sets. build twine +# The build backend. The jobs run `python -m build --no-isolation`, so the backend is this locked one +# and not whatever is newest on PyPI when the job runs. It must satisfy `build-system.requires` in +# stable.toml and dev.toml (tests/test_workflow_actions.py). +setuptools diff --git a/.github/requirements/publish.txt b/.github/requirements/publish.txt index 5eec725..ba43a8a 100644 --- a/.github/requirements/publish.txt +++ b/.github/requirements/publish.txt @@ -455,6 +455,10 @@ secretstorage==3.5.0 \ --hash=sha256:0ce65888c0725fcb2c5bc0fdb8e5438eece02c523557ea40ce0703c266248137 \ --hash=sha256:f04b8e4689cbce351744d5537bf6b1329c6fc68f91fa666f60a380edddcd11be # via keyring +setuptools==84.0.0 \ + --hash=sha256:51a52592b3b99e102b609654876bd65f19f999935166d1352678931132b0c670 \ + --hash=sha256:f4695c21257f0d9b537ec2692c941d02ee143b7cc1276941349a546573b2ef73 + # via -r .github/requirements/publish.in twine==7.0.0 \ --hash=sha256:85cdb29c518efef867360ae4acd4b0dfd61c8654a22fca08e6f8539f05022177 \ --hash=sha256:b854164df26db268af05f49aa5c0344b10e27a494343ff05b1e0bad3b135f5a7 diff --git a/.github/workflows/ci-dev.yml b/.github/workflows/ci-dev.yml index 5ea6465..8940b0c 100644 --- a/.github/workflows/ci-dev.yml +++ b/.github/workflows/ci-dev.yml @@ -94,8 +94,9 @@ jobs: python -m pip install --require-hashes --only-binary :all: -r .github/requirements/publish.txt - name: Write pyproject.toml from dev.toml with the next version run: python scripts/dev_release.py prepare + # --no-isolation: the backend is the setuptools locked in publish.txt, not a fresh download. - name: Build distribution - run: python -m build + run: python -m build --no-isolation - name: Verify distribution metadata run: python -m twine check dist/* - name: Compare with the newest published wheel diff --git a/.github/workflows/publish.yml b/.github/workflows/publish.yml index 3a4c7e1..10c3204 100644 --- a/.github/workflows/publish.yml +++ b/.github/workflows/publish.yml @@ -70,8 +70,9 @@ jobs: - name: Use stable.toml as pyproject.toml run: cp stable.toml pyproject.toml + # --no-isolation: the backend is the setuptools locked in publish.txt, not a fresh download. - name: Build distribution - run: python -m build + run: python -m build --no-isolation - name: Publish to PyPI env: diff --git a/CLAUDE.md b/CLAUDE.md index 1c6b35a..75b9ea9 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -72,7 +72,8 @@ automation_file/ - CI steps: `lint` (ruff check + ruff format --check + mypy) → `pytest` with coverage → uploads `coverage.xml` as an artifact. - Stable publishing lives in a separate workflow (`.github/workflows/publish.yml`) that runs on push to `main`: bumps both TOMLs, copies `stable.toml` to `pyproject.toml`, builds the sdist + wheel, `twine upload` via `PYPI_API_TOKEN`, then commits + tags + pushes and creates `gh release create v --generate-notes`. - Dev publishing is the `publish-dev` job at the end of `ci-dev.yml`. It runs only on a push to `dev`, after `lint` and `pytest` pass: `scripts/dev_release.py prepare` writes `pyproject.toml` from `dev.toml` with one patch above the newest `automation_file_dev` on PyPI, the job builds and runs `twine check`, and it uploads (same `PYPI_API_TOKEN`) only when the commit is still the tip of `dev` and the wheel differs from the newest published one. Nothing is committed back. -- Both publish jobs hold `PYPI_API_TOKEN`, so they install their tools (`build`, `twine`) with one command and nothing else: `python -m pip install --require-hashes --only-binary :all: -r .github/requirements/publish.txt`. No `pip install --upgrade pip`, no unpinned install; `tests/test_workflow_actions.py` fails on any other `pip install` in a job that is given the token. To add or raise a tool, edit `.github/requirements/publish.in` and regenerate `publish.txt` with the `uv pip compile` command written in that file. Dependabot reads the directory and proposes updates on `dev`. +- Both publish jobs hold `PYPI_API_TOKEN`, so they install their tools (`build`, `twine`, and the build backend `setuptools`) with one command and nothing else: `python -m pip install --require-hashes --only-binary :all: -r .github/requirements/publish.txt`. No `pip install --upgrade pip`, no unpinned install; `tests/test_workflow_actions.py` fails on any other `pip install` in a job that is given the token. To add or raise a tool, edit `.github/requirements/publish.in` and regenerate `publish.txt` with the `uv pip compile` command written in that file. Dependabot reads the directory and proposes updates on `dev`. +- Both publish jobs build with `python -m build --no-isolation`, so the backend is the locked `setuptools` and nothing is downloaded at build time. `--no-isolation` checks `[build-system] requires` against what is installed instead of installing it: when you raise that floor in `stable.toml` and `dev.toml`, or add a build requirement, regenerate `publish.txt` in the same commit. `tests/test_workflow_actions.py` fails on a build without `--no-isolation` in those jobs and on a build requirement the lock does not satisfy. - `pre-commit` is configured (`.pre-commit-config.yaml`): trailing-whitespace, eof-fixer, check-yaml, check-toml, check-added-large-files, ruff, ruff-format, mypy. Install with `pre-commit install` after cloning. ## Development diff --git a/architecture.md b/architecture.md index 3ec0050..bf590df 100644 --- a/architecture.md +++ b/architecture.md @@ -30,7 +30,7 @@ the CLI, over loopback TCP or HTTP servers, as MCP tools, or from the PySide6 GU | `stable.toml`, `dev.toml` | Packaging for `automation_file` and `automation_file_dev`. No `pyproject.toml` is committed; CI and the publish jobs write one of these TOMLs into place. Apart from the name, version and description they say the same thing (`tests/test_dev_toml_parity.py`) | | `MANIFEST.in` | Keeps `tests/` out of both source distributions (`tests/test_sdist_manifest.py`); package discovery in the TOMLs already keeps it out of the wheels | | `scripts/dev_release.py` | Release helper for the dev channel (standard library only): picks the next `automation_file_dev` version from PyPI and tells whether the built wheel differs from the newest published one | -| `.github/requirements/publish.in`, `publish.txt` | The tools of the two publish jobs (`build`, `twine`) and their hash-locked resolution for Python 3.12 on Linux. `publish.in` holds the `uv pip compile` command that regenerates `publish.txt`; Dependabot reads the directory | +| `.github/requirements/publish.in`, `publish.txt` | The tools of the two publish jobs (`build`, `twine`, and the build backend `setuptools`) and their hash-locked resolution for Python 3.12 on Linux. `publish.in` holds the `uv pip compile` command that regenerates `publish.txt`; Dependabot reads the directory | | `main_ui.py` | Development shortcut for `launch_ui()` | | `tests/`, `docs/`, `examples/mcp/` | pytest suite (fixtures in `tests/conftest.py`); Sphinx docs; MCP host configuration example | @@ -71,6 +71,9 @@ the CLI, over loopback TCP or HTTP servers, as MCP tools, or from the PySide6 GU - Both jobs hold the PyPI token and install only the wheels pinned by hash in `.github/requirements/publish.txt` (`pip install --require-hashes --only-binary :all:`); `tests/test_workflow_actions.py` fails on any other `pip install` in them. + - Both jobs build with `python -m build --no-isolation`, so the build backend is the locked + `setuptools` and not a download made at build time. The same test file fails on a build without + the flag and when the lock does not satisfy `[build-system] requires` of `stable.toml` or `dev.toml`. ## 4. Main flows diff --git a/docs/updates/2026-10.md b/docs/updates/2026-10.md index 5ddcc42..ac8c919 100644 --- a/docs/updates/2026-10.md +++ b/docs/updates/2026-10.md @@ -177,3 +177,18 @@ Index and query commands: [README.md](README.md). New entries go at the end. - **Docs**: `CLAUDE.md` › Branching & CI and `architecture.md` §2 and §3 describe the lock. The READMEs and the Sphinx pages do not describe the publish jobs, so they are unchanged. - **Files**: `.github/requirements/publish.in`, `.github/requirements/publish.txt`, `.github/workflows/ci-dev.yml`, `.github/workflows/publish.yml`, `.github/dependabot.yml`, `tests/test_workflow_actions.py`, `CLAUDE.md`, `architecture.md`, `progress.md`. - **Open items**: `progress.md` #9 (the build backend is still fetched unpinned by `python -m build`). + +## U-20261001-11 · 2026-10-01 · The publish jobs build with the locked setuptools · #done #ci #security #X-13 + +- **What**: `progress.md` #9, decided by the owner (workspace X-13). The two jobs that are given `PYPI_API_TOKEN` installed their tools from the hash-locked `publish.txt` (U-20261001-10), but `python -m build` then created an isolated environment and downloaded whatever `setuptools` was newest (`[build-system] requires` is `setuptools>=77`), outside the lock, and ran it in the job that uploads. + - `publish.in` now lists `setuptools`. `publish.txt`, regenerated with the command in `publish.in` and the same `--exclude-newer 2026-09-24T00:00:00Z`, gains `setuptools==84.0.0` with its two hashes; the other 29 pins and their hashes are unchanged (30 packages, 397 hashes). + - Both jobs (`publish-dev` in `ci-dev.yml`, `publish` in `publish.yml`) run `python -m build --no-isolation`, so the backend is the locked one. Nothing else about the jobs changed. +- **Tests**: `tests/test_workflow_actions.py` gains two guards. + - Every build command (`python -m build`, `pyproject-build`) in a job that is given the token carries `--no-isolation`, and each such job has one. The previous steps fail it, as do a second build without the flag and the flag written only in a comment. + - Every entry of `[build-system] requires` in `stable.toml` and `dev.toml` names a package pinned in `publish.txt` at a version the entry accepts. `--no-isolation` checks the requirement instead of installing it, so a raised floor (`setuptools>=85`), a cap below the pin or a new build requirement (`wheel`) without a regenerated lock fails here and not in the publish job. +- **Result / numbers**: built in a throwaway worktree with Python 3.12 and a scratch environment holding `build` 1.6.1, `setuptools` 84.0.0, `packaging` 26.3, `pyproject-hooks` 1.3.3 and `twine` 7.0.0 (no `wheel`, no `pip`): the dev pair from `scripts/dev_release.py prepare` (0.0.36), the stable pair from `stable.toml` copied to `pyproject.toml` (0.0.31, without the job's version bump), each once isolated and once with `--no-isolation`. + - Both ways give the same files: wheels of 167 members (161 under `automation_file/`, none under `tests/`), sdists of 173 members, same member lists and every member byte-identical. The newest `setuptools` on PyPI today is 84.0.0 as well, so the isolated builds used the same backend version. + - `twine check` passes for all eight files. 829 passed, 8 skipped; `ruff check` and `ruff format --check` pass. +- **Docs**: `CLAUDE.md` › Branching & CI and `architecture.md` §2 and §3 say how the jobs build and what to regenerate when the build requirement changes. The READMEs and the Sphinx pages do not describe the publish jobs, so they are unchanged. +- **Files**: `.github/requirements/publish.in`, `.github/requirements/publish.txt`, `.github/workflows/ci-dev.yml`, `.github/workflows/publish.yml`, `tests/test_workflow_actions.py`, `tests/test_dev_release.py`, `CLAUDE.md`, `architecture.md`, `progress.md`. +- **Open items**: none. `publish.yml` runs only on `main`, so its build step takes effect, and is first exercised, when `dev` is merged. diff --git a/docs/updates/README.md b/docs/updates/README.md index ab9fe3c..5308af4 100644 --- a/docs/updates/README.md +++ b/docs/updates/README.md @@ -58,6 +58,7 @@ In the same commit: delete the item from `progress.md`, add a `#done` entry here | ID | Date | Title | Tags | Batch | |---|---|---|---|---| +| U-20261001-11 | 2026-10-01 | The publish jobs build with the locked setuptools | #done #ci #security #X-13 | [2026-10](2026-10.md) | | U-20261001-10 | 2026-10-01 | The publish jobs install hash-locked build tools | #done #ci #security #deps | [2026-10](2026-10.md) | | U-20261001-09 | 2026-10-01 | The source distributions stop carrying the tests | #done #packaging #tests | [2026-10](2026-10.md) | | U-20261001-08 | 2026-10-01 | The wheels stop installing the test suite | #done #packaging #tests | [2026-10](2026-10.md) | @@ -94,5 +95,5 @@ In the same commit: delete the item from `progress.md`, add a `#done` entry here | File | Period | Entries | |---|---|---:| -| [2026-10.md](2026-10.md) | 2026-10 | 10 | +| [2026-10.md](2026-10.md) | 2026-10 | 11 | | [2026-09.md](2026-09.md) | 2026-09 | 21 | diff --git a/progress.md b/progress.md index f93d1b4..d683eef 100644 --- a/progress.md +++ b/progress.md @@ -5,5 +5,3 @@ Item numbers (`#n`) are never reused. Tags: [DECIDE] needs the owner's decision, Cross-repo and workspace items live in `D:\Codes\progress.md` (relevant here: X-12, X-13). ## Open - -- **#9** [DECIDE] The publish jobs still fetch the build backend unpinned. `python -m build` (`.github/workflows/ci-dev.yml:98`, `.github/workflows/publish.yml:74`) creates an isolated environment and installs `setuptools>=77` from `[build-system] requires` (`stable.toml:3`, `dev.toml:5`) at whatever version is newest, outside `.github/requirements/publish.txt`. Closing it means adding `setuptools` to `publish.in` and building with `python -m build --no-isolation`, or pointing the isolated install at a hash-locked constraints file. The other repositories locked under workspace X-13 build the same way. diff --git a/tests/test_dev_release.py b/tests/test_dev_release.py index 35bd189..90c6cc3 100644 --- a/tests/test_dev_release.py +++ b/tests/test_dev_release.py @@ -160,7 +160,8 @@ def test_the_workflow_publishes_only_a_tested_push_to_dev(): def test_the_workflow_uploads_only_a_changed_build_and_keeps_no_credentials(): job = _publish_job() upload = job.index("twine upload") - assert job.index("dev_release.py prepare") < job.index("python -m build") < upload + build = job.index("python -m build --no-isolation") + assert job.index("dev_release.py prepare") < build < upload assert job.index("dev_release.py changed dist") < upload assert job.index("git ls-remote origin refs/heads/dev") < upload guard = "if: steps.compare.outputs.changed == 'true' && steps.tip.outputs.current == 'true'" diff --git a/tests/test_workflow_actions.py b/tests/test_workflow_actions.py index fb6ab27..4e7a321 100644 --- a/tests/test_workflow_actions.py +++ b/tests/test_workflow_actions.py @@ -10,10 +10,18 @@ from __future__ import annotations import re +import sys from pathlib import Path import pytest import yaml +from packaging.requirements import Requirement +from packaging.utils import canonicalize_name + +if sys.version_info >= (3, 11): + import tomllib +else: + import tomli as tomllib # declared in *.toml for Python<3.11 _ROOT = next(p for p in Path(__file__).resolve().parents if (p / ".github" / "workflows").is_dir()) _WORKFLOWS = sorted((_ROOT / ".github" / "workflows").glob("*.yml")) @@ -148,9 +156,12 @@ def test_every_job_has_a_timeout(workflow): "python -m pip install --require-hashes --only-binary :all: -r .github/requirements/publish.txt" ) _PIP_INSTALL = re.compile(r"\bpip\d*\s+install\b") +_BUILD = re.compile(r"\bpython\S*\s+-m\s+build\b|\bpyproject-build\b") _STEP_PREFIX = re.compile(r"^\s*(?:-\s*)?(?:run:\s*)?") _TOOL = re.compile(r"^([A-Za-z0-9][A-Za-z0-9_.-]*)", re.MULTILINE) -_PIN = re.compile(r"^([A-Za-z0-9][A-Za-z0-9_.-]*)==", re.MULTILINE) +_PIN = re.compile(r"^([A-Za-z0-9][A-Za-z0-9_.-]*)==(\S+)", re.MULTILINE) +# The metadata the publish jobs build from: each writes one of these to pyproject.toml. +_METADATA = ["stable.toml", "dev.toml"] def _publish_jobs() -> list[tuple[str, str]]: @@ -163,19 +174,32 @@ def _publish_jobs() -> list[tuple[str, str]]: ] -def _pip_installs(job: str) -> list[str]: - """Return each ``pip install`` command of a job, without its YAML key and any comment.""" +def _commands(job: str, pattern: re.Pattern[str]) -> list[str]: + """Return each command of a job that ``pattern`` finds, without its YAML key and any comment.""" commands = [] for line in job.splitlines(): code = line.split("#", 1)[0] - if _PIP_INSTALL.search(code): + if pattern.search(code): commands.append(_STEP_PREFIX.sub("", code).strip()) return commands -def _normalised(names: list[str]) -> set[str]: - """Return project names in the form PyPI compares them: lower case, runs of ``-_.`` as ``-``.""" - return {re.sub(r"[-_.]+", "-", name).lower() for name in names} +def _pins() -> dict[str, str]: + """Return ``{name: version}`` for each pin in ``publish.txt``, names as PyPI compares them.""" + text = (_REQUIREMENTS / "publish.txt").read_text(encoding="utf-8") + return {canonicalize_name(name): version for name, version in _PIN.findall(text)} + + +def _build_requires(metadata: str) -> list[Requirement]: + """Return ``build-system.requires`` of a metadata file in the repository root.""" + with (_ROOT / metadata).open("rb") as handle: + return [Requirement(item) for item in tomllib.load(handle)["build-system"]["requires"]] + + +def _is_locked(requirement: Requirement, pins: dict[str, str]) -> bool: + """Tell whether ``pins`` holds a version of the package that ``requirement`` accepts.""" + version = pins.get(canonicalize_name(requirement.name)) + return version is not None and requirement.specifier.contains(version) def test_the_publish_jobs_are_the_two_known_ones(): @@ -192,14 +216,33 @@ def test_publish_jobs_install_only_hash_locked_tools(job): # install takes wheels whose hashes are in publish.txt: no `pip install --upgrade pip` and no # unpinned package, so a release published a minute ago cannot reach the job. _name, body = job - assert _pip_installs(body) == [_LOCKED_INSTALL] + assert _commands(body, _PIP_INSTALL) == [_LOCKED_INSTALL] + + +@pytest.mark.parametrize("job", _publish_jobs(), ids=lambda job: job[0]) +def test_publish_jobs_build_with_the_locked_backend(job): + # An isolated build downloads the newest setuptools of that minute, outside publish.txt, and runs + # it beside the token. --no-isolation builds with the backend the locked install put in the job. + _name, body = job + builds = _commands(body, _BUILD) + assert builds and all("--no-isolation" in build.split() for build in builds) def test_publish_lock_pins_every_tool_in_publish_in(): # A tool named in publish.in but not pinned in publish.txt means the lock was not regenerated. - tools = _normalised(_TOOL.findall((_REQUIREMENTS / "publish.in").read_text(encoding="utf-8"))) - pinned = _normalised(_PIN.findall((_REQUIREMENTS / "publish.txt").read_text(encoding="utf-8"))) - assert tools and tools <= pinned + listed = _TOOL.findall((_REQUIREMENTS / "publish.in").read_text(encoding="utf-8")) + tools = {canonicalize_name(name) for name in listed} + assert tools and tools <= set(_pins()) + + +@pytest.mark.parametrize("metadata", _METADATA) +def test_publish_lock_satisfies_build_system_requires(metadata): + # --no-isolation checks build-system.requires against what is installed and installs nothing, so + # a backend that is not locked, or a floor raised without regenerating publish.txt, has to fail + # here and not in the publish job. + pins = _pins() + requires = _build_requires(metadata) + assert requires and [str(item) for item in requires if not _is_locked(item, pins)] == [] def test_dependabot_reads_the_publish_lock(): From 9f487c38a792b6d077c19432f09fb4e1d409775b Mon Sep 17 00:00:00 2001 From: JeffreyChen Date: Thu, 8 Oct 2026 11:45:26 +0800 Subject: [PATCH 20/59] feat: add the universal storage layer with local and memory backends --- CLAUDE.md | 12 + README.md | 59 +++ README.zh-CN.md | 56 +++ README.zh-TW.md | 56 +++ architecture.md | 35 ++ automation_file/__init__.py | 45 +++ automation_file/exceptions.py | 40 ++ automation_file/storage/__init__.py | 56 +++ automation_file/storage/backend.py | 412 +++++++++++++++++++ automation_file/storage/file.py | 195 +++++++++ automation_file/storage/local_storage.py | 252 ++++++++++++ automation_file/storage/memory_storage.py | 108 +++++ automation_file/storage/resolver.py | 152 ++++++++ automation_file/storage/storage.py | 151 +++++++ automation_file/storage/types.py | 121 ++++++ automation_file/storage/uri.py | 160 ++++++++ docs/source/API/api_index.rst | 14 + docs/source/API/storage.rst | 46 +++ docs/source/Eng/eng_index.rst | 15 + docs/source/Eng/usage/storage.rst | 272 +++++++++++++ docs/source/Zh-CN/usage/storage.rst | 255 ++++++++++++ docs/source/Zh-CN/zh_cn_index.rst | 15 + docs/source/Zh-TW/usage/storage.rst | 255 ++++++++++++ docs/source/Zh-TW/zh_tw_index.rst | 15 + docs/source/index.rst | 6 +- docs/updates/2026-10.md | 24 ++ docs/updates/README.md | 3 +- progress.md | 36 ++ tests/storage_contract.py | 456 ++++++++++++++++++++++ tests/test_storage_file.py | 386 ++++++++++++++++++ tests/test_storage_imports.py | 66 ++++ tests/test_storage_local.py | 208 ++++++++++ tests/test_storage_memory.py | 57 +++ tests/test_storage_resolver.py | 199 ++++++++++ tests/test_storage_types.py | 112 ++++++ tests/test_storage_uri.py | 241 ++++++++++++ 36 files changed, 4587 insertions(+), 4 deletions(-) create mode 100644 automation_file/storage/__init__.py create mode 100644 automation_file/storage/backend.py create mode 100644 automation_file/storage/file.py create mode 100644 automation_file/storage/local_storage.py create mode 100644 automation_file/storage/memory_storage.py create mode 100644 automation_file/storage/resolver.py create mode 100644 automation_file/storage/storage.py create mode 100644 automation_file/storage/types.py create mode 100644 automation_file/storage/uri.py create mode 100644 docs/source/API/storage.rst create mode 100644 docs/source/Eng/usage/storage.rst create mode 100644 docs/source/Zh-CN/usage/storage.rst create mode 100644 docs/source/Zh-TW/usage/storage.rst create mode 100644 tests/storage_contract.py create mode 100644 tests/test_storage_file.py create mode 100644 tests/test_storage_imports.py create mode 100644 tests/test_storage_local.py create mode 100644 tests/test_storage_memory.py create mode 100644 tests/test_storage_resolver.py create mode 100644 tests/test_storage_types.py create mode 100644 tests/test_storage_uri.py diff --git a/CLAUDE.md b/CLAUDE.md index 75b9ea9..c2f9a16 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -24,6 +24,9 @@ automation_file/ │ # subpackage per backend: google_drive, s3, azure_blob, dropbox_api, sftp, ftp, │ # onedrive, box (client.py + *_ops.py + register__ops); smb and webdav │ # have a client only +├── storage/ # Universal storage layer: uri (StorageURI), types (FileInfo, Checksum, +│ # StorageCapabilities), backend (StorageBackend contract), local_storage, +│ # memory_storage, resolver (StorageResolver), file (File), storage (Storage) ├── server/ # tcp_server, http_server, mcp_server (MCP over stdio), web_ui, metrics_server, │ # action_acl (ActionACL), network_guards (ensure_loopback) ├── client/ # HTTPActionClient for the HTTP action server @@ -61,6 +64,9 @@ automation_file/ - `Quota` — frozen dataclass capping bytes and wall-clock seconds per action or block (`check_size`, `time_budget` context manager, `wraps` decorator). `0` disables each cap. - `retry_on_transient(max_attempts, backoff_base, backoff_cap, retriable)` — decorator that retries with capped exponential back-off and raises `RetryExhaustedException` chained to the last error. - `safe_join(root, user_path)` / `is_within(root, path)` — path traversal guard; `safe_join` raises `PathTraversalException` when the resolved path escapes `root`. +- `File(uri)` / `Storage(uri)` — the universal storage layer's application API: one file, one directory, in any backend. Both resolve their backend on every call through `StorageResolver` (`Storage.mount`, `Storage.register_scheme`). +- `StorageBackend` — the contract a storage backend implements. The public operations (`exists`, `stat`, `list_dir`, `mkdir`, `upload`, `download`, `delete`, `checksum`, `read_bytes`, `write_bytes`, `copy_from`, `move_from`) are template methods; a backend supplies only the `_`-prefixed primitives. `LocalStorage` and `MemoryStorage` are built in. +- `StorageURI` / `parse_storage_uri` — `:///`; `FileInfo`, `Checksum`, `StorageCapabilities` are the frozen value types the layer returns. ## Branching & CI @@ -144,6 +150,12 @@ All code must follow secure-by-default principles. Review every change against t ### Path traversal - Any caller resolving a user-supplied path against a trusted root must go through `automation_file.local.safe_paths.safe_join` (raises `PathTraversalException`) or the `is_within` check. Never concatenate + `Path.resolve()` yourself and skip the containment check — symlinks and `..` segments bypass naive string checks. +### Storage layer +- A new storage backend subclasses `StorageBackend` and passes `tests/storage_contract.py` through a `StorageContract` subclass. Do not weaken a contract case to make a backend pass: fix the backend, or branch on `capabilities` when backends legitimately differ. +- Keep the checks that live in the base class: `normalize_path` refuses `..`, `parse_storage_uri` refuses credentials in the authority (and its error does not repeat them), `delete` refuses the storage root, and `LocalStorage` deletes a symbolic link without following it. Never log a storage URI's credentials or a backend's secrets. +- When paths come from outside the process, use `LocalStorage(root)` behind a scheme or authority of its own (`Storage.mount("sandbox://jobs", LocalStorage(root))`), not the rootless `local://` backend. +- At module level, `automation_file/storage/` imports only the standard library, `exceptions`, `core.checksum` and `local.safe_paths`; `tests/test_storage_imports.py` fails on anything else. A backend SDK is imported lazily, inside the function that needs it. + ### SFTP host verification - `SFTPClient` uses `paramiko.RejectPolicy()` — unknown hosts are rejected, never auto-added. Callers pass `known_hosts=` explicitly or rely on `~/.ssh/known_hosts`. Do not swap in `AutoAddPolicy` for convenience. diff --git a/README.md b/README.md index ed85aee..13585ef 100644 --- a/README.md +++ b/README.md @@ -48,6 +48,7 @@ facade. - **HTTP server observability** — `GET /healthz` / `GET /readyz` probes, `GET /openapi.json` spec, and `GET /progress` WebSocket stream of live transfer snapshots - **HTMX Web UI** — `start_web_ui()` serves a read-only dashboard (health, progress, registry) that polls HTML fragments; stdlib-only HTTP plus one CDN script with SRI - **MCP (Model Context Protocol) server** — `MCPServer` bridges the registry to any MCP host (Claude Desktop, MCP CLIs) over newline-delimited JSON-RPC 2.0 on stdio; every `FA_*` action becomes an MCP tool with an auto-generated input schema +- **Universal storage layer** — `File` / `Storage` address local and remote storage with one URI syntax (`local:///…`, `memory://…`), one `StorageBackend` contract and one error hierarchy; `LocalStorage` and `MemoryStorage` are built in, and a 70-case contract suite checks any backend - PySide6 GUI (`python -m automation_file ui`) with a tab per backend, the JSON-action runner, and dedicated tabs for Triggers, Scheduler, and live Progress - Rich CLI with one-shot subcommands plus legacy JSON-batch flags - Project scaffolding (`ProjectBuilder`) for executor-based automations @@ -146,6 +147,12 @@ flowchart TD Cross["cross_backend
local:// s3:// azure://
dropbox:// sftp:// ftp://"] end + subgraph StorageLayer["storage (universal layer)"] + FileAPI["File · Storage
local:// memory:// …"] + Resolver["StorageResolver
mounts · scheme factories"] + Backends["StorageBackend contract
LocalStorage · MemoryStorage"] + end + subgraph Notify["notifications"] NM["NotificationManager
fanout · dedup · SSRF guard"] Sinks["Sinks
Webhook · Slack · Email
Telegram · Discord · Teams · PagerDuty"] @@ -177,6 +184,11 @@ flowchart TD PublicAPI ==> NM PublicAPI ==> Trigger PublicAPI ==> Sched + PublicAPI ==> FileAPI + FileAPI ==> Resolver + Resolver ==> Backends + Backends ==> SafeP + Backends ==> Check TCP ==> Executor HTTPS ==> Executor @@ -268,6 +280,7 @@ flowchart TD classDef remote fill:#D5F5E3,stroke:#196F3D,stroke-width:3px,color:#000,font-weight:bold; classDef notify fill:#F9E79F,stroke:#7D6608,stroke-width:3px,color:#000,font-weight:bold; classDef utils fill:#EAEDED,stroke:#212F3C,stroke-width:3px,color:#000,font-weight:bold; + classDef storage fill:#D4E6F1,stroke:#1A5276,stroke-width:3px,color:#000,font-weight:bold; class CLI,GUIUser,ClientSDK,MCPHost,Plugins entry; class PublicAPI facade; @@ -282,6 +295,7 @@ flowchart TD class UrlVal,Http,Drive,S3M,Azure,Dropbox,SFTP,FTP,OneD,Box,WebDAV,SMB,Fsspec,Cross remote; class NM,Sinks notify; class Fast,Dedup,Grep,Rotate,Discovery,Builder utils; + class FileAPI,Resolver,Backends storage; linkStyle default stroke:#1F2A44,stroke-width:2.5px; ``` @@ -421,6 +435,51 @@ All backends (`s3`, `azure_blob`, `dropbox_api`, `sftp`) expose the same five operations: `upload_file`, `upload_dir`, `download_file`, `delete_*`, `list_*`. SFTP uses `paramiko.RejectPolicy` — unknown hosts are rejected, not auto-added. +### Universal storage layer (File / Storage) +One URI syntax, one set of operations and one set of errors for every storage. +`File` is a single file, `Storage` a directory, and `StorageBackend` the contract a +backend implements. The `FA_*` actions and the per-backend functions keep working +unchanged next to it. + +```python +from automation_file import File, LocalStorage, Storage + +report = File("local:///data/reports/q1.csv") # a plain path works too +report.write("region,total\nEMEA,42\n") +report.size, report.modified_at, report.content_type +report.checksum() # Checksum("sha256", "…") +report.copy_to("memory://scratch/archive/q1.csv") # any backend to any backend +report.move_to("local:///data/done/q1.csv") + +reports = Storage("local:///data/reports") +for info in reports.list_dir(recursive=True): + print(info.path, info.size) + +# Confine untrusted paths: nothing under sandbox://jobs/ can leave /srv/jobs. +Storage.mount("sandbox://jobs", LocalStorage("/srv/jobs")) +File("sandbox://jobs/42/out.csv").write(b"done") +``` + +- **URIs** — `:///`: `local:///data/a.csv`, `s3://bucket/a.csv`, + `sftp://server/data/a.csv`. The path is literal (nothing is percent-decoded), `..` segments + are rejected, and credentials in the authority are refused. Text without `://` is a local path. +- **Operations** — `exists`, `stat`, `list_dir`, `mkdir`, `upload`, `download`, `delete`, + `checksum`, `read_bytes`, `write_bytes`, `copy_from`, `move_from`, identical on every backend. + Downloads and local writes are atomic, deleting a directory with entries needs + `recursive=True`, and the storage root is never deleted. +- **Errors** — `StorageException` and its subclasses: `StorageNotFoundException`, + `StorageAlreadyExistsException`, `StoragePathTypeException`, `StorageNotEmptyException`, + `StoragePermissionException`, `StorageTransientException`, `StorageUnavailableException`, + `StorageUnsupportedException`, `StorageURIException`. +- **Backends today** — `LocalStorage` (`local://`, optionally confined to a root through + `safe_join`) and `MemoryStorage` (`memory://`, for tests and dry runs). The cloud and SFTP + backends are still used through their own clients and `FA_*` actions; their adapters for + this layer are not written yet. Write your own by subclassing `StorageBackend` and check it with + the 70-case contract suite in `tests/storage_contract.py`. + +The API is new and may still change before 1.0. Full reference: the *Universal Storage Layer* +chapter of the documentation. + ### File-watcher triggers Run an action list whenever a filesystem event fires on a watched path: diff --git a/README.zh-CN.md b/README.zh-CN.md index ce4e224..b42c534 100644 --- a/README.zh-CN.md +++ b/README.zh-CN.md @@ -46,6 +46,7 @@ TCP / HTTP 服务器执行的 JSON 驱动动作。内附 PySide6 GUI，每个功 - **HTTP 服务器观测端点** — `GET /healthz` / `GET /readyz` 探针、`GET /openapi.json` 规格，以及 `GET /progress`（通过 WebSocket 推送实时传输快照） - **HTMX Web UI** — `start_web_ui()` 启动只读观测仪表板（health、progress、registry），通过 HTML 片段轮询；仅用标准库 HTTP，搭配一个带 SRI 的 CDN 脚本 - **MCP（Model Context Protocol）服务器** — `MCPServer` 通过 stdio 上的 JSON-RPC 2.0（换行分隔 JSON）将注册表桥接到任意 MCP 主机（Claude Desktop、MCP CLI）；每个 `FA_*` 动作都会自动生成输入 schema 并成为 MCP 工具 +- **通用存储层** — `File` / `Storage` 以同一套 URI 语法（`local:///…`、`memory://…`）、同一份 `StorageBackend` 契约与同一组异常层级访问本地与远端存储；内置 `LocalStorage` 与 `MemoryStorage`，并附带 70 个用例的契约测试套件可检查任何后端 - PySide6 GUI（`python -m automation_file ui`）每个后端一个页签，含 JSON 动作执行器，另有 Triggers、Scheduler、实时 Progress 专属页签 - 功能丰富的 CLI，包含一次性子命令与旧式 JSON 批量标志 - 项目脚手架（`ProjectBuilder`）协助构建以 executor 为核心的自动化项目 @@ -144,6 +145,12 @@ flowchart TD Cross["cross_backend
local:// s3:// azure://
dropbox:// sftp:// ftp://"] end + subgraph StorageLayer["通用存储层"] + FileAPI["File · Storage
local:// memory:// …"] + Resolver["StorageResolver
mounts · scheme factories"] + Backends["StorageBackend contract
LocalStorage · MemoryStorage"] + end + subgraph Notify["通知"] NM["NotificationManager
fanout · dedup · SSRF guard"] Sinks["Sinks
Webhook · Slack · Email
Telegram · Discord · Teams · PagerDuty"] @@ -175,6 +182,11 @@ flowchart TD PublicAPI ==> NM PublicAPI ==> Trigger PublicAPI ==> Sched + PublicAPI ==> FileAPI + FileAPI ==> Resolver + Resolver ==> Backends + Backends ==> SafeP + Backends ==> Check TCP ==> Executor HTTPS ==> Executor @@ -266,6 +278,7 @@ flowchart TD classDef remote fill:#D5F5E3,stroke:#196F3D,stroke-width:3px,color:#000,font-weight:bold; classDef notify fill:#F9E79F,stroke:#7D6608,stroke-width:3px,color:#000,font-weight:bold; classDef utils fill:#EAEDED,stroke:#212F3C,stroke-width:3px,color:#000,font-weight:bold; + classDef storage fill:#D4E6F1,stroke:#1A5276,stroke-width:3px,color:#000,font-weight:bold; class CLI,GUIUser,ClientSDK,MCPHost,Plugins entry; class PublicAPI facade; @@ -280,6 +293,7 @@ flowchart TD class UrlVal,Http,Drive,S3M,Azure,Dropbox,SFTP,FTP,OneD,Box,WebDAV,SMB,Fsspec,Cross remote; class NM,Sinks notify; class Fast,Dedup,Grep,Rotate,Discovery,Builder utils; + class FileAPI,Resolver,Backends storage; linkStyle default stroke:#1F2A44,stroke-width:2.5px; ``` @@ -419,6 +433,48 @@ execute_action([ `upload_file`、`upload_dir`、`download_file`、`delete_*`、`list_*`。 SFTP 使用 `paramiko.RejectPolicy` — 未知主机会被拒绝，不会自动加入。 +### 通用存储层（File / Storage） +每一种存储都使用同一套 URI 语法、同一组操作和同一组异常。`File` 代表单个文件， +`Storage` 代表目录，`StorageBackend` 则是后端需要实现的契约。`FA_*` 动作以及各后端原有的 +函数完全不变，可以与本层同时使用。 + +```python +from automation_file import File, LocalStorage, Storage + +report = File("local:///data/reports/q1.csv") # 也可以直接写普通路径 +report.write("region,total\nEMEA,42\n") +report.size, report.modified_at, report.content_type +report.checksum() # Checksum("sha256", "…") +report.copy_to("memory://scratch/archive/q1.csv") # 任何后端到任何后端 +report.move_to("local:///data/done/q1.csv") + +reports = Storage("local:///data/reports") +for info in reports.list_dir(recursive=True): + print(info.path, info.size) + +# 限制不受信任的路径：sandbox://jobs/ 之下的任何东西都离不开 /srv/jobs。 +Storage.mount("sandbox://jobs", LocalStorage("/srv/jobs")) +File("sandbox://jobs/42/out.csv").write(b"done") +``` + +- **URI** — `:///`：`local:///data/a.csv`、`s3://bucket/a.csv`、 + `sftp://server/data/a.csv`。路径按字面理解（不做百分号解码），`..` 段会被拒绝， + authority 中的凭据也会被拒绝。不含 `://` 的文本视为本地路径。 +- **操作** — `exists`、`stat`、`list_dir`、`mkdir`、`upload`、`download`、`delete`、 + `checksum`、`read_bytes`、`write_bytes`、`copy_from`、`move_from`，在每个后端上都相同。 + 下载与本地写入均为原子操作，删除内有条目的目录需要 `recursive=True`，存储的根目录 + 永远不会被删除。 +- **异常** — `StorageException` 及其子类：`StorageNotFoundException`、 + `StorageAlreadyExistsException`、`StoragePathTypeException`、`StorageNotEmptyException`、 + `StoragePermissionException`、`StorageTransientException`、`StorageUnavailableException`、 + `StorageUnsupportedException`、`StorageURIException`。 +- **目前的后端** — `LocalStorage`（`local://`，可通过 `safe_join` 限制在某个根目录内）与 + `MemoryStorage`（`memory://`，用于测试与试运行）。云端与 SFTP 后端目前仍通过各自的客户端与 + `FA_*` 动作使用，接入本层的适配器尚未完成。你可以继承 `StorageBackend` 编写自己的后端， + 并用 `tests/storage_contract.py` 中 70 个用例的契约测试套件检查。 + +此 API 为新功能，在 1.0 之前仍可能调整。完整说明请见文档的“通用存储层”章节。 + ### 文件监听触发 每当被监听路径发生文件系统事件，就执行动作清单： diff --git a/README.zh-TW.md b/README.zh-TW.md index 49352f4..5910c11 100644 --- a/README.zh-TW.md +++ b/README.zh-TW.md @@ -46,6 +46,7 @@ TCP / HTTP 伺服器執行的 JSON 驅動動作。內附 PySide6 GUI，每個功 - **HTTP 伺服器觀測端點** — `GET /healthz` / `GET /readyz` 探針、`GET /openapi.json` 規格、以及 `GET /progress`（以 WebSocket 推送即時傳輸快照） - **HTMX Web UI** — `start_web_ui()` 啟動唯讀觀測儀表板（health、progress、registry），以 HTML 片段輪詢；僅用標準函式庫 HTTP，搭配一支帶 SRI 的 CDN 腳本 - **MCP（Model Context Protocol）伺服器** — `MCPServer` 透過 stdio 上的 JSON-RPC 2.0（行分隔 JSON）將登錄表橋接到任何 MCP 主機（Claude Desktop、MCP CLI）；每個 `FA_*` 動作都會自動生成輸入 schema 並成為 MCP 工具 +- **通用儲存層** — `File` / `Storage` 以同一套 URI 語法（`local:///…`、`memory://…`）、同一份 `StorageBackend` 契約與同一組例外階層存取本機與遠端儲存；內建 `LocalStorage` 與 `MemoryStorage`，並附 70 個案例的契約測試套件可檢查任何後端 - PySide6 GUI（`python -m automation_file ui`）每個後端一個分頁，含 JSON 動作執行器，另有 Triggers、Scheduler、即時 Progress 專屬分頁 - 功能豐富的 CLI，包含一次性子指令與舊式 JSON 批次旗標 - 專案鷹架（`ProjectBuilder`）協助建立以 executor 為核心的自動化專案 @@ -144,6 +145,12 @@ flowchart TD Cross["cross_backend
local:// s3:// azure://
dropbox:// sftp:// ftp://"] end + subgraph StorageLayer["通用儲存層"] + FileAPI["File · Storage
local:// memory:// …"] + Resolver["StorageResolver
mounts · scheme factories"] + Backends["StorageBackend contract
LocalStorage · MemoryStorage"] + end + subgraph Notify["通知"] NM["NotificationManager
fanout · dedup · SSRF guard"] Sinks["Sinks
Webhook · Slack · Email
Telegram · Discord · Teams · PagerDuty"] @@ -175,6 +182,11 @@ flowchart TD PublicAPI ==> NM PublicAPI ==> Trigger PublicAPI ==> Sched + PublicAPI ==> FileAPI + FileAPI ==> Resolver + Resolver ==> Backends + Backends ==> SafeP + Backends ==> Check TCP ==> Executor HTTPS ==> Executor @@ -266,6 +278,7 @@ flowchart TD classDef remote fill:#D5F5E3,stroke:#196F3D,stroke-width:3px,color:#000,font-weight:bold; classDef notify fill:#F9E79F,stroke:#7D6608,stroke-width:3px,color:#000,font-weight:bold; classDef utils fill:#EAEDED,stroke:#212F3C,stroke-width:3px,color:#000,font-weight:bold; + classDef storage fill:#D4E6F1,stroke:#1A5276,stroke-width:3px,color:#000,font-weight:bold; class CLI,GUIUser,ClientSDK,MCPHost,Plugins entry; class PublicAPI facade; @@ -280,6 +293,7 @@ flowchart TD class UrlVal,Http,Drive,S3M,Azure,Dropbox,SFTP,FTP,OneD,Box,WebDAV,SMB,Fsspec,Cross remote; class NM,Sinks notify; class Fast,Dedup,Grep,Rotate,Discovery,Builder utils; + class FileAPI,Resolver,Backends storage; linkStyle default stroke:#1F2A44,stroke-width:2.5px; ``` @@ -419,6 +433,48 @@ execute_action([ 操作：`upload_file`、`upload_dir`、`download_file`、`delete_*`、`list_*`。 SFTP 使用 `paramiko.RejectPolicy` — 未知主機會被拒絕，不會自動加入。 +### 通用儲存層（File / Storage） +每一種儲存都使用同一套 URI 語法、同一組操作與同一組例外。`File` 代表單一檔案， +`Storage` 代表目錄，`StorageBackend` 則是後端要實作的契約。`FA_*` 動作與各後端原有的 +函式完全不變，可與本層並用。 + +```python +from automation_file import File, LocalStorage, Storage + +report = File("local:///data/reports/q1.csv") # 也可以直接寫一般路徑 +report.write("region,total\nEMEA,42\n") +report.size, report.modified_at, report.content_type +report.checksum() # Checksum("sha256", "…") +report.copy_to("memory://scratch/archive/q1.csv") # 任何後端到任何後端 +report.move_to("local:///data/done/q1.csv") + +reports = Storage("local:///data/reports") +for info in reports.list_dir(recursive=True): + print(info.path, info.size) + +# 限制不受信任的路徑：sandbox://jobs/ 之下的任何東西都離不開 /srv/jobs。 +Storage.mount("sandbox://jobs", LocalStorage("/srv/jobs")) +File("sandbox://jobs/42/out.csv").write(b"done") +``` + +- **URI** — `:///`：`local:///data/a.csv`、`s3://bucket/a.csv`、 + `sftp://server/data/a.csv`。路徑按字面解讀（不做百分比解碼），`..` 區段會被拒絕， + authority 中的憑證也會被拒絕。不含 `://` 的文字視為本機路徑。 +- **操作** — `exists`、`stat`、`list_dir`、`mkdir`、`upload`、`download`、`delete`、 + `checksum`、`read_bytes`、`write_bytes`、`copy_from`、`move_from`，在每個後端上都相同。 + 下載與本機寫入皆為原子操作，刪除內有項目的目錄需要 `recursive=True`，儲存的根目錄 + 永遠不會被刪除。 +- **例外** — `StorageException` 及其子類別：`StorageNotFoundException`、 + `StorageAlreadyExistsException`、`StoragePathTypeException`、`StorageNotEmptyException`、 + `StoragePermissionException`、`StorageTransientException`、`StorageUnavailableException`、 + `StorageUnsupportedException`、`StorageURIException`。 +- **目前的後端** — `LocalStorage`（`local://`，可透過 `safe_join` 限制在某個根目錄內）與 + `MemoryStorage`（`memory://`，用於測試與試跑）。雲端與 SFTP 後端目前仍透過各自的用戶端與 + `FA_*` 動作使用，接上本層的轉接器尚未完成。你可以繼承 `StorageBackend` 撰寫自己的後端， + 並用 `tests/storage_contract.py` 中 70 個案例的契約測試套件檢查。 + +此 API 為新功能，在 1.0 之前仍可能調整。完整說明請見文件的「通用儲存層」章節。 + ### 檔案監看觸發 每當被監看路徑發生檔案系統事件，就執行動作清單： diff --git a/architecture.md b/architecture.md index bf590df..2761d94 100644 --- a/architecture.md +++ b/architecture.md @@ -11,6 +11,10 @@ Dropbox, SFTP, FTP, OneDrive, Box, plus SMB and WebDAV clients). Every operation command in one `ActionRegistry`, so JSON action lists run the same way in-process, from files, from the CLI, over loopback TCP or HTTP servers, as MCP tools, or from the PySide6 GUI. +Next to the actions, `automation_file.storage` is the universal storage layer: one URI syntax, one +`StorageBackend` contract and one error hierarchy, used through `File` and `Storage`. It is the first +piece of the 1.0 roadmap (`docs/FILEAUTOMATION-1.0-ROADMAP.md`, PR #107); what is still open is in `progress.md`. + ## 2. Layers and directories | Path | Responsibility | @@ -20,6 +24,7 @@ the CLI, over loopback TCP or HTTP servers, as MCP tools, or from the PySide6 GU | `automation_file/core/` | Engine, on je_action_core: `action_registry.py` (`ActionRegistry`, a `CommandRegistry`; `build_default_registry`), `action_executor.py` (`ActionExecutor`, an `ActionExecutor` with strict actions, indexed records and the dry-run, validate, substitute and parallel extras; shared `executor`), `callback_executor.py`, `package_loader.py`, `plugins.py`, `dag_executor.py`, `action_queue.py`, `json_store.py`, `substitution.py`. Also cross-cutting helpers: `retry`, `quota`, `rate_limit`, `circuit_breaker`, `file_lock`, `sqlite_lock`, `checksum`, `manifest`, `crypto`, `secrets`, `config`, `config_watcher`, `audit`, `metrics`, `tracing`, `progress`, `fim`, `content_store` | | `automation_file/local/` | Local strategy modules: file, dir, zip, tar and archive ops, sync, diff, text/JSON/data edits, templates, versioning, trash, `shell_ops` (argv-only subprocess), conditional branches. `safe_paths.py` guards against path traversal | | `automation_file/remote/` | `url_validator.py` (SSRF guard), `http_download.py`, `cross_backend.py`, `fsspec_bridge.py`. One subpackage per backend: `google_drive/`, `s3/`, `azure_blob/`, `dropbox_api/`, `sftp/`, `ftp/`, `onedrive/`, `box/`, each with `client.py`, `*_ops.py` and `register__ops`. `smb/` and `webdav/` have a client only | +| `automation_file/storage/` | Universal storage layer. `uri.py` (`StorageURI`, `parse_storage_uri`, `normalize_path`), `types.py` (`FileInfo`, `Checksum`, `StorageCapabilities`), `backend.py` (`StorageBackend`: the public operations are template methods over the `_`-prefixed primitives a backend supplies), `local_storage.py` (`LocalStorage`, confined through `safe_join` when given a root), `memory_storage.py` (`MemoryStorage`), `resolver.py` (`StorageResolver`, `default_resolver`: mounts first, then scheme factories), `file.py` (`File`), `storage.py` (`Storage`). It imports only `exceptions`, `core.checksum` and `local.safe_paths`: no registry, no GUI, no backend SDK | | `automation_file/server/` | `tcp_server.py`, `http_server.py`, `mcp_server.py`, `web_ui.py`, `metrics_server.py`, `action_acl.py` (`ActionACL`), `network_guards.py` (`ensure_loopback`) | | `automation_file/client/` | `HTTPActionClient` for the HTTP action server | | `automation_file/trigger/`, `scheduler/`, `notify/` | Watchdog file triggers, cron scheduler, notification sinks. Each registers its own `FA_*` ops | @@ -41,6 +46,11 @@ the CLI, over loopback TCP or HTTP servers, as MCP tools, or from the PySide6 GU `executor`, `callback_executor`, `package_manager`, `ActionRegistry`, `build_default_registry`, `driver_instance` (Google Drive), `start_autocontrol_socket_server`, `start_http_action_server`, `HTTPActionClient`, `MCPServer`, `create_project_dir`, `launch_ui` (lazy). +- **Storage layer** (same facade): `File`, `Storage`, `StorageBackend`, `StorageResolver`, `StorageURI`, + `parse_storage_uri`, `FileInfo`, `Checksum`, `StorageCapabilities`, `LocalStorage`, `MemoryStorage`, and + `StorageException` with its nine subclasses. Storage URIs are `:///`; the built-in + schemes are `local` (alias `file`) and `memory`, and text without `://` is a local path. The API is + provisional until 1.0. It is not reachable through `FA_*` actions yet. - **Action format**: an action is `[name]`, `[name, {kwargs}]` or `[name, [args]]`. A file holds a list of actions or `{"auto_control": [...]}`. - **CLI** (`python -m automation_file`; no console script for it): @@ -103,6 +113,16 @@ ActionExecutor() → build_default_registry(): local + http + utils + drive comm → executor adds FA_execute_action, FA_execute_files, FA_execute_action_parallel, FA_validate ``` +**Storage URI → backend** + +``` +File(uri) / Storage(uri) → parse_storage_uri (scheme alias, authority check, path normalised, ".." refused) + → StorageResolver.resolve: longest mount at or above the URI, else the scheme's factory → (backend, path) + → StorageBackend public method: normalise, check what exists, make parents → _primitive of the backend + → FileInfo / Checksum / bytes, or a StorageException subclass +copy_to / move_to → target_backend.copy_from(source_backend, ...) → native (_copy_from / _move_from) or a local staging file +``` + ## 5. Extension points - **New local or utility action**: function in `local/_ops.py` (or `utils/`, `core/`) using @@ -117,6 +137,17 @@ ActionExecutor() → build_default_registry(): local + http + utils + drive comm 3. Add the SDK to `dependencies` in both `stable.toml` and `dev.toml`, then add facade exports. 4. Add `ui/tabs/_tab.py` and wire it into `ui/tabs/transfer_tab.py`. 5. Add tests; paths that need the network are not exercised in CI. +- **New storage backend** (the universal layer; separate from the `FA_*` backend above): + 1. Subclass `StorageBackend` in `storage/_storage.py`: set `scheme` and `capabilities`, implement + `_stat`, `_list_dir`, `_upload`, `_download`, `_delete_file`, plus `_mkdir` and `_rmdir` when + `capabilities.directories` is true. Map the SDK's errors to the `StorageException` subclasses and + import the SDK lazily. + 2. Register its factory in `register_default_schemes` (`storage/resolver.py`), or leave it to callers + to `Storage.mount(...)` when it needs connection arguments. + 3. Add `tests/test_storage_.py` with a `StorageContract` subclass (`tests/storage_contract.py`); + every backend passes the same suite. + 4. Export it from `storage/__init__.py` and the facade, and document its URI form in the three + `usage/storage.rst` pages and the READMEs. - **Outbound HTTP**: always call `validate_http_url` (`remote/url_validator.py`) first. - **Plugins**: an entry point in the group `automation_file.actions` (`core/plugins.py`), or `add_command_to_executor({...})` at runtime. `package_manager.add_package_to_executor` registers a @@ -174,6 +205,9 @@ ActionExecutor() → build_default_registry(): local + http + utils + drive comm TCP reads one `recv(8192)` payload; HTTP bodies are capped at 1 MB (§ Security › TCP server; › HTTP server). - Resolve user paths through `safe_join` / `is_within` (§ Security › Path traversal). SFTP keeps `paramiko.RejectPolicy()` (§ Security › SFTP host verification). +- The storage layer does not import the registry, the GUI or a backend SDK at import time. Storage paths + never contain `..`, storage URIs never carry credentials, `delete` never removes a storage root and + never follows a symbolic link, and every backend passes `tests/storage_contract.py`. - `retry_on_transient` retries only the listed exception types (§ Security › Reliability (retry / quota)). `PackageLoader` is eval-grade; never expose it remotely (§ Security › Plugin / package loading). No `FA_*` command reaches it, so je_action_core's package gate is off here (`tests/test_package_loader.py` fails if one @@ -194,6 +228,7 @@ ActionExecutor() → build_default_registry(): local + http + utils + drive comm - How either PyPI package is built or published changes. - The action format, the `auto_control` key, the registry build order, or plugin override semantics change. - Server defaults (host, port, auth, ACL, terminator) or HTTP routes change. +- The storage URI syntax, the `StorageBackend` contract, the built-in schemes or the resolver order change. - A §6 contract changes: PyBreeze invocation, the Windows double decode, the facade names TestPioneer uses. - A CLAUDE.md section referenced in §7 is renamed or its rule changes. - Refresh the "Last verified" line whenever this file is re-checked against HEAD. diff --git a/automation_file/__init__.py b/automation_file/__init__.py index 6a27f81..c7aa8e8 100644 --- a/automation_file/__init__.py +++ b/automation_file/__init__.py @@ -79,6 +79,16 @@ DataOpsException, DiffException, OneDriveException, + StorageAlreadyExistsException, + StorageException, + StorageNotEmptyException, + StorageNotFoundException, + StoragePathTypeException, + StoragePermissionException, + StorageTransientException, + StorageUnavailableException, + StorageUnsupportedException, + StorageURIException, TextOpsException, TracingException, ) @@ -255,6 +265,19 @@ start_autocontrol_socket_server, ) from automation_file.server.web_ui import WebUIServer, start_web_ui +from automation_file.storage import ( + Checksum, + File, + FileInfo, + LocalStorage, + MemoryStorage, + Storage, + StorageBackend, + StorageCapabilities, + StorageResolver, + StorageURI, + parse_storage_uri, +) from automation_file.trigger import ( FileWatcher, TriggerManager, @@ -452,6 +475,28 @@ def __getattr__(name: str) -> Any: "fsspec_mkdir", "fsspec_upload", "get_fs", + # Universal storage layer + "File", + "Storage", + "StorageBackend", + "StorageResolver", + "StorageURI", + "parse_storage_uri", + "FileInfo", + "Checksum", + "StorageCapabilities", + "LocalStorage", + "MemoryStorage", + "StorageException", + "StorageURIException", + "StorageNotFoundException", + "StorageAlreadyExistsException", + "StoragePathTypeException", + "StorageNotEmptyException", + "StoragePermissionException", + "StorageTransientException", + "StorageUnavailableException", + "StorageUnsupportedException", # Server / Project / Utils "TCPActionServer", "start_autocontrol_socket_server", diff --git a/automation_file/exceptions.py b/automation_file/exceptions.py index d541ad9..ee8d225 100644 --- a/automation_file/exceptions.py +++ b/automation_file/exceptions.py @@ -143,6 +143,46 @@ class TracingException(FileAutomationException): """Raised when OpenTelemetry tracing setup cannot be completed.""" +class StorageException(FileAutomationException): + """Root of the errors raised by the universal storage layer (``automation_file.storage``).""" + + +class StorageURIException(StorageException): + """Raised when a storage URI or path is malformed, ambiguous, or has no backend.""" + + +class StorageNotFoundException(StorageException, FileNotExistsException): + """Raised when a storage path, or the local source of an upload, does not exist.""" + + +class StorageAlreadyExistsException(StorageException): + """Raised when a write would replace a path and ``overwrite`` / ``exist_ok`` is off.""" + + +class StoragePathTypeException(StorageException): + """Raised when a file operation targets a directory, or a directory operation a file.""" + + +class StorageNotEmptyException(StorageException): + """Raised when a directory with entries is deleted without ``recursive=True``.""" + + +class StoragePermissionException(StorageException): + """Raised when the backend denies access to a path.""" + + +class StorageTransientException(StorageException): + """Raised for failures worth retrying: timeouts, dropped connections, throttling.""" + + +class StorageUnavailableException(StorageException): + """Raised when a backend is not initialised or its SDK is not installed.""" + + +class StorageUnsupportedException(StorageException): + """Raised when a backend cannot perform the requested operation.""" + + _ARGPARSE_EMPTY_MESSAGE = "argparse received no actionable argument" _BAD_TRIGGER_FUNCTION = "trigger name is not registered in the executor" _BAD_CALLBACK_METHOD = "callback_param_method must be 'kwargs' or 'args'" diff --git a/automation_file/storage/__init__.py b/automation_file/storage/__init__.py new file mode 100644 index 0000000..f6e228f --- /dev/null +++ b/automation_file/storage/__init__.py @@ -0,0 +1,56 @@ +"""Universal storage layer: one contract, one URI syntax, any backend. + +* :class:`File` and :class:`Storage` are the application API. +* :class:`StorageBackend` is the contract a backend implements; + :class:`LocalStorage` and :class:`MemoryStorage` are the built-in ones. +* :class:`StorageURI` / :func:`parse_storage_uri` define the address syntax, and + :class:`StorageResolver` maps an address to a backend. +""" + +from __future__ import annotations + +from automation_file.storage.backend import StorageBackend +from automation_file.storage.file import File +from automation_file.storage.local_storage import LocalStorage +from automation_file.storage.memory_storage import ( + MemoryStorage, + clear_memory_stores, + memory_store, +) +from automation_file.storage.resolver import ( + BackendFactory, + StorageResolver, + default_resolver, + register_default_schemes, +) +from automation_file.storage.storage import Storage +from automation_file.storage.types import Checksum, FileInfo, StorageCapabilities +from automation_file.storage.uri import ( + StorageURI, + URILike, + local_path_to_uri, + normalize_path, + parse_storage_uri, +) + +__all__ = [ + "BackendFactory", + "Checksum", + "File", + "FileInfo", + "LocalStorage", + "MemoryStorage", + "Storage", + "StorageBackend", + "StorageCapabilities", + "StorageResolver", + "StorageURI", + "URILike", + "clear_memory_stores", + "default_resolver", + "local_path_to_uri", + "memory_store", + "normalize_path", + "parse_storage_uri", + "register_default_schemes", +] diff --git a/automation_file/storage/backend.py b/automation_file/storage/backend.py new file mode 100644 index 0000000..2c05f65 --- /dev/null +++ b/automation_file/storage/backend.py @@ -0,0 +1,412 @@ +"""The contract every storage backend implements. + +:class:`StorageBackend` is a Template Method class. Its public methods -- +``exists``, ``stat``, ``list_dir``, ``mkdir``, ``upload``, ``download``, +``delete``, ``checksum``, ``read_bytes``, ``write_bytes``, ``copy_from`` and +``move_from`` -- are the same for every backend: they normalise the path, check +what is already there, raise the same :class:`StorageException` subclasses and +create missing parent directories. A backend only supplies the primitives named +with a leading underscore, so a new backend gets the shared behaviour for free +and the contract suite (``tests/storage_contract.py``) checks it the same way as +the built-in ones. + +Paths are relative to the backend's root, use ``/`` and never start with one. +The root itself is the empty string. +""" + +from __future__ import annotations + +import hashlib +import os +import tempfile +import uuid +from abc import ABC, abstractmethod +from collections.abc import Iterable +from pathlib import Path +from types import TracebackType +from typing import ClassVar, TypeVar + +from automation_file.core.checksum import file_checksum +from automation_file.exceptions import ( + StorageAlreadyExistsException, + StorageException, + StorageNotEmptyException, + StorageNotFoundException, + StoragePathTypeException, + StorageUnsupportedException, +) +from automation_file.storage.types import Checksum, FileInfo, StorageCapabilities +from automation_file.storage.uri import normalize_path + +DEFAULT_CHECKSUM_ALGORITHM = "sha256" +_STAGED_NAME = "staged" +# Digests of these need an explicit length, so they have no fixed hex form. +_VARIABLE_LENGTH_PREFIX = "shake" + +_BackendT = TypeVar("_BackendT", bound="StorageBackend") + + +def checked_algorithm(algorithm: str) -> str: + """Return the lower-case name of a fixed-length hash ``hashlib`` provides.""" + name = algorithm.strip().lower() + if name not in hashlib.algorithms_available or name.startswith(_VARIABLE_LENGTH_PREFIX): + raise StorageUnsupportedException(f"unsupported checksum algorithm: {algorithm!r}") + return name + + +def parent_of(path: str) -> str: + """Return the directory part of a normalised path (empty for a top-level entry).""" + return path.rpartition("/")[0] + + +def join_path(directory: str, name: str) -> str: + """Append ``name`` to a normalised directory path.""" + return f"{directory}/{name}" if directory else name + + +def missing_error(location: str) -> StorageNotFoundException: + return StorageNotFoundException(f"{location} does not exist") + + +def not_a_file_error(location: str) -> StoragePathTypeException: + return StoragePathTypeException(f"{location} is a directory, not a file") + + +def not_empty_error(location: str) -> StorageNotEmptyException: + return StorageNotEmptyException( + f"{location} is not empty; pass recursive=True to delete its contents" + ) + + +def _by_path(info: FileInfo) -> str: + return info.path + + +def _by_depth(info: FileInfo) -> int: + return info.path.count("/") + + +class StorageBackend(ABC): + """One storage root: a directory tree, a bucket, a share, a remote session.""" + + scheme: ClassVar[str] = "" + capabilities: ClassVar[StorageCapabilities] = StorageCapabilities() + + # ------------------------------------------------------------------ primitives + + @abstractmethod + def _stat(self, path: str) -> FileInfo | None: + """Return the entry at ``path`` with ``FileInfo.path == path``, or ``None`` if absent. + + The root (``""``) is a directory. + """ + + @abstractmethod + def _list_dir(self, path: str) -> Iterable[FileInfo]: + """Return the immediate children of the existing directory ``path``.""" + + @abstractmethod + def _upload(self, source: Path, path: str) -> None: + """Store the local file ``source`` at ``path``, replacing a file already there. + + The parent directory exists and ``path`` is not a directory. + """ + + @abstractmethod + def _download(self, path: str, target: Path) -> None: + """Write the existing file ``path`` to the local file ``target``.""" + + @abstractmethod + def _delete_file(self, path: str) -> None: + """Remove the existing file ``path``.""" + + def _mkdir(self, path: str) -> None: + """Create the directory ``path``; its parent exists. An existing directory is not an error. + + Called only when ``capabilities.directories`` is true. + """ + raise StorageUnsupportedException(f"{self.uri_for(path)}: this backend cannot mkdir") + + def _rmdir(self, path: str) -> None: + """Remove the empty directory ``path``. + + Called only when ``capabilities.directories`` is true. + """ + raise StorageUnsupportedException(f"{self.uri_for(path)}: this backend cannot rmdir") + + def _walk(self, path: str) -> Iterable[FileInfo]: + """Return every descendant of the directory ``path``, directories included. + + The default recurses through :meth:`_list_dir`. An object store overrides it + with one flat listing. + """ + found: list[FileInfo] = [] + pending = [path] + while pending: + for info in self._list_dir(pending.pop()): + found.append(info) + if info.is_dir: + pending.append(info.path) + return found + + def _copy_from(self, source: StorageBackend, source_path: str, path: str) -> bool: + """Copy a file from ``source`` without a local staging copy; ``False`` if unable.""" + return False + + def _move_from(self, source: StorageBackend, source_path: str, path: str) -> bool: + """Move a file from ``source`` natively (a rename); ``False`` if unable.""" + return False + + def _checksum(self, path: str, algorithm: str) -> str: + """Return the hex digest of the file ``path``. The default hashes a staged copy.""" + with tempfile.TemporaryDirectory() as scratch: + staged = Path(scratch) / _STAGED_NAME + self._download(path, staged) + return file_checksum(staged, algorithm) + + def _read_bytes(self, path: str) -> bytes: + """Return the content of the file ``path``. The default reads a staged copy.""" + with tempfile.TemporaryDirectory() as scratch: + staged = Path(scratch) / _STAGED_NAME + self._download(path, staged) + return staged.read_bytes() + + def _delete_directory(self, path: str, recursive: bool) -> None: + """Remove the directory ``path``, which is not the root.""" + if not recursive: + if any(True for _ in self._list_dir(path)): + raise not_empty_error(self.uri_for(path)) + else: + for info in sorted(self._walk(path), key=_by_depth, reverse=True): + if not info.is_dir: + self._delete_file(info.path) + elif self.capabilities.directories: + self._rmdir(info.path) + if self.capabilities.directories: + self._rmdir(path) + + def _normalize(self, path: str) -> str: + """Normalise a caller-supplied path. Backends with extra separators extend it.""" + return normalize_path(path) + + def _is_root(self, path: str) -> bool: + """Say whether ``path`` is a root that :meth:`delete` must refuse to remove.""" + return not path + + # ------------------------------------------------------------------ public API + + def uri_for(self, path: str = "") -> str: + """Return the storage URI of ``path`` in this backend, for messages and logs.""" + return f"{self.scheme}:///{self._normalize(path)}" + + def exists(self, path: str) -> bool: + """Return True when a file or a directory is at ``path``.""" + return self._stat(self._normalize(path)) is not None + + def stat(self, path: str) -> FileInfo: + """Return the :class:`FileInfo` of ``path``; raise ``StorageNotFoundException`` if absent.""" + clean = self._normalize(path) + info = self._stat(clean) + if info is None: + raise missing_error(self.uri_for(clean)) + return info + + def list_dir(self, path: str = "", *, recursive: bool = False) -> list[FileInfo]: + """Return the entries under the directory ``path``, sorted by path. + + ``recursive=True`` returns every descendant, directories included. + """ + clean = self._normalize(path) + if not self.stat(clean).is_dir: + raise StoragePathTypeException(f"{self.uri_for(clean)} is not a directory") + entries = self._walk(clean) if recursive else self._list_dir(clean) + return sorted(entries, key=_by_path) + + def mkdir(self, path: str, *, parents: bool = True, exist_ok: bool = True) -> None: + """Create the directory ``path``. + + Where directories are only implied by file paths (``capabilities.directories`` + is false) nothing is created and nothing needs to be. + """ + clean = self._normalize(path) + info = self._stat(clean) + if info is not None: + if not info.is_dir: + raise StoragePathTypeException(f"{self.uri_for(clean)} is a file") + if not exist_ok: + raise StorageAlreadyExistsException(f"{self.uri_for(clean)} already exists") + return + if not self.capabilities.directories: + return + self._make_parents(clean, create=parents) + self._mkdir(clean) + + def upload( + self, local_path: str | os.PathLike[str], path: str, *, overwrite: bool = True + ) -> FileInfo: + """Store the local file ``local_path`` at ``path`` and return its ``FileInfo``. + + Missing parent directories are created. + """ + source = Path(local_path) + if not source.is_file(): + raise StorageNotFoundException(f"local source is not a file: {source}") + clean = self._writable_file(path, overwrite) + self._make_parents(clean) + self._upload(source, clean) + return self.stat(clean) + + def download( + self, path: str, local_path: str | os.PathLike[str], *, overwrite: bool = True + ) -> Path: + """Write the file ``path`` to ``local_path`` and return that path. + + The content lands in a sibling ``.part`` file that replaces the target once + complete, so a failed download never leaves a truncated target behind. + """ + clean = self._existing_file(path) + target = Path(local_path) + if target.is_dir(): + raise StoragePathTypeException(f"local target is a directory: {target}") + if not overwrite and target.exists(): + raise StorageAlreadyExistsException(f"local target already exists: {target}") + target.parent.mkdir(parents=True, exist_ok=True) + partial = target.with_name(f".{target.name}.{uuid.uuid4().hex}.part") + try: + self._download(clean, partial) + os.replace(partial, target) + finally: + partial.unlink(missing_ok=True) + return target + + def delete(self, path: str, *, recursive: bool = False, missing_ok: bool = False) -> None: + """Remove the file or directory at ``path``. + + A directory with entries needs ``recursive=True``. The storage root is never + removed. + """ + clean = self._normalize(path) + info = self._stat(clean) + if info is None: + if missing_ok: + return + raise missing_error(self.uri_for(clean)) + if not info.is_dir: + self._delete_file(clean) + return + if self._is_root(clean): + raise StorageUnsupportedException( + f"refusing to delete the storage root {self.uri_for(clean)}" + ) + self._delete_directory(clean, recursive) + + def checksum(self, path: str, algorithm: str = DEFAULT_CHECKSUM_ALGORITHM) -> Checksum: + """Return the :class:`Checksum` of the file ``path`` (SHA-256 by default).""" + name = checked_algorithm(algorithm) + return Checksum(name, self._checksum(self._existing_file(path), name)) + + def read_bytes(self, path: str) -> bytes: + """Return the whole content of the file ``path``.""" + return self._read_bytes(self._existing_file(path)) + + def write_bytes(self, path: str, data: bytes, *, overwrite: bool = True) -> FileInfo: + """Store ``data`` as the file ``path`` and return its ``FileInfo``.""" + with tempfile.TemporaryDirectory() as scratch: + staged = Path(scratch) / _STAGED_NAME + staged.write_bytes(data) + return self.upload(staged, path, overwrite=overwrite) + + def copy_from( + self, source: StorageBackend, source_path: str, path: str, *, overwrite: bool = True + ) -> FileInfo: + """Copy the file ``source_path`` of ``source`` (which may be this backend) to ``path``. + + The copy is native when the two backends can do it between themselves and + goes through a local staging file otherwise. + """ + origin, target = self._transfer_paths(source, source_path, path, overwrite) + self._pull(source, origin, target) + return self.stat(target) + + def move_from( + self, source: StorageBackend, source_path: str, path: str, *, overwrite: bool = True + ) -> FileInfo: + """Move the file ``source_path`` of ``source`` to ``path``: a rename, or copy then delete.""" + origin, target = self._transfer_paths(source, source_path, path, overwrite) + if not self._move_from(source, origin, target): + self._pull(source, origin, target) + source.delete(origin) + return self.stat(target) + + def close(self) -> None: # noqa: B027 - optional hook: most backends hold nothing open + """Release what the backend holds open. The default holds nothing.""" + + def __enter__(self: _BackendT) -> _BackendT: + return self + + def __exit__( + self, + exc_type: type[BaseException] | None, + exc: BaseException | None, + tb: TracebackType | None, + ) -> None: + self.close() + + # ------------------------------------------------------------------ shared steps + + def _existing_file(self, path: str) -> str: + clean = self._normalize(path) + if self.stat(clean).is_dir: + raise not_a_file_error(self.uri_for(clean)) + return clean + + def _writable_file(self, path: str, overwrite: bool) -> str: + clean = self._normalize(path) + if not clean: + raise StoragePathTypeException(f"{self.uri_for(clean)} is the storage root, not a file") + existing = self._stat(clean) + if existing is None: + return clean + if existing.is_dir: + raise not_a_file_error(self.uri_for(clean)) + if not overwrite: + raise StorageAlreadyExistsException(f"{self.uri_for(clean)} already exists") + return clean + + def _make_parents(self, path: str, *, create: bool = True) -> None: + if not self.capabilities.directories: + return + missing: list[str] = [] + parent = parent_of(path) + while parent: + info = self._stat(parent) + if info is not None: + if not info.is_dir: + raise StoragePathTypeException(f"{self.uri_for(parent)} is a file") + break + missing.append(parent) + parent = parent_of(parent) + if missing and not create: + raise missing_error(self.uri_for(missing[0])) + for directory in reversed(missing): + self._mkdir(directory) + + def _transfer_paths( + self, source: StorageBackend, source_path: str, path: str, overwrite: bool + ) -> tuple[str, str]: + info = source.stat(source_path) + if info.is_dir: + raise not_a_file_error(source.uri_for(info.path)) + target = self._normalize(path) + if source is self and info.path == target: + raise StorageException(f"{self.uri_for(target)}: source and target are the same file") + target = self._writable_file(target, overwrite) + self._make_parents(target) + return info.path, target + + def _pull(self, source: StorageBackend, origin: str, target: str) -> None: + if self._copy_from(source, origin, target): + return + with tempfile.TemporaryDirectory() as scratch: + staged = source.download(origin, Path(scratch) / _STAGED_NAME) + self._upload(staged, target) diff --git a/automation_file/storage/file.py b/automation_file/storage/file.py new file mode 100644 index 0000000..fbd3c50 --- /dev/null +++ b/automation_file/storage/file.py @@ -0,0 +1,195 @@ +"""``File``: one file in any storage backend, addressed by URI. + +.. code-block:: python + + from automation_file import File + + report = File("s3://reports/2026/q1.csv") + report.exists() + report.size + data = report.read() + report.copy_to("sftp://nas/archive/q1.csv") + report.checksum().matches("sha256:9f86d0...") + +The backend is looked up on every call, so a ``File`` can be created before its +backend is initialised or mounted. +""" + +from __future__ import annotations + +import os +from collections.abc import Mapping +from dataclasses import replace +from datetime import datetime +from pathlib import Path + +from automation_file.exceptions import StorageNotFoundException, StoragePathTypeException +from automation_file.storage.backend import DEFAULT_CHECKSUM_ALGORITHM, StorageBackend +from automation_file.storage.resolver import StorageResolver, default_resolver +from automation_file.storage.types import Checksum, FileInfo +from automation_file.storage.uri import StorageURI, URILike, parse_storage_uri + +_DEFAULT_ENCODING = "utf-8" + + +def _expected_algorithm(expected: str | Checksum, fallback: str) -> str: + if isinstance(expected, Checksum): + return expected.algorithm + if ":" in expected: + return Checksum.parse(expected).algorithm + return fallback + + +class File: + """A file somewhere in storage. Creating one touches nothing.""" + + def __init__(self, uri: URILike, *, resolver: StorageResolver | None = None) -> None: + self._uri = parse_storage_uri(uri) + self._resolver = resolver if resolver is not None else default_resolver + + @property + def uri(self) -> StorageURI: + return self._uri + + @property + def name(self) -> str: + return self._uri.name + + def exists(self) -> bool: + """Return True when anything -- a file or a directory -- is at this URI.""" + backend, path = self._locate() + return backend.exists(path) + + def is_file(self) -> bool: + return self._type_is(directory=False) + + def is_dir(self) -> bool: + return self._type_is(directory=True) + + def stat(self) -> FileInfo: + """Return this file's :class:`FileInfo`; its ``path`` is the path of the URI.""" + backend, path = self._locate() + return replace(backend.stat(path), path=self._uri.path) + + @property + def size(self) -> int | None: + return self.stat().size + + @property + def modified_at(self) -> datetime | None: + return self.stat().modified_at + + @property + def etag(self) -> str | None: + return self.stat().etag + + @property + def version(self) -> str | None: + return self.stat().version + + @property + def content_type(self) -> str | None: + return self.stat().content_type + + @property + def metadata(self) -> Mapping[str, str]: + return self.stat().metadata + + def read(self) -> bytes: + """Return the whole content.""" + backend, path = self._locate() + return backend.read_bytes(path) + + def read_text(self, encoding: str = _DEFAULT_ENCODING) -> str: + return self.read().decode(encoding) + + def write( + self, data: bytes | str, *, overwrite: bool = True, encoding: str = _DEFAULT_ENCODING + ) -> FileInfo: + """Store ``data`` as the content; text is encoded with ``encoding``.""" + payload = data.encode(encoding) if isinstance(data, str) else data + backend, path = self._locate() + return replace(backend.write_bytes(path, payload, overwrite=overwrite), path=self._uri.path) + + def upload_from( + self, local_path: str | os.PathLike[str], *, overwrite: bool = True + ) -> FileInfo: + """Store the local file ``local_path`` as the content.""" + backend, path = self._locate() + return replace(backend.upload(local_path, path, overwrite=overwrite), path=self._uri.path) + + def download_to(self, local_path: str | os.PathLike[str], *, overwrite: bool = True) -> Path: + """Write the content to the local file ``local_path`` and return that path.""" + backend, path = self._locate() + return backend.download(path, local_path, overwrite=overwrite) + + def copy_to(self, target: URILike | File, *, overwrite: bool = True) -> File: + """Copy this file to ``target`` in any backend and return the new ``File``.""" + destination = self._as_file(target) + source_backend, source_path = self._locate() + target_backend, target_path = destination._locate() + target_backend.copy_from(source_backend, source_path, target_path, overwrite=overwrite) + return destination + + def move_to(self, target: URILike | File, *, overwrite: bool = True) -> File: + """Move this file to ``target`` in any backend and return the new ``File``.""" + destination = self._as_file(target) + source_backend, source_path = self._locate() + target_backend, target_path = destination._locate() + target_backend.move_from(source_backend, source_path, target_path, overwrite=overwrite) + return destination + + def delete(self, *, missing_ok: bool = False) -> None: + """Remove this file. A directory is refused: delete it through ``Storage``.""" + backend, path = self._locate() + try: + info = backend.stat(path) + except StorageNotFoundException: + if missing_ok: + return + raise + if info.is_dir: + raise StoragePathTypeException( + f"{self._uri} is a directory; remove it with Storage(...).delete(recursive=True)" + ) + backend.delete(path) + + def checksum(self, algorithm: str = DEFAULT_CHECKSUM_ALGORITHM) -> Checksum: + """Return the :class:`Checksum` of the content (SHA-256 by default).""" + backend, path = self._locate() + return backend.checksum(path, algorithm) + + def verify( + self, expected: str | Checksum, *, algorithm: str = DEFAULT_CHECKSUM_ALGORITHM + ) -> bool: + """Return whether the content has the digest ``expected``. + + ``expected`` is a ``Checksum``, ``"algorithm:digest"`` or a bare digest of + ``algorithm``. + """ + return self.checksum(_expected_algorithm(expected, algorithm)).matches(expected) + + def _locate(self) -> tuple[StorageBackend, str]: + return self._resolver.resolve(self._uri) + + def _as_file(self, target: URILike | File) -> File: + return target if isinstance(target, File) else File(target, resolver=self._resolver) + + def _type_is(self, *, directory: bool) -> bool: + backend, path = self._locate() + try: + return backend.stat(path).is_dir is directory + except StorageNotFoundException: + return False + + def __eq__(self, other: object) -> bool: + return isinstance(other, File) and other._uri == self._uri + + def __hash__(self) -> int: + return hash(self._uri) + + def __str__(self) -> str: + return str(self._uri) + + def __repr__(self) -> str: + return f"File({str(self._uri)!r})" diff --git a/automation_file/storage/local_storage.py b/automation_file/storage/local_storage.py new file mode 100644 index 0000000..927f8af --- /dev/null +++ b/automation_file/storage/local_storage.py @@ -0,0 +1,252 @@ +"""Local filesystem backend. + +``LocalStorage()`` spans the whole filesystem: its paths are absolute paths +without the leading slash (``data/report.csv``, or ``C:/data/report.csv`` on +Windows), which is what ``local:///data/report.csv`` resolves to. + +``LocalStorage(root)`` is confined to one directory tree. Every path goes through +:func:`~automation_file.local.safe_paths.safe_join`, so a path that leaves the +root -- through a symlink or an absolute Windows path -- raises +:class:`~automation_file.exceptions.PathTraversalException`. + +Symbolic links are followed when reading and writing. Deleting never follows +them: the link is removed and its target is left alone. +""" + +from __future__ import annotations + +import contextlib +import errno +import mimetypes +import os +import re +import shutil +import stat +import uuid +from collections.abc import Iterable, Iterator +from datetime import datetime, timezone +from pathlib import Path, PurePosixPath + +from automation_file.core.checksum import file_checksum +from automation_file.exceptions import StorageException, StoragePermissionException +from automation_file.local.safe_paths import safe_join +from automation_file.storage.backend import ( + StorageBackend, + join_path, + missing_error, + not_empty_error, +) +from automation_file.storage.types import FileInfo, StorageCapabilities +from automation_file.storage.uri import LOCAL_SCHEME, StorageURI, local_path_to_uri, normalize_path + +_WINDOWS = os.sep == "\\" +_DRIVE = re.compile(r"[A-Za-z]:") + + +@contextlib.contextmanager +def _os_errors(location: str) -> Iterator[None]: + """Turn the ``OSError`` family into the storage layer's exceptions.""" + try: + yield + except FileNotFoundError as error: + raise missing_error(location) from error + except PermissionError as error: + raise StoragePermissionException(f"access to {location} was denied") from error + except OSError as error: + raise StorageException(f"{location}: {error}") from error + + +def _anchored(path: str) -> Path: + """Return the absolute filesystem path a rootless storage path stands for.""" + if _WINDOWS: + drive, _, rest = path.partition("/") + if _DRIVE.fullmatch(drive): + return Path(f"{drive}/{rest}") + return Path(f"/{path}") + + +def _content_type(name: str) -> str | None: + # Only the suffixes are passed on: guess_type() parses its argument as a URL. + return mimetypes.guess_type("_" + "".join(PurePosixPath(name).suffixes))[0] + + +def _file_info(path: str, result: os.stat_result) -> FileInfo: + is_dir = stat.S_ISDIR(result.st_mode) + return FileInfo( + path=path, + is_dir=is_dir, + size=None if is_dir else result.st_size, + modified_at=datetime.fromtimestamp(result.st_mtime, tz=timezone.utc), + content_type=None if is_dir else _content_type(path), + ) + + +def _stat_entry(entry: Path) -> os.stat_result: + """Stat ``entry``, falling back to the link itself when its target is gone.""" + try: + return entry.stat() + except FileNotFoundError: + return entry.lstat() + + +def _remove_link(link: Path) -> None: + try: + link.unlink() + except (PermissionError, IsADirectoryError): + # A Windows directory link is removed like a directory. + os.rmdir(link) + + +def _replace_with_copy(source: Path, target: Path) -> None: + """Copy ``source`` over ``target`` through a sibling file, so readers never see half of it.""" + partial = target.with_name(f".{target.name}.{uuid.uuid4().hex}.part") + try: + shutil.copyfile(source, partial) + os.replace(partial, target) + finally: + partial.unlink(missing_ok=True) + + +def _raise(error: OSError) -> None: + raise error + + +class LocalStorage(StorageBackend): + """The local filesystem, whole (``root=None``) or confined to ``root``.""" + + scheme = LOCAL_SCHEME + capabilities = StorageCapabilities(directories=True, modified_at=True, content_type=True) + + def __init__(self, root: str | os.PathLike[str] | None = None) -> None: + self._root: Path | None = Path(root).resolve() if root is not None else None + + @property + def root(self) -> Path | None: + """The directory this backend is confined to, or ``None`` for the whole filesystem.""" + return self._root + + def local_path(self, path: str = "") -> Path: + """Return the filesystem path of ``path``, checked against the root when there is one.""" + clean = self._normalize(path) + if self._root is None: + return _anchored(clean) + return safe_join(self._root, clean) if clean else self._root + + def uri_for(self, path: str = "") -> str: + clean = self._normalize(path) + if self._root is None: + return str(StorageURI(LOCAL_SCHEME, "", clean)) + return str(local_path_to_uri(self._root / clean)) + + def __repr__(self) -> str: + return "LocalStorage()" if self._root is None else f"LocalStorage({str(self._root)!r})" + + def _normalize(self, path: str) -> str: + return normalize_path(path.replace("\\", "/") if _WINDOWS else path) + + def _entry_path(self, path: str) -> Path: + """Like :meth:`local_path`, but the last segment is not resolved: a link stays a link.""" + parent, _, name = path.rpartition("/") + return self.local_path(parent) / name if name else self.local_path(parent) + + def _is_root(self, path: str) -> bool: + if not path: + return True + target = self.local_path(path) + return target == target.parent + + def _stat(self, path: str) -> FileInfo | None: + with _os_errors(self.uri_for(path)): + try: + return _file_info(path, self.local_path(path).stat()) + except (FileNotFoundError, NotADirectoryError): + return self._dangling_link(path) + + def _dangling_link(self, path: str) -> FileInfo | None: + """Report a link whose target is gone as a file, so it can still be deleted.""" + try: + entry = self._entry_path(path) + if not entry.is_symlink(): + return None + return _file_info(path, entry.lstat()) + except (FileNotFoundError, NotADirectoryError): + return None + + def _list_dir(self, path: str) -> Iterable[FileInfo]: + directory = self.local_path(path) + with _os_errors(self.uri_for(path)): + return [ + _file_info(join_path(path, entry.name), _stat_entry(entry)) + for entry in directory.iterdir() + ] + + def _walk(self, path: str) -> Iterable[FileInfo]: + top = self.local_path(path) + found: list[FileInfo] = [] + with _os_errors(self.uri_for(path)): + # os.walk lists linked directories but does not descend into them. + for current, directories, files in os.walk(top, onerror=_raise): + here = Path(current) + relative = here.relative_to(top).as_posix() + base = path if relative == "." else join_path(path, relative) + found.extend( + _file_info(join_path(base, name), _stat_entry(here / name)) + for name in (*directories, *files) + ) + return found + + def _upload(self, source: Path, path: str) -> None: + with _os_errors(self.uri_for(path)): + _replace_with_copy(source, self.local_path(path)) + + def _download(self, path: str, target: Path) -> None: + with _os_errors(self.uri_for(path)): + shutil.copyfile(self.local_path(path), target) + + def _delete_file(self, path: str) -> None: + with _os_errors(self.uri_for(path)): + self._entry_path(path).unlink() + + def _mkdir(self, path: str) -> None: + with _os_errors(self.uri_for(path)): + self.local_path(path).mkdir(parents=True, exist_ok=True) + + def _delete_directory(self, path: str, recursive: bool) -> None: + entry = self._entry_path(path) + with _os_errors(self.uri_for(path)): + if entry.is_symlink(): + _remove_link(entry) + elif recursive: + shutil.rmtree(entry) + elif any(entry.iterdir()): + raise not_empty_error(self.uri_for(path)) + else: + entry.rmdir() + + def _copy_from(self, source: StorageBackend, source_path: str, path: str) -> bool: + if not isinstance(source, LocalStorage): + return False + with _os_errors(self.uri_for(path)): + _replace_with_copy(source.local_path(source_path), self.local_path(path)) + return True + + def _move_from(self, source: StorageBackend, source_path: str, path: str) -> bool: + if not isinstance(source, LocalStorage): + return False + origin = source._entry_path(source_path) + with _os_errors(self.uri_for(path)): + try: + os.replace(origin, self.local_path(path)) + except OSError as error: + if error.errno == errno.EXDEV: + return False + raise + return True + + def _checksum(self, path: str, algorithm: str) -> str: + with _os_errors(self.uri_for(path)): + return file_checksum(self.local_path(path), algorithm) + + def _read_bytes(self, path: str) -> bytes: + with _os_errors(self.uri_for(path)): + return self.local_path(path).read_bytes() diff --git a/automation_file/storage/memory_storage.py b/automation_file/storage/memory_storage.py new file mode 100644 index 0000000..7dc7d55 --- /dev/null +++ b/automation_file/storage/memory_storage.py @@ -0,0 +1,108 @@ +"""In-memory backend: a storage that lives in the process and needs no setup. + +Meant for tests, dry runs and examples. ``memory:///`` addresses the +store called ````, created on first use and kept until the process exits or +:func:`clear_memory_stores` runs. +""" + +from __future__ import annotations + +import threading +from collections.abc import Iterable +from dataclasses import dataclass +from datetime import datetime, timezone +from pathlib import Path + +from automation_file.storage.backend import StorageBackend, parent_of +from automation_file.storage.types import FileInfo, StorageCapabilities + +MEMORY_SCHEME = "memory" + + +@dataclass(frozen=True) +class _Blob: + data: bytes + modified_at: datetime + + +class MemoryStorage(StorageBackend): + """A thread-safe tree of files and directories held in memory.""" + + scheme = MEMORY_SCHEME + capabilities = StorageCapabilities(directories=True, modified_at=True) + + def __init__(self, name: str = "") -> None: + self._name = name + self._lock = threading.RLock() + self._files: dict[str, _Blob] = {} + self._directories: set[str] = set() + + @property + def name(self) -> str: + """The store name, which is the authority of its ``memory://`` URIs.""" + return self._name + + def uri_for(self, path: str = "") -> str: + clean = self._normalize(path) + root = f"{MEMORY_SCHEME}://{self._name}" + return f"{root}/{clean}" if clean or not self._name else root + + def __repr__(self) -> str: + return f"MemoryStorage({self._name!r})" + + def _stat(self, path: str) -> FileInfo | None: + with self._lock: + blob = self._files.get(path) + if blob is not None: + return FileInfo(path=path, size=len(blob.data), modified_at=blob.modified_at) + if not path or path in self._directories: + return FileInfo(path=path, is_dir=True) + return None + + def _list_dir(self, path: str) -> Iterable[FileInfo]: + with self._lock: + names = [name for name in (*self._directories, *self._files) if parent_of(name) == path] + return [info for info in map(self._stat, names) if info is not None] + + def _upload(self, source: Path, path: str) -> None: + blob = _Blob(source.read_bytes(), datetime.now(timezone.utc)) + with self._lock: + self._files[path] = blob + + def _download(self, path: str, target: Path) -> None: + target.write_bytes(self._read_bytes(path)) + + def _read_bytes(self, path: str) -> bytes: + with self._lock: + return self._files[path].data + + def _delete_file(self, path: str) -> None: + with self._lock: + del self._files[path] + + def _mkdir(self, path: str) -> None: + with self._lock: + self._directories.add(path) + + def _rmdir(self, path: str) -> None: + with self._lock: + self._directories.discard(path) + + +_stores: dict[str, MemoryStorage] = {} +_stores_lock = threading.Lock() + + +def memory_store(name: str = "") -> MemoryStorage: + """Return the shared in-memory store called ``name``, creating it on first use.""" + with _stores_lock: + store = _stores.get(name) + if store is None: + store = _stores[name] = MemoryStorage(name) + return store + + +def clear_memory_stores() -> None: + """Forget every shared in-memory store and what it holds.""" + with _stores_lock: + _stores.clear() diff --git a/automation_file/storage/resolver.py b/automation_file/storage/resolver.py new file mode 100644 index 0000000..3a2ee53 --- /dev/null +++ b/automation_file/storage/resolver.py @@ -0,0 +1,152 @@ +"""Resolve a storage URI to the backend that serves it. + +A :class:`StorageResolver` answers "which backend, and which path inside it?" in +two steps: + +1. **Mounts.** ``mount("sftp://nas/archive", backend)`` binds one backend + instance to a URI. A URI at or below a mount goes to that backend, with the + mount's own path stripped. The longest matching mount wins. +2. **Scheme factories.** ``register_scheme("local", factory)`` handles every URI + of a scheme no mount claimed. The factory receives the parsed URI and returns + ``(backend, path)``. + +:data:`default_resolver` is the process-wide instance behind +:class:`~automation_file.storage.File` and :class:`~automation_file.storage.Storage`. +""" + +from __future__ import annotations + +import os +import threading +from collections.abc import Callable + +from automation_file.exceptions import StorageURIException +from automation_file.storage.backend import StorageBackend +from automation_file.storage.local_storage import LocalStorage +from automation_file.storage.memory_storage import MEMORY_SCHEME, memory_store +from automation_file.storage.types import StorageCapabilities +from automation_file.storage.uri import ( + LOCAL_SCHEME, + StorageURI, + URILike, + canonical_scheme, + parse_storage_uri, +) + +BackendFactory = Callable[[StorageURI], tuple[StorageBackend, str]] +_MountKey = tuple[str, str, str] + + +def _mount_key(uri: StorageURI) -> _MountKey: + return uri.scheme, uri.authority.casefold(), uri.path + + +def _below(path: str, prefix: str) -> str | None: + """Return ``path`` relative to ``prefix``, or ``None`` when it is not at or below it.""" + if not prefix: + return path + if path == prefix: + return "" + if path.startswith(f"{prefix}/"): + return path[len(prefix) + 1 :] + return None + + +class StorageResolver: + """A thread-safe table of mounts and scheme factories.""" + + def __init__(self, *, defaults: bool = True) -> None: + self._lock = threading.RLock() + self._factories: dict[str, BackendFactory] = {} + self._mounts: dict[_MountKey, StorageBackend] = {} + if defaults: + register_default_schemes(self) + + def register_scheme(self, scheme: str, factory: BackendFactory) -> None: + """Serve every unmounted URI of ``scheme`` through ``factory``.""" + if not callable(factory): + raise TypeError(f"storage factory for {scheme!r} is not callable") + with self._lock: + self._factories[canonical_scheme(scheme)] = factory + + def unregister_scheme(self, scheme: str) -> bool: + """Drop the factory of ``scheme``; return whether there was one.""" + with self._lock: + return self._factories.pop(canonical_scheme(scheme), None) is not None + + def mount(self, uri: URILike, backend: StorageBackend) -> None: + """Serve ``uri`` and everything below it from ``backend``.""" + if not isinstance(backend, StorageBackend): + raise TypeError(f"cannot mount {type(backend).__name__}: not a StorageBackend") + with self._lock: + self._mounts[_mount_key(parse_storage_uri(uri))] = backend + + def unmount(self, uri: URILike) -> bool: + """Remove the mount at exactly ``uri``; return whether there was one.""" + with self._lock: + return self._mounts.pop(_mount_key(parse_storage_uri(uri)), None) is not None + + def schemes(self) -> list[str]: + """Return every scheme that has a factory or a mount, sorted.""" + with self._lock: + return sorted({*self._factories, *(key[0] for key in self._mounts)}) + + def resolve(self, uri: URILike) -> tuple[StorageBackend, str]: + """Return the backend that serves ``uri`` and the path inside that backend.""" + parsed = parse_storage_uri(uri) + with self._lock: + mounted = self._find_mount(parsed) + factory = self._factories.get(parsed.scheme) + if mounted is not None: + return mounted + if factory is None: + known = ", ".join(self.schemes()) or "none" + raise StorageURIException( + f"no storage backend serves {str(parsed)!r}: scheme {parsed.scheme!r} has no " + f"factory and no mount covers the URI (known schemes: {known})" + ) + return factory(parsed) + + def capabilities(self, uri: URILike) -> StorageCapabilities: + """Return the capabilities of the backend that serves ``uri``.""" + return self.resolve(uri)[0].capabilities + + def _find_mount(self, uri: StorageURI) -> tuple[StorageBackend, str] | None: + authority = uri.authority.casefold() + best: tuple[int, StorageBackend, str] | None = None + for (scheme, mounted_authority, prefix), backend in self._mounts.items(): + if scheme != uri.scheme or mounted_authority != authority: + continue + relative = _below(uri.path, prefix) + if relative is not None and (best is None or len(prefix) > best[0]): + best = (len(prefix), backend, relative) + return None if best is None else (best[1], best[2]) + + +_whole_filesystem = LocalStorage() + + +def _local_factory(uri: StorageURI) -> tuple[StorageBackend, str]: + if not uri.authority: + return _whole_filesystem, uri.path + share, _, rest = uri.path.partition("/") + if os.sep != "\\" or not share: + raise StorageURIException( + f"{str(uri)!r} names the host {uri.authority!r}; a local file takes an empty " + f"authority, as in 'local:///{uri.authority}/{uri.path}'. Only Windows reads " + "'local://server/share/path' as a UNC path" + ) + return LocalStorage(f"//{uri.authority}/{share}/"), rest + + +def _memory_factory(uri: StorageURI) -> tuple[StorageBackend, str]: + return memory_store(uri.authority), uri.path + + +def register_default_schemes(resolver: StorageResolver) -> None: + """Register the factory of every built-in backend on ``resolver``.""" + resolver.register_scheme(LOCAL_SCHEME, _local_factory) + resolver.register_scheme(MEMORY_SCHEME, _memory_factory) + + +default_resolver = StorageResolver() diff --git a/automation_file/storage/storage.py b/automation_file/storage/storage.py new file mode 100644 index 0000000..37f1921 --- /dev/null +++ b/automation_file/storage/storage.py @@ -0,0 +1,151 @@ +"""``Storage``: a directory in any backend, and the table of backends itself. + +.. code-block:: python + + from automation_file import Storage + + reports = Storage("s3://reports/2026") + for info in reports.list_dir(recursive=True): + print(info.path, info.size) + reports.upload("q1.csv", "q1.csv") + reports.file("q1.csv").copy_to("local:///backup/q1.csv") + +Paths given to an instance, and the ``FileInfo.path`` values it returns, are +relative to the URI the instance was created with. + +The static methods manage the process-wide table: ``Storage.mount`` binds a +backend instance to a URI, ``Storage.register_scheme`` installs a factory for a +whole scheme, ``Storage.resolve`` shows where a URI leads. +""" + +from __future__ import annotations + +import os +from dataclasses import replace +from pathlib import Path + +from automation_file.storage.backend import DEFAULT_CHECKSUM_ALGORITHM, StorageBackend +from automation_file.storage.file import File +from automation_file.storage.resolver import BackendFactory, StorageResolver, default_resolver +from automation_file.storage.types import Checksum, FileInfo, StorageCapabilities +from automation_file.storage.uri import StorageURI, URILike, normalize_path, parse_storage_uri + + +def _rebased(info: FileInfo, backend_base: str, asked: str) -> FileInfo: + """Swap the backend's own prefix of ``info.path`` for the path the caller asked with.""" + tail = info.path[len(backend_base) :].lstrip("/") + visible_base = normalize_path(asked) + return replace( + info, path=f"{visible_base}/{tail}" if visible_base and tail else visible_base or tail + ) + + +class Storage: + """A directory somewhere in storage. Creating one touches nothing.""" + + def __init__(self, uri: URILike, *, resolver: StorageResolver | None = None) -> None: + self._uri = parse_storage_uri(uri) + self._resolver = resolver if resolver is not None else default_resolver + + # ------------------------------------------------------------------ the backend table + + @staticmethod + def mount(uri: URILike, backend: StorageBackend) -> None: + """Serve ``uri`` and everything below it from ``backend``.""" + default_resolver.mount(uri, backend) + + @staticmethod + def unmount(uri: URILike) -> bool: + """Remove the mount at exactly ``uri``; return whether there was one.""" + return default_resolver.unmount(uri) + + @staticmethod + def register_scheme(scheme: str, factory: BackendFactory) -> None: + """Serve every unmounted URI of ``scheme`` through ``factory``.""" + default_resolver.register_scheme(scheme, factory) + + @staticmethod + def schemes() -> list[str]: + """Return every scheme that has a factory or a mount.""" + return default_resolver.schemes() + + @staticmethod + def resolve(uri: URILike) -> tuple[StorageBackend, str]: + """Return the backend that serves ``uri`` and the path inside that backend.""" + return default_resolver.resolve(uri) + + # ------------------------------------------------------------------ one directory + + @property + def uri(self) -> StorageURI: + return self._uri + + @property + def backend(self) -> StorageBackend: + """The backend that serves this storage right now.""" + return self._resolver.resolve(self._uri)[0] + + @property + def capabilities(self) -> StorageCapabilities: + return self.backend.capabilities + + def file(self, path: str) -> File: + """Return the :class:`File` at ``path`` below this storage.""" + return File(self._uri.joinpath(path), resolver=self._resolver) + + def exists(self, path: str = "") -> bool: + backend, target = self._locate(path) + return backend.exists(target) + + def stat(self, path: str = "") -> FileInfo: + backend, target = self._locate(path) + return _rebased(backend.stat(target), target, path) + + def list_dir(self, path: str = "", *, recursive: bool = False) -> list[FileInfo]: + """Return the entries under ``path``, sorted; ``recursive`` adds every descendant.""" + backend, target = self._locate(path) + return [ + _rebased(info, target, path) for info in backend.list_dir(target, recursive=recursive) + ] + + def mkdir(self, path: str = "", *, parents: bool = True, exist_ok: bool = True) -> None: + backend, target = self._locate(path) + backend.mkdir(target, parents=parents, exist_ok=exist_ok) + + def upload( + self, local_path: str | os.PathLike[str], path: str, *, overwrite: bool = True + ) -> FileInfo: + """Store the local file ``local_path`` at ``path``.""" + backend, target = self._locate(path) + return _rebased(backend.upload(local_path, target, overwrite=overwrite), target, path) + + def download( + self, path: str, local_path: str | os.PathLike[str], *, overwrite: bool = True + ) -> Path: + """Write the file ``path`` to the local file ``local_path``.""" + backend, target = self._locate(path) + return backend.download(target, local_path, overwrite=overwrite) + + def delete(self, path: str, *, recursive: bool = False, missing_ok: bool = False) -> None: + """Remove the file or directory ``path``; ``""`` is this storage's own directory.""" + backend, target = self._locate(path) + backend.delete(target, recursive=recursive, missing_ok=missing_ok) + + def checksum(self, path: str, algorithm: str = DEFAULT_CHECKSUM_ALGORITHM) -> Checksum: + backend, target = self._locate(path) + return backend.checksum(target, algorithm) + + def _locate(self, path: str) -> tuple[StorageBackend, str]: + return self._resolver.resolve(self._uri.joinpath(path)) + + def __eq__(self, other: object) -> bool: + return isinstance(other, Storage) and other._uri == self._uri + + def __hash__(self) -> int: + return hash(self._uri) + + def __str__(self) -> str: + return str(self._uri) + + def __repr__(self) -> str: + return f"Storage({str(self._uri)!r})" diff --git a/automation_file/storage/types.py b/automation_file/storage/types.py new file mode 100644 index 0000000..7297476 --- /dev/null +++ b/automation_file/storage/types.py @@ -0,0 +1,121 @@ +"""Value types shared by every storage backend. + +``FileInfo`` is what ``stat`` and ``list`` return, ``Checksum`` what ``checksum`` +returns, and ``StorageCapabilities`` says which optional ``FileInfo`` fields and +which directory semantics a backend provides. All three are frozen and turn into +JSON-friendly dictionaries with ``to_dict`` so action results can cross the TCP, +HTTP and MCP transports unchanged. +""" + +from __future__ import annotations + +import hmac +from collections.abc import Mapping +from dataclasses import asdict, dataclass, field +from datetime import datetime +from typing import Any + +_CHECKSUM_SEPARATOR = ":" + + +@dataclass(frozen=True) +class Checksum: + """A digest and the algorithm that produced it, both lower-case.""" + + algorithm: str + value: str + + def __post_init__(self) -> None: + object.__setattr__(self, "algorithm", self.algorithm.strip().lower()) + object.__setattr__(self, "value", self.value.strip().lower()) + + @classmethod + def parse(cls, text: str) -> Checksum: + """Build a checksum from ``":"``.""" + algorithm, separator, value = text.partition(_CHECKSUM_SEPARATOR) + if not separator or not algorithm.strip() or not value.strip(): + raise ValueError(f"checksum must look like 'sha256:', got {text!r}") + return cls(algorithm, value) + + def matches(self, expected: str | Checksum) -> bool: + """Compare in constant time against a digest, ``"algorithm:digest"`` or a ``Checksum``. + + A different algorithm never matches. A bare digest is compared as is. + """ + if isinstance(expected, str) and _CHECKSUM_SEPARATOR in expected: + expected = Checksum.parse(expected) + if isinstance(expected, Checksum): + if expected.algorithm != self.algorithm: + return False + expected = expected.value + return hmac.compare_digest( + self.value.encode("utf-8"), expected.strip().lower().encode("utf-8") + ) + + def to_dict(self) -> dict[str, str]: + return {"algorithm": self.algorithm, "value": self.value} + + def __str__(self) -> str: + return f"{self.algorithm}{_CHECKSUM_SEPARATOR}{self.value}" + + +@dataclass(frozen=True) +class FileInfo: + """One file or directory as a backend reports it. + + ``path`` uses ``/`` separators and has no leading slash. A backend reports it + relative to its own root; :class:`~automation_file.storage.File` and + :class:`~automation_file.storage.Storage` report it relative to the object that + was asked. Fields a backend cannot provide are ``None`` (``metadata`` is empty); + :class:`StorageCapabilities` says which ones to expect. + """ + + path: str + is_dir: bool = False + size: int | None = None + modified_at: datetime | None = None + etag: str | None = None + version: str | None = None + content_type: str | None = None + metadata: Mapping[str, str] = field(default_factory=dict, hash=False) + + @property + def name(self) -> str: + """The last path segment (empty for a storage root).""" + return self.path.rsplit("/", 1)[-1] + + def to_dict(self) -> dict[str, Any]: + """Return a JSON-serialisable mapping; ``modified_at`` becomes ISO 8601.""" + return { + "path": self.path, + "name": self.name, + "is_dir": self.is_dir, + "size": self.size, + "modified_at": self.modified_at.isoformat() if self.modified_at else None, + "etag": self.etag, + "version": self.version, + "content_type": self.content_type, + "metadata": dict(self.metadata), + } + + +@dataclass(frozen=True) +class StorageCapabilities: + """What a backend provides beyond the mandatory contract. + + ``directories`` is ``True`` when directories exist on their own (a filesystem) + and ``False`` when they are only implied by the paths of the files under them + (an object store): there ``mkdir`` creates nothing and an empty directory + cannot exist. The other flags name the optional :class:`FileInfo` fields the + backend fills in. + """ + + directories: bool = True + modified_at: bool = True + etag: bool = False + version: bool = False + content_type: bool = False + metadata: bool = False + + def to_dict(self) -> dict[str, bool]: + return asdict(self) diff --git a/automation_file/storage/uri.py b/automation_file/storage/uri.py new file mode 100644 index 0000000..10b3d74 --- /dev/null +++ b/automation_file/storage/uri.py @@ -0,0 +1,160 @@ +"""Storage URIs: ``:///``. + +One syntax addresses every backend:: + + local:///data/report.csv s3://bucket/report.csv + local:///C:/data/report.csv azure://container/report.csv + sftp://server/data/report.csv dropbox:///reports/report.csv + +* **scheme** picks the backend. It is lower-cased; ``file`` is an alias of ``local`` + and ``az`` of ``azure``. +* **authority** is what the backend needs to find its root: a bucket, a container, + a host. It is kept as written. Credentials never belong in it, so ``user@host`` + is rejected. +* **path** is taken literally. Nothing is percent-decoded and ``?`` / ``#`` are + ordinary characters, so ``s3://bucket/Q1 #3?.csv`` names exactly that key. Empty + and ``.`` segments are dropped; a ``..`` segment is an error. + +Text without ``://`` is a local filesystem path and is made absolute, so +``reports/a.csv`` and ``C:\\data\\a.csv`` work as they are. Text that starts with +something scheme-like but has no ``//`` (``sftp:/data/a.csv``) could mean either, +so it is rejected with the two unambiguous spellings. +""" + +from __future__ import annotations + +import os +import re +from dataclasses import dataclass +from typing import TypeAlias + +from automation_file.exceptions import StorageURIException + +LOCAL_SCHEME = "local" +_URI_SEPARATOR = "://" +_SCHEME_ALIASES = {"file": LOCAL_SCHEME, "az": "azure"} +_SCHEME_PATTERN = re.compile(r"[a-z][a-z0-9+.-]*") +# Two or more characters before the colon: one character is a Windows drive letter. +_SCHEME_LIKE_PREFIX = re.compile(r"([A-Za-z][A-Za-z0-9+.-]+):") +_AUTHORITY_FORBIDDEN = re.compile(r"[\s/\\]") + + +def normalize_path(path: str) -> str: + """Return ``path`` with ``/`` separators and no empty, ``.`` or leading segments. + + Raises :class:`StorageURIException` for a ``..`` segment or a NUL character, so + a normalised path can never climb out of the root it is joined to. + """ + if "\x00" in path: + raise StorageURIException("storage paths cannot contain a NUL character") + segments: list[str] = [] + for segment in path.split("/"): + if segment in ("", "."): + continue + if segment == "..": + raise StorageURIException(f"storage paths cannot contain '..' segments: {path!r}") + segments.append(segment) + return "/".join(segments) + + +def canonical_scheme(scheme: str) -> str: + """Lower-case ``scheme``, resolve its alias and check its syntax.""" + lowered = scheme.strip().lower() + if not _SCHEME_PATTERN.fullmatch(lowered): + raise StorageURIException(f"invalid storage URI scheme: {scheme!r}") + return _SCHEME_ALIASES.get(lowered, lowered) + + +def _checked_authority(authority: str, scheme: str) -> str: + cleaned = authority.strip() + if "@" in cleaned: + raise StorageURIException( + f"{scheme} URI authority {cleaned.rsplit('@', 1)[-1]!r} carries user information; " + "credentials do not belong in a storage URI, initialise the backend with them instead" + ) + if _AUTHORITY_FORBIDDEN.search(cleaned): + raise StorageURIException(f"invalid {scheme} URI authority: {authority!r}") + return cleaned + + +@dataclass(frozen=True) +class StorageURI: + """A parsed storage URI. Construction validates and normalises all three parts.""" + + scheme: str + authority: str = "" + path: str = "" + + def __post_init__(self) -> None: + scheme = canonical_scheme(self.scheme) + object.__setattr__(self, "scheme", scheme) + object.__setattr__(self, "authority", _checked_authority(self.authority, scheme)) + object.__setattr__(self, "path", normalize_path(self.path)) + + @property + def name(self) -> str: + """The last path segment (empty at the root).""" + return self.path.rsplit("/", 1)[-1] + + @property + def parent(self) -> StorageURI: + """The URI one level up; the root is its own parent.""" + head, _, _ = self.path.rpartition("/") + return StorageURI(self.scheme, self.authority, head) + + def joinpath(self, *segments: str) -> StorageURI: + """Return this URI with ``segments`` appended to its path.""" + return StorageURI(self.scheme, self.authority, "/".join((self.path, *segments))) + + def __str__(self) -> str: + root = f"{self.scheme}{_URI_SEPARATOR}{self.authority}" + if self.path or not self.authority: + return f"{root}/{self.path}" + return root + + +URILike: TypeAlias = str | os.PathLike[str] | StorageURI + + +def local_path_to_uri(path: str | os.PathLike[str]) -> StorageURI: + """Return the ``local`` URI of a filesystem path, made absolute first. + + A Windows UNC path ``\\\\server\\share\\x`` becomes ``local://server/share/x``. + """ + absolute = os.path.abspath(os.fspath(path)) + if os.sep != "\\": + return StorageURI(LOCAL_SCHEME, "", absolute) + posix = absolute.replace("\\", "/") + if posix.startswith("//"): + host, _, tail = posix[2:].partition("/") + return StorageURI(LOCAL_SCHEME, host, tail) + return StorageURI(LOCAL_SCHEME, "", posix) + + +def parse_storage_uri(value: URILike) -> StorageURI: + """Turn a URI string, a filesystem path or a ``StorageURI`` into a ``StorageURI``. + + Raises :class:`StorageURIException` when the text is empty, malformed or could + be read as either a URI or a local path. + """ + if isinstance(value, StorageURI): + return value + text = os.fspath(value) + if not isinstance(text, str): + raise StorageURIException("storage URIs and paths must be text, not bytes") + if not text.strip(): + raise StorageURIException("storage URI is empty") + if not isinstance(value, str): + return local_path_to_uri(text) + scheme, separator, rest = text.partition(_URI_SEPARATOR) + if separator and len(scheme) > 1 and _SCHEME_PATTERN.fullmatch(scheme.lower()): + authority, _, path = rest.partition("/") + return StorageURI(scheme, authority, path) + scheme_like = _SCHEME_LIKE_PREFIX.match(text) + if scheme_like: + prefix = scheme_like.group(1) + raise StorageURIException( + f"{text!r} is ambiguous: write '{prefix.lower()}://...' for a storage URI, " + f"or './{text}' for a local file of that name" + ) + return local_path_to_uri(text) diff --git a/docs/source/API/api_index.rst b/docs/source/API/api_index.rst index 335e64f..43c3feb 100644 --- a/docs/source/API/api_index.rst +++ b/docs/source/API/api_index.rst @@ -172,3 +172,17 @@ File-discovery, fast-find, deduplicate, grep, and rotate helpers. :caption: Utils utils + +.. _api-storage: + +Chapter M — Universal Storage Layer +=================================== + +``File``, ``Storage``, the ``StorageBackend`` contract, storage URIs, the +resolver, and the built-in local and in-memory backends. + +.. toctree:: + :maxdepth: 2 + :caption: Universal Storage Layer + + storage \ No newline at end of file diff --git a/docs/source/API/storage.rst b/docs/source/API/storage.rst new file mode 100644 index 0000000..951cd82 --- /dev/null +++ b/docs/source/API/storage.rst @@ -0,0 +1,46 @@ +Universal storage layer +======================= + +The application API (``File``, ``Storage``), the backend contract +(``StorageBackend``), the URI syntax and the resolver. Usage and the full +contract are described in the manual chapter *Universal Storage Layer*. + +File and Storage +---------------- + +.. automodule:: automation_file.storage.file + :members: + +.. automodule:: automation_file.storage.storage + :members: + +Storage URIs +------------ + +.. automodule:: automation_file.storage.uri + :members: + +Backend contract +---------------- + +.. automodule:: automation_file.storage.backend + :members: + :private-members: _stat, _list_dir, _upload, _download, _delete_file, _mkdir, _rmdir, _walk, _copy_from, _move_from, _checksum, _read_bytes + +.. automodule:: automation_file.storage.types + :members: + +Resolver +-------- + +.. automodule:: automation_file.storage.resolver + :members: + +Built-in backends +----------------- + +.. automodule:: automation_file.storage.local_storage + :members: + +.. automodule:: automation_file.storage.memory_storage + :members: diff --git a/docs/source/Eng/eng_index.rst b/docs/source/Eng/eng_index.rst index 5a94bdd..71100e8 100644 --- a/docs/source/Eng/eng_index.rst +++ b/docs/source/Eng/eng_index.rst @@ -242,3 +242,18 @@ imports a Python package and registers its top-level members as :caption: Plugins usage/plugins + +.. _eng-storage: + +Chapter 16 — Universal Storage Layer +==================================== + +``File`` and ``Storage`` address local and remote storage through one URI +syntax; ``StorageBackend`` is the single contract a backend implements, with +shared operations, shared errors and a reusable contract test suite. + +.. toctree:: + :maxdepth: 2 + :caption: Universal Storage Layer + + usage/storage \ No newline at end of file diff --git a/docs/source/Eng/usage/storage.rst b/docs/source/Eng/usage/storage.rst new file mode 100644 index 0000000..bcd5a1b --- /dev/null +++ b/docs/source/Eng/usage/storage.rst @@ -0,0 +1,272 @@ +Universal storage layer +======================= + +``automation_file.storage`` gives every storage one address syntax, one set of +operations and one set of errors. :class:`~automation_file.File` and +:class:`~automation_file.Storage` are the application API; +:class:`~automation_file.StorageBackend` is the contract a backend implements. + +The ``FA_*`` actions and the per-backend functions (``s3_upload_file``, +``sftp_download_file`` …) are unchanged and keep working next to it. + +.. note:: + + The layer is new and its API may still change before 1.0. The local + filesystem and an in-memory store are built in today. S3, Azure Blob, Google + Drive, Dropbox, SFTP, FTP, WebDAV, SMB and fsspec are reached through their + existing clients and actions (:doc:`cloud`) until their adapters land; you can + already put any of them behind the layer by writing a backend + (`Writing a backend`_). + +Quick start +----------- + +.. code-block:: python + + from automation_file import File, Storage + + report = File("local:///data/reports/q1.csv") # or just "reports/q1.csv" + report.write("region,total\nEMEA,42\n") + report.exists() # True + report.size # 21 + report.read_text() + report.checksum() # Checksum("sha256", "…") + report.verify("sha256:9f86d081…") # constant-time compare + + archive = report.copy_to("memory://scratch/archive/q1.csv") + report.move_to("local:///data/done/q1.csv") + + reports = Storage("local:///data/reports") + for info in reports.list_dir(recursive=True): + print(info.path, info.size, info.modified_at) + reports.upload("q2.csv", "2026/q2.csv") + reports.file("2026/q2.csv").download_to("copy-of-q2.csv") + reports.delete("2026", recursive=True) + +Creating a ``File`` or a ``Storage`` touches nothing. The backend is looked up +on every call, so an object can be created before its backend is initialised or +mounted. + +Storage URIs +------------ + +.. code-block:: text + + :/// + + local:///data/report.csv s3://bucket/report.csv + local:///C:/data/report.csv azure://container/report.csv + sftp://server/data/report.csv dropbox:///reports/report.csv + memory://scratch/report.csv smb://server/share/report.csv + +``scheme`` + Picks the backend. Lower-cased. ``file`` is an alias of ``local`` and ``az`` + of ``azure``. + +``authority`` + What the backend needs to find its root: a bucket, a container, a host. Kept + as written. Credentials never belong in a URI, so ``user@host`` and + ``user:password@host`` are rejected, and the error message does not repeat + them. + +``path`` + Taken literally. Nothing is percent-decoded and ``?`` and ``#`` are ordinary + characters, so ``s3://bucket/Q1 #3?.csv`` names exactly that key. Empty and + ``.`` segments are dropped. A ``..`` segment is an error, so a path can never + climb out of the root it is joined to. + +Local paths + Text without ``://`` is a filesystem path and is made absolute: + ``reports/a.csv``, ``/data/a.csv`` and ``C:\data\a.csv`` work as they are, + and so does a :class:`pathlib.Path`. On Windows a UNC path + ``\\server\share\a.csv`` is ``local://server/share/a.csv``. + +Ambiguous text + ``sftp:/data/a.csv`` could be a URI with a slash missing or a local file + called ``sftp:``. It is rejected, and the message gives both unambiguous + spellings: ``sftp://…`` for the URI, ``./sftp:/data/a.csv`` for the file. + +:func:`~automation_file.parse_storage_uri` returns a frozen +:class:`~automation_file.StorageURI` with ``scheme``, ``authority``, ``path``, +``name``, ``parent`` and ``joinpath()``. Malformed input raises +:class:`~automation_file.StorageURIException`. + +Operations +---------- + +Every backend has the same methods. ``File`` and ``Storage`` forward to them. + +.. list-table:: + :header-rows: 1 + :widths: 34 66 + + * - Method + - Behaviour + * - ``exists(path)`` + - ``True`` for a file or a directory. + * - ``stat(path)`` + - Returns a :class:`~automation_file.FileInfo`. Missing path: + ``StorageNotFoundException``. + * - ``list_dir(path="", recursive=False)`` + - Entries sorted by path. ``recursive=True`` returns every descendant, + directories included. A file: ``StoragePathTypeException``. + * - ``mkdir(path, parents=True, exist_ok=True)`` + - Creates a directory. On a backend where directories are only implied by + file paths, nothing is created and nothing needs to be. + * - ``upload(local_path, path, overwrite=True)`` + - Stores a local file and creates missing parent directories. With + ``overwrite=False`` an existing file raises + ``StorageAlreadyExistsException``. + * - ``download(path, local_path, overwrite=True)`` + - Writes to a sibling ``.part`` file that replaces the target when complete, + so a failed download never leaves a truncated file. + * - ``delete(path, recursive=False, missing_ok=False)`` + - A directory with entries needs ``recursive=True`` + (``StorageNotEmptyException`` otherwise). The storage root is never + deleted. + * - ``checksum(path, algorithm="sha256")`` + - Returns a :class:`~automation_file.Checksum`. Any fixed-length + ``hashlib`` algorithm: ``sha256``, ``sha512``, ``blake2b``, ``md5`` (for + compatibility, not for security). + * - ``read_bytes(path)`` / ``write_bytes(path, data)`` + - Whole-file content. + * - ``copy_from(source, source_path, path)`` / ``move_from(…)`` + - Transfer from any backend, this one included. Native when the two backends + can do it between themselves (a local rename), through a local staging + file otherwise. + +``FileInfo`` carries ``path``, ``name``, ``is_dir``, ``size``, ``modified_at`` +(timezone-aware UTC), ``etag``, ``version``, ``content_type`` and ``metadata``. +Fields a backend cannot provide are ``None``. ``FileInfo.to_dict()`` and +``Checksum.to_dict()`` are JSON-serialisable. + +``backend.capabilities`` is a :class:`~automation_file.StorageCapabilities` that +says which optional ``FileInfo`` fields the backend fills in and whether its +directories are real (``directories=True``, a filesystem) or implied by file +paths (``directories=False``, an object store). + +Errors +------ + +All derive from :class:`~automation_file.StorageException`, itself a +``FileAutomationException``. + +.. list-table:: + :header-rows: 1 + :widths: 40 60 + + * - Exception + - Raised when + * - ``StorageURIException`` + - A URI or path is malformed or ambiguous, or no backend serves it. + * - ``StorageNotFoundException`` + - A path, or the local source of an upload, is missing. Also a + ``FileNotExistsException``, so existing handlers keep working. + * - ``StorageAlreadyExistsException`` + - A write would replace something and ``overwrite`` / ``exist_ok`` is off. + * - ``StoragePathTypeException`` + - A file operation targets a directory, or the reverse. + * - ``StorageNotEmptyException`` + - A directory with entries is deleted without ``recursive=True``. + * - ``StoragePermissionException`` + - The backend denies access. + * - ``StorageTransientException`` + - A failure worth retrying: timeout, dropped connection, throttling. Use it + as the ``retriable=`` type of ``retry_on_transient``. + * - ``StorageUnavailableException`` + - The backend is not initialised or its SDK is not installed. + * - ``StorageUnsupportedException`` + - The backend cannot do what was asked (an unknown checksum algorithm, + deleting the root). + +Built-in backends +----------------- + +``LocalStorage`` (``local://``, alias ``file://``) + ``LocalStorage()`` spans the whole filesystem and is what ``local:///…`` + resolves to. ``LocalStorage(root)`` is confined to one directory tree: every + path goes through :func:`~automation_file.safe_join`, so a path that leaves + the root through a symbolic link raises ``PathTraversalException``. Use a + rooted instance whenever paths come from outside the process. + + Symbolic links are followed when reading and writing. Deleting never follows + them: the link is removed and its target is left alone. Recursive listing + does not descend into linked directories. Writes go through a temporary + sibling file and replace the target atomically. + +``MemoryStorage`` (``memory:///…``) + A thread-safe tree held in memory, for tests, dry runs and examples. Each + ```` is a separate store, created on first use. + +Mounting and registering backends +--------------------------------- + +A URI is resolved in two steps. **Mounts** come first: a mount binds one backend +instance to a URI, and every URI at or below it goes to that backend with the +mount's own path stripped. The longest matching mount wins. **Scheme factories** +handle every URI no mount claimed. + +.. code-block:: python + + from automation_file import File, LocalStorage, Storage + + # A name of your own for one directory tree. Nothing addressed through + # sandbox://jobs/… can leave /srv/jobs, not even through a symbolic link. + Storage.mount("sandbox://jobs", LocalStorage("/srv/jobs")) + File("sandbox://jobs/42/out.csv").write(b"done") # /srv/jobs/42/out.csv + + # A whole scheme: the factory gets the parsed URI and returns (backend, path). + Storage.register_scheme("vault", lambda uri: (vault_backend(uri.authority), uri.path)) + + Storage.resolve("sandbox://jobs/42/out.csv") # (LocalStorage('/srv/jobs'), '42/out.csv') + Storage.schemes() # ['local', 'memory', 'sandbox', 'vault'] + +``Storage.mount`` / ``unmount`` / ``register_scheme`` / ``schemes`` / ``resolve`` +work on the process-wide table. A private table is a +:class:`~automation_file.StorageResolver`; pass it as ``resolver=`` to ``File`` +and ``Storage``. + +A mount is matched on the text of the URI. Mounting a rooted backend over a +``local://`` path therefore routes that spelling of the path, but another +spelling of the same directory (a symbolic link to it) still reaches the whole +filesystem. To confine untrusted paths, give the rooted backend a scheme or +authority of its own, as above, and accept only URIs below it. + +Writing a backend +----------------- + +Subclass :class:`~automation_file.StorageBackend` and implement the primitives. +The public methods above are inherited: they normalise paths, check what is +already there, raise the shared exceptions and create parent directories. + +.. code-block:: python + + from automation_file import FileInfo, StorageBackend, StorageCapabilities + + class VaultStorage(StorageBackend): + scheme = "vault" + capabilities = StorageCapabilities(directories=False, etag=True) + + def _stat(self, path): ... # FileInfo, or None when absent; "" is the root + def _list_dir(self, path): ... # immediate children of a directory + def _upload(self, source, path): ... + def _download(self, path, target): ... + def _delete_file(self, path): ... + # With real directories (capabilities.directories=True) also: + # _mkdir(path), _rmdir(path) + # Optional overrides: _walk, _copy_from, _move_from, _checksum, _read_bytes + +Check it with the contract suite. ``tests/storage_contract.py`` holds 70 cases — +nested directories, empty and large files, Unicode paths, binary data, overwrite +and missing-path behaviour, path normalisation, copy and move — and reads +``capabilities`` where backends legitimately differ: + +.. code-block:: python + + import pytest + from tests.storage_contract import StorageContract + + class TestVaultStorageContract(StorageContract): + @pytest.fixture + def backend(self): + return VaultStorage(...) # an empty storage for each test diff --git a/docs/source/Zh-CN/usage/storage.rst b/docs/source/Zh-CN/usage/storage.rst new file mode 100644 index 0000000..2434d30 --- /dev/null +++ b/docs/source/Zh-CN/usage/storage.rst @@ -0,0 +1,255 @@ +通用存储层 +========== + +``automation_file.storage`` 让每一种存储都使用同一套地址语法、同一组操作和同一组 +异常。:class:`~automation_file.File` 与 :class:`~automation_file.Storage` 是应用层 +API；:class:`~automation_file.StorageBackend` 则是后端需要实现的契约。 + +``FA_*`` 动作以及各后端原有的函数（``s3_upload_file``、``sftp_download_file`` ……） +完全不变，可以与本层同时使用。 + +.. note:: + + 本层是新功能，API 在 1.0 之前仍可能调整。目前内置本地文件系统与内存存储两种 + 后端。S3、Azure Blob、Google Drive、Dropbox、SFTP、FTP、WebDAV、SMB 与 fsspec + 在各自的适配器完成之前，仍通过已有的客户端与动作使用（见 :doc:`cloud`）；你也 + 可以现在就自行编写后端，把它们接到本层之后（见 `编写后端`_）。 + +快速开始 +-------- + +.. code-block:: python + + from automation_file import File, Storage + + report = File("local:///data/reports/q1.csv") # 也可以直接写 "reports/q1.csv" + report.write("region,total\nEMEA,42\n") + report.exists() # True + report.size # 21 + report.read_text() + report.checksum() # Checksum("sha256", "…") + report.verify("sha256:9f86d081…") # 常数时间比较 + + archive = report.copy_to("memory://scratch/archive/q1.csv") + report.move_to("local:///data/done/q1.csv") + + reports = Storage("local:///data/reports") + for info in reports.list_dir(recursive=True): + print(info.path, info.size, info.modified_at) + reports.upload("q2.csv", "2026/q2.csv") + reports.file("2026/q2.csv").download_to("copy-of-q2.csv") + reports.delete("2026", recursive=True) + +创建 ``File`` 或 ``Storage`` 对象不会触碰任何存储。每次调用时才查找后端，因此可以在 +后端初始化或挂载之前先创建对象。 + +存储 URI +-------- + +.. code-block:: text + + :/// + + local:///data/report.csv s3://bucket/report.csv + local:///C:/data/report.csv azure://container/report.csv + sftp://server/data/report.csv dropbox:///reports/report.csv + memory://scratch/report.csv smb://server/share/report.csv + +``scheme`` + 决定使用哪个后端，一律转为小写。``file`` 是 ``local`` 的别名，``az`` 是 + ``azure`` 的别名。 + +``authority`` + 后端找到自身根目录所需的信息：bucket、container 或主机名，保持原样。凭据 + 不属于 URI，因此 ``user@host`` 与 ``user:password@host`` 会被拒绝，并且错误 + 信息不会重复显示这些内容。 + +``path`` + 按字面理解。不做百分号解码，``?`` 与 ``#`` 都是普通字符，所以 + ``s3://bucket/Q1 #3?.csv`` 指的就是那个 key。空段与 ``.`` 段会被丢弃； + ``..`` 段视为错误，因此路径永远无法跳出它所拼接的根目录。 + +本地路径 + 不含 ``://`` 的文本是文件系统路径，会先转换为绝对路径：``reports/a.csv``、 + ``/data/a.csv`` 与 ``C:\data\a.csv`` 都可以直接使用，:class:`pathlib.Path` 也 + 一样。在 Windows 上，UNC 路径 ``\\server\share\a.csv`` 等同于 + ``local://server/share/a.csv``。 + +有歧义的文本 + ``sftp:/data/a.csv`` 可能是少写一个斜杠的 URI，也可能是名为 ``sftp:`` 的本地 + 文件。这种写法会被拒绝，信息中会给出两种明确的写法：URI 写成 ``sftp://…``， + 本地文件写成 ``./sftp:/data/a.csv``。 + +:func:`~automation_file.parse_storage_uri` 返回不可变的 +:class:`~automation_file.StorageURI`，具有 ``scheme``、``authority``、``path``、 +``name``、``parent`` 与 ``joinpath()``。格式错误的输入会抛出 +:class:`~automation_file.StorageURIException`。 + +操作 +---- + +每个后端的方法都相同，``File`` 与 ``Storage`` 会转调它们。 + +.. list-table:: + :header-rows: 1 + :widths: 34 66 + + * - 方法 + - 行为 + * - ``exists(path)`` + - 文件或目录存在时返回 ``True``。 + * - ``stat(path)`` + - 返回 :class:`~automation_file.FileInfo`。路径不存在时抛出 + ``StorageNotFoundException``。 + * - ``list_dir(path="", recursive=False)`` + - 按路径排序的条目。``recursive=True`` 返回所有后代条目，包含目录。对象是 + 文件时抛出 ``StoragePathTypeException``。 + * - ``mkdir(path, parents=True, exist_ok=True)`` + - 创建目录。如果后端的目录只是由文件路径隐含而来，则不会创建任何东西，也不 + 需要创建。 + * - ``upload(local_path, path, overwrite=True)`` + - 存入本地文件，并创建缺少的上级目录。``overwrite=False`` 时如果文件已存在， + 抛出 ``StorageAlreadyExistsException``。 + * - ``download(path, local_path, overwrite=True)`` + - 先写入同目录下的 ``.part`` 文件，完成后才替换目标，所以下载失败不会留下 + 被截断的文件。 + * - ``delete(path, recursive=False, missing_ok=False)`` + - 目录内有条目时需要 ``recursive=True``（否则抛出 + ``StorageNotEmptyException``）。存储的根目录永远不会被删除。 + * - ``checksum(path, algorithm="sha256")`` + - 返回 :class:`~automation_file.Checksum`。支持 ``hashlib`` 中所有固定长度 + 的算法：``sha256``、``sha512``、``blake2b``、``md5``（仅用于兼容，不用于 + 安全目的）。 + * - ``read_bytes(path)`` / ``write_bytes(path, data)`` + - 读写整个文件的内容。 + * - ``copy_from(source, source_path, path)`` / ``move_from(…)`` + - 从任何后端（包含自己）传输。两个后端之间能直接完成时走原生方式（例如 + 本地重命名），否则经由本地暂存文件。 + +``FileInfo`` 包含 ``path``、``name``、``is_dir``、``size``、``modified_at``（带 +时区的 UTC 时间）、``etag``、``version``、``content_type`` 与 ``metadata``。后端 +无法提供的字段为 ``None``。``FileInfo.to_dict()`` 与 ``Checksum.to_dict()`` 的结果 +可以直接序列化为 JSON。 + +``backend.capabilities`` 是 :class:`~automation_file.StorageCapabilities`，说明后端 +会填入哪些可选的 ``FileInfo`` 字段，以及它的目录是真实存在（``directories=True``， +文件系统）还是由文件路径隐含（``directories=False``，对象存储）。 + +异常 +---- + +全部派生自 :class:`~automation_file.StorageException`，而它本身是 +``FileAutomationException`` 的子类。 + +.. list-table:: + :header-rows: 1 + :widths: 40 60 + + * - 异常 + - 抛出时机 + * - ``StorageURIException`` + - URI 或路径格式错误、有歧义，或没有后端能处理。 + * - ``StorageNotFoundException`` + - 路径或上传的本地源文件不存在。它同时是 ``FileNotExistsException``，所以 + 已有的异常处理仍然有效。 + * - ``StorageAlreadyExistsException`` + - 写入会替换已有内容，而 ``overwrite`` / ``exist_ok`` 为关闭。 + * - ``StoragePathTypeException`` + - 对目录执行文件操作，或对文件执行目录操作。 + * - ``StorageNotEmptyException`` + - 删除内有条目的目录却没有指定 ``recursive=True``。 + * - ``StoragePermissionException`` + - 后端拒绝访问。 + * - ``StorageTransientException`` + - 值得重试的失败：超时、连接中断、被限流。可以作为 ``retry_on_transient`` 的 + ``retriable=`` 类型。 + * - ``StorageUnavailableException`` + - 后端尚未初始化，或其 SDK 未安装。 + * - ``StorageUnsupportedException`` + - 后端无法执行所要求的操作（未知的校验算法、删除根目录）。 + +内置后端 +-------- + +``LocalStorage``（``local://``，别名 ``file://``） + ``LocalStorage()`` 覆盖整个文件系统，也就是 ``local:///…`` 解析的结果。 + ``LocalStorage(root)`` 则被限制在单个目录树内：每个路径都经过 + :func:`~automation_file.safe_join`，因此通过符号链接离开根目录的路径会抛出 + ``PathTraversalException``。只要路径来自进程之外，就应使用带根目录的实例。 + + 读写时会跟随符号链接；删除时绝不跟随：只移除链接本身，不动它指向的目标。 + 递归列出时不会进入被链接的目录。写入会先写到同目录的临时文件，再以原子操作 + 替换目标。 + +``MemoryStorage``（``memory:///…``） + 保存在内存中的线程安全目录树，用于测试、试运行与示例。每个 ```` 是 + 独立的存储，首次使用时创建。 + +挂载与注册后端 +-------------- + +URI 的解析分两步。**挂载** 优先：挂载把一个后端实例绑定到某个 URI，位于该 URI +或其下的所有 URI 都交给这个后端，并去掉挂载点本身的路径；匹配的挂载中最长者 +优先。没有任何挂载认领的 URI 则交给 **scheme 工厂**。 + +.. code-block:: python + + from automation_file import File, LocalStorage, Storage + + # 为某个目录树取一个自己的名称。通过 sandbox://jobs/… 访问的任何东西都离不开 + # /srv/jobs，即使经由符号链接也一样。 + Storage.mount("sandbox://jobs", LocalStorage("/srv/jobs")) + File("sandbox://jobs/42/out.csv").write(b"done") # /srv/jobs/42/out.csv + + # 整个 scheme：工厂收到解析后的 URI，返回 (backend, path)。 + Storage.register_scheme("vault", lambda uri: (vault_backend(uri.authority), uri.path)) + + Storage.resolve("sandbox://jobs/42/out.csv") # (LocalStorage('/srv/jobs'), '42/out.csv') + Storage.schemes() # ['local', 'memory', 'sandbox', 'vault'] + +``Storage.mount`` / ``unmount`` / ``register_scheme`` / ``schemes`` / ``resolve`` +操作的是整个进程共用的表。需要私有的表时使用 +:class:`~automation_file.StorageResolver`，并以 ``resolver=`` 传给 ``File`` 与 +``Storage``。 + +挂载是按 URI 的文本匹配。因此把带根目录的后端挂在某个 ``local://`` 路径上，只会 +路由该路径的那一种写法；同一个目录的另一种写法（例如指向它的符号链接）仍然会 +连到整个文件系统。要限制不受信任的路径，请像上面那样给带根目录的后端一个专属的 +scheme 或 authority，并且只接受其下的 URI。 + +编写后端 +-------- + +继承 :class:`~automation_file.StorageBackend` 并实现基本操作即可。上述公开方法 +都由基类提供：它们会规范化路径、检查已有内容、抛出共用的异常并创建上级目录。 + +.. code-block:: python + + from automation_file import FileInfo, StorageBackend, StorageCapabilities + + class VaultStorage(StorageBackend): + scheme = "vault" + capabilities = StorageCapabilities(directories=False, etag=True) + + def _stat(self, path): ... # FileInfo；不存在时返回 None；"" 是根目录 + def _list_dir(self, path): ... # 目录的直接子条目 + def _upload(self, source, path): ... + def _download(self, path, target): ... + def _delete_file(self, path): ... + # 目录真实存在时（capabilities.directories=True）还需要： + # _mkdir(path)、_rmdir(path) + # 可选择重写：_walk、_copy_from、_move_from、_checksum、_read_bytes + +请用契约测试套件检查。``tests/storage_contract.py`` 包含 70 个用例——嵌套目录、 +空文件与大文件、Unicode 路径、二进制数据、覆盖与路径不存在时的行为、路径规范化、 +复制与移动——并在后端确实有差异之处读取 ``capabilities``： + +.. code-block:: python + + import pytest + from tests.storage_contract import StorageContract + + class TestVaultStorageContract(StorageContract): + @pytest.fixture + def backend(self): + return VaultStorage(...) # 每个测试都要是空的存储 diff --git a/docs/source/Zh-CN/zh_cn_index.rst b/docs/source/Zh-CN/zh_cn_index.rst index 136f4bf..49001ee 100644 --- a/docs/source/Zh-CN/zh_cn_index.rst +++ b/docs/source/Zh-CN/zh_cn_index.rst @@ -237,3 +237,18 @@ Slack、Email（SMTP）、Discord、Telegram、Microsoft Teams、PagerDuty :caption: 插件 usage/plugins + +.. _zh-cn-storage: + +第 16 章 — 通用存储层 +===================== + +``File`` 与 ``Storage`` 以同一套 URI 语法访问本地与远端存储； +``StorageBackend`` 是后端需要实现的唯一契约，提供共用的操作、共用的异常， +以及可重复使用的契约测试套件。 + +.. toctree:: + :maxdepth: 2 + :caption: 通用存储层 + + usage/storage \ No newline at end of file diff --git a/docs/source/Zh-TW/usage/storage.rst b/docs/source/Zh-TW/usage/storage.rst new file mode 100644 index 0000000..4a3667d --- /dev/null +++ b/docs/source/Zh-TW/usage/storage.rst @@ -0,0 +1,255 @@ +通用儲存層 +========== + +``automation_file.storage`` 讓每一種儲存都使用同一套位址語法、同一組操作與同一組 +例外。:class:`~automation_file.File` 與 :class:`~automation_file.Storage` 是應用層 +API；:class:`~automation_file.StorageBackend` 則是後端要實作的契約。 + +``FA_*`` 動作與各後端原有的函式（``s3_upload_file``、``sftp_download_file`` ……） +完全不變，可與本層並用。 + +.. note:: + + 本層是新功能，API 在 1.0 之前仍可能調整。目前內建本機檔案系統與記憶體儲存兩種 + 後端。S3、Azure Blob、Google Drive、Dropbox、SFTP、FTP、WebDAV、SMB 與 fsspec + 在各自的轉接器完成之前，仍透過既有的用戶端與動作使用（見 :doc:`cloud`）；你也 + 可以現在就自行撰寫後端，把它們接到本層之後（見 `撰寫後端`_）。 + +快速開始 +-------- + +.. code-block:: python + + from automation_file import File, Storage + + report = File("local:///data/reports/q1.csv") # 也可以直接寫 "reports/q1.csv" + report.write("region,total\nEMEA,42\n") + report.exists() # True + report.size # 21 + report.read_text() + report.checksum() # Checksum("sha256", "…") + report.verify("sha256:9f86d081…") # 常數時間比對 + + archive = report.copy_to("memory://scratch/archive/q1.csv") + report.move_to("local:///data/done/q1.csv") + + reports = Storage("local:///data/reports") + for info in reports.list_dir(recursive=True): + print(info.path, info.size, info.modified_at) + reports.upload("q2.csv", "2026/q2.csv") + reports.file("2026/q2.csv").download_to("copy-of-q2.csv") + reports.delete("2026", recursive=True) + +建立 ``File`` 或 ``Storage`` 物件不會碰觸任何儲存。每次呼叫才查找後端，因此可以在 +後端初始化或掛載之前先建立物件。 + +儲存 URI +-------- + +.. code-block:: text + + :/// + + local:///data/report.csv s3://bucket/report.csv + local:///C:/data/report.csv azure://container/report.csv + sftp://server/data/report.csv dropbox:///reports/report.csv + memory://scratch/report.csv smb://server/share/report.csv + +``scheme`` + 決定使用哪個後端，一律轉為小寫。``file`` 是 ``local`` 的別名，``az`` 是 + ``azure`` 的別名。 + +``authority`` + 後端找到自身根目錄所需的資訊：bucket、container 或主機名稱，保留原樣。憑證 + 不屬於 URI，因此 ``user@host`` 與 ``user:password@host`` 會被拒絕，而且錯誤 + 訊息不會重複顯示這些內容。 + +``path`` + 按字面解讀。不做百分比解碼，``?`` 與 ``#`` 都是一般字元，所以 + ``s3://bucket/Q1 #3?.csv`` 指的就是那個 key。空區段與 ``.`` 區段會被捨棄； + ``..`` 區段視為錯誤，因此路徑永遠無法跳出它所接上的根目錄。 + +本機路徑 + 不含 ``://`` 的文字是檔案系統路徑，會先轉成絕對路徑：``reports/a.csv``、 + ``/data/a.csv`` 與 ``C:\data\a.csv`` 都能直接使用，:class:`pathlib.Path` 也 + 一樣。在 Windows 上，UNC 路徑 ``\\server\share\a.csv`` 等同 + ``local://server/share/a.csv``。 + +有歧義的文字 + ``sftp:/data/a.csv`` 可能是少打一個斜線的 URI，也可能是名為 ``sftp:`` 的本機 + 檔案。這種寫法會被拒絕，訊息中會列出兩種明確的寫法：URI 寫成 ``sftp://…``， + 本機檔案寫成 ``./sftp:/data/a.csv``。 + +:func:`~automation_file.parse_storage_uri` 回傳不可變的 +:class:`~automation_file.StorageURI`，具有 ``scheme``、``authority``、``path``、 +``name``、``parent`` 與 ``joinpath()``。格式錯誤的輸入會拋出 +:class:`~automation_file.StorageURIException`。 + +操作 +---- + +每個後端的方法都相同，``File`` 與 ``Storage`` 會轉呼叫它們。 + +.. list-table:: + :header-rows: 1 + :widths: 34 66 + + * - 方法 + - 行為 + * - ``exists(path)`` + - 檔案或目錄存在時回傳 ``True``。 + * - ``stat(path)`` + - 回傳 :class:`~automation_file.FileInfo`。路徑不存在時拋出 + ``StorageNotFoundException``。 + * - ``list_dir(path="", recursive=False)`` + - 依路徑排序的項目。``recursive=True`` 回傳所有子孫項目，包含目錄。對象是 + 檔案時拋出 ``StoragePathTypeException``。 + * - ``mkdir(path, parents=True, exist_ok=True)`` + - 建立目錄。若後端的目錄只是由檔案路徑隱含而來，則不會建立任何東西，也不 + 需要建立。 + * - ``upload(local_path, path, overwrite=True)`` + - 存入本機檔案，並建立缺少的上層目錄。``overwrite=False`` 時若檔案已存在， + 拋出 ``StorageAlreadyExistsException``。 + * - ``download(path, local_path, overwrite=True)`` + - 先寫入同目錄下的 ``.part`` 檔，完成後才取代目標，所以下載失敗不會留下 + 被截斷的檔案。 + * - ``delete(path, recursive=False, missing_ok=False)`` + - 目錄內有項目時需要 ``recursive=True``（否則拋出 + ``StorageNotEmptyException``）。儲存的根目錄永遠不會被刪除。 + * - ``checksum(path, algorithm="sha256")`` + - 回傳 :class:`~automation_file.Checksum`。支援 ``hashlib`` 中所有固定長度 + 的演算法：``sha256``、``sha512``、``blake2b``、``md5``（僅供相容，不作 + 安全用途）。 + * - ``read_bytes(path)`` / ``write_bytes(path, data)`` + - 讀寫整個檔案的內容。 + * - ``copy_from(source, source_path, path)`` / ``move_from(…)`` + - 從任何後端（包含自己）傳輸。兩個後端之間能直接完成時走原生方式（例如 + 本機重新命名），否則經由本機暫存檔。 + +``FileInfo`` 包含 ``path``、``name``、``is_dir``、``size``、``modified_at``（帶 +時區的 UTC 時間）、``etag``、``version``、``content_type`` 與 ``metadata``。後端 +無法提供的欄位為 ``None``。``FileInfo.to_dict()`` 與 ``Checksum.to_dict()`` 的結果 +可直接序列化為 JSON。 + +``backend.capabilities`` 是 :class:`~automation_file.StorageCapabilities`，說明後端 +會填入哪些選用的 ``FileInfo`` 欄位，以及它的目錄是真實存在（``directories=True``， +檔案系統）還是由檔案路徑隱含（``directories=False``，物件儲存）。 + +例外 +---- + +全部衍生自 :class:`~automation_file.StorageException`，而它本身是 +``FileAutomationException`` 的子類別。 + +.. list-table:: + :header-rows: 1 + :widths: 40 60 + + * - 例外 + - 拋出時機 + * - ``StorageURIException`` + - URI 或路徑格式錯誤、有歧義，或沒有後端能處理。 + * - ``StorageNotFoundException`` + - 路徑或上傳的本機來源檔不存在。它同時是 ``FileNotExistsException``，所以 + 既有的例外處理仍然有效。 + * - ``StorageAlreadyExistsException`` + - 寫入會取代既有內容，而 ``overwrite`` / ``exist_ok`` 為關閉。 + * - ``StoragePathTypeException`` + - 對目錄執行檔案操作，或對檔案執行目錄操作。 + * - ``StorageNotEmptyException`` + - 刪除內有項目的目錄卻沒有指定 ``recursive=True``。 + * - ``StoragePermissionException`` + - 後端拒絕存取。 + * - ``StorageTransientException`` + - 值得重試的失敗：逾時、連線中斷、被限流。可作為 ``retry_on_transient`` 的 + ``retriable=`` 型別。 + * - ``StorageUnavailableException`` + - 後端尚未初始化，或其 SDK 未安裝。 + * - ``StorageUnsupportedException`` + - 後端無法執行所要求的操作（未知的校驗演算法、刪除根目錄）。 + +內建後端 +-------- + +``LocalStorage``（``local://``，別名 ``file://``） + ``LocalStorage()`` 涵蓋整個檔案系統，也就是 ``local:///…`` 解析的結果。 + ``LocalStorage(root)`` 則被限制在單一目錄樹內：每個路徑都經過 + :func:`~automation_file.safe_join`，因此透過符號連結離開根目錄的路徑會拋出 + ``PathTraversalException``。只要路徑來自行程之外，就應使用有根目錄的實例。 + + 讀寫時會跟隨符號連結；刪除時絕不跟隨：只移除連結本身，不動它指向的目標。 + 遞迴列出時不會進入被連結的目錄。寫入會先寫到同目錄的暫存檔，再以原子操作 + 取代目標。 + +``MemoryStorage``（``memory:///…``） + 存在記憶體中的執行緒安全目錄樹，用於測試、試跑與範例。每個 ```` 是 + 獨立的儲存，首次使用時建立。 + +掛載與註冊後端 +-------------- + +URI 的解析分兩步。**掛載** 優先：掛載把一個後端實例綁定到某個 URI，位於該 URI +或其下的所有 URI 都交給這個後端，並去掉掛載點本身的路徑；符合的掛載中最長者 +優先。沒有任何掛載認領的 URI 則交給 **scheme 工廠**。 + +.. code-block:: python + + from automation_file import File, LocalStorage, Storage + + # 為某個目錄樹取一個自己的名稱。透過 sandbox://jobs/… 存取的任何東西都離不開 + # /srv/jobs，即使經由符號連結也一樣。 + Storage.mount("sandbox://jobs", LocalStorage("/srv/jobs")) + File("sandbox://jobs/42/out.csv").write(b"done") # /srv/jobs/42/out.csv + + # 整個 scheme：工廠收到解析後的 URI，回傳 (backend, path)。 + Storage.register_scheme("vault", lambda uri: (vault_backend(uri.authority), uri.path)) + + Storage.resolve("sandbox://jobs/42/out.csv") # (LocalStorage('/srv/jobs'), '42/out.csv') + Storage.schemes() # ['local', 'memory', 'sandbox', 'vault'] + +``Storage.mount`` / ``unmount`` / ``register_scheme`` / ``schemes`` / ``resolve`` +操作的是整個行程共用的表。需要私有的表時使用 +:class:`~automation_file.StorageResolver`，並以 ``resolver=`` 傳給 ``File`` 與 +``Storage``。 + +掛載是依 URI 的文字比對。因此把有根目錄的後端掛在某個 ``local://`` 路徑上，只會 +導向該路徑的那一種寫法；同一個目錄的另一種寫法（例如指向它的符號連結）仍然會 +連到整個檔案系統。要限制不受信任的路徑，請像上面那樣給有根目錄的後端一個專屬的 +scheme 或 authority，並且只接受其下的 URI。 + +撰寫後端 +-------- + +繼承 :class:`~automation_file.StorageBackend` 並實作基本操作即可。上述公開方法 +都由基底類別提供：它們會正規化路徑、檢查既有內容、拋出共用的例外並建立上層目錄。 + +.. code-block:: python + + from automation_file import FileInfo, StorageBackend, StorageCapabilities + + class VaultStorage(StorageBackend): + scheme = "vault" + capabilities = StorageCapabilities(directories=False, etag=True) + + def _stat(self, path): ... # FileInfo；不存在時回傳 None；"" 是根目錄 + def _list_dir(self, path): ... # 目錄的直接子項目 + def _upload(self, source, path): ... + def _download(self, path, target): ... + def _delete_file(self, path): ... + # 目錄真實存在時（capabilities.directories=True）還需要： + # _mkdir(path)、_rmdir(path) + # 可選擇覆寫：_walk、_copy_from、_move_from、_checksum、_read_bytes + +請用契約測試套件檢查。``tests/storage_contract.py`` 包含 70 個案例——巢狀目錄、 +空檔與大檔、Unicode 路徑、二進位資料、覆寫與路徑不存在時的行為、路徑正規化、 +複製與搬移——並在後端確實有差異之處讀取 ``capabilities``： + +.. code-block:: python + + import pytest + from tests.storage_contract import StorageContract + + class TestVaultStorageContract(StorageContract): + @pytest.fixture + def backend(self): + return VaultStorage(...) # 每個測試都要是空的儲存 diff --git a/docs/source/Zh-TW/zh_tw_index.rst b/docs/source/Zh-TW/zh_tw_index.rst index ddcbf3f..3507308 100644 --- a/docs/source/Zh-TW/zh_tw_index.rst +++ b/docs/source/Zh-TW/zh_tw_index.rst @@ -237,3 +237,18 @@ Slack、Email（SMTP）、Discord、Telegram、Microsoft Teams、PagerDuty :caption: 外掛 usage/plugins + +.. _zh-tw-storage: + +第 16 章 — 通用儲存層 +===================== + +``File`` 與 ``Storage`` 以同一套 URI 語法存取本機與遠端儲存； +``StorageBackend`` 是後端要實作的唯一契約，提供共用的操作、共用的例外， +以及可重複使用的契約測試套件。 + +.. toctree:: + :maxdepth: 2 + :caption: 通用儲存層 + + usage/storage \ No newline at end of file diff --git a/docs/source/index.rst b/docs/source/index.rst index 74b40f5..aecb821 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -24,9 +24,9 @@ The documentation is split by language and by content type. Each language manual is organised into chapters (Getting Started, CLI, Architecture, Local Operations, HTTP Transfers, Cloud and SFTP Backends, Action Servers, MCP Server, GUI, Reliability, Triggers and Scheduler, Notifications, -Configuration, DAG, Plugins); the API book holds the auto-generated Python -reference for every public module. Pick a language from the table of -contents on the left, or jump straight to a section below. +Configuration, DAG, Plugins, Universal Storage Layer); the API book holds the +auto-generated Python reference for every public module. Pick a language from +the table of contents on the left, or jump straight to a section below. .. toctree:: :maxdepth: 2 diff --git a/docs/updates/2026-10.md b/docs/updates/2026-10.md index ac8c919..52f22ff 100644 --- a/docs/updates/2026-10.md +++ b/docs/updates/2026-10.md @@ -192,3 +192,27 @@ Index and query commands: [README.md](README.md). New entries go at the end. - **Docs**: `CLAUDE.md` › Branching & CI and `architecture.md` §2 and §3 say how the jobs build and what to regenerate when the build requirement changes. The READMEs and the Sphinx pages do not describe the publish jobs, so they are unchanged. - **Files**: `.github/requirements/publish.in`, `.github/requirements/publish.txt`, `.github/workflows/ci-dev.yml`, `.github/workflows/publish.yml`, `tests/test_workflow_actions.py`, `tests/test_dev_release.py`, `CLAUDE.md`, `architecture.md`, `progress.md`. - **Open items**: none. `publish.yml` runs only on `main`, so its build step takes effect, and is first exercised, when `dev` is merged. + +## U-20261008-01 · 2026-10-08 · Universal storage layer: contract, URIs, local and memory backends · #storage #roadmap #tests + +- **What**: the first stage of the 1.0 roadmap (`docs/FILEAUTOMATION-1.0-ROADMAP.md`, PR #107, not on `dev` yet). It covers the roadmap's "Define StorageBackend contract", "Define URI specification", "URI resolver", "File abstraction", "Local migration", "Storage contract suite" and "Local tests". New package `automation_file/storage/`, exported from the facade (`from automation_file import File, Storage`): + - `StorageURI` / `parse_storage_uri`: `:///`. The scheme is lower-cased (`file` is `local`, `az` is `azure`); the path is literal, with no percent-decoding; a `..` segment, a NUL, and `user@` or `user:password@` in the authority are refused, and that last message does not repeat the credentials. Text without `://` is a local path, made absolute; a Windows UNC path becomes `local://server/share/...`. Scheme-like text without `//` (`sftp:/data/a.csv`) is refused as ambiguous, with both unambiguous spellings in the message. + - `StorageBackend`: `exists`, `stat`, `list_dir`, `mkdir`, `upload`, `download`, `delete`, `checksum`, `read_bytes`, `write_bytes`, `copy_from`, `move_from` are template methods over the primitives a backend supplies (`_stat`, `_list_dir`, `_upload`, `_download`, `_delete_file`, and `_mkdir` / `_rmdir` where directories are real). `download` writes a `.part` file and replaces the target, `delete` refuses the storage root and needs `recursive=True` for a directory with entries, `checksum` takes any fixed-length `hashlib` algorithm. The listing method is `list_dir`, not `list`, which would shadow the builtin inside the class. + - `FileInfo`, `Checksum` (constant-time `matches`), `StorageCapabilities`: frozen, with `to_dict()` for JSON. + - `LocalStorage`: the whole filesystem, or one tree through `safe_join` when given a root. Writes go through a sibling temporary file and `os.replace`. Deleting removes a symbolic link and leaves its target; a recursive delete is `shutil.rmtree`; a recursive listing does not descend into linked directories. + - `MemoryStorage` (`memory:///...`): for tests and dry runs, and the second implementation that shows the contract suite is not tied to a filesystem. + - `StorageResolver` / `default_resolver`: mounts first (the longest one at or above the URI), then the scheme's factory. `Storage.mount("sandbox://jobs", LocalStorage(root))` is the documented way to confine untrusted paths. + - `File` (one file) and `Storage` (one directory, plus the static methods on the process-wide table). Both resolve the backend on every call. + - Ten exceptions in `exceptions.py`: `StorageException` and `StorageURIException`, `StorageNotFoundException` (also a `FileNotExistsException`), `StorageAlreadyExistsException`, `StoragePathTypeException`, `StorageNotEmptyException`, `StoragePermissionException`, `StorageTransientException`, `StorageUnavailableException`, `StorageUnsupportedException`. + - Nothing existing changed: no `FA_*` action, no backend function, no dependency. The layer is not registered in the action registry. +- **Tests**: 315 new cases. + - `tests/storage_contract.py`: 70 cases a backend must pass (57 test functions), run for `LocalStorage` and `MemoryStorage`: nested directories, empty and 5 MiB files, four Unicode paths, binary data, overwrite on and off, missing paths, file against directory, path spellings, `..`, the root, checksums against `hashlib`, copy and move. + - `test_storage_local.py`, `test_storage_memory.py`, `test_storage_uri.py`, `test_storage_resolver.py`, `test_storage_file.py`, `test_storage_types.py`, and `test_storage_imports.py`, which reads the module-level imports of the package and fails on anything outside the standard library, `exceptions`, `core.checksum` and `local.safe_paths`. +- **Result / numbers**: `ruff check` and `ruff format --check` pass; `mypy automation_file` finds no issues in 169 files. `pytest tests/`: 1141 passed, 8 skipped, 11 failed, on Python 3.14.7 on Windows. The same 11 fail on `a0dd11f` without this change (834 passed, 11 failed, in a separate worktree with the same environment), and none of them touches this code: + - 6 in `test_data_ops_yaml_parquet.py`: `pyarrow` was not installed in the environment of this run; + - 5 in `test_versioning.py`: `WinError 206` on this machine's long temp path (`progress.md` #28). +- **Not verified**: 5 of the 8 skips are the symbolic-link tests of `test_storage_local.py`; this machine may not create links, so that code has not run (`progress.md` #18). The other three are a POSIX-only path test, a Windows-only expectation, and a content-type case `MemoryStorage` does not support. `PySide6` was only partly installed in the environment (path-length limit); the 18 UI smoke tests passed with it. +- **Docs**: a "Universal Storage Layer" chapter in the three manuals (`docs/source/Eng`, `Zh-TW`, `Zh-CN` `usage/storage.rst`, chapter 16 of each index) and an API page (`docs/source/API/storage.rst`, chapter M); a feature bullet, three diagram nodes and a usage section in `README.md`, `README.zh-TW.md` and `README.zh-CN.md`; `architecture.md` §1 to §5, §7 and §8; `CLAUDE.md` (the package map, key types, and a Storage layer section under Security). +- **Files**: `automation_file/storage/` (9 modules), `automation_file/exceptions.py`, `automation_file/__init__.py`, `tests/storage_contract.py`, `tests/test_storage_*.py` (7 files), the documentation above, `progress.md`. +- **Evidence**: branch `feat/universal-storage-layer`, based on `a0dd11f`. +- **Open items**: `progress.md` #10 to #26 (the rest of the roadmap), #27 and #28 (found on the way). diff --git a/docs/updates/README.md b/docs/updates/README.md index 5308af4..5f8cd29 100644 --- a/docs/updates/README.md +++ b/docs/updates/README.md @@ -58,6 +58,7 @@ In the same commit: delete the item from `progress.md`, add a `#done` entry here | ID | Date | Title | Tags | Batch | |---|---|---|---|---| +| U-20261008-01 | 2026-10-08 | Universal storage layer: contract, URIs, local and memory | #storage #roadmap #tests | [2026-10](2026-10.md) | | U-20261001-11 | 2026-10-01 | The publish jobs build with the locked setuptools | #done #ci #security #X-13 | [2026-10](2026-10.md) | | U-20261001-10 | 2026-10-01 | The publish jobs install hash-locked build tools | #done #ci #security #deps | [2026-10](2026-10.md) | | U-20261001-09 | 2026-10-01 | The source distributions stop carrying the tests | #done #packaging #tests | [2026-10](2026-10.md) | @@ -95,5 +96,5 @@ In the same commit: delete the item from `progress.md`, add a `#done` entry here | File | Period | Entries | |---|---|---:| -| [2026-10.md](2026-10.md) | 2026-10 | 11 | +| [2026-10.md](2026-10.md) | 2026-10 | 12 | | [2026-09.md](2026-09.md) | 2026-09 | 21 | diff --git a/progress.md b/progress.md index d683eef..440a47e 100644 --- a/progress.md +++ b/progress.md @@ -5,3 +5,39 @@ Item numbers (`#n`) are never reused. Tags: [DECIDE] needs the owner's decision, Cross-repo and workspace items live in `D:\Codes\progress.md` (relevant here: X-12, X-13). ## Open + +Items #10 to #26 are what is left of the 1.0 roadmap (`docs/FILEAUTOMATION-1.0-ROADMAP.md`, PR #107, not on `dev` yet), in the order its §20 recommends. The storage core is done (U-20261008-01). + +### Architecture and packaging (roadmap M1) + +- **#10** [DECIDE] Optional extras. Roadmap §3 wants a light base install with the SDKs and the GUI under `[s3]`, `[gdrive]`, `[azure]`, `[dropbox]`, `[sftp]`, `[ftp]`, `[webdav]`, `[smb]`, `[fsspec]`, `[gui]`, `[all]`, `[test]`. `CLAUDE.md` › Branching & CI and `architecture.md` §7 say the opposite: the backends and PySide6 are first-class runtime dependencies and must not move under extras. PyBreeze declares `automation-file` and gets the GUI and the SDKs through it (`architecture.md` §6), so moving them breaks it unless it asks for `[all]` in the same round. Decide which rule wins and whether `automation_file_dev` follows. +- **#11** [DECIDE] Public API policy and deprecation policy (roadmap M1, M9): which names are frozen at 1.0 and how a name is retired. The storage layer is documented as provisional until then. +- **#12** `import automation_file` loads `requests`, `cryptography`, `watchdog`, `googleapiclient`, `google.auth`, `google_auth_oauthlib`, `prometheus_client`, `opentelemetry`, `tqdm` and `defusedxml` at module level, because the facade imports every module. Roadmap §3 requires that the base package imports no optional SDK at import time. `boto3`, `azure`, `dropbox`, `paramiko`, `PySide6`, `pyarrow`, `msal` and `boxsdk` are already lazy. What counts as optional depends on #10. + +### Universal storage layer (roadmap M2) + +- **#13** Storage adapters over the existing clients, each in `storage/_storage.py` with a `StorageContract` class against a fake client: S3 (`s3://bucket/key`), Azure Blob (`azure://container/blob`), Dropbox (`dropbox:///path`), SFTP (`sftp://host/path`), FTP and FTPS, WebDAV, SMB (`smb://server/share/path`), fsspec. The object stores set `capabilities.directories=False` and override `_walk` with one flat listing. A session backend must refuse a URI whose host is not the one it is connected to; `SFTPClient` does not keep its host today. `tests/test_storage_imports.py` then needs the client modules on its allowlist. +- **#14** Google Drive adapter (`gdrive://`). Drive addresses files by ID and allows two files of one name in a folder, so the path-to-ID lookup and the duplicate-name rule have to be designed first. +- **#15** [DECIDE] The eleventh backend slot, and whether OneDrive and Box are promoted to the storage contract or documented as action-only (roadmap §4). +- **#16** Cross-backend operations through the layer: `copy_between` / `FA_copy_between` on `File.copy_to`, and `FA_storage_*` actions. `copy_between` accepts `local:`, `sftp:/path`, `s3:bucket/key` and http(s) sources today; `parse_storage_uri` rejects the first three as ambiguous, so the action needs a translation step to stay compatible. +- **#17** Streams and directory trees: `open()` or chunked reads and writes (`read_bytes` holds the whole file in memory, and the default `checksum` stages a full local copy of a remote file), directory copy and sync through the layer, and a backend's native checksum where it has one. +- **#18** [UNVERIFIED] The five symbolic-link tests of `tests/test_storage_local.py` have not run anywhere: the development machine may not create links (they skip there), CI's Windows runners may. Read the first CI run of the branch and fix `LocalStorage` if one fails. + +### Backend integration tests (roadmap M3) + +- **#19** Integration environments for the contract suite in CI: MinIO, Azurite, SFTP, FTP/FTPS, WebDAV and Samba, plus credential-gated jobs for the cloud adapters, and Linux and macOS legs (`ci-dev.yml` runs pytest on Windows only). Needs #13. +- **#20** Failure cases in the contract suite: access denied, transient failures mapped to `StorageTransientException` and retried, metadata kept where the backend supports it. Only `LocalStorage` has permission-error tests today (`tests/test_storage_local.py`). + +### Later milestones + +- **#21** IntegrityMonitor 2.0 (roadmap §6, M4): snapshot and manifest schema with a version, baseline management, change detection over `FileInfo`, watch and continuous modes, alert and audit hooks, opt-in remediation. `core/fim.py` and `core/manifest.py` are the starting point; build it on the storage layer. +- **#22** Pipeline runtime (roadmap §7, M5): `Pipeline` domain model, DAG runtime v2 with retry, timeout, cancellation, conditions, idempotency, checkpoint and resume, dry run, execution history, and versioned YAML/JSON definitions with schema validation. `core/dag_executor.py` is the starting point. +- **#23** Scheduler, events, notifications and audit (roadmap §8 to §10, M6): one scheduler with cron (time-zone aware), manual, file-event, webhook and pipeline-dependency triggers; an event model that drives a `NotificationRouter`; audit schema v2 with correlation IDs behind a storage interface. +- **#24** UI 2.0 (roadmap §11, M7). Not before the APIs of #13 to #23 are stable (roadmap §20). +- **#25** Semantic MCP tools (roadmap §12, M8): `file_*`, `storage_*`, `pipeline_*`, `integrity_status`, `audit_search`, with a permission model and dry run, next to the existing `FA_*` bridge. +- **#26** Release engineering and 1.0 (roadmap §13, M9): contract and integration tests in the PR gate, PyPI Trusted Publishing, SemVer, migration guide, API freeze. + +### Found on the way + +- **#27** The three `usage/cloud.rst` pages (`docs/source/Eng`, `Zh-TW`, `Zh-CN`) show a `FA_cross_copy` action with `src` / `dst` and a `drive://` prefix. Neither exists: the action is `FA_copy_between(source, target)` and `remote/cross_backend.py` has no `drive` scheme. The READMEs were corrected in `bc13101`; these pages were not. Fix them with #16, which rewrites that section anyway. +- **#28** Five tests of `tests/test_versioning.py` fail on a machine with a long temp directory (`FileNotFoundError: [WinError 206]`, the file name or extension is too long): `FileVersioner` names a version directory after the whole source path (`C__sep__Users__sep__...`), so the path roughly doubles and passes Windows' 260-character limit. Seen on `a0dd11f` before any change of this branch; the suite passes where the temp path is shorter (U-20261001-11). A real source file with a long path fails the same way. diff --git a/tests/storage_contract.py b/tests/storage_contract.py new file mode 100644 index 0000000..a6541c7 --- /dev/null +++ b/tests/storage_contract.py @@ -0,0 +1,456 @@ +"""The contract every storage backend must pass. + +Subclass :class:`StorageContract` in a ``test_*.py`` module and provide a +``backend`` fixture that returns an empty :class:`StorageBackend`: + +.. code-block:: python + + class TestMyStorageContract(StorageContract): + @pytest.fixture + def backend(self) -> StorageBackend: + return MyStorage(...) + +The suite reads ``backend.capabilities`` to pick the expected behaviour where +backends legitimately differ (real directories versus implied ones, optional +``FileInfo`` fields). Everything else is the same for every backend. +""" + +from __future__ import annotations + +import hashlib +from datetime import timedelta +from pathlib import Path + +import pytest + +from automation_file.exceptions import ( + FileNotExistsException, + StorageAlreadyExistsException, + StorageException, + StorageNotEmptyException, + StorageNotFoundException, + StoragePathTypeException, + StorageUnsupportedException, + StorageURIException, +) +from automation_file.storage import StorageBackend + +BINARY = bytes(range(256)) * 4 +LARGE_SIZE = 5 * 1024 * 1024 + 123 +UNICODE_PATHS = [ + "資料/報告 2026 ✓.txt", + "données/été.bin", + "ファイル/メモ.md", + "with space/and (parens) & more.txt", +] + + +def _local_file(directory: Path, data: bytes, name: str = "source.bin") -> Path: + path = directory / name + path.write_bytes(data) + return path + + +class StorageContract: + """Behaviour shared by every backend. Not collected on its own.""" + + @pytest.fixture + def backend(self) -> StorageBackend: + raise NotImplementedError("the contract subclass provides the backend fixture") + + # ------------------------------------------------------------------ exists / stat + + def test_root_exists_and_is_a_directory(self, backend: StorageBackend) -> None: + assert backend.exists("") is True + assert backend.stat("").is_dir is True + + def test_missing_path_does_not_exist(self, backend: StorageBackend) -> None: + assert backend.exists("nope.txt") is False + + def test_stat_of_a_missing_path_raises_not_found(self, backend: StorageBackend) -> None: + with pytest.raises(StorageNotFoundException): + backend.stat("nope.txt") + + def test_not_found_is_also_the_legacy_file_not_exists(self, backend: StorageBackend) -> None: + with pytest.raises(FileNotExistsException): + backend.stat("nope.txt") + + def test_stat_describes_a_file(self, backend: StorageBackend) -> None: + info = backend.write_bytes("dir/report.txt", b"hello world") + assert info == backend.stat("dir/report.txt") + assert info.path == "dir/report.txt" + assert info.name == "report.txt" + assert info.is_dir is False + assert info.size == 11 + + def test_stat_reports_an_aware_modification_time(self, backend: StorageBackend) -> None: + if not backend.capabilities.modified_at: + pytest.skip("backend does not report modification times") + modified = backend.write_bytes("a.txt", b"x").modified_at + assert modified is not None + assert modified.utcoffset() == timedelta(0) + + def test_stat_reports_a_content_type_where_supported(self, backend: StorageBackend) -> None: + if not backend.capabilities.content_type: + pytest.skip("backend does not report content types") + assert backend.write_bytes("notes.txt", b"x").content_type == "text/plain" + + def test_file_info_is_json_friendly(self, backend: StorageBackend) -> None: + document = backend.write_bytes("a.txt", b"x").to_dict() + assert document["path"] == "a.txt" + assert document["is_dir"] is False + assert document["size"] == 1 + + # ------------------------------------------------------------------ upload / download + + def test_upload_then_download_round_trips_binary_data( + self, backend: StorageBackend, tmp_path: Path + ) -> None: + info = backend.upload(_local_file(tmp_path, BINARY), "data.bin") + assert info.size == len(BINARY) + target = backend.download("data.bin", tmp_path / "out" / "copy.bin") + assert target == tmp_path / "out" / "copy.bin" + assert target.read_bytes() == BINARY + + def test_upload_creates_missing_parent_directories( + self, backend: StorageBackend, tmp_path: Path + ) -> None: + backend.upload(_local_file(tmp_path, b"deep"), "a/b/c/d.txt") + assert backend.read_bytes("a/b/c/d.txt") == b"deep" + assert backend.stat("a/b/c").is_dir is True + assert backend.stat("a").is_dir is True + + def test_empty_file_round_trips(self, backend: StorageBackend, tmp_path: Path) -> None: + info = backend.upload(_local_file(tmp_path, b""), "empty.bin") + assert info.size == 0 + assert backend.read_bytes("empty.bin") == b"" + assert backend.download("empty.bin", tmp_path / "empty.out").read_bytes() == b"" + + def test_large_file_round_trips(self, backend: StorageBackend, tmp_path: Path) -> None: + data = bytes(range(251)) * (LARGE_SIZE // 251 + 1) + data = data[:LARGE_SIZE] + info = backend.upload(_local_file(tmp_path, data), "large.bin") + assert info.size == LARGE_SIZE + digest = hashlib.sha256(data).hexdigest() + assert backend.checksum("large.bin").value == digest + downloaded = backend.download("large.bin", tmp_path / "large.out") + assert hashlib.sha256(downloaded.read_bytes()).hexdigest() == digest + + @pytest.mark.parametrize("path", UNICODE_PATHS) + def test_unicode_paths_round_trip(self, backend: StorageBackend, path: str) -> None: + backend.write_bytes(path, BINARY) + assert backend.exists(path) is True + assert backend.read_bytes(path) == BINARY + directory, _, name = path.rpartition("/") + assert [info.name for info in backend.list_dir(directory)] == [name] + + def test_upload_overwrites_by_default(self, backend: StorageBackend) -> None: + backend.write_bytes("a.txt", b"first") + info = backend.write_bytes("a.txt", b"second, longer") + assert info.size == 14 + assert backend.read_bytes("a.txt") == b"second, longer" + + def test_upload_without_overwrite_keeps_the_existing_file( + self, backend: StorageBackend, tmp_path: Path + ) -> None: + backend.write_bytes("a.txt", b"first") + with pytest.raises(StorageAlreadyExistsException): + backend.upload(_local_file(tmp_path, b"second"), "a.txt", overwrite=False) + assert backend.read_bytes("a.txt") == b"first" + + def test_upload_of_a_missing_local_file_raises_not_found( + self, backend: StorageBackend, tmp_path: Path + ) -> None: + with pytest.raises(StorageNotFoundException): + backend.upload(tmp_path / "missing.bin", "a.bin") + assert backend.exists("a.bin") is False + + def test_upload_onto_a_directory_is_refused( + self, backend: StorageBackend, tmp_path: Path + ) -> None: + backend.write_bytes("dir/a.txt", b"x") + with pytest.raises(StoragePathTypeException): + backend.upload(_local_file(tmp_path, b"y"), "dir") + + def test_upload_onto_the_root_is_refused(self, backend: StorageBackend, tmp_path: Path) -> None: + with pytest.raises(StoragePathTypeException): + backend.upload(_local_file(tmp_path, b"y"), "") + + def test_download_of_a_missing_file_raises_not_found( + self, backend: StorageBackend, tmp_path: Path + ) -> None: + target = tmp_path / "out.bin" + with pytest.raises(StorageNotFoundException): + backend.download("nope.bin", target) + assert not target.exists() + + def test_download_of_a_directory_is_refused( + self, backend: StorageBackend, tmp_path: Path + ) -> None: + backend.write_bytes("dir/a.txt", b"x") + with pytest.raises(StoragePathTypeException): + backend.download("dir", tmp_path / "out.bin") + + def test_download_overwrites_a_local_file_by_default( + self, backend: StorageBackend, tmp_path: Path + ) -> None: + backend.write_bytes("a.txt", b"remote") + target = _local_file(tmp_path, b"local", "target.txt") + backend.download("a.txt", target) + assert target.read_bytes() == b"remote" + + def test_download_without_overwrite_keeps_the_local_file( + self, backend: StorageBackend, tmp_path: Path + ) -> None: + backend.write_bytes("a.txt", b"remote") + target = _local_file(tmp_path, b"local", "target.txt") + with pytest.raises(StorageAlreadyExistsException): + backend.download("a.txt", target, overwrite=False) + assert target.read_bytes() == b"local" + + def test_download_leaves_no_partial_file_behind( + self, backend: StorageBackend, tmp_path: Path + ) -> None: + backend.write_bytes("a.txt", b"remote") + backend.download("a.txt", tmp_path / "out" / "a.txt") + assert [entry.name for entry in (tmp_path / "out").iterdir()] == ["a.txt"] + + def test_read_of_a_missing_file_raises_not_found(self, backend: StorageBackend) -> None: + with pytest.raises(StorageNotFoundException): + backend.read_bytes("nope.bin") + + # ------------------------------------------------------------------ list_dir + + def test_list_dir_of_an_empty_root_is_empty(self, backend: StorageBackend) -> None: + assert backend.list_dir() == [] + + def test_list_dir_returns_immediate_children_sorted(self, backend: StorageBackend) -> None: + for path in ("b.txt", "a.txt", "sub/c.txt", "sub/deeper/d.txt"): + backend.write_bytes(path, b"x") + listing = backend.list_dir("") + assert [info.path for info in listing] == ["a.txt", "b.txt", "sub"] + assert [info.is_dir for info in listing] == [False, False, True] + assert [info.path for info in backend.list_dir("sub")] == ["sub/c.txt", "sub/deeper"] + + def test_list_dir_recursive_returns_every_descendant(self, backend: StorageBackend) -> None: + for path in ("b.txt", "sub/c.txt", "sub/deeper/d.txt"): + backend.write_bytes(path, b"xy") + listing = backend.list_dir("", recursive=True) + assert [info.path for info in listing] == [ + "b.txt", + "sub", + "sub/c.txt", + "sub/deeper", + "sub/deeper/d.txt", + ] + assert [info.size for info in listing if not info.is_dir] == [2, 2, 2] + assert [info.path for info in backend.list_dir("sub", recursive=True)] == [ + "sub/c.txt", + "sub/deeper", + "sub/deeper/d.txt", + ] + + def test_list_dir_of_a_missing_directory_raises_not_found( + self, backend: StorageBackend + ) -> None: + with pytest.raises(StorageNotFoundException): + backend.list_dir("nope") + + def test_list_dir_of_a_file_is_refused(self, backend: StorageBackend) -> None: + backend.write_bytes("a.txt", b"x") + with pytest.raises(StoragePathTypeException): + backend.list_dir("a.txt") + + # ------------------------------------------------------------------ mkdir + + def test_mkdir_creates_a_directory_where_directories_are_real( + self, backend: StorageBackend + ) -> None: + backend.mkdir("new/nested") + if backend.capabilities.directories: + assert backend.stat("new/nested").is_dir is True + assert [info.path for info in backend.list_dir("new")] == ["new/nested"] + else: + assert backend.exists("new/nested") is False + + def test_mkdir_of_an_existing_directory_is_fine_by_default( + self, backend: StorageBackend + ) -> None: + backend.write_bytes("dir/a.txt", b"x") + backend.mkdir("dir") + assert backend.read_bytes("dir/a.txt") == b"x" + + def test_mkdir_with_exist_ok_off_refuses_an_existing_directory( + self, backend: StorageBackend + ) -> None: + backend.write_bytes("dir/a.txt", b"x") + with pytest.raises(StorageAlreadyExistsException): + backend.mkdir("dir", exist_ok=False) + + def test_mkdir_over_a_file_is_refused(self, backend: StorageBackend) -> None: + backend.write_bytes("a.txt", b"x") + with pytest.raises(StoragePathTypeException): + backend.mkdir("a.txt") + + def test_mkdir_without_parents_needs_the_parent(self, backend: StorageBackend) -> None: + if not backend.capabilities.directories: + pytest.skip("directories are implied on this backend") + with pytest.raises(StorageNotFoundException): + backend.mkdir("missing/child", parents=False) + assert backend.exists("missing") is False + + def test_a_file_cannot_be_created_below_a_file(self, backend: StorageBackend) -> None: + if not backend.capabilities.directories: + pytest.skip("directories are implied on this backend") + backend.write_bytes("a.txt", b"x") + with pytest.raises(StoragePathTypeException): + backend.write_bytes("a.txt/child.txt", b"y") + + # ------------------------------------------------------------------ delete + + def test_delete_removes_a_file(self, backend: StorageBackend) -> None: + backend.write_bytes("a.txt", b"x") + backend.delete("a.txt") + assert backend.exists("a.txt") is False + + def test_delete_of_a_missing_path_raises_not_found(self, backend: StorageBackend) -> None: + with pytest.raises(StorageNotFoundException): + backend.delete("nope.txt") + + def test_delete_with_missing_ok_ignores_a_missing_path(self, backend: StorageBackend) -> None: + backend.delete("nope.txt", missing_ok=True) + + def test_delete_refuses_a_directory_with_entries(self, backend: StorageBackend) -> None: + backend.write_bytes("dir/a.txt", b"x") + with pytest.raises(StorageNotEmptyException): + backend.delete("dir") + assert backend.read_bytes("dir/a.txt") == b"x" + + def test_delete_recursive_removes_the_whole_tree(self, backend: StorageBackend) -> None: + for path in ("dir/a.txt", "dir/sub/b.txt", "dir/sub/deeper/c.txt", "keep.txt"): + backend.write_bytes(path, b"x") + backend.delete("dir", recursive=True) + assert backend.exists("dir") is False + assert [info.path for info in backend.list_dir("", recursive=True)] == ["keep.txt"] + + def test_delete_removes_an_empty_directory(self, backend: StorageBackend) -> None: + if not backend.capabilities.directories: + pytest.skip("an empty directory cannot exist on this backend") + backend.mkdir("empty") + backend.delete("empty") + assert backend.exists("empty") is False + + def test_delete_refuses_the_root(self, backend: StorageBackend) -> None: + backend.write_bytes("a.txt", b"x") + with pytest.raises(StorageUnsupportedException): + backend.delete("", recursive=True) + assert backend.read_bytes("a.txt") == b"x" + + # ------------------------------------------------------------------ checksum + + @pytest.mark.parametrize("algorithm", ["sha256", "sha512", "blake2b", "md5"]) + def test_checksum_matches_hashlib(self, backend: StorageBackend, algorithm: str) -> None: + backend.write_bytes("data.bin", BINARY) + checksum = backend.checksum("data.bin", algorithm) + assert checksum.algorithm == algorithm + assert checksum.value == hashlib.new(algorithm, BINARY).hexdigest() + + def test_checksum_defaults_to_sha256(self, backend: StorageBackend) -> None: + backend.write_bytes("data.bin", BINARY) + checksum = backend.checksum("data.bin") + assert str(checksum) == f"sha256:{hashlib.sha256(BINARY).hexdigest()}" + assert checksum.matches(hashlib.sha256(BINARY).hexdigest().upper()) is True + + @pytest.mark.parametrize("algorithm", ["no-such-hash", "shake_128", ""]) + def test_checksum_refuses_an_unusable_algorithm( + self, backend: StorageBackend, algorithm: str + ) -> None: + backend.write_bytes("data.bin", BINARY) + with pytest.raises(StorageUnsupportedException): + backend.checksum("data.bin", algorithm) + + def test_checksum_of_a_missing_file_raises_not_found(self, backend: StorageBackend) -> None: + with pytest.raises(StorageNotFoundException): + backend.checksum("nope.bin") + + def test_checksum_of_a_directory_is_refused(self, backend: StorageBackend) -> None: + backend.write_bytes("dir/a.txt", b"x") + with pytest.raises(StoragePathTypeException): + backend.checksum("dir") + + # ------------------------------------------------------------------ paths + + @pytest.mark.parametrize( + "spelling", ["/dir/a.txt", "dir//a.txt", "./dir/./a.txt", "dir/a.txt/"] + ) + def test_equivalent_spellings_name_the_same_file( + self, backend: StorageBackend, spelling: str + ) -> None: + backend.write_bytes("dir/a.txt", b"x") + assert backend.stat(spelling).path == "dir/a.txt" + + @pytest.mark.parametrize("path", ["../escape.txt", "dir/../../escape.txt", "dir/.."]) + def test_parent_segments_are_rejected(self, backend: StorageBackend, path: str) -> None: + with pytest.raises(StorageURIException): + backend.exists(path) + with pytest.raises(StorageURIException): + backend.write_bytes(path, b"x") + + # ------------------------------------------------------------------ copy / move + + def test_copy_from_the_same_backend(self, backend: StorageBackend) -> None: + backend.write_bytes("a.txt", BINARY) + info = backend.copy_from(backend, "a.txt", "copies/b.txt") + assert info.path == "copies/b.txt" + assert backend.read_bytes("copies/b.txt") == BINARY + assert backend.read_bytes("a.txt") == BINARY + + def test_copy_from_respects_overwrite(self, backend: StorageBackend) -> None: + backend.write_bytes("a.txt", b"new") + backend.write_bytes("b.txt", b"old") + with pytest.raises(StorageAlreadyExistsException): + backend.copy_from(backend, "a.txt", "b.txt", overwrite=False) + assert backend.read_bytes("b.txt") == b"old" + backend.copy_from(backend, "a.txt", "b.txt") + assert backend.read_bytes("b.txt") == b"new" + + def test_copy_onto_itself_is_refused(self, backend: StorageBackend) -> None: + backend.write_bytes("a.txt", b"x") + with pytest.raises(StorageException): + backend.copy_from(backend, "a.txt", "/a.txt") + assert backend.read_bytes("a.txt") == b"x" + + def test_copy_of_a_missing_file_raises_not_found(self, backend: StorageBackend) -> None: + with pytest.raises(StorageNotFoundException): + backend.copy_from(backend, "nope.txt", "b.txt") + assert backend.exists("b.txt") is False + + def test_copy_of_a_directory_is_refused(self, backend: StorageBackend) -> None: + backend.write_bytes("dir/a.txt", b"x") + with pytest.raises(StoragePathTypeException): + backend.copy_from(backend, "dir", "other") + + def test_move_from_the_same_backend(self, backend: StorageBackend) -> None: + backend.write_bytes("a.txt", BINARY) + info = backend.move_from(backend, "a.txt", "moved/b.txt") + assert info.path == "moved/b.txt" + assert backend.read_bytes("moved/b.txt") == BINARY + assert backend.exists("a.txt") is False + + def test_move_from_respects_overwrite(self, backend: StorageBackend) -> None: + backend.write_bytes("a.txt", b"new") + backend.write_bytes("b.txt", b"old") + with pytest.raises(StorageAlreadyExistsException): + backend.move_from(backend, "a.txt", "b.txt", overwrite=False) + assert backend.read_bytes("a.txt") == b"new" + assert backend.read_bytes("b.txt") == b"old" + backend.move_from(backend, "a.txt", "b.txt") + assert backend.read_bytes("b.txt") == b"new" + assert backend.exists("a.txt") is False + + # ------------------------------------------------------------------ lifecycle + + def test_backend_is_a_context_manager(self, backend: StorageBackend) -> None: + with backend as entered: + assert entered is backend + entered.write_bytes("a.txt", b"x") diff --git a/tests/test_storage_file.py b/tests/test_storage_file.py new file mode 100644 index 0000000..30b3745 --- /dev/null +++ b/tests/test_storage_file.py @@ -0,0 +1,386 @@ +"""File and Storage: the object API over the resolver, including cross-backend transfers.""" + +from __future__ import annotations + +import hashlib +from collections.abc import Iterator +from pathlib import Path + +import pytest + +import automation_file +from automation_file.exceptions import ( + StorageAlreadyExistsException, + StorageNotFoundException, + StoragePathTypeException, + StorageURIException, +) +from automation_file.storage import ( + Checksum, + File, + LocalStorage, + MemoryStorage, + Storage, + StorageResolver, + clear_memory_stores, + parse_storage_uri, +) + +PAYLOAD = bytes(range(256)) * 3 +SHA256 = hashlib.sha256(PAYLOAD).hexdigest() + + +@pytest.fixture(autouse=True) +def _fresh_stores() -> Iterator[None]: + clear_memory_stores() + yield + clear_memory_stores() + + +@pytest.fixture +def resolver() -> StorageResolver: + return StorageResolver() + + +@pytest.fixture +def remote(resolver: StorageResolver) -> MemoryStorage: + """A stand-in for a remote backend, mounted below ``vault://bucket/data``.""" + backend = MemoryStorage("remote") + resolver.mount("vault://bucket/data", backend) + return backend + + +# ---------------------------------------------------------------------- File + + +def test_creating_a_file_touches_nothing(resolver: StorageResolver) -> None: + file = File("nowhere://host/a.txt", resolver=resolver) + assert str(file) == "nowhere://host/a.txt" + assert repr(file) == "File('nowhere://host/a.txt')" + assert file.name == "a.txt" + assert file.uri == parse_storage_uri("nowhere://host/a.txt") + with pytest.raises(StorageURIException): + file.exists() + + +def test_write_then_read(resolver: StorageResolver) -> None: + file = File("memory://scratch/dir/a.bin", resolver=resolver) + info = file.write(PAYLOAD) + assert info.path == "dir/a.bin" + assert info.size == len(PAYLOAD) + assert file.read() == PAYLOAD + assert file.exists() is True + assert file.is_file() is True + assert file.is_dir() is False + + +def test_text_round_trips_as_utf8_by_default(resolver: StorageResolver) -> None: + file = File("memory://scratch/note.txt", resolver=resolver) + file.write("報告 ✓") + assert file.read() == "報告 ✓".encode() + assert file.read_text() == "報告 ✓" + file.write("été", encoding="latin-1") + assert file.read() == b"\xe9t\xe9" + assert file.read_text(encoding="latin-1") == "été" + + +def test_write_respects_overwrite(resolver: StorageResolver) -> None: + file = File("memory://scratch/a.txt", resolver=resolver) + file.write(b"first") + with pytest.raises(StorageAlreadyExistsException): + file.write(b"second", overwrite=False) + assert file.read() == b"first" + + +def test_metadata_properties_come_from_stat(resolver: StorageResolver, tmp_path: Path) -> None: + file = File(tmp_path / "notes.txt", resolver=resolver) + file.write(b"hello") + info = file.stat() + assert info.path == parse_storage_uri(tmp_path / "notes.txt").path + assert file.size == 5 + assert file.modified_at == info.modified_at + assert file.content_type == "text/plain" + assert file.etag is None + assert file.version is None + assert dict(file.metadata) == {} + + +def test_a_missing_file(resolver: StorageResolver) -> None: + file = File("memory://scratch/nope.txt", resolver=resolver) + assert file.exists() is False + assert file.is_file() is False + assert file.is_dir() is False + with pytest.raises(StorageNotFoundException): + file.stat() + with pytest.raises(StorageNotFoundException): + file.read() + with pytest.raises(StorageNotFoundException): + file.delete() + file.delete(missing_ok=True) + + +def test_a_directory_is_not_a_file(resolver: StorageResolver) -> None: + File("memory://scratch/dir/a.txt", resolver=resolver).write(b"x") + directory = File("memory://scratch/dir", resolver=resolver) + assert directory.exists() is True + assert directory.is_dir() is True + assert directory.is_file() is False + with pytest.raises(StoragePathTypeException): + directory.read() + with pytest.raises(StoragePathTypeException, match="recursive=True"): + directory.delete() + assert File("memory://scratch/dir/a.txt", resolver=resolver).read() == b"x" + + +def test_delete(resolver: StorageResolver) -> None: + file = File("memory://scratch/a.txt", resolver=resolver) + file.write(b"x") + file.delete() + assert file.exists() is False + + +def test_upload_from_and_download_to(resolver: StorageResolver, tmp_path: Path) -> None: + source = tmp_path / "source.bin" + source.write_bytes(PAYLOAD) + file = File("memory://scratch/up/a.bin", resolver=resolver) + assert file.upload_from(source).path == "up/a.bin" + target = file.download_to(tmp_path / "out" / "a.bin") + assert target.read_bytes() == PAYLOAD + with pytest.raises(StorageAlreadyExistsException): + file.download_to(target, overwrite=False) + with pytest.raises(StorageAlreadyExistsException): + file.upload_from(source, overwrite=False) + + +def test_checksum_and_verify(resolver: StorageResolver) -> None: + file = File("memory://scratch/a.bin", resolver=resolver) + file.write(PAYLOAD) + assert file.checksum() == Checksum("sha256", SHA256) + assert file.checksum("md5").value == hashlib.md5(PAYLOAD, usedforsecurity=False).hexdigest() + assert file.verify(SHA256) is True + assert file.verify(SHA256.upper()) is True + assert file.verify(f"sha256:{SHA256}") is True + assert file.verify(Checksum("sha256", SHA256)) is True + assert file.verify("0" * 64) is False + md5 = hashlib.md5(PAYLOAD, usedforsecurity=False).hexdigest() + assert file.verify(md5, algorithm="md5") is True + assert file.verify(f"md5:{md5}") is True + assert file.verify(md5) is False + + +def test_copy_to_another_backend( + resolver: StorageResolver, remote: MemoryStorage, tmp_path: Path +) -> None: + source = File(tmp_path / "report.csv", resolver=resolver) + source.write(PAYLOAD) + copy = source.copy_to("vault://bucket/data/2026/report.csv") + assert copy == File("vault://bucket/data/2026/report.csv", resolver=resolver) + assert remote.read_bytes("2026/report.csv") == PAYLOAD + assert source.read() == PAYLOAD + assert copy.checksum().matches(source.checksum()) is True + + +def test_copy_back_from_another_backend( + resolver: StorageResolver, remote: MemoryStorage, tmp_path: Path +) -> None: + remote.write_bytes("in/report.csv", PAYLOAD) + target = tmp_path / "downloads" / "report.csv" + File("vault://bucket/data/in/report.csv", resolver=resolver).copy_to(target) + assert target.read_bytes() == PAYLOAD + assert [entry.name for entry in target.parent.iterdir()] == ["report.csv"] + + +def test_copy_to_accepts_a_file_and_respects_overwrite(resolver: StorageResolver) -> None: + source = File("memory://one/a.txt", resolver=resolver) + target = File("memory://two/a.txt", resolver=resolver) + source.write(b"new") + target.write(b"old") + with pytest.raises(StorageAlreadyExistsException): + source.copy_to(target, overwrite=False) + assert target.read() == b"old" + assert source.copy_to(target) is target + assert target.read() == b"new" + + +def test_move_to_another_backend( + resolver: StorageResolver, remote: MemoryStorage, tmp_path: Path +) -> None: + source = File(tmp_path / "report.csv", resolver=resolver) + source.write(PAYLOAD) + moved = source.move_to("vault://bucket/data/report.csv") + assert moved.read() == PAYLOAD + assert remote.exists("report.csv") is True + assert source.exists() is False + + +def test_move_within_the_local_filesystem(resolver: StorageResolver, tmp_path: Path) -> None: + source = File(tmp_path / "a.txt", resolver=resolver) + source.write(b"x") + moved = source.move_to(tmp_path / "sub" / "b.txt") + assert (tmp_path / "sub" / "b.txt").read_bytes() == b"x" + assert not (tmp_path / "a.txt").exists() + assert moved.name == "b.txt" + + +def test_copy_of_a_missing_file_creates_nothing( + resolver: StorageResolver, remote: MemoryStorage +) -> None: + with pytest.raises(StorageNotFoundException): + File("memory://scratch/nope.txt", resolver=resolver).copy_to("vault://bucket/data/a.txt") + assert remote.list_dir() == [] + + +def test_copy_onto_a_directory_is_refused(resolver: StorageResolver, remote: MemoryStorage) -> None: + remote.write_bytes("dir/a.txt", b"x") + source = File("memory://scratch/a.txt", resolver=resolver) + source.write(b"y") + with pytest.raises(StoragePathTypeException): + source.copy_to("vault://bucket/data/dir") + + +def test_files_compare_by_uri(resolver: StorageResolver) -> None: + first = File("memory://scratch/a.txt", resolver=resolver) + assert first == File("memory://scratch//a.txt/") + assert first != File("memory://scratch/b.txt") + assert first != "memory://scratch/a.txt" + assert len({first, File("memory://scratch/a.txt")}) == 1 + + +def test_a_file_follows_a_backend_mounted_later(resolver: StorageResolver) -> None: + file = File("vault://later/a.txt", resolver=resolver) + with pytest.raises(StorageURIException): + file.exists() + resolver.mount("vault://later", MemoryStorage("later")) + file.write(b"x") + assert file.read() == b"x" + + +# ---------------------------------------------------------------------- Storage + + +def test_storage_paths_are_relative_to_its_uri( + resolver: StorageResolver, remote: MemoryStorage +) -> None: + for path in ("reports/2026/q1.csv", "reports/2026/q2.csv", "reports/2025/q4.csv", "other.txt"): + remote.write_bytes(path, b"xy") + reports = Storage("vault://bucket/data/reports", resolver=resolver) + assert str(reports) == "vault://bucket/data/reports" + assert repr(reports) == "Storage('vault://bucket/data/reports')" + assert reports.uri == parse_storage_uri("vault://bucket/data/reports") + assert reports.backend is remote + assert reports.capabilities == remote.capabilities + assert [info.path for info in reports.list_dir()] == ["2025", "2026"] + assert [info.path for info in reports.list_dir("2026")] == ["2026/q1.csv", "2026/q2.csv"] + assert [info.path for info in reports.list_dir(recursive=True)] == [ + "2025", + "2025/q4.csv", + "2026", + "2026/q1.csv", + "2026/q2.csv", + ] + assert reports.stat("2026/q1.csv").path == "2026/q1.csv" + assert reports.stat("2026/q1.csv").size == 2 + assert reports.stat().path == "" + assert reports.stat().is_dir is True + assert reports.exists("2026/q1.csv") is True + assert reports.exists("other.txt") is False + assert reports.exists() is True + + +def test_storage_file_returns_a_file_below_it( + resolver: StorageResolver, remote: MemoryStorage +) -> None: + reports = Storage("vault://bucket/data/reports", resolver=resolver) + file = reports.file("2026/q1.csv") + assert str(file) == "vault://bucket/data/reports/2026/q1.csv" + file.write(b"x") + assert remote.read_bytes("reports/2026/q1.csv") == b"x" + with pytest.raises(StorageURIException): + reports.file("../escape.txt") + + +def test_storage_upload_download_checksum_delete( + resolver: StorageResolver, remote: MemoryStorage, tmp_path: Path +) -> None: + source = tmp_path / "source.bin" + source.write_bytes(PAYLOAD) + storage = Storage("vault://bucket/data/in", resolver=resolver) + assert storage.upload(source, "a/b.bin").path == "a/b.bin" + assert remote.read_bytes("in/a/b.bin") == PAYLOAD + assert storage.checksum("a/b.bin") == Checksum("sha256", SHA256) + assert storage.download("a/b.bin", tmp_path / "out.bin").read_bytes() == PAYLOAD + storage.mkdir("empty") + assert storage.stat("empty").is_dir is True + storage.delete("a", recursive=True) + assert [info.path for info in storage.list_dir()] == ["empty"] + storage.delete("", recursive=True) + assert remote.exists("in") is False + storage.delete("gone", missing_ok=True) + + +def test_storage_over_a_local_directory(resolver: StorageResolver, tmp_path: Path) -> None: + (tmp_path / "work").mkdir() + storage = Storage(tmp_path / "work", resolver=resolver) + storage.file("a/b.txt").write(b"x") + assert (tmp_path / "work" / "a" / "b.txt").read_bytes() == b"x" + assert [info.path for info in storage.list_dir(recursive=True)] == ["a", "a/b.txt"] + assert isinstance(storage.backend, LocalStorage) + + +def test_storage_spanning_a_nested_mount_reports_the_asked_paths( + resolver: StorageResolver, remote: MemoryStorage +) -> None: + nested = MemoryStorage("nested") + resolver.mount("vault://bucket/data/hot", nested) + nested.write_bytes("a.txt", b"x") + data = Storage("vault://bucket/data", resolver=resolver) + assert [info.path for info in data.list_dir("hot")] == ["hot/a.txt"] + assert data.stat("hot/a.txt").path == "hot/a.txt" + assert remote.exists("hot") is False + + +def test_storages_compare_by_uri(resolver: StorageResolver) -> None: + assert Storage("memory://scratch/a", resolver=resolver) == Storage("memory://scratch/a/") + assert Storage("memory://scratch/a") != Storage("memory://scratch/b") + assert len({Storage("memory://scratch/a"), Storage("memory://scratch/a")}) == 1 + + +# ---------------------------------------------------------------------- facade + + +@pytest.mark.parametrize( + "name", + [ + "Checksum", + "File", + "FileInfo", + "LocalStorage", + "MemoryStorage", + "Storage", + "StorageBackend", + "StorageCapabilities", + "StorageResolver", + "StorageURI", + "parse_storage_uri", + "StorageException", + "StorageAlreadyExistsException", + "StorageNotEmptyException", + "StorageNotFoundException", + "StoragePathTypeException", + "StoragePermissionException", + "StorageTransientException", + "StorageURIException", + "StorageUnavailableException", + "StorageUnsupportedException", + ], +) +def test_the_facade_exports_the_storage_layer(name: str) -> None: + assert name in automation_file.__all__ + assert hasattr(automation_file, name) + + +def test_the_default_resolver_serves_local_files(tmp_path: Path) -> None: + file = automation_file.File(tmp_path / "a.txt") + file.write("hello") + assert (tmp_path / "a.txt").read_text(encoding="utf-8") == "hello" + assert automation_file.Storage(tmp_path).list_dir()[0].path == "a.txt" diff --git a/tests/test_storage_imports.py b/tests/test_storage_imports.py new file mode 100644 index 0000000..d0e4c8a --- /dev/null +++ b/tests/test_storage_imports.py @@ -0,0 +1,66 @@ +"""The storage layer stays independent of the registry, the GUI and the backend SDKs. + +It reads the module-level imports of every file under ``automation_file/storage``: +each one is either from the standard library or from the short list of first-party +modules below. An SDK imported lazily inside a function is not a module-level +import and stays allowed. +""" + +from __future__ import annotations + +import ast +import sys +from pathlib import Path + +import pytest + +STORAGE_PACKAGE = Path(__file__).resolve().parent.parent / "automation_file" / "storage" +ALLOWED_FIRST_PARTY = ( + "automation_file.exceptions", + "automation_file.core.checksum", + "automation_file.local.safe_paths", + "automation_file.storage", +) +MODULES = sorted(STORAGE_PACKAGE.glob("*.py")) + + +def _module_level_imports(path: Path) -> list[str]: + tree = ast.parse(path.read_text(encoding="utf-8"), filename=str(path)) + names: list[str] = [] + for node in tree.body: + if isinstance(node, ast.Import): + names.extend(alias.name for alias in node.names) + elif isinstance(node, ast.ImportFrom): + assert node.level == 0, f"{path.name}: relative import" + names.append(node.module or "") + return names + + +def _is_allowed(name: str) -> bool: + top_level = name.partition(".")[0] + if top_level == "automation_file": + return any( + name == allowed or name.startswith(f"{allowed}.") for allowed in ALLOWED_FIRST_PARTY + ) + return top_level in sys.stdlib_module_names or name == "__future__" + + +def test_the_package_has_modules() -> None: + assert len(MODULES) >= 9 + + +@pytest.mark.parametrize("path", MODULES, ids=lambda path: path.name) +def test_module_level_imports_stay_inside_the_layer(path: Path) -> None: + offending = [name for name in _module_level_imports(path) if not _is_allowed(name)] + assert offending == [] + + +def test_the_guard_recognises_what_it_must_refuse() -> None: + assert _is_allowed("os.path") is True + assert _is_allowed("automation_file.storage.uri") is True + assert _is_allowed("automation_file.exceptions") is True + assert _is_allowed("boto3") is False + assert _is_allowed("PySide6.QtWidgets") is False + assert _is_allowed("automation_file.core.action_registry") is False + assert _is_allowed("automation_file.remote.s3.client") is False + assert _is_allowed("automation_file.exceptions_extra") is False diff --git a/tests/test_storage_local.py b/tests/test_storage_local.py new file mode 100644 index 0000000..7a8136b --- /dev/null +++ b/tests/test_storage_local.py @@ -0,0 +1,208 @@ +"""LocalStorage: the storage contract plus what is specific to a filesystem.""" + +from __future__ import annotations + +import os +import shutil +from pathlib import Path + +import pytest + +from automation_file.exceptions import ( + PathTraversalException, + StorageException, + StorageNotFoundException, + StoragePermissionException, + StorageUnsupportedException, +) +from automation_file.storage import LocalStorage, StorageBackend +from tests.storage_contract import StorageContract + + +def _symlink(link: Path, target: Path, *, directory: bool = False) -> None: + try: + link.symlink_to(target, target_is_directory=directory) + except (OSError, NotImplementedError): + pytest.skip("symbolic links cannot be created here") + + +def _rootless(path: Path) -> str: + """Return ``path`` the way a rootless LocalStorage addresses it.""" + return path.resolve().as_posix().lstrip("/") + + +class TestRootedLocalStorageContract(StorageContract): + @pytest.fixture + def backend(self, tmp_path: Path) -> StorageBackend: + root = tmp_path / "storage-root" + root.mkdir() + return LocalStorage(root) + + +@pytest.fixture +def root(tmp_path: Path) -> Path: + directory = tmp_path / "root" + directory.mkdir() + return directory + + +@pytest.fixture +def storage(root: Path) -> LocalStorage: + return LocalStorage(root) + + +def test_files_land_under_the_root(storage: LocalStorage, root: Path) -> None: + storage.write_bytes("a/b.txt", b"x") + assert (root / "a" / "b.txt").read_bytes() == b"x" + assert storage.root == root.resolve() + assert storage.local_path("a/b.txt") == (root / "a" / "b.txt").resolve() + + +def test_uri_for_is_the_absolute_local_uri(storage: LocalStorage, root: Path) -> None: + expected = "local:///" + (root.resolve() / "a" / "b.txt").as_posix().lstrip("/") + assert storage.uri_for("a/b.txt") == expected + + +def test_rootless_storage_addresses_absolute_paths(tmp_path: Path) -> None: + (tmp_path / "abs.txt").write_bytes(b"absolute") + storage = LocalStorage() + path = _rootless(tmp_path / "abs.txt") + assert storage.root is None + assert storage.read_bytes(path) == b"absolute" + assert storage.local_path(path) == (tmp_path / "abs.txt").resolve() + assert storage.uri_for(path) == f"local:///{path}" + + +def test_rootless_storage_refuses_to_delete_a_filesystem_root( + monkeypatch: pytest.MonkeyPatch, +) -> None: + def _must_not_run(*_args: object, **_kwargs: object) -> None: + raise AssertionError("the root guard let a delete through") + + # The guard is what is under test; nothing may reach the filesystem if it fails. + monkeypatch.setattr(LocalStorage, "_delete_directory", _must_not_run) + monkeypatch.setattr(shutil, "rmtree", _must_not_run) + anchor = Path(os.path.abspath(os.sep)).as_posix().strip("/") + with pytest.raises(StorageUnsupportedException): + LocalStorage().delete(anchor, recursive=True) + + +def test_a_link_out_of_the_root_is_a_path_traversal(storage: LocalStorage, root: Path) -> None: + outside = root.parent / "outside" + outside.mkdir() + (outside / "secret.txt").write_bytes(b"secret") + _symlink(root / "escape", outside, directory=True) + with pytest.raises(PathTraversalException): + storage.read_bytes("escape/secret.txt") + with pytest.raises(PathTraversalException): + storage.write_bytes("escape/planted.txt", b"x") + assert not (outside / "planted.txt").exists() + + +def test_deleting_a_linked_directory_leaves_its_target_alone( + storage: LocalStorage, root: Path +) -> None: + (root / "real").mkdir() + (root / "real" / "keep.txt").write_bytes(b"keep") + _symlink(root / "link", root / "real", directory=True) + storage.delete("link", recursive=True) + assert not (root / "link").exists() + assert (root / "real" / "keep.txt").read_bytes() == b"keep" + + +def test_deleting_a_linked_file_leaves_its_target_alone(storage: LocalStorage, root: Path) -> None: + (root / "real.txt").write_bytes(b"keep") + _symlink(root / "link.txt", root / "real.txt") + storage.delete("link.txt") + assert not (root / "link.txt").is_symlink() + assert (root / "real.txt").read_bytes() == b"keep" + + +def test_a_dangling_link_is_listed_and_can_be_deleted(storage: LocalStorage, root: Path) -> None: + _symlink(root / "dangling", root / "gone.txt") + assert [info.path for info in storage.list_dir()] == ["dangling"] + assert storage.exists("dangling") is True + storage.delete("dangling") + assert storage.list_dir() == [] + + +def test_recursive_listing_does_not_descend_into_linked_directories( + storage: LocalStorage, root: Path +) -> None: + (root / "real").mkdir() + (root / "real" / "a.txt").write_bytes(b"x") + _symlink(root / "real" / "loop", root, directory=True) + assert [info.path for info in storage.list_dir(recursive=True)] == [ + "real", + "real/a.txt", + "real/loop", + ] + + +@pytest.mark.skipif(os.sep != "\\", reason="backslash is a separator only on Windows") +def test_backslashes_are_separators_on_windows(storage: LocalStorage) -> None: + storage.write_bytes("dir\\a.txt", b"x") + assert storage.stat("dir/a.txt").path == "dir/a.txt" + assert storage.stat("dir\\a.txt").path == "dir/a.txt" + + +def test_copy_between_two_local_roots(tmp_path: Path) -> None: + source = LocalStorage(tmp_path) + (tmp_path / "target").mkdir() + target = LocalStorage(tmp_path / "target") + source.write_bytes("a.txt", b"payload") + info = target.copy_from(source, "a.txt", "nested/b.txt") + assert info.path == "nested/b.txt" + assert (tmp_path / "target" / "nested" / "b.txt").read_bytes() == b"payload" + assert (tmp_path / "a.txt").read_bytes() == b"payload" + + +def test_move_between_two_local_roots(tmp_path: Path) -> None: + source = LocalStorage(tmp_path) + (tmp_path / "target").mkdir() + target = LocalStorage(tmp_path / "target") + source.write_bytes("a.txt", b"payload") + target.move_from(source, "a.txt", "b.txt") + assert (tmp_path / "target" / "b.txt").read_bytes() == b"payload" + assert not (tmp_path / "a.txt").exists() + + +def test_upload_leaves_no_partial_file_behind(storage: LocalStorage, root: Path) -> None: + storage.write_bytes("a.txt", b"x") + assert [entry.name for entry in root.iterdir()] == ["a.txt"] + + +def test_a_failed_upload_keeps_the_previous_content( + storage: LocalStorage, root: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + storage.write_bytes("a.txt", b"previous") + + def _fail(*_args: object, **_kwargs: object) -> None: + raise OSError("disk full") + + monkeypatch.setattr(shutil, "copyfile", _fail) + with pytest.raises(StorageException, match="disk full"): + storage.write_bytes("a.txt", b"next") + assert [entry.name for entry in root.iterdir()] == ["a.txt"] + assert (root / "a.txt").read_bytes() == b"previous" + + +def test_a_denied_operation_raises_the_permission_error( + storage: LocalStorage, monkeypatch: pytest.MonkeyPatch +) -> None: + storage.write_bytes("a.txt", b"x") + + def _deny(*_args: object, **_kwargs: object) -> None: + raise PermissionError("denied") + + monkeypatch.setattr(shutil, "copyfile", _deny) + with pytest.raises(StoragePermissionException): + storage.write_bytes("b.txt", b"y") + assert storage.exists("b.txt") is False + + +def test_a_root_that_does_not_exist_reports_not_found(tmp_path: Path) -> None: + storage = LocalStorage(tmp_path / "never-created") + assert storage.exists("") is False + with pytest.raises(StorageNotFoundException): + storage.list_dir() diff --git a/tests/test_storage_memory.py b/tests/test_storage_memory.py new file mode 100644 index 0000000..07091aa --- /dev/null +++ b/tests/test_storage_memory.py @@ -0,0 +1,57 @@ +"""MemoryStorage: the storage contract plus the shared named stores.""" + +from __future__ import annotations + +from collections.abc import Iterator + +import pytest + +from automation_file.storage import ( + MemoryStorage, + StorageBackend, + clear_memory_stores, + memory_store, +) +from tests.storage_contract import StorageContract + + +class TestMemoryStorageContract(StorageContract): + @pytest.fixture + def backend(self) -> StorageBackend: + return MemoryStorage() + + +@pytest.fixture(autouse=True) +def _fresh_stores() -> Iterator[None]: + clear_memory_stores() + yield + clear_memory_stores() + + +def test_uri_for_carries_the_store_name() -> None: + assert MemoryStorage("scratch").uri_for("a/b.txt") == "memory://scratch/a/b.txt" + assert MemoryStorage("scratch").uri_for("") == "memory://scratch" + assert MemoryStorage().uri_for("a.txt") == "memory:///a.txt" + assert MemoryStorage("scratch").name == "scratch" + + +def test_named_stores_are_shared_and_separate() -> None: + first = memory_store("one") + first.write_bytes("a.txt", b"x") + assert memory_store("one") is first + assert memory_store("one").read_bytes("a.txt") == b"x" + assert memory_store("two").exists("a.txt") is False + + +def test_clear_memory_stores_forgets_everything() -> None: + memory_store("one").write_bytes("a.txt", b"x") + clear_memory_stores() + assert memory_store("one").exists("a.txt") is False + + +def test_two_instances_do_not_share_content() -> None: + first, second = MemoryStorage(), MemoryStorage() + first.write_bytes("a.txt", b"x") + assert second.exists("a.txt") is False + second.copy_from(first, "a.txt", "copy.txt") + assert second.read_bytes("copy.txt") == b"x" diff --git a/tests/test_storage_resolver.py b/tests/test_storage_resolver.py new file mode 100644 index 0000000..5a7de7a --- /dev/null +++ b/tests/test_storage_resolver.py @@ -0,0 +1,199 @@ +"""StorageResolver: mounts, scheme factories, and the default table.""" + +from __future__ import annotations + +import os +import re +from collections.abc import Iterator +from pathlib import Path +from typing import Any + +import pytest + +from automation_file.exceptions import StorageURIException +from automation_file.storage import ( + File, + LocalStorage, + MemoryStorage, + Storage, + StorageBackend, + StorageResolver, + StorageURI, + clear_memory_stores, + default_resolver, + memory_store, + parse_storage_uri, +) + + +@pytest.fixture(autouse=True) +def _fresh_stores() -> Iterator[None]: + clear_memory_stores() + yield + clear_memory_stores() + + +@pytest.fixture +def resolver() -> StorageResolver: + return StorageResolver() + + +def test_default_schemes(resolver: StorageResolver) -> None: + assert resolver.schemes() == ["local", "memory"] + assert StorageResolver(defaults=False).schemes() == [] + + +def test_local_uri_resolves_to_the_whole_filesystem( + resolver: StorageResolver, tmp_path: Path +) -> None: + (tmp_path / "a.txt").write_bytes(b"x") + backend, path = resolver.resolve(tmp_path / "a.txt") + assert isinstance(backend, LocalStorage) + assert backend.root is None + assert backend.read_bytes(path) == b"x" + assert resolver.resolve(str(parse_storage_uri(tmp_path / "a.txt"))) == (backend, path) + + +@pytest.mark.skipif(os.sep == "\\", reason="Windows reads a local authority as a UNC host") +def test_a_local_authority_is_rejected_off_windows(resolver: StorageResolver) -> None: + with pytest.raises(StorageURIException, match=re.escape("local:///data/report.csv")): + resolver.resolve("local://data/report.csv") + + +def test_a_unc_uri_without_a_share_is_rejected(resolver: StorageResolver) -> None: + with pytest.raises(StorageURIException, match="empty authority"): + resolver.resolve("local://server") + + +def test_memory_uri_resolves_to_the_named_store(resolver: StorageResolver) -> None: + backend, path = resolver.resolve("memory://scratch/dir/a.txt") + assert backend is memory_store("scratch") + assert path == "dir/a.txt" + assert resolver.resolve("memory://other/a.txt")[0] is not backend + + +def test_an_unknown_scheme_names_the_known_ones(resolver: StorageResolver) -> None: + with pytest.raises(StorageURIException) as caught: + resolver.resolve("gopher://host/a.txt") + message = str(caught.value) + assert "'gopher'" in message + assert "gopher://host/a.txt" in message + assert "local, memory" in message + + +def test_register_scheme_installs_a_factory(resolver: StorageResolver) -> None: + store = MemoryStorage("custom") + + def factory(uri: StorageURI) -> tuple[StorageBackend, str]: + return store, f"{uri.authority}/{uri.path}" + + resolver.register_scheme("Custom", factory) + assert "custom" in resolver.schemes() + assert resolver.resolve("custom://bucket/a.txt") == (store, "bucket/a.txt") + assert resolver.unregister_scheme("custom") is True + assert resolver.unregister_scheme("custom") is False + with pytest.raises(StorageURIException): + resolver.resolve("custom://bucket/a.txt") + + +def test_register_scheme_rejects_a_non_callable(resolver: StorageResolver) -> None: + not_a_factory: Any = "nope" + with pytest.raises(TypeError): + resolver.register_scheme("custom", not_a_factory) + + +def test_mount_serves_a_uri_and_everything_below(resolver: StorageResolver) -> None: + nas = MemoryStorage("nas") + resolver.mount("vault://nas/archive", nas) + assert resolver.resolve("vault://nas/archive") == (nas, "") + assert resolver.resolve("vault://nas/archive/2026/a.txt") == (nas, "2026/a.txt") + assert "vault" in resolver.schemes() + with pytest.raises(StorageURIException): + resolver.resolve("vault://nas/archived/a.txt") + with pytest.raises(StorageURIException): + resolver.resolve("vault://other/archive/a.txt") + + +def test_the_longest_mount_wins(resolver: StorageResolver) -> None: + whole, part = MemoryStorage("whole"), MemoryStorage("part") + resolver.mount("vault://bucket", whole) + resolver.mount("vault://bucket/reports/2026", part) + assert resolver.resolve("vault://bucket/reports/2026/q1.csv") == (part, "q1.csv") + assert resolver.resolve("vault://bucket/reports/2025/q1.csv") == (whole, "reports/2025/q1.csv") + assert resolver.resolve("vault://bucket") == (whole, "") + + +def test_a_mount_takes_precedence_over_the_scheme_factory(resolver: StorageResolver) -> None: + pinned = MemoryStorage("pinned") + resolver.mount("memory://scratch/pinned", pinned) + assert resolver.resolve("memory://scratch/pinned/a.txt") == (pinned, "a.txt") + assert resolver.resolve("memory://scratch/free/a.txt")[0] is memory_store("scratch") + + +def test_mount_authority_matches_case_insensitively(resolver: StorageResolver) -> None: + nas = MemoryStorage("nas") + resolver.mount("vault://NAS.example.com", nas) + assert resolver.resolve("vault://nas.example.com/a.txt") == (nas, "a.txt") + + +def test_unmount(resolver: StorageResolver) -> None: + resolver.mount("vault://nas", MemoryStorage("nas")) + assert resolver.unmount("vault://nas/other") is False + assert resolver.unmount("vault://nas") is True + assert resolver.unmount("vault://nas") is False + assert "vault" not in resolver.schemes() + + +def test_mount_rejects_what_is_not_a_backend(resolver: StorageResolver) -> None: + not_a_backend: Any = object() + with pytest.raises(TypeError, match="StorageBackend"): + resolver.mount("vault://nas", not_a_backend) + + +def test_capabilities_come_from_the_resolved_backend(resolver: StorageResolver) -> None: + assert resolver.capabilities("memory://scratch").directories is True + assert resolver.capabilities("local:///").content_type is True + + +def test_resolvers_are_independent(resolver: StorageResolver) -> None: + resolver.mount("vault://nas", MemoryStorage("nas")) + assert "vault" not in StorageResolver().schemes() + assert "vault" not in default_resolver.schemes() + + +def test_storage_static_methods_drive_the_default_resolver() -> None: + nas = MemoryStorage("nas") + Storage.mount("vault://storage-test-nas", nas) + try: + assert Storage.resolve("vault://storage-test-nas/a.txt") == (nas, "a.txt") + assert default_resolver.resolve("vault://storage-test-nas/a.txt") == (nas, "a.txt") + assert "vault" in Storage.schemes() + finally: + assert Storage.unmount("vault://storage-test-nas") is True + Storage.register_scheme("storage-test", lambda uri: (nas, uri.path)) + try: + assert Storage.resolve("storage-test://x/a.txt") == (nas, "a.txt") + finally: + assert default_resolver.unregister_scheme("storage-test") is True + + +def test_a_rooted_backend_behind_its_own_authority_is_a_sandbox( + resolver: StorageResolver, tmp_path: Path +) -> None: + jobs = tmp_path / "jobs" + jobs.mkdir() + (tmp_path / "secret.txt").write_bytes(b"secret") + resolver.mount("sandbox://jobs", LocalStorage(jobs)) + File("sandbox://jobs/42/out.csv", resolver=resolver).write(b"done") + assert (jobs / "42" / "out.csv").read_bytes() == b"done" + backend, path = resolver.resolve("sandbox://jobs/42/out.csv") + assert repr(backend) == f"LocalStorage({str(jobs.resolve())!r})" + assert path == "42/out.csv" + with pytest.raises(StorageURIException): + File("sandbox://jobs/../secret.txt", resolver=resolver) + assert [info.path for info in Storage("sandbox://jobs", resolver=resolver).list_dir()] == ["42"] + + +def test_backend_reprs() -> None: + assert repr(LocalStorage()) == "LocalStorage()" + assert repr(MemoryStorage("scratch")) == "MemoryStorage('scratch')" diff --git a/tests/test_storage_types.py b/tests/test_storage_types.py new file mode 100644 index 0000000..6b5ddf2 --- /dev/null +++ b/tests/test_storage_types.py @@ -0,0 +1,112 @@ +"""Checksum, FileInfo and StorageCapabilities value types.""" + +from __future__ import annotations + +import dataclasses +import json +from datetime import datetime, timezone + +import pytest + +from automation_file.storage import Checksum, FileInfo, StorageCapabilities + +DIGEST = "9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08" + + +def test_checksum_is_normalised_to_lower_case() -> None: + checksum = Checksum(" SHA256 ", f" {DIGEST.upper()} ") + assert checksum == Checksum("sha256", DIGEST) + assert str(checksum) == f"sha256:{DIGEST}" + assert checksum.to_dict() == {"algorithm": "sha256", "value": DIGEST} + + +def test_checksum_parse() -> None: + assert Checksum.parse(f"SHA256:{DIGEST}") == Checksum("sha256", DIGEST) + assert Checksum.parse(str(Checksum("md5", "abc"))) == Checksum("md5", "abc") + + +@pytest.mark.parametrize("text", ["", DIGEST, "sha256:", ":abc", " : "]) +def test_checksum_parse_rejects_text_without_both_parts(text: str) -> None: + with pytest.raises(ValueError, match="sha256:"): + Checksum.parse(text) + + +def test_checksum_matches() -> None: + checksum = Checksum("sha256", DIGEST) + assert checksum.matches(DIGEST) is True + assert checksum.matches(f" {DIGEST.upper()}\n") is True + assert checksum.matches(f"sha256:{DIGEST}") is True + assert checksum.matches(Checksum("SHA256", DIGEST)) is True + assert checksum.matches(DIGEST[:-1] + "0") is False + assert checksum.matches("") is False + + +def test_checksum_of_another_algorithm_never_matches() -> None: + checksum = Checksum("sha256", DIGEST) + assert checksum.matches(Checksum("sha512", DIGEST)) is False + assert checksum.matches(f"md5:{DIGEST}") is False + + +def test_checksum_matches_tolerates_non_ascii_input() -> None: + assert Checksum("sha256", DIGEST).matches("不是摘要") is False + + +def test_file_info_defaults_and_name() -> None: + info = FileInfo("dir/sub/report.csv") + assert info.name == "report.csv" + assert info.is_dir is False + assert info.size is None + assert info.modified_at is None + assert info.etag is None + assert info.version is None + assert info.content_type is None + assert dict(info.metadata) == {} + assert FileInfo("top.txt").name == "top.txt" + assert FileInfo("", is_dir=True).name == "" + + +def test_file_info_is_frozen_and_hashable() -> None: + info = FileInfo("a.txt", size=1, metadata={"owner": "ops"}) + with pytest.raises(dataclasses.FrozenInstanceError): + info.size = 2 # type: ignore[misc] + assert info == FileInfo("a.txt", size=1, metadata={"owner": "ops"}) + assert info != FileInfo("a.txt", size=1, metadata={"owner": "dev"}) + assert len({info, FileInfo("a.txt", size=1, metadata={"owner": "ops"})}) == 1 + + +def test_file_info_to_dict_is_json_serialisable() -> None: + info = FileInfo( + "dir/report.csv", + size=12, + modified_at=datetime(2026, 10, 8, 2, 30, tzinfo=timezone.utc), + etag="abc", + version="7", + content_type="text/csv", + metadata={"owner": "ops"}, + ) + document = json.loads(json.dumps(info.to_dict())) + assert document == { + "path": "dir/report.csv", + "name": "report.csv", + "is_dir": False, + "size": 12, + "modified_at": "2026-10-08T02:30:00+00:00", + "etag": "abc", + "version": "7", + "content_type": "text/csv", + "metadata": {"owner": "ops"}, + } + assert FileInfo("dir", is_dir=True).to_dict()["modified_at"] is None + + +def test_capabilities_defaults_and_to_dict() -> None: + capabilities = StorageCapabilities() + assert capabilities.to_dict() == { + "directories": True, + "modified_at": True, + "etag": False, + "version": False, + "content_type": False, + "metadata": False, + } + assert StorageCapabilities(directories=False, etag=True).to_dict()["directories"] is False diff --git a/tests/test_storage_uri.py b/tests/test_storage_uri.py new file mode 100644 index 0000000..447438d --- /dev/null +++ b/tests/test_storage_uri.py @@ -0,0 +1,241 @@ +"""Storage URI syntax: parsing, normalisation, and what is rejected.""" + +from __future__ import annotations + +import os +from pathlib import Path +from typing import Any + +import pytest + +from automation_file.exceptions import StorageURIException +from automation_file.storage import ( + StorageURI, + local_path_to_uri, + normalize_path, + parse_storage_uri, +) + +_WINDOWS = os.sep == "\\" + + +@pytest.mark.parametrize( + "text,scheme,authority,path", + [ + ("local:///data/report.csv", "local", "", "data/report.csv"), + ("local:///C:/data/report.csv", "local", "", "C:/data/report.csv"), + ("s3://bucket/report.csv", "s3", "bucket", "report.csv"), + ("s3://bucket", "s3", "bucket", ""), + ("s3://bucket/", "s3", "bucket", ""), + ("azure://container/dir/report.csv", "azure", "container", "dir/report.csv"), + ("gdrive://folder/report.csv", "gdrive", "folder", "report.csv"), + ("dropbox:///reports/report.csv", "dropbox", "", "reports/report.csv"), + ("sftp://server/data/report.csv", "sftp", "server", "data/report.csv"), + ("sftp://server:2222/data/report.csv", "sftp", "server:2222", "data/report.csv"), + ("ftp://server/data/report.csv", "ftp", "server", "data/report.csv"), + ("webdav://server/files/report.csv", "webdav", "server", "files/report.csv"), + ("smb://server/share/report.csv", "smb", "server", "share/report.csv"), + ("memory://scratch/a.txt", "memory", "scratch", "a.txt"), + ], +) +def test_parse_splits_scheme_authority_and_path( + text: str, scheme: str, authority: str, path: str +) -> None: + uri = parse_storage_uri(text) + assert (uri.scheme, uri.authority, uri.path) == (scheme, authority, path) + + +@pytest.mark.parametrize( + "text", + [ + "local:///data/report.csv", + "local:///", + "s3://bucket/report.csv", + "s3://bucket", + "sftp://server:2222/data/report.csv", + "dropbox:///reports/report.csv", + ], +) +def test_str_round_trips(text: str) -> None: + assert str(parse_storage_uri(text)) == text + assert parse_storage_uri(str(parse_storage_uri(text))) == parse_storage_uri(text) + + +@pytest.mark.parametrize( + "text,expected", + [ + ("S3://bucket/Key.CSV", "s3://bucket/Key.CSV"), + ("file:///data/a.txt", "local:///data/a.txt"), + ("az://container/blob", "azure://container/blob"), + ("AZ://Container/Blob", "azure://Container/Blob"), + ], +) +def test_scheme_is_lower_cased_and_aliases_resolve(text: str, expected: str) -> None: + assert str(parse_storage_uri(text)) == expected + + +def test_the_path_is_taken_literally() -> None: + uri = parse_storage_uri("s3://bucket/reports/Q1 #3?.csv%20") + assert uri.path == "reports/Q1 #3?.csv%20" + assert uri.name == "Q1 #3?.csv%20" + + +def test_unicode_paths_are_kept() -> None: + assert parse_storage_uri("s3://bucket/資料/報告.csv").path == "資料/報告.csv" + + +@pytest.mark.parametrize( + "path,expected", + [ + ("", ""), + ("/", ""), + ("a/b", "a/b"), + ("/a/b/", "a/b"), + ("a//b", "a/b"), + ("./a/./b/.", "a/b"), + ("a/...", "a/..."), + ("a/..b/c..", "a/..b/c.."), + ], +) +def test_normalize_path(path: str, expected: str) -> None: + assert normalize_path(path) == expected + + +@pytest.mark.parametrize("path", ["..", "../a", "a/../b", "a/b/..", "/../etc/passwd"]) +def test_normalize_path_rejects_parent_segments(path: str) -> None: + with pytest.raises(StorageURIException, match=r"'\.\.'"): + normalize_path(path) + + +def test_normalize_path_rejects_nul() -> None: + with pytest.raises(StorageURIException, match="NUL"): + normalize_path("a\x00b") + + +@pytest.mark.parametrize("text", ["s3://bucket/../other", "local:///data/../../etc/passwd"]) +def test_parse_rejects_parent_segments(text: str) -> None: + with pytest.raises(StorageURIException): + parse_storage_uri(text) + + +@pytest.mark.parametrize("text", ["", " "]) +def test_parse_rejects_empty_text(text: str) -> None: + with pytest.raises(StorageURIException, match="empty"): + parse_storage_uri(text) + + +def test_parse_rejects_bytes() -> None: + raw: Any = b"s3://bucket/key" + with pytest.raises(StorageURIException, match="bytes"): + parse_storage_uri(raw) + + +@pytest.mark.parametrize( + "text", + ["sftp://user@server/data", "sftp://user:secret@server/data", "s3://key:secret@bucket/k"], +) +def test_parse_rejects_credentials_without_echoing_them(text: str) -> None: + with pytest.raises(StorageURIException, match="credentials") as caught: + parse_storage_uri(text) + assert "secret" not in str(caught.value) + assert "user" not in str(caught.value).replace("user information", "") + + +@pytest.mark.parametrize("text", ["s3://bad bucket/key", "sftp://ser\\ver/data"]) +def test_parse_rejects_a_malformed_authority(text: str) -> None: + with pytest.raises(StorageURIException, match="authority"): + parse_storage_uri(text) + + +@pytest.mark.parametrize( + "text,spelling", + [ + ("sftp:/remote/a.csv", "sftp://"), + ("s3:bucket/key", "s3://"), + ("local:C:\\data\\a.csv", "local://"), + ("backup:2026.tar", "./backup:2026.tar"), + ], +) +def test_scheme_like_text_without_slashes_is_ambiguous(text: str, spelling: str) -> None: + with pytest.raises(StorageURIException, match="ambiguous") as caught: + parse_storage_uri(text) + assert spelling in str(caught.value) + + +def test_a_relative_path_becomes_an_absolute_local_uri( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + monkeypatch.chdir(tmp_path) + uri = parse_storage_uri("reports/a.csv") + assert uri.scheme == "local" + assert uri == local_path_to_uri(tmp_path / "reports" / "a.csv") + assert uri.path.endswith("reports/a.csv") + + +def test_a_dot_prefix_names_a_local_file_with_a_colon( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + monkeypatch.chdir(tmp_path) + assert parse_storage_uri("./backup:2026.tar").name == "backup:2026.tar" + + +def test_a_path_object_is_always_local(tmp_path: Path) -> None: + uri = parse_storage_uri(tmp_path / "a.txt") + assert uri.scheme == "local" + assert uri.name == "a.txt" + assert uri == parse_storage_uri(str(tmp_path / "a.txt")) + + +def test_parent_segments_of_a_local_path_are_collapsed_not_rejected(tmp_path: Path) -> None: + assert parse_storage_uri(str(tmp_path / "a" / ".." / "b.txt")) == local_path_to_uri( + tmp_path / "b.txt" + ) + + +def test_a_storage_uri_passes_through() -> None: + uri = StorageURI("s3", "bucket", "key") + assert parse_storage_uri(uri) is uri + + +@pytest.mark.skipif(not _WINDOWS, reason="drive letters and UNC paths are Windows syntax") +def test_windows_paths() -> None: + assert str(parse_storage_uri("C:\\data\\a.csv")) == "local:///C:/data/a.csv" + assert str(parse_storage_uri("C:/data/a.csv")) == "local:///C:/data/a.csv" + assert ( + str(parse_storage_uri("\\\\server\\share\\dir\\a.csv")) == "local://server/share/dir/a.csv" + ) + + +@pytest.mark.skipif(_WINDOWS, reason="POSIX absolute paths") +def test_posix_paths() -> None: + assert str(parse_storage_uri("/data/a.csv")) == "local:///data/a.csv" + assert parse_storage_uri("/data/back\\slash.csv").name == "back\\slash.csv" + + +def test_storage_uri_validates_on_construction() -> None: + assert StorageURI("S3", "bucket", "/a//b/").path == "a/b" + assert StorageURI("FILE").scheme == "local" + with pytest.raises(StorageURIException, match="scheme"): + StorageURI("not a scheme", "bucket") + with pytest.raises(StorageURIException, match="scheme"): + StorageURI("", "bucket") + with pytest.raises(StorageURIException): + StorageURI("s3", "bucket", "a/../b") + + +def test_name_parent_and_joinpath() -> None: + uri = parse_storage_uri("s3://bucket/a/b/c.txt") + assert uri.name == "c.txt" + assert str(uri.parent) == "s3://bucket/a/b" + assert str(uri.parent.parent.parent) == "s3://bucket" + assert uri.parent.parent.parent.parent == StorageURI("s3", "bucket") + assert str(StorageURI("s3", "bucket").joinpath("a", "b/c.txt")) == "s3://bucket/a/b/c.txt" + assert str(uri.joinpath("")) == "s3://bucket/a/b/c.txt" + with pytest.raises(StorageURIException): + uri.joinpath("..") + + +def test_storage_uri_is_hashable_and_comparable() -> None: + assert parse_storage_uri("s3://bucket/a") == parse_storage_uri("S3://bucket//a/") + assert len({parse_storage_uri("s3://bucket/a"), parse_storage_uri("s3://bucket/a/")}) == 1 + assert parse_storage_uri("s3://bucket/a") != parse_storage_uri("s3://Bucket/a") From f20a150c47e4b413b7aeaa020619451c1f766659 Mon Sep 17 00:00:00 2001 From: JeffreyChen Date: Thu, 8 Oct 2026 11:58:49 +0800 Subject: [PATCH 21/59] feat: put S3 and Azure Blob behind the storage layer --- CLAUDE.md | 5 +- README.md | 20 +- README.zh-CN.md | 20 +- README.zh-TW.md | 20 +- architecture.md | 10 +- automation_file/__init__.py | 6 + automation_file/storage/__init__.py | 11 +- automation_file/storage/azure_storage.py | 187 ++++++++++ automation_file/storage/backend.py | 11 +- automation_file/storage/local_storage.py | 11 +- automation_file/storage/object_storage.py | 134 +++++++ automation_file/storage/resolver.py | 20 ++ automation_file/storage/s3_storage.py | 211 +++++++++++ docs/source/API/storage.rst | 13 + docs/source/Eng/usage/storage.rst | 41 ++- docs/source/Zh-CN/usage/storage.rst | 43 ++- docs/source/Zh-TW/usage/storage.rst | 43 ++- docs/updates/2026-10.md | 20 ++ docs/updates/README.md | 3 +- progress.md | 4 +- tests/test_storage_azure.py | 339 ++++++++++++++++++ tests/test_storage_resolver.py | 4 +- tests/test_storage_s3.py | 410 ++++++++++++++++++++++ 23 files changed, 1530 insertions(+), 56 deletions(-) create mode 100644 automation_file/storage/azure_storage.py create mode 100644 automation_file/storage/object_storage.py create mode 100644 automation_file/storage/s3_storage.py create mode 100644 tests/test_storage_azure.py create mode 100644 tests/test_storage_s3.py diff --git a/CLAUDE.md b/CLAUDE.md index c2f9a16..de6e37e 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -26,7 +26,8 @@ automation_file/ │ # have a client only ├── storage/ # Universal storage layer: uri (StorageURI), types (FileInfo, Checksum, │ # StorageCapabilities), backend (StorageBackend contract), local_storage, -│ # memory_storage, resolver (StorageResolver), file (File), storage (Storage) +│ # memory_storage, object_storage (ObjectStorage), s3_storage, azure_storage, +│ # resolver (StorageResolver), file (File), storage (Storage) ├── server/ # tcp_server, http_server, mcp_server (MCP over stdio), web_ui, metrics_server, │ # action_acl (ActionACL), network_guards (ensure_loopback) ├── client/ # HTTPActionClient for the HTTP action server @@ -65,7 +66,7 @@ automation_file/ - `retry_on_transient(max_attempts, backoff_base, backoff_cap, retriable)` — decorator that retries with capped exponential back-off and raises `RetryExhaustedException` chained to the last error. - `safe_join(root, user_path)` / `is_within(root, path)` — path traversal guard; `safe_join` raises `PathTraversalException` when the resolved path escapes `root`. - `File(uri)` / `Storage(uri)` — the universal storage layer's application API: one file, one directory, in any backend. Both resolve their backend on every call through `StorageResolver` (`Storage.mount`, `Storage.register_scheme`). -- `StorageBackend` — the contract a storage backend implements. The public operations (`exists`, `stat`, `list_dir`, `mkdir`, `upload`, `download`, `delete`, `checksum`, `read_bytes`, `write_bytes`, `copy_from`, `move_from`) are template methods; a backend supplies only the `_`-prefixed primitives. `LocalStorage` and `MemoryStorage` are built in. +- `StorageBackend` — the contract a storage backend implements. The public operations (`exists`, `stat`, `list_dir`, `mkdir`, `upload`, `download`, `delete`, `checksum`, `read_bytes`, `write_bytes`, `copy_from`, `move_from`) are template methods; a backend supplies only the `_`-prefixed primitives. `LocalStorage`, `MemoryStorage`, `S3Storage` (`s3://bucket/key`) and `AzureStorage` (`azure://container/blob`) are built in; the last two extend `ObjectStorage` and use the shared `s3_instance` / `azure_blob_instance` unless given a client. - `StorageURI` / `parse_storage_uri` — `:///`; `FileInfo`, `Checksum`, `StorageCapabilities` are the frozen value types the layer returns. ## Branching & CI diff --git a/README.md b/README.md index 13585ef..92d15fc 100644 --- a/README.md +++ b/README.md @@ -48,7 +48,7 @@ facade. - **HTTP server observability** — `GET /healthz` / `GET /readyz` probes, `GET /openapi.json` spec, and `GET /progress` WebSocket stream of live transfer snapshots - **HTMX Web UI** — `start_web_ui()` serves a read-only dashboard (health, progress, registry) that polls HTML fragments; stdlib-only HTTP plus one CDN script with SRI - **MCP (Model Context Protocol) server** — `MCPServer` bridges the registry to any MCP host (Claude Desktop, MCP CLIs) over newline-delimited JSON-RPC 2.0 on stdio; every `FA_*` action becomes an MCP tool with an auto-generated input schema -- **Universal storage layer** — `File` / `Storage` address local and remote storage with one URI syntax (`local:///…`, `memory://…`), one `StorageBackend` contract and one error hierarchy; `LocalStorage` and `MemoryStorage` are built in, and a 70-case contract suite checks any backend +- **Universal storage layer** — `File` / `Storage` address local and remote storage with one URI syntax (`local:///…`, `s3://…`, `azure://…`, `memory://…`), one `StorageBackend` contract and one error hierarchy; local, S3, Azure Blob and in-memory backends are built in, and a 70-case contract suite checks any backend - PySide6 GUI (`python -m automation_file ui`) with a tab per backend, the JSON-action runner, and dedicated tabs for Triggers, Scheduler, and live Progress - Rich CLI with one-shot subcommands plus legacy JSON-batch flags - Project scaffolding (`ProjectBuilder`) for executor-based automations @@ -148,9 +148,9 @@ flowchart TD end subgraph StorageLayer["storage (universal layer)"] - FileAPI["File · Storage
local:// memory:// …"] + FileAPI["File · Storage
local:// memory:// s3:// azure://"] Resolver["StorageResolver
mounts · scheme factories"] - Backends["StorageBackend contract
LocalStorage · MemoryStorage"] + Backends["StorageBackend contract
Local · Memory · S3 · Azure"] end subgraph Notify["notifications"] @@ -189,6 +189,8 @@ flowchart TD Resolver ==> Backends Backends ==> SafeP Backends ==> Check + Backends ==> S3M + Backends ==> Azure TCP ==> Executor HTTPS ==> Executor @@ -472,10 +474,14 @@ File("sandbox://jobs/42/out.csv").write(b"done") `StoragePermissionException`, `StorageTransientException`, `StorageUnavailableException`, `StorageUnsupportedException`, `StorageURIException`. - **Backends today** — `LocalStorage` (`local://`, optionally confined to a root through - `safe_join`) and `MemoryStorage` (`memory://`, for tests and dry runs). The cloud and SFTP - backends are still used through their own clients and `FA_*` actions; their adapters for - this layer are not written yet. Write your own by subclassing `StorageBackend` and check it with - the 70-case contract suite in `tests/storage_contract.py`. + `safe_join`), `S3Storage` (`s3://bucket/key`), `AzureStorage` (`azure://container/blob`) and + `MemoryStorage` (`memory://`, for tests and dry runs). S3 and Azure use the clients you already + initialise (`s3_instance.later_init(...)`, `azure_blob_instance.later_init(...)`), so + `File("s3://reports/q1.csv").copy_to("azure://backups/q1.csv")` works once both are ready. + Google Drive, Dropbox, SFTP, FTP, WebDAV, SMB and fsspec are still used through their own + clients and `FA_*` actions; their adapters are not written yet. Write your own by subclassing + `StorageBackend` (or `ObjectStorage` for an object store) and check it with the 70-case + contract suite in `tests/storage_contract.py`. The API is new and may still change before 1.0. Full reference: the *Universal Storage Layer* chapter of the documentation. diff --git a/README.zh-CN.md b/README.zh-CN.md index b42c534..fad6b9a 100644 --- a/README.zh-CN.md +++ b/README.zh-CN.md @@ -46,7 +46,7 @@ TCP / HTTP 服务器执行的 JSON 驱动动作。内附 PySide6 GUI，每个功 - **HTTP 服务器观测端点** — `GET /healthz` / `GET /readyz` 探针、`GET /openapi.json` 规格，以及 `GET /progress`（通过 WebSocket 推送实时传输快照） - **HTMX Web UI** — `start_web_ui()` 启动只读观测仪表板（health、progress、registry），通过 HTML 片段轮询；仅用标准库 HTTP，搭配一个带 SRI 的 CDN 脚本 - **MCP（Model Context Protocol）服务器** — `MCPServer` 通过 stdio 上的 JSON-RPC 2.0（换行分隔 JSON）将注册表桥接到任意 MCP 主机（Claude Desktop、MCP CLI）；每个 `FA_*` 动作都会自动生成输入 schema 并成为 MCP 工具 -- **通用存储层** — `File` / `Storage` 以同一套 URI 语法（`local:///…`、`memory://…`）、同一份 `StorageBackend` 契约与同一组异常层级访问本地与远端存储；内置 `LocalStorage` 与 `MemoryStorage`，并附带 70 个用例的契约测试套件可检查任何后端 +- **通用存储层** — `File` / `Storage` 以同一套 URI 语法（`local:///…`、`s3://…`、`azure://…`、`memory://…`）、同一份 `StorageBackend` 契约与同一组异常层级访问本地与远端存储；内置本地、S3、Azure Blob 与内存后端，并附带 70 个用例的契约测试套件可检查任何后端 - PySide6 GUI（`python -m automation_file ui`）每个后端一个页签，含 JSON 动作执行器，另有 Triggers、Scheduler、实时 Progress 专属页签 - 功能丰富的 CLI，包含一次性子命令与旧式 JSON 批量标志 - 项目脚手架（`ProjectBuilder`）协助构建以 executor 为核心的自动化项目 @@ -146,9 +146,9 @@ flowchart TD end subgraph StorageLayer["通用存储层"] - FileAPI["File · Storage
local:// memory:// …"] + FileAPI["File · Storage
local:// memory:// s3:// azure://"] Resolver["StorageResolver
mounts · scheme factories"] - Backends["StorageBackend contract
LocalStorage · MemoryStorage"] + Backends["StorageBackend contract
Local · Memory · S3 · Azure"] end subgraph Notify["通知"] @@ -187,6 +187,8 @@ flowchart TD Resolver ==> Backends Backends ==> SafeP Backends ==> Check + Backends ==> S3M + Backends ==> Azure TCP ==> Executor HTTPS ==> Executor @@ -468,10 +470,14 @@ File("sandbox://jobs/42/out.csv").write(b"done") `StorageAlreadyExistsException`、`StoragePathTypeException`、`StorageNotEmptyException`、 `StoragePermissionException`、`StorageTransientException`、`StorageUnavailableException`、 `StorageUnsupportedException`、`StorageURIException`。 -- **目前的后端** — `LocalStorage`（`local://`，可通过 `safe_join` 限制在某个根目录内）与 - `MemoryStorage`（`memory://`，用于测试与试运行）。云端与 SFTP 后端目前仍通过各自的客户端与 - `FA_*` 动作使用，接入本层的适配器尚未完成。你可以继承 `StorageBackend` 编写自己的后端， - 并用 `tests/storage_contract.py` 中 70 个用例的契约测试套件检查。 +- **目前的后端** — `LocalStorage`（`local://`，可通过 `safe_join` 限制在某个根目录内）、 + `S3Storage`（`s3://bucket/key`）、`AzureStorage`（`azure://container/blob`）与 + `MemoryStorage`（`memory://`，用于测试与试运行）。S3 与 Azure 使用你原本就会初始化的客户端 + （`s3_instance.later_init(...)`、`azure_blob_instance.later_init(...)`），两者都就绪后， + `File("s3://reports/q1.csv").copy_to("azure://backups/q1.csv")` 即可运行。 + Google Drive、Dropbox、SFTP、FTP、WebDAV、SMB 与 fsspec 目前仍通过各自的客户端与 `FA_*` + 动作使用，其适配器尚未完成。你可以继承 `StorageBackend`（对象存储则继承 `ObjectStorage`） + 编写自己的后端，并用 `tests/storage_contract.py` 中 70 个用例的契约测试套件检查。 此 API 为新功能，在 1.0 之前仍可能调整。完整说明请见文档的“通用存储层”章节。 diff --git a/README.zh-TW.md b/README.zh-TW.md index 5910c11..9d3960c 100644 --- a/README.zh-TW.md +++ b/README.zh-TW.md @@ -46,7 +46,7 @@ TCP / HTTP 伺服器執行的 JSON 驅動動作。內附 PySide6 GUI，每個功 - **HTTP 伺服器觀測端點** — `GET /healthz` / `GET /readyz` 探針、`GET /openapi.json` 規格、以及 `GET /progress`（以 WebSocket 推送即時傳輸快照） - **HTMX Web UI** — `start_web_ui()` 啟動唯讀觀測儀表板（health、progress、registry），以 HTML 片段輪詢；僅用標準函式庫 HTTP，搭配一支帶 SRI 的 CDN 腳本 - **MCP（Model Context Protocol）伺服器** — `MCPServer` 透過 stdio 上的 JSON-RPC 2.0（行分隔 JSON）將登錄表橋接到任何 MCP 主機（Claude Desktop、MCP CLI）；每個 `FA_*` 動作都會自動生成輸入 schema 並成為 MCP 工具 -- **通用儲存層** — `File` / `Storage` 以同一套 URI 語法（`local:///…`、`memory://…`）、同一份 `StorageBackend` 契約與同一組例外階層存取本機與遠端儲存；內建 `LocalStorage` 與 `MemoryStorage`，並附 70 個案例的契約測試套件可檢查任何後端 +- **通用儲存層** — `File` / `Storage` 以同一套 URI 語法（`local:///…`、`s3://…`、`azure://…`、`memory://…`）、同一份 `StorageBackend` 契約與同一組例外階層存取本機與遠端儲存；內建本機、S3、Azure Blob 與記憶體後端，並附 70 個案例的契約測試套件可檢查任何後端 - PySide6 GUI（`python -m automation_file ui`）每個後端一個分頁，含 JSON 動作執行器，另有 Triggers、Scheduler、即時 Progress 專屬分頁 - 功能豐富的 CLI，包含一次性子指令與舊式 JSON 批次旗標 - 專案鷹架（`ProjectBuilder`）協助建立以 executor 為核心的自動化專案 @@ -146,9 +146,9 @@ flowchart TD end subgraph StorageLayer["通用儲存層"] - FileAPI["File · Storage
local:// memory:// …"] + FileAPI["File · Storage
local:// memory:// s3:// azure://"] Resolver["StorageResolver
mounts · scheme factories"] - Backends["StorageBackend contract
LocalStorage · MemoryStorage"] + Backends["StorageBackend contract
Local · Memory · S3 · Azure"] end subgraph Notify["通知"] @@ -187,6 +187,8 @@ flowchart TD Resolver ==> Backends Backends ==> SafeP Backends ==> Check + Backends ==> S3M + Backends ==> Azure TCP ==> Executor HTTPS ==> Executor @@ -468,10 +470,14 @@ File("sandbox://jobs/42/out.csv").write(b"done") `StorageAlreadyExistsException`、`StoragePathTypeException`、`StorageNotEmptyException`、 `StoragePermissionException`、`StorageTransientException`、`StorageUnavailableException`、 `StorageUnsupportedException`、`StorageURIException`。 -- **目前的後端** — `LocalStorage`（`local://`，可透過 `safe_join` 限制在某個根目錄內）與 - `MemoryStorage`（`memory://`，用於測試與試跑）。雲端與 SFTP 後端目前仍透過各自的用戶端與 - `FA_*` 動作使用，接上本層的轉接器尚未完成。你可以繼承 `StorageBackend` 撰寫自己的後端， - 並用 `tests/storage_contract.py` 中 70 個案例的契約測試套件檢查。 +- **目前的後端** — `LocalStorage`（`local://`，可透過 `safe_join` 限制在某個根目錄內）、 + `S3Storage`（`s3://bucket/key`）、`AzureStorage`（`azure://container/blob`）與 + `MemoryStorage`（`memory://`，用於測試與試跑）。S3 與 Azure 使用你原本就會初始化的用戶端 + （`s3_instance.later_init(...)`、`azure_blob_instance.later_init(...)`），兩者都就緒後， + `File("s3://reports/q1.csv").copy_to("azure://backups/q1.csv")` 即可運作。 + Google Drive、Dropbox、SFTP、FTP、WebDAV、SMB 與 fsspec 目前仍透過各自的用戶端與 `FA_*` + 動作使用，其轉接器尚未完成。你可以繼承 `StorageBackend`（物件儲存則繼承 `ObjectStorage`） + 撰寫自己的後端，並用 `tests/storage_contract.py` 中 70 個案例的契約測試套件檢查。 此 API 為新功能，在 1.0 之前仍可能調整。完整說明請見文件的「通用儲存層」章節。 diff --git a/architecture.md b/architecture.md index 2761d94..f41fee8 100644 --- a/architecture.md +++ b/architecture.md @@ -24,7 +24,7 @@ piece of the 1.0 roadmap (`docs/FILEAUTOMATION-1.0-ROADMAP.md`, PR #107); what i | `automation_file/core/` | Engine, on je_action_core: `action_registry.py` (`ActionRegistry`, a `CommandRegistry`; `build_default_registry`), `action_executor.py` (`ActionExecutor`, an `ActionExecutor` with strict actions, indexed records and the dry-run, validate, substitute and parallel extras; shared `executor`), `callback_executor.py`, `package_loader.py`, `plugins.py`, `dag_executor.py`, `action_queue.py`, `json_store.py`, `substitution.py`. Also cross-cutting helpers: `retry`, `quota`, `rate_limit`, `circuit_breaker`, `file_lock`, `sqlite_lock`, `checksum`, `manifest`, `crypto`, `secrets`, `config`, `config_watcher`, `audit`, `metrics`, `tracing`, `progress`, `fim`, `content_store` | | `automation_file/local/` | Local strategy modules: file, dir, zip, tar and archive ops, sync, diff, text/JSON/data edits, templates, versioning, trash, `shell_ops` (argv-only subprocess), conditional branches. `safe_paths.py` guards against path traversal | | `automation_file/remote/` | `url_validator.py` (SSRF guard), `http_download.py`, `cross_backend.py`, `fsspec_bridge.py`. One subpackage per backend: `google_drive/`, `s3/`, `azure_blob/`, `dropbox_api/`, `sftp/`, `ftp/`, `onedrive/`, `box/`, each with `client.py`, `*_ops.py` and `register__ops`. `smb/` and `webdav/` have a client only | -| `automation_file/storage/` | Universal storage layer. `uri.py` (`StorageURI`, `parse_storage_uri`, `normalize_path`), `types.py` (`FileInfo`, `Checksum`, `StorageCapabilities`), `backend.py` (`StorageBackend`: the public operations are template methods over the `_`-prefixed primitives a backend supplies), `local_storage.py` (`LocalStorage`, confined through `safe_join` when given a root), `memory_storage.py` (`MemoryStorage`), `resolver.py` (`StorageResolver`, `default_resolver`: mounts first, then scheme factories), `file.py` (`File`), `storage.py` (`Storage`). It imports only `exceptions`, `core.checksum` and `local.safe_paths`: no registry, no GUI, no backend SDK | +| `automation_file/storage/` | Universal storage layer. `uri.py` (`StorageURI`, `parse_storage_uri`, `normalize_path`), `types.py` (`FileInfo`, `Checksum`, `StorageCapabilities`), `backend.py` (`StorageBackend`: the public operations are template methods over the `_`-prefixed primitives a backend supplies), `local_storage.py` (`LocalStorage`, confined through `safe_join` when given a root), `memory_storage.py` (`MemoryStorage`), `object_storage.py` (`ObjectStorage`: directories as key prefixes over `_head`, `_scan`, `_put`, `_get`, `_remove`), `s3_storage.py` (`S3Storage`, over `s3_instance` or a given boto3 client), `azure_storage.py` (`AzureStorage`, over `azure_blob_instance` or a given `BlobServiceClient`), `resolver.py` (`StorageResolver`, `default_resolver`: mounts first, then scheme factories), `file.py` (`File`), `storage.py` (`Storage`). At module level it imports only `exceptions`, `core.checksum` and `local.safe_paths`: no registry, no GUI, no backend SDK. The adapters import their SDK's exceptions and the shared client inside the functions that use them | | `automation_file/server/` | `tcp_server.py`, `http_server.py`, `mcp_server.py`, `web_ui.py`, `metrics_server.py`, `action_acl.py` (`ActionACL`), `network_guards.py` (`ensure_loopback`) | | `automation_file/client/` | `HTTPActionClient` for the HTTP action server | | `automation_file/trigger/`, `scheduler/`, `notify/` | Watchdog file triggers, cron scheduler, notification sinks. Each registers its own `FA_*` ops | @@ -47,9 +47,10 @@ piece of the 1.0 roadmap (`docs/FILEAUTOMATION-1.0-ROADMAP.md`, PR #107); what i `driver_instance` (Google Drive), `start_autocontrol_socket_server`, `start_http_action_server`, `HTTPActionClient`, `MCPServer`, `create_project_dir`, `launch_ui` (lazy). - **Storage layer** (same facade): `File`, `Storage`, `StorageBackend`, `StorageResolver`, `StorageURI`, - `parse_storage_uri`, `FileInfo`, `Checksum`, `StorageCapabilities`, `LocalStorage`, `MemoryStorage`, and + `parse_storage_uri`, `FileInfo`, `Checksum`, `StorageCapabilities`, `LocalStorage`, `MemoryStorage`, + `ObjectStorage`, `S3Storage`, `AzureStorage`, and `StorageException` with its nine subclasses. Storage URIs are `:///`; the built-in - schemes are `local` (alias `file`) and `memory`, and text without `://` is a local path. The API is + schemes are `local` (alias `file`), `memory`, `s3` and `azure` (alias `az`), and text without `://` is a local path. `s3://` and `azure://` use the shared `s3_instance` / `azure_blob_instance`. The API is provisional until 1.0. It is not reachable through `FA_*` actions yet. - **Action format**: an action is `[name]`, `[name, {kwargs}]` or `[name, [args]]`. A file holds a list of actions or `{"auto_control": [...]}`. @@ -141,7 +142,8 @@ copy_to / move_to → target_backend.copy_from(source_backend, ...) → native ( 1. Subclass `StorageBackend` in `storage/_storage.py`: set `scheme` and `capabilities`, implement `_stat`, `_list_dir`, `_upload`, `_download`, `_delete_file`, plus `_mkdir` and `_rmdir` when `capabilities.directories` is true. Map the SDK's errors to the `StorageException` subclasses and - import the SDK lazily. + import the SDK lazily. An object store subclasses `ObjectStorage` and implements `_head`, `_scan`, + `_put`, `_get`, `_remove` instead. 2. Register its factory in `register_default_schemes` (`storage/resolver.py`), or leave it to callers to `Storage.mount(...)` when it needs connection arguments. 3. Add `tests/test_storage_.py` with a `StorageContract` subclass (`tests/storage_contract.py`); diff --git a/automation_file/__init__.py b/automation_file/__init__.py index c7aa8e8..72d8584 100644 --- a/automation_file/__init__.py +++ b/automation_file/__init__.py @@ -266,11 +266,14 @@ ) from automation_file.server.web_ui import WebUIServer, start_web_ui from automation_file.storage import ( + AzureStorage, Checksum, File, FileInfo, LocalStorage, MemoryStorage, + ObjectStorage, + S3Storage, Storage, StorageBackend, StorageCapabilities, @@ -487,6 +490,9 @@ def __getattr__(name: str) -> Any: "StorageCapabilities", "LocalStorage", "MemoryStorage", + "ObjectStorage", + "S3Storage", + "AzureStorage", "StorageException", "StorageURIException", "StorageNotFoundException", diff --git a/automation_file/storage/__init__.py b/automation_file/storage/__init__.py index f6e228f..5b9f0a3 100644 --- a/automation_file/storage/__init__.py +++ b/automation_file/storage/__init__.py @@ -1,14 +1,16 @@ """Universal storage layer: one contract, one URI syntax, any backend. * :class:`File` and :class:`Storage` are the application API. -* :class:`StorageBackend` is the contract a backend implements; - :class:`LocalStorage` and :class:`MemoryStorage` are the built-in ones. +* :class:`StorageBackend` is the contract a backend implements. The built-in ones + are :class:`LocalStorage`, :class:`MemoryStorage`, :class:`S3Storage` and + :class:`AzureStorage`; :class:`ObjectStorage` is the shared base of the last two. * :class:`StorageURI` / :func:`parse_storage_uri` define the address syntax, and :class:`StorageResolver` maps an address to a backend. """ from __future__ import annotations +from automation_file.storage.azure_storage import AzureStorage from automation_file.storage.backend import StorageBackend from automation_file.storage.file import File from automation_file.storage.local_storage import LocalStorage @@ -17,12 +19,14 @@ clear_memory_stores, memory_store, ) +from automation_file.storage.object_storage import ObjectStorage from automation_file.storage.resolver import ( BackendFactory, StorageResolver, default_resolver, register_default_schemes, ) +from automation_file.storage.s3_storage import S3Storage from automation_file.storage.storage import Storage from automation_file.storage.types import Checksum, FileInfo, StorageCapabilities from automation_file.storage.uri import ( @@ -34,12 +38,15 @@ ) __all__ = [ + "AzureStorage", "BackendFactory", "Checksum", "File", "FileInfo", "LocalStorage", "MemoryStorage", + "ObjectStorage", + "S3Storage", "Storage", "StorageBackend", "StorageCapabilities", diff --git a/automation_file/storage/azure_storage.py b/automation_file/storage/azure_storage.py new file mode 100644 index 0000000..e372b57 --- /dev/null +++ b/automation_file/storage/azure_storage.py @@ -0,0 +1,187 @@ +"""Azure Blob backend: ``azure:///`` (alias ``az://``). + +``AzureStorage("backups")`` serves one container through the shared +:data:`~automation_file.remote.azure_blob.client.azure_blob_instance`, which the +caller initialises as before (``azure_blob_instance.later_init(...)`` or +``FA_azure_blob_later_init``). Pass ``service=`` to use another +``BlobServiceClient``, for another account or the Azurite emulator. + +Uploads set the blob's content type from its name. ``stat`` reports the size, +modification time, ETag, content type, version ID and metadata of the blob. +""" + +from __future__ import annotations + +import contextlib +from collections.abc import Iterable, Iterator +from pathlib import Path +from typing import Any + +from automation_file.exceptions import ( + StorageException, + StorageNotFoundException, + StoragePermissionException, + StorageTransientException, + StorageUnavailableException, + StorageURIException, +) +from automation_file.storage.backend import guess_content_type, missing_error +from automation_file.storage.object_storage import ObjectStorage +from automation_file.storage.types import FileInfo, StorageCapabilities +from automation_file.storage.uri import StorageURI + +AZURE_SCHEME = "azure" +_DENIED_STATUS = frozenset({401, 403}) +_MISSING_STATUS = frozenset({404}) +_TRANSIENT_STATUS = frozenset({408, 429, 500, 502, 503, 504}) +_NOT_INSTALLED = "azure-storage-blob is not installed; the Azure Blob backend needs it" + + +@contextlib.contextmanager +def _azure_errors(location: str) -> Iterator[None]: + """Turn azure-core errors into the storage layer's exceptions.""" + try: + from azure.core import exceptions as azure_errors + except ImportError as error: + raise StorageUnavailableException(_NOT_INSTALLED) from error + try: + yield + except azure_errors.ResourceNotFoundError as error: + raise missing_error(location) from error + except azure_errors.ClientAuthenticationError as error: + raise StoragePermissionException(f"access to {location} was denied") from error + except azure_errors.HttpResponseError as error: + status = getattr(error, "status_code", None) + if status in _MISSING_STATUS: + raise missing_error(location) from error + if status in _DENIED_STATUS: + raise StoragePermissionException( + f"access to {location} was denied ({status})" + ) from error + if status in _TRANSIENT_STATUS: + raise StorageTransientException(f"{location}: Azure answered {status}") from error + raise StorageException(f"{location}: Azure error {status or 'unknown'}") from error + except (azure_errors.ServiceRequestError, azure_errors.ServiceResponseError) as error: + raise StorageTransientException(f"{location}: {type(error).__name__}") from error + except azure_errors.AzureError as error: + raise StorageException(f"{location}: {type(error).__name__}") from error + + +def _blob_info(properties: Any) -> FileInfo: + settings = getattr(properties, "content_settings", None) + etag = getattr(properties, "etag", None) + return FileInfo( + path=properties.name, + size=int(getattr(properties, "size", 0) or 0), + modified_at=getattr(properties, "last_modified", None), + etag=str(etag).strip('"') if etag else None, + version=getattr(properties, "version_id", None), + content_type=getattr(settings, "content_type", None), + metadata=dict(getattr(properties, "metadata", None) or {}), + ) + + +class AzureStorage(ObjectStorage): + """One Azure Blob container, or the blobs below one prefix of it.""" + + scheme = AZURE_SCHEME + capabilities = StorageCapabilities( + directories=False, + modified_at=True, + etag=True, + version=True, + content_type=True, + metadata=True, + ) + + def __init__(self, container: str, *, service: Any = None, prefix: str = "") -> None: + if not container: + raise StorageURIException("an Azure Blob storage needs a container name") + super().__init__(prefix) + self._container = container + self._explicit_service = service + + @property + def container(self) -> str: + return self._container + + @property + def _service(self) -> Any: + if self._explicit_service is not None: + return self._explicit_service + from automation_file.remote.azure_blob.client import azure_blob_instance + + try: + return azure_blob_instance.require_service() + except RuntimeError as error: + raise StorageUnavailableException( + "the Azure Blob client is not initialised; call azure_blob_instance.later_init() " + "or pass service= to AzureStorage" + ) from error + + def uri_for(self, path: str = "") -> str: + key = self._key(self._normalize(path)) + return str(StorageURI(AZURE_SCHEME, self._container, key)) + + def _location(self, key: str) -> str: + return f"{AZURE_SCHEME}://{self._container}/{key}" + + def _blob(self, key: str) -> Any: + return self._service.get_blob_client(container=self._container, blob=key) + + def _head(self, key: str) -> FileInfo | None: + try: + with _azure_errors(self._location(key)): + properties = self._blob(key).get_blob_properties() + except StorageNotFoundException: + return None + return _blob_info(properties) + + def _scan(self, key_prefix: str, *, shallow: bool) -> Iterable[FileInfo]: + found: list[FileInfo] = [] + with _azure_errors(self._location(key_prefix)): + container = self._service.get_container_client(self._container) + starts_with = key_prefix or None + if not shallow: + return [ + _blob_info(item) for item in container.list_blobs(name_starts_with=starts_with) + ] + for item in container.walk_blobs(name_starts_with=starts_with, delimiter="/"): + # A deeper level arrives as a prefix whose name ends with the delimiter. + if item.name.endswith("/") and item.name != key_prefix: + found.append(FileInfo(path=item.name.rstrip("/"), is_dir=True)) + else: + found.append(_blob_info(item)) + return found + + def _put(self, source: Path, key: str) -> None: + try: + from azure.storage.blob import ContentSettings + except ImportError as error: + raise StorageUnavailableException(_NOT_INSTALLED) from error + content_type = guess_content_type(key) + settings = ContentSettings(content_type=content_type) if content_type else None + with _azure_errors(self._location(key)), open(source, "rb") as handle: + self._blob(key).upload_blob(handle, overwrite=True, content_settings=settings) + + def _get(self, key: str, target: Path) -> None: + with _azure_errors(self._location(key)), open(target, "wb") as handle: + self._blob(key).download_blob().readinto(handle) + + def _remove(self, key: str) -> None: + with _azure_errors(self._location(key)): + self._blob(key).delete_blob() + + def __eq__(self, other: object) -> bool: + return ( + isinstance(other, AzureStorage) + and other._container == self._container + and other._prefix == self._prefix + and other._explicit_service is self._explicit_service + ) + + def __hash__(self) -> int: + return hash((AZURE_SCHEME, self._container, self._prefix, id(self._explicit_service))) + + def __repr__(self) -> str: + return f"AzureStorage({self._container!r}, prefix={self._prefix!r})" diff --git a/automation_file/storage/backend.py b/automation_file/storage/backend.py index 2c05f65..c937eac 100644 --- a/automation_file/storage/backend.py +++ b/automation_file/storage/backend.py @@ -17,12 +17,13 @@ from __future__ import annotations import hashlib +import mimetypes import os import tempfile import uuid from abc import ABC, abstractmethod from collections.abc import Iterable -from pathlib import Path +from pathlib import Path, PurePosixPath from types import TracebackType from typing import ClassVar, TypeVar @@ -64,6 +65,12 @@ def join_path(directory: str, name: str) -> str: return f"{directory}/{name}" if directory else name +def guess_content_type(name: str) -> str | None: + """Guess a MIME type from the suffixes of ``name``.""" + # Only the suffixes are passed on: guess_type() parses its argument as a URL. + return mimetypes.guess_type("_" + "".join(PurePosixPath(name).suffixes))[0] + + def missing_error(location: str) -> StorageNotFoundException: return StorageNotFoundException(f"{location} does not exist") @@ -398,7 +405,7 @@ def _transfer_paths( if info.is_dir: raise not_a_file_error(source.uri_for(info.path)) target = self._normalize(path) - if source is self and info.path == target: + if source == self and info.path == target: raise StorageException(f"{self.uri_for(target)}: source and target are the same file") target = self._writable_file(target, overwrite) self._make_parents(target) diff --git a/automation_file/storage/local_storage.py b/automation_file/storage/local_storage.py index 927f8af..5f3e680 100644 --- a/automation_file/storage/local_storage.py +++ b/automation_file/storage/local_storage.py @@ -17,7 +17,6 @@ import contextlib import errno -import mimetypes import os import re import shutil @@ -25,13 +24,14 @@ import uuid from collections.abc import Iterable, Iterator from datetime import datetime, timezone -from pathlib import Path, PurePosixPath +from pathlib import Path from automation_file.core.checksum import file_checksum from automation_file.exceptions import StorageException, StoragePermissionException from automation_file.local.safe_paths import safe_join from automation_file.storage.backend import ( StorageBackend, + guess_content_type, join_path, missing_error, not_empty_error, @@ -65,11 +65,6 @@ def _anchored(path: str) -> Path: return Path(f"/{path}") -def _content_type(name: str) -> str | None: - # Only the suffixes are passed on: guess_type() parses its argument as a URL. - return mimetypes.guess_type("_" + "".join(PurePosixPath(name).suffixes))[0] - - def _file_info(path: str, result: os.stat_result) -> FileInfo: is_dir = stat.S_ISDIR(result.st_mode) return FileInfo( @@ -77,7 +72,7 @@ def _file_info(path: str, result: os.stat_result) -> FileInfo: is_dir=is_dir, size=None if is_dir else result.st_size, modified_at=datetime.fromtimestamp(result.st_mtime, tz=timezone.utc), - content_type=None if is_dir else _content_type(path), + content_type=None if is_dir else guess_content_type(path), ) diff --git a/automation_file/storage/object_storage.py b/automation_file/storage/object_storage.py new file mode 100644 index 0000000..b234986 --- /dev/null +++ b/automation_file/storage/object_storage.py @@ -0,0 +1,134 @@ +"""Shared behaviour of object stores: flat keys, where a directory is only a key prefix. + +:class:`ObjectStorage` turns the :class:`StorageBackend` primitives into five +calls on a key/value store -- ``_head``, ``_scan``, ``_put``, ``_get`` and +``_remove`` -- so an adapter for S3, Azure Blob or a similar service is a thin +translation of those calls and of the service's errors. + +* A directory exists while a key lies below it; ``mkdir`` creates nothing. +* A key that ends with ``/`` is a placeholder some tools write to make an empty + "folder" visible. It is never listed as a file, and deleting the directory + removes it too. +* ``prefix`` confines the backend to the keys below one prefix of the container. +""" + +from __future__ import annotations + +from abc import abstractmethod +from collections.abc import Iterable +from dataclasses import replace +from pathlib import Path + +from automation_file.exceptions import StorageNotFoundException +from automation_file.storage.backend import StorageBackend, join_path, not_empty_error, parent_of +from automation_file.storage.types import FileInfo, StorageCapabilities +from automation_file.storage.uri import normalize_path + +_PLACEHOLDER_SUFFIX = "/" + + +def implied_directories(paths: Iterable[str], base: str) -> list[FileInfo]: + """Return the directories that lie between ``base`` and each of ``paths``.""" + found: set[str] = set() + for path in paths: + parent = parent_of(path) + while parent != base and parent not in found: + found.add(parent) + parent = parent_of(parent) + return [FileInfo(path=directory, is_dir=True) for directory in sorted(found)] + + +class ObjectStorage(StorageBackend): + """A :class:`StorageBackend` over a flat key space.""" + + capabilities = StorageCapabilities(directories=False) + + def __init__(self, prefix: str = "") -> None: + self._prefix = normalize_path(prefix) + + @property + def prefix(self) -> str: + """The key prefix this backend is confined to (empty for the whole container).""" + return self._prefix + + # ------------------------------------------------------------------ the key/value store + + @abstractmethod + def _head(self, key: str) -> FileInfo | None: + """Return the object at exactly ``key`` (``FileInfo.path`` is free), or ``None``.""" + + @abstractmethod + def _scan(self, key_prefix: str, *, shallow: bool) -> Iterable[FileInfo]: + """Yield the objects whose key starts with ``key_prefix``; ``FileInfo.path`` is the key. + + With ``shallow=True`` only one level is returned: the objects directly below + the prefix, and one ``FileInfo(is_dir=True)`` per deeper prefix, its path + without the trailing ``/``. + """ + + @abstractmethod + def _put(self, source: Path, key: str) -> None: + """Store the local file ``source`` as the object ``key``.""" + + @abstractmethod + def _get(self, key: str, target: Path) -> None: + """Write the object ``key`` to the local file ``target``.""" + + @abstractmethod + def _remove(self, key: str) -> None: + """Delete the object ``key``.""" + + # ------------------------------------------------------------------ StorageBackend primitives + + def _key(self, path: str) -> str: + return join_path(self._prefix, path) if path else self._prefix + + def _below(self, path: str) -> str: + key = self._key(path) + return f"{key}/" if key else "" + + def _visible(self, info: FileInfo) -> FileInfo: + """Re-express a key as a path relative to this backend's prefix.""" + return replace(info, path=info.path[len(self._prefix) + 1 :]) if self._prefix else info + + def _stat(self, path: str) -> FileInfo | None: + if not path: + return FileInfo(path="", is_dir=True) + info = self._head(self._key(path)) + if info is not None: + return replace(info, path=path, is_dir=False) + try: + has_entries = any(True for _ in self._scan(self._below(path), shallow=True)) + except StorageNotFoundException: + # The container itself is missing, so nothing exists inside it. + return None + return FileInfo(path=path, is_dir=True) if has_entries else None + + def _list_dir(self, path: str) -> Iterable[FileInfo]: + return [ + self._visible(info) + for info in self._scan(self._below(path), shallow=True) + if not info.path.endswith(_PLACEHOLDER_SUFFIX) + ] + + def _walk(self, path: str) -> Iterable[FileInfo]: + entries = [self._visible(info) for info in self._scan(self._below(path), shallow=False)] + files = [info for info in entries if not info.path.endswith(_PLACEHOLDER_SUFFIX)] + # A placeholder's parent is the directory it marks, so it implies that directory too. + return [*implied_directories((info.path for info in entries), path), *files] + + def _upload(self, source: Path, path: str) -> None: + self._put(source, self._key(path)) + + def _download(self, path: str, target: Path) -> None: + self._get(self._key(path), target) + + def _delete_file(self, path: str) -> None: + self._remove(self._key(path)) + + def _delete_directory(self, path: str, recursive: bool) -> None: + keys = [info.path for info in self._scan(self._below(path), shallow=False)] + if not recursive and any(not key.endswith(_PLACEHOLDER_SUFFIX) for key in keys): + raise not_empty_error(self.uri_for(path)) + for key in keys: + self._remove(key) diff --git a/automation_file/storage/resolver.py b/automation_file/storage/resolver.py index 3a2ee53..357ebf0 100644 --- a/automation_file/storage/resolver.py +++ b/automation_file/storage/resolver.py @@ -21,9 +21,11 @@ from collections.abc import Callable from automation_file.exceptions import StorageURIException +from automation_file.storage.azure_storage import AZURE_SCHEME, AzureStorage from automation_file.storage.backend import StorageBackend from automation_file.storage.local_storage import LocalStorage from automation_file.storage.memory_storage import MEMORY_SCHEME, memory_store +from automation_file.storage.s3_storage import S3_SCHEME, S3Storage from automation_file.storage.types import StorageCapabilities from automation_file.storage.uri import ( LOCAL_SCHEME, @@ -143,10 +145,28 @@ def _memory_factory(uri: StorageURI) -> tuple[StorageBackend, str]: return memory_store(uri.authority), uri.path +def _container_of(uri: StorageURI, kind: str) -> str: + if not uri.authority: + raise StorageURIException( + f"{str(uri)!r} names no {kind}; write '{uri.scheme}://<{kind}>/'" + ) + return uri.authority + + +def _s3_factory(uri: StorageURI) -> tuple[StorageBackend, str]: + return S3Storage(_container_of(uri, "bucket")), uri.path + + +def _azure_factory(uri: StorageURI) -> tuple[StorageBackend, str]: + return AzureStorage(_container_of(uri, "container")), uri.path + + def register_default_schemes(resolver: StorageResolver) -> None: """Register the factory of every built-in backend on ``resolver``.""" resolver.register_scheme(LOCAL_SCHEME, _local_factory) resolver.register_scheme(MEMORY_SCHEME, _memory_factory) + resolver.register_scheme(S3_SCHEME, _s3_factory) + resolver.register_scheme(AZURE_SCHEME, _azure_factory) default_resolver = StorageResolver() diff --git a/automation_file/storage/s3_storage.py b/automation_file/storage/s3_storage.py new file mode 100644 index 0000000..7316bcb --- /dev/null +++ b/automation_file/storage/s3_storage.py @@ -0,0 +1,211 @@ +"""S3 backend: ``s3:///``. + +``S3Storage("reports")`` serves one bucket through the shared +:data:`~automation_file.remote.s3.client.s3_instance`, which the caller +initialises as before (``s3_instance.later_init(...)`` or ``FA_s3_later_init``). +Pass ``client=`` to use another boto3 S3 client, for another account or an +S3-compatible endpoint such as MinIO. + +Uploads set ``ContentType`` from the key's suffix. ``stat`` reports the size, +modification time, ETag, content type, version ID and user metadata that +``HeadObject`` returns. Checksums are computed from the content: an S3 ETag is +not a digest of a multipart upload, so it is never used as one. +""" + +from __future__ import annotations + +import contextlib +from collections.abc import Iterable, Iterator +from pathlib import Path +from typing import Any + +from automation_file.exceptions import ( + StorageException, + StorageNotFoundException, + StoragePermissionException, + StorageTransientException, + StorageUnavailableException, + StorageURIException, +) +from automation_file.storage.backend import ( + StorageBackend, + guess_content_type, + missing_error, +) +from automation_file.storage.object_storage import ObjectStorage +from automation_file.storage.types import FileInfo, StorageCapabilities +from automation_file.storage.uri import StorageURI + +S3_SCHEME = "s3" +_MISSING_CODES = frozenset({"404", "NoSuchKey", "NoSuchBucket", "NotFound"}) +_DENIED_CODES = frozenset( + {"403", "AccessDenied", "Forbidden", "InvalidAccessKeyId", "SignatureDoesNotMatch"} +) +_TRANSIENT_CODES = frozenset( + { + "500", + "502", + "503", + "504", + "InternalError", + "RequestTimeout", + "ServiceUnavailable", + "SlowDown", + "Throttling", + "ThrottlingException", + } +) + + +def _error_code(error: Any) -> str: + response = getattr(error, "response", None) or {} + return str(response.get("Error", {}).get("Code", "")) + + +@contextlib.contextmanager +def _s3_errors(location: str) -> Iterator[None]: + """Turn boto3 and botocore errors into the storage layer's exceptions.""" + try: + from boto3.exceptions import Boto3Error + from botocore import exceptions as botocore_errors + except ImportError as error: + raise StorageUnavailableException( + "boto3 is not installed; the S3 backend needs it" + ) from error + try: + yield + except botocore_errors.ClientError as error: + code = _error_code(error) + if code in _MISSING_CODES: + raise missing_error(location) from error + if code in _DENIED_CODES: + raise StoragePermissionException(f"access to {location} was denied ({code})") from error + if code in _TRANSIENT_CODES: + raise StorageTransientException(f"{location}: S3 answered {code}") from error + raise StorageException(f"{location}: S3 error {code or 'unknown'}") from error + except (botocore_errors.ConnectionError, botocore_errors.HTTPClientError) as error: + raise StorageTransientException(f"{location}: {type(error).__name__}") from error + except (botocore_errors.BotoCoreError, Boto3Error) as error: + raise StorageException(f"{location}: {type(error).__name__}") from error + + +def _etag(raw: Any) -> str | None: + return str(raw).strip('"') if raw else None + + +def _listed(entry: dict[str, Any]) -> FileInfo: + return FileInfo( + path=entry["Key"], + size=int(entry.get("Size", 0)), + modified_at=entry.get("LastModified"), + etag=_etag(entry.get("ETag")), + ) + + +class S3Storage(ObjectStorage): + """One S3 bucket, or the keys below one prefix of it.""" + + scheme = S3_SCHEME + capabilities = StorageCapabilities( + directories=False, + modified_at=True, + etag=True, + version=True, + content_type=True, + metadata=True, + ) + + def __init__(self, bucket: str, *, client: Any = None, prefix: str = "") -> None: + if not bucket: + raise StorageURIException("an S3 storage needs a bucket name") + super().__init__(prefix) + self._bucket = bucket + self._explicit_client = client + + @property + def bucket(self) -> str: + return self._bucket + + @property + def _client(self) -> Any: + if self._explicit_client is not None: + return self._explicit_client + from automation_file.remote.s3.client import s3_instance + + try: + return s3_instance.require_client() + except RuntimeError as error: + raise StorageUnavailableException( + "the S3 client is not initialised; call s3_instance.later_init() " + "or pass client= to S3Storage" + ) from error + + def uri_for(self, path: str = "") -> str: + key = self._key(self._normalize(path)) + return str(StorageURI(S3_SCHEME, self._bucket, key)) + + def _head(self, key: str) -> FileInfo | None: + try: + with _s3_errors(f"{S3_SCHEME}://{self._bucket}/{key}"): + head = self._client.head_object(Bucket=self._bucket, Key=key) + except StorageNotFoundException: + return None + return FileInfo( + path=key, + size=int(head.get("ContentLength", 0)), + modified_at=head.get("LastModified"), + etag=_etag(head.get("ETag")), + version=head.get("VersionId"), + content_type=head.get("ContentType"), + metadata=dict(head.get("Metadata") or {}), + ) + + def _scan(self, key_prefix: str, *, shallow: bool) -> Iterable[FileInfo]: + arguments = {"Bucket": self._bucket, "Prefix": key_prefix} + if shallow: + arguments["Delimiter"] = "/" + found: list[FileInfo] = [] + with _s3_errors(f"{S3_SCHEME}://{self._bucket}/{key_prefix}"): + for page in self._client.get_paginator("list_objects_v2").paginate(**arguments): + found.extend( + FileInfo(path=item["Prefix"].rstrip("/"), is_dir=True) + for item in page.get("CommonPrefixes", []) + ) + found.extend(_listed(entry) for entry in page.get("Contents", [])) + return found + + def _put(self, source: Path, key: str) -> None: + content_type = guess_content_type(key) + extra = {"ContentType": content_type} if content_type else None + with _s3_errors(f"{S3_SCHEME}://{self._bucket}/{key}"): + self._client.upload_file(str(source), self._bucket, key, ExtraArgs=extra) + + def _get(self, key: str, target: Path) -> None: + with _s3_errors(f"{S3_SCHEME}://{self._bucket}/{key}"): + self._client.download_file(self._bucket, key, str(target)) + + def _remove(self, key: str) -> None: + with _s3_errors(f"{S3_SCHEME}://{self._bucket}/{key}"): + self._client.delete_object(Bucket=self._bucket, Key=key) + + def _copy_from(self, source: StorageBackend, source_path: str, path: str) -> bool: + if not isinstance(source, S3Storage) or source._client is not self._client: + return False + origin = {"Bucket": source.bucket, "Key": source._key(source_path)} + with _s3_errors(self.uri_for(path)): + self._client.copy(origin, self._bucket, self._key(path)) + return True + + def __eq__(self, other: object) -> bool: + return ( + isinstance(other, S3Storage) + and other._bucket == self._bucket + and other._prefix == self._prefix + and other._explicit_client is self._explicit_client + ) + + def __hash__(self) -> int: + return hash((S3_SCHEME, self._bucket, self._prefix, id(self._explicit_client))) + + def __repr__(self) -> str: + return f"S3Storage({self._bucket!r}, prefix={self._prefix!r})" diff --git a/docs/source/API/storage.rst b/docs/source/API/storage.rst index 951cd82..e50154b 100644 --- a/docs/source/API/storage.rst +++ b/docs/source/API/storage.rst @@ -44,3 +44,16 @@ Built-in backends .. automodule:: automation_file.storage.memory_storage :members: + +Object stores +------------- + +.. automodule:: automation_file.storage.object_storage + :members: + :private-members: _head, _scan, _put, _get, _remove + +.. automodule:: automation_file.storage.s3_storage + :members: + +.. automodule:: automation_file.storage.azure_storage + :members: diff --git a/docs/source/Eng/usage/storage.rst b/docs/source/Eng/usage/storage.rst index bcd5a1b..12c5d70 100644 --- a/docs/source/Eng/usage/storage.rst +++ b/docs/source/Eng/usage/storage.rst @@ -12,7 +12,7 @@ The ``FA_*`` actions and the per-backend functions (``s3_upload_file``, .. note:: The layer is new and its API may still change before 1.0. The local - filesystem and an in-memory store are built in today. S3, Azure Blob, Google + filesystem, an in-memory store, S3 and Azure Blob are built in today. Google Drive, Dropbox, SFTP, FTP, WebDAV, SMB and fsspec are reached through their existing clients and actions (:doc:`cloud`) until their adapters land; you can already put any of them behind the layer by writing a backend @@ -198,6 +198,38 @@ Built-in backends A thread-safe tree held in memory, for tests, dry runs and examples. Each ```` is a separate store, created on first use. +``S3Storage`` (``s3:///``) + One bucket through the shared ``s3_instance``, initialised as before with + ``s3_instance.later_init(...)`` or ``FA_s3_later_init``. + ``S3Storage(bucket, client=...)`` takes another boto3 client (another + account, MinIO), and ``prefix=`` confines the backend to the keys below one + prefix. Uploads set ``ContentType`` from the key's suffix. ``stat`` reports + size, modification time, ETag, content type, version ID and metadata. A copy + between two S3 locations that share a client is done by S3 itself. + +``AzureStorage`` (``azure:///``, alias ``az://``) + One container through the shared ``azure_blob_instance`` + (``azure_blob_instance.later_init(...)`` or ``FA_azure_blob_later_init``), or + ``AzureStorage(container, service=...)`` for another ``BlobServiceClient`` + such as the Azurite emulator. ``prefix=`` works as for S3, and ``stat`` + reports the same fields. + +S3 and Azure Blob are object stores. A directory exists only while a key lies +below it, so ``mkdir`` creates nothing and an empty directory cannot exist +(``capabilities.directories`` is ``False``). A key ending in ``/`` that another +tool wrote as a folder placeholder is shown as a directory, never as a file. +Checksums are computed from the content and not taken from the ETag, which is +not a digest of a multipart upload. Until the client is initialised, every call +raises ``StorageUnavailableException``. + +.. code-block:: python + + from automation_file import File, azure_blob_instance, s3_instance + + s3_instance.later_init(region_name="us-east-1") + azure_blob_instance.later_init(connection_string=connection_string) + File("s3://reports/2026/q1.csv").copy_to("azure://backups/2026/q1.csv") + Mounting and registering backends --------------------------------- @@ -219,7 +251,7 @@ handle every URI no mount claimed. Storage.register_scheme("vault", lambda uri: (vault_backend(uri.authority), uri.path)) Storage.resolve("sandbox://jobs/42/out.csv") # (LocalStorage('/srv/jobs'), '42/out.csv') - Storage.schemes() # ['local', 'memory', 'sandbox', 'vault'] + Storage.schemes() # ['azure', 'local', 'memory', 's3', 'sandbox', 'vault'] ``Storage.mount`` / ``unmount`` / ``register_scheme`` / ``schemes`` / ``resolve`` work on the process-wide table. A private table is a @@ -256,6 +288,11 @@ already there, raise the shared exceptions and create parent directories. # _mkdir(path), _rmdir(path) # Optional overrides: _walk, _copy_from, _move_from, _checksum, _read_bytes +For an object store, subclass :class:`~automation_file.ObjectStorage` instead and +implement ``_head``, ``_scan``, ``_put``, ``_get`` and ``_remove``. It supplies +the directory behaviour described under `Built-in backends`_, and is what +``S3Storage`` and ``AzureStorage`` are built on. + Check it with the contract suite. ``tests/storage_contract.py`` holds 70 cases — nested directories, empty and large files, Unicode paths, binary data, overwrite and missing-path behaviour, path normalisation, copy and move — and reads diff --git a/docs/source/Zh-CN/usage/storage.rst b/docs/source/Zh-CN/usage/storage.rst index 2434d30..af804f1 100644 --- a/docs/source/Zh-CN/usage/storage.rst +++ b/docs/source/Zh-CN/usage/storage.rst @@ -10,10 +10,10 @@ API；:class:`~automation_file.StorageBackend` 则是后端需要实现的契约 .. note:: - 本层是新功能，API 在 1.0 之前仍可能调整。目前内置本地文件系统与内存存储两种 - 后端。S3、Azure Blob、Google Drive、Dropbox、SFTP、FTP、WebDAV、SMB 与 fsspec - 在各自的适配器完成之前，仍通过已有的客户端与动作使用（见 :doc:`cloud`）；你也 - 可以现在就自行编写后端，把它们接到本层之后（见 `编写后端`_）。 + 本层是新功能，API 在 1.0 之前仍可能调整。目前内置本地文件系统、内存存储、 + S3 与 Azure Blob 四种后端。Google Drive、Dropbox、SFTP、FTP、WebDAV、SMB 与 + fsspec 在各自的适配器完成之前，仍通过已有的客户端与动作使用（见 :doc:`cloud`）； + 你也可以现在就自行编写后端，把它们接到本层之后（见 `编写后端`_）。 快速开始 -------- @@ -185,6 +185,35 @@ API；:class:`~automation_file.StorageBackend` 则是后端需要实现的契约 保存在内存中的线程安全目录树，用于测试、试运行与示例。每个 ```` 是 独立的存储，首次使用时创建。 +``S3Storage``（``s3:///``） + 通过共用的 ``s3_instance`` 访问一个 bucket，初始化方式与以往相同： + ``s3_instance.later_init(...)`` 或 ``FA_s3_later_init``。 + ``S3Storage(bucket, client=...)`` 可以改用另一个 boto3 客户端（其他账号、 + MinIO），``prefix=`` 则把后端限制在某个前缀之下的 key。上传时会按 key 的 + 扩展名设置 ``ContentType``。``stat`` 报告大小、修改时间、ETag、内容类型、 + 版本 ID 与元数据。共用同一个客户端的两个 S3 位置之间的复制由 S3 本身完成。 + +``AzureStorage``（``azure:///``，别名 ``az://``） + 通过共用的 ``azure_blob_instance`` 访问一个 container + （``azure_blob_instance.later_init(...)`` 或 ``FA_azure_blob_later_init``）， + 或以 ``AzureStorage(container, service=...)`` 改用另一个 + ``BlobServiceClient``，例如 Azurite 模拟器。``prefix=`` 的用法与 S3 相同， + ``stat`` 报告的字段也相同。 + +S3 与 Azure Blob 都是对象存储。目录只在其下还有 key 时才存在，因此 ``mkdir`` 不会 +创建任何东西，空目录也无法存在（``capabilities.directories`` 为 ``False``）。其他 +工具写入、以 ``/`` 结尾的文件夹占位 key 会显示为目录，绝不会显示为文件。校验码是 +由内容计算而来，不取自 ETag，因为分段上传的 ETag 并不是摘要。客户端尚未初始化时， +每个调用都会抛出 ``StorageUnavailableException``。 + +.. code-block:: python + + from automation_file import File, azure_blob_instance, s3_instance + + s3_instance.later_init(region_name="us-east-1") + azure_blob_instance.later_init(connection_string=connection_string) + File("s3://reports/2026/q1.csv").copy_to("azure://backups/2026/q1.csv") + 挂载与注册后端 -------------- @@ -205,7 +234,7 @@ URI 的解析分两步。**挂载** 优先：挂载把一个后端实例绑定 Storage.register_scheme("vault", lambda uri: (vault_backend(uri.authority), uri.path)) Storage.resolve("sandbox://jobs/42/out.csv") # (LocalStorage('/srv/jobs'), '42/out.csv') - Storage.schemes() # ['local', 'memory', 'sandbox', 'vault'] + Storage.schemes() # ['azure', 'local', 'memory', 's3', 'sandbox', 'vault'] ``Storage.mount`` / ``unmount`` / ``register_scheme`` / ``schemes`` / ``resolve`` 操作的是整个进程共用的表。需要私有的表时使用 @@ -240,6 +269,10 @@ scheme 或 authority，并且只接受其下的 URI。 # _mkdir(path)、_rmdir(path) # 可选择重写：_walk、_copy_from、_move_from、_checksum、_read_bytes +如果是对象存储，请改为继承 :class:`~automation_file.ObjectStorage`，并实现 +``_head``、``_scan``、``_put``、``_get`` 与 ``_remove``。它提供 `内置后端`_ 一节 +所述的目录行为，``S3Storage`` 与 ``AzureStorage`` 都建立在它之上。 + 请用契约测试套件检查。``tests/storage_contract.py`` 包含 70 个用例——嵌套目录、 空文件与大文件、Unicode 路径、二进制数据、覆盖与路径不存在时的行为、路径规范化、 复制与移动——并在后端确实有差异之处读取 ``capabilities``： diff --git a/docs/source/Zh-TW/usage/storage.rst b/docs/source/Zh-TW/usage/storage.rst index 4a3667d..5956939 100644 --- a/docs/source/Zh-TW/usage/storage.rst +++ b/docs/source/Zh-TW/usage/storage.rst @@ -10,10 +10,10 @@ API；:class:`~automation_file.StorageBackend` 則是後端要實作的契約。 .. note:: - 本層是新功能，API 在 1.0 之前仍可能調整。目前內建本機檔案系統與記憶體儲存兩種 - 後端。S3、Azure Blob、Google Drive、Dropbox、SFTP、FTP、WebDAV、SMB 與 fsspec - 在各自的轉接器完成之前，仍透過既有的用戶端與動作使用（見 :doc:`cloud`）；你也 - 可以現在就自行撰寫後端，把它們接到本層之後（見 `撰寫後端`_）。 + 本層是新功能，API 在 1.0 之前仍可能調整。目前內建本機檔案系統、記憶體儲存、 + S3 與 Azure Blob 四種後端。Google Drive、Dropbox、SFTP、FTP、WebDAV、SMB 與 + fsspec 在各自的轉接器完成之前，仍透過既有的用戶端與動作使用（見 :doc:`cloud`）； + 你也可以現在就自行撰寫後端，把它們接到本層之後（見 `撰寫後端`_）。 快速開始 -------- @@ -185,6 +185,35 @@ API；:class:`~automation_file.StorageBackend` 則是後端要實作的契約。 存在記憶體中的執行緒安全目錄樹，用於測試、試跑與範例。每個 ```` 是 獨立的儲存，首次使用時建立。 +``S3Storage``（``s3:///``） + 透過共用的 ``s3_instance`` 存取一個 bucket，初始化方式與以往相同： + ``s3_instance.later_init(...)`` 或 ``FA_s3_later_init``。 + ``S3Storage(bucket, client=...)`` 可改用另一個 boto3 用戶端（其他帳號、 + MinIO），``prefix=`` 則把後端限制在某個前綴之下的 key。上傳時會依 key 的 + 副檔名設定 ``ContentType``。``stat`` 回報大小、修改時間、ETag、內容類型、 + 版本 ID 與中繼資料。共用同一個用戶端的兩個 S3 位置之間的複製由 S3 本身完成。 + +``AzureStorage``（``azure:///``，別名 ``az://``） + 透過共用的 ``azure_blob_instance`` 存取一個 container + （``azure_blob_instance.later_init(...)`` 或 ``FA_azure_blob_later_init``）， + 或以 ``AzureStorage(container, service=...)`` 改用另一個 + ``BlobServiceClient``，例如 Azurite 模擬器。``prefix=`` 的用法與 S3 相同， + ``stat`` 回報的欄位也相同。 + +S3 與 Azure Blob 都是物件儲存。目錄只在其下還有 key 時才存在，因此 ``mkdir`` 不會 +建立任何東西，空目錄也無法存在（``capabilities.directories`` 為 ``False``）。其他 +工具寫入、以 ``/`` 結尾的資料夾佔位 key 會顯示為目錄，絕不會顯示為檔案。校驗碼是 +由內容計算而來，不取自 ETag，因為分段上傳的 ETag 並不是摘要。用戶端尚未初始化時， +每個呼叫都會拋出 ``StorageUnavailableException``。 + +.. code-block:: python + + from automation_file import File, azure_blob_instance, s3_instance + + s3_instance.later_init(region_name="us-east-1") + azure_blob_instance.later_init(connection_string=connection_string) + File("s3://reports/2026/q1.csv").copy_to("azure://backups/2026/q1.csv") + 掛載與註冊後端 -------------- @@ -205,7 +234,7 @@ URI 的解析分兩步。**掛載** 優先：掛載把一個後端實例綁定 Storage.register_scheme("vault", lambda uri: (vault_backend(uri.authority), uri.path)) Storage.resolve("sandbox://jobs/42/out.csv") # (LocalStorage('/srv/jobs'), '42/out.csv') - Storage.schemes() # ['local', 'memory', 'sandbox', 'vault'] + Storage.schemes() # ['azure', 'local', 'memory', 's3', 'sandbox', 'vault'] ``Storage.mount`` / ``unmount`` / ``register_scheme`` / ``schemes`` / ``resolve`` 操作的是整個行程共用的表。需要私有的表時使用 @@ -240,6 +269,10 @@ scheme 或 authority，並且只接受其下的 URI。 # _mkdir(path)、_rmdir(path) # 可選擇覆寫：_walk、_copy_from、_move_from、_checksum、_read_bytes +若是物件儲存，請改為繼承 :class:`~automation_file.ObjectStorage`，並實作 +``_head``、``_scan``、``_put``、``_get`` 與 ``_remove``。它提供 `內建後端`_ 一節 +所述的目錄行為，``S3Storage`` 與 ``AzureStorage`` 都建立在它之上。 + 請用契約測試套件檢查。``tests/storage_contract.py`` 包含 70 個案例——巢狀目錄、 空檔與大檔、Unicode 路徑、二進位資料、覆寫與路徑不存在時的行為、路徑正規化、 複製與搬移——並在後端確實有差異之處讀取 ``capabilities``： diff --git a/docs/updates/2026-10.md b/docs/updates/2026-10.md index 52f22ff..2863b65 100644 --- a/docs/updates/2026-10.md +++ b/docs/updates/2026-10.md @@ -216,3 +216,23 @@ Index and query commands: [README.md](README.md). New entries go at the end. - **Files**: `automation_file/storage/` (9 modules), `automation_file/exceptions.py`, `automation_file/__init__.py`, `tests/storage_contract.py`, `tests/test_storage_*.py` (7 files), the documentation above, `progress.md`. - **Evidence**: branch `feat/universal-storage-layer`, based on `a0dd11f`. - **Open items**: `progress.md` #10 to #26 (the rest of the roadmap), #27 and #28 (found on the way). + +## U-20261008-02 · 2026-10-08 · S3 and Azure Blob behind the storage layer · #storage #roadmap #s3 #azure + +- **What**: the first two adapters of `progress.md` #13, which stays open for the other backends. + - `storage/object_storage.py`, `ObjectStorage`: the `StorageBackend` primitives for a flat key space, over five calls a service adapter supplies (`_head`, `_scan`, `_put`, `_get`, `_remove`). A directory exists while a key lies below it; `mkdir` creates nothing; a key ending in `/` (a folder placeholder written by another tool) is listed as a directory, never as a file, and deleting the directory removes it. `prefix=` confines a backend to the keys below one prefix. `exists` on a missing container is `False`, not an error. + - `storage/s3_storage.py`, `S3Storage(bucket, client=None, prefix="")`: `head_object`, the `list_objects_v2` paginator (with `Delimiter="/"` for one level), `upload_file` with `ContentType` from the key's suffix, `download_file`, `delete_object`, and `copy` for a server-side copy between two S3 locations that share a client. `stat` reports size, modification time, ETag (unquoted), content type, version ID and metadata. botocore error codes map to `StorageNotFoundException`, `StoragePermissionException`, `StorageTransientException` (throttling, 5xx, connection and HTTP client errors) or `StorageException`. + - `storage/azure_storage.py`, `AzureStorage(container, service=None, prefix="")`: `get_blob_properties`, `walk_blobs` / `list_blobs`, `upload_blob` with `ContentSettings`, `download_blob().readinto`, `delete_blob`; azure-core errors map the same way. + - Without `client=` / `service=` they use the shared `s3_instance` / `azure_blob_instance`, so the existing `later_init` calls and `FA_*_later_init` actions initialise them; before that every call raises `StorageUnavailableException`. The SDK's exception classes and the shared client are imported inside functions, so the layer's module-level imports are unchanged (`tests/test_storage_imports.py`). + - `s3:///` and `azure:///` (alias `az://`) are default schemes of `StorageResolver`; a URI without a bucket or container is refused with the correct form in the message. + - `StorageBackend._transfer_paths` compares the two backends with `==`, so copying `s3://bucket/a` onto itself is refused although each resolution builds a new `S3Storage`. `guess_content_type` moved from `local_storage.py` to `backend.py` for the adapters to share. + - The existing `FA_s3_*` and `FA_azure_blob_*` actions and functions are untouched. +- **Tests**: 323 new cases (638 for the layer). + - `tests/test_storage_s3.py` and `tests/test_storage_azure.py`: the 70-case contract for each adapter, once on a whole container and once behind `prefix=` next to another tenant's keys, against in-memory stand-ins for the boto3 client and `BlobServiceClient` that raise the real SDK exceptions and page their listings. + - Held against the installed SDKs, not only the stand-ins: a real boto3 client with botocore's `Stubber`, which checks each `head_object`, `list_objects_v2` and `delete_object` request against the S3 service model and feeds back modelled responses and errors; `inspect.signature` checks of `upload_file`, `download_file`, `copy`, and of the azure-storage-blob calls and `BlobProperties` fields the adapter reads. boto3 1.43.109, azure-storage-blob 12.31.0. + - Folder placeholders, pagination, server-side copy against staged copy, error mapping with the cause chained, the uninitialised shared client, and `File("s3://…")` / `File("az://…")` through a resolver. +- **Result / numbers**: `ruff check` and `ruff format --check` pass; `mypy automation_file` finds no issues in 172 files. `pytest tests/`: 1458 passed, 20 skipped, 5 failed, on Python 3.14.7 on Windows. The 5 are the `test_versioning.py` path-length failures of `progress.md` #28, present on `a0dd11f`. `pyarrow` is installed in this run, so the 6 Parquet tests that failed in U-20261008-01 pass. +- **Not verified**: no request has reached S3, MinIO, Azure or Azurite (`progress.md` #19). 12 of the 20 skips are contract cases that do not apply where directories are implied (three cases for each of the four object-store classes); the other 8 are those of U-20261008-01, the five symbolic-link tests among them (`progress.md` #18). +- **Docs**: the three `usage/storage.rst` pages (the two backends, object-store behaviour, `ObjectStorage` for new backends), `docs/source/API/storage.rst`, the three READMEs (feature bullet, diagram, "Backends today"), `architecture.md` §2, §3 and §5, `CLAUDE.md` (package map, key types). +- **Files**: `automation_file/storage/object_storage.py`, `s3_storage.py`, `azure_storage.py`, `backend.py`, `local_storage.py`, `resolver.py`, `__init__.py`, `automation_file/__init__.py`, `tests/test_storage_s3.py`, `tests/test_storage_azure.py`, `tests/test_storage_resolver.py`, the documentation above, `progress.md`. +- **Open items**: `progress.md` #13 (Dropbox, SFTP, FTP, WebDAV, SMB, fsspec), #19. diff --git a/docs/updates/README.md b/docs/updates/README.md index 5f8cd29..ab472dd 100644 --- a/docs/updates/README.md +++ b/docs/updates/README.md @@ -58,6 +58,7 @@ In the same commit: delete the item from `progress.md`, add a `#done` entry here | ID | Date | Title | Tags | Batch | |---|---|---|---|---| +| U-20261008-02 | 2026-10-08 | S3 and Azure Blob behind the storage layer | #storage #roadmap #s3 #azure | [2026-10](2026-10.md) | | U-20261008-01 | 2026-10-08 | Universal storage layer: contract, URIs, local and memory | #storage #roadmap #tests | [2026-10](2026-10.md) | | U-20261001-11 | 2026-10-01 | The publish jobs build with the locked setuptools | #done #ci #security #X-13 | [2026-10](2026-10.md) | | U-20261001-10 | 2026-10-01 | The publish jobs install hash-locked build tools | #done #ci #security #deps | [2026-10](2026-10.md) | @@ -96,5 +97,5 @@ In the same commit: delete the item from `progress.md`, add a `#done` entry here | File | Period | Entries | |---|---|---:| -| [2026-10.md](2026-10.md) | 2026-10 | 12 | +| [2026-10.md](2026-10.md) | 2026-10 | 13 | | [2026-09.md](2026-09.md) | 2026-09 | 21 | diff --git a/progress.md b/progress.md index 440a47e..f31ce7e 100644 --- a/progress.md +++ b/progress.md @@ -16,7 +16,7 @@ Items #10 to #26 are what is left of the 1.0 roadmap (`docs/FILEAUTOMATION-1.0-R ### Universal storage layer (roadmap M2) -- **#13** Storage adapters over the existing clients, each in `storage/_storage.py` with a `StorageContract` class against a fake client: S3 (`s3://bucket/key`), Azure Blob (`azure://container/blob`), Dropbox (`dropbox:///path`), SFTP (`sftp://host/path`), FTP and FTPS, WebDAV, SMB (`smb://server/share/path`), fsspec. The object stores set `capabilities.directories=False` and override `_walk` with one flat listing. A session backend must refuse a URI whose host is not the one it is connected to; `SFTPClient` does not keep its host today. `tests/test_storage_imports.py` then needs the client modules on its allowlist. +- **#13** Storage adapters over the remaining clients, each in `storage/_storage.py` with a `StorageContract` class against a stand-in client: Dropbox (`dropbox:///path`), SFTP (`sftp://host/path`), FTP and FTPS, WebDAV, SMB (`smb://server/share/path`), fsspec. S3 and Azure Blob are done (U-20261008-02) and show the pattern: import the shared client and the SDK's exceptions inside functions, so `tests/test_storage_imports.py` keeps passing. A session backend must refuse a URI whose host is not the one it is connected to; `SFTPClient` does not keep its host today. - **#14** Google Drive adapter (`gdrive://`). Drive addresses files by ID and allows two files of one name in a folder, so the path-to-ID lookup and the duplicate-name rule have to be designed first. - **#15** [DECIDE] The eleventh backend slot, and whether OneDrive and Box are promoted to the storage contract or documented as action-only (roadmap §4). - **#16** Cross-backend operations through the layer: `copy_between` / `FA_copy_between` on `File.copy_to`, and `FA_storage_*` actions. `copy_between` accepts `local:`, `sftp:/path`, `s3:bucket/key` and http(s) sources today; `parse_storage_uri` rejects the first three as ambiguous, so the action needs a translation step to stay compatible. @@ -25,7 +25,7 @@ Items #10 to #26 are what is left of the 1.0 roadmap (`docs/FILEAUTOMATION-1.0-R ### Backend integration tests (roadmap M3) -- **#19** Integration environments for the contract suite in CI: MinIO, Azurite, SFTP, FTP/FTPS, WebDAV and Samba, plus credential-gated jobs for the cloud adapters, and Linux and macOS legs (`ci-dev.yml` runs pytest on Windows only). Needs #13. +- **#19** Integration environments for the contract suite in CI: MinIO and Azurite (`S3Storage` and `AzureStorage` have only met stand-in clients and, for S3, botocore's Stubber; no request has reached a real service), SFTP, FTP/FTPS, WebDAV and Samba, plus credential-gated jobs for the cloud adapters, and Linux and macOS legs (`ci-dev.yml` runs pytest on Windows only). Needs #13. - **#20** Failure cases in the contract suite: access denied, transient failures mapped to `StorageTransientException` and retried, metadata kept where the backend supports it. Only `LocalStorage` has permission-error tests today (`tests/test_storage_local.py`). ### Later milestones diff --git a/tests/test_storage_azure.py b/tests/test_storage_azure.py new file mode 100644 index 0000000..fd63602 --- /dev/null +++ b/tests/test_storage_azure.py @@ -0,0 +1,339 @@ +"""AzureStorage: the storage contract against an in-memory stand-in for BlobServiceClient. + +The stand-in answers the calls the adapter makes -- ``get_blob_client`` (with +``get_blob_properties``, ``upload_blob``, ``download_blob().readinto`` and +``delete_blob``) and ``get_container_client`` (with ``list_blobs`` and +``walk_blobs``) -- and raises the real ``azure.core`` exceptions. No request +leaves the process. +""" + +from __future__ import annotations + +import hashlib +import inspect +from dataclasses import dataclass +from datetime import datetime, timezone +from pathlib import Path +from types import SimpleNamespace +from typing import Any, BinaryIO + +import pytest +from azure.core.exceptions import ( + ClientAuthenticationError, + HttpResponseError, + ResourceExistsError, + ResourceNotFoundError, + ServiceRequestError, +) +from azure.storage.blob import ( + BlobClient, + BlobProperties, + BlobServiceClient, + ContainerClient, + ContentSettings, + StorageStreamDownloader, +) + +from automation_file.exceptions import ( + StorageException, + StorageNotFoundException, + StoragePermissionException, + StorageTransientException, + StorageUnavailableException, + StorageURIException, +) +from automation_file.remote.azure_blob.client import azure_blob_instance +from automation_file.storage import AzureStorage, File, StorageBackend, StorageResolver +from tests.storage_contract import StorageContract + + +def _http_error(status: int) -> HttpResponseError: + error = HttpResponseError(message=f"status {status}") + error.status_code = status + return error + + +@dataclass +class _Blob: + data: bytes + content_type: str | None + modified: datetime + + def properties(self, name: str) -> SimpleNamespace: + return SimpleNamespace( + name=name, + size=len(self.data), + last_modified=self.modified, + etag=f'"0x{hashlib.md5(self.data, usedforsecurity=False).hexdigest()[:16].upper()}"', + content_settings=SimpleNamespace(content_type=self.content_type), + metadata={}, + version_id=None, + ) + + +class _Download: + def __init__(self, data: bytes) -> None: + self._data = data + + def readinto(self, stream: BinaryIO) -> int: + return stream.write(self._data) + + +class _BlobClient: + def __init__(self, service: FakeBlobService, container: str, name: str) -> None: + self._service = service + self._container = container + self._name = name + + def _existing(self) -> _Blob: + blobs = self._service.blobs(self._container) + if self._name not in blobs: + raise ResourceNotFoundError("The specified blob does not exist.") + return blobs[self._name] + + def get_blob_properties(self) -> SimpleNamespace: + return self._existing().properties(self._name) + + def upload_blob( + self, data: BinaryIO, overwrite: bool = False, content_settings: Any = None + ) -> None: + blobs = self._service.blobs(self._container) + if self._name in blobs and not overwrite: + raise ResourceExistsError("The specified blob already exists.") + content_type = getattr(content_settings, "content_type", None) + blobs[self._name] = _Blob(data.read(), content_type, datetime.now(timezone.utc)) + + def download_blob(self) -> _Download: + return _Download(self._existing().data) + + def delete_blob(self) -> None: + self._existing() + del self._service.blobs(self._container)[self._name] + + +class _ContainerClient: + def __init__(self, service: FakeBlobService, container: str) -> None: + self._service = service + self._container = container + + def list_blobs(self, name_starts_with: str | None = None): + blobs = self._service.blobs(self._container) + prefix = name_starts_with or "" + for name in sorted(blobs): + if name.startswith(prefix): + yield blobs[name].properties(name) + + def walk_blobs(self, name_starts_with: str | None = None, delimiter: str = "/"): + blobs = self._service.blobs(self._container) + prefix = name_starts_with or "" + seen: set[str] = set() + for name in sorted(blobs): + if not name.startswith(prefix): + continue + rest = name[len(prefix) :] + if delimiter in rest: + deeper = prefix + rest.split(delimiter, 1)[0] + delimiter + if deeper not in seen: + seen.add(deeper) + yield SimpleNamespace(name=deeper, prefix=deeper) + continue + yield blobs[name].properties(name) + + +class FakeBlobService: + """The subset of BlobServiceClient that AzureStorage calls.""" + + def __init__(self, *containers: str) -> None: + self.containers: dict[str, dict[str, _Blob]] = { + name: {} for name in containers or ("container",) + } + self.fail_with: Exception | None = None + + def blobs(self, container: str) -> dict[str, _Blob]: + if self.fail_with is not None: + raise self.fail_with + if container not in self.containers: + raise ResourceNotFoundError("The specified container does not exist.") + return self.containers[container] + + def get_blob_client(self, container: str, blob: str) -> _BlobClient: + return _BlobClient(self, container, blob) + + def get_container_client(self, container: str) -> _ContainerClient: + return _ContainerClient(self, container) + + +class TestAzureStorageContract(StorageContract): + @pytest.fixture + def backend(self) -> StorageBackend: + return AzureStorage("container", service=FakeBlobService()) + + +class TestPrefixedAzureStorageContract(StorageContract): + @pytest.fixture + def backend(self) -> StorageBackend: + service = FakeBlobService() + service.containers["container"]["other-tenant/keep.txt"] = _Blob( + b"keep", "text/plain", datetime.now(timezone.utc) + ) + return AzureStorage("container", service=service, prefix="tenant/a") + + +@pytest.fixture +def service() -> FakeBlobService: + return FakeBlobService("container", "archive") + + +@pytest.fixture +def storage(service: FakeBlobService) -> AzureStorage: + return AzureStorage("container", service=service) + + +def test_stat_reports_the_blob_properties(storage: AzureStorage) -> None: + info = storage.write_bytes("reports/q1.json", b"{}") + assert info.path == "reports/q1.json" + assert info.size == 2 + assert info.etag is not None + assert not info.etag.startswith('"') + assert info.content_type == "application/json" + assert info.version is None + assert dict(info.metadata) == {} + assert info.modified_at is not None + + +def test_a_blob_without_a_known_suffix_has_no_content_type(storage: AzureStorage) -> None: + assert storage.write_bytes("blob", b"x").content_type is None + + +def test_a_prefix_confines_the_backend_to_its_blobs(service: FakeBlobService) -> None: + service.containers["container"]["other/keep.txt"] = _Blob( + b"k", None, datetime.now(timezone.utc) + ) + tenant = AzureStorage("container", service=service, prefix="tenant/a") + tenant.write_bytes("docs/a.txt", b"x") + assert sorted(service.containers["container"]) == ["other/keep.txt", "tenant/a/docs/a.txt"] + assert [info.path for info in tenant.list_dir("", recursive=True)] == ["docs", "docs/a.txt"] + assert tenant.uri_for("docs/a.txt") == "azure://container/tenant/a/docs/a.txt" + tenant.delete("docs", recursive=True) + assert sorted(service.containers["container"]) == ["other/keep.txt"] + + +def test_a_folder_placeholder_is_a_directory_not_a_file( + storage: AzureStorage, service: FakeBlobService +) -> None: + service.containers["container"]["folder/"] = _Blob(b"", None, datetime.now(timezone.utc)) + assert storage.stat("folder").is_dir is True + assert storage.list_dir("folder") == [] + assert [(info.path, info.is_dir) for info in storage.list_dir()] == [("folder", True)] + storage.delete("folder") + assert service.containers["container"] == {} + + +def test_copy_between_containers(storage: AzureStorage, service: FakeBlobService) -> None: + storage.write_bytes("a.txt", b"payload") + AzureStorage("archive", service=service).copy_from(storage, "a.txt", "2026/a.txt") + assert service.containers["archive"]["2026/a.txt"].data == b"payload" + assert service.containers["archive"]["2026/a.txt"].content_type == "text/plain" + + +@pytest.mark.parametrize( + "error,expected", + [ + (ClientAuthenticationError("bad key"), StoragePermissionException), + (_http_error(403), StoragePermissionException), + (_http_error(429), StorageTransientException), + (_http_error(503), StorageTransientException), + (_http_error(400), StorageException), + (ServiceRequestError("connection refused"), StorageTransientException), + ], +) +def test_sdk_errors_become_storage_errors( + storage: AzureStorage, service: FakeBlobService, error: Exception, expected: type[Exception] +) -> None: + service.fail_with = error + with pytest.raises(expected) as caught: + storage.stat("a.txt") + assert caught.value.__cause__ is error + assert type(caught.value) is expected + + +def test_a_missing_container_is_not_found(service: FakeBlobService) -> None: + missing = AzureStorage("no-such-container", service=service) + assert missing.exists("a.txt") is False + with pytest.raises(StorageNotFoundException): + missing.list_dir() + + +def test_a_container_name_is_required() -> None: + with pytest.raises(StorageURIException): + AzureStorage("") + + +def test_equality_and_repr(service: FakeBlobService) -> None: + assert AzureStorage("container", service=service) == AzureStorage("container", service=service) + assert AzureStorage("container", service=service) != AzureStorage("archive", service=service) + assert AzureStorage("container", service=service) != AzureStorage( + "container", service=FakeBlobService() + ) + assert ( + repr(AzureStorage("container", service=service, prefix="a")) + == "AzureStorage('container', prefix='a')" + ) + assert AzureStorage("container", service=service).container == "container" + assert AzureStorage("container", service=service).uri_for("") == "azure://container" + + +def test_the_shared_client_must_be_initialised(monkeypatch: pytest.MonkeyPatch) -> None: + monkeypatch.setattr(azure_blob_instance, "service", None) + with pytest.raises(StorageUnavailableException, match="later_init"): + AzureStorage("container").exists("a.txt") + + +def test_azure_uris_use_the_shared_client( + monkeypatch: pytest.MonkeyPatch, service: FakeBlobService, tmp_path: Path +) -> None: + monkeypatch.setattr(azure_blob_instance, "service", service) + resolver = StorageResolver() + report = File("azure://container/reports/q1.csv", resolver=resolver) + report.write(b"a,b\n") + assert service.containers["container"]["reports/q1.csv"].data == b"a,b\n" + assert File("az://container/reports/q1.csv", resolver=resolver).read() == b"a,b\n" + assert resolver.resolve("az://container/reports/q1.csv") == ( + AzureStorage("container"), + "reports/q1.csv", + ) + report.move_to(tmp_path / "q1.csv") + assert (tmp_path / "q1.csv").read_bytes() == b"a,b\n" + assert service.containers["container"] == {} + + +def test_an_azure_uri_needs_a_container() -> None: + with pytest.raises(StorageURIException, match="container"): + StorageResolver().resolve("azure:///blob-without-container") + + +# ---------------------------------------------------------------------- the real azure SDK + + +def test_the_sdk_has_the_calls_the_adapter_makes() -> None: + """The stand-in above is only as good as its match with azure-storage-blob.""" + blob_client = inspect.signature(BlobServiceClient.get_blob_client).parameters + assert {"container", "blob"} <= set(blob_client) + assert "container" in inspect.signature(BlobServiceClient.get_container_client).parameters + assert "name_starts_with" in inspect.signature(ContainerClient.list_blobs).parameters + assert {"name_starts_with", "delimiter"} <= set( + inspect.signature(ContainerClient.walk_blobs).parameters + ) + # overwrite= and content_settings= travel through **kwargs. + assert "kwargs" in inspect.signature(BlobClient.upload_blob).parameters + for method in ("get_blob_properties", "download_blob", "delete_blob"): + assert callable(getattr(BlobClient, method)) + assert list(inspect.signature(StorageStreamDownloader.readinto).parameters) == [ + "self", + "stream", + ] + properties = BlobProperties() + for attribute in ("name", "size", "last_modified", "etag", "content_settings", "metadata"): + assert hasattr(properties, attribute) + assert hasattr(properties, "version_id") + assert ContentSettings(content_type="text/plain").content_type == "text/plain" diff --git a/tests/test_storage_resolver.py b/tests/test_storage_resolver.py index 5a7de7a..a90036f 100644 --- a/tests/test_storage_resolver.py +++ b/tests/test_storage_resolver.py @@ -39,7 +39,7 @@ def resolver() -> StorageResolver: def test_default_schemes(resolver: StorageResolver) -> None: - assert resolver.schemes() == ["local", "memory"] + assert resolver.schemes() == ["azure", "local", "memory", "s3"] assert StorageResolver(defaults=False).schemes() == [] @@ -78,7 +78,7 @@ def test_an_unknown_scheme_names_the_known_ones(resolver: StorageResolver) -> No message = str(caught.value) assert "'gopher'" in message assert "gopher://host/a.txt" in message - assert "local, memory" in message + assert "azure, local, memory, s3" in message def test_register_scheme_installs_a_factory(resolver: StorageResolver) -> None: diff --git a/tests/test_storage_s3.py b/tests/test_storage_s3.py new file mode 100644 index 0000000..37f58ad --- /dev/null +++ b/tests/test_storage_s3.py @@ -0,0 +1,410 @@ +"""S3Storage: the storage contract against an in-memory stand-in for the boto3 client. + +The stand-in answers the calls the adapter makes -- ``head_object``, the +``list_objects_v2`` paginator, ``upload_file``, ``download_file``, +``delete_object`` and ``copy`` -- with the shapes boto3 documents, and raises the +real ``botocore`` exceptions. No request leaves the process. +""" + +from __future__ import annotations + +import hashlib +import inspect +from dataclasses import dataclass +from datetime import datetime, timezone +from pathlib import Path +from typing import Any + +import boto3 +import pytest +from botocore import UNSIGNED +from botocore.config import Config +from botocore.exceptions import ClientError, EndpointConnectionError, NoCredentialsError +from botocore.stub import Stubber + +from automation_file.exceptions import ( + StorageException, + StorageNotFoundException, + StoragePermissionException, + StorageTransientException, + StorageUnavailableException, + StorageURIException, +) +from automation_file.remote.s3.client import s3_instance +from automation_file.storage import File, S3Storage, StorageBackend, StorageResolver +from tests.storage_contract import StorageContract + +PAGE_SIZE = 2 + + +def _client_error(code: str, operation: str) -> ClientError: + return ClientError({"Error": {"Code": code, "Message": code}}, operation) + + +@dataclass +class _Object: + data: bytes + content_type: str + modified: datetime + + @property + def etag(self) -> str: + return f'"{hashlib.md5(self.data, usedforsecurity=False).hexdigest()}"' + + +class _Paginator: + def __init__(self, client: FakeS3Client) -> None: + self._client = client + + def paginate(self, **arguments: Any): + prefix: str = arguments.get("Prefix", "") + delimiter: str | None = arguments.get("Delimiter") + objects = self._client.objects(arguments["Bucket"], "ListObjectsV2") + contents: list[dict[str, Any]] = [] + prefixes: list[str] = [] + for key in sorted(objects): + if not key.startswith(prefix): + continue + rest = key[len(prefix) :] + if delimiter and delimiter in rest: + common = prefix + rest.split(delimiter, 1)[0] + delimiter + if common not in prefixes: + prefixes.append(common) + continue + item = objects[key] + contents.append( + { + "Key": key, + "Size": len(item.data), + "LastModified": item.modified, + "ETag": item.etag, + } + ) + entries: list[tuple[str, Any]] = [("CommonPrefixes", {"Prefix": p}) for p in prefixes] + entries += [("Contents", item) for item in contents] + if not entries: + yield {"KeyCount": 0} + for start in range(0, len(entries), PAGE_SIZE): + page: dict[str, Any] = {"KeyCount": 0} + for kind, value in entries[start : start + PAGE_SIZE]: + page.setdefault(kind, []).append(value) + page["KeyCount"] += 1 + yield page + + +class FakeS3Client: + """The subset of the boto3 S3 client that S3Storage calls.""" + + def __init__(self, *buckets: str) -> None: + self.buckets: dict[str, dict[str, _Object]] = {name: {} for name in buckets or ("bucket",)} + self.calls: list[str] = [] + self.fail_with: Exception | None = None + + def objects(self, bucket: str, operation: str) -> dict[str, _Object]: + self.calls.append(operation) + if self.fail_with is not None: + raise self.fail_with + if bucket not in self.buckets: + raise _client_error("NoSuchBucket", operation) + return self.buckets[bucket] + + def head_object(self, **arguments: Any) -> dict[str, Any]: + objects = self.objects(arguments["Bucket"], "HeadObject") + if arguments["Key"] not in objects: + raise _client_error("404", "HeadObject") + item = objects[arguments["Key"]] + return { + "ContentLength": len(item.data), + "LastModified": item.modified, + "ETag": item.etag, + "ContentType": item.content_type, + "Metadata": {}, + } + + def get_paginator(self, name: str) -> _Paginator: + assert name == "list_objects_v2" + return _Paginator(self) + + def upload_file(self, filename: str, bucket: str, key: str, **options: Any) -> None: + extra = options.get("ExtraArgs") or {} + self.objects(bucket, "PutObject")[key] = _Object( + Path(filename).read_bytes(), + extra.get("ContentType", "binary/octet-stream"), + datetime.now(timezone.utc), + ) + + def download_file(self, bucket: str, key: str, filename: str) -> None: + objects = self.objects(bucket, "GetObject") + if key not in objects: + raise _client_error("404", "GetObject") + Path(filename).write_bytes(objects[key].data) + + def delete_object(self, **arguments: Any) -> None: + self.objects(arguments["Bucket"], "DeleteObject").pop(arguments["Key"], None) + + def copy(self, copy_source: dict[str, str], bucket: str, key: str) -> None: + source = self.objects(copy_source["Bucket"], "CopyObject") + if copy_source["Key"] not in source: + raise _client_error("NoSuchKey", "CopyObject") + item = source[copy_source["Key"]] + self.buckets[bucket][key] = _Object( + item.data, item.content_type, datetime.now(timezone.utc) + ) + + +class TestS3StorageContract(StorageContract): + @pytest.fixture + def backend(self) -> StorageBackend: + return S3Storage("bucket", client=FakeS3Client()) + + +class TestPrefixedS3StorageContract(StorageContract): + @pytest.fixture + def backend(self) -> StorageBackend: + client = FakeS3Client() + client.buckets["bucket"]["other-tenant/keep.txt"] = _Object( + b"keep", "text/plain", datetime.now(timezone.utc) + ) + return S3Storage("bucket", client=client, prefix="tenant/a") + + +@pytest.fixture +def client() -> FakeS3Client: + return FakeS3Client("bucket", "archive") + + +@pytest.fixture +def storage(client: FakeS3Client) -> S3Storage: + return S3Storage("bucket", client=client) + + +def test_stat_reports_what_head_object_returns(storage: S3Storage) -> None: + info = storage.write_bytes("reports/q1.json", b"{}") + assert info.path == "reports/q1.json" + assert info.size == 2 + assert info.etag == hashlib.md5(b"{}", usedforsecurity=False).hexdigest() + assert info.content_type == "application/json" + assert info.version is None + assert dict(info.metadata) == {} + assert info.modified_at is not None + + +def test_a_key_without_a_known_suffix_keeps_the_default_content_type(storage: S3Storage) -> None: + assert storage.write_bytes("blob", b"x").content_type == "binary/octet-stream" + + +def test_listing_reads_every_page(storage: S3Storage, client: FakeS3Client) -> None: + for index in range(7): + storage.write_bytes(f"dir/{index}.txt", b"x") + listing = storage.list_dir("dir") + assert [info.path for info in listing] == [f"dir/{index}.txt" for index in range(7)] + assert all(info.etag for info in listing) + assert len(storage.list_dir("", recursive=True)) == 8 + + +def test_a_prefix_confines_the_backend_to_its_keys(client: FakeS3Client) -> None: + client.buckets["bucket"]["other/keep.txt"] = _Object( + b"k", "text/plain", datetime.now(timezone.utc) + ) + tenant = S3Storage("bucket", client=client, prefix="/tenant//a/") + assert tenant.prefix == "tenant/a" + tenant.write_bytes("docs/a.txt", b"x") + assert sorted(client.buckets["bucket"]) == ["other/keep.txt", "tenant/a/docs/a.txt"] + assert [info.path for info in tenant.list_dir("", recursive=True)] == ["docs", "docs/a.txt"] + assert tenant.uri_for("docs/a.txt") == "s3://bucket/tenant/a/docs/a.txt" + tenant.delete("docs", recursive=True) + assert sorted(client.buckets["bucket"]) == ["other/keep.txt"] + + +def test_a_folder_placeholder_is_a_directory_not_a_file( + storage: S3Storage, client: FakeS3Client +) -> None: + client.buckets["bucket"]["folder/"] = _Object( + b"", "application/x-directory", datetime.now(timezone.utc) + ) + assert storage.stat("folder").is_dir is True + assert storage.list_dir("folder") == [] + assert [(info.path, info.is_dir) for info in storage.list_dir()] == [("folder", True)] + assert [(info.path, info.is_dir) for info in storage.list_dir("", recursive=True)] == [ + ("folder", True) + ] + storage.delete("folder") + assert client.buckets["bucket"] == {} + assert storage.exists("folder") is False + + +def test_copy_within_s3_is_server_side(storage: S3Storage, client: FakeS3Client) -> None: + storage.write_bytes("a.txt", b"payload") + client.calls.clear() + storage.copy_from(storage, "a.txt", "copies/b.txt") + assert "CopyObject" in client.calls + assert "GetObject" not in client.calls + archive = S3Storage("archive", client=client) + archive.copy_from(storage, "a.txt", "a.txt") + assert client.buckets["archive"]["a.txt"].data == b"payload" + + +def test_copy_between_two_clients_goes_through_a_staging_file(storage: S3Storage) -> None: + other_client = FakeS3Client("bucket") + other = S3Storage("bucket", client=other_client) + storage.write_bytes("a.txt", b"payload") + other.copy_from(storage, "a.txt", "a.txt") + assert other_client.buckets["bucket"]["a.txt"].data == b"payload" + assert "CopyObject" not in other_client.calls + + +@pytest.mark.parametrize( + "error,expected", + [ + (_client_error("AccessDenied", "HeadObject"), StoragePermissionException), + (_client_error("403", "HeadObject"), StoragePermissionException), + (_client_error("SlowDown", "HeadObject"), StorageTransientException), + (_client_error("503", "HeadObject"), StorageTransientException), + (_client_error("InvalidBucketName", "HeadObject"), StorageException), + (EndpointConnectionError(endpoint_url="https://s3.invalid"), StorageTransientException), + (NoCredentialsError(), StorageException), + ], +) +def test_sdk_errors_become_storage_errors( + storage: S3Storage, client: FakeS3Client, error: Exception, expected: type[Exception] +) -> None: + client.fail_with = error + with pytest.raises(expected) as caught: + storage.stat("a.txt") + assert caught.value.__cause__ is error + assert type(caught.value) is expected + + +def test_a_missing_bucket_is_not_found(client: FakeS3Client) -> None: + missing = S3Storage("no-such-bucket", client=client) + assert missing.exists("a.txt") is False + with pytest.raises(StorageNotFoundException): + missing.list_dir() + + +def test_a_bucket_name_is_required() -> None: + with pytest.raises(StorageURIException): + S3Storage("") + + +def test_equality_and_repr(client: FakeS3Client) -> None: + assert S3Storage("bucket", client=client) == S3Storage("bucket", client=client) + assert S3Storage("bucket", client=client) != S3Storage("archive", client=client) + assert S3Storage("bucket", client=client) != S3Storage("bucket", client=client, prefix="a") + assert S3Storage("bucket", client=client) != S3Storage("bucket", client=FakeS3Client()) + assert len({S3Storage("bucket", client=client), S3Storage("bucket", client=client)}) == 1 + assert repr(S3Storage("bucket", client=client, prefix="a")) == "S3Storage('bucket', prefix='a')" + assert S3Storage("bucket", client=client).bucket == "bucket" + assert S3Storage("bucket", client=client).uri_for("") == "s3://bucket" + + +def test_the_shared_client_must_be_initialised(monkeypatch: pytest.MonkeyPatch) -> None: + monkeypatch.setattr(s3_instance, "client", None) + with pytest.raises(StorageUnavailableException, match="later_init"): + S3Storage("bucket").exists("a.txt") + + +def test_s3_uris_use_the_shared_client( + monkeypatch: pytest.MonkeyPatch, client: FakeS3Client, tmp_path: Path +) -> None: + monkeypatch.setattr(s3_instance, "client", client) + resolver = StorageResolver() + report = File("s3://bucket/reports/q1.csv", resolver=resolver) + report.write(b"a,b\n") + assert client.buckets["bucket"]["reports/q1.csv"].data == b"a,b\n" + assert resolver.resolve("s3://bucket/reports/q1.csv") == (S3Storage("bucket"), "reports/q1.csv") + report.copy_to(tmp_path / "q1.csv") + assert (tmp_path / "q1.csv").read_bytes() == b"a,b\n" + File(tmp_path / "q1.csv", resolver=resolver).copy_to("s3://archive/2026/q1.csv") + assert client.buckets["archive"]["2026/q1.csv"].data == b"a,b\n" + with pytest.raises(StorageException): + report.copy_to("s3://bucket//reports/q1.csv/") + + +def test_an_s3_uri_needs_a_bucket() -> None: + with pytest.raises(StorageURIException, match="bucket"): + StorageResolver().resolve("s3:///key-without-bucket") + + +# ---------------------------------------------------------------------- the real boto3 client + + +@pytest.fixture +def real_client() -> Any: + """A real boto3 S3 client that signs nothing; the Stubber answers for the network.""" + return boto3.client("s3", region_name="us-east-1", config=Config(signature_version=UNSIGNED)) + + +def test_requests_and_responses_fit_the_service_model(real_client: Any) -> None: + """Stubber checks every request against botocore's model of S3, parameter by parameter.""" + storage = S3Storage("bucket", client=real_client) + modified = datetime(2026, 10, 8, 2, 30, tzinfo=timezone.utc) + listing = { + "KeyCount": 2, + "IsTruncated": False, + "CommonPrefixes": [{"Prefix": "dir/sub/"}], + "Contents": [{"Key": "dir/a.txt", "Size": 3, "LastModified": modified, "ETag": '"abc"'}], + } + shallow = {"Bucket": "bucket", "Prefix": "dir/", "Delimiter": "/"} + with Stubber(real_client) as stub: + stub.add_response( + "head_object", + { + "ContentLength": 3, + "LastModified": modified, + "ETag": '"abc"', + "ContentType": "text/plain", + "VersionId": "v1", + "Metadata": {"owner": "ops"}, + }, + {"Bucket": "bucket", "Key": "dir/a.txt"}, + ) + info = storage.stat("dir/a.txt") + assert (info.size, info.modified_at, info.etag) == (3, modified, "abc") + assert (info.content_type, info.version, dict(info.metadata)) == ( + "text/plain", + "v1", + {"owner": "ops"}, + ) + + stub.add_client_error( + "head_object", + service_error_code="404", + http_status_code=404, + expected_params={"Bucket": "bucket", "Key": "dir"}, + ) + stub.add_response("list_objects_v2", listing, shallow) + stub.add_response("list_objects_v2", listing, shallow) + assert [(entry.path, entry.is_dir, entry.size) for entry in storage.list_dir("dir")] == [ + ("dir/a.txt", False, 3), + ("dir/sub", True, None), + ] + + stub.add_response( + "head_object", {"ContentLength": 3}, {"Bucket": "bucket", "Key": "dir/a.txt"} + ) + stub.add_response("delete_object", {}, {"Bucket": "bucket", "Key": "dir/a.txt"}) + storage.delete("dir/a.txt") + + stub.add_client_error( + "head_object", service_error_code="AccessDenied", http_status_code=403 + ) + with pytest.raises(StoragePermissionException): + storage.stat("secret.txt") + stub.assert_no_pending_responses() + + +def test_the_transfer_calls_exist_with_the_arguments_the_adapter_passes(real_client: Any) -> None: + upload = inspect.signature(real_client.upload_file).parameters + assert list(upload)[:4] == ["Filename", "Bucket", "Key", "ExtraArgs"] + assert list(inspect.signature(real_client.download_file).parameters)[:3] == [ + "Bucket", + "Key", + "Filename", + ] + assert list(inspect.signature(real_client.copy).parameters)[:3] == [ + "CopySource", + "Bucket", + "Key", + ] + assert real_client.can_paginate("list_objects_v2") is True From 2cfb09f80077a3a3cddf19f55e2e1f4f8acee8ce Mon Sep 17 00:00:00 2001 From: JeffreyChen Date: Thu, 8 Oct 2026 12:04:09 +0800 Subject: [PATCH 22/59] feat: add FA_storage actions for the storage layer --- CLAUDE.md | 5 +- README.md | 16 ++ README.zh-CN.md | 15 ++ README.zh-TW.md | 15 ++ architecture.md | 7 +- automation_file/__init__.py | 2 + automation_file/core/action_registry.py | 7 + automation_file/storage/__init__.py | 3 + automation_file/storage/actions.py | 162 ++++++++++++++++++++ docs/source/API/storage.rst | 6 + docs/source/Eng/usage/storage.rst | 78 ++++++++++ docs/source/Zh-CN/usage/storage.rst | 77 ++++++++++ docs/source/Zh-TW/usage/storage.rst | 77 ++++++++++ docs/updates/2026-10.md | 15 ++ docs/updates/README.md | 3 +- progress.md | 2 +- tests/test_storage_actions.py | 196 ++++++++++++++++++++++++ tests/test_storage_imports.py | 5 +- 18 files changed, 683 insertions(+), 8 deletions(-) create mode 100644 automation_file/storage/actions.py create mode 100644 tests/test_storage_actions.py diff --git a/CLAUDE.md b/CLAUDE.md index de6e37e..3ca06ef 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -27,7 +27,8 @@ automation_file/ ├── storage/ # Universal storage layer: uri (StorageURI), types (FileInfo, Checksum, │ # StorageCapabilities), backend (StorageBackend contract), local_storage, │ # memory_storage, object_storage (ObjectStorage), s3_storage, azure_storage, -│ # resolver (StorageResolver), file (File), storage (Storage) +│ # resolver (StorageResolver), file (File), storage (Storage), +│ # actions (FA_storage_* and register_storage_ops) ├── server/ # tcp_server, http_server, mcp_server (MCP over stdio), web_ui, metrics_server, │ # action_acl (ActionACL), network_guards (ensure_loopback) ├── client/ # HTTPActionClient for the HTTP action server @@ -155,7 +156,7 @@ All code must follow secure-by-default principles. Review every change against t - A new storage backend subclasses `StorageBackend` and passes `tests/storage_contract.py` through a `StorageContract` subclass. Do not weaken a contract case to make a backend pass: fix the backend, or branch on `capabilities` when backends legitimately differ. - Keep the checks that live in the base class: `normalize_path` refuses `..`, `parse_storage_uri` refuses credentials in the authority (and its error does not repeat them), `delete` refuses the storage root, and `LocalStorage` deletes a symbolic link without following it. Never log a storage URI's credentials or a backend's secrets. - When paths come from outside the process, use `LocalStorage(root)` behind a scheme or authority of its own (`Storage.mount("sandbox://jobs", LocalStorage(root))`), not the rootless `local://` backend. -- At module level, `automation_file/storage/` imports only the standard library, `exceptions`, `core.checksum` and `local.safe_paths`; `tests/test_storage_imports.py` fails on anything else. A backend SDK is imported lazily, inside the function that needs it. +- At module level, `automation_file/storage/` imports only the standard library, `exceptions`, `logging_config`, `core.checksum` and `local.safe_paths`; `tests/test_storage_imports.py` fails on anything else. A backend SDK is imported lazily, inside the function that needs it. ### SFTP host verification - `SFTPClient` uses `paramiko.RejectPolicy()` — unknown hosts are rejected, never auto-added. Callers pass `known_hosts=` explicitly or rely on `~/.ssh/known_hosts`. Do not swap in `AutoAddPolicy` for convenience. diff --git a/README.md b/README.md index 92d15fc..55ce393 100644 --- a/README.md +++ b/README.md @@ -483,6 +483,22 @@ File("sandbox://jobs/42/out.csv").write(b"done") `StorageBackend` (or `ObjectStorage` for an object store) and check it with the 70-case contract suite in `tests/storage_contract.py`. +- **Actions** — `FA_storage_exists`, `FA_storage_stat`, `FA_storage_list`, `FA_storage_mkdir`, + `FA_storage_upload`, `FA_storage_download`, `FA_storage_delete`, `FA_storage_checksum`, + `FA_storage_verify`, `FA_storage_copy`, `FA_storage_move`, `FA_storage_read_text`, + `FA_storage_write_text`, `FA_storage_schemes`. They take URIs + as strings and return JSON-friendly values, so the layer works from action files, the CLI, the + TCP and HTTP servers and as MCP tools. Restrict them on a server with `ActionACL`, as for any + file action. + +```json +[ + ["FA_storage_copy", {"source": "s3://reports/q1.csv", "target": "local:///backup/q1.csv"}], + ["FA_storage_verify", {"uri": "local:///backup/q1.csv", "expected": "sha256:9f86d081884c7d65…"}], + ["FA_storage_list", {"uri": "s3://reports", "recursive": true}] +] +``` + The API is new and may still change before 1.0. Full reference: the *Universal Storage Layer* chapter of the documentation. diff --git a/README.zh-CN.md b/README.zh-CN.md index fad6b9a..a5ea642 100644 --- a/README.zh-CN.md +++ b/README.zh-CN.md @@ -479,6 +479,21 @@ File("sandbox://jobs/42/out.csv").write(b"done") 动作使用，其适配器尚未完成。你可以继承 `StorageBackend`（对象存储则继承 `ObjectStorage`） 编写自己的后端，并用 `tests/storage_contract.py` 中 70 个用例的契约测试套件检查。 +- **动作** — `FA_storage_exists`, `FA_storage_stat`, `FA_storage_list`, `FA_storage_mkdir`, + `FA_storage_upload`, `FA_storage_download`, `FA_storage_delete`, `FA_storage_checksum`, + `FA_storage_verify`, `FA_storage_copy`, `FA_storage_move`, `FA_storage_read_text`, + `FA_storage_write_text`, `FA_storage_schemes`。它们以字符串 + 形式接收 URI，并返回可以序列化为 JSON 的值，因此本层可用于动作文件、CLI、TCP 与 HTTP 服务器， + 也能作为 MCP 工具。在服务器上请像其他文件动作一样用 `ActionACL` 加以限制。 + +```json +[ + ["FA_storage_copy", {"source": "s3://reports/q1.csv", "target": "local:///backup/q1.csv"}], + ["FA_storage_verify", {"uri": "local:///backup/q1.csv", "expected": "sha256:9f86d081884c7d65…"}], + ["FA_storage_list", {"uri": "s3://reports", "recursive": true}] +] +``` + 此 API 为新功能，在 1.0 之前仍可能调整。完整说明请见文档的“通用存储层”章节。 ### 文件监听触发 diff --git a/README.zh-TW.md b/README.zh-TW.md index 9d3960c..97d656e 100644 --- a/README.zh-TW.md +++ b/README.zh-TW.md @@ -479,6 +479,21 @@ File("sandbox://jobs/42/out.csv").write(b"done") 動作使用，其轉接器尚未完成。你可以繼承 `StorageBackend`（物件儲存則繼承 `ObjectStorage`） 撰寫自己的後端，並用 `tests/storage_contract.py` 中 70 個案例的契約測試套件檢查。 +- **動作** — `FA_storage_exists`, `FA_storage_stat`, `FA_storage_list`, `FA_storage_mkdir`, + `FA_storage_upload`, `FA_storage_download`, `FA_storage_delete`, `FA_storage_checksum`, + `FA_storage_verify`, `FA_storage_copy`, `FA_storage_move`, `FA_storage_read_text`, + `FA_storage_write_text`, `FA_storage_schemes`。它們以字串 + 形式接收 URI，並回傳可序列化為 JSON 的值，因此本層可用於動作檔、CLI、TCP 與 HTTP 伺服器， + 也能作為 MCP 工具。在伺服器上請像其他檔案動作一樣以 `ActionACL` 加以限制。 + +```json +[ + ["FA_storage_copy", {"source": "s3://reports/q1.csv", "target": "local:///backup/q1.csv"}], + ["FA_storage_verify", {"uri": "local:///backup/q1.csv", "expected": "sha256:9f86d081884c7d65…"}], + ["FA_storage_list", {"uri": "s3://reports", "recursive": true}] +] +``` + 此 API 為新功能，在 1.0 之前仍可能調整。完整說明請見文件的「通用儲存層」章節。 ### 檔案監看觸發 diff --git a/architecture.md b/architecture.md index f41fee8..d00557a 100644 --- a/architecture.md +++ b/architecture.md @@ -24,7 +24,7 @@ piece of the 1.0 roadmap (`docs/FILEAUTOMATION-1.0-ROADMAP.md`, PR #107); what i | `automation_file/core/` | Engine, on je_action_core: `action_registry.py` (`ActionRegistry`, a `CommandRegistry`; `build_default_registry`), `action_executor.py` (`ActionExecutor`, an `ActionExecutor` with strict actions, indexed records and the dry-run, validate, substitute and parallel extras; shared `executor`), `callback_executor.py`, `package_loader.py`, `plugins.py`, `dag_executor.py`, `action_queue.py`, `json_store.py`, `substitution.py`. Also cross-cutting helpers: `retry`, `quota`, `rate_limit`, `circuit_breaker`, `file_lock`, `sqlite_lock`, `checksum`, `manifest`, `crypto`, `secrets`, `config`, `config_watcher`, `audit`, `metrics`, `tracing`, `progress`, `fim`, `content_store` | | `automation_file/local/` | Local strategy modules: file, dir, zip, tar and archive ops, sync, diff, text/JSON/data edits, templates, versioning, trash, `shell_ops` (argv-only subprocess), conditional branches. `safe_paths.py` guards against path traversal | | `automation_file/remote/` | `url_validator.py` (SSRF guard), `http_download.py`, `cross_backend.py`, `fsspec_bridge.py`. One subpackage per backend: `google_drive/`, `s3/`, `azure_blob/`, `dropbox_api/`, `sftp/`, `ftp/`, `onedrive/`, `box/`, each with `client.py`, `*_ops.py` and `register__ops`. `smb/` and `webdav/` have a client only | -| `automation_file/storage/` | Universal storage layer. `uri.py` (`StorageURI`, `parse_storage_uri`, `normalize_path`), `types.py` (`FileInfo`, `Checksum`, `StorageCapabilities`), `backend.py` (`StorageBackend`: the public operations are template methods over the `_`-prefixed primitives a backend supplies), `local_storage.py` (`LocalStorage`, confined through `safe_join` when given a root), `memory_storage.py` (`MemoryStorage`), `object_storage.py` (`ObjectStorage`: directories as key prefixes over `_head`, `_scan`, `_put`, `_get`, `_remove`), `s3_storage.py` (`S3Storage`, over `s3_instance` or a given boto3 client), `azure_storage.py` (`AzureStorage`, over `azure_blob_instance` or a given `BlobServiceClient`), `resolver.py` (`StorageResolver`, `default_resolver`: mounts first, then scheme factories), `file.py` (`File`), `storage.py` (`Storage`). At module level it imports only `exceptions`, `core.checksum` and `local.safe_paths`: no registry, no GUI, no backend SDK. The adapters import their SDK's exceptions and the shared client inside the functions that use them | +| `automation_file/storage/` | Universal storage layer. `uri.py` (`StorageURI`, `parse_storage_uri`, `normalize_path`), `types.py` (`FileInfo`, `Checksum`, `StorageCapabilities`), `backend.py` (`StorageBackend`: the public operations are template methods over the `_`-prefixed primitives a backend supplies), `local_storage.py` (`LocalStorage`, confined through `safe_join` when given a root), `memory_storage.py` (`MemoryStorage`), `object_storage.py` (`ObjectStorage`: directories as key prefixes over `_head`, `_scan`, `_put`, `_get`, `_remove`), `s3_storage.py` (`S3Storage`, over `s3_instance` or a given boto3 client), `azure_storage.py` (`AzureStorage`, over `azure_blob_instance` or a given `BlobServiceClient`), `resolver.py` (`StorageResolver`, `default_resolver`: mounts first, then scheme factories), `file.py` (`File`), `storage.py` (`Storage`), `actions.py` (the `FA_storage_*` functions and `register_storage_ops`). At module level it imports only `exceptions`, `logging_config`, `core.checksum` and `local.safe_paths`: no registry, no GUI, no backend SDK. The adapters import their SDK's exceptions and the shared client inside the functions that use them | | `automation_file/server/` | `tcp_server.py`, `http_server.py`, `mcp_server.py`, `web_ui.py`, `metrics_server.py`, `action_acl.py` (`ActionACL`), `network_guards.py` (`ensure_loopback`) | | `automation_file/client/` | `HTTPActionClient` for the HTTP action server | | `automation_file/trigger/`, `scheduler/`, `notify/` | Watchdog file triggers, cron scheduler, notification sinks. Each registers its own `FA_*` ops | @@ -51,7 +51,9 @@ piece of the 1.0 roadmap (`docs/FILEAUTOMATION-1.0-ROADMAP.md`, PR #107); what i `ObjectStorage`, `S3Storage`, `AzureStorage`, and `StorageException` with its nine subclasses. Storage URIs are `:///`; the built-in schemes are `local` (alias `file`), `memory`, `s3` and `azure` (alias `az`), and text without `://` is a local path. `s3://` and `azure://` use the shared `s3_instance` / `azure_blob_instance`. The API is - provisional until 1.0. It is not reachable through `FA_*` actions yet. + provisional until 1.0. Fourteen `FA_storage_*` actions (`exists`, `stat`, `list`, `mkdir`, `upload`, + `download`, `delete`, `checksum`, `verify`, `copy`, `move`, `read_text`, `write_text`, `schemes`) put + it in the default registry; `register_storage_ops` adds them to another one. - **Action format**: an action is `[name]`, `[name, {kwargs}]` or `[name, [args]]`. A file holds a list of actions or `{"auto_control": [...]}`. - **CLI** (`python -m automation_file`; no console script for it): @@ -110,6 +112,7 @@ MCP host → automation_file_mcp (stdio JSON-RPC) → tools/call → MCPServer r ``` ActionExecutor() → build_default_registry(): local + http + utils + drive commands → _register_cloud_backends (register__ops) → trigger / scheduler / progress / notify ops + → storage ops (FA_storage_*) → _load_plugins (entry points; may override built-ins) → executor adds FA_execute_action, FA_execute_files, FA_execute_action_parallel, FA_validate ``` diff --git a/automation_file/__init__.py b/automation_file/__init__.py index 72d8584..c3cf927 100644 --- a/automation_file/__init__.py +++ b/automation_file/__init__.py @@ -280,6 +280,7 @@ StorageResolver, StorageURI, parse_storage_uri, + register_storage_ops, ) from automation_file.trigger import ( FileWatcher, @@ -485,6 +486,7 @@ def __getattr__(name: str) -> Any: "StorageResolver", "StorageURI", "parse_storage_uri", + "register_storage_ops", "FileInfo", "Checksum", "StorageCapabilities", diff --git a/automation_file/core/action_registry.py b/automation_file/core/action_registry.py index 80fd705..57f19bf 100644 --- a/automation_file/core/action_registry.py +++ b/automation_file/core/action_registry.py @@ -218,6 +218,12 @@ def _register_notify_ops(registry: ActionRegistry) -> None: register_notify_ops(registry) +def _register_storage_ops(registry: ActionRegistry) -> None: + from automation_file.storage.actions import register_storage_ops + + register_storage_ops(registry) + + def build_default_registry() -> ActionRegistry: """Return a registry pre-populated with every built-in ``FA_*`` action. @@ -235,6 +241,7 @@ def build_default_registry() -> ActionRegistry: _register_scheduler_ops(registry) _register_progress_ops(registry) _register_notify_ops(registry) + _register_storage_ops(registry) _load_plugins(registry) # DEBUG, not INFO: this runs at import, and INFO is mirrored to stderr, so every import -- # `python -m automation_file --help` included -- printed it. diff --git a/automation_file/storage/__init__.py b/automation_file/storage/__init__.py index 5b9f0a3..661f710 100644 --- a/automation_file/storage/__init__.py +++ b/automation_file/storage/__init__.py @@ -6,10 +6,12 @@ :class:`AzureStorage`; :class:`ObjectStorage` is the shared base of the last two. * :class:`StorageURI` / :func:`parse_storage_uri` define the address syntax, and :class:`StorageResolver` maps an address to a backend. +* :func:`register_storage_ops` adds the ``FA_storage_*`` actions to a registry. """ from __future__ import annotations +from automation_file.storage.actions import register_storage_ops from automation_file.storage.azure_storage import AzureStorage from automation_file.storage.backend import StorageBackend from automation_file.storage.file import File @@ -60,4 +62,5 @@ "normalize_path", "parse_storage_uri", "register_default_schemes", + "register_storage_ops", ] diff --git a/automation_file/storage/actions.py b/automation_file/storage/actions.py new file mode 100644 index 0000000..2e97670 --- /dev/null +++ b/automation_file/storage/actions.py @@ -0,0 +1,162 @@ +"""``FA_storage_*`` actions: the storage layer for JSON action lists. + +Each function takes storage URIs as plain strings and returns JSON-friendly +values, so the same call works from Python, an action file, the CLI, the TCP and +HTTP action servers and as an MCP tool: + +.. code-block:: json + + [ + ["FA_storage_copy", {"source": "s3://reports/q1.csv", "target": "local:///backup/q1.csv"}], + ["FA_storage_checksum", {"uri": "local:///backup/q1.csv"}] + ] + +A ``FileInfo`` comes back as its ``to_dict()`` form plus a ``"uri"`` key. Failures +raise the storage layer's exceptions; the executor records them per action. +""" + +from __future__ import annotations + +from collections.abc import Callable +from typing import TYPE_CHECKING, Any + +from automation_file.logging_config import file_automation_logger +from automation_file.storage.backend import DEFAULT_CHECKSUM_ALGORITHM +from automation_file.storage.file import File +from automation_file.storage.storage import Storage +from automation_file.storage.types import FileInfo + +if TYPE_CHECKING: + from automation_file.core.action_registry import ActionRegistry + +_DEFAULT_ENCODING = "utf-8" + + +def _described(info: FileInfo, uri: str) -> dict[str, Any]: + return {"uri": uri, **info.to_dict()} + + +def storage_exists(uri: str) -> bool: + """Return whether a file or a directory is at ``uri``.""" + return File(uri).exists() + + +def storage_stat(uri: str) -> dict[str, Any]: + """Return the size, modification time and other metadata of ``uri``.""" + target = File(uri) + return _described(target.stat(), str(target)) + + +def storage_list(uri: str, recursive: bool = False) -> list[dict[str, Any]]: + """List the directory ``uri``; ``recursive`` adds every descendant. + + Each entry's ``path`` is relative to ``uri`` and its ``uri`` is absolute. + """ + directory = Storage(uri) + return [ + _described(info, str(directory.uri.joinpath(info.path))) + for info in directory.list_dir(recursive=recursive) + ] + + +def storage_mkdir(uri: str, parents: bool = True, exist_ok: bool = True) -> bool: + """Create the directory ``uri``.""" + Storage(uri).mkdir(parents=parents, exist_ok=exist_ok) + file_automation_logger.info("storage_mkdir: %s", uri) + return True + + +def storage_upload(local_path: str, uri: str, overwrite: bool = True) -> dict[str, Any]: + """Store the local file ``local_path`` at ``uri``.""" + target = File(uri) + info = target.upload_from(local_path, overwrite=overwrite) + file_automation_logger.info("storage_upload: %s -> %s", local_path, target) + return _described(info, str(target)) + + +def storage_download(uri: str, local_path: str, overwrite: bool = True) -> str: + """Write the file ``uri`` to ``local_path`` and return that path.""" + source = File(uri) + written = source.download_to(local_path, overwrite=overwrite) + file_automation_logger.info("storage_download: %s -> %s", source, written) + return str(written) + + +def storage_delete(uri: str, recursive: bool = False, missing_ok: bool = False) -> bool: + """Remove the file or directory at ``uri``; a directory with entries needs ``recursive``.""" + target = Storage(uri) + target.delete("", recursive=recursive, missing_ok=missing_ok) + file_automation_logger.info("storage_delete: %s (recursive=%s)", target, recursive) + return True + + +def storage_checksum(uri: str, algorithm: str = DEFAULT_CHECKSUM_ALGORITHM) -> dict[str, str]: + """Return ``{"algorithm": ..., "value": ...}`` for the content of ``uri``.""" + return File(uri).checksum(algorithm).to_dict() + + +def storage_verify(uri: str, expected: str, algorithm: str = DEFAULT_CHECKSUM_ALGORITHM) -> bool: + """Return whether ``uri`` has the digest ``expected`` (``"sha256:..."`` or a bare digest).""" + matched = File(uri).verify(expected, algorithm=algorithm) + if not matched: + file_automation_logger.warning("storage_verify mismatch: %s", uri) + return matched + + +def storage_copy(source: str, target: str, overwrite: bool = True) -> dict[str, Any]: + """Copy the file ``source`` to ``target``, in the same backend or another one.""" + copied = File(source).copy_to(target, overwrite=overwrite) + file_automation_logger.info("storage_copy: %s -> %s", source, copied) + return _described(copied.stat(), str(copied)) + + +def storage_move(source: str, target: str, overwrite: bool = True) -> dict[str, Any]: + """Move the file ``source`` to ``target``, in the same backend or another one.""" + moved = File(source).move_to(target, overwrite=overwrite) + file_automation_logger.info("storage_move: %s -> %s", source, moved) + return _described(moved.stat(), str(moved)) + + +def storage_read_text(uri: str, encoding: str = _DEFAULT_ENCODING) -> str: + """Return the content of ``uri`` decoded as text.""" + return File(uri).read_text(encoding) + + +def storage_write_text( + uri: str, text: str, overwrite: bool = True, encoding: str = _DEFAULT_ENCODING +) -> dict[str, Any]: + """Store ``text`` as the content of ``uri``.""" + target = File(uri) + info = target.write(text, overwrite=overwrite, encoding=encoding) + file_automation_logger.info("storage_write_text: %s", target) + return _described(info, str(target)) + + +def storage_schemes() -> list[str]: + """Return the URI schemes a backend is registered or mounted for.""" + return Storage.schemes() + + +def storage_commands() -> dict[str, Callable[..., Any]]: + """Return every ``FA_storage_*`` action by name.""" + return { + "FA_storage_exists": storage_exists, + "FA_storage_stat": storage_stat, + "FA_storage_list": storage_list, + "FA_storage_mkdir": storage_mkdir, + "FA_storage_upload": storage_upload, + "FA_storage_download": storage_download, + "FA_storage_delete": storage_delete, + "FA_storage_checksum": storage_checksum, + "FA_storage_verify": storage_verify, + "FA_storage_copy": storage_copy, + "FA_storage_move": storage_move, + "FA_storage_read_text": storage_read_text, + "FA_storage_write_text": storage_write_text, + "FA_storage_schemes": storage_schemes, + } + + +def register_storage_ops(registry: ActionRegistry) -> None: + """Register every ``FA_storage_*`` command into ``registry``.""" + registry.register_many(storage_commands()) diff --git a/docs/source/API/storage.rst b/docs/source/API/storage.rst index e50154b..3946318 100644 --- a/docs/source/API/storage.rst +++ b/docs/source/API/storage.rst @@ -14,6 +14,12 @@ File and Storage .. automodule:: automation_file.storage.storage :members: +Actions +------- + +.. automodule:: automation_file.storage.actions + :members: + Storage URIs ------------ diff --git a/docs/source/Eng/usage/storage.rst b/docs/source/Eng/usage/storage.rst index 12c5d70..3e3f6df 100644 --- a/docs/source/Eng/usage/storage.rst +++ b/docs/source/Eng/usage/storage.rst @@ -145,6 +145,84 @@ says which optional ``FileInfo`` fields the backend fills in and whether its directories are real (``directories=True``, a filesystem) or implied by file paths (``directories=False``, an object store). +Actions +------- + +The layer is also reachable from JSON action lists, and so from the CLI, the TCP +and HTTP action servers and MCP hosts. Every ``FA_storage_*`` action takes URIs as +strings and returns JSON-friendly values. + +.. list-table:: + :header-rows: 1 + :widths: 28 40 32 + + * - Action + - Parameters + - Returns + * - ``FA_storage_exists`` + - ``uri`` + - ``true`` / ``false`` + * - ``FA_storage_stat`` + - ``uri`` + - The file information + * - ``FA_storage_list`` + - ``uri, recursive=False`` + - A list of file information + * - ``FA_storage_mkdir`` + - ``uri, parents=True, exist_ok=True`` + - ``True`` + * - ``FA_storage_upload`` + - ``local_path, uri, overwrite=True`` + - The file information + * - ``FA_storage_download`` + - ``uri, local_path, overwrite=True`` + - The local path + * - ``FA_storage_delete`` + - ``uri, recursive=False, missing_ok=False`` + - ``True`` + * - ``FA_storage_checksum`` + - ``uri, algorithm="sha256"`` + - ``{"algorithm": …, "value": …}`` + * - ``FA_storage_verify`` + - ``uri, expected, algorithm="sha256"`` + - ``true`` / ``false`` + * - ``FA_storage_copy`` + - ``source, target, overwrite=True`` + - The target's file information + * - ``FA_storage_move`` + - ``source, target, overwrite=True`` + - The target's file information + * - ``FA_storage_read_text`` + - ``uri, encoding="utf-8"`` + - The content as text + * - ``FA_storage_write_text`` + - ``uri, text, overwrite=True, encoding="utf-8"`` + - The file information + * - ``FA_storage_schemes`` + - — + - The registered schemes + +File information is ``FileInfo.to_dict()`` plus a ``uri`` key: ``uri``, ``path``, +``name``, ``is_dir``, ``size``, ``modified_at`` (ISO 8601), ``etag``, ``version``, +``content_type`` and ``metadata``. In ``FA_storage_list`` each ``path`` is relative +to the listed URI. A failure raises the exception from `Errors`_, which the +executor records for that action without stopping the list. + +.. code-block:: json + + [ + ["FA_storage_copy", {"source": "s3://reports/2026/q1.csv", + "target": "local:///backup/2026/q1.csv"}], + ["FA_storage_verify", {"uri": "local:///backup/2026/q1.csv", + "expected": "sha256:9f86d081884c7d65…"}], + ["FA_storage_list", {"uri": "s3://reports/2026", "recursive": true}] + ] + +Like the other file actions, these reach whatever the process can reach. On a TCP +or HTTP action server pass an :class:`~automation_file.ActionACL`, and on the MCP +server ``--allowed-actions``, to expose only the ones a client needs. +:func:`~automation_file.register_storage_ops` adds them to a registry of your own. + Errors ------ diff --git a/docs/source/Zh-CN/usage/storage.rst b/docs/source/Zh-CN/usage/storage.rst index af804f1..8c5392d 100644 --- a/docs/source/Zh-CN/usage/storage.rst +++ b/docs/source/Zh-CN/usage/storage.rst @@ -135,6 +135,83 @@ API；:class:`~automation_file.StorageBackend` 则是后端需要实现的契约 会填入哪些可选的 ``FileInfo`` 字段，以及它的目录是真实存在（``directories=True``， 文件系统）还是由文件路径隐含（``directories=False``，对象存储）。 +动作 +---- + +本层也可以从 JSON 动作列表使用，因此 CLI、TCP 与 HTTP 动作服务器以及 MCP 主机都能调用。 +每个 ``FA_storage_*`` 动作都以字符串形式接收 URI，并返回可以序列化为 JSON 的值。 + +.. list-table:: + :header-rows: 1 + :widths: 28 40 32 + + * - 动作 + - 参数 + - 返回值 + * - ``FA_storage_exists`` + - ``uri`` + - ``true`` / ``false`` + * - ``FA_storage_stat`` + - ``uri`` + - 文件信息 + * - ``FA_storage_list`` + - ``uri, recursive=False`` + - 文件信息的列表 + * - ``FA_storage_mkdir`` + - ``uri, parents=True, exist_ok=True`` + - ``True`` + * - ``FA_storage_upload`` + - ``local_path, uri, overwrite=True`` + - 文件信息 + * - ``FA_storage_download`` + - ``uri, local_path, overwrite=True`` + - 本地路径 + * - ``FA_storage_delete`` + - ``uri, recursive=False, missing_ok=False`` + - ``True`` + * - ``FA_storage_checksum`` + - ``uri, algorithm="sha256"`` + - ``{"algorithm": …, "value": …}`` + * - ``FA_storage_verify`` + - ``uri, expected, algorithm="sha256"`` + - ``true`` / ``false`` + * - ``FA_storage_copy`` + - ``source, target, overwrite=True`` + - 目标的文件信息 + * - ``FA_storage_move`` + - ``source, target, overwrite=True`` + - 目标的文件信息 + * - ``FA_storage_read_text`` + - ``uri, encoding="utf-8"`` + - 文本内容 + * - ``FA_storage_write_text`` + - ``uri, text, overwrite=True, encoding="utf-8"`` + - 文件信息 + * - ``FA_storage_schemes`` + - — + - 已注册的 scheme + +文件信息是 ``FileInfo.to_dict()`` 再加上 ``uri`` 键：``uri``、``path``、``name``、 +``is_dir``、``size``、``modified_at``（ISO 8601）、``etag``、``version``、 +``content_type`` 与 ``metadata``。在 ``FA_storage_list`` 中，每个 ``path`` 都相对于 +被列出的 URI。失败时会抛出 `异常`_ 一节中的异常，执行器会把它记录在该动作上， +不会中断整份列表。 + +.. code-block:: json + + [ + ["FA_storage_copy", {"source": "s3://reports/2026/q1.csv", + "target": "local:///backup/2026/q1.csv"}], + ["FA_storage_verify", {"uri": "local:///backup/2026/q1.csv", + "expected": "sha256:9f86d081884c7d65…"}], + ["FA_storage_list", {"uri": "s3://reports/2026", "recursive": true}] + ] + +与其他文件动作一样，这些动作能访问进程所能访问的一切。在 TCP 或 HTTP 动作服务器上 +请传入 :class:`~automation_file.ActionACL`，在 MCP 服务器上请使用 +``--allowed-actions``，只开放客户端需要的动作。 +:func:`~automation_file.register_storage_ops` 可以把它们加入你自己的注册表。 + 异常 ---- diff --git a/docs/source/Zh-TW/usage/storage.rst b/docs/source/Zh-TW/usage/storage.rst index 5956939..39e99a6 100644 --- a/docs/source/Zh-TW/usage/storage.rst +++ b/docs/source/Zh-TW/usage/storage.rst @@ -135,6 +135,83 @@ API；:class:`~automation_file.StorageBackend` 則是後端要實作的契約。 會填入哪些選用的 ``FileInfo`` 欄位，以及它的目錄是真實存在（``directories=True``， 檔案系統）還是由檔案路徑隱含（``directories=False``，物件儲存）。 +動作 +---- + +本層也能從 JSON 動作清單使用，因此 CLI、TCP 與 HTTP 動作伺服器以及 MCP 主機都能呼叫。 +每個 ``FA_storage_*`` 動作都以字串形式接收 URI，並回傳可序列化為 JSON 的值。 + +.. list-table:: + :header-rows: 1 + :widths: 28 40 32 + + * - 動作 + - 參數 + - 回傳值 + * - ``FA_storage_exists`` + - ``uri`` + - ``true`` / ``false`` + * - ``FA_storage_stat`` + - ``uri`` + - 檔案資訊 + * - ``FA_storage_list`` + - ``uri, recursive=False`` + - 檔案資訊的清單 + * - ``FA_storage_mkdir`` + - ``uri, parents=True, exist_ok=True`` + - ``True`` + * - ``FA_storage_upload`` + - ``local_path, uri, overwrite=True`` + - 檔案資訊 + * - ``FA_storage_download`` + - ``uri, local_path, overwrite=True`` + - 本機路徑 + * - ``FA_storage_delete`` + - ``uri, recursive=False, missing_ok=False`` + - ``True`` + * - ``FA_storage_checksum`` + - ``uri, algorithm="sha256"`` + - ``{"algorithm": …, "value": …}`` + * - ``FA_storage_verify`` + - ``uri, expected, algorithm="sha256"`` + - ``true`` / ``false`` + * - ``FA_storage_copy`` + - ``source, target, overwrite=True`` + - 目標的檔案資訊 + * - ``FA_storage_move`` + - ``source, target, overwrite=True`` + - 目標的檔案資訊 + * - ``FA_storage_read_text`` + - ``uri, encoding="utf-8"`` + - 文字內容 + * - ``FA_storage_write_text`` + - ``uri, text, overwrite=True, encoding="utf-8"`` + - 檔案資訊 + * - ``FA_storage_schemes`` + - — + - 已註冊的 scheme + +檔案資訊是 ``FileInfo.to_dict()`` 再加上 ``uri`` 鍵：``uri``、``path``、``name``、 +``is_dir``、``size``、``modified_at``（ISO 8601）、``etag``、``version``、 +``content_type`` 與 ``metadata``。在 ``FA_storage_list`` 中，每個 ``path`` 都相對於 +被列出的 URI。失敗時會拋出 `例外`_ 一節中的例外，執行器會把它記錄在該動作上， +不會中斷整份清單。 + +.. code-block:: json + + [ + ["FA_storage_copy", {"source": "s3://reports/2026/q1.csv", + "target": "local:///backup/2026/q1.csv"}], + ["FA_storage_verify", {"uri": "local:///backup/2026/q1.csv", + "expected": "sha256:9f86d081884c7d65…"}], + ["FA_storage_list", {"uri": "s3://reports/2026", "recursive": true}] + ] + +與其他檔案動作一樣，這些動作能存取行程所能存取的一切。在 TCP 或 HTTP 動作伺服器上 +請傳入 :class:`~automation_file.ActionACL`，在 MCP 伺服器上請使用 +``--allowed-actions``，只開放用戶端需要的動作。 +:func:`~automation_file.register_storage_ops` 可把它們加入你自己的註冊表。 + 例外 ---- diff --git a/docs/updates/2026-10.md b/docs/updates/2026-10.md index 2863b65..f84749a 100644 --- a/docs/updates/2026-10.md +++ b/docs/updates/2026-10.md @@ -236,3 +236,18 @@ Index and query commands: [README.md](README.md). New entries go at the end. - **Docs**: the three `usage/storage.rst` pages (the two backends, object-store behaviour, `ObjectStorage` for new backends), `docs/source/API/storage.rst`, the three READMEs (feature bullet, diagram, "Backends today"), `architecture.md` §2, §3 and §5, `CLAUDE.md` (package map, key types). - **Files**: `automation_file/storage/object_storage.py`, `s3_storage.py`, `azure_storage.py`, `backend.py`, `local_storage.py`, `resolver.py`, `__init__.py`, `automation_file/__init__.py`, `tests/test_storage_s3.py`, `tests/test_storage_azure.py`, `tests/test_storage_resolver.py`, the documentation above, `progress.md`. - **Open items**: `progress.md` #13 (Dropbox, SFTP, FTP, WebDAV, SMB, fsspec), #19. + +## U-20261008-03 · 2026-10-08 · FA_storage_* actions put the storage layer in the registry · #storage #roadmap #actions #mcp + +- **What**: the action half of `progress.md` #16, which stays open for `copy_between`. `automation_file/storage/actions.py` holds fourteen functions over `File` and `Storage`, registered by `build_default_registry()` after the notify ops and before the plugins: + - `FA_storage_exists`, `FA_storage_stat`, `FA_storage_list`, `FA_storage_mkdir`, `FA_storage_upload`, `FA_storage_download`, `FA_storage_delete`, `FA_storage_checksum`, `FA_storage_verify`, `FA_storage_copy`, `FA_storage_move`, `FA_storage_read_text`, `FA_storage_write_text`, `FA_storage_schemes`. + - They take URIs as strings and return JSON-friendly values: a `FileInfo` as `to_dict()` plus a `uri` key, a checksum as `{"algorithm", "value"}`. In `FA_storage_list` each `path` is relative to the listed URI and each `uri` is absolute. + - The mutating ones log one line through `file_automation_logger`, like the other ops. A storage URI cannot carry credentials, so the line cannot leak one. + - `register_storage_ops(registry)` is exported from the facade for custom registries. The registry is imported only for type checking, so the storage package still does not depend on it at run time; `logging_config` joins the allowed module-level imports (`tests/test_storage_imports.py`). + - Because the MCP bridge builds its tools from the registry, the fourteen actions are MCP tools with their parameters without further code. +- **Security**: no new exposure. The actions reach what the process reaches, as `FA_copy_file`, `FA_remove_dir_tree` or `FA_run_shell` already do; the servers stay loopback-only with the optional secret, and `ActionACL` / `--allowed-actions` limit which actions a client may call. The documentation says so next to the action table. +- **Tests**: `tests/test_storage_actions.py`, 11 cases: the default and a custom registry hold exactly the fourteen names; each action's result shape; overwrite, encoding and error cases; an action list through `execute_action` with keyword and positional arguments, a JSON round trip of the results and one failing action recorded without stopping the list; the MCP catalogue's schema for `FA_storage_copy`. +- **Result / numbers**: `ruff check` and `ruff format --check` pass; `mypy automation_file` finds no issues in 173 files. `pytest tests/`: 1470 passed, 20 skipped, 5 failed, on Python 3.14.7 on Windows. The 5 are the `test_versioning.py` path-length failures of `progress.md` #28. +- **Docs**: an "Actions" section with the table and a JSON example in the three `usage/storage.rst` pages, `docs/source/API/storage.rst`, an "Actions" bullet and example in the three READMEs, `architecture.md` §2, §3 and §4, `CLAUDE.md` (package map, the import rule). +- **Files**: `automation_file/storage/actions.py`, `automation_file/storage/__init__.py`, `automation_file/core/action_registry.py`, `automation_file/__init__.py`, `tests/test_storage_actions.py`, `tests/test_storage_imports.py`, the documentation above, `progress.md`. +- **Open items**: `progress.md` #16 (`copy_between` on the layer), #25 (the semantic MCP tools of the roadmap, with a permission model and dry run). diff --git a/docs/updates/README.md b/docs/updates/README.md index ab472dd..b33fdbe 100644 --- a/docs/updates/README.md +++ b/docs/updates/README.md @@ -58,6 +58,7 @@ In the same commit: delete the item from `progress.md`, add a `#done` entry here | ID | Date | Title | Tags | Batch | |---|---|---|---|---| +| U-20261008-03 | 2026-10-08 | FA_storage_* actions put the storage layer in the registry | #storage #roadmap #actions #mcp | [2026-10](2026-10.md) | | U-20261008-02 | 2026-10-08 | S3 and Azure Blob behind the storage layer | #storage #roadmap #s3 #azure | [2026-10](2026-10.md) | | U-20261008-01 | 2026-10-08 | Universal storage layer: contract, URIs, local and memory | #storage #roadmap #tests | [2026-10](2026-10.md) | | U-20261001-11 | 2026-10-01 | The publish jobs build with the locked setuptools | #done #ci #security #X-13 | [2026-10](2026-10.md) | @@ -97,5 +98,5 @@ In the same commit: delete the item from `progress.md`, add a `#done` entry here | File | Period | Entries | |---|---|---:| -| [2026-10.md](2026-10.md) | 2026-10 | 13 | +| [2026-10.md](2026-10.md) | 2026-10 | 14 | | [2026-09.md](2026-09.md) | 2026-09 | 21 | diff --git a/progress.md b/progress.md index f31ce7e..8eaf639 100644 --- a/progress.md +++ b/progress.md @@ -19,7 +19,7 @@ Items #10 to #26 are what is left of the 1.0 roadmap (`docs/FILEAUTOMATION-1.0-R - **#13** Storage adapters over the remaining clients, each in `storage/_storage.py` with a `StorageContract` class against a stand-in client: Dropbox (`dropbox:///path`), SFTP (`sftp://host/path`), FTP and FTPS, WebDAV, SMB (`smb://server/share/path`), fsspec. S3 and Azure Blob are done (U-20261008-02) and show the pattern: import the shared client and the SDK's exceptions inside functions, so `tests/test_storage_imports.py` keeps passing. A session backend must refuse a URI whose host is not the one it is connected to; `SFTPClient` does not keep its host today. - **#14** Google Drive adapter (`gdrive://`). Drive addresses files by ID and allows two files of one name in a folder, so the path-to-ID lookup and the duplicate-name rule have to be designed first. - **#15** [DECIDE] The eleventh backend slot, and whether OneDrive and Box are promoted to the storage contract or documented as action-only (roadmap §4). -- **#16** Cross-backend operations through the layer: `copy_between` / `FA_copy_between` on `File.copy_to`, and `FA_storage_*` actions. `copy_between` accepts `local:`, `sftp:/path`, `s3:bucket/key` and http(s) sources today; `parse_storage_uri` rejects the first three as ambiguous, so the action needs a translation step to stay compatible. +- **#16** `copy_between` / `FA_copy_between` on `File.copy_to`. The `FA_storage_*` actions exist (U-20261008-03), so the layer is reachable from action lists; the older action still has its own dispatcher in `remote/cross_backend.py`. `copy_between` accepts `local:`, `sftp:/path`, `s3:bucket/key` and http(s) sources today; `parse_storage_uri` rejects the first three as ambiguous, so the action needs a translation step to stay compatible. - **#17** Streams and directory trees: `open()` or chunked reads and writes (`read_bytes` holds the whole file in memory, and the default `checksum` stages a full local copy of a remote file), directory copy and sync through the layer, and a backend's native checksum where it has one. - **#18** [UNVERIFIED] The five symbolic-link tests of `tests/test_storage_local.py` have not run anywhere: the development machine may not create links (they skip there), CI's Windows runners may. Read the first CI run of the branch and fix `LocalStorage` if one fails. diff --git a/tests/test_storage_actions.py b/tests/test_storage_actions.py new file mode 100644 index 0000000..2179d1e --- /dev/null +++ b/tests/test_storage_actions.py @@ -0,0 +1,196 @@ +"""FA_storage_* actions: the storage layer through the registry, the executor and MCP.""" + +from __future__ import annotations + +import hashlib +import json +from collections.abc import Iterator +from pathlib import Path + +import pytest + +from automation_file import ( + ActionRegistry, + build_default_registry, + execute_action, + register_storage_ops, + tools_from_registry, +) +from automation_file.exceptions import ( + StorageAlreadyExistsException, + StorageNotEmptyException, + StorageNotFoundException, + StorageUnsupportedException, +) +from automation_file.storage import actions, clear_memory_stores, memory_store + +NAMES = [ + "FA_storage_checksum", + "FA_storage_copy", + "FA_storage_delete", + "FA_storage_download", + "FA_storage_exists", + "FA_storage_list", + "FA_storage_mkdir", + "FA_storage_move", + "FA_storage_read_text", + "FA_storage_schemes", + "FA_storage_stat", + "FA_storage_upload", + "FA_storage_verify", + "FA_storage_write_text", +] +SHA256_HELLO = hashlib.sha256(b"hello").hexdigest() + + +@pytest.fixture(autouse=True) +def _fresh_stores() -> Iterator[None]: + clear_memory_stores() + yield + clear_memory_stores() + + +def test_the_default_registry_has_every_storage_action() -> None: + registry = build_default_registry() + assert sorted(name for name in registry.event_dict if name.startswith("FA_storage_")) == NAMES + + +def test_register_storage_ops_fills_a_custom_registry() -> None: + registry = ActionRegistry() + register_storage_ops(registry) + assert sorted(registry.event_dict) == NAMES + assert sorted(actions.storage_commands()) == NAMES + + +def test_write_stat_and_read() -> None: + written = actions.storage_write_text("memory://scratch/dir/note.txt", "hello") + assert written["uri"] == "memory://scratch/dir/note.txt" + assert written["path"] == "dir/note.txt" + assert written["size"] == 5 + assert written["is_dir"] is False + assert actions.storage_exists("memory://scratch/dir/note.txt") is True + assert actions.storage_exists("memory://scratch/dir/nope.txt") is False + assert actions.storage_stat("memory://scratch/dir/note.txt") == written + assert actions.storage_read_text("memory://scratch/dir/note.txt") == "hello" + assert actions.storage_stat("memory://scratch/dir")["is_dir"] is True + + +def test_write_respects_overwrite_and_encoding() -> None: + actions.storage_write_text("memory://scratch/a.txt", "été", encoding="latin-1") + assert memory_store("scratch").read_bytes("a.txt") == b"\xe9t\xe9" + assert actions.storage_read_text("memory://scratch/a.txt", encoding="latin-1") == "été" + with pytest.raises(StorageAlreadyExistsException): + actions.storage_write_text("memory://scratch/a.txt", "again", overwrite=False) + + +def test_list_reports_relative_paths_and_absolute_uris() -> None: + for path in ("reports/2026/q1.csv", "reports/2026/q2.csv", "reports/readme.txt"): + actions.storage_write_text(f"memory://scratch/{path}", "x") + shallow = actions.storage_list("memory://scratch/reports") + assert [(entry["path"], entry["is_dir"]) for entry in shallow] == [ + ("2026", True), + ("readme.txt", False), + ] + assert [entry["uri"] for entry in shallow] == [ + "memory://scratch/reports/2026", + "memory://scratch/reports/readme.txt", + ] + deep = actions.storage_list("memory://scratch/reports", recursive=True) + assert [entry["path"] for entry in deep] == [ + "2026", + "2026/q1.csv", + "2026/q2.csv", + "readme.txt", + ] + assert deep[1]["uri"] == "memory://scratch/reports/2026/q1.csv" + + +def test_upload_download_copy_and_move(tmp_path: Path) -> None: + source = tmp_path / "source.txt" + source.write_bytes(b"hello") + uploaded = actions.storage_upload(str(source), "memory://scratch/in/a.txt") + assert (uploaded["uri"], uploaded["size"]) == ("memory://scratch/in/a.txt", 5) + + copied = actions.storage_copy("memory://scratch/in/a.txt", str(tmp_path / "copy" / "a.txt")) + assert (tmp_path / "copy" / "a.txt").read_bytes() == b"hello" + assert copied["uri"].startswith("local:///") + assert copied["size"] == 5 + with pytest.raises(StorageAlreadyExistsException): + actions.storage_copy( + "memory://scratch/in/a.txt", str(tmp_path / "copy" / "a.txt"), overwrite=False + ) + + moved = actions.storage_move("memory://scratch/in/a.txt", "memory://archive/2026/a.txt") + assert moved["uri"] == "memory://archive/2026/a.txt" + assert actions.storage_exists("memory://scratch/in/a.txt") is False + + target = tmp_path / "down" / "a.txt" + assert actions.storage_download("memory://archive/2026/a.txt", str(target)) == str(target) + assert target.read_bytes() == b"hello" + + +def test_checksum_and_verify() -> None: + actions.storage_write_text("memory://scratch/a.txt", "hello") + assert actions.storage_checksum("memory://scratch/a.txt") == { + "algorithm": "sha256", + "value": SHA256_HELLO, + } + md5 = hashlib.md5(b"hello", usedforsecurity=False).hexdigest() + assert actions.storage_checksum("memory://scratch/a.txt", "md5")["value"] == md5 + assert actions.storage_verify("memory://scratch/a.txt", SHA256_HELLO) is True + assert actions.storage_verify("memory://scratch/a.txt", f"md5:{md5}") is True + assert actions.storage_verify("memory://scratch/a.txt", md5, algorithm="md5") is True + assert actions.storage_verify("memory://scratch/a.txt", "0" * 64) is False + with pytest.raises(StorageUnsupportedException): + actions.storage_checksum("memory://scratch/a.txt", "no-such-hash") + + +def test_mkdir_and_delete() -> None: + assert actions.storage_mkdir("memory://scratch/empty/nested") is True + assert actions.storage_stat("memory://scratch/empty/nested")["is_dir"] is True + actions.storage_write_text("memory://scratch/empty/nested/a.txt", "x") + with pytest.raises(StorageNotEmptyException): + actions.storage_delete("memory://scratch/empty") + assert actions.storage_delete("memory://scratch/empty/nested/a.txt") is True + assert actions.storage_delete("memory://scratch/empty", recursive=True) is True + assert actions.storage_exists("memory://scratch/empty") is False + with pytest.raises(StorageNotFoundException): + actions.storage_delete("memory://scratch/empty") + assert actions.storage_delete("memory://scratch/empty", missing_ok=True) is True + with pytest.raises(StorageUnsupportedException): + actions.storage_delete("memory://scratch", recursive=True) + + +def test_schemes_lists_the_registered_backends() -> None: + assert {"local", "memory", "s3", "azure"} <= set(actions.storage_schemes()) + + +def test_an_action_list_runs_through_the_executor(tmp_path: Path) -> None: + results = execute_action( + [ + ["FA_storage_write_text", {"uri": "memory://scratch/in/a.txt", "text": "hello"}], + ["FA_storage_copy", ["memory://scratch/in/a.txt", str(tmp_path / "a.txt")]], + ["FA_storage_checksum", {"uri": str(tmp_path / "a.txt")}], + ["FA_storage_list", {"uri": "memory://scratch", "recursive": True}], + ["FA_storage_read_text", {"uri": "memory://scratch/in/missing.txt"}], + ["FA_storage_schemes"], + ] + ) + values = list(results.values()) + assert json.loads(json.dumps(values[:4])) == values[:4] + assert values[0]["uri"] == "memory://scratch/in/a.txt" + assert (tmp_path / "a.txt").read_bytes() == b"hello" + assert values[2] == {"algorithm": "sha256", "value": SHA256_HELLO} + assert [entry["path"] for entry in values[3]] == ["in", "in/a.txt"] + assert "StorageNotFoundException" in values[4] + assert "memory" in values[5] + + +def test_the_actions_are_mcp_tools_with_their_parameters() -> None: + tools = {tool["name"]: tool for tool in tools_from_registry(build_default_registry())} + assert set(NAMES) <= set(tools) + copy = tools["FA_storage_copy"]["inputSchema"] + assert list(copy["properties"]) == ["source", "target", "overwrite"] + assert copy["required"] == ["source", "target"] + assert tools["FA_storage_schemes"]["inputSchema"].get("required", []) == [] + assert "Copy the file" in tools["FA_storage_copy"]["description"] diff --git a/tests/test_storage_imports.py b/tests/test_storage_imports.py index d0e4c8a..c74f1ad 100644 --- a/tests/test_storage_imports.py +++ b/tests/test_storage_imports.py @@ -2,8 +2,8 @@ It reads the module-level imports of every file under ``automation_file/storage``: each one is either from the standard library or from the short list of first-party -modules below. An SDK imported lazily inside a function is not a module-level -import and stays allowed. +modules below. An SDK imported lazily inside a function, or a name imported only +for type checking, is not a module-level import and stays allowed. """ from __future__ import annotations @@ -17,6 +17,7 @@ STORAGE_PACKAGE = Path(__file__).resolve().parent.parent / "automation_file" / "storage" ALLOWED_FIRST_PARTY = ( "automation_file.exceptions", + "automation_file.logging_config", "automation_file.core.checksum", "automation_file.local.safe_paths", "automation_file.storage", From 2cd734309b030035d6c18af183fadc19be3ae884 Mon Sep 17 00:00:00 2001 From: JeffreyChen Date: Thu, 8 Oct 2026 12:08:47 +0800 Subject: [PATCH 23/59] refactor: let a storage backend instance carry its own scheme and capabilities --- automation_file/storage/backend.py | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/automation_file/storage/backend.py b/automation_file/storage/backend.py index c937eac..f4b5be1 100644 --- a/automation_file/storage/backend.py +++ b/automation_file/storage/backend.py @@ -25,7 +25,7 @@ from collections.abc import Iterable from pathlib import Path, PurePosixPath from types import TracebackType -from typing import ClassVar, TypeVar +from typing import TypeVar from automation_file.core.checksum import file_checksum from automation_file.exceptions import ( @@ -96,8 +96,10 @@ def _by_depth(info: FileInfo) -> int: class StorageBackend(ABC): """One storage root: a directory tree, a bucket, a share, a remote session.""" - scheme: ClassVar[str] = "" - capabilities: ClassVar[StorageCapabilities] = StorageCapabilities() + #: Class-wide defaults. A backend whose instances differ (one adapter over many + #: kinds of filesystem) assigns its own in ``__init__``. + scheme: str = "" + capabilities: StorageCapabilities = StorageCapabilities() # ------------------------------------------------------------------ primitives From c6f49689805c7ccde2607babe6a0483924b9ddd5 Mon Sep 17 00:00:00 2001 From: JeffreyChen Date: Thu, 8 Oct 2026 12:12:22 +0800 Subject: [PATCH 24/59] fix: keep version directories short enough for long source paths --- automation_file/local/versioning.py | 25 ++++++++++- docs/updates/2026-10.md | 12 +++++ docs/updates/README.md | 3 +- progress.md | 1 - tests/test_versioning.py | 69 +++++++++++++++++++++++++++++ 5 files changed, 106 insertions(+), 4 deletions(-) diff --git a/automation_file/local/versioning.py b/automation_file/local/versioning.py index d75db91..3b1a4f8 100644 --- a/automation_file/local/versioning.py +++ b/automation_file/local/versioning.py @@ -7,6 +7,7 @@ from __future__ import annotations +import hashlib import os import re import shutil @@ -17,6 +18,11 @@ from automation_file.exceptions import VersioningException _VERSION_RE = re.compile(r"^v(\d+)__(\d+)$") +# A flattened source path longer than this is shortened to its tail plus a digest: +# the directory sits below the versions root, and Windows caps a whole path at 260. +_MAX_BUCKET_NAME = 80 +_BUCKET_TAIL = 48 +_BUCKET_DIGEST = 16 @dataclass(frozen=True) @@ -33,7 +39,10 @@ class FileVersioner: Each source file is versioned in its own subdirectory so multiple files can coexist. The subdirectory name is the source path's POSIX form with - path separators replaced by ``__sep__`` to flatten safely. + path separators replaced by ``__sep__`` to flatten safely. A long name is + cut to its last characters plus a digest of the whole path, so a deep + source path does not push the snapshot past the platform's path limit; a + directory already created under the long name keeps being used. """ def __init__(self, root: str | os.PathLike[str]) -> None: @@ -97,7 +106,12 @@ def prune(self, path: str | os.PathLike[str], keep: int) -> int: def _bucket_for(self, src: Path) -> Path: safe = _flatten_path(src) - return self._root / safe + if len(safe) <= _MAX_BUCKET_NAME: + return self._root / safe + legacy = self._root / safe + if legacy.is_dir(): + return legacy + return self._root / _shortened(safe) def _next_version(self, bucket: Path) -> int: highest = 0 @@ -115,3 +129,10 @@ def _flatten_path(src: Path) -> str: flat = (drive.replace(":", "") + body).replace(os.sep, "__sep__") flat = flat.replace("/", "__sep__") return flat.strip("_") or "root" + + +def _shortened(flat: str) -> str: + # normcase: the same file spelled in another case must land in the same directory. + digest = hashlib.sha256(os.path.normcase(flat).encode("utf-8")).hexdigest() + tail = flat[-_BUCKET_TAIL:].lstrip("_") + return f"{tail}__{digest[:_BUCKET_DIGEST]}" diff --git a/docs/updates/2026-10.md b/docs/updates/2026-10.md index f84749a..eb1d00b 100644 --- a/docs/updates/2026-10.md +++ b/docs/updates/2026-10.md @@ -251,3 +251,15 @@ Index and query commands: [README.md](README.md). New entries go at the end. - **Docs**: an "Actions" section with the table and a JSON example in the three `usage/storage.rst` pages, `docs/source/API/storage.rst`, an "Actions" bullet and example in the three READMEs, `architecture.md` §2, §3 and §4, `CLAUDE.md` (package map, the import rule). - **Files**: `automation_file/storage/actions.py`, `automation_file/storage/__init__.py`, `automation_file/core/action_registry.py`, `automation_file/__init__.py`, `tests/test_storage_actions.py`, `tests/test_storage_imports.py`, the documentation above, `progress.md`. - **Open items**: `progress.md` #16 (`copy_between` on the layer), #25 (the semantic MCP tools of the roadmap, with a permission model and dry run). + +## U-20261008-04 · 2026-10-08 · Version directories stay short for long source paths · #done #versioning #windows + +- **What**: `progress.md` #28. `FileVersioner` named the directory that holds a file's snapshots after the whole source path (`C__sep__Users__sep__...`), so a snapshot's path was roughly twice the source's and passed Windows' 260-character limit: five tests of `tests/test_versioning.py` failed with `WinError 206` on a machine with a long temp directory, and a real source file with a long path failed the same way. + - A flattened name of more than 80 characters is now cut to its last 48 characters plus the first 16 hex digits of the SHA-256 of the whole name (after `os.path.normcase`, so another spelling of the same Windows path lands in the same directory). Shorter names are unchanged. + - A directory that already exists under the long name keeps being used, so snapshots taken before this change stay listed and restorable. +- **Tests**: four new cases in `tests/test_versioning.py`: a long source path gets a short directory and still saves, lists, restores and prunes; two long paths with the same tail get different directories; a short path keeps the readable name; a directory created under the long name is reused. The last two skip where the temp path is too long to create the long name at all (this machine). +- **Result / numbers**: the five failing tests pass. `pytest tests/`: 1532 passed, 22 skipped, 0 failed, on Python 3.14.7 on Windows; `ruff check`, `ruff format --check` and `mypy automation_file` pass. +- **Docs**: the class docstring. The READMEs and the manuals do not describe the directory layout, so they are unchanged. +- **Files**: `automation_file/local/versioning.py`, `tests/test_versioning.py`, `progress.md`. +- **Evidence**: the commit that adds this entry. +- **Open items**: none. diff --git a/docs/updates/README.md b/docs/updates/README.md index b33fdbe..d63af2b 100644 --- a/docs/updates/README.md +++ b/docs/updates/README.md @@ -58,6 +58,7 @@ In the same commit: delete the item from `progress.md`, add a `#done` entry here | ID | Date | Title | Tags | Batch | |---|---|---|---|---| +| U-20261008-04 | 2026-10-08 | Version directories stay short for long source paths | #done #versioning #windows | [2026-10](2026-10.md) | | U-20261008-03 | 2026-10-08 | FA_storage_* actions put the storage layer in the registry | #storage #roadmap #actions #mcp | [2026-10](2026-10.md) | | U-20261008-02 | 2026-10-08 | S3 and Azure Blob behind the storage layer | #storage #roadmap #s3 #azure | [2026-10](2026-10.md) | | U-20261008-01 | 2026-10-08 | Universal storage layer: contract, URIs, local and memory | #storage #roadmap #tests | [2026-10](2026-10.md) | @@ -98,5 +99,5 @@ In the same commit: delete the item from `progress.md`, add a `#done` entry here | File | Period | Entries | |---|---|---:| -| [2026-10.md](2026-10.md) | 2026-10 | 14 | +| [2026-10.md](2026-10.md) | 2026-10 | 15 | | [2026-09.md](2026-09.md) | 2026-09 | 21 | diff --git a/progress.md b/progress.md index 8eaf639..9ab261b 100644 --- a/progress.md +++ b/progress.md @@ -40,4 +40,3 @@ Items #10 to #26 are what is left of the 1.0 roadmap (`docs/FILEAUTOMATION-1.0-R ### Found on the way - **#27** The three `usage/cloud.rst` pages (`docs/source/Eng`, `Zh-TW`, `Zh-CN`) show a `FA_cross_copy` action with `src` / `dst` and a `drive://` prefix. Neither exists: the action is `FA_copy_between(source, target)` and `remote/cross_backend.py` has no `drive` scheme. The READMEs were corrected in `bc13101`; these pages were not. Fix them with #16, which rewrites that section anyway. -- **#28** Five tests of `tests/test_versioning.py` fail on a machine with a long temp directory (`FileNotFoundError: [WinError 206]`, the file name or extension is too long): `FileVersioner` names a version directory after the whole source path (`C__sep__Users__sep__...`), so the path roughly doubles and passes Windows' 260-character limit. Seen on `a0dd11f` before any change of this branch; the suite passes where the temp path is shorter (U-20261001-11). A real source file with a long path fails the same way. diff --git a/tests/test_versioning.py b/tests/test_versioning.py index b90308f..cb82656 100644 --- a/tests/test_versioning.py +++ b/tests/test_versioning.py @@ -7,6 +7,7 @@ import pytest from automation_file.exceptions import VersioningException +from automation_file.local import versioning from automation_file.local.versioning import FileVersioner @@ -74,3 +75,71 @@ def test_prune_negative_keep_rejected(tmp_path: Path) -> None: versioner.save_version(src) with pytest.raises(VersioningException): versioner.prune(src, keep=-1) + + +def _long_source(tmp_path: Path, leaf: str = "data.txt") -> Path: + directory = tmp_path / ("d" * 40) / ("e" * 40) + directory.mkdir(parents=True, exist_ok=True) + return directory / leaf + + +def test_a_long_source_path_gets_a_short_directory(tmp_path: Path) -> None: + src = _long_source(tmp_path) + src.write_text("one", encoding="utf-8") + versioner = FileVersioner(tmp_path / "v") + entry = versioner.save_version(src) + bucket = entry.path.parent + assert len(bucket.name) <= 80 + assert bucket.name.rsplit("__", 1)[0].endswith("data.txt") + assert [e.version for e in versioner.list_versions(src)] == [1] + src.write_text("two", encoding="utf-8") + assert versioner.save_version(src).path.parent == bucket + versioner.restore(src, 1) + assert src.read_text(encoding="utf-8") == "one" + assert versioner.prune(src, keep=1) == 1 + + +def test_long_paths_with_the_same_tail_do_not_share_a_directory(tmp_path: Path) -> None: + first = _long_source(tmp_path / "a") + second = _long_source(tmp_path / "b") + first.write_text("first", encoding="utf-8") + second.write_text("second", encoding="utf-8") + versioner = FileVersioner(tmp_path / "v") + one = versioner.save_version(first) + two = versioner.save_version(second) + assert one.path.parent != two.path.parent + assert [e.path.read_text(encoding="utf-8") for e in versioner.list_versions(first)] == ["first"] + assert [e.path.read_text(encoding="utf-8") for e in versioner.list_versions(second)] == [ + "second" + ] + + +def test_a_short_source_path_keeps_the_readable_directory_name( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + monkeypatch.setattr(versioning, "_MAX_BUCKET_NAME", 10_000) + src = tmp_path / "data.txt" + src.write_text("one", encoding="utf-8") + try: + entry = FileVersioner(tmp_path / "v").save_version(src) + except OSError: + pytest.skip("this temp path is too long for the readable directory name") + assert entry.path.parent.name.endswith("__sep__data.txt") + + +def test_a_directory_created_under_the_long_name_keeps_being_used( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + src = tmp_path / "data.txt" + src.write_text("old", encoding="utf-8") + versioner = FileVersioner(tmp_path / "v") + monkeypatch.setattr(versioning, "_MAX_BUCKET_NAME", 10_000) + try: + legacy = versioner.save_version(src) + except OSError: + pytest.skip("this temp path is too long for the readable directory name") + monkeypatch.setattr(versioning, "_MAX_BUCKET_NAME", 1) + src.write_text("new", encoding="utf-8") + again = versioner.save_version(src) + assert again.path.parent == legacy.path.parent + assert [e.version for e in versioner.list_versions(src)] == [1, 2] From 7ae82f95253be2567a48cc980e56b98e4d41c825 Mon Sep 17 00:00:00 2001 From: JeffreyChen Date: Thu, 8 Oct 2026 12:19:15 +0800 Subject: [PATCH 25/59] feat: add streams and directory trees to the storage layer --- CLAUDE.md | 3 +- README.md | 9 +- README.zh-CN.md | 9 +- README.zh-TW.md | 9 +- architecture.md | 7 +- automation_file/__init__.py | 2 + automation_file/storage/__init__.py | 4 + automation_file/storage/actions.py | 19 +++ automation_file/storage/backend.py | 35 ++++- automation_file/storage/file.py | 19 ++- automation_file/storage/local_storage.py | 5 + automation_file/storage/storage.py | 30 ++++ automation_file/storage/streams.py | 82 ++++++++++ automation_file/storage/tree.py | 191 +++++++++++++++++++++++ docs/source/API/storage.rst | 9 ++ docs/source/Eng/usage/storage.rst | 48 +++++- docs/source/Zh-CN/usage/storage.rst | 44 +++++- docs/source/Zh-TW/usage/storage.rst | 44 +++++- docs/updates/2026-10.md | 13 ++ docs/updates/README.md | 3 +- progress.md | 2 +- tests/storage_contract.py | 44 ++++++ tests/test_storage_actions.py | 27 ++++ tests/test_storage_tree.py | 178 +++++++++++++++++++++ 24 files changed, 813 insertions(+), 23 deletions(-) create mode 100644 automation_file/storage/streams.py create mode 100644 automation_file/storage/tree.py create mode 100644 tests/test_storage_tree.py diff --git a/CLAUDE.md b/CLAUDE.md index 3ca06ef..0d853d1 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -27,7 +27,8 @@ automation_file/ ├── storage/ # Universal storage layer: uri (StorageURI), types (FileInfo, Checksum, │ # StorageCapabilities), backend (StorageBackend contract), local_storage, │ # memory_storage, object_storage (ObjectStorage), s3_storage, azure_storage, -│ # resolver (StorageResolver), file (File), storage (Storage), +│ # resolver (StorageResolver), file (File), storage (Storage), streams, +│ # tree (copy_tree, sync_tree), │ # actions (FA_storage_* and register_storage_ops) ├── server/ # tcp_server, http_server, mcp_server (MCP over stdio), web_ui, metrics_server, │ # action_acl (ActionACL), network_guards (ensure_loopback) diff --git a/README.md b/README.md index 55ce393..90c4af8 100644 --- a/README.md +++ b/README.md @@ -48,7 +48,7 @@ facade. - **HTTP server observability** — `GET /healthz` / `GET /readyz` probes, `GET /openapi.json` spec, and `GET /progress` WebSocket stream of live transfer snapshots - **HTMX Web UI** — `start_web_ui()` serves a read-only dashboard (health, progress, registry) that polls HTML fragments; stdlib-only HTTP plus one CDN script with SRI - **MCP (Model Context Protocol) server** — `MCPServer` bridges the registry to any MCP host (Claude Desktop, MCP CLIs) over newline-delimited JSON-RPC 2.0 on stdio; every `FA_*` action becomes an MCP tool with an auto-generated input schema -- **Universal storage layer** — `File` / `Storage` address local and remote storage with one URI syntax (`local:///…`, `s3://…`, `azure://…`, `memory://…`), one `StorageBackend` contract and one error hierarchy; local, S3, Azure Blob and in-memory backends are built in, and a 70-case contract suite checks any backend +- **Universal storage layer** — `File` / `Storage` address local and remote storage with one URI syntax (`local:///…`, `s3://…`, `azure://…`, `memory://…`), one `StorageBackend` contract and one error hierarchy; local, S3, Azure Blob and in-memory backends are built in, and a 77-case contract suite checks any backend - PySide6 GUI (`python -m automation_file ui`) with a tab per backend, the JSON-action runner, and dedicated tabs for Triggers, Scheduler, and live Progress - Rich CLI with one-shot subcommands plus legacy JSON-batch flags - Project scaffolding (`ProjectBuilder`) for executor-based automations @@ -469,6 +469,9 @@ File("sandbox://jobs/42/out.csv").write(b"done") `checksum`, `read_bytes`, `write_bytes`, `copy_from`, `move_from`, identical on every backend. Downloads and local writes are atomic, deleting a directory with entries needs `recursive=True`, and the storage root is never deleted. +- **Streams and trees** — `File.open_read()` / `open_write()` / `iter_chunks()` for content too + large for memory; `Storage.copy_to(target)` copies a directory tree to any backend and + `Storage.sync_to(target, delete=False, checksum=False, dry_run=False)` copies only what changed. - **Errors** — `StorageException` and its subclasses: `StorageNotFoundException`, `StorageAlreadyExistsException`, `StoragePathTypeException`, `StorageNotEmptyException`, `StoragePermissionException`, `StorageTransientException`, `StorageUnavailableException`, @@ -480,13 +483,13 @@ File("sandbox://jobs/42/out.csv").write(b"done") `File("s3://reports/q1.csv").copy_to("azure://backups/q1.csv")` works once both are ready. Google Drive, Dropbox, SFTP, FTP, WebDAV, SMB and fsspec are still used through their own clients and `FA_*` actions; their adapters are not written yet. Write your own by subclassing - `StorageBackend` (or `ObjectStorage` for an object store) and check it with the 70-case + `StorageBackend` (or `ObjectStorage` for an object store) and check it with the 77-case contract suite in `tests/storage_contract.py`. - **Actions** — `FA_storage_exists`, `FA_storage_stat`, `FA_storage_list`, `FA_storage_mkdir`, `FA_storage_upload`, `FA_storage_download`, `FA_storage_delete`, `FA_storage_checksum`, `FA_storage_verify`, `FA_storage_copy`, `FA_storage_move`, `FA_storage_read_text`, - `FA_storage_write_text`, `FA_storage_schemes`. They take URIs + `FA_storage_write_text`, `FA_storage_copy_tree`, `FA_storage_sync`, `FA_storage_schemes`. They take URIs as strings and return JSON-friendly values, so the layer works from action files, the CLI, the TCP and HTTP servers and as MCP tools. Restrict them on a server with `ActionACL`, as for any file action. diff --git a/README.zh-CN.md b/README.zh-CN.md index a5ea642..3c89b8e 100644 --- a/README.zh-CN.md +++ b/README.zh-CN.md @@ -46,7 +46,7 @@ TCP / HTTP 服务器执行的 JSON 驱动动作。内附 PySide6 GUI，每个功 - **HTTP 服务器观测端点** — `GET /healthz` / `GET /readyz` 探针、`GET /openapi.json` 规格，以及 `GET /progress`（通过 WebSocket 推送实时传输快照） - **HTMX Web UI** — `start_web_ui()` 启动只读观测仪表板（health、progress、registry），通过 HTML 片段轮询；仅用标准库 HTTP，搭配一个带 SRI 的 CDN 脚本 - **MCP（Model Context Protocol）服务器** — `MCPServer` 通过 stdio 上的 JSON-RPC 2.0（换行分隔 JSON）将注册表桥接到任意 MCP 主机（Claude Desktop、MCP CLI）；每个 `FA_*` 动作都会自动生成输入 schema 并成为 MCP 工具 -- **通用存储层** — `File` / `Storage` 以同一套 URI 语法（`local:///…`、`s3://…`、`azure://…`、`memory://…`）、同一份 `StorageBackend` 契约与同一组异常层级访问本地与远端存储；内置本地、S3、Azure Blob 与内存后端，并附带 70 个用例的契约测试套件可检查任何后端 +- **通用存储层** — `File` / `Storage` 以同一套 URI 语法（`local:///…`、`s3://…`、`azure://…`、`memory://…`）、同一份 `StorageBackend` 契约与同一组异常层级访问本地与远端存储；内置本地、S3、Azure Blob 与内存后端，并附带 77 个用例的契约测试套件可检查任何后端 - PySide6 GUI（`python -m automation_file ui`）每个后端一个页签，含 JSON 动作执行器，另有 Triggers、Scheduler、实时 Progress 专属页签 - 功能丰富的 CLI，包含一次性子命令与旧式 JSON 批量标志 - 项目脚手架（`ProjectBuilder`）协助构建以 executor 为核心的自动化项目 @@ -466,6 +466,9 @@ File("sandbox://jobs/42/out.csv").write(b"done") `checksum`、`read_bytes`、`write_bytes`、`copy_from`、`move_from`，在每个后端上都相同。 下载与本地写入均为原子操作，删除内有条目的目录需要 `recursive=True`，存储的根目录 永远不会被删除。 +- **流与目录树** — `File.open_read()` / `open_write()` / `iter_chunks()` 用于大到放不进内存的 + 内容；`Storage.copy_to(target)` 把整个目录树复制到任何后端， + `Storage.sync_to(target, delete=False, checksum=False, dry_run=False)` 只复制有变动的部分。 - **异常** — `StorageException` 及其子类：`StorageNotFoundException`、 `StorageAlreadyExistsException`、`StoragePathTypeException`、`StorageNotEmptyException`、 `StoragePermissionException`、`StorageTransientException`、`StorageUnavailableException`、 @@ -477,12 +480,12 @@ File("sandbox://jobs/42/out.csv").write(b"done") `File("s3://reports/q1.csv").copy_to("azure://backups/q1.csv")` 即可运行。 Google Drive、Dropbox、SFTP、FTP、WebDAV、SMB 与 fsspec 目前仍通过各自的客户端与 `FA_*` 动作使用，其适配器尚未完成。你可以继承 `StorageBackend`（对象存储则继承 `ObjectStorage`） - 编写自己的后端，并用 `tests/storage_contract.py` 中 70 个用例的契约测试套件检查。 + 编写自己的后端，并用 `tests/storage_contract.py` 中 77 个用例的契约测试套件检查。 - **动作** — `FA_storage_exists`, `FA_storage_stat`, `FA_storage_list`, `FA_storage_mkdir`, `FA_storage_upload`, `FA_storage_download`, `FA_storage_delete`, `FA_storage_checksum`, `FA_storage_verify`, `FA_storage_copy`, `FA_storage_move`, `FA_storage_read_text`, - `FA_storage_write_text`, `FA_storage_schemes`。它们以字符串 + `FA_storage_write_text`, `FA_storage_copy_tree`, `FA_storage_sync`, `FA_storage_schemes`。它们以字符串 形式接收 URI，并返回可以序列化为 JSON 的值，因此本层可用于动作文件、CLI、TCP 与 HTTP 服务器， 也能作为 MCP 工具。在服务器上请像其他文件动作一样用 `ActionACL` 加以限制。 diff --git a/README.zh-TW.md b/README.zh-TW.md index 97d656e..c4cf67b 100644 --- a/README.zh-TW.md +++ b/README.zh-TW.md @@ -46,7 +46,7 @@ TCP / HTTP 伺服器執行的 JSON 驅動動作。內附 PySide6 GUI，每個功 - **HTTP 伺服器觀測端點** — `GET /healthz` / `GET /readyz` 探針、`GET /openapi.json` 規格、以及 `GET /progress`（以 WebSocket 推送即時傳輸快照） - **HTMX Web UI** — `start_web_ui()` 啟動唯讀觀測儀表板（health、progress、registry），以 HTML 片段輪詢；僅用標準函式庫 HTTP，搭配一支帶 SRI 的 CDN 腳本 - **MCP（Model Context Protocol）伺服器** — `MCPServer` 透過 stdio 上的 JSON-RPC 2.0（行分隔 JSON）將登錄表橋接到任何 MCP 主機（Claude Desktop、MCP CLI）；每個 `FA_*` 動作都會自動生成輸入 schema 並成為 MCP 工具 -- **通用儲存層** — `File` / `Storage` 以同一套 URI 語法（`local:///…`、`s3://…`、`azure://…`、`memory://…`）、同一份 `StorageBackend` 契約與同一組例外階層存取本機與遠端儲存；內建本機、S3、Azure Blob 與記憶體後端，並附 70 個案例的契約測試套件可檢查任何後端 +- **通用儲存層** — `File` / `Storage` 以同一套 URI 語法（`local:///…`、`s3://…`、`azure://…`、`memory://…`）、同一份 `StorageBackend` 契約與同一組例外階層存取本機與遠端儲存；內建本機、S3、Azure Blob 與記憶體後端，並附 77 個案例的契約測試套件可檢查任何後端 - PySide6 GUI（`python -m automation_file ui`）每個後端一個分頁，含 JSON 動作執行器，另有 Triggers、Scheduler、即時 Progress 專屬分頁 - 功能豐富的 CLI，包含一次性子指令與舊式 JSON 批次旗標 - 專案鷹架（`ProjectBuilder`）協助建立以 executor 為核心的自動化專案 @@ -466,6 +466,9 @@ File("sandbox://jobs/42/out.csv").write(b"done") `checksum`、`read_bytes`、`write_bytes`、`copy_from`、`move_from`，在每個後端上都相同。 下載與本機寫入皆為原子操作，刪除內有項目的目錄需要 `recursive=True`，儲存的根目錄 永遠不會被刪除。 +- **串流與目錄樹** — `File.open_read()` / `open_write()` / `iter_chunks()` 用於大到放不進記憶體的 + 內容；`Storage.copy_to(target)` 把整個目錄樹複製到任何後端， + `Storage.sync_to(target, delete=False, checksum=False, dry_run=False)` 只複製有變動的部分。 - **例外** — `StorageException` 及其子類別：`StorageNotFoundException`、 `StorageAlreadyExistsException`、`StoragePathTypeException`、`StorageNotEmptyException`、 `StoragePermissionException`、`StorageTransientException`、`StorageUnavailableException`、 @@ -477,12 +480,12 @@ File("sandbox://jobs/42/out.csv").write(b"done") `File("s3://reports/q1.csv").copy_to("azure://backups/q1.csv")` 即可運作。 Google Drive、Dropbox、SFTP、FTP、WebDAV、SMB 與 fsspec 目前仍透過各自的用戶端與 `FA_*` 動作使用，其轉接器尚未完成。你可以繼承 `StorageBackend`（物件儲存則繼承 `ObjectStorage`） - 撰寫自己的後端，並用 `tests/storage_contract.py` 中 70 個案例的契約測試套件檢查。 + 撰寫自己的後端，並用 `tests/storage_contract.py` 中 77 個案例的契約測試套件檢查。 - **動作** — `FA_storage_exists`, `FA_storage_stat`, `FA_storage_list`, `FA_storage_mkdir`, `FA_storage_upload`, `FA_storage_download`, `FA_storage_delete`, `FA_storage_checksum`, `FA_storage_verify`, `FA_storage_copy`, `FA_storage_move`, `FA_storage_read_text`, - `FA_storage_write_text`, `FA_storage_schemes`。它們以字串 + `FA_storage_write_text`, `FA_storage_copy_tree`, `FA_storage_sync`, `FA_storage_schemes`。它們以字串 形式接收 URI，並回傳可序列化為 JSON 的值，因此本層可用於動作檔、CLI、TCP 與 HTTP 伺服器， 也能作為 MCP 工具。在伺服器上請像其他檔案動作一樣以 `ActionACL` 加以限制。 diff --git a/architecture.md b/architecture.md index d00557a..98fdaf8 100644 --- a/architecture.md +++ b/architecture.md @@ -24,7 +24,7 @@ piece of the 1.0 roadmap (`docs/FILEAUTOMATION-1.0-ROADMAP.md`, PR #107); what i | `automation_file/core/` | Engine, on je_action_core: `action_registry.py` (`ActionRegistry`, a `CommandRegistry`; `build_default_registry`), `action_executor.py` (`ActionExecutor`, an `ActionExecutor` with strict actions, indexed records and the dry-run, validate, substitute and parallel extras; shared `executor`), `callback_executor.py`, `package_loader.py`, `plugins.py`, `dag_executor.py`, `action_queue.py`, `json_store.py`, `substitution.py`. Also cross-cutting helpers: `retry`, `quota`, `rate_limit`, `circuit_breaker`, `file_lock`, `sqlite_lock`, `checksum`, `manifest`, `crypto`, `secrets`, `config`, `config_watcher`, `audit`, `metrics`, `tracing`, `progress`, `fim`, `content_store` | | `automation_file/local/` | Local strategy modules: file, dir, zip, tar and archive ops, sync, diff, text/JSON/data edits, templates, versioning, trash, `shell_ops` (argv-only subprocess), conditional branches. `safe_paths.py` guards against path traversal | | `automation_file/remote/` | `url_validator.py` (SSRF guard), `http_download.py`, `cross_backend.py`, `fsspec_bridge.py`. One subpackage per backend: `google_drive/`, `s3/`, `azure_blob/`, `dropbox_api/`, `sftp/`, `ftp/`, `onedrive/`, `box/`, each with `client.py`, `*_ops.py` and `register__ops`. `smb/` and `webdav/` have a client only | -| `automation_file/storage/` | Universal storage layer. `uri.py` (`StorageURI`, `parse_storage_uri`, `normalize_path`), `types.py` (`FileInfo`, `Checksum`, `StorageCapabilities`), `backend.py` (`StorageBackend`: the public operations are template methods over the `_`-prefixed primitives a backend supplies), `local_storage.py` (`LocalStorage`, confined through `safe_join` when given a root), `memory_storage.py` (`MemoryStorage`), `object_storage.py` (`ObjectStorage`: directories as key prefixes over `_head`, `_scan`, `_put`, `_get`, `_remove`), `s3_storage.py` (`S3Storage`, over `s3_instance` or a given boto3 client), `azure_storage.py` (`AzureStorage`, over `azure_blob_instance` or a given `BlobServiceClient`), `resolver.py` (`StorageResolver`, `default_resolver`: mounts first, then scheme factories), `file.py` (`File`), `storage.py` (`Storage`), `actions.py` (the `FA_storage_*` functions and `register_storage_ops`). At module level it imports only `exceptions`, `logging_config`, `core.checksum` and `local.safe_paths`: no registry, no GUI, no backend SDK. The adapters import their SDK's exceptions and the shared client inside the functions that use them | +| `automation_file/storage/` | Universal storage layer. `uri.py` (`StorageURI`, `parse_storage_uri`, `normalize_path`), `types.py` (`FileInfo`, `Checksum`, `StorageCapabilities`), `backend.py` (`StorageBackend`: the public operations are template methods over the `_`-prefixed primitives a backend supplies), `local_storage.py` (`LocalStorage`, confined through `safe_join` when given a root), `memory_storage.py` (`MemoryStorage`), `object_storage.py` (`ObjectStorage`: directories as key prefixes over `_head`, `_scan`, `_put`, `_get`, `_remove`), `s3_storage.py` (`S3Storage`, over `s3_instance` or a given boto3 client), `azure_storage.py` (`AzureStorage`, over `azure_blob_instance` or a given `BlobServiceClient`), `resolver.py` (`StorageResolver`, `default_resolver`: mounts first, then scheme factories), `file.py` (`File`), `storage.py` (`Storage`), `streams.py` (staged file objects behind `open_read` / `open_write`), `tree.py` (`copy_tree`, `sync_tree`, `TreeResult`), `actions.py` (the `FA_storage_*` functions and `register_storage_ops`). At module level it imports only `exceptions`, `logging_config`, `core.checksum` and `local.safe_paths`: no registry, no GUI, no backend SDK. The adapters import their SDK's exceptions and the shared client inside the functions that use them | | `automation_file/server/` | `tcp_server.py`, `http_server.py`, `mcp_server.py`, `web_ui.py`, `metrics_server.py`, `action_acl.py` (`ActionACL`), `network_guards.py` (`ensure_loopback`) | | `automation_file/client/` | `HTTPActionClient` for the HTTP action server | | `automation_file/trigger/`, `scheduler/`, `notify/` | Watchdog file triggers, cron scheduler, notification sinks. Each registers its own `FA_*` ops | @@ -51,8 +51,9 @@ piece of the 1.0 roadmap (`docs/FILEAUTOMATION-1.0-ROADMAP.md`, PR #107); what i `ObjectStorage`, `S3Storage`, `AzureStorage`, and `StorageException` with its nine subclasses. Storage URIs are `:///`; the built-in schemes are `local` (alias `file`), `memory`, `s3` and `azure` (alias `az`), and text without `://` is a local path. `s3://` and `azure://` use the shared `s3_instance` / `azure_blob_instance`. The API is - provisional until 1.0. Fourteen `FA_storage_*` actions (`exists`, `stat`, `list`, `mkdir`, `upload`, - `download`, `delete`, `checksum`, `verify`, `copy`, `move`, `read_text`, `write_text`, `schemes`) put + provisional until 1.0. Sixteen `FA_storage_*` actions (`exists`, `stat`, `list`, `mkdir`, `upload`, + `download`, `delete`, `checksum`, `verify`, `copy`, `move`, `read_text`, `write_text`, `copy_tree`, + `sync`, `schemes`) put it in the default registry; `register_storage_ops` adds them to another one. - **Action format**: an action is `[name]`, `[name, {kwargs}]` or `[name, [args]]`. A file holds a list of actions or `{"auto_control": [...]}`. diff --git a/automation_file/__init__.py b/automation_file/__init__.py index c3cf927..0f4b114 100644 --- a/automation_file/__init__.py +++ b/automation_file/__init__.py @@ -279,6 +279,7 @@ StorageCapabilities, StorageResolver, StorageURI, + TreeResult, parse_storage_uri, register_storage_ops, ) @@ -487,6 +488,7 @@ def __getattr__(name: str) -> Any: "StorageURI", "parse_storage_uri", "register_storage_ops", + "TreeResult", "FileInfo", "Checksum", "StorageCapabilities", diff --git a/automation_file/storage/__init__.py b/automation_file/storage/__init__.py index 661f710..3faf32f 100644 --- a/automation_file/storage/__init__.py +++ b/automation_file/storage/__init__.py @@ -30,6 +30,7 @@ ) from automation_file.storage.s3_storage import S3Storage from automation_file.storage.storage import Storage +from automation_file.storage.tree import TreeResult, copy_tree, sync_tree from automation_file.storage.types import Checksum, FileInfo, StorageCapabilities from automation_file.storage.uri import ( StorageURI, @@ -54,8 +55,10 @@ "StorageCapabilities", "StorageResolver", "StorageURI", + "TreeResult", "URILike", "clear_memory_stores", + "copy_tree", "default_resolver", "local_path_to_uri", "memory_store", @@ -63,4 +66,5 @@ "parse_storage_uri", "register_default_schemes", "register_storage_ops", + "sync_tree", ] diff --git a/automation_file/storage/actions.py b/automation_file/storage/actions.py index 2e97670..318d6b0 100644 --- a/automation_file/storage/actions.py +++ b/automation_file/storage/actions.py @@ -132,6 +132,23 @@ def storage_write_text( return _described(info, str(target)) +def storage_copy_tree(source: str, target: str, overwrite: bool = True) -> dict[str, Any]: + """Copy every file below the directory ``source`` to ``target``; returns a summary.""" + return Storage(source).copy_to(target, overwrite=overwrite).to_dict() + + +def storage_sync( + source: str, + target: str, + delete: bool = False, + checksum: bool = False, + dry_run: bool = False, +) -> dict[str, Any]: + """Mirror the directory ``source`` into ``target``, copying only what changed.""" + result = Storage(source).sync_to(target, delete=delete, checksum=checksum, dry_run=dry_run) + return result.to_dict() + + def storage_schemes() -> list[str]: """Return the URI schemes a backend is registered or mounted for.""" return Storage.schemes() @@ -153,6 +170,8 @@ def storage_commands() -> dict[str, Callable[..., Any]]: "FA_storage_move": storage_move, "FA_storage_read_text": storage_read_text, "FA_storage_write_text": storage_write_text, + "FA_storage_copy_tree": storage_copy_tree, + "FA_storage_sync": storage_sync, "FA_storage_schemes": storage_schemes, } diff --git a/automation_file/storage/backend.py b/automation_file/storage/backend.py index f4b5be1..1eb771a 100644 --- a/automation_file/storage/backend.py +++ b/automation_file/storage/backend.py @@ -19,13 +19,14 @@ import hashlib import mimetypes import os +import shutil import tempfile import uuid from abc import ABC, abstractmethod from collections.abc import Iterable from pathlib import Path, PurePosixPath from types import TracebackType -from typing import TypeVar +from typing import BinaryIO, TypeVar from automation_file.core.checksum import file_checksum from automation_file.exceptions import ( @@ -36,6 +37,7 @@ StoragePathTypeException, StorageUnsupportedException, ) +from automation_file.storage.streams import StagedReader, StagedWriter, new_scratch_file from automation_file.storage.types import Checksum, FileInfo, StorageCapabilities from automation_file.storage.uri import normalize_path @@ -180,6 +182,21 @@ def _read_bytes(self, path: str) -> bytes: self._download(path, staged) return staged.read_bytes() + def _open_read(self, path: str) -> BinaryIO: + """Return a binary file object over the file ``path``. + + The default serves a staged copy that is removed when the object is closed. + """ + scratch, staged = new_scratch_file() + reader: BinaryIO | None = None + try: + self._download(path, staged) + reader = StagedReader(staged, scratch) + return reader + finally: + if reader is None: + shutil.rmtree(scratch, ignore_errors=True) + def _delete_directory(self, path: str, recursive: bool) -> None: """Remove the directory ``path``, which is not the root.""" if not recursive: @@ -318,6 +335,22 @@ def read_bytes(self, path: str) -> bytes: """Return the whole content of the file ``path``.""" return self._read_bytes(self._existing_file(path)) + def open_read(self, path: str) -> BinaryIO: + """Return a binary file object over the file ``path``; close it when done. + + Use it for a file too large to hold in memory with :meth:`read_bytes`. + """ + return self._open_read(self._existing_file(path)) + + def open_write(self, path: str, *, overwrite: bool = True) -> BinaryIO: + """Return a binary file object whose content becomes the file ``path`` on close. + + Nothing is stored when a ``with`` block is left through an exception. The + ``overwrite`` check runs now and again when the content is stored. + """ + clean = self._writable_file(path, overwrite) + return StagedWriter(lambda staged: self.upload(staged, clean, overwrite=overwrite)) + def write_bytes(self, path: str, data: bytes, *, overwrite: bool = True) -> FileInfo: """Store ``data`` as the file ``path`` and return its ``FileInfo``.""" with tempfile.TemporaryDirectory() as scratch: diff --git a/automation_file/storage/file.py b/automation_file/storage/file.py index fbd3c50..0280d7f 100644 --- a/automation_file/storage/file.py +++ b/automation_file/storage/file.py @@ -18,10 +18,11 @@ from __future__ import annotations import os -from collections.abc import Mapping +from collections.abc import Iterator, Mapping from dataclasses import replace from datetime import datetime from pathlib import Path +from typing import BinaryIO from automation_file.exceptions import StorageNotFoundException, StoragePathTypeException from automation_file.storage.backend import DEFAULT_CHECKSUM_ALGORITHM, StorageBackend @@ -30,6 +31,7 @@ from automation_file.storage.uri import StorageURI, URILike, parse_storage_uri _DEFAULT_ENCODING = "utf-8" +_DEFAULT_CHUNK = 1024 * 1024 def _expected_algorithm(expected: str | Checksum, fallback: str) -> str: @@ -111,6 +113,21 @@ def write( backend, path = self._locate() return replace(backend.write_bytes(path, payload, overwrite=overwrite), path=self._uri.path) + def open_read(self) -> BinaryIO: + """Return a binary file object over the content; close it when done.""" + backend, path = self._locate() + return backend.open_read(path) + + def open_write(self, *, overwrite: bool = True) -> BinaryIO: + """Return a binary file object whose content is stored when it is closed.""" + backend, path = self._locate() + return backend.open_write(path, overwrite=overwrite) + + def iter_chunks(self, chunk_size: int = _DEFAULT_CHUNK) -> Iterator[bytes]: + """Yield the content in blocks of at most ``chunk_size`` bytes.""" + with self.open_read() as stream: + yield from iter(lambda: stream.read(chunk_size), b"") + def upload_from( self, local_path: str | os.PathLike[str], *, overwrite: bool = True ) -> FileInfo: diff --git a/automation_file/storage/local_storage.py b/automation_file/storage/local_storage.py index 5f3e680..bee045c 100644 --- a/automation_file/storage/local_storage.py +++ b/automation_file/storage/local_storage.py @@ -25,6 +25,7 @@ from collections.abc import Iterable, Iterator from datetime import datetime, timezone from pathlib import Path +from typing import BinaryIO from automation_file.core.checksum import file_checksum from automation_file.exceptions import StorageException, StoragePermissionException @@ -242,6 +243,10 @@ def _checksum(self, path: str, algorithm: str) -> str: with _os_errors(self.uri_for(path)): return file_checksum(self.local_path(path), algorithm) + def _open_read(self, path: str) -> BinaryIO: + with _os_errors(self.uri_for(path)): + return self.local_path(path).open("rb") + def _read_bytes(self, path: str) -> bytes: with _os_errors(self.uri_for(path)): return self.local_path(path).read_bytes() diff --git a/automation_file/storage/storage.py b/automation_file/storage/storage.py index 37f1921..3c75f28 100644 --- a/automation_file/storage/storage.py +++ b/automation_file/storage/storage.py @@ -23,6 +23,7 @@ import os from dataclasses import replace from pathlib import Path +from typing import TYPE_CHECKING from automation_file.storage.backend import DEFAULT_CHECKSUM_ALGORITHM, StorageBackend from automation_file.storage.file import File @@ -30,6 +31,9 @@ from automation_file.storage.types import Checksum, FileInfo, StorageCapabilities from automation_file.storage.uri import StorageURI, URILike, normalize_path, parse_storage_uri +if TYPE_CHECKING: + from automation_file.storage.tree import TreeResult + def _rebased(info: FileInfo, backend_base: str, asked: str) -> FileInfo: """Swap the backend's own prefix of ``info.path`` for the path the caller asked with.""" @@ -135,6 +139,32 @@ def checksum(self, path: str, algorithm: str = DEFAULT_CHECKSUM_ALGORITHM) -> Ch backend, target = self._locate(path) return backend.checksum(target, algorithm) + def copy_to(self, target: URILike | Storage, *, overwrite: bool = True) -> TreeResult: + """Copy every file below this storage to ``target``, in any backend.""" + from automation_file.storage.tree import copy_tree + + return copy_tree(self, self._as_storage(target), overwrite=overwrite) + + def sync_to( + self, + target: URILike | Storage, + *, + delete: bool = False, + checksum: bool = False, + dry_run: bool = False, + ) -> TreeResult: + """Make ``target`` hold what this storage holds, copying only what changed.""" + from automation_file.storage.tree import sync_tree + + return sync_tree( + self, self._as_storage(target), delete=delete, checksum=checksum, dry_run=dry_run + ) + + def _as_storage(self, target: URILike | Storage) -> Storage: + if isinstance(target, Storage): + return target + return Storage(target, resolver=self._resolver) + def _locate(self, path: str) -> tuple[StorageBackend, str]: return self._resolver.resolve(self._uri.joinpath(path)) diff --git a/automation_file/storage/streams.py b/automation_file/storage/streams.py new file mode 100644 index 0000000..4863c0c --- /dev/null +++ b/automation_file/storage/streams.py @@ -0,0 +1,82 @@ +"""File objects over a staged local copy. + +A backend that cannot stream natively still offers ``open_read`` and +``open_write`` through these two classes: the reader serves a downloaded copy and +removes it when closed; the writer collects what is written and hands the +finished file to the backend when closed. +""" + +from __future__ import annotations + +import io +import shutil +import tempfile +from collections.abc import Callable +from pathlib import Path +from types import TracebackType + +_STAGED_NAME = "staged" + + +def new_scratch_file() -> tuple[Path, Path]: + """Create a private scratch directory and return ``(directory, file path inside it)``.""" + scratch = Path(tempfile.mkdtemp()) + return scratch, scratch / _STAGED_NAME + + +class StagedReader(io.BufferedReader): + """Read a staged copy; closing it removes the copy.""" + + def __init__(self, staged: Path, scratch: Path) -> None: + super().__init__(io.FileIO(staged, "rb")) + self._scratch = scratch + + def close(self) -> None: + try: + super().close() + finally: + shutil.rmtree(self._scratch, ignore_errors=True) + + +class StagedWriter(io.BufferedWriter): + """Collect written bytes; closing it stores them through ``commit``. + + Leaving a ``with`` block through an exception, calling :meth:`discard`, or + dropping the object without closing it stores nothing. + """ + + def __init__(self, commit: Callable[[Path], object]) -> None: + self._scratch, self._staged = new_scratch_file() + super().__init__(io.FileIO(self._staged, "wb")) + self._commit: Callable[[Path], object] | None = commit + + def discard(self) -> None: + """Close without storing anything.""" + self._commit = None + self.close() + + def close(self) -> None: + if self.closed: + return + commit, self._commit = self._commit, None + try: + super().close() + if commit is not None: + commit(self._staged) + finally: + shutil.rmtree(self._scratch, ignore_errors=True) + + def __exit__( + self, + exc_type: type[BaseException] | None, + exc: BaseException | None, + tb: TracebackType | None, + ) -> None: + if exc_type is not None: + self._commit = None + super().__exit__(exc_type, exc, tb) + + def __del__(self) -> None: + # A writer that was never closed must not store a half-written file. + self._commit = None + self.close() diff --git a/automation_file/storage/tree.py b/automation_file/storage/tree.py new file mode 100644 index 0000000..1e37c6f --- /dev/null +++ b/automation_file/storage/tree.py @@ -0,0 +1,191 @@ +"""Directory trees across backends: copy one, or mirror one into another. + +Both functions list the source once, then work file by file through +``File.copy_to``, so the two sides may be any two backends. A file that fails is +recorded in ``TreeResult.errors`` and the others still run, like ``sync_dir``. + +``sync_tree`` copies a file when the target lacks it, when the sizes differ, or +when the source is newer; ``checksum=True`` compares SHA-256 digests instead of +times, which costs a read of both sides. ``delete=True`` also removes what the +source does not have. ``dry_run=True`` reports what would happen and changes +nothing. +""" + +from __future__ import annotations + +from dataclasses import dataclass, field +from typing import Any + +from automation_file.exceptions import StorageException, StorageNotFoundException +from automation_file.logging_config import file_automation_logger +from automation_file.storage.storage import Storage +from automation_file.storage.types import FileInfo + + +@dataclass +class TreeResult: + """What a tree copy or sync did, as paths relative to the two roots.""" + + copied: list[str] = field(default_factory=list) + skipped: list[str] = field(default_factory=list) + deleted: list[str] = field(default_factory=list) + errors: dict[str, str] = field(default_factory=dict) + dry_run: bool = False + + @property + def ok(self) -> bool: + """True when no file failed.""" + return not self.errors + + def to_dict(self) -> dict[str, Any]: + return { + "copied": list(self.copied), + "skipped": list(self.skipped), + "deleted": list(self.deleted), + "errors": dict(self.errors), + "dry_run": self.dry_run, + } + + +def _entries(storage: Storage, *, missing_ok: bool) -> tuple[dict[str, FileInfo], list[str]]: + """Return the files (by relative path) and the directories below ``storage``.""" + try: + listing = storage.list_dir(recursive=True) + except StorageNotFoundException: + if missing_ok: + return {}, [] + raise + files = {info.path: info for info in listing if not info.is_dir} + return files, [info.path for info in listing if info.is_dir] + + +def _checked_roots(source: Storage, target: Storage) -> None: + if source.uri == target.uri: + raise StorageException(f"{source} is both the source and the target") + + +def _failure(error: StorageException) -> str: + return f"{type(error).__name__}: {error}" + + +def _copy_file(source: Storage, target: Storage, path: str, result: TreeResult) -> None: + if result.dry_run: + result.copied.append(path) + return + try: + source.file(path).copy_to(target.file(path)) + except StorageException as error: + result.errors[path] = _failure(error) + else: + result.copied.append(path) + + +def _make_directories(target: Storage, directories: list[str], result: TreeResult) -> None: + if result.dry_run: + return + for directory in directories: + try: + target.mkdir(directory) + except StorageException as error: + result.errors[directory] = _failure(error) + + +def _newer(ours: FileInfo, theirs: FileInfo) -> bool: + if ours.modified_at is None or theirs.modified_at is None: + return False + return ours.modified_at > theirs.modified_at + + +def _sync_file( + source: Storage, target: Storage, pair: tuple[FileInfo, FileInfo | None], checksum: bool +) -> bool: + """Say whether the target lacks the source file or holds another version of it.""" + ours, theirs = pair + if theirs is None or ours.size != theirs.size: + return True + if checksum: + return not source.checksum(ours.path).matches(target.checksum(ours.path)) + return _newer(ours, theirs) + + +def _delete_extras( + target: Storage, extra_files: list[str], extra_directories: list[str], result: TreeResult +) -> None: + # Deepest directories first, after the files inside them are gone. + ordered = [*extra_files, *sorted(extra_directories, key=lambda path: -path.count("/"))] + for path in ordered: + if result.dry_run: + result.deleted.append(path) + continue + try: + target.delete(path, missing_ok=True) + except StorageException as error: + result.errors[path] = _failure(error) + else: + result.deleted.append(path) + + +def _log(action: str, source: Storage, target: Storage, result: TreeResult) -> None: + file_automation_logger.info( + "%s %s -> %s: copied=%d skipped=%d deleted=%d errors=%d (dry_run=%s)", + action, + source, + target, + len(result.copied), + len(result.skipped), + len(result.deleted), + len(result.errors), + result.dry_run, + ) + + +def copy_tree(source: Storage, target: Storage, *, overwrite: bool = True) -> TreeResult: + """Copy every file below ``source`` to the same relative path below ``target``. + + With ``overwrite=False`` a file the target already has is skipped. Empty + directories are created where the target backend has real directories. + """ + _checked_roots(source, target) + files, directories = _entries(source, missing_ok=False) + existing, _ = _entries(target, missing_ok=True) + result = TreeResult() + _make_directories(target, directories, result) + for path in sorted(files): + if not overwrite and path in existing: + result.skipped.append(path) + else: + _copy_file(source, target, path, result) + _log("copy_tree", source, target, result) + return result + + +def sync_tree( + source: Storage, + target: Storage, + *, + delete: bool = False, + checksum: bool = False, + dry_run: bool = False, +) -> TreeResult: + """Make ``target`` hold what ``source`` holds, copying only what changed.""" + _checked_roots(source, target) + files, directories = _entries(source, missing_ok=False) + existing, existing_directories = _entries(target, missing_ok=True) + result = TreeResult(dry_run=dry_run) + _make_directories(target, directories, result) + for path in sorted(files): + try: + changed = _sync_file(source, target, (files[path], existing.get(path)), checksum) + except StorageException as error: + result.errors[path] = _failure(error) + continue + if changed: + _copy_file(source, target, path, result) + else: + result.skipped.append(path) + if delete: + extra_files = sorted(set(existing) - set(files)) + extra_directories = sorted(set(existing_directories) - set(directories)) + _delete_extras(target, extra_files, extra_directories, result) + _log("sync_tree", source, target, result) + return result diff --git a/docs/source/API/storage.rst b/docs/source/API/storage.rst index 3946318..ee39343 100644 --- a/docs/source/API/storage.rst +++ b/docs/source/API/storage.rst @@ -20,6 +20,15 @@ Actions .. automodule:: automation_file.storage.actions :members: +Streams and directory trees +--------------------------- + +.. automodule:: automation_file.storage.tree + :members: + +.. automodule:: automation_file.storage.streams + :members: + Storage URIs ------------ diff --git a/docs/source/Eng/usage/storage.rst b/docs/source/Eng/usage/storage.rst index 3e3f6df..85f393a 100644 --- a/docs/source/Eng/usage/storage.rst +++ b/docs/source/Eng/usage/storage.rst @@ -130,6 +130,10 @@ Every backend has the same methods. ``File`` and ``Storage`` forward to them. compatibility, not for security). * - ``read_bytes(path)`` / ``write_bytes(path, data)`` - Whole-file content. + * - ``open_read(path)`` / ``open_write(path, overwrite=True)`` + - Binary file objects, for content too large to hold in memory. What is + written is stored when the object is closed; leaving a ``with`` block + through an exception stores nothing. * - ``copy_from(source, source_path, path)`` / ``move_from(…)`` - Transfer from any backend, this one included. Native when the two backends can do it between themselves (a local rename), through a local staging @@ -198,6 +202,12 @@ strings and returns JSON-friendly values. * - ``FA_storage_write_text`` - ``uri, text, overwrite=True, encoding="utf-8"`` - The file information + * - ``FA_storage_copy_tree`` + - ``source, target, overwrite=True`` + - A summary: ``copied``, ``skipped``, ``deleted``, ``errors``, ``dry_run`` + * - ``FA_storage_sync`` + - ``source, target, delete=False, checksum=False, dry_run=False`` + - A summary: ``copied``, ``skipped``, ``deleted``, ``errors``, ``dry_run`` * - ``FA_storage_schemes`` - — - The registered schemes @@ -308,6 +318,40 @@ raises ``StorageUnavailableException``. azure_blob_instance.later_init(connection_string=connection_string) File("s3://reports/2026/q1.csv").copy_to("azure://backups/2026/q1.csv") +Streams and directory trees +--------------------------- + +``File.open_read()`` and ``File.open_write()`` return binary file objects, and +``File.iter_chunks()`` yields the content block by block. The local backend +reads in place; the others serve a staged local copy that is removed when the +object is closed, so memory stays bounded either way. + +``Storage.copy_to(target)`` copies every file below a directory to the same +relative path below ``target``, in any backend. ``Storage.sync_to(target)`` +copies only what changed: a file the target lacks, one whose size differs, or +one the source holds in a newer version. ``checksum=True`` compares SHA-256 +digests instead of times, at the cost of reading both sides. ``delete=True`` +also removes what the source does not have, and ``dry_run=True`` reports what +would happen without changing anything. Both return a ``TreeResult`` with +``copied``, ``skipped``, ``deleted`` and ``errors``; a file that fails is +recorded in ``errors`` and the others still run. + +.. code-block:: python + + from automation_file import File, Storage + + with File("s3://logs/2026/big.log").open_read() as stream: + for line in stream: + ... + + with File("local:///exports/report.csv").open_write() as stream: + stream.write(b"region,total\n") + + reports = Storage("s3://reports/2026") + reports.copy_to("local:///backup/2026") # every file, any backend + result = reports.sync_to("azure://backups/2026", delete=True, dry_run=True) + result.copied, result.skipped, result.deleted, result.errors + Mounting and registering backends --------------------------------- @@ -371,9 +415,9 @@ implement ``_head``, ``_scan``, ``_put``, ``_get`` and ``_remove``. It supplies the directory behaviour described under `Built-in backends`_, and is what ``S3Storage`` and ``AzureStorage`` are built on. -Check it with the contract suite. ``tests/storage_contract.py`` holds 70 cases — +Check it with the contract suite. ``tests/storage_contract.py`` holds 77 cases — nested directories, empty and large files, Unicode paths, binary data, overwrite -and missing-path behaviour, path normalisation, copy and move — and reads +and missing-path behaviour, path normalisation, streams, copy and move — and reads ``capabilities`` where backends legitimately differ: .. code-block:: python diff --git a/docs/source/Zh-CN/usage/storage.rst b/docs/source/Zh-CN/usage/storage.rst index 8c5392d..d2775b3 100644 --- a/docs/source/Zh-CN/usage/storage.rst +++ b/docs/source/Zh-CN/usage/storage.rst @@ -122,6 +122,9 @@ API；:class:`~automation_file.StorageBackend` 则是后端需要实现的契约 安全目的）。 * - ``read_bytes(path)`` / ``write_bytes(path, data)`` - 读写整个文件的内容。 + * - ``open_read(path)`` / ``open_write(path, overwrite=True)`` + - 二进制文件对象，用于大到无法整个放进内存的内容。写入的内容在对象关闭时 + 才会存入；如果因异常离开 ``with`` 块，则不会存入任何东西。 * - ``copy_from(source, source_path, path)`` / ``move_from(…)`` - 从任何后端（包含自己）传输。两个后端之间能直接完成时走原生方式（例如 本地重命名），否则经由本地暂存文件。 @@ -187,6 +190,12 @@ API；:class:`~automation_file.StorageBackend` 则是后端需要实现的契约 * - ``FA_storage_write_text`` - ``uri, text, overwrite=True, encoding="utf-8"`` - 文件信息 + * - ``FA_storage_copy_tree`` + - ``source, target, overwrite=True`` + - 摘要：``copied``、``skipped``、``deleted``、``errors``、``dry_run`` + * - ``FA_storage_sync`` + - ``source, target, delete=False, checksum=False, dry_run=False`` + - 摘要：``copied``、``skipped``、``deleted``、``errors``、``dry_run`` * - ``FA_storage_schemes`` - — - 已注册的 scheme @@ -291,6 +300,37 @@ S3 与 Azure Blob 都是对象存储。目录只在其下还有 key 时才存在 azure_blob_instance.later_init(connection_string=connection_string) File("s3://reports/2026/q1.csv").copy_to("azure://backups/2026/q1.csv") +流与目录树 +---------- + +``File.open_read()`` 与 ``File.open_write()`` 返回二进制文件对象， +``File.iter_chunks()`` 则逐块产出内容。本地后端直接就地读取；其他后端提供一份 +暂存的本地副本，对象关闭时即移除，因此两种情况下内存用量都有上限。 + +``Storage.copy_to(target)`` 会把目录下的每个文件复制到 ``target`` 之下相同的相对 +路径，后端不限。``Storage.sync_to(target)`` 只复制有变动的部分：目标缺少的文件、 +大小不同的文件，或来源版本较新的文件。``checksum=True`` 改为比较 SHA-256 摘要而非 +时间，代价是两边都要读取一次。``delete=True`` 还会移除来源没有的条目， +``dry_run=True`` 只报告会发生什么而不做任何更改。两者都返回 ``TreeResult``，包含 +``copied``、``skipped``、``deleted`` 与 ``errors``；失败的文件会记在 ``errors`` +中，其余文件照常处理。 + +.. code-block:: python + + from automation_file import File, Storage + + with File("s3://logs/2026/big.log").open_read() as stream: + for line in stream: + ... + + with File("local:///exports/report.csv").open_write() as stream: + stream.write(b"region,total\n") + + reports = Storage("s3://reports/2026") + reports.copy_to("local:///backup/2026") # every file, any backend + result = reports.sync_to("azure://backups/2026", delete=True, dry_run=True) + result.copied, result.skipped, result.deleted, result.errors + 挂载与注册后端 -------------- @@ -350,9 +390,9 @@ scheme 或 authority，并且只接受其下的 URI。 ``_head``、``_scan``、``_put``、``_get`` 与 ``_remove``。它提供 `内置后端`_ 一节 所述的目录行为，``S3Storage`` 与 ``AzureStorage`` 都建立在它之上。 -请用契约测试套件检查。``tests/storage_contract.py`` 包含 70 个用例——嵌套目录、 +请用契约测试套件检查。``tests/storage_contract.py`` 包含 77 个用例——嵌套目录、 空文件与大文件、Unicode 路径、二进制数据、覆盖与路径不存在时的行为、路径规范化、 -复制与移动——并在后端确实有差异之处读取 ``capabilities``： +流、复制与移动——并在后端确实有差异之处读取 ``capabilities``： .. code-block:: python diff --git a/docs/source/Zh-TW/usage/storage.rst b/docs/source/Zh-TW/usage/storage.rst index 39e99a6..bd7f5df 100644 --- a/docs/source/Zh-TW/usage/storage.rst +++ b/docs/source/Zh-TW/usage/storage.rst @@ -122,6 +122,9 @@ API；:class:`~automation_file.StorageBackend` 則是後端要實作的契約。 安全用途）。 * - ``read_bytes(path)`` / ``write_bytes(path, data)`` - 讀寫整個檔案的內容。 + * - ``open_read(path)`` / ``open_write(path, overwrite=True)`` + - 二進位檔案物件，用於大到無法整個放進記憶體的內容。寫入的內容在物件關閉時 + 才會存入；若因例外離開 ``with`` 區塊，則不會存入任何東西。 * - ``copy_from(source, source_path, path)`` / ``move_from(…)`` - 從任何後端（包含自己）傳輸。兩個後端之間能直接完成時走原生方式（例如 本機重新命名），否則經由本機暫存檔。 @@ -187,6 +190,12 @@ API；:class:`~automation_file.StorageBackend` 則是後端要實作的契約。 * - ``FA_storage_write_text`` - ``uri, text, overwrite=True, encoding="utf-8"`` - 檔案資訊 + * - ``FA_storage_copy_tree`` + - ``source, target, overwrite=True`` + - 摘要：``copied``、``skipped``、``deleted``、``errors``、``dry_run`` + * - ``FA_storage_sync`` + - ``source, target, delete=False, checksum=False, dry_run=False`` + - 摘要：``copied``、``skipped``、``deleted``、``errors``、``dry_run`` * - ``FA_storage_schemes`` - — - 已註冊的 scheme @@ -291,6 +300,37 @@ S3 與 Azure Blob 都是物件儲存。目錄只在其下還有 key 時才存在 azure_blob_instance.later_init(connection_string=connection_string) File("s3://reports/2026/q1.csv").copy_to("azure://backups/2026/q1.csv") +串流與目錄樹 +------------ + +``File.open_read()`` 與 ``File.open_write()`` 回傳二進位檔案物件， +``File.iter_chunks()`` 則逐塊產出內容。本機後端直接就地讀取；其他後端提供一份 +暫存的本機副本，物件關閉時即移除，因此兩種情況下記憶體用量都有上限。 + +``Storage.copy_to(target)`` 會把目錄下的每個檔案複製到 ``target`` 之下相同的相對 +路徑，後端不限。``Storage.sync_to(target)`` 只複製有變動的部分：目標缺少的檔案、 +大小不同的檔案，或來源版本較新的檔案。``checksum=True`` 改為比對 SHA-256 摘要而非 +時間，代價是兩邊都要讀取一次。``delete=True`` 還會移除來源沒有的項目， +``dry_run=True`` 只回報會發生什麼而不做任何變更。兩者都回傳 ``TreeResult``，內含 +``copied``、``skipped``、``deleted`` 與 ``errors``；失敗的檔案會記在 ``errors`` +中，其餘檔案照常處理。 + +.. code-block:: python + + from automation_file import File, Storage + + with File("s3://logs/2026/big.log").open_read() as stream: + for line in stream: + ... + + with File("local:///exports/report.csv").open_write() as stream: + stream.write(b"region,total\n") + + reports = Storage("s3://reports/2026") + reports.copy_to("local:///backup/2026") # every file, any backend + result = reports.sync_to("azure://backups/2026", delete=True, dry_run=True) + result.copied, result.skipped, result.deleted, result.errors + 掛載與註冊後端 -------------- @@ -350,9 +390,9 @@ scheme 或 authority，並且只接受其下的 URI。 ``_head``、``_scan``、``_put``、``_get`` 與 ``_remove``。它提供 `內建後端`_ 一節 所述的目錄行為，``S3Storage`` 與 ``AzureStorage`` 都建立在它之上。 -請用契約測試套件檢查。``tests/storage_contract.py`` 包含 70 個案例——巢狀目錄、 +請用契約測試套件檢查。``tests/storage_contract.py`` 包含 77 個案例——巢狀目錄、 空檔與大檔、Unicode 路徑、二進位資料、覆寫與路徑不存在時的行為、路徑正規化、 -複製與搬移——並在後端確實有差異之處讀取 ``capabilities``： +串流、複製與搬移——並在後端確實有差異之處讀取 ``capabilities``： .. code-block:: python diff --git a/docs/updates/2026-10.md b/docs/updates/2026-10.md index eb1d00b..8eeffbe 100644 --- a/docs/updates/2026-10.md +++ b/docs/updates/2026-10.md @@ -263,3 +263,16 @@ Index and query commands: [README.md](README.md). New entries go at the end. - **Files**: `automation_file/local/versioning.py`, `tests/test_versioning.py`, `progress.md`. - **Evidence**: the commit that adds this entry. - **Open items**: none. + +## U-20261008-05 · 2026-10-08 · Streams and directory trees in the storage layer · #storage #roadmap #streams + +- **What**: most of `progress.md` #17; the native streams and checksums of the remote backends stay open there. + - `StorageBackend.open_read(path)` and `open_write(path, overwrite=True)` return binary file objects (`storage/streams.py`). `LocalStorage` reads in place; every other backend serves a staged local copy that is removed on close, so memory stays bounded. What is written is stored when the object is closed; a `with` block left through an exception, `discard()`, or a writer dropped without closing stores nothing. The names are `open_read` / `open_write` because `open` would shadow the builtin. + - `File.open_read()`, `File.open_write()`, `File.iter_chunks(chunk_size)`. + - `storage/tree.py`: `copy_tree(source, target, overwrite=True)` and `sync_tree(source, target, delete=False, checksum=False, dry_run=False)` between any two backends, also as `Storage.copy_to` / `Storage.sync_to`. A sync copies a file the target lacks, one whose size differs, or one the source holds in a newer version; `checksum=True` compares SHA-256 digests instead of times. `delete=True` removes extra files and then the directories they leave empty. The result is a `TreeResult` (`copied`, `skipped`, `deleted`, `errors`, `dry_run`); a file that fails is recorded and the others still run. Copying a tree onto itself is refused. + - Actions `FA_storage_copy_tree` and `FA_storage_sync` (sixteen `FA_storage_*` actions in all). +- **Tests**: seven stream cases join the contract (77 per backend, run for six contract classes), `tests/test_storage_tree.py` (10 cases: other backend, overwrite off, missing or non-directory source, same tree, change detection by size, time and checksum, delete, dry run, a failing file) and one action case. +- **Result / numbers**: `ruff check`, `ruff format --check` pass; `mypy automation_file` finds no issues in 175 files; `pytest tests/`: 1532 passed, 22 skipped, 0 failed, on Python 3.14.7 on Windows. +- **Docs**: a "Streams and directory trees" section, two operation rows and two action rows in the three `usage/storage.rst` pages, `docs/source/API/storage.rst`, a bullet in the three READMEs (which now say 77 cases), `architecture.md` §2 and §3, `CLAUDE.md` (package map). +- **Files**: `automation_file/storage/streams.py`, `tree.py`, `backend.py`, `local_storage.py`, `file.py`, `storage.py`, `actions.py`, `__init__.py`, `automation_file/__init__.py`, `tests/storage_contract.py`, `tests/test_storage_tree.py`, `tests/test_storage_actions.py`, the documentation above, `progress.md`. +- **Open items**: `progress.md` #17 (native streams and checksums for the remote backends). diff --git a/docs/updates/README.md b/docs/updates/README.md index d63af2b..1ddffb5 100644 --- a/docs/updates/README.md +++ b/docs/updates/README.md @@ -58,6 +58,7 @@ In the same commit: delete the item from `progress.md`, add a `#done` entry here | ID | Date | Title | Tags | Batch | |---|---|---|---|---| +| U-20261008-05 | 2026-10-08 | Streams and directory trees in the storage layer | #storage #roadmap #streams | [2026-10](2026-10.md) | | U-20261008-04 | 2026-10-08 | Version directories stay short for long source paths | #done #versioning #windows | [2026-10](2026-10.md) | | U-20261008-03 | 2026-10-08 | FA_storage_* actions put the storage layer in the registry | #storage #roadmap #actions #mcp | [2026-10](2026-10.md) | | U-20261008-02 | 2026-10-08 | S3 and Azure Blob behind the storage layer | #storage #roadmap #s3 #azure | [2026-10](2026-10.md) | @@ -99,5 +100,5 @@ In the same commit: delete the item from `progress.md`, add a `#done` entry here | File | Period | Entries | |---|---|---:| -| [2026-10.md](2026-10.md) | 2026-10 | 15 | +| [2026-10.md](2026-10.md) | 2026-10 | 16 | | [2026-09.md](2026-09.md) | 2026-09 | 21 | diff --git a/progress.md b/progress.md index 9ab261b..797e619 100644 --- a/progress.md +++ b/progress.md @@ -20,7 +20,7 @@ Items #10 to #26 are what is left of the 1.0 roadmap (`docs/FILEAUTOMATION-1.0-R - **#14** Google Drive adapter (`gdrive://`). Drive addresses files by ID and allows two files of one name in a folder, so the path-to-ID lookup and the duplicate-name rule have to be designed first. - **#15** [DECIDE] The eleventh backend slot, and whether OneDrive and Box are promoted to the storage contract or documented as action-only (roadmap §4). - **#16** `copy_between` / `FA_copy_between` on `File.copy_to`. The `FA_storage_*` actions exist (U-20261008-03), so the layer is reachable from action lists; the older action still has its own dispatcher in `remote/cross_backend.py`. `copy_between` accepts `local:`, `sftp:/path`, `s3:bucket/key` and http(s) sources today; `parse_storage_uri` rejects the first three as ambiguous, so the action needs a translation step to stay compatible. -- **#17** Streams and directory trees: `open()` or chunked reads and writes (`read_bytes` holds the whole file in memory, and the default `checksum` stages a full local copy of a remote file), directory copy and sync through the layer, and a backend's native checksum where it has one. +- **#17** Native streams and checksums for the remote backends. `open_read` / `open_write`, `copy_tree` and `sync_tree` exist (U-20261008-04), but outside `LocalStorage` a stream is a staged local copy and the default `checksum` downloads the file: S3 could read `get_object()["Body"]`, and a backend with a server-side digest could answer `checksum` from it. - **#18** [UNVERIFIED] The five symbolic-link tests of `tests/test_storage_local.py` have not run anywhere: the development machine may not create links (they skip there), CI's Windows runners may. Read the first CI run of the branch and fix `LocalStorage` if one fails. ### Backend integration tests (roadmap M3) diff --git a/tests/storage_contract.py b/tests/storage_contract.py index a6541c7..45aeb4f 100644 --- a/tests/storage_contract.py +++ b/tests/storage_contract.py @@ -219,6 +219,50 @@ def test_read_of_a_missing_file_raises_not_found(self, backend: StorageBackend) with pytest.raises(StorageNotFoundException): backend.read_bytes("nope.bin") + # ------------------------------------------------------------------ streams + + def test_open_read_streams_the_content(self, backend: StorageBackend) -> None: + backend.write_bytes("dir/data.bin", BINARY) + with backend.open_read("dir/data.bin") as stream: + assert stream.read(10) == BINARY[:10] + assert stream.read() == BINARY[10:] + assert stream.read() == b"" + assert stream.closed is True + + def test_open_read_of_a_missing_file_raises_not_found(self, backend: StorageBackend) -> None: + with pytest.raises(StorageNotFoundException): + backend.open_read("nope.bin") + + def test_open_read_of_a_directory_is_refused(self, backend: StorageBackend) -> None: + backend.write_bytes("dir/a.txt", b"x") + with pytest.raises(StoragePathTypeException): + backend.open_read("dir") + + def test_open_write_stores_the_content_on_close(self, backend: StorageBackend) -> None: + with backend.open_write("deep/dir/out.bin") as stream: + stream.write(BINARY[:100]) + stream.write(BINARY[100:]) + assert backend.exists("deep/dir/out.bin") is False + assert backend.read_bytes("deep/dir/out.bin") == BINARY + + def test_open_write_stores_nothing_when_the_block_fails(self, backend: StorageBackend) -> None: + backend.write_bytes("a.txt", b"previous") + with pytest.raises(RuntimeError, match="boom"), backend.open_write("a.txt") as stream: + stream.write(b"half") + raise RuntimeError("boom") + assert backend.read_bytes("a.txt") == b"previous" + + def test_open_write_respects_overwrite(self, backend: StorageBackend) -> None: + backend.write_bytes("a.txt", b"previous") + with pytest.raises(StorageAlreadyExistsException): + backend.open_write("a.txt", overwrite=False) + assert backend.read_bytes("a.txt") == b"previous" + + def test_open_write_onto_a_directory_is_refused(self, backend: StorageBackend) -> None: + backend.write_bytes("dir/a.txt", b"x") + with pytest.raises(StoragePathTypeException): + backend.open_write("dir") + # ------------------------------------------------------------------ list_dir def test_list_dir_of_an_empty_root_is_empty(self, backend: StorageBackend) -> None: diff --git a/tests/test_storage_actions.py b/tests/test_storage_actions.py index 2179d1e..3ffd27d 100644 --- a/tests/test_storage_actions.py +++ b/tests/test_storage_actions.py @@ -27,6 +27,7 @@ NAMES = [ "FA_storage_checksum", "FA_storage_copy", + "FA_storage_copy_tree", "FA_storage_delete", "FA_storage_download", "FA_storage_exists", @@ -36,6 +37,7 @@ "FA_storage_read_text", "FA_storage_schemes", "FA_storage_stat", + "FA_storage_sync", "FA_storage_upload", "FA_storage_verify", "FA_storage_write_text", @@ -194,3 +196,28 @@ def test_the_actions_are_mcp_tools_with_their_parameters() -> None: assert copy["required"] == ["source", "target"] assert tools["FA_storage_schemes"]["inputSchema"].get("required", []) == [] assert "Copy the file" in tools["FA_storage_copy"]["description"] + + +def test_copy_tree_and_sync_actions(tmp_path: Path) -> None: + for path in ("a.txt", "sub/b.txt"): + actions.storage_write_text(f"memory://scratch/src/{path}", "x") + copied = actions.storage_copy_tree("memory://scratch/src", str(tmp_path / "out")) + assert copied == { + "copied": ["a.txt", "sub/b.txt"], + "skipped": [], + "deleted": [], + "errors": {}, + "dry_run": False, + } + assert (tmp_path / "out" / "sub" / "b.txt").read_bytes() == b"x" + (tmp_path / "out" / "extra.txt").write_bytes(b"extra") + preview = actions.storage_sync( + "memory://scratch/src", str(tmp_path / "out"), delete=True, dry_run=True + ) + assert preview["deleted"] == ["extra.txt"] + assert preview["dry_run"] is True + assert (tmp_path / "out" / "extra.txt").exists() + synced = actions.storage_sync("memory://scratch/src", str(tmp_path / "out"), delete=True) + assert synced["deleted"] == ["extra.txt"] + assert not (tmp_path / "out" / "extra.txt").exists() + assert json.loads(json.dumps(synced)) == synced diff --git a/tests/test_storage_tree.py b/tests/test_storage_tree.py new file mode 100644 index 0000000..d572f05 --- /dev/null +++ b/tests/test_storage_tree.py @@ -0,0 +1,178 @@ +"""copy_tree and sync_tree: directory trees between any two backends.""" + +from __future__ import annotations + +import os +from collections.abc import Iterator +from pathlib import Path + +import pytest + +from automation_file.exceptions import ( + StorageException, + StorageNotFoundException, + StoragePathTypeException, +) +from automation_file.storage import ( + File, + MemoryStorage, + Storage, + StorageResolver, + TreeResult, + clear_memory_stores, + copy_tree, + sync_tree, +) + + +@pytest.fixture(autouse=True) +def _fresh_stores() -> Iterator[None]: + clear_memory_stores() + yield + clear_memory_stores() + + +@pytest.fixture +def resolver() -> StorageResolver: + return StorageResolver() + + +@pytest.fixture +def source(resolver: StorageResolver) -> Storage: + storage = Storage("memory://one/src", resolver=resolver) + for path, data in (("a.txt", b"a"), ("sub/b.txt", b"bb"), ("sub/deep/c.txt", b"ccc")): + storage.file(path).write(data) + storage.mkdir("empty") + return storage + + +def _files(storage: Storage) -> dict[str, bytes]: + return { + info.path: storage.file(info.path).read() + for info in storage.list_dir(recursive=True) + if not info.is_dir + } + + +def test_copy_tree_to_another_backend(source: Storage, tmp_path: Path) -> None: + target = Storage(tmp_path / "out", resolver=StorageResolver()) + result = source.copy_to(target) + assert isinstance(result, TreeResult) + assert result.copied == ["a.txt", "sub/b.txt", "sub/deep/c.txt"] + assert (result.skipped, result.deleted, result.errors, result.dry_run) == ([], [], {}, False) + assert result.ok is True + assert _files(target) == {"a.txt": b"a", "sub/b.txt": b"bb", "sub/deep/c.txt": b"ccc"} + assert (tmp_path / "out" / "empty").is_dir() + + +def test_copy_tree_accepts_a_uri_and_uses_the_same_resolver( + source: Storage, resolver: StorageResolver +) -> None: + resolver.mount("vault://bucket", MemoryStorage("vault")) + result = source.copy_to("vault://bucket/backup") + assert result.copied == ["a.txt", "sub/b.txt", "sub/deep/c.txt"] + assert File("vault://bucket/backup/sub/b.txt", resolver=resolver).read() == b"bb" + + +def test_copy_tree_without_overwrite_skips_what_exists( + source: Storage, resolver: StorageResolver +) -> None: + target = Storage("memory://two/dst", resolver=resolver) + target.file("a.txt").write(b"kept") + result = copy_tree(source, target, overwrite=False) + assert result.skipped == ["a.txt"] + assert result.copied == ["sub/b.txt", "sub/deep/c.txt"] + assert target.file("a.txt").read() == b"kept" + assert copy_tree(source, target).copied == ["a.txt", "sub/b.txt", "sub/deep/c.txt"] + assert target.file("a.txt").read() == b"a" + + +def test_copy_tree_needs_an_existing_source_directory(resolver: StorageResolver) -> None: + target = Storage("memory://two/dst", resolver=resolver) + with pytest.raises(StorageNotFoundException): + copy_tree(Storage("memory://one/nope", resolver=resolver), target) + File("memory://one/file.txt", resolver=resolver).write(b"x") + with pytest.raises(StoragePathTypeException): + copy_tree(Storage("memory://one/file.txt", resolver=resolver), target) + + +def test_a_tree_cannot_be_copied_onto_itself(source: Storage, resolver: StorageResolver) -> None: + with pytest.raises(StorageException, match="both the source and the target"): + source.copy_to("memory://one/src/") + with pytest.raises(StorageException, match="both the source and the target"): + sync_tree(source, Storage("memory://one/src", resolver=resolver)) + + +def test_sync_copies_only_what_changed(source: Storage, resolver: StorageResolver) -> None: + target = Storage("memory://two/dst", resolver=resolver) + assert source.sync_to(target).copied == ["a.txt", "sub/b.txt", "sub/deep/c.txt"] + again = source.sync_to(target) + assert again.copied == [] + assert again.skipped == ["a.txt", "sub/b.txt", "sub/deep/c.txt"] + source.file("sub/b.txt").write(b"longer now") + changed = source.sync_to(target) + assert changed.copied == ["sub/b.txt"] + assert changed.skipped == ["a.txt", "sub/deep/c.txt"] + assert target.file("sub/b.txt").read() == b"longer now" + + +def test_sync_copies_a_newer_file_of_the_same_size(tmp_path: Path) -> None: + resolver = StorageResolver() + (tmp_path / "src").mkdir() + (tmp_path / "dst").mkdir() + (tmp_path / "src" / "a.txt").write_bytes(b"new!") + (tmp_path / "dst" / "a.txt").write_bytes(b"old!") + os.utime(tmp_path / "dst" / "a.txt", (1_600_000_000, 1_600_000_000)) + source = Storage(tmp_path / "src", resolver=resolver) + target = Storage(tmp_path / "dst", resolver=resolver) + assert source.sync_to(target).copied == ["a.txt"] + assert (tmp_path / "dst" / "a.txt").read_bytes() == b"new!" + os.utime(tmp_path / "src" / "a.txt", (1_500_000_000, 1_500_000_000)) + (tmp_path / "dst" / "a.txt").write_bytes(b"tgt!") + assert source.sync_to(target).skipped == ["a.txt"] + assert (tmp_path / "dst" / "a.txt").read_bytes() == b"tgt!" + assert source.sync_to(target, checksum=True).copied == ["a.txt"] + assert (tmp_path / "dst" / "a.txt").read_bytes() == b"new!" + assert source.sync_to(target, checksum=True).skipped == ["a.txt"] + + +def test_sync_with_delete_removes_what_the_source_lacks( + source: Storage, resolver: StorageResolver +) -> None: + target = Storage("memory://two/dst", resolver=resolver) + source.sync_to(target) + target.file("stale.txt").write(b"x") + target.file("old/dir/stale.txt").write(b"x") + kept = source.sync_to(target) + assert kept.deleted == [] + assert target.exists("stale.txt") is True + removed = source.sync_to(target, delete=True) + assert removed.deleted == ["old/dir/stale.txt", "stale.txt", "old/dir", "old"] + assert _files(target) == _files(source) + assert target.exists("old") is False + assert target.exists("empty") is True + + +def test_a_dry_run_changes_nothing(source: Storage, resolver: StorageResolver) -> None: + target = Storage("memory://two/dst", resolver=resolver) + target.file("stale.txt").write(b"x") + preview = source.sync_to(target, delete=True, dry_run=True) + assert preview.dry_run is True + assert preview.copied == ["a.txt", "sub/b.txt", "sub/deep/c.txt"] + assert preview.deleted == ["stale.txt"] + assert _files(target) == {"stale.txt": b"x"} + assert target.exists("empty") is False + assert preview.to_dict()["dry_run"] is True + + +def test_a_failing_file_is_recorded_and_the_rest_still_run( + source: Storage, resolver: StorageResolver +) -> None: + target = Storage("memory://two/dst", resolver=resolver) + target.file("sub/b.txt/blocker").write(b"x") + result = source.copy_to(target) + assert result.copied == ["a.txt", "sub/deep/c.txt"] + assert list(result.errors) == ["sub/b.txt"] + assert result.errors["sub/b.txt"].startswith("StoragePathTypeException:") + assert result.ok is False + assert target.file("a.txt").read() == b"a" From 16e5eabdf2f8beeb7538353ad82c534e1e094cff Mon Sep 17 00:00:00 2001 From: JeffreyChen Date: Thu, 8 Oct 2026 12:27:50 +0800 Subject: [PATCH 26/59] feat: add the event model, the event bus and storage observers --- CLAUDE.md | 3 + README.md | 24 ++ README.zh-CN.md | 24 ++ README.zh-TW.md | 24 ++ architecture.md | 17 +- automation_file/__init__.py | 37 +++ automation_file/events/__init__.py | 81 ++++++ automation_file/events/bus.py | 154 ++++++++++ automation_file/events/context.py | 67 +++++ automation_file/events/model.py | 174 ++++++++++++ automation_file/events/storage_bridge.py | 66 +++++ automation_file/storage/backend.py | 161 +++++++---- automation_file/storage/observe.py | 111 ++++++++ docs/source/API/api_index.rst | 16 +- docs/source/API/events.rst | 20 ++ docs/source/Eng/eng_index.rst | 17 +- docs/source/Eng/usage/event_bus.rst | 143 ++++++++++ docs/source/Zh-CN/usage/event_bus.rst | 139 +++++++++ docs/source/Zh-CN/zh_cn_index.rst | 16 +- docs/source/Zh-TW/usage/event_bus.rst | 139 +++++++++ docs/source/Zh-TW/zh_tw_index.rst | 16 +- docs/source/index.rst | 2 +- docs/updates/2026-10.md | 16 ++ docs/updates/README.md | 3 +- tests/test_events.py | 343 +++++++++++++++++++++++ tests/test_storage_observe.py | 161 +++++++++++ 26 files changed, 1910 insertions(+), 64 deletions(-) create mode 100644 automation_file/events/__init__.py create mode 100644 automation_file/events/bus.py create mode 100644 automation_file/events/context.py create mode 100644 automation_file/events/model.py create mode 100644 automation_file/events/storage_bridge.py create mode 100644 automation_file/storage/observe.py create mode 100644 docs/source/API/events.rst create mode 100644 docs/source/Eng/usage/event_bus.rst create mode 100644 docs/source/Zh-CN/usage/event_bus.rst create mode 100644 docs/source/Zh-TW/usage/event_bus.rst create mode 100644 tests/test_events.py create mode 100644 tests/test_storage_observe.py diff --git a/CLAUDE.md b/CLAUDE.md index 0d853d1..f422201 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -30,6 +30,8 @@ automation_file/ │ # resolver (StorageResolver), file (File), storage (Storage), streams, │ # tree (copy_tree, sync_tree), │ # actions (FA_storage_* and register_storage_ops) +├── events/ # Event model: model (Event, Severity, the ten core events), bus (EventBus, +│ # event_bus, emit), context (correlation_scope, actor_scope), storage_bridge ├── server/ # tcp_server, http_server, mcp_server (MCP over stdio), web_ui, metrics_server, │ # action_acl (ActionACL), network_guards (ensure_loopback) ├── client/ # HTTPActionClient for the HTTP action server @@ -69,6 +71,7 @@ automation_file/ - `safe_join(root, user_path)` / `is_within(root, path)` — path traversal guard; `safe_join` raises `PathTraversalException` when the resolved path escapes `root`. - `File(uri)` / `Storage(uri)` — the universal storage layer's application API: one file, one directory, in any backend. Both resolve their backend on every call through `StorageResolver` (`Storage.mount`, `Storage.register_scheme`). - `StorageBackend` — the contract a storage backend implements. The public operations (`exists`, `stat`, `list_dir`, `mkdir`, `upload`, `download`, `delete`, `checksum`, `read_bytes`, `write_bytes`, `copy_from`, `move_from`) are template methods; a backend supplies only the `_`-prefixed primitives. `LocalStorage`, `MemoryStorage`, `S3Storage` (`s3://bucket/key`) and `AzureStorage` (`azure://container/blob`) are built in; the last two extend `ObjectStorage` and use the shared `s3_instance` / `azure_blob_instance` unless given a client. +- `Event` / `EventBus` / `event_bus` — every component reports through events (`PipelineFailed`, `TaskFailed`, `IntegrityViolation`, `StorageError`, ...) with a severity, a correlation ID and an actor; consumers subscribe on the bus by class, type name or prefix. New code that has something to report publishes an event; it does not call a notification sink or the audit log directly. - `StorageURI` / `parse_storage_uri` — `:///`; `FileInfo`, `Checksum`, `StorageCapabilities` are the frozen value types the layer returns. ## Branching & CI diff --git a/README.md b/README.md index 90c4af8..c32c9b1 100644 --- a/README.md +++ b/README.md @@ -49,6 +49,7 @@ facade. - **HTMX Web UI** — `start_web_ui()` serves a read-only dashboard (health, progress, registry) that polls HTML fragments; stdlib-only HTTP plus one CDN script with SRI - **MCP (Model Context Protocol) server** — `MCPServer` bridges the registry to any MCP host (Claude Desktop, MCP CLIs) over newline-delimited JSON-RPC 2.0 on stdio; every `FA_*` action becomes an MCP tool with an auto-generated input schema - **Universal storage layer** — `File` / `Storage` address local and remote storage with one URI syntax (`local:///…`, `s3://…`, `azure://…`, `memory://…`), one `StorageBackend` contract and one error hierarchy; local, S3, Azure Blob and in-memory backends are built in, and a 77-case contract suite checks any backend +- **Event bus** — one `Event` model with ten core events (`pipeline.*`, `task.*`, `integrity.violation`, `storage.error`, `scheduler.error`, `system.error`), severities, correlation IDs and actors; subscribe on `event_bus` by class, type or prefix - PySide6 GUI (`python -m automation_file ui`) with a tab per backend, the JSON-action runner, and dedicated tabs for Triggers, Scheduler, and live Progress - Rich CLI with one-shot subcommands plus legacy JSON-batch flags - Project scaffolding (`ProjectBuilder`) for executor-based automations @@ -505,6 +506,29 @@ File("sandbox://jobs/42/out.csv").write(b"done") The API is new and may still change before 1.0. Full reference: the *Universal Storage Layer* chapter of the documentation. +### Events +Every component reports through one event model instead of calling a sink or the audit log itself. + +```python +from automation_file import Severity, actor_scope, correlation_scope, event_bus + +event_bus.subscribe(print, types=["pipeline.*", "integrity.violation"]) +event_bus.subscribe(alert, min_severity=Severity.ERROR) + +with actor_scope("scheduler"), correlation_scope() as run_id: + ... # every event and storage operation in here carries run_id and the actor +event_bus.recent(limit=20, correlation_id=run_id) +``` + +- **Core events** — `PipelineStarted`, `PipelineCompleted`, `PipelineFailed`, `TaskStarted`, + `TaskCompleted`, `TaskFailed`, `IntegrityViolation`, `StorageError`, `SchedulerError`, + `SystemErrorEvent`. Each has a `type` (`pipeline.failed`), a `severity`, a `source`, a `subject`, + a structured `payload`, a `correlation_id` and an `actor`, and turns into JSON with `to_dict()`. +- **Bus** — `event_bus.subscribe(handler, types=..., min_severity=...)` by class, type name or + prefix; a handler that raises is logged and skipped; `event_bus.recent()` returns the latest events. +- **Storage operations** — uploads, downloads, reads, deletes, copies and moves are reported to + `automation_file.storage.observe` listeners, and a failing backend becomes a `StorageError` event. + ### File-watcher triggers Run an action list whenever a filesystem event fires on a watched path: diff --git a/README.zh-CN.md b/README.zh-CN.md index 3c89b8e..3a25e74 100644 --- a/README.zh-CN.md +++ b/README.zh-CN.md @@ -47,6 +47,7 @@ TCP / HTTP 服务器执行的 JSON 驱动动作。内附 PySide6 GUI，每个功 - **HTMX Web UI** — `start_web_ui()` 启动只读观测仪表板（health、progress、registry），通过 HTML 片段轮询；仅用标准库 HTTP，搭配一个带 SRI 的 CDN 脚本 - **MCP（Model Context Protocol）服务器** — `MCPServer` 通过 stdio 上的 JSON-RPC 2.0（换行分隔 JSON）将注册表桥接到任意 MCP 主机（Claude Desktop、MCP CLI）；每个 `FA_*` 动作都会自动生成输入 schema 并成为 MCP 工具 - **通用存储层** — `File` / `Storage` 以同一套 URI 语法（`local:///…`、`s3://…`、`azure://…`、`memory://…`）、同一份 `StorageBackend` 契约与同一组异常层级访问本地与远端存储；内置本地、S3、Azure Blob 与内存后端，并附带 77 个用例的契约测试套件可检查任何后端 +- **事件总线** — 单一 `Event` 模型与十种核心事件（`pipeline.*`、`task.*`、`integrity.violation`、`storage.error`、`scheduler.error`、`system.error`），具备严重程度、关联 ID 与 actor；可以在 `event_bus` 上按类、type 或前缀订阅 - PySide6 GUI（`python -m automation_file ui`）每个后端一个页签，含 JSON 动作执行器，另有 Triggers、Scheduler、实时 Progress 专属页签 - 功能丰富的 CLI，包含一次性子命令与旧式 JSON 批量标志 - 项目脚手架（`ProjectBuilder`）协助构建以 executor 为核心的自动化项目 @@ -499,6 +500,29 @@ File("sandbox://jobs/42/out.csv").write(b"done") 此 API 为新功能，在 1.0 之前仍可能调整。完整说明请见文档的“通用存储层”章节。 +### 事件 +每个组件都通过同一套事件模型报告，而不是自行调用通知接收端或审计记录。 + +```python +from automation_file import Severity, actor_scope, correlation_scope, event_bus + +event_bus.subscribe(print, types=["pipeline.*", "integrity.violation"]) +event_bus.subscribe(alert, min_severity=Severity.ERROR) + +with actor_scope("scheduler"), correlation_scope() as run_id: + ... # 这里面的每个事件与存储操作都带有 run_id 与 actor +event_bus.recent(limit=20, correlation_id=run_id) +``` + +- **核心事件** — `PipelineStarted`、`PipelineCompleted`、`PipelineFailed`、`TaskStarted`、 + `TaskCompleted`、`TaskFailed`、`IntegrityViolation`、`StorageError`、`SchedulerError`、 + `SystemErrorEvent`。每个事件都有 `type`（`pipeline.failed`）、`severity`、`source`、`subject`、 + 结构化的 `payload`、`correlation_id` 与 `actor`，并可以用 `to_dict()` 转成 JSON。 +- **总线** — `event_bus.subscribe(handler, types=..., min_severity=...)` 可以按类、type 名称或 + 前缀订阅；处理函数抛出异常时只会被记录并跳过；`event_bus.recent()` 返回最近的事件。 +- **存储操作** — 上传、下载、读取、删除、复制与移动都会报告给 + `automation_file.storage.observe` 的监听者，后端失败时会产生 `StorageError` 事件。 + ### 文件监听触发 每当被监听路径发生文件系统事件，就执行动作清单： diff --git a/README.zh-TW.md b/README.zh-TW.md index c4cf67b..c8eb857 100644 --- a/README.zh-TW.md +++ b/README.zh-TW.md @@ -47,6 +47,7 @@ TCP / HTTP 伺服器執行的 JSON 驅動動作。內附 PySide6 GUI，每個功 - **HTMX Web UI** — `start_web_ui()` 啟動唯讀觀測儀表板（health、progress、registry），以 HTML 片段輪詢；僅用標準函式庫 HTTP，搭配一支帶 SRI 的 CDN 腳本 - **MCP（Model Context Protocol）伺服器** — `MCPServer` 透過 stdio 上的 JSON-RPC 2.0（行分隔 JSON）將登錄表橋接到任何 MCP 主機（Claude Desktop、MCP CLI）；每個 `FA_*` 動作都會自動生成輸入 schema 並成為 MCP 工具 - **通用儲存層** — `File` / `Storage` 以同一套 URI 語法（`local:///…`、`s3://…`、`azure://…`、`memory://…`）、同一份 `StorageBackend` 契約與同一組例外階層存取本機與遠端儲存；內建本機、S3、Azure Blob 與記憶體後端，並附 77 個案例的契約測試套件可檢查任何後端 +- **事件匯流排** — 單一 `Event` 模型與十種核心事件（`pipeline.*`、`task.*`、`integrity.violation`、`storage.error`、`scheduler.error`、`system.error`），具備嚴重程度、關聯 ID 與 actor；可在 `event_bus` 上依類別、type 或前綴訂閱 - PySide6 GUI（`python -m automation_file ui`）每個後端一個分頁，含 JSON 動作執行器，另有 Triggers、Scheduler、即時 Progress 專屬分頁 - 功能豐富的 CLI，包含一次性子指令與舊式 JSON 批次旗標 - 專案鷹架（`ProjectBuilder`）協助建立以 executor 為核心的自動化專案 @@ -499,6 +500,29 @@ File("sandbox://jobs/42/out.csv").write(b"done") 此 API 為新功能，在 1.0 之前仍可能調整。完整說明請見文件的「通用儲存層」章節。 +### 事件 +每個元件都透過同一套事件模型回報，而不是自行呼叫通知接收端或稽核紀錄。 + +```python +from automation_file import Severity, actor_scope, correlation_scope, event_bus + +event_bus.subscribe(print, types=["pipeline.*", "integrity.violation"]) +event_bus.subscribe(alert, min_severity=Severity.ERROR) + +with actor_scope("scheduler"), correlation_scope() as run_id: + ... # 這裡面的每個事件與儲存操作都帶有 run_id 與 actor +event_bus.recent(limit=20, correlation_id=run_id) +``` + +- **核心事件** — `PipelineStarted`、`PipelineCompleted`、`PipelineFailed`、`TaskStarted`、 + `TaskCompleted`、`TaskFailed`、`IntegrityViolation`、`StorageError`、`SchedulerError`、 + `SystemErrorEvent`。每個事件都有 `type`（`pipeline.failed`）、`severity`、`source`、`subject`、 + 結構化的 `payload`、`correlation_id` 與 `actor`，並可用 `to_dict()` 轉成 JSON。 +- **匯流排** — `event_bus.subscribe(handler, types=..., min_severity=...)` 可依類別、type 名稱或 + 前綴訂閱；處理函式拋出例外時只會被記錄並略過；`event_bus.recent()` 回傳最近的事件。 +- **儲存操作** — 上傳、下載、讀取、刪除、複製與搬移都會回報給 + `automation_file.storage.observe` 的監聽者，後端失敗時會產生 `StorageError` 事件。 + ### 檔案監看觸發 每當被監看路徑發生檔案系統事件，就執行動作清單： diff --git a/architecture.md b/architecture.md index 98fdaf8..cf03f21 100644 --- a/architecture.md +++ b/architecture.md @@ -24,7 +24,8 @@ piece of the 1.0 roadmap (`docs/FILEAUTOMATION-1.0-ROADMAP.md`, PR #107); what i | `automation_file/core/` | Engine, on je_action_core: `action_registry.py` (`ActionRegistry`, a `CommandRegistry`; `build_default_registry`), `action_executor.py` (`ActionExecutor`, an `ActionExecutor` with strict actions, indexed records and the dry-run, validate, substitute and parallel extras; shared `executor`), `callback_executor.py`, `package_loader.py`, `plugins.py`, `dag_executor.py`, `action_queue.py`, `json_store.py`, `substitution.py`. Also cross-cutting helpers: `retry`, `quota`, `rate_limit`, `circuit_breaker`, `file_lock`, `sqlite_lock`, `checksum`, `manifest`, `crypto`, `secrets`, `config`, `config_watcher`, `audit`, `metrics`, `tracing`, `progress`, `fim`, `content_store` | | `automation_file/local/` | Local strategy modules: file, dir, zip, tar and archive ops, sync, diff, text/JSON/data edits, templates, versioning, trash, `shell_ops` (argv-only subprocess), conditional branches. `safe_paths.py` guards against path traversal | | `automation_file/remote/` | `url_validator.py` (SSRF guard), `http_download.py`, `cross_backend.py`, `fsspec_bridge.py`. One subpackage per backend: `google_drive/`, `s3/`, `azure_blob/`, `dropbox_api/`, `sftp/`, `ftp/`, `onedrive/`, `box/`, each with `client.py`, `*_ops.py` and `register__ops`. `smb/` and `webdav/` have a client only | -| `automation_file/storage/` | Universal storage layer. `uri.py` (`StorageURI`, `parse_storage_uri`, `normalize_path`), `types.py` (`FileInfo`, `Checksum`, `StorageCapabilities`), `backend.py` (`StorageBackend`: the public operations are template methods over the `_`-prefixed primitives a backend supplies), `local_storage.py` (`LocalStorage`, confined through `safe_join` when given a root), `memory_storage.py` (`MemoryStorage`), `object_storage.py` (`ObjectStorage`: directories as key prefixes over `_head`, `_scan`, `_put`, `_get`, `_remove`), `s3_storage.py` (`S3Storage`, over `s3_instance` or a given boto3 client), `azure_storage.py` (`AzureStorage`, over `azure_blob_instance` or a given `BlobServiceClient`), `resolver.py` (`StorageResolver`, `default_resolver`: mounts first, then scheme factories), `file.py` (`File`), `storage.py` (`Storage`), `streams.py` (staged file objects behind `open_read` / `open_write`), `tree.py` (`copy_tree`, `sync_tree`, `TreeResult`), `actions.py` (the `FA_storage_*` functions and `register_storage_ops`). At module level it imports only `exceptions`, `logging_config`, `core.checksum` and `local.safe_paths`: no registry, no GUI, no backend SDK. The adapters import their SDK's exceptions and the shared client inside the functions that use them | +| `automation_file/storage/` | Universal storage layer. `uri.py` (`StorageURI`, `parse_storage_uri`, `normalize_path`), `types.py` (`FileInfo`, `Checksum`, `StorageCapabilities`), `backend.py` (`StorageBackend`: the public operations are template methods over the `_`-prefixed primitives a backend supplies), `local_storage.py` (`LocalStorage`, confined through `safe_join` when given a root), `memory_storage.py` (`MemoryStorage`), `object_storage.py` (`ObjectStorage`: directories as key prefixes over `_head`, `_scan`, `_put`, `_get`, `_remove`), `s3_storage.py` (`S3Storage`, over `s3_instance` or a given boto3 client), `azure_storage.py` (`AzureStorage`, over `azure_blob_instance` or a given `BlobServiceClient`), `resolver.py` (`StorageResolver`, `default_resolver`: mounts first, then scheme factories), `file.py` (`File`), `storage.py` (`Storage`), `observe.py` (listeners for `upload`, `download`, `read`, `delete`, `mkdir`, `copy`, `move`), `streams.py` (staged file objects behind `open_read` / `open_write`), `tree.py` (`copy_tree`, `sync_tree`, `TreeResult`), `actions.py` (the `FA_storage_*` functions and `register_storage_ops`). At module level it imports only `exceptions`, `logging_config`, `core.checksum` and `local.safe_paths`: no registry, no GUI, no backend SDK. The adapters import their SDK's exceptions and the shared client inside the functions that use them | +| `automation_file/events/` | The event model every component reports through. `model.py` (`Event`, `Severity`, the ten core events), `bus.py` (`EventBus`, the process-wide `event_bus`, `emit`), `context.py` (`correlation_scope`, `actor_scope`), `storage_bridge.py` (failed storage operations become `StorageError` events; installed when the package is imported). It imports only the standard library, `logging_config` and `storage.observe` | | `automation_file/server/` | `tcp_server.py`, `http_server.py`, `mcp_server.py`, `web_ui.py`, `metrics_server.py`, `action_acl.py` (`ActionACL`), `network_guards.py` (`ensure_loopback`) | | `automation_file/client/` | `HTTPActionClient` for the HTTP action server | | `automation_file/trigger/`, `scheduler/`, `notify/` | Watchdog file triggers, cron scheduler, notification sinks. Each registers its own `FA_*` ops | @@ -55,6 +56,11 @@ piece of the 1.0 roadmap (`docs/FILEAUTOMATION-1.0-ROADMAP.md`, PR #107); what i `download`, `delete`, `checksum`, `verify`, `copy`, `move`, `read_text`, `write_text`, `copy_tree`, `sync`, `schemes`) put it in the default registry; `register_storage_ops` adds them to another one. +- **Events** (same facade): `Event`, `Severity`, `EventBus`, `event_bus`, `emit`, `correlation_scope`, + `actor_scope`, and the core events `PipelineStarted`, `PipelineCompleted`, `PipelineFailed`, + `TaskStarted`, `TaskCompleted`, `TaskFailed`, `IntegrityViolation`, `StorageError`, `SchedulerError`, + `SystemErrorEvent`. Event `type` names (`pipeline.failed`, ...) and the payload keys in + `events.model.PAYLOAD_KEYS` are what consumers match on. - **Action format**: an action is `[name]`, `[name, {kwargs}]` or `[name, [args]]`. A file holds a list of actions or `{"auto_control": [...]}`. - **CLI** (`python -m automation_file`; no console script for it): @@ -126,6 +132,15 @@ File(uri) / Storage(uri) → parse_storage_uri (scheme alias, authority check, p → StorageBackend public method: normalise, check what exists, make parents → _primitive of the backend → FileInfo / Checksum / bytes, or a StorageException subclass copy_to / move_to → target_backend.copy_from(source_backend, ...) → native (_copy_from / _move_from) or a local staging file +every upload / download / read / delete / mkdir / copy / move → storage.observe listeners (StorageOperation) +``` + +**Event → consumers** + +``` +component → Event (type, severity, source, subject, payload, correlation_id, actor) → event_bus.publish + → each matching subscriber, in the publisher's thread; one that raises is logged and skipped +storage.observe → events.storage_bridge → StorageError (only for a failing backend, not a caller mistake) ``` ## 5. Extension points diff --git a/automation_file/__init__.py b/automation_file/__init__.py index 0f4b114..4e61bb9 100644 --- a/automation_file/__init__.py +++ b/automation_file/__init__.py @@ -74,6 +74,25 @@ from automation_file.core.sqlite_lock import SQLiteLock from automation_file.core.substitution import SubstitutionException, substitute from automation_file.core.tracing import action_span, init_tracing +from automation_file.events import ( + Event, + EventBus, + IntegrityViolation, + PipelineCompleted, + PipelineFailed, + PipelineStarted, + SchedulerError, + Severity, + StorageError, + SystemErrorEvent, + TaskCompleted, + TaskFailed, + TaskStarted, + actor_scope, + correlation_scope, + emit, + event_bus, +) from automation_file.exceptions import ( BoxException, DataOpsException, @@ -507,6 +526,24 @@ def __getattr__(name: str) -> Any: "StorageTransientException", "StorageUnavailableException", "StorageUnsupportedException", + # Events + "Event", + "EventBus", + "Severity", + "event_bus", + "emit", + "correlation_scope", + "actor_scope", + "PipelineStarted", + "PipelineCompleted", + "PipelineFailed", + "TaskStarted", + "TaskCompleted", + "TaskFailed", + "IntegrityViolation", + "StorageError", + "SchedulerError", + "SystemErrorEvent", # Server / Project / Utils "TCPActionServer", "start_autocontrol_socket_server", diff --git a/automation_file/events/__init__.py b/automation_file/events/__init__.py new file mode 100644 index 0000000..7b3485b --- /dev/null +++ b/automation_file/events/__init__.py @@ -0,0 +1,81 @@ +"""Events: one way for every component to say what happened. + +Components publish :class:`Event` objects on the :data:`event_bus`; the +notification router, the audit trail and the metrics subscribe to it. Nothing +calls a sink or writes an audit row directly. + +Importing this package installs the bridge that turns failed storage operations +into :class:`StorageError` events. +""" + +from __future__ import annotations + +from automation_file.events.bus import ( + EventBus, + EventFilter, + EventHandler, + Subscription, + emit, + event_bus, +) +from automation_file.events.context import ( + actor_scope, + correlation_scope, + current_actor, + current_correlation_id, + new_correlation_id, +) +from automation_file.events.model import ( + CORE_EVENTS, + PAYLOAD_KEYS, + Event, + IntegrityViolation, + PipelineCompleted, + PipelineFailed, + PipelineStarted, + SchedulerError, + Severity, + StorageError, + SystemErrorEvent, + TaskCompleted, + TaskFailed, + TaskStarted, +) +from automation_file.events.storage_bridge import ( + StorageErrorBridge, + install_storage_bridge, + uninstall_storage_bridge, +) + +install_storage_bridge() + +__all__ = [ + "CORE_EVENTS", + "PAYLOAD_KEYS", + "Event", + "EventBus", + "EventFilter", + "EventHandler", + "IntegrityViolation", + "PipelineCompleted", + "PipelineFailed", + "PipelineStarted", + "SchedulerError", + "Severity", + "StorageError", + "StorageErrorBridge", + "Subscription", + "SystemErrorEvent", + "TaskCompleted", + "TaskFailed", + "TaskStarted", + "actor_scope", + "correlation_scope", + "current_actor", + "current_correlation_id", + "emit", + "event_bus", + "install_storage_bridge", + "new_correlation_id", + "uninstall_storage_bridge", +] diff --git a/automation_file/events/bus.py b/automation_file/events/bus.py new file mode 100644 index 0000000..9bd7a9b --- /dev/null +++ b/automation_file/events/bus.py @@ -0,0 +1,154 @@ +"""The event bus: publishers on one side, subscribers on the other. + +``publish`` delivers an event to every matching subscriber in the publisher's +thread, in subscription order. A subscriber that raises is logged and skipped, +so one broken consumer never reaches the code that reported the event. The bus +also keeps the most recent events for dashboards and status tools. + +A subscription matches by event class (subclasses included), by type name +(``"task.failed"``), by type prefix (``"pipeline.*"``), and by a minimum +severity; with no filter it receives everything. +""" + +from __future__ import annotations + +import threading +from collections import deque +from collections.abc import Callable, Iterable +from dataclasses import dataclass + +from automation_file.events.model import Event, Severity +from automation_file.logging_config import file_automation_logger + +EventHandler = Callable[[Event], object] +EventFilter = type[Event] | str +_DEFAULT_HISTORY = 500 +_PREFIX_SUFFIX = ".*" + + +def _matches_one(event: Event, wanted: EventFilter) -> bool: + if isinstance(wanted, str): + if wanted.endswith(_PREFIX_SUFFIX): + return event.type.startswith(wanted[: -len(_PREFIX_SUFFIX)] + ".") + return event.type == wanted + return isinstance(event, wanted) + + +@dataclass(frozen=True) +class Subscription: + """A handle returned by :meth:`EventBus.subscribe`; pass it to ``unsubscribe``.""" + + handler: EventHandler + types: tuple[EventFilter, ...] = () + min_severity: Severity = Severity.INFO + + def matches(self, event: Event) -> bool: + if not event.severity.at_least(self.min_severity): + return False + return not self.types or any(_matches_one(event, wanted) for wanted in self.types) + + +class EventBus: + """A thread-safe, synchronous publish/subscribe hub.""" + + def __init__(self, history: int = _DEFAULT_HISTORY) -> None: + self._lock = threading.RLock() + self._subscriptions: list[Subscription] = [] + self._recent: deque[Event] = deque(maxlen=max(history, 0)) + + def subscribe( + self, + handler: EventHandler, + *, + types: Iterable[EventFilter] | EventFilter | None = None, + min_severity: Severity = Severity.INFO, + ) -> Subscription: + """Call ``handler`` for every published event that matches the filters.""" + if not callable(handler): + raise TypeError("event handler is not callable") + if types is None: + wanted: tuple[EventFilter, ...] = () + elif isinstance(types, (str, type)): + wanted = (types,) + else: + wanted = tuple(types) + subscription = Subscription(handler, wanted, Severity(min_severity)) + with self._lock: + self._subscriptions.append(subscription) + return subscription + + def unsubscribe(self, subscription: Subscription) -> bool: + """Remove a subscription; return whether it was registered.""" + with self._lock: + try: + self._subscriptions.remove(subscription) + except ValueError: + return False + return True + + def publish(self, event: Event) -> int: + """Deliver ``event`` and return how many subscribers received it.""" + with self._lock: + self._recent.append(event) + targets = [entry for entry in self._subscriptions if entry.matches(event)] + delivered = 0 + for subscription in targets: + try: + subscription.handler(event) + except Exception as error: # pylint: disable=broad-except + # Boundary: a subscriber's failure must not reach the publisher. + file_automation_logger.error( + "event bus: subscriber %r failed on %s: %r", + getattr(subscription.handler, "__qualname__", subscription.handler), + event.type, + error, + ) + else: + delivered += 1 + return delivered + + def recent( + self, + limit: int = 100, + *, + types: Iterable[EventFilter] | EventFilter | None = None, + min_severity: Severity = Severity.INFO, + correlation_id: str | None = None, + ) -> list[Event]: + """Return up to ``limit`` of the latest events, newest first.""" + probe = Subscription(_ignore, _as_filters(types), Severity(min_severity)) + with self._lock: + events = list(self._recent) + chosen = [ + event + for event in reversed(events) + if probe.matches(event) + and (correlation_id is None or event.correlation_id == correlation_id) + ] + return chosen[: max(limit, 0)] + + def clear(self) -> None: + """Drop every subscription and the remembered events.""" + with self._lock: + self._subscriptions.clear() + self._recent.clear() + + +def _ignore(_event: Event) -> None: + return None + + +def _as_filters(types: Iterable[EventFilter] | EventFilter | None) -> tuple[EventFilter, ...]: + if types is None: + return () + if isinstance(types, (str, type)): + return (types,) + return tuple(types) + + +event_bus: EventBus = EventBus() + + +def emit(event: Event) -> int: + """Publish ``event`` on the process-wide :data:`event_bus`.""" + return event_bus.publish(event) diff --git a/automation_file/events/context.py b/automation_file/events/context.py new file mode 100644 index 0000000..57944e4 --- /dev/null +++ b/automation_file/events/context.py @@ -0,0 +1,67 @@ +"""Who is acting and which run an event belongs to, carried through ``contextvars``. + +A pipeline run, a scheduler firing or a server request opens a scope; everything +that happens inside it -- events, audit records, storage operations -- picks up +the same correlation ID and actor without passing them through every call. +Threads started by ``concurrent.futures`` do not inherit a context on their own: +the code that fans work out re-enters the scope in each worker. +""" + +from __future__ import annotations + +import contextlib +import getpass +import uuid +from collections.abc import Iterator +from contextvars import ContextVar + +_UNKNOWN_ACTOR = "unknown" +_correlation_id: ContextVar[str | None] = ContextVar("fa_correlation_id", default=None) +_actor: ContextVar[str | None] = ContextVar("fa_actor", default=None) + + +def new_correlation_id() -> str: + """Return a fresh correlation ID (32 hex characters).""" + return uuid.uuid4().hex + + +def current_correlation_id() -> str | None: + """Return the correlation ID of the enclosing scope, or ``None`` outside any scope.""" + return _correlation_id.get() + + +def _process_user() -> str: + try: + return getpass.getuser() + except (OSError, KeyError, ImportError): + return _UNKNOWN_ACTOR + + +def current_actor() -> str: + """Return the actor of the enclosing scope; the process's user outside any scope.""" + return _actor.get() or _process_user() + + +@contextlib.contextmanager +def correlation_scope(correlation_id: str | None = None) -> Iterator[str]: + """Run a block under one correlation ID and yield it. + + Without an argument the enclosing scope's ID is kept, or a new one is made + when there is none, so nested scopes share the outermost run's ID. + """ + chosen = correlation_id or _correlation_id.get() or new_correlation_id() + token = _correlation_id.set(chosen) + try: + yield chosen + finally: + _correlation_id.reset(token) + + +@contextlib.contextmanager +def actor_scope(actor: str) -> Iterator[str]: + """Run a block on behalf of ``actor`` (a user, ``"scheduler"``, ``"mcp"`` ...).""" + token = _actor.set(actor) + try: + yield actor + finally: + _actor.reset(token) diff --git a/automation_file/events/model.py b/automation_file/events/model.py new file mode 100644 index 0000000..dab948a --- /dev/null +++ b/automation_file/events/model.py @@ -0,0 +1,174 @@ +"""The event model: what happened, how bad it is, and which run it belongs to. + +Every component reports through :class:`Event` objects instead of calling a +notification sink or the audit log itself. An event is frozen and JSON-friendly +(:meth:`Event.to_dict`), carries a severity and a correlation ID, and keeps its +details in ``payload`` under the keys listed in :data:`PAYLOAD_KEYS`. + +The ten core events are subclasses that fix the ``type`` and the default +severity (which a caller may still override). ``SystemErrorEvent`` is the roadmap's "SystemError"; the shorter name +would shadow Python's builtin exception. +""" + +from __future__ import annotations + +import uuid +from collections.abc import Mapping +from dataclasses import dataclass, field +from datetime import datetime, timezone +from enum import Enum +from typing import Any, ClassVar + +from automation_file.events.context import ( + current_actor, + current_correlation_id, + new_correlation_id, +) + +#: Conventional ``payload`` keys, so consumers can rely on them across emitters. +PAYLOAD_KEYS = ( + "pipeline", # pipeline name + "run_id", # one execution of a pipeline + "task", # task ID inside the pipeline + "attempt", # 1-based attempt number of a task + "action", # FA_* action or operation name + "resource", # storage URI or other target + "backend", # storage scheme + "status", # outcome word of the emitter + "duration_ms", + "error", # ": " + "job", # scheduler job name + "trigger", # what fired a run +) + + +class Severity(str, Enum): + """How much attention an event needs, in rising order.""" + + INFO = "info" + WARNING = "warning" + ERROR = "error" + CRITICAL = "critical" + + @property + def rank(self) -> int: + return _SEVERITY_ORDER.index(self) + + def at_least(self, other: Severity) -> bool: + """Return whether this severity is ``other`` or worse.""" + return self.rank >= other.rank + + +_SEVERITY_ORDER = (Severity.INFO, Severity.WARNING, Severity.ERROR, Severity.CRITICAL) + + +def _now() -> datetime: + return datetime.now(timezone.utc) + + +def _event_id() -> str: + return uuid.uuid4().hex + + +def _correlation() -> str: + return current_correlation_id() or new_correlation_id() + + +@dataclass(frozen=True, kw_only=True) +class Event: + """Something that happened. Subclasses name the kind; ``payload`` holds the details.""" + + type: ClassVar[str] = "event" + + source: str = "" + subject: str = "" + severity: Severity = Severity.INFO + payload: Mapping[str, Any] = field(default_factory=dict, hash=False) + correlation_id: str = field(default_factory=_correlation) + actor: str = field(default_factory=current_actor) + id: str = field(default_factory=_event_id) + timestamp: datetime = field(default_factory=_now) + + def to_dict(self) -> dict[str, Any]: + """Return a JSON-serialisable mapping of the event.""" + return { + "id": self.id, + "type": self.type, + "severity": self.severity.value, + "source": self.source, + "subject": self.subject, + "correlation_id": self.correlation_id, + "actor": self.actor, + "timestamp": self.timestamp.isoformat(), + "payload": dict(self.payload), + } + + +@dataclass(frozen=True, kw_only=True) +class PipelineStarted(Event): + type: ClassVar[str] = "pipeline.started" + + +@dataclass(frozen=True, kw_only=True) +class PipelineCompleted(Event): + type: ClassVar[str] = "pipeline.completed" + + +@dataclass(frozen=True, kw_only=True) +class PipelineFailed(Event): + type: ClassVar[str] = "pipeline.failed" + severity: Severity = Severity.ERROR + + +@dataclass(frozen=True, kw_only=True) +class TaskStarted(Event): + type: ClassVar[str] = "task.started" + + +@dataclass(frozen=True, kw_only=True) +class TaskCompleted(Event): + type: ClassVar[str] = "task.completed" + + +@dataclass(frozen=True, kw_only=True) +class TaskFailed(Event): + type: ClassVar[str] = "task.failed" + severity: Severity = Severity.ERROR + + +@dataclass(frozen=True, kw_only=True) +class IntegrityViolation(Event): + type: ClassVar[str] = "integrity.violation" + severity: Severity = Severity.ERROR + + +@dataclass(frozen=True, kw_only=True) +class StorageError(Event): + type: ClassVar[str] = "storage.error" + severity: Severity = Severity.ERROR + + +@dataclass(frozen=True, kw_only=True) +class SchedulerError(Event): + type: ClassVar[str] = "scheduler.error" + severity: Severity = Severity.ERROR + + +@dataclass(frozen=True, kw_only=True) +class SystemErrorEvent(Event): + type: ClassVar[str] = "system.error" + severity: Severity = Severity.CRITICAL + + +CORE_EVENTS: tuple[type[Event], ...] = ( + PipelineStarted, + PipelineCompleted, + PipelineFailed, + TaskStarted, + TaskCompleted, + TaskFailed, + IntegrityViolation, + StorageError, + SchedulerError, + SystemErrorEvent, +) diff --git a/automation_file/events/storage_bridge.py b/automation_file/events/storage_bridge.py new file mode 100644 index 0000000..fc534d4 --- /dev/null +++ b/automation_file/events/storage_bridge.py @@ -0,0 +1,66 @@ +"""Turn failed storage operations into :class:`StorageError` events. + +Only failures of the storage itself are reported: a denied, unavailable or +transiently failing backend, or an error the backend could not classify. A +missing file, an existing target or a malformed URI is the caller's mistake and +raises to the caller without an event. +""" + +from __future__ import annotations + +from automation_file.events.bus import EventBus, event_bus +from automation_file.events.model import StorageError +from automation_file.storage import observe + +#: Exception type names that describe the caller's request, not the storage's health. +_CALLER_ERRORS = frozenset( + { + "StorageNotFoundException", + "StorageAlreadyExistsException", + "StoragePathTypeException", + "StorageNotEmptyException", + "StorageURIException", + "StorageUnsupportedException", + "PathTraversalException", + } +) +_SOURCE = "storage" + + +class StorageErrorBridge: + """A storage observer that publishes on one :class:`EventBus`.""" + + def __init__(self, bus: EventBus) -> None: + self._bus = bus + + def __call__(self, operation: observe.StorageOperation) -> None: + if operation.ok or operation.error_type in _CALLER_ERRORS: + return + self._bus.publish( + StorageError( + source=_SOURCE, + subject=f"{operation.operation} failed: {operation.uri}", + payload={ + "action": operation.operation, + "resource": operation.uri, + "backend": operation.backend, + "status": operation.status, + "duration_ms": operation.duration_ms, + "error": operation.error, + "error_type": operation.error_type, + }, + ) + ) + + +_default_bridge = StorageErrorBridge(event_bus) + + +def install_storage_bridge() -> None: + """Report storage failures on the process-wide bus. Calling it again changes nothing.""" + observe.add_listener(_default_bridge) + + +def uninstall_storage_bridge() -> bool: + """Stop reporting storage failures on the process-wide bus.""" + return observe.remove_listener(_default_bridge) diff --git a/automation_file/storage/backend.py b/automation_file/storage/backend.py index 1eb771a..b3241a0 100644 --- a/automation_file/storage/backend.py +++ b/automation_file/storage/backend.py @@ -16,14 +16,16 @@ from __future__ import annotations +import contextlib import hashlib import mimetypes import os import shutil import tempfile +import time import uuid from abc import ABC, abstractmethod -from collections.abc import Iterable +from collections.abc import Iterable, Iterator from pathlib import Path, PurePosixPath from types import TracebackType from typing import BinaryIO, TypeVar @@ -37,6 +39,7 @@ StoragePathTypeException, StorageUnsupportedException, ) +from automation_file.storage import observe from automation_file.storage.streams import StagedReader, StagedWriter, new_scratch_file from automation_file.storage.types import Checksum, FileInfo, StorageCapabilities from automation_file.storage.uri import normalize_path @@ -254,18 +257,19 @@ def mkdir(self, path: str, *, parents: bool = True, exist_ok: bool = True) -> No Where directories are only implied by file paths (``capabilities.directories`` is false) nothing is created and nothing needs to be. """ - clean = self._normalize(path) - info = self._stat(clean) - if info is not None: - if not info.is_dir: - raise StoragePathTypeException(f"{self.uri_for(clean)} is a file") - if not exist_ok: - raise StorageAlreadyExistsException(f"{self.uri_for(clean)} already exists") - return - if not self.capabilities.directories: - return - self._make_parents(clean, create=parents) - self._mkdir(clean) + with self._observing("mkdir", path): + clean = self._normalize(path) + info = self._stat(clean) + if info is not None: + if not info.is_dir: + raise StoragePathTypeException(f"{self.uri_for(clean)} is a file") + if not exist_ok: + raise StorageAlreadyExistsException(f"{self.uri_for(clean)} already exists") + return + if not self.capabilities.directories: + return + self._make_parents(clean, create=parents) + self._mkdir(clean) def upload( self, local_path: str | os.PathLike[str], path: str, *, overwrite: bool = True @@ -274,13 +278,14 @@ def upload( Missing parent directories are created. """ - source = Path(local_path) - if not source.is_file(): - raise StorageNotFoundException(f"local source is not a file: {source}") - clean = self._writable_file(path, overwrite) - self._make_parents(clean) - self._upload(source, clean) - return self.stat(clean) + with self._observing("upload", path): + source = Path(local_path) + if not source.is_file(): + raise StorageNotFoundException(f"local source is not a file: {source}") + clean = self._writable_file(path, overwrite) + self._make_parents(clean) + self._upload(source, clean) + return self.stat(clean) def download( self, path: str, local_path: str | os.PathLike[str], *, overwrite: bool = True @@ -290,20 +295,21 @@ def download( The content lands in a sibling ``.part`` file that replaces the target once complete, so a failed download never leaves a truncated target behind. """ - clean = self._existing_file(path) - target = Path(local_path) - if target.is_dir(): - raise StoragePathTypeException(f"local target is a directory: {target}") - if not overwrite and target.exists(): - raise StorageAlreadyExistsException(f"local target already exists: {target}") - target.parent.mkdir(parents=True, exist_ok=True) - partial = target.with_name(f".{target.name}.{uuid.uuid4().hex}.part") - try: - self._download(clean, partial) - os.replace(partial, target) - finally: - partial.unlink(missing_ok=True) - return target + with self._observing("download", path): + clean = self._existing_file(path) + target = Path(local_path) + if target.is_dir(): + raise StoragePathTypeException(f"local target is a directory: {target}") + if not overwrite and target.exists(): + raise StorageAlreadyExistsException(f"local target already exists: {target}") + target.parent.mkdir(parents=True, exist_ok=True) + partial = target.with_name(f".{target.name}.{uuid.uuid4().hex}.part") + try: + self._download(clean, partial) + os.replace(partial, target) + finally: + partial.unlink(missing_ok=True) + return target def delete(self, path: str, *, recursive: bool = False, missing_ok: bool = False) -> None: """Remove the file or directory at ``path``. @@ -311,20 +317,21 @@ def delete(self, path: str, *, recursive: bool = False, missing_ok: bool = False A directory with entries needs ``recursive=True``. The storage root is never removed. """ - clean = self._normalize(path) - info = self._stat(clean) - if info is None: - if missing_ok: + with self._observing("delete", path): + clean = self._normalize(path) + info = self._stat(clean) + if info is None: + if missing_ok: + return + raise missing_error(self.uri_for(clean)) + if not info.is_dir: + self._delete_file(clean) return - raise missing_error(self.uri_for(clean)) - if not info.is_dir: - self._delete_file(clean) - return - if self._is_root(clean): - raise StorageUnsupportedException( - f"refusing to delete the storage root {self.uri_for(clean)}" - ) - self._delete_directory(clean, recursive) + if self._is_root(clean): + raise StorageUnsupportedException( + f"refusing to delete the storage root {self.uri_for(clean)}" + ) + self._delete_directory(clean, recursive) def checksum(self, path: str, algorithm: str = DEFAULT_CHECKSUM_ALGORITHM) -> Checksum: """Return the :class:`Checksum` of the file ``path`` (SHA-256 by default).""" @@ -333,14 +340,16 @@ def checksum(self, path: str, algorithm: str = DEFAULT_CHECKSUM_ALGORITHM) -> Ch def read_bytes(self, path: str) -> bytes: """Return the whole content of the file ``path``.""" - return self._read_bytes(self._existing_file(path)) + with self._observing("read", path): + return self._read_bytes(self._existing_file(path)) def open_read(self, path: str) -> BinaryIO: """Return a binary file object over the file ``path``; close it when done. Use it for a file too large to hold in memory with :meth:`read_bytes`. """ - return self._open_read(self._existing_file(path)) + with self._observing("read", path): + return self._open_read(self._existing_file(path)) def open_write(self, path: str, *, overwrite: bool = True) -> BinaryIO: """Return a binary file object whose content becomes the file ``path`` on close. @@ -366,19 +375,21 @@ def copy_from( The copy is native when the two backends can do it between themselves and goes through a local staging file otherwise. """ - origin, target = self._transfer_paths(source, source_path, path, overwrite) - self._pull(source, origin, target) - return self.stat(target) + with self._observing("copy", path, (source, source_path)), observe.suppressed(): + origin, target = self._transfer_paths(source, source_path, path, overwrite) + self._pull(source, origin, target) + return self.stat(target) def move_from( self, source: StorageBackend, source_path: str, path: str, *, overwrite: bool = True ) -> FileInfo: """Move the file ``source_path`` of ``source`` to ``path``: a rename, or copy then delete.""" - origin, target = self._transfer_paths(source, source_path, path, overwrite) - if not self._move_from(source, origin, target): - self._pull(source, origin, target) - source.delete(origin) - return self.stat(target) + with self._observing("move", path, (source, source_path)), observe.suppressed(): + origin, target = self._transfer_paths(source, source_path, path, overwrite) + if not self._move_from(source, origin, target): + self._pull(source, origin, target) + source.delete(origin) + return self.stat(target) def close(self) -> None: # noqa: B027 - optional hook: most backends hold nothing open """Release what the backend holds open. The default holds nothing.""" @@ -396,6 +407,42 @@ def __exit__( # ------------------------------------------------------------------ shared steps + def _display_uri(self, path: str) -> str: + """Return the URI of ``path`` for a report, even when the path itself is invalid.""" + try: + return self.uri_for(path) + except StorageException: + return f"{self.scheme}:{path}" + + @contextlib.contextmanager + def _observing( + self, operation: str, path: str, origin: tuple[StorageBackend, str] | None = None + ) -> Iterator[None]: + """Report the enclosed operation, with its outcome and duration, to the observers.""" + if not observe.has_listeners(): + yield + return + started = time.perf_counter() + failure: BaseException | None = None + try: + yield + except BaseException as error: + failure = error + raise + finally: + observe.notify( + observe.StorageOperation( + operation=operation, + uri=self._display_uri(path), + backend=self.scheme, + status=observe.STATUS_ERROR if failure else observe.STATUS_OK, + duration_ms=(time.perf_counter() - started) * 1000.0, + source_uri=origin[0]._display_uri(origin[1]) if origin else None, + error=f"{type(failure).__name__}: {failure}" if failure else None, + error_type=type(failure).__name__ if failure else None, + ) + ) + def _existing_file(self, path: str) -> str: clean = self._normalize(path) if self.stat(clean).is_dir: diff --git a/automation_file/storage/observe.py b/automation_file/storage/observe.py new file mode 100644 index 0000000..cb17807 --- /dev/null +++ b/automation_file/storage/observe.py @@ -0,0 +1,111 @@ +"""Listeners for what the storage layer does. + +Every backend reports its file operations -- ``upload``, ``download``, ``read``, +``delete``, ``mkdir``, ``copy`` and ``move`` -- to the listeners registered here, +with the outcome and the duration. The audit trail and the event bridge are +listeners; the storage layer itself knows nothing about either. + +Lookups (``exists``, ``stat``, ``list_dir``, ``checksum``) are not reported: they +change nothing and would drown the operations that matter. +""" + +from __future__ import annotations + +import contextlib +import threading +from collections.abc import Callable, Iterator +from contextvars import ContextVar +from dataclasses import dataclass +from typing import Any + +from automation_file.logging_config import file_automation_logger + +STATUS_OK = "ok" +STATUS_ERROR = "error" + + +@dataclass(frozen=True) +class StorageOperation: + """One finished storage operation.""" + + operation: str + uri: str + backend: str + status: str + duration_ms: float + source_uri: str | None = None + error: str | None = None + error_type: str | None = None + + @property + def ok(self) -> bool: + return self.status == STATUS_OK + + def to_dict(self) -> dict[str, Any]: + return { + "operation": self.operation, + "uri": self.uri, + "backend": self.backend, + "status": self.status, + "duration_ms": self.duration_ms, + "source_uri": self.source_uri, + "error": self.error, + "error_type": self.error_type, + } + + +StorageListener = Callable[[StorageOperation], object] + +_lock = threading.Lock() +_listeners: list[StorageListener] = [] +_suppressed: ContextVar[bool] = ContextVar("fa_storage_observe_suppressed", default=False) + + +def add_listener(listener: StorageListener) -> None: + """Call ``listener`` after every storage operation. Adding it twice has no effect.""" + if not callable(listener): + raise TypeError("storage listener is not callable") + with _lock: + if listener not in _listeners: + _listeners.append(listener) + + +def remove_listener(listener: StorageListener) -> bool: + """Stop calling ``listener``; return whether it was registered.""" + with _lock: + try: + _listeners.remove(listener) + except ValueError: + return False + return True + + +def has_listeners() -> bool: + """Return whether an operation starting now should be reported.""" + return bool(_listeners) and not _suppressed.get() + + +@contextlib.contextmanager +def suppressed() -> Iterator[None]: + """Do not report the operations started inside the block. + + A copy or a move is one operation to an observer, although it is carried out + as a download, an upload and a delete. + """ + token = _suppressed.set(True) + try: + yield + finally: + _suppressed.reset(token) + + +def notify(operation: StorageOperation) -> None: + """Hand ``operation`` to every listener; a listener that raises is logged and skipped.""" + with _lock: + targets = list(_listeners) + for listener in targets: + try: + listener(operation) + except Exception as error: # pylint: disable=broad-except + # Boundary: an observer's failure must not fail the storage operation. + file_automation_logger.error("storage observer %r failed: %r", listener, error) diff --git a/docs/source/API/api_index.rst b/docs/source/API/api_index.rst index 43c3feb..f01e25c 100644 --- a/docs/source/API/api_index.rst +++ b/docs/source/API/api_index.rst @@ -185,4 +185,18 @@ resolver, and the built-in local and in-memory backends. :maxdepth: 2 :caption: Universal Storage Layer - storage \ No newline at end of file + storage + +.. _api-events: + +Chapter N — Events +================== + +The event model, the event bus, the correlation and actor scopes, and the +storage observers. + +.. toctree:: + :maxdepth: 2 + :caption: Events + + events diff --git a/docs/source/API/events.rst b/docs/source/API/events.rst new file mode 100644 index 0000000..b14e7bc --- /dev/null +++ b/docs/source/API/events.rst @@ -0,0 +1,20 @@ +Events +====== + +.. automodule:: automation_file.events.model + :members: + +.. automodule:: automation_file.events.bus + :members: + +.. automodule:: automation_file.events.context + :members: + +Storage observers +----------------- + +.. automodule:: automation_file.storage.observe + :members: + +.. automodule:: automation_file.events.storage_bridge + :members: diff --git a/docs/source/Eng/eng_index.rst b/docs/source/Eng/eng_index.rst index 71100e8..d6e0e24 100644 --- a/docs/source/Eng/eng_index.rst +++ b/docs/source/Eng/eng_index.rst @@ -256,4 +256,19 @@ shared operations, shared errors and a reusable contract test suite. :maxdepth: 2 :caption: Universal Storage Layer - usage/storage \ No newline at end of file + usage/storage + +.. _eng-event-bus: + +Chapter 17 — Events +=================== + +One event model for every component: ``Event``, the ten core events, the +``EventBus`` that subscribers listen on, correlation IDs and actors, and the +observers that report storage operations. + +.. toctree:: + :maxdepth: 2 + :caption: Events + + usage/event_bus diff --git a/docs/source/Eng/usage/event_bus.rst b/docs/source/Eng/usage/event_bus.rst new file mode 100644 index 0000000..77535aa --- /dev/null +++ b/docs/source/Eng/usage/event_bus.rst @@ -0,0 +1,143 @@ +Events +====== + +Every component reports what happened as an :class:`~automation_file.Event` on +the :data:`~automation_file.event_bus`. Consumers subscribe to the bus; nothing +calls a notification sink or writes an audit row directly. + +.. code-block:: python + + from automation_file import PipelineFailed, Severity, event_bus + + def page_someone(event): + print(event.severity.value, event.subject, event.payload.get("error")) + + subscription = event_bus.subscribe(page_someone, min_severity=Severity.ERROR) + event_bus.subscribe(print, types=["pipeline.*", "integrity.violation"]) + event_bus.subscribe(print, types=PipelineFailed) + event_bus.recent(limit=20, min_severity=Severity.WARNING) # newest first + event_bus.unsubscribe(subscription) + +The event +--------- + +An event is frozen and JSON-friendly (``event.to_dict()``). + +.. list-table:: + :header-rows: 1 + :widths: 24 76 + + * - Field + - Meaning + * - ``type`` + - The kind, as a dotted name: ``pipeline.failed``. + * - ``severity`` + - ``Severity.INFO``, ``WARNING``, ``ERROR`` or ``CRITICAL``. Each event class + has a default; the emitter may override it. + * - ``source`` + - The component that reports: ``pipeline``, ``integrity``, ``scheduler``, + ``storage``, ``system``. + * - ``subject`` + - One line for a human. + * - ``payload`` + - The details, under conventional keys: ``pipeline``, ``run_id``, ``task``, + ``attempt``, ``action``, ``resource``, ``backend``, ``status``, + ``duration_ms``, ``error``, ``job``, ``trigger``. + * - ``correlation_id`` + - Ties together everything that belongs to one run. + * - ``actor`` + - On whose behalf it happened. + * - ``id``, ``timestamp`` + - A unique ID and the UTC time. + +Core events +----------- + +.. list-table:: + :header-rows: 1 + :widths: 34 30 36 + + * - Class + - ``type`` + - Default severity + * - ``PipelineStarted`` + - ``pipeline.started`` + - info + * - ``PipelineCompleted`` + - ``pipeline.completed`` + - info + * - ``PipelineFailed`` + - ``pipeline.failed`` + - error + * - ``TaskStarted`` + - ``task.started`` + - info + * - ``TaskCompleted`` + - ``task.completed`` + - info + * - ``TaskFailed`` + - ``task.failed`` + - error + * - ``IntegrityViolation`` + - ``integrity.violation`` + - error + * - ``StorageError`` + - ``storage.error`` + - error + * - ``SchedulerError`` + - ``scheduler.error`` + - error + * - ``SystemErrorEvent`` + - ``system.error`` + - critical + +``SystemErrorEvent`` is the roadmap's "SystemError"; the shorter name would +shadow Python's builtin exception. + +Subscribing +----------- + +``event_bus.subscribe(handler, types=None, min_severity=Severity.INFO)`` matches +by event class (subclasses included), by exact type name, or by prefix +(``"pipeline.*"``); without ``types`` the handler receives everything. +``publish`` delivers in the publisher's thread, in subscription order, and +returns how many handlers received the event. A handler that raises is logged +and skipped, so a broken consumer never reaches the code that reported the +event. Keep handlers quick; hand slow work to a queue or a thread. + +``event_bus.recent(limit, types, min_severity, correlation_id)`` returns the +latest events the bus remembers (500 by default), newest first. A private bus is +an :class:`~automation_file.EventBus`. + +Correlation and actor +--------------------- + +.. code-block:: python + + from automation_file import actor_scope, correlation_scope, emit, PipelineStarted + + with actor_scope("scheduler"), correlation_scope() as run_id: + emit(PipelineStarted(source="pipeline", subject="daily-report started", + payload={"pipeline": "daily-report", "run_id": run_id})) + ... # every event and storage operation in here carries run_id and the actor + +``correlation_scope()`` keeps the ID of an enclosing scope, so nested work shares +the outermost run's ID. Outside any scope each event gets an ID of its own, and +the actor is the user the process runs as. A scope does not follow work into +another thread by itself; the code that fans work out re-enters it there. + +Storage operations +------------------ + +The storage layer reports ``upload``, ``download``, ``read``, ``delete``, +``mkdir``, ``copy`` and ``move`` to the listeners registered with +``automation_file.storage.observe.add_listener``: one +``StorageOperation(operation, uri, backend, status, duration_ms, source_uri, +error, error_type)`` per call, whether it succeeded or not. A copy or a move is +one operation, although it may be carried out as a download and an upload. +Lookups (``exists``, ``stat``, ``list_dir``, ``checksum``) are not reported. + +A built-in listener publishes a ``StorageError`` event when the storage itself +fails: access denied, the backend unavailable, a transient failure, or an error +the backend could not classify. A missing file, an existing target or a +malformed URI is the caller's mistake: it raises to the caller without an event. diff --git a/docs/source/Zh-CN/usage/event_bus.rst b/docs/source/Zh-CN/usage/event_bus.rst new file mode 100644 index 0000000..9a450ec --- /dev/null +++ b/docs/source/Zh-CN/usage/event_bus.rst @@ -0,0 +1,139 @@ +事件 +==== + +每个组件都以 :class:`~automation_file.Event` 的形式，把发生的事报告到 +:data:`~automation_file.event_bus`。使用方订阅事件总线即可；没有任何组件会直接 +调用通知接收端或写入审计记录。 + +.. code-block:: python + + from automation_file import PipelineFailed, Severity, event_bus + + def page_someone(event): + print(event.severity.value, event.subject, event.payload.get("error")) + + subscription = event_bus.subscribe(page_someone, min_severity=Severity.ERROR) + event_bus.subscribe(print, types=["pipeline.*", "integrity.violation"]) + event_bus.subscribe(print, types=PipelineFailed) + event_bus.recent(limit=20, min_severity=Severity.WARNING) # 最新的在前 + event_bus.unsubscribe(subscription) + +事件本身 +-------- + +事件不可变，并且可以直接转成 JSON（``event.to_dict()``）。 + +.. list-table:: + :header-rows: 1 + :widths: 24 76 + + * - 字段 + - 含义 + * - ``type`` + - 事件种类，以点分隔的名称表示：``pipeline.failed``。 + * - ``severity`` + - ``Severity.INFO``、``WARNING``、``ERROR`` 或 ``CRITICAL``。每个事件类都有 + 默认值，发出者可以覆盖。 + * - ``source`` + - 报告的组件：``pipeline``、``integrity``、``scheduler``、``storage``、 + ``system``。 + * - ``subject`` + - 给人看的一行摘要。 + * - ``payload`` + - 细节，使用约定的键：``pipeline``、``run_id``、``task``、``attempt``、 + ``action``、``resource``、``backend``、``status``、``duration_ms``、 + ``error``、``job``、``trigger``。 + * - ``correlation_id`` + - 把属于同一次运行的所有事物串在一起。 + * - ``actor`` + - 这件事是代表谁执行的。 + * - ``id``、``timestamp`` + - 唯一 ID 与 UTC 时间。 + +核心事件 +-------- + +.. list-table:: + :header-rows: 1 + :widths: 34 30 36 + + * - 类 + - ``type`` + - 默认严重程度 + * - ``PipelineStarted`` + - ``pipeline.started`` + - info + * - ``PipelineCompleted`` + - ``pipeline.completed`` + - info + * - ``PipelineFailed`` + - ``pipeline.failed`` + - error + * - ``TaskStarted`` + - ``task.started`` + - info + * - ``TaskCompleted`` + - ``task.completed`` + - info + * - ``TaskFailed`` + - ``task.failed`` + - error + * - ``IntegrityViolation`` + - ``integrity.violation`` + - error + * - ``StorageError`` + - ``storage.error`` + - error + * - ``SchedulerError`` + - ``scheduler.error`` + - error + * - ``SystemErrorEvent`` + - ``system.error`` + - critical + +``SystemErrorEvent`` 就是路线图中的“SystemError”；较短的名称会遮蔽 Python 内置的 +异常。 + +订阅 +---- + +``event_bus.subscribe(handler, types=None, min_severity=Severity.INFO)`` 可以按事件 +类（包含子类）、完整的 type 名称或前缀（``"pipeline.*"``）匹配；不指定 ``types`` +时，处理函数会收到所有事件。``publish`` 在发布者的线程中，按订阅顺序逐一传递，并 +返回收到事件的处理函数数量。处理函数如果抛出异常，只会被记录并跳过，因此有问题的 +使用方永远不会影响报告事件的代码。处理函数应保持快速；耗时的工作请交给队列或 +线程。 + +``event_bus.recent(limit, types, min_severity, correlation_id)`` 返回总线记得的 +最近事件（默认 500 条），最新的在前。需要私有的总线时使用 +:class:`~automation_file.EventBus`。 + +关联 ID 与 actor +---------------- + +.. code-block:: python + + from automation_file import actor_scope, correlation_scope, emit, PipelineStarted + + with actor_scope("scheduler"), correlation_scope() as run_id: + emit(PipelineStarted(source="pipeline", subject="daily-report started", + payload={"pipeline": "daily-report", "run_id": run_id})) + ... # 这里面的每个事件与存储操作都带有 run_id 与 actor + +``correlation_scope()`` 会沿用外层范围的 ID，因此嵌套的工作共用最外层那次运行的 +ID。在任何范围之外，每个事件都有自己的 ID，actor 则是运行进程的用户。范围不会 +自动跟着工作进入另一个线程；把工作分派出去的代码要在那里重新进入范围。 + +存储操作 +-------- + +存储层会把 ``upload``、``download``、``read``、``delete``、``mkdir``、``copy`` 与 +``move`` 报告给通过 ``automation_file.storage.observe.add_listener`` 注册的监听者： +每次调用一条 ``StorageOperation(operation, uri, backend, status, duration_ms, +source_uri, error, error_type)``，无论成功还是失败。复制或移动算作一条操作，即使它 +实际上是由一次下载与一次上传完成。查询类操作（``exists``、``stat``、 +``list_dir``、``checksum``）不会报告。 + +内置的监听者会在存储本身失败时发布 ``StorageError`` 事件：访问被拒、后端不可用、 +暂时性失败，或后端无法分类的错误。文件不存在、目标已存在或 URI 格式错误属于调用方 +的错误：只会抛给调用方，不会产生事件。 diff --git a/docs/source/Zh-CN/zh_cn_index.rst b/docs/source/Zh-CN/zh_cn_index.rst index 49001ee..d901a28 100644 --- a/docs/source/Zh-CN/zh_cn_index.rst +++ b/docs/source/Zh-CN/zh_cn_index.rst @@ -251,4 +251,18 @@ Slack、Email（SMTP）、Discord、Telegram、Microsoft Teams、PagerDuty :maxdepth: 2 :caption: 通用存储层 - usage/storage \ No newline at end of file + usage/storage + +.. _zh-cn-event-bus: + +第 17 章 — 事件 +=============== + +所有组件共用的事件模型：``Event``、十种核心事件、供订阅者监听的 +``EventBus``、关联 ID 与 actor，以及报告存储操作的观察者。 + +.. toctree:: + :maxdepth: 2 + :caption: 事件 + + usage/event_bus diff --git a/docs/source/Zh-TW/usage/event_bus.rst b/docs/source/Zh-TW/usage/event_bus.rst new file mode 100644 index 0000000..95ee84d --- /dev/null +++ b/docs/source/Zh-TW/usage/event_bus.rst @@ -0,0 +1,139 @@ +事件 +==== + +每個元件都以 :class:`~automation_file.Event` 的形式，把發生的事回報到 +:data:`~automation_file.event_bus`。使用端訂閱事件匯流排即可；沒有任何元件會直接 +呼叫通知接收端或寫入稽核紀錄。 + +.. code-block:: python + + from automation_file import PipelineFailed, Severity, event_bus + + def page_someone(event): + print(event.severity.value, event.subject, event.payload.get("error")) + + subscription = event_bus.subscribe(page_someone, min_severity=Severity.ERROR) + event_bus.subscribe(print, types=["pipeline.*", "integrity.violation"]) + event_bus.subscribe(print, types=PipelineFailed) + event_bus.recent(limit=20, min_severity=Severity.WARNING) # 最新的在前 + event_bus.unsubscribe(subscription) + +事件本身 +-------- + +事件不可變，且可直接轉成 JSON（``event.to_dict()``）。 + +.. list-table:: + :header-rows: 1 + :widths: 24 76 + + * - 欄位 + - 意義 + * - ``type`` + - 事件種類，以點分隔的名稱表示：``pipeline.failed``。 + * - ``severity`` + - ``Severity.INFO``、``WARNING``、``ERROR`` 或 ``CRITICAL``。每個事件類別都有 + 預設值，發出者可以覆寫。 + * - ``source`` + - 回報的元件：``pipeline``、``integrity``、``scheduler``、``storage``、 + ``system``。 + * - ``subject`` + - 給人看的一行摘要。 + * - ``payload`` + - 細節，使用約定的鍵：``pipeline``、``run_id``、``task``、``attempt``、 + ``action``、``resource``、``backend``、``status``、``duration_ms``、 + ``error``、``job``、``trigger``。 + * - ``correlation_id`` + - 把屬於同一次執行的所有事物串在一起。 + * - ``actor`` + - 這件事是代表誰執行的。 + * - ``id``、``timestamp`` + - 唯一 ID 與 UTC 時間。 + +核心事件 +-------- + +.. list-table:: + :header-rows: 1 + :widths: 34 30 36 + + * - 類別 + - ``type`` + - 預設嚴重程度 + * - ``PipelineStarted`` + - ``pipeline.started`` + - info + * - ``PipelineCompleted`` + - ``pipeline.completed`` + - info + * - ``PipelineFailed`` + - ``pipeline.failed`` + - error + * - ``TaskStarted`` + - ``task.started`` + - info + * - ``TaskCompleted`` + - ``task.completed`` + - info + * - ``TaskFailed`` + - ``task.failed`` + - error + * - ``IntegrityViolation`` + - ``integrity.violation`` + - error + * - ``StorageError`` + - ``storage.error`` + - error + * - ``SchedulerError`` + - ``scheduler.error`` + - error + * - ``SystemErrorEvent`` + - ``system.error`` + - critical + +``SystemErrorEvent`` 就是路線圖中的「SystemError」；較短的名稱會遮蔽 Python 內建的 +例外。 + +訂閱 +---- + +``event_bus.subscribe(handler, types=None, min_severity=Severity.INFO)`` 可依事件 +類別（包含子類別）、完整的 type 名稱或前綴（``"pipeline.*"``）比對；不指定 +``types`` 時，處理函式會收到所有事件。``publish`` 在發布者的執行緒中，依訂閱順序 +逐一傳遞，並回傳收到事件的處理函式數量。處理函式若拋出例外，只會被記錄並略過， +因此有問題的使用端永遠不會影響回報事件的程式。處理函式應保持快速；耗時的工作請 +交給佇列或執行緒。 + +``event_bus.recent(limit, types, min_severity, correlation_id)`` 回傳匯流排記得的 +最近事件（預設 500 筆），最新的在前。需要私有的匯流排時使用 +:class:`~automation_file.EventBus`。 + +關聯 ID 與 actor +---------------- + +.. code-block:: python + + from automation_file import actor_scope, correlation_scope, emit, PipelineStarted + + with actor_scope("scheduler"), correlation_scope() as run_id: + emit(PipelineStarted(source="pipeline", subject="daily-report started", + payload={"pipeline": "daily-report", "run_id": run_id})) + ... # 這裡面的每個事件與儲存操作都帶有 run_id 與 actor + +``correlation_scope()`` 會沿用外層範圍的 ID，因此巢狀的工作共用最外層那次執行的 +ID。在任何範圍之外，每個事件都有自己的 ID，actor 則是執行行程的使用者。範圍不會 +自動跟著工作進入另一個執行緒；把工作分派出去的程式要在那裡重新進入範圍。 + +儲存操作 +-------- + +儲存層會把 ``upload``、``download``、``read``、``delete``、``mkdir``、``copy`` 與 +``move`` 回報給以 ``automation_file.storage.observe.add_listener`` 註冊的監聽者： +每次呼叫一筆 ``StorageOperation(operation, uri, backend, status, duration_ms, +source_uri, error, error_type)``，無論成功或失敗。複製或搬移算作一筆操作，即使它 +實際上是由一次下載與一次上傳完成。查詢類操作（``exists``、``stat``、 +``list_dir``、``checksum``）不會回報。 + +內建的監聽者會在儲存本身失敗時發布 ``StorageError`` 事件：存取被拒、後端無法 +使用、暫時性失敗，或後端無法分類的錯誤。檔案不存在、目標已存在或 URI 格式錯誤 +屬於呼叫方的錯誤：只會拋給呼叫方，不會產生事件。 diff --git a/docs/source/Zh-TW/zh_tw_index.rst b/docs/source/Zh-TW/zh_tw_index.rst index 3507308..415bd30 100644 --- a/docs/source/Zh-TW/zh_tw_index.rst +++ b/docs/source/Zh-TW/zh_tw_index.rst @@ -251,4 +251,18 @@ Slack、Email（SMTP）、Discord、Telegram、Microsoft Teams、PagerDuty :maxdepth: 2 :caption: 通用儲存層 - usage/storage \ No newline at end of file + usage/storage + +.. _zh-tw-event-bus: + +第 17 章 — 事件 +=============== + +所有元件共用的事件模型：``Event``、十種核心事件、供訂閱者監聽的 +``EventBus``、關聯 ID 與 actor，以及回報儲存操作的觀察者。 + +.. toctree:: + :maxdepth: 2 + :caption: 事件 + + usage/event_bus diff --git a/docs/source/index.rst b/docs/source/index.rst index aecb821..0b4994a 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -24,7 +24,7 @@ The documentation is split by language and by content type. Each language manual is organised into chapters (Getting Started, CLI, Architecture, Local Operations, HTTP Transfers, Cloud and SFTP Backends, Action Servers, MCP Server, GUI, Reliability, Triggers and Scheduler, Notifications, -Configuration, DAG, Plugins, Universal Storage Layer); the API book holds the +Configuration, DAG, Plugins, Universal Storage Layer, Events); the API book holds the auto-generated Python reference for every public module. Pick a language from the table of contents on the left, or jump straight to a section below. diff --git a/docs/updates/2026-10.md b/docs/updates/2026-10.md index 8eeffbe..db9b702 100644 --- a/docs/updates/2026-10.md +++ b/docs/updates/2026-10.md @@ -276,3 +276,19 @@ Index and query commands: [README.md](README.md). New entries go at the end. - **Docs**: a "Streams and directory trees" section, two operation rows and two action rows in the three `usage/storage.rst` pages, `docs/source/API/storage.rst`, a bullet in the three READMEs (which now say 77 cases), `architecture.md` §2 and §3, `CLAUDE.md` (package map). - **Files**: `automation_file/storage/streams.py`, `tree.py`, `backend.py`, `local_storage.py`, `file.py`, `storage.py`, `actions.py`, `__init__.py`, `automation_file/__init__.py`, `tests/storage_contract.py`, `tests/test_storage_tree.py`, `tests/test_storage_actions.py`, the documentation above, `progress.md`. - **Open items**: `progress.md` #17 (native streams and checksums for the remote backends). + +## U-20261008-06 · 2026-10-08 · Event model, event bus and storage observers · #events #roadmap #storage + +- **What**: the foundation of `progress.md` #23 (events, notifications, audit) and of the pipeline and integrity work that will report through it. New package `automation_file/events/`: + - `Event`: frozen, keyword-only, JSON-friendly (`to_dict`). Fields `type` (a dotted name on the class), `severity`, `source`, `subject`, `payload`, `correlation_id`, `actor`, `id`, `timestamp` (aware UTC). `PAYLOAD_KEYS` lists the conventional payload keys. + - Ten core events with their type and default severity: `PipelineStarted`, `PipelineCompleted`, `PipelineFailed`, `TaskStarted`, `TaskCompleted`, `TaskFailed`, `IntegrityViolation`, `StorageError`, `SchedulerError`, `SystemErrorEvent`. The last is the roadmap's "SystemError", renamed because that name is a Python builtin. + - `Severity` (`info`, `warning`, `error`, `critical`) with `at_least`. + - `EventBus` / the process-wide `event_bus` / `emit`: synchronous delivery in the publisher's thread, in subscription order; filters by event class, type name, prefix (`pipeline.*`) and minimum severity; a subscriber that raises is logged and skipped; `recent()` returns the latest events (500 kept), newest first, optionally for one correlation ID. + - `correlation_scope` and `actor_scope` on `contextvars`: nested scopes share the outermost run's ID; outside a scope each event gets its own ID and the actor is the process's user. +- **Storage observers**: `automation_file/storage/observe.py`. `StorageBackend` reports `upload`, `download`, `read`, `delete`, `mkdir`, `copy` and `move` to registered listeners as a `StorageOperation` (URI, backend, status, duration, source URI, error and its type), on success and on failure. A copy or a move is one operation: the download, upload and delete it is made of are suppressed. Lookups are not reported. With no listener the bookkeeping is skipped. + - `events/storage_bridge.py` is such a listener, installed when `automation_file.events` is imported: a failing backend (denied, unavailable, transient, unclassified) becomes a `StorageError` event; a missing file, an existing target or a bad URI raises to the caller without one. +- **Tests**: `tests/test_events.py` (the model, severities, scopes and their isolation between threads, every bus filter, a failing subscriber, history bounds, the bridge, the facade) and `tests/test_storage_observe.py` (one report per operation, lookups, copy and move, failures, an invalid path, a failing observer, suppression): 38 cases. +- **Result / numbers**: `ruff check`, `ruff format --check` pass; `mypy automation_file` finds no issues in 181 files; `pytest tests/`: 1570 passed, 22 skipped, 0 failed, on Python 3.14.7 on Windows. +- **Docs**: an "Events" chapter in the three manuals (`usage/event_bus.rst`, chapter 17) and an API page (chapter N), a feature bullet and an "Events" section in the three READMEs, `architecture.md` §2, §3 and §4, `CLAUDE.md` (package map, key types). +- **Files**: `automation_file/events/` (5 modules), `automation_file/storage/observe.py`, `automation_file/storage/backend.py`, `automation_file/__init__.py`, `tests/test_events.py`, `tests/test_storage_observe.py`, the documentation above. +- **Open items**: `progress.md` #23 (notification router, audit schema v2, scheduler states), #21 and #22, which publish on this bus. diff --git a/docs/updates/README.md b/docs/updates/README.md index 1ddffb5..05c8ad7 100644 --- a/docs/updates/README.md +++ b/docs/updates/README.md @@ -58,6 +58,7 @@ In the same commit: delete the item from `progress.md`, add a `#done` entry here | ID | Date | Title | Tags | Batch | |---|---|---|---|---| +| U-20261008-06 | 2026-10-08 | Event model, event bus and storage observers | #events #roadmap #storage | [2026-10](2026-10.md) | | U-20261008-05 | 2026-10-08 | Streams and directory trees in the storage layer | #storage #roadmap #streams | [2026-10](2026-10.md) | | U-20261008-04 | 2026-10-08 | Version directories stay short for long source paths | #done #versioning #windows | [2026-10](2026-10.md) | | U-20261008-03 | 2026-10-08 | FA_storage_* actions put the storage layer in the registry | #storage #roadmap #actions #mcp | [2026-10](2026-10.md) | @@ -100,5 +101,5 @@ In the same commit: delete the item from `progress.md`, add a `#done` entry here | File | Period | Entries | |---|---|---:| -| [2026-10.md](2026-10.md) | 2026-10 | 16 | +| [2026-10.md](2026-10.md) | 2026-10 | 17 | | [2026-09.md](2026-09.md) | 2026-09 | 21 | diff --git a/tests/test_events.py b/tests/test_events.py new file mode 100644 index 0000000..13a240d --- /dev/null +++ b/tests/test_events.py @@ -0,0 +1,343 @@ +"""The event model, the bus, the scopes, and the storage-error bridge.""" + +from __future__ import annotations + +import dataclasses +import json +import threading +from collections.abc import Iterator +from datetime import timedelta + +import pytest + +import automation_file +from automation_file.events import ( + CORE_EVENTS, + Event, + EventBus, + IntegrityViolation, + PipelineCompleted, + PipelineFailed, + PipelineStarted, + Severity, + StorageError, + StorageErrorBridge, + SystemErrorEvent, + TaskFailed, + actor_scope, + correlation_scope, + current_actor, + current_correlation_id, + emit, + event_bus, + new_correlation_id, +) +from automation_file.exceptions import StorageNotFoundException, StoragePermissionException +from automation_file.storage import MemoryStorage, observe + + +@pytest.fixture +def bus() -> EventBus: + return EventBus() + + +# ---------------------------------------------------------------------- model + + +def test_the_ten_core_events_have_distinct_types() -> None: + assert [event.type for event in CORE_EVENTS] == [ + "pipeline.started", + "pipeline.completed", + "pipeline.failed", + "task.started", + "task.completed", + "task.failed", + "integrity.violation", + "storage.error", + "scheduler.error", + "system.error", + ] + + +@pytest.mark.parametrize( + "event_class,severity", + [ + (PipelineStarted, Severity.INFO), + (PipelineCompleted, Severity.INFO), + (PipelineFailed, Severity.ERROR), + (TaskFailed, Severity.ERROR), + (IntegrityViolation, Severity.ERROR), + (StorageError, Severity.ERROR), + (SystemErrorEvent, Severity.CRITICAL), + ], +) +def test_each_core_event_has_its_default_severity( + event_class: type[Event], severity: Severity +) -> None: + assert event_class().severity is severity + assert event_class(severity=Severity.WARNING).severity is Severity.WARNING + + +def test_an_event_fills_in_its_identity() -> None: + first, second = PipelineStarted(source="pipeline"), PipelineStarted(source="pipeline") + assert first.id != second.id + assert len(first.id) == 32 + assert first.timestamp.utcoffset() == timedelta(0) + assert first.correlation_id != second.correlation_id + assert first.actor == current_actor() + assert dict(first.payload) == {} + + +def test_an_event_is_frozen_and_keyword_only() -> None: + event = TaskFailed(subject="boom", payload={"task": "load"}) + with pytest.raises(dataclasses.FrozenInstanceError): + event.subject = "changed" # type: ignore[misc] + with pytest.raises(TypeError): + TaskFailed("positional") # type: ignore[misc] + assert hash(event) == hash(event) + + +def test_to_dict_is_json_serialisable() -> None: + event = TaskFailed( + source="pipeline", + subject="load failed", + payload={"pipeline": "daily", "task": "load", "attempt": 2, "error": "X: y"}, + correlation_id="run-1", + actor="scheduler", + ) + document = json.loads(json.dumps(event.to_dict())) + assert document["type"] == "task.failed" + assert document["severity"] == "error" + assert document["correlation_id"] == "run-1" + assert document["actor"] == "scheduler" + assert document["payload"]["attempt"] == 2 + assert document["timestamp"].endswith("+00:00") + assert document["id"] == event.id + + +def test_severity_order() -> None: + assert [severity.rank for severity in Severity] == [0, 1, 2, 3] + assert Severity.ERROR.at_least(Severity.WARNING) is True + assert Severity.ERROR.at_least(Severity.ERROR) is True + assert Severity.WARNING.at_least(Severity.ERROR) is False + assert Severity("critical") is Severity.CRITICAL + + +# ---------------------------------------------------------------------- scopes + + +def test_correlation_scope_is_shared_by_nested_scopes() -> None: + assert current_correlation_id() is None + with correlation_scope() as outer: + assert current_correlation_id() == outer + assert PipelineStarted().correlation_id == outer + with correlation_scope() as inner: + assert inner == outer + with correlation_scope("explicit") as explicit: + assert explicit == "explicit" + assert PipelineStarted().correlation_id == "explicit" + assert current_correlation_id() == outer + assert current_correlation_id() is None + assert len(new_correlation_id()) == 32 + + +def test_actor_scope() -> None: + default = current_actor() + assert default + with actor_scope("scheduler"): + assert current_actor() == "scheduler" + assert PipelineStarted().actor == "scheduler" + with actor_scope("mcp"): + assert current_actor() == "mcp" + assert current_actor() == "scheduler" + assert current_actor() == default + + +def test_scopes_do_not_leak_into_other_threads() -> None: + seen: list[str | None] = [] + with correlation_scope("main-run"): + thread = threading.Thread(target=lambda: seen.append(current_correlation_id())) + thread.start() + thread.join() + assert seen == [None] + + +# ---------------------------------------------------------------------- bus + + +def test_publish_reaches_every_matching_subscriber_in_order(bus: EventBus) -> None: + calls: list[str] = [] + bus.subscribe(lambda event: calls.append(f"all:{event.type}")) + bus.subscribe(lambda event: calls.append("class"), types=PipelineFailed) + bus.subscribe(lambda event: calls.append("name"), types="task.failed") + bus.subscribe(lambda event: calls.append("prefix"), types="pipeline.*") + bus.subscribe(lambda event: calls.append("several"), types=[TaskFailed, "storage.error"]) + assert bus.publish(PipelineFailed()) == 3 + assert calls == ["all:pipeline.failed", "class", "prefix"] + calls.clear() + assert bus.publish(TaskFailed()) == 3 + assert calls == ["all:task.failed", "name", "several"] + calls.clear() + assert bus.publish(StorageError()) == 2 + assert calls == ["all:storage.error", "several"] + + +def test_a_prefix_does_not_match_a_longer_word(bus: EventBus) -> None: + calls: list[str] = [] + bus.subscribe(lambda event: calls.append(event.type), types="task.*") + + @dataclasses.dataclass(frozen=True, kw_only=True) + class Tasklist(Event): + type = "tasklist.updated" + + bus.publish(Tasklist()) + bus.publish(TaskFailed()) + assert calls == ["task.failed"] + + +def test_min_severity_filters(bus: EventBus) -> None: + calls: list[str] = [] + bus.subscribe(lambda event: calls.append(event.type), min_severity=Severity.ERROR) + bus.publish(PipelineStarted()) + bus.publish(PipelineStarted(severity=Severity.CRITICAL)) + bus.publish(TaskFailed()) + bus.publish(TaskFailed(severity=Severity.WARNING)) + assert calls == ["pipeline.started", "task.failed"] + + +def test_a_failing_subscriber_does_not_stop_the_others(bus: EventBus) -> None: + calls: list[str] = [] + + def broken(_event: Event) -> None: + raise RuntimeError("subscriber bug") + + bus.subscribe(broken) + bus.subscribe(lambda event: calls.append(event.type)) + assert bus.publish(PipelineStarted()) == 1 + assert calls == ["pipeline.started"] + + +def test_unsubscribe(bus: EventBus) -> None: + calls: list[str] = [] + subscription = bus.subscribe(lambda event: calls.append(event.type)) + assert bus.unsubscribe(subscription) is True + assert bus.unsubscribe(subscription) is False + assert bus.publish(PipelineStarted()) == 0 + assert calls == [] + + +def test_subscribe_rejects_a_non_callable(bus: EventBus) -> None: + with pytest.raises(TypeError): + bus.subscribe("not callable") # type: ignore[arg-type] + + +def test_recent_returns_the_latest_events_newest_first(bus: EventBus) -> None: + with correlation_scope("run-1"): + started = PipelineStarted() + failed = TaskFailed() + other = StorageError() + for event in (started, failed, other): + bus.publish(event) + assert bus.recent() == [other, failed, started] + assert bus.recent(limit=1) == [other] + assert bus.recent(types="task.*") == [failed] + assert bus.recent(min_severity=Severity.ERROR) == [other, failed] + assert bus.recent(correlation_id="run-1") == [failed, started] + assert bus.recent(limit=0) == [] + + +def test_history_is_bounded() -> None: + bus = EventBus(history=3) + events = [PipelineStarted() for _ in range(5)] + for event in events: + bus.publish(event) + assert bus.recent() == list(reversed(events[2:])) + + +def test_clear_forgets_subscriptions_and_history(bus: EventBus) -> None: + calls: list[str] = [] + bus.subscribe(lambda event: calls.append(event.type)) + bus.publish(PipelineStarted()) + bus.clear() + assert bus.recent() == [] + assert bus.publish(PipelineStarted()) == 0 + assert calls == ["pipeline.started"] + + +def test_emit_publishes_on_the_process_wide_bus() -> None: + calls: list[Event] = [] + subscription = event_bus.subscribe(calls.append, types="pipeline.completed") + try: + event = PipelineCompleted(subject="done") + assert emit(event) >= 1 + finally: + event_bus.unsubscribe(subscription) + assert calls == [event] + + +# ---------------------------------------------------------------------- storage bridge + + +@pytest.fixture +def bridged(bus: EventBus) -> Iterator[list[Event]]: + bridge = StorageErrorBridge(bus) + received: list[Event] = [] + bus.subscribe(received.append) + observe.add_listener(bridge) + yield received + observe.remove_listener(bridge) + + +def test_a_storage_failure_becomes_a_storage_error_event( + bridged: list[Event], monkeypatch: pytest.MonkeyPatch +) -> None: + storage = MemoryStorage("events") + + def _deny(*_args: object) -> None: + raise StoragePermissionException("memory://events/a.txt: denied") + + monkeypatch.setattr(storage, "_upload", _deny) + with correlation_scope("run-7"), pytest.raises(StoragePermissionException): + storage.write_bytes("a.txt", b"x") + assert len(bridged) == 1 + event = bridged[0] + assert isinstance(event, StorageError) + assert event.source == "storage" + assert event.severity is Severity.ERROR + assert event.correlation_id == "run-7" + assert event.subject == "upload failed: memory://events/a.txt" + assert event.payload["resource"] == "memory://events/a.txt" + assert event.payload["backend"] == "memory" + assert event.payload["error_type"] == "StoragePermissionException" + + +def test_a_caller_mistake_raises_without_an_event(bridged: list[Event]) -> None: + storage = MemoryStorage("events") + with pytest.raises(StorageNotFoundException): + storage.read_bytes("nope.txt") + storage.write_bytes("a.txt", b"x") + assert bridged == [] + + +def test_the_facade_exports_the_events() -> None: + for name in ( + "Event", + "EventBus", + "Severity", + "event_bus", + "emit", + "correlation_scope", + "actor_scope", + "PipelineStarted", + "PipelineCompleted", + "PipelineFailed", + "TaskStarted", + "TaskCompleted", + "TaskFailed", + "IntegrityViolation", + "StorageError", + "SchedulerError", + "SystemErrorEvent", + ): + assert name in automation_file.__all__ + assert hasattr(automation_file, name) diff --git a/tests/test_storage_observe.py b/tests/test_storage_observe.py new file mode 100644 index 0000000..f326d6e --- /dev/null +++ b/tests/test_storage_observe.py @@ -0,0 +1,161 @@ +"""Storage observers: what each backend operation reports.""" + +from __future__ import annotations + +from collections.abc import Iterator +from pathlib import Path + +import pytest + +from automation_file.exceptions import ( + StorageAlreadyExistsException, + StorageNotFoundException, + StorageURIException, +) +from automation_file.storage import LocalStorage, MemoryStorage, observe +from automation_file.storage.observe import StorageOperation + + +@pytest.fixture +def seen() -> Iterator[list[StorageOperation]]: + operations: list[StorageOperation] = [] + observe.add_listener(operations.append) + yield operations + observe.remove_listener(operations.append) + + +@pytest.fixture +def storage() -> MemoryStorage: + return MemoryStorage("observed") + + +def _summary(operations: list[StorageOperation]) -> list[tuple[str, str, str]]: + return [(entry.operation, entry.uri, entry.status) for entry in operations] + + +def test_each_operation_is_reported_once( + storage: MemoryStorage, seen: list[StorageOperation], tmp_path: Path +) -> None: + storage.write_bytes("dir/a.txt", b"hello") + storage.read_bytes("dir/a.txt") + with storage.open_read("dir/a.txt") as stream: + stream.read() + storage.download("dir/a.txt", tmp_path / "a.txt") + storage.mkdir("empty") + storage.delete("empty") + with storage.open_write("dir/b.txt") as stream: + stream.write(b"x") + assert _summary(seen) == [ + ("upload", "memory://observed/dir/a.txt", "ok"), + ("read", "memory://observed/dir/a.txt", "ok"), + ("read", "memory://observed/dir/a.txt", "ok"), + ("download", "memory://observed/dir/a.txt", "ok"), + ("mkdir", "memory://observed/empty", "ok"), + ("delete", "memory://observed/empty", "ok"), + ("upload", "memory://observed/dir/b.txt", "ok"), + ] + assert all(entry.backend == "memory" and entry.ok for entry in seen) + assert all(entry.duration_ms >= 0 for entry in seen) + assert all(entry.error is None and entry.error_type is None for entry in seen) + + +def test_lookups_are_not_reported(storage: MemoryStorage, seen: list[StorageOperation]) -> None: + storage.write_bytes("a.txt", b"x") + seen.clear() + storage.exists("a.txt") + storage.stat("a.txt") + storage.list_dir("") + storage.checksum("a.txt") + assert seen == [] + + +def test_a_copy_or_a_move_is_one_operation( + storage: MemoryStorage, seen: list[StorageOperation], tmp_path: Path +) -> None: + storage.write_bytes("a.txt", b"x") + local = LocalStorage(tmp_path) + seen.clear() + local.copy_from(storage, "a.txt", "copy.txt") + storage.move_from(local, "copy.txt", "back.txt") + assert [(entry.operation, entry.backend, entry.status) for entry in seen] == [ + ("copy", "local", "ok"), + ("move", "memory", "ok"), + ] + assert seen[0].source_uri == "memory://observed/a.txt" + assert seen[0].uri == local.uri_for("copy.txt") + assert seen[1].source_uri == local.uri_for("copy.txt") + assert seen[1].uri == "memory://observed/back.txt" + + +def test_a_failure_is_reported_with_its_error( + storage: MemoryStorage, seen: list[StorageOperation] +) -> None: + with pytest.raises(StorageNotFoundException): + storage.read_bytes("nope.txt") + storage.write_bytes("a.txt", b"x") + with pytest.raises(StorageAlreadyExistsException): + storage.write_bytes("a.txt", b"y", overwrite=False) + failures = [entry for entry in seen if not entry.ok] + assert [(entry.operation, entry.error_type) for entry in failures] == [ + ("read", "StorageNotFoundException"), + ("upload", "StorageAlreadyExistsException"), + ] + assert failures[0].error is not None + assert failures[0].error.startswith("StorageNotFoundException: ") + assert failures[0].status == "error" + + +def test_an_invalid_path_is_still_reported( + storage: MemoryStorage, seen: list[StorageOperation] +) -> None: + with pytest.raises(StorageURIException): + storage.delete("../escape") + assert _summary(seen) == [("delete", "memory:../escape", "error")] + + +def test_a_failing_observer_does_not_fail_the_operation(storage: MemoryStorage) -> None: + def broken(_operation: StorageOperation) -> None: + raise RuntimeError("observer bug") + + observe.add_listener(broken) + try: + assert storage.write_bytes("a.txt", b"x").size == 1 + finally: + assert observe.remove_listener(broken) is True + assert observe.remove_listener(broken) is False + + +def test_listeners_are_registered_once_and_must_be_callable() -> None: + calls: list[StorageOperation] = [] + observe.add_listener(calls.append) + observe.add_listener(calls.append) + try: + MemoryStorage("once").write_bytes("a.txt", b"x") + finally: + observe.remove_listener(calls.append) + assert len(calls) == 1 + with pytest.raises(TypeError): + observe.add_listener("nope") # type: ignore[arg-type] + + +def test_suppressed_hides_the_operations_inside( + storage: MemoryStorage, seen: list[StorageOperation] +) -> None: + with observe.suppressed(): + storage.write_bytes("a.txt", b"x") + storage.write_bytes("b.txt", b"x") + assert _summary(seen) == [("upload", "memory://observed/b.txt", "ok")] + + +def test_to_dict() -> None: + operation = StorageOperation("upload", "memory://x/a", "memory", "ok", 1.5) + assert operation.to_dict() == { + "operation": "upload", + "uri": "memory://x/a", + "backend": "memory", + "status": "ok", + "duration_ms": 1.5, + "source_uri": None, + "error": None, + "error_type": None, + } From 810e5078265f7ae301ee1684e477101715d44933 Mon Sep 17 00:00:00 2001 From: JeffreyChen Date: Thu, 8 Oct 2026 12:32:24 +0800 Subject: [PATCH 27/59] docs: describe FA_copy_between on the cloud pages instead of a FA_cross_copy that does not exist --- docs/source/Eng/usage/cloud.rst | 22 ++++++++++++++++------ docs/source/Zh-CN/usage/cloud.rst | 19 +++++++++++++------ docs/source/Zh-TW/usage/cloud.rst | 19 +++++++++++++------ docs/updates/2026-10.md | 8 ++++++++ docs/updates/README.md | 3 ++- progress.md | 1 - 6 files changed, 52 insertions(+), 20 deletions(-) diff --git a/docs/source/Eng/usage/cloud.rst b/docs/source/Eng/usage/cloud.rst index 232933b..5ce9582 100644 --- a/docs/source/Eng/usage/cloud.rst +++ b/docs/source/Eng/usage/cloud.rst @@ -46,17 +46,27 @@ swap in ``AutoAddPolicy`` for convenience. Cross-backend copy ------------------ -The cross-backend dispatcher accepts URI syntax for every backend: +``FA_copy_between`` (``copy_between(source, target)``) copies one file from a +backend to another through a local temporary file and returns ``True`` when +both halves succeeded: .. code-block:: python from automation_file import execute_action execute_action([ - ["FA_cross_copy", - {"src": "s3://reports/2026-04.csv", - "dst": "drive:///Backups/april.csv"}], + ["FA_copy_between", + {"source": "s3://reports/2026-04.csv", + "target": "azure://backups/april.csv"}], ]) -URI prefixes: ``local://``, ``s3://``, ``drive://``, ``sftp://``, -``azure://``, ``dropbox://``, ``ftp://``. +It accepts ``s3://bucket/key``, ``azure://container/blob`` (or ``az://``), +``dropbox:/path``, ``sftp:/path``, ``ftp:/path``, ``local:/path`` or a plain +filesystem path, and ``http://`` / ``https://`` as a source only. Each backend +must be initialised first (``s3_instance.later_init(...)`` and so on). There is +no Google Drive scheme: Drive addresses files by ID, so use the ``FA_drive_*`` +actions for it. + +For new code prefer the storage layer (:doc:`storage`): ``FA_storage_copy`` +takes the same kind of URIs, reports what it copied, and raises a specific +error instead of returning ``False``. diff --git a/docs/source/Zh-CN/usage/cloud.rst b/docs/source/Zh-CN/usage/cloud.rst index 2efe62d..4848d1e 100644 --- a/docs/source/Zh-CN/usage/cloud.rst +++ b/docs/source/Zh-CN/usage/cloud.rst @@ -44,17 +44,24 @@ SFTP 跨后端复制 ---------- -跨后端调度器接受 URI 语法： +``FA_copy_between``（``copy_between(source, target)``）通过本地临时文件，把一个文件 +从某个后端复制到另一个后端，两个阶段都成功时返回 ``True``： .. code-block:: python from automation_file import execute_action execute_action([ - ["FA_cross_copy", - {"src": "s3://reports/2026-04.csv", - "dst": "drive:///Backups/april.csv"}], + ["FA_copy_between", + {"source": "s3://reports/2026-04.csv", + "target": "azure://backups/april.csv"}], ]) -URI 前缀：``local://``、``s3://``、``drive://``、``sftp://``、 -``azure://``、``dropbox://``、``ftp://``。 +它接受 ``s3://bucket/key``、``azure://container/blob``（或 ``az://``）、 +``dropbox:/path``、``sftp:/path``、``ftp:/path``、``local:/path`` 或普通的文件系统 +路径；``http://`` / ``https://`` 只能作为来源。每个后端都必须先初始化 +（``s3_instance.later_init(...)`` 等）。没有 Google Drive 的 scheme：Drive 以 ID +定位文件，请改用 ``FA_drive_*`` 动作。 + +新的代码建议使用存储层（:doc:`storage`）：``FA_storage_copy`` 接受同类型的 URI， +会报告复制的结果，失败时抛出明确的异常，而不是返回 ``False``。 diff --git a/docs/source/Zh-TW/usage/cloud.rst b/docs/source/Zh-TW/usage/cloud.rst index c47be33..39f1e07 100644 --- a/docs/source/Zh-TW/usage/cloud.rst +++ b/docs/source/Zh-TW/usage/cloud.rst @@ -44,17 +44,24 @@ SFTP 跨後端複製 ---------- -跨後端調度器接受 URI 語法： +``FA_copy_between``（``copy_between(source, target)``）透過本機暫存檔，把一個檔案 +從某個後端複製到另一個後端，兩個階段都成功時回傳 ``True``： .. code-block:: python from automation_file import execute_action execute_action([ - ["FA_cross_copy", - {"src": "s3://reports/2026-04.csv", - "dst": "drive:///Backups/april.csv"}], + ["FA_copy_between", + {"source": "s3://reports/2026-04.csv", + "target": "azure://backups/april.csv"}], ]) -URI 前綴：``local://``、``s3://``、``drive://``、``sftp://``、 -``azure://``、``dropbox://``、``ftp://``。 +它接受 ``s3://bucket/key``、``azure://container/blob``（或 ``az://``）、 +``dropbox:/path``、``sftp:/path``、``ftp:/path``、``local:/path`` 或一般的檔案系統 +路徑；``http://`` / ``https://`` 只能作為來源。每個後端都必須先初始化 +（``s3_instance.later_init(...)`` 等）。沒有 Google Drive 的 scheme：Drive 以 ID +定位檔案，請改用 ``FA_drive_*`` 動作。 + +新的程式建議使用儲存層（:doc:`storage`）：``FA_storage_copy`` 接受同類型的 URI， +會回報複製的結果，失敗時拋出明確的例外，而不是回傳 ``False``。 diff --git a/docs/updates/2026-10.md b/docs/updates/2026-10.md index db9b702..896928f 100644 --- a/docs/updates/2026-10.md +++ b/docs/updates/2026-10.md @@ -292,3 +292,11 @@ Index and query commands: [README.md](README.md). New entries go at the end. - **Docs**: an "Events" chapter in the three manuals (`usage/event_bus.rst`, chapter 17) and an API page (chapter N), a feature bullet and an "Events" section in the three READMEs, `architecture.md` §2, §3 and §4, `CLAUDE.md` (package map, key types). - **Files**: `automation_file/events/` (5 modules), `automation_file/storage/observe.py`, `automation_file/storage/backend.py`, `automation_file/__init__.py`, `tests/test_events.py`, `tests/test_storage_observe.py`, the documentation above. - **Open items**: `progress.md` #23 (notification router, audit schema v2, scheduler states), #21 and #22, which publish on this bus. + +## U-20261008-07 · 2026-10-08 · The cloud pages describe FA_copy_between, not FA_cross_copy · #done #docs + +- **What**: `progress.md` #27. The "Cross-backend copy" section of `docs/source/Eng/usage/cloud.rst` and its `Zh-TW` and `Zh-CN` translations showed an action `FA_cross_copy` with `src` / `dst` and a `drive://` prefix. Neither exists: the action is `FA_copy_between(source, target)` and `remote/cross_backend.py` has no Drive scheme. The READMEs had been corrected in `bc13101`; these pages had not. + - The section now shows `FA_copy_between` with `source` / `target`, lists the forms it accepts (`s3://bucket/key`, `azure://` or `az://`, `dropbox:/path`, `sftp:/path`, `ftp:/path`, `local:/path` or a plain path, and `http(s)://` as a source only), says each backend must be initialised first and that Drive has no scheme, and points new code at `FA_storage_copy` in the storage layer. +- **Result / numbers**: no occurrence of `FA_cross_copy` or `drive://` is left under `docs/source/` or in the READMEs. Documentation only. +- **Files**: `docs/source/Eng/usage/cloud.rst`, `docs/source/Zh-TW/usage/cloud.rst`, `docs/source/Zh-CN/usage/cloud.rst`, `progress.md`. +- **Open items**: `progress.md` #16 (moving `copy_between` itself onto the storage layer). diff --git a/docs/updates/README.md b/docs/updates/README.md index 05c8ad7..28c8511 100644 --- a/docs/updates/README.md +++ b/docs/updates/README.md @@ -58,6 +58,7 @@ In the same commit: delete the item from `progress.md`, add a `#done` entry here | ID | Date | Title | Tags | Batch | |---|---|---|---|---| +| U-20261008-07 | 2026-10-08 | The cloud pages describe FA_copy_between, not FA_cross_copy | #done #docs | [2026-10](2026-10.md) | | U-20261008-06 | 2026-10-08 | Event model, event bus and storage observers | #events #roadmap #storage | [2026-10](2026-10.md) | | U-20261008-05 | 2026-10-08 | Streams and directory trees in the storage layer | #storage #roadmap #streams | [2026-10](2026-10.md) | | U-20261008-04 | 2026-10-08 | Version directories stay short for long source paths | #done #versioning #windows | [2026-10](2026-10.md) | @@ -101,5 +102,5 @@ In the same commit: delete the item from `progress.md`, add a `#done` entry here | File | Period | Entries | |---|---|---:| -| [2026-10.md](2026-10.md) | 2026-10 | 17 | +| [2026-10.md](2026-10.md) | 2026-10 | 18 | | [2026-09.md](2026-09.md) | 2026-09 | 21 | diff --git a/progress.md b/progress.md index 797e619..81367af 100644 --- a/progress.md +++ b/progress.md @@ -39,4 +39,3 @@ Items #10 to #26 are what is left of the 1.0 roadmap (`docs/FILEAUTOMATION-1.0-R ### Found on the way -- **#27** The three `usage/cloud.rst` pages (`docs/source/Eng`, `Zh-TW`, `Zh-CN`) show a `FA_cross_copy` action with `src` / `dst` and a `drive://` prefix. Neither exists: the action is `FA_copy_between(source, target)` and `remote/cross_backend.py` has no `drive` scheme. The READMEs were corrected in `bc13101`; these pages were not. Fix them with #16, which rewrites that section anyway. From c394651892096d9604ea40e1ea75fc75eccc0e40 Mon Sep 17 00:00:00 2001 From: JeffreyChen Date: Thu, 8 Oct 2026 12:32:59 +0800 Subject: [PATCH 28/59] docs: drop drive:// from the architecture diagrams --- docs/source/Eng/architecture.rst | 2 +- docs/source/Zh-CN/architecture.rst | 2 +- docs/source/Zh-TW/architecture.rst | 2 +- docs/updates/2026-10.md | 9 ++++++++- docs/updates/README.md | 3 ++- progress.md | 3 --- 6 files changed, 13 insertions(+), 8 deletions(-) diff --git a/docs/source/Eng/architecture.rst b/docs/source/Eng/architecture.rst index dc23e8a..6334c92 100644 --- a/docs/source/Eng/architecture.rst +++ b/docs/source/Eng/architecture.rst @@ -104,7 +104,7 @@ dispatchers. WebDAV["webdav"] SMB["smb / cifs"] Fsspec["fsspec_bridge"] - Cross["cross_backend
local:// s3:// drive:// azure://
dropbox:// sftp:// ftp://"] + Cross["cross_backend
local:// s3:// azure://
dropbox:// sftp:// ftp://"] end subgraph Notify["notifications"] diff --git a/docs/source/Zh-CN/architecture.rst b/docs/source/Zh-CN/architecture.rst index 3cb95b4..5be3522 100644 --- a/docs/source/Zh-CN/architecture.rst +++ b/docs/source/Zh-CN/architecture.rst @@ -101,7 +101,7 @@ WebDAV["webdav"] SMB["smb / cifs"] Fsspec["fsspec_bridge"] - Cross["cross_backend
local:// s3:// drive:// azure://
dropbox:// sftp:// ftp://"] + Cross["cross_backend
local:// s3:// azure://
dropbox:// sftp:// ftp://"] end subgraph Notify["通知"] diff --git a/docs/source/Zh-TW/architecture.rst b/docs/source/Zh-TW/architecture.rst index 706d584..3ae73a0 100644 --- a/docs/source/Zh-TW/architecture.rst +++ b/docs/source/Zh-TW/architecture.rst @@ -101,7 +101,7 @@ WebDAV["webdav"] SMB["smb / cifs"] Fsspec["fsspec_bridge"] - Cross["cross_backend
local:// s3:// drive:// azure://
dropbox:// sftp:// ftp://"] + Cross["cross_backend
local:// s3:// azure://
dropbox:// sftp:// ftp://"] end subgraph Notify["通知"] diff --git a/docs/updates/2026-10.md b/docs/updates/2026-10.md index 896928f..d1b0caa 100644 --- a/docs/updates/2026-10.md +++ b/docs/updates/2026-10.md @@ -297,6 +297,13 @@ Index and query commands: [README.md](README.md). New entries go at the end. - **What**: `progress.md` #27. The "Cross-backend copy" section of `docs/source/Eng/usage/cloud.rst` and its `Zh-TW` and `Zh-CN` translations showed an action `FA_cross_copy` with `src` / `dst` and a `drive://` prefix. Neither exists: the action is `FA_copy_between(source, target)` and `remote/cross_backend.py` has no Drive scheme. The READMEs had been corrected in `bc13101`; these pages had not. - The section now shows `FA_copy_between` with `source` / `target`, lists the forms it accepts (`s3://bucket/key`, `azure://` or `az://`, `dropbox:/path`, `sftp:/path`, `ftp:/path`, `local:/path` or a plain path, and `http(s)://` as a source only), says each backend must be initialised first and that Drive has no scheme, and points new code at `FA_storage_copy` in the storage layer. -- **Result / numbers**: no occurrence of `FA_cross_copy` or `drive://` is left under `docs/source/` or in the READMEs. Documentation only. +- **Result / numbers**: no occurrence of `FA_cross_copy` or `drive://` is left under `docs/source/` or in the READMEs. Documentation only. → corrected in U-20261008-08: three diagrams still had `drive://`. - **Files**: `docs/source/Eng/usage/cloud.rst`, `docs/source/Zh-TW/usage/cloud.rst`, `docs/source/Zh-CN/usage/cloud.rst`, `progress.md`. - **Open items**: `progress.md` #16 (moving `copy_between` itself onto the storage layer). + +## U-20261008-08 · 2026-10-08 · Three architecture diagrams still listed drive:// · #incident #docs + +- **What**: U-20261008-07 said no `drive://` was left under `docs/source/`. That was wrong: the check was read before its output was complete. The `cross_backend` node of the Mermaid diagram in `docs/source/Eng/architecture.rst`, `Zh-TW/architecture.rst` and `Zh-CN/architecture.rst` still listed `drive://` among its prefixes. +- **Fix**: the node lists `local:// s3:// azure:// dropbox:// sftp:// ftp://`, as the READMEs' diagram does. `grep -rn "FA_cross_copy\|[^g]drive://" docs/source README*.md` now prints nothing (`gdrive://`, the storage layer's scheme, is a different thing). +- **Files**: the three `architecture.rst` pages, `docs/updates/2026-10.md` (the pointer added to U-20261008-07), `progress.md` (an empty heading removed). +- **Open items**: none. diff --git a/docs/updates/README.md b/docs/updates/README.md index 28c8511..2944cde 100644 --- a/docs/updates/README.md +++ b/docs/updates/README.md @@ -58,6 +58,7 @@ In the same commit: delete the item from `progress.md`, add a `#done` entry here | ID | Date | Title | Tags | Batch | |---|---|---|---|---| +| U-20261008-08 | 2026-10-08 | Three architecture diagrams still listed drive:// | #incident #docs | [2026-10](2026-10.md) | | U-20261008-07 | 2026-10-08 | The cloud pages describe FA_copy_between, not FA_cross_copy | #done #docs | [2026-10](2026-10.md) | | U-20261008-06 | 2026-10-08 | Event model, event bus and storage observers | #events #roadmap #storage | [2026-10](2026-10.md) | | U-20261008-05 | 2026-10-08 | Streams and directory trees in the storage layer | #storage #roadmap #streams | [2026-10](2026-10.md) | @@ -102,5 +103,5 @@ In the same commit: delete the item from `progress.md`, add a `#done` entry here | File | Period | Entries | |---|---|---:| -| [2026-10.md](2026-10.md) | 2026-10 | 18 | +| [2026-10.md](2026-10.md) | 2026-10 | 19 | | [2026-09.md](2026-09.md) | 2026-09 | 21 | diff --git a/progress.md b/progress.md index 81367af..5975449 100644 --- a/progress.md +++ b/progress.md @@ -36,6 +36,3 @@ Items #10 to #26 are what is left of the 1.0 roadmap (`docs/FILEAUTOMATION-1.0-R - **#24** UI 2.0 (roadmap §11, M7). Not before the APIs of #13 to #23 are stable (roadmap §20). - **#25** Semantic MCP tools (roadmap §12, M8): `file_*`, `storage_*`, `pipeline_*`, `integrity_status`, `audit_search`, with a permission model and dry run, next to the existing `FA_*` bridge. - **#26** Release engineering and 1.0 (roadmap §13, M9): contract and integration tests in the PR gate, PyPI Trusted Publishing, SemVer, migration guide, API freeze. - -### Found on the way - From 43432c7b91e2e4e3e846c4e031ae9bd594c9ebb8 Mon Sep 17 00:00:00 2001 From: JeffreyChen Date: Thu, 8 Oct 2026 12:50:59 +0800 Subject: [PATCH 29/59] build: move the backend SDKs and the GUI toolkit under extras --- .github/workflows/ci-dev.yml | 55 +++++- .github/workflows/ci-stable.yml | 53 ++++- CLAUDE.md | 8 +- README.md | 40 +++- README.zh-CN.md | 40 +++- README.zh-TW.md | 40 +++- architecture.md | 14 +- automation_file/core/optional.py | 52 +++++ automation_file/exceptions.py | 4 + automation_file/local/data_ops.py | 8 +- automation_file/remote/azure_blob/client.py | 10 +- automation_file/remote/box/client.py | 3 +- automation_file/remote/dropbox_api/client.py | 9 +- .../remote/dropbox_api/upload_ops.py | 8 +- automation_file/remote/fsspec_bridge.py | 4 +- automation_file/remote/google_drive/client.py | 65 ++++--- .../remote/google_drive/delete_ops.py | 7 +- .../remote/google_drive/download_ops.py | 17 +- .../remote/google_drive/folder_ops.py | 7 +- .../remote/google_drive/search_ops.py | 13 +- .../remote/google_drive/share_ops.py | 7 +- .../remote/google_drive/upload_ops.py | 14 +- automation_file/remote/s3/client.py | 9 +- dev.toml | 55 ++++-- dev_requirements.txt | 2 +- docs/source/API/remote.rst | 10 +- docs/source/Eng/usage/cloud.rst | 12 +- docs/source/Zh-CN/usage/cloud.rst | 11 +- docs/source/Zh-TW/usage/cloud.rst | 11 +- docs/updates/2026-10.md | 16 ++ docs/updates/README.md | 3 +- progress.md | 8 +- requirements.txt | 2 +- stable.toml | 55 ++++-- tests/test_box_ops.py | 4 + tests/test_data_ops_yaml_parquet.py | 11 ++ tests/test_dev_release.py | 3 +- tests/test_optional_dependencies.py | 183 ++++++++++++++++++ tests/test_storage_azure.py | 11 +- tests/test_storage_s3.py | 25 ++- 40 files changed, 722 insertions(+), 187 deletions(-) create mode 100644 automation_file/core/optional.py create mode 100644 tests/test_optional_dependencies.py diff --git a/.github/workflows/ci-dev.yml b/.github/workflows/ci-dev.yml index 8940b0c..5add4f6 100644 --- a/.github/workflows/ci-dev.yml +++ b/.github/workflows/ci-dev.yml @@ -56,8 +56,8 @@ jobs: run: | python -m pip install --upgrade pip wheel Copy-Item dev.toml pyproject.toml -Force - pip install -e . - pip install pytest pytest-cov + # Every extra, so every backend's tests run. The minimal job runs without any. + pip install -e ".[all,test]" - name: Run pytest with coverage run: python -m pytest tests/ -v --tb=short --cov=automation_file --cov-report=term-missing --cov-report=xml - name: Upload coverage artifact @@ -67,12 +67,61 @@ jobs: name: coverage-xml path: coverage.xml + minimal: + # The base install: no cloud SDK and no GUI toolkit. The tests of a backend whose extra is + # missing skip; everything else, the import guard included, has to pass. + needs: lint + runs-on: windows-latest + timeout-minutes: 15 # no run yet: the floor of 15 minutes, to revisit after the first runs + steps: + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + persist-credentials: false + - name: Set up Python + uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0 + with: + python-version: "3.12" + cache: pip + - name: Install the base package + run: | + python -m pip install --upgrade pip wheel + Copy-Item dev.toml pyproject.toml -Force + pip install -e ".[test]" + - name: Run pytest + run: python -m pytest tests/ -v --tb=short + + extras: + # Each extra on its own: it installs, and its backend's tests run with no other SDK present. + needs: lint + runs-on: windows-latest + timeout-minutes: 15 # no run yet: the floor of 15 minutes, to revisit after the first runs + strategy: + fail-fast: false + matrix: + extra: [ s3, azure, gdrive, dropbox, sftp, ftp, webdav, smb, fsspec, onedrive, box, parquet, gui ] + steps: + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + persist-credentials: false + - name: Set up Python + uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0 + with: + python-version: "3.12" + cache: pip + - name: Install the package with one extra + run: | + python -m pip install --upgrade pip wheel + Copy-Item dev.toml pyproject.toml -Force + pip install -e ".[${{ matrix.extra }},test]" + - name: Run pytest + run: python -m pytest tests/ -v --tb=short + publish-dev: # The dev channel. A push to dev that passes lint and the tests is built from dev.toml and uploaded # when it is still the tip of dev and ships something the newest automation_file_dev does not. # scripts/dev_release.py picks the version from PyPI, so nothing is committed back. name: Publish automation_file_dev to PyPI - needs: [lint, pytest] + needs: [lint, pytest, minimal, extras] if: github.event_name == 'push' && github.ref == 'refs/heads/dev' runs-on: ubuntu-latest timeout-minutes: 15 # about 3x the slowest recent run, at least 15 diff --git a/.github/workflows/ci-stable.yml b/.github/workflows/ci-stable.yml index 56855bf..ce50e92 100644 --- a/.github/workflows/ci-stable.yml +++ b/.github/workflows/ci-stable.yml @@ -56,8 +56,8 @@ jobs: run: | python -m pip install --upgrade pip wheel Copy-Item stable.toml pyproject.toml -Force - pip install -e . - pip install pytest pytest-cov + # Every extra, so every backend's tests run. The minimal job runs without any. + pip install -e ".[all,test]" - name: Run pytest with coverage run: python -m pytest tests/ -v --tb=short --cov=automation_file --cov-report=term-missing --cov-report=xml - name: Upload coverage artifact @@ -66,3 +66,52 @@ jobs: with: name: coverage-xml path: coverage.xml + + minimal: + # The base install: no cloud SDK and no GUI toolkit. The tests of a backend whose extra is + # missing skip; everything else, the import guard included, has to pass. + needs: lint + runs-on: windows-latest + timeout-minutes: 15 # no run yet: the floor of 15 minutes, to revisit after the first runs + steps: + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + persist-credentials: false + - name: Set up Python + uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0 + with: + python-version: "3.12" + cache: pip + - name: Install the base package + run: | + python -m pip install --upgrade pip wheel + Copy-Item stable.toml pyproject.toml -Force + pip install -e ".[test]" + - name: Run pytest + run: python -m pytest tests/ -v --tb=short + + extras: + # Each extra on its own: it installs, and its backend's tests run with no other SDK present. + needs: lint + runs-on: windows-latest + timeout-minutes: 15 # no run yet: the floor of 15 minutes, to revisit after the first runs + strategy: + fail-fast: false + matrix: + extra: [ s3, azure, gdrive, dropbox, sftp, ftp, webdav, smb, fsspec, onedrive, box, parquet, gui ] + steps: + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + persist-credentials: false + - name: Set up Python + uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0 + with: + python-version: "3.12" + cache: pip + - name: Install the package with one extra + run: | + python -m pip install --upgrade pip wheel + Copy-Item stable.toml pyproject.toml -Force + pip install -e ".[${{ matrix.extra }},test]" + - name: Run pytest + run: python -m pytest tests/ -v --tb=short diff --git a/CLAUDE.md b/CLAUDE.md index f422201..dd422aa 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -61,7 +61,7 @@ automation_file/ - `CallbackExecutor` — runs a registered trigger, then a user callback, sharing the executor's registry. - `PackageLoader` — imports a package by name and registers its top-level functions / classes / builtins as `_`. - `GoogleDriveClient` — wraps OAuth2 credential loading; exposes `service` lazily. `later_init(token_path, credentials_path)` bootstraps; `require_service()` raises if not initialised. -- `S3Client` / `AzureBlobClient` / `DropboxClient` / `SFTPClient` — singleton wrappers around the required SDKs. Each exposes `later_init(...)` plus `close()` where relevant. Their ops are auto-registered by `build_default_registry()`; `register__ops(registry)` is still exported so callers can populate custom registries. +- `S3Client` / `AzureBlobClient` / `DropboxClient` / `SFTPClient` — singleton wrappers around the SDKs of their extras. Each exposes `later_init(...)` plus `close()` where relevant. Their ops are auto-registered by `build_default_registry()`; `register__ops(registry)` is still exported so callers can populate custom registries. - `MainWindow` — PySide6 tabbed control surface (`ui/main_window.py`). Nine tabs — Local, HTTP, Google Drive, S3, Azure Blob, Dropbox, SFTP, JSON actions, Servers — share a `LogPanel` and dispatch work through `ActionWorker(QRunnable)` on the global `QThreadPool`. - `launch_ui(argv=None)` — boots / reuses a `QApplication`, shows `MainWindow`, and returns the exec code. Exposed lazily on the facade via `__getattr__` so the Qt runtime isn't paid for by non-UI importers. - `TCPActionServer` — threaded TCP server that deserialises a JSON action list per connection. Defaults to loopback; optional `shared_secret` enforces `AUTH \n` prefix. @@ -78,10 +78,10 @@ automation_file/ - `main` branch: stable releases, publishes `automation_file` to PyPI (version in `stable.toml`). - `dev` branch: development, publishes `automation_file_dev` to PyPI from CI. The version in `dev.toml` is only a floor. -- Keep `dependencies` and `[project.optional-dependencies]` (`dev`) in sync across both TOMLs; `tests/test_dev_toml_parity.py` fails when those, the entry points, `requires-python`, `[build-system]` or `[tool.setuptools]` differ. Backends (`boto3`, `azure-storage-blob`, `dropbox`, `paramiko`) and `PySide6` are first-class runtime deps — do not move them back under extras. +- Keep `dependencies` and `[project.optional-dependencies]` (`dev`) in sync across both TOMLs; `tests/test_dev_toml_parity.py` fails when those, the entry points, `requires-python`, `[build-system]` or `[tool.setuptools]` differ. The base `dependencies` carry no cloud SDK and no GUI toolkit: each backend's SDK, `pyarrow` and `PySide6` live in an extra (`s3`, `azure`, `gdrive`, `dropbox`, `sftp`, `smb`, `fsspec`, `onedrive`, `box`, `parquet`, `gui`; `ftp` and `webdav` are empty; `all` lists every one). Do not move one into `dependencies`, and do not import one at module level: code asks for it at the moment of use with `automation_file.core.optional.require_module(name, extra=...)`, which raises `OptionalDependencyException` naming the extra. `tests/test_optional_dependencies.py` fails when the package cannot be imported without them, when importing it loads one, or when the extras and `all` drift apart. A new optional package needs its extra in both TOMLs, its line in `all`, and its entry in `core.optional.EXTRAS`. - **Version bumping is automatic.** A dedicated publish workflow bumps the patch in both `stable.toml` and `dev.toml`, builds, uploads to PyPI, then commits the bump back to `main` tagged as `vX.Y.Z`. Do not hand-bump before merging to `main`. The next publish run is skipped via a commit-message guard (`chore: bump version`), so the bump itself never re-triggers publishing. The dev channel takes its number from PyPI, so never hand-bump `dev.toml` either. - CI: GitHub Actions — a `lint` job on Ubuntu (Python 3.12), then `pytest` on Windows across Python 3.10 / 3.11 / 3.12 / 3.13 / 3.14. One workflow per branch: `.github/workflows/ci-dev.yml`, `.github/workflows/ci-stable.yml`. -- CI steps: `lint` (ruff check + ruff format --check + mypy) → `pytest` with coverage → uploads `coverage.xml` as an artifact. +- CI steps: `lint` (ruff check + ruff format --check + mypy) → `pytest` with coverage, installed with `.[all,test]` → uploads `coverage.xml` as an artifact. Two more jobs follow `lint`: `minimal` installs `.[test]` only and runs the whole suite (the tests of a missing extra skip), and `extras` installs each extra on its own. `publish-dev` needs all four. - Stable publishing lives in a separate workflow (`.github/workflows/publish.yml`) that runs on push to `main`: bumps both TOMLs, copies `stable.toml` to `pyproject.toml`, builds the sdist + wheel, `twine upload` via `PYPI_API_TOKEN`, then commits + tags + pushes and creates `gh release create v --generate-notes`. - Dev publishing is the `publish-dev` job at the end of `ci-dev.yml`. It runs only on a push to `dev`, after `lint` and `pytest` pass: `scripts/dev_release.py prepare` writes `pyproject.toml` from `dev.toml` with one patch above the newest `automation_file_dev` on PyPI, the job builds and runs `twine check`, and it uploads (same `PYPI_API_TOKEN`) only when the commit is still the tip of `dev` and the wheel differs from the newest published one. Nothing is committed back. - Both publish jobs hold `PYPI_API_TOKEN`, so they install their tools (`build`, `twine`, and the build backend `setuptools`) with one command and nothing else: `python -m pip install --require-hashes --only-binary :all: -r .github/requirements/publish.txt`. No `pip install --upgrade pip`, no unpinned install; `tests/test_workflow_actions.py` fails on any other `pip install` in a job that is given the token. To add or raise a tool, edit `.github/requirements/publish.in` and regenerate `publish.txt` with the `uv pip compile` command written in that file. Dependabot reads the directory and proposes updates on `dev`. @@ -92,7 +92,7 @@ automation_file/ ```bash python -m pip install -r dev_requirements.txt pytest pytest-cov -python -m pip install -e ".[dev]" # ruff, mypy, pre-commit +python -m pip install -e ".[all,dev]" # every backend and the GUI, plus ruff, mypy, pre-commit python -m pytest tests/ -v --tb=short ruff check automation_file/ tests/ ruff format --check automation_file/ tests/ diff --git a/README.md b/README.md index c32c9b1..4b5156f 100644 --- a/README.md +++ b/README.md @@ -311,23 +311,45 @@ through the same shared registry instance exposed as `executor.registry`. ## Installation ```bash -pip install automation_file +pip install automation_file # the base: no cloud SDK, no GUI toolkit +pip install "automation_file[s3,sftp]" # add the backends you use +pip install "automation_file[all]" # every backend and the GUI ``` -A single install pulls in every backend (Google Drive, S3, Azure Blob, Dropbox, -SFTP, OneDrive, Box) and the PySide6 GUI — no extras required for day-to-day use. +The base install runs JSON actions, local file operations, HTTP downloads, the storage +layer's local and in-memory backends, pipelines, events, triggers, the scheduler and the +servers. Each backend's SDK and the GUI toolkit live in an extra, imported only when the +feature is used. Calling a feature whose extra is missing raises +`OptionalDependencyException` with the command to run. + +| Extra | Installs | Gives you | +|---|---|---| +| `s3` | `boto3` | S3 (`FA_s3_*`, `s3://`) | +| `azure` | `azure-storage-blob` | Azure Blob (`FA_azure_blob_*`, `azure://`) | +| `gdrive` | `google-api-python-client`, `google-auth-httplib2`, `google-auth-oauthlib` | Google Drive (`FA_drive_*`) | +| `dropbox` | `dropbox` | Dropbox (`FA_dropbox_*`) | +| `sftp` | `paramiko` | SFTP (`FA_sftp_*`) | +| `ftp` | — | FTP / FTPS (`FA_ftp_*`); standard library only | +| `webdav` | — | WebDAV (`WebDAVClient`); the base dependencies suffice | +| `smb` | `smbprotocol` | SMB / CIFS (`SMBClient`) | +| `fsspec` | `fsspec` | The fsspec bridge | +| `onedrive` | `msal` | OneDrive (`FA_onedrive_*`) | +| `box` | `boxsdk` | Box (`FA_box_*`) | +| `parquet` | `pyarrow` | Parquet data operations (`FA_parquet_*`, `FA_csv_to_parquet`) | +| `gui` | `PySide6` | The desktop GUI (`python -m automation_file ui`) | +| `all` | everything above | Every backend and the GUI, as before the split | ```bash -pip install "automation_file[dev]" # ruff, mypy, pre-commit, pytest-cov, build, twine +pip install "automation_file[all,dev]" # plus ruff, mypy, pre-commit, pytest-cov, build, twine ``` +Upgrading from a release that bundled everything: install `automation_file[all]` to keep +what you had. + Requirements: - Python 3.10+ -- Bundled dependencies: `google-api-python-client`, `google-auth-httplib2`, `google-auth-oauthlib`, `requests`, - `tqdm`, `boto3`, `azure-storage-blob`, `dropbox`, - `paramiko`, `msal`, `boxsdk`, `PySide6`, - `watchdog`, `cryptography`, `prometheus_client`, `defusedxml`, - `PyYAML`, `pyarrow`, `opentelemetry-api`, `opentelemetry-sdk`, `je_action_core` (the action executor +- Base dependencies: `requests`, `tqdm`, `watchdog`, `cryptography`, `prometheus_client`, `defusedxml`, + `PyYAML`, `opentelemetry-api`, `opentelemetry-sdk`, `je_action_core` (the action executor shared with APITestka, LoadDensity and MailThunder) ## Usage diff --git a/README.zh-CN.md b/README.zh-CN.md index 3a25e74..3c86b1a 100644 --- a/README.zh-CN.md +++ b/README.zh-CN.md @@ -309,24 +309,44 @@ flowchart TD ## 安装 ```bash -pip install automation_file +pip install automation_file # 基础安装：不含云端 SDK，也不含 GUI 工具包 +pip install "automation_file[s3,sftp]" # 加上你会用到的后端 +pip install "automation_file[all]" # 所有后端与 GUI ``` -单次安装即涵盖所有后端（Google Drive、S3、Azure Blob、Dropbox、SFTP、OneDrive、Box）以及 -PySide6 GUI — 日常使用不需要任何 extras。 +基础安装即可运行 JSON 动作、本地文件操作、HTTP 下载、存储层的本地与内存后端、pipeline、 +事件、触发器、调度器与各种服务器。各后端的 SDK 与 GUI 工具包都放在 extra 中，只有在用到该 +功能时才会导入。调用缺少 extra 的功能时，会抛出 `OptionalDependencyException`，信息中附有 +要执行的安装命令。 + +| Extra | 安装的包 | 提供的功能 | +|---|---|---| +| `s3` | `boto3` | S3（`FA_s3_*`、`s3://`） | +| `azure` | `azure-storage-blob` | Azure Blob（`FA_azure_blob_*`、`azure://`） | +| `gdrive` | `google-api-python-client`、`google-auth-httplib2`、`google-auth-oauthlib` | Google Drive（`FA_drive_*`） | +| `dropbox` | `dropbox` | Dropbox（`FA_dropbox_*`） | +| `sftp` | `paramiko` | SFTP（`FA_sftp_*`） | +| `ftp` | — | FTP / FTPS（`FA_ftp_*`）；只需要标准库 | +| `webdav` | — | WebDAV（`WebDAVClient`）；基础依赖已足够 | +| `smb` | `smbprotocol` | SMB / CIFS（`SMBClient`） | +| `fsspec` | `fsspec` | fsspec 桥接 | +| `onedrive` | `msal` | OneDrive（`FA_onedrive_*`） | +| `box` | `boxsdk` | Box（`FA_box_*`） | +| `parquet` | `pyarrow` | Parquet 数据操作（`FA_parquet_*`、`FA_csv_to_parquet`） | +| `gui` | `PySide6` | 桌面 GUI（`python -m automation_file ui`） | +| `all` | 以上全部 | 所有后端与 GUI，与拆分之前相同 | ```bash -pip install "automation_file[dev]" # ruff, mypy, pre-commit, pytest-cov, build, twine +pip install "automation_file[all,dev]" # 另含 ruff、mypy、pre-commit、pytest-cov、build、twine ``` +从先前包含所有包的版本升级时：安装 `automation_file[all]` 即可保留原有的全部功能。 + 要求： - Python 3.10+ -- 内置依赖：`google-api-python-client`、`google-auth-httplib2`、`google-auth-oauthlib`、`requests`、 - `tqdm`、`boto3`、`azure-storage-blob`、`dropbox`、 - `paramiko`、`msal`、`boxsdk`、`PySide6`、 - `watchdog`、`cryptography`、`prometheus_client`、`defusedxml`、 - `PyYAML`、`pyarrow`、`opentelemetry-api`、`opentelemetry-sdk`、`je_action_core`(与 APITestka、 - LoadDensity、MailThunder 共用的 action 执行器) +- 基础依赖：`requests`、`tqdm`、`watchdog`、`cryptography`、`prometheus_client`、`defusedxml`、 + `PyYAML`、`opentelemetry-api`、`opentelemetry-sdk`、`je_action_core`（与 APITestka、 + LoadDensity、MailThunder 共用的 action 执行器） ## 使用方式 diff --git a/README.zh-TW.md b/README.zh-TW.md index c8eb857..9037e24 100644 --- a/README.zh-TW.md +++ b/README.zh-TW.md @@ -309,24 +309,44 @@ flowchart TD ## 安裝 ```bash -pip install automation_file +pip install automation_file # 基礎安裝：不含雲端 SDK，也不含 GUI 工具組 +pip install "automation_file[s3,sftp]" # 加上你會用到的後端 +pip install "automation_file[all]" # 所有後端與 GUI ``` -單一安裝即涵蓋所有後端（Google Drive、S3、Azure Blob、Dropbox、SFTP、OneDrive、Box）以及 -PySide6 GUI — 日常使用不需要任何 extras。 +基礎安裝即可執行 JSON 動作、本機檔案操作、HTTP 下載、儲存層的本機與記憶體後端、pipeline、 +事件、觸發器、排程器與各種伺服器。各後端的 SDK 與 GUI 工具組都放在 extra 中，只有在用到該 +功能時才會匯入。呼叫缺少 extra 的功能時，會拋出 `OptionalDependencyException`，訊息中附有 +要執行的安裝指令。 + +| Extra | 安裝的套件 | 提供的功能 | +|---|---|---| +| `s3` | `boto3` | S3（`FA_s3_*`、`s3://`） | +| `azure` | `azure-storage-blob` | Azure Blob（`FA_azure_blob_*`、`azure://`） | +| `gdrive` | `google-api-python-client`、`google-auth-httplib2`、`google-auth-oauthlib` | Google Drive（`FA_drive_*`） | +| `dropbox` | `dropbox` | Dropbox（`FA_dropbox_*`） | +| `sftp` | `paramiko` | SFTP（`FA_sftp_*`） | +| `ftp` | — | FTP / FTPS（`FA_ftp_*`）；只需要標準函式庫 | +| `webdav` | — | WebDAV（`WebDAVClient`）；基礎相依套件已足夠 | +| `smb` | `smbprotocol` | SMB / CIFS（`SMBClient`） | +| `fsspec` | `fsspec` | fsspec 橋接 | +| `onedrive` | `msal` | OneDrive（`FA_onedrive_*`） | +| `box` | `boxsdk` | Box（`FA_box_*`） | +| `parquet` | `pyarrow` | Parquet 資料操作（`FA_parquet_*`、`FA_csv_to_parquet`） | +| `gui` | `PySide6` | 桌面 GUI（`python -m automation_file ui`） | +| `all` | 以上全部 | 所有後端與 GUI，與拆分之前相同 | ```bash -pip install "automation_file[dev]" # ruff, mypy, pre-commit, pytest-cov, build, twine +pip install "automation_file[all,dev]" # 另含 ruff、mypy、pre-commit、pytest-cov、build、twine ``` +從先前內含所有套件的版本升級時：安裝 `automation_file[all]` 即可保留原有的全部功能。 + 需求： - Python 3.10+ -- 內建相依套件：`google-api-python-client`、`google-auth-httplib2`、`google-auth-oauthlib`、`requests`、 - `tqdm`、`boto3`、`azure-storage-blob`、`dropbox`、 - `paramiko`、`msal`、`boxsdk`、`PySide6`、 - `watchdog`、`cryptography`、`prometheus_client`、`defusedxml`、 - `PyYAML`、`pyarrow`、`opentelemetry-api`、`opentelemetry-sdk`、`je_action_core`(與 APITestka、 - LoadDensity、MailThunder 共用的 action 執行器) +- 基礎相依套件：`requests`、`tqdm`、`watchdog`、`cryptography`、`prometheus_client`、`defusedxml`、 + `PyYAML`、`opentelemetry-api`、`opentelemetry-sdk`、`je_action_core`（與 APITestka、 + LoadDensity、MailThunder 共用的 action 執行器） ## 使用方式 diff --git a/architecture.md b/architecture.md index cf03f21..866076c 100644 --- a/architecture.md +++ b/architecture.md @@ -21,7 +21,7 @@ piece of the 1.0 roadmap (`docs/FILEAUTOMATION-1.0-ROADMAP.md`, PR #107); what i | --- | --- | | `automation_file/__init__.py` | Public facade (`__all__`). Wires the shared `executor`, `callback_executor` and `package_manager` over one registry. `launch_ui` is loaded lazily through `__getattr__` | | `automation_file/__main__.py` | CLI: legacy flags plus subcommands | -| `automation_file/core/` | Engine, on je_action_core: `action_registry.py` (`ActionRegistry`, a `CommandRegistry`; `build_default_registry`), `action_executor.py` (`ActionExecutor`, an `ActionExecutor` with strict actions, indexed records and the dry-run, validate, substitute and parallel extras; shared `executor`), `callback_executor.py`, `package_loader.py`, `plugins.py`, `dag_executor.py`, `action_queue.py`, `json_store.py`, `substitution.py`. Also cross-cutting helpers: `retry`, `quota`, `rate_limit`, `circuit_breaker`, `file_lock`, `sqlite_lock`, `checksum`, `manifest`, `crypto`, `secrets`, `config`, `config_watcher`, `audit`, `metrics`, `tracing`, `progress`, `fim`, `content_store` | +| `automation_file/core/` | Engine, on je_action_core: `action_registry.py` (`ActionRegistry`, a `CommandRegistry`; `build_default_registry`), `action_executor.py` (`ActionExecutor`, an `ActionExecutor` with strict actions, indexed records and the dry-run, validate, substitute and parallel extras; shared `executor`), `callback_executor.py`, `package_loader.py`, `plugins.py`, `dag_executor.py`, `action_queue.py`, `json_store.py`, `substitution.py`. Also cross-cutting helpers: `optional` (`require_module`, the extras table), `retry`, `quota`, `rate_limit`, `circuit_breaker`, `file_lock`, `sqlite_lock`, `checksum`, `manifest`, `crypto`, `secrets`, `config`, `config_watcher`, `audit`, `metrics`, `tracing`, `progress`, `fim`, `content_store` | | `automation_file/local/` | Local strategy modules: file, dir, zip, tar and archive ops, sync, diff, text/JSON/data edits, templates, versioning, trash, `shell_ops` (argv-only subprocess), conditional branches. `safe_paths.py` guards against path traversal | | `automation_file/remote/` | `url_validator.py` (SSRF guard), `http_download.py`, `cross_backend.py`, `fsspec_bridge.py`. One subpackage per backend: `google_drive/`, `s3/`, `azure_blob/`, `dropbox_api/`, `sftp/`, `ftp/`, `onedrive/`, `box/`, each with `client.py`, `*_ops.py` and `register__ops`. `smb/` and `webdav/` have a client only | | `automation_file/storage/` | Universal storage layer. `uri.py` (`StorageURI`, `parse_storage_uri`, `normalize_path`), `types.py` (`FileInfo`, `Checksum`, `StorageCapabilities`), `backend.py` (`StorageBackend`: the public operations are template methods over the `_`-prefixed primitives a backend supplies), `local_storage.py` (`LocalStorage`, confined through `safe_join` when given a root), `memory_storage.py` (`MemoryStorage`), `object_storage.py` (`ObjectStorage`: directories as key prefixes over `_head`, `_scan`, `_put`, `_get`, `_remove`), `s3_storage.py` (`S3Storage`, over `s3_instance` or a given boto3 client), `azure_storage.py` (`AzureStorage`, over `azure_blob_instance` or a given `BlobServiceClient`), `resolver.py` (`StorageResolver`, `default_resolver`: mounts first, then scheme factories), `file.py` (`File`), `storage.py` (`Storage`), `observe.py` (listeners for `upload`, `download`, `read`, `delete`, `mkdir`, `copy`, `move`), `streams.py` (staged file objects behind `open_read` / `open_write`), `tree.py` (`copy_tree`, `sync_tree`, `TreeResult`), `actions.py` (the `FA_storage_*` functions and `register_storage_ops`). At module level it imports only `exceptions`, `logging_config`, `core.checksum` and `local.safe_paths`: no registry, no GUI, no backend SDK. The adapters import their SDK's exceptions and the shared client inside the functions that use them | @@ -154,7 +154,8 @@ storage.observe → events.storage_bridge → StorageError (only for a failing b 1. `remote//` with `client.py` (module singleton `_instance` with `later_init`, plus `close` where relevant), the `*_ops.py` modules, and `register__ops(registry)` in `__init__.py`. 2. Call it from `_register_cloud_backends`. - 3. Add the SDK to `dependencies` in both `stable.toml` and `dev.toml`, then add facade exports. + 3. Add the SDK as an extra in both `stable.toml` and `dev.toml` (and to `all`), name it in + `core.optional.EXTRAS`, import it with `require_module` where it is used, then add facade exports. 4. Add `ui/tabs/_tab.py` and wire it into `ui/tabs/transfer_tab.py`. 5. Add tests; paths that need the network are not exercised in CI. - **New storage backend** (the universal layer; separate from the `FA_*` backend above): @@ -183,7 +184,7 @@ storage.observe → events.storage_bridge → StorageError (only for a failing b (`PyBreeze/pybreeze/extend/process_executor/python_task_process_manager.py`; the package name is in `.../process_executor/file_automation/file_automation_process.py`). PyBreeze double-encodes the JSON on Windows, so `_execute_str`'s `isinstance`-guarded second decode and the legacy flag names are a - contract, guarded by `tests/test_legacy_cli_contract.py`. PyBreeze also declares `automation-file` as + contract, guarded by `tests/test_legacy_cli_contract.py`. PyBreeze declares `automation-file` and, now that the SDKs and the GUI are extras, needs `automation-file[all]` to keep what it had (`progress.md` #29); it lists the package as a dependency. - **TestPioneer** imports `download_file` and `unzip_all` from the facade in-process (`test_pioneer/executor/file/file_processing.py`). Its `parallel_run` does not spawn this package. @@ -234,9 +235,10 @@ storage.observe → events.storage_bridge → StorageError (only for a failing b command reaches it, so je_action_core's package gate is off here (`tests/test_package_loader.py` fails if one is added; workspace X-12). - No `shell=True`; subprocesses use argument lists and a timeout (§ Security › General rules; › Subprocess execution). -- Backends and PySide6 are first-class runtime dependencies. Keep `stable.toml` and `dev.toml` - in sync (`tests/test_dev_toml_parity.py`), and let CI number both channels: never bump a version by - hand (§ Branching & CI). +- The base install has no cloud SDK and no GUI toolkit: each lives in an extra and is imported at the + moment of use through `core.optional.require_module` (`tests/test_optional_dependencies.py`). Keep + `stable.toml` and `dev.toml` in sync (`tests/test_dev_toml_parity.py`), and let CI number both + channels: never bump a version by hand (§ Branching & CI). - Limits: cyclomatic complexity ≤ 15 (hard cap 20), cognitive complexity ≤ 15, functions ≤ 75 lines, ≤ 7 parameters, nesting ≤ 4, files ≤ 1000 lines (§ Code quality › Complexity & size). - Run `ruff check`, `ruff format --check`, `mypy` and `pytest` before committing (§ Development). diff --git a/automation_file/core/optional.py b/automation_file/core/optional.py new file mode 100644 index 0000000..dabe28a --- /dev/null +++ b/automation_file/core/optional.py @@ -0,0 +1,52 @@ +"""Optional dependencies: import an SDK when a feature needs it, or say which extra to install. + +The base install carries no cloud SDK and no GUI toolkit. Each backend's SDK +lives in an extra (``pip install "automation_file[s3]"``), and the code that +needs it calls :func:`require_module` at the moment of use, so importing +``automation_file`` never depends on a package the user did not ask for. +""" + +from __future__ import annotations + +import importlib +from types import ModuleType + +from automation_file.exceptions import OptionalDependencyException + +#: Extra name -> what it enables, for messages and documentation. +EXTRAS: dict[str, str] = { + "s3": "the S3 backend", + "azure": "the Azure Blob backend", + "gdrive": "the Google Drive backend", + "dropbox": "the Dropbox backend", + "sftp": "the SFTP backend", + "ftp": "the FTP / FTPS backend", + "webdav": "the WebDAV backend", + "smb": "the SMB / CIFS backend", + "fsspec": "the fsspec bridge and adapter", + "onedrive": "the OneDrive backend", + "box": "the Box backend", + "parquet": "the Parquet data operations", + "gui": "the desktop GUI", +} +_DISTRIBUTION = "automation_file" + + +def install_hint(extra: str) -> str: + """Return the ``pip install`` command that provides ``extra``.""" + return f'pip install "{_DISTRIBUTION}[{extra}]"' + + +def require_module(name: str, *, extra: str) -> ModuleType: + """Import and return the module ``name``, which the extra ``extra`` provides. + + Raises :class:`OptionalDependencyException` naming the extra when the module + cannot be imported. + """ + try: + return importlib.import_module(name) + except ImportError as error: + feature = EXTRAS.get(extra, f"the {extra} feature") + raise OptionalDependencyException( + f"{name.partition('.')[0]} is not installed; {feature} needs it: {install_hint(extra)}" + ) from error diff --git a/automation_file/exceptions.py b/automation_file/exceptions.py index ee8d225..07569c6 100644 --- a/automation_file/exceptions.py +++ b/automation_file/exceptions.py @@ -143,6 +143,10 @@ class TracingException(FileAutomationException): """Raised when OpenTelemetry tracing setup cannot be completed.""" +class OptionalDependencyException(FileAutomationException, RuntimeError): + """Raised when a feature needs a package that only an optional extra installs.""" + + class StorageException(FileAutomationException): """Root of the errors raised by the universal storage layer (``automation_file.storage``).""" diff --git a/automation_file/local/data_ops.py b/automation_file/local/data_ops.py index 86e2857..6afee14 100644 --- a/automation_file/local/data_ops.py +++ b/automation_file/local/data_ops.py @@ -22,10 +22,12 @@ from pathlib import Path from typing import Any +from automation_file.core.optional import require_module from automation_file.exceptions import DataOpsException, FileNotExistsException from automation_file.logging_config import file_automation_logger _MISSING = object() +_PARQUET_EXTRA = "parquet" def csv_filter( @@ -267,7 +269,7 @@ def parquet_read( ``limit`` caps the number of rows returned (reads the whole file but slices before conversion — handy for previews of multi-GB files). """ - import pyarrow.parquet as pq + pq = require_module("pyarrow.parquet", extra=_PARQUET_EXTRA) source = Path(path) if not source.is_file(): @@ -283,8 +285,8 @@ def parquet_read( def parquet_write(path: str, records: list[dict[str, Any]]) -> int: """Write ``records`` (list of dicts) as a Parquet file; return the row count.""" - import pyarrow as pa - import pyarrow.parquet as pq + pa = require_module("pyarrow", extra=_PARQUET_EXTRA) + pq = require_module("pyarrow.parquet", extra=_PARQUET_EXTRA) if not isinstance(records, list): raise DataOpsException("records must be a list of dicts") diff --git a/automation_file/remote/azure_blob/client.py b/automation_file/remote/azure_blob/client.py index 3cab75a..3b92df0 100644 --- a/automation_file/remote/azure_blob/client.py +++ b/automation_file/remote/azure_blob/client.py @@ -4,18 +4,12 @@ from typing import Any +from automation_file.core.optional import require_module from automation_file.logging_config import file_automation_logger def _import_blob_service_client() -> Any: - try: - from azure.storage.blob import BlobServiceClient - except ImportError as error: - raise RuntimeError( - "azure-storage-blob import failed — reinstall `automation_file` to restore" - " the Azure Blob backend" - ) from error - return BlobServiceClient + return require_module("azure.storage.blob", extra="azure").BlobServiceClient class AzureBlobClient: diff --git a/automation_file/remote/box/client.py b/automation_file/remote/box/client.py index 86c2f62..91ad6b7 100644 --- a/automation_file/remote/box/client.py +++ b/automation_file/remote/box/client.py @@ -12,6 +12,7 @@ from typing import Any +from automation_file.core.optional import install_hint from automation_file.exceptions import BoxException from automation_file.logging_config import file_automation_logger @@ -22,7 +23,7 @@ def import_box_sdk_gen() -> Any: import box_sdk_gen except ImportError as error: raise BoxException( - "box_sdk_gen import failed — install `boxsdk>=10` to restore the Box backend" + f"box_sdk_gen is not installed; the Box backend needs it: {install_hint('box')}" ) from error return box_sdk_gen diff --git a/automation_file/remote/dropbox_api/client.py b/automation_file/remote/dropbox_api/client.py index 13a1115..cdf4c3e 100644 --- a/automation_file/remote/dropbox_api/client.py +++ b/automation_file/remote/dropbox_api/client.py @@ -4,17 +4,12 @@ from typing import Any +from automation_file.core.optional import require_module from automation_file.logging_config import file_automation_logger def _import_dropbox() -> Any: - try: - import dropbox - except ImportError as error: - raise RuntimeError( - "dropbox import failed — reinstall `automation_file` to restore the Dropbox backend" - ) from error - return dropbox + return require_module("dropbox", extra="dropbox") class DropboxClient: diff --git a/automation_file/remote/dropbox_api/upload_ops.py b/automation_file/remote/dropbox_api/upload_ops.py index 2bc3a65..25ea5c1 100644 --- a/automation_file/remote/dropbox_api/upload_ops.py +++ b/automation_file/remote/dropbox_api/upload_ops.py @@ -4,6 +4,7 @@ from pathlib import Path +from automation_file.core.optional import require_module from automation_file.exceptions import FileNotExistsException from automation_file.logging_config import file_automation_logger from automation_file.remote._upload_tree import walk_and_upload @@ -20,12 +21,7 @@ def dropbox_upload_file(file_path: str, remote_path: str) -> bool: if not path.is_file(): raise FileNotExistsException(str(path)) client = dropbox_instance.require_client() - try: - from dropbox import files as dropbox_files - except ImportError as error: - raise RuntimeError( - "dropbox import failed — reinstall `automation_file` to restore the Dropbox backend" - ) from error + dropbox_files = require_module("dropbox.files", extra="dropbox") try: with open(path, "rb") as fp: client.files_upload( diff --git a/automation_file/remote/fsspec_bridge.py b/automation_file/remote/fsspec_bridge.py index 55d42df..b79a144 100644 --- a/automation_file/remote/fsspec_bridge.py +++ b/automation_file/remote/fsspec_bridge.py @@ -21,6 +21,7 @@ from pathlib import Path from typing import Any +from automation_file.core.optional import install_hint from automation_file.exceptions import FsspecException @@ -38,7 +39,8 @@ def _import_fsspec() -> Any: import fsspec except ImportError as error: raise FsspecException( - "fsspec import failed — install `fsspec` (and any backend extras) to use the bridge" + "fsspec is not installed; the fsspec bridge needs it (plus the package of the " + f"filesystem you address): {install_hint('fsspec')}" ) from error return fsspec diff --git a/automation_file/remote/google_drive/client.py b/automation_file/remote/google_drive/client.py index b361c32..0646edb 100644 --- a/automation_file/remote/google_drive/client.py +++ b/automation_file/remote/google_drive/client.py @@ -1,7 +1,8 @@ """Google Drive client (Singleton Facade). Wraps OAuth2 credential loading and exposes a lazily-built ``service`` attribute -that every operation module calls through. +that every operation module calls through. The Google SDK is imported when it is +first needed, so importing this module does not require the ``gdrive`` extra. """ from __future__ import annotations @@ -9,15 +10,22 @@ from pathlib import Path from typing import Any -from google.auth.transport.requests import Request -from google.oauth2.credentials import Credentials -from google_auth_oauthlib.flow import InstalledAppFlow -from googleapiclient.discovery import build -from googleapiclient.errors import HttpError - +from automation_file.core.optional import require_module from automation_file.logging_config import file_automation_logger _DEFAULT_SCOPES = ("https://www.googleapis.com/auth/drive",) +_EXTRA = "gdrive" + + +def drive_http_error() -> type[Exception]: + """Return ``googleapiclient.errors.HttpError``, imported on first use.""" + error_class: type[Exception] = require_module("googleapiclient.errors", extra=_EXTRA).HttpError + return error_class + + +def drive_media() -> Any: + """Return the ``googleapiclient.http`` module, imported on first use.""" + return require_module("googleapiclient.http", extra=_EXTRA) class GoogleDriveClient: @@ -25,7 +33,7 @@ class GoogleDriveClient: def __init__(self, scopes: tuple[str, ...] = _DEFAULT_SCOPES) -> None: self.scopes: tuple[str, ...] = scopes - self.creds: Credentials | None = None + self.creds: Any = None self.service: Any = None def later_init(self, token_path: str, credentials_path: str) -> Any: @@ -33,36 +41,45 @@ def later_init(self, token_path: str, credentials_path: str) -> Any: Writes the refreshed token back to ``token_path`` with UTF-8 encoding. """ + http_error = drive_http_error() token_file = Path(token_path) - credentials_file = Path(credentials_path) - creds: Credentials | None = None - - if token_file.exists(): - file_automation_logger.info("GoogleDriveClient: loading token from %s", token_file) - creds = Credentials.from_authorized_user_file(str(token_file), list(self.scopes)) + creds = self._load_credentials(token_file) if creds is None or not creds.valid: - if creds and creds.expired and creds.refresh_token: - creds.refresh(Request()) - else: - flow = InstalledAppFlow.from_client_secrets_file( - str(credentials_file), - list(self.scopes), - ) - creds = flow.run_local_server(port=0) + creds = self._renew_credentials(creds, Path(credentials_path)) with open(token_file, "w", encoding="utf-8") as token_fp: token_fp.write(creds.to_json()) try: self.creds = creds - self.service = build("drive", "v3", credentials=creds) + discovery = require_module("googleapiclient.discovery", extra=_EXTRA) + self.service = discovery.build("drive", "v3", credentials=creds) file_automation_logger.info("GoogleDriveClient: service ready") return self.service - except HttpError as error: + except http_error as error: file_automation_logger.error("GoogleDriveClient init failed: %r", error) self.service = None raise + def _load_credentials(self, token_file: Path) -> Any: + if not token_file.exists(): + return None + file_automation_logger.info("GoogleDriveClient: loading the saved token") + credentials = require_module("google.oauth2.credentials", extra=_EXTRA) + return credentials.Credentials.from_authorized_user_file(str(token_file), list(self.scopes)) + + def _renew_credentials(self, creds: Any, credentials_file: Path) -> Any: + if creds and creds.expired and creds.refresh_token: + transport = require_module("google.auth.transport.requests", extra=_EXTRA) + creds.refresh(transport.Request()) + return creds + flow_module = require_module("google_auth_oauthlib.flow", extra=_EXTRA) + flow = flow_module.InstalledAppFlow.from_client_secrets_file( + str(credentials_file), + list(self.scopes), + ) + return flow.run_local_server(port=0) + def require_service(self) -> Any: """Return ``self.service`` or raise if the client has not been initialised.""" if self.service is None: diff --git a/automation_file/remote/google_drive/delete_ops.py b/automation_file/remote/google_drive/delete_ops.py index 6225a24..3137a94 100644 --- a/automation_file/remote/google_drive/delete_ops.py +++ b/automation_file/remote/google_drive/delete_ops.py @@ -4,18 +4,17 @@ from typing import Any -from googleapiclient.errors import HttpError - from automation_file.logging_config import file_automation_logger -from automation_file.remote.google_drive.client import driver_instance +from automation_file.remote.google_drive.client import drive_http_error, driver_instance def drive_delete_file(file_id: str) -> Any | None: """Delete a file by Drive ID. Returns the API response or None.""" + http_error = drive_http_error() try: result = driver_instance.require_service().files().delete(fileId=file_id).execute() file_automation_logger.info("drive_delete_file: %s", file_id) return result - except HttpError as error: + except http_error as error: file_automation_logger.error("drive_delete_file failed: %r", error) return None diff --git a/automation_file/remote/google_drive/download_ops.py b/automation_file/remote/google_drive/download_ops.py index 1c38108..e61ea73 100644 --- a/automation_file/remote/google_drive/download_ops.py +++ b/automation_file/remote/google_drive/download_ops.py @@ -4,11 +4,12 @@ import io -from googleapiclient.errors import HttpError -from googleapiclient.http import MediaIoBaseDownload - from automation_file.logging_config import file_automation_logger -from automation_file.remote.google_drive.client import driver_instance +from automation_file.remote.google_drive.client import ( + drive_http_error, + drive_media, + driver_instance, +) def drive_download_file(file_id: str, file_name: str) -> io.BytesIO | None: @@ -18,11 +19,12 @@ def drive_download_file(file_id: str, file_name: str) -> io.BytesIO | None: is **only** written after the download completes cleanly, so a failed request cannot leave an empty file behind. """ + http_error = drive_http_error() service = driver_instance.require_service() buffer = io.BytesIO() try: request = service.files().get_media(fileId=file_id) - downloader = MediaIoBaseDownload(buffer, request) + downloader = drive_media().MediaIoBaseDownload(buffer, request) done = False while not done: status, done = downloader.next_chunk() @@ -32,7 +34,7 @@ def drive_download_file(file_id: str, file_name: str) -> io.BytesIO | None: file_name, int(status.progress() * 100), ) - except HttpError as error: + except http_error as error: file_automation_logger.error("drive_download_file failed: %r", error) return None @@ -44,6 +46,7 @@ def drive_download_file(file_id: str, file_name: str) -> io.BytesIO | None: def drive_download_file_from_folder(folder_name: str) -> dict[str, str] | None: """Download every file inside the Drive folder named ``folder_name``.""" + http_error = drive_http_error() service = driver_instance.require_service() try: folders = ( @@ -60,7 +63,7 @@ def drive_download_file_from_folder(folder_name: str) -> dict[str, str] | None: return None folder_id = folder_list[0].get("id") response = service.files().list(q=f"'{folder_id}' in parents").execute() - except HttpError as error: + except http_error as error: file_automation_logger.error("drive_download_file_from_folder failed: %r", error) return None diff --git a/automation_file/remote/google_drive/folder_ops.py b/automation_file/remote/google_drive/folder_ops.py index a10f026..236ebff 100644 --- a/automation_file/remote/google_drive/folder_ops.py +++ b/automation_file/remote/google_drive/folder_ops.py @@ -2,10 +2,8 @@ from __future__ import annotations -from googleapiclient.errors import HttpError - from automation_file.logging_config import file_automation_logger -from automation_file.remote.google_drive.client import driver_instance +from automation_file.remote.google_drive.client import drive_http_error, driver_instance _FOLDER_MIME = "application/vnd.google-apps.folder" @@ -13,12 +11,13 @@ def drive_add_folder(folder_name: str) -> str | None: """Create a folder on Drive. Returns the new folder's ID or None.""" metadata = {"name": folder_name, "mimeType": _FOLDER_MIME} + http_error = drive_http_error() try: response = ( driver_instance.require_service().files().create(body=metadata, fields="id").execute() ) file_automation_logger.info("drive_add_folder: %s", folder_name) return response.get("id") - except HttpError as error: + except http_error as error: file_automation_logger.error("drive_add_folder failed: %r", error) return None diff --git a/automation_file/remote/google_drive/search_ops.py b/automation_file/remote/google_drive/search_ops.py index cb3adb0..4de2645 100644 --- a/automation_file/remote/google_drive/search_ops.py +++ b/automation_file/remote/google_drive/search_ops.py @@ -2,17 +2,16 @@ from __future__ import annotations -from googleapiclient.errors import HttpError - from automation_file.logging_config import file_automation_logger -from automation_file.remote.google_drive.client import driver_instance +from automation_file.remote.google_drive.client import drive_http_error, driver_instance def drive_search_all_file() -> dict[str, str] | None: """Return ``{name: id}`` for every file visible to the current token.""" + http_error = drive_http_error() try: response = driver_instance.require_service().files().list().execute() - except HttpError as error: + except http_error as error: file_automation_logger.error("drive_search_all_file failed: %r", error) return None result = {file.get("name"): file.get("id") for file in response.get("files", [])} @@ -24,6 +23,7 @@ def drive_search_file_mimetype(mime_type: str) -> dict[str, str] | None: """Return ``{name: id}`` for files matching ``mime_type`` (all pages).""" results: dict[str, str] = {} page_token: str | None = None + http_error = drive_http_error() service = driver_instance.require_service() try: while True: @@ -41,7 +41,7 @@ def drive_search_file_mimetype(mime_type: str) -> dict[str, str] | None: page_token = response.get("nextPageToken") if page_token is None: break - except HttpError as error: + except http_error as error: file_automation_logger.error("drive_search_file_mimetype failed: %r", error) return None file_automation_logger.info( @@ -52,9 +52,10 @@ def drive_search_file_mimetype(mime_type: str) -> dict[str, str] | None: def drive_search_field(field_pattern: str) -> dict[str, str] | None: """Return ``{name: id}`` for a list call with a custom ``fields=`` pattern.""" + http_error = drive_http_error() try: response = driver_instance.require_service().files().list(fields=field_pattern).execute() - except HttpError as error: + except http_error as error: file_automation_logger.error("drive_search_field failed: %r", error) return None result = {file.get("name"): file.get("id") for file in response.get("files", [])} diff --git a/automation_file/remote/google_drive/share_ops.py b/automation_file/remote/google_drive/share_ops.py index 921e964..b29d2a9 100644 --- a/automation_file/remote/google_drive/share_ops.py +++ b/automation_file/remote/google_drive/share_ops.py @@ -2,13 +2,12 @@ from __future__ import annotations -from googleapiclient.errors import HttpError - from automation_file.logging_config import file_automation_logger -from automation_file.remote.google_drive.client import driver_instance +from automation_file.remote.google_drive.client import drive_http_error, driver_instance def _create_permission(file_id: str, body: dict, description: str) -> dict | None: + http_error = drive_http_error() try: response = ( driver_instance.require_service() @@ -18,7 +17,7 @@ def _create_permission(file_id: str, body: dict, description: str) -> dict | Non ) file_automation_logger.info("drive_share (%s): file=%s", description, file_id) return response - except HttpError as error: + except http_error as error: file_automation_logger.error("drive_share (%s) failed: %r", description, error) return None diff --git a/automation_file/remote/google_drive/upload_ops.py b/automation_file/remote/google_drive/upload_ops.py index 530fc90..025aac1 100644 --- a/automation_file/remote/google_drive/upload_ops.py +++ b/automation_file/remote/google_drive/upload_ops.py @@ -5,12 +5,13 @@ import mimetypes from pathlib import Path -from googleapiclient.errors import HttpError -from googleapiclient.http import MediaFileUpload - from automation_file.exceptions import FileNotExistsException from automation_file.logging_config import file_automation_logger -from automation_file.remote.google_drive.client import driver_instance +from automation_file.remote.google_drive.client import ( + drive_http_error, + drive_media, + driver_instance, +) def _guess_mime(path: Path) -> str: @@ -19,8 +20,9 @@ def _guess_mime(path: Path) -> str: def _upload(path: Path, metadata: dict, description: str) -> dict | None: + http_error = drive_http_error() try: - media = MediaFileUpload(str(path), mimetype=_guess_mime(path), resumable=True) + media = drive_media().MediaFileUpload(str(path), mimetype=_guess_mime(path), resumable=True) response = ( driver_instance.require_service() .files() @@ -29,7 +31,7 @@ def _upload(path: Path, metadata: dict, description: str) -> dict | None: ) file_automation_logger.info("drive_upload (%s): %s", description, path) return response - except HttpError as error: + except http_error as error: file_automation_logger.error("drive_upload (%s) failed: %r", description, error) return None diff --git a/automation_file/remote/s3/client.py b/automation_file/remote/s3/client.py index 05b6cfc..66f62d9 100644 --- a/automation_file/remote/s3/client.py +++ b/automation_file/remote/s3/client.py @@ -4,17 +4,12 @@ from typing import Any +from automation_file.core.optional import require_module from automation_file.logging_config import file_automation_logger def _import_boto3() -> Any: - try: - import boto3 - except ImportError as error: - raise RuntimeError( - "boto3 import failed — reinstall `automation_file` to restore the S3 backend" - ) from error - return boto3 + return require_module("boto3", extra="s3") class S3Client: diff --git a/dev.toml b/dev.toml index f21b8fb..45e79f0 100644 --- a/dev.toml +++ b/dev.toml @@ -16,28 +16,20 @@ readme = { file = "README.md", content-type = "text/markdown" } requires-python = ">=3.10" license = "MIT" license-files = ["LICENSE"] +# The base install: the action engine, local operations, HTTP, the storage layer's local and +# in-memory backends, pipelines, events, audit, triggers and the servers. No cloud SDK and no GUI +# toolkit: each of those is an extra below, imported only when its feature is used. dependencies = [ - "google-api-python-client>=2.100.0", - "google-auth-httplib2>=0.2.0", - "google-auth-oauthlib>=1.2.0", "requests>=2.31.0", "tqdm>=4.66.0", - "boto3>=1.34.0", - "azure-storage-blob>=12.19.0", - "dropbox>=11.36.2", - "paramiko>=3.4.0", - "PySide6>=6.6.0", "watchdog>=4.0.0", "cryptography>=50.0.0", "prometheus_client>=0.26.0", "defusedxml>=0.7.1", "je_action_core>=0.0.2", "PyYAML>=6.0.3", - "pyarrow>=25.0.1", "opentelemetry-api>=1.44.0", "opentelemetry-sdk>=1.44.0", - "msal>=1.39.0", - "boxsdk>=10.15.0,<11", "tomli>=2.0.1; python_version<\"3.11\"" ] classifiers = [ @@ -54,6 +46,47 @@ classifiers = [ ] [project.optional-dependencies] +# One extra per backend, so each can be installed on its own. `ftp` and `webdav` need nothing beyond +# the base install (ftplib is in the standard library; requests and defusedxml are base +# dependencies); they exist so that every backend has an extra of its name. +s3 = ["boto3>=1.34.0"] +azure = ["azure-storage-blob>=12.19.0"] +gdrive = [ + "google-api-python-client>=2.100.0", + "google-auth-httplib2>=0.2.0", + "google-auth-oauthlib>=1.2.0" +] +dropbox = ["dropbox>=11.36.2"] +sftp = ["paramiko>=3.4.0"] +ftp = [] +webdav = [] +smb = ["smbprotocol>=1.13.0"] +fsspec = ["fsspec>=2024.2.0"] +onedrive = ["msal>=1.39.0"] +box = ["boxsdk>=10.15.0,<11"] +parquet = ["pyarrow>=25.0.1"] +gui = ["PySide6>=6.6.0"] +# Everything above. The list is written out because the two channels have different package names, +# so neither can refer to its own extras by name and stay identical to the other. +all = [ + "boto3>=1.34.0", + "azure-storage-blob>=12.19.0", + "google-api-python-client>=2.100.0", + "google-auth-httplib2>=0.2.0", + "google-auth-oauthlib>=1.2.0", + "dropbox>=11.36.2", + "paramiko>=3.4.0", + "smbprotocol>=1.13.0", + "fsspec>=2024.2.0", + "msal>=1.39.0", + "boxsdk>=10.15.0,<11", + "pyarrow>=25.0.1", + "PySide6>=6.6.0" +] +test = [ + "pytest>=8.0.0", + "pytest-cov>=5.0.0" +] dev = [ "pytest>=8.0.0", "pytest-cov>=5.0.0", diff --git a/dev_requirements.txt b/dev_requirements.txt index aa1b519..f88c603 100644 --- a/dev_requirements.txt +++ b/dev_requirements.txt @@ -1,4 +1,4 @@ -automation_file_dev +automation_file_dev[all] google-api-python-client google-auth-httplib2 google-auth-oauthlib diff --git a/docs/source/API/remote.rst b/docs/source/API/remote.rst index 611bc8b..043dcd2 100644 --- a/docs/source/API/remote.rst +++ b/docs/source/API/remote.rst @@ -34,7 +34,7 @@ Google Drive S3 --- -Bundled with ``automation_file``; registered automatically by +Its SDK comes with the extra of its name; its actions are registered automatically by :func:`automation_file.core.action_registry.build_default_registry`. .. automodule:: automation_file.remote.s3.client @@ -55,7 +55,7 @@ Bundled with ``automation_file``; registered automatically by Azure Blob ---------- -Bundled with ``automation_file``; registered automatically by +Its SDK comes with the extra of its name; its actions are registered automatically by :func:`automation_file.core.action_registry.build_default_registry`. .. automodule:: automation_file.remote.azure_blob.client @@ -76,7 +76,7 @@ Bundled with ``automation_file``; registered automatically by Dropbox ------- -Bundled with ``automation_file``; registered automatically by +Its SDK comes with the extra of its name; its actions are registered automatically by :func:`automation_file.core.action_registry.build_default_registry`. .. automodule:: automation_file.remote.dropbox_api.client @@ -97,7 +97,7 @@ Bundled with ``automation_file``; registered automatically by SFTP ---- -Bundled with ``automation_file``; registered automatically by +Its SDK comes with the extra of its name; its actions are registered automatically by :func:`automation_file.core.action_registry.build_default_registry`. Uses :class:`paramiko.RejectPolicy` — unknown hosts are never auto-added. @@ -119,7 +119,7 @@ Bundled with ``automation_file``; registered automatically by FTP / FTPS ---------- -Bundled with ``automation_file``; registered automatically by +Its SDK comes with the extra of its name; its actions are registered automatically by :func:`automation_file.core.action_registry.build_default_registry`. Supports plain FTP and explicit FTPS (via ``FTP_TLS`` + ``auth()``). diff --git a/docs/source/Eng/usage/cloud.rst b/docs/source/Eng/usage/cloud.rst index 5ce9582..86e765b 100644 --- a/docs/source/Eng/usage/cloud.rst +++ b/docs/source/Eng/usage/cloud.rst @@ -1,10 +1,14 @@ Cloud and SFTP backends ======================= -Every backend (Google Drive, S3, Azure Blob, Dropbox, SFTP) is bundled -with ``automation_file`` and auto-registered by -:func:`~automation_file.core.action_registry.build_default_registry`. -There is no extra install step — call ``later_init`` on the singleton and go: +Every backend's actions are registered by +:func:`~automation_file.core.action_registry.build_default_registry`, whether +or not its SDK is installed. The SDK itself comes with an extra: +``pip install "automation_file[s3]"`` (``azure``, ``gdrive``, ``dropbox``, +``sftp``, ``onedrive``, ``box``, ``smb``, ``fsspec``), or ``[all]`` for every +backend. Using a backend whose extra is missing raises +``OptionalDependencyException`` with the command to run. With the SDK in place, +call ``later_init`` on the singleton and go: .. code-block:: python diff --git a/docs/source/Zh-CN/usage/cloud.rst b/docs/source/Zh-CN/usage/cloud.rst index 4848d1e..f3e76ed 100644 --- a/docs/source/Zh-CN/usage/cloud.rst +++ b/docs/source/Zh-CN/usage/cloud.rst @@ -1,10 +1,13 @@ 云与 SFTP 后端 ============== -每个后端（Google Drive、S3、Azure Blob、Dropbox、SFTP）都是 -``automation_file`` 自带的，并由 -:func:`~automation_file.core.action_registry.build_default_registry` 自动注册。 -无需额外安装步骤——在单例上调用 ``later_init`` 即可使用： +每个后端的动作都由 +:func:`~automation_file.core.action_registry.build_default_registry` 注册，无论其 +SDK 是否已安装。SDK 本身则由 extra 提供：``pip install "automation_file[s3]"`` +（另有 ``azure``、``gdrive``、``dropbox``、``sftp``、``onedrive``、``box``、 +``smb``、``fsspec``），或用 ``[all]`` 安装所有后端。使用缺少 extra 的后端时会抛出 +``OptionalDependencyException``，信息中附有要执行的命令。SDK 就绪后，在单例上调用 +``later_init`` 即可使用： .. code-block:: python diff --git a/docs/source/Zh-TW/usage/cloud.rst b/docs/source/Zh-TW/usage/cloud.rst index 39f1e07..07c627c 100644 --- a/docs/source/Zh-TW/usage/cloud.rst +++ b/docs/source/Zh-TW/usage/cloud.rst @@ -1,10 +1,13 @@ 雲端與 SFTP 後端 ================ -每個後端（Google Drive、S3、Azure Blob、Dropbox、SFTP）皆內建於 -``automation_file``，並由 -:func:`~automation_file.core.action_registry.build_default_registry` 自動註冊。 -無須額外安裝步驟——在單例上呼叫 ``later_init`` 即可使用： +每個後端的動作都由 +:func:`~automation_file.core.action_registry.build_default_registry` 註冊，無論其 +SDK 是否已安裝。SDK 本身則由 extra 提供：``pip install "automation_file[s3]"`` +（另有 ``azure``、``gdrive``、``dropbox``、``sftp``、``onedrive``、``box``、 +``smb``、``fsspec``），或以 ``[all]`` 安裝所有後端。使用缺少 extra 的後端時會拋出 +``OptionalDependencyException``，訊息中附有要執行的指令。SDK 就緒後，在單例上呼叫 +``later_init`` 即可使用： .. code-block:: python diff --git a/docs/updates/2026-10.md b/docs/updates/2026-10.md index d1b0caa..ec38e8d 100644 --- a/docs/updates/2026-10.md +++ b/docs/updates/2026-10.md @@ -307,3 +307,19 @@ Index and query commands: [README.md](README.md). New entries go at the end. - **Fix**: the node lists `local:// s3:// azure:// dropbox:// sftp:// ftp://`, as the READMEs' diagram does. `grep -rn "FA_cross_copy\|[^g]drive://" docs/source README*.md` now prints nothing (`gdrive://`, the storage layer's scheme, is a different thing). - **Files**: the three `architecture.rst` pages, `docs/updates/2026-10.md` (the pointer added to U-20261008-07), `progress.md` (an empty heading removed). - **Open items**: none. + +## U-20261008-09 · 2026-10-08 · Backend SDKs and the GUI toolkit become extras · #done #packaging #roadmap #decision + +- **What**: `progress.md` #10 and #12 (roadmap section 3). The owner's instruction of 2026-10-08 to carry out the whole roadmap settles #10 in the roadmap's favour, so the earlier rule that the backends and PySide6 are first-class runtime dependencies is replaced in `CLAUDE.md` and `architecture.md` §7. + - **Base dependencies** (`stable.toml`, `dev.toml`): `requests`, `tqdm`, `watchdog`, `cryptography`, `prometheus_client`, `defusedxml`, `je_action_core`, `PyYAML`, `opentelemetry-api`, `opentelemetry-sdk`, and `tomli` below Python 3.11. + - **Extras**: `s3` (boto3), `azure` (azure-storage-blob), `gdrive` (the three Google packages), `dropbox`, `sftp` (paramiko), `smb` (smbprotocol), `fsspec`, `onedrive` (msal), `box` (boxsdk), `parquet` (pyarrow), `gui` (PySide6); `ftp` and `webdav` exist and are empty (standard library and base dependencies suffice); `all` lists every package, written out because the two channels cannot name their own extras and stay identical; `test` (pytest, pytest-cov); `dev` as before. + - **Import time**: the Google Drive modules were the only ones importing their SDK at module level; they now import it at the moment of use (`drive_http_error()`, `drive_media()`, and inside `GoogleDriveClient.later_init`, which is split into `_load_credentials` / `_renew_credentials`). `import automation_file` works with every optional package impossible to import, builds all its commands, and loads none of them when they are installed. + - **Missing extra**: `automation_file/core/optional.py` (`require_module(name, extra=...)`, `install_hint`, the `EXTRAS` table) and `OptionalDependencyException` (a `FileAutomationException` and a `RuntimeError`): "boto3 is not installed; the S3 backend needs it: pip install "automation_file[s3]"". S3, Azure Blob, Dropbox, Google Drive and the Parquet operations use it; Box and the fsspec bridge keep their own exception types and name the extra in the message. The SFTP, OneDrive and SMB clients still say "reinstall" or name the bare package: they are being changed on other branches and follow when those merge. + - Found on the way and fixed: `GoogleDriveClient` logged the token path, against `CLAUDE.md` › Secrets and credentials. It logs "loading the saved token". +- **CI** (`ci-dev.yml`, `ci-stable.yml`): the `pytest` matrix installs `.[all,test]`. New jobs after `lint`: `minimal` (`.[test]` only, the whole suite) and `extras` (a matrix of the thirteen extras, each installed alone, the whole suite). `publish-dev` needs all four. The two new jobs have never run. +- **Tests**: `tests/test_optional_dependencies.py`: a subprocess with an import blocker proves the import and the registry without any optional package and checks the hint of four features; a second one proves nothing optional is loaded by the import; the extras of both TOMLs match `core.optional.EXTRAS`, `all` is their union, and no optional distribution is a base dependency. `tests/test_storage_s3.py`, `test_storage_azure.py`, `test_box_ops.py` and the six Parquet tests skip where their extra is missing. +- **Result / numbers**: with every extra installed: 1581 passed, 22 skipped, 0 failed. In a virtualenv holding only the base dependencies and pytest: 1191 passed, 28 skipped, 0 failed. `ruff check`, `ruff format --check` and `mypy automation_file` (182 files) pass. Python 3.14.7 on Windows. +- **Consumers**: PyBreeze declares `automation-file` and relied on it for the SDKs. Its local branch `deps/automation-file-all-extra` (one commit on `origin/dev`, not pushed) declares `automation-file[all]` in `dev.toml`, `pyproject.toml` and `requirements.txt`; pip accepts the extra against today's release with a warning, so it can merge before or after. TestPioneer uses `download_file` and `unzip_all`, which are in the base install. +- **Docs**: the installation section with the extras table in the three READMEs, the three `usage/cloud.rst` pages and `API/remote.rst` (no longer "bundled"), `CLAUDE.md` (Branching & CI, Development, key types), `architecture.md` §2, §5, §6 and §7, `requirements.txt` and `dev_requirements.txt` (`[all]`). +- **Files**: `stable.toml`, `dev.toml`, `requirements.txt`, `dev_requirements.txt`, `.github/workflows/ci-dev.yml`, `.github/workflows/ci-stable.yml`, `automation_file/core/optional.py`, `automation_file/exceptions.py`, `automation_file/remote/google_drive/*.py`, `automation_file/remote/s3/client.py`, `automation_file/remote/azure_blob/client.py`, `automation_file/remote/dropbox_api/client.py`, `automation_file/remote/dropbox_api/upload_ops.py`, `automation_file/remote/box/client.py`, `automation_file/remote/fsspec_bridge.py`, `automation_file/local/data_ops.py`, the tests named above, the documentation above, `progress.md`. +- **Open items**: `progress.md` #29 (PyBreeze's branch), #30 (the three clients that still name no extra). diff --git a/docs/updates/README.md b/docs/updates/README.md index 2944cde..29bce48 100644 --- a/docs/updates/README.md +++ b/docs/updates/README.md @@ -58,6 +58,7 @@ In the same commit: delete the item from `progress.md`, add a `#done` entry here | ID | Date | Title | Tags | Batch | |---|---|---|---|---| +| U-20261008-09 | 2026-10-08 | Backend SDKs and the GUI toolkit become extras | #done #packaging #roadmap #decision | [2026-10](2026-10.md) | | U-20261008-08 | 2026-10-08 | Three architecture diagrams still listed drive:// | #incident #docs | [2026-10](2026-10.md) | | U-20261008-07 | 2026-10-08 | The cloud pages describe FA_copy_between, not FA_cross_copy | #done #docs | [2026-10](2026-10.md) | | U-20261008-06 | 2026-10-08 | Event model, event bus and storage observers | #events #roadmap #storage | [2026-10](2026-10.md) | @@ -103,5 +104,5 @@ In the same commit: delete the item from `progress.md`, add a `#done` entry here | File | Period | Entries | |---|---|---:| -| [2026-10.md](2026-10.md) | 2026-10 | 19 | +| [2026-10.md](2026-10.md) | 2026-10 | 20 | | [2026-09.md](2026-09.md) | 2026-09 | 21 | diff --git a/progress.md b/progress.md index 5975449..9b5a39d 100644 --- a/progress.md +++ b/progress.md @@ -10,9 +10,7 @@ Items #10 to #26 are what is left of the 1.0 roadmap (`docs/FILEAUTOMATION-1.0-R ### Architecture and packaging (roadmap M1) -- **#10** [DECIDE] Optional extras. Roadmap §3 wants a light base install with the SDKs and the GUI under `[s3]`, `[gdrive]`, `[azure]`, `[dropbox]`, `[sftp]`, `[ftp]`, `[webdav]`, `[smb]`, `[fsspec]`, `[gui]`, `[all]`, `[test]`. `CLAUDE.md` › Branching & CI and `architecture.md` §7 say the opposite: the backends and PySide6 are first-class runtime dependencies and must not move under extras. PyBreeze declares `automation-file` and gets the GUI and the SDKs through it (`architecture.md` §6), so moving them breaks it unless it asks for `[all]` in the same round. Decide which rule wins and whether `automation_file_dev` follows. - **#11** [DECIDE] Public API policy and deprecation policy (roadmap M1, M9): which names are frozen at 1.0 and how a name is retired. The storage layer is documented as provisional until then. -- **#12** `import automation_file` loads `requests`, `cryptography`, `watchdog`, `googleapiclient`, `google.auth`, `google_auth_oauthlib`, `prometheus_client`, `opentelemetry`, `tqdm` and `defusedxml` at module level, because the facade imports every module. Roadmap §3 requires that the base package imports no optional SDK at import time. `boto3`, `azure`, `dropbox`, `paramiko`, `PySide6`, `pyarrow`, `msal` and `boxsdk` are already lazy. What counts as optional depends on #10. ### Universal storage layer (roadmap M2) @@ -36,3 +34,9 @@ Items #10 to #26 are what is left of the 1.0 roadmap (`docs/FILEAUTOMATION-1.0-R - **#24** UI 2.0 (roadmap §11, M7). Not before the APIs of #13 to #23 are stable (roadmap §20). - **#25** Semantic MCP tools (roadmap §12, M8): `file_*`, `storage_*`, `pipeline_*`, `integrity_status`, `audit_search`, with a permission model and dry run, next to the existing `FA_*` bridge. - **#26** Release engineering and 1.0 (roadmap §13, M9): contract and integration tests in the PR gate, PyPI Trusted Publishing, SemVer, migration guide, API freeze. + +### Packaging follow-ups + +- **#29** [BLOCKED] PyBreeze has to declare `automation-file[all]` before the stable release that splits the extras reaches users, or it installs without the SDKs it relied on. The change exists on PyBreeze's local branch `deps/automation-file-all-extra` (one commit on its `origin/dev`: `dev.toml`, `pyproject.toml`, `requirements.txt`), not pushed: it waits for someone to open the PR there and for PyBreeze's own update log. PyBreeze's checkout was on `docs/tutorials` with other work, so nothing else was touched. +- **#30** `remote/sftp/client.py` ("reinstall `automation_file`"), `remote/onedrive/client.py` (the same) and `remote/smb/client.py` ("install `smbprotocol`") still report a missing SDK without naming the extra. Change them to `core.optional.require_module` or `install_hint` once the adapter branches that edit those files are merged. + diff --git a/requirements.txt b/requirements.txt index c4c1b88..2c0f480 100644 --- a/requirements.txt +++ b/requirements.txt @@ -1,4 +1,4 @@ -automation_file +automation_file[all] google-api-python-client google-auth-httplib2 google-auth-oauthlib diff --git a/stable.toml b/stable.toml index 5e45eab..7e3bf47 100644 --- a/stable.toml +++ b/stable.toml @@ -14,28 +14,20 @@ readme = { file = "README.md", content-type = "text/markdown" } requires-python = ">=3.10" license = "MIT" license-files = ["LICENSE"] +# The base install: the action engine, local operations, HTTP, the storage layer's local and +# in-memory backends, pipelines, events, audit, triggers and the servers. No cloud SDK and no GUI +# toolkit: each of those is an extra below, imported only when its feature is used. dependencies = [ - "google-api-python-client>=2.100.0", - "google-auth-httplib2>=0.2.0", - "google-auth-oauthlib>=1.2.0", "requests>=2.31.0", "tqdm>=4.66.0", - "boto3>=1.34.0", - "azure-storage-blob>=12.19.0", - "dropbox>=11.36.2", - "paramiko>=3.4.0", - "PySide6>=6.6.0", "watchdog>=4.0.0", "cryptography>=50.0.0", "prometheus_client>=0.26.0", "defusedxml>=0.7.1", "je_action_core>=0.0.2", "PyYAML>=6.0.3", - "pyarrow>=25.0.1", "opentelemetry-api>=1.44.0", "opentelemetry-sdk>=1.44.0", - "msal>=1.39.0", - "boxsdk>=10.15.0,<11", "tomli>=2.0.1; python_version<\"3.11\"" ] classifiers = [ @@ -52,6 +44,47 @@ classifiers = [ ] [project.optional-dependencies] +# One extra per backend, so each can be installed on its own. `ftp` and `webdav` need nothing beyond +# the base install (ftplib is in the standard library; requests and defusedxml are base +# dependencies); they exist so that every backend has an extra of its name. +s3 = ["boto3>=1.34.0"] +azure = ["azure-storage-blob>=12.19.0"] +gdrive = [ + "google-api-python-client>=2.100.0", + "google-auth-httplib2>=0.2.0", + "google-auth-oauthlib>=1.2.0" +] +dropbox = ["dropbox>=11.36.2"] +sftp = ["paramiko>=3.4.0"] +ftp = [] +webdav = [] +smb = ["smbprotocol>=1.13.0"] +fsspec = ["fsspec>=2024.2.0"] +onedrive = ["msal>=1.39.0"] +box = ["boxsdk>=10.15.0,<11"] +parquet = ["pyarrow>=25.0.1"] +gui = ["PySide6>=6.6.0"] +# Everything above. The list is written out because the two channels have different package names, +# so neither can refer to its own extras by name and stay identical to the other. +all = [ + "boto3>=1.34.0", + "azure-storage-blob>=12.19.0", + "google-api-python-client>=2.100.0", + "google-auth-httplib2>=0.2.0", + "google-auth-oauthlib>=1.2.0", + "dropbox>=11.36.2", + "paramiko>=3.4.0", + "smbprotocol>=1.13.0", + "fsspec>=2024.2.0", + "msal>=1.39.0", + "boxsdk>=10.15.0,<11", + "pyarrow>=25.0.1", + "PySide6>=6.6.0" +] +test = [ + "pytest>=8.0.0", + "pytest-cov>=5.0.0" +] dev = [ "pytest>=8.0.0", "pytest-cov>=5.0.0", diff --git a/tests/test_box_ops.py b/tests/test_box_ops.py index 10337fb..a093479 100644 --- a/tests/test_box_ops.py +++ b/tests/test_box_ops.py @@ -12,6 +12,10 @@ from typing import Any import pytest + +pytest.importorskip("box_sdk_gen", reason="needs the box extra") + +# pylint: disable=wrong-import-position # importorskip must precede these imports from box_sdk_gen.schemas.file_base import FileBaseTypeField from automation_file import ( diff --git a/tests/test_data_ops_yaml_parquet.py b/tests/test_data_ops_yaml_parquet.py index a8afb19..6b91835 100644 --- a/tests/test_data_ops_yaml_parquet.py +++ b/tests/test_data_ops_yaml_parquet.py @@ -2,6 +2,7 @@ from __future__ import annotations +import importlib.util from pathlib import Path import pytest @@ -18,6 +19,10 @@ ) from automation_file.exceptions import FileNotExistsException +needs_pyarrow = pytest.mark.skipif( + importlib.util.find_spec("pyarrow") is None, reason="needs the parquet extra" +) + # --- YAML ----------------------------------------------------------------- @@ -87,6 +92,7 @@ def test_yaml_handles_missing_file(tmp_path: Path) -> None: # --- Parquet -------------------------------------------------------------- +@needs_pyarrow def test_parquet_write_and_read_roundtrip(tmp_path: Path) -> None: path = tmp_path / "data.parquet" records = [{"id": 1, "name": "alice"}, {"id": 2, "name": "bob"}] @@ -94,29 +100,34 @@ def test_parquet_write_and_read_roundtrip(tmp_path: Path) -> None: assert parquet_read(str(path)) == records +@needs_pyarrow def test_parquet_read_respects_limit(tmp_path: Path) -> None: path = tmp_path / "data.parquet" parquet_write(str(path), [{"i": n} for n in range(5)]) assert parquet_read(str(path), limit=2) == [{"i": 0}, {"i": 1}] +@needs_pyarrow def test_parquet_read_projects_columns(tmp_path: Path) -> None: path = tmp_path / "data.parquet" parquet_write(str(path), [{"id": 1, "name": "alice", "team": "red"}]) assert parquet_read(str(path), columns=["id", "team"]) == [{"id": 1, "team": "red"}] +@needs_pyarrow def test_parquet_write_rejects_non_list(tmp_path: Path) -> None: path = tmp_path / "data.parquet" with pytest.raises(DataOpsException): parquet_write(str(path), {"not": "a list"}) # type: ignore[arg-type] +@needs_pyarrow def test_parquet_read_rejects_missing_file(tmp_path: Path) -> None: with pytest.raises(FileNotExistsException): parquet_read(str(tmp_path / "gone.parquet")) +@needs_pyarrow def test_csv_to_parquet_roundtrip(tmp_path: Path) -> None: csv_path = tmp_path / "a.csv" csv_path.write_text("id,name\n1,alice\n2,bob\n", encoding="utf-8") diff --git a/tests/test_dev_release.py b/tests/test_dev_release.py index 90c6cc3..7d4db35 100644 --- a/tests/test_dev_release.py +++ b/tests/test_dev_release.py @@ -153,7 +153,8 @@ def _publish_job() -> str: def test_the_workflow_publishes_only_a_tested_push_to_dev(): job = _publish_job() - assert "needs: [lint, pytest]" in job + # Lint, the full matrix, the base install and each extra on its own all come first. + assert "needs: [lint, pytest, minimal, extras]" in job assert "if: github.event_name == 'push' && github.ref == 'refs/heads/dev'" in job diff --git a/tests/test_optional_dependencies.py b/tests/test_optional_dependencies.py new file mode 100644 index 0000000..b703f3f --- /dev/null +++ b/tests/test_optional_dependencies.py @@ -0,0 +1,183 @@ +"""The base install needs no cloud SDK and no GUI toolkit; each lives in an extra. + +Three things are held here: + +* ``import automation_file`` succeeds when every optional package is impossible to + import, and builds the whole action registry; +* it loads none of them even when they are installed; +* the extras in ``stable.toml`` / ``dev.toml`` match what the code asks for, and no + optional package sits among the base dependencies. +""" + +from __future__ import annotations + +import json +import re +import subprocess +import sys +from pathlib import Path + +import pytest + +from automation_file.core.optional import EXTRAS, install_hint, require_module +from automation_file.exceptions import FileAutomationException, OptionalDependencyException + +if sys.version_info >= (3, 11): + import tomllib +else: + import tomli as tomllib # declared in *.toml for Python<3.11 + +REPO_ROOT = Path(__file__).resolve().parents[1] + +#: Top-level import names of every package that only an extra installs. +OPTIONAL_ROOTS = ( + "boto3", + "botocore", + "azure", + "dropbox", + "paramiko", + "googleapiclient", + "google_auth_oauthlib", + "google_auth_httplib2", + "msal", + "box_sdk_gen", + "boxsdk", + "pyarrow", + "fsspec", + "smbclient", + "smbprotocol", + "PySide6", +) +#: Distribution names that must never be base dependencies. +OPTIONAL_DISTRIBUTIONS = { + "boto3", + "azure-storage-blob", + "dropbox", + "paramiko", + "google-api-python-client", + "google-auth-httplib2", + "google-auth-oauthlib", + "msal", + "boxsdk", + "pyarrow", + "fsspec", + "smbprotocol", + "pyside6", +} + +_PROBE = """ +import importlib.abc, json, sys + +blocked = set(json.loads(sys.argv[1])) + + +class Blocker(importlib.abc.MetaPathFinder): + def find_spec(self, fullname, path=None, target=None): + if fullname.split(".")[0] in blocked: + raise ImportError("blocked: " + fullname) + return None + + +if blocked: + sys.meta_path.insert(0, Blocker()) +import automation_file +from automation_file.exceptions import OptionalDependencyException + +# What the import itself pulled in, measured before any feature is used. +loaded = sorted({name.split(".")[0] for name in sys.modules} & set(json.loads(sys.argv[2]))) +messages = {} +features = { + "s3": lambda: automation_file.s3_instance.later_init(), + "azure": lambda: automation_file.azure_blob_instance.later_init(connection_string="x"), + "dropbox": lambda: automation_file.dropbox_instance.later_init("token"), + "gdrive": lambda: automation_file.drive_search_all_file(), +} +for name, call in features.items() if blocked else (): + try: + call() + except OptionalDependencyException as error: + messages[name] = str(error) + except Exception as error: + messages[name] = "OTHER " + type(error).__name__ + else: + messages[name] = "NO ERROR" +print(json.dumps({ + "commands": len(automation_file.executor.registry.event_dict), + "loaded": loaded, + "messages": messages, +})) +""" + + +def _probe(blocked: tuple[str, ...]) -> dict: + result = subprocess.run( # nosec B603 - fixed argv: this interpreter and a script defined above + [sys.executable, "-c", _PROBE, json.dumps(blocked), json.dumps(OPTIONAL_ROOTS)], + cwd=REPO_ROOT, + capture_output=True, + text=True, + timeout=120, + check=False, + ) + assert result.returncode == 0, result.stderr[-2000:] + return json.loads(result.stdout.strip().splitlines()[-1]) + + +def test_the_package_imports_without_any_optional_dependency() -> None: + report = _probe((*OPTIONAL_ROOTS, "google")) + assert report["commands"] > 100 + assert report["loaded"] == [] + for extra in ("s3", "azure", "dropbox", "gdrive"): + assert install_hint(extra) in report["messages"][extra], report["messages"] + + +def test_importing_the_package_loads_no_optional_dependency() -> None: + report = _probe(()) + assert report["loaded"] == [] + + +def test_require_module_returns_the_module_or_names_the_extra() -> None: + assert require_module("json", extra="s3").dumps({}) == "{}" + with pytest.raises(OptionalDependencyException) as caught: + require_module("no_such_sdk_for_automation_file.sub", extra="s3") + message = str(caught.value) + assert message.startswith("no_such_sdk_for_automation_file is not installed; the S3 backend") + assert message.endswith('pip install "automation_file[s3]"') + assert isinstance(caught.value, FileAutomationException) + assert isinstance(caught.value, RuntimeError) + assert isinstance(caught.value.__cause__, ImportError) + with pytest.raises(OptionalDependencyException, match="the reports feature"): + require_module("no_such_sdk_for_automation_file", extra="reports") + + +def _name(requirement: str) -> str: + return re.split(r"[<>=!~;\[ ]", requirement, maxsplit=1)[0].strip().lower().replace("_", "-") + + +@pytest.fixture(params=["stable.toml", "dev.toml"]) +def project(request: pytest.FixtureRequest) -> dict: + with (REPO_ROOT / request.param).open("rb") as handle: + return tomllib.load(handle)["project"] + + +def test_no_optional_package_is_a_base_dependency(project: dict) -> None: + base = {_name(requirement) for requirement in project["dependencies"]} + assert base & OPTIONAL_DISTRIBUTIONS == set() + + +def test_every_extra_the_code_names_is_declared(project: dict) -> None: + extras = project["optional-dependencies"] + assert set(EXTRAS) <= set(extras) + assert {"all", "test", "dev"} <= set(extras) + + +def test_all_is_the_union_of_the_feature_extras(project: dict) -> None: + extras = project["optional-dependencies"] + union = {requirement for name in EXTRAS for requirement in extras[name]} + assert set(extras["all"]) == union + assert {_name(requirement) for requirement in union} == OPTIONAL_DISTRIBUTIONS + + +def test_the_extras_that_need_nothing_are_empty(project: dict) -> None: + extras = project["optional-dependencies"] + assert extras["ftp"] == [] + assert extras["webdav"] == [] diff --git a/tests/test_storage_azure.py b/tests/test_storage_azure.py index fd63602..92b4eae 100644 --- a/tests/test_storage_azure.py +++ b/tests/test_storage_azure.py @@ -18,6 +18,10 @@ from typing import Any, BinaryIO import pytest + +pytest.importorskip("azure.storage.blob", reason="needs the azure extra") + +# pylint: disable=wrong-import-position # importorskip must precede these imports from azure.core.exceptions import ( ClientAuthenticationError, HttpResponseError, @@ -43,7 +47,12 @@ StorageURIException, ) from automation_file.remote.azure_blob.client import azure_blob_instance -from automation_file.storage import AzureStorage, File, StorageBackend, StorageResolver +from automation_file.storage import ( + AzureStorage, + File, + StorageBackend, + StorageResolver, +) from tests.storage_contract import StorageContract diff --git a/tests/test_storage_s3.py b/tests/test_storage_s3.py index 37f58ad..41fd1d4 100644 --- a/tests/test_storage_s3.py +++ b/tests/test_storage_s3.py @@ -15,14 +15,21 @@ from pathlib import Path from typing import Any -import boto3 import pytest -from botocore import UNSIGNED -from botocore.config import Config -from botocore.exceptions import ClientError, EndpointConnectionError, NoCredentialsError -from botocore.stub import Stubber -from automation_file.exceptions import ( +boto3 = pytest.importorskip("boto3", reason="needs the s3 extra") + +# pylint: disable=wrong-import-position # importorskip must precede these imports +from botocore import UNSIGNED # noqa: E402 +from botocore.config import Config # noqa: E402 +from botocore.exceptions import ( # noqa: E402 + ClientError, + EndpointConnectionError, + NoCredentialsError, +) +from botocore.stub import Stubber # noqa: E402 + +from automation_file.exceptions import ( # noqa: E402 StorageException, StorageNotFoundException, StoragePermissionException, @@ -30,9 +37,9 @@ StorageUnavailableException, StorageURIException, ) -from automation_file.remote.s3.client import s3_instance -from automation_file.storage import File, S3Storage, StorageBackend, StorageResolver -from tests.storage_contract import StorageContract +from automation_file.remote.s3.client import s3_instance # noqa: E402 +from automation_file.storage import File, S3Storage, StorageBackend, StorageResolver # noqa: E402 +from tests.storage_contract import StorageContract # noqa: E402 PAGE_SIZE = 2 From e90fc87bc9cc0915fd3d3c8c324f8e80ecd2a3be Mon Sep 17 00:00:00 2001 From: JeffreyChen Date: Thu, 8 Oct 2026 12:56:23 +0800 Subject: [PATCH 30/59] test: add denied and transient failure cases to the storage contract --- README.md | 4 +- README.zh-CN.md | 4 +- README.zh-TW.md | 4 +- automation_file/storage/local_storage.py | 9 ++- docs/source/Eng/usage/storage.rst | 12 +++- docs/source/Zh-CN/usage/storage.rst | 12 +++- docs/source/Zh-TW/usage/storage.rst | 12 +++- docs/updates/2026-10.md | 11 ++++ docs/updates/README.md | 3 +- progress.md | 3 +- tests/storage_contract.py | 78 ++++++++++++++++++++++++ tests/test_storage_azure.py | 25 +++++++- tests/test_storage_local.py | 24 ++++++++ tests/test_storage_s3.py | 27 +++++++- 14 files changed, 211 insertions(+), 17 deletions(-) diff --git a/README.md b/README.md index 4b5156f..2fa783b 100644 --- a/README.md +++ b/README.md @@ -48,7 +48,7 @@ facade. - **HTTP server observability** — `GET /healthz` / `GET /readyz` probes, `GET /openapi.json` spec, and `GET /progress` WebSocket stream of live transfer snapshots - **HTMX Web UI** — `start_web_ui()` serves a read-only dashboard (health, progress, registry) that polls HTML fragments; stdlib-only HTTP plus one CDN script with SRI - **MCP (Model Context Protocol) server** — `MCPServer` bridges the registry to any MCP host (Claude Desktop, MCP CLIs) over newline-delimited JSON-RPC 2.0 on stdio; every `FA_*` action becomes an MCP tool with an auto-generated input schema -- **Universal storage layer** — `File` / `Storage` address local and remote storage with one URI syntax (`local:///…`, `s3://…`, `azure://…`, `memory://…`), one `StorageBackend` contract and one error hierarchy; local, S3, Azure Blob and in-memory backends are built in, and a 77-case contract suite checks any backend +- **Universal storage layer** — `File` / `Storage` address local and remote storage with one URI syntax (`local:///…`, `s3://…`, `azure://…`, `memory://…`), one `StorageBackend` contract and one error hierarchy; local, S3, Azure Blob and in-memory backends are built in, and an 81-case contract suite checks any backend - **Event bus** — one `Event` model with ten core events (`pipeline.*`, `task.*`, `integrity.violation`, `storage.error`, `scheduler.error`, `system.error`), severities, correlation IDs and actors; subscribe on `event_bus` by class, type or prefix - PySide6 GUI (`python -m automation_file ui`) with a tab per backend, the JSON-action runner, and dedicated tabs for Triggers, Scheduler, and live Progress - Rich CLI with one-shot subcommands plus legacy JSON-batch flags @@ -506,7 +506,7 @@ File("sandbox://jobs/42/out.csv").write(b"done") `File("s3://reports/q1.csv").copy_to("azure://backups/q1.csv")` works once both are ready. Google Drive, Dropbox, SFTP, FTP, WebDAV, SMB and fsspec are still used through their own clients and `FA_*` actions; their adapters are not written yet. Write your own by subclassing - `StorageBackend` (or `ObjectStorage` for an object store) and check it with the 77-case + `StorageBackend` (or `ObjectStorage` for an object store) and check it with the 81-case contract suite in `tests/storage_contract.py`. - **Actions** — `FA_storage_exists`, `FA_storage_stat`, `FA_storage_list`, `FA_storage_mkdir`, diff --git a/README.zh-CN.md b/README.zh-CN.md index 3c86b1a..84ed40d 100644 --- a/README.zh-CN.md +++ b/README.zh-CN.md @@ -46,7 +46,7 @@ TCP / HTTP 服务器执行的 JSON 驱动动作。内附 PySide6 GUI，每个功 - **HTTP 服务器观测端点** — `GET /healthz` / `GET /readyz` 探针、`GET /openapi.json` 规格，以及 `GET /progress`（通过 WebSocket 推送实时传输快照） - **HTMX Web UI** — `start_web_ui()` 启动只读观测仪表板（health、progress、registry），通过 HTML 片段轮询；仅用标准库 HTTP，搭配一个带 SRI 的 CDN 脚本 - **MCP（Model Context Protocol）服务器** — `MCPServer` 通过 stdio 上的 JSON-RPC 2.0（换行分隔 JSON）将注册表桥接到任意 MCP 主机（Claude Desktop、MCP CLI）；每个 `FA_*` 动作都会自动生成输入 schema 并成为 MCP 工具 -- **通用存储层** — `File` / `Storage` 以同一套 URI 语法（`local:///…`、`s3://…`、`azure://…`、`memory://…`）、同一份 `StorageBackend` 契约与同一组异常层级访问本地与远端存储；内置本地、S3、Azure Blob 与内存后端，并附带 77 个用例的契约测试套件可检查任何后端 +- **通用存储层** — `File` / `Storage` 以同一套 URI 语法（`local:///…`、`s3://…`、`azure://…`、`memory://…`）、同一份 `StorageBackend` 契约与同一组异常层级访问本地与远端存储；内置本地、S3、Azure Blob 与内存后端，并附带 81 个用例的契约测试套件可检查任何后端 - **事件总线** — 单一 `Event` 模型与十种核心事件（`pipeline.*`、`task.*`、`integrity.violation`、`storage.error`、`scheduler.error`、`system.error`），具备严重程度、关联 ID 与 actor；可以在 `event_bus` 上按类、type 或前缀订阅 - PySide6 GUI（`python -m automation_file ui`）每个后端一个页签，含 JSON 动作执行器，另有 Triggers、Scheduler、实时 Progress 专属页签 - 功能丰富的 CLI，包含一次性子命令与旧式 JSON 批量标志 @@ -501,7 +501,7 @@ File("sandbox://jobs/42/out.csv").write(b"done") `File("s3://reports/q1.csv").copy_to("azure://backups/q1.csv")` 即可运行。 Google Drive、Dropbox、SFTP、FTP、WebDAV、SMB 与 fsspec 目前仍通过各自的客户端与 `FA_*` 动作使用，其适配器尚未完成。你可以继承 `StorageBackend`（对象存储则继承 `ObjectStorage`） - 编写自己的后端，并用 `tests/storage_contract.py` 中 77 个用例的契约测试套件检查。 + 编写自己的后端，并用 `tests/storage_contract.py` 中 81 个用例的契约测试套件检查。 - **动作** — `FA_storage_exists`, `FA_storage_stat`, `FA_storage_list`, `FA_storage_mkdir`, `FA_storage_upload`, `FA_storage_download`, `FA_storage_delete`, `FA_storage_checksum`, diff --git a/README.zh-TW.md b/README.zh-TW.md index 9037e24..a81d23d 100644 --- a/README.zh-TW.md +++ b/README.zh-TW.md @@ -46,7 +46,7 @@ TCP / HTTP 伺服器執行的 JSON 驅動動作。內附 PySide6 GUI，每個功 - **HTTP 伺服器觀測端點** — `GET /healthz` / `GET /readyz` 探針、`GET /openapi.json` 規格、以及 `GET /progress`（以 WebSocket 推送即時傳輸快照） - **HTMX Web UI** — `start_web_ui()` 啟動唯讀觀測儀表板（health、progress、registry），以 HTML 片段輪詢；僅用標準函式庫 HTTP，搭配一支帶 SRI 的 CDN 腳本 - **MCP（Model Context Protocol）伺服器** — `MCPServer` 透過 stdio 上的 JSON-RPC 2.0（行分隔 JSON）將登錄表橋接到任何 MCP 主機（Claude Desktop、MCP CLI）；每個 `FA_*` 動作都會自動生成輸入 schema 並成為 MCP 工具 -- **通用儲存層** — `File` / `Storage` 以同一套 URI 語法（`local:///…`、`s3://…`、`azure://…`、`memory://…`）、同一份 `StorageBackend` 契約與同一組例外階層存取本機與遠端儲存；內建本機、S3、Azure Blob 與記憶體後端，並附 77 個案例的契約測試套件可檢查任何後端 +- **通用儲存層** — `File` / `Storage` 以同一套 URI 語法（`local:///…`、`s3://…`、`azure://…`、`memory://…`）、同一份 `StorageBackend` 契約與同一組例外階層存取本機與遠端儲存；內建本機、S3、Azure Blob 與記憶體後端，並附 81 個案例的契約測試套件可檢查任何後端 - **事件匯流排** — 單一 `Event` 模型與十種核心事件（`pipeline.*`、`task.*`、`integrity.violation`、`storage.error`、`scheduler.error`、`system.error`），具備嚴重程度、關聯 ID 與 actor；可在 `event_bus` 上依類別、type 或前綴訂閱 - PySide6 GUI（`python -m automation_file ui`）每個後端一個分頁，含 JSON 動作執行器，另有 Triggers、Scheduler、即時 Progress 專屬分頁 - 功能豐富的 CLI，包含一次性子指令與舊式 JSON 批次旗標 @@ -501,7 +501,7 @@ File("sandbox://jobs/42/out.csv").write(b"done") `File("s3://reports/q1.csv").copy_to("azure://backups/q1.csv")` 即可運作。 Google Drive、Dropbox、SFTP、FTP、WebDAV、SMB 與 fsspec 目前仍透過各自的用戶端與 `FA_*` 動作使用，其轉接器尚未完成。你可以繼承 `StorageBackend`（物件儲存則繼承 `ObjectStorage`） - 撰寫自己的後端，並用 `tests/storage_contract.py` 中 77 個案例的契約測試套件檢查。 + 撰寫自己的後端，並用 `tests/storage_contract.py` 中 81 個案例的契約測試套件檢查。 - **動作** — `FA_storage_exists`, `FA_storage_stat`, `FA_storage_list`, `FA_storage_mkdir`, `FA_storage_upload`, `FA_storage_download`, `FA_storage_delete`, `FA_storage_checksum`, diff --git a/automation_file/storage/local_storage.py b/automation_file/storage/local_storage.py index bee045c..cd47ab4 100644 --- a/automation_file/storage/local_storage.py +++ b/automation_file/storage/local_storage.py @@ -28,7 +28,11 @@ from typing import BinaryIO from automation_file.core.checksum import file_checksum -from automation_file.exceptions import StorageException, StoragePermissionException +from automation_file.exceptions import ( + StorageException, + StoragePermissionException, + StorageTransientException, +) from automation_file.local.safe_paths import safe_join from automation_file.storage.backend import ( StorageBackend, @@ -53,6 +57,9 @@ def _os_errors(location: str) -> Iterator[None]: raise missing_error(location) from error except PermissionError as error: raise StoragePermissionException(f"access to {location} was denied") from error + except (TimeoutError, ConnectionError, InterruptedError) as error: + # A network filesystem mounted as a local path fails this way. + raise StorageTransientException(f"{location}: {error}") from error except OSError as error: raise StorageException(f"{location}: {error}") from error diff --git a/docs/source/Eng/usage/storage.rst b/docs/source/Eng/usage/storage.rst index 85f393a..a0b3d98 100644 --- a/docs/source/Eng/usage/storage.rst +++ b/docs/source/Eng/usage/storage.rst @@ -415,7 +415,7 @@ implement ``_head``, ``_scan``, ``_put``, ``_get`` and ``_remove``. It supplies the directory behaviour described under `Built-in backends`_, and is what ``S3Storage`` and ``AzureStorage`` are built on. -Check it with the contract suite. ``tests/storage_contract.py`` holds 77 cases — +Check it with the contract suite. ``tests/storage_contract.py`` holds 81 cases — nested directories, empty and large files, Unicode paths, binary data, overwrite and missing-path behaviour, path normalisation, streams, copy and move — and reads ``capabilities`` where backends legitimately differ: @@ -429,3 +429,13 @@ and missing-path behaviour, path normalisation, streams, copy and move — and r @pytest.fixture def backend(self): return VaultStorage(...) # an empty storage for each test + + @pytest.fixture + def break_storage(self, backend): + def fail(kind, times=1): # kind: "denied" or "transient" + backend.client.fail_next(kind, times) + return fail + +The four failure cases (access denied, a transient failure, a retry that succeeds, a +denied call that is not retried) need the ``break_storage`` fixture, which makes the +next calls to the service fail. Without it those four skip and the rest still run. diff --git a/docs/source/Zh-CN/usage/storage.rst b/docs/source/Zh-CN/usage/storage.rst index d2775b3..8b6754f 100644 --- a/docs/source/Zh-CN/usage/storage.rst +++ b/docs/source/Zh-CN/usage/storage.rst @@ -390,7 +390,7 @@ scheme 或 authority，并且只接受其下的 URI。 ``_head``、``_scan``、``_put``、``_get`` 与 ``_remove``。它提供 `内置后端`_ 一节 所述的目录行为，``S3Storage`` 与 ``AzureStorage`` 都建立在它之上。 -请用契约测试套件检查。``tests/storage_contract.py`` 包含 77 个用例——嵌套目录、 +请用契约测试套件检查。``tests/storage_contract.py`` 包含 81 个用例——嵌套目录、 空文件与大文件、Unicode 路径、二进制数据、覆盖与路径不存在时的行为、路径规范化、 流、复制与移动——并在后端确实有差异之处读取 ``capabilities``： @@ -403,3 +403,13 @@ scheme 或 authority，并且只接受其下的 URI。 @pytest.fixture def backend(self): return VaultStorage(...) # 每个测试都要是空的存储 + + @pytest.fixture + def break_storage(self, backend): + def fail(kind, times=1): # kind: "denied" or "transient" + backend.client.fail_next(kind, times) + return fail + +四个失败用例（访问被拒、暂时性失败、重试后成功、被拒的调用不会重试）需要 +``break_storage`` fixture，它会让接下来对服务的调用失败。没有它时，这四个用例会跳过， +其余用例照常运行。 diff --git a/docs/source/Zh-TW/usage/storage.rst b/docs/source/Zh-TW/usage/storage.rst index bd7f5df..a2ba355 100644 --- a/docs/source/Zh-TW/usage/storage.rst +++ b/docs/source/Zh-TW/usage/storage.rst @@ -390,7 +390,7 @@ scheme 或 authority，並且只接受其下的 URI。 ``_head``、``_scan``、``_put``、``_get`` 與 ``_remove``。它提供 `內建後端`_ 一節 所述的目錄行為，``S3Storage`` 與 ``AzureStorage`` 都建立在它之上。 -請用契約測試套件檢查。``tests/storage_contract.py`` 包含 77 個案例——巢狀目錄、 +請用契約測試套件檢查。``tests/storage_contract.py`` 包含 81 個案例——巢狀目錄、 空檔與大檔、Unicode 路徑、二進位資料、覆寫與路徑不存在時的行為、路徑正規化、 串流、複製與搬移——並在後端確實有差異之處讀取 ``capabilities``： @@ -403,3 +403,13 @@ scheme 或 authority，並且只接受其下的 URI。 @pytest.fixture def backend(self): return VaultStorage(...) # 每個測試都要是空的儲存 + + @pytest.fixture + def break_storage(self, backend): + def fail(kind, times=1): # kind: "denied" or "transient" + backend.client.fail_next(kind, times) + return fail + +四個失敗案例（存取被拒、暫時性失敗、重試後成功、被拒的呼叫不會重試）需要 +``break_storage`` fixture，它會讓接下來對服務的呼叫失敗。沒有它時，這四個案例會略過， +其餘案例照常執行。 diff --git a/docs/updates/2026-10.md b/docs/updates/2026-10.md index ec38e8d..1a9497c 100644 --- a/docs/updates/2026-10.md +++ b/docs/updates/2026-10.md @@ -323,3 +323,14 @@ Index and query commands: [README.md](README.md). New entries go at the end. - **Docs**: the installation section with the extras table in the three READMEs, the three `usage/cloud.rst` pages and `API/remote.rst` (no longer "bundled"), `CLAUDE.md` (Branching & CI, Development, key types), `architecture.md` §2, §5, §6 and §7, `requirements.txt` and `dev_requirements.txt` (`[all]`). - **Files**: `stable.toml`, `dev.toml`, `requirements.txt`, `dev_requirements.txt`, `.github/workflows/ci-dev.yml`, `.github/workflows/ci-stable.yml`, `automation_file/core/optional.py`, `automation_file/exceptions.py`, `automation_file/remote/google_drive/*.py`, `automation_file/remote/s3/client.py`, `automation_file/remote/azure_blob/client.py`, `automation_file/remote/dropbox_api/client.py`, `automation_file/remote/dropbox_api/upload_ops.py`, `automation_file/remote/box/client.py`, `automation_file/remote/fsspec_bridge.py`, `automation_file/local/data_ops.py`, the tests named above, the documentation above, `progress.md`. - **Open items**: `progress.md` #29 (PyBreeze's branch), #30 (the three clients that still name no extra). + +## U-20261008-10 · 2026-10-08 · Failure cases join the storage contract · #storage #roadmap #tests + +- **What**: the failure half of `progress.md` #20 (roadmap section 5: "permission/access errors", "retryable failures"); the metadata half stays open there. + - Four cases in `tests/storage_contract.py`: a denied call raises `StoragePermissionException` with its cause chained and the next call works; a transient failure raises `StorageTransientException`; under `retry_on_transient(retriable=(StorageTransientException,))` two transient failures are retried away and five exhaust three attempts into `RetryExhaustedException`; a denied call is attempted once and not retried. + - They use a `break_storage` fixture that returns `fail(kind, times=1)`. The default skips; a contract class overrides it for its stand-in. The local class fails `Path.read_bytes`, the S3 and Azure classes set `fail_with` / `fail_times` on their stand-in clients, and the two prefixed classes now inherit from the unprefixed ones, so they run the cases too. `MemoryStorage` has nothing that can fail and skips them. + - `LocalStorage` maps `TimeoutError`, `ConnectionError` and `InterruptedError` to `StorageTransientException`: a network filesystem mounted as a local path fails that way, and until now those became a plain `StorageException` that a retry policy would not pick up. +- **Result / numbers**: the contract has 81 cases per backend. With every extra: 1606 passed, 26 skipped, 0 failed; base dependencies only: 1200 passed, 32 skipped, 0 failed. `ruff check`, `ruff format --check` and `mypy automation_file` pass. Python 3.14.7 on Windows. +- **Docs**: the three `usage/storage.rst` pages (the count and the `break_storage` fixture under "Writing a backend") and the three READMEs (the count). +- **Files**: `tests/storage_contract.py`, `tests/test_storage_local.py`, `tests/test_storage_s3.py`, `tests/test_storage_azure.py`, `automation_file/storage/local_storage.py`, the documentation above, `progress.md`. +- **Open items**: `progress.md` #20 (metadata cases); the adapters on other branches add the fixture when they merge. diff --git a/docs/updates/README.md b/docs/updates/README.md index 29bce48..5ff8482 100644 --- a/docs/updates/README.md +++ b/docs/updates/README.md @@ -58,6 +58,7 @@ In the same commit: delete the item from `progress.md`, add a `#done` entry here | ID | Date | Title | Tags | Batch | |---|---|---|---|---| +| U-20261008-10 | 2026-10-08 | Failure cases join the storage contract | #storage #roadmap #tests | [2026-10](2026-10.md) | | U-20261008-09 | 2026-10-08 | Backend SDKs and the GUI toolkit become extras | #done #packaging #roadmap #decision | [2026-10](2026-10.md) | | U-20261008-08 | 2026-10-08 | Three architecture diagrams still listed drive:// | #incident #docs | [2026-10](2026-10.md) | | U-20261008-07 | 2026-10-08 | The cloud pages describe FA_copy_between, not FA_cross_copy | #done #docs | [2026-10](2026-10.md) | @@ -104,5 +105,5 @@ In the same commit: delete the item from `progress.md`, add a `#done` entry here | File | Period | Entries | |---|---|---:| -| [2026-10.md](2026-10.md) | 2026-10 | 20 | +| [2026-10.md](2026-10.md) | 2026-10 | 21 | | [2026-09.md](2026-09.md) | 2026-09 | 21 | diff --git a/progress.md b/progress.md index 9b5a39d..98c9c8f 100644 --- a/progress.md +++ b/progress.md @@ -24,7 +24,7 @@ Items #10 to #26 are what is left of the 1.0 roadmap (`docs/FILEAUTOMATION-1.0-R ### Backend integration tests (roadmap M3) - **#19** Integration environments for the contract suite in CI: MinIO and Azurite (`S3Storage` and `AzureStorage` have only met stand-in clients and, for S3, botocore's Stubber; no request has reached a real service), SFTP, FTP/FTPS, WebDAV and Samba, plus credential-gated jobs for the cloud adapters, and Linux and macOS legs (`ci-dev.yml` runs pytest on Windows only). Needs #13. -- **#20** Failure cases in the contract suite: access denied, transient failures mapped to `StorageTransientException` and retried, metadata kept where the backend supports it. Only `LocalStorage` has permission-error tests today (`tests/test_storage_local.py`). +- **#20** Metadata cases in the contract suite: user metadata and content type kept across an upload and a copy where `capabilities` says the backend supports them. The failure cases exist (U-20261008-10): a backend's contract class gets them by providing the `break_storage` fixture, as the local, S3 and Azure classes do. ### Later milestones @@ -39,4 +39,3 @@ Items #10 to #26 are what is left of the 1.0 roadmap (`docs/FILEAUTOMATION-1.0-R - **#29** [BLOCKED] PyBreeze has to declare `automation-file[all]` before the stable release that splits the extras reaches users, or it installs without the SDKs it relied on. The change exists on PyBreeze's local branch `deps/automation-file-all-extra` (one commit on its `origin/dev`: `dev.toml`, `pyproject.toml`, `requirements.txt`), not pushed: it waits for someone to open the PR there and for PyBreeze's own update log. PyBreeze's checkout was on `docs/tutorials` with other work, so nothing else was touched. - **#30** `remote/sftp/client.py` ("reinstall `automation_file`"), `remote/onedrive/client.py` (the same) and `remote/smb/client.py` ("install `smbprotocol`") still report a missing SDK without naming the extra. Change them to `core.optional.require_module` or `install_hint` once the adapter branches that edit those files are merged. - diff --git a/tests/storage_contract.py b/tests/storage_contract.py index 45aeb4f..ad3c26a 100644 --- a/tests/storage_contract.py +++ b/tests/storage_contract.py @@ -13,23 +13,33 @@ def backend(self) -> StorageBackend: The suite reads ``backend.capabilities`` to pick the expected behaviour where backends legitimately differ (real directories versus implied ones, optional ``FileInfo`` fields). Everything else is the same for every backend. + +The failure cases need a way to make the storage fail. Override the +``break_storage`` fixture to return ``fail(kind, times=1)``, which makes the next +``times`` calls to the service fail as ``"denied"`` or ``"transient"``; without it +those cases skip. """ from __future__ import annotations import hashlib +from collections.abc import Callable from datetime import timedelta from pathlib import Path import pytest +from automation_file.core.retry import retry_on_transient from automation_file.exceptions import ( FileNotExistsException, + RetryExhaustedException, StorageAlreadyExistsException, StorageException, StorageNotEmptyException, StorageNotFoundException, StoragePathTypeException, + StoragePermissionException, + StorageTransientException, StorageUnsupportedException, StorageURIException, ) @@ -58,6 +68,10 @@ class StorageContract: def backend(self) -> StorageBackend: raise NotImplementedError("the contract subclass provides the backend fixture") + @pytest.fixture + def break_storage(self, backend: StorageBackend) -> Callable[..., None]: + pytest.skip("this backend's stand-in cannot be made to fail") + # ------------------------------------------------------------------ exists / stat def test_root_exists_and_is_a_directory(self, backend: StorageBackend) -> None: @@ -492,6 +506,70 @@ def test_move_from_respects_overwrite(self, backend: StorageBackend) -> None: assert backend.read_bytes("b.txt") == b"new" assert backend.exists("a.txt") is False + # ------------------------------------------------------------------ failures + + def test_a_denied_call_raises_the_permission_error( + self, backend: StorageBackend, break_storage: Callable[..., None] + ) -> None: + backend.write_bytes("a.txt", b"x") + break_storage("denied") + with pytest.raises(StoragePermissionException) as caught: + backend.read_bytes("a.txt") + assert caught.value.__cause__ is not None + assert backend.read_bytes("a.txt") == b"x" + + def test_a_transient_failure_raises_the_retryable_error( + self, backend: StorageBackend, break_storage: Callable[..., None] + ) -> None: + backend.write_bytes("a.txt", b"x") + break_storage("transient") + with pytest.raises(StorageTransientException) as caught: + backend.read_bytes("a.txt") + assert caught.value.__cause__ is not None + + def test_a_transient_failure_goes_away_on_retry( + self, backend: StorageBackend, break_storage: Callable[..., None] + ) -> None: + backend.write_bytes("a.txt", b"payload") + + @retry_on_transient( + max_attempts=3, + backoff_base=0.0, + backoff_cap=0.0, + retriable=(StorageTransientException,), + ) + def read() -> bytes: + return backend.read_bytes("a.txt") + + break_storage("transient", times=2) + assert read() == b"payload" + break_storage("transient", times=5) + with pytest.raises(RetryExhaustedException) as caught: + read() + assert isinstance(caught.value.__cause__, StorageTransientException) + + def test_a_denied_call_is_not_retried( + self, backend: StorageBackend, break_storage: Callable[..., None] + ) -> None: + backend.write_bytes("a.txt", b"x") + attempts = 0 + + @retry_on_transient( + max_attempts=3, + backoff_base=0.0, + backoff_cap=0.0, + retriable=(StorageTransientException,), + ) + def read() -> bytes: + nonlocal attempts + attempts += 1 + return backend.read_bytes("a.txt") + + break_storage("denied", times=3) + with pytest.raises(StoragePermissionException): + read() + assert attempts == 1 + # ------------------------------------------------------------------ lifecycle def test_backend_is_a_context_manager(self, backend: StorageBackend) -> None: diff --git a/tests/test_storage_azure.py b/tests/test_storage_azure.py index 92b4eae..b9350f1 100644 --- a/tests/test_storage_azure.py +++ b/tests/test_storage_azure.py @@ -11,6 +11,7 @@ import hashlib import inspect +from collections.abc import Callable from dataclasses import dataclass from datetime import datetime, timezone from pathlib import Path @@ -157,9 +158,12 @@ def __init__(self, *containers: str) -> None: name: {} for name in containers or ("container",) } self.fail_with: Exception | None = None + self.fail_times: int | None = None # None: every call while fail_with is set def blobs(self, container: str) -> dict[str, _Blob]: - if self.fail_with is not None: + if self.fail_with is not None and self.fail_times != 0: + if self.fail_times is not None: + self.fail_times -= 1 raise self.fail_with if container not in self.containers: raise ResourceNotFoundError("The specified container does not exist.") @@ -172,13 +176,30 @@ def get_container_client(self, container: str) -> _ContainerClient: return _ContainerClient(self, container) +_FAILURES = { + "denied": lambda: _http_error(403), + "transient": lambda: _http_error(503), +} + + class TestAzureStorageContract(StorageContract): @pytest.fixture def backend(self) -> StorageBackend: return AzureStorage("container", service=FakeBlobService()) + @pytest.fixture + def break_storage(self, backend: StorageBackend) -> Callable[..., None]: + assert isinstance(backend, AzureStorage) + service = backend._service + + def fail(kind: str, times: int = 1) -> None: + service.fail_with = _FAILURES[kind]() + service.fail_times = times + + return fail + -class TestPrefixedAzureStorageContract(StorageContract): +class TestPrefixedAzureStorageContract(TestAzureStorageContract): @pytest.fixture def backend(self) -> StorageBackend: service = FakeBlobService() diff --git a/tests/test_storage_local.py b/tests/test_storage_local.py index 7a8136b..2d79807 100644 --- a/tests/test_storage_local.py +++ b/tests/test_storage_local.py @@ -4,6 +4,7 @@ import os import shutil +from collections.abc import Callable from pathlib import Path import pytest @@ -31,6 +32,9 @@ def _rootless(path: Path) -> str: return path.resolve().as_posix().lstrip("/") +_FAILURES: dict[str, type[OSError]] = {"denied": PermissionError, "transient": TimeoutError} + + class TestRootedLocalStorageContract(StorageContract): @pytest.fixture def backend(self, tmp_path: Path) -> StorageBackend: @@ -38,6 +42,26 @@ def backend(self, tmp_path: Path) -> StorageBackend: root.mkdir() return LocalStorage(root) + @pytest.fixture + def break_storage( + self, backend: StorageBackend, monkeypatch: pytest.MonkeyPatch + ) -> Callable[..., None]: + real_read = Path.read_bytes + remaining = {"kind": "", "times": 0} + + def read_bytes(path: Path) -> bytes: + if remaining["times"]: + remaining["times"] = int(remaining["times"]) - 1 + raise _FAILURES[str(remaining["kind"])]("injected") + return real_read(path) + + monkeypatch.setattr(Path, "read_bytes", read_bytes) + + def fail(kind: str, times: int = 1) -> None: + remaining.update(kind=kind, times=times) + + return fail + @pytest.fixture def root(tmp_path: Path) -> Path: diff --git a/tests/test_storage_s3.py b/tests/test_storage_s3.py index 41fd1d4..3d630af 100644 --- a/tests/test_storage_s3.py +++ b/tests/test_storage_s3.py @@ -10,6 +10,7 @@ import hashlib import inspect +from collections.abc import Callable from dataclasses import dataclass from datetime import datetime, timezone from pathlib import Path @@ -106,10 +107,13 @@ def __init__(self, *buckets: str) -> None: self.buckets: dict[str, dict[str, _Object]] = {name: {} for name in buckets or ("bucket",)} self.calls: list[str] = [] self.fail_with: Exception | None = None + self.fail_times: int | None = None # None: every call while fail_with is set def objects(self, bucket: str, operation: str) -> dict[str, _Object]: self.calls.append(operation) - if self.fail_with is not None: + if self.fail_with is not None and self.fail_times != 0: + if self.fail_times is not None: + self.fail_times -= 1 raise self.fail_with if bucket not in self.buckets: raise _client_error("NoSuchBucket", operation) @@ -159,13 +163,32 @@ def copy(self, copy_source: dict[str, str], bucket: str, key: str) -> None: ) +_FAILURES = { + "denied": lambda: _client_error("AccessDenied", "HeadObject"), + "transient": lambda: _client_error("SlowDown", "HeadObject"), +} + + +def _breaker(client: FakeS3Client) -> Callable[..., None]: + def fail(kind: str, times: int = 1) -> None: + client.fail_with = _FAILURES[kind]() + client.fail_times = times + + return fail + + class TestS3StorageContract(StorageContract): @pytest.fixture def backend(self) -> StorageBackend: return S3Storage("bucket", client=FakeS3Client()) + @pytest.fixture + def break_storage(self, backend: StorageBackend) -> Callable[..., None]: + assert isinstance(backend, S3Storage) + return _breaker(backend._client) + -class TestPrefixedS3StorageContract(StorageContract): +class TestPrefixedS3StorageContract(TestS3StorageContract): @pytest.fixture def backend(self) -> StorageBackend: client = FakeS3Client() From 22f9cde51da5fc2300f87971f56163ddedd76290 Mon Sep 17 00:00:00 2001 From: JeffreyChen Date: Thu, 8 Oct 2026 13:00:16 +0800 Subject: [PATCH 31/59] feat: add a storage subcommand to the CLI --- CLAUDE.md | 1 + README.md | 6 + README.zh-CN.md | 6 + README.zh-TW.md | 6 + architecture.md | 7 +- automation_file/__main__.py | 6 +- automation_file/cli_storage.py | 192 ++++++++++++++++++++++++++++++++ docs/source/Eng/usage/cli.rst | 32 ++++++ docs/source/Zh-CN/usage/cli.rst | 30 +++++ docs/source/Zh-TW/usage/cli.rst | 30 +++++ docs/updates/2026-10.md | 12 ++ docs/updates/README.md | 3 +- tests/test_cli_storage.py | 132 ++++++++++++++++++++++ 13 files changed, 458 insertions(+), 5 deletions(-) create mode 100644 automation_file/cli_storage.py create mode 100644 tests/test_cli_storage.py diff --git a/CLAUDE.md b/CLAUDE.md index dd422aa..a9e7c77 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -10,6 +10,7 @@ Automation-first Python library for local file / directory / zip operations, HTT automation_file/ ├── __init__.py # Public API facade (__all__); launch_ui is loaded lazily via __getattr__ ├── __main__.py # CLI entry: subcommands plus the legacy -e/-d/-c/--execute_str flags +├── cli_storage.py # the `storage` subcommand (ls, cp, mv, rm, sync, checksum, ...) ├── exceptions.py # FileAutomationException hierarchy ├── logging_config.py # file_automation_logger (file + stderr handlers) ├── core/ # Engine: action_registry (ActionRegistry, build_default_registry), action_executor diff --git a/README.md b/README.md index 2fa783b..d758261 100644 --- a/README.md +++ b/README.md @@ -1052,6 +1052,12 @@ python -m automation_file drive-upload my.txt --token token.json --credentials c python -m automation_file mcp --allowed-actions FA_file_checksum,FA_fast_find automation_file_mcp --allowed-actions FA_file_checksum,FA_fast_find # installed console script +# Storage layer: ls, stat, cat, cp, mv, rm, mkdir, sync, checksum, verify, schemes (JSON output) +python -m automation_file storage ls s3://reports/2026 --recursive +python -m automation_file storage cp report.csv s3://reports/2026/report.csv +python -m automation_file storage sync ./site s3://www --delete --dry-run +python -m automation_file storage checksum s3://reports/2026/q1.csv + # Legacy flags (JSON action lists) python -m automation_file --execute_file actions.json python -m automation_file --execute_dir ./actions/ diff --git a/README.zh-CN.md b/README.zh-CN.md index 84ed40d..d2bb93a 100644 --- a/README.zh-CN.md +++ b/README.zh-CN.md @@ -1027,6 +1027,12 @@ python -m automation_file drive-upload my.txt --token token.json --credentials c python -m automation_file mcp --allowed-actions FA_file_checksum,FA_fast_find automation_file_mcp --allowed-actions FA_file_checksum,FA_fast_find # 已安装的 console script +# 存储层：ls、stat、cat、cp、mv、rm、mkdir、sync、checksum、verify、schemes（输出 JSON） +python -m automation_file storage ls s3://reports/2026 --recursive +python -m automation_file storage cp report.csv s3://reports/2026/report.csv +python -m automation_file storage sync ./site s3://www --delete --dry-run +python -m automation_file storage checksum s3://reports/2026/q1.csv + # 旧式标志（JSON 动作清单） python -m automation_file --execute_file actions.json python -m automation_file --execute_dir ./actions/ diff --git a/README.zh-TW.md b/README.zh-TW.md index a81d23d..9530127 100644 --- a/README.zh-TW.md +++ b/README.zh-TW.md @@ -1027,6 +1027,12 @@ python -m automation_file drive-upload my.txt --token token.json --credentials c python -m automation_file mcp --allowed-actions FA_file_checksum,FA_fast_find automation_file_mcp --allowed-actions FA_file_checksum,FA_fast_find # 已安裝的 console script +# 儲存層：ls、stat、cat、cp、mv、rm、mkdir、sync、checksum、verify、schemes（輸出 JSON） +python -m automation_file storage ls s3://reports/2026 --recursive +python -m automation_file storage cp report.csv s3://reports/2026/report.csv +python -m automation_file storage sync ./site s3://www --delete --dry-run +python -m automation_file storage checksum s3://reports/2026/q1.csv + # 舊式旗標（JSON 動作清單） python -m automation_file --execute_file actions.json python -m automation_file --execute_dir ./actions/ diff --git a/architecture.md b/architecture.md index 866076c..1d182bc 100644 --- a/architecture.md +++ b/architecture.md @@ -20,7 +20,7 @@ piece of the 1.0 roadmap (`docs/FILEAUTOMATION-1.0-ROADMAP.md`, PR #107); what i | Path | Responsibility | | --- | --- | | `automation_file/__init__.py` | Public facade (`__all__`). Wires the shared `executor`, `callback_executor` and `package_manager` over one registry. `launch_ui` is loaded lazily through `__getattr__` | -| `automation_file/__main__.py` | CLI: legacy flags plus subcommands | +| `automation_file/__main__.py`, `cli_storage.py` | CLI: legacy flags plus subcommands; `cli_storage.py` holds the `storage` subcommand | | `automation_file/core/` | Engine, on je_action_core: `action_registry.py` (`ActionRegistry`, a `CommandRegistry`; `build_default_registry`), `action_executor.py` (`ActionExecutor`, an `ActionExecutor` with strict actions, indexed records and the dry-run, validate, substitute and parallel extras; shared `executor`), `callback_executor.py`, `package_loader.py`, `plugins.py`, `dag_executor.py`, `action_queue.py`, `json_store.py`, `substitution.py`. Also cross-cutting helpers: `optional` (`require_module`, the extras table), `retry`, `quota`, `rate_limit`, `circuit_breaker`, `file_lock`, `sqlite_lock`, `checksum`, `manifest`, `crypto`, `secrets`, `config`, `config_watcher`, `audit`, `metrics`, `tracing`, `progress`, `fim`, `content_store` | | `automation_file/local/` | Local strategy modules: file, dir, zip, tar and archive ops, sync, diff, text/JSON/data edits, templates, versioning, trash, `shell_ops` (argv-only subprocess), conditional branches. `safe_paths.py` guards against path traversal | | `automation_file/remote/` | `url_validator.py` (SSRF guard), `http_download.py`, `cross_backend.py`, `fsspec_bridge.py`. One subpackage per backend: `google_drive/`, `s3/`, `azure_blob/`, `dropbox_api/`, `sftp/`, `ftp/`, `onedrive/`, `box/`, each with `client.py`, `*_ops.py` and `register__ops`. `smb/` and `webdav/` have a client only | @@ -66,7 +66,10 @@ piece of the 1.0 roadmap (`docs/FILEAUTOMATION-1.0-ROADMAP.md`, PR #107); what i - **CLI** (`python -m automation_file`; no console script for it): - legacy flags `-e/--execute_file`, `-d/--execute_dir`, `-c/--create_project` and `--execute_str`. `_execute_str` decodes a second time when the first `json.loads` yields a string; - - subcommands `zip`, `unzip`, `download`, `create-file`, `server`, `http-server`, `ui`, `mcp`, `drive-upload`. + - subcommands `zip`, `unzip`, `download`, `create-file`, `server`, `http-server`, `ui`, `mcp`, `drive-upload`; + - `storage` with its own subcommands `ls`, `stat`, `cat`, `cp`, `mv`, `rm`, `mkdir`, `sync`, `checksum`, + `verify`, `schemes` (`cli_storage.py`): one JSON document per call, and `--init ` to + initialise a backend's client first. - **MCP**: `automation_file_mcp` (`automation_file.server.mcp_server:_cli`) or `python -m automation_file mcp [--allowed-actions ...]`. It is a standard-library JSON-RPC stdio server whose tools come from the registry (`tools_from_registry`). diff --git a/automation_file/__main__.py b/automation_file/__main__.py index 5e48c92..9025fb7 100644 --- a/automation_file/__main__.py +++ b/automation_file/__main__.py @@ -5,8 +5,8 @@ * Legacy flags (``-e``, ``-d``, ``-c``, ``--execute_str``) — run JSON action lists without writing Python. * Subcommands (``zip``, ``unzip``, ``download``, ``server``, ``http-server``, - ``drive-upload``, ``ui``) — wrap the most common facade calls so users do - not need to hand-author JSON for one-shot operations. + ``drive-upload``, ``ui``, ``mcp``, ``storage``) — wrap the most common facade + calls so users do not need to hand-author JSON for one-shot operations. * No arguments — prints help and exits non-zero. """ @@ -19,6 +19,7 @@ from collections.abc import Callable from typing import Any +from automation_file.cli_storage import add_storage_commands from automation_file.core.action_executor import execute_action, execute_files from automation_file.core.json_store import read_action_json from automation_file.exceptions import ArgparseException @@ -224,6 +225,7 @@ def _build_parser() -> argparse.ArgumentParser: _add_file_commands(subparsers) _add_server_commands(subparsers) _add_integration_commands(subparsers) + add_storage_commands(subparsers) return parser diff --git a/automation_file/cli_storage.py b/automation_file/cli_storage.py new file mode 100644 index 0000000..60e1dc8 --- /dev/null +++ b/automation_file/cli_storage.py @@ -0,0 +1,192 @@ +"""``python -m automation_file storage ...``: the storage layer from a shell. + +Every subcommand takes storage URIs (or plain local paths) and prints one JSON +document, so the output can be piped into ``jq`` or read by another program: + +.. code-block:: text + + python -m automation_file storage ls s3://reports/2026 --recursive + python -m automation_file storage cp report.csv s3://reports/2026/report.csv + python -m automation_file storage sync ./site s3://www --delete --dry-run + python -m automation_file storage checksum s3://reports/2026/report.csv + +A remote backend needs its client initialised first. ``--init`` takes a JSON +action list that runs before the command: + +.. code-block:: text + + python -m automation_file storage \\ + --init '[["FA_s3_later_init", {"region_name": "us-east-1"}]]' \\ + ls s3://reports +""" + +from __future__ import annotations + +import argparse +import json +import sys +from typing import Any + +from automation_file.storage import actions + +_SOURCE = "source" +_TARGET = "target" +_URI = "uri" + + +def _emit(document: Any) -> int: + sys.stdout.write(json.dumps(document, ensure_ascii=False, indent=2, default=str) + "\n") + return 0 + + +def _run_init(raw: str | None) -> None: + if not raw: + return + from automation_file.core.action_executor import execute_action + + execute_action(json.loads(raw)) + + +def _cmd_ls(args: argparse.Namespace) -> int: + return _emit(actions.storage_list(args.uri, recursive=args.recursive)) + + +def _cmd_stat(args: argparse.Namespace) -> int: + return _emit(actions.storage_stat(args.uri)) + + +def _cmd_cat(args: argparse.Namespace) -> int: + sys.stdout.write(actions.storage_read_text(args.uri, encoding=args.encoding)) + return 0 + + +def _cmd_cp(args: argparse.Namespace) -> int: + if args.recursive: + summary = actions.storage_copy_tree( + args.source, args.target, overwrite=not args.no_overwrite + ) + _emit(summary) + return 1 if summary["errors"] else 0 + return _emit(actions.storage_copy(args.source, args.target, overwrite=not args.no_overwrite)) + + +def _cmd_mv(args: argparse.Namespace) -> int: + return _emit(actions.storage_move(args.source, args.target, overwrite=not args.no_overwrite)) + + +def _cmd_rm(args: argparse.Namespace) -> int: + actions.storage_delete(args.uri, recursive=args.recursive, missing_ok=args.missing_ok) + return _emit({"deleted": args.uri}) + + +def _cmd_mkdir(args: argparse.Namespace) -> int: + actions.storage_mkdir(args.uri) + return _emit({"created": args.uri}) + + +def _cmd_checksum(args: argparse.Namespace) -> int: + return _emit(actions.storage_checksum(args.uri, algorithm=args.algorithm)) + + +def _cmd_verify(args: argparse.Namespace) -> int: + matched = actions.storage_verify(args.uri, args.expected, algorithm=args.algorithm) + _emit({"uri": args.uri, "matches": matched}) + return 0 if matched else 1 + + +def _cmd_sync(args: argparse.Namespace) -> int: + summary = actions.storage_sync( + args.source, + args.target, + delete=args.delete, + checksum=args.checksum, + dry_run=args.dry_run, + ) + _emit(summary) + return 1 if summary["errors"] else 0 + + +def _cmd_schemes(_args: argparse.Namespace) -> int: + return _emit(actions.storage_schemes()) + + +def _dispatch(args: argparse.Namespace) -> int: + _run_init(args.init) + return int(args.storage_handler(args)) + + +def _add_read_commands(commands: argparse._SubParsersAction) -> None: + ls_parser = commands.add_parser("ls", help="list a directory") + ls_parser.add_argument(_URI) + ls_parser.add_argument("-r", "--recursive", action="store_true") + ls_parser.set_defaults(storage_handler=_cmd_ls) + + stat_parser = commands.add_parser("stat", help="show one entry's metadata") + stat_parser.add_argument(_URI) + stat_parser.set_defaults(storage_handler=_cmd_stat) + + cat_parser = commands.add_parser("cat", help="print a file's text") + cat_parser.add_argument(_URI) + cat_parser.add_argument("--encoding", default="utf-8") + cat_parser.set_defaults(storage_handler=_cmd_cat) + + checksum_parser = commands.add_parser("checksum", help="print a file's digest") + checksum_parser.add_argument(_URI) + checksum_parser.add_argument("--algorithm", default="sha256") + checksum_parser.set_defaults(storage_handler=_cmd_checksum) + + verify_parser = commands.add_parser("verify", help="compare a file with an expected digest") + verify_parser.add_argument(_URI) + verify_parser.add_argument("expected", help="a digest, or 'algorithm:digest'") + verify_parser.add_argument("--algorithm", default="sha256") + verify_parser.set_defaults(storage_handler=_cmd_verify) + + schemes_parser = commands.add_parser("schemes", help="list the registered URI schemes") + schemes_parser.set_defaults(storage_handler=_cmd_schemes) + + +def _add_write_commands(commands: argparse._SubParsersAction) -> None: + cp_parser = commands.add_parser("cp", help="copy a file, or a directory tree with -r") + cp_parser.add_argument(_SOURCE) + cp_parser.add_argument(_TARGET) + cp_parser.add_argument("-r", "--recursive", action="store_true") + cp_parser.add_argument("--no-overwrite", action="store_true") + cp_parser.set_defaults(storage_handler=_cmd_cp) + + mv_parser = commands.add_parser("mv", help="move a file") + mv_parser.add_argument(_SOURCE) + mv_parser.add_argument(_TARGET) + mv_parser.add_argument("--no-overwrite", action="store_true") + mv_parser.set_defaults(storage_handler=_cmd_mv) + + rm_parser = commands.add_parser("rm", help="delete a file or a directory") + rm_parser.add_argument(_URI) + rm_parser.add_argument("-r", "--recursive", action="store_true") + rm_parser.add_argument("--missing-ok", action="store_true") + rm_parser.set_defaults(storage_handler=_cmd_rm) + + mkdir_parser = commands.add_parser("mkdir", help="create a directory") + mkdir_parser.add_argument(_URI) + mkdir_parser.set_defaults(storage_handler=_cmd_mkdir) + + sync_parser = commands.add_parser("sync", help="mirror a directory tree into another") + sync_parser.add_argument(_SOURCE) + sync_parser.add_argument(_TARGET) + sync_parser.add_argument("--delete", action="store_true", help="remove what the source lacks") + sync_parser.add_argument("--checksum", action="store_true", help="compare digests, not times") + sync_parser.add_argument("--dry-run", action="store_true", help="report without changing") + sync_parser.set_defaults(storage_handler=_cmd_sync) + + +def add_storage_commands(subparsers: argparse._SubParsersAction) -> None: + """Register the ``storage`` subcommand and its own subcommands.""" + parser = subparsers.add_parser("storage", help="files and directories in any storage backend") + parser.add_argument( + "--init", + default=None, + help="JSON action list to run first, e.g. to initialise a backend's client", + ) + commands = parser.add_subparsers(dest="storage_command", required=True) + _add_read_commands(commands) + _add_write_commands(commands) + parser.set_defaults(handler=_dispatch) diff --git a/docs/source/Eng/usage/cli.rst b/docs/source/Eng/usage/cli.rst index 1b5c543..e64ab6b 100644 --- a/docs/source/Eng/usage/cli.rst +++ b/docs/source/Eng/usage/cli.rst @@ -23,3 +23,35 @@ Subcommands for one-shot operations:: The ``mcp`` subcommand starts a Model Context Protocol server over stdio so hosts such as Claude Desktop can call ``FA_*`` actions as MCP tools — see :doc:`mcp` for the full integration guide. + +Storage +------- + +The ``storage`` subcommand reaches the storage layer (:doc:`storage`) from a +shell. Every command takes storage URIs or plain local paths:: + + python -m automation_file storage ls s3://reports/2026 --recursive + python -m automation_file storage stat s3://reports/2026/q1.csv + python -m automation_file storage cat local:///etc/hostname + python -m automation_file storage cp report.csv s3://reports/2026/report.csv + python -m automation_file storage cp -r ./site s3://www/site --no-overwrite + python -m automation_file storage mv s3://inbox/a.csv s3://archive/a.csv + python -m automation_file storage rm s3://tmp/old --recursive --missing-ok + python -m automation_file storage mkdir local:///data/new + python -m automation_file storage sync ./site s3://www --delete --dry-run + python -m automation_file storage checksum s3://reports/2026/q1.csv --algorithm sha512 + python -m automation_file storage verify s3://reports/2026/q1.csv sha256:9f86d081... + python -m automation_file storage schemes + +Each command prints one JSON document (``cat`` prints the file's text), so the +output can be piped into ``jq`` or read by another program. The exit code is 0 on +success. ``verify`` exits 1 when the digest does not match; ``cp -r`` and ``sync`` +exit 1 when a file failed, with the failures under ``errors``. Any other failure +prints the exception and exits 1. + +A remote backend needs its client initialised first. ``--init`` takes a JSON +action list that runs before the command:: + + python -m automation_file storage \ + --init '[["FA_s3_later_init", {"region_name": "us-east-1"}]]' \ + ls s3://reports diff --git a/docs/source/Zh-CN/usage/cli.rst b/docs/source/Zh-CN/usage/cli.rst index 1de5e1b..4ddb8ab 100644 --- a/docs/source/Zh-CN/usage/cli.rst +++ b/docs/source/Zh-CN/usage/cli.rst @@ -23,3 +23,33 @@ CLI ``mcp`` 子命令通过 stdio 启动 Model Context Protocol 服务器， 让 Claude Desktop 这类宿主可以把 ``FA_*`` 动作当作 MCP 工具调用——完整集成 说明请见 :doc:`mcp`。 + +存储 +---- + +``storage`` 子命令让你从 shell 使用存储层（:doc:`storage`）。每个命令都接受存储 URI +或普通的本地路径:: + + python -m automation_file storage ls s3://reports/2026 --recursive + python -m automation_file storage stat s3://reports/2026/q1.csv + python -m automation_file storage cat local:///etc/hostname + python -m automation_file storage cp report.csv s3://reports/2026/report.csv + python -m automation_file storage cp -r ./site s3://www/site --no-overwrite + python -m automation_file storage mv s3://inbox/a.csv s3://archive/a.csv + python -m automation_file storage rm s3://tmp/old --recursive --missing-ok + python -m automation_file storage mkdir local:///data/new + python -m automation_file storage sync ./site s3://www --delete --dry-run + python -m automation_file storage checksum s3://reports/2026/q1.csv --algorithm sha512 + python -m automation_file storage verify s3://reports/2026/q1.csv sha256:9f86d081... + python -m automation_file storage schemes + +每个命令都会输出一份 JSON 文档（``cat`` 输出文件的文本内容），因此可以接到 ``jq`` +或交给其他程序读取。成功时退出码为 0。``verify`` 在摘要不符时以 1 退出；``cp -r`` 与 +``sync`` 在有文件失败时以 1 退出，失败项列在 ``errors`` 之下。其他失败会打印异常并 +以 1 退出。 + +远端后端必须先初始化客户端。``--init`` 接受一份 JSON 动作列表，会在命令之前执行:: + + python -m automation_file storage \ + --init '[["FA_s3_later_init", {"region_name": "us-east-1"}]]' \ + ls s3://reports diff --git a/docs/source/Zh-TW/usage/cli.rst b/docs/source/Zh-TW/usage/cli.rst index fec45d0..738d21b 100644 --- a/docs/source/Zh-TW/usage/cli.rst +++ b/docs/source/Zh-TW/usage/cli.rst @@ -23,3 +23,33 @@ CLI ``mcp`` 子指令以 stdio 啟動 Model Context Protocol 伺服器， 讓 Claude Desktop 之類的宿主可把 ``FA_*`` 動作當成 MCP 工具呼叫—— 完整整合說明請見 :doc:`mcp`。 + +儲存 +---- + +``storage`` 子指令讓你從 shell 使用儲存層（:doc:`storage`）。每個指令都接受儲存 URI +或一般的本機路徑:: + + python -m automation_file storage ls s3://reports/2026 --recursive + python -m automation_file storage stat s3://reports/2026/q1.csv + python -m automation_file storage cat local:///etc/hostname + python -m automation_file storage cp report.csv s3://reports/2026/report.csv + python -m automation_file storage cp -r ./site s3://www/site --no-overwrite + python -m automation_file storage mv s3://inbox/a.csv s3://archive/a.csv + python -m automation_file storage rm s3://tmp/old --recursive --missing-ok + python -m automation_file storage mkdir local:///data/new + python -m automation_file storage sync ./site s3://www --delete --dry-run + python -m automation_file storage checksum s3://reports/2026/q1.csv --algorithm sha512 + python -m automation_file storage verify s3://reports/2026/q1.csv sha256:9f86d081... + python -m automation_file storage schemes + +每個指令都會輸出一份 JSON 文件（``cat`` 輸出檔案的文字內容），因此可以接到 ``jq`` +或交給其他程式讀取。成功時結束碼為 0。``verify`` 在摘要不符時以 1 結束；``cp -r`` 與 +``sync`` 在有檔案失敗時以 1 結束，失敗項目列在 ``errors`` 之下。其他失敗會印出例外並 +以 1 結束。 + +遠端後端必須先初始化用戶端。``--init`` 接受一份 JSON 動作清單，會在指令之前執行:: + + python -m automation_file storage \ + --init '[["FA_s3_later_init", {"region_name": "us-east-1"}]]' \ + ls s3://reports diff --git a/docs/updates/2026-10.md b/docs/updates/2026-10.md index 1a9497c..9657606 100644 --- a/docs/updates/2026-10.md +++ b/docs/updates/2026-10.md @@ -334,3 +334,15 @@ Index and query commands: [README.md](README.md). New entries go at the end. - **Docs**: the three `usage/storage.rst` pages (the count and the `break_storage` fixture under "Writing a backend") and the three READMEs (the count). - **Files**: `tests/storage_contract.py`, `tests/test_storage_local.py`, `tests/test_storage_s3.py`, `tests/test_storage_azure.py`, `automation_file/storage/local_storage.py`, the documentation above, `progress.md`. - **Open items**: `progress.md` #20 (metadata cases); the adapters on other branches add the fixture when they merge. + +## U-20261008-11 · 2026-10-08 · A storage subcommand for the CLI · #storage #cli #roadmap + +- **What**: `python -m automation_file storage ` (`automation_file/cli_storage.py`, registered from `__main__.py`): `ls` (`--recursive`), `stat`, `cat`, `cp` (`-r` for a tree, `--no-overwrite`), `mv`, `rm` (`--recursive`, `--missing-ok`), `mkdir`, `sync` (`--delete`, `--checksum`, `--dry-run`), `checksum` (`--algorithm`), `verify`, `schemes`. Each is a thin call into the `FA_storage_*` functions, takes storage URIs or plain local paths, and prints one JSON document (`cat` prints the file's text). + - Exit codes: 0 on success; `verify` exits 1 on a mismatch; `cp -r` and `sync` exit 1 when a file failed (the failures are under `errors`). + - `storage --init '' ...` runs an action list first, which is how a one-shot command initialises a backend's client (`FA_s3_later_init`, ...). + - The legacy flags and the other subcommands are untouched (`tests/test_legacy_cli_contract.py` passes). +- **Tests**: `tests/test_cli_storage.py`, 7 cases through `main([...])`: listings and text with non-ASCII content, copy / move / delete / mkdir, a tree copy with and without overwrite, the exit codes of `verify` and of a `sync` with a failing file, `--init`, and the missing subcommand. +- **Result / numbers**: 1613 passed, 26 skipped, 0 failed with every extra; `ruff check`, `ruff format --check` and `mypy automation_file` (184 files) pass. Python 3.14.7 on Windows. +- **Docs**: a "Storage" section in the three `usage/cli.rst` pages, the CLI block of the three READMEs, `architecture.md` §2 and §3, `CLAUDE.md` (package map). +- **Files**: `automation_file/cli_storage.py`, `automation_file/__main__.py`, `tests/test_cli_storage.py`, the documentation above. +- **Open items**: none for the storage commands; the pipeline, integrity and audit subcommands follow with their packages. diff --git a/docs/updates/README.md b/docs/updates/README.md index 5ff8482..9d7948d 100644 --- a/docs/updates/README.md +++ b/docs/updates/README.md @@ -58,6 +58,7 @@ In the same commit: delete the item from `progress.md`, add a `#done` entry here | ID | Date | Title | Tags | Batch | |---|---|---|---|---| +| U-20261008-11 | 2026-10-08 | A storage subcommand for the CLI | #storage #cli #roadmap | [2026-10](2026-10.md) | | U-20261008-10 | 2026-10-08 | Failure cases join the storage contract | #storage #roadmap #tests | [2026-10](2026-10.md) | | U-20261008-09 | 2026-10-08 | Backend SDKs and the GUI toolkit become extras | #done #packaging #roadmap #decision | [2026-10](2026-10.md) | | U-20261008-08 | 2026-10-08 | Three architecture diagrams still listed drive:// | #incident #docs | [2026-10](2026-10.md) | @@ -105,5 +106,5 @@ In the same commit: delete the item from `progress.md`, add a `#done` entry here | File | Period | Entries | |---|---|---:| -| [2026-10.md](2026-10.md) | 2026-10 | 21 | +| [2026-10.md](2026-10.md) | 2026-10 | 22 | | [2026-09.md](2026-09.md) | 2026-09 | 21 | diff --git a/tests/test_cli_storage.py b/tests/test_cli_storage.py new file mode 100644 index 0000000..b552bb7 --- /dev/null +++ b/tests/test_cli_storage.py @@ -0,0 +1,132 @@ +"""The ``storage`` subcommand: one JSON document per call, exit codes that mean something.""" + +from __future__ import annotations + +import json +from collections.abc import Iterator +from pathlib import Path + +import pytest + +from automation_file.__main__ import main +from automation_file.exceptions import StorageNotFoundException +from automation_file.storage import clear_memory_stores, memory_store + + +@pytest.fixture(autouse=True) +def _fresh_stores() -> Iterator[None]: + clear_memory_stores() + yield + clear_memory_stores() + + +def _run(capsys: pytest.CaptureFixture[str], *argv: str) -> tuple[int, str]: + code = main(["storage", *argv]) + return code, capsys.readouterr().out + + +def test_ls_stat_and_cat(capsys: pytest.CaptureFixture[str]) -> None: + store = memory_store("cli") + store.write_bytes("docs/a.txt", "報告 ✓".encode()) + store.write_bytes("docs/sub/b.txt", b"x") + code, out = _run(capsys, "ls", "memory://cli/docs") + assert code == 0 + assert [(entry["path"], entry["is_dir"]) for entry in json.loads(out)] == [ + ("a.txt", False), + ("sub", True), + ] + code, out = _run(capsys, "ls", "memory://cli/docs", "--recursive") + assert [entry["path"] for entry in json.loads(out)] == ["a.txt", "sub", "sub/b.txt"] + code, out = _run(capsys, "stat", "memory://cli/docs/a.txt") + assert json.loads(out)["uri"] == "memory://cli/docs/a.txt" + code, out = _run(capsys, "cat", "memory://cli/docs/a.txt") + assert (code, out) == (0, "報告 ✓") + + +def test_cp_mv_rm_and_mkdir(capsys: pytest.CaptureFixture[str], tmp_path: Path) -> None: + source = tmp_path / "a.txt" + source.write_bytes(b"payload") + code, out = _run(capsys, "cp", str(source), "memory://cli/in/a.txt") + assert code == 0 + assert json.loads(out)["size"] == 7 + code, out = _run(capsys, "mv", "memory://cli/in/a.txt", "memory://cli/out/b.txt") + assert json.loads(out)["uri"] == "memory://cli/out/b.txt" + assert memory_store("cli").exists("in/a.txt") is False + code, out = _run(capsys, "mkdir", "memory://cli/empty") + assert json.loads(out) == {"created": "memory://cli/empty"} + code, out = _run(capsys, "rm", "memory://cli/out", "-r") + assert json.loads(out) == {"deleted": "memory://cli/out"} + assert memory_store("cli").exists("out") is False + code, _ = _run(capsys, "rm", "memory://cli/out", "--missing-ok") + assert code == 0 + with pytest.raises(StorageNotFoundException): + main(["storage", "rm", "memory://cli/out"]) + + +def test_copy_without_overwrite_and_recursive_copy( + capsys: pytest.CaptureFixture[str], tmp_path: Path +) -> None: + store = memory_store("cli") + store.write_bytes("src/a.txt", b"a") + store.write_bytes("src/sub/b.txt", b"b") + code, out = _run(capsys, "cp", "-r", "memory://cli/src", str(tmp_path / "out")) + assert code == 0 + assert json.loads(out)["copied"] == ["a.txt", "sub/b.txt"] + assert (tmp_path / "out" / "sub" / "b.txt").read_bytes() == b"b" + code, out = _run( + capsys, "cp", "-r", "--no-overwrite", "memory://cli/src", str(tmp_path / "out") + ) + assert json.loads(out)["skipped"] == ["a.txt", "sub/b.txt"] + + +def test_checksum_and_verify_exit_codes(capsys: pytest.CaptureFixture[str]) -> None: + memory_store("cli").write_bytes("a.txt", b"hello") + code, out = _run(capsys, "checksum", "memory://cli/a.txt") + digest = json.loads(out) + assert digest["algorithm"] == "sha256" + code, out = _run(capsys, "verify", "memory://cli/a.txt", digest["value"]) + assert (code, json.loads(out)["matches"]) == (0, True) + code, out = _run(capsys, "verify", "memory://cli/a.txt", "0" * 64) + assert (code, json.loads(out)["matches"]) == (1, False) + code, out = _run(capsys, "checksum", "memory://cli/a.txt", "--algorithm", "md5") + assert json.loads(out)["algorithm"] == "md5" + + +def test_sync_reports_and_signals_errors(capsys: pytest.CaptureFixture[str]) -> None: + store = memory_store("cli") + store.write_bytes("src/a.txt", b"a") + store.write_bytes("dst/stale.txt", b"s") + code, out = _run( + capsys, "sync", "memory://cli/src", "memory://cli/dst", "--delete", "--dry-run" + ) + preview = json.loads(out) + assert (code, preview["copied"], preview["deleted"], preview["dry_run"]) == ( + 0, + ["a.txt"], + ["stale.txt"], + True, + ) + assert store.exists("dst/stale.txt") is True + code, out = _run(capsys, "sync", "memory://cli/src", "memory://cli/dst", "--delete") + assert code == 0 + assert store.exists("dst/stale.txt") is False + store.write_bytes("blocked/a.txt/inner", b"x") + code, out = _run(capsys, "sync", "memory://cli/src", "memory://cli/blocked") + assert code == 1 + assert list(json.loads(out)["errors"]) == ["a.txt"] + + +def test_schemes_and_init(capsys: pytest.CaptureFixture[str], tmp_path: Path) -> None: + code, out = _run(capsys, "schemes") + assert {"local", "memory"} <= set(json.loads(out)) + marker = tmp_path / "made-by-init" + init = json.dumps([["FA_create_dir", {"dir_path": str(marker)}]]) + code, out = _run(capsys, "--init", init, "schemes") + assert code == 0 + assert marker.is_dir() + + +def test_a_subcommand_is_required(capsys: pytest.CaptureFixture[str]) -> None: + with pytest.raises(SystemExit): + main(["storage"]) + assert "storage" in capsys.readouterr().err From c58b5b396de2813947079f67a77cdc4b082d547b Mon Sep 17 00:00:00 2001 From: JeffreyChen Date: Thu, 8 Oct 2026 13:27:05 +0800 Subject: [PATCH 32/59] feat: add storage adapters for eight more backends SFTP, FTP/FTPS, Google Drive, OneDrive and Dropbox resolve by URI through the shared clients; WebDAV, SMB and fsspec are mounted. A copy or a move of a file onto itself through two views of one store is refused (file identity), the WebDAV client only talks to its own server and no longer leaves redirects to requests, and the SFTP, OneDrive and SMB clients name the extra to install when their SDK is missing. --- CLAUDE.md | 5 +- README.md | 33 +- README.zh-CN.md | 31 +- README.zh-TW.md | 31 +- architecture.md | 8 +- automation_file/__init__.py | 18 + automation_file/exceptions.py | 10 +- automation_file/remote/ftp/client.py | 21 +- automation_file/remote/onedrive/client.py | 33 +- automation_file/remote/sftp/client.py | 25 +- automation_file/remote/smb/client.py | 105 ++- automation_file/remote/webdav/client.py | 202 ++++- automation_file/storage/__init__.py | 28 +- automation_file/storage/azure_storage.py | 5 +- automation_file/storage/backend.py | 14 +- automation_file/storage/dropbox_storage.py | 342 +++++++ automation_file/storage/fsspec_storage.py | 331 +++++++ automation_file/storage/ftp_storage.py | 380 ++++++++ automation_file/storage/gdrive_storage.py | 557 ++++++++++++ automation_file/storage/local_storage.py | 6 +- automation_file/storage/object_storage.py | 9 +- automation_file/storage/onedrive_storage.py | 495 +++++++++++ automation_file/storage/resolver.py | 16 + automation_file/storage/s3_storage.py | 5 +- automation_file/storage/session_storage.py | 283 ++++++ automation_file/storage/sftp_storage.py | 262 ++++++ automation_file/storage/smb_storage.py | 226 +++++ automation_file/storage/timestamps.py | 40 + automation_file/storage/webdav_storage.py | 264 ++++++ docs/source/API/storage.rst | 36 + docs/source/Eng/usage/storage.rst | 300 ++++++- docs/source/Zh-CN/usage/storage.rst | 261 +++++- docs/source/Zh-TW/usage/storage.rst | 261 +++++- docs/updates/2026-10.md | 58 ++ docs/updates/README.md | 6 +- progress.md | 8 +- tests/drive_stand_in.py | 418 +++++++++ tests/ftp_stand_in.py | 380 ++++++++ tests/graph_stand_in.py | 418 +++++++++ tests/test_backends.py | 47 + tests/test_ftp_ops.py | 52 ++ tests/test_optional_dependencies.py | 17 +- tests/test_smb_client.py | 98 +- tests/test_storage_azure.py | 17 + tests/test_storage_dropbox.py | 734 +++++++++++++++ tests/test_storage_fsspec.py | 606 +++++++++++++ tests/test_storage_ftp.py | 865 ++++++++++++++++++ tests/test_storage_gdrive.py | 787 ++++++++++++++++ tests/test_storage_local.py | 21 + tests/test_storage_onedrive.py | 715 +++++++++++++++ tests/test_storage_resolver.py | 15 +- tests/test_storage_s3.py | 16 + tests/test_storage_sftp.py | 939 ++++++++++++++++++++ tests/test_storage_sftp_loopback.py | 239 +++++ tests/test_storage_smb.py | 508 +++++++++++ tests/test_storage_timestamps.py | 58 ++ tests/test_storage_webdav.py | 608 +++++++++++++ tests/test_webdav_client.py | 241 ++++- 58 files changed, 12362 insertions(+), 152 deletions(-) create mode 100644 automation_file/storage/dropbox_storage.py create mode 100644 automation_file/storage/fsspec_storage.py create mode 100644 automation_file/storage/ftp_storage.py create mode 100644 automation_file/storage/gdrive_storage.py create mode 100644 automation_file/storage/onedrive_storage.py create mode 100644 automation_file/storage/session_storage.py create mode 100644 automation_file/storage/sftp_storage.py create mode 100644 automation_file/storage/smb_storage.py create mode 100644 automation_file/storage/timestamps.py create mode 100644 automation_file/storage/webdav_storage.py create mode 100644 tests/drive_stand_in.py create mode 100644 tests/ftp_stand_in.py create mode 100644 tests/graph_stand_in.py create mode 100644 tests/test_storage_dropbox.py create mode 100644 tests/test_storage_fsspec.py create mode 100644 tests/test_storage_ftp.py create mode 100644 tests/test_storage_gdrive.py create mode 100644 tests/test_storage_onedrive.py create mode 100644 tests/test_storage_sftp.py create mode 100644 tests/test_storage_sftp_loopback.py create mode 100644 tests/test_storage_smb.py create mode 100644 tests/test_storage_timestamps.py create mode 100644 tests/test_storage_webdav.py diff --git a/CLAUDE.md b/CLAUDE.md index a9e7c77..cb3671d 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -28,6 +28,8 @@ automation_file/ ├── storage/ # Universal storage layer: uri (StorageURI), types (FileInfo, Checksum, │ # StorageCapabilities), backend (StorageBackend contract), local_storage, │ # memory_storage, object_storage (ObjectStorage), s3_storage, azure_storage, +│ # session_storage (SessionStorage), sftp_storage, ftp_storage, gdrive_storage, +│ # onedrive_storage, dropbox_storage, webdav_storage, smb_storage, fsspec_storage, │ # resolver (StorageResolver), file (File), storage (Storage), streams, │ # tree (copy_tree, sync_tree), │ # actions (FA_storage_* and register_storage_ops) @@ -71,7 +73,7 @@ automation_file/ - `retry_on_transient(max_attempts, backoff_base, backoff_cap, retriable)` — decorator that retries with capped exponential back-off and raises `RetryExhaustedException` chained to the last error. - `safe_join(root, user_path)` / `is_within(root, path)` — path traversal guard; `safe_join` raises `PathTraversalException` when the resolved path escapes `root`. - `File(uri)` / `Storage(uri)` — the universal storage layer's application API: one file, one directory, in any backend. Both resolve their backend on every call through `StorageResolver` (`Storage.mount`, `Storage.register_scheme`). -- `StorageBackend` — the contract a storage backend implements. The public operations (`exists`, `stat`, `list_dir`, `mkdir`, `upload`, `download`, `delete`, `checksum`, `read_bytes`, `write_bytes`, `copy_from`, `move_from`) are template methods; a backend supplies only the `_`-prefixed primitives. `LocalStorage`, `MemoryStorage`, `S3Storage` (`s3://bucket/key`) and `AzureStorage` (`azure://container/blob`) are built in; the last two extend `ObjectStorage` and use the shared `s3_instance` / `azure_blob_instance` unless given a client. +- `StorageBackend` — the contract a storage backend implements. The public operations (`exists`, `stat`, `list_dir`, `mkdir`, `upload`, `download`, `delete`, `checksum`, `read_bytes`, `write_bytes`, `copy_from`, `move_from`) are template methods; a backend supplies only the `_`-prefixed primitives. Twelve are built in: `LocalStorage`, `MemoryStorage`, `S3Storage` and `AzureStorage` (both on `ObjectStorage`), `SFTPStorage` and `FTPStorage` (both on `SessionStorage`), `GoogleDriveStorage`, `OneDriveStorage`, `DropboxStorage`, and the mounted `WebDAVStorage`, `SMBStorage` and `FsspecStorage`. Each uses its backend's shared client singleton unless given one, and reports a missing SDK with the extra to install. - `Event` / `EventBus` / `event_bus` — every component reports through events (`PipelineFailed`, `TaskFailed`, `IntegrityViolation`, `StorageError`, ...) with a severity, a correlation ID and an actor; consumers subscribe on the bus by class, type name or prefix. New code that has something to report publishes an event; it does not call a notification sink or the audit log directly. - `StorageURI` / `parse_storage_uri` — `:///`; `FileInfo`, `Checksum`, `StorageCapabilities` are the frozen value types the layer returns. @@ -160,6 +162,7 @@ All code must follow secure-by-default principles. Review every change against t ### Storage layer - A new storage backend subclasses `StorageBackend` and passes `tests/storage_contract.py` through a `StorageContract` subclass. Do not weaken a contract case to make a backend pass: fix the backend, or branch on `capabilities` when backends legitimately differ. - Keep the checks that live in the base class: `normalize_path` refuses `..`, `parse_storage_uri` refuses credentials in the authority (and its error does not repeat them), `delete` refuses the storage root, and `LocalStorage` deletes a symbolic link without following it. Never log a storage URI's credentials or a backend's secrets. +- A backend that can show one stored file through two instances (two roots, two prefixes, a link) overrides `_identity`, so a copy or a move of a file onto itself is refused instead of deleting it. - When paths come from outside the process, use `LocalStorage(root)` behind a scheme or authority of its own (`Storage.mount("sandbox://jobs", LocalStorage(root))`), not the rootless `local://` backend. - At module level, `automation_file/storage/` imports only the standard library, `exceptions`, `logging_config`, `core.checksum` and `local.safe_paths`; `tests/test_storage_imports.py` fails on anything else. A backend SDK is imported lazily, inside the function that needs it. diff --git a/README.md b/README.md index d758261..8872c1b 100644 --- a/README.md +++ b/README.md @@ -48,7 +48,7 @@ facade. - **HTTP server observability** — `GET /healthz` / `GET /readyz` probes, `GET /openapi.json` spec, and `GET /progress` WebSocket stream of live transfer snapshots - **HTMX Web UI** — `start_web_ui()` serves a read-only dashboard (health, progress, registry) that polls HTML fragments; stdlib-only HTTP plus one CDN script with SRI - **MCP (Model Context Protocol) server** — `MCPServer` bridges the registry to any MCP host (Claude Desktop, MCP CLIs) over newline-delimited JSON-RPC 2.0 on stdio; every `FA_*` action becomes an MCP tool with an auto-generated input schema -- **Universal storage layer** — `File` / `Storage` address local and remote storage with one URI syntax (`local:///…`, `s3://…`, `azure://…`, `memory://…`), one `StorageBackend` contract and one error hierarchy; local, S3, Azure Blob and in-memory backends are built in, and an 81-case contract suite checks any backend +- **Universal storage layer** — `File` / `Storage` address local and remote storage with one URI syntax (`local:///…`, `s3://…`, `azure://…`, `gdrive://…`, `sftp://…`, …), one `StorageBackend` contract and one error hierarchy; twelve backends are built in (local, in-memory, S3, Azure Blob, Google Drive, Dropbox, OneDrive, SFTP, FTP / FTPS, WebDAV, SMB, fsspec), and an 81-case contract suite checks any backend - **Event bus** — one `Event` model with ten core events (`pipeline.*`, `task.*`, `integrity.violation`, `storage.error`, `scheduler.error`, `system.error`), severities, correlation IDs and actors; subscribe on `event_bus` by class, type or prefix - PySide6 GUI (`python -m automation_file ui`) with a tab per backend, the JSON-action runner, and dedicated tabs for Triggers, Scheduler, and live Progress - Rich CLI with one-shot subcommands plus legacy JSON-batch flags @@ -149,9 +149,9 @@ flowchart TD end subgraph StorageLayer["storage (universal layer)"] - FileAPI["File · Storage
local:// memory:// s3:// azure://"] + FileAPI["File · Storage
local:// s3:// azure:// gdrive:// sftp:// …"] Resolver["StorageResolver
mounts · scheme factories"] - Backends["StorageBackend contract
Local · Memory · S3 · Azure"] + Backends["StorageBackend contract
Local · Memory · S3 · Azure · Drive · Dropbox
OneDrive · SFTP · FTP · WebDAV · SMB · fsspec"] end subgraph Notify["notifications"] @@ -192,6 +192,14 @@ flowchart TD Backends ==> Check Backends ==> S3M Backends ==> Azure + Backends ==> Drive + Backends ==> Dropbox + Backends ==> OneD + Backends ==> SFTP + Backends ==> FTP + Backends ==> WebDAV + Backends ==> SMB + Backends ==> Fsspec TCP ==> Executor HTTPS ==> Executor @@ -499,15 +507,16 @@ File("sandbox://jobs/42/out.csv").write(b"done") `StorageAlreadyExistsException`, `StoragePathTypeException`, `StorageNotEmptyException`, `StoragePermissionException`, `StorageTransientException`, `StorageUnavailableException`, `StorageUnsupportedException`, `StorageURIException`. -- **Backends today** — `LocalStorage` (`local://`, optionally confined to a root through - `safe_join`), `S3Storage` (`s3://bucket/key`), `AzureStorage` (`azure://container/blob`) and - `MemoryStorage` (`memory://`, for tests and dry runs). S3 and Azure use the clients you already - initialise (`s3_instance.later_init(...)`, `azure_blob_instance.later_init(...)`), so - `File("s3://reports/q1.csv").copy_to("azure://backups/q1.csv")` works once both are ready. - Google Drive, Dropbox, SFTP, FTP, WebDAV, SMB and fsspec are still used through their own - clients and `FA_*` actions; their adapters are not written yet. Write your own by subclassing - `StorageBackend` (or `ObjectStorage` for an object store) and check it with the 81-case - contract suite in `tests/storage_contract.py`. +- **Backends** — twelve are built in. Addressed by URI through the shared clients you already + initialise: `local://`, `memory://`, `s3://bucket/key`, `azure://container/blob`, `gdrive:///path`, + `dropbox:///path`, `onedrive:///path`, `sftp://host/path`, `ftp://host/path` and `ftps://host/path`. + Mounted, because they need a client or a filesystem of their own: `WebDAVStorage`, `SMBStorage` and + `FsspecStorage` (`Storage.mount("webdav://files.example.com", WebDAVStorage(client))`). Each remote + backend needs its extra (`pip install "automation_file[sftp]"`). An `sftp://` or `ftp://` URI must + name the host the session is connected to, so a typo cannot write to another server. Box has no + adapter and stays on its `FA_box_*` actions. Write your own by subclassing `StorageBackend` + (`ObjectStorage` for an object store, `SessionStorage` for a login session) and check it with the + 81-case contract suite in `tests/storage_contract.py`. - **Actions** — `FA_storage_exists`, `FA_storage_stat`, `FA_storage_list`, `FA_storage_mkdir`, `FA_storage_upload`, `FA_storage_download`, `FA_storage_delete`, `FA_storage_checksum`, diff --git a/README.zh-CN.md b/README.zh-CN.md index d2bb93a..474a4bf 100644 --- a/README.zh-CN.md +++ b/README.zh-CN.md @@ -46,7 +46,7 @@ TCP / HTTP 服务器执行的 JSON 驱动动作。内附 PySide6 GUI，每个功 - **HTTP 服务器观测端点** — `GET /healthz` / `GET /readyz` 探针、`GET /openapi.json` 规格，以及 `GET /progress`（通过 WebSocket 推送实时传输快照） - **HTMX Web UI** — `start_web_ui()` 启动只读观测仪表板（health、progress、registry），通过 HTML 片段轮询；仅用标准库 HTTP，搭配一个带 SRI 的 CDN 脚本 - **MCP（Model Context Protocol）服务器** — `MCPServer` 通过 stdio 上的 JSON-RPC 2.0（换行分隔 JSON）将注册表桥接到任意 MCP 主机（Claude Desktop、MCP CLI）；每个 `FA_*` 动作都会自动生成输入 schema 并成为 MCP 工具 -- **通用存储层** — `File` / `Storage` 以同一套 URI 语法（`local:///…`、`s3://…`、`azure://…`、`memory://…`）、同一份 `StorageBackend` 契约与同一组异常层级访问本地与远端存储；内置本地、S3、Azure Blob 与内存后端，并附带 81 个用例的契约测试套件可检查任何后端 +- **通用存储层** — `File` / `Storage` 以同一套 URI 语法（`local:///…`、`s3://…`、`azure://…`、`gdrive://…`、`sftp://…`、…）、同一份 `StorageBackend` 契约与同一组异常层级访问本地与远端存储；内置十二种后端（本地、内存、S3、Azure Blob、Google Drive、Dropbox、OneDrive、SFTP、FTP / FTPS、WebDAV、SMB、fsspec），并附带 81 个用例的契约测试套件可检查任何后端 - **事件总线** — 单一 `Event` 模型与十种核心事件（`pipeline.*`、`task.*`、`integrity.violation`、`storage.error`、`scheduler.error`、`system.error`），具备严重程度、关联 ID 与 actor；可以在 `event_bus` 上按类、type 或前缀订阅 - PySide6 GUI（`python -m automation_file ui`）每个后端一个页签，含 JSON 动作执行器，另有 Triggers、Scheduler、实时 Progress 专属页签 - 功能丰富的 CLI，包含一次性子命令与旧式 JSON 批量标志 @@ -147,9 +147,9 @@ flowchart TD end subgraph StorageLayer["通用存储层"] - FileAPI["File · Storage
local:// memory:// s3:// azure://"] + FileAPI["File · Storage
local:// s3:// azure:// gdrive:// sftp:// …"] Resolver["StorageResolver
mounts · scheme factories"] - Backends["StorageBackend contract
Local · Memory · S3 · Azure"] + Backends["StorageBackend contract
Local · Memory · S3 · Azure · Drive · Dropbox
OneDrive · SFTP · FTP · WebDAV · SMB · fsspec"] end subgraph Notify["通知"] @@ -190,6 +190,14 @@ flowchart TD Backends ==> Check Backends ==> S3M Backends ==> Azure + Backends ==> Drive + Backends ==> Dropbox + Backends ==> OneD + Backends ==> SFTP + Backends ==> FTP + Backends ==> WebDAV + Backends ==> SMB + Backends ==> Fsspec TCP ==> Executor HTTPS ==> Executor @@ -494,14 +502,15 @@ File("sandbox://jobs/42/out.csv").write(b"done") `StorageAlreadyExistsException`、`StoragePathTypeException`、`StorageNotEmptyException`、 `StoragePermissionException`、`StorageTransientException`、`StorageUnavailableException`、 `StorageUnsupportedException`、`StorageURIException`。 -- **目前的后端** — `LocalStorage`（`local://`，可通过 `safe_join` 限制在某个根目录内）、 - `S3Storage`（`s3://bucket/key`）、`AzureStorage`（`azure://container/blob`）与 - `MemoryStorage`（`memory://`，用于测试与试运行）。S3 与 Azure 使用你原本就会初始化的客户端 - （`s3_instance.later_init(...)`、`azure_blob_instance.later_init(...)`），两者都就绪后， - `File("s3://reports/q1.csv").copy_to("azure://backups/q1.csv")` 即可运行。 - Google Drive、Dropbox、SFTP、FTP、WebDAV、SMB 与 fsspec 目前仍通过各自的客户端与 `FA_*` - 动作使用，其适配器尚未完成。你可以继承 `StorageBackend`（对象存储则继承 `ObjectStorage`） - 编写自己的后端，并用 `tests/storage_contract.py` 中 81 个用例的契约测试套件检查。 +- **后端** — 内置十二种。可以直接用 URI 访问、并使用你原本就会初始化的共用客户端的有：`local://`、 + `memory://`、`s3://bucket/key`、`azure://container/blob`、`gdrive:///path`、`dropbox:///path`、 + `onedrive:///path`、`sftp://host/path`、`ftp://host/path` 与 `ftps://host/path`。需要自己的客户端或 + 文件系统、因此以挂载方式使用的有：`WebDAVStorage`、`SMBStorage` 与 `FsspecStorage` + （`Storage.mount("webdav://files.example.com", WebDAVStorage(client))`）。每个远端后端都需要对应的 + extra（`pip install "automation_file[sftp]"`）。`sftp://` 或 `ftp://` URI 必须写出会话实际连接 + 的主机，打错字就不会写到另一台服务器。Box 没有适配器，仍使用它的 `FA_box_*` 动作。你可以继承 + `StorageBackend`（对象存储继承 `ObjectStorage`，登录会话继承 `SessionStorage`）编写自己的后端， + 并用 `tests/storage_contract.py` 中 81 个用例的契约测试套件检查。 - **动作** — `FA_storage_exists`, `FA_storage_stat`, `FA_storage_list`, `FA_storage_mkdir`, `FA_storage_upload`, `FA_storage_download`, `FA_storage_delete`, `FA_storage_checksum`, diff --git a/README.zh-TW.md b/README.zh-TW.md index 9530127..fcd11f7 100644 --- a/README.zh-TW.md +++ b/README.zh-TW.md @@ -46,7 +46,7 @@ TCP / HTTP 伺服器執行的 JSON 驅動動作。內附 PySide6 GUI，每個功 - **HTTP 伺服器觀測端點** — `GET /healthz` / `GET /readyz` 探針、`GET /openapi.json` 規格、以及 `GET /progress`（以 WebSocket 推送即時傳輸快照） - **HTMX Web UI** — `start_web_ui()` 啟動唯讀觀測儀表板（health、progress、registry），以 HTML 片段輪詢；僅用標準函式庫 HTTP，搭配一支帶 SRI 的 CDN 腳本 - **MCP（Model Context Protocol）伺服器** — `MCPServer` 透過 stdio 上的 JSON-RPC 2.0（行分隔 JSON）將登錄表橋接到任何 MCP 主機（Claude Desktop、MCP CLI）；每個 `FA_*` 動作都會自動生成輸入 schema 並成為 MCP 工具 -- **通用儲存層** — `File` / `Storage` 以同一套 URI 語法（`local:///…`、`s3://…`、`azure://…`、`memory://…`）、同一份 `StorageBackend` 契約與同一組例外階層存取本機與遠端儲存；內建本機、S3、Azure Blob 與記憶體後端，並附 81 個案例的契約測試套件可檢查任何後端 +- **通用儲存層** — `File` / `Storage` 以同一套 URI 語法（`local:///…`、`s3://…`、`azure://…`、`gdrive://…`、`sftp://…`、…）、同一份 `StorageBackend` 契約與同一組例外階層存取本機與遠端儲存；內建十二種後端（本機、記憶體、S3、Azure Blob、Google Drive、Dropbox、OneDrive、SFTP、FTP / FTPS、WebDAV、SMB、fsspec），並附 81 個案例的契約測試套件可檢查任何後端 - **事件匯流排** — 單一 `Event` 模型與十種核心事件（`pipeline.*`、`task.*`、`integrity.violation`、`storage.error`、`scheduler.error`、`system.error`），具備嚴重程度、關聯 ID 與 actor；可在 `event_bus` 上依類別、type 或前綴訂閱 - PySide6 GUI（`python -m automation_file ui`）每個後端一個分頁，含 JSON 動作執行器，另有 Triggers、Scheduler、即時 Progress 專屬分頁 - 功能豐富的 CLI，包含一次性子指令與舊式 JSON 批次旗標 @@ -147,9 +147,9 @@ flowchart TD end subgraph StorageLayer["通用儲存層"] - FileAPI["File · Storage
local:// memory:// s3:// azure://"] + FileAPI["File · Storage
local:// s3:// azure:// gdrive:// sftp:// …"] Resolver["StorageResolver
mounts · scheme factories"] - Backends["StorageBackend contract
Local · Memory · S3 · Azure"] + Backends["StorageBackend contract
Local · Memory · S3 · Azure · Drive · Dropbox
OneDrive · SFTP · FTP · WebDAV · SMB · fsspec"] end subgraph Notify["通知"] @@ -190,6 +190,14 @@ flowchart TD Backends ==> Check Backends ==> S3M Backends ==> Azure + Backends ==> Drive + Backends ==> Dropbox + Backends ==> OneD + Backends ==> SFTP + Backends ==> FTP + Backends ==> WebDAV + Backends ==> SMB + Backends ==> Fsspec TCP ==> Executor HTTPS ==> Executor @@ -494,14 +502,15 @@ File("sandbox://jobs/42/out.csv").write(b"done") `StorageAlreadyExistsException`、`StoragePathTypeException`、`StorageNotEmptyException`、 `StoragePermissionException`、`StorageTransientException`、`StorageUnavailableException`、 `StorageUnsupportedException`、`StorageURIException`。 -- **目前的後端** — `LocalStorage`（`local://`，可透過 `safe_join` 限制在某個根目錄內）、 - `S3Storage`（`s3://bucket/key`）、`AzureStorage`（`azure://container/blob`）與 - `MemoryStorage`（`memory://`，用於測試與試跑）。S3 與 Azure 使用你原本就會初始化的用戶端 - （`s3_instance.later_init(...)`、`azure_blob_instance.later_init(...)`），兩者都就緒後， - `File("s3://reports/q1.csv").copy_to("azure://backups/q1.csv")` 即可運作。 - Google Drive、Dropbox、SFTP、FTP、WebDAV、SMB 與 fsspec 目前仍透過各自的用戶端與 `FA_*` - 動作使用，其轉接器尚未完成。你可以繼承 `StorageBackend`（物件儲存則繼承 `ObjectStorage`） - 撰寫自己的後端，並用 `tests/storage_contract.py` 中 81 個案例的契約測試套件檢查。 +- **後端** — 內建十二種。可直接以 URI 存取、並使用你原本就會初始化的共用用戶端的有：`local://`、 + `memory://`、`s3://bucket/key`、`azure://container/blob`、`gdrive:///path`、`dropbox:///path`、 + `onedrive:///path`、`sftp://host/path`、`ftp://host/path` 與 `ftps://host/path`。需要自己的用戶端或 + 檔案系統、因此以掛載方式使用的有：`WebDAVStorage`、`SMBStorage` 與 `FsspecStorage` + （`Storage.mount("webdav://files.example.com", WebDAVStorage(client))`）。每個遠端後端都需要對應的 + extra（`pip install "automation_file[sftp]"`）。`sftp://` 或 `ftp://` URI 必須寫出工作階段實際連線 + 的主機，打錯字就不會寫到另一台伺服器。Box 沒有轉接器，仍使用它的 `FA_box_*` 動作。你可以繼承 + `StorageBackend`（物件儲存繼承 `ObjectStorage`，登入工作階段繼承 `SessionStorage`）撰寫自己的後端， + 並用 `tests/storage_contract.py` 中 81 個案例的契約測試套件檢查。 - **動作** — `FA_storage_exists`, `FA_storage_stat`, `FA_storage_list`, `FA_storage_mkdir`, `FA_storage_upload`, `FA_storage_download`, `FA_storage_delete`, `FA_storage_checksum`, diff --git a/architecture.md b/architecture.md index 1d182bc..3d65590 100644 --- a/architecture.md +++ b/architecture.md @@ -24,7 +24,7 @@ piece of the 1.0 roadmap (`docs/FILEAUTOMATION-1.0-ROADMAP.md`, PR #107); what i | `automation_file/core/` | Engine, on je_action_core: `action_registry.py` (`ActionRegistry`, a `CommandRegistry`; `build_default_registry`), `action_executor.py` (`ActionExecutor`, an `ActionExecutor` with strict actions, indexed records and the dry-run, validate, substitute and parallel extras; shared `executor`), `callback_executor.py`, `package_loader.py`, `plugins.py`, `dag_executor.py`, `action_queue.py`, `json_store.py`, `substitution.py`. Also cross-cutting helpers: `optional` (`require_module`, the extras table), `retry`, `quota`, `rate_limit`, `circuit_breaker`, `file_lock`, `sqlite_lock`, `checksum`, `manifest`, `crypto`, `secrets`, `config`, `config_watcher`, `audit`, `metrics`, `tracing`, `progress`, `fim`, `content_store` | | `automation_file/local/` | Local strategy modules: file, dir, zip, tar and archive ops, sync, diff, text/JSON/data edits, templates, versioning, trash, `shell_ops` (argv-only subprocess), conditional branches. `safe_paths.py` guards against path traversal | | `automation_file/remote/` | `url_validator.py` (SSRF guard), `http_download.py`, `cross_backend.py`, `fsspec_bridge.py`. One subpackage per backend: `google_drive/`, `s3/`, `azure_blob/`, `dropbox_api/`, `sftp/`, `ftp/`, `onedrive/`, `box/`, each with `client.py`, `*_ops.py` and `register__ops`. `smb/` and `webdav/` have a client only | -| `automation_file/storage/` | Universal storage layer. `uri.py` (`StorageURI`, `parse_storage_uri`, `normalize_path`), `types.py` (`FileInfo`, `Checksum`, `StorageCapabilities`), `backend.py` (`StorageBackend`: the public operations are template methods over the `_`-prefixed primitives a backend supplies), `local_storage.py` (`LocalStorage`, confined through `safe_join` when given a root), `memory_storage.py` (`MemoryStorage`), `object_storage.py` (`ObjectStorage`: directories as key prefixes over `_head`, `_scan`, `_put`, `_get`, `_remove`), `s3_storage.py` (`S3Storage`, over `s3_instance` or a given boto3 client), `azure_storage.py` (`AzureStorage`, over `azure_blob_instance` or a given `BlobServiceClient`), `resolver.py` (`StorageResolver`, `default_resolver`: mounts first, then scheme factories), `file.py` (`File`), `storage.py` (`Storage`), `observe.py` (listeners for `upload`, `download`, `read`, `delete`, `mkdir`, `copy`, `move`), `streams.py` (staged file objects behind `open_read` / `open_write`), `tree.py` (`copy_tree`, `sync_tree`, `TreeResult`), `actions.py` (the `FA_storage_*` functions and `register_storage_ops`). At module level it imports only `exceptions`, `logging_config`, `core.checksum` and `local.safe_paths`: no registry, no GUI, no backend SDK. The adapters import their SDK's exceptions and the shared client inside the functions that use them | +| `automation_file/storage/` | Universal storage layer. `uri.py` (`StorageURI`, `parse_storage_uri`, `normalize_path`), `types.py` (`FileInfo`, `Checksum`, `StorageCapabilities`), `backend.py` (`StorageBackend`: the public operations are template methods over the `_`-prefixed primitives a backend supplies), `local_storage.py` (`LocalStorage`, confined through `safe_join` when given a root), `memory_storage.py` (`MemoryStorage`), `object_storage.py` (`ObjectStorage`: directories as key prefixes over `_head`, `_scan`, `_put`, `_get`, `_remove`), `s3_storage.py` (`S3Storage`, over `s3_instance` or a given boto3 client), `azure_storage.py` (`AzureStorage`, over `azure_blob_instance` or a given `BlobServiceClient`), `session_storage.py` (`SessionStorage`: one login session, one operation at a time, and `require_session_host`), `sftp_storage.py` (`SFTPStorage`), `ftp_storage.py` (`FTPStorage`, for `ftp` and `ftps`), `gdrive_storage.py` (`GoogleDriveStorage`: paths resolved to file IDs, duplicate names refused), `onedrive_storage.py` (`OneDriveStorage`, Microsoft Graph), `dropbox_storage.py` (`DropboxStorage`), `webdav_storage.py` (`WebDAVStorage`), `smb_storage.py` (`SMBStorage`), `fsspec_storage.py` (`FsspecStorage`, any fsspec filesystem), `timestamps.py` (RFC 3339 parsing), `resolver.py` (`StorageResolver`, `default_resolver`: mounts first, then scheme factories), `file.py` (`File`), `storage.py` (`Storage`), `observe.py` (listeners for `upload`, `download`, `read`, `delete`, `mkdir`, `copy`, `move`), `streams.py` (staged file objects behind `open_read` / `open_write`), `tree.py` (`copy_tree`, `sync_tree`, `TreeResult`), `actions.py` (the `FA_storage_*` functions and `register_storage_ops`). At module level it imports only `exceptions`, `logging_config`, `core.checksum` and `local.safe_paths`: no registry, no GUI, no backend SDK. The adapters import their SDK's exceptions and the shared client inside the functions that use them | | `automation_file/events/` | The event model every component reports through. `model.py` (`Event`, `Severity`, the ten core events), `bus.py` (`EventBus`, the process-wide `event_bus`, `emit`), `context.py` (`correlation_scope`, `actor_scope`), `storage_bridge.py` (failed storage operations become `StorageError` events; installed when the package is imported). It imports only the standard library, `logging_config` and `storage.observe` | | `automation_file/server/` | `tcp_server.py`, `http_server.py`, `mcp_server.py`, `web_ui.py`, `metrics_server.py`, `action_acl.py` (`ActionACL`), `network_guards.py` (`ensure_loopback`) | | `automation_file/client/` | `HTTPActionClient` for the HTTP action server | @@ -49,9 +49,11 @@ piece of the 1.0 roadmap (`docs/FILEAUTOMATION-1.0-ROADMAP.md`, PR #107); what i `HTTPActionClient`, `MCPServer`, `create_project_dir`, `launch_ui` (lazy). - **Storage layer** (same facade): `File`, `Storage`, `StorageBackend`, `StorageResolver`, `StorageURI`, `parse_storage_uri`, `FileInfo`, `Checksum`, `StorageCapabilities`, `LocalStorage`, `MemoryStorage`, - `ObjectStorage`, `S3Storage`, `AzureStorage`, and + `ObjectStorage`, `S3Storage`, `AzureStorage`, `SessionStorage`, `SFTPStorage`, `FTPStorage`, + `GoogleDriveStorage`, `OneDriveStorage`, `DropboxStorage`, `WebDAVStorage`, `SMBStorage`, + `FsspecStorage`, and `StorageException` with its nine subclasses. Storage URIs are `:///`; the built-in - schemes are `local` (alias `file`), `memory`, `s3` and `azure` (alias `az`), and text without `://` is a local path. `s3://` and `azure://` use the shared `s3_instance` / `azure_blob_instance`. The API is + schemes are `local` (alias `file`), `memory`, `s3`, `azure` (alias `az`), `gdrive`, `dropbox`, `onedrive`, `sftp`, `ftp` and `ftps`, each through its shared client singleton; text without `://` is a local path. WebDAV, SMB and fsspec backends are mounted with `Storage.mount`. An `sftp://` or `ftp://` URI with a host is refused unless the session is connected to that host. The API is provisional until 1.0. Sixteen `FA_storage_*` actions (`exists`, `stat`, `list`, `mkdir`, `upload`, `download`, `delete`, `checksum`, `verify`, `copy`, `move`, `read_text`, `write_text`, `copy_tree`, `sync`, `schemes`) put diff --git a/automation_file/__init__.py b/automation_file/__init__.py index 4e61bb9..f9f8cc1 100644 --- a/automation_file/__init__.py +++ b/automation_file/__init__.py @@ -287,18 +287,27 @@ from automation_file.storage import ( AzureStorage, Checksum, + DropboxStorage, File, FileInfo, + FsspecStorage, + FTPStorage, + GoogleDriveStorage, LocalStorage, MemoryStorage, ObjectStorage, + OneDriveStorage, S3Storage, + SessionStorage, + SFTPStorage, + SMBStorage, Storage, StorageBackend, StorageCapabilities, StorageResolver, StorageURI, TreeResult, + WebDAVStorage, parse_storage_uri, register_storage_ops, ) @@ -516,6 +525,15 @@ def __getattr__(name: str) -> Any: "ObjectStorage", "S3Storage", "AzureStorage", + "GoogleDriveStorage", + "DropboxStorage", + "OneDriveStorage", + "SFTPStorage", + "FTPStorage", + "WebDAVStorage", + "SMBStorage", + "FsspecStorage", + "SessionStorage", "StorageException", "StorageURIException", "StorageNotFoundException", diff --git a/automation_file/exceptions.py b/automation_file/exceptions.py index 07569c6..09b6958 100644 --- a/automation_file/exceptions.py +++ b/automation_file/exceptions.py @@ -108,7 +108,15 @@ class ArchiveException(FileAutomationException): class WebDAVException(FileAutomationException): - """Raised by the WebDAV client on transport / protocol failures.""" + """Raised by the WebDAV client on transport / protocol failures. + + ``status_code`` is the HTTP status the server answered with, or ``None`` when + the failure happened before a response arrived. + """ + + def __init__(self, *args: object, status_code: int | None = None) -> None: + super().__init__(*args) + self.status_code = status_code class SMBException(FileAutomationException): diff --git a/automation_file/remote/ftp/client.py b/automation_file/remote/ftp/client.py index a2767ae..ce6f6d4 100644 --- a/automation_file/remote/ftp/client.py +++ b/automation_file/remote/ftp/client.py @@ -37,7 +37,23 @@ class FTPClient: def __init__(self) -> None: self._ftp: FTP | None = None - self._host: str = "" + self._host: str | None = None + self._port: int | None = None + + @property + def host(self) -> str | None: + """The host of the open session, or ``None`` while there is none.""" + return self._host + + @property + def port(self) -> int | None: + """The port of the open session, or ``None`` while there is none.""" + return self._port + + @property + def tls(self) -> bool: + """Whether the open session is an FTPS (``FTP_TLS``) session.""" + return isinstance(self._ftp, FTP_TLS) def later_init(self, options: FTPConnectOptions | None = None, **kwargs: Any) -> FTP: """Open an FTP control connection. TLS is negotiated when ``tls=True``.""" @@ -62,6 +78,7 @@ def later_init(self, options: FTPConnectOptions | None = None, **kwargs: Any) -> raise FTPException(f"FTP connect failed: {err}") from err self._ftp = ftp self._host = opts.host + self._port = opts.port file_automation_logger.info( "FTPClient: connected to %s@%s:%d (tls=%s)", opts.username, @@ -77,6 +94,8 @@ def require_ftp(self) -> FTP: return self._ftp def close(self) -> bool: + self._host = None + self._port = None if self._ftp is not None: try: self._ftp.quit() diff --git a/automation_file/remote/onedrive/client.py b/automation_file/remote/onedrive/client.py index 3455a0f..a303285 100644 --- a/automation_file/remote/onedrive/client.py +++ b/automation_file/remote/onedrive/client.py @@ -23,6 +23,7 @@ import requests +from automation_file.core.optional import install_hint from automation_file.exceptions import OneDriveException from automation_file.logging_config import file_automation_logger @@ -36,7 +37,7 @@ def _import_msal() -> Any: import msal except ImportError as error: raise OneDriveException( - "msal import failed — reinstall `automation_file` to restore the OneDrive backend" + f"msal is not installed; the OneDrive backend needs it: {install_hint('onedrive')}" ) from error return msal @@ -102,6 +103,32 @@ def require_session(self) -> requests.Session: ) return self._session + def graph_send( + self, + method: str, + path: str, + *, + timeout: float = 30.0, + authorized: bool = True, + **request_kwargs: Any, + ) -> requests.Response: + """Send one request and return the response, whatever its status. + + ``path`` is resolved as in :meth:`graph_request`. ``authorized=False`` + leaves the bearer token out: the pre-authenticated URLs Graph hands out + (an upload session) must not receive it. A + :class:`requests.RequestException` is raised as it is, so the caller + decides what a status or a transport failure means. + """ + session = self.require_session() + url = path if path.startswith("http") else f"{_GRAPH_BASE}{path}" + if not authorized: + # requests leaves out a session header whose per-request value is None. + headers: dict[str, Any] = dict(request_kwargs.pop("headers", None) or {}) + headers["Authorization"] = None + request_kwargs["headers"] = headers + return session.request(method, url, timeout=timeout, **request_kwargs) + def graph_request( self, method: str, @@ -119,10 +146,8 @@ def graph_request( forwarded to :meth:`requests.Session.request` — ``params``, ``json``, ``data``, and ``headers`` are the common ones. """ - session = self.require_session() - url = path if path.startswith("http") else f"{_GRAPH_BASE}{path}" try: - response = session.request(method, url, timeout=timeout, **request_kwargs) + response = self.graph_send(method, path, timeout=timeout, **request_kwargs) except requests.RequestException as error: raise OneDriveException(f"graph request failed: {error}") from error if not response.ok: diff --git a/automation_file/remote/sftp/client.py b/automation_file/remote/sftp/client.py index 0c4861e..8a020fc 100644 --- a/automation_file/remote/sftp/client.py +++ b/automation_file/remote/sftp/client.py @@ -12,17 +12,12 @@ from pathlib import Path from typing import Any +from automation_file.core.optional import require_module from automation_file.logging_config import file_automation_logger def _import_paramiko() -> Any: - try: - import paramiko - except ImportError as error: - raise RuntimeError( - "paramiko import failed — reinstall `automation_file` to restore the SFTP backend" - ) from error - return paramiko + return require_module("paramiko", extra="sftp") @dataclass(frozen=True) @@ -44,6 +39,18 @@ class SFTPClient: def __init__(self) -> None: self._ssh: Any = None self._sftp: Any = None + self._host: str | None = None + self._port: int | None = None + + @property + def host(self) -> str | None: + """The host of the open session, or ``None`` while there is none.""" + return self._host + + @property + def port(self) -> int | None: + """The port of the open session, or ``None`` while there is none.""" + return self._port def later_init(self, options: SFTPConnectOptions | None = None, **kwargs: Any) -> Any: """Open the SSH + SFTP session. Raises if the host key is not pinned. @@ -76,6 +83,8 @@ def later_init(self, options: SFTPConnectOptions | None = None, **kwargs: Any) - ) self._ssh = ssh self._sftp = ssh.open_sftp() + self._host = opts.host + self._port = opts.port file_automation_logger.info( "SFTPClient: connected to %s@%s:%d", opts.username, opts.host, opts.port ) @@ -88,6 +97,8 @@ def require_sftp(self) -> Any: def close(self) -> bool: """Close the underlying SFTP and SSH connections.""" + self._host = None + self._port = None if self._sftp is not None: try: self._sftp.close() diff --git a/automation_file/remote/smb/client.py b/automation_file/remote/smb/client.py index 9c71e04..cb140d3 100644 --- a/automation_file/remote/smb/client.py +++ b/automation_file/remote/smb/client.py @@ -1,20 +1,23 @@ """SMB / CIFS client built on ``smbprotocol``'s high-level ``smbclient`` API. Scope mirrors :mod:`automation_file.remote.webdav.client` — existence check, -upload, download, delete, directory create, and shallow listing. The -underlying session is registered per ``(server, username)`` pair and torn +stat, upload, download, delete, rename, directory create, and shallow listing. +The underlying session is registered per ``(server, username)`` pair and torn down when :meth:`SMBClient.close` runs. ``smbprotocol`` is imported lazily so importing this module never touches the optional dependency. """ from __future__ import annotations +import errno import os from dataclasses import dataclass from pathlib import Path +from stat import S_ISDIR from types import TracebackType from typing import Any +from automation_file.core.optional import install_hint from automation_file.exceptions import SMBException _DEFAULT_PORT = 445 @@ -23,11 +26,15 @@ @dataclass(frozen=True) class SMBEntry: - """A single directory listing entry returned by :meth:`SMBClient.list_dir`.""" + """One file or directory, from :meth:`SMBClient.list_dir` or :meth:`SMBClient.stat`. + + ``mtime`` is the modification time in seconds since the epoch. + """ name: str is_dir: bool size: int | None + mtime: float | None = None def _import_smbclient() -> Any: @@ -35,11 +42,34 @@ def _import_smbclient() -> Any: import smbclient except ImportError as error: raise SMBException( - "smbprotocol import failed — install `smbprotocol` to use the SMB backend" + f"smbprotocol is not installed; the SMB backend needs it: {install_hint('smb')}" ) from error return smbclient +def _is_missing(error: OSError) -> bool: + """smbprotocol reports a missing path as an ``OSError`` subclass of its own with ``ENOENT``.""" + return isinstance(error, FileNotFoundError) or error.errno == errno.ENOENT + + +def _entry(name: str, is_dir: bool, details: Any) -> SMBEntry: + return SMBEntry( + name=name, + is_dir=is_dir, + size=None if is_dir else int(details.st_size), + mtime=float(details.st_mtime), + ) + + +def _listed(item: Any) -> SMBEntry: + is_dir = bool(item.is_dir()) + try: + details = item.stat() + except OSError: + return SMBEntry(name=item.name, is_dir=is_dir, size=None) + return _entry(item.name, is_dir, details) + + class SMBClient: """Minimal SMB client scoped to the operations used by this project.""" @@ -77,6 +107,16 @@ def __exit__( ) -> None: self.close() + @property + def server(self) -> str: + """The host this client connects to.""" + return self._server + + @property + def share(self) -> str: + """The share every remote path is resolved against.""" + return self._share + def close(self) -> None: if not self._registered: return @@ -119,13 +159,24 @@ def exists(self, remote_path: str) -> bool: self._ensure_session() smbclient = _import_smbclient() try: - smbclient.stat(self._unc(remote_path)) - except FileNotFoundError: - return False + smbclient.stat(self._unc(remote_path), port=self._port) except OSError as error: + if _is_missing(error): + return False raise SMBException(f"stat failed for {remote_path}: {error}") from error return True + def stat(self, remote_path: str) -> SMBEntry: + """Return the kind, size and modification time of ``remote_path``.""" + self._ensure_session() + smbclient = _import_smbclient() + try: + details = smbclient.stat(self._unc(remote_path), port=self._port) + except OSError as error: + raise SMBException(f"stat failed for {remote_path}: {error}") from error + name = remote_path.replace("\\", "/").rstrip("/").rsplit("/", 1)[-1] + return _entry(name, S_ISDIR(details.st_mode), details) + def upload(self, local_path: str | os.PathLike[str], remote_path: str) -> None: """Copy the contents of ``local_path`` to ``remote_path`` on the share.""" source = Path(local_path) @@ -136,7 +187,7 @@ def upload(self, local_path: str | os.PathLike[str], remote_path: str) -> None: try: with ( open(source, "rb") as src, - smbclient.open_file(self._unc(remote_path), mode="wb") as dst, + smbclient.open_file(self._unc(remote_path), mode="wb", port=self._port) as dst, ): while True: chunk = src.read(_CHUNK_SIZE) @@ -154,7 +205,7 @@ def download(self, remote_path: str, local_path: str | os.PathLike[str]) -> None smbclient = _import_smbclient() try: with ( - smbclient.open_file(self._unc(remote_path), mode="rb") as src, + smbclient.open_file(self._unc(remote_path), mode="rb", port=self._port) as src, open(dest, "wb") as out, ): while True: @@ -170,16 +221,30 @@ def delete(self, remote_path: str) -> None: self._ensure_session() smbclient = _import_smbclient() try: - smbclient.remove(self._unc(remote_path)) + smbclient.remove(self._unc(remote_path), port=self._port) except OSError as error: raise SMBException(f"delete failed for {remote_path}: {error}") from error + def rename(self, remote_path: str, new_path: str, *, overwrite: bool = False) -> None: + """Rename ``remote_path`` to ``new_path`` on the share. + + ``overwrite=True`` replaces a file already at ``new_path``; otherwise that + is an error. + """ + self._ensure_session() + smbclient = _import_smbclient() + rename = smbclient.replace if overwrite else smbclient.rename + try: + rename(self._unc(remote_path), self._unc(new_path), port=self._port) + except OSError as error: + raise SMBException(f"rename failed for {remote_path}: {error}") from error + def mkdir(self, remote_path: str) -> None: """Create the remote directory at ``remote_path`` (parents must exist).""" self._ensure_session() smbclient = _import_smbclient() try: - smbclient.makedirs(self._unc(remote_path), exist_ok=True) + smbclient.makedirs(self._unc(remote_path), exist_ok=True, port=self._port) except OSError as error: raise SMBException(f"mkdir failed for {remote_path}: {error}") from error @@ -188,7 +253,7 @@ def rmdir(self, remote_path: str) -> None: self._ensure_session() smbclient = _import_smbclient() try: - smbclient.rmdir(self._unc(remote_path)) + smbclient.rmdir(self._unc(remote_path), port=self._port) except OSError as error: raise SMBException(f"rmdir failed for {remote_path}: {error}") from error @@ -197,19 +262,7 @@ def list_dir(self, remote_path: str) -> list[SMBEntry]: self._ensure_session() smbclient = _import_smbclient() try: - dir_entries = list(smbclient.scandir(self._unc(remote_path))) + dir_entries = list(smbclient.scandir(self._unc(remote_path), port=self._port)) except OSError as error: raise SMBException(f"list_dir failed for {remote_path}: {error}") from error - entries: list[SMBEntry] = [] - for item in dir_entries: - is_dir = bool(item.is_dir()) - size: int | None - if is_dir: - size = None - else: - try: - size = int(item.stat().st_size) - except OSError: - size = None - entries.append(SMBEntry(name=item.name, is_dir=is_dir, size=size)) - return entries + return [_listed(item) for item in dir_entries] diff --git a/automation_file/remote/webdav/client.py b/automation_file/remote/webdav/client.py index ab787f6..4b13f40 100644 --- a/automation_file/remote/webdav/client.py +++ b/automation_file/remote/webdav/client.py @@ -1,8 +1,9 @@ """WebDAV client built on ``requests``. -Supports the minimal set used for file automation — ``PUT`` upload, ``GET`` -download, ``DELETE``, ``MKCOL`` directory create, ``HEAD`` existence check, and -``PROPFIND`` listing. All URLs pass through +Supports the set used for file automation — ``PUT`` upload, ``GET`` download, +``DELETE``, ``MKCOL`` directory create, ``HEAD`` existence check, ``PROPFIND`` +lookup and listing, and ``COPY`` / ``MOVE`` on the server. The base URL, and a +``COPY`` / ``MOVE`` destination outside it, pass through :func:`automation_file.remote.url_validator.validate_http_url`; private / loopback hosts require ``allow_private_hosts=True``. """ @@ -14,7 +15,7 @@ from pathlib import Path from types import TracebackType from typing import Any -from urllib.parse import quote, unquote, urlparse +from urllib.parse import quote, unquote, urljoin, urlsplit import requests from defusedxml.ElementTree import ParseError as DefusedParseError @@ -26,11 +27,20 @@ _DAV_NS = "{DAV:}" _DEFAULT_TIMEOUT = 30.0 _ABSOLUTE_URL_PREFIXES = ("http" + "://", "https://") +_MULTI_STATUS = 207 +_REDIRECT_STATUS = frozenset({301, 302, 303, 307, 308}) +_MAX_REDIRECTS = 5 +# A redirect is followed only for requests that change nothing on the server. +_SAFE_METHODS = frozenset({"GET", "HEAD", "OPTIONS", "PROPFIND"}) +_DEPTH_SELF = "0" +_DEPTH_MEMBERS = "1" +_PROPFIND_CONTENT_TYPE = 'application/xml; charset="utf-8"' _PROPFIND_BODY = ( '' '' "" "" + "" "" "" ) @@ -38,13 +48,18 @@ @dataclass(frozen=True) class WebDAVEntry: - """A single directory listing entry returned by :meth:`WebDAVClient.list_dir`.""" + """One resource as ``PROPFIND`` describes it. + + Returned by :meth:`WebDAVClient.list_dir` and :meth:`WebDAVClient.stat`. + """ href: str name: str is_dir: bool size: int | None last_modified: str | None + etag: str | None = None + content_type: str | None = None class WebDAVClient: @@ -62,6 +77,7 @@ def __init__( ) -> None: validate_http_url(base_url, allow_private=allow_private_hosts) self._base_url = base_url.rstrip("/") + self._allow_private_hosts = allow_private_hosts self._auth: tuple[str, str] | None = ( (username, password) if username is not None and password is not None else None ) @@ -83,6 +99,11 @@ def __exit__( def close(self) -> None: self._session.close() + @property + def base_url(self) -> str: + """The URL every remote path is resolved against, without a trailing slash.""" + return self._base_url + def _url_for(self, remote_path: str) -> str: remote_path = remote_path.strip() if remote_path.startswith(_ABSOLUTE_URL_PREFIXES): @@ -92,37 +113,99 @@ def _url_for(self, remote_path: str) -> str: return self._base_url + "/" return f"{self._base_url}/{quote(remote_path, safe='/')}" - def _request(self, method: str, remote_path: str, **kwargs: Any) -> requests.Response: + def _own_url(self, remote_path: str) -> str: + """Return the URL of ``remote_path``, refusing one outside this client's server.""" url = self._url_for(remote_path) - try: + self._require_own_server(url) + return url + + def _require_own_server(self, url: str) -> None: + """Raise unless ``url`` is on the scheme, host and port this client was built for. + + A request carries the client's credentials, so it never goes anywhere else. + """ + ours, theirs = urlsplit(self._base_url), urlsplit(url) + own = (ours.scheme, ours.netloc.rpartition("@")[2].lower()) + other = (theirs.scheme, theirs.netloc.rpartition("@")[2].lower()) + if other != own: + raise WebDAVException( + f"refusing a request to {other[0]}://{other[1]}: " + f"this client only talks to {own[0]}://{own[1]}" + ) + + def _send(self, method: str, url: str, **kwargs: Any) -> requests.Response: + """Send one request. A redirect is followed by hand, and only when it is safe. + + ``requests`` would follow a redirect anywhere, past the URL validation the + client was built with. Here a redirect is followed for a read-only method + to a location on the same server, and reported as an error otherwise. + """ + for _ in range(_MAX_REDIRECTS + 1): response = self._session.request( method, url, auth=self._auth, timeout=self._timeout, verify=self._verify_tls, + allow_redirects=False, **kwargs, ) + location = self._redirect_target(method, url, response) + if location is None: + return response + response.close() + url = location + raise WebDAVException(f"{method} {url}: more than {_MAX_REDIRECTS} redirects") + + def _redirect_target(self, method: str, url: str, response: requests.Response) -> str | None: + """Return where to repeat the request, or ``None`` when ``response`` is the answer.""" + if response.status_code not in _REDIRECT_STATUS: + return None + location = response.headers.get("Location") + if not location or method not in _SAFE_METHODS: + response.close() + raise WebDAVException( + f"{method} {url} -> HTTP {response.status_code}: the redirect is not followed", + status_code=response.status_code, + ) + target = urljoin(url, location) + self._require_own_server(target) + return target + + def _request(self, method: str, remote_path: str, **kwargs: Any) -> requests.Response: + url = self._own_url(remote_path) + try: + response = self._send(method, url, **kwargs) except requests.RequestException as error: raise WebDAVException(f"{method} {url} failed: {error}") from error if response.status_code >= 400: response.close() raise WebDAVException( - f"{method} {url} -> HTTP {response.status_code}: {response.reason}" + f"{method} {url} -> HTTP {response.status_code}: {response.reason}", + status_code=response.status_code, ) return response + def _change(self, method: str, remote_path: str, **kwargs: Any) -> None: + """Send a request that changes a resource and everything below it. + + A 207 answer to ``DELETE``, ``COPY`` or ``MOVE`` lists the members the + server could not change, so it is a failure. + """ + response = self._request(method, remote_path, **kwargs) + response.close() + if response.status_code == _MULTI_STATUS: + raise WebDAVException( + f"{method} {self._url_for(remote_path)} -> HTTP {_MULTI_STATUS}: " + "the server could not complete it for every member", + status_code=_MULTI_STATUS, + ) + def exists(self, remote_path: str) -> bool: """Return True if the remote resource exists (HEAD 200-299).""" - url = self._url_for(remote_path) + url = self._own_url(remote_path) try: - response = self._session.request( - "HEAD", - url, - auth=self._auth, - timeout=self._timeout, - verify=self._verify_tls, - ) + response = self._send("HEAD", url) except requests.RequestException as error: raise WebDAVException(f"HEAD {url} failed: {error}") from error response.close() @@ -134,7 +217,10 @@ def upload(self, local_path: str | os.PathLike[str], remote_path: str) -> None: if not source.is_file(): raise WebDAVException(f"local source is not a file: {source}") with open(source, "rb") as fh: - response = self._request("PUT", remote_path, data=fh) + # requests sends an empty stream chunked, which some servers refuse; + # empty bytes go out with ``Content-Length: 0``. + body: Any = fh if source.stat().st_size else b"" + response = self._request("PUT", remote_path, data=body) response.close() def download(self, remote_path: str, local_path: str | os.PathLike[str]) -> None: @@ -151,23 +237,54 @@ def download(self, remote_path: str, local_path: str | os.PathLike[str]) -> None response.close() def delete(self, remote_path: str) -> None: - """DELETE the remote resource.""" - response = self._request("DELETE", remote_path) - response.close() + """DELETE the remote resource; a collection goes with everything in it.""" + self._change("DELETE", remote_path) def mkcol(self, remote_path: str) -> None: """MKCOL — create a collection (directory) at the remote path.""" response = self._request("MKCOL", remote_path) response.close() - def list_dir(self, remote_path: str) -> list[WebDAVEntry]: - """PROPFIND depth=1 against ``remote_path`` and return its entries.""" - headers = {"Depth": "1", "Content-Type": 'application/xml; charset="utf-8"'} + def copy(self, remote_path: str, destination: str, *, overwrite: bool = True) -> None: + """COPY ``remote_path`` to ``destination`` on the server.""" + self._relocate("COPY", remote_path, destination, overwrite) + + def move(self, remote_path: str, destination: str, *, overwrite: bool = True) -> None: + """MOVE ``remote_path`` to ``destination`` on the server.""" + self._relocate("MOVE", remote_path, destination, overwrite) + + def _relocate(self, method: str, remote_path: str, destination: str, overwrite: bool) -> None: + target = self._url_for(destination) + if not target.startswith(f"{self._base_url}/"): + validate_http_url(target, allow_private=self._allow_private_hosts) + headers = {"Destination": target, "Overwrite": "T" if overwrite else "F"} + self._change(method, remote_path, headers=headers) + + def stat(self, remote_path: str) -> WebDAVEntry: + """PROPFIND depth=0 against ``remote_path`` and return its own entry.""" + entries = self._propfind(remote_path, _DEPTH_SELF) + if not entries: + raise WebDAVException(f"PROPFIND {self._url_for(remote_path)} described no resource") + return entries[0] + + def list_dir(self, remote_path: str, *, include_self: bool = True) -> list[WebDAVEntry]: + """PROPFIND depth=1 against ``remote_path`` and return its entries. + + A server lists the collection itself next to its members; + ``include_self=False`` leaves that entry out. + """ + entries = self._propfind(remote_path, _DEPTH_MEMBERS) + if include_self: + return entries + own_path = _decoded_path(self._url_for(remote_path)) + return [entry for entry in entries if _decoded_path(entry.href) != own_path] + + def _propfind(self, remote_path: str, depth: str) -> list[WebDAVEntry]: response = self._request( "PROPFIND", remote_path, data=_PROPFIND_BODY, - headers=headers, + headers={"Depth": depth, "Content-Type": _PROPFIND_CONTENT_TYPE}, ) try: payload = response.text @@ -176,6 +293,17 @@ def list_dir(self, remote_path: str) -> list[WebDAVEntry]: return _parse_propfind(payload) +def _decoded_path(href: str) -> str: + """Return the decoded path of an href or a URL, without a trailing slash.""" + return unquote(urlsplit(href).path).rstrip("/") + + +def _text(element: Any) -> str | None: + """Return the stripped text of an XML element, or ``None`` when it has none.""" + text = element.text.strip() if element is not None and element.text else "" + return text or None + + def _parse_propfind(xml_text: str) -> list[WebDAVEntry]: try: root = defused_fromstring(xml_text) @@ -183,19 +311,21 @@ def _parse_propfind(xml_text: str) -> list[WebDAVEntry]: raise WebDAVException(f"malformed PROPFIND response: {error}") from error entries: list[WebDAVEntry] = [] for response in root.findall(f"{_DAV_NS}response"): - href_elem = response.find(f"{_DAV_NS}href") - if href_elem is None or href_elem.text is None: + href = _text(response.find(f"{_DAV_NS}href")) + if href is None: continue - href = href_elem.text.strip() - is_dir = response.find(f".//{_DAV_NS}collection") is not None - size_elem = response.find(f".//{_DAV_NS}getcontentlength") - size = int(size_elem.text) if size_elem is not None and size_elem.text else None - modified_elem = response.find(f".//{_DAV_NS}getlastmodified") - modified = ( - modified_elem.text.strip() if modified_elem is not None and modified_elem.text else None - ) - name = unquote(urlparse(href).path.rstrip("/").rsplit("/", 1)[-1]) + size = _text(response.find(f".//{_DAV_NS}getcontentlength")) + # urlsplit, not urlparse: a ";" in the last segment belongs to the name. + name = unquote(urlsplit(href).path.rstrip("/").rsplit("/", 1)[-1]) entries.append( - WebDAVEntry(href=href, name=name, is_dir=is_dir, size=size, last_modified=modified) + WebDAVEntry( + href=href, + name=name, + is_dir=response.find(f".//{_DAV_NS}collection") is not None, + size=int(size) if size is not None and size.isdigit() else None, + last_modified=_text(response.find(f".//{_DAV_NS}getlastmodified")), + etag=_text(response.find(f".//{_DAV_NS}getetag")), + content_type=_text(response.find(f".//{_DAV_NS}getcontenttype")), + ) ) return entries diff --git a/automation_file/storage/__init__.py b/automation_file/storage/__init__.py index 3faf32f..feae433 100644 --- a/automation_file/storage/__init__.py +++ b/automation_file/storage/__init__.py @@ -1,9 +1,13 @@ """Universal storage layer: one contract, one URI syntax, any backend. * :class:`File` and :class:`Storage` are the application API. -* :class:`StorageBackend` is the contract a backend implements. The built-in ones - are :class:`LocalStorage`, :class:`MemoryStorage`, :class:`S3Storage` and - :class:`AzureStorage`; :class:`ObjectStorage` is the shared base of the last two. +* :class:`StorageBackend` is the contract a backend implements. Built in: + :class:`LocalStorage`, :class:`MemoryStorage`, :class:`S3Storage`, + :class:`AzureStorage`, :class:`GoogleDriveStorage`, :class:`DropboxStorage`, + :class:`OneDriveStorage`, :class:`SFTPStorage`, :class:`FTPStorage`, + :class:`WebDAVStorage`, :class:`SMBStorage` and :class:`FsspecStorage`. + :class:`ObjectStorage` and :class:`SessionStorage` are the shared bases of the + object stores and of the login-session backends. * :class:`StorageURI` / :func:`parse_storage_uri` define the address syntax, and :class:`StorageResolver` maps an address to a backend. * :func:`register_storage_ops` adds the ``FA_storage_*`` actions to a registry. @@ -14,7 +18,11 @@ from automation_file.storage.actions import register_storage_ops from automation_file.storage.azure_storage import AzureStorage from automation_file.storage.backend import StorageBackend +from automation_file.storage.dropbox_storage import DropboxStorage from automation_file.storage.file import File +from automation_file.storage.fsspec_storage import FsspecStorage +from automation_file.storage.ftp_storage import FTPStorage +from automation_file.storage.gdrive_storage import GoogleDriveStorage from automation_file.storage.local_storage import LocalStorage from automation_file.storage.memory_storage import ( MemoryStorage, @@ -22,6 +30,7 @@ memory_store, ) from automation_file.storage.object_storage import ObjectStorage +from automation_file.storage.onedrive_storage import OneDriveStorage from automation_file.storage.resolver import ( BackendFactory, StorageResolver, @@ -29,6 +38,9 @@ register_default_schemes, ) from automation_file.storage.s3_storage import S3Storage +from automation_file.storage.session_storage import SessionStorage +from automation_file.storage.sftp_storage import SFTPStorage +from automation_file.storage.smb_storage import SMBStorage from automation_file.storage.storage import Storage from automation_file.storage.tree import TreeResult, copy_tree, sync_tree from automation_file.storage.types import Checksum, FileInfo, StorageCapabilities @@ -39,17 +51,26 @@ normalize_path, parse_storage_uri, ) +from automation_file.storage.webdav_storage import WebDAVStorage __all__ = [ "AzureStorage", "BackendFactory", "Checksum", + "DropboxStorage", + "FTPStorage", "File", "FileInfo", + "FsspecStorage", + "GoogleDriveStorage", "LocalStorage", "MemoryStorage", "ObjectStorage", + "OneDriveStorage", "S3Storage", + "SFTPStorage", + "SMBStorage", + "SessionStorage", "Storage", "StorageBackend", "StorageCapabilities", @@ -57,6 +78,7 @@ "StorageURI", "TreeResult", "URILike", + "WebDAVStorage", "clear_memory_stores", "copy_tree", "default_resolver", diff --git a/automation_file/storage/azure_storage.py b/automation_file/storage/azure_storage.py index e372b57..bdc3952 100644 --- a/automation_file/storage/azure_storage.py +++ b/automation_file/storage/azure_storage.py @@ -13,7 +13,7 @@ from __future__ import annotations import contextlib -from collections.abc import Iterable, Iterator +from collections.abc import Hashable, Iterable, Iterator from pathlib import Path from typing import Any @@ -119,6 +119,9 @@ def _service(self) -> Any: "or pass service= to AzureStorage" ) from error + def _store_identity(self) -> Hashable: + return (AZURE_SCHEME, id(self._service), self._container) + def uri_for(self, path: str = "") -> str: key = self._key(self._normalize(path)) return str(StorageURI(AZURE_SCHEME, self._container, key)) diff --git a/automation_file/storage/backend.py b/automation_file/storage/backend.py index b3241a0..e13dca7 100644 --- a/automation_file/storage/backend.py +++ b/automation_file/storage/backend.py @@ -25,7 +25,7 @@ import time import uuid from abc import ABC, abstractmethod -from collections.abc import Iterable, Iterator +from collections.abc import Hashable, Iterable, Iterator from pathlib import Path, PurePosixPath from types import TracebackType from typing import BinaryIO, TypeVar @@ -214,6 +214,16 @@ def _delete_directory(self, path: str, recursive: bool) -> None: if self.capabilities.directories: self._rmdir(path) + def _identity(self, path: str) -> Hashable: + """Return a key that two paths share exactly when they are the same stored file. + + The key is compared across backend instances. Two views of one store (two + roots, two prefixes, a link) must give the same key for the same file, or a + move between them would overwrite the file with itself and then delete it. + The default only recognises this instance's own paths. + """ + return (id(self), path) + def _normalize(self, path: str) -> str: """Normalise a caller-supplied path. Backends with extra separators extend it.""" return normalize_path(path) @@ -487,7 +497,7 @@ def _transfer_paths( if info.is_dir: raise not_a_file_error(source.uri_for(info.path)) target = self._normalize(path) - if source == self and info.path == target: + if source._identity(info.path) == self._identity(target): raise StorageException(f"{self.uri_for(target)}: source and target are the same file") target = self._writable_file(target, overwrite) self._make_parents(target) diff --git a/automation_file/storage/dropbox_storage.py b/automation_file/storage/dropbox_storage.py new file mode 100644 index 0000000..467df0b --- /dev/null +++ b/automation_file/storage/dropbox_storage.py @@ -0,0 +1,342 @@ +"""Dropbox backend: ``dropbox:///``. + +``DropboxStorage()`` serves the Dropbox of the shared +:data:`~automation_file.remote.dropbox_api.client.dropbox_instance`, which the +caller initialises as before (``dropbox_instance.later_init(token)`` or +``FA_dropbox_later_init``). Pass ``client=`` to use another ``dropbox.Dropbox`` +client and ``root=`` to confine the backend to one folder. + +Folders are real directories. ``stat`` reports the size, the server's +modification time, the revision as ``version`` and Dropbox's content hash as +``etag``. A file larger than :data:`UPLOAD_SESSION_THRESHOLD` goes up through an +upload session, :data:`UPLOAD_CHUNK_SIZE` bytes at a time, so it is never held in +memory as a whole. + +Dropbox never replaces a file on copy or move: when the target exists it is +deleted first, then the copy or move runs. +""" + +from __future__ import annotations + +import contextlib +from collections.abc import Callable, Hashable, Iterable, Iterator +from datetime import datetime, timezone +from pathlib import Path +from typing import Any, BinaryIO + +from automation_file.exceptions import ( + StorageAlreadyExistsException, + StorageException, + StorageNotFoundException, + StoragePermissionException, + StorageTransientException, + StorageUnavailableException, + StorageURIException, +) +from automation_file.storage.backend import ( + StorageBackend, + join_path, + missing_error, + not_empty_error, +) +from automation_file.storage.types import FileInfo, StorageCapabilities +from automation_file.storage.uri import StorageURI, normalize_path + +DROPBOX_SCHEME = "dropbox" +#: The size of one piece of an upload session. Dropbox asks for a multiple of 4 MiB. +UPLOAD_CHUNK_SIZE = 8 * 1024 * 1024 +#: A file larger than this goes up through an upload session instead of one request. +UPLOAD_SESSION_THRESHOLD = UPLOAD_CHUNK_SIZE +_NOT_INSTALLED = "dropbox is not installed; the Dropbox backend needs it" +# The branches of a Dropbox error under which "not_found" means the path, not an upload session. +_LOOKUP_BRANCHES = frozenset({"path", "path_lookup", "from_lookup"}) +_MISSING_TAG = "not_found" +_CONFLICT_TAG = "conflict" +_FILE_CONFLICT = (_CONFLICT_TAG, "file") +_DENIED_TAGS = frozenset( + {"access_restricted", "no_permission", "no_write_permission", "restricted_content"} +) +_TRANSIENT_TAGS = frozenset({"internal_error", "too_many_write_operations"}) +_DENIED_STATUS = frozenset({401, 403}) +_TRANSIENT_STATUS = frozenset({408, 429}) +_SERVER_ERROR = 500 + + +def _tags(error: Any) -> tuple[str, ...]: + """Return the tag of a Dropbox error union and of every union nested in it. + + ``GetMetadataError('path', LookupError('not_found'))`` gives ``("path", "not_found")``. + """ + tags: list[str] = [] + value = error + while value is not None: + tag = getattr(value, "_tag", None) + if tag is None: + # UploadWriteFailed is a struct that carries its WriteError as ``reason``. + value = getattr(value, "reason", None) + else: + tags.append(str(tag)) + value = getattr(value, "_value", None) + return tuple(tags) + + +def _api_error(tags: tuple[str, ...], location: str) -> StorageException: + """Translate the tags of a route error (a 409 answer) into a storage exception.""" + reason = "/".join(tags) or "unknown" + leaf = tags[-1] if tags else "" + if leaf == _MISSING_TAG and len(tags) > 1 and tags[-2] in _LOOKUP_BRANCHES: + return missing_error(location) + if leaf in _DENIED_TAGS: + return StoragePermissionException(f"access to {location} was denied ({reason})") + if leaf in _TRANSIENT_TAGS: + return StorageTransientException(f"{location}: Dropbox answered {reason}") + if _CONFLICT_TAG in tags: + return StorageAlreadyExistsException( + f"{location}: something is already at that path ({reason})" + ) + return StorageException(f"{location}: Dropbox error {reason}") + + +def _http_error(error: Any, location: str) -> StorageException: + """Translate an error of the HTTP layer: bad credentials, throttling, a server fault.""" + status = getattr(error, "status_code", None) + tags = _tags(getattr(error, "error", None)) + if status in _DENIED_STATUS or (tags and tags[-1] in _DENIED_TAGS): + return StoragePermissionException(f"access to {location} was denied ({status})") + if status in _TRANSIENT_STATUS or (isinstance(status, int) and status >= _SERVER_ERROR): + return StorageTransientException(f"{location}: Dropbox answered {status}") + return StorageException(f"{location}: Dropbox answered {status}") + + +@contextlib.contextmanager +def _dropbox_errors(location: str) -> Iterator[None]: + """Turn Dropbox SDK errors and dropped connections into the storage layer's exceptions.""" + try: + from dropbox import exceptions as dropbox_errors + from requests import exceptions as request_errors + except ImportError as error: + raise StorageUnavailableException(_NOT_INSTALLED) from error + try: + yield + except dropbox_errors.ApiError as error: + raise _api_error(_tags(error.error), location) from error + except dropbox_errors.HttpError as error: + raise _http_error(error, location) from error + except dropbox_errors.DropboxException as error: + raise StorageException(f"{location}: {type(error).__name__}") from error + except ( + request_errors.ConnectionError, + request_errors.Timeout, + request_errors.ChunkedEncodingError, + ) as error: + raise StorageTransientException(f"{location}: {type(error).__name__}") from error + + +def _dropbox_files() -> Any: + """Return the ``dropbox.files`` module, which holds the SDK's data types.""" + try: + from dropbox import files + except ImportError as error: + raise StorageUnavailableException(_NOT_INSTALLED) from error + return files + + +def _utc(moment: Any) -> datetime | None: + """Return a Dropbox timestamp, which the SDK hands over naive and in UTC, as aware UTC.""" + if not isinstance(moment, datetime): + return None + if moment.tzinfo is None: + return moment.replace(tzinfo=timezone.utc) + return moment.astimezone(timezone.utc) + + +def _file_info(path: str, metadata: Any, files: Any) -> FileInfo | None: + """Describe a metadata entry; ``None`` for one that is neither a file nor a folder.""" + if isinstance(metadata, files.FolderMetadata): + return FileInfo(path=path, is_dir=True) + if isinstance(metadata, files.FileMetadata): + return FileInfo( + path=path, + size=int(metadata.size), + modified_at=_utc(metadata.server_modified), + etag=metadata.content_hash, + version=metadata.rev, + ) + return None + + +def _upload_in_session(client: Any, files: Any, handle: BinaryIO, remote: str) -> None: + """Send an open file to Dropbox one chunk at a time and commit it at ``remote``.""" + session = client.files_upload_session_start(handle.read(UPLOAD_CHUNK_SIZE)) + cursor = files.UploadSessionCursor(session_id=session.session_id, offset=handle.tell()) + pending = handle.read(UPLOAD_CHUNK_SIZE) + # One chunk is read ahead, so the last one is known when it is sent. + while following := handle.read(UPLOAD_CHUNK_SIZE): + client.files_upload_session_append_v2(pending, cursor) + cursor.offset += len(pending) + pending = following + commit = files.CommitInfo(path=remote, mode=files.WriteMode.overwrite) + client.files_upload_session_finish(pending, cursor, commit) + + +class DropboxStorage(StorageBackend): + """A Dropbox account, whole or confined to the folder ``root``.""" + + scheme = DROPBOX_SCHEME + capabilities = StorageCapabilities(directories=True, modified_at=True, etag=True, version=True) + + def __init__(self, client: Any = None, *, root: str = "") -> None: + self._explicit_client = client + self._root = normalize_path(root) + + @property + def root(self) -> str: + """The folder this backend is confined to (empty for the whole Dropbox).""" + return self._root + + @property + def _client(self) -> Any: + if self._explicit_client is not None: + return self._explicit_client + from automation_file.remote.dropbox_api.client import dropbox_instance + + try: + return dropbox_instance.require_client() + except RuntimeError as error: + raise StorageUnavailableException( + "the Dropbox client is not initialised; call dropbox_instance.later_init() " + "or pass client= to DropboxStorage" + ) from error + + def _identity(self, path: str) -> Hashable: + # Two roots of one account; Dropbox ignores case name a file by the same full path. + return (DROPBOX_SCHEME, id(self._client), self._remote(path).lower()) + + def uri_for(self, path: str = "") -> str: + return str(StorageURI(DROPBOX_SCHEME, "", self._below_root(self._normalize(path)))) + + def _below_root(self, path: str) -> str: + return join_path(self._root, path) if path else self._root + + def _remote(self, path: str) -> str: + """Return the Dropbox path of ``path``: empty for the account root, ``/a/b`` otherwise.""" + joined = self._below_root(path) + return f"/{joined}" if joined else "" + + def _stat(self, path: str) -> FileInfo | None: + remote = self._remote(path) + if not remote: + # The account root has no metadata of its own; it is always there. + return FileInfo(path=path, is_dir=True) + files = _dropbox_files() + try: + with _dropbox_errors(self.uri_for(path)): + metadata = self._client.files_get_metadata(remote) + except StorageNotFoundException: + return None + return _file_info(path, metadata, files) + + def _list_dir(self, path: str) -> Iterable[FileInfo]: + files = _dropbox_files() + with _dropbox_errors(self.uri_for(path)): + page = self._client.files_list_folder(self._remote(path)) + entries = list(page.entries) + while page.has_more: + page = self._client.files_list_folder_continue(page.cursor) + entries.extend(page.entries) + described = (_file_info(join_path(path, entry.name), entry, files) for entry in entries) + return [info for info in described if info is not None] + + def _upload(self, source: Path, path: str) -> None: + files = _dropbox_files() + remote = self._remote(path) + with _dropbox_errors(self.uri_for(path)), open(source, "rb") as handle: + if source.stat().st_size > UPLOAD_SESSION_THRESHOLD: + _upload_in_session(self._client, files, handle, remote) + else: + self._client.files_upload(handle.read(), remote, mode=files.WriteMode.overwrite) + + def _download(self, path: str, target: Path) -> None: + with _dropbox_errors(self.uri_for(path)): + self._client.files_download_to_file(str(target), self._remote(path)) + + def _delete_file(self, path: str) -> None: + with _dropbox_errors(self.uri_for(path)): + self._client.files_delete_v2(self._remote(path)) + + def _mkdir(self, path: str) -> None: + try: + with _dropbox_errors(self.uri_for(path)): + self._client.files_create_folder_v2(self._remote(path)) + except StorageAlreadyExistsException: + existing = self._stat(path) + if existing is None or not existing.is_dir: + raise + + def _rmdir(self, path: str) -> None: + # One call deletes a file or a folder, and a folder goes with everything in it. + self._delete_file(path) + + def _delete_directory(self, path: str, recursive: bool) -> None: + if not recursive and any(True for _ in self._list_dir(path)): + raise not_empty_error(self.uri_for(path)) + self._rmdir(path) + + def _copy_from(self, source: StorageBackend, source_path: str, path: str) -> bool: + if not isinstance(source, DropboxStorage) or source._client is not self._client: + return False + self._relocate(self._client.files_copy_v2, source._remote(source_path), path) + return True + + def _move_from(self, source: StorageBackend, source_path: str, path: str) -> bool: + if not isinstance(source, DropboxStorage) or source._client is not self._client: + return False + self._relocate(self._client.files_move_v2, source._remote(source_path), path) + return True + + def _relocate(self, relocate: Callable[[str, str], Any], origin: str, path: str) -> None: + """Run a Dropbox copy or move to ``path``, replacing a file that is in the way.""" + location = self.uri_for(path) + target = self._remote(path) + try: + with _dropbox_errors(location): + relocate(origin, target) + return + except StorageAlreadyExistsException as error: + # Only a file is ever removed to make room: deleting a folder takes its content. + if _tags(getattr(error.__cause__, "error", None))[-2:] != _FILE_CONFLICT: + raise + with _dropbox_errors(location): + if self._entry_id(origin) == self._entry_id(target): + # Dropbox compares names without case, so another spelling is the same file. + raise StorageException(f"{location}: source and target are the same file") + self._client.files_delete_v2(target) + relocate(origin, target) + + def _entry_id(self, remote: str) -> Any: + """Return the identifier of the entry at ``remote``, the same for every spelling of it.""" + return getattr(self._client.files_get_metadata(remote), "id", None) + + def __eq__(self, other: object) -> bool: + return ( + isinstance(other, DropboxStorage) + and other._root == self._root + and other._explicit_client is self._explicit_client + ) + + def __hash__(self) -> int: + return hash((DROPBOX_SCHEME, self._root, id(self._explicit_client))) + + def __repr__(self) -> str: + return f"DropboxStorage(root={self._root!r})" + + +def dropbox_factory(uri: StorageURI) -> tuple[StorageBackend, str]: + """Serve ``dropbox:///`` from the shared Dropbox client.""" + if uri.authority: + correct = StorageURI(DROPBOX_SCHEME, "", f"{uri.authority}/{uri.path}") + raise StorageURIException( + f"{str(uri)!r} names the host {uri.authority!r}; a Dropbox path follows an empty " + f"authority, as in {str(correct)!r}" + ) + return DropboxStorage(), uri.path diff --git a/automation_file/storage/fsspec_storage.py b/automation_file/storage/fsspec_storage.py new file mode 100644 index 0000000..05eab0b --- /dev/null +++ b/automation_file/storage/fsspec_storage.py @@ -0,0 +1,331 @@ +"""fsspec backend: any `fsspec `_ filesystem as a storage. + +fsspec has a filesystem for most services (Google Cloud Storage, HDFS, FTP, +archives, ...). ``FsspecStorage`` puts one of them behind the storage contract. +The filesystem object carries its own connection and credentials, so there is no +URI factory: a backend is mounted where its files should appear, under any scheme. + +.. code-block:: python + + Storage.mount("gcs://reports", FsspecStorage.from_url("gcs://reports", directories=False)) + File("gcs://reports/2026/q1.csv").read() + +``directories`` says whether the filesystem keeps a directory that has no files in +it. Pass ``False`` for an object store, where a directory is only a key prefix. + +Paths are passed to the filesystem literally. ``copy()`` and ``mv()`` of fsspec +expand glob patterns, so a copy uses ``cp_file`` and a move uses ``mv`` only for +paths without ``*``, ``?`` or ``[``; other moves are a copy followed by a delete. +""" + +from __future__ import annotations + +import contextlib +from collections.abc import Hashable, Iterable, Iterator, Mapping +from dataclasses import replace +from datetime import datetime, timedelta, timezone +from pathlib import Path +from typing import Any + +from automation_file.exceptions import ( + StorageException, + StorageNotFoundException, + StoragePermissionException, + StorageTransientException, + StorageUnavailableException, + StorageUnsupportedException, + StorageURIException, +) +from automation_file.storage.backend import StorageBackend, join_path, missing_error +from automation_file.storage.types import FileInfo, StorageCapabilities +from automation_file.storage.uri import canonical_scheme, normalize_path + +FSSPEC_SCHEME = "fsspec" +_NOT_INSTALLED = "fsspec is not installed; the fsspec backend needs it" +_URI_SEPARATOR = "://" +_DIRECTORY = "directory" +# The keys under which filesystems report a modification time in info() and ls(). +_MODIFIED_KEYS = ("mtime", "LastModified", "last_modified", "modified", "updated") +_GLOB_CHARACTERS = frozenset("*?[") +_EPOCH = datetime(1970, 1, 1, tzinfo=timezone.utc) + + +@contextlib.contextmanager +def _fsspec_errors(location: str) -> Iterator[None]: + """Turn what an fsspec filesystem raises into the storage layer's exceptions.""" + try: + yield + except StorageException: + raise + except ImportError as error: + raise StorageUnavailableException(f"{location}: {error}") from error + except (FileNotFoundError, NotADirectoryError) as error: + raise missing_error(location) from error + except PermissionError as error: + raise StoragePermissionException(f"access to {location} was denied") from error + except (ConnectionError, TimeoutError) as error: + raise StorageTransientException(f"{location}: {type(error).__name__}") from error + except NotImplementedError as error: + raise StorageUnsupportedException( + f"{location}: the filesystem does not implement this operation" + ) from error + except Exception as error: + # A filesystem raises whatever the client library behind it raises. + raise StorageException(f"{location}: {type(error).__name__}") from error + + +def _utc(value: Any) -> datetime | None: + """Read a modification time as filesystems report one: datetime, epoch seconds, ISO 8601.""" + if isinstance(value, str): + text = f"{value[:-1]}+00:00" if value.endswith("Z") else value + try: + value = datetime.fromisoformat(text) + except ValueError: + return None + if isinstance(value, datetime): + if value.tzinfo is None: + return value.replace(tzinfo=timezone.utc) + return value.astimezone(timezone.utc) + if isinstance(value, (int, float)) and not isinstance(value, bool): + try: + return _EPOCH + timedelta(seconds=value) + except (OverflowError, ValueError): + return None + return None + + +def _listed_time(details: Mapping[str, Any]) -> datetime | None: + """Return the modification time an ``info()`` or ``ls()`` entry carries, if it has one.""" + for key in _MODIFIED_KEYS: + moment = _utc(details.get(key)) + if moment is not None: + return moment + return None + + +def _implements_modified(filesystem: Any) -> bool: + """Say whether the filesystem has a ``modified()`` of its own; the base class has none.""" + try: + from fsspec import AbstractFileSystem + except ImportError as error: + raise StorageUnavailableException(_NOT_INSTALLED) from error + modified = getattr(filesystem, "modified", None) + inherited = getattr(modified, "__func__", None) is AbstractFileSystem.modified + return callable(modified) and not inherited + + +def _spelled(filesystem: Any, root: str) -> str: + """Return ``root`` the way the filesystem spells its own paths.""" + if not root: + return str(getattr(filesystem, "root_marker", "")) + strip = getattr(filesystem, "_strip_protocol", None) + return str(strip(root)) if callable(strip) else root.rstrip("/") + + +def _without_credentials(url: str) -> str: + """Return ``url`` without the user information it may carry, for messages.""" + scheme, separator, rest = url.partition(_URI_SEPARATOR) + if not separator: + return url + authority, slash, path = rest.partition("/") + return f"{scheme}{separator}{authority.rpartition('@')[2]}{slash}{path}" + + +def _scheme_of(url: str, filesystem: Any) -> str: + """Pick the scheme of a storage built from ``url``: as written, else the filesystem's.""" + written, separator, _ = url.partition(_URI_SEPARATOR) + protocol = getattr(filesystem, "protocol", ()) + candidates = [written] if separator else [] + candidates.extend([protocol] if isinstance(protocol, str) else protocol) + for candidate in candidates: + with contextlib.suppress(StorageURIException): + return canonical_scheme(candidate) + return FSSPEC_SCHEME + + +class FsspecStorage(StorageBackend): + """An fsspec filesystem, whole or confined to the directory ``root``. + + ``scheme`` and ``capabilities`` belong to the instance: they depend on the + filesystem it was given. + """ + + scheme = FSSPEC_SCHEME + + def __init__( + self, + filesystem: Any, + *, + root: str = "", + scheme: str = FSSPEC_SCHEME, + directories: bool = True, + ) -> None: + self._fs = filesystem + self._root = _spelled(filesystem, root) + self._prefix = f"{self._root.rstrip('/')}/" if self._root else "" + self.scheme = canonical_scheme(scheme) + self.capabilities = StorageCapabilities( + directories=directories, modified_at=_implements_modified(filesystem) + ) + + @classmethod + def from_url( + cls, url: str, *, directories: bool = True, **storage_options: Any + ) -> FsspecStorage: + """Build the storage of an fsspec URL; the path of the URL becomes the root. + + ``storage_options`` go to the constructor of the filesystem. + """ + try: + from fsspec.core import url_to_fs + except ImportError as error: + raise StorageUnavailableException(_NOT_INSTALLED) from error + location = _without_credentials(url) + with _fsspec_errors(location): + try: + filesystem, root = url_to_fs(url, **storage_options) + except ValueError as error: + raise StorageURIException(f"fsspec cannot open {location!r}: {error}") from error + return cls( + filesystem, root=root, scheme=_scheme_of(url, filesystem), directories=directories + ) + + @property + def filesystem(self) -> Any: + """The fsspec filesystem this backend talks to.""" + return self._fs + + @property + def root(self) -> str: + """The directory this backend is confined to, as the filesystem spells it.""" + return self._root + + def _identity(self, path: str) -> Hashable: + # Two roots of one filesystem name a file by the same full path. + return (id(self._fs), self._full(path)) + + def uri_for(self, path: str = "") -> str: + return f"{self.scheme}{_URI_SEPARATOR}{self._full(self._normalize(path))}" + + def _normalize(self, path: str) -> str: + clean = normalize_path(path) + if ".." in clean.replace("\\", "/").split("/"): + # Some filesystems (local on Windows, SMB) read a backslash as a separator. + raise StorageURIException(f"storage paths cannot contain '..' segments: {path!r}") + return clean + + def _full(self, path: str) -> str: + """Return the path the filesystem knows ``path`` by.""" + return f"{self._prefix}{path}" if path else self._root + + def _stat(self, path: str) -> FileInfo | None: + if not path and not self.capabilities.directories: + # Without real directories the root is a key prefix, and a prefix is always there. + return FileInfo(path=path, is_dir=True) + full = self._full(path) + try: + with _fsspec_errors(self.uri_for(path)): + info = self._describe(path, self._fs.info(full)) + if info.is_dir or info.modified_at is not None: + return info + return replace(info, modified_at=self._asked_time(full)) + except StorageNotFoundException: + return None + + def _describe(self, path: str, details: Mapping[str, Any]) -> FileInfo: + modified = _listed_time(details) + if details.get("type") == _DIRECTORY: + return FileInfo(path=path, is_dir=True, modified_at=modified) + size = details.get("size") + return FileInfo( + path=path, size=int(size) if isinstance(size, int) else None, modified_at=modified + ) + + def _asked_time(self, full: str) -> datetime | None: + """Ask ``modified()`` for a time that ``info()`` did not carry.""" + if not self.capabilities.modified_at: + return None + try: + return _utc(self._fs.modified(full)) + except NotImplementedError: + return None + + def _list_dir(self, path: str) -> Iterable[FileInfo]: + full = self._full(path) + try: + with _fsspec_errors(self.uri_for(path)): + listing = self._fs.ls(full, detail=True) + except StorageNotFoundException: + if path or self.capabilities.directories: + raise + # A root that is only a key prefix has no keys below it yet. + return [] + found: list[FileInfo] = [] + for details in listing: + name = str(details.get("name", "")).rstrip("/") + # A placeholder object some stores keep for the directory itself is not a child. + if name and name != full.rstrip("/"): + child = join_path(path, name.rsplit("/", 1)[-1]) + found.append(self._describe(child, details)) + return found + + def _upload(self, source: Path, path: str) -> None: + with _fsspec_errors(self.uri_for(path)): + self._fs.put_file(str(source), self._full(path)) + + def _download(self, path: str, target: Path) -> None: + with _fsspec_errors(self.uri_for(path)): + self._fs.get_file(self._full(path), str(target)) + + def _read_bytes(self, path: str) -> bytes: + with _fsspec_errors(self.uri_for(path)): + return bytes(self._fs.cat_file(self._full(path))) + + def _delete_file(self, path: str) -> None: + with _fsspec_errors(self.uri_for(path)): + self._fs.rm_file(self._full(path)) + + def _mkdir(self, path: str) -> None: + with _fsspec_errors(self.uri_for(path)): + self._fs.makedirs(self._full(path), exist_ok=True) + + def _rmdir(self, path: str) -> None: + # A directory that a filesystem only implies is gone once its last file is. + with contextlib.suppress(StorageNotFoundException), _fsspec_errors(self.uri_for(path)): + self._fs.rmdir(self._full(path)) + + def _copy_from(self, source: StorageBackend, source_path: str, path: str) -> bool: + if not isinstance(source, FsspecStorage) or source._fs is not self._fs: + return False + try: + with _fsspec_errors(self.uri_for(path)): + # cp_file() takes both paths literally; copy() expands glob patterns. + self._fs.cp_file(source._full(source_path), self._full(path)) + except StorageUnsupportedException: + return False + return True + + def _move_from(self, source: StorageBackend, source_path: str, path: str) -> bool: + if not isinstance(source, FsspecStorage) or source._fs is not self._fs: + return False + origin, target = source._full(source_path), self._full(path) + if _GLOB_CHARACTERS.intersection(origin + target): + # mv() would read the path as a pattern; copy then delete names it literally. + return False + try: + with _fsspec_errors(self.uri_for(path)): + self._fs.mv(origin, target) + except StorageUnsupportedException: + return False + return True + + def __eq__(self, other: object) -> bool: + return ( + isinstance(other, FsspecStorage) and other._fs is self._fs and other._root == self._root + ) + + def __hash__(self) -> int: + return hash((id(self._fs), self._root)) + + def __repr__(self) -> str: + return f"FsspecStorage({type(self._fs).__name__}, root={self._root!r})" diff --git a/automation_file/storage/ftp_storage.py b/automation_file/storage/ftp_storage.py new file mode 100644 index 0000000..7ef2dd4 --- /dev/null +++ b/automation_file/storage/ftp_storage.py @@ -0,0 +1,380 @@ +"""FTP and FTPS backend: ``ftp://[:]/`` and ``ftps://...``. + +``FTPStorage()`` serves the files the shared +:data:`~automation_file.remote.ftp.client.ftp_instance` session can reach. The +caller opens that session as before (``ftp_instance.later_init(...)`` or +``FA_ftp_later_init``, with ``tls=True`` for FTPS). Pass another connected +``FTPClient`` to work with a second host, and ``root=`` to join every path to one +remote directory. + +A server that offers ``MLST`` (RFC 3659) is asked for the ``type``, ``size`` and +``modify`` facts of an entry. Any other server is probed: a directory is what +``CWD`` enters, a file is what ``SIZE`` and ``MDTM`` answer for, and a listing is +``NLST`` followed by one probe for each name. The working directory is put back +after a probe, so callers that use relative paths on the same session keep theirs. + +FTP does not say reliably which names are symbolic links. Deleting never follows +one all the same: ``DELE`` removes a link and refuses a directory, so it is tried +before anything is descended into. A listing shows a link as the server shows it. +""" + +from __future__ import annotations + +import contextlib +import ftplib # nosec B402 - error types and reply parsing; FTPClient opens the session +import posixpath +import weakref +from collections.abc import Iterable, Iterator +from contextlib import AbstractContextManager +from datetime import datetime, timezone +from pathlib import Path +from typing import TYPE_CHECKING, Any + +from automation_file.exceptions import ( + StorageException, + StorageNotFoundException, + StoragePermissionException, + StorageTransientException, + StorageUnavailableException, + StorageURIException, +) +from automation_file.storage.backend import ( + StorageBackend, + join_path, + missing_error, + not_empty_error, +) +from automation_file.storage.session_storage import SessionStorage, require_session_host +from automation_file.storage.types import FileInfo +from automation_file.storage.uri import StorageURI + +if TYPE_CHECKING: + from automation_file.remote.ftp.client import FTPClient + +FTP_SCHEME = "ftp" +FTPS_SCHEME = "ftps" +_FTP_PORT = 21 +# "Requested action not taken, file unavailable": in a lookup, there is no such entry. +_UNAVAILABLE = "550" +_BINARY = "TYPE I" +_WANTED_FACTS = "OPTS MLST type;size;modify;" +_DIRECTORY_TYPES = frozenset({"dir", "cdir", "pdir"}) +_SELF_AND_PARENT = frozenset({"cdir", "pdir"}) +_NOT_ENTRIES = frozenset({"", ".", ".."}) +_TIMESTAMP = "%Y%m%d%H%M%S" + +# Whether the server behind a session offers MLST, asked once for each session. +_mlst_support: weakref.WeakKeyDictionary[Any, bool] = weakref.WeakKeyDictionary() + + +def _reply_code(error: ftplib.Error) -> str: + return str(error)[:3] + + +@contextlib.contextmanager +def _ftp_errors(location: str) -> Iterator[None]: + """Turn what ftplib raises into the storage layer's exceptions.""" + try: + yield + except ftplib.error_perm as error: + raise StoragePermissionException( + f"access to {location} was denied ({_reply_code(error)})" + ) from error + except ftplib.error_temp as error: + raise StorageTransientException(f"{location}: FTP answered {_reply_code(error)}") from error + # TimeoutError is what socket.timeout has been since Python 3.10. + except (EOFError, TimeoutError, ConnectionError) as error: + raise StorageTransientException(f"{location}: {type(error).__name__}") from error + # UnicodeError: a name or a reply that is not in the session's encoding. + except (ftplib.Error, OSError, UnicodeError) as error: + raise StorageException(f"{location}: {error}") from error + + +def _one_line(text: str) -> str: + """Return ``text``, or refuse it: a line break would end the FTP command it is sent in.""" + if "\r" in text or "\n" in text: + raise StorageURIException(f"FTP paths cannot contain a line break: {text!r}") + return text + + +def _negotiate_mlst(ftp: Any) -> bool: + try: + features = ftp.sendcmd("FEAT").splitlines()[1:-1] + if "MLST" not in {line.strip().partition(" ")[0].upper() for line in features}: + return False + ftp.sendcmd(_WANTED_FACTS) + except ftplib.error_perm: + # No FEAT, or the facts cannot be switched on: probe instead. + return False + return True + + +def _has_mlst(ftp: Any) -> bool: + known = _mlst_support.get(ftp) + if known is None: + known = _mlst_support[ftp] = _negotiate_mlst(ftp) + return known + + +def _timestamp(text: str | None) -> datetime | None: + """Parse the UTC time of a ``modify`` fact or an ``MDTM`` reply: ``YYYYMMDDHHMMSS[.sss]``.""" + whole, _, fraction = (text or "").strip().partition(".") + try: + moment = datetime.strptime(whole, _TIMESTAMP) + microsecond = int(fraction[:6].ljust(6, "0")) if fraction else 0 + except ValueError: + return None + return moment.replace(microsecond=microsecond, tzinfo=timezone.utc) + + +def _fact_info(path: str, facts: dict[str, str]) -> FileInfo: + is_dir = facts.get("type", "").lower() in _DIRECTORY_TYPES + size = facts.get("size", "") + return FileInfo( + path=path, + is_dir=is_dir, + size=int(size) if size.isascii() and size.isdigit() and not is_dir else None, + modified_at=_timestamp(facts.get("modify")), + ) + + +def _mlst(ftp: Any, path: str, remote: str) -> FileInfo: + """Describe ``remote`` from the one entry line of an ``MLST`` reply.""" + reply = ftp.sendcmd(f"MLST {remote}") + lines = reply.split("\n") + if len(lines) < 3 or not lines[1].startswith(" "): + raise ftplib.error_reply(reply) + facts: dict[str, str] = {} + for fact in filter(None, lines[1][1:].partition(" ")[0].split(";")): + name, _, value = fact.partition("=") + facts[name.lower()] = value + return _fact_info(path, facts) + + +def _mlsd(ftp: Any, path: str, remote: str) -> list[FileInfo]: + found: list[FileInfo] = [] + for listed, facts in ftp.mlsd(remote): + name = posixpath.basename(listed.rstrip("/")) + if name in _NOT_ENTRIES or facts.get("type", "").lower() in _SELF_AND_PARENT: + continue + found.append(_fact_info(join_path(path, name), facts)) + return found + + +def _begin_probing(ftp: Any) -> str: + """Return the working directory to go back to, and set the mode ``SIZE`` needs.""" + home = ftp.pwd() + ftp.voidcmd(_BINARY) + return home + + +def _enters(ftp: Any, remote: str, home: str) -> bool: + """Say whether ``CWD`` enters ``remote``, going back to ``home`` when it does.""" + try: + ftp.cwd(remote) + except ftplib.error_perm as error: + if _reply_code(error) != _UNAVAILABLE: + raise + return False + ftp.cwd(home) + return True + + +def _modified(ftp: Any, remote: str) -> datetime | None: + try: + return _timestamp(ftp.sendcmd(f"MDTM {remote}")[3:]) + except ftplib.error_perm: + return None + + +def _probe(ftp: Any, path: str, remote: str, home: str) -> FileInfo: + """Describe ``remote`` on a server without ``MLST``; 550 from ``SIZE`` means it is absent.""" + if _enters(ftp, remote, home): + return FileInfo(path=path, is_dir=True) + return FileInfo(path=path, size=ftp.size(remote), modified_at=_modified(ftp, remote)) + + +def _probe_listed(ftp: Any, path: str, remote: str, home: str) -> FileInfo: + """Like :func:`_probe` for a name ``NLST`` returned, which exists whatever ``SIZE`` says.""" + try: + return _probe(ftp, path, remote, home) + except ftplib.error_perm as error: + if _reply_code(error) != _UNAVAILABLE: + raise + return FileInfo(path=path) + + +def _nlst(ftp: Any, remote: str) -> list[str]: + """Return the names in the directory ``remote``, whether the server sends names or paths.""" + try: + lines = ftp.nlst(remote) + except ftplib.error_perm as error: + if _reply_code(error) != _UNAVAILABLE: + raise + # How some servers say that an existing directory is empty. + return [] + return sorted({posixpath.basename(line.rstrip("/")) for line in lines} - _NOT_ENTRIES) + + +def _names(ftp: Any, remote: str) -> list[str]: + """Return the names in the directory ``remote``, from ``MLSD`` where the server has it.""" + if _has_mlst(ftp): + return [info.path for info in _mlsd(ftp, "", remote)] + return _nlst(ftp, remote) + + +def _unlinked(ftp: Any, remote: str) -> bool: + """Try ``DELE``, which removes a file or a link and never a directory; say whether it did.""" + try: + ftp.delete(remote) + except ftplib.error_perm: + return False + return True + + +def _remove_tree(ftp: Any, remote: str) -> None: + """Remove the directory ``remote`` and what it holds. A link is removed, never followed. + + FTP does not say reliably which names are links, so every name gets ``DELE`` + first: what that refuses is a directory to descend into. + """ + directories = [remote] + pending = [remote] + while pending: + directory = pending.pop() + for name in _names(ftp, directory): + child = posixpath.join(directory, name) + if not _unlinked(ftp, child): + directories.append(child) + pending.append(child) + for directory in reversed(directories): + ftp.rmd(directory) + + +class FTPStorage(SessionStorage): + """The directory tree below ``root`` on the host of one FTP or FTPS session.""" + + scheme = FTP_SCHEME + default_port = _FTP_PORT + + def __init__(self, client: FTPClient | None = None, *, root: str = "/") -> None: + super().__init__(client, root=_one_line(root)) + + def _shared_client(self) -> Any: + from automation_file.remote.ftp.client import ftp_instance + + return ftp_instance + + def _open_session(self) -> Any: + from automation_file.remote.ftp.client import FTPException + + try: + return self.client.require_ftp() + except FTPException as error: + raise StorageUnavailableException( + "the FTP session is not open; call ftp_instance.later_init(...) first, " + "or give FTPStorage a connected FTPClient" + ) from error + + def _errors(self, session: Any, location: str) -> AbstractContextManager[None]: + return _ftp_errors(location) + + def _uri_scheme(self) -> str: + return FTPS_SCHEME if self.client.tls else FTP_SCHEME + + def _normalize(self, path: str) -> str: + return super()._normalize(_one_line(path)) + + @contextlib.contextmanager + def _lookup(self, path: str) -> Iterator[Any]: + """Like :meth:`_session`, for a lookup: there 550 means that nothing has that name.""" + with self._session(path) as ftp: + try: + yield ftp + except ftplib.error_perm as error: + if _reply_code(error) != _UNAVAILABLE: + raise + raise missing_error(self.uri_for(path)) from error + + def _stat(self, path: str) -> FileInfo | None: + remote = self._remote(path) + try: + with self._lookup(path) as ftp: + if _has_mlst(ftp): + return _mlst(ftp, path, remote) + return _probe(ftp, path, remote, _begin_probing(ftp)) + except StorageNotFoundException: + return None + + def _list_dir(self, path: str) -> Iterable[FileInfo]: + remote = self._remote(path) + with self._lookup(path) as ftp: + if _has_mlst(ftp): + return _mlsd(ftp, path, remote) + names = _nlst(ftp, remote) + # NLST switched the session to ASCII mode, so probing starts after it. + home = _begin_probing(ftp) + return [ + _probe_listed(ftp, join_path(path, name), posixpath.join(remote, name), home) + for name in names + ] + + def _store(self, session: Any, source: Path, remote: str) -> None: + with open(source, "rb") as handle: + session.storbinary(f"STOR {remote}", handle) + + def _rename(self, session: Any, origin: str, target: str) -> Exception | None: + try: + # A Unix server replaces a file at the target; others answer 5xx. + session.rename(origin, target) + except ftplib.error_perm as error: + return error + return None + + def _unlink(self, session: Any, remote: str) -> None: + session.delete(remote) + + def _download(self, path: str, target: Path) -> None: + with open(target, "wb") as handle, self._session(path) as ftp: + ftp.retrbinary(f"RETR {self._remote(path)}", handle.write) + + def _mkdir(self, path: str) -> None: + with self._session(path) as ftp: + try: + ftp.mkd(self._remote(path)) + except ftplib.error_perm: + # The refusal stands unless the directory is there after all. + existing = self._stat(path) + if existing is None or not existing.is_dir: + raise + + def _delete_directory(self, path: str, recursive: bool) -> None: + remote = self._remote(path) + with self._session(path) as ftp: + if _unlinked(ftp, remote): + # It was a link to a directory: the link is gone, the directory stays. + return + if recursive: + _remove_tree(ftp, remote) + elif _names(ftp, remote): + raise not_empty_error(self.uri_for(path)) + else: + ftp.rmd(remote) + + +def ftp_factory(uri: StorageURI) -> tuple[StorageBackend, str]: + """Serve ``ftp://`` and ``ftps://`` URIs through the shared FTP session. + + The URI names no host, or the host that session is connected to: see + :func:`~automation_file.storage.session_storage.require_session_host`. + ``ftps://`` is refused while the open session is not an FTPS one. + """ + from automation_file.remote.ftp.client import ftp_instance + + require_session_host(uri, ftp_instance, FTPStorage.__name__) + if uri.scheme == FTPS_SCHEME and ftp_instance.host and not ftp_instance.tls: + raise StorageURIException( + f"{str(uri)!r} asks for FTPS, but the open FTP session is not encrypted; " + "open it with ftp_instance.later_init(..., tls=True)" + ) + return FTPStorage(), uri.path diff --git a/automation_file/storage/gdrive_storage.py b/automation_file/storage/gdrive_storage.py new file mode 100644 index 0000000..93ffbd4 --- /dev/null +++ b/automation_file/storage/gdrive_storage.py @@ -0,0 +1,557 @@ +"""Google Drive backend: ``gdrive:///``. + +``GoogleDriveStorage()`` serves My Drive through the shared +:data:`~automation_file.remote.google_drive.client.driver_instance`, which the +caller initialises as before (``driver_instance.later_init(token_path, +credentials_path)`` or ``FA_drive_later_init``). ``root_id=`` makes another +folder, or a shared drive, the root; in a URI that ID is the authority +(``gdrive:///q1.csv``), and an empty authority or ``root`` is My Drive. + +Drive addresses entries by ID and lets several entries in one folder carry the +same name, so a path is resolved one segment at a time, on every call: + +* A name that two or more entries of a folder share cannot be addressed. The + call raises :class:`~automation_file.exceptions.StorageException` with the + count; it never picks one. A listing still shows each of them. +* A name that contains ``/`` cannot be addressed either and is left out of + listings. +* Entries in the trash do not exist as far as this backend is concerned. + +Folders are real directories. Writing to a path that holds a file uploads a new +revision of that file, so its ID, links and sharing stay; this is also what a +copy or a move onto an existing file does. A copy or a move to a new path +between two locations of one client is done by Drive itself. ``delete`` removes +permanently, without the trash, and a folder goes with everything in it. + +Google Docs, Sheets, Slides and the other ``application/vnd.google-apps.*`` +types have no binary content: they are listed with ``size=None`` and can be +moved, copied and deleted, but ``download``, ``read_bytes`` and ``checksum`` +raise :class:`~automation_file.exceptions.StorageUnsupportedException`. +""" + +from __future__ import annotations + +import contextlib +import json +import ssl +from collections.abc import Iterable, Iterator, Mapping +from pathlib import Path +from typing import TYPE_CHECKING, Any + +from automation_file.exceptions import ( + StorageException, + StorageNotFoundException, + StoragePathTypeException, + StoragePermissionException, + StorageTransientException, + StorageUnavailableException, + StorageUnsupportedException, +) +from automation_file.logging_config import file_automation_logger +from automation_file.storage.backend import ( + StorageBackend, + guess_content_type, + join_path, + missing_error, + not_a_file_error, + not_empty_error, + parent_of, +) +from automation_file.storage.timestamps import parse_rfc3339 +from automation_file.storage.types import FileInfo, StorageCapabilities +from automation_file.storage.uri import StorageURI + +if TYPE_CHECKING: + from automation_file.remote.google_drive.client import GoogleDriveClient + +GDRIVE_SCHEME = "gdrive" +MY_DRIVE = "root" +FOLDER_MIME_TYPE = "application/vnd.google-apps.folder" +_WORKSPACE_PREFIX = "application/vnd.google-apps." +_DEFAULT_MIME_TYPE = "application/octet-stream" +_ENTRY_FIELDS = ( + "id, name, mimeType, size, modifiedTime, md5Checksum, sha1Checksum, sha256Checksum, " + "version, parents" +) +_LIST_FIELDS = f"nextPageToken, files({_ENTRY_FIELDS})" +_ROOT_FIELDS = "id, name, mimeType, modifiedTime, trashed" +_ID = "id" +_NAME = "name" +_MIME_TYPE = "mimeType" +_PARENTS = "parents" +_PAGE_SIZE = 1000 +_DOWNLOAD_CHUNK_SIZE = 8 * 1024 * 1024 +_NOT_FOUND = 404 +_FORBIDDEN = 403 +_DENIED_STATUS = frozenset({401, _FORBIDDEN}) +_TOO_MANY_REQUESTS = 429 +_SERVER_ERROR = 500 +# The reasons googleapiclient itself retries a 403 for. +_RATE_LIMIT_REASONS = frozenset({"rateLimitExceeded", "userRateLimitExceeded"}) +# hashlib name -> the Drive field that carries that digest of a file's content. +_SERVER_DIGESTS = {"md5": "md5Checksum", "sha1": "sha1Checksum", "sha256": "sha256Checksum"} +_NOT_INSTALLED = "google-api-python-client is not installed; the Google Drive backend needs it" +_SAME_FILE = "source and target are the same file" + +_Entry = dict[str, Any] + + +def _error_reasons(error: Any) -> set[str]: + """Return the ``reason`` codes in the error body Drive sent (none when it is not JSON).""" + try: + entries = json.loads(error.content.decode("utf-8"))["error"]["errors"] + except (AttributeError, UnicodeDecodeError, ValueError, KeyError, TypeError): + return set() + if not isinstance(entries, list): + return set() + return { + str(entry["reason"]) for entry in entries if isinstance(entry, dict) and entry.get("reason") + } + + +def _translated(error: Any, location: str) -> StorageException: + """Turn a ``googleapiclient`` ``HttpError`` into the storage exception it stands for.""" + status = int(error.resp.status) + reasons = _error_reasons(error) + detail = " ".join([str(status), *sorted(reasons)]) + if status == _NOT_FOUND: + return missing_error(location) + throttled = status == _FORBIDDEN and bool(reasons & _RATE_LIMIT_REASONS) + if throttled or status == _TOO_MANY_REQUESTS or status >= _SERVER_ERROR: + return StorageTransientException(f"{location}: Google Drive answered {detail}") + if status in _DENIED_STATUS: + return StoragePermissionException(f"access to {location} was denied ({detail})") + return StorageException(f"{location}: Google Drive error {detail}") + + +@contextlib.contextmanager +def _drive_errors(location: str) -> Iterator[None]: + """Turn googleapiclient, google-auth and transport errors into the storage layer's exceptions. + + A message carries the status and Drive's reason codes, never the text of the + SDK error: that text quotes the request URL. The SDK error stays the cause. + """ + try: + import httplib2 + from google.auth import exceptions as auth_errors + from googleapiclient import errors as api_errors + except ImportError as error: + raise StorageUnavailableException(_NOT_INSTALLED) from error + try: + yield + except api_errors.HttpError as error: + raise _translated(error, location) from error + except auth_errors.RefreshError as error: + raise StoragePermissionException( + f"access to {location} was denied: the Google credentials could not be refreshed" + ) from error + except ssl.SSLCertVerificationError as error: + raise StorageException(f"{location}: {type(error).__name__}") from error + except ( + auth_errors.TransportError, + httplib2.ServerNotFoundError, + ssl.SSLError, + ConnectionError, + TimeoutError, + ) as error: + raise StorageTransientException(f"{location}: {type(error).__name__}") from error + except (api_errors.Error, auth_errors.GoogleAuthError, httplib2.HttpLib2Error) as error: + raise StorageException(f"{location}: {type(error).__name__}") from error + + +def _quoted(value: str) -> str: + """Return ``value`` as a string literal of the Drive query language.""" + escaped = value.replace("\\", "\\\\").replace("'", "\\'") + return f"'{escaped}'" + + +def _children_query(folder_id: str, name: str | None = None) -> str: + """Return the query for what a folder holds outside the trash, or for one name in it.""" + named = "" if name is None else f" and name = {_quoted(name)}" + return f"{_quoted(folder_id)} in parents{named} and trashed = false" + + +def _leaf(path: str) -> str: + return path.rpartition("/")[2] + + +def _is_folder(entry: Mapping[str, Any]) -> bool: + return entry.get(_MIME_TYPE) == FOLDER_MIME_TYPE + + +def _is_workspace_document(entry: Mapping[str, Any]) -> bool: + """Say whether ``entry`` is a Docs / Sheets / Slides style entry without binary content.""" + return str(entry.get(_MIME_TYPE) or "").startswith(_WORKSPACE_PREFIX) and not _is_folder(entry) + + +def _is_addressable(name: str) -> bool: + """Say whether a Drive name can be one segment of a storage path.""" + return bool(name) and name not in (".", "..") and "/" not in name and "\x00" not in name + + +def _file_info(path: str, entry: Mapping[str, Any]) -> FileInfo: + modified_at = parse_rfc3339(entry.get("modifiedTime")) + if _is_folder(entry): + return FileInfo(path=path, is_dir=True, modified_at=modified_at) + size = entry.get("size") + version = entry.get("version") + return FileInfo( + path=path, + # Drive reports a storage size for Workspace documents; it is not a content length. + size=None if size is None or _is_workspace_document(entry) else int(size), + modified_at=modified_at, + etag=entry.get("md5Checksum"), + version=None if version is None else str(version), + content_type=entry.get(_MIME_TYPE), + ) + + +class GoogleDriveStorage(StorageBackend): + """My Drive, or the tree below the folder (or shared drive) ``root_id``.""" + + scheme = GDRIVE_SCHEME + capabilities = StorageCapabilities( + directories=True, modified_at=True, etag=True, version=True, content_type=True + ) + + def __init__(self, client: GoogleDriveClient | None = None, *, root_id: str = MY_DRIVE) -> None: + self._explicit_client = client + self._root_id = root_id.strip() or MY_DRIVE + # The ID is the authority of this backend's URIs, so it has to be usable as one. + StorageURI(GDRIVE_SCHEME, self._root_id) + + @property + def root_id(self) -> str: + """The ID of the folder this backend treats as its root (``root`` is My Drive).""" + return self._root_id + + @property + def _service(self) -> Any: + client = self._explicit_client + if client is None: + from automation_file.remote.google_drive.client import driver_instance + + client = driver_instance + try: + return client.require_service() + except RuntimeError as error: + raise StorageUnavailableException( + "the Google Drive client is not initialised; call " + "driver_instance.later_init(token_path, credentials_path) " + "or pass client= to GoogleDriveStorage" + ) from error + + def uri_for(self, path: str = "") -> str: + authority = "" if self._root_id == MY_DRIVE else self._root_id + return str(StorageURI(GDRIVE_SCHEME, authority, self._normalize(path))) + + # ------------------------------------------------------------------ resolving paths + + def _matches(self, query: str, location: str, *, first_only: bool = False) -> list[_Entry]: + """Return the entries ``query`` selects, reading every page (or until one is found).""" + files = self._service.files() + found: list[_Entry] = [] + token: str | None = None + with _drive_errors(location): + while True: + page = files.list( + q=query, + fields=_LIST_FIELDS, + pageSize=1 if first_only else _PAGE_SIZE, + pageToken=token, + supportsAllDrives=True, + includeItemsFromAllDrives=True, + ).execute() + found.extend(page.get("files", [])) + token = page.get("nextPageToken") + if not token or (first_only and found): + return found + + def _child(self, folder_id: str, name: str, location: str) -> _Entry | None: + """Return the entry called ``name`` in the folder, or ``None``; refuse a shared name.""" + found = self._matches(_children_query(folder_id, name), location) + # Compared again here: the query's "=" is not relied on to tell upper from lower case. + matches = [entry for entry in found if entry.get(_NAME) == name] + if len(matches) > 1: + raise StorageException( + f"{location}: {len(matches)} entries share the name {name!r} in that Google Drive " + "folder, so the path does not say which one is meant; rename or remove the others" + ) + return matches[0] if matches else None + + def _root_entry(self) -> _Entry | None: + with _drive_errors(self.uri_for("")): + entry: _Entry = ( + self._service.files() + .get(fileId=self._root_id, fields=_ROOT_FIELDS, supportsAllDrives=True) + .execute() + ) + return None if entry.get("trashed") else entry + + def _descend(self, path: str) -> _Entry | None: + folder_id = self._root_id + entry: _Entry | None = None + walked = "" + for name in path.split("/"): + if entry is not None and not _is_folder(entry): + return None + walked = join_path(walked, name) + entry = self._child(folder_id, name, self.uri_for(walked)) + if entry is None: + return None + folder_id = str(entry[_ID]) + return entry + + def _entry(self, path: str) -> _Entry | None: + """Return the Drive resource at ``path``, or ``None`` when nothing is there.""" + try: + return self._descend(path) if path else self._root_entry() + except StorageNotFoundException: + # The root, or a folder on the way, is gone: nothing exists below it. + return None + + def _existing(self, path: str) -> _Entry: + entry = self._entry(path) + if entry is None: + raise missing_error(self.uri_for(path)) + return entry + + def _folder_id(self, path: str) -> str: + """Return the ID of the existing folder ``path``.""" + if not path: + return self._root_id + entry = self._existing(path) + if not _is_folder(entry): + raise StoragePathTypeException(f"{self.uri_for(path)} is not a directory") + return str(entry[_ID]) + + def _content_entry(self, path: str) -> _Entry: + """Return the entry of the file ``path``, which must have binary content.""" + location = self.uri_for(path) + entry = self._existing(path) + if _is_folder(entry): + raise not_a_file_error(location) + if _is_workspace_document(entry): + raise StorageUnsupportedException( + f"{location} is a Google Workspace document ({entry.get(_MIME_TYPE)}): it has no " + "binary content to download, read or hash" + ) + return entry + + def _scan(self, folder_id: str, path: str) -> list[tuple[str, _Entry]]: + """Return ``(path, entry)`` for every addressable child of the folder at ``path``.""" + location = self.uri_for(path) + found: list[tuple[str, _Entry]] = [] + for entry in self._matches(_children_query(folder_id), location): + name = str(entry.get(_NAME) or "") + if _is_addressable(name): + found.append((join_path(path, name), entry)) + else: + file_automation_logger.warning( + "GoogleDriveStorage: %s holds an entry named %r that no path can address; " + "it is left out of the listing", + location, + name, + ) + return found + + # ------------------------------------------------------------------ StorageBackend primitives + + def _stat(self, path: str) -> FileInfo | None: + entry = self._entry(path) + return None if entry is None else _file_info(path, entry) + + def _list_dir(self, path: str) -> Iterable[FileInfo]: + return [ + _file_info(child, entry) for child, entry in self._scan(self._folder_id(path), path) + ] + + def _walk(self, path: str) -> Iterable[FileInfo]: + # By ID, not by path: one listing per folder, and two folders of one name are both read. + found: list[FileInfo] = [] + pending = [(self._folder_id(path), path)] + while pending: + folder_id, base = pending.pop() + for child, entry in self._scan(folder_id, base): + found.append(_file_info(child, entry)) + if _is_folder(entry): + pending.append((str(entry[_ID]), child)) + return found + + def _upload(self, source: Path, path: str) -> None: + location = self.uri_for(path) + name = _leaf(path) + folder_id = self._folder_id(parent_of(path)) + existing = self._child(folder_id, name, location) + if existing is not None and _is_folder(existing): + raise not_a_file_error(location) + if existing is not None and _is_workspace_document(existing): + raise StorageUnsupportedException( + f"{location} is a Google Workspace document ({existing.get(_MIME_TYPE)}); " + "a file cannot replace it" + ) + mime_type = guess_content_type(name) or _DEFAULT_MIME_TYPE + files = self._service.files() + with _drive_errors(location): + from googleapiclient.http import MediaFileUpload + + media = MediaFileUpload(str(source), mimetype=mime_type, resumable=True) + try: + if existing is None: + request = files.create( + body={_NAME: name, _PARENTS: [folder_id], _MIME_TYPE: mime_type}, + media_body=media, + fields=_ID, + supportsAllDrives=True, + ) + else: + # A new revision of the same file: its ID, links and sharing stay. + request = files.update( + fileId=existing[_ID], + media_body=media, + fields=_ID, + supportsAllDrives=True, + ) + request.execute() + finally: + # MediaFileUpload only closes its file when it is collected, and an error + # in flight keeps it alive: on Windows the source could not be removed. + media.stream().close() + + def _download(self, path: str, target: Path) -> None: + location = self.uri_for(path) + entry = self._content_entry(path) + files = self._service.files() + with _drive_errors(location), open(target, "wb") as handle: + from googleapiclient.http import MediaIoBaseDownload + + request = files.get_media(fileId=entry[_ID], supportsAllDrives=True) + downloader = MediaIoBaseDownload(handle, request, chunksize=_DOWNLOAD_CHUNK_SIZE) + done = False + while not done: + _, done = downloader.next_chunk() + + def _checksum(self, path: str, algorithm: str) -> str: + entry = self._content_entry(path) + digest = entry.get(_SERVER_DIGESTS.get(algorithm, "")) + return str(digest).lower() if digest else super()._checksum(path, algorithm) + + def _remove(self, file_id: str, location: str) -> None: + with _drive_errors(location): + self._service.files().delete(fileId=file_id, supportsAllDrives=True).execute() + + def _delete_file(self, path: str) -> None: + self._remove(str(self._existing(path)[_ID]), self.uri_for(path)) + + def _mkdir(self, path: str) -> None: + location = self.uri_for(path) + name = _leaf(path) + folder_id = self._folder_id(parent_of(path)) + existing = self._child(folder_id, name, location) + if existing is not None: + if not _is_folder(existing): + raise StoragePathTypeException(f"{location} is a file") + return + with _drive_errors(location): + self._service.files().create( + body={_NAME: name, _MIME_TYPE: FOLDER_MIME_TYPE, _PARENTS: [folder_id]}, + fields=_ID, + supportsAllDrives=True, + ).execute() + + def _delete_directory(self, path: str, recursive: bool) -> None: + location = self.uri_for(path) + folder_id = self._folder_id(path) + # Drive removes a folder with everything in it, so "empty" is checked here, against + # every child and not only the ones a path can address. + if not recursive and self._matches(_children_query(folder_id), location, first_only=True): + raise not_empty_error(location) + self._remove(folder_id, location) + + # ------------------------------------------------------------------ copy and move inside Drive + + def _native_transfer( + self, source: StorageBackend, source_path: str, path: str + ) -> tuple[_Entry, str] | None: + """Return the source entry and the target folder's ID when Drive can do the transfer. + + ``None`` sends the caller through a staged copy: the source is another backend + or another client, or the target exists and is replaced in place to keep its ID. + """ + if not isinstance(source, GoogleDriveStorage): + return None + location = self.uri_for(path) + origin = source._existing(source_path) + folder_id = self._folder_id(parent_of(path)) + existing = self._child(folder_id, _leaf(path), location) + if existing is not None: + # Two roots can show one file under two paths; replacing it with itself and + # then deleting the "source" would lose it. + if existing[_ID] == origin[_ID]: + raise StorageException(f"{location}: {_SAME_FILE}") + return None + if source._service is not self._service: + return None + return origin, folder_id + + def _copy_from(self, source: StorageBackend, source_path: str, path: str) -> bool: + transfer = self._native_transfer(source, source_path, path) + if transfer is None: + return False + origin, folder_id = transfer + with _drive_errors(self.uri_for(path)): + self._service.files().copy( + fileId=origin[_ID], + body={_NAME: _leaf(path), _PARENTS: [folder_id]}, + fields=_ID, + supportsAllDrives=True, + ).execute() + return True + + def _move_from(self, source: StorageBackend, source_path: str, path: str) -> bool: + transfer = self._native_transfer(source, source_path, path) + if transfer is None: + return False + origin, folder_id = transfer + location = self.uri_for(path) + files = self._service.files() + with _drive_errors(location): + if folder_id == MY_DRIVE: + # "parents" holds real IDs, so the alias is resolved before the two are compared. + my_drive = files.get(fileId=MY_DRIVE, fields=_ID, supportsAllDrives=True) + folder_id = str(my_drive.execute()[_ID]) + parents = [str(parent) for parent in origin.get(_PARENTS) or []] + moved: dict[str, str] = {} + if folder_id not in parents: + moved["addParents"] = folder_id + if parents: + moved["removeParents"] = ",".join(parents) + files.update( + fileId=origin[_ID], + body={_NAME: _leaf(path)}, + fields=_ID, + supportsAllDrives=True, + **moved, + ).execute() + return True + + def __eq__(self, other: object) -> bool: + return ( + isinstance(other, GoogleDriveStorage) + and other._root_id == self._root_id + and other._explicit_client is self._explicit_client + ) + + def __hash__(self) -> int: + return hash((GDRIVE_SCHEME, self._root_id, id(self._explicit_client))) + + def __repr__(self) -> str: + return f"GoogleDriveStorage(root_id={self._root_id!r})" + + +def gdrive_factory(uri: StorageURI) -> tuple[StorageBackend, str]: + """Scheme factory for ``gdrive:///`` on the shared ``driver_instance``. + + The authority is the ID of the root folder; empty or ``root`` means My Drive. + """ + return GoogleDriveStorage(root_id=uri.authority or MY_DRIVE), uri.path diff --git a/automation_file/storage/local_storage.py b/automation_file/storage/local_storage.py index cd47ab4..62f6679 100644 --- a/automation_file/storage/local_storage.py +++ b/automation_file/storage/local_storage.py @@ -22,7 +22,7 @@ import shutil import stat import uuid -from collections.abc import Iterable, Iterator +from collections.abc import Hashable, Iterable, Iterator from datetime import datetime, timezone from pathlib import Path from typing import BinaryIO @@ -152,6 +152,10 @@ def _entry_path(self, path: str) -> Path: parent, _, name = path.rpartition("/") return self.local_path(parent) / name if name else self.local_path(parent) + def _identity(self, path: str) -> Hashable: + # Resolved, so a rooted view, the rootless one and a link all name one file alike. + return (LOCAL_SCHEME, os.path.normcase(os.path.realpath(self.local_path(path)))) + def _is_root(self, path: str) -> bool: if not path: return True diff --git a/automation_file/storage/object_storage.py b/automation_file/storage/object_storage.py index b234986..4ca5086 100644 --- a/automation_file/storage/object_storage.py +++ b/automation_file/storage/object_storage.py @@ -15,7 +15,7 @@ from __future__ import annotations from abc import abstractmethod -from collections.abc import Iterable +from collections.abc import Hashable, Iterable from dataclasses import replace from pathlib import Path @@ -80,6 +80,13 @@ def _remove(self, key: str) -> None: # ------------------------------------------------------------------ StorageBackend primitives + def _store_identity(self) -> Hashable: + """Identify the container behind this backend, the same for every prefix of it.""" + return id(self) + + def _identity(self, path: str) -> Hashable: + return (self._store_identity(), self._key(path)) + def _key(self, path: str) -> str: return join_path(self._prefix, path) if path else self._prefix diff --git a/automation_file/storage/onedrive_storage.py b/automation_file/storage/onedrive_storage.py new file mode 100644 index 0000000..aa04bc4 --- /dev/null +++ b/automation_file/storage/onedrive_storage.py @@ -0,0 +1,495 @@ +"""OneDrive backend: ``onedrive:///``. + +``OneDriveStorage()`` serves the signed-in user's drive (Microsoft Graph +``/me/drive``) through the shared +:data:`~automation_file.remote.onedrive.client.onedrive_instance`, which the +caller initialises as before (``onedrive_instance.later_init(access_token)``, +``device_code_login(...)`` or the ``FA_onedrive_*`` actions). ``root=`` confines +the backend to one folder of the drive. The URI authority is always empty. + +Folders are real directories. OneDrive compares names without regard to case and +keeps the case they were written with, so ``Report.txt`` and ``report.txt`` are +one item; a move between two such spellings renames it. + +A file up to 4 MiB goes up in one request, a larger one through an upload session +in fragments read from the file as they are sent. A download is streamed to the +target file. Writing to a path that holds a file replaces its content and keeps +the item, which is also what a copy or a move onto an existing file does. A move +to a new path between two locations of one client is done by OneDrive itself; a +copy goes through a local staging file. ``delete`` sends the item to the +recycle bin, a folder together with everything in it. + +``stat`` reports the size, modification time, ETag and MIME type of a file. +""" + +from __future__ import annotations + +import contextlib +import os +from collections.abc import Iterable, Iterator, Mapping +from pathlib import Path +from typing import TYPE_CHECKING, Any, BinaryIO +from urllib.parse import quote + +from automation_file.exceptions import ( + OneDriveException, + StorageException, + StoragePathTypeException, + StoragePermissionException, + StorageTransientException, + StorageUnavailableException, + StorageURIException, +) +from automation_file.logging_config import file_automation_logger +from automation_file.storage.backend import ( + StorageBackend, + guess_content_type, + join_path, + missing_error, + not_empty_error, + parent_of, +) +from automation_file.storage.timestamps import parse_rfc3339 +from automation_file.storage.types import FileInfo, StorageCapabilities +from automation_file.storage.uri import StorageURI, normalize_path + +if TYPE_CHECKING: + from automation_file.remote.onedrive.client import OneDriveClient + +ONEDRIVE_SCHEME = "onedrive" +_DRIVE_ROOT = "/me/drive/root" +_CHILDREN = "/children" +_CONTENT = "/content" +_UPLOAD_SESSION = "/createUploadSession" +_ITEM_FIELDS = "id,name,size,lastModifiedDateTime,eTag,file,folder,parentReference" +_SELECT = "$select" +_TOP = "$top" +_PARENT_REFERENCE = "parentReference" +_FOLDER = "folder" +_ID = "id" +_NAME = "name" +_GET = "GET" +_PUT = "PUT" +_POST = "POST" +_PATCH = "PATCH" +_DELETE = "DELETE" +_NEXT_LINK = "@odata.nextLink" +_CONFLICT_BEHAVIOR = "@microsoft.graph.conflictBehavior" +_DEFAULT_MIME_TYPE = "application/octet-stream" +_HTTPS = "https://" +# Graph takes a whole file in one PUT up to this size; larger ones need an upload session. +_SIMPLE_UPLOAD_MAX = 4 * 1024 * 1024 +# Every fragment of an upload session but the last must be a multiple of 320 KiB. +_UPLOAD_FRAGMENT_UNIT = 320 * 1024 +_UPLOAD_CHUNK_SIZE = 32 * _UPLOAD_FRAGMENT_UNIT +_DOWNLOAD_CHUNK_SIZE = 1024 * 1024 +_TRANSFER_TIMEOUT = 120.0 +_OK_STATUS = range(200, 300) +_UPLOAD_DONE_STATUS = frozenset({200, 201}) +_NOT_FOUND = 404 +_CONFLICT = 409 +_DENIED_STATUS = frozenset({401, 403}) +_RETRY_STATUS = frozenset({408, 429}) +_SERVER_ERROR = 500 +_NO_STATUS: frozenset[int] = frozenset() +_MISSING_OK = frozenset({_NOT_FOUND}) +_MKDIR_TOLERATED = frozenset({_NOT_FOUND, _CONFLICT}) +_ERROR_CODE_LIMIT = 64 +_NOT_INSTALLED = "requests is not installed; the OneDrive backend needs it" +_SAME_FILE = "source and target are the same file" + +_Item = dict[str, Any] + + +@contextlib.contextmanager +def _transport_errors(location: str, *, chain_cause: bool = True) -> Iterator[None]: + """Turn the errors of ``requests`` into the storage layer's exceptions. + + ``chain_cause=False`` is for requests that reach a pre-authenticated URL (an + upload session, the redirect of a download): the text of a ``requests`` error + quotes the URL, so that error is not kept as the cause. + """ + try: + import requests + except ImportError as error: + raise StorageUnavailableException(_NOT_INSTALLED) from error + dropped = ( + requests.ConnectionError, + requests.Timeout, + requests.exceptions.ChunkedEncodingError, + ) + try: + yield + except requests.RequestException as error: + transient = isinstance(error, dropped) and not isinstance( + error, requests.exceptions.SSLError + ) + kind = StorageTransientException if transient else StorageException + raise kind(f"{location}: {type(error).__name__}") from (error if chain_cause else None) + + +def _error_code(response: Any) -> str: + """Return the ``error.code`` of a Graph error body (empty when there is none).""" + try: + code = response.json()["error"]["code"] + except (ValueError, KeyError, TypeError): + return "" + return code[:_ERROR_CODE_LIMIT] if isinstance(code, str) else "" + + +def _failure(response: Any, location: str) -> StorageException: + """Return the storage exception a failed Graph response stands for.""" + status = int(response.status_code) + detail = f"{status} {_error_code(response)}".strip() + if status == _NOT_FOUND: + return missing_error(location) + if status in _DENIED_STATUS: + return StoragePermissionException(f"access to {location} was denied ({detail})") + if status in _RETRY_STATUS or status >= _SERVER_ERROR: + return StorageTransientException(f"{location}: OneDrive answered {detail}") + return StorageException(f"{location}: OneDrive error {detail}") + + +def _document(response: Any, location: str) -> _Item: + """Return the JSON object in the body of ``response``.""" + try: + document = response.json() + except ValueError as error: + raise StorageException( + f"{location}: OneDrive answered with a body that is not JSON" + ) from error + if not isinstance(document, dict): + raise StorageException(f"{location}: OneDrive answered with an unexpected JSON value") + return document + + +def _leaf(path: str) -> str: + return path.rpartition("/")[2] + + +def _is_same_item(first: Mapping[str, Any], second: Mapping[str, Any]) -> bool: + """Say whether two driveItem resources are one item of one drive.""" + + def identity(item: Mapping[str, Any]) -> tuple[Any, Any]: + reference = item.get(_PARENT_REFERENCE) + drive = reference.get("driveId") if isinstance(reference, Mapping) else None + return drive, item.get(_ID) + + return first.get(_ID) is not None and identity(first) == identity(second) + + +def _item_info(path: str, item: Mapping[str, Any]) -> FileInfo: + modified_at = parse_rfc3339(item.get("lastModifiedDateTime")) + if _FOLDER in item: + return FileInfo(path=path, is_dir=True, modified_at=modified_at) + etag = item.get("eTag") + facet = item.get("file") + return FileInfo( + path=path, + size=int(item.get("size") or 0), + modified_at=modified_at, + etag=str(etag).strip('"') if etag else None, + content_type=facet.get("mimeType") if isinstance(facet, Mapping) else None, + ) + + +class OneDriveStorage(StorageBackend): + """The signed-in user's OneDrive, whole or confined to the folder ``root``.""" + + scheme = ONEDRIVE_SCHEME + capabilities = StorageCapabilities( + directories=True, modified_at=True, etag=True, content_type=True + ) + + def __init__(self, client: OneDriveClient | None = None, *, root: str = "") -> None: + self._explicit_client = client + self._root = normalize_path(root) + + @property + def root(self) -> str: + """The folder this backend is confined to (empty for the whole drive).""" + return self._root + + @property + def _client(self) -> OneDriveClient: + client = self._explicit_client + if client is None: + from automation_file.remote.onedrive.client import onedrive_instance + + client = onedrive_instance + try: + client.require_session() + except OneDriveException as error: + raise StorageUnavailableException( + "the OneDrive client is not initialised; call onedrive_instance.later_init() " + "or device_code_login(), or pass client= to OneDriveStorage" + ) from error + return client + + def uri_for(self, path: str = "") -> str: + return str(StorageURI(ONEDRIVE_SCHEME, "", self._drive_path(self._normalize(path)))) + + # ------------------------------------------------------------------ talking to Graph + + def _drive_path(self, path: str) -> str: + """Return ``path`` relative to the root of the drive.""" + return join_path(self._root, path) if path else self._root + + def _item_url(self, path: str, facet: str = "") -> str: + """Return the Graph path of the item ``path``, or of one of its facets (``/children``).""" + drive_path = self._drive_path(path) + if not drive_path: + return f"{_DRIVE_ROOT}{facet}" + address = f"{_DRIVE_ROOT}:/{quote(drive_path, safe='/')}" + return f"{address}:{facet}" if facet else address + + def _send( + self, + method: str, + target: str, + location: str, + *, + tolerate: frozenset[int] = _NO_STATUS, + chain_cause: bool = True, + **options: Any, + ) -> Any: + """Send one request through the client and return the response. + + A status outside 2xx and outside ``tolerate`` raises the storage exception + it stands for. + """ + client = self._client + with _transport_errors(location, chain_cause=chain_cause): + response = client.graph_send(method, target, **options) + if response.status_code in _OK_STATUS or response.status_code in tolerate: + return response + failure = _failure(response, location) + response.close() + raise failure + + def _item(self, path: str) -> _Item | None: + """Return the driveItem at ``path``, or ``None`` when nothing is there.""" + location = self.uri_for(path) + response = self._send( + _GET, + self._item_url(path), + location, + tolerate=_MISSING_OK, + params={_SELECT: _ITEM_FIELDS}, + ) + return None if response.status_code == _NOT_FOUND else _document(response, location) + + def _existing_item(self, path: str) -> _Item: + item = self._item(path) + if item is None: + raise missing_error(self.uri_for(path)) + return item + + # ------------------------------------------------------------------ StorageBackend primitives + + def _stat(self, path: str) -> FileInfo | None: + item = self._item(path) + return None if item is None else _item_info(path, item) + + def _list_dir(self, path: str) -> Iterable[FileInfo]: + location = self.uri_for(path) + found: list[FileInfo] = [] + target: str | None = self._item_url(path, _CHILDREN) + options: dict[str, Any] = {"params": {_SELECT: _ITEM_FIELDS}} + while target: + page = _document(self._send(_GET, target, location, **options), location) + found.extend( + _item_info(join_path(path, str(item[_NAME])), item) + for item in page.get("value", []) + if item.get(_NAME) + ) + # The link to the next page is a full URL that carries the query of this one. + following = page.get(_NEXT_LINK) + target = following if isinstance(following, str) else None + options = {} + return found + + def _upload(self, source: Path, path: str) -> None: + location = self.uri_for(path) + with open(source, "rb") as handle: + size = os.fstat(handle.fileno()).st_size + if size > _SIMPLE_UPLOAD_MAX: + self._upload_in_session(handle, size, path, location) + return + self._send( + _PUT, + self._item_url(path, _CONTENT), + location, + # Bytes, not the handle: requests would send an empty stream chunked. + data=handle.read(), + headers={"Content-Type": guess_content_type(path) or _DEFAULT_MIME_TYPE}, + timeout=_TRANSFER_TIMEOUT, + ) + + def _upload_in_session(self, handle: BinaryIO, size: int, path: str, location: str) -> None: + opened = self._send( + _POST, + self._item_url(path, _UPLOAD_SESSION), + location, + json={"item": {_CONFLICT_BEHAVIOR: "replace"}}, + ) + upload_url = _document(opened, location).get("uploadUrl") + if not isinstance(upload_url, str) or not upload_url.startswith(_HTTPS): + raise StorageException(f"{location}: OneDrive opened an upload session without a URL") + completed = False + try: + self._send_fragments(handle, size, upload_url, location) + completed = True + finally: + if not completed: + self._cancel_session(upload_url, location) + + def _send_fragments(self, handle: BinaryIO, size: int, upload_url: str, location: str) -> None: + sent = 0 + status = 0 + while sent < size: + fragment = handle.read(min(_UPLOAD_CHUNK_SIZE, size - sent)) + if not fragment: + raise StorageException(f"{location}: the local source shrank during the upload") + end = sent + len(fragment) + # The upload URL is pre-authenticated: it gets no bearer token and stays out of errors. + status = self._send( + _PUT, + upload_url, + location, + chain_cause=False, + authorized=False, + data=fragment, + headers={"Content-Range": f"bytes {sent}-{end - 1}/{size}"}, + timeout=_TRANSFER_TIMEOUT, + ).status_code + sent = end + if status not in _UPLOAD_DONE_STATUS: + raise StorageException( + f"{location}: OneDrive did not complete the upload session ({status})" + ) + + def _cancel_session(self, upload_url: str, location: str) -> None: + try: + self._send(_DELETE, upload_url, location, chain_cause=False, authorized=False) + except StorageException as error: + file_automation_logger.warning( + "OneDriveStorage: the upload session for %s could not be cancelled (%s); " + "it expires on its own", + location, + type(error).__name__, + ) + + def _download(self, path: str, target: Path) -> None: + location = self.uri_for(path) + # Graph redirects to a pre-authenticated download URL, so no error keeps its cause. + response = self._send( + _GET, + self._item_url(path, _CONTENT), + location, + chain_cause=False, + stream=True, + timeout=_TRANSFER_TIMEOUT, + ) + with ( + response, + _transport_errors(location, chain_cause=False), + open(target, "wb") as handle, + ): + for chunk in response.iter_content(chunk_size=_DOWNLOAD_CHUNK_SIZE): + handle.write(chunk) + + def _delete_file(self, path: str) -> None: + self._send(_DELETE, self._item_url(path), self.uri_for(path)) + + def _mkdir(self, path: str) -> None: + location = self.uri_for(path) + response = self._send( + _POST, + self._item_url(parent_of(path), _CHILDREN), + location, + tolerate=_MKDIR_TOLERATED, + json={_NAME: _leaf(path), _FOLDER: {}, _CONFLICT_BEHAVIOR: "fail"}, + ) + if response.status_code == _NOT_FOUND: + # Only the folder ``root=`` names can be missing here: the caller made the others. + raise missing_error(self.uri_for(parent_of(path))) + if response.status_code != _CONFLICT: + return + existing = self._item(path) + if existing is None: + raise StorageException(f"{location}: OneDrive reported a name conflict") + if _FOLDER not in existing: + raise StoragePathTypeException(f"{location} is a file") + + def _delete_directory(self, path: str, recursive: bool) -> None: + location = self.uri_for(path) + if not recursive: + # OneDrive removes a folder with everything in it, so "empty" is checked here. + page = self._send( + _GET, + self._item_url(path, _CHILDREN), + location, + params={_SELECT: _ID, _TOP: 1}, + ) + if _document(page, location).get("value"): + raise not_empty_error(location) + self._send(_DELETE, self._item_url(path), location) + + # ------------------------------------------------------------------ copy and move inside OneDrive + + def _copy_from(self, source: StorageBackend, source_path: str, path: str) -> bool: + if isinstance(source, OneDriveStorage): + existing = self._item(path) + # Two roots, or two spellings, can name one item: copying it onto itself is refused. + if existing is not None and _is_same_item(source._existing_item(source_path), existing): + raise StorageException(f"{self.uri_for(path)}: {_SAME_FILE}") + return False + + def _move_from(self, source: StorageBackend, source_path: str, path: str) -> bool: + if not isinstance(source, OneDriveStorage): + return False + location = self.uri_for(path) + one_client = source._client is self._client + existing = self._item(path) + if existing is not None: + if not _is_same_item(source._existing_item(source_path), existing): + # Another file is there: it is replaced in place by a staged copy. + return False + respelled = source._drive_path(source_path) != self._drive_path(path) + if not (one_client and respelled): + raise StorageException(f"{location}: {_SAME_FILE}") + elif not one_client: + return False + folder = self._existing_item(parent_of(path)) + self._send( + _PATCH, + source._item_url(source_path), + location, + json={_PARENT_REFERENCE: {_ID: folder[_ID]}, _NAME: _leaf(path)}, + ) + return True + + def __eq__(self, other: object) -> bool: + return ( + isinstance(other, OneDriveStorage) + and other._root == self._root + and other._explicit_client is self._explicit_client + ) + + def __hash__(self) -> int: + return hash((ONEDRIVE_SCHEME, self._root, id(self._explicit_client))) + + def __repr__(self) -> str: + return f"OneDriveStorage(root={self._root!r})" + + +def onedrive_factory(uri: StorageURI) -> tuple[StorageBackend, str]: + """Scheme factory for ``onedrive:///`` on the shared ``onedrive_instance``.""" + if uri.authority: + intended = StorageURI(ONEDRIVE_SCHEME, "", join_path(uri.authority, uri.path)) + raise StorageURIException( + f"{str(uri)!r} names {uri.authority!r} as its authority; a OneDrive URI addresses " + f"the signed-in user's drive and takes an empty one, as in '{intended}'" + ) + return OneDriveStorage(), uri.path diff --git a/automation_file/storage/resolver.py b/automation_file/storage/resolver.py index 357ebf0..7173544 100644 --- a/automation_file/storage/resolver.py +++ b/automation_file/storage/resolver.py @@ -10,6 +10,11 @@ of a scheme no mount claimed. The factory receives the parsed URI and returns ``(backend, path)``. +The default factories serve ``local``, ``memory``, ``s3``, ``azure``, ``gdrive``, +``dropbox``, ``onedrive``, ``sftp``, ``ftp`` and ``ftps`` through the shared clients. +WebDAV, SMB and fsspec backends need a client or a filesystem of their own, so they +are mounted. + :data:`default_resolver` is the process-wide instance behind :class:`~automation_file.storage.File` and :class:`~automation_file.storage.Storage`. """ @@ -23,9 +28,14 @@ from automation_file.exceptions import StorageURIException from automation_file.storage.azure_storage import AZURE_SCHEME, AzureStorage from automation_file.storage.backend import StorageBackend +from automation_file.storage.dropbox_storage import DROPBOX_SCHEME, dropbox_factory +from automation_file.storage.ftp_storage import FTP_SCHEME, FTPS_SCHEME, ftp_factory +from automation_file.storage.gdrive_storage import GDRIVE_SCHEME, gdrive_factory from automation_file.storage.local_storage import LocalStorage from automation_file.storage.memory_storage import MEMORY_SCHEME, memory_store +from automation_file.storage.onedrive_storage import ONEDRIVE_SCHEME, onedrive_factory from automation_file.storage.s3_storage import S3_SCHEME, S3Storage +from automation_file.storage.sftp_storage import SFTP_SCHEME, sftp_factory from automation_file.storage.types import StorageCapabilities from automation_file.storage.uri import ( LOCAL_SCHEME, @@ -167,6 +177,12 @@ def register_default_schemes(resolver: StorageResolver) -> None: resolver.register_scheme(MEMORY_SCHEME, _memory_factory) resolver.register_scheme(S3_SCHEME, _s3_factory) resolver.register_scheme(AZURE_SCHEME, _azure_factory) + resolver.register_scheme(GDRIVE_SCHEME, gdrive_factory) + resolver.register_scheme(DROPBOX_SCHEME, dropbox_factory) + resolver.register_scheme(ONEDRIVE_SCHEME, onedrive_factory) + resolver.register_scheme(SFTP_SCHEME, sftp_factory) + resolver.register_scheme(FTP_SCHEME, ftp_factory) + resolver.register_scheme(FTPS_SCHEME, ftp_factory) default_resolver = StorageResolver() diff --git a/automation_file/storage/s3_storage.py b/automation_file/storage/s3_storage.py index 7316bcb..c84ba65 100644 --- a/automation_file/storage/s3_storage.py +++ b/automation_file/storage/s3_storage.py @@ -15,7 +15,7 @@ from __future__ import annotations import contextlib -from collections.abc import Iterable, Iterator +from collections.abc import Hashable, Iterable, Iterator from pathlib import Path from typing import Any @@ -140,6 +140,9 @@ def _client(self) -> Any: "or pass client= to S3Storage" ) from error + def _store_identity(self) -> Hashable: + return (S3_SCHEME, id(self._client), self._bucket) + def uri_for(self, path: str = "") -> str: key = self._key(self._normalize(path)) return str(StorageURI(S3_SCHEME, self._bucket, key)) diff --git a/automation_file/storage/session_storage.py b/automation_file/storage/session_storage.py new file mode 100644 index 0000000..e151532 --- /dev/null +++ b/automation_file/storage/session_storage.py @@ -0,0 +1,283 @@ +"""Shared behaviour of the backends that work through one login session: SFTP and FTP. + +A session backend serves the directory tree below ``root`` on the host its client +is connected to. :class:`SessionStorage` supplies what does not depend on the +protocol: + +* Every storage path is joined to ``root`` and sent as an absolute remote path. +* An upload goes to a sibling ``.part`` name that is renamed over the target, so + a failed upload never leaves a truncated file behind. +* A move within one session is a rename. +* One operation runs on a session at a time: neither a paramiko SFTP channel nor + an FTP control connection can serve two callers at once. + +:func:`require_session_host` is the rule the ``sftp://`` and ``ftp://`` factories +share: a URI names no host, or the host of the open session. +""" + +from __future__ import annotations + +import contextlib +import posixpath +import threading +import uuid +import weakref +from abc import abstractmethod +from collections.abc import Hashable, Iterator +from contextlib import AbstractContextManager +from pathlib import Path +from typing import Any + +from automation_file.exceptions import StorageException, StorageURIException +from automation_file.logging_config import file_automation_logger +from automation_file.storage.backend import StorageBackend +from automation_file.storage.types import StorageCapabilities +from automation_file.storage.uri import StorageURI, normalize_path + +_PARTIAL_SUFFIX = "part" +_ASIDE_SUFFIX = "old" + +_locks: weakref.WeakKeyDictionary[Any, threading.RLock] = weakref.WeakKeyDictionary() +_locks_guard = threading.Lock() + + +def session_lock(session: Any) -> threading.RLock: + """Return the lock that lets one caller at a time use ``session``.""" + with _locks_guard: + return _locks.setdefault(session, threading.RLock()) + + +def absolute_root(root: str) -> str: + """Return ``root`` as an absolute remote path with no trailing slash (``/`` stays ``/``).""" + return f"/{normalize_path(root)}" + + +def hidden_sibling(remote: str, suffix: str) -> str: + """Return a unique dot-name next to ``remote``: ``...``.""" + directory, name = posixpath.split(remote) + return posixpath.join(directory, f".{name}.{uuid.uuid4().hex}.{suffix}") + + +def split_authority(authority: str) -> tuple[str, int | None]: + """Split ``host``, ``host:port`` or ``[v6-address]:port`` into the host and the port.""" + if authority.startswith("["): + host, _, rest = authority[1:].partition("]") + port = rest.removeprefix(":") + elif authority.count(":") == 1: + host, _, port = authority.partition(":") + else: + host, port = authority, "" + if not port: + return host, None + if not (port.isascii() and port.isdigit()): + raise StorageURIException(f"invalid port in the storage URI authority {authority!r}") + return host, int(port) + + +def session_authority(host: str | None, port: int | None, default_port: int) -> str: + """Return the URI authority of a session: empty without one, the port only when unusual.""" + if not host: + return "" + name = f"[{host}]" if ":" in host else host + return name if port is None or port == default_port else f"{name}:{port}" + + +def require_session_host(uri: StorageURI, client: Any, backend: str) -> None: + """Refuse a URI that names another host than the one ``client`` is connected to. + + An empty authority means "the open session". So does any authority while no + session is open: the first operation then raises ``StorageUnavailableException``. + ``backend`` is the class name the message tells the caller to mount. + """ + connected = client.host + if not uri.authority or not connected: + return + host, port = split_authority(uri.authority) + if host.casefold() == connected.strip("[]").casefold() and port in (None, client.port): + return + session = f"{connected}:{client.port}" + mount = f'Storage.mount("{uri.scheme}://{uri.authority}", {backend}(client))' + raise StorageURIException( + f"{str(uri)!r} names the host {uri.authority!r}, but the open {uri.scheme} session is " + f"connected to {session!r}. To reach another host, connect a second client to it and " + f"mount a backend for it: {mount}" + ) + + +class SessionStorage(StorageBackend): + """A :class:`StorageBackend` over the directory tree one login session can reach.""" + + capabilities = StorageCapabilities(directories=True, modified_at=True) + #: The port a URI of this backend leaves out. + default_port = 0 + + def __init__(self, client: Any = None, *, root: str = "/") -> None: + self._explicit_client = client + self._root = absolute_root(root) + + @property + def root(self) -> str: + """The absolute remote directory every path of this backend is joined to.""" + return self._root + + @property + def client(self) -> Any: + """The client whose session this backend works through.""" + if self._explicit_client is not None: + return self._explicit_client + return self._shared_client() + + # ------------------------------------------------------------------ the protocol + + @abstractmethod + def _shared_client(self) -> Any: + """Return the process-wide client, used when none was passed in.""" + + @abstractmethod + def _open_session(self) -> Any: + """Return the client's open session, or raise ``StorageUnavailableException``.""" + + @abstractmethod + def _errors(self, session: Any, location: str) -> AbstractContextManager[None]: + """Return a context manager that turns the errors of ``session`` into storage errors.""" + + @abstractmethod + def _store(self, session: Any, source: Path, remote: str) -> None: + """Write the local file ``source`` to the remote file ``remote``.""" + + @abstractmethod + def _rename(self, session: Any, origin: str, target: str) -> Exception | None: + """Rename ``origin`` to ``target`` and return ``None``, or return the server's refusal. + + A refusal is an error the server answers with, which it may do because a + file is at ``target``. A session that broke is raised as usual. + """ + + @abstractmethod + def _unlink(self, session: Any, remote: str) -> None: + """Remove the remote file ``remote``.""" + + def _uri_scheme(self) -> str: + """Return the scheme :meth:`uri_for` writes; a backend with two schemes picks one.""" + return self.scheme + + # ------------------------------------------------------------------ shared steps + + def _remote(self, path: str) -> str: + """Return the absolute remote path of the normalised storage path ``path``.""" + return posixpath.join(self._root, path) if path else self._root + + def _identity(self, path: str) -> Hashable: + # Two roots of one client name a file by the same absolute path. + return (self.scheme, id(self.client), self._remote(path)) + + @contextlib.contextmanager + def _session(self, path: str) -> Iterator[Any]: + """Yield the open session for one operation on ``path``, its errors translated.""" + session = self._open_session() + with session_lock(session), self._errors(session, self.uri_for(path)): + yield session + + def uri_for(self, path: str = "") -> str: + client = self.client + authority = session_authority(client.host, client.port, self.default_port) + remote = self._remote(self._normalize(path)).lstrip("/") + root = f"{self._uri_scheme()}://{authority}" + return f"{root}/{remote}" if remote or not authority else root + + def _upload(self, source: Path, path: str) -> None: + remote = self._remote(path) + partial = hidden_sibling(remote, _PARTIAL_SUFFIX) + with self._session(path) as session: + stored = False + try: + self._store(session, source, partial) + self._replace(session, partial, remote) + stored = True + finally: + if not stored: + self._discard(session, partial) + + def _replace(self, session: Any, origin: str, target: str) -> None: + """Rename ``origin`` to ``target``, replacing a file that is already there. + + Most servers do that in one step. A server that will not rename onto an + existing file has that file moved aside first, and put back when the rename + still does not happen: the one case in which the replacement is not atomic. + """ + refusal = self._rename(session, origin, target) + if refusal is None: + return + aside = hidden_sibling(target, _ASIDE_SUFFIX) + if self._rename(session, target, aside) is not None: + # Nothing is in the way, or nothing can be renamed: the refusal stands. + raise refusal + replaced = False + try: + refusal = self._rename(session, origin, target) + replaced = refusal is None + finally: + if not replaced: + self._put_back(session, aside, target) + if refusal is not None: + raise refusal + self._discard(session, aside) + + def _put_back(self, session: Any, aside: str, target: str) -> None: + """Give ``target`` its file back. The failed rename is the news, so this only logs.""" + try: + with self._errors(session, target): + refusal = self._rename(session, aside, target) + except StorageException as error: + refusal = error + if refusal is not None: + file_automation_logger.warning( + "%s: %s was not replaced and its content is now at %s (%s)", + type(self).__name__, + target, + aside, + type(refusal).__name__, + ) + + def _discard(self, session: Any, remote: str) -> None: + """Remove the leftover ``remote``. What happened before is the news, so this only logs.""" + try: + with self._errors(session, remote): + self._unlink(session, remote) + except StorageException as error: + file_automation_logger.warning( + "%s: %s may have been left behind (%s)", + type(self).__name__, + remote, + type(error).__name__, + ) + + def _delete_file(self, path: str) -> None: + with self._session(path) as session: + self._unlink(session, self._remote(path)) + + def _move_from(self, source: StorageBackend, source_path: str, path: str) -> bool: + if not isinstance(source, SessionStorage) or not self._shares_session(source): + return False + origin, target = source._remote(source_path), self._remote(path) + if origin == target: + raise StorageException(f"{self.uri_for(path)}: source and target are the same file") + with self._session(path) as session: + self._replace(session, origin, target) + return True + + def _shares_session(self, other: SessionStorage) -> bool: + return other.scheme == self.scheme and other.client is self.client + + def __eq__(self, other: object) -> bool: + return ( + isinstance(other, SessionStorage) + and self._shares_session(other) + and other._root == self._root + ) + + def __hash__(self) -> int: + return hash((self.scheme, self._root, id(self.client))) + + def __repr__(self) -> str: + return f"{type(self).__name__}(root={self._root!r})" diff --git a/automation_file/storage/sftp_storage.py b/automation_file/storage/sftp_storage.py new file mode 100644 index 0000000..ce729d3 --- /dev/null +++ b/automation_file/storage/sftp_storage.py @@ -0,0 +1,262 @@ +"""SFTP backend: ``sftp://[:]/``. + +``SFTPStorage()`` serves the files the shared +:data:`~automation_file.remote.sftp.client.sftp_instance` session can reach. The +caller opens that session as before (``sftp_instance.later_init(...)`` or +``FA_sftp_later_init``), with the host key pinned. Pass another connected +``SFTPClient`` to work with a second host, and ``root=`` to join every path to +one remote directory. + +``stat`` reports the size and the modification time the server returns. Symbolic +links are followed when reading and writing. Deleting never follows them: the +link is removed and its target is left alone. A recursive listing does not +descend into a linked directory. +""" + +from __future__ import annotations + +import contextlib +import posixpath +import stat +from collections.abc import Iterable, Iterator +from contextlib import AbstractContextManager +from datetime import datetime, timezone +from pathlib import Path +from typing import TYPE_CHECKING, Any + +from automation_file.exceptions import ( + StorageException, + StoragePermissionException, + StorageTransientException, + StorageUnavailableException, +) +from automation_file.storage.backend import ( + StorageBackend, + join_path, + missing_error, + not_empty_error, +) +from automation_file.storage.session_storage import SessionStorage, require_session_host +from automation_file.storage.types import FileInfo +from automation_file.storage.uri import StorageURI + +if TYPE_CHECKING: + from automation_file.remote.sftp.client import SFTPClient + +SFTP_SCHEME = "sftp" +_SSH_PORT = 22 + + +def _closed(sftp: Any) -> bool: + """Say whether the channel under ``sftp`` is closed; using it then raises a plain ``OSError``.""" + channel = sftp.get_channel() + return channel is None or bool(channel.closed) + + +@contextlib.contextmanager +def _sftp_errors(sftp: Any, location: str) -> Iterator[None]: + """Turn what paramiko raises into the storage layer's exceptions. + + paramiko reports an SFTP status as an ``OSError``: ``ENOENT`` for "no such + file", ``EACCES`` for "permission denied" and no ``errno`` for any other. + """ + try: + import paramiko + except ImportError as error: + raise StorageUnavailableException( + "paramiko is not installed; the SFTP backend needs it" + ) from error + try: + yield + except FileNotFoundError as error: + raise missing_error(location) from error + except PermissionError as error: + raise StoragePermissionException(f"access to {location} was denied") from error + # TimeoutError is what socket.timeout has been since Python 3.10. + except (paramiko.SSHException, EOFError, TimeoutError, ConnectionError) as error: + raise StorageTransientException(f"{location}: {type(error).__name__}") from error + except OSError as error: + if _closed(sftp): + raise StorageTransientException(f"{location}: the SFTP session is closed") from error + raise StorageException(f"{location}: {error}") from error + # UnicodeError: paramiko decodes names as UTF-8 and a server may send another encoding. + except (paramiko.SFTPError, UnicodeError) as error: + raise StorageException(f"{location}: {error}") from error + + +def _is_link(attributes: Any) -> bool: + return stat.S_ISLNK(attributes.st_mode or 0) + + +def _is_directory(attributes: Any) -> bool: + return stat.S_ISDIR(attributes.st_mode or 0) + + +def _file_info(path: str, attributes: Any) -> FileInfo: + is_dir = _is_directory(attributes) + modified = attributes.st_mtime + return FileInfo( + path=path, + is_dir=is_dir, + size=None if is_dir else attributes.st_size, + modified_at=None if modified is None else datetime.fromtimestamp(modified, timezone.utc), + ) + + +def _status(sftp: Any, error: OSError) -> OSError: + """Return ``error`` when it is an SFTP status, and raise it when the session broke.""" + if isinstance(error, (TimeoutError, ConnectionError)) or _closed(sftp): + raise error + return error + + +def _followed(sftp: Any, remote: str, link: Any) -> Any: + """Return the attributes of what the link ``remote`` points to, or its own when that is gone.""" + try: + return sftp.stat(remote) + except FileNotFoundError: + return link + + +def _directory_exists(sftp: Any, remote: str) -> bool: + try: + return _is_directory(sftp.stat(remote)) + except FileNotFoundError: + return False + + +def _remove_tree(sftp: Any, remote: str) -> None: + """Remove the directory ``remote`` and what it holds. A link is removed, never followed.""" + directories = [remote] + pending = [remote] + while pending: + directory = pending.pop() + for attributes in sftp.listdir_attr(directory): + child = posixpath.join(directory, attributes.filename) + if _is_directory(attributes): + directories.append(child) + pending.append(child) + else: + sftp.remove(child) + for directory in reversed(directories): + sftp.rmdir(directory) + + +class SFTPStorage(SessionStorage): + """The directory tree below ``root`` on the host of one SFTP session.""" + + scheme = SFTP_SCHEME + default_port = _SSH_PORT + + def __init__(self, client: SFTPClient | None = None, *, root: str = "/") -> None: + super().__init__(client, root=root) + + def _shared_client(self) -> Any: + from automation_file.remote.sftp.client import sftp_instance + + return sftp_instance + + def _open_session(self) -> Any: + try: + return self.client.require_sftp() + except RuntimeError as error: + raise StorageUnavailableException( + "the SFTP session is not open; call sftp_instance.later_init(...) first, " + "or give SFTPStorage a connected SFTPClient" + ) from error + + def _errors(self, session: Any, location: str) -> AbstractContextManager[None]: + return _sftp_errors(session, location) + + def _stat(self, path: str) -> FileInfo | None: + with self._session(path) as sftp: + try: + return _file_info(path, sftp.stat(self._remote(path))) + except FileNotFoundError: + return None + + def _entries(self, sftp: Any, path: str) -> list[tuple[FileInfo, bool]]: + """Return every child of the directory ``path`` and whether it is a symbolic link.""" + remote = self._remote(path) + found: list[tuple[FileInfo, bool]] = [] + for listed in sftp.listdir_attr(remote): + linked = _is_link(listed) + child = posixpath.join(remote, listed.filename) + attributes = _followed(sftp, child, listed) if linked else listed + found.append((_file_info(join_path(path, listed.filename), attributes), linked)) + return found + + def _list_dir(self, path: str) -> Iterable[FileInfo]: + with self._session(path) as sftp: + return [info for info, _ in self._entries(sftp, path)] + + def _walk(self, path: str) -> Iterable[FileInfo]: + found: list[FileInfo] = [] + pending = [path] + with self._session(path) as sftp: + while pending: + for info, linked in self._entries(sftp, pending.pop()): + found.append(info) + if info.is_dir and not linked: + pending.append(info.path) + return found + + def _store(self, session: Any, source: Path, remote: str) -> None: + session.put(str(source), remote) + + def _rename(self, session: Any, origin: str, target: str) -> Exception | None: + try: + # posix-rename@openssh.com replaces a file at the target in one step. + session.posix_rename(origin, target) + return None + except OSError as error: + if _status(session, error).errno is not None: + return error + # A status without an errno: this server may not have the extension. The + # plain rename it does have replaces nothing on most servers. + try: + session.rename(origin, target) + except OSError as error: + return _status(session, error) + return None + + def _unlink(self, session: Any, remote: str) -> None: + session.remove(remote) + + def _download(self, path: str, target: Path) -> None: + with self._session(path) as sftp: + sftp.get(self._remote(path), str(target)) + + def _mkdir(self, path: str) -> None: + remote = self._remote(path) + with self._session(path) as sftp: + try: + sftp.mkdir(remote) + except OSError as error: + # "It is already there" arrives as a status without an errno. + if _status(sftp, error).errno is not None or not _directory_exists(sftp, remote): + raise + + def _delete_directory(self, path: str, recursive: bool) -> None: + remote = self._remote(path) + with self._session(path) as sftp: + if _is_link(sftp.lstat(remote)): + sftp.remove(remote) + elif recursive: + _remove_tree(sftp, remote) + elif sftp.listdir_attr(remote): + raise not_empty_error(self.uri_for(path)) + else: + sftp.rmdir(remote) + + +def sftp_factory(uri: StorageURI) -> tuple[StorageBackend, str]: + """Serve ``sftp://[[:]]/`` through the shared SFTP session. + + The URI names no host, or the host that session is connected to: see + :func:`~automation_file.storage.session_storage.require_session_host`. + """ + from automation_file.remote.sftp.client import sftp_instance + + require_session_host(uri, sftp_instance, SFTPStorage.__name__) + return SFTPStorage(), uri.path diff --git a/automation_file/storage/smb_storage.py b/automation_file/storage/smb_storage.py new file mode 100644 index 0000000..9f27238 --- /dev/null +++ b/automation_file/storage/smb_storage.py @@ -0,0 +1,226 @@ +"""SMB / CIFS backend, mounted on an :class:`~automation_file.remote.smb.client.SMBClient`. + +The client carries the server, the share and the credentials, so there is no URI +factory: a backend is mounted where its files should appear. + +.. code-block:: python + + client = SMBClient("nas.example.com", "projects", "user", password) + Storage.mount("smb://nas.example.com/projects", SMBStorage(client)) + File("smb://nas.example.com/projects/2026/plan.docx").download_to("plan.docx") + +Directories are real. ``stat`` reports the size and the modification time. A move +between two paths of one client is a rename on the server; a copy goes through a +local staging file. +""" + +from __future__ import annotations + +import contextlib +import errno +from collections.abc import Hashable, Iterable, Iterator +from datetime import datetime, timedelta, timezone +from pathlib import Path +from typing import TYPE_CHECKING + +from automation_file.exceptions import ( + SMBException, + StorageException, + StorageNotFoundException, + StoragePermissionException, + StorageTransientException, + StorageUnavailableException, +) +from automation_file.storage.backend import StorageBackend, join_path, missing_error +from automation_file.storage.types import FileInfo, StorageCapabilities +from automation_file.storage.uri import StorageURI, normalize_path + +if TYPE_CHECKING: + from automation_file.remote.smb.client import SMBClient, SMBEntry + +SMB_SCHEME = "smb" +_EPOCH = datetime(1970, 1, 1, tzinfo=timezone.utc) +_MISSING_ERRNO = frozenset({errno.ENOENT, errno.ENOTDIR}) +# NTSTATUS STATUS_ACCESS_DENIED, which smbprotocol passes on without an errno. +_STATUS_ACCESS_DENIED = 0xC0000022 +_PROTOCOL_ERROR = ("SMBException",) +_LOGON_ERRORS = ("AccessDenied", "LogonFailure", "SMBAuthenticationError") +_SELF_AND_PARENT = frozenset({".", ".."}) +_MAX_CAUSES = 5 + + +def _protocol_errors(names: Iterable[str]) -> tuple[type[BaseException], ...]: + """Return the named ``smbprotocol.exceptions`` classes this installation has.""" + try: + from smbprotocol import exceptions as smb_errors + except ImportError: + # Without smbprotocol none of its errors can have been raised. + return () + found = (getattr(smb_errors, name, None) for name in names) + return tuple(error for error in found if isinstance(error, type)) + + +def _causes(error: BaseException) -> list[BaseException]: + """Return ``error`` and the errors it was raised from, outermost first.""" + found: list[BaseException] = [] + current: BaseException | None = error + while current is not None and len(found) < _MAX_CAUSES: + found.append(current) + current = current.__cause__ + return found + + +def _is_missing(error: BaseException) -> bool: + # smbprotocol raises an OSError subclass of its own, so the errno decides. + return ( + isinstance(error, (FileNotFoundError, NotADirectoryError)) + or getattr(error, "errno", None) in _MISSING_ERRNO + ) + + +def _is_denied(error: BaseException, logon_errors: tuple[type[BaseException], ...]) -> bool: + return ( + isinstance(error, (PermissionError, *logon_errors)) + or getattr(error, "errno", None) == errno.EACCES + or getattr(error, "ntstatus", None) == _STATUS_ACCESS_DENIED + ) + + +def _translated(error: BaseException, location: str) -> StorageException: + """Translate a client error by what it, or an error behind it, says went wrong.""" + causes = _causes(error) + if any(isinstance(cause, ImportError) for cause in causes): + return StorageUnavailableException("smbprotocol is not installed; the SMB backend needs it") + if any(_is_missing(cause) for cause in causes): + return missing_error(location) + logon_errors = _protocol_errors(_LOGON_ERRORS) + if any(_is_denied(cause, logon_errors) for cause in causes): + return StoragePermissionException(f"access to {location} was denied") + if any(isinstance(cause, (ConnectionError, TimeoutError)) for cause in causes): + return StorageTransientException(f"{location}: {error}") + return StorageException(f"{location}: {error}") + + +def _client_failures() -> tuple[type[BaseException], ...]: + """Return what a client call raises. + + The client wraps an ``OSError``; smbprotocol's own errors and the ``ValueError`` + of a failed connection come through as they are. + """ + return (SMBException, OSError, ValueError, *_protocol_errors(_PROTOCOL_ERROR)) + + +@contextlib.contextmanager +def _smb_errors(location: str) -> Iterator[None]: + """Turn SMB client and smbprotocol errors into the storage layer's exceptions.""" + try: + yield + except _client_failures() as error: + raise _translated(error, location) from error + + +def _moment(seconds: float | None) -> datetime | None: + """Return seconds since the epoch as aware UTC; before 1970 is fine, out of range is not.""" + if seconds is None: + return None + try: + return _EPOCH + timedelta(seconds=seconds) + except OverflowError: + return None + + +def _file_info(path: str, entry: SMBEntry) -> FileInfo: + return FileInfo( + path=path, + is_dir=entry.is_dir, + size=None if entry.is_dir else entry.size, + modified_at=_moment(entry.mtime), + ) + + +class SMBStorage(StorageBackend): + """The share an ``SMBClient`` is connected to, or the directory ``root`` on it.""" + + scheme = SMB_SCHEME + capabilities = StorageCapabilities(directories=True, modified_at=True) + + def __init__(self, client: SMBClient, *, root: str = "") -> None: + self._client = client + self._root = self._normalize(root) + + @property + def root(self) -> str: + """The directory this backend is confined to, relative to the share.""" + return self._root + + def _identity(self, path: str) -> Hashable: + # Two roots of one share; SMB ignores case name a file by the same full path. + return (SMB_SCHEME, id(self._client), self._remote(path).lower()) + + def uri_for(self, path: str = "") -> str: + inside = "/".join((self._client.share, self._root, self._normalize(path))) + return str(StorageURI(SMB_SCHEME, self._client.server, inside)) + + def _normalize(self, path: str) -> str: + # A backslash separates SMB path segments too, so "..\\.." must not get past the check. + return normalize_path(path.replace("\\", "/")) + + def _remote(self, path: str) -> str: + return join_path(self._root, path) if path else self._root + + def _stat(self, path: str) -> FileInfo | None: + try: + with _smb_errors(self.uri_for(path)): + entry = self._client.stat(self._remote(path)) + except StorageNotFoundException: + return None + return _file_info(path, entry) + + def _list_dir(self, path: str) -> Iterable[FileInfo]: + with _smb_errors(self.uri_for(path)): + entries = self._client.list_dir(self._remote(path)) + return [ + _file_info(join_path(path, entry.name), entry) + for entry in entries + if entry.name not in _SELF_AND_PARENT + ] + + def _upload(self, source: Path, path: str) -> None: + with _smb_errors(self.uri_for(path)): + self._client.upload(source, self._remote(path)) + + def _download(self, path: str, target: Path) -> None: + with _smb_errors(self.uri_for(path)): + self._client.download(self._remote(path), target) + + def _delete_file(self, path: str) -> None: + with _smb_errors(self.uri_for(path)): + self._client.delete(self._remote(path)) + + def _mkdir(self, path: str) -> None: + with _smb_errors(self.uri_for(path)): + self._client.mkdir(self._remote(path)) + + def _rmdir(self, path: str) -> None: + with _smb_errors(self.uri_for(path)): + self._client.rmdir(self._remote(path)) + + def _move_from(self, source: StorageBackend, source_path: str, path: str) -> bool: + if not isinstance(source, SMBStorage) or source._client is not self._client: + return False + with _smb_errors(self.uri_for(path)): + self._client.rename(source._remote(source_path), self._remote(path), overwrite=True) + return True + + def __eq__(self, other: object) -> bool: + return ( + isinstance(other, SMBStorage) + and other._client is self._client + and other._root == self._root + ) + + def __hash__(self) -> int: + return hash((SMB_SCHEME, id(self._client), self._root)) + + def __repr__(self) -> str: + return f"SMBStorage({self.uri_for()!r})" diff --git a/automation_file/storage/timestamps.py b/automation_file/storage/timestamps.py new file mode 100644 index 0000000..968986b --- /dev/null +++ b/automation_file/storage/timestamps.py @@ -0,0 +1,40 @@ +"""RFC 3339 timestamps as cloud APIs send them (``2026-10-08T02:30:00.123Z``).""" + +from __future__ import annotations + +import re +from datetime import datetime, timedelta, timezone + +_RFC3339 = re.compile( + r"(\d{4})-(\d{2})-(\d{2})[Tt ](\d{2}):(\d{2}):(\d{2})(?:\.(\d+))?([Zz]|[+-]\d{2}:\d{2})" +) +_MICROSECOND_DIGITS = 6 + + +def _utc_offset(zone: str) -> timedelta: + if zone in ("Z", "z"): + return timedelta(0) + offset = timedelta(hours=int(zone[1:3]), minutes=int(zone[4:6])) + return -offset if zone.startswith("-") else offset + + +def parse_rfc3339(value: object) -> datetime | None: + """Return ``value`` as an aware UTC ``datetime``, or ``None`` when it is not a timestamp. + + A fraction of any length is accepted and cut to microseconds. Before Python + 3.11 ``datetime.fromisoformat`` reads neither that nor the ``Z`` suffix. + """ + if not isinstance(value, str): + return None + match = _RFC3339.fullmatch(value.strip()) + if match is None: + return None + year, month, day, hour, minute, second = (int(part) for part in match.groups()[:6]) + fraction = (match.group(7) or "")[:_MICROSECOND_DIGITS].ljust(_MICROSECOND_DIGITS, "0") + try: + zone = timezone(_utc_offset(match.group(8))) + moment = datetime(year, month, day, hour, minute, second, int(fraction), tzinfo=zone) + return moment.astimezone(timezone.utc) + except (ValueError, OverflowError): + # Not a real date (month 13, second 60), or one that leaves the calendar in UTC. + return None diff --git a/automation_file/storage/webdav_storage.py b/automation_file/storage/webdav_storage.py new file mode 100644 index 0000000..2ff3331 --- /dev/null +++ b/automation_file/storage/webdav_storage.py @@ -0,0 +1,264 @@ +"""WebDAV backend, mounted on a :class:`~automation_file.remote.webdav.client.WebDAVClient`. + +The client carries the base URL and the credentials, so there is no URI factory: +a backend is mounted where its files should appear. + +.. code-block:: python + + client = WebDAVClient("https://files.example.com/remote.php/dav", "user", password) + Storage.mount("webdav://files.example.com", WebDAVStorage(client)) + File("webdav://files.example.com/reports/q1.csv").read() + +Collections are real directories. ``stat`` is a ``PROPFIND`` with ``Depth: 0`` and +reports the size, ``getlastmodified``, ``getetag`` and ``getcontenttype``. Copy +and move between two paths of one client are done by the server (``COPY`` / +``MOVE``), and deleting a directory is one ``DELETE``. +""" + +from __future__ import annotations + +import contextlib +from collections.abc import Hashable, Iterable, Iterator +from datetime import datetime, timezone +from email.utils import parsedate_to_datetime +from pathlib import Path +from typing import TYPE_CHECKING +from urllib.parse import unquote, urlsplit + +from automation_file.exceptions import ( + StorageException, + StorageNotFoundException, + StoragePermissionException, + StorageTransientException, + StorageUnavailableException, + StorageURIException, + WebDAVException, +) +from automation_file.storage.backend import ( + StorageBackend, + join_path, + missing_error, + not_empty_error, +) +from automation_file.storage.types import FileInfo, StorageCapabilities +from automation_file.storage.uri import StorageURI, normalize_path + +if TYPE_CHECKING: + from automation_file.remote.webdav.client import WebDAVClient, WebDAVEntry + +WEBDAV_SCHEME = "webdav" +_MISSING_STATUS = 404 +_DENIED_STATUS = frozenset({401, 403}) +_TRANSIENT_STATUS = frozenset({408, 429}) +_SERVER_ERROR = 500 +# MKCOL answers 405 when something is already at the path. +_ALREADY_THERE_STATUS = 405 +# What a server without COPY / MOVE answers to them. +_NO_SUCH_METHOD_STATUS = frozenset({405, 501}) + + +def _status_error(status: int | None, location: str) -> StorageException: + """Translate the HTTP status of a failed request into a storage exception.""" + if status is None: + return StorageException(f"{location}: the WebDAV request failed") + if status == _MISSING_STATUS: + return missing_error(location) + if status in _DENIED_STATUS: + return StoragePermissionException(f"access to {location} was denied ({status})") + if status in _TRANSIENT_STATUS or status >= _SERVER_ERROR: + return StorageTransientException(f"{location}: the WebDAV server answered {status}") + return StorageException(f"{location}: the WebDAV server answered {status}") + + +@contextlib.contextmanager +def _webdav_errors(location: str) -> Iterator[None]: + """Turn WebDAV client errors and dropped connections into the storage layer's exceptions.""" + try: + from requests import exceptions as request_errors + except ImportError as error: + raise StorageUnavailableException( + "requests is not installed; the WebDAV backend needs it" + ) from error + dropped = ( + request_errors.ConnectionError, + request_errors.Timeout, + request_errors.ChunkedEncodingError, + ) + try: + yield + except WebDAVException as error: + cause = error.__cause__ + if error.status_code is None and isinstance(cause, dropped): + raise StorageTransientException(f"{location}: {type(cause).__name__}") from error + raise _status_error(error.status_code, location) from error + except dropped as error: + # A download is streamed after the client has checked the response. + raise StorageTransientException(f"{location}: {type(error).__name__}") from error + + +def _status_of(error: StorageException) -> int | None: + """Return the HTTP status behind a translated error.""" + return getattr(error.__cause__, "status_code", None) + + +def _modified(text: str | None) -> datetime | None: + """Parse ``getlastmodified``, an HTTP date, into an aware UTC datetime.""" + if not text: + return None + try: + moment = parsedate_to_datetime(text) + except (TypeError, ValueError): + return None + if moment.tzinfo is None: + return moment.replace(tzinfo=timezone.utc) + return moment.astimezone(timezone.utc) + + +def _etag(text: str | None) -> str | None: + """Drop the quotes of a strong validator; a weak one (``W/"..."``) is kept as it is.""" + if text and len(text) > 1 and text.startswith('"') and text.endswith('"'): + return text[1:-1] + return text + + +def _file_info(path: str, entry: WebDAVEntry) -> FileInfo: + modified = _modified(entry.last_modified) + if entry.is_dir: + return FileInfo(path=path, is_dir=True, modified_at=modified) + return FileInfo( + path=path, + size=entry.size, + modified_at=modified, + etag=_etag(entry.etag), + content_type=entry.content_type, + ) + + +class WebDAVStorage(StorageBackend): + """The collection a ``WebDAVClient`` points at, or the collection ``root`` below it.""" + + scheme = WEBDAV_SCHEME + capabilities = StorageCapabilities( + directories=True, modified_at=True, etag=True, content_type=True + ) + + def __init__(self, client: WebDAVClient, *, root: str = "") -> None: + self._client = client + self._root = self._normalize(root) + + @property + def root(self) -> str: + """The collection this backend is confined to, relative to the client's base URL.""" + return self._root + + def _identity(self, path: str) -> Hashable: + # Two roots of one server name a file by the same full path. + return (WEBDAV_SCHEME, id(self._client), self._remote(path)) + + def uri_for(self, path: str = "") -> str: + base = urlsplit(self._client.base_url) + # The host without any user information the base URL may carry. + authority = base.netloc.rpartition("@")[2] + return str( + StorageURI(WEBDAV_SCHEME, authority, "/".join((unquote(base.path), self._root, path))) + ) + + def _normalize(self, path: str) -> str: + clean = normalize_path(path) + if clean != clean.rstrip(): + # WebDAVClient trims the path it is given, so the name could not be addressed. + raise StorageURIException(f"a WebDAV path cannot end with white space: {path!r}") + return clean + + def _remote(self, path: str) -> str: + """Return the path to ask the client for. + + With the leading slash the client neither trims white space off the first + segment nor reads the path as an absolute URL. + """ + joined = join_path(self._root, path) if path else self._root + return f"/{joined}" + + def _collection(self, path: str) -> str: + """Like :meth:`_remote`, with the trailing slash a collection is addressed by.""" + remote = self._remote(path) + return remote if remote.endswith("/") else f"{remote}/" + + def _stat(self, path: str) -> FileInfo | None: + try: + with _webdav_errors(self.uri_for(path)): + entry = self._client.stat(self._remote(path)) + except StorageNotFoundException: + return None + return _file_info(path, entry) + + def _list_dir(self, path: str) -> Iterable[FileInfo]: + with _webdav_errors(self.uri_for(path)): + entries = self._client.list_dir(self._collection(path), include_self=False) + return [_file_info(join_path(path, entry.name), entry) for entry in entries if entry.name] + + def _upload(self, source: Path, path: str) -> None: + with _webdav_errors(self.uri_for(path)): + self._client.upload(source, self._remote(path)) + + def _download(self, path: str, target: Path) -> None: + with _webdav_errors(self.uri_for(path)): + self._client.download(self._remote(path), target) + + def _delete_file(self, path: str) -> None: + with _webdav_errors(self.uri_for(path)): + self._client.delete(self._remote(path)) + + def _mkdir(self, path: str) -> None: + try: + with _webdav_errors(self.uri_for(path)): + self._client.mkcol(self._collection(path)) + except StorageException as error: + if _status_of(error) != _ALREADY_THERE_STATUS: + raise + existing = self._stat(path) + if existing is None or not existing.is_dir: + raise + + def _rmdir(self, path: str) -> None: + with _webdav_errors(self.uri_for(path)): + self._client.delete(self._collection(path)) + + def _delete_directory(self, path: str, recursive: bool) -> None: + # DELETE removes a collection with everything in it, in one request. + if not recursive and any(True for _ in self._list_dir(path)): + raise not_empty_error(self.uri_for(path)) + self._rmdir(path) + + def _copy_from(self, source: StorageBackend, source_path: str, path: str) -> bool: + return self._relocate(source, source_path, path, move=False) + + def _move_from(self, source: StorageBackend, source_path: str, path: str) -> bool: + return self._relocate(source, source_path, path, move=True) + + def _relocate(self, source: StorageBackend, source_path: str, path: str, *, move: bool) -> bool: + """COPY or MOVE on the server; ``False`` when it has to go through a staging file.""" + if not isinstance(source, WebDAVStorage) or source._client is not self._client: + return False + relocate = self._client.move if move else self._client.copy + try: + with _webdav_errors(self.uri_for(path)): + relocate(source._remote(source_path), self._remote(path), overwrite=True) + except StorageException as error: + if _status_of(error) in _NO_SUCH_METHOD_STATUS: + return False + raise + return True + + def __eq__(self, other: object) -> bool: + return ( + isinstance(other, WebDAVStorage) + and other._client is self._client + and other._root == self._root + ) + + def __hash__(self) -> int: + return hash((WEBDAV_SCHEME, id(self._client), self._root)) + + def __repr__(self) -> str: + return f"WebDAVStorage({self.uri_for()!r})" diff --git a/docs/source/API/storage.rst b/docs/source/API/storage.rst index ee39343..66ad794 100644 --- a/docs/source/API/storage.rst +++ b/docs/source/API/storage.rst @@ -72,3 +72,39 @@ Object stores .. automodule:: automation_file.storage.azure_storage :members: + +Login-session backends +---------------------- + +.. automodule:: automation_file.storage.session_storage + :members: + +.. automodule:: automation_file.storage.sftp_storage + :members: + +.. automodule:: automation_file.storage.ftp_storage + :members: + +Drive-style and mounted backends +-------------------------------- + +.. automodule:: automation_file.storage.gdrive_storage + :members: + +.. automodule:: automation_file.storage.onedrive_storage + :members: + +.. automodule:: automation_file.storage.dropbox_storage + :members: + +.. automodule:: automation_file.storage.webdav_storage + :members: + +.. automodule:: automation_file.storage.smb_storage + :members: + +.. automodule:: automation_file.storage.fsspec_storage + :members: + +.. automodule:: automation_file.storage.timestamps + :members: diff --git a/docs/source/Eng/usage/storage.rst b/docs/source/Eng/usage/storage.rst index a0b3d98..b39e51d 100644 --- a/docs/source/Eng/usage/storage.rst +++ b/docs/source/Eng/usage/storage.rst @@ -11,12 +11,13 @@ The ``FA_*`` actions and the per-backend functions (``s3_upload_file``, .. note:: - The layer is new and its API may still change before 1.0. The local - filesystem, an in-memory store, S3 and Azure Blob are built in today. Google - Drive, Dropbox, SFTP, FTP, WebDAV, SMB and fsspec are reached through their - existing clients and actions (:doc:`cloud`) until their adapters land; you can - already put any of them behind the layer by writing a backend - (`Writing a backend`_). + The layer is new and its API may still change before 1.0. Twelve backends are + built in: the local filesystem, an in-memory store, S3, Azure Blob, Google + Drive, Dropbox, OneDrive, SFTP, FTP / FTPS, WebDAV, SMB and anything fsspec + can address. Box has no adapter: it is reached through its ``FA_box_*`` + actions only (:doc:`cloud`). Each remote backend needs its extra installed + (``pip install "automation_file[s3]"``) and its client initialised, as its + entry under `Built-in backends`_ says. Quick start ----------- @@ -318,6 +319,290 @@ raises ``StorageUnavailableException``. azure_blob_instance.later_init(connection_string=connection_string) File("s3://reports/2026/q1.csv").copy_to("azure://backups/2026/q1.csv") +``SFTPStorage`` (``sftp://[:]/``) + The files one SFTP session can reach, through the shared ``sftp_instance``. + Open the session as before, with ``sftp_instance.later_init(host=..., + username=..., ...)`` or ``FA_sftp_later_init``: the host key is checked + against ``known_hosts`` and an unknown host is rejected. The path of the URI + is the absolute path on the server, so ``sftp://nas/data/q1.csv`` is + ``/data/q1.csv`` and not a path below the login directory. + + The host may be left out (``sftp:///data/q1.csv``), which means "the open + session". A host that is named must be the one the session is connected to; + letter case is ignored, and a port, when given, must match as well. Any other + host raises ``StorageURIException``. To reach a second host, connect another + ``SFTPClient`` and mount a backend for it: + ``Storage.mount("sftp://backup", SFTPStorage(client))``. + ``SFTPStorage(client, root="/srv/data")`` joins every path to one remote + directory. ``root`` is a path prefix and not a jail: a symbolic link on the + server can still lead out of it. + + ``stat`` reports the size and the modification time the server returns, in + UTC and whole seconds. A move within one session is a rename; a copy goes + through a local temporary file, because SFTP has no copy of its own. + + Symbolic links are followed when reading and writing. Deleting never follows + them: the link is removed and its target is left alone. Recursive listing + does not descend into linked directories. A link whose target is gone is + listed, while ``exists`` and ``stat`` report it missing. + +``FTPStorage`` (``ftp://[:]/``, ``ftps://…``) + The files one FTP or FTPS session can reach, through the shared + ``ftp_instance``. Open the session as before, with + ``ftp_instance.later_init(host=..., username=..., password=..., tls=True)`` + or ``FA_ftp_later_init``. The host rule, ``root=`` and the absolute paths + are those of ``SFTPStorage``; for a second host, mount + ``FTPStorage(client)`` with another connected ``FTPClient``. ``ftps://`` is + refused with ``StorageURIException`` unless the open session was started + with ``tls=True``; ``ftp://`` accepts either kind. Plain FTP sends the + password and the files unencrypted. + + On a server that offers ``MLST`` / ``MLSD`` (RFC 3659), ``stat`` reports the + type, the size and the modification time (UTC) from the server's facts. Any + other server is probed: a directory is what ``CWD`` enters, a file is what + ``SIZE`` and ``MDTM`` answer for, and a listing is ``NLST`` followed by up to + three commands for each name. That is slower, a directory then has no + modification time, and files the server hides from ``NLST`` (often those + whose name starts with a dot) are not listed. The working directory of the + session is put back after each probe. + + FTP answers "no such file" and "not allowed" with the same reply code, 550. + ``exists``, ``stat`` and listings read it as "missing"; an upload, a + download or a delete reads it as ``StoragePermissionException``. A path that + contains a line break is refused with ``StorageURIException``. + + Deleting never follows a symbolic link, on either kind of server. Listing + depends on the server: where ``MLSD`` marks links they are listed as files + and not entered, while a probed server shows a link to a directory as a + directory, and a recursive listing descends into it. + +SFTP and FTP have real directories (``capabilities.directories`` is ``True``): +``mkdir`` creates one and an empty directory can exist. Neither reports an ETag, +a version, a content type or metadata, and checksums are computed from the +downloaded content. An upload is written to a hidden ``.part`` file next to the +target and renamed over it, so a failed upload never leaves a truncated file. A +server that will not rename onto an existing file (SFTP without the +``posix-rename@openssh.com`` extension, FTP on Windows) has that file moved +aside first and removed afterwards, and put back if the rename still fails; that +replacement is not atomic. + +A session carries one operation at a time, so calls on the same session wait for +each other. The ``FA_sftp_*`` / ``FA_ftp_*`` actions do not take part in that: +do not run them on a session while another thread uses it through the storage +layer. Until ``later_init`` has run, every call raises +``StorageUnavailableException``. A lost or timed-out connection raises +``StorageTransientException``; the layer does not reconnect, so call +``later_init`` again before retrying. + +.. code-block:: python + + from automation_file import ( + File, SFTPClient, SFTPStorage, Storage, ftp_instance, sftp_instance, + ) + + sftp_instance.later_init(host="nas.example", username="ops", + key_filename="/home/ops/.ssh/id_ed25519") + ftp_instance.later_init(host="files.example", username="ops", + password=password, tls=True) + + File("sftp://nas.example/exports/q1.csv").copy_to("ftps://files.example/incoming/q1.csv") + File("sftp:///exports/q1.csv").move_to("sftp:///archive/2026/q1.csv") # one rename + + # A second host: its own client, mounted under its own authority. + backup = SFTPClient() + backup.later_init(host="backup.example", username="ops") + Storage.mount("sftp://backup.example", SFTPStorage(backup, root="/srv/backups")) + File("sftp:///archive/2026/q1.csv").copy_to("sftp://backup.example/2026/q1.csv") + +``DropboxStorage`` (``dropbox:///``) + The Dropbox of the shared ``dropbox_instance``, initialised as before with + ``dropbox_instance.later_init(token)`` or ``FA_dropbox_later_init``. The + authority is empty: ``dropbox:///reports/q1.csv`` is the file + ``/reports/q1.csv``, and ``dropbox://reports/q1.csv`` is refused with the + correct spelling. ``DropboxStorage(client)`` takes another + ``dropbox.Dropbox`` client and ``root=`` confines the backend to one folder; + mount such an instance to give it a URI. Folders are real directories. + + ``stat`` reports size, the server's modification time, the revision as + ``version`` and Dropbox's content hash as ``etag``. A file larger than 8 MiB + goes up through an upload session, 8 MiB at a time, so it is never held in + memory as a whole. Copy and move between two paths of one client are done by + Dropbox, and deleting a folder is one request. + + Dropbox compares names without regard to case. It never replaces a file on + copy or move, so an existing target is deleted first and that step is not + atomic. Until the client is initialised, every call raises + ``StorageUnavailableException``. + +``WebDAVStorage`` (mounted, for example at ``webdav://``) + A WebDAV server through a :class:`~automation_file.WebDAVClient`. The client + carries the base URL and the credentials, so no URI resolves on its own: + mount the backend where its files should appear. ``root=`` confines it to + one collection below the base URL. Collections are real directories. + + ``stat`` is a ``PROPFIND`` with ``Depth: 0`` and reports size, modification + time (``getlastmodified``), ``getetag`` and ``getcontenttype`` as the server + gives them. Copy and move between two paths of one client are done by the + server with ``COPY`` and ``MOVE``; a server without them gets a transfer + through a local staging file. Deleting a directory is one ``DELETE``. HTTP + 404 raises ``StorageNotFoundException``, 401 and 403 + ``StoragePermissionException``, 408, 429, 5xx and a dropped connection + ``StorageTransientException``. + + The base URL goes through the SSRF check of ``WebDAVClient`` (pass + ``allow_private_hosts=True`` for a server on a private network) and TLS is + verified by default. A path cannot end with white space. The caller closes + the client. + +``SMBStorage`` (mounted, for example at ``smb:///``) + One SMB / CIFS share through an :class:`~automation_file.SMBClient`, which + carries the server, the share and the credentials. It needs ``smbprotocol`` + (``pip install smbprotocol``); without it every call raises + ``StorageUnavailableException``. Mount the backend where its files should + appear. ``root=`` confines it to one directory of the share. Directories are + real. + + ``stat`` reports size and modification time. A move between two paths of one + client is a rename on the server; a copy goes through a local staging file. + ``/`` and ``\`` both separate path segments, and a ``..`` segment is refused + in either spelling. The caller closes the client. + +``FsspecStorage`` (mounted under any scheme) + Any `fsspec `_ filesystem — Google + Cloud Storage, HDFS, FTP, an archive — behind the storage contract. It needs + ``fsspec`` and the driver of the service (``gcsfs``, ``adlfs`` …); a missing + one raises ``StorageUnavailableException``. + ``FsspecStorage(filesystem, root=..., scheme=..., directories=...)`` wraps a + filesystem object, and + ``FsspecStorage.from_url(url, directories=..., **storage_options)`` builds + one from an fsspec URL, whose path becomes the root. Mount the backend under + the scheme of your choice. + + ``directories`` says whether the filesystem keeps a directory that has no + files in it. Leave it ``True`` for a real filesystem and pass ``False`` for + an object store, where a directory is only a key prefix. + ``stat`` reports size and, where the filesystem provides one, the + modification time: ``capabilities.modified_at`` says whether to expect it, + and a listing carries it only when the filesystem lists it. Copy and move + inside one filesystem object are done by the filesystem. + + Paths are literal: a name containing ``*``, ``?`` or ``[`` is never expanded + as a pattern. fsspec is not covered by the SSRF check, so build the backend + from configuration and never from request input. For a local directory use + ``LocalStorage(root)``, which also keeps symbolic links from leaving the + root. + +.. code-block:: python + + from automation_file import ( + DropboxStorage, File, FsspecStorage, SMBClient, SMBStorage, Storage, + WebDAVClient, WebDAVStorage, dropbox_instance, + ) + + dropbox_instance.later_init(token) + File("dropbox:///reports/q1.csv").copy_to("local:///backup/q1.csv") + Storage.mount("dropbox://team", DropboxStorage(root="team/shared")) + + dav = WebDAVClient("https://files.example.com/remote.php/dav", "user", password) + Storage.mount("webdav://files.example.com", WebDAVStorage(dav)) + + nas = SMBClient("nas.example.com", "projects", "user", password) + Storage.mount("smb://nas.example.com/projects", SMBStorage(nas, root="2026")) + + Storage.mount("gcs://reports", FsspecStorage.from_url("gcs://reports", directories=False)) + + File("webdav://files.example.com/reports/q1.csv").copy_to("gcs://reports/2026/q1.csv") + +``GoogleDriveStorage`` (``gdrive:///``) + My Drive through the shared ``driver_instance``, initialised as before with + ``driver_instance.later_init(token_path, credentials_path)`` or + ``FA_drive_later_init``. The URI authority is the ID of the folder that + serves as the root, and an empty one, or ``root``, is My Drive: + ``gdrive:///reports/q1.csv``, ``gdrive:///q1.csv``. In code that + is ``GoogleDriveStorage(root_id="")``; the ID of a shared drive + works too, and ``GoogleDriveStorage(client)`` takes another + ``GoogleDriveClient``. + + Drive addresses entries by ID, not by path, so a path is looked up one + folder at a time on every call and nothing is remembered in between. Names + are compared exactly: ``Report.txt`` and ``report.txt`` are two entries. + Drive also lets several entries of one folder share a name. Such a path + names no single entry, so every call on it raises ``StorageException`` + with the number of entries that share the name; none of them is ever + picked. A listing still shows each of them. A name that contains ``/`` + cannot be written as a path either; it is left out of listings, with a + warning in the log. Entries in the trash do not exist for this backend. + + Folders are real directories. Writing to a path that holds a file uploads + a new revision of it, so the file keeps its ID, its links and its sharing; + copying or moving onto an existing file does the same. A copy or a move to + a new path within one client is done by Drive, and a move keeps the ID. + ``delete`` removes permanently, without the trash, and a folder goes with + everything in it. + + Google Docs, Sheets, Slides and the other ``application/vnd.google-apps.*`` + types have no binary content. They are listed with ``size=None`` and can be + copied, moved and deleted, but ``download``, ``read_bytes`` and ``checksum`` + raise ``StorageUnsupportedException``, and a file cannot be written over + one. Nothing is exported to another format, and shortcuts are not followed. + + ``stat`` reports size, modification time, Drive's MD5 as the ETag, the + version number and the MIME type. ``checksum`` returns the MD5, SHA-1 or + SHA-256 Drive holds for the file without downloading it, and hashes the + content for any other algorithm. + + Limitations: each path segment costs one request; and because Drive does + not keep names unique, two writers that create the same new path at the + same moment leave two entries of that name. + +``OneDriveStorage`` (``onedrive:///``) + The signed-in user's OneDrive through the shared ``onedrive_instance``, + initialised as before with ``onedrive_instance.later_init(access_token)``, + ``onedrive_instance.device_code_login(client_id)`` or the matching + ``FA_onedrive_*`` actions. The URI authority is always empty: + ``onedrive:///reports/q1.csv``. Something written in its place + (``onedrive://reports/q1.csv``) is refused with the correct spelling. + ``OneDriveStorage(root="backups/2026")`` confines the backend to one folder, + which has to exist, and ``OneDriveStorage(client)`` takes another + ``OneDriveClient``. + + Folders are real directories. OneDrive compares names without regard to + case and keeps the case they were written with, so ``Report.txt`` and + ``report.txt`` are the same item and a name is unique in its folder. A + name with a character OneDrive forbids (``" * : < > ? \ |``) is refused by + the service and raises ``StorageException``. + + A file up to 4 MiB is uploaded in one request. A larger one goes through an + upload session in 10 MiB fragments read from the file as they are sent, and + a download is streamed to disk, so neither holds a whole file in memory. + Writing to a path that holds a file replaces its content and keeps the + item; copying or moving onto an existing file does the same. A move to a + new path within one client is done by OneDrive, and a copy goes through a + local staging file. ``delete`` sends the item to the recycle bin, a folder + together with everything in it. + + ``stat`` reports size, modification time, ETag and MIME type; there is no + version. Checksums are computed from the content. + + Limitations: only the signed-in user's own drive is served; and the client + does not renew its access token, so once the token expires every call + raises ``StoragePermissionException`` until a new one is installed. + +Google Drive and OneDrive have real directories, so ``mkdir`` creates a folder +and an empty one can exist (``capabilities.directories`` is ``True``). Both +services throttle: a rate-limit answer, a server error or a dropped connection +raises ``StorageTransientException``, which ``retry_on_transient`` can retry. +Until the client is initialised, every call raises +``StorageUnavailableException``. + +.. code-block:: python + + from automation_file import File, driver_instance, onedrive_instance + + driver_instance.later_init("token.json", "credentials.json") + onedrive_instance.later_init(access_token) + File("gdrive:///reports/2026/q1.csv").copy_to("onedrive:///backups/2026/q1.csv") + Streams and directory trees --------------------------- @@ -373,7 +658,8 @@ handle every URI no mount claimed. Storage.register_scheme("vault", lambda uri: (vault_backend(uri.authority), uri.path)) Storage.resolve("sandbox://jobs/42/out.csv") # (LocalStorage('/srv/jobs'), '42/out.csv') - Storage.schemes() # ['azure', 'local', 'memory', 's3', 'sandbox', 'vault'] + Storage.schemes() # ['azure', 'dropbox', 'ftp', 'ftps', 'gdrive', 'local', 'memory', + # 'onedrive', 's3', 'sandbox', 'sftp', 'vault'] ``Storage.mount`` / ``unmount`` / ``register_scheme`` / ``schemes`` / ``resolve`` work on the process-wide table. A private table is a diff --git a/docs/source/Zh-CN/usage/storage.rst b/docs/source/Zh-CN/usage/storage.rst index 8b6754f..37ecde2 100644 --- a/docs/source/Zh-CN/usage/storage.rst +++ b/docs/source/Zh-CN/usage/storage.rst @@ -10,10 +10,12 @@ API；:class:`~automation_file.StorageBackend` 则是后端需要实现的契约 .. note:: - 本层是新功能，API 在 1.0 之前仍可能调整。目前内置本地文件系统、内存存储、 - S3 与 Azure Blob 四种后端。Google Drive、Dropbox、SFTP、FTP、WebDAV、SMB 与 - fsspec 在各自的适配器完成之前，仍通过已有的客户端与动作使用（见 :doc:`cloud`）； - 你也可以现在就自行编写后端，把它们接到本层之后（见 `编写后端`_）。 + 本层是新功能，API 在 1.0 之前仍可能调整。目前内置十二种后端：本地文件系统、 + 内存存储、S3、Azure Blob、Google Drive、Dropbox、OneDrive、SFTP、FTP / FTPS、 + WebDAV、SMB，以及 fsspec 能访问的任何存储。Box 没有适配器，只能通过它的 + ``FA_box_*`` 动作使用（见 :doc:`cloud`）。每个远端后端都需要安装对应的 extra + （``pip install "automation_file[s3]"``）并初始化其客户端，详见 `内置后端`_ + 中各自的条目。 快速开始 -------- @@ -300,6 +302,254 @@ S3 与 Azure Blob 都是对象存储。目录只在其下还有 key 时才存在 azure_blob_instance.later_init(connection_string=connection_string) File("s3://reports/2026/q1.csv").copy_to("azure://backups/2026/q1.csv") +``SFTPStorage``（``sftp://[:]/<绝对路径>``） + 通过共用的 ``sftp_instance`` 访问一个 SFTP 会话所能到达的文件。打开会话的方式与 + 以往相同：``sftp_instance.later_init(host=..., username=..., ...)`` 或 + ``FA_sftp_later_init``；主机密钥会与 ``known_hosts`` 比对，未知的主机一律拒绝。 + URI 的路径就是服务器上的绝对路径，因此 ``sftp://nas/data/q1.csv`` 指的是 + ``/data/q1.csv``，而不是登录目录之下的路径。 + + 主机可以省略（``sftp:///data/q1.csv``），代表“已打开的会话”。如果写出主机， + 它必须是会话所连接的那一台；比对时不区分大小写，如果同时写了端口，端口也必须 + 一致。其他主机会抛出 ``StorageURIException``。要访问第二台主机，请另外连接一个 + ``SFTPClient`` 并为它挂载后端： + ``Storage.mount("sftp://backup", SFTPStorage(client))``。 + ``SFTPStorage(client, root="/srv/data")`` 会把每个路径都接在某个远程目录之下。 + ``root`` 只是路径前缀，并不是隔离环境：服务器上的符号链接仍可能通往它之外。 + + ``stat`` 报告服务器返回的大小与修改时间（UTC，精确到秒）。同一个会话内的移动 + 是一次重命名；复制则经由本地临时文件，因为 SFTP 本身没有复制功能。 + + 读写时会跟随符号链接；删除时绝不跟随：只移除链接本身，不动它指向的目标。 + 递归列出时不会进入被链接的目录。目标已不存在的链接仍会被列出，但 ``exists`` + 与 ``stat`` 会报告它不存在。 + +``FTPStorage``（``ftp://[:]/<绝对路径>``、``ftps://…``） + 通过共用的 ``ftp_instance`` 访问一个 FTP 或 FTPS 会话所能到达的文件。打开会话的 + 方式与以往相同： + ``ftp_instance.later_init(host=..., username=..., password=..., tls=True)`` 或 + ``FA_ftp_later_init``。主机规则、``root=`` 与绝对路径都和 ``SFTPStorage`` + 相同；要访问第二台主机，请用另一个已连接的 ``FTPClient`` 挂载 + ``FTPStorage(client)``。除非已打开的会话是以 ``tls=True`` 建立的，否则 + ``ftps://`` 会以 ``StorageURIException`` 拒绝；``ftp://`` 则两种会话都接受。 + 未加密的 FTP 会以明文传送密码与文件内容。 + + 服务器如果提供 ``MLST`` / ``MLSD``（RFC 3659），``stat`` 会按服务器的 fact 报告 + 类型、大小与修改时间（UTC）。其他服务器则以探测的方式判断：``CWD`` 进得去的是 + 目录，``SIZE`` 与 ``MDTM`` 有应答的是文件，列出目录则是 ``NLST`` 再加上每个名称 + 最多三条命令。这种方式比较慢，目录没有修改时间，而且服务器不在 ``NLST`` 中显示 + 的文件（通常是名称以点开头的文件）不会被列出。每次探测后都会把会话的工作目录 + 切回原处。 + + FTP 对“没有这个文件”与“不允许”使用同一个应答码 550。``exists``、``stat`` + 与列出目录会把它视为“不存在”；上传、下载与删除则把它视为 + ``StoragePermissionException``。含有换行符的路径会以 ``StorageURIException`` + 拒绝。 + + 不论哪一种服务器，删除时都绝不跟随符号链接。列出时则视服务器而定：``MLSD`` + 会标示链接的服务器，链接会列为文件且不会被进入；以探测方式处理的服务器则把 + 指向目录的链接显示为目录，递归列出时会进入其中。 + +SFTP 与 FTP 都有真正的目录（``capabilities.directories`` 为 ``True``）：``mkdir`` +会创建目录，空目录也可以存在。两者都不报告 ETag、版本、内容类型与元数据，校验码 +则由下载回来的内容计算。上传时会先写入目标旁边的隐藏 ``.part`` 文件，再重命名覆盖 +目标，因此失败的上传绝不会留下被截断的文件。如果服务器不允许重命名到已存在的文件 +之上（没有 ``posix-rename@openssh.com`` 扩展的 SFTP、Windows 上的 FTP），会先把该 +文件移到一旁，完成后再删除，如果重命名仍然失败则放回原处；这种替换方式不是原子 +操作。 + +一个会话一次只能执行一个操作，因此同一个会话上的调用会互相等待。 +``FA_sftp_*`` / ``FA_ftp_*`` 动作不受这个机制保护：其他线程正通过存储层使用某个 +会话时，不要同时对它执行这些动作。在调用 ``later_init`` 之前，每个调用都会抛出 +``StorageUnavailableException``。连接中断或超时会抛出 +``StorageTransientException``；存储层不会自动重新连接，重试之前请再调用一次 +``later_init``。 + +.. code-block:: python + + from automation_file import ( + File, SFTPClient, SFTPStorage, Storage, ftp_instance, sftp_instance, + ) + + sftp_instance.later_init(host="nas.example", username="ops", + key_filename="/home/ops/.ssh/id_ed25519") + ftp_instance.later_init(host="files.example", username="ops", + password=password, tls=True) + + File("sftp://nas.example/exports/q1.csv").copy_to("ftps://files.example/incoming/q1.csv") + File("sftp:///exports/q1.csv").move_to("sftp:///archive/2026/q1.csv") # 一次重命名 + + # 第二台主机：使用自己的客户端，挂载在自己的 authority 之下。 + backup = SFTPClient() + backup.later_init(host="backup.example", username="ops") + Storage.mount("sftp://backup.example", SFTPStorage(backup, root="/srv/backups")) + File("sftp:///archive/2026/q1.csv").copy_to("sftp://backup.example/2026/q1.csv") + +``DropboxStorage``（``dropbox:///``） + 通过共用的 ``dropbox_instance`` 访问 Dropbox，初始化方式与以往相同： + ``dropbox_instance.later_init(token)`` 或 ``FA_dropbox_later_init``。authority + 必须留空：``dropbox:///reports/q1.csv`` 就是文件 ``/reports/q1.csv``，而 + ``dropbox://reports/q1.csv`` 会被拒绝，并在错误信息中给出正确写法。 + ``DropboxStorage(client)`` 可以改用另一个 ``dropbox.Dropbox`` 客户端，``root=`` + 则把后端限制在某个文件夹内；这样的实例要挂载后才有 URI。文件夹是真正的目录。 + + ``stat`` 报告大小、服务器端的修改时间，并以修订版本（rev）作为 ``version``、 + 以 Dropbox 的内容哈希作为 ``etag``。超过 8 MiB 的文件会通过上传会话、每次 + 8 MiB 分段上传，因此不会整个读入内存。同一个客户端内两个路径之间的复制与 + 移动由 Dropbox 本身完成，删除文件夹只需要一次请求。 + + Dropbox 比较名称时不区分大小写。复制或移动时它不会替换已有文件，因此会先删除 + 已存在的目标，这一步并非原子操作。客户端尚未初始化时，每个调用都会抛出 + ``StorageUnavailableException``。 + +``WebDAVStorage``（以挂载方式使用，例如挂在 ``webdav://``） + 通过 :class:`~automation_file.WebDAVClient` 访问 WebDAV 服务器。基础 URL 与 + 凭据都在客户端上，因此没有任何 URI 能自行解析：请把后端挂载到文件应该出现的 + 位置。``root=`` 把后端限制在基础 URL 之下的某个集合（collection）内。集合是 + 真正的目录。 + + ``stat`` 是一次 ``Depth: 0`` 的 ``PROPFIND``，报告服务器提供的大小、修改时间 + （``getlastmodified``）、``getetag`` 与 ``getcontenttype``。同一个客户端内两个 + 路径之间的复制与移动由服务器以 ``COPY`` 与 ``MOVE`` 完成；不支持这两个方法的 + 服务器则改经本地暂存文件传输。删除目录只需要一次 ``DELETE``。HTTP 404 会抛出 + ``StorageNotFoundException``，401 与 403 抛出 ``StoragePermissionException``， + 408、429、5xx 与连接中断则抛出 ``StorageTransientException``。 + + 基础 URL 会经过 ``WebDAVClient`` 的 SSRF 检查（服务器位于私有网络时请传入 + ``allow_private_hosts=True``），而且默认会验证 TLS。路径不能以空白字符结尾。 + 客户端由调用方负责关闭。 + +``SMBStorage``（以挂载方式使用，例如挂在 ``smb:///``） + 通过 :class:`~automation_file.SMBClient` 访问一个 SMB / CIFS 共享；服务器、 + 共享名称与凭据都在客户端上。需要安装 ``smbprotocol`` + （``pip install smbprotocol``），未安装时每个调用都会抛出 + ``StorageUnavailableException``。请把后端挂载到文件应该出现的位置。``root=`` + 把后端限制在共享内的某个目录。目录是真正的目录。 + + ``stat`` 报告大小与修改时间。同一个客户端内两个路径之间的移动是服务器上的 + 重命名；复制则经由本地暂存文件。``/`` 与 ``\`` 都是路径分隔符，不论用哪一种 + 写法，``..`` 段都会被拒绝。客户端由调用方负责关闭。 + +``FsspecStorage``（可以挂载在任何 scheme 之下） + 把任何 `fsspec `_ 文件系统（Google + Cloud Storage、HDFS、FTP、压缩包……）放到存储契约之后。需要安装 ``fsspec`` 与 + 该服务的驱动（``gcsfs``、``adlfs`` ……），缺少时会抛出 + ``StorageUnavailableException``。 + ``FsspecStorage(filesystem, root=..., scheme=..., directories=...)`` 包装一个 + 文件系统对象； + ``FsspecStorage.from_url(url, directories=..., **storage_options)`` 则由 fsspec + URL 创建，URL 的路径会成为根目录。请用你选择的 scheme 挂载这个后端。 + + ``directories`` 表示文件系统是否保留没有任何文件的目录。真正的文件系统保持 + ``True``；对象存储的目录只是 key 的前缀，请传入 ``False``。``stat`` 报告大小， + 并在文件系统提供时报告修改时间：``capabilities.modified_at`` 说明是否可以 + 期待这个字段，而列出目录时只有在文件系统的列表本身带有时间时才会报告。同一个 + 文件系统对象内的复制与移动由文件系统本身完成。 + + 路径一律按字面解读：名称中含有 ``*``、``?`` 或 ``[`` 时绝不会被当成通配符 + 展开。fsspec 不在 SSRF 检查的范围内，因此后端应由配置创建，绝不要由请求输入 + 创建。本地目录请使用 ``LocalStorage(root)``，它还能阻止符号链接离开根目录。 + +.. code-block:: python + + from automation_file import ( + DropboxStorage, File, FsspecStorage, SMBClient, SMBStorage, Storage, + WebDAVClient, WebDAVStorage, dropbox_instance, + ) + + dropbox_instance.later_init(token) + File("dropbox:///reports/q1.csv").copy_to("local:///backup/q1.csv") + Storage.mount("dropbox://team", DropboxStorage(root="team/shared")) + + dav = WebDAVClient("https://files.example.com/remote.php/dav", "user", password) + Storage.mount("webdav://files.example.com", WebDAVStorage(dav)) + + nas = SMBClient("nas.example.com", "projects", "user", password) + Storage.mount("smb://nas.example.com/projects", SMBStorage(nas, root="2026")) + + Storage.mount("gcs://reports", FsspecStorage.from_url("gcs://reports", directories=False)) + + File("webdav://files.example.com/reports/q1.csv").copy_to("gcs://reports/2026/q1.csv") + +``GoogleDriveStorage``（``gdrive:///``） + 通过共用的 ``driver_instance`` 访问“我的云端硬盘”，初始化方式与以往相同： + ``driver_instance.later_init(token_path, credentials_path)`` 或 + ``FA_drive_later_init``。URI 的 authority 是作为根目录的文件夹 ID，留空或 + 写成 ``root`` 则代表“我的云端硬盘”：``gdrive:///reports/q1.csv``、 + ``gdrive:///q1.csv``。在代码中即 + ``GoogleDriveStorage(root_id="")``；共享云端硬盘的 ID 也可以， + ``GoogleDriveStorage(client)`` 则可以改用另一个 ``GoogleDriveClient``。 + + Drive 以 ID 而非路径来寻址，因此每次调用都会逐层文件夹查找路径，调用之间 + 不保留任何结果。名称采用完全匹配：``Report.txt`` 与 ``report.txt`` 是两个 + 条目。Drive 也允许同一个文件夹内有多个同名条目；这样的路径无法指向单个 + 条目，因此对它的每个调用都会抛出 ``StorageException``，并说明有几个条目 + 共用该名称，绝不会从中挑选一个。列出目录时仍会显示每一个同名条目。名称 + 含有 ``/`` 的条目同样无法写成路径，列出时会略过它，并在日志中留下警告。 + 回收站中的条目对这个后端而言并不存在。 + + 文件夹是真实的目录。写入已有文件的路径时，会上传该文件的新修订版本，因此 + 文件的 ID、链接与共享设置都会保留；复制或移动到已有文件上也是如此。在同一 + 个客户端之内复制或移动到新路径时由 Drive 本身完成，移动会保留 ID。 + ``delete`` 是永久删除，不经过回收站，文件夹会连同其中所有内容一并删除。 + + Google 文档、表格、幻灯片以及其他 ``application/vnd.google-apps.*`` 类型 + 没有二进制内容。它们在列出时 ``size=None``，可以复制、移动与删除，但 + ``download``、``read_bytes`` 与 ``checksum`` 会抛出 + ``StorageUnsupportedException``，也不能用文件覆盖它们。不会导出为其他 + 格式，也不会跟随快捷方式。 + + ``stat`` 报告大小、修改时间、作为 ETag 的 Drive MD5、版本号与 MIME 类型。 + ``checksum`` 对 MD5、SHA-1 与 SHA-256 直接返回 Drive 为该文件保存的值， + 无需下载；其他算法则由内容计算。 + + 限制：路径的每一层都要一次请求；而且 Drive 不保证名称唯一，两个写入者若 + 同时创建同一个新路径，会留下两个同名条目。 + +``OneDriveStorage``（``onedrive:///``） + 通过共用的 ``onedrive_instance`` 访问已登录用户的 OneDrive，初始化方式 + 与以往相同：``onedrive_instance.later_init(access_token)``、 + ``onedrive_instance.device_code_login(client_id)`` 或对应的 + ``FA_onedrive_*`` 动作。URI 的 authority 一律留空： + ``onedrive:///reports/q1.csv``。在该位置写了东西 + （``onedrive://reports/q1.csv``）会被拒绝，并提示正确写法。 + ``OneDriveStorage(root="backups/2026")`` 把后端限制在某个文件夹内，该 + 文件夹必须已经存在；``OneDriveStorage(client)`` 则可以改用另一个 + ``OneDriveClient``。 + + 文件夹是真实的目录。OneDrive 比较名称时不区分大小写，但会保留写入时的 + 大小写，因此 ``Report.txt`` 与 ``report.txt`` 是同一个条目，名称在所属 + 文件夹内是唯一的。名称含有 OneDrive 禁用的字符（``" * : < > ? \ |``）时会 + 被服务拒绝，并抛出 ``StorageException``。 + + 4 MiB 以内的文件以单个请求上传。更大的文件通过上传会话，以 10 MiB 的 + 分段边读边发；下载则以流式写入磁盘，因此两者都不会把整个文件放进内存。 + 写入已有文件的路径时会替换其内容并保留该条目；复制或移动到已有文件上也是 + 如此。在同一个客户端之内移动到新路径时由 OneDrive 本身完成，复制则经过 + 本地临时文件。``delete`` 会把条目送进回收站，文件夹会连同其中所有内容 + 一并送入。 + + ``stat`` 报告大小、修改时间、ETag 与 MIME 类型，没有版本。校验码由内容 + 计算。 + + 限制：只能访问已登录用户自己的云端硬盘；而且客户端不会刷新访问令牌， + 令牌过期后每个调用都会抛出 ``StoragePermissionException``，直到安装新的 + 令牌为止。 + +Google Drive 与 OneDrive 都有真实的目录，因此 ``mkdir`` 会创建文件夹，空目录 +也可以存在（``capabilities.directories`` 为 ``True``）。两个服务都会限流：收到 +限流响应、服务器错误或连接中断时会抛出 ``StorageTransientException``，可以交给 +``retry_on_transient`` 重试。客户端尚未初始化时，每个调用都会抛出 +``StorageUnavailableException``。 + +.. code-block:: python + + from automation_file import File, driver_instance, onedrive_instance + + driver_instance.later_init("token.json", "credentials.json") + onedrive_instance.later_init(access_token) + File("gdrive:///reports/2026/q1.csv").copy_to("onedrive:///backups/2026/q1.csv") + 流与目录树 ---------- @@ -351,7 +601,8 @@ URI 的解析分两步。**挂载** 优先：挂载把一个后端实例绑定 Storage.register_scheme("vault", lambda uri: (vault_backend(uri.authority), uri.path)) Storage.resolve("sandbox://jobs/42/out.csv") # (LocalStorage('/srv/jobs'), '42/out.csv') - Storage.schemes() # ['azure', 'local', 'memory', 's3', 'sandbox', 'vault'] + Storage.schemes() # ['azure', 'dropbox', 'ftp', 'ftps', 'gdrive', 'local', 'memory', + # 'onedrive', 's3', 'sandbox', 'sftp', 'vault'] ``Storage.mount`` / ``unmount`` / ``register_scheme`` / ``schemes`` / ``resolve`` 操作的是整个进程共用的表。需要私有的表时使用 diff --git a/docs/source/Zh-TW/usage/storage.rst b/docs/source/Zh-TW/usage/storage.rst index a2ba355..ca7bd72 100644 --- a/docs/source/Zh-TW/usage/storage.rst +++ b/docs/source/Zh-TW/usage/storage.rst @@ -10,10 +10,12 @@ API；:class:`~automation_file.StorageBackend` 則是後端要實作的契約。 .. note:: - 本層是新功能，API 在 1.0 之前仍可能調整。目前內建本機檔案系統、記憶體儲存、 - S3 與 Azure Blob 四種後端。Google Drive、Dropbox、SFTP、FTP、WebDAV、SMB 與 - fsspec 在各自的轉接器完成之前，仍透過既有的用戶端與動作使用（見 :doc:`cloud`）； - 你也可以現在就自行撰寫後端，把它們接到本層之後（見 `撰寫後端`_）。 + 本層是新功能，API 在 1.0 之前仍可能調整。目前內建十二種後端：本機檔案系統、 + 記憶體儲存、S3、Azure Blob、Google Drive、Dropbox、OneDrive、SFTP、FTP / FTPS、 + WebDAV、SMB，以及 fsspec 能存取的任何儲存。Box 沒有轉接器，只能透過它的 + ``FA_box_*`` 動作使用（見 :doc:`cloud`）。每個遠端後端都需要安裝對應的 extra + （``pip install "automation_file[s3]"``）並初始化其用戶端，詳見 `內建後端`_ + 中各自的條目。 快速開始 -------- @@ -300,6 +302,254 @@ S3 與 Azure Blob 都是物件儲存。目錄只在其下還有 key 時才存在 azure_blob_instance.later_init(connection_string=connection_string) File("s3://reports/2026/q1.csv").copy_to("azure://backups/2026/q1.csv") +``SFTPStorage``（``sftp://[:]/<絕對路徑>``） + 透過共用的 ``sftp_instance`` 存取一個 SFTP 工作階段所能到達的檔案。開啟工作階段 + 的方式與以往相同：``sftp_instance.later_init(host=..., username=..., ...)`` 或 + ``FA_sftp_later_init``；主機金鑰會與 ``known_hosts`` 比對，未知的主機一律拒絕。 + URI 的路徑就是伺服器上的絕對路徑，因此 ``sftp://nas/data/q1.csv`` 指的是 + ``/data/q1.csv``，而不是登入目錄之下的路徑。 + + 主機可以省略（``sftp:///data/q1.csv``），代表「已開啟的工作階段」。若寫出主機， + 它必須是工作階段所連線的那一台；比對時不分大小寫，若同時寫了連接埠，連接埠也 + 必須相符。其他主機會拋出 ``StorageURIException``。要存取第二台主機，請另外 + 連線一個 ``SFTPClient`` 並為它掛載後端： + ``Storage.mount("sftp://backup", SFTPStorage(client))``。 + ``SFTPStorage(client, root="/srv/data")`` 會把每個路徑都接在某個遠端目錄之下。 + ``root`` 只是路徑前綴，並不是隔離環境：伺服器上的符號連結仍可能通往它之外。 + + ``stat`` 回報伺服器傳回的大小與修改時間（UTC，精確到秒）。同一個工作階段內的 + 移動是一次重新命名；複製則經由本機暫存檔，因為 SFTP 本身沒有複製功能。 + + 讀寫時會跟隨符號連結；刪除時絕不跟隨：只移除連結本身，不動它指向的目標。 + 遞迴列出時不會進入被連結的目錄。目標已不存在的連結仍會被列出，但 ``exists`` + 與 ``stat`` 會回報它不存在。 + +``FTPStorage``（``ftp://[:]/<絕對路徑>``、``ftps://…``） + 透過共用的 ``ftp_instance`` 存取一個 FTP 或 FTPS 工作階段所能到達的檔案。開啟 + 工作階段的方式與以往相同： + ``ftp_instance.later_init(host=..., username=..., password=..., tls=True)`` 或 + ``FA_ftp_later_init``。主機規則、``root=`` 與絕對路徑都和 ``SFTPStorage`` + 相同；要存取第二台主機，請以另一個已連線的 ``FTPClient`` 掛載 + ``FTPStorage(client)``。除非已開啟的工作階段是以 ``tls=True`` 建立的，否則 + ``ftps://`` 會以 ``StorageURIException`` 拒絕；``ftp://`` 則兩種工作階段都接受。 + 未加密的 FTP 會以明文傳送密碼與檔案內容。 + + 伺服器若提供 ``MLST`` / ``MLSD``（RFC 3659），``stat`` 會依伺服器的 fact 回報 + 類型、大小與修改時間（UTC）。其他伺服器則以探測的方式判斷：``CWD`` 進得去的是 + 目錄，``SIZE`` 與 ``MDTM`` 有回應的是檔案，列出目錄則是 ``NLST`` 再加上每個名稱 + 最多三個指令。這種方式比較慢，目錄沒有修改時間，而且伺服器不在 ``NLST`` 中顯示 + 的檔案（通常是名稱以點開頭的檔案）不會被列出。每次探測後都會把工作階段的工作 + 目錄切回原處。 + + FTP 對「沒有這個檔案」與「不允許」使用同一個回覆碼 550。``exists``、``stat`` + 與列出目錄會把它視為「不存在」；上傳、下載與刪除則把它視為 + ``StoragePermissionException``。含有換行字元的路徑會以 ``StorageURIException`` + 拒絕。 + + 不論哪一種伺服器，刪除時都絕不跟隨符號連結。列出時則視伺服器而定：``MLSD`` + 會標示連結的伺服器，連結會列為檔案且不會被進入；以探測方式處理的伺服器則把 + 指向目錄的連結顯示為目錄，遞迴列出時會進入其中。 + +SFTP 與 FTP 都有真正的目錄（``capabilities.directories`` 為 ``True``）：``mkdir`` +會建立目錄，空目錄也可以存在。兩者都不回報 ETag、版本、內容類型與中繼資料，校驗碼 +則由下載回來的內容計算。上傳時會先寫入目標旁邊的隱藏 ``.part`` 檔，再重新命名蓋過 +目標，因此失敗的上傳絕不會留下被截斷的檔案。若伺服器不允許重新命名到已存在的檔案 +之上（沒有 ``posix-rename@openssh.com`` 擴充的 SFTP、Windows 上的 FTP），會先把該 +檔案移到一旁，完成後再刪除，若重新命名仍然失敗則放回原處；這種取代方式不是原子 +操作。 + +一個工作階段一次只能執行一個操作，因此同一個工作階段上的呼叫會互相等待。 +``FA_sftp_*`` / ``FA_ftp_*`` 動作不受這個機制保護：其他執行緒正透過儲存層使用某個 +工作階段時，不要同時對它執行這些動作。在呼叫 ``later_init`` 之前，每個呼叫都會拋出 +``StorageUnavailableException``。連線中斷或逾時會拋出 +``StorageTransientException``；儲存層不會自動重新連線，重試之前請再呼叫一次 +``later_init``。 + +.. code-block:: python + + from automation_file import ( + File, SFTPClient, SFTPStorage, Storage, ftp_instance, sftp_instance, + ) + + sftp_instance.later_init(host="nas.example", username="ops", + key_filename="/home/ops/.ssh/id_ed25519") + ftp_instance.later_init(host="files.example", username="ops", + password=password, tls=True) + + File("sftp://nas.example/exports/q1.csv").copy_to("ftps://files.example/incoming/q1.csv") + File("sftp:///exports/q1.csv").move_to("sftp:///archive/2026/q1.csv") # 一次重新命名 + + # 第二台主機：使用自己的用戶端，掛載在自己的 authority 之下。 + backup = SFTPClient() + backup.later_init(host="backup.example", username="ops") + Storage.mount("sftp://backup.example", SFTPStorage(backup, root="/srv/backups")) + File("sftp:///archive/2026/q1.csv").copy_to("sftp://backup.example/2026/q1.csv") + +``DropboxStorage``（``dropbox:///``） + 透過共用的 ``dropbox_instance`` 存取 Dropbox，初始化方式與以往相同： + ``dropbox_instance.later_init(token)`` 或 ``FA_dropbox_later_init``。authority + 必須留空：``dropbox:///reports/q1.csv`` 就是檔案 ``/reports/q1.csv``，而 + ``dropbox://reports/q1.csv`` 會被拒絕，並在錯誤訊息中給出正確寫法。 + ``DropboxStorage(client)`` 可改用另一個 ``dropbox.Dropbox`` 用戶端，``root=`` + 則把後端限制在某個資料夾內；這樣的實例要掛載後才有 URI。資料夾是真正的目錄。 + + ``stat`` 回報大小、伺服器端的修改時間，並以修訂版本（rev）作為 ``version``、 + 以 Dropbox 的內容雜湊作為 ``etag``。超過 8 MiB 的檔案會透過上傳工作階段、每次 + 8 MiB 分段上傳，因此不會整個讀進記憶體。同一個用戶端內兩個路徑之間的複製與 + 搬移由 Dropbox 本身完成，刪除資料夾只需要一次請求。 + + Dropbox 比對名稱時不分大小寫。複製或搬移時它不會取代既有檔案，因此會先刪除 + 已存在的目標，這一步並非原子操作。用戶端尚未初始化時，每個呼叫都會拋出 + ``StorageUnavailableException``。 + +``WebDAVStorage``（以掛載方式使用，例如掛在 ``webdav://``） + 透過 :class:`~automation_file.WebDAVClient` 存取 WebDAV 伺服器。基底 URL 與 + 憑證都在用戶端上，因此沒有任何 URI 能自行解析：請把後端掛載到檔案應該出現的 + 位置。``root=`` 把後端限制在基底 URL 之下的某個集合（collection）內。集合是 + 真正的目錄。 + + ``stat`` 是一次 ``Depth: 0`` 的 ``PROPFIND``，回報伺服器提供的大小、修改時間 + （``getlastmodified``）、``getetag`` 與 ``getcontenttype``。同一個用戶端內兩個 + 路徑之間的複製與搬移由伺服器以 ``COPY`` 與 ``MOVE`` 完成；不支援這兩個方法的 + 伺服器則改經本機暫存檔傳輸。刪除目錄只需要一次 ``DELETE``。HTTP 404 會拋出 + ``StorageNotFoundException``，401 與 403 拋出 ``StoragePermissionException``， + 408、429、5xx 與連線中斷則拋出 ``StorageTransientException``。 + + 基底 URL 會經過 ``WebDAVClient`` 的 SSRF 檢查（伺服器位於私有網路時請傳入 + ``allow_private_hosts=True``），而且預設會驗證 TLS。路徑不能以空白字元結尾。 + 用戶端由呼叫端負責關閉。 + +``SMBStorage``（以掛載方式使用，例如掛在 ``smb:///``） + 透過 :class:`~automation_file.SMBClient` 存取一個 SMB / CIFS 共用資料夾； + 伺服器、共用名稱與憑證都在用戶端上。需要安裝 ``smbprotocol`` + （``pip install smbprotocol``），未安裝時每個呼叫都會拋出 + ``StorageUnavailableException``。請把後端掛載到檔案應該出現的位置。``root=`` + 把後端限制在共用資料夾內的某個目錄。目錄是真正的目錄。 + + ``stat`` 回報大小與修改時間。同一個用戶端內兩個路徑之間的搬移是伺服器上的 + 重新命名；複製則經由本機暫存檔。``/`` 與 ``\`` 都是路徑分隔符號，不論用哪一種 + 寫法，``..`` 區段都會被拒絕。用戶端由呼叫端負責關閉。 + +``FsspecStorage``（可掛載在任何 scheme 之下） + 把任何 `fsspec `_ 檔案系統（Google + Cloud Storage、HDFS、FTP、壓縮檔……）放到儲存契約之後。需要安裝 ``fsspec`` 與 + 該服務的驅動程式（``gcsfs``、``adlfs`` ……），缺少時會拋出 + ``StorageUnavailableException``。 + ``FsspecStorage(filesystem, root=..., scheme=..., directories=...)`` 包裝一個 + 檔案系統物件； + ``FsspecStorage.from_url(url, directories=..., **storage_options)`` 則由 fsspec + URL 建立，URL 的路徑會成為根目錄。請用你選擇的 scheme 掛載這個後端。 + + ``directories`` 表示檔案系統是否保留沒有任何檔案的目錄。真正的檔案系統維持 + ``True``；物件儲存的目錄只是 key 的前綴，請傳入 ``False``。``stat`` 回報大小， + 並在檔案系統有提供時回報修改時間：``capabilities.modified_at`` 說明是否可以 + 期待這個欄位，而列出目錄時只有在檔案系統的清單本身帶有時間時才會回報。同一個 + 檔案系統物件內的複製與搬移由檔案系統本身完成。 + + 路徑一律照字面解讀：名稱中含有 ``*``、``?`` 或 ``[`` 時絕不會被當成萬用字元 + 展開。fsspec 不在 SSRF 檢查的範圍內，因此後端應由設定建立，絕不要由請求輸入 + 建立。本機目錄請使用 ``LocalStorage(root)``，它還能阻止符號連結離開根目錄。 + +.. code-block:: python + + from automation_file import ( + DropboxStorage, File, FsspecStorage, SMBClient, SMBStorage, Storage, + WebDAVClient, WebDAVStorage, dropbox_instance, + ) + + dropbox_instance.later_init(token) + File("dropbox:///reports/q1.csv").copy_to("local:///backup/q1.csv") + Storage.mount("dropbox://team", DropboxStorage(root="team/shared")) + + dav = WebDAVClient("https://files.example.com/remote.php/dav", "user", password) + Storage.mount("webdav://files.example.com", WebDAVStorage(dav)) + + nas = SMBClient("nas.example.com", "projects", "user", password) + Storage.mount("smb://nas.example.com/projects", SMBStorage(nas, root="2026")) + + Storage.mount("gcs://reports", FsspecStorage.from_url("gcs://reports", directories=False)) + + File("webdav://files.example.com/reports/q1.csv").copy_to("gcs://reports/2026/q1.csv") + +``GoogleDriveStorage``（``gdrive:///``） + 透過共用的 ``driver_instance`` 存取「我的雲端硬碟」，初始化方式與以往相同： + ``driver_instance.later_init(token_path, credentials_path)`` 或 + ``FA_drive_later_init``。URI 的 authority 是作為根目錄的資料夾 ID，留空或 + 寫成 ``root`` 則代表「我的雲端硬碟」：``gdrive:///reports/q1.csv``、 + ``gdrive:///q1.csv``。在程式中即 + ``GoogleDriveStorage(root_id="")``；共用雲端硬碟的 ID 也可以， + ``GoogleDriveStorage(client)`` 則可改用另一個 ``GoogleDriveClient``。 + + Drive 以 ID 而非路徑來定址，因此每次呼叫都會逐層資料夾查找路徑，呼叫之間 + 不保留任何結果。名稱採完全比對：``Report.txt`` 與 ``report.txt`` 是兩個 + 項目。Drive 也允許同一個資料夾內有多個同名項目；這樣的路徑無法指向單一 + 項目，因此對它的每個呼叫都會拋出 ``StorageException``，並說明有幾個項目 + 共用該名稱，絕不會從中挑選一個。列出目錄時仍會顯示每一個同名項目。名稱 + 含有 ``/`` 的項目同樣無法寫成路徑，列出時會略過它，並在記錄檔留下警告。 + 垃圾桶中的項目對這個後端而言並不存在。 + + 資料夾是真實的目錄。寫入已有檔案的路徑時，會上傳該檔案的新修訂版本，因此 + 檔案的 ID、連結與共用設定都會保留；複製或搬移到既有檔案上也是如此。在同一 + 個用戶端之內複製或搬移到新路徑時由 Drive 本身完成，搬移會保留 ID。 + ``delete`` 是永久刪除，不經過垃圾桶，資料夾會連同其中所有內容一併刪除。 + + Google 文件、試算表、簡報以及其他 ``application/vnd.google-apps.*`` 類型 + 沒有二進位內容。它們在列出時 ``size=None``，可以複製、搬移與刪除，但 + ``download``、``read_bytes`` 與 ``checksum`` 會拋出 + ``StorageUnsupportedException``，也不能用檔案覆寫它們。不會匯出成其他 + 格式，也不會跟隨捷徑。 + + ``stat`` 回報大小、修改時間、作為 ETag 的 Drive MD5、版本號與 MIME 類型。 + ``checksum`` 對 MD5、SHA-1 與 SHA-256 直接回傳 Drive 為該檔案保存的值， + 不需下載；其他演算法則由內容計算。 + + 限制：路徑的每一層都要一次請求；而且 Drive 不保證名稱唯一，兩個寫入者若 + 同時建立同一個新路徑，會留下兩個同名項目。 + +``OneDriveStorage``（``onedrive:///``） + 透過共用的 ``onedrive_instance`` 存取已登入使用者的 OneDrive，初始化方式 + 與以往相同：``onedrive_instance.later_init(access_token)``、 + ``onedrive_instance.device_code_login(client_id)`` 或對應的 + ``FA_onedrive_*`` 動作。URI 的 authority 一律留空： + ``onedrive:///reports/q1.csv``。在該位置寫了東西 + （``onedrive://reports/q1.csv``）會被拒絕，並提示正確寫法。 + ``OneDriveStorage(root="backups/2026")`` 把後端限制在某個資料夾內，該 + 資料夾必須已經存在；``OneDriveStorage(client)`` 則可改用另一個 + ``OneDriveClient``。 + + 資料夾是真實的目錄。OneDrive 比對名稱時不分大小寫，但會保留寫入時的 + 大小寫，因此 ``Report.txt`` 與 ``report.txt`` 是同一個項目，名稱在所屬 + 資料夾內是唯一的。名稱含有 OneDrive 禁用的字元（``" * : < > ? \ |``）時會 + 被服務拒絕，並拋出 ``StorageException``。 + + 4 MiB 以內的檔案以單一請求上傳。更大的檔案透過上傳工作階段，以 10 MiB 的 + 分段邊讀邊送；下載則以串流寫入磁碟，因此兩者都不會把整個檔案放進記憶體。 + 寫入已有檔案的路徑時會取代其內容並保留該項目；複製或搬移到既有檔案上也是 + 如此。在同一個用戶端之內搬移到新路徑時由 OneDrive 本身完成，複製則經過 + 本機暫存檔。``delete`` 會把項目送進資源回收筒，資料夾會連同其中所有內容 + 一併送入。 + + ``stat`` 回報大小、修改時間、ETag 與 MIME 類型，沒有版本。校驗碼由內容 + 計算。 + + 限制：只能存取已登入使用者自己的雲端硬碟；而且用戶端不會更新存取權杖， + 權杖過期後每個呼叫都會拋出 ``StoragePermissionException``，直到安裝新的 + 權杖為止。 + +Google Drive 與 OneDrive 都有真實的目錄，因此 ``mkdir`` 會建立資料夾，空目錄 +也可以存在（``capabilities.directories`` 為 ``True``）。兩個服務都會限流：收到 +限流回應、伺服器錯誤或連線中斷時會拋出 ``StorageTransientException``，可交給 +``retry_on_transient`` 重試。用戶端尚未初始化時，每個呼叫都會拋出 +``StorageUnavailableException``。 + +.. code-block:: python + + from automation_file import File, driver_instance, onedrive_instance + + driver_instance.later_init("token.json", "credentials.json") + onedrive_instance.later_init(access_token) + File("gdrive:///reports/2026/q1.csv").copy_to("onedrive:///backups/2026/q1.csv") + 串流與目錄樹 ------------ @@ -351,7 +601,8 @@ URI 的解析分兩步。**掛載** 優先：掛載把一個後端實例綁定 Storage.register_scheme("vault", lambda uri: (vault_backend(uri.authority), uri.path)) Storage.resolve("sandbox://jobs/42/out.csv") # (LocalStorage('/srv/jobs'), '42/out.csv') - Storage.schemes() # ['azure', 'local', 'memory', 's3', 'sandbox', 'vault'] + Storage.schemes() # ['azure', 'dropbox', 'ftp', 'ftps', 'gdrive', 'local', 'memory', + # 'onedrive', 's3', 'sandbox', 'sftp', 'vault'] ``Storage.mount`` / ``unmount`` / ``register_scheme`` / ``schemes`` / ``resolve`` 操作的是整個行程共用的表。需要私有的表時使用 diff --git a/docs/updates/2026-10.md b/docs/updates/2026-10.md index 9657606..1600b98 100644 --- a/docs/updates/2026-10.md +++ b/docs/updates/2026-10.md @@ -346,3 +346,61 @@ Index and query commands: [README.md](README.md). New entries go at the end. - **Docs**: a "Storage" section in the three `usage/cli.rst` pages, the CLI block of the three READMEs, `architecture.md` §2 and §3, `CLAUDE.md` (package map). - **Files**: `automation_file/cli_storage.py`, `automation_file/__main__.py`, `tests/test_cli_storage.py`, the documentation above. - **Open items**: none for the storage commands; the pipeline, integrity and audit subcommands follow with their packages. + +## U-20261008-12 · 2026-10-08 · Storage adapters for eight more backends · #storage #roadmap #done + +- **What**: eight `StorageBackend` implementations over the existing clients, which closes `progress.md` #13, #14 and #15. Twelve backends are now built in. + - Resolved by URI through the shared client singletons (`register_default_schemes`): `SFTPStorage` (`sftp://`), `FTPStorage` (`ftp://`, `ftps://`), `GoogleDriveStorage` (`gdrive:///path`), `OneDriveStorage` (`onedrive:///path`), `DropboxStorage` (`dropbox:///path`). + - Mounted, because each needs a client or a filesystem of its own: `WebDAVStorage`, `SMBStorage`, `FsspecStorage` (`FsspecStorage.from_url(url, directories=...)`). + - `SessionStorage` (`storage/session_storage.py`) is the shared base of SFTP and FTP: paths are absolute below `root`, an upload goes to a hidden `.part` sibling that is renamed over the target, a move within one session is a rename, and one operation runs on a session at a time. A server that refuses to rename onto an existing file has the old file moved aside and put back if the rename still fails; that path is not atomic. + - `require_session_host`: an `sftp://` or `ftp://` URI that names a host other than the one the session is connected to is refused with `StorageURIException`. `SFTPClient` and `FTPClient` gained read-only `host` / `port` (and `tls`) for it. + - Google Drive: a path is resolved to a file ID folder by folder; two entries of one name in a folder make the path ambiguous and are refused; an overwrite keeps the file's ID. OneDrive goes through Microsoft Graph by path (`OneDriveClient.graph_send`). + - Client additions the adapters needed: `WebDAVException.status_code`; `WebDAVClient.base_url`, `stat`, `copy`, `move`, `list_dir(include_self=)` and a 207 answer to `DELETE` / `COPY` / `MOVE` treated as a failure; `SMBClient.server`, `share`, `stat` and `rename`. +- **Decision (#15)**: OneDrive is the eleventh remote backend of the storage contract. Box stays action-only (`FA_box_*`); the manuals say so. +- **Tests**: one `StorageContract` class (81 cases) per backend and per variant: SFTP (whole, rooted, without posix-rename), FTP, Drive, OneDrive, Dropbox, WebDAV, SMB, fsspec (memory, rooted memory, local, key/value), plus adapter-specific cases. `tests/test_storage_sftp_loopback.py` runs the contract through the real paramiko client against paramiko's own `SFTPServer` over a socket pair. Stand-ins: `tests/ftp_stand_in.py` (an in-memory FTP server behind a real `ftplib.FTP`), `tests/drive_stand_in.py` (the Drive v3 discovery document as an HTTP transport), `tests/graph_stand_in.py`. Test modules that need an SDK skip without it (`pytest.importorskip`). +- **Result / numbers**: 4015 passed, 149 skipped, 0 failed with every extra; 2298 passed, 89 skipped with the base dependencies only. `ruff check`, `ruff format --check` and `mypy automation_file` (194 files) pass. Importing `automation_file` loads none of the SDKs. Python 3.14.7 on Windows. +- **Not verified**: no adapter has met a real service. FTP ran against an in-memory model of RFC 959 / 3659 and FTPS data channels did not run at all; SFTP met paramiko's server, not OpenSSH; Graph, Drive, Dropbox, WebDAV and SMB met stand-ins, and the `smbprotocol` error shapes were written from its documentation. Recorded as `progress.md` #32. +- **Docs**: the "Built-in backends" section and the opening note of the three `usage/storage.rst` pages, `docs/source/API/storage.rst`, the feature list, diagram and storage section of the three READMEs, `architecture.md` §2 and §4, `CLAUDE.md` (package map, key types). +- **Files**: `automation_file/storage/{session,sftp,ftp,gdrive,onedrive,dropbox,webdav,smb,fsspec}_storage.py`, `storage/timestamps.py`, `storage/resolver.py`, `storage/__init__.py`, `automation_file/__init__.py`, `automation_file/exceptions.py`, `remote/{sftp,ftp,onedrive,smb,webdav}/client.py`, the tests and stand-ins above, the documentation above. +- **Open items**: #32 (real services), #31 (fsspec folder placeholders), #17 (native streams and checksums). + +## U-20261008-13 · 2026-10-08 · A move between two views of one store could delete the file · #storage #incident + +- **What happened**: `StorageBackend.move_from` and `copy_from` refused a transfer of a file onto itself only when the two backends compared equal. Two instances that show one store under different roots are not equal, so a move such as `AzureStorage("c").move_from(AzureStorage("c", prefix="team/a"), "x.txt", "team/a/x.txt")` wrote the file over itself and then deleted the "source": the file was gone. Reproduced for overlapping Azure prefixes; the same held for S3 prefixes and for two `LocalStorage` roots. Found while reviewing the adapter branches; the storage layer has not been released, so no user data was affected. +- **Fix**: a backend names the stored file behind a path with `_identity(path)`, and `_transfer_paths` refuses a transfer whose source and target identities are equal (`StorageException`, "source and target are the same file"), whatever the two instances are. + - `LocalStorage`: the real, case-normalised filesystem path (`os.path.realpath`), so two roots are recognised, and a symbolic link should be: links cannot be created on the development machine, so that half is not verified (`progress.md` #18). + - `ObjectStorage`: `_store_identity()` plus the full key; S3 is the client and the bucket, Azure the service client and the container. + - `SessionStorage` (SFTP, FTP): the client and the absolute remote path. Dropbox and SMB: the client and the lower-cased full path, since both ignore case. WebDAV: the client and the full path. fsspec: the filesystem object and the full path. + - Google Drive and OneDrive compare item IDs in their own `_copy_from` / `_move_from`. + - The default is `(id(self), path)`, which only recognises an instance's own paths; `CLAUDE.md` tells a backend with more than one view of a file to override it. +- **Tests**: `test_two_roots_of_one__do_not_lose_a_file_to_itself` (or the Drive / OneDrive equivalent) in the test modules of the local, S3, Azure, SFTP, Dropbox, WebDAV, SMB, fsspec, Drive and OneDrive backends: a move and a copy in both directions are refused and the content is still there. +- **Result / numbers**: part of the run recorded in U-20261008-12. +- **Limit**: two different clients connected to one account or one host are not recognised as one store. Nothing identifies the account behind a client without a request, and two logins of one host may see different trees. +- **Files**: `automation_file/storage/{backend,local_storage,object_storage,s3_storage,azure_storage,session_storage,dropbox_storage,webdav_storage,smb_storage,fsspec_storage}.py`, the test modules above, `CLAUDE.md`. +- **Open items**: none. + +## U-20261008-14 · 2026-10-08 · The WebDAV client only talks to its own server · #security #incident + +- **What was wrong**: `WebDAVClient` validated `base_url` once, in the constructor, and then: + - `_url_for` returned a path that began with `http://` or `https://` unchanged, so `client.download("https://other.example/x", ...)` sent the request, with the client's credentials, to another host that was never validated; + - every request went through `requests` with its default redirect handling, so an answer from the server could send the client on to an address the SSRF guard would have refused. `requests` drops the credentials when a redirect changes host, but it still makes the request. + Both predate this branch; they were found while reviewing the WebDAV adapter. +- **Fix** (`automation_file/remote/webdav/client.py`): + - every request URL must have the scheme, host and port of `base_url` (`_require_own_server`); anything else raises `WebDAVException` before a request is sent. An absolute URL on the same server is still accepted. + - requests are sent with `allow_redirects=False`. A redirect is followed by hand, at most five times, only for `GET`, `HEAD`, `OPTIONS` and `PROPFIND`, and only to a location on the same server. A redirect of a method that changes something (`PUT`, `DELETE`, `MKCOL`, `COPY`, `MOVE`) is reported as a `WebDAVException` with the status code. + - The `Destination` header of `COPY` / `MOVE` keeps its existing validation. +- **Tests**: eight cases in `tests/test_webdav_client.py`: no request leaves for another host, another port or another scheme; a same-server absolute URL works; `allow_redirects` is always `False`; a read follows a same-server redirect; a redirect to another server, a redirect of a write, a redirect loop and a redirect without `Location` are errors. +- **Result / numbers**: part of the run recorded in U-20261008-12. +- **Not verified**: a real WebDAV server. Servers that answer a collection without a trailing slash with a 301 are the case the same-server follow is for. +- **Files**: `automation_file/remote/webdav/client.py`, `tests/test_webdav_client.py`. +- **Open items**: none. + +## U-20261008-15 · 2026-10-08 · The SFTP, OneDrive and SMB clients name the extra to install · #packaging #done + +- **What**: the last three clients that reported a missing SDK without saying how to get it now do. Closes `progress.md` #30. + - `remote/sftp/client.py`: `require_module("paramiko", extra="sftp")`, which raises `OptionalDependencyException` (a `RuntimeError`, as before). + - `remote/onedrive/client.py` and `remote/smb/client.py`: the `OneDriveException` / `SMBException` they already raised now ends with `install_hint(...)`: `pip install "automation_file[onedrive]"`, `pip install "automation_file[smb]"`. +- **Tests**: `tests/test_optional_dependencies.py::test_the_package_imports_without_any_optional_dependency` now also blocks the three SDKs in a subprocess and checks that each client's error carries its `install_hint`. +- **Result / numbers**: part of the run recorded in U-20261008-12. +- **Files**: `automation_file/remote/sftp/client.py`, `automation_file/remote/onedrive/client.py`, `automation_file/remote/smb/client.py`, `tests/test_optional_dependencies.py`. +- **Open items**: none. diff --git a/docs/updates/README.md b/docs/updates/README.md index 9d7948d..f450c92 100644 --- a/docs/updates/README.md +++ b/docs/updates/README.md @@ -58,6 +58,10 @@ In the same commit: delete the item from `progress.md`, add a `#done` entry here | ID | Date | Title | Tags | Batch | |---|---|---|---|---| +| U-20261008-15 | 2026-10-08 | The SFTP, OneDrive and SMB clients name the extra to install | #packaging #done | [2026-10](2026-10.md) | +| U-20261008-14 | 2026-10-08 | The WebDAV client only talks to its own server | #security #incident | [2026-10](2026-10.md) | +| U-20261008-13 | 2026-10-08 | A move between two views of one store could delete the file | #storage #incident | [2026-10](2026-10.md) | +| U-20261008-12 | 2026-10-08 | Storage adapters for eight more backends | #storage #roadmap #done | [2026-10](2026-10.md) | | U-20261008-11 | 2026-10-08 | A storage subcommand for the CLI | #storage #cli #roadmap | [2026-10](2026-10.md) | | U-20261008-10 | 2026-10-08 | Failure cases join the storage contract | #storage #roadmap #tests | [2026-10](2026-10.md) | | U-20261008-09 | 2026-10-08 | Backend SDKs and the GUI toolkit become extras | #done #packaging #roadmap #decision | [2026-10](2026-10.md) | @@ -106,5 +110,5 @@ In the same commit: delete the item from `progress.md`, add a `#done` entry here | File | Period | Entries | |---|---|---:| -| [2026-10.md](2026-10.md) | 2026-10 | 22 | +| [2026-10.md](2026-10.md) | 2026-10 | 26 | | [2026-09.md](2026-09.md) | 2026-09 | 21 | diff --git a/progress.md b/progress.md index 98c9c8f..1a883b9 100644 --- a/progress.md +++ b/progress.md @@ -14,16 +14,15 @@ Items #10 to #26 are what is left of the 1.0 roadmap (`docs/FILEAUTOMATION-1.0-R ### Universal storage layer (roadmap M2) -- **#13** Storage adapters over the remaining clients, each in `storage/_storage.py` with a `StorageContract` class against a stand-in client: Dropbox (`dropbox:///path`), SFTP (`sftp://host/path`), FTP and FTPS, WebDAV, SMB (`smb://server/share/path`), fsspec. S3 and Azure Blob are done (U-20261008-02) and show the pattern: import the shared client and the SDK's exceptions inside functions, so `tests/test_storage_imports.py` keeps passing. A session backend must refuse a URI whose host is not the one it is connected to; `SFTPClient` does not keep its host today. -- **#14** Google Drive adapter (`gdrive://`). Drive addresses files by ID and allows two files of one name in a folder, so the path-to-ID lookup and the duplicate-name rule have to be designed first. -- **#15** [DECIDE] The eleventh backend slot, and whether OneDrive and Box are promoted to the storage contract or documented as action-only (roadmap §4). - **#16** `copy_between` / `FA_copy_between` on `File.copy_to`. The `FA_storage_*` actions exist (U-20261008-03), so the layer is reachable from action lists; the older action still has its own dispatcher in `remote/cross_backend.py`. `copy_between` accepts `local:`, `sftp:/path`, `s3:bucket/key` and http(s) sources today; `parse_storage_uri` rejects the first three as ambiguous, so the action needs a translation step to stay compatible. - **#17** Native streams and checksums for the remote backends. `open_read` / `open_write`, `copy_tree` and `sync_tree` exist (U-20261008-04), but outside `LocalStorage` a stream is a staged local copy and the default `checksum` downloads the file: S3 could read `get_object()["Body"]`, and a backend with a server-side digest could answer `checksum` from it. +- **#31** `FsspecStorage(directories=False)`: a `/` placeholder key that another tool wrote survives `delete(dir, recursive=True)`, so the directory still exists afterwards, and an empty directory that only a placeholder holds cannot be deleted at all. `ObjectStorage` removes placeholders because it lists raw keys; through fsspec the key has to be addressed with the filesystem's own call, and `_strip_protocol` of s3fs / gcsfs removes a trailing slash, so `rm_file(path + "/")` is probably wrong. Needs a real s3fs or gcsfs (MinIO under #19) before it is written. +- **#32** [UNVERIFIED] No storage adapter has met a real service: FTP ran against an in-memory model of RFC 959 / 3659 and FTPS data channels did not run at all; SFTP met paramiko's own server, not OpenSSH; Microsoft Graph (`OneDriveClient.graph_send`), Drive (the discovery document as a transport), Dropbox, WebDAV and SMB met stand-ins, and the `smbprotocol` error shapes were written from its documentation. The hand-written redirect handling of `WebDAVClient` (U-20261008-14) has not met a server that redirects. Run each against the service (#19) and fix what differs. - **#18** [UNVERIFIED] The five symbolic-link tests of `tests/test_storage_local.py` have not run anywhere: the development machine may not create links (they skip there), CI's Windows runners may. Read the first CI run of the branch and fix `LocalStorage` if one fails. ### Backend integration tests (roadmap M3) -- **#19** Integration environments for the contract suite in CI: MinIO and Azurite (`S3Storage` and `AzureStorage` have only met stand-in clients and, for S3, botocore's Stubber; no request has reached a real service), SFTP, FTP/FTPS, WebDAV and Samba, plus credential-gated jobs for the cloud adapters, and Linux and macOS legs (`ci-dev.yml` runs pytest on Windows only). Needs #13. +- **#19** Integration environments for the contract suite in CI: MinIO and Azurite (`S3Storage` and `AzureStorage` have only met stand-in clients and, for S3, botocore's Stubber; no request has reached a real service), SFTP, FTP/FTPS, WebDAV and Samba, plus credential-gated jobs for the cloud adapters, and Linux and macOS legs (`ci-dev.yml` runs pytest on Windows only). The adapters exist (U-20261008-12); what each has not met is listed in #32. - **#20** Metadata cases in the contract suite: user metadata and content type kept across an upload and a copy where `capabilities` says the backend supports them. The failure cases exist (U-20261008-10): a backend's contract class gets them by providing the `break_storage` fixture, as the local, S3 and Azure classes do. ### Later milestones @@ -38,4 +37,3 @@ Items #10 to #26 are what is left of the 1.0 roadmap (`docs/FILEAUTOMATION-1.0-R ### Packaging follow-ups - **#29** [BLOCKED] PyBreeze has to declare `automation-file[all]` before the stable release that splits the extras reaches users, or it installs without the SDKs it relied on. The change exists on PyBreeze's local branch `deps/automation-file-all-extra` (one commit on its `origin/dev`: `dev.toml`, `pyproject.toml`, `requirements.txt`), not pushed: it waits for someone to open the PR there and for PyBreeze's own update log. PyBreeze's checkout was on `docs/tutorials` with other work, so nothing else was touched. -- **#30** `remote/sftp/client.py` ("reinstall `automation_file`"), `remote/onedrive/client.py` (the same) and `remote/smb/client.py` ("install `smbprotocol`") still report a missing SDK without naming the extra. Change them to `core.optional.require_module` or `install_hint` once the adapter branches that edit those files are merged. diff --git a/tests/drive_stand_in.py b/tests/drive_stand_in.py new file mode 100644 index 0000000..bee4930 --- /dev/null +++ b/tests/drive_stand_in.py @@ -0,0 +1,418 @@ +"""An in-memory Google Drive for tests, served to the real ``googleapiclient``. + +:class:`FakeDrive` is an ``httplib2`` transport. Hand it to +``googleapiclient.discovery.build("drive", "v3", http=FakeDrive(), static_discovery=True)`` +and the service is the real one, built from the Drive v3 discovery document the +package ships: every ``files()`` call has its parameters checked against that +document, an upload and a download speak the resumable and ranged protocols of +``MediaFileUpload`` and ``MediaIoBaseDownload``, and a failure arrives as the +``HttpError`` googleapiclient builds from the response. No request leaves the +process. + +What the stand-in answers is a reading of Drive's documentation, not a recording +of the service. It assumes that a folder can hold several entries of one name, +that ``name =`` in a query ignores case, that a query below a folder that does +not exist is a 404, that ``root`` is an alias of the My Drive folder wherever a +file ID goes, that a ranged read of an empty file is a 416 with +``Content-Range: bytes */0``, and that deleting a folder deletes what is in it. +Where it is stricter than Drive (``addParents`` takes real IDs only) a comment +says so. +""" + +from __future__ import annotations + +import hashlib +import itertools +import json +import re +from dataclasses import dataclass, field +from datetime import datetime, timezone +from pathlib import Path +from typing import Any +from urllib.parse import parse_qsl, urlsplit + +import googleapiclient +import httplib2 + +from automation_file.storage.gdrive_storage import FOLDER_MIME_TYPE + +MY_DRIVE_ID = "0AFakeMyDriveRootId" +DOCUMENT_MIME_TYPE = "application/vnd.google-apps.document" +BINARY_MIME_TYPE = "application/octet-stream" +LIST_PAGE = 2 +UPLOAD_CHUNK_UNIT = 256 * 1024 +UPLOAD_URL = "https://upload.drive.invalid/session/" +DISCOVERY = json.loads( + ( + Path(googleapiclient.__file__).parent / "discovery_cache" / "documents" / "drive.v3.json" + ).read_text(encoding="utf-8") +) +FILE_SCHEMA = DISCOVERY["schemas"]["File"]["properties"] +FILE_METHODS = DISCOVERY["resources"]["files"]["methods"] +_LITERAL = r"'((?:[^'\\]|\\.)*)'" +_QUERY = re.compile(rf"{_LITERAL} in parents(?: and name = {_LITERAL})? and trashed = false") + + +class _Failure(Exception): + """An error answer of the stand-in: an HTTP status and a Drive ``reason`` code.""" + + def __init__(self, status: int, reason: str) -> None: + super().__init__(f"{status} {reason}") + self.status = status + self.reason = reason + + +def error_answer(status: int, reason: str) -> tuple[httplib2.Response, bytes]: + body = { + "error": { + "code": status, + "message": f"stand-in says {reason}", + "errors": [{"domain": "global", "reason": reason, "message": reason}], + } + } + headers = {"status": str(status), "content-type": "application/json"} + return httplib2.Response(headers), json.dumps(body).encode("utf-8") + + +def _answer(document: Any) -> tuple[httplib2.Response, bytes]: + headers = {"status": "200", "content-type": "application/json"} + return httplib2.Response(headers), json.dumps(document).encode("utf-8") + + +def _unescaped(literal: str) -> str: + return re.sub(r"\\(.)", r"\1", literal) + + +def _field_names(fields: str) -> list[str]: + """Split a ``fields`` selection of File properties; Drive refuses one it does not know.""" + names = [name.strip() for name in fields.split(",")] + if any(name not in FILE_SCHEMA for name in names): + raise _Failure(400, "invalid") + return names + + +@dataclass +class _Entry: + file_id: str + name: str + mime_type: str + parent: str | None + data: bytes | None = None + version: int = 1 + trashed: bool = False + modified: datetime = field(default_factory=lambda: datetime.now(timezone.utc)) + + def resource(self, *, digests: bool = True) -> dict[str, Any]: + stamp = self.modified.strftime("%Y-%m-%dT%H:%M:%S") + resource: dict[str, Any] = { + "id": self.file_id, + "name": self.name, + "mimeType": self.mime_type, + "parents": [self.parent] if self.parent else [], + "trashed": self.trashed, + "modifiedTime": f"{stamp}.{self.modified.microsecond // 1000:03d}Z", + "version": str(self.version), + } + if self.data is not None: + resource["size"] = str(len(self.data)) + if digests: + resource["md5Checksum"] = hashlib.md5(self.data, usedforsecurity=False).hexdigest() + resource["sha1Checksum"] = hashlib.sha1( + self.data, usedforsecurity=False + ).hexdigest() + resource["sha256Checksum"] = hashlib.sha256(self.data).hexdigest() + elif self.mime_type != FOLDER_MIME_TYPE: + # Drive reports the storage a Workspace document uses, which is not a content length. + resource["size"] = "1024" + return resource + + +@dataclass +class _Upload: + file_id: str | None + metadata: dict[str, Any] + mime_type: str + size: int + fields: str + received: bytearray = field(default_factory=bytearray) + + +@dataclass +class _Seen: + call: str + params: dict[str, str] + headers: dict[str, str] + body: Any = None + + +class FakeDrive: + """The part of the Drive v3 REST API that GoogleDriveStorage reaches, as an httplib2 transport. + + Like Drive, it lets a folder hold several entries of one name, matches ``name =`` + without regard to case, pages its listings, returns only the fields that were + asked for and removes a folder together with everything below it. + """ + + def __init__(self) -> None: + self.entries: dict[str, _Entry] = { + MY_DRIVE_ID: _Entry(MY_DRIVE_ID, "My Drive", FOLDER_MIME_TYPE, None) + } + self.seen: list[_Seen] = [] + self.fail_with: tuple[int, str] | Exception | None = None + self.fail_methods: frozenset[str] | None = None + self.digests = True + self._ids = itertools.count(1) + self._uploads: dict[str, _Upload] = {} + + # ------------------------------------------------------------------ for the tests + + @property + def calls(self) -> list[str]: + return [seen.call for seen in self.seen] + + def last(self, call: str) -> _Seen: + return next(seen for seen in reversed(self.seen) if seen.call == call) + + def add_folder(self, name: str, parent: str = MY_DRIVE_ID) -> str: + return self._new(name, FOLDER_MIME_TYPE, parent, None).file_id + + def add_file( + self, name: str, data: bytes, parent: str = MY_DRIVE_ID, mime_type: str = BINARY_MIME_TYPE + ) -> str: + return self._new(name, mime_type, parent, data).file_id + + def add_document(self, name: str, parent: str = MY_DRIVE_ID) -> str: + return self._new(name, DOCUMENT_MIME_TYPE, parent, None).file_id + + def named(self, name: str) -> list[_Entry]: + return [entry for entry in self.entries.values() if entry.name == name] + + def only(self, name: str) -> _Entry: + (entry,) = self.named(name) + return entry + + # ------------------------------------------------------------------ httplib2.Http + + def request( + self, + uri: str, + method: str = "GET", + body: Any = None, + headers: dict[str, str] | None = None, + redirections: int = 1, + connection_type: Any = None, + ) -> tuple[httplib2.Response, bytes]: + del redirections, connection_type + if self.fail_with is not None and ( + self.fail_methods is None or method in self.fail_methods + ): + if isinstance(self.fail_with, Exception): + raise self.fail_with + return error_answer(*self.fail_with) + lowered = {key.lower(): value for key, value in (headers or {}).items()} + try: + if uri.startswith(UPLOAD_URL): + return self._receive(uri, body, lowered) + split = urlsplit(uri) + return self._route(method, split.path, dict(parse_qsl(split.query)), body, lowered) + except _Failure as failure: + return error_answer(failure.status, failure.reason) + + def close(self) -> None: + return None + + # ------------------------------------------------------------------ routing + + def _route( + self, method: str, path: str, params: dict[str, str], body: Any, headers: dict[str, str] + ) -> tuple[httplib2.Response, bytes]: + assert params.get("supportsAllDrives") == "true", ( + f"{method} {path} without supportsAllDrives" + ) + if path.startswith("/upload/drive/v3/files"): + file_id = path.removeprefix("/upload/drive/v3/files").strip("/") + return self._open_upload(method, file_id, params, body, headers) + assert path.startswith("/drive/v3/files"), f"unexpected Drive path {path}" + file_id, _, verb = path.removeprefix("/drive/v3/files").strip("/").partition("/") + if not file_id: + if method == "GET": + return self._list(params, headers) + return self._create(params, json.loads(body), headers) + if verb == "copy": + return self._copy(file_id, params, json.loads(body), headers) + assert not verb, f"unexpected Drive path {path}" + if method == "DELETE": + return self._delete(file_id, params, headers) + if method == "PATCH": + return self._update(file_id, params, json.loads(body), headers) + if params.get("alt") == "media": + return self._media(file_id, params, headers) + self.seen.append(_Seen("get", params, headers)) + return _answer(self._projected(self._entry(file_id), params["fields"])) + + def _new(self, name: str, mime_type: str, parent: str, data: bytes | None) -> _Entry: + entry = _Entry(f"id-{next(self._ids):04d}", name, mime_type, parent, data) + self.entries[entry.file_id] = entry + return entry + + def _entry(self, file_id: str) -> _Entry: + """Look an ID up; ``root`` is Drive's alias of the My Drive folder.""" + entry = self.entries.get(MY_DRIVE_ID if file_id == "root" else file_id) + if entry is None: + raise _Failure(404, "notFound") + return entry + + def _folder(self, file_id: str) -> _Entry: + entry = self._entry(file_id) + if entry.mime_type != FOLDER_MIME_TYPE: + raise _Failure(400, "invalid") + return entry + + def _projected(self, entry: _Entry, fields: str) -> dict[str, Any]: + resource = entry.resource(digests=self.digests) + return {name: resource[name] for name in _field_names(fields) if name in resource} + + # ------------------------------------------------------------------ files.list / get / delete + + def _list(self, params: dict[str, str], headers: dict[str, str]) -> Any: + self.seen.append(_Seen("list", params, headers)) + assert params.get("includeItemsFromAllDrives") == "true" + query = _QUERY.fullmatch(params["q"]) + selection = re.fullmatch(r"nextPageToken, files\((.*)\)", params["fields"]) + if query is None or selection is None: + raise _Failure(400, "invalid") + folder = self._entry(_unescaped(query.group(1))) + name = None if query.group(2) is None else _unescaped(query.group(2)).casefold() + matches = [ + entry + for entry in self.entries.values() + if entry.parent == folder.file_id + and not entry.trashed + and (name is None or entry.name.casefold() == name) + ] + start = int(params.get("pageToken", "0")) + end = start + min(int(params.get("pageSize", "100")), LIST_PAGE) + page: dict[str, Any] = { + "files": [self._projected(entry, selection.group(1)) for entry in matches[start:end]] + } + if end < len(matches): + page["nextPageToken"] = str(end) + return _answer(page) + + def _media(self, file_id: str, params: dict[str, str], headers: dict[str, str]) -> Any: + self.seen.append(_Seen("media", params, headers)) + data = self._entry(file_id).data + if data is None: + raise _Failure(403, "fileNotDownloadable") + wanted = re.fullmatch(r"bytes=(\d+)-(\d+)", headers["range"]) + assert wanted is not None + if not data: + return httplib2.Response({"status": "416", "content-range": "bytes */0"}), b"" + start = int(wanted.group(1)) + chunk = data[start : int(wanted.group(2)) + 1] + span = f"bytes {start}-{start + len(chunk) - 1}/{len(data)}" + return httplib2.Response({"status": "206", "content-range": span}), chunk + + def _delete(self, file_id: str, params: dict[str, str], headers: dict[str, str]) -> Any: + self.seen.append(_Seen("delete", params, headers)) + doomed = [self._entry(file_id).file_id] + if doomed == [MY_DRIVE_ID]: + raise _Failure(403, "insufficientFilePermissions") + for parent in doomed: + doomed.extend(key for key, entry in self.entries.items() if entry.parent == parent) + for key in doomed: + del self.entries[key] + return httplib2.Response({"status": "204"}), b"" + + # ------------------------------------------------------------------ files.create / update / copy + + def _create(self, params: dict[str, str], metadata: dict[str, Any], headers: Any) -> Any: + self.seen.append(_Seen("create", params, headers, metadata)) + # Without media the adapter only ever creates folders. + assert metadata["mimeType"] == FOLDER_MIME_TYPE + (parent,) = metadata["parents"] + entry = self._new(metadata["name"], FOLDER_MIME_TYPE, self._folder(parent).file_id, None) + return _answer(self._projected(entry, params["fields"])) + + def _update(self, file_id: str, params: dict[str, str], metadata: Any, headers: Any) -> Any: + self.seen.append(_Seen("update", params, headers, metadata)) + entry = self._entry(file_id) + if "addParents" in params: + # Stricter than Drive on purpose: real IDs only, so the adapter may not lean on + # the "root" alias being accepted here. + if params.get("removeParents") != entry.parent or params["addParents"] == "root": + raise _Failure(400, "invalid") + entry.parent = self._folder(params["addParents"]).file_id + else: + assert "removeParents" not in params + entry.name = metadata.get("name", entry.name) + entry.version += 1 + return _answer(self._projected(entry, params["fields"])) + + def _copy(self, file_id: str, params: dict[str, str], metadata: Any, headers: Any) -> Any: + self.seen.append(_Seen("copy", params, headers, metadata)) + source = self._entry(file_id) + (parent,) = metadata["parents"] + copied = self._new( + metadata["name"], source.mime_type, self._folder(parent).file_id, source.data + ) + return _answer(self._projected(copied, params["fields"])) + + # ------------------------------------------------------------------ resumable upload + + def _open_upload( + self, method: str, file_id: str, params: dict[str, str], body: Any, headers: dict[str, str] + ) -> Any: + metadata = json.loads(body) if body else {} + self.seen.append(_Seen("update" if file_id else "create", params, headers, metadata)) + assert params["uploadType"] == "resumable" + assert method == ("PATCH" if file_id else "POST") + if file_id: + self._entry(file_id) + else: + (parent,) = metadata["parents"] + self._folder(parent) + token = f"{UPLOAD_URL}{next(self._ids)}" + self._uploads[token] = _Upload( + file_id or None, + metadata, + headers["x-upload-content-type"], + int(headers["x-upload-content-length"]), + params["fields"], + ) + return httplib2.Response({"status": "200", "location": token}), b"" + + def _receive(self, uri: str, body: Any, headers: dict[str, str]) -> Any: + self.seen.append(_Seen("upload", {}, headers)) + upload = self._uploads[uri] + data = body.read() if hasattr(body, "read") else (body or b"") + assert int(headers["content-length"]) == len(data) + if data: + first = len(upload.received) + span = f"bytes {first}-{first + len(data) - 1}/{upload.size}" + assert headers["content-range"] == span + else: + # googleapiclient sends an empty file without a Content-Range. + assert "content-range" not in headers + upload.received.extend(data) + if len(upload.received) < upload.size: + assert len(data) % UPLOAD_CHUNK_UNIT == 0, "a chunk must be a multiple of 256 KiB" + last = len(upload.received) - 1 + return httplib2.Response({"status": "308", "range": f"bytes=0-{last}"}), b"" + del self._uploads[uri] + return _answer(self._projected(self._stored(upload), upload.fields)) + + def _stored(self, upload: _Upload) -> _Entry: + content = bytes(upload.received) + if upload.file_id is None: + (parent,) = upload.metadata["parents"] + mime_type = upload.metadata.get("mimeType", upload.mime_type) + return self._new( + upload.metadata["name"], mime_type, self._folder(parent).file_id, content + ) + entry = self._entry(upload.file_id) + entry.data = content + entry.mime_type = upload.mime_type + entry.version += 1 + entry.modified = datetime.now(timezone.utc) + return entry diff --git a/tests/ftp_stand_in.py b/tests/ftp_stand_in.py new file mode 100644 index 0000000..e8a9cd0 --- /dev/null +++ b/tests/ftp_stand_in.py @@ -0,0 +1,380 @@ +"""An FTP server held in memory, and an ``ftplib.FTP`` connected to it without sockets. + +:class:`FakeFTP` is a real ``ftplib.FTP`` whose control and data connections lead +to a :class:`FakeFTPServer`. ftplib itself therefore builds every command, parses +every reply and raises its own exceptions; the server only answers as an FTP +server does. Options select the behaviours real servers differ in: ``MLST`` / +``MLSD`` or not, ``NLST`` answering with names or with paths, an empty directory +answered with 550, and a rename that will not replace an existing file. +""" + +from __future__ import annotations + +import ftplib # nosec B402 - the sessions built here lead to an in-memory server +import io +import posixpath +from collections import deque +from collections.abc import Callable +from dataclasses import dataclass +from datetime import datetime, timezone +from typing import Any + +MODIFIED = datetime(2026, 10, 8, 2, 30, 15, 250000, tzinfo=timezone.utc) +NO_SUCH_FILE = "550 No such file or directory." + + +@dataclass +class _Entry: + """A directory (neither data nor target), a file (data) or a symbolic link (target).""" + + data: bytes | None = None + target: str | None = None + modified: datetime = MODIFIED + + @property + def is_dir(self) -> bool: + return self.data is None and self.target is None + + @property + def is_file(self) -> bool: + return self.data is not None + + +class _DataConnection: + """What ``ftplib`` gets from ``transfercmd``: a socket that is read or written, then closed.""" + + def __init__(self, payload: bytes, on_close: Callable[[bytes], None]) -> None: + self._outgoing = io.BytesIO(payload) + self._incoming = bytearray() + self._on_close = on_close + + def recv(self, size: int) -> bytes: + return self._outgoing.read(size) + + def sendall(self, data: bytes) -> None: + self._incoming += data + + def makefile(self, mode: str, encoding: str | None = None) -> io.TextIOWrapper: + return io.TextIOWrapper(io.BytesIO(self._outgoing.read()), encoding=encoding) + + def close(self) -> None: + self._on_close(bytes(self._incoming)) + + def __enter__(self) -> _DataConnection: + return self + + def __exit__(self, *exc_info: object) -> None: + self.close() + + +class FakeFTPServer: + """An FTP server in memory: a tree of entries and the far end of one control connection.""" + + def __init__( + self, + *, + mlst: bool = True, + rename_replaces: bool = True, + nlst_paths: bool = False, + empty_nlst_refused: bool = False, + ) -> None: + self.entries: dict[str, _Entry] = {"/": _Entry()} + self.cwd = "/" + self.commands: list[str] = [] + self.refusals: dict[str, str] = {} + self.transfer_error: str | None = None + self.send_error: Exception | None = None + self.hung_up = False + self.extra_listing: list[str] = [] + self.raw_listing = b"" + self._offers_mlst = mlst + self._rename_replaces = rename_replaces + self._nlst_paths = nlst_paths + self._empty_nlst_refused = empty_nlst_refused + self._binary = False + self._rename_from: str | None = None + self._replies: deque[str] = deque() + self._scripted: dict[tuple[str, int], str] = {} + self._data: _DataConnection | None = None + + # ------------------------------------------------------------------ the tree + + def mkdir(self, path: str) -> None: + self.entries[path] = _Entry() + + def write(self, path: str, data: bytes) -> None: + self.entries[path] = _Entry(data=data) + + def link(self, path: str, target: str) -> None: + self.entries[path] = _Entry(target=target) + + def read(self, path: str) -> bytes: + data = self.entries[path].data + assert data is not None + return data + + def paths(self) -> list[str]: + return sorted(key for key in self.entries if key != "/") + + def verbs(self) -> list[str]: + return [command.partition(" ")[0] for command in self.commands] + + def answer_the_nth(self, verb: str, nth: int, reply: str) -> None: + """Answer the ``nth`` command ``verb`` with ``reply`` instead of carrying it out.""" + self._scripted[(verb, nth)] = reply + + def _path(self, argument: str) -> str: + return posixpath.normpath(posixpath.join(self.cwd, argument)) + + def _real(self, path: str, *, follow: bool = True) -> str: + """Resolve the links in ``path`` as a server does; the last one only when ``follow``.""" + parts = [part for part in path.split("/") if part] + current = "/" + for index, part in enumerate(parts): + current = posixpath.join(current, part) + entry = self.entries.get(current) + linked = entry is not None and entry.target is not None + if linked and (follow or index < len(parts) - 1): + current = self._real(posixpath.join(posixpath.dirname(current), entry.target)) + return current + + def _found(self, argument: str, *, follow: bool = True) -> tuple[str, _Entry | None]: + """Return where the argument of a command leads and the entry there, if any.""" + real = self._real(self._path(argument), follow=follow) + return real, self.entries.get(real) + + def _holds(self, real: str) -> bool: + """Say whether an entry can be made at ``real``: its parent is a directory.""" + parent = self.entries.get(posixpath.dirname(real)) + return parent is not None and parent.is_dir + + def _children(self, real: str) -> list[str]: + return sorted( + posixpath.basename(key) + for key in self.entries + if key != "/" and posixpath.dirname(key) == real + ) + + def _facts(self, entry: _Entry, kind: str | None = None) -> str: + modify = f"{entry.modified:%Y%m%d%H%M%S}.{entry.modified.microsecond // 1000:03d}" + if entry.target is not None: + return f"Type=OS.unix=slink:{entry.target};size={len(entry.target)};Modify={modify};" + if entry.is_dir: + return f"Type={kind or 'dir'};sizd=4096;Modify={modify};" + return f"Type=file;size={len(entry.data or b'')};Modify={modify};" + + # ------------------------------------------------------------------ the control connection + + def sendall(self, data: bytes) -> None: + """Receive one command line from the client, as the control socket would.""" + if self.send_error is not None: + raise self.send_error + line = data.decode("utf-8") + assert line.endswith("\r\n") + self.commands.append(line[:-2]) + if self.hung_up: + return + verb, _, argument = line[:-2].partition(" ") + handler = getattr(self, f"_do_{verb.lower()}", None) + scripted = self._scripted.pop((verb, self.verbs().count(verb)), None) + if scripted is not None or verb in self.refusals: + self._reply(scripted or self.refusals[verb]) + elif handler is None: + self._reply("500 Unknown command.") + else: + handler(argument) + + def readline(self, limit: int) -> str: + """Hand the client the next reply line; an empty one is how a closed connection reads.""" + return self._replies.popleft() if self._replies else "" + + def close(self) -> None: + self.hung_up = True + + def data_connection(self) -> _DataConnection: + assert self._data is not None + connection, self._data = self._data, None + return connection + + def _reply(self, text: str) -> None: + self._replies.extend(f"{line}\r\n" for line in text.split("\n")) + + def _transfer(self, payload: bytes, done: Callable[[bytes], None]) -> None: + self._data = _DataConnection(payload, done) + self._reply("150 Opening data connection.") + + def _send(self, payload: bytes) -> None: + self._transfer( + payload, lambda _: self._reply(self.transfer_error or "226 Transfer complete.") + ) + + def _send_lines(self, lines: list[str]) -> None: + self._send("".join(f"{line}\r\n" for line in lines).encode("utf-8") + self.raw_listing) + + # ------------------------------------------------------------------ the commands + + def _do_pwd(self, _: str) -> None: + self._reply(f'257 "{self.cwd}" is the current directory') + + def _do_cwd(self, argument: str) -> None: + entry = self._found(argument)[1] + if entry is None or not entry.is_dir: + self._reply("550 Failed to change directory.") + return + self.cwd = self._path(argument) + self._reply("250 Directory successfully changed.") + + def _do_type(self, argument: str) -> None: + self._binary = argument == "I" + self._reply(f"200 Type set to {argument}.") + + def _do_feat(self, _: str) -> None: + features = [" MDTM", " SIZE", " UTF8"] + if self._offers_mlst: + features.insert(1, " MLST type*;size*;sizd*;modify*;perm;") + self._reply("\n".join(["211-Features:", *features, "211 End"])) + + def _do_opts(self, argument: str) -> None: + if self._offers_mlst and argument.upper().startswith("MLST"): + self._reply(f"200 {argument}") + else: + self._reply("501 Option not understood.") + + def _do_mlst(self, argument: str) -> None: + path = self._path(argument) + entry = self._found(argument, follow=False)[1] + if not self._offers_mlst: + self._reply("500 Unknown command.") + elif entry is None: + self._reply(NO_SUCH_FILE) + else: + listed = f" {self._facts(entry)} {path}" + self._reply("\n".join([f"250-Listing {path}", listed, "250 End"])) + + def _do_mlsd(self, argument: str) -> None: + real, entry = self._found(argument) + if not self._offers_mlst: + self._reply("500 Unknown command.") + elif entry is None or not entry.is_dir: + self._reply(NO_SUCH_FILE) + else: + lines = [f"{self._facts(entry, 'cdir')} .", f"{self._facts(entry, 'pdir')} .."] + lines += [ + f"{self._facts(self.entries[posixpath.join(real, name)])} {name}" + for name in self._children(real) + ] + self._send_lines(lines + self.extra_listing) + + def _do_nlst(self, argument: str) -> None: + real, entry = self._found(argument) + names = self._children(real) + if entry is None or not entry.is_dir or (self._empty_nlst_refused and not names): + self._reply("550 No files found.") + return + self._binary = False + shown = self._path(argument) + self._send_lines( + [posixpath.join(shown, name) if self._nlst_paths else name for name in names] + ) + + def _do_size(self, argument: str) -> None: + entry = self._found(argument)[1] + if not self._binary: + self._reply("550 SIZE not allowed in ASCII mode") + elif entry is None or entry.data is None: + self._reply("550 Could not get file size.") + else: + self._reply(f"213 {len(entry.data)}") + + def _do_mdtm(self, argument: str) -> None: + entry = self._found(argument)[1] + if entry is None or not entry.is_file: + self._reply("550 Could not get file modification time.") + else: + self._reply(f"213 {entry.modified:%Y%m%d%H%M%S}") + + def _do_retr(self, argument: str) -> None: + entry = self._found(argument)[1] + if entry is None or entry.data is None: + self._reply("550 Failed to open file.") + else: + self._send(entry.data) + + def _do_stor(self, argument: str) -> None: + real, entry = self._found(argument) + if not self._holds(real) or (entry is not None and entry.is_dir): + self._reply("553 Could not create file.") + return + + def stored(data: bytes) -> None: + kept = data if self.transfer_error is None else data[: len(data) // 2] + self.entries[real] = _Entry(data=kept, modified=datetime.now(timezone.utc)) + self._reply(self.transfer_error or "226 Transfer complete.") + + self._transfer(b"", stored) + + def _do_dele(self, argument: str) -> None: + real, entry = self._found(argument, follow=False) + if entry is None or entry.is_dir: + self._reply("550 Delete operation failed.") + return + del self.entries[real] + self._reply("250 Delete operation successful.") + + def _do_mkd(self, argument: str) -> None: + real, entry = self._found(argument, follow=False) + if entry is not None or not self._holds(real): + self._reply("550 Create directory operation failed.") + return + self.entries[real] = _Entry(modified=datetime.now(timezone.utc)) + self._reply(f'257 "{real}" created') + + def _do_rmd(self, argument: str) -> None: + real, entry = self._found(argument, follow=False) + if entry is None or not entry.is_dir or self._children(real) or real == "/": + self._reply("550 Remove directory operation failed.") + return + del self.entries[real] + self._reply("250 Remove directory operation successful.") + + def _do_rnfr(self, argument: str) -> None: + real, entry = self._found(argument, follow=False) + if entry is None: + self._reply("550 RNFR command failed.") + return + self._rename_from = real + self._reply("350 Ready for RNTO.") + + def _do_rnto(self, argument: str) -> None: + origin, self._rename_from = self._rename_from, None + target, existing = self._found(argument, follow=False) + if origin is None: + self._reply("503 RNFR required first.") + elif not self._holds(target) or (existing is not None and existing.is_dir): + self._reply("550 Rename failed.") + elif existing is not None and not self._rename_replaces: + self._reply("550 Cannot create a file when that file already exists.") + else: + self.entries[target] = self.entries.pop(origin) + self._reply("250 Rename successful.") + + +class FakeFTP(ftplib.FTP): + """A real ``ftplib.FTP`` connected to a :class:`FakeFTPServer` instead of to sockets.""" + + def __init__(self, server: FakeFTPServer) -> None: + super().__init__() + self.server = server + self.sock = server + self.file = server + + def ntransfercmd(self, cmd: str, rest: Any = None) -> tuple[Any, None]: + """Open the data connection: where ftplib would dial the server's passive port.""" + reply = self.sendcmd(cmd) + if reply[0] != "1": + raise ftplib.error_reply(reply) + return self.server.data_connection(), None + + +class FakeFTPS(FakeFTP, ftplib.FTP_TLS): + """The same stand-in as an ``FTP_TLS`` session, which is what marks a session as FTPS.""" diff --git a/tests/graph_stand_in.py b/tests/graph_stand_in.py new file mode 100644 index 0000000..839794a --- /dev/null +++ b/tests/graph_stand_in.py @@ -0,0 +1,418 @@ +"""An in-memory OneDrive for tests, served to a real ``requests.Session``. + +:class:`FakeGraph` is a ``requests`` transport adapter. Mount it on the session of +a real ``OneDriveClient`` and every request is prepared by ``requests`` itself -- +the merged headers, the encoded URL, the body and its length, the redirect of a +download -- before the stand-in answers it the way Microsoft Graph documents its +``driveItem`` API. No request leaves the process. + +What the stand-in answers is a reading of that documentation, not a recording of +the service, and nothing installed describes Graph. It assumes that names are +compared without regard to case, that a path through a file is a 404, that an +upload creates the folders missing on its path, that creating a folder whose name +is taken is a 409, that ``/content`` redirects to a pre-authenticated URL, that +an upload session answers 202 until its last fragment, and that deleting a folder +deletes what is in it. +""" + +from __future__ import annotations + +import io +import itertools +import json +import mimetypes +from dataclasses import dataclass, field +from datetime import datetime, timezone +from typing import Any +from urllib.parse import parse_qsl, quote, unquote, urlsplit + +import requests +from requests.adapters import BaseAdapter +from requests.structures import CaseInsensitiveDict + +GRAPH_ORIGIN = "https://graph.microsoft.com" +DRIVE_ROOT = "/v1.0/me/drive/root" +UPLOAD_ORIGIN = "https://upload.onedrive.invalid" +DOWNLOAD_ORIGIN = "https://download.onedrive.invalid" +# Not credentials: markers the stand-in hands out and the tests look for in messages. +FAKE_TOKEN = "fake-token" # nosec B105 +URL_SECRET = "tempauth=fake-url-secret" # nosec B105 +ROOT_ID = "ROOT" +DRIVE_ID = "fake-drive" +LIST_PAGE = 2 +KIB = 1024 +FRAGMENT_UNIT = 320 * KIB +FRAGMENT_LIMIT = 60 * KIB * KIB +SIMPLE_UPLOAD_LIMIT = 4 * KIB * KIB +FORBIDDEN_IN_NAMES = frozenset('"*:<>?/\\|') +ERROR_CODES = { + 400: "invalidRequest", + 401: "unauthenticated", + 403: "accessDenied", + 404: "itemNotFound", + 408: "requestTimeout", + 409: "nameAlreadyExists", + 429: "activityLimitReached", + 500: "generalException", + 503: "serviceNotAvailable", + 504: "gatewayTimeout", + 507: "quotaLimitReached", +} +CALLS = { + (GRAPH_ORIGIN, "GET", ""): "item", + (GRAPH_ORIGIN, "PATCH", ""): "patch", + (GRAPH_ORIGIN, "DELETE", ""): "delete", + (GRAPH_ORIGIN, "GET", "/children"): "children", + (GRAPH_ORIGIN, "POST", "/children"): "mkdir", + (GRAPH_ORIGIN, "PUT", "/content"): "put", + (GRAPH_ORIGIN, "GET", "/content"): "content", + (GRAPH_ORIGIN, "POST", "/createUploadSession"): "session", + (UPLOAD_ORIGIN, "PUT", ""): "fragment", + (UPLOAD_ORIGIN, "DELETE", ""): "cancel", + (DOWNLOAD_ORIGIN, "GET", ""): "download", +} + + +class _Failure(Exception): + """An error answer of the stand-in.""" + + def __init__(self, status: int) -> None: + super().__init__(str(status)) + self.status = status + + +@dataclass +class _Item: + item_id: str + name: str + parent: str | None + data: bytes | None = None + revision: int = 1 + modified: datetime = field(default_factory=lambda: datetime.now(timezone.utc)) + + def resource(self, child_count: int) -> dict[str, Any]: + stamp = self.modified.strftime("%Y-%m-%dT%H:%M:%S") + resource: dict[str, Any] = { + "id": self.item_id, + "name": self.name, + "eTag": f'"{{{self.item_id}}},{self.revision}"', + # Seven fractional digits, as Graph sometimes sends. + "lastModifiedDateTime": f"{stamp}.{self.modified.microsecond:06d}0Z", + "parentReference": {"driveId": DRIVE_ID, "id": self.parent}, + "size": len(self.data or b""), + } + if self.data is None: + resource["folder"] = {"childCount": child_count} + else: + # OneDrive derives the type from the name, not from the upload's Content-Type. + mime_type = mimetypes.guess_type(self.name)[0] or "application/octet-stream" + resource["file"] = {"mimeType": mime_type} + return resource + + +@dataclass +class _Seen: + call: str + url: str + headers: dict[str, str] + stream: bool + body: bytes + + +def graph_answer( + request: requests.PreparedRequest, + status: int, + document: Any = None, + *, + content: bytes = b"", + location: str | None = None, +) -> requests.Response: + response = requests.Response() + response.status_code = status + response.request = request + response.url = request.url or "" + body = content if document is None else json.dumps(document).encode("utf-8") + response.headers = CaseInsensitiveDict({"Content-Length": str(len(body))}) + if document is not None: + response.headers["Content-Type"] = "application/json" + if location is not None: + response.headers["Location"] = location + response.raw = io.BytesIO(body) + return response + + +def _error(request: requests.PreparedRequest, status: int) -> requests.Response: + code = ERROR_CODES.get(status, "generalException") + return graph_answer(request, status, {"error": {"code": code, "message": f"stand-in: {code}"}}) + + +def _address(origin: str, raw_path: str) -> tuple[str, str]: + """Split the path of a URL into the drive path it names and the facet asked for.""" + if origin != GRAPH_ORIGIN: + return raw_path, "" + assert raw_path.startswith(DRIVE_ROOT), f"unexpected Graph path {raw_path}" + rest = raw_path.removeprefix(DRIVE_ROOT) + if not rest.startswith(":"): + return "", rest + # A colon inside a name would arrive percent-encoded, so these colons are the delimiters. + item, _, facet = rest[1:].partition(":") + return unquote(item).strip("/"), facet + + +class FakeGraph(BaseAdapter): + """The part of Microsoft Graph that OneDriveStorage reaches, as a requests transport. + + Like OneDrive, it compares names without regard to case, pages its listings, + returns only the selected properties, redirects a download to a + pre-authenticated URL, takes a large file through an upload session and + removes a folder together with everything below it. + """ + + def __init__(self) -> None: + super().__init__() + self.items: dict[str, _Item] = {ROOT_ID: _Item(ROOT_ID, "root", None)} + self.seen: list[_Seen] = [] + self.fail_with: int | Exception | None = None + self.fail_calls: frozenset[str] | None = None + self.sessions: dict[str, tuple[str, bytearray]] = {} + self._ids = itertools.count(1) + self._downloads: dict[str, bytes] = {} + + # ------------------------------------------------------------------ for the tests + + @property + def calls(self) -> list[str]: + return [seen.call for seen in self.seen] + + def all(self, call: str) -> list[_Seen]: + return [seen for seen in self.seen if seen.call == call] + + def find(self, path: str) -> _Item | None: + item = self.items[ROOT_ID] + for segment in filter(None, path.split("/")): + wanted = segment.casefold() + named = [child for child in self._children(item) if child.name.casefold() == wanted] + if not named: + return None + (item,) = named + return item + + def at(self, path: str) -> _Item: + item = self.find(path) + assert item is not None, f"nothing at {path}" + return item + + def add_file(self, path: str, data: bytes) -> _Item: + return self._store(path, data)[0] + + def add_folder(self, path: str) -> _Item: + return self._folder(path) + + def paths(self) -> list[str]: + found: list[str] = [] + pending = [("", self.items[ROOT_ID])] + while pending: + base, folder = pending.pop() + for child in self._children(folder): + path = f"{base}/{child.name}" if base else child.name + found.append(path) + if child.data is None: + pending.append((path, child)) + return sorted(found) + + # ------------------------------------------------------------------ requests transport + + def send( + self, + request: requests.PreparedRequest, + stream: bool = False, + timeout: Any = None, + verify: Any = True, + cert: Any = None, + proxies: Any = None, + ) -> requests.Response: + del cert, proxies + assert verify is not False, "TLS verification was switched off" + assert timeout is not None, "a request without a timeout" + assert "Transfer-Encoding" not in request.headers, "a body without a length" + url = request.url or "" + split = urlsplit(url) + origin = f"{split.scheme}://{split.netloc}" + if origin == GRAPH_ORIGIN: + assert request.headers.get("Authorization") == f"Bearer {FAKE_TOKEN}" + else: + assert "Authorization" not in request.headers, f"the bearer token reached {origin}" + path, facet = _address(origin, split.path) + call = CALLS[origin, request.method or "", facet] + body = request.body or b"" + assert isinstance(body, bytes) + self.seen.append(_Seen(call, url, dict(request.headers), stream, body)) + if self.fail_with is not None and (self.fail_calls is None or call in self.fail_calls): + if isinstance(self.fail_with, Exception): + raise self.fail_with + return _error(request, self.fail_with) + try: + handler = getattr(self, f"_on_{call}") + return handler(request, path, dict(parse_qsl(split.query)), body) + except _Failure as failure: + return _error(request, failure.status) + + def close(self) -> None: + return None + + # ------------------------------------------------------------------ the drive + + def _children(self, folder: _Item) -> list[_Item]: + return [item for item in self.items.values() if item.parent == folder.item_id] + + def _existing(self, path: str) -> _Item: + item = self.find(path) + if item is None: + raise _Failure(404) + return item + + def _resource(self, item: _Item, params: dict[str, str]) -> dict[str, Any]: + resource = item.resource(len(self._children(item))) + selected = params.get("$select") + if selected is None: + return resource + return {key: value for key, value in resource.items() if key in selected.split(",")} + + def _new(self, name: str, parent: _Item, data: bytes | None) -> _Item: + if not name or FORBIDDEN_IN_NAMES & set(name): + raise _Failure(400) + if parent.data is not None: + raise _Failure(404) + item = _Item(f"ITEM{next(self._ids):04d}", name, parent.item_id, data) + self.items[item.item_id] = item + return item + + def _folder(self, path: str) -> _Item: + """Return the folder at ``path``, creating what is missing, as an upload does.""" + folder = self.items[ROOT_ID] + walked = "" + for segment in filter(None, path.split("/")): + walked = f"{walked}/{segment}" + folder = self.find(walked) or self._new(segment, folder, None) + return folder + + def _store(self, path: str, data: bytes) -> tuple[_Item, bool]: + """Create or replace the file at ``path``; say whether it is new.""" + existing = self.find(path) + if existing is None: + directory, _, name = path.rpartition("/") + return self._new(name, self._folder(directory), data), True + if existing.data is None: + raise _Failure(409) + existing.data = data + existing.revision += 1 + existing.modified = datetime.now(timezone.utc) + return existing, False + + # ------------------------------------------------------------------ Graph calls + + def _on_item(self, request: Any, path: str, params: dict[str, str], body: bytes) -> Any: + del body + return graph_answer(request, 200, self._resource(self._existing(path), params)) + + def _on_children(self, request: Any, path: str, params: dict[str, str], body: bytes) -> Any: + del body + children = self._children(self._existing(path)) + start = int(params.get("$skiptoken", "0")) + end = start + min(int(params.get("$top", LIST_PAGE)), LIST_PAGE) + page: dict[str, Any] = { + "value": [self._resource(child, params) for child in children[start:end]] + } + if end < len(children): + following = {**params, "$skiptoken": str(end)} + query = "&".join(f"{key}={quote(value, safe='')}" for key, value in following.items()) + page["@odata.nextLink"] = f"{request.url.partition('?')[0]}?{query}" + return graph_answer(request, 200, page) + + def _on_mkdir(self, request: Any, path: str, params: dict[str, str], body: bytes) -> Any: + wanted = json.loads(body) + assert wanted["folder"] == {} + assert wanted["@microsoft.graph.conflictBehavior"] == "fail" + parent = self._existing(path) + folded = wanted["name"].casefold() + if any(child.name.casefold() == folded for child in self._children(parent)): + raise _Failure(409) + created = self._new(wanted["name"], parent, None) + return graph_answer(request, 201, self._resource(created, params)) + + def _on_put(self, request: Any, path: str, params: dict[str, str], body: bytes) -> Any: + assert len(body) <= SIMPLE_UPLOAD_LIMIT, "too large for a simple upload" + assert int(request.headers["Content-Length"]) == len(body) + item, created = self._store(path, body) + return graph_answer(request, 201 if created else 200, self._resource(item, params)) + + def _on_content(self, request: Any, path: str, params: dict[str, str], body: bytes) -> Any: + del params, body + data = self._existing(path).data + if data is None: + raise _Failure(400) + ticket = f"{DOWNLOAD_ORIGIN}/content/{next(self._ids)}?{URL_SECRET}" + self._downloads[ticket] = data + return graph_answer(request, 302, location=ticket) + + def _on_download(self, request: Any, path: str, params: dict[str, str], body: bytes) -> Any: + del path, params, body + return graph_answer(request, 200, content=self._downloads.pop(request.url)) + + def _on_session(self, request: Any, path: str, params: dict[str, str], body: bytes) -> Any: + del params + assert json.loads(body) == {"item": {"@microsoft.graph.conflictBehavior": "replace"}} + existing = self.find(path) + if existing is not None and existing.data is None: + raise _Failure(409) + upload_url = f"{UPLOAD_ORIGIN}/session/{next(self._ids)}?{URL_SECRET}" + self.sessions[upload_url] = (path, bytearray()) + return graph_answer(request, 200, {"uploadUrl": upload_url, "nextExpectedRanges": ["0-"]}) + + def _on_fragment(self, request: Any, path: str, params: dict[str, str], body: bytes) -> Any: + del path, params + target, received = self.sessions[request.url] + span, _, total = request.headers["Content-Range"].removeprefix("bytes ").partition("/") + first, _, last = span.partition("-") + assert int(first) == len(received), "fragments out of order" + assert int(last) - int(first) + 1 == len(body) == int(request.headers["Content-Length"]) + assert 0 < len(body) <= FRAGMENT_LIMIT + received.extend(body) + if len(received) < int(total): + assert len(body) % FRAGMENT_UNIT == 0, "a fragment must be a multiple of 320 KiB" + expected = f"{len(received)}-{int(total) - 1}" + return graph_answer(request, 202, {"nextExpectedRanges": [expected]}) + del self.sessions[request.url] + item, created = self._store(target, bytes(received)) + return graph_answer(request, 201 if created else 200, self._resource(item, {})) + + def _on_cancel(self, request: Any, path: str, params: dict[str, str], body: bytes) -> Any: + del path, params, body + del self.sessions[request.url] + return graph_answer(request, 204) + + def _on_patch(self, request: Any, path: str, params: dict[str, str], body: bytes) -> Any: + item = self._existing(path) + wanted = json.loads(body) + name = wanted["name"] + parent = self.items.get(wanted["parentReference"]["id"]) + if parent is None or parent.data is not None: + raise _Failure(404) + if FORBIDDEN_IN_NAMES & set(name): + raise _Failure(400) + siblings = [other for other in self._children(parent) if other is not item] + if any(other.name.casefold() == name.casefold() for other in siblings): + raise _Failure(409) + item.name, item.parent = name, parent.item_id + return graph_answer(request, 200, self._resource(item, params)) + + def _on_delete(self, request: Any, path: str, params: dict[str, str], body: bytes) -> Any: + del params, body + doomed = [self._existing(path)] + if doomed[0].item_id == ROOT_ID: + raise _Failure(400) + for item in doomed: + doomed.extend(self._children(item)) + for item in doomed: + del self.items[item.item_id] + return graph_answer(request, 204) diff --git a/tests/test_backends.py b/tests/test_backends.py index c5f5134..24dc28f 100644 --- a/tests/test_backends.py +++ b/tests/test_backends.py @@ -10,6 +10,8 @@ from __future__ import annotations import importlib +from pathlib import Path +from types import SimpleNamespace import pytest @@ -88,6 +90,51 @@ def test_register_sftp_ops_adds_entries() -> None: assert "FA_sftp_upload_file" in registry +def test_sftp_client_reports_the_host_and_port_of_its_session( + monkeypatch: pytest.MonkeyPatch, tmp_path: Path +) -> None: + from automation_file.remote.sftp import client as client_module + + class _StubSSH: + """What ``later_init`` drives on ``paramiko.SSHClient``.""" + + def __init__(self) -> None: + self.connected_to: tuple[str, int] | None = None + + def load_host_keys(self, filename: str) -> None: + return None + + def set_missing_host_key_policy(self, policy: object) -> None: + return None + + def connect(self, **options: object) -> None: + self.connected_to = (str(options["hostname"]), int(str(options["port"]))) + + def open_sftp(self) -> _StubSSH: + return self + + def close(self) -> None: + self.connected_to = None + + stub = SimpleNamespace(SSHClient=_StubSSH, RejectPolicy=object) + monkeypatch.setattr(client_module, "_import_paramiko", lambda: stub) + known_hosts = tmp_path / "known_hosts" + known_hosts.write_text("", encoding="utf-8") + client = client_module.SFTPClient() + assert (client.host, client.port) == (None, None) + session = client.later_init( + host="nas.example", port=2222, username="ops", known_hosts=str(known_hosts) + ) + assert session.connected_to == ("nas.example", 2222) + assert (client.host, client.port) == ("nas.example", 2222) + assert set(vars(client)) == {"_ssh", "_sftp", "_host", "_port"} + for name in ("host", "port"): + with pytest.raises(AttributeError): + setattr(client, name, None) + client.close() + assert (client.host, client.port) == (None, None) + + def test_register_onedrive_ops_adds_entries() -> None: from automation_file.core.action_registry import ActionRegistry from automation_file.remote.onedrive import register_onedrive_ops diff --git a/tests/test_ftp_ops.py b/tests/test_ftp_ops.py index 15658b2..5ac8482 100644 --- a/tests/test_ftp_ops.py +++ b/tests/test_ftp_ops.py @@ -136,3 +136,55 @@ def test_list_dir_returns_names(fake_ftp: _FakeFTP) -> None: def test_close_on_fresh_client_is_noop() -> None: client = FTPClient() assert client.close() is True + + +class _StubSession: + """What ``later_init`` drives: connect, log in, choose the transfer mode, quit.""" + + def __init__(self, timeout: float | None = None) -> None: + self.steps: list[str] = [] + + def connect(self, host: str, port: int, timeout: float | None = None) -> None: + self.steps.append(f"connect {host}:{port}") + + def auth(self) -> None: + self.steps.append("auth") + + def login(self, user: str = "", passwd: str = "") -> None: + self.steps.append("login") + + def prot_p(self) -> None: + self.steps.append("prot_p") + + def set_pasv(self, value: bool) -> None: + self.steps.append("set_pasv") + + def quit(self) -> None: + self.steps.append("quit") + + +class _StubTLSSession(_StubSession): + """Stands in for ``FTP_TLS``, the session type that marks a session as FTPS.""" + + +@pytest.mark.parametrize("tls", [False, True]) +def test_client_reports_the_host_port_and_tls_of_its_session( + monkeypatch: pytest.MonkeyPatch, tls: bool +) -> None: + from automation_file.remote.ftp import client as client_module + + monkeypatch.setattr(client_module, "FTP", _StubSession) + monkeypatch.setattr(client_module, "FTP_TLS", _StubTLSSession) + client = FTPClient() + assert (client.host, client.port, client.tls) == (None, None, False) + session = client.later_init(host="files.example", port=2121, username="ops", tls=tls) + assert (client.host, client.port, client.tls) == ("files.example", 2121, tls) + assert session.steps[0] == "connect files.example:2121" + assert ("prot_p" in session.steps) is tls + assert set(vars(client)) == {"_ftp", "_host", "_port"} + for name in ("host", "port", "tls"): + with pytest.raises(AttributeError): + setattr(client, name, None) + client.close() + assert (client.host, client.port, client.tls) == (None, None, False) + assert session.steps[-1] == "quit" diff --git a/tests/test_optional_dependencies.py b/tests/test_optional_dependencies.py index b703f3f..fb5ac68 100644 --- a/tests/test_optional_dependencies.py +++ b/tests/test_optional_dependencies.py @@ -66,7 +66,7 @@ } _PROBE = """ -import importlib.abc, json, sys +import importlib, importlib.abc, json, sys blocked = set(json.loads(sys.argv[1])) @@ -81,7 +81,12 @@ def find_spec(self, fullname, path=None, target=None): if blocked: sys.meta_path.insert(0, Blocker()) import automation_file -from automation_file.exceptions import OptionalDependencyException +from automation_file.exceptions import FileAutomationException, OptionalDependencyException + + +def client(backend): + return importlib.import_module("automation_file.remote." + backend + ".client") + # What the import itself pulled in, measured before any feature is used. loaded = sorted({name.split(".")[0] for name in sys.modules} & set(json.loads(sys.argv[2]))) @@ -91,12 +96,18 @@ def find_spec(self, fullname, path=None, target=None): "azure": lambda: automation_file.azure_blob_instance.later_init(connection_string="x"), "dropbox": lambda: automation_file.dropbox_instance.later_init("token"), "gdrive": lambda: automation_file.drive_search_all_file(), + "sftp": lambda: client("sftp")._import_paramiko(), + "onedrive": lambda: client("onedrive")._import_msal(), + "smb": lambda: client("smb")._import_smbclient(), } for name, call in features.items() if blocked else (): try: call() except OptionalDependencyException as error: messages[name] = str(error) + except FileAutomationException as error: + # A client with an exception type of its own keeps it, and adds the hint. + messages[name] = str(error) except Exception as error: messages[name] = "OTHER " + type(error).__name__ else: @@ -126,7 +137,7 @@ def test_the_package_imports_without_any_optional_dependency() -> None: report = _probe((*OPTIONAL_ROOTS, "google")) assert report["commands"] > 100 assert report["loaded"] == [] - for extra in ("s3", "azure", "dropbox", "gdrive"): + for extra in ("s3", "azure", "dropbox", "gdrive", "sftp", "onedrive", "smb"): assert install_hint(extra) in report["messages"][extra], report["messages"] diff --git a/tests/test_smb_client.py b/tests/test_smb_client.py index 900107b..fc772ed 100644 --- a/tests/test_smb_client.py +++ b/tests/test_smb_client.py @@ -3,9 +3,11 @@ # pylint: disable=redefined-outer-name,undefined-variable # pytest fixtures + lazy annotations from __future__ import annotations +import errno +import stat import sys from pathlib import Path -from types import ModuleType +from types import ModuleType, SimpleNamespace from unittest.mock import MagicMock import pytest @@ -26,6 +28,7 @@ def is_dir(self) -> bool: def stat(self) -> MagicMock: stat_result = MagicMock() stat_result.st_size = self._size + stat_result.st_mtime = 1_791_000_000.5 return stat_result @@ -42,10 +45,16 @@ def smbclient_module(monkeypatch: pytest.MonkeyPatch) -> ModuleType: fake.makedirs = MagicMock() # type: ignore[attr-defined] fake.rmdir = MagicMock() # type: ignore[attr-defined] fake.scandir = MagicMock() # type: ignore[attr-defined] + fake.rename = MagicMock() # type: ignore[attr-defined] + fake.replace = MagicMock() # type: ignore[attr-defined] monkeypatch.setitem(sys.modules, "smbclient", fake) return fake +class _ProtocolOSError(OSError): + """Like smbprotocol's own error: an ``OSError`` subclass, so the errno is all there is.""" + + def test_rejects_empty_server() -> None: with pytest.raises(SMBException): SMBClient("", "share") @@ -69,11 +78,87 @@ def test_exists_false_on_file_not_found(smbclient_module: ModuleType) -> None: assert client.exists("missing") is False +def test_exists_false_on_the_errno_of_a_missing_path(smbclient_module: ModuleType) -> None: + missing = _ProtocolOSError(errno.ENOENT, "No such file or directory") + assert not isinstance(missing, FileNotFoundError) + smbclient_module.stat.side_effect = missing # type: ignore[attr-defined] + client = SMBClient("fs", "pub") + assert client.exists("missing") is False + + def test_exists_wraps_os_error(smbclient_module: ModuleType) -> None: smbclient_module.stat.side_effect = OSError("boom") # type: ignore[attr-defined] client = SMBClient("fs", "pub") with pytest.raises(SMBException): client.exists("x") + denied = _ProtocolOSError(errno.EACCES, "denied") + smbclient_module.stat.side_effect = denied # type: ignore[attr-defined] + with pytest.raises(SMBException): + client.exists("x") + + +def test_stat_reports_kind_size_and_modification_time(smbclient_module: ModuleType) -> None: + smbclient_module.stat.return_value = SimpleNamespace( # type: ignore[attr-defined] + st_mode=stat.S_IFREG | 0o644, st_size=7, st_mtime=1_791_000_000.5 + ) + client = SMBClient("fs", "pub", port=4455) + entry = client.stat("folder/data.bin") + assert (entry.name, entry.is_dir, entry.size, entry.mtime) == ( + "data.bin", + False, + 7, + 1_791_000_000.5, + ) + call_args, call_kwargs = smbclient_module.stat.call_args # type: ignore[attr-defined] + assert call_args[0] == "\\\\fs\\pub\\folder\\data.bin" + assert call_kwargs == {"port": 4455} + smbclient_module.stat.return_value = SimpleNamespace( # type: ignore[attr-defined] + st_mode=stat.S_IFDIR | 0o755, st_size=0, st_mtime=1.0 + ) + folder = client.stat("folder\\") + assert (folder.name, folder.is_dir, folder.size) == ("folder", True, None) + + +def test_stat_wraps_os_error(smbclient_module: ModuleType) -> None: + missing = _ProtocolOSError(errno.ENOENT, "No such file or directory") + smbclient_module.stat.side_effect = missing # type: ignore[attr-defined] + client = SMBClient("fs", "pub") + with pytest.raises(SMBException) as caught: + client.stat("missing") + assert caught.value.__cause__ is missing + + +def test_rename_keeps_or_replaces_the_target(smbclient_module: ModuleType) -> None: + client = SMBClient("fs", "pub", port=4455) + client.rename("a.txt", "dir/b.txt") + smbclient_module.rename.assert_called_once_with( # type: ignore[attr-defined] + "\\\\fs\\pub\\a.txt", "\\\\fs\\pub\\dir\\b.txt", port=4455 + ) + assert smbclient_module.replace.call_count == 0 # type: ignore[attr-defined] + client.rename("a.txt", "dir/b.txt", overwrite=True) + smbclient_module.replace.assert_called_once_with( # type: ignore[attr-defined] + "\\\\fs\\pub\\a.txt", "\\\\fs\\pub\\dir\\b.txt", port=4455 + ) + smbclient_module.rename.side_effect = FileExistsError # type: ignore[attr-defined] + with pytest.raises(SMBException): + client.rename("a.txt", "dir/b.txt") + + +def test_every_call_names_the_port(smbclient_module: ModuleType, tmp_path: Path) -> None: + local = tmp_path / "data.bin" + local.write_bytes(b"x") + smbclient_module.scandir.return_value = iter([]) # type: ignore[attr-defined] + client = SMBClient("fs", "pub", port=4455) + client.exists("a") + client.upload(local, "a") + client.delete("a") + client.mkdir("d") + client.rmdir("d") + client.list_dir("") + for name in ("register_session", "stat", "open_file", "remove", "makedirs", "rmdir", "scandir"): + _, call_kwargs = getattr(smbclient_module, name).call_args + assert call_kwargs["port"] == 4455, name + assert (client.server, client.share) == ("fs", "pub") def test_upload_streams_file(smbclient_module: ModuleType, tmp_path: Path) -> None: @@ -168,6 +253,17 @@ def test_list_dir_returns_entries(smbclient_module: ModuleType) -> None: assert entries[1].name == "data.bin" assert entries[1].is_dir is False assert entries[1].size == 7 + assert entries[1].mtime == 1_791_000_000.5 + + +def test_list_dir_keeps_an_entry_it_cannot_stat(smbclient_module: ModuleType) -> None: + unreadable = MagicMock() + unreadable.name = "locked.bin" + unreadable.is_dir.return_value = False + unreadable.stat.side_effect = OSError("sharing violation") + smbclient_module.scandir.return_value = iter([unreadable]) # type: ignore[attr-defined] + entries = SMBClient("fs", "pub").list_dir("folder") + assert (entries[0].name, entries[0].size, entries[0].mtime) == ("locked.bin", None, None) def test_close_is_idempotent(smbclient_module: ModuleType) -> None: diff --git a/tests/test_storage_azure.py b/tests/test_storage_azure.py index b9350f1..da98798 100644 --- a/tests/test_storage_azure.py +++ b/tests/test_storage_azure.py @@ -367,3 +367,20 @@ def test_the_sdk_has_the_calls_the_adapter_makes() -> None: assert hasattr(properties, attribute) assert hasattr(properties, "version_id") assert ContentSettings(content_type="text/plain").content_type == "text/plain" + + +def test_two_prefixes_of_one_container_do_not_lose_a_blob_to_itself( + service: FakeBlobService, +) -> None: + whole = AzureStorage("container", service=service) + tenant = AzureStorage("container", service=service, prefix="tenant/a") + tenant.write_bytes("docs/a.txt", b"payload") + for operation in (whole.move_from, whole.copy_from): + with pytest.raises(StorageException, match="same file"): + operation(tenant, "docs/a.txt", "tenant/a/docs/a.txt") + with pytest.raises(StorageException, match="same file"): + tenant.move_from(whole, "tenant/a/docs/a.txt", "docs/a.txt") + assert service.containers["container"]["tenant/a/docs/a.txt"].data == b"payload" + elsewhere = AzureStorage("container", service=FakeBlobService()) + elsewhere.copy_from(tenant, "docs/a.txt", "tenant/a/docs/a.txt") + assert elsewhere.read_bytes("tenant/a/docs/a.txt") == b"payload" diff --git a/tests/test_storage_dropbox.py b/tests/test_storage_dropbox.py new file mode 100644 index 0000000..71e6aaa --- /dev/null +++ b/tests/test_storage_dropbox.py @@ -0,0 +1,734 @@ +"""DropboxStorage: the storage contract against an in-memory stand-in for ``dropbox.Dropbox``. + +The stand-in answers the calls the adapter makes -- ``files_get_metadata``, +``files_list_folder`` (paged), ``files_upload``, the upload session calls, +``files_download_to_file``, ``files_delete_v2``, ``files_create_folder_v2``, +``files_copy_v2`` and ``files_move_v2`` -- with the SDK's own data types and +raises the SDK's own exceptions. No request leaves the process. +""" + +from __future__ import annotations + +import hashlib +import inspect +import io +from dataclasses import dataclass +from datetime import datetime, timedelta, timezone +from pathlib import Path +from typing import Any + +import pytest + +pytest.importorskip("dropbox", reason="needs the dropbox extra") + +# pylint: disable=wrong-import-position # importorskip must precede these imports +import dropbox +import requests +from dropbox import auth as dropbox_auth +from dropbox import files +from dropbox.exceptions import ( + ApiError, + AuthError, + BadInputError, + DropboxException, + HttpError, + InternalServerError, + RateLimitError, +) +from dropbox.stone_serializers import json_compat_obj_decode + +from automation_file.exceptions import ( + StorageAlreadyExistsException, + StorageException, + StoragePermissionException, + StorageTransientException, + StorageUnavailableException, + StorageURIException, +) +from automation_file.remote.dropbox_api.client import dropbox_instance +from automation_file.storage import File, StorageBackend, StorageResolver +from automation_file.storage import dropbox_storage as dropbox_storage_module +from automation_file.storage.dropbox_storage import ( + DROPBOX_SCHEME, + DropboxStorage, + dropbox_factory, +) +from tests.storage_contract import StorageContract + +PAGE_SIZE = 2 +REQUEST_ID = "request-1" +_HASH_BLOCK = 4 * 1024 * 1024 + + +def _api_error(error: Any) -> ApiError: + return ApiError(REQUEST_ID, error, None, None) + + +def _content_hash(data: bytes) -> str: + """Dropbox's content hash: the SHA-256 of the SHA-256 digests of 4 MiB blocks.""" + digests = b"".join( + hashlib.sha256(data[start : start + _HASH_BLOCK]).digest() + for start in range(0, len(data), _HASH_BLOCK) + ) + return hashlib.sha256(digests).hexdigest() + + +def _parent(path: str) -> str: + return path.rpartition("/")[0] + + +@dataclass +class _Stored: + data: bytes + revision: int + modified: datetime + + def metadata(self, path: str) -> files.FileMetadata: + return files.FileMetadata( + name=path.rpartition("/")[2], + id=f"id:{self.revision}", + client_modified=self.modified, + server_modified=self.modified, + rev=f"{self.revision:016x}", + size=len(self.data), + path_lower=path.lower(), + path_display=path, + content_hash=_content_hash(self.data), + ) + + +def _folder_metadata(path: str) -> files.FolderMetadata: + return files.FolderMetadata( + name=path.rpartition("/")[2], id="id:folder", path_lower=path.lower(), path_display=path + ) + + +class FakeDropbox: + """The subset of ``dropbox.Dropbox`` that DropboxStorage calls. + + Paths are ``""`` for the root and ``/a/b`` otherwise, as in the HTTP API. Unlike + Dropbox, names are compared case-sensitively. + """ + + def __init__(self) -> None: + self.stored: dict[str, _Stored] = {} + self.folders: set[str] = set() + self.calls: list[str] = [] + self.chunks: list[int] = [] + self.fail_with: Exception | None = None + self._sessions: dict[str, bytearray] = {} + self._cursors: dict[str, list[Any]] = {} + self._revision = 0 + + # ------------------------------------------------------------------ helpers + + def _begin(self, call: str, *paths: str) -> None: + self.calls.append(call) + if self.fail_with is not None: + raise self.fail_with + for path in paths: + if path and (not path.startswith("/") or path.endswith("/")): + raise BadInputError(REQUEST_ID, f"path: {path!r} did not match the pattern") + + def _is_folder(self, path: str) -> bool: + return not path or path in self.folders + + def _write_conflict(self, path: str) -> Any: + """Return the WriteError that stops a write at ``path``, or ``None``.""" + if path in self.folders: + return files.WriteError.conflict(files.WriteConflictError.folder) + ancestor = _parent(path) + while ancestor: + if ancestor in self.stored: + return files.WriteError.conflict(files.WriteConflictError.file_ancestor) + ancestor = _parent(ancestor) + return None + + def _make_parents(self, path: str) -> None: + ancestor = _parent(path) + while ancestor: + self.folders.add(ancestor) + ancestor = _parent(ancestor) + + def _store(self, path: str, data: bytes, mode: Any) -> files.FileMetadata: + conflict = self._write_conflict(path) + if conflict is None and path in self.stored and not mode.is_overwrite(): + conflict = files.WriteError.conflict(files.WriteConflictError.file) + if conflict is not None: + failed = files.UploadWriteFailed(reason=conflict, upload_session_id="session") + raise _api_error(files.UploadError.path(failed)) + self._make_parents(path) + self._revision += 1 + moment = datetime.now(timezone.utc).replace(tzinfo=None, microsecond=0) + self.stored[path] = _Stored(data, self._revision, moment) + return self.stored[path].metadata(path) + + def _metadata(self, path: str) -> Any: + if path in self.stored: + return self.stored[path].metadata(path) + if path in self.folders: + return _folder_metadata(path) + return None + + # ------------------------------------------------------------------ the SDK surface + + def files_get_metadata(self, path: str) -> Any: + self._begin("files_get_metadata", path) + if not path: + raise BadInputError(REQUEST_ID, "path: The root folder is unsupported.") + metadata = self._metadata(path) + if metadata is None: + raise _api_error(files.GetMetadataError.path(files.LookupError.not_found)) + return metadata + + def files_list_folder(self, path: str) -> files.ListFolderResult: + self._begin("files_list_folder", path) + if path in self.stored: + raise _api_error(files.ListFolderError.path(files.LookupError.not_folder)) + if not self._is_folder(path): + raise _api_error(files.ListFolderError.path(files.LookupError.not_found)) + names = sorted(name for name in (*self.folders, *self.stored) if _parent(name) == path) + return self._page([self._metadata(name) for name in names]) + + def files_list_folder_continue(self, cursor: str) -> files.ListFolderResult: + self._begin("files_list_folder_continue") + return self._page(self._cursors.pop(cursor)) + + def _page(self, entries: list[Any]) -> files.ListFolderResult: + cursor = f"cursor-{len(self._cursors)}-{len(entries)}" + rest = entries[PAGE_SIZE:] + if rest: + self._cursors[cursor] = rest + return files.ListFolderResult( + entries=entries[:PAGE_SIZE], cursor=cursor, has_more=bool(rest) + ) + + def files_upload(self, f: bytes, path: str, mode: Any = files.WriteMode.add) -> Any: + self._begin("files_upload", path) + if not isinstance(f, bytes): + raise TypeError(f"expected request_binary as binary type, got {type(f)}") + return self._store(path, f, mode) + + def files_upload_session_start(self, f: bytes) -> files.UploadSessionStartResult: + self._begin("files_upload_session_start") + session_id = f"session-{len(self._sessions)}" + self._sessions[session_id] = bytearray(f) + self.chunks.append(len(f)) + return files.UploadSessionStartResult(session_id=session_id) + + def _append(self, f: bytes, cursor: Any) -> bytearray: + lookup = files.UploadSessionLookupError + if cursor.session_id not in self._sessions: + raise _api_error(files.UploadSessionFinishError.lookup_failed(lookup.not_found)) + session = self._sessions[cursor.session_id] + if cursor.offset != len(session): + offset = files.UploadSessionOffsetError(correct_offset=len(session)) + raise _api_error( + files.UploadSessionFinishError.lookup_failed(lookup.incorrect_offset(offset)) + ) + session.extend(f) + self.chunks.append(len(f)) + return session + + def files_upload_session_append_v2(self, f: bytes, cursor: Any) -> None: + self._begin("files_upload_session_append_v2") + self._append(f, cursor) + + def files_upload_session_finish(self, f: bytes, cursor: Any, commit: Any) -> Any: + self._begin("files_upload_session_finish", commit.path) + data = bytes(self._append(f, cursor)) + del self._sessions[cursor.session_id] + return self._store(commit.path, data, commit.mode) + + def files_download_to_file(self, download_path: str, path: str) -> Any: + self._begin("files_download_to_file", path) + if path in self.folders: + raise _api_error(files.DownloadError.path(files.LookupError.not_file)) + if path not in self.stored: + raise _api_error(files.DownloadError.path(files.LookupError.not_found)) + Path(download_path).write_bytes(self.stored[path].data) + return self.stored[path].metadata(path) + + def files_delete_v2(self, path: str) -> files.DeleteResult: + self._begin("files_delete_v2", path) + metadata = self._metadata(path) + if metadata is None: + raise _api_error(files.DeleteError.path_lookup(files.LookupError.not_found)) + below = f"{path}/" + for name in [name for name in self.stored if name == path or name.startswith(below)]: + del self.stored[name] + self.folders = { + name for name in self.folders if name != path and not name.startswith(below) + } + return files.DeleteResult(metadata=metadata) + + def files_create_folder_v2(self, path: str) -> None: + self._begin("files_create_folder_v2", path) + conflict = self._write_conflict(path) + if path in self.stored: + conflict = files.WriteError.conflict(files.WriteConflictError.file) + if conflict is not None: + raise _api_error(files.CreateFolderError.path(conflict)) + self._make_parents(path) + self.folders.add(path) + + def _relocate(self, call: str, from_path: str, to_path: str) -> _Stored: + self._begin(call, from_path, to_path) + if from_path not in self.stored: + raise _api_error(files.RelocationError.from_lookup(files.LookupError.not_found)) + conflict = self._write_conflict(to_path) + if to_path in self.stored: + conflict = files.WriteError.conflict(files.WriteConflictError.file) + if conflict is not None: + raise _api_error(files.RelocationError.to(conflict)) + self._make_parents(to_path) + self._revision += 1 + source = self.stored[from_path] + self.stored[to_path] = _Stored(source.data, self._revision, source.modified) + return source + + def files_copy_v2(self, from_path: str, to_path: str) -> None: + self._relocate("files_copy_v2", from_path, to_path) + + def files_move_v2(self, from_path: str, to_path: str) -> None: + self._relocate("files_move_v2", from_path, to_path) + del self.stored[from_path] + + +class TestDropboxStorageContract(StorageContract): + @pytest.fixture + def backend(self) -> StorageBackend: + return DropboxStorage(FakeDropbox()) + + +class TestRootedDropboxStorageContract(StorageContract): + @pytest.fixture + def backend(self) -> StorageBackend: + client = FakeDropbox() + client.files_create_folder_v2("/team/a") + client.files_upload(b"keep", "/other-team/keep.txt") + return DropboxStorage(client, root="team/a") + + +class TestSessionUploadDropboxStorageContract(StorageContract): + """The whole contract again, with every upload going through an upload session.""" + + @pytest.fixture + def backend(self, monkeypatch: pytest.MonkeyPatch) -> StorageBackend: + monkeypatch.setattr(dropbox_storage_module, "UPLOAD_SESSION_THRESHOLD", -1) + monkeypatch.setattr(dropbox_storage_module, "UPLOAD_CHUNK_SIZE", 1024 * 1024) + return DropboxStorage(FakeDropbox()) + + +@pytest.fixture +def client() -> FakeDropbox: + return FakeDropbox() + + +@pytest.fixture +def storage(client: FakeDropbox) -> DropboxStorage: + return DropboxStorage(client) + + +def test_stat_reports_what_the_metadata_carries( + storage: DropboxStorage, client: FakeDropbox +) -> None: + info = storage.write_bytes("reports/q1.json", b"{}") + stored = client.stored["/reports/q1.json"] + assert info.path == "reports/q1.json" + assert info.size == 2 + assert info.modified_at == stored.modified.replace(tzinfo=timezone.utc) + assert info.modified_at.utcoffset() == timedelta(0) + assert info.version == f"{stored.revision:016x}" + assert info.etag == _content_hash(b"{}") + assert info.content_type is None + folder = storage.stat("reports") + assert (folder.is_dir, folder.size, folder.modified_at) == (True, None, None) + + +def test_the_account_root_needs_no_request(storage: DropboxStorage, client: FakeDropbox) -> None: + assert storage.stat("").is_dir is True + assert client.calls == [] + + +def test_a_missing_root_folder_does_not_exist(client: FakeDropbox) -> None: + rooted = DropboxStorage(client, root="/no/such/folder/") + assert rooted.root == "no/such/folder" + assert rooted.exists("") is False + rooted.write_bytes("a.txt", b"x") + assert rooted.exists("") is True + assert sorted(client.stored) == ["/no/such/folder/a.txt"] + + +def test_listing_reads_every_page(storage: DropboxStorage, client: FakeDropbox) -> None: + for index in range(5): + storage.write_bytes(f"dir/{index}.txt", b"x") + client.calls.clear() + listing = storage.list_dir("dir") + assert [info.path for info in listing] == [f"dir/{index}.txt" for index in range(5)] + assert all(info.version and info.etag for info in listing) + assert client.calls.count("files_list_folder_continue") == 2 + + +def test_a_small_file_goes_up_in_one_request(storage: DropboxStorage, client: FakeDropbox) -> None: + storage.write_bytes("a.bin", b"x" * 64) + assert "files_upload" in client.calls + assert "files_upload_session_start" not in client.calls + + +@pytest.mark.parametrize( + "size,chunks", + [ + (11, [4, 4, 3]), + (12, [4, 4, 4]), + (5, [4, 1]), + (9, [4, 4, 1]), + ], +) +def test_a_large_file_goes_up_through_an_upload_session( + monkeypatch: pytest.MonkeyPatch, + storage: DropboxStorage, + client: FakeDropbox, + size: int, + chunks: list[int], +) -> None: + monkeypatch.setattr(dropbox_storage_module, "UPLOAD_SESSION_THRESHOLD", 4) + monkeypatch.setattr(dropbox_storage_module, "UPLOAD_CHUNK_SIZE", 4) + data = bytes(range(size)) + storage.write_bytes("old.bin", b"old!") + client.calls.clear() + info = storage.write_bytes("big.bin", data) + assert info.size == size + assert client.stored["/big.bin"].data == data + assert client.chunks == chunks + assert "files_upload" not in client.calls + assert client.calls.count("files_upload_session_start") == 1 + assert client.calls.count("files_upload_session_append_v2") == len(chunks) - 2 + assert client.calls.count("files_upload_session_finish") == 1 + + +def test_a_session_upload_replaces_an_existing_file( + monkeypatch: pytest.MonkeyPatch, storage: DropboxStorage, client: FakeDropbox +) -> None: + monkeypatch.setattr(dropbox_storage_module, "UPLOAD_SESSION_THRESHOLD", 4) + monkeypatch.setattr(dropbox_storage_module, "UPLOAD_CHUNK_SIZE", 4) + storage.write_bytes("a.bin", b"first content") + storage.write_bytes("a.bin", b"second") + assert client.stored["/a.bin"].data == b"second" + + +def test_a_file_at_the_threshold_still_goes_up_in_one_request( + monkeypatch: pytest.MonkeyPatch, storage: DropboxStorage, client: FakeDropbox +) -> None: + monkeypatch.setattr(dropbox_storage_module, "UPLOAD_SESSION_THRESHOLD", 4) + storage.write_bytes("a.bin", b"1234") + assert client.calls.count("files_upload") == 1 + assert client.chunks == [] + + +def test_the_default_chunk_is_a_multiple_of_four_mebibytes() -> None: + assert dropbox_storage_module.UPLOAD_CHUNK_SIZE % (4 * 1024 * 1024) == 0 + assert dropbox_storage_module.UPLOAD_SESSION_THRESHOLD <= 150 * 1024 * 1024 + + +def test_copy_and_move_within_dropbox_are_done_by_dropbox( + storage: DropboxStorage, client: FakeDropbox +) -> None: + storage.write_bytes("a.txt", b"payload") + client.calls.clear() + storage.copy_from(storage, "a.txt", "copies/b.txt") + assert client.stored["/copies/b.txt"].data == b"payload" + storage.move_from(storage, "a.txt", "moved/c.txt") + assert sorted(client.stored) == ["/copies/b.txt", "/moved/c.txt"] + assert {"files_copy_v2", "files_move_v2"} <= set(client.calls) + assert not {"files_upload", "files_download_to_file"} & set(client.calls) + + +def test_a_copy_replaces_a_file_in_the_way(storage: DropboxStorage, client: FakeDropbox) -> None: + storage.write_bytes("a.txt", b"new") + storage.write_bytes("b.txt", b"old") + client.calls.clear() + storage.copy_from(storage, "a.txt", "b.txt") + assert client.stored["/b.txt"].data == b"new" + relocations = [call for call in client.calls if call in ("files_copy_v2", "files_delete_v2")] + assert relocations == ["files_copy_v2", "files_delete_v2", "files_copy_v2"] + + +@pytest.mark.parametrize("relocate", ["_copy_from", "_move_from"]) +def test_a_file_is_never_deleted_to_make_room_for_itself( + storage: DropboxStorage, client: FakeDropbox, relocate: str +) -> None: + """Dropbox ignores case, so ``A.txt`` over ``a.txt`` reaches it as a file in its own way.""" + storage.write_bytes("a.txt", b"only copy") + client.calls.clear() + with pytest.raises(StorageException, match="the same file"): + getattr(storage, relocate)(storage, "a.txt", "a.txt") + assert "files_delete_v2" not in client.calls + assert client.stored["/a.txt"].data == b"only copy" + + +def test_a_folder_in_the_way_is_never_deleted(storage: DropboxStorage, client: FakeDropbox) -> None: + storage.write_bytes("a.txt", b"new") + storage.write_bytes("dir/keep.txt", b"keep") + client.calls.clear() + with pytest.raises(StorageAlreadyExistsException): + storage._copy_from(storage, "a.txt", "dir") + with pytest.raises(StorageAlreadyExistsException): + storage._move_from(storage, "a.txt", "dir") + assert "files_delete_v2" not in client.calls + assert client.stored["/dir/keep.txt"].data == b"keep" + assert client.stored["/a.txt"].data == b"new" + + +def test_copy_between_two_clients_goes_through_a_staging_file(storage: DropboxStorage) -> None: + other_client = FakeDropbox() + other = DropboxStorage(other_client) + storage.write_bytes("a.txt", b"payload") + other.copy_from(storage, "a.txt", "a.txt") + assert other_client.stored["/a.txt"].data == b"payload" + assert "files_copy_v2" not in other_client.calls + other.move_from(storage, "a.txt", "b.txt") + assert other_client.stored["/b.txt"].data == b"payload" + assert storage.exists("a.txt") is False + + +def test_mkdir_accepts_a_folder_that_appeared_meanwhile( + storage: DropboxStorage, client: FakeDropbox +) -> None: + client.files_create_folder_v2("/dir") + storage._mkdir("dir") + client.files_upload(b"x", "/file") + with pytest.raises(StorageAlreadyExistsException): + storage._mkdir("file") + + +def test_deleting_a_directory_is_one_request(storage: DropboxStorage, client: FakeDropbox) -> None: + for path in ("dir/a.txt", "dir/sub/b.txt", "dir/sub/deeper/c.txt"): + storage.write_bytes(path, b"x") + client.calls.clear() + storage.delete("dir", recursive=True) + assert client.calls.count("files_delete_v2") == 1 + assert client.stored == {} + assert client.folders == set() + + +def _lookup(error: Any) -> ApiError: + return _api_error(files.GetMetadataError.path(error)) + + +@pytest.mark.parametrize( + "error,expected", + [ + (AuthError(REQUEST_ID, dropbox_auth.AuthError.invalid_access_token), "permission"), + (HttpError(REQUEST_ID, 403, "forbidden"), "permission"), + (_lookup(files.LookupError.restricted_content), "permission"), + ( + _api_error(files.CreateFolderError.path(files.WriteError.no_write_permission)), + "permission", + ), + (RateLimitError(REQUEST_ID, None, 1), "transient"), + (InternalServerError(REQUEST_ID, 503, "unavailable"), "transient"), + (HttpError(REQUEST_ID, 408, "timeout"), "transient"), + (_api_error(files.DeleteError.too_many_write_operations), "transient"), + (_api_error(files.RelocationError.internal_error), "transient"), + (requests.ConnectionError("connection refused"), "transient"), + (requests.Timeout("read timed out"), "transient"), + (requests.exceptions.ChunkedEncodingError("cut off"), "transient"), + (BadInputError(REQUEST_ID, "bad request"), "other"), + (HttpError(REQUEST_ID, 418, "teapot"), "other"), + (DropboxException(REQUEST_ID), "other"), + (_lookup(files.LookupError.malformed_path(None)), "other"), + (_api_error(files.ListFolderError.other), "other"), + ], +) +def test_sdk_errors_become_storage_errors( + storage: DropboxStorage, client: FakeDropbox, error: Exception, expected: str +) -> None: + kinds: dict[str, type[Exception]] = { + "permission": StoragePermissionException, + "transient": StorageTransientException, + "other": StorageException, + } + client.fail_with = error + with pytest.raises(kinds[expected]) as caught: + storage.stat("a.txt") + assert caught.value.__cause__ is error + assert type(caught.value) is kinds[expected] + + +def test_a_lost_upload_session_is_not_a_missing_path( + monkeypatch: pytest.MonkeyPatch, storage: DropboxStorage, client: FakeDropbox, tmp_path: Path +) -> None: + monkeypatch.setattr(dropbox_storage_module, "UPLOAD_SESSION_THRESHOLD", 1) + monkeypatch.setattr(client, "files_upload_session_start", lambda _data: _Lost()) + source = tmp_path / "source.bin" + source.write_bytes(b"0123456789") + with pytest.raises(StorageException) as caught: + storage.upload(source, "a.bin") + assert type(caught.value) is StorageException + assert "lookup_failed/not_found" in str(caught.value) + + +class _Lost: + session_id = "no-such-session" + + +def test_error_tags_follow_the_sdk_unions() -> None: + tags = dropbox_storage_module._tags + assert tags(files.GetMetadataError.path(files.LookupError.not_found)) == ("path", "not_found") + conflict = files.WriteError.conflict(files.WriteConflictError.file) + failed = files.UploadWriteFailed(reason=conflict, upload_session_id="session") + assert tags(files.UploadError.path(failed)) == ("path", "conflict", "file") + assert tags(files.RelocationError.to(conflict)) == ("to", "conflict", "file") + assert tags(files.DeleteError.path_lookup(files.LookupError.not_found)) == ( + "path_lookup", + "not_found", + ) + assert tags(files.LookupError.malformed_path("bad")) == ("malformed_path",) + assert tags(None) == () + + +def test_the_shared_client_must_be_initialised(monkeypatch: pytest.MonkeyPatch) -> None: + monkeypatch.setattr(dropbox_instance, "client", None) + with pytest.raises(StorageUnavailableException, match="later_init"): + DropboxStorage().exists("a.txt") + + +def test_dropbox_uris_use_the_shared_client( + monkeypatch: pytest.MonkeyPatch, client: FakeDropbox, tmp_path: Path +) -> None: + monkeypatch.setattr(dropbox_instance, "client", client) + resolver = StorageResolver() + resolver.register_scheme(DROPBOX_SCHEME, dropbox_factory) + report = File("dropbox:///reports/q1.csv", resolver=resolver) + report.write(b"a,b\n") + assert client.stored["/reports/q1.csv"].data == b"a,b\n" + assert resolver.resolve("dropbox:///reports/q1.csv") == (DropboxStorage(), "reports/q1.csv") + assert resolver.resolve("dropbox:///") == (DropboxStorage(), "") + report.copy_to(tmp_path / "q1.csv") + assert (tmp_path / "q1.csv").read_bytes() == b"a,b\n" + File(tmp_path / "q1.csv", resolver=resolver).move_to("dropbox:///archive/2026/q1.csv") + assert client.stored["/archive/2026/q1.csv"].data == b"a,b\n" + assert not (tmp_path / "q1.csv").exists() + + +def test_a_dropbox_uri_takes_no_authority() -> None: + resolver = StorageResolver() + resolver.register_scheme(DROPBOX_SCHEME, dropbox_factory) + with pytest.raises(StorageURIException) as caught: + resolver.resolve("dropbox://team/reports/q1.csv") + assert "as in 'dropbox:///team/reports/q1.csv'" in str(caught.value) + with pytest.raises(StorageURIException) as caught: + resolver.resolve("dropbox://team") + assert str(caught.value).endswith("as in 'dropbox:///team'") + + +def test_a_rooted_backend_can_be_mounted(client: FakeDropbox) -> None: + resolver = StorageResolver() + archive = DropboxStorage(client, root="archive/2026") + resolver.mount("dropbox://archive", archive) + File("dropbox://archive/q1.csv", resolver=resolver).write(b"x") + assert sorted(client.stored) == ["/archive/2026/q1.csv"] + assert resolver.resolve("dropbox://archive/q1.csv") == (archive, "q1.csv") + assert resolver.capabilities("dropbox://archive").directories is True + + +def test_uri_equality_and_repr(client: FakeDropbox) -> None: + assert DropboxStorage(client).uri_for("") == "dropbox:///" + assert DropboxStorage(client).uri_for("/a//b.txt") == "dropbox:///a/b.txt" + assert DropboxStorage(client, root="team").uri_for("a.txt") == "dropbox:///team/a.txt" + assert DropboxStorage(client, root="team").uri_for() == "dropbox:///team" + assert DropboxStorage(client) == DropboxStorage(client) + assert DropboxStorage(client) != DropboxStorage(client, root="team") + assert DropboxStorage(client) != DropboxStorage(FakeDropbox()) + assert DropboxStorage() == DropboxStorage() + assert len({DropboxStorage(client), DropboxStorage(client)}) == 1 + assert repr(DropboxStorage(client, root="team/a")) == "DropboxStorage(root='team/a')" + assert DropboxStorage.scheme == DROPBOX_SCHEME == "dropbox" + capabilities = DropboxStorage.capabilities + assert (capabilities.directories, capabilities.etag, capabilities.version) == (True, True, True) + assert (capabilities.content_type, capabilities.metadata) == (False, False) + + +# ---------------------------------------------------------------------- the real dropbox SDK + + +def _parameters(method: Any) -> list[str]: + return list(inspect.signature(method).parameters)[1:] + + +def test_the_sdk_has_the_calls_the_adapter_makes() -> None: + """The stand-in above is only as good as its match with the installed SDK.""" + client = dropbox.Dropbox + assert _parameters(client.files_get_metadata)[0] == "path" + assert _parameters(client.files_list_folder)[0] == "path" + assert _parameters(client.files_list_folder_continue) == ["cursor"] + assert _parameters(client.files_upload)[:3] == ["f", "path", "mode"] + assert _parameters(client.files_upload_session_start)[0] == "f" + assert _parameters(client.files_upload_session_append_v2)[:2] == ["f", "cursor"] + assert _parameters(client.files_upload_session_finish)[:3] == ["f", "cursor", "commit"] + assert _parameters(client.files_download_to_file)[:2] == ["download_path", "path"] + assert _parameters(client.files_delete_v2)[0] == "path" + assert _parameters(client.files_create_folder_v2)[0] == "path" + assert _parameters(client.files_copy_v2)[:2] == ["from_path", "to_path"] + assert _parameters(client.files_move_v2)[:2] == ["from_path", "to_path"] + assert files.WriteMode.overwrite.is_overwrite() is True + commit = files.CommitInfo(path="/a.bin", mode=files.WriteMode.overwrite) + assert (commit.path, commit.mode.is_overwrite()) == ("/a.bin", True) + cursor = files.UploadSessionCursor(session_id="session", offset=0) + cursor.offset += 4 + assert (cursor.session_id, cursor.offset) == ("session", 4) + + +def test_the_sdk_takes_bytes_not_a_file_handle() -> None: + """Why a large file needs a session: one request holds its whole body in memory.""" + client = dropbox.Dropbox("placeholder") + with pytest.raises(TypeError, match="binary type"): + client.files_upload(io.BytesIO(b"x"), "/a.bin") + + +def test_the_sdk_decodes_metadata_the_way_the_adapter_reads_it() -> None: + decoded = json_compat_obj_decode( + files.Metadata_validator, + { + ".tag": "file", + "name": "a.txt", + "id": "id:a", + "client_modified": "2026-10-08T02:30:00Z", + "server_modified": "2026-10-08T02:30:05Z", + "rev": "0123456789abcdef", + "size": 3, + "path_lower": "/a.txt", + "path_display": "/a.txt", + "content_hash": "a" * 64, + }, + ) + assert isinstance(decoded, files.FileMetadata) + assert decoded.server_modified.tzinfo is None + info = dropbox_storage_module._file_info("a.txt", decoded, files) + assert info is not None + assert info.modified_at == datetime(2026, 10, 8, 2, 30, 5, tzinfo=timezone.utc) + assert (info.size, info.version, info.etag) == (3, "0123456789abcdef", "a" * 64) + folder = json_compat_obj_decode( + files.Metadata_validator, + {".tag": "folder", "name": "d", "id": "id:d", "path_lower": "/d", "path_display": "/d"}, + ) + assert dropbox_storage_module._file_info("d", folder, files).is_dir is True + deleted = json_compat_obj_decode( + files.Metadata_validator, + {".tag": "deleted", "name": "x", "path_lower": "/x", "path_display": "/x"}, + ) + assert dropbox_storage_module._file_info("x", deleted, files) is None + + +def test_two_roots_of_one_account_do_not_lose_a_file_to_itself(client: FakeDropbox) -> None: + whole = DropboxStorage(client) + inner = DropboxStorage(client, root="team/a") + whole.mkdir("team/a") + inner.write_bytes("docs/a.txt", b"payload") + for operation in (whole.move_from, whole.copy_from): + with pytest.raises(StorageException, match="same file"): + operation(inner, "docs/a.txt", "team/a/docs/a.txt") + with pytest.raises(StorageException, match="same file"): + inner.move_from(whole, "team/a/docs/a.txt", "docs/a.txt") + assert whole.read_bytes("team/a/docs/a.txt") == b"payload" diff --git a/tests/test_storage_fsspec.py b/tests/test_storage_fsspec.py new file mode 100644 index 0000000..343c1af --- /dev/null +++ b/tests/test_storage_fsspec.py @@ -0,0 +1,606 @@ +"""FsspecStorage: the storage contract against real fsspec filesystems. + +fsspec ships a memory filesystem and a local one, so the contract runs against +both without a stand-in. A third filesystem written here keeps flat keys the way +an object store does, for the ``directories=False`` mode. +""" + +from __future__ import annotations + +import inspect +import os +import secrets +import sys +from collections.abc import Iterator +from datetime import datetime, timedelta, timezone +from pathlib import Path +from typing import Any + +import pytest + +fsspec = pytest.importorskip("fsspec") + +# pylint: disable=wrong-import-position # importorskip must precede these imports +from fsspec import AbstractFileSystem # noqa: E402 +from fsspec.core import url_to_fs # noqa: E402 +from fsspec.implementations.local import LocalFileSystem # noqa: E402 +from fsspec.implementations.memory import MemoryFileSystem # noqa: E402 + +from automation_file.exceptions import ( # noqa: E402 + StorageException, + StorageNotFoundException, + StoragePermissionException, + StorageTransientException, + StorageUnavailableException, + StorageUnsupportedException, + StorageURIException, +) +from automation_file.storage import File, StorageBackend, StorageResolver # noqa: E402 +from automation_file.storage import fsspec_storage as fsspec_storage_module # noqa: E402 +from automation_file.storage.fsspec_storage import FSSPEC_SCHEME, FsspecStorage # noqa: E402 +from tests.storage_contract import StorageContract # noqa: E402 + +CLOSE_ENOUGH = timedelta(milliseconds=1) + + +def _wipe_memory_filesystem() -> None: + """MemoryFileSystem keeps one store per process, files and directories alike.""" + MemoryFileSystem.store.clear() + MemoryFileSystem.pseudo_dirs[:] = [""] + + +@pytest.fixture(autouse=True) +def _isolated_memory_filesystem() -> Iterator[None]: + _wipe_memory_filesystem() + yield + _wipe_memory_filesystem() + + +class KeyValueFileSystem(AbstractFileSystem): + """Flat keys, as in s3fs or gcsfs: a directory is only a prefix of the keys below it.""" + + protocol = "keyvalue" + cachable = False + + def __init__(self, **options: Any) -> None: + super().__init__(**options) + self.objects: dict[str, tuple[bytes, datetime]] = {} + self.calls: list[str] = [] + + def _key(self, path: str) -> str: + return str(self._strip_protocol(path)).strip("/") + + def info(self, path: str, **kwargs: Any) -> dict[str, Any]: + key = self._key(path) + if key in self.objects: + data, modified = self.objects[key] + return {"name": key, "size": len(data), "type": "file", "LastModified": modified} + if not key or any(name.startswith(f"{key}/") for name in self.objects): + return {"name": key, "size": 0, "type": "directory"} + raise FileNotFoundError(path) + + def ls(self, path: str, detail: bool = True, **kwargs: Any) -> list[Any]: + key = self._key(path) + if key in self.objects: + return [self.info(key)] if detail else [key] + prefix = f"{key}/" if key else "" + below = (name[len(prefix) :] for name in self.objects if name.startswith(prefix)) + children = sorted({prefix + rest.split("/", 1)[0] for rest in below}) + if not children and key: + raise FileNotFoundError(path) + return [self.info(child) for child in children] if detail else children + + def put_file(self, lpath: str, rpath: str, callback: Any = None, **kwargs: Any) -> None: + self.calls.append("put_file") + self.objects[self._key(rpath)] = (Path(lpath).read_bytes(), datetime.now(timezone.utc)) + + def get_file(self, rpath: str, lpath: str, callback: Any = None, **kwargs: Any) -> None: + self.calls.append("get_file") + Path(lpath).write_bytes(self.cat_file(rpath)) + + def cat_file(self, path: str, start: Any = None, end: Any = None, **kwargs: Any) -> bytes: + key = self._key(path) + if key not in self.objects: + raise FileNotFoundError(path) + return self.objects[key][0][start:end] + + def rm_file(self, path: str) -> None: + self.calls.append("rm_file") + key = self._key(path) + if key not in self.objects: + raise FileNotFoundError(path) + del self.objects[key] + + def cp_file(self, path1: str, path2: str, **kwargs: Any) -> None: + self.calls.append("cp_file") + self.objects[self._key(path2)] = (self.cat_file(path1), datetime.now(timezone.utc)) + + def mkdir(self, path: str, create_parents: bool = True, **kwargs: Any) -> None: + """A prefix needs no creating.""" + + def makedirs(self, path: str, exist_ok: bool = False) -> None: + """A prefix needs no creating.""" + + def rmdir(self, path: str) -> None: + # Like s3fs: a prefix whose keys are gone is not there to remove. + raise FileNotFoundError(path) + + def modified(self, path: str) -> datetime: + return self.info(path)["LastModified"] + + +class TimelessFileSystem(KeyValueFileSystem): + """A filesystem that knows no modification times, like fsspec's HTTP one.""" + + modified = AbstractFileSystem.modified + + def info(self, path: str, **kwargs: Any) -> dict[str, Any]: + details = super().info(path, **kwargs) + details.pop("LastModified", None) + return details + + +class CopylessFileSystem(KeyValueFileSystem): + """A filesystem that cannot copy on its own side, like the abstract base.""" + + cp_file = AbstractFileSystem.cp_file + + +class TestMemoryFsspecStorageContract(StorageContract): + @pytest.fixture + def backend(self) -> StorageBackend: + return FsspecStorage(MemoryFileSystem()) + + +class TestRootedMemoryFsspecStorageContract(StorageContract): + @pytest.fixture + def backend(self) -> StorageBackend: + filesystem = MemoryFileSystem() + filesystem.makedirs("/team/a") + filesystem.pipe_file("/other-team/keep.txt", b"keep") + return FsspecStorage(filesystem, root="team/a") + + +class TestLocalFsspecStorageContract(StorageContract): + @pytest.fixture + def backend(self, tmp_path: Path) -> StorageBackend: + root = tmp_path / "fsspec-root" + root.mkdir() + (tmp_path / "outside.txt").write_bytes(b"outside") + return FsspecStorage(LocalFileSystem(), root=str(root)) + + +class TestKeyValueFsspecStorageContract(StorageContract): + """An object store: ``directories=False`` and a root that is only a key prefix.""" + + @pytest.fixture + def backend(self) -> StorageBackend: + filesystem = KeyValueFileSystem() + filesystem.objects["bucket/other-tenant/keep.txt"] = (b"keep", datetime.now(timezone.utc)) + return FsspecStorage(filesystem, root="bucket/tenant/a", directories=False) + + +@pytest.fixture +def memory() -> FsspecStorage: + return FsspecStorage(MemoryFileSystem()) + + +@pytest.fixture +def keyvalue() -> FsspecStorage: + return FsspecStorage(KeyValueFileSystem(), root="bucket", scheme="gcs", directories=False) + + +def _stored() -> dict[str, bytes]: + return {name: bytes(item.getbuffer()) for name, item in MemoryFileSystem.store.items()} + + +# ---------------------------------------------------------------------- stat and listing + + +def test_stat_takes_the_time_from_info_when_it_is_there(tmp_path: Path) -> None: + storage = FsspecStorage(LocalFileSystem(), root=str(tmp_path)) + info = storage.write_bytes("dir/a.txt", b"xyz") + expected = datetime.fromtimestamp(os.stat(tmp_path / "dir" / "a.txt").st_mtime, timezone.utc) + assert info.size == 3 + assert abs(info.modified_at - expected) < CLOSE_ENOUGH + assert info.modified_at.utcoffset() == timedelta(0) + assert (info.etag, info.version, info.content_type) == (None, None, None) + folder = storage.stat("dir") + assert (folder.is_dir, folder.size) == (True, None) + assert folder.modified_at is not None + assert all(entry.modified_at is not None for entry in storage.list_dir("", recursive=True)) + + +def test_stat_asks_modified_when_info_has_no_time(memory: FsspecStorage) -> None: + info = memory.write_bytes("dir/a.txt", b"x") + assert "mtime" not in memory.filesystem.info("/dir/a.txt") + assert info.modified_at == memory.filesystem.modified("/dir/a.txt") + assert info.modified_at.utcoffset() == timedelta(0) + assert memory.stat("dir").modified_at is None + # A listing stays one call: it carries a time only when the filesystem lists one. + assert [entry.modified_at for entry in memory.list_dir("dir")] == [None] + + +def test_a_filesystem_without_times_reports_none() -> None: + storage = FsspecStorage(TimelessFileSystem(), root="bucket", directories=False) + assert storage.capabilities.modified_at is False + info = storage.write_bytes("a.txt", b"x") + assert (info.size, info.modified_at) == (1, None) + + +def test_capabilities_and_scheme_belong_to_the_instance(tmp_path: Path) -> None: + local = FsspecStorage(LocalFileSystem(), root=str(tmp_path)) + store = FsspecStorage(KeyValueFileSystem(), root="bucket", scheme="GCS", directories=False) + assert (local.scheme, local.capabilities.directories) == (FSSPEC_SCHEME, True) + assert (store.scheme, store.capabilities.directories) == ("gcs", False) + assert local.capabilities.modified_at is True + assert store.capabilities.modified_at is True + # The instances carry their own; the class keeps the defaults. + assert FsspecStorage.scheme == FSSPEC_SCHEME == "fsspec" + assert FsspecStorage.capabilities.directories is True + with pytest.raises(StorageURIException): + FsspecStorage(KeyValueFileSystem(), scheme="not a scheme") + + +AWARE = datetime(2026, 10, 8, 2, 30, tzinfo=timezone.utc) + + +@pytest.mark.parametrize( + "value,expected", + [ + (AWARE, AWARE), + (AWARE.astimezone(timezone(timedelta(hours=8))), AWARE), + (AWARE.replace(tzinfo=None), AWARE), + (AWARE.timestamp(), AWARE), + (int(AWARE.timestamp()), AWARE), + ("2026-10-08T02:30:00Z", AWARE), + ("2026-10-08T10:30:00+08:00", AWARE), + ("2026-10-08T02:30:00.000Z", AWARE), + ("Thu Oct 8 02:30:00 2026", None), + ("", None), + (None, None), + (True, None), + (float("nan"), None), + (1e30, None), + (b"2026", None), + ], +) +def test_modification_times_are_read_in_every_usual_form(value: Any, expected: Any) -> None: + assert fsspec_storage_module._utc(value) == expected + + +@pytest.mark.parametrize("key", ["mtime", "LastModified", "last_modified", "modified", "updated"]) +def test_the_usual_info_keys_are_understood(key: str) -> None: + assert fsspec_storage_module._listed_time({key: "2026-10-08T02:30:00Z"}) == AWARE + assert fsspec_storage_module._listed_time({"created": AWARE}) is None + + +def test_a_placeholder_for_the_directory_itself_is_not_listed(keyvalue: FsspecStorage) -> None: + filesystem = keyvalue.filesystem + keyvalue.write_bytes("dir/a.txt", b"x") + + def _with_placeholder(path: str, detail: bool = True, **kwargs: Any) -> list[Any]: + listing = KeyValueFileSystem.ls(filesystem, path, detail, **kwargs) + return [{"name": f"{path}/", "size": 0, "type": "directory"}, *listing] + + filesystem.ls = _with_placeholder + assert [info.path for info in keyvalue.list_dir("dir")] == ["dir/a.txt"] + + +# ---------------------------------------------------------------------- copy and move + + +def test_copy_and_move_within_one_filesystem_are_native(keyvalue: FsspecStorage) -> None: + filesystem = keyvalue.filesystem + keyvalue.write_bytes("a.txt", b"payload") + filesystem.calls.clear() + keyvalue.copy_from(keyvalue, "a.txt", "copies/b.txt") + assert filesystem.objects["bucket/copies/b.txt"][0] == b"payload" + assert filesystem.calls == ["cp_file"] + filesystem.calls.clear() + other_root = FsspecStorage(filesystem, root="bucket/moved", directories=False) + other_root.move_from(keyvalue, "a.txt", "c.txt") + assert sorted(filesystem.objects) == ["bucket/copies/b.txt", "bucket/moved/c.txt"] + assert not {"get_file", "put_file"} & set(filesystem.calls) + + +def test_a_local_move_is_a_rename(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: + storage = FsspecStorage(LocalFileSystem(), root=str(tmp_path)) + storage.write_bytes("a.txt", b"payload") + storage.write_bytes("moved/b.txt", b"old") + copies: list[str] = [] + monkeypatch.setattr(LocalFileSystem, "cp_file", lambda *paths, **_options: copies.append("cp")) + storage.move_from(storage, "a.txt", "moved/b.txt") + assert copies == [] + assert (tmp_path / "moved" / "b.txt").read_bytes() == b"payload" + assert not (tmp_path / "a.txt").exists() + + +def test_names_with_glob_characters_are_taken_literally(memory: FsspecStorage) -> None: + """fsspec's copy(), mv() and rm() would expand ``report[1].txt`` to ``report1.txt``.""" + memory.write_bytes("report1.txt", b"one") + memory.write_bytes("report[1].txt", b"bracket") + memory.write_bytes("dir1/keep.txt", b"keep") + memory.write_bytes("dir[1]/a?.txt", b"question") + memory.write_bytes("dir[1]/b*.txt", b"star") + memory.copy_from(memory, "report[1].txt", "copy.txt") + assert memory.read_bytes("copy.txt") == b"bracket" + memory.move_from(memory, "report[1].txt", "moved.txt") + memory.move_from(memory, "dir[1]/a?.txt", "dir[1]/renamed.txt") + memory.move_from(memory, "copy.txt", "dir[1]/c[2].txt") + memory.delete("dir[1]/b*.txt") + assert _stored() == { + "/report1.txt": b"one", + "/moved.txt": b"bracket", + "/dir1/keep.txt": b"keep", + "/dir[1]/renamed.txt": b"question", + "/dir[1]/c[2].txt": b"bracket", + } + memory.delete("dir[1]", recursive=True) + assert _stored() == { + "/report1.txt": b"one", + "/moved.txt": b"bracket", + "/dir1/keep.txt": b"keep", + } + assert [info.path for info in memory.list_dir("")] == ["dir1", "moved.txt", "report1.txt"] + + +def test_a_filesystem_that_cannot_copy_gets_a_staged_transfer() -> None: + filesystem = CopylessFileSystem() + storage = FsspecStorage(filesystem, root="bucket", directories=False) + storage.write_bytes("a.txt", b"payload") + filesystem.calls.clear() + storage.copy_from(storage, "a.txt", "b.txt") + assert filesystem.objects["bucket/b.txt"][0] == b"payload" + assert filesystem.calls == ["get_file", "put_file"] + storage.move_from(storage, "a.txt", "c.txt") + assert sorted(filesystem.objects) == ["bucket/b.txt", "bucket/c.txt"] + + +def test_two_filesystem_objects_do_not_share_a_native_copy() -> None: + first = FsspecStorage(KeyValueFileSystem(), root="bucket", directories=False) + second = FsspecStorage(KeyValueFileSystem(), root="bucket", directories=False) + first.write_bytes("a.txt", b"payload") + second.copy_from(first, "a.txt", "a.txt") + assert second.filesystem.objects["bucket/a.txt"][0] == b"payload" + assert "cp_file" not in second.filesystem.calls + assert first != second + + +def test_copy_from_another_backend(memory: FsspecStorage, tmp_path: Path) -> None: + local = FsspecStorage(LocalFileSystem(), root=str(tmp_path)) + local.write_bytes("a.txt", b"payload") + memory.move_from(local, "a.txt", "in/memory.txt") + assert _stored() == {"/in/memory.txt": b"payload"} + assert not (tmp_path / "a.txt").exists() + + +# ---------------------------------------------------------------------- errors + + +class _DriverError(Exception): + """What a client library behind a filesystem might raise.""" + + +@pytest.mark.parametrize( + "error,expected", + [ + (PermissionError(13, "denied"), StoragePermissionException), + (ConnectionResetError(104, "reset"), StorageTransientException), + (TimeoutError("timed out"), StorageTransientException), + (NotImplementedError(), StorageUnsupportedException), + (ImportError("Install s3fs to access S3"), StorageUnavailableException), + (OSError("disk on fire"), StorageException), + (ValueError("bad argument"), StorageException), + (_DriverError("client library error"), StorageException), + ], +) +def test_filesystem_errors_become_storage_errors( + keyvalue: FsspecStorage, error: Exception, expected: type[Exception] +) -> None: + def _fail(*_arguments: Any, **_options: Any) -> None: + raise error + + keyvalue.write_bytes("a.txt", b"x") + for method in ("info", "ls", "cat_file", "rm_file", "put_file", "get_file"): + setattr(keyvalue.filesystem, method, _fail) + with pytest.raises(expected) as caught: + keyvalue.stat("a.txt") + assert type(caught.value) is expected + assert caught.value.__cause__ is error + with pytest.raises(expected): + keyvalue._list_dir("") + with pytest.raises(expected): + keyvalue._read_bytes("a.txt") + with pytest.raises(expected): + keyvalue._delete_file("a.txt") + + +def test_a_missing_path_is_none_not_an_error(keyvalue: FsspecStorage, tmp_path: Path) -> None: + assert keyvalue.exists("nope.txt") is False + with pytest.raises(StorageNotFoundException): + keyvalue._delete_file("nope.txt") + local = FsspecStorage(LocalFileSystem(), root=str(tmp_path / "no-such-directory")) + assert local.exists("") is False + with pytest.raises(StorageNotFoundException): + local.list_dir("") + # Only a root that is a key prefix lists as empty when nothing is below it. + with pytest.raises(StorageNotFoundException): + local._list_dir("") + assert list(keyvalue._list_dir("")) == [] + with pytest.raises(StorageNotFoundException): + keyvalue._list_dir("nope") + + +def test_a_directory_that_was_only_implied_goes_with_its_last_file(memory: FsspecStorage) -> None: + """Written behind the adapter's back, these directories were never created as such.""" + memory.filesystem.pipe_file("/implied/sub/a.txt", b"x") + memory.filesystem.pipe_file("/keep.txt", b"keep") + assert memory.stat("implied/sub").is_dir is True + memory.delete("implied", recursive=True) + assert memory.exists("implied") is False + assert _stored() == {"/keep.txt": b"keep"} + + +def test_a_read_only_filesystem_refuses_a_write(keyvalue: FsspecStorage) -> None: + def _read_only(*_arguments: Any, **_options: Any) -> None: + raise NotImplementedError + + keyvalue.filesystem.put_file = _read_only + with pytest.raises(StorageUnsupportedException): + keyvalue.write_bytes("a.txt", b"x") + assert keyvalue.exists("a.txt") is False + + +@pytest.mark.parametrize("path", ["a\\..\\..\\secret.txt", "..\\secret.txt", "dir/..\\..\\x"]) +def test_a_backslash_cannot_smuggle_a_parent_segment(tmp_path: Path, path: str) -> None: + root = tmp_path / "root" + root.mkdir() + (tmp_path / "secret.txt").write_bytes(b"secret") + storage = FsspecStorage(LocalFileSystem(), root=str(root)) + with pytest.raises(StorageURIException): + storage.exists(path) + with pytest.raises(StorageURIException): + storage.write_bytes(path, b"overwritten") + assert (tmp_path / "secret.txt").read_bytes() == b"secret" + + +def test_a_backslash_is_otherwise_part_of_the_name(keyvalue: FsspecStorage) -> None: + keyvalue.write_bytes("dir/a\\b.txt", b"x") + assert sorted(keyvalue.filesystem.objects) == ["bucket/dir/a\\b.txt"] + + +# ---------------------------------------------------------------------- from_url, mounting + + +def test_from_url_roots_the_storage_at_the_path_of_the_url() -> None: + storage = FsspecStorage.from_url("memory://jobs/2026/") + assert isinstance(storage.filesystem, MemoryFileSystem) + assert (storage.root, storage.scheme) == ("/jobs/2026", "memory") + assert storage.capabilities.directories is True + storage.write_bytes("a.txt", b"x") + assert _stored() == {"/jobs/2026/a.txt": b"x"} + assert storage.uri_for("a.txt") == "memory:///jobs/2026/a.txt" + whole = FsspecStorage.from_url("memory://", directories=False) + assert (whole.root, whole.capabilities.directories) == ("/", False) + assert whole.read_bytes("jobs/2026/a.txt") == b"x" + + +def test_from_url_passes_storage_options_to_the_filesystem(tmp_path: Path) -> None: + storage = FsspecStorage.from_url(f"file://{tmp_path.as_posix()}", auto_mkdir=True) + assert isinstance(storage.filesystem, LocalFileSystem) + assert storage.filesystem.auto_mkdir is True + assert storage.scheme == "local" + assert Path(storage.root) == tmp_path + storage.write_bytes("dir/a.txt", b"x") + assert (tmp_path / "dir" / "a.txt").read_bytes() == b"x" + + +def test_from_url_refuses_what_fsspec_cannot_open(monkeypatch: pytest.MonkeyPatch) -> None: + password = secrets.token_hex(8) + with pytest.raises(StorageURIException, match="Protocol not known") as caught: + FsspecStorage.from_url(f"no-such-protocol://user:{password}@host/data") + assert "no-such-protocol://host/data" in str(caught.value) + assert password not in str(caught.value) + + def _needs_a_driver(url: str, **_options: Any) -> None: + raise ImportError("Install s3fs to access S3") + + monkeypatch.setattr("fsspec.core.url_to_fs", _needs_a_driver) + with pytest.raises(StorageUnavailableException, match="Install s3fs"): + FsspecStorage.from_url("s3://bucket/data") + + +def test_fsspec_must_be_installed(monkeypatch: pytest.MonkeyPatch) -> None: + filesystem = KeyValueFileSystem() + monkeypatch.setitem(sys.modules, "fsspec", None) + monkeypatch.setitem(sys.modules, "fsspec.core", None) + with pytest.raises(StorageUnavailableException, match="fsspec is not installed"): + FsspecStorage.from_url("memory://jobs") + with pytest.raises(StorageUnavailableException, match="fsspec is not installed"): + FsspecStorage(filesystem) + + +def test_any_scheme_can_be_mounted_on_it(keyvalue: FsspecStorage, tmp_path: Path) -> None: + resolver = StorageResolver() + resolver.mount("gcs://reports", keyvalue) + report = File("gcs://reports/2026/q1.csv", resolver=resolver) + report.write(b"a,b\n") + assert keyvalue.filesystem.objects["bucket/2026/q1.csv"][0] == b"a,b\n" + assert resolver.resolve("gcs://reports/2026/q1.csv") == (keyvalue, "2026/q1.csv") + assert resolver.capabilities("gcs://reports").directories is False + assert "gcs" in resolver.schemes() + report.copy_to(tmp_path / "q1.csv") + assert (tmp_path / "q1.csv").read_bytes() == b"a,b\n" + report.move_to("gcs://reports/archive/q1.csv") + assert sorted(keyvalue.filesystem.objects) == ["bucket/archive/q1.csv"] + with pytest.raises(StorageURIException, match="no mount"): + resolver.resolve("gcs://another-bucket/a.txt") + + +def test_uri_equality_and_repr(tmp_path: Path) -> None: + filesystem = KeyValueFileSystem() + storage = FsspecStorage(filesystem, root="bucket/tenant/", scheme="gcs", directories=False) + assert storage.root == "bucket/tenant" + assert storage.uri_for("") == "gcs://bucket/tenant" + assert storage.uri_for("/a//b.txt") == "gcs://bucket/tenant/a/b.txt" + assert FsspecStorage(filesystem).uri_for("bucket/a.txt") == "fsspec://bucket/a.txt" + assert FsspecStorage(MemoryFileSystem()).uri_for("a.txt") == "fsspec:///a.txt" + assert repr(storage) == "FsspecStorage(KeyValueFileSystem, root='bucket/tenant')" + assert storage == FsspecStorage(filesystem, root="bucket/tenant") + assert storage != FsspecStorage(filesystem, root="bucket") + assert storage != FsspecStorage(KeyValueFileSystem(), root="bucket/tenant") + assert len({storage, FsspecStorage(filesystem, root="bucket/tenant")}) == 1 + # fsspec hands out one object per filesystem configuration, so these two are one storage. + assert FsspecStorage(LocalFileSystem(), root=str(tmp_path)) == FsspecStorage( + LocalFileSystem(), root=str(tmp_path) + ) + + +# ---------------------------------------------------------------------- the real fsspec + + +def _parameters(method: Any) -> list[str]: + return list(inspect.signature(method).parameters)[1:] + + +@pytest.mark.parametrize("filesystem", [AbstractFileSystem, LocalFileSystem, MemoryFileSystem]) +def test_fsspec_has_the_calls_the_adapter_makes(filesystem: type) -> None: + assert _parameters(filesystem.info)[0] == "path" + assert _parameters(filesystem.ls)[:2] == ["path", "detail"] + assert len(_parameters(filesystem.put_file)) >= 2 + assert len(_parameters(filesystem.get_file)) >= 2 + assert _parameters(filesystem.rm_file) == ["path"] + assert _parameters(filesystem.makedirs) == ["path", "exist_ok"] + assert _parameters(filesystem.rmdir) == ["path"] + assert _parameters(filesystem.cp_file)[:2] == ["path1", "path2"] + assert _parameters(filesystem.mv)[:2] == ["path1", "path2"] + assert _parameters(filesystem.cat_file)[0] == "path" + assert _parameters(filesystem.modified) == ["path"] + assert isinstance(filesystem.root_marker, str) + assert callable(filesystem._strip_protocol) + + +def test_fsspec_resolves_a_url_to_a_filesystem_and_a_path() -> None: + assert _parameters(AbstractFileSystem.put_file)[:2] == ["lpath", "rpath"] + assert _parameters(AbstractFileSystem.get_file)[:2] == ["rpath", "lpath"] + assert list(inspect.signature(url_to_fs).parameters) == ["url", "kwargs"] + filesystem, path = url_to_fs("memory://jobs/2026") + assert isinstance(filesystem, MemoryFileSystem) + assert path == "/jobs/2026" + assert MemoryFileSystem() is MemoryFileSystem() + with pytest.raises(NotImplementedError): + AbstractFileSystem(skip_instance_cache=True).modified("a.txt") + + +def test_two_roots_of_one_filesystem_do_not_lose_a_file_to_itself() -> None: + whole = FsspecStorage(MemoryFileSystem()) + inner = FsspecStorage(MemoryFileSystem(), root="team/a") + whole.mkdir("team/a") + inner.write_bytes("docs/a.txt", b"payload") + for operation in (whole.move_from, whole.copy_from): + with pytest.raises(StorageException, match="same file"): + operation(inner, "docs/a.txt", "team/a/docs/a.txt") + with pytest.raises(StorageException, match="same file"): + inner.move_from(whole, "team/a/docs/a.txt", "docs/a.txt") + assert whole.read_bytes("team/a/docs/a.txt") == b"payload" diff --git a/tests/test_storage_ftp.py b/tests/test_storage_ftp.py new file mode 100644 index 0000000..d8f8f8a --- /dev/null +++ b/tests/test_storage_ftp.py @@ -0,0 +1,865 @@ +"""FTPStorage: the storage contract against an FTP server held in memory. + +The session is a real ``ftplib.FTP`` connected to the in-memory server of +``tests/ftp_stand_in.py``, so ftplib builds the commands, parses the replies and +raises its own exceptions. The server comes in the two kinds the adapter tells +apart: one that offers ``MLST`` / ``MLSD`` and one that has to be probed with +``CWD``, ``SIZE``, ``MDTM`` and ``NLST``. +""" + +from __future__ import annotations + +import errno +import ftplib # nosec B402 - the sessions under test lead to an in-memory server +import inspect +import io +import posixpath +import re +import ssl +from pathlib import Path +from typing import Any + +import pytest + +from automation_file.exceptions import ( + StorageException, + StorageNotFoundException, + StoragePermissionException, + StorageTransientException, + StorageUnavailableException, + StorageURIException, +) +from automation_file.remote.ftp.client import FTPClient, ftp_instance +from automation_file.storage import File, Storage, StorageBackend, StorageResolver +from automation_file.storage.ftp_storage import FTP_SCHEME, FTPS_SCHEME, FTPStorage, ftp_factory +from tests._insecure_fixtures import insecure_url +from tests.ftp_stand_in import MODIFIED, FakeFTP, FakeFTPS, FakeFTPServer +from tests.storage_contract import StorageContract + +HOST = "files.example" + + +def ftp(rest: str) -> str: + """Return the ``ftp`` URI of ``rest``. + + Assembled from parts, so that no clear-text scheme literal is written in this + module (SonarCloud python:S5332). The URIs lead to the in-memory server. + """ + return insecure_url(FTP_SCHEME, rest) + + +def connected( + session: ftplib.FTP | None, host: str | None = HOST, port: int | None = 21 +) -> FTPClient: + """Return an ``FTPClient`` whose session is ``session``, as ``later_init`` leaves it.""" + client = FTPClient() + client._ftp = session + client._host = host + client._port = port + return client + + +def storage_on(server: FakeFTPServer, root: str = "/") -> FTPStorage: + return FTPStorage(connected(FakeFTP(server)), root=root) + + +class TestFTPStorageContract(StorageContract): + @pytest.fixture + def backend(self) -> StorageBackend: + return storage_on(FakeFTPServer()) + + +class TestFTPStorageWithoutMlstContract(StorageContract): + @pytest.fixture + def backend(self) -> StorageBackend: + return storage_on(FakeFTPServer(mlst=False)) + + +class TestRootedFTPStorageContract(StorageContract): + @pytest.fixture + def backend(self) -> StorageBackend: + server = FakeFTPServer() + server.mkdir("/srv") + server.mkdir("/srv/data") + server.write("/srv/keep.txt", b"keep") + return storage_on(server, "/srv/data") + + +class TestRootedFTPStorageWithoutMlstContract(StorageContract): + """The probed kind as vsftpd and older servers behave: paths from NLST, 550 when empty.""" + + @pytest.fixture + def backend(self) -> StorageBackend: + server = FakeFTPServer(mlst=False, nlst_paths=True, empty_nlst_refused=True) + server.mkdir("/home") + server.mkdir("/srv") + server.mkdir("/srv/data") + server.write("/srv/keep.txt", b"keep") + server.cwd = "/home" + return storage_on(server, "/srv/data") + + +class TestFTPStorageThatCannotRenameOverAFileContract(StorageContract): + @pytest.fixture + def backend(self) -> StorageBackend: + return storage_on(FakeFTPServer(rename_replaces=False)) + + +@pytest.fixture +def server() -> FakeFTPServer: + return FakeFTPServer() + + +@pytest.fixture +def storage(server: FakeFTPServer) -> FTPStorage: + return storage_on(server) + + +@pytest.fixture +def plain_server() -> FakeFTPServer: + """A server without MLST, whose client stands in a home directory.""" + server = FakeFTPServer(mlst=False) + server.mkdir("/home") + server.cwd = "/home" + return server + + +@pytest.fixture +def shared(monkeypatch: pytest.MonkeyPatch, server: FakeFTPServer) -> FakeFTPServer: + """Make a session with ``server`` the open session of the shared ``ftp_instance``.""" + monkeypatch.setattr(ftp_instance, "_ftp", FakeFTP(server)) + monkeypatch.setattr(ftp_instance, "_host", HOST) + monkeypatch.setattr(ftp_instance, "_port", 21) + return server + + +@pytest.fixture +def resolver() -> StorageResolver: + table = StorageResolver() + table.register_scheme(FTP_SCHEME, ftp_factory) + table.register_scheme(FTPS_SCHEME, ftp_factory) + return table + + +# ---------------------------------------------------------------------- a server with MLST + + +def test_stat_reads_the_facts_of_an_mlst_reply(storage: FTPStorage, server: FakeFTPServer) -> None: + server.mkdir("/reports") + server.write("/reports/q1 final.csv", b"a,b\n") + info = storage.stat("reports/q1 final.csv") + assert (info.path, info.is_dir, info.size) == ("reports/q1 final.csv", False, 4) + assert info.modified_at == MODIFIED + assert (info.etag, info.version, info.content_type, dict(info.metadata)) == ( + None, + None, + None, + {}, + ) + directory = storage.stat("reports") + assert (directory.is_dir, directory.size, directory.modified_at) == (True, None, MODIFIED) + assert server.commands[-1] == "MLST /reports" + assert storage.capabilities.to_dict() == { + "directories": True, + "modified_at": True, + "etag": False, + "version": False, + "content_type": False, + "metadata": False, + } + + +def test_the_server_is_asked_for_its_features_once_for_each_session(server: FakeFTPServer) -> None: + session = FakeFTP(server) + client = connected(session) + FTPStorage(client).write_bytes("a.txt", b"x") + FTPStorage(client, root="/").list_dir() + assert server.commands[:2] == ["FEAT", "OPTS MLST type;size;modify;"] + assert server.verbs().count("FEAT") == 1 + FTPStorage(connected(FakeFTP(server))).exists("a.txt") + assert server.verbs().count("FEAT") == 2 + assert not {"CWD", "PWD", "SIZE", "MDTM", "NLST"} & set(server.verbs()) + + +def test_a_listing_skips_the_directory_itself_and_reads_a_link_as_a_file( + storage: FTPStorage, server: FakeFTPServer +) -> None: + server.mkdir("/dir") + server.write("/dir/a.txt", b"abc") + server.mkdir("/dir/sub") + server.extra_listing = ["type=OS.unix=slink:/elsewhere;size=10;modify=20261008023015; link"] + assert [(info.path, info.is_dir, info.size) for info in storage.list_dir("dir")] == [ + ("dir/a.txt", False, 3), + ("dir/link", False, 10), + ("dir/sub", True, None), + ] + + +@pytest.mark.parametrize( + "reply", + ["250 Nothing to see", "250-Listing\ntype=file;size=3; /a.txt\n250 End"], +) +def test_an_mlst_reply_without_an_entry_line_is_an_error( + storage: FTPStorage, server: FakeFTPServer, reply: str +) -> None: + storage.exists("a.txt") + server.refusals["MLST"] = reply + with pytest.raises(StorageException) as caught: + storage.stat("a.txt") + assert type(caught.value) is StorageException + assert isinstance(caught.value.__cause__, ftplib.error_reply) + + +def test_facts_a_server_leaves_out_stay_unknown(storage: FTPStorage, server: FakeFTPServer) -> None: + storage.exists("a.txt") + server.refusals["MLST"] = "250-Listing\n modify=2026; /a.txt\n250 End" + info = storage.stat("a.txt") + assert (info.is_dir, info.size, info.modified_at) == (False, None, None) + + +# ---------------------------------------------------------------------- a server without MLST + + +def test_a_server_without_mlst_is_probed(plain_server: FakeFTPServer) -> None: + storage = storage_on(plain_server) + plain_server.mkdir("/reports") + plain_server.write("/reports/q1.csv", b"a,b\n") + info = storage.stat("reports/q1.csv") + assert (info.is_dir, info.size) == (False, 4) + assert info.modified_at == MODIFIED.replace(microsecond=0) + assert plain_server.commands == [ + "FEAT", + "PWD", + "TYPE I", + "CWD /reports/q1.csv", + "SIZE /reports/q1.csv", + "MDTM /reports/q1.csv", + ] + plain_server.commands.clear() + directory = storage.stat("reports") + assert (directory.is_dir, directory.size, directory.modified_at) == (True, None, None) + assert plain_server.commands == ["PWD", "TYPE I", "CWD /reports", "CWD /home"] + assert storage.exists("reports/nope.csv") is False + assert not {"MLST", "MLSD", "OPTS"} & set(plain_server.verbs()) + + +def test_probing_puts_the_working_directory_back(plain_server: FakeFTPServer) -> None: + storage = storage_on(plain_server) + for path in ("dir/sub/a.txt", "dir/b.txt"): + storage.write_bytes(path, b"x") + assert [(info.path, info.is_dir) for info in storage.list_dir("dir")] == [ + ("dir/b.txt", False), + ("dir/sub", True), + ] + storage.delete("dir", recursive=True) + assert plain_server.cwd == "/home" + assert plain_server.paths() == ["/home"] + + +def test_a_server_without_feat_is_probed() -> None: + server = FakeFTPServer(mlst=False) + server.refusals["FEAT"] = "502 Command not implemented." + storage = storage_on(server) + assert storage.write_bytes("a.txt", b"abc").size == 3 + assert "MLST" not in server.verbs() + + +def test_a_server_that_will_not_switch_the_facts_on_is_probed() -> None: + server = FakeFTPServer() + server.refusals["OPTS"] = "501 Option not understood." + storage = storage_on(server) + assert storage.write_bytes("a.txt", b"abc").size == 3 + assert "MLST" not in server.verbs() + + +def test_a_listed_name_the_server_will_not_describe_is_still_listed( + plain_server: FakeFTPServer, +) -> None: + storage = storage_on(plain_server) + plain_server.write("/locked.bin", b"secret") + plain_server.refusals["SIZE"] = "550 Permission denied." + listed = {info.path: info for info in storage.list_dir()} + assert (listed["locked.bin"].is_dir, listed["locked.bin"].size) == (False, None) + assert listed["home"].is_dir is True + assert storage.exists("locked.bin") is False + + +def test_a_server_without_mdtm_reports_no_modification_time(plain_server: FakeFTPServer) -> None: + storage = storage_on(plain_server) + plain_server.refusals["MDTM"] = "502 Command not implemented." + info = storage.write_bytes("a.txt", b"abc") + assert (info.size, info.modified_at) == (3, None) + + +# ---------------------------------------------------------------------- paths and URIs + + +def test_every_path_is_joined_to_the_root(server: FakeFTPServer) -> None: + server.mkdir("/srv") + server.mkdir("/srv/data") + server.write("/srv/keep.txt", b"keep") + rooted = storage_on(server, "srv//data/") + assert rooted.root == "/srv/data" + rooted.write_bytes("docs/a.txt", b"x") + assert server.paths() == [ + "/srv", + "/srv/data", + "/srv/data/docs", + "/srv/data/docs/a.txt", + "/srv/keep.txt", + ] + assert [info.path for info in rooted.list_dir("", recursive=True)] == ["docs", "docs/a.txt"] + assert rooted.uri_for("docs/a.txt") == ftp("files.example/srv/data/docs/a.txt") + rooted.delete("docs", recursive=True) + assert server.paths() == ["/srv", "/srv/data", "/srv/keep.txt"] + with pytest.raises(StorageURIException): + storage_on(server, "/srv/../etc") + + +@pytest.mark.parametrize("path", ["a.txt\r\nDELE /b.txt", "dir/a\nb.txt", "a\r.txt"]) +def test_a_path_with_a_line_break_is_refused( + storage: FTPStorage, server: FakeFTPServer, path: str +) -> None: + for call in (storage.exists, storage.delete, lambda name: storage.write_bytes(name, b"x")): + with pytest.raises(StorageURIException, match="line break"): + call(path) + with pytest.raises(StorageURIException, match="line break"): + storage_on(server, "/srv\r\nDELE /b.txt") + assert server.commands == [] + + +def test_uri_for_names_the_host_of_the_session(server: FakeFTPServer) -> None: + session = FakeFTP(server) + assert FTPStorage(connected(session)).uri_for("data/a.txt") == ftp("files.example/data/a.txt") + assert FTPStorage(connected(session)).uri_for("") == ftp("files.example") + assert FTPStorage(connected(session, port=2121)).uri_for("a.txt") == ( + ftp("files.example:2121/a.txt") + ) + assert FTPStorage(connected(FakeFTPS(server))).uri_for("a.txt") == "ftps://files.example/a.txt" + assert FTPStorage(connected(None, host=None, port=None)).uri_for("a.txt") == ftp("/a.txt") + + +def test_equality_and_repr(server: FakeFTPServer) -> None: + client = connected(FakeFTP(server)) + assert FTPStorage(client) == FTPStorage(client) + assert FTPStorage(client) != FTPStorage(client, root="/srv") + assert FTPStorage(client) != FTPStorage(connected(FakeFTP(server))) + assert FTPStorage() == FTPStorage(ftp_instance) + assert len({FTPStorage(client), FTPStorage(client, root="/")}) == 1 + assert repr(FTPStorage(client, root="srv/data")) == "FTPStorage(root='/srv/data')" + assert FTPStorage().client is ftp_instance + + +# ---------------------------------------------------------------------- errors + + +@pytest.mark.parametrize( + "reply,expected,cause", + [ + ("530 Please login with USER and PASS.", StoragePermissionException, ftplib.error_perm), + ( + "421 Service not available, closing control connection.", + StorageTransientException, + ftplib.error_temp, + ), + ("450 Requested file action not taken.", StorageTransientException, ftplib.error_temp), + ("350 Unexpected", StorageException, ftplib.error_reply), + ("999 Not a reply", StorageException, ftplib.error_proto), + ], +) +def test_replies_become_storage_errors( + storage: FTPStorage, + server: FakeFTPServer, + reply: str, + expected: type[Exception], + cause: type[Exception], +) -> None: + server.write("/a.txt", b"x") + assert storage.exists("a.txt") is True + server.refusals["DELE"] = reply + with pytest.raises(expected) as caught: + storage.delete("a.txt") + assert type(caught.value) is expected + assert type(caught.value.__cause__) is cause + assert str(caught.value.__cause__) == reply + assert ftp("files.example/a.txt") in str(caught.value) + + +@pytest.mark.parametrize( + "error,expected", + [ + (TimeoutError("timed out"), StorageTransientException), + (ConnectionResetError(errno.ECONNRESET, "reset by peer"), StorageTransientException), + (BrokenPipeError(errno.EPIPE, "broken pipe"), StorageTransientException), + (ssl.SSLError("decryption failed"), StorageException), + (OSError("unexpected"), StorageException), + ], +) +def test_connection_errors_become_storage_errors( + storage: FTPStorage, server: FakeFTPServer, error: Exception, expected: type[Exception] +) -> None: + server.send_error = error + with pytest.raises(expected) as caught: + storage.stat("a.txt") + assert type(caught.value) is expected + assert caught.value.__cause__ is error + + +@pytest.mark.parametrize("mlst", [True, False]) +def test_a_name_in_another_encoding_is_a_storage_error(mlst: bool) -> None: + server = FakeFTPServer(mlst=mlst) + storage = storage_on(server) + server.raw_listing = b"caf\xe9.txt\r\n" + with pytest.raises(StorageException) as caught: + storage.list_dir() + assert type(caught.value) is StorageException + assert isinstance(caught.value.__cause__, UnicodeDecodeError) + + +def test_a_closed_connection_is_transient(storage: FTPStorage, server: FakeFTPServer) -> None: + server.hung_up = True + with pytest.raises(StorageTransientException) as caught: + storage.list_dir() + assert isinstance(caught.value.__cause__, EOFError) + + +def test_550_means_absent_in_a_lookup_and_refused_elsewhere( + storage: FTPStorage, server: FakeFTPServer +) -> None: + assert storage.exists("nope.txt") is False + with pytest.raises(StorageNotFoundException, match=re.escape(ftp("files.example/nope"))): + storage.list_dir("nope") + server.write("/a.txt", b"x") + for verb in ("DELE", "RETR"): + server.refusals[verb] = "550 Permission denied." + with pytest.raises(StoragePermissionException, match=r"denied \(550\)"): + storage.delete("a.txt") + with pytest.raises(StoragePermissionException): + storage.read_bytes("a.txt") + server.refusals["MLST"] = "550 Permission denied." + assert storage.exists("a.txt") is False + + +def test_another_refusal_in_a_lookup_is_not_read_as_absent( + storage: FTPStorage, server: FakeFTPServer, plain_server: FakeFTPServer +) -> None: + refusal = "530 Please login with USER and PASS." + storage.exists("a.txt") + server.refusals["MLST"] = refusal + with pytest.raises(StoragePermissionException, match=r"denied \(530\)"): + storage.exists("a.txt") + + probed = storage_on(plain_server) + plain_server.write("/a.txt", b"x") + for verb in ("CWD", "SIZE", "NLST"): + plain_server.refusals = {verb: refusal} + with pytest.raises(StoragePermissionException, match=r"denied \(530\)"): + probed.list_dir() + + +def test_a_session_must_be_open(monkeypatch: pytest.MonkeyPatch) -> None: + with pytest.raises(StorageUnavailableException, match="later_init"): + FTPStorage(FTPClient()).exists("a.txt") + monkeypatch.setattr(ftp_instance, "_ftp", None) + with pytest.raises(StorageUnavailableException, match="later_init") as caught: + FTPStorage().write_bytes("a.txt", b"x") + assert type(caught.value.__cause__).__name__ == "FTPException" + + +# ---------------------------------------------------------------------- uploads and moves + + +def test_an_upload_arrives_under_a_part_name_and_is_renamed( + storage: FTPStorage, server: FakeFTPServer +) -> None: + storage.write_bytes("a.txt", b"payload") + stor, rename_from, rename_to = [ + command for command in server.commands if command[:4] in ("STOR", "RNFR", "RNTO") + ] + partial = stor.removeprefix("STOR ") + assert posixpath.dirname(partial) == "/" + assert posixpath.basename(partial).startswith(".a.txt.") + assert partial.endswith(".part") + assert (rename_from, rename_to) == (f"RNFR {partial}", "RNTO /a.txt") + assert server.paths() == ["/a.txt"] + + +@pytest.mark.parametrize( + "reply,expected", + [ + ("426 Connection closed; transfer aborted.", StorageTransientException), + ( + "552 Requested file action aborted. Exceeded storage allocation.", + StoragePermissionException, + ), + ], +) +def test_a_failed_upload_leaves_no_partial_file_and_keeps_the_target( + storage: FTPStorage, server: FakeFTPServer, reply: str, expected: type[Exception] +) -> None: + server.write("/a.txt", b"old") + server.transfer_error = reply + with pytest.raises(expected) as caught: + storage.write_bytes("a.txt", b"new and longer") + assert str(caught.value.__cause__) == reply + assert server.verbs()[-1] == "DELE" + assert server.paths() == ["/a.txt"] + assert server.read("/a.txt") == b"old" + + +def test_a_refused_rename_leaves_no_partial_file_and_keeps_the_target( + storage: FTPStorage, server: FakeFTPServer +) -> None: + server.write("/a.txt", b"old") + server.refusals["RNFR"] = "550 Permission denied." + with pytest.raises(StoragePermissionException) as caught: + storage.write_bytes("a.txt", b"new") + assert str(caught.value.__cause__) == "550 Permission denied." + assert server.paths() == ["/a.txt"] + assert server.read("/a.txt") == b"old" + + +def test_the_upload_error_is_reported_when_the_cleanup_fails_too( + storage: FTPStorage, server: FakeFTPServer +) -> None: + server.transfer_error = "426 Connection closed; transfer aborted." + server.refusals["DELE"] = "421 Timeout." + with pytest.raises(StorageTransientException) as caught: + storage.write_bytes("a.txt", b"payload") + assert str(caught.value.__cause__) == server.transfer_error + assert [posixpath.basename(path)[:7] for path in server.paths()] == [".a.txt."] + + +@pytest.fixture +def strict_server() -> FakeFTPServer: + """A server that will not rename onto an existing file, as FTP servers on Windows do.""" + return FakeFTPServer(rename_replaces=False) + + +def test_a_server_that_will_not_rename_over_a_file_has_it_moved_aside( + strict_server: FakeFTPServer, +) -> None: + storage = storage_on(strict_server) + storage.write_bytes("a.txt", b"first") + assert "DELE" not in strict_server.verbs() + strict_server.commands.clear() + storage.write_bytes("a.txt", b"second") + commands = strict_server.commands + partial = next(c for c in commands if c.startswith("STOR ")).removeprefix("STOR ") + aside = next(c for c in commands if c.endswith(".old")).removeprefix("RNTO ") + assert posixpath.basename(aside).startswith(".a.txt.") + assert commands[commands.index(f"RNFR {partial}") :] == [ + f"RNFR {partial}", + "RNTO /a.txt", + "RNFR /a.txt", + f"RNTO {aside}", + f"RNFR {partial}", + "RNTO /a.txt", + f"DELE {aside}", + "MLST /a.txt", + ] + assert strict_server.paths() == ["/a.txt"] + assert strict_server.read("/a.txt") == b"second" + + +def test_a_rename_refused_for_another_reason_keeps_the_target( + storage: FTPStorage, server: FakeFTPServer +) -> None: + server.refusals["RNTO"] = "553 Rename not allowed." + with pytest.raises(StoragePermissionException) as caught: + storage.write_bytes("new.txt", b"payload") + assert str(caught.value.__cause__) == "553 Rename not allowed." + assert server.paths() == [] + + server.write("/a.txt", b"old") + with pytest.raises(StoragePermissionException) as caught: + storage.write_bytes("a.txt", b"payload") + assert str(caught.value.__cause__) == "553 Rename not allowed." + assert server.paths() == ["/a.txt"] + assert server.read("/a.txt") == b"old" + + server.write("/b.txt", b"other") + server.refusals = {"RNFR": "550 RNFR command failed."} + with pytest.raises(StoragePermissionException): + storage.move_from(storage, "b.txt", "a.txt") + assert (server.read("/a.txt"), server.read("/b.txt")) == (b"old", b"other") + assert "DELE" not in server.verbs()[-4:] + + +@pytest.mark.parametrize( + "reply,expected", + [ + ("553 Could not rename.", StoragePermissionException), + ("451 Requested action aborted: local error in processing.", StorageTransientException), + ], +) +def test_the_old_file_is_put_back_when_the_rename_still_fails( + strict_server: FakeFTPServer, reply: str, expected: type[Exception] +) -> None: + storage = storage_on(strict_server) + strict_server.write("/a.txt", b"old") + # RNTO: onto the file (refused), the file aside, onto the freed name, the file back. + strict_server.answer_the_nth("RNTO", 3, reply) + with pytest.raises(expected) as caught: + storage.write_bytes("a.txt", b"new") + assert str(caught.value.__cause__) == reply + assert strict_server.verbs().count("RNTO") == 4 + assert strict_server.paths() == ["/a.txt"] + assert strict_server.read("/a.txt") == b"old" + + +def test_the_old_content_stays_aside_when_it_cannot_be_put_back( + strict_server: FakeFTPServer, +) -> None: + storage = storage_on(strict_server) + strict_server.write("/a.txt", b"old") + for ordinal in (3, 4): + strict_server.answer_the_nth("RNTO", ordinal, "553 Could not rename.") + with pytest.raises(StoragePermissionException): + storage.write_bytes("a.txt", b"new") + (aside,) = strict_server.paths() + assert posixpath.basename(aside).startswith(".a.txt.") + assert aside.endswith(".old") + assert strict_server.read(aside) == b"old" + + +def test_a_move_within_one_session_is_a_rename(storage: FTPStorage, server: FakeFTPServer) -> None: + storage.write_bytes("a.txt", b"payload") + server.commands.clear() + info = storage.move_from(storage, "a.txt", "moved/b.txt") + assert (info.path, info.size) == ("moved/b.txt", 7) + assert server.commands[-3:-1] == ["RNFR /a.txt", "RNTO /moved/b.txt"] + assert not {"RETR", "STOR", "DELE"} & set(server.verbs()) + assert server.paths() == ["/moved", "/moved/b.txt"] + + archive = FTPStorage(storage.client, root="/moved") + archive.move_from(storage, "moved/b.txt", "c.txt") + assert server.paths() == ["/moved", "/moved/c.txt"] + with pytest.raises(StorageException, match="same file"): + archive.move_from(storage, "moved/c.txt", "c.txt") + assert server.read("/moved/c.txt") == b"payload" + + +def test_a_move_between_two_sessions_is_copy_then_delete( + storage: FTPStorage, server: FakeFTPServer +) -> None: + other_server = FakeFTPServer() + other = FTPStorage(connected(FakeFTP(other_server), host="backup.example")) + storage.write_bytes("a.txt", b"payload") + other.move_from(storage, "a.txt", "a.txt") + assert other_server.read("/a.txt") == b"payload" + assert server.paths() == [] + assert "RETR" in server.verbs() + + +def test_mkdir_accepts_a_directory_that_appeared_meanwhile( + storage: FTPStorage, server: FakeFTPServer +) -> None: + server.mkdir("/dir") + storage._mkdir("dir") + server.write("/a.txt", b"x") + with pytest.raises(StoragePermissionException): + storage._mkdir("a.txt") + + +# ---------------------------------------------------------------------- symbolic links + + +@pytest.fixture(params=[True, False], ids=["mlst", "probed"]) +def linked(request: pytest.FixtureRequest) -> tuple[FTPStorage, FakeFTPServer]: + """``/data`` with a link to a file and a link to a directory outside it, on both kinds.""" + server = FakeFTPServer(mlst=request.param) + storage = storage_on(server) + for path in ("outside/keep.txt", "data/real.txt"): + storage.write_bytes(path, b"content") + server.link("/data/file-link", "real.txt") + server.link("/data/dir-link", "/outside") + return storage, server + + +def test_reading_follows_a_link(linked: tuple[FTPStorage, FakeFTPServer]) -> None: + storage, _ = linked + assert storage.read_bytes("data/file-link") == b"content" + assert storage.read_bytes("data/dir-link/keep.txt") == b"content" + assert [info.name for info in storage.list_dir("data")] == [ + "dir-link", + "file-link", + "real.txt", + ] + + +def test_deleting_never_follows_a_link(linked: tuple[FTPStorage, FakeFTPServer]) -> None: + storage, server = linked + storage.delete("data/dir-link", recursive=True) + assert "/data/dir-link" not in server.entries + assert server.read("/outside/keep.txt") == b"content" + storage.delete("data/file-link") + assert server.read("/data/real.txt") == b"content" + + server.link("/data/dir-link", "/outside") + storage.delete("data", recursive=True) + assert server.paths() == ["/outside", "/outside/keep.txt"] + + +def test_a_recursive_delete_asks_for_no_more_than_it_needs() -> None: + server = FakeFTPServer(mlst=False) + storage = storage_on(server) + for path in ("dir/a.txt", "dir/sub/b.txt"): + storage.write_bytes(path, b"x") + server.commands.clear() + storage.delete("dir", recursive=True) + assert server.commands[server.commands.index("DELE /dir") :] == [ + "DELE /dir", + "TYPE A", + "NLST /dir", + "DELE /dir/a.txt", + "DELE /dir/sub", + "TYPE A", + "NLST /dir/sub", + "DELE /dir/sub/b.txt", + "RMD /dir/sub", + "RMD /dir", + ] + assert server.paths() == [] + + +# ---------------------------------------------------------------------- ftp:// and ftps:// URIs + + +def test_ftp_uris_use_the_shared_session( + shared: FakeFTPServer, resolver: StorageResolver, tmp_path: Path +) -> None: + report = File(ftp("files.example/reports/q1.csv"), resolver=resolver) + report.write(b"a,b\n") + assert shared.read("/reports/q1.csv") == b"a,b\n" + assert resolver.resolve(ftp("files.example/reports/q1.csv")) == ( + FTPStorage(), + "reports/q1.csv", + ) + report.copy_to(tmp_path / "q1.csv") + assert (tmp_path / "q1.csv").read_bytes() == b"a,b\n" + File(tmp_path / "q1.csv", resolver=resolver).move_to(ftp("/archive/2026/q1.csv")) + assert shared.read("/archive/2026/q1.csv") == b"a,b\n" + archive = Storage(ftp("files.example/archive"), resolver=resolver) + assert [info.path for info in archive.list_dir(recursive=True)] == ["2026", "2026/q1.csv"] + archive.delete("2026", recursive=True) + assert shared.paths() == ["/archive", "/reports", "/reports/q1.csv"] + + +@pytest.mark.parametrize( + "uri", + [ + ftp("/data/a.txt"), + ftp("files.example/data/a.txt"), + ftp("FILES.example/data/a.txt"), + ftp("files.example:21/data/a.txt"), + ], +) +def test_a_uri_may_name_no_host_or_the_connected_one( + shared: FakeFTPServer, resolver: StorageResolver, uri: str +) -> None: + assert resolver.resolve(uri) == (FTPStorage(), "data/a.txt") + + +@pytest.mark.parametrize( + "uri,named", + [ + (ftp("backup.example/data/a.txt"), "backup.example"), + (ftp("files.example:2121/data/a.txt"), "files.example:2121"), + ], +) +def test_a_uri_for_another_host_is_refused( + shared: FakeFTPServer, resolver: StorageResolver, uri: str, named: str +) -> None: + with pytest.raises(StorageURIException) as caught: + resolver.resolve(uri) + message = str(caught.value) + assert repr(named) in message + assert "'files.example:21'" in message + assert f'Storage.mount("ftp://{named}", FTPStorage(client))' in message + + +def test_ftps_needs_an_ftps_session( + shared: FakeFTPServer, resolver: StorageResolver, monkeypatch: pytest.MonkeyPatch +) -> None: + assert ftp_instance.tls is False + for uri in ("ftps://files.example/data/a.txt", "ftps:///data/a.txt"): + with pytest.raises(StorageURIException, match="tls=True"): + resolver.resolve(uri) + with pytest.raises(StorageURIException, match=re.escape("backup.example")): + resolver.resolve("ftps://backup.example/data/a.txt") + + monkeypatch.setattr(ftp_instance, "_ftp", FakeFTPS(shared)) + assert ftp_instance.tls is True + for uri in ("ftps://files.example/data/a.txt", "ftps:///data/a.txt", ftp("/data/a.txt")): + assert resolver.resolve(uri) == (FTPStorage(), "data/a.txt") + File("ftps://files.example/data/a.txt", resolver=resolver).write(b"x") + assert shared.read("/data/a.txt") == b"x" + assert resolver.resolve("ftps:///data/a.txt")[0].uri_for("data/a.txt") == ( + "ftps://files.example/data/a.txt" + ) + + +def test_any_host_resolves_until_a_session_is_open( + monkeypatch: pytest.MonkeyPatch, resolver: StorageResolver +) -> None: + for name in ("_ftp", "_host", "_port"): + monkeypatch.setattr(ftp_instance, name, None) + for uri in (ftp("anywhere.example/data/a.txt"), "ftps://anywhere.example/data/a.txt"): + backend, path = resolver.resolve(uri) + assert (backend, path) == (FTPStorage(), "data/a.txt") + with pytest.raises(StorageUnavailableException, match="later_init"): + backend.exists(path) + + +def test_another_host_is_reached_through_a_mount( + shared: FakeFTPServer, resolver: StorageResolver +) -> None: + backup = FakeFTPServer() + resolver.mount( + ftp("backup.example"), FTPStorage(connected(FakeFTP(backup), host="backup.example")) + ) + File(ftp("files.example/a.txt"), resolver=resolver).write(b"payload") + File(ftp("files.example/a.txt"), resolver=resolver).copy_to( + File(ftp("backup.example/copies/a.txt"), resolver=resolver) + ) + assert backup.read("/copies/a.txt") == b"payload" + assert shared.paths() == ["/a.txt"] + + +# ---------------------------------------------------------------------- the real ftplib + + +@pytest.mark.parametrize( + "method,arguments", + [ + ("sendcmd", ("MLST /a",)), + ("voidcmd", ("TYPE I",)), + ("mlsd", ("/a",)), + ("nlst", ("/a",)), + ("size", ("/a",)), + ("pwd", ()), + ("cwd", ("/a",)), + ("storbinary", ("STOR /a", io.BytesIO())), + ("retrbinary", ("RETR /a", print)), + ("delete", ("/a",)), + ("mkd", ("/a",)), + ("rmd", ("/a",)), + ("rename", ("/a", "/b")), + ], +) +def test_ftplib_has_the_calls_the_adapter_makes(method: str, arguments: tuple[Any, ...]) -> None: + for session_type in (ftplib.FTP, ftplib.FTP_TLS): + inspect.signature(getattr(session_type, method)).bind(None, *arguments) + + +def test_ftplib_has_the_seams_the_stand_in_replaces() -> None: + assert list(inspect.signature(ftplib.FTP.ntransfercmd).parameters) == ["self", "cmd", "rest"] + assert list(inspect.signature(ftplib.FTP.putline).parameters) == ["self", "line"] + assert list(inspect.signature(ftplib.FTP.getline).parameters) == ["self"] + session = ftplib.FTP() # nosec B321 - never connected: only its attributes are read + assert (session.sock, session.file, session.encoding) == (None, None, "utf-8") + assert ftplib.error_perm("550 No such file").args[0][:3] == "550" + for error in (ftplib.error_perm, ftplib.error_temp, ftplib.error_reply, ftplib.error_proto): + assert issubclass(error, ftplib.Error) + assert not issubclass(error, OSError) diff --git a/tests/test_storage_gdrive.py b/tests/test_storage_gdrive.py new file mode 100644 index 0000000..f63c8d2 --- /dev/null +++ b/tests/test_storage_gdrive.py @@ -0,0 +1,787 @@ +"""GoogleDriveStorage: the storage contract against an in-memory Drive behind the real client. + +The stand-in (``tests/drive_stand_in.py``) is the HTTP transport of a real +``googleapiclient`` service, built from the Drive v3 discovery document the +package ships. Every ``files()`` call the adapter makes is therefore the real +one, and the last section names what that pins and what it leaves open. +""" + +from __future__ import annotations + +import hashlib +import inspect +import ssl +from datetime import datetime, timedelta, timezone +from pathlib import Path +from typing import Any + +import pytest + +pytest.importorskip("googleapiclient", reason="needs the gdrive extra") + +# pylint: disable=wrong-import-position # importorskip must precede these imports +import httplib2 +from google.auth import exceptions as auth_errors +from googleapiclient import http as googleapiclient_http +from googleapiclient.discovery import build +from googleapiclient.errors import HttpError +from googleapiclient.http import MediaFileUpload, MediaIoBaseDownload + +from automation_file.exceptions import ( + StorageException, + StorageNotEmptyException, + StorageNotFoundException, + StoragePermissionException, + StorageTransientException, + StorageUnavailableException, + StorageUnsupportedException, + StorageURIException, +) +from automation_file.remote.google_drive.client import ( + GoogleDriveClient, + driver_instance, +) +from automation_file.storage import ( + File, + StorageBackend, + StorageResolver, + gdrive_storage, +) +from automation_file.storage.gdrive_storage import ( + GDRIVE_SCHEME, + GoogleDriveStorage, + gdrive_factory, +) +from tests.drive_stand_in import ( + BINARY_MIME_TYPE, + DISCOVERY, + DOCUMENT_MIME_TYPE, + FILE_METHODS, + FILE_SCHEMA, + MY_DRIVE_ID, + FakeDrive, + error_answer, +) +from tests.storage_contract import StorageContract + + +def _client(drive: FakeDrive) -> GoogleDriveClient: + """A GoogleDriveClient whose service is the real one, talking to the stand-in.""" + client = GoogleDriveClient() + client.service = build("drive", "v3", http=drive, static_discovery=True) + return client + + +class TestGoogleDriveStorageContract(StorageContract): + @pytest.fixture + def backend(self) -> StorageBackend: + return GoogleDriveStorage(_client(FakeDrive())) + + +class TestRootedGoogleDriveStorageContract(StorageContract): + @pytest.fixture + def backend(self) -> StorageBackend: + drive = FakeDrive() + drive.add_file("keep.txt", b"keep") + drive.add_file("keep.txt", b"keep", drive.add_folder("other-team")) + return GoogleDriveStorage(_client(drive), root_id=drive.add_folder("team")) + + +@pytest.fixture +def drive() -> FakeDrive: + return FakeDrive() + + +@pytest.fixture +def client(drive: FakeDrive) -> GoogleDriveClient: + return _client(drive) + + +@pytest.fixture +def storage(client: GoogleDriveClient) -> GoogleDriveStorage: + return GoogleDriveStorage(client) + + +# ---------------------------------------------------------------------- stat and listing + + +def test_stat_reports_what_drive_holds(storage: GoogleDriveStorage, drive: FakeDrive) -> None: + info = storage.write_bytes("reports/q1.json", b"{}") + assert info.path == "reports/q1.json" + assert info.size == 2 + assert info.etag == hashlib.md5(b"{}", usedforsecurity=False).hexdigest() + assert info.version == "1" + assert info.content_type == "application/json" + assert info.modified_at is not None + assert info.modified_at.utcoffset() == timedelta(0) + assert datetime.now(timezone.utc) - info.modified_at < timedelta(minutes=1) + assert dict(info.metadata) == {} + assert drive.only("q1.json").mime_type == "application/json" + + +def test_a_folder_is_a_directory_with_a_modification_time(storage: GoogleDriveStorage) -> None: + storage.mkdir("reports") + info = storage.stat("reports") + assert (info.is_dir, info.size, info.etag, info.content_type) == (True, None, None, None) + assert info.modified_at is not None + assert storage.stat("").is_dir is True + + +def test_a_name_without_a_known_suffix_is_uploaded_as_octet_stream( + storage: GoogleDriveStorage, drive: FakeDrive +) -> None: + assert storage.write_bytes("blob", b"x").content_type == BINARY_MIME_TYPE + opened = drive.last("create") + assert opened.headers["x-upload-content-type"] == BINARY_MIME_TYPE + assert opened.body == {"name": "blob", "parents": ["root"], "mimeType": BINARY_MIME_TYPE} + + +def test_listing_reads_every_page(storage: GoogleDriveStorage, drive: FakeDrive) -> None: + for index in range(7): + storage.write_bytes(f"dir/{index}.txt", b"x") + drive.seen.clear() + listing = storage.list_dir("dir") + assert [info.path for info in listing] == [f"dir/{index}.txt" for index in range(7)] + assert all(info.etag and info.size == 1 for info in listing) + # "dir" is looked up twice (nothing is remembered between calls), then come four pages. + assert drive.calls == ["list"] * 6 + assert len(storage.list_dir("", recursive=True)) == 8 + + +def test_recursive_listing_asks_once_per_folder( + storage: GoogleDriveStorage, drive: FakeDrive +) -> None: + storage.write_bytes("a/b/c.txt", b"x") + drive.seen.clear() + assert [info.path for info in storage.list_dir("", recursive=True)] == ["a", "a/b", "a/b/c.txt"] + # The root is checked with files.get; each of the three folders is listed once. + assert drive.calls == ["get", "list", "list", "list"] + + +def test_entries_in_the_trash_do_not_exist(storage: GoogleDriveStorage, drive: FakeDrive) -> None: + storage.write_bytes("dir/a.txt", b"x") + drive.only("a.txt").trashed = True + assert storage.exists("dir/a.txt") is False + assert storage.list_dir("dir") == [] + storage.write_bytes("dir/a.txt", b"new") + assert storage.read_bytes("dir/a.txt") == b"new" + + +def test_a_missing_root_folder_is_not_found(client: GoogleDriveClient) -> None: + missing = GoogleDriveStorage(client, root_id="no-such-folder") + assert missing.exists("") is False + assert missing.exists("a.txt") is False + with pytest.raises(StorageNotFoundException): + missing.list_dir() + with pytest.raises(StorageNotFoundException): + missing.write_bytes("a.txt", b"x") + + +def test_a_trashed_root_folder_is_not_found(client: GoogleDriveClient, drive: FakeDrive) -> None: + folder = drive.add_folder("old") + drive.entries[folder].trashed = True + assert GoogleDriveStorage(client, root_id=folder).exists("") is False + + +# ---------------------------------------------------------------------- names + + +def test_duplicate_names_are_refused(storage: GoogleDriveStorage, drive: FakeDrive) -> None: + folder = drive.add_folder("dir") + first = drive.add_file("a.txt", b"first", folder) + second = drive.add_file("a.txt", b"second", folder) + for attempt in ( + lambda: storage.stat("dir/a.txt"), + lambda: storage.exists("dir/a.txt"), + lambda: storage.read_bytes("dir/a.txt"), + lambda: storage.write_bytes("dir/a.txt", b"third"), + lambda: storage.delete("dir/a.txt"), + lambda: storage.delete("dir/a.txt", missing_ok=True), + lambda: storage.copy_from(storage, "dir/a.txt", "b.txt"), + ): + with pytest.raises(StorageException, match=r"2 entries share the name 'a\.txt'") as caught: + attempt() + assert type(caught.value) is StorageException + assert "gdrive:///dir/a.txt" in str(caught.value) + # Neither was picked: both are untouched, and a listing still shows both. + assert (drive.entries[first].data, drive.entries[second].data) == (b"first", b"second") + assert [info.path for info in storage.list_dir("dir")] == ["dir/a.txt", "dir/a.txt"] + assert storage.exists("b.txt") is False + + +def test_a_duplicate_folder_on_the_way_is_refused( + storage: GoogleDriveStorage, drive: FakeDrive +) -> None: + for _ in range(3): + drive.add_file("inner.txt", b"x", drive.add_folder("twin")) + with pytest.raises(StorageException, match="3 entries share the name 'twin'") as caught: + storage.exists("twin/inner.txt") + assert "gdrive:///twin:" in str(caught.value) + with pytest.raises(StorageException, match="3 entries"): + storage.write_bytes("twin/new.txt", b"x") + with pytest.raises(StorageException, match="3 entries"): + storage.mkdir("twin/sub") + assert len(drive.named("new.txt")) == 0 + # A recursive listing reads the folders by ID, so it still shows what is there. + assert [info.path for info in storage.list_dir("", recursive=True)] == [ + "twin", + "twin", + "twin", + "twin/inner.txt", + "twin/inner.txt", + "twin/inner.txt", + ] + + +def test_names_that_differ_only_in_case_are_different_entries( + storage: GoogleDriveStorage, drive: FakeDrive +) -> None: + # The stand-in's "name =" ignores case, as Drive's is reported to. + storage.write_bytes("Report.txt", b"upper") + assert storage.exists("report.txt") is False + storage.write_bytes("report.txt", b"lower") + assert storage.read_bytes("Report.txt") == b"upper" + assert storage.read_bytes("report.txt") == b"lower" + assert sorted(info.name for info in storage.list_dir()) == ["Report.txt", "report.txt"] + storage.delete("Report.txt") + assert [entry.data for entry in drive.named("report.txt")] == [b"lower"] + + +@pytest.mark.parametrize( + "name", ["it's.txt", "back\\slash.txt", "both \\' at once", "'", "\\", "q='x' or name='y'"] +) +def test_quotes_and_backslashes_in_names_are_escaped( + storage: GoogleDriveStorage, drive: FakeDrive, name: str +) -> None: + drive.add_file("decoy", b"decoy") + storage.write_bytes(f"dir/{name}", b"payload") + assert storage.read_bytes(f"dir/{name}") == b"payload" + assert drive.only(name).data == b"payload" + storage.delete(f"dir/{name}") + assert drive.named(name) == [] + assert drive.only("decoy").data == b"decoy" + + +def test_an_entry_no_path_can_address_is_left_out_but_protects_its_folder( + storage: GoogleDriveStorage, drive: FakeDrive +) -> None: + folder = drive.add_folder("dir") + drive.add_file("2026/10 notes.txt", b"x", folder) + drive.add_file("..", b"x", folder) + assert storage.list_dir("dir") == [] + assert [info.path for info in storage.list_dir("", recursive=True)] == ["dir"] + with pytest.raises(StorageNotEmptyException): + storage.delete("dir") + assert len(drive.entries) == 4 + storage.delete("dir", recursive=True) + assert list(drive.entries) == [MY_DRIVE_ID] + + +# ---------------------------------------------------------------------- Google Workspace documents + + +def test_a_google_doc_is_listed_without_a_size_and_cannot_be_downloaded( + storage: GoogleDriveStorage, drive: FakeDrive, tmp_path: Path +) -> None: + drive.add_document("Minutes", drive.add_folder("docs")) + info = storage.stat("docs/Minutes") + assert (info.is_dir, info.size, info.etag) == (False, None, None) + assert info.content_type == DOCUMENT_MIME_TYPE + assert storage.list_dir("docs") == [info] + target = tmp_path / "minutes.bin" + for attempt in ( + lambda: storage.download("docs/Minutes", target), + lambda: storage.read_bytes("docs/Minutes"), + lambda: storage.checksum("docs/Minutes"), + lambda: storage.checksum("docs/Minutes", "md5"), + ): + with pytest.raises(StorageUnsupportedException, match="Google Workspace document"): + attempt() + assert not target.exists() + assert list(tmp_path.iterdir()) == [] + assert "media" not in drive.calls + + +def test_a_google_doc_cannot_be_replaced_by_a_file( + storage: GoogleDriveStorage, drive: FakeDrive +) -> None: + document = drive.add_document("Minutes") + storage.write_bytes("a.txt", b"x") + with pytest.raises(StorageUnsupportedException, match="cannot replace"): + storage.write_bytes("Minutes", b"x") + with pytest.raises(StorageUnsupportedException): + storage.copy_from(storage, "a.txt", "Minutes") + assert drive.entries[document].mime_type == DOCUMENT_MIME_TYPE + assert len(drive.named("Minutes")) == 1 + + +def test_a_google_doc_can_be_copied_moved_and_deleted( + storage: GoogleDriveStorage, drive: FakeDrive +) -> None: + document = drive.add_document("Minutes") + copied = storage.copy_from(storage, "Minutes", "archive/Minutes 2026") + assert (copied.size, copied.content_type) == (None, DOCUMENT_MIME_TYPE) + storage.move_from(storage, "Minutes", "archive/Minutes") + assert drive.entries[document].parent == drive.only("archive").file_id + storage.delete("archive/Minutes") + assert [info.name for info in storage.list_dir("archive")] == ["Minutes 2026"] + + +# ---------------------------------------------------------------------- upload and download + + +def test_overwrite_keeps_the_file_id(storage: GoogleDriveStorage, drive: FakeDrive) -> None: + storage.write_bytes("dir/a.txt", b"first") + original = drive.only("a.txt").file_id + drive.seen.clear() + info = storage.write_bytes("dir/a.txt", b"second, longer") + entry = drive.only("a.txt") + assert (entry.file_id, entry.data, entry.version) == (original, b"second, longer", 2) + assert (info.size, info.version) == (14, "2") + assert "update" in drive.calls + assert "create" not in drive.calls + assert "delete" not in drive.calls + + +def test_an_upload_is_resumable_and_typed_by_the_target_name( + storage: GoogleDriveStorage, drive: FakeDrive, tmp_path: Path +) -> None: + source = tmp_path / "staged-without-a-suffix" + source.write_bytes(b"a,b\n") + storage.mkdir("reports") + storage.upload(source, "reports/q1.csv") + opened = drive.last("create") + assert opened.params["uploadType"] == "resumable" + assert opened.headers["x-upload-content-type"] == "text/csv" + assert opened.headers["x-upload-content-length"] == "4" + assert opened.body == { + "name": "q1.csv", + "parents": [drive.only("reports").file_id], + "mimeType": "text/csv", + } + assert drive.last("upload").headers["content-range"] == "bytes 0-3/4" + + +def test_a_failed_upload_leaves_the_local_file_closed( + storage: GoogleDriveStorage, drive: FakeDrive, tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + opened: list[MediaFileUpload] = [] + construct = MediaFileUpload.__init__ + + def recording(media: MediaFileUpload, *arguments: Any, **options: Any) -> None: + construct(media, *arguments, **options) + opened.append(media) + + monkeypatch.setattr(MediaFileUpload, "__init__", recording) + source = tmp_path / "source.bin" + source.write_bytes(b"payload") + drive.fail_with, drive.fail_methods = (503, "backendError"), frozenset({"PUT"}) + with pytest.raises(StorageTransientException): + storage.upload(source, "a.bin") + assert [media.stream().closed for media in opened] == [True] + # On Windows an open handle would make this fail. + source.unlink() + assert storage.exists("a.bin") is False + + +def test_a_download_arrives_in_chunks( + storage: GoogleDriveStorage, drive: FakeDrive, tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + data = bytes(range(256)) * 20 + storage.write_bytes("data.bin", data) + monkeypatch.setattr(gdrive_storage, "_DOWNLOAD_CHUNK_SIZE", 1024) + drive.seen.clear() + assert storage.download("data.bin", tmp_path / "data.bin").read_bytes() == data + ranges = [seen.headers["range"] for seen in drive.seen if seen.call == "media"] + assert ranges == [f"bytes={start}-{start + 1023}" for start in range(0, len(data), 1024)] + + +def test_a_failed_download_leaves_no_file_behind( + storage: GoogleDriveStorage, drive: FakeDrive, tmp_path: Path +) -> None: + storage.write_bytes("a.txt", b"remote") + target = tmp_path / "out" / "a.txt" + target.parent.mkdir() + target.write_bytes(b"local") + drive.fail_with, drive.fail_methods = (500, "backendError"), frozenset({"GET"}) + with pytest.raises(StorageTransientException): + storage.download("a.txt", target) + assert target.read_bytes() == b"local" + assert [entry.name for entry in target.parent.iterdir()] == ["a.txt"] + + +# ---------------------------------------------------------------------- checksum + + +@pytest.mark.parametrize("algorithm", ["md5", "sha1", "sha256"]) +def test_checksum_uses_the_digest_drive_holds( + storage: GoogleDriveStorage, drive: FakeDrive, algorithm: str +) -> None: + storage.write_bytes("data.bin", b"payload") + drive.seen.clear() + assert ( + storage.checksum("data.bin", algorithm).value + == hashlib.new(algorithm, b"payload").hexdigest() + ) + assert "media" not in drive.calls + + +@pytest.mark.parametrize("algorithm", ["md5", "sha256", "sha512"]) +def test_checksum_falls_back_to_hashing_the_content( + storage: GoogleDriveStorage, drive: FakeDrive, algorithm: str +) -> None: + storage.write_bytes("data.bin", b"payload") + drive.digests = False + drive.seen.clear() + assert storage.stat("data.bin").etag is None + assert ( + storage.checksum("data.bin", algorithm).value + == hashlib.new(algorithm, b"payload").hexdigest() + ) + assert "media" in drive.calls + + +# ---------------------------------------------------------------------- copy and move + + +def test_copy_within_drive_is_done_by_drive(storage: GoogleDriveStorage, drive: FakeDrive) -> None: + storage.write_bytes("a.txt", b"payload") + original = drive.only("a.txt").file_id + drive.seen.clear() + storage.copy_from(storage, "a.txt", "copies/b.txt") + assert "copy" in drive.calls + assert not {"media", "upload"} & set(drive.calls) + copied = drive.only("b.txt") + assert (copied.data, copied.parent) == (b"payload", drive.only("copies").file_id) + assert copied.file_id != original + assert drive.only("a.txt").file_id == original + assert drive.last("copy").body == {"name": "b.txt", "parents": [copied.parent]} + + +def test_move_within_drive_keeps_the_file_id(storage: GoogleDriveStorage, drive: FakeDrive) -> None: + storage.write_bytes("inbox/a.txt", b"payload") + original = drive.only("a.txt").file_id + inbox = drive.only("inbox").file_id + drive.seen.clear() + storage.move_from(storage, "inbox/a.txt", "archive/2026/b.txt") + assert not {"media", "upload", "copy", "delete"} & set(drive.calls) + moved = drive.only("b.txt") + assert (moved.file_id, moved.data) == (original, b"payload") + assert moved.parent == drive.only("2026").file_id + assert drive.named("a.txt") == [] + patch = drive.last("update") + assert (patch.params["addParents"], patch.params["removeParents"]) == (moved.parent, inbox) + assert patch.body == {"name": "b.txt"} + + +def test_a_rename_in_place_changes_no_parent(storage: GoogleDriveStorage, drive: FakeDrive) -> None: + storage.write_bytes("a.txt", b"payload") + storage.write_bytes("dir/c.txt", b"other") + for source, target in (("a.txt", "b.txt"), ("dir/c.txt", "dir/d.txt")): + drive.seen.clear() + storage.move_from(storage, source, target) + assert not {"addParents", "removeParents"} & set(drive.last("update").params) + assert (drive.only("b.txt").parent, drive.only("b.txt").data) == (MY_DRIVE_ID, b"payload") + assert drive.only("d.txt").parent == drive.only("dir").file_id + + +def test_a_move_to_my_drive_names_the_folder_by_its_id( + storage: GoogleDriveStorage, drive: FakeDrive +) -> None: + storage.write_bytes("dir/a.txt", b"payload") + storage.move_from(storage, "dir/a.txt", "a.txt") + assert drive.only("a.txt").parent == MY_DRIVE_ID + assert drive.last("update").params["addParents"] == MY_DRIVE_ID + + +def test_copy_and_move_onto_an_existing_file_keep_its_id( + storage: GoogleDriveStorage, drive: FakeDrive +) -> None: + storage.write_bytes("a.txt", b"from a") + storage.write_bytes("c.txt", b"from c") + storage.write_bytes("b.txt", b"old") + target = drive.only("b.txt").file_id + storage.copy_from(storage, "a.txt", "b.txt") + assert (drive.only("b.txt").file_id, drive.only("b.txt").data) == (target, b"from a") + storage.move_from(storage, "c.txt", "b.txt") + assert (drive.only("b.txt").file_id, drive.only("b.txt").data) == (target, b"from c") + assert drive.named("c.txt") == [] + assert "copy" not in drive.calls + + +def test_two_roots_that_show_one_file_never_transfer_it_onto_itself( + client: GoogleDriveClient, drive: FakeDrive +) -> None: + whole = GoogleDriveStorage(client) + whole.write_bytes("team/a.txt", b"payload") + team = GoogleDriveStorage(client, root_id=drive.only("team").file_id) + other_client = GoogleDriveStorage(_client(drive), root_id=drive.only("team").file_id) + for target in (team, other_client): + with pytest.raises(StorageException, match="same file"): + target.move_from(whole, "team/a.txt", "a.txt") + with pytest.raises(StorageException, match="same file"): + target.copy_from(whole, "team/a.txt", "a.txt") + assert drive.only("a.txt").data == b"payload" + + +def test_copy_between_two_clients_goes_through_a_staging_file( + storage: GoogleDriveStorage, drive: FakeDrive +) -> None: + other_drive = FakeDrive() + other = GoogleDriveStorage(_client(other_drive)) + storage.write_bytes("a.txt", b"payload") + other.copy_from(storage, "a.txt", "a.txt") + assert other_drive.only("a.txt").data == b"payload" + assert "copy" not in other_drive.calls + other.move_from(storage, "a.txt", "moved.txt") + assert other_drive.only("moved.txt").data == b"payload" + assert drive.named("a.txt") == [] + + +def test_a_folder_goes_in_one_call_with_everything_in_it( + storage: GoogleDriveStorage, drive: FakeDrive +) -> None: + for path in ("dir/a.txt", "dir/sub/b.txt", "keep.txt"): + storage.write_bytes(path, b"x") + drive.seen.clear() + storage.delete("dir", recursive=True) + assert drive.calls.count("delete") == 1 + assert sorted(entry.name for entry in drive.entries.values()) == ["My Drive", "keep.txt"] + + +# ---------------------------------------------------------------------- errors + + +@pytest.mark.parametrize( + "status,reason,expected", + [ + (403, "rateLimitExceeded", StorageTransientException), + (403, "userRateLimitExceeded", StorageTransientException), + (429, "rateLimitExceeded", StorageTransientException), + (500, "internalError", StorageTransientException), + (503, "backendError", StorageTransientException), + (401, "authError", StoragePermissionException), + (403, "insufficientFilePermissions", StoragePermissionException), + (403, "dailyLimitExceeded", StoragePermissionException), + (400, "invalid", StorageException), + (409, "conflict", StorageException), + ], +) +def test_http_errors_become_storage_errors( + storage: GoogleDriveStorage, + drive: FakeDrive, + status: int, + reason: str, + expected: type[Exception], +) -> None: + drive.fail_with = (status, reason) + with pytest.raises(expected) as caught: + storage.stat("dir/a.txt") + assert type(caught.value) is expected + cause = caught.value.__cause__ + assert isinstance(cause, HttpError) + assert cause.status_code == status + message = str(caught.value) + assert message.startswith("gdrive:///dir") or "access to gdrive:///dir" in message + assert f"{status} {reason}" in message + # The request URL, which can carry an API key, stays out of the message. + assert "googleapis" not in message + + +def test_a_404_is_not_found_with_the_sdk_error_as_its_cause( + storage: GoogleDriveStorage, drive: FakeDrive +) -> None: + storage.write_bytes("a.txt", b"x") + drive.fail_with, drive.fail_methods = (404, "notFound"), frozenset({"DELETE"}) + with pytest.raises(StorageNotFoundException) as caught: + storage.delete("a.txt") + assert isinstance(caught.value.__cause__, HttpError) + drive.fail_methods = None + assert storage.exists("a.txt") is False + + +@pytest.mark.parametrize( + "error,expected", + [ + (ConnectionResetError("reset by peer"), StorageTransientException), + (TimeoutError("timed out"), StorageTransientException), + (httplib2.ServerNotFoundError("no such host"), StorageTransientException), + (ssl.SSLEOFError("EOF in violation of protocol"), StorageTransientException), + (auth_errors.TransportError("token endpoint unreachable"), StorageTransientException), + (auth_errors.RefreshError("invalid_grant"), StoragePermissionException), + (ssl.SSLCertVerificationError("self-signed certificate"), StorageException), + (httplib2.HttpLib2Error("redirected too often"), StorageException), + (auth_errors.DefaultCredentialsError("no credentials"), StorageException), + ], +) +def test_transport_and_credential_errors_become_storage_errors( + storage: GoogleDriveStorage, drive: FakeDrive, error: Exception, expected: type[Exception] +) -> None: + drive.fail_with = error + with pytest.raises(expected) as caught: + storage.stat("a.txt") + assert type(caught.value) is expected + assert caught.value.__cause__ is error + assert str(error) not in str(caught.value) + + +def test_the_shared_client_must_be_initialised(monkeypatch: pytest.MonkeyPatch) -> None: + monkeypatch.setattr(driver_instance, "service", None) + with pytest.raises(StorageUnavailableException, match="later_init") as caught: + GoogleDriveStorage().exists("a.txt") + assert isinstance(caught.value.__cause__, RuntimeError) + + +# ---------------------------------------------------------------------- identity and URIs + + +def test_uri_for(client: GoogleDriveClient) -> None: + assert GoogleDriveStorage(client).uri_for("") == "gdrive:///" + assert GoogleDriveStorage(client).uri_for("/reports//q1.csv") == "gdrive:///reports/q1.csv" + rooted = GoogleDriveStorage(client, root_id="1AbC-dEf_9") + assert rooted.uri_for("") == "gdrive://1AbC-dEf_9" + assert rooted.uri_for("q1.csv") == "gdrive://1AbC-dEf_9/q1.csv" + + +def test_equality_and_repr(client: GoogleDriveClient, drive: FakeDrive) -> None: + assert GoogleDriveStorage(client) == GoogleDriveStorage(client, root_id="root") + assert GoogleDriveStorage(client) == GoogleDriveStorage(client, root_id=" ") + assert GoogleDriveStorage(client) != GoogleDriveStorage(client, root_id="folder") + assert GoogleDriveStorage(client) != GoogleDriveStorage(_client(drive)) + assert GoogleDriveStorage(client) != GoogleDriveStorage() + assert len({GoogleDriveStorage(client), GoogleDriveStorage(client)}) == 1 + assert ( + repr(GoogleDriveStorage(client, root_id="folder")) == "GoogleDriveStorage(root_id='folder')" + ) + assert GoogleDriveStorage(client).root_id == "root" + assert GoogleDriveStorage.scheme == GDRIVE_SCHEME == "gdrive" + assert GoogleDriveStorage.capabilities.to_dict() == { + "directories": True, + "modified_at": True, + "etag": True, + "version": True, + "content_type": True, + "metadata": False, + } + + +@pytest.mark.parametrize("root_id", ["two words", "a/b", "user@host"]) +def test_a_root_id_that_cannot_be_a_uri_authority_is_refused(root_id: str) -> None: + with pytest.raises(StorageURIException): + GoogleDriveStorage(root_id=root_id) + + +def test_gdrive_uris_use_the_shared_client( + monkeypatch: pytest.MonkeyPatch, client: GoogleDriveClient, drive: FakeDrive, tmp_path: Path +) -> None: + monkeypatch.setattr(driver_instance, "service", client.service) + resolver = StorageResolver() + resolver.register_scheme(GDRIVE_SCHEME, gdrive_factory) + report = File("gdrive:///reports/q1.csv", resolver=resolver) + report.write(b"a,b\n") + assert drive.only("q1.csv").data == b"a,b\n" + assert resolver.resolve("gdrive:///reports/q1.csv") == (GoogleDriveStorage(), "reports/q1.csv") + assert resolver.resolve("gdrive://root/reports/q1.csv") == ( + GoogleDriveStorage(), + "reports/q1.csv", + ) + folder = drive.only("reports").file_id + assert resolver.resolve(f"gdrive://{folder}/q1.csv") == ( + GoogleDriveStorage(root_id=folder), + "q1.csv", + ) + assert File(f"gdrive://{folder}/q1.csv", resolver=resolver).read() == b"a,b\n" + report.copy_to(tmp_path / "q1.csv") + assert (tmp_path / "q1.csv").read_bytes() == b"a,b\n" + File(tmp_path / "q1.csv", resolver=resolver).copy_to(f"gdrive://{folder}/2026/q1.csv") + assert [info.path for info in GoogleDriveStorage().list_dir("reports", recursive=True)] == [ + "reports/2026", + "reports/2026/q1.csv", + "reports/q1.csv", + ] + with pytest.raises(StorageException, match="same file"): + report.copy_to(f"gdrive://{folder}/q1.csv") + assert "gdrive" in resolver.schemes() + + +# ---------------------------------------------------------------------- the installed googleapiclient + + +def _parameters(method: str) -> set[str]: + return set(FILE_METHODS[method]["parameters"]) | set(DISCOVERY["parameters"]) + + +def test_the_discovery_document_has_the_calls_the_adapter_makes() -> None: + """The stand-in serves a real service; this names what that service is held to.""" + assert DISCOVERY["name"] == "drive" + assert DISCOVERY["version"] == "v3" + everywhere = {"supportsAllDrives", "fields"} + assert everywhere | {"q", "pageSize", "pageToken", "includeItemsFromAllDrives"} <= _parameters( + "list" + ) + assert everywhere | {"fileId"} <= _parameters("get") + assert everywhere <= _parameters("create") + assert everywhere | {"fileId", "addParents", "removeParents"} <= _parameters("update") + assert everywhere | {"fileId"} <= _parameters("copy") + assert {"supportsAllDrives", "fileId"} <= _parameters("delete") + assert FILE_METHODS["get"]["supportsMediaDownload"] is True + for method in ("create", "update"): + assert "resumable" in FILE_METHODS[method]["mediaUpload"]["protocols"] + assert {"files", "nextPageToken"} <= set(DISCOVERY["schemas"]["FileList"]["properties"]) + assert ( + int(FILE_METHODS["list"]["parameters"]["pageSize"]["maximum"]) >= gdrive_storage._PAGE_SIZE + ) + + +def test_the_fields_the_adapter_reads_are_in_the_file_schema() -> None: + asked = {name.strip() for name in gdrive_storage._ENTRY_FIELDS.split(",")} + asked |= {name.strip() for name in gdrive_storage._ROOT_FIELDS.split(",")} + asked |= set(gdrive_storage._SERVER_DIGESTS.values()) + assert asked <= set(FILE_SCHEMA) + assert asked >= {"id", "name", "mimeType", "size", "modifiedTime", "md5Checksum", "version"} + # 64-bit integers travel as strings, and the time as RFC 3339 text. + assert (FILE_SCHEMA["size"]["type"], FILE_SCHEMA["size"]["format"]) == ("string", "int64") + assert (FILE_SCHEMA["version"]["type"], FILE_SCHEMA["version"]["format"]) == ("string", "int64") + assert FILE_SCHEMA["modifiedTime"]["format"] == "date-time" + assert FILE_SCHEMA["parents"]["type"] == "array" + + +def test_the_media_classes_take_the_arguments_the_adapter_passes() -> None: + assert list(inspect.signature(MediaFileUpload.__init__).parameters) == [ + "self", + "filename", + "mimetype", + "chunksize", + "resumable", + ] + assert list(inspect.signature(MediaIoBaseDownload.__init__).parameters) == [ + "self", + "fd", + "request", + "chunksize", + ] + assert callable(MediaFileUpload.stream) + assert callable(MediaIoBaseDownload.next_chunk) + assert isinstance(HttpError.status_code, property) + + +@pytest.mark.parametrize( + "reason", + [ + "rateLimitExceeded", + "userRateLimitExceeded", + "dailyLimitExceeded", + "sharingRateLimitExceeded", + "insufficientFilePermissions", + "fileNotDownloadable", + ], +) +def test_a_403_is_transient_exactly_when_googleapiclient_would_retry_it(reason: str) -> None: + retried = getattr(googleapiclient_http, "_should_retry_response", None) + if retried is None: + pytest.skip("this googleapiclient no longer has the private retry rule to compare with") + _, body = error_answer(403, reason) + assert retried(403, body) is (reason in gdrive_storage._RATE_LIMIT_REASONS) + assert retried(429, b"") is True + assert retried(500, b"") is True + assert retried(404, b"") is False diff --git a/tests/test_storage_local.py b/tests/test_storage_local.py index 2d79807..3c07c5d 100644 --- a/tests/test_storage_local.py +++ b/tests/test_storage_local.py @@ -230,3 +230,24 @@ def test_a_root_that_does_not_exist_reports_not_found(tmp_path: Path) -> None: assert storage.exists("") is False with pytest.raises(StorageNotFoundException): storage.list_dir() + + +def test_a_rooted_and_the_rootless_view_of_one_file_are_the_same_file(tmp_path: Path) -> None: + rooted = LocalStorage(tmp_path) + rootless = LocalStorage() + rooted.write_bytes("dir/a.txt", b"payload") + absolute = _rootless(tmp_path / "dir" / "a.txt") + for operation in (rootless.move_from, rootless.copy_from): + with pytest.raises(StorageException, match="same file"): + operation(rooted, "dir/a.txt", absolute) + with pytest.raises(StorageException, match="same file"): + rooted.move_from(rootless, absolute, "dir/a.txt") + assert (tmp_path / "dir" / "a.txt").read_bytes() == b"payload" + + +def test_a_link_to_a_file_is_the_same_file(storage: LocalStorage, root: Path) -> None: + (root / "real.txt").write_bytes(b"payload") + _symlink(root / "link.txt", root / "real.txt") + with pytest.raises(StorageException, match="same file"): + storage.move_from(storage, "link.txt", "real.txt") + assert (root / "real.txt").read_bytes() == b"payload" diff --git a/tests/test_storage_onedrive.py b/tests/test_storage_onedrive.py new file mode 100644 index 0000000..ff2eb9f --- /dev/null +++ b/tests/test_storage_onedrive.py @@ -0,0 +1,715 @@ +"""OneDriveStorage: the storage contract against an in-memory OneDrive behind a real session. + +The stand-in (``tests/graph_stand_in.py``) is a transport adapter mounted on the +``requests.Session`` of a real ``OneDriveClient``, so every request is prepared by +``requests`` itself. What Microsoft Graph answers is the stand-in's reading of the +``driveItem`` documentation: nothing installed describes that API. +""" + +# pylint: disable=protected-access # the shared client's session is swapped for the stand-in's + +from __future__ import annotations + +import inspect +import json +import logging +import traceback +from collections.abc import Iterator +from datetime import timedelta +from pathlib import Path +from typing import Any + +import pytest +import requests + +from automation_file.exceptions import ( + OneDriveException, + StorageException, + StorageNotFoundException, + StoragePathTypeException, + StoragePermissionException, + StorageTransientException, + StorageUnavailableException, + StorageURIException, +) +from automation_file.logging_config import file_automation_logger +from automation_file.remote.onedrive.client import OneDriveClient, onedrive_instance +from automation_file.storage import File, StorageBackend, StorageResolver, onedrive_storage +from automation_file.storage.onedrive_storage import ( + ONEDRIVE_SCHEME, + OneDriveStorage, + onedrive_factory, +) +from tests.graph_stand_in import ( + DOWNLOAD_ORIGIN, + DRIVE_ROOT, + ERROR_CODES, + FAKE_TOKEN, + FRAGMENT_LIMIT, + FRAGMENT_UNIT, + GRAPH_ORIGIN, + KIB, + SIMPLE_UPLOAD_LIMIT, + UPLOAD_ORIGIN, + URL_SECRET, + FakeGraph, + graph_answer, +) +from tests.storage_contract import StorageContract + + +def _client(graph: FakeGraph) -> OneDriveClient: + """A OneDriveClient whose session is a real one, with the stand-in as its transport.""" + client = OneDriveClient() + client.later_init(FAKE_TOKEN) + client.require_session().mount("https://", graph) + return client + + +def _printed(error: BaseException) -> str: + """Everything a logger would print for ``error``: its text, its cause, its traceback.""" + return "".join(traceback.format_exception(type(error), error, error.__traceback__)) + + +class TestOneDriveStorageContract(StorageContract): + @pytest.fixture + def backend(self) -> StorageBackend: + return OneDriveStorage(_client(FakeGraph())) + + +class TestRootedOneDriveStorageContract(StorageContract): + @pytest.fixture + def backend(self) -> StorageBackend: + graph = FakeGraph() + graph.add_file("keep.txt", b"keep") + graph.add_file("tenants/b/keep.txt", b"keep") + graph.add_folder("tenants/a") + return OneDriveStorage(_client(graph), root="tenants/a") + + +@pytest.fixture +def graph() -> FakeGraph: + return FakeGraph() + + +@pytest.fixture +def client(graph: FakeGraph) -> OneDriveClient: + return _client(graph) + + +@pytest.fixture +def storage(client: OneDriveClient) -> OneDriveStorage: + return OneDriveStorage(client) + + +@pytest.fixture +def logged() -> Iterator[list[str]]: + """The messages the library logs while the test runs.""" + messages: list[str] = [] + + class _Collect(logging.Handler): + def emit(self, record: logging.LogRecord) -> None: + messages.append(record.getMessage()) + + handler = _Collect(level=logging.DEBUG) + file_automation_logger.addHandler(handler) + yield messages + file_automation_logger.removeHandler(handler) + + +# ---------------------------------------------------------------------- stat and listing + + +def test_stat_reports_the_drive_item(storage: OneDriveStorage, graph: FakeGraph) -> None: + info = storage.write_bytes("reports/q1.json", b"{}") + item = graph.at("reports/q1.json") + assert info.path == "reports/q1.json" + assert info.size == 2 + assert info.etag == f"{{{item.item_id}}},1" + assert info.content_type == "application/json" + assert info.version is None + assert dict(info.metadata) == {} + assert info.modified_at is not None + assert info.modified_at.utcoffset() == timedelta(0) + assert info.modified_at == item.modified + + +def test_a_folder_is_a_directory_with_a_modification_time(storage: OneDriveStorage) -> None: + storage.mkdir("reports") + info = storage.stat("reports") + assert (info.is_dir, info.size, info.etag, info.content_type) == (True, None, None, None) + assert info.modified_at is not None + assert storage.stat("").is_dir is True + + +def test_listing_follows_the_next_link(storage: OneDriveStorage, graph: FakeGraph) -> None: + for index in range(7): + storage.write_bytes(f"dir/{index}.txt", b"x") + graph.seen.clear() + listing = storage.list_dir("dir") + assert [info.path for info in listing] == [f"dir/{index}.txt" for index in range(7)] + assert all(info.etag and info.size == 1 for info in listing) + pages = graph.all("children") + assert len(pages) == 4 + assert "skiptoken" not in pages[0].url + assert all("skiptoken" in page.url and "select" in page.url for page in pages[1:]) + assert len(storage.list_dir("", recursive=True)) == 8 + + +@pytest.mark.parametrize( + "name", + ["100% #1 'draft' +v2.txt", "a%20b.txt", "q=1&r=2;s.txt", "~[x]{y}^`!$@,.txt", "Ünïcödé ß.txt"], +) +def test_reserved_characters_in_a_name_reach_onedrive_intact( + storage: OneDriveStorage, graph: FakeGraph, name: str +) -> None: + storage.write_bytes(f"odd names/{name}", b"payload") + assert graph.paths() == ["odd names", f"odd names/{name}"] + assert storage.read_bytes(f"odd names/{name}") == b"payload" + assert [info.name for info in storage.list_dir("odd names")] == [name] + storage.move_from(storage, f"odd names/{name}", f"moved/{name}") + storage.delete(f"moved/{name}") + assert graph.paths() == ["moved", "odd names"] + + +def test_a_name_onedrive_does_not_allow_is_a_storage_error(storage: OneDriveStorage) -> None: + with pytest.raises(StorageException, match="400 invalidRequest") as caught: + storage.write_bytes("what?.txt", b"x") + assert type(caught.value) is StorageException + + +def test_names_compare_without_regard_to_case(storage: OneDriveStorage, graph: FakeGraph) -> None: + storage.write_bytes("Docs/Report.txt", b"first") + original = graph.at("Docs/Report.txt").item_id + assert storage.exists("docs/report.txt") is True + assert storage.stat("DOCS/REPORT.TXT").path == "DOCS/REPORT.TXT" + storage.write_bytes("docs/report.txt", b"second") + assert graph.paths() == ["Docs", "Docs/Report.txt"] + assert (graph.at("Docs/Report.txt").item_id, graph.at("Docs/Report.txt").data) == ( + original, + b"second", + ) + assert [info.path for info in storage.list_dir("docs")] == ["docs/Report.txt"] + + +# ---------------------------------------------------------------------- upload and download + + +def test_a_small_file_goes_up_in_one_request(storage: OneDriveStorage, graph: FakeGraph) -> None: + storage.mkdir("reports") + graph.seen.clear() + storage.write_bytes("reports/q1.csv", b"a,b\n") + (put,) = graph.all("put") + assert put.url == f"{GRAPH_ORIGIN}{DRIVE_ROOT}:/reports/q1.csv:/content" + assert (put.headers["Content-Type"], put.headers["Content-Length"]) == ("text/csv", "4") + assert put.body == b"a,b\n" + assert not {"session", "fragment"} & set(graph.calls) + + +def test_overwrite_keeps_the_item(storage: OneDriveStorage, graph: FakeGraph) -> None: + storage.write_bytes("dir/a.txt", b"first") + original = graph.at("dir/a.txt").item_id + graph.seen.clear() + info = storage.write_bytes("dir/a.txt", b"second, longer") + item = graph.at("dir/a.txt") + assert (item.item_id, item.data, item.revision) == (original, b"second, longer", 2) + assert (info.size, info.etag) == (14, f"{{{original}}},2") + assert "delete" not in graph.calls + + +def test_the_simple_upload_limit_decides_between_one_request_and_a_session( + storage: OneDriveStorage, graph: FakeGraph, monkeypatch: pytest.MonkeyPatch +) -> None: + monkeypatch.setattr(onedrive_storage, "_SIMPLE_UPLOAD_MAX", 100) + storage.write_bytes("at-the-limit.bin", b"x" * 100) + assert graph.calls.count("put") == 1 + assert "session" not in graph.calls + storage.write_bytes("over-the-limit.bin", b"x" * 101) + assert graph.calls.count("put") == 1 + assert (graph.calls.count("session"), graph.calls.count("fragment")) == (1, 1) + assert graph.at("over-the-limit.bin").data == b"x" * 101 + + +def test_a_large_file_goes_up_in_fragments_read_from_the_file( + storage: OneDriveStorage, graph: FakeGraph, tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + monkeypatch.setattr(onedrive_storage, "_SIMPLE_UPLOAD_MAX", KIB) + monkeypatch.setattr(onedrive_storage, "_UPLOAD_CHUNK_SIZE", FRAGMENT_UNIT) + data = bytes(range(251)) * 4200 + data = data[: KIB * KIB + 17] + source = tmp_path / "large.bin" + source.write_bytes(data) + storage.write_bytes("large.bin", b"the file it replaces") + original = graph.at("large.bin").item_id + graph.seen.clear() + info = storage.upload(source, "large.bin") + assert info.size == len(data) + assert (graph.at("large.bin").item_id, graph.at("large.bin").data) == (original, data) + (opened,) = graph.all("session") + assert opened.url == f"{GRAPH_ORIGIN}{DRIVE_ROOT}:/large.bin:/createUploadSession" + fragments = graph.all("fragment") + assert [fragment.headers["Content-Range"] for fragment in fragments] == [ + f"bytes 0-327679/{len(data)}", + f"bytes 327680-655359/{len(data)}", + f"bytes 655360-983039/{len(data)}", + f"bytes 983040-{len(data) - 1}/{len(data)}", + ] + assert max(len(fragment.body) for fragment in fragments) == FRAGMENT_UNIT + assert all(fragment.url.startswith(UPLOAD_ORIGIN) for fragment in fragments) + assert not {"put", "cancel"} & set(graph.calls) + assert graph.sessions == {} + + +def test_the_default_fragment_size_is_a_multiple_of_320_kib() -> None: + assert onedrive_storage._UPLOAD_CHUNK_SIZE % FRAGMENT_UNIT == 0 + assert 0 < onedrive_storage._UPLOAD_CHUNK_SIZE <= FRAGMENT_LIMIT + assert onedrive_storage._SIMPLE_UPLOAD_MAX == SIMPLE_UPLOAD_LIMIT + + +@pytest.mark.parametrize( + "failure", + [503, requests.ConnectionError(f"Max retries exceeded with url: /session/1?{URL_SECRET}")], +) +def test_a_failed_fragment_cancels_the_session_and_keeps_its_url_secret( + storage: OneDriveStorage, + graph: FakeGraph, + monkeypatch: pytest.MonkeyPatch, + logged: list[str], + failure: int | Exception, +) -> None: + monkeypatch.setattr(onedrive_storage, "_SIMPLE_UPLOAD_MAX", 10) + graph.fail_with, graph.fail_calls = failure, frozenset({"fragment"}) + with pytest.raises(StorageTransientException) as caught: + storage.write_bytes("big.bin", b"x" * 100) + assert graph.calls[-2:] == ["fragment", "cancel"] + assert graph.sessions == {} + assert storage.exists("big.bin") is False + assert caught.value.__cause__ is None + assert "onedrive:///big.bin" in str(caught.value) + assert URL_SECRET not in _printed(caught.value) + assert not any(URL_SECRET in message or FAKE_TOKEN in message for message in logged) + + +def test_a_session_that_cannot_be_cancelled_is_logged_and_the_first_error_wins( + storage: OneDriveStorage, + graph: FakeGraph, + monkeypatch: pytest.MonkeyPatch, + logged: list[str], +) -> None: + monkeypatch.setattr(onedrive_storage, "_SIMPLE_UPLOAD_MAX", 10) + graph.fail_with, graph.fail_calls = 403, frozenset({"fragment", "cancel"}) + with pytest.raises(StoragePermissionException, match="403 accessDenied"): + storage.write_bytes("big.bin", b"x" * 100) + (warning,) = [message for message in logged if "could not be cancelled" in message] + assert "onedrive:///big.bin" in warning + assert URL_SECRET not in warning + + +def test_an_upload_session_without_an_https_url_is_refused( + storage: OneDriveStorage, graph: FakeGraph, monkeypatch: pytest.MonkeyPatch +) -> None: + monkeypatch.setattr(onedrive_storage, "_SIMPLE_UPLOAD_MAX", 10) + plain = {"uploadUrl": "http://upload.onedrive.invalid/session/1"} + monkeypatch.setattr(graph, "_on_session", lambda request, *_: graph_answer(request, 200, plain)) + with pytest.raises(StorageException, match="without a URL"): + storage.write_bytes("big.bin", b"x" * 100) + assert "fragment" not in graph.calls + + +def test_a_download_is_streamed_through_the_redirect( + storage: OneDriveStorage, graph: FakeGraph, tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + data = bytes(range(256)) * 20 + storage.write_bytes("data.bin", data) + read: list[tuple[int, int]] = [] + iter_content = requests.Response.iter_content + + def recording(response: requests.Response, chunk_size: int = 1, **options: Any) -> Any: + for chunk in iter_content(response, chunk_size, **options): + read.append((chunk_size, len(chunk))) + yield chunk + + monkeypatch.setattr(requests.Response, "iter_content", recording) + monkeypatch.setattr(onedrive_storage, "_DOWNLOAD_CHUNK_SIZE", KIB) + graph.seen.clear() + assert storage.download("data.bin", tmp_path / "data.bin").read_bytes() == data + assert graph.calls == ["item", "content", "download"] + assert all(seen.stream for seen in graph.seen[1:]) + assert graph.seen[2].url.startswith(DOWNLOAD_ORIGIN) + # requests reads an unstreamed body in one piece; the download came in 1 KiB pieces. + assert [size for asked, size in read if asked == KIB] == [KIB] * 5 + + +def test_a_failed_download_keeps_the_download_url_secret_and_leaves_no_file( + storage: OneDriveStorage, graph: FakeGraph, tmp_path: Path +) -> None: + storage.write_bytes("a.txt", b"remote") + target = tmp_path / "out" / "a.txt" + target.parent.mkdir() + target.write_bytes(b"local") + graph.fail_with = requests.ConnectionError(f"Max retries exceeded with url: /c/1?{URL_SECRET}") + graph.fail_calls = frozenset({"download"}) + with pytest.raises(StorageTransientException) as caught: + storage.download("a.txt", target) + assert caught.value.__cause__ is None + assert URL_SECRET not in _printed(caught.value) + assert target.read_bytes() == b"local" + assert [entry.name for entry in target.parent.iterdir()] == ["a.txt"] + + +# ---------------------------------------------------------------------- directories + + +def test_a_folder_goes_in_one_call_with_everything_in_it( + storage: OneDriveStorage, graph: FakeGraph +) -> None: + for path in ("dir/a.txt", "dir/sub/b.txt", "keep.txt"): + storage.write_bytes(path, b"x") + graph.seen.clear() + storage.delete("dir", recursive=True) + assert graph.calls == ["item", "delete"] + assert graph.paths() == ["keep.txt"] + + +def test_creating_a_folder_that_appeared_meanwhile_is_fine( + storage: OneDriveStorage, graph: FakeGraph +) -> None: + graph.add_file("dir/a.txt", b"x") + storage._mkdir("dir") + assert graph.paths() == ["dir", "dir/a.txt"] + with pytest.raises(StoragePathTypeException, match="is a file"): + storage._mkdir("dir/a.txt") + + +# ---------------------------------------------------------------------- copy and move + + +def test_move_within_onedrive_is_done_by_onedrive( + storage: OneDriveStorage, graph: FakeGraph +) -> None: + storage.write_bytes("inbox/a.txt", b"payload") + original = graph.at("inbox/a.txt").item_id + graph.seen.clear() + info = storage.move_from(storage, "inbox/a.txt", "archive/2026/b.txt") + assert info.path == "archive/2026/b.txt" + assert graph.paths() == ["archive", "archive/2026", "archive/2026/b.txt", "inbox"] + moved = graph.at("archive/2026/b.txt") + assert (moved.item_id, moved.data) == (original, b"payload") + (patch,) = graph.all("patch") + assert patch.url == f"{GRAPH_ORIGIN}{DRIVE_ROOT}:/inbox/a.txt" + assert json.loads(patch.body) == { + "parentReference": {"id": graph.at("archive/2026").item_id}, + "name": "b.txt", + } + assert not {"content", "put", "session", "delete"} & set(graph.calls) + + +def test_a_move_between_two_spellings_renames_the_item( + storage: OneDriveStorage, graph: FakeGraph +) -> None: + storage.write_bytes("dir/report.txt", b"payload") + original = graph.at("dir/report.txt").item_id + graph.seen.clear() + storage.move_from(storage, "dir/report.txt", "dir/Report.TXT") + assert graph.paths() == ["dir", "dir/Report.TXT"] + assert (graph.at("dir/Report.TXT").item_id, graph.at("dir/Report.TXT").data) == ( + original, + b"payload", + ) + assert not {"content", "put", "delete"} & set(graph.calls) + + +def test_a_copy_between_two_spellings_of_one_item_is_refused( + storage: OneDriveStorage, graph: FakeGraph +) -> None: + storage.write_bytes("report.txt", b"payload") + with pytest.raises(StorageException, match="same file"): + storage.copy_from(storage, "report.txt", "REPORT.txt") + assert graph.at("report.txt").data == b"payload" + + +def test_two_roots_that_show_one_file_never_transfer_it_onto_itself( + client: OneDriveClient, graph: FakeGraph +) -> None: + whole = OneDriveStorage(client) + whole.write_bytes("team/a.txt", b"payload") + team = OneDriveStorage(client, root="team") + other_client = OneDriveStorage(_client(graph), root="TEAM") + for target in (team, other_client): + with pytest.raises(StorageException, match="same file"): + target.move_from(whole, "team/a.txt", "a.txt") + with pytest.raises(StorageException, match="same file"): + target.copy_from(whole, "team/a.txt", "a.txt") + with pytest.raises(StorageException, match="same file"): + other_client.move_from(whole, "team/a.txt", "A.TXT") + assert graph.paths() == ["team", "team/a.txt"] + assert graph.at("team/a.txt").data == b"payload" + + +def test_copy_and_move_onto_an_existing_file_keep_its_item( + storage: OneDriveStorage, graph: FakeGraph +) -> None: + storage.write_bytes("a.txt", b"from a") + storage.write_bytes("c.txt", b"from c") + storage.write_bytes("b.txt", b"old") + target = graph.at("b.txt").item_id + storage.copy_from(storage, "a.txt", "b.txt") + assert (graph.at("b.txt").item_id, graph.at("b.txt").data) == (target, b"from a") + storage.move_from(storage, "c.txt", "b.txt") + assert (graph.at("b.txt").item_id, graph.at("b.txt").data) == (target, b"from c") + assert graph.paths() == ["a.txt", "b.txt"] + assert "patch" not in graph.calls + + +def test_copy_goes_through_a_staging_file(storage: OneDriveStorage, graph: FakeGraph) -> None: + storage.write_bytes("a.txt", b"payload") + graph.seen.clear() + storage.copy_from(storage, "a.txt", "copies/b.txt") + assert graph.at("copies/b.txt").data == b"payload" + assert graph.at("copies/b.txt").item_id != graph.at("a.txt").item_id + assert {"content", "download", "put"} <= set(graph.calls) + + +def test_a_move_between_two_clients_copies_then_deletes(storage: OneDriveStorage) -> None: + other_graph = FakeGraph() + other = OneDriveStorage(_client(other_graph)) + storage.write_bytes("a.txt", b"payload") + other.move_from(storage, "a.txt", "moved/a.txt") + assert other_graph.at("moved/a.txt").data == b"payload" + assert "patch" not in other_graph.calls + assert storage.exists("a.txt") is False + + +# ---------------------------------------------------------------------- errors + + +@pytest.mark.parametrize( + "status,expected", + [ + (401, StoragePermissionException), + (403, StoragePermissionException), + (408, StorageTransientException), + (429, StorageTransientException), + (500, StorageTransientException), + (503, StorageTransientException), + (504, StorageTransientException), + (400, StorageException), + (409, StorageException), + ], +) +def test_http_errors_become_storage_errors( + storage: OneDriveStorage, graph: FakeGraph, status: int, expected: type[Exception] +) -> None: + graph.fail_with = status + with pytest.raises(expected) as caught: + storage.stat("dir/a.txt") + assert type(caught.value) is expected + message = str(caught.value) + assert "onedrive:///dir/a.txt" in message + assert f"{status} {ERROR_CODES[status]}" in message + assert "graph.microsoft.com" not in message + assert FAKE_TOKEN not in _printed(caught.value) + + +def test_a_404_is_not_found(storage: OneDriveStorage, graph: FakeGraph) -> None: + storage.write_bytes("a.txt", b"x") + graph.fail_with, graph.fail_calls = 404, frozenset({"delete"}) + with pytest.raises(StorageNotFoundException, match=r"onedrive:///a\.txt does not exist"): + storage.delete("a.txt") + graph.fail_calls = None + assert storage.exists("a.txt") is False + with pytest.raises(StorageNotFoundException): + storage.list_dir() + + +@pytest.mark.parametrize( + "error,expected", + [ + (requests.ConnectionError("connection refused"), StorageTransientException), + (requests.ConnectTimeout("connect timed out"), StorageTransientException), + (requests.ReadTimeout("read timed out"), StorageTransientException), + (requests.exceptions.ChunkedEncodingError("connection broken"), StorageTransientException), + (requests.exceptions.ProxyError("proxy unreachable"), StorageTransientException), + (requests.exceptions.SSLError("certificate verify failed"), StorageException), + (requests.TooManyRedirects("30 redirects"), StorageException), + (requests.exceptions.InvalidURL("no host"), StorageException), + ], +) +def test_transport_errors_become_storage_errors_with_their_cause( + storage: OneDriveStorage, graph: FakeGraph, error: Exception, expected: type[Exception] +) -> None: + graph.fail_with = error + with pytest.raises(expected) as caught: + storage.stat("a.txt") + assert type(caught.value) is expected + assert caught.value.__cause__ is error + assert str(caught.value) == f"onedrive:///a.txt: {type(error).__name__}" + + +def test_an_answer_that_is_not_a_json_object_is_a_storage_error( + storage: OneDriveStorage, graph: FakeGraph, monkeypatch: pytest.MonkeyPatch +) -> None: + monkeypatch.setattr(graph, "_on_item", lambda request, *_: graph_answer(request, 200, [1, 2])) + with pytest.raises(StorageException, match="unexpected JSON"): + storage.stat("a.txt") + monkeypatch.setattr( + graph, "_on_item", lambda request, *_: graph_answer(request, 200, content=b"") + ) + with pytest.raises(StorageException, match="not JSON"): + storage.stat("a.txt") + + +def test_an_uninitialised_client_is_unavailable(monkeypatch: pytest.MonkeyPatch) -> None: + with pytest.raises(StorageUnavailableException, match="later_init") as caught: + OneDriveStorage(OneDriveClient()).exists("a.txt") + assert isinstance(caught.value.__cause__, OneDriveException) + monkeypatch.setattr(onedrive_instance, "_session", None) + with pytest.raises(StorageUnavailableException, match="device_code_login"): + OneDriveStorage().write_bytes("a.txt", b"x") + + +def test_a_closed_client_is_unavailable(storage: OneDriveStorage, client: OneDriveClient) -> None: + storage.write_bytes("a.txt", b"x") + client.close() + with pytest.raises(StorageUnavailableException): + storage.read_bytes("a.txt") + + +# ---------------------------------------------------------------------- identity and URIs + + +def test_uri_for(client: OneDriveClient) -> None: + assert OneDriveStorage(client).uri_for("") == "onedrive:///" + assert OneDriveStorage(client).uri_for("/reports//q1.csv") == "onedrive:///reports/q1.csv" + rooted = OneDriveStorage(client, root="/backups//2026/") + assert rooted.root == "backups/2026" + assert rooted.uri_for("") == "onedrive:///backups/2026" + assert rooted.uri_for("q1.csv") == "onedrive:///backups/2026/q1.csv" + + +def test_a_root_confines_the_backend_to_one_folder( + client: OneDriveClient, graph: FakeGraph +) -> None: + graph.add_file("other/keep.txt", b"k") + tenant = OneDriveStorage(client, root="tenants/a") + assert tenant.exists("") is False + with pytest.raises(StorageNotFoundException, match="onedrive:///tenants/a does not exist"): + tenant.write_bytes("docs/a.txt", b"x") + OneDriveStorage(client).mkdir("tenants/a") + tenant.write_bytes("docs/a.txt", b"x") + assert tenant.stat("").is_dir is True + assert graph.paths() == [ + "other", + "other/keep.txt", + "tenants", + "tenants/a", + "tenants/a/docs", + "tenants/a/docs/a.txt", + ] + assert [info.path for info in tenant.list_dir("", recursive=True)] == ["docs", "docs/a.txt"] + tenant.delete("docs", recursive=True) + assert graph.paths() == ["other", "other/keep.txt", "tenants", "tenants/a"] + with pytest.raises(StorageURIException): + OneDriveStorage(client, root="tenants/../other") + + +def test_equality_and_repr(client: OneDriveClient, graph: FakeGraph) -> None: + assert OneDriveStorage(client) == OneDriveStorage(client, root="/") + assert OneDriveStorage(client) != OneDriveStorage(client, root="a") + assert OneDriveStorage(client) != OneDriveStorage(_client(graph)) + assert OneDriveStorage(client) != OneDriveStorage() + assert OneDriveStorage() == OneDriveStorage() + assert len({OneDriveStorage(client), OneDriveStorage(client)}) == 1 + assert repr(OneDriveStorage(client, root="a/b")) == "OneDriveStorage(root='a/b')" + assert OneDriveStorage.scheme == ONEDRIVE_SCHEME == "onedrive" + assert OneDriveStorage.capabilities.to_dict() == { + "directories": True, + "modified_at": True, + "etag": True, + "version": False, + "content_type": True, + "metadata": False, + } + + +def test_onedrive_uris_use_the_shared_client( + monkeypatch: pytest.MonkeyPatch, client: OneDriveClient, graph: FakeGraph, tmp_path: Path +) -> None: + monkeypatch.setattr(onedrive_instance, "_session", client.require_session()) + resolver = StorageResolver() + resolver.register_scheme(ONEDRIVE_SCHEME, onedrive_factory) + report = File("onedrive:///reports/q1.csv", resolver=resolver) + report.write(b"a,b\n") + assert graph.at("reports/q1.csv").data == b"a,b\n" + assert resolver.resolve("onedrive:///reports/q1.csv") == (OneDriveStorage(), "reports/q1.csv") + report.copy_to(tmp_path / "q1.csv") + assert (tmp_path / "q1.csv").read_bytes() == b"a,b\n" + File(tmp_path / "q1.csv", resolver=resolver).copy_to("onedrive:///archive/2026/q1.csv") + report.move_to("onedrive:///archive/q1.csv") + assert graph.paths() == [ + "archive", + "archive/2026", + "archive/2026/q1.csv", + "archive/q1.csv", + "reports", + ] + assert "onedrive" in resolver.schemes() + + +def test_a_onedrive_uri_takes_no_authority() -> None: + resolver = StorageResolver() + resolver.register_scheme(ONEDRIVE_SCHEME, onedrive_factory) + with pytest.raises(StorageURIException, match=r"'onedrive:///reports/q1\.csv'") as caught: + resolver.resolve("onedrive://reports/q1.csv") + assert "'reports' as its authority" in str(caught.value) + with pytest.raises(StorageURIException, match="'onedrive:///reports'"): + resolver.resolve("onedrive://reports") + + +# ---------------------------------------------------------------------- the client and requests + + +def test_graph_send_returns_a_failed_response_instead_of_raising( + client: OneDriveClient, graph: FakeGraph +) -> None: + graph.fail_with = 503 + response = client.graph_send("GET", "/me/drive/root") + assert (response.status_code, response.ok) == (503, False) + with pytest.raises(OneDriveException, match="503"): + client.graph_request("GET", "/me/drive/root") + graph.fail_with = requests.ConnectionError("down") + with pytest.raises(requests.ConnectionError): + client.graph_send("GET", "/me/drive/root") + with pytest.raises(OneDriveException, match="graph request failed"): + client.graph_request("GET", "/me/drive/root") + + +def test_graph_send_can_leave_the_bearer_token_out( + client: OneDriveClient, graph: FakeGraph +) -> None: + graph.sessions[f"{UPLOAD_ORIGIN}/session/9"] = ("a.bin", bytearray()) + headers = {"Content-Range": "bytes 0-2/3"} + response = client.graph_send( + "PUT", f"{UPLOAD_ORIGIN}/session/9", authorized=False, data=b"abc", headers=headers + ) + assert response.status_code == 201 + assert headers == {"Content-Range": "bytes 0-2/3"} + assert "Authorization" not in graph.seen[-1].headers + # The session keeps its token for the next Graph call. + assert client.graph_send("GET", "/me/drive/root:/a.bin").json()["size"] == 3 + assert graph.seen[-1].headers["Authorization"] == f"Bearer {FAKE_TOKEN}" + + +def test_requests_has_what_the_adapter_relies_on() -> None: + """The stand-in is a transport; this names what the adapter needs of requests itself.""" + assert {"method", "url", "params", "data", "headers", "json", "timeout", "stream"} <= set( + inspect.signature(requests.Session.request).parameters + ) + assert "chunk_size" in inspect.signature(requests.Response.iter_content).parameters + assert callable(requests.Response.__enter__) + assert issubclass(requests.ConnectTimeout, (requests.ConnectionError, requests.Timeout)) + assert issubclass(requests.exceptions.SSLError, requests.ConnectionError) + assert issubclass(requests.exceptions.JSONDecodeError, ValueError) + for error in (requests.ConnectionError, requests.Timeout, requests.TooManyRedirects): + assert issubclass(error, requests.RequestException) + assert not issubclass(requests.exceptions.ChunkedEncodingError, requests.ConnectionError) diff --git a/tests/test_storage_resolver.py b/tests/test_storage_resolver.py index a90036f..c893c4c 100644 --- a/tests/test_storage_resolver.py +++ b/tests/test_storage_resolver.py @@ -39,7 +39,18 @@ def resolver() -> StorageResolver: def test_default_schemes(resolver: StorageResolver) -> None: - assert resolver.schemes() == ["azure", "local", "memory", "s3"] + assert resolver.schemes() == [ + "azure", + "dropbox", + "ftp", + "ftps", + "gdrive", + "local", + "memory", + "onedrive", + "s3", + "sftp", + ] assert StorageResolver(defaults=False).schemes() == [] @@ -78,7 +89,7 @@ def test_an_unknown_scheme_names_the_known_ones(resolver: StorageResolver) -> No message = str(caught.value) assert "'gopher'" in message assert "gopher://host/a.txt" in message - assert "azure, local, memory, s3" in message + assert "known schemes: azure, dropbox, ftp, ftps, gdrive, local, memory" in message def test_register_scheme_installs_a_factory(resolver: StorageResolver) -> None: diff --git a/tests/test_storage_s3.py b/tests/test_storage_s3.py index 3d630af..32a2910 100644 --- a/tests/test_storage_s3.py +++ b/tests/test_storage_s3.py @@ -438,3 +438,19 @@ def test_the_transfer_calls_exist_with_the_arguments_the_adapter_passes(real_cli "Key", ] assert real_client.can_paginate("list_objects_v2") is True + + +def test_two_prefixes_of_one_bucket_do_not_lose_a_file_to_itself(client: FakeS3Client) -> None: + whole = S3Storage("bucket", client=client) + tenant = S3Storage("bucket", client=client, prefix="tenant/a") + tenant.write_bytes("docs/a.txt", b"payload") + for operation in (whole.move_from, whole.copy_from): + with pytest.raises(StorageException, match="same file"): + operation(tenant, "docs/a.txt", "tenant/a/docs/a.txt") + with pytest.raises(StorageException, match="same file"): + tenant.move_from(whole, "tenant/a/docs/a.txt", "docs/a.txt") + assert client.buckets["bucket"]["tenant/a/docs/a.txt"].data == b"payload" + # Another client's bucket of the same name is another store. + elsewhere = S3Storage("bucket", client=FakeS3Client()) + elsewhere.copy_from(tenant, "docs/a.txt", "tenant/a/docs/a.txt") + assert elsewhere.read_bytes("tenant/a/docs/a.txt") == b"payload" diff --git a/tests/test_storage_sftp.py b/tests/test_storage_sftp.py new file mode 100644 index 0000000..ffb4480 --- /dev/null +++ b/tests/test_storage_sftp.py @@ -0,0 +1,939 @@ +"""SFTPStorage: the storage contract against an in-memory stand-in for paramiko's SFTP client. + +The stand-in keeps a tree of directories, files and symbolic links. It answers +the calls the adapter makes -- ``stat``, ``lstat``, ``listdir_attr``, ``put``, +``get``, ``remove``, ``mkdir``, ``rmdir``, ``posix_rename`` and ``rename`` -- with +paramiko's own ``SFTPAttributes`` and with the exceptions paramiko turns an SFTP +status into. No connection is opened. +""" + +from __future__ import annotations + +import errno +import inspect +import posixpath +import re +import stat +import threading +import time +from collections.abc import Callable +from dataclasses import dataclass +from datetime import datetime, timezone +from pathlib import Path + +import pytest + +pytest.importorskip("paramiko", reason="needs the sftp extra") + +# pylint: disable=wrong-import-position # importorskip must precede these imports +import paramiko +from paramiko.sftp import ( + SFTP_EOF, + SFTP_FAILURE, + SFTP_NO_SUCH_FILE, + SFTP_OP_UNSUPPORTED, + SFTP_PERMISSION_DENIED, +) + +from automation_file.exceptions import ( + StorageException, + StorageNotFoundException, + StoragePermissionException, + StorageTransientException, + StorageUnavailableException, + StorageURIException, +) +from automation_file.remote.sftp.client import SFTPClient, sftp_instance +from automation_file.storage import File, Storage, StorageBackend, StorageResolver +from automation_file.storage.session_storage import session_lock +from automation_file.storage.sftp_storage import ( + SFTP_SCHEME, + SFTPStorage, + sftp_factory, +) +from tests.storage_contract import StorageContract + +HOST = "nas.example" +MODIFIED = 1_791_426_600.0 + + +@dataclass +class _Directory: + modified: float + + +@dataclass +class _Regular: + data: bytes + modified: float + + +@dataclass +class _Link: + target: str + + +_Node = _Directory | _Regular | _Link + + +def _no_such_file() -> OSError: + """What paramiko raises for SSH_FX_NO_SUCH_FILE: a ``FileNotFoundError``.""" + return OSError(errno.ENOENT, "No such file") + + +def _denied() -> OSError: + """What paramiko raises for SSH_FX_PERMISSION_DENIED: a ``PermissionError``.""" + return OSError(errno.EACCES, "Permission denied") + + +def _failure(text: str = "Failure") -> OSError: + """What paramiko raises for every other status: an ``OSError`` without an errno.""" + return OSError(text) + + +@dataclass +class _Channel: + """What ``get_channel`` returns; paramiko refuses to send on a closed one.""" + + closed: bool = False + + +class FakeSFTP: + """The subset of ``paramiko.SFTPClient`` that SFTPStorage calls, over an in-memory tree.""" + + def __init__(self, *, posix_rename: bool = True) -> None: + self.nodes: dict[str, _Node] = {"/": _Directory(time.time())} + self.channel = _Channel() + self.calls: list[tuple[str, str]] = [] + self.fail_with: Exception | None = None + self.failures: dict[str, Exception] = {} + self.put_error: Exception | None = None + self.on_call: Callable[[], None] | None = None + self._posix_rename = posix_rename + + # ------------------------------------------------------------------ the tree + + def _called(self, name: str, path: str) -> None: + self.calls.append((name, path)) + if self.on_call is not None: + self.on_call() + if self.channel.closed: + raise OSError("Socket is closed") + error = self.fail_with or self.failures.get(name) + if error is not None: + raise error + + def _real(self, path: str, *, follow: bool = True) -> str: + """Resolve the links in ``path`` as a server does; the last one only when ``follow``.""" + parts = [part for part in path.split("/") if part] + current = "/" + for index, part in enumerate(parts): + current = posixpath.join(current, part) + node = self.nodes.get(current) + if isinstance(node, _Link) and (follow or index < len(parts) - 1): + current = self._real(posixpath.join(posixpath.dirname(current), node.target)) + return current + + def _node(self, path: str, *, follow: bool = True) -> tuple[str, _Node]: + real = self._real(path, follow=follow) + if real not in self.nodes: + raise _no_such_file() + return real, self.nodes[real] + + def _writable(self, path: str, *, follow: bool) -> str: + real = self._real(path, follow=follow) + if not isinstance(self.nodes.get(posixpath.dirname(real)), _Directory): + raise _no_such_file() + return real + + def _attributes(self, node: _Node, filename: str | None = None) -> paramiko.SFTPAttributes: + attributes = paramiko.SFTPAttributes() + if isinstance(node, _Directory): + attributes.st_mode, attributes.st_size = stat.S_IFDIR | 0o755, 4096 + attributes.st_mtime = int(node.modified) + elif isinstance(node, _Regular): + attributes.st_mode, attributes.st_size = stat.S_IFREG | 0o644, len(node.data) + attributes.st_mtime = int(node.modified) + else: + attributes.st_mode, attributes.st_size = stat.S_IFLNK | 0o777, len(node.target) + attributes.st_mtime = int(MODIFIED) + if filename is not None: + attributes.filename = filename + return attributes + + def _move(self, oldpath: str, newpath: str, *, replace: bool) -> None: + origin, _ = self._node(oldpath, follow=False) + target = self._writable(newpath, follow=False) + existing = self.nodes.get(target) + if isinstance(existing, _Directory) or (existing is not None and not replace): + raise _failure() + for key in [key for key in self.nodes if key == origin or key.startswith(f"{origin}/")]: + self.nodes[target + key[len(origin) :]] = self.nodes.pop(key) + + def paths(self) -> list[str]: + return sorted(key for key in self.nodes if key != "/") + + def write(self, path: str, data: bytes, modified: float = MODIFIED) -> None: + self.nodes[path] = _Regular(data, modified) + + def read(self, path: str) -> bytes: + node = self.nodes[path] + assert isinstance(node, _Regular) + return node.data + + # ------------------------------------------------------------------ paramiko.SFTPClient + + def get_channel(self) -> _Channel: + return self.channel + + def stat(self, path: str) -> paramiko.SFTPAttributes: + self._called("stat", path) + return self._attributes(self._node(path)[1]) + + def lstat(self, path: str) -> paramiko.SFTPAttributes: + self._called("lstat", path) + return self._attributes(self._node(path, follow=False)[1]) + + def listdir_attr(self, path: str = ".") -> list[paramiko.SFTPAttributes]: + self._called("listdir_attr", path) + real, node = self._node(path) + if not isinstance(node, _Directory): + raise _no_such_file() + return [ + self._attributes(child, posixpath.basename(key)) + for key, child in self.nodes.items() + if key != "/" and posixpath.dirname(key) == real + ] + + def put(self, localpath: str, remotepath: str, callback=None, confirm: bool = True): + self._called("put", remotepath) + data = Path(localpath).read_bytes() + real = self._writable(remotepath, follow=True) + if isinstance(self.nodes.get(real), _Directory): + raise _failure() + if self.put_error is not None: + self.nodes[real] = _Regular(data[: len(data) // 2], time.time()) + raise self.put_error + self.nodes[real] = _Regular(data, time.time()) + return self._attributes(self.nodes[real]) + + def get( + self, + remotepath: str, + localpath: str, + callback=None, + prefetch: bool = True, + max_concurrent_prefetch_requests=None, + ) -> None: + self._called("get", remotepath) + node = self._node(remotepath)[1] + if not isinstance(node, _Regular): + raise _failure() + Path(localpath).write_bytes(node.data) + + def remove(self, path: str) -> None: + self._called("remove", path) + real, node = self._node(path, follow=False) + if isinstance(node, _Directory): + raise _failure() + del self.nodes[real] + + def mkdir(self, path: str, mode: int = 0o777) -> None: + self._called("mkdir", path) + real = self._writable(path, follow=False) + if real in self.nodes: + raise _failure() + self.nodes[real] = _Directory(time.time()) + + def rmdir(self, path: str) -> None: + self._called("rmdir", path) + real, node = self._node(path, follow=False) + if not isinstance(node, _Directory): + raise _no_such_file() + if any(posixpath.dirname(key) == real for key in self.nodes if key != "/"): + raise _failure() + del self.nodes[real] + + def posix_rename(self, oldpath: str, newpath: str) -> None: + self._called("posix_rename", newpath) + if not self._posix_rename: + raise _failure("Operation unsupported") + self._move(oldpath, newpath, replace=True) + + def rename(self, oldpath: str, newpath: str) -> None: + self._called("rename", newpath) + self._move(oldpath, newpath, replace=False) + + def symlink(self, source: str, dest: str) -> None: + self.nodes[dest] = _Link(source) + + +def connected( + session: FakeSFTP | None, host: str | None = HOST, port: int | None = 22 +) -> SFTPClient: + """Return an ``SFTPClient`` whose session is ``session``, as ``later_init`` leaves it.""" + client = SFTPClient() + client._sftp = session + client._host = host + client._port = port + return client + + +class TestSFTPStorageContract(StorageContract): + @pytest.fixture + def backend(self) -> StorageBackend: + return SFTPStorage(connected(FakeSFTP())) + + +class TestRootedSFTPStorageContract(StorageContract): + @pytest.fixture + def backend(self) -> StorageBackend: + session = FakeSFTP() + session.mkdir("/srv") + session.mkdir("/srv/data") + session.write("/srv/keep.txt", b"keep") + return SFTPStorage(connected(session), root="/srv/data") + + +class TestSFTPStorageWithoutPosixRenameContract(StorageContract): + @pytest.fixture + def backend(self) -> StorageBackend: + return SFTPStorage(connected(FakeSFTP(posix_rename=False))) + + +@pytest.fixture +def session() -> FakeSFTP: + return FakeSFTP() + + +@pytest.fixture +def storage(session: FakeSFTP) -> SFTPStorage: + return SFTPStorage(connected(session)) + + +@pytest.fixture +def shared(monkeypatch: pytest.MonkeyPatch, session: FakeSFTP) -> FakeSFTP: + """Make ``session`` the open session of the shared ``sftp_instance``.""" + monkeypatch.setattr(sftp_instance, "_sftp", session) + monkeypatch.setattr(sftp_instance, "_host", HOST) + monkeypatch.setattr(sftp_instance, "_port", 22) + return session + + +@pytest.fixture +def resolver() -> StorageResolver: + table = StorageResolver() + table.register_scheme(SFTP_SCHEME, sftp_factory) + return table + + +# ---------------------------------------------------------------------- stat and paths + + +def test_stat_reports_the_size_and_the_modification_time( + storage: SFTPStorage, session: FakeSFTP +) -> None: + session.mkdir("/reports") + session.write("/reports/q1.csv", b"a,b\n") + info = storage.stat("reports/q1.csv") + assert (info.path, info.is_dir, info.size) == ("reports/q1.csv", False, 4) + assert info.modified_at == datetime.fromtimestamp(MODIFIED, timezone.utc) + assert (info.etag, info.version, info.content_type, dict(info.metadata)) == ( + None, + None, + None, + {}, + ) + directory = storage.stat("reports") + assert (directory.is_dir, directory.size) == (True, None) + assert storage.capabilities.to_dict() == { + "directories": True, + "modified_at": True, + "etag": False, + "version": False, + "content_type": False, + "metadata": False, + } + + +def test_attributes_a_server_leaves_out_stay_unknown( + storage: SFTPStorage, session: FakeSFTP, monkeypatch: pytest.MonkeyPatch +) -> None: + session.write("/bare", b"x") + monkeypatch.setattr( + session, "_attributes", lambda node, filename=None: paramiko.SFTPAttributes() + ) + info = storage.stat("bare") + assert (info.is_dir, info.size, info.modified_at) == (False, None, None) + + +def test_every_path_is_joined_to_the_root(session: FakeSFTP) -> None: + session.mkdir("/srv") + session.mkdir("/srv/data") + session.write("/srv/keep.txt", b"keep") + rooted = SFTPStorage(connected(session), root="srv//data/") + assert rooted.root == "/srv/data" + rooted.write_bytes("docs/a.txt", b"x") + assert session.paths() == [ + "/srv", + "/srv/data", + "/srv/data/docs", + "/srv/data/docs/a.txt", + "/srv/keep.txt", + ] + assert [info.path for info in rooted.list_dir("", recursive=True)] == ["docs", "docs/a.txt"] + assert rooted.uri_for("docs/a.txt") == "sftp://nas.example/srv/data/docs/a.txt" + rooted.delete("docs", recursive=True) + assert session.paths() == ["/srv", "/srv/data", "/srv/keep.txt"] + with pytest.raises(StorageURIException): + SFTPStorage(connected(session), root="/srv/../etc") + + +def test_uri_for_names_the_host_of_the_session(session: FakeSFTP) -> None: + assert SFTPStorage(connected(session)).uri_for("data/a.txt") == "sftp://nas.example/data/a.txt" + assert SFTPStorage(connected(session)).uri_for("") == "sftp://nas.example" + assert ( + SFTPStorage(connected(session, port=2222)).uri_for("a.txt") + == "sftp://nas.example:2222/a.txt" + ) + assert SFTPStorage(connected(session, host="::1")).uri_for("a.txt") == "sftp://[::1]/a.txt" + assert SFTPStorage(connected(None, host=None, port=None)).uri_for("a.txt") == "sftp:///a.txt" + + +def test_equality_and_repr(session: FakeSFTP) -> None: + client = connected(session) + assert SFTPStorage(client) == SFTPStorage(client) + assert SFTPStorage(client) != SFTPStorage(client, root="/srv") + assert SFTPStorage(client) != SFTPStorage(connected(session)) + assert SFTPStorage() == SFTPStorage(sftp_instance) + assert len({SFTPStorage(client), SFTPStorage(client, root="/")}) == 1 + assert repr(SFTPStorage(client, root="srv/data")) == "SFTPStorage(root='/srv/data')" + assert SFTPStorage(client).client is client + assert SFTPStorage().client is sftp_instance + + +# ---------------------------------------------------------------------- errors + + +@pytest.mark.parametrize( + "error,expected", + [ + (_denied(), StoragePermissionException), + (paramiko.SSHException("Server connection dropped: "), StorageTransientException), + (EOFError(), StorageTransientException), + (TimeoutError("timed out"), StorageTransientException), + (ConnectionResetError(errno.ECONNRESET, "reset by peer"), StorageTransientException), + (_failure(), StorageException), + (paramiko.SFTPError("Expected attributes"), StorageException), + (UnicodeDecodeError("utf-8", b"caf\xe9", 3, 4, "invalid start byte"), StorageException), + ], +) +def test_session_errors_become_storage_errors( + storage: SFTPStorage, session: FakeSFTP, error: Exception, expected: type[Exception] +) -> None: + session.fail_with = error + with pytest.raises(expected) as caught: + storage.stat("a.txt") + assert caught.value.__cause__ is error + assert type(caught.value) is expected + + +def test_a_closed_channel_is_transient(storage: SFTPStorage, session: FakeSFTP) -> None: + storage.write_bytes("a.txt", b"old") + session.channel.closed = True + for call in (storage.exists, storage.list_dir, storage.read_bytes, storage.mkdir): + with pytest.raises(StorageTransientException, match="session is closed") as caught: + call("a.txt") + assert str(caught.value.__cause__) == "Socket is closed" + + +def test_a_channel_that_closes_during_a_rename_is_not_a_refusal( + storage: SFTPStorage, session: FakeSFTP +) -> None: + storage.write_bytes("a.txt", b"old") + session.calls.clear() + + def close_before_the_rename() -> None: + if session.calls[-1][0] == "posix_rename": + session.channel.closed = True + + session.on_call = close_before_the_rename + with pytest.raises(StorageTransientException, match="session is closed"): + storage.write_bytes("a.txt", b"new") + assert [name for name, _ in session.calls] == ["stat", "put", "posix_rename", "remove"] + assert session.read("/a.txt") == b"old" + + +def test_a_missing_path_is_not_found(storage: SFTPStorage) -> None: + assert storage.exists("nope/a.txt") is False + with pytest.raises(StorageNotFoundException, match=re.escape("sftp://nas.example/nope")): + storage.list_dir("nope") + + +def test_paramiko_reports_a_status_the_way_the_adapter_reads_it() -> None: + """The adapter tells the SFTP statuses apart by what paramiko's own converter raises.""" + + def convert(code: int) -> None: + message = paramiko.Message() + message.add_int(code) + message.add_string("text") + message.rewind() + paramiko.SFTPClient._convert_status(None, message) + + with pytest.raises(FileNotFoundError): + convert(SFTP_NO_SUCH_FILE) + with pytest.raises(PermissionError): + convert(SFTP_PERMISSION_DENIED) + with pytest.raises(EOFError): + convert(SFTP_EOF) + for code in (SFTP_FAILURE, SFTP_OP_UNSUPPORTED): + with pytest.raises(OSError) as caught: + convert(code) + assert type(caught.value) is OSError + assert caught.value.errno is None + assert type(_no_such_file()) is FileNotFoundError + assert type(_denied()) is PermissionError + + +def test_a_session_must_be_open(monkeypatch: pytest.MonkeyPatch) -> None: + with pytest.raises(StorageUnavailableException, match="later_init"): + SFTPStorage(SFTPClient()).exists("a.txt") + monkeypatch.setattr(sftp_instance, "_sftp", None) + with pytest.raises(StorageUnavailableException, match="later_init") as caught: + SFTPStorage().write_bytes("a.txt", b"x") + assert isinstance(caught.value.__cause__, RuntimeError) + + +# ---------------------------------------------------------------------- uploads and moves + + +def test_an_upload_arrives_under_a_part_name_and_is_renamed( + storage: SFTPStorage, session: FakeSFTP +) -> None: + storage.write_bytes("a.txt", b"payload") + (put, partial), (rename, target) = [ + call for call in session.calls if call[0] in ("put", "posix_rename", "rename") + ] + assert (put, rename, target) == ("put", "posix_rename", "/a.txt") + assert posixpath.dirname(partial) == "/" + assert posixpath.basename(partial).startswith(".a.txt.") + assert partial.endswith(".part") + assert session.paths() == ["/a.txt"] + + +@pytest.mark.parametrize( + "error,expected", + [ + (TimeoutError("timed out"), StorageTransientException), + (_failure("size mismatch in put! 3 != 7"), StorageException), + ], +) +def test_a_failed_upload_leaves_no_partial_file_and_keeps_the_target( + storage: SFTPStorage, session: FakeSFTP, error: Exception, expected: type[Exception] +) -> None: + session.write("/a.txt", b"old") + session.put_error = error + with pytest.raises(expected) as caught: + storage.write_bytes("a.txt", b"new and longer") + assert caught.value.__cause__ is error + assert session.paths() == ["/a.txt"] + assert session.read("/a.txt") == b"old" + + +def test_a_refused_rename_leaves_no_partial_file_and_keeps_the_target( + storage: SFTPStorage, session: FakeSFTP +) -> None: + session.write("/a.txt", b"old") + session.failures["posix_rename"] = _denied() + with pytest.raises(StoragePermissionException): + storage.write_bytes("a.txt", b"new") + assert session.paths() == ["/a.txt"] + assert session.read("/a.txt") == b"old" + + +def test_the_upload_error_is_reported_when_the_cleanup_fails_too( + storage: SFTPStorage, session: FakeSFTP +) -> None: + session.put_error = EOFError() + session.failures["remove"] = paramiko.SSHException("Server connection dropped: ") + with pytest.raises(StorageTransientException) as caught: + storage.write_bytes("a.txt", b"payload") + assert caught.value.__cause__ is session.put_error + assert [posixpath.basename(path)[:7] for path in session.paths()] == [".a.txt."] + + +@pytest.fixture +def old_session() -> FakeSFTP: + """A server without ``posix-rename@openssh.com``: its rename never replaces a file.""" + return FakeSFTP(posix_rename=False) + + +def fail_the_nth(session: FakeSFTP, name: str, nth: set[int], error: Exception) -> None: + """Make the calls of the method ``name`` whose ordinal is in ``nth`` raise ``error``.""" + seen = 0 + + def before_each_call() -> None: + nonlocal seen + session.failures.pop(name, None) + if session.calls[-1][0] == name: + seen += 1 + if seen in nth: + session.failures[name] = error + + session.on_call = before_each_call + + +def test_a_server_without_posix_rename_has_the_old_file_moved_aside( + old_session: FakeSFTP, +) -> None: + storage = SFTPStorage(connected(old_session)) + storage.write_bytes("a.txt", b"first") + assert [name for name, _ in old_session.calls if "rename" in name] == [ + "posix_rename", + "rename", + ] + old_session.calls.clear() + storage.write_bytes("a.txt", b"second") + partial = next(path for name, path in old_session.calls if name == "put") + aside = next(path for name, path in old_session.calls if path.endswith(".old")) + assert posixpath.basename(aside).startswith(".a.txt.") + assert [call for call in old_session.calls if call[0] != "stat"] == [ + ("put", partial), + ("posix_rename", "/a.txt"), + ("rename", "/a.txt"), + ("posix_rename", aside), + ("rename", aside), + ("posix_rename", "/a.txt"), + ("rename", "/a.txt"), + ("remove", aside), + ] + assert old_session.paths() == ["/a.txt"] + assert old_session.read("/a.txt") == b"second" + + +def test_a_rename_refused_for_another_reason_keeps_the_target(old_session: FakeSFTP) -> None: + storage = SFTPStorage(connected(old_session)) + refusal = _failure("Quota exceeded") + old_session.failures["rename"] = refusal + with pytest.raises(StorageException) as caught: + storage.write_bytes("new.txt", b"payload") + assert caught.value.__cause__ is refusal + assert old_session.paths() == [] + + old_session.write("/a.txt", b"old") + with pytest.raises(StorageException) as caught: + storage.write_bytes("a.txt", b"payload") + assert caught.value.__cause__ is refusal + assert old_session.paths() == ["/a.txt"] + assert old_session.read("/a.txt") == b"old" + + +@pytest.mark.parametrize( + "error,expected", + [ + (_failure("Disk full"), StorageException), + (paramiko.SSHException("Server connection dropped: "), StorageTransientException), + ], +) +def test_the_old_file_is_put_back_when_the_rename_still_fails( + old_session: FakeSFTP, error: Exception, expected: type[Exception] +) -> None: + storage = SFTPStorage(connected(old_session)) + old_session.write("/a.txt", b"old") + # Renames: onto the file (refused), the file aside, onto the freed name, the file back. + fail_the_nth(old_session, "rename", {3}, error) + with pytest.raises(expected) as caught: + storage.write_bytes("a.txt", b"new") + assert caught.value.__cause__ is error + assert old_session.paths() == ["/a.txt"] + assert old_session.read("/a.txt") == b"old" + + +@pytest.mark.parametrize( + "error", [_failure("Disk full"), paramiko.SSHException("Server connection dropped: ")] +) +def test_the_old_content_stays_aside_when_it_cannot_be_put_back( + old_session: FakeSFTP, error: Exception +) -> None: + storage = SFTPStorage(connected(old_session)) + old_session.write("/a.txt", b"old") + fail_the_nth(old_session, "rename", {3, 4}, error) + with pytest.raises(StorageException) as caught: + storage.write_bytes("a.txt", b"new") + assert caught.value.__cause__ is error + (aside,) = old_session.paths() + assert posixpath.basename(aside).startswith(".a.txt.") + assert aside.endswith(".old") + assert old_session.read(aside) == b"old" + + +def test_a_move_within_one_session_is_a_rename(storage: SFTPStorage, session: FakeSFTP) -> None: + storage.write_bytes("a.txt", b"payload") + session.calls.clear() + info = storage.move_from(storage, "a.txt", "moved/b.txt") + assert (info.path, info.size) == ("moved/b.txt", 7) + assert ("posix_rename", "/moved/b.txt") in session.calls + assert not [call for call in session.calls if call[0] in ("get", "put", "remove")] + assert session.paths() == ["/moved", "/moved/b.txt"] + + archive = SFTPStorage(storage.client, root="/moved") + archive.move_from(storage, "moved/b.txt", "c.txt") + assert session.paths() == ["/moved", "/moved/c.txt"] + with pytest.raises(StorageException, match="same file"): + archive.move_from(storage, "moved/c.txt", "c.txt") + assert session.read("/moved/c.txt") == b"payload" + + +def test_a_move_between_two_sessions_is_copy_then_delete( + storage: SFTPStorage, session: FakeSFTP +) -> None: + other_session = FakeSFTP() + other = SFTPStorage(connected(other_session, host="backup.example")) + storage.write_bytes("a.txt", b"payload") + other.move_from(storage, "a.txt", "a.txt") + assert other_session.read("/a.txt") == b"payload" + assert session.paths() == [] + assert "get" in [name for name, _ in session.calls] + + +def test_mkdir_accepts_a_directory_that_appeared_meanwhile( + storage: SFTPStorage, session: FakeSFTP +) -> None: + session.mkdir("/dir") + storage._mkdir("dir") + session.write("/a.txt", b"x") + with pytest.raises(StorageException) as caught: + storage._mkdir("a.txt") + assert type(caught.value) is StorageException + session.failures["mkdir"] = _failure("Quota exceeded") + with pytest.raises(StorageException) as caught: + storage._mkdir("new") + assert caught.value.__cause__ is session.failures.pop("mkdir") + with pytest.raises(StorageNotFoundException): + SFTPStorage(storage.client, root="/missing")._mkdir("child") + + +def test_a_file_that_vanished_is_not_found(storage: SFTPStorage, session: FakeSFTP) -> None: + storage.write_bytes("a.txt", b"x") + + def vanish_before_the_download() -> None: + if session.calls[-1][0] == "get": + del session.nodes["/a.txt"] + + session.on_call = vanish_before_the_download + with pytest.raises(StorageNotFoundException, match="does not exist"): + storage.read_bytes("a.txt") + + +# ---------------------------------------------------------------------- symbolic links + + +@pytest.fixture +def linked(storage: SFTPStorage, session: FakeSFTP) -> SFTPStorage: + """``/data`` with a link to a file, to a directory outside it, and to nothing.""" + for path in ("outside/keep.txt", "data/real.txt"): + storage.write_bytes(path, b"content") + session.symlink("real.txt", "/data/file-link") + session.symlink("/outside", "/data/dir-link") + session.symlink("/gone", "/data/dangling") + return storage + + +def test_links_are_followed_when_reading_and_listing(linked: SFTPStorage) -> None: + assert linked.read_bytes("data/file-link") == b"content" + assert linked.read_bytes("data/dir-link/keep.txt") == b"content" + assert [(info.name, info.is_dir, info.size) for info in linked.list_dir("data")] == [ + ("dangling", False, 5), + ("dir-link", True, None), + ("file-link", False, 7), + ("real.txt", False, 7), + ] + assert linked.exists("data/dangling") is False + + +def test_a_recursive_listing_does_not_descend_into_a_linked_directory(linked: SFTPStorage) -> None: + assert [info.path for info in linked.list_dir("data", recursive=True)] == [ + "data/dangling", + "data/dir-link", + "data/file-link", + "data/real.txt", + ] + + +def test_deleting_never_follows_a_link(linked: SFTPStorage, session: FakeSFTP) -> None: + linked.delete("data/dir-link", recursive=True) + assert "/data/dir-link" not in session.nodes + assert session.read("/outside/keep.txt") == b"content" + linked.delete("data/file-link") + assert session.read("/data/real.txt") == b"content" + + session.symlink("/outside", "/data/dir-link") + linked.delete("data", recursive=True) + assert session.paths() == ["/outside", "/outside/keep.txt"] + + +# ---------------------------------------------------------------------- sftp:// URIs + + +def test_sftp_uris_use_the_shared_session( + shared: FakeSFTP, resolver: StorageResolver, tmp_path: Path +) -> None: + report = File("sftp://nas.example/reports/q1.csv", resolver=resolver) + report.write(b"a,b\n") + assert shared.read("/reports/q1.csv") == b"a,b\n" + assert resolver.resolve("sftp://nas.example/reports/q1.csv") == ( + SFTPStorage(), + "reports/q1.csv", + ) + report.copy_to(tmp_path / "q1.csv") + assert (tmp_path / "q1.csv").read_bytes() == b"a,b\n" + File(tmp_path / "q1.csv", resolver=resolver).move_to("sftp:///archive/2026/q1.csv") + assert shared.read("/archive/2026/q1.csv") == b"a,b\n" + archive = Storage("sftp://nas.example/archive", resolver=resolver) + assert [info.path for info in archive.list_dir(recursive=True)] == ["2026", "2026/q1.csv"] + archive.delete("2026", recursive=True) + assert shared.paths() == ["/archive", "/reports", "/reports/q1.csv"] + + +@pytest.mark.parametrize( + "uri", + [ + "sftp:///data/a.txt", + "sftp://nas.example/data/a.txt", + "sftp://NAS.Example/data/a.txt", + "sftp://nas.example:22/data/a.txt", + ], +) +def test_a_uri_may_name_no_host_or_the_connected_one( + shared: FakeSFTP, resolver: StorageResolver, uri: str +) -> None: + assert resolver.resolve(uri) == (SFTPStorage(), "data/a.txt") + + +@pytest.mark.parametrize( + "uri,named", + [ + ("sftp://backup.example/data/a.txt", "backup.example"), + ("sftp://nas.example:2222/data/a.txt", "nas.example:2222"), + ], +) +def test_a_uri_for_another_host_is_refused( + shared: FakeSFTP, resolver: StorageResolver, uri: str, named: str +) -> None: + with pytest.raises(StorageURIException) as caught: + resolver.resolve(uri) + message = str(caught.value) + assert repr(named) in message + assert "'nas.example:22'" in message + assert f'Storage.mount("sftp://{named}", SFTPStorage(client))' in message + + +def test_a_malformed_port_is_refused(shared: FakeSFTP, resolver: StorageResolver) -> None: + with pytest.raises(StorageURIException, match="port"): + resolver.resolve("sftp://nas.example:ssh/data/a.txt") + + +def test_an_ipv6_host_is_written_in_brackets( + shared: FakeSFTP, resolver: StorageResolver, monkeypatch: pytest.MonkeyPatch +) -> None: + monkeypatch.setattr(sftp_instance, "_host", "fd00::1") + assert resolver.resolve("sftp://[FD00::1]:22/a.txt") == (SFTPStorage(), "a.txt") + assert resolver.resolve("sftp://[fd00::1]/a.txt") == (SFTPStorage(), "a.txt") + with pytest.raises(StorageURIException): + resolver.resolve("sftp://[fd00::2]/a.txt") + + +def test_any_host_resolves_until_a_session_is_open( + monkeypatch: pytest.MonkeyPatch, resolver: StorageResolver +) -> None: + for name in ("_sftp", "_host", "_port"): + monkeypatch.setattr(sftp_instance, name, None) + backend, path = resolver.resolve("sftp://anywhere.example/data/a.txt") + assert (backend, path) == (SFTPStorage(), "data/a.txt") + with pytest.raises(StorageUnavailableException, match="later_init"): + backend.exists(path) + + +def test_another_host_is_reached_through_a_mount( + shared: FakeSFTP, resolver: StorageResolver +) -> None: + backup_session = FakeSFTP() + resolver.mount( + "sftp://backup.example", SFTPStorage(connected(backup_session, host="backup.example")) + ) + File("sftp://nas.example/a.txt", resolver=resolver).write(b"payload") + File("sftp://nas.example/a.txt", resolver=resolver).copy_to( + File("sftp://backup.example/copies/a.txt", resolver=resolver) + ) + assert backup_session.read("/copies/a.txt") == b"payload" + assert shared.paths() == ["/a.txt"] + + +# ---------------------------------------------------------------------- one caller at a time + + +def test_a_session_serves_one_operation_at_a_time(storage: SFTPStorage, session: FakeSFTP) -> None: + acquired: list[bool] = [] + + def from_another_thread() -> None: + thread = threading.Thread( + target=lambda: acquired.append(session_lock(session).acquire(blocking=False)) + ) + thread.start() + thread.join() + + session.on_call = from_another_thread + storage.write_bytes("dir/a.txt", b"x") + storage.list_dir("", recursive=True) + storage.delete("dir", recursive=True) + assert acquired + assert not any(acquired) + assert session_lock(session) is session_lock(session) + assert session_lock(session) is not session_lock(FakeSFTP()) + + +# ---------------------------------------------------------------------- the real paramiko client + + +@pytest.mark.parametrize( + "method,arguments", + [ + ("stat", ("/a",)), + ("lstat", ("/a",)), + ("listdir_attr", ("/a",)), + ("put", ("local", "/a")), + ("get", ("/a", "local")), + ("remove", ("/a",)), + ("mkdir", ("/a",)), + ("rmdir", ("/a",)), + ("posix_rename", ("/a", "/b")), + ("rename", ("/a", "/b")), + ("get_channel", ()), + ], +) +def test_paramiko_has_the_calls_the_adapter_makes(method: str, arguments: tuple[str, ...]) -> None: + real = inspect.signature(getattr(paramiko.SFTPClient, method)) + real.bind(None, *arguments) + stand_in = list(inspect.signature(getattr(FakeSFTP, method)).parameters) + assert list(real.parameters)[: len(stand_in)] == stand_in + + +def test_paramiko_attributes_carry_the_fields_the_adapter_reads() -> None: + attributes = paramiko.SFTPAttributes() + assert (attributes.st_mode, attributes.st_size, attributes.st_mtime) == (None, None, None) + assert issubclass(paramiko.SSHException, Exception) + assert not issubclass(paramiko.SFTPError, OSError) + + +def test_two_roots_of_one_session_do_not_lose_a_file_to_itself(session: FakeSFTP) -> None: + client = connected(session) + whole = SFTPStorage(client) + whole.mkdir("team/a") + inner = SFTPStorage(client, root="/team/a") + inner.write_bytes("docs/a.txt", b"payload") + for operation in (whole.move_from, whole.copy_from): + with pytest.raises(StorageException, match="same file"): + operation(inner, "docs/a.txt", "team/a/docs/a.txt") + with pytest.raises(StorageException, match="same file"): + inner.move_from(whole, "team/a/docs/a.txt", "docs/a.txt") + assert whole.read_bytes("team/a/docs/a.txt") == b"payload" diff --git a/tests/test_storage_sftp_loopback.py b/tests/test_storage_sftp_loopback.py new file mode 100644 index 0000000..02832ee --- /dev/null +++ b/tests/test_storage_sftp_loopback.py @@ -0,0 +1,239 @@ +"""SFTPStorage through the real paramiko client. + +The client talks to paramiko's own ``SFTPServer`` over a socket pair inside the +process: no SSH daemon and no network, but every request is encoded, sent and +answered by paramiko exactly as it is against a remote host. The server keeps +its files in a temporary directory. + +``test_storage_sftp.py`` runs the same contract against an in-memory stand-in, +which can fail on demand and can hold symbolic links. This module is what shows +that the stand-in answers the way paramiko does. +""" + +from __future__ import annotations + +import contextlib +import os +import socket +import threading +from collections.abc import Iterator +from pathlib import Path +from typing import Any + +import pytest + +pytest.importorskip("paramiko", reason="needs the sftp extra") + +# pylint: disable=wrong-import-position # importorskip must precede these imports +import paramiko +from paramiko.sftp import SFTP_FAILURE, SFTP_OK, SFTP_OP_UNSUPPORTED + +from automation_file.exceptions import ( + StorageException, + StorageNotEmptyException, + StorageTransientException, +) +from automation_file.remote.sftp.client import SFTPClient +from automation_file.storage import StorageBackend +from automation_file.storage.sftp_storage import SFTPStorage +from tests.storage_contract import StorageContract + +TIMEOUT = 30.0 +_BINARY = getattr(os, "O_BINARY", 0) + + +class _Server(paramiko.ServerInterface): + """Lets anyone in: the peer is this very process.""" + + def check_auth_none(self, username: str) -> int: + return paramiko.AUTH_SUCCESSFUL + + def get_allowed_auths(self, username: str) -> str: + return "none" + + def check_channel_request(self, kind: str, chanid: int) -> int: + return paramiko.OPEN_SUCCEEDED + + +class _Files(paramiko.SFTPServerInterface): + """The files below one local directory, served as the SFTP root ``/``.""" + + def __init__(self, server: Any, root: Path, posix_rename: bool) -> None: + super().__init__(server) + self._root = root + self._posix_rename = posix_rename + + def _local(self, path: str) -> str: + return str(self._root) + self.canonicalize(path) + + def _attempt(self, action: Any, *arguments: Any) -> Any: + """Run a filesystem call and answer its ``OSError`` with the SFTP status for it.""" + try: + return action(*arguments) + except OSError as error: + return paramiko.SFTPServer.convert_errno(error.errno) + + def _listing(self, directory: str) -> list[paramiko.SFTPAttributes]: + return [ + paramiko.SFTPAttributes.from_stat(os.lstat(os.path.join(directory, name)), name) + for name in os.listdir(directory) + ] + + def _opened(self, local: str, flags: int) -> paramiko.SFTPHandle: + mode = "wb" if flags & os.O_WRONLY else "r+b" if flags & os.O_RDWR else "rb" + stream = os.fdopen(os.open(local, flags | _BINARY, 0o666), mode) + handle = paramiko.SFTPHandle(flags) + handle.readfile = handle.writefile = stream + return handle + + def _renamed(self, origin: str, target: str) -> int: + os.rename(origin, target) + return SFTP_OK + + def _replaced(self, origin: str, target: str) -> int: + os.replace(origin, target) + return SFTP_OK + + def list_folder(self, path: str) -> Any: + return self._attempt(self._listing, self._local(path)) + + def stat(self, path: str) -> Any: + return self._attempt(lambda: paramiko.SFTPAttributes.from_stat(os.stat(self._local(path)))) + + def lstat(self, path: str) -> Any: + return self._attempt(lambda: paramiko.SFTPAttributes.from_stat(os.lstat(self._local(path)))) + + def open(self, path: str, flags: int, attr: Any) -> Any: + return self._attempt(self._opened, self._local(path), flags) + + def remove(self, path: str) -> int: + return self._attempt(os.remove, self._local(path)) or SFTP_OK + + def mkdir(self, path: str, attr: Any) -> int: + return self._attempt(os.mkdir, self._local(path)) or SFTP_OK + + def rmdir(self, path: str) -> int: + return self._attempt(os.rmdir, self._local(path)) or SFTP_OK + + def rename(self, oldpath: str, newpath: str) -> int: + """Plain SFTP rename as OpenSSH does it: it never replaces what is at the new name.""" + if os.path.lexists(self._local(newpath)): + return SFTP_FAILURE + return self._attempt(self._renamed, self._local(oldpath), self._local(newpath)) + + def posix_rename(self, oldpath: str, newpath: str) -> int: + if not self._posix_rename: + return SFTP_OP_UNSUPPORTED + return self._attempt(self._replaced, self._local(oldpath), self._local(newpath)) + + +@pytest.fixture(scope="module") +def host_key() -> paramiko.PKey: + return paramiko.RSAKey.generate(2048) + + +@contextlib.contextmanager +def loopback( + root: Path, host_key: paramiko.PKey, *, posix_rename: bool = True +) -> Iterator[SFTPClient]: + """Yield an ``SFTPClient`` whose session is a real paramiko one, served from ``root``. + + The transports are joined directly, without ``SSHClient``: there is no host to + verify, both ends of the socket pair are this process. + """ + server_socket, client_socket = socket.socketpair() + server = paramiko.Transport(server_socket) + client = paramiko.Transport(client_socket) + try: + server.add_server_key(host_key) + server.set_subsystem_handler("sftp", paramiko.SFTPServer, _Files, root, posix_rename) + server.start_server(event=threading.Event(), server=_Server()) + client.start_client(timeout=TIMEOUT) + client.auth_none("tester") + session = paramiko.SFTPClient.from_transport(client) + assert session is not None + session.get_channel().settimeout(TIMEOUT) + connected = SFTPClient() + connected._sftp = session + connected._host = "loopback" + connected._port = 22 + yield connected + finally: + client.close() + server.close() + + +@pytest.fixture +def served(tmp_path: Path) -> Path: + """The local directory the loopback server serves as ``/``.""" + root = tmp_path / "served" + root.mkdir() + return root + + +@pytest.fixture +def storage(served: Path, host_key: paramiko.PKey) -> Iterator[SFTPStorage]: + with loopback(served, host_key) as client: + yield SFTPStorage(client) + + +class TestSFTPStorageThroughParamikoContract(StorageContract): + @pytest.fixture + def backend(self, storage: SFTPStorage) -> StorageBackend: + return storage + + +class TestSFTPStorageThroughParamikoWithoutPosixRenameContract(StorageContract): + @pytest.fixture + def backend(self, served: Path, host_key: paramiko.PKey) -> Iterator[StorageBackend]: + with loopback(served, host_key, posix_rename=False) as client: + yield SFTPStorage(client) + + +def test_stat_reports_what_the_server_returns(storage: SFTPStorage, served: Path) -> None: + (served / "reports").mkdir() + (served / "reports" / "q1.csv").write_bytes(b"a,b\n") + info = storage.stat("reports/q1.csv") + assert (info.path, info.is_dir, info.size) == ("reports/q1.csv", False, 4) + assert info.modified_at is not None + assert int(info.modified_at.timestamp()) == int((served / "reports" / "q1.csv").stat().st_mtime) + assert (storage.stat("reports").is_dir, storage.stat("reports").size) == (True, None) + assert storage.uri_for("reports/q1.csv") == "sftp://loopback/reports/q1.csv" + + +def test_an_upload_replaces_the_file_and_leaves_nothing_else( + storage: SFTPStorage, served: Path +) -> None: + storage.write_bytes("dir/a.txt", b"first") + storage.write_bytes("dir/a.txt", b"second") + storage.move_from(storage, "dir/a.txt", "dir/b.txt") + assert [entry.name for entry in (served / "dir").iterdir()] == ["b.txt"] + assert (served / "dir" / "b.txt").read_bytes() == b"second" + with pytest.raises(StorageNotEmptyException): + storage.delete("dir") + storage.delete("dir", recursive=True) + assert list(served.iterdir()) == [] + + +def test_a_refusal_the_server_gives_no_reason_for_is_a_storage_error( + storage: SFTPStorage, served: Path +) -> None: + (served / "a.txt").write_bytes(b"x") + with pytest.raises(StorageException) as caught: + storage._mkdir("a.txt") + assert type(caught.value) is StorageException + assert type(caught.value.__cause__) is OSError + assert caught.value.__cause__.errno is None + + +def test_a_closed_session_is_transient(served: Path, host_key: paramiko.PKey) -> None: + with loopback(served, host_key) as client: + storage = SFTPStorage(client) + storage.write_bytes("a.txt", b"x") + client.require_sftp().close() + with pytest.raises(StorageTransientException) as caught: + storage.stat("a.txt") + assert isinstance(caught.value.__cause__, OSError) + with pytest.raises(StorageTransientException): + storage.write_bytes("b.txt", b"y") + assert sorted(entry.name for entry in served.iterdir()) == ["a.txt"] diff --git a/tests/test_storage_smb.py b/tests/test_storage_smb.py new file mode 100644 index 0000000..a9b9b92 --- /dev/null +++ b/tests/test_storage_smb.py @@ -0,0 +1,508 @@ +"""SMBStorage: the storage contract against an in-memory stand-in for ``smbclient``. + +``smbprotocol`` is an optional dependency, so the tests install a module named +``smbclient`` that keeps a share in memory. It answers the calls ``SMBClient`` +makes -- ``register_session``, ``stat``, ``scandir``, ``open_file``, ``remove``, +``rmdir``, ``makedirs``, ``rename`` and ``replace`` -- and fails the way +smbprotocol does: with an ``OSError`` subclass of its own that carries an errno +and an NTSTATUS code. The real ``SMBClient`` runs on top of it. +""" + +from __future__ import annotations + +import errno +import io +import os +import stat +import sys +from collections.abc import Iterator +from datetime import datetime, timedelta, timezone +from pathlib import Path +from types import ModuleType, SimpleNamespace, TracebackType +from typing import Any + +import pytest + +from automation_file.exceptions import ( + StorageException, + StoragePermissionException, + StorageTransientException, + StorageUnavailableException, + StorageURIException, +) +from automation_file.remote.smb.client import SMBClient +from automation_file.storage import File, StorageBackend, StorageResolver +from automation_file.storage.smb_storage import SMB_SCHEME, SMBStorage +from tests.storage_contract import StorageContract + +SERVER = "nas.example.com" +SHARE = "projects" +SHARE_ROOT = f"\\\\{SERVER}\\{SHARE}" +PORT = 4455 +STATUS_ACCESS_DENIED = 0xC0000022 +STATUS_UNSUCCESSFUL = 0xC0000001 + + +class FakeSMBOSError(OSError): + """Shaped like ``smbprotocol.exceptions.SMBOSError``: an errno and the NTSTATUS behind it.""" + + def __init__(self, code: int, path: str, ntstatus: int = STATUS_UNSUCCESSFUL) -> None: + super().__init__(code, os.strerror(code) if code else "Unknown NtStatus error", path) + self.ntstatus = ntstatus + + +class _Writer(io.BytesIO): + def __init__(self, share: FakeShare, path: str) -> None: + super().__init__() + self._share = share + self._path = path + + def __exit__( + self, + exc_type: type[BaseException] | None, + exc: BaseException | None, + tb: TracebackType | None, + ) -> None: + self._share.nodes[self._path] = self.getvalue() + self._share.modified[self._path] = self._share.clock() + super().__exit__(exc_type, exc, tb) + + +class _DirEntry: + def __init__(self, share: FakeShare, path: str) -> None: + self._share = share + self._path = path + self.name = path.rpartition("\\")[2] + + def is_dir(self) -> bool: + return self._share.nodes[self._path] is None + + def stat(self) -> SimpleNamespace: + return self._share.stat_result(self._path) + + +class FakeShare: + """One share, keyed by UNC path. ``None`` marks a directory, bytes a file.""" + + def __init__(self) -> None: + self.nodes: dict[str, bytes | None] = {SHARE_ROOT: None} + self.modified: dict[str, float] = {SHARE_ROOT: 0.0} + self.calls: list[str] = [] + self.ports: set[Any] = set() + self.fail_with: Exception | None = None + self.refuse_session_with: Exception | None = None + self._ticks = 0 + + # ------------------------------------------------------------------ helpers + + def clock(self) -> float: + self._ticks += 1 + return 1_791_000_000.0 + self._ticks + + def stat_result(self, path: str) -> SimpleNamespace: + data = self.nodes[path] + return SimpleNamespace( + st_mode=(stat.S_IFDIR | 0o755) if data is None else (stat.S_IFREG | 0o644), + st_size=0 if data is None else len(data), + st_mtime=self.modified[path], + ) + + def _begin(self, call: str, options: dict[str, Any]) -> None: + self.calls.append(call) + self.ports.add(options.get("port")) + if self.fail_with is not None: + raise self.fail_with + + def _existing(self, path: str) -> bytes | None: + """Return the node at ``path``; fail like the server for a missing one.""" + if path in self.nodes: + return self.nodes[path] + assert path.startswith(f"{SHARE_ROOT}\\"), f"request left the share: {path}" + parent = path.rpartition("\\")[0] + while parent not in self.nodes: + parent = parent.rpartition("\\")[0] + code = errno.ENOTDIR if self.nodes[parent] is not None else errno.ENOENT + raise FakeSMBOSError(code, path) + + def _children(self, path: str) -> list[str]: + below = f"{path}\\" + return sorted( + name for name in self.nodes if name.startswith(below) and "\\" not in name[len(below) :] + ) + + def _require_directory(self, path: str) -> None: + if self._existing(path) is not None: + raise FakeSMBOSError(errno.ENOTDIR, path) + + # ------------------------------------------------------------------ the smbclient surface + + def register_session(self, server: str, **options: Any) -> None: + self._begin("register_session", options) + assert server == SERVER + if self.refuse_session_with is not None: + raise self.refuse_session_with + + def delete_session(self, server: str, **options: Any) -> None: + self._begin("delete_session", options) + assert server == SERVER + + def stat(self, path: str, **options: Any) -> SimpleNamespace: + self._begin("stat", options) + self._existing(path) + return self.stat_result(path) + + def scandir(self, path: str, **options: Any) -> Iterator[_DirEntry]: + self._begin("scandir", options) + self._require_directory(path) + return iter([_DirEntry(self, name) for name in self._children(path)]) + + def open_file(self, path: str, mode: str = "r", **options: Any) -> io.BytesIO: + self._begin(f"open_file:{mode}", options) + if mode == "rb": + data = self._existing(path) + if data is None: + raise FakeSMBOSError(errno.EISDIR, path) + return io.BytesIO(data) + assert mode == "wb" + self._require_directory(path.rpartition("\\")[0]) + if self.nodes.get(path, b"") is None: + raise FakeSMBOSError(errno.EISDIR, path) + return _Writer(self, path) + + def remove(self, path: str, **options: Any) -> None: + self._begin("remove", options) + if self._existing(path) is None: + raise FakeSMBOSError(errno.EISDIR, path) + del self.nodes[path] + + def rmdir(self, path: str, **options: Any) -> None: + self._begin("rmdir", options) + self._require_directory(path) + if self._children(path): + raise FakeSMBOSError(errno.ENOTEMPTY, path) + del self.nodes[path] + + def makedirs(self, path: str, exist_ok: bool = False, **options: Any) -> None: + self._begin("makedirs", options) + if path in self.nodes: + if not exist_ok or self.nodes[path] is not None: + raise FakeSMBOSError(errno.EEXIST, path) + return + missing: list[str] = [] + current = path + while current not in self.nodes: + missing.append(current) + current = current.rpartition("\\")[0] + self._require_directory(current) + for directory in reversed(missing): + self.nodes[directory] = None + self.modified[directory] = self.clock() + + def _rename(self, source: str, target: str, *, replace: bool) -> None: + data = self._existing(source) + self._require_directory(target.rpartition("\\")[0]) + if target in self.nodes and (not replace or self.nodes[target] is None): + raise FakeSMBOSError(errno.EEXIST, target) + self.nodes[target] = data + self.modified[target] = self.modified[source] + del self.nodes[source] + + def rename(self, source: str, target: str, **options: Any) -> None: + self._begin("rename", options) + self._rename(source, target, replace=False) + + def replace(self, source: str, target: str, **options: Any) -> None: + self._begin("replace", options) + self._rename(source, target, replace=True) + + +class _ProtocolError(Exception): + """Stands in for ``smbprotocol.exceptions.SMBException``, the base of its own errors.""" + + +class _LogonFailure(_ProtocolError): + """Stands in for ``smbprotocol.exceptions.LogonFailure``.""" + + +@pytest.fixture +def share(monkeypatch: pytest.MonkeyPatch) -> FakeShare: + """Install an in-memory ``smbclient`` and the ``smbprotocol.exceptions`` it raises from.""" + fake = FakeShare() + module = ModuleType("smbclient") + for name in ( + "register_session", + "delete_session", + "stat", + "scandir", + "open_file", + "remove", + "rmdir", + "makedirs", + "rename", + "replace", + ): + setattr(module, name, getattr(fake, name)) + errors = ModuleType("smbprotocol.exceptions") + errors.SMBException = _ProtocolError # type: ignore[attr-defined] + errors.LogonFailure = _LogonFailure # type: ignore[attr-defined] + package = ModuleType("smbprotocol") + package.exceptions = errors # type: ignore[attr-defined] + monkeypatch.setitem(sys.modules, "smbclient", module) + monkeypatch.setitem(sys.modules, "smbprotocol", package) + monkeypatch.setitem(sys.modules, "smbprotocol.exceptions", errors) + return fake + + +@pytest.fixture +def client(share: FakeShare) -> SMBClient: # pylint: disable=unused-argument + return SMBClient(SERVER, SHARE, port=PORT) + + +@pytest.fixture +def storage(client: SMBClient) -> SMBStorage: + return SMBStorage(client) + + +class TestSMBStorageContract(StorageContract): + @pytest.fixture + def backend(self, client: SMBClient) -> StorageBackend: + return SMBStorage(client) + + +class TestRootedSMBStorageContract(StorageContract): + @pytest.fixture + def backend(self, client: SMBClient, share: FakeShare) -> StorageBackend: + share.makedirs(f"{SHARE_ROOT}\\team\\a", port=PORT) + share.makedirs(f"{SHARE_ROOT}\\other-team", port=PORT) + share.nodes[f"{SHARE_ROOT}\\other-team\\keep.txt"] = b"keep" + share.modified[f"{SHARE_ROOT}\\other-team\\keep.txt"] = share.clock() + return SMBStorage(client, root="team/a") + + +def test_stat_reports_size_and_modification_time(storage: SMBStorage, share: FakeShare) -> None: + info = storage.write_bytes("reports/q1.json", b"{}") + path = f"{SHARE_ROOT}\\reports\\q1.json" + assert info.path == "reports/q1.json" + assert info.size == 2 + assert info.modified_at == datetime.fromtimestamp(share.modified[path], tz=timezone.utc) + assert info.modified_at.utcoffset() == timedelta(0) + assert (info.etag, info.version, info.content_type) == (None, None, None) + folder = storage.stat("reports") + assert (folder.is_dir, folder.size) == (True, None) + assert folder.modified_at is not None + + +def test_a_time_before_1970_is_reported(storage: SMBStorage, share: FakeShare) -> None: + storage.write_bytes("old.txt", b"x") + # A server that keeps no time sends the FILETIME epoch, the year 1601. + share.modified[f"{SHARE_ROOT}\\old.txt"] = -11_644_473_600.0 + assert storage.stat("old.txt").modified_at == datetime(1601, 1, 1, tzinfo=timezone.utc) + share.modified[f"{SHARE_ROOT}\\old.txt"] = 1e20 + assert storage.stat("old.txt").modified_at is None + + +def test_listing_carries_sizes_and_times(storage: SMBStorage) -> None: + for path in ("dir/b.txt", "dir/a.txt", "dir/sub/c.txt"): + storage.write_bytes(path, b"xy") + listing = storage.list_dir("dir") + assert [(info.path, info.is_dir, info.size) for info in listing] == [ + ("dir/a.txt", False, 2), + ("dir/b.txt", False, 2), + ("dir/sub", True, None), + ] + assert all(info.modified_at is not None for info in listing) + + +def test_every_call_goes_to_the_port_of_the_client(storage: SMBStorage, share: FakeShare) -> None: + storage.write_bytes("dir/a.txt", b"x") + storage.move_from(storage, "dir/a.txt", "dir/b.txt") + storage.list_dir("dir") + storage.read_bytes("dir/b.txt") + storage.delete("dir", recursive=True) + assert share.ports == {PORT} + assert {"stat", "scandir", "open_file:wb", "open_file:rb", "remove", "rmdir"} <= set( + share.calls + ) + + +def test_a_move_within_one_client_is_a_rename(storage: SMBStorage, share: FakeShare) -> None: + storage.write_bytes("a.txt", b"payload") + storage.write_bytes("moved/b.txt", b"old") + share.calls.clear() + storage.move_from(storage, "a.txt", "moved/b.txt") + assert share.nodes[f"{SHARE_ROOT}\\moved\\b.txt"] == b"payload" + assert f"{SHARE_ROOT}\\a.txt" not in share.nodes + assert share.calls.count("replace") == 1 + assert not {"open_file:rb", "open_file:wb", "remove"} & set(share.calls) + + +def test_a_copy_goes_through_a_staging_file(storage: SMBStorage, share: FakeShare) -> None: + storage.write_bytes("a.txt", b"payload") + share.calls.clear() + storage.copy_from(storage, "a.txt", "copies/b.txt") + assert share.nodes[f"{SHARE_ROOT}\\copies\\b.txt"] == b"payload" + assert share.nodes[f"{SHARE_ROOT}\\a.txt"] == b"payload" + assert {"open_file:rb", "open_file:wb"} <= set(share.calls) + + +def test_a_move_between_two_clients_copies_then_deletes( + storage: SMBStorage, share: FakeShare +) -> None: + other = SMBStorage(SMBClient(SERVER, SHARE, port=PORT)) + storage.write_bytes("a.txt", b"payload") + share.calls.clear() + other.move_from(storage, "a.txt", "b.txt") + assert share.nodes[f"{SHARE_ROOT}\\b.txt"] == b"payload" + assert f"{SHARE_ROOT}\\a.txt" not in share.nodes + assert "replace" not in share.calls + + +def test_a_backslash_is_a_separator(storage: SMBStorage, share: FakeShare) -> None: + storage.write_bytes("dir\\sub\\a.txt", b"x") + assert share.nodes[f"{SHARE_ROOT}\\dir\\sub\\a.txt"] == b"x" + assert storage.stat("dir/sub\\a.txt").path == "dir/sub/a.txt" + assert [info.path for info in storage.list_dir("dir\\sub")] == ["dir/sub/a.txt"] + + +@pytest.mark.parametrize("path", ["a\\..\\..\\secret.txt", "..\\secret.txt", "dir/..\\..\\x"]) +def test_a_backslash_cannot_smuggle_a_parent_segment( + client: SMBClient, share: FakeShare, path: str +) -> None: + share.nodes[f"{SHARE_ROOT}\\secret.txt"] = b"secret" + share.modified[f"{SHARE_ROOT}\\secret.txt"] = share.clock() + rooted = SMBStorage(client, root="team") + with pytest.raises(StorageURIException): + rooted.exists(path) + with pytest.raises(StorageURIException): + rooted.write_bytes(path, b"overwritten") + with pytest.raises(StorageURIException): + SMBStorage(client, root="team\\..\\..") + assert share.nodes[f"{SHARE_ROOT}\\secret.txt"] == b"secret" + assert share.calls == [] + + +def _denied_without_errno() -> FakeSMBOSError: + return FakeSMBOSError(0, "a.txt", STATUS_ACCESS_DENIED) + + +@pytest.mark.parametrize( + "error,expected", + [ + (FakeSMBOSError(errno.EACCES, "a.txt"), StoragePermissionException), + (_denied_without_errno(), StoragePermissionException), + (PermissionError(errno.EPERM, "not permitted"), StoragePermissionException), + (ConnectionResetError(errno.ECONNRESET, "reset"), StorageTransientException), + (TimeoutError("timed out"), StorageTransientException), + (FakeSMBOSError(errno.EIO, "a.txt"), StorageException), + (OSError("boom"), StorageException), + (ValueError("Failed to connect to 'nas.example.com:445'"), StorageException), + (_ProtocolError("Socket connection has been closed"), StorageException), + ], +) +def test_client_errors_become_storage_errors( + storage: SMBStorage, share: FakeShare, error: Exception, expected: type[Exception] +) -> None: + storage.write_bytes("a.txt", b"x") + share.fail_with = error + with pytest.raises(expected) as caught: + storage.read_bytes("a.txt") + assert type(caught.value) is expected + chain = [caught.value.__cause__, caught.value.__cause__.__cause__] + assert error in chain + + +def _refused_connection() -> ValueError: + error = ValueError("Failed to connect to 'nas.example.com:445'") + error.__cause__ = ConnectionRefusedError(errno.ECONNREFUSED, "refused") + return error + + +@pytest.mark.parametrize( + "refusal,expected", + [ + (_LogonFailure("bad credentials"), StoragePermissionException), + (_refused_connection(), StorageTransientException), + (_ProtocolError("negotiation failed"), StorageException), + ], +) +def test_a_session_that_cannot_be_opened_is_reported( + storage: SMBStorage, share: FakeShare, refusal: Exception, expected: type[Exception] +) -> None: + share.refuse_session_with = refusal + with pytest.raises(expected) as caught: + storage.exists("a.txt") + assert type(caught.value) is expected + assert caught.value.__cause__.__cause__ is refusal + + +def test_a_missing_path_is_told_by_its_errno(storage: SMBStorage, share: FakeShare) -> None: + storage.write_bytes("a.txt", b"x") + assert storage.exists("nope.txt") is False + assert storage.exists("nope/deeper.txt") is False + assert storage.exists("a.txt/child.txt") is False + assert not isinstance(FakeSMBOSError(errno.ENOENT, "x"), FileNotFoundError) + share.fail_with = FileNotFoundError(errno.ENOENT, "gone") + assert storage.exists("a.txt") is False + + +def test_smbprotocol_must_be_installed(monkeypatch: pytest.MonkeyPatch) -> None: + monkeypatch.setitem(sys.modules, "smbclient", None) + monkeypatch.setitem(sys.modules, "smbprotocol", None) + storage = SMBStorage(SMBClient(SERVER, SHARE)) + with pytest.raises(StorageUnavailableException, match="smbprotocol is not installed"): + storage.exists("a.txt") + + +def test_a_mounted_backend_serves_its_uris( + storage: SMBStorage, share: FakeShare, tmp_path: Path +) -> None: + resolver = StorageResolver() + resolver.mount(f"smb://{SERVER}/{SHARE}", storage) + plan = File(f"smb://{SERVER}/{SHARE}/2026/plan.txt", resolver=resolver) + plan.write(b"plan") + assert share.nodes[f"{SHARE_ROOT}\\2026\\plan.txt"] == b"plan" + assert resolver.resolve(f"smb://{SERVER.upper()}/{SHARE}/2026/plan.txt") == ( + storage, + "2026/plan.txt", + ) + assert resolver.capabilities(f"smb://{SERVER}/{SHARE}").directories is True + plan.copy_to(tmp_path / "plan.txt") + assert (tmp_path / "plan.txt").read_bytes() == b"plan" + plan.move_to(f"smb://{SERVER}/{SHARE}/archive/plan.txt") + assert f"{SHARE_ROOT}\\2026\\plan.txt" not in share.nodes + with pytest.raises(StorageURIException, match="no mount"): + resolver.resolve(f"smb://{SERVER}/another-share/a.txt") + + +def test_uri_equality_and_repr(client: SMBClient) -> None: + assert SMBStorage(client).uri_for("") == f"smb://{SERVER}/{SHARE}" + assert SMBStorage(client).uri_for("a\\b.txt") == f"smb://{SERVER}/{SHARE}/a/b.txt" + rooted = SMBStorage(client, root="\\team\\\\a/") + assert rooted.root == "team/a" + assert rooted.uri_for("b.txt") == f"smb://{SERVER}/{SHARE}/team/a/b.txt" + assert repr(rooted) == f"SMBStorage('smb://{SERVER}/{SHARE}/team/a')" + assert SMBStorage(client) == SMBStorage(client) + assert SMBStorage(client) != rooted + assert SMBStorage(client) != SMBStorage(SMBClient(SERVER, SHARE)) + assert len({SMBStorage(client), SMBStorage(client)}) == 1 + assert SMBStorage.scheme == SMB_SCHEME == "smb" + capabilities = SMBStorage.capabilities + assert (capabilities.directories, capabilities.modified_at) == (True, True) + assert (capabilities.etag, capabilities.version, capabilities.content_type) == ( + False, + False, + False, + ) + + +def test_two_roots_of_one_share_do_not_lose_a_file_to_itself(client: SMBClient) -> None: + whole = SMBStorage(client) + inner = SMBStorage(client, root="team/a") + whole.mkdir("team/a") + inner.write_bytes("docs/a.txt", b"payload") + for operation in (whole.move_from, whole.copy_from): + with pytest.raises(StorageException, match="same file"): + operation(inner, "docs/a.txt", "team/a/docs/a.txt") + with pytest.raises(StorageException, match="same file"): + inner.move_from(whole, "team/a/docs/a.txt", "docs/a.txt") + assert whole.read_bytes("team/a/docs/a.txt") == b"payload" diff --git a/tests/test_storage_timestamps.py b/tests/test_storage_timestamps.py new file mode 100644 index 0000000..f69f30c --- /dev/null +++ b/tests/test_storage_timestamps.py @@ -0,0 +1,58 @@ +"""parse_rfc3339: the timestamps of the cloud APIs as aware UTC datetimes.""" + +from __future__ import annotations + +from datetime import datetime, timedelta, timezone + +import pytest + +from automation_file.storage.timestamps import parse_rfc3339 + + +@pytest.mark.parametrize( + "text,expected", + [ + ("2026-10-08T02:30:00Z", datetime(2026, 10, 8, 2, 30, tzinfo=timezone.utc)), + ("2026-10-08T02:30:00.123Z", datetime(2026, 10, 8, 2, 30, 0, 123000, tzinfo=timezone.utc)), + # Seven digits, as Microsoft Graph sends: cut to microseconds, not rounded. + ( + "2026-10-08T02:30:00.1234567Z", + datetime(2026, 10, 8, 2, 30, 0, 123456, tzinfo=timezone.utc), + ), + ("2026-10-08T02:30:00.5z", datetime(2026, 10, 8, 2, 30, 0, 500000, tzinfo=timezone.utc)), + ("2026-10-08T10:30:00+08:00", datetime(2026, 10, 8, 2, 30, tzinfo=timezone.utc)), + ( + "2026-10-07T21:00:00.25-05:30", + datetime(2026, 10, 8, 2, 30, 0, 250000, tzinfo=timezone.utc), + ), + ("2026-10-08t02:30:00+00:00", datetime(2026, 10, 8, 2, 30, tzinfo=timezone.utc)), + (" 2026-10-08 02:30:00Z ", datetime(2026, 10, 8, 2, 30, tzinfo=timezone.utc)), + ("0001-01-01T00:00:00Z", datetime(1, 1, 1, tzinfo=timezone.utc)), + ], +) +def test_a_timestamp_becomes_an_aware_utc_datetime(text: str, expected: datetime) -> None: + parsed = parse_rfc3339(text) + assert parsed == expected + assert parsed is not None + assert parsed.utcoffset() == timedelta(0) + + +@pytest.mark.parametrize( + "value", + [ + None, + 1759890600, + "", + "yesterday", + "2026-10-08", + "2026-10-08T02:30:00", + "2026-10-08T02:30Z", + "2026-13-08T02:30:00Z", + "2026-10-08T02:30:60Z", + "2026-10-08T02:30:00+99:00", + "0001-01-01T00:00:00+05:00", + "2026-10-08T02:30:00Z trailing", + ], +) +def test_anything_else_is_none(value: object) -> None: + assert parse_rfc3339(value) is None diff --git a/tests/test_storage_webdav.py b/tests/test_storage_webdav.py new file mode 100644 index 0000000..5869be1 --- /dev/null +++ b/tests/test_storage_webdav.py @@ -0,0 +1,608 @@ +"""WebDAVStorage: the storage contract against an in-memory WebDAV server. + +The server stands in for ``requests.Session``, so the real ``WebDAVClient`` runs: +it builds the URLs and headers, and parses the ``207 Multi-Status`` documents the +server writes the way Apache and Nextcloud write them. No request leaves the +process. +""" + +from __future__ import annotations + +import hashlib +import http.client +import inspect +import mimetypes +import secrets +from collections.abc import Iterator +from dataclasses import dataclass +from datetime import datetime, timedelta, timezone +from email.utils import format_datetime +from pathlib import Path +from typing import Any +from urllib.parse import quote, unquote, urlsplit +from xml.sax.saxutils import escape + +import pytest +import requests + +from automation_file.exceptions import ( + StorageException, + StorageNotFoundException, + StoragePermissionException, + StorageTransientException, + StorageURIException, +) +from automation_file.remote.webdav.client import WebDAVClient +from automation_file.storage import File, StorageBackend, StorageResolver +from automation_file.storage.webdav_storage import WEBDAV_SCHEME, WebDAVStorage +from tests.storage_contract import StorageContract + +HOST = "files.example.com" +DAV_ROOT = "/remote.php/dav" +BASE_URL = f"https://{HOST}{DAV_ROOT}" +CHUNK = 1 << 16 +OK = "HTTP/1.1 200 OK" +NOT_FOUND = "HTTP/1.1 404 Not Found" +MULTISTATUS = ( + '{}' +) + + +def _dav_path(url: str) -> str: + """Return the decoded path a URL names on the fake server; it must be below the DAV root.""" + parts = urlsplit(url) + assert (parts.scheme, parts.hostname) == ("https", HOST), f"request left the server: {url}" + assert parts.path.startswith(f"{DAV_ROOT}/"), f"request left the DAV root: {url}" + return unquote(parts.path).rstrip("/") + + +@dataclass +class _Resource: + data: bytes + modified: datetime + + @property + def etag(self) -> str: + return f'"{hashlib.md5(self.data, usedforsecurity=False).hexdigest()}"' + + +class _Response: + def __init__( + self, + status: int, + *, + text: str = "", + body: bytes = b"", + cut_off: Exception | None = None, + ) -> None: + self.status_code = status + self.reason = http.client.responses.get(status, "Unknown") + self.text = text + self.closed = False + self._body = body + self._cut_off = cut_off + + def iter_content(self, chunk_size: int = CHUNK) -> Iterator[bytes]: + for start in range(0, len(self._body), chunk_size): + yield self._body[start : start + chunk_size] + if self._cut_off is not None: + raise self._cut_off + + def close(self) -> None: + self.closed = True + + +def _propstat(properties: str, status: str) -> str: + return f"{properties}{status}" + + +def _response(path: str, resource: _Resource | None, modified: datetime) -> str: + """Describe one resource the way Apache does: a second propstat names what it lacks.""" + stamp = f"{format_datetime(modified, usegmt=True)}" + if resource is None: + # Like Apache, a ";" in a name is left as it is and a collection ends with "/". + href = quote(path, safe="/;") + "/" + found = f"{stamp}" + absent = _propstat("", NOT_FOUND) + else: + href = quote(path, safe="/;") + content_type = mimetypes.guess_type(path)[0] or "application/octet-stream" + found = ( + f"{len(resource.data)}" + f"{stamp}{escape(resource.etag)}" + f"{content_type}" + ) + absent = "" + return f"{escape(href)}{_propstat(found, OK)}{absent}" + + +class FakeDavServer: + """A WebDAV server in memory, behind the one ``requests.Session`` call the client makes.""" + + def __init__(self) -> None: + self.files: dict[str, _Resource] = {} + self.collections: dict[str, datetime] = {DAV_ROOT: datetime.now(timezone.utc)} + self.requests: list[tuple[str, str]] = [] + self.headers: list[dict[str, str]] = [] + self.auth: list[Any] = [] + self.fail_with: Exception | None = None + self.status_for: dict[str, int] = {} + self.cut_downloads_with: Exception | None = None + + # ------------------------------------------------------------------ requests.Session + + def request(self, method: str, url: str, **options: Any) -> _Response: + path = _dav_path(url) + assert options["verify"] is True + assert options["timeout"] > 0 + self.requests.append((method, url)) + self.headers.append(dict(options.get("headers") or {})) + self.auth.append(options.get("auth")) + if self.fail_with is not None: + raise self.fail_with + if method in self.status_for: + return _Response(self.status_for[method]) + handler = getattr(self, f"_{method.lower()}") + return handler(path, options) + + def close(self) -> None: + """Nothing to close.""" + + # ------------------------------------------------------------------ the tree + + def methods(self) -> list[str]: + return [method for method, _ in self.requests] + + def _exists(self, path: str) -> bool: + return path in self.files or path in self.collections + + def _parent_exists(self, path: str) -> bool: + return path.rpartition("/")[0] in self.collections + + def _members(self, path: str) -> list[str]: + below = f"{path}/" + names = (name for name in (*self.files, *self.collections) if name.startswith(below)) + return sorted(name for name in names if "/" not in name[len(below) :]) + + def _remove(self, path: str) -> None: + below = f"{path}/" + for name in [name for name in self.files if name == path or name.startswith(below)]: + del self.files[name] + for name in [name for name in self.collections if name == path or name.startswith(below)]: + del self.collections[name] + + def _describe(self, path: str) -> str: + if path in self.files: + return _response(path, self.files[path], self.files[path].modified) + return _response(path, None, self.collections[path]) + + # ------------------------------------------------------------------ the methods + + def _propfind(self, path: str, options: dict[str, Any]) -> _Response: + assert " _Response: + return _Response(200 if self._exists(path) else 404) + + def _put(self, path: str, options: dict[str, Any]) -> _Response: + if path in self.collections: + return _Response(405) + if not self._parent_exists(path): + return _Response(409) + body = options["data"] + data = body if isinstance(body, bytes) else body.read() + created = path not in self.files + self.files[path] = _Resource(data, datetime.now(timezone.utc).replace(microsecond=0)) + return _Response(201 if created else 204) + + def _get(self, path: str, options: dict[str, Any]) -> _Response: + assert options["stream"] is True + if path not in self.files: + return _Response(404 if path not in self.collections else 405) + return _Response(200, body=self.files[path].data, cut_off=self.cut_downloads_with) + + def _delete(self, path: str, _options: dict[str, Any]) -> _Response: + if not self._exists(path): + return _Response(404) + self._remove(path) + return _Response(204) + + def _mkcol(self, path: str, _options: dict[str, Any]) -> _Response: + if self._exists(path): + return _Response(405) + if not self._parent_exists(path): + return _Response(409) + self.collections[path] = datetime.now(timezone.utc).replace(microsecond=0) + return _Response(201) + + def _relocate(self, path: str, options: dict[str, Any], *, move: bool) -> _Response: + target = _dav_path(options["headers"]["Destination"]) + if path not in self.files: + return _Response(404) + if not self._parent_exists(target): + return _Response(409) + replaced = self._exists(target) + if replaced and options["headers"]["Overwrite"] != "T": + return _Response(412) + self._remove(target) + self.files[target] = _Resource(self.files[path].data, self.files[path].modified) + if move: + del self.files[path] + return _Response(204 if replaced else 201) + + def _copy(self, path: str, options: dict[str, Any]) -> _Response: + return self._relocate(path, options, move=False) + + def _move(self, path: str, options: dict[str, Any]) -> _Response: + return self._relocate(path, options, move=True) + + +@pytest.fixture +def server(monkeypatch: pytest.MonkeyPatch) -> FakeDavServer: + """Put the in-memory server behind every ``WebDAVClient`` and keep DNS out of the test.""" + fake = FakeDavServer() + monkeypatch.setattr("automation_file.remote.webdav.client.requests.Session", lambda: fake) + monkeypatch.setattr( + "automation_file.remote.webdav.client.validate_http_url", lambda url, **_options: url + ) + return fake + + +@pytest.fixture +def client(server: FakeDavServer) -> WebDAVClient: # pylint: disable=unused-argument + return WebDAVClient(BASE_URL) + + +@pytest.fixture +def storage(client: WebDAVClient) -> WebDAVStorage: + return WebDAVStorage(client) + + +class TestWebDAVStorageContract(StorageContract): + @pytest.fixture + def backend(self, client: WebDAVClient) -> StorageBackend: + return WebDAVStorage(client) + + +class TestRootedWebDAVStorageContract(StorageContract): + @pytest.fixture + def backend(self, client: WebDAVClient, server: FakeDavServer) -> StorageBackend: + moment = datetime.now(timezone.utc) + for collection in ("team", "team/a b", "other-team"): + server.collections[f"{DAV_ROOT}/{collection}"] = moment + server.files[f"{DAV_ROOT}/other-team/keep.txt"] = _Resource(b"keep", moment) + return WebDAVStorage(client, root="team/a b") + + +def test_stat_reports_what_propfind_returns(storage: WebDAVStorage, server: FakeDavServer) -> None: + info = storage.write_bytes("reports/q1.json", b"{}") + stored = server.files[f"{DAV_ROOT}/reports/q1.json"] + assert info.path == "reports/q1.json" + assert info.size == 2 + assert info.modified_at == stored.modified + assert info.modified_at.utcoffset() == timedelta(0) + assert info.etag == hashlib.md5(b"{}", usedforsecurity=False).hexdigest() + assert info.content_type == "application/json" + assert info.version is None + folder = storage.stat("reports") + assert (folder.is_dir, folder.size, folder.etag, folder.content_type) == ( + True, + None, + None, + None, + ) + assert folder.modified_at is not None + + +def test_stat_asks_for_the_resource_only(storage: WebDAVStorage, server: FakeDavServer) -> None: + storage.write_bytes("dir/a.txt", b"x") + server.requests.clear() + server.headers.clear() + assert storage.stat("dir").is_dir is True + assert server.requests == [("PROPFIND", f"{BASE_URL}/dir")] + assert server.headers[0]["Depth"] == "0" + + +def test_listing_leaves_out_the_collection_itself( + storage: WebDAVStorage, server: FakeDavServer +) -> None: + for path in ("dir/b.txt", "dir/a b;c.txt", "dir/sub/d.txt", "dir/報告.txt"): + storage.write_bytes(path, b"xy") + server.requests.clear() + server.headers.clear() + listing = storage.list_dir("dir") + assert [(info.path, info.is_dir, info.size) for info in listing] == [ + ("dir/a b;c.txt", False, 2), + ("dir/b.txt", False, 2), + ("dir/sub", True, None), + ("dir/報告.txt", False, 2), + ] + assert all(info.etag and info.modified_at for info in listing if not info.is_dir) + # One PROPFIND for the stat of the directory, one for its members. + assert server.requests[-1] == ("PROPFIND", f"{BASE_URL}/dir/") + assert server.headers[-1]["Depth"] == "1" + assert storage.list_dir("dir/sub")[0].path == "dir/sub/d.txt" + + +def test_paths_are_percent_encoded_once(storage: WebDAVStorage, server: FakeDavServer) -> None: + storage.write_bytes("100% done/a#b?.txt", b"x") + assert f"{DAV_ROOT}/100% done/a#b?.txt" in server.files + assert ("PUT", f"{BASE_URL}/100%25%20done/a%23b%3F.txt") in server.requests + assert storage.read_bytes("100% done/a#b?.txt") == b"x" + + +def test_a_path_cannot_point_the_client_at_another_host( + storage: WebDAVStorage, server: FakeDavServer +) -> None: + resolver = StorageResolver() + resolver.mount(f"webdav://{HOST}", storage) + smuggled = File(f"webdav://{HOST}/https://internal.example.com/secret", resolver=resolver) + assert smuggled.exists() is False + assert server.requests == [ + ("PROPFIND", f"{BASE_URL}/https%3A/internal.example.com/secret"), + ] + + +def test_copy_and_move_within_one_client_are_done_by_the_server( + storage: WebDAVStorage, server: FakeDavServer +) -> None: + storage.write_bytes("a.txt", b"payload") + storage.mkdir("copies and moves") + server.requests.clear() + server.headers.clear() + storage.copy_from(storage, "a.txt", "copies and moves/b ü.txt") + assert server.files[f"{DAV_ROOT}/copies and moves/b ü.txt"].data == b"payload" + copy = server.headers[server.methods().index("COPY")] + assert copy["Destination"] == f"{BASE_URL}/copies%20and%20moves/b%20%C3%BC.txt" + assert copy["Overwrite"] == "T" + storage.move_from(storage, "a.txt", "copies and moves/c.txt") + assert f"{DAV_ROOT}/a.txt" not in server.files + assert server.files[f"{DAV_ROOT}/copies and moves/c.txt"].data == b"payload" + assert {"COPY", "MOVE"} <= set(server.methods()) + assert not {"GET", "PUT"} & set(server.methods()) + + +@pytest.mark.parametrize("status", [405, 501]) +def test_a_server_without_copy_and_move_gets_a_staged_transfer( + storage: WebDAVStorage, server: FakeDavServer, status: int +) -> None: + storage.write_bytes("a.txt", b"payload") + server.status_for = {"COPY": status, "MOVE": status} + storage.copy_from(storage, "a.txt", "b.txt") + assert server.files[f"{DAV_ROOT}/b.txt"].data == b"payload" + storage.move_from(storage, "a.txt", "c.txt") + assert server.files[f"{DAV_ROOT}/c.txt"].data == b"payload" + assert f"{DAV_ROOT}/a.txt" not in server.files + assert {"COPY", "MOVE", "GET", "PUT"} <= set(server.methods()) + + +def test_a_failed_copy_is_reported(storage: WebDAVStorage, server: FakeDavServer) -> None: + storage.write_bytes("a.txt", b"payload") + server.status_for = {"COPY": 507} + with pytest.raises(StorageTransientException): + storage.copy_from(storage, "a.txt", "b.txt") + server.status_for = {"COPY": 207} + with pytest.raises(StorageException, match="207"): + storage.copy_from(storage, "a.txt", "b.txt") + assert f"{DAV_ROOT}/b.txt" not in server.files + + +def test_copy_between_two_clients_goes_through_a_staging_file( + storage: WebDAVStorage, server: FakeDavServer +) -> None: + other = WebDAVStorage(WebDAVClient(BASE_URL)) + storage.write_bytes("a.txt", b"payload") + server.requests.clear() + other.copy_from(storage, "a.txt", "b.txt") + assert server.files[f"{DAV_ROOT}/b.txt"].data == b"payload" + assert "COPY" not in server.methods() + + +def test_deleting_a_directory_is_one_request(storage: WebDAVStorage, server: FakeDavServer) -> None: + for path in ("dir/a.txt", "dir/sub/b.txt", "dir/sub/deeper/c.txt"): + storage.write_bytes(path, b"x") + server.requests.clear() + storage.delete("dir", recursive=True) + assert server.methods().count("DELETE") == 1 + assert ("DELETE", f"{BASE_URL}/dir/") in server.requests + assert server.files == {} + assert list(server.collections) == [DAV_ROOT] + + +def test_a_delete_the_server_could_not_finish_is_an_error( + storage: WebDAVStorage, server: FakeDavServer +) -> None: + storage.write_bytes("dir/a.txt", b"x") + server.status_for = {"DELETE": 207} + with pytest.raises(StorageException, match="207") as caught: + storage.delete("dir", recursive=True) + assert type(caught.value) is StorageException + + +def test_mkdir_accepts_a_collection_that_appeared_meanwhile( + storage: WebDAVStorage, server: FakeDavServer +) -> None: + server.collections[f"{DAV_ROOT}/dir"] = datetime.now(timezone.utc) + storage._mkdir("dir") + server.files[f"{DAV_ROOT}/file"] = _Resource(b"x", datetime.now(timezone.utc)) + with pytest.raises(StorageException) as caught: + storage._mkdir("file") + assert caught.value.__cause__.status_code == 405 + + +@pytest.mark.parametrize( + "status,expected", + [ + (401, StoragePermissionException), + (403, StoragePermissionException), + (404, StorageNotFoundException), + (408, StorageTransientException), + (429, StorageTransientException), + (500, StorageTransientException), + (503, StorageTransientException), + (400, StorageException), + (423, StorageException), + ], +) +def test_http_statuses_become_storage_errors( + storage: WebDAVStorage, + server: FakeDavServer, + tmp_path: Path, + status: int, + expected: type[Exception], +) -> None: + storage.write_bytes("dir/a.txt", b"x") + server.status_for = {"PROPFIND": status, "GET": status, "PUT": status, "DELETE": status} + with pytest.raises(expected) as caught: + storage._list_dir("dir") + assert type(caught.value) is expected + assert caught.value.__cause__.status_code == status + with pytest.raises(expected): + storage._delete_file("dir/a.txt") + with pytest.raises(expected): + storage._download("dir/a.txt", tmp_path / "never-written.bin") + assert not (tmp_path / "never-written.bin").exists() + + +@pytest.mark.parametrize( + "error,expected", + [ + (requests.ConnectionError("connection refused"), StorageTransientException), + (requests.Timeout("read timed out"), StorageTransientException), + (requests.exceptions.ConnectTimeout("connect timed out"), StorageTransientException), + (requests.TooManyRedirects("redirect loop"), StorageException), + ], +) +def test_transport_errors_become_storage_errors( + storage: WebDAVStorage, server: FakeDavServer, error: Exception, expected: type[Exception] +) -> None: + server.fail_with = error + with pytest.raises(expected) as caught: + storage.stat("a.txt") + assert type(caught.value) is expected + assert caught.value.__cause__.__cause__ is error + assert BASE_URL not in str(caught.value) + + +def test_a_download_cut_off_midway_is_transient_and_leaves_nothing( + storage: WebDAVStorage, server: FakeDavServer, tmp_path: Path +) -> None: + storage.write_bytes("a.bin", b"x" * (3 * CHUNK)) + server.cut_downloads_with = requests.exceptions.ChunkedEncodingError("connection broken") + with pytest.raises(StorageTransientException) as caught: + storage.download("a.bin", tmp_path / "out" / "a.bin") + assert caught.value.__cause__ is server.cut_downloads_with + assert list((tmp_path / "out").iterdir()) == [] + + +def test_a_path_ending_in_white_space_is_refused( + storage: WebDAVStorage, server: FakeDavServer +) -> None: + for path in ("a.txt ", "dir/a.txt\t", "dir /"): + with pytest.raises(StorageURIException, match="white space"): + storage.exists(path) + with pytest.raises(StorageURIException): + WebDAVStorage(storage._client, root="team ") + assert server.requests == [] + storage.write_bytes(" leading and inner spaces/ a.txt", b"x") + assert list(server.files) == [f"{DAV_ROOT}/ leading and inner spaces/ a.txt"] + assert storage.read_bytes(" leading and inner spaces/ a.txt") == b"x" + + +def test_a_directory_whose_name_ends_in_white_space_can_still_be_listed_and_deleted( + storage: WebDAVStorage, server: FakeDavServer +) -> None: + moment = datetime.now(timezone.utc) + server.collections[f"{DAV_ROOT}/dir"] = moment + server.collections[f"{DAV_ROOT}/dir/odd "] = moment + server.files[f"{DAV_ROOT}/dir/odd /a.txt"] = _Resource(b"x", moment) + assert [info.path for info in storage.list_dir("dir", recursive=True)] == [ + "dir/odd ", + "dir/odd /a.txt", + ] + storage.delete("dir", recursive=True) + assert server.files == {} + + +def test_credentials_go_to_the_server_and_not_into_uris(server: FakeDavServer) -> None: + password = secrets.token_hex(8) + client = WebDAVClient(f"https://user:{password}@{HOST}{DAV_ROOT}/", "user", password) + storage = WebDAVStorage(client, root="team") + assert storage.exists("a.txt") is False + assert server.auth == [("user", password)] + assert storage.uri_for("a.txt") == f"webdav://{HOST}{DAV_ROOT}/team/a.txt" + assert password not in repr(storage) + server.fail_with = requests.ConnectionError(f"refused: {password}") + with pytest.raises(StorageTransientException) as caught: + storage.exists("a.txt") + assert password not in str(caught.value) + + +def test_a_mounted_backend_serves_its_uris( + storage: WebDAVStorage, server: FakeDavServer, tmp_path: Path +) -> None: + resolver = StorageResolver() + resolver.mount(f"webdav://{HOST}", storage) + report = File(f"webdav://{HOST}/reports/q1.csv", resolver=resolver) + report.write(b"a,b\n") + assert server.files[f"{DAV_ROOT}/reports/q1.csv"].data == b"a,b\n" + assert resolver.resolve(f"webdav://{HOST}/reports/q1.csv") == (storage, "reports/q1.csv") + assert resolver.capabilities(f"webdav://{HOST}").content_type is True + report.copy_to(tmp_path / "q1.csv") + assert (tmp_path / "q1.csv").read_bytes() == b"a,b\n" + report.move_to(f"webdav://{HOST}/archive/q1.csv") + assert f"{DAV_ROOT}/reports/q1.csv" not in server.files + assert "MOVE" in server.methods() + with pytest.raises(StorageURIException, match="no mount"): + resolver.resolve("webdav://another-host.example.com/a.txt") + + +def test_uri_equality_and_repr(client: WebDAVClient) -> None: + assert WebDAVStorage(client).uri_for("") == f"webdav://{HOST}{DAV_ROOT}" + assert WebDAVStorage(client).uri_for("a/b.txt") == f"webdav://{HOST}{DAV_ROOT}/a/b.txt" + rooted = WebDAVStorage(client, root="/team//a/") + assert rooted.root == "team/a" + assert rooted.uri_for("b.txt") == f"webdav://{HOST}{DAV_ROOT}/team/a/b.txt" + assert repr(rooted) == f"WebDAVStorage('webdav://{HOST}{DAV_ROOT}/team/a')" + assert WebDAVStorage(client) == WebDAVStorage(client) + assert WebDAVStorage(client) != rooted + assert WebDAVStorage(client) != WebDAVStorage(WebDAVClient(BASE_URL)) + assert len({WebDAVStorage(client), WebDAVStorage(client)}) == 1 + assert WebDAVStorage.scheme == WEBDAV_SCHEME == "webdav" + capabilities = WebDAVStorage.capabilities + assert (capabilities.directories, capabilities.etag, capabilities.content_type) == ( + True, + True, + True, + ) + assert (capabilities.version, capabilities.metadata) == (False, False) + + +def test_requests_takes_the_arguments_the_client_passes() -> None: + parameters = inspect.signature(requests.Session.request).parameters + assert {"method", "url", "data", "headers", "auth", "timeout", "verify", "stream"} <= set( + parameters + ) + assert issubclass(requests.exceptions.ConnectTimeout, requests.ConnectionError) + assert issubclass(requests.exceptions.ReadTimeout, requests.Timeout) + assert issubclass(requests.exceptions.ChunkedEncodingError, requests.RequestException) + + +def test_two_roots_of_one_server_do_not_lose_a_file_to_itself(client: WebDAVClient) -> None: + whole = WebDAVStorage(client) + inner = WebDAVStorage(client, root="team/a") + whole.mkdir("team/a") + inner.write_bytes("docs/a.txt", b"payload") + for operation in (whole.move_from, whole.copy_from): + with pytest.raises(StorageException, match="same file"): + operation(inner, "docs/a.txt", "team/a/docs/a.txt") + with pytest.raises(StorageException, match="same file"): + inner.move_from(whole, "team/a/docs/a.txt", "docs/a.txt") + assert whole.read_bytes("team/a/docs/a.txt") == b"payload" diff --git a/tests/test_webdav_client.py b/tests/test_webdav_client.py index 8d906c0..f9e6f75 100644 --- a/tests/test_webdav_client.py +++ b/tests/test_webdav_client.py @@ -3,11 +3,13 @@ # pylint: disable=redefined-outer-name # pytest passes fixtures by matching name from __future__ import annotations +import re from collections.abc import Iterator from pathlib import Path from unittest.mock import MagicMock, patch import pytest +import requests from automation_file.exceptions import UrlValidationException, WebDAVException from automation_file.remote.webdav.client import WebDAVClient, _parse_propfind @@ -98,8 +100,78 @@ def test_mkcol_sends_mkcol(session_patch: MagicMock, _allow_example_com: None) - def test_error_status_raises(session_patch: MagicMock, _allow_example_com: None) -> None: session_patch.request.return_value = _make_response(status=500) client = WebDAVClient("https://example.com/dav") - with pytest.raises(WebDAVException): + with pytest.raises(WebDAVException) as caught: + client.delete("x") + assert caught.value.status_code == 500 + + +def test_transport_error_has_no_status(session_patch: MagicMock, _allow_example_com: None) -> None: + session_patch.request.side_effect = requests.ConnectionError("refused") + client = WebDAVClient("https://example.com/dav") + with pytest.raises(WebDAVException) as caught: client.delete("x") + assert caught.value.status_code is None + assert isinstance(caught.value.__cause__, requests.ConnectionError) + + +def test_delete_treats_multi_status_as_failure( + session_patch: MagicMock, _allow_example_com: None +) -> None: + session_patch.request.return_value = _make_response(status=207) + client = WebDAVClient("https://example.com/dav") + with pytest.raises(WebDAVException) as caught: + client.delete("folder/") + assert caught.value.status_code == 207 + + +def test_upload_of_an_empty_file_sends_a_body_with_a_length( + session_patch: MagicMock, _allow_example_com: None, tmp_path: Path +) -> None: + local = tmp_path / "empty.bin" + local.write_bytes(b"") + session_patch.request.return_value = _make_response(status=201) + client = WebDAVClient("https://example.com/dav") + client.upload(local, "empty.bin") + _, kwargs = session_patch.request.call_args + assert kwargs["data"] == b"" + + +@pytest.mark.parametrize("method", ["copy", "move"]) +def test_copy_and_move_name_the_destination( + session_patch: MagicMock, _allow_example_com: None, method: str +) -> None: + session_patch.request.return_value = _make_response(status=201) + client = WebDAVClient("https://example.com/dav/") + getattr(client, method)("a b.txt", "/folder/c d.txt") + args, kwargs = session_patch.request.call_args + assert args == (method.upper(), "https://example.com/dav/a%20b.txt") + assert kwargs["headers"] == { + "Destination": "https://example.com/dav/folder/c%20d.txt", + "Overwrite": "T", + } + getattr(client, method)("a.txt", "b.txt", overwrite=False) + _, kwargs = session_patch.request.call_args + assert kwargs["headers"]["Overwrite"] == "F" + session_patch.request.return_value = _make_response(status=207) + with pytest.raises(WebDAVException): + getattr(client, method)("a.txt", "b.txt") + + +def test_a_destination_outside_the_base_url_is_validated(session_patch: MagicMock) -> None: + session_patch.request.return_value = _make_response(status=201) + with patch("automation_file.remote.webdav.client.validate_http_url") as validator: + client = WebDAVClient("https://example.com/dav") + client.copy("a.txt", "b.txt") + assert validator.call_count == 1 + validator.side_effect = UrlValidationException("disallowed ip") + with pytest.raises(UrlValidationException): + client.move("a.txt", "https://internal.example.com/b.txt") + validator.assert_called_with("https://internal.example.com/b.txt", allow_private=False) + assert session_patch.request.call_count == 1 + + +def test_base_url_has_no_trailing_slash(_allow_example_com: None) -> None: + assert WebDAVClient("https://example.com/dav/").base_url == "https://example.com/dav" def test_parse_propfind_multi_entry() -> None: @@ -135,6 +207,46 @@ def test_parse_propfind_multi_entry() -> None: assert entries[1].name == "file.txt" +def test_parse_propfind_reads_etag_and_content_type() -> None: + xml = """ + + + /dav/a%20b;v1.txt + + + + 42 + "abc123" + text/plain + + HTTP/1.1 200 OK + + + + https://example.com/dav/folder/ + + + HTTP/1.1 200 OK + + + + HTTP/1.1 404 Not Found + + + +""" + entries = _parse_propfind(xml) + # A ";" in the last segment belongs to the name: urlparse would cut it off as a parameter. + assert entries[0].name == "a b;v1.txt" + assert (entries[0].size, entries[0].etag, entries[0].content_type) == ( + 42, + '"abc123"', + "text/plain", + ) + assert (entries[1].name, entries[1].is_dir) == ("folder", True) + assert (entries[1].size, entries[1].etag, entries[1].content_type) == (None, None, None) + + def test_parse_propfind_rejects_malformed() -> None: with pytest.raises(WebDAVException): _parse_propfind("") @@ -154,3 +266,130 @@ def test_list_dir_returns_entries(session_patch: MagicMock, _allow_example_com: entries = client.list_dir("") assert len(entries) == 1 assert entries[0].size == 7 + + +_COLLECTION_AND_MEMBER = ( + '' + '' + "/dav/my%20folder/" + "" + "" + "/dav/my%20folder/a.txt" + "7" + "" + "" +) + + +def test_list_dir_can_leave_out_the_collection_itself( + session_patch: MagicMock, _allow_example_com: None +) -> None: + session_patch.request.return_value = _make_response(status=207, text=_COLLECTION_AND_MEMBER) + client = WebDAVClient("https://example.com/dav") + assert [entry.name for entry in client.list_dir("my folder")] == ["my folder", "a.txt"] + assert [entry.name for entry in client.list_dir("my folder/", include_self=False)] == ["a.txt"] + assert [entry.name for entry in client.list_dir("my folder", include_self=False)] == ["a.txt"] + _, kwargs = session_patch.request.call_args + assert kwargs["headers"]["Depth"] == "1" + + +def test_stat_asks_for_the_resource_itself( + session_patch: MagicMock, _allow_example_com: None +) -> None: + session_patch.request.return_value = _make_response(status=207, text=_COLLECTION_AND_MEMBER) + client = WebDAVClient("https://example.com/dav") + entry = client.stat("my folder") + assert (entry.name, entry.is_dir) == ("my folder", True) + args, kwargs = session_patch.request.call_args + assert args == ("PROPFIND", "https://example.com/dav/my%20folder") + assert kwargs["headers"]["Depth"] == "0" + assert "" in kwargs["data"] + empty = '' + session_patch.request.return_value = _make_response(status=207, text=empty) + with pytest.raises(WebDAVException): + client.stat("my folder") + + +def _redirect(status: int, location: str | None) -> MagicMock: + response = _make_response(status=status) + response.headers = {"Location": location} if location else {} + return response + + +def test_a_request_never_goes_to_another_server( + session_patch: MagicMock, _allow_example_com: None, tmp_path: Path +) -> None: + client = WebDAVClient("https://example.com/dav", "user", "secret") + for call in ( + lambda: client.exists("https://attacker.example/steal"), + lambda: client.download("https://attacker.example/steal", tmp_path / "out"), + lambda: client.delete("https://example.com:8443/dav/a.txt"), + lambda: client.list_dir(insecure_url("http", "example.com/dav/")), + ): + with pytest.raises(WebDAVException, match=re.escape("only talks to https://example.com")): + call() + session_patch.request.assert_not_called() + + +def test_an_absolute_url_on_the_same_server_is_allowed( + session_patch: MagicMock, _allow_example_com: None +) -> None: + session_patch.request.return_value = _make_response(status=200) + client = WebDAVClient("https://example.com/dav") + assert client.exists("https://EXAMPLE.com/dav/folder/file.txt") is True + + +def test_redirects_are_never_left_to_requests( + session_patch: MagicMock, _allow_example_com: None +) -> None: + session_patch.request.return_value = _make_response(status=200) + WebDAVClient("https://example.com/dav").exists("a.txt") + assert session_patch.request.call_args.kwargs["allow_redirects"] is False + + +def test_a_read_follows_a_redirect_on_the_same_server( + session_patch: MagicMock, _allow_example_com: None +) -> None: + session_patch.request.side_effect = [ + _redirect(301, "/dav/folder/"), + _make_response(status=200), + ] + assert WebDAVClient("https://example.com/dav").exists("folder") is True + urls = [call.args[1] for call in session_patch.request.call_args_list] + assert urls == ["https://example.com/dav/folder", "https://example.com/dav/folder/"] + + +def test_a_redirect_to_another_server_is_refused( + session_patch: MagicMock, _allow_example_com: None +) -> None: + session_patch.request.return_value = _redirect(302, "https://internal.example/admin") + with pytest.raises(WebDAVException, match=re.escape("only talks to https://example.com")): + WebDAVClient("https://example.com/dav").exists("a.txt") + assert session_patch.request.call_count == 1 + + +def test_a_write_does_not_follow_a_redirect( + session_patch: MagicMock, _allow_example_com: None, tmp_path: Path +) -> None: + local = tmp_path / "data.bin" + local.write_bytes(b"payload") + session_patch.request.return_value = _redirect(307, "/dav/elsewhere.bin") + with pytest.raises(WebDAVException, match="redirect is not followed") as caught: + WebDAVClient("https://example.com/dav").upload(local, "data.bin") + assert caught.value.status_code == 307 + assert session_patch.request.call_count == 1 + + +def test_a_redirect_loop_ends(session_patch: MagicMock, _allow_example_com: None) -> None: + session_patch.request.return_value = _redirect(301, "/dav/again") + with pytest.raises(WebDAVException, match="more than 5 redirects"): + WebDAVClient("https://example.com/dav").exists("a.txt") + assert session_patch.request.call_count == 6 + + +def test_a_redirect_without_a_location_is_an_error( + session_patch: MagicMock, _allow_example_com: None +) -> None: + session_patch.request.return_value = _redirect(302, None) + with pytest.raises(WebDAVException, match="redirect is not followed"): + WebDAVClient("https://example.com/dav").exists("a.txt") From d9416a6b3e8813bbf474baff9ecde61631eb4274 Mon Sep 17 00:00:00 2001 From: JeffreyChen Date: Thu, 8 Oct 2026 13:39:51 +0800 Subject: [PATCH 33/59] feat: add IntegrityMonitor 2.0 The integrity package compares a tree at any storage URI with an approved baseline: snapshot, verify, watch and continuous modes, six kinds of change, a versioned manifest, one IntegrityViolation event per pass that finds drift, and opt-in remediation. The first monitor's call, summary and notification are kept. --- CLAUDE.md | 4 + README.md | 52 +- README.zh-CN.md | 48 +- README.zh-TW.md | 48 +- architecture.md | 10 +- automation_file/__init__.py | 18 +- automation_file/core/action_registry.py | 7 + automation_file/core/fim.py | 161 +----- automation_file/core/manifest.py | 6 + automation_file/integrity/__init__.py | 80 +++ automation_file/integrity/actions.py | 156 ++++++ automation_file/integrity/alerts.py | 187 +++++++ automation_file/integrity/baseline.py | 113 ++++ automation_file/integrity/detector.py | 196 +++++++ automation_file/integrity/errors.py | 9 + automation_file/integrity/hashing.py | 97 ++++ automation_file/integrity/legacy.py | 146 ++++++ automation_file/integrity/local_watcher.py | 112 ++++ automation_file/integrity/manifest.py | 138 +++++ automation_file/integrity/monitor.py | 573 +++++++++++++++++++++ automation_file/integrity/remediation.py | 241 +++++++++ automation_file/integrity/report.py | 108 ++++ automation_file/integrity/snapshot.py | 325 ++++++++++++ automation_file/integrity/target.py | 168 ++++++ automation_file/integrity/watcher.py | 200 +++++++ docs/source/API/api_index.rst | 15 + docs/source/API/integrity.rst | 82 +++ docs/source/Eng/eng_index.rst | 18 +- docs/source/Eng/usage/integrity.rst | 514 ++++++++++++++++++ docs/source/Zh-CN/usage/integrity.rst | 472 +++++++++++++++++ docs/source/Zh-CN/zh_cn_index.rst | 14 + docs/source/Zh-TW/usage/integrity.rst | 471 +++++++++++++++++ docs/source/Zh-TW/zh_tw_index.rst | 14 + docs/updates/2026-10.md | 21 + docs/updates/README.md | 3 +- progress.md | 2 +- tests/test_integrity_actions.py | 241 +++++++++ tests/test_integrity_detector.py | 317 ++++++++++++ tests/test_integrity_legacy.py | 311 +++++++++++ tests/test_integrity_monitor.py | 572 ++++++++++++++++++++ tests/test_integrity_object_store.py | 264 ++++++++++ tests/test_integrity_remediation.py | 414 +++++++++++++++ tests/test_integrity_snapshot.py | 453 ++++++++++++++++ tests/test_integrity_watch.py | 364 +++++++++++++ 44 files changed, 7565 insertions(+), 200 deletions(-) create mode 100644 automation_file/integrity/__init__.py create mode 100644 automation_file/integrity/actions.py create mode 100644 automation_file/integrity/alerts.py create mode 100644 automation_file/integrity/baseline.py create mode 100644 automation_file/integrity/detector.py create mode 100644 automation_file/integrity/errors.py create mode 100644 automation_file/integrity/hashing.py create mode 100644 automation_file/integrity/legacy.py create mode 100644 automation_file/integrity/local_watcher.py create mode 100644 automation_file/integrity/manifest.py create mode 100644 automation_file/integrity/monitor.py create mode 100644 automation_file/integrity/remediation.py create mode 100644 automation_file/integrity/report.py create mode 100644 automation_file/integrity/snapshot.py create mode 100644 automation_file/integrity/target.py create mode 100644 automation_file/integrity/watcher.py create mode 100644 docs/source/API/integrity.rst create mode 100644 docs/source/Eng/usage/integrity.rst create mode 100644 docs/source/Zh-CN/usage/integrity.rst create mode 100644 docs/source/Zh-TW/usage/integrity.rst create mode 100644 tests/test_integrity_actions.py create mode 100644 tests/test_integrity_detector.py create mode 100644 tests/test_integrity_legacy.py create mode 100644 tests/test_integrity_monitor.py create mode 100644 tests/test_integrity_object_store.py create mode 100644 tests/test_integrity_remediation.py create mode 100644 tests/test_integrity_snapshot.py create mode 100644 tests/test_integrity_watch.py diff --git a/CLAUDE.md b/CLAUDE.md index cb3671d..0895a99 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -35,6 +35,9 @@ automation_file/ │ # actions (FA_storage_* and register_storage_ops) ├── events/ # Event model: model (Event, Severity, the ten core events), bus (EventBus, │ # event_bus, emit), context (correlation_scope, actor_scope), storage_bridge +├── integrity/ # IntegrityMonitor 2.0: target, hashing, snapshot, manifest (schema 2), baseline, +│ # detector, report, alerts, remediation, watcher / local_watcher, legacy, monitor, +│ # actions (FA_integrity_*); core/fim.py re-exports IntegrityMonitor ├── server/ # tcp_server, http_server, mcp_server (MCP over stdio), web_ui, metrics_server, │ # action_acl (ActionACL), network_guards (ensure_loopback) ├── client/ # HTTPActionClient for the HTTP action server @@ -74,6 +77,7 @@ automation_file/ - `safe_join(root, user_path)` / `is_within(root, path)` — path traversal guard; `safe_join` raises `PathTraversalException` when the resolved path escapes `root`. - `File(uri)` / `Storage(uri)` — the universal storage layer's application API: one file, one directory, in any backend. Both resolve their backend on every call through `StorageResolver` (`Storage.mount`, `Storage.register_scheme`). - `StorageBackend` — the contract a storage backend implements. The public operations (`exists`, `stat`, `list_dir`, `mkdir`, `upload`, `download`, `delete`, `checksum`, `read_bytes`, `write_bytes`, `copy_from`, `move_from`) are template methods; a backend supplies only the `_`-prefixed primitives. Twelve are built in: `LocalStorage`, `MemoryStorage`, `S3Storage` and `AzureStorage` (both on `ObjectStorage`), `SFTPStorage` and `FTPStorage` (both on `SessionStorage`), `GoogleDriveStorage`, `OneDriveStorage`, `DropboxStorage`, and the mounted `WebDAVStorage`, `SMBStorage` and `FsspecStorage`. Each uses its backend's shared client singleton unless given one, and reports a missing SDK with the extra to install. +- `IntegrityMonitor` — compares a tree at any storage URI with an approved baseline (`create_baseline`, `verify`, `accept`, `watch`, `start` / `stop`, `snapshot`) and returns a `DriftReport`; drift is published as one `IntegrityViolation` per pass. It only reads unless a `RemediationPolicy` is passed. Its options are keyword arguments (`MonitorKeywords`). The first monitor's call and `check_once()` summary are kept, including the notification through `manager` or the process-wide `notification_manager`. - `Event` / `EventBus` / `event_bus` — every component reports through events (`PipelineFailed`, `TaskFailed`, `IntegrityViolation`, `StorageError`, ...) with a severity, a correlation ID and an actor; consumers subscribe on the bus by class, type name or prefix. New code that has something to report publishes an event; it does not call a notification sink or the audit log directly. - `StorageURI` / `parse_storage_uri` — `:///`; `FileInfo`, `Checksum`, `StorageCapabilities` are the frozen value types the layer returns. diff --git a/README.md b/README.md index 8872c1b..8c3a8cc 100644 --- a/README.md +++ b/README.md @@ -38,7 +38,7 @@ facade. - **Variable substitution** — opt-in `${env:VAR}` / `${date:%Y-%m-%d}` / `${uuid}` / `${cwd}` expansion in action arguments via `execute_action(..., substitute=True)` - **Conditional execution** — `FA_if_exists` / `FA_if_newer` / `FA_if_size_gt` run a nested action list only when a guard passes - **SQLite audit log** — `AuditLog(db_path)` records every action execution with actor / status / duration; query via `recent` / `count` / `purge` -- **File integrity monitor** — `IntegrityMonitor` polls a tree against a manifest and fires a callback + notification on drift +- **File integrity monitoring** — `IntegrityMonitor` keeps a versioned baseline of a tree in any storage backend, detects created / modified / deleted / renamed files and metadata or permission changes, publishes drift as an event, and quarantines or restores only when a policy asks for it - **HTTPActionClient SDK** — typed Python client for the HTTP action server with shared-secret auth, loopback guard, and OPTIONS-based ping - **AES-256-GCM file encryption** — `encrypt_file` / `decrypt_file` with `generate_key()` / `key_from_password()` (PBKDF2-HMAC-SHA256); JSON actions `FA_encrypt_file` / `FA_decrypt_file` - **Prometheus metrics exporter** — `start_metrics_server()` exposes `automation_file_actions_total{action,status}` counters and `automation_file_action_duration_seconds{action}` histograms @@ -835,26 +835,44 @@ for row in audit.recent(limit=50): print(row["timestamp"], row["action"], row["status"]) ``` -### File integrity monitor -Poll a tree against a manifest and fire a callback + notification on drift: +### File integrity monitoring +`IntegrityMonitor` checks that a directory tree, in any storage backend, is still what was +approved: it stores a baseline, compares the tree with it, and publishes every drift as an event. ```python -from automation_file import IntegrityMonitor, notification_manager, write_manifest - -write_manifest("/srv/site", "/srv/MANIFEST.json") - -mon = IntegrityMonitor( - root="/srv/site", - manifest_path="/srv/MANIFEST.json", - interval=60.0, - manager=notification_manager, - on_drift=lambda summary: print("drift:", summary), -) -mon.start() +from automation_file import IntegrityMonitor + +monitor = IntegrityMonitor("s3://reports/2026", + baseline="local:///var/lib/fa/reports-2026.json") +monitor.create_baseline() # approve what is there now +report = monitor.verify() # hashes every file; verify(deep=False) is the quick pass +if not report.ok: + print(report.counts) # {'created': 0, 'modified': 1, 'deleted': 0, ...} + monitor.accept(report) # after review: approve what the report saw +monitor.start() # continuous mode: verify every `interval` seconds +handle = monitor.watch() # or react to changes as they happen; handle.stop() ends it ``` -Manifest-load errors are surfaced as drift so tamper and config issues -aren't silently different code paths. +- **Four modes** — `snapshot()`, `verify()`, `watch()` (filesystem events for a local target, + polling for any other backend) and continuous `start()` / `stop()`. +- **Six kinds of change** — `created`, `modified`, `deleted`, `renamed`, `metadata_changed` and + `permission_changed`, in a `DriftReport` with counts per kind and `to_dict()`. +- **Baseline anywhere** — a versioned JSON manifest at any storage URI, written atomically; the + `write_manifest` format is still read. SHA-256 by default, `sha512` and `blake2b` on request, + `md5` and `sha1` only with `allow_weak=True`. +- **Events and opt-in remediation** — one `IntegrityViolation` per verification that finds drift + (`error` when something was modified or deleted, `warning` for additions and metadata). The + monitor only reads unless a `RemediationPolicy` tells it to quarantine or to restore from a + mirror, which it verifies by checksum. +- **Actions** — `FA_integrity_snapshot`, `FA_integrity_baseline`, `FA_integrity_verify`, + `FA_integrity_accept`, `FA_integrity_watch_start`, `FA_integrity_watch_stop`, + `FA_integrity_status`. + +Code written for the first monitor keeps working: `IntegrityMonitor(root=..., manifest_path=..., +interval=..., manager=..., on_drift=...)` reads a manifest written by `write_manifest`, `check_once()` +returns the same summary, and the notification still goes through `manager` or, when none is passed, +the process-wide `notification_manager`. Pass `notify=False` when the published event is routed to +your sinks instead, so one drift is not announced twice. ### AES-256-GCM file encryption Authenticated encryption with a self-describing envelope. Derive a key from diff --git a/README.zh-CN.md b/README.zh-CN.md index 474a4bf..0407abe 100644 --- a/README.zh-CN.md +++ b/README.zh-CN.md @@ -36,7 +36,7 @@ TCP / HTTP 服务器执行的 JSON 驱动动作。内附 PySide6 GUI，每个功 - **变量替换** — 动作参数中可选使用 `${env:VAR}` / `${date:%Y-%m-%d}` / `${uuid}` / `${cwd}`,通过 `execute_action(..., substitute=True)` 展开 - **条件执行** — `FA_if_exists` / `FA_if_newer` / `FA_if_size_gt` 仅在路径守卫通过时执行嵌套动作清单 - **SQLite 审计日志** — `AuditLog(db_path)` 为每个动作记录 actor / status / duration;通过 `recent` / `count` / `purge` 查询 -- **文件完整性监控** — `IntegrityMonitor` 按 manifest 轮询整棵树,检测到 drift 时触发 callback + 通知 +- **文件完整性监控** — `IntegrityMonitor` 为任何存储后端中的目录树保存带版本的基准，检测新增、修改、删除、重命名以及元数据或权限的变更，把偏移发布为事件，并且只在策略要求时才隔离或还原 - **HTTPActionClient SDK** — HTTP 动作服务器的类型化 Python 客户端,具 shared-secret 认证、loopback 守护与 OPTIONS ping - **AES-256-GCM 文件加密** — `encrypt_file` / `decrypt_file` 搭配 `generate_key()` / `key_from_password()`(PBKDF2-HMAC-SHA256);JSON 动作 `FA_encrypt_file` / `FA_decrypt_file` - **Prometheus metrics 导出器** — `start_metrics_server()` 提供 `automation_file_actions_total{action,status}` 计数器与 `automation_file_action_duration_seconds{action}` 直方图 @@ -819,25 +819,41 @@ for row in audit.recent(limit=50): ``` ### 文件完整性监控 -按 manifest 轮询整棵树,检测到 drift 时触发 callback + 通知: +`IntegrityMonitor` 检查任何存储后端中的目录树是否仍然是当初批准的样子：它存储基线、拿目录树与 +基线比较，并把每一次偏移以事件的形式发布。 ```python -from automation_file import IntegrityMonitor, notification_manager, write_manifest - -write_manifest("/srv/site", "/srv/MANIFEST.json") - -mon = IntegrityMonitor( - root="/srv/site", - manifest_path="/srv/MANIFEST.json", - interval=60.0, - manager=notification_manager, - on_drift=lambda summary: print("drift:", summary), -) -mon.start() +from automation_file import IntegrityMonitor + +monitor = IntegrityMonitor("s3://reports/2026", + baseline="local:///var/lib/fa/reports-2026.json") +monitor.create_baseline() # 批准当前的内容 +report = monitor.verify() # 对每个文件计算哈希；verify(deep=False) 是快速验证 +if not report.ok: + print(report.counts) # {'created': 0, 'modified': 1, 'deleted': 0, ...} + monitor.accept(report) # 审查之后：批准这份报告所看到的状态 +monitor.start() # 持续模式：每隔 `interval` 秒验证一次 +handle = monitor.watch() # 或在变更发生时即时响应；handle.stop() 结束监视 ``` -加载 manifest 时的错误也会被视为 drift,让篡改与配置问题走同一条处理 -路径。 +- **四种模式** — `snapshot()`、`verify()`、`watch()`（本地目标使用文件系统事件，其他后端使用 + 轮询）以及持续模式的 `start()` / `stop()`。 +- **六种变更** — `created`、`modified`、`deleted`、`renamed`、`metadata_changed` 与 + `permission_changed`，汇总在带有各种类数量与 `to_dict()` 的 `DriftReport` 中。 +- **基线可放在任何地方** — 位于任意存储 URI、带有版本的 JSON manifest，以原子方式写入；仍可读取 + `write_manifest` 的格式。默认使用 SHA-256，可改用 `sha512` 与 `blake2b`，`md5` 与 `sha1` 只有在 + `allow_weak=True` 时才能使用。 +- **事件与需显式开启的补救** — 每一次发现偏移的验证发布一个 `IntegrityViolation`（有东西被修改 + 或删除时为 `error`，新增与元数据变更为 `warning`）。除非以 `RemediationPolicy` 要求隔离， + 或要求从镜像还原（会以校验码验证），否则监控器只会读取。 +- **动作** — `FA_integrity_snapshot`、`FA_integrity_baseline`、`FA_integrity_verify`、 + `FA_integrity_accept`、`FA_integrity_watch_start`、`FA_integrity_watch_stop`、 + `FA_integrity_status`。 + +为第一代监控器写的代码照常工作：`IntegrityMonitor(root=..., manifest_path=..., interval=..., +manager=..., on_drift=...)` 会读取 `write_manifest` 写出的 manifest，`check_once()` 返回同样的摘要， +通知也仍然通过 `manager` 发送，没有传入时则使用整个进程共用的 `notification_manager`。如果改由 +发布的事件把偏移送到通知渠道，请传入 `notify=False`，同一次偏移才不会被通知两次。 ### AES-256-GCM 文件加密 带认证的加密与自描述封包格式。可由密码派生密钥或直接生成密钥: diff --git a/README.zh-TW.md b/README.zh-TW.md index fcd11f7..7115f02 100644 --- a/README.zh-TW.md +++ b/README.zh-TW.md @@ -36,7 +36,7 @@ TCP / HTTP 伺服器執行的 JSON 驅動動作。內附 PySide6 GUI，每個功 - **變數替換** — 動作參數中可選使用 `${env:VAR}` / `${date:%Y-%m-%d}` / `${uuid}` / `${cwd}`，透過 `execute_action(..., substitute=True)` 展開 - **條件式執行** — `FA_if_exists` / `FA_if_newer` / `FA_if_size_gt` 僅在路徑守護通過時執行巢狀動作清單 - **SQLite 稽核日誌** — `AuditLog(db_path)` 為每個動作記錄 actor / status / duration；以 `recent` / `count` / `purge` 查詢 -- **檔案完整性監控** — `IntegrityMonitor` 依 manifest 輪詢整棵樹，偵測到 drift 時觸發 callback + 通知 +- **檔案完整性監控** — `IntegrityMonitor` 為任何儲存後端中的目錄樹保存帶版本的基準，偵測新增、修改、刪除、重新命名以及中繼資料或權限的變更，把偏移發布為事件，並且只在政策要求時才隔離或還原 - **HTTPActionClient SDK** — HTTP 動作伺服器的型別化 Python 客戶端，具 shared-secret 驗證、loopback 防護與 OPTIONS ping - **AES-256-GCM 檔案加密** — `encrypt_file` / `decrypt_file` 搭配 `generate_key()` / `key_from_password()`（PBKDF2-HMAC-SHA256）；JSON 動作 `FA_encrypt_file` / `FA_decrypt_file` - **Prometheus metrics 匯出器** — `start_metrics_server()` 提供 `automation_file_actions_total{action,status}` 計數器與 `automation_file_action_duration_seconds{action}` 直方圖 @@ -819,25 +819,41 @@ for row in audit.recent(limit=50): ``` ### 檔案完整性監控 -依 manifest 輪詢整棵樹，偵測到 drift 時觸發 callback + 通知： +`IntegrityMonitor` 檢查任何儲存後端中的目錄樹是否仍然是當初核可的樣子：它儲存基準、拿目錄樹與 +基準比對，並把每一次偏移以事件的形式發布。 ```python -from automation_file import IntegrityMonitor, notification_manager, write_manifest - -write_manifest("/srv/site", "/srv/MANIFEST.json") - -mon = IntegrityMonitor( - root="/srv/site", - manifest_path="/srv/MANIFEST.json", - interval=60.0, - manager=notification_manager, - on_drift=lambda summary: print("drift:", summary), -) -mon.start() +from automation_file import IntegrityMonitor + +monitor = IntegrityMonitor("s3://reports/2026", + baseline="local:///var/lib/fa/reports-2026.json") +monitor.create_baseline() # 核可目前的內容 +report = monitor.verify() # 雜湊每個檔案；verify(deep=False) 是快速驗證 +if not report.ok: + print(report.counts) # {'created': 0, 'modified': 1, 'deleted': 0, ...} + monitor.accept(report) # 檢視之後：核可這份報告所看到的狀態 +monitor.start() # 持續模式：每隔 `interval` 秒驗證一次 +handle = monitor.watch() # 或在變更發生時即時反應；handle.stop() 結束監看 ``` -載入 manifest 時的錯誤也會被視為 drift，讓竄改與設定問題走同一條處理 -路徑。 +- **四種模式** — `snapshot()`、`verify()`、`watch()`（本機目標使用檔案系統事件，其他後端使用 + 輪詢）以及持續模式的 `start()` / `stop()`。 +- **六種變更** — `created`、`modified`、`deleted`、`renamed`、`metadata_changed` 與 + `permission_changed`，彙整在帶有各種類數量與 `to_dict()` 的 `DriftReport` 中。 +- **基準可放在任何地方** — 位於任意儲存 URI、帶有版本的 JSON manifest，以原子方式寫入；仍可讀取 + `write_manifest` 的格式。預設使用 SHA-256，可改用 `sha512` 與 `blake2b`，`md5` 與 `sha1` 只有在 + `allow_weak=True` 時才能使用。 +- **事件與需明確開啟的補救** — 每一次發現偏移的驗證發布一個 `IntegrityViolation`（有東西被修改 + 或刪除時為 `error`，新增與中繼資料變更為 `warning`）。除非以 `RemediationPolicy` 要求隔離， + 或要求從鏡像還原（會以校驗碼驗證），否則監控器只會讀取。 +- **動作** — `FA_integrity_snapshot`、`FA_integrity_baseline`、`FA_integrity_verify`、 + `FA_integrity_accept`、`FA_integrity_watch_start`、`FA_integrity_watch_stop`、 + `FA_integrity_status`。 + +為第一代監控器寫的程式照常運作：`IntegrityMonitor(root=..., manifest_path=..., interval=..., +manager=..., on_drift=...)` 會讀取 `write_manifest` 寫出的 manifest，`check_once()` 回傳同樣的摘要， +通知也仍然透過 `manager` 送出，沒有傳入時則使用整個行程共用的 `notification_manager`。如果改由 +發布的事件把偏移送到通知管道，請傳入 `notify=False`，同一次偏移才不會被通知兩次。 ### AES-256-GCM 檔案加密 具驗證的加密與自述式封包格式。可由密碼衍生金鑰或直接產生金鑰： diff --git a/architecture.md b/architecture.md index 3d65590..132e298 100644 --- a/architecture.md +++ b/architecture.md @@ -26,6 +26,7 @@ piece of the 1.0 roadmap (`docs/FILEAUTOMATION-1.0-ROADMAP.md`, PR #107); what i | `automation_file/remote/` | `url_validator.py` (SSRF guard), `http_download.py`, `cross_backend.py`, `fsspec_bridge.py`. One subpackage per backend: `google_drive/`, `s3/`, `azure_blob/`, `dropbox_api/`, `sftp/`, `ftp/`, `onedrive/`, `box/`, each with `client.py`, `*_ops.py` and `register__ops`. `smb/` and `webdav/` have a client only | | `automation_file/storage/` | Universal storage layer. `uri.py` (`StorageURI`, `parse_storage_uri`, `normalize_path`), `types.py` (`FileInfo`, `Checksum`, `StorageCapabilities`), `backend.py` (`StorageBackend`: the public operations are template methods over the `_`-prefixed primitives a backend supplies), `local_storage.py` (`LocalStorage`, confined through `safe_join` when given a root), `memory_storage.py` (`MemoryStorage`), `object_storage.py` (`ObjectStorage`: directories as key prefixes over `_head`, `_scan`, `_put`, `_get`, `_remove`), `s3_storage.py` (`S3Storage`, over `s3_instance` or a given boto3 client), `azure_storage.py` (`AzureStorage`, over `azure_blob_instance` or a given `BlobServiceClient`), `session_storage.py` (`SessionStorage`: one login session, one operation at a time, and `require_session_host`), `sftp_storage.py` (`SFTPStorage`), `ftp_storage.py` (`FTPStorage`, for `ftp` and `ftps`), `gdrive_storage.py` (`GoogleDriveStorage`: paths resolved to file IDs, duplicate names refused), `onedrive_storage.py` (`OneDriveStorage`, Microsoft Graph), `dropbox_storage.py` (`DropboxStorage`), `webdav_storage.py` (`WebDAVStorage`), `smb_storage.py` (`SMBStorage`), `fsspec_storage.py` (`FsspecStorage`, any fsspec filesystem), `timestamps.py` (RFC 3339 parsing), `resolver.py` (`StorageResolver`, `default_resolver`: mounts first, then scheme factories), `file.py` (`File`), `storage.py` (`Storage`), `observe.py` (listeners for `upload`, `download`, `read`, `delete`, `mkdir`, `copy`, `move`), `streams.py` (staged file objects behind `open_read` / `open_write`), `tree.py` (`copy_tree`, `sync_tree`, `TreeResult`), `actions.py` (the `FA_storage_*` functions and `register_storage_ops`). At module level it imports only `exceptions`, `logging_config`, `core.checksum` and `local.safe_paths`: no registry, no GUI, no backend SDK. The adapters import their SDK's exceptions and the shared client inside the functions that use them | | `automation_file/events/` | The event model every component reports through. `model.py` (`Event`, `Severity`, the ten core events), `bus.py` (`EventBus`, the process-wide `event_bus`, `emit`), `context.py` (`correlation_scope`, `actor_scope`), `storage_bridge.py` (failed storage operations become `StorageError` events; installed when the package is imported). It imports only the standard library, `logging_config` and `storage.observe` | +| `automation_file/integrity/` | IntegrityMonitor 2.0, on the storage layer and the event bus. `target.py` (`Target`: the monitored tree behind a storage URI), `hashing.py` (`HashEngine`; `md5` and `sha1` only with `allow_weak`), `snapshot.py` (`Snapshot`, `SnapshotEntry`, `build_snapshot`), `manifest.py` (schema version 2; the `write_manifest` format is read and converted), `baseline.py` (`BaselineManager`: an atomic write at any storage URI), `detector.py` (`Change`, `ChangeKind`, `detect_changes`: six kinds of change), `report.py` (`DriftReport`), `alerts.py` (`AlertEngine`, `AlertPolicy`: one `IntegrityViolation` per pass that finds drift), `remediation.py` (`RemediationPolicy`, `Remediator`: quarantine or restore, opt-in), `watcher.py` and `local_watcher.py` (polling, and watchdog events for a local target), `legacy.py` (the first monitor's summary, callback and notification), `monitor.py` (`IntegrityMonitor`), `actions.py` (`FA_integrity_*`). `core/fim.py` re-exports the class | | `automation_file/server/` | `tcp_server.py`, `http_server.py`, `mcp_server.py`, `web_ui.py`, `metrics_server.py`, `action_acl.py` (`ActionACL`), `network_guards.py` (`ensure_loopback`) | | `automation_file/client/` | `HTTPActionClient` for the HTTP action server | | `automation_file/trigger/`, `scheduler/`, `notify/` | Watchdog file triggers, cron scheduler, notification sinks. Each registers its own `FA_*` ops | @@ -58,6 +59,13 @@ piece of the 1.0 roadmap (`docs/FILEAUTOMATION-1.0-ROADMAP.md`, PR #107); what i `download`, `delete`, `checksum`, `verify`, `copy`, `move`, `read_text`, `write_text`, `copy_tree`, `sync`, `schemes`) put it in the default registry; `register_storage_ops` adds them to another one. +- **Integrity** (same facade): `IntegrityMonitor`, `DriftReport`, `BaselineManager`, `AlertPolicy`, + `RemediationPolicy`, `IntegrityRemediated`, `IntegrityException`, `register_integrity_ops`; the + rest (`Snapshot`, `Change`, `HashEngine`, the manifest functions) is in `automation_file.integrity`. + Seven actions: `FA_integrity_snapshot`, `FA_integrity_baseline`, `FA_integrity_verify`, + `FA_integrity_accept`, `FA_integrity_watch_start`, `FA_integrity_watch_stop`, `FA_integrity_status`. + The first monitor's call, `IntegrityMonitor(root, manifest_path, interval=, on_drift=, manager=, + alert_on_extra=)`, and its `check_once()` summary are kept. - **Events** (same facade): `Event`, `Severity`, `EventBus`, `event_bus`, `emit`, `correlation_scope`, `actor_scope`, and the core events `PipelineStarted`, `PipelineCompleted`, `PipelineFailed`, `TaskStarted`, `TaskCompleted`, `TaskFailed`, `IntegrityViolation`, `StorageError`, `SchedulerError`, @@ -124,7 +132,7 @@ MCP host → automation_file_mcp (stdio JSON-RPC) → tools/call → MCPServer r ``` ActionExecutor() → build_default_registry(): local + http + utils + drive commands → _register_cloud_backends (register__ops) → trigger / scheduler / progress / notify ops - → storage ops (FA_storage_*) + → storage ops (FA_storage_*) → integrity ops (FA_integrity_*) → _load_plugins (entry points; may override built-ins) → executor adds FA_execute_action, FA_execute_files, FA_execute_action_parallel, FA_validate ``` diff --git a/automation_file/__init__.py b/automation_file/__init__.py index f9f8cc1..8a1a00f 100644 --- a/automation_file/__init__.py +++ b/automation_file/__init__.py @@ -41,7 +41,6 @@ ) from automation_file.core.dag_executor import execute_action_dag from automation_file.core.file_lock import FileLock -from automation_file.core.fim import IntegrityMonitor from automation_file.core.json_store import read_action_json, write_action_json from automation_file.core.manifest import ManifestException, verify_manifest, write_manifest from automation_file.core.metrics import ACTION_COUNT, ACTION_DURATION, record_action @@ -111,6 +110,16 @@ TextOpsException, TracingException, ) +from automation_file.integrity import ( + AlertPolicy, + BaselineManager, + DriftReport, + IntegrityException, + IntegrityMonitor, + IntegrityRemediated, + RemediationPolicy, + register_integrity_ops, +) from automation_file.local.archive_ops import ( detect_archive_format, extract_archive, @@ -592,6 +601,13 @@ def __getattr__(name: str) -> Any: "AuditException", "AuditLog", "IntegrityMonitor", + "IntegrityException", + "DriftReport", + "BaselineManager", + "AlertPolicy", + "RemediationPolicy", + "IntegrityRemediated", + "register_integrity_ops", "CryptoException", "encrypt_file", "decrypt_file", diff --git a/automation_file/core/action_registry.py b/automation_file/core/action_registry.py index 57f19bf..d087408 100644 --- a/automation_file/core/action_registry.py +++ b/automation_file/core/action_registry.py @@ -224,6 +224,12 @@ def _register_storage_ops(registry: ActionRegistry) -> None: register_storage_ops(registry) +def _register_integrity_ops(registry: ActionRegistry) -> None: + from automation_file.integrity.actions import register_integrity_ops + + register_integrity_ops(registry) + + def build_default_registry() -> ActionRegistry: """Return a registry pre-populated with every built-in ``FA_*`` action. @@ -242,6 +248,7 @@ def build_default_registry() -> ActionRegistry: _register_progress_ops(registry) _register_notify_ops(registry) _register_storage_ops(registry) + _register_integrity_ops(registry) _load_plugins(registry) # DEBUG, not INFO: this runs at import, and INFO is mirrored to stderr, so every import -- # `python -m automation_file --help` included -- printed it. diff --git a/automation_file/core/fim.py b/automation_file/core/fim.py index 6ab6541..4b154ea 100644 --- a/automation_file/core/fim.py +++ b/automation_file/core/fim.py @@ -1,151 +1,20 @@ -"""File integrity monitoring (FIM) — periodic manifest verification. - -``IntegrityMonitor(root, manifest_path)`` runs a background thread that -re-verifies the tree against a previously-written manifest at a fixed -interval. When drift is detected (missing or modified files) the monitor - -1. invokes the optional ``on_drift`` callback with the verification summary, -2. emits an ``error``-level notification through the supplied - :class:`~automation_file.notify.NotificationManager` (defaults to the - process-wide singleton), and -3. logs a single warning describing the counts. - -This is the "watchdog" side of manifests: once a baseline has been written -with :func:`write_manifest`, a monitor keeps checking that the tree still -matches and alerts when it does not. Extras (new files not in the manifest) -do not count as drift by default — mirrors the posture of ``verify_manifest``. +"""File integrity monitoring (FIM): the legacy import path. + +The monitor lives in :mod:`automation_file.integrity`. This module keeps +``from automation_file.core.fim import IntegrityMonitor`` working: the legacy +call ``IntegrityMonitor(root, manifest_path, interval=..., on_drift=..., +manager=..., alert_on_extra=...)`` runs on the new class, reads a manifest +written by :func:`automation_file.core.manifest.write_manifest`, and +``check_once()`` returns the same summary dictionary. + +The notification still goes through the ``manager`` passed to the constructor, +or through the process-wide ``notification_manager`` when none is. One thing was +added: drift is also published as an ``IntegrityViolation`` event. """ from __future__ import annotations -import threading -from collections.abc import Callable -from pathlib import Path -from typing import Any - -from automation_file.core.manifest import verify_manifest -from automation_file.exceptions import FileAutomationException -from automation_file.logging_config import file_automation_logger -from automation_file.notify import NotificationManager, notification_manager - -OnDrift = Callable[[dict[str, Any]], None] - - -class IntegrityMonitor: - """Periodically verify a manifest and fire alerts on drift.""" - - def __init__( - self, - root: str | Path, - manifest_path: str | Path, - *, - interval: float = 60.0, - on_drift: OnDrift | None = None, - manager: NotificationManager | None = None, - alert_on_extra: bool = False, - ) -> None: - if interval <= 0: - raise FileAutomationException("interval must be positive") - self._root = Path(root) - self._manifest_path = Path(manifest_path) - self._interval = float(interval) - self._on_drift = on_drift - self._manager = manager or notification_manager - self._alert_on_extra = bool(alert_on_extra) - self._stop = threading.Event() - self._thread: threading.Thread | None = None - self._last_summary: dict[str, Any] | None = None - - @property - def last_summary(self) -> dict[str, Any] | None: - return self._last_summary - - def start(self) -> None: - """Arm the monitor. The first verification runs on the next tick.""" - if self._thread is not None and self._thread.is_alive(): - return - self._stop.clear() - thread = threading.Thread(target=self._run, name="fa-integrity-monitor", daemon=True) - thread.start() - self._thread = thread - file_automation_logger.info( - "integrity_monitor: watching %s against %s (interval=%.1fs)", - self._root, - self._manifest_path, - self._interval, - ) - - def stop(self, timeout: float = 5.0) -> None: - self._stop.set() - thread = self._thread - self._thread = None - if thread is not None and thread.is_alive(): - thread.join(timeout=timeout) - - def check_once(self) -> dict[str, Any]: - """Run one verification pass and return the summary.""" - try: - summary = verify_manifest(self._root, self._manifest_path) - except FileAutomationException as err: - file_automation_logger.error("integrity_monitor: verify failed: %r", err) - summary = { - "matched": [], - "missing": [], - "modified": [], - "extra": [], - "ok": False, - "error": repr(err), - } - self._last_summary = summary - if self._is_drift(summary): - self._handle_drift(summary) - return summary - - def _is_drift(self, summary: dict[str, Any]) -> bool: - if summary.get("error"): - return True - if summary.get("missing") or summary.get("modified"): - return True - return bool(self._alert_on_extra and summary.get("extra")) - - def _handle_drift(self, summary: dict[str, Any]) -> None: - file_automation_logger.warning( - "integrity_monitor: drift detected missing=%d modified=%d extra=%d", - len(summary.get("missing") or []), - len(summary.get("modified") or []), - len(summary.get("extra") or []), - ) - if self._on_drift is not None: - try: - self._on_drift(summary) - except FileAutomationException as err: - file_automation_logger.error("integrity_monitor: on_drift raised: %r", err) - body = _format_body(summary) - try: - self._manager.notify( - subject=f"integrity drift: {self._root}", - body=body, - level="error", - ) - except FileAutomationException as err: - file_automation_logger.error("integrity_monitor: notify failed: %r", err) - - def _run(self) -> None: - while not self._stop.is_set(): - self._stop.wait(self._interval) - if self._stop.is_set(): - break - self.check_once() - +from automation_file.integrity.legacy import OnDrift +from automation_file.integrity.monitor import IntegrityMonitor -def _format_body(summary: dict[str, Any]) -> str: - parts: list[str] = [] - if summary.get("error"): - parts.append(f"error: {summary['error']}") - for key in ("missing", "modified", "extra"): - items = summary.get(key) or [] - if items: - preview = ", ".join(items[:5]) - suffix = f" (+{len(items) - 5} more)" if len(items) > 5 else "" - parts.append(f"{key}: {preview}{suffix}") - return "\n".join(parts) if parts else "no drift detected" +__all__ = ["IntegrityMonitor", "OnDrift"] diff --git a/automation_file/core/manifest.py b/automation_file/core/manifest.py index be409dc..e44b1e1 100644 --- a/automation_file/core/manifest.py +++ b/automation_file/core/manifest.py @@ -164,6 +164,12 @@ def _load_manifest(path: str | os.PathLike[str]) -> dict[str, Any]: data = json.loads(manifest_path.read_text(encoding="utf-8")) except (OSError, json.JSONDecodeError) as err: raise ManifestException(f"cannot read manifest {manifest_path}: {err}") from err + if isinstance(data, dict) and "schema_version" in data: + raise ManifestException( + f"{manifest_path} is an integrity baseline (schema version " + f"{data['schema_version']}), which verify_manifest does not read: verify it " + "with IntegrityMonitor.verify() or FA_integrity_verify" + ) if not isinstance(data, dict) or "files" not in data: raise ManifestException(f"manifest missing 'files' mapping: {manifest_path}") return data diff --git a/automation_file/integrity/__init__.py b/automation_file/integrity/__init__.py new file mode 100644 index 0000000..6378d9d --- /dev/null +++ b/automation_file/integrity/__init__.py @@ -0,0 +1,80 @@ +"""IntegrityMonitor 2.0: file integrity monitoring over any storage backend. + +* :class:`IntegrityMonitor` ties the parts together: ``snapshot()``, + ``create_baseline()``, ``verify()``, ``accept()``, ``watch()`` and + ``start()`` / ``stop()``. +* :class:`Snapshot` / :class:`SnapshotEntry` describe a tree; + :func:`dump_manifest` / :func:`load_manifest` are its versioned JSON form, and + :class:`BaselineManager` keeps the approved one at a storage URI. +* :class:`HashEngine` hashes files in parallel and refuses weak digests. +* :func:`detect_changes` compares two snapshots into :class:`Change` records, + and a :class:`DriftReport` is the outcome of one verification. +* :class:`AlertEngine` publishes drift as events; :class:`RemediationPolicy` + turns on quarantine and restore, which are off by default. +* :func:`register_integrity_ops` adds the ``FA_integrity_*`` actions to a registry. +""" + +from __future__ import annotations + +from automation_file.integrity.actions import register_integrity_ops +from automation_file.integrity.alerts import ( + DEFAULT_SEVERITIES, + AlertEngine, + AlertPolicy, + IntegrityRemediated, +) +from automation_file.integrity.baseline import BaselineManager +from automation_file.integrity.detector import Change, ChangeKind, detect_changes +from automation_file.integrity.errors import IntegrityException +from automation_file.integrity.hashing import ( + DEFAULT_ALGORITHM, + STRONG_ALGORITHMS, + WEAK_ALGORITHMS, + HashEngine, +) +from automation_file.integrity.manifest import ( + MANIFEST_SCHEMA_VERSION, + dump_manifest, + from_manifest, + load_manifest, + to_manifest, +) +from automation_file.integrity.monitor import IntegrityMonitor, MonitorKeywords +from automation_file.integrity.remediation import RemediationPolicy, Remediator +from automation_file.integrity.report import DriftReport, RemediationStep +from automation_file.integrity.snapshot import Snapshot, SnapshotEntry, build_snapshot +from automation_file.integrity.target import Target +from automation_file.integrity.watcher import WatchHandle + +__all__ = [ + "DEFAULT_ALGORITHM", + "DEFAULT_SEVERITIES", + "MANIFEST_SCHEMA_VERSION", + "STRONG_ALGORITHMS", + "WEAK_ALGORITHMS", + "AlertEngine", + "AlertPolicy", + "BaselineManager", + "Change", + "ChangeKind", + "DriftReport", + "HashEngine", + "IntegrityException", + "IntegrityMonitor", + "IntegrityRemediated", + "MonitorKeywords", + "RemediationPolicy", + "RemediationStep", + "Remediator", + "Snapshot", + "SnapshotEntry", + "Target", + "WatchHandle", + "build_snapshot", + "detect_changes", + "dump_manifest", + "from_manifest", + "load_manifest", + "register_integrity_ops", + "to_manifest", +] diff --git a/automation_file/integrity/actions.py b/automation_file/integrity/actions.py new file mode 100644 index 0000000..bda95eb --- /dev/null +++ b/automation_file/integrity/actions.py @@ -0,0 +1,156 @@ +"""``FA_integrity_*`` actions: the integrity monitor for JSON action lists. + +Each function takes storage URIs as plain strings and returns JSON-friendly +values, so the same call works from Python, an action file, the CLI, the TCP +and HTTP action servers and as an MCP tool: + +.. code-block:: json + + [ + ["FA_integrity_baseline", {"target": "s3://reports/2026", + "baseline": "local:///var/lib/fa/reports.json"}], + ["FA_integrity_verify", {"target": "s3://reports/2026", + "baseline": "local:///var/lib/fa/reports.json"}] + ] + +``FA_integrity_watch_start`` keeps a named monitor verifying on a thread until +``FA_integrity_watch_stop``; ``FA_integrity_status`` shows what each one last +found. Drift is published on the process-wide event bus. None of the actions +remediates: a remediation policy can only be given in Python. +""" + +from __future__ import annotations + +import threading +from collections.abc import Callable +from typing import TYPE_CHECKING, Any + +from automation_file.integrity.errors import IntegrityException +from automation_file.integrity.hashing import DEFAULT_ALGORITHM +from automation_file.integrity.monitor import IntegrityMonitor +from automation_file.integrity.snapshot import Snapshot +from automation_file.logging_config import file_automation_logger + +if TYPE_CHECKING: + from automation_file.core.action_registry import ActionRegistry + +_DEFAULT_INTERVAL = 60.0 + +_monitors: dict[str, IntegrityMonitor] = {} +_monitors_lock = threading.Lock() + + +def _stored(snapshot: Snapshot, baseline: str | None) -> dict[str, Any]: + return { + "target": snapshot.root, + "baseline": baseline, + "backend": snapshot.backend, + "algorithm": snapshot.algorithm, + "created_at": snapshot.created_at.isoformat(), + "entries": len(snapshot), + } + + +def _status(name: str, monitor: IntegrityMonitor) -> dict[str, Any]: + return {"name": name, **monitor.status()} + + +def integrity_snapshot(target: str, algorithm: str = DEFAULT_ALGORITHM) -> dict[str, Any]: + """Return every file below ``target`` with its size, time and checksum. Stores nothing.""" + return IntegrityMonitor(target, algorithm=algorithm).snapshot().to_dict() + + +def integrity_baseline( + target: str, baseline: str, algorithm: str = DEFAULT_ALGORITHM +) -> dict[str, Any]: + """Snapshot ``target`` and store it at ``baseline`` as the approved state.""" + monitor = IntegrityMonitor(target, baseline, algorithm=algorithm) + return _stored(monitor.create_baseline(), monitor.baseline) + + +def integrity_verify(target: str, baseline: str, deep: bool = True) -> dict[str, Any]: + """Compare ``target`` with ``baseline`` and return the drift report. + + ``deep=False`` hashes only the files whose size, modification time or etag changed. + """ + return IntegrityMonitor(target, baseline).verify(deep=bool(deep)).to_dict() + + +def integrity_accept(target: str, baseline: str) -> dict[str, Any]: + """Approve the current state of ``target``: store it at ``baseline`` as the new baseline.""" + monitor = IntegrityMonitor(target, baseline) + return _stored(monitor.accept(), monitor.baseline) + + +def integrity_watch_start( + name: str, target: str, baseline: str, interval: float = _DEFAULT_INTERVAL +) -> dict[str, Any]: + """Start the monitor ``name``: verify ``target`` against ``baseline`` every ``interval`` s.""" + if not isinstance(name, str) or not name.strip(): + raise IntegrityException("an integrity monitor needs a name") + monitor = IntegrityMonitor(target, baseline, interval=interval) + if not monitor.has_baseline(): + raise IntegrityException( + f"no baseline at {monitor.baseline}; create it first with FA_integrity_baseline" + ) + with _monitors_lock: + if name in _monitors: + raise IntegrityException(f"an integrity monitor named {name!r} is already running") + monitor.start() + _monitors[name] = monitor + file_automation_logger.info("integrity: monitor %r started on %s", name, monitor.target) + return _status(name, monitor) + + +def integrity_watch_stop(name: str) -> dict[str, Any]: + """Stop the monitor ``name`` and return its last status.""" + with _monitors_lock: + monitor = _monitors.pop(name, None) + if monitor is None: + raise IntegrityException(f"no integrity monitor named {name!r}") + monitor.stop() + file_automation_logger.info("integrity: monitor %r stopped", name) + return _status(name, monitor) + + +def integrity_status(name: str | None = None) -> list[dict[str, Any]]: + """Return the status of the monitor ``name``, or of every named monitor. + + Each status has ``running``, ``last_run``, ``last_error`` and ``last_report``. + """ + with _monitors_lock: + if name is None: + chosen = dict(_monitors) + elif name in _monitors: + chosen = {name: _monitors[name]} + else: + raise IntegrityException(f"no integrity monitor named {name!r}") + return [_status(key, monitor) for key, monitor in chosen.items()] + + +def stop_all_monitors() -> list[dict[str, Any]]: + """Stop every named monitor, for an orderly shutdown; returns their last statuses.""" + with _monitors_lock: + stopped = dict(_monitors) + _monitors.clear() + for monitor in stopped.values(): + monitor.stop() + return [_status(name, monitor) for name, monitor in stopped.items()] + + +def integrity_commands() -> dict[str, Callable[..., Any]]: + """Return every ``FA_integrity_*`` action by name.""" + return { + "FA_integrity_snapshot": integrity_snapshot, + "FA_integrity_baseline": integrity_baseline, + "FA_integrity_verify": integrity_verify, + "FA_integrity_accept": integrity_accept, + "FA_integrity_watch_start": integrity_watch_start, + "FA_integrity_watch_stop": integrity_watch_stop, + "FA_integrity_status": integrity_status, + } + + +def register_integrity_ops(registry: ActionRegistry) -> None: + """Register every ``FA_integrity_*`` command into ``registry``.""" + registry.register_many(integrity_commands()) diff --git a/automation_file/integrity/alerts.py b/automation_file/integrity/alerts.py new file mode 100644 index 0000000..a6ce04c --- /dev/null +++ b/automation_file/integrity/alerts.py @@ -0,0 +1,187 @@ +"""Alert Engine: drift becomes events on the bus. + +The integrity subsystem never calls a notification sink or the audit log. It +publishes, and whoever cares subscribes: + +* one :class:`~automation_file.events.IntegrityViolation` per verification that + found drift, or that could not run at all; +* one :class:`IntegrityRemediated` per remediation step. + +Every event has ``source="integrity"`` and, in its payload, ``resource`` (the +target URI) and ``backend``. The severity of a violation is the worst severity +among the kinds of change it found; :class:`AlertPolicy` sets the severity of +each kind. +""" + +from __future__ import annotations + +from collections.abc import Iterable, Mapping +from dataclasses import dataclass, field +from typing import Any, ClassVar + +from automation_file.events.bus import EventBus, event_bus +from automation_file.events.model import Event, IntegrityViolation, Severity +from automation_file.integrity.detector import Change, ChangeKind +from automation_file.integrity.errors import IntegrityException +from automation_file.integrity.report import DriftReport, RemediationStep + +SOURCE = "integrity" +STATUS_DRIFT = "drift" +STATUS_ERROR = "error" +STATUS_OK = "ok" +STATUS_FAILED = "failed" +_ACTION_VERIFY = "verify" +_DEFAULT_MAX_CHANGES = 20 + +#: Additions and metadata are worth a look; anything that alters or removes what the +#: baseline holds is an error. +DEFAULT_SEVERITIES: Mapping[ChangeKind, Severity] = { + ChangeKind.CREATED: Severity.WARNING, + ChangeKind.METADATA_CHANGED: Severity.WARNING, + ChangeKind.MODIFIED: Severity.ERROR, + ChangeKind.DELETED: Severity.ERROR, + ChangeKind.RENAMED: Severity.ERROR, + ChangeKind.PERMISSION_CHANGED: Severity.ERROR, +} + + +@dataclass(frozen=True, kw_only=True) +class IntegrityRemediated(Event): + """A remediation step ran: a file was quarantined or restored, or the attempt failed.""" + + type: ClassVar[str] = "integrity.remediated" + + +@dataclass(frozen=True) +class AlertPolicy: + """How loud each kind of change is, and how many changes an event lists. + + ``severities`` overrides :data:`DEFAULT_SEVERITIES` kind by kind; keys and + values may be the enum members or their names (``{"created": "error"}``). + """ + + severities: Mapping[Any, Any] = field(default_factory=dict, hash=False) + max_changes: int = _DEFAULT_MAX_CHANGES + + def __post_init__(self) -> None: + if self.max_changes < 0: + raise IntegrityException("max_changes must not be negative") + table = dict(DEFAULT_SEVERITIES) + try: + table.update( + (ChangeKind(kind), Severity(severity)) for kind, severity in self.severities.items() + ) + except ValueError as error: + raise IntegrityException(f"invalid alert severity setting: {error}") from error + object.__setattr__(self, "severities", table) + + def severity_of(self, kind: ChangeKind) -> Severity: + """Return the severity of one kind of change.""" + return Severity(self.severities[kind]) + + def severity_for(self, changes: Iterable[Change]) -> Severity: + """Return the worst severity among ``changes`` (``info`` when there are none).""" + worst = Severity.INFO + for change in changes: + severity = self.severity_of(change.kind) + if severity.rank > worst.rank: + worst = severity + return worst + + +def _described(counts: Mapping[str, int]) -> str: + return ", ".join(f"{count} {kind}" for kind, count in counts.items() if count) + + +def _payload( + action: str, resource: str, backend: str, status: str, **details: Any +) -> dict[str, Any]: + """Return a payload that opens with the four keys every integrity event carries.""" + return { + "action": action, + "resource": resource, + "backend": backend, + "status": status, + **details, + } + + +class AlertEngine: + """Publishes what a monitor finds on one :class:`~automation_file.events.EventBus`.""" + + def __init__(self, bus: EventBus | None = None, policy: AlertPolicy | None = None) -> None: + self._bus = bus if bus is not None else event_bus + self._policy = policy if policy is not None else AlertPolicy() + + def violation(self, report: DriftReport) -> IntegrityViolation | None: + """Publish the drift ``report`` found; a clean report publishes nothing.""" + if report.ok: + return None + limit = self._policy.max_changes + payload = _payload( + _ACTION_VERIFY, + report.target, + report.backend, + STATUS_DRIFT, + baseline=report.baseline, + algorithm=report.algorithm, + deep=report.deep, + partial=report.partial, + counts=report.counts, + total=len(report.changes), + changes=[change.brief() for change in report.changes[:limit]], + truncated=len(report.changes) > limit, + ) + if report.remediation: + succeeded = sum(1 for step in report.remediation if step.ok) + payload["remediation"] = { + STATUS_OK: succeeded, + STATUS_FAILED: len(report.remediation) - succeeded, + } + event = IntegrityViolation( + source=SOURCE, + subject=f"integrity drift: {report.target} ({_described(report.counts)})", + severity=self._policy.severity_for(report.changes), + payload=payload, + ) + self._bus.publish(event) + return event + + def failure(self, resource: str, backend: str, error: BaseException) -> IntegrityViolation: + """Publish that ``resource`` could not be verified at all.""" + event = IntegrityViolation( + source=SOURCE, + subject=f"integrity verification failed: {resource}", + severity=Severity.ERROR, + payload=_payload( + _ACTION_VERIFY, + resource, + backend, + STATUS_ERROR, + error=f"{type(error).__name__}: {error}", + ), + ) + self._bus.publish(event) + return event + + def remediated(self, step: RemediationStep, resource: str, backend: str) -> IntegrityRemediated: + """Publish one remediation step; ``resource`` is the URI of the file it concerned.""" + outcome = "done" if step.ok else "failed" + event = IntegrityRemediated( + source=SOURCE, + subject=f"integrity {step.action} {outcome}: {resource}", + severity=Severity.INFO if step.ok else Severity.ERROR, + payload=_payload( + step.action, + resource, + backend, + STATUS_OK if step.ok else STATUS_FAILED, + path=step.path, + kind=step.kind, + source=step.source, + destination=step.destination, + error=step.error, + ), + ) + self._bus.publish(event) + return event diff --git a/automation_file/integrity/baseline.py b/automation_file/integrity/baseline.py new file mode 100644 index 0000000..79b8457 --- /dev/null +++ b/automation_file/integrity/baseline.py @@ -0,0 +1,113 @@ +"""Baseline Manager: where the approved state of a tree is kept. + +A baseline is one manifest file, addressed by a storage URI, so it can live in +another backend than the tree it describes. Keeping it elsewhere is the point: +whoever can change the tree should not be able to change what it is compared +with. + +``save`` writes the manifest to a sibling temporary file and moves it over the +baseline, so a reader never sees a half-written document. The move is a rename +where the backend has one (the local filesystem); elsewhere it is one whole +write of the finished file. +""" + +from __future__ import annotations + +import uuid + +from automation_file.exceptions import ( + StorageException, + StorageNotFoundException, + StoragePathTypeException, +) +from automation_file.integrity.errors import IntegrityException +from automation_file.integrity.manifest import dump_manifest, load_manifest +from automation_file.integrity.snapshot import Snapshot +from automation_file.logging_config import file_automation_logger +from automation_file.storage.file import File +from automation_file.storage.resolver import StorageResolver +from automation_file.storage.uri import StorageURI, URILike, parse_storage_uri + +_TEMPORARY_SUFFIX = ".tmp" + + +def _temporary_prefix(name: str) -> str: + return f".{name}." + + +def is_baseline_path(candidate: str, baseline: str) -> bool: + """Say whether ``candidate`` is the baseline file ``baseline`` or a temporary sibling of it. + + Both are paths inside the same tree. A monitor uses it to leave its own + baseline out of the snapshots when the baseline is kept inside the target. + """ + if candidate == baseline: + return True + directory, _, name = baseline.rpartition("/") + parent, _, leaf = candidate.rpartition("/") + return ( + parent == directory + and leaf.startswith(_temporary_prefix(name)) + and leaf.endswith(_TEMPORARY_SUFFIX) + ) + + +class BaselineManager: + """Reads and writes the baseline manifest at one storage URI.""" + + def __init__(self, location: URILike, *, resolver: StorageResolver | None = None) -> None: + self._uri = parse_storage_uri(location) + if not self._uri.name: + raise IntegrityException(f"a baseline must be a file, not the storage root {self._uri}") + self._resolver = resolver + self._file = File(self._uri, resolver=resolver) + + @property + def uri(self) -> StorageURI: + return self._uri + + def exists(self) -> bool: + """Return whether a baseline file is stored.""" + return self._file.is_file() + + def load(self) -> Snapshot: + """Return the stored baseline; the legacy manifest format is converted on the way.""" + try: + data = self._file.read() + except StorageNotFoundException as error: + raise IntegrityException( + f"no baseline at {self._uri}; create one with create_baseline()" + ) from error + except StoragePathTypeException as error: + raise IntegrityException(f"baseline {self._uri} is a directory") from error + return load_manifest(data, origin=f"baseline {self._uri}") + + def save(self, snapshot: Snapshot) -> None: + """Store ``snapshot`` as the baseline, replacing the previous one in one step.""" + temporary = File( + self._uri.parent.joinpath( + f"{_temporary_prefix(self._uri.name)}{uuid.uuid4().hex}{_TEMPORARY_SUFFIX}" + ), + resolver=self._resolver, + ) + try: + temporary.write(dump_manifest(snapshot)) + temporary.move_to(self._file) + finally: + _discard(temporary) + file_automation_logger.info( + "integrity: baseline %s written (%d files, %s)", + self._uri, + len(snapshot), + snapshot.algorithm, + ) + + +def _discard(temporary: File) -> None: + """Remove what a failed ``save`` left behind; after a successful one nothing is there.""" + try: + temporary.delete(missing_ok=True) + except StorageException as error: + file_automation_logger.warning( + "integrity: could not remove the temporary baseline %s: %r", temporary, error + ) diff --git a/automation_file/integrity/detector.py b/automation_file/integrity/detector.py new file mode 100644 index 0000000..b94ac43 --- /dev/null +++ b/automation_file/integrity/detector.py @@ -0,0 +1,196 @@ +"""Change Detector: what differs between two snapshots of one tree. + +Six kinds of change are told apart: + +``created`` + A path the baseline does not have. +``modified`` + The checksum (or the recorded size) differs. +``deleted`` + A path of the baseline that is gone. +``renamed`` + One deleted and one created file with the same checksum and size. When + several deleted or several created files share that content the pairing is + ambiguous: they are reported as ``deleted`` and ``created`` with a note + that says so. +``metadata_changed`` + Same checksum, but the modification time, content type, version or etag + differs. A field one side did not record is not compared. +``permission_changed`` + The permission bits differ. It is reported next to ``modified`` when both + happened. +""" + +from __future__ import annotations + +from collections.abc import Iterable +from dataclasses import dataclass +from enum import Enum +from typing import Any + +from automation_file.integrity.errors import IntegrityException +from automation_file.integrity.snapshot import Snapshot, SnapshotEntry + +_METADATA_FIELDS = ("modified_at", "content_type", "version", "etag") +_DIGEST_PREVIEW = 12 +_ContentKey = tuple[str, int | None, str] + + +class ChangeKind(str, Enum): + """The kinds of drift the detector reports.""" + + CREATED = "created" + MODIFIED = "modified" + DELETED = "deleted" + RENAMED = "renamed" + METADATA_CHANGED = "metadata_changed" + PERMISSION_CHANGED = "permission_changed" + + +_KIND_ORDER = tuple(ChangeKind) + + +@dataclass(frozen=True) +class Change: + """One difference. ``before`` is the baseline's entry, ``after`` the current one. + + ``previous_path`` is set for a rename, ``fields`` names the metadata that + differs, and ``note`` carries a remark such as an ambiguous rename. + """ + + kind: ChangeKind + path: str + previous_path: str | None = None + before: SnapshotEntry | None = None + after: SnapshotEntry | None = None + fields: tuple[str, ...] = () + note: str = "" + + def brief(self) -> dict[str, Any]: + """Return the change without its two entries, for an event payload or a log line.""" + summary: dict[str, Any] = {"kind": self.kind.value, "path": self.path} + if self.previous_path is not None: + summary["previous_path"] = self.previous_path + if self.fields: + summary["fields"] = list(self.fields) + if self.note: + summary["note"] = self.note + return summary + + def to_dict(self) -> dict[str, Any]: + """Return a JSON-serialisable mapping of the change with both entries.""" + return { + "kind": self.kind.value, + "path": self.path, + "previous_path": self.previous_path, + "fields": list(self.fields), + "note": self.note, + "before": self.before.to_dict() if self.before else None, + "after": self.after.to_dict() if self.after else None, + } + + +def _content_differs(before: SnapshotEntry, after: SnapshotEntry) -> bool: + if before.checksum != after.checksum: + return True + return before.size is not None and after.size is not None and before.size != after.size + + +def _metadata_differences(before: SnapshotEntry, after: SnapshotEntry) -> tuple[str, ...]: + differing: list[str] = [] + for name in _METADATA_FIELDS: + old, new = getattr(before, name), getattr(after, name) + if old is not None and new is not None and old != new: + differing.append(name) + return tuple(differing) + + +def _compare(before: SnapshotEntry, after: SnapshotEntry) -> list[Change]: + """Return what changed in a file that both snapshots hold.""" + changes: list[Change] = [] + if _content_differs(before, after): + changes.append(Change(ChangeKind.MODIFIED, after.path, before=before, after=after)) + else: + fields = _metadata_differences(before, after) + if fields: + changes.append( + Change( + ChangeKind.METADATA_CHANGED, + after.path, + before=before, + after=after, + fields=fields, + ) + ) + if before.mode is not None and after.mode is not None and before.mode != after.mode: + changes.append( + Change( + ChangeKind.PERMISSION_CHANGED, + after.path, + before=before, + after=after, + fields=("mode",), + ) + ) + return changes + + +def _content_key(entry: SnapshotEntry) -> _ContentKey: + # An entry without a checksum can be paired with nothing, so its key is its own. + return entry.checksum, entry.size, "" if entry.checksum else entry.path + + +def _by_content(entries: Iterable[SnapshotEntry]) -> dict[_ContentKey, list[SnapshotEntry]]: + grouped: dict[_ContentKey, list[SnapshotEntry]] = {} + for entry in entries: + grouped.setdefault(_content_key(entry), []).append(entry) + return grouped + + +def _pair(gone: list[SnapshotEntry], new: list[SnapshotEntry]) -> list[Change]: + """Turn files of one content that left and arrived into a rename, or into its two halves.""" + if len(gone) == 1 and len(new) == 1: + return [ + Change( + ChangeKind.RENAMED, + new[0].path, + previous_path=gone[0].path, + before=gone[0], + after=new[0], + ) + ] + note = "" + if gone and new: + note = ( + f"ambiguous rename: {len(gone)} deleted and {len(new)} created files share the " + f"checksum {gone[0].checksum[:_DIGEST_PREVIEW]}...; reported separately" + ) + return [ + *(Change(ChangeKind.DELETED, entry.path, before=entry, note=note) for entry in gone), + *(Change(ChangeKind.CREATED, entry.path, after=entry, note=note) for entry in new), + ] + + +def _order(change: Change) -> tuple[str, int]: + return change.path, _KIND_ORDER.index(change.kind) + + +def detect_changes(baseline: Snapshot, current: Snapshot) -> list[Change]: + """Return the changes that turn ``baseline`` into ``current``, sorted by path.""" + if baseline.algorithm != current.algorithm: + raise IntegrityException( + f"cannot compare a {baseline.algorithm} snapshot with a {current.algorithm} one: " + "hash the current tree with the baseline's algorithm" + ) + changes: list[Change] = [] + for before in baseline: + after = current.get(before.path) + if after is not None: + changes.extend(_compare(before, after)) + gone = _by_content(entry for entry in baseline if entry.path not in current) + new = _by_content(entry for entry in current if entry.path not in baseline) + for key, entries in gone.items(): + changes.extend(_pair(entries, new.pop(key, []))) + for entries in new.values(): + changes.extend(_pair([], entries)) + return sorted(changes, key=_order) diff --git a/automation_file/integrity/errors.py b/automation_file/integrity/errors.py new file mode 100644 index 0000000..4feed4a --- /dev/null +++ b/automation_file/integrity/errors.py @@ -0,0 +1,9 @@ +"""Errors of the integrity subsystem.""" + +from __future__ import annotations + +from automation_file.exceptions import FileAutomationException + + +class IntegrityException(FileAutomationException): + """Raised when a snapshot, a baseline or a verification cannot be produced or trusted.""" diff --git a/automation_file/integrity/hashing.py b/automation_file/integrity/hashing.py new file mode 100644 index 0000000..ff39a18 --- /dev/null +++ b/automation_file/integrity/hashing.py @@ -0,0 +1,97 @@ +"""Hash Engine: which digests may prove integrity, and hashing many files at once. + +``sha256`` is the default; ``sha512`` and ``blake2b`` are the alternatives. ``md5`` +and ``sha1`` are refused unless the caller passes ``allow_weak=True``: both have +practical collisions, so a file can be replaced by another with the same digest. +They exist only to keep reading a baseline that was written with one of them. + +Files are hashed through the storage layer's ``checksum``, several at a time on +a thread pool, so the engine works on any backend. +""" + +from __future__ import annotations + +from collections.abc import Callable, Iterable +from concurrent.futures import ThreadPoolExecutor +from typing import TypeVar + +from automation_file.exceptions import FileNotExistsException, StoragePathTypeException +from automation_file.integrity.errors import IntegrityException +from automation_file.storage.storage import Storage + +DEFAULT_ALGORITHM = "sha256" +STRONG_ALGORITHMS = ("sha256", "sha512", "blake2b") +WEAK_ALGORITHMS = ("md5", "sha1") +_DEFAULT_WORKERS = 8 + +_ResultT = TypeVar("_ResultT") + + +def checked_algorithm(algorithm: str, *, allow_weak: bool = False) -> str: + """Return the lower-case name of ``algorithm``, or raise when it may not be used.""" + name = str(algorithm).strip().lower() + if name in STRONG_ALGORITHMS: + return name + choices = ", ".join(STRONG_ALGORITHMS) + if name not in WEAK_ALGORITHMS: + raise IntegrityException( + f"unsupported integrity algorithm {algorithm!r}: choose {choices} " + f"({', '.join(WEAK_ALGORITHMS)} only with allow_weak=True)" + ) + if not allow_weak: + raise IntegrityException( + f"{name} is refused for integrity checks: it is not collision-resistant, so a file " + f"can be replaced by another with the same digest and the change goes unnoticed. " + f"Use {choices}; pass allow_weak=True only to keep reading a baseline that was " + f"written with {name}" + ) + return name + + +class HashEngine: + """Hashes files of one :class:`~automation_file.storage.Storage` with one algorithm.""" + + def __init__( + self, + algorithm: str = DEFAULT_ALGORITHM, + *, + allow_weak: bool = False, + max_workers: int | None = None, + ) -> None: + self._algorithm = checked_algorithm(algorithm, allow_weak=allow_weak) + if max_workers is not None and max_workers < 1: + raise IntegrityException("max_workers must be at least 1") + self._max_workers = max_workers or _DEFAULT_WORKERS + + @property + def algorithm(self) -> str: + return self._algorithm + + def hash_file(self, storage: Storage, path: str) -> str | None: + """Return the hex digest of ``path``, or ``None`` when the file is no longer there.""" + try: + return storage.checksum(path, self._algorithm).value + except (FileNotExistsException, StoragePathTypeException): + return None + + def hash_many(self, storage: Storage, paths: Iterable[str]) -> dict[str, str]: + """Return the digest of every path; a file that vanished meanwhile is left out.""" + digests = self.map(lambda path: self.hash_file(storage, path), paths) + return {path: digest for path, digest in digests.items() if digest is not None} + + def map(self, work: Callable[[str], _ResultT], paths: Iterable[str]) -> dict[str, _ResultT]: + """Call ``work(path)`` for every path on the thread pool and return the results by path. + + The first failure is raised and the calls that have not started are dropped. + """ + todo = list(dict.fromkeys(paths)) + if len(todo) <= 1 or self._max_workers == 1: + return {path: work(path) for path in todo} + workers = min(self._max_workers, len(todo)) + with ThreadPoolExecutor(max_workers=workers, thread_name_prefix="fa-integrity") as pool: + futures = {path: pool.submit(work, path) for path in todo} + try: + return {path: future.result() for path, future in futures.items()} + finally: + for future in futures.values(): + future.cancel() diff --git a/automation_file/integrity/legacy.py b/automation_file/integrity/legacy.py new file mode 100644 index 0000000..2335577 --- /dev/null +++ b/automation_file/integrity/legacy.py @@ -0,0 +1,146 @@ +"""What the first ``IntegrityMonitor`` (``automation_file.core.fim``) gave its callers. + +``check_once()`` returned a summary dictionary -- ``matched``, ``missing``, +``modified``, ``extra``, ``ok``, and ``error`` when the pass failed -- handed +the same dictionary to ``on_drift`` and sent one notification. Those three +things are kept here so code written for that monitor keeps working. New code +reads the :class:`~automation_file.integrity.report.DriftReport` and subscribes +to the event bus instead. + +Additions do not count as drift in this summary unless ``alert_on_extra`` is +set, and a rename appears as its two halves, as it always did. +""" + +from __future__ import annotations + +from collections.abc import Callable, Iterable +from typing import TYPE_CHECKING, Any + +from automation_file.exceptions import FileAutomationException +from automation_file.integrity.detector import ChangeKind +from automation_file.integrity.report import DriftReport +from automation_file.logging_config import file_automation_logger + +if TYPE_CHECKING: + from automation_file.notify.manager import NotificationManager + +OnDrift = Callable[[dict[str, Any]], None] + +_MISSING = "missing" +_MODIFIED = "modified" +_EXTRA = "extra" +_ERROR = "error" +_PREVIEW = 5 +_NOTIFICATION_LEVEL = "error" + + +def summary_of(report: DriftReport, examined: Iterable[str]) -> dict[str, Any]: + """Return the legacy summary of ``report``; ``examined`` are the baseline paths it covered.""" + missing: list[str] = [] + modified: list[str] = [] + extra: list[str] = [] + for change in report.changes: + if change.kind is ChangeKind.DELETED: + missing.append(change.path) + elif change.kind is ChangeKind.MODIFIED: + modified.append(change.path) + elif change.kind is ChangeKind.CREATED: + extra.append(change.path) + elif change.kind is ChangeKind.RENAMED and change.previous_path is not None: + missing.append(change.previous_path) + extra.append(change.path) + unmatched = {*missing, *modified} + return { + "matched": [path for path in examined if path not in unmatched], + _MISSING: sorted(missing), + _MODIFIED: sorted(modified), + _EXTRA: sorted(extra), + "ok": not missing and not modified, + } + + +def error_summary(error: BaseException) -> dict[str, Any]: + """Return the legacy summary of a pass that could not run.""" + return { + "matched": [], + _MISSING: [], + _MODIFIED: [], + _EXTRA: [], + "ok": False, + _ERROR: repr(error), + } + + +def format_body(summary: dict[str, Any]) -> str: + """Return the text of the legacy notification: the first few paths of each list.""" + parts: list[str] = [] + if summary.get(_ERROR): + parts.append(f"error: {summary[_ERROR]}") + for key in (_MISSING, _MODIFIED, _EXTRA): + items = summary.get(key) or [] + if items: + preview = ", ".join(items[:_PREVIEW]) + suffix = f" (+{len(items) - _PREVIEW} more)" if len(items) > _PREVIEW else "" + parts.append(f"{key}: {preview}{suffix}") + return "\n".join(parts) if parts else "no drift detected" + + +def _process_wide_manager() -> NotificationManager: + """Return the shared manager, looked up when it is needed so a test may replace it.""" + from automation_file import notify + + return notify.notification_manager + + +class LegacyHooks: + """The ``on_drift`` callback and the notification of the first monitor. + + The notification goes through the manager the caller passed, or through the + process-wide ``notification_manager`` when none was: what the first monitor + did. ``notify=False`` sends none. + """ + + def __init__( + self, + subject: str, + *, + on_drift: OnDrift | None = None, + manager: NotificationManager | None = None, + alert_on_extra: bool = False, + notify: bool = True, + ) -> None: + self._subject = subject + self._on_drift = on_drift + self._manager = manager + self._notifies = bool(notify) + self._alert_on_extra = bool(alert_on_extra) + + def is_drift(self, summary: dict[str, Any]) -> bool: + """Say whether ``summary`` counts as drift by the first monitor's rule.""" + if summary.get(_ERROR) or summary.get(_MISSING) or summary.get(_MODIFIED): + return True + return bool(self._alert_on_extra and summary.get(_EXTRA)) + + def handle(self, summary: dict[str, Any]) -> None: + """Call ``on_drift`` and notify the manager when ``summary`` shows drift.""" + if not self.is_drift(summary): + return + if self._on_drift is not None: + self._call_back(self._on_drift, summary) + if self._notifies: + self._notify(self._manager or _process_wide_manager(), summary) + + def _call_back(self, on_drift: OnDrift, summary: dict[str, Any]) -> None: + try: + on_drift(summary) + except Exception as error: # pylint: disable=broad-except + # Boundary: a caller's callback must not stop the monitor that called it. + file_automation_logger.error("integrity: on_drift raised: %r", error) + + def _notify(self, manager: NotificationManager, summary: dict[str, Any]) -> None: + try: + manager.notify( + subject=self._subject, body=format_body(summary), level=_NOTIFICATION_LEVEL + ) + except FileAutomationException as error: + file_automation_logger.error("integrity: notify failed: %r", error) diff --git a/automation_file/integrity/local_watcher.py b/automation_file/integrity/local_watcher.py new file mode 100644 index 0000000..cc314d9 --- /dev/null +++ b/automation_file/integrity/local_watcher.py @@ -0,0 +1,112 @@ +"""Watcher for a local directory: filesystem events, debounced into batches of paths. + +Imported only when a local target is watched, so the rest of the integrity +package does not load watchdog. + +An observer reports what the operating system tells it, and an overflowing +event queue drops events without saying so. Watching narrows the time to +detection; a periodic full verification is still what proves the tree. +""" + +from __future__ import annotations + +import os +from collections.abc import Callable +from pathlib import Path + +from watchdog.events import FileSystemEvent, FileSystemEventHandler +from watchdog.observers import Observer +from watchdog.observers.api import BaseObserver + +from automation_file.integrity.errors import IntegrityException +from automation_file.integrity.watcher import Debouncer, WatchHandle +from automation_file.logging_config import file_automation_logger + +_CHANGE_EVENTS = frozenset({"created", "modified", "deleted", "moved"}) +_DIRECTORY_NOISE = "modified" +_DEFAULT_JOIN_TIMEOUT = 5.0 + + +class PathCollector(FileSystemEventHandler): + """Turns watchdog events into paths relative to the watched directory. + + Only ``created``, ``modified``, ``deleted`` and ``moved`` count; a move + reports both of its paths. A path outside ``root`` is ignored. + """ + + def __init__(self, root: Path, report: Callable[[str], None]) -> None: + super().__init__() + self._root = root + self._report = report + + def on_any_event(self, event: FileSystemEvent) -> None: + if event.event_type not in _CHANGE_EVENTS: + return + # A directory is "modified" whenever an entry in it changes; that entry reports itself. + if event.is_directory and event.event_type == _DIRECTORY_NOISE: + return + # Before watchdog 5 only a move carries a destination. + for raw in (event.src_path, getattr(event, "dest_path", "")): + relative = self._relative(raw) + if relative is not None: + self._report(relative) + + def _relative(self, raw: bytes | str) -> str | None: + if not raw: + return None + try: + relative = Path(os.fsdecode(raw)).relative_to(self._root).as_posix() + except ValueError: + return None + return "" if relative == "." else relative + + +class LocalWatcher(WatchHandle): + """Watches ``root`` recursively and calls ``on_paths`` with each batch of changed paths.""" + + kind = "events" + + def __init__( + self, + root: Path, + on_paths: Callable[[frozenset[str]], object], + *, + debounce: float = 0.5, + ignore: Callable[[str], bool] | None = None, + ) -> None: + if not root.is_dir(): + raise IntegrityException(f"cannot watch {root}: it is not a directory") + self._root = root + self._ignore = ignore + self._debouncer = Debouncer(on_paths, debounce) + self._observer: BaseObserver | None = None + + @property + def is_running(self) -> bool: + observer = self._observer + return observer is not None and observer.is_alive() + + def start(self) -> None: + if self.is_running: + return + self._debouncer.start() + observer = Observer() + observer.schedule(PathCollector(self._root, self.feed), str(self._root), recursive=True) + observer.daemon = True + observer.start() + self._observer = observer + file_automation_logger.info("integrity: watching %s for changes", self._root) + + def feed(self, path: str) -> None: + """Report that ``path``, relative to the root, changed. The observer calls this.""" + if self._ignore is not None and self._ignore(path): + return + self._debouncer.add(path) + + def stop(self, timeout: float = _DEFAULT_JOIN_TIMEOUT) -> None: + observer, self._observer = self._observer, None + if observer is not None: + observer.stop() + observer.join(timeout=timeout) + self._debouncer.stop(timeout) + file_automation_logger.info("integrity: stopped watching %s", self._root) diff --git a/automation_file/integrity/manifest.py b/automation_file/integrity/manifest.py new file mode 100644 index 0000000..e9d0b18 --- /dev/null +++ b/automation_file/integrity/manifest.py @@ -0,0 +1,138 @@ +"""Manifest: the JSON document a baseline is stored as. + +Schema version 2:: + + { + "schema_version": 2, + "created_at": "2026-10-08T10:15:30.123456+00:00", + "root": "s3://reports/2026", + "backend": "s3", + "algorithm": "sha256", + "entries": [ + {"path": "q1.csv", "size": 1024, "modified_at": "2026-10-01T08:00:00+00:00", + "checksum": "9f86d0...", "algorithm": "sha256", "content_type": "text/csv", + "backend": "s3", "version": null, "etag": "5d41402a...", "mode": null} + ] + } + +The format written by :func:`automation_file.core.manifest.write_manifest` (no +``schema_version`` key, a ``files`` mapping with ``size`` and ``checksum``) is +read as well and converted. Any other version is refused, so a newer document +is never half-understood. +""" + +from __future__ import annotations + +import json +from collections.abc import Mapping +from datetime import datetime, timezone +from typing import Any + +from automation_file.exceptions import StorageURIException +from automation_file.integrity.errors import IntegrityException +from automation_file.integrity.hashing import DEFAULT_ALGORITHM +from automation_file.integrity.snapshot import Snapshot, SnapshotEntry, parse_timestamp +from automation_file.storage.uri import LOCAL_SCHEME, local_path_to_uri + +MANIFEST_SCHEMA_VERSION = 2 +LEGACY_MANIFEST_VERSION = 1 +_SCHEMA_KEY = "schema_version" +_LEGACY_FILES_KEY = "files" +_LEGACY_VERSION_KEY = "version" +_ENCODING = "utf-8" + + +def to_manifest(snapshot: Snapshot) -> dict[str, Any]: + """Return the schema-version-2 document of ``snapshot``.""" + return {_SCHEMA_KEY: MANIFEST_SCHEMA_VERSION, **snapshot.to_dict()} + + +def from_manifest(document: Any, *, origin: str = "manifest") -> Snapshot: + """Return the snapshot a manifest document describes. + + ``origin`` names the document in error messages. Raises + :class:`IntegrityException` for a document that is not a manifest or whose + version this release does not read. + """ + if not isinstance(document, Mapping): + raise IntegrityException(f"{origin} is not a manifest: expected a JSON object") + if _SCHEMA_KEY not in document: + return _from_legacy(document, origin) + version = document[_SCHEMA_KEY] + if isinstance(version, bool) or version != MANIFEST_SCHEMA_VERSION: + raise IntegrityException( + f"{origin} has manifest schema version {version!r}; this release reads version " + f"{MANIFEST_SCHEMA_VERSION} and the legacy format without a version" + ) + try: + return Snapshot.from_dict(document) + except IntegrityException as error: + raise IntegrityException(f"{origin} is not a valid manifest: {error}") from error + + +def dump_manifest(snapshot: Snapshot) -> bytes: + """Return ``snapshot`` as an encoded schema-version-2 JSON document.""" + return json.dumps(to_manifest(snapshot), indent=2, ensure_ascii=False).encode(_ENCODING) + + +def load_manifest(data: bytes | str, *, origin: str = "manifest") -> Snapshot: + """Return the snapshot held by the JSON text ``data`` (version 2 or legacy).""" + try: + text = data.decode(_ENCODING) if isinstance(data, bytes) else data + document = json.loads(text) + except (UnicodeDecodeError, json.JSONDecodeError) as error: + raise IntegrityException(f"{origin} is not readable JSON: {error}") from error + return from_manifest(document, origin=origin) + + +def _legacy_root(value: object) -> str: + """Return the storage URI of the directory a legacy manifest was written for.""" + if not isinstance(value, str) or not value.strip(): + return "" + try: + return str(local_path_to_uri(value)) + except StorageURIException: + return "" + + +def _legacy_entry(path: object, meta: object, algorithm: str) -> SnapshotEntry: + details = meta if isinstance(meta, Mapping) else {} + size = details.get("size") + checksum = details.get("checksum") + return SnapshotEntry.from_dict( + { + "path": path, + "size": size if isinstance(size, int) and not isinstance(size, bool) else None, + "checksum": checksum if isinstance(checksum, str) else "", + }, + algorithm=algorithm, + backend=LOCAL_SCHEME, + ) + + +def _from_legacy(document: Mapping[str, Any], origin: str) -> Snapshot: + files = document.get(_LEGACY_FILES_KEY) + if not isinstance(files, Mapping): + raise IntegrityException( + f"{origin} is not a manifest: it has neither {_SCHEMA_KEY!r} nor a legacy " + f"{_LEGACY_FILES_KEY!r} mapping" + ) + version = document.get(_LEGACY_VERSION_KEY, LEGACY_MANIFEST_VERSION) + if isinstance(version, bool) or version != LEGACY_MANIFEST_VERSION: + raise IntegrityException( + f"{origin} is a legacy manifest of version {version!r}; only version " + f"{LEGACY_MANIFEST_VERSION} is known" + ) + algorithm = document.get("algorithm") + name = algorithm if isinstance(algorithm, str) and algorithm else DEFAULT_ALGORITHM + created_at = parse_timestamp(document.get("created_at"), f"{origin} 'created_at'") + try: + return Snapshot( + root=_legacy_root(document.get("root")), + backend=LOCAL_SCHEME, + algorithm=name, + created_at=created_at or datetime.now(timezone.utc), + entries=tuple(_legacy_entry(path, meta, name) for path, meta in files.items()), + ) + except IntegrityException as error: + raise IntegrityException(f"{origin} is not a valid legacy manifest: {error}") from error diff --git a/automation_file/integrity/monitor.py b/automation_file/integrity/monitor.py new file mode 100644 index 0000000..951e5dc --- /dev/null +++ b/automation_file/integrity/monitor.py @@ -0,0 +1,573 @@ +"""IntegrityMonitor: is this tree still what was approved? + +.. code-block:: python + + from automation_file.integrity import IntegrityMonitor + + monitor = IntegrityMonitor( + "s3://reports/2026", + baseline="local:///var/lib/fa/reports.baseline.json", + ) + monitor.create_baseline() # approve what is there now + report = monitor.verify() # a DriftReport; drift is published as an event + monitor.accept(report) # approve what the report saw + +Four modes share one comparison: + +``snapshot`` + :meth:`IntegrityMonitor.snapshot` reads the tree and stores nothing. +``verify`` + :meth:`IntegrityMonitor.verify` compares the tree with the baseline once. +``watch`` + :meth:`IntegrityMonitor.watch` reacts to changes: filesystem events for a + local target, a quick re-read on a timer for any other backend. +``continuous`` + :meth:`IntegrityMonitor.start` verifies every ``interval`` seconds on a + thread until :meth:`IntegrityMonitor.stop`. + +The monitor only reads, unless a +:class:`~automation_file.integrity.remediation.RemediationPolicy` is passed. +""" + +from __future__ import annotations + +import threading +from collections.abc import Callable, Iterable, Mapping +from dataclasses import dataclass, fields +from datetime import datetime, timezone +from typing import TYPE_CHECKING, Any, TypedDict + +from automation_file.events.bus import EventBus +from automation_file.events.context import correlation_scope +from automation_file.exceptions import FileAutomationException +from automation_file.integrity.alerts import AlertEngine, AlertPolicy +from automation_file.integrity.baseline import BaselineManager, is_baseline_path +from automation_file.integrity.detector import detect_changes +from automation_file.integrity.errors import IntegrityException +from automation_file.integrity.hashing import DEFAULT_ALGORITHM, HashEngine +from automation_file.integrity.legacy import LegacyHooks, OnDrift, error_summary, summary_of +from automation_file.integrity.remediation import RemediationPolicy, Remediator +from automation_file.integrity.report import DriftReport, RemediationStep +from automation_file.integrity.snapshot import ( + Snapshot, + SnapshotEntry, + build_snapshot, + quick_matches, +) +from automation_file.integrity.target import Target +from automation_file.integrity.watcher import IntervalRunner, PollingWatcher, WatchHandle +from automation_file.logging_config import file_automation_logger +from automation_file.storage.resolver import StorageResolver +from automation_file.storage.types import FileInfo +from automation_file.storage.uri import URILike, normalize_path + +if TYPE_CHECKING: + from typing_extensions import Unpack + + from automation_file.notify.manager import NotificationManager + +_DEFAULT_INTERVAL = 60.0 +_DEFAULT_DEBOUNCE = 0.5 +_DEFAULT_JOIN_TIMEOUT = 5.0 +_Signature = tuple[tuple[str, ...], ...] + + +def _named( + value: URILike | None, legacy: URILike | None, name: str, legacy_name: str +) -> URILike | None: + """Return the argument given under its name or under the first monitor's name for it.""" + if value is not None and legacy is not None: + raise IntegrityException(f"pass {name}= or its older name {legacy_name}=, not both") + return value if value is not None else legacy + + +class MonitorKeywords(TypedDict, total=False): + """The keyword arguments of :class:`IntegrityMonitor`; each one is optional.""" + + algorithm: str + interval: float + allow_weak: bool + max_workers: int | None + alerts: AlertPolicy | None + remediation: RemediationPolicy | None + bus: EventBus | None + resolver: StorageResolver | None + on_drift: OnDrift | None + manager: NotificationManager | None + notify: bool + alert_on_extra: bool + root: URILike | None + manifest_path: URILike | None + + +@dataclass(frozen=True) +class _Settings: + """The keyword arguments one monitor was given, with the default of each.""" + + algorithm: str = DEFAULT_ALGORITHM + interval: float = _DEFAULT_INTERVAL + allow_weak: bool = False + max_workers: int | None = None + alerts: AlertPolicy | None = None + remediation: RemediationPolicy | None = None + bus: EventBus | None = None + resolver: StorageResolver | None = None + on_drift: OnDrift | None = None + manager: NotificationManager | None = None + notify: bool = True + alert_on_extra: bool = False + root: URILike | None = None + manifest_path: URILike | None = None + + +def _settings(keywords: Mapping[str, Any]) -> _Settings: + """Return the settings ``keywords`` describe, refusing a name the monitor does not take.""" + unknown = sorted(set(keywords) - {field.name for field in fields(_Settings)}) + if unknown: + raise TypeError(f"IntegrityMonitor() got an unexpected keyword argument {unknown[0]!r}") + return _Settings(**keywords) + + +@dataclass(frozen=True) +class _Pass: + """One comparison: the part of the baseline examined, what was found, and the whole tree.""" + + before: Snapshot + after: Snapshot + tree: Snapshot + deep: bool + partial: bool + hashed: int + + def notes(self) -> tuple[str, ...]: + if self.partial: + return ( + f"partial pass: only the paths that changed were examined ({len(self.after)} " + f"current and {len(self.before)} baseline files), not the whole tree", + ) + if not self.deep: + return ( + f"quick pass: {self.hashed} of {len(self.after)} files hashed; size, " + "modification time and etag decided the rest", + ) + return () + + +class IntegrityMonitor: + """Compares the tree at ``target`` with the baseline stored at ``baseline``. + + Both are storage URIs (or local paths), so either may live in any backend. + Everything else is a keyword argument (:class:`MonitorKeywords`): + + ``algorithm`` + Used for :meth:`snapshot` and :meth:`create_baseline`; a verification + hashes with the algorithm of the baseline it reads. ``allow_weak=True`` + admits ``md5`` and ``sha1``. + ``interval``, ``max_workers`` + Seconds between two passes of continuous mode, and the size of the + thread pool that hashes. + ``alerts``, ``bus`` + The :class:`AlertPolicy` and the event bus drift is published on (the + process-wide one by default). + ``remediation`` + A :class:`RemediationPolicy`; without one the monitor only reads. + ``resolver`` + The :class:`StorageResolver` both URIs are resolved with. + ``on_drift``, ``manager``, ``alert_on_extra``, ``notify`` + The hooks of the first monitor, which work as they did: the callback and + the notification receive the summary :meth:`check_once` returns, and the + notification goes through ``manager``, or through the process-wide + ``notification_manager`` when none is passed. ``notify=False`` sends no + notification, for when the published event is routed to the sinks instead. + ``root``, ``manifest_path`` + The first monitor's names for ``target`` and ``baseline``. + """ + + def __init__( + self, + target: URILike | None = None, + baseline: URILike | None = None, + **keywords: Unpack[MonitorKeywords], + ) -> None: + chosen = _settings(keywords) + self._runner = IntervalRunner( + self._continuous_tick, chosen.interval, name="fa-integrity-monitor" + ) + location = _named(target, chosen.root, "target", "root") + if location is None: + raise IntegrityException("an IntegrityMonitor needs a target: a storage URI or a path") + self._target = Target(location, resolver=chosen.resolver) + self._allow_weak = bool(chosen.allow_weak) + self._max_workers = chosen.max_workers + self._algorithm = self._engine(chosen.algorithm).algorithm + self._baselines = self._baseline_manager( + _named(baseline, chosen.manifest_path, "baseline", "manifest_path"), chosen.resolver + ) + self._alerts = AlertEngine(chosen.bus, chosen.alerts) + self._remediator = ( + None if chosen.remediation is None else Remediator(chosen.remediation, self._target) + ) + self._legacy = LegacyHooks( + f"integrity drift: {self._target.uri}", + on_drift=chosen.on_drift, + manager=chosen.manager, + alert_on_extra=chosen.alert_on_extra, + notify=chosen.notify, + ) + self._lock = threading.RLock() + self._last_report: DriftReport | None = None + self._last_summary: dict[str, Any] | None = None + self._last_run: datetime | None = None + self._last_error: str | None = None + self._announced: _Signature | None = None + + # ------------------------------------------------------------------ what it watches + + @property + def target(self) -> str: + """The storage URI of the monitored tree.""" + return str(self._target.uri) + + @property + def baseline(self) -> str | None: + """The storage URI of the baseline, or ``None`` when the monitor has none.""" + return None if self._baselines is None else str(self._baselines.uri) + + @property + def algorithm(self) -> str: + return self._algorithm + + @property + def interval(self) -> float: + return self._runner.interval + + @property + def is_running(self) -> bool: + """Whether continuous mode is active.""" + return self._runner.is_running + + @property + def last_report(self) -> DriftReport | None: + return self._last_report + + @property + def last_summary(self) -> dict[str, Any] | None: + return self._last_summary + + @property + def last_run(self) -> datetime | None: + return self._last_run + + @property + def last_error(self) -> str | None: + """Why the latest pass could not run, or ``None`` when it did.""" + return self._last_error + + def has_baseline(self) -> bool: + """Return whether a baseline is configured and stored.""" + return self._baselines is not None and self._baselines.exists() + + def status(self) -> dict[str, Any]: + """Return a JSON-friendly view: running or not, the last run and the last report.""" + report, last_run = self._last_report, self._last_run + return { + "target": self.target, + "baseline": self.baseline, + "algorithm": self._algorithm, + "interval": self.interval, + "running": self.is_running, + "last_run": last_run.isoformat() if last_run else None, + "last_error": self._last_error, + "last_report": report.to_dict() if report else None, + } + + # ------------------------------------------------------------------ snapshot and baseline + + def snapshot(self) -> Snapshot: + """Return the tree as it is now. Nothing is stored.""" + return self._snapshot_with(self._engine(self._algorithm)) + + def create_baseline(self) -> Snapshot: + """Take a snapshot with the monitor's algorithm and store it as the baseline.""" + baselines = self._require_baselines() + with self._lock: + snapshot = self.snapshot() + baselines.save(snapshot) + self._announced = None + return snapshot + + def accept(self, report: DriftReport | None = None) -> Snapshot: + """Approve the current state as the new baseline and return it. + + With ``report``, exactly the tree that verification saw is stored, so a + change made since is not approved unseen. Without one, or when the + verification remediated something, the tree is read again. The + baseline keeps its algorithm. + """ + baselines = self._require_baselines() + with self._lock: + snapshot = self._approved(report, baselines) + baselines.save(snapshot) + self._announced = None + file_automation_logger.info( + "integrity: accepted the state of %s as its baseline (%d files)", + self._target.uri, + len(snapshot), + ) + return snapshot + + # ------------------------------------------------------------------ verify + + def verify(self, deep: bool = True) -> DriftReport: + """Compare the tree with the baseline and return a :class:`DriftReport`. + + ``deep=True`` hashes every file. ``deep=False`` hashes only the files + whose size, modification time or etag differ from the baseline, and the + report says it was a quick pass: a change that keeps all three goes + unnoticed. Drift is published as one ``IntegrityViolation`` event. + """ + return self._verify(deep=deep, repeat=True)[0] + + def verify_paths(self, paths: Iterable[str]) -> DriftReport: + """Verify only the files at ``paths`` (relative to the target) and under them. + + This is what watch mode runs for the paths that changed; call it when + something else tells you what changed. Each path is hashed and compared + with its baseline entry, a directory stands for everything below it, + and the report is marked ``partial``: the rest of the tree is not read. + """ + chosen = frozenset(normalize_path(path) for path in paths) + return self._verify_paths(chosen, repeat=True)[0] + + def check_once(self) -> dict[str, Any]: + """Verify once and return the first monitor's summary; a failure is in ``error``.""" + try: + return self._verify(deep=True, repeat=True)[1] + except FileAutomationException as error: + return self._record_failure(error, repeat=True) + + # ------------------------------------------------------------------ watch and continuous + + def watch( + self, *, debounce: float = _DEFAULT_DEBOUNCE, poll_interval: float | None = None + ) -> WatchHandle: + """React to changes as they happen and return a handle with ``stop()``. + + A local target is observed through filesystem events: paths that change + within ``debounce`` seconds of each other are verified together, and + only those paths are read. Any other backend is polled with a quick + pass every ``poll_interval`` seconds (``interval`` by default). Either + way a drift is reported when it appears, not again while it stays the + same. Watching does not verify the tree when it starts. + """ + self._require_baselines() + root = self._target.local_root() + handle: WatchHandle + if root is None: + period = poll_interval if poll_interval is not None else self.interval + handle = PollingWatcher(self._poll_tick, period) + else: + from automation_file.integrity.local_watcher import LocalWatcher + + handle = LocalWatcher( + root, self._paths_changed, debounce=debounce, ignore=self._target.is_left_out + ) + handle.start() + return handle + + def start(self) -> None: + """Verify every ``interval`` seconds on a thread; the first pass runs after one interval.""" + if self._runner.is_running: + return + self._runner.start() + file_automation_logger.info( + "integrity: monitoring %s against %s (interval=%.1fs)", + self._target.uri, + self.baseline, + self.interval, + ) + + def stop(self, timeout: float = _DEFAULT_JOIN_TIMEOUT) -> None: + """End continuous mode.""" + self._runner.stop(timeout) + + # ------------------------------------------------------------------ internals + + def _engine(self, algorithm: str) -> HashEngine: + return HashEngine(algorithm, allow_weak=self._allow_weak, max_workers=self._max_workers) + + def _baseline_manager( + self, baseline: URILike | None, resolver: StorageResolver | None + ) -> BaselineManager | None: + if baseline is None: + return None + baselines = BaselineManager(baseline, resolver=resolver) + inside = self._target.relative(baselines.uri) + if inside == "": + raise IntegrityException( + f"the baseline and the target are the same location: {baselines.uri}" + ) + if inside is not None: + # A baseline kept inside the tree is not part of what the tree is checked for. + own = self._target.fold(inside) + self._target.leave_out(lambda path: is_baseline_path(path, own)) + return baselines + + def _require_baselines(self) -> BaselineManager: + if self._baselines is None: + raise IntegrityException( + f"the monitor of {self._target.uri} has no baseline; pass baseline=" + ) + return self._baselines + + def _snapshot_with(self, engine: HashEngine) -> Snapshot: + return build_snapshot(self._target, engine, self._target.files()) + + def _approved(self, report: DriftReport | None, baselines: BaselineManager) -> Snapshot: + if report is not None: + if report.target != self.target: + raise IntegrityException( + f"the report describes {report.target}, not this monitor's {self.target}" + ) + if report.snapshot is not None and not report.remediation: + return report.snapshot + algorithm = baselines.load().algorithm if baselines.exists() else self._algorithm + return self._snapshot_with(self._engine(algorithm)) + + def _verify(self, *, deep: bool, repeat: bool) -> tuple[DriftReport, dict[str, Any]]: + with self._lock, correlation_scope() as correlation_id: + baseline = self._require_baselines().load() + engine = self._engine(baseline.algorithm) + infos = self._target.files() + known = {} if deep else quick_matches(infos, baseline) + current = build_snapshot(self._target, engine, infos, known=known) + done = _Pass( + before=baseline, + after=current, + tree=current, + deep=deep, + partial=False, + hashed=sum(1 for entry in current if entry.path not in known), + ) + return self._conclude(done, engine, correlation_id, repeat=repeat) + + def _verify_paths( + self, paths: frozenset[str], *, repeat: bool + ) -> tuple[DriftReport, dict[str, Any]]: + """Verify only what lies at ``paths``: the watcher's share of a verification.""" + with self._lock, correlation_scope() as correlation_id: + baseline = self._require_baselines().load() + engine = self._engine(baseline.algorithm) + recorded: dict[str, SnapshotEntry] = {} + found: dict[str, FileInfo] = {} + for path in sorted(paths): + recorded.update((entry.path, entry) for entry in baseline.below(path)) + found.update((info.path, info) for info in self._target.at(path)) + after = build_snapshot(self._target, engine, list(found.values())) + before = Snapshot( + root=baseline.root, + backend=baseline.backend, + algorithm=baseline.algorithm, + created_at=baseline.created_at, + entries=tuple(recorded.values()), + ) + tree = baseline.merged( + removed=recorded, added=after.entries, root=after.root, backend=after.backend + ) + done = _Pass( + before=before, after=after, tree=tree, deep=True, partial=True, hashed=len(after) + ) + return self._conclude(done, engine, correlation_id, repeat=repeat) + + def _conclude( + self, done: _Pass, engine: HashEngine, correlation_id: str, *, repeat: bool + ) -> tuple[DriftReport, dict[str, Any]]: + """Turn a comparison into a report, remediate, remember it and raise the alerts.""" + changes = detect_changes(done.before, done.after) + signature: _Signature = tuple( + ( + change.kind.value, + change.path, + change.previous_path or "", + change.after.checksum if change.after else "", + ) + for change in changes + ) + announce = repeat or signature != self._announced + steps: list[RemediationStep] = [] + if announce and changes and self._remediator is not None: + steps = self._remediator.apply(changes, engine) + report = DriftReport( + target=self.target, + baseline=self.baseline, + backend=done.tree.backend, + algorithm=done.tree.algorithm, + deep=done.deep, + partial=done.partial, + changes=tuple(changes), + checked=len(done.after), + hashed=done.hashed, + notes=done.notes(), + remediation=tuple(steps), + correlation_id=correlation_id, + snapshot=done.tree, + ) + summary = summary_of(report, done.before.paths) + self._last_report, self._last_summary = report, summary + self._last_run, self._last_error = report.verified_at, None + self._announced = signature + if report.ok: + file_automation_logger.debug("integrity: %s matches its baseline", self._target.uri) + elif announce: + file_automation_logger.warning( + "integrity: drift in %s: %s", self._target.uri, report.counts + ) + self._publish(report) + self._legacy.handle(summary) + return report, summary + + def _publish(self, report: DriftReport) -> None: + self._alerts.violation(report) + for step in report.remediation: + resource = str(self._target.uri.joinpath(step.path)) + self._alerts.remediated(step, resource, report.backend) + + def _backend_name(self) -> str: + try: + return self._target.backend_name + except FileAutomationException: + return "" + + def _record_failure(self, error: Exception, *, repeat: bool) -> dict[str, Any]: + """Remember that a pass could not run, and say so on the bus and to the legacy hooks.""" + file_automation_logger.error( + "integrity: verification of %s failed: %r", self._target.uri, error + ) + summary = error_summary(error) + message = f"{type(error).__name__}: {error}" + signature: _Signature = (("error", message),) + announce = repeat or signature != self._announced + self._last_summary, self._last_error = summary, message + self._last_run = datetime.now(timezone.utc) + self._announced = signature + if announce: + with correlation_scope(): + self._alerts.failure(self.target, self._backend_name(), error) + self._legacy.handle(summary) + return summary + + def _guarded(self, work: Callable[[], object], *, repeat: bool) -> None: + try: + work() + except Exception as error: # pylint: disable=broad-except + # Boundary: a monitor thread must outlive whatever one pass runs into. + self._record_failure(error, repeat=repeat) + + def _continuous_tick(self) -> None: + self._guarded(self.check_once, repeat=True) + + def _poll_tick(self) -> None: + self._guarded(lambda: self._verify(deep=False, repeat=False), repeat=False) + + def _paths_changed(self, paths: frozenset[str]) -> None: + self._guarded(lambda: self._verify_paths(paths, repeat=False), repeat=False) diff --git a/automation_file/integrity/remediation.py b/automation_file/integrity/remediation.py new file mode 100644 index 0000000..25d4620 --- /dev/null +++ b/automation_file/integrity/remediation.py @@ -0,0 +1,241 @@ +"""Remediation: what a monitor may do about drift. Off unless a policy is passed. + +Monitoring is read-only. A :class:`RemediationPolicy` given to the monitor +turns on, per kind of change, one of two actions: + +``quarantine`` + Move the offending file out of the tree into + ``//``. Nothing there is ever overwritten. +``restore`` + Copy the file back from ``restore_from``, a mirror of the baseline. The + mirror's copy is hashed first and refused when it does not match the + baseline; the restored file is hashed again before the step counts as done. + A modified file is moved to the quarantine first when one is configured; + without one its content is overwritten. + +A rename is treated as its two halves: the old path as deleted, the new one as +created. Every step is returned as a +:class:`~automation_file.integrity.report.RemediationStep`; a step that fails is +reported, never raised. +""" + +from __future__ import annotations + +from collections.abc import Iterable +from dataclasses import dataclass +from datetime import datetime, timezone + +from automation_file.exceptions import FileAutomationException +from automation_file.integrity.detector import Change, ChangeKind +from automation_file.integrity.errors import IntegrityException +from automation_file.integrity.hashing import HashEngine +from automation_file.integrity.report import RemediationStep +from automation_file.integrity.snapshot import SnapshotEntry +from automation_file.integrity.target import Target +from automation_file.logging_config import file_automation_logger +from automation_file.storage.storage import Storage +from automation_file.storage.uri import StorageURI, URILike, parse_storage_uri + +ACTION_NONE = "none" +ACTION_QUARANTINE = "quarantine" +ACTION_RESTORE = "restore" +_ON_CREATED = (ACTION_NONE, ACTION_QUARANTINE) +_ON_MODIFIED = (ACTION_NONE, ACTION_QUARANTINE, ACTION_RESTORE) +_ON_DELETED = (ACTION_NONE, ACTION_RESTORE) +_STAMP_FORMAT = "%Y%m%dT%H%M%S%fZ" + + +def _checked_action(name: str, value: str, allowed: tuple[str, ...]) -> None: + if value not in allowed: + raise IntegrityException(f"{name} must be one of {', '.join(allowed)}; got {value!r}") + + +@dataclass(frozen=True) +class RemediationPolicy: + """Which changes are acted on, and with which storage.""" + + quarantine: URILike | None = None + restore_from: URILike | None = None + on_created: str = ACTION_NONE + on_modified: str = ACTION_NONE + on_deleted: str = ACTION_NONE + + def __post_init__(self) -> None: + _checked_action("on_created", self.on_created, _ON_CREATED) + _checked_action("on_modified", self.on_modified, _ON_MODIFIED) + _checked_action("on_deleted", self.on_deleted, _ON_DELETED) + chosen = (self.on_created, self.on_modified, self.on_deleted) + if ACTION_QUARANTINE in chosen and self.quarantine is None: + raise IntegrityException("a policy that quarantines needs a quarantine= storage URI") + if ACTION_RESTORE in chosen and self.restore_from is None: + raise IntegrityException("a policy that restores needs a restore_from= storage URI") + + @property + def active(self) -> bool: + """True when at least one kind of change is acted on.""" + return (self.on_created, self.on_modified, self.on_deleted) != (ACTION_NONE,) * 3 + + @property + def quarantine_uri(self) -> StorageURI | None: + return None if self.quarantine is None else parse_storage_uri(self.quarantine) + + @property + def restore_uri(self) -> StorageURI | None: + return None if self.restore_from is None else parse_storage_uri(self.restore_from) + + +def _failure(error: FileAutomationException) -> str: + return f"{type(error).__name__}: {error}" + + +class Remediator: + """Carries out a :class:`RemediationPolicy` on one :class:`Target`.""" + + def __init__(self, policy: RemediationPolicy, target: Target) -> None: + if not isinstance(policy, RemediationPolicy): + raise IntegrityException( + f"remediation must be a RemediationPolicy, got {type(policy).__name__}" + ) + self._policy = policy + self._target = target + self._quarantine = self._outside_storage(policy.quarantine_uri, "quarantine") + self._mirror = self._outside_storage(policy.restore_uri, "restore_from") + + def _outside_storage(self, uri: StorageURI | None, name: str) -> Storage | None: + if uri is None: + return None + if self._target.relative(uri) is not None: + raise IntegrityException( + f"{name} {uri} lies inside the monitored target {self._target.uri}; " + "keep it outside the tree it serves" + ) + return Storage(uri, resolver=self._target.resolver) + + def apply(self, changes: Iterable[Change], engine: HashEngine) -> list[RemediationStep]: + """Act on ``changes`` as the policy says and return every step taken. + + ``engine`` must hash with the baseline's algorithm: a restore is checked + against the baseline's checksums. + """ + if not self._policy.active: + return [] + stamp = datetime.now(timezone.utc).strftime(_STAMP_FORMAT) + steps: list[RemediationStep] = [] + for change in changes: + steps.extend(self._handle(change, stamp, engine)) + for step in steps: + file_automation_logger.info( + "integrity: %s of %s %s", step.action, step.path, "done" if step.ok else "failed" + ) + return steps + + def _handle(self, change: Change, stamp: str, engine: HashEngine) -> list[RemediationStep]: + if change.kind is ChangeKind.CREATED: + return self._created(change.path, change.kind, stamp) + if change.kind is ChangeKind.MODIFIED: + return self._modified(change, stamp, engine) + if change.kind is ChangeKind.DELETED: + return self._deleted(change.path, change.before, change.kind, engine) + if change.kind is ChangeKind.RENAMED and change.previous_path is not None: + return [ + *self._deleted(change.previous_path, change.before, change.kind, engine), + *self._created(change.path, change.kind, stamp), + ] + return [] + + def _created(self, path: str, kind: ChangeKind, stamp: str) -> list[RemediationStep]: + if self._policy.on_created != ACTION_QUARANTINE: + return [] + return [self._quarantined(path, kind, stamp)] + + def _modified(self, change: Change, stamp: str, engine: HashEngine) -> list[RemediationStep]: + if self._policy.on_modified == ACTION_QUARANTINE: + return [self._quarantined(change.path, change.kind, stamp)] + if self._policy.on_modified != ACTION_RESTORE: + return [] + refusal = self._mirror_refusal(change.path, change.before, engine) + if refusal is not None: + return [self._restore_step(change.path, change.kind, refusal)] + if self._quarantine is None: + return [self._restored(change.path, change.before, change.kind, engine)] + set_aside = self._quarantined(change.path, change.kind, stamp) + if not set_aside.ok: + reason = "the modified file could not be quarantined, so it was left in place" + return [set_aside, self._restore_step(change.path, change.kind, reason)] + return [set_aside, self._restored(change.path, change.before, change.kind, engine)] + + def _deleted( + self, path: str, expected: SnapshotEntry | None, kind: ChangeKind, engine: HashEngine + ) -> list[RemediationStep]: + if self._policy.on_deleted != ACTION_RESTORE: + return [] + refusal = self._mirror_refusal(path, expected, engine) + if refusal is not None: + return [self._restore_step(path, kind, refusal)] + return [self._restored(path, expected, kind, engine)] + + def _quarantined(self, path: str, kind: ChangeKind, stamp: str) -> RemediationStep: + """Move ``path`` out of the tree into this pass's quarantine directory.""" + if self._quarantine is None: + raise IntegrityException("no quarantine storage is configured") + source = self._target.storage.file(path) + destination = self._quarantine.file(f"{stamp}/{path}") + error: str | None = None + try: + source.move_to(destination, overwrite=False) + except FileAutomationException as failure: + error = _failure(failure) + return RemediationStep( + action=ACTION_QUARANTINE, + path=path, + kind=kind.value, + ok=error is None, + source=str(source), + destination=str(destination), + error=error, + ) + + def _mirror_refusal( + self, path: str, expected: SnapshotEntry | None, engine: HashEngine + ) -> str | None: + """Say why the mirror's copy of ``path`` must not be restored, or ``None`` if it may.""" + if self._mirror is None: + raise IntegrityException("no restore_from storage is configured") + if expected is None or not expected.checksum: + return "the baseline records no checksum for it, so a restore could not be verified" + try: + digest = engine.hash_file(self._mirror, path) + except FileAutomationException as failure: + return f"the mirror could not be read: {_failure(failure)}" + if digest is None: + return f"the mirror {self._mirror} has no copy of it" + if digest != expected.checksum: + return "the mirror's copy does not match the baseline checksum; nothing was copied" + return None + + def _restored( + self, path: str, expected: SnapshotEntry | None, kind: ChangeKind, engine: HashEngine + ) -> RemediationStep: + """Copy the verified mirror copy of ``path`` into place and check it again there.""" + if self._mirror is None or expected is None: + raise IntegrityException("a restore needs a mirror and a baseline entry") + try: + self._mirror.file(path).copy_to(self._target.storage.file(path)) + digest = engine.hash_file(self._target.storage, path) + except FileAutomationException as failure: + return self._restore_step(path, kind, _failure(failure)) + if digest != expected.checksum: + reason = "the restored file does not match the baseline checksum" + return self._restore_step(path, kind, reason) + return self._restore_step(path, kind, None) + + def _restore_step(self, path: str, kind: ChangeKind, error: str | None) -> RemediationStep: + return RemediationStep( + action=ACTION_RESTORE, + path=path, + kind=kind.value, + ok=error is None, + source=str(self._mirror.file(path)) if self._mirror is not None else "", + destination=str(self._target.storage.file(path)), + error=error, + ) diff --git a/automation_file/integrity/report.py b/automation_file/integrity/report.py new file mode 100644 index 0000000..3f014d2 --- /dev/null +++ b/automation_file/integrity/report.py @@ -0,0 +1,108 @@ +"""DriftReport: the outcome of one verification. + +A report holds the changes, how many there are of each kind, how thorough the +pass was (``deep``, ``partial``, ``hashed`` of ``checked`` files, and ``notes`` +that say so in words) and what remediation did. ``to_dict`` is JSON-friendly. +""" + +from __future__ import annotations + +from dataclasses import dataclass, field +from datetime import datetime, timezone +from typing import Any + +from automation_file.integrity.detector import Change, ChangeKind +from automation_file.integrity.hashing import DEFAULT_ALGORITHM +from automation_file.integrity.snapshot import Snapshot + + +def _now() -> datetime: + return datetime.now(timezone.utc) + + +@dataclass(frozen=True) +class RemediationStep: + """One thing remediation did, or tried to do, about a change.""" + + action: str + path: str + kind: str + ok: bool + source: str = "" + destination: str = "" + error: str | None = None + + def to_dict(self) -> dict[str, Any]: + return { + "action": self.action, + "path": self.path, + "kind": self.kind, + "ok": self.ok, + "source": self.source, + "destination": self.destination, + "error": self.error, + } + + +@dataclass(frozen=True) +class DriftReport: + """What one verification of ``target`` against ``baseline`` found. + + ``deep`` is false for a quick pass, which hashed only the files whose size, + modification time or etag differ from the baseline. ``partial`` is true + when only the paths a watcher saw change were examined. ``snapshot`` is the + tree as this pass saw it; :meth:`IntegrityMonitor.accept` stores it. + """ + + target: str + baseline: str | None = None + backend: str = "" + algorithm: str = DEFAULT_ALGORITHM + deep: bool = True + partial: bool = False + changes: tuple[Change, ...] = () + checked: int = 0 + hashed: int = 0 + notes: tuple[str, ...] = () + remediation: tuple[RemediationStep, ...] = () + verified_at: datetime = field(default_factory=_now) + correlation_id: str = "" + snapshot: Snapshot | None = field(default=None, repr=False, compare=False) + + @property + def ok(self) -> bool: + """True when nothing differs from the baseline.""" + return not self.changes + + @property + def counts(self) -> dict[str, int]: + """The number of changes of every kind, kinds without a change included.""" + totals = {kind.value: 0 for kind in ChangeKind} + for change in self.changes: + totals[change.kind.value] += 1 + return totals + + def paths(self, kind: ChangeKind | str) -> list[str]: + """Return the paths of the changes of one kind.""" + wanted = ChangeKind(kind) + return [change.path for change in self.changes if change.kind is wanted] + + def to_dict(self) -> dict[str, Any]: + """Return a JSON-serialisable mapping of the report, without the snapshot.""" + return { + "target": self.target, + "baseline": self.baseline, + "backend": self.backend, + "algorithm": self.algorithm, + "ok": self.ok, + "deep": self.deep, + "partial": self.partial, + "checked": self.checked, + "hashed": self.hashed, + "counts": self.counts, + "changes": [change.to_dict() for change in self.changes], + "notes": list(self.notes), + "remediation": [step.to_dict() for step in self.remediation], + "verified_at": self.verified_at.isoformat(), + "correlation_id": self.correlation_id, + } diff --git a/automation_file/integrity/snapshot.py b/automation_file/integrity/snapshot.py new file mode 100644 index 0000000..2c5dd01 --- /dev/null +++ b/automation_file/integrity/snapshot.py @@ -0,0 +1,325 @@ +"""Snapshot: what a directory tree holds at one moment. + +A :class:`Snapshot` lists every file below a target with its size, modification +time, checksum and whatever else the backend reports. It is frozen and turns +into a JSON-friendly dictionary with ``to_dict``; ``from_dict`` reads it back +and rejects anything that is not a well-formed entry. + +:func:`build_snapshot` produces one through the storage layer, so it works on +any backend. Directories are not recorded: an empty directory is invisible. +""" + +from __future__ import annotations + +from bisect import bisect_left +from collections.abc import Iterable, Iterator, Mapping, Sequence +from dataclasses import dataclass, field, replace +from datetime import datetime, timezone +from typing import Any + +from automation_file.exceptions import FileNotExistsException, StorageURIException +from automation_file.integrity.errors import IntegrityException +from automation_file.integrity.hashing import DEFAULT_ALGORITHM, HashEngine +from automation_file.integrity.target import Target +from automation_file.storage.types import FileInfo, StorageCapabilities +from automation_file.storage.uri import normalize_path + +# The character that sorts right after "/": every path below "a/" lies in ["a/", "a0"). +_AFTER_SEPARATOR = chr(ord("/") + 1) + + +def _now() -> datetime: + return datetime.now(timezone.utc) + + +def as_utc(moment: datetime | None) -> datetime | None: + """Return ``moment`` as an aware UTC time; a naive one is taken to be UTC already.""" + if moment is None: + return None + if moment.tzinfo is None: + return moment.replace(tzinfo=timezone.utc) + return moment.astimezone(timezone.utc) + + +def parse_timestamp(value: object, name: str) -> datetime | None: + """Read an ISO 8601 timestamp written by ``to_dict``; ``None`` stays ``None``.""" + if value is None: + return None + if not isinstance(value, str): + raise IntegrityException(f"{name} must be an ISO 8601 timestamp, got {value!r}") + text = f"{value[:-1]}+00:00" if value.endswith("Z") else value + try: + return as_utc(datetime.fromisoformat(text)) + except ValueError as error: + raise IntegrityException(f"{name} is not an ISO 8601 timestamp: {value!r}") from error + + +def _text(data: Mapping[str, Any], key: str, default: str | None = None) -> str | None: + value = data.get(key, default) + if value is None or isinstance(value, str): + return value + raise IntegrityException(f"snapshot entry field {key!r} must be text, got {value!r}") + + +def _count(data: Mapping[str, Any], key: str) -> int | None: + value = data.get(key) + if value is None: + return None + if isinstance(value, bool) or not isinstance(value, int) or value < 0: + raise IntegrityException( + f"snapshot entry field {key!r} must be a non-negative integer, got {value!r}" + ) + return value + + +def _entry_path(value: object) -> str: + if not isinstance(value, str): + raise IntegrityException(f"snapshot entry has no usable 'path': {value!r}") + try: + clean = normalize_path(value) + except StorageURIException as error: + raise IntegrityException(f"snapshot entry path {value!r} is not valid: {error}") from error + if not clean: + raise IntegrityException("snapshot entry has an empty 'path'") + return clean + + +@dataclass(frozen=True) +class SnapshotEntry: + """One file of a snapshot. Fields a backend cannot provide are ``None``. + + ``mode`` holds the permission bits and is recorded only for a local file. + """ + + path: str + size: int | None = None + modified_at: datetime | None = None + checksum: str = "" + algorithm: str = DEFAULT_ALGORITHM + content_type: str | None = None + backend: str = "" + version: str | None = None + etag: str | None = None + mode: int | None = None + + def to_dict(self) -> dict[str, Any]: + """Return a JSON-serialisable mapping; ``modified_at`` becomes ISO 8601.""" + return { + "path": self.path, + "size": self.size, + "modified_at": self.modified_at.isoformat() if self.modified_at else None, + "checksum": self.checksum, + "algorithm": self.algorithm, + "content_type": self.content_type, + "backend": self.backend, + "version": self.version, + "etag": self.etag, + "mode": self.mode, + } + + @classmethod + def from_dict( + cls, data: Any, *, algorithm: str = DEFAULT_ALGORITHM, backend: str = "" + ) -> SnapshotEntry: + """Build an entry from ``to_dict`` output; ``algorithm`` and ``backend`` fill gaps.""" + if not isinstance(data, Mapping): + raise IntegrityException(f"snapshot entry must be an object, got {data!r}") + return cls( + path=_entry_path(data.get("path")), + size=_count(data, "size"), + modified_at=parse_timestamp(data.get("modified_at"), "snapshot entry 'modified_at'"), + checksum=(_text(data, "checksum") or "").strip().lower(), + algorithm=_text(data, "algorithm") or algorithm, + content_type=_text(data, "content_type"), + backend=_text(data, "backend") or backend, + version=_text(data, "version"), + etag=_text(data, "etag"), + mode=_count(data, "mode"), + ) + + +def _by_path(entry: SnapshotEntry) -> str: + return entry.path + + +@dataclass(frozen=True) +class Snapshot: + """Every file below ``root`` (a storage URI) at ``created_at``, sorted by path.""" + + root: str + backend: str = "" + algorithm: str = DEFAULT_ALGORITHM + created_at: datetime = field(default_factory=_now) + entries: tuple[SnapshotEntry, ...] = () + _index: Mapping[str, SnapshotEntry] = field(init=False, repr=False, compare=False) + + def __post_init__(self) -> None: + ordered = tuple(sorted(self.entries, key=_by_path)) + index = {entry.path: entry for entry in ordered} + if len(index) != len(ordered): + raise IntegrityException(f"a snapshot of {self.root} lists a path more than once") + object.__setattr__(self, "entries", ordered) + object.__setattr__(self, "_index", index) + + @property + def paths(self) -> tuple[str, ...]: + return tuple(self._index) + + def get(self, path: str) -> SnapshotEntry | None: + """Return the entry at ``path``, or ``None``.""" + return self._index.get(path) + + def below(self, path: str) -> tuple[SnapshotEntry, ...]: + """Return the entry at ``path`` and every entry under it as a directory.""" + if not path: + return self.entries + start = bisect_left(self.entries, f"{path}/", key=_by_path) + end = bisect_left(self.entries, f"{path}{_AFTER_SEPARATOR}", key=_by_path) + exact = self._index.get(path) + return (*((exact,) if exact is not None else ()), *self.entries[start:end]) + + def merged( + self, *, removed: Iterable[str], added: Iterable[SnapshotEntry], root: str, backend: str + ) -> Snapshot: + """Return a new snapshot of ``root`` without ``removed`` and with ``added`` on top.""" + dropped = set(removed) + kept = {entry.path: entry for entry in self.entries if entry.path not in dropped} + kept.update((entry.path, entry) for entry in added) + return Snapshot( + root=root, backend=backend, algorithm=self.algorithm, entries=tuple(kept.values()) + ) + + def __len__(self) -> int: + return len(self.entries) + + def __iter__(self) -> Iterator[SnapshotEntry]: + return iter(self.entries) + + def __contains__(self, path: object) -> bool: + return path in self._index + + def to_dict(self) -> dict[str, Any]: + """Return a JSON-serialisable mapping of the snapshot.""" + return { + "created_at": self.created_at.isoformat(), + "root": self.root, + "backend": self.backend, + "algorithm": self.algorithm, + "entries": [entry.to_dict() for entry in self.entries], + } + + @classmethod + def from_dict(cls, data: Any) -> Snapshot: + """Build a snapshot from ``to_dict`` output.""" + if not isinstance(data, Mapping): + raise IntegrityException(f"a snapshot must be an object, got {data!r}") + entries = data.get("entries") + if not isinstance(entries, list): + raise IntegrityException("a snapshot needs an 'entries' list") + algorithm = _text(data, "algorithm") or DEFAULT_ALGORITHM + backend = _text(data, "backend") or "" + return cls( + root=_text(data, "root") or "", + backend=backend, + algorithm=algorithm, + created_at=parse_timestamp(data.get("created_at"), "snapshot 'created_at'") or _now(), + entries=tuple( + SnapshotEntry.from_dict(entry, algorithm=algorithm, backend=backend) + for entry in entries + ), + ) + + +def quick_matches(infos: Iterable[FileInfo], baseline: Snapshot) -> dict[str, SnapshotEntry]: + """Return the baseline entries a quick pass takes on trust, by path. + + A file is trusted when its size, modification time and etag are the ones the + baseline recorded. An entry that recorded neither a time nor an etag gives + nothing to compare beyond the size, so its file is hashed. + """ + trusted: dict[str, SnapshotEntry] = {} + for info in infos: + known = baseline.get(info.path) + if known is not None and _same_stamp(info, known): + trusted[info.path] = known + return trusted + + +def _same_stamp(info: FileInfo, known: SnapshotEntry) -> bool: + if known.size is None or info.size != known.size: + return False + if known.modified_at is None and known.etag is None: + return False + return as_utc(info.modified_at) == known.modified_at and info.etag == known.etag + + +def _detailed(target: Target, capabilities: StorageCapabilities, info: FileInfo) -> FileInfo | None: + """Add what a listing leaves out (version, content type); ``None`` when the file is gone.""" + lacks_version = capabilities.version and info.version is None + lacks_type = capabilities.content_type and info.content_type is None + if not (lacks_version or lacks_type): + return info + try: + detail = target.storage.stat(info.path) + except FileNotExistsException: + return None + return replace( + info, + version=info.version or detail.version, + content_type=info.content_type or detail.content_type, + ) + + +def build_snapshot( + target: Target, + engine: HashEngine, + infos: Sequence[FileInfo], + *, + known: Mapping[str, SnapshotEntry] | None = None, +) -> Snapshot: + """Return the snapshot of the files ``infos`` of ``target``. + + A path in ``known`` keeps the checksum of its entry there instead of being + hashed; that is how a quick pass skips the files it trusts. A file that + vanishes while the snapshot is taken is left out. + """ + trusted = known or {} + served_by = target.backend + backend, capabilities = served_by.scheme, served_by.capabilities + read_mode = target.mode_reader() + + def describe(info: FileInfo) -> SnapshotEntry | None: + carried = trusted.get(info.path) + if carried is not None: + digest: str | None = carried.checksum + detail: FileInfo | None = replace( + info, + version=info.version or carried.version, + content_type=info.content_type or carried.content_type, + ) + else: + digest = engine.hash_file(target.storage, info.path) + detail = _detailed(target, capabilities, info) if digest is not None else None + if digest is None or detail is None: + return None + return SnapshotEntry( + path=info.path, + size=detail.size, + modified_at=as_utc(detail.modified_at), + checksum=digest, + algorithm=engine.algorithm, + content_type=detail.content_type, + backend=backend, + version=detail.version, + etag=detail.etag, + mode=read_mode(info.path), + ) + + by_path = {info.path: info for info in infos} + described = engine.map(lambda path: describe(by_path[path]), by_path) + return Snapshot( + root=str(target.uri), + backend=backend, + algorithm=engine.algorithm, + entries=tuple(entry for entry in described.values() if entry is not None), + ) diff --git a/automation_file/integrity/target.py b/automation_file/integrity/target.py new file mode 100644 index 0000000..315b1a3 --- /dev/null +++ b/automation_file/integrity/target.py @@ -0,0 +1,168 @@ +"""The monitored tree: where it is, which backend serves it, and what is left out of it. + +Everything the integrity subsystem reads goes through :class:`Target`, and +:class:`Target` goes through the storage layer, so a tree may live in any +backend. Two things depend on the backend being a +:class:`~automation_file.storage.LocalStorage`: the permission bits of a file, +and the directory a watcher can observe. +""" + +from __future__ import annotations + +import os +import stat +from collections.abc import Callable +from pathlib import Path + +from automation_file.exceptions import ( + FileNotExistsException, + PathTraversalException, + StorageNotFoundException, + StoragePathTypeException, +) +from automation_file.integrity.errors import IntegrityException +from automation_file.storage.backend import StorageBackend, join_path +from automation_file.storage.local_storage import LocalStorage +from automation_file.storage.resolver import StorageResolver, default_resolver +from automation_file.storage.storage import Storage +from automation_file.storage.types import FileInfo, StorageCapabilities +from automation_file.storage.uri import LOCAL_SCHEME, StorageURI, URILike, parse_storage_uri + +PathRule = Callable[[str], bool] +ModeReader = Callable[[str], int | None] + + +def _no_mode(_path: str) -> int | None: + return None + + +class Target: + """A directory somewhere in storage, read for snapshots.""" + + def __init__(self, uri: URILike, *, resolver: StorageResolver | None = None) -> None: + self._resolver = resolver if resolver is not None else default_resolver + self._uri = parse_storage_uri(uri) + self._storage = Storage(self._uri, resolver=self._resolver) + self._left_out: list[PathRule] = [] + # Windows compares local paths without regard to case. + self._case_blind = self._uri.scheme == LOCAL_SCHEME and os.sep == "\\" + + @property + def uri(self) -> StorageURI: + return self._uri + + @property + def storage(self) -> Storage: + return self._storage + + @property + def resolver(self) -> StorageResolver: + return self._resolver + + @property + def backend(self) -> StorageBackend: + """The backend that serves the tree right now.""" + return self._resolver.resolve(self._uri)[0] + + @property + def backend_name(self) -> str: + return self.backend.scheme + + @property + def capabilities(self) -> StorageCapabilities: + return self.backend.capabilities + + def fold(self, path: str) -> str: + """Return ``path`` in the form paths of this tree are compared in.""" + return path.casefold() if self._case_blind else path + + def relative(self, other: URILike) -> str | None: + """Return the path of ``other`` inside this tree: ``""`` for the tree itself, ``None`` outside.""" + candidate = parse_storage_uri(other) + if candidate.scheme != self._uri.scheme: + return None + if self.fold(candidate.authority) != self.fold(self._uri.authority): + return None + own = self._uri.path.split("/") if self._uri.path else [] + theirs = candidate.path.split("/") if candidate.path else [] + head = theirs[: len(own)] + if [self.fold(segment) for segment in head] != [self.fold(segment) for segment in own]: + return None + return "/".join(theirs[len(own) :]) + + def leave_out(self, rule: PathRule) -> None: + """Skip every file whose folded path ``rule`` accepts.""" + self._left_out.append(rule) + + def is_left_out(self, path: str) -> bool: + folded = self.fold(path) + return any(rule(folded) for rule in self._left_out) + + def files(self, below: str = "") -> list[FileInfo]: + """Return every file at any depth under ``below``, with paths relative to the tree. + + A prefix of an object store that holds nothing is an empty tree. A + directory of a filesystem that is missing is an error. + """ + try: + listing = self._storage.list_dir(below, recursive=True) + except StorageNotFoundException as error: + if below or not self.capabilities.directories: + return [] + raise IntegrityException(f"target {self._uri} does not exist") from error + except StoragePathTypeException as error: + if below: + return [] + raise IntegrityException(f"target {self._uri} is not a directory") from error + return [info for info in listing if not info.is_dir and not self.is_left_out(info.path)] + + def at(self, path: str) -> list[FileInfo]: + """Return the files at ``path``: the file itself, all below a directory, none if gone. + + A file is described the way :meth:`files` describes it, so the result + compares cleanly with a baseline taken from a listing. + """ + try: + info = self._storage.stat(path) + if info.is_dir: + return self.files(path) + if self.is_left_out(info.path): + return [] + return [self._as_listed(info)] + except FileNotExistsException: + return [] + + def _as_listed(self, info: FileInfo) -> FileInfo: + """Return the listing's view of the file ``info``. + + On a filesystem ``stat`` and a listing agree. An object store answers + ``stat`` from an HTTP header with whole seconds and a listing with the + stored time, so the two can differ for the same object. + """ + if self.capabilities.directories: + return info + parent = info.path.rpartition("/")[0] + for listed in self._storage.list_dir(parent): + if listed.path == info.path and not listed.is_dir: + return listed + return info + + def local_root(self) -> Path | None: + """Return the directory of the tree on this machine, or ``None`` for another backend.""" + backend, base = self._resolver.resolve(self._uri) + return backend.local_path(base) if isinstance(backend, LocalStorage) else None + + def mode_reader(self) -> ModeReader: + """Return a function that reads the permission bits of a path; always ``None`` off-disk.""" + backend, base = self._resolver.resolve(self._uri) + if not isinstance(backend, LocalStorage): + return _no_mode + local = backend + + def read(path: str) -> int | None: + try: + return stat.S_IMODE(local.local_path(join_path(base, path)).stat().st_mode) + except (OSError, PathTraversalException): + return None + + return read diff --git a/automation_file/integrity/watcher.py b/automation_file/integrity/watcher.py new file mode 100644 index 0000000..44760fb --- /dev/null +++ b/automation_file/integrity/watcher.py @@ -0,0 +1,200 @@ +"""Watcher: the threads behind the watch and continuous modes. + +* :class:`IntervalRunner` calls a function every N seconds. Continuous mode is + one of these, and so is :class:`PollingWatcher`, the watcher for a backend + that cannot report changes by itself. +* :class:`Debouncer` collects the paths a filesystem observer reports and hands + them over as one batch once they stop arriving, so saving one file ten times + is one verification. +* :class:`WatchHandle` is what ``IntegrityMonitor.watch()`` returns. + +The watchdog-based watcher for a local directory is in +:mod:`automation_file.integrity.local_watcher`, which is imported only when a +local target is watched. +""" + +from __future__ import annotations + +import threading +import time +from abc import ABC, abstractmethod +from collections.abc import Callable +from types import TracebackType +from typing import TypeVar + +from automation_file.integrity.errors import IntegrityException + +_DEFAULT_JOIN_TIMEOUT = 5.0 +# A batch is handed over at the latest after this many quiet periods, however busy the tree. +_MAX_WAIT_FACTOR = 10.0 + +_HandleT = TypeVar("_HandleT", bound="WatchHandle") + + +class WatchHandle(ABC): + """A running watch. ``stop()`` ends it; it is also a context manager.""" + + #: ``"events"`` when the backend reports changes, ``"poll"`` when the tree is re-read. + kind: str = "" + + @abstractmethod + def start(self) -> None: + """Begin watching. Starting a running watch changes nothing.""" + + @abstractmethod + def stop(self, timeout: float = _DEFAULT_JOIN_TIMEOUT) -> None: + """Stop watching and wait up to ``timeout`` seconds for the threads to end.""" + + @property + @abstractmethod + def is_running(self) -> bool: + """Whether the watch is active.""" + + def __enter__(self: _HandleT) -> _HandleT: + return self + + def __exit__( + self, + exc_type: type[BaseException] | None, + exc: BaseException | None, + tb: TracebackType | None, + ) -> None: + self.stop() + + +class IntervalRunner: + """Calls ``tick`` every ``interval`` seconds on a daemon thread, first after one interval.""" + + def __init__(self, tick: Callable[[], object], interval: float, *, name: str) -> None: + if interval <= 0: + raise IntegrityException("interval must be positive") + self._tick = tick + self._interval = float(interval) + self._name = name + self._lock = threading.Lock() + self._thread: threading.Thread | None = None + self._stop: threading.Event | None = None + + @property + def interval(self) -> float: + return self._interval + + @property + def is_running(self) -> bool: + thread = self._thread + return thread is not None and thread.is_alive() + + def start(self) -> None: + with self._lock: + if self.is_running: + return + stop = threading.Event() + thread = threading.Thread(target=self._run, args=(stop,), name=self._name, daemon=True) + thread.start() + self._thread, self._stop = thread, stop + + def stop(self, timeout: float = _DEFAULT_JOIN_TIMEOUT) -> None: + with self._lock: + thread, stop = self._thread, self._stop + self._thread = self._stop = None + if stop is not None: + stop.set() + if thread is not None and thread.is_alive() and thread is not threading.current_thread(): + thread.join(timeout=timeout) + + def _run(self, stop: threading.Event) -> None: + while not stop.wait(self._interval): + self._tick() + + +class PollingWatcher(WatchHandle): + """Watches by re-reading: calls ``tick`` every ``interval`` seconds.""" + + kind = "poll" + + def __init__(self, tick: Callable[[], object], interval: float) -> None: + self._runner = IntervalRunner(tick, interval, name="fa-integrity-poll") + + @property + def is_running(self) -> bool: + return self._runner.is_running + + def start(self) -> None: + self._runner.start() + + def stop(self, timeout: float = _DEFAULT_JOIN_TIMEOUT) -> None: + self._runner.stop(timeout) + + +class Debouncer: + """Collects paths and delivers them as one batch after ``delay`` seconds of quiet.""" + + def __init__(self, deliver: Callable[[frozenset[str]], object], delay: float) -> None: + if delay < 0: + raise IntegrityException("debounce must not be negative") + self._deliver = deliver + self._delay = float(delay) + self._wake = threading.Condition() + self._pending: set[str] = set() + self._first = 0.0 + self._last = 0.0 + self._stopped = False + self._thread: threading.Thread | None = None + + @property + def is_running(self) -> bool: + thread = self._thread + return thread is not None and thread.is_alive() + + def start(self) -> None: + if self.is_running: + return + with self._wake: + self._stopped = False + thread = threading.Thread(target=self._run, name="fa-integrity-debounce", daemon=True) + thread.start() + self._thread = thread + + def add(self, path: str) -> None: + """Note that ``path`` changed; the batch it joins is delivered once things settle.""" + with self._wake: + now = time.monotonic() + if not self._pending: + self._first = now + self._pending.add(path) + self._last = now + self._wake.notify_all() + + def stop(self, timeout: float = _DEFAULT_JOIN_TIMEOUT) -> None: + """Stop the thread; paths that were not delivered yet are dropped.""" + with self._wake: + self._stopped = True + self._pending.clear() + self._wake.notify_all() + thread, self._thread = self._thread, None + if thread is not None and thread.is_alive() and thread is not threading.current_thread(): + thread.join(timeout=timeout) + + def _run(self) -> None: + while True: + batch = self._next_batch() + if batch is None: + return + self._deliver(batch) + + def _next_batch(self) -> frozenset[str] | None: + """Wait for paths, then for them to stop arriving; ``None`` once stopped.""" + with self._wake: + while not self._pending and not self._stopped: + self._wake.wait() + while not self._stopped: + due = min(self._last + self._delay, self._first + self._delay * _MAX_WAIT_FACTOR) + remaining = due - time.monotonic() + if remaining <= 0: + break + self._wake.wait(remaining) + if self._stopped: + return None + batch = frozenset(self._pending) + self._pending.clear() + return batch diff --git a/docs/source/API/api_index.rst b/docs/source/API/api_index.rst index f01e25c..d7f662c 100644 --- a/docs/source/API/api_index.rst +++ b/docs/source/API/api_index.rst @@ -200,3 +200,18 @@ storage observers. :caption: Events events + +.. _api-integrity: + +Chapter O — File Integrity Monitoring +===================================== + +``IntegrityMonitor``, snapshots and the manifest, the change detector, the +drift report, alerts, remediation, the watchers and the ``FA_integrity_*`` +actions. + +.. toctree:: + :maxdepth: 2 + :caption: File Integrity Monitoring + + integrity diff --git a/docs/source/API/integrity.rst b/docs/source/API/integrity.rst new file mode 100644 index 0000000..927303f --- /dev/null +++ b/docs/source/API/integrity.rst @@ -0,0 +1,82 @@ +File integrity monitoring +========================= + +``IntegrityMonitor`` and its parts: snapshots, the manifest schema, the hash +engine, the baseline manager, the change detector, the watchers, the alert +engine and remediation. Usage is described in the manual chapter *File integrity +monitoring*. + +Monitor +------- + +.. automodule:: automation_file.integrity.monitor + :members: + +Snapshot and target +------------------- + +.. automodule:: automation_file.integrity.snapshot + :members: + +.. automodule:: automation_file.integrity.target + :members: + +Manifest and baseline +--------------------- + +.. automodule:: automation_file.integrity.manifest + :members: + +.. automodule:: automation_file.integrity.baseline + :members: + +Hash engine +----------- + +.. automodule:: automation_file.integrity.hashing + :members: + +Change detector and report +-------------------------- + +.. automodule:: automation_file.integrity.detector + :members: + +.. automodule:: automation_file.integrity.report + :members: + +Alert engine +------------ + +.. automodule:: automation_file.integrity.alerts + :members: + +Remediation +----------- + +.. automodule:: automation_file.integrity.remediation + :members: + +Watchers +-------- + +.. automodule:: automation_file.integrity.watcher + :members: + +.. automodule:: automation_file.integrity.local_watcher + :members: + +Actions +------- + +.. automodule:: automation_file.integrity.actions + :members: + +Legacy summary and errors +------------------------- + +.. automodule:: automation_file.integrity.legacy + :members: + +.. automodule:: automation_file.integrity.errors + :members: diff --git a/docs/source/Eng/eng_index.rst b/docs/source/Eng/eng_index.rst index d6e0e24..131fc43 100644 --- a/docs/source/Eng/eng_index.rst +++ b/docs/source/Eng/eng_index.rst @@ -158,7 +158,8 @@ Chapter 10 — Reliability ``retry_on_transient`` with capped exponential back-off, ``Quota`` size and time budgets, ``CircuitBreaker``, ``RateLimiter``, ``FileLock`` / ``SQLiteLock``, persistent ``ActionQueue``, SQLite ``AuditLog``, -``IntegrityMonitor`` for periodic manifest verification, and the typed +``IntegrityMonitor`` for periodic manifest verification (now +:doc:`usage/integrity`), and the typed ``FileAutomationException`` hierarchy. .. toctree:: @@ -272,3 +273,18 @@ observers that report storage operations. :caption: Events usage/event_bus + +.. _eng-integrity: + +Chapter 18 — File Integrity Monitoring +====================================== + +``IntegrityMonitor`` keeps an approved baseline of a directory tree in any +storage backend and reports what drifted from it: snapshots, the manifest +schema, the four modes, alerts, and opt-in remediation. + +.. toctree:: + :maxdepth: 2 + :caption: File Integrity Monitoring + + usage/integrity diff --git a/docs/source/Eng/usage/integrity.rst b/docs/source/Eng/usage/integrity.rst new file mode 100644 index 0000000..c970de7 --- /dev/null +++ b/docs/source/Eng/usage/integrity.rst @@ -0,0 +1,514 @@ +File integrity monitoring +========================= + +``automation_file.integrity`` answers one question about a directory tree: is it +still what was approved? An +:class:`~automation_file.integrity.monitor.IntegrityMonitor` records the approved +state as a *baseline*, compares the tree with it, and reports every difference +as a change of one of six kinds. + +It reads through the :doc:`storage layer `, so the tree and the baseline +are storage URIs and may live in any backend. Drift is published as an event +(:doc:`event_bus`); the monitor never calls a notification sink itself. It only +reads, unless you pass a remediation policy. + +``IntegrityMonitor`` is still importable from ``automation_file`` and from +``automation_file.core.fim``, and the call written for the first monitor keeps +working (`The first monitor`_). + +Minimal example +--------------- + +.. code-block:: python + + from automation_file.integrity import IntegrityMonitor + + monitor = IntegrityMonitor("/srv/site", baseline="/var/lib/fa/site.baseline.json") + monitor.create_baseline() # approve what is there now + + report = monitor.verify() # hash every file, compare with the baseline + if not report.ok: + print(report.counts) # {'created': 0, 'modified': 1, 'deleted': 0, ...} + for change in report.changes: + print(change.kind.value, change.path) + monitor.accept(report) # after review: approve what the report saw + +Production example +------------------ + +The baseline is kept outside the bucket it describes, the monitor verifies on a +thread, the event is routed to a notification sink, and remediation is switched +on explicitly. + +.. code-block:: python + + import json + + from automation_file import Severity, SlackSink, event_bus, notification_manager, s3_instance + from automation_file.integrity import IntegrityMonitor, RemediationPolicy + + s3_instance.later_init(region_name="eu-west-1") + notification_manager.register(SlackSink(slack_webhook_url)) + + def route(event): + level = "error" if event.severity.at_least(Severity.ERROR) else "warning" + details = event.payload.get("error") or json.dumps(event.payload.get("counts", {})) + notification_manager.notify(event.subject, details, level) + + # integrity.violation and integrity.remediated; a successful remediation is "info". + event_bus.subscribe(route, types="integrity.*", min_severity=Severity.WARNING) + + monitor = IntegrityMonitor( + "s3://reports/2026", + baseline="local:///var/lib/fa/baselines/reports-2026.json", + algorithm="sha256", + interval=900, # continuous mode: every 15 minutes + remediation=RemediationPolicy( # opt-in; without it nothing is changed + quarantine="s3://reports-quarantine/2026", + restore_from="s3://reports-mirror/2026", + on_created="quarantine", + on_modified="restore", + on_deleted="restore", + ), + ) + if not monitor.has_baseline(): + monitor.create_baseline() + + monitor.start() # a daemon thread; returns at once + ... + monitor.status() # running, last_run, last_error, last_report + monitor.stop() + +Keep the baseline where whoever can change the tree cannot change the baseline. +A baseline inside the target works, and the monitor leaves that file out of its +snapshots, but then one write access covers both. + +The four modes +-------------- + +.. list-table:: + :header-rows: 1 + :widths: 16 30 54 + + * - Mode + - Call + - What it does + * - snapshot + - ``monitor.snapshot()`` + - Reads the tree and returns a + :class:`~automation_file.integrity.snapshot.Snapshot`. Nothing is stored + and no baseline is needed. + * - verify + - ``monitor.verify(deep=True)`` + - Compares the tree with the baseline once and returns a + :class:`~automation_file.integrity.report.DriftReport`. + * - watch + - ``monitor.watch()`` + - Reacts to changes as they happen and returns a handle with ``stop()``. A + local target is observed through filesystem events: paths that change + within ``debounce`` seconds (0.5 by default) are verified together, and + only those paths are read. Any other backend is polled with a quick pass + every ``poll_interval`` seconds. A drift is reported when it appears, not + again while it stays the same. + * - continuous + - ``monitor.start()`` / ``monitor.stop()`` + - Verifies every ``interval`` seconds (60 by default) on a daemon thread; + the first pass runs after one interval. Every pass that finds drift + publishes an event. + +``monitor.create_baseline()`` stores a snapshot as the baseline, and +``monitor.accept(report)`` approves a drift. With a report, ``accept`` stores +exactly the tree that verification saw, so a change made after the report is not +approved unseen; without one it reads the tree again. + +``monitor.verify_paths(["a.txt", "config"])`` verifies only those paths (a +directory stands for everything below it) and marks the report ``partial``. Watch +mode runs it for the paths that changed; call it yourself when something else, +such as a bucket notification, tells you what changed. + +Watching does not verify the tree when it starts, and an operating system drops +events without notice when its queue overflows. Watch mode shortens the time to +detection; a periodic deep verification is still what proves the tree. + +Deep and quick verification +--------------------------- + +``verify(deep=True)`` hashes every file. On a remote backend that means reading +every file. + +``verify(deep=False)`` first compares the size, the modification time and the +etag of each file with the baseline and hashes only the files where one of them +differs. The report says so: ``report.deep`` is ``False``, ``report.hashed`` +counts the files that were read out of ``report.checked``, and ``report.notes`` +holds ``"quick pass: 2 of 1840 files hashed; size, modification time and etag +decided the rest"``. A change that keeps all three goes unnoticed by a quick +pass, so schedule a deep one as well. A baseline entry that recorded neither a +time nor an etag (the legacy format) is always hashed. + +A report holds ``changes``, ``counts`` (one number per kind, zeros included), +``ok``, ``deep``, ``partial``, ``checked``, ``hashed``, ``notes``, +``remediation``, ``verified_at`` and ``correlation_id``. ``report.to_dict()`` is +JSON-serialisable. + +The manifest +------------ + +A baseline is one JSON document, the *manifest*, with a schema version: + +.. code-block:: json + + { + "schema_version": 2, + "created_at": "2026-10-08T10:15:30.123456+00:00", + "root": "s3://reports/2026", + "backend": "s3", + "algorithm": "sha256", + "entries": [ + { + "path": "q1.csv", + "size": 1024, + "modified_at": "2026-10-01T08:00:00+00:00", + "checksum": "9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08", + "algorithm": "sha256", + "content_type": "text/csv", + "backend": "s3", + "version": null, + "etag": "5d41402abc4b2a76b9719d911017c592", + "mode": null + } + ] + } + +.. list-table:: + :header-rows: 1 + :widths: 22 78 + + * - Field + - Meaning + * - ``schema_version`` + - ``2``. Any other number is refused with an ``IntegrityException``, so a + document from a newer release is never half-understood. + * - ``created_at`` + - When the snapshot was taken, ISO 8601 in UTC. + * - ``root``, ``backend`` + - The URI of the tree and the scheme of the backend that served it. + * - ``algorithm`` + - The hash algorithm of every checksum. A verification hashes with it. + * - ``entries`` + - One object per file, sorted by ``path`` (relative to ``root``, ``/`` + separators). Directories are not recorded, so an empty directory is + invisible. + * - ``size``, ``modified_at``, ``content_type``, ``version``, ``etag`` + - What the backend reports. A field it cannot provide is ``null``. + * - ``mode`` + - The permission bits as an integer (``420`` is ``0o644``). Recorded only + for a file on the local filesystem. + +The manifest written by ``write_manifest`` / ``FA_write_manifest`` (no +``schema_version``, a ``files`` mapping with ``size`` and ``checksum``) is read +too and converted on the way. ``create_baseline()`` and ``accept()`` always write +version 2; after that ``verify_manifest`` refuses the file with a +``ManifestException`` that names ``FA_integrity_verify``. + +The Baseline Manager writes the manifest to a temporary sibling file and moves +it over the baseline, so a reader never sees a half-written document. The move +is a rename on the local filesystem and one whole write of the finished file +elsewhere. + +Change kinds +------------ + +.. list-table:: + :header-rows: 1 + :widths: 24 60 16 + + * - Kind + - Reported when + - Severity + * - ``created`` + - A path the baseline does not have. + - warning + * - ``modified`` + - The checksum, or the recorded size, differs. + - error + * - ``deleted`` + - A path of the baseline is gone. + - error + * - ``renamed`` + - One deleted and one created file have the same checksum and size. + ``change.previous_path`` is the old path. + - error + * - ``metadata_changed`` + - Same checksum, but the modification time, content type, version or etag + differs; ``change.fields`` names which. A field one side did not record is + not compared. + - warning + * - ``permission_changed`` + - The permission bits differ. Reported next to ``modified`` when both + happened. + - error + +When several deleted or several created files share one checksum, the pairing is +ambiguous. They are then reported as ``deleted`` and ``created``, each with +``change.note`` set to ``"ambiguous rename: 2 deleted and 1 created files share +the checksum 9f86d081884c...; reported separately"``. + +Events +------ + +Each verification that finds drift publishes one +:class:`~automation_file.events.model.IntegrityViolation` on ``event_bus``, or on +the bus passed as ``bus=``. The verification runs in a correlation scope, so the +event, the remediation events and ``report.correlation_id`` share one ID, and an +enclosing scope's ID is kept. + +.. list-table:: + :header-rows: 1 + :widths: 26 74 + + * - Field + - Value + * - ``type``, ``source`` + - ``integrity.violation``, ``integrity`` + * - ``severity`` + - The worst severity among the kinds found: ``error`` when something was + modified, deleted, renamed or had its permissions changed, ``warning`` + when there are only additions or metadata changes. + * - ``payload["resource"]``, ``["backend"]`` + - The URI of the target and its backend. + * - ``payload["status"]`` + - ``drift``, or ``error`` when the pass could not run (then + ``payload["error"]`` says why). + * - ``payload["counts"]`` + - The number of changes of each kind. + * - ``payload["changes"]`` + - The first 20 changes as ``{"kind": ..., "path": ...}``; ``total`` is the + real number and ``truncated`` says whether some were left out. + * - ``payload["baseline"]``, ``["algorithm"]``, ``["deep"]``, ``["partial"]`` + - What the pass was compared with and how thorough it was. + +Pass ``alerts=AlertPolicy(severities={"created": "error"}, max_changes=50)`` to +change the severity of a kind or the number of changes listed. + +``verify()`` raises when it cannot run. Continuous mode, watch mode and +``check_once()`` have nobody to raise to: they publish the same event with +``status: "error"``, keep the reason in ``monitor.last_error``, and go on. + +Remediation +----------- + +Off by default. Nothing is moved or copied unless a +:class:`~automation_file.integrity.remediation.RemediationPolicy` is passed to +the monitor, and a policy with every action left at ``"none"`` does nothing. + +.. code-block:: python + + RemediationPolicy( + quarantine="s3://reports-quarantine/2026", # or None + restore_from="s3://reports-mirror/2026", # or None; a mirror of the baseline + on_created="quarantine", # "none" | "quarantine" + on_modified="restore", # "none" | "quarantine" | "restore" + on_deleted="restore", # "none" | "restore" + ) + +``quarantine`` + Moves the offending file to ``//``. All + files of one pass share a timestamp directory, and nothing in the quarantine + is ever overwritten. + +``restore`` + Copies the file back from ``restore_from``. The mirror's copy is hashed + first and refused when it does not match the baseline, so a stale or + tampered mirror is never copied into place; the restored file is hashed + again before the step counts as done. A modified file is moved to the + quarantine first when one is configured. Without a quarantine its content is + overwritten. + +A rename is handled as its two halves: the old path as deleted, the new one as +created. Metadata and permission changes are never remediated. The quarantine +and the mirror must lie outside the target. + +Each step is recorded in ``report.remediation`` (``action``, ``path``, ``kind``, +``ok``, ``source``, ``destination``, ``error``) and published as an +``IntegrityRemediated`` event of type ``integrity.remediated``: ``info`` when it +worked, ``error`` when it failed. A step that fails is reported and never raised; +the file stays as it was. The report describes the tree before remediation, so +``accept(report)`` reads the tree again when steps were taken. + +A restored file has a new modification time, which the next verification reports +as ``metadata_changed`` until the baseline is accepted. + +Algorithms +---------- + +.. list-table:: + :header-rows: 1 + :widths: 30 70 + + * - Algorithm + - Use + * - ``sha256`` + - The default. + * - ``sha512``, ``blake2b`` + - Alternatives of at least the same strength. + * - ``md5``, ``sha1`` + - Refused unless ``allow_weak=True`` is passed. Both have practical + collisions: a file can be replaced by another with the same digest and + the change goes unnoticed. They exist to keep reading a baseline that was + written with one of them, never as a default. + +``algorithm=`` is what ``snapshot()`` and ``create_baseline()`` hash with. A +verification always hashes with the algorithm of the baseline it reads, and +``accept()`` keeps it. To move a baseline to another algorithm, review the tree +and call ``create_baseline()`` on a monitor created with the new one. + +Files are hashed in parallel on a thread pool (``max_workers=8`` by default) +through the storage layer's ``checksum``. + +Actions +------- + +.. list-table:: + :header-rows: 1 + :widths: 30 36 34 + + * - Action + - Parameters + - Returns + * - ``FA_integrity_snapshot`` + - ``target, algorithm="sha256"`` + - The snapshot: ``root``, ``backend``, ``algorithm``, ``created_at``, + ``entries`` + * - ``FA_integrity_baseline`` + - ``target, baseline, algorithm="sha256"`` + - ``target``, ``baseline``, ``backend``, ``algorithm``, ``created_at`` and + the number of ``entries`` + * - ``FA_integrity_verify`` + - ``target, baseline, deep=True`` + - The drift report + * - ``FA_integrity_accept`` + - ``target, baseline`` + - As ``FA_integrity_baseline`` + * - ``FA_integrity_watch_start`` + - ``name, target, baseline, interval=60.0`` + - The status of the new monitor + * - ``FA_integrity_watch_stop`` + - ``name`` + - Its last status + * - ``FA_integrity_status`` + - ``name=None`` + - A list of statuses: one monitor, or all of them + +``FA_integrity_watch_start`` keeps a named monitor in continuous mode until +``FA_integrity_watch_stop``; the baseline must exist first. A status holds +``name``, ``target``, ``baseline``, ``algorithm``, ``interval``, ``running``, +``last_run``, ``last_error`` and ``last_report``. + +.. code-block:: json + + [ + ["FA_integrity_baseline", {"target": "s3://reports/2026", + "baseline": "local:///var/lib/fa/reports-2026.json"}], + ["FA_integrity_verify", {"target": "s3://reports/2026", + "baseline": "local:///var/lib/fa/reports-2026.json", + "deep": false}], + ["FA_integrity_watch_start", {"name": "reports", "target": "s3://reports/2026", + "baseline": "local:///var/lib/fa/reports-2026.json", + "interval": 900}], + ["FA_integrity_status", {"name": "reports"}] + ] + +The actions publish drift on the process-wide ``event_bus``. They take no weak +algorithm and no remediation policy: both can only be chosen in Python. Like the +storage actions they reach whatever the process can reach, and +``FA_integrity_baseline`` and ``FA_integrity_accept`` write a file, so on a TCP +or HTTP action server pass an ``ActionACL``, and on the MCP server +``--allowed-actions``, to expose only what a client needs. +``register_integrity_ops(registry)`` adds them to a registry of your own. + +The first monitor +----------------- + +Code written for the first ``IntegrityMonitor`` keeps working: + +.. code-block:: python + + from automation_file import IntegrityMonitor, notification_manager, write_manifest + + write_manifest("/srv/site", "/srv/MANIFEST.json") + monitor = IntegrityMonitor( + "/srv/site", # root= and manifest_path= are still accepted as keywords + "/srv/MANIFEST.json", + interval=60.0, + manager=notification_manager, + on_drift=lambda summary: print("drift:", summary), + ) + summary = monitor.check_once() # {"matched": [...], "missing": [...], "modified": [...], + # "extra": [...], "ok": False} + monitor.start() + +``check_once()`` returns the same summary dictionary, with ``error`` when the +pass could not run; ``on_drift`` receives it, and ``last_summary`` keeps it. As +before, additions do not count as drift for ``on_drift`` and the notification +unless ``alert_on_extra=True``, and a rename appears as ``missing`` plus +``extra``. + +The notification goes where it went before: through the ``manager`` you pass, +or through the process-wide ``notification_manager`` when you pass none. One +thing was added: every drift, additions included, is also published as an +``IntegrityViolation`` event. If you deliver that event to your sinks yourself +(a subscriber, or a notification route), pass ``notify=False`` so one drift is +not announced twice. + +When something fails +-------------------- + +.. list-table:: + :header-rows: 1 + :widths: 40 60 + + * - What you see + - What it means and what to do + * - ``IntegrityException: no baseline at …`` + - Nothing is stored at the baseline URI. Check the URI, then call + ``create_baseline()``. A baseline that disappears from a running monitor + is itself a finding: it arrives as an event with ``status: "error"``. + * - ``… is not readable JSON`` / ``… is not a valid manifest`` + - The baseline is damaged or was edited. Restore it from a copy, or review + the tree and create it again. Do not accept a tree you cannot compare. + * - ``… has manifest schema version 3`` + - The baseline was written by a newer release. Upgrade, or create the + baseline again with this one. + * - ``md5 is refused for integrity checks …`` + - The baseline uses a weak algorithm. Pass ``allow_weak=True`` to read it, + then call ``create_baseline()`` with ``sha256``. + * - ``target … does not exist`` + - The directory is missing on a filesystem. On an object store a prefix + that holds nothing is an empty tree instead: every file is ``deleted``. + * - ``StorageUnavailableException`` + - The backend is not initialised: call ``s3_instance.later_init(...)`` or + the equivalent first. + * - ``StoragePermissionException`` or ``StorageTransientException`` during a + pass + - The pass fails as a whole; a tree that could not be read completely is + never reported as clean. Continuous mode tries again after ``interval``. + * - A quick pass is clean, a deep pass reports ``modified`` + - The content changed while size and modification time stayed the same. + Ordinary tools do not do that; treat it as tampering. + * - Everything is ``metadata_changed`` + - The files were copied or restored, which gives them new times. Review, + then ``accept()``. + * - A remediation step has ``ok: False`` + - ``step.error`` says why (no copy in the mirror, a mirror that does not + match the baseline, access denied). The file was left as it was; an + ``integrity.remediated`` event of severity ``error`` was published. + * - The same event on every interval + - Continuous mode reports a drift on every pass for as long as it lasts. + Fix the tree or ``accept()`` it, or deduplicate in the subscriber (the + ``NotificationManager`` does). + * - Watch mode missed a change + - Filesystem events can be lost. Run a deep ``verify()`` on a schedule next + to the watch. + +One monitor runs one pass at a time: a ``verify()`` called while the continuous +thread is in a pass waits for it. diff --git a/docs/source/Zh-CN/usage/integrity.rst b/docs/source/Zh-CN/usage/integrity.rst new file mode 100644 index 0000000..afc481c --- /dev/null +++ b/docs/source/Zh-CN/usage/integrity.rst @@ -0,0 +1,472 @@ +文件完整性监控 +============== + +``automation_file.integrity`` 只回答一个关于目录树的问题：它是否仍然是当初批准的 +样子？:class:`~automation_file.integrity.monitor.IntegrityMonitor` 把批准的状态记录 +为 *基线*\ （baseline），拿目录树与它比较，并把每一处差异报告为六种变更之一。 + +它通过\ :doc:`存储层 `\ 读取，所以目录树与基线都是存储 URI，可以放在任何 +后端。偏移（drift）会以事件的形式发布（见 :doc:`event_bus`）；监控器本身绝不调用通知 +接收端。除非你传入补救策略，否则它只会读取。 + +``IntegrityMonitor`` 仍然可以从 ``automation_file`` 与 ``automation_file.core.fim`` +导入，为第一代监控器写的调用方式也照常工作（见 `第一代监控器`_）。 + +最小示例 +-------- + +.. code-block:: python + + from automation_file.integrity import IntegrityMonitor + + monitor = IntegrityMonitor("/srv/site", baseline="/var/lib/fa/site.baseline.json") + monitor.create_baseline() # 批准当前的内容 + + report = monitor.verify() # 对每个文件计算哈希，与基线比较 + if not report.ok: + print(report.counts) # {'created': 0, 'modified': 1, 'deleted': 0, ...} + for change in report.changes: + print(change.kind.value, change.path) + monitor.accept(report) # 审查之后：批准这份报告所看到的状态 + +生产环境示例 +------------ + +基线放在它所描述的 bucket 之外，监控器在线程上验证，事件被转发到通知接收端，补救则 +显式开启。 + +.. code-block:: python + + import json + + from automation_file import Severity, SlackSink, event_bus, notification_manager, s3_instance + from automation_file.integrity import IntegrityMonitor, RemediationPolicy + + s3_instance.later_init(region_name="eu-west-1") + notification_manager.register(SlackSink(slack_webhook_url)) + + def route(event): + level = "error" if event.severity.at_least(Severity.ERROR) else "warning" + details = event.payload.get("error") or json.dumps(event.payload.get("counts", {})) + notification_manager.notify(event.subject, details, level) + + # integrity.violation 与 integrity.remediated；补救成功的事件是 "info"。 + event_bus.subscribe(route, types="integrity.*", min_severity=Severity.WARNING) + + monitor = IntegrityMonitor( + "s3://reports/2026", + baseline="local:///var/lib/fa/baselines/reports-2026.json", + algorithm="sha256", + interval=900, # 持续模式：每 15 分钟一次 + remediation=RemediationPolicy( # 需显式开启；不传就不会更改任何东西 + quarantine="s3://reports-quarantine/2026", + restore_from="s3://reports-mirror/2026", + on_created="quarantine", + on_modified="restore", + on_deleted="restore", + ), + ) + if not monitor.has_baseline(): + monitor.create_baseline() + + monitor.start() # 守护线程；立即返回 + ... + monitor.status() # running、last_run、last_error、last_report + monitor.stop() + +请把基线放在“能改动目录树的人改不到”的地方。把基线放在目标之内也能工作，监控器会 +把那个文件排除在快照之外，但这样一来同一份写入权限就同时覆盖两者。 + +四种模式 +-------- + +.. list-table:: + :header-rows: 1 + :widths: 16 30 54 + + * - 模式 + - 调用 + - 作用 + * - snapshot（快照） + - ``monitor.snapshot()`` + - 读取目录树并返回 + :class:`~automation_file.integrity.snapshot.Snapshot`。不存储任何东西， + 也不需要基线。 + * - verify（验证） + - ``monitor.verify(deep=True)`` + - 拿目录树与基线比较一次，返回 + :class:`~automation_file.integrity.report.DriftReport`。 + * - watch（监视） + - ``monitor.watch()`` + - 在变更发生时即时响应，返回带有 ``stop()`` 的句柄。本地目标通过文件系统事件 + 观察：在 ``debounce`` 秒（默认 0.5）之内变更的路径会一起验证，而且只读取这些 + 路径。其他后端则每隔 ``poll_interval`` 秒以快速验证轮询一次。偏移在出现时 + 报告一次，保持不变期间不会重复报告。 + * - continuous（持续） + - ``monitor.start()`` / ``monitor.stop()`` + - 在守护线程上每隔 ``interval`` 秒（默认 60）验证一次；第一次验证在经过一个 + 间隔之后执行。每一次发现偏移的验证都会发布一个事件。 + +``monitor.create_baseline()`` 把快照存为基线，``monitor.accept(report)`` 则批准 +一次偏移。传入报告时，``accept`` 存下的正是那次验证所看到的目录树，因此报告之后才 +发生的变更不会在没人看过的情况下被批准；不传报告时则重新读取目录树。 + +``monitor.verify_paths(["a.txt", "config"])`` 只验证这些路径（目录代表其下的所有 +文件），并把报告标记为 ``partial``。监视模式就是对变更的路径执行它；当有其他来源 +（例如 bucket 通知）告诉你哪些东西变了，也可以自己调用。 + +监视模式启动时不会先验证整棵目录树，而且操作系统的事件队列溢出时会悄悄丢弃事件。 +监视模式缩短的是检测所需的时间；真正能证明目录树完好的，仍然是定期的深度验证。 + +深度验证与快速验证 +------------------ + +``verify(deep=True)`` 会对每一个文件计算哈希。在远程后端上，这意味着要读取每一个 +文件。 + +``verify(deep=False)`` 会先拿每个文件的大小、修改时间与 etag 与基线比较，只对其中 +任何一项不同的文件计算哈希。报告会说明这一点：``report.deep`` 为 ``False``， +``report.hashed`` 是 ``report.checked`` 个文件中实际被读取的数量，``report.notes`` +则包含 ``"quick pass: 2 of 1840 files hashed; size, modification time and etag +decided the rest"``。三项都没变的变更，快速验证察觉不到，所以也请安排深度验证。 +基线条目若既没有记录时间也没有记录 etag（旧格式），一律会被计算哈希。 + +报告包含 ``changes``、``counts``\ （每种变更一个数字，包含零）、``ok``、``deep``、 +``partial``、``checked``、``hashed``、``notes``、``remediation``、``verified_at`` +与 ``correlation_id``。``report.to_dict()`` 的结果可直接序列化为 JSON。 + +Manifest 格式 +------------- + +基线是一份带有结构版本的 JSON 文档，称为 *manifest*： + +.. code-block:: json + + { + "schema_version": 2, + "created_at": "2026-10-08T10:15:30.123456+00:00", + "root": "s3://reports/2026", + "backend": "s3", + "algorithm": "sha256", + "entries": [ + { + "path": "q1.csv", + "size": 1024, + "modified_at": "2026-10-01T08:00:00+00:00", + "checksum": "9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08", + "algorithm": "sha256", + "content_type": "text/csv", + "backend": "s3", + "version": null, + "etag": "5d41402abc4b2a76b9719d911017c592", + "mode": null + } + ] + } + +.. list-table:: + :header-rows: 1 + :widths: 22 78 + + * - 字段 + - 含义 + * - ``schema_version`` + - ``2``。其他数字一律以 ``IntegrityException`` 拒绝，因此较新版本写出的文档绝 + 不会被一知半解地读取。 + * - ``created_at`` + - 快照的创建时间，UTC 的 ISO 8601 格式。 + * - ``root``、``backend`` + - 目录树的 URI，以及提供它的后端的 scheme。 + * - ``algorithm`` + - 所有校验码使用的哈希算法。验证时就用它来计算哈希。 + * - ``entries`` + - 每个文件一个对象，按 ``path`` 排序（相对于 ``root``，以 ``/`` 分隔）。目录 + 不会被记录，因此空目录是看不见的。 + * - ``size``、``modified_at``、``content_type``、``version``、``etag`` + - 后端报告的内容。后端无法提供的字段为 ``null``。 + * - ``mode`` + - 以整数表示的权限位（``420`` 即 ``0o644``）。只有本地文件系统上的文件才会 + 记录。 + +``write_manifest`` / ``FA_write_manifest`` 写出的 manifest（没有 +``schema_version``，以 ``files`` 映射记录 ``size`` 与 ``checksum``）同样可以读取， +并在读取时转换。``create_baseline()`` 与 ``accept()`` 一律写出第 2 版；之后 +``verify_manifest`` 会以 ``ManifestException`` 拒绝该文件，并在消息中指出 +``FA_integrity_verify``。 + +基线管理器会先把 manifest 写到同目录的临时文件，再把它移过去取代基线，所以读取方永远 +不会看到写到一半的文档。这个移动在本地文件系统上是重命名，在其他后端则是把完成的 +文件整个写入一次。 + +变更种类 +-------- + +.. list-table:: + :header-rows: 1 + :widths: 24 60 16 + + * - 种类 + - 报告时机 + - 严重程度 + * - ``created`` + - 基线中没有的路径。 + - warning + * - ``modified`` + - 校验码或记录的大小不同。 + - error + * - ``deleted`` + - 基线中的路径不见了。 + - error + * - ``renamed`` + - 一个被删除的文件与一个新创建的文件有相同的校验码与大小。 + ``change.previous_path`` 是旧路径。 + - error + * - ``metadata_changed`` + - 校验码相同，但修改时间、内容类型、版本或 etag 不同；``change.fields`` 会 + 指出是哪些。任何一边没有记录的字段不会被比较。 + - warning + * - ``permission_changed`` + - 权限位不同。两者都发生时，会与 ``modified`` 一并报告。 + - error + +当多个被删除或多个新创建的文件共用同一个校验码时，配对就有歧义。此时它们会被报告为 +``deleted`` 与 ``created``，各自的 ``change.note`` 会设为 ``"ambiguous rename: 2 +deleted and 1 created files share the checksum 9f86d081884c...; reported +separately"``。 + +事件 +---- + +每一次发现偏移的验证都会在 ``event_bus``\ （或以 ``bus=`` 传入的事件总线）上发布 +一个 :class:`~automation_file.events.model.IntegrityViolation`。验证在关联范围 +（correlation scope）内执行，所以这个事件、补救事件与 ``report.correlation_id`` 共用 +同一个 ID；若外层已有范围，则沿用外层的 ID。 + +.. list-table:: + :header-rows: 1 + :widths: 26 74 + + * - 字段 + - 值 + * - ``type``、``source`` + - ``integrity.violation``、``integrity`` + * - ``severity`` + - 所发现的变更种类中最严重的一级：有东西被修改、删除、重命名或权限被更改时 + 为 ``error``；只有新增或元数据变更时为 ``warning``。 + * - ``payload["resource"]``、``["backend"]`` + - 目标的 URI 与它的后端。 + * - ``payload["status"]`` + - ``drift``；验证无法执行时为 ``error``\ （此时 ``payload["error"]`` 说明原因）。 + * - ``payload["counts"]`` + - 每种变更的数量。 + * - ``payload["changes"]`` + - 前 20 条变更，格式为 ``{"kind": ..., "path": ...}``；``total`` 是实际数量， + ``truncated`` 表示是否有省略。 + * - ``payload["baseline"]``、``["algorithm"]``、``["deep"]``、``["partial"]`` + - 这次验证比较的对象，以及它有多彻底。 + +传入 ``alerts=AlertPolicy(severities={"created": "error"}, max_changes=50)`` 可以 +改变某种变更的严重程度，或事件中列出的变更数量。 + +``verify()`` 无法执行时会抛出异常。持续模式、监视模式与 ``check_once()`` 没有对象 +可以抛出：它们会发布同一种事件并带有 ``status: "error"``，把原因留在 +``monitor.last_error``，然后继续执行。 + +补救 +---- + +默认关闭。除非把 :class:`~automation_file.integrity.remediation.RemediationPolicy` +传给监控器，否则不会移动或复制任何东西；而所有动作都保持 ``"none"`` 的策略同样什么 +都不做。 + +.. code-block:: python + + RemediationPolicy( + quarantine="s3://reports-quarantine/2026", # 或 None + restore_from="s3://reports-mirror/2026", # 或 None；基线的镜像 + on_created="quarantine", # "none" | "quarantine" + on_modified="restore", # "none" | "quarantine" | "restore" + on_deleted="restore", # "none" | "restore" + ) + +``quarantine``\ （隔离） + 把有问题的文件移到 ``//``。同一次验证的所有文件 + 共用一个时间戳目录，而且隔离区中的任何东西都不会被覆盖。 + +``restore``\ （还原） + 从 ``restore_from`` 把文件复制回来。镜像中的副本会先被计算哈希，与基线不符就 + 拒绝，因此过期或被篡改的镜像绝不会被复制到原位；还原后的文件会再计算一次哈希， + 通过之后这个步骤才算完成。若配置了隔离区，被修改的文件会先移进隔离区；没有 + 隔离区时，它的内容会被覆盖。 + +重命名会拆成两半处理：旧路径视为被删除，新路径视为新创建。元数据与权限的变更绝不会 +被补救。隔离区与镜像都必须位于目标之外。 + +每个步骤都会记录在 ``report.remediation``\ （``action``、``path``、``kind``、 +``ok``、``source``、``destination``、``error``），并以类型为 +``integrity.remediated`` 的 ``IntegrityRemediated`` 事件发布：成功时为 ``info``， +失败时为 ``error``。失败的步骤只会被报告，绝不会抛出异常；文件保持原状。报告描述的 +是补救之前的目录树，因此执行过补救步骤时，``accept(report)`` 会重新读取目录树。 + +还原后的文件有新的修改时间，下一次验证会把它报告为 ``metadata_changed``，直到基线被 +批准为止。 + +算法 +---- + +.. list-table:: + :header-rows: 1 + :widths: 30 70 + + * - 算法 + - 用途 + * - ``sha256`` + - 默认值。 + * - ``sha512``、``blake2b`` + - 强度至少相同的替代选择。 + * - ``md5``、``sha1`` + - 除非传入 ``allow_weak=True``，否则一律拒绝。两者都有实际可行的碰撞：文件可以 + 被换成另一个摘要相同的文件，而这个变更不会被察觉。保留它们只是为了继续读取 + 当初以它们写出的基线，绝不会作为默认值。 + +``algorithm=`` 是 ``snapshot()`` 与 ``create_baseline()`` 使用的哈希算法。验证一律 +使用所读取基线的算法，``accept()`` 也会沿用它。要把基线换成另一种算法，请先审查 +目录树，再用以新算法创建的监控器调用 ``create_baseline()``。 + +文件通过存储层的 ``checksum``，在线程池上并行计算哈希（默认 ``max_workers=8``）。 + +动作 +---- + +.. list-table:: + :header-rows: 1 + :widths: 30 36 34 + + * - 动作 + - 参数 + - 返回值 + * - ``FA_integrity_snapshot`` + - ``target, algorithm="sha256"`` + - 快照：``root``、``backend``、``algorithm``、``created_at``、``entries`` + * - ``FA_integrity_baseline`` + - ``target, baseline, algorithm="sha256"`` + - ``target``、``baseline``、``backend``、``algorithm``、``created_at`` 以及 + ``entries`` 的数量 + * - ``FA_integrity_verify`` + - ``target, baseline, deep=True`` + - 偏移报告 + * - ``FA_integrity_accept`` + - ``target, baseline`` + - 与 ``FA_integrity_baseline`` 相同 + * - ``FA_integrity_watch_start`` + - ``name, target, baseline, interval=60.0`` + - 新监控器的状态 + * - ``FA_integrity_watch_stop`` + - ``name`` + - 它最后的状态 + * - ``FA_integrity_status`` + - ``name=None`` + - 状态的列表：单个监控器，或全部 + +``FA_integrity_watch_start`` 让一个具名的监控器保持在持续模式，直到 +``FA_integrity_watch_stop``；基线必须先存在。状态包含 ``name``、``target``、 +``baseline``、``algorithm``、``interval``、``running``、``last_run``、 +``last_error`` 与 ``last_report``。 + +.. code-block:: json + + [ + ["FA_integrity_baseline", {"target": "s3://reports/2026", + "baseline": "local:///var/lib/fa/reports-2026.json"}], + ["FA_integrity_verify", {"target": "s3://reports/2026", + "baseline": "local:///var/lib/fa/reports-2026.json", + "deep": false}], + ["FA_integrity_watch_start", {"name": "reports", "target": "s3://reports/2026", + "baseline": "local:///var/lib/fa/reports-2026.json", + "interval": 900}], + ["FA_integrity_status", {"name": "reports"}] + ] + +这些动作会把偏移发布到整个进程共用的 ``event_bus``。它们不接受弱算法，也不接受 +补救策略：这两者都只能在 Python 中选用。与存储动作一样，它们能访问进程所能访问的 +一切，而且 ``FA_integrity_baseline`` 与 ``FA_integrity_accept`` 会写入文件，因此在 +TCP 或 HTTP 动作服务器上请传入 ``ActionACL``，在 MCP 服务器上请使用 +``--allowed-actions``，只开放客户端需要的动作。 +``register_integrity_ops(registry)`` 可把它们加入你自己的注册表。 + +第一代监控器 +------------ + +为第一代 ``IntegrityMonitor`` 写的代码照常工作： + +.. code-block:: python + + from automation_file import IntegrityMonitor, notification_manager, write_manifest + + write_manifest("/srv/site", "/srv/MANIFEST.json") + monitor = IntegrityMonitor( + "/srv/site", # 仍然接受 root= 与 manifest_path= 这两个关键字 + "/srv/MANIFEST.json", + interval=60.0, + manager=notification_manager, + on_drift=lambda summary: print("drift:", summary), + ) + summary = monitor.check_once() # {"matched": [...], "missing": [...], "modified": [...], + # "extra": [...], "ok": False} + monitor.start() + +``check_once()`` 返回同样的摘要字典，验证无法执行时会带有 ``error``；``on_drift`` +会收到它，``last_summary`` 会保留它。与以往一样，除非 ``alert_on_extra=True``，否则 +新增的文件对 ``on_drift`` 与通知而言不算偏移，而重命名会以 ``missing`` 加上 +``extra`` 的形式出现。 + +通知的去向与以往相同：通过你传入的 ``manager`` 发送，没有传入时则使用整个进程共用的 +``notification_manager``。新增的只有一点：每一次偏移（包含新增）也都会以 +``IntegrityViolation`` 事件的形式发布。如果你改由这个事件（订阅者或通知路由）把偏移 +送到通知渠道，请传入 ``notify=False``，同一次偏移才不会被通知两次。 + +出现问题时 +---------- + +.. list-table:: + :header-rows: 1 + :widths: 40 60 + + * - 你看到的现象 + - 它的含义与处理方式 + * - ``IntegrityException: no baseline at …`` + - 基线 URI 上没有任何东西。请检查 URI，然后调用 ``create_baseline()``。运行中 + 的监控器的基线消失，本身就是一项发现：它会以带有 ``status: "error"`` 的事件 + 送达。 + * - ``… is not readable JSON`` / ``… is not a valid manifest`` + - 基线已损坏或被编辑过。请从副本还原，或审查目录树之后重新创建。不要批准一棵 + 你无从比较的目录树。 + * - ``… has manifest schema version 3`` + - 基线是由较新的版本写出的。请升级，或用当前的版本重新创建基线。 + * - ``md5 is refused for integrity checks …`` + - 基线使用了弱算法。传入 ``allow_weak=True`` 以便读取，再以 ``sha256`` 调用 + ``create_baseline()``。 + * - ``target … does not exist`` + - 文件系统上的目录不见了。在对象存储上，什么都没有的前缀则是一棵空的目录树： + 每个文件都是 ``deleted``。 + * - ``StorageUnavailableException`` + - 后端尚未初始化：请先调用 ``s3_instance.later_init(...)`` 或对应的函数。 + * - 验证过程中出现 ``StoragePermissionException`` 或 + ``StorageTransientException`` + - 整次验证失败；无法完整读取的目录树绝不会被报告为完好。持续模式会在 + ``interval`` 之后再试一次。 + * - 快速验证没问题，深度验证却报告 ``modified`` + - 内容变了，大小与修改时间却没变。一般工具不会这样做；请视为篡改。 + * - 所有文件都是 ``metadata_changed`` + - 文件被复制或还原过，因而有了新的时间。审查之后调用 ``accept()``。 + * - 补救步骤的 ``ok`` 为 ``False`` + - ``step.error`` 说明原因（镜像中没有副本、镜像与基线不符、访问被拒）。文件 + 保持原状；并已发布严重程度为 ``error`` 的 ``integrity.remediated`` 事件。 + * - 每个间隔都收到同一个事件 + - 只要偏移还在，持续模式每次验证都会报告。请修正目录树或调用 ``accept()``， + 或在订阅方去除重复（``NotificationManager`` 会这么做）。 + * - 监视模式漏掉了某个变更 + - 文件系统事件可能丢失。请在监视之外，另外定期执行深度的 ``verify()``。 + +一个监控器一次只执行一次验证：在持续模式的线程正在验证时调用 ``verify()``，会等它 +完成。 diff --git a/docs/source/Zh-CN/zh_cn_index.rst b/docs/source/Zh-CN/zh_cn_index.rst index d901a28..9c26030 100644 --- a/docs/source/Zh-CN/zh_cn_index.rst +++ b/docs/source/Zh-CN/zh_cn_index.rst @@ -266,3 +266,17 @@ Slack、Email（SMTP）、Discord、Telegram、Microsoft Teams、PagerDuty :caption: 事件 usage/event_bus + +.. _zh-cn-integrity: + +第 18 章 — 文件完整性监控 +========================= + +``IntegrityMonitor`` 为任何存储后端中的目录树保存一份经过核准的基准，并报告 +与基准不符之处：快照、manifest 结构、四种模式、告警，以及可选的修复。 + +.. toctree:: + :maxdepth: 2 + :caption: 文件完整性监控 + + usage/integrity diff --git a/docs/source/Zh-TW/usage/integrity.rst b/docs/source/Zh-TW/usage/integrity.rst new file mode 100644 index 0000000..4e872be --- /dev/null +++ b/docs/source/Zh-TW/usage/integrity.rst @@ -0,0 +1,471 @@ +檔案完整性監控 +============== + +``automation_file.integrity`` 只回答一個關於目錄樹的問題：它是否仍然是當初核可的 +樣子？:class:`~automation_file.integrity.monitor.IntegrityMonitor` 把核可的狀態記錄 +為 *基準*\ （baseline），拿目錄樹與它比對，並把每一處差異回報為六種變更之一。 + +它透過\ :doc:`儲存層 `\ 讀取，所以目錄樹與基準都是儲存 URI，可以放在任何 +後端。偏移（drift）會以事件的形式發布（見 :doc:`event_bus`）；監控器本身絕不呼叫通知 +接收端。除非你傳入補救政策，否則它只會讀取。 + +``IntegrityMonitor`` 仍然可以從 ``automation_file`` 與 ``automation_file.core.fim`` +匯入，為第一代監控器寫的呼叫方式也照常運作（見 `第一代監控器`_）。 + +最小範例 +-------- + +.. code-block:: python + + from automation_file.integrity import IntegrityMonitor + + monitor = IntegrityMonitor("/srv/site", baseline="/var/lib/fa/site.baseline.json") + monitor.create_baseline() # 核可目前的內容 + + report = monitor.verify() # 雜湊每個檔案，與基準比對 + if not report.ok: + print(report.counts) # {'created': 0, 'modified': 1, 'deleted': 0, ...} + for change in report.changes: + print(change.kind.value, change.path) + monitor.accept(report) # 檢視之後：核可這份報告所看到的狀態 + +正式環境範例 +------------ + +基準放在它所描述的 bucket 之外，監控器在執行緒上驗證，事件被導向通知接收端，補救則 +明確地開啟。 + +.. code-block:: python + + import json + + from automation_file import Severity, SlackSink, event_bus, notification_manager, s3_instance + from automation_file.integrity import IntegrityMonitor, RemediationPolicy + + s3_instance.later_init(region_name="eu-west-1") + notification_manager.register(SlackSink(slack_webhook_url)) + + def route(event): + level = "error" if event.severity.at_least(Severity.ERROR) else "warning" + details = event.payload.get("error") or json.dumps(event.payload.get("counts", {})) + notification_manager.notify(event.subject, details, level) + + # integrity.violation 與 integrity.remediated；補救成功的事件是 "info"。 + event_bus.subscribe(route, types="integrity.*", min_severity=Severity.WARNING) + + monitor = IntegrityMonitor( + "s3://reports/2026", + baseline="local:///var/lib/fa/baselines/reports-2026.json", + algorithm="sha256", + interval=900, # 持續模式：每 15 分鐘一次 + remediation=RemediationPolicy( # 需明確開啟；不傳就不會變更任何東西 + quarantine="s3://reports-quarantine/2026", + restore_from="s3://reports-mirror/2026", + on_created="quarantine", + on_modified="restore", + on_deleted="restore", + ), + ) + if not monitor.has_baseline(): + monitor.create_baseline() + + monitor.start() # 常駐執行緒；立即返回 + ... + monitor.status() # running、last_run、last_error、last_report + monitor.stop() + +請把基準放在「能改動目錄樹的人改不到」的地方。把基準放在目標之內也能運作，監控器會 +把那個檔案排除在快照之外，但這樣一來同一份寫入權限就同時涵蓋兩者。 + +四種模式 +-------- + +.. list-table:: + :header-rows: 1 + :widths: 16 30 54 + + * - 模式 + - 呼叫 + - 作用 + * - snapshot（快照） + - ``monitor.snapshot()`` + - 讀取目錄樹並回傳 + :class:`~automation_file.integrity.snapshot.Snapshot`。不儲存任何東西， + 也不需要基準。 + * - verify（驗證） + - ``monitor.verify(deep=True)`` + - 拿目錄樹與基準比對一次，回傳 + :class:`~automation_file.integrity.report.DriftReport`。 + * - watch（監看） + - ``monitor.watch()`` + - 在變更發生時即時反應，回傳帶有 ``stop()`` 的控制代碼。本機目標透過檔案系統 + 事件觀察：在 ``debounce`` 秒（預設 0.5）之內變更的路徑會一起驗證，而且只讀取 + 這些路徑。其他後端則每隔 ``poll_interval`` 秒以快速驗證輪詢一次。偏移在出現時 + 回報一次，維持不變期間不會重複回報。 + * - continuous（持續） + - ``monitor.start()`` / ``monitor.stop()`` + - 在常駐執行緒上每隔 ``interval`` 秒（預設 60）驗證一次；第一次驗證在經過一個 + 間隔之後執行。每一次發現偏移的驗證都會發布一個事件。 + +``monitor.create_baseline()`` 把快照存為基準，``monitor.accept(report)`` 則核可 +一次偏移。傳入報告時，``accept`` 存下的正是那次驗證所看到的目錄樹，因此報告之後才 +發生的變更不會在沒人看過的情況下被核可；不傳報告時則重新讀取目錄樹。 + +``monitor.verify_paths(["a.txt", "config"])`` 只驗證這些路徑（目錄代表其下的所有 +檔案），並把報告標記為 ``partial``。監看模式就是對變更的路徑執行它；當有其他來源 +（例如 bucket 通知）告訴你哪些東西變了，也可以自己呼叫。 + +監看模式啟動時不會先驗證整棵目錄樹，而且作業系統的事件佇列溢位時會無聲地丟棄事件。 +監看模式縮短的是偵測所需的時間；真正能證明目錄樹完好的，仍然是定期的深度驗證。 + +深度驗證與快速驗證 +------------------ + +``verify(deep=True)`` 會雜湊每一個檔案。在遠端後端上，這表示要讀取每一個檔案。 + +``verify(deep=False)`` 會先拿每個檔案的大小、修改時間與 etag 與基準比對，只雜湊其中 +任何一項不同的檔案。報告會說明這一點：``report.deep`` 為 ``False``， +``report.hashed`` 是 ``report.checked`` 個檔案中實際被讀取的數量，``report.notes`` +則包含 ``"quick pass: 2 of 1840 files hashed; size, modification time and etag +decided the rest"``。三項都沒變的變更，快速驗證察覺不到，所以也請排定深度驗證。 +基準項目若既沒有記錄時間也沒有記錄 etag（舊格式），一律會被雜湊。 + +報告包含 ``changes``、``counts``\ （每種變更一個數字，包含零）、``ok``、``deep``、 +``partial``、``checked``、``hashed``、``notes``、``remediation``、``verified_at`` +與 ``correlation_id``。``report.to_dict()`` 的結果可直接序列化為 JSON。 + +Manifest 格式 +------------- + +基準是一份帶有結構版本的 JSON 文件，稱為 *manifest*： + +.. code-block:: json + + { + "schema_version": 2, + "created_at": "2026-10-08T10:15:30.123456+00:00", + "root": "s3://reports/2026", + "backend": "s3", + "algorithm": "sha256", + "entries": [ + { + "path": "q1.csv", + "size": 1024, + "modified_at": "2026-10-01T08:00:00+00:00", + "checksum": "9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08", + "algorithm": "sha256", + "content_type": "text/csv", + "backend": "s3", + "version": null, + "etag": "5d41402abc4b2a76b9719d911017c592", + "mode": null + } + ] + } + +.. list-table:: + :header-rows: 1 + :widths: 22 78 + + * - 欄位 + - 意義 + * - ``schema_version`` + - ``2``。其他數字一律以 ``IntegrityException`` 拒絕，因此較新版本寫出的文件絕 + 不會被一知半解地讀取。 + * - ``created_at`` + - 快照的建立時間，UTC 的 ISO 8601 格式。 + * - ``root``、``backend`` + - 目錄樹的 URI，以及提供它的後端的 scheme。 + * - ``algorithm`` + - 所有校驗碼使用的雜湊演算法。驗證時就用它來雜湊。 + * - ``entries`` + - 每個檔案一個物件，依 ``path`` 排序（相對於 ``root``，以 ``/`` 分隔）。目錄 + 不會被記錄，因此空目錄是看不見的。 + * - ``size``、``modified_at``、``content_type``、``version``、``etag`` + - 後端回報的內容。後端無法提供的欄位為 ``null``。 + * - ``mode`` + - 以整數表示的權限位元（``420`` 即 ``0o644``）。只有本機檔案系統上的檔案才會 + 記錄。 + +``write_manifest`` / ``FA_write_manifest`` 寫出的 manifest（沒有 +``schema_version``，以 ``files`` 對應表記錄 ``size`` 與 ``checksum``）同樣可以讀取， +並在讀取時轉換。``create_baseline()`` 與 ``accept()`` 一律寫出第 2 版；之後 +``verify_manifest`` 會以 ``ManifestException`` 拒絕該檔案，並在訊息中指出 +``FA_integrity_verify``。 + +基準管理員會先把 manifest 寫到同目錄的暫存檔，再把它移過去取代基準，所以讀取端永遠 +不會看到寫到一半的文件。這個移動在本機檔案系統上是重新命名，在其他後端則是把完成的 +檔案整個寫入一次。 + +變更種類 +-------- + +.. list-table:: + :header-rows: 1 + :widths: 24 60 16 + + * - 種類 + - 回報時機 + - 嚴重程度 + * - ``created`` + - 基準中沒有的路徑。 + - warning + * - ``modified`` + - 校驗碼或記錄的大小不同。 + - error + * - ``deleted`` + - 基準中的路徑不見了。 + - error + * - ``renamed`` + - 一個被刪除的檔案與一個新建立的檔案有相同的校驗碼與大小。 + ``change.previous_path`` 是舊路徑。 + - error + * - ``metadata_changed`` + - 校驗碼相同，但修改時間、內容類型、版本或 etag 不同；``change.fields`` 會 + 指出是哪些。任何一邊沒有記錄的欄位不會被比對。 + - warning + * - ``permission_changed`` + - 權限位元不同。兩者都發生時，會與 ``modified`` 一併回報。 + - error + +當多個被刪除或多個新建立的檔案共用同一個校驗碼時，配對就有歧義。此時它們會被回報為 +``deleted`` 與 ``created``，各自的 ``change.note`` 會設為 ``"ambiguous rename: 2 +deleted and 1 created files share the checksum 9f86d081884c...; reported +separately"``。 + +事件 +---- + +每一次發現偏移的驗證都會在 ``event_bus``\ （或以 ``bus=`` 傳入的事件匯流排）上發布 +一個 :class:`~automation_file.events.model.IntegrityViolation`。驗證在關聯範圍 +（correlation scope）內執行，所以這個事件、補救事件與 ``report.correlation_id`` 共用 +同一個 ID；若外層已有範圍，則沿用外層的 ID。 + +.. list-table:: + :header-rows: 1 + :widths: 26 74 + + * - 欄位 + - 值 + * - ``type``、``source`` + - ``integrity.violation``、``integrity`` + * - ``severity`` + - 所發現的變更種類中最嚴重的一級：有東西被修改、刪除、重新命名或權限被變更時 + 為 ``error``；只有新增或中繼資料變更時為 ``warning``。 + * - ``payload["resource"]``、``["backend"]`` + - 目標的 URI 與它的後端。 + * - ``payload["status"]`` + - ``drift``；驗證無法執行時為 ``error``\ （此時 ``payload["error"]`` 說明原因）。 + * - ``payload["counts"]`` + - 每種變更的數量。 + * - ``payload["changes"]`` + - 前 20 筆變更，格式為 ``{"kind": ..., "path": ...}``；``total`` 是實際數量， + ``truncated`` 表示是否有省略。 + * - ``payload["baseline"]``、``["algorithm"]``、``["deep"]``、``["partial"]`` + - 這次驗證比對的對象，以及它有多徹底。 + +傳入 ``alerts=AlertPolicy(severities={"created": "error"}, max_changes=50)`` 可以 +改變某種變更的嚴重程度，或事件中列出的變更數量。 + +``verify()`` 無法執行時會拋出例外。持續模式、監看模式與 ``check_once()`` 沒有對象 +可以拋出：它們會發布同一種事件並帶有 ``status: "error"``，把原因留在 +``monitor.last_error``，然後繼續執行。 + +補救 +---- + +預設關閉。除非把 :class:`~automation_file.integrity.remediation.RemediationPolicy` +傳給監控器，否則不會搬移或複製任何東西；而所有動作都維持 ``"none"`` 的政策同樣什麼 +都不做。 + +.. code-block:: python + + RemediationPolicy( + quarantine="s3://reports-quarantine/2026", # 或 None + restore_from="s3://reports-mirror/2026", # 或 None；基準的鏡像 + on_created="quarantine", # "none" | "quarantine" + on_modified="restore", # "none" | "quarantine" | "restore" + on_deleted="restore", # "none" | "restore" + ) + +``quarantine``\ （隔離） + 把有問題的檔案搬到 ``//``。同一次驗證的所有檔案 + 共用一個時間戳記目錄，而且隔離區中的任何東西都不會被覆寫。 + +``restore``\ （還原） + 從 ``restore_from`` 把檔案複製回來。鏡像中的副本會先被雜湊，與基準不符就拒絕， + 因此過期或被竄改的鏡像絕不會被複製到原位；還原後的檔案會再雜湊一次，通過之後 + 這個步驟才算完成。若設定了隔離區，被修改的檔案會先搬進隔離區；沒有隔離區時， + 它的內容會被覆寫。 + +重新命名會拆成兩半處理：舊路徑視為被刪除，新路徑視為新建立。中繼資料與權限的變更 +絕不會被補救。隔離區與鏡像都必須位於目標之外。 + +每個步驟都會記錄在 ``report.remediation``\ （``action``、``path``、``kind``、 +``ok``、``source``、``destination``、``error``），並以型別為 +``integrity.remediated`` 的 ``IntegrityRemediated`` 事件發布：成功時為 ``info``， +失敗時為 ``error``。失敗的步驟只會被回報，絕不會拋出例外；檔案維持原狀。報告描述的 +是補救之前的目錄樹，因此有執行過補救步驟時，``accept(report)`` 會重新讀取目錄樹。 + +還原後的檔案有新的修改時間，下一次驗證會把它回報為 ``metadata_changed``，直到基準被 +核可為止。 + +演算法 +------ + +.. list-table:: + :header-rows: 1 + :widths: 30 70 + + * - 演算法 + - 用途 + * - ``sha256`` + - 預設值。 + * - ``sha512``、``blake2b`` + - 強度至少相同的替代選擇。 + * - ``md5``、``sha1`` + - 除非傳入 ``allow_weak=True``，否則一律拒絕。兩者都有實際可行的碰撞：檔案可以 + 被換成另一個摘要相同的檔案，而這個變更不會被察覺。保留它們只是為了繼續讀取 + 當初以它們寫出的基準，絕不會作為預設值。 + +``algorithm=`` 是 ``snapshot()`` 與 ``create_baseline()`` 使用的雜湊演算法。驗證一律 +使用所讀取基準的演算法，``accept()`` 也會沿用它。要把基準換成另一種演算法，請先檢視 +目錄樹，再用以新演算法建立的監控器呼叫 ``create_baseline()``。 + +檔案透過儲存層的 ``checksum``，在執行緒池上平行雜湊（預設 ``max_workers=8``）。 + +動作 +---- + +.. list-table:: + :header-rows: 1 + :widths: 30 36 34 + + * - 動作 + - 參數 + - 回傳值 + * - ``FA_integrity_snapshot`` + - ``target, algorithm="sha256"`` + - 快照：``root``、``backend``、``algorithm``、``created_at``、``entries`` + * - ``FA_integrity_baseline`` + - ``target, baseline, algorithm="sha256"`` + - ``target``、``baseline``、``backend``、``algorithm``、``created_at`` 以及 + ``entries`` 的數量 + * - ``FA_integrity_verify`` + - ``target, baseline, deep=True`` + - 偏移報告 + * - ``FA_integrity_accept`` + - ``target, baseline`` + - 與 ``FA_integrity_baseline`` 相同 + * - ``FA_integrity_watch_start`` + - ``name, target, baseline, interval=60.0`` + - 新監控器的狀態 + * - ``FA_integrity_watch_stop`` + - ``name`` + - 它最後的狀態 + * - ``FA_integrity_status`` + - ``name=None`` + - 狀態的清單：單一監控器，或全部 + +``FA_integrity_watch_start`` 讓一個具名的監控器維持在持續模式，直到 +``FA_integrity_watch_stop``；基準必須先存在。狀態包含 ``name``、``target``、 +``baseline``、``algorithm``、``interval``、``running``、``last_run``、 +``last_error`` 與 ``last_report``。 + +.. code-block:: json + + [ + ["FA_integrity_baseline", {"target": "s3://reports/2026", + "baseline": "local:///var/lib/fa/reports-2026.json"}], + ["FA_integrity_verify", {"target": "s3://reports/2026", + "baseline": "local:///var/lib/fa/reports-2026.json", + "deep": false}], + ["FA_integrity_watch_start", {"name": "reports", "target": "s3://reports/2026", + "baseline": "local:///var/lib/fa/reports-2026.json", + "interval": 900}], + ["FA_integrity_status", {"name": "reports"}] + ] + +這些動作會把偏移發布到整個行程共用的 ``event_bus``。它們不接受弱演算法，也不接受 +補救政策：這兩者都只能在 Python 中選用。與儲存動作一樣，它們能存取行程所能存取的 +一切，而且 ``FA_integrity_baseline`` 與 ``FA_integrity_accept`` 會寫入檔案，因此在 +TCP 或 HTTP 動作伺服器上請傳入 ``ActionACL``，在 MCP 伺服器上請使用 +``--allowed-actions``，只開放用戶端需要的動作。 +``register_integrity_ops(registry)`` 可把它們加入你自己的註冊表。 + +第一代監控器 +------------ + +為第一代 ``IntegrityMonitor`` 寫的程式照常運作： + +.. code-block:: python + + from automation_file import IntegrityMonitor, notification_manager, write_manifest + + write_manifest("/srv/site", "/srv/MANIFEST.json") + monitor = IntegrityMonitor( + "/srv/site", # 仍然接受 root= 與 manifest_path= 這兩個關鍵字 + "/srv/MANIFEST.json", + interval=60.0, + manager=notification_manager, + on_drift=lambda summary: print("drift:", summary), + ) + summary = monitor.check_once() # {"matched": [...], "missing": [...], "modified": [...], + # "extra": [...], "ok": False} + monitor.start() + +``check_once()`` 回傳同樣的摘要字典，驗證無法執行時會帶有 ``error``；``on_drift`` +會收到它，``last_summary`` 會保留它。與以往一樣，除非 ``alert_on_extra=True``，否則 +新增的檔案對 ``on_drift`` 與通知而言不算偏移，而重新命名會以 ``missing`` 加上 +``extra`` 的形式出現。 + +通知的去向與以往相同：透過你傳入的 ``manager`` 送出，沒有傳入時則使用整個行程共用的 +``notification_manager``。新增的只有一點：每一次偏移（包含新增）也都會以 +``IntegrityViolation`` 事件的形式發布。如果你改由這個事件（訂閱者或通知路由）把偏移 +送到通知管道，請傳入 ``notify=False``，同一次偏移才不會被通知兩次。 + +發生問題時 +---------- + +.. list-table:: + :header-rows: 1 + :widths: 40 60 + + * - 你看到的現象 + - 它的意思與處理方式 + * - ``IntegrityException: no baseline at …`` + - 基準 URI 上沒有任何東西。請檢查 URI，然後呼叫 ``create_baseline()``。執行中 + 的監控器的基準消失，本身就是一項發現：它會以帶有 ``status: "error"`` 的事件 + 送達。 + * - ``… is not readable JSON`` / ``… is not a valid manifest`` + - 基準已損毀或被編輯過。請從副本還原，或檢視目錄樹之後重新建立。不要核可一棵 + 你無從比對的目錄樹。 + * - ``… has manifest schema version 3`` + - 基準是由較新的版本寫出的。請升級，或用目前的版本重新建立基準。 + * - ``md5 is refused for integrity checks …`` + - 基準使用了弱演算法。傳入 ``allow_weak=True`` 以便讀取，再以 ``sha256`` 呼叫 + ``create_baseline()``。 + * - ``target … does not exist`` + - 檔案系統上的目錄不見了。在物件儲存上，什麼都沒有的前綴則是一棵空的目錄樹： + 每個檔案都是 ``deleted``。 + * - ``StorageUnavailableException`` + - 後端尚未初始化：請先呼叫 ``s3_instance.later_init(...)`` 或對應的函式。 + * - 驗證過程中出現 ``StoragePermissionException`` 或 + ``StorageTransientException`` + - 整次驗證失敗；無法完整讀取的目錄樹絕不會被回報為完好。持續模式會在 + ``interval`` 之後再試一次。 + * - 快速驗證沒問題，深度驗證卻回報 ``modified`` + - 內容變了，大小與修改時間卻沒變。一般工具不會這樣做；請視為竄改。 + * - 所有檔案都是 ``metadata_changed`` + - 檔案被複製或還原過，因而有了新的時間。檢視之後呼叫 ``accept()``。 + * - 補救步驟的 ``ok`` 為 ``False`` + - ``step.error`` 說明原因（鏡像中沒有副本、鏡像與基準不符、存取被拒）。檔案 + 維持原狀；並已發布嚴重程度為 ``error`` 的 ``integrity.remediated`` 事件。 + * - 每個間隔都收到同一個事件 + - 只要偏移還在，持續模式每次驗證都會回報。請修正目錄樹或呼叫 ``accept()``， + 或在訂閱端去除重複（``NotificationManager`` 會這麼做）。 + * - 監看模式漏掉了某個變更 + - 檔案系統事件可能遺失。請在監看之外，另外排程執行深度的 ``verify()``。 + +一個監控器一次只執行一次驗證：在持續模式的執行緒正在驗證時呼叫 ``verify()``，會等它 +完成。 diff --git a/docs/source/Zh-TW/zh_tw_index.rst b/docs/source/Zh-TW/zh_tw_index.rst index 415bd30..4e82cac 100644 --- a/docs/source/Zh-TW/zh_tw_index.rst +++ b/docs/source/Zh-TW/zh_tw_index.rst @@ -266,3 +266,17 @@ Slack、Email（SMTP）、Discord、Telegram、Microsoft Teams、PagerDuty :caption: 事件 usage/event_bus + +.. _zh-tw-integrity: + +第 18 章 — 檔案完整性監控 +========================= + +``IntegrityMonitor`` 為任何儲存後端中的目錄樹保存一份經過核可的基準，並回報 +與基準不符之處：快照、manifest 結構、四種模式、警示，以及選用的修復。 + +.. toctree:: + :maxdepth: 2 + :caption: 檔案完整性監控 + + usage/integrity diff --git a/docs/updates/2026-10.md b/docs/updates/2026-10.md index 1600b98..82df21c 100644 --- a/docs/updates/2026-10.md +++ b/docs/updates/2026-10.md @@ -404,3 +404,24 @@ Index and query commands: [README.md](README.md). New entries go at the end. - **Result / numbers**: part of the run recorded in U-20261008-12. - **Files**: `automation_file/remote/sftp/client.py`, `automation_file/remote/onedrive/client.py`, `automation_file/remote/smb/client.py`, `tests/test_optional_dependencies.py`. - **Open items**: none. + +## U-20261008-16 · 2026-10-08 · IntegrityMonitor 2.0 · #integrity #roadmap #done + +- **What**: the package `automation_file/integrity/` (roadmap §6, M4), which closes `progress.md` #21. `IntegrityMonitor(target, baseline)` compares a tree at any storage URI with an approved baseline stored at any storage URI. + - Four modes over one comparison: `snapshot()`, `verify(deep=True)` (`deep=False` hashes only the files whose size, modification time or etag changed), `watch()` (watchdog events for a local target, a quick pass on a timer for any other backend) and continuous `start()` / `stop()`. `create_baseline()` and `accept(report)` approve a state. + - Six kinds of change (`created`, `modified`, `deleted`, `renamed`, `metadata_changed`, `permission_changed`) in a `DriftReport`. + - Manifest schema version 2 (`schema_version`, `root`, `backend`, `algorithm`, one entry per file with size, times, checksum, etag, version, mode). Any other version is refused. The `write_manifest` format is read and converted. SHA-256 by default; `md5` and `sha1` only with `allow_weak=True`. + - One `IntegrityViolation` event per pass that finds drift, with a severity per kind (`AlertPolicy`), inside a correlation scope. A pass that cannot run in `check_once`, continuous or watch mode publishes the event with `status: "error"`; `verify()` raises. + - Remediation is opt-in (`RemediationPolicy`): quarantine, or restore from a mirror that is verified by checksum before and after the copy. Without a policy the monitor only reads. + - Seven actions: `FA_integrity_snapshot`, `_baseline`, `_verify`, `_accept`, `_watch_start`, `_watch_stop`, `_status`, registered by `build_default_registry`. + - `core/fim.py` re-exports the class; `tests/test_fim.py` passes unchanged. +- **Changed while integrating**: + - The constructor had fifteen parameters. It is now `IntegrityMonitor(target=None, baseline=None, **keywords)` with the keywords typed by `MonitorKeywords` (`Unpack`), so a call site reads as before and a misspelt keyword is a `TypeError`. + - The branch sent the first monitor's notification only through a `manager` the caller passed. The first monitor used the process-wide `notification_manager` when none was passed, and code that relies on it would have stopped being notified without an error. That default is back; `notify=False` turns the notification off for callers who route the event to their sinks instead. + - `verify_manifest` met a schema-2 baseline with "manifest missing 'files' mapping". It now says what the file is and names `IntegrityMonitor.verify()` / `FA_integrity_verify`. +- **Tests**: `tests/test_integrity_{snapshot,detector,monitor,object_store,remediation,watch,legacy,actions}.py` (196 cases), among them the nine cases of `tests/test_fim.py` repeated against the new class, the default and the disabled notification, the unknown keyword and the legacy reader's message. +- **Result / numbers**: 4211 passed, 149 skipped, 0 failed with every extra; 2494 passed, 89 skipped with the base dependencies only. `ruff check`, `ruff format --check` and `mypy automation_file` (210 files) pass. Python 3.14.7 on Windows. +- **Not verified**: a real S3 or Azure target (an in-memory object-store stand-in was used); symbolic links; the Sphinx build of the new pages (headings, markup and the imports of the examples were checked by script); any platform other than Windows. +- **Docs**: chapter 18 in the three manuals (`usage/integrity.rst`), `docs/source/API/integrity.rst`, the indexes, the feature list and the integrity section of the three READMEs, `architecture.md` §2 to §4, `CLAUDE.md` (package map, key types). +- **Files**: `automation_file/integrity/` (16 modules), `automation_file/core/{fim,manifest,action_registry}.py`, `automation_file/__init__.py`, the tests above, the documentation above. +- **Open items**: an `integrity` subcommand for the CLI comes with the pipeline and audit subcommands (#33). diff --git a/docs/updates/README.md b/docs/updates/README.md index f450c92..037bd90 100644 --- a/docs/updates/README.md +++ b/docs/updates/README.md @@ -58,6 +58,7 @@ In the same commit: delete the item from `progress.md`, add a `#done` entry here | ID | Date | Title | Tags | Batch | |---|---|---|---|---| +| U-20261008-16 | 2026-10-08 | IntegrityMonitor 2.0 | #integrity #roadmap #done | [2026-10](2026-10.md) | | U-20261008-15 | 2026-10-08 | The SFTP, OneDrive and SMB clients name the extra to install | #packaging #done | [2026-10](2026-10.md) | | U-20261008-14 | 2026-10-08 | The WebDAV client only talks to its own server | #security #incident | [2026-10](2026-10.md) | | U-20261008-13 | 2026-10-08 | A move between two views of one store could delete the file | #storage #incident | [2026-10](2026-10.md) | @@ -110,5 +111,5 @@ In the same commit: delete the item from `progress.md`, add a `#done` entry here | File | Period | Entries | |---|---|---:| -| [2026-10.md](2026-10.md) | 2026-10 | 26 | +| [2026-10.md](2026-10.md) | 2026-10 | 27 | | [2026-09.md](2026-09.md) | 2026-09 | 21 | diff --git a/progress.md b/progress.md index 1a883b9..ac6d891 100644 --- a/progress.md +++ b/progress.md @@ -27,7 +27,7 @@ Items #10 to #26 are what is left of the 1.0 roadmap (`docs/FILEAUTOMATION-1.0-R ### Later milestones -- **#21** IntegrityMonitor 2.0 (roadmap §6, M4): snapshot and manifest schema with a version, baseline management, change detection over `FileInfo`, watch and continuous modes, alert and audit hooks, opt-in remediation. `core/fim.py` and `core/manifest.py` are the starting point; build it on the storage layer. +- **#33** CLI subcommands for the packages that have none: `integrity` (snapshot, baseline, verify, accept, status), `pipeline` and `audit`, as thin calls into their `FA_*` functions like `storage` (U-20261008-11). - **#22** Pipeline runtime (roadmap §7, M5): `Pipeline` domain model, DAG runtime v2 with retry, timeout, cancellation, conditions, idempotency, checkpoint and resume, dry run, execution history, and versioned YAML/JSON definitions with schema validation. `core/dag_executor.py` is the starting point. - **#23** Scheduler, events, notifications and audit (roadmap §8 to §10, M6): one scheduler with cron (time-zone aware), manual, file-event, webhook and pipeline-dependency triggers; an event model that drives a `NotificationRouter`; audit schema v2 with correlation IDs behind a storage interface. - **#24** UI 2.0 (roadmap §11, M7). Not before the APIs of #13 to #23 are stable (roadmap §20). diff --git a/tests/test_integrity_actions.py b/tests/test_integrity_actions.py new file mode 100644 index 0000000..63fb526 --- /dev/null +++ b/tests/test_integrity_actions.py @@ -0,0 +1,241 @@ +"""FA_integrity_* actions: the monitor through the registry, the executor and MCP.""" + +from __future__ import annotations + +import hashlib +import json +import threading +from collections.abc import Iterator +from pathlib import Path + +import pytest + +from automation_file import ActionExecutor, ActionRegistry, tools_from_registry +from automation_file.events import Event, IntegrityViolation, event_bus +from automation_file.integrity import IntegrityException, actions, register_integrity_ops +from automation_file.storage import File, Storage, clear_memory_stores + +NAMES = [ + "FA_integrity_accept", + "FA_integrity_baseline", + "FA_integrity_snapshot", + "FA_integrity_status", + "FA_integrity_verify", + "FA_integrity_watch_start", + "FA_integrity_watch_stop", +] +TREE = "memory://actions/tree" +BASELINE = "memory://actions-state/tree.json" +WAIT = 10.0 + + +@pytest.fixture(autouse=True) +def _fresh_state() -> Iterator[None]: + clear_memory_stores() + yield + actions.stop_all_monitors() + clear_memory_stores() + + +@pytest.fixture +def tree() -> Storage: + storage = Storage(TREE) + storage.file("a.txt").write(b"alpha") + storage.file("sub/b.txt").write(b"bravo") + return storage + + +@pytest.fixture +def executor() -> ActionExecutor: + registry = ActionRegistry() + register_integrity_ops(registry) + return ActionExecutor(registry) + + +@pytest.fixture +def drift() -> Iterator[threading.Event]: + """Set once an IntegrityViolation for the tree reaches the process-wide bus.""" + seen = threading.Event() + + def _note(event: Event) -> None: + if event.payload.get("resource") == TREE: + seen.set() + + subscription = event_bus.subscribe(_note, types=IntegrityViolation) + yield seen + event_bus.unsubscribe(subscription) + + +def test_register_integrity_ops_fills_a_registry() -> None: + registry = ActionRegistry() + register_integrity_ops(registry) + assert sorted(registry.event_dict) == NAMES + assert sorted(actions.integrity_commands()) == NAMES + + +def test_snapshot_returns_the_tree_and_stores_nothing(tree: Storage) -> None: + snapshot = actions.integrity_snapshot(TREE) + assert json.loads(json.dumps(snapshot)) == snapshot + assert (snapshot["root"], snapshot["backend"], snapshot["algorithm"]) == ( + TREE, + "memory", + "sha256", + ) + assert [entry["path"] for entry in snapshot["entries"]] == ["a.txt", "sub/b.txt"] + assert snapshot["entries"][0]["checksum"] == hashlib.sha256(b"alpha").hexdigest() + assert actions.integrity_snapshot(TREE, "sha512")["entries"][0]["checksum"] == ( + hashlib.sha512(b"alpha").hexdigest() + ) + assert Storage("memory://actions-state").exists("tree.json") is False + + +def test_a_weak_algorithm_cannot_be_chosen_through_an_action(tree: Storage) -> None: + with pytest.raises(IntegrityException, match="not collision-resistant"): + actions.integrity_snapshot(TREE, "md5") + with pytest.raises(IntegrityException, match="not collision-resistant"): + actions.integrity_baseline(TREE, BASELINE, "md5") + assert File(BASELINE).exists() is False + + +def test_baseline_verify_and_accept(tree: Storage) -> None: + stored = actions.integrity_baseline(TREE, BASELINE) + assert stored == { + "target": TREE, + "baseline": BASELINE, + "backend": "memory", + "algorithm": "sha256", + "created_at": stored["created_at"], + "entries": 2, + } + assert json.loads(File(BASELINE).read_text())["schema_version"] == 2 + clean = actions.integrity_verify(TREE, BASELINE) + assert (clean["ok"], clean["deep"], clean["checked"], clean["changes"]) == (True, True, 2, []) + + tree.file("a.txt").write(b"tampered") + tree.file("new.txt").write(b"novel") + report = actions.integrity_verify(TREE, BASELINE) + assert json.loads(json.dumps(report)) == report + assert report["ok"] is False + assert (report["counts"]["modified"], report["counts"]["created"]) == (1, 1) + assert [(change["kind"], change["path"]) for change in report["changes"]] == [ + ("modified", "a.txt"), + ("created", "new.txt"), + ] + quick = actions.integrity_verify(TREE, BASELINE, deep=False) + assert (quick["deep"], quick["hashed"], quick["counts"]["modified"]) == (False, 2, 1) + + accepted = actions.integrity_accept(TREE, BASELINE) + assert (accepted["entries"], accepted["baseline"], accepted["algorithm"]) == ( + 3, + BASELINE, + "sha256", + ) + assert actions.integrity_verify(TREE, BASELINE)["ok"] is True + + +def test_the_baseline_may_be_a_local_path(tree: Storage, tmp_path: Path) -> None: + location = str(tmp_path / "state" / "tree.json") + stored = actions.integrity_baseline(TREE, location, "blake2b") + assert stored["baseline"].startswith("local:///") + assert stored["algorithm"] == "blake2b" + assert (tmp_path / "state" / "tree.json").is_file() + assert actions.integrity_verify(TREE, location)["algorithm"] == "blake2b" + + +def test_an_action_list_runs_through_the_executor(tree: Storage, executor: ActionExecutor) -> None: + results = executor.execute_action( + [ + ["FA_integrity_snapshot", {"target": TREE}], + ["FA_integrity_baseline", {"target": TREE, "baseline": BASELINE}], + ["FA_integrity_verify", [TREE, BASELINE]], + ["FA_integrity_verify", {"target": TREE, "baseline": "memory://nowhere/b.json"}], + ["FA_integrity_accept", {"target": TREE, "baseline": BASELINE}], + ["FA_integrity_status"], + ["FA_integrity_status", {"name": "nobody"}], + ] + ) + values = list(results.values()) + assert json.loads(json.dumps(values)) == values + assert len(values[0]["entries"]) == 2 + assert values[1]["entries"] == 2 + assert values[2]["ok"] is True + assert "IntegrityException" in values[3] and "no baseline at" in values[3] + assert values[4]["entries"] == 2 + assert values[5] == [] + assert "no integrity monitor named 'nobody'" in values[6] + + +def test_a_named_monitor_runs_until_it_is_stopped( + tree: Storage, executor: ActionExecutor, drift: threading.Event +) -> None: + actions.integrity_baseline(TREE, BASELINE) + tree.file("a.txt").write(b"tampered") + started = executor.execute_action( + [ + [ + "FA_integrity_watch_start", + {"name": "tree", "target": TREE, "baseline": BASELINE, "interval": 0.02}, + ] + ] + ) + (status,) = started.values() + assert (status["name"], status["running"], status["interval"]) == ("tree", True, 0.02) + assert (status["target"], status["baseline"]) == (TREE, BASELINE) + assert drift.wait(WAIT), "the named monitor did not verify" + + (listed,) = actions.integrity_status() + (named,) = actions.integrity_status("tree") + assert listed["name"] == named["name"] == "tree" + assert named["running"] is True + assert named["last_run"] is not None + assert named["last_error"] is None + assert named["last_report"]["counts"]["modified"] == 1 + assert json.loads(json.dumps(named)) == named + + with pytest.raises(IntegrityException, match="already running"): + actions.integrity_watch_start("tree", TREE, BASELINE) + stopped = actions.integrity_watch_stop("tree") + assert (stopped["name"], stopped["running"]) == ("tree", False) + assert stopped["last_report"]["ok"] is False + assert actions.integrity_status() == [] + with pytest.raises(IntegrityException, match="no integrity monitor named 'tree'"): + actions.integrity_watch_stop("tree") + + +def test_a_named_monitor_needs_a_name_a_baseline_and_a_positive_interval(tree: Storage) -> None: + with pytest.raises(IntegrityException, match="create it first with FA_integrity_baseline"): + actions.integrity_watch_start("tree", TREE, BASELINE) + actions.integrity_baseline(TREE, BASELINE) + with pytest.raises(IntegrityException, match="needs a name"): + actions.integrity_watch_start(" ", TREE, BASELINE) + with pytest.raises(IntegrityException, match="interval must be positive"): + actions.integrity_watch_start("tree", TREE, BASELINE, interval=0) + assert actions.integrity_status() == [] + + +def test_stop_all_monitors_stops_every_named_monitor(tree: Storage) -> None: + actions.integrity_baseline(TREE, BASELINE) + actions.integrity_watch_start("one", TREE, BASELINE, interval=30.0) + actions.integrity_watch_start("two", TREE, BASELINE, interval=30.0) + assert [status["name"] for status in actions.integrity_status()] == ["one", "two"] + stopped = actions.stop_all_monitors() + assert [(status["name"], status["running"]) for status in stopped] == [ + ("one", False), + ("two", False), + ] + assert actions.integrity_status() == [] + + +def test_the_actions_are_mcp_tools_with_their_parameters() -> None: + registry = ActionRegistry() + register_integrity_ops(registry) + tools = {tool["name"]: tool for tool in tools_from_registry(registry)} + assert set(NAMES) <= set(tools) + verify = tools["FA_integrity_verify"]["inputSchema"] + assert list(verify["properties"]) == ["target", "baseline", "deep"] + assert verify["required"] == ["target", "baseline"] + start = tools["FA_integrity_watch_start"]["inputSchema"] + assert list(start["properties"]) == ["name", "target", "baseline", "interval"] + assert start["required"] == ["name", "target", "baseline"] + assert tools["FA_integrity_status"]["inputSchema"].get("required", []) == [] + assert "Compare" in tools["FA_integrity_verify"]["description"] diff --git a/tests/test_integrity_detector.py b/tests/test_integrity_detector.py new file mode 100644 index 0000000..c37d2a4 --- /dev/null +++ b/tests/test_integrity_detector.py @@ -0,0 +1,317 @@ +"""The change detector and the drift report.""" + +from __future__ import annotations + +import json +from datetime import datetime, timedelta, timezone +from typing import Any + +import pytest + +from automation_file.integrity import ( + Change, + ChangeKind, + DriftReport, + IntegrityException, + RemediationStep, + Snapshot, + SnapshotEntry, + detect_changes, +) + +ROOT = "memory://detect/tree" +MOMENT = datetime(2026, 10, 8, 10, 15, 30, tzinfo=timezone.utc) +LATER = MOMENT + timedelta(hours=1) + + +def _entry(path: str, checksum: str = "aa", size: int | None = 5, **fields: Any) -> SnapshotEntry: + return SnapshotEntry(path=path, checksum=checksum, size=size, **fields) + + +def _snapshot(*entries: SnapshotEntry, algorithm: str = "sha256") -> Snapshot: + return Snapshot(root=ROOT, algorithm=algorithm, entries=entries) + + +def _kinds(changes: list[Change]) -> list[tuple[str, str]]: + return [(change.kind.value, change.path) for change in changes] + + +def test_identical_snapshots_have_no_changes() -> None: + entries = (_entry("a.txt", modified_at=MOMENT, mode=0o644), _entry("b.txt", "bb")) + assert detect_changes(_snapshot(*entries), _snapshot(*entries)) == [] + assert detect_changes(_snapshot(), _snapshot()) == [] + + +def test_created() -> None: + new = _entry("new.txt", "nn") + (change,) = detect_changes(_snapshot(_entry("a.txt")), _snapshot(_entry("a.txt"), new)) + assert (change.kind, change.path) == (ChangeKind.CREATED, "new.txt") + assert (change.before, change.after, change.previous_path, change.note) == ( + None, + new, + None, + "", + ) + + +def test_deleted() -> None: + gone = _entry("gone.txt", "gg") + (change,) = detect_changes(_snapshot(_entry("a.txt"), gone), _snapshot(_entry("a.txt"))) + assert (change.kind, change.path) == (ChangeKind.DELETED, "gone.txt") + assert (change.before, change.after) == (gone, None) + + +def test_modified_when_the_checksum_differs() -> None: + before = _entry("a.txt", "aa", modified_at=MOMENT) + after = _entry("a.txt", "zz", size=9, modified_at=LATER) + (change,) = detect_changes(_snapshot(before), _snapshot(after)) + assert (change.kind, change.path) == (ChangeKind.MODIFIED, "a.txt") + assert (change.before, change.after) == (before, after) + assert change.fields == () + + +def test_a_size_that_differs_is_modified_even_with_the_same_checksum() -> None: + changes = detect_changes(_snapshot(_entry("a.txt", size=5)), _snapshot(_entry("a.txt", size=6))) + assert _kinds(changes) == [("modified", "a.txt")] + unknown = detect_changes( + _snapshot(_entry("a.txt", size=None)), _snapshot(_entry("a.txt", size=6)) + ) + assert unknown == [] + + +def test_renamed_when_one_deleted_and_one_created_file_share_the_content() -> None: + old, new = _entry("old/name.txt", "rr", size=7), _entry("new/name.txt", "rr", size=7) + (change,) = detect_changes( + _snapshot(old, _entry("keep.txt")), _snapshot(new, _entry("keep.txt")) + ) + assert change.kind is ChangeKind.RENAMED + assert (change.path, change.previous_path) == ("new/name.txt", "old/name.txt") + assert (change.before, change.after, change.note) == (old, new, "") + + +def test_the_same_checksum_with_another_size_is_not_a_rename() -> None: + changes = detect_changes( + _snapshot(_entry("old.txt", "rr", size=7)), _snapshot(_entry("new.txt", "rr", size=8)) + ) + assert _kinds(changes) == [("created", "new.txt"), ("deleted", "old.txt")] + assert [change.note for change in changes] == ["", ""] + + +def test_an_ambiguous_rename_falls_back_to_created_and_deleted_and_says_so() -> None: + baseline = _snapshot( + _entry("one.txt", "dddddddddddddddddd"), _entry("two.txt", "dddddddddddddddddd") + ) + current = _snapshot(_entry("three.txt", "dddddddddddddddddd")) + changes = detect_changes(baseline, current) + assert _kinds(changes) == [ + ("deleted", "one.txt"), + ("created", "three.txt"), + ("deleted", "two.txt"), + ] + notes = {change.note for change in changes} + assert notes == { + "ambiguous rename: 2 deleted and 1 created files share the checksum dddddddddddd...; " + "reported separately" + } + + +def test_one_deleted_file_and_two_copies_of_it_is_ambiguous_too() -> None: + changes = detect_changes( + _snapshot(_entry("orig.txt", "cc")), + _snapshot(_entry("copy1.txt", "cc"), _entry("copy2.txt", "cc")), + ) + assert _kinds(changes) == [ + ("created", "copy1.txt"), + ("created", "copy2.txt"), + ("deleted", "orig.txt"), + ] + assert all( + change.note.startswith("ambiguous rename: 1 deleted and 2 created") for change in changes + ) + + +def test_entries_without_a_checksum_are_never_paired() -> None: + changes = detect_changes( + _snapshot(_entry("old.txt", "", size=None)), _snapshot(_entry("new.txt", "", size=None)) + ) + assert _kinds(changes) == [("created", "new.txt"), ("deleted", "old.txt")] + assert [change.note for change in changes] == ["", ""] + + +def test_metadata_changed_when_only_other_metadata_differs() -> None: + before = _entry("a.txt", modified_at=MOMENT, content_type="text/plain", version="v1", etag="e1") + after = _entry("a.txt", modified_at=LATER, content_type="text/plain", version="v2", etag="e1") + (change,) = detect_changes(_snapshot(before), _snapshot(after)) + assert change.kind is ChangeKind.METADATA_CHANGED + assert change.fields == ("modified_at", "version") + assert (change.before, change.after) == (before, after) + + +def test_metadata_one_side_did_not_record_is_not_compared() -> None: + before = _entry("a.txt") + after = _entry("a.txt", modified_at=LATER, content_type="text/plain", version="v2", etag="e9") + assert detect_changes(_snapshot(before), _snapshot(after)) == [] + assert detect_changes(_snapshot(after), _snapshot(before)) == [] + + +def test_permission_changed_when_the_mode_differs() -> None: + before, after = _entry("run.sh", mode=0o644), _entry("run.sh", mode=0o755) + (change,) = detect_changes(_snapshot(before), _snapshot(after)) + assert change.kind is ChangeKind.PERMISSION_CHANGED + assert change.fields == ("mode",) + assert detect_changes(_snapshot(_entry("run.sh")), _snapshot(after)) == [] + + +def test_a_file_can_be_modified_and_have_its_permissions_changed() -> None: + changes = detect_changes( + _snapshot(_entry("run.sh", "aa", mode=0o644, modified_at=MOMENT)), + _snapshot(_entry("run.sh", "bb", mode=0o4755, modified_at=LATER)), + ) + assert _kinds(changes) == [("modified", "run.sh"), ("permission_changed", "run.sh")] + + +def test_changes_are_sorted_by_path() -> None: + baseline = _snapshot(_entry("b.txt", "bb"), _entry("d.txt", "dd"), _entry("m.txt", "mm")) + current = _snapshot(_entry("a.txt", "new"), _entry("b.txt", "changed"), _entry("m.txt", "mm")) + assert _kinds(detect_changes(baseline, current)) == [ + ("created", "a.txt"), + ("modified", "b.txt"), + ("deleted", "d.txt"), + ] + + +def test_snapshots_of_two_algorithms_cannot_be_compared() -> None: + with pytest.raises(IntegrityException, match="sha256 snapshot with a sha512 one"): + detect_changes(_snapshot(), _snapshot(algorithm="sha512")) + + +def test_the_kinds() -> None: + assert [kind.value for kind in ChangeKind] == [ + "created", + "modified", + "deleted", + "renamed", + "metadata_changed", + "permission_changed", + ] + + +# ---------------------------------------------------------------------- report + + +def _report() -> DriftReport: + baseline = _snapshot( + _entry("a.txt", "aa"), + _entry("gone.txt", "gg"), + _entry("old.txt", "rr"), + _entry("meta.txt", "mm", modified_at=MOMENT), + ) + current = _snapshot( + _entry("a.txt", "zz"), + _entry("new.txt", "nn"), + _entry("renamed.txt", "rr"), + _entry("meta.txt", "mm", modified_at=LATER), + ) + return DriftReport( + target=ROOT, + baseline="memory://detect/baseline.json", + backend="memory", + changes=tuple(detect_changes(baseline, current)), + checked=4, + hashed=4, + verified_at=MOMENT, + correlation_id="run-1", + snapshot=current, + ) + + +def test_a_report_counts_every_kind() -> None: + report = _report() + assert report.ok is False + assert report.counts == { + "created": 1, + "modified": 1, + "deleted": 1, + "renamed": 1, + "metadata_changed": 1, + "permission_changed": 0, + } + assert report.paths("modified") == ["a.txt"] + assert report.paths(ChangeKind.RENAMED) == ["renamed.txt"] + assert report.paths("permission_changed") == [] + + +def test_a_clean_report_is_ok() -> None: + report = DriftReport(target=ROOT) + assert report.ok is True + assert set(report.counts.values()) == {0} + assert (report.deep, report.partial, report.notes, report.remediation) == (True, False, (), ()) + assert report.verified_at.utcoffset() == timedelta(0) + + +def test_a_report_is_json_friendly() -> None: + step = RemediationStep( + action="restore", path="gone.txt", kind="deleted", ok=False, error="no copy" + ) + report = DriftReport( + target=ROOT, + changes=_report().changes, + deep=False, + notes=("quick pass",), + remediation=(step,), + verified_at=MOMENT, + ) + document = json.loads(json.dumps(report.to_dict())) + assert list(document) == [ + "target", + "baseline", + "backend", + "algorithm", + "ok", + "deep", + "partial", + "checked", + "hashed", + "counts", + "changes", + "notes", + "remediation", + "verified_at", + "correlation_id", + ] + assert (document["ok"], document["deep"], document["notes"]) == (False, False, ["quick pass"]) + assert document["verified_at"] == "2026-10-08T10:15:30+00:00" + assert document["remediation"] == [ + { + "action": "restore", + "path": "gone.txt", + "kind": "deleted", + "ok": False, + "source": "", + "destination": "", + "error": "no copy", + } + ] + renamed = next(change for change in document["changes"] if change["kind"] == "renamed") + assert (renamed["path"], renamed["previous_path"]) == ("renamed.txt", "old.txt") + assert renamed["before"]["path"] == "old.txt" + assert renamed["after"]["checksum"] == "rr" + created = next(change for change in document["changes"] if change["kind"] == "created") + assert (created["before"], created["previous_path"], created["fields"]) == (None, None, []) + + +def test_the_brief_form_of_a_change_leaves_out_what_is_empty() -> None: + changes = {change.kind: change for change in _report().changes} + assert changes[ChangeKind.MODIFIED].brief() == {"kind": "modified", "path": "a.txt"} + assert changes[ChangeKind.RENAMED].brief() == { + "kind": "renamed", + "path": "renamed.txt", + "previous_path": "old.txt", + } + assert changes[ChangeKind.METADATA_CHANGED].brief() == { + "kind": "metadata_changed", + "path": "meta.txt", + "fields": ["modified_at"], + } + noted = Change(ChangeKind.CREATED, "x.txt", note="ambiguous rename") + assert noted.brief() == {"kind": "created", "path": "x.txt", "note": "ambiguous rename"} diff --git a/tests/test_integrity_legacy.py b/tests/test_integrity_legacy.py new file mode 100644 index 0000000..5d3cc2c --- /dev/null +++ b/tests/test_integrity_legacy.py @@ -0,0 +1,311 @@ +"""The first monitor's behaviour, kept by ``automation_file.integrity.IntegrityMonitor``. + +The first half repeats the cases of ``tests/test_fim.py`` against the new class: +the legacy call ``IntegrityMonitor(root, manifest_path, ...)`` on a manifest +written by ``write_manifest``. The rest covers what the summary does with the +change kinds the first monitor did not know. +""" + +from __future__ import annotations + +from pathlib import Path +from typing import Any + +import pytest + +import automation_file +from automation_file.core import fim +from automation_file.core.manifest import ManifestException, verify_manifest, write_manifest +from automation_file.events import EventBus, IntegrityViolation +from automation_file.exceptions import FileAutomationException +from automation_file.integrity import IntegrityException, IntegrityMonitor +from automation_file.integrity.legacy import format_body +from automation_file.notify import NotificationManager, notification_manager +from automation_file.notify.sinks import NotificationSink + + +class _Recorder(NotificationSink): + name = "recorder" + + def __init__(self) -> None: + self.messages: list[tuple[str, str, str]] = [] + + def send(self, subject: str, body: str, level: str = "info") -> None: + self.messages.append((subject, body, level)) + + +def _build_tree(tmp_path: Path) -> tuple[Path, Path]: + root = tmp_path / "tree" + root.mkdir() + (root / "a.txt").write_text("alpha", encoding="utf-8") + (root / "b.txt").write_text("bravo", encoding="utf-8") + manifest_path = tmp_path / "manifest.json" + write_manifest(root, manifest_path) + return root, manifest_path + + +def _recording_manager() -> tuple[NotificationManager, _Recorder]: + manager = NotificationManager(dedup_seconds=0.0) + recorder = _Recorder() + manager.register(recorder) + return manager, recorder + + +# ---------------------------------------------------------------------- the cases of test_fim.py + + +def test_check_once_clean_tree(tmp_path: Path) -> None: + root, manifest_path = _build_tree(tmp_path) + monitor = IntegrityMonitor(root, manifest_path) + summary = monitor.check_once() + assert summary["ok"] is True + assert monitor.last_summary is summary + + +def test_check_once_detects_modified_file_and_notifies(tmp_path: Path) -> None: + root, manifest_path = _build_tree(tmp_path) + (root / "a.txt").write_text("tampered", encoding="utf-8") + manager, recorder = _recording_manager() + monitor = IntegrityMonitor(root, manifest_path, manager=manager) + summary = monitor.check_once() + assert summary["ok"] is False + assert "a.txt" in summary["modified"] + assert recorder.messages, "expected a drift notification" + subject, body, level = recorder.messages[0] + assert level == "error" + assert "integrity drift" in subject + assert "a.txt" in body + + +def test_check_once_detects_missing_file(tmp_path: Path) -> None: + root, manifest_path = _build_tree(tmp_path) + (root / "b.txt").unlink() + manager, recorder = _recording_manager() + monitor = IntegrityMonitor(root, manifest_path, manager=manager) + summary = monitor.check_once() + assert "b.txt" in summary["missing"] + assert recorder.messages + + +def test_extras_ignored_by_default(tmp_path: Path) -> None: + root, manifest_path = _build_tree(tmp_path) + (root / "new.txt").write_text("novel", encoding="utf-8") + manager, recorder = _recording_manager() + monitor = IntegrityMonitor(root, manifest_path, manager=manager) + summary = monitor.check_once() + assert "new.txt" in summary["extra"] + assert summary["ok"] is True + assert not recorder.messages + + +def test_alert_on_extra_flag(tmp_path: Path) -> None: + root, manifest_path = _build_tree(tmp_path) + (root / "new.txt").write_text("novel", encoding="utf-8") + manager, recorder = _recording_manager() + monitor = IntegrityMonitor(root, manifest_path, manager=manager, alert_on_extra=True) + monitor.check_once() + assert recorder.messages + + +def test_on_drift_callback_invoked(tmp_path: Path) -> None: + root, manifest_path = _build_tree(tmp_path) + (root / "a.txt").write_text("tampered", encoding="utf-8") + seen: list[dict[str, Any]] = [] + monitor = IntegrityMonitor( + root, + manifest_path, + manager=NotificationManager(dedup_seconds=0.0), + on_drift=seen.append, + ) + monitor.check_once() + assert len(seen) == 1 + assert "a.txt" in seen[0]["modified"] + + +def test_missing_manifest_treated_as_drift(tmp_path: Path) -> None: + root = tmp_path / "tree" + root.mkdir() + (root / "a.txt").write_text("alpha", encoding="utf-8") + manager, recorder = _recording_manager() + monitor = IntegrityMonitor(root, tmp_path / "missing.json", manager=manager) + summary = monitor.check_once() + assert summary["ok"] is False + assert "error" in summary + assert recorder.messages + + +def test_positive_interval_required(tmp_path: Path) -> None: + root, manifest_path = _build_tree(tmp_path) + with pytest.raises(FileAutomationException): + IntegrityMonitor(root, manifest_path, interval=0) + + +def test_start_and_stop_thread(tmp_path: Path) -> None: + root, manifest_path = _build_tree(tmp_path) + monitor = IntegrityMonitor(root, manifest_path, interval=0.05) + monitor.start() + try: + # Second start is a no-op. + monitor.start() + finally: + monitor.stop(timeout=1.0) + + +# ---------------------------------------------------------------------- beyond test_fim.py + + +def test_the_legacy_import_path_is_the_new_class() -> None: + assert fim.IntegrityMonitor is IntegrityMonitor + assert automation_file.IntegrityMonitor is IntegrityMonitor + + +def test_the_legacy_keyword_names_still_work(tmp_path: Path) -> None: + root, manifest_path = _build_tree(tmp_path) + (root / "a.txt").write_text("tampered", encoding="utf-8") + manager, recorder = _recording_manager() + seen: list[dict[str, Any]] = [] + monitor = IntegrityMonitor( + root=str(root), + manifest_path=str(manifest_path), + interval=60.0, + manager=manager, + on_drift=seen.append, + ) + assert monitor.check_once()["modified"] == ["a.txt"] + assert len(seen) == len(recorder.messages) == 1 + with pytest.raises(IntegrityException, match="target= or its older name root="): + IntegrityMonitor(root, root=root) + with pytest.raises(IntegrityException, match="baseline= or its older name manifest_path="): + IntegrityMonitor(root, manifest_path, manifest_path=manifest_path) + with pytest.raises(IntegrityException, match="needs a target"): + IntegrityMonitor(manifest_path=manifest_path) + + +def test_the_summary_keeps_the_legacy_shape(tmp_path: Path) -> None: + root, manifest_path = _build_tree(tmp_path) + (root / "a.txt").write_text("tampered", encoding="utf-8") + (root / "new.txt").write_text("novel", encoding="utf-8") + summary = IntegrityMonitor(root, manifest_path, bus=EventBus()).check_once() + assert summary == { + "matched": ["b.txt"], + "missing": [], + "modified": ["a.txt"], + "extra": ["new.txt"], + "ok": False, + } + + +def test_a_rename_is_missing_plus_extra_in_the_summary(tmp_path: Path) -> None: + root, manifest_path = _build_tree(tmp_path) + (root / "a.txt").rename(root / "renamed.txt") + monitor = IntegrityMonitor(root, manifest_path, bus=EventBus()) + summary = monitor.check_once() + assert (summary["missing"], summary["extra"], summary["ok"]) == ( + ["a.txt"], + ["renamed.txt"], + False, + ) + report = monitor.last_report + assert report is not None + assert report.counts["renamed"] == 1 + + +def test_without_a_manager_the_process_wide_one_is_notified(tmp_path: Path) -> None: + root, manifest_path = _build_tree(tmp_path) + (root / "a.txt").write_text("tampered", encoding="utf-8") + recorder = _Recorder() + notification_manager.register(recorder) + try: + bus = EventBus() + summary = IntegrityMonitor(root, manifest_path, bus=bus).check_once() + finally: + notification_manager.unregister(recorder.name) + assert summary["ok"] is False + assert [level for _, _, level in recorder.messages] == ["error"] + assert [event.type for event in bus.recent()] == ["integrity.violation"] + + +def test_notify_false_sends_no_notification(tmp_path: Path) -> None: + root, manifest_path = _build_tree(tmp_path) + (root / "a.txt").write_text("tampered", encoding="utf-8") + passed, own = _recording_manager() + shared = _Recorder() + notification_manager.register(shared) + seen: list[dict[str, Any]] = [] + try: + bus = EventBus() + IntegrityMonitor( + root, manifest_path, manager=passed, notify=False, on_drift=seen.append, bus=bus + ).check_once() + finally: + notification_manager.unregister(shared.name) + assert own.messages == [] + assert shared.messages == [] + assert len(seen) == 1 + assert [event.type for event in bus.recent()] == ["integrity.violation"] + + +def test_an_unknown_keyword_is_a_type_error(tmp_path: Path) -> None: + with pytest.raises(TypeError, match="unexpected keyword argument 'intervall'"): + IntegrityMonitor(tmp_path, intervall=5.0) # type: ignore[call-arg] + + +def test_verify_manifest_names_the_new_reader_for_a_baseline(tmp_path: Path) -> None: + root, manifest_path = _build_tree(tmp_path) + IntegrityMonitor(root, manifest_path, bus=EventBus()).accept() + with pytest.raises(ManifestException, match="FA_integrity_verify"): + verify_manifest(root, manifest_path) + + +def test_a_failed_pass_is_an_event_and_stays_in_the_status(tmp_path: Path) -> None: + root = tmp_path / "tree" + root.mkdir() + bus = EventBus() + monitor = IntegrityMonitor(root, tmp_path / "missing.json", bus=bus) + summary = monitor.check_once() + assert summary["matched"] == summary["missing"] == summary["modified"] == summary["extra"] == [] + assert "IntegrityException" in summary["error"] + (event,) = bus.recent() + assert isinstance(event, IntegrityViolation) + assert event.payload["status"] == "error" + assert event.payload["error"].startswith("IntegrityException: no baseline at ") + assert monitor.last_error == event.payload["error"] + assert monitor.last_report is None + + +def test_a_missing_target_is_an_error_not_a_tree_of_deleted_files(tmp_path: Path) -> None: + _, manifest_path = _build_tree(tmp_path) + monitor = IntegrityMonitor(tmp_path / "gone", manifest_path, bus=EventBus()) + with pytest.raises(IntegrityException, match="does not exist"): + monitor.verify() + assert "does not exist" in monitor.check_once()["error"] + + +def test_an_on_drift_failure_does_not_stop_the_notification(tmp_path: Path) -> None: + root, manifest_path = _build_tree(tmp_path) + (root / "a.txt").write_text("tampered", encoding="utf-8") + manager, recorder = _recording_manager() + + def broken(_summary: dict[str, Any]) -> None: + raise RuntimeError("callback bug") + + monitor = IntegrityMonitor( + root, manifest_path, manager=manager, on_drift=broken, bus=EventBus() + ) + assert monitor.check_once()["modified"] == ["a.txt"] + assert len(recorder.messages) == 1 + + +def test_the_notification_body_lists_the_first_paths() -> None: + summary = { + "missing": [f"m{index}.txt" for index in range(7)], + "modified": ["changed.txt"], + "extra": [], + "error": "IntegrityException('boom')", + } + assert format_body(summary).splitlines() == [ + "error: IntegrityException('boom')", + "missing: m0.txt, m1.txt, m2.txt, m3.txt, m4.txt (+2 more)", + "modified: changed.txt", + ] + assert format_body({"missing": [], "modified": [], "extra": []}) == "no drift detected" diff --git a/tests/test_integrity_monitor.py b/tests/test_integrity_monitor.py new file mode 100644 index 0000000..64b67de --- /dev/null +++ b/tests/test_integrity_monitor.py @@ -0,0 +1,572 @@ +"""IntegrityMonitor: snapshot, baseline, verify, accept, alerts and continuous mode.""" + +from __future__ import annotations + +import hashlib +import json +import os +import stat +import threading +from collections.abc import Iterator +from pathlib import Path + +import pytest + +from automation_file.core.manifest import write_manifest +from automation_file.events import ( + Event, + EventBus, + IntegrityViolation, + Severity, + correlation_scope, + event_bus, +) +from automation_file.exceptions import StoragePermissionException +from automation_file.integrity import ( + AlertPolicy, + BaselineManager, + DriftReport, + IntegrityException, + IntegrityMonitor, + Snapshot, + load_manifest, +) +from automation_file.storage import File, Storage, clear_memory_stores + +TREE = "memory://monitor/tree" +BASELINE = "memory://baselines/tree.json" +FILES = {"a.txt": b"alpha", "b.txt": b"bravo", "sub/c.txt": b"charlie"} +WAIT = 10.0 + + +@pytest.fixture(autouse=True) +def _fresh_stores() -> Iterator[None]: + clear_memory_stores() + yield + clear_memory_stores() + + +@pytest.fixture +def bus() -> EventBus: + return EventBus() + + +@pytest.fixture +def tree() -> Storage: + storage = Storage(TREE) + for path, data in FILES.items(): + storage.file(path).write(data) + return storage + + +@pytest.fixture +def monitor(tree: Storage, bus: EventBus) -> IntegrityMonitor: + watching = IntegrityMonitor(TREE, baseline=BASELINE, bus=bus) + watching.create_baseline() + return watching + + +def _local_tree(tmp_path: Path) -> Path: + root = tmp_path / "tree" + (root / "sub").mkdir(parents=True) + for path, data in FILES.items(): + (root / path).write_bytes(data) + return root + + +def _rewrite_in_place(path: Path, data: bytes) -> None: + """Replace the content of ``path`` and put its modification time back.""" + stamp = path.stat() + path.write_bytes(data) + os.utime(path, ns=(stamp.st_atime_ns, stamp.st_mtime_ns)) + + +# ---------------------------------------------------------------------- snapshot and baseline + + +def test_snapshot_mode_reads_the_tree_and_stores_nothing(tree: Storage) -> None: + watching = IntegrityMonitor(TREE) + snapshot = watching.snapshot() + assert isinstance(snapshot, Snapshot) + assert snapshot.paths == ("a.txt", "b.txt", "sub/c.txt") + assert (watching.target, watching.baseline, watching.algorithm) == (TREE, None, "sha256") + assert watching.has_baseline() is False + assert [info.path for info in Storage("memory://monitor").list_dir()] == ["tree"] + assert tree.file("a.txt").read() == b"alpha" + for call in (watching.create_baseline, watching.verify, watching.accept, watching.watch): + with pytest.raises(IntegrityException, match="has no baseline"): + call() + + +def test_create_baseline_stores_a_schema_2_manifest(monitor: IntegrityMonitor) -> None: + assert monitor.has_baseline() is True + document = json.loads(File(BASELINE).read_text()) + assert document["schema_version"] == 2 + assert (document["root"], document["backend"], document["algorithm"]) == ( + TREE, + "memory", + "sha256", + ) + assert [entry["path"] for entry in document["entries"]] == ["a.txt", "b.txt", "sub/c.txt"] + assert document["entries"][0]["checksum"] == hashlib.sha256(b"alpha").hexdigest() + assert [info.path for info in Storage("memory://baselines").list_dir()] == ["tree.json"] + + +def test_a_clean_tree_verifies_ok_and_publishes_nothing( + monitor: IntegrityMonitor, bus: EventBus +) -> None: + report = monitor.verify() + assert isinstance(report, DriftReport) + assert report.ok is True + assert report.changes == () + assert (report.target, report.baseline, report.backend) == (TREE, BASELINE, "memory") + assert (report.deep, report.partial, report.checked, report.hashed) == (True, False, 3, 3) + assert report.notes == () + assert monitor.last_report is report + assert monitor.last_run == report.verified_at + assert monitor.last_error is None + assert bus.recent() == [] + + +def test_verify_reports_every_kind_of_change(monitor: IntegrityMonitor, tree: Storage) -> None: + tree.file("a.txt").write(b"tampered") + tree.file("b.txt").move_to(tree.file("moved/b.txt")) + tree.file("sub/c.txt").delete() + tree.file("new.txt").write(b"novel") + report = monitor.verify() + assert [(change.kind.value, change.path) for change in report.changes] == [ + ("modified", "a.txt"), + ("renamed", "moved/b.txt"), + ("created", "new.txt"), + ("deleted", "sub/c.txt"), + ] + assert report.changes[1].previous_path == "b.txt" + assert report.ok is False + + +def test_the_baseline_can_live_in_another_backend(tmp_path: Path, bus: EventBus) -> None: + root = _local_tree(tmp_path) + in_memory = IntegrityMonitor(root, baseline="memory://elsewhere/base/tree.json", bus=bus) + in_memory.create_baseline() + assert load_manifest(File("memory://elsewhere/base/tree.json").read()).backend == "local" + (root / "a.txt").write_bytes(b"tampered") + assert in_memory.verify().paths("modified") == ["a.txt"] + + storage = Storage(TREE) + storage.file("a.txt").write(b"alpha") + on_disk = IntegrityMonitor(TREE, baseline=tmp_path / "state" / "tree.json", bus=bus) + on_disk.create_baseline() + assert (tmp_path / "state" / "tree.json").is_file() + assert sorted(path.name for path in (tmp_path / "state").iterdir()) == ["tree.json"] + storage.file("a.txt").delete() + assert on_disk.verify().paths("deleted") == ["a.txt"] + + +def test_a_failed_save_leaves_the_previous_baseline_and_no_temporary_file( + monitor: IntegrityMonitor, tree: Storage, monkeypatch: pytest.MonkeyPatch +) -> None: + before = File(BASELINE).read() + tree.file("new.txt").write(b"novel") + + def _deny(*_args: object, **_options: object) -> None: + raise StoragePermissionException("access to the baseline was denied") + + monkeypatch.setattr(File, "move_to", _deny) + with pytest.raises(StoragePermissionException): + monitor.create_baseline() + assert File(BASELINE).read() == before + assert [info.path for info in Storage("memory://baselines").list_dir()] == ["tree.json"] + + +def test_a_baseline_inside_the_target_is_not_part_of_the_tree(tree: Storage, bus: EventBus) -> None: + inside = f"{TREE}/.fa/baseline.json" + watching = IntegrityMonitor(TREE, baseline=inside, bus=bus) + assert watching.create_baseline().paths == ("a.txt", "b.txt", "sub/c.txt") + assert tree.file(".fa/baseline.json").is_file() + assert watching.verify().ok is True + assert watching.snapshot().paths == ("a.txt", "b.txt", "sub/c.txt") + tree.file(".fa/.baseline.json.0123abcd.tmp").write(b"half-written") + tree.file(".fa/other.json").write(b"{}") + assert watching.verify().paths("created") == [".fa/other.json"] + + +def test_the_baseline_cannot_be_the_target_itself_or_a_storage_root(tree: Storage) -> None: + with pytest.raises(IntegrityException, match="same location"): + IntegrityMonitor(TREE, baseline=TREE) + with pytest.raises(IntegrityException, match="must be a file"): + BaselineManager("memory://baselines") + + +def test_a_missing_or_broken_baseline_is_an_error(tree: Storage, bus: EventBus) -> None: + watching = IntegrityMonitor(TREE, baseline=BASELINE, bus=bus) + with pytest.raises(IntegrityException, match=r"no baseline at memory://baselines/tree\.json"): + watching.verify() + File(BASELINE).write(b"{not json") + with pytest.raises(IntegrityException, match="not readable JSON"): + watching.verify() + File(BASELINE).write(json.dumps({"schema_version": 9, "entries": []})) + with pytest.raises(IntegrityException, match="schema version 9"): + watching.verify() + Storage("memory://baselines").mkdir("folder.json") + with pytest.raises(IntegrityException, match="is a directory"): + IntegrityMonitor(TREE, baseline="memory://baselines/folder.json").verify() + assert bus.recent() == [] + + +# ---------------------------------------------------------------------- the event + + +def test_drift_is_published_as_one_integrity_violation( + monitor: IntegrityMonitor, tree: Storage, bus: EventBus +) -> None: + tree.file("a.txt").write(b"tampered") + tree.file("new.txt").write(b"novel") + received: list[Event] = [] + bus.subscribe(received.append) + report = monitor.verify() + (event,) = received + assert isinstance(event, IntegrityViolation) + assert event.type == "integrity.violation" + assert event.source == "integrity" + assert event.severity is Severity.ERROR + assert event.subject == f"integrity drift: {TREE} (1 created, 1 modified)" + assert len(event.correlation_id) == 32 + assert event.correlation_id == report.correlation_id + assert json.loads(json.dumps(event.to_dict()))["payload"] == { + "action": "verify", + "resource": TREE, + "backend": "memory", + "status": "drift", + "baseline": BASELINE, + "algorithm": "sha256", + "deep": True, + "partial": False, + "counts": { + "created": 1, + "modified": 1, + "deleted": 0, + "renamed": 0, + "metadata_changed": 0, + "permission_changed": 0, + }, + "total": 2, + "changes": [ + {"kind": "modified", "path": "a.txt"}, + {"kind": "created", "path": "new.txt"}, + ], + "truncated": False, + } + + +def test_the_event_joins_the_enclosing_correlation_scope( + monitor: IntegrityMonitor, tree: Storage, bus: EventBus +) -> None: + tree.file("a.txt").delete() + with correlation_scope("run-42"): + report = monitor.verify() + assert report.correlation_id == "run-42" + assert [event.correlation_id for event in bus.recent()] == ["run-42"] + again = monitor.verify() + assert again.correlation_id != "run-42" + assert bus.recent(limit=1)[0].correlation_id == again.correlation_id + + +def test_only_additions_are_a_warning( + monitor: IntegrityMonitor, tree: Storage, bus: EventBus +) -> None: + tree.file("new.txt").write(b"novel") + monitor.verify() + assert [event.severity for event in bus.recent()] == [Severity.WARNING] + + +@pytest.mark.parametrize( + "change", + ["modified", "deleted", "renamed"], +) +def test_anything_that_alters_the_baseline_content_is_an_error( + monitor: IntegrityMonitor, tree: Storage, bus: EventBus, change: str +) -> None: + tree.file("new.txt").write(b"novel") + if change == "modified": + tree.file("a.txt").write(b"tampered") + elif change == "deleted": + tree.file("a.txt").delete() + else: + tree.file("a.txt").move_to(tree.file("z.txt")) + assert monitor.verify().counts[change] == 1 + assert [event.severity for event in bus.recent()] == [Severity.ERROR] + + +def test_metadata_and_permission_changes(tmp_path: Path, bus: EventBus) -> None: + root = _local_tree(tmp_path) + watching = IntegrityMonitor(root, baseline=tmp_path / "baseline.json", bus=bus) + watching.create_baseline() + stamp = (root / "a.txt").stat() + os.utime(root / "a.txt", ns=(stamp.st_atime_ns, stamp.st_mtime_ns + 5_000_000_000)) + report = watching.verify() + (touched,) = report.changes + assert (touched.kind.value, touched.path, touched.fields) == ( + "metadata_changed", + "a.txt", + ("modified_at",), + ) + assert bus.recent(limit=1)[0].severity is Severity.WARNING + + watching.accept(report) + os.chmod(root / "b.txt", stat.S_IREAD) + try: + (locked,) = watching.verify().changes + finally: + os.chmod(root / "b.txt", stat.S_IREAD | stat.S_IWRITE) + assert (locked.kind.value, locked.path, locked.fields) == ( + "permission_changed", + "b.txt", + ("mode",), + ) + assert locked.before is not None and locked.after is not None + assert locked.before.mode != locked.after.mode + assert bus.recent(limit=1)[0].severity is Severity.ERROR + + +def test_the_severity_of_each_kind_is_configurable(tree: Storage, bus: EventBus) -> None: + policy = AlertPolicy(severities={"created": "critical", "modified": Severity.INFO}) + watching = IntegrityMonitor(TREE, baseline=BASELINE, bus=bus, alerts=policy) + watching.create_baseline() + tree.file("a.txt").write(b"tampered") + watching.verify() + tree.file("new.txt").write(b"novel") + watching.verify() + assert [event.severity for event in bus.recent()] == [Severity.CRITICAL, Severity.INFO] + with pytest.raises(IntegrityException, match="invalid alert severity"): + AlertPolicy(severities={"created": "loud"}) + with pytest.raises(IntegrityException, match="invalid alert severity"): + AlertPolicy(severities={"vanished": "error"}) + with pytest.raises(IntegrityException, match="max_changes"): + AlertPolicy(max_changes=-1) + + +def test_the_payload_lists_only_the_first_changes(tree: Storage, bus: EventBus) -> None: + watching = IntegrityMonitor(TREE, baseline=BASELINE, bus=bus, alerts=AlertPolicy(max_changes=2)) + watching.create_baseline() + for index in range(5): + tree.file(f"new{index}.txt").write(f"novel {index}") + watching.verify() + payload = bus.recent()[0].payload + assert [change["path"] for change in payload["changes"]] == ["new0.txt", "new1.txt"] + assert (payload["total"], payload["truncated"], payload["counts"]["created"]) == (5, True, 5) + + +def test_events_go_to_the_process_wide_bus_by_default(tree: Storage) -> None: + received: list[Event] = [] + subscription = event_bus.subscribe(received.append, types=IntegrityViolation) + try: + watching = IntegrityMonitor(TREE, baseline=BASELINE) + watching.create_baseline() + tree.file("a.txt").write(b"tampered") + watching.verify() + finally: + event_bus.unsubscribe(subscription) + assert [event.payload["resource"] for event in received] == [TREE] + + +# ---------------------------------------------------------------------- deep and quick + + +def test_a_quick_pass_hashes_only_what_looks_different(tmp_path: Path, bus: EventBus) -> None: + root = _local_tree(tmp_path) + watching = IntegrityMonitor(root, baseline=tmp_path / "baseline.json", bus=bus) + watching.create_baseline() + quick = watching.verify(deep=False) + assert (quick.ok, quick.deep, quick.checked, quick.hashed) == (True, False, 3, 0) + assert quick.notes == ( + "quick pass: 0 of 3 files hashed; size, modification time and etag decided the rest", + ) + assert quick.to_dict()["deep"] is False + + (root / "a.txt").write_bytes(b"tampered with") + (root / "new.txt").write_bytes(b"novel") + quick = watching.verify(deep=False) + assert [(change.kind.value, change.path) for change in quick.changes] == [ + ("modified", "a.txt"), + ("created", "new.txt"), + ] + assert (quick.checked, quick.hashed) == (4, 2) + assert bus.recent(limit=1)[0].payload["deep"] is False + + +def test_only_a_deep_pass_sees_a_change_that_keeps_size_and_time( + tmp_path: Path, bus: EventBus +) -> None: + root = _local_tree(tmp_path) + watching = IntegrityMonitor(root, baseline=tmp_path / "baseline.json", bus=bus) + watching.create_baseline() + _rewrite_in_place(root / "a.txt", b"ALPHA") + quick = watching.verify(deep=False) + assert (quick.ok, quick.hashed) == (True, 0) + deep = watching.verify() + assert (deep.deep, deep.hashed, deep.notes) == (True, 3, ()) + assert deep.paths("modified") == ["a.txt"] + + +def test_a_quick_pass_against_a_legacy_baseline_hashes_everything( + tmp_path: Path, bus: EventBus +) -> None: + root = _local_tree(tmp_path) + write_manifest(root, tmp_path / "manifest.json") + watching = IntegrityMonitor(root, tmp_path / "manifest.json", bus=bus) + quick = watching.verify(deep=False) + assert (quick.ok, quick.checked, quick.hashed) == (True, 3, 3) + + +# ---------------------------------------------------------------------- accept + + +def test_accept_stores_exactly_what_the_report_saw( + monitor: IntegrityMonitor, tree: Storage +) -> None: + tree.file("a.txt").write(b"approved change") + report = monitor.verify() + tree.file("late.txt").write(b"arrived after the report") + accepted = monitor.accept(report) + assert accepted is report.snapshot + assert accepted.paths == ("a.txt", "b.txt", "sub/c.txt") + after = monitor.verify() + assert [(change.kind.value, change.path) for change in after.changes] == [ + ("created", "late.txt") + ] + + +def test_accept_without_a_report_reads_the_tree_again( + monitor: IntegrityMonitor, tree: Storage +) -> None: + tree.file("a.txt").write(b"approved change") + tree.file("new.txt").write(b"novel") + assert monitor.accept().paths == ("a.txt", "b.txt", "new.txt", "sub/c.txt") + assert monitor.verify().ok is True + + +def test_accept_refuses_a_report_about_another_target(monitor: IntegrityMonitor) -> None: + with pytest.raises(IntegrityException, match="memory://other/tree, not this monitor's"): + monitor.accept(DriftReport(target="memory://other/tree")) + + +def test_accept_keeps_the_algorithm_of_the_baseline(tree: Storage, bus: EventBus) -> None: + IntegrityMonitor(TREE, baseline=BASELINE, algorithm="sha512", bus=bus).create_baseline() + plain = IntegrityMonitor(TREE, baseline=BASELINE, bus=bus) + assert plain.algorithm == "sha256" + report = plain.verify() + assert (report.ok, report.algorithm) == (True, "sha512") + tree.file("a.txt").write(b"approved change") + assert plain.accept().algorithm == "sha512" + assert plain.accept(plain.verify()).algorithm == "sha512" + assert load_manifest(File(BASELINE).read()).algorithm == "sha512" + assert plain.create_baseline().algorithm == "sha256" + + +def test_accept_turns_a_legacy_baseline_into_schema_2(tmp_path: Path, bus: EventBus) -> None: + root = _local_tree(tmp_path) + manifest_path = tmp_path / "manifest.json" + write_manifest(root, manifest_path) + watching = IntegrityMonitor(root, manifest_path, bus=bus) + (root / "a.txt").write_bytes(b"approved change") + watching.accept(watching.verify()) + document = json.loads(manifest_path.read_text(encoding="utf-8")) + assert document["schema_version"] == 2 + assert document["entries"][0]["modified_at"] is not None + assert watching.verify().ok is True + + +# ---------------------------------------------------------------------- algorithms + + +def test_a_monitor_refuses_a_weak_algorithm_unless_told_otherwise(tree: Storage) -> None: + with pytest.raises(IntegrityException, match="not collision-resistant"): + IntegrityMonitor(TREE, baseline=BASELINE, algorithm="md5") + with pytest.raises(IntegrityException, match="unsupported integrity algorithm"): + IntegrityMonitor(TREE, baseline=BASELINE, algorithm="crc32", allow_weak=True) + allowed = IntegrityMonitor(TREE, baseline=BASELINE, algorithm="md5", allow_weak=True) + entry = allowed.snapshot().get("a.txt") + assert entry is not None + assert entry.checksum == hashlib.md5(b"alpha", usedforsecurity=False).hexdigest() + + +def test_a_weak_baseline_is_not_verified_without_allow_weak(tree: Storage, bus: EventBus) -> None: + IntegrityMonitor( + TREE, baseline=BASELINE, algorithm="md5", allow_weak=True, bus=bus + ).create_baseline() + strict = IntegrityMonitor(TREE, baseline=BASELINE, bus=bus) + with pytest.raises(IntegrityException, match="md5 is refused"): + strict.verify() + with pytest.raises(IntegrityException, match="md5 is refused"): + strict.accept() + lenient = IntegrityMonitor(TREE, baseline=BASELINE, allow_weak=True, bus=bus) + assert lenient.verify().algorithm == "md5" + + +# ---------------------------------------------------------------------- continuous mode + + +def test_continuous_mode_verifies_on_a_thread( + monitor: IntegrityMonitor, tree: Storage, bus: EventBus +) -> None: + tree.file("a.txt").write(b"tampered") + found = threading.Event() + bus.subscribe(lambda _event: found.set(), types=IntegrityViolation) + watching = IntegrityMonitor(TREE, baseline=BASELINE, bus=bus, interval=0.02) + assert watching.is_running is False + watching.start() + try: + watching.start() + assert watching.is_running is True + assert found.wait(WAIT), "continuous mode did not verify" + finally: + watching.stop() + assert watching.is_running is False + report = watching.last_report + assert report is not None + assert report.paths("modified") == ["a.txt"] + assert watching.last_summary is not None + assert watching.last_summary["modified"] == ["a.txt"] + watching.stop() + + +def test_continuous_mode_survives_a_failing_pass(tree: Storage, bus: EventBus) -> None: + failed = threading.Event() + recovered = threading.Event() + + def _route(event: Event) -> None: + (failed if event.payload["status"] == "error" else recovered).set() + + bus.subscribe(_route, types=IntegrityViolation) + watching = IntegrityMonitor(TREE, baseline=BASELINE, bus=bus, interval=0.02) + watching.start() + try: + assert failed.wait(WAIT), "the missing baseline was not reported" + assert watching.last_error is not None + assert watching.last_error.startswith("IntegrityException: no baseline at ") + watching.create_baseline() + tree.file("a.txt").write(b"tampered") + assert recovered.wait(WAIT), "the monitor did not keep running after a failed pass" + finally: + watching.stop() + assert watching.last_error is None + + +def test_status_is_json_friendly(monitor: IntegrityMonitor, tree: Storage) -> None: + idle = monitor.status() + assert idle == { + "target": TREE, + "baseline": BASELINE, + "algorithm": "sha256", + "interval": 60.0, + "running": False, + "last_run": None, + "last_error": None, + "last_report": None, + } + tree.file("a.txt").write(b"tampered") + report = monitor.verify() + status = json.loads(json.dumps(monitor.status())) + assert status["last_run"] == report.verified_at.isoformat() + assert status["last_report"]["counts"]["modified"] == 1 + assert status["last_report"]["ok"] is False diff --git a/tests/test_integrity_object_store.py b/tests/test_integrity_object_store.py new file mode 100644 index 0000000..aa72116 --- /dev/null +++ b/tests/test_integrity_object_store.py @@ -0,0 +1,264 @@ +"""The monitor on an object store: etags, versions, an emptied prefix, and polling. + +``_Bucket`` is an in-memory :class:`ObjectStorage` that behaves like S3 where it +matters here: a listing reports size, time and etag, and only a ``HEAD`` adds +the version and the content type. It counts its reads, so the tests can tell a +quick pass from a deep one. +""" + +from __future__ import annotations + +import hashlib +import threading +from collections.abc import Iterable, Iterator +from dataclasses import dataclass +from datetime import datetime, timedelta, timezone +from pathlib import Path + +import pytest + +from automation_file.events import Event, EventBus, IntegrityViolation +from automation_file.integrity import IntegrityMonitor +from automation_file.storage import ( + FileInfo, + ObjectStorage, + Storage, + StorageCapabilities, + StorageResolver, + clear_memory_stores, +) + +TARGET = "bucket://reports/2026" +BASELINE = "memory://baselines/reports.json" +WAIT = 10.0 + + +@dataclass +class _Object: + data: bytes + modified: datetime + version: str + + +class _Bucket(ObjectStorage): + scheme = "bucket" + capabilities = StorageCapabilities( + directories=False, modified_at=True, etag=True, version=True, content_type=True + ) + + def __init__(self) -> None: + super().__init__() + self.objects: dict[str, _Object] = {} + self.downloads: list[str] = [] + self.heads: list[str] = [] + self.listings = threading.Semaphore(0) + self._clock = datetime(2026, 10, 8, tzinfo=timezone.utc) + self._lock = threading.Lock() + + def uri_for(self, path: str = "") -> str: + return f"bucket://reports/{self._normalize(path)}" + + def _head(self, key: str) -> FileInfo | None: + with self._lock: + item = self.objects.get(key) + self.heads.append(key) + if item is None: + return None + return FileInfo( + path=key, + size=len(item.data), + # An HTTP Last-Modified header carries whole seconds; the listing has the stored time. + modified_at=item.modified.replace(microsecond=0), + etag=hashlib.md5(item.data, usedforsecurity=False).hexdigest(), + version=item.version, + content_type="text/csv" if key.endswith(".csv") else "application/octet-stream", + ) + + def _scan(self, key_prefix: str, *, shallow: bool) -> Iterable[FileInfo]: + found: list[FileInfo] = [] + prefixes: set[str] = set() + with self._lock: + items = sorted(self.objects.items()) + for key, item in items: + if not key.startswith(key_prefix): + continue + rest = key[len(key_prefix) :] + if shallow and "/" in rest: + prefixes.add(key_prefix + rest.split("/", 1)[0]) + continue + found.append( + FileInfo( + path=key, + size=len(item.data), + modified_at=item.modified, + etag=hashlib.md5(item.data, usedforsecurity=False).hexdigest(), + ) + ) + if not shallow: + self.listings.release() + return [*(FileInfo(path=prefix, is_dir=True) for prefix in sorted(prefixes)), *found] + + def _put(self, source: Path, key: str) -> None: + with self._lock: + self._clock += timedelta(seconds=1, microseconds=250_000) + generation = int(self.objects[key].version[1:]) + 1 if key in self.objects else 1 + self.objects[key] = _Object(source.read_bytes(), self._clock, f"v{generation}") + + def _get(self, key: str, target: Path) -> None: + with self._lock: + self.downloads.append(key) + data = self.objects[key].data + target.write_bytes(data) + + def _remove(self, key: str) -> None: + with self._lock: + self.objects.pop(key, None) + + +@pytest.fixture(autouse=True) +def _fresh_stores() -> Iterator[None]: + clear_memory_stores() + yield + clear_memory_stores() + + +@pytest.fixture +def bucket() -> _Bucket: + return _Bucket() + + +@pytest.fixture +def resolver(bucket: _Bucket) -> StorageResolver: + table = StorageResolver() + table.mount("bucket://reports", bucket) + return table + + +@pytest.fixture +def bus() -> EventBus: + return EventBus() + + +@pytest.fixture +def reports(resolver: StorageResolver) -> Storage: + storage = Storage(TARGET, resolver=resolver) + storage.file("q1.csv").write(b"region,total\nEMEA,42\n") + storage.file("q2.csv").write(b"region,total\nEMEA,43\n") + storage.file("raw/dump.bin").write(b"\x00\x01\x02") + return storage + + +@pytest.fixture +def monitor(reports: Storage, resolver: StorageResolver, bus: EventBus) -> IntegrityMonitor: + watching = IntegrityMonitor(TARGET, baseline=BASELINE, resolver=resolver, bus=bus) + watching.create_baseline() + return watching + + +def test_a_snapshot_takes_the_etag_from_the_listing_and_the_rest_from_a_head( + monitor: IntegrityMonitor, bucket: _Bucket +) -> None: + bucket.heads.clear() + snapshot = monitor.snapshot() + assert (snapshot.root, snapshot.backend) == (TARGET, "bucket") + assert snapshot.paths == ("q1.csv", "q2.csv", "raw/dump.bin") + entry = snapshot.get("q1.csv") + assert entry is not None + assert entry.checksum == hashlib.sha256(b"region,total\nEMEA,42\n").hexdigest() + assert entry.etag == hashlib.md5(b"region,total\nEMEA,42\n", usedforsecurity=False).hexdigest() + assert (entry.version, entry.content_type, entry.backend, entry.mode) == ( + "v1", + "text/csv", + "bucket", + None, + ) + assert entry.modified_at == bucket.objects["2026/q1.csv"].modified + assert sorted(bucket.heads).count("2026/q1.csv") >= 1 + + +def test_a_quick_pass_downloads_nothing_while_the_listing_matches( + monitor: IntegrityMonitor, reports: Storage, bucket: _Bucket +) -> None: + bucket.downloads.clear() + quick = monitor.verify(deep=False) + assert (quick.ok, quick.deep, quick.checked, quick.hashed) == (True, False, 3, 0) + assert bucket.downloads == [] + assert monitor.verify().hashed == 3 + assert sorted(bucket.downloads) == ["2026/q1.csv", "2026/q2.csv", "2026/raw/dump.bin"] + + bucket.downloads.clear() + reports.file("q2.csv").write(b"region,total\nEMEA,99\n") + quick = monitor.verify(deep=False) + assert quick.paths("modified") == ["q2.csv"] + assert (quick.hashed, bucket.downloads) == (1, ["2026/q2.csv"]) + + +def test_the_same_content_stored_again_is_a_metadata_change( + monitor: IntegrityMonitor, reports: Storage +) -> None: + reports.file("q1.csv").write(b"region,total\nEMEA,42\n") + for deep in (False, True): + (change,) = monitor.verify(deep=deep).changes + assert (change.kind.value, change.path) == ("metadata_changed", "q1.csv") + assert change.fields == ("modified_at", "version") + accepted = monitor.accept(monitor.verify(deep=False)).get("q2.csv") + assert accepted is not None + assert (accepted.version, accepted.content_type) == ("v1", "text/csv") + assert monitor.verify(deep=False).ok is True + + +def test_a_prefix_that_holds_nothing_any_more_is_an_empty_tree( + monitor: IntegrityMonitor, reports: Storage, bucket: _Bucket +) -> None: + bucket.objects.clear() + report = monitor.verify() + assert report.counts["deleted"] == 3 + assert report.paths("deleted") == ["q1.csv", "q2.csv", "raw/dump.bin"] + assert monitor.snapshot().paths == () + + +def test_a_single_path_is_verified_with_the_listing_view_of_it( + monitor: IntegrityMonitor, reports: Storage, bucket: _Bucket +) -> None: + listed = bucket.objects["2026/q1.csv"].modified + assert reports.stat("q1.csv").modified_at != listed + clean = monitor.verify_paths(["q1.csv", "raw"]) + assert (clean.ok, clean.partial, clean.checked) == (True, True, 2) + reports.file("q1.csv").write(b"region,total\nEMEA,0\n") + bucket.objects.pop("2026/q2.csv") + report = monitor.verify_paths(["q1.csv", "q2.csv", "raw/dump.bin"]) + assert [(change.kind.value, change.path) for change in report.changes] == [ + ("modified", "q1.csv"), + ("deleted", "q2.csv"), + ] + + +def test_another_backend_is_watched_by_polling_and_reports_a_drift_once( + monitor: IntegrityMonitor, reports: Storage, bucket: _Bucket, bus: EventBus +) -> None: + received: list[Event] = [] + arrived = threading.Semaphore(0) + + def _collect(event: Event) -> None: + received.append(event) + arrived.release() + + bus.subscribe(_collect, types=IntegrityViolation) + reports.file("q1.csv").write(b"region,total\nEMEA,0\n") + with monitor.watch(poll_interval=0.01) as handle: + assert (handle.kind, handle.is_running) == ("poll", True) + assert arrived.acquire(timeout=WAIT), "the poll did not report the drift" + while bucket.listings.acquire(blocking=False): + pass + for _ in range(3): + assert bucket.listings.acquire(timeout=WAIT), "the poll stopped" + assert len(received) == 1 + reports.file("new.csv").write(b"region,total\n") + assert arrived.acquire(timeout=WAIT), "the poll did not report the second drift" + assert handle.is_running is False + first, second = received + assert (first.payload["deep"], first.payload["counts"]["modified"]) == (False, 1) + assert second.payload["counts"]["created"] == 1 + report = monitor.last_report + assert report is not None + assert (report.deep, report.partial) == (False, False) diff --git a/tests/test_integrity_remediation.py b/tests/test_integrity_remediation.py new file mode 100644 index 0000000..bba8425 --- /dev/null +++ b/tests/test_integrity_remediation.py @@ -0,0 +1,414 @@ +"""Remediation: off by default, quarantine, restore with checksum verification.""" + +from __future__ import annotations + +from collections.abc import Iterator +from typing import Any + +import pytest + +from automation_file.events import Event, EventBus, IntegrityViolation, Severity +from automation_file.exceptions import StoragePermissionException +from automation_file.integrity import ( + IntegrityException, + IntegrityMonitor, + IntegrityRemediated, + RemediationPolicy, + RemediationStep, + Remediator, + Target, +) +from automation_file.storage import File, Storage, clear_memory_stores + +TREE = "memory://prod/tree" +BASELINE = "memory://state/tree.json" +QUARANTINE = "memory://quarantine/tree" +MIRROR = "memory://mirror/tree" +FILES = {"a.txt": b"alpha", "b.txt": b"bravo", "sub/c.txt": b"charlie"} + + +@pytest.fixture(autouse=True) +def _fresh_stores() -> Iterator[None]: + clear_memory_stores() + yield + clear_memory_stores() + + +@pytest.fixture +def bus() -> EventBus: + return EventBus() + + +@pytest.fixture +def tree() -> Storage: + storage = Storage(TREE) + for path, data in FILES.items(): + storage.file(path).write(data) + IntegrityMonitor(TREE, baseline=BASELINE, bus=EventBus()).create_baseline() + return storage + + +@pytest.fixture +def mirror(tree: Storage) -> Storage: + copy = Storage(MIRROR) + tree.copy_to(copy) + return copy + + +def _monitor(bus: EventBus, **policy: Any) -> IntegrityMonitor: + return IntegrityMonitor( + TREE, baseline=BASELINE, bus=bus, remediation=RemediationPolicy(**policy) + ) + + +def _files(uri: str) -> dict[str, bytes]: + storage = Storage(uri) + if not storage.exists(): + return {} + return { + info.path: storage.file(info.path).read() + for info in storage.list_dir(recursive=True) + if not info.is_dir + } + + +def _quarantined() -> dict[str, bytes]: + """Return the quarantined files without the timestamp directory they are under.""" + return {path.split("/", 1)[1]: data for path, data in _files(QUARANTINE).items()} + + +# ---------------------------------------------------------------------- off by default + + +def test_a_monitor_without_a_policy_changes_nothing(tree: Storage, bus: EventBus) -> None: + tree.file("a.txt").write(b"tampered") + tree.file("b.txt").delete() + tree.file("new.txt").write(b"novel") + expected = _files(TREE) + report = IntegrityMonitor(TREE, baseline=BASELINE, bus=bus).verify() + assert report.remediation == () + assert _files(TREE) == expected + assert [event.type for event in bus.recent()] == ["integrity.violation"] + assert "remediation" not in bus.recent()[0].payload + + +def test_a_policy_does_nothing_until_an_action_is_chosen( + tree: Storage, mirror: Storage, bus: EventBus +) -> None: + policy = RemediationPolicy(quarantine=QUARANTINE, restore_from=MIRROR) + assert policy.active is False + assert (policy.on_created, policy.on_modified, policy.on_deleted) == ("none", "none", "none") + tree.file("a.txt").write(b"tampered") + tree.file("b.txt").delete() + tree.file("new.txt").write(b"novel") + expected = _files(TREE) + report = IntegrityMonitor(TREE, baseline=BASELINE, bus=bus, remediation=policy).verify() + assert report.remediation == () + assert _files(TREE) == expected + assert _files(QUARANTINE) == {} + + +@pytest.mark.parametrize( + "options,message", + [ + ({"on_created": "delete"}, "on_created must be one of none, quarantine"), + ({"on_created": "restore", "restore_from": MIRROR}, "on_created must be one of"), + ({"on_modified": "purge"}, "on_modified must be one of none, quarantine, restore"), + ({"on_deleted": "quarantine", "quarantine": QUARANTINE}, "on_deleted must be one of"), + ({"on_created": "quarantine"}, "needs a quarantine= storage URI"), + ({"on_modified": "restore"}, "needs a restore_from= storage URI"), + ({"on_deleted": "restore", "quarantine": QUARANTINE}, "needs a restore_from="), + ], +) +def test_a_policy_that_cannot_work_is_refused(options: dict[str, str], message: str) -> None: + with pytest.raises(IntegrityException, match=message): + RemediationPolicy(**options) + + +def test_the_quarantine_and_the_mirror_must_lie_outside_the_target(tree: Storage) -> None: + inside = RemediationPolicy(quarantine=f"{TREE}/.quarantine", on_created="quarantine") + with pytest.raises(IntegrityException, match="lies inside the monitored target"): + IntegrityMonitor(TREE, baseline=BASELINE, remediation=inside) + itself = RemediationPolicy(restore_from=TREE, on_deleted="restore") + with pytest.raises(IntegrityException, match=r"restore_from .* lies inside"): + IntegrityMonitor(TREE, baseline=BASELINE, remediation=itself) + with pytest.raises(IntegrityException, match="must be a RemediationPolicy"): + Remediator({"on_created": "quarantine"}, Target(TREE)) # type: ignore[arg-type] + sibling = RemediationPolicy(quarantine="memory://prod/tree-quarantine", on_created="quarantine") + assert IntegrityMonitor(TREE, baseline=BASELINE, remediation=sibling).target == TREE + + +# ---------------------------------------------------------------------- quarantine + + +def test_a_created_file_is_moved_to_a_timestamped_quarantine(tree: Storage, bus: EventBus) -> None: + tree.file("drop/new.txt").write(b"novel") + received: list[Event] = [] + bus.subscribe(received.append) + report = _monitor(bus, quarantine=QUARANTINE, on_created="quarantine").verify() + + assert tree.file("drop/new.txt").exists() is False + (stored,) = _files(QUARANTINE).items() + stamp, _, path = stored[0].partition("/") + assert (path, stored[1]) == ("drop/new.txt", b"novel") + assert len(stamp) == len("20261008T101530123456Z") and stamp.endswith("Z") + assert stamp[:8].isdigit() and stamp[8] == "T" + + (step,) = report.remediation + assert step == RemediationStep( + action="quarantine", + path="drop/new.txt", + kind="created", + ok=True, + source=f"{TREE}/drop/new.txt", + destination=f"{QUARANTINE}/{stamp}/drop/new.txt", + error=None, + ) + assert report.paths("created") == ["drop/new.txt"] + assert report.to_dict()["remediation"][0]["ok"] is True + + violation, remediated = received + assert isinstance(violation, IntegrityViolation) + assert violation.payload["remediation"] == {"ok": 1, "failed": 0} + assert isinstance(remediated, IntegrityRemediated) + assert remediated.type == "integrity.remediated" + assert (remediated.source, remediated.severity) == ("integrity", Severity.INFO) + assert remediated.subject == f"integrity quarantine done: {TREE}/drop/new.txt" + assert remediated.correlation_id == violation.correlation_id == report.correlation_id + assert dict(remediated.payload) == { + "action": "quarantine", + "resource": f"{TREE}/drop/new.txt", + "backend": "memory", + "status": "ok", + "path": "drop/new.txt", + "kind": "created", + "source": f"{TREE}/drop/new.txt", + "destination": f"{QUARANTINE}/{stamp}/drop/new.txt", + "error": None, + } + assert _monitor(bus, quarantine=QUARANTINE, on_created="quarantine").verify().ok is True + + +def test_a_modified_file_can_be_quarantined(tree: Storage, bus: EventBus) -> None: + tree.file("a.txt").write(b"tampered") + report = _monitor(bus, quarantine=QUARANTINE, on_modified="quarantine").verify() + assert [(step.action, step.path, step.kind, step.ok) for step in report.remediation] == [ + ("quarantine", "a.txt", "modified", True) + ] + assert _quarantined() == {"a.txt": b"tampered"} + assert tree.file("a.txt").exists() is False + + +def test_a_failed_quarantine_is_reported_and_leaves_the_file( + tree: Storage, bus: EventBus, monkeypatch: pytest.MonkeyPatch +) -> None: + tree.file("new.txt").write(b"novel") + + def _deny(*_args: object, **_options: object) -> None: + raise StoragePermissionException("access to the quarantine was denied") + + monkeypatch.setattr(File, "move_to", _deny) + report = _monitor(bus, quarantine=QUARANTINE, on_created="quarantine").verify() + (step,) = report.remediation + assert (step.ok, step.error) == ( + False, + "StoragePermissionException: access to the quarantine was denied", + ) + assert tree.file("new.txt").read() == b"novel" + remediated = bus.recent(types=IntegrityRemediated)[0] + assert (remediated.severity, remediated.payload["status"]) == (Severity.ERROR, "failed") + assert bus.recent(types=IntegrityViolation)[0].payload["remediation"] == {"ok": 0, "failed": 1} + + +# ---------------------------------------------------------------------- restore + + +def test_a_deleted_file_is_restored_and_checked_against_the_baseline( + tree: Storage, mirror: Storage, bus: EventBus +) -> None: + tree.file("sub/c.txt").delete() + monitor = _monitor(bus, restore_from=MIRROR, on_deleted="restore") + report = monitor.verify() + (step,) = report.remediation + assert step == RemediationStep( + action="restore", + path="sub/c.txt", + kind="deleted", + ok=True, + source=f"{MIRROR}/sub/c.txt", + destination=f"{TREE}/sub/c.txt", + error=None, + ) + assert tree.file("sub/c.txt").read() == b"charlie" + assert report.ok is False + after = monitor.verify() + assert after.counts["deleted"] == after.counts["modified"] == 0 + assert after.remediation == () + + +def test_a_modified_file_is_set_aside_before_it_is_restored( + tree: Storage, mirror: Storage, bus: EventBus +) -> None: + tree.file("a.txt").write(b"tampered") + report = _monitor( + bus, quarantine=QUARANTINE, restore_from=MIRROR, on_modified="restore" + ).verify() + assert [(step.action, step.kind, step.ok) for step in report.remediation] == [ + ("quarantine", "modified", True), + ("restore", "modified", True), + ] + assert tree.file("a.txt").read() == b"alpha" + assert _quarantined() == {"a.txt": b"tampered"} + assert [ + event.payload["action"] for event in reversed(bus.recent(types=IntegrityRemediated)) + ] == [ + "quarantine", + "restore", + ] + + +def test_without_a_quarantine_a_restore_overwrites_the_modified_file( + tree: Storage, mirror: Storage, bus: EventBus +) -> None: + tree.file("a.txt").write(b"tampered") + report = _monitor(bus, restore_from=MIRROR, on_modified="restore").verify() + assert [(step.action, step.ok) for step in report.remediation] == [("restore", True)] + assert tree.file("a.txt").read() == b"alpha" + + +def test_a_restore_fails_when_the_mirror_has_no_copy( + tree: Storage, mirror: Storage, bus: EventBus +) -> None: + mirror.file("b.txt").delete() + tree.file("b.txt").delete() + report = _monitor(bus, restore_from=MIRROR, on_deleted="restore").verify() + (step,) = report.remediation + assert (step.action, step.path, step.ok) == ("restore", "b.txt", False) + assert step.error == f"the mirror {MIRROR} has no copy of it" + assert tree.file("b.txt").exists() is False + remediated = bus.recent(types=IntegrityRemediated)[0] + assert remediated.severity is Severity.ERROR + assert remediated.subject == f"integrity restore failed: {TREE}/b.txt" + assert (remediated.payload["status"], remediated.payload["error"]) == ("failed", step.error) + assert report.to_dict()["remediation"][0]["error"] == step.error + + +def test_a_mirror_copy_that_differs_from_the_baseline_is_never_copied( + tree: Storage, mirror: Storage, bus: EventBus +) -> None: + mirror.file("a.txt").write(b"a stale or tampered mirror") + tree.file("a.txt").write(b"tampered") + report = _monitor( + bus, quarantine=QUARANTINE, restore_from=MIRROR, on_modified="restore" + ).verify() + (step,) = report.remediation + assert (step.action, step.ok) == ("restore", False) + assert step.error == ( + "the mirror's copy does not match the baseline checksum; nothing was copied" + ) + assert tree.file("a.txt").read() == b"tampered" + assert _files(QUARANTINE) == {} + + +def test_a_restore_that_arrives_damaged_is_reported_as_failed( + tree: Storage, mirror: Storage, bus: EventBus, monkeypatch: pytest.MonkeyPatch +) -> None: + tree.file("b.txt").delete() + + def _corrupting_copy(_self: File, target: File, **_options: object) -> File: + target.write(b"damaged in transit") + return target + + monkeypatch.setattr(File, "copy_to", _corrupting_copy) + report = _monitor(bus, restore_from=MIRROR, on_deleted="restore").verify() + (step,) = report.remediation + assert (step.ok, step.error) == ( + False, + "the restored file does not match the baseline checksum", + ) + + +def test_a_restore_whose_copy_fails_is_reported_as_failed( + tree: Storage, mirror: Storage, bus: EventBus, monkeypatch: pytest.MonkeyPatch +) -> None: + tree.file("b.txt").delete() + + def _deny(*_args: object, **_options: object) -> None: + raise StoragePermissionException("access to the target was denied") + + monkeypatch.setattr(File, "copy_to", _deny) + report = _monitor(bus, restore_from=MIRROR, on_deleted="restore").verify() + (step,) = report.remediation + assert (step.ok, step.error) == ( + False, + "StoragePermissionException: access to the target was denied", + ) + assert tree.file("b.txt").exists() is False + + +def test_a_modified_file_that_cannot_be_set_aside_is_not_overwritten( + tree: Storage, mirror: Storage, bus: EventBus, monkeypatch: pytest.MonkeyPatch +) -> None: + tree.file("a.txt").write(b"tampered") + + def _deny(*_args: object, **_options: object) -> None: + raise StoragePermissionException("access to the quarantine was denied") + + monkeypatch.setattr(File, "move_to", _deny) + report = _monitor( + bus, quarantine=QUARANTINE, restore_from=MIRROR, on_modified="restore" + ).verify() + quarantine, restore = report.remediation + assert (quarantine.action, quarantine.ok) == ("quarantine", False) + assert (restore.action, restore.ok) == ("restore", False) + assert restore.error == "the modified file could not be quarantined, so it was left in place" + assert tree.file("a.txt").read() == b"tampered" + + +def test_a_rename_is_remediated_as_its_two_halves( + tree: Storage, mirror: Storage, bus: EventBus +) -> None: + tree.file("a.txt").move_to(tree.file("renamed.txt")) + report = _monitor( + bus, + quarantine=QUARANTINE, + restore_from=MIRROR, + on_created="quarantine", + on_deleted="restore", + ).verify() + assert report.counts["renamed"] == 1 + assert [(step.action, step.path, step.kind, step.ok) for step in report.remediation] == [ + ("restore", "a.txt", "renamed", True), + ("quarantine", "renamed.txt", "renamed", True), + ] + assert _files(TREE) == FILES + assert _quarantined() == {"renamed.txt": b"alpha"} + + +def test_metadata_and_permission_changes_are_never_remediated( + tree: Storage, mirror: Storage, bus: EventBus +) -> None: + tree.file("a.txt").write(b"alpha") + report = _monitor( + bus, + quarantine=QUARANTINE, + restore_from=MIRROR, + on_created="quarantine", + on_modified="restore", + on_deleted="restore", + ).verify() + assert report.counts["metadata_changed"] == 1 + assert report.remediation == () + + +def test_accept_reads_the_tree_again_after_a_verification_that_remediated( + tree: Storage, bus: EventBus +) -> None: + tree.file("a.txt").write(b"approved change") + tree.file("new.txt").write(b"novel") + monitor = _monitor(bus, quarantine=QUARANTINE, on_created="quarantine") + report = monitor.verify() + assert report.snapshot is not None and "new.txt" in report.snapshot + accepted = monitor.accept(report) + assert accepted.paths == ("a.txt", "b.txt", "sub/c.txt") + assert monitor.verify().ok is True diff --git a/tests/test_integrity_snapshot.py b/tests/test_integrity_snapshot.py new file mode 100644 index 0000000..6e865dc --- /dev/null +++ b/tests/test_integrity_snapshot.py @@ -0,0 +1,453 @@ +"""Snapshots, the manifest schema and the hash engine.""" + +from __future__ import annotations + +import dataclasses +import hashlib +import json +import os +import stat +import threading +from collections.abc import Iterator +from datetime import datetime, timedelta, timezone +from pathlib import Path +from typing import Any + +import pytest + +from automation_file.core.manifest import write_manifest +from automation_file.integrity import ( + MANIFEST_SCHEMA_VERSION, + HashEngine, + IntegrityException, + Snapshot, + SnapshotEntry, + Target, + build_snapshot, + dump_manifest, + from_manifest, + load_manifest, + to_manifest, +) +from automation_file.integrity.hashing import checked_algorithm +from automation_file.integrity.snapshot import quick_matches +from automation_file.storage import FileInfo, Storage, clear_memory_stores + +TREE = "memory://snap/tree" +FILES = {"a.txt": b"alpha", "sub/b.bin": b"bravo!", "sub/deep/c.txt": b""} +MOMENT = datetime(2026, 10, 8, 10, 15, 30, 123456, tzinfo=timezone.utc) + + +@pytest.fixture(autouse=True) +def _fresh_stores() -> Iterator[None]: + clear_memory_stores() + yield + clear_memory_stores() + + +def _write(uri: str, files: dict[str, bytes]) -> Storage: + storage = Storage(uri) + for path, data in files.items(): + storage.file(path).write(data) + return storage + + +def _snapshot(uri: str = TREE, algorithm: str = "sha256") -> Snapshot: + target = Target(uri) + return build_snapshot(target, HashEngine(algorithm), target.files()) + + +def _entry(path: str = "a.txt", **fields: Any) -> SnapshotEntry: + return SnapshotEntry(path=path, **fields) + + +# ---------------------------------------------------------------------- snapshot contents + + +def test_a_snapshot_lists_every_file_with_what_the_backend_reports() -> None: + storage = _write(TREE, FILES) + storage.mkdir("empty") + before = datetime.now(timezone.utc) - timedelta(seconds=5) + snapshot = _snapshot() + assert snapshot.root == TREE + assert snapshot.backend == "memory" + assert snapshot.algorithm == "sha256" + assert snapshot.created_at.utcoffset() == timedelta(0) + assert snapshot.paths == ("a.txt", "sub/b.bin", "sub/deep/c.txt") + assert len(snapshot) == 3 + assert "a.txt" in snapshot and "empty" not in snapshot + entry = snapshot.get("sub/b.bin") + assert entry is not None + assert entry.size == 6 + assert entry.checksum == hashlib.sha256(b"bravo!").hexdigest() + assert entry.algorithm == "sha256" + assert entry.backend == "memory" + assert entry.modified_at is not None and entry.modified_at > before + assert entry.modified_at.utcoffset() == timedelta(0) + assert (entry.content_type, entry.version, entry.etag, entry.mode) == (None, None, None, None) + assert snapshot.get("nope.txt") is None + + +def test_a_local_snapshot_records_permission_bits_and_content_type(tmp_path: Path) -> None: + (tmp_path / "tree" / "sub").mkdir(parents=True) + (tmp_path / "tree" / "a.txt").write_bytes(b"alpha") + (tmp_path / "tree" / "sub" / "data").write_bytes(b"\x00\x01") + snapshot = _snapshot(str(tmp_path / "tree")) + assert snapshot.root.startswith("local:///") + assert snapshot.backend == "local" + entry = snapshot.get("a.txt") + assert entry is not None + assert entry.mode == stat.S_IMODE(os.stat(tmp_path / "tree" / "a.txt").st_mode) + assert entry.content_type == "text/plain" + assert entry.backend == "local" + nested = snapshot.get("sub/data") + assert nested is not None + assert (nested.content_type, nested.size) == (None, 2) + assert nested.mode is not None + + +def test_a_file_that_vanishes_while_the_snapshot_is_taken_is_left_out() -> None: + _write(TREE, {"a.txt": b"alpha"}) + target = Target(TREE) + listed = [*target.files(), FileInfo(path="gone.txt", size=4, modified_at=MOMENT)] + snapshot = build_snapshot(target, HashEngine(), listed) + assert snapshot.paths == ("a.txt",) + + +def test_a_known_entry_keeps_its_checksum_instead_of_being_hashed() -> None: + _write(TREE, {"a.txt": b"alpha", "b.txt": b"bravo"}) + target = Target(TREE) + carried = _entry("a.txt", checksum="carried-over", content_type="text/x-kept", version="v7") + snapshot = build_snapshot(target, HashEngine(), target.files(), known={"a.txt": carried}) + first, second = snapshot.entries + assert (first.checksum, first.content_type, first.version) == ( + "carried-over", + "text/x-kept", + "v7", + ) + assert second.checksum == hashlib.sha256(b"bravo").hexdigest() + + +def test_quick_matches_trusts_only_an_identical_stamp() -> None: + baseline = Snapshot( + root=TREE, + entries=( + _entry("same.txt", size=5, modified_at=MOMENT, checksum="1"), + _entry("resized.txt", size=5, modified_at=MOMENT, checksum="2"), + _entry("touched.txt", size=5, modified_at=MOMENT, checksum="3"), + _entry("retagged.txt", size=5, modified_at=MOMENT, etag="old", checksum="4"), + _entry("legacy.txt", size=5, checksum="5"), + _entry("etag-only.txt", size=5, etag="tag", checksum="6"), + ), + ) + listed = [ + FileInfo(path="same.txt", size=5, modified_at=MOMENT), + FileInfo(path="resized.txt", size=6, modified_at=MOMENT), + FileInfo(path="touched.txt", size=5, modified_at=MOMENT + timedelta(seconds=1)), + FileInfo(path="retagged.txt", size=5, modified_at=MOMENT, etag="new"), + FileInfo(path="legacy.txt", size=5), + FileInfo(path="etag-only.txt", size=5, etag="tag"), + FileInfo(path="new.txt", size=5, modified_at=MOMENT), + ] + assert sorted(quick_matches(listed, baseline)) == ["etag-only.txt", "same.txt"] + + +def test_quick_matches_compares_instants_not_time_zones() -> None: + baseline = Snapshot( + root=TREE, entries=(_entry("a.txt", size=1, modified_at=MOMENT, checksum="1"),) + ) + elsewhere = MOMENT.astimezone(timezone(timedelta(hours=8))) + naive = MOMENT.replace(tzinfo=None) + assert list(quick_matches([FileInfo(path="a.txt", size=1, modified_at=elsewhere)], baseline)) + assert list(quick_matches([FileInfo(path="a.txt", size=1, modified_at=naive)], baseline)) + + +# ---------------------------------------------------------------------- value types + + +def test_an_entry_round_trips_through_json() -> None: + entry = SnapshotEntry( + path="報告/q1 ✓.csv", + size=1024, + modified_at=MOMENT, + checksum="9f86d0", + algorithm="sha512", + content_type="text/csv", + backend="s3", + version="v3", + etag="5d41", + mode=0o640, + ) + document = json.loads(json.dumps(entry.to_dict())) + assert document["modified_at"] == "2026-10-08T10:15:30.123456+00:00" + assert document["mode"] == 0o640 + assert list(document) == [ + "path", + "size", + "modified_at", + "checksum", + "algorithm", + "content_type", + "backend", + "version", + "etag", + "mode", + ] + assert SnapshotEntry.from_dict(document) == entry + + +def test_from_dict_fills_the_gaps_and_reads_a_z_suffix() -> None: + entry = SnapshotEntry.from_dict( + {"path": "a.txt", "checksum": " ABC ", "modified_at": "2026-10-08T10:15:30Z"}, + algorithm="blake2b", + backend="memory", + ) + assert (entry.algorithm, entry.backend, entry.checksum) == ("blake2b", "memory", "abc") + assert entry.modified_at == datetime(2026, 10, 8, 10, 15, 30, tzinfo=timezone.utc) + assert (entry.size, entry.mode, entry.etag) == (None, None, None) + + +@pytest.mark.parametrize( + "document", + [ + "not an object", + {}, + {"path": ""}, + {"path": 7}, + {"path": "../outside.txt"}, + {"path": "a/../../outside.txt"}, + {"path": "a.txt", "size": -1}, + {"path": "a.txt", "size": True}, + {"path": "a.txt", "size": "12"}, + {"path": "a.txt", "checksum": 12}, + {"path": "a.txt", "mode": "0644"}, + {"path": "a.txt", "modified_at": "yesterday"}, + {"path": "a.txt", "modified_at": 1728382530}, + ], +) +def test_from_dict_rejects_a_malformed_entry(document: Any) -> None: + with pytest.raises(IntegrityException): + SnapshotEntry.from_dict(document) + + +def test_a_snapshot_is_frozen_sorted_and_refuses_a_duplicate_path() -> None: + snapshot = Snapshot(root=TREE, entries=(_entry("b.txt"), _entry("a.txt"))) + assert snapshot.paths == ("a.txt", "b.txt") + assert [entry.path for entry in snapshot] == ["a.txt", "b.txt"] + with pytest.raises(dataclasses.FrozenInstanceError): + snapshot.root = "elsewhere" # type: ignore[misc] + with pytest.raises(dataclasses.FrozenInstanceError): + snapshot.entries[0].size = 3 # type: ignore[misc] + with pytest.raises(IntegrityException, match="more than once"): + Snapshot(root=TREE, entries=(_entry("a.txt"), _entry("a.txt"))) + + +def test_below_returns_a_path_and_everything_under_it() -> None: + paths = ["a", "a.txt", "a/x.txt", "a/y/z.txt", "a0", "ab/c.txt", "b/a/x.txt"] + snapshot = Snapshot(root=TREE, entries=tuple(_entry(path) for path in paths)) + assert [entry.path for entry in snapshot.below("a")] == ["a", "a/x.txt", "a/y/z.txt"] + assert [entry.path for entry in snapshot.below("a/y")] == ["a/y/z.txt"] + assert [entry.path for entry in snapshot.below("a.txt")] == ["a.txt"] + assert snapshot.below("nope") == () + assert snapshot.below("") == snapshot.entries + + +def test_merged_replaces_removes_and_adds() -> None: + snapshot = Snapshot( + root="old", entries=(_entry("a.txt", checksum="1"), _entry("b.txt", checksum="2")) + ) + merged = snapshot.merged( + removed=["a.txt", "b.txt"], + added=[_entry("b.txt", checksum="3"), _entry("c.txt", checksum="4")], + root=TREE, + backend="memory", + ) + assert (merged.root, merged.backend) == (TREE, "memory") + assert [(entry.path, entry.checksum) for entry in merged] == [("b.txt", "3"), ("c.txt", "4")] + + +# ---------------------------------------------------------------------- manifest + + +def test_a_manifest_round_trips() -> None: + _write(TREE, FILES) + snapshot = _snapshot() + document = to_manifest(snapshot) + assert list(document) == [ + "schema_version", + "created_at", + "root", + "backend", + "algorithm", + "entries", + ] + assert document["schema_version"] == MANIFEST_SCHEMA_VERSION == 2 + assert [entry["path"] for entry in document["entries"]] == list(snapshot.paths) + assert from_manifest(json.loads(json.dumps(document))) == snapshot + encoded = dump_manifest(snapshot) + assert isinstance(encoded, bytes) + assert load_manifest(encoded) == snapshot + assert load_manifest(encoded.decode("utf-8")) == snapshot + assert Snapshot.from_dict(snapshot.to_dict()) == snapshot + + +def test_a_manifest_keeps_non_ascii_paths_readable() -> None: + _write(TREE, {"報告/q1 ✓.csv": b"x"}) + encoded = dump_manifest(_snapshot()) + assert "報告/q1 ✓.csv" in encoded.decode("utf-8") + assert load_manifest(encoded).paths == ("報告/q1 ✓.csv",) + + +def test_the_legacy_manifest_format_is_converted(tmp_path: Path) -> None: + root = tmp_path / "tree" + (root / "nested").mkdir(parents=True) + (root / "a.txt").write_bytes(b"alpha") + (root / "nested" / "b.txt").write_bytes(b"bravo") + legacy = write_manifest(root, tmp_path / "manifest.json") + assert "schema_version" not in legacy + snapshot = load_manifest((tmp_path / "manifest.json").read_bytes()) + assert snapshot.paths == ("a.txt", "nested/b.txt") + assert snapshot.algorithm == "sha256" + assert snapshot.backend == "local" + assert snapshot.root == _snapshot(str(root)).root + assert snapshot.created_at == datetime.fromisoformat(legacy["created_at"]) + entry = snapshot.get("nested/b.txt") + assert entry is not None + assert (entry.size, entry.checksum) == (5, hashlib.sha256(b"bravo").hexdigest()) + assert (entry.modified_at, entry.mode, entry.algorithm) == (None, None, "sha256") + assert to_manifest(snapshot)["schema_version"] == 2 + + +def test_a_legacy_entry_without_a_checksum_is_kept_without_one() -> None: + snapshot = from_manifest({"files": {"a.txt": {"size": "big"}, "b.txt": None}}) + assert [(entry.path, entry.size, entry.checksum) for entry in snapshot] == [ + ("a.txt", None, ""), + ("b.txt", None, ""), + ] + assert snapshot.root == "" + + +@pytest.mark.parametrize("version", [1, 3, "2", 2.5, None, True]) +def test_an_unknown_schema_version_is_refused(version: Any) -> None: + document = {"schema_version": version, "entries": [], "root": TREE, "algorithm": "sha256"} + with pytest.raises(IntegrityException, match="schema version"): + from_manifest(document, origin="baseline memory://b/m.json") + + +def test_a_legacy_manifest_of_an_unknown_version_is_refused() -> None: + with pytest.raises(IntegrityException, match="legacy manifest of version 4"): + from_manifest({"version": 4, "files": {}}) + + +@pytest.mark.parametrize( + "document", + [ + [], + "text", + {"version": 1}, + {"files": ["a.txt"]}, + {"schema_version": 2}, + {"schema_version": 2, "entries": {"a.txt": {}}}, + {"schema_version": 2, "entries": [{"path": "../x"}]}, + {"schema_version": 2, "entries": [{"path": "a"}, {"path": "a"}]}, + {"schema_version": 2, "entries": [], "created_at": "soon"}, + {"files": {"../x": {"size": 1, "checksum": "ab"}}}, + ], +) +def test_a_document_that_is_not_a_manifest_is_refused(document: Any) -> None: + with pytest.raises(IntegrityException, match="the baseline"): + from_manifest(document, origin="the baseline") + + +@pytest.mark.parametrize("data", [b"{not json", b"\xff\xfe\x00", ""]) +def test_unreadable_json_is_refused(data: bytes | str) -> None: + with pytest.raises(IntegrityException, match="not readable JSON"): + load_manifest(data) + + +# ---------------------------------------------------------------------- hash engine + + +@pytest.mark.parametrize("algorithm", ["sha256", "sha512", "blake2b"]) +def test_the_strong_algorithms_hash_through_the_storage_layer(algorithm: str) -> None: + storage = _write(TREE, FILES) + engine = HashEngine(algorithm.upper()) + assert engine.algorithm == algorithm + assert engine.hash_many(storage, FILES) == { + path: hashlib.new(algorithm, data).hexdigest() for path, data in FILES.items() + } + assert {entry.algorithm for entry in _snapshot(algorithm=algorithm)} == {algorithm} + + +@pytest.mark.parametrize("algorithm", ["md5", "MD5", "sha1"]) +def test_a_weak_algorithm_is_refused_with_the_reason(algorithm: str) -> None: + with pytest.raises(IntegrityException) as refusal: + HashEngine(algorithm) + message = str(refusal.value) + assert "not collision-resistant" in message + assert "allow_weak=True" in message + assert "sha256" in message + + +def test_a_weak_algorithm_works_when_explicitly_allowed() -> None: + storage = _write(TREE, {"a.txt": b"alpha"}) + engine = HashEngine("md5", allow_weak=True) + assert engine.algorithm == "md5" + assert engine.hash_file(storage, "a.txt") == ( + hashlib.md5(b"alpha", usedforsecurity=False).hexdigest() + ) + + +@pytest.mark.parametrize("algorithm", ["sha384", "crc32", "", "shake_128"]) +def test_an_unsupported_algorithm_is_refused_even_when_weak_ones_are_allowed( + algorithm: str, +) -> None: + with pytest.raises(IntegrityException, match="unsupported integrity algorithm"): + checked_algorithm(algorithm, allow_weak=True) + + +def test_the_default_algorithm_is_sha256() -> None: + assert HashEngine().algorithm == "sha256" + with pytest.raises(IntegrityException, match="at least 1"): + HashEngine(max_workers=0) + + +def test_a_missing_file_has_no_digest() -> None: + storage = _write(TREE, {"a.txt": b"alpha"}) + storage.mkdir("folder") + engine = HashEngine() + assert engine.hash_file(storage, "nope.txt") is None + assert engine.hash_file(storage, "folder") is None + assert list(engine.hash_many(storage, ["a.txt", "nope.txt", "folder"])) == ["a.txt"] + + +def test_files_are_hashed_in_parallel() -> None: + together = threading.Barrier(3, timeout=10) + + def work(path: str) -> str: + together.wait() + return path.upper() + + # Three calls can only pass the barrier when three threads are inside it at once. + assert HashEngine(max_workers=3).map(work, ["a", "b", "c"]) == {"a": "A", "b": "B", "c": "C"} + + +def test_one_worker_hashes_in_the_calling_thread() -> None: + threads: set[int] = set() + + def work(path: str) -> str: + threads.add(threading.get_ident()) + return path + + assert list(HashEngine(max_workers=1).map(work, ["a", "b", "a"])) == ["a", "b"] + assert threads == {threading.get_ident()} + + +def test_the_first_failure_is_raised() -> None: + def work(path: str) -> str: + if path == "bad": + raise IntegrityException("cannot read bad") + return path + + with pytest.raises(IntegrityException, match="cannot read bad"): + HashEngine(max_workers=2).map(work, ["a", "bad", "c", "d"]) diff --git a/tests/test_integrity_watch.py b/tests/test_integrity_watch.py new file mode 100644 index 0000000..ab718b1 --- /dev/null +++ b/tests/test_integrity_watch.py @@ -0,0 +1,364 @@ +"""Watch mode: the threads, the path collector, partial verification and a real watch. + +Everything but the last case is independent of the operating system delivering +filesystem events: the collector is given watchdog event objects, and partial +verification is called with the paths a watcher would report. The last case +writes to a watched directory for real and is skipped where no event arrives. +""" + +from __future__ import annotations + +import threading +from collections.abc import Iterator +from pathlib import Path + +import pytest +from watchdog.events import ( + DirDeletedEvent, + DirModifiedEvent, + FileClosedEvent, + FileCreatedEvent, + FileDeletedEvent, + FileModifiedEvent, + FileMovedEvent, + FileOpenedEvent, +) + +from automation_file.events import Event, EventBus, IntegrityViolation +from automation_file.exceptions import StorageURIException +from automation_file.integrity import IntegrityException, IntegrityMonitor, WatchHandle +from automation_file.integrity.local_watcher import LocalWatcher, PathCollector +from automation_file.integrity.watcher import Debouncer, IntervalRunner, PollingWatcher + +WAIT = 10.0 +FILES = {"a.txt": b"alpha", "b.txt": b"bravo", "sub/c.txt": b"charlie", "sub/d.txt": b"delta"} + + +class _Inbox: + """Collects what a thread delivers and lets a test wait for the next item.""" + + def __init__(self) -> None: + self.items: list = [] + self._arrived = threading.Semaphore(0) + + def put(self, item: object) -> None: + self.items.append(item) + self._arrived.release() + + def wait(self, timeout: float = WAIT) -> bool: + return self._arrived.acquire(timeout=timeout) + + +@pytest.fixture +def root(tmp_path: Path) -> Path: + directory = tmp_path / "tree" + (directory / "sub").mkdir(parents=True) + for path, data in FILES.items(): + (directory / path).write_bytes(data) + return directory + + +@pytest.fixture +def bus() -> EventBus: + return EventBus() + + +@pytest.fixture +def events(bus: EventBus) -> _Inbox: + inbox = _Inbox() + bus.subscribe(inbox.put, types=IntegrityViolation) + return inbox + + +@pytest.fixture +def monitor(root: Path, tmp_path: Path, bus: EventBus) -> IntegrityMonitor: + watching = IntegrityMonitor(root, baseline=tmp_path / "baseline.json", bus=bus) + watching.create_baseline() + return watching + + +@pytest.fixture +def watch(monitor: IntegrityMonitor) -> Iterator[WatchHandle]: + handle = monitor.watch(debounce=0.01) + yield handle + handle.stop() + + +# ---------------------------------------------------------------------- the threads + + +def test_the_debouncer_delivers_what_arrives_together_as_one_batch() -> None: + inbox = _Inbox() + debouncer = Debouncer(inbox.put, 0.05) + assert debouncer.is_running is False + debouncer.start() + try: + debouncer.start() + for path in ("a.txt", "b.txt", "a.txt"): + debouncer.add(path) + assert inbox.wait() + assert inbox.items == [frozenset({"a.txt", "b.txt"})] + debouncer.add("c.txt") + assert inbox.wait() + assert inbox.items[1] == frozenset({"c.txt"}) + finally: + debouncer.stop() + assert debouncer.is_running is False + + +def test_a_busy_tree_cannot_hold_a_batch_back_forever() -> None: + inbox = _Inbox() + debouncer = Debouncer(inbox.put, 0.02) + stop = threading.Event() + + def _keep_changing() -> None: + while not stop.wait(0.001): + debouncer.add("busy.log") + + writer = threading.Thread(target=_keep_changing, daemon=True) + debouncer.start() + writer.start() + try: + assert inbox.wait(), "a batch was never delivered while paths kept arriving" + finally: + stop.set() + writer.join(WAIT) + debouncer.stop() + assert inbox.items[0] == frozenset({"busy.log"}) + + +def test_stopping_the_debouncer_drops_what_was_not_delivered() -> None: + inbox = _Inbox() + debouncer = Debouncer(inbox.put, 30.0) + debouncer.start() + debouncer.add("a.txt") + debouncer.stop() + assert debouncer.is_running is False + assert inbox.items == [] + with pytest.raises(IntegrityException, match="must not be negative"): + Debouncer(inbox.put, -1.0) + + +def test_the_interval_runner_ticks_until_stopped_and_can_start_again() -> None: + ticks = _Inbox() + runner = IntervalRunner(lambda: ticks.put("tick"), 0.01, name="test-runner") + assert (runner.interval, runner.is_running) == (0.01, False) + runner.start() + runner.start() + assert ticks.wait() and ticks.wait() + runner.stop() + assert runner.is_running is False + count = len(ticks.items) + runner.start() + assert ticks.wait() + runner.stop() + assert len(ticks.items) > count + with pytest.raises(IntegrityException, match="interval must be positive"): + IntervalRunner(lambda: None, 0, name="never") + + +def test_the_polling_watcher_is_a_watch_handle() -> None: + ticks = _Inbox() + with PollingWatcher(lambda: ticks.put("tick"), 0.01) as handle: + assert isinstance(handle, WatchHandle) + handle.start() + assert (handle.kind, handle.is_running) == ("poll", True) + assert ticks.wait() + assert handle.is_running is False + + +# ---------------------------------------------------------------------- the path collector + + +def test_the_collector_reports_paths_relative_to_the_root(root: Path) -> None: + seen: list[str] = [] + collector = PathCollector(root, seen.append) + collector.dispatch(FileModifiedEvent(str(root / "a.txt"))) + collector.dispatch(FileCreatedEvent(str(root / "sub" / "new.txt"))) + collector.dispatch(FileDeletedEvent(str(root / "sub" / "c.txt"))) + collector.dispatch(DirDeletedEvent(str(root / "sub"))) + collector.dispatch(FileMovedEvent(str(root / "b.txt"), str(root / "sub" / "moved.txt"))) + assert seen == ["a.txt", "sub/new.txt", "sub/c.txt", "sub", "b.txt", "sub/moved.txt"] + + +def test_the_collector_leaves_out_what_is_not_a_change(root: Path) -> None: + seen: list[str] = [] + collector = PathCollector(root, seen.append) + collector.dispatch(DirModifiedEvent(str(root / "sub"))) + collector.dispatch(FileOpenedEvent(str(root / "a.txt"))) + collector.dispatch(FileClosedEvent(str(root / "a.txt"))) + collector.dispatch(FileModifiedEvent(str(root.parent / "elsewhere.txt"))) + assert seen == [] + collector.dispatch(FileMovedEvent(str(root / "a.txt"), str(root.parent / "moved-out.txt"))) + collector.dispatch(FileModifiedEvent(bytes(root / "b.txt"))) + collector.dispatch(DirDeletedEvent(str(root))) + assert seen == ["a.txt", "b.txt", ""] + + +def test_the_local_watcher_batches_the_paths_it_is_fed_and_skips_the_ignored(root: Path) -> None: + batches = _Inbox() + watcher = LocalWatcher( + root, batches.put, debounce=0.02, ignore=lambda path: path.endswith(".tmp") + ) + assert (watcher.kind, watcher.is_running) == ("events", False) + with watcher: + watcher.start() + watcher.start() + assert watcher.is_running is True + watcher.feed("scratch.tmp") + watcher.feed("a.txt") + watcher.feed("sub/c.txt") + delivered: set[str] = set() + while not {"a.txt", "sub/c.txt"} <= delivered: + assert batches.wait(), "the fed paths were not delivered" + delivered = set().union(*batches.items) + assert "scratch.tmp" not in delivered + assert watcher.is_running is False + watcher.stop() + with pytest.raises(IntegrityException, match="is not a directory"): + LocalWatcher(root / "a.txt", batches.put) + + +# ---------------------------------------------------------------------- verifying only what changed + + +def test_only_the_paths_that_changed_are_verified( + monitor: IntegrityMonitor, root: Path, events: _Inbox +) -> None: + (root / "a.txt").write_bytes(b"tampered") + (root / "b.txt").write_bytes(b"also tampered, but nothing reported it") + report = monitor.verify_paths(["a.txt"]) + assert [(change.kind.value, change.path) for change in report.changes] == [ + ("modified", "a.txt") + ] + assert (report.partial, report.deep, report.checked, report.hashed) == (True, True, 1, 1) + assert report.notes == ( + "partial pass: only the paths that changed were examined (1 current and 1 baseline " + "files), not the whole tree", + ) + assert monitor.last_report is report + (event,) = events.items + assert isinstance(event, Event) + assert (event.payload["partial"], event.payload["total"]) == (True, 1) + assert event.correlation_id == report.correlation_id + assert monitor.last_summary == { + "matched": [], + "missing": [], + "modified": ["a.txt"], + "extra": [], + "ok": False, + } + + +def test_a_path_that_did_not_really_change_raises_no_alert( + monitor: IntegrityMonitor, root: Path, events: _Inbox +) -> None: + report = monitor.verify_paths(["b.txt", "never-existed.tmp", "sub"]) + assert (report.ok, report.partial, report.checked) == (True, True, 3) + assert monitor.verify_paths([]).ok is True + assert events.items == [] + assert monitor.last_summary is not None + assert monitor.last_summary["matched"] == [] + with pytest.raises(StorageURIException): + monitor.verify_paths(["../outside.txt"]) + + +def test_a_directory_stands_for_everything_below_it(monitor: IntegrityMonitor, root: Path) -> None: + (root / "sub" / "c.txt").unlink() + (root / "sub" / "d.txt").unlink() + (root / "sub").rmdir() + gone = monitor.verify_paths(["sub"]) + assert gone.paths("deleted") == ["sub/c.txt", "sub/d.txt"] + assert (gone.checked, gone.hashed) == (0, 0) + + (root / "fresh").mkdir() + (root / "fresh" / "one.txt").write_bytes(b"one") + (root / "fresh" / "two.txt").write_bytes(b"two") + assert monitor.verify_paths(["fresh/"]).paths("created") == ["fresh/one.txt", "fresh/two.txt"] + everything = monitor.verify_paths([""]) + assert everything.counts["deleted"] == everything.counts["created"] == 2 + assert everything.checked == 4 + + +def test_a_move_reported_with_both_paths_is_a_rename(monitor: IntegrityMonitor, root: Path) -> None: + (root / "a.txt").rename(root / "sub" / "moved.txt") + (change,) = monitor.verify_paths(["a.txt", "sub/moved.txt"]).changes + assert (change.kind.value, change.path, change.previous_path) == ( + "renamed", + "sub/moved.txt", + "a.txt", + ) + assert monitor.verify_paths(["a.txt"]).paths("deleted") == ["a.txt"] + + +def test_accepting_a_partial_report_stores_the_whole_tree( + monitor: IntegrityMonitor, root: Path +) -> None: + (root / "a.txt").write_bytes(b"approved change") + (root / "b.txt").unlink() + report = monitor.verify_paths(["a.txt", "b.txt"]) + assert report.partial is True + assert monitor.accept(report).paths == ("a.txt", "sub/c.txt", "sub/d.txt") + assert monitor.verify().ok is True + + +def test_the_baseline_kept_inside_the_tree_is_not_verified_as_part_of_it( + root: Path, bus: EventBus, events: _Inbox +) -> None: + watching = IntegrityMonitor(root, baseline=root / ".baseline.json", bus=bus) + watching.create_baseline() + (root / ".baseline.json").write_bytes((root / ".baseline.json").read_bytes() + b" ") + assert watching.verify_paths([".baseline.json"]).ok is True + assert watching.verify_paths([""]).ok is True + assert events.items == [] + + +# ---------------------------------------------------------------------- watching a local directory + + +def test_watching_needs_a_baseline_and_a_directory(root: Path, tmp_path: Path) -> None: + with pytest.raises(IntegrityException, match="has no baseline"): + IntegrityMonitor(root).watch() + with pytest.raises(IntegrityException, match="is not a directory"): + IntegrityMonitor(tmp_path / "nowhere", baseline=tmp_path / "b.json").watch() + + +def test_a_local_target_is_watched_through_events(watch: WatchHandle) -> None: + assert isinstance(watch, LocalWatcher) + assert (watch.kind, watch.is_running) == ("events", True) + watch.stop() + assert watch.is_running is False + + +def test_a_failing_pass_does_not_end_the_watch( + monitor: IntegrityMonitor, watch: WatchHandle, root: Path, tmp_path: Path, events: _Inbox +) -> None: + assert isinstance(watch, LocalWatcher) + baseline = (tmp_path / "baseline.json").read_bytes() + (tmp_path / "baseline.json").unlink() + watch.feed("a.txt") + assert events.wait(), "the pass that could not run was not reported" + assert events.items[0].payload["status"] == "error" + assert monitor.last_error is not None + (tmp_path / "restored.json").write_bytes(baseline) + (tmp_path / "restored.json").replace(tmp_path / "baseline.json") + (root / "a.txt").write_bytes(b"tampered") + watch.feed("a.txt") + assert events.wait(), "the watch did not keep running after a failed pass" + assert events.items[1].payload["status"] == "drift" + assert monitor.last_error is None + + +def test_a_real_change_on_disk_reaches_the_monitor( + monitor: IntegrityMonitor, watch: WatchHandle, root: Path, events: _Inbox +) -> None: + for attempt in range(40): + (root / "a.txt").write_bytes(f"tampered {attempt}".encode()) + if events.wait(0.25): + break + else: + pytest.skip("no filesystem event arrived from watchdog in this environment") + report = monitor.last_report + assert report is not None + assert report.partial is True + assert report.paths("modified") == ["a.txt"] From 80f0e0d910bb06c6d5502c4bfe6db3a36b5d68ec Mon Sep 17 00:00:00 2001 From: JeffreyChen Date: Thu, 8 Oct 2026 13:47:15 +0800 Subject: [PATCH 34/59] feat: add the notification router and audit schema v2 Routes deliver events to named sinks with deduplication and a rate limit per route, and the audit trail records one row per event and per storage operation in a searchable store. Both are opt-in. While the router is active, notify_on_failure and the integrity monitor leave the direct notification to it. --- CLAUDE.md | 6 + README.md | 73 +- README.zh-CN.md | 68 +- README.zh-TW.md | 68 +- architecture.md | 12 +- automation_file/__init__.py | 46 +- automation_file/audit/__init__.py | 70 ++ automation_file/audit/actions.py | 61 ++ automation_file/audit/record.py | 235 +++++ automation_file/audit/sqlite_store.py | 321 +++++++ automation_file/audit/store.py | 270 ++++++ automation_file/audit/trail.py | 215 +++++ automation_file/core/action_registry.py | 7 + automation_file/core/config.py | 116 ++- automation_file/core/fim.py | 4 +- automation_file/core/metrics.py | 170 +++- automation_file/integrity/legacy.py | 28 +- automation_file/integrity/monitor.py | 9 +- automation_file/notify/__init__.py | 22 + automation_file/notify/manager.py | 145 +++- automation_file/notify/router.py | 529 ++++++++++++ docs/source/API/api_index.rst | 14 + docs/source/API/audit.rst | 19 + docs/source/API/notify.rst | 3 + docs/source/Eng/eng_index.rst | 16 + docs/source/Eng/usage/audit.rst | 309 +++++++ docs/source/Eng/usage/config.rst | 8 +- docs/source/Eng/usage/integrity.rst | 8 +- docs/source/Eng/usage/notifications.rst | 188 ++++ docs/source/Zh-CN/usage/audit.rst | 287 +++++++ docs/source/Zh-CN/usage/config.rst | 8 +- docs/source/Zh-CN/usage/integrity.rst | 5 +- docs/source/Zh-CN/usage/notifications.rst | 179 ++++ docs/source/Zh-CN/zh_cn_index.rst | 15 + docs/source/Zh-TW/usage/audit.rst | 288 +++++++ docs/source/Zh-TW/usage/config.rst | 8 +- docs/source/Zh-TW/usage/integrity.rst | 5 +- docs/source/Zh-TW/usage/notifications.rst | 179 ++++ docs/source/Zh-TW/zh_tw_index.rst | 15 + docs/updates/2026-10.md | 16 + docs/updates/README.md | 3 +- progress.md | 2 +- tests/test_audit_v2.py | 993 ++++++++++++++++++++++ tests/test_config.py | 250 +++++- tests/test_integrity_legacy.py | 54 +- tests/test_notify.py | 98 +++ tests/test_notify_router.py | 803 +++++++++++++++++ tests/test_operational_metrics.py | 230 +++++ 48 files changed, 6398 insertions(+), 80 deletions(-) create mode 100644 automation_file/audit/__init__.py create mode 100644 automation_file/audit/actions.py create mode 100644 automation_file/audit/record.py create mode 100644 automation_file/audit/sqlite_store.py create mode 100644 automation_file/audit/store.py create mode 100644 automation_file/audit/trail.py create mode 100644 automation_file/notify/router.py create mode 100644 docs/source/API/audit.rst create mode 100644 docs/source/Eng/usage/audit.rst create mode 100644 docs/source/Zh-CN/usage/audit.rst create mode 100644 docs/source/Zh-TW/usage/audit.rst create mode 100644 tests/test_audit_v2.py create mode 100644 tests/test_notify_router.py create mode 100644 tests/test_operational_metrics.py diff --git a/CLAUDE.md b/CLAUDE.md index 0895a99..528717a 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -41,7 +41,11 @@ automation_file/ ├── server/ # tcp_server, http_server, mcp_server (MCP over stdio), web_ui, metrics_server, │ # action_acl (ActionACL), network_guards (ensure_loopback) ├── client/ # HTTPActionClient for the HTTP action server +├── audit/ # Audit schema v2: record (AuditRecord), store (AuditStore, AuditQuery, +│ # MemoryAuditStore), sqlite_store (SQLiteAuditStore), trail (AuditTrail, +│ # audit_trail, configure_audit), actions (FA_audit_*) ├── trigger/, scheduler/, notify/ # watchdog file triggers, cron scheduler, notification sinks; +│ # notify/router.py routes events to sinks (NotificationRouter); │ # each registers its own FA_* ops ├── project/ # ProjectBuilder, create_project_dir ├── ui/ # PySide6 GUI: launcher.launch_ui, main_window.MainWindow, worker.ActionWorker, @@ -78,6 +82,8 @@ automation_file/ - `File(uri)` / `Storage(uri)` — the universal storage layer's application API: one file, one directory, in any backend. Both resolve their backend on every call through `StorageResolver` (`Storage.mount`, `Storage.register_scheme`). - `StorageBackend` — the contract a storage backend implements. The public operations (`exists`, `stat`, `list_dir`, `mkdir`, `upload`, `download`, `delete`, `checksum`, `read_bytes`, `write_bytes`, `copy_from`, `move_from`) are template methods; a backend supplies only the `_`-prefixed primitives. Twelve are built in: `LocalStorage`, `MemoryStorage`, `S3Storage` and `AzureStorage` (both on `ObjectStorage`), `SFTPStorage` and `FTPStorage` (both on `SessionStorage`), `GoogleDriveStorage`, `OneDriveStorage`, `DropboxStorage`, and the mounted `WebDAVStorage`, `SMBStorage` and `FsspecStorage`. Each uses its backend's shared client singleton unless given one, and reports a missing SDK with the extra to install. - `IntegrityMonitor` — compares a tree at any storage URI with an approved baseline (`create_baseline`, `verify`, `accept`, `watch`, `start` / `stop`, `snapshot`) and returns a `DriftReport`; drift is published as one `IntegrityViolation` per pass. It only reads unless a `RemediationPolicy` is passed. Its options are keyword arguments (`MonitorKeywords`). The first monitor's call and `check_once()` summary are kept, including the notification through `manager` or the process-wide `notification_manager`. +- `NotificationRouter` / `Route` / `notification_router` — delivers events to named sinks by type, source and minimum severity, with deduplication and a rate limit per route and sink. Opt-in: nothing is routed until a route exists and the router is started (`FA_notify_route_add` and `AutomationConfig.apply_to(manager, router)` start it). While it is active, `notify_on_failure` and the integrity monitor leave the direct notification to it, so nothing is announced twice. +- `AuditTrail` / `audit_trail` / `configure_audit(path)` — audit schema v2: one `AuditRecord` per event and per storage operation in an `AuditStore` (`SQLiteAuditStore`, `MemoryAuditStore`), searched with `audit_search` / `FA_audit_search`. Records nothing until configured, and never raises into the code it audits. The v1 `AuditLog` is unchanged. - `Event` / `EventBus` / `event_bus` — every component reports through events (`PipelineFailed`, `TaskFailed`, `IntegrityViolation`, `StorageError`, ...) with a severity, a correlation ID and an actor; consumers subscribe on the bus by class, type name or prefix. New code that has something to report publishes an event; it does not call a notification sink or the audit log directly. - `StorageURI` / `parse_storage_uri` — `:///`; `FileInfo`, `Checksum`, `StorageCapabilities` are the frozen value types the layer returns. diff --git a/README.md b/README.md index 8c3a8cc..1241791 100644 --- a/README.md +++ b/README.md @@ -50,6 +50,8 @@ facade. - **MCP (Model Context Protocol) server** — `MCPServer` bridges the registry to any MCP host (Claude Desktop, MCP CLIs) over newline-delimited JSON-RPC 2.0 on stdio; every `FA_*` action becomes an MCP tool with an auto-generated input schema - **Universal storage layer** — `File` / `Storage` address local and remote storage with one URI syntax (`local:///…`, `s3://…`, `azure://…`, `gdrive://…`, `sftp://…`, …), one `StorageBackend` contract and one error hierarchy; twelve backends are built in (local, in-memory, S3, Azure Blob, Google Drive, Dropbox, OneDrive, SFTP, FTP / FTPS, WebDAV, SMB, fsspec), and an 81-case contract suite checks any backend - **Event bus** — one `Event` model with ten core events (`pipeline.*`, `task.*`, `integrity.violation`, `storage.error`, `scheduler.error`, `system.error`), severities, correlation IDs and actors; subscribe on `event_bus` by class, type or prefix +- **Notification router** — routes decide which sinks hear about which events (by type, source and minimum severity), with deduplication and rate limiting per route; declare them in code, in `automation_file.toml` or with `FA_notify_route_*` +- **Audit trail** — `configure_audit(path)` records one row per event and per storage operation (actor, source, pipeline, task, action, resource, backend, status, duration, correlation ID), searchable with `audit_search` / `FA_audit_search` - PySide6 GUI (`python -m automation_file ui`) with a tab per backend, the JSON-action runner, and dedicated tabs for Triggers, Scheduler, and live Progress - Rich CLI with one-shot subcommands plus legacy JSON-batch flags - Project scaffolding (`ProjectBuilder`) for executor-based automations @@ -560,6 +562,68 @@ event_bus.recent(limit=20, correlation_id=run_id) - **Storage operations** — uploads, downloads, reads, deletes, copies and moves are reported to `automation_file.storage.observe` listeners, and a failing backend becomes a `StorageError` event. +### Notification router +Notifications are driven by events: a module publishes an event, and routes decide which +sinks hear about it. + +```python +from automation_file import Route, Severity, notification_router + +notification_router.add_route(Route( + "pipeline-failures", + sinks=("team-alerts",), # empty = every registered sink + types=("pipeline.*", "task.failed"), # event class, type name or prefix + min_severity=Severity.ERROR, + dedup_seconds=600, rate_limit=10, rate_period=60, +)) +notification_router.start() # subscribe on the event bus +``` + +- **Routes** — by event type, source and minimum severity, to named sinks. Declare them in + code, as `[[notify.routes]]` tables in `automation_file.toml` (hot-reloaded with the sinks), + or with `FA_notify_route_add` / `FA_notify_route_remove` / `FA_notify_route_list`. +- **Deduplication and rate limiting** — per route and sink: a repeat of the same type, source + and subject within `dedup_seconds` is dropped, and at most `rate_limit` messages go out per + `rate_period`. +- **Structured messages** — the subject and the body are built from the event: severity, + source, correlation ID, actor and the JSON of `event.to_dict()`. `critical` is sent at the + sinks' `error` level. +- **Failure isolation** — one failing sink never affects another. The failure is published as + a `system.error` event from the source `notify`, which the router never routes, so a broken + sink cannot feed a loop. +- **`notify_on_failure`** — always publishes an event. With the router active the routes + deliver it; otherwise the direct notification is sent as before, so nobody is notified twice + and nobody stops being notified. + +### Audit trail (schema v2) +The audit trail records who did what, when, against which resource, using which backend and +with what result: one record per event and per storage operation. + +```python +from automation_file import audit_search, configure_audit, correlation_scope + +configure_audit("audit.sqlite") # SQLite store; starts recording + +with correlation_scope() as run_id: + ... # events and storage operations are recorded +audit_search(correlation_id=run_id) # the whole run, newest first +audit_search(status="error", resource_prefix="s3://reports/", limit=20) +``` + +- **Record** — `id`, `timestamp` (UTC), `actor`, `source`, `pipeline`, `task`, `action`, + `resource`, `backend`, `status`, `duration_ms`, `error`, `metadata`, `correlation_id`. +- **Search** — by `since` / `until`, `actor`, `source`, `pipeline`, `task`, `action`, + `resource_prefix`, `backend`, `status`, `correlation_id` and free `text`; newest first, with + `limit` / `offset`. +- **Stores** — `SQLiteAuditStore` (parameterised SQL, a schema-version table, WAL) and + `MemoryAuditStore` for tests; `AuditStore` is the interface a PostgreSQL or remote store + implements. `SQLiteAuditStore.import_v1()` copies the rows of a v1 `AuditLog`. +- **Never in the way** — a record that cannot be written is logged and dropped, never raised + into the code being audited. A failed storage operation is recorded once, not twice. +- **Actions and metrics** — `FA_audit_configure` / `FA_audit_search` / `FA_audit_count` / + `FA_audit_purge`; `install_operational_metrics()` adds Prometheus counters for events, + notifications and storage operations. + ### File-watcher triggers Run an action list whenever a filesystem event fires on a watched path: @@ -779,10 +843,10 @@ password = "${file:smtp_password}" ``` ```python -from automation_file import AutomationConfig, notification_manager +from automation_file import AutomationConfig, notification_manager, notification_router config = AutomationConfig.load("automation_file.toml") -config.apply_to(notification_manager) +config.apply_to(notification_manager, notification_router) # sinks, and [[notify.routes]] ``` Unresolved `${…}` references raise `SecretNotFoundException` rather than @@ -871,8 +935,9 @@ handle = monitor.watch() # or react to changes as they happen; handle.st Code written for the first monitor keeps working: `IntegrityMonitor(root=..., manifest_path=..., interval=..., manager=..., on_drift=...)` reads a manifest written by `write_manifest`, `check_once()` returns the same summary, and the notification still goes through `manager` or, when none is passed, -the process-wide `notification_manager`. Pass `notify=False` when the published event is routed to -your sinks instead, so one drift is not announced twice. +the process-wide `notification_manager`. While the notification router is active its routes deliver the +`IntegrityViolation` event in place of that direct notification, so one drift is not announced twice; +`notify=False` turns the direct notification off altogether. ### AES-256-GCM file encryption Authenticated encryption with a self-describing envelope. Derive a key from diff --git a/README.zh-CN.md b/README.zh-CN.md index 0407abe..835304c 100644 --- a/README.zh-CN.md +++ b/README.zh-CN.md @@ -48,6 +48,8 @@ TCP / HTTP 服务器执行的 JSON 驱动动作。内附 PySide6 GUI，每个功 - **MCP（Model Context Protocol）服务器** — `MCPServer` 通过 stdio 上的 JSON-RPC 2.0（换行分隔 JSON）将注册表桥接到任意 MCP 主机（Claude Desktop、MCP CLI）；每个 `FA_*` 动作都会自动生成输入 schema 并成为 MCP 工具 - **通用存储层** — `File` / `Storage` 以同一套 URI 语法（`local:///…`、`s3://…`、`azure://…`、`gdrive://…`、`sftp://…`、…）、同一份 `StorageBackend` 契约与同一组异常层级访问本地与远端存储；内置十二种后端（本地、内存、S3、Azure Blob、Google Drive、Dropbox、OneDrive、SFTP、FTP / FTPS、WebDAV、SMB、fsspec），并附带 81 个用例的契约测试套件可检查任何后端 - **事件总线** — 单一 `Event` 模型与十种核心事件（`pipeline.*`、`task.*`、`integrity.violation`、`storage.error`、`scheduler.error`、`system.error`），具备严重程度、关联 ID 与 actor；可以在 `event_bus` 上按类、type 或前缀订阅 +- **通知路由器** — 以路由决定哪些事件（按类型、来源与最低严重程度）发送到哪些 sink，每条路由各自去重与限流；可在代码、`automation_file.toml` 或通过 `FA_notify_route_*` 声明 +- **审计轨迹** — `configure_audit(path)` 为每个事件与每次存储操作记录一条（actor、来源、pipeline、task、动作、资源、后端、状态、耗时、关联 ID），可用 `audit_search` / `FA_audit_search` 查询 - PySide6 GUI（`python -m automation_file ui`）每个后端一个页签，含 JSON 动作执行器，另有 Triggers、Scheduler、实时 Progress 专属页签 - 功能丰富的 CLI，包含一次性子命令与旧式 JSON 批量标志 - 项目脚手架（`ProjectBuilder`）协助构建以 executor 为核心的自动化项目 @@ -552,6 +554,63 @@ event_bus.recent(limit=20, correlation_id=run_id) - **存储操作** — 上传、下载、读取、删除、复制与移动都会报告给 `automation_file.storage.observe` 的监听者，后端失败时会产生 `StorageError` 事件。 +### 通知路由器 +通知改由事件驱动：模块发布事件，再由路由决定哪些 sink 会收到。 + +```python +from automation_file import Route, Severity, notification_router + +notification_router.add_route(Route( + "pipeline-failures", + sinks=("team-alerts",), # 留空 = 所有已注册的 sink + types=("pipeline.*", "task.failed"), # 事件类、type 名称或前缀 + min_severity=Severity.ERROR, + dedup_seconds=600, rate_limit=10, rate_period=60, +)) +notification_router.start() # 在事件总线上订阅 +``` + +- **路由** — 按事件 type、来源与最低严重程度，送往指定名称的 sink。可以在代码中声明、 + 在 `automation_file.toml` 以 `[[notify.routes]]` 表声明（与 sink 一起热重载），或使用 + `FA_notify_route_add` / `FA_notify_route_remove` / `FA_notify_route_list`。 +- **去重与速率限制** — 以每条路由、每个 sink 为单位：type、来源与主题都相同的事件在 + `dedup_seconds` 内重复出现时会被丢弃，每个 `rate_period` 内最多送出 `rate_limit` 条消息。 +- **结构化消息** — 主题与正文由事件组成：严重程度、来源、关联 ID、actor 以及 + `event.to_dict()` 的 JSON。`critical` 会以 sink 的 `error` 级别发送。 +- **失败隔离** — 单个 sink 失败绝对不会影响其他 sink。失败会以来源为 `notify` 的 + `system.error` 事件发布，而路由器绝对不会路由这类事件，因此故障的 sink 不会形成循环。 +- **`notify_on_failure`** — 总是会发布事件。路由器工作时由路由投递；否则照旧直接发送通知， + 因此不会有人收到两次，也不会有人收不到。 + +### 审计轨迹（schema v2） +审计轨迹记录谁在什么时候做了什么、对象是哪个资源、使用哪个后端以及结果如何：每个事件与 +每次存储操作各一条记录。 + +```python +from automation_file import audit_search, configure_audit, correlation_scope + +configure_audit("audit.sqlite") # SQLite 存储库；开始记录 + +with correlation_scope() as run_id: + ... # 事件与存储操作都会被记录 +audit_search(correlation_id=run_id) # 整次运行，最新的在前 +audit_search(status="error", resource_prefix="s3://reports/", limit=20) +``` + +- **记录** — `id`、`timestamp`（UTC）、`actor`、`source`、`pipeline`、`task`、`action`、 + `resource`、`backend`、`status`、`duration_ms`、`error`、`metadata`、`correlation_id`。 +- **搜索** — 可以按 `since` / `until`、`actor`、`source`、`pipeline`、`task`、`action`、 + `resource_prefix`、`backend`、`status`、`correlation_id` 与自由文本 `text` 筛选；最新的 + 在前，并支持 `limit` / `offset`。 +- **存储库** — `SQLiteAuditStore`（参数化 SQL、模式版本表、WAL）与测试用的 + `MemoryAuditStore`；`AuditStore` 是 PostgreSQL 或远程存储库要实现的接口。 + `SQLiteAuditStore.import_v1()` 可以复制 v1 `AuditLog` 的行。 +- **绝不碍事** — 无法写入的记录只会被记录到日志并丢弃，绝对不会抛进被审计的代码。失败的 + 存储操作只记录一次，不会重复。 +- **动作与指标** — `FA_audit_configure` / `FA_audit_search` / `FA_audit_count` / + `FA_audit_purge`；`install_operational_metrics()` 会加入事件、通知与存储操作的 + Prometheus 计数器。 + ### 文件监听触发 每当被监听路径发生文件系统事件，就执行动作清单： @@ -763,10 +822,10 @@ password = "${file:smtp_password}" ``` ```python -from automation_file import AutomationConfig, notification_manager +from automation_file import AutomationConfig, notification_manager, notification_router config = AutomationConfig.load("automation_file.toml") -config.apply_to(notification_manager) +config.apply_to(notification_manager, notification_router) # sinks, and [[notify.routes]] ``` 未解析的 `${…}` 引用会抛出 `SecretNotFoundException`，而不是默默变成空 @@ -852,8 +911,9 @@ handle = monitor.watch() # 或在变更发生时即时响应；handle.st 为第一代监控器写的代码照常工作：`IntegrityMonitor(root=..., manifest_path=..., interval=..., manager=..., on_drift=...)` 会读取 `write_manifest` 写出的 manifest，`check_once()` 返回同样的摘要， -通知也仍然通过 `manager` 发送，没有传入时则使用整个进程共用的 `notification_manager`。如果改由 -发布的事件把偏移送到通知渠道，请传入 `notify=False`，同一次偏移才不会被通知两次。 +通知也仍然通过 `manager` 发送，没有传入时则使用整个进程共用的 `notification_manager`。通知路由器 +启用期间改由路由送达 `IntegrityViolation` 事件，不再另外直接通知，同一次偏移不会被通知两次； +`notify=False` 会完全关闭这项直接通知。 ### AES-256-GCM 文件加密 带认证的加密与自描述封包格式。可由密码派生密钥或直接生成密钥: diff --git a/README.zh-TW.md b/README.zh-TW.md index 7115f02..da8fc06 100644 --- a/README.zh-TW.md +++ b/README.zh-TW.md @@ -48,6 +48,8 @@ TCP / HTTP 伺服器執行的 JSON 驅動動作。內附 PySide6 GUI，每個功 - **MCP（Model Context Protocol）伺服器** — `MCPServer` 透過 stdio 上的 JSON-RPC 2.0（行分隔 JSON）將登錄表橋接到任何 MCP 主機（Claude Desktop、MCP CLI）；每個 `FA_*` 動作都會自動生成輸入 schema 並成為 MCP 工具 - **通用儲存層** — `File` / `Storage` 以同一套 URI 語法（`local:///…`、`s3://…`、`azure://…`、`gdrive://…`、`sftp://…`、…）、同一份 `StorageBackend` 契約與同一組例外階層存取本機與遠端儲存；內建十二種後端（本機、記憶體、S3、Azure Blob、Google Drive、Dropbox、OneDrive、SFTP、FTP / FTPS、WebDAV、SMB、fsspec），並附 81 個案例的契約測試套件可檢查任何後端 - **事件匯流排** — 單一 `Event` 模型與十種核心事件（`pipeline.*`、`task.*`、`integrity.violation`、`storage.error`、`scheduler.error`、`system.error`），具備嚴重程度、關聯 ID 與 actor；可在 `event_bus` 上依類別、type 或前綴訂閱 +- **通知路由器** — 以路由決定哪些事件（依類型、來源與最低嚴重程度）送到哪些 sink，每條路由各自去重與限流；可在程式、`automation_file.toml` 或以 `FA_notify_route_*` 宣告 +- **稽核軌跡** — `configure_audit(path)` 為每個事件與每次儲存操作記錄一筆（actor、來源、pipeline、task、動作、資源、後端、狀態、耗時、關聯 ID），可用 `audit_search` / `FA_audit_search` 查詢 - PySide6 GUI（`python -m automation_file ui`）每個後端一個分頁，含 JSON 動作執行器，另有 Triggers、Scheduler、即時 Progress 專屬分頁 - 功能豐富的 CLI，包含一次性子指令與舊式 JSON 批次旗標 - 專案鷹架（`ProjectBuilder`）協助建立以 executor 為核心的自動化專案 @@ -552,6 +554,63 @@ event_bus.recent(limit=20, correlation_id=run_id) - **儲存操作** — 上傳、下載、讀取、刪除、複製與搬移都會回報給 `automation_file.storage.observe` 的監聽者，後端失敗時會產生 `StorageError` 事件。 +### 通知路由器 +通知改由事件驅動：模組發布事件，再由路由決定哪些 sink 會收到。 + +```python +from automation_file import Route, Severity, notification_router + +notification_router.add_route(Route( + "pipeline-failures", + sinks=("team-alerts",), # 留空 = 所有已註冊的 sink + types=("pipeline.*", "task.failed"), # 事件類別、type 名稱或前綴 + min_severity=Severity.ERROR, + dedup_seconds=600, rate_limit=10, rate_period=60, +)) +notification_router.start() # 在事件匯流排上訂閱 +``` + +- **路由** — 依事件 type、來源與最低嚴重程度，送往指定名稱的 sink。可以在程式中宣告、 + 在 `automation_file.toml` 以 `[[notify.routes]]` 表格宣告（與 sink 一起熱重載），或使用 + `FA_notify_route_add` / `FA_notify_route_remove` / `FA_notify_route_list`。 +- **去重與速率限制** — 以每條路由、每個 sink 為單位：type、來源與主旨都相同的事件在 + `dedup_seconds` 內重複出現時會被丟棄，每個 `rate_period` 內最多送出 `rate_limit` 則訊息。 +- **結構化訊息** — 主旨與內文由事件組成：嚴重程度、來源、關聯 ID、actor 以及 + `event.to_dict()` 的 JSON。`critical` 會以 sink 的 `error` 等級發送。 +- **失敗隔離** — 單一 sink 失敗絕對不會影響其他 sink。失敗會以來源為 `notify` 的 + `system.error` 事件發布，而路由器絕對不會路由這類事件，因此故障的 sink 不會形成迴圈。 +- **`notify_on_failure`** — 一律會發布事件。路由器運作時由路由投遞；否則照舊直接發送通知， + 因此不會有人收到兩次，也不會有人收不到。 + +### 稽核軌跡（schema v2） +稽核軌跡記錄誰在什麼時候做了什麼、對象是哪個資源、使用哪個後端以及結果如何：每個事件與 +每次儲存操作各一筆紀錄。 + +```python +from automation_file import audit_search, configure_audit, correlation_scope + +configure_audit("audit.sqlite") # SQLite 儲存庫；開始記錄 + +with correlation_scope() as run_id: + ... # 事件與儲存操作都會被記錄 +audit_search(correlation_id=run_id) # 整次執行，最新的在前 +audit_search(status="error", resource_prefix="s3://reports/", limit=20) +``` + +- **紀錄** — `id`、`timestamp`（UTC）、`actor`、`source`、`pipeline`、`task`、`action`、 + `resource`、`backend`、`status`、`duration_ms`、`error`、`metadata`、`correlation_id`。 +- **搜尋** — 可依 `since` / `until`、`actor`、`source`、`pipeline`、`task`、`action`、 + `resource_prefix`、`backend`、`status`、`correlation_id` 與自由文字 `text` 篩選；最新的 + 在前，並支援 `limit` / `offset`。 +- **儲存庫** — `SQLiteAuditStore`（參數化 SQL、結構描述版本資料表、WAL）與測試用的 + `MemoryAuditStore`；`AuditStore` 是 PostgreSQL 或遠端儲存庫要實作的介面。 + `SQLiteAuditStore.import_v1()` 可複製 v1 `AuditLog` 的資料列。 +- **絕不礙事** — 無法寫入的紀錄只會被記錄到日誌並捨棄，絕對不會拋進被稽核的程式。失敗的 + 儲存操作只記錄一次，不會重複。 +- **動作與指標** — `FA_audit_configure` / `FA_audit_search` / `FA_audit_count` / + `FA_audit_purge`；`install_operational_metrics()` 會加入事件、通知與儲存操作的 + Prometheus 計數器。 + ### 檔案監看觸發 每當被監看路徑發生檔案系統事件，就執行動作清單： @@ -763,10 +822,10 @@ password = "${file:smtp_password}" ``` ```python -from automation_file import AutomationConfig, notification_manager +from automation_file import AutomationConfig, notification_manager, notification_router config = AutomationConfig.load("automation_file.toml") -config.apply_to(notification_manager) +config.apply_to(notification_manager, notification_router) # sinks, and [[notify.routes]] ``` 未解析的 `${…}` 參考會拋出 `SecretNotFoundException`，而非默默變成空字串。 @@ -852,8 +911,9 @@ handle = monitor.watch() # 或在變更發生時即時反應；handle.st 為第一代監控器寫的程式照常運作：`IntegrityMonitor(root=..., manifest_path=..., interval=..., manager=..., on_drift=...)` 會讀取 `write_manifest` 寫出的 manifest，`check_once()` 回傳同樣的摘要， -通知也仍然透過 `manager` 送出，沒有傳入時則使用整個行程共用的 `notification_manager`。如果改由 -發布的事件把偏移送到通知管道，請傳入 `notify=False`，同一次偏移才不會被通知兩次。 +通知也仍然透過 `manager` 送出，沒有傳入時則使用整個行程共用的 `notification_manager`。通知路由器 +啟用期間改由路由送達 `IntegrityViolation` 事件，不再另外直接通知，同一次偏移不會被通知兩次； +`notify=False` 會完全關閉這項直接通知。 ### AES-256-GCM 檔案加密 具驗證的加密與自述式封包格式。可由密碼衍生金鑰或直接產生金鑰： diff --git a/architecture.md b/architecture.md index 132e298..2df035e 100644 --- a/architecture.md +++ b/architecture.md @@ -29,7 +29,8 @@ piece of the 1.0 roadmap (`docs/FILEAUTOMATION-1.0-ROADMAP.md`, PR #107); what i | `automation_file/integrity/` | IntegrityMonitor 2.0, on the storage layer and the event bus. `target.py` (`Target`: the monitored tree behind a storage URI), `hashing.py` (`HashEngine`; `md5` and `sha1` only with `allow_weak`), `snapshot.py` (`Snapshot`, `SnapshotEntry`, `build_snapshot`), `manifest.py` (schema version 2; the `write_manifest` format is read and converted), `baseline.py` (`BaselineManager`: an atomic write at any storage URI), `detector.py` (`Change`, `ChangeKind`, `detect_changes`: six kinds of change), `report.py` (`DriftReport`), `alerts.py` (`AlertEngine`, `AlertPolicy`: one `IntegrityViolation` per pass that finds drift), `remediation.py` (`RemediationPolicy`, `Remediator`: quarantine or restore, opt-in), `watcher.py` and `local_watcher.py` (polling, and watchdog events for a local target), `legacy.py` (the first monitor's summary, callback and notification), `monitor.py` (`IntegrityMonitor`), `actions.py` (`FA_integrity_*`). `core/fim.py` re-exports the class | | `automation_file/server/` | `tcp_server.py`, `http_server.py`, `mcp_server.py`, `web_ui.py`, `metrics_server.py`, `action_acl.py` (`ActionACL`), `network_guards.py` (`ensure_loopback`) | | `automation_file/client/` | `HTTPActionClient` for the HTTP action server | -| `automation_file/trigger/`, `scheduler/`, `notify/` | Watchdog file triggers, cron scheduler, notification sinks. Each registers its own `FA_*` ops | +| `automation_file/trigger/`, `scheduler/`, `notify/` | Watchdog file triggers, cron scheduler, notification sinks. Each registers its own `FA_*` ops. `notify/router.py` (`Route`, `NotificationRouter`, the process-wide `notification_router`) subscribes on the event bus and delivers events to named sinks by type, source and minimum severity, with deduplication and a rate limit per route and sink; a failing sink becomes a `system.error` event from the source `notify`, which is never routed | +| `automation_file/audit/` | Audit schema v2. `record.py` (`AuditRecord`, built from an event or from a storage operation), `store.py` (`AuditStore`, `AuditQuery`, `MemoryAuditStore`), `sqlite_store.py` (`SQLiteAuditStore`: parameterised SQL, a schema-version table, `import_v1`), `trail.py` (`AuditTrail`, the process-wide `audit_trail`, `configure_audit`), `actions.py` (`FA_audit_*`). The trail records nothing until it is configured; the v1 `core/audit.py` `AuditLog` is unchanged | | `automation_file/project/` | `ProjectBuilder`, `create_project_dir` | | `automation_file/ui/` | PySide6 GUI: `launcher.launch_ui`, `main_window.MainWindow`, `worker.ActionWorker`, `log_widget.LogPanel`, `tabs/` (backend panels are grouped under `TransferTab`) | | `automation_file/utils/` | File discovery, fast find, grep, duplicate finder, backup rotation | @@ -66,6 +67,13 @@ piece of the 1.0 roadmap (`docs/FILEAUTOMATION-1.0-ROADMAP.md`, PR #107); what i `FA_integrity_accept`, `FA_integrity_watch_start`, `FA_integrity_watch_stop`, `FA_integrity_status`. The first monitor's call, `IntegrityMonitor(root, manifest_path, interval=, on_drift=, manager=, alert_on_extra=)`, and its `check_once()` summary are kept. +- **Notification routes and audit** (same facade): `Route`, `NotificationRouter`, `notification_router`; + `AuditRecord`, `AuditQuery`, `AuditStore`, `SQLiteAuditStore`, `MemoryAuditStore`, `AuditTrail`, + `audit_trail`, `configure_audit`, `audit_search`, `register_audit_ops`; `install_operational_metrics` + and the counters `EVENT_COUNT`, `NOTIFICATION_COUNT`, `STORAGE_OPERATION_COUNT`, + `STORAGE_OPERATION_DURATION`. Actions: `FA_notify_route_add` / `_remove` / `_list`, + `FA_audit_configure` / `_search` / `_count` / `_purge`. Both are opt-in: the router delivers + nothing until it has a route and is started, and the trail records nothing until it has a store. - **Events** (same facade): `Event`, `Severity`, `EventBus`, `event_bus`, `emit`, `correlation_scope`, `actor_scope`, and the core events `PipelineStarted`, `PipelineCompleted`, `PipelineFailed`, `TaskStarted`, `TaskCompleted`, `TaskFailed`, `IntegrityViolation`, `StorageError`, `SchedulerError`, @@ -132,7 +140,7 @@ MCP host → automation_file_mcp (stdio JSON-RPC) → tools/call → MCPServer r ``` ActionExecutor() → build_default_registry(): local + http + utils + drive commands → _register_cloud_backends (register__ops) → trigger / scheduler / progress / notify ops - → storage ops (FA_storage_*) → integrity ops (FA_integrity_*) + → storage ops (FA_storage_*) → integrity ops (FA_integrity_*) → audit ops (FA_audit_*) → _load_plugins (entry points; may override built-ins) → executor adds FA_execute_action, FA_execute_files, FA_execute_action_parallel, FA_validate ``` diff --git a/automation_file/__init__.py b/automation_file/__init__.py index 8a1a00f..922a546 100644 --- a/automation_file/__init__.py +++ b/automation_file/__init__.py @@ -9,6 +9,18 @@ from typing import TYPE_CHECKING, Any +from automation_file.audit import ( + AuditQuery, + AuditRecord, + AuditStore, + AuditTrail, + MemoryAuditStore, + SQLiteAuditStore, + audit_search, + audit_trail, + configure_audit, + register_audit_ops, +) from automation_file.client import HTTPActionClient, HTTPActionClientException from automation_file.core.action_executor import ( ActionExecutor, @@ -43,7 +55,17 @@ from automation_file.core.file_lock import FileLock from automation_file.core.json_store import read_action_json, write_action_json from automation_file.core.manifest import ManifestException, verify_manifest, write_manifest -from automation_file.core.metrics import ACTION_COUNT, ACTION_DURATION, record_action +from automation_file.core.metrics import ( + ACTION_COUNT, + ACTION_DURATION, + EVENT_COUNT, + NOTIFICATION_COUNT, + STORAGE_OPERATION_COUNT, + STORAGE_OPERATION_DURATION, + install_operational_metrics, + record_action, + uninstall_operational_metrics, +) from automation_file.core.metrics import render as render_metrics from automation_file.core.package_loader import PackageLoader from automation_file.core.progress import ( @@ -199,13 +221,16 @@ EmailSink, NotificationException, NotificationManager, + NotificationRouter, NotificationSink, PagerDutySink, + Route, SlackSink, TeamsSink, TelegramSink, WebhookSink, notification_manager, + notification_router, notify_send, register_notify_ops, ) @@ -600,6 +625,16 @@ def __getattr__(name: str) -> Any: "verify_manifest", "AuditException", "AuditLog", + "AuditRecord", + "AuditQuery", + "AuditStore", + "SQLiteAuditStore", + "MemoryAuditStore", + "AuditTrail", + "audit_trail", + "configure_audit", + "audit_search", + "register_audit_ops", "IntegrityMonitor", "IntegrityException", "DriftReport", @@ -616,6 +651,12 @@ def __getattr__(name: str) -> Any: "ACTION_COUNT", "ACTION_DURATION", "record_action", + "EVENT_COUNT", + "NOTIFICATION_COUNT", + "STORAGE_OPERATION_COUNT", + "STORAGE_OPERATION_DURATION", + "install_operational_metrics", + "uninstall_operational_metrics", "render_metrics", "MetricsServer", "start_metrics_server", @@ -657,6 +698,9 @@ def __getattr__(name: str) -> Any: "EmailSink", "NotificationException", "NotificationManager", + "NotificationRouter", + "Route", + "notification_router", "NotificationSink", "PagerDutySink", "SlackSink", diff --git a/automation_file/audit/__init__.py b/automation_file/audit/__init__.py new file mode 100644 index 0000000..0309ccc --- /dev/null +++ b/automation_file/audit/__init__.py @@ -0,0 +1,70 @@ +"""Audit schema v2: who did what, when, against which resource, with what result. + +An :class:`AuditTrail` listens to the event bus and to the storage observers +and appends one :class:`AuditRecord` per event and per storage operation to an +:class:`AuditStore`. :class:`SQLiteAuditStore` is the store that ships; +:class:`MemoryAuditStore` serves tests; a PostgreSQL or remote store implements +the same five methods. + +.. code-block:: python + + from automation_file import audit_search, configure_audit + + configure_audit("audit.sqlite") + ... # events and storage operations are recorded + audit_search(status="error", resource_prefix="s3://reports/", limit=20) + +The v1 :class:`~automation_file.core.audit.AuditLog` stays as it is; +:meth:`SQLiteAuditStore.import_v1` copies its rows. +""" + +from __future__ import annotations + +from automation_file.audit.actions import ( + audit_configure, + audit_count, + audit_purge, + register_audit_ops, +) +from automation_file.audit.record import ( + AuditRecord, + record_from_event, + record_from_operation, +) +from automation_file.audit.sqlite_store import SCHEMA_VERSION, SQLiteAuditStore +from automation_file.audit.store import ( + DEFAULT_LIMIT, + MAX_LIMIT, + AuditQuery, + AuditStore, + MemoryAuditStore, +) +from automation_file.audit.trail import ( + AuditTrail, + audit_search, + audit_trail, + configure_audit, +) +from automation_file.core.audit import AuditException + +__all__ = [ + "DEFAULT_LIMIT", + "MAX_LIMIT", + "SCHEMA_VERSION", + "AuditException", + "AuditQuery", + "AuditRecord", + "AuditStore", + "AuditTrail", + "MemoryAuditStore", + "SQLiteAuditStore", + "audit_configure", + "audit_count", + "audit_purge", + "audit_search", + "audit_trail", + "configure_audit", + "record_from_event", + "record_from_operation", + "register_audit_ops", +] diff --git a/automation_file/audit/actions.py b/automation_file/audit/actions.py new file mode 100644 index 0000000..fde178e --- /dev/null +++ b/automation_file/audit/actions.py @@ -0,0 +1,61 @@ +"""``FA_audit_*`` actions: the audit trail for JSON action lists. + +.. code-block:: json + + [ + ["FA_audit_configure", {"db_path": "/var/lib/automation_file/audit.sqlite"}], + ["FA_audit_search", {"status": "error", "since": "2026-10-01T00:00:00+00:00", "limit": 50}], + ["FA_audit_count", {"correlation_id": "4f0c2b6e9d5a4c1f8a7b3e2d1c0f9a8b"}], + ["FA_audit_purge", {"older_than_seconds": 7776000}] + ] + +The search and count actions take the filters of +:class:`~automation_file.audit.store.AuditStore` by name and act on the +process-wide trail that ``FA_audit_configure`` set up. +""" + +from __future__ import annotations + +from typing import TYPE_CHECKING, Any + +from automation_file.audit.sqlite_store import SQLiteAuditStore +from automation_file.audit.trail import audit_search, audit_trail, configure_audit + +if TYPE_CHECKING: + from automation_file.core.action_registry import ActionRegistry + + +def audit_configure(db_path: str) -> dict[str, Any]: + """Keep the audit trail in the SQLite database ``db_path`` and start recording. + + Returns ``{"active": ..., "db_path": ..., "schema_version": ...}``. + """ + trail = configure_audit(db_path) + store = trail.store + described: dict[str, Any] = {"active": trail.active} + if isinstance(store, SQLiteAuditStore): + described["db_path"] = str(store.path) + described["schema_version"] = store.schema_version + return described + + +def audit_count(**filters: Any) -> int: + """Return how many audit records pass the filters (the same ones as ``FA_audit_search``).""" + return audit_trail.count(**filters) + + +def audit_purge(older_than_seconds: float) -> int: + """Delete the audit records older than ``older_than_seconds``; return how many.""" + return audit_trail.purge(older_than_seconds) + + +def register_audit_ops(registry: ActionRegistry) -> None: + """Wire the ``FA_audit_*`` actions into a registry.""" + registry.register_many( + { + "FA_audit_configure": audit_configure, + "FA_audit_search": audit_search, + "FA_audit_count": audit_count, + "FA_audit_purge": audit_purge, + } + ) diff --git a/automation_file/audit/record.py b/automation_file/audit/record.py new file mode 100644 index 0000000..48e8b72 --- /dev/null +++ b/automation_file/audit/record.py @@ -0,0 +1,235 @@ +"""The audit record: who did what, when, against which resource, with what result. + +One :class:`AuditRecord` answers the audit questions for one thing that +happened: ``actor`` did ``action`` at ``timestamp`` against ``resource`` using +``backend``, and it ended with ``status`` (and ``error``). ``pipeline``, +``task`` and ``correlation_id`` place it in a run; ``metadata`` keeps the +rest. + +Records are frozen and JSON-friendly. :func:`record_from_event` and +:func:`record_from_operation` build them from the two things the audit trail +listens to. +""" + +from __future__ import annotations + +import json +import uuid +from collections.abc import Mapping +from dataclasses import dataclass, field, fields +from datetime import datetime, timedelta, timezone +from typing import TYPE_CHECKING, Any + +from automation_file.core.audit import AuditException +from automation_file.events.context import current_actor, current_correlation_id +from automation_file.events.model import Severity + +if TYPE_CHECKING: + from automation_file.events.model import Event + from automation_file.storage.observe import StorageOperation + +STATUS_OK = "ok" +STATUS_WARNING = "warning" +STATUS_ERROR = "error" +#: The ``source`` of the records made from storage operations. +STORAGE_SOURCE = "storage" + +_EPOCH = datetime(1970, 1, 1, tzinfo=timezone.utc) +_MICROSECOND = timedelta(microseconds=1) +_ZULU = "Z" +_UTC_OFFSET = "+00:00" +#: Payload keys of an event that fill the record field of the same name, as text. +_PAYLOAD_TEXT = ("pipeline", "task", "resource", "backend", "error") +_REQUIRED_TEXT = ("id", "actor", "source", "action", "status") +_OPTIONAL_TEXT = (*_PAYLOAD_TEXT, "correlation_id") +_DURATION = "duration_ms" +_STATUS_BY_SEVERITY = { + Severity.INFO: STATUS_OK, + Severity.WARNING: STATUS_WARNING, + Severity.ERROR: STATUS_ERROR, + Severity.CRITICAL: STATUS_ERROR, +} + + +def _new_id() -> str: + return uuid.uuid4().hex + + +def _now() -> datetime: + return datetime.now(timezone.utc) + + +def to_microseconds(moment: datetime) -> int: + """Return an aware ``moment`` as whole microseconds since the Unix epoch.""" + return (moment - _EPOCH) // _MICROSECOND + + +def from_microseconds(value: int) -> datetime: + """Return the UTC time that is ``value`` microseconds after the Unix epoch.""" + return _EPOCH + timedelta(microseconds=value) + + +def parse_time(value: object, name: str = "timestamp") -> datetime: + """Return ``value`` as an aware UTC time. + + Accepts an aware ``datetime``, an ISO 8601 string with an offset (or a + trailing ``Z``), or a number of seconds since the Unix epoch. A time without + a time zone is rejected: it would be a guess. + """ + if isinstance(value, bool): + raise AuditException(f"{name} must be a time, got {value!r}") + if isinstance(value, (int, float)): + try: + return _EPOCH + timedelta(seconds=value) + except (OverflowError, ValueError) as err: + raise AuditException(f"{name} is out of range: {value!r}") from err + if isinstance(value, str): + text = value.strip() + if text.endswith(_ZULU): + text = text[: -len(_ZULU)] + _UTC_OFFSET + try: + value = datetime.fromisoformat(text) + except ValueError as err: + raise AuditException(f"{name} is not an ISO 8601 time: {value!r}") from err + if not isinstance(value, datetime): + raise AuditException(f"{name} must be a time, got {value!r}") + if value.utcoffset() is None: + raise AuditException(f"{name} needs a time zone, got {value.isoformat()!r}") + return value.astimezone(timezone.utc) + + +def _check_text(name: str, value: object, *, optional: bool) -> None: + if isinstance(value, str) or (optional and value is None): + return + wanted = "a string or None" if optional else "a string" + raise AuditException(f"{name} must be {wanted}, got {value!r}") + + +def _json_safe(metadata: Mapping[str, Any]) -> dict[str, Any]: + """Return ``metadata`` as plain JSON values; what JSON cannot hold becomes its ``repr``.""" + try: + return dict(json.loads(json.dumps(dict(metadata), default=repr))) + except (TypeError, ValueError): + return {str(key): repr(value) for key, value in dict(metadata).items()} + + +@dataclass(frozen=True, kw_only=True) +class AuditRecord: + """One audited thing that happened. Fields left out are filled from the current scopes.""" + + id: str = field(default_factory=_new_id) + timestamp: datetime = field(default_factory=_now) + actor: str = field(default_factory=current_actor) + source: str = "" + pipeline: str | None = None + task: str | None = None + action: str = "" + resource: str | None = None + backend: str | None = None + status: str = STATUS_OK + duration_ms: float | None = None + error: str | None = None + metadata: Mapping[str, Any] = field(default_factory=dict, hash=False) + correlation_id: str | None = field(default_factory=current_correlation_id) + + def __post_init__(self) -> None: + for name in _REQUIRED_TEXT: + _check_text(name, getattr(self, name), optional=False) + for name in _OPTIONAL_TEXT: + _check_text(name, getattr(self, name), optional=True) + if not self.id: + raise AuditException("an audit record needs a non-empty id") + if not isinstance(self.metadata, Mapping): + raise AuditException(f"metadata must be a mapping, got {self.metadata!r}") + object.__setattr__(self, "timestamp", parse_time(self.timestamp)) + object.__setattr__(self, "metadata", _json_safe(self.metadata)) + if self.duration_ms is not None: + object.__setattr__(self, _DURATION, _as_duration(self.duration_ms)) + + def to_dict(self) -> dict[str, Any]: + """Return a JSON-serialisable mapping of the record, its keys in field order.""" + document = {entry.name: getattr(self, entry.name) for entry in fields(self)} + document["timestamp"] = self.timestamp.isoformat() + document["metadata"] = dict(self.metadata) + return document + + @classmethod + def from_dict(cls, document: Mapping[str, Any]) -> AuditRecord: + """Build a record from what :meth:`to_dict` returned.""" + if not isinstance(document, Mapping): + raise AuditException(f"an audit record is a mapping, got {document!r}") + known = {entry.name for entry in fields(cls)} + unknown = sorted(set(document) - known) + if unknown: + raise AuditException(f"unknown audit record field(s) {unknown}") + try: + return cls(**dict(document)) + except TypeError as err: + raise AuditException(f"invalid audit record: {err}") from err + + +def _as_duration(value: object) -> float: + if isinstance(value, bool) or not isinstance(value, (int, float)): + raise AuditException(f"{_DURATION} must be a number, got {value!r}") + return float(value) + + +def _text(value: object) -> str | None: + return None if value is None else str(value) + + +def record_from_event(event: Event) -> AuditRecord: + """Return the audit record of ``event``. + + ``action`` is the event type. ``pipeline``, ``task``, ``resource``, + ``backend``, ``status``, ``duration_ms`` and ``error`` come from the payload + keys of the same name; the event's subject and severity and every other + payload key go under ``metadata``. Without a ``status`` in the payload the + severity decides: ``ok`` for info, ``warning``, and ``error`` for error and + critical. + """ + details = dict(event.payload) + status = details.pop("status", None) + duration = details.get(_DURATION) + if isinstance(duration, bool) or not isinstance(duration, (int, float)): + duration = None # not a number: it stays under the metadata as it is + else: + del details[_DURATION] + placed: dict[str, Any] = {key: _text(details.pop(key, None)) for key in _PAYLOAD_TEXT} + return AuditRecord( + id=event.id, + timestamp=event.timestamp, + actor=event.actor, + correlation_id=event.correlation_id, + source=event.source, + action=event.type, + status=str(status) if status else _STATUS_BY_SEVERITY[event.severity], + duration_ms=duration, + metadata={**details, "subject": event.subject, "severity": event.severity.value}, + **placed, + ) + + +def record_from_operation(operation: StorageOperation) -> AuditRecord: + """Return the audit record of one storage operation. + + ``source`` is ``"storage"``, ``action`` the operation (``upload``, + ``download``, ``read``, ``delete``, ``mkdir``, ``copy``, ``move``), + ``resource`` its URI. The actor and the correlation ID are those of the + scope the operation ran in. + """ + metadata: dict[str, Any] = {} + if operation.source_uri is not None: + metadata["source_uri"] = operation.source_uri + if operation.error_type is not None: + metadata["error_type"] = operation.error_type + return AuditRecord( + source=STORAGE_SOURCE, + action=operation.operation, + resource=operation.uri, + backend=operation.backend, + status=operation.status, + duration_ms=operation.duration_ms, + error=operation.error, + metadata=metadata, + ) diff --git a/automation_file/audit/sqlite_store.py b/automation_file/audit/sqlite_store.py new file mode 100644 index 0000000..d271031 --- /dev/null +++ b/automation_file/audit/sqlite_store.py @@ -0,0 +1,321 @@ +"""The SQLite audit store. + +:class:`SQLiteAuditStore` keeps the records of audit schema v2 in one table, +with a second table that states the schema version and indexes on the +timestamp, the correlation ID, the resource and the action. One connection is +shared by every thread behind a lock, in WAL mode so another process can read +while this one writes. + +Every value reaches SQLite as a bound parameter. The text of a statement is +only ever assembled from the constant fragments of this module, which depend +on *which* filters are set and never on what they contain, and the ``LIKE`` +wildcards in a ``resource_prefix`` or ``text`` filter are escaped. + +:meth:`SQLiteAuditStore.import_v1` copies the rows of a v1 +:class:`~automation_file.core.audit.AuditLog` database. +""" + +from __future__ import annotations + +import hashlib +import json +import os +import sqlite3 +import threading +from collections.abc import Iterator, Sequence +from contextlib import closing +from pathlib import Path +from typing import Any + +from automation_file.audit.record import ( + STATUS_ERROR, + STATUS_OK, + AuditRecord, + from_microseconds, + parse_time, + to_microseconds, +) +from automation_file.audit.store import ( + EXACT_FILTERS, + LIKE_ESCAPE, + TEXT_FIELDS, + AuditQuery, + AuditStore, + check_record, + escape_like, + metadata_json, + purge_cutoff, +) +from automation_file.core.audit import AuditException +from automation_file.logging_config import file_automation_logger + +SCHEMA_VERSION = 2 +#: The ``source`` of the records copied from a v1 audit log. +V1_SOURCE = "audit.v1" + +_TIMEOUT_SECONDS = 5.0 +_IMPORT_BATCH = 1000 +_V1_ACTOR = "unknown" +_ID_LENGTH = 32 +_SCHEMA = """ +CREATE TABLE IF NOT EXISTS audit_schema_version ( + version INTEGER NOT NULL +); +CREATE TABLE IF NOT EXISTS audit_records ( + seq INTEGER PRIMARY KEY AUTOINCREMENT, + id TEXT NOT NULL UNIQUE, + ts_us INTEGER NOT NULL, + actor TEXT NOT NULL, + source TEXT NOT NULL, + pipeline TEXT, + task TEXT, + action TEXT NOT NULL, + resource TEXT, + backend TEXT, + status TEXT NOT NULL, + duration_ms REAL, + error TEXT, + metadata TEXT NOT NULL, + correlation_id TEXT +); +CREATE INDEX IF NOT EXISTS idx_audit_records_ts ON audit_records (ts_us DESC, seq DESC); +CREATE INDEX IF NOT EXISTS idx_audit_records_correlation ON audit_records (correlation_id); +CREATE INDEX IF NOT EXISTS idx_audit_records_resource ON audit_records (resource); +CREATE INDEX IF NOT EXISTS idx_audit_records_action ON audit_records (action); +""" +_COLUMNS = ( + "id, ts_us, actor, source, pipeline, task, action, resource, backend, status, " + "duration_ms, error, metadata, correlation_id" +) +_PLACEHOLDERS = "?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?" +_INSERT = f"INSERT INTO audit_records ({_COLUMNS}) VALUES ({_PLACEHOLDERS})" +_INSERT_NEW = f"INSERT OR IGNORE INTO audit_records ({_COLUMNS}) VALUES ({_PLACEHOLDERS})" +_SELECT = f"SELECT {_COLUMNS} FROM audit_records" +_COUNT = "SELECT COUNT(*) FROM audit_records" +_NEWEST_FIRST = " ORDER BY ts_us DESC, seq DESC LIMIT ? OFFSET ?" +_PURGE = "DELETE FROM audit_records WHERE ts_us < ?" +_SINCE = "ts_us >= ?" +_UNTIL = "ts_us < ?" +_LIKE = f" LIKE ? ESCAPE '{LIKE_ESCAPE}'" +_EXACT = {name: f"{name} = ?" for name in EXACT_FILTERS} +_RESOURCE_PREFIX = f"resource{_LIKE}" +_TEXT_COLUMNS = (*TEXT_FIELDS, "metadata") +_TEXT = "(" + " OR ".join(f"{column}{_LIKE}" for column in _TEXT_COLUMNS) + ")" +_VERSION = "SELECT MAX(version) FROM audit_schema_version" +_V1_ROWS = "SELECT id, ts, action, payload, result, error, duration_ms FROM audit ORDER BY id" + + +def _where(query: AuditQuery) -> tuple[str, list[Any]]: + """Return the ``WHERE`` clause of ``query`` and the values bound to its placeholders.""" + clauses: list[str] = [] + values: list[Any] = [] + if query.since is not None: + clauses.append(_SINCE) + values.append(to_microseconds(query.since)) + if query.until is not None: + clauses.append(_UNTIL) + values.append(to_microseconds(query.until)) + for name in EXACT_FILTERS: + wanted = getattr(query, name) + if wanted is not None: + clauses.append(_EXACT[name]) + values.append(wanted) + if query.resource_prefix: + clauses.append(_RESOURCE_PREFIX) + values.append(escape_like(query.resource_prefix) + "%") + if query.text: + clauses.append(_TEXT) + values.extend([f"%{escape_like(query.text)}%"] * len(_TEXT_COLUMNS)) + return (" WHERE " + " AND ".join(clauses) if clauses else ""), values + + +def _row(record: AuditRecord) -> tuple[Any, ...]: + return ( + record.id, + to_microseconds(record.timestamp), + record.actor, + record.source, + record.pipeline, + record.task, + record.action, + record.resource, + record.backend, + record.status, + record.duration_ms, + record.error, + metadata_json(record), + record.correlation_id, + ) + + +def _record(row: Sequence[Any]) -> AuditRecord: + return AuditRecord( + id=row[0], + timestamp=from_microseconds(row[1]), + actor=row[2], + source=row[3], + pipeline=row[4], + task=row[5], + action=row[6], + resource=row[7], + backend=row[8], + status=row[9], + duration_ms=row[10], + error=row[11], + metadata=json.loads(row[12]), + correlation_id=row[13], + ) + + +def _decoded(text: str | None) -> Any: + """Return the value a v1 row kept as JSON text, or the text itself when it is not JSON.""" + if not text: + return None + try: + return json.loads(text) + except ValueError: + return text + + +def _v1_record(row: Sequence[Any]) -> AuditRecord: + row_id, moment, action, payload, result, error, duration_ms = row + # A digest of the row is its ID, so importing the same log twice adds nothing. + digest = hashlib.sha256(f"{row_id}|{moment!r}|{action}|{payload}".encode()).hexdigest() + return AuditRecord( + id=digest[:_ID_LENGTH], + timestamp=parse_time(float(moment)), + actor=_V1_ACTOR, + correlation_id=None, + source=V1_SOURCE, + action=str(action), + status=STATUS_ERROR if error else STATUS_OK, + duration_ms=float(duration_ms or 0.0), + error=error, + metadata={"v1_id": row_id, "payload": _decoded(payload), "result": _decoded(result)}, + ) + + +class SQLiteAuditStore(AuditStore): + """Audit records in a SQLite database at ``path``; created when missing.""" + + def __init__(self, path: str | os.PathLike[str]) -> None: + self._path = Path(path) + self._lock = threading.Lock() + self._closed = False + try: + self._path.parent.mkdir(parents=True, exist_ok=True) + self._conn = sqlite3.connect( + self._path, timeout=_TIMEOUT_SECONDS, check_same_thread=False + ) + except (OSError, sqlite3.Error) as err: + raise AuditException(f"cannot open audit store {self._path}: {err}") from err + try: + self._prepare() + except (AuditException, sqlite3.Error) as err: + self._conn.close() + raise AuditException(f"cannot open audit store {self._path}: {err}") from err + + @property + def path(self) -> Path: + return self._path + + @property + def schema_version(self) -> int: + """The schema version the database states.""" + return int(self._fetch(_VERSION, ())[0][0]) + + def append(self, record: AuditRecord) -> None: + check_record(record) + try: + with self._lock, self._conn: + self._conn.execute(_INSERT, _row(record)) + except sqlite3.IntegrityError as err: + raise AuditException(f"audit record {record.id} is already stored") from err + except sqlite3.Error as err: + raise AuditException(f"cannot write audit record {record.id}: {err}") from err + + def search(self, **filters: Any) -> list[AuditRecord]: + query = AuditQuery.from_filters(filters) + if query.limit == 0: + return [] + where, values = _where(query) + rows = self._fetch(_SELECT + where + _NEWEST_FIRST, [*values, query.limit, query.offset]) + return [_record(row) for row in rows] + + def count(self, **filters: Any) -> int: + where, values = _where(AuditQuery.from_filters(filters)) + return int(self._fetch(_COUNT + where, values)[0][0]) + + def purge(self, older_than_seconds: float) -> int: + cutoff = to_microseconds(purge_cutoff(older_than_seconds)) + try: + with self._lock, self._conn: + return int(self._conn.execute(_PURGE, (cutoff,)).rowcount) + except sqlite3.Error as err: + raise AuditException(f"cannot purge the audit store: {err}") from err + + def close(self) -> None: + with self._lock: + if self._closed: + return + self._closed = True + self._conn.close() + + def import_v1(self, db_path: str | os.PathLike[str]) -> int: + """Copy the rows of the v1 ``AuditLog`` database at ``db_path``. + + Each row becomes a record with the source ``audit.v1``; its payload and + result go under ``metadata``. Returns how many rows were new: importing + the same log again adds nothing. The v1 database is only read. + """ + source = Path(db_path) + if not source.is_file(): + raise AuditException(f"v1 audit log not found: {source}") + try: + with closing(sqlite3.connect(source, timeout=_TIMEOUT_SECONDS)) as old: + imported = self._copy(_v1_batches(old)) + except (sqlite3.Error, TypeError, ValueError) as err: + raise AuditException(f"cannot import the v1 audit log {source}: {err}") from err + file_automation_logger.info("audit: imported %d v1 row(s) from %s", imported, source) + return imported + + def _copy(self, batches: Iterator[list[tuple[Any, ...]]]) -> int: + with self._lock, self._conn: + before = self._conn.total_changes + for batch in batches: + self._conn.executemany(_INSERT_NEW, batch) + return self._conn.total_changes - before + + def _fetch(self, statement: str, values: Sequence[Any]) -> list[Any]: + # ``statement`` is one of this module's constants, with the WHERE clause that + # _where() joined from constants; ``values`` are bound, never interpolated. + try: + with self._lock: + return self._conn.execute(statement, values).fetchall() + except sqlite3.Error as err: + raise AuditException(f"cannot read the audit store: {err}") from err + + def _prepare(self) -> None: + self._conn.execute("PRAGMA journal_mode=WAL") + self._conn.execute("PRAGMA synchronous=NORMAL") + self._conn.executescript(_SCHEMA) + found = self._conn.execute(_VERSION).fetchone()[0] + if found is None: + with self._conn: + self._conn.execute( + "INSERT INTO audit_schema_version (version) VALUES (?)", (SCHEMA_VERSION,) + ) + elif found != SCHEMA_VERSION: + raise AuditException( + f"audit schema version {found} is not supported (this version reads " + f"{SCHEMA_VERSION})" + ) + + +def _v1_batches(old: sqlite3.Connection) -> Iterator[list[tuple[Any, ...]]]: + cursor = old.execute(_V1_ROWS) + while True: + rows = cursor.fetchmany(_IMPORT_BATCH) + if not rows: + return + yield [_row(_v1_record(row)) for row in rows] diff --git a/automation_file/audit/store.py b/automation_file/audit/store.py new file mode 100644 index 0000000..630876b --- /dev/null +++ b/automation_file/audit/store.py @@ -0,0 +1,270 @@ +"""Where audit records are kept: the store interface, its filters, an in-memory store. + +:class:`AuditStore` is the interface every store implements -- the SQLite store +of this package today, a PostgreSQL or remote store later. A store appends +records and never changes one; it answers searches newest first. + +Filters are passed by name to ``search`` and ``count``: + +``since`` / ``until`` + The time range, ``since`` included and ``until`` excluded. An aware + ``datetime``, an ISO 8601 string with an offset, or seconds since the epoch. +``actor``, ``source``, ``pipeline``, ``task``, ``action``, ``backend``, ``status``, ``correlation_id`` + Exact matches. +``resource_prefix`` + Records whose resource starts with the text. +``text`` + Records that contain the text in the action, resource, error, actor, + source, pipeline, task, backend or the JSON of the metadata. +``limit`` / ``offset`` + Paging; ``limit`` defaults to :data:`DEFAULT_LIMIT` and may not exceed + :data:`MAX_LIMIT`. ``count`` ignores both. + +``resource_prefix`` and ``text`` take the text literally (``%`` and ``_`` are +ordinary characters) and ignore the case of ASCII letters. A filter left out, +or given as ``None``, does not restrict the search; an unknown filter name is +an error rather than a search that silently returns everything. +""" + +from __future__ import annotations + +import json +import string +import threading +from abc import ABC, abstractmethod +from collections.abc import Mapping +from dataclasses import dataclass, fields +from datetime import datetime, timedelta, timezone +from types import TracebackType +from typing import Any, TypeVar + +from automation_file.audit.record import AuditRecord, parse_time +from automation_file.core.audit import AuditException + +DEFAULT_LIMIT = 100 +MAX_LIMIT = 10_000 +#: Filters compared for equality with the record field of the same name. +EXACT_FILTERS = ( + "actor", + "source", + "pipeline", + "task", + "action", + "backend", + "status", + "correlation_id", +) +#: Record fields the ``text`` filter looks into, next to the metadata. +TEXT_FIELDS = ("action", "resource", "error", "actor", "source", "pipeline", "task", "backend") +LIKE_ESCAPE = "\\" + +_TIME_FILTERS = ("since", "until") +_TEXT_FILTERS = (*EXACT_FILTERS, "resource_prefix", "text") +#: The paging filters and the highest value each may take (``None``: no ceiling). +_PAGE_FILTERS: dict[str, int | None] = {"limit": MAX_LIMIT, "offset": None} +_ASCII_LOWER = str.maketrans(string.ascii_uppercase, string.ascii_lowercase) +_StoreT = TypeVar("_StoreT", bound="AuditStore") + + +def escape_like(text: str) -> str: + """Return ``text`` with the SQL ``LIKE`` wildcards escaped by :data:`LIKE_ESCAPE`.""" + return ( + text.replace(LIKE_ESCAPE, LIKE_ESCAPE * 2) + .replace("%", LIKE_ESCAPE + "%") + .replace("_", LIKE_ESCAPE + "_") + ) + + +def metadata_json(record: AuditRecord) -> str: + """Return the metadata of ``record`` as the JSON text a store keeps and searches.""" + return json.dumps(dict(record.metadata), ensure_ascii=False, sort_keys=True) + + +def _fold(text: str) -> str: + """Lower the ASCII letters only, the way SQL ``LIKE`` compares.""" + return text.translate(_ASCII_LOWER) + + +def _page_number(value: object, name: str, maximum: int | None) -> int: + if isinstance(value, bool) or not isinstance(value, int) or value < 0: + raise AuditException(f"{name} must be an integer, 0 or more, got {value!r}") + if maximum is not None and value > maximum: + raise AuditException(f"{name} may not exceed {maximum}, got {value}") + return value + + +@dataclass(frozen=True) +class AuditQuery: + """The validated filters of one search.""" + + since: datetime | None = None + until: datetime | None = None + actor: str | None = None + source: str | None = None + pipeline: str | None = None + task: str | None = None + action: str | None = None + resource_prefix: str | None = None + backend: str | None = None + status: str | None = None + correlation_id: str | None = None + text: str | None = None + limit: int = DEFAULT_LIMIT + offset: int = 0 + + @classmethod + def from_filters(cls, filters: Mapping[str, Any]) -> AuditQuery: + """Validate the keyword filters of ``search`` / ``count`` into a query.""" + known = {entry.name for entry in fields(cls)} + unknown = sorted(set(filters) - known) + if unknown: + raise AuditException(f"unknown audit filter(s) {unknown}; known: {sorted(known)}") + given = {name: value for name, value in filters.items() if value is not None} + for name in _TIME_FILTERS: + if name in given: + given[name] = parse_time(given[name], name) + for name in _TEXT_FILTERS: + if name in given and not isinstance(given[name], str): + raise AuditException(f"{name} must be a string, got {given[name]!r}") + for name, maximum in _PAGE_FILTERS.items(): + if name in given: + given[name] = _page_number(given[name], name, maximum) + return cls(**given) + + def matches(self, record: AuditRecord) -> bool: + """Return whether ``record`` passes every filter but the paging.""" + if self.since is not None and record.timestamp < self.since: + return False + if self.until is not None and record.timestamp >= self.until: + return False + for name in EXACT_FILTERS: + wanted = getattr(self, name) + if wanted is not None and getattr(record, name) != wanted: + return False + return self._matches_resource(record) and self._matches_text(record) + + def _matches_resource(self, record: AuditRecord) -> bool: + if not self.resource_prefix: + return True + return _fold(record.resource or "").startswith(_fold(self.resource_prefix)) + + def _matches_text(self, record: AuditRecord) -> bool: + if not self.text: + return True + needle = _fold(self.text) + haystacks = [getattr(record, name) or "" for name in TEXT_FIELDS] + haystacks.append(metadata_json(record)) + return any(needle in _fold(haystack) for haystack in haystacks) + + +class AuditStore(ABC): + """The interface of a place that keeps audit records. + + A store is append-only, safe to share between threads, and answers + ``search`` newest first. Every failure is an + :class:`~automation_file.AuditException`. Implement the five methods and + parse the filters with :meth:`AuditQuery.from_filters` to plug a new store + into :class:`~automation_file.audit.trail.AuditTrail`. + """ + + @abstractmethod + def append(self, record: AuditRecord) -> None: + """Keep ``record``. A record whose ``id`` is already kept is an error.""" + + @abstractmethod + def search(self, **filters: Any) -> list[AuditRecord]: + """Return the records that pass ``filters``, newest first.""" + + @abstractmethod + def count(self, **filters: Any) -> int: + """Return how many records pass ``filters`` (``limit`` and ``offset`` are ignored).""" + + @abstractmethod + def purge(self, older_than_seconds: float) -> int: + """Delete the records older than ``older_than_seconds``; return how many.""" + + @abstractmethod + def close(self) -> None: + """Release what the store holds open. Closing twice is harmless.""" + + def __enter__(self: _StoreT) -> _StoreT: + return self + + def __exit__( + self, + exc_type: type[BaseException] | None, + exc: BaseException | None, + tb: TracebackType | None, + ) -> None: + self.close() + + +def purge_cutoff(older_than_seconds: float) -> datetime: + """Return the time before which ``purge(older_than_seconds)`` deletes.""" + if isinstance(older_than_seconds, bool) or not isinstance(older_than_seconds, (int, float)): + raise AuditException(f"older_than_seconds must be a number, got {older_than_seconds!r}") + if not older_than_seconds > 0: + raise AuditException("older_than_seconds must be positive") + try: + return datetime.now(timezone.utc) - timedelta(seconds=older_than_seconds) + except (OverflowError, ValueError) as err: + raise AuditException(f"older_than_seconds is out of range: {older_than_seconds!r}") from err + + +class MemoryAuditStore(AuditStore): + """An audit store that lives in the process; for tests and short-lived tools.""" + + def __init__(self) -> None: + self._lock = threading.Lock() + self._records: list[AuditRecord] = [] + self._ids: set[str] = set() + self._closed = False + + def append(self, record: AuditRecord) -> None: + check_record(record) + with self._lock: + self._check_open() + if record.id in self._ids: + raise AuditException(f"audit record {record.id} is already stored") + self._ids.add(record.id) + self._records.append(record) + + def search(self, **filters: Any) -> list[AuditRecord]: + query = AuditQuery.from_filters(filters) + matched = self._matching(query) + return matched[query.offset : query.offset + query.limit] + + def count(self, **filters: Any) -> int: + return len(self._matching(AuditQuery.from_filters(filters))) + + def purge(self, older_than_seconds: float) -> int: + cutoff = purge_cutoff(older_than_seconds) + with self._lock: + self._check_open() + kept = [record for record in self._records if record.timestamp >= cutoff] + removed = len(self._records) - len(kept) + self._records = kept + self._ids = {record.id for record in kept} + return removed + + def close(self) -> None: + with self._lock: + self._closed = True + + def _matching(self, query: AuditQuery) -> list[AuditRecord]: + """Return the matching records, newest first; the later of two equal times first.""" + with self._lock: + self._check_open() + numbered = list(enumerate(self._records)) + matched = [entry for entry in numbered if query.matches(entry[1])] + matched.sort(key=lambda entry: (entry[1].timestamp, entry[0]), reverse=True) + return [record for _, record in matched] + + def _check_open(self) -> None: + if self._closed: + raise AuditException("the audit store is closed") + + +def check_record(record: object) -> None: + if not isinstance(record, AuditRecord): + raise AuditException(f"expected AuditRecord, got {type(record).__name__}") diff --git a/automation_file/audit/trail.py b/automation_file/audit/trail.py new file mode 100644 index 0000000..c49a9a5 --- /dev/null +++ b/automation_file/audit/trail.py @@ -0,0 +1,215 @@ +"""The audit trail: turn what happens into audit records. + +An :class:`AuditTrail` listens to the event bus and to the storage observers +and appends one :class:`~automation_file.audit.record.AuditRecord` per event +and per storage operation to its store. Nothing writes an audit row itself: +components publish events, the storage layer reports its operations, and the +trail records both. + +A failed storage operation is reported twice by the layers below -- as the +operation and as the ``storage.error`` event the storage bridge publishes for +it. The trail keeps the operation and skips that event, so it is recorded once. + +Auditing never breaks what it audits: a record that cannot be built or written +is logged and dropped. + +The process-wide :data:`audit_trail` has no store and records nothing until +:func:`configure_audit` gives it one. +""" + +from __future__ import annotations + +import os +import threading +from collections.abc import Callable +from typing import Any, TypeVar + +from automation_file.audit.record import ( + STORAGE_SOURCE, + AuditRecord, + record_from_event, + record_from_operation, +) +from automation_file.audit.sqlite_store import SQLiteAuditStore +from automation_file.audit.store import AuditStore +from automation_file.core.audit import AuditException +from automation_file.events import Event, EventBus, StorageError, Subscription, event_bus +from automation_file.logging_config import file_automation_logger +from automation_file.storage import observe +from automation_file.storage.observe import StorageOperation + +_SubjectT = TypeVar("_SubjectT") +_BRIDGE_MARK = "error_type" + + +def _is_reported_operation(event: Event) -> bool: + """Return whether ``event`` is the storage bridge's report of a failed operation.""" + return ( + event.type == StorageError.type + and event.source == STORAGE_SOURCE + and _BRIDGE_MARK in event.payload + ) + + +class AuditTrail: + """Record events and storage operations into an :class:`AuditStore`.""" + + def __init__(self, store: AuditStore | None = None, bus: EventBus | None = None) -> None: + self._store = store + self._owns_store = False + self._bus = bus if bus is not None else event_bus + self._lock = threading.RLock() + self._subscription: Subscription | None = None + + @property + def store(self) -> AuditStore | None: + return self._store + + @property + def bus(self) -> EventBus: + return self._bus + + @property + def active(self) -> bool: + """Whether the trail is listening (between ``start`` and ``stop``).""" + with self._lock: + return self._subscription is not None + + def attach(self, store: AuditStore, *, owned: bool = False) -> None: + """Record into ``store`` from now on. + + An ``owned`` store is closed by the trail when it is replaced or the + trail is closed; a store the caller built stays the caller's to close. + """ + if not isinstance(store, AuditStore): + raise AuditException(f"expected AuditStore, got {type(store).__name__}") + with self._lock: + previous, was_owned = self._store, self._owns_store + self._store, self._owns_store = store, owned + if was_owned and previous is not None and previous is not store: + previous.close() + + def start(self) -> None: + """Listen to the bus and to the storage observers. Starting twice changes nothing.""" + with self._lock: + if self._subscription is not None: + return + if self._store is None: + raise AuditException("the audit trail has no store; attach one first") + self._subscription = self._bus.subscribe(self._on_event) + observe.add_listener(self._on_operation) + file_automation_logger.info("audit trail: started") + + def stop(self) -> None: + """Stop listening. Stopping an inactive trail changes nothing.""" + with self._lock: + subscription, self._subscription = self._subscription, None + if subscription is None: + return + self._bus.unsubscribe(subscription) + observe.remove_listener(self._on_operation) + file_automation_logger.info("audit trail: stopped") + + def close(self) -> None: + """Stop listening and let go of the store, closing it when the trail owns it.""" + self.stop() + with self._lock: + store, owned = self._store, self._owns_store + self._store, self._owns_store = None, False + if owned and store is not None: + store.close() + + def record(self, action: str, **details: Any) -> AuditRecord | None: + """Append one record by hand and return it. + + ``details`` are the other :class:`AuditRecord` fields (``resource``, + ``backend``, ``status``, ``source``, ``pipeline``, ``task``, + ``duration_ms``, ``error``, ``metadata`` ...). The actor and the + correlation ID default to those of the current scopes. A field that + does not exist is the caller's mistake and raises; a failure to write + is logged and gives ``None``, as does a trail without a store. + """ + try: + entry = AuditRecord(action=action, **details) + except TypeError as err: + raise AuditException(f"invalid audit record: {err}") from err + return entry if self._append(entry) else None + + def search(self, **filters: Any) -> list[AuditRecord]: + """Return the records of the store that pass ``filters``, newest first.""" + return self._require_store().search(**filters) + + def count(self, **filters: Any) -> int: + """Return how many records of the store pass ``filters``.""" + return self._require_store().count(**filters) + + def purge(self, older_than_seconds: float) -> int: + """Delete the records older than ``older_than_seconds``; return how many.""" + return self._require_store().purge(older_than_seconds) + + def _require_store(self) -> AuditStore: + store = self._store + if store is None: + raise AuditException("audit is not configured; call configure_audit first") + return store + + def _on_event(self, event: Event) -> None: + if _is_reported_operation(event): + return + self._capture(record_from_event, event) + + def _on_operation(self, operation: StorageOperation) -> None: + self._capture(record_from_operation, operation) + + def _capture(self, build: Callable[[_SubjectT], AuditRecord], subject: _SubjectT) -> None: + try: + entry = build(subject) + except Exception as error: # pylint: disable=broad-except + # Boundary: auditing must never fail the code that is audited. + file_automation_logger.error("audit trail: cannot build a record: %r", error) + return + self._append(entry) + + def _append(self, entry: AuditRecord) -> bool: + store = self._store + if store is None: + return False + try: + # A store built on the storage layer must not audit its own writes. + with observe.suppressed(): + store.append(entry) + except Exception as error: # pylint: disable=broad-except + # Boundary: auditing must never fail the code that is audited. + file_automation_logger.error( + "audit trail: cannot write the record of %r: %r", entry.action, error + ) + return False + return True + + +audit_trail: AuditTrail = AuditTrail() + + +def configure_audit(target: AuditStore | str | os.PathLike[str]) -> AuditTrail: + """Point the process-wide :data:`audit_trail` at ``target`` and start it. + + ``target`` is the path of a SQLite database (created when missing) or a + ready :class:`AuditStore`. Calling it again switches the store; a store the + trail opened from a path is closed when it is replaced. + """ + if isinstance(target, AuditStore): + audit_trail.attach(target) + else: + audit_trail.attach(SQLiteAuditStore(target), owned=True) + audit_trail.start() + return audit_trail + + +def audit_search(**filters: Any) -> list[dict[str, Any]]: + """Search the process-wide audit trail; each record comes back as its ``to_dict()``. + + Filters: ``since``, ``until``, ``actor``, ``source``, ``pipeline``, ``task``, + ``action``, ``resource_prefix``, ``backend``, ``status``, ``correlation_id``, + ``text``, ``limit``, ``offset``. Newest first. + """ + return [entry.to_dict() for entry in audit_trail.search(**filters)] diff --git a/automation_file/core/action_registry.py b/automation_file/core/action_registry.py index d087408..52d4ba9 100644 --- a/automation_file/core/action_registry.py +++ b/automation_file/core/action_registry.py @@ -224,6 +224,12 @@ def _register_storage_ops(registry: ActionRegistry) -> None: register_storage_ops(registry) +def _register_audit_ops(registry: ActionRegistry) -> None: + from automation_file.audit.actions import register_audit_ops + + register_audit_ops(registry) + + def _register_integrity_ops(registry: ActionRegistry) -> None: from automation_file.integrity.actions import register_integrity_ops @@ -249,6 +255,7 @@ def build_default_registry() -> ActionRegistry: _register_notify_ops(registry) _register_storage_ops(registry) _register_integrity_ops(registry) + _register_audit_ops(registry) _load_plugins(registry) # DEBUG, not INFO: this runs at import, and INFO is mirrored to stderr, so every import -- # `python -m automation_file --help` included -- printed it. diff --git a/automation_file/core/config.py b/automation_file/core/config.py index b9f4961..41db9fa 100644 --- a/automation_file/core/config.py +++ b/automation_file/core/config.py @@ -1,10 +1,11 @@ """TOML-based configuration for automation_file. -Callers describe notification sinks, secret-provider roots, and scheduler -defaults in a single ``automation_file.toml`` file. :class:`AutomationConfig` -loads it, resolves ``${env:…}`` / ``${file:…}`` references via the secret -provider chain, and exposes helpers to materialise runtime objects (sinks, -etc.) without the caller poking at the raw dict. +Callers describe notification sinks and routes, secret-provider roots, and +scheduler defaults in a single ``automation_file.toml`` file. +:class:`AutomationConfig` loads it, resolves ``${env:…}`` / ``${file:…}`` +references via the secret provider chain, and exposes helpers to materialise +runtime objects (sinks, routes, etc.) without the caller poking at the raw +dict. Minimal example:: @@ -26,6 +27,15 @@ username = "${env:SMTP_USER}" password = "${file:smtp_password}" + [[notify.routes]] + name = "pipeline-failures" + sinks = ["team-alerts", "ops-email"] + types = ["pipeline.*", "task.failed"] + min_severity = "error" + dedup_seconds = 600 + rate_limit = 10 + rate_period = 60 + [defaults] dedup_seconds = 120 @@ -35,6 +45,7 @@ from __future__ import annotations import sys +from collections.abc import Iterable from pathlib import Path from typing import Any @@ -46,8 +57,10 @@ from automation_file.exceptions import FileAutomationException from automation_file.logging_config import file_automation_logger from automation_file.notify.manager import NotificationManager +from automation_file.notify.router import CONFIG_ORIGIN, NotificationRouter, Route from automation_file.notify.sinks import ( EmailSink, + NotificationException, NotificationSink, SlackSink, WebhookSink, @@ -119,27 +132,51 @@ def section(self, name: str) -> dict[str, Any]: def notification_sinks(self) -> list[NotificationSink]: """Instantiate every sink declared under ``[[notify.sinks]]``.""" - notify_section = self.section("notify") - entries = notify_section.get("sinks") or [] - if not isinstance(entries, list): - raise ConfigException("'notify.sinks' must be an array of tables") - sinks: list[NotificationSink] = [] - for entry in entries: - if not isinstance(entry, dict): - raise ConfigException("each 'notify.sinks' entry must be a table") - sinks.append(_build_sink(entry)) - return sinks + return [_build_sink(entry) for entry in self._notify_tables("sinks")] + + def notification_routes(self, known_sinks: Iterable[str] = ()) -> list[Route]: + """Build every route declared under ``[[notify.routes]]``. - def apply_to(self, manager: NotificationManager) -> int: + A route may only name a sink declared under ``[[notify.sinks]]`` or + listed in ``known_sinks`` (the sinks registered in code). An unknown + sink, an unknown severity, an unknown option or a name used twice + raises :class:`ConfigException`. + """ + known = set(known_sinks) + known.update(_sink_name(entry) for entry in self._notify_tables("sinks")) + routes: dict[str, Route] = {} + for entry in self._notify_tables("routes"): + route = _build_route(entry) + if route.name in routes: + raise ConfigException(f"route {route.name!r} is declared twice") + unknown = [sink for sink in route.sinks if sink not in known] + if unknown: + raise ConfigException( + f"route {route.name!r} names unknown sink(s) {unknown} (known: {sorted(known)})" + ) + routes[route.name] = route + return list(routes.values()) + + def apply_to( + self, manager: NotificationManager, router: NotificationRouter | None = None + ) -> int: """Register every configured sink into ``manager``. Returns the count. Existing registrations are preserved; duplicates by name are replaced (see :meth:`NotificationManager.register`). + + With a ``router``, the ``[[notify.routes]]`` tables become its + configured routes: the ones this file declared before and no longer + does are removed, while routes added in code stay. The router is + started when the file declares a route, and stopped when removing the + file's routes leaves it with none, because a router without routes + would deliver nothing. Sinks and routes are both validated before + anything is registered. """ - count = 0 - for sink in self.notification_sinks(): + sinks = self.notification_sinks() + routes = self.notification_routes(known_sinks=manager.names()) + for sink in sinks: manager.register(sink) - count += 1 defaults = self.section("defaults") if "dedup_seconds" in defaults: try: @@ -148,7 +185,46 @@ def apply_to(self, manager: NotificationManager) -> int: raise ConfigException( f"defaults.dedup_seconds must be a number, got {defaults['dedup_seconds']!r}" ) from err - return count + if router is not None: + _apply_routes(router, routes) + elif routes: + file_automation_logger.warning( + "config: %d notify route(s) declared, but apply_to was given no router", + len(routes), + ) + return len(sinks) + + def _notify_tables(self, key: str) -> list[dict[str, Any]]: + entries = self.section("notify").get(key) or [] + if not isinstance(entries, list): + raise ConfigException(f"'notify.{key}' must be an array of tables") + for entry in entries: + if not isinstance(entry, dict): + raise ConfigException(f"each 'notify.{key}' entry must be a table") + return entries + + +def _apply_routes(router: NotificationRouter, routes: list[Route]) -> None: + removed = router.sync_routes(routes, origin=CONFIG_ORIGIN) + if routes: + router.start() + elif removed and not router.routes(): + # The file took the last route away: an active router would now deliver nothing. + router.stop() + + +def _sink_name(entry: dict[str, Any]) -> str: + """Return the name a sink entry registers under: its ``name``, else its ``type``.""" + return str(entry.get("name") or entry.get("type") or "") + + +def _build_route(entry: dict[str, Any]) -> Route: + try: + return Route.from_mapping(entry) + except (NotificationException, TypeError, ValueError) as err: + raise ConfigException( + f"invalid config for route {entry.get('name') or ''!r}: {err}" + ) from err def _build_sink(entry: dict[str, Any]) -> NotificationSink: diff --git a/automation_file/core/fim.py b/automation_file/core/fim.py index 4b154ea..02763da 100644 --- a/automation_file/core/fim.py +++ b/automation_file/core/fim.py @@ -9,7 +9,9 @@ The notification still goes through the ``manager`` passed to the constructor, or through the process-wide ``notification_manager`` when none is. One thing was -added: drift is also published as an ``IntegrityViolation`` event. +added: drift is also published as an ``IntegrityViolation`` event, and while the +notification router is active its routes deliver that event in place of the +direct notification to the process-wide manager. """ from __future__ import annotations diff --git a/automation_file/core/metrics.py b/automation_file/core/metrics.py index ee8fcc4..5916dea 100644 --- a/automation_file/core/metrics.py +++ b/automation_file/core/metrics.py @@ -1,26 +1,54 @@ -"""Prometheus metrics — per-action counters and duration histogram. +"""Prometheus metrics: actions, events, notifications and storage operations. -The module exposes two metrics that are updated from -:class:`~automation_file.core.action_executor.ActionExecutor` on every -call: +Per-action metrics, updated by +:class:`~automation_file.core.action_executor.ActionExecutor` on every call: * ``automation_file_actions_total{action, status}`` — counter incremented with ``status="ok"`` or ``status="error"`` per action. * ``automation_file_action_duration_seconds{action}`` — histogram of wall time spent inside the registered callable. +Operational metrics: + +* ``automation_file_events_total{type, severity}`` — events published on the + event bus. +* ``automation_file_notifications_total{sink, outcome}`` — notifications per + sink, with ``outcome`` one of ``sent``, ``dedup``, ``rate_limited`` or + ``error``. +* ``automation_file_storage_operations_total{operation, backend, status}`` — + uploads, downloads, reads, deletes, mkdirs, copies and moves of the storage + layer. +* ``automation_file_storage_operation_duration_seconds{operation, backend}`` — + histogram of the time one storage operation took. + +The event and storage metrics are fed by a bus subscriber and a storage +observer that :func:`install_operational_metrics` registers once. The +notification counter is fed by the notification manager and router, which +are the only places that know a delivery succeeded. + +No label ever carries a path, a URI or a correlation ID, and each label keeps +at most :data:`MAX_LABEL_VALUES` distinct values; further values are counted +under ``other``. + :func:`render` returns the wire-format text and matching ``Content-Type`` -suitable for a ``GET /metrics`` handler. :func:`record_action` is the -single write path — failures are swallowed so a broken metrics backend -can never abort a real action. +suitable for a ``GET /metrics`` handler. Every write path swallows its own +failures so a broken metrics backend can never abort a real action. """ from __future__ import annotations +import threading +from typing import TYPE_CHECKING + from prometheus_client import CONTENT_TYPE_LATEST, REGISTRY, Counter, Histogram, generate_latest from automation_file.logging_config import file_automation_logger +if TYPE_CHECKING: + from automation_file.events.bus import EventBus, Subscription + from automation_file.events.model import Event + from automation_file.storage.observe import StorageOperation + _DURATION_BUCKETS = ( 0.005, 0.01, @@ -37,6 +65,12 @@ 60.0, ) +#: Distinct values one label may take before the rest is counted as ``other``. +MAX_LABEL_VALUES = 100 +OTHER_LABEL = "other" +_UNKNOWN_LABEL = "unknown" +_MILLISECONDS = 1000.0 + ACTION_COUNT = Counter( "automation_file_actions_total", "Total actions executed, partitioned by outcome.", @@ -48,6 +82,57 @@ labelnames=("action",), buckets=_DURATION_BUCKETS, ) +EVENT_COUNT = Counter( + "automation_file_events_total", + "Events published on the event bus, by type and severity.", + labelnames=("type", "severity"), +) +NOTIFICATION_COUNT = Counter( + "automation_file_notifications_total", + "Notifications per sink, by outcome (sent, dedup, rate_limited, error).", + labelnames=("sink", "outcome"), +) +STORAGE_OPERATION_COUNT = Counter( + "automation_file_storage_operations_total", + "Storage operations, by operation, backend and status.", + labelnames=("operation", "backend", "status"), +) +STORAGE_OPERATION_DURATION = Histogram( + "automation_file_storage_operation_duration_seconds", + "Time one storage operation took.", + labelnames=("operation", "backend"), + buckets=_DURATION_BUCKETS, +) + + +class _BoundedLabel: + """Let a label take a limited number of distinct values; the rest become ``other``.""" + + def __init__(self, limit: int = MAX_LABEL_VALUES) -> None: + self._limit = limit + self._lock = threading.Lock() + self._known: set[str] = set() + + def __call__(self, value: object) -> str: + text = str(value) if value else _UNKNOWN_LABEL + with self._lock: + if text in self._known: + return text + if len(self._known) < self._limit: + self._known.add(text) + return text + return OTHER_LABEL + + +_event_type = _BoundedLabel() +_sink_name = _BoundedLabel() +_notification_outcome = _BoundedLabel() +_storage_operation = _BoundedLabel() +_storage_backend = _BoundedLabel() +_storage_status = _BoundedLabel() + +_install_lock = threading.Lock() +_subscriptions: list[tuple[EventBus, Subscription]] = [] def record_action(action: str, duration_seconds: float, ok: bool) -> None: @@ -60,6 +145,77 @@ def record_action(action: str, duration_seconds: float, ok: bool) -> None: file_automation_logger.error("metrics.record_action failed: %r", err) +def record_event(event: Event) -> None: + """Count one published event by type and severity. Never raises.""" + try: + EVENT_COUNT.labels(type=_event_type(event.type), severity=event.severity.value).inc() + except Exception as err: # pylint: disable=broad-except # pragma: no cover - defensive + file_automation_logger.error("metrics.record_event failed: %r", err) + + +def record_notification(sink: str, outcome: str) -> None: + """Count one notification for ``sink`` with its ``outcome``. Never raises.""" + try: + NOTIFICATION_COUNT.labels( + sink=_sink_name(sink), outcome=_notification_outcome(outcome) + ).inc() + except Exception as err: # pylint: disable=broad-except # pragma: no cover - defensive + file_automation_logger.error("metrics.record_notification failed: %r", err) + + +def record_storage_operation(operation: StorageOperation) -> None: + """Count one storage operation and observe how long it took. Never raises.""" + try: + name = _storage_operation(operation.operation) + backend = _storage_backend(operation.backend) + STORAGE_OPERATION_COUNT.labels( + operation=name, backend=backend, status=_storage_status(operation.status) + ).inc() + STORAGE_OPERATION_DURATION.labels(operation=name, backend=backend).observe( + max(0.0, float(operation.duration_ms)) / _MILLISECONDS + ) + except Exception as err: # pylint: disable=broad-except # pragma: no cover - defensive + file_automation_logger.error("metrics.record_storage_operation failed: %r", err) + + +def install_operational_metrics(bus: EventBus | None = None) -> bool: + """Feed the event and storage metrics from ``bus`` and the storage observers. + + ``bus`` defaults to the process-wide event bus. Calling it again for the + same bus changes nothing; the return value tells whether this call + subscribed. + """ + from automation_file.events.bus import event_bus + from automation_file.storage import observe + + target = bus if bus is not None else event_bus + with _install_lock: + if any(known is target for known, _ in _subscriptions): + return False + _subscriptions.append((target, target.subscribe(record_event))) + observe.add_listener(record_storage_operation) + return True + + +def uninstall_operational_metrics(bus: EventBus | None = None) -> bool: + """Stop feeding the metrics from ``bus``; return whether it was installed. + + The storage observer goes with the last bus. The collected values stay. + """ + from automation_file.events.bus import event_bus + from automation_file.storage import observe + + target = bus if bus is not None else event_bus + with _install_lock: + installed = [entry for entry in _subscriptions if entry[0] is target] + for entry in installed: + _subscriptions.remove(entry) + target.unsubscribe(entry[1]) + if not _subscriptions: + observe.remove_listener(record_storage_operation) + return bool(installed) + + def render() -> tuple[bytes, str]: """Return ``(payload, content_type)`` for a ``/metrics`` response.""" return generate_latest(REGISTRY), CONTENT_TYPE_LATEST diff --git a/automation_file/integrity/legacy.py b/automation_file/integrity/legacy.py index 2335577..3360e67 100644 --- a/automation_file/integrity/legacy.py +++ b/automation_file/integrity/legacy.py @@ -85,10 +85,19 @@ def format_body(summary: dict[str, Any]) -> str: return "\n".join(parts) if parts else "no drift detected" -def _process_wide_manager() -> NotificationManager: - """Return the shared manager, looked up when it is needed so a test may replace it.""" +def _process_wide_manager(on_shared_bus: bool) -> NotificationManager | None: + """Return the shared manager, or ``None`` while the notification router delivers. + + An active router hears the drift event on the process-wide bus and its routes + decide who is told, so a direct notification would announce one drift twice: + the rule of ``notify_on_failure``. Both are looked up when they are needed, so + a test may replace them. + """ from automation_file import notify + from automation_file.notify import router + if on_shared_bus and router.notification_router.active: + return None return notify.notification_manager @@ -97,7 +106,9 @@ class LegacyHooks: The notification goes through the manager the caller passed, or through the process-wide ``notification_manager`` when none was: what the first monitor - did. ``notify=False`` sends none. + did. The process-wide manager is left alone while the notification router is + active and the drift event is published on the process-wide bus + (``on_shared_bus``). ``notify=False`` sends none. """ def __init__( @@ -108,11 +119,13 @@ def __init__( manager: NotificationManager | None = None, alert_on_extra: bool = False, notify: bool = True, + on_shared_bus: bool = True, ) -> None: self._subject = subject self._on_drift = on_drift self._manager = manager self._notifies = bool(notify) + self._on_shared_bus = bool(on_shared_bus) self._alert_on_extra = bool(alert_on_extra) def is_drift(self, summary: dict[str, Any]) -> bool: @@ -127,8 +140,13 @@ def handle(self, summary: dict[str, Any]) -> None: return if self._on_drift is not None: self._call_back(self._on_drift, summary) - if self._notifies: - self._notify(self._manager or _process_wide_manager(), summary) + if not self._notifies: + return + manager = self._manager + if manager is None: + manager = _process_wide_manager(self._on_shared_bus) + if manager is not None: + self._notify(manager, summary) def _call_back(self, on_drift: OnDrift, summary: dict[str, Any]) -> None: try: diff --git a/automation_file/integrity/monitor.py b/automation_file/integrity/monitor.py index 951e5dc..1758c0b 100644 --- a/automation_file/integrity/monitor.py +++ b/automation_file/integrity/monitor.py @@ -37,7 +37,7 @@ from datetime import datetime, timezone from typing import TYPE_CHECKING, Any, TypedDict -from automation_file.events.bus import EventBus +from automation_file.events.bus import EventBus, event_bus from automation_file.events.context import correlation_scope from automation_file.exceptions import FileAutomationException from automation_file.integrity.alerts import AlertEngine, AlertPolicy @@ -177,8 +177,10 @@ class IntegrityMonitor: The hooks of the first monitor, which work as they did: the callback and the notification receive the summary :meth:`check_once` returns, and the notification goes through ``manager``, or through the process-wide - ``notification_manager`` when none is passed. ``notify=False`` sends no - notification, for when the published event is routed to the sinks instead. + ``notification_manager`` when none is passed. While the notification + router is active its routes deliver the published event and the + process-wide manager is not notified directly. ``notify=False`` sends no + direct notification at all. ``root``, ``manifest_path`` The first monitor's names for ``target`` and ``baseline``. """ @@ -213,6 +215,7 @@ def __init__( manager=chosen.manager, alert_on_extra=chosen.alert_on_extra, notify=chosen.notify, + on_shared_bus=chosen.bus is None or chosen.bus is event_bus, ) self._lock = threading.RLock() self._last_report: DriftReport | None = None diff --git a/automation_file/notify/__init__.py b/automation_file/notify/__init__.py index a5600b7..dbc5f8c 100644 --- a/automation_file/notify/__init__.py +++ b/automation_file/notify/__init__.py @@ -5,6 +5,10 @@ deduplicates identical messages within a sliding window so a stuck trigger cannot flood the channel, and it catches per-sink failures so one broken sink cannot starve the others. + +The :class:`NotificationRouter` drives the sinks from the event bus: a +:class:`Route` says which events reach which sinks, with deduplication and a +rate limit per route and sink. """ from __future__ import annotations @@ -16,6 +20,16 @@ notify_send, register_notify_ops, ) +from automation_file.notify.router import ( + NotificationMessage, + NotificationRouter, + Route, + message_for, + notification_router, + notify_route_add, + notify_route_list, + notify_route_remove, +) from automation_file.notify.sinks import ( DiscordSink, EmailSink, @@ -32,13 +46,21 @@ "EmailSink", "NotificationException", "NotificationManager", + "NotificationMessage", + "NotificationRouter", "NotificationSink", "PagerDutySink", + "Route", "SlackSink", "TeamsSink", "TelegramSink", "WebhookSink", + "message_for", "notification_manager", + "notification_router", + "notify_route_add", + "notify_route_list", + "notify_route_remove", "notify_send", "register_notify_ops", ] diff --git a/automation_file/notify/manager.py b/automation_file/notify/manager.py index fc05164..d24a7be 100644 --- a/automation_file/notify/manager.py +++ b/automation_file/notify/manager.py @@ -8,15 +8,22 @@ messages seen within the window, which is the minimum safety net against a stuck trigger flooding a channel. ``dedup_seconds=0`` disables the guard. + +:meth:`NotificationManager.send_to` delivers to one named sink without that +window; the :class:`~automation_file.notify.router.NotificationRouter` uses it +and applies its own deduplication and rate limits per route. """ from __future__ import annotations +import re import threading import time from typing import Any from automation_file.core.action_registry import ActionRegistry +from automation_file.core.metrics import record_notification +from automation_file.events import Event, SchedulerError, SystemErrorEvent, emit from automation_file.exceptions import FileAutomationException from automation_file.logging_config import file_automation_logger from automation_file.notify.sinks import ( @@ -26,6 +33,29 @@ ) _DEFAULT_DEDUP_SECONDS = 60.0 +OUTCOME_SENT = "sent" +OUTCOME_DEDUP = "dedup" +OUTCOME_ERROR = "error" +_REDACTED = "" +# A webhook URL or a bot token is a secret, and the HTTP client quotes the URL +# in its error text. Keep the host, drop everything after it. +_URL_PATTERN = re.compile(r"(?i)\b(https?://)(?:[^/\s@'\"]*@)?([^/\s'\"]+)[^\s'\")]*") +_URL_PATH_PATTERN = re.compile(r"(?i)(\burl: )\S+") +_CONTEXT_PATTERN = re.compile(r"(?P[A-Za-z_]+)\[(?P.*)\]", re.DOTALL) +_SCHEDULER_KIND = "scheduler" +_TRIGGER_KIND = "trigger" +_SYSTEM_SOURCE = "system" + + +def redact_urls(text: str) -> str: + """Return ``text`` with the path and credentials of every URL removed.""" + text = _URL_PATTERN.sub(rf"\1\2/{_REDACTED}", text) + return _URL_PATH_PATTERN.sub(rf"\1{_REDACTED}", text) + + +def describe_error(error: BaseException) -> str: + """Return ``": "`` for ``error`` with its URLs redacted.""" + return f"{type(error).__name__}: {redact_urls(str(error))}" class NotificationManager: @@ -67,6 +97,11 @@ def list(self) -> list[dict[str, Any]]: sinks = list(self._sinks.values()) return [_describe(sink) for sink in sinks] + def names(self) -> tuple[str, ...]: + """Return the names of the registered sinks, in registration order.""" + with self._lock: + return tuple(self._sinks) + def has_sinks(self) -> bool: """Return whether at least one sink is currently registered.""" with self._lock: @@ -84,17 +119,35 @@ def notify( Missing sinks return an empty dict — callers can use that to detect an unconfigured notifier rather than silently succeeding. """ - if not isinstance(subject, str) or not subject: - raise NotificationException("subject must be a non-empty string") + _check_subject(subject) with self._lock: sinks = list(self._sinks.values()) - if self._should_dedup(subject, body, level): - return {sink.name: "dedup" for sink in sinks} + duplicate = self._should_dedup(subject, body, level) + if duplicate: + for sink in sinks: + record_notification(sink.name, OUTCOME_DEDUP) + return {sink.name: OUTCOME_DEDUP for sink in sinks} results: dict[str, Any] = {} for sink in sinks: results[sink.name] = self._deliver(sink, subject, body, level) return results + def send_to(self, name: str, subject: str, body: str = "", level: str = "info") -> str: + """Deliver one message to the sink registered as ``name``. + + Returns ``"sent"``, or ``": "`` when the sink + failed; the failure is logged and never raised. The deduplication + window of :meth:`notify` does not apply. Raises + :class:`NotificationException` when no sink has that name. + """ + _check_subject(subject) + with self._lock: + sink = self._sinks.get(name) + if sink is None: + raise NotificationException(f"no notification sink is registered as {name!r}") + failure = self._attempt(sink, subject, body, level) + return OUTCOME_SENT if failure is None else describe_error(failure) + def _deliver( self, sink: NotificationSink, @@ -102,15 +155,34 @@ def _deliver( body: str, level: str, ) -> str: + failure = self._attempt(sink, subject, body, level) + return OUTCOME_SENT if failure is None else redact_urls(repr(failure)) + + def _attempt( + self, + sink: NotificationSink, + subject: str, + body: str, + level: str, + ) -> Exception | None: + """Send through ``sink``; return what it raised, or ``None`` when it delivered.""" try: sink.send(subject, body, level) except NotificationException as err: - file_automation_logger.error("notify: sink %r failed: %r", sink.name, err) - return repr(err) + file_automation_logger.error( + "notify: sink %r failed: %s", sink.name, describe_error(err) + ) + record_notification(sink.name, OUTCOME_ERROR) + return err except Exception as err: # pylint: disable=broad-except - file_automation_logger.error("notify: sink %r raised unexpectedly: %r", sink.name, err) - return repr(err) - return "sent" + # Boundary: one sink's bug must not reach the other sinks or the caller. + file_automation_logger.error( + "notify: sink %r raised unexpectedly: %s", sink.name, describe_error(err) + ) + record_notification(sink.name, OUTCOME_ERROR) + return err + record_notification(sink.name, OUTCOME_SENT) + return None def _should_dedup(self, subject: str, body: str, level: str) -> bool: if self.dedup_seconds <= 0.0: @@ -130,6 +202,11 @@ def _prune(self, now: float) -> None: self._recent.pop(key, None) +def _check_subject(subject: str) -> None: + if not isinstance(subject, str) or not subject: + raise NotificationException("subject must be a non-empty string") + + notification_manager: NotificationManager = NotificationManager() @@ -147,13 +224,48 @@ def notify_list() -> list[dict[str, Any]]: return notification_manager.list() +def failure_event(context: str, error: BaseException) -> Event: + """Return the event that reports ``context`` failing with ``error``. + + ``scheduler[]`` becomes a :class:`SchedulerError` whose payload names + the ``job``. Any other context becomes a :class:`SystemErrorEvent`; its + source is the word before the brackets (``trigger[inbox]`` gives + ``trigger``, with the name under the ``trigger`` key) or ``system`` when + the context has none. + """ + subject = f"{context} failed" + payload: dict[str, Any] = { + "status": OUTCOME_ERROR, + "error": describe_error(error), + "context": context, + } + match = _CONTEXT_PATTERN.fullmatch(context) + if match is None: + return SystemErrorEvent(source=_SYSTEM_SOURCE, subject=subject, payload=payload) + kind, name = match.group("kind"), match.group("name") + if kind == _SCHEDULER_KIND: + payload["job"] = name + return SchedulerError(source=_SCHEDULER_KIND, subject=subject, payload=payload) + if kind == _TRIGGER_KIND: + payload["trigger"] = name + return SystemErrorEvent(source=kind, subject=subject, payload=payload) + + def notify_on_failure(context: str, error: FileAutomationException | Exception) -> None: - """Helper for auto-notify hooks — sends an ``error``-level message. + """Report that ``context`` failed. + + The failure is always published on the event bus. When the notification + router is active it delivers the event and nothing else is sent; when it is + not, the ``error``-level message goes straight to every registered sink, as + it did before the router existed. - Does nothing when no sinks are registered, so callers can call this + Does nothing more when no sinks are registered, so callers can call this unconditionally without having to check the configuration. """ - if not notification_manager.has_sinks(): + from automation_file.notify.router import notification_router + + emit(failure_event(context, error)) + if notification_router.active or not notification_manager.has_sinks(): return try: notification_manager.notify( @@ -165,9 +277,18 @@ def notify_on_failure(context: str, error: FileAutomationException | Exception) def register_notify_ops(registry: ActionRegistry) -> None: """Wire ``FA_notify_*`` actions into a registry.""" + from automation_file.notify.router import ( + notify_route_add, + notify_route_list, + notify_route_remove, + ) + registry.register_many( { "FA_notify_send": notify_send, "FA_notify_list": notify_list, + "FA_notify_route_add": notify_route_add, + "FA_notify_route_remove": notify_route_remove, + "FA_notify_route_list": notify_route_list, } ) diff --git a/automation_file/notify/router.py b/automation_file/notify/router.py new file mode 100644 index 0000000..999c2b4 --- /dev/null +++ b/automation_file/notify/router.py @@ -0,0 +1,529 @@ +"""Route events to notification sinks. + +Modules publish events; they do not call a sink. A :class:`Route` says which +events go to which sinks, and the :class:`NotificationRouter` subscribes to the +event bus and delivers every matching event through the +:class:`~automation_file.notify.manager.NotificationManager`: + +.. code-block:: python + + from automation_file import Route, Severity, notification_router + + notification_router.add_route( + Route("pipeline-failures", sinks=("team-alerts",), types=("pipeline.*", "task.failed"), + min_severity=Severity.ERROR, dedup_seconds=600, rate_limit=10, rate_period=60) + ) + notification_router.start() + +Per route and sink the router drops a repeat of the same event (same type, +source and subject) inside ``dedup_seconds`` and sends at most ``rate_limit`` +messages per ``rate_period``. One sink failing never affects another; the +failure is published as a ``system.error`` event from the source ``notify``, +which the router itself never routes, so a broken sink cannot feed a loop. +""" + +from __future__ import annotations + +import itertools +import json +import math +import threading +import time +from collections import deque +from collections.abc import Callable, Iterable, Mapping +from dataclasses import dataclass, fields +from typing import Any + +from automation_file.core.metrics import record_notification +from automation_file.events import ( + Event, + EventBus, + EventFilter, + Severity, + Subscription, + SystemErrorEvent, + event_bus, +) +from automation_file.logging_config import file_automation_logger +from automation_file.notify.manager import ( + OUTCOME_DEDUP, + OUTCOME_ERROR, + OUTCOME_SENT, + NotificationManager, + describe_error, + notification_manager, +) +from automation_file.notify.sinks import NotificationException + +OUTCOME_RATE_LIMITED = "rate_limited" +#: The ``source`` of the events the router publishes about its own deliveries. +NOTIFY_SOURCE = "notify" +#: The origin of the routes loaded from ``automation_file.toml``. +CONFIG_ORIGIN = "config" + +_THROTTLED = frozenset({OUTCOME_DEDUP, OUTCOME_RATE_LIMITED}) +_FAILURE_ACTION = "notify.deliver" +_LEVELS = { + Severity.INFO: "info", + Severity.WARNING: "warning", + Severity.ERROR: "error", + Severity.CRITICAL: "error", +} +_DEFAULT_DEDUP_SECONDS = 300.0 +_DEFAULT_RATE_PERIOD = 60.0 +_PRUNE_INTERVAL = 30.0 +_MAX_DEDUP_KEYS = 10_000 +#: When the dedup memory is full, one part in this many is forgotten at once. +_FLOOD_SHARE = 10 +_MAX_SUBJECT = 200 +_MISSING = "-" + +_DedupKey = tuple[str, str, str, str, str] +_RateKey = tuple[str, str] + + +def _ignore(_event: Event) -> None: + return None + + +def _as_names(value: object, option: str) -> tuple[str, ...]: + if value is None: + return () + if isinstance(value, str): + value = (value,) + if not isinstance(value, Iterable): + raise NotificationException(f"route {option} must be a list of names, got {value!r}") + names = tuple(value) + for name in names: + if not isinstance(name, str) or not name: + raise NotificationException(f"route {option} must be non-empty strings, got {name!r}") + return tuple(dict.fromkeys(names)) + + +def _as_filters(value: object) -> tuple[EventFilter, ...]: + if value is None: + return () + if isinstance(value, (str, type)): + value = (value,) + if not isinstance(value, Iterable): + raise NotificationException(f"route types must be a list of event types, got {value!r}") + filters = tuple(value) + for wanted in filters: + named = isinstance(wanted, str) and bool(wanted) + if not named and not (isinstance(wanted, type) and issubclass(wanted, Event)): + raise NotificationException( + f"a route type is an event class, a type name or a 'prefix.*', got {wanted!r}" + ) + return filters + + +def _as_severity(value: object) -> Severity: + if isinstance(value, Severity): + return value + allowed = [severity.value for severity in Severity] + wanted = value.strip().lower() if isinstance(value, str) else None + if wanted in allowed: + return Severity(wanted) + raise NotificationException(f"min_severity must be one of {allowed}, got {value!r}") + + +def _as_seconds(value: object, option: str, *, positive: bool = False) -> float: + number = math.nan + if isinstance(value, (int, float)) and not isinstance(value, bool): + number = float(value) + if math.isfinite(number) and (number > 0 or (number == 0 and not positive)): + return number + wanted = "a positive number" if positive else "a number, 0 or more" + raise NotificationException(f"route {option} must be {wanted}, got {value!r}") + + +def _as_count(value: object, option: str) -> int: + if isinstance(value, int) and not isinstance(value, bool) and value >= 0: + return value + raise NotificationException(f"route {option} must be an integer, 0 or more, got {value!r}") + + +def _filter_name(wanted: EventFilter) -> str: + return wanted if isinstance(wanted, str) else wanted.type + + +@dataclass(frozen=True) +class Route: + """Which events reach which sinks, and how often. + + ``sinks`` are names registered on the manager; empty means every sink. + ``types`` are the bus's filters (an event class, a type name such as + ``"task.failed"``, or a prefix such as ``"pipeline.*"``); empty means every + type. ``sources`` are exact ``event.source`` values; empty means every + source. ``dedup_seconds=0`` and ``rate_limit=0`` switch each guard off. + """ + + name: str + sinks: tuple[str, ...] = () + types: tuple[EventFilter, ...] = () + sources: tuple[str, ...] = () + min_severity: Severity = Severity.WARNING + dedup_seconds: float = _DEFAULT_DEDUP_SECONDS + rate_limit: int = 0 + rate_period: float = _DEFAULT_RATE_PERIOD + + def __post_init__(self) -> None: + if not isinstance(self.name, str) or not self.name: + raise NotificationException(f"a route needs a non-empty name, got {self.name!r}") + normalized = { + "sinks": _as_names(self.sinks, "sinks"), + "types": _as_filters(self.types), + "sources": _as_names(self.sources, "sources"), + "min_severity": _as_severity(self.min_severity), + "dedup_seconds": _as_seconds(self.dedup_seconds, "dedup_seconds"), + "rate_limit": _as_count(self.rate_limit, "rate_limit"), + "rate_period": _as_seconds(self.rate_period, "rate_period", positive=True), + } + for option, value in normalized.items(): + object.__setattr__(self, option, value) + + @classmethod + def from_mapping(cls, options: Mapping[str, Any]) -> Route: + """Build a route from a TOML table or a JSON action's arguments.""" + known = {entry.name for entry in fields(cls)} + unknown = sorted(set(options) - known) + if unknown: + raise NotificationException(f"unknown route option(s) {unknown}") + if options.get("name") is None: + raise NotificationException("a route needs a 'name'") + given = {key: value for key, value in options.items() if value is not None} + return cls(**given) + + def matches(self, event: Event) -> bool: + """Return whether ``event`` is one this route delivers.""" + if self.sources and event.source not in self.sources: + return False + return Subscription(_ignore, self.types, self.min_severity).matches(event) + + def to_dict(self) -> dict[str, Any]: + """Return a JSON-serialisable mapping; an event class appears as its type name.""" + return { + "name": self.name, + "sinks": list(self.sinks), + "types": [_filter_name(wanted) for wanted in self.types], + "sources": list(self.sources), + "min_severity": self.min_severity.value, + "dedup_seconds": self.dedup_seconds, + "rate_limit": self.rate_limit, + "rate_period": self.rate_period, + } + + +@dataclass(frozen=True) +class NotificationMessage: + """What one event looks like to a sink.""" + + subject: str + body: str + level: str + + +def message_for(event: Event) -> NotificationMessage: + """Build the subject, the body and the sink level of ``event``. + + The body lists the severity, type, source, subject, time, correlation ID and + actor, then the JSON of ``event.to_dict()``. ``critical`` is sent at the + ``error`` level, the highest one a sink accepts. + """ + severity = event.severity.value + headline = ( + " ".join(event.subject.split()) or f"reported by {event.source or 'an unnamed source'}" + ) + subject = f"[{severity.upper()}] {event.type}: {headline}" + if len(subject) > _MAX_SUBJECT: + subject = subject[: _MAX_SUBJECT - 1] + "…" + lines = [ + f"Severity: {severity}", + f"Type: {event.type}", + f"Source: {event.source or _MISSING}", + f"Subject: {event.subject or _MISSING}", + f"Time: {event.timestamp.isoformat()}", + f"Correlation ID: {event.correlation_id}", + f"Actor: {event.actor}", + ] + error = event.payload.get("error") + if error: + lines.append(f"Error: {error}") + document = json.dumps( + event.to_dict(), indent=2, sort_keys=True, ensure_ascii=False, default=repr + ) + return NotificationMessage( + subject=subject, + body="\n".join([*lines, "", "Event:", document]), + level=_LEVELS[event.severity], + ) + + +def _is_delivery_failure(event: Event) -> bool: + """Return whether ``event`` is a router's report about a failed delivery.""" + return event.source == NOTIFY_SOURCE and event.type == SystemErrorEvent.type + + +class NotificationRouter: + """Deliver events to sinks along configurable routes.""" + + def __init__( + self, + manager: NotificationManager | None = None, + bus: EventBus | None = None, + *, + clock: Callable[[], float] = time.monotonic, + ) -> None: + self._manager = manager if manager is not None else notification_manager + self._bus = bus if bus is not None else event_bus + self._clock = clock + self._lock = threading.RLock() + self._routes: dict[str, Route] = {} + self._origins: dict[str, str] = {} + self._subscription: Subscription | None = None + self._seen: dict[_DedupKey, float] = {} + self._sent: dict[_RateKey, deque[float]] = {} + self._next_prune = -math.inf + + @property + def manager(self) -> NotificationManager: + return self._manager + + @property + def bus(self) -> EventBus: + return self._bus + + @property + def active(self) -> bool: + """Whether the router is subscribed to its bus (between ``start`` and ``stop``).""" + with self._lock: + return self._subscription is not None + + def add_route(self, route: Route, *, origin: str = "") -> None: + """Add ``route``, replacing the one with the same name. + + ``origin`` records where the route came from, for :meth:`sync_routes`. + """ + _check_route(route) + with self._lock: + self._put(route, origin) + file_automation_logger.info("notify router: route %r added", route.name) + + def remove_route(self, name: str) -> bool: + """Remove the route called ``name``; return whether there was one.""" + with self._lock: + removed = self._drop(name) + if removed: + file_automation_logger.info("notify router: route %r removed", name) + return removed + + def routes(self) -> list[Route]: + """Return the routes, in the order they were added.""" + with self._lock: + return list(self._routes.values()) + + def sync_routes(self, routes: Iterable[Route], *, origin: str) -> int: + """Make ``routes`` the complete set of routes that came from ``origin``. + + Routes added earlier under the same origin and missing from ``routes`` + are removed; routes added under another origin stay. A reloaded + configuration file uses this so a deleted table stops routing. Returns + how many routes were removed. + """ + wanted = list(routes) + for route in wanted: + _check_route(route) + names = {route.name for route in wanted} + with self._lock: + stale = [ + name + for name, owner in self._origins.items() + if owner == origin and name not in names + ] + for name in stale: + self._drop(name) + for route in wanted: + self._put(route, origin) + file_automation_logger.info( + "notify router: %d route(s) from %r, %d removed", len(wanted), origin, len(stale) + ) + return len(stale) + + def start(self) -> None: + """Subscribe to the bus. Starting an active router changes nothing.""" + with self._lock: + if self._subscription is not None: + return + self._subscription = self._bus.subscribe(self.handle) + file_automation_logger.info("notify router: started") + + def stop(self) -> None: + """Unsubscribe from the bus. Stopping an inactive router changes nothing.""" + with self._lock: + subscription, self._subscription = self._subscription, None + if subscription is None: + return + self._bus.unsubscribe(subscription) + file_automation_logger.info("notify router: stopped") + + def handle(self, event: Event) -> dict[str, str]: + """Deliver ``event`` along every matching route. + + Returns one outcome per sink: ``"sent"``, ``"dedup"``, ``"rate_limited"`` + or the error as ``": "``. A sink reached by + several routes gets the event once: the first route that is allowed to + send delivers it. An event that matches no route gives an empty mapping. + """ + if _is_delivery_failure(event): + return {} + outcomes: dict[str, str] = {} + message: NotificationMessage | None = None + for route in self._matching(event): + for sink in route.sinks or self._manager.names(): + previous = outcomes.get(sink) + if previous is not None and previous not in _THROTTLED: + continue # an earlier route already tried this sink + verdict = self._admit(route, sink, event) + if verdict is not None: + outcomes.setdefault(sink, verdict) + continue + message = message or message_for(event) + outcomes[sink] = self._send(route, sink, event, message) + for sink, outcome in outcomes.items(): + if outcome in _THROTTLED: + record_notification(sink, outcome) + return outcomes + + def _matching(self, event: Event) -> list[Route]: + with self._lock: + return [route for route in self._routes.values() if route.matches(event)] + + def _admit(self, route: Route, sink: str, event: Event) -> str | None: + """Return why ``event`` must not go to ``sink`` now, or ``None`` and count it as sent.""" + now = self._clock() + key = (route.name, sink, event.type, event.source, event.subject) + with self._lock: + self._prune(now) + if route.dedup_seconds > 0 and self._seen.get(key, -math.inf) > now: + return OUTCOME_DEDUP + if route.rate_limit > 0: + times = self._sent.setdefault((route.name, sink), deque()) + while times and times[0] <= now - route.rate_period: + times.popleft() + if len(times) >= route.rate_limit: + return OUTCOME_RATE_LIMITED + times.append(now) + if route.dedup_seconds > 0: + self._seen[key] = now + route.dedup_seconds + return None + + def _send(self, route: Route, sink: str, event: Event, message: NotificationMessage) -> str: + try: + outcome = self._manager.send_to(sink, message.subject, message.body, message.level) + except NotificationException as error: + # The route names a sink the manager does not have. + outcome = describe_error(error) + record_notification(sink, OUTCOME_ERROR) + file_automation_logger.error("notify router: route %r: %s", route.name, outcome) + if outcome != OUTCOME_SENT: + self._report_failure(route, sink, event, outcome) + return outcome + + def _report_failure(self, route: Route, sink: str, event: Event, error: str) -> None: + self._bus.publish( + SystemErrorEvent( + source=NOTIFY_SOURCE, + subject=f"notification sink {sink!r} failed", + payload={ + "action": _FAILURE_ACTION, + "resource": sink, + "status": OUTCOME_ERROR, + "error": error, + "route": route.name, + "event_type": event.type, + "event_id": event.id, + }, + correlation_id=event.correlation_id, + ) + ) + + def _put(self, route: Route, origin: str) -> None: + if self._routes.get(route.name) != route: + self._forget(route.name) + self._routes[route.name] = route + self._origins[route.name] = origin + + def _drop(self, name: str) -> bool: + self._origins.pop(name, None) + self._forget(name) + return self._routes.pop(name, None) is not None + + def _forget(self, name: str) -> None: + """Drop what is remembered about the deliveries of the route ``name``.""" + for seen in [key for key in self._seen if key[0] == name]: + del self._seen[seen] + for sent in [key for key in self._sent if key[0] == name]: + del self._sent[sent] + + def _prune(self, now: float) -> None: + if len(self._seen) > _MAX_DEDUP_KEYS: + # A flood of distinct subjects: forget the oldest share of them. + for key in list(itertools.islice(self._seen, _MAX_DEDUP_KEYS // _FLOOD_SHARE)): + del self._seen[key] + if now < self._next_prune: + return + self._next_prune = now + _PRUNE_INTERVAL + for seen in [key for key, expires in self._seen.items() if expires <= now]: + del self._seen[seen] + idle = [key for key, times in self._sent.items() if not times or key[0] not in self._routes] + for sent in idle: + del self._sent[sent] + + +def _check_route(route: object) -> None: + if not isinstance(route, Route): + raise NotificationException(f"expected Route, got {type(route).__name__}") + + +notification_router: NotificationRouter = NotificationRouter() + + +def notify_route_add( + name: str, + sinks: list[str] | None = None, + types: list[str] | None = None, + sources: list[str] | None = None, + min_severity: str = Severity.WARNING.value, + **throttle: Any, +) -> dict[str, Any]: + """Add or replace a route on the process-wide router, and start routing. + + ``throttle`` takes ``dedup_seconds``, ``rate_limit`` and ``rate_period``. + Returns the route as stored. + """ + route = Route.from_mapping( + { + "name": name, + "sinks": sinks, + "types": types, + "sources": sources, + "min_severity": min_severity, + **throttle, + } + ) + notification_router.add_route(route) + notification_router.start() + return route.to_dict() + + +def notify_route_remove(name: str) -> bool: + """Remove a route from the process-wide router; stop routing when none is left.""" + removed = notification_router.remove_route(name) + if removed and not notification_router.routes(): + notification_router.stop() + return removed + + +def notify_route_list() -> list[dict[str, Any]]: + """Return every route of the process-wide router.""" + return [route.to_dict() for route in notification_router.routes()] diff --git a/docs/source/API/api_index.rst b/docs/source/API/api_index.rst index d7f662c..e90ef8e 100644 --- a/docs/source/API/api_index.rst +++ b/docs/source/API/api_index.rst @@ -215,3 +215,17 @@ actions. :caption: File Integrity Monitoring integrity + +.. _api-audit: + +Chapter P — Audit Trail +======================= + +Audit schema v2: the record, the stores, the trail and the ``FA_audit_*`` +actions. + +.. toctree:: + :maxdepth: 2 + :caption: Audit Trail + + audit diff --git a/docs/source/API/audit.rst b/docs/source/API/audit.rst new file mode 100644 index 0000000..8b5397a --- /dev/null +++ b/docs/source/API/audit.rst @@ -0,0 +1,19 @@ +Audit +===== + +Audit schema v2. The v1 ``AuditLog`` is documented with the core modules. + +.. automodule:: automation_file.audit.record + :members: + +.. automodule:: automation_file.audit.store + :members: + +.. automodule:: automation_file.audit.sqlite_store + :members: + +.. automodule:: automation_file.audit.trail + :members: + +.. automodule:: automation_file.audit.actions + :members: diff --git a/docs/source/API/notify.rst b/docs/source/API/notify.rst index 066c645..44e879d 100644 --- a/docs/source/API/notify.rst +++ b/docs/source/API/notify.rst @@ -6,3 +6,6 @@ Notifications .. automodule:: automation_file.notify.manager :members: + +.. automodule:: automation_file.notify.router + :members: diff --git a/docs/source/Eng/eng_index.rst b/docs/source/Eng/eng_index.rst index 131fc43..edebc93 100644 --- a/docs/source/Eng/eng_index.rst +++ b/docs/source/Eng/eng_index.rst @@ -288,3 +288,19 @@ schema, the four modes, alerts, and opt-in remediation. :caption: File Integrity Monitoring usage/integrity + +.. _eng-audit: + +Chapter 19 — Audit Trail +======================== + +Audit schema v2: one record per event and per storage operation, with the +actor, the resource, the backend, the result and a correlation ID; the +stores, the search filters, the migration from the v1 ``AuditLog`` and the +operational metrics. Notification routes are in :doc:`usage/notifications`. + +.. toctree:: + :maxdepth: 2 + :caption: Audit Trail + + usage/audit diff --git a/docs/source/Eng/usage/audit.rst b/docs/source/Eng/usage/audit.rst new file mode 100644 index 0000000..1a7f764 --- /dev/null +++ b/docs/source/Eng/usage/audit.rst @@ -0,0 +1,309 @@ +Audit trail +=========== + +The audit trail answers one question about everything the library does: **who** +did **what**, **when**, against **which resource**, using **which backend**, and +with **what result**. It is audit schema v2. Nothing writes an audit row itself: +components publish :doc:`events `, the storage layer reports its +operations, and an :class:`~automation_file.audit.trail.AuditTrail` turns both +into :class:`~automation_file.audit.record.AuditRecord` rows in an +:class:`~automation_file.audit.store.AuditStore`. + +The v1 :class:`~automation_file.core.audit.AuditLog` stays as it is; see +`Migrating from v1`_. + +Minimal example +--------------- + +.. code-block:: python + + from automation_file import audit_search, configure_audit + + configure_audit("audit.sqlite") # creates the database, starts recording + + ... # run pipelines, copy files, publish events + + for entry in audit_search(status="error", limit=20): + print(entry["timestamp"], entry["actor"], entry["action"], entry["resource"], + entry["error"]) + +``configure_audit`` takes the path of a SQLite database or a ready store, +points the process-wide ``audit_trail`` at it and starts it. Until it is called +the trail has no store and records nothing. ``audit_search`` returns plain +dictionaries, newest first. + +Production example +------------------ + +.. code-block:: python + + from automation_file import ( + File, PipelineCompleted, PipelineStarted, actor_scope, audit_search, + audit_trail, configure_audit, correlation_scope, emit, + ) + + configure_audit("/var/lib/automation_file/audit.sqlite") + + with actor_scope("scheduler"), correlation_scope() as run_id: + emit(PipelineStarted(source="pipeline", subject="daily-report started", + payload={"pipeline": "daily-report", "run_id": run_id})) + File("s3://reports/q1.csv").copy_to("local:///backup/q1.csv") # recorded + audit_trail.record("approve", resource="s3://reports/q1.csv", backend="s3", + source="review", metadata={"ticket": "OPS-12"}) + emit(PipelineCompleted(source="pipeline", subject="daily-report completed", + payload={"pipeline": "daily-report", "duration_ms": 812.0})) + + audit_search(correlation_id=run_id) # the whole run, newest first + audit_search(resource_prefix="s3://reports/", since="2026-10-01T00:00:00+00:00") + audit_trail.count(actor="scheduler", status="error") + audit_trail.purge(older_than_seconds=90 * 24 * 3600) # keep 90 days + +Open the store once at start-up and let it fail fast: ``configure_audit`` +raises :class:`~automation_file.AuditException` when the database cannot be +opened. Run the purge from a scheduled job. Everything inside the two scopes +carries the same correlation ID and actor, so one filter returns the run. + +A trail of your own, on a private bus or with another store: + +.. code-block:: python + + from automation_file import AuditTrail, EventBus, SQLiteAuditStore + + trail = AuditTrail(SQLiteAuditStore("tenant-a.sqlite"), bus=EventBus()) + trail.start() + ... + trail.stop() + trail.store.close() + +The record +---------- + +An ``AuditRecord`` is frozen and JSON-friendly (``to_dict()`` / +``AuditRecord.from_dict()``). + +.. list-table:: + :header-rows: 1 + :widths: 22 78 + + * - Field + - Meaning + * - ``id`` + - A unique ID. The record of an event has the event's ID. + * - ``timestamp`` + - When it happened, as an aware UTC ``datetime``. A time without a time + zone is rejected. + * - ``actor`` + - **Who**: the actor of the enclosing ``actor_scope``, or the user the + process runs as. + * - ``source`` + - The component that reported: ``pipeline``, ``storage``, ``scheduler``, + ``notify``, ... + * - ``pipeline``, ``task`` + - The pipeline and the task it belongs to, when there is one. + * - ``action`` + - **What**: the event type (``task.failed``) or the storage operation + (``upload``, ``download``, ``read``, ``delete``, ``mkdir``, ``copy``, + ``move``). + * - ``resource`` + - **Which resource**: a storage URI or another target. + * - ``backend`` + - **Which backend**: the storage scheme (``s3``, ``local``, ...). + * - ``status`` + - **What result**: ``ok``, ``warning``, ``error``, or the word the emitter + chose. + * - ``duration_ms`` + - How long it took, when that is known. + * - ``error`` + - ``": "`` when it failed. + * - ``metadata`` + - Everything else, as JSON values. What JSON cannot hold is kept as its + ``repr``. + * - ``correlation_id`` + - The run it belongs to; ``None`` outside any ``correlation_scope``. + +What is recorded +---------------- + +**Events.** Every event on the bus becomes one record. ``source``, ``actor``, +``timestamp`` and ``correlation_id`` are the event's; ``action`` is the event +type; ``pipeline``, ``task``, ``resource``, ``backend``, ``status``, +``duration_ms`` and ``error`` come from the payload keys of the same name. The +event's ``subject`` and ``severity`` and every other payload key go under +``metadata``. When the payload has no ``status``, the severity decides: ``ok`` +for info, ``warning`` for warning, ``error`` for error and critical. + +**Storage operations.** Every upload, download, read, delete, mkdir, copy and +move becomes one record with ``source="storage"``, the operation as the +``action``, the URI as the ``resource`` and the scheme as the ``backend``, +whether it succeeded or not. A copy or a move is one record; its origin is +``metadata["source_uri"]``. + +**A failed storage operation is recorded once.** The storage layer reports the +operation, and the storage bridge also publishes a ``storage.error`` event for +it. The trail keeps the operation and skips that event. + +**By hand.** ``audit_trail.record(action, **fields)`` appends one record; the +actor and the correlation ID default to the current scopes. + +Filters +------- + +``search`` and ``count`` take the same filters by name; ``search`` returns the +newest records first. + +.. list-table:: + :header-rows: 1 + :widths: 28 72 + + * - Filter + - Matches + * - ``since``, ``until`` + - The time range: ``since`` is included, ``until`` is excluded. An aware + ``datetime``, an ISO 8601 string with an offset or a ``Z``, or seconds + since the epoch. + * - ``actor``, ``source``, ``pipeline``, ``task``, ``action``, ``backend``, + ``status``, ``correlation_id`` + - The field is exactly this value. + * - ``resource_prefix`` + - The resource starts with this text. + * - ``text`` + - The text occurs in the action, resource, error, actor, source, pipeline, + task, backend or the JSON of the metadata. + * - ``limit``, ``offset`` + - Paging. ``limit`` defaults to 100 and may not exceed 10 000. ``count`` + ignores both. + +``resource_prefix`` and ``text`` take the text literally -- ``%`` and ``_`` are +ordinary characters -- and ignore the case of ASCII letters. A filter given as +``None`` does not restrict the search. An unknown filter name or a value of the +wrong kind raises ``AuditException`` instead of quietly returning everything. + +The store interface +------------------- + +``AuditStore`` is the interface a store implements: the SQLite store today, a +PostgreSQL or remote store later. + +.. code-block:: python + + from automation_file import AuditQuery, AuditRecord, AuditStore + + class MyStore(AuditStore): + def append(self, record: AuditRecord) -> None: ... + def search(self, **filters) -> list[AuditRecord]: + query = AuditQuery.from_filters(filters) # validated filters + ... + def count(self, **filters) -> int: ... + def purge(self, older_than_seconds: float) -> int: ... + def close(self) -> None: ... + +A store is append-only, safe to share between threads, answers ``search`` +newest first, rejects a record whose ``id`` it already holds, and raises +``AuditException`` for every failure. ``AuditQuery.from_filters`` validates the +filters the same way for every store. + +``SQLiteAuditStore(path)`` keeps the records in one table with a second table +that states the schema version (``2``), and indexes on the timestamp, the +correlation ID, the resource and the action. Every value reaches SQLite as a +bound parameter and the ``LIKE`` wildcards in a filter are escaped. One +connection is shared by all threads behind a lock, in WAL mode, so another +process can read while this one writes. ``MemoryAuditStore()`` keeps the +records in the process and is meant for tests. + +Migrating from v1 +----------------- + +.. code-block:: python + + from automation_file import SQLiteAuditStore + + store = SQLiteAuditStore("audit.sqlite") + store.import_v1("old-audit.sqlite3") # returns how many rows were new + +Each v1 row becomes a record with ``source="audit.v1"`` and the actor +``unknown``; its payload and result go under ``metadata``. The v1 database is +only read, and importing it again adds nothing. + +Actions +------- + +``register_audit_ops(registry)`` wires four actions; they act on the +process-wide trail. + +.. code-block:: json + + [ + ["FA_audit_configure", {"db_path": "/var/lib/automation_file/audit.sqlite"}], + ["FA_audit_search", {"status": "error", "since": "2026-10-01T00:00:00+00:00", "limit": 50}], + ["FA_audit_count", {"correlation_id": "4f0c2b6e9d5a4c1f8a7b3e2d1c0f9a8b"}], + ["FA_audit_purge", {"older_than_seconds": 7776000}] + ] + +``FA_audit_search`` and ``FA_audit_count`` take the filters above; +``FA_audit_search`` returns each record as its ``to_dict()``. + +A client that may purge the trail can erase its own tracks. On a TCP or HTTP +action server pass an :class:`~automation_file.ActionACL` that denies +``FA_audit_purge`` and ``FA_audit_configure``, and on the MCP server leave them +out of ``--allowed-actions``, unless remote clients are meant to manage the +trail. + +When something goes wrong +------------------------- + +- **A record cannot be written.** The failure is logged + (``audit trail: cannot write the record of ...``) and the record is dropped. + It is never raised into the code that is being audited: a full disk must not + stop a pipeline. Watch the log for that line, and alert on it. +- **The database cannot be opened.** ``configure_audit`` and + ``SQLiteAuditStore`` raise ``AuditException``. Open the store at start-up so + this is seen at once. +- **The database was written by a newer version.** The store refuses to open + it rather than guess at its layout. +- **Durability.** The SQLite store commits every record. In WAL mode a crash of + the process loses nothing; a power failure may lose the last moments. Keep + the database on a local disk: WAL does not work on a network share. +- **Growth.** Nothing is deleted on its own. Call ``purge`` or + ``FA_audit_purge`` on a schedule with your retention period. +- **Searching before configuring.** ``audit_search`` raises ``AuditException`` + when no store is configured, so an empty answer always means "no match". +- **Threads.** A scope does not follow work into another thread by itself; the + code that fans work out re-enters ``correlation_scope`` and ``actor_scope`` + there, or its records carry no correlation ID. +- **Secrets.** Payloads are stored as they are. Do not put a credential in an + event payload or in ``metadata``. + +Operational metrics +------------------- + +The same events and storage operations feed Prometheus counters, next to the +per-action metrics ``automation_file_actions_total`` and +``automation_file_action_duration_seconds``: + +.. code-block:: python + + from automation_file import install_operational_metrics, start_metrics_server + + install_operational_metrics() # once; calling it again changes nothing + start_metrics_server(host="127.0.0.1", port=9945) + +.. list-table:: + :header-rows: 1 + :widths: 62 38 + + * - Metric + - Labels + * - ``automation_file_events_total`` + - ``type``, ``severity`` + * - ``automation_file_notifications_total`` + - ``sink``, ``outcome`` (``sent``, ``dedup``, ``rate_limited``, ``error``) + * - ``automation_file_storage_operations_total`` + - ``operation``, ``backend``, ``status`` + * - ``automation_file_storage_operation_duration_seconds`` (histogram) + - ``operation``, ``backend`` + +``install_operational_metrics()`` subscribes to the event bus and to the +storage observers. The notification counter needs no installation: the +notification manager and the router count every delivery themselves. No label +ever carries a path, a URI or a correlation ID, and each label keeps at most +100 distinct values; further values are counted as ``other``. diff --git a/docs/source/Eng/usage/config.rst b/docs/source/Eng/usage/config.rst index 10b6553..9282733 100644 --- a/docs/source/Eng/usage/config.rst +++ b/docs/source/Eng/usage/config.rst @@ -32,10 +32,14 @@ root (Docker / K8s style): .. code-block:: python - from automation_file import AutomationConfig, notification_manager + from automation_file import AutomationConfig, notification_manager, notification_router config = AutomationConfig.load("automation_file.toml") - config.apply_to(notification_manager) + config.apply_to(notification_manager, notification_router) + +``apply_to`` registers the sinks. Given the router as well, it also applies the +``[[notify.routes]]`` tables and starts the router when the file declares a route +(:doc:`notifications`); without it the routes are validated and not applied. Unresolved ``${…}`` references raise :class:`~automation_file.SecretNotFoundException` rather than silently diff --git a/docs/source/Eng/usage/integrity.rst b/docs/source/Eng/usage/integrity.rst index c970de7..ea994ae 100644 --- a/docs/source/Eng/usage/integrity.rst +++ b/docs/source/Eng/usage/integrity.rst @@ -456,9 +456,11 @@ unless ``alert_on_extra=True``, and a rename appears as ``missing`` plus The notification goes where it went before: through the ``manager`` you pass, or through the process-wide ``notification_manager`` when you pass none. One thing was added: every drift, additions included, is also published as an -``IntegrityViolation`` event. If you deliver that event to your sinks yourself -(a subscriber, or a notification route), pass ``notify=False`` so one drift is -not announced twice. +``IntegrityViolation`` event. While the notification router is active +(:doc:`notifications`), its routes deliver that event and the process-wide +manager is not notified directly, so one drift is not announced twice; a +``manager`` you pass is always notified. ``notify=False`` turns the direct +notification off altogether. When something fails -------------------- diff --git a/docs/source/Eng/usage/notifications.rst b/docs/source/Eng/usage/notifications.rst index 278d6a0..57cd1ed 100644 --- a/docs/source/Eng/usage/notifications.rst +++ b/docs/source/Eng/usage/notifications.rst @@ -27,3 +27,191 @@ the fanout :class:`~automation_file.NotificationManager` handles: Scheduler and trigger dispatchers auto-notify on failure at ``level="error"`` — registering a sink is all that's needed to get production alerts. JSON forms: ``FA_notify_send`` / ``FA_notify_list``. + +Routing events +-------------- + +Notifications can be driven by :doc:`events ` instead of by modules +calling a sink: a component publishes an event, and the +:class:`~automation_file.notify.router.NotificationRouter` decides which sinks +hear about it. + +.. code-block:: python + + from automation_file import ( + Route, Severity, SlackSink, notification_manager, notification_router, + ) + + notification_manager.register(SlackSink("https://hooks.slack.com/services/T/B/X", + name="team-alerts")) + notification_router.add_route(Route( + "pipeline-failures", + sinks=("team-alerts",), + types=("pipeline.*", "task.failed"), + min_severity=Severity.ERROR, + dedup_seconds=600, + rate_limit=10, + rate_period=60, + )) + notification_router.start() # subscribe on the event bus + +The router is inactive until ``start()``; ``stop()`` unsubscribes it and +``active`` tells which state it is in. ``add_route`` replaces a route with the +same name, ``remove_route(name)`` drops one and ``routes()`` lists them. A +private router is ``NotificationRouter(manager, bus)``. + +Routes +~~~~~~ + +.. list-table:: + :header-rows: 1 + :widths: 22 18 60 + + * - Field + - Default + - Meaning + * - ``name`` + - required + - Identifies the route. + * - ``sinks`` + - ``()`` + - Names registered on the ``NotificationManager``. Empty means every sink. + * - ``types`` + - ``()`` + - The bus's filters: an event class, a type name (``"task.failed"``) or a + prefix (``"pipeline.*"``). Empty means every type. + * - ``sources`` + - ``()`` + - Exact ``event.source`` values. Empty means every source. + * - ``min_severity`` + - ``Severity.WARNING`` + - The lowest severity the route delivers. + * - ``dedup_seconds`` + - ``300.0`` + - The deduplication window. ``0`` switches it off. + * - ``rate_limit`` + - ``0`` + - Messages allowed per ``rate_period``. ``0`` means no limit. + * - ``rate_period`` + - ``60.0`` + - The length of the rate-limit window, in seconds. + +A sink reached by several routes gets an event once: the first route that is +allowed to send delivers it. An event that matches no route is not delivered. + +What a sink receives +~~~~~~~~~~~~~~~~~~~~ + +The message is built from the event. The subject reads +``[ERROR] task.failed: load failed``; the body lists the severity, the type, +the source, the subject, the time, the correlation ID and the actor, then the +JSON of ``event.to_dict()``. The severity becomes a level the sinks accept: +``info``, ``warning`` and ``error`` keep their name, and ``critical`` is sent +as ``error``. + +Deduplication and rate limiting +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Both are kept per route and per sink. + +- **Deduplication** — a repeat of the same event (same type, source and + subject) within ``dedup_seconds`` of the first one is dropped; the payload + is not compared. A failed attempt counts too, so a dead sink is not tried + again for every repeat. +- **Rate limiting** — at most ``rate_limit`` messages per ``rate_period``. + An event held back by the limit is not remembered as sent, so its next + occurrence can still go out; a duplicate does not use up the limit. + +``notification_router.handle(event)`` returns one outcome per sink: ``sent``, +``dedup``, ``rate_limited``, or the error as +``": "``. + +Delivery happens in the thread that published the event. Keep sink timeouts +short; the two guards bound how often a slow sink is called. + +Failures +~~~~~~~~ + +One sink failing never affects another. Each failure is logged and published +as a ``SystemErrorEvent`` with ``source="notify"``; its payload names the sink +(``resource``), the ``route``, the ``error``, and the ``event_type`` and +``event_id`` of the event that could not be delivered. The router never routes +these events, so a broken sink cannot feed a loop; subscribe to the bus, or +search the :doc:`audit trail `, to see them. URLs in an error text are +cut down to their host, because a webhook URL or a bot token is a secret. + +Routes in ``automation_file.toml`` +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. code-block:: toml + + [[notify.sinks]] + type = "slack" + name = "team-alerts" + webhook_url = "${env:SLACK_WEBHOOK}" + + [[notify.routes]] + name = "pipeline-failures" + sinks = ["team-alerts"] + types = ["pipeline.*", "task.failed"] + sources = ["pipeline"] + min_severity = "error" + dedup_seconds = 600 + rate_limit = 10 + rate_period = 60 + +.. code-block:: python + + from automation_file import ( + AutomationConfig, ConfigWatcher, notification_manager, notification_router, + ) + + def apply(config): + config.apply_to(notification_manager, notification_router) + + watcher = ConfigWatcher("automation_file.toml", apply) + apply(watcher.start()) # load now, and again whenever the file changes + +``apply_to(manager, router)`` registers the sinks and makes the +``[[notify.routes]]`` tables the router's configured routes: a table removed +from the file stops routing at the next reload, while routes added in code +stay. The router is started when the file declares a route, and stopped when a +reload takes its last route away. A route may only name a sink declared in the file or +already registered on the manager; an unknown sink, an unknown severity, an +unknown option or a name used twice raises +:class:`~automation_file.ConfigException` before anything is changed, so a bad +reload leaves the previous routes in place. Without the ``router`` argument +the routes are not applied. + +Actions +~~~~~~~ + +.. code-block:: json + + [ + ["FA_notify_route_add", {"name": "pipeline-failures", "sinks": ["team-alerts"], + "types": ["pipeline.*", "task.failed"], "min_severity": "error", + "dedup_seconds": 600, "rate_limit": 10, "rate_period": 60}], + ["FA_notify_route_list"], + ["FA_notify_route_remove", {"name": "pipeline-failures"}] + ] + +They act on the process-wide router. ``FA_notify_route_add`` starts it, and +``FA_notify_route_remove`` stops it when the last route is gone. + +``notify_on_failure`` +~~~~~~~~~~~~~~~~~~~~~ + +The scheduler and the triggers still call +``notify_on_failure(context, error)``. It now always publishes an event: a +``SchedulerError`` when the context is a scheduler job (``scheduler[nightly]``), +a ``SystemErrorEvent`` otherwise. Then: + +- **router active** — the router delivers the event along its routes, and + nothing else is sent, so nobody is notified twice; +- **router inactive** — the ``error``-level message also goes straight to + every registered sink, exactly as before, so nobody stops being notified. + +With the router active, the routes decide: a failure that no route matches is +not delivered. A route such as ``Route("failures", types=("scheduler.error", +"system.error"))`` keeps those alerts coming. diff --git a/docs/source/Zh-CN/usage/audit.rst b/docs/source/Zh-CN/usage/audit.rst new file mode 100644 index 0000000..08bdd95 --- /dev/null +++ b/docs/source/Zh-CN/usage/audit.rst @@ -0,0 +1,287 @@ +审计轨迹 +================ + +审计轨迹针对库所做的每一件事回答同一个问题：**谁** 在 **什么时候** 做了 +**什么**，对象是 **哪个资源**，使用 **哪个后端**，以及 **结果如何**。这是审计 +模式 v2。没有任何组件会自行写入审计行：组件发布 :doc:`事件 `， +存储层报告它的操作，再由 :class:`~automation_file.audit.trail.AuditTrail` +把两者转成 :class:`~automation_file.audit.record.AuditRecord`，写进 +:class:`~automation_file.audit.store.AuditStore`。 + +v1 的 :class:`~automation_file.core.audit.AuditLog` 保持原样；请见 +`从 v1 迁移`_。 + +最小示例 +---------------- + +.. code-block:: python + + from automation_file import audit_search, configure_audit + + configure_audit("audit.sqlite") # 创建数据库并开始记录 + + ... # 运行管道、复制文件、发布事件 + + for entry in audit_search(status="error", limit=20): + print(entry["timestamp"], entry["actor"], entry["action"], entry["resource"], + entry["error"]) + +``configure_audit`` 接受 SQLite 数据库的路径或一个现成的存储库，把进程级的 +``audit_trail`` 指向它并启动。在调用之前，轨迹没有存储库，也不会记录任何内容。 +``audit_search`` 返回普通的字典，最新的在前。 + +生产环境示例 +------------------------ + +.. code-block:: python + + from automation_file import ( + File, PipelineCompleted, PipelineStarted, actor_scope, audit_search, + audit_trail, configure_audit, correlation_scope, emit, + ) + + configure_audit("/var/lib/automation_file/audit.sqlite") + + with actor_scope("scheduler"), correlation_scope() as run_id: + emit(PipelineStarted(source="pipeline", subject="daily-report started", + payload={"pipeline": "daily-report", "run_id": run_id})) + File("s3://reports/q1.csv").copy_to("local:///backup/q1.csv") # 会被记录 + audit_trail.record("approve", resource="s3://reports/q1.csv", backend="s3", + source="review", metadata={"ticket": "OPS-12"}) + emit(PipelineCompleted(source="pipeline", subject="daily-report completed", + payload={"pipeline": "daily-report", "duration_ms": 812.0})) + + audit_search(correlation_id=run_id) # 整次运行，最新的在前 + audit_search(resource_prefix="s3://reports/", since="2026-10-01T00:00:00+00:00") + audit_trail.count(actor="scheduler", status="error") + audit_trail.purge(older_than_seconds=90 * 24 * 3600) # 保留 90 天 + +请在启动时就打开存储库，让问题立刻暴露：数据库无法打开时，``configure_audit`` +会抛出 :class:`~automation_file.AuditException`。清理旧记录的工作请交给调度任务。 +两个范围内的所有事物都带有相同的关联 ID 与 actor，因此只要一个筛选条件就能取回 +整次运行。 + +自行创建轨迹，使用私有的总线或其他存储库： + +.. code-block:: python + + from automation_file import AuditTrail, EventBus, SQLiteAuditStore + + trail = AuditTrail(SQLiteAuditStore("tenant-a.sqlite"), bus=EventBus()) + trail.start() + ... + trail.stop() + trail.store.close() + +记录 +-------- + +``AuditRecord`` 不可变，并且可以直接转成 JSON（``to_dict()`` / +``AuditRecord.from_dict()``）。 + +.. list-table:: + :header-rows: 1 + :widths: 22 78 + + * - 字段 + - 含义 + * - ``id`` + - 唯一 ID。由事件产生的记录沿用该事件的 ID。 + * - ``timestamp`` + - 发生的时间，为带时区的 UTC ``datetime``。不含时区的时间会被拒绝。 + * - ``actor`` + - **谁**：外层 ``actor_scope`` 的 actor，或运行进程的用户。 + * - ``source`` + - 报告的组件：``pipeline``、``storage``、``scheduler``、``notify`` 等。 + * - ``pipeline``、``task`` + - 所属的管道与任务（如果有的话）。 + * - ``action`` + - **做了什么**：事件的 type（``task.failed``）或存储操作（``upload``、 + ``download``、``read``、``delete``、``mkdir``、``copy``、 + ``move``）。 + * - ``resource`` + - **哪个资源**：存储 URI 或其他对象。 + * - ``backend`` + - **哪个后端**：存储的 scheme（``s3``、``local`` 等）。 + * - ``status`` + - **结果如何**：``ok``、``warning``、``error``，或发出者自行选用的词。 + * - ``duration_ms`` + - 花费的时间（已知时）。 + * - ``error`` + - 失败时为 ``": "``。 + * - ``metadata`` + - 其余的所有信息，以 JSON 值保存。JSON 无法表示的值会保存为它的 ``repr``。 + * - ``correlation_id`` + - 所属的那次运行；在任何 ``correlation_scope`` 之外为 ``None``。 + +记录哪些内容 +------------------------ + +**事件。** 总线上的每个事件都会变成一条记录。``source``、``actor``、 +``timestamp`` 与 ``correlation_id`` 取自事件；``action`` 是事件的 type； +``pipeline``、``task``、``resource``、``backend``、``status``、 +``duration_ms`` 与 ``error`` 取自 payload 中同名的键。事件的 ``subject``、 +``severity`` 以及 payload 的其他键都放在 ``metadata`` 之下。payload 没有 +``status`` 时由严重程度决定：info 为 ``ok``，warning 为 ``warning``，error 与 +critical 为 ``error``。 + +**存储操作。** 每一次上传、下载、读取、删除、创建目录、复制与移动都会变成一条 +记录：``source="storage"``，``action`` 是操作名称，``resource`` 是 URI， +``backend`` 是 scheme，无论成功还是失败。复制或移动算作一条记录，其来源位于 +``metadata["source_uri"]``。 + +**失败的存储操作只记录一次。** 存储层会报告该操作，存储桥接器也会为它发布一个 +``storage.error`` 事件。轨迹保留操作本身，跳过那个事件。 + +**手动记录。** ``audit_trail.record(action, **fields)`` 会追加一条记录；actor 与 +关联 ID 默认取自当前的范围。 + +筛选条件 +---------------- + +``search`` 与 ``count`` 接受相同的具名筛选条件；``search`` 返回的记录最新的在前。 + +.. list-table:: + :header-rows: 1 + :widths: 28 72 + + * - 筛选条件 + - 匹配的记录 + * - ``since``、``until`` + - 时间范围：包含 ``since``，不包含 ``until``。可以是带时区的 ``datetime``、 + 带有时差或 ``Z`` 的 ISO 8601 字符串，或自 epoch 起算的秒数。 + * - ``actor``、``source``、``pipeline``、``task``、``action``、 + ``backend``、``status``、``correlation_id`` + - 字段与此值完全相同。 + * - ``resource_prefix`` + - 资源以此文本开头。 + * - ``text`` + - 此文本出现在 action、resource、error、actor、source、pipeline、task、 + backend 或 metadata 的 JSON 之中。 + * - ``limit``、``offset`` + - 分页。``limit`` 默认为 100，且不得超过 10 000。``count`` 会忽略这两者。 + +``resource_prefix`` 与 ``text`` 会把文本当成字面值（``%`` 与 ``_`` 都是普通 +字符），并忽略 ASCII 字母的大小写。值为 ``None`` 的筛选条件不会限制搜索。未知的 +筛选条件名称或类型错误的值会抛出 ``AuditException``，而不是悄悄返回所有记录。 + +存储接口 +---------------- + +``AuditStore`` 是存储库要实现的接口：目前是 SQLite 存储库，日后可以是 +PostgreSQL 或远程存储库。 + +.. code-block:: python + + from automation_file import AuditQuery, AuditRecord, AuditStore + + class MyStore(AuditStore): + def append(self, record: AuditRecord) -> None: ... + def search(self, **filters) -> list[AuditRecord]: + query = AuditQuery.from_filters(filters) # 校验过的筛选条件 + ... + def count(self, **filters) -> int: ... + def purge(self, older_than_seconds: float) -> int: ... + def close(self) -> None: ... + +存储库只能追加、可以在线程之间共用、``search`` 以最新的在前返回、拒绝 ``id`` +已存在的记录，并且所有失败都以 ``AuditException`` 抛出。 +``AuditQuery.from_filters`` 以相同的方式为每一种存储库校验筛选条件。 + +``SQLiteAuditStore(path)`` 把记录保存在一张表中，另有一张表记载模式版本 +（``2``），并在时间戳、关联 ID、资源与 action 上建立索引。所有的值都以绑定参数 +传给 SQLite，筛选条件中的 ``LIKE`` 通配符也会被转义。所有线程通过一把锁共用同一条 +连接，并使用 WAL 模式，因此另一个进程可以在这个进程写入时读取。 +``MemoryAuditStore()`` 把记录保存在进程内，供测试使用。 + +从 v1 迁移 +-------------------- + +.. code-block:: python + + from automation_file import SQLiteAuditStore + + store = SQLiteAuditStore("audit.sqlite") + store.import_v1("old-audit.sqlite3") # 返回新增的行数 + +每一条 v1 行都会变成一条 ``source="audit.v1"``、actor 为 ``unknown`` 的记录； +它的 payload 与 result 放在 ``metadata`` 之下。v1 数据库只会被读取，重复导入不会 +新增任何记录。 + +动作 +-------- + +``register_audit_ops(registry)`` 会注册四个动作；它们作用于进程级的轨迹。 + +.. code-block:: json + + [ + ["FA_audit_configure", {"db_path": "/var/lib/automation_file/audit.sqlite"}], + ["FA_audit_search", {"status": "error", "since": "2026-10-01T00:00:00+00:00", "limit": 50}], + ["FA_audit_count", {"correlation_id": "4f0c2b6e9d5a4c1f8a7b3e2d1c0f9a8b"}], + ["FA_audit_purge", {"older_than_seconds": 7776000}] + ] + +``FA_audit_search`` 与 ``FA_audit_count`` 接受上述的筛选条件； +``FA_audit_search`` 以 ``to_dict()`` 的形式返回每一条记录。 + +能够清除轨迹的客户端，就能抹去自己的痕迹。除非远程客户端本来就该管理轨迹，否则在 +TCP 或 HTTP 动作服务器上请传入拒绝 ``FA_audit_purge`` 与 ``FA_audit_configure`` 的 +:class:`~automation_file.ActionACL`，在 MCP 服务器上则不要把它们列入 +``--allowed-actions``。 + +出现问题时 +-------------------- + +- **记录无法写入。** 失败会被记录到日志 + （``audit trail: cannot write the record of ...``），该条记录则被丢弃。它绝对 + 不会抛进被审计的代码：磁盘已满不应该让管道停摆。请监控日志中的这一行并设置 + 告警。 +- **数据库无法打开。** ``configure_audit`` 与 ``SQLiteAuditStore`` 会抛出 + ``AuditException``。请在启动时打开存储库，才能立刻发现问题。 +- **数据库是由较新的版本写入的。** 存储库会拒绝打开，而不是猜测它的结构。 +- **持久性。** SQLite 存储库每写入一条记录就提交一次。在 WAL 模式下，进程崩溃 + 不会丢失任何数据；断电则可能丢失最后一小段时间的记录。请把数据库放在本地 + 磁盘：WAL 无法在网络共享盘上工作。 +- **增长。** 没有任何记录会自动删除。请按照保留期限，定期调用 ``purge`` 或 + ``FA_audit_purge``。 +- **尚未配置就搜索。** 没有配置存储库时，``audit_search`` 会抛出 + ``AuditException``，因此空的结果永远代表“没有匹配的记录”。 +- **线程。** 范围不会自动跟着工作进入另一个线程；把工作分派出去的代码要在那里 + 重新进入 ``correlation_scope`` 与 ``actor_scope``，否则它的记录不会带有关联 + ID。 +- **机密。** payload 会原样保存。请不要把凭据放进事件的 payload 或 ``metadata``。 + +运维指标 +---------------- + +同一批事件与存储操作也会送入 Prometheus 计数器，与每个动作的指标 +``automation_file_actions_total``、``automation_file_action_duration_seconds`` +并列： + +.. code-block:: python + + from automation_file import install_operational_metrics, start_metrics_server + + install_operational_metrics() # 一次即可；重复调用不会有任何改变 + start_metrics_server(host="127.0.0.1", port=9945) + +.. list-table:: + :header-rows: 1 + :widths: 62 38 + + * - 指标 + - 标签 + * - ``automation_file_events_total`` + - ``type``、``severity`` + * - ``automation_file_notifications_total`` + - ``sink``、``outcome`` （``sent``、``dedup``、``rate_limited``、 + ``error``） + * - ``automation_file_storage_operations_total`` + - ``operation``、``backend``、``status`` + * - ``automation_file_storage_operation_duration_seconds`` （直方图） + - ``operation``、``backend`` + +``install_operational_metrics()`` 会订阅事件总线与存储观察者。通知计数器不需要 +安装：通知管理器与路由器会自行统计每一次投递。标签绝对不会带有路径、URI 或关联 +ID，而且每个标签最多保留 100 个不同的值；超出的值会计入 ``other``。 diff --git a/docs/source/Zh-CN/usage/config.rst b/docs/source/Zh-CN/usage/config.rst index 72d715e..509e567 100644 --- a/docs/source/Zh-CN/usage/config.rst +++ b/docs/source/Zh-CN/usage/config.rst @@ -31,10 +31,14 @@ .. code-block:: python - from automation_file import AutomationConfig, notification_manager + from automation_file import AutomationConfig, notification_manager, notification_router config = AutomationConfig.load("automation_file.toml") - config.apply_to(notification_manager) + config.apply_to(notification_manager, notification_router) + +``apply_to`` 会注册各个 sink。同时传入路由器时，也会应用 ``[[notify.routes]]`` +表格，并在文件声明了路由时启动路由器（见 :doc:`notifications`）；没有传入时， +路由只会被验证而不会应用。 未解析的 ``${…}`` 引用会抛出 :class:`~automation_file.SecretNotFoundException`， diff --git a/docs/source/Zh-CN/usage/integrity.rst b/docs/source/Zh-CN/usage/integrity.rst index afc481c..23a6bee 100644 --- a/docs/source/Zh-CN/usage/integrity.rst +++ b/docs/source/Zh-CN/usage/integrity.rst @@ -422,8 +422,9 @@ TCP 或 HTTP 动作服务器上请传入 ``ActionACL``，在 MCP 服务器上请 通知的去向与以往相同：通过你传入的 ``manager`` 发送，没有传入时则使用整个进程共用的 ``notification_manager``。新增的只有一点：每一次偏移（包含新增）也都会以 -``IntegrityViolation`` 事件的形式发布。如果你改由这个事件（订阅者或通知路由）把偏移 -送到通知渠道，请传入 ``notify=False``，同一次偏移才不会被通知两次。 +``IntegrityViolation`` 事件的形式发布。通知路由器启用期间（见 :doc:`notifications`）， +这个事件由路由负责送达，不再直接通知进程共用的管理器，同一次偏移因此不会被通知两次； +你自己传入的 ``manager`` 则一律会收到通知。``notify=False`` 会完全关闭这项直接通知。 出现问题时 ---------- diff --git a/docs/source/Zh-CN/usage/notifications.rst b/docs/source/Zh-CN/usage/notifications.rst index d18769e..f33bd81 100644 --- a/docs/source/Zh-CN/usage/notifications.rst +++ b/docs/source/Zh-CN/usage/notifications.rst @@ -25,3 +25,182 @@ 调度器与触发器在失败时会自动以 ``level="error"`` 通知—— 只要注册任意一个 sink，就能拿到生产告警。JSON 形式： ``FA_notify_send`` / ``FA_notify_list``。 + +事件路由 +---------------- + +通知可以改由 :doc:`事件 ` 驱动，而不是由各个模块自行调用 sink：组件 +发布事件，再由 :class:`~automation_file.notify.router.NotificationRouter` 决定 +哪些 sink 会收到。 + +.. code-block:: python + + from automation_file import ( + Route, Severity, SlackSink, notification_manager, notification_router, + ) + + notification_manager.register(SlackSink("https://hooks.slack.com/services/T/B/X", + name="team-alerts")) + notification_router.add_route(Route( + "pipeline-failures", + sinks=("team-alerts",), + types=("pipeline.*", "task.failed"), + min_severity=Severity.ERROR, + dedup_seconds=600, + rate_limit=10, + rate_period=60, + )) + notification_router.start() # 在事件总线上订阅 + +路由器在调用 ``start()`` 之前不会工作；``stop()`` 会取消订阅，``active`` 则表示 +当前的状态。``add_route`` 会替换同名的路由，``remove_route(name)`` 移除一条路由， +``routes()`` 列出所有路由。需要私有的路由器时使用 +``NotificationRouter(manager, bus)``。 + +路由 +~~~~~~~~ + +.. list-table:: + :header-rows: 1 + :widths: 22 18 60 + + * - 字段 + - 默认值 + - 含义 + * - ``name`` + - 必填 + - 路由的标识名称。 + * - ``sinks`` + - ``()`` + - 在 ``NotificationManager`` 上注册的名称。留空代表所有 sink。 + * - ``types`` + - ``()`` + - 总线的筛选条件：事件类、type 名称（``"task.failed"``）或前缀 + （``"pipeline.*"``）。留空代表所有 type。 + * - ``sources`` + - ``()`` + - 完全相同的 ``event.source`` 值。留空代表所有来源。 + * - ``min_severity`` + - ``Severity.WARNING`` + - 这条路由会投递的最低严重程度。 + * - ``dedup_seconds`` + - ``300.0`` + - 去重窗口。``0`` 表示关闭。 + * - ``rate_limit`` + - ``0`` + - 每个 ``rate_period`` 内允许的消息数。``0`` 表示不限制。 + * - ``rate_period`` + - ``60.0`` + - 速率限制窗口的长度，单位为秒。 + +同一个 sink 即使被多条路由覆盖，同一个事件也只会收到一次：由第一条获准发送的 +路由负责投递。不匹配任何路由的事件不会被投递。 + +sink 收到的内容 +~~~~~~~~~~~~~~~~~~~~~~~~ + +消息由事件组成。主题类似 ``[ERROR] task.failed: load failed``；正文依次列出 +严重程度、type、来源、主题、时间、关联 ID 与 actor，接着是 +``event.to_dict()`` 的 JSON。严重程度会对应到 sink 接受的级别：``info``、 +``warning`` 与 ``error`` 保持原名，``critical`` 则以 ``error`` 发送。 + +去重与速率限制 +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +两者都是以“每条路由、每个 sink”为单位分别计算。 + +- **去重** —— 同一个事件（type、来源与主题都相同）在第一次之后的 + ``dedup_seconds`` 内再次出现时会被丢弃；不会比较 payload。失败的尝试也算在内， + 因此故障的 sink 不会在每次重复时都被再试一次。 +- **速率限制** —— 每个 ``rate_period`` 内最多发送 ``rate_limit`` 条消息。被限制 + 挡下的事件不会被记成已发送，因此它下一次出现时仍然可以送出；重复的事件不会 + 消耗额度。 + +``notification_router.handle(event)`` 会针对每个 sink 返回一个结果：``sent``、 +``dedup``、``rate_limited``，或以 ``": "`` 表示的错误。 + +投递是在发布事件的线程中进行的。请让 sink 的超时时间保持简短；这两道防护限制了 +缓慢的 sink 被调用的频率。 + +失败 +~~~~~~~~ + +单个 sink 失败绝对不会影响其他 sink。每一次失败都会被记录到日志，并以 +``source="notify"`` 的 ``SystemErrorEvent`` 发布；它的 payload 会指出 sink +（``resource``）、``route``、``error``，以及无法投递的那个事件的 ``event_type`` 与 +``event_id``。路由器绝对不会路由这类事件，因此故障的 sink 不会形成循环；要查看 +它们，请订阅总线，或搜索 :doc:`审计轨迹 `。错误文本中的 URL 只会保留 +主机名，因为 webhook URL 或 bot token 都属于机密。 + +``automation_file.toml`` 中的路由 +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. code-block:: toml + + [[notify.sinks]] + type = "slack" + name = "team-alerts" + webhook_url = "${env:SLACK_WEBHOOK}" + + [[notify.routes]] + name = "pipeline-failures" + sinks = ["team-alerts"] + types = ["pipeline.*", "task.failed"] + sources = ["pipeline"] + min_severity = "error" + dedup_seconds = 600 + rate_limit = 10 + rate_period = 60 + +.. code-block:: python + + from automation_file import ( + AutomationConfig, ConfigWatcher, notification_manager, notification_router, + ) + + def apply(config): + config.apply_to(notification_manager, notification_router) + + watcher = ConfigWatcher("automation_file.toml", apply) + apply(watcher.start()) # 立即加载，之后每当文件变更就重新加载 + +``apply_to(manager, router)`` 会注册 sink，并让 ``[[notify.routes]]`` 表成为 +路由器的配置路由：从文件中移除的表会在下一次重新加载时停止路由，而在代码中 +加入的路由则保留。文件声明了路由时会启动路由器；重新加载移除了最后一条路由时 +则会停止。 +路由只能指向文件中声明的、或已经在管理器上注册的 sink；未知的 sink、未知的 +严重程度、未知的选项或重复的名称，都会在任何变更发生之前抛出 +:class:`~automation_file.ConfigException`，因此失败的重新加载会保留先前的路由。 +没有传入 ``router`` 参数时，路由不会被应用。 + +动作 +~~~~~~~~ + +.. code-block:: json + + [ + ["FA_notify_route_add", {"name": "pipeline-failures", "sinks": ["team-alerts"], + "types": ["pipeline.*", "task.failed"], "min_severity": "error", + "dedup_seconds": 600, "rate_limit": 10, "rate_period": 60}], + ["FA_notify_route_list"], + ["FA_notify_route_remove", {"name": "pipeline-failures"}] + ] + +这些动作作用于进程级的路由器。``FA_notify_route_add`` 会启动它， +``FA_notify_route_remove`` 则在最后一条路由被移除时停止它。 + +``notify_on_failure`` +~~~~~~~~~~~~~~~~~~~~~ + +调度器与触发器仍然调用 ``notify_on_failure(context, error)``。它现在总是会发布 +一个事件：context 是调度作业（``scheduler[nightly]``）时为 ``SchedulerError``， +其余情况为 ``SystemErrorEvent``。接着： + +- **路由器工作中** —— 由路由器按照路由投递该事件，不会再发送其他消息，因此 + 没有人会收到两次通知； +- **路由器未工作** —— ``error`` 级别的消息也会直接送到每一个已注册的 sink，与 + 以往完全相同，因此不会有人收不到通知。 + +路由器工作时由路由决定：没有任何路由匹配的失败不会被投递。像 +``Route("failures", types=("scheduler.error", "system.error"))`` 这样的路由可以 +让这些告警持续送达。 diff --git a/docs/source/Zh-CN/zh_cn_index.rst b/docs/source/Zh-CN/zh_cn_index.rst index 9c26030..78aad84 100644 --- a/docs/source/Zh-CN/zh_cn_index.rst +++ b/docs/source/Zh-CN/zh_cn_index.rst @@ -280,3 +280,18 @@ Slack、Email（SMTP）、Discord、Telegram、Microsoft Teams、PagerDuty :caption: 文件完整性监控 usage/integrity + +.. _zh-cn-audit: + +第 19 章 — 审计轨迹 +=================== + +审计模式 v2：每个事件与每次存储操作各记录一条，包含 actor、资源、后端、 +结果与关联 ID；以及各种存储、搜索条件、从 v1 ``AuditLog`` 的迁移与运行指标。 +通知路由请见 :doc:`usage/notifications`。 + +.. toctree:: + :maxdepth: 2 + :caption: 审计轨迹 + + usage/audit diff --git a/docs/source/Zh-TW/usage/audit.rst b/docs/source/Zh-TW/usage/audit.rst new file mode 100644 index 0000000..871ca66 --- /dev/null +++ b/docs/source/Zh-TW/usage/audit.rst @@ -0,0 +1,288 @@ +稽核軌跡 +================ + +稽核軌跡針對函式庫所做的每一件事回答同一個問題：**誰** 在 **什麼時候** 做了 +**什麼**，對象是 **哪個資源**，使用 **哪個後端**，以及 **結果如何**。這是稽核 +結構描述 v2。沒有任何元件會自行寫入稽核資料列：元件發布 +:doc:`事件 `，儲存層回報它的操作，再由 +:class:`~automation_file.audit.trail.AuditTrail` 把兩者轉成 +:class:`~automation_file.audit.record.AuditRecord`，寫進 +:class:`~automation_file.audit.store.AuditStore`。 + +v1 的 :class:`~automation_file.core.audit.AuditLog` 維持原樣；請見 +`從 v1 遷移`_。 + +最小範例 +---------------- + +.. code-block:: python + + from automation_file import audit_search, configure_audit + + configure_audit("audit.sqlite") # 建立資料庫並開始記錄 + + ... # 執行管線、複製檔案、發布事件 + + for entry in audit_search(status="error", limit=20): + print(entry["timestamp"], entry["actor"], entry["action"], entry["resource"], + entry["error"]) + +``configure_audit`` 接受 SQLite 資料庫的路徑或一個現成的儲存庫，把行程層級的 +``audit_trail`` 指向它並啟動。在呼叫之前，軌跡沒有儲存庫，也不會記錄任何內容。 +``audit_search`` 回傳一般的字典，最新的在前。 + +正式環境範例 +------------------------ + +.. code-block:: python + + from automation_file import ( + File, PipelineCompleted, PipelineStarted, actor_scope, audit_search, + audit_trail, configure_audit, correlation_scope, emit, + ) + + configure_audit("/var/lib/automation_file/audit.sqlite") + + with actor_scope("scheduler"), correlation_scope() as run_id: + emit(PipelineStarted(source="pipeline", subject="daily-report started", + payload={"pipeline": "daily-report", "run_id": run_id})) + File("s3://reports/q1.csv").copy_to("local:///backup/q1.csv") # 會被記錄 + audit_trail.record("approve", resource="s3://reports/q1.csv", backend="s3", + source="review", metadata={"ticket": "OPS-12"}) + emit(PipelineCompleted(source="pipeline", subject="daily-report completed", + payload={"pipeline": "daily-report", "duration_ms": 812.0})) + + audit_search(correlation_id=run_id) # 整次執行，最新的在前 + audit_search(resource_prefix="s3://reports/", since="2026-10-01T00:00:00+00:00") + audit_trail.count(actor="scheduler", status="error") + audit_trail.purge(older_than_seconds=90 * 24 * 3600) # 保留 90 天 + +請在啟動時就開啟儲存庫，讓問題立刻浮現：資料庫無法開啟時，``configure_audit`` +會拋出 :class:`~automation_file.AuditException`。清除舊紀錄的工作請交給排程。 +兩個範圍內的所有事物都帶有相同的關聯 ID 與 actor，因此只要一個篩選條件就能取回 +整次執行。 + +自行建立軌跡，使用私有的匯流排或其他儲存庫： + +.. code-block:: python + + from automation_file import AuditTrail, EventBus, SQLiteAuditStore + + trail = AuditTrail(SQLiteAuditStore("tenant-a.sqlite"), bus=EventBus()) + trail.start() + ... + trail.stop() + trail.store.close() + +紀錄 +-------- + +``AuditRecord`` 不可變，且可直接轉成 JSON（``to_dict()`` / +``AuditRecord.from_dict()``）。 + +.. list-table:: + :header-rows: 1 + :widths: 22 78 + + * - 欄位 + - 意義 + * - ``id`` + - 唯一 ID。由事件產生的紀錄沿用該事件的 ID。 + * - ``timestamp`` + - 發生的時間，為帶時區的 UTC ``datetime``。不含時區的時間會被拒絕。 + * - ``actor`` + - **誰**：外層 ``actor_scope`` 的 actor，或執行行程的使用者。 + * - ``source`` + - 回報的元件：``pipeline``、``storage``、``scheduler``、``notify`` 等。 + * - ``pipeline``、``task`` + - 所屬的管線與任務（如果有的話）。 + * - ``action`` + - **做了什麼**：事件的 type（``task.failed``）或儲存操作（``upload``、 + ``download``、``read``、``delete``、``mkdir``、``copy``、 + ``move``）。 + * - ``resource`` + - **哪個資源**：儲存 URI 或其他對象。 + * - ``backend`` + - **哪個後端**：儲存的 scheme（``s3``、``local`` 等）。 + * - ``status`` + - **結果如何**：``ok``、``warning``、``error``，或發出者自行選用的字詞。 + * - ``duration_ms`` + - 花費的時間（已知時）。 + * - ``error`` + - 失敗時為 ``": "``。 + * - ``metadata`` + - 其餘的所有資訊，以 JSON 值保存。JSON 無法表示的值會保存為它的 ``repr``。 + * - ``correlation_id`` + - 所屬的那次執行；在任何 ``correlation_scope`` 之外為 ``None``。 + +記錄哪些內容 +------------------------ + +**事件。** 匯流排上的每個事件都會變成一筆紀錄。``source``、``actor``、 +``timestamp`` 與 ``correlation_id`` 取自事件；``action`` 是事件的 type； +``pipeline``、``task``、``resource``、``backend``、``status``、 +``duration_ms`` 與 ``error`` 取自 payload 中同名的鍵。事件的 ``subject``、 +``severity`` 以及 payload 的其他鍵都放在 ``metadata`` 之下。payload 沒有 +``status`` 時由嚴重程度決定：info 為 ``ok``，warning 為 ``warning``，error 與 +critical 為 ``error``。 + +**儲存操作。** 每一次上傳、下載、讀取、刪除、建立目錄、複製與搬移都會變成一筆 +紀錄：``source="storage"``，``action`` 是操作名稱，``resource`` 是 URI， +``backend`` 是 scheme，無論成功或失敗。複製或搬移算作一筆紀錄，其來源位於 +``metadata["source_uri"]``。 + +**失敗的儲存操作只記錄一次。** 儲存層會回報該操作，儲存橋接器也會為它發布一個 +``storage.error`` 事件。軌跡保留操作本身，略過那個事件。 + +**手動記錄。** ``audit_trail.record(action, **fields)`` 會附加一筆紀錄；actor 與 +關聯 ID 預設取自目前的範圍。 + +篩選條件 +---------------- + +``search`` 與 ``count`` 接受相同的具名篩選條件；``search`` 回傳的紀錄最新的在前。 + +.. list-table:: + :header-rows: 1 + :widths: 28 72 + + * - 篩選條件 + - 符合的紀錄 + * - ``since``、``until`` + - 時間範圍：包含 ``since``，不包含 ``until``。可以是帶時區的 ``datetime``、 + 帶有時差或 ``Z`` 的 ISO 8601 字串，或自 epoch 起算的秒數。 + * - ``actor``、``source``、``pipeline``、``task``、``action``、 + ``backend``、``status``、``correlation_id`` + - 欄位與此值完全相同。 + * - ``resource_prefix`` + - 資源以此文字開頭。 + * - ``text`` + - 此文字出現在 action、resource、error、actor、source、pipeline、task、 + backend 或 metadata 的 JSON 之中。 + * - ``limit``、``offset`` + - 分頁。``limit`` 預設為 100，且不得超過 10 000。``count`` 會忽略這兩者。 + +``resource_prefix`` 與 ``text`` 會把文字當成字面值（``%`` 與 ``_`` 都是普通 +字元），並忽略 ASCII 字母的大小寫。值為 ``None`` 的篩選條件不會限制搜尋。未知的 +篩選條件名稱或型別錯誤的值會拋出 ``AuditException``，而不是悄悄回傳所有紀錄。 + +儲存介面 +---------------- + +``AuditStore`` 是儲存庫要實作的介面：目前是 SQLite 儲存庫，日後可以是 +PostgreSQL 或遠端儲存庫。 + +.. code-block:: python + + from automation_file import AuditQuery, AuditRecord, AuditStore + + class MyStore(AuditStore): + def append(self, record: AuditRecord) -> None: ... + def search(self, **filters) -> list[AuditRecord]: + query = AuditQuery.from_filters(filters) # 驗證過的篩選條件 + ... + def count(self, **filters) -> int: ... + def purge(self, older_than_seconds: float) -> int: ... + def close(self) -> None: ... + +儲存庫只能附加、可在執行緒之間共用、``search`` 以最新的在前回傳、拒絕 ``id`` +已存在的紀錄，並且所有失敗都以 ``AuditException`` 拋出。 +``AuditQuery.from_filters`` 以相同的方式為每一種儲存庫驗證篩選條件。 + +``SQLiteAuditStore(path)`` 把紀錄保存在一張資料表中，另有一張資料表記載結構描述 +版本（``2``），並在時間戳記、關聯 ID、資源與 action 上建立索引。所有的值都以 +繫結參數傳給 SQLite，篩選條件中的 ``LIKE`` 萬用字元也會被跳脫。所有執行緒透過一把 +鎖共用同一條連線，並使用 WAL 模式，因此另一個行程可以在這個行程寫入時讀取。 +``MemoryAuditStore()`` 把紀錄保存在行程內，供測試使用。 + +從 v1 遷移 +-------------------- + +.. code-block:: python + + from automation_file import SQLiteAuditStore + + store = SQLiteAuditStore("audit.sqlite") + store.import_v1("old-audit.sqlite3") # 回傳新增的資料列數 + +每一筆 v1 資料列都會變成一筆 ``source="audit.v1"``、actor 為 ``unknown`` 的 +紀錄；它的 payload 與 result 放在 ``metadata`` 之下。v1 資料庫只會被讀取，重複 +匯入不會新增任何紀錄。 + +動作 +-------- + +``register_audit_ops(registry)`` 會註冊四個動作；它們作用於行程層級的軌跡。 + +.. code-block:: json + + [ + ["FA_audit_configure", {"db_path": "/var/lib/automation_file/audit.sqlite"}], + ["FA_audit_search", {"status": "error", "since": "2026-10-01T00:00:00+00:00", "limit": 50}], + ["FA_audit_count", {"correlation_id": "4f0c2b6e9d5a4c1f8a7b3e2d1c0f9a8b"}], + ["FA_audit_purge", {"older_than_seconds": 7776000}] + ] + +``FA_audit_search`` 與 ``FA_audit_count`` 接受上述的篩選條件； +``FA_audit_search`` 以 ``to_dict()`` 的形式回傳每一筆紀錄。 + +能夠清除軌跡的用戶端，就能抹去自己的痕跡。除非遠端用戶端本來就該管理軌跡，否則在 +TCP 或 HTTP 動作伺服器上請傳入拒絕 ``FA_audit_purge`` 與 ``FA_audit_configure`` 的 +:class:`~automation_file.ActionACL`，在 MCP 伺服器上則不要把它們列入 +``--allowed-actions``。 + +發生問題時 +-------------------- + +- **紀錄無法寫入。** 失敗會被記錄到日誌 + （``audit trail: cannot write the record of ...``），該筆紀錄則被捨棄。它絕對 + 不會拋進被稽核的程式：磁碟已滿不應該讓管線停擺。請監看日誌中的這一行並設定 + 告警。 +- **資料庫無法開啟。** ``configure_audit`` 與 ``SQLiteAuditStore`` 會拋出 + ``AuditException``。請在啟動時開啟儲存庫，才能立刻發現問題。 +- **資料庫是由較新的版本寫入的。** 儲存庫會拒絕開啟，而不是猜測它的結構。 +- **持久性。** SQLite 儲存庫每寫入一筆紀錄就提交一次。在 WAL 模式下，行程當掉 + 不會遺失任何資料；斷電則可能遺失最後一小段時間的紀錄。請把資料庫放在本機 + 磁碟：WAL 無法在網路共用磁碟上運作。 +- **成長。** 沒有任何紀錄會自動刪除。請依照保存期限，定期呼叫 ``purge`` 或 + ``FA_audit_purge``。 +- **尚未設定就搜尋。** 沒有設定儲存庫時，``audit_search`` 會拋出 + ``AuditException``，因此空的結果永遠代表「沒有符合的紀錄」。 +- **執行緒。** 範圍不會自動跟著工作進入另一個執行緒；把工作分派出去的程式要在 + 那裡重新進入 ``correlation_scope`` 與 ``actor_scope``，否則它的紀錄不會帶有 + 關聯 ID。 +- **機密。** payload 會原樣保存。請不要把憑證放進事件的 payload 或 ``metadata``。 + +維運指標 +---------------- + +同一批事件與儲存操作也會餵給 Prometheus 計數器，與每個動作的指標 +``automation_file_actions_total``、``automation_file_action_duration_seconds`` +並列： + +.. code-block:: python + + from automation_file import install_operational_metrics, start_metrics_server + + install_operational_metrics() # 一次即可；重複呼叫不會有任何改變 + start_metrics_server(host="127.0.0.1", port=9945) + +.. list-table:: + :header-rows: 1 + :widths: 62 38 + + * - 指標 + - 標籤 + * - ``automation_file_events_total`` + - ``type``、``severity`` + * - ``automation_file_notifications_total`` + - ``sink``、``outcome`` （``sent``、``dedup``、``rate_limited``、 + ``error``） + * - ``automation_file_storage_operations_total`` + - ``operation``、``backend``、``status`` + * - ``automation_file_storage_operation_duration_seconds`` （直方圖） + - ``operation``、``backend`` + +``install_operational_metrics()`` 會訂閱事件匯流排與儲存觀察者。通知計數器不需要 +安裝：通知管理器與路由器會自行計算每一次投遞。標籤絕對不會帶有路徑、URI 或關聯 +ID，而且每個標籤最多保留 100 個不同的值；超出的值會計入 ``other``。 diff --git a/docs/source/Zh-TW/usage/config.rst b/docs/source/Zh-TW/usage/config.rst index 16e973e..088b8ed 100644 --- a/docs/source/Zh-TW/usage/config.rst +++ b/docs/source/Zh-TW/usage/config.rst @@ -31,10 +31,14 @@ .. code-block:: python - from automation_file import AutomationConfig, notification_manager + from automation_file import AutomationConfig, notification_manager, notification_router config = AutomationConfig.load("automation_file.toml") - config.apply_to(notification_manager) + config.apply_to(notification_manager, notification_router) + +``apply_to`` 會註冊各個 sink。同時傳入路由器時，也會套用 ``[[notify.routes]]`` +表格，並在檔案宣告了路由時啟動路由器（見 :doc:`notifications`）；沒有傳入時， +路由只會被驗證而不會套用。 未解析的 ``${…}`` 參考會擲出 :class:`~automation_file.SecretNotFoundException`， diff --git a/docs/source/Zh-TW/usage/integrity.rst b/docs/source/Zh-TW/usage/integrity.rst index 4e872be..0794ca4 100644 --- a/docs/source/Zh-TW/usage/integrity.rst +++ b/docs/source/Zh-TW/usage/integrity.rst @@ -421,8 +421,9 @@ TCP 或 HTTP 動作伺服器上請傳入 ``ActionACL``，在 MCP 伺服器上請 通知的去向與以往相同：透過你傳入的 ``manager`` 送出，沒有傳入時則使用整個行程共用的 ``notification_manager``。新增的只有一點：每一次偏移（包含新增）也都會以 -``IntegrityViolation`` 事件的形式發布。如果你改由這個事件（訂閱者或通知路由）把偏移 -送到通知管道，請傳入 ``notify=False``，同一次偏移才不會被通知兩次。 +``IntegrityViolation`` 事件的形式發布。通知路由器啟用期間（見 :doc:`notifications`）， +這個事件由路由負責送達，不再直接通知行程共用的管理器，同一次偏移因此不會被通知兩次； +你自己傳入的 ``manager`` 則一律會收到通知。``notify=False`` 會完全關閉這項直接通知。 發生問題時 ---------- diff --git a/docs/source/Zh-TW/usage/notifications.rst b/docs/source/Zh-TW/usage/notifications.rst index fdf81d8..e0f6739 100644 --- a/docs/source/Zh-TW/usage/notifications.rst +++ b/docs/source/Zh-TW/usage/notifications.rst @@ -25,3 +25,182 @@ 排程器與觸發器在失敗時會自動以 ``level="error"`` 通知—— 只要註冊任意一個 sink，就能取得正式環境告警。JSON 形式： ``FA_notify_send`` / ``FA_notify_list``。 + +事件路由 +---------------- + +通知可以改由 :doc:`事件 ` 驅動，而不是由各個模組自行呼叫 sink：元件 +發布事件，再由 :class:`~automation_file.notify.router.NotificationRouter` 決定 +哪些 sink 會收到。 + +.. code-block:: python + + from automation_file import ( + Route, Severity, SlackSink, notification_manager, notification_router, + ) + + notification_manager.register(SlackSink("https://hooks.slack.com/services/T/B/X", + name="team-alerts")) + notification_router.add_route(Route( + "pipeline-failures", + sinks=("team-alerts",), + types=("pipeline.*", "task.failed"), + min_severity=Severity.ERROR, + dedup_seconds=600, + rate_limit=10, + rate_period=60, + )) + notification_router.start() # 在事件匯流排上訂閱 + +路由器在呼叫 ``start()`` 之前不會運作；``stop()`` 會取消訂閱，``active`` 則表示 +目前的狀態。``add_route`` 會取代同名的路由，``remove_route(name)`` 移除一條路由， +``routes()`` 列出所有路由。需要私有的路由器時使用 +``NotificationRouter(manager, bus)``。 + +路由 +~~~~~~~~ + +.. list-table:: + :header-rows: 1 + :widths: 22 18 60 + + * - 欄位 + - 預設值 + - 意義 + * - ``name`` + - 必填 + - 路由的識別名稱。 + * - ``sinks`` + - ``()`` + - 在 ``NotificationManager`` 上註冊的名稱。留空代表所有 sink。 + * - ``types`` + - ``()`` + - 匯流排的篩選條件：事件類別、type 名稱（``"task.failed"``）或前綴 + （``"pipeline.*"``）。留空代表所有 type。 + * - ``sources`` + - ``()`` + - 完全相同的 ``event.source`` 值。留空代表所有來源。 + * - ``min_severity`` + - ``Severity.WARNING`` + - 這條路由會投遞的最低嚴重程度。 + * - ``dedup_seconds`` + - ``300.0`` + - 去重視窗。``0`` 表示關閉。 + * - ``rate_limit`` + - ``0`` + - 每個 ``rate_period`` 內允許的訊息數。``0`` 表示不限制。 + * - ``rate_period`` + - ``60.0`` + - 速率限制視窗的長度，單位為秒。 + +同一個 sink 即使被多條路由涵蓋，同一個事件也只會收到一次：由第一條獲准發送的 +路由負責投遞。不符合任何路由的事件不會被投遞。 + +sink 收到的內容 +~~~~~~~~~~~~~~~~~~~~~~~~ + +訊息由事件組成。主旨類似 ``[ERROR] task.failed: load failed``；內文依序列出 +嚴重程度、type、來源、主旨、時間、關聯 ID 與 actor，接著是 +``event.to_dict()`` 的 JSON。嚴重程度會對應到 sink 接受的等級：``info``、 +``warning`` 與 ``error`` 維持原名，``critical`` 則以 ``error`` 發送。 + +去重與速率限制 +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +兩者都是以「每條路由、每個 sink」為單位分別計算。 + +- **去重** —— 同一個事件（type、來源與主旨都相同）在第一次之後的 + ``dedup_seconds`` 內再次出現時會被丟棄；不會比較 payload。失敗的嘗試也算在內， + 因此故障的 sink 不會在每次重複時都被再試一次。 +- **速率限制** —— 每個 ``rate_period`` 內最多發送 ``rate_limit`` 則訊息。被限制 + 擋下的事件不會被記成已發送，因此它下一次出現時仍然可以送出；重複的事件不會 + 消耗額度。 + +``notification_router.handle(event)`` 會針對每個 sink 回傳一個結果：``sent``、 +``dedup``、``rate_limited``，或以 ``": "`` 表示的錯誤。 + +投遞是在發布事件的執行緒中進行的。請讓 sink 的逾時時間保持簡短；這兩道防護限制了 +緩慢的 sink 被呼叫的頻率。 + +失敗 +~~~~~~~~ + +單一 sink 失敗絕對不會影響其他 sink。每一次失敗都會被記錄到日誌，並以 +``source="notify"`` 的 ``SystemErrorEvent`` 發布；它的 payload 會指出 sink +（``resource``）、``route``、``error``，以及無法投遞的那個事件的 ``event_type`` 與 +``event_id``。路由器絕對不會路由這類事件，因此故障的 sink 不會形成迴圈；要查看 +它們，請訂閱匯流排，或搜尋 :doc:`稽核軌跡 `。錯誤文字中的 URL 只會保留 +主機名稱，因為 webhook URL 或 bot token 都屬於機密。 + +``automation_file.toml`` 中的路由 +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. code-block:: toml + + [[notify.sinks]] + type = "slack" + name = "team-alerts" + webhook_url = "${env:SLACK_WEBHOOK}" + + [[notify.routes]] + name = "pipeline-failures" + sinks = ["team-alerts"] + types = ["pipeline.*", "task.failed"] + sources = ["pipeline"] + min_severity = "error" + dedup_seconds = 600 + rate_limit = 10 + rate_period = 60 + +.. code-block:: python + + from automation_file import ( + AutomationConfig, ConfigWatcher, notification_manager, notification_router, + ) + + def apply(config): + config.apply_to(notification_manager, notification_router) + + watcher = ConfigWatcher("automation_file.toml", apply) + apply(watcher.start()) # 立即載入，之後每當檔案變更就重新載入 + +``apply_to(manager, router)`` 會註冊 sink，並讓 ``[[notify.routes]]`` 表格成為 +路由器的設定路由：從檔案中移除的表格會在下一次重新載入時停止路由，而在程式中 +加入的路由則保留。檔案宣告了路由時會啟動路由器；重新載入移除了最後一條路由時 +則會停止。 +路由只能指向檔案中宣告的、或已經在管理器上註冊的 sink；未知的 sink、未知的 +嚴重程度、未知的選項或重複的名稱，都會在任何變更發生之前拋出 +:class:`~automation_file.ConfigException`，因此失敗的重新載入會保留先前的路由。 +沒有傳入 ``router`` 引數時，路由不會被套用。 + +動作 +~~~~~~~~ + +.. code-block:: json + + [ + ["FA_notify_route_add", {"name": "pipeline-failures", "sinks": ["team-alerts"], + "types": ["pipeline.*", "task.failed"], "min_severity": "error", + "dedup_seconds": 600, "rate_limit": 10, "rate_period": 60}], + ["FA_notify_route_list"], + ["FA_notify_route_remove", {"name": "pipeline-failures"}] + ] + +這些動作作用於行程層級的路由器。``FA_notify_route_add`` 會啟動它， +``FA_notify_route_remove`` 則在最後一條路由被移除時停止它。 + +``notify_on_failure`` +~~~~~~~~~~~~~~~~~~~~~ + +排程器與觸發器仍然呼叫 ``notify_on_failure(context, error)``。它現在一律會發布 +一個事件：context 是排程工作（``scheduler[nightly]``）時為 ``SchedulerError``， +其餘情況為 ``SystemErrorEvent``。接著： + +- **路由器運作中** —— 由路由器依照路由投遞該事件，不會再發送其他訊息，因此 + 沒有人會收到兩次通知； +- **路由器未運作** —— ``error`` 等級的訊息也會直接送到每一個已註冊的 sink，與 + 以往完全相同，因此不會有人收不到通知。 + +路由器運作時由路由決定：沒有任何路由符合的失敗不會被投遞。像 +``Route("failures", types=("scheduler.error", "system.error"))`` 這樣的路由可以 +讓這些告警持續送達。 diff --git a/docs/source/Zh-TW/zh_tw_index.rst b/docs/source/Zh-TW/zh_tw_index.rst index 4e82cac..d48398e 100644 --- a/docs/source/Zh-TW/zh_tw_index.rst +++ b/docs/source/Zh-TW/zh_tw_index.rst @@ -280,3 +280,18 @@ Slack、Email（SMTP）、Discord、Telegram、Microsoft Teams、PagerDuty :caption: 檔案完整性監控 usage/integrity + +.. _zh-tw-audit: + +第 19 章 — 稽核軌跡 +=================== + +稽核結構描述 v2：每個事件與每次儲存操作各記錄一筆，包含 actor、資源、後端、 +結果與關聯 ID；以及各種儲存、搜尋條件、從 v1 ``AuditLog`` 的遷移與營運指標。 +通知路由請見 :doc:`usage/notifications`。 + +.. toctree:: + :maxdepth: 2 + :caption: 稽核軌跡 + + usage/audit diff --git a/docs/updates/2026-10.md b/docs/updates/2026-10.md index 82df21c..380e495 100644 --- a/docs/updates/2026-10.md +++ b/docs/updates/2026-10.md @@ -425,3 +425,19 @@ Index and query commands: [README.md](README.md). New entries go at the end. - **Docs**: chapter 18 in the three manuals (`usage/integrity.rst`), `docs/source/API/integrity.rst`, the indexes, the feature list and the integrity section of the three READMEs, `architecture.md` §2 to §4, `CLAUDE.md` (package map, key types). - **Files**: `automation_file/integrity/` (16 modules), `automation_file/core/{fim,manifest,action_registry}.py`, `automation_file/__init__.py`, the tests above, the documentation above. - **Open items**: an `integrity` subcommand for the CLI comes with the pipeline and audit subcommands (#33). + +## U-20261008-17 · 2026-10-08 · Notification router and audit schema v2 · #notify #audit #roadmap + +- **What**: two consumers of the event bus (roadmap §9 and §10, part of M6). Both are opt-in. + - **Router** (`automation_file/notify/router.py`): a `Route` names the events it wants (class, type name or prefix, source, minimum severity) and the sinks that hear about them; `NotificationRouter` subscribes on the bus and delivers a structured message built from the event. Deduplication (`dedup_seconds`) and a rate limit (`rate_limit` per `rate_period`) are kept per route and sink. A failing sink never affects another; its failure is published as a `system.error` event from the source `notify`, which the router never routes, so a broken sink cannot feed a loop. Routes come from code, from `[[notify.routes]]` in `automation_file.toml` (`AutomationConfig.apply_to(manager, router)`; a reload drops the routes the file no longer declares and keeps the ones added in code) or from `FA_notify_route_add` / `_remove` / `_list`. `FA_notify_route_add` starts the process-wide router and removing its last route with `FA_notify_route_remove` stops it, as does `apply_to`; in code the router is started with `notification_router.start()`. + - **`notify_on_failure`** always publishes an event. While the router is active its routes deliver it; otherwise the direct notification is sent as before. + - **Audit trail** (`automation_file/audit/`): `AuditTrail` turns every event and every storage operation into one `AuditRecord` (id, timestamp, actor, source, pipeline, task, action, resource, backend, status, duration, error, metadata, correlation ID) in an `AuditStore`. `SQLiteAuditStore` uses parameterised SQL, a schema-version table and WAL, and `import_v1()` copies the rows of a v1 `AuditLog`; `MemoryAuditStore` is for tests. `configure_audit(path)` starts the process-wide trail; `audit_search(...)` filters by time, actor, source, pipeline, task, action, resource prefix, backend, status, correlation ID and free text. A failed storage operation is recorded once, not also as its `storage.error` event. A record that cannot be written is logged and dropped. Actions: `FA_audit_configure`, `_search`, `_count`, `_purge`. + - **Metrics**: `install_operational_metrics()` adds the Prometheus counters `EVENT_COUNT`, `NOTIFICATION_COUNT`, `STORAGE_OPERATION_COUNT` and the histogram `STORAGE_OPERATION_DURATION`. + - Sink error text now has URLs cut to the host, in `NotificationManager.notify()` results and in the log: the Slack and Telegram errors carried the token in the path. Existing error strings change accordingly. +- **Changed while integrating**: the integrity monitor follows the rule of `notify_on_failure`. Its direct notification to the process-wide manager is left to the router while the router is active and the drift event is on the process-wide bus; a `manager` passed to the monitor is always notified, and `notify=False` (U-20261008-16) still turns the direct notification off. The check also no longer treats an empty manager as "none passed". +- **Tests**: `tests/test_notify_router.py`, `tests/test_audit_v2.py`, `tests/test_operational_metrics.py`, additions to `tests/test_notify.py` and `tests/test_config.py`, and four cases for the monitor's rule in `tests/test_integrity_legacy.py`. +- **Result / numbers**: 4506 passed, 149 skipped, 0 failed with every extra; 2789 passed, 89 skipped with the base dependencies only. `ruff check`, `ruff format --check` and `mypy automation_file` (217 files) pass. 169 registered commands. Python 3.14.7 on Windows. +- **Not verified**: delivery to a real sink over the network; the Sphinx build of the new pages (headings, markup and the imports of the examples were checked by script); Python 3.10 to 3.13. +- **Docs**: chapter 19 in the three manuals (`usage/audit.rst`, with an "Operational metrics" section), the route sections of the three `usage/notifications.rst`, the three `usage/config.rst`, `docs/source/API/audit.rst` and `API/notify.rst`, the indexes, two feature bullets and two sections in the three READMEs, `architecture.md` §2 to §4, `CLAUDE.md` (package map, key types). +- **Files**: `automation_file/notify/{router,manager,__init__}.py`, `automation_file/audit/` (6 modules), `automation_file/core/{metrics,config,action_registry,fim}.py`, `automation_file/integrity/{legacy,monitor}.py`, `automation_file/__init__.py`, the tests above, the documentation above. +- **Open items**: the scheduler half of M6 stays open as #23; an `audit` subcommand for the CLI is #33. diff --git a/docs/updates/README.md b/docs/updates/README.md index 037bd90..5ec41eb 100644 --- a/docs/updates/README.md +++ b/docs/updates/README.md @@ -58,6 +58,7 @@ In the same commit: delete the item from `progress.md`, add a `#done` entry here | ID | Date | Title | Tags | Batch | |---|---|---|---|---| +| U-20261008-17 | 2026-10-08 | Notification router and audit schema v2 | #notify #audit #roadmap | [2026-10](2026-10.md) | | U-20261008-16 | 2026-10-08 | IntegrityMonitor 2.0 | #integrity #roadmap #done | [2026-10](2026-10.md) | | U-20261008-15 | 2026-10-08 | The SFTP, OneDrive and SMB clients name the extra to install | #packaging #done | [2026-10](2026-10.md) | | U-20261008-14 | 2026-10-08 | The WebDAV client only talks to its own server | #security #incident | [2026-10](2026-10.md) | @@ -111,5 +112,5 @@ In the same commit: delete the item from `progress.md`, add a `#done` entry here | File | Period | Entries | |---|---|---:| -| [2026-10.md](2026-10.md) | 2026-10 | 27 | +| [2026-10.md](2026-10.md) | 2026-10 | 28 | | [2026-09.md](2026-09.md) | 2026-09 | 21 | diff --git a/progress.md b/progress.md index ac6d891..1559d91 100644 --- a/progress.md +++ b/progress.md @@ -29,7 +29,7 @@ Items #10 to #26 are what is left of the 1.0 roadmap (`docs/FILEAUTOMATION-1.0-R - **#33** CLI subcommands for the packages that have none: `integrity` (snapshot, baseline, verify, accept, status), `pipeline` and `audit`, as thin calls into their `FA_*` functions like `storage` (U-20261008-11). - **#22** Pipeline runtime (roadmap §7, M5): `Pipeline` domain model, DAG runtime v2 with retry, timeout, cancellation, conditions, idempotency, checkpoint and resume, dry run, execution history, and versioned YAML/JSON definitions with schema validation. `core/dag_executor.py` is the starting point. -- **#23** Scheduler, events, notifications and audit (roadmap §8 to §10, M6): one scheduler with cron (time-zone aware), manual, file-event, webhook and pipeline-dependency triggers; an event model that drives a `NotificationRouter`; audit schema v2 with correlation IDs behind a storage interface. +- **#23** Scheduler v2 (roadmap §8, the open half of M6): one scheduler with cron (time-zone aware), manual, file-event, webhook and pipeline-dependency triggers, run states and overlap protection, reading a pipeline's `schedule`. The event model, the `NotificationRouter` and audit schema v2 are done (U-20261008-05, U-20261008-17); the scheduler in `scheduler/` still dispatches action lists on its own cron loop. - **#24** UI 2.0 (roadmap §11, M7). Not before the APIs of #13 to #23 are stable (roadmap §20). - **#25** Semantic MCP tools (roadmap §12, M8): `file_*`, `storage_*`, `pipeline_*`, `integrity_status`, `audit_search`, with a permission model and dry run, next to the existing `FA_*` bridge. - **#26** Release engineering and 1.0 (roadmap §13, M9): contract and integration tests in the PR gate, PyPI Trusted Publishing, SemVer, migration guide, API freeze. diff --git a/tests/test_audit_v2.py b/tests/test_audit_v2.py new file mode 100644 index 0000000..bac4ad9 --- /dev/null +++ b/tests/test_audit_v2.py @@ -0,0 +1,993 @@ +"""Audit schema v2: the record, the stores, the trail, the v1 import and the actions.""" + +from __future__ import annotations + +import dataclasses +import json +import sqlite3 +import threading +from collections.abc import Iterator +from contextlib import closing +from datetime import datetime, timedelta, timezone +from pathlib import Path +from typing import Any + +import pytest + +from automation_file.audit import ( + DEFAULT_LIMIT, + MAX_LIMIT, + SCHEMA_VERSION, + AuditException, + AuditQuery, + AuditRecord, + AuditStore, + AuditTrail, + MemoryAuditStore, + SQLiteAuditStore, + audit_search, + audit_trail, + configure_audit, + record_from_event, + record_from_operation, + register_audit_ops, +) +from automation_file.core.action_executor import ActionExecutor +from automation_file.core.action_registry import ActionRegistry +from automation_file.core.audit import AuditLog +from automation_file.events import ( + Event, + EventBus, + PipelineCompleted, + PipelineFailed, + PipelineStarted, + Severity, + StorageError, + StorageErrorBridge, + TaskCompleted, + TaskFailed, + TaskStarted, + actor_scope, + correlation_scope, + current_actor, + event_bus, +) +from automation_file.exceptions import StorageNotFoundException, StoragePermissionException +from automation_file.storage import MemoryStorage, observe +from automation_file.storage.observe import StorageOperation + +_T0 = datetime(2026, 10, 1, 12, 0, 0, tzinfo=timezone.utc) +_FIELDS = ( + "id timestamp actor source pipeline task action resource backend status duration_ms error " + "metadata correlation_id" +) +#: The text fields of a record, and the filters that find a record by one of them. +_TEXT_FIELDS = "action source actor resource backend status pipeline task error correlation_id" +_TEXT_FILTERS = ( + "actor source pipeline task action backend status correlation_id resource_prefix text" +) +_RUN_1 = {"correlation_id": "run-1", "actor": "scheduler"} + + +def _at(minutes: float) -> datetime: + return _T0 + timedelta(minutes=minutes) + + +def _record(minutes: float = 0.0, **fields: Any) -> AuditRecord: + fields.setdefault("action", "upload") + fields.setdefault("source", "storage") + fields.setdefault("actor", "ops") + return AuditRecord(timestamp=_at(minutes), **fields) + + +@pytest.fixture(params=["memory", "sqlite"]) +def store(request: pytest.FixtureRequest, tmp_path: Path) -> Iterator[AuditStore]: + built: AuditStore + if request.param == "memory": + built = MemoryAuditStore() + else: + built = SQLiteAuditStore(tmp_path / "audit.sqlite") + yield built + built.close() + + +@pytest.fixture +def sqlite_store(tmp_path: Path) -> Iterator[SQLiteAuditStore]: + built = SQLiteAuditStore(tmp_path / "audit.sqlite") + yield built + built.close() + + +@pytest.fixture +def bus() -> EventBus: + return EventBus() + + +@pytest.fixture +def trail(bus: EventBus) -> Iterator[AuditTrail]: + built = AuditTrail(MemoryAuditStore(), bus=bus) + built.start() + yield built + built.close() + + +def _stored(trail: AuditTrail) -> list[AuditRecord]: + """Return what the trail recorded, oldest first.""" + return list(reversed(trail.search(limit=MAX_LIMIT))) + + +# ---------------------------------------------------------------------- the record + + +def test_a_record_fills_in_its_identity() -> None: + first, second = AuditRecord(action="upload"), AuditRecord(action="upload") + assert first.id != second.id + assert len(first.id) == 32 + assert first.timestamp.utcoffset() == timedelta(0) + assert first.actor == current_actor() + assert first.correlation_id is None + assert first.status == "ok" + assert dict(first.metadata) == {} + with actor_scope("scheduler"), correlation_scope("run-1"): + scoped = AuditRecord(action="upload") + assert (scoped.actor, scoped.correlation_id) == ("scheduler", "run-1") + + +def test_a_record_is_frozen_and_keyword_only() -> None: + record = _record() + with pytest.raises(dataclasses.FrozenInstanceError): + record.status = "error" + with pytest.raises(TypeError): + AuditRecord("upload") + assert hash(record) == hash(record) + + +def test_a_record_turns_into_json_and_back() -> None: + record = AuditRecord( + action="upload", + source="storage", + resource="s3://reports/q1.csv", + backend="s3", + status="error", + pipeline="daily", + task="load", + duration_ms=12.5, + error="StoragePermissionException: denied", + metadata={"attempt": 2, "tags": ("a", "b")}, + actor="ops", + correlation_id="run-1", + timestamp=_at(0), + ) + document = json.loads(json.dumps(record.to_dict())) + assert list(document) == _FIELDS.split() + assert document["timestamp"] == "2026-10-01T12:00:00+00:00" + assert document["metadata"] == {"attempt": 2, "tags": ["a", "b"]} + assert AuditRecord.from_dict(document) == record + + +def test_a_timestamp_is_always_aware_utc() -> None: + taipei = timezone(timedelta(hours=8)) + record = AuditRecord(action="x", timestamp=datetime(2026, 10, 1, 20, 0, tzinfo=taipei)) + assert record.timestamp == _at(0) + assert record.timestamp.utcoffset() == timedelta(0) + assert AuditRecord(action="x", timestamp="2026-10-01T12:00:00Z").timestamp == _at(0) + with pytest.raises(AuditException, match="time zone"): + AuditRecord(action="x", timestamp=datetime(2026, 10, 1, 12, 0)) + with pytest.raises(AuditException): + AuditRecord(action="x", timestamp="yesterday") + + +def test_metadata_that_json_cannot_hold_is_kept_as_text() -> None: + record = AuditRecord(action="x", metadata={"when": _at(0), "path": Path("a/b"), 7: "seven"}) + assert record.metadata["when"] == repr(_at(0)) + assert record.metadata["7"] == "seven" + json.dumps(record.to_dict()) + + +@pytest.mark.parametrize( + "fields", + [ + {"action": 5}, + {"status": None}, + {"resource": 5}, + {"id": ""}, + {"metadata": ["not", "a", "mapping"]}, + {"duration_ms": "fast"}, + {"duration_ms": True}, + ], +) +def test_a_record_rejects_fields_of_the_wrong_kind(fields: dict) -> None: + with pytest.raises(AuditException): + AuditRecord(**fields) + + +def test_from_dict_rejects_what_is_not_a_record() -> None: + with pytest.raises(AuditException, match="colour"): + AuditRecord.from_dict({"action": "x", "colour": "red"}) + with pytest.raises(AuditException): + AuditRecord.from_dict(["action"]) + + +# ---------------------------------------------------------------------- the store contract + + +def test_a_store_returns_what_it_was_given(store: AuditStore) -> None: + record = _record( + resource="s3://reports/q1.csv", + backend="s3", + status="error", + pipeline="daily", + task="load", + duration_ms=0.125, + error="X: y", + metadata={"attempt": 2, "nested": {"名稱": "報表"}}, + correlation_id="run-1", + ) + store.append(record) + assert store.search() == [record] + assert store.count() == 1 + + +def test_search_is_newest_first(store: AuditStore) -> None: + records = [_record(minutes, action=f"a{minutes}") for minutes in (2, 0, 3, 1)] + for record in records: + store.append(record) + assert [record.action for record in store.search()] == ["a3", "a2", "a1", "a0"] + + +def test_equal_timestamps_keep_the_later_append_first(store: AuditStore) -> None: + for name in ("first", "second", "third"): + store.append(_record(action=name)) + assert [record.action for record in store.search()] == ["third", "second", "first"] + + +def _populate(store: AuditStore) -> None: + """Store five records, one per minute; ``_record`` makes each an upload by ops unless told. + + Minutes 0 to 2 belong to run-1 of the scheduler: the start of the daily pipeline, an upload + to the reports bucket, and an upload to the archive bucket that was denied. Minute 3 is a + local delete of run-2, minute 4 the failed load task of the weekly pipeline (run-3). + """ + denied = {"status": "error", "error": "StoragePermissionException: Access Denied"} + timeout = {"status": "error", "error": "TimeoutError: no answer"} + cleanup = {"correlation_id": "run-2", "metadata": {"reason": "Cleanup after publish"}} + weekly = {"source": "pipeline", "pipeline": "weekly", "task": "load", "correlation_id": "run-3"} + reports, archive = "s3://reports/2026/q1.csv", "s3://archive/2026/q1.csv" + rows: list[dict[str, Any]] = [ + {"action": "pipeline.started", "source": "pipeline", "pipeline": "daily", **_RUN_1}, + {"resource": reports, "backend": "s3", "pipeline": "daily", "task": "publish", **_RUN_1}, + {"resource": archive, "backend": "s3", **denied, **_RUN_1}, + {"action": "delete", "resource": "local:///tmp/q1.csv", "backend": "local", **cleanup}, + {"action": "task.failed", **weekly, **timeout}, + ] + for minutes, fields in enumerate(rows): + store.append(_record(minutes, **fields)) + + +@pytest.mark.parametrize( + "filters,minutes", + [ + ({}, [4, 3, 2, 1, 0]), + ({"since": _at(2)}, [4, 3, 2]), + ({"until": _at(2)}, [1, 0]), + ({"since": _at(1), "until": _at(4)}, [3, 2, 1]), + ({"since": "2026-10-01T12:03:00+00:00"}, [4, 3]), + ({"since": "2026-10-01T20:03:00+08:00"}, [4, 3]), + ({"until": _at(1).timestamp()}, [0]), + ({"actor": "scheduler"}, [2, 1, 0]), + ({"actor": "ops"}, [4, 3]), + ({"source": "pipeline"}, [4, 0]), + ({"pipeline": "daily"}, [1, 0]), + ({"task": "load"}, [4]), + ({"action": "upload"}, [2, 1]), + ({"resource_prefix": "s3://reports/"}, [1]), + ({"resource_prefix": "s3://"}, [2, 1]), + ({"resource_prefix": "S3://REPORTS"}, [1]), + ({"backend": "local"}, [3]), + ({"status": "error"}, [4, 2]), + ({"correlation_id": "run-1"}, [2, 1, 0]), + ({"text": "denied"}, [2]), + ({"text": "q1.csv"}, [3, 2, 1]), + ({"text": "cleanup after"}, [3]), + ({"text": "weekly"}, [4]), + ({"text": "scheduler"}, [2, 1, 0]), + ({"limit": 2}, [4, 3]), + ({"limit": 2, "offset": 2}, [2, 1]), + ({"offset": 4}, [0]), + ({"limit": 0}, []), + ({"status": "error", "backend": "s3", "correlation_id": "run-1"}, [2]), + ({"action": "upload", "status": "ok", "text": "reports", "since": _at(1)}, [1]), + ({"actor": None, "status": None, "limit": None}, [4, 3, 2, 1, 0]), + ({"pipeline": "monthly"}, []), + ], +) +def test_every_filter(store: AuditStore, filters: dict[str, Any], minutes: list[int]) -> None: + _populate(store) + found = store.search(**filters) + assert [record.timestamp for record in found] == [_at(minute) for minute in minutes] + unpaged = {key: value for key, value in filters.items() if key not in ("limit", "offset")} + assert store.count(**filters) == len(store.search(**unpaged)) + + +def test_count_ignores_the_paging(store: AuditStore) -> None: + _populate(store) + assert store.count(limit=1, offset=3) == 5 + assert store.count(status="error", limit=1) == 2 + + +@pytest.mark.parametrize( + "filters", + [ + {"colour": "red"}, + {"resource": "s3://reports/q1.csv"}, + {"limit": -1}, + {"limit": MAX_LIMIT + 1}, + {"limit": "ten"}, + {"limit": True}, + {"offset": -1}, + {"offset": 1.5}, + {"actor": 5}, + {"text": ["a"]}, + {"since": "last week"}, + {"since": datetime(2026, 10, 1, 12, 0)}, + {"until": "2026-10-01T12:00:00"}, + {"since": True}, + ], +) +def test_a_bad_filter_is_an_error_not_an_empty_answer( + store: AuditStore, filters: dict[str, Any] +) -> None: + _populate(store) + with pytest.raises(AuditException): + store.search(**filters) + with pytest.raises(AuditException): + store.count(**filters) + + +def test_the_default_limit(store: AuditStore) -> None: + for index in range(DEFAULT_LIMIT + 5): + store.append(_record(index)) + assert len(store.search()) == DEFAULT_LIMIT + assert len(store.search(limit=DEFAULT_LIMIT + 5)) == DEFAULT_LIMIT + 5 + assert AuditQuery.from_filters({}).limit == DEFAULT_LIMIT + + +_INJECTIONS = [ + "x' OR '1'='1", + "x'; DROP TABLE audit_records; --", + 'x" OR ""="', + "x') UNION SELECT * FROM audit_schema_version --", + "1; DELETE FROM audit_records", + "\\' OR 1=1 --", +] + + +@pytest.mark.parametrize("attack", _INJECTIONS) +def test_sql_in_a_filter_is_literal_text(store: AuditStore, attack: str) -> None: + _populate(store) + for name in ("actor", "source", "pipeline", "task", "action", "backend", "status", "text"): + assert store.search(**{name: attack}) == [] + assert store.count(**{name: attack}) == 0 + assert store.search(correlation_id=attack) == [] + assert store.search(resource_prefix=attack) == [] + assert store.count() == 5 + + +@pytest.mark.parametrize("attack", _INJECTIONS) +def test_sql_in_a_record_is_stored_and_found_as_text(store: AuditStore, attack: str) -> None: + _populate(store) + everywhere = dict.fromkeys(_TEXT_FIELDS.split(), attack) + record = _record(10, metadata={"note": attack}, **everywhere) + store.append(record) + assert store.count() == 6 + for name in _TEXT_FILTERS.split(): + assert store.search(**{name: attack}) == [record] + + +def test_like_wildcards_in_a_filter_are_literal(store: AuditStore) -> None: + percent = _record(0, resource="local:///data/100%_done.txt", error="50% of the files") + letter = _record(1, resource="local:///data/100x_done.txt", error="500 of the files") + under = _record(2, resource="local:///data/100xxdone.txt", error="a_b") + slash = _record(3, resource="local:///data/C:\\temp\\x.txt", error="axb") + for record in (percent, letter, under, slash): + store.append(record) + assert store.search(resource_prefix="local:///data/100%") == [percent] + assert store.search(resource_prefix="local:///data/100%_") == [percent] + assert store.search(resource_prefix="local:///data/100x_") == [letter] + assert store.search(resource_prefix="%") == [] + assert store.search(resource_prefix="_") == [] + assert store.search(text="50%") == [percent] + assert store.search(text="0% of") == [percent] + assert store.search(text="a_b") == [under] + assert store.search(text="%") == [percent] + assert store.search(text="C:\\temp\\") == [slash] + assert store.search(resource_prefix="local:///data/C:\\") == [slash] + assert store.count(text="\\") == 1 + + +def test_text_search_ignores_ascii_case_only(store: AuditStore) -> None: + store.append(_record(0, error="Access DENIED for Ärger")) + assert store.count(text="access denied") == 1 + assert store.count(text="Ärger") == 1 + assert store.count(text="ärger") == 0 + + +def test_text_search_looks_into_the_metadata(store: AuditStore) -> None: + store.append(_record(0, metadata={"job": "nightly-報表", "attempt": 3})) + assert store.count(text="nightly-報表") == 1 + assert store.count(text='"attempt": 3') == 1 + assert store.count(text="weekly") == 0 + + +def test_an_id_is_stored_once(store: AuditStore) -> None: + record = _record() + store.append(record) + with pytest.raises(AuditException, match="already stored"): + store.append(record) + with pytest.raises(AuditException): + store.append({"action": "not a record"}) + assert store.count() == 1 + + +def test_purge_removes_what_is_older(store: AuditStore) -> None: + now = datetime.now(timezone.utc) + old = AuditRecord(action="old", timestamp=now - timedelta(days=2)) + recent = AuditRecord(action="recent", timestamp=now - timedelta(minutes=1)) + store.append(old) + store.append(recent) + assert store.purge(older_than_seconds=3600) == 1 + assert store.search() == [recent] + assert store.purge(older_than_seconds=3600) == 0 + for bad in (0, -5, "soon", True, float("nan")): + with pytest.raises(AuditException): + store.purge(bad) + store.append(old) + assert store.count() == 2 + + +def test_a_closed_store_refuses_work(store: AuditStore) -> None: + store.append(_record()) + store.close() + store.close() + with pytest.raises(AuditException): + store.append(_record(1)) + with pytest.raises(AuditException): + store.search() + with pytest.raises(AuditException): + store.count() + with pytest.raises(AuditException): + store.purge(60) + + +def test_a_store_is_a_context_manager(tmp_path: Path) -> None: + with SQLiteAuditStore(tmp_path / "audit.sqlite") as opened: + opened.append(_record()) + with pytest.raises(AuditException): + opened.count() + + +# ---------------------------------------------------------------------- the SQLite store + + +def _names(path: Path, kind: str) -> set[str]: + with closing(sqlite3.connect(path)) as conn: + rows = conn.execute("SELECT name FROM sqlite_master WHERE type = ?", (kind,)).fetchall() + return {row[0] for row in rows} + + +def test_the_database_states_its_schema_version(sqlite_store: SQLiteAuditStore) -> None: + assert SCHEMA_VERSION == 2 + assert sqlite_store.schema_version == 2 + assert {"audit_records", "audit_schema_version"} <= _names(sqlite_store.path, "table") + with closing(sqlite3.connect(sqlite_store.path)) as conn: + assert conn.execute("SELECT version FROM audit_schema_version").fetchall() == [(2,)] + + +def test_the_database_has_its_indexes(sqlite_store: SQLiteAuditStore) -> None: + with closing(sqlite3.connect(sqlite_store.path)) as conn: + indexed = { + conn.execute(f"PRAGMA index_info({name})").fetchall()[0][2] + for name in _names(sqlite_store.path, "index") + if name.startswith("idx_audit_records_") + } + plan = conn.execute( + "EXPLAIN QUERY PLAN SELECT id FROM audit_records WHERE correlation_id = ?", ("run",) + ).fetchall() + assert indexed == {"ts_us", "correlation_id", "resource", "action"} + assert "idx_audit_records_correlation" in plan[0][3] + + +def test_the_database_outlives_the_store(tmp_path: Path) -> None: + path = tmp_path / "nested" / "deeper" / "audit.sqlite" + record = _record(metadata={"kept": True}, correlation_id="run-1") + with SQLiteAuditStore(path) as first: + first.append(record) + with SQLiteAuditStore(path) as second: + assert second.search() == [record] + assert second.schema_version == 2 + with closing(sqlite3.connect(path)) as conn: + assert conn.execute("SELECT COUNT(*) FROM audit_schema_version").fetchone() == (1,) + + +def test_a_newer_schema_is_refused(tmp_path: Path) -> None: + path = tmp_path / "audit.sqlite" + SQLiteAuditStore(path).close() + with closing(sqlite3.connect(path)) as conn: + conn.execute("UPDATE audit_schema_version SET version = 99") + conn.commit() + with pytest.raises(AuditException, match="schema version 99"): + SQLiteAuditStore(path) + + +def test_a_path_that_cannot_be_a_database_is_refused(tmp_path: Path) -> None: + blocker = tmp_path / "blocker" + blocker.write_text("x", encoding="utf-8") + with pytest.raises(AuditException): + SQLiteAuditStore(blocker / "child" / "audit.sqlite") + garbage = tmp_path / "garbage.sqlite" + garbage.write_bytes(b"this is not a sqlite database, not even close" * 40) + with pytest.raises(AuditException): + SQLiteAuditStore(garbage) + + +def test_many_threads_share_one_store(sqlite_store: SQLiteAuditStore) -> None: + threads, per_thread = 8, 40 + failures: list[Exception] = [] + barrier = threading.Barrier(threads) + + def work(worker: int) -> None: + try: + barrier.wait(timeout=10) + for index in range(per_thread): + sqlite_store.append( + _record(index, action=f"w{worker}", correlation_id=f"run-{worker}") + ) + if index % 10 == 0: + sqlite_store.search(correlation_id=f"run-{worker}", limit=5) + sqlite_store.count(action=f"w{worker}") + except Exception as error: # pylint: disable=broad-except # asserted on below + failures.append(error) + + workers = [threading.Thread(target=work, args=(number,)) for number in range(threads)] + for worker in workers: + worker.start() + for worker in workers: + worker.join(timeout=60) + assert failures == [] + assert sqlite_store.count() == threads * per_thread + for number in range(threads): + assert sqlite_store.count(correlation_id=f"run-{number}") == per_thread + ids = {record.id for record in sqlite_store.search(limit=MAX_LIMIT)} + assert len(ids) == threads * per_thread + + +def test_two_stores_can_share_one_database(tmp_path: Path) -> None: + path = tmp_path / "audit.sqlite" + with SQLiteAuditStore(path) as writer, SQLiteAuditStore(path) as reader: + writer.append(_record(0, action="one")) + assert reader.count() == 1 + reader.append(_record(1, action="two")) + assert [record.action for record in writer.search()] == ["two", "one"] + + +# ---------------------------------------------------------------------- v1 import + + +def test_import_v1_copies_the_rows_of_an_audit_log( + sqlite_store: SQLiteAuditStore, tmp_path: Path +) -> None: + v1_path = tmp_path / "v1.sqlite" + log = AuditLog(v1_path) + log.record("FA_copy_file", {"src": "a", "dst": "b"}, result={"ok": True}, duration_ms=12.5) + log.record("FA_delete", {"path": "x' OR '1'='1"}, error=ValueError("boom"), duration_ms=1.0) + v1_times = {row["action"]: row["ts"] for row in log.recent()} + assert sqlite_store.import_v1(v1_path) == 2 + failed, copied = sqlite_store.search(source="audit.v1") + assert copied.action == "FA_copy_file" + assert copied.status == "ok" + assert copied.error is None + assert copied.duration_ms == pytest.approx(12.5) + assert copied.metadata["payload"] == {"src": "a", "dst": "b"} + assert copied.metadata["result"] == {"ok": True} + assert copied.correlation_id is None + assert copied.actor == "unknown" + assert abs(copied.timestamp.timestamp() - v1_times["FA_copy_file"]) < 0.001 + assert failed.action == "FA_delete" + assert failed.status == "error" + assert failed.error is not None + assert "boom" in failed.error + assert failed.metadata["payload"] == {"path": "x' OR '1'='1"} + assert failed.metadata["result"] is None + + +def test_import_v1_twice_adds_nothing(sqlite_store: SQLiteAuditStore, tmp_path: Path) -> None: + v1_path = tmp_path / "v1.sqlite" + log = AuditLog(v1_path) + log.record("first", {}) + assert sqlite_store.import_v1(v1_path) == 1 + assert sqlite_store.import_v1(v1_path) == 0 + log.record("second", {}) + assert sqlite_store.import_v1(v1_path) == 1 + assert sqlite_store.count(source="audit.v1") == 2 + assert log.count() == 2 + + +def test_import_v1_refuses_what_is_not_a_v1_log( + sqlite_store: SQLiteAuditStore, tmp_path: Path +) -> None: + with pytest.raises(AuditException, match="not found"): + sqlite_store.import_v1(tmp_path / "missing.sqlite") + assert not (tmp_path / "missing.sqlite").exists() + empty = tmp_path / "empty.sqlite" + sqlite3.connect(empty).close() + with pytest.raises(AuditException, match="cannot import"): + sqlite_store.import_v1(empty) + assert sqlite_store.count() == 0 + + +# ---------------------------------------------------------------------- event and operation records + + +def test_an_event_becomes_a_record() -> None: + target = {"pipeline": "daily", "task": "load", "resource": "s3://reports/q1.csv"} + outcome = {"backend": "s3", "status": "failed", "error": "TimeoutError: no answer"} + rest = {"run_id": "run-7", "attempt": 2, "action": "FA_storage_copy"} + payload = {**target, **outcome, **rest, "duration_ms": 1500} + with actor_scope("scheduler"), correlation_scope("run-7"): + event = TaskFailed(source="pipeline", subject="load failed", payload=payload) + record = record_from_event(event) + assert (record.id, record.timestamp) == (event.id, event.timestamp) + assert (record.actor, record.correlation_id) == ("scheduler", "run-7") + assert (record.source, record.action) == ("pipeline", "task.failed") + for name, value in {**target, **outcome}.items(): + assert getattr(record, name) == value + assert record.duration_ms == pytest.approx(1500.0) + assert isinstance(record.duration_ms, float) + assert record.metadata == {**rest, "subject": "load failed", "severity": "error"} + + +@pytest.mark.parametrize( + "severity,status", + [ + (Severity.INFO, "ok"), + (Severity.WARNING, "warning"), + (Severity.ERROR, "error"), + (Severity.CRITICAL, "error"), + ], +) +def test_an_event_without_a_status_takes_it_from_its_severity( + severity: Severity, status: str +) -> None: + record = record_from_event(Event(source="system", severity=severity)) + assert record.status == status + assert record.action == "event" + assert record.pipeline is None + assert record.duration_ms is None + + +def test_a_duration_that_is_not_a_number_stays_in_the_metadata() -> None: + record = record_from_event(PipelineCompleted(payload={"duration_ms": "fast", "task": 7})) + assert record.duration_ms is None + assert record.metadata["duration_ms"] == "fast" + assert record.task == "7" + + +def test_a_storage_operation_becomes_a_record() -> None: + operation = StorageOperation( + operation="copy", + uri="local:///backup/q1.csv", + backend="local", + status="error", + duration_ms=3.5, + source_uri="s3://reports/q1.csv", + error="StoragePermissionException: denied", + error_type="StoragePermissionException", + ) + with actor_scope("mcp"), correlation_scope("run-3"): + record = record_from_operation(operation) + assert record.source == "storage" + assert record.action == "copy" + assert record.resource == "local:///backup/q1.csv" + assert record.backend == "local" + assert record.status == "error" + assert record.duration_ms == pytest.approx(3.5) + assert record.error == "StoragePermissionException: denied" + assert record.metadata == { + "source_uri": "s3://reports/q1.csv", + "error_type": "StoragePermissionException", + } + assert (record.actor, record.correlation_id) == ("mcp", "run-3") + + +# ---------------------------------------------------------------------- the trail + + +@pytest.fixture(params=["bridge first", "trail first"]) +def bridged(request: pytest.FixtureRequest, bus: EventBus) -> Iterator[None]: + """Publish failed storage operations on the private bus, as the process-wide bridge does. + + The bridge listens before the trail in one run and after it in the other: which of the + two hears about a failed operation first must not matter. + """ + bridge = StorageErrorBridge(bus) + if request.param == "trail first": + request.getfixturevalue("trail") + observe.add_listener(bridge) + yield + observe.remove_listener(bridge) + + +def _deny(*_args: object) -> None: + raise StoragePermissionException("memory://audit/out/report.csv: denied") + + +@pytest.mark.usefixtures("bridged") +def test_a_pipeline_run_is_recorded_once_under_one_correlation_id( + trail: AuditTrail, bus: EventBus, monkeypatch: pytest.MonkeyPatch +) -> None: + storage = MemoryStorage("audit") + published: list[Event] = [] + bus.subscribe(published.append) + with actor_scope("scheduler"), correlation_scope() as run_id: + bus.publish(PipelineStarted(source="pipeline", payload={"pipeline": "daily"})) + bus.publish( + TaskStarted(source="pipeline", payload={"pipeline": "daily", "task": "extract"}) + ) + storage.write_bytes("in/raw.csv", b"a,b\n") + storage.read_bytes("in/raw.csv") + bus.publish( + TaskCompleted( + source="pipeline", + payload={"pipeline": "daily", "task": "extract", "duration_ms": 4.0}, + ) + ) + bus.publish(TaskStarted(source="pipeline", payload={"pipeline": "daily", "task": "load"})) + monkeypatch.setattr(storage, "_upload", _deny) + with pytest.raises(StoragePermissionException): + storage.write_bytes("out/report.csv", b"x") + bus.publish( + TaskFailed( + source="pipeline", + payload={ + "pipeline": "daily", + "task": "load", + "error": "StoragePermissionException: denied", + }, + ) + ) + bus.publish(PipelineFailed(source="pipeline", payload={"pipeline": "daily"})) + records = _stored(trail) + assert [(record.source, record.action, record.status) for record in records] == [ + ("pipeline", "pipeline.started", "ok"), + ("pipeline", "task.started", "ok"), + ("storage", "upload", "ok"), + ("storage", "read", "ok"), + ("pipeline", "task.completed", "ok"), + ("pipeline", "task.started", "ok"), + ("storage", "upload", "error"), + ("pipeline", "task.failed", "error"), + ("pipeline", "pipeline.failed", "error"), + ] + assert {record.correlation_id for record in records} == {run_id} + assert {record.actor for record in records} == {"scheduler"} + assert [event.type for event in published].count("storage.error") == 1 + assert trail.count(action="storage.error") == 0 + failed = trail.search(resource_prefix="memory://audit/out/") + assert len(failed) == 1 + assert failed[0].backend == "memory" + assert failed[0].error is not None + assert failed[0].error.startswith("StoragePermissionException: ") + assert failed[0].metadata["error_type"] == "StoragePermissionException" + assert trail.count(pipeline="daily", task="load") == 2 + assert trail.count(correlation_id=run_id) == len(records) == 9 + + +@pytest.mark.usefixtures("bridged") +def test_a_caller_mistake_in_storage_is_recorded_once(trail: AuditTrail) -> None: + storage = MemoryStorage("audit") + with pytest.raises(StorageNotFoundException): + storage.read_bytes("nope.txt") + records = _stored(trail) + assert [(record.action, record.status) for record in records] == [("read", "error")] + assert records[0].correlation_id is None + + +def test_a_storage_error_from_elsewhere_is_recorded(trail: AuditTrail, bus: EventBus) -> None: + bus.publish(StorageError(source="replication", subject="replica is behind")) + assert [(record.source, record.action) for record in _stored(trail)] == [ + ("replication", "storage.error") + ] + + +def test_a_copy_between_backends_is_one_record(trail: AuditTrail) -> None: + source = MemoryStorage("audit-source") + target = MemoryStorage("audit-target") + source.write_bytes("a.txt", b"x") + target.copy_from(source, "a.txt", "copy.txt") + records = _stored(trail) + assert [record.action for record in records] == ["upload", "copy"] + assert records[1].resource == "memory://audit-target/copy.txt" + assert records[1].metadata == {"source_uri": "memory://audit-source/a.txt"} + + +def test_the_trail_listens_only_between_start_and_stop(bus: EventBus) -> None: + store = MemoryAuditStore() + trail = AuditTrail(store, bus=bus) + storage = MemoryStorage("audit") + assert trail.active is False + bus.publish(PipelineStarted()) + storage.write_bytes("before.txt", b"x") + trail.start() + trail.start() + assert trail.active is True + bus.publish(PipelineStarted()) + storage.write_bytes("during.txt", b"x") + trail.stop() + trail.stop() + assert trail.active is False + bus.publish(PipelineStarted()) + storage.write_bytes("after.txt", b"x") + assert [record.action for record in store.search()] == ["upload", "pipeline.started"] + assert store.search()[0].resource == "memory://audit/during.txt" + + +def test_a_trail_without_a_store_cannot_start(bus: EventBus) -> None: + trail = AuditTrail(bus=bus) + assert trail.store is None + with pytest.raises(AuditException, match="no store"): + trail.start() + assert trail.record("manual") is None + with pytest.raises(AuditException, match="not configured"): + trail.search() + with pytest.raises(AuditException): + trail.attach("audit.sqlite") + + +def test_record_appends_one_by_hand(trail: AuditTrail) -> None: + with actor_scope("alice"), correlation_scope("run-5"): + written = trail.record( + "approve", + resource="s3://reports/q1.csv", + backend="s3", + source="review", + metadata={"ticket": "OPS-12"}, + ) + assert written is not None + assert trail.search() == [written] + assert (written.actor, written.correlation_id) == ("alice", "run-5") + assert (written.action, written.source, written.status) == ("approve", "review", "ok") + outside = trail.record("approve", actor="bob", correlation_id="run-6") + assert outside is not None + assert (outside.actor, outside.correlation_id) == ("bob", "run-6") + with pytest.raises(AuditException, match="colour"): + trail.record("approve", colour="red") + assert trail.count() == 2 + + +class _FailingStore(MemoryAuditStore): + def __init__(self) -> None: + super().__init__() + self.attempts = 0 + + def append(self, record: AuditRecord) -> None: + self.attempts += 1 + raise OSError("the audit disk is gone") + + +def test_a_store_that_cannot_write_does_not_break_what_is_audited(bus: EventBus) -> None: + store = _FailingStore() + trail = AuditTrail(store, bus=bus) + received: list[Event] = [] + bus.subscribe(received.append) + trail.start() + try: + assert bus.publish(PipelineStarted()) == 2 + storage = MemoryStorage("audit") + info = storage.write_bytes("a.txt", b"hello") + assert info.size == 5 + assert storage.read_bytes("a.txt") == b"hello" + assert trail.record("manual") is None + finally: + trail.close() + assert len(received) == 1 + assert store.attempts == 4 + assert store.search() == [] + + +def test_a_record_that_cannot_be_built_is_dropped(trail: AuditTrail, bus: EventBus) -> None: + assert bus.publish(PipelineStarted(subject="not a mapping", payload=5)) == 1 + bus.publish(PipelineStarted()) + assert [record.action for record in _stored(trail)] == ["pipeline.started"] + + +def test_close_closes_only_a_store_the_trail_owns(tmp_path: Path, bus: EventBus) -> None: + mine = MemoryAuditStore() + trail = AuditTrail(mine, bus=bus) + trail.close() + assert trail.store is None + mine.append(_record()) + owned = SQLiteAuditStore(tmp_path / "owned.sqlite") + trail.attach(owned, owned=True) + trail.attach(mine) + with pytest.raises(AuditException): + owned.count() + assert mine.count() == 1 + + +# ---------------------------------------------------------------------- the process-wide trail + + +@pytest.fixture +def process_wide() -> Iterator[None]: + """The process-wide audit trail, left without a store as it started.""" + assert audit_trail.store is None + yield + audit_trail.close() + + +def test_the_process_wide_trail_is_inactive_until_configured() -> None: + assert audit_trail.active is False + assert audit_trail.store is None + assert audit_trail.bus is event_bus + with pytest.raises(AuditException, match="not configured"): + audit_search() + + +@pytest.mark.usefixtures("process_wide") +def test_configure_audit_with_a_path(tmp_path: Path) -> None: + path = tmp_path / "audit.sqlite" + assert configure_audit(path) is audit_trail + assert audit_trail.active is True + assert isinstance(audit_trail.store, SQLiteAuditStore) + with correlation_scope("configured-run"): + event_bus.publish(PipelineStarted(source="pipeline", subject="configured")) + MemoryStorage("audit-configured").write_bytes("a.txt", b"x") + found = audit_search(correlation_id="configured-run") + assert [entry["action"] for entry in found] == ["upload", "pipeline.started"] + assert found[1]["metadata"]["subject"] == "configured" + json.dumps(found) + first = audit_trail.store + configure_audit(tmp_path / "second.sqlite") + with pytest.raises(AuditException): + first.count() + assert audit_search(correlation_id="configured-run") == [] + + +@pytest.mark.usefixtures("process_wide") +def test_configure_audit_with_a_store() -> None: + store = MemoryAuditStore() + configure_audit(store) + assert audit_trail.store is store + event_bus.publish(PipelineStarted(source="pipeline", subject="into my store")) + audit_trail.close() + assert store.count(text="into my store") == 1 + assert audit_trail.active is False + + +@pytest.mark.usefixtures("process_wide") +def test_the_audit_actions(tmp_path: Path) -> None: + registry = ActionRegistry() + register_audit_ops(registry) + for name in ("FA_audit_configure", "FA_audit_search", "FA_audit_count", "FA_audit_purge"): + assert name in registry + executor = ActionExecutor(registry) + path = tmp_path / "actions.sqlite" + (configured,) = executor.execute_action( + [["FA_audit_configure", {"db_path": str(path)}]] + ).values() + assert configured == {"active": True, "db_path": str(path), "schema_version": 2} + with correlation_scope("action-run"): + event_bus.publish(TaskFailed(source="pipeline", subject="load failed")) + event_bus.publish(PipelineStarted(source="pipeline", subject="started")) + found, counted, purged, bad = executor.execute_action( + [ + ["FA_audit_search", {"correlation_id": "action-run", "status": "error"}], + ["FA_audit_count", {"correlation_id": "action-run"}], + ["FA_audit_purge", {"older_than_seconds": 3600}], + ["FA_audit_search", {"colour": "red"}], + ] + ).values() + assert [entry["action"] for entry in found] == ["task.failed"] + assert counted == 2 + assert purged == 0 + assert "unknown audit filter" in bad diff --git a/tests/test_config.py b/tests/test_config.py index ee87e05..a709c1a 100644 --- a/tests/test_config.py +++ b/tests/test_config.py @@ -2,13 +2,18 @@ from __future__ import annotations +import time from pathlib import Path +from unittest.mock import patch import pytest from automation_file.core.config import AutomationConfig, ConfigException +from automation_file.core.config_watcher import ConfigWatcher +from automation_file.events import EventBus, PipelineFailed, Severity, TaskFailed from automation_file.notify.manager import NotificationManager -from automation_file.notify.sinks import EmailSink, SlackSink, WebhookSink +from automation_file.notify.router import NotificationRouter, Route +from automation_file.notify.sinks import EmailSink, NotificationSink, SlackSink, WebhookSink def _write_toml(path: Path, body: str) -> None: @@ -174,3 +179,246 @@ def test_file_secret_provider_resolved_from_config(tmp_path: Path) -> None: config = AutomationConfig.load(path) sinks = config.notification_sinks() assert sinks[0].url == "https://example.com/alerts" + + +# ---------------------------------------------------------------------- notify routes + +_ROUTED = """ +[[notify.sinks]] +type = "webhook" +name = "team-alerts" +url = "https://example.com/alerts" + +[[notify.sinks]] +type = "webhook" +url = "https://example.com/everything" + +[[notify.routes]] +name = "pipeline-failures" +sinks = ["team-alerts"] +types = ["pipeline.*", "task.failed"] +sources = ["pipeline"] +min_severity = "error" +dedup_seconds = 600 +rate_limit = 10 +rate_period = 30 + +[[notify.routes]] +name = "everything" +sinks = ["webhook"] +""" + + +class _Recorder(NotificationSink): + def __init__(self, name: str) -> None: + self.name = name + self.subjects: list[str] = [] + + def send(self, subject: str, body: str, level: str = "info") -> None: + self.subjects.append(subject) + + +def _config(tmp_path: Path, body: str) -> AutomationConfig: + path = tmp_path / "automation_file.toml" + _write_toml(path, body) + return AutomationConfig.load(path) + + +def _private_router() -> NotificationRouter: + return NotificationRouter(NotificationManager(dedup_seconds=0.0), EventBus()) + + +def test_routes_are_built_from_toml(tmp_path: Path) -> None: + routes = _config(tmp_path, _ROUTED).notification_routes() + assert routes == [ + Route( + "pipeline-failures", + sinks=("team-alerts",), + types=("pipeline.*", "task.failed"), + sources=("pipeline",), + min_severity=Severity.ERROR, + dedup_seconds=600.0, + rate_limit=10, + rate_period=30.0, + ), + Route("everything", sinks=("webhook",)), + ] + assert routes[1].min_severity is Severity.WARNING + assert routes[1].dedup_seconds == pytest.approx(300.0) + + +def test_apply_to_loads_the_routes_and_starts_the_router(tmp_path: Path) -> None: + router = _private_router() + config = _config(tmp_path, _ROUTED) + assert config.apply_to(router.manager, router) == 2 + assert router.manager.names() == ("team-alerts", "webhook") + assert [route.name for route in router.routes()] == ["pipeline-failures", "everything"] + assert router.active is True + with patch("automation_file.notify.sinks.requests.post") as post: + post.return_value.status_code = 200 + router.bus.publish(TaskFailed(source="pipeline", subject="load failed")) + assert post.call_count == 2 + assert {call.args[0] for call in post.call_args_list} == { + "https://example.com/alerts", + "https://example.com/everything", + } + + +def test_apply_to_without_a_router_leaves_routing_alone(tmp_path: Path) -> None: + manager = NotificationManager(dedup_seconds=0.0) + assert _config(tmp_path, _ROUTED).apply_to(manager) == 2 + assert manager.names() == ("team-alerts", "webhook") + + +def test_a_route_may_name_a_sink_registered_in_code(tmp_path: Path) -> None: + body = """ + [[notify.routes]] + name = "to-code-sink" + sinks = ["pager"] + """ + config = _config(tmp_path, body) + with pytest.raises(ConfigException, match="unknown sink"): + config.notification_routes() + assert config.notification_routes(known_sinks=["pager"]) == [ + Route("to-code-sink", sinks=("pager",)) + ] + router = _private_router() + pager = _Recorder("pager") + router.manager.register(pager) + assert config.apply_to(router.manager, router) == 0 + router.bus.publish(PipelineFailed(source="pipeline", subject="daily failed")) + assert pager.subjects == ["[ERROR] pipeline.failed: daily failed"] + + +@pytest.mark.parametrize( + "route,message", + [ + ('name = "r"\nsinks = ["nobody"]', "unknown sink"), + ('name = "r"\nmin_severity = "fatal"', "min_severity"), + ('name = "r"\nmin_severity = 3', "min_severity"), + ('name = "r"\ncolour = "red"', "colour"), + ('sinks = ["team-alerts"]', "name"), + ('name = ""', "name"), + ('name = "r"\nrate_limit = -1', "rate_limit"), + ('name = "r"\nrate_period = 0', "rate_period"), + ('name = "r"\ndedup_seconds = "long"', "dedup_seconds"), + ('name = "r"\ntypes = [5]', "route type"), + ('name = "r"\nsinks = "team-alerts"\n\n[[notify.routes]]\nname = "r"', "declared twice"), + ], +) +def test_a_bad_route_is_rejected(tmp_path: Path, route: str, message: str) -> None: + body = f""" +[[notify.sinks]] +type = "email" +name = "team-alerts" +host = "smtp.example.com" +port = 587 +sender = "alerts@example.com" +recipients = ["ops@example.com"] + +[[notify.routes]] +{route} +""" + config = _config(tmp_path, body) + with pytest.raises(ConfigException, match=message): + config.notification_routes() + router = _private_router() + router.add_route(Route("by-hand")) + with pytest.raises(ConfigException, match=message): + config.apply_to(router.manager, router) + assert router.manager.names() == () + assert router.routes() == [Route("by-hand")] + assert router.active is False + + +@pytest.mark.parametrize( + "body,message", + [ + ('[notify]\nroutes = "all"', "array of tables"), + ('[notify]\nroutes = ["all"]', "must be a table"), + ('[notify]\nsinks = "all"', "array of tables"), + ("[notify]\nsinks = [3]", "must be a table"), + ], +) +def test_notify_tables_must_be_arrays_of_tables(tmp_path: Path, body: str, message: str) -> None: + config = _config(tmp_path, body) + with pytest.raises(ConfigException, match=message): + config.apply_to(NotificationManager()) + + +def test_applying_again_follows_the_file(tmp_path: Path) -> None: + router = _private_router() + router.manager.register(_Recorder("chat")) + router.add_route(Route("by-hand", sinks=("chat",))) + first = """ + [[notify.routes]] + name = "a" + sinks = ["chat"] + + [[notify.routes]] + name = "b" + sinks = ["chat"] + """ + second = """ + [[notify.routes]] + name = "b" + sinks = ["chat"] + min_severity = "critical" + + [[notify.routes]] + name = "c" + """ + _config(tmp_path, first).apply_to(router.manager, router) + assert [route.name for route in router.routes()] == ["by-hand", "a", "b"] + _config(tmp_path, second).apply_to(router.manager, router) + assert [route.name for route in router.routes()] == ["by-hand", "b", "c"] + assert router.routes()[1].min_severity is Severity.CRITICAL + _config(tmp_path, "# no routes any more\n").apply_to(router.manager, router) + assert router.routes() == [Route("by-hand", sinks=("chat",))] + assert router.active is True + + +def test_the_router_stops_when_the_file_leaves_it_without_routes(tmp_path: Path) -> None: + router = _private_router() + router.manager.register(_Recorder("chat")) + _config(tmp_path, '[[notify.routes]]\nname = "a"\n').apply_to(router.manager, router) + assert router.active is True + _config(tmp_path, "# no routes any more\n").apply_to(router.manager, router) + assert router.routes() == [] + assert router.active is False + + +def test_a_file_without_routes_does_not_stop_a_router_it_never_configured(tmp_path: Path) -> None: + router = _private_router() + router.start() + _config(tmp_path, "# no routes at all\n").apply_to(router.manager, router) + assert router.active is True + router.stop() + + +def test_routes_are_hot_reloaded_with_the_file(tmp_path: Path) -> None: + router = _private_router() + chat = _Recorder("chat") + router.manager.register(chat) + path = tmp_path / "automation_file.toml" + _write_toml(path, '[[notify.routes]]\nname = "errors"\nmin_severity = "error"\n') + + def apply(config: AutomationConfig) -> None: + config.apply_to(router.manager, router) + + watcher = ConfigWatcher(path, apply, interval=60.0) + try: + apply(watcher.start()) + router.bus.publish(TaskFailed(source="pipeline", subject="before reload")) + time.sleep(0.01) + _write_toml(path, '[[notify.routes]]\nname = "critical-only"\nmin_severity = "critical"\n') + assert watcher.check_once() is True + assert [route.name for route in router.routes()] == ["critical-only"] + router.bus.publish(TaskFailed(source="pipeline", subject="after reload")) + time.sleep(0.01) + _write_toml(path, '[[notify.routes]]\nname = "broken"\nmin_severity = "fatal"\n') + assert watcher.check_once() is True + assert [route.name for route in router.routes()] == ["critical-only"] + finally: + watcher.stop() + assert chat.subjects == ["[ERROR] task.failed: before reload"] diff --git a/tests/test_integrity_legacy.py b/tests/test_integrity_legacy.py index 5d3cc2c..14063a4 100644 --- a/tests/test_integrity_legacy.py +++ b/tests/test_integrity_legacy.py @@ -9,6 +9,7 @@ from __future__ import annotations from pathlib import Path +from types import SimpleNamespace from typing import Any import pytest @@ -19,7 +20,7 @@ from automation_file.events import EventBus, IntegrityViolation from automation_file.exceptions import FileAutomationException from automation_file.integrity import IntegrityException, IntegrityMonitor -from automation_file.integrity.legacy import format_body +from automation_file.integrity.legacy import LegacyHooks, format_body from automation_file.notify import NotificationManager, notification_manager from automation_file.notify.sinks import NotificationSink @@ -309,3 +310,54 @@ def test_the_notification_body_lists_the_first_paths() -> None: "modified: changed.txt", ] assert format_body({"missing": [], "modified": [], "extra": []}) == "no drift detected" + + +_DRIFT = {"matched": [], "missing": ["a.txt"], "modified": [], "extra": [], "ok": False} + + +@pytest.fixture +def shared_recorder() -> Any: + recorder = _Recorder() + notification_manager.register(recorder) + yield recorder + notification_manager.unregister(recorder.name) + + +def _router(monkeypatch: pytest.MonkeyPatch, *, active: bool) -> None: + monkeypatch.setattr( + "automation_file.notify.router.notification_router", SimpleNamespace(active=active) + ) + + +def test_an_active_router_delivers_in_place_of_the_shared_manager( + monkeypatch: pytest.MonkeyPatch, shared_recorder: _Recorder +) -> None: + _router(monkeypatch, active=True) + LegacyHooks("drift: an active router").handle(dict(_DRIFT)) + assert shared_recorder.messages == [] + + +def test_an_idle_router_leaves_the_shared_manager_notified( + monkeypatch: pytest.MonkeyPatch, shared_recorder: _Recorder +) -> None: + _router(monkeypatch, active=False) + LegacyHooks("drift: an idle router").handle(dict(_DRIFT)) + assert [subject for subject, _, _ in shared_recorder.messages] == ["drift: an idle router"] + + +def test_a_private_bus_is_not_routed_so_the_shared_manager_is_notified( + monkeypatch: pytest.MonkeyPatch, shared_recorder: _Recorder +) -> None: + _router(monkeypatch, active=True) + LegacyHooks("drift: a private bus", on_shared_bus=False).handle(dict(_DRIFT)) + assert [subject for subject, _, _ in shared_recorder.messages] == ["drift: a private bus"] + + +def test_a_manager_that_was_passed_is_notified_whatever_the_router_does( + monkeypatch: pytest.MonkeyPatch, shared_recorder: _Recorder +) -> None: + _router(monkeypatch, active=True) + manager, recorder = _recording_manager() + LegacyHooks("drift: an own manager", manager=manager).handle(dict(_DRIFT)) + assert [subject for subject, _, _ in recorder.messages] == ["drift: an own manager"] + assert shared_recorder.messages == [] diff --git a/tests/test_notify.py b/tests/test_notify.py index 7ef94ba..1730bce 100644 --- a/tests/test_notify.py +++ b/tests/test_notify.py @@ -286,3 +286,101 @@ def test_manager_list_describes_sinks() -> None: webhook_desc = next(d for d in descriptions if d["name"] == "hook-a") assert webhook_desc["type"] == "WebhookSink" assert webhook_desc["url_host"] == "example.com" + + +class _Named(NotificationSink): + def __init__(self, name: str, error: Exception | None = None) -> None: + self.name = name + self.sent: list[tuple[str, str, str]] = [] + self._error = error + + def send(self, subject: str, body: str, level: str = "info") -> None: + if self._error is not None: + raise self._error + self.sent.append((subject, body, level)) + + +def test_manager_names_are_in_registration_order() -> None: + manager = NotificationManager(dedup_seconds=0.0) + assert manager.names() == () + for name in ("mail", "chat", "pager"): + manager.register(_Named(name)) + manager.register(_Named("mail")) + assert manager.names() == ("mail", "chat", "pager") + manager.unregister("chat") + assert manager.names() == ("mail", "pager") + + +def test_send_to_delivers_to_one_named_sink() -> None: + manager = NotificationManager(dedup_seconds=60.0) + chat, mail = _Named("chat"), _Named("mail") + manager.register(chat) + manager.register(mail) + assert manager.send_to("chat", "subject", "body", "warning") == "sent" + assert manager.send_to("chat", "subject", "body", "warning") == "sent" + assert chat.sent == [("subject", "body", "warning")] * 2 + assert mail.sent == [] + assert manager.send_to("mail", "only a subject") == "sent" + assert mail.sent == [("only a subject", "", "info")] + + +def test_send_to_reports_a_failure_without_raising() -> None: + manager = NotificationManager(dedup_seconds=0.0) + manager.register(_Named("down", NotificationException("channel is down"))) + manager.register(_Named("buggy", KeyError("oops"))) + assert manager.send_to("down", "subject") == "NotificationException: channel is down" + assert manager.send_to("buggy", "subject") == "KeyError: 'oops'" + + +def test_send_to_rejects_an_unknown_sink_and_an_empty_subject() -> None: + manager = NotificationManager(dedup_seconds=0.0) + manager.register(_Named("chat")) + with pytest.raises(NotificationException, match="'pager'"): + manager.send_to("pager", "subject") + with pytest.raises(NotificationException, match="subject"): + manager.send_to("chat", "") + + +def test_a_secret_url_in_a_sink_error_is_redacted() -> None: + leak = NotificationException( + "slack sink 'slack' post failed: HTTPSConnectionPool(host='hooks.slack.com', port=443): " + "Max retries exceeded with url: /services/T000/B000/SECRET (Caused by timeout) " + "while calling https://bot:pw@hooks.slack.com/services/T000/B000/SECRET" + ) + manager = NotificationManager(dedup_seconds=0.0) + manager.register(_Named("slack", leak)) + for outcome in (manager.notify("s", "b")["slack"], manager.send_to("slack", "s")): + assert "SECRET" not in outcome + assert "bot:pw" not in outcome + assert "hooks.slack.com" in outcome + assert "(Caused by timeout)" in outcome + assert manager.notify("s", "b")["slack"].startswith("NotificationException(") + + +def test_route_actions_are_registered() -> None: + from automation_file.core.action_registry import build_default_registry + + registry = build_default_registry() + for name in ("FA_notify_route_add", "FA_notify_route_remove", "FA_notify_route_list"): + assert name in registry + + +def test_notify_on_failure_publishes_an_event() -> None: + from automation_file.events import event_bus + from automation_file.notify.manager import notify_on_failure + + published: list = [] + subscription = event_bus.subscribe(published.append, types=["scheduler.error", "system.error"]) + try: + notify_on_failure("scheduler[published]", RuntimeError("boom")) + notify_on_failure("trigger[published]", RuntimeError("boom")) + finally: + event_bus.unsubscribe(subscription) + mine = [event for event in published if "[published]" in event.subject] + assert [(event.type, event.source) for event in mine] == [ + ("scheduler.error", "scheduler"), + ("system.error", "trigger"), + ] + assert mine[0].payload["job"] == "published" + assert mine[1].payload["trigger"] == "published" + assert mine[0].payload["error"] == "RuntimeError: boom" diff --git a/tests/test_notify_router.py b/tests/test_notify_router.py new file mode 100644 index 0000000..b87d3d1 --- /dev/null +++ b/tests/test_notify_router.py @@ -0,0 +1,803 @@ +"""The notification router: routes, deduplication, rate limits, failures, notify_on_failure.""" + +from __future__ import annotations + +import dataclasses +import json +import threading +from collections.abc import Iterator + +import pytest + +from automation_file.core.action_executor import ActionExecutor +from automation_file.core.action_registry import ActionRegistry +from automation_file.events import ( + Event, + EventBus, + PipelineFailed, + PipelineStarted, + SchedulerError, + Severity, + StorageError, + SystemErrorEvent, + TaskFailed, + actor_scope, + correlation_scope, + event_bus, +) +from automation_file.notify import ( + NotificationException, + NotificationManager, + NotificationRouter, + NotificationSink, + Route, + message_for, + notification_manager, + notification_router, + register_notify_ops, +) +from automation_file.notify.manager import failure_event, notify_on_failure + + +class _Clock: + """A clock the test moves by hand.""" + + def __init__(self) -> None: + self.now = 1000.0 + + def __call__(self) -> float: + return self.now + + +class _Recorder(NotificationSink): + def __init__(self, name: str) -> None: + self.name = name + self.messages: list[tuple[str, str, str]] = [] + + def send(self, subject: str, body: str, level: str = "info") -> None: + self.messages.append((subject, body, level)) + + +class _Broken(NotificationSink): + def __init__(self, name: str, error: Exception | None = None) -> None: + self.name = name + self.calls = 0 + self._error = error or NotificationException("channel is down") + + def send(self, subject: str, body: str, level: str = "info") -> None: + self.calls += 1 + raise self._error + + +@pytest.fixture +def bus() -> EventBus: + return EventBus() + + +@pytest.fixture +def clock() -> _Clock: + return _Clock() + + +@pytest.fixture +def manager() -> NotificationManager: + return NotificationManager(dedup_seconds=0.0) + + +@pytest.fixture +def chat(manager: NotificationManager) -> _Recorder: + sink = _Recorder("chat") + manager.register(sink) + return sink + + +@pytest.fixture +def mail(manager: NotificationManager) -> _Recorder: + sink = _Recorder("mail") + manager.register(sink) + return sink + + +@pytest.fixture +def router(manager: NotificationManager, bus: EventBus, clock: _Clock) -> NotificationRouter: + return NotificationRouter(manager, bus, clock=clock) + + +def _failed(subject: str = "daily failed", source: str = "pipeline") -> PipelineFailed: + return PipelineFailed(source=source, subject=subject, payload={"pipeline": "daily"}) + + +# ---------------------------------------------------------------------- Route + + +def test_a_route_has_the_documented_defaults() -> None: + route = Route("all") + assert route.sinks == () + assert route.types == () + assert route.sources == () + assert route.min_severity is Severity.WARNING + assert route.dedup_seconds == pytest.approx(300.0) + assert route.rate_limit == 0 + assert route.rate_period == pytest.approx(60.0) + + +def test_a_route_is_frozen_and_normalises_its_fields() -> None: + route = Route( + "ops", + sinks=["chat", "mail", "chat"], + types="pipeline.*", + sources=["pipeline"], + min_severity="ERROR", + dedup_seconds=10, + ) + assert route.sinks == ("chat", "mail") + assert route.types == ("pipeline.*",) + assert route.sources == ("pipeline",) + assert route.min_severity is Severity.ERROR + assert isinstance(route.dedup_seconds, float) + with pytest.raises(dataclasses.FrozenInstanceError): + route.name = "changed" + assert hash(route) == hash(dataclasses.replace(route)) + + +@pytest.mark.parametrize( + "options", + [ + {"name": ""}, + {"name": 7}, + {"name": "r", "sinks": [""]}, + {"name": "r", "sinks": 5}, + {"name": "r", "sinks": [["nested"]]}, + {"name": "r", "types": [3]}, + {"name": "r", "types": [int]}, + {"name": "r", "sources": [None]}, + {"name": "r", "min_severity": "fatal"}, + {"name": "r", "min_severity": 2}, + {"name": "r", "dedup_seconds": -1}, + {"name": "r", "dedup_seconds": "soon"}, + {"name": "r", "dedup_seconds": float("nan")}, + {"name": "r", "rate_limit": -1}, + {"name": "r", "rate_limit": 1.5}, + {"name": "r", "rate_limit": True}, + {"name": "r", "rate_period": 0}, + ], +) +def test_a_route_rejects_bad_options(options: dict) -> None: + with pytest.raises(NotificationException): + Route(**options) + + +def test_a_route_from_a_mapping_and_back() -> None: + options = { + "name": "ops", + "sinks": ["chat"], + "types": ["task.failed", "pipeline.*"], + "sources": ["pipeline"], + "min_severity": "critical", + "dedup_seconds": 0.0, + "rate_limit": 3, + "rate_period": 30.0, + } + route = Route.from_mapping(options) + assert route.to_dict() == options + assert Route.from_mapping(route.to_dict()) == route + assert Route("classes", types=(TaskFailed,)).to_dict()["types"] == ["task.failed"] + assert Route.from_mapping({"name": "bare", "sinks": None}) == Route("bare") + + +def test_a_mapping_with_an_unknown_option_or_no_name_is_rejected() -> None: + with pytest.raises(NotificationException, match="colour"): + Route.from_mapping({"name": "ops", "colour": "red"}) + with pytest.raises(NotificationException, match="name"): + Route.from_mapping({"sinks": ["chat"]}) + + +# ---------------------------------------------------------------------- routing + + +def test_routing_by_type(router: NotificationRouter, chat: _Recorder) -> None: + router.add_route(Route("by-class", types=(TaskFailed,), dedup_seconds=0)) + router.add_route(Route("by-name", types=("storage.error",), dedup_seconds=0)) + router.add_route(Route("by-prefix", types=("pipeline.*",), dedup_seconds=0)) + assert router.handle(TaskFailed(subject="load")) == {"chat": "sent"} + assert router.handle(StorageError(subject="s3")) == {"chat": "sent"} + assert router.handle(_failed()) == {"chat": "sent"} + assert router.handle(SchedulerError(subject="nightly")) == {} + assert len(chat.messages) == 3 + + +def test_routing_by_source(router: NotificationRouter, chat: _Recorder) -> None: + router.add_route(Route("pipelines", sources=("pipeline",), dedup_seconds=0)) + assert router.handle(_failed(source="pipeline")) == {"chat": "sent"} + assert router.handle(_failed(source="scheduler")) == {} + assert len(chat.messages) == 1 + + +def test_routing_by_severity(router: NotificationRouter, chat: _Recorder) -> None: + router.add_route(Route("default")) + router.add_route(Route("loud", sinks=("chat",), min_severity=Severity.CRITICAL)) + assert router.handle(PipelineStarted(subject="info is below the default")) == {} + assert router.handle(PipelineStarted(subject="warned", severity=Severity.WARNING)) == { + "chat": "sent" + } + assert router.handle(TaskFailed(subject="errored")) == {"chat": "sent"} + assert len(chat.messages) == 2 + + +def test_a_route_without_sinks_reaches_every_sink( + router: NotificationRouter, chat: _Recorder, mail: _Recorder +) -> None: + router.add_route(Route("everyone", dedup_seconds=0)) + assert router.handle(_failed()) == {"chat": "sent", "mail": "sent"} + assert len(chat.messages) == len(mail.messages) == 1 + + +def test_a_route_with_sinks_reaches_only_those( + router: NotificationRouter, chat: _Recorder, mail: _Recorder +) -> None: + router.add_route(Route("mail-only", sinks=("mail",), dedup_seconds=0)) + assert router.handle(_failed()) == {"mail": "sent"} + assert chat.messages == [] + assert len(mail.messages) == 1 + + +def test_a_sink_on_two_routes_gets_the_event_once( + router: NotificationRouter, chat: _Recorder, mail: _Recorder +) -> None: + router.add_route(Route("first", sinks=("chat",), dedup_seconds=0)) + router.add_route(Route("second", sinks=("chat", "mail"), dedup_seconds=0)) + assert router.handle(_failed()) == {"chat": "sent", "mail": "sent"} + assert len(chat.messages) == 1 + assert len(mail.messages) == 1 + + +def test_a_later_route_may_send_what_an_earlier_one_held_back( + router: NotificationRouter, chat: _Recorder +) -> None: + router.add_route(Route("quiet", sinks=("chat",), dedup_seconds=600)) + router.add_route( + Route("critical", sinks=("chat",), min_severity=Severity.CRITICAL, dedup_seconds=0) + ) + event = SystemErrorEvent(source="system", subject="disk gone") + assert router.handle(event) == {"chat": "sent"} + assert router.handle(event) == {"chat": "sent"} + assert router.handle(TaskFailed(subject="load")) == {"chat": "sent"} + assert router.handle(TaskFailed(subject="load")) == {"chat": "dedup"} + assert len(chat.messages) == 3 + + +def test_routes_are_listed_replaced_and_removed(router: NotificationRouter) -> None: + first, second = Route("first"), Route("second", min_severity=Severity.ERROR) + router.add_route(first) + router.add_route(second) + assert router.routes() == [first, second] + replacement = Route("first", sinks=("chat",)) + router.add_route(replacement) + assert router.routes() == [replacement, second] + assert router.remove_route("first") is True + assert router.remove_route("first") is False + assert router.routes() == [second] + with pytest.raises(NotificationException): + router.add_route({"name": "not a route"}) + + +def test_sync_routes_replaces_only_the_routes_of_its_origin(router: NotificationRouter) -> None: + router.add_route(Route("by-hand")) + assert router.sync_routes([Route("a"), Route("b")], origin="config") == 0 + assert [route.name for route in router.routes()] == ["by-hand", "a", "b"] + changed = [Route("b", min_severity=Severity.ERROR), Route("c")] + assert router.sync_routes(changed, origin="config") == 1 + assert [route.name for route in router.routes()] == ["by-hand", "b", "c"] + assert router.routes()[1].min_severity is Severity.ERROR + assert router.sync_routes([], origin="config") == 2 + assert [route.name for route in router.routes()] == ["by-hand"] + with pytest.raises(NotificationException): + router.sync_routes(["not a route"], origin="config") + + +# ---------------------------------------------------------------------- the bus + + +def test_start_and_stop_subscribe_on_the_bus( + router: NotificationRouter, bus: EventBus, chat: _Recorder +) -> None: + router.add_route(Route("all", dedup_seconds=0)) + assert router.active is False + bus.publish(_failed("before start")) + assert chat.messages == [] + router.start() + router.start() + assert router.active is True + bus.publish(_failed("while active")) + assert len(chat.messages) == 1 + router.stop() + router.stop() + assert router.active is False + bus.publish(_failed("after stop")) + assert len(chat.messages) == 1 + + +def test_the_default_router_uses_the_process_wide_manager_and_bus() -> None: + assert notification_router.manager is notification_manager + assert notification_router.bus is event_bus + private = NotificationRouter() + assert private.manager is notification_manager + assert private.bus is event_bus + assert private.active is False + + +# ---------------------------------------------------------------------- the message + + +def test_the_message_is_built_from_the_event(router: NotificationRouter, chat: _Recorder) -> None: + router.add_route(Route("all", dedup_seconds=0)) + with actor_scope("scheduler"), correlation_scope("run-42"): + event = TaskFailed( + source="pipeline", + subject="load failed", + payload={"pipeline": "daily", "task": "load", "error": "OSError: disk full"}, + ) + router.handle(event) + subject, body, level = chat.messages[0] + assert subject == "[ERROR] task.failed: load failed" + assert level == "error" + for line in ( + "Severity: error", + "Type: task.failed", + "Source: pipeline", + "Subject: load failed", + "Correlation ID: run-42", + "Actor: scheduler", + "Error: OSError: disk full", + ): + assert line in body.splitlines() + assert json.loads(body.split("Event:\n", 1)[1]) == event.to_dict() + + +@pytest.mark.parametrize( + "severity,level", + [ + (Severity.INFO, "info"), + (Severity.WARNING, "warning"), + (Severity.ERROR, "error"), + (Severity.CRITICAL, "error"), + ], +) +def test_severity_maps_onto_a_level_the_sinks_accept(severity: Severity, level: str) -> None: + message = message_for(Event(source="system", subject="something", severity=severity)) + assert message.level == level + assert message.subject.startswith(f"[{severity.value.upper()}] event: ") + + +def test_a_subject_stays_one_short_line() -> None: + message = message_for(TaskFailed(source="pipeline", subject="first line\nsecond " + "x" * 500)) + assert "\n" not in message.subject + assert len(message.subject) <= 200 + assert message_for(TaskFailed(source="pipeline")).subject == ( + "[ERROR] task.failed: reported by pipeline" + ) + + +def test_a_payload_json_cannot_hold_is_still_delivered() -> None: + message = message_for(TaskFailed(subject="odd", payload={"value": object()})) + assert " None: + router.add_route(Route("all", dedup_seconds=300)) + assert router.handle(_failed()) == {"chat": "sent"} + clock.now += 299 + assert router.handle(_failed()) == {"chat": "dedup"} + clock.now += 1 + assert router.handle(_failed()) == {"chat": "sent"} + assert len(chat.messages) == 2 + + +def test_dedup_compares_type_source_and_subject( + router: NotificationRouter, chat: _Recorder +) -> None: + router.add_route(Route("all", dedup_seconds=300)) + assert router.handle(_failed("daily failed", "pipeline")) == {"chat": "sent"} + assert router.handle(_failed("weekly failed", "pipeline")) == {"chat": "sent"} + assert router.handle(_failed("daily failed", "scheduler")) == {"chat": "sent"} + assert router.handle(TaskFailed(source="pipeline", subject="daily failed")) == {"chat": "sent"} + other_payload = PipelineFailed(source="pipeline", subject="daily failed", payload={"n": 2}) + assert router.handle(other_payload) == {"chat": "dedup"} + assert len(chat.messages) == 4 + + +def test_dedup_can_be_switched_off(router: NotificationRouter, chat: _Recorder) -> None: + router.add_route(Route("all", dedup_seconds=0)) + assert [router.handle(_failed()) for _ in range(3)] == [{"chat": "sent"}] * 3 + assert len(chat.messages) == 3 + + +def test_repeats_from_many_threads_are_sent_once( + router: NotificationRouter, chat: _Recorder +) -> None: + router.add_route(Route("all", dedup_seconds=300)) + workers = 8 + barrier = threading.Barrier(workers) + outcomes: list[str] = [] + + def publish() -> None: + barrier.wait(timeout=10) + outcomes.append(router.handle(_failed())["chat"]) + + threads = [threading.Thread(target=publish) for _ in range(workers)] + for thread in threads: + thread.start() + for thread in threads: + thread.join(timeout=30) + assert sorted(outcomes) == ["dedup"] * (workers - 1) + ["sent"] + assert len(chat.messages) == 1 + + +def test_each_route_has_its_own_window( + router: NotificationRouter, chat: _Recorder, mail: _Recorder, clock: _Clock +) -> None: + router.add_route(Route("short", sinks=("chat",), dedup_seconds=10)) + router.add_route(Route("long", sinks=("mail",), dedup_seconds=100)) + router.handle(_failed()) + clock.now += 50 + assert router.handle(_failed()) == {"chat": "sent", "mail": "dedup"} + + +def test_replacing_a_route_forgets_what_it_sent( + router: NotificationRouter, chat: _Recorder +) -> None: + route = Route("all", dedup_seconds=300) + router.add_route(route) + router.handle(_failed()) + router.add_route(Route("all", dedup_seconds=300)) + assert router.handle(_failed()) == {"chat": "dedup"} + router.add_route(Route("all", dedup_seconds=301)) + assert router.handle(_failed()) == {"chat": "sent"} + assert len(chat.messages) == 2 + + +# ---------------------------------------------------------------------- rate limiting + + +def test_a_route_sends_at_most_its_limit_per_period( + router: NotificationRouter, chat: _Recorder, clock: _Clock +) -> None: + router.add_route(Route("all", dedup_seconds=0, rate_limit=2, rate_period=60)) + assert router.handle(_failed("one")) == {"chat": "sent"} + clock.now += 10 + assert router.handle(_failed("two")) == {"chat": "sent"} + clock.now += 10 + assert router.handle(_failed("three")) == {"chat": "rate_limited"} + clock.now += 40 + assert router.handle(_failed("four")) == {"chat": "sent"} + assert router.handle(_failed("five")) == {"chat": "rate_limited"} + assert [subject for subject, _, _ in chat.messages] == [ + "[ERROR] pipeline.failed: one", + "[ERROR] pipeline.failed: two", + "[ERROR] pipeline.failed: four", + ] + + +def test_the_limit_is_per_sink_and_route( + router: NotificationRouter, chat: _Recorder, mail: _Recorder +) -> None: + router.add_route(Route("tight", sinks=("chat",), dedup_seconds=0, rate_limit=1)) + router.add_route(Route("loose", sinks=("mail",), dedup_seconds=0, rate_limit=3)) + assert router.handle(_failed("one")) == {"chat": "sent", "mail": "sent"} + assert router.handle(_failed("two")) == {"chat": "rate_limited", "mail": "sent"} + assert len(chat.messages) == 1 + assert len(mail.messages) == 2 + + +def test_a_rate_limited_event_is_not_remembered_as_sent( + router: NotificationRouter, chat: _Recorder, clock: _Clock +) -> None: + router.add_route(Route("all", dedup_seconds=300, rate_limit=1, rate_period=60)) + assert router.handle(_failed("one")) == {"chat": "sent"} + assert router.handle(_failed("two")) == {"chat": "rate_limited"} + clock.now += 60 + assert router.handle(_failed("two")) == {"chat": "sent"} + assert router.handle(_failed("one")) == {"chat": "dedup"} + + +def test_a_duplicate_does_not_use_up_the_limit(router: NotificationRouter, chat: _Recorder) -> None: + router.add_route(Route("all", dedup_seconds=300, rate_limit=2, rate_period=60)) + assert router.handle(_failed("one")) == {"chat": "sent"} + assert router.handle(_failed("one")) == {"chat": "dedup"} + assert router.handle(_failed("one")) == {"chat": "dedup"} + assert router.handle(_failed("two")) == {"chat": "sent"} + assert len(chat.messages) == 2 + + +# ---------------------------------------------------------------------- failures + + +def test_one_failing_sink_does_not_affect_another( + router: NotificationRouter, manager: NotificationManager, chat: _Recorder +) -> None: + broken = _Broken("pager") + crashing = _Broken("legacy", RuntimeError("not a NotificationException")) + manager.register(broken) + manager.register(crashing) + router.add_route(Route("all", dedup_seconds=0)) + assert router.handle(_failed()) == { + "chat": "sent", + "pager": "NotificationException: channel is down", + "legacy": "RuntimeError: not a NotificationException", + } + assert len(chat.messages) == 1 + assert broken.calls == crashing.calls == 1 + + +def test_a_sink_failure_is_published_as_a_system_error( + router: NotificationRouter, manager: NotificationManager, bus: EventBus +) -> None: + manager.register(_Broken("pager")) + router.add_route(Route("ops", dedup_seconds=0)) + published: list[Event] = [] + bus.subscribe(published.append) + with correlation_scope("run-9"): + event = _failed() + router.handle(event) + assert len(published) == 1 + failure = published[0] + assert isinstance(failure, SystemErrorEvent) + assert failure.source == "notify" + assert failure.severity is Severity.CRITICAL + assert failure.subject == "notification sink 'pager' failed" + assert failure.correlation_id == "run-9" + assert failure.payload["resource"] == "pager" + assert failure.payload["route"] == "ops" + assert failure.payload["status"] == "error" + assert failure.payload["error"] == "NotificationException: channel is down" + assert failure.payload["event_type"] == "pipeline.failed" + assert failure.payload["event_id"] == event.id + + +def test_a_failure_event_is_never_routed( + router: NotificationRouter, manager: NotificationManager, bus: EventBus, chat: _Recorder +) -> None: + broken = _Broken("pager") + manager.register(broken) + router.add_route(Route("everything", min_severity=Severity.INFO, dedup_seconds=0)) + router.start() + published: list[Event] = [] + bus.subscribe(published.append) + bus.publish(_failed()) + assert [event.type for event in published] == ["system.error", "pipeline.failed"] + assert broken.calls == 1 + assert len(chat.messages) == 1 + failure = published[0] + assert router.handle(failure) == {} + assert broken.calls == 1 + + +def test_two_routers_on_one_bus_do_not_feed_each_other( + manager: NotificationManager, bus: EventBus, clock: _Clock +) -> None: + broken = _Broken("pager") + manager.register(broken) + routers = [NotificationRouter(manager, bus, clock=clock) for _ in range(2)] + published: list[Event] = [] + bus.subscribe(published.append) + for router in routers: + router.add_route(Route("everything", min_severity=Severity.INFO, dedup_seconds=0)) + router.start() + bus.publish(_failed()) + assert broken.calls == 2 + assert sorted(event.type for event in published) == [ + "pipeline.failed", + "system.error", + "system.error", + ] + + +def test_another_system_error_is_routed(router: NotificationRouter, chat: _Recorder) -> None: + router.add_route(Route("all", dedup_seconds=0)) + assert router.handle(SystemErrorEvent(source="system", subject="out of disk")) == { + "chat": "sent" + } + assert chat.messages[0][2] == "error" + + +def test_a_route_to_an_unknown_sink_reports_the_failure( + router: NotificationRouter, bus: EventBus, chat: _Recorder +) -> None: + router.add_route(Route("typo", sinks=("chat", "chta"), dedup_seconds=0)) + published: list[Event] = [] + bus.subscribe(published.append) + outcomes = router.handle(_failed()) + assert outcomes["chat"] == "sent" + assert outcomes["chta"].startswith("NotificationException: no notification sink") + assert [event.payload["resource"] for event in published] == ["chta"] + + +def test_a_secret_url_never_reaches_the_outcome_or_the_event( + router: NotificationRouter, manager: NotificationManager, bus: EventBus +) -> None: + leak = ( + "post failed: HTTPSConnectionPool(host='api.telegram.org', port=443): Max retries " + "exceeded with url: /bot123456:TOPSECRET/sendMessage; see " + "https://user:hunter2@hooks.example.com/services/T000/B000/XXXX?token=abc" + ) + manager.register(_Broken("telegram", NotificationException(leak))) + router.add_route(Route("all", dedup_seconds=0)) + published: list[Event] = [] + bus.subscribe(published.append) + outcome = router.handle(_failed())["telegram"] + reported = json.dumps(published[0].to_dict()) + for text in (outcome, reported): + assert "TOPSECRET" not in text + assert "hunter2" not in text + assert "XXXX" not in text + assert "token=abc" not in text + assert "hooks.example.com" in text + assert "api.telegram.org" in text + + +# ---------------------------------------------------------------------- notify_on_failure + + +@pytest.fixture +def process_wide() -> Iterator[list[Event]]: + """The process-wide manager, router and bus, put back the way they were.""" + routes, was_active = notification_router.routes(), notification_router.active + notification_manager.unregister_all() + published: list[Event] = [] + subscription = event_bus.subscribe(published.append) + yield published + event_bus.unsubscribe(subscription) + notification_router.stop() + for route in notification_router.routes(): + notification_router.remove_route(route.name) + for route in routes: + notification_router.add_route(route) + if was_active: + notification_router.start() + notification_manager.unregister_all() + + +def test_failure_event_for_a_scheduler_job() -> None: + event = failure_event("scheduler[nightly]", RuntimeError("disk full")) + assert isinstance(event, SchedulerError) + assert event.source == "scheduler" + assert event.severity is Severity.ERROR + assert event.subject == "scheduler[nightly] failed" + assert event.payload["job"] == "nightly" + assert event.payload["status"] == "error" + assert event.payload["error"] == "RuntimeError: disk full" + + +def test_failure_event_for_any_other_context() -> None: + trigger = failure_event("trigger[inbox]", ValueError("bad action")) + assert isinstance(trigger, SystemErrorEvent) + assert trigger.source == "trigger" + assert trigger.payload["trigger"] == "inbox" + assert trigger.payload["error"] == "ValueError: bad action" + plain = failure_event("nightly backup", OSError("no space")) + assert isinstance(plain, SystemErrorEvent) + assert plain.source == "system" + assert plain.subject == "nightly backup failed" + assert plain.payload["context"] == "nightly backup" + odd = failure_event("error[x]", OSError("no space")) + assert odd.payload["error"] == "OSError: no space" + + +def test_notify_on_failure_without_the_router_notifies_directly( + process_wide: list[Event], +) -> None: + sink = _Recorder("chat") + notification_manager.register(sink) + assert notification_router.active is False + notify_on_failure("scheduler[router-off]", RuntimeError("disk full")) + assert sink.messages == [ + ("automation_file: scheduler[router-off] failed", "RuntimeError('disk full')", "error") + ] + assert [event.type for event in process_wide] == ["scheduler.error"] + assert process_wide[0].payload["job"] == "router-off" + + +def test_notify_on_failure_with_the_router_notifies_once_through_it( + process_wide: list[Event], +) -> None: + sink = _Recorder("chat") + notification_manager.register(sink) + notification_router.add_route(Route("failures", types=("scheduler.error", "system.error"))) + notification_router.start() + notify_on_failure("scheduler[router-on]", RuntimeError("disk full")) + notify_on_failure("trigger[router-on]", RuntimeError("disk full")) + assert [(subject, level) for subject, _, level in sink.messages] == [ + ("[ERROR] scheduler.error: scheduler[router-on] failed", "error"), + ("[CRITICAL] system.error: trigger[router-on] failed", "error"), + ] + assert "RuntimeError: disk full" in sink.messages[0][1] + assert [event.type for event in process_wide] == ["scheduler.error", "system.error"] + + +def test_notify_on_failure_with_the_router_obeys_its_routes(process_wide: list[Event]) -> None: + sink = _Recorder("chat") + notification_manager.register(sink) + notification_router.add_route(Route("pipelines-only", types=("pipeline.*",))) + notification_router.start() + notify_on_failure("scheduler[unrouted]", RuntimeError("disk full")) + assert sink.messages == [] + assert [event.type for event in process_wide] == ["scheduler.error"] + + +def test_notify_on_failure_without_sinks_still_publishes(process_wide: list[Event]) -> None: + notify_on_failure("scheduler[no-sinks]", RuntimeError("disk full")) + assert [event.subject for event in process_wide] == ["scheduler[no-sinks] failed"] + + +# ---------------------------------------------------------------------- actions + + +def test_the_route_actions(process_wide: list[Event]) -> None: + registry = ActionRegistry() + register_notify_ops(registry) + for name in ("FA_notify_route_add", "FA_notify_route_remove", "FA_notify_route_list"): + assert name in registry + sink = _Recorder("chat") + notification_manager.register(sink) + executor = ActionExecutor(registry) + added, listed = executor.execute_action( + [ + [ + "FA_notify_route_add", + { + "name": "ops", + "sinks": ["chat"], + "types": ["pipeline.*"], + "min_severity": "error", + "dedup_seconds": 0, + "rate_limit": 5, + }, + ], + ["FA_notify_route_list"], + ] + ).values() + assert added["name"] == "ops" + assert added["rate_limit"] == 5 + assert listed == [added] + assert notification_router.active is True + event_bus.publish(_failed("through the action")) + assert [subject for subject, _, _ in sink.messages] == [ + "[ERROR] pipeline.failed: through the action" + ] + removed, missing = executor.execute_action( + [["FA_notify_route_remove", {"name": "ops"}], ["FA_notify_route_remove", ["ops"]]] + ).values() + assert (removed, missing) == (True, False) + assert notification_router.routes() == [] + assert notification_router.active is False + + +def test_removing_an_unknown_route_leaves_a_started_router_alone( + process_wide: list[Event], +) -> None: + registry = ActionRegistry() + register_notify_ops(registry) + notification_router.start() + (removed,) = ( + ActionExecutor(registry).execute_action([["FA_notify_route_remove", ["nobody"]]]).values() + ) + assert removed is False + assert notification_router.active is True + + +def test_a_bad_route_action_is_reported_not_stored(process_wide: list[Event]) -> None: + registry = ActionRegistry() + register_notify_ops(registry) + (result,) = ( + ActionExecutor(registry) + .execute_action([["FA_notify_route_add", {"name": "ops", "min_severity": "fatal"}]]) + .values() + ) + assert "min_severity" in result + assert notification_router.routes() == [] + assert notification_router.active is False diff --git a/tests/test_operational_metrics.py b/tests/test_operational_metrics.py new file mode 100644 index 0000000..2aac2f2 --- /dev/null +++ b/tests/test_operational_metrics.py @@ -0,0 +1,230 @@ +"""Operational metrics: events, notifications and storage operations.""" + +from __future__ import annotations + +from collections.abc import Iterator + +import pytest +from prometheus_client import REGISTRY + +from automation_file.core import metrics +from automation_file.core.metrics import ( + install_operational_metrics, + record_event, + record_notification, + record_storage_operation, + render, + uninstall_operational_metrics, +) +from automation_file.events import ( + EventBus, + PipelineStarted, + Severity, + TaskFailed, + correlation_scope, + event_bus, +) +from automation_file.exceptions import StorageNotFoundException +from automation_file.notify import ( + NotificationException, + NotificationManager, + NotificationRouter, + NotificationSink, + Route, +) +from automation_file.storage import MemoryStorage +from automation_file.storage.observe import StorageOperation + +_EVENTS = "automation_file_events_total" +_NOTIFICATIONS = "automation_file_notifications_total" +_OPERATIONS = "automation_file_storage_operations_total" +_DURATION = "automation_file_storage_operation_duration_seconds" + + +def _sample(name: str, **labels: str) -> float: + return REGISTRY.get_sample_value(name, labels) or 0.0 + + +@pytest.fixture +def bus() -> Iterator[EventBus]: + """A private bus that feeds the metrics, removed again afterwards.""" + private = EventBus() + assert install_operational_metrics(private) is True + yield private + assert uninstall_operational_metrics(private) is True + + +class _Sink(NotificationSink): + def __init__(self, name: str, fail: bool = False) -> None: + self.name = name + self._fail = fail + + def send(self, subject: str, body: str, level: str = "info") -> None: + if self._fail: + raise NotificationException("channel is down") + + +def test_published_events_are_counted_by_type_and_severity(bus: EventBus) -> None: + failed = _sample(_EVENTS, type="task.failed", severity="error") + started = _sample(_EVENTS, type="pipeline.started", severity="info") + warned = _sample(_EVENTS, type="pipeline.started", severity="warning") + bus.publish(TaskFailed(source="pipeline", subject="load failed")) + bus.publish(PipelineStarted()) + bus.publish(PipelineStarted()) + bus.publish(PipelineStarted(severity=Severity.WARNING)) + assert _sample(_EVENTS, type="task.failed", severity="error") == failed + 1 + assert _sample(_EVENTS, type="pipeline.started", severity="info") == started + 2 + assert _sample(_EVENTS, type="pipeline.started", severity="warning") == warned + 1 + + +def test_installing_twice_counts_an_event_once(bus: EventBus) -> None: + assert install_operational_metrics(bus) is False + before = _sample(_EVENTS, type="task.failed", severity="error") + bus.publish(TaskFailed()) + assert _sample(_EVENTS, type="task.failed", severity="error") == before + 1 + + +def test_uninstalling_stops_the_counting() -> None: + private = EventBus() + assert uninstall_operational_metrics(private) is False + install_operational_metrics(private) + assert uninstall_operational_metrics(private) is True + before = _sample(_EVENTS, type="task.failed", severity="error") + private.publish(TaskFailed()) + assert _sample(_EVENTS, type="task.failed", severity="error") == before + + +def test_the_process_wide_bus_is_the_default() -> None: + installed_here = install_operational_metrics() + try: + assert install_operational_metrics() is False + before = _sample(_EVENTS, type="pipeline.started", severity="info") + event_bus.publish(PipelineStarted(subject="metrics on the process-wide bus")) + assert _sample(_EVENTS, type="pipeline.started", severity="info") == before + 1 + finally: + if installed_here: + assert uninstall_operational_metrics() is True + + +@pytest.mark.usefixtures("bus") +def test_storage_operations_are_counted_with_their_duration() -> None: + uploads = _sample(_OPERATIONS, operation="upload", backend="memory", status="ok") + reads = _sample(_OPERATIONS, operation="read", backend="memory", status="ok") + misses = _sample(_OPERATIONS, operation="read", backend="memory", status="error") + timed = _sample(f"{_DURATION}_count", operation="read", backend="memory") + spent = _sample(f"{_DURATION}_sum", operation="read", backend="memory") + storage = MemoryStorage("metrics") + storage.write_bytes("a.txt", b"x") + storage.read_bytes("a.txt") + with pytest.raises(StorageNotFoundException): + storage.read_bytes("missing.txt") + assert _sample(_OPERATIONS, operation="upload", backend="memory", status="ok") == uploads + 1 + assert _sample(_OPERATIONS, operation="read", backend="memory", status="ok") == reads + 1 + assert _sample(_OPERATIONS, operation="read", backend="memory", status="error") == misses + 1 + assert _sample(f"{_DURATION}_count", operation="read", backend="memory") == timed + 2 + assert _sample(f"{_DURATION}_sum", operation="read", backend="memory") >= spent + + +def test_a_duration_is_observed_in_seconds() -> None: + labels = {"operation": "copy", "backend": "metrics-seconds"} + record_storage_operation( + StorageOperation( + operation="copy", + uri="metrics-seconds://bucket/a.txt", + backend="metrics-seconds", + status="ok", + duration_ms=2500.0, + ) + ) + assert _sample(f"{_DURATION}_sum", **labels) == pytest.approx(2.5) + assert _sample(f"{_DURATION}_bucket", le="2.5", **labels) == 1 + assert _sample(f"{_DURATION}_bucket", le="1.0", **labels) == 0 + + +def test_notifications_are_counted_by_sink_and_outcome() -> None: + manager = NotificationManager(dedup_seconds=60.0) + manager.register(_Sink("metrics-good")) + manager.register(_Sink("metrics-bad", fail=True)) + manager.notify("subject", "body", "info") + manager.notify("subject", "body", "info") + manager.send_to("metrics-good", "direct") + assert _sample(_NOTIFICATIONS, sink="metrics-good", outcome="sent") == 2 + assert _sample(_NOTIFICATIONS, sink="metrics-good", outcome="dedup") == 1 + assert _sample(_NOTIFICATIONS, sink="metrics-bad", outcome="error") == 1 + assert _sample(_NOTIFICATIONS, sink="metrics-bad", outcome="dedup") == 1 + assert _sample(_NOTIFICATIONS, sink="metrics-bad", outcome="sent") == 0 + + +def test_the_router_counts_what_it_holds_back() -> None: + manager = NotificationManager(dedup_seconds=0.0) + manager.register(_Sink("metrics-routed")) + router = NotificationRouter(manager, EventBus()) + router.add_route(Route("limited", types=("task.failed",), rate_limit=1)) + router.add_route(Route("typo", sinks=("metrics-missing",), types=("task.failed",))) + router.handle(TaskFailed(subject="one")) + router.handle(TaskFailed(subject="one")) + router.handle(TaskFailed(subject="two")) + assert _sample(_NOTIFICATIONS, sink="metrics-routed", outcome="sent") == 1 + assert _sample(_NOTIFICATIONS, sink="metrics-routed", outcome="dedup") == 1 + assert _sample(_NOTIFICATIONS, sink="metrics-routed", outcome="rate_limited") == 1 + assert _sample(_NOTIFICATIONS, sink="metrics-missing", outcome="error") == 2 + assert _sample(_NOTIFICATIONS, sink="metrics-missing", outcome="dedup") == 1 + + +def test_no_label_carries_a_path_or_a_correlation_id(bus: EventBus) -> None: + with correlation_scope("metrics-correlation-id"): + bus.publish(TaskFailed(source="pipeline", subject="metrics-secret-subject")) + MemoryStorage("metrics-private-bucket").write_bytes("metrics-private-file.txt", b"x") + exposition = render()[0].decode("utf-8") + for name in (_EVENTS, _NOTIFICATIONS, _OPERATIONS, _DURATION): + assert name in exposition + for leaked in ( + "metrics-correlation-id", + "metrics-secret-subject", + "metrics-private-bucket", + "metrics-private-file.txt", + ): + assert leaked not in exposition + label_names = { + label + for metric in REGISTRY.collect() + if metric.name.startswith("automation_file_") + for sample in metric.samples + for label in sample.labels + } + assert label_names <= { + "action", + "status", + "type", + "severity", + "sink", + "outcome", + "operation", + "backend", + "le", + } + + +def test_a_label_takes_a_bounded_number_of_values() -> None: + label = metrics._BoundedLabel(limit=2) # pylint: disable=protected-access + assert [label(value) for value in ("a", "b", "c", "a", "d", "b", "", None)] == [ + "a", + "b", + "other", + "a", + "other", + "b", + "other", + "other", + ] + assert metrics.MAX_LABEL_VALUES == 100 + roomy = metrics._BoundedLabel() # pylint: disable=protected-access + assert roomy("") == "unknown" + + +def test_recording_never_raises() -> None: + record_event(object()) + record_storage_operation(object()) + record_notification("metrics-odd", "sent") + record_notification("", "") + assert _sample(_NOTIFICATIONS, sink="unknown", outcome="unknown") >= 1 From bfa6f19e116024b35239a580a29240333c80564c Mon Sep 17 00:00:00 2001 From: JeffreyChen Date: Thu, 8 Oct 2026 14:01:02 +0800 Subject: [PATCH 35/59] feat: add the pipeline runtime Pipelines run tasks in dependency order with retry, timeout, cancellation, conditions, idempotency, checkpoint and resume, a dry run and an execution history, from Python or from a versioned YAML or JSON definition. FA_storage_verify can raise on a mismatch (strict), so a verification can fail its task. The action ACL and the MCP server now check action names nested in the arguments of another action. --- CLAUDE.md | 5 + README.md | 62 ++ README.zh-CN.md | 59 ++ README.zh-TW.md | 59 ++ architecture.md | 9 + automation_file/__init__.py | 32 + automation_file/core/action_registry.py | 7 + automation_file/exceptions.py | 4 + automation_file/pipeline/__init__.py | 73 ++ automation_file/pipeline/actions.py | 110 +++ automation_file/pipeline/definition.py | 608 +++++++++++++++ automation_file/pipeline/errors.py | 25 + automation_file/pipeline/graph.py | 122 +++ automation_file/pipeline/model.py | 353 +++++++++ automation_file/pipeline/pipeline.py | 479 ++++++++++++ automation_file/pipeline/reporting.py | 100 +++ automation_file/pipeline/runner.py | 356 +++++++++ automation_file/pipeline/store.py | 360 +++++++++ automation_file/pipeline/substitution.py | 164 ++++ automation_file/pipeline/worker.py | 267 +++++++ automation_file/server/action_acl.py | 42 +- automation_file/server/mcp_server.py | 16 + automation_file/storage/actions.py | 20 +- docs/source/API/api_index.rst | 14 + docs/source/API/pipeline.rst | 60 ++ docs/source/Eng/eng_index.rst | 15 + docs/source/Eng/usage/pipeline.rst | 737 +++++++++++++++++ docs/source/Eng/usage/storage.rst | 8 +- docs/source/Zh-CN/usage/pipeline.rst | 696 +++++++++++++++++ docs/source/Zh-CN/usage/storage.rst | 7 +- docs/source/Zh-CN/zh_cn_index.rst | 14 + docs/source/Zh-TW/usage/pipeline.rst | 696 +++++++++++++++++ docs/source/Zh-TW/usage/storage.rst | 7 +- docs/source/Zh-TW/zh_tw_index.rst | 14 + docs/updates/2026-10.md | 34 + docs/updates/README.md | 4 +- progress.md | 1 - tests/test_action_acl.py | 41 + tests/test_mcp_server.py | 37 + tests/test_pipeline_actions.py | 308 ++++++++ tests/test_pipeline_definition.py | 861 ++++++++++++++++++++ tests/test_pipeline_events.py | 443 +++++++++++ tests/test_pipeline_imports.py | 112 +++ tests/test_pipeline_rejected.py | 188 +++++ tests/test_pipeline_run.py | 955 +++++++++++++++++++++++ tests/test_pipeline_store.py | 783 +++++++++++++++++++ tests/test_storage_actions.py | 4 + 47 files changed, 9357 insertions(+), 14 deletions(-) create mode 100644 automation_file/pipeline/__init__.py create mode 100644 automation_file/pipeline/actions.py create mode 100644 automation_file/pipeline/definition.py create mode 100644 automation_file/pipeline/errors.py create mode 100644 automation_file/pipeline/graph.py create mode 100644 automation_file/pipeline/model.py create mode 100644 automation_file/pipeline/pipeline.py create mode 100644 automation_file/pipeline/reporting.py create mode 100644 automation_file/pipeline/runner.py create mode 100644 automation_file/pipeline/store.py create mode 100644 automation_file/pipeline/substitution.py create mode 100644 automation_file/pipeline/worker.py create mode 100644 docs/source/API/pipeline.rst create mode 100644 docs/source/Eng/usage/pipeline.rst create mode 100644 docs/source/Zh-CN/usage/pipeline.rst create mode 100644 docs/source/Zh-TW/usage/pipeline.rst create mode 100644 tests/test_pipeline_actions.py create mode 100644 tests/test_pipeline_definition.py create mode 100644 tests/test_pipeline_events.py create mode 100644 tests/test_pipeline_imports.py create mode 100644 tests/test_pipeline_rejected.py create mode 100644 tests/test_pipeline_run.py create mode 100644 tests/test_pipeline_store.py diff --git a/CLAUDE.md b/CLAUDE.md index 528717a..d3a87e2 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -41,6 +41,9 @@ automation_file/ ├── server/ # tcp_server, http_server, mcp_server (MCP over stdio), web_ui, metrics_server, │ # action_acl (ActionACL), network_guards (ensure_loopback) ├── client/ # HTTPActionClient for the HTTP action server +├── pipeline/ # Pipeline runtime: model (Task, RetryPolicy, PipelineRun, ...), graph, pipeline +│ # (Pipeline), runner + worker, substitution, store (RunStore, MemoryRunStore, +│ # SQLiteRunStore), definition (YAML/JSON + PIPELINE_SCHEMA), reporting, actions ├── audit/ # Audit schema v2: record (AuditRecord), store (AuditStore, AuditQuery, │ # MemoryAuditStore), sqlite_store (SQLiteAuditStore), trail (AuditTrail, │ # audit_trail, configure_audit), actions (FA_audit_*) @@ -82,6 +85,7 @@ automation_file/ - `File(uri)` / `Storage(uri)` — the universal storage layer's application API: one file, one directory, in any backend. Both resolve their backend on every call through `StorageResolver` (`Storage.mount`, `Storage.register_scheme`). - `StorageBackend` — the contract a storage backend implements. The public operations (`exists`, `stat`, `list_dir`, `mkdir`, `upload`, `download`, `delete`, `checksum`, `read_bytes`, `write_bytes`, `copy_from`, `move_from`) are template methods; a backend supplies only the `_`-prefixed primitives. Twelve are built in: `LocalStorage`, `MemoryStorage`, `S3Storage` and `AzureStorage` (both on `ObjectStorage`), `SFTPStorage` and `FTPStorage` (both on `SessionStorage`), `GoogleDriveStorage`, `OneDriveStorage`, `DropboxStorage`, and the mounted `WebDAVStorage`, `SMBStorage` and `FsspecStorage`. Each uses its backend's shared client singleton unless given one, and reports a missing SDK with the extra to install. - `IntegrityMonitor` — compares a tree at any storage URI with an approved baseline (`create_baseline`, `verify`, `accept`, `watch`, `start` / `stop`, `snapshot`) and returns a `DriftReport`; drift is published as one `IntegrityViolation` per pass. It only reads unless a `RemediationPolicy` is passed. Its options are keyword arguments (`MonitorKeywords`). The first monitor's call and `check_once()` summary are kept, including the notification through `manager` or the process-wide `notification_manager`. +- `Pipeline` — tasks (a callable taking a `TaskContext`, or an `FA_*` action) with `depends_on`, run in dependency order with `RetryPolicy`, a timeout, `when` conditions and idempotency keys. `run` executes in the calling thread, `start` in the background, `resume(run_id)` repeats only what did not succeed, `run(dry_run=True)` plans. Every transition is checkpointed in a `RunStore` (`MemoryRunStore`, `SQLiteRunStore`) and reported as a `pipeline.*` / `task.*` event. A task fails only by raising: an action that reports through its return value needs its raising form (`FA_storage_verify` with `strict=True`). - `NotificationRouter` / `Route` / `notification_router` — delivers events to named sinks by type, source and minimum severity, with deduplication and a rate limit per route and sink. Opt-in: nothing is routed until a route exists and the router is started (`FA_notify_route_add` and `AutomationConfig.apply_to(manager, router)` start it). While it is active, `notify_on_failure` and the integrity monitor leave the direct notification to it, so nothing is announced twice. - `AuditTrail` / `audit_trail` / `configure_audit(path)` — audit schema v2: one `AuditRecord` per event and per storage operation in an `AuditStore` (`SQLiteAuditStore`, `MemoryAuditStore`), searched with `audit_search` / `FA_audit_search`. Records nothing until configured, and never raises into the code it audits. The v1 `AuditLog` is unchanged. - `Event` / `EventBus` / `event_bus` — every component reports through events (`PipelineFailed`, `TaskFailed`, `IntegrityViolation`, `StorageError`, ...) with a severity, a correlation ID and an actor; consumers subscribe on the bus by class, type name or prefix. New code that has something to report publishes an event; it does not call a notification sink or the audit log directly. @@ -159,6 +163,7 @@ All code must follow secure-by-default principles. Review every change against t - Do not remove the loopback guard to "make it easier to test remotely". The server dispatches arbitrary registry commands; exposing it to the network is equivalent to exposing a Python REPL. - The server accepts a single JSON payload per connection (`recv(8192)`). Do not raise that limit without also adding a length-framed protocol. - `quit_server` triggers an orderly shutdown; do not add an administrative bypass that skips the loopback check. +- `ActionACL` checks every registered action name anywhere in a request, so an action nested in the arguments of another (`FA_execute_action`, a pipeline definition, a scheduled list) is covered; the MCP server applies the same check against the tools it exposes (`nested_action_names`). Neither can see what a request only points to: an action file, a definition file, a stored pipeline run. A new action that runs other actions from such a place must say so in its documentation, and must never be added to a default allow list. - Optional `shared_secret=` enforces an `AUTH \n` prefix; the comparison uses `hmac.compare_digest` (constant time). Never log the secret or the raw payload. ### HTTP server diff --git a/README.md b/README.md index 1241791..6ca1414 100644 --- a/README.md +++ b/README.md @@ -52,6 +52,7 @@ facade. - **Event bus** — one `Event` model with ten core events (`pipeline.*`, `task.*`, `integrity.violation`, `storage.error`, `scheduler.error`, `system.error`), severities, correlation IDs and actors; subscribe on `event_bus` by class, type or prefix - **Notification router** — routes decide which sinks hear about which events (by type, source and minimum severity), with deduplication and rate limiting per route; declare them in code, in `automation_file.toml` or with `FA_notify_route_*` - **Audit trail** — `configure_audit(path)` records one row per event and per storage operation (actor, source, pipeline, task, action, resource, backend, status, duration, correlation ID), searchable with `audit_search` / `FA_audit_search` +- **Pipelines** — `Pipeline` runs tasks (callables or `FA_*` actions) in dependency order, independent ones in parallel, with retry, timeout, cancellation, conditions, idempotency keys, checkpoint and resume, a dry run and an execution history; definitions in Python, YAML or JSON - PySide6 GUI (`python -m automation_file ui`) with a tab per backend, the JSON-action runner, and dedicated tabs for Triggers, Scheduler, and live Progress - Rich CLI with one-shot subcommands plus legacy JSON-batch flags - Project scaffolding (`ProjectBuilder`) for executor-based automations @@ -624,6 +625,67 @@ audit_search(status="error", resource_prefix="s3://reports/", limit=20) `FA_audit_purge`; `install_operational_metrics()` adds Prometheus counters for events, notifications and storage operations. +### Pipelines + +`automation_file.pipeline` runs tasks in dependency order, with independent tasks +in parallel, and records every step. A task is a Python callable or an `FA_*` +action; a pipeline is built in Python or loaded from a YAML / JSON definition. A +run reports only through `pipeline.*` and `task.*` events on the event bus. + +```python +from automation_file import Pipeline, RetryPolicy, SQLiteRunStore + +def check(ctx): + if ctx.results["download"]["size"] == 0: + raise ValueError("the report is empty") + +pipeline = Pipeline("daily-report", max_workers=4) +pipeline.task( + "download", + ["FA_storage_copy", {"source": "s3://input/${params.date}.csv", + "target": "local:///tmp/report.csv"}], + retry=RetryPolicy(max_attempts=3, backoff_base=1.0, backoff_cap=30.0), + timeout=300.0, +) +pipeline.task("check", check, depends_on=["download"]) +pipeline.task( + "publish", + ["FA_storage_copy", {"source": "local:///tmp/report.csv", + "target": "azure://reports/${params.date}.csv"}], + depends_on=["check"], + idempotency_key="publish-${params.date}", # at most once per date +) +pipeline.task( + "withdraw", # clean-up when publish failed + ["FA_storage_delete", {"uri": "azure://reports/${params.date}.csv", + "missing_ok": True}], + depends_on=["publish"], + when="on_failure", +) + +store = SQLiteRunStore("pipelines.db") +run = pipeline.run(params={"date": "2026-10-08"}, store=store) +if run.status != "succeeded": + run = pipeline.resume(run.run_id, store=store) # keeps what succeeded +``` + +- **Retry, timeout, cancellation.** `RetryPolicy` retries transient errors with + capped exponential back-off; a task past its `timeout` is marked `timeout` and + the run goes on; `pipeline.start()` runs in the background and `run.cancel()` + stops it. +- **Conditions and idempotency.** `when` is `on_success`, `on_failure`, `always` + or a callable; an `idempotency_key` skips a task that already succeeded under + the same key and reuses its result. +- **Checkpoint, resume, history.** Every task transition is written to a + `RunStore` (`MemoryRunStore`, `SQLiteRunStore`); `resume(run_id)` runs only what + did not succeed, and `store.list_runs()` is the execution history. +- **Definitions.** `Pipeline.from_file("daily-report.yaml")`, `from_dict` / + `to_dict`, `validate_definition()` with the path of every problem, and + `PIPELINE_SCHEMA` (JSON Schema). `run(dry_run=True)` plans without executing. +- **Actions.** `FA_pipeline_run`, `FA_pipeline_validate`, `FA_pipeline_status`, + `FA_pipeline_history` and `FA_pipeline_resume` for JSON action lists, the CLI, + the action servers and MCP. + ### File-watcher triggers Run an action list whenever a filesystem event fires on a watched path: diff --git a/README.zh-CN.md b/README.zh-CN.md index 835304c..2e3c37c 100644 --- a/README.zh-CN.md +++ b/README.zh-CN.md @@ -50,6 +50,7 @@ TCP / HTTP 服务器执行的 JSON 驱动动作。内附 PySide6 GUI，每个功 - **事件总线** — 单一 `Event` 模型与十种核心事件（`pipeline.*`、`task.*`、`integrity.violation`、`storage.error`、`scheduler.error`、`system.error`），具备严重程度、关联 ID 与 actor；可以在 `event_bus` 上按类、type 或前缀订阅 - **通知路由器** — 以路由决定哪些事件（按类型、来源与最低严重程度）发送到哪些 sink，每条路由各自去重与限流；可在代码、`automation_file.toml` 或通过 `FA_notify_route_*` 声明 - **审计轨迹** — `configure_audit(path)` 为每个事件与每次存储操作记录一条（actor、来源、pipeline、task、动作、资源、后端、状态、耗时、关联 ID），可用 `audit_search` / `FA_audit_search` 查询 +- **流水线（Pipeline）** — `Pipeline` 按依赖顺序执行任务（可调用对象或 `FA_*` 动作），互不依赖者并行执行，并支持重试、超时、取消、条件、幂等键、检查点与续跑、试运行以及执行历史；定义可以用 Python、YAML 或 JSON 编写 - PySide6 GUI（`python -m automation_file ui`）每个后端一个页签，含 JSON 动作执行器，另有 Triggers、Scheduler、实时 Progress 专属页签 - 功能丰富的 CLI，包含一次性子命令与旧式 JSON 批量标志 - 项目脚手架（`ProjectBuilder`）协助构建以 executor 为核心的自动化项目 @@ -611,6 +612,64 @@ audit_search(status="error", resource_prefix="s3://reports/", limit=20) `FA_audit_purge`；`install_operational_metrics()` 会加入事件、通知与存储操作的 Prometheus 计数器。 +### 流水线（Pipeline） + +`automation_file.pipeline` 按依赖顺序执行任务，互不依赖的任务并行执行，并记录每一个 +步骤。任务可以是 Python 可调用对象或 `FA_*` 动作；流水线可以用 Python 构建，也可以从 +YAML / JSON 定义载入。一次运行只通过事件总线上的 `pipeline.*` 与 `task.*` 事件报告。 + +```python +from automation_file import Pipeline, RetryPolicy, SQLiteRunStore + +def check(ctx): + if ctx.results["download"]["size"] == 0: + raise ValueError("the report is empty") + +pipeline = Pipeline("daily-report", max_workers=4) +pipeline.task( + "download", + ["FA_storage_copy", {"source": "s3://input/${params.date}.csv", + "target": "local:///tmp/report.csv"}], + retry=RetryPolicy(max_attempts=3, backoff_base=1.0, backoff_cap=30.0), + timeout=300.0, +) +pipeline.task("check", check, depends_on=["download"]) +pipeline.task( + "publish", + ["FA_storage_copy", {"source": "local:///tmp/report.csv", + "target": "azure://reports/${params.date}.csv"}], + depends_on=["check"], + idempotency_key="publish-${params.date}", # 同一个日期最多一次 +) +pipeline.task( + "withdraw", # publish 失败时清理 + ["FA_storage_delete", {"uri": "azure://reports/${params.date}.csv", + "missing_ok": True}], + depends_on=["publish"], + when="on_failure", +) + +store = SQLiteRunStore("pipelines.db") +run = pipeline.run(params={"date": "2026-10-08"}, store=store) +if run.status != "succeeded": + run = pipeline.resume(run.run_id, store=store) # 保留已成功的部分 +``` + +- **重试、超时、取消。** `RetryPolicy` 以有上限的指数退避重试暂时性错误；超过 + `timeout` 的任务会被标记为 `timeout`，运行则继续进行；`pipeline.start()` 在后台 + 运行，`run.cancel()` 可以停止它。 +- **条件与幂等。** `when` 可以是 `on_success`、`on_failure`、`always` 或可调用对象； + `idempotency_key` 会跳过已经以相同的键成功过的任务，并沿用它的结果。 +- **检查点、续跑、历史。** 任务的每一次状态转换都会写入 `RunStore` + （`MemoryRunStore`、`SQLiteRunStore`）；`resume(run_id)` 只执行尚未成功的部分， + `store.list_runs()` 就是运行历史。 +- **定义文件。** `Pipeline.from_file("daily-report.yaml")`、`from_dict` / `to_dict`、 + 会报告每一项问题路径的 `validate_definition()`，以及 `PIPELINE_SCHEMA` + （JSON Schema）。`run(dry_run=True)` 只规划而不执行。 +- **动作。** `FA_pipeline_run`、`FA_pipeline_validate`、`FA_pipeline_status`、 + `FA_pipeline_history` 与 `FA_pipeline_resume`，可用于 JSON 动作列表、CLI、动作 + 服务器与 MCP。 + ### 文件监听触发 每当被监听路径发生文件系统事件，就执行动作清单： diff --git a/README.zh-TW.md b/README.zh-TW.md index da8fc06..6add872 100644 --- a/README.zh-TW.md +++ b/README.zh-TW.md @@ -50,6 +50,7 @@ TCP / HTTP 伺服器執行的 JSON 驅動動作。內附 PySide6 GUI，每個功 - **事件匯流排** — 單一 `Event` 模型與十種核心事件（`pipeline.*`、`task.*`、`integrity.violation`、`storage.error`、`scheduler.error`、`system.error`），具備嚴重程度、關聯 ID 與 actor；可在 `event_bus` 上依類別、type 或前綴訂閱 - **通知路由器** — 以路由決定哪些事件（依類型、來源與最低嚴重程度）送到哪些 sink，每條路由各自去重與限流；可在程式、`automation_file.toml` 或以 `FA_notify_route_*` 宣告 - **稽核軌跡** — `configure_audit(path)` 為每個事件與每次儲存操作記錄一筆（actor、來源、pipeline、task、動作、資源、後端、狀態、耗時、關聯 ID），可用 `audit_search` / `FA_audit_search` 查詢 +- **管線（Pipeline）** — `Pipeline` 依相依順序執行任務（可呼叫物件或 `FA_*` 動作），互不相依者平行執行，並支援重試、逾時、取消、條件、冪等鍵、檢查點與續跑、試跑以及執行歷史；定義可用 Python、YAML 或 JSON 撰寫 - PySide6 GUI（`python -m automation_file ui`）每個後端一個分頁，含 JSON 動作執行器，另有 Triggers、Scheduler、即時 Progress 專屬分頁 - 功能豐富的 CLI，包含一次性子指令與舊式 JSON 批次旗標 - 專案鷹架（`ProjectBuilder`）協助建立以 executor 為核心的自動化專案 @@ -611,6 +612,64 @@ audit_search(status="error", resource_prefix="s3://reports/", limit=20) `FA_audit_purge`；`install_operational_metrics()` 會加入事件、通知與儲存操作的 Prometheus 計數器。 +### 管線（Pipeline） + +`automation_file.pipeline` 依相依順序執行任務，互不相依的任務平行執行，並記錄每一個 +步驟。任務可以是 Python 可呼叫物件或 `FA_*` 動作；管線可以用 Python 建立，也可以從 +YAML / JSON 定義載入。一次執行只透過事件匯流排上的 `pipeline.*` 與 `task.*` 事件回報。 + +```python +from automation_file import Pipeline, RetryPolicy, SQLiteRunStore + +def check(ctx): + if ctx.results["download"]["size"] == 0: + raise ValueError("the report is empty") + +pipeline = Pipeline("daily-report", max_workers=4) +pipeline.task( + "download", + ["FA_storage_copy", {"source": "s3://input/${params.date}.csv", + "target": "local:///tmp/report.csv"}], + retry=RetryPolicy(max_attempts=3, backoff_base=1.0, backoff_cap=30.0), + timeout=300.0, +) +pipeline.task("check", check, depends_on=["download"]) +pipeline.task( + "publish", + ["FA_storage_copy", {"source": "local:///tmp/report.csv", + "target": "azure://reports/${params.date}.csv"}], + depends_on=["check"], + idempotency_key="publish-${params.date}", # 同一個日期最多一次 +) +pipeline.task( + "withdraw", # publish 失敗時清理 + ["FA_storage_delete", {"uri": "azure://reports/${params.date}.csv", + "missing_ok": True}], + depends_on=["publish"], + when="on_failure", +) + +store = SQLiteRunStore("pipelines.db") +run = pipeline.run(params={"date": "2026-10-08"}, store=store) +if run.status != "succeeded": + run = pipeline.resume(run.run_id, store=store) # 保留已成功的部分 +``` + +- **重試、逾時、取消。** `RetryPolicy` 以有上限的指數退避重試暫時性錯誤；超過 + `timeout` 的任務會被標記為 `timeout`，執行則繼續進行；`pipeline.start()` 在背景 + 執行，`run.cancel()` 可以停止它。 +- **條件與冪等。** `when` 可以是 `on_success`、`on_failure`、`always` 或可呼叫物件； + `idempotency_key` 會略過已經以相同的鍵成功過的任務，並沿用它的結果。 +- **檢查點、續跑、歷史。** 任務的每一次狀態轉換都會寫入 `RunStore` + （`MemoryRunStore`、`SQLiteRunStore`）；`resume(run_id)` 只執行尚未成功的部分， + `store.list_runs()` 就是執行歷史。 +- **定義檔。** `Pipeline.from_file("daily-report.yaml")`、`from_dict` / `to_dict`、 + 會回報每一項問題路徑的 `validate_definition()`，以及 `PIPELINE_SCHEMA` + （JSON Schema）。`run(dry_run=True)` 只規劃而不執行。 +- **動作。** `FA_pipeline_run`、`FA_pipeline_validate`、`FA_pipeline_status`、 + `FA_pipeline_history` 與 `FA_pipeline_resume`，可用於 JSON 動作清單、CLI、動作 + 伺服器與 MCP。 + ### 檔案監看觸發 每當被監看路徑發生檔案系統事件，就執行動作清單： diff --git a/architecture.md b/architecture.md index 2df035e..f37da6c 100644 --- a/architecture.md +++ b/architecture.md @@ -30,6 +30,7 @@ piece of the 1.0 roadmap (`docs/FILEAUTOMATION-1.0-ROADMAP.md`, PR #107); what i | `automation_file/server/` | `tcp_server.py`, `http_server.py`, `mcp_server.py`, `web_ui.py`, `metrics_server.py`, `action_acl.py` (`ActionACL`), `network_guards.py` (`ensure_loopback`) | | `automation_file/client/` | `HTTPActionClient` for the HTTP action server | | `automation_file/trigger/`, `scheduler/`, `notify/` | Watchdog file triggers, cron scheduler, notification sinks. Each registers its own `FA_*` ops. `notify/router.py` (`Route`, `NotificationRouter`, the process-wide `notification_router`) subscribes on the event bus and delivers events to named sinks by type, source and minimum severity, with deduplication and a rate limit per route and sink; a failing sink becomes a `system.error` event from the source `notify`, which is never routed | +| `automation_file/pipeline/` | The pipeline runtime. `model.py` (`Task`, `TaskContext`, `RetryPolicy`, `Schedule`, `PipelineRun`, `TaskRun`, `RunStatus`, `TaskStatus`), `graph.py` (dependency order, cycles), `pipeline.py` (`Pipeline`: `task`, `run`, `start`, `resume`, `from_file` / `from_dict` / `to_dict`, `problems` / `validate`), `runner.py` and `worker.py` (one daemon thread per running task, capped at `max_workers`; retry, timeout, cancellation, conditions, idempotency), `substitution.py` (`${params.x}`, `${tasks.id.result}`), `store.py` (`RunStore`, `MemoryRunStore`, `SQLiteRunStore`: checkpoints and history), `definition.py` (`load_definition`, `validate_definition`, `PIPELINE_SCHEMA`), `reporting.py` (the `pipeline.*` and `task.*` events), `actions.py` (`FA_pipeline_*`). `core/dag_executor.py` is the older, unrecorded DAG helper and is unchanged | | `automation_file/audit/` | Audit schema v2. `record.py` (`AuditRecord`, built from an event or from a storage operation), `store.py` (`AuditStore`, `AuditQuery`, `MemoryAuditStore`), `sqlite_store.py` (`SQLiteAuditStore`: parameterised SQL, a schema-version table, `import_v1`), `trail.py` (`AuditTrail`, the process-wide `audit_trail`, `configure_audit`), `actions.py` (`FA_audit_*`). The trail records nothing until it is configured; the v1 `core/audit.py` `AuditLog` is unchanged | | `automation_file/project/` | `ProjectBuilder`, `create_project_dir` | | `automation_file/ui/` | PySide6 GUI: `launcher.launch_ui`, `main_window.MainWindow`, `worker.ActionWorker`, `log_widget.LogPanel`, `tabs/` (backend panels are grouped under `TransferTab`) | @@ -67,6 +68,13 @@ piece of the 1.0 roadmap (`docs/FILEAUTOMATION-1.0-ROADMAP.md`, PR #107); what i `FA_integrity_accept`, `FA_integrity_watch_start`, `FA_integrity_watch_stop`, `FA_integrity_status`. The first monitor's call, `IntegrityMonitor(root, manifest_path, interval=, on_drift=, manager=, alert_on_extra=)`, and its `check_once()` summary are kept. +- **Pipelines** (same facade): `Pipeline`, `Task`, `TaskContext`, `RetryPolicy`, `PipelineRun`, `TaskRun`, + `RunStatus`, `TaskStatus`, `RunStore`, `MemoryRunStore`, `SQLiteRunStore`, `PipelineException`, + `PipelineDefinitionException`, `register_pipeline_ops`; `Schedule`, `load_definition`, + `validate_definition` and `PIPELINE_SCHEMA` are in `automation_file.pipeline`. Actions: + `FA_pipeline_run`, `FA_pipeline_validate`, `FA_pipeline_status`, `FA_pipeline_history`, + `FA_pipeline_resume`. A run reports only through `pipeline.*` and `task.*` events, with the run ID as + the correlation ID. - **Notification routes and audit** (same facade): `Route`, `NotificationRouter`, `notification_router`; `AuditRecord`, `AuditQuery`, `AuditStore`, `SQLiteAuditStore`, `MemoryAuditStore`, `AuditTrail`, `audit_trail`, `configure_audit`, `audit_search`, `register_audit_ops`; `install_operational_metrics` @@ -141,6 +149,7 @@ MCP host → automation_file_mcp (stdio JSON-RPC) → tools/call → MCPServer r ActionExecutor() → build_default_registry(): local + http + utils + drive commands → _register_cloud_backends (register__ops) → trigger / scheduler / progress / notify ops → storage ops (FA_storage_*) → integrity ops (FA_integrity_*) → audit ops (FA_audit_*) + → pipeline ops (FA_pipeline_*) → _load_plugins (entry points; may override built-ins) → executor adds FA_execute_action, FA_execute_files, FA_execute_action_parallel, FA_validate ``` diff --git a/automation_file/__init__.py b/automation_file/__init__.py index 922a546..65618fe 100644 --- a/automation_file/__init__.py +++ b/automation_file/__init__.py @@ -120,6 +120,7 @@ DiffException, OneDriveException, StorageAlreadyExistsException, + StorageChecksumException, StorageException, StorageNotEmptyException, StorageNotFoundException, @@ -234,6 +235,22 @@ notify_send, register_notify_ops, ) +from automation_file.pipeline import ( + MemoryRunStore, + Pipeline, + PipelineDefinitionException, + PipelineException, + PipelineRun, + RetryPolicy, + RunStatus, + RunStore, + SQLiteRunStore, + Task, + TaskContext, + TaskRun, + TaskStatus, + register_pipeline_ops, +) from automation_file.project.project_builder import ProjectBuilder, create_project_dir from automation_file.remote.azure_blob import ( AzureBlobClient, @@ -578,6 +595,21 @@ def __getattr__(name: str) -> Any: "StorageTransientException", "StorageUnavailableException", "StorageUnsupportedException", + "StorageChecksumException", + "Pipeline", + "Task", + "TaskContext", + "RetryPolicy", + "PipelineRun", + "TaskRun", + "RunStatus", + "TaskStatus", + "RunStore", + "MemoryRunStore", + "SQLiteRunStore", + "PipelineException", + "PipelineDefinitionException", + "register_pipeline_ops", # Events "Event", "EventBus", diff --git a/automation_file/core/action_registry.py b/automation_file/core/action_registry.py index 52d4ba9..37d28ea 100644 --- a/automation_file/core/action_registry.py +++ b/automation_file/core/action_registry.py @@ -224,6 +224,12 @@ def _register_storage_ops(registry: ActionRegistry) -> None: register_storage_ops(registry) +def _register_pipeline_ops(registry: ActionRegistry) -> None: + from automation_file.pipeline.actions import register_pipeline_ops + + register_pipeline_ops(registry) + + def _register_audit_ops(registry: ActionRegistry) -> None: from automation_file.audit.actions import register_audit_ops @@ -256,6 +262,7 @@ def build_default_registry() -> ActionRegistry: _register_storage_ops(registry) _register_integrity_ops(registry) _register_audit_ops(registry) + _register_pipeline_ops(registry) _load_plugins(registry) # DEBUG, not INFO: this runs at import, and INFO is mirrored to stderr, so every import -- # `python -m automation_file --help` included -- printed it. diff --git a/automation_file/exceptions.py b/automation_file/exceptions.py index 09b6958..3343ee8 100644 --- a/automation_file/exceptions.py +++ b/automation_file/exceptions.py @@ -195,6 +195,10 @@ class StorageUnsupportedException(StorageException): """Raised when a backend cannot perform the requested operation.""" +class StorageChecksumException(StorageException): + """Raised when a file does not have the digest a strict verification expected.""" + + _ARGPARSE_EMPTY_MESSAGE = "argparse received no actionable argument" _BAD_TRIGGER_FUNCTION = "trigger name is not registered in the executor" _BAD_CALLBACK_METHOD = "callback_param_method must be 'kwargs' or 'args'" diff --git a/automation_file/pipeline/__init__.py b/automation_file/pipeline/__init__.py new file mode 100644 index 0000000..92ce1f6 --- /dev/null +++ b/automation_file/pipeline/__init__.py @@ -0,0 +1,73 @@ +"""Pipelines: tasks with dependencies, run in order with retry, timeout, resume and history. + +* :class:`Pipeline` holds the tasks; ``run`` executes them in the calling thread, + ``start`` in the background, ``resume`` continues a stored run. +* A task is a callable taking a :class:`TaskContext`, or an ``FA_*`` action. + :class:`RetryPolicy`, a timeout, a ``when`` condition and an idempotency key + say how it runs. +* A :class:`PipelineRun` and its :class:`TaskRun` entries say what happened. A + :class:`RunStore` records them: :class:`MemoryRunStore` or :class:`SQLiteRunStore`. +* :func:`validate_definition`, :data:`PIPELINE_SCHEMA` and ``Pipeline.from_file`` + cover pipelines written as YAML or JSON. +* :func:`register_pipeline_ops` adds the ``FA_pipeline_*`` actions to a registry. + +A run reports only through events on the bus (``pipeline.*`` and ``task.*``, all +carrying the run ID as their correlation ID). +""" + +from __future__ import annotations + +from automation_file.pipeline.actions import register_pipeline_ops +from automation_file.pipeline.definition import ( + PIPELINE_SCHEMA, + RETRYABLE_EXCEPTIONS, + SCHEMA_VERSION, + load_definition, + validate_definition, +) +from automation_file.pipeline.errors import PipelineDefinitionException, PipelineException +from automation_file.pipeline.model import ( + DEFAULT_RETRY_ON, + PipelineRun, + RetryPolicy, + RunStatus, + Schedule, + Task, + TaskContext, + TaskRun, + TaskStatus, +) +from automation_file.pipeline.pipeline import Pipeline +from automation_file.pipeline.store import ( + MemoryRunStore, + RunStore, + SQLiteRunStore, + default_run_store, + set_default_run_store, +) + +__all__ = [ + "DEFAULT_RETRY_ON", + "PIPELINE_SCHEMA", + "RETRYABLE_EXCEPTIONS", + "SCHEMA_VERSION", + "MemoryRunStore", + "Pipeline", + "PipelineDefinitionException", + "PipelineException", + "PipelineRun", + "RetryPolicy", + "RunStatus", + "RunStore", + "SQLiteRunStore", + "Schedule", + "Task", + "TaskContext", + "TaskRun", + "TaskStatus", + "default_run_store", + "load_definition", + "register_pipeline_ops", + "set_default_run_store", + "validate_definition", +] diff --git a/automation_file/pipeline/actions.py b/automation_file/pipeline/actions.py new file mode 100644 index 0000000..cc44599 --- /dev/null +++ b/automation_file/pipeline/actions.py @@ -0,0 +1,110 @@ +"""``FA_pipeline_*`` actions: pipelines for JSON action lists. + +Each function takes a definition as a mapping or as the path of a YAML / JSON +file and returns JSON-friendly values, so the same call works from Python, an +action file, the CLI, the TCP and HTTP action servers and as an MCP tool: + +.. code-block:: json + + [ + ["FA_pipeline_validate", {"definition": "pipelines/daily-report.yaml"}], + ["FA_pipeline_run", {"definition": "pipelines/daily-report.yaml", + "params": {"date": "2026-10-08"}}] + ] + +Runs are recorded in the default run store (:func:`set_default_run_store`), which +is where ``FA_pipeline_status``, ``FA_pipeline_history`` and ``FA_pipeline_resume`` +look them up. + +A definition names the actions its tasks call, so ``FA_pipeline_run`` and +``FA_pipeline_resume`` can reach every registered action. On an action server +allow them only for clients that may call all of those. +""" + +from __future__ import annotations + +import os +from collections.abc import Callable, Mapping +from typing import TYPE_CHECKING, Any + +from automation_file.pipeline.definition import load_definition, validate_definition +from automation_file.pipeline.errors import PipelineDefinitionException, PipelineException +from automation_file.pipeline.pipeline import Pipeline +from automation_file.pipeline.store import default_run_store + +if TYPE_CHECKING: + from automation_file.core.action_registry import ActionRegistry + +Definition = Mapping[str, Any] | str | os.PathLike[str] + + +def _document(definition: Definition) -> Any: + """Return the definition document: the mapping itself, or the content of the file.""" + if isinstance(definition, Mapping): + return definition + if isinstance(definition, (str, os.PathLike)): + return load_definition(definition) + raise PipelineDefinitionException( + f"definition: expected a mapping or a file path, got {type(definition).__name__}" + ) + + +def pipeline_run( + definition: Definition, + params: dict[str, Any] | None = None, + dry_run: bool = False, +) -> dict[str, Any]: + """Run a pipeline definition (a mapping or a YAML/JSON file path) and return the run. + + A failed task does not raise: read ``status`` and ``tasks`` of the result. + ``dry_run`` plans the run without executing or recording anything. + """ + pipeline = Pipeline.from_dict(_document(definition)) + return pipeline.run(params=params, dry_run=dry_run).to_dict() + + +def pipeline_validate(definition: Definition) -> dict[str, Any]: + """Check a pipeline definition and return ``{"valid": ..., "errors": [...]}``. + + A file that cannot be read or parsed is reported the same way instead of raising. + """ + try: + errors = validate_definition(_document(definition)) + except PipelineDefinitionException as error: + errors = list(error.problems) + return {"valid": not errors, "errors": errors} + + +def pipeline_status(run_id: str) -> dict[str, Any]: + """Return the recorded state of the run ``run_id`` and of its tasks.""" + run = default_run_store().get_run(run_id) + if run is None: + raise PipelineException(f"unknown run {run_id!r}") + return run.to_dict() + + +def pipeline_history(pipeline: str | None = None, limit: int = 20) -> list[dict[str, Any]]: + """Return the latest recorded runs, newest first, of one pipeline or of all.""" + return [run.to_dict() for run in default_run_store().list_runs(pipeline, limit)] + + +def pipeline_resume(run_id: str, definition: Definition) -> dict[str, Any]: + """Continue the recorded run ``run_id``: keep what succeeded and run the rest.""" + pipeline = Pipeline.from_dict(_document(definition)) + return pipeline.resume(run_id).to_dict() + + +def pipeline_commands() -> dict[str, Callable[..., Any]]: + """Return every ``FA_pipeline_*`` action by name.""" + return { + "FA_pipeline_run": pipeline_run, + "FA_pipeline_validate": pipeline_validate, + "FA_pipeline_status": pipeline_status, + "FA_pipeline_history": pipeline_history, + "FA_pipeline_resume": pipeline_resume, + } + + +def register_pipeline_ops(registry: ActionRegistry) -> None: + """Register every ``FA_pipeline_*`` command into ``registry``.""" + registry.register_many(pipeline_commands()) diff --git a/automation_file/pipeline/definition.py b/automation_file/pipeline/definition.py new file mode 100644 index 0000000..f5cf904 --- /dev/null +++ b/automation_file/pipeline/definition.py @@ -0,0 +1,608 @@ +"""Pipeline definitions as documents: the schema, the validator and the file loader. + +A definition is a mapping (from YAML, JSON or Python) with ``schema_version: 1``. +:func:`validate_definition` checks one by hand and returns every problem with the +path of the offending entry; :data:`PIPELINE_SCHEMA` describes the same shape as +a JSON Schema document for editors and other tools. +""" + +from __future__ import annotations + +import json +import math +import os +from collections.abc import Callable, Mapping +from pathlib import Path +from types import MappingProxyType +from typing import Any + +from automation_file import exceptions +from automation_file.pipeline.errors import PipelineDefinitionException +from automation_file.pipeline.graph import dependency_problems, upstream_tasks +from automation_file.pipeline.model import ( + DEFAULT_RETRY_ON, + ON_SUCCESS, + WHEN_CHOICES, + RetryPolicy, + Task, +) +from automation_file.pipeline.substitution import ( + NAME_PATTERN, + NAME_RULE, + action_reference_problems, + is_name, + text_problems, +) + +SCHEMA_VERSION = 1 +_TOP_KEYS = ("schema_version", "name", "description", "max_workers", "schedule", "params", "tasks") +_TASK_KEYS = ("action", "depends_on", "retry", "timeout", "when", "idempotency_key") +_RETRY_KEYS = ("max_attempts", "backoff", "backoff_cap", "on") +_SCHEDULE_KEYS = ("cron", "timezone") +_YAML_SUFFIXES = (".yaml", ".yml") +_JSON_SUFFIX = ".json" +_MAX_YAML_NODES = 200_000 +_MAX_ACTION_ITEMS = 2 +_MISSING: Any = object() +_JSON_SCALARS = (str, int, float, bool, type(None)) +_DEFAULT_RETRY = RetryPolicy() +_CALLABLE_NOT_WRITABLE = "a Python callable cannot be written to a definition" +_YAML_ON_HINT = ' (YAML reads a bare on as true; write "on")' +_TOO_DEEP = "the definition is nested too deeply" + + +def _retryable_exceptions() -> dict[str, type[BaseException]]: + found: dict[str, type[BaseException]] = { + name: member + for name, member in vars(exceptions).items() + if isinstance(member, type) and issubclass(member, exceptions.FileAutomationException) + } + found.update({kind.__name__: kind for kind in (TimeoutError, ConnectionError, OSError)}) + return dict(sorted(found.items())) + + +#: The exception classes a definition may name under ``retry.on``, by name. +RETRYABLE_EXCEPTIONS: Mapping[str, type[BaseException]] = MappingProxyType(_retryable_exceptions()) + + +def _schema() -> dict[str, Any]: + return { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "urn:automation_file:pipeline:1", + "title": "automation_file pipeline definition", + "type": "object", + "required": ["schema_version", "name", "tasks"], + "additionalProperties": False, + "properties": { + "schema_version": {"const": SCHEMA_VERSION}, + "name": {"type": "string", "pattern": "\\S"}, + "description": {"type": "string"}, + "max_workers": {"type": "integer", "minimum": 1}, + "schedule": {"$ref": "#/$defs/schedule"}, + "params": {"type": "object", "propertyNames": {"pattern": NAME_PATTERN}}, + "tasks": { + "type": "object", + "minProperties": 1, + "propertyNames": {"pattern": NAME_PATTERN}, + "additionalProperties": {"$ref": "#/$defs/task"}, + }, + }, + "$defs": { + "schedule": { + "type": "object", + "required": ["cron"], + "additionalProperties": False, + "properties": { + "cron": {"type": "string", "pattern": "\\S"}, + "timezone": {"type": "string", "pattern": "\\S"}, + }, + }, + "action": { + "type": "array", + "minItems": 1, + "maxItems": _MAX_ACTION_ITEMS, + "prefixItems": [ + {"type": "string", "minLength": 1}, + {"type": ["object", "array"]}, + ], + }, + "retry": { + "type": "object", + "additionalProperties": False, + "properties": { + "max_attempts": {"type": "integer", "minimum": 1}, + "backoff": {"type": "number", "minimum": 0}, + "backoff_cap": {"type": "number", "minimum": 0}, + "on": {"type": "array", "items": {"enum": list(RETRYABLE_EXCEPTIONS)}}, + }, + }, + "task": { + "type": "object", + "required": ["action"], + "additionalProperties": False, + "properties": { + "action": {"$ref": "#/$defs/action"}, + "depends_on": { + "type": "array", + "items": {"type": "string"}, + "uniqueItems": True, + }, + "retry": {"$ref": "#/$defs/retry"}, + "timeout": {"type": "number", "exclusiveMinimum": 0}, + "when": {"enum": list(WHEN_CHOICES)}, + "idempotency_key": {"type": "string", "minLength": 1}, + }, + }, + }, + } + + +#: JSON Schema (draft 2020-12) of a ``schema_version: 1`` definition. +PIPELINE_SCHEMA: dict[str, Any] = _schema() + + +# ---------------------------------------------------------------------- small checks + + +def _kind(value: Any) -> str: + return type(value).__name__ + + +def _is_integer(value: Any) -> bool: + return isinstance(value, int) and not isinstance(value, bool) + + +def _is_number(value: Any) -> bool: + return isinstance(value, (int, float)) and not isinstance(value, bool) and math.isfinite(value) + + +def _is_text(value: Any) -> bool: + return isinstance(value, str) and bool(value.strip()) + + +def _unknown_keys(mapping: Mapping[Any, Any], allowed: tuple[str, ...], prefix: str) -> list[str]: + return [f"{prefix}{key}: unknown key" for key in mapping if key not in allowed] + + +def action_problems(action: Any, path: str) -> list[str]: + """Return what keeps ``action`` from being one of the three action shapes.""" + if not isinstance(action, list) or not action: + found = "an empty list" if isinstance(action, list) else _kind(action) + return [f"{path}: expected [name], [name, {{kwargs}}] or [name, [args]], got {found}"] + problems: list[str] = [] + if not (isinstance(action[0], str) and action[0]): + problems.append(f"{path}[0]: expected an action name, got {_kind(action[0])}") + if len(action) > _MAX_ACTION_ITEMS: + problems.append(f"{path}: expected a name and at most one argument set, got {len(action)}") + elif len(action) == _MAX_ACTION_ITEMS: + problems.extend(_arguments_problems(action[1], f"{path}[1]")) + return problems + + +def _arguments_problems(arguments: Any, path: str) -> list[str]: + if isinstance(arguments, list): + return [] + if not isinstance(arguments, dict): + return [f"{path}: expected a mapping or a list of arguments, got {_kind(arguments)}"] + return [ + f"{path}: argument name {key!r} is not a string" + for key in arguments + if not isinstance(key, str) + ] + + +def _json_problems(value: Any, path: str) -> list[str]: + """Return the values inside ``value`` that JSON cannot hold, such as an unquoted YAML date.""" + if isinstance(value, _JSON_SCALARS): + return [] + if isinstance(value, list): + return [ + problem + for index, item in enumerate(value) + for problem in _json_problems(item, f"{path}[{index}]") + ] + if isinstance(value, Mapping): + return [ + problem + for key, item in value.items() + for problem in _json_problems(item, f"{path}.{key}") + ] + return [f"{path}: expected a JSON value, got {_kind(value)} (quote dates and times in YAML)"] + + +def depends_on_problems(value: Any, path: str) -> list[str]: + """Return the shape problems of a ``depends_on`` list (the graph is checked separately).""" + if not isinstance(value, (list, tuple)): + return [f"{path}: expected a list of task IDs, got {_kind(value)}"] + return [ + f"{path}[{index}]: expected a task ID, got {_kind(item)}" + for index, item in enumerate(value) + if not isinstance(item, str) + ] + + +def timeout_problems(value: Any, path: str) -> list[str]: + """Return the problem of a ``timeout`` that is not a positive number of seconds.""" + if _is_number(value) and value > 0: + return [] + return [f"{path}: expected a number of seconds > 0, got {value!r}"] + + +def when_problems(value: Any, path: str) -> list[str]: + """Return the problem of a ``when`` that is not one of the named conditions.""" + if isinstance(value, str) and value in WHEN_CHOICES: + return [] + return [f"{path}: expected one of {', '.join(WHEN_CHOICES)}, got {value!r}"] + + +def idempotency_key_problems(value: Any, path: str) -> list[str]: + """Return the problems of an idempotency key: its type and its placeholders.""" + if not (isinstance(value, str) and value): + return [f"{path}: expected a non-empty string, got {value!r}"] + return [f"{path}: {problem}" for problem in text_problems(value, None)] + + +def _retry_problems(value: Any, path: str) -> list[str]: + if not isinstance(value, Mapping): + return [f"{path}: expected a mapping, got {_kind(value)}"] + problems = [ + f"{path}.{key}: unknown key{_YAML_ON_HINT if key is True else ''}" + for key in value + if key not in _RETRY_KEYS + ] + attempts = value.get("max_attempts", 1) + if not _is_integer(attempts) or attempts < 1: + problems.append(f"{path}.max_attempts: expected an integer >= 1, got {attempts!r}") + for key in ("backoff", "backoff_cap"): + seconds = value.get(key, 0) + if not _is_number(seconds) or seconds < 0: + problems.append(f"{path}.{key}: expected a number of seconds >= 0, got {seconds!r}") + if "on" in value: + problems.extend(_retry_on_problems(value["on"], f"{path}.on")) + return problems + + +def _retry_on_problems(names: Any, path: str) -> list[str]: + if not isinstance(names, list): + return [f"{path}: expected a list of exception names, got {_kind(names)}"] + return [ + f"{path}[{index}]: unknown exception {name!r}" + for index, name in enumerate(names) + if not (isinstance(name, str) and name in RETRYABLE_EXCEPTIONS) + ] + + +_TASK_CHECKS: Mapping[str, Callable[[Any, str], list[str]]] = MappingProxyType( + { + "action": action_problems, + "depends_on": depends_on_problems, + "retry": _retry_problems, + "timeout": timeout_problems, + "when": when_problems, + "idempotency_key": idempotency_key_problems, + } +) + + +# ---------------------------------------------------------------------- the document + + +def _version_problem(document: Mapping[Any, Any]) -> str | None: + if "schema_version" not in document: + return f"schema_version: required (supported: {SCHEMA_VERSION})" + version = document["schema_version"] + if isinstance(version, bool) or version != SCHEMA_VERSION: + return f"schema_version: unsupported version {version!r} (supported: {SCHEMA_VERSION})" + return None + + +def _cron_problem(cron: Any) -> str | None: + from automation_file.scheduler.cron import CronException, CronExpression + + if not isinstance(cron, str): + return f"schedule.cron: expected a cron expression, got {_kind(cron)}" + try: + CronExpression.parse(cron) + except CronException as error: + return f"schedule.cron: {str(error).removeprefix('cron: ')}" + return None + + +def _schedule_problems(schedule: Any) -> list[str]: + if not isinstance(schedule, Mapping): + return [f"schedule: expected a mapping, got {_kind(schedule)}"] + problems = _unknown_keys(schedule, _SCHEDULE_KEYS, "schedule.") + if "cron" not in schedule: + problems.append("schedule.cron: required") + else: + cron = _cron_problem(schedule["cron"]) + if cron is not None: + problems.append(cron) + if "timezone" in schedule and not _is_text(schedule["timezone"]): + problems.append( + f"schedule.timezone: expected a time zone name, got {schedule['timezone']!r}" + ) + return problems + + +def params_problems(params: Any) -> list[str]: + """Return the problems of a ``params`` mapping: its type and its names.""" + if not isinstance(params, Mapping): + return [f"params: expected a mapping, got {_kind(params)}"] + return [ + f"params.{name}: invalid parameter name, {NAME_RULE}" + for name in params + if not is_name(name) + ] + + +def _header_problems(document: Mapping[Any, Any]) -> list[str]: + problems: list[str] = [] + if "name" not in document: + problems.append("name: required") + elif not _is_text(document["name"]): + problems.append(f"name: expected a non-empty string, got {document['name']!r}") + if not isinstance(document.get("description", ""), str): + problems.append(f"description: expected a string, got {_kind(document['description'])}") + workers = document.get("max_workers", 1) + if not _is_integer(workers) or workers < 1: + problems.append(f"max_workers: expected an integer >= 1, got {workers!r}") + if "schedule" in document: + problems.extend(_schedule_problems(document["schedule"])) + if "params" in document: + problems.extend(params_problems(document["params"])) + if isinstance(document["params"], Mapping): + problems.extend(_json_problems(document["params"], "params")) + return problems + + +def _task_problems(spec: Any, path: str) -> list[str]: + if not isinstance(spec, Mapping): + return [f"{path}: expected a mapping, got {_kind(spec)}"] + problems = _unknown_keys(spec, _TASK_KEYS, f"{path}.") + if "action" not in spec: + problems.append(f"{path}.action: required") + elif isinstance(spec["action"], list): + problems.extend(_json_problems(spec["action"], f"{path}.action")) + for key, check in _TASK_CHECKS.items(): + if key in spec: + problems.extend(check(spec[key], f"{path}.{key}")) + return problems + + +def _listed_dependencies(spec: Any) -> list[str]: + """Return a task's dependencies when they are all task IDs, else nothing to check.""" + wanted = spec.get("depends_on", []) if isinstance(spec, Mapping) else [] + if isinstance(wanted, list) and all(isinstance(item, str) for item in wanted): + return wanted + return [] + + +def _reference_problems(tasks: Mapping[Any, Any], dependencies: dict[str, list[str]]) -> list[str]: + problems: list[str] = [] + for task_id, upstream in upstream_tasks(dependencies).items(): + spec = tasks[task_id] + action = spec.get("action") if isinstance(spec, Mapping) else None + if isinstance(action, list): + problems.extend(action_reference_problems(action, f"tasks.{task_id}.action", upstream)) + return problems + + +def _tasks_problems(document: Mapping[Any, Any]) -> list[str]: + tasks = document.get("tasks", _MISSING) + if tasks is _MISSING: + return ["tasks: required"] + if not isinstance(tasks, Mapping): + return [f"tasks: expected a mapping of task ID to task, got {_kind(tasks)}"] + if not tasks: + return ["tasks: at least one task is required"] + problems: list[str] = [] + dependencies: dict[str, list[str]] = {} + for task_id, spec in tasks.items(): + if not is_name(task_id): + problems.append(f"tasks.{task_id}: invalid task ID, {NAME_RULE}") + continue + dependencies[task_id] = _listed_dependencies(spec) + problems.extend(_task_problems(spec, f"tasks.{task_id}")) + problems.extend(dependency_problems(dependencies)) + problems.extend(_reference_problems(tasks, dependencies)) + return problems + + +def validate_definition(document: Any) -> list[str]: + """Return every problem of a definition document; an empty list means it is valid. + + Each problem starts with the path of the entry it is about, for example + ``tasks.verify.depends_on[0]: unknown task 'x'``. A document without + ``schema_version``, or with one this version does not know, gets that single + finding: its other entries cannot be judged. + """ + if not isinstance(document, Mapping): + return [f"document: expected a mapping, got {_kind(document)}"] + version = _version_problem(document) + if version is not None: + return [version] + problems = _unknown_keys(document, _TOP_KEYS, "") + problems.extend(_header_problems(document)) + problems.extend(_tasks_problems(document)) + return problems + + +# ---------------------------------------------------------------------- to and from tasks + + +def retry_from_dict(spec: Mapping[str, Any]) -> RetryPolicy: + """Build a :class:`RetryPolicy` from the ``retry`` entry of a valid definition.""" + kinds = DEFAULT_RETRY_ON + if "on" in spec: + kinds = tuple(RETRYABLE_EXCEPTIONS[name] for name in spec["on"]) + return RetryPolicy( + max_attempts=spec.get("max_attempts", _DEFAULT_RETRY.max_attempts), + backoff_base=spec.get("backoff", _DEFAULT_RETRY.backoff_base), + backoff_cap=spec.get("backoff_cap", _DEFAULT_RETRY.backoff_cap), + retry_on=kinds, + ) + + +def _retry_to_dict(policy: RetryPolicy, path: str) -> dict[str, Any]: + spec: dict[str, Any] = {"max_attempts": policy.max_attempts} + if policy.backoff_base != _DEFAULT_RETRY.backoff_base: + spec["backoff"] = policy.backoff_base + if policy.backoff_cap != _DEFAULT_RETRY.backoff_cap: + spec["backoff_cap"] = policy.backoff_cap + if policy.retry_on != DEFAULT_RETRY_ON: + for kind in policy.retry_on: + if RETRYABLE_EXCEPTIONS.get(kind.__name__) is not kind: + raise PipelineDefinitionException( + f"{path}.on: {kind.__name__} cannot be named in a definition" + ) + spec["on"] = [kind.__name__ for kind in policy.retry_on] + return spec + + +def _copied(value: Any) -> Any: + """Copy nested mappings and lists; leave every other value as it is.""" + if isinstance(value, Mapping): + return {key: _copied(item) for key, item in value.items()} + if isinstance(value, list): + return [_copied(item) for item in value] + return value + + +def task_to_dict(task: Task) -> dict[str, Any]: + """Return the definition entry of ``task``. + + Raises :class:`PipelineDefinitionException` for what a document cannot hold: + a Python callable as the work or as ``when``, or a ``retry_on`` class outside + :data:`RETRYABLE_EXCEPTIONS`. + """ + path = f"tasks.{task.task_id}" + if callable(task.work): + raise PipelineDefinitionException(f"{path}: {_CALLABLE_NOT_WRITABLE}") + if callable(task.when): + raise PipelineDefinitionException(f"{path}.when: {_CALLABLE_NOT_WRITABLE}") + spec: dict[str, Any] = {"action": _copied(task.work)} + if task.depends_on: + spec["depends_on"] = list(task.depends_on) + if task.retry != _DEFAULT_RETRY: + spec["retry"] = _retry_to_dict(task.retry, f"{path}.retry") + if task.timeout is not None: + spec["timeout"] = task.timeout + if task.when != ON_SUCCESS: + spec["when"] = task.when + if task.idempotency_key is not None: + spec["idempotency_key"] = task.idempotency_key + return spec + + +# ---------------------------------------------------------------------- files + + +def _unique_keys(source: Path) -> Callable[[list[tuple[str, Any]]], dict[str, Any]]: + def build(pairs: list[tuple[str, Any]]) -> dict[str, Any]: + built: dict[str, Any] = {} + for key, value in pairs: + if key in built: + raise PipelineDefinitionException(f"{source}: duplicate key {key!r}") + built[key] = value + return built + + return build + + +def _parse_json(text: str, source: Path) -> Any: + try: + return json.loads(text, object_pairs_hook=_unique_keys(source)) + except json.JSONDecodeError as error: + raise PipelineDefinitionException( + f"{source}: invalid JSON: {error.msg} at line {error.lineno}, column {error.colno}" + ) from error + except RecursionError as error: + raise PipelineDefinitionException(f"{source}: {_TOO_DEEP}") from error + + +def _check_yaml_nodes(root: Any, source: Path) -> None: + """Refuse a repeated mapping key, and a document that aliases make huge or endless.""" + stack = [] if root is None else [root] + budget = _MAX_YAML_NODES + while stack: + node = stack.pop() + budget -= 1 + if budget < 0: + raise PipelineDefinitionException( + f"{source}: the definition is too large or refers to itself" + ) + if node.id == "sequence": + stack.extend(node.value) + elif node.id == "mapping": + _reject_repeated_keys(node, source) + stack.extend(value for _key, value in node.value) + + +def _reject_repeated_keys(node: Any, source: Path) -> None: + seen: set[str] = set() + for key, _value in node.value: + if key.id != "scalar": + continue + if key.value in seen: + raise PipelineDefinitionException( + f"{source}: duplicate key {key.value!r} at line {key.start_mark.line + 1}" + ) + seen.add(key.value) + + +def _yaml_reason(error: Exception) -> str: + """Say what the parser objected to and where, without quoting the file's content.""" + parts = [getattr(error, "context", None), getattr(error, "problem", None)] + reason = " ".join(str(part) for part in parts if part) or "syntax error" + mark = getattr(error, "problem_mark", None) + if mark is None: + return reason + return f"{reason} at line {mark.line + 1}, column {mark.column + 1}" + + +def _restore_on_keys(document: Any) -> Any: + """Give every ``retry`` entry its ``on`` key back: YAML 1.1 reads a bare ``on`` as ``true``.""" + tasks = document.get("tasks") if isinstance(document, dict) else None + if not isinstance(tasks, dict): + return document + for spec in tasks.values(): + retry = spec.get("retry") if isinstance(spec, dict) else None + if isinstance(retry, dict) and "on" not in retry and any(key is True for key in retry): + spec["retry"] = {("on" if key is True else key): item for key, item in retry.items()} + return document + + +def _parse_yaml(text: str, source: Path) -> Any: + import yaml + + try: + _check_yaml_nodes(yaml.compose(text, Loader=yaml.SafeLoader), source) + return _restore_on_keys(yaml.safe_load(text)) + except yaml.YAMLError as error: + raise PipelineDefinitionException( + f"{source}: invalid YAML: {_yaml_reason(error)}" + ) from error + except RecursionError as error: + raise PipelineDefinitionException(f"{source}: {_TOO_DEEP}") from error + + +def load_definition(path: str | os.PathLike[str]) -> Any: + """Read a definition document from a ``.yaml`` / ``.yml`` or ``.json`` file. + + The content is returned as parsed, without validation. YAML goes through + ``yaml.safe_load``; a key repeated in one mapping is an error in both formats, + so a second task with the same ID cannot silently replace the first. YAML 1.1 + reads a bare ``on`` as ``true``: under ``retry`` that key is given back as ``"on"``. + """ + source = Path(path) + suffix = source.suffix.lower() + if suffix not in (*_YAML_SUFFIXES, _JSON_SUFFIX): + raise PipelineDefinitionException(f"{source}: expected a .yaml, .yml or .json file") + try: + text = source.read_text(encoding="utf-8") + except (OSError, UnicodeDecodeError) as error: + raise PipelineDefinitionException( + f"{source}: cannot read the definition: {error}" + ) from error + return _parse_json(text, source) if suffix == _JSON_SUFFIX else _parse_yaml(text, source) diff --git a/automation_file/pipeline/errors.py b/automation_file/pipeline/errors.py new file mode 100644 index 0000000..4363991 --- /dev/null +++ b/automation_file/pipeline/errors.py @@ -0,0 +1,25 @@ +"""Exceptions raised by the pipeline runtime.""" + +from __future__ import annotations + +from collections.abc import Iterable + +from automation_file.exceptions import FileAutomationException + + +class PipelineException(FileAutomationException): + """Raised when a pipeline cannot be run, resumed, or written to its store.""" + + +class PipelineDefinitionException(PipelineException): + """Raised when a pipeline definition is invalid. + + ``problems`` holds every finding, each one starting with the path of the + offending entry (``tasks.verify.depends_on[0]: unknown task 'x'``). + """ + + def __init__(self, problems: Iterable[str] | str) -> None: + self.problems: tuple[str, ...] = ( + (problems,) if isinstance(problems, str) else tuple(problems) + ) + super().__init__("; ".join(self.problems)) diff --git a/automation_file/pipeline/graph.py b/automation_file/pipeline/graph.py new file mode 100644 index 0000000..2950d3b --- /dev/null +++ b/automation_file/pipeline/graph.py @@ -0,0 +1,122 @@ +"""Dependency graph checks and ordering for a pipeline. + +Every function takes the graph as ``{task ID: IDs it depends on}`` in definition +order, so the same checks serve a :class:`~automation_file.pipeline.pipeline.Pipeline` +built in Python and a definition document that has not become one yet. +""" + +from __future__ import annotations + +from collections.abc import Iterator, Mapping, Sequence + +from automation_file.pipeline.errors import PipelineDefinitionException + +Dependencies = Mapping[str, Sequence[str]] +_ON_PATH = 1 +_DONE = 2 + + +def dependency_problems(dependencies: Dependencies) -> list[str]: + """Return every unknown, repeated or self dependency and any cycle, each with its path.""" + problems: list[str] = [] + for task_id, wanted in dependencies.items(): + seen: set[str] = set() + for index, dependency in enumerate(wanted): + path = f"tasks.{task_id}.depends_on[{index}]" + if dependency == task_id: + problems.append(f"{path}: a task cannot depend on itself") + elif dependency not in dependencies: + problems.append(f"{path}: unknown task {dependency!r}") + elif dependency in seen: + problems.append(f"{path}: {dependency!r} is listed twice") + seen.add(dependency) + cycle = find_cycle(dependencies) + if cycle is not None: + problems.append(f"tasks: dependency cycle: {' -> '.join(cycle)}") + return problems + + +def _edges(dependencies: Dependencies, task_id: str) -> Iterator[str]: + """Yield the dependencies of ``task_id`` that exist and are not the task itself.""" + return ( + dependency + for dependency in dependencies[task_id] + if dependency in dependencies and dependency != task_id + ) + + +def find_cycle(dependencies: Dependencies) -> list[str] | None: + """Return one dependency cycle as ``[a, b, ..., a]``, or ``None`` when there is none.""" + marks: dict[str, int] = {} + for root in dependencies: + if root in marks: + continue + cycle = _cycle_from(root, dependencies, marks) + if cycle is not None: + return cycle + return None + + +def _cycle_from(root: str, dependencies: Dependencies, marks: dict[str, int]) -> list[str] | None: + path = [root] + pending = [_edges(dependencies, root)] + marks[root] = _ON_PATH + while pending: + following = next(pending[-1], None) + if following is None: + marks[path.pop()] = _DONE + pending.pop() + elif marks.get(following) == _ON_PATH: + return [*path[path.index(following) :], following] + elif following not in marks: + marks[following] = _ON_PATH + path.append(following) + pending.append(_edges(dependencies, following)) + return None + + +def task_levels(dependencies: Dependencies) -> dict[str, int]: + """Return ``{task ID: level}`` in dependency order. + + A task without dependencies is at level 0; any other is one level above its + deepest dependency. Tasks of one level keep their definition order. The graph + must be free of the problems :func:`dependency_problems` reports. + """ + levels: dict[str, int] = {} + remaining = list(dependencies) + while remaining: + waiting: list[str] = [] + for task_id in remaining: + wanted = dependencies[task_id] + if all(dependency in levels for dependency in wanted): + levels[task_id] = 1 + max((levels[dependency] for dependency in wanted), default=-1) + else: + waiting.append(task_id) + if len(waiting) == len(remaining): + raise PipelineDefinitionException( + f"tasks: cannot order {', '.join(waiting)}: unknown dependency or cycle" + ) + remaining = waiting + position = {task_id: index for index, task_id in enumerate(dependencies)} + ordered = sorted(levels, key=lambda task_id: (levels[task_id], position[task_id])) + return {task_id: levels[task_id] for task_id in ordered} + + +def upstream_tasks(dependencies: Dependencies) -> dict[str, tuple[str, ...]]: + """Return ``{task ID: every task it depends on, directly or through others}``. + + Each tuple is in definition order. Unknown dependencies are left out and a + cycle ends the walk, so this is safe to call on a graph that still has problems. + """ + upstream: dict[str, tuple[str, ...]] = {} + for task_id in dependencies: + found: set[str] = set() + stack = list(_edges(dependencies, task_id)) + while stack: + current = stack.pop() + if current in found or current == task_id: + continue + found.add(current) + stack.extend(_edges(dependencies, current)) + upstream[task_id] = tuple(other for other in dependencies if other in found) + return upstream diff --git a/automation_file/pipeline/model.py b/automation_file/pipeline/model.py new file mode 100644 index 0000000..b89aa77 --- /dev/null +++ b/automation_file/pipeline/model.py @@ -0,0 +1,353 @@ +"""The pipeline's data model: tasks, retry policies, and the recorded state of a run. + +A :class:`Task` is what a pipeline was told to do; a :class:`TaskRun` is what +happened to it in one :class:`PipelineRun`. Both run records turn into +JSON-friendly mappings (``to_dict``) and back (``from_dict``), which is the form +a :class:`~automation_file.pipeline.store.RunStore` keeps. +""" + +from __future__ import annotations + +import json +import threading +from collections.abc import Callable, Mapping +from dataclasses import dataclass, field +from datetime import datetime, timezone +from enum import Enum +from typing import Any + +from automation_file.core.progress import CancellationToken +from automation_file.exceptions import StorageTransientException +from automation_file.pipeline.errors import PipelineDefinitionException + +#: ``when`` values: run when every dependency succeeded / when one failed / in any case. +ON_SUCCESS = "on_success" +ON_FAILURE = "on_failure" +ALWAYS = "always" +WHEN_CHOICES: tuple[str, ...] = (ON_SUCCESS, ON_FAILURE, ALWAYS) + +#: ``TaskRun.reason`` values of a skipped task. +REASON_IDEMPOTENT = "idempotent" +REASON_CONDITION = "condition" +REASON_UPSTREAM_FAILED = "upstream_failed" +REASON_UPSTREAM_SKIPPED = "upstream_skipped" + +#: Failures worth a second attempt; anything else is a bug or a wrong input. +DEFAULT_RETRY_ON: tuple[type[BaseException], ...] = ( + StorageTransientException, + ConnectionError, + TimeoutError, +) +_MAX_BACKOFF_EXPONENT = 62 + + +class TaskStatus(str, Enum): + """Where a task stands in one run.""" + + PENDING = "pending" + RUNNING = "running" + SUCCEEDED = "succeeded" + FAILED = "failed" + SKIPPED = "skipped" + TIMEOUT = "timeout" + CANCELLED = "cancelled" + PLANNED = "planned" + + @property + def is_final(self) -> bool: + """Whether the task will not change any more in this run.""" + return self not in (TaskStatus.PENDING, TaskStatus.RUNNING) + + @property + def is_failure(self) -> bool: + """Whether the task failed, timed out or was cancelled.""" + return self in (TaskStatus.FAILED, TaskStatus.TIMEOUT, TaskStatus.CANCELLED) + + +class RunStatus(str, Enum): + """Where a whole run stands.""" + + RUNNING = "running" + SUCCEEDED = "succeeded" + FAILED = "failed" + CANCELLED = "cancelled" + + +def utc_now() -> datetime: + """Return the current time as a timezone-aware UTC ``datetime``.""" + return datetime.now(timezone.utc) + + +def _iso(moment: datetime | None) -> str | None: + return None if moment is None else moment.isoformat(timespec="microseconds") + + +def _moment(text: str | None) -> datetime | None: + return None if text is None else datetime.fromisoformat(text) + + +def json_safe(value: Any) -> tuple[Any, bool]: + """Return ``value`` as plain JSON data, or ``(repr(value), True)`` when JSON cannot hold it.""" + try: + return json.loads(json.dumps(value)), False + except (TypeError, ValueError, RecursionError): + return repr(value), True + + +def describe_error(error: BaseException) -> str: + """Return ``": "``, the form the events and the store keep.""" + message = str(error) + return f"{type(error).__name__}: {message}" if message else type(error).__name__ + + +@dataclass(frozen=True) +class RetryPolicy: + """How often a task is tried again, how long to wait, and after which errors. + + The wait after attempt ``n`` is ``backoff_base * 2 ** (n - 1)`` seconds, at most + ``backoff_cap``. Only an error that is an instance of one of ``retry_on`` is + retried; the default is the transient kind, so a bug fails on its first attempt. + """ + + max_attempts: int = 1 + backoff_base: float = 0.0 + backoff_cap: float = 60.0 + retry_on: tuple[type[BaseException], ...] = DEFAULT_RETRY_ON + + def __post_init__(self) -> None: + if isinstance(self.max_attempts, bool) or not isinstance(self.max_attempts, int): + raise PipelineDefinitionException("retry.max_attempts: expected an integer >= 1") + if self.max_attempts < 1: + raise PipelineDefinitionException("retry.max_attempts: expected an integer >= 1") + if self.backoff_base < 0 or self.backoff_cap < 0: + raise PipelineDefinitionException("retry: a back-off cannot be negative") + kinds = tuple(self.retry_on) + for kind in kinds: + if not (isinstance(kind, type) and issubclass(kind, BaseException)): + raise PipelineDefinitionException(f"retry.retry_on: {kind!r} is not an exception") + object.__setattr__(self, "retry_on", kinds) + + def delay(self, attempt: int) -> float: + """Return the seconds to wait after the failed attempt number ``attempt`` (1-based).""" + exponent = min(max(attempt - 1, 0), _MAX_BACKOFF_EXPONENT) + return float(min(self.backoff_cap, self.backoff_base * 2**exponent)) + + def retries(self, error: BaseException) -> bool: + """Return whether ``error`` is one this policy tries again after.""" + return isinstance(error, self.retry_on) + + +@dataclass(frozen=True) +class Schedule: + """When a pipeline should run by itself; kept for the scheduler, not acted on here.""" + + cron: str + timezone: str | None = None + + def to_dict(self) -> dict[str, str]: + """Return the ``schedule`` entry of a definition.""" + document = {"cron": self.cron} + if self.timezone is not None: + document["timezone"] = self.timezone + return document + + +@dataclass(frozen=True) +class TaskContext: + """What a task callable (or a ``when`` callable) is given. + + ``results`` maps the ID of every upstream task that succeeded to its result. + ``cancel`` is set when the run is cancelled or the task's timeout has passed; + a long task must look at it, because a running thread cannot be stopped from + outside. ``attempt`` is 1-based (``0`` in a ``when`` callable). + """ + + pipeline: str + run_id: str + task: str + attempt: int + params: Mapping[str, Any] + results: Mapping[str, Any] + cancel: CancellationToken + dry_run: bool = False + + +TaskCallable = Callable[[TaskContext], Any] +Condition = Callable[[TaskContext], bool] +TaskWork = TaskCallable | list[Any] + + +@dataclass(frozen=True) +class Task: + """One unit of work in a pipeline and the rules for running it. + + ``work`` is a callable taking a :class:`TaskContext`, or an action + (``[name]``, ``[name, {kwargs}]``, ``[name, [args]]``). + """ + + task_id: str + work: TaskWork + depends_on: tuple[str, ...] = () + retry: RetryPolicy = field(default_factory=RetryPolicy) + timeout: float | None = None + when: str | Condition = ON_SUCCESS + idempotency_key: str | None = None + + @property + def action_name(self) -> str | None: + """Return the ``FA_*`` name of an action task, ``None`` for a callable.""" + return None if callable(self.work) else str(self.work[0]) + + +@dataclass +class TaskRun: + """What happened to one task in one run. + + ``result`` is the task's return value while the run is in memory. A store + keeps it as JSON; a value JSON cannot hold is kept as its ``repr`` and + ``result_is_repr`` is set, so a resumed run sees that text, not the object. + ``reason`` says why a ``skipped`` task did not run. + """ + + task: str + status: TaskStatus = TaskStatus.PENDING + level: int = 0 + attempts: int = 0 + result: Any = None + error: str | None = None + reason: str | None = None + idempotency_key: str | None = None + started_at: datetime | None = None + finished_at: datetime | None = None + result_is_repr: bool = False + + @property + def satisfied(self) -> bool: + """Whether dependents can count on this task: it succeeded now or in an earlier run.""" + if self.status is TaskStatus.SUCCEEDED: + return True + return self.status is TaskStatus.SKIPPED and self.reason == REASON_IDEMPOTENT + + @property + def duration_ms(self) -> float | None: + """Return the milliseconds between start and finish, ``None`` while either is missing.""" + if self.started_at is None or self.finished_at is None: + return None + return round((self.finished_at - self.started_at).total_seconds() * 1000, 3) + + def to_dict(self) -> dict[str, Any]: + """Return a JSON-serialisable mapping of the task's state.""" + result, converted = json_safe(self.result) + return { + "task": self.task, + "status": self.status.value, + "level": self.level, + "attempts": self.attempts, + "result": result, + "result_is_repr": self.result_is_repr or converted, + "error": self.error, + "reason": self.reason, + "idempotency_key": self.idempotency_key, + "started_at": _iso(self.started_at), + "finished_at": _iso(self.finished_at), + "duration_ms": self.duration_ms, + } + + @classmethod + def from_dict(cls, record: Mapping[str, Any]) -> TaskRun: + """Rebuild a task state from :meth:`to_dict`.""" + return cls( + task=record["task"], + status=TaskStatus(record["status"]), + level=int(record.get("level", 0)), + attempts=int(record.get("attempts", 0)), + result=record.get("result"), + error=record.get("error"), + reason=record.get("reason"), + idempotency_key=record.get("idempotency_key"), + started_at=_moment(record.get("started_at")), + finished_at=_moment(record.get("finished_at")), + result_is_repr=bool(record.get("result_is_repr", False)), + ) + + +@dataclass +class PipelineRun: + """One execution of a pipeline: its status, its parameters and every task's state. + + ``tasks`` is in dependency order. :meth:`cancel` asks a run to stop and + :meth:`wait` blocks until it has ended; both matter for a run started in the + background with :meth:`~automation_file.pipeline.pipeline.Pipeline.start`. + """ + + run_id: str + pipeline: str + status: RunStatus = RunStatus.RUNNING + params: dict[str, Any] = field(default_factory=dict) + dry_run: bool = False + tasks: dict[str, TaskRun] = field(default_factory=dict) + started_at: datetime | None = None + finished_at: datetime | None = None + error: str | None = None + cancel_token: CancellationToken = field( + default_factory=CancellationToken, repr=False, compare=False + ) + _ended: threading.Event = field( + default_factory=threading.Event, init=False, repr=False, compare=False + ) + + @property + def done(self) -> bool: + """Whether the run has ended.""" + return self._ended.is_set() and self.status is not RunStatus.RUNNING + + def cancel(self) -> None: + """Ask the run to stop: unstarted tasks become ``cancelled``, running ones are told.""" + self.cancel_token.cancel() + + def wait(self, timeout: float | None = None) -> bool: + """Block until the run has ended or ``timeout`` seconds passed; return whether it ended. + + A run read back from a store is a snapshot that nothing will change, so + this returns at once, ``False`` if the snapshot is of a run still going. + """ + self._ended.wait(timeout) + return self.done + + def mark_done(self) -> None: + """Release everything blocked in :meth:`wait`. The runtime calls this when the run ends.""" + self._ended.set() + + def to_dict(self) -> dict[str, Any]: + """Return a JSON-serialisable mapping of the run and its tasks.""" + return { + "run_id": self.run_id, + "pipeline": self.pipeline, + "status": self.status.value, + "dry_run": self.dry_run, + "params": {str(name): json_safe(value)[0] for name, value in self.params.items()}, + "started_at": _iso(self.started_at), + "finished_at": _iso(self.finished_at), + "error": self.error, + "tasks": {task_id: state.to_dict() for task_id, state in self.tasks.items()}, + } + + @classmethod + def from_dict(cls, record: Mapping[str, Any]) -> PipelineRun: + """Rebuild a run from :meth:`to_dict`: a snapshot, with nothing left to wait for.""" + run = cls( + run_id=record["run_id"], + pipeline=record["pipeline"], + status=RunStatus(record["status"]), + params=dict(record.get("params") or {}), + dry_run=bool(record.get("dry_run", False)), + tasks={ + task_id: TaskRun.from_dict(state) + for task_id, state in (record.get("tasks") or {}).items() + }, + started_at=_moment(record.get("started_at")), + finished_at=_moment(record.get("finished_at")), + error=record.get("error"), + ) + run.mark_done() + return run diff --git a/automation_file/pipeline/pipeline.py b/automation_file/pipeline/pipeline.py new file mode 100644 index 0000000..f148892 --- /dev/null +++ b/automation_file/pipeline/pipeline.py @@ -0,0 +1,479 @@ +"""The pipeline: a named set of tasks with dependencies, and the ways to run it. + +.. code-block:: python + + from automation_file.pipeline import Pipeline, RetryPolicy + + pipeline = Pipeline("daily-report") + pipeline.task("download", ["FA_storage_copy", {"source": "s3://in/report.csv", + "target": "local:///tmp/report.csv"}]) + pipeline.task("validate", validate, depends_on=["download"], + retry=RetryPolicy(max_attempts=3, backoff_base=1.0), timeout=60.0) + run = pipeline.run(params={"date": "2026-10-08"}) + run.status, run.tasks["validate"].result +""" + +from __future__ import annotations + +import os +import threading +from collections.abc import Collection, Iterable, Mapping +from typing import TYPE_CHECKING, Any + +from automation_file.core.progress import CancellationToken +from automation_file.events import EventBus, current_actor, event_bus, new_correlation_id +from automation_file.logging_config import file_automation_logger +from automation_file.pipeline.definition import ( + SCHEMA_VERSION, + action_problems, + depends_on_problems, + idempotency_key_problems, + load_definition, + params_problems, + retry_from_dict, + task_to_dict, + timeout_problems, + validate_definition, + when_problems, +) +from automation_file.pipeline.errors import PipelineDefinitionException, PipelineException +from automation_file.pipeline.graph import dependency_problems, task_levels, upstream_tasks +from automation_file.pipeline.model import ( + ON_SUCCESS, + Condition, + PipelineRun, + RetryPolicy, + RunStatus, + Schedule, + Task, + TaskRun, + TaskStatus, + TaskWork, + utc_now, +) +from automation_file.pipeline.runner import Engine, Plan, Services +from automation_file.pipeline.store import RunStore, default_run_store +from automation_file.pipeline.substitution import ( + NAME_RULE, + action_reference_problems, + is_name, + text_problems, +) + +if TYPE_CHECKING: + from automation_file.core.action_registry import ActionRegistry + +DEFAULT_MAX_WORKERS = 4 + + +def _shared_registry() -> ActionRegistry: + """Return the registry of the shared executor (imported late: it is built at import).""" + from automation_file.core.action_executor import executor + + return executor.registry + + +def _task_problems(task: Task, taken: Collection[str]) -> list[str]: + path = f"tasks.{task.task_id}" + if not is_name(task.task_id): + return [f"{path}: invalid task ID, {NAME_RULE}"] + problems: list[str] = [] + if task.task_id in taken: + problems.append(f"{path}: duplicate task ID") + if not callable(task.work): + problems.extend(action_problems(task.work, f"{path}.action")) + problems.extend(depends_on_problems(task.depends_on, f"{path}.depends_on")) + if task.timeout is not None: + problems.extend(timeout_problems(task.timeout, f"{path}.timeout")) + if not callable(task.when): + problems.extend(when_problems(task.when, f"{path}.when")) + if task.idempotency_key is not None: + problems.extend(idempotency_key_problems(task.idempotency_key, f"{path}.idempotency_key")) + return problems + + +def _carried_over(earlier: TaskRun | None, fresh: TaskRun) -> TaskRun: + """Return the stored state of a task that succeeded, else the fresh one to run again.""" + if earlier is None or earlier.status is not TaskStatus.SUCCEEDED: + return fresh + earlier.level = fresh.level + return earlier + + +def _run_in_background(engine: Engine) -> None: + try: + engine.execute() + except Exception as error: # pylint: disable=broad-except + # Boundary of the background thread: the engine has already marked the run failed. + file_automation_logger.error( + "pipeline %s run %s ended with %r", engine.run.pipeline, engine.run.run_id, error + ) + + +class Pipeline: + """A named set of tasks, run in dependency order with independent tasks in parallel. + + ``max_workers`` bounds how many tasks run at once. ``params`` are defaults + that ``run(params=...)`` adds to or overrides. ``schedule`` is kept for the + scheduler and not acted on here. ``registry`` is where action tasks are looked + up; it defaults to the shared executor's registry. + """ + + def __init__( + self, + name: str, + description: str = "", + max_workers: int = DEFAULT_MAX_WORKERS, + *, + params: Mapping[str, Any] | None = None, + schedule: Schedule | None = None, + registry: ActionRegistry | None = None, + ) -> None: + problems: list[str] = [] + if not (isinstance(name, str) and name.strip()): + problems.append(f"name: expected a non-empty string, got {name!r}") + if isinstance(max_workers, bool) or not isinstance(max_workers, int) or max_workers < 1: + problems.append(f"max_workers: expected an integer >= 1, got {max_workers!r}") + if params is not None: + problems.extend(params_problems(params)) + if problems: + raise PipelineDefinitionException(problems) + self.name = name + self.description = description + self.max_workers = max_workers + self.params: dict[str, Any] = dict(params or {}) + self.schedule = schedule + self._registry = registry + self._tasks: dict[str, Task] = {} + + def __repr__(self) -> str: + return f"Pipeline({self.name!r}, tasks={list(self._tasks)})" + + @property + def tasks(self) -> tuple[Task, ...]: + """The tasks in the order they were added.""" + return tuple(self._tasks.values()) + + # ------------------------------------------------------------------ building + + def task( + self, + task_id: str, + work: TaskWork, + *, + depends_on: Iterable[str] | str | None = None, + retry: RetryPolicy | None = None, + timeout: float | None = None, + when: str | Condition = ON_SUCCESS, + idempotency_key: str | None = None, + ) -> Task: + """Add a task and return it. + + ``work`` is a callable taking a :class:`TaskContext`, or an action + (``[name]``, ``[name, {kwargs}]``, ``[name, [args]]``) whose string + arguments may hold ``${params.}`` and ``${tasks..result}``. + + ``timeout`` is the budget in seconds for the whole task, every attempt and + the waits between them included. A thread cannot be killed: when the + budget is spent the task is recorded as ``timeout``, its cancellation + token is set and the run goes on, but the callable keeps running until it + returns. A long callable must therefore watch ``context.cancel``. + + ``when`` is ``"on_success"`` (every dependency succeeded), ``"on_failure"`` + (at least one failed, timed out or was cancelled), ``"always"``, or a + callable ``(TaskContext) -> bool``. ``idempotency_key`` (with + ``${params.}`` placeholders) skips the task when the store already + holds a succeeded execution under the same key, and reuses its result. + + A dependency may name a task that is added later; the graph is checked + when the pipeline runs. Anything else that is wrong raises + :class:`PipelineDefinitionException` here. + """ + wanted = (depends_on,) if isinstance(depends_on, str) else tuple(depends_on or ()) + added = Task( + task_id=task_id, + work=work, + depends_on=wanted, + retry=RetryPolicy() if retry is None else retry, + timeout=timeout, + when=when, + idempotency_key=idempotency_key, + ) + problems = _task_problems(added, self._tasks) + if problems: + raise PipelineDefinitionException(problems) + self._tasks[task_id] = added + return added + + def _dependencies(self) -> dict[str, tuple[str, ...]]: + return {task.task_id: task.depends_on for task in self._tasks.values()} + + def problems(self) -> list[str]: + """Return what keeps the pipeline from running, each finding with its path. + + Checked here: an empty pipeline, unknown, repeated and self dependencies, + cycles, and placeholders that are malformed or name a task that is not + upstream. + """ + if not self._tasks: + return ["tasks: at least one task is required"] + dependencies = self._dependencies() + found = dependency_problems(dependencies) + upstream = upstream_tasks(dependencies) + for task in self._tasks.values(): + if not callable(task.work): + found.extend( + action_reference_problems( + task.work, f"tasks.{task.task_id}.action", upstream[task.task_id] + ) + ) + return found + + def validate(self) -> None: + """Raise :class:`PipelineDefinitionException` when :meth:`problems` finds any.""" + problems = self.problems() + if problems: + raise PipelineDefinitionException(problems) + + # ------------------------------------------------------------------ running + + def run( + self, + params: Mapping[str, Any] | None = None, + *, + dry_run: bool = False, + store: RunStore | None = None, + cancel: CancellationToken | None = None, + bus: EventBus | None = None, + ) -> PipelineRun: + """Run the pipeline in the calling thread and return the finished run. + + A task that fails does not raise here: look at ``run.status`` and + ``run.tasks``. What does raise, before anything runs, is a definition + problem (:class:`PipelineDefinitionException`): a cycle, an unknown + dependency, a placeholder for a parameter the run was not given. + + ``dry_run=True`` executes nothing, records nothing and publishes nothing: + every task comes back ``planned``, in dependency order with its level, + and an unknown action name or a missing parameter is reported in the + task's ``error``. ``store`` records the run (the default store when + omitted), ``cancel`` stops it from outside, and ``bus`` receives its + events instead of the process-wide bus. + """ + merged = {**self.params, **(params or {})} + if dry_run: + return self._dry_run(merged) + engine = self._engine(merged, store, cancel, bus) + engine.execute() + return engine.run + + def start( + self, + params: Mapping[str, Any] | None = None, + *, + store: RunStore | None = None, + cancel: CancellationToken | None = None, + bus: EventBus | None = None, + ) -> PipelineRun: + """Run the pipeline on a background thread and return its run at once. + + ``run.wait(timeout)`` blocks until it has ended and ``run.cancel()`` + stops it. A definition problem raises here, before the thread starts. + """ + engine = self._engine({**self.params, **(params or {})}, store, cancel, bus) + threading.Thread( + target=_run_in_background, args=(engine,), name=f"pipeline-{self.name}" + ).start() + return engine.run + + def resume( + self, + run_id: str, + *, + store: RunStore | None = None, + cancel: CancellationToken | None = None, + bus: EventBus | None = None, + ) -> PipelineRun: + """Continue a stored run: keep the tasks that succeeded, run the rest. + + The run keeps its ID and its parameters. Results come from the store, so + they are JSON values (or a ``repr`` when ``result_is_repr`` is set). A run + that already succeeded is returned as stored. Do not resume a run that is + still executing somewhere else. + """ + chosen = default_run_store() if store is None else store + earlier = chosen.get_run(run_id) + if earlier is None: + raise PipelineException(f"unknown run {run_id!r}") + if earlier.pipeline != self.name: + raise PipelineException( + f"run {run_id!r} belongs to pipeline {earlier.pipeline!r}, not {self.name!r}" + ) + if earlier.status is RunStatus.SUCCEEDED: + return earlier + plan = self._plan(earlier.params) + run = self._blank_run(plan, earlier.params, cancel) + run.run_id = run_id + run.started_at = earlier.started_at or run.started_at + run.tasks = { + task_id: _carried_over(earlier.tasks.get(task_id), state) + for task_id, state in run.tasks.items() + } + Engine(plan, run, self._services(chosen, bus)).execute() + return run + + def _plan(self, params: Mapping[str, Any], *, lenient: bool = False) -> Plan: + """Order the tasks, or raise with every problem. ``lenient`` leaves missing parameters.""" + problems = self.problems() + if not problems and not lenient: + problems = [note for notes in self._input_problems(params).values() for note in notes] + if problems: + raise PipelineDefinitionException(problems) + dependencies = self._dependencies() + levels = task_levels(dependencies) + return Plan( + pipeline=self.name, + tasks=tuple(self._tasks[task_id] for task_id in levels), + levels=levels, + upstream=upstream_tasks(dependencies), + max_workers=self.max_workers, + ) + + def _input_problems(self, params: Mapping[str, Any]) -> dict[str, list[str]]: + """Return, per task, the parameters its action or key uses and the run was not given.""" + upstream = upstream_tasks(self._dependencies()) + found: dict[str, list[str]] = {} + for task in self._tasks.values(): + path = f"tasks.{task.task_id}" + notes: list[str] = [] + if not callable(task.work): + notes.extend( + action_reference_problems( + task.work, f"{path}.action", upstream[task.task_id], params + ) + ) + if task.idempotency_key is not None: + notes.extend( + f"{path}.idempotency_key: {problem}" + for problem in text_problems(task.idempotency_key, None, params) + ) + if notes: + found[task.task_id] = notes + return found + + def _blank_run( + self, plan: Plan, params: Mapping[str, Any], cancel: CancellationToken | None + ) -> PipelineRun: + return PipelineRun( + run_id=new_correlation_id(), + pipeline=self.name, + params=dict(params), + tasks={ + task.task_id: TaskRun(task=task.task_id, level=plan.levels[task.task_id]) + for task in plan.tasks + }, + started_at=utc_now(), + cancel_token=CancellationToken() if cancel is None else cancel, + ) + + def _action_registry(self) -> ActionRegistry: + return _shared_registry() if self._registry is None else self._registry + + def _services(self, store: RunStore | None, bus: EventBus | None) -> Services: + return Services( + store=default_run_store() if store is None else store, + bus=event_bus if bus is None else bus, + registry=self._action_registry(), + actor=current_actor(), + ) + + def _engine( + self, + params: Mapping[str, Any], + store: RunStore | None, + cancel: CancellationToken | None, + bus: EventBus | None, + ) -> Engine: + plan = self._plan(params) + return Engine(plan, self._blank_run(plan, params, cancel), self._services(store, bus)) + + def _dry_run(self, params: Mapping[str, Any]) -> PipelineRun: + plan = self._plan(params, lenient=True) + run = self._blank_run(plan, params, None) + run.dry_run = True + registry = self._action_registry() + notes = self._input_problems(params) + for task in plan.tasks: + state = run.tasks[task.task_id] + state.status = TaskStatus.PLANNED + found = list(notes.get(task.task_id, ())) + name = task.action_name + if name is not None and registry.resolve(name) is None: + found.append(f"tasks.{task.task_id}.action[0]: unknown action {name!r}") + state.error = "; ".join(found) or None + flagged = [state.task for state in run.tasks.values() if state.error is not None] + run.status = RunStatus.FAILED if flagged else RunStatus.SUCCEEDED + run.error = f"would not run as planned: {', '.join(flagged)}" if flagged else None + run.finished_at = utc_now() + run.mark_done() + return run + + # ------------------------------------------------------------------ definitions + + @classmethod + def from_dict( + cls, document: Mapping[str, Any], *, registry: ActionRegistry | None = None + ) -> Pipeline: + """Build a pipeline from a definition document (``schema_version: 1``). + + Raises :class:`PipelineDefinitionException` carrying every problem + :func:`validate_definition` finds. + """ + problems = validate_definition(document) + if problems: + raise PipelineDefinitionException(problems) + schedule = document.get("schedule") + pipeline = cls( + document["name"], + description=document.get("description", ""), + max_workers=document.get("max_workers", DEFAULT_MAX_WORKERS), + params=document.get("params"), + schedule=None if schedule is None else Schedule(**schedule), + registry=registry, + ) + for task_id, spec in document["tasks"].items(): + pipeline.task( + task_id, + spec["action"], + depends_on=spec.get("depends_on"), + retry=retry_from_dict(spec["retry"]) if "retry" in spec else None, + timeout=spec.get("timeout"), + when=spec.get("when", ON_SUCCESS), + idempotency_key=spec.get("idempotency_key"), + ) + return pipeline + + @classmethod + def from_file( + cls, path: str | os.PathLike[str], *, registry: ActionRegistry | None = None + ) -> Pipeline: + """Build a pipeline from a ``.yaml`` / ``.yml`` or ``.json`` definition file.""" + return cls.from_dict(load_definition(path), registry=registry) + + def to_dict(self) -> dict[str, Any]: + """Return the definition document of this pipeline. + + A default (``when: on_success``, no retry ...) is left out. A pipeline + with a Python callable in it has no document and raises + :class:`PipelineDefinitionException`. + """ + document: dict[str, Any] = {"schema_version": SCHEMA_VERSION, "name": self.name} + if self.description: + document["description"] = self.description + document["max_workers"] = self.max_workers + if self.schedule is not None: + document["schedule"] = self.schedule.to_dict() + if self.params: + document["params"] = dict(self.params) + document["tasks"] = {task.task_id: task_to_dict(task) for task in self._tasks.values()} + return document diff --git a/automation_file/pipeline/reporting.py b/automation_file/pipeline/reporting.py new file mode 100644 index 0000000..9b0ee79 --- /dev/null +++ b/automation_file/pipeline/reporting.py @@ -0,0 +1,100 @@ +"""How a run tells the world what happened: events on the bus, checkpoints in the store. + +The pipeline calls no notification sink and writes no audit row. It publishes +``pipeline.*`` and ``task.*`` events with ``source="pipeline"``; whoever needs +them subscribes to the bus. A failing store is logged and the run goes on: the +work matters more than its record. +""" + +from __future__ import annotations + +from dataclasses import dataclass +from typing import Any + +from automation_file.events import Event, EventBus, Severity +from automation_file.logging_config import file_automation_logger +from automation_file.pipeline.errors import PipelineException +from automation_file.pipeline.model import PipelineRun, TaskRun +from automation_file.pipeline.store import RunStore + +SOURCE = "pipeline" + + +@dataclass(frozen=True) +class Outcome: + """The optional details of an event: how long it took, what went wrong, how bad it is.""" + + duration_ms: float | None = None + error: str | None = None + severity: Severity | None = None + + +_PLAIN = Outcome() + + +class Reporter: + """Publishes the events of one run and writes its checkpoints.""" + + def __init__(self, run: PipelineRun, store: RunStore, bus: EventBus) -> None: + self._run = run + self._store = store + self._bus = bus + + def save_run(self) -> None: + """Checkpoint the run and all of its tasks.""" + try: + self._store.save_run(self._run) + except PipelineException as error: + self._lost("the run", error) + + def save_task(self, state: TaskRun) -> None: + """Checkpoint one task transition.""" + try: + self._store.save_task(self._run.run_id, state) + except PipelineException as error: + self._lost(f"task {state.task!r}", error) + + def _lost(self, what: str, error: PipelineException) -> None: + file_automation_logger.error( + "pipeline %s run %s: cannot record %s: %r", + self._run.pipeline, + self._run.run_id, + what, + error, + ) + + def pipeline_event(self, kind: type[Event], status: str, outcome: Outcome = _PLAIN) -> None: + """Publish a ``pipeline.*`` event for the run.""" + payload: dict[str, Any] = { + "pipeline": self._run.pipeline, + "run_id": self._run.run_id, + "status": status, + } + self._publish(kind, f"{self._run.pipeline} {status}", payload, outcome) + + def task_event( + self, kind: type[Event], state: TaskRun, status: str, outcome: Outcome = _PLAIN + ) -> None: + """Publish a ``task.*`` event for the current attempt of ``state``.""" + payload: dict[str, Any] = { + "pipeline": self._run.pipeline, + "run_id": self._run.run_id, + "task": state.task, + "attempt": state.attempts, + "status": status, + } + subject = f"{self._run.pipeline}/{state.task} {status} (attempt {state.attempts})" + self._publish(kind, subject, payload, outcome) + + def _publish( + self, kind: type[Event], subject: str, payload: dict[str, Any], outcome: Outcome + ) -> None: + if outcome.duration_ms is not None: + payload["duration_ms"] = outcome.duration_ms + if outcome.error is not None: + payload["error"] = outcome.error + if outcome.severity is None: + event = kind(source=SOURCE, subject=subject, payload=payload) + else: + event = kind(source=SOURCE, subject=subject, payload=payload, severity=outcome.severity) + self._bus.publish(event) diff --git a/automation_file/pipeline/runner.py b/automation_file/pipeline/runner.py new file mode 100644 index 0000000..d67bdee --- /dev/null +++ b/automation_file/pipeline/runner.py @@ -0,0 +1,356 @@ +"""The run loop: order the tasks, fan them out on threads, watch deadlines and cancellation. + +One thread coordinates a run (the caller's for ``Pipeline.run``, a background +one for ``Pipeline.start``). When a task's dependencies have all ended it decides +whether the task runs: its ``when`` condition, then its idempotency key. A task +that runs gets a thread of its own, with at most ``max_workers`` tasks in the air. + +Python cannot stop a thread. When a timeout passes, the coordinator records +``timeout``, sets the task's cancellation token and moves on; the thread is left +to finish by itself and what it does afterwards is ignored. Task threads are +daemon threads, so one that never returns does not keep the interpreter alive. +A cancelled run is different: its running tasks are told through their tokens +and the run waits for them, because their outcome still counts. +""" + +from __future__ import annotations + +import queue +import threading +import time +from collections import deque +from collections.abc import Mapping +from dataclasses import dataclass +from types import MappingProxyType +from typing import Any + +from automation_file.events import ( + EventBus, + PipelineCompleted, + PipelineFailed, + PipelineStarted, + Severity, + TaskFailed, + actor_scope, + correlation_scope, +) +from automation_file.logging_config import file_automation_logger +from automation_file.pipeline.errors import PipelineException +from automation_file.pipeline.model import ( + ALWAYS, + ON_FAILURE, + REASON_CONDITION, + REASON_IDEMPOTENT, + REASON_UPSTREAM_FAILED, + REASON_UPSTREAM_SKIPPED, + Condition, + PipelineRun, + RunStatus, + Task, + TaskRun, + TaskStatus, + describe_error, + utc_now, +) +from automation_file.pipeline.reporting import Outcome, Reporter +from automation_file.pipeline.store import RunStore +from automation_file.pipeline.substitution import render_key +from automation_file.pipeline.worker import Flight, TaskToken, Worker, elapsed_ms, make_context + +_POLL_SECONDS = 0.05 +_NO_RESULTS: Mapping[str, Any] = MappingProxyType({}) + + +@dataclass(frozen=True) +class Plan: + """What one run executes: the tasks in dependency order and how they relate.""" + + pipeline: str + tasks: tuple[Task, ...] + levels: Mapping[str, int] + upstream: Mapping[str, tuple[str, ...]] + max_workers: int + + +@dataclass(frozen=True) +class Services: + """What a run works with: where it is recorded, where it reports, what it may call.""" + + store: RunStore + bus: EventBus + registry: Any + actor: str + + +@dataclass(frozen=True) +class _Verdict: + """Why a task ends without running.""" + + status: TaskStatus + reason: str | None = None + error: str | None = None + + +class Engine: + """Coordinates one run of a :class:`Plan` and leaves the outcome on its ``run``.""" + + def __init__(self, plan: Plan, run: PipelineRun, services: Services) -> None: + self.run = run + self._plan = plan + self._services = services + self._reporter = Reporter(run, services.store, services.bus) + self._inbox: queue.SimpleQueue[str] = queue.SimpleQueue() + self._worker = Worker( + run, self._reporter, services.registry, services.actor, self._inbox.put + ) + self._tasks = {task.task_id: task for task in plan.tasks} + self._dependents: dict[str, list[str]] = {task.task_id: [] for task in plan.tasks} + self._blocked: dict[str, int] = {} + self._ready: deque[str] = deque() + self._waiting: deque[str] = deque() + self._flights: dict[str, Flight] = {} + self._cancel_seen = False + self._started = time.monotonic() + self._prepare() + + def _prepare(self) -> None: + """Work out which tasks are still to run (a resumed run keeps what succeeded).""" + states = self.run.tasks + for task in self._plan.tasks: + for dependency in task.depends_on: + self._dependents[dependency].append(task.task_id) + open_dependencies = [ + dependency + for dependency in task.depends_on + if not states[dependency].status.is_final + ] + self._blocked[task.task_id] = len(open_dependencies) + if not open_dependencies and not states[task.task_id].status.is_final: + self._ready.append(task.task_id) + + # ------------------------------------------------------------------ the run + + def execute(self) -> None: + """Run to the end in the calling thread, inside the run's correlation scope.""" + with correlation_scope(self.run.run_id), actor_scope(self._services.actor): + try: + self._reporter.save_run() + self._reporter.pipeline_event(PipelineStarted, RunStatus.RUNNING.value) + self._loop() + self._conclude() + except BaseException as error: + # Ctrl-C included: the run must not stay "running", and nothing may wait for ever. + self._abort(error) + raise + finally: + self.run.mark_done() + + def _loop(self) -> None: + while self._ready or self._waiting or self._flights: + self._observe_cancel() + self._admit() + self._launch() + if self._flights: + self._await() + + def _observe_cancel(self) -> None: + if self._cancel_seen or not self.run.cancel_token.is_cancelled: + return + self._cancel_seen = True + self._ready.clear() + self._waiting.clear() + for flight in self._flights.values(): + flight.token.cancel() + for task in self._plan.tasks: + state = self.run.tasks[task.task_id] + if state.status is TaskStatus.PENDING: + self._end_unstarted(state, _Verdict(TaskStatus.CANCELLED)) + + def _conclude(self) -> None: + states = list(self.run.tasks.values()) + broken = [state.task for state in states if state.status.is_failure] + stopped = any(state.status is TaskStatus.CANCELLED for state in states) + if self._cancel_seen and stopped: + self._close(RunStatus.CANCELLED, "the run was cancelled") + elif broken: + self._close(RunStatus.FAILED, f"did not succeed: {', '.join(broken)}") + else: + self._close(RunStatus.SUCCEEDED, None) + + def _close(self, status: RunStatus, error: str | None) -> None: + self.run.status = status + self.run.error = error + self.run.finished_at = utc_now() + self._reporter.save_run() + duration = elapsed_ms(self._started) + if status is RunStatus.SUCCEEDED: + self._reporter.pipeline_event(PipelineCompleted, status.value, Outcome(duration)) + else: + severity = Severity.WARNING if status is RunStatus.CANCELLED else None + self._reporter.pipeline_event( + PipelineFailed, status.value, Outcome(duration, error, severity) + ) + file_automation_logger.info( + "pipeline %s run %s: %s", self.run.pipeline, self.run.run_id, status.value + ) + + def _abort(self, error: BaseException) -> None: + """Leave a consistent record when the coordinator itself is interrupted.""" + file_automation_logger.error( + "pipeline %s run %s aborted: %r", self.run.pipeline, self.run.run_id, error + ) + for flight in self._flights.values(): + with flight.lock: + flight.abandoned = not flight.settled + flight.token.cancel() + for state in self.run.tasks.values(): + if not state.status.is_final: + state.status = TaskStatus.CANCELLED + state.finished_at = utc_now() + self._close(RunStatus.FAILED, describe_error(error)) + + # ------------------------------------------------------------------ admission + + def _admit(self) -> None: + """Decide for every ready task whether it runs; ending one may make others ready.""" + while self._ready: + task = self._tasks[self._ready.popleft()] + state = self.run.tasks[task.task_id] + verdict = self._gate(task, state) + if verdict is None: + self._waiting.append(task.task_id) + else: + self._end_unstarted(state, verdict) + self._release(task.task_id) + + def _gate(self, task: Task, state: TaskRun) -> _Verdict | None: + verdict = self._condition(task) + if verdict is not None or task.idempotency_key is None: + return verdict + return self._seen_before(task, state) + + def _condition(self, task: Task) -> _Verdict | None: + when = task.when + if callable(when): + return self._ask(task, when) + if when == ALWAYS: + return None + dependencies = [self.run.tasks[dependency] for dependency in task.depends_on] + failed = any(dependency.status.is_failure for dependency in dependencies) + if when == ON_FAILURE: + return None if failed else _Verdict(TaskStatus.SKIPPED, REASON_CONDITION) + if all(dependency.satisfied for dependency in dependencies): + return None + reason = REASON_UPSTREAM_FAILED if failed else REASON_UPSTREAM_SKIPPED + return _Verdict(TaskStatus.SKIPPED, reason) + + def _ask(self, task: Task, when: Condition) -> _Verdict | None: + probe = Flight(task, self.run.tasks[task.task_id], TaskToken(), self._results_for(task)) + try: + wanted = when(make_context(self.run, probe, 0)) + except Exception as error: # pylint: disable=broad-except + # Boundary: a broken condition fails its task, not the run loop. + return _Verdict(TaskStatus.FAILED, error=f"when: {describe_error(error)}") + return None if wanted else _Verdict(TaskStatus.SKIPPED, REASON_CONDITION) + + def _seen_before(self, task: Task, state: TaskRun) -> _Verdict | None: + """Skip a task whose idempotency key already has a succeeded execution.""" + try: + key = render_key(str(task.idempotency_key), self.run.params) + earlier = self._services.store.find_idempotent(self._plan.pipeline, task.task_id, key) + except PipelineException as error: + # Running it anyway could repeat the very thing the key protects. + return _Verdict(TaskStatus.FAILED, error=describe_error(error)) + state.idempotency_key = key + if earlier is None: + return None + state.result = earlier.result + state.result_is_repr = earlier.result_is_repr + return _Verdict(TaskStatus.SKIPPED, REASON_IDEMPOTENT) + + def _end_unstarted(self, state: TaskRun, verdict: _Verdict) -> None: + state.status = verdict.status + state.reason = verdict.reason + state.error = verdict.error + state.finished_at = utc_now() + self._reporter.save_task(state) + if verdict.status is TaskStatus.FAILED: + self._reporter.task_event( + TaskFailed, state, verdict.status.value, Outcome(error=verdict.error) + ) + + def _release(self, task_id: str) -> None: + """A task has ended: its dependents wait for one dependency less.""" + for dependent in self._dependents[task_id]: + self._blocked[dependent] -= 1 + if ( + self._blocked[dependent] == 0 + and self.run.tasks[dependent].status is TaskStatus.PENDING + ): + self._ready.append(dependent) + + # ------------------------------------------------------------------ flights + + def _results_for(self, task: Task) -> Mapping[str, Any]: + states = self.run.tasks + upstream = self._plan.upstream[task.task_id] + if not upstream: + return _NO_RESULTS + return MappingProxyType( + {other: states[other].result for other in upstream if states[other].satisfied} + ) + + def _launch(self) -> None: + while self._waiting and len(self._flights) < self._plan.max_workers: + task = self._tasks[self._waiting.popleft()] + state = self.run.tasks[task.task_id] + state.status = TaskStatus.RUNNING + state.started_at = utc_now() + flight = Flight(task, state, TaskToken(), self._results_for(task)) + self._flights[task.task_id] = flight + threading.Thread( + target=self._worker, + args=(flight,), + name=f"pipeline-{self._plan.pipeline}-{task.task_id}", + daemon=True, + ).start() + + def _patience(self) -> float: + """Seconds to wait for a landing before looking at deadlines and cancellation again.""" + deadlines = [ + flight.deadline for flight in self._flights.values() if flight.deadline is not None + ] + if not deadlines: + return _POLL_SECONDS + return max(0.0, min(_POLL_SECONDS, min(deadlines) - time.monotonic())) + + def _await(self) -> None: + try: + landed: str | None = self._inbox.get(timeout=self._patience()) + except queue.Empty: + landed = None + if landed is not None and self._flights.pop(landed, None) is not None: + self._release(landed) + self._expire() + + def _expire(self) -> None: + now = time.monotonic() + for task_id, flight in list(self._flights.items()): + if flight.deadline is None or now < flight.deadline: + continue + with flight.lock: + if flight.settled: + continue # it finished in time; its landing is on the way + flight.abandoned = True + flight.token.cancel() + del self._flights[task_id] + self._time_out(flight) + self._release(task_id) + + def _time_out(self, flight: Flight) -> None: + state = flight.state + state.status = TaskStatus.TIMEOUT + state.error = f"TimeoutError: no result within {flight.task.timeout:g} s" + state.finished_at = utc_now() + self._reporter.save_task(state) + self._reporter.task_event(TaskFailed, state, state.status.value, Outcome(error=state.error)) diff --git a/automation_file/pipeline/store.py b/automation_file/pipeline/store.py new file mode 100644 index 0000000..1887b3d --- /dev/null +++ b/automation_file/pipeline/store.py @@ -0,0 +1,360 @@ +"""Where runs are recorded: every task transition as it happens, and the history afterwards. + +A :class:`RunStore` is what makes checkpoint and resume, idempotency keys and the +execution history work. :class:`MemoryRunStore` keeps runs for the life of the +process and is the default; :class:`SQLiteRunStore` keeps them in a file, so a run +can be resumed and an idempotency key honoured after a restart. + +Both keep the JSON form of a run (:meth:`PipelineRun.to_dict`): what comes back +from a store is a copy, and a result JSON cannot hold comes back as its ``repr``. +Parameters and results are stored as given, so keep secrets out of them. +""" + +from __future__ import annotations + +import copy +import json +import os +import sqlite3 +import threading +from abc import ABC, abstractmethod +from collections.abc import Iterator +from contextlib import closing, contextmanager +from pathlib import Path +from typing import Any + +from automation_file.pipeline.errors import PipelineException +from automation_file.pipeline.model import PipelineRun, TaskRun, TaskStatus + +_SCHEMA_VERSION = "1" +_VERSION_KEY = "schema_version" +_DEFAULT_MEMORY_RUNS = 1000 +_BUSY_SECONDS = 5.0 +_SCHEMA = """ +CREATE TABLE IF NOT EXISTS pipeline_meta ( + key TEXT PRIMARY KEY, + value TEXT NOT NULL +); +CREATE TABLE IF NOT EXISTS pipeline_runs ( + run_id TEXT PRIMARY KEY, + pipeline TEXT NOT NULL, + status TEXT NOT NULL, + dry_run INTEGER NOT NULL, + params TEXT NOT NULL, + error TEXT, + started_at TEXT, + finished_at TEXT +); +CREATE INDEX IF NOT EXISTS idx_pipeline_runs_started ON pipeline_runs (pipeline, started_at); +CREATE TABLE IF NOT EXISTS pipeline_tasks ( + run_id TEXT NOT NULL, + task TEXT NOT NULL, + position INTEGER NOT NULL, + status TEXT NOT NULL, + level INTEGER NOT NULL, + attempts INTEGER NOT NULL, + result TEXT NOT NULL, + result_is_repr INTEGER NOT NULL, + error TEXT, + reason TEXT, + idempotency_key TEXT, + started_at TEXT, + finished_at TEXT, + PRIMARY KEY (run_id, task) +); +CREATE INDEX IF NOT EXISTS idx_pipeline_tasks_key ON pipeline_tasks (task, idempotency_key); +""" +_SELECT_VERSION = "SELECT value FROM pipeline_meta WHERE key = ?" +_INSERT_VERSION = "INSERT INTO pipeline_meta (key, value) VALUES (?, ?)" +_UPSERT_RUN = ( + "INSERT INTO pipeline_runs" + " (run_id, pipeline, status, dry_run, params, error, started_at, finished_at)" + " VALUES (?, ?, ?, ?, ?, ?, ?, ?)" + " ON CONFLICT (run_id) DO UPDATE SET pipeline = excluded.pipeline," + " status = excluded.status, dry_run = excluded.dry_run, params = excluded.params," + " error = excluded.error, started_at = excluded.started_at," + " finished_at = excluded.finished_at" +) +_RUN_EXISTS = "SELECT 1 FROM pipeline_runs WHERE run_id = ?" +_DELETE_TASKS = "DELETE FROM pipeline_tasks WHERE run_id = ?" +_UPSERT_TASK = ( + "INSERT INTO pipeline_tasks" + " (run_id, position, task, status, level, attempts, result, result_is_repr, error," + " reason, idempotency_key, started_at, finished_at)" + " VALUES (?, (SELECT COALESCE(MAX(position), -1) + 1 FROM pipeline_tasks WHERE run_id = ?)," + " ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)" + " ON CONFLICT (run_id, task) DO UPDATE SET status = excluded.status," + " level = excluded.level, attempts = excluded.attempts, result = excluded.result," + " result_is_repr = excluded.result_is_repr, error = excluded.error," + " reason = excluded.reason, idempotency_key = excluded.idempotency_key," + " started_at = excluded.started_at, finished_at = excluded.finished_at" +) +_SELECT_RUN = ( + "SELECT run_id, pipeline, status, dry_run, params, error, started_at, finished_at" + " FROM pipeline_runs WHERE run_id = ?" +) +_SELECT_RUNS = ( + "SELECT run_id, pipeline, status, dry_run, params, error, started_at, finished_at" + " FROM pipeline_runs WHERE (? IS NULL OR pipeline = ?)" + " ORDER BY started_at DESC, rowid DESC LIMIT ?" +) +_SELECT_TASKS = ( + "SELECT task, status, level, attempts, result, result_is_repr, error, reason," + " idempotency_key, started_at, finished_at" + " FROM pipeline_tasks WHERE run_id = ? ORDER BY position" +) +_SELECT_IDEMPOTENT = ( + "SELECT t.task, t.status, t.level, t.attempts, t.result, t.result_is_repr, t.error," + " t.reason, t.idempotency_key, t.started_at, t.finished_at" + " FROM pipeline_tasks AS t JOIN pipeline_runs AS r ON r.run_id = t.run_id" + " WHERE r.pipeline = ? AND t.task = ? AND t.idempotency_key = ? AND t.status = ?" + " ORDER BY t.finished_at DESC LIMIT 1" +) + + +class RunStore(ABC): + """The contract of a run store. Every method is safe to call from several threads.""" + + @abstractmethod + def save_run(self, run: PipelineRun) -> None: + """Record ``run`` with all of its tasks, replacing what was stored under its ID.""" + + @abstractmethod + def save_task(self, run_id: str, task: TaskRun) -> None: + """Record one task of a stored run; an unknown ``run_id`` raises ``PipelineException``.""" + + @abstractmethod + def get_run(self, run_id: str) -> PipelineRun | None: + """Return the stored run, or ``None`` when there is none with that ID.""" + + @abstractmethod + def list_runs(self, pipeline: str | None = None, limit: int = 50) -> list[PipelineRun]: + """Return up to ``limit`` runs, newest first, of one pipeline or of all.""" + + @abstractmethod + def find_idempotent(self, pipeline: str, task: str, key: str) -> TaskRun | None: + """Return the latest succeeded execution of ``task`` in ``pipeline`` under ``key``.""" + + +class MemoryRunStore(RunStore): + """Runs kept in memory, for the life of the process. + + At most ``max_runs`` are kept; the oldest are dropped first, and with them the + idempotency keys they held. + """ + + def __init__(self, max_runs: int = _DEFAULT_MEMORY_RUNS) -> None: + if max_runs < 1: + raise ValueError("max_runs must be >= 1") + self._max_runs = max_runs + self._lock = threading.Lock() + self._runs: dict[str, dict[str, Any]] = {} + + def save_run(self, run: PipelineRun) -> None: + record = run.to_dict() + with self._lock: + self._runs[run.run_id] = record + while len(self._runs) > self._max_runs: + del self._runs[next(iter(self._runs))] + + def save_task(self, run_id: str, task: TaskRun) -> None: + record = task.to_dict() + with self._lock: + run = self._runs.get(run_id) + if run is None: + raise PipelineException(f"unknown run {run_id!r}") + run["tasks"][task.task] = record + + def get_run(self, run_id: str) -> PipelineRun | None: + with self._lock: + record = copy.deepcopy(self._runs.get(run_id)) + return None if record is None else PipelineRun.from_dict(record) + + def list_runs(self, pipeline: str | None = None, limit: int = 50) -> list[PipelineRun]: + with self._lock: + chosen = [ + record + for record in reversed(self._runs.values()) + if pipeline is None or record["pipeline"] == pipeline + ] + chosen.sort(key=lambda record: record["started_at"] or "", reverse=True) + records = copy.deepcopy(chosen[: max(limit, 0)]) + return [PipelineRun.from_dict(record) for record in records] + + def find_idempotent(self, pipeline: str, task: str, key: str) -> TaskRun | None: + latest: dict[str, Any] | None = None + with self._lock: + for run in self._runs.values(): + record = run["tasks"].get(task) if run["pipeline"] == pipeline else None + if record is None or not _is_execution(record, key): + continue + if latest is None or _finished(record) >= _finished(latest): + latest = record + found = copy.deepcopy(latest) + return None if found is None else TaskRun.from_dict(found) + + +def _is_execution(record: dict[str, Any], key: str) -> bool: + return record["status"] == TaskStatus.SUCCEEDED.value and record["idempotency_key"] == key + + +def _finished(record: dict[str, Any]) -> str: + return record["finished_at"] or "" + + +class SQLiteRunStore(RunStore): + """Runs kept in a SQLite file, so they outlive the process. + + Every call opens a short-lived connection under one lock, uses parameterised + statements only, and commits before it returns: a crash loses at most the + transition that was being written. The file carries a schema version; a file + written by an unknown version is refused. + """ + + def __init__(self, path: str | os.PathLike[str]) -> None: + self._path = Path(path) + self._lock = threading.Lock() + try: + self._path.parent.mkdir(parents=True, exist_ok=True) + except OSError as error: + raise PipelineException(f"run store {self._path}: {error}") from error + with self._session() as connection: + connection.executescript(_SCHEMA) + row = connection.execute(_SELECT_VERSION, (_VERSION_KEY,)).fetchone() + if row is None: + connection.execute(_INSERT_VERSION, (_VERSION_KEY, _SCHEMA_VERSION)) + elif row[0] != _SCHEMA_VERSION: + raise PipelineException( + f"run store {self._path}: schema version {row[0]} is not supported" + f" (expected {_SCHEMA_VERSION})" + ) + + @property + def path(self) -> Path: + """The SQLite file.""" + return self._path + + @contextmanager + def _session(self) -> Iterator[sqlite3.Connection]: + try: + with ( + self._lock, + closing(sqlite3.connect(self._path, timeout=_BUSY_SECONDS)) as connection, + connection, + ): + yield connection + except sqlite3.Error as error: + raise PipelineException(f"run store {self._path}: {error}") from error + + def save_run(self, run: PipelineRun) -> None: + record = run.to_dict() + with self._session() as connection: + connection.execute(_UPSERT_RUN, _run_row(record)) + connection.execute(_DELETE_TASKS, (run.run_id,)) + for task in record["tasks"].values(): + connection.execute(_UPSERT_TASK, _task_row(run.run_id, task)) + + def save_task(self, run_id: str, task: TaskRun) -> None: + row = _task_row(run_id, task.to_dict()) + with self._session() as connection: + if connection.execute(_RUN_EXISTS, (run_id,)).fetchone() is None: + raise PipelineException(f"unknown run {run_id!r}") + connection.execute(_UPSERT_TASK, row) + + def get_run(self, run_id: str) -> PipelineRun | None: + with self._session() as connection: + row = connection.execute(_SELECT_RUN, (run_id,)).fetchone() + return None if row is None else _load_run(connection, row) + + def list_runs(self, pipeline: str | None = None, limit: int = 50) -> list[PipelineRun]: + with self._session() as connection: + rows = connection.execute(_SELECT_RUNS, (pipeline, pipeline, max(limit, 0))).fetchall() + return [_load_run(connection, row) for row in rows] + + def find_idempotent(self, pipeline: str, task: str, key: str) -> TaskRun | None: + with self._session() as connection: + row = connection.execute( + _SELECT_IDEMPOTENT, (pipeline, task, key, TaskStatus.SUCCEEDED.value) + ).fetchone() + return None if row is None else TaskRun.from_dict(_task_record(row)) + + +def _run_row(record: dict[str, Any]) -> tuple[Any, ...]: + return ( + record["run_id"], + record["pipeline"], + record["status"], + int(record["dry_run"]), + json.dumps(record["params"]), + record["error"], + record["started_at"], + record["finished_at"], + ) + + +def _task_row(run_id: str, record: dict[str, Any]) -> tuple[Any, ...]: + return ( + run_id, + run_id, + record["task"], + record["status"], + record["level"], + record["attempts"], + json.dumps(record["result"]), + int(record["result_is_repr"]), + record["error"], + record["reason"], + record["idempotency_key"], + record["started_at"], + record["finished_at"], + ) + + +def _task_record(row: tuple[Any, ...]) -> dict[str, Any]: + return { + "task": row[0], + "status": row[1], + "level": row[2], + "attempts": row[3], + "result": json.loads(row[4]), + "result_is_repr": bool(row[5]), + "error": row[6], + "reason": row[7], + "idempotency_key": row[8], + "started_at": row[9], + "finished_at": row[10], + } + + +def _load_run(connection: sqlite3.Connection, row: tuple[Any, ...]) -> PipelineRun: + tasks = connection.execute(_SELECT_TASKS, (row[0],)).fetchall() + return PipelineRun.from_dict( + { + "run_id": row[0], + "pipeline": row[1], + "status": row[2], + "dry_run": bool(row[3]), + "params": json.loads(row[4]), + "error": row[5], + "started_at": row[6], + "finished_at": row[7], + "tasks": {task[0]: _task_record(task) for task in tasks}, + } + ) + + +_default: dict[str, RunStore] = {"store": MemoryRunStore()} + + +def default_run_store() -> RunStore: + """Return the store used when ``run``, ``resume`` and the actions are given none.""" + return _default["store"] + + +def set_default_run_store(store: RunStore) -> RunStore: + """Make ``store`` the default one and return the store it replaces.""" + if not isinstance(store, RunStore): + raise PipelineException(f"not a RunStore: {store!r}") + previous = _default["store"] + _default["store"] = store + return previous diff --git a/automation_file/pipeline/substitution.py b/automation_file/pipeline/substitution.py new file mode 100644 index 0000000..3faacde --- /dev/null +++ b/automation_file/pipeline/substitution.py @@ -0,0 +1,164 @@ +"""``${params.}`` and ``${tasks..result}`` in action arguments. + +There is no expression language and nothing is evaluated: a placeholder names +one run parameter or the result of one upstream task. + +* ``${params.}`` may stand anywhere in a string and is replaced by the + parameter as text. A string that is exactly one such placeholder becomes the + parameter itself, so a number stays a number. +* ``${tasks..result}`` must be the whole string and is replaced by the result + object of that upstream task (``None`` when the task did not succeed). + +Any other ``${...}`` text is left alone. +""" + +from __future__ import annotations + +import re +from collections.abc import Collection, Iterator, Mapping, Sequence +from dataclasses import dataclass +from typing import Any + +from automation_file.pipeline.errors import PipelineException + +#: What a task ID or a parameter name looks like (a regular expression, also used by the schema). +NAME_PATTERN = r"^[^\s.${}]+$" +NAME_RULE = "expected a non-empty string without whitespace, '.', '$', '{' or '}'" +_NAME = re.compile(NAME_PATTERN) +_PLACEHOLDER = re.compile(r"\$\{(params|tasks)\.([^}]*)\}") +_TASKS = "tasks" +_RESULT_SUFFIX = ".result" +_SYNTAX = "use ${params.} or ${tasks..result}" + + +def is_name(value: object) -> bool: + """Return whether ``value`` can be a task ID or a parameter name.""" + return isinstance(value, str) and _NAME.fullmatch(value) is not None + + +@dataclass(frozen=True) +class Reference: + """One placeholder found in a string.""" + + kind: str # "params" or "tasks" + name: str | None # the parameter name or task ID; ``None`` when the placeholder is malformed + text: str # the placeholder as written + whole: bool # whether the placeholder is the entire string + + +def _reference(match: re.Match[str], whole: bool) -> Reference: + kind, body = match.group(1), match.group(2) + if kind == _TASKS: + body = body[: -len(_RESULT_SUFFIX)] if body.endswith(_RESULT_SUFFIX) else "" + return Reference(kind, body if is_name(body) else None, match.group(0), whole) + + +def references(text: str) -> list[Reference]: + """Return the pipeline placeholders of ``text``, in order.""" + matches = list(_PLACEHOLDER.finditer(text)) + whole = len(matches) == 1 and matches[0].span() == (0, len(text)) + return [_reference(match, whole) for match in matches] + + +def _task_reference_problem(reference: Reference, upstream: Collection[str] | None) -> str | None: + if upstream is None: + return f"{reference.text}: only ${{params.}} can be used here" + if not reference.whole: + return f"{reference.text} must be the whole string" + if reference.name not in upstream: + return f"{reference.text}: {reference.name!r} is not an upstream task (see depends_on)" + return None + + +def text_problems( + text: str, + upstream: Collection[str] | None, + params: Mapping[str, Any] | None = None, +) -> list[str]: + """Return what is wrong with the placeholders of ``text``. + + ``upstream`` holds the task IDs whose result may be used; ``None`` forbids + task results altogether (an idempotency key). With ``params``, a parameter + the run was not given is a problem too. + """ + problems: list[str] = [] + for reference in references(text): + problem: str | None = None + if reference.name is None: + problem = f"malformed placeholder {reference.text} ({_SYNTAX})" + elif reference.kind == _TASKS: + problem = _task_reference_problem(reference, upstream) + elif params is not None and reference.name not in params: + problem = f"unknown parameter {reference.name!r}" + if problem is not None: + problems.append(problem) + return problems + + +def walk_strings(value: Any, path: str) -> Iterator[tuple[str, str]]: + """Yield ``(path, text)`` for every string inside nested mappings and lists.""" + if isinstance(value, str): + yield path, value + elif isinstance(value, Mapping): + for key, item in value.items(): + yield from walk_strings(item, f"{path}.{key}") + elif isinstance(value, (list, tuple)): + for index, item in enumerate(value): + yield from walk_strings(item, f"{path}[{index}]") + + +def action_reference_problems( + action: Sequence[Any], + path: str, + upstream: Collection[str], + params: Mapping[str, Any] | None = None, +) -> list[str]: + """Return the placeholder problems of an action's arguments, each with its path.""" + problems: list[str] = [] + for index, payload in enumerate(action[1:], start=1): + for where, text in walk_strings(payload, f"{path}[{index}]"): + problems.extend( + f"{where}: {problem}" for problem in text_problems(text, upstream, params) + ) + return problems + + +def _resolve(reference: Reference, params: Mapping[str, Any], results: Mapping[str, Any]) -> Any: + if reference.name is None: + raise PipelineException(f"malformed placeholder {reference.text} ({_SYNTAX})") + if reference.kind == _TASKS: + return results.get(reference.name) + if reference.name not in params: + raise PipelineException(f"unknown parameter {reference.name!r} in {reference.text}") + return params[reference.name] + + +def _render_text(text: str, params: Mapping[str, Any], results: Mapping[str, Any]) -> Any: + found = references(text) + if not found: + return text + if found[0].whole: + return _resolve(found[0], params, results) + return _PLACEHOLDER.sub( + lambda match: str(_resolve(_reference(match, False), params, results)), text + ) + + +def render(value: Any, params: Mapping[str, Any], results: Mapping[str, Any]) -> Any: + """Return a copy of ``value`` with every placeholder replaced. + + Mappings and lists are walked; mapping keys and non-string values are kept. + An unknown parameter raises :class:`PipelineException`. + """ + if isinstance(value, str): + return _render_text(value, params, results) + if isinstance(value, Mapping): + return {key: render(item, params, results) for key, item in value.items()} + if isinstance(value, list): + return [render(item, params, results) for item in value] + return value + + +def render_key(key: str, params: Mapping[str, Any]) -> str: + """Return an idempotency key with its ``${params.}`` placeholders filled in.""" + return str(render(key, params, {})) diff --git a/automation_file/pipeline/worker.py b/automation_file/pipeline/worker.py new file mode 100644 index 0000000..906e8ec --- /dev/null +++ b/automation_file/pipeline/worker.py @@ -0,0 +1,267 @@ +"""A task on a thread of its own: the attempts, the retries, their events and the outcome. + +The coordinator (:mod:`automation_file.pipeline.runner`) starts one thread per +task and hands it a :class:`Flight`. The thread makes the attempts, publishes +``task.started`` / ``task.completed`` / ``task.failed`` for each of them, records +the outcome and tells the coordinator it has landed. + +A flight that passed its deadline is *abandoned*: the coordinator has already +recorded ``timeout``, and whatever the thread still does is ignored. The flight's +lock makes "the task finished" and "the task timed out" exclude each other. +""" + +from __future__ import annotations + +import threading +import time +from collections.abc import Callable, Mapping +from dataclasses import dataclass, field +from types import MappingProxyType +from typing import Any + +from automation_file.core.progress import CancellationToken, CancelledException +from automation_file.events import ( + Severity, + TaskCompleted, + TaskFailed, + TaskStarted, + actor_scope, + correlation_scope, +) +from automation_file.logging_config import file_automation_logger +from automation_file.pipeline.errors import PipelineException +from automation_file.pipeline.model import ( + PipelineRun, + Task, + TaskContext, + TaskRun, + TaskStatus, + describe_error, + utc_now, +) +from automation_file.pipeline.reporting import Outcome, Reporter +from automation_file.pipeline.substitution import render + +RETRYING = "retrying" +_NO_OUTCOME = "PipelineException: the task ended without an outcome" + + +class TaskToken(CancellationToken): + """A cancellation token a back-off can sleep on.""" + + def __init__(self) -> None: + super().__init__() + self._woken = threading.Event() + + def cancel(self) -> None: + super().cancel() + self._woken.set() + + def wait(self, seconds: float) -> bool: + """Wait up to ``seconds``; return whether the token was cancelled by then.""" + return self._woken.wait(seconds) + + +def pause(token: TaskToken, seconds: float) -> bool: + """Wait out a back-off; return ``True`` when the task was cancelled meanwhile.""" + if seconds <= 0: + return token.is_cancelled + return token.wait(seconds) + + +def elapsed_ms(started: float) -> float: + """Return the milliseconds since the ``time.monotonic()`` reading ``started``.""" + return round((time.monotonic() - started) * 1000, 3) + + +@dataclass +class Flight: + """One task in the air: its definition, its state, its token and its deadline. + + ``deadline`` is a ``time.monotonic()`` reading, set when the first attempt + starts, so the timeout measures the task and not the wait for its thread. + """ + + task: Task + state: TaskRun + token: TaskToken + results: Mapping[str, Any] + deadline: float | None = None + lock: threading.Lock = field(default_factory=threading.Lock) + settled: bool = False + abandoned: bool = False + + +def make_context(run: PipelineRun, flight: Flight, attempt: int) -> TaskContext: + """Build what a task callable or a ``when`` callable receives.""" + return TaskContext( + pipeline=run.pipeline, + run_id=run.run_id, + task=flight.task.task_id, + attempt=attempt, + params=MappingProxyType(run.params), + results=flight.results, + cancel=flight.token, + dry_run=False, + ) + + +def call_action(action: list[Any], registry: Any) -> Any: + """Call the registered command of ``action`` directly, so a failure raises.""" + name = action[0] + command = registry.resolve(name) + if command is None: + raise PipelineException(f"unknown action {name!r}") + if len(action) == 1: + return command() + arguments = action[1] + if isinstance(arguments, Mapping): + return command(**arguments) + return command(*arguments) + + +class Worker: + """Runs flights; one instance serves every task thread of a run.""" + + def __init__( + self, + run: PipelineRun, + reporter: Reporter, + registry: Any, + actor: str, + landed: Callable[[str], None], + ) -> None: + self._run = run + self._reporter = reporter + self._registry = registry + self._actor = actor + self._landed = landed + + def __call__(self, flight: Flight) -> None: + """Thread entry point. A new thread has no context: re-enter the run's scopes.""" + with correlation_scope(self._run.run_id), actor_scope(self._actor): + try: + self._fly(flight) + finally: + self._landed(flight.task.task_id) + + def _fly(self, flight: Flight) -> None: + try: + self._attempts(flight) + except Exception as error: # pylint: disable=broad-except + # Boundary of the task thread: the runtime itself failed, for instance a store that + # raised something other than PipelineException. The task is failed just below. + file_automation_logger.error( + "pipeline %s run %s task %s: %r", + self._run.pipeline, + self._run.run_id, + flight.task.task_id, + error, + ) + finally: + # Does nothing when the attempts recorded an outcome, which is the normal case. + self._settle(flight, TaskStatus.FAILED, error=_NO_OUTCOME) + + def _attempts(self, flight: Flight) -> None: + for attempt in range(1, flight.task.retry.max_attempts + 1): + if not self._begin(flight, attempt): + return + started = time.monotonic() + try: + result = self._invoke(flight, attempt) + except CancelledException as error: + self._settle( + flight, TaskStatus.CANCELLED, error=describe_error(error), started=started + ) + return + except Exception as error: # pylint: disable=broad-except + # Boundary: a task's failure is recorded on the run, never raised into the runtime. + if not self._try_again(flight, attempt, error, started): + return + except SystemExit as error: + # sys.exit() in a task ends only this thread; what it leaves behind is a failure. + self._settle( + flight, TaskStatus.FAILED, error=describe_error(error), started=started + ) + return + else: + self._settle(flight, TaskStatus.SUCCEEDED, result=result, started=started) + return + + def _begin(self, flight: Flight, attempt: int) -> bool: + """Announce an attempt; ``False`` when the task was cancelled or timed out first.""" + if flight.token.is_cancelled: + self._settle(flight, TaskStatus.CANCELLED, error=flight.state.error, quiet=True) + return False + with flight.lock: + if flight.abandoned: + return False + if flight.deadline is None and flight.task.timeout is not None: + flight.deadline = time.monotonic() + flight.task.timeout + flight.state.attempts = attempt + self._reporter.save_task(flight.state) + self._reporter.task_event(TaskStarted, flight.state, TaskStatus.RUNNING.value) + return True + + def _invoke(self, flight: Flight, attempt: int) -> Any: + work = flight.task.work + if callable(work): + return work(make_context(self._run, flight, attempt)) + return call_action(render(work, self._run.params, flight.results), self._registry) + + def _try_again(self, flight: Flight, attempt: int, error: Exception, started: float) -> bool: + """Record a failed attempt; return whether another one follows.""" + policy = flight.task.retry + text = describe_error(error) + if attempt >= policy.max_attempts or not policy.retries(error): + self._settle(flight, TaskStatus.FAILED, error=text, started=started) + return False + if flight.token.is_cancelled: + # It would have been tried again; the cancellation is why it was not. + self._settle(flight, TaskStatus.CANCELLED, error=text, started=started) + return False + with flight.lock: + if flight.abandoned: + return False + flight.state.error = text + self._reporter.save_task(flight.state) + self._reporter.task_event( + TaskFailed, + flight.state, + RETRYING, + Outcome(elapsed_ms(started), text, Severity.WARNING), + ) + pause(flight.token, policy.delay(attempt)) + return True + + def _settle( + self, + flight: Flight, + status: TaskStatus, + *, + result: Any = None, + error: str | None = None, + started: float | None = None, + quiet: bool = False, + ) -> None: + """Record the task's outcome once; a flight already settled or abandoned is left alone.""" + with flight.lock: + if flight.settled or flight.abandoned: + return + flight.settled = True + state = flight.state + state.status = status + state.result = result + state.error = error + state.finished_at = utc_now() + self._reporter.save_task(state) + if quiet: + return + duration = None if started is None else elapsed_ms(started) + if status is TaskStatus.SUCCEEDED: + self._reporter.task_event(TaskCompleted, state, status.value, Outcome(duration)) + return + severity = Severity.WARNING if status is TaskStatus.CANCELLED else None + self._reporter.task_event( + TaskFailed, state, status.value, Outcome(duration, error, severity) + ) diff --git a/automation_file/server/action_acl.py b/automation_file/server/action_acl.py index e6f2472..b4dc6af 100644 --- a/automation_file/server/action_acl.py +++ b/automation_file/server/action_acl.py @@ -4,11 +4,23 @@ :meth:`ActionACL.filter` before dispatch. If any referenced action is denied the whole payload is rejected — partial execution would leave the caller in an ambiguous state. + +An action may carry other actions in its arguments: ``FA_execute_action`` takes +an action list, ``FA_pipeline_run`` a definition whose tasks name actions, the +scheduler and the triggers a list to run later. The ACL therefore checks every +registered action name that appears anywhere in the arguments, not only the +names at the top of the payload. A string argument that happens to equal a +registered action name is checked as that action: the cautious reading. + +What the payload does not contain cannot be checked: an action list or a +pipeline definition named by a file path (``FA_execute_files``, +``FA_pipeline_run`` with a path) and a stored run that ``FA_pipeline_resume`` +continues. Deny those actions for a client that must stay inside the list. """ from __future__ import annotations -from collections.abc import Iterable +from collections.abc import Container, Iterable, Iterator from dataclasses import dataclass, field from automation_file.exceptions import FileAutomationException @@ -18,6 +30,32 @@ class ActionNotPermittedException(FileAutomationException): """Raised when a payload references an action the ACL forbids.""" +def _registered_names() -> Container[str]: + """Return the names the shared executor dispatches, which is what a server runs.""" + from automation_file.core.action_executor import executor + + return executor.registry.event_dict + + +def nested_action_names(arguments: object, known: Container[str]) -> Iterator[str]: + """Yield every name of ``known`` found at any depth of ``arguments``. + + A name counts wherever it stands: as a list element, a mapping value or a + mapping key. The walk is iterative, so nesting cannot exhaust the stack. + """ + pending = [arguments] + while pending: + value = pending.pop() + if isinstance(value, str): + if value in known: + yield value + elif isinstance(value, dict): + pending.extend(value.keys()) + pending.extend(value.values()) + elif isinstance(value, (list, tuple)): + pending.extend(value) + + @dataclass(frozen=True) class ActionACL: """Allow/deny list for inbound action names. @@ -59,6 +97,8 @@ def _iter_names(payload: object) -> Iterable[str]: payload = payload.get("actions", []) if not isinstance(payload, list): return + known = _registered_names() for entry in payload: if isinstance(entry, list) and entry and isinstance(entry[0], str): yield entry[0] + yield from nested_action_names(entry[1:], known) diff --git a/automation_file/server/mcp_server.py b/automation_file/server/mcp_server.py index fb4088f..d4951a2 100644 --- a/automation_file/server/mcp_server.py +++ b/automation_file/server/mcp_server.py @@ -30,6 +30,7 @@ from automation_file.core.action_registry import ActionRegistry from automation_file.exceptions import MCPServerException from automation_file.logging_config import file_automation_logger +from automation_file.server.action_acl import nested_action_names _JSONRPC_VERSION = "2.0" _PROTOCOL_VERSION = "2024-11-05" @@ -137,6 +138,7 @@ def _handle_tools_call(self, params: dict[str, Any]) -> dict[str, Any]: command = self._registry.resolve(name) if command is None: raise MCPServerException(f"unknown tool: {name}") + self._require_exposed(name, arguments) try: value = command(**arguments) except TypeError as error: @@ -146,6 +148,20 @@ def _handle_tools_call(self, params: dict[str, Any]) -> dict[str, Any]: "isError": False, } + def _require_exposed(self, name: str, arguments: dict[str, Any]) -> None: + """Refuse a call whose arguments name an action this server does not expose. + + A tool such as ``FA_execute_action`` or ``FA_pipeline_run`` runs the actions + its arguments name through the shared executor, whatever this server's + registry was narrowed to. Without the check an allow list would stop at the + tool name. + """ + for nested in nested_action_names(arguments, executor.registry.event_dict): + if self._registry.resolve(nested) is None: + raise MCPServerException( + f"{name} names the action {nested}, which this server does not expose" + ) + @staticmethod def _write(writer: TextIO, response: dict[str, Any]) -> None: writer.write(json.dumps(response, default=repr) + "\n") diff --git a/automation_file/storage/actions.py b/automation_file/storage/actions.py index 318d6b0..a97cc9c 100644 --- a/automation_file/storage/actions.py +++ b/automation_file/storage/actions.py @@ -20,6 +20,7 @@ from collections.abc import Callable from typing import TYPE_CHECKING, Any +from automation_file.exceptions import StorageChecksumException from automation_file.logging_config import file_automation_logger from automation_file.storage.backend import DEFAULT_CHECKSUM_ALGORITHM from automation_file.storage.file import File @@ -95,12 +96,21 @@ def storage_checksum(uri: str, algorithm: str = DEFAULT_CHECKSUM_ALGORITHM) -> d return File(uri).checksum(algorithm).to_dict() -def storage_verify(uri: str, expected: str, algorithm: str = DEFAULT_CHECKSUM_ALGORITHM) -> bool: - """Return whether ``uri`` has the digest ``expected`` (``"sha256:..."`` or a bare digest).""" +def storage_verify( + uri: str, expected: str, algorithm: str = DEFAULT_CHECKSUM_ALGORITHM, strict: bool = False +) -> bool: + """Return whether ``uri`` has the digest ``expected`` (``"sha256:..."`` or a bare digest). + + With ``strict=True`` a mismatch raises ``StorageChecksumException`` instead of + returning ``False``, so a pipeline task or an action list stops at it. + """ matched = File(uri).verify(expected, algorithm=algorithm) - if not matched: - file_automation_logger.warning("storage_verify mismatch: %s", uri) - return matched + if matched: + return True + file_automation_logger.warning("storage_verify mismatch: %s", uri) + if strict: + raise StorageChecksumException(f"{uri} does not have the expected digest") + return False def storage_copy(source: str, target: str, overwrite: bool = True) -> dict[str, Any]: diff --git a/docs/source/API/api_index.rst b/docs/source/API/api_index.rst index e90ef8e..834eff8 100644 --- a/docs/source/API/api_index.rst +++ b/docs/source/API/api_index.rst @@ -229,3 +229,17 @@ actions. :caption: Audit Trail audit + +.. _api-pipeline: + +Chapter Q — Pipelines +===================== + +``Pipeline``, the task and run model, the run stores, the definition format +and its schema, and the ``FA_pipeline_*`` actions. + +.. toctree:: + :maxdepth: 2 + :caption: Pipelines + + pipeline diff --git a/docs/source/API/pipeline.rst b/docs/source/API/pipeline.rst new file mode 100644 index 0000000..9fbd58b --- /dev/null +++ b/docs/source/API/pipeline.rst @@ -0,0 +1,60 @@ +Pipelines +========= + +The pipeline runtime: tasks with dependencies, retry, timeout, cancellation, +conditions, idempotency keys, checkpoint and resume, a dry run and the execution +history. Usage is described in the manual chapter *Pipelines*. + +Pipeline +-------- + +.. automodule:: automation_file.pipeline.pipeline + :members: + +Tasks, retry policies and run state +----------------------------------- + +.. automodule:: automation_file.pipeline.model + :members: + +Run stores +---------- + +.. automodule:: automation_file.pipeline.store + :members: + +Definitions +----------- + +.. automodule:: automation_file.pipeline.definition + :members: + +.. automodule:: automation_file.pipeline.substitution + :members: + +.. automodule:: automation_file.pipeline.graph + :members: + +Actions +------- + +.. automodule:: automation_file.pipeline.actions + :members: + +Runtime +------- + +.. automodule:: automation_file.pipeline.runner + :members: + +.. automodule:: automation_file.pipeline.worker + :members: + +.. automodule:: automation_file.pipeline.reporting + :members: + +Exceptions +---------- + +.. automodule:: automation_file.pipeline.errors + :members: diff --git a/docs/source/Eng/eng_index.rst b/docs/source/Eng/eng_index.rst index edebc93..d8d41da 100644 --- a/docs/source/Eng/eng_index.rst +++ b/docs/source/Eng/eng_index.rst @@ -304,3 +304,18 @@ operational metrics. Notification routes are in :doc:`usage/notifications`. :caption: Audit Trail usage/audit + +.. _eng-pipeline: + +Chapter 20 — Pipelines +====================== + +Tasks with dependencies, run in order with retry, timeout, cancellation, +conditions, idempotency, checkpoint and resume, a dry run and an execution +history; written in Python or as a versioned YAML / JSON definition. + +.. toctree:: + :maxdepth: 2 + :caption: Pipelines + + usage/pipeline diff --git a/docs/source/Eng/usage/pipeline.rst b/docs/source/Eng/usage/pipeline.rst new file mode 100644 index 0000000..3bbdcbf --- /dev/null +++ b/docs/source/Eng/usage/pipeline.rst @@ -0,0 +1,737 @@ +Pipelines +========= + +``automation_file.pipeline`` runs a set of tasks in dependency order, with +independent tasks in parallel, and adds what a recurring job needs: retry, +timeout, cancellation, conditions, idempotency keys, checkpoint and resume, a +dry run and an execution history. + +A task is a Python callable or an ``FA_*`` action. A pipeline is built in Python +or written as a YAML / JSON document. A run reports only through events on the +bus (:doc:`event_bus`); it calls no notification sink and writes no audit row. + +:func:`~automation_file.execute_action_dag` (:doc:`dag`) is unchanged. It runs a +list of actions once and returns their results; use a pipeline when the run has +to be recorded, retried, resumed or observed. + +Minimal example +--------------- + +.. code-block:: python + + from automation_file.pipeline import Pipeline + + def count_rows(ctx): + return len(ctx.results["read"].splitlines()) + + pipeline = Pipeline("row-count") + pipeline.task("read", ["FA_storage_read_text", {"uri": "local:///data/report.csv"}]) + pipeline.task("count", count_rows, depends_on=["read"]) + + run = pipeline.run() + run.status # RunStatus.SUCCEEDED; equal to "succeeded" + run.tasks["count"].result # 42 + run.tasks["count"].attempts # 1 + +``run()`` returns when every task has ended. A task that fails does not raise: +the outcome is on ``run.status`` and on each entry of ``run.tasks``. + +Production example +------------------ + +A nightly job: fetch a file with retry and a timeout, check it, publish it at +most once per date, take a half-published report back if publishing failed, and +keep the runs in a SQLite file so a failed run can be resumed. + +.. code-block:: python + + from automation_file.pipeline import Pipeline, RetryPolicy, SQLiteRunStore + + WORK = "local:///var/tmp/report-${params.date}.csv" + TARGET = "azure://reports/${params.date}.csv" + store = SQLiteRunStore("/var/lib/automation/pipelines.db") + + def check(ctx): + ctx.cancel.raise_if_cancelled() # stop when cancelled or timed out + if ctx.results["download"]["size"] == 0: + raise ValueError("the report is empty") # a task fails by raising + return {"bytes": ctx.results["download"]["size"]} + + pipeline = Pipeline("daily-report", description="Fetch, check, publish", max_workers=4) + pipeline.task( + "download", + ["FA_storage_copy", {"source": "s3://input/${params.date}.csv", "target": WORK}], + retry=RetryPolicy(max_attempts=5, backoff_base=2.0, backoff_cap=60.0), + timeout=300.0, + ) + pipeline.task("check", check, depends_on=["download"], timeout=60.0) + pipeline.task( + "publish", + ["FA_storage_copy", {"source": WORK, "target": TARGET}], + depends_on=["check"], + retry=RetryPolicy(max_attempts=3, backoff_base=1.0), + idempotency_key="publish-${params.date}", # never published twice for one date + ) + pipeline.task( + "withdraw", # clean-up: runs only when publish failed + ["FA_storage_delete", {"uri": TARGET, "missing_ok": True}], + depends_on=["publish"], + when="on_failure", + ) + pipeline.task("tidy", ["FA_storage_delete", {"uri": WORK}], depends_on=["publish"]) + + run = pipeline.run(params={"date": "2026-10-08"}, store=store) + if run.status != "succeeded": + for task_id, state in run.tasks.items(): + print(task_id, state.status.value, state.error or state.reason or "") + + # Later, in this process or another one, once the cause is fixed: + run = pipeline.resume(run.run_id, store=store) # runs only what did not succeed + +When ``publish`` fails after its three attempts, ``withdraw`` runs, ``tidy`` is +skipped and the run is ``failed``. ``resume`` keeps ``download`` and ``check``, +runs ``publish`` again and then ``tidy``. A later run for the same date +downloads and checks again but skips ``publish``: its key has succeeded. + +The same pipeline without Python, as ``daily-report.yaml``. ``check`` is a +callable and has no place in a document, so this version publishes what it +downloaded: + +.. code-block:: yaml + + schema_version: 1 + name: daily-report + description: Fetch and publish the daily report + max_workers: 4 + schedule: {cron: "0 2 * * *", timezone: Asia/Taipei} + params: {date: "2026-10-08"} + tasks: + download: + action: ["FA_storage_copy", {"source": "s3://input/${params.date}.csv", + "target": "local:///var/tmp/report-${params.date}.csv"}] + retry: {max_attempts: 5, backoff: 2, backoff_cap: 60, + on: [StorageTransientException, ConnectionError]} + timeout: 300 + publish: + action: ["FA_storage_copy", {"source": "local:///var/tmp/report-${params.date}.csv", + "target": "azure://reports/${params.date}.csv"}] + depends_on: [download] + retry: {max_attempts: 3, backoff: 1} + idempotency_key: "publish-${params.date}" + withdraw: + action: ["FA_storage_delete", {"uri": "azure://reports/${params.date}.csv", + "missing_ok": true}] + depends_on: [publish] + when: on_failure + tidy: + action: ["FA_storage_delete", {"uri": "local:///var/tmp/report-${params.date}.csv"}] + depends_on: [publish] + +.. code-block:: python + + pipeline = Pipeline.from_file("daily-report.yaml") + run = pipeline.run(params={"date": "2026-10-09"}, store=store) + +Tasks +----- + +A task's work is one of two things. + +**A callable** taking one :class:`~automation_file.pipeline.model.TaskContext`. +What it returns is the task's result; raising fails the task. + +.. list-table:: + :header-rows: 1 + :widths: 20 80 + + * - Field + - Meaning + * - ``pipeline`` + - The pipeline's name. + * - ``run_id`` + - The ID of this run. It is also the correlation ID of every event. + * - ``task`` + - The task's ID. + * - ``attempt`` + - The attempt number, from 1 (``0`` in a ``when`` callable). + * - ``params`` + - The run's parameters, read-only: the pipeline's defaults overridden by + ``run(params=...)``. + * - ``results`` + - Task ID to result, read-only, for every upstream task that succeeded, + direct or not. A task that failed or was skipped has no entry. + * - ``cancel`` + - A ``CancellationToken``, set when the run is cancelled or the task's + timeout has passed. See `Timeout`_. + * - ``dry_run`` + - Always ``False``: a dry run executes nothing. + +**An action** in one of the three shapes ``[name]``, ``[name, {kwargs}]`` and +``[name, [args]]``. The name is looked up in the shared executor's registry (or +in the ``registry=`` given to the pipeline) and the command is called directly, +so a failure raises into the task. In the arguments, at any depth: + +* ``${params.}`` is replaced by the parameter. Inside longer text it is + inserted as text; a string that is exactly one placeholder becomes the + parameter itself, so ``"${params.limit}"`` stays the number ``20``. +* A string that is exactly ``${tasks..result}`` is replaced by the result + object of that upstream task (``None`` if it did not succeed). The task must + be upstream, and the placeholder cannot be part of longer text. + +There is no expression language and nothing is evaluated. Any other ``${...}`` +text is passed on unchanged. Because every string of the arguments is looked at, +a pipeline definition handed to ``FA_pipeline_run`` as an argument has its +placeholders filled in by the outer pipeline: pass a file path instead. + +Task IDs and parameter names are non-empty strings without whitespace, ``.``, +``$``, ``{`` or ``}``. + +Options +------- + +``Pipeline(name, description="", max_workers=4, *, params=None, schedule=None, registry=None)`` + +.. list-table:: + :header-rows: 1 + :widths: 22 78 + + * - Option + - Meaning + * - ``name`` + - Identifies the pipeline in events, in the history and in idempotency keys. + * - ``description`` + - Free text, kept in the definition. + * - ``max_workers`` + - How many tasks run at the same time. Default ``4``. + * - ``params`` + - Default parameters; ``run(params=...)`` adds to them and overrides them. + * - ``schedule`` + - A ``Schedule(cron, timezone=None)``. It is kept on ``pipeline.schedule`` + for the scheduler and not acted on by the pipeline. + * - ``registry`` + - Where action names are looked up. Default: the shared executor's registry. + +``pipeline.task(task_id, work, *, depends_on=None, retry=None, timeout=None, when="on_success", idempotency_key=None)`` + +.. list-table:: + :header-rows: 1 + :widths: 22 78 + + * - Option + - Meaning + * - ``task_id`` + - Unique in the pipeline. A second task with the same ID is rejected at once. + * - ``work`` + - The callable or the action. + * - ``depends_on`` + - IDs of the tasks that must end first. A task may be named before it is + added; the graph is checked when the pipeline runs. + * - ``retry`` + - A ``RetryPolicy``. Default: one attempt. See `Retry`_. + * - ``timeout`` + - Seconds for the whole task. Default: no limit. See `Timeout`_. + * - ``when`` + - ``"on_success"`` (default), ``"on_failure"``, ``"always"`` or a callable. + See `Conditions`_. + * - ``idempotency_key`` + - Text with ``${params.}`` placeholders. See `Idempotency`_. + +``RetryPolicy(max_attempts=1, backoff_base=0.0, backoff_cap=60.0, retry_on=(...))`` + +.. list-table:: + :header-rows: 1 + :widths: 22 78 + + * - Option + - Meaning + * - ``max_attempts`` + - Attempts in total; ``1`` means no retry. + * - ``backoff_base`` + - Seconds to wait after the first failed attempt; doubled after each + further one. + * - ``backoff_cap`` + - The longest wait. + * - ``retry_on`` + - The exception classes worth another attempt. Default: + ``StorageTransientException``, ``ConnectionError``, ``TimeoutError``. + +``pipeline.run(params=None, *, dry_run=False, store=None, cancel=None, bus=None)``, +``pipeline.start(params=None, *, store=None, cancel=None, bus=None)`` and +``pipeline.resume(run_id, *, store=None, cancel=None, bus=None)`` + +.. list-table:: + :header-rows: 1 + :widths: 22 78 + + * - Option + - Meaning + * - ``params`` + - Parameters of this run. + * - ``dry_run`` + - Plan without executing. See `Dry run`_. + * - ``store`` + - The ``RunStore`` that records the run. Default: the default store. + * - ``cancel`` + - A ``CancellationToken`` that stops the run when it is set. + * - ``bus`` + - The ``EventBus`` that receives the events. Default: the process-wide bus. + * - ``run_id`` + - For ``resume``: the run to continue. + +``run`` works in the calling thread and returns the finished +:class:`~automation_file.pipeline.model.PipelineRun`. ``start`` returns the run at +once and works on a background thread: ``run.wait(timeout)`` blocks until it has +ended (and returns whether it has), ``run.done`` tells without blocking, and +``run.cancel()`` stops it. The interpreter does not exit while a started run is +still going. + +Before anything runs, ``run``, ``start`` and ``resume`` raise +``PipelineDefinitionException`` for an empty pipeline, an unknown or repeated +dependency, a task that depends on itself, a cycle, a malformed placeholder, a +result placeholder that names a task which is not upstream, and a placeholder +for a parameter the run was not given. ``error.problems`` lists every finding +with its path; ``pipeline.problems()`` returns the same list without raising. + +Statuses +-------- + +``run.tasks[task_id].status`` is a ``TaskStatus``, ``run.status`` a +``RunStatus``. Both compare equal to their text. + +.. list-table:: + :header-rows: 1 + :widths: 18 82 + + * - Task status + - Meaning + * - ``pending`` + - Not started yet. + * - ``running`` + - An attempt is in progress, or the task waits between two attempts. + * - ``succeeded`` + - It returned; ``result`` holds the value. + * - ``failed`` + - It raised and no attempt is left; ``error`` is + ``": "``. + * - ``skipped`` + - It did not run; ``reason`` says why (below). + * - ``timeout`` + - It did not finish within its timeout. + * - ``cancelled`` + - The run was cancelled before it started or while it ran, or the task + raised ``CancelledException``. + * - ``planned`` + - A dry run: it would be considered in this order. + +.. list-table:: + :header-rows: 1 + :widths: 24 76 + + * - ``reason`` of a skip + - Meaning + * - ``idempotent`` + - Its idempotency key already has a succeeded execution; the stored result + is reused and its dependents run. + * - ``condition`` + - Its own ``on_failure`` or callable condition was not met. + * - ``upstream_failed`` + - An ``on_success`` task one of whose dependencies failed, timed out or was + cancelled. + * - ``upstream_skipped`` + - An ``on_success`` task one of whose dependencies was skipped. + +.. list-table:: + :header-rows: 1 + :widths: 18 82 + + * - Run status + - Meaning + * - ``running`` + - Not ended yet. + * - ``succeeded`` + - Every task succeeded or was skipped. + * - ``failed`` + - At least one task failed, timed out or was cancelled; ``run.error`` names + them. A clean-up task that succeeds does not change this. + * - ``cancelled`` + - The run was cancelled and at least one task did not run because of it. + +A task state also has ``attempts``, ``started_at`` and ``finished_at`` (UTC), +``duration_ms``, ``level`` (0 for a task without dependencies) and +``idempotency_key`` (the key with its placeholders filled in). ``run.tasks`` is +in dependency order, and ``run.to_dict()`` is JSON-serialisable. + +Conditions +---------- + +``when`` is looked at once, when every dependency of the task has ended. + +``"on_success"`` + Every dependency succeeded (or was skipped as ``idempotent``). Otherwise the + task is skipped, and that reaches its own ``on_success`` dependents. + +``"on_failure"`` + At least one dependency failed, timed out or was cancelled. For clean-up. + A dependency that was merely skipped does not count. + +``"always"`` + Whatever happened to the dependencies. + +A callable ``(TaskContext) -> bool`` + The task runs when it returns true. It decides alone: the outcome of the + dependencies is not consulted, but ``ctx.results`` holds only those that + succeeded. If the callable raises, the task is ``failed``. + +Retry +----- + +After a failed attempt the task is tried again when attempts are left and the +exception is an instance of one of ``retry_on``. The wait before attempt ``n + 1`` +is ``backoff_base * 2 ** (n - 1)`` seconds, at most ``backoff_cap``. + +The default ``retry_on`` holds the transient kind only. A ``ValueError`` or a +``KeyError`` is a bug or a wrong input and fails on its first attempt. Widen +``retry_on`` to the errors you know to be transient, never to ``Exception``. + +Timeout +------- + +``timeout`` is the budget in seconds for the whole task: every attempt and the +waits between them. When it is spent, the task is recorded as ``timeout``, its +cancellation token is set, and the run goes on with the other tasks. + +**A thread cannot be killed.** The callable keeps running until it returns, and +whatever it returns or raises after the timeout is ignored. A long callable must +therefore look at its token: + +.. code-block:: python + + def export(ctx): + for chunk in chunks(): + ctx.cancel.raise_if_cancelled() # raises CancelledException + write(chunk) + +An action cannot look at a token; give long transfers a timeout of their own +where the action has one. A task that timed out no longer counts towards +``max_workers``. Task threads are daemon threads, so one that never returns does +not keep the interpreter alive. + +Cancellation +------------ + +``run.cancel()``, or setting the ``CancellationToken`` passed as ``cancel=``, +stops a run within a few hundredths of a second: + +* every task that has not started becomes ``cancelled``, clean-up tasks included; +* every running task has its token set. The run waits for these tasks, and each + keeps the outcome it really had: ``cancelled`` if it raised + ``CancelledException``, ``succeeded`` if it finished anyway; +* a task waiting between two attempts stops waiting and becomes ``cancelled``. + +.. code-block:: python + + run = pipeline.start(params={"date": "2026-10-08"}) + ... + run.cancel() + run.wait(30) + run.status # "cancelled" + +Idempotency +----------- + +A task with an ``idempotency_key`` is not executed again once it has succeeded +under that key. Before the task starts, the key is rendered with the run's +parameters and looked up in the store for the same pipeline name and task ID. +When a succeeded execution is found, the task is ``skipped`` with the reason +``idempotent``, its ``result`` is the stored one, and its dependents run as if it +had succeeded. + +The key is only as durable as the store: with the in-memory default it lasts +until the process ends, with a ``SQLiteRunStore`` it survives a restart. It is +not a lock: two runs that start at the same moment can both find nothing and +both execute the task. If the store cannot be read, the task fails instead of +running. + +Checkpoint and resume +--------------------- + +Every transition of a task is written to the store as it happens: the start of +each attempt, each failed attempt, and the outcome. ``pipeline.resume(run_id)`` +loads the run, keeps the tasks that ``succeeded`` together with their results, +and runs the others again under the same run ID and the same parameters. + +.. code-block:: python + + store = SQLiteRunStore("pipelines.db") + run = pipeline.run(params={"date": "2026-10-08"}, store=store) + # ... the process may end here ... + pipeline = build_pipeline() # the same tasks + run = pipeline.resume(run.run_id, store=SQLiteRunStore("pipelines.db")) + +The store holds the state of a run, not the pipeline: ``resume`` is called on a +pipeline with the same name. A result is stored as JSON; a value JSON cannot +hold is stored as its ``repr`` and ``result_is_repr`` is set, so after a resume +the downstream tasks see that text. Let a task whose result others need return +JSON data. A run that already succeeded is returned as stored. Do not resume a +run that is still executing somewhere else. + +A kept task is not run again, so what it produced must still be there. A +clean-up task should remove what the failed task left behind, not the output of +a task that succeeded and that a later task still needs. + +Dry run +------- + +``pipeline.run(dry_run=True)`` executes nothing, records nothing and publishes +nothing. Every task comes back ``planned``, in dependency order with its +``level``. An action name the registry does not know and a placeholder for a +missing parameter are reported in the task's ``error``; the run is then +``failed``, otherwise ``succeeded``. A broken graph still raises. + +.. code-block:: python + + plan = pipeline.run(params={"date": "2026-10-08"}, dry_run=True) + for task_id, state in plan.tasks.items(): + print(state.level, task_id, state.error or "ok") + +Run stores and history +---------------------- + +.. list-table:: + :header-rows: 1 + :widths: 30 70 + + * - Store + - Keeps runs + * - ``MemoryRunStore(max_runs=1000)`` + - In memory, for the life of the process; the oldest runs are dropped + first. This is the default store. + * - ``SQLiteRunStore(path)`` + - In a SQLite file. Thread-safe, parameterised statements only, one commit + per transition, and a schema version in the file. + +.. code-block:: python + + from automation_file.pipeline import SQLiteRunStore, set_default_run_store + + store = SQLiteRunStore("/var/lib/automation/pipelines.db") + set_default_run_store(store) # for run(), resume() and the actions + + store.get_run(run_id) # a PipelineRun, or None + store.list_runs("daily-report", limit=10) # newest first + store.find_idempotent("daily-report", "publish", "publish-2026-10-08") + +A store of your own subclasses ``RunStore`` and implements ``save_run``, +``save_task``, ``get_run``, ``list_runs`` and ``find_idempotent``; it raises +``PipelineException`` when it cannot read or write. Parameters and results are +stored as given: keep passwords and tokens out of both. + +Definitions +----------- + +A definition is a mapping with ``schema_version: 1``, from YAML, JSON or Python. + +.. list-table:: + :header-rows: 1 + :widths: 30 70 + + * - Key + - Value + * - ``schema_version`` + - ``1``. Required; a document without it or with another value is rejected. + * - ``name`` + - Required. + * - ``description``, ``max_workers``, ``params`` + - As the options of ``Pipeline``. + * - ``schedule`` + - ``{cron: "0 2 * * *", timezone: Asia/Taipei}``. ``cron`` has five fields; + ``timezone`` is optional. + * - ``tasks`` + - Required. Task ID to task. + * - ``tasks..action`` + - Required. ``[name]``, ``[name, {kwargs}]`` or ``[name, [args]]``. + * - ``tasks..depends_on`` + - A list of task IDs. + * - ``tasks..retry`` + - ``{max_attempts, backoff, backoff_cap, on}``. ``on`` lists exception + names: the classes of ``automation_file.exceptions``, ``TimeoutError``, + ``ConnectionError`` and ``OSError``. Any other name is an error. + * - ``tasks..timeout`` + - Seconds, above 0. + * - ``tasks..when`` + - ``on_success``, ``on_failure`` or ``always``. + * - ``tasks..idempotency_key`` + - Text with ``${params.}`` placeholders. + +.. code-block:: python + + from automation_file.pipeline import PIPELINE_SCHEMA, Pipeline, validate_definition + + validate_definition(document) # [] when valid, else every problem with its path + # ["max_workers: expected an integer >= 1, got 0", + # "tasks.verify.depends_on[0]: unknown task 'x'"] + + pipeline = Pipeline.from_dict(document) # raises PipelineDefinitionException + pipeline = Pipeline.from_file("daily-report.yaml") # .yaml, .yml or .json + pipeline.to_dict() # the document, defaults left out + PIPELINE_SCHEMA # JSON Schema (draft 2020-12), a dict + +An unknown key is an error at every level. ``to_dict`` raises for a pipeline +that holds a Python callable, which a document cannot express. + +YAML is read with ``yaml.safe_load``. Three things YAML does are handled: + +* A key repeated in one mapping is an error (in JSON too), so a second task with + the same ID cannot silently replace the first. +* YAML 1.1 reads a bare ``on`` as ``true``. ``from_file`` gives the ``retry`` + key back as ``on``; if you parse the YAML yourself, write ``"on"``. +* An unquoted date such as ``2026-10-08`` becomes a date object, which a + definition cannot hold. Validation points at it: quote dates and times. + +Events +------ + +Every event has ``source="pipeline"`` and the run ID as its ``correlation_id``, +also when it is published from a task's thread. Events published inside a task, +storage errors for instance, carry the same correlation ID and the actor of the +code that started the run. + +.. list-table:: + :header-rows: 1 + :widths: 24 16 60 + + * - Event + - Severity + - Published + * - ``pipeline.started`` + - info + - Once, before the first task. Again when a run is resumed. + * - ``task.started`` + - info + - At the start of every attempt. + * - ``task.completed`` + - info + - When an attempt succeeds. + * - ``task.failed`` + - warning / error + - When an attempt fails. ``status`` is ``retrying`` (warning: another + attempt follows), ``failed``, ``timeout`` or ``cancelled`` (warning). + * - ``pipeline.completed`` + - info + - When the run ends ``succeeded``. + * - ``pipeline.failed`` + - error / warning + - When the run ends ``failed``, or ``cancelled`` (warning). + +The payload uses the shared keys: ``pipeline``, ``run_id``, ``status``, and for +task events ``task`` and ``attempt``; ``duration_ms`` when something ended and +``error`` when it went wrong. A task that is skipped, or cancelled before it +started, publishes nothing: its state is in the run. A task whose ``when`` +callable raised, or whose idempotency key could not be looked up, publishes one +``task.failed`` with ``attempt`` 0. A dry run publishes nothing. + +.. code-block:: python + + from automation_file import Severity, event_bus + + def alert(event): + print(event.subject, event.payload.get("error")) + + event_bus.subscribe(alert, types=["pipeline.failed", "task.failed"], + min_severity=Severity.ERROR) + +Actions +------- + +Pipelines are also reachable from JSON action lists, and so from the CLI, the +TCP and HTTP action servers and MCP hosts. ``definition`` is a mapping or the +path of a ``.yaml`` / ``.yml`` / ``.json`` file. + +.. list-table:: + :header-rows: 1 + :widths: 26 40 34 + + * - Action + - Parameters + - Returns + * - ``FA_pipeline_run`` + - ``definition, params=None, dry_run=False`` + - The run (``PipelineRun.to_dict()``) + * - ``FA_pipeline_validate`` + - ``definition`` + - ``{"valid": …, "errors": […]}`` + * - ``FA_pipeline_status`` + - ``run_id`` + - The recorded run + * - ``FA_pipeline_history`` + - ``pipeline=None, limit=20`` + - Recorded runs, newest first + * - ``FA_pipeline_resume`` + - ``run_id, definition`` + - The run after it was continued + +.. code-block:: json + + [ + ["FA_pipeline_validate", {"definition": "pipelines/daily-report.yaml"}], + ["FA_pipeline_run", {"definition": "pipelines/daily-report.yaml", + "params": {"date": "2026-10-08"}}], + ["FA_pipeline_history", {"pipeline": "daily-report", "limit": 5}] + ] + +The actions use the default run store, so ``FA_pipeline_status``, +``FA_pipeline_history`` and ``FA_pipeline_resume`` see the runs of the same +process unless ``set_default_run_store`` was given a ``SQLiteRunStore``. +``FA_pipeline_run`` returns a run whose ``status`` is ``failed`` when a task +failed; it raises only for a definition that cannot be loaded or is invalid. +``register_pipeline_ops(registry)`` adds the actions to a registry of your own. + +A definition names the actions its tasks call. As long as the definition is part +of the request, an :class:`~automation_file.ActionACL` on a TCP or HTTP action +server checks those names too, and so does ``--allowed-actions`` on the MCP +server. Neither can see inside a definition given as a file path, nor inside the +stored run ``FA_pipeline_resume`` continues: allow ``FA_pipeline_run`` and +``FA_pipeline_resume`` only for clients that may call every registered action, +or keep the definition files where those clients cannot write. + +When something goes wrong +------------------------- + +A task failed + ``run.status`` is ``failed``, ``run.error`` names the tasks, and each state + has ``error``, ``attempts`` and its times. Fix the cause and call + ``resume(run_id)``: what succeeded is not repeated. + +A task did not run + Look at ``reason``. ``upstream_failed`` and ``upstream_skipped`` point at a + dependency; give a task that must run anyway ``when="always"``. + +A task succeeds although the work went wrong + A task fails only by raising. An action that reports through its return + value succeeds as a task: ``FA_storage_verify`` returns ``false`` on a + mismatch unless it is given ``strict: true``, which makes it raise + ``StorageChecksumException`` and so fail the task. Check any other such + value in a callable that raises, or in a callable ``when`` of the next task. + +A task is never retried + Its exception is not in ``retry_on``. The default covers + ``StorageTransientException``, ``ConnectionError`` and ``TimeoutError`` only. + +A task is ``timeout`` but still seems to work + Its thread could not be stopped. Make the callable watch ``ctx.cancel``, and + make sure a second run cannot collide with what is left of the first. + +A run was interrupted (the process died, Ctrl-C) + The store has every transition up to that point, with tasks possibly left + ``running``. ``resume(run_id)`` runs everything that had not succeeded. + +``PipelineDefinitionException`` before anything ran + The definition is wrong. ``error.problems`` holds every finding with its + path; ``validate_definition`` and a dry run give them without running. + +The history is incomplete + A store that cannot write is logged as an error and the run goes on. The + run itself is right; its record is not. + +Both exceptions, ``PipelineException`` and its subclass +``PipelineDefinitionException``, derive from ``FileAutomationException``. diff --git a/docs/source/Eng/usage/storage.rst b/docs/source/Eng/usage/storage.rst index b39e51d..9e38a03 100644 --- a/docs/source/Eng/usage/storage.rst +++ b/docs/source/Eng/usage/storage.rst @@ -189,8 +189,9 @@ strings and returns JSON-friendly values. - ``uri, algorithm="sha256"`` - ``{"algorithm": …, "value": …}`` * - ``FA_storage_verify`` - - ``uri, expected, algorithm="sha256"`` - - ``true`` / ``false`` + - ``uri, expected, algorithm="sha256", strict=False`` + - ``true`` / ``false``; with ``strict=True`` a mismatch raises + ``StorageChecksumException`` * - ``FA_storage_copy`` - ``source, target, overwrite=True`` - The target's file information @@ -267,6 +268,9 @@ All derive from :class:`~automation_file.StorageException`, itself a * - ``StorageUnsupportedException`` - The backend cannot do what was asked (an unknown checksum algorithm, deleting the root). + * - ``StorageChecksumException`` + - A strict verification (``FA_storage_verify`` with ``strict=True``) found + another digest than the expected one. Built-in backends ----------------- diff --git a/docs/source/Zh-CN/usage/pipeline.rst b/docs/source/Zh-CN/usage/pipeline.rst new file mode 100644 index 0000000..ddd8fff --- /dev/null +++ b/docs/source/Zh-CN/usage/pipeline.rst @@ -0,0 +1,696 @@ +流水线（Pipeline） +====================== + +``automation_file.pipeline`` 按依赖顺序执行一组任务，互不依赖的任务并行执行，并补上 +周期性作业需要的功能：重试、超时、取消、条件、幂等键、检查点与续跑、试运行，以及运行 +历史。 + +任务可以是 Python 可调用对象，也可以是 ``FA_*`` 动作。流水线可以用 Python 构建，也可以 +写成 YAML / JSON 文档。一次运行只通过事件总线上的事件报告（见 :doc:`event_bus`）； +它不会调用任何通知 sink，也不会写入审计记录。 + +:func:`~automation_file.execute_action_dag`\ （见 :doc:`dag`）保持不变。它把一份动作 +列表执行一次并返回结果；当运行需要被记录、重试、续跑或观察时，请改用流水线。 + +最小示例 +---------------- + +.. code-block:: python + + from automation_file.pipeline import Pipeline + + def count_rows(ctx): + return len(ctx.results["read"].splitlines()) + + pipeline = Pipeline("row-count") + pipeline.task("read", ["FA_storage_read_text", {"uri": "local:///data/report.csv"}]) + pipeline.task("count", count_rows, depends_on=["read"]) + + run = pipeline.run() + run.status # RunStatus.SUCCEEDED；等于 "succeeded" + run.tasks["count"].result # 42 + run.tasks["count"].attempts # 1 + +``run()`` 在所有任务都结束后才返回。任务失败不会抛出异常：结果记在 ``run.status`` +以及 ``run.tasks`` 的每一项上。 + +生产环境示例 +------------------------ + +一个每晚运行的作业：带重试与超时地获取文件、检查内容、同一个日期最多发布一次、发布 +失败时撤回发布到一半的报表，并把运行记录存进 SQLite 文件，让失败的运行可以续跑。 + +.. code-block:: python + + from automation_file.pipeline import Pipeline, RetryPolicy, SQLiteRunStore + + WORK = "local:///var/tmp/report-${params.date}.csv" + TARGET = "azure://reports/${params.date}.csv" + store = SQLiteRunStore("/var/lib/automation/pipelines.db") + + def check(ctx): + ctx.cancel.raise_if_cancelled() # 被取消或超时就停下来 + if ctx.results["download"]["size"] == 0: + raise ValueError("the report is empty") # 任务通过抛出异常表示失败 + return {"bytes": ctx.results["download"]["size"]} + + pipeline = Pipeline("daily-report", description="Fetch, check, publish", max_workers=4) + pipeline.task( + "download", + ["FA_storage_copy", {"source": "s3://input/${params.date}.csv", "target": WORK}], + retry=RetryPolicy(max_attempts=5, backoff_base=2.0, backoff_cap=60.0), + timeout=300.0, + ) + pipeline.task("check", check, depends_on=["download"], timeout=60.0) + pipeline.task( + "publish", + ["FA_storage_copy", {"source": WORK, "target": TARGET}], + depends_on=["check"], + retry=RetryPolicy(max_attempts=3, backoff_base=1.0), + idempotency_key="publish-${params.date}", # 同一个日期绝不发布两次 + ) + pipeline.task( + "withdraw", # 清理：只在 publish 失败时执行 + ["FA_storage_delete", {"uri": TARGET, "missing_ok": True}], + depends_on=["publish"], + when="on_failure", + ) + pipeline.task("tidy", ["FA_storage_delete", {"uri": WORK}], depends_on=["publish"]) + + run = pipeline.run(params={"date": "2026-10-08"}, store=store) + if run.status != "succeeded": + for task_id, state in run.tasks.items(): + print(task_id, state.status.value, state.error or state.reason or "") + + # 稍后，在同一个或另一个进程中，等原因排除之后： + run = pipeline.resume(run.run_id, store=store) # 只执行尚未成功的部分 + +``publish`` 三次尝试都失败时，``withdraw`` 会执行，``tidy`` 被跳过，这次运行的状态为 +``failed``。``resume`` 保留 ``download`` 与 ``check``，重新执行 ``publish``，接着执行 +``tidy``。之后同一个日期的另一次运行会重新下载与检查，但跳过 ``publish``：它的键已经 +成功过。 + +同一条流水线不写 Python 的版本，保存为 ``daily-report.yaml``。``check`` 是可调用对象， +无法写进文档，所以这个版本直接发布下载到的内容： + +.. code-block:: yaml + + schema_version: 1 + name: daily-report + description: Fetch and publish the daily report + max_workers: 4 + schedule: {cron: "0 2 * * *", timezone: Asia/Taipei} + params: {date: "2026-10-08"} + tasks: + download: + action: ["FA_storage_copy", {"source": "s3://input/${params.date}.csv", + "target": "local:///var/tmp/report-${params.date}.csv"}] + retry: {max_attempts: 5, backoff: 2, backoff_cap: 60, + on: [StorageTransientException, ConnectionError]} + timeout: 300 + publish: + action: ["FA_storage_copy", {"source": "local:///var/tmp/report-${params.date}.csv", + "target": "azure://reports/${params.date}.csv"}] + depends_on: [download] + retry: {max_attempts: 3, backoff: 1} + idempotency_key: "publish-${params.date}" + withdraw: + action: ["FA_storage_delete", {"uri": "azure://reports/${params.date}.csv", + "missing_ok": true}] + depends_on: [publish] + when: on_failure + tidy: + action: ["FA_storage_delete", {"uri": "local:///var/tmp/report-${params.date}.csv"}] + depends_on: [publish] + +.. code-block:: python + + pipeline = Pipeline.from_file("daily-report.yaml") + run = pipeline.run(params={"date": "2026-10-09"}, store=store) + +任务 +-------- + +任务要做的事有两种写法。 + +**可调用对象**：接收一个 :class:`~automation_file.pipeline.model.TaskContext`。它的 +返回值就是任务的结果；抛出异常则任务失败。 + +.. list-table:: + :header-rows: 1 + :widths: 20 80 + + * - 字段 + - 含义 + * - ``pipeline`` + - 流水线的名称。 + * - ``run_id`` + - 这次运行的 ID，同时也是每个事件的关联 ID。 + * - ``task`` + - 任务的 ID。 + * - ``attempt`` + - 第几次尝试，从 1 开始（在 ``when`` 可调用对象中为 ``0``）。 + * - ``params`` + - 这次运行的参数，只读：流水线的默认值，再由 ``run(params=...)`` 覆盖。 + * - ``results`` + - 任务 ID 到结果的映射，只读，涵盖所有成功的上游任务，不论是否直接依赖。失败 + 或被跳过的任务不会出现在其中。 + * - ``cancel`` + - ``CancellationToken``。运行被取消或任务超时后会被置位。见 `超时`_。 + * - ``dry_run`` + - 永远是 ``False``：试运行不会执行任何东西。 + +**动作**：三种形式之一，``[name]``、``[name, {kwargs}]`` 与 ``[name, [args]]``。名称 +会在共享执行器的注册表（或传给流水线的 ``registry=``）中查找，并直接调用该命令，因此 +失败时异常会抛进任务。在调用参数中，不论嵌套多深： + +* ``${params.}`` 会被替换为该参数。夹在较长的文本中时以文本插入；整个字符串 + 恰好就是一个占位符时，会替换为参数本身，所以 ``"${params.limit}"`` 仍然是数字 ``20``。 +* 整个字符串恰好是 ``${tasks..result}`` 时，会替换为该上游任务的结果对象（如果它 + 没有成功则为 ``None``）。该任务必须是上游任务，而且这个占位符不能夹在较长的文本中。 + +这里没有表达式语言，也不会对任何内容求值。其他的 ``${...}`` 文本会原样传递。由于 +调用参数中的每个字符串都会被检查，把流水线定义作为参数交给 ``FA_pipeline_run`` 时， +其中的占位符会被外层流水线填入：请改为传入文件路径。 + +任务 ID 与参数名称是非空字符串，不能包含空白、``.``、``$``、``{`` 或 ``}``。 + +选项 +-------- + +``Pipeline(name, description="", max_workers=4, *, params=None, schedule=None, registry=None)`` + +.. list-table:: + :header-rows: 1 + :widths: 22 78 + + * - 选项 + - 含义 + * - ``name`` + - 在事件、历史与幂等键中用来标识这条流水线。 + * - ``description`` + - 自由文本，保存在定义中。 + * - ``max_workers`` + - 同时执行的任务数量上限。默认为 ``4``。 + * - ``params`` + - 默认参数；``run(params=...)`` 会加入并覆盖它们。 + * - ``schedule`` + - ``Schedule(cron, timezone=None)``。保存在 ``pipeline.schedule`` 供调度器 + 使用，流水线本身不会据此行动。 + * - ``registry`` + - 查找动作名称的地方。默认：共享执行器的注册表。 + +``pipeline.task(task_id, work, *, depends_on=None, retry=None, timeout=None, when="on_success", idempotency_key=None)`` + +.. list-table:: + :header-rows: 1 + :widths: 22 78 + + * - 选项 + - 含义 + * - ``task_id`` + - 在流水线中必须唯一。加入第二个相同 ID 的任务会立刻被拒绝。 + * - ``work`` + - 可调用对象或动作。 + * - ``depends_on`` + - 必须先结束的任务 ID。可以先写出尚未加入的任务；依赖图在流水线运行时才 + 检查。 + * - ``retry`` + - ``RetryPolicy``。默认：只尝试一次。见 `重试`_。 + * - ``timeout`` + - 整个任务可用的秒数。默认：没有限制。见 `超时`_。 + * - ``when`` + - ``"on_success"``\ （默认）、``"on_failure"``、``"always"`` 或可调用对象。 + 见 `条件`_。 + * - ``idempotency_key`` + - 可含 ``${params.}`` 占位符的文本。见 `幂等`_。 + +``RetryPolicy(max_attempts=1, backoff_base=0.0, backoff_cap=60.0, retry_on=(...))`` + +.. list-table:: + :header-rows: 1 + :widths: 22 78 + + * - 选项 + - 含义 + * - ``max_attempts`` + - 总共尝试几次；``1`` 表示不重试。 + * - ``backoff_base`` + - 第一次尝试失败后等待的秒数；之后每失败一次就加倍。 + * - ``backoff_cap`` + - 等待时间的上限。 + * - ``retry_on`` + - 值得再试一次的异常类。默认：``StorageTransientException``、 + ``ConnectionError``、``TimeoutError``。 + +``pipeline.run(params=None, *, dry_run=False, store=None, cancel=None, bus=None)``、 +``pipeline.start(params=None, *, store=None, cancel=None, bus=None)`` 与 +``pipeline.resume(run_id, *, store=None, cancel=None, bus=None)`` + +.. list-table:: + :header-rows: 1 + :widths: 22 78 + + * - 选项 + - 含义 + * - ``params`` + - 这次运行的参数。 + * - ``dry_run`` + - 只规划而不执行。见 `试运行`_。 + * - ``store`` + - 记录这次运行的 ``RunStore``。默认：默认的存储。 + * - ``cancel`` + - ``CancellationToken``，被置位时会停止这次运行。 + * - ``bus`` + - 接收事件的 ``EventBus``。默认：整个进程共用的总线。 + * - ``run_id`` + - 用于 ``resume``：要接续的那次运行。 + +``run`` 在调用它的线程中工作，并返回已结束的 +:class:`~automation_file.pipeline.model.PipelineRun`。``start`` 立刻返回运行对象， +并在后台线程中工作：``run.wait(timeout)`` 会等到它结束（并返回是否已结束）， +``run.done`` 不等待就能得知，``run.cancel()`` 则会停止它。已启动的运行还没结束时， +解释器不会退出。 + +在任何任务开始之前，``run``、``start`` 与 ``resume`` 会对以下情况抛出 +``PipelineDefinitionException``：流水线是空的、依赖的任务不存在或重复、任务依赖于 +自己、依赖图有环、占位符格式错误、结果占位符指向不是上游的任务，以及占位符用到 +这次运行没有提供的参数。``error.problems`` 列出每一项问题及其路径； +``pipeline.problems()`` 返回同一份列表但不抛出异常。 + +状态 +-------- + +``run.tasks[task_id].status`` 是 ``TaskStatus``，``run.status`` 是 ``RunStatus``。 +两者都可以直接与它们的文本比较。 + +.. list-table:: + :header-rows: 1 + :widths: 18 82 + + * - 任务状态 + - 含义 + * - ``pending`` + - 尚未开始。 + * - ``running`` + - 正在进行某一次尝试，或正在两次尝试之间等待。 + * - ``succeeded`` + - 已返回；``result`` 保存返回值。 + * - ``failed`` + - 抛出了异常，而且没有剩余的尝试次数；``error`` 的形式为 + ``": "``。 + * - ``skipped`` + - 没有执行；``reason`` 说明原因（见下表）。 + * - ``timeout`` + - 没有在超时时间内完成。 + * - ``cancelled`` + - 运行在它开始之前或执行期间被取消，或任务抛出了 ``CancelledException``。 + * - ``planned`` + - 试运行：它会按这个顺序被考虑。 + +.. list-table:: + :header-rows: 1 + :widths: 24 76 + + * - 跳过的 ``reason`` + - 含义 + * - ``idempotent`` + - 它的幂等键已经有一次成功的执行；沿用存储的结果，依赖于它的任务照常执行。 + * - ``condition`` + - 它自己的 ``on_failure`` 条件或可调用条件没有成立。 + * - ``upstream_failed`` + - ``on_success`` 任务，而它的某个依赖任务失败、超时或被取消。 + * - ``upstream_skipped`` + - ``on_success`` 任务，而它的某个依赖任务被跳过。 + +.. list-table:: + :header-rows: 1 + :widths: 18 82 + + * - 运行状态 + - 含义 + * - ``running`` + - 尚未结束。 + * - ``succeeded`` + - 每个任务都成功或被跳过。 + * - ``failed`` + - 至少有一个任务失败、超时或被取消；``run.error`` 会列出它们。清理任务成功 + 并不会改变这个结果。 + * - ``cancelled`` + - 运行被取消，而且至少有一个任务因此没有执行。 + +任务状态还有 ``attempts``、``started_at`` 与 ``finished_at``\ （UTC）、 +``duration_ms``、``level``\ （没有依赖的任务为 0）以及 ``idempotency_key``\ （已填入 +占位符的键）。``run.tasks`` 按依赖顺序排列，``run.to_dict()`` 可以序列化为 JSON。 + +条件 +-------- + +``when`` 只在任务的所有依赖任务都结束时检查一次。 + +``"on_success"`` + 每个依赖任务都成功（或以 ``idempotent`` 被跳过）。否则任务被跳过，而且这会 + 传到它自己的 ``on_success`` 下游任务。 + +``"on_failure"`` + 至少有一个依赖任务失败、超时或被取消。用于清理。只是被跳过的依赖任务不算。 + +``"always"`` + 不论依赖任务的结果如何。 + +可调用对象 ``(TaskContext) -> bool`` + 返回真值时任务才执行。它独自决定：不会参考依赖任务的结果，但 ``ctx.results`` + 只包含成功的那些。可调用对象抛出异常时，任务为 ``failed``。 + +重试 +-------- + +一次尝试失败后，如果还有剩余次数，而且异常是 ``retry_on`` 中某个类的实例，任务就会 +再试一次。第 ``n + 1`` 次尝试之前等待 ``backoff_base * 2 ** (n - 1)`` 秒，最多 +``backoff_cap`` 秒。 + +默认的 ``retry_on`` 只包含暂时性的错误。``ValueError`` 或 ``KeyError`` 意味着程序错误 +或输入有误，第一次尝试就会失败。请把 ``retry_on`` 放宽到你确知是暂时性的错误，绝对 +不要放宽到 ``Exception``。 + +超时 +-------- + +``timeout`` 是整个任务可用的秒数：包含每一次尝试以及尝试之间的等待。用完之后，任务 +被记录为 ``timeout``，它的取消令牌被置位，其余任务继续执行。 + +**线程无法被强制终止。**\ 可调用对象会继续运行直到它自己返回，而它在超时之后返回 +或抛出的任何东西都会被忽略。因此运行时间长的可调用对象必须检查自己的令牌： + +.. code-block:: python + + def export(ctx): + for chunk in chunks(): + ctx.cancel.raise_if_cancelled() # 抛出 CancelledException + write(chunk) + +动作无法检查令牌；动作本身若有超时参数，请为长时间的传输设置它。超时的任务不再 +计入 ``max_workers``。任务线程是 daemon 线程，所以永不返回的线程不会让解释器无法 +退出。 + +取消 +-------- + +``run.cancel()``，或置位以 ``cancel=`` 传入的 ``CancellationToken``，会在百分之几秒内 +停止一次运行： + +* 所有尚未开始的任务变成 ``cancelled``，清理任务也一样； +* 所有执行中任务的令牌被置位。运行会等待这些任务，而每个任务保留它实际的结果： + 抛出 ``CancelledException`` 的为 ``cancelled``，照样完成的为 ``succeeded``； +* 正在两次尝试之间等待的任务会停止等待，变成 ``cancelled``。 + +.. code-block:: python + + run = pipeline.start(params={"date": "2026-10-08"}) + ... + run.cancel() + run.wait(30) + run.status # "cancelled" + +幂等 +-------- + +带有 ``idempotency_key`` 的任务，一旦以该键成功过，就不会再次执行。任务开始之前， +会用这次运行的参数算出键，并在存储中查找相同流水线名称与任务 ID 的记录。找到成功的 +执行时，任务为 ``skipped``，原因是 ``idempotent``，它的 ``result`` 是存储的那一份， +依赖于它的任务会像它成功了一样照常执行。 + +键的持久程度取决于存储：使用默认的内存存储时，只维持到进程结束；使用 +``SQLiteRunStore`` 时，重新启动后仍然有效。它不是锁：同时开始的两次运行可能都查不到 +记录，于是都执行该任务。如果存储无法读取，任务会失败而不是照常执行。 + +检查点与续跑 +------------------------ + +任务的每一次状态转换都会在发生时写入存储：每次尝试的开始、每次失败的尝试，以及 +最后的结果。``pipeline.resume(run_id)`` 载入该次运行，保留 ``succeeded`` 的任务及其 +结果，并以相同的运行 ID 与相同的参数重新执行其余任务。 + +.. code-block:: python + + store = SQLiteRunStore("pipelines.db") + run = pipeline.run(params={"date": "2026-10-08"}, store=store) + # ... 进程可能在这里结束 ... + pipeline = build_pipeline() # 相同的任务 + run = pipeline.resume(run.run_id, store=SQLiteRunStore("pipelines.db")) + +存储保存的是一次运行的状态，而不是流水线本身：``resume`` 要在名称相同的流水线上 +调用。结果以 JSON 存储；JSON 无法表示的值会以它的 ``repr`` 存储，并设置 +``result_is_repr``，所以续跑之后下游任务看到的是那段文本。其他任务需要用到的结果， +请让任务返回 JSON 数据。已经成功的运行会按存储的内容原样返回。不要续跑仍在别处 +执行中的运行。 + +被保留的任务不会再执行一次，所以它产生的东西必须还在。清理任务应该移除失败任务 +留下的东西，而不是某个已成功、后续任务还需要的任务的输出。 + +试运行 +------------ + +``pipeline.run(dry_run=True)`` 不执行任何东西、不记录任何东西，也不发布任何事件。 +每个任务都以 ``planned`` 返回，按依赖顺序排列并带有 ``level``。注册表中找不到的动作 +名称，以及用到缺少参数的占位符，会报告在任务的 ``error`` 中；这时运行状态为 +``failed``，否则为 ``succeeded``。依赖图有问题时仍然会抛出异常。 + +.. code-block:: python + + plan = pipeline.run(params={"date": "2026-10-08"}, dry_run=True) + for task_id, state in plan.tasks.items(): + print(state.level, task_id, state.error or "ok") + +运行记录存储与历史 +------------------------------------ + +.. list-table:: + :header-rows: 1 + :widths: 30 70 + + * - 存储 + - 运行记录保存在 + * - ``MemoryRunStore(max_runs=1000)`` + - 内存中，维持到进程结束；最旧的运行最先被丢弃。这是默认的存储。 + * - ``SQLiteRunStore(path)`` + - SQLite 文件中。线程安全、只使用参数化语句、每次转换提交一次，文件中带有 + 结构版本。 + +.. code-block:: python + + from automation_file.pipeline import SQLiteRunStore, set_default_run_store + + store = SQLiteRunStore("/var/lib/automation/pipelines.db") + set_default_run_store(store) # 供 run()、resume() 与动作使用 + + store.get_run(run_id) # PipelineRun，或 None + store.list_runs("daily-report", limit=10) # 最新的在前 + store.find_idempotent("daily-report", "publish", "publish-2026-10-08") + +自定义的存储要继承 ``RunStore``，并实现 ``save_run``、``save_task``、``get_run``、 +``list_runs`` 与 ``find_idempotent``；无法读取或写入时抛出 ``PipelineException``。 +参数与结果会原样存储：不要把密码与令牌放进其中任何一个。 + +定义文件 +---------------- + +定义是一份带有 ``schema_version: 1`` 的映射，来源可以是 YAML、JSON 或 Python。 + +.. list-table:: + :header-rows: 1 + :widths: 30 70 + + * - 键 + - 值 + * - ``schema_version`` + - ``1``。必填；没有这个键或值不同的文档会被拒绝。 + * - ``name`` + - 必填。 + * - ``description``、``max_workers``、``params`` + - 与 ``Pipeline`` 的选项相同。 + * - ``schedule`` + - ``{cron: "0 2 * * *", timezone: Asia/Taipei}``。``cron`` 有五个字段； + ``timezone`` 可省略。 + * - ``tasks`` + - 必填。任务 ID 到任务的映射。 + * - ``tasks..action`` + - 必填。``[name]``、``[name, {kwargs}]`` 或 ``[name, [args]]``。 + * - ``tasks..depends_on`` + - 任务 ID 的列表。 + * - ``tasks..retry`` + - ``{max_attempts, backoff, backoff_cap, on}``。``on`` 列出异常名称： + ``automation_file.exceptions`` 中的类、``TimeoutError``、 + ``ConnectionError`` 与 ``OSError``。其他名称都是错误。 + * - ``tasks..timeout`` + - 秒数，必须大于 0。 + * - ``tasks..when`` + - ``on_success``、``on_failure`` 或 ``always``。 + * - ``tasks..idempotency_key`` + - 可含 ``${params.}`` 占位符的文本。 + +.. code-block:: python + + from automation_file.pipeline import PIPELINE_SCHEMA, Pipeline, validate_definition + + validate_definition(document) # 有效时为 []，否则是每一项问题及其路径 + # ["max_workers: expected an integer >= 1, got 0", + # "tasks.verify.depends_on[0]: unknown task 'x'"] + + pipeline = Pipeline.from_dict(document) # 会抛出 PipelineDefinitionException + pipeline = Pipeline.from_file("daily-report.yaml") # .yaml、.yml 或 .json + pipeline.to_dict() # 定义文档，省略默认值 + PIPELINE_SCHEMA # JSON Schema（draft 2020-12），是一个 dict + +每一层出现未知的键都是错误。流水线中含有 Python 可调用对象时，``to_dict`` 会抛出 +异常，因为文档无法表示它。 + +YAML 以 ``yaml.safe_load`` 读取。YAML 的三种行为已经处理： + +* 同一个映射中重复的键是错误（JSON 也一样），所以第二个相同 ID 的任务不会悄悄地 + 取代第一个。 +* YAML 1.1 会把没有加引号的 ``on`` 读成 ``true``。``from_file`` 会把 ``retry`` 下面 + 的这个键还原为 ``on``；如果你自己解析 YAML，请写成 ``"on"``。 +* 没有加引号的日期（例如 ``2026-10-08``）会变成日期对象，定义无法保存它。验证会 + 指出这一点：请为日期与时间加上引号。 + +事件 +-------- + +每个事件的 ``source`` 都是 ``"pipeline"``，``correlation_id`` 都是运行 ID，从任务的 +线程发布时也一样。在任务内部发布的事件（例如存储错误）带有相同的关联 ID，以及 +启动这次运行的代码的 actor。 + +.. list-table:: + :header-rows: 1 + :widths: 24 16 60 + + * - 事件 + - 严重程度 + - 发布时机 + * - ``pipeline.started`` + - info + - 一次，在第一个任务之前。续跑时会再发布一次。 + * - ``task.started`` + - info + - 每次尝试开始时。 + * - ``task.completed`` + - info + - 某次尝试成功时。 + * - ``task.failed`` + - warning / error + - 某次尝试失败时。``status`` 为 ``retrying``\ （warning：接着还有一次 + 尝试）、``failed``、``timeout`` 或 ``cancelled``\ （warning）。 + * - ``pipeline.completed`` + - info + - 运行以 ``succeeded`` 结束时。 + * - ``pipeline.failed`` + - error / warning + - 运行以 ``failed`` 或 ``cancelled``\ （warning）结束时。 + +payload 使用共用的键：``pipeline``、``run_id``、``status``，任务事件另有 ``task`` 与 +``attempt``；某件事结束时有 ``duration_ms``，出错时有 ``error``。被跳过的任务，以及 +开始之前就被取消的任务，不会发布任何事件：它的状态记在运行上。``when`` 可调用对象 +抛出异常，或幂等键无法查找的任务，会发布一个 ``attempt`` 为 0 的 ``task.failed``。 +试运行不会发布任何事件。 + +.. code-block:: python + + from automation_file import Severity, event_bus + + def alert(event): + print(event.subject, event.payload.get("error")) + + event_bus.subscribe(alert, types=["pipeline.failed", "task.failed"], + min_severity=Severity.ERROR) + +动作 +-------- + +流水线也可以从 JSON 动作列表使用，因此 CLI、TCP 与 HTTP 动作服务器以及 MCP 主机都能 +调用。``definition`` 是一份映射，或 ``.yaml`` / ``.yml`` / ``.json`` 文件的路径。 + +.. list-table:: + :header-rows: 1 + :widths: 26 40 34 + + * - 动作 + - 参数 + - 返回值 + * - ``FA_pipeline_run`` + - ``definition, params=None, dry_run=False`` + - 这次运行（``PipelineRun.to_dict()``） + * - ``FA_pipeline_validate`` + - ``definition`` + - ``{"valid": …, "errors": […]}`` + * - ``FA_pipeline_status`` + - ``run_id`` + - 已记录的运行 + * - ``FA_pipeline_history`` + - ``pipeline=None, limit=20`` + - 已记录的运行，最新的在前 + * - ``FA_pipeline_resume`` + - ``run_id, definition`` + - 续跑之后的运行 + +.. code-block:: json + + [ + ["FA_pipeline_validate", {"definition": "pipelines/daily-report.yaml"}], + ["FA_pipeline_run", {"definition": "pipelines/daily-report.yaml", + "params": {"date": "2026-10-08"}}], + ["FA_pipeline_history", {"pipeline": "daily-report", "limit": 5}] + ] + +这些动作使用默认的运行记录存储，因此 ``FA_pipeline_status``、``FA_pipeline_history`` +与 ``FA_pipeline_resume`` 看到的是同一个进程中的运行，除非已用 +``set_default_run_store`` 指定 ``SQLiteRunStore``。任务失败时，``FA_pipeline_run`` +返回 ``status`` 为 ``failed`` 的运行；只有定义无法载入或无效时才会抛出异常。 +``register_pipeline_ops(registry)`` 可以把这些动作加入你自己的注册表。 + +定义会写出它的任务要调用哪些动作。只要定义本身包含在请求里，TCP 或 HTTP 动作服务器上的 +:class:`~automation_file.ActionACL` 与 MCP 服务器的 ``--allowed-actions`` 也都会检查 +这些名称。两者都看不到以文件路径指定的定义，也看不到 ``FA_pipeline_resume`` 所接续的 +已存储运行：请只对可以调用全部已注册动作的客户端开放 ``FA_pipeline_run`` 与 +``FA_pipeline_resume``，或把定义文件放在这些客户端无法写入的位置。 + +出问题时 +---------------- + +任务失败 + ``run.status`` 为 ``failed``，``run.error`` 列出这些任务，每个任务状态都有 + ``error``、``attempts`` 与时间。排除原因之后调用 ``resume(run_id)``：已成功的 + 部分不会重做。 + +任务没有执行 + 请看 ``reason``。``upstream_failed`` 与 ``upstream_skipped`` 指向某个依赖任务； + 无论如何都必须执行的任务请设置 ``when="always"``。 + +工作出了问题，任务却成功 + 任务只有在抛出异常时才算失败。以返回值报告的动作会让任务成功： + ``FA_storage_verify`` 在不匹配时返回 ``false``，除非传入 ``strict: true``，此时它会 + 抛出 ``StorageChecksumException``，任务因而失败。其他这类返回值，请在会抛出异常的 + 可调用对象中，或在下一个任务的可调用 ``when`` 中检查。 + +任务从不重试 + 它的异常不在 ``retry_on`` 中。默认只涵盖 ``StorageTransientException``、 + ``ConnectionError`` 与 ``TimeoutError``。 + +任务已经 ``timeout``，却好像还在工作 + 它的线程无法被停止。请让可调用对象检查 ``ctx.cancel``，并确保第二次运行不会 + 与第一次遗留的工作相冲突。 + +运行被中断（进程结束、Ctrl-C） + 存储中有到那一刻为止的每一次转换，其中可能有任务停在 ``running``。 + ``resume(run_id)`` 会执行所有尚未成功的任务。 + +还没执行任何任务就抛出 ``PipelineDefinitionException`` + 定义有误。``error.problems`` 保存每一项问题及其路径；``validate_definition`` + 与试运行可以在不执行的情况下取得它们。 + +历史不完整 + 存储无法写入时会记录为错误，运行则继续进行。运行本身是正确的；它的记录则 + 不是。 + +两个异常，``PipelineException`` 与其子类 ``PipelineDefinitionException``，都派生自 +``FileAutomationException``。 diff --git a/docs/source/Zh-CN/usage/storage.rst b/docs/source/Zh-CN/usage/storage.rst index 37ecde2..7c59dc5 100644 --- a/docs/source/Zh-CN/usage/storage.rst +++ b/docs/source/Zh-CN/usage/storage.rst @@ -178,8 +178,9 @@ API；:class:`~automation_file.StorageBackend` 则是后端需要实现的契约 - ``uri, algorithm="sha256"`` - ``{"algorithm": …, "value": …}`` * - ``FA_storage_verify`` - - ``uri, expected, algorithm="sha256"`` - - ``true`` / ``false`` + - ``uri, expected, algorithm="sha256", strict=False`` + - ``true`` / ``false``；``strict=True`` 时，不匹配会抛出 + ``StorageChecksumException`` * - ``FA_storage_copy`` - ``source, target, overwrite=True`` - 目标的文件信息 @@ -255,6 +256,8 @@ API；:class:`~automation_file.StorageBackend` 则是后端需要实现的契约 - 后端尚未初始化，或其 SDK 未安装。 * - ``StorageUnsupportedException`` - 后端无法执行所要求的操作（未知的校验算法、删除根目录）。 + * - ``StorageChecksumException`` + - 严格验证（``FA_storage_verify`` 搭配 ``strict=True``）发现摘要与预期不同。 内置后端 -------- diff --git a/docs/source/Zh-CN/zh_cn_index.rst b/docs/source/Zh-CN/zh_cn_index.rst index 78aad84..a7411a0 100644 --- a/docs/source/Zh-CN/zh_cn_index.rst +++ b/docs/source/Zh-CN/zh_cn_index.rst @@ -295,3 +295,17 @@ Slack、Email（SMTP）、Discord、Telegram、Microsoft Teams、PagerDuty :caption: 审计轨迹 usage/audit + +.. _zh-cn-pipeline: + +第 20 章 — 流水线（Pipeline） +============================= + +具有依赖关系的任务，按顺序执行，并支持重试、超时、取消、条件、幂等、检查点与续跑、 +试运行（dry run）以及执行历史；可以用 Python 编写，或写成带版本的 YAML / JSON 定义。 + +.. toctree:: + :maxdepth: 2 + :caption: 流水线 + + usage/pipeline diff --git a/docs/source/Zh-TW/usage/pipeline.rst b/docs/source/Zh-TW/usage/pipeline.rst new file mode 100644 index 0000000..068d112 --- /dev/null +++ b/docs/source/Zh-TW/usage/pipeline.rst @@ -0,0 +1,696 @@ +管線（Pipeline） +==================== + +``automation_file.pipeline`` 依相依順序執行一組任務，互不相依的任務平行執行，並補上 +週期性工作需要的功能：重試、逾時、取消、條件、冪等鍵、檢查點與續跑、試跑，以及執行 +歷史。 + +任務可以是 Python 可呼叫物件，也可以是 ``FA_*`` 動作。管線可以用 Python 建立，也可以 +寫成 YAML / JSON 文件。一次執行只透過事件匯流排上的事件回報（見 :doc:`event_bus`）； +它不會呼叫任何通知 sink，也不會寫入稽核紀錄。 + +:func:`~automation_file.execute_action_dag`\ （見 :doc:`dag`）維持不變。它把一份動作 +清單執行一次並回傳結果；當執行需要被記錄、重試、續跑或觀察時，請改用管線。 + +最小範例 +---------------- + +.. code-block:: python + + from automation_file.pipeline import Pipeline + + def count_rows(ctx): + return len(ctx.results["read"].splitlines()) + + pipeline = Pipeline("row-count") + pipeline.task("read", ["FA_storage_read_text", {"uri": "local:///data/report.csv"}]) + pipeline.task("count", count_rows, depends_on=["read"]) + + run = pipeline.run() + run.status # RunStatus.SUCCEEDED；等於 "succeeded" + run.tasks["count"].result # 42 + run.tasks["count"].attempts # 1 + +``run()`` 在所有任務都結束後才回傳。任務失敗不會拋出例外：結果記在 ``run.status`` +以及 ``run.tasks`` 的每個項目上。 + +正式環境範例 +------------------------ + +一個每晚執行的工作：以重試與逾時抓取檔案、檢查內容、同一個日期最多發布一次、發布 +失敗時收回發布到一半的報表，並把執行紀錄存進 SQLite 檔案，讓失敗的執行可以續跑。 + +.. code-block:: python + + from automation_file.pipeline import Pipeline, RetryPolicy, SQLiteRunStore + + WORK = "local:///var/tmp/report-${params.date}.csv" + TARGET = "azure://reports/${params.date}.csv" + store = SQLiteRunStore("/var/lib/automation/pipelines.db") + + def check(ctx): + ctx.cancel.raise_if_cancelled() # 被取消或逾時就停下來 + if ctx.results["download"]["size"] == 0: + raise ValueError("the report is empty") # 任務以拋出例外表示失敗 + return {"bytes": ctx.results["download"]["size"]} + + pipeline = Pipeline("daily-report", description="Fetch, check, publish", max_workers=4) + pipeline.task( + "download", + ["FA_storage_copy", {"source": "s3://input/${params.date}.csv", "target": WORK}], + retry=RetryPolicy(max_attempts=5, backoff_base=2.0, backoff_cap=60.0), + timeout=300.0, + ) + pipeline.task("check", check, depends_on=["download"], timeout=60.0) + pipeline.task( + "publish", + ["FA_storage_copy", {"source": WORK, "target": TARGET}], + depends_on=["check"], + retry=RetryPolicy(max_attempts=3, backoff_base=1.0), + idempotency_key="publish-${params.date}", # 同一個日期絕不發布兩次 + ) + pipeline.task( + "withdraw", # 清理：只在 publish 失敗時執行 + ["FA_storage_delete", {"uri": TARGET, "missing_ok": True}], + depends_on=["publish"], + when="on_failure", + ) + pipeline.task("tidy", ["FA_storage_delete", {"uri": WORK}], depends_on=["publish"]) + + run = pipeline.run(params={"date": "2026-10-08"}, store=store) + if run.status != "succeeded": + for task_id, state in run.tasks.items(): + print(task_id, state.status.value, state.error or state.reason or "") + + # 稍後，在同一個或另一個行程中，等原因排除之後： + run = pipeline.resume(run.run_id, store=store) # 只執行尚未成功的部分 + +``publish`` 三次嘗試都失敗時，``withdraw`` 會執行，``tidy`` 被略過，這次執行的狀態為 +``failed``。``resume`` 保留 ``download`` 與 ``check``，重新執行 ``publish``，接著執行 +``tidy``。之後同一個日期的另一次執行會重新下載與檢查，但略過 ``publish``：它的鍵已經 +成功過。 + +同一條管線不寫 Python 的版本，存成 ``daily-report.yaml``。``check`` 是可呼叫物件，無法 +寫進文件，所以這個版本直接發布下載到的內容： + +.. code-block:: yaml + + schema_version: 1 + name: daily-report + description: Fetch and publish the daily report + max_workers: 4 + schedule: {cron: "0 2 * * *", timezone: Asia/Taipei} + params: {date: "2026-10-08"} + tasks: + download: + action: ["FA_storage_copy", {"source": "s3://input/${params.date}.csv", + "target": "local:///var/tmp/report-${params.date}.csv"}] + retry: {max_attempts: 5, backoff: 2, backoff_cap: 60, + on: [StorageTransientException, ConnectionError]} + timeout: 300 + publish: + action: ["FA_storage_copy", {"source": "local:///var/tmp/report-${params.date}.csv", + "target": "azure://reports/${params.date}.csv"}] + depends_on: [download] + retry: {max_attempts: 3, backoff: 1} + idempotency_key: "publish-${params.date}" + withdraw: + action: ["FA_storage_delete", {"uri": "azure://reports/${params.date}.csv", + "missing_ok": true}] + depends_on: [publish] + when: on_failure + tidy: + action: ["FA_storage_delete", {"uri": "local:///var/tmp/report-${params.date}.csv"}] + depends_on: [publish] + +.. code-block:: python + + pipeline = Pipeline.from_file("daily-report.yaml") + run = pipeline.run(params={"date": "2026-10-09"}, store=store) + +任務 +-------- + +任務要做的事有兩種寫法。 + +**可呼叫物件**：接收一個 :class:`~automation_file.pipeline.model.TaskContext`。它的 +回傳值就是任務的結果；拋出例外則任務失敗。 + +.. list-table:: + :header-rows: 1 + :widths: 20 80 + + * - 欄位 + - 意義 + * - ``pipeline`` + - 管線的名稱。 + * - ``run_id`` + - 這次執行的 ID，同時也是每個事件的關聯 ID。 + * - ``task`` + - 任務的 ID。 + * - ``attempt`` + - 第幾次嘗試，從 1 開始（在 ``when`` 可呼叫物件中為 ``0``）。 + * - ``params`` + - 這次執行的參數，唯讀：管線的預設值，再由 ``run(params=...)`` 覆寫。 + * - ``results`` + - 任務 ID 對應到結果，唯讀，涵蓋所有成功的上游任務，不論是否直接相依。失敗 + 或被略過的任務不會出現在其中。 + * - ``cancel`` + - ``CancellationToken``。執行被取消或任務逾時後會被設定。見 `逾時`_。 + * - ``dry_run`` + - 永遠是 ``False``：試跑不會執行任何東西。 + +**動作**：三種形式之一，``[name]``、``[name, {kwargs}]`` 與 ``[name, [args]]``。名稱 +會在共用執行器的註冊表（或傳給管線的 ``registry=``）中查找，並直接呼叫該指令，因此 +失敗時例外會拋進任務。在引數中，不論巢狀多深： + +* ``${params.}`` 會被換成該參數。夾在較長的文字中時以文字插入；整個字串剛好 + 就是一個占位符時，會換成參數本身，所以 ``"${params.limit}"`` 仍然是數字 ``20``。 +* 整個字串剛好是 ``${tasks..result}`` 時，會換成該上游任務的結果物件（若它沒有 + 成功則為 ``None``）。該任務必須是上游任務，而且這個占位符不能夾在較長的文字中。 + +這裡沒有運算式語言，也不會對任何內容求值。其他的 ``${...}`` 文字會原樣傳遞。由於 +引數中的每個字串都會被檢查，把管線定義當成引數交給 ``FA_pipeline_run`` 時，其中的 +占位符會被外層管線填入：請改為傳入檔案路徑。 + +任務 ID 與參數名稱是非空字串，不能包含空白、``.``、``$``、``{`` 或 ``}``。 + +選項 +-------- + +``Pipeline(name, description="", max_workers=4, *, params=None, schedule=None, registry=None)`` + +.. list-table:: + :header-rows: 1 + :widths: 22 78 + + * - 選項 + - 意義 + * - ``name`` + - 在事件、歷史與冪等鍵中用來辨識這條管線。 + * - ``description`` + - 自由文字，保存在定義中。 + * - ``max_workers`` + - 同時執行的任務數量上限。預設為 ``4``。 + * - ``params`` + - 預設參數；``run(params=...)`` 會加入並覆寫它們。 + * - ``schedule`` + - ``Schedule(cron, timezone=None)``。保存在 ``pipeline.schedule`` 供排程器 + 使用，管線本身不會據此行動。 + * - ``registry`` + - 查找動作名稱的地方。預設：共用執行器的註冊表。 + +``pipeline.task(task_id, work, *, depends_on=None, retry=None, timeout=None, when="on_success", idempotency_key=None)`` + +.. list-table:: + :header-rows: 1 + :widths: 22 78 + + * - 選項 + - 意義 + * - ``task_id`` + - 在管線中必須唯一。加入第二個相同 ID 的任務會立刻被拒絕。 + * - ``work`` + - 可呼叫物件或動作。 + * - ``depends_on`` + - 必須先結束的任務 ID。可以先寫出尚未加入的任務；相依圖在管線執行時才 + 檢查。 + * - ``retry`` + - ``RetryPolicy``。預設：只嘗試一次。見 `重試`_。 + * - ``timeout`` + - 整個任務可用的秒數。預設：沒有限制。見 `逾時`_。 + * - ``when`` + - ``"on_success"``\ （預設）、``"on_failure"``、``"always"`` 或可呼叫物件。 + 見 `條件`_。 + * - ``idempotency_key`` + - 可含 ``${params.}`` 占位符的文字。見 `冪等`_。 + +``RetryPolicy(max_attempts=1, backoff_base=0.0, backoff_cap=60.0, retry_on=(...))`` + +.. list-table:: + :header-rows: 1 + :widths: 22 78 + + * - 選項 + - 意義 + * - ``max_attempts`` + - 總共嘗試幾次；``1`` 表示不重試。 + * - ``backoff_base`` + - 第一次嘗試失敗後等待的秒數；之後每失敗一次就加倍。 + * - ``backoff_cap`` + - 等待時間的上限。 + * - ``retry_on`` + - 值得再試一次的例外類別。預設：``StorageTransientException``、 + ``ConnectionError``、``TimeoutError``。 + +``pipeline.run(params=None, *, dry_run=False, store=None, cancel=None, bus=None)``、 +``pipeline.start(params=None, *, store=None, cancel=None, bus=None)`` 與 +``pipeline.resume(run_id, *, store=None, cancel=None, bus=None)`` + +.. list-table:: + :header-rows: 1 + :widths: 22 78 + + * - 選項 + - 意義 + * - ``params`` + - 這次執行的參數。 + * - ``dry_run`` + - 只規劃而不執行。見 `試跑`_。 + * - ``store`` + - 記錄這次執行的 ``RunStore``。預設：預設的儲存。 + * - ``cancel`` + - ``CancellationToken``，被設定時會停止這次執行。 + * - ``bus`` + - 接收事件的 ``EventBus``。預設：整個行程共用的匯流排。 + * - ``run_id`` + - 用於 ``resume``：要接續的那次執行。 + +``run`` 在呼叫它的執行緒中工作，並回傳已結束的 +:class:`~automation_file.pipeline.model.PipelineRun`。``start`` 立刻回傳執行物件， +並在背景執行緒中工作：``run.wait(timeout)`` 會等到它結束（並回傳是否已結束）， +``run.done`` 不等待就能得知，``run.cancel()`` 則會停止它。已啟動的執行還沒結束時， +直譯器不會結束。 + +在任何任務開始之前，``run``、``start`` 與 ``resume`` 會對以下情況拋出 +``PipelineDefinitionException``：管線是空的、相依的任務不存在或重複、任務相依於 +自己、相依圖有循環、占位符格式錯誤、結果占位符指向不是上游的任務，以及占位符用到 +這次執行沒有提供的參數。``error.problems`` 列出每一項問題及其路徑； +``pipeline.problems()`` 回傳同一份清單但不拋出例外。 + +狀態 +-------- + +``run.tasks[task_id].status`` 是 ``TaskStatus``，``run.status`` 是 ``RunStatus``。 +兩者都可以直接與它們的文字比較。 + +.. list-table:: + :header-rows: 1 + :widths: 18 82 + + * - 任務狀態 + - 意義 + * - ``pending`` + - 尚未開始。 + * - ``running`` + - 正在進行某一次嘗試，或正在兩次嘗試之間等待。 + * - ``succeeded`` + - 已回傳；``result`` 保存回傳值。 + * - ``failed`` + - 拋出了例外，而且沒有剩餘的嘗試次數；``error`` 的形式為 + ``": "``。 + * - ``skipped`` + - 沒有執行；``reason`` 說明原因（見下表）。 + * - ``timeout`` + - 沒有在逾時時間內完成。 + * - ``cancelled`` + - 執行在它開始之前或執行期間被取消，或任務拋出了 ``CancelledException``。 + * - ``planned`` + - 試跑：它會依這個順序被考慮。 + +.. list-table:: + :header-rows: 1 + :widths: 24 76 + + * - 略過的 ``reason`` + - 意義 + * - ``idempotent`` + - 它的冪等鍵已經有一次成功的執行；沿用儲存的結果，相依於它的任務照常執行。 + * - ``condition`` + - 它自己的 ``on_failure`` 條件或可呼叫條件沒有成立。 + * - ``upstream_failed`` + - ``on_success`` 任務，而它的某個相依任務失敗、逾時或被取消。 + * - ``upstream_skipped`` + - ``on_success`` 任務，而它的某個相依任務被略過。 + +.. list-table:: + :header-rows: 1 + :widths: 18 82 + + * - 執行狀態 + - 意義 + * - ``running`` + - 尚未結束。 + * - ``succeeded`` + - 每個任務都成功或被略過。 + * - ``failed`` + - 至少有一個任務失敗、逾時或被取消；``run.error`` 會列出它們。清理任務成功 + 並不會改變這個結果。 + * - ``cancelled`` + - 執行被取消，而且至少有一個任務因此沒有執行。 + +任務狀態還有 ``attempts``、``started_at`` 與 ``finished_at``\ （UTC）、 +``duration_ms``、``level``\ （沒有相依的任務為 0）以及 ``idempotency_key``\ （已填入 +占位符的鍵）。``run.tasks`` 依相依順序排列，``run.to_dict()`` 可以序列化為 JSON。 + +條件 +-------- + +``when`` 只在任務的所有相依任務都結束時檢查一次。 + +``"on_success"`` + 每個相依任務都成功（或以 ``idempotent`` 被略過）。否則任務被略過，而且這會 + 傳到它自己的 ``on_success`` 下游任務。 + +``"on_failure"`` + 至少有一個相依任務失敗、逾時或被取消。用於清理。只是被略過的相依任務不算。 + +``"always"`` + 不論相依任務的結果如何。 + +可呼叫物件 ``(TaskContext) -> bool`` + 回傳真值時任務才執行。它獨自決定：不會參考相依任務的結果，但 ``ctx.results`` + 只包含成功的那些。可呼叫物件拋出例外時，任務為 ``failed``。 + +重試 +-------- + +一次嘗試失敗後，如果還有剩餘次數，而且例外是 ``retry_on`` 中某個類別的實例，任務 +就會再試一次。第 ``n + 1`` 次嘗試之前等待 ``backoff_base * 2 ** (n - 1)`` 秒，最多 +``backoff_cap`` 秒。 + +預設的 ``retry_on`` 只包含暫時性的錯誤。``ValueError`` 或 ``KeyError`` 代表程式錯誤 +或輸入有誤，第一次嘗試就會失敗。請把 ``retry_on`` 放寬到你確知是暫時性的錯誤，絕對 +不要放寬到 ``Exception``。 + +逾時 +-------- + +``timeout`` 是整個任務可用的秒數：包含每一次嘗試以及嘗試之間的等待。用完之後，任務 +被記錄為 ``timeout``，它的取消權杖被設定，其餘任務繼續執行。 + +**執行緒無法被強制終止。**\ 可呼叫物件會繼續執行直到它自己返回，而它在逾時之後回傳 +或拋出的任何東西都會被忽略。因此執行時間長的可呼叫物件必須檢查自己的權杖： + +.. code-block:: python + + def export(ctx): + for chunk in chunks(): + ctx.cancel.raise_if_cancelled() # 拋出 CancelledException + write(chunk) + +動作無法檢查權杖；動作本身若有逾時參數，請為長時間的傳輸設定它。逾時的任務不再 +計入 ``max_workers``。任務執行緒是 daemon 執行緒，所以永不返回的執行緒不會讓直譯器 +無法結束。 + +取消 +-------- + +``run.cancel()``，或設定以 ``cancel=`` 傳入的 ``CancellationToken``，會在百分之幾秒內 +停止一次執行： + +* 所有尚未開始的任務變成 ``cancelled``，清理任務也一樣； +* 所有執行中任務的權杖被設定。執行會等待這些任務，而每個任務保留它實際的結果： + 拋出 ``CancelledException`` 的為 ``cancelled``，照樣完成的為 ``succeeded``； +* 正在兩次嘗試之間等待的任務會停止等待，變成 ``cancelled``。 + +.. code-block:: python + + run = pipeline.start(params={"date": "2026-10-08"}) + ... + run.cancel() + run.wait(30) + run.status # "cancelled" + +冪等 +-------- + +帶有 ``idempotency_key`` 的任務，一旦以該鍵成功過，就不會再次執行。任務開始之前， +會以這次執行的參數算出鍵，並在儲存中查找相同管線名稱與任務 ID 的紀錄。找到成功的 +執行時，任務為 ``skipped``，原因是 ``idempotent``，它的 ``result`` 是儲存的那一份， +相依於它的任務會像它成功了一樣照常執行。 + +鍵的持久程度取決於儲存：使用預設的記憶體儲存時，只維持到行程結束；使用 +``SQLiteRunStore`` 時，重新啟動後仍然有效。它不是鎖：同時開始的兩次執行可能都查不到 +紀錄，於是都執行該任務。如果儲存無法讀取，任務會失敗而不是照常執行。 + +檢查點與續跑 +------------------------ + +任務的每一次狀態轉換都會在發生當下寫入儲存：每次嘗試的開始、每次失敗的嘗試，以及 +最後的結果。``pipeline.resume(run_id)`` 載入該次執行，保留 ``succeeded`` 的任務及其 +結果，並以相同的執行 ID 與相同的參數重新執行其餘任務。 + +.. code-block:: python + + store = SQLiteRunStore("pipelines.db") + run = pipeline.run(params={"date": "2026-10-08"}, store=store) + # ... 行程可能在這裡結束 ... + pipeline = build_pipeline() # 相同的任務 + run = pipeline.resume(run.run_id, store=SQLiteRunStore("pipelines.db")) + +儲存保存的是一次執行的狀態，而不是管線本身：``resume`` 要在名稱相同的管線上呼叫。 +結果以 JSON 儲存；JSON 無法表示的值會以它的 ``repr`` 儲存，並設定 +``result_is_repr``，所以續跑之後下游任務看到的是那段文字。其他任務需要用到的結果， +請讓任務回傳 JSON 資料。已經成功的執行會照儲存的內容原樣回傳。不要續跑仍在別處 +執行中的執行。 + +被保留的任務不會再執行一次，所以它產生的東西必須還在。清理任務應該移除失敗任務 +留下的東西，而不是某個已成功、後續任務還需要的任務的輸出。 + +試跑 +-------- + +``pipeline.run(dry_run=True)`` 不執行任何東西、不記錄任何東西，也不發布任何事件。 +每個任務都以 ``planned`` 回傳，依相依順序排列並帶有 ``level``。註冊表中找不到的動作 +名稱，以及用到缺少參數的占位符，會回報在任務的 ``error`` 中；這時執行狀態為 +``failed``，否則為 ``succeeded``。相依圖有問題時仍然會拋出例外。 + +.. code-block:: python + + plan = pipeline.run(params={"date": "2026-10-08"}, dry_run=True) + for task_id, state in plan.tasks.items(): + print(state.level, task_id, state.error or "ok") + +執行紀錄儲存與歷史 +------------------------------------ + +.. list-table:: + :header-rows: 1 + :widths: 30 70 + + * - 儲存 + - 執行紀錄保存在 + * - ``MemoryRunStore(max_runs=1000)`` + - 記憶體中，維持到行程結束；最舊的執行最先被捨棄。這是預設的儲存。 + * - ``SQLiteRunStore(path)`` + - SQLite 檔案中。執行緒安全、只使用參數化陳述式、每次轉換提交一次，檔案中 + 帶有結構版本。 + +.. code-block:: python + + from automation_file.pipeline import SQLiteRunStore, set_default_run_store + + store = SQLiteRunStore("/var/lib/automation/pipelines.db") + set_default_run_store(store) # 供 run()、resume() 與動作使用 + + store.get_run(run_id) # PipelineRun，或 None + store.list_runs("daily-report", limit=10) # 最新的在前 + store.find_idempotent("daily-report", "publish", "publish-2026-10-08") + +自訂的儲存要繼承 ``RunStore``，並實作 ``save_run``、``save_task``、``get_run``、 +``list_runs`` 與 ``find_idempotent``；無法讀取或寫入時拋出 ``PipelineException``。 +參數與結果會原樣儲存：不要把密碼與權杖放進其中任何一個。 + +定義檔 +------------ + +定義是一份帶有 ``schema_version: 1`` 的對應表，來源可以是 YAML、JSON 或 Python。 + +.. list-table:: + :header-rows: 1 + :widths: 30 70 + + * - 鍵 + - 值 + * - ``schema_version`` + - ``1``。必填；沒有這個鍵或值不同的文件會被拒絕。 + * - ``name`` + - 必填。 + * - ``description``、``max_workers``、``params`` + - 與 ``Pipeline`` 的選項相同。 + * - ``schedule`` + - ``{cron: "0 2 * * *", timezone: Asia/Taipei}``。``cron`` 有五個欄位； + ``timezone`` 可省略。 + * - ``tasks`` + - 必填。任務 ID 對應到任務。 + * - ``tasks..action`` + - 必填。``[name]``、``[name, {kwargs}]`` 或 ``[name, [args]]``。 + * - ``tasks..depends_on`` + - 任務 ID 的清單。 + * - ``tasks..retry`` + - ``{max_attempts, backoff, backoff_cap, on}``。``on`` 列出例外名稱： + ``automation_file.exceptions`` 中的類別、``TimeoutError``、 + ``ConnectionError`` 與 ``OSError``。其他名稱都是錯誤。 + * - ``tasks..timeout`` + - 秒數，必須大於 0。 + * - ``tasks..when`` + - ``on_success``、``on_failure`` 或 ``always``。 + * - ``tasks..idempotency_key`` + - 可含 ``${params.}`` 占位符的文字。 + +.. code-block:: python + + from automation_file.pipeline import PIPELINE_SCHEMA, Pipeline, validate_definition + + validate_definition(document) # 有效時為 []，否則是每一項問題及其路徑 + # ["max_workers: expected an integer >= 1, got 0", + # "tasks.verify.depends_on[0]: unknown task 'x'"] + + pipeline = Pipeline.from_dict(document) # 會拋出 PipelineDefinitionException + pipeline = Pipeline.from_file("daily-report.yaml") # .yaml、.yml 或 .json + pipeline.to_dict() # 定義文件，省略預設值 + PIPELINE_SCHEMA # JSON Schema（draft 2020-12），是一個 dict + +每一層出現未知的鍵都是錯誤。管線中含有 Python 可呼叫物件時，``to_dict`` 會拋出 +例外，因為文件無法表示它。 + +YAML 以 ``yaml.safe_load`` 讀取。YAML 的三種行為已經處理： + +* 同一個對應表中重複的鍵是錯誤（JSON 也一樣），所以第二個相同 ID 的任務不會無聲地 + 取代第一個。 +* YAML 1.1 會把沒有加引號的 ``on`` 讀成 ``true``。``from_file`` 會把 ``retry`` 底下 + 的這個鍵還原為 ``on``；如果你自己解析 YAML，請寫成 ``"on"``。 +* 沒有加引號的日期（例如 ``2026-10-08``）會變成日期物件，定義無法保存它。驗證會 + 指出這一點：請為日期與時間加上引號。 + +事件 +-------- + +每個事件的 ``source`` 都是 ``"pipeline"``，``correlation_id`` 都是執行 ID，從任務的 +執行緒發布時也一樣。在任務內部發布的事件（例如儲存錯誤）帶有相同的關聯 ID，以及 +啟動這次執行的程式的 actor。 + +.. list-table:: + :header-rows: 1 + :widths: 24 16 60 + + * - 事件 + - 嚴重程度 + - 發布時機 + * - ``pipeline.started`` + - info + - 一次，在第一個任務之前。續跑時會再發布一次。 + * - ``task.started`` + - info + - 每次嘗試開始時。 + * - ``task.completed`` + - info + - 某次嘗試成功時。 + * - ``task.failed`` + - warning / error + - 某次嘗試失敗時。``status`` 為 ``retrying``\ （warning：接著還有一次 + 嘗試）、``failed``、``timeout`` 或 ``cancelled``\ （warning）。 + * - ``pipeline.completed`` + - info + - 執行以 ``succeeded`` 結束時。 + * - ``pipeline.failed`` + - error / warning + - 執行以 ``failed`` 或 ``cancelled``\ （warning）結束時。 + +payload 使用共用的鍵：``pipeline``、``run_id``、``status``，任務事件另有 ``task`` 與 +``attempt``；某件事結束時有 ``duration_ms``，出錯時有 ``error``。被略過的任務，以及 +開始之前就被取消的任務，不會發布任何事件：它的狀態記在執行上。``when`` 可呼叫物件 +拋出例外，或冪等鍵無法查找的任務，會發布一個 ``attempt`` 為 0 的 ``task.failed``。 +試跑不會發布任何事件。 + +.. code-block:: python + + from automation_file import Severity, event_bus + + def alert(event): + print(event.subject, event.payload.get("error")) + + event_bus.subscribe(alert, types=["pipeline.failed", "task.failed"], + min_severity=Severity.ERROR) + +動作 +-------- + +管線也能從 JSON 動作清單使用，因此 CLI、TCP 與 HTTP 動作伺服器以及 MCP 主機都能 +呼叫。``definition`` 是一份對應表，或 ``.yaml`` / ``.yml`` / ``.json`` 檔案的路徑。 + +.. list-table:: + :header-rows: 1 + :widths: 26 40 34 + + * - 動作 + - 參數 + - 回傳值 + * - ``FA_pipeline_run`` + - ``definition, params=None, dry_run=False`` + - 這次執行（``PipelineRun.to_dict()``） + * - ``FA_pipeline_validate`` + - ``definition`` + - ``{"valid": …, "errors": […]}`` + * - ``FA_pipeline_status`` + - ``run_id`` + - 已記錄的執行 + * - ``FA_pipeline_history`` + - ``pipeline=None, limit=20`` + - 已記錄的執行，最新的在前 + * - ``FA_pipeline_resume`` + - ``run_id, definition`` + - 續跑之後的執行 + +.. code-block:: json + + [ + ["FA_pipeline_validate", {"definition": "pipelines/daily-report.yaml"}], + ["FA_pipeline_run", {"definition": "pipelines/daily-report.yaml", + "params": {"date": "2026-10-08"}}], + ["FA_pipeline_history", {"pipeline": "daily-report", "limit": 5}] + ] + +這些動作使用預設的執行紀錄儲存，因此 ``FA_pipeline_status``、``FA_pipeline_history`` +與 ``FA_pipeline_resume`` 看到的是同一個行程中的執行，除非已用 +``set_default_run_store`` 指定 ``SQLiteRunStore``。任務失敗時，``FA_pipeline_run`` +回傳 ``status`` 為 ``failed`` 的執行；只有定義無法載入或無效時才會拋出例外。 +``register_pipeline_ops(registry)`` 可把這些動作加入你自己的註冊表。 + +定義會寫出它的任務要呼叫哪些動作。只要定義本身包含在請求裡，TCP 或 HTTP 動作伺服器上的 +:class:`~automation_file.ActionACL` 與 MCP 伺服器的 ``--allowed-actions`` 也都會檢查 +這些名稱。兩者都看不到以檔案路徑指定的定義，也看不到 ``FA_pipeline_resume`` 所接續的 +已儲存執行：請只對可以呼叫全部已註冊動作的用戶端開放 ``FA_pipeline_run`` 與 +``FA_pipeline_resume``，或把定義檔放在這些用戶端無法寫入的位置。 + +出問題時 +---------------- + +任務失敗 + ``run.status`` 為 ``failed``，``run.error`` 列出這些任務，每個任務狀態都有 + ``error``、``attempts`` 與時間。排除原因之後呼叫 ``resume(run_id)``：已成功的 + 部分不會重做。 + +任務沒有執行 + 請看 ``reason``。``upstream_failed`` 與 ``upstream_skipped`` 指向某個相依任務； + 無論如何都必須執行的任務請設定 ``when="always"``。 + +工作出了問題，任務卻成功 + 任務只有在拋出例外時才算失敗。以回傳值回報的動作會讓任務成功： + ``FA_storage_verify`` 在不相符時回傳 ``false``，除非傳入 ``strict: true``，此時它會 + 拋出 ``StorageChecksumException``，任務因而失敗。其他這類回傳值，請在會拋出例外的 + 可呼叫物件中，或在下一個任務的可呼叫 ``when`` 中檢查。 + +任務從不重試 + 它的例外不在 ``retry_on`` 中。預設只涵蓋 ``StorageTransientException``、 + ``ConnectionError`` 與 ``TimeoutError``。 + +任務已經 ``timeout``，卻好像還在工作 + 它的執行緒無法被停止。請讓可呼叫物件檢查 ``ctx.cancel``，並確保第二次執行不會 + 與第一次遺留的工作相衝突。 + +執行被中斷（行程結束、Ctrl-C） + 儲存中有到那一刻為止的每一次轉換，其中可能有任務停在 ``running``。 + ``resume(run_id)`` 會執行所有尚未成功的任務。 + +還沒執行任何任務就拋出 ``PipelineDefinitionException`` + 定義有誤。``error.problems`` 保存每一項問題及其路徑；``validate_definition`` + 與試跑可以在不執行的情況下取得它們。 + +歷史不完整 + 儲存無法寫入時會記錄為錯誤，執行則繼續進行。執行本身是正確的；它的紀錄則 + 不是。 + +兩個例外，``PipelineException`` 與其子類別 ``PipelineDefinitionException``，都衍生自 +``FileAutomationException``。 diff --git a/docs/source/Zh-TW/usage/storage.rst b/docs/source/Zh-TW/usage/storage.rst index ca7bd72..d9f5336 100644 --- a/docs/source/Zh-TW/usage/storage.rst +++ b/docs/source/Zh-TW/usage/storage.rst @@ -178,8 +178,9 @@ API；:class:`~automation_file.StorageBackend` 則是後端要實作的契約。 - ``uri, algorithm="sha256"`` - ``{"algorithm": …, "value": …}`` * - ``FA_storage_verify`` - - ``uri, expected, algorithm="sha256"`` - - ``true`` / ``false`` + - ``uri, expected, algorithm="sha256", strict=False`` + - ``true`` / ``false``；``strict=True`` 時，不相符會擲出 + ``StorageChecksumException`` * - ``FA_storage_copy`` - ``source, target, overwrite=True`` - 目標的檔案資訊 @@ -255,6 +256,8 @@ API；:class:`~automation_file.StorageBackend` 則是後端要實作的契約。 - 後端尚未初始化，或其 SDK 未安裝。 * - ``StorageUnsupportedException`` - 後端無法執行所要求的操作（未知的校驗演算法、刪除根目錄）。 + * - ``StorageChecksumException`` + - 嚴格驗證（``FA_storage_verify`` 搭配 ``strict=True``）發現摘要與預期不同。 內建後端 -------- diff --git a/docs/source/Zh-TW/zh_tw_index.rst b/docs/source/Zh-TW/zh_tw_index.rst index d48398e..99d59b2 100644 --- a/docs/source/Zh-TW/zh_tw_index.rst +++ b/docs/source/Zh-TW/zh_tw_index.rst @@ -295,3 +295,17 @@ Slack、Email（SMTP）、Discord、Telegram、Microsoft Teams、PagerDuty :caption: 稽核軌跡 usage/audit + +.. _zh-tw-pipeline: + +第 20 章 — 管線（Pipeline） +=========================== + +具有相依關係的任務，依序執行，並支援重試、逾時、取消、條件、冪等、檢查點與續跑、 +試跑（dry run）以及執行歷史；可用 Python 撰寫，或寫成帶版本的 YAML / JSON 定義。 + +.. toctree:: + :maxdepth: 2 + :caption: 管線 + + usage/pipeline diff --git a/docs/updates/2026-10.md b/docs/updates/2026-10.md index 380e495..c42965e 100644 --- a/docs/updates/2026-10.md +++ b/docs/updates/2026-10.md @@ -441,3 +441,37 @@ Index and query commands: [README.md](README.md). New entries go at the end. - **Docs**: chapter 19 in the three manuals (`usage/audit.rst`, with an "Operational metrics" section), the route sections of the three `usage/notifications.rst`, the three `usage/config.rst`, `docs/source/API/audit.rst` and `API/notify.rst`, the indexes, two feature bullets and two sections in the three READMEs, `architecture.md` §2 to §4, `CLAUDE.md` (package map, key types). - **Files**: `automation_file/notify/{router,manager,__init__}.py`, `automation_file/audit/` (6 modules), `automation_file/core/{metrics,config,action_registry,fim}.py`, `automation_file/integrity/{legacy,monitor}.py`, `automation_file/__init__.py`, the tests above, the documentation above. - **Open items**: the scheduler half of M6 stays open as #23; an `audit` subcommand for the CLI is #33. + +## U-20261008-18 · 2026-10-08 · Pipeline runtime · #pipeline #roadmap #done + +- **What**: the package `automation_file/pipeline/` (roadmap §7, M5), which closes `progress.md` #22. + - `Pipeline(name, max_workers=...)` holds tasks: a callable taking a `TaskContext`, or an `FA_*` action, with `depends_on`. `run` executes in the calling thread, `start` in the background, `resume(run_id)` continues a stored run and repeats only what did not succeed, `run(dry_run=True)` plans without executing, publishing or storing. + - Per task: `RetryPolicy` (capped exponential back-off; by default only `StorageTransientException`, `ConnectionError` and `TimeoutError` are retried), a `timeout` that covers all attempts and is terminal, a `when` condition (`on_success`, `on_failure`, `always` or a callable), an `idempotency_key` that reuses the result of a task that already succeeded under the key. + - Independent tasks run in parallel: one daemon thread per running task, capped at `max_workers`. A thread pool was not used because a timed-out thread cannot be killed and would hold a slot and block shutdown. + - `${params.name}` and `${tasks.id.result}` are substituted in action arguments; a string that is exactly one placeholder keeps the value's type. A parameter the run was not given, an unknown upstream task, a cycle and every other definition problem are reported together, with their path, before anything runs. + - `RunStore` (`MemoryRunStore`, `SQLiteRunStore`) records every task transition: the checkpoint `resume` reads and the execution history. A failed write is logged and the run continues. + - Definitions as YAML or JSON with `schema_version: 1`: `Pipeline.from_file`, `from_dict` / `to_dict`, `validate_definition`, `PIPELINE_SCHEMA`. YAML is read with `yaml.safe_load`; duplicate keys and unquoted dates are rejected. + - A run reports only through `pipeline.*` and `task.*` events, all carrying the run ID as correlation ID. A retried attempt publishes `task.failed` with status `retrying` at warning severity. + - Five actions: `FA_pipeline_run`, `_validate`, `_status`, `_history`, `_resume`, registered by `build_default_registry`. +- **Changed while integrating**: + - A task fails only by raising, and `FA_storage_verify` returns `false` on a mismatch, so the roadmap's copy, verify, notify pipeline would have published a corrupt copy. `FA_storage_verify` takes `strict=True`, which raises the new `StorageChecksumException`; the default is unchanged. + - `tests/test_pipeline_run.py` had 1132 lines; the cases about what is rejected before a run moved to `tests/test_pipeline_rejected.py`. +- **Tests**: `tests/test_pipeline_{run,rejected,store,definition,events,actions,imports}.py` (313 cases), among them a pipeline whose strict verification fails its task and skips the task after it. +- **Result / numbers**: 4825 passed, 149 skipped, 0 failed with every extra; 3108 passed, 89 skipped with the base dependencies only. `ruff check`, `ruff format --check` and `mypy automation_file` (229 files) pass. 174 registered commands. Python 3.14.7 on Windows. +- **Not verified**: `PIPELINE_SCHEMA` against a JSON Schema validator (none is a dependency); the Sphinx build of the new pages (headings, markup and the imports of the examples were checked by script); Python 3.10 to 3.13. +- **Docs**: chapter 20 in the three manuals (`usage/pipeline.rst`), `docs/source/API/pipeline.rst`, the indexes, a feature bullet and a section in the three READMEs, the `FA_storage_verify` row and the error table of the three `usage/storage.rst`, `architecture.md` §2 to §4, `CLAUDE.md` (package map, key types). +- **Files**: `automation_file/pipeline/` (12 modules), `automation_file/storage/actions.py`, `automation_file/exceptions.py`, `automation_file/core/action_registry.py`, `automation_file/__init__.py`, the tests above, the documentation above. +- **Open items**: the scheduler does not read a pipeline's `schedule` yet (#23); a `pipeline` subcommand for the CLI is #33. `core/dag_executor.py` (`execute_action_dag`) is unchanged and not built on the new runtime. + +## U-20261008-19 · 2026-10-08 · The action ACL and the MCP server check nested action names · #security #incident + +- **What was wrong**: `ActionACL.enforce` looked only at the first element of each top-level entry of a request. An action that carries other actions in its arguments therefore ran them unchecked: a server whose ACL denied `FA_run_shell` still ran it inside `["FA_execute_action", [[["FA_run_shell", ...]]]]`. The same held for `FA_execute_action_parallel`, the scheduler and trigger actions that store a list to run later, and, on this branch, `FA_pipeline_run` with a definition. It predates the branch; the pipeline work made it visible. +- **Fix** (`automation_file/server/action_acl.py`): every registered action name that appears anywhere in the arguments of an entry, at any depth, as a list element, a mapping value or a mapping key, is checked like a top-level name. The names come from the shared executor's registry, which is what the TCP and HTTP servers dispatch through. The walk is iterative, so a deeply nested request cannot exhaust the stack. + - A string argument that equals a registered action name is checked as that action. With an allow list this can refuse a harmless request whose argument happens to be such a name; that is the cautious side for an access control. + - Arguments that are not action names (paths, URIs, lists of them) are not affected. +- **What it still cannot see**: what a request only points to. An action list in a file (`FA_execute_files`), a pipeline definition given as a path (`FA_pipeline_run`) and a stored run (`FA_pipeline_resume`) run what the file or the store says. The module docstring, the pipeline manuals and `CLAUDE.md` say to deny those actions for a client that must stay inside the list. +- **MCP server**: `--allowed-actions` narrowed the registry the server lists and calls, but a listed tool such as `FA_execute_action` ran the actions in its arguments through the shared executor. `MCPServer` now refuses a `tools/call` whose arguments name a registered action the server does not expose (`nested_action_names`, shared with the ACL). +- **Tests**: four cases in `tests/test_action_acl.py`: a denied action inside an action list, an action named by a pipeline definition under an allow list, a name under a mapping key and 5000 levels deep, and ordinary arguments passing an allow list. `tests/test_server_acl.py` passes unchanged. Two cases in `tests/test_mcp_server.py`: a narrowed server refuses a nested action it does not expose and runs one it does; an unfiltered server runs nested actions. +- **Result / numbers**: part of the run recorded in U-20261008-18. +- **Files**: `automation_file/server/action_acl.py`, `automation_file/server/mcp_server.py`, `tests/test_action_acl.py`, `tests/test_mcp_server.py`, the three `usage/pipeline.rst`, `CLAUDE.md`. +- **Open items**: none. diff --git a/docs/updates/README.md b/docs/updates/README.md index 5ec41eb..ad74f6f 100644 --- a/docs/updates/README.md +++ b/docs/updates/README.md @@ -58,6 +58,8 @@ In the same commit: delete the item from `progress.md`, add a `#done` entry here | ID | Date | Title | Tags | Batch | |---|---|---|---|---| +| U-20261008-19 | 2026-10-08 | The action ACL and the MCP server check nested action names | #security #incident | [2026-10](2026-10.md) | +| U-20261008-18 | 2026-10-08 | Pipeline runtime | #pipeline #roadmap #done | [2026-10](2026-10.md) | | U-20261008-17 | 2026-10-08 | Notification router and audit schema v2 | #notify #audit #roadmap | [2026-10](2026-10.md) | | U-20261008-16 | 2026-10-08 | IntegrityMonitor 2.0 | #integrity #roadmap #done | [2026-10](2026-10.md) | | U-20261008-15 | 2026-10-08 | The SFTP, OneDrive and SMB clients name the extra to install | #packaging #done | [2026-10](2026-10.md) | @@ -112,5 +114,5 @@ In the same commit: delete the item from `progress.md`, add a `#done` entry here | File | Period | Entries | |---|---|---:| -| [2026-10.md](2026-10.md) | 2026-10 | 28 | +| [2026-10.md](2026-10.md) | 2026-10 | 30 | | [2026-09.md](2026-09.md) | 2026-09 | 21 | diff --git a/progress.md b/progress.md index 1559d91..84f0e9b 100644 --- a/progress.md +++ b/progress.md @@ -28,7 +28,6 @@ Items #10 to #26 are what is left of the 1.0 roadmap (`docs/FILEAUTOMATION-1.0-R ### Later milestones - **#33** CLI subcommands for the packages that have none: `integrity` (snapshot, baseline, verify, accept, status), `pipeline` and `audit`, as thin calls into their `FA_*` functions like `storage` (U-20261008-11). -- **#22** Pipeline runtime (roadmap §7, M5): `Pipeline` domain model, DAG runtime v2 with retry, timeout, cancellation, conditions, idempotency, checkpoint and resume, dry run, execution history, and versioned YAML/JSON definitions with schema validation. `core/dag_executor.py` is the starting point. - **#23** Scheduler v2 (roadmap §8, the open half of M6): one scheduler with cron (time-zone aware), manual, file-event, webhook and pipeline-dependency triggers, run states and overlap protection, reading a pipeline's `schedule`. The event model, the `NotificationRouter` and audit schema v2 are done (U-20261008-05, U-20261008-17); the scheduler in `scheduler/` still dispatches action lists on its own cron loop. - **#24** UI 2.0 (roadmap §11, M7). Not before the APIs of #13 to #23 are stable (roadmap §20). - **#25** Semantic MCP tools (roadmap §12, M8): `file_*`, `storage_*`, `pipeline_*`, `integrity_status`, `audit_search`, with a permission model and dry run, next to the existing `FA_*` bridge. diff --git a/tests/test_action_acl.py b/tests/test_action_acl.py index 2aa407e..99af104 100644 --- a/tests/test_action_acl.py +++ b/tests/test_action_acl.py @@ -41,3 +41,44 @@ def test_enforce_ignores_malformed_entries() -> None: acl = ActionACL.build(allowed=["FA_foo"]) # Non-list / empty / non-string-first entries are skipped silently. acl.enforce([[], [1, 2], "garbage"]) # type: ignore[list-item] + + +def test_an_action_nested_in_an_action_list_is_checked() -> None: + acl = ActionACL.build(denied=["FA_run_shell"]) + nested = [["FA_execute_action", [[["FA_run_shell", {"argv": ["echo"]}]]]]] + with pytest.raises(ActionNotPermittedException, match="FA_run_shell"): + acl.enforce(nested) + acl.enforce([["FA_execute_action", [[["FA_create_file", {"file_path": "x"}]]]]]) + + +def test_an_action_named_by_a_pipeline_definition_is_checked() -> None: + acl = ActionACL.build(allowed=["FA_pipeline_run", "FA_storage_copy"]) + definition = { + "schema_version": 1, + "name": "copy", + "tasks": { + "copy": {"action": ["FA_storage_copy", {"source": "a", "target": "b"}]}, + "wipe": {"action": ["FA_storage_delete", {"uri": "b"}], "depends_on": ["copy"]}, + }, + } + with pytest.raises(ActionNotPermittedException, match="FA_storage_delete"): + acl.enforce([["FA_pipeline_run", {"definition": definition}]]) + del definition["tasks"]["wipe"] + acl.enforce([["FA_pipeline_run", {"definition": definition}]]) + + +def test_a_nested_name_is_found_under_a_key_and_at_any_depth() -> None: + acl = ActionACL.build(denied=["FA_storage_delete"]) + deep: object = "FA_storage_delete" + for _ in range(5000): + deep = [deep] + with pytest.raises(ActionNotPermittedException): + acl.enforce([["FA_execute_action", deep]]) + with pytest.raises(ActionNotPermittedException): + acl.enforce([["FA_schedule_add", {"job": {"FA_storage_delete": {"uri": "b"}}}]]) + + +def test_arguments_that_are_not_action_names_pass_an_allow_list() -> None: + acl = ActionACL.build(allowed=["FA_storage_copy"]) + acl.enforce([["FA_storage_copy", {"source": "reports/q1.csv", "target": ["a", "b"]}]]) + acl.enforce([["FA_storage_copy", ["local:///a.txt", "local:///b.txt"]]]) diff --git a/tests/test_mcp_server.py b/tests/test_mcp_server.py index 986154f..198f319 100644 --- a/tests/test_mcp_server.py +++ b/tests/test_mcp_server.py @@ -193,3 +193,40 @@ def serve_stdio(self) -> None: assert captured["name"] == "t" assert captured["version"] == "2.0.0" assert captured["served"] is True + + +def _call(server: MCPServer, name: str, arguments: dict) -> dict: + return server.handle_message( + { + "jsonrpc": "2.0", + "id": 1, + "method": "tools/call", + "params": {"name": name, "arguments": arguments}, + } + ) + + +def test_a_tool_cannot_run_an_action_the_server_does_not_expose() -> None: + from automation_file import executor + + exposed = _filtered_registry(executor.registry, ["FA_execute_action", "FA_storage_exists"]) + server = MCPServer(exposed) + refused = _call( + server, "FA_execute_action", {"action_list": [["FA_run_shell", {"argv": ["echo"]}]]} + ) + assert "FA_run_shell, which this server does not expose" in refused["error"]["message"] + allowed = _call( + server, + "FA_execute_action", + {"action_list": [["FA_storage_exists", {"uri": "memory://mcp/absent.txt"}]]}, + ) + assert allowed["result"]["isError"] is False + + +def test_an_unfiltered_server_runs_nested_actions() -> None: + answer = _call( + MCPServer(), + "FA_execute_action", + {"action_list": [["FA_storage_exists", {"uri": "memory://mcp/absent.txt"}]]}, + ) + assert answer["result"]["isError"] is False diff --git a/tests/test_pipeline_actions.py b/tests/test_pipeline_actions.py new file mode 100644 index 0000000..8b35eb5 --- /dev/null +++ b/tests/test_pipeline_actions.py @@ -0,0 +1,308 @@ +"""FA_pipeline_* actions: pipelines through the registry and ``execute_action``.""" + +from __future__ import annotations + +import json +from collections.abc import Iterator +from pathlib import Path +from typing import Any + +import pytest + +from automation_file import ActionRegistry, execute_action, executor +from automation_file.pipeline import ( + MemoryRunStore, + PipelineDefinitionException, + PipelineException, + actions, + register_pipeline_ops, + set_default_run_store, +) +from automation_file.storage import clear_memory_stores + +NAMES = [ + "FA_pipeline_history", + "FA_pipeline_resume", + "FA_pipeline_run", + "FA_pipeline_status", + "FA_pipeline_validate", +] + +YAML_TEXT = """\ +schema_version: 1 +name: notes +params: {text: default} +tasks: + write: + action: ["FA_storage_write_text", {"uri": "memory://pipeline/notes/${params.name}.txt", "text": "${params.text}"}] + read: + action: ["FA_storage_read_text", {"uri": "memory://pipeline/notes/${params.name}.txt"}] + depends_on: [write] +""" + + +def definition(name: str = "notes") -> dict[str, Any]: + return { + "schema_version": 1, + "name": name, + "tasks": { + "write": { + "action": [ + "FA_storage_write_text", + {"uri": "memory://pipeline/a.txt", "text": "${params.text}"}, + ] + }, + "read": { + "action": ["FA_storage_read_text", {"uri": "memory://pipeline/a.txt"}], + "depends_on": ["write"], + }, + "size": { + "action": ["FA_storage_stat", ["memory://pipeline/a.txt"]], + "depends_on": ["write"], + }, + }, + } + + +@pytest.fixture(autouse=True) +def _isolated() -> Iterator[MemoryRunStore]: + """A fresh default run store, clean memory storage, and the actions on the shared executor.""" + clear_memory_stores() + fresh = MemoryRunStore() + previous = set_default_run_store(fresh) + before = {name: executor.registry.resolve(name) for name in NAMES} + register_pipeline_ops(executor.registry) + yield fresh + for name, command in before.items(): + if command is None: + executor.registry.unregister(name) + else: + executor.registry.register(name, command) + set_default_run_store(previous) + clear_memory_stores() + + +def run_action(name: str, arguments: dict[str, Any]) -> Any: + """Call one action through the shared executor and return its recorded result.""" + (result,) = execute_action([[name, arguments]]).values() + return result + + +def test_register_pipeline_ops_fills_a_registry() -> None: + registry = ActionRegistry() + register_pipeline_ops(registry) + assert sorted(registry.event_dict) == NAMES + assert sorted(actions.pipeline_commands()) == NAMES + assert all(callable(command) for command in actions.pipeline_commands().values()) + + +def test_run_executes_a_definition_and_returns_the_run(_isolated: MemoryRunStore) -> None: + result = run_action( + "FA_pipeline_run", {"definition": definition(), "params": {"text": "hello"}} + ) + assert json.loads(json.dumps(result)) == result + assert result["pipeline"] == "notes" + assert result["status"] == "succeeded" + assert result["dry_run"] is False + assert result["params"] == {"text": "hello"} + assert result["error"] is None + assert list(result["tasks"]) == ["write", "read", "size"] + assert result["tasks"]["read"]["result"] == "hello" + assert result["tasks"]["size"]["result"]["size"] == 5 + assert result["tasks"]["write"]["attempts"] == 1 + assert result["tasks"]["write"]["duration_ms"] >= 0 + assert _isolated.get_run(result["run_id"]).to_dict() == result + + +def test_run_takes_the_path_of_a_yaml_or_a_json_file(tmp_path: Path) -> None: + yaml_file = tmp_path / "notes.yaml" + yaml_file.write_text(YAML_TEXT, encoding="utf-8") + from_yaml = run_action( + "FA_pipeline_run", {"definition": str(yaml_file), "params": {"name": "a"}} + ) + assert from_yaml["status"] == "succeeded" + assert from_yaml["params"] == {"text": "default", "name": "a"} + assert from_yaml["tasks"]["read"]["result"] == "default" + json_file = tmp_path / "notes.json" + json_file.write_text(json.dumps(definition("from-json")), encoding="utf-8") + from_json = actions.pipeline_run(json_file, params={"text": "json"}) + assert (from_json["pipeline"], from_json["tasks"]["read"]["result"]) == ("from-json", "json") + + +def test_a_failed_task_does_not_raise_from_the_action() -> None: + document = definition() + document["tasks"]["read"]["action"][1]["uri"] = "memory://pipeline/absent.txt" + document["tasks"]["after"] = {"action": ["FA_storage_schemes"], "depends_on": ["read"]} + result = run_action("FA_pipeline_run", {"definition": document, "params": {"text": "x"}}) + assert result["status"] == "failed" + assert result["error"] == "did not succeed: read" + assert result["tasks"]["read"]["status"] == "failed" + assert result["tasks"]["read"]["error"].startswith("StorageNotFoundException: ") + assert result["tasks"]["after"] == { + **result["tasks"]["after"], + "status": "skipped", + "reason": "upstream_failed", + } + + +def test_a_dry_run_plans_without_executing_or_recording(_isolated: MemoryRunStore) -> None: + document = definition() + document["tasks"]["typo"] = {"action": ["FA_storage_wrte_text"]} + result = run_action("FA_pipeline_run", {"definition": document, "dry_run": True}) + assert result["dry_run"] is True + assert result["status"] == "failed" + assert {task["status"] for task in result["tasks"].values()} == {"planned"} + assert result["tasks"]["write"]["error"] == ( + "tasks.write.action[1].text: unknown parameter 'text'" + ) + assert result["tasks"]["typo"]["error"] == ( + "tasks.typo.action[0]: unknown action 'FA_storage_wrte_text'" + ) + assert [task["level"] for task in result["tasks"].values()] == [0, 0, 1, 1] + assert _isolated.list_runs() == [] + assert run_action("FA_storage_exists", {"uri": "memory://pipeline/a.txt"}) is False + + +def test_an_invalid_definition_is_recorded_as_the_action_s_error(_isolated: MemoryRunStore) -> None: + document = definition() + document["tasks"]["read"]["depends_on"] = ["missing"] + recorded = run_action("FA_pipeline_run", {"definition": document}) + assert "PipelineDefinitionException" in recorded + assert "tasks.read.depends_on[0]: unknown task 'missing'" in recorded + with pytest.raises(PipelineDefinitionException, match="unknown task 'missing'"): + actions.pipeline_run(document) + with pytest.raises(PipelineDefinitionException, match="expected a mapping or a file path"): + actions.pipeline_run(["not", "a", "definition"]) # type: ignore[arg-type] + with pytest.raises(PipelineDefinitionException, match="unknown parameter 'text'"): + actions.pipeline_run(definition()) + assert _isolated.list_runs() == [] + + +def test_validate_reports_every_problem(tmp_path: Path) -> None: + assert run_action("FA_pipeline_validate", {"definition": definition()}) == { + "valid": True, + "errors": [], + } + document = definition() + document["max_workers"] = 0 + document["tasks"]["read"]["depends_on"] = ["missing"] + assert run_action("FA_pipeline_validate", {"definition": document}) == { + "valid": False, + "errors": [ + "max_workers: expected an integer >= 1, got 0", + "tasks.read.depends_on[0]: unknown task 'missing'", + ], + } + good = tmp_path / "good.yaml" + good.write_text(YAML_TEXT, encoding="utf-8") + assert actions.pipeline_validate(str(good)) == {"valid": True, "errors": []} + broken = tmp_path / "broken.yaml" + broken.write_text("tasks: [unclosed\n", encoding="utf-8") + unreadable = actions.pipeline_validate(str(broken)) + assert unreadable["valid"] is False + assert unreadable["errors"][0].startswith(f"{broken}: invalid YAML: ") + missing = actions.pipeline_validate(str(tmp_path / "absent.json")) + assert missing["valid"] is False + assert "cannot read the definition" in missing["errors"][0] + assert actions.pipeline_validate(7) == { # type: ignore[arg-type] + "valid": False, + "errors": ["definition: expected a mapping or a file path, got int"], + } + + +def test_status_returns_a_recorded_run() -> None: + result = run_action("FA_pipeline_run", {"definition": definition(), "params": {"text": "x"}}) + assert run_action("FA_pipeline_status", {"run_id": result["run_id"]}) == result + assert actions.pipeline_status(result["run_id"]) == result + unknown = run_action("FA_pipeline_status", {"run_id": "no-such-run"}) + assert "PipelineException" in unknown + assert "unknown run 'no-such-run'" in unknown + with pytest.raises(PipelineException, match="unknown run"): + actions.pipeline_status("no-such-run") + + +def test_history_lists_the_recorded_runs_newest_first() -> None: + assert run_action("FA_pipeline_history", {}) == [] + first = actions.pipeline_run(definition("alpha"), params={"text": "1"}) + second = actions.pipeline_run(definition("beta"), params={"text": "2"}) + third = actions.pipeline_run(definition("alpha"), params={"text": "3"}) + history = run_action("FA_pipeline_history", {}) + assert history == [third, second, first] + assert json.loads(json.dumps(history)) == history + assert run_action("FA_pipeline_history", {"pipeline": "alpha"}) == [third, first] + assert run_action("FA_pipeline_history", {"pipeline": "alpha", "limit": 1}) == [third] + assert run_action("FA_pipeline_history", {"pipeline": "gamma"}) == [] + assert actions.pipeline_history(limit=2) == [third, second] + (listed,) = execute_action([["FA_pipeline_history", ["beta", 5]]]).values() + assert listed == [second] + + +def test_resume_continues_a_recorded_run(_isolated: MemoryRunStore) -> None: + document = definition() + document["tasks"]["read"]["action"][1]["uri"] = "memory://pipeline/late.txt" + failed = run_action("FA_pipeline_run", {"definition": document, "params": {"text": "kept"}}) + assert failed["status"] == "failed" + assert failed["tasks"]["write"]["status"] == "succeeded" + run_action("FA_storage_write_text", {"uri": "memory://pipeline/late.txt", "text": "arrived"}) + run_action("FA_storage_write_text", {"uri": "memory://pipeline/a.txt", "text": "overwritten"}) + resumed = run_action("FA_pipeline_resume", {"run_id": failed["run_id"], "definition": document}) + assert resumed["run_id"] == failed["run_id"] + assert resumed["status"] == "succeeded" + assert resumed["params"] == {"text": "kept"} + assert resumed["tasks"]["read"]["result"] == "arrived" + assert resumed["tasks"]["write"] == failed["tasks"]["write"] # not run again + assert run_action("FA_storage_read_text", {"uri": "memory://pipeline/a.txt"}) == "overwritten" + assert len(_isolated.list_runs()) == 1 + assert run_action("FA_pipeline_status", {"run_id": failed["run_id"]}) == resumed + unknown = run_action("FA_pipeline_resume", {"run_id": "no-such-run", "definition": document}) + assert "unknown run 'no-such-run'" in unknown + + +def test_a_pipeline_action_can_be_a_task_of_another_pipeline(tmp_path: Path) -> None: + inner = tmp_path / "inner.json" + inner.write_text(json.dumps(definition("inner")), encoding="utf-8") + outer = { + "schema_version": 1, + "name": "outer", + "params": {"inner": str(inner)}, + "tasks": { + "check": {"action": ["FA_pipeline_validate", {"definition": "${params.inner}"}]}, + "inner": { + "action": [ + "FA_pipeline_run", + {"definition": "${params.inner}", "params": {"text": "${params.text}"}}, + ], + "depends_on": ["check"], + }, + }, + } + result = actions.pipeline_run(outer, params={"text": "nested"}) + assert result["status"] == "succeeded" + assert result["tasks"]["check"]["result"] == {"valid": True, "errors": []} + assert result["tasks"]["inner"]["result"]["tasks"]["read"]["result"] == "nested" + assert sorted(run["pipeline"] for run in actions.pipeline_history()) == ["inner", "outer"] + + +def test_a_strict_verification_fails_its_task_on_a_mismatch() -> None: + execute_action([["FA_storage_write_text", ["memory://pipeline/verified.txt", "hello"]]]) + described = { + "schema_version": 1, + "name": "verified", + "tasks": { + "verify": { + "action": [ + "FA_storage_verify", + {"uri": "memory://pipeline/verified.txt", "expected": "0" * 64, "strict": True}, + ] + }, + "publish": { + "action": ["FA_storage_read_text", {"uri": "memory://pipeline/verified.txt"}], + "depends_on": ["verify"], + }, + }, + } + run = actions.pipeline_run(described) + assert run["status"] == "failed" + assert run["tasks"]["verify"]["status"] == "failed" + assert "StorageChecksumException" in str(run["tasks"]["verify"]) + assert run["tasks"]["publish"]["status"] == "skipped" diff --git a/tests/test_pipeline_definition.py b/tests/test_pipeline_definition.py new file mode 100644 index 0000000..633e317 --- /dev/null +++ b/tests/test_pipeline_definition.py @@ -0,0 +1,861 @@ +"""Pipeline definitions: validation with paths, the schema, round trips, YAML and JSON files.""" + +from __future__ import annotations + +import copy +import datetime +import json +from pathlib import Path +from typing import Any + +import pytest + +from automation_file import ActionRegistry, exceptions +from automation_file.events import EventBus +from automation_file.exceptions import FileAutomationException, StorageTransientException +from automation_file.pipeline import ( + DEFAULT_RETRY_ON, + PIPELINE_SCHEMA, + RETRYABLE_EXCEPTIONS, + SCHEMA_VERSION, + MemoryRunStore, + Pipeline, + PipelineDefinitionException, + PipelineException, + RetryPolicy, + RunStatus, + Schedule, + load_definition, + validate_definition, + worker, +) +from automation_file.pipeline.substitution import render, render_key + +NAME_RULE = "expected a non-empty string without whitespace, '.', '$', '{' or '}'" +SYNTAX = "(use ${params.} or ${tasks..result})" + +DOCUMENT: dict[str, Any] = { + "schema_version": 1, + "name": "daily-report", + "max_workers": 4, + "schedule": {"cron": "0 2 * * *", "timezone": "Asia/Taipei"}, + "params": {"date": "2026-10-08"}, + "tasks": { + "download": { + "action": [ + "FA_storage_copy", + {"source": "s3://input/report.csv", "target": "local:///tmp/report.csv"}, + ] + }, + "verify": { + "action": [ + "FA_storage_verify", + {"uri": "local:///tmp/report.csv", "expected": "sha256:abc"}, + ], + "depends_on": ["download"], + "retry": { + "max_attempts": 3, + "backoff": 2, + "backoff_cap": 30, + "on": ["StorageTransientException"], + }, + "timeout": 120, + "when": "on_success", + "idempotency_key": "verify-${params.date}", + }, + }, +} + +YAML_TEXT = """\ +schema_version: 1 +name: daily-report +max_workers: 4 +schedule: {cron: "0 2 * * *", timezone: Asia/Taipei} +params: {date: "2026-10-08"} +tasks: + download: + action: ["FA_storage_copy", {"source": "s3://input/report.csv", "target": "local:///tmp/report.csv"}] + verify: + action: ["FA_storage_verify", {"uri": "local:///tmp/report.csv", "expected": "sha256:abc"}] + depends_on: [download] + retry: {max_attempts: 3, backoff: 2, backoff_cap: 30, on: [StorageTransientException]} + timeout: 120 + when: on_success + idempotency_key: "verify-${params.date}" +""" + +_DELETE = object() + + +def changed(*path: Any, to: Any = _DELETE) -> dict[str, Any]: + """Return the example document with the entry at ``path`` replaced (or removed).""" + document = copy.deepcopy(DOCUMENT) + node: Any = document + for key in path[:-1]: + node = node[key] + if to is _DELETE: + del node[path[-1]] + else: + node[path[-1]] = to + return document + + +# ---------------------------------------------------------------------- validation + + +def test_the_example_definition_is_valid() -> None: + assert validate_definition(DOCUMENT) == [] + assert validate_definition(copy.deepcopy(DOCUMENT)) == [] + + +def test_a_minimal_definition_is_valid() -> None: + assert ( + validate_definition( + { + "schema_version": 1, + "name": "tiny", + "tasks": {"only": {"action": ["FA_storage_schemes"]}}, + } + ) + == [] + ) + + +def test_task_ids_may_be_written_in_any_script() -> None: + document = changed("tasks", "下載", to={"action": ["FA_storage_schemes"]}) + document["tasks"]["驗證"] = {"action": ["FA_storage_schemes"], "depends_on": ["下載"]} + assert validate_definition(document) == [] + assert [task.task_id for task in Pipeline.from_dict(document).tasks][-2:] == ["下載", "驗證"] + + +@pytest.mark.parametrize( + "document,problem", + [ + ([], "document: expected a mapping, got list"), + (None, "document: expected a mapping, got NoneType"), + ("daily-report.yaml", "document: expected a mapping, got str"), + (changed("schema_version"), "schema_version: required (supported: 1)"), + (changed("schema_version", to=2), "schema_version: unsupported version 2 (supported: 1)"), + ( + changed("schema_version", to="1"), + "schema_version: unsupported version '1' (supported: 1)", + ), + ( + changed("schema_version", to=True), + "schema_version: unsupported version True (supported: 1)", + ), + ], +) +def test_a_document_of_an_unknown_shape_or_version_is_rejected(document: Any, problem: str) -> None: + assert validate_definition(document) == [problem] + + +def test_an_unknown_version_hides_every_other_finding() -> None: + document = changed("schema_version", to=2) + document["name"] = "" + document["tasks"] = [] + assert validate_definition(document) == ["schema_version: unsupported version 2 (supported: 1)"] + + +@pytest.mark.parametrize( + "document,problem", + [ + (changed("name"), "name: required"), + (changed("name", to=" "), "name: expected a non-empty string, got ' '"), + (changed("name", to=5), "name: expected a non-empty string, got 5"), + (changed("description", to=5), "description: expected a string, got int"), + (changed("max_workers", to=0), "max_workers: expected an integer >= 1, got 0"), + (changed("max_workers", to="4"), "max_workers: expected an integer >= 1, got '4'"), + (changed("max_workers", to=True), "max_workers: expected an integer >= 1, got True"), + (changed("max_workers", to=2.5), "max_workers: expected an integer >= 1, got 2.5"), + (changed("owner", to="ops"), "owner: unknown key"), + (changed("schedule", to="0 2 * * *"), "schedule: expected a mapping, got str"), + (changed("schedule", to=None), "schedule: expected a mapping, got NoneType"), + (changed("schedule", to={}), "schedule.cron: required"), + ( + changed("schedule", "cron", to="every day"), + "schedule.cron: expected 5 fields, got 2: 'every day'", + ), + (changed("schedule", "cron", to="61 2 * * *"), "schedule.cron: value 61 outside [0,59]"), + ( + changed("schedule", "cron", to=5), + "schedule.cron: expected a cron expression, got int", + ), + ( + changed("schedule", "timezone", to=""), + "schedule.timezone: expected a time zone name, got ''", + ), + (changed("schedule", "jitter", to=1), "schedule.jitter: unknown key"), + (changed("params", to=["date"]), "params: expected a mapping, got list"), + ( + changed("params", to={"a b": 1, "fine": 2}), + f"params.a b: invalid parameter name, {NAME_RULE}", + ), + ( + changed("params", "date", to=datetime.date(2026, 10, 8)), + "params.date: expected a JSON value, got date (quote dates and times in YAML)", + ), + (changed("tasks"), "tasks: required"), + (changed("tasks", to=[]), "tasks: expected a mapping of task ID to task, got list"), + (changed("tasks", to={}), "tasks: at least one task is required"), + ( + changed("tasks", "bad id", to={"action": ["FA_x"]}), + f"tasks.bad id: invalid task ID, {NAME_RULE}", + ), + ( + changed("tasks", "a.b", to={"action": ["FA_x"]}), + f"tasks.a.b: invalid task ID, {NAME_RULE}", + ), + (changed("tasks", 7, to={"action": ["FA_x"]}), f"tasks.7: invalid task ID, {NAME_RULE}"), + (changed("tasks", "verify", to="FA_x"), "tasks.verify: expected a mapping, got str"), + (changed("tasks", "verify", "priority", to=1), "tasks.verify.priority: unknown key"), + ], +) +def test_a_problem_in_the_header_or_the_task_list_is_reported_with_its_path( + document: Any, problem: str +) -> None: + assert validate_definition(document) == [problem] + + +@pytest.mark.parametrize( + "path,value,problem", + [ + ( + ("action",), + "FA_x", + "tasks.verify.action: expected [name], [name, {kwargs}] or [name, [args]], got str", + ), + ( + ("action",), + [], + "tasks.verify.action: expected [name], [name, {kwargs}] or [name, [args]]," + " got an empty list", + ), + (("action",), [5], "tasks.verify.action[0]: expected an action name, got int"), + (("action",), [""], "tasks.verify.action[0]: expected an action name, got str"), + ( + ("action",), + ["FA_x", "y"], + "tasks.verify.action[1]: expected a mapping or a list of arguments, got str", + ), + ( + ("action",), + ["FA_x", {}, {}], + "tasks.verify.action: expected a name and at most one argument set, got 3", + ), + ( + ("action",), + ["FA_x", {3: "x"}], + "tasks.verify.action[1]: argument name 3 is not a string", + ), + ( + ("action",), + ["FA_x", {"since": datetime.date(2026, 1, 1)}], + "tasks.verify.action[1].since: expected a JSON value, got date" + " (quote dates and times in YAML)", + ), + ( + ("action",), + ["FA_x", {"a": "${tasks.nope.result}"}], + "tasks.verify.action[1].a: ${tasks.nope.result}:" + " 'nope' is not an upstream task (see depends_on)", + ), + ( + ("action",), + ["FA_x", {"a": "see ${tasks.download.result}"}], + "tasks.verify.action[1].a: ${tasks.download.result} must be the whole string", + ), + ( + ("action",), + ["FA_x", [{"deep": ["${params.}"]}]], + f"tasks.verify.action[1][0].deep[0]: malformed placeholder ${{params.}} {SYNTAX}", + ), + ( + ("action",), + ["FA_x", {"a": "${tasks.download}"}], + f"tasks.verify.action[1].a: malformed placeholder ${{tasks.download}} {SYNTAX}", + ), + ( + ("depends_on",), + "download", + "tasks.verify.depends_on: expected a list of task IDs, got str", + ), + (("depends_on",), ["x"], "tasks.verify.depends_on[0]: unknown task 'x'"), + ( + ("depends_on",), + ["download", 3], + "tasks.verify.depends_on[1]: expected a task ID, got int", + ), + (("depends_on",), ["verify"], "tasks.verify.depends_on[0]: a task cannot depend on itself"), + ( + ("depends_on",), + ["download", "download"], + "tasks.verify.depends_on[1]: 'download' is listed twice", + ), + (("retry",), 3, "tasks.verify.retry: expected a mapping, got int"), + ( + ("retry", "max_attempts"), + 0, + "tasks.verify.retry.max_attempts: expected an integer >= 1, got 0", + ), + ( + ("retry", "max_attempts"), + 1.5, + "tasks.verify.retry.max_attempts: expected an integer >= 1, got 1.5", + ), + ( + ("retry", "backoff"), + -1, + "tasks.verify.retry.backoff: expected a number of seconds >= 0, got -1", + ), + ( + ("retry", "backoff_cap"), + "30", + "tasks.verify.retry.backoff_cap: expected a number of seconds >= 0, got '30'", + ), + ( + ("retry", "on"), + "StorageTransientException", + "tasks.verify.retry.on: expected a list of exception names, got str", + ), + (("retry", "on"), ["Exception"], "tasks.verify.retry.on[0]: unknown exception 'Exception'"), + ( + ("retry", "on"), + ["OSError", "ValueError"], + "tasks.verify.retry.on[1]: unknown exception 'ValueError'", + ), + (("retry", "on"), [3], "tasks.verify.retry.on[0]: unknown exception 3"), + (("retry", "jitter"), 1, "tasks.verify.retry.jitter: unknown key"), + ( + ("retry",), + {"max_attempts": 2, True: ["OSError"]}, + 'tasks.verify.retry.True: unknown key (YAML reads a bare on as true; write "on")', + ), + (("timeout",), 0, "tasks.verify.timeout: expected a number of seconds > 0, got 0"), + (("timeout",), "120", "tasks.verify.timeout: expected a number of seconds > 0, got '120'"), + ( + ("timeout",), + float("inf"), + "tasks.verify.timeout: expected a number of seconds > 0, got inf", + ), + ( + ("when",), + "sometimes", + "tasks.verify.when: expected one of on_success, on_failure, always, got 'sometimes'", + ), + ( + ("when",), + None, + "tasks.verify.when: expected one of on_success, on_failure, always, got None", + ), + ( + ("idempotency_key",), + "", + "tasks.verify.idempotency_key: expected a non-empty string, got ''", + ), + ( + ("idempotency_key",), + 5, + "tasks.verify.idempotency_key: expected a non-empty string, got 5", + ), + ( + ("idempotency_key",), + "k-${tasks.download.result}", + "tasks.verify.idempotency_key: ${tasks.download.result}:" + " only ${params.} can be used here", + ), + ( + ("idempotency_key",), + "k-${params.a b}", + f"tasks.verify.idempotency_key: malformed placeholder ${{params.a b}} {SYNTAX}", + ), + ], +) +def test_a_problem_in_a_task_is_reported_with_its_path( + path: tuple[str, ...], value: Any, problem: str +) -> None: + assert validate_definition(changed("tasks", "verify", *path, to=value)) == [problem] + + +def test_a_task_without_an_action_is_reported() -> None: + assert validate_definition(changed("tasks", "verify", "action")) == [ + "tasks.verify.action: required" + ] + + +def test_a_cycle_is_reported_with_its_tasks() -> None: + document = changed("tasks", "download", "depends_on", to=["verify"]) + assert validate_definition(document) == [ + "tasks: dependency cycle: download -> verify -> download" + ] + + +def test_every_problem_is_reported_at_once() -> None: + document = changed("name", to="") + document["max_workers"] = 0 + document["tasks"]["download"]["action"] = ["FA_x", {"a": "${params.}"}] + document["tasks"]["verify"]["depends_on"] = ["missing"] + document["tasks"]["verify"]["retry"] = {"max_attempts": 0, "on": ["Nope"]} + document["tasks"]["verify"]["timeout"] = -5 + assert validate_definition(document) == [ + "name: expected a non-empty string, got ''", + "max_workers: expected an integer >= 1, got 0", + "tasks.verify.retry.max_attempts: expected an integer >= 1, got 0", + "tasks.verify.retry.on[0]: unknown exception 'Nope'", + "tasks.verify.timeout: expected a number of seconds > 0, got -5", + "tasks.verify.depends_on[0]: unknown task 'missing'", + f"tasks.download.action[1].a: malformed placeholder ${{params.}} {SYNTAX}", + ] + + +# ---------------------------------------------------------------------- the schema + + +def test_the_schema_is_a_json_schema_document_for_version_1() -> None: + assert SCHEMA_VERSION == 1 + assert json.loads(json.dumps(PIPELINE_SCHEMA)) == PIPELINE_SCHEMA + assert PIPELINE_SCHEMA["$schema"] == "https://json-schema.org/draft/2020-12/schema" + assert PIPELINE_SCHEMA["type"] == "object" + assert PIPELINE_SCHEMA["required"] == ["schema_version", "name", "tasks"] + assert PIPELINE_SCHEMA["properties"]["schema_version"] == {"const": 1} + assert PIPELINE_SCHEMA["additionalProperties"] is False + + +def test_the_schema_names_the_keys_the_validator_accepts() -> None: + definitions = PIPELINE_SCHEMA["$defs"] + assert set(PIPELINE_SCHEMA["properties"]) == set(DOCUMENT) | {"description"} + assert set(definitions["task"]["properties"]) == set(DOCUMENT["tasks"]["verify"]) + assert definitions["task"]["required"] == ["action"] + assert set(definitions["retry"]["properties"]) == set(DOCUMENT["tasks"]["verify"]["retry"]) + assert set(definitions["schedule"]["properties"]) == set(DOCUMENT["schedule"]) + assert definitions["task"]["properties"]["when"]["enum"] == [ + "on_success", + "on_failure", + "always", + ] + assert definitions["retry"]["properties"]["on"]["items"]["enum"] == list(RETRYABLE_EXCEPTIONS) + assert definitions["action"]["maxItems"] == 2 + for part in ("task", "retry", "schedule"): + assert definitions[part]["additionalProperties"] is False + every_reference = json.dumps(PIPELINE_SCHEMA).count('"$ref"') + assert every_reference == 4 + for name in ("schedule", "action", "retry", "task"): + assert f'"#/$defs/{name}"' in json.dumps(PIPELINE_SCHEMA) + + +def test_the_exceptions_a_definition_may_name() -> None: + project = { + name + for name, member in vars(exceptions).items() + if isinstance(member, type) and issubclass(member, FileAutomationException) + } + assert set(RETRYABLE_EXCEPTIONS) == project | {"TimeoutError", "ConnectionError", "OSError"} + assert RETRYABLE_EXCEPTIONS["StorageTransientException"] is StorageTransientException + assert RETRYABLE_EXCEPTIONS["OSError"] is OSError + assert "Exception" not in RETRYABLE_EXCEPTIONS + assert "ValueError" not in RETRYABLE_EXCEPTIONS + assert list(RETRYABLE_EXCEPTIONS) == sorted(RETRYABLE_EXCEPTIONS) + with pytest.raises(TypeError): + RETRYABLE_EXCEPTIONS["Exception"] = Exception # type: ignore[index] + + +# ---------------------------------------------------------------------- from_dict and to_dict + + +def test_from_dict_builds_the_pipeline() -> None: + pipeline = Pipeline.from_dict(DOCUMENT) + assert pipeline.name == "daily-report" + assert pipeline.description == "" + assert pipeline.max_workers == 4 + assert pipeline.schedule == Schedule(cron="0 2 * * *", timezone="Asia/Taipei") + assert pipeline.params == {"date": "2026-10-08"} + download, verify = pipeline.tasks + assert download.task_id == "download" + assert download.action_name == "FA_storage_copy" + assert download.depends_on == () + assert download.retry == RetryPolicy() + assert download.timeout is None + assert download.when == "on_success" + assert download.idempotency_key is None + assert verify.depends_on == ("download",) + assert verify.retry == RetryPolicy( + max_attempts=3, backoff_base=2, backoff_cap=30, retry_on=(StorageTransientException,) + ) + assert verify.timeout == 120 + assert verify.idempotency_key == "verify-${params.date}" + assert pipeline.problems() == [] + + +def test_from_dict_fills_in_the_defaults() -> None: + pipeline = Pipeline.from_dict( + { + "schema_version": 1, + "name": "tiny", + "description": "one task", + "tasks": {"only": {"action": ["FA_x"], "retry": {"max_attempts": 2}}}, + } + ) + assert pipeline.description == "one task" + assert pipeline.max_workers == 4 + assert pipeline.schedule is None + assert pipeline.params == {} + assert pipeline.tasks[0].retry == RetryPolicy(max_attempts=2) + assert pipeline.tasks[0].retry.retry_on is DEFAULT_RETRY_ON + + +def test_an_empty_retry_list_never_retries() -> None: + document = changed("tasks", "verify", "retry", "on", to=[]) + assert validate_definition(document) == [] + policy = Pipeline.from_dict(document).tasks[1].retry + assert policy.retry_on == () + assert policy.retries(StorageTransientException("throttled")) is False + + +def test_from_dict_raises_with_every_problem() -> None: + document = changed("tasks", "verify", "depends_on", to=["x"]) + document["max_workers"] = 0 + with pytest.raises(PipelineDefinitionException) as raised: + Pipeline.from_dict(document) + assert raised.value.problems == ( + "max_workers: expected an integer >= 1, got 0", + "tasks.verify.depends_on[0]: unknown task 'x'", + ) + assert str(raised.value) == "; ".join(raised.value.problems) + assert isinstance(raised.value, PipelineException) + + +def test_a_definition_round_trip() -> None: + canonical = changed("tasks", "verify", "when") # on_success is the default and is left out + written = Pipeline.from_dict(DOCUMENT).to_dict() + assert written == canonical + assert list(written) == ["schema_version", "name", "max_workers", "schedule", "params", "tasks"] + assert validate_definition(written) == [] + assert Pipeline.from_dict(written).to_dict() == written + assert json.loads(json.dumps(written)) == written + + +def test_to_dict_writes_what_differs_from_the_defaults() -> None: + pipeline = Pipeline("built", description="made in Python", max_workers=2) + pipeline.task("first", ["FA_a"]) + pipeline.task( + "second", + ["FA_b", ["${tasks.first.result}"]], + depends_on="first", + retry=RetryPolicy(max_attempts=2), + when="always", + ) + pipeline.task( + "third", + ["FA_c", {"x": 1}], + depends_on=["first", "second"], + retry=RetryPolicy(max_attempts=4, backoff_base=0.5, retry_on=(OSError, TimeoutError)), + timeout=1.5, + when="on_failure", + idempotency_key="third", + ) + assert pipeline.to_dict() == { + "schema_version": 1, + "name": "built", + "description": "made in Python", + "max_workers": 2, + "tasks": { + "first": {"action": ["FA_a"]}, + "second": { + "action": ["FA_b", ["${tasks.first.result}"]], + "depends_on": ["first"], + "retry": {"max_attempts": 2}, + "when": "always", + }, + "third": { + "action": ["FA_c", {"x": 1}], + "depends_on": ["first", "second"], + "retry": {"max_attempts": 4, "backoff": 0.5, "on": ["OSError", "TimeoutError"]}, + "timeout": 1.5, + "when": "on_failure", + "idempotency_key": "third", + }, + }, + } + assert validate_definition(pipeline.to_dict()) == [] + assert Pipeline.from_dict(pipeline.to_dict()).to_dict() == pipeline.to_dict() + + +def test_to_dict_hands_out_a_copy() -> None: + pipeline = Pipeline.from_dict(DOCUMENT) + written = pipeline.to_dict() + written["tasks"]["download"]["action"][1]["source"] = "changed" + written["params"]["date"] = "changed" + assert pipeline.to_dict() == changed("tasks", "verify", "when") + + +@pytest.mark.parametrize( + "options,problem", + [ + ( + {"work": lambda ctx: None}, + "tasks.t: a Python callable cannot be written to a definition", + ), + ( + {"when": lambda ctx: True}, + "tasks.t.when: a Python callable cannot be written to a definition", + ), + ( + {"retry": RetryPolicy(max_attempts=2, retry_on=(KeyError,))}, + "tasks.t.retry.on: KeyError cannot be named in a definition", + ), + ], +) +def test_to_dict_refuses_what_a_document_cannot_hold(options: dict[str, Any], problem: str) -> None: + pipeline = Pipeline("python-only") + work = options.pop("work", ["FA_a"]) + pipeline.task("t", work, **options) + with pytest.raises(PipelineDefinitionException) as raised: + pipeline.to_dict() + assert raised.value.problems == (problem,) + + +def test_a_schedule_without_a_time_zone() -> None: + document = changed("schedule", to={"cron": "*/5 * * * *"}) + pipeline = Pipeline.from_dict(document) + assert pipeline.schedule == Schedule("*/5 * * * *") + assert pipeline.schedule.timezone is None + assert pipeline.to_dict()["schedule"] == {"cron": "*/5 * * * *"} + + +# ---------------------------------------------------------------------- files + + +def test_a_yaml_and_a_json_file_give_the_same_pipeline(tmp_path: Path) -> None: + yaml_file = tmp_path / "daily-report.yaml" + yml_file = tmp_path / "daily-report.YML" + json_file = tmp_path / "daily-report.json" + yaml_file.write_text(YAML_TEXT, encoding="utf-8") + yml_file.write_text(YAML_TEXT, encoding="utf-8") + json_file.write_text(json.dumps(DOCUMENT), encoding="utf-8") + assert load_definition(yaml_file) == DOCUMENT + assert load_definition(str(yml_file)) == DOCUMENT + assert load_definition(json_file) == DOCUMENT + expected = Pipeline.from_dict(DOCUMENT).to_dict() + for path in (yaml_file, yml_file, json_file): + assert Pipeline.from_file(path).to_dict() == expected + assert Pipeline.from_file(str(yaml_file)).schedule == Schedule("0 2 * * *", "Asia/Taipei") + + +def test_a_bare_on_in_a_yaml_file_is_the_retry_key(tmp_path: Path) -> None: + import yaml + + assert ( + True in yaml.safe_load(YAML_TEXT)["tasks"]["verify"]["retry"] + ) # what YAML 1.1 makes of it + for spelling in ("on", '"on"', "'on'"): + path = tmp_path / "retry.yaml" + path.write_text( + YAML_TEXT.replace("on: [Storage", f"{spelling}: [Storage"), encoding="utf-8" + ) + retry = load_definition(path)["tasks"]["verify"]["retry"] + assert list(retry) == ["max_attempts", "backoff", "backoff_cap", "on"] + assert Pipeline.from_file(path).tasks[1].retry.retry_on == (StorageTransientException,) + shared = tmp_path / "shared.yaml" + shared.write_text( + "schema_version: 1\nname: shared\ntasks:\n" + " a: &task {action: [FA_a], retry: {max_attempts: 2, on: [OSError]}}\n" + " b: *task\n", + encoding="utf-8", + ) + assert [task.retry.retry_on for task in Pipeline.from_file(shared).tasks] == [(OSError,)] * 2 + + +def test_a_file_may_use_yaml_anchors_and_any_script(tmp_path: Path) -> None: + path = tmp_path / "anchors.yaml" + path.write_text( + "schema_version: 1\n" + "name: 每日報表\n" + "tasks:\n" + " 下載: &base {action: [FA_storage_schemes], timeout: 5}\n" + " 備份: *base\n", + encoding="utf-8", + ) + pipeline = Pipeline.from_file(path) + assert pipeline.name == "每日報表" + assert [(task.task_id, task.timeout) for task in pipeline.tasks] == [("下載", 5), ("備份", 5)] + + +@pytest.mark.parametrize( + "name,content,message", + [ + ("pipeline.txt", YAML_TEXT, "expected a .yaml, .yml or .json file"), + ("pipeline", YAML_TEXT, "expected a .yaml, .yml or .json file"), + ("broken.yaml", "name: [unclosed\n", "invalid YAML: "), + ("two.yaml", "a: 1\n---\nb: 2\n", "invalid YAML: "), + ("broken.json", '{"name": }', "invalid JSON: Expecting value at line 1, column 10"), + ("binary.yaml", b"\xff\xfe\x00name", "cannot read the definition"), + ( + "twice.yaml", + "schema_version: 1\nname: x\ntasks:\n load: {action: [FA_a]}\n load: {action: [FA_b]}\n", + "duplicate key 'load' at line 5", + ), + ( + "twice.json", + '{"schema_version": 1, "tasks": {"load": {}, "load": {}}}', + "duplicate key 'load'", + ), + ("endless.yaml", "a: &a [*a]\n", "the definition is too large or refers to itself"), + ( + "bomb.yaml", + "a: &a [x, x, x, x, x, x, x, x, x, x]\n" + + "".join( + f"{name}: &{name} [{', '.join([f'*{previous}'] * 10)}]\n" + for previous, name in zip("abcde", "bcdef", strict=True) + ), + "the definition is too large or refers to itself", + ), + ], +) +def test_a_file_that_cannot_be_loaded_says_why( + tmp_path: Path, name: str, content: str | bytes, message: str +) -> None: + path = tmp_path / name + if isinstance(content, bytes): + path.write_bytes(content) + else: + path.write_text(content, encoding="utf-8") + with pytest.raises(PipelineDefinitionException) as raised: + load_definition(path) + assert str(raised.value).startswith(f"{path}: {message}") + with pytest.raises(PipelineDefinitionException): + Pipeline.from_file(path) + + +def test_a_file_nested_too_deeply_is_refused( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + import yaml + + def too_deep(*_args: object, **_kwargs: object) -> None: + raise RecursionError("maximum recursion depth exceeded") + + json_file = tmp_path / "deep.json" + json_file.write_text("[" * 100_000 + "]" * 100_000, encoding="utf-8") + yaml_file = tmp_path / "deep.yaml" + yaml_file.write_text("[[[[]]]]", encoding="utf-8") + monkeypatch.setattr(yaml, "compose", too_deep) # what PyYAML's composer does past the limit + for path in (json_file, yaml_file): + with pytest.raises(PipelineDefinitionException) as raised: + load_definition(path) + assert str(raised.value) == f"{path}: the definition is nested too deeply" + + +def test_a_missing_file_says_so(tmp_path: Path) -> None: + with pytest.raises(PipelineDefinitionException, match="cannot read the definition"): + load_definition(tmp_path / "absent.yaml") + + +def test_an_invalid_yaml_file_names_the_line(tmp_path: Path) -> None: + path = tmp_path / "broken.yaml" + path.write_text("schema_version: 1\nname: x\ntasks:\n a: {action: [FA_a\n", encoding="utf-8") + with pytest.raises( + PipelineDefinitionException, match=r"invalid YAML: .* at line \d+, column \d+" + ): + load_definition(path) + + +def test_an_empty_file_is_not_a_definition(tmp_path: Path) -> None: + path = tmp_path / "empty.yaml" + path.write_text("", encoding="utf-8") + assert load_definition(path) is None + with pytest.raises(PipelineDefinitionException, match="document: expected a mapping"): + Pipeline.from_file(path) + + +def test_an_unquoted_yaml_date_is_pointed_out(tmp_path: Path) -> None: + path = tmp_path / "dated.yaml" + path.write_text( + YAML_TEXT.replace('{date: "2026-10-08"}', "{date: 2026-10-08}"), encoding="utf-8" + ) + with pytest.raises(PipelineDefinitionException) as raised: + Pipeline.from_file(path) + assert raised.value.problems == ( + "params.date: expected a JSON value, got date (quote dates and times in YAML)", + ) + + +# ---------------------------------------------------------------------- a definition at work + + +def test_a_definition_runs_with_its_retry_and_its_placeholders( + monkeypatch: pytest.MonkeyPatch, +) -> None: + monkeypatch.setattr(worker, "pause", lambda _token, _seconds: False) + calls: list[str] = [] + + def fetch(source: str) -> dict[str, str]: + calls.append(source) + if len(calls) < 3: + raise StorageTransientException("throttled") + return {"path": f"/tmp/{source.rsplit('/', 1)[-1]}"} + + registry = ActionRegistry() + registry.register("T_fetch", fetch) + registry.register("T_report", lambda fetched, day: f"{fetched['path']} for {day}") + pipeline = Pipeline.from_dict( + { + "schema_version": 1, + "name": "from-a-document", + "params": {"date": "2026-01-01"}, + "tasks": { + "fetch": { + "action": ["T_fetch", {"source": "s3://in/${params.date}.csv"}], + "retry": {"max_attempts": 3, "on": ["StorageTransientException"]}, + }, + "report": { + "action": ["T_report", ["${tasks.fetch.result}", "${params.date}"]], + "depends_on": ["fetch"], + }, + }, + }, + registry=registry, + ) + run = pipeline.run(params={"date": "2026-10-08"}, store=MemoryRunStore(), bus=EventBus()) + assert run.status is RunStatus.SUCCEEDED + assert calls == ["s3://in/2026-10-08.csv"] * 3 + assert run.tasks["fetch"].attempts == 3 + assert run.tasks["report"].result == "/tmp/2026-10-08.csv for 2026-10-08" + + +# ---------------------------------------------------------------------- substitution + + +def test_render_replaces_parameters_and_results() -> None: + params = {"date": "2026-10-08", "limit": 20, "flag": False} + results = {"load": {"rows": 3}} + assert render("${params.limit}", params, results) == 20 + assert render("${params.flag}", params, results) is False + assert render("limit=${params.limit} on ${params.date}", params, results) == ( + "limit=20 on 2026-10-08" + ) + assert render("${params.date}${params.limit}", params, results) == "2026-10-0820" + assert render("${tasks.load.result}", params, results) is results["load"] + assert render("${tasks.absent.result}", params, results) is None + assert render(["${params.limit}", {"k": "${params.date}"}, 7, None], params, results) == [ + 20, + {"k": "2026-10-08"}, + 7, + None, + ] + assert render({"${params.date}": "key stays"}, params, results) == { + "${params.date}": "key stays" + } + assert render("plain ${HOME} ${env:X} $5", params, results) == "plain ${HOME} ${env:X} $5" + + +def test_render_raises_for_what_it_cannot_resolve() -> None: + with pytest.raises(PipelineException, match="unknown parameter 'absent'"): + render("x-${params.absent}", {}, {}) + with pytest.raises(PipelineException, match="malformed placeholder"): + render("${tasks.load}", {}, {"load": 1}) + + +def test_render_key_is_always_text() -> None: + assert render_key("batch-${params.number}", {"number": 5}) == "batch-5" + assert render_key("${params.number}", {"number": 5}) == "5" + assert render_key("fixed", {}) == "fixed" diff --git a/tests/test_pipeline_events.py b/tests/test_pipeline_events.py new file mode 100644 index 0000000..93c72df --- /dev/null +++ b/tests/test_pipeline_events.py @@ -0,0 +1,443 @@ +"""What a pipeline run publishes: the events, their order, payload and correlation ID.""" + +from __future__ import annotations + +import threading +import time + +import pytest + +from automation_file.core.progress import CancellationToken, CancelledException +from automation_file.events import ( + PAYLOAD_KEYS, + Event, + EventBus, + PipelineCompleted, + PipelineFailed, + PipelineStarted, + Severity, + TaskCompleted, + TaskFailed, + TaskStarted, + actor_scope, + correlation_scope, + current_actor, + current_correlation_id, + event_bus, +) +from automation_file.pipeline import ( + MemoryRunStore, + Pipeline, + RetryPolicy, + RunStatus, + TaskContext, + worker, +) + +WAIT = 5.0 + + +@pytest.fixture +def bus() -> EventBus: + return EventBus() + + +@pytest.fixture +def events(bus: EventBus) -> list[Event]: + """Every event published on the test's bus, in order.""" + received: list[Event] = [] + bus.subscribe(received.append) + return received + + +@pytest.fixture +def store() -> MemoryRunStore: + return MemoryRunStore() + + +def summary(events: list[Event]) -> list[tuple[str, str | None, int | None, str]]: + return [ + ( + event.type, + event.payload.get("task"), + event.payload.get("attempt"), + event.payload["status"], + ) + for event in events + ] + + +def test_a_successful_run_publishes_its_events_in_order( + bus: EventBus, events: list[Event], store: MemoryRunStore +) -> None: + pipeline = Pipeline("daily") + pipeline.task("load", lambda ctx: 1) + pipeline.task("report", lambda ctx: 2, depends_on=["load"]) + run = pipeline.run(store=store, bus=bus) + assert summary(events) == [ + ("pipeline.started", None, None, "running"), + ("task.started", "load", 1, "running"), + ("task.completed", "load", 1, "succeeded"), + ("task.started", "report", 1, "running"), + ("task.completed", "report", 1, "succeeded"), + ("pipeline.completed", None, None, "succeeded"), + ] + assert [type(event) for event in events] == [ + PipelineStarted, + TaskStarted, + TaskCompleted, + TaskStarted, + TaskCompleted, + PipelineCompleted, + ] + assert {event.correlation_id for event in events} == {run.run_id} + assert {event.source for event in events} == {"pipeline"} + assert {event.severity for event in events} == {Severity.INFO} + assert {event.actor for event in events} == {current_actor()} + assert [event.subject for event in events] == [ + "daily running", + "daily/load running (attempt 1)", + "daily/load succeeded (attempt 1)", + "daily/report running (attempt 1)", + "daily/report succeeded (attempt 1)", + "daily succeeded", + ] + + +def test_the_payload_uses_the_shared_keys( + bus: EventBus, events: list[Event], store: MemoryRunStore +) -> None: + def boom(_ctx: TaskContext) -> None: + raise ValueError("boom") + + pipeline = Pipeline("payloads") + pipeline.task("fine", lambda ctx: 1) + pipeline.task("broken", boom, depends_on=["fine"]) + run = pipeline.run(store=store, bus=bus) + for event in events: + assert set(event.payload) <= set(PAYLOAD_KEYS) + assert event.payload["pipeline"] == "payloads" + assert event.payload["run_id"] == run.run_id + assert isinstance(event.to_dict()["payload"], dict) + by_type = {event.type: event for event in events} + assert set(by_type["pipeline.started"].payload) == {"pipeline", "run_id", "status"} + assert set(by_type["task.started"].payload) == { + "pipeline", + "run_id", + "task", + "attempt", + "status", + } + completed = by_type["task.completed"].payload + assert set(completed) == {"pipeline", "run_id", "task", "attempt", "status", "duration_ms"} + assert completed["duration_ms"] >= 0 + failed = by_type["task.failed"] + assert failed.payload["error"] == "ValueError: boom" + assert failed.payload["status"] == "failed" + assert failed.payload["duration_ms"] >= 0 + assert failed.severity is Severity.ERROR + ended = by_type["pipeline.failed"] + assert isinstance(ended, PipelineFailed) + assert ended.payload["status"] == "failed" + assert ended.payload["error"] == "did not succeed: broken" + assert ended.payload["duration_ms"] >= 0 + assert ended.severity is Severity.ERROR + assert "pipeline.completed" not in by_type + + +def test_each_attempt_has_its_own_events( + bus: EventBus, events: list[Event], store: MemoryRunStore, monkeypatch: pytest.MonkeyPatch +) -> None: + monkeypatch.setattr(worker, "pause", lambda _token, _seconds: False) + + def down(_ctx: TaskContext) -> None: + raise ConnectionError("refused") + + pipeline = Pipeline("attempts") + pipeline.task("down", down, retry=RetryPolicy(max_attempts=3)) + pipeline.run(store=store, bus=bus) + assert summary(events) == [ + ("pipeline.started", None, None, "running"), + ("task.started", "down", 1, "running"), + ("task.failed", "down", 1, "retrying"), + ("task.started", "down", 2, "running"), + ("task.failed", "down", 2, "retrying"), + ("task.started", "down", 3, "running"), + ("task.failed", "down", 3, "failed"), + ("pipeline.failed", None, None, "failed"), + ] + failures = [event for event in events if isinstance(event, TaskFailed)] + assert [event.severity for event in failures] == [ + Severity.WARNING, # it will be tried again + Severity.WARNING, + Severity.ERROR, + ] + assert {event.payload["error"] for event in failures} == {"ConnectionError: refused"} + + +def test_a_timeout_is_a_failed_task_event( + bus: EventBus, events: list[Event], store: MemoryRunStore +) -> None: + release = threading.Event() + pipeline = Pipeline("slow") + pipeline.task("stuck", lambda ctx: release.wait(WAIT), timeout=0.05) + try: + pipeline.run(store=store, bus=bus) + finally: + release.set() + assert summary(events) == [ + ("pipeline.started", None, None, "running"), + ("task.started", "stuck", 1, "running"), + ("task.failed", "stuck", 1, "timeout"), + ("pipeline.failed", None, None, "failed"), + ] + assert events[2].payload["error"] == "TimeoutError: no result within 0.05 s" + assert events[2].severity is Severity.ERROR + + +def test_a_cancelled_run_ends_with_a_warning( + bus: EventBus, events: list[Event], store: MemoryRunStore +) -> None: + entered = threading.Event() + + def watches(ctx: TaskContext) -> None: + entered.set() + deadline = time.monotonic() + WAIT + while not ctx.cancel.is_cancelled and time.monotonic() < deadline: + time.sleep(0.002) + ctx.cancel.raise_if_cancelled() + + pipeline = Pipeline("stopped") + pipeline.task("first", watches) + pipeline.task("second", lambda ctx: None, depends_on=["first"]) + run = pipeline.start(store=store, bus=bus) + assert entered.wait(WAIT) + run.cancel() + assert run.wait(WAIT) is True + assert summary(events) == [ + ("pipeline.started", None, None, "running"), + ("task.started", "first", 1, "running"), + ("task.failed", "first", 1, "cancelled"), + ("pipeline.failed", None, None, "cancelled"), + ] + assert events[2].severity is Severity.WARNING + assert events[2].payload["error"] == "CancelledException: operation cancelled" + assert events[3].severity is Severity.WARNING + assert events[3].payload["error"] == "the run was cancelled" + assert {event.correlation_id for event in events} == {run.run_id} + + +def test_tasks_that_do_not_run_publish_nothing( + bus: EventBus, events: list[Event], store: MemoryRunStore +) -> None: + def build() -> Pipeline: + pipeline = Pipeline("quiet") + pipeline.task("once", lambda ctx: "done", idempotency_key="once") + pipeline.task("never", lambda ctx: None, depends_on=["once"], when="on_failure") + pipeline.task("pruned", lambda ctx: None, depends_on=["never"]) + return pipeline + + build().run(store=store, bus=bus) + events.clear() + second = build().run(store=store, bus=bus) + assert second.status is RunStatus.SUCCEEDED + assert second.tasks["once"].reason == "idempotent" + assert summary(events) == [ + ("pipeline.started", None, None, "running"), + ("pipeline.completed", None, None, "succeeded"), + ] + events.clear() + token = CancellationToken() + token.cancel() + build().run(store=store, bus=bus, cancel=token) + assert summary(events) == [ + ("pipeline.started", None, None, "running"), + ("pipeline.failed", None, None, "cancelled"), + ] + events.clear() + build().run(dry_run=True, store=store, bus=bus) + assert events == [] + + +def test_a_condition_that_raises_is_reported( + bus: EventBus, events: list[Event], store: MemoryRunStore +) -> None: + def broken(_ctx: TaskContext) -> bool: + raise LookupError("missing") + + pipeline = Pipeline("conditions") + pipeline.task("guarded", lambda ctx: None, when=broken) + pipeline.run(store=store, bus=bus) + assert summary(events) == [ + ("pipeline.started", None, None, "running"), + ("task.failed", "guarded", 0, "failed"), + ("pipeline.failed", None, None, "failed"), + ] + assert events[1].payload["error"] == "when: LookupError: missing" + + +def test_a_task_cancelled_on_its_own_is_reported( + bus: EventBus, events: list[Event], store: MemoryRunStore +) -> None: + def gives_up(_ctx: TaskContext) -> None: + raise CancelledException("transfer cancelled") + + pipeline = Pipeline("self-cancelled") + pipeline.task("transfer", gives_up) + pipeline.run(store=store, bus=bus) + assert summary(events)[2] == ("task.failed", "transfer", 1, "cancelled") + assert summary(events)[3] == ("pipeline.failed", None, None, "failed") + assert events[3].severity is Severity.ERROR + + +# ---------------------------------------------------------------------- scopes + + +def test_worker_threads_carry_the_correlation_id_and_the_actor( + bus: EventBus, events: list[Event], store: MemoryRunStore +) -> None: + def inspect(_ctx: TaskContext) -> dict[str, object]: + return { + "correlation_id": current_correlation_id(), + "actor": current_actor(), + "event": Event().correlation_id, # what a storage event raised in here would carry + "thread": threading.current_thread().name, + } + + pipeline = Pipeline("scoped", max_workers=2) + pipeline.task("one", inspect) + pipeline.task("two", inspect) + pipeline.task( + "when", inspect, depends_on=["one"], when=lambda ctx: current_actor() == "scheduler" + ) + with actor_scope("scheduler"), correlation_scope("an-outer-scope"): + run = pipeline.run(store=store, bus=bus) + assert current_correlation_id() == "an-outer-scope" # the run's scope is closed again + assert current_correlation_id() is None + for task_id in ("one", "two", "when"): + seen = run.tasks[task_id].result + assert seen["correlation_id"] == run.run_id + assert seen["event"] == run.run_id + assert seen["actor"] == "scheduler" + assert seen["thread"] == f"pipeline-scoped-{task_id}" + assert seen["thread"] != threading.current_thread().name + assert {event.actor for event in events} == {"scheduler"} + assert {event.correlation_id for event in events} == {run.run_id} + assert run.run_id != "an-outer-scope" + + +def test_a_background_run_keeps_the_actor_of_its_caller( + bus: EventBus, events: list[Event], store: MemoryRunStore +) -> None: + pipeline = Pipeline("background") + pipeline.task("who", lambda ctx: (current_actor(), current_correlation_id())) + with actor_scope("mcp"): + run = pipeline.start(store=store, bus=bus) + assert run.wait(WAIT) is True + assert run.tasks["who"].result == ("mcp", run.run_id) + assert {event.actor for event in events} == {"mcp"} + assert {event.correlation_id for event in events} == {run.run_id} + assert [event.type for event in events] == [ + "pipeline.started", + "task.started", + "task.completed", + "pipeline.completed", + ] + + +def test_two_runs_have_two_correlation_ids( + bus: EventBus, events: list[Event], store: MemoryRunStore +) -> None: + pipeline = Pipeline("twice") + pipeline.task("only", lambda ctx: None) + first = pipeline.run(store=store, bus=bus) + second = pipeline.run(store=store, bus=bus) + assert first.run_id != second.run_id + assert len(first.run_id) == 32 + assert [event.correlation_id for event in events] == [first.run_id] * 4 + [second.run_id] * 4 + + +def test_a_resumed_run_reports_under_the_same_correlation_id( + bus: EventBus, events: list[Event], store: MemoryRunStore +) -> None: + def build(fail: bool) -> Pipeline: + def second(_ctx: TaskContext) -> str: + if fail: + raise ValueError("not yet") + return "done" + + pipeline = Pipeline("resumed") + pipeline.task("first", lambda ctx: "kept") + pipeline.task("second", second, depends_on=["first"]) + return pipeline + + run = build(fail=True).run(store=store, bus=bus) + events.clear() + build(fail=False).resume(run.run_id, store=store, bus=bus) + assert summary(events) == [ + ("pipeline.started", None, None, "running"), + ("task.started", "second", 1, "running"), + ("task.completed", "second", 1, "succeeded"), + ("pipeline.completed", None, None, "succeeded"), + ] + assert {event.correlation_id for event in events} == {run.run_id} + + +# ---------------------------------------------------------------------- the bus + + +def test_without_a_bus_the_events_go_to_the_process_wide_one(store: MemoryRunStore) -> None: + received: list[Event] = [] + subscription = event_bus.subscribe(received.append, types=["pipeline.*", "task.*"]) + try: + pipeline = Pipeline("global-bus") + pipeline.task("only", lambda ctx: None) + run = pipeline.run(store=store) + finally: + event_bus.unsubscribe(subscription) + mine = [event for event in received if event.correlation_id == run.run_id] + assert [event.type for event in mine] == [ + "pipeline.started", + "task.started", + "task.completed", + "pipeline.completed", + ] + recent = event_bus.recent(correlation_id=run.run_id) + assert [event.type for event in reversed(recent)] == [event.type for event in mine] + + +def test_a_failing_subscriber_does_not_disturb_the_run( + bus: EventBus, store: MemoryRunStore +) -> None: + def broken(_event: Event) -> None: + raise RuntimeError("subscriber bug") + + bus.subscribe(broken) + pipeline = Pipeline("robust") + pipeline.task("only", lambda ctx: "fine") + run = pipeline.run(store=store, bus=bus) + assert run.status is RunStatus.SUCCEEDED + assert run.tasks["only"].result == "fine" + + +def test_a_subscriber_sees_the_state_the_event_announces( + bus: EventBus, store: MemoryRunStore +) -> None: + seen: list[tuple[str, str, str | None]] = [] + + def look(event: Event) -> None: + stored = store.get_run(event.correlation_id) + task = event.payload.get("task") + status = stored.tasks[task].status.value if task else stored.status.value + seen.append((event.type, status, task)) + + bus.subscribe(look) + pipeline = Pipeline("consistent") + pipeline.task("only", lambda ctx: "fine") + pipeline.run(store=store, bus=bus) + assert seen == [ + ("pipeline.started", "running", None), + ("task.started", "running", "only"), + ("task.completed", "succeeded", "only"), + ("pipeline.completed", "succeeded", None), + ] diff --git a/tests/test_pipeline_imports.py b/tests/test_pipeline_imports.py new file mode 100644 index 0000000..cfe5ea6 --- /dev/null +++ b/tests/test_pipeline_imports.py @@ -0,0 +1,112 @@ +"""What the pipeline package may import at module level, and what it exports. + +``build_default_registry()`` runs while ``automation_file.core.action_executor`` +is still being imported. For the ``FA_pipeline_*`` actions to be registered there, +no module of the package may import the executor at module level (the pipeline +reaches the shared registry inside a function), and nothing outside the standard +library and the project itself: PyYAML is imported where a YAML file is read. +""" + +from __future__ import annotations + +import ast +import sys +from pathlib import Path + +import pytest + +import automation_file.pipeline as pipeline_package + +PACKAGE = Path(pipeline_package.__file__).resolve().parent +MODULES = sorted(PACKAGE.glob("*.py")) +FORBIDDEN = ("automation_file.core.action_executor", "automation_file.ui", "automation_file.server") +PUBLIC_NAMES = [ + "DEFAULT_RETRY_ON", + "PIPELINE_SCHEMA", + "RETRYABLE_EXCEPTIONS", + "SCHEMA_VERSION", + "MemoryRunStore", + "Pipeline", + "PipelineDefinitionException", + "PipelineException", + "PipelineRun", + "RetryPolicy", + "RunStatus", + "RunStore", + "SQLiteRunStore", + "Schedule", + "Task", + "TaskContext", + "TaskRun", + "TaskStatus", + "default_run_store", + "load_definition", + "register_pipeline_ops", + "set_default_run_store", + "validate_definition", +] + + +def _module_level_imports(path: Path) -> list[str]: + tree = ast.parse(path.read_text(encoding="utf-8"), filename=str(path)) + names: list[str] = [] + for node in tree.body: + if isinstance(node, ast.Import): + names.extend(alias.name for alias in node.names) + elif isinstance(node, ast.ImportFrom): + assert node.level == 0, f"{path.name}: relative import" + names.append(node.module or "") + names.extend(f"{node.module}.{alias.name}" for alias in node.names) + return names + + +def _is_allowed(name: str) -> bool: + top_level = name.partition(".")[0] + if top_level == "automation_file": + return not any(name == banned or name.startswith(f"{banned}.") for banned in FORBIDDEN) + return top_level in sys.stdlib_module_names or name == "__future__" + + +def test_the_package_has_its_modules() -> None: + assert {path.name for path in MODULES} >= { + "__init__.py", + "actions.py", + "definition.py", + "errors.py", + "graph.py", + "model.py", + "pipeline.py", + "reporting.py", + "runner.py", + "store.py", + "substitution.py", + "worker.py", + } + + +@pytest.mark.parametrize("path", MODULES, ids=lambda path: path.name) +def test_module_level_imports_are_safe_while_the_registry_is_built(path: Path) -> None: + offending = [name for name in _module_level_imports(path) if not _is_allowed(name)] + assert offending == [] + + +def test_the_guard_recognises_what_it_must_refuse() -> None: + assert _is_allowed("threading") is True + assert _is_allowed("automation_file.events") is True + assert _is_allowed("automation_file.core.progress") is True + assert _is_allowed("automation_file.core.action_executor") is False + assert _is_allowed("automation_file.core.action_executor.executor") is False + assert _is_allowed("automation_file.ui.launcher") is False + assert _is_allowed("yaml") is False + assert _is_allowed("jsonschema") is False + + +def test_the_package_exports_its_public_names() -> None: + assert sorted(pipeline_package.__all__) == sorted(PUBLIC_NAMES) + for name in PUBLIC_NAMES: + assert hasattr(pipeline_package, name), name + + +def test_every_module_starts_with_the_future_import() -> None: + for path in MODULES: + assert "from __future__ import annotations" in path.read_text(encoding="utf-8"), path.name diff --git a/tests/test_pipeline_rejected.py b/tests/test_pipeline_rejected.py new file mode 100644 index 0000000..6f107e1 --- /dev/null +++ b/tests/test_pipeline_rejected.py @@ -0,0 +1,188 @@ +"""The pipeline runtime: what is rejected before anything runs.""" + +from __future__ import annotations + +import re +from collections.abc import Callable +from typing import Any + +import pytest + +from automation_file import ActionRegistry +from automation_file.events import EventBus +from automation_file.pipeline import ( + MemoryRunStore, + Pipeline, + PipelineDefinitionException, + PipelineException, +) + + +def test_a_duplicate_task_id_is_rejected_when_it_is_added() -> None: + pipeline = Pipeline("duplicates") + pipeline.task("load", lambda ctx: None) + with pytest.raises(PipelineDefinitionException) as raised: + pipeline.task("load", lambda ctx: None) + assert raised.value.problems == ("tasks.load: duplicate task ID",) + assert [task.task_id for task in pipeline.tasks] == ["load"] + + +@pytest.mark.parametrize( + "build,problem", + [ + ( + lambda p: p.task("b", lambda ctx: None, depends_on=["missing"]), + "tasks.b.depends_on[0]: unknown task 'missing'", + ), + ( + lambda p: p.task("b", lambda ctx: None, depends_on=["b"]), + "tasks.b.depends_on[0]: a task cannot depend on itself", + ), + ( + lambda p: p.task("b", lambda ctx: None, depends_on=["a", "a"]), + "tasks.b.depends_on[1]: 'a' is listed twice", + ), + ( + lambda p: ( + p.task("b", lambda ctx: None, depends_on=["c"]), + p.task("c", lambda ctx: None, depends_on=["b"]), + ), + "tasks: dependency cycle: b -> c -> b", + ), + ( + lambda p: p.task("b", ["T_echo", {"x": "${tasks.a.result}"}]), + "tasks.b.action[1].x: ${tasks.a.result}: 'a' is not an upstream task (see depends_on)", + ), + ( + lambda p: p.task("b", ["T_echo", ["see ${tasks.a.result}"]], depends_on=["a"]), + "tasks.b.action[1][0]: ${tasks.a.result} must be the whole string", + ), + ( + lambda p: p.task("b", ["T_echo", {"x": "${params.}"}]), + "tasks.b.action[1].x: malformed placeholder ${params.}" + " (use ${params.} or ${tasks..result})", + ), + ( + lambda p: p.task("b", ["T_echo", {"x": "${tasks.a}"}], depends_on=["a"]), + "tasks.b.action[1].x: malformed placeholder ${tasks.a}" + " (use ${params.} or ${tasks..result})", + ), + ( + lambda p: p.task("b", ["T_echo", {"x": "${params.absent}"}]), + "tasks.b.action[1].x: unknown parameter 'absent'", + ), + ( + lambda p: p.task("b", lambda ctx: None, idempotency_key="b-${params.absent}"), + "tasks.b.idempotency_key: unknown parameter 'absent'", + ), + ], +) +def test_a_definition_problem_stops_the_run_before_any_task( + build: Callable[[Pipeline], object], problem: str +) -> None: + store, bus, registry = MemoryRunStore(), EventBus(), ActionRegistry() + registry.register("T_echo", lambda *args, **kwargs: {"args": list(args), "kwargs": kwargs}) + ran: list[str] = [] + pipeline = Pipeline("rejected", registry=registry) + pipeline.task("a", lambda ctx: ran.append("a")) + build(pipeline) + for launch in (pipeline.run, pipeline.start): + with pytest.raises(PipelineDefinitionException) as raised: + launch(store=store, bus=bus) + assert problem in raised.value.problems + assert problem in str(raised.value) + assert ran == [] + assert store.list_runs() == [] + assert bus.recent() == [] + + +def test_every_graph_problem_is_reported_at_once() -> None: + pipeline = Pipeline("several") + pipeline.task("a", lambda ctx: None, depends_on=["nowhere"]) + pipeline.task("b", lambda ctx: None, depends_on=["b"]) + assert pipeline.problems() == [ + "tasks.a.depends_on[0]: unknown task 'nowhere'", + "tasks.b.depends_on[0]: a task cannot depend on itself", + ] + with pytest.raises(PipelineDefinitionException) as raised: + pipeline.validate() + assert len(raised.value.problems) == 2 + + +def test_an_empty_pipeline_does_not_run() -> None: + with pytest.raises(PipelineDefinitionException, match="at least one task"): + Pipeline("empty").run() + + +@pytest.mark.parametrize( + "arguments,problem", + [ + (("bad id", lambda ctx: None), "tasks.bad id: invalid task ID"), + (("a.b", lambda ctx: None), "tasks.a.b: invalid task ID"), + (("", lambda ctx: None), "tasks.: invalid task ID"), + (("t", "FA_storage_schemes"), "tasks.t.action: expected [name]"), + (("t", []), "tasks.t.action: expected [name]"), + (("t", [3]), "tasks.t.action[0]: expected an action name, got int"), + (("t", ["FA_x", "text"]), "tasks.t.action[1]: expected a mapping or a list"), + (("t", ["FA_x", {}, {}]), "tasks.t.action: expected a name and at most one argument set"), + (("t", ["FA_x", {1: "x"}]), "tasks.t.action[1]: argument name 1 is not a string"), + ], +) +def test_a_malformed_task_is_rejected_when_it_is_added( + arguments: tuple[Any, Any], problem: str +) -> None: + with pytest.raises(PipelineDefinitionException, match=re.escape(problem)): + Pipeline("malformed").task(*arguments) + + +@pytest.mark.parametrize( + "options,problem", + [ + ({"timeout": 0}, "tasks.t.timeout: expected a number of seconds > 0, got 0"), + ({"timeout": -1.5}, "tasks.t.timeout: expected a number of seconds > 0, got -1.5"), + ({"timeout": True}, "tasks.t.timeout: expected a number of seconds > 0, got True"), + ({"when": "sometimes"}, "tasks.t.when: expected one of on_success, on_failure, always"), + ({"depends_on": [3]}, "tasks.t.depends_on[0]: expected a task ID, got int"), + ({"idempotency_key": ""}, "tasks.t.idempotency_key: expected a non-empty string"), + ( + {"idempotency_key": "k-${tasks.a.result}"}, + "tasks.t.idempotency_key: ${tasks.a.result}: only ${params.} can be used here", + ), + ], +) +def test_a_malformed_option_is_rejected_when_the_task_is_added( + options: dict[str, Any], problem: str +) -> None: + with pytest.raises(PipelineDefinitionException) as raised: + Pipeline("malformed").task("t", lambda ctx: None, **options) + assert any(found.startswith(problem) for found in raised.value.problems) + + +@pytest.mark.parametrize( + "options", + [ + {"name": ""}, + {"name": " "}, + {"max_workers": 0}, + {"max_workers": True}, + {"params": {"a b": 1}}, + ], +) +def test_a_malformed_pipeline_is_rejected(options: dict[str, Any]) -> None: + arguments = {"name": "fine", **options} + with pytest.raises(PipelineDefinitionException): + Pipeline(**arguments) + + +def test_depends_on_takes_one_id_as_a_string() -> None: + pipeline = Pipeline("single") + pipeline.task("first", lambda ctx: None) + assert pipeline.task("second", lambda ctx: None, depends_on="first").depends_on == ("first",) + + +def test_the_exceptions_belong_to_the_project_hierarchy() -> None: + from automation_file.exceptions import FileAutomationException + + assert issubclass(PipelineException, FileAutomationException) + assert issubclass(PipelineDefinitionException, PipelineException) + assert PipelineDefinitionException("one problem").problems == ("one problem",) diff --git a/tests/test_pipeline_run.py b/tests/test_pipeline_run.py new file mode 100644 index 0000000..7167f9e --- /dev/null +++ b/tests/test_pipeline_run.py @@ -0,0 +1,955 @@ +"""The pipeline runtime: ordering, fan-out, statuses, retry, timeout, cancellation, conditions.""" + +from __future__ import annotations + +import threading +import time +from collections.abc import Callable +from typing import Any + +import pytest + +from automation_file import ActionRegistry +from automation_file.core.progress import CancellationToken, CancelledException +from automation_file.events import EventBus +from automation_file.exceptions import StorageTransientException +from automation_file.pipeline import ( + DEFAULT_RETRY_ON, + MemoryRunStore, + Pipeline, + PipelineDefinitionException, + PipelineRun, + RetryPolicy, + RunStatus, + TaskContext, + TaskStatus, + worker, +) +from automation_file.pipeline.runner import Engine + +WAIT = 5.0 # an upper bound for things that happen at once; never slept through + + +def until(condition: Callable[[], bool], timeout: float = WAIT) -> bool: + """Poll ``condition`` until it holds; for state another thread is about to reach.""" + deadline = time.monotonic() + timeout + while time.monotonic() < deadline: + if condition(): + return True + time.sleep(0.002) + return condition() + + +def join_task_thread(pipeline: str, task: str) -> None: + """Wait for the thread of a task the run has given up on.""" + for thread in threading.enumerate(): + if thread.name == f"pipeline-{pipeline}-{task}": + thread.join(WAIT) + assert not thread.is_alive() + + +@pytest.fixture +def store() -> MemoryRunStore: + return MemoryRunStore() + + +@pytest.fixture +def bus() -> EventBus: + return EventBus() + + +@pytest.fixture +def execute(store: MemoryRunStore, bus: EventBus) -> Callable[..., PipelineRun]: + """Run a pipeline against a store and a bus of this test alone.""" + + def _execute(pipeline: Pipeline, **options: Any) -> PipelineRun: + return pipeline.run(store=store, bus=bus, **options) + + return _execute + + +@pytest.fixture +def registry() -> ActionRegistry: + registered = ActionRegistry() + registered.register("T_echo", lambda *args, **kwargs: {"args": list(args), "kwargs": kwargs}) + registered.register("T_constant", lambda: "constant") + registered.register("T_same", lambda value: value) + return registered + + +def statuses(run: PipelineRun) -> dict[str, str]: + return {task_id: state.status.value for task_id, state in run.tasks.items()} + + +# ---------------------------------------------------------------------- ordering and fan-out + + +def test_a_chain_runs_in_dependency_order(execute: Callable[..., PipelineRun]) -> None: + order: list[str] = [] + pipeline = Pipeline("chain") + pipeline.task("c", lambda ctx: order.append("c"), depends_on=["b"]) + pipeline.task("b", lambda ctx: order.append("b"), depends_on=["a"]) + pipeline.task("a", lambda ctx: order.append("a")) + run = execute(pipeline) + assert order == ["a", "b", "c"] + assert run.status is RunStatus.SUCCEEDED + assert run.status == "succeeded" + assert list(run.tasks) == ["a", "b", "c"] + assert [state.level for state in run.tasks.values()] == [0, 1, 2] + assert run.error is None + assert run.done is True + assert run.started_at is not None and run.finished_at is not None + assert run.started_at <= run.finished_at + + +def test_a_diamond_joins_after_both_branches(execute: Callable[..., PipelineRun]) -> None: + order: list[str] = [] + guard = threading.Lock() + + def note(ctx: TaskContext) -> str: + with guard: + order.append(ctx.task) + return ctx.task + + pipeline = Pipeline("diamond") + pipeline.task("top", note) + pipeline.task("left", note, depends_on=["top"]) + pipeline.task("right", note, depends_on=["top"]) + pipeline.task("bottom", note, depends_on=["left", "right"]) + run = execute(pipeline) + assert order[0] == "top" + assert order[-1] == "bottom" + assert sorted(order[1:3]) == ["left", "right"] + assert [state.level for state in run.tasks.values()] == [0, 1, 1, 2] + assert statuses(run) == dict.fromkeys(["top", "left", "right", "bottom"], "succeeded") + + +def test_independent_tasks_run_at_the_same_time(execute: Callable[..., PipelineRun]) -> None: + together = threading.Barrier(3) + pipeline = Pipeline("fan-out", max_workers=3) + for name in ("one", "two", "three"): + # The barrier only opens when all three are inside at once. + pipeline.task(name, lambda ctx: together.wait(WAIT)) + run = execute(pipeline) + assert statuses(run) == {"one": "succeeded", "two": "succeeded", "three": "succeeded"} + assert sorted(state.result for state in run.tasks.values()) == [0, 1, 2] + + +@pytest.mark.parametrize("workers", [1, 2]) +def test_max_workers_bounds_the_fan_out(execute: Callable[..., PipelineRun], workers: int) -> None: + guard = threading.Lock() + inside = {"now": 0, "peak": 0} + pair = threading.Barrier(workers) + + def work(_ctx: TaskContext) -> None: + with guard: + inside["now"] += 1 + inside["peak"] = max(inside["peak"], inside["now"]) + pair.wait(WAIT) + with guard: + inside["now"] -= 1 + + pipeline = Pipeline("bounded", max_workers=workers) + for index in range(4): + pipeline.task(f"task{index}", work) + run = execute(pipeline) + assert run.status is RunStatus.SUCCEEDED + assert inside["peak"] == workers + + +def test_pending_and_running_are_visible_while_a_run_is_busy( + store: MemoryRunStore, bus: EventBus +) -> None: + entered, release = threading.Event(), threading.Event() + + def first(_ctx: TaskContext) -> str: + entered.set() + release.wait(WAIT) + return "done" + + pipeline = Pipeline("busy") + pipeline.task("first", first) + pipeline.task("second", lambda ctx: ctx.results["first"], depends_on=["first"]) + run = pipeline.start(store=store, bus=bus) + assert entered.wait(WAIT) + assert run.status is RunStatus.RUNNING + assert run.done is False + assert run.wait(0.01) is False + assert statuses(run) == {"first": "running", "second": "pending"} + assert run.tasks["first"].started_at is not None + assert run.tasks["first"].finished_at is None + release.set() + assert run.wait(WAIT) is True + assert run.done is True + assert statuses(run) == {"first": "succeeded", "second": "succeeded"} + assert run.tasks["second"].result == "done" + + +# ---------------------------------------------------------------------- failure + + +def test_a_failing_task_is_recorded_and_its_dependents_are_skipped( + execute: Callable[..., PipelineRun], +) -> None: + def boom(_ctx: TaskContext) -> None: + raise ValueError("boom") + + ran: list[str] = [] + pipeline = Pipeline("failing") + pipeline.task("bad", boom) + pipeline.task("good", lambda ctx: "fine") + pipeline.task("child", lambda ctx: ran.append("child"), depends_on=["bad"]) + pipeline.task("grandchild", lambda ctx: ran.append("grandchild"), depends_on=["child"]) + run = execute(pipeline) + assert ran == [] + assert statuses(run) == { + "bad": "failed", + "good": "succeeded", + "child": "skipped", + "grandchild": "skipped", + } + bad = run.tasks["bad"] + assert bad.error == "ValueError: boom" + assert bad.attempts == 1 + assert bad.result is None + assert bad.finished_at is not None + assert run.tasks["child"].reason == "upstream_failed" + assert run.tasks["child"].attempts == 0 + assert run.tasks["grandchild"].reason == "upstream_skipped" + assert run.status is RunStatus.FAILED + assert run.error == "did not succeed: bad" + + +def test_a_task_that_calls_sys_exit_is_recorded_as_failed( + execute: Callable[..., PipelineRun], +) -> None: + def leaves(_ctx: TaskContext) -> None: + raise SystemExit(3) + + pipeline = Pipeline("exits") + pipeline.task("leaves", leaves, retry=RetryPolicy(max_attempts=3)) + pipeline.task("after", lambda ctx: "never", depends_on=["leaves"]) + run = execute(pipeline) + assert run.tasks["leaves"].status is TaskStatus.FAILED + assert run.tasks["leaves"].error == "SystemExit: 3" + assert run.tasks["leaves"].attempts == 1 + assert run.tasks["after"].status is TaskStatus.SKIPPED + assert run.status is RunStatus.FAILED + + +def test_an_interrupted_coordinator_leaves_a_failed_run( + store: MemoryRunStore, bus: EventBus, monkeypatch: pytest.MonkeyPatch +) -> None: + def explode(_engine: Engine) -> None: + raise RuntimeError("no threads left") + + monkeypatch.setattr(Engine, "_launch", explode) + pipeline = Pipeline("aborted") + pipeline.task("only", lambda ctx: "never") + with pytest.raises(RuntimeError, match="no threads left"): + pipeline.run(store=store, bus=bus) + (recorded,) = store.list_runs("aborted") + assert recorded.status is RunStatus.FAILED + assert recorded.error == "RuntimeError: no threads left" + assert recorded.done is True + assert recorded.tasks["only"].status is TaskStatus.CANCELLED + assert [event.type for event in reversed(bus.recent())] == [ + "pipeline.started", + "pipeline.failed", + ] + + +def test_ctrl_c_gives_up_on_what_is_in_the_air( + store: MemoryRunStore, bus: EventBus, monkeypatch: pytest.MonkeyPatch +) -> None: + entered, release = threading.Event(), threading.Event() + + def interrupted(_engine: Engine) -> None: + assert entered.wait(WAIT) + raise KeyboardInterrupt + + def slow(ctx: TaskContext) -> str: + entered.set() + release.wait(WAIT) + return f"late, cancelled={ctx.cancel.is_cancelled}" + + monkeypatch.setattr(Engine, "_await", interrupted) + pipeline = Pipeline("interrupted") + pipeline.task("slow", slow) + pipeline.task("next", lambda ctx: "never", depends_on=["slow"]) + try: + with pytest.raises(KeyboardInterrupt): + pipeline.run(store=store, bus=bus) + (recorded,) = store.list_runs("interrupted") + assert recorded.status is RunStatus.FAILED + assert recorded.error == "KeyboardInterrupt" + assert recorded.done is True + assert statuses(recorded) == {"slow": "cancelled", "next": "cancelled"} + finally: + release.set() + join_task_thread("interrupted", "slow") + # The task returned after the run had given up on it; its result is not recorded. + assert store.get_run(recorded.run_id).to_dict() == recorded.to_dict() + assert bus.recent(limit=1)[0].type == "pipeline.failed" + + +def test_a_background_run_that_aborts_still_ends( + store: MemoryRunStore, bus: EventBus, monkeypatch: pytest.MonkeyPatch +) -> None: + def explode(_engine: Engine) -> None: + raise RuntimeError("no threads left") + + monkeypatch.setattr(Engine, "_launch", explode) + pipeline = Pipeline("aborted-in-background") + pipeline.task("only", lambda ctx: "never") + run = pipeline.start(store=store, bus=bus) + assert run.wait(WAIT) is True + assert run.status is RunStatus.FAILED + assert run.error == "RuntimeError: no threads left" + + +# ---------------------------------------------------------------------- retry + + +def test_the_default_retry_types_are_the_transient_ones() -> None: + policy = RetryPolicy() + assert policy.max_attempts == 1 + assert policy.backoff_base == pytest.approx(0.0) + assert policy.backoff_cap == pytest.approx(60.0) + assert policy.retry_on == (StorageTransientException, ConnectionError, TimeoutError) + assert policy.retry_on is DEFAULT_RETRY_ON + assert Exception not in policy.retry_on + assert policy.retries(StorageTransientException("throttled")) is True + assert policy.retries(ConnectionResetError("reset")) is True + assert policy.retries(ValueError("a bug")) is False + + +def test_the_back_off_doubles_up_to_its_cap() -> None: + policy = RetryPolicy(max_attempts=9, backoff_base=2.0, backoff_cap=5.0) + assert [policy.delay(attempt) for attempt in (1, 2, 3, 4)] == [2.0, 4.0, 5.0, 5.0] + assert policy.delay(5000) == pytest.approx(5.0) + assert RetryPolicy(max_attempts=3).delay(2) == pytest.approx(0.0) + + +@pytest.mark.parametrize( + "options", + [ + {"max_attempts": 0}, + {"max_attempts": True}, + {"max_attempts": 2.5}, + {"backoff_base": -1.0}, + {"backoff_cap": -0.1}, + {"retry_on": ("ValueError",)}, + {"retry_on": (int,)}, + ], +) +def test_a_retry_policy_rejects_bad_values(options: dict[str, Any]) -> None: + with pytest.raises(PipelineDefinitionException): + RetryPolicy(**options) + + +def test_retry_on_accepts_a_list() -> None: + assert RetryPolicy(retry_on=[KeyError]).retry_on == (KeyError,) # type: ignore[arg-type] + + +def test_a_transient_failure_is_retried_with_back_off( + execute: Callable[..., PipelineRun], monkeypatch: pytest.MonkeyPatch +) -> None: + waits: list[float] = [] + + def no_sleep(_token: object, seconds: float) -> bool: + waits.append(seconds) + return False + + monkeypatch.setattr(worker, "pause", no_sleep) + attempts: list[int] = [] + + def flaky(ctx: TaskContext) -> str: + attempts.append(ctx.attempt) + if ctx.attempt < 5: + raise StorageTransientException("throttled") + return "through" + + pipeline = Pipeline("retrying") + pipeline.task( + "flaky", flaky, retry=RetryPolicy(max_attempts=5, backoff_base=1.0, backoff_cap=3.0) + ) + run = execute(pipeline) + state = run.tasks["flaky"] + assert attempts == [1, 2, 3, 4, 5] + assert waits == [1.0, 2.0, 3.0, 3.0] + assert state.status is TaskStatus.SUCCEEDED + assert state.attempts == 5 + assert state.result == "through" + assert state.error is None + assert run.status is RunStatus.SUCCEEDED + + +def test_retry_gives_up_after_max_attempts( + execute: Callable[..., PipelineRun], monkeypatch: pytest.MonkeyPatch +) -> None: + waits: list[float] = [] + monkeypatch.setattr(worker, "pause", lambda _token, seconds: waits.append(seconds) or False) + + def down(_ctx: TaskContext) -> None: + raise ConnectionError("refused") + + pipeline = Pipeline("giving-up") + pipeline.task("down", down, retry=RetryPolicy(max_attempts=3, backoff_base=0.5)) + run = execute(pipeline) + assert run.tasks["down"].status is TaskStatus.FAILED + assert run.tasks["down"].attempts == 3 + assert run.tasks["down"].error == "ConnectionError: refused" + assert waits == [0.5, 1.0] + + +def test_an_error_outside_retry_on_fails_at_once( + execute: Callable[..., PipelineRun], monkeypatch: pytest.MonkeyPatch +) -> None: + waits: list[float] = [] + monkeypatch.setattr(worker, "pause", lambda _token, seconds: waits.append(seconds) or False) + + def bug(_ctx: TaskContext) -> None: + raise KeyError("typo") + + pipeline = Pipeline("bug") + pipeline.task("bug", bug, retry=RetryPolicy(max_attempts=4, backoff_base=1.0)) + pipeline.task( + "wanted", + bug, + retry=RetryPolicy(max_attempts=2, retry_on=(KeyError,)), + ) + run = execute(pipeline) + assert run.tasks["bug"].attempts == 1 + assert run.tasks["wanted"].attempts == 2 + assert waits == [0.0] + + +def test_a_back_off_ends_when_the_run_is_cancelled( + store: MemoryRunStore, bus: EventBus, monkeypatch: pytest.MonkeyPatch +) -> None: + waiting = threading.Event() + real_pause = worker.pause + + def announced(token: worker.TaskToken, seconds: float) -> bool: + waiting.set() + return real_pause(token, seconds) + + monkeypatch.setattr(worker, "pause", announced) + + def flaky(_ctx: TaskContext) -> None: + raise ConnectionError("down") + + pipeline = Pipeline("cancelled-back-off") + pipeline.task("flaky", flaky, retry=RetryPolicy(max_attempts=3, backoff_base=600.0)) + run = pipeline.start(store=store, bus=bus) + assert waiting.wait(WAIT) + run.cancel() + assert run.wait(WAIT) is True # the ten-minute back-off was not slept through + state = run.tasks["flaky"] + assert state.status is TaskStatus.CANCELLED + assert state.attempts == 1 + assert state.error == "ConnectionError: down" + assert run.status is RunStatus.CANCELLED + + +# ---------------------------------------------------------------------- timeout + + +def test_a_task_past_its_timeout_is_marked_and_the_run_goes_on( + execute: Callable[..., PipelineRun], bus: EventBus +) -> None: + release, saw_cancel = threading.Event(), threading.Event() + + def stuck(ctx: TaskContext) -> str: + release.wait(WAIT) # ignores its token until the test lets it go + if ctx.cancel.is_cancelled: + saw_cancel.set() + return "too late" + + pipeline = Pipeline("slow") + pipeline.task("stuck", stuck, timeout=0.05) + pipeline.task("other", lambda ctx: "ran") + pipeline.task("after", lambda ctx: "never", depends_on=["stuck"]) + pipeline.task("cleanup", lambda ctx: "cleaned", depends_on=["stuck"], when="on_failure") + try: + run = execute(pipeline) + state = run.tasks["stuck"] + assert state.status is TaskStatus.TIMEOUT + assert state.error == "TimeoutError: no result within 0.05 s" + assert state.result is None + assert state.finished_at is not None + assert run.tasks["other"].status is TaskStatus.SUCCEEDED + assert run.tasks["after"].status is TaskStatus.SKIPPED + assert run.tasks["after"].reason == "upstream_failed" + assert run.tasks["cleanup"].result == "cleaned" + assert run.status is RunStatus.FAILED + assert run.error == "did not succeed: stuck" + finally: + release.set() + join_task_thread("slow", "stuck") + # The thread could not be killed: it finished later, saw its token, and changed nothing. + assert saw_cancel.is_set() + assert run.tasks["stuck"].status is TaskStatus.TIMEOUT + assert run.tasks["stuck"].result is None + assert bus.recent(limit=1)[0].type == "pipeline.failed" + + +def test_a_task_inside_its_timeout_succeeds(execute: Callable[..., PipelineRun]) -> None: + pipeline = Pipeline("quick") + pipeline.task("quick", lambda ctx: "in time", timeout=WAIT) + run = execute(pipeline) + assert run.tasks["quick"].status is TaskStatus.SUCCEEDED + assert run.tasks["quick"].result == "in time" + + +def test_the_timeout_covers_the_waits_between_attempts( + execute: Callable[..., PipelineRun], +) -> None: + def down(_ctx: TaskContext) -> None: + raise ConnectionError("down") + + pipeline = Pipeline("budget") + pipeline.task("down", down, retry=RetryPolicy(max_attempts=5, backoff_base=600.0), timeout=0.05) + run = execute(pipeline) + assert run.tasks["down"].status is TaskStatus.TIMEOUT + assert run.tasks["down"].attempts == 1 + join_task_thread("budget", "down") + assert run.tasks["down"].status is TaskStatus.TIMEOUT + + +# ---------------------------------------------------------------------- cancellation + + +def test_cancel_stops_a_background_run(store: MemoryRunStore, bus: EventBus) -> None: + entered = threading.Event() + ran: list[str] = [] + + def watches(ctx: TaskContext) -> None: + entered.set() + assert until(lambda: ctx.cancel.is_cancelled) + ctx.cancel.raise_if_cancelled() + + pipeline = Pipeline("cancelled") + pipeline.task("first", watches) + pipeline.task("second", lambda ctx: ran.append("second"), depends_on=["first"]) + pipeline.task("cleanup", lambda ctx: ran.append("cleanup"), depends_on=["first"], when="always") + run = pipeline.start(store=store, bus=bus) + assert entered.wait(WAIT) + run.cancel() + assert run.wait(WAIT) is True + assert ran == [] + assert statuses(run) == {"first": "cancelled", "second": "cancelled", "cleanup": "cancelled"} + assert run.tasks["first"].error == "CancelledException: operation cancelled" + assert run.tasks["second"].attempts == 0 + assert run.status is RunStatus.CANCELLED + assert run.error == "the run was cancelled" + assert store.get_run(run.run_id).to_dict() == run.to_dict() + + +def test_a_token_cancelled_beforehand_runs_nothing(execute: Callable[..., PipelineRun]) -> None: + ran: list[str] = [] + token = CancellationToken() + token.cancel() + pipeline = Pipeline("pre-cancelled") + pipeline.task("first", lambda ctx: ran.append("first")) + pipeline.task("second", lambda ctx: ran.append("second"), depends_on=["first"]) + run = execute(pipeline, cancel=token) + assert ran == [] + assert statuses(run) == {"first": "cancelled", "second": "cancelled"} + assert run.status is RunStatus.CANCELLED + assert run.cancel_token is token + + +def test_a_running_task_that_ignores_cancellation_keeps_its_outcome( + store: MemoryRunStore, bus: EventBus +) -> None: + entered, release = threading.Event(), threading.Event() + + def deaf(_ctx: TaskContext) -> str: + entered.set() + release.wait(WAIT) + return "finished anyway" + + token = CancellationToken() + pipeline = Pipeline("deaf") + pipeline.task("deaf", deaf) + pipeline.task("next", lambda ctx: "never", depends_on=["deaf"]) + run = pipeline.start(store=store, bus=bus, cancel=token) + assert entered.wait(WAIT) + token.cancel() + assert until(lambda: run.tasks["next"].status is TaskStatus.CANCELLED) + assert run.done is False # the run waits for what is still in the air + release.set() + assert run.wait(WAIT) is True + assert run.tasks["deaf"].status is TaskStatus.SUCCEEDED + assert run.tasks["deaf"].result == "finished anyway" + assert run.status is RunStatus.CANCELLED + + +def test_a_task_cancelled_on_its_own_counts_as_a_failure( + execute: Callable[..., PipelineRun], +) -> None: + def gives_up(_ctx: TaskContext) -> None: + raise CancelledException("transfer cancelled") + + pipeline = Pipeline("self-cancelled") + pipeline.task("transfer", gives_up) + pipeline.task("cleanup", lambda ctx: "cleaned", depends_on=["transfer"], when="on_failure") + run = execute(pipeline) + assert run.tasks["transfer"].status is TaskStatus.CANCELLED + assert run.tasks["cleanup"].status is TaskStatus.SUCCEEDED + assert run.status is RunStatus.FAILED + + +# ---------------------------------------------------------------------- conditions + + +def _outcomes(execute: Callable[..., PipelineRun], upstream_fails: bool, when: Any) -> PipelineRun: + def upstream(_ctx: TaskContext) -> str: + if upstream_fails: + raise ValueError("upstream broke") + return "fine" + + pipeline = Pipeline("conditions") + pipeline.task("upstream", upstream) + pipeline.task("conditional", lambda ctx: "ran", depends_on=["upstream"], when=when) + return execute(pipeline) + + +@pytest.mark.parametrize( + "when,upstream_fails,status,reason", + [ + ("on_success", False, "succeeded", None), + ("on_success", True, "skipped", "upstream_failed"), + ("on_failure", True, "succeeded", None), + ("on_failure", False, "skipped", "condition"), + ("always", False, "succeeded", None), + ("always", True, "succeeded", None), + ], +) +def test_when_decides_whether_a_task_runs( + execute: Callable[..., PipelineRun], + when: str, + upstream_fails: bool, + status: str, + reason: str | None, +) -> None: + run = _outcomes(execute, upstream_fails, when) + state = run.tasks["conditional"] + assert state.status == status + assert state.reason == reason + assert state.result == ("ran" if status == "succeeded" else None) + + +def test_on_success_is_the_default() -> None: + pipeline = Pipeline("default") + assert pipeline.task("a", lambda ctx: None).when == "on_success" + + +def test_on_failure_without_dependencies_never_runs(execute: Callable[..., PipelineRun]) -> None: + pipeline = Pipeline("orphan") + pipeline.task("cleanup", lambda ctx: "ran", when="on_failure") + run = execute(pipeline) + assert run.tasks["cleanup"].status is TaskStatus.SKIPPED + assert run.tasks["cleanup"].reason == "condition" + assert run.status is RunStatus.SUCCEEDED + + +def test_a_callable_condition_sees_the_context(execute: Callable[..., PipelineRun]) -> None: + seen: list[TaskContext] = [] + + def big_enough(ctx: TaskContext) -> bool: + seen.append(ctx) + return ctx.results["count"] >= ctx.params["minimum"] + + pipeline = Pipeline("callable-when") + pipeline.task("count", lambda ctx: 7) + pipeline.task("report", lambda ctx: "reported", depends_on=["count"], when=big_enough) + run = execute(pipeline, params={"minimum": 5}) + assert run.tasks["report"].status is TaskStatus.SUCCEEDED + assert (seen[0].task, seen[0].attempt, seen[0].run_id) == ("report", 0, run.run_id) + skipped = execute(pipeline, params={"minimum": 50}) + assert skipped.tasks["report"].status is TaskStatus.SKIPPED + assert skipped.tasks["report"].reason == "condition" + assert len(seen) == 2 # asked exactly once per run + + +def test_a_condition_that_raises_fails_its_task(execute: Callable[..., PipelineRun]) -> None: + def broken(_ctx: TaskContext) -> bool: + raise LookupError("no such result") + + ran: list[str] = [] + pipeline = Pipeline("broken-when") + pipeline.task("guarded", lambda ctx: ran.append("guarded"), when=broken) + run = execute(pipeline) + assert ran == [] + assert run.tasks["guarded"].status is TaskStatus.FAILED + assert run.tasks["guarded"].error == "when: LookupError: no such result" + assert run.tasks["guarded"].attempts == 0 + assert run.status is RunStatus.FAILED + + +def test_a_skip_propagates_to_on_success_dependents_only( + execute: Callable[..., PipelineRun], +) -> None: + pipeline = Pipeline("propagation") + pipeline.task("gate", lambda ctx: "ran", when=lambda ctx: False) + pipeline.task("child", lambda ctx: "ran", depends_on=["gate"]) + pipeline.task("grandchild", lambda ctx: "ran", depends_on=["child"]) + pipeline.task("regardless", lambda ctx: "ran", depends_on=["gate"], when="always") + pipeline.task("on-error", lambda ctx: "ran", depends_on=["gate"], when="on_failure") + run = execute(pipeline) + assert statuses(run) == { + "gate": "skipped", + "child": "skipped", + "regardless": "succeeded", + "on-error": "skipped", + "grandchild": "skipped", + } + assert run.tasks["gate"].reason == "condition" + assert run.tasks["child"].reason == "upstream_skipped" + assert run.tasks["grandchild"].reason == "upstream_skipped" + assert run.tasks["on-error"].reason == "condition" + assert run.status is RunStatus.SUCCEEDED # nothing failed + + +def test_a_failure_outranks_a_skip_as_the_reason(execute: Callable[..., PipelineRun]) -> None: + def boom(_ctx: TaskContext) -> None: + raise ValueError("boom") + + pipeline = Pipeline("mixed") + pipeline.task("skipped", lambda ctx: "ran", when=lambda ctx: False) + pipeline.task("failed", boom) + pipeline.task("both", lambda ctx: "ran", depends_on=["skipped", "failed"]) + run = execute(pipeline) + assert run.tasks["both"].reason == "upstream_failed" + + +# ---------------------------------------------------------------------- context and results + + +def test_a_callable_gets_its_context(execute: Callable[..., PipelineRun]) -> None: + seen: dict[str, TaskContext] = {} + + def remember(ctx: TaskContext) -> str: + seen[ctx.task] = ctx + return f"{ctx.task}-result" + + pipeline = Pipeline("context", params={"date": "2026-01-01", "region": "emea"}) + pipeline.task("a", remember) + pipeline.task("b", remember, depends_on=["a"]) + pipeline.task("c", remember, depends_on=["b"]) + pipeline.task("unrelated", remember) + run = execute(pipeline, params={"date": "2026-10-08"}) + ctx = seen["c"] + assert ctx.pipeline == "context" + assert ctx.run_id == run.run_id + assert ctx.task == "c" + assert ctx.attempt == 1 + assert ctx.dry_run is False + assert dict(ctx.params) == {"date": "2026-10-08", "region": "emea"} + assert dict(ctx.results) == {"a": "a-result", "b": "b-result"} # every upstream task + assert isinstance(ctx.cancel, CancellationToken) + assert ctx.cancel.is_cancelled is False + assert dict(seen["a"].results) == {} + with pytest.raises(TypeError): + ctx.params["date"] = "changed" # type: ignore[index] + with pytest.raises(TypeError): + ctx.results["a"] = "changed" # type: ignore[index] + assert run.params == {"date": "2026-10-08", "region": "emea"} + assert pipeline.params == {"date": "2026-01-01", "region": "emea"} + + +def test_results_hold_only_the_upstream_tasks_that_succeeded( + execute: Callable[..., PipelineRun], +) -> None: + def boom(_ctx: TaskContext) -> None: + raise ValueError("boom") + + pipeline = Pipeline("partial") + pipeline.task("good", lambda ctx: 1) + pipeline.task("bad", boom) + pipeline.task( + "report", lambda ctx: dict(ctx.results), depends_on=["good", "bad"], when="always" + ) + run = execute(pipeline) + assert run.tasks["report"].result == {"good": 1} + + +def test_a_result_json_cannot_hold_stays_an_object_in_memory( + execute: Callable[..., PipelineRun], store: MemoryRunStore +) -> None: + marker = object() + pipeline = Pipeline("objects") + pipeline.task("make", lambda ctx: marker) + pipeline.task("use", lambda ctx: ctx.results["make"] is marker, depends_on=["make"]) + run = execute(pipeline) + assert run.tasks["make"].result is marker + assert run.tasks["make"].result_is_repr is False + assert run.tasks["use"].result is True + document = run.to_dict()["tasks"]["make"] + assert document["result"] == repr(marker) + assert document["result_is_repr"] is True + stored = store.get_run(run.run_id).tasks["make"] + assert stored.result == repr(marker) + assert stored.result_is_repr is True + + +# ---------------------------------------------------------------------- action tasks + + +def test_the_three_action_shapes( + execute: Callable[..., PipelineRun], registry: ActionRegistry +) -> None: + pipeline = Pipeline("actions", registry=registry) + pipeline.task("bare", ["T_constant"]) + pipeline.task("keywords", ["T_echo", {"a": 1, "b": "two"}]) + pipeline.task("positional", ["T_echo", [1, "two"]]) + run = execute(pipeline) + assert run.tasks["bare"].result == "constant" + assert run.tasks["keywords"].result == {"args": [], "kwargs": {"a": 1, "b": "two"}} + assert run.tasks["positional"].result == {"args": [1, "two"], "kwargs": {}} + + +def test_an_action_resolves_through_the_shared_registry( + execute: Callable[..., PipelineRun], +) -> None: + pipeline = Pipeline("shared") + pipeline.task("schemes", ["FA_storage_schemes"]) + run = execute(pipeline) + assert "memory" in run.tasks["schemes"].result + + +def test_an_action_that_fails_raises_into_its_task( + execute: Callable[..., PipelineRun], registry: ActionRegistry +) -> None: + def refuse(path: str) -> None: + raise PermissionError(f"denied: {path}") + + registry.register("T_refuse", refuse) + pipeline = Pipeline("action-errors", registry=registry) + pipeline.task("refused", ["T_refuse", {"path": "/etc"}]) + pipeline.task("unknown", ["T_missing"]) + pipeline.task("wrong-arguments", ["T_constant", {"surplus": 1}]) + run = execute(pipeline) + assert run.tasks["refused"].error == "PermissionError: denied: /etc" + assert run.tasks["unknown"].error == "PipelineException: unknown action 'T_missing'" + assert run.tasks["wrong-arguments"].error.startswith("TypeError: ") + assert run.status is RunStatus.FAILED + + +def test_parameters_and_results_are_substituted( + execute: Callable[..., PipelineRun], registry: ActionRegistry +) -> None: + listing = {"files": ["a.csv", "b.csv"]} + pipeline = Pipeline("substitution", registry=registry, params={"limit": 20}) + pipeline.task("list", lambda ctx: listing) + pipeline.task( + "use", + [ + "T_echo", + { + "text": "report-${params.date}-${params.limit}.csv", + "limit": "${params.limit}", + "listing": "${tasks.list.result}", + "nested": {"again": ["${tasks.list.result}", "x-${params.date}"], "number": 3}, + "untouched": "${env:HOME} and ${date:%Y} stay", + }, + ], + depends_on=["list"], + ) + run = execute(pipeline, params={"date": "2026-10-08"}) + kwargs = run.tasks["use"].result["kwargs"] + assert kwargs["text"] == "report-2026-10-08-20.csv" + assert kwargs["limit"] == 20 # a whole-string parameter keeps its type + assert kwargs["listing"] is listing # the result object itself + assert kwargs["nested"] == {"again": [listing, "x-2026-10-08"], "number": 3} + assert kwargs["untouched"] == "${env:HOME} and ${date:%Y} stay" + assert pipeline.tasks[1].work[1]["limit"] == "${params.limit}" # the definition is not changed + + +def test_positional_arguments_are_substituted( + execute: Callable[..., PipelineRun], registry: ActionRegistry +) -> None: + pipeline = Pipeline("positional-substitution", registry=registry) + pipeline.task("first", lambda ctx: [1, 2]) + pipeline.task("second", ["T_same", ["${tasks.first.result}"]], depends_on=["first"]) + run = execute(pipeline) + assert run.tasks["second"].result == [1, 2] + + +def test_the_result_of_a_failed_upstream_task_is_none( + execute: Callable[..., PipelineRun], registry: ActionRegistry +) -> None: + def boom(_ctx: TaskContext) -> None: + raise ValueError("boom") + + pipeline = Pipeline("failed-upstream", registry=registry) + pipeline.task("broken", boom) + pipeline.task( + "notify", + ["T_same", {"value": "${tasks.broken.result}"}], + depends_on=["broken"], + when="always", + ) + run = execute(pipeline) + assert run.tasks["notify"].status is TaskStatus.SUCCEEDED + assert run.tasks["notify"].result is None + + +# ---------------------------------------------------------------------- dry run + + +def test_a_dry_run_plans_every_task_and_executes_nothing( + store: MemoryRunStore, bus: EventBus, registry: ActionRegistry +) -> None: + ran: list[str] = [] + pipeline = Pipeline("planned", registry=registry) + pipeline.task("report", ["T_echo", {"rows": "${tasks.load.result}"}], depends_on=["load"]) + pipeline.task("load", lambda ctx: ran.append("load"), depends_on=["fetch"]) + pipeline.task("fetch", ["T_constant"], idempotency_key="fetch-${params.date}") + pipeline.task("side", lambda ctx: ran.append("side"), when=lambda ctx: ran.append("when")) + run = pipeline.run(params={"date": "2026-10-08"}, dry_run=True, store=store, bus=bus) + assert ran == [] + assert run.dry_run is True + assert run.done is True + assert run.status is RunStatus.SUCCEEDED + assert run.error is None + assert [(task_id, state.status.value, state.level) for task_id, state in run.tasks.items()] == [ + ("fetch", "planned", 0), + ("side", "planned", 0), + ("load", "planned", 1), + ("report", "planned", 2), + ] + assert all(state.error is None and state.attempts == 0 for state in run.tasks.values()) + assert store.list_runs() == [] + assert bus.recent() == [] + assert run.to_dict()["dry_run"] is True + + +def test_a_dry_run_reports_unknown_actions_and_missing_parameters( + registry: ActionRegistry, +) -> None: + pipeline = Pipeline("planned-with-problems", registry=registry) + pipeline.task("typo", ["T_ecko", {"path": "${params.path}"}]) + pipeline.task("fine", ["T_constant"], depends_on=["typo"]) + pipeline.task("keyed", ["T_constant"], idempotency_key="k-${params.date}") + run = pipeline.run(dry_run=True) + assert statuses(run) == {"typo": "planned", "keyed": "planned", "fine": "planned"} + assert run.tasks["typo"].error == ( + "tasks.typo.action[1].path: unknown parameter 'path';" + " tasks.typo.action[0]: unknown action 'T_ecko'" + ) + assert run.tasks["keyed"].error == "tasks.keyed.idempotency_key: unknown parameter 'date'" + assert run.tasks["fine"].error is None + assert run.status is RunStatus.FAILED + assert run.error == "would not run as planned: typo, keyed" + + +def test_a_dry_run_still_rejects_a_broken_graph() -> None: + pipeline = Pipeline("planned-cycle") + pipeline.task("a", lambda ctx: None, depends_on=["b"]) + pipeline.task("b", lambda ctx: None, depends_on=["a"]) + with pytest.raises(PipelineDefinitionException, match="dependency cycle: a -> b -> a"): + pipeline.run(dry_run=True) diff --git a/tests/test_pipeline_store.py b/tests/test_pipeline_store.py new file mode 100644 index 0000000..e3d68e8 --- /dev/null +++ b/tests/test_pipeline_store.py @@ -0,0 +1,783 @@ +"""Run stores, checkpoints, resume, idempotency and the execution history.""" + +from __future__ import annotations + +import inspect +import sqlite3 +import threading +import time +from collections.abc import Callable, Iterator +from contextlib import closing +from datetime import datetime, timedelta, timezone +from pathlib import Path +from typing import Any + +import pytest + +from automation_file.events import EventBus +from automation_file.pipeline import ( + MemoryRunStore, + Pipeline, + PipelineDefinitionException, + PipelineException, + PipelineRun, + RetryPolicy, + RunStatus, + RunStore, + SQLiteRunStore, + TaskContext, + TaskRun, + TaskStatus, + default_run_store, + set_default_run_store, + worker, +) + +WAIT = 5.0 +START = datetime(2026, 10, 8, 2, 0, tzinfo=timezone.utc) + + +@pytest.fixture(params=["memory", "sqlite"]) +def store(request: pytest.FixtureRequest, tmp_path: Path) -> RunStore: + if request.param == "memory": + return MemoryRunStore() + return SQLiteRunStore(tmp_path / "runs" / "pipelines.db") + + +@pytest.fixture +def bus() -> EventBus: + return EventBus() + + +def until(condition: Callable[[], bool], timeout: float = WAIT) -> bool: + """Poll ``condition`` until it holds; for state another thread is about to reach.""" + deadline = time.monotonic() + timeout + while time.monotonic() < deadline: + if condition(): + return True + time.sleep(0.002) + return condition() + + +def a_run(run_id: str = "run-1", pipeline: str = "daily", minutes: int = 0) -> PipelineRun: + return PipelineRun( + run_id=run_id, + pipeline=pipeline, + params={"date": "2026-10-08"}, + started_at=START + timedelta(minutes=minutes), + tasks={"load": TaskRun(task="load"), "report": TaskRun(task="report", level=1)}, + ) + + +def a_success( + task: str = "load", key: str | None = None, result: Any = None, minutes: int = 0 +) -> TaskRun: + return TaskRun( + task=task, + status=TaskStatus.SUCCEEDED, + attempts=2, + result=result, + idempotency_key=key, + started_at=START + timedelta(minutes=minutes), + finished_at=START + timedelta(minutes=minutes, seconds=3), + ) + + +# ---------------------------------------------------------------------- the store contract + + +def test_a_saved_run_comes_back_equal(store: RunStore) -> None: + run = a_run() + run.tasks["load"] = a_success(key="load-2026-10-08", result={"rows": 3, "files": ["a", "b"]}) + store.save_run(run) + loaded = store.get_run("run-1") + assert loaded is not None and loaded is not run + assert loaded.to_dict() == run.to_dict() + assert list(loaded.tasks) == ["load", "report"] + assert loaded.tasks["load"].started_at == START + assert loaded.tasks["load"].duration_ms == pytest.approx(3000.0) + assert loaded.tasks["report"].level == 1 + assert loaded.status is RunStatus.RUNNING + assert loaded.done is False + assert loaded.params == {"date": "2026-10-08"} + + +def test_an_unknown_run_is_none(store: RunStore) -> None: + assert store.get_run("nope") is None + + +def test_save_task_records_one_transition(store: RunStore) -> None: + store.save_run(a_run()) + store.save_task("run-1", a_success(result=[1, 2])) + failed = TaskRun(task="report", status=TaskStatus.FAILED, level=1, attempts=1, error="X: y") + store.save_task("run-1", failed) + store.save_task( + "run-1", TaskRun(task="added-later", status=TaskStatus.SKIPPED, reason="condition") + ) + loaded = store.get_run("run-1") + assert list(loaded.tasks) == ["load", "report", "added-later"] + assert loaded.tasks["load"].status is TaskStatus.SUCCEEDED + assert loaded.tasks["load"].result == [1, 2] + assert loaded.tasks["report"].error == "X: y" + assert loaded.tasks["added-later"].reason == "condition" + + +def test_save_task_needs_a_stored_run(store: RunStore) -> None: + with pytest.raises(PipelineException, match="unknown run 'ghost'"): + store.save_task("ghost", TaskRun(task="load")) + + +def test_save_run_replaces_the_whole_run(store: RunStore) -> None: + store.save_run(a_run()) + changed = a_run() + changed.status = RunStatus.SUCCEEDED + changed.finished_at = START + timedelta(seconds=9) + changed.error = None + changed.tasks = {"report": TaskRun(task="report"), "extra": TaskRun(task="extra", level=1)} + store.save_run(changed) + loaded = store.get_run("run-1") + assert list(loaded.tasks) == ["report", "extra"] + assert loaded.status is RunStatus.SUCCEEDED + assert loaded.done is True + assert loaded.finished_at == changed.finished_at + assert len(store.list_runs()) == 1 + + +def test_what_a_store_returns_is_a_copy(store: RunStore) -> None: + result = {"rows": [1]} + run = a_run() + run.tasks["load"] = a_success(result=result) + store.save_run(run) + result["rows"].append(2) # the live object changes after the checkpoint + loaded = store.get_run("run-1") + assert loaded.tasks["load"].result == {"rows": [1]} + loaded.tasks["load"].result["rows"].append(99) + loaded.params["date"] = "changed" + again = store.get_run("run-1") + assert again.tasks["load"].result == {"rows": [1]} + assert again.params == {"date": "2026-10-08"} + assert store.list_runs()[0].tasks["load"].result == {"rows": [1]} + + +def test_list_runs_is_newest_first_and_filters(store: RunStore) -> None: + store.save_run(a_run("run-1", "daily", minutes=0)) + store.save_run(a_run("run-3", "daily", minutes=20)) + store.save_run(a_run("run-2", "weekly", minutes=10)) + assert [run.run_id for run in store.list_runs()] == ["run-3", "run-2", "run-1"] + assert [run.run_id for run in store.list_runs("daily")] == ["run-3", "run-1"] + assert [run.run_id for run in store.list_runs(pipeline="weekly")] == ["run-2"] + assert store.list_runs("unknown") == [] + assert [run.run_id for run in store.list_runs(limit=2)] == ["run-3", "run-2"] + assert store.list_runs(limit=0) == [] + assert list(store.list_runs()[0].tasks) == ["load", "report"] + assert inspect.signature(RunStore.list_runs).parameters["limit"].default == 50 + + +def test_runs_started_at_the_same_moment_list_the_latest_saved_first(store: RunStore) -> None: + for run_id in ("first", "second", "third"): + store.save_run(a_run(run_id)) + store.save_run(a_run("first")) # saving again does not move it + assert [run.run_id for run in store.list_runs()] == ["third", "second", "first"] + + +def test_find_idempotent_returns_the_latest_succeeded_execution(store: RunStore) -> None: + def record(run_id: str, pipeline: str, state: TaskRun) -> None: + run = a_run(run_id, pipeline) + run.tasks["load"] = state + store.save_run(run) + + record("run-1", "daily", a_success(key="k1", result="old", minutes=1)) + record("run-2", "daily", a_success(key="k1", result="new", minutes=5)) + record("run-3", "daily", a_success(key="k1", result="middle", minutes=3)) + failed = TaskRun(task="load", status=TaskStatus.FAILED, idempotency_key="k1") + record("run-4", "daily", failed) + reused = TaskRun( + task="load", + status=TaskStatus.SKIPPED, + reason="idempotent", + idempotency_key="k1", + result="reused", + finished_at=START + timedelta(minutes=9), + ) + record("run-5", "daily", reused) + record("run-6", "weekly", a_success(key="k1", result="weekly", minutes=7)) + found = store.find_idempotent("daily", "load", "k1") + assert found is not None + assert (found.result, found.status, found.attempts) == ("new", TaskStatus.SUCCEEDED, 2) + assert found.idempotency_key == "k1" + assert store.find_idempotent("weekly", "load", "k1").result == "weekly" + assert store.find_idempotent("daily", "load", "k2") is None + assert store.find_idempotent("daily", "report", "k1") is None + assert store.find_idempotent("monthly", "load", "k1") is None + + +def test_a_result_json_cannot_hold_is_stored_as_its_repr_and_flagged(store: RunStore) -> None: + marker = object() + run = a_run() + run.tasks["load"] = a_success(result=marker) + run.tasks["report"] = a_success("report", result=(1, {"when": START})) + run.params["client"] = marker + store.save_run(run) + store.save_task("run-1", a_success("tuple", result=(1, 2))) + loaded = store.get_run("run-1") + assert loaded.tasks["load"].result == repr(marker) + assert loaded.tasks["load"].result_is_repr is True + assert loaded.tasks["report"].result == repr((1, {"when": START})) + assert loaded.tasks["report"].result_is_repr is True + assert loaded.tasks["tuple"].result == [1, 2] # JSON has no tuple + assert loaded.tasks["tuple"].result_is_repr is False + assert loaded.params == {"date": "2026-10-08", "client": repr(marker)} + assert loaded.to_dict()["tasks"]["load"]["result_is_repr"] is True # the flag stays + + +def test_names_are_never_part_of_the_sql(store: RunStore) -> None: + hostile = "x'; DROP TABLE pipeline_runs; --" + run = a_run(hostile, hostile) + run.tasks = {hostile: a_success(hostile, key=hostile, result=hostile)} + store.save_run(run) + store.save_task(hostile, run.tasks[hostile]) + assert store.get_run(hostile).to_dict() == run.to_dict() + assert [found.run_id for found in store.list_runs(hostile)] == [hostile] + assert store.find_idempotent(hostile, hostile, hostile).result == hostile + + +def test_a_store_is_safe_to_share_between_threads(store: RunStore) -> None: + store.save_run(a_run()) + errors: list[Exception] = [] + + def write(worker_index: int) -> None: + try: + for step in range(5): + store.save_task("run-1", a_success(f"w{worker_index}-{step}", result=step)) + store.list_runs() + except Exception as error: # collected for the assertion below + errors.append(error) + + threads = [threading.Thread(target=write, args=(index,)) for index in range(4)] + for thread in threads: + thread.start() + for thread in threads: + thread.join(WAIT) + assert errors == [] + assert len(store.get_run("run-1").tasks) == 2 + 20 + + +def test_run_store_is_abstract() -> None: + with pytest.raises(TypeError): + RunStore() # type: ignore[abstract] + + +# ---------------------------------------------------------------------- memory + + +def test_the_memory_store_drops_its_oldest_runs() -> None: + store = MemoryRunStore(max_runs=2) + for index in (1, 2, 3): + store.save_run(a_run(f"run-{index}", minutes=index)) + assert [run.run_id for run in store.list_runs()] == ["run-3", "run-2"] + assert store.get_run("run-1") is None + with pytest.raises(ValueError, match="max_runs"): + MemoryRunStore(max_runs=0) + + +# ---------------------------------------------------------------------- sqlite + + +def test_the_sqlite_store_outlives_its_object(tmp_path: Path) -> None: + path = tmp_path / "deep" / "er" / "runs.db" + first = SQLiteRunStore(path) + assert first.path == path + run = a_run() + run.tasks["load"] = a_success(key="k", result={"rows": 3}) + first.save_run(run) + second = SQLiteRunStore(str(path)) + assert second.get_run("run-1").to_dict() == run.to_dict() + assert second.find_idempotent("daily", "load", "k").result == {"rows": 3} + + +def test_the_sqlite_file_carries_a_schema_version(tmp_path: Path) -> None: + path = tmp_path / "runs.db" + SQLiteRunStore(path) + with closing(sqlite3.connect(path)) as connection: + rows = connection.execute("SELECT key, value FROM pipeline_meta").fetchall() + assert rows == [("schema_version", "1")] + connection.execute( + "UPDATE pipeline_meta SET value = ? WHERE key = ?", ("99", "schema_version") + ) + connection.commit() + with pytest.raises(PipelineException, match="schema version 99 is not supported"): + SQLiteRunStore(path) + + +def test_a_sqlite_store_that_cannot_be_opened_raises(tmp_path: Path) -> None: + with pytest.raises(PipelineException, match="run store"): + SQLiteRunStore(tmp_path) # a directory, not a file + blocker = tmp_path / "file.txt" + blocker.write_text("not a directory", encoding="utf-8") + with pytest.raises(PipelineException, match="run store"): + SQLiteRunStore(blocker / "runs.db") + + +# ---------------------------------------------------------------------- the default store + + +@pytest.fixture +def fresh_default() -> Iterator[MemoryRunStore]: + fresh = MemoryRunStore() + previous = set_default_run_store(fresh) + yield fresh + set_default_run_store(previous) + + +def test_the_default_store_is_in_memory_and_can_be_replaced() -> None: + original = default_run_store() + assert isinstance(original, MemoryRunStore) + replacement = MemoryRunStore() + try: + assert set_default_run_store(replacement) is original + assert default_run_store() is replacement + with pytest.raises(PipelineException, match="not a RunStore"): + set_default_run_store("runs.db") # type: ignore[arg-type] + assert default_run_store() is replacement + finally: + set_default_run_store(original) + assert default_run_store() is original + + +def test_a_run_without_a_store_lands_in_the_default_one( + fresh_default: MemoryRunStore, bus: EventBus +) -> None: + pipeline = Pipeline("defaulted") + pipeline.task("only", lambda ctx: "ran") + run = pipeline.run(bus=bus) + assert fresh_default.get_run(run.run_id).to_dict() == run.to_dict() + assert pipeline.resume(run.run_id, bus=bus).run_id == run.run_id + + +# ---------------------------------------------------------------------- checkpoints + + +class RecordingStore(MemoryRunStore): + """A memory store that remembers every write, in order.""" + + def __init__(self) -> None: + super().__init__() + self.writes: list[tuple[Any, ...]] = [] + + def save_run(self, run: PipelineRun) -> None: + tasks = {task_id: state.status.value for task_id, state in run.tasks.items()} + self.writes.append(("run", run.status.value, tasks)) + super().save_run(run) + + def save_task(self, run_id: str, task: TaskRun) -> None: + self.writes.append(("task", task.task, task.status.value, task.attempts)) + super().save_task(run_id, task) + + +def test_every_transition_is_written_as_it_happens( + bus: EventBus, monkeypatch: pytest.MonkeyPatch +) -> None: + monkeypatch.setattr(worker, "pause", lambda _token, _seconds: False) + + def flaky(ctx: TaskContext) -> str: + if ctx.attempt == 1: + raise ConnectionError("once") + return "through" + + def boom(_ctx: TaskContext) -> None: + raise ValueError("boom") + + store = RecordingStore() + pipeline = Pipeline("checkpointed", max_workers=1) + pipeline.task("a", lambda ctx: "a") + pipeline.task("b", flaky, depends_on=["a"], retry=RetryPolicy(max_attempts=2)) + pipeline.task("c", boom, depends_on=["b"]) + pipeline.task("d", lambda ctx: "never", depends_on=["c"]) + pipeline.run(store=store, bus=bus) + assert store.writes == [ + ("run", "running", {"a": "pending", "b": "pending", "c": "pending", "d": "pending"}), + ("task", "a", "running", 1), + ("task", "a", "succeeded", 1), + ("task", "b", "running", 1), + ("task", "b", "running", 1), # the failed first attempt, with its error + ("task", "b", "running", 2), + ("task", "b", "succeeded", 2), + ("task", "c", "running", 1), + ("task", "c", "failed", 1), + ("task", "d", "skipped", 0), + ("run", "failed", {"a": "succeeded", "b": "succeeded", "c": "failed", "d": "skipped"}), + ] + + +def test_the_store_shows_a_run_in_progress(store: RunStore, bus: EventBus) -> None: + entered, release = threading.Event(), threading.Event() + + def first(_ctx: TaskContext) -> str: + entered.set() + release.wait(WAIT) + return "done" + + pipeline = Pipeline("in-progress") + pipeline.task("first", first) + pipeline.task("second", lambda ctx: "after", depends_on=["first"]) + run = pipeline.start(store=store, bus=bus) + try: + assert entered.wait(WAIT) + stored = store.get_run(run.run_id) + assert stored.status is RunStatus.RUNNING + assert stored.done is False + assert stored.tasks["first"].status is TaskStatus.RUNNING + assert stored.tasks["first"].attempts == 1 + assert stored.tasks["second"].status is TaskStatus.PENDING + asked = time.monotonic() + assert stored.wait(WAIT) is False # a snapshot: nothing will ever end it + assert time.monotonic() - asked < WAIT / 2 + finally: + release.set() + assert run.wait(WAIT) is True + assert store.get_run(run.run_id).to_dict() == run.to_dict() + assert store.get_run(run.run_id).status is RunStatus.SUCCEEDED + + +def test_a_store_that_cannot_write_does_not_stop_the_run(bus: EventBus) -> None: + class ReadOnly(MemoryRunStore): + def save_run(self, run: PipelineRun) -> None: + raise PipelineException("disk full") + + def save_task(self, run_id: str, task: TaskRun) -> None: + raise PipelineException("disk full") + + store = ReadOnly() + pipeline = Pipeline("unrecorded") + pipeline.task("a", lambda ctx: 1) + pipeline.task("b", lambda ctx: ctx.results["a"] + 1, depends_on=["a"]) + run = pipeline.run(store=store, bus=bus) + assert run.status is RunStatus.SUCCEEDED + assert run.tasks["b"].result == 2 + assert store.list_runs() == [] + + +def test_a_store_that_breaks_its_contract_fails_the_task_not_the_run_loop(bus: EventBus) -> None: + class Misbehaving(MemoryRunStore): + raised = False + + def save_task(self, run_id: str, task: TaskRun) -> None: + if not self.raised: + self.raised = True + raise RuntimeError("not a PipelineException") + super().save_task(run_id, task) + + ran: list[str] = [] + pipeline = Pipeline("misbehaving-store") + pipeline.task("only", lambda ctx: ran.append("only")) + run = pipeline.run(store=Misbehaving(), bus=bus) + assert ran == [] + assert run.tasks["only"].status is TaskStatus.FAILED + assert run.tasks["only"].error == "PipelineException: the task ended without an outcome" + assert run.status is RunStatus.FAILED + assert [event.type for event in reversed(bus.recent())] == [ + "pipeline.started", + "task.failed", + "pipeline.failed", + ] + + +# ---------------------------------------------------------------------- resume + + +def build_etl(ran: list[str], failing: frozenset[str] = frozenset()) -> Pipeline: + def step(ctx: TaskContext) -> dict[str, Any]: + ran.append(ctx.task) + if ctx.task in failing: + raise ValueError(f"{ctx.task} broke") + return {"task": ctx.task, "date": ctx.params["date"], "upstream": sorted(ctx.results)} + + pipeline = Pipeline("etl", params={"date": "a-default"}) + pipeline.task("extract", step) + pipeline.task("transform", step, depends_on=["extract"]) + pipeline.task("load", step, depends_on=["transform"]) + pipeline.task("audit", step, depends_on=["extract"]) + return pipeline + + +def test_resume_keeps_what_succeeded_and_runs_the_rest(tmp_path: Path, bus: EventBus) -> None: + path = tmp_path / "runs.db" + first_ran: list[str] = [] + first = build_etl(first_ran, frozenset({"transform"})).run( + params={"date": "2026-10-08"}, store=SQLiteRunStore(path), bus=bus + ) + assert sorted(first_ran) == ["audit", "extract", "transform"] + assert first.status is RunStatus.FAILED + assert first.tasks["load"].status is TaskStatus.SKIPPED + + # A new process: another store object on the same file, another Pipeline object. + store = SQLiteRunStore(path) + second_ran: list[str] = [] + resumed = build_etl(second_ran).resume(first.run_id, store=store, bus=bus) + assert second_ran == ["transform", "load"] + assert resumed.run_id == first.run_id + assert resumed.status is RunStatus.SUCCEEDED + assert resumed.error is None + assert resumed.params == {"date": "2026-10-08"} # the run's parameters, not the defaults + assert resumed.started_at == first.started_at + assert {state.status for state in resumed.tasks.values()} == {TaskStatus.SUCCEEDED} + assert list(resumed.tasks) == ["extract", "transform", "audit", "load"] + assert resumed.tasks["extract"].finished_at == first.tasks["extract"].finished_at + assert resumed.tasks["extract"].attempts == 1 + assert resumed.tasks["transform"].result == { + "task": "transform", + "date": "2026-10-08", + "upstream": ["extract"], + } + assert resumed.tasks["load"].result["upstream"] == ["extract", "transform"] + assert resumed.tasks["transform"].error is None + assert store.get_run(first.run_id).to_dict() == resumed.to_dict() + assert [run.run_id for run in store.list_runs()] == [first.run_id] + + +def test_resume_hands_stored_results_to_the_remaining_tasks(store: RunStore, bus: EventBus) -> None: + marker = object() + + def build(fail: bool) -> Pipeline: + def use(ctx: TaskContext) -> dict[str, Any]: + if fail: + raise ValueError("not yet") + return dict(ctx.results) + + pipeline = Pipeline("handover") + pipeline.task("numbers", lambda ctx: (1, 2, 3)) + pipeline.task("object", lambda ctx: marker) + pipeline.task("use", use, depends_on=["numbers", "object"]) + return pipeline + + first = build(fail=True).run(store=store, bus=bus) + assert first.tasks["numbers"].result == (1, 2, 3) + resumed = build(fail=False).resume(first.run_id, store=store, bus=bus) + assert resumed.tasks["use"].result == {"numbers": [1, 2, 3], "object": repr(marker)} + assert resumed.tasks["object"].result_is_repr is True + assert resumed.tasks["numbers"].result_is_repr is False + + +def test_resume_finishes_a_cancelled_run(store: RunStore, bus: EventBus) -> None: + ran: list[str] = [] + entered = threading.Event() + + def first(ctx: TaskContext) -> str: + ran.append("first") + if len(ran) == 1: # the first time round, stay until the run is cancelled + entered.set() + assert until(lambda: ctx.cancel.is_cancelled) + ctx.cancel.raise_if_cancelled() + return "first" + + def build() -> Pipeline: + pipeline = Pipeline("stopped") + pipeline.task("first", first) + pipeline.task("second", lambda ctx: ran.append("second"), depends_on=["first"]) + return pipeline + + run = build().start(store=store, bus=bus) + assert entered.wait(WAIT) + run.cancel() + assert run.wait(WAIT) is True + assert run.status is RunStatus.CANCELLED + assert store.get_run(run.run_id).status is RunStatus.CANCELLED + resumed = build().resume(run.run_id, store=store, bus=bus) + assert ran == ["first", "first", "second"] + assert resumed.status is RunStatus.SUCCEEDED + assert resumed.cancel_token is not run.cancel_token + + +def test_resume_refuses_what_it_cannot_continue(store: RunStore, bus: EventBus) -> None: + ran: list[str] = [] + pipeline = Pipeline("one") + pipeline.task("only", lambda ctx: ran.append("only")) + with pytest.raises(PipelineException, match="unknown run 'missing'"): + pipeline.resume("missing", store=store, bus=bus) + run = pipeline.run(store=store, bus=bus) + other = Pipeline("two") + other.task("only", lambda ctx: ran.append("other")) + with pytest.raises(PipelineException, match="belongs to pipeline 'one', not 'two'"): + other.resume(run.run_id, store=store, bus=bus) + assert ran == ["only"] + + +def test_resuming_a_succeeded_run_runs_nothing(store: RunStore, bus: EventBus) -> None: + ran: list[str] = [] + pipeline = Pipeline("finished") + pipeline.task("only", lambda ctx: ran.append("only")) + run = pipeline.run(store=store, bus=bus) + events_before = len(bus.recent()) + again = pipeline.resume(run.run_id, store=store, bus=bus) + assert ran == ["only"] + assert again.to_dict() == run.to_dict() + assert again.done is True + assert len(bus.recent()) == events_before + + +def test_resume_checks_the_definition_first(store: RunStore, bus: EventBus) -> None: + def boom(_ctx: TaskContext) -> None: + raise ValueError("boom") + + failing = Pipeline("changed") + failing.task("a", boom) + run = failing.run(store=store, bus=bus) + changed = Pipeline("changed") + changed.task("a", lambda ctx: None, depends_on=["gone"]) + with pytest.raises(PipelineDefinitionException, match="unknown task 'gone'"): + changed.resume(run.run_id, store=store, bus=bus) + assert store.get_run(run.run_id).status is RunStatus.FAILED + + +def test_resume_follows_a_changed_definition(store: RunStore, bus: EventBus) -> None: + def boom(_ctx: TaskContext) -> None: + raise ValueError("boom") + + before = Pipeline("evolving") + before.task("keep", lambda ctx: "kept") + before.task("drop", boom) + run = before.run(store=store, bus=bus) + after = Pipeline("evolving") + after.task("new", lambda ctx: ctx.results["keep"], depends_on=["keep"]) + after.task("keep", lambda ctx: "ran again") + resumed = after.resume(run.run_id, store=store, bus=bus) + assert list(resumed.tasks) == ["keep", "new"] + assert resumed.tasks["keep"].result == "kept" + assert resumed.tasks["new"].result == "kept" + assert list(store.get_run(run.run_id).tasks) == ["keep", "new"] + + +# ---------------------------------------------------------------------- idempotency + + +def build_billing(charged: list[str], name: str = "billing", fail: bool = False) -> Pipeline: + def charge(ctx: TaskContext) -> dict[str, str]: + charged.append(ctx.params["date"]) + if fail: + raise ValueError("card declined") + return {"charged": ctx.params["date"]} + + pipeline = Pipeline(name) + pipeline.task("charge", charge, idempotency_key="charge-${params.date}") + pipeline.task("receipt", lambda ctx: ctx.results["charge"], depends_on=["charge"]) + return pipeline + + +def test_a_key_that_already_succeeded_skips_the_task_and_reuses_its_result( + store: RunStore, bus: EventBus +) -> None: + charged: list[str] = [] + first = build_billing(charged).run(params={"date": "2026-10-08"}, store=store, bus=bus) + assert first.tasks["charge"].status is TaskStatus.SUCCEEDED + assert first.tasks["charge"].idempotency_key == "charge-2026-10-08" + + second = build_billing(charged).run(params={"date": "2026-10-08"}, store=store, bus=bus) + state = second.tasks["charge"] + assert charged == ["2026-10-08"] # not charged twice + assert state.status is TaskStatus.SKIPPED + assert state.reason == "idempotent" + assert state.result == {"charged": "2026-10-08"} + assert state.attempts == 0 + assert state.idempotency_key == "charge-2026-10-08" + assert state.satisfied is True + assert second.tasks["receipt"].status is TaskStatus.SUCCEEDED + assert second.tasks["receipt"].result == {"charged": "2026-10-08"} + assert second.status is RunStatus.SUCCEEDED + assert second.run_id != first.run_id + + third = build_billing(charged).run(params={"date": "2026-10-09"}, store=store, bus=bus) + assert charged == ["2026-10-08", "2026-10-09"] # another key + assert third.tasks["charge"].status is TaskStatus.SUCCEEDED + + fourth = build_billing(charged).run(params={"date": "2026-10-08"}, store=store, bus=bus) + assert fourth.tasks["charge"].reason == "idempotent" # a skipped run does not hide the original + assert charged == ["2026-10-08", "2026-10-09"] + + +def test_a_failed_execution_does_not_count(store: RunStore, bus: EventBus) -> None: + charged: list[str] = [] + failed = build_billing(charged, fail=True).run(params={"date": "d"}, store=store, bus=bus) + assert failed.tasks["charge"].status is TaskStatus.FAILED + assert failed.tasks["charge"].idempotency_key == "charge-d" + retried = build_billing(charged).run(params={"date": "d"}, store=store, bus=bus) + assert retried.tasks["charge"].status is TaskStatus.SUCCEEDED + assert charged == ["d", "d"] + + +def test_a_key_belongs_to_one_pipeline_and_one_task(store: RunStore, bus: EventBus) -> None: + charged: list[str] = [] + build_billing(charged, "billing").run(params={"date": "d"}, store=store, bus=bus) + other = build_billing(charged, "other-billing").run(params={"date": "d"}, store=store, bus=bus) + assert other.tasks["charge"].status is TaskStatus.SUCCEEDED + assert charged == ["d", "d"] + renamed = Pipeline("billing") + renamed.task("charge-again", lambda ctx: charged.append("again"), idempotency_key="charge-d") + assert renamed.run(store=store, bus=bus).tasks["charge-again"].status is TaskStatus.SUCCEEDED + + +def test_a_key_is_remembered_after_a_restart(tmp_path: Path, bus: EventBus) -> None: + path = tmp_path / "runs.db" + charged: list[str] = [] + build_billing(charged).run(params={"date": "d"}, store=SQLiteRunStore(path), bus=bus) + later = build_billing(charged).run(params={"date": "d"}, store=SQLiteRunStore(path), bus=bus) + assert later.tasks["charge"].reason == "idempotent" + assert later.tasks["charge"].result == {"charged": "d"} + assert charged == ["d"] + + +def test_a_key_is_rendered_as_text(store: RunStore, bus: EventBus) -> None: + pipeline = Pipeline("numbered") + pipeline.task("batch", lambda ctx: "done", idempotency_key="${params.number}") + pipeline.task("fixed", lambda ctx: "done", idempotency_key="always-the-same") + run = pipeline.run(params={"number": 5}, store=store, bus=bus) + assert run.tasks["batch"].idempotency_key == "5" + assert run.tasks["fixed"].idempotency_key == "always-the-same" + assert store.find_idempotent("numbered", "batch", "5").result == "done" + + +def test_a_condition_is_checked_before_the_key(store: RunStore, bus: EventBus) -> None: + pipeline = Pipeline("conditional-key") + pipeline.task("skipped", lambda ctx: "ran", when="on_failure", idempotency_key="k") + run = pipeline.run(store=store, bus=bus) + assert run.tasks["skipped"].reason == "condition" + assert run.tasks["skipped"].idempotency_key is None + + +def test_a_key_that_cannot_be_looked_up_fails_the_task_without_running_it(bus: EventBus) -> None: + class Offline(MemoryRunStore): + def find_idempotent(self, pipeline: str, task: str, key: str) -> TaskRun | None: + raise PipelineException("store offline") + + charged: list[str] = [] + run = build_billing(charged).run(params={"date": "d"}, store=Offline(), bus=bus) + assert charged == [] + assert run.tasks["charge"].status is TaskStatus.FAILED + assert run.tasks["charge"].error == "PipelineException: store offline" + assert run.tasks["receipt"].reason == "upstream_failed" + assert run.status is RunStatus.FAILED + + +# ---------------------------------------------------------------------- history + + +def test_the_history_of_a_pipeline(store: RunStore, bus: EventBus) -> None: + def build(name: str, fail: bool) -> Pipeline: + def step(_ctx: TaskContext) -> str: + if fail: + raise ValueError("boom") + return "ok" + + pipeline = Pipeline(name) + pipeline.task("step", step) + return pipeline + + first = build("nightly", fail=False).run(store=store, bus=bus) + second = build("nightly", fail=True).run(store=store, bus=bus) + other = build("hourly", fail=False).run(store=store, bus=bus) + history = store.list_runs("nightly") + assert [run.run_id for run in history] == [second.run_id, first.run_id] + assert [run.status for run in history] == [RunStatus.FAILED, RunStatus.SUCCEEDED] + assert history[0].tasks["step"].error == "ValueError: boom" + assert history[0].error == "did not succeed: step" + assert all(run.done for run in history) + assert [run.run_id for run in store.list_runs(limit=1)] == [other.run_id] diff --git a/tests/test_storage_actions.py b/tests/test_storage_actions.py index 3ffd27d..d36edf1 100644 --- a/tests/test_storage_actions.py +++ b/tests/test_storage_actions.py @@ -18,6 +18,7 @@ ) from automation_file.exceptions import ( StorageAlreadyExistsException, + StorageChecksumException, StorageNotEmptyException, StorageNotFoundException, StorageUnsupportedException, @@ -143,6 +144,9 @@ def test_checksum_and_verify() -> None: assert actions.storage_verify("memory://scratch/a.txt", f"md5:{md5}") is True assert actions.storage_verify("memory://scratch/a.txt", md5, algorithm="md5") is True assert actions.storage_verify("memory://scratch/a.txt", "0" * 64) is False + assert actions.storage_verify("memory://scratch/a.txt", SHA256_HELLO, strict=True) is True + with pytest.raises(StorageChecksumException, match="expected digest"): + actions.storage_verify("memory://scratch/a.txt", "0" * 64, strict=True) with pytest.raises(StorageUnsupportedException): actions.storage_checksum("memory://scratch/a.txt", "no-such-hash") From 821df87b4d922cc8e0cce24cd8e3e04aa20eac92 Mon Sep 17 00:00:00 2001 From: JeffreyChen Date: Thu, 8 Oct 2026 14:16:03 +0800 Subject: [PATCH 36/59] feat: add integrity, pipeline and audit subcommands to the CLI --- CLAUDE.md | 2 + README.md | 8 + README.zh-CN.md | 8 + README.zh-TW.md | 8 + architecture.md | 9 +- automation_file/__main__.py | 2 + automation_file/cli_common.py | 80 ++++++++ automation_file/cli_operations.py | 301 ++++++++++++++++++++++++++++++ automation_file/cli_storage.py | 27 +-- docs/source/Eng/usage/cli.rst | 69 +++++++ docs/source/Zh-CN/usage/cli.rst | 62 ++++++ docs/source/Zh-TW/usage/cli.rst | 62 ++++++ docs/updates/2026-10.md | 16 ++ docs/updates/README.md | 3 +- progress.md | 1 - tests/test_cli_operations.py | 285 ++++++++++++++++++++++++++++ 16 files changed, 917 insertions(+), 26 deletions(-) create mode 100644 automation_file/cli_common.py create mode 100644 automation_file/cli_operations.py create mode 100644 tests/test_cli_operations.py diff --git a/CLAUDE.md b/CLAUDE.md index d3a87e2..8e18255 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -11,6 +11,8 @@ automation_file/ ├── __init__.py # Public API facade (__all__); launch_ui is loaded lazily via __getattr__ ├── __main__.py # CLI entry: subcommands plus the legacy -e/-d/-c/--execute_str flags ├── cli_storage.py # the `storage` subcommand (ls, cp, mv, rm, sync, checksum, ...) +├── cli_operations.py # the `integrity`, `pipeline` and `audit` subcommands +├── cli_common.py # what the subcommands share: JSON output, --init, --audit, the cli actor ├── exceptions.py # FileAutomationException hierarchy ├── logging_config.py # file_automation_logger (file + stderr handlers) ├── core/ # Engine: action_registry (ActionRegistry, build_default_registry), action_executor diff --git a/README.md b/README.md index 6ca1414..0fc7060 100644 --- a/README.md +++ b/README.md @@ -1212,6 +1212,14 @@ python -m automation_file storage cp report.csv s3://reports/2026/report.csv python -m automation_file storage sync ./site s3://www --delete --dry-run python -m automation_file storage checksum s3://reports/2026/q1.csv +# Integrity, pipelines and the audit trail (JSON output; exit code 1 on drift or a failed run) +python -m automation_file integrity baseline s3://reports/2026 reports.baseline.json +python -m automation_file integrity verify s3://reports/2026 reports.baseline.json +python -m automation_file pipeline run daily.yaml --param date=2026-10-08 --store runs.db +python -m automation_file pipeline history --store runs.db +python -m automation_file pipeline --audit audit.sqlite run daily.yaml --store runs.db +python -m automation_file audit search --db audit.sqlite --status error --limit 20 + # Legacy flags (JSON action lists) python -m automation_file --execute_file actions.json python -m automation_file --execute_dir ./actions/ diff --git a/README.zh-CN.md b/README.zh-CN.md index 2e3c37c..ad5de3a 100644 --- a/README.zh-CN.md +++ b/README.zh-CN.md @@ -1177,6 +1177,14 @@ python -m automation_file storage cp report.csv s3://reports/2026/report.csv python -m automation_file storage sync ./site s3://www --delete --dry-run python -m automation_file storage checksum s3://reports/2026/q1.csv +# 完整性、流水线与审计轨迹（输出 JSON；出现偏移或运行失败时退出码为 1） +python -m automation_file integrity baseline s3://reports/2026 reports.baseline.json +python -m automation_file integrity verify s3://reports/2026 reports.baseline.json +python -m automation_file pipeline run daily.yaml --param date=2026-10-08 --store runs.db +python -m automation_file pipeline history --store runs.db +python -m automation_file pipeline --audit audit.sqlite run daily.yaml --store runs.db +python -m automation_file audit search --db audit.sqlite --status error --limit 20 + # 旧式标志（JSON 动作清单） python -m automation_file --execute_file actions.json python -m automation_file --execute_dir ./actions/ diff --git a/README.zh-TW.md b/README.zh-TW.md index 6add872..8fcb266 100644 --- a/README.zh-TW.md +++ b/README.zh-TW.md @@ -1177,6 +1177,14 @@ python -m automation_file storage cp report.csv s3://reports/2026/report.csv python -m automation_file storage sync ./site s3://www --delete --dry-run python -m automation_file storage checksum s3://reports/2026/q1.csv +# 完整性、管線與稽核軌跡（輸出 JSON；出現偏移或執行失敗時結束碼為 1） +python -m automation_file integrity baseline s3://reports/2026 reports.baseline.json +python -m automation_file integrity verify s3://reports/2026 reports.baseline.json +python -m automation_file pipeline run daily.yaml --param date=2026-10-08 --store runs.db +python -m automation_file pipeline history --store runs.db +python -m automation_file pipeline --audit audit.sqlite run daily.yaml --store runs.db +python -m automation_file audit search --db audit.sqlite --status error --limit 20 + # 舊式旗標（JSON 動作清單） python -m automation_file --execute_file actions.json python -m automation_file --execute_dir ./actions/ diff --git a/architecture.md b/architecture.md index f37da6c..398e6f0 100644 --- a/architecture.md +++ b/architecture.md @@ -20,7 +20,7 @@ piece of the 1.0 roadmap (`docs/FILEAUTOMATION-1.0-ROADMAP.md`, PR #107); what i | Path | Responsibility | | --- | --- | | `automation_file/__init__.py` | Public facade (`__all__`). Wires the shared `executor`, `callback_executor` and `package_manager` over one registry. `launch_ui` is loaded lazily through `__getattr__` | -| `automation_file/__main__.py`, `cli_storage.py` | CLI: legacy flags plus subcommands; `cli_storage.py` holds the `storage` subcommand | +| `automation_file/__main__.py`, `cli_*.py` | CLI: legacy flags plus subcommands. `cli_storage.py` holds `storage`, `cli_operations.py` holds `integrity`, `pipeline` and `audit`, `cli_common.py` what they share (JSON output, `--init`, `--audit`, the `cli:` actor) | | `automation_file/core/` | Engine, on je_action_core: `action_registry.py` (`ActionRegistry`, a `CommandRegistry`; `build_default_registry`), `action_executor.py` (`ActionExecutor`, an `ActionExecutor` with strict actions, indexed records and the dry-run, validate, substitute and parallel extras; shared `executor`), `callback_executor.py`, `package_loader.py`, `plugins.py`, `dag_executor.py`, `action_queue.py`, `json_store.py`, `substitution.py`. Also cross-cutting helpers: `optional` (`require_module`, the extras table), `retry`, `quota`, `rate_limit`, `circuit_breaker`, `file_lock`, `sqlite_lock`, `checksum`, `manifest`, `crypto`, `secrets`, `config`, `config_watcher`, `audit`, `metrics`, `tracing`, `progress`, `fim`, `content_store` | | `automation_file/local/` | Local strategy modules: file, dir, zip, tar and archive ops, sync, diff, text/JSON/data edits, templates, versioning, trash, `shell_ops` (argv-only subprocess), conditional branches. `safe_paths.py` guards against path traversal | | `automation_file/remote/` | `url_validator.py` (SSRF guard), `http_download.py`, `cross_backend.py`, `fsspec_bridge.py`. One subpackage per backend: `google_drive/`, `s3/`, `azure_blob/`, `dropbox_api/`, `sftp/`, `ftp/`, `onedrive/`, `box/`, each with `client.py`, `*_ops.py` and `register__ops`. `smb/` and `webdav/` have a client only | @@ -95,7 +95,12 @@ piece of the 1.0 roadmap (`docs/FILEAUTOMATION-1.0-ROADMAP.md`, PR #107); what i - subcommands `zip`, `unzip`, `download`, `create-file`, `server`, `http-server`, `ui`, `mcp`, `drive-upload`; - `storage` with its own subcommands `ls`, `stat`, `cat`, `cp`, `mv`, `rm`, `mkdir`, `sync`, `checksum`, `verify`, `schemes` (`cli_storage.py`): one JSON document per call, and `--init ` to - initialise a backend's client first. + initialise a backend's client first; + - `integrity` (`snapshot`, `baseline`, `verify`, `accept`), `pipeline` (`validate`, `run`, `status`, + `history`, `resume`; `--store ` keeps runs between commands) and `audit` (`search`, + `count`, `purge`; `--db `) in `cli_operations.py`. `integrity verify` exits 1 on drift, + `pipeline validate` on an invalid definition, `pipeline run` / `resume` unless the run succeeded; + - `--audit ` on `storage`, `integrity` and `pipeline` records the command in an audit trail. - **MCP**: `automation_file_mcp` (`automation_file.server.mcp_server:_cli`) or `python -m automation_file mcp [--allowed-actions ...]`. It is a standard-library JSON-RPC stdio server whose tools come from the registry (`tools_from_registry`). diff --git a/automation_file/__main__.py b/automation_file/__main__.py index 9025fb7..0d74585 100644 --- a/automation_file/__main__.py +++ b/automation_file/__main__.py @@ -19,6 +19,7 @@ from collections.abc import Callable from typing import Any +from automation_file.cli_operations import add_operation_commands from automation_file.cli_storage import add_storage_commands from automation_file.core.action_executor import execute_action, execute_files from automation_file.core.json_store import read_action_json @@ -226,6 +227,7 @@ def _build_parser() -> argparse.ArgumentParser: _add_server_commands(subparsers) _add_integration_commands(subparsers) add_storage_commands(subparsers) + add_operation_commands(subparsers) return parser diff --git a/automation_file/cli_common.py b/automation_file/cli_common.py new file mode 100644 index 0000000..1aef1fd --- /dev/null +++ b/automation_file/cli_common.py @@ -0,0 +1,80 @@ +"""What the subcommand modules of ``python -m automation_file`` share. + +A subcommand prints one JSON document, may run a JSON action list first +(``--init``, to initialise a backend's client) and may record what it does in an +audit trail (``--audit``). :func:`command_scope` does the last two around one +command, on behalf of the user who typed it. +""" + +from __future__ import annotations + +import argparse +import contextlib +import getpass +import json +import sys +from collections.abc import Iterator +from typing import Any + +from automation_file.events import actor_scope + +_ACTOR_PREFIX = "cli" + + +def emit(document: Any) -> int: + """Print ``document`` as indented JSON and return the exit code 0.""" + sys.stdout.write(json.dumps(document, ensure_ascii=False, indent=2, default=str) + "\n") + return 0 + + +def add_setup_arguments(parser: argparse.ArgumentParser) -> None: + """Add ``--init`` and ``--audit`` to a subcommand.""" + parser.add_argument( + "--init", + default=None, + help="JSON action list to run first, e.g. to initialise a backend's client", + ) + parser.add_argument( + "--audit", + default=None, + metavar="DB", + help="record this command's events and storage operations in this SQLite audit trail", + ) + + +def cli_actor() -> str: + """Return the actor a command runs as: ``cli:``, or ``cli`` when the user is unknown.""" + try: + return f"{_ACTOR_PREFIX}:{getpass.getuser()}" + except (OSError, KeyError, ImportError): + # getpass finds no user in a stripped environment (no USERNAME, no passwd entry). + return _ACTOR_PREFIX + + +def _run_init(raw: str | None) -> None: + if not raw: + return + from automation_file.core.action_executor import execute_action + + execute_action(json.loads(raw)) + + +@contextlib.contextmanager +def command_scope(args: argparse.Namespace) -> Iterator[None]: + """Run one command: start the audit trail it asked for, name the actor, run ``--init``. + + The trail is closed afterwards, so the database is complete when the process ends. + """ + audit_db = getattr(args, "audit", None) + trail = None + if audit_db: + from automation_file.audit import configure_audit + + trail = configure_audit(audit_db) + try: + with actor_scope(cli_actor()): + _run_init(getattr(args, "init", None)) + yield + finally: + if trail is not None: + trail.close() diff --git a/automation_file/cli_operations.py b/automation_file/cli_operations.py new file mode 100644 index 0000000..6919645 --- /dev/null +++ b/automation_file/cli_operations.py @@ -0,0 +1,301 @@ +"""``python -m automation_file integrity|pipeline|audit ...``: the runtime from a shell. + +Each subcommand is a thin call into the ``FA_*`` function of the same name and +prints one JSON document: + +.. code-block:: text + + python -m automation_file integrity baseline s3://reports/2026 reports.baseline.json + python -m automation_file integrity verify s3://reports/2026 reports.baseline.json + python -m automation_file pipeline run daily.yaml --param date=2026-10-08 --store runs.db + python -m automation_file pipeline history --store runs.db + python -m automation_file audit search --db audit.sqlite --status error --limit 20 + +Exit codes: ``integrity verify`` exits 1 when the tree drifted, ``pipeline +validate`` when the definition is invalid, ``pipeline run`` / ``resume`` when the +run did not succeed. Everything else exits 0, or 1 with the exception. + +A pipeline run is recorded in the process's memory unless ``--store`` names a +SQLite file; ``status``, ``history`` and ``resume`` need that file to find a run +of an earlier command. +""" + +from __future__ import annotations + +import argparse +import json +from datetime import datetime +from typing import Any + +from automation_file.audit import actions as audit +from automation_file.audit import audit_search, configure_audit +from automation_file.cli_common import add_setup_arguments, command_scope, emit +from automation_file.exceptions import ArgparseException +from automation_file.integrity import DEFAULT_ALGORITHM +from automation_file.integrity import actions as integrity +from automation_file.pipeline import SQLiteRunStore, set_default_run_store +from automation_file.pipeline import actions as pipeline + +_TARGET = "target" +_BASELINE = "baseline" +_DEFINITION = "definition" +_STORE = "--store" +_SUCCEEDED = "succeeded" +_SECONDS_PER_DAY = 86400.0 +_TIME_FILTERS = ("since", "until") +#: The filters of ``audit search`` / ``audit count``, as their option names say them. +_AUDIT_FILTERS = ( + "since", + "until", + "actor", + "source", + "pipeline", + "task", + "action", + "resource_prefix", + "backend", + "status", + "correlation_id", + "text", +) + +# ---------------------------------------------------------------------- integrity + + +def _cmd_snapshot(args: argparse.Namespace) -> int: + return emit(integrity.integrity_snapshot(args.target, algorithm=args.algorithm)) + + +def _cmd_baseline(args: argparse.Namespace) -> int: + return emit(integrity.integrity_baseline(args.target, args.baseline, algorithm=args.algorithm)) + + +def _cmd_verify(args: argparse.Namespace) -> int: + report = integrity.integrity_verify(args.target, args.baseline, deep=not args.quick) + emit(report) + return 0 if report["ok"] else 1 + + +def _cmd_accept(args: argparse.Namespace) -> int: + return emit(integrity.integrity_accept(args.target, args.baseline)) + + +def _add_integrity_commands(subparsers: argparse._SubParsersAction) -> None: + parser = subparsers.add_parser("integrity", help="baseline and verify a directory tree") + add_setup_arguments(parser) + commands = parser.add_subparsers(dest="integrity_command", required=True) + + snapshot = commands.add_parser("snapshot", help="hash the tree and print its manifest") + snapshot.add_argument(_TARGET) + snapshot.add_argument("--algorithm", default=DEFAULT_ALGORITHM) + snapshot.set_defaults(operation_handler=_cmd_snapshot) + + baseline = commands.add_parser("baseline", help="approve the tree as it is now") + baseline.add_argument(_TARGET) + baseline.add_argument(_BASELINE, help="where the baseline is stored (a path or a URI)") + baseline.add_argument("--algorithm", default=DEFAULT_ALGORITHM) + baseline.set_defaults(operation_handler=_cmd_baseline) + + verify = commands.add_parser("verify", help="compare the tree with its baseline") + verify.add_argument(_TARGET) + verify.add_argument(_BASELINE) + verify.add_argument( + "--quick", + action="store_true", + help="hash only the files whose size, time or etag changed", + ) + verify.set_defaults(operation_handler=_cmd_verify) + + accept = commands.add_parser("accept", help="approve the current tree as the new baseline") + accept.add_argument(_TARGET) + accept.add_argument(_BASELINE) + accept.set_defaults(operation_handler=_cmd_accept) + + parser.set_defaults(handler=_dispatch) + + +# ---------------------------------------------------------------------- pipeline + + +def _parameters(pairs: list[str] | None) -> dict[str, Any] | None: + """Turn ``name=value`` pairs into run parameters; a value that is JSON keeps its type.""" + if not pairs: + return None + parameters: dict[str, Any] = {} + for pair in pairs: + name, separator, raw = pair.partition("=") + if not separator or not name: + raise ArgparseException(f"--param takes name=value, got {pair!r}") + try: + parameters[name] = json.loads(raw) + except json.JSONDecodeError: + parameters[name] = raw + return parameters + + +def _use_store(path: str | None) -> None: + if path: + set_default_run_store(SQLiteRunStore(path)) + + +def _cmd_run(args: argparse.Namespace) -> int: + _use_store(args.store) + run = pipeline.pipeline_run( + args.definition, params=_parameters(args.param), dry_run=args.dry_run + ) + emit(run) + return 0 if run["status"] == _SUCCEEDED else 1 + + +def _cmd_validate(args: argparse.Namespace) -> int: + result = pipeline.pipeline_validate(args.definition) + emit(result) + return 0 if result["valid"] else 1 + + +def _cmd_status(args: argparse.Namespace) -> int: + _use_store(args.store) + return emit(pipeline.pipeline_status(args.run_id)) + + +def _cmd_history(args: argparse.Namespace) -> int: + _use_store(args.store) + return emit(pipeline.pipeline_history(args.pipeline, limit=args.limit)) + + +def _cmd_resume(args: argparse.Namespace) -> int: + _use_store(args.store) + run = pipeline.pipeline_resume(args.run_id, args.definition) + emit(run) + return 0 if run["status"] == _SUCCEEDED else 1 + + +def _add_pipeline_commands(subparsers: argparse._SubParsersAction) -> None: + parser = subparsers.add_parser("pipeline", help="validate, run and inspect pipelines") + add_setup_arguments(parser) + commands = parser.add_subparsers(dest="pipeline_command", required=True) + store_help = "SQLite file that records runs (default: this process's memory)" + + validate = commands.add_parser("validate", help="check a YAML or JSON definition") + validate.add_argument(_DEFINITION) + validate.set_defaults(operation_handler=_cmd_validate) + + run = commands.add_parser("run", help="run a definition") + run.add_argument(_DEFINITION) + run.add_argument( + "--param", action="append", metavar="NAME=VALUE", help="a run parameter; repeatable" + ) + run.add_argument("--dry-run", action="store_true", help="plan without executing") + run.add_argument(_STORE, default=None, help=store_help) + run.set_defaults(operation_handler=_cmd_run) + + status = commands.add_parser("status", help="show one recorded run") + status.add_argument("run_id") + status.add_argument(_STORE, default=None, help=store_help) + status.set_defaults(operation_handler=_cmd_status) + + history = commands.add_parser("history", help="list the latest recorded runs") + history.add_argument("--pipeline", default=None, help="only runs of this pipeline") + history.add_argument("--limit", type=int, default=20) + history.add_argument(_STORE, default=None, help=store_help) + history.set_defaults(operation_handler=_cmd_history) + + resume = commands.add_parser("resume", help="continue a recorded run") + resume.add_argument("run_id") + resume.add_argument(_DEFINITION) + resume.add_argument(_STORE, default=None, help=store_help) + resume.set_defaults(operation_handler=_cmd_resume) + + parser.set_defaults(handler=_dispatch) + + +# ---------------------------------------------------------------------- audit + + +def _local_time(value: str, option: str) -> str: + """Return ``value`` with a UTC offset; a time typed without one is local time.""" + try: + moment = datetime.fromisoformat(value) + except ValueError as error: + raise ArgparseException( + f"--{option} takes an ISO 8601 date or time, got {value!r}" + ) from error + return (moment if moment.tzinfo is not None else moment.astimezone()).isoformat() + + +def _filters(args: argparse.Namespace) -> dict[str, Any]: + filters = { + name: getattr(args, name) for name in _AUDIT_FILTERS if getattr(args, name) is not None + } + for name in _TIME_FILTERS: + if name in filters: + filters[name] = _local_time(filters[name], name) + return filters + + +def _cmd_search(args: argparse.Namespace) -> int: + return emit(audit_search(**_filters(args), limit=args.limit, offset=args.offset)) + + +def _cmd_count(args: argparse.Namespace) -> int: + return emit({"count": audit.audit_count(**_filters(args))}) + + +def _cmd_purge(args: argparse.Namespace) -> int: + return emit({"purged": audit.audit_purge(args.older_than_days * _SECONDS_PER_DAY)}) + + +def _add_database_argument(parser: argparse.ArgumentParser) -> None: + parser.add_argument("--db", required=True, help="the SQLite file of the audit trail") + + +def _add_filter_arguments(parser: argparse.ArgumentParser) -> None: + for name in _AUDIT_FILTERS: + parser.add_argument(f"--{name.replace('_', '-')}", dest=name, default=None) + + +def _dispatch_audit(args: argparse.Namespace) -> int: + trail = configure_audit(args.db) + try: + return int(args.operation_handler(args)) + finally: + trail.close() + + +def _add_audit_commands(subparsers: argparse._SubParsersAction) -> None: + parser = subparsers.add_parser("audit", help="search the audit trail") + commands = parser.add_subparsers(dest="audit_command", required=True) + + search = commands.add_parser("search", help="list matching records, newest first") + _add_database_argument(search) + _add_filter_arguments(search) + search.add_argument("--limit", type=int, default=50) + search.add_argument("--offset", type=int, default=0) + search.set_defaults(operation_handler=_cmd_search) + + count = commands.add_parser("count", help="count matching records") + _add_database_argument(count) + _add_filter_arguments(count) + count.set_defaults(operation_handler=_cmd_count) + + purge = commands.add_parser("purge", help="delete the records older than a number of days") + _add_database_argument(purge) + purge.add_argument("--older-than-days", type=float, required=True) + purge.set_defaults(operation_handler=_cmd_purge) + + parser.set_defaults(handler=_dispatch_audit) + + +# ---------------------------------------------------------------------- registration + + +def _dispatch(args: argparse.Namespace) -> int: + with command_scope(args): + return int(args.operation_handler(args)) + + +def add_operation_commands(subparsers: argparse._SubParsersAction) -> None: + """Register the ``integrity``, ``pipeline`` and ``audit`` subcommands.""" + _add_integrity_commands(subparsers) + _add_pipeline_commands(subparsers) + _add_audit_commands(subparsers) diff --git a/automation_file/cli_storage.py b/automation_file/cli_storage.py index 60e1dc8..748b0c2 100644 --- a/automation_file/cli_storage.py +++ b/automation_file/cli_storage.py @@ -23,10 +23,10 @@ from __future__ import annotations import argparse -import json import sys -from typing import Any +from automation_file.cli_common import add_setup_arguments, command_scope +from automation_file.cli_common import emit as _emit from automation_file.storage import actions _SOURCE = "source" @@ -34,19 +34,6 @@ _URI = "uri" -def _emit(document: Any) -> int: - sys.stdout.write(json.dumps(document, ensure_ascii=False, indent=2, default=str) + "\n") - return 0 - - -def _run_init(raw: str | None) -> None: - if not raw: - return - from automation_file.core.action_executor import execute_action - - execute_action(json.loads(raw)) - - def _cmd_ls(args: argparse.Namespace) -> int: return _emit(actions.storage_list(args.uri, recursive=args.recursive)) @@ -111,8 +98,8 @@ def _cmd_schemes(_args: argparse.Namespace) -> int: def _dispatch(args: argparse.Namespace) -> int: - _run_init(args.init) - return int(args.storage_handler(args)) + with command_scope(args): + return int(args.storage_handler(args)) def _add_read_commands(commands: argparse._SubParsersAction) -> None: @@ -181,11 +168,7 @@ def _add_write_commands(commands: argparse._SubParsersAction) -> None: def add_storage_commands(subparsers: argparse._SubParsersAction) -> None: """Register the ``storage`` subcommand and its own subcommands.""" parser = subparsers.add_parser("storage", help="files and directories in any storage backend") - parser.add_argument( - "--init", - default=None, - help="JSON action list to run first, e.g. to initialise a backend's client", - ) + add_setup_arguments(parser) commands = parser.add_subparsers(dest="storage_command", required=True) _add_read_commands(commands) _add_write_commands(commands) diff --git a/docs/source/Eng/usage/cli.rst b/docs/source/Eng/usage/cli.rst index e64ab6b..c1f81dd 100644 --- a/docs/source/Eng/usage/cli.rst +++ b/docs/source/Eng/usage/cli.rst @@ -55,3 +55,72 @@ action list that runs before the command:: python -m automation_file storage \ --init '[["FA_s3_later_init", {"region_name": "us-east-1"}]]' \ ls s3://reports + +Integrity +--------- + +The ``integrity`` subcommand baselines and verifies a directory tree in any +storage backend (:doc:`integrity`). The target and the baseline are storage URIs +or plain local paths:: + + python -m automation_file integrity baseline s3://reports/2026 reports.baseline.json + python -m automation_file integrity verify s3://reports/2026 reports.baseline.json + python -m automation_file integrity verify ./site site.baseline.json --quick + python -m automation_file integrity accept ./site site.baseline.json + python -m automation_file integrity snapshot ./site --algorithm sha512 + +``baseline`` approves the tree as it is now, ``verify`` compares it with that +baseline and prints the drift report, ``accept`` approves the current tree after a +review, and ``snapshot`` prints the manifest without storing anything. ``verify`` +exits 1 when the tree drifted, so a shell script or a CI job can gate on it; +``--quick`` hashes only the files whose size, modification time or etag changed. +``--init`` works as it does for ``storage``. Continuous monitoring and watching +need a process that stays alive: use the Python API or the ``FA_integrity_watch_*`` +actions for those. + +Pipelines +--------- + +The ``pipeline`` subcommand validates, runs and inspects pipelines written as +YAML or JSON definitions (:doc:`pipeline`):: + + python -m automation_file pipeline validate daily.yaml + python -m automation_file pipeline run daily.yaml --param date=2026-10-08 --store runs.db + python -m automation_file pipeline run daily.yaml --dry-run + python -m automation_file pipeline status --store runs.db + python -m automation_file pipeline history --pipeline daily-report --limit 10 --store runs.db + python -m automation_file pipeline resume daily.yaml --store runs.db + +``--param name=value`` may be repeated; a value that is valid JSON keeps its type +(``--param retries=3``, ``--param tags='["a","b"]'``), anything else is a string. +``validate`` exits 1 when the definition is invalid and prints every problem with +its path. ``run`` and ``resume`` print the run and exit 1 unless it succeeded. + +A run is recorded in the memory of the process that made it. Pass ``--store`` with +the path of a SQLite file to keep it: ``status``, ``history`` and ``resume`` read +the same file to find a run of an earlier command, and without it they only know +the runs of their own process, which is none. + +Audit +----- + +``--audit `` on ``storage``, ``integrity`` and ``pipeline`` records the +command's events and storage operations in an audit trail (:doc:`audit`), with +the actor ``cli:``. The ``audit`` subcommand reads that trail:: + + python -m automation_file pipeline --audit audit.sqlite run daily.yaml --store runs.db + python -m automation_file storage --audit audit.sqlite cp report.csv s3://reports/report.csv + python -m automation_file audit search --db audit.sqlite --status error --limit 20 + python -m automation_file audit search --db audit.sqlite --correlation-id + python -m automation_file audit count --db audit.sqlite --since 2026-10-01 --backend s3 + python -m automation_file audit purge --db audit.sqlite --older-than-days 90 + +``search`` prints the matching records newest first and takes ``--since``, +``--until``, ``--actor``, ``--source``, ``--pipeline``, ``--task``, ``--action``, +``--resource-prefix``, ``--backend``, ``--status``, ``--correlation-id``, +``--text``, ``--limit`` and ``--offset``. ``count`` takes the same filters. +``--since`` and ``--until`` take an ISO 8601 date or time; one without a UTC +offset is read as local time. +``purge`` deletes the records older than a number of days and prints how many it +removed. A pipeline run's ID is its correlation ID, so one search shows everything +a run did. diff --git a/docs/source/Zh-CN/usage/cli.rst b/docs/source/Zh-CN/usage/cli.rst index 4ddb8ab..e0dbd28 100644 --- a/docs/source/Zh-CN/usage/cli.rst +++ b/docs/source/Zh-CN/usage/cli.rst @@ -53,3 +53,65 @@ CLI python -m automation_file storage \ --init '[["FA_s3_later_init", {"region_name": "us-east-1"}]]' \ ls s3://reports + +完整性 +------ + +``integrity`` 子命令可以为任何存储后端中的目录树建立基准并加以验证（:doc:`integrity`）。 +目标与基准都是存储 URI 或普通的本地路径:: + + python -m automation_file integrity baseline s3://reports/2026 reports.baseline.json + python -m automation_file integrity verify s3://reports/2026 reports.baseline.json + python -m automation_file integrity verify ./site site.baseline.json --quick + python -m automation_file integrity accept ./site site.baseline.json + python -m automation_file integrity snapshot ./site --algorithm sha512 + +``baseline`` 把目录树当前的状态核准为基准，``verify`` 把它与基准比对并输出偏移报告， +``accept`` 在检查之后把当前的状态核准为新基准，``snapshot`` 只输出 manifest 而不存储 +任何东西。``verify`` 在目录树出现偏移时以 1 退出，因此 shell 脚本或 CI 作业可以据此 +把关；``--quick`` 只对大小、修改时间或 etag 有变动的文件计算哈希。``--init`` 的用法与 +``storage`` 相同。持续监控与监听需要一个持续存活的进程：请使用 Python API 或 +``FA_integrity_watch_*`` 动作。 + +流水线 +------ + +``pipeline`` 子命令可以验证、运行并查看以 YAML 或 JSON 定义编写的流水线 +（:doc:`pipeline`）:: + + python -m automation_file pipeline validate daily.yaml + python -m automation_file pipeline run daily.yaml --param date=2026-10-08 --store runs.db + python -m automation_file pipeline run daily.yaml --dry-run + python -m automation_file pipeline status --store runs.db + python -m automation_file pipeline history --pipeline daily-report --limit 10 --store runs.db + python -m automation_file pipeline resume daily.yaml --store runs.db + +``--param name=value`` 可以重复指定；值如果是合法的 JSON 会保留其类型 +（``--param retries=3``、``--param tags='["a","b"]'``），其余一律视为字符串。 +``validate`` 在定义无效时以 1 退出，并输出每个问题及其路径。``run`` 与 ``resume`` +会输出该次运行，除非运行成功，否则以 1 退出。 + +运行记录默认只存在于运行它的进程的内存中。传入 ``--store`` 与一个 SQLite 文件的 +路径即可保存：``status``、``history`` 与 ``resume`` 会读取同一个文件来找到先前命令 +的运行；没有它，这些命令只知道自己进程中的运行，也就是没有。 + +审计 +---- + +在 ``storage``、``integrity`` 与 ``pipeline`` 加上 ``--audit <文件>``，就会把该命令的 +事件与存储操作记录到审计轨迹（:doc:`audit`），actor 为 ``cli:<用户>``。``audit`` +子命令用来读取这份轨迹:: + + python -m automation_file pipeline --audit audit.sqlite run daily.yaml --store runs.db + python -m automation_file storage --audit audit.sqlite cp report.csv s3://reports/report.csv + python -m automation_file audit search --db audit.sqlite --status error --limit 20 + python -m automation_file audit search --db audit.sqlite --correlation-id + python -m automation_file audit count --db audit.sqlite --since 2026-10-01 --backend s3 + python -m automation_file audit purge --db audit.sqlite --older-than-days 90 + +``search`` 由新到旧输出符合条件的记录，可以使用 ``--since``、``--until``、``--actor``、 +``--source``、``--pipeline``、``--task``、``--action``、``--resource-prefix``、 +``--backend``、``--status``、``--correlation-id``、``--text``、``--limit`` 与 +``--offset``。``count`` 接受相同的筛选条件。``--since`` 与 ``--until`` 接受 ISO 8601 的 +日期或时间，没有 UTC 偏移时视为本地时间。``purge`` 会删除早于指定天数的记录，并输出 +删除的条数。流水线运行的 ID 就是它的关联 ID，因此一次搜索就能看到一次运行所做的一切。 diff --git a/docs/source/Zh-TW/usage/cli.rst b/docs/source/Zh-TW/usage/cli.rst index 738d21b..1c2d43c 100644 --- a/docs/source/Zh-TW/usage/cli.rst +++ b/docs/source/Zh-TW/usage/cli.rst @@ -53,3 +53,65 @@ CLI python -m automation_file storage \ --init '[["FA_s3_later_init", {"region_name": "us-east-1"}]]' \ ls s3://reports + +完整性 +------ + +``integrity`` 子指令可為任何儲存後端中的目錄樹建立基準並加以驗證（:doc:`integrity`）。 +目標與基準都是儲存 URI 或一般的本機路徑:: + + python -m automation_file integrity baseline s3://reports/2026 reports.baseline.json + python -m automation_file integrity verify s3://reports/2026 reports.baseline.json + python -m automation_file integrity verify ./site site.baseline.json --quick + python -m automation_file integrity accept ./site site.baseline.json + python -m automation_file integrity snapshot ./site --algorithm sha512 + +``baseline`` 把目錄樹目前的狀態核可為基準，``verify`` 把它與基準比對並輸出偏移報告， +``accept`` 在檢視之後把目前的狀態核可為新基準，``snapshot`` 只輸出 manifest 而不儲存 +任何東西。``verify`` 在目錄樹出現偏移時以 1 結束，因此 shell 腳本或 CI 工作可以據此 +把關；``--quick`` 只對大小、修改時間或 etag 有變動的檔案計算雜湊。``--init`` 的用法與 +``storage`` 相同。持續監控與監看需要一個持續存活的行程：請使用 Python API 或 +``FA_integrity_watch_*`` 動作。 + +管線 +---- + +``pipeline`` 子指令可驗證、執行並檢視以 YAML 或 JSON 定義撰寫的管線 +（:doc:`pipeline`）:: + + python -m automation_file pipeline validate daily.yaml + python -m automation_file pipeline run daily.yaml --param date=2026-10-08 --store runs.db + python -m automation_file pipeline run daily.yaml --dry-run + python -m automation_file pipeline status --store runs.db + python -m automation_file pipeline history --pipeline daily-report --limit 10 --store runs.db + python -m automation_file pipeline resume daily.yaml --store runs.db + +``--param name=value`` 可以重複指定；值若是合法的 JSON 會保留其型別 +（``--param retries=3``、``--param tags='["a","b"]'``），其餘一律視為字串。 +``validate`` 在定義無效時以 1 結束，並輸出每個問題及其路徑。``run`` 與 ``resume`` +會輸出該次執行，除非執行成功，否則以 1 結束。 + +執行紀錄預設只存在於執行它的行程的記憶體中。傳入 ``--store`` 與一個 SQLite 檔案的 +路徑即可保存：``status``、``history`` 與 ``resume`` 會讀取同一個檔案來找到先前指令 +的執行；沒有它，這些指令只知道自己行程中的執行，也就是沒有。 + +稽核 +---- + +在 ``storage``、``integrity`` 與 ``pipeline`` 加上 ``--audit <檔案>``，就會把該指令的 +事件與儲存操作記錄到稽核軌跡（:doc:`audit`），actor 為 ``cli:<使用者>``。``audit`` +子指令用來讀取這份軌跡:: + + python -m automation_file pipeline --audit audit.sqlite run daily.yaml --store runs.db + python -m automation_file storage --audit audit.sqlite cp report.csv s3://reports/report.csv + python -m automation_file audit search --db audit.sqlite --status error --limit 20 + python -m automation_file audit search --db audit.sqlite --correlation-id + python -m automation_file audit count --db audit.sqlite --since 2026-10-01 --backend s3 + python -m automation_file audit purge --db audit.sqlite --older-than-days 90 + +``search`` 由新到舊輸出符合條件的紀錄，可使用 ``--since``、``--until``、``--actor``、 +``--source``、``--pipeline``、``--task``、``--action``、``--resource-prefix``、 +``--backend``、``--status``、``--correlation-id``、``--text``、``--limit`` 與 +``--offset``。``count`` 接受相同的篩選條件。``--since`` 與 ``--until`` 接受 ISO 8601 的 +日期或時間，沒有 UTC 偏移時視為本地時間。``purge`` 會刪除早於指定天數的紀錄，並輸出 +刪除的筆數。管線執行的 ID 就是它的關聯 ID，因此一次搜尋就能看到一次執行所做的一切。 diff --git a/docs/updates/2026-10.md b/docs/updates/2026-10.md index c42965e..a557672 100644 --- a/docs/updates/2026-10.md +++ b/docs/updates/2026-10.md @@ -475,3 +475,19 @@ Index and query commands: [README.md](README.md). New entries go at the end. - **Result / numbers**: part of the run recorded in U-20261008-18. - **Files**: `automation_file/server/action_acl.py`, `automation_file/server/mcp_server.py`, `tests/test_action_acl.py`, `tests/test_mcp_server.py`, the three `usage/pipeline.rst`, `CLAUDE.md`. - **Open items**: none. + +## U-20261008-20 · 2026-10-08 · CLI subcommands for integrity, pipelines and the audit trail · #cli #roadmap #done + +- **What**: three subcommands of `python -m automation_file`, each a thin call into the `FA_*` function of the same name that prints one JSON document. Closes `progress.md` #33. + - `integrity snapshot | baseline | verify | accept` on a target and a baseline given as storage URIs or local paths. `verify` exits 1 when the tree drifted; `--quick` hashes only what changed by size, time or etag. + - `pipeline validate | run | status | history | resume` on a YAML or JSON definition. `--param name=value` is repeatable and a value that is JSON keeps its type. `validate` exits 1 on an invalid definition; `run` and `resume` exit 1 unless the run succeeded. `--store ` keeps runs between commands (`SQLiteRunStore`); without it a run lives in the memory of the command that made it. + - `audit search | count | purge` with `--db ` and one option per filter of `audit_search`. `--since` and `--until` given without a UTC offset are read as local time; the Python API still requires a zone. + - `--audit ` on `storage`, `integrity` and `pipeline` records the command's events and storage operations in an audit trail, which is closed when the command ends. Every command runs as the actor `cli:`. + - `automation_file/cli_common.py` holds what the subcommand modules share; `cli_storage.py` now uses it, with unchanged behaviour. +- **Checked while writing it**: a storage operation inside a pipeline task carries the run ID as its correlation ID and the caller's actor, although the task runs on another thread. So `audit search --correlation-id ` shows everything a run did. +- **Tests**: `tests/test_cli_operations.py`, 12 cases through `main([...])`: baseline, verify, accept and the exit code on drift, a snapshot that stores nothing, a `memory://` target, validation problems, a run found again by `status` and `history` through the store file only, a dry run, a failed run that is resumed, parameter types, a command recorded with `--audit` and found by `audit search`, paging and purge, a local time, the required `--db`. `tests/test_cli_storage.py` and `tests/test_legacy_cli_contract.py` pass unchanged. +- **Result / numbers**: 4837 passed, 149 skipped, 0 failed with every extra; 3120 passed, 89 skipped with the base dependencies only. `ruff check`, `ruff format --check` and `mypy automation_file` (231 files) pass. Python 3.14.7 on Windows. +- **Not done**: a foreground `integrity watch`. Watching and continuous monitoring need a process that stays alive; the manual points to the Python API and the `FA_integrity_watch_*` actions. +- **Docs**: three new sections in the three `usage/cli.rst` pages, the CLI block of the three READMEs, `architecture.md` §2 and §3, `CLAUDE.md` (package map). +- **Files**: `automation_file/cli_common.py`, `automation_file/cli_operations.py`, `automation_file/cli_storage.py`, `automation_file/__main__.py`, `tests/test_cli_operations.py`, the documentation above. +- **Open items**: none. diff --git a/docs/updates/README.md b/docs/updates/README.md index ad74f6f..aafc109 100644 --- a/docs/updates/README.md +++ b/docs/updates/README.md @@ -58,6 +58,7 @@ In the same commit: delete the item from `progress.md`, add a `#done` entry here | ID | Date | Title | Tags | Batch | |---|---|---|---|---| +| U-20261008-20 | 2026-10-08 | CLI subcommands for integrity, pipelines and the audit trail | #cli #roadmap #done | [2026-10](2026-10.md) | | U-20261008-19 | 2026-10-08 | The action ACL and the MCP server check nested action names | #security #incident | [2026-10](2026-10.md) | | U-20261008-18 | 2026-10-08 | Pipeline runtime | #pipeline #roadmap #done | [2026-10](2026-10.md) | | U-20261008-17 | 2026-10-08 | Notification router and audit schema v2 | #notify #audit #roadmap | [2026-10](2026-10.md) | @@ -114,5 +115,5 @@ In the same commit: delete the item from `progress.md`, add a `#done` entry here | File | Period | Entries | |---|---|---:| -| [2026-10.md](2026-10.md) | 2026-10 | 30 | +| [2026-10.md](2026-10.md) | 2026-10 | 31 | | [2026-09.md](2026-09.md) | 2026-09 | 21 | diff --git a/progress.md b/progress.md index 84f0e9b..3c460e0 100644 --- a/progress.md +++ b/progress.md @@ -27,7 +27,6 @@ Items #10 to #26 are what is left of the 1.0 roadmap (`docs/FILEAUTOMATION-1.0-R ### Later milestones -- **#33** CLI subcommands for the packages that have none: `integrity` (snapshot, baseline, verify, accept, status), `pipeline` and `audit`, as thin calls into their `FA_*` functions like `storage` (U-20261008-11). - **#23** Scheduler v2 (roadmap §8, the open half of M6): one scheduler with cron (time-zone aware), manual, file-event, webhook and pipeline-dependency triggers, run states and overlap protection, reading a pipeline's `schedule`. The event model, the `NotificationRouter` and audit schema v2 are done (U-20261008-05, U-20261008-17); the scheduler in `scheduler/` still dispatches action lists on its own cron loop. - **#24** UI 2.0 (roadmap §11, M7). Not before the APIs of #13 to #23 are stable (roadmap §20). - **#25** Semantic MCP tools (roadmap §12, M8): `file_*`, `storage_*`, `pipeline_*`, `integrity_status`, `audit_search`, with a permission model and dry run, next to the existing `FA_*` bridge. diff --git a/tests/test_cli_operations.py b/tests/test_cli_operations.py new file mode 100644 index 0000000..4397dd6 --- /dev/null +++ b/tests/test_cli_operations.py @@ -0,0 +1,285 @@ +"""The ``integrity``, ``pipeline`` and ``audit`` subcommands: JSON out, exit codes that mean something.""" + +from __future__ import annotations + +import json +from collections.abc import Iterator +from pathlib import Path +from typing import Any + +import pytest + +from automation_file.__main__ import main +from automation_file.audit import audit_trail +from automation_file.cli_common import cli_actor +from automation_file.cli_operations import _parameters +from automation_file.exceptions import ArgparseException +from automation_file.pipeline import MemoryRunStore, PipelineException, set_default_run_store +from automation_file.storage import clear_memory_stores, memory_store + + +@pytest.fixture(autouse=True) +def _isolated() -> Iterator[None]: + clear_memory_stores() + previous = set_default_run_store(MemoryRunStore()) + yield + set_default_run_store(previous) + audit_trail.close() + clear_memory_stores() + + +def _run(capsys: pytest.CaptureFixture[str], *argv: str) -> tuple[int, Any]: + code = main(list(argv)) + return code, json.loads(capsys.readouterr().out) + + +def _forget_runs() -> None: + """Stand in for a new process: the next command has only what its store file holds.""" + set_default_run_store(MemoryRunStore()) + + +def _definition(tmp_path: Path, name: str = "cli-notes", source: str = "a.txt") -> Path: + document = { + "schema_version": 1, + "name": name, + "params": {"text": "default"}, + "tasks": { + "write": { + "action": [ + "FA_storage_write_text", + {"uri": "memory://cli-pipe/a.txt", "text": "${params.text}"}, + ] + }, + "read": { + "action": ["FA_storage_read_text", {"uri": f"memory://cli-pipe/{source}"}], + "depends_on": ["write"], + }, + }, + } + path = tmp_path / f"{name}.json" + path.write_text(json.dumps(document), encoding="utf-8") + return path + + +# ---------------------------------------------------------------------- integrity + + +def test_integrity_baseline_verify_and_accept( + capsys: pytest.CaptureFixture[str], tmp_path: Path +) -> None: + tree = tmp_path / "tree" + tree.mkdir() + (tree / "a.txt").write_text("alpha", encoding="utf-8") + baseline = tmp_path / "baseline.json" + + code, stored = _run(capsys, "integrity", "baseline", str(tree), str(baseline)) + assert code == 0 + assert baseline.is_file() + assert stored["algorithm"] == "sha256" + + code, report = _run(capsys, "integrity", "verify", str(tree), str(baseline)) + assert (code, report["ok"], report["deep"]) == (0, True, True) + + (tree / "a.txt").write_text("changed", encoding="utf-8") + (tree / "new.txt").write_text("new", encoding="utf-8") + code, report = _run(capsys, "integrity", "verify", str(tree), str(baseline)) + assert code == 1 + assert (report["counts"]["modified"], report["counts"]["created"]) == (1, 1) + + code, _ = _run(capsys, "integrity", "accept", str(tree), str(baseline)) + assert code == 0 + code, report = _run(capsys, "integrity", "verify", str(tree), str(baseline), "--quick") + assert (code, report["ok"], report["deep"]) == (0, True, False) + + +def test_integrity_snapshot_prints_the_manifest_and_stores_nothing( + capsys: pytest.CaptureFixture[str], tmp_path: Path +) -> None: + tree = tmp_path / "tree" + tree.mkdir() + (tree / "報告.txt").write_text("alpha", encoding="utf-8") + code, snapshot = _run(capsys, "integrity", "snapshot", str(tree), "--algorithm", "sha512") + assert code == 0 + assert snapshot["algorithm"] == "sha512" + assert [entry["path"] for entry in snapshot["entries"]] == ["報告.txt"] + assert sorted(path.name for path in tmp_path.iterdir()) == ["tree"] + + +def test_integrity_works_on_a_storage_uri( + capsys: pytest.CaptureFixture[str], tmp_path: Path +) -> None: + memory_store("cli-tree").write_bytes("docs/a.txt", b"alpha") + baseline = tmp_path / "memory.baseline.json" + code, _ = _run(capsys, "integrity", "baseline", "memory://cli-tree/docs", str(baseline)) + assert code == 0 + memory_store("cli-tree").delete("docs/a.txt") + code, report = _run(capsys, "integrity", "verify", "memory://cli-tree/docs", str(baseline)) + assert code == 1 + assert report["counts"]["deleted"] == 1 + + +# ---------------------------------------------------------------------- pipeline + + +def test_pipeline_validate_says_what_is_wrong( + capsys: pytest.CaptureFixture[str], tmp_path: Path +) -> None: + code, result = _run(capsys, "pipeline", "validate", str(_definition(tmp_path))) + assert (code, result) == (0, {"valid": True, "errors": []}) + + broken = tmp_path / "broken.json" + broken.write_text( + json.dumps( + { + "schema_version": 1, + "name": "broken", + "tasks": {"a": {"action": ["FA_storage_schemes"], "depends_on": ["nowhere"]}}, + } + ), + encoding="utf-8", + ) + code, result = _run(capsys, "pipeline", "validate", str(broken)) + assert code == 1 + assert result["valid"] is False + assert any("nowhere" in problem for problem in result["errors"]) + + +def test_pipeline_run_status_and_history_share_a_store_file( + capsys: pytest.CaptureFixture[str], tmp_path: Path +) -> None: + definition, store = str(_definition(tmp_path)), str(tmp_path / "runs.db") + code, run = _run( + capsys, "pipeline", "run", definition, "--param", "text=hello", "--store", store + ) + assert code == 0 + assert run["status"] == "succeeded" + assert run["tasks"]["read"]["result"] == "hello" + + _forget_runs() + code, status = _run(capsys, "pipeline", "status", run["run_id"], "--store", store) + assert (code, status["run_id"], status["status"]) == (0, run["run_id"], "succeeded") + + _forget_runs() + code, history = _run(capsys, "pipeline", "history", "--pipeline", "cli-notes", "--store", store) + assert code == 0 + assert [entry["run_id"] for entry in history] == [run["run_id"]] + + _forget_runs() + with pytest.raises(PipelineException, match="unknown run"): + main(["pipeline", "status", run["run_id"]]) + + +def test_pipeline_dry_run_plans_and_executes_nothing( + capsys: pytest.CaptureFixture[str], tmp_path: Path +) -> None: + code, plan = _run(capsys, "pipeline", "run", str(_definition(tmp_path)), "--dry-run") + assert code == 0 + assert {state["status"] for state in plan["tasks"].values()} == {"planned"} + assert not memory_store("cli-pipe").exists("a.txt") + + +def test_a_failed_run_exits_1_and_can_be_resumed( + capsys: pytest.CaptureFixture[str], tmp_path: Path +) -> None: + definition = str(_definition(tmp_path, name="cli-late", source="late.txt")) + store = str(tmp_path / "runs.db") + code, run = _run(capsys, "pipeline", "run", definition, "--store", store) + assert code == 1 + assert (run["status"], run["tasks"]["read"]["status"]) == ("failed", "failed") + + memory_store("cli-pipe").write_bytes("late.txt", b"arrived") + _forget_runs() + code, resumed = _run(capsys, "pipeline", "resume", run["run_id"], definition, "--store", store) + assert code == 0 + assert resumed["tasks"]["read"]["result"] == "arrived" + + +def test_run_parameters_keep_json_types() -> None: + assert _parameters(None) is None + assert _parameters(["a=1", "b=x", 'c={"k": [1]}', "d=", "e=a=b"]) == { + "a": 1, + "b": "x", + "c": {"k": [1]}, + "d": "", + "e": "a=b", + } + for pair in ("novalue", "=x"): + with pytest.raises(ArgparseException, match="name=value"): + _parameters([pair]) + + +# ---------------------------------------------------------------------- audit + + +def test_a_command_run_with_audit_is_found_by_audit_search( + capsys: pytest.CaptureFixture[str], tmp_path: Path +) -> None: + database = str(tmp_path / "audit.sqlite") + memory_store("cli-audit").write_bytes("a.txt", b"alpha") + code, _ = _run( + capsys, + "storage", + "--audit", + database, + "cp", + "memory://cli-audit/a.txt", + "memory://cli-audit/b.txt", + ) + assert code == 0 + assert audit_trail.active is False + + code, records = _run( + capsys, "audit", "search", "--db", database, "--resource-prefix", "memory://cli-audit/" + ) + assert code == 0 + copies = [record for record in records if record["action"] == "copy"] + assert [record["resource"] for record in copies] == ["memory://cli-audit/b.txt"] + assert copies[0]["actor"] == cli_actor() + assert copies[0]["status"] == "ok" + + code, counted = _run(capsys, "audit", "count", "--db", database, "--action", "copy") + assert (code, counted) == (0, {"count": 1}) + code, counted = _run(capsys, "audit", "count", "--db", database, "--status", "error") + assert (code, counted) == (0, {"count": 0}) + + +def test_audit_search_pages_and_purge_empties( + capsys: pytest.CaptureFixture[str], tmp_path: Path +) -> None: + database = str(tmp_path / "audit.sqlite") + store = memory_store("cli-audit") + for name in ("a", "b", "c"): + store.write_bytes(f"{name}.txt", b"x") + main(["storage", "--audit", database, "cat", f"memory://cli-audit/{name}.txt"]) + capsys.readouterr() + + code, first = _run(capsys, "audit", "search", "--db", database, "--limit", "2") + code, rest = _run(capsys, "audit", "search", "--db", database, "--limit", "2", "--offset", "2") + assert (len(first), len(rest)) == (2, 1) + assert {record["id"] for record in first}.isdisjoint(record["id"] for record in rest) + + code, purged = _run(capsys, "audit", "purge", "--db", database, "--older-than-days", "30") + assert (code, purged) == (0, {"purged": 0}) + + +def test_a_time_without_a_zone_is_local_time( + capsys: pytest.CaptureFixture[str], tmp_path: Path +) -> None: + database = str(tmp_path / "audit.sqlite") + memory_store("cli-audit").write_bytes("a.txt", b"x") + main(["storage", "--audit", database, "cat", "memory://cli-audit/a.txt"]) + capsys.readouterr() + code, counted = _run(capsys, "audit", "count", "--db", database, "--since", "2020-01-01") + assert (code, counted) == (0, {"count": 1}) + code, counted = _run( + capsys, "audit", "count", "--db", database, "--until", "2020-01-01T00:00:00+00:00" + ) + assert (code, counted) == (0, {"count": 0}) + with pytest.raises(ArgparseException, match="--since takes an ISO 8601"): + main(["audit", "count", "--db", database, "--since", "yesterday"]) + + +def test_the_audit_database_is_required(capsys: pytest.CaptureFixture[str]) -> None: + with pytest.raises(SystemExit): + main(["audit", "search"]) + assert "--db" in capsys.readouterr().err From ba7c455a295787d52b1ee081a00b12844bc97cd7 Mon Sep 17 00:00:00 2001 From: JeffreyChen Date: Thu, 8 Oct 2026 14:23:26 +0800 Subject: [PATCH 37/59] refactor: run copy_between on the storage layer --- README.md | 2 +- README.zh-CN.md | 2 +- README.zh-TW.md | 2 +- automation_file/remote/cross_backend.py | 239 +++++++++++++----------- docs/source/Eng/usage/cloud.rst | 41 ++-- docs/source/Zh-CN/usage/cloud.rst | 34 +++- docs/source/Zh-TW/usage/cloud.rst | 34 +++- docs/updates/2026-10.md | 19 ++ docs/updates/README.md | 3 +- progress.md | 1 - tests/test_cross_backend_storage.py | 208 +++++++++++++++++++++ 11 files changed, 436 insertions(+), 149 deletions(-) create mode 100644 tests/test_cross_backend_storage.py diff --git a/README.md b/README.md index 0fc7060..7304d5a 100644 --- a/README.md +++ b/README.md @@ -32,7 +32,7 @@ facade. - **Config hot reload** — `ConfigWatcher` polls `automation_file.toml` and re-applies sinks / defaults on change without restart - **Shell / grep / JSON edit / tar / backup rotation** — `FA_run_shell` (argument-list subprocess with timeout), `FA_grep` (streaming text search), `FA_json_get` / `FA_json_set` / `FA_json_delete` (in-place JSON editing), `FA_create_tar` / `FA_extract_tar`, `FA_rotate_backups` - **FTP / FTPS backend** — plain FTP or explicit FTPS via `FTP_TLS.auth()`; auto-registered as `FA_ftp_*` -- **Cross-backend copy** — `FA_copy_between` moves data between any two backends via `local://`, `s3://`, `azure://`, `dropbox://`, `sftp://`, `ftp://` URIs +- **Cross-backend copy** — `FA_copy_between` copies a file between any two storage locations (`local://`, `s3://`, `azure://`, `gdrive://`, `dropbox://`, `sftp://`, `ftp://`, a mount, or an `http(s)://` source) on the storage layer; the older `s3:bucket/key` and `sftp:/path` spellings still work - **Scheduler overlap guard** — running jobs are skipped on the next fire unless `allow_overlap=True` - **Server action ACL** — `allowed_actions=(...)` restricts which commands TCP / HTTP servers will dispatch - **Variable substitution** — opt-in `${env:VAR}` / `${date:%Y-%m-%d}` / `${uuid}` / `${cwd}` expansion in action arguments via `execute_action(..., substitute=True)` diff --git a/README.zh-CN.md b/README.zh-CN.md index ad5de3a..cba460f 100644 --- a/README.zh-CN.md +++ b/README.zh-CN.md @@ -30,7 +30,7 @@ TCP / HTTP 服务器执行的 JSON 驱动动作。内附 PySide6 GUI，每个功 - **配置热加载** — `ConfigWatcher` 轮询 `automation_file.toml`，变更时即时应用 sink / 默认值,无需重启 - **Shell / grep / JSON 编辑 / tar / 备份轮转** — `FA_run_shell`(参数列表式 subprocess,含超时)、`FA_grep`(流式文本搜索)、`FA_json_get` / `FA_json_set` / `FA_json_delete`(原地 JSON 编辑)、`FA_create_tar` / `FA_extract_tar`、`FA_rotate_backups` - **FTP / FTPS 后端** — 纯 FTP 或通过 `FTP_TLS.auth()` 的显式 FTPS;自动注册为 `FA_ftp_*` -- **跨后端复制** — `FA_copy_between` 通过 `local://`、`s3://`、`azure://`、`dropbox://`、`sftp://`、`ftp://` URI 在任意两个后端之间搬运数据 +- **跨后端复制** — `FA_copy_between` 建立在存储层之上，在任意两个存储位置之间复制文件（`local://`、`s3://`、`azure://`、`gdrive://`、`dropbox://`、`sftp://`、`ftp://`、挂载点，或以 `http(s)://` 作为来源）；旧的 `s3:bucket/key` 与 `sftp:/path` 写法仍然可用 - **调度器重叠防护** — 正在执行的作业在下次触发时会被跳过,除非显式传入 `allow_overlap=True` - **服务器动作 ACL** — `allowed_actions=(...)` 限制 TCP / HTTP 服务器可派发的命令 - **变量替换** — 动作参数中可选使用 `${env:VAR}` / `${date:%Y-%m-%d}` / `${uuid}` / `${cwd}`,通过 `execute_action(..., substitute=True)` 展开 diff --git a/README.zh-TW.md b/README.zh-TW.md index 8fcb266..1531004 100644 --- a/README.zh-TW.md +++ b/README.zh-TW.md @@ -30,7 +30,7 @@ TCP / HTTP 伺服器執行的 JSON 驅動動作。內附 PySide6 GUI，每個功 - **設定熱重載** — `ConfigWatcher` 輪詢 `automation_file.toml`，變更時即時套用 sink / 預設值，無需重啟 - **Shell / grep / JSON 編輯 / tar / 備份輪替** — `FA_run_shell`（參數列表式 subprocess，含逾時）、`FA_grep`（串流文字搜尋）、`FA_json_get` / `FA_json_set` / `FA_json_delete`（原地 JSON 編輯）、`FA_create_tar` / `FA_extract_tar`、`FA_rotate_backups` - **FTP / FTPS 後端** — 純 FTP 或透過 `FTP_TLS.auth()` 的顯式 FTPS；自動註冊為 `FA_ftp_*` -- **跨後端複製** — `FA_copy_between` 透過 `local://`、`s3://`、`azure://`、`dropbox://`、`sftp://`、`ftp://` URI 在任意兩個後端之間搬運資料 +- **跨後端複製** — `FA_copy_between` 建立在儲存層之上，在任意兩個儲存位置之間複製檔案（`local://`、`s3://`、`azure://`、`gdrive://`、`dropbox://`、`sftp://`、`ftp://`、掛載點，或以 `http(s)://` 作為來源）；舊的 `s3:bucket/key` 與 `sftp:/path` 寫法仍然可用 - **排程器重疊防護** — 正在執行的工作在下次觸發時會被跳過，除非明確傳入 `allow_overlap=True` - **伺服器動作 ACL** — `allowed_actions=(...)` 限制 TCP / HTTP 伺服器可派送的指令 - **變數替換** — 動作參數中可選使用 `${env:VAR}` / `${date:%Y-%m-%d}` / `${uuid}` / `${cwd}`，透過 `execute_action(..., substitute=True)` 展開 diff --git a/automation_file/remote/cross_backend.py b/automation_file/remote/cross_backend.py index 0f80c1f..51eaffa 100644 --- a/automation_file/remote/cross_backend.py +++ b/automation_file/remote/cross_backend.py @@ -1,128 +1,159 @@ -"""Cross-backend copy — stream a file from one storage backend to another. - -``copy_between(source, target)`` resolves each URI to (backend, parameters), -downloads the source to a private temp file, then uploads from that temp -file to the target. Every backend already exposes ``*_download_file`` / -``*_upload_file`` primitives — this module just picks the right pair and -cleans up the intermediate temp file. - -Supported URI schemes: - -* ``local:/absolute/path`` or a bare filesystem path -* ``s3://bucket/key`` -* ``azure://container/blob`` -* ``dropbox:/path`` -* ``sftp:/remote/path`` -* ``ftp:/remote/path`` -* ``http://...`` / ``https://...`` (source only) - -Callers must have previously initialised every backend they reference -(``s3_instance.later_init``, etc.) — this helper does not manage sessions. +"""Cross-backend copy: one file from a storage location to another. + +``copy_between(source, target)`` is the older spelling of +``File(source).copy_to(target)``. It resolves both locations to a storage +backend and lets the storage layer do the transfer, so the copy is native where +two backends can do it between themselves, is reported to the storage +observers, and shows up in the audit trail. + +Locations it accepts: + +* any storage URI (``s3://bucket/key``, ``azure://container/blob``, + ``gdrive:///path``, ``sftp://host/absolute/path``, ``memory://name/path``, + a mounted prefix, ...), see :mod:`automation_file.storage`; +* a bare filesystem path, ``local:`` or ``local:/absolute/path``; +* ``s3:bucket/key`` and ``azure:container/blob`` without the slashes; +* ``dropbox:/path``; +* ``sftp:/path`` and ``ftp:/path`` with one slash or none: the path is taken + relative to the directory the session logged in to, as it always was. With + two slashes (``sftp://host/path``) the URI names a host and an absolute path; +* ``http://...`` / ``https://...`` as a source only, fetched with + :func:`automation_file.remote.http_download.download_file` and its SSRF guard. + +Every backend involved must have been initialised (``s3_instance.later_init``, +...). The function returns ``False`` when the transfer itself fails (a missing +source, a refused write), and raises for a location it cannot make sense of +(:class:`CrossBackendException`) or a backend that is not initialised. """ from __future__ import annotations -import shutil +import os import tempfile -from collections.abc import Callable from pathlib import Path +from typing import TYPE_CHECKING from urllib.parse import urlparse -from automation_file.exceptions import FileAutomationException +from automation_file.exceptions import ( + FileAutomationException, + StorageUnavailableException, + StorageURIException, +) from automation_file.logging_config import file_automation_logger +if TYPE_CHECKING: + from automation_file.storage.backend import StorageBackend + +_LOCAL = "local" +_SFTP = "sftp" +_HTTP_SCHEMES = ("http", "https") +_BUCKET_SCHEMES = ("s3", "azure", "az") +_SESSION_SCHEMES = (_SFTP, "ftp") +_STAGED_NAME = "download" +_AUTHORITY_MARK = "//" +_KNOWN_SCHEMES = frozenset({_LOCAL, *_BUCKET_SCHEMES, "dropbox", *_SESSION_SCHEMES, *_HTTP_SCHEMES}) + class CrossBackendException(FileAutomationException): - """Raised when a URI is malformed or refers to an unknown backend.""" + """Raised when a location is malformed or names an unknown backend.""" def copy_between(source: str, target: str) -> bool: - """Copy the object at ``source`` to ``target`` via a local temp file. + """Copy the file at ``source`` to ``target`` and return whether it was transferred. - Returns True when both the download and the upload reported success. + A file already at ``target`` is replaced. ``False`` means the transfer failed + and the reason was logged. """ - downloader = _resolve_downloader(source) - uploader = _resolve_uploader(target) - with tempfile.NamedTemporaryFile(delete=False) as handle: - tmp_path = handle.name + if _scheme_of(target) in _HTTP_SCHEMES: + raise CrossBackendException(f"unknown target backend: {_scheme_of(target)!r}") try: - if not downloader(tmp_path): - file_automation_logger.error("copy_between: download failed (%s)", source) - return False - if not uploader(tmp_path): - file_automation_logger.error("copy_between: upload failed (%s)", target) - return False + destination, path = _locate(target, "target") + if _scheme_of(source) in _HTTP_SCHEMES: + transferred = _fetch_into(source, destination, path) + else: + origin, origin_path = _locate(source, "source") + destination.copy_from(origin, origin_path, path) + transferred = True + except (CrossBackendException, StorageUnavailableException): + raise + except FileAutomationException as error: + file_automation_logger.error( + "copy_between: %s -> %s failed: %s: %s", source, target, type(error).__name__, error + ) + return False + if transferred: file_automation_logger.info("copy_between: %s -> %s", source, target) - return True - finally: - Path(tmp_path).unlink(missing_ok=True) + return transferred -def _resolve_downloader(uri: str) -> Callable[[str], bool]: - scheme, remainder = _split(uri) - if scheme in ("local", ""): - return lambda dest: _local_download(remainder, dest) - if scheme == "s3": - bucket, key = _split_bucket(remainder, "s3") - from automation_file.remote.s3.download_ops import s3_download_file - - return lambda dest: bool(s3_download_file(bucket, key, dest)) - if scheme in ("azure", "az"): - container, blob = _split_bucket(remainder, "azure") - from automation_file.remote.azure_blob.download_ops import azure_blob_download_file - - return lambda dest: bool(azure_blob_download_file(container, blob, dest)) - if scheme == "dropbox": - from automation_file.remote.dropbox_api.download_ops import dropbox_download_file +def _fetch_into(url: str, destination: StorageBackend, path: str) -> bool: + """Download ``url`` to a staging file and store it at ``path``.""" + from automation_file.remote.http_download import download_file - return lambda dest: bool(dropbox_download_file(remainder, dest)) - if scheme == "sftp": - from automation_file.remote.sftp.download_ops import sftp_download_file - - return lambda dest: bool(sftp_download_file(remainder, dest)) - if scheme == "ftp": - from automation_file.remote.ftp.download_ops import ftp_download_file + with tempfile.TemporaryDirectory() as scratch: + staged = Path(scratch) / _STAGED_NAME + if not download_file(url, str(staged)): + file_automation_logger.error("copy_between: download failed (%s)", url) + return False + destination.upload(staged, path) + return True - return lambda dest: bool(ftp_download_file(remainder, dest)) - if scheme in ("http", "https"): - from automation_file.remote.http_download import download_file - return lambda dest: bool(download_file(uri, dest)) - raise CrossBackendException(f"unknown source backend: {scheme!r}") +def _scheme_of(uri: str) -> str: + """Return the lower-cased scheme of ``uri``; a drive letter or no scheme is ``""``.""" + scheme = urlparse(uri).scheme.lower() + return scheme if len(scheme) > 1 else "" -def _resolve_uploader(uri: str) -> Callable[[str], bool]: +def _locate(uri: str, role: str) -> tuple[StorageBackend, str]: + """Return the backend and the path of ``uri``, which is the ``role`` of a copy.""" + scheme = _scheme_of(uri) + if scheme and scheme not in _KNOWN_SCHEMES: + return _resolve(uri, role) + if scheme in _SESSION_SCHEMES and uri[len(scheme) + 1 :].startswith(_AUTHORITY_MARK): + return _resolve(uri, role) scheme, remainder = _split(uri) - if scheme in ("local", ""): - return lambda src: _local_upload(src, remainder) - if scheme == "s3": - bucket, key = _split_bucket(remainder, "s3") - from automation_file.remote.s3.upload_ops import s3_upload_file + if scheme in ("", _LOCAL): + # abspath folds the '..' segments a storage URI may not contain. + return _resolve(os.path.abspath(remainder), role) + if scheme in _BUCKET_SCHEMES: + container, key = _split_bucket(remainder, scheme) + return _resolve(f"{scheme}://{container}/{key}", role) + if scheme in _SESSION_SCHEMES: + return _in_login_directory(scheme), remainder + return _resolve(f"{scheme}:///{remainder}", role) - return lambda src: bool(s3_upload_file(src, bucket, key)) - if scheme in ("azure", "az"): - container, blob = _split_bucket(remainder, "azure") - from automation_file.remote.azure_blob.upload_ops import azure_blob_upload_file - return lambda src: bool(azure_blob_upload_file(src, container, blob)) - if scheme == "dropbox": - from automation_file.remote.dropbox_api.upload_ops import dropbox_upload_file +def _resolve(uri: str, role: str) -> tuple[StorageBackend, str]: + from automation_file.storage.resolver import default_resolver - return lambda src: bool(dropbox_upload_file(src, remainder)) - if scheme == "sftp": - from automation_file.remote.sftp.upload_ops import sftp_upload_file + try: + return default_resolver.resolve(uri) + except StorageURIException as error: + raise CrossBackendException(f"unknown {role} backend: {error}") from error - return lambda src: bool(sftp_upload_file(src, remainder)) - if scheme == "ftp": - from automation_file.remote.ftp.upload_ops import ftp_upload_file - return lambda src: bool(ftp_upload_file(src, remainder)) - raise CrossBackendException(f"unknown target backend: {scheme!r}") +def _in_login_directory(scheme: str) -> StorageBackend: + """Return a backend rooted where the open session logged in. + ``sftp:/path`` and ``ftp:/path`` have always been sent to the server without + their leading slash, which a server resolves against the login directory. + """ + if scheme == _SFTP: + from automation_file.remote.sftp.client import sftp_instance + from automation_file.storage.sftp_storage import SFTPStorage -_KNOWN_SCHEMES = frozenset( - {"local", "s3", "azure", "az", "dropbox", "sftp", "ftp", "http", "https"} -) + try: + return SFTPStorage(root=sftp_instance.require_sftp().normalize(".")) + except RuntimeError as error: + raise StorageUnavailableException(str(error)) from error + from automation_file.remote.ftp.client import FTPException, ftp_instance + from automation_file.storage.ftp_storage import FTPStorage + + try: + return FTPStorage(root=ftp_instance.require_ftp().pwd()) + except FTPException as error: + raise StorageUnavailableException(str(error)) from error def _split(uri: str) -> tuple[str, str]: @@ -134,17 +165,17 @@ def _split(uri: str) -> tuple[str, str]: return "", uri if scheme not in _KNOWN_SCHEMES: raise CrossBackendException(f"unknown backend scheme: {scheme!r}") - if scheme in ("http", "https"): + if scheme in _HTTP_SCHEMES: return scheme, uri - if scheme in ("s3", "azure", "az"): + if scheme in _BUCKET_SCHEMES: if parsed.netloc: tail = parsed.path.lstrip("/") return scheme, f"{parsed.netloc}/{tail}" if tail else parsed.netloc return scheme, parsed.path.lstrip("/") - if scheme == "local": + if scheme == _LOCAL: if parsed.netloc: - return "local", f"{parsed.netloc}{parsed.path}" - return "local", parsed.path + return _LOCAL, f"{parsed.netloc}{parsed.path}" + return _LOCAL, parsed.path # Generic remote path (dropbox, sftp, ftp) — keep the path as given. combined = f"{parsed.netloc}{parsed.path}" if parsed.netloc else parsed.path return scheme, combined.lstrip("/") @@ -157,19 +188,3 @@ def _split_bucket(remainder: str, scheme: str) -> tuple[str, str]: if not bucket or not key: raise CrossBackendException(f"{scheme} URI must be /: {remainder!r}") return bucket, key - - -def _local_download(source_path: str, dest_path: str) -> bool: - src = Path(source_path) - if not src.is_file(): - file_automation_logger.error("copy_between: local source missing: %s", src) - return False - shutil.copyfile(src, dest_path) - return True - - -def _local_upload(source_path: str, target_path: str) -> bool: - target = Path(target_path) - target.parent.mkdir(parents=True, exist_ok=True) - shutil.copyfile(source_path, target) - return True diff --git a/docs/source/Eng/usage/cloud.rst b/docs/source/Eng/usage/cloud.rst index 86e765b..7821eff 100644 --- a/docs/source/Eng/usage/cloud.rst +++ b/docs/source/Eng/usage/cloud.rst @@ -51,8 +51,11 @@ Cross-backend copy ------------------ ``FA_copy_between`` (``copy_between(source, target)``) copies one file from a -backend to another through a local temporary file and returns ``True`` when -both halves succeeded: +location to another and returns ``True`` when it was transferred. It is the +older spelling of ``File(source).copy_to(target)`` and runs on the storage layer +(:doc:`storage`): the copy is native where two backends can do it between +themselves, a file already at the target is replaced, and the operation reaches +the storage observers and the audit trail. .. code-block:: python @@ -64,13 +67,27 @@ both halves succeeded: "target": "azure://backups/april.csv"}], ]) -It accepts ``s3://bucket/key``, ``azure://container/blob`` (or ``az://``), -``dropbox:/path``, ``sftp:/path``, ``ftp:/path``, ``local:/path`` or a plain -filesystem path, and ``http://`` / ``https://`` as a source only. Each backend -must be initialised first (``s3_instance.later_init(...)`` and so on). There is -no Google Drive scheme: Drive addresses files by ID, so use the ``FA_drive_*`` -actions for it. - -For new code prefer the storage layer (:doc:`storage`): ``FA_storage_copy`` -takes the same kind of URIs, reports what it copied, and raises a specific -error instead of returning ``False``. +It accepts: + +* any storage URI: ``s3://bucket/key``, ``azure://container/blob`` (or + ``az://``), ``gdrive:///path``, ``onedrive:///path``, ``dropbox:///path``, + ``sftp://host/absolute/path``, ``memory://name/path``, a mounted prefix; +* a plain filesystem path, ``local:`` or ``local:/path``; +* the spellings it has always taken: ``s3:bucket/key``, ``azure:container/blob`` + and ``dropbox:/path``; +* ``sftp:/path`` and ``ftp:/path`` with one slash or none, where the path is + relative to the directory the session logged in to, as it always was. With two + slashes (``sftp://host/path``) the URI names a host and an absolute path, and + the host must be the one the session is connected to; +* ``http://`` / ``https://`` as a source only, fetched through the validated + downloader (:doc:`transfer`). + +Each backend must be initialised first (``s3_instance.later_init(...)`` and so +on). The function returns ``False`` when the transfer itself fails (a missing +source, a refused write, a copy of a file onto itself) and logs the reason. It +raises ``CrossBackendException`` for a location it cannot make sense of and +``StorageUnavailableException`` for a backend that is not initialised. + +For new code prefer ``FA_storage_copy`` (:doc:`storage`): it takes storage +URIs, reports what it copied, and raises a specific error instead of returning +``False``. diff --git a/docs/source/Zh-CN/usage/cloud.rst b/docs/source/Zh-CN/usage/cloud.rst index f3e76ed..e7d607f 100644 --- a/docs/source/Zh-CN/usage/cloud.rst +++ b/docs/source/Zh-CN/usage/cloud.rst @@ -47,8 +47,10 @@ SFTP 跨后端复制 ---------- -``FA_copy_between``（``copy_between(source, target)``）通过本地临时文件，把一个文件 -从某个后端复制到另一个后端，两个阶段都成功时返回 ``True``： +``FA_copy_between``（``copy_between(source, target)``）把一个文件从某个位置复制到 +另一个位置，传输完成时返回 ``True``。它是 ``File(source).copy_to(target)`` 的旧写法， +建立在存储层之上（:doc:`storage`）：两个后端能直接互传时会采用原生复制，目标位置已有 +的文件会被替换，而且这次操作会送达存储观察者与审计轨迹。 .. code-block:: python @@ -60,11 +62,23 @@ SFTP "target": "azure://backups/april.csv"}], ]) -它接受 ``s3://bucket/key``、``azure://container/blob``（或 ``az://``）、 -``dropbox:/path``、``sftp:/path``、``ftp:/path``、``local:/path`` 或普通的文件系统 -路径；``http://`` / ``https://`` 只能作为来源。每个后端都必须先初始化 -（``s3_instance.later_init(...)`` 等）。没有 Google Drive 的 scheme：Drive 以 ID -定位文件，请改用 ``FA_drive_*`` 动作。 - -新的代码建议使用存储层（:doc:`storage`）：``FA_storage_copy`` 接受同类型的 URI， -会报告复制的结果，失败时抛出明确的异常，而不是返回 ``False``。 +它接受： + +* 任何存储 URI：``s3://bucket/key``、``azure://container/blob``（或 ``az://``）、 + ``gdrive:///path``、``onedrive:///path``、``dropbox:///path``、 + ``sftp://host/absolute/path``、``memory://name/path``，或挂载的前缀； +* 普通的文件系统路径、``local:`` 或 ``local:/path``； +* 它一直以来接受的写法：``s3:bucket/key``、``azure:container/blob`` 与 + ``dropbox:/path``； +* 只有一个斜线或没有斜线的 ``sftp:/path`` 与 ``ftp:/path``，路径相对于会话登录时 + 所在的目录，与以往相同。写成两个斜线（``sftp://host/path``）时，URI 指定的是主机 + 与绝对路径，而且主机必须是会话实际连接的那一台； +* ``http://`` / ``https://`` 只能作为来源，通过经过验证的下载器获取（:doc:`transfer`）。 + +每个后端都必须先初始化（``s3_instance.later_init(...)`` 等）。传输本身失败时（来源不 +存在、写入被拒、把文件复制到它自己）函数返回 ``False`` 并记录原因；无法理解的位置会 +抛出 ``CrossBackendException``，尚未初始化的后端会抛出 +``StorageUnavailableException``。 + +新的代码建议使用 ``FA_storage_copy``（:doc:`storage`）：它接受存储 URI，会报告复制的 +结果，失败时抛出明确的异常，而不是返回 ``False``。 diff --git a/docs/source/Zh-TW/usage/cloud.rst b/docs/source/Zh-TW/usage/cloud.rst index 07c627c..62b2e65 100644 --- a/docs/source/Zh-TW/usage/cloud.rst +++ b/docs/source/Zh-TW/usage/cloud.rst @@ -47,8 +47,10 @@ SFTP 跨後端複製 ---------- -``FA_copy_between``（``copy_between(source, target)``）透過本機暫存檔，把一個檔案 -從某個後端複製到另一個後端，兩個階段都成功時回傳 ``True``： +``FA_copy_between``（``copy_between(source, target)``）把一個檔案從某個位置複製到 +另一個位置，傳輸完成時回傳 ``True``。它是 ``File(source).copy_to(target)`` 的舊寫法， +建立在儲存層之上（:doc:`storage`）：兩個後端能直接互傳時會採用原生複製，目標位置已有 +的檔案會被取代，而且這次操作會送達儲存觀察者與稽核軌跡。 .. code-block:: python @@ -60,11 +62,23 @@ SFTP "target": "azure://backups/april.csv"}], ]) -它接受 ``s3://bucket/key``、``azure://container/blob``（或 ``az://``）、 -``dropbox:/path``、``sftp:/path``、``ftp:/path``、``local:/path`` 或一般的檔案系統 -路徑；``http://`` / ``https://`` 只能作為來源。每個後端都必須先初始化 -（``s3_instance.later_init(...)`` 等）。沒有 Google Drive 的 scheme：Drive 以 ID -定位檔案，請改用 ``FA_drive_*`` 動作。 - -新的程式建議使用儲存層（:doc:`storage`）：``FA_storage_copy`` 接受同類型的 URI， -會回報複製的結果，失敗時拋出明確的例外，而不是回傳 ``False``。 +它接受： + +* 任何儲存 URI：``s3://bucket/key``、``azure://container/blob``（或 ``az://``）、 + ``gdrive:///path``、``onedrive:///path``、``dropbox:///path``、 + ``sftp://host/absolute/path``、``memory://name/path``，或掛載的前綴； +* 一般的檔案系統路徑、``local:`` 或 ``local:/path``； +* 它一直以來接受的寫法：``s3:bucket/key``、``azure:container/blob`` 與 + ``dropbox:/path``； +* 只有一個斜線或沒有斜線的 ``sftp:/path`` 與 ``ftp:/path``，路徑相對於工作階段登入時 + 所在的目錄，與以往相同。寫成兩個斜線（``sftp://host/path``）時，URI 指定的是主機 + 與絕對路徑，而且主機必須是工作階段實際連線的那一台； +* ``http://`` / ``https://`` 只能作為來源，透過經過驗證的下載器取得（:doc:`transfer`）。 + +每個後端都必須先初始化（``s3_instance.later_init(...)`` 等）。傳輸本身失敗時（來源不 +存在、寫入被拒、把檔案複製到它自己）函式回傳 ``False`` 並記錄原因；無法理解的位置會 +拋出 ``CrossBackendException``，尚未初始化的後端會拋出 +``StorageUnavailableException``。 + +新的程式建議使用 ``FA_storage_copy``（:doc:`storage`）：它接受儲存 URI，會回報複製的 +結果，失敗時拋出明確的例外，而不是回傳 ``False``。 diff --git a/docs/updates/2026-10.md b/docs/updates/2026-10.md index a557672..1bbbecc 100644 --- a/docs/updates/2026-10.md +++ b/docs/updates/2026-10.md @@ -491,3 +491,22 @@ Index and query commands: [README.md](README.md). New entries go at the end. - **Docs**: three new sections in the three `usage/cli.rst` pages, the CLI block of the three READMEs, `architecture.md` §2 and §3, `CLAUDE.md` (package map). - **Files**: `automation_file/cli_common.py`, `automation_file/cli_operations.py`, `automation_file/cli_storage.py`, `automation_file/__main__.py`, `tests/test_cli_operations.py`, the documentation above. - **Open items**: none. + +## U-20261008-21 · 2026-10-08 · copy_between runs on the storage layer · #storage #roadmap #done + +- **What**: `copy_between(source, target)` / `FA_copy_between` no longer has a dispatcher of its own that picks a `*_download_file` and a `*_upload_file` per scheme. It resolves both locations to a storage backend and calls `copy_from`, so it is the older spelling of `File(source).copy_to(target)`. Closes `progress.md` #16 (roadmap rule 4: cross-backend operations use the common File/Storage layer). + - It now takes every storage URI the resolver knows (Google Drive, OneDrive, `memory://`, mounts, ...), copies natively where two backends can, and is reported to the storage observers and the audit trail. + - The spellings it always took keep their meaning: a bare path, `local:`, `s3:bucket/key`, `azure:` / `az:`, `dropbox:/path`, and an `http(s)://` source fetched with `download_file` and its SSRF guard. A local path may contain `..` segments (it is made absolute before it becomes a URI). + - `sftp:/path` and `ftp:/path` with one slash or none were always sent to the server without their leading slash, which a server resolves against the login directory. That is kept: the backend is rooted at the directory the session reports (`normalize(".")`, `pwd()`). + - Still `True` / `False`: `False` when the transfer itself fails, with the reason logged. `CrossBackendException` for a location it cannot make sense of, as before. +- **What changed for callers**: + - `sftp://host/path` and `ftp://host/path` with two slashes used to treat the host as the first path segment. They are storage URIs now: a host and an absolute path, refused unless the session is connected to that host. The documented form was the one-slash one. + - A backend that is not initialised raises `StorageUnavailableException`; the per-backend functions raised `RuntimeError` or their own exception from the same place. + - Copying a file onto itself returns `False` (the storage layer refuses it); it used to rewrite the file and return `True`. + - A target that is replaced is replaced by the backend's own write (a temporary sibling and a rename where the backend has one), not by the per-backend upload function. +- **Tests**: `tests/test_cross_backend_storage.py`, 18 cases: storage URIs of several schemes, replacement, `..` in a local path, the copy as an observed storage operation, both S3 spellings and the `az:` and `dropbox:` ones against mounted stand-ins, a bucket without a key, the one-slash session forms and the login directory, an unopened session, the two-slash form, an HTTP source and a refused download, HTTP as a target, a failed transfer and an uninitialised backend, the action. `tests/test_cross_backend.py` (11 cases) passes unchanged. +- **Result / numbers**: 4855 passed, 149 skipped, 0 failed with every extra; 3137 passed, 90 skipped with the base dependencies only. `ruff check`, `ruff format --check` and `mypy automation_file` (231 files) pass. Python 3.14.7 on Windows. +- **Not verified**: a transfer to or from a real service through this function; the login-directory lookup against a real SFTP or FTP server (a stub session was used). +- **Docs**: the "Cross-backend copy" section of the three `usage/cloud.rst` pages and the feature bullet of the three READMEs. +- **Files**: `automation_file/remote/cross_backend.py`, `tests/test_cross_backend_storage.py`, the documentation above. +- **Open items**: none. diff --git a/docs/updates/README.md b/docs/updates/README.md index aafc109..1c8aa9a 100644 --- a/docs/updates/README.md +++ b/docs/updates/README.md @@ -58,6 +58,7 @@ In the same commit: delete the item from `progress.md`, add a `#done` entry here | ID | Date | Title | Tags | Batch | |---|---|---|---|---| +| U-20261008-21 | 2026-10-08 | copy_between runs on the storage layer | #storage #roadmap #done | [2026-10](2026-10.md) | | U-20261008-20 | 2026-10-08 | CLI subcommands for integrity, pipelines and the audit trail | #cli #roadmap #done | [2026-10](2026-10.md) | | U-20261008-19 | 2026-10-08 | The action ACL and the MCP server check nested action names | #security #incident | [2026-10](2026-10.md) | | U-20261008-18 | 2026-10-08 | Pipeline runtime | #pipeline #roadmap #done | [2026-10](2026-10.md) | @@ -115,5 +116,5 @@ In the same commit: delete the item from `progress.md`, add a `#done` entry here | File | Period | Entries | |---|---|---:| -| [2026-10.md](2026-10.md) | 2026-10 | 31 | +| [2026-10.md](2026-10.md) | 2026-10 | 32 | | [2026-09.md](2026-09.md) | 2026-09 | 21 | diff --git a/progress.md b/progress.md index 3c460e0..e8be4ac 100644 --- a/progress.md +++ b/progress.md @@ -14,7 +14,6 @@ Items #10 to #26 are what is left of the 1.0 roadmap (`docs/FILEAUTOMATION-1.0-R ### Universal storage layer (roadmap M2) -- **#16** `copy_between` / `FA_copy_between` on `File.copy_to`. The `FA_storage_*` actions exist (U-20261008-03), so the layer is reachable from action lists; the older action still has its own dispatcher in `remote/cross_backend.py`. `copy_between` accepts `local:`, `sftp:/path`, `s3:bucket/key` and http(s) sources today; `parse_storage_uri` rejects the first three as ambiguous, so the action needs a translation step to stay compatible. - **#17** Native streams and checksums for the remote backends. `open_read` / `open_write`, `copy_tree` and `sync_tree` exist (U-20261008-04), but outside `LocalStorage` a stream is a staged local copy and the default `checksum` downloads the file: S3 could read `get_object()["Body"]`, and a backend with a server-side digest could answer `checksum` from it. - **#31** `FsspecStorage(directories=False)`: a `/` placeholder key that another tool wrote survives `delete(dir, recursive=True)`, so the directory still exists afterwards, and an empty directory that only a placeholder holds cannot be deleted at all. `ObjectStorage` removes placeholders because it lists raw keys; through fsspec the key has to be addressed with the filesystem's own call, and `_strip_protocol` of s3fs / gcsfs removes a trailing slash, so `rm_file(path + "/")` is probably wrong. Needs a real s3fs or gcsfs (MinIO under #19) before it is written. - **#32** [UNVERIFIED] No storage adapter has met a real service: FTP ran against an in-memory model of RFC 959 / 3659 and FTPS data channels did not run at all; SFTP met paramiko's own server, not OpenSSH; Microsoft Graph (`OneDriveClient.graph_send`), Drive (the discovery document as a transport), Dropbox, WebDAV and SMB met stand-ins, and the `smbprotocol` error shapes were written from its documentation. The hand-written redirect handling of `WebDAVClient` (U-20261008-14) has not met a server that redirects. Run each against the service (#19) and fix what differs. diff --git a/tests/test_cross_backend_storage.py b/tests/test_cross_backend_storage.py new file mode 100644 index 0000000..59443af --- /dev/null +++ b/tests/test_cross_backend_storage.py @@ -0,0 +1,208 @@ +"""``copy_between`` on the storage layer: storage URIs, the older spellings, and what fails how.""" + +from __future__ import annotations + +from collections.abc import Iterator +from pathlib import Path +from types import SimpleNamespace +from typing import Any + +import pytest + +from automation_file import CrossBackendException, Storage, copy_between, execute_action +from automation_file.exceptions import StorageUnavailableException +from automation_file.remote import cross_backend +from automation_file.storage import MemoryStorage, clear_memory_stores, memory_store, observe +from automation_file.storage.types import FileInfo + +MOUNTS = ("s3://legacy-bucket", "azure://legacy-container", "dropbox://", "vault://down") + + +class _Unavailable(MemoryStorage): + """A backend whose client was never initialised.""" + + def _stat(self, path: str) -> FileInfo | None: + raise StorageUnavailableException("the vault client is not initialised") + + +@pytest.fixture(autouse=True) +def _fresh() -> Iterator[None]: + clear_memory_stores() + yield + for prefix in MOUNTS: + Storage.unmount(prefix) + clear_memory_stores() + + +def test_storage_uris_of_any_scheme_are_copied(tmp_path: Path) -> None: + source = tmp_path / "source.bin" + source.write_bytes(b"payload") + assert copy_between(str(source), "memory://xb/in/a.bin") is True + assert copy_between("memory://xb/in/a.bin", "memory://xb/out/b.bin") is True + assert copy_between("memory://xb/out/b.bin", f"local:{tmp_path / 'back' / 'c.bin'}") is True + assert (tmp_path / "back" / "c.bin").read_bytes() == b"payload" + + +def test_an_existing_target_is_replaced(tmp_path: Path) -> None: + store = memory_store("xb") + store.write_bytes("a.txt", b"new") + store.write_bytes("b.txt", b"old") + assert copy_between("memory://xb/a.txt", "memory://xb/b.txt") is True + assert store.read_bytes("b.txt") == b"new" + + +def test_a_local_path_may_contain_parent_segments(tmp_path: Path) -> None: + (tmp_path / "a.bin").write_bytes(b"x") + (tmp_path / "sub").mkdir() + roundabout = tmp_path / "sub" / ".." / "a.bin" + assert copy_between(str(roundabout), str(tmp_path / "sub" / ".." / "b.bin")) is True + assert (tmp_path / "b.bin").read_bytes() == b"x" + + +def test_the_copy_is_a_storage_operation_others_can_observe() -> None: + memory_store("xb").write_bytes("a.txt", b"x") + seen: list[observe.StorageOperation] = [] + observe.add_listener(seen.append) + try: + assert copy_between("memory://xb/a.txt", "memory://xb/b.txt") is True + finally: + observe.remove_listener(seen.append) + copies = [operation for operation in seen if operation.operation == "copy"] + assert [(operation.uri, operation.status) for operation in copies] == [ + ("memory://xb/b.txt", "ok") + ] + + +@pytest.mark.parametrize( + "spelling", + ["s3://legacy-bucket/reports/q1.csv", "s3:legacy-bucket/reports/q1.csv"], +) +def test_both_s3_spellings_reach_the_bucket(spelling: str) -> None: + bucket = MemoryStorage() + Storage.mount("s3://legacy-bucket", bucket) + memory_store("xb").write_bytes("q1.csv", b"1,2") + assert copy_between("memory://xb/q1.csv", spelling) is True + assert bucket.read_bytes("reports/q1.csv") == b"1,2" + assert copy_between(spelling, "memory://xb/back.csv") is True + assert memory_store("xb").read_bytes("back.csv") == b"1,2" + + +def test_az_and_dropbox_spellings_are_translated() -> None: + container, dropbox = MemoryStorage(), MemoryStorage() + Storage.mount("azure://legacy-container", container) + Storage.mount("dropbox://", dropbox) + memory_store("xb").write_bytes("a.txt", b"x") + assert copy_between("memory://xb/a.txt", "az://legacy-container/docs/a.txt") is True + assert container.read_bytes("docs/a.txt") == b"x" + assert copy_between("memory://xb/a.txt", "dropbox:/team/a.txt") is True + assert dropbox.read_bytes("team/a.txt") == b"x" + + +def test_a_bucket_location_needs_a_key() -> None: + memory_store("xb").write_bytes("a.txt", b"x") + with pytest.raises(CrossBackendException, match="/"): + copy_between("memory://xb/a.txt", "s3://legacy-bucket") + + +@pytest.mark.parametrize("scheme", ["sftp", "ftp"]) +def test_one_slash_means_relative_to_the_login_directory( + monkeypatch: pytest.MonkeyPatch, scheme: str +) -> None: + home = MemoryStorage() + asked: list[str] = [] + + def _login_directory(name: str) -> MemoryStorage: + asked.append(name) + return home + + monkeypatch.setattr(cross_backend, "_in_login_directory", _login_directory) + memory_store("xb").write_bytes("a.txt", b"x") + assert copy_between("memory://xb/a.txt", f"{scheme}:/inbox/a.txt") is True + assert copy_between("memory://xb/a.txt", f"{scheme}:inbox/b.txt") is True + assert sorted(info.path for info in home.list_dir("inbox")) == ["inbox/a.txt", "inbox/b.txt"] + assert asked == [scheme, scheme] + + +def test_the_login_directory_is_asked_of_the_open_session(monkeypatch: pytest.MonkeyPatch) -> None: + pytest.importorskip("paramiko") + from automation_file.remote.sftp.client import sftp_instance + + session = SimpleNamespace(normalize=lambda path: "/home/ops" if path == "." else path) + monkeypatch.setattr(sftp_instance, "require_sftp", lambda: session) + backend: Any = cross_backend._in_login_directory("sftp") + assert backend.root == "/home/ops" + + +def test_a_session_that_is_not_open_is_reported_as_unavailable( + monkeypatch: pytest.MonkeyPatch, +) -> None: + from automation_file.remote.ftp.client import ftp_instance + + monkeypatch.setattr(ftp_instance, "_ftp", None) + memory_store("xb").write_bytes("a.txt", b"x") + with pytest.raises(StorageUnavailableException, match="not initialised"): + copy_between("memory://xb/a.txt", "ftp:/inbox/a.txt") + + +def test_two_slashes_name_a_host_and_go_to_the_storage_layer( + monkeypatch: pytest.MonkeyPatch, +) -> None: + seen: list[str] = [] + + def _resolve(uri: str, role: str) -> tuple[MemoryStorage, str]: + seen.append(uri) + return memory_store("xb"), "a.txt" if role == "source" else "landed.txt" + + monkeypatch.setattr(cross_backend, "_resolve", _resolve) + memory_store("xb").write_bytes("a.txt", b"x") + assert copy_between("memory://xb/a.txt", "sftp://nas.example/srv/a.txt") is True + assert seen == ["sftp://nas.example/srv/a.txt", "memory://xb/a.txt"] + + +def test_an_http_source_is_fetched_with_the_validated_downloader( + monkeypatch: pytest.MonkeyPatch, +) -> None: + fetched: list[str] = [] + + def _download(url: str, target: str) -> bool: + fetched.append(url) + Path(target).write_bytes(b"from the web") + return True + + monkeypatch.setattr("automation_file.remote.http_download.download_file", _download) + assert copy_between("https://example.org/a.bin", "memory://xb/web/a.bin") is True + assert fetched == ["https://example.org/a.bin"] + assert memory_store("xb").read_bytes("web/a.bin") == b"from the web" + + +def test_a_refused_download_is_false_and_writes_nothing(monkeypatch: pytest.MonkeyPatch) -> None: + monkeypatch.setattr( + "automation_file.remote.http_download.download_file", lambda url, target: False + ) + assert copy_between("https://example.org/a.bin", "memory://xb/web/a.bin") is False + assert not memory_store("xb").exists("web/a.bin") + + +def test_http_is_never_a_target() -> None: + memory_store("xb").write_bytes("a.txt", b"x") + with pytest.raises(CrossBackendException, match="unknown target backend: 'https'"): + copy_between("memory://xb/a.txt", "https://example.org/upload") + + +def test_a_failed_transfer_is_false_and_an_uninitialised_backend_raises() -> None: + assert copy_between("memory://xb/absent.txt", "memory://xb/b.txt") is False + memory_store("xb").write_bytes("a.txt", b"x") + assert copy_between("memory://xb/a.txt", "memory://xb/a.txt") is False + assert memory_store("xb").read_bytes("a.txt") == b"x" + Storage.mount("vault://down", _Unavailable()) + with pytest.raises(StorageUnavailableException, match="not initialised"): + copy_between("vault://down/a.txt", "memory://xb/b.txt") + + +def test_the_action_takes_the_same_locations() -> None: + memory_store("xb").write_bytes("a.txt", b"x") + results = execute_action( + [["FA_copy_between", {"source": "memory://xb/a.txt", "target": "memory://xb/b.txt"}]] + ) + assert list(results.values()) == [True] + assert memory_store("xb").read_bytes("b.txt") == b"x" From 41f37825817f377f4df73caf2e9755343f4a07c3 Mon Sep 17 00:00:00 2001 From: JeffreyChen Date: Thu, 8 Oct 2026 14:30:26 +0800 Subject: [PATCH 38/59] docs: add the public API and deprecation policy Says what is public, the stability levels, what a version number promises and how a name is retired, and adds the helper every deprecation goes through. --- CLAUDE.md | 1 + README.md | 11 ++ README.zh-CN.md | 9 ++ README.zh-TW.md | 9 ++ architecture.md | 4 + automation_file/core/deprecation.py | 81 ++++++++++++ docs/source/API/core.rst | 3 + docs/source/Eng/eng_index.rst | 14 ++ docs/source/Eng/usage/api_policy.rst | 176 +++++++++++++++++++++++++ docs/source/Zh-CN/usage/api_policy.rst | 154 ++++++++++++++++++++++ docs/source/Zh-CN/zh_cn_index.rst | 13 ++ docs/source/Zh-TW/usage/api_policy.rst | 154 ++++++++++++++++++++++ docs/source/Zh-TW/zh_tw_index.rst | 13 ++ docs/updates/2026-10.md | 17 +++ docs/updates/README.md | 3 +- progress.md | 1 - tests/test_deprecation.py | 81 ++++++++++++ 17 files changed, 742 insertions(+), 2 deletions(-) create mode 100644 automation_file/core/deprecation.py create mode 100644 docs/source/Eng/usage/api_policy.rst create mode 100644 docs/source/Zh-CN/usage/api_policy.rst create mode 100644 docs/source/Zh-TW/usage/api_policy.rst create mode 100644 tests/test_deprecation.py diff --git a/CLAUDE.md b/CLAUDE.md index 8e18255..dbdefee 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -134,6 +134,7 @@ python -m automation_file --help - Action-list shape: `[name]`, `[name, {kwargs}]`, or `[name, [args]]` — nothing else. - Delete all unused code — no dead imports, commented-out blocks, unreachable branches, or `_old_`-prefixed names. Git history is the archive. - Prefer updating the registry over extending the executor class. Plugins register via `add_command_to_executor({name: callable})`. +- Public API: what `docs/source/Eng/usage/api_policy.rst` lists (the facade and package `__all__`s, `FA_*` actions with their parameters and result shapes, CLI flags and exit codes, storage URI syntax, versioned data formats, event types and payload keys, the exception hierarchy). Never rename, remove or change the meaning of one of those in place. Retire it with `automation_file.core.deprecation.deprecated(since=, removal=, replacement=)` or `warn_deprecated`, keep it working for at least two minor releases, and remove it only in a major release. A format written to disk carries a schema version, and its reader refuses a version it does not know. ## Security diff --git a/README.md b/README.md index 7304d5a..ac5107c 100644 --- a/README.md +++ b/README.md @@ -1240,6 +1240,17 @@ Each entry is either a bare command name, a `[name, kwargs]` pair, or a ] ``` +## Compatibility + +Releases follow semantic versioning. The public surface is everything in `automation_file.__all__` +and in the `__all__` of the documented packages, the `FA_*` actions, the command line, the storage +URI syntax, the data formats (each carries a schema version) and the event types. Until 1.0 the +storage layer, the event bus, pipelines, the integrity monitor, the audit trail, the notification +router and the semantic MCP tools are provisional: they may still change in a minor release, and +the release notes say how. A deprecated name keeps working for at least two minor releases, warns +with its replacement, and is removed only in a major release. The full policy is in the manual: +*Public API and compatibility* (`docs/source/Eng/usage/api_policy.rst`). + ## Documentation Full API documentation lives under `docs/` and can be built with Sphinx: diff --git a/README.zh-CN.md b/README.zh-CN.md index cba460f..a3f0ef4 100644 --- a/README.zh-CN.md +++ b/README.zh-CN.md @@ -1205,6 +1205,15 @@ python -m automation_file --create_project ./my_project ] ``` +## 兼容性 + +版本采用语义化版本。公开接口包含 `automation_file.__all__` 与已写入文档的各个包 `__all__` 中的 +所有名称、`FA_*` 动作、命令行、存储 URI 语法、数据格式（每一种都带有 schema 版本）以及事件类型。 +在 1.0 之前，存储层、事件总线、流水线、完整性监控、审计轨迹、通知路由器与语义化 MCP 工具属于 +暂定功能：仍可能在次版本中变动，版本说明会交代如何应对。被弃用的名称至少会保留两个次版本、 +发出附带替代方案的警告，并且只会在主版本中移除。完整的政策请见手册的“公开 API 与兼容性” +（`docs/source/Zh-CN/usage/api_policy.rst`）。 + ## 文档 完整 API 文档位于 `docs/`，可用 Sphinx 生成： diff --git a/README.zh-TW.md b/README.zh-TW.md index 1531004..5fa7df0 100644 --- a/README.zh-TW.md +++ b/README.zh-TW.md @@ -1205,6 +1205,15 @@ python -m automation_file --create_project ./my_project ] ``` +## 相容性 + +版本採用語意化版本。公開介面包含 `automation_file.__all__` 與已寫入文件的各套件 `__all__` 中的 +所有名稱、`FA_*` 動作、命令列、儲存 URI 語法、資料格式（每一種都帶有 schema 版本）以及事件類型。 +在 1.0 之前，儲存層、事件匯流排、管線、完整性監控、稽核軌跡、通知路由器與語意化 MCP 工具屬於 +暫定功能：仍可能在次版本中變動，版本說明會交代如何因應。被棄用的名稱至少會保留兩個次版本、 +發出附帶替代方案的警告，並且只會在主版本中移除。完整的政策請見手冊的「公開 API 與相容性」 +（`docs/source/Zh-TW/usage/api_policy.rst`）。 + ## 文件 完整 API 文件位於 `docs/`，可用 Sphinx 產生： diff --git a/architecture.md b/architecture.md index 398e6f0..c124c32 100644 --- a/architecture.md +++ b/architecture.md @@ -252,6 +252,10 @@ storage.observe → events.storage_bridge → StorageError (only for a failing b ## 7. Design constraints +- The public surface (facade and package `__all__`s, `FA_*` actions, CLI, storage URIs, versioned data + formats, event types, the exception hierarchy) changes only by deprecation: `core/deprecation.py` + warns, the name stays for at least two minor releases, a major release removes it + (`docs/source/Eng/usage/api_policy.rst`; CLAUDE.md § Conventions). - Only the three action shapes in §3. Extend through the registry, not by subclassing the executor. Python 3.10+, `X | Y` unions, `from __future__ import annotations` (CLAUDE.md § Conventions). - Exceptions derive from `FileAutomationException`. Log through `file_automation_logger`; no diff --git a/automation_file/core/deprecation.py b/automation_file/core/deprecation.py new file mode 100644 index 0000000..126921c --- /dev/null +++ b/automation_file/core/deprecation.py @@ -0,0 +1,81 @@ +"""Retire a public name without breaking the code that still uses it. + +The deprecation policy (``docs``: *Public API and compatibility*) is that a +deprecated name keeps working for at least two minor releases, warns each time it +is used, names its replacement, and is removed only in a major release. This +module is the one way to do the warning half. Python hides a +``DeprecationWarning`` raised outside ``__main__`` by default, and an action list +run from JSON has no ``__main__`` of the user's, so each message is also logged +once per process: + +.. code-block:: python + + @deprecated(since="1.1", removal="2.0", replacement="automation_file.File.copy_to") + def copy_between(source, target): ... + + warn_deprecated("the 'manager=' argument", since="1.1", removal="2.0", + replacement="a notification route") +""" + +from __future__ import annotations + +import functools +import threading +import warnings +from collections.abc import Callable +from typing import ParamSpec, TypeVar + +from automation_file.logging_config import file_automation_logger + +_P = ParamSpec("_P") +_R = TypeVar("_R") +_logged: set[str] = set() +_logged_guard = threading.Lock() + + +def deprecation_message(what: str, *, since: str, removal: str, replacement: str | None) -> str: + """Return the standard wording: what is deprecated, since when, until when, what to use.""" + message = f"{what} is deprecated since {since} and will be removed in {removal}" + return f"{message}; use {replacement} instead" if replacement else message + + +def warn_deprecated( + what: str, + *, + since: str, + removal: str, + replacement: str | None = None, + stacklevel: int = 2, +) -> None: + """Emit a :class:`DeprecationWarning` attributed to the caller of the deprecated thing. + + The same message is logged at warning level the first time it occurs in a process. + """ + message = deprecation_message(what, since=since, removal=removal, replacement=replacement) + with _logged_guard: + first = message not in _logged + _logged.add(message) + if first: + file_automation_logger.warning("%s", message) + warnings.warn(message, DeprecationWarning, stacklevel=stacklevel + 1) + + +def deprecated( + *, since: str, removal: str, replacement: str | None = None +) -> Callable[[Callable[_P, _R]], Callable[_P, _R]]: + """Mark a function or method as deprecated; calling it warns, then runs it unchanged.""" + + def decorate(function: Callable[_P, _R]) -> Callable[_P, _R]: + what = f"{function.__module__}.{function.__qualname__}" + + @functools.wraps(function) + def wrapper(*args: _P.args, **kwargs: _P.kwargs) -> _R: + warn_deprecated(what, since=since, removal=removal, replacement=replacement) + return function(*args, **kwargs) + + wrapper.__deprecated__ = deprecation_message( # type: ignore[attr-defined] + what, since=since, removal=removal, replacement=replacement + ) + return wrapper + + return decorate diff --git a/docs/source/API/core.rst b/docs/source/API/core.rst index e25e807..14e0219 100644 --- a/docs/source/API/core.rst +++ b/docs/source/API/core.rst @@ -79,6 +79,9 @@ Core .. automodule:: automation_file.core.secrets :members: +.. automodule:: automation_file.core.deprecation + :members: + .. automodule:: automation_file.exceptions :members: diff --git a/docs/source/Eng/eng_index.rst b/docs/source/Eng/eng_index.rst index d8d41da..aebfee7 100644 --- a/docs/source/Eng/eng_index.rst +++ b/docs/source/Eng/eng_index.rst @@ -319,3 +319,17 @@ history; written in Python or as a versioned YAML / JSON definition. :caption: Pipelines usage/pipeline + +.. _eng-api-policy: + +Chapter 21 — Public API and Compatibility +========================================= + +What is public and what is not, the stability levels, what a version number +promises, and how a name is deprecated and removed. + +.. toctree:: + :maxdepth: 2 + :caption: Public API and Compatibility + + usage/api_policy diff --git a/docs/source/Eng/usage/api_policy.rst b/docs/source/Eng/usage/api_policy.rst new file mode 100644 index 0000000..8d3b811 --- /dev/null +++ b/docs/source/Eng/usage/api_policy.rst @@ -0,0 +1,176 @@ +Public API and compatibility +============================ + +This page says what you may rely on, how a release number tells you what +changed, and how a name is retired. It is the contract behind the 1.0 release. + +What is public +-------------- + +.. list-table:: + :header-rows: 1 + :widths: 30 70 + + * - Surface + - What it covers + * - Python names + - Everything in ``automation_file.__all__``, and everything in the + ``__all__`` of the documented packages: ``automation_file.storage``, + ``automation_file.events``, ``automation_file.pipeline``, + ``automation_file.integrity``, ``automation_file.audit``, + ``automation_file.notify``, ``automation_file.scheduler``. That means + the name, its parameters, what it returns and the exceptions it + documents. + * - Actions + - Every ``FA_*`` action name, its parameters and the shape of its result. + An action list written today keeps running. + * - Command line + - The subcommands and flags of ``python -m automation_file``, the shape of + their JSON output and their exit codes. + * - Storage URIs + - The syntax (:doc:`storage`), the built-in schemes and their aliases. + * - Data formats + - Pipeline definitions (``schema_version: 1``), integrity manifests + (``schema_version: 2``), the audit record and its SQLite schema + (version 2), the configuration file. + * - Events + - The type name of each core event (``pipeline.failed``, ...), its severity + and its payload keys (:doc:`event_bus`). + * - Exceptions + - The hierarchy below ``FileAutomationException``: which class an error + is, and which classes it inherits from. + * - Extension points + - The methods a ``StorageBackend`` subclass overrides, the ``AuditStore``, + ``RunStore`` and ``NotificationSink`` interfaces, and the + ``automation_file.actions`` entry point. + +What is not public +------------------ + +* A name that starts with an underscore, in any module. +* The module a public name lives in. Import ``S3Storage`` from + ``automation_file`` or ``automation_file.storage``; the path + ``automation_file.storage.s3_storage`` may change. +* The exact text of an error message and of a log line. Rely on the exception + class and its attributes. +* The GUI's widget classes. +* Anything under ``tests/``, including the stand-ins. The storage contract suite + (``tests/storage_contract.py``) is published for backend authors, but it is + test code and is versioned with the repository, not with the package. + +Stability levels +---------------- + +Stable + Covered by everything below. From 1.0 on this is: ``StorageBackend`` and + the storage URI syntax, ``File`` and ``Storage``, ``Pipeline`` and its + definition format, ``IntegrityMonitor`` and its manifest, the event model, + the audit record and ``AuditStore``, the ``FA_*`` actions, the command line. + +Provisional + New and still allowed to change in a minor release, with the change listed + in the release notes. A provisional feature is marked as such at the top of + its manual page. Until 1.0 the storage layer, the event bus, the pipeline + runtime, the integrity monitor, the audit trail, the notification router and + the semantic MCP tools are provisional. + +Private + Everything listed under `What is not public`_. It changes without notice. + +Version numbers +--------------- + +Releases follow semantic versioning, ``MAJOR.MINOR.PATCH``. + +.. list-table:: + :header-rows: 1 + :widths: 16 84 + + * - Part + - Goes up when + * - ``PATCH`` + - A bug is fixed. Nothing public is added, removed or changed in meaning. + * - ``MINOR`` + - Something is added, a provisional feature changes, or something is + deprecated. Code written for the previous minor release keeps working. + * - ``MAJOR`` + - Something stable is removed or changes in a way existing code can + notice. The release notes carry a migration guide. + +Two cases deserve a sentence each. A **security fix** may change behaviour in a +patch release when keeping the behaviour would keep the hole; the release notes +say so. And **0.x releases** are before the contract: the next minor release may +change anything, although the ``FA_*`` actions have been kept compatible +throughout. + +Supported Python versions are the CPython releases that still receive security +fixes upstream. Dropping one that has reached its end of life is a minor change. + +How a name is retired +--------------------- + +1. A minor release marks the name as deprecated. It keeps working exactly as + before, and each use raises a ``DeprecationWarning`` that names the release + it was deprecated in, the release that will remove it, and its replacement. + The message is also logged once per process, because Python hides the + warning outside ``__main__`` and an action list run from JSON would never + show it. A deprecated ``FA_*`` action stays registered. +2. The manual and the release notes list it, with the replacement. +3. It stays for at least two minor releases. +4. Only a major release removes it. + +The warning is written in one way throughout the code base: + +.. code-block:: python + + from automation_file.core.deprecation import deprecated, warn_deprecated + + @deprecated(since="1.2", removal="2.0", replacement="automation_file.File.copy_to") + def copy_between(source, target): + ... + + def start(self, *, legacy_flag=None): + if legacy_flag is not None: + warn_deprecated("the 'legacy_flag' argument", since="1.2", removal="2.0") + +To find the deprecated names your own code uses, run your tests with warnings +turned into errors:: + + python -W error::DeprecationWarning -m pytest + +Nothing is deprecated at the time of writing. The older interfaces that have a +newer counterpart (``copy_between`` and ``File.copy_to``, ``AuditLog`` and the +audit trail, ``execute_action_dag`` and ``Pipeline``) are all supported side by +side. + +Data formats +------------ + +A format that is written to disk carries a version, and a reader refuses a +version it does not know instead of guessing: a manifest with +``schema_version: 3`` raises ``IntegrityException`` today, and a pipeline +definition with an unknown ``schema_version`` is reported by +``validate_definition``. When a format gets a new version, the release that +introduces it still reads the previous one, and says how to convert. + +When something goes wrong +------------------------- + +.. list-table:: + :header-rows: 1 + :widths: 40 60 + + * - What you see + - What it means and what to do + * - ``DeprecationWarning: … is deprecated since 1.2 and will be removed in 2.0; use … instead`` + - The name still works. Switch to the replacement before the release the + message names. + * - An import of a module path fails after an upgrade + - The path was not public. Import the name from ``automation_file`` or + from its documented package. + * - A test that compared an error message fails after an upgrade + - Message text is not part of the contract. Compare the exception class, + or match only the part you depend on. + * - ``… has manifest schema version 3`` + - The file was written by a newer release. Upgrade the package that reads + it. diff --git a/docs/source/Zh-CN/usage/api_policy.rst b/docs/source/Zh-CN/usage/api_policy.rst new file mode 100644 index 0000000..db6192f --- /dev/null +++ b/docs/source/Zh-CN/usage/api_policy.rst @@ -0,0 +1,154 @@ +公开 API 与兼容性 +================== + +本页说明哪些东西可以依赖、版本号如何告诉你改了什么，以及一个名称如何退场。 +这是 1.0 版背后的契约。 + +哪些是公开的 +------------ + +.. list-table:: + :header-rows: 1 + :widths: 30 70 + + * - 接口 + - 涵盖范围 + * - Python 名称 + - ``automation_file.__all__`` 中的所有名称，以及下列已写入文档的包其 + ``__all__`` 中的所有名称：``automation_file.storage``、 + ``automation_file.events``、``automation_file.pipeline``、 + ``automation_file.integrity``、``automation_file.audit``、 + ``automation_file.notify``、``automation_file.scheduler``。这包含名称本身、 + 它的参数、返回值，以及文档中记载的异常。 + * - 动作 + - 每一个 ``FA_*`` 动作的名称、参数与结果的形状。今天写下的动作列表之后仍然能运行。 + * - 命令行 + - ``python -m automation_file`` 的子命令与选项、JSON 输出的形状，以及退出码。 + * - 存储 URI + - 语法（:doc:`storage`）、内置的 scheme 及其别名。 + * - 数据格式 + - 流水线定义（``schema_version: 1``）、完整性 manifest（``schema_version: 2``）、 + 审计记录及其 SQLite 结构（第 2 版）、配置文件。 + * - 事件 + - 每一种核心事件的类型名称（``pipeline.failed`` 等）、严重程度与 payload 的键 + （:doc:`event_bus`）。 + * - 异常 + - ``FileAutomationException`` 之下的层级：一个错误属于哪个类，以及它继承自 + 哪些类。 + * - 扩展点 + - ``StorageBackend`` 子类要重写的方法，``AuditStore``、``RunStore`` 与 + ``NotificationSink`` 接口，以及 ``automation_file.actions`` 入口点。 + +哪些不是公开的 +-------------- + +* 任何模块中以下划线开头的名称。 +* 公开名称所在的模块。请从 ``automation_file`` 或 ``automation_file.storage`` 导入 + ``S3Storage``；``automation_file.storage.s3_storage`` 这个路径可能会变。 +* 错误消息与日志的确切文字。请依赖异常类与它的属性。 +* GUI 的 widget 类。 +* ``tests/`` 之下的一切，包含各种替身。存储契约测试套件 + （``tests/storage_contract.py``）是提供给后端作者使用的，但它是测试代码， + 随着代码仓库而不是随着包的发行版本演进。 + +稳定等级 +-------- + +稳定（Stable） + 受以下所有规则保障。自 1.0 起包含：``StorageBackend`` 与存储 URI 语法、 + ``File`` 与 ``Storage``、``Pipeline`` 及其定义格式、``IntegrityMonitor`` 及其 + manifest、事件模型、审计记录与 ``AuditStore``、``FA_*`` 动作，以及命令行。 + +暂定（Provisional） + 新功能，仍可能在次版本中变动，变动会列在版本说明中。暂定的功能会在其手册页面 + 开头标示。在 1.0 之前，存储层、事件总线、流水线运行环境、完整性监控、审计轨迹、 + 通知路由器与语义化 MCP 工具都属于暂定。 + +私有（Private） + `哪些不是公开的`_ 列出的一切。它们会在没有通知的情况下变动。 + +版本号 +------ + +版本采用语义化版本，``MAJOR.MINOR.PATCH``。 + +.. list-table:: + :header-rows: 1 + :widths: 16 84 + + * - 部分 + - 何时递增 + * - ``PATCH`` + - 修复错误。没有任何公开的东西被新增、移除或改变含义。 + * - ``MINOR`` + - 新增功能、暂定功能有所变动，或有东西被标为弃用。为前一个次版本写的代码仍然 + 能运行。 + * - ``MAJOR`` + - 稳定的东西被移除，或以既有代码能察觉的方式改变。版本说明会附上迁移指南。 + +有两种情况各值得一句话。**安全修复** 可以在修订版中改变行为，只要保留原行为就等于 +保留漏洞；版本说明会注明。而 **0.x 版** 还在契约之前：下一个次版本可以改动任何东西， +不过 ``FA_*`` 动作一路以来都保持兼容。 + +支持的 Python 版本是上游仍提供安全修复的 CPython 版本。停止支持已到生命周期终点的 +版本属于次版本变动。 + +名称如何退场 +------------ + +1. 某个次版本把名称标为弃用。它的运行与以往完全相同，每次使用都会发出 + ``DeprecationWarning``，说明它在哪个版本被弃用、哪个版本会移除，以及替代方案。 + 同一条消息在每个进程中也会写入日志一次，因为 Python 在 ``__main__`` 之外会隐藏 + 这种警告，而由 JSON 运行的动作列表永远看不到它。被弃用的 ``FA_*`` 动作仍然保持 + 注册。 +2. 手册与版本说明会列出它以及替代方案。 +3. 它至少保留两个次版本。 +4. 只有主版本才会移除它。 + +整个代码库都用同一种方式发出这个警告： + +.. code-block:: python + + from automation_file.core.deprecation import deprecated, warn_deprecated + + @deprecated(since="1.2", removal="2.0", replacement="automation_file.File.copy_to") + def copy_between(source, target): + ... + + def start(self, *, legacy_flag=None): + if legacy_flag is not None: + warn_deprecated("the 'legacy_flag' argument", since="1.2", removal="2.0") + +要找出你自己的代码用到哪些已弃用的名称，请把警告当成错误来运行测试:: + + python -W error::DeprecationWarning -m pytest + +撰写本文时没有任何东西被弃用。已有较新对应物的旧接口（``copy_between`` 与 +``File.copy_to``、``AuditLog`` 与审计轨迹、``execute_action_dag`` 与 ``Pipeline``） +全部并行支持。 + +数据格式 +-------- + +写入磁盘的格式都带有版本，读取端遇到不认识的版本会直接拒绝而不是猜测： +``schema_version: 3`` 的 manifest 目前会抛出 ``IntegrityException``， +``schema_version`` 不明的流水线定义会由 ``validate_definition`` 报告。当某个格式推出新 +版本时，引入它的那个发行版本仍然读得懂前一个版本，并说明如何转换。 + +出现问题时 +---------- + +.. list-table:: + :header-rows: 1 + :widths: 40 60 + + * - 你看到的 + - 代表什么、该怎么做 + * - ``DeprecationWarning: … is deprecated since 1.2 and will be removed in 2.0; use … instead`` + - 这个名称仍然可用。请在消息所说的版本之前改用替代方案。 + * - 升级后某个模块路径的导入失败 + - 那个路径不是公开的。请从 ``automation_file`` 或它已写入文档的包导入该名称。 + * - 升级后，一个比对错误消息的测试失败 + - 消息文字不属于契约。请比对异常类，或只比对你真正依赖的部分。 + * - ``… has manifest schema version 3`` + - 这个文件是由较新的版本写出的。请升级读取它的包。 diff --git a/docs/source/Zh-CN/zh_cn_index.rst b/docs/source/Zh-CN/zh_cn_index.rst index a7411a0..71abb99 100644 --- a/docs/source/Zh-CN/zh_cn_index.rst +++ b/docs/source/Zh-CN/zh_cn_index.rst @@ -309,3 +309,16 @@ Slack、Email（SMTP）、Discord、Telegram、Microsoft Teams、PagerDuty :caption: 流水线 usage/pipeline + +.. _zh-cn-api-policy: + +第 21 章 — 公开 API 与兼容性 +============================ + +哪些是公开的、哪些不是，稳定等级，版本号所作的承诺，以及名称如何被弃用与移除。 + +.. toctree:: + :maxdepth: 2 + :caption: 公开 API 与兼容性 + + usage/api_policy diff --git a/docs/source/Zh-TW/usage/api_policy.rst b/docs/source/Zh-TW/usage/api_policy.rst new file mode 100644 index 0000000..48aed0d --- /dev/null +++ b/docs/source/Zh-TW/usage/api_policy.rst @@ -0,0 +1,154 @@ +公開 API 與相容性 +================== + +本頁說明哪些東西可以依賴、版本號碼如何告訴你改了什麼，以及一個名稱如何退場。 +這是 1.0 版背後的契約。 + +哪些是公開的 +------------ + +.. list-table:: + :header-rows: 1 + :widths: 30 70 + + * - 介面 + - 涵蓋範圍 + * - Python 名稱 + - ``automation_file.__all__`` 中的所有名稱，以及下列已寫入文件的套件其 + ``__all__`` 中的所有名稱：``automation_file.storage``、 + ``automation_file.events``、``automation_file.pipeline``、 + ``automation_file.integrity``、``automation_file.audit``、 + ``automation_file.notify``、``automation_file.scheduler``。這包含名稱本身、 + 它的參數、回傳值，以及文件中記載的例外。 + * - 動作 + - 每一個 ``FA_*`` 動作的名稱、參數與結果的形狀。今天寫下的動作清單之後仍然能執行。 + * - 命令列 + - ``python -m automation_file`` 的子指令與旗標、JSON 輸出的形狀，以及結束碼。 + * - 儲存 URI + - 語法（:doc:`storage`）、內建的 scheme 及其別名。 + * - 資料格式 + - 管線定義（``schema_version: 1``）、完整性 manifest（``schema_version: 2``）、 + 稽核紀錄及其 SQLite 結構（第 2 版）、設定檔。 + * - 事件 + - 每一種核心事件的類型名稱（``pipeline.failed`` 等）、嚴重程度與 payload 的鍵 + （:doc:`event_bus`）。 + * - 例外 + - ``FileAutomationException`` 之下的階層：一個錯誤屬於哪個類別，以及它繼承自 + 哪些類別。 + * - 擴充點 + - ``StorageBackend`` 子類別要覆寫的方法，``AuditStore``、``RunStore`` 與 + ``NotificationSink`` 介面，以及 ``automation_file.actions`` 進入點。 + +哪些不是公開的 +-------------- + +* 任何模組中以底線開頭的名稱。 +* 公開名稱所在的模組。請從 ``automation_file`` 或 ``automation_file.storage`` 匯入 + ``S3Storage``；``automation_file.storage.s3_storage`` 這個路徑可能會變。 +* 錯誤訊息與日誌的確切文字。請依賴例外類別與它的屬性。 +* GUI 的 widget 類別。 +* ``tests/`` 之下的一切，包含各種替身。儲存契約測試套件 + （``tests/storage_contract.py``）是提供給後端作者使用的，但它是測試程式碼， + 隨著儲存庫而不是隨著套件發行版本演進。 + +穩定等級 +-------- + +穩定（Stable） + 受以下所有規則保障。自 1.0 起包含：``StorageBackend`` 與儲存 URI 語法、 + ``File`` 與 ``Storage``、``Pipeline`` 及其定義格式、``IntegrityMonitor`` 及其 + manifest、事件模型、稽核紀錄與 ``AuditStore``、``FA_*`` 動作，以及命令列。 + +暫定（Provisional） + 新功能，仍可能在次版本中變動，變動會列在版本說明中。暫定的功能會在其手冊頁面 + 開頭標示。在 1.0 之前，儲存層、事件匯流排、管線執行環境、完整性監控、稽核軌跡、 + 通知路由器與語意化 MCP 工具都屬於暫定。 + +私有（Private） + `哪些不是公開的`_ 列出的一切。它們會在沒有通知的情況下變動。 + +版本號碼 +-------- + +版本採用語意化版本，``MAJOR.MINOR.PATCH``。 + +.. list-table:: + :header-rows: 1 + :widths: 16 84 + + * - 部分 + - 何時遞增 + * - ``PATCH`` + - 修正錯誤。沒有任何公開的東西被新增、移除或改變意義。 + * - ``MINOR`` + - 新增功能、暫定功能有所變動，或有東西被標為棄用。為前一個次版本寫的程式仍然 + 能運作。 + * - ``MAJOR`` + - 穩定的東西被移除，或以既有程式能察覺的方式改變。版本說明會附上遷移指南。 + +有兩種情況各值得一句話。**安全性修正** 可以在修訂版中改變行為，只要保留原行為就等於 +保留漏洞；版本說明會註明。而 **0.x 版** 還在契約之前：下一個次版本可以改動任何東西， +不過 ``FA_*`` 動作一路以來都維持相容。 + +支援的 Python 版本是上游仍提供安全性修正的 CPython 版本。停止支援已到生命週期終點的 +版本屬於次版本變動。 + +名稱如何退場 +------------ + +1. 某個次版本把名稱標為棄用。它的運作與以往完全相同，每次使用都會發出 + ``DeprecationWarning``，說明它在哪個版本被棄用、哪個版本會移除，以及替代方案。 + 同一則訊息在每個行程中也會寫入日誌一次，因為 Python 在 ``__main__`` 之外會隱藏 + 這種警告，而由 JSON 執行的動作清單永遠看不到它。被棄用的 ``FA_*`` 動作仍然保持 + 註冊。 +2. 手冊與版本說明會列出它以及替代方案。 +3. 它至少保留兩個次版本。 +4. 只有主版本才會移除它。 + +整個程式庫都用同一種方式發出這個警告： + +.. code-block:: python + + from automation_file.core.deprecation import deprecated, warn_deprecated + + @deprecated(since="1.2", removal="2.0", replacement="automation_file.File.copy_to") + def copy_between(source, target): + ... + + def start(self, *, legacy_flag=None): + if legacy_flag is not None: + warn_deprecated("the 'legacy_flag' argument", since="1.2", removal="2.0") + +要找出你自己的程式用到哪些已棄用的名稱，請把警告當成錯誤來執行測試:: + + python -W error::DeprecationWarning -m pytest + +撰寫本文時沒有任何東西被棄用。已有較新對應物的舊介面（``copy_between`` 與 +``File.copy_to``、``AuditLog`` 與稽核軌跡、``execute_action_dag`` 與 ``Pipeline``） +全部並行支援。 + +資料格式 +-------- + +寫入磁碟的格式都帶有版本，讀取端遇到不認識的版本會直接拒絕而不是猜測： +``schema_version: 3`` 的 manifest 目前會拋出 ``IntegrityException``， +``schema_version`` 不明的管線定義會由 ``validate_definition`` 回報。當某個格式推出新 +版本時，引入它的那個發行版本仍然讀得懂前一個版本，並說明如何轉換。 + +發生問題時 +---------- + +.. list-table:: + :header-rows: 1 + :widths: 40 60 + + * - 你看到的 + - 代表什麼、該怎麼做 + * - ``DeprecationWarning: … is deprecated since 1.2 and will be removed in 2.0; use … instead`` + - 這個名稱仍然可用。請在訊息所說的版本之前改用替代方案。 + * - 升級後某個模組路徑的匯入失敗 + - 那個路徑不是公開的。請從 ``automation_file`` 或它已寫入文件的套件匯入該名稱。 + * - 升級後，一個比對錯誤訊息的測試失敗 + - 訊息文字不屬於契約。請比對例外類別，或只比對你真正依賴的部分。 + * - ``… has manifest schema version 3`` + - 這個檔案是由較新的版本寫出的。請升級讀取它的套件。 diff --git a/docs/source/Zh-TW/zh_tw_index.rst b/docs/source/Zh-TW/zh_tw_index.rst index 99d59b2..e9e246a 100644 --- a/docs/source/Zh-TW/zh_tw_index.rst +++ b/docs/source/Zh-TW/zh_tw_index.rst @@ -309,3 +309,16 @@ Slack、Email（SMTP）、Discord、Telegram、Microsoft Teams、PagerDuty :caption: 管線 usage/pipeline + +.. _zh-tw-api-policy: + +第 21 章 — 公開 API 與相容性 +============================ + +哪些是公開的、哪些不是，穩定等級，版本號碼所作的承諾，以及名稱如何被棄用與移除。 + +.. toctree:: + :maxdepth: 2 + :caption: 公開 API 與相容性 + + usage/api_policy diff --git a/docs/updates/2026-10.md b/docs/updates/2026-10.md index 1bbbecc..a0f2099 100644 --- a/docs/updates/2026-10.md +++ b/docs/updates/2026-10.md @@ -510,3 +510,20 @@ Index and query commands: [README.md](README.md). New entries go at the end. - **Docs**: the "Cross-backend copy" section of the three `usage/cloud.rst` pages and the feature bullet of the three READMEs. - **Files**: `automation_file/remote/cross_backend.py`, `tests/test_cross_backend_storage.py`, the documentation above. - **Open items**: none. + +## U-20261008-22 · 2026-10-08 · Public API and deprecation policy · #decision #docs #roadmap #done + +- **Decision** (closes `progress.md` #11; roadmap M1 and M9). The owner asked for the whole roadmap to be carried out, so the policy was written rather than left open; it is theirs to amend before 1.0. + - **Public**: every name in `automation_file.__all__` and in the `__all__` of the documented packages (`storage`, `events`, `pipeline`, `integrity`, `audit`, `notify`, `scheduler`); the `FA_*` action names, parameters and result shapes; the CLI subcommands, flags, JSON output and exit codes; the storage URI syntax; the data formats, each with a schema version; the event type names, severities and payload keys; the exception hierarchy; the extension points (`StorageBackend` hooks, `AuditStore`, `RunStore`, `NotificationSink`, the `automation_file.actions` entry point). + - **Not public**: underscore names, the module a public name lives in, message and log text, the GUI widget classes, everything under `tests/`. + - **Levels**: stable, provisional, private. Until 1.0 the storage layer, the event bus, pipelines, the integrity monitor, the audit trail, the notification router and the semantic MCP tools are provisional. + - **Versions**: semantic versioning. PATCH for fixes, MINOR for additions, provisional changes and deprecations, MAJOR for removing or changing something stable. A security fix may change behaviour in a PATCH and says so. Supported Python versions are the ones that still get upstream security fixes. + - **Deprecation**: a MINOR marks the name; it keeps working and warns with the release it was deprecated in, the release that removes it and its replacement; it stays at least two minor releases; only a MAJOR removes it. A deprecated `FA_*` action stays registered. Nothing is deprecated today: `copy_between`, `AuditLog` and `execute_action_dag` stay supported next to their newer counterparts. + - **Data formats**: a reader refuses a version it does not know, and the release that introduces a new version still reads the previous one. +- **Code**: `automation_file/core/deprecation.py` with `deprecated(since=, removal=, replacement=)`, `warn_deprecated(...)` and `deprecation_message(...)`. The warning points at the caller's line. Each message is also logged once per process, because Python hides a `DeprecationWarning` raised outside `__main__` and an action list run from JSON would never show it. +- **Tests**: `tests/test_deprecation.py`, 6 cases: the wording, a function and a method that warn and still work, the warning's location, no warning before the call, one log line per message. +- **Result / numbers**: 4861 passed, 149 skipped, 0 failed with every extra; 3143 passed, 90 skipped with the base dependencies only. `ruff check`, `ruff format --check` and `mypy automation_file` (231 files) pass. Python 3.14.7 on Windows. +- **Not done here**: the publish workflow still bumps only the patch number; choosing a MINOR or MAJOR bump is part of `progress.md` #26. The migration guide is written once the scheduler, MCP and GUI work has landed. +- **Docs**: chapter 21 in the three manuals (`usage/api_policy.rst`), the indexes, `docs/source/API/core.rst`, a "Compatibility" section in the three READMEs, `CLAUDE.md` § Conventions, `architecture.md` §7. +- **Files**: `automation_file/core/deprecation.py`, `tests/test_deprecation.py`, the documentation above. +- **Open items**: #26 (release engineering, migration guide). diff --git a/docs/updates/README.md b/docs/updates/README.md index 1c8aa9a..c2b1648 100644 --- a/docs/updates/README.md +++ b/docs/updates/README.md @@ -58,6 +58,7 @@ In the same commit: delete the item from `progress.md`, add a `#done` entry here | ID | Date | Title | Tags | Batch | |---|---|---|---|---| +| U-20261008-22 | 2026-10-08 | Public API and deprecation policy | #decision #docs #roadmap #done | [2026-10](2026-10.md) | | U-20261008-21 | 2026-10-08 | copy_between runs on the storage layer | #storage #roadmap #done | [2026-10](2026-10.md) | | U-20261008-20 | 2026-10-08 | CLI subcommands for integrity, pipelines and the audit trail | #cli #roadmap #done | [2026-10](2026-10.md) | | U-20261008-19 | 2026-10-08 | The action ACL and the MCP server check nested action names | #security #incident | [2026-10](2026-10.md) | @@ -116,5 +117,5 @@ In the same commit: delete the item from `progress.md`, add a `#done` entry here | File | Period | Entries | |---|---|---:| -| [2026-10.md](2026-10.md) | 2026-10 | 32 | +| [2026-10.md](2026-10.md) | 2026-10 | 33 | | [2026-09.md](2026-09.md) | 2026-09 | 21 | diff --git a/progress.md b/progress.md index e8be4ac..9a3aad8 100644 --- a/progress.md +++ b/progress.md @@ -10,7 +10,6 @@ Items #10 to #26 are what is left of the 1.0 roadmap (`docs/FILEAUTOMATION-1.0-R ### Architecture and packaging (roadmap M1) -- **#11** [DECIDE] Public API policy and deprecation policy (roadmap M1, M9): which names are frozen at 1.0 and how a name is retired. The storage layer is documented as provisional until then. ### Universal storage layer (roadmap M2) diff --git a/tests/test_deprecation.py b/tests/test_deprecation.py new file mode 100644 index 0000000..489ccf5 --- /dev/null +++ b/tests/test_deprecation.py @@ -0,0 +1,81 @@ +"""The deprecation helpers: the wording, the warning category, and where the warning points.""" + +from __future__ import annotations + +import logging +import warnings + +import pytest + +from automation_file.core.deprecation import deprecated, deprecation_message, warn_deprecated +from automation_file.logging_config import file_automation_logger + + +def test_message_names_the_versions_and_the_replacement() -> None: + assert ( + deprecation_message("old()", since="1.1", removal="2.0", replacement="new()") + == "old() is deprecated since 1.1 and will be removed in 2.0; use new() instead" + ) + assert ( + deprecation_message("old()", since="1.1", removal="2.0", replacement=None) + == "old() is deprecated since 1.1 and will be removed in 2.0" + ) + + +def test_a_deprecated_function_warns_and_still_works() -> None: + @deprecated(since="1.1", removal="2.0", replacement="add_many") + def add(left: int, right: int = 1) -> int: + """Add two numbers.""" + return left + right + + with pytest.warns(DeprecationWarning, match="add is deprecated since 1.1") as caught: + assert add(2, right=3) == 5 + assert caught[0].filename == __file__ + assert "use add_many instead" in str(caught[0].message) + assert add.__name__ == "add" + assert add.__doc__ == "Add two numbers." + assert "removed in 2.0" in add.__deprecated__ # type: ignore[attr-defined] + + +def test_a_deprecated_method_warns_at_the_call_site() -> None: + class Thing: + @deprecated(since="1.2", removal="2.0") + def old(self) -> str: + return "ok" + + with pytest.warns(DeprecationWarning) as caught: + assert Thing().old() == "ok" + assert caught[0].filename == __file__ + assert "Thing.old is deprecated since 1.2" in str(caught[0].message) + + +def test_warn_deprecated_points_at_the_caller_of_the_deprecated_code() -> None: + def library_function() -> None: + warn_deprecated("the 'manager=' argument", since="1.1", removal="2.0") + + with pytest.warns(DeprecationWarning, match="the 'manager=' argument") as caught: + library_function() + assert caught[0].filename == __file__ + + +def test_nothing_warns_until_the_function_is_called() -> None: + with warnings.catch_warnings(): + warnings.simplefilter("error") + + @deprecated(since="1.1", removal="2.0") + def unused() -> None: + return None + + assert callable(unused) + + +def test_a_message_is_logged_once_per_process(caplog: pytest.LogCaptureFixture) -> None: + caplog.set_level(logging.WARNING, logger=file_automation_logger.name) + with warnings.catch_warnings(): + warnings.simplefilter("ignore", DeprecationWarning) + for _ in range(3): + warn_deprecated("logged_once()", since="1.4", removal="2.0", replacement="newer()") + logged = [record.getMessage() for record in caplog.records if "logged_once()" in record.message] + assert logged == [ + "logged_once() is deprecated since 1.4 and will be removed in 2.0; use newer() instead" + ] From 6910e4db1636752dfc237b707025bc51594ca540 Mon Sep 17 00:00:00 2001 From: JeffreyChen Date: Thu, 8 Oct 2026 14:39:03 +0800 Subject: [PATCH 39/59] test: add integration tests against real services and their workflow The storage contract suite runs against MinIO, Azurite, OpenSSH, FTP, WebDAV and Samba in containers, and the unit tests on Linux and macOS. Nothing here has run yet: there is no Docker on the development machine, so the first workflow run is the test. --- .gitattributes | 2 + .github/workflows/integration.yml | 87 ++++++++++++++ CLAUDE.md | 1 + README.md | 14 +++ README.zh-CN.md | 13 ++ README.zh-TW.md | 13 ++ docs/source/Eng/eng_index.rst | 14 +++ docs/source/Eng/usage/integration_tests.rst | 109 +++++++++++++++++ docs/source/Zh-CN/usage/integration_tests.rst | 96 +++++++++++++++ docs/source/Zh-CN/zh_cn_index.rst | 14 +++ docs/source/Zh-TW/usage/integration_tests.rst | 96 +++++++++++++++ docs/source/Zh-TW/zh_tw_index.rst | 14 +++ docs/updates/2026-10.md | 15 +++ docs/updates/README.md | 3 +- progress.md | 4 +- tests/integration/__init__.py | 8 ++ tests/integration/service_env.py | 37 ++++++ tests/integration/start_service.sh | 113 ++++++++++++++++++ tests/integration/test_azure_azurite.py | 59 +++++++++ tests/integration/test_ftp_server.py | 53 ++++++++ tests/integration/test_s3_minio.py | 74 ++++++++++++ tests/integration/test_sftp_openssh.py | 59 +++++++++ tests/integration/test_smb_samba.py | 53 ++++++++ tests/integration/test_webdav_server.py | 50 ++++++++ 24 files changed, 998 insertions(+), 3 deletions(-) create mode 100644 .github/workflows/integration.yml create mode 100644 docs/source/Eng/usage/integration_tests.rst create mode 100644 docs/source/Zh-CN/usage/integration_tests.rst create mode 100644 docs/source/Zh-TW/usage/integration_tests.rst create mode 100644 tests/integration/__init__.py create mode 100644 tests/integration/service_env.py create mode 100755 tests/integration/start_service.sh create mode 100644 tests/integration/test_azure_azurite.py create mode 100644 tests/integration/test_ftp_server.py create mode 100644 tests/integration/test_s3_minio.py create mode 100644 tests/integration/test_sftp_openssh.py create mode 100644 tests/integration/test_smb_samba.py create mode 100644 tests/integration/test_webdav_server.py diff --git a/.gitattributes b/.gitattributes index dfe0770..19ee0e7 100644 --- a/.gitattributes +++ b/.gitattributes @@ -1,2 +1,4 @@ # Auto detect text files and perform LF normalization * text=auto +# Shell scripts run under bash on Linux runners, which does not accept CRLF. +*.sh text eol=lf diff --git a/.github/workflows/integration.yml b/.github/workflows/integration.yml new file mode 100644 index 0000000..e72c17f --- /dev/null +++ b/.github/workflows/integration.yml @@ -0,0 +1,87 @@ +name: Integration + +# The storage contract suite against real services, and the unit tests on the other two platforms. +# Neither job gates a release: publishing depends on the jobs of ci-dev.yml and ci-stable.yml. + +on: + pull_request: + branches: [ "dev", "main" ] + push: + branches: [ "dev" ] + schedule: + - cron: "30 3 * * *" + workflow_dispatch: + +permissions: + contents: read + +concurrency: + group: integration-${{ github.ref }} + cancel-in-progress: true + +jobs: + services: + # One service per job, each started in a container by tests/integration/start_service.sh. + # FA_IT_REQUIRED turns a test module that would skip (its variables are missing) into an error. + runs-on: ubuntu-latest + timeout-minutes: 20 # no run yet: to revisit after the first runs + strategy: + fail-fast: false + matrix: + service: [ s3, azure, sftp, ftp, webdav, smb ] + env: + SERVICE: ${{ matrix.service }} + FA_IT_REQUIRED: "1" + steps: + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + persist-credentials: false + - name: Set up Python + uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0 + with: + python-version: "3.12" + cache: pip + - name: Install the package with every extra + run: | + python -m pip install --upgrade pip wheel + cp dev.toml pyproject.toml + pip install -e ".[all,test]" + - name: Start the service + run: bash tests/integration/start_service.sh "$SERVICE" + - name: Run the contract suite against it + run: python -m pytest tests/integration/test_"$SERVICE"_*.py -v --tb=short + - name: Show the service log + if: failure() + run: docker logs "fa-it-$SERVICE" || true + + platforms: + # ci-dev.yml and ci-stable.yml run the unit tests on Windows. These are the other two. + runs-on: ${{ matrix.os }} + timeout-minutes: 20 # no run yet: to revisit after the first runs + strategy: + fail-fast: false + matrix: + os: [ ubuntu-latest, macos-latest ] + env: + QT_QPA_PLATFORM: offscreen + steps: + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + persist-credentials: false + - name: Set up Python + uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0 + with: + python-version: "3.12" + cache: pip + - name: Install the libraries Qt needs without a display + if: runner.os == 'Linux' + run: | + sudo apt-get update + sudo apt-get install -y --no-install-recommends libegl1 libgl1 libxkbcommon0 libdbus-1-3 libfontconfig1 + - name: Install the package with every extra + run: | + python -m pip install --upgrade pip wheel + cp dev.toml pyproject.toml + pip install -e ".[all,test]" + - name: Run pytest + run: python -m pytest tests/ -v --tb=short diff --git a/CLAUDE.md b/CLAUDE.md index dbdefee..3e71702 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -123,6 +123,7 @@ python -m automation_file --help - Unit tests live under `tests/` (pytest). Fixtures in `tests/conftest.py` (`sample_file`, `sample_dir`). - Tests cover every module in `core/`, `local/`, `remote/url_validator`, `project/`, `server/`, `utils/`, plus a facade smoke test, retry/quota/safe_paths, HTTP+TCP auth, and optional-backend registration. - Google Drive / HTTP-download / S3 / Azure / Dropbox / SFTP code paths that require real credentials or network access are **not** exercised in CI — only their URL-validation, auth, and guard-clause behaviour are. +- `tests/integration/` runs the storage contract suite against real services (MinIO, Azurite, OpenSSH, FTP, WebDAV, Samba). Each module is skipped unless its `FA_IT_*` variables are set; `tests/integration/start_service.sh ` starts the container and exports them, and `.github/workflows/integration.yml` runs one service per job with `FA_IT_REQUIRED=1`, which turns a skip into a failure. The same workflow runs the unit tests on Linux and macOS. It does not gate publishing. A backend whose service can run in a container gets a module there, an entry in the script and one in the workflow matrix. - Run all tests before submitting changes: `python -m pytest tests/ -v`. ## Conventions diff --git a/README.md b/README.md index ac5107c..4e34241 100644 --- a/README.md +++ b/README.md @@ -1240,6 +1240,20 @@ Each entry is either a bare command name, a `[name, kwargs]` pair, or a ] ``` +## Tests + +```bash +pip install -e ".[all,test]" +python -m pytest tests/ # unit tests; a backend whose extra is missing is skipped + +# The storage contract against a real service in a container (needs Docker) +eval "$(bash tests/integration/start_service.sh s3)" # or azure, sftp, ftp, webdav, smb +python -m pytest tests/integration/test_s3_minio.py +``` + +The integration tests are skipped unless their `FA_IT_*` variables are set; see the manual chapter +*Integration tests*. + ## Compatibility Releases follow semantic versioning. The public surface is everything in `automation_file.__all__` diff --git a/README.zh-CN.md b/README.zh-CN.md index a3f0ef4..3dedbf2 100644 --- a/README.zh-CN.md +++ b/README.zh-CN.md @@ -1205,6 +1205,19 @@ python -m automation_file --create_project ./my_project ] ``` +## 测试 + +```bash +pip install -e ".[all,test]" +python -m pytest tests/ # 单元测试；缺少 extra 的后端会被跳过 + +# 对容器中的真实服务运行存储契约测试（需要 Docker） +eval "$(bash tests/integration/start_service.sh s3)" # 或 azure、sftp、ftp、webdav、smb +python -m pytest tests/integration/test_s3_minio.py +``` + +除非设置了对应的 `FA_IT_*` 变量，否则集成测试会被跳过；详见手册的“集成测试”一章。 + ## 兼容性 版本采用语义化版本。公开接口包含 `automation_file.__all__` 与已写入文档的各个包 `__all__` 中的 diff --git a/README.zh-TW.md b/README.zh-TW.md index 5fa7df0..f5c0129 100644 --- a/README.zh-TW.md +++ b/README.zh-TW.md @@ -1205,6 +1205,19 @@ python -m automation_file --create_project ./my_project ] ``` +## 測試 + +```bash +pip install -e ".[all,test]" +python -m pytest tests/ # 單元測試；缺少 extra 的後端會被略過 + +# 對容器中的真實服務執行儲存契約測試（需要 Docker） +eval "$(bash tests/integration/start_service.sh s3)" # 或 azure、sftp、ftp、webdav、smb +python -m pytest tests/integration/test_s3_minio.py +``` + +除非設定了對應的 `FA_IT_*` 變數，否則整合測試會被略過；詳見手冊的「整合測試」一章。 + ## 相容性 版本採用語意化版本。公開介面包含 `automation_file.__all__` 與已寫入文件的各套件 `__all__` 中的 diff --git a/docs/source/Eng/eng_index.rst b/docs/source/Eng/eng_index.rst index aebfee7..66b756b 100644 --- a/docs/source/Eng/eng_index.rst +++ b/docs/source/Eng/eng_index.rst @@ -333,3 +333,17 @@ promises, and how a name is deprecated and removed. :caption: Public API and Compatibility usage/api_policy + +.. _eng-integration-tests: + +Chapter 22 — Integration Tests +============================== + +The storage contract suite against real services in containers: how to run it +locally, the variables each module reads, and what CI runs. + +.. toctree:: + :maxdepth: 2 + :caption: Integration Tests + + usage/integration_tests diff --git a/docs/source/Eng/usage/integration_tests.rst b/docs/source/Eng/usage/integration_tests.rst new file mode 100644 index 0000000..efca2f3 --- /dev/null +++ b/docs/source/Eng/usage/integration_tests.rst @@ -0,0 +1,109 @@ +Integration tests +================= + +The unit tests give every storage backend a stand-in for its service. The +integration tests run the same contract suite (``tests/storage_contract.py``, 81 +cases) against a real one: MinIO for S3, Azurite for Azure Blob, an OpenSSH server +for SFTP, and an FTP, a WebDAV and a Samba server. They live in +``tests/integration/``. + +They are skipped unless you point them at a service, so ``python -m pytest tests/`` +behaves the same with and without them. + +Running one locally +------------------- + +With Docker installed, one script starts a service in a container and prints the +variables its test module reads: + +.. code-block:: bash + + eval "$(bash tests/integration/start_service.sh s3)" + python -m pytest tests/integration/test_s3_minio.py -v + docker rm -f fa-it-s3 + +The argument is ``s3``, ``azure``, ``sftp``, ``ftp``, ``webdav`` or ``smb``. The +script generates the password or key for that run and stores nothing. To test +against a service of your own, set the variables yourself instead. + +Variables +--------- + +.. list-table:: + :header-rows: 1 + :widths: 24 76 + + * - Module + - Variables + * - ``test_s3_minio.py`` + - ``FA_IT_S3_ENDPOINT``, ``FA_IT_S3_ACCESS_KEY``, ``FA_IT_S3_SECRET_KEY``; + optional ``FA_IT_S3_REGION`` (``us-east-1``) + * - ``test_azure_azurite.py`` + - ``FA_IT_AZURE_CONNECTION_STRING`` + * - ``test_sftp_openssh.py`` + - ``FA_IT_SFTP_HOST``, ``FA_IT_SFTP_USER``, ``FA_IT_SFTP_PASSWORD``, + ``FA_IT_SFTP_KNOWN_HOSTS`` (a file with the server's host key; an unknown + host is rejected); optional ``FA_IT_SFTP_PORT`` (``22``), + ``FA_IT_SFTP_ROOT`` (``/upload``) + * - ``test_ftp_server.py`` + - ``FA_IT_FTP_HOST``, ``FA_IT_FTP_USER``, ``FA_IT_FTP_PASSWORD``; optional + ``FA_IT_FTP_PORT`` (``21``), ``FA_IT_FTP_ROOT`` (``/``), ``FA_IT_FTP_TLS`` + (``1`` for FTPS) + * - ``test_webdav_server.py`` + - ``FA_IT_WEBDAV_URL``, ``FA_IT_WEBDAV_USER``, ``FA_IT_WEBDAV_PASSWORD`` + * - ``test_smb_samba.py`` + - ``FA_IT_SMB_SERVER``, ``FA_IT_SMB_SHARE``, ``FA_IT_SMB_USER``, + ``FA_IT_SMB_PASSWORD``; optional ``FA_IT_SMB_PORT`` (``445``), + ``FA_IT_SMB_ENCRYPT`` (``1``) + +A module whose first variable is not set is skipped. With ``FA_IT_REQUIRED=1`` it +fails instead, which is how CI makes sure a job did not pass by skipping +everything. + +Each test works in a bucket, a container or a directory of its own with a random +name, and removes it afterwards. Point the tests at an account you can afford to +write to all the same. + +In CI +----- + +``.github/workflows/integration.yml`` runs on every pull request, on pushes to +``dev``, nightly and on demand. Its ``services`` job starts one service per matrix +entry with the same script and runs that service's module. Its ``platforms`` job +runs the unit tests on Linux and macOS; the ``pytest`` job of the main workflow +covers Windows. Neither gates a release. + +Google Drive, OneDrive and Dropbox have no emulator, so they are covered by their +stand-ins only. To check one of them against the real service, write a module on +the same pattern: a guard on its variables, then a ``StorageContract`` subclass +whose ``backend`` fixture yields the adapter rooted in a scratch folder. + +Adding a backend +---------------- + +A new backend ships with a contract class against a stand-in (:doc:`storage`, +"Writing a backend"), and, where its service can run in a container, with a module +here and an entry in the script and in the workflow matrix. + +When something goes wrong +------------------------- + +.. list-table:: + :header-rows: 1 + :widths: 40 60 + + * - What you see + - What it means and what to do + * - ``SKIPPED … set FA_IT_S3_ENDPOINT to run against an S3 service`` + - The module's variables are not set. Start the service with the script, or + set them. + * - ``FA_IT_REQUIRED is set, but FA_IT_… is not`` + - CI mode, and the service did not start or did not export its variables. + Read the "Start the service" step and the service log printed after a + failure. + * - ``nothing is listening on 127.0.0.1:`` + - The container did not come up within 90 seconds. The script prints + ``docker ps -a`` and the container's log. + * - ``StorageUnavailableException`` in every test + - The service is reachable but refused the credentials, or, for SFTP, its + host key is not in the known-hosts file. diff --git a/docs/source/Zh-CN/usage/integration_tests.rst b/docs/source/Zh-CN/usage/integration_tests.rst new file mode 100644 index 0000000..989a885 --- /dev/null +++ b/docs/source/Zh-CN/usage/integration_tests.rst @@ -0,0 +1,96 @@ +集成测试 +======== + +单元测试为每个存储后端准备了服务的替身。集成测试则把同一套契约测试 +（``tests/storage_contract.py``，81 个用例）拿去对真正的服务运行：S3 用 MinIO、 +Azure Blob 用 Azurite、SFTP 用 OpenSSH 服务器，另外还有 FTP、WebDAV 与 Samba 服务器。 +这些测试放在 ``tests/integration/``。 + +除非你把它们指向某个服务，否则它们会被跳过，因此不论有没有这些测试， +``python -m pytest tests/`` 的行为都一样。 + +在本地运行其中一个 +------------------ + +安装 Docker 之后，用一个脚本就能在容器中启动服务，并打印出该测试模块要读取的变量： + +.. code-block:: bash + + eval "$(bash tests/integration/start_service.sh s3)" + python -m pytest tests/integration/test_s3_minio.py -v + docker rm -f fa-it-s3 + +参数可以是 ``s3``、``azure``、``sftp``、``ftp``、``webdav`` 或 ``smb``。脚本会为这一次 +运行生成密码或密钥，不会存储任何东西。如果要对你自己的服务测试，请改为自行设置这些变量。 + +变量 +---- + +.. list-table:: + :header-rows: 1 + :widths: 24 76 + + * - 模块 + - 变量 + * - ``test_s3_minio.py`` + - ``FA_IT_S3_ENDPOINT``、``FA_IT_S3_ACCESS_KEY``、``FA_IT_S3_SECRET_KEY``； + 可选 ``FA_IT_S3_REGION``（``us-east-1``） + * - ``test_azure_azurite.py`` + - ``FA_IT_AZURE_CONNECTION_STRING`` + * - ``test_sftp_openssh.py`` + - ``FA_IT_SFTP_HOST``、``FA_IT_SFTP_USER``、``FA_IT_SFTP_PASSWORD``、 + ``FA_IT_SFTP_KNOWN_HOSTS``（内含服务器主机密钥的文件；未知的主机会被拒绝）； + 可选 ``FA_IT_SFTP_PORT``（``22``）、``FA_IT_SFTP_ROOT``（``/upload``） + * - ``test_ftp_server.py`` + - ``FA_IT_FTP_HOST``、``FA_IT_FTP_USER``、``FA_IT_FTP_PASSWORD``；可选 + ``FA_IT_FTP_PORT``（``21``）、``FA_IT_FTP_ROOT``（``/``）、``FA_IT_FTP_TLS`` + （``1`` 代表 FTPS） + * - ``test_webdav_server.py`` + - ``FA_IT_WEBDAV_URL``、``FA_IT_WEBDAV_USER``、``FA_IT_WEBDAV_PASSWORD`` + * - ``test_smb_samba.py`` + - ``FA_IT_SMB_SERVER``、``FA_IT_SMB_SHARE``、``FA_IT_SMB_USER``、 + ``FA_IT_SMB_PASSWORD``；可选 ``FA_IT_SMB_PORT``（``445``）、 + ``FA_IT_SMB_ENCRYPT``（``1``） + +第一个变量没有设置的模块会被跳过。设置 ``FA_IT_REQUIRED=1`` 时则改为失败，CI 就是用 +这个方式确保作业不是靠着跳过所有测试而通过。 + +每个测试都在自己专属、名称随机的 bucket、container 或目录中运行，结束后会把它删除。 +即使如此，仍请把测试指向一个你承担得起写入的账号。 + +在 CI 中 +-------- + +``.github/workflows/integration.yml`` 会在每个 pull request、每次推送到 ``dev``、每晚 +以及手动触发时运行。其中的 ``services`` 作业会用同一个脚本，为矩阵中的每个条目启动一个 +服务，并运行该服务的模块。``platforms`` 作业则在 Linux 与 macOS 上运行单元测试； +Windows 由主要流程的 ``pytest`` 作业负责。两者都不会阻挡发布。 + +Google Drive、OneDrive 与 Dropbox 没有模拟器，因此只由它们的替身覆盖。如果要对真正的 +服务检查其中之一，请按同样的模式编写模块：先检查变量，再写一个 ``StorageContract`` +子类，让它的 ``backend`` fixture 产生以某个临时文件夹为根目录的适配器。 + +新增后端 +-------- + +新的后端会附上一个针对替身的契约类（:doc:`storage` 的“编写后端”），而且只要它的 +服务能在容器中运行，就会在这里附上一个模块，并在脚本与流程矩阵中各加上一个条目。 + +出现问题时 +---------- + +.. list-table:: + :header-rows: 1 + :widths: 40 60 + + * - 你看到的 + - 代表什么、该怎么做 + * - ``SKIPPED … set FA_IT_S3_ENDPOINT to run against an S3 service`` + - 这个模块的变量没有设置。请用脚本启动服务，或自行设置变量。 + * - ``FA_IT_REQUIRED is set, but FA_IT_… is not`` + - 当前是 CI 模式，而服务没有启动，或没有导出它的变量。请查看“Start the service” + 步骤，以及失败后打印的服务日志。 + * - ``nothing is listening on 127.0.0.1:`` + - 容器在 90 秒内没有启动完成。脚本会打印 ``docker ps -a`` 与该容器的日志。 + * - 每个测试都出现 ``StorageUnavailableException`` + - 服务连得上，但拒绝了凭据；或者对 SFTP 而言，它的主机密钥不在 known-hosts 文件中。 diff --git a/docs/source/Zh-CN/zh_cn_index.rst b/docs/source/Zh-CN/zh_cn_index.rst index 71abb99..dc8320e 100644 --- a/docs/source/Zh-CN/zh_cn_index.rst +++ b/docs/source/Zh-CN/zh_cn_index.rst @@ -322,3 +322,17 @@ Slack、Email（SMTP）、Discord、Telegram、Microsoft Teams、PagerDuty :caption: 公开 API 与兼容性 usage/api_policy + +.. _zh-cn-integration-tests: + +第 22 章 — 集成测试 +=================== + +把存储契约测试套件拿去对容器中的真实服务运行：如何在本地运行、各模块读取的变量， +以及 CI 运行了什么。 + +.. toctree:: + :maxdepth: 2 + :caption: 集成测试 + + usage/integration_tests diff --git a/docs/source/Zh-TW/usage/integration_tests.rst b/docs/source/Zh-TW/usage/integration_tests.rst new file mode 100644 index 0000000..82e5363 --- /dev/null +++ b/docs/source/Zh-TW/usage/integration_tests.rst @@ -0,0 +1,96 @@ +整合測試 +======== + +單元測試為每個儲存後端準備了服務的替身。整合測試則把同一套契約測試 +（``tests/storage_contract.py``，81 個案例）拿去對真正的服務執行：S3 用 MinIO、 +Azure Blob 用 Azurite、SFTP 用 OpenSSH 伺服器，另外還有 FTP、WebDAV 與 Samba 伺服器。 +這些測試放在 ``tests/integration/``。 + +除非你把它們指向某個服務，否則它們會被略過，因此不論有沒有這些測試， +``python -m pytest tests/`` 的行為都一樣。 + +在本機執行其中一個 +------------------ + +安裝 Docker 之後，用一支腳本就能在容器中啟動服務，並印出該測試模組要讀取的變數： + +.. code-block:: bash + + eval "$(bash tests/integration/start_service.sh s3)" + python -m pytest tests/integration/test_s3_minio.py -v + docker rm -f fa-it-s3 + +參數可以是 ``s3``、``azure``、``sftp``、``ftp``、``webdav`` 或 ``smb``。腳本會為這一次 +執行產生密碼或金鑰，不會儲存任何東西。若要對你自己的服務測試，請改為自行設定這些變數。 + +變數 +---- + +.. list-table:: + :header-rows: 1 + :widths: 24 76 + + * - 模組 + - 變數 + * - ``test_s3_minio.py`` + - ``FA_IT_S3_ENDPOINT``、``FA_IT_S3_ACCESS_KEY``、``FA_IT_S3_SECRET_KEY``； + 選用 ``FA_IT_S3_REGION``（``us-east-1``） + * - ``test_azure_azurite.py`` + - ``FA_IT_AZURE_CONNECTION_STRING`` + * - ``test_sftp_openssh.py`` + - ``FA_IT_SFTP_HOST``、``FA_IT_SFTP_USER``、``FA_IT_SFTP_PASSWORD``、 + ``FA_IT_SFTP_KNOWN_HOSTS``（內含伺服器主機金鑰的檔案；未知的主機會被拒絕）； + 選用 ``FA_IT_SFTP_PORT``（``22``）、``FA_IT_SFTP_ROOT``（``/upload``） + * - ``test_ftp_server.py`` + - ``FA_IT_FTP_HOST``、``FA_IT_FTP_USER``、``FA_IT_FTP_PASSWORD``；選用 + ``FA_IT_FTP_PORT``（``21``）、``FA_IT_FTP_ROOT``（``/``）、``FA_IT_FTP_TLS`` + （``1`` 代表 FTPS） + * - ``test_webdav_server.py`` + - ``FA_IT_WEBDAV_URL``、``FA_IT_WEBDAV_USER``、``FA_IT_WEBDAV_PASSWORD`` + * - ``test_smb_samba.py`` + - ``FA_IT_SMB_SERVER``、``FA_IT_SMB_SHARE``、``FA_IT_SMB_USER``、 + ``FA_IT_SMB_PASSWORD``；選用 ``FA_IT_SMB_PORT``（``445``）、 + ``FA_IT_SMB_ENCRYPT``（``1``） + +第一個變數沒有設定的模組會被略過。設定 ``FA_IT_REQUIRED=1`` 時則改為失敗，CI 就是用 +這個方式確保工作不是靠著略過所有測試而通過。 + +每個測試都在自己專屬、名稱隨機的 bucket、container 或目錄中運作，結束後會把它移除。 +即使如此，仍請把測試指向一個你承擔得起寫入的帳號。 + +在 CI 中 +-------- + +``.github/workflows/integration.yml`` 會在每個 pull request、每次推送到 ``dev``、每晚 +以及手動觸發時執行。其中的 ``services`` 工作會用同一支腳本，為矩陣中的每個項目啟動一個 +服務，並執行該服務的模組。``platforms`` 工作則在 Linux 與 macOS 上執行單元測試； +Windows 由主要流程的 ``pytest`` 工作負責。兩者都不會阻擋發行。 + +Google Drive、OneDrive 與 Dropbox 沒有模擬器，因此只由它們的替身涵蓋。若要對真正的 +服務檢查其中之一，請依同樣的模式撰寫模組：先檢查變數，再寫一個 ``StorageContract`` +子類別，讓它的 ``backend`` fixture 產生以某個暫存資料夾為根目錄的轉接器。 + +新增後端 +-------- + +新的後端會附上一個針對替身的契約類別（:doc:`storage` 的「撰寫後端」），而且只要它的 +服務能在容器中執行，就會在這裡附上一個模組，並在腳本與流程矩陣中各加上一個項目。 + +發生問題時 +---------- + +.. list-table:: + :header-rows: 1 + :widths: 40 60 + + * - 你看到的 + - 代表什麼、該怎麼做 + * - ``SKIPPED … set FA_IT_S3_ENDPOINT to run against an S3 service`` + - 這個模組的變數沒有設定。請用腳本啟動服務，或自行設定變數。 + * - ``FA_IT_REQUIRED is set, but FA_IT_… is not`` + - 目前是 CI 模式，而服務沒有啟動，或沒有匯出它的變數。請查看「Start the service」 + 步驟，以及失敗後印出的服務日誌。 + * - ``nothing is listening on 127.0.0.1:`` + - 容器在 90 秒內沒有啟動完成。腳本會印出 ``docker ps -a`` 與該容器的日誌。 + * - 每個測試都出現 ``StorageUnavailableException`` + - 服務連得到，但拒絕了憑證；或者以 SFTP 而言，它的主機金鑰不在 known-hosts 檔案中。 diff --git a/docs/source/Zh-TW/zh_tw_index.rst b/docs/source/Zh-TW/zh_tw_index.rst index e9e246a..727367f 100644 --- a/docs/source/Zh-TW/zh_tw_index.rst +++ b/docs/source/Zh-TW/zh_tw_index.rst @@ -322,3 +322,17 @@ Slack、Email（SMTP）、Discord、Telegram、Microsoft Teams、PagerDuty :caption: 公開 API 與相容性 usage/api_policy + +.. _zh-tw-integration-tests: + +第 22 章 — 整合測試 +=================== + +把儲存契約測試套件拿去對容器中的真實服務執行：如何在本機執行、各模組讀取的變數， +以及 CI 執行了什麼。 + +.. toctree:: + :maxdepth: 2 + :caption: 整合測試 + + usage/integration_tests diff --git a/docs/updates/2026-10.md b/docs/updates/2026-10.md index a0f2099..d7d8dd7 100644 --- a/docs/updates/2026-10.md +++ b/docs/updates/2026-10.md @@ -527,3 +527,18 @@ Index and query commands: [README.md](README.md). New entries go at the end. - **Docs**: chapter 21 in the three manuals (`usage/api_policy.rst`), the indexes, `docs/source/API/core.rst`, a "Compatibility" section in the three READMEs, `CLAUDE.md` § Conventions, `architecture.md` §7. - **Files**: `automation_file/core/deprecation.py`, `tests/test_deprecation.py`, the documentation above. - **Open items**: #26 (release engineering, migration guide). + +## U-20261008-23 · 2026-10-08 · Integration tests and their workflow · #ci #tests #roadmap + +- **What** (roadmap §5, M3; `progress.md` #19 stays open until the first run): + - `tests/integration/`: the storage contract suite (81 cases) against a real service, one module per service: `test_s3_minio.py` (whole bucket and a prefix), `test_azure_azurite.py` (whole container and a prefix), `test_sftp_openssh.py`, `test_ftp_server.py`, `test_webdav_server.py`, `test_smb_samba.py`. Each test works in a bucket, container or directory with a random name and removes it. + - `tests/integration/service_env.py`: a module is skipped unless its `FA_IT_*` variables are set, so the ordinary test run is unchanged (six module-level skips). With `FA_IT_REQUIRED=1` a missing variable is a collection error, so a CI job cannot pass by skipping everything. + - `tests/integration/start_service.sh ` starts the service in a container (MinIO, Azurite, `atmoz/sftp`, `delfer/alpine-ftp-server`, `bytemark/webdav`, `dperson/samba`), waits for its port and exports the variables to `$GITHUB_ENV`, or prints `export` lines outside GitHub Actions. Passwords and keys are generated per run with `openssl rand`; Azurite gets a generated account key through `AZURITE_ACCOUNTS`, not its well-known development key. For SFTP the host key is read with `ssh-keyscan` into a known-hosts file, because the client rejects unknown hosts. + - `.github/workflows/integration.yml`: a `services` job (one service per matrix entry, `fail-fast: false`) and a `platforms` job that runs the unit tests on `ubuntu-latest` and `macos-latest`. It runs on pull requests, pushes to `dev`, nightly and on demand. It does not gate publishing: `publish-dev` still needs only the jobs of `ci-dev.yml`. + - `.gitattributes`: `*.sh` keeps LF line endings. +- **Verified here**: the modules import and skip in both environments; `FA_IT_REQUIRED=1` turns the skip into an error; the script passes `bash -n` and refuses an unknown service; the workflow is valid YAML and passes the repository's workflow rules (`tests/test_workflow_actions.py`: actions pinned to commits with their version, `persist-credentials` set, a timeout on every job). +- **Not verified, and it matters**: nothing in this entry has run against a service or on a GitHub runner. There is no Docker on the development machine. Unknown until the first run: whether each container starts with the options given (the Samba share options and the FTP passive-port mapping are the least certain), whether the image tags still resolve, whether the adapters pass the contract against real servers (`progress.md` #32), and whether the unit tests pass on Linux and macOS, where the symbolic-link tests run for the first time (#18) and Qt needs the system libraries the job installs. +- **Result / numbers**: 4865 passed, 155 skipped, 0 failed with every extra; 3147 passed, 96 skipped with the base dependencies only (six more skips than U-20261008-22, the integration modules, and four more cases, the workflow rules applied to the new workflow). `ruff check` and `ruff format --check` pass. Python 3.14.7 on Windows. +- **Docs**: chapter 22 in the three manuals (`usage/integration_tests.rst`), the indexes, a "Tests" section in the three READMEs, `CLAUDE.md` § Testing. +- **Files**: `tests/integration/` (`__init__.py`, `service_env.py`, six test modules, `start_service.sh`), `.github/workflows/integration.yml`, `.gitattributes`, the documentation above. +- **Open items**: #19 (first run, image pinning, credential-gated cloud jobs, FTPS), #32, #18. diff --git a/docs/updates/README.md b/docs/updates/README.md index c2b1648..f52b217 100644 --- a/docs/updates/README.md +++ b/docs/updates/README.md @@ -58,6 +58,7 @@ In the same commit: delete the item from `progress.md`, add a `#done` entry here | ID | Date | Title | Tags | Batch | |---|---|---|---|---| +| U-20261008-23 | 2026-10-08 | Integration tests and their workflow | #ci #tests #roadmap | [2026-10](2026-10.md) | | U-20261008-22 | 2026-10-08 | Public API and deprecation policy | #decision #docs #roadmap #done | [2026-10](2026-10.md) | | U-20261008-21 | 2026-10-08 | copy_between runs on the storage layer | #storage #roadmap #done | [2026-10](2026-10.md) | | U-20261008-20 | 2026-10-08 | CLI subcommands for integrity, pipelines and the audit trail | #cli #roadmap #done | [2026-10](2026-10.md) | @@ -117,5 +118,5 @@ In the same commit: delete the item from `progress.md`, add a `#done` entry here | File | Period | Entries | |---|---|---:| -| [2026-10.md](2026-10.md) | 2026-10 | 33 | +| [2026-10.md](2026-10.md) | 2026-10 | 34 | | [2026-09.md](2026-09.md) | 2026-09 | 21 | diff --git a/progress.md b/progress.md index 9a3aad8..52dcbf7 100644 --- a/progress.md +++ b/progress.md @@ -16,11 +16,11 @@ Items #10 to #26 are what is left of the 1.0 roadmap (`docs/FILEAUTOMATION-1.0-R - **#17** Native streams and checksums for the remote backends. `open_read` / `open_write`, `copy_tree` and `sync_tree` exist (U-20261008-04), but outside `LocalStorage` a stream is a staged local copy and the default `checksum` downloads the file: S3 could read `get_object()["Body"]`, and a backend with a server-side digest could answer `checksum` from it. - **#31** `FsspecStorage(directories=False)`: a `/` placeholder key that another tool wrote survives `delete(dir, recursive=True)`, so the directory still exists afterwards, and an empty directory that only a placeholder holds cannot be deleted at all. `ObjectStorage` removes placeholders because it lists raw keys; through fsspec the key has to be addressed with the filesystem's own call, and `_strip_protocol` of s3fs / gcsfs removes a trailing slash, so `rm_file(path + "/")` is probably wrong. Needs a real s3fs or gcsfs (MinIO under #19) before it is written. - **#32** [UNVERIFIED] No storage adapter has met a real service: FTP ran against an in-memory model of RFC 959 / 3659 and FTPS data channels did not run at all; SFTP met paramiko's own server, not OpenSSH; Microsoft Graph (`OneDriveClient.graph_send`), Drive (the discovery document as a transport), Dropbox, WebDAV and SMB met stand-ins, and the `smbprotocol` error shapes were written from its documentation. The hand-written redirect handling of `WebDAVClient` (U-20261008-14) has not met a server that redirects. Run each against the service (#19) and fix what differs. -- **#18** [UNVERIFIED] The five symbolic-link tests of `tests/test_storage_local.py` have not run anywhere: the development machine may not create links (they skip there), CI's Windows runners may. Read the first CI run of the branch and fix `LocalStorage` if one fails. +- **#18** [UNVERIFIED] The five symbolic-link tests of `tests/test_storage_local.py` have not run anywhere: the development machine may not create links (they skip there). The Linux and macOS jobs of `integration.yml` can; read their first run and fix `LocalStorage` if one fails. ### Backend integration tests (roadmap M3) -- **#19** Integration environments for the contract suite in CI: MinIO and Azurite (`S3Storage` and `AzureStorage` have only met stand-in clients and, for S3, botocore's Stubber; no request has reached a real service), SFTP, FTP/FTPS, WebDAV and Samba, plus credential-gated jobs for the cloud adapters, and Linux and macOS legs (`ci-dev.yml` runs pytest on Windows only). The adapters exist (U-20261008-12); what each has not met is listed in #32. +- **#19** [UNVERIFIED] `.github/workflows/integration.yml` and `tests/integration/` (U-20261008-23) have never run: there is no Docker on the development machine. The first run of the workflow decides whether each of the six service jobs (`s3`, `azure`, `sftp`, `ftp`, `webdav`, `smb`) and the two platform jobs (Linux, macOS) work. Expect to adjust the container options in `tests/integration/start_service.sh` (image tags are floating; pin them to digests once a run is green; the Samba share options and the FTP passive ports are the least certain) and to fix what a real service or another platform shows (#32, #18). Still missing: credential-gated jobs for Google Drive, OneDrive and Dropbox, which have no emulator, and FTPS. - **#20** Metadata cases in the contract suite: user metadata and content type kept across an upload and a copy where `capabilities` says the backend supports them. The failure cases exist (U-20261008-10): a backend's contract class gets them by providing the `break_storage` fixture, as the local, S3 and Azure classes do. ### Later milestones diff --git a/tests/integration/__init__.py b/tests/integration/__init__.py new file mode 100644 index 0000000..50ec8e4 --- /dev/null +++ b/tests/integration/__init__.py @@ -0,0 +1,8 @@ +"""Storage adapters against real services. + +Each module here runs the same ``StorageContract`` as the unit tests, but against +a live service (MinIO, Azurite, ...) instead of a stand-in. A module skips as a +whole unless its ``FA_IT_*`` environment variables say where the service is, so a +plain ``pytest tests/`` never needs the network. ``.github/workflows/integration.yml`` +starts the services in containers and sets the variables. +""" diff --git a/tests/integration/service_env.py b/tests/integration/service_env.py new file mode 100644 index 0000000..37e9571 --- /dev/null +++ b/tests/integration/service_env.py @@ -0,0 +1,37 @@ +"""How an integration test module finds the service it runs against. + +Each module reads ``FA_IT_*`` environment variables. Without them the module is +skipped, so the ordinary test run is unaffected. In CI ``FA_IT_REQUIRED`` is +set, and a missing variable is then an error: a job that skipped every test +would look green without having touched the service. +""" + +from __future__ import annotations + +import os +import uuid + +import pytest + +_REQUIRED = "FA_IT_REQUIRED" + + +def service_setting(name: str, service: str) -> str: + """Return the variable ``name``; skip the calling module (or fail in CI) when it is unset.""" + value = os.environ.get(name, "") + if value: + return value + message = f"set {name} to run against {service}" + if os.environ.get(_REQUIRED): + pytest.fail(f"{_REQUIRED} is set, but {name} is not: {message}", pytrace=False) + pytest.skip(message, allow_module_level=True) + + +def optional_setting(name: str, default: str) -> str: + """Return the variable ``name``, or ``default`` when it is unset or empty.""" + return os.environ.get(name, "") or default + + +def unique_name() -> str: + """Return a name no other test run uses, for a bucket, a container or a directory.""" + return f"fa-it-{uuid.uuid4().hex[:16]}" diff --git a/tests/integration/start_service.sh b/tests/integration/start_service.sh new file mode 100755 index 0000000..38760e8 --- /dev/null +++ b/tests/integration/start_service.sh @@ -0,0 +1,113 @@ +#!/usr/bin/env bash +# Start one backend service in a container and say how the integration tests reach it. +# +# bash tests/integration/start_service.sh s3|azure|sftp|ftp|webdav|smb +# +# The FA_IT_* variables the tests read are appended to $GITHUB_ENV in GitHub +# Actions. Anywhere else they are printed as `export` lines: +# +# eval "$(bash tests/integration/start_service.sh s3)" +# python -m pytest tests/integration/test_s3_minio.py +# +# Every password and key is generated here, for this run only. Nothing is stored. +set -euo pipefail + +service="${1:?usage: start_service.sh s3|azure|sftp|ftp|webdav|smb}" +user="fa" +secret="$(openssl rand -hex 16)" + +emit() { + if [ -n "${GITHUB_ENV:-}" ]; then + printf '%s=%s\n' "$1" "$2" >> "$GITHUB_ENV" + else + printf 'export %s=%q\n' "$1" "$2" + fi +} + +wait_for_port() { + local attempt + for attempt in $(seq 1 90); do + if (exec 3<>"/dev/tcp/127.0.0.1/$1") 2>/dev/null; then + return 0 + fi + sleep 1 + done + echo "nothing is listening on 127.0.0.1:$1 after ${attempt}s" >&2 + docker ps -a >&2 + docker logs "fa-it-$service" >&2 || true + return 1 +} + +case "$service" in + s3) + docker run -d --name fa-it-s3 -p 9000:9000 \ + -e "MINIO_ROOT_USER=$user-integration" -e "MINIO_ROOT_PASSWORD=$secret" \ + minio/minio server /data >&2 + wait_for_port 9000 + emit FA_IT_S3_ENDPOINT "http://127.0.0.1:9000" + emit FA_IT_S3_ACCESS_KEY "$user-integration" + emit FA_IT_S3_SECRET_KEY "$secret" + ;; + azure) + account="faintegration" + key="$(openssl rand -base64 32)" + docker run -d --name fa-it-azure -p 10000:10000 \ + -e "AZURITE_ACCOUNTS=$account:$key" \ + mcr.microsoft.com/azure-storage/azurite \ + azurite-blob --blobHost 0.0.0.0 --skipApiVersionCheck >&2 + wait_for_port 10000 + emit FA_IT_AZURE_CONNECTION_STRING \ + "DefaultEndpointsProtocol=http;AccountName=$account;AccountKey=$key;BlobEndpoint=http://127.0.0.1:10000/$account;" + ;; + sftp) + docker run -d --name fa-it-sftp -p 2222:22 atmoz/sftp "$user:$secret:1001::upload" >&2 + wait_for_port 2222 + known_hosts="${RUNNER_TEMP:-${TMPDIR:-/tmp}}/fa-it-known-hosts" + # The port accepts connections a moment before sshd hands out its keys. + for _ in $(seq 1 30); do + ssh-keyscan -p 2222 127.0.0.1 > "$known_hosts" 2>/dev/null || true + [ -s "$known_hosts" ] && break + sleep 1 + done + [ -s "$known_hosts" ] || { echo "the SFTP server offered no host key" >&2; exit 1; } + emit FA_IT_SFTP_HOST 127.0.0.1 + emit FA_IT_SFTP_PORT 2222 + emit FA_IT_SFTP_USER "$user" + emit FA_IT_SFTP_PASSWORD "$secret" + emit FA_IT_SFTP_KNOWN_HOSTS "$known_hosts" + emit FA_IT_SFTP_ROOT /upload + ;; + ftp) + docker run -d --name fa-it-ftp -p 2121:21 -p 21000-21010:21000-21010 \ + -e "USERS=$user|$secret" -e ADDRESS=127.0.0.1 -e MIN_PORT=21000 -e MAX_PORT=21010 \ + delfer/alpine-ftp-server >&2 + wait_for_port 2121 + emit FA_IT_FTP_HOST 127.0.0.1 + emit FA_IT_FTP_PORT 2121 + emit FA_IT_FTP_USER "$user" + emit FA_IT_FTP_PASSWORD "$secret" + ;; + webdav) + docker run -d --name fa-it-webdav -p 8080:80 \ + -e AUTH_TYPE=Basic -e "USERNAME=$user" -e "PASSWORD=$secret" \ + bytemark/webdav >&2 + wait_for_port 8080 + emit FA_IT_WEBDAV_URL "http://127.0.0.1:8080" + emit FA_IT_WEBDAV_USER "$user" + emit FA_IT_WEBDAV_PASSWORD "$secret" + ;; + smb) + docker run -d --name fa-it-smb -p 4450:445 \ + dperson/samba -p -u "$user;$secret" -s "share;/share;yes;no;no;$user" >&2 + wait_for_port 4450 + emit FA_IT_SMB_SERVER 127.0.0.1 + emit FA_IT_SMB_PORT 4450 + emit FA_IT_SMB_SHARE share + emit FA_IT_SMB_USER "$user" + emit FA_IT_SMB_PASSWORD "$secret" + ;; + *) + echo "unknown service: $service" >&2 + exit 2 + ;; +esac diff --git a/tests/integration/test_azure_azurite.py b/tests/integration/test_azure_azurite.py new file mode 100644 index 0000000..67a19d1 --- /dev/null +++ b/tests/integration/test_azure_azurite.py @@ -0,0 +1,59 @@ +"""AzureStorage against Azure Blob or its emulator (Azurite in CI). + +Environment: ``FA_IT_AZURE_CONNECTION_STRING``. +""" + +from __future__ import annotations + +from collections.abc import Iterator +from typing import Any + +import pytest + +from tests.integration.service_env import service_setting, unique_name + +CONNECTION_STRING = service_setting("FA_IT_AZURE_CONNECTION_STRING", "a Blob service") +blob = pytest.importorskip("azure.storage.blob", reason="needs the azure extra") + +# pylint: disable=wrong-import-position # the two guards above must come first +from automation_file.storage import AzureStorage, StorageBackend # noqa: E402 +from tests.storage_contract import StorageContract # noqa: E402 + + +@pytest.fixture(scope="module") +def service() -> Any: + return blob.BlobServiceClient.from_connection_string(CONNECTION_STRING) + + +class TestAzureServiceContract(StorageContract): + @pytest.fixture + def backend(self, service: Any) -> Iterator[StorageBackend]: + container = unique_name() + service.create_container(container) + yield AzureStorage(container, service=service) + service.delete_container(container) + + +class TestPrefixedAzureServiceContract(StorageContract): + @pytest.fixture + def backend(self, service: Any) -> Iterator[StorageBackend]: + container = unique_name() + service.create_container(container) + keep = service.get_blob_client(container=container, blob="other-tenant/keep.txt") + keep.upload_blob(b"keep") + yield AzureStorage(container, service=service, prefix="tenant/a") + assert keep.download_blob().readall() == b"keep" + service.delete_container(container) + + +def test_stat_reports_what_the_service_returns(service: Any) -> None: + container = unique_name() + service.create_container(container) + try: + info = AzureStorage(container, service=service).write_bytes("reports/q1.json", b"{}") + assert info.size == 2 + assert info.etag + assert info.content_type == "application/json" + assert info.modified_at is not None + finally: + service.delete_container(container) diff --git a/tests/integration/test_ftp_server.py b/tests/integration/test_ftp_server.py new file mode 100644 index 0000000..0ece239 --- /dev/null +++ b/tests/integration/test_ftp_server.py @@ -0,0 +1,53 @@ +"""FTPStorage against a real FTP server (a container in CI). + +Environment: ``FA_IT_FTP_HOST``, ``FA_IT_FTP_USER``, ``FA_IT_FTP_PASSWORD``, and +optionally ``FA_IT_FTP_PORT`` (21), ``FA_IT_FTP_ROOT`` (``/``, an absolute +directory the user may write to) and ``FA_IT_FTP_TLS`` (``1`` for FTPS). +""" + +from __future__ import annotations + +from collections.abc import Iterator + +import pytest + +from automation_file import FTPClient +from automation_file.storage import FTPStorage, StorageBackend +from tests.integration.service_env import optional_setting, service_setting, unique_name +from tests.storage_contract import StorageContract + +HOST = service_setting("FA_IT_FTP_HOST", "an FTP server") +ROOT = optional_setting("FA_IT_FTP_ROOT", "/") + + +@pytest.fixture(scope="module") +def client() -> Iterator[FTPClient]: + connected = FTPClient() + connected.later_init( + host=HOST, + port=int(optional_setting("FA_IT_FTP_PORT", "21")), + username=service_setting("FA_IT_FTP_USER", "an FTP server"), + password=service_setting("FA_IT_FTP_PASSWORD", "an FTP server"), + tls=optional_setting("FA_IT_FTP_TLS", "0") == "1", + ) + yield connected + connected.close() + + +class TestFTPServerContract(StorageContract): + @pytest.fixture + def backend(self, client: FTPClient) -> Iterator[StorageBackend]: + whole, name = FTPStorage(client, root=ROOT), unique_name() + whole.mkdir(name) + yield FTPStorage(client, root=f"{ROOT.rstrip('/')}/{name}") + whole.delete(name, recursive=True) + + +def test_stat_reports_what_the_server_returns(client: FTPClient) -> None: + whole, name = FTPStorage(client, root=ROOT), unique_name() + try: + info = whole.write_bytes(f"{name}/reports/q1.json", b"{}") + assert info.size == 2 + assert whole.read_bytes(f"{name}/reports/q1.json") == b"{}" + finally: + whole.delete(name, recursive=True) diff --git a/tests/integration/test_s3_minio.py b/tests/integration/test_s3_minio.py new file mode 100644 index 0000000..6057e43 --- /dev/null +++ b/tests/integration/test_s3_minio.py @@ -0,0 +1,74 @@ +"""S3Storage against an S3-compatible service (MinIO in CI). + +Environment: ``FA_IT_S3_ENDPOINT`` (for example ``http://127.0.0.1:9000``), +``FA_IT_S3_ACCESS_KEY``, ``FA_IT_S3_SECRET_KEY`` and optionally ``FA_IT_S3_REGION``. +""" + +from __future__ import annotations + +from collections.abc import Iterator +from typing import Any + +import pytest + +from tests.integration.service_env import optional_setting, service_setting, unique_name + +ENDPOINT = service_setting("FA_IT_S3_ENDPOINT", "an S3 service") +boto3 = pytest.importorskip("boto3", reason="needs the s3 extra") + +# pylint: disable=wrong-import-position # the two guards above must come first +from automation_file.storage import S3Storage, StorageBackend # noqa: E402 +from tests.storage_contract import StorageContract # noqa: E402 + + +@pytest.fixture(scope="module") +def client() -> Any: + return boto3.client( + "s3", + endpoint_url=ENDPOINT, + aws_access_key_id=service_setting("FA_IT_S3_ACCESS_KEY", "an S3 service"), + aws_secret_access_key=service_setting("FA_IT_S3_SECRET_KEY", "an S3 service"), + region_name=optional_setting("FA_IT_S3_REGION", "us-east-1"), + ) + + +def _empty_and_delete(client: Any, bucket: str) -> None: + for page in client.get_paginator("list_objects_v2").paginate(Bucket=bucket): + for entry in page.get("Contents", []): + client.delete_object(Bucket=bucket, Key=entry["Key"]) + client.delete_bucket(Bucket=bucket) + + +class TestS3ServiceContract(StorageContract): + @pytest.fixture + def backend(self, client: Any) -> Iterator[StorageBackend]: + bucket = unique_name() + client.create_bucket(Bucket=bucket) + yield S3Storage(bucket, client=client) + _empty_and_delete(client, bucket) + + +class TestPrefixedS3ServiceContract(StorageContract): + @pytest.fixture + def backend(self, client: Any) -> Iterator[StorageBackend]: + bucket = unique_name() + client.create_bucket(Bucket=bucket) + client.put_object(Bucket=bucket, Key="other-tenant/keep.txt", Body=b"keep") + yield S3Storage(bucket, client=client, prefix="tenant/a") + assert ( + client.get_object(Bucket=bucket, Key="other-tenant/keep.txt")["Body"].read() == b"keep" + ) + _empty_and_delete(client, bucket) + + +def test_stat_reports_what_the_service_returns(client: Any) -> None: + bucket = unique_name() + client.create_bucket(Bucket=bucket) + try: + info = S3Storage(bucket, client=client).write_bytes("reports/q1.json", b"{}") + assert info.size == 2 + assert info.etag + assert info.content_type == "application/json" + assert info.modified_at is not None + finally: + _empty_and_delete(client, bucket) diff --git a/tests/integration/test_sftp_openssh.py b/tests/integration/test_sftp_openssh.py new file mode 100644 index 0000000..c6d5ce1 --- /dev/null +++ b/tests/integration/test_sftp_openssh.py @@ -0,0 +1,59 @@ +"""SFTPStorage against a real SFTP server (OpenSSH in a container in CI). + +Environment: ``FA_IT_SFTP_HOST``, ``FA_IT_SFTP_USER``, ``FA_IT_SFTP_PASSWORD``, +``FA_IT_SFTP_KNOWN_HOSTS`` (a file holding the server's host key: unknown hosts +are rejected), and optionally ``FA_IT_SFTP_PORT`` (22) and ``FA_IT_SFTP_ROOT`` +(``/upload``, an absolute directory the user may write to). +""" + +from __future__ import annotations + +from collections.abc import Iterator + +import pytest + +from tests.integration.service_env import optional_setting, service_setting, unique_name + +HOST = service_setting("FA_IT_SFTP_HOST", "an SFTP server") +pytest.importorskip("paramiko", reason="needs the sftp extra") + +# pylint: disable=wrong-import-position # the two guards above must come first +from automation_file import SFTPClient # noqa: E402 +from automation_file.storage import SFTPStorage, StorageBackend # noqa: E402 +from tests.storage_contract import StorageContract # noqa: E402 + +ROOT = optional_setting("FA_IT_SFTP_ROOT", "/upload") + + +@pytest.fixture(scope="module") +def client() -> Iterator[SFTPClient]: + connected = SFTPClient() + connected.later_init( + host=HOST, + port=int(optional_setting("FA_IT_SFTP_PORT", "22")), + username=service_setting("FA_IT_SFTP_USER", "an SFTP server"), + password=service_setting("FA_IT_SFTP_PASSWORD", "an SFTP server"), + known_hosts=service_setting("FA_IT_SFTP_KNOWN_HOSTS", "an SFTP server"), + ) + yield connected + connected.close() + + +class TestSFTPServerContract(StorageContract): + @pytest.fixture + def backend(self, client: SFTPClient) -> Iterator[StorageBackend]: + whole, name = SFTPStorage(client, root=ROOT), unique_name() + whole.mkdir(name) + yield SFTPStorage(client, root=f"{ROOT}/{name}") + whole.delete(name, recursive=True) + + +def test_stat_reports_what_the_server_returns(client: SFTPClient) -> None: + whole, name = SFTPStorage(client, root=ROOT), unique_name() + try: + info = whole.write_bytes(f"{name}/reports/q1.json", b"{}") + assert info.size == 2 + assert info.modified_at is not None + assert whole.uri_for(f"{name}/reports/q1.json").startswith(f"sftp://{HOST}") + finally: + whole.delete(name, recursive=True) diff --git a/tests/integration/test_smb_samba.py b/tests/integration/test_smb_samba.py new file mode 100644 index 0000000..5207dc2 --- /dev/null +++ b/tests/integration/test_smb_samba.py @@ -0,0 +1,53 @@ +"""SMBStorage against a real SMB server (Samba in a container in CI). + +Environment: ``FA_IT_SMB_SERVER``, ``FA_IT_SMB_SHARE``, ``FA_IT_SMB_USER``, +``FA_IT_SMB_PASSWORD``, and optionally ``FA_IT_SMB_PORT`` (445) and +``FA_IT_SMB_ENCRYPT`` (``1``; ``0`` for a server without SMB3 encryption). +""" + +from __future__ import annotations + +from collections.abc import Iterator + +import pytest + +from tests.integration.service_env import optional_setting, service_setting, unique_name + +SERVER = service_setting("FA_IT_SMB_SERVER", "an SMB server") +pytest.importorskip("smbclient", reason="needs the smb extra") + +# pylint: disable=wrong-import-position # the two guards above must come first +from automation_file import SMBClient # noqa: E402 +from automation_file.storage import SMBStorage, StorageBackend # noqa: E402 +from tests.storage_contract import StorageContract # noqa: E402 + + +@pytest.fixture(scope="module") +def client() -> SMBClient: + return SMBClient( + SERVER, + service_setting("FA_IT_SMB_SHARE", "an SMB server"), + service_setting("FA_IT_SMB_USER", "an SMB server"), + service_setting("FA_IT_SMB_PASSWORD", "an SMB server"), + port=int(optional_setting("FA_IT_SMB_PORT", "445")), + encrypt=optional_setting("FA_IT_SMB_ENCRYPT", "1") == "1", + ) + + +class TestSMBServerContract(StorageContract): + @pytest.fixture + def backend(self, client: SMBClient) -> Iterator[StorageBackend]: + whole, name = SMBStorage(client), unique_name() + whole.mkdir(name) + yield SMBStorage(client, root=name) + whole.delete(name, recursive=True) + + +def test_stat_reports_what_the_server_returns(client: SMBClient) -> None: + whole, name = SMBStorage(client), unique_name() + try: + info = whole.write_bytes(f"{name}/reports/q1.json", b"{}") + assert info.size == 2 + assert info.modified_at is not None + finally: + whole.delete(name, recursive=True) diff --git a/tests/integration/test_webdav_server.py b/tests/integration/test_webdav_server.py new file mode 100644 index 0000000..02eeb76 --- /dev/null +++ b/tests/integration/test_webdav_server.py @@ -0,0 +1,50 @@ +"""WebDAVStorage against a real WebDAV server (Apache mod_dav in a container in CI). + +Environment: ``FA_IT_WEBDAV_URL`` (for example ``http://127.0.0.1:8080``), +``FA_IT_WEBDAV_USER`` and ``FA_IT_WEBDAV_PASSWORD``. +""" + +from __future__ import annotations + +from collections.abc import Iterator + +import pytest + +from automation_file import WebDAVClient +from automation_file.storage import StorageBackend, WebDAVStorage +from tests.integration.service_env import service_setting, unique_name +from tests.storage_contract import StorageContract + +URL = service_setting("FA_IT_WEBDAV_URL", "a WebDAV server") + + +@pytest.fixture(scope="module") +def client() -> Iterator[WebDAVClient]: + # The server of a test run is on the loopback interface, which the URL guard refuses + # unless it is told the host is meant to be private. + with WebDAVClient( + URL, + service_setting("FA_IT_WEBDAV_USER", "a WebDAV server"), + service_setting("FA_IT_WEBDAV_PASSWORD", "a WebDAV server"), + allow_private_hosts=True, + ) as connected: + yield connected + + +class TestWebDAVServerContract(StorageContract): + @pytest.fixture + def backend(self, client: WebDAVClient) -> Iterator[StorageBackend]: + whole, name = WebDAVStorage(client), unique_name() + whole.mkdir(name) + yield WebDAVStorage(client, root=name) + whole.delete(name, recursive=True) + + +def test_stat_reports_what_the_server_returns(client: WebDAVClient) -> None: + whole, name = WebDAVStorage(client), unique_name() + try: + info = whole.write_bytes(f"{name}/reports/q1.json", b"{}") + assert info.size == 2 + assert info.modified_at is not None + finally: + whole.delete(name, recursive=True) From 541adc9ae9fde218dc9ca0ffbd6c66ed5e399e8d Mon Sep 17 00:00:00 2001 From: JeffreyChen Date: Thu, 8 Oct 2026 14:54:05 +0800 Subject: [PATCH 40/59] ci: let a release raise MINOR or MAJOR, and check the package build --- .github/workflows/ci-dev.yml | 22 +++++ .github/workflows/ci-stable.yml | 22 +++++ .github/workflows/publish.yml | 36 +-------- CLAUDE.md | 7 +- docs/updates/2026-10.md | 16 ++++ docs/updates/README.md | 3 +- progress.md | 4 +- scripts/stable_release.py | 134 ++++++++++++++++++++++++++++++ tests/test_stable_release.py | 139 ++++++++++++++++++++++++++++++++ 9 files changed, 346 insertions(+), 37 deletions(-) create mode 100644 scripts/stable_release.py create mode 100644 tests/test_stable_release.py diff --git a/.github/workflows/ci-dev.yml b/.github/workflows/ci-dev.yml index 5add4f6..0809a88 100644 --- a/.github/workflows/ci-dev.yml +++ b/.github/workflows/ci-dev.yml @@ -116,6 +116,28 @@ jobs: - name: Run pytest run: python -m pytest tests/ -v --tb=short + package: + # The sdist and the wheel build from dev.toml and their metadata renders, before anything is + # merged. It gates nothing: the publish job builds again from the locked tools. + needs: lint + runs-on: ubuntu-latest + timeout-minutes: 15 # no run yet: the floor of 15 minutes, to revisit after the first runs + steps: + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + persist-credentials: false + - name: Set up Python + uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0 + with: + python-version: "3.12" + cache: pip + - name: Build the distribution and check its metadata + run: | + python -m pip install --upgrade pip build twine + cp dev.toml pyproject.toml + python -m build + twine check dist/* + publish-dev: # The dev channel. A push to dev that passes lint and the tests is built from dev.toml and uploaded # when it is still the tip of dev and ships something the newest automation_file_dev does not. diff --git a/.github/workflows/ci-stable.yml b/.github/workflows/ci-stable.yml index ce50e92..38aec62 100644 --- a/.github/workflows/ci-stable.yml +++ b/.github/workflows/ci-stable.yml @@ -115,3 +115,25 @@ jobs: pip install -e ".[${{ matrix.extra }},test]" - name: Run pytest run: python -m pytest tests/ -v --tb=short + + package: + # The sdist and the wheel build from stable.toml and their metadata renders, before anything is + # merged. It gates nothing: the publish job builds again from the locked tools. + needs: lint + runs-on: ubuntu-latest + timeout-minutes: 15 # no run yet: the floor of 15 minutes, to revisit after the first runs + steps: + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + persist-credentials: false + - name: Set up Python + uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0 + with: + python-version: "3.12" + cache: pip + - name: Build the distribution and check its metadata + run: | + python -m pip install --upgrade pip build twine + cp stable.toml pyproject.toml + python -m build + twine check dist/* diff --git a/.github/workflows/publish.yml b/.github/workflows/publish.yml index 10c3204..131553f 100644 --- a/.github/workflows/publish.yml +++ b/.github/workflows/publish.yml @@ -33,39 +33,11 @@ jobs: run: | python -m pip install --require-hashes --only-binary :all: -r .github/requirements/publish.txt - - name: Bump patch version in stable.toml and dev.toml + # A patch release unless the pull request raised MAJOR or MINOR in stable.toml and dev.toml. + # scripts/stable_release.py says how the two are told apart. + - name: Pick the release version in stable.toml and dev.toml id: bump - run: | - python <<'EOF' - import os - import pathlib - import re - - def bump(path: pathlib.Path) -> str: - text = path.read_text(encoding="utf-8") - match = re.search(r'^version\s*=\s*"(\d+)\.(\d+)\.(\d+)"', text, re.MULTILINE) - if not match: - raise SystemExit(f"Could not find version in {path}") - major, minor, patch = (int(part) for part in match.groups()) - new_version = f"{major}.{minor}.{patch + 1}" - new_text = re.sub( - r'^(version\s*=\s*)"\d+\.\d+\.\d+"', - rf'\1"{new_version}"', - text, - count=1, - flags=re.MULTILINE, - ) - path.write_text(new_text, encoding="utf-8") - return new_version - - stable_version = bump(pathlib.Path("stable.toml")) - dev_version = bump(pathlib.Path("dev.toml")) - with open(os.environ["GITHUB_OUTPUT"], "a", encoding="utf-8") as output: - output.write(f"new_version={stable_version}\n") - output.write(f"dev_version={dev_version}\n") - print(f"stable.toml -> {stable_version}") - print(f"dev.toml -> {dev_version}") - EOF + run: python scripts/stable_release.py bump - name: Use stable.toml as pyproject.toml run: cp stable.toml pyproject.toml diff --git a/CLAUDE.md b/CLAUDE.md index 3e71702..b9cbe1b 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -98,10 +98,11 @@ automation_file/ - `main` branch: stable releases, publishes `automation_file` to PyPI (version in `stable.toml`). - `dev` branch: development, publishes `automation_file_dev` to PyPI from CI. The version in `dev.toml` is only a floor. - Keep `dependencies` and `[project.optional-dependencies]` (`dev`) in sync across both TOMLs; `tests/test_dev_toml_parity.py` fails when those, the entry points, `requires-python`, `[build-system]` or `[tool.setuptools]` differ. The base `dependencies` carry no cloud SDK and no GUI toolkit: each backend's SDK, `pyarrow` and `PySide6` live in an extra (`s3`, `azure`, `gdrive`, `dropbox`, `sftp`, `smb`, `fsspec`, `onedrive`, `box`, `parquet`, `gui`; `ftp` and `webdav` are empty; `all` lists every one). Do not move one into `dependencies`, and do not import one at module level: code asks for it at the moment of use with `automation_file.core.optional.require_module(name, extra=...)`, which raises `OptionalDependencyException` naming the extra. `tests/test_optional_dependencies.py` fails when the package cannot be imported without them, when importing it loads one, or when the extras and `all` drift apart. A new optional package needs its extra in both TOMLs, its line in `all`, and its entry in `core.optional.EXTRAS`. -- **Version bumping is automatic.** A dedicated publish workflow bumps the patch in both `stable.toml` and `dev.toml`, builds, uploads to PyPI, then commits the bump back to `main` tagged as `vX.Y.Z`. Do not hand-bump before merging to `main`. The next publish run is skipped via a commit-message guard (`chore: bump version`), so the bump itself never re-triggers publishing. The dev channel takes its number from PyPI, so never hand-bump `dev.toml` either. +- **Patch releases bump themselves.** The publish workflow runs `scripts/stable_release.py bump`, which raises the patch in both `stable.toml` and `dev.toml`, then builds, uploads to PyPI and commits the bump back to `main` tagged as `vX.Y.Z`. Do not hand-bump the patch before merging to `main`. The next publish run is skipped via a commit-message guard (`chore: bump version`), so the bump itself never re-triggers publishing. The dev channel takes its number from PyPI, so never hand-bump `dev.toml` for a patch either. +- **A MINOR or MAJOR release is written in the pull request.** Set the version to `X.Y.0` in both `stable.toml` and `dev.toml` in the pull request that goes to `main`. When a file's `MAJOR.MINOR` is above the newest `vX.Y.Z` tag's, the script publishes the version as written instead of adding a patch; it stops the job when the stable version would not be above the newest tag. What counts as MINOR and MAJOR is in `docs/source/Eng/usage/api_policy.rst`. - CI: GitHub Actions — a `lint` job on Ubuntu (Python 3.12), then `pytest` on Windows across Python 3.10 / 3.11 / 3.12 / 3.13 / 3.14. One workflow per branch: `.github/workflows/ci-dev.yml`, `.github/workflows/ci-stable.yml`. -- CI steps: `lint` (ruff check + ruff format --check + mypy) → `pytest` with coverage, installed with `.[all,test]` → uploads `coverage.xml` as an artifact. Two more jobs follow `lint`: `minimal` installs `.[test]` only and runs the whole suite (the tests of a missing extra skip), and `extras` installs each extra on its own. `publish-dev` needs all four. -- Stable publishing lives in a separate workflow (`.github/workflows/publish.yml`) that runs on push to `main`: bumps both TOMLs, copies `stable.toml` to `pyproject.toml`, builds the sdist + wheel, `twine upload` via `PYPI_API_TOKEN`, then commits + tags + pushes and creates `gh release create v --generate-notes`. +- CI steps: `lint` (ruff check + ruff format --check + mypy) → `pytest` with coverage, installed with `.[all,test]` → uploads `coverage.xml` as an artifact. Two more jobs follow `lint`: `minimal` installs `.[test]` only and runs the whole suite (the tests of a missing extra skip), and `extras` installs each extra on its own. `publish-dev` needs all four. A `package` job builds the sdist and the wheel and runs `twine check`; it gates nothing. `.github/workflows/integration.yml` (§ Testing) runs next to them and gates nothing either. +- Stable publishing lives in a separate workflow (`.github/workflows/publish.yml`) that runs on push to `main`: picks the version in both TOMLs (`scripts/stable_release.py`), copies `stable.toml` to `pyproject.toml`, builds the sdist + wheel, `twine upload` via `PYPI_API_TOKEN`, then commits + tags + pushes and creates `gh release create v --generate-notes`. - Dev publishing is the `publish-dev` job at the end of `ci-dev.yml`. It runs only on a push to `dev`, after `lint` and `pytest` pass: `scripts/dev_release.py prepare` writes `pyproject.toml` from `dev.toml` with one patch above the newest `automation_file_dev` on PyPI, the job builds and runs `twine check`, and it uploads (same `PYPI_API_TOKEN`) only when the commit is still the tip of `dev` and the wheel differs from the newest published one. Nothing is committed back. - Both publish jobs hold `PYPI_API_TOKEN`, so they install their tools (`build`, `twine`, and the build backend `setuptools`) with one command and nothing else: `python -m pip install --require-hashes --only-binary :all: -r .github/requirements/publish.txt`. No `pip install --upgrade pip`, no unpinned install; `tests/test_workflow_actions.py` fails on any other `pip install` in a job that is given the token. To add or raise a tool, edit `.github/requirements/publish.in` and regenerate `publish.txt` with the `uv pip compile` command written in that file. Dependabot reads the directory and proposes updates on `dev`. - Both publish jobs build with `python -m build --no-isolation`, so the backend is the locked `setuptools` and nothing is downloaded at build time. `--no-isolation` checks `[build-system] requires` against what is installed instead of installing it: when you raise that floor in `stable.toml` and `dev.toml`, or add a build requirement, regenerate `publish.txt` in the same commit. `tests/test_workflow_actions.py` fails on a build without `--no-isolation` in those jobs and on a build requirement the lock does not satisfy. diff --git a/docs/updates/2026-10.md b/docs/updates/2026-10.md index d7d8dd7..96bdce3 100644 --- a/docs/updates/2026-10.md +++ b/docs/updates/2026-10.md @@ -542,3 +542,19 @@ Index and query commands: [README.md](README.md). New entries go at the end. - **Docs**: chapter 22 in the three manuals (`usage/integration_tests.rst`), the indexes, a "Tests" section in the three READMEs, `CLAUDE.md` § Testing. - **Files**: `tests/integration/` (`__init__.py`, `service_env.py`, six test modules, `start_service.sh`), `.github/workflows/integration.yml`, `.gitattributes`, the documentation above. - **Open items**: #19 (first run, image pinning, credential-gated cloud jobs, FTPS), #32, #18. + +## U-20261008-24 · 2026-10-08 · A release can raise MINOR or MAJOR · #release #ci #roadmap + +- **Why**: the publish workflow could only add one to the patch number, so the versioning policy (U-20261008-22) had no way to produce `1.0.0` or `1.1.0`. +- **What**: + - `scripts/stable_release.py bump` replaces the inline script of `.github/workflows/publish.yml`. For `stable.toml` and `dev.toml` it compares the file's `MAJOR.MINOR` with the newest `vX.Y.Z` tag: when the file is above it, the version written in the file is published as it stands; otherwise the patch goes up by one, as before. So a MINOR or MAJOR release is made by writing `X.Y.0` in both files in the pull request to `main`, where it is reviewed with the rest. + - It stops the job, before anything is built, when the stable version would not be above the newest tag, and writes nothing unless both files can be written. + - The outputs (`new_version`, `dev_version`), the commit, the tag and the GitHub release are unchanged. The checkout already fetched every tag (`fetch-depth: 0`). + - A `package` job in `ci-dev.yml` and `ci-stable.yml` builds the sdist and the wheel and runs `twine check` on pull requests. It gates nothing; the publish jobs build again from their locked tools. +- **Tests**: `tests/test_stable_release.py`, 18 cases: the newest tag among mixed tags, the patch and the raised cases for both files, a refused version, a file without a version, `main` with `$GITHUB_OUTPUT`, the exit codes, the tags of this checkout, and the order of the workflow's steps. The repository's workflow rules (`tests/test_workflow_actions.py`) pass for the changed and the new jobs. +- **Result / numbers**: 4883 passed, 155 skipped, 0 failed with every extra in four of five runs; 3165 passed, 96 skipped with the base dependencies only. One of the five full runs reported `1 failed, 4882 passed` while three other test suites were running on the machine; the failing test was not captured and did not fail again (`progress.md` #35). `ruff check` and `ruff format --check` pass. Python 3.14.7 on Windows. +- **Not verified**: the workflow itself. It runs only on a push to `main`; the script was tested with the tag list stubbed and against this checkout's real tags, not inside the job. The `package` job has not run: `build` and `twine` are not installed on the development machine. +- **Not done**: PyPI Trusted Publishing, which needs the owner to register the publishers on PyPI first (`progress.md` #34). +- **Docs**: `CLAUDE.md` § Branching & CI (how a patch and how a MINOR or MAJOR release is made). +- **Files**: `scripts/stable_release.py`, `tests/test_stable_release.py`, `.github/workflows/publish.yml`, `.github/workflows/ci-dev.yml`, `.github/workflows/ci-stable.yml`, `CLAUDE.md`, `progress.md`. +- **Open items**: #26 (migration guide, documentation audit, the 1.0.0 release), #34. diff --git a/docs/updates/README.md b/docs/updates/README.md index f52b217..eaa10e8 100644 --- a/docs/updates/README.md +++ b/docs/updates/README.md @@ -58,6 +58,7 @@ In the same commit: delete the item from `progress.md`, add a `#done` entry here | ID | Date | Title | Tags | Batch | |---|---|---|---|---| +| U-20261008-24 | 2026-10-08 | A release can raise MINOR or MAJOR | #release #ci #roadmap | [2026-10](2026-10.md) | | U-20261008-23 | 2026-10-08 | Integration tests and their workflow | #ci #tests #roadmap | [2026-10](2026-10.md) | | U-20261008-22 | 2026-10-08 | Public API and deprecation policy | #decision #docs #roadmap #done | [2026-10](2026-10.md) | | U-20261008-21 | 2026-10-08 | copy_between runs on the storage layer | #storage #roadmap #done | [2026-10](2026-10.md) | @@ -118,5 +119,5 @@ In the same commit: delete the item from `progress.md`, add a `#done` entry here | File | Period | Entries | |---|---|---:| -| [2026-10.md](2026-10.md) | 2026-10 | 34 | +| [2026-10.md](2026-10.md) | 2026-10 | 35 | | [2026-09.md](2026-09.md) | 2026-09 | 21 | diff --git a/progress.md b/progress.md index 52dcbf7..6c58aad 100644 --- a/progress.md +++ b/progress.md @@ -28,7 +28,9 @@ Items #10 to #26 are what is left of the 1.0 roadmap (`docs/FILEAUTOMATION-1.0-R - **#23** Scheduler v2 (roadmap §8, the open half of M6): one scheduler with cron (time-zone aware), manual, file-event, webhook and pipeline-dependency triggers, run states and overlap protection, reading a pipeline's `schedule`. The event model, the `NotificationRouter` and audit schema v2 are done (U-20261008-05, U-20261008-17); the scheduler in `scheduler/` still dispatches action lists on its own cron loop. - **#24** UI 2.0 (roadmap §11, M7). Not before the APIs of #13 to #23 are stable (roadmap §20). - **#25** Semantic MCP tools (roadmap §12, M8): `file_*`, `storage_*`, `pipeline_*`, `integrity_status`, `audit_search`, with a permission model and dry run, next to the existing `FA_*` bridge. -- **#26** Release engineering and 1.0 (roadmap §13, M9): contract and integration tests in the PR gate, PyPI Trusted Publishing, SemVer, migration guide, API freeze. +- **#26** Release engineering and 1.0 (roadmap §13, M9). Done: semantic versioning with a way to release a MINOR or MAJOR (U-20261008-24), the public API policy (U-20261008-22), a package-build check and the integration workflow next to the PR checks. Open: the migration guide and the final documentation audit (after the scheduler, MCP and GUI work lands); making the integration jobs required once they are green (#19); the 1.0.0 release itself, which is the owner's call: write `1.0.0` in both TOMLs in the release pull request. +- **#35** [UNVERIFIED] One full run of the suite on 2026-10-08 reported `1 failed, 4882 passed` and four runs around it passed. The machine was running three other test suites at the time, and the run was not started with `-rf`, so the test is not known. A test that depends on timing is the likely cause (the pipeline timeout and cancellation cases, the integrity watchers, the SFTP loopback, the scheduler). Run the suite with `-rf` under load, or read the first CI runs, to find it. +- **#34** [BLOCKED] PyPI Trusted Publishing (roadmap §13). `publish.yml` and `publish-dev` still upload with the `PYPI_API_TOKEN` secret. Switching needs the owner to add a trusted publisher for each project on PyPI (`automation_file`: workflow `publish.yml`; `automation_file_dev`: workflow `ci-dev.yml`; an environment name if one is wanted) before the workflows can drop the token for `id-token: write` and `pypa/gh-action-pypi-publish`. Changing the workflows first would break both channels. ### Packaging follow-ups diff --git a/scripts/stable_release.py b/scripts/stable_release.py new file mode 100644 index 0000000..d664444 --- /dev/null +++ b/scripts/stable_release.py @@ -0,0 +1,134 @@ +"""Pick the version of a stable release of ``automation_file``. + +The ``publish`` job of ``.github/workflows/publish.yml`` runs it on a push to ``main``:: + + python scripts/stable_release.py bump + +For ``stable.toml`` and for ``dev.toml`` it decides between two cases: + +* **A patch release**, the usual one: the patch number goes up by one. +* **A minor or major release**: a pull request raises ``MAJOR`` or ``MINOR`` by writing + the version it wants (``1.1.0``, ``2.0.0``) in the file. When the file's ``MAJOR.MINOR`` + is above the newest release tag's, the version in the file is the release and nothing is + added to it. + +The stable version must end up above the newest ``vX.Y.Z`` tag; anything else stops the +job before it builds. The two versions are written to ``$GITHUB_OUTPUT`` as ``new_version`` +and ``dev_version``. +""" + +from __future__ import annotations + +import os +import re +import subprocess +import sys +from collections.abc import Iterable +from pathlib import Path + +Version = tuple[int, int, int] + +VERSION_LINE = re.compile(r'^(version\s*=\s*)"(\d+)\.(\d+)\.(\d+)"', re.MULTILINE) +TAG = re.compile(r"^v(\d+)\.(\d+)\.(\d+)$") +GIT_TIMEOUT_SECONDS = 60 +STABLE = "stable.toml" +DEV = "dev.toml" + + +class ReleaseError(ValueError): + """The release cannot be made as the files and the tags stand. + + A plain ``ValueError``: the publish job runs this script without the package installed. + """ + + +def text_of(version: Version) -> str: + return ".".join(str(part) for part in version) + + +def version_in(text: str, name: str) -> Version: + """Return the ``version = "X.Y.Z"`` of a metadata file's text.""" + match = VERSION_LINE.search(text) + if match is None: + raise ReleaseError(f'{name} has no version = "X.Y.Z" line') + return int(match.group(2)), int(match.group(3)), int(match.group(4)) + + +def newest_release(tags: Iterable[str]) -> Version | None: + """Return the highest ``vX.Y.Z`` among ``tags``, or ``None`` when there is none.""" + found = [TAG.match(tag.strip()) for tag in tags] + versions = [(int(m.group(1)), int(m.group(2)), int(m.group(3))) for m in found if m] + return max(versions, default=None) + + +def next_version(current: Version, released: Version | None) -> Version: + """Return the version to publish for a file that says ``current``. + + A file whose ``MAJOR.MINOR`` was raised above the newest release keeps what it + says. Without a release to compare with, the patch goes up as it always did. + """ + if released is not None and current[:2] > released[:2]: + return current + return current[0], current[1], current[2] + 1 + + +def write_version(path: Path, version: Version) -> None: + text = path.read_text(encoding="utf-8") + replaced = VERSION_LINE.sub(rf'\g<1>"{text_of(version)}"', text, count=1) + path.write_text(replaced, encoding="utf-8") + + +def git_tags(root: Path) -> list[str]: + """Return the release tags of the checkout at ``root``.""" + # A fixed argument list and no shell: nothing here comes from outside the workflow. + result = subprocess.run( # nosec B603 B607 + ["git", "tag", "--list", "v*"], + cwd=root, + capture_output=True, + text=True, + timeout=GIT_TIMEOUT_SECONDS, + check=True, + ) + return result.stdout.splitlines() + + +def bump(root: Path, released: Version | None) -> tuple[Version, Version]: + """Write the release versions into both metadata files and return ``(stable, dev)``.""" + chosen: list[Version] = [] + for name in (STABLE, DEV): + path = root / name + version = next_version(version_in(path.read_text(encoding="utf-8"), name), released) + chosen.append(version) + stable, dev = chosen + if released is not None and stable <= released: + raise ReleaseError( + f"{STABLE} would release {text_of(stable)}, which is not above the newest tag " + f"v{text_of(released)}" + ) + write_version(root / STABLE, stable) + write_version(root / DEV, dev) + return stable, dev + + +def main(argv: list[str] | None = None) -> int: + arguments = sys.argv[1:] if argv is None else argv + if arguments != ["bump"]: + sys.stderr.write("usage: stable_release.py bump\n") + return 2 + root = Path.cwd() + try: + stable, dev = bump(root, newest_release(git_tags(root))) + except ReleaseError as error: + sys.stderr.write(f"{error}\n") + return 1 + output = os.environ.get("GITHUB_OUTPUT") + if output: + with open(output, "a", encoding="utf-8") as handle: + handle.write(f"new_version={text_of(stable)}\n") + handle.write(f"dev_version={text_of(dev)}\n") + sys.stdout.write(f"{STABLE} -> {text_of(stable)}\n{DEV} -> {text_of(dev)}\n") + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/tests/test_stable_release.py b/tests/test_stable_release.py new file mode 100644 index 0000000..48e590c --- /dev/null +++ b/tests/test_stable_release.py @@ -0,0 +1,139 @@ +"""The stable release helper: which version a push to ``main`` publishes.""" + +from __future__ import annotations + +import importlib.util +import re +from pathlib import Path + +import pytest + +REPO_ROOT = Path(__file__).resolve().parents[1] +SCRIPT = REPO_ROOT / "scripts" / "stable_release.py" +WORKFLOW = REPO_ROOT / ".github" / "workflows" / "publish.yml" + + +def _load_script(): + spec = importlib.util.spec_from_file_location("stable_release", SCRIPT) + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + return module + + +stable_release = _load_script() + + +def _metadata(root: Path, stable: str, dev: str) -> None: + (root / "stable.toml").write_text( + f'[project]\nname = "automation_file"\nversion = "{stable}"\nrequires-python = ">=3.10"\n', + encoding="utf-8", + ) + (root / "dev.toml").write_text( + f'[project]\nname = "automation_file_dev"\nversion = "{dev}"\n', encoding="utf-8" + ) + + +def _versions(root: Path) -> tuple[str, str]: + stable = (root / "stable.toml").read_text(encoding="utf-8") + dev = (root / "dev.toml").read_text(encoding="utf-8") + return ( + re.search(r'^version = "([^"]+)"', stable, re.MULTILINE).group(1), + re.search(r'^version = "([^"]+)"', dev, re.MULTILINE).group(1), + ) + + +@pytest.mark.parametrize( + "tags,expected", + [ + (["v0.0.9", "v0.0.51", "v0.0.10"], (0, 0, 51)), + (["v1.2.0", "v1.10.0", "v1.9.9"], (1, 10, 0)), + (["v0.0.51\n", "release-1", "v2", "v1.0.0rc1", "1.0.0"], (0, 0, 51)), + ([], None), + ], +) +def test_the_newest_release_is_the_highest_plain_tag(tags, expected): + assert stable_release.newest_release(tags) == expected + + +@pytest.mark.parametrize( + "current,released,expected", + [ + ((0, 0, 51), (0, 0, 51), (0, 0, 52)), # the usual patch release + ((0, 0, 33), (0, 0, 51), (0, 0, 34)), # dev.toml is a floor of its own + ((1, 0, 0), (0, 0, 51), (1, 0, 0)), # MAJOR was raised: published as written + ((1, 1, 0), (1, 0, 7), (1, 1, 0)), # MINOR was raised + ((1, 1, 0), (1, 1, 0), (1, 1, 1)), # the release after it is a patch again + ((0, 0, 5), None, (0, 0, 6)), # no tag to compare with + ], +) +def test_the_next_version_is_a_patch_unless_major_or_minor_was_raised(current, released, expected): + assert stable_release.next_version(current, released) == expected + + +def test_bump_writes_a_patch_release_into_both_files(tmp_path): + _metadata(tmp_path, "0.0.51", "0.0.33") + assert stable_release.bump(tmp_path, (0, 0, 51)) == ((0, 0, 52), (0, 0, 34)) + assert _versions(tmp_path) == ("0.0.52", "0.0.34") + assert 'requires-python = ">=3.10"' in (tmp_path / "stable.toml").read_text(encoding="utf-8") + + +def test_bump_keeps_a_raised_major_or_minor_as_written(tmp_path): + _metadata(tmp_path, "1.0.0", "1.0.0") + assert stable_release.bump(tmp_path, (0, 0, 51)) == ((1, 0, 0), (1, 0, 0)) + assert _versions(tmp_path) == ("1.0.0", "1.0.0") + _metadata(tmp_path, "1.1.0", "1.0.4") + assert stable_release.bump(tmp_path, (1, 0, 3)) == ((1, 1, 0), (1, 0, 5)) + + +def test_bump_refuses_a_version_that_is_not_above_the_newest_tag(tmp_path): + _metadata(tmp_path, "0.0.40", "0.0.33") + with pytest.raises( + stable_release.ReleaseError, match=r"0\.0\.41, which is not above .* v0\.0\.51" + ): + stable_release.bump(tmp_path, (0, 0, 51)) + assert _versions(tmp_path) == ("0.0.40", "0.0.33") + + +def test_a_file_without_a_version_is_an_error(tmp_path): + _metadata(tmp_path, "0.0.51", "0.0.33") + (tmp_path / "dev.toml").write_text('[project]\nname = "x"\n', encoding="utf-8") + with pytest.raises(stable_release.ReleaseError, match="has no version"): + stable_release.bump(tmp_path, (0, 0, 51)) + # Nothing is written unless both files can be: stable.toml keeps its version. + assert 'version = "0.0.51"' in (tmp_path / "stable.toml").read_text(encoding="utf-8") + + +def test_main_reads_the_tags_and_writes_what_the_workflow_reads(tmp_path, monkeypatch, capsys): + _metadata(tmp_path, "0.0.51", "0.0.33") + output = tmp_path / "github_output" + monkeypatch.chdir(tmp_path) + monkeypatch.setenv("GITHUB_OUTPUT", str(output)) + monkeypatch.setattr(stable_release, "git_tags", lambda root: ["v0.0.50", "v0.0.51"]) + assert stable_release.main(["bump"]) == 0 + assert output.read_text(encoding="utf-8") == "new_version=0.0.52\ndev_version=0.0.34\n" + assert "stable.toml -> 0.0.52" in capsys.readouterr().out + + +def test_main_stops_the_job_when_the_release_cannot_be_made(tmp_path, monkeypatch, capsys): + _metadata(tmp_path, "0.0.40", "0.0.33") + monkeypatch.chdir(tmp_path) + monkeypatch.delenv("GITHUB_OUTPUT", raising=False) + monkeypatch.setattr(stable_release, "git_tags", lambda root: ["v0.0.51"]) + assert stable_release.main(["bump"]) == 1 + assert "not above the newest tag" in capsys.readouterr().err + assert stable_release.main(["release"]) == 2 + + +def test_git_tags_lists_the_tags_of_this_checkout(): + tags = stable_release.git_tags(REPO_ROOT) + assert all(tag.startswith("v") for tag in tags) + + +def test_the_workflow_picks_the_version_before_it_builds_and_has_every_tag(): + job = WORKFLOW.read_text(encoding="utf-8") + pick = job.index("python scripts/stable_release.py bump") + assert pick < job.index("cp stable.toml pyproject.toml") < job.index("python -m build") + assert "id: bump" in job + # The newest tag decides between a patch and a raised version, so the checkout needs them all. + assert "fetch-depth: 0" in job + assert "steps.bump.outputs.new_version" in job From 036ebfa22f687c0c00b20315f9e75abd6fec8f60 Mon Sep 17 00:00:00 2001 From: JeffreyChen Date: Thu, 8 Oct 2026 14:54:58 +0800 Subject: [PATCH 41/59] docs: correct the README line that said the backends are installed by default --- README.md | 2 +- README.zh-CN.md | 2 +- README.zh-TW.md | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index 4e34241..9a858a7 100644 --- a/README.md +++ b/README.md @@ -12,7 +12,7 @@ facade. - Local file / directory / ZIP operations with path traversal guard (`safe_join`) - Validated HTTP downloads with SSRF protections, retry, and size / time caps - Google Drive CRUD (upload, download, search, delete, share, folders) -- First-class S3, Azure Blob, Dropbox, and SFTP backends — installed by default +- S3, Azure Blob, Dropbox, SFTP and seven more remote backends, each installed with its own extra (`pip install "automation_file[s3]"`, or `[all]` for every one) - JSON action lists executed by a shared `ActionExecutor` — validate, dry-run, parallel - Loopback-first TCP **and** HTTP servers that accept JSON command batches with optional shared-secret auth - Reliability primitives: `retry_on_transient` decorator, `Quota` size / time budgets diff --git a/README.zh-CN.md b/README.zh-CN.md index 3dedbf2..ef841e6 100644 --- a/README.zh-CN.md +++ b/README.zh-CN.md @@ -10,7 +10,7 @@ TCP / HTTP 服务器执行的 JSON 驱动动作。内附 PySide6 GUI，每个功 - 本地文件 / 目录 / ZIP 操作，内置路径穿越防护（`safe_join`） - 经 SSRF 验证的 HTTP 下载，支持重试与大小 / 时间上限 - Google Drive CRUD（上传、下载、搜索、删除、分享、文件夹） -- 一等公民的 S3、Azure Blob、Dropbox、SFTP 后端 — 默认安装 +- S3、Azure Blob、Dropbox、SFTP 以及另外七种远端后端，各自通过对应的 extra 安装（`pip install "automation_file[s3]"`，或通过 `[all]` 一次安装全部） - JSON 动作清单由共享的 `ActionExecutor` 执行 — 支持验证、干跑、并行 - Loopback 优先的 TCP **与** HTTP 服务器，接受 JSON 指令批量并可选 shared-secret 验证 - 可靠性原语：`retry_on_transient` 装饰器、`Quota` 大小 / 时间预算 diff --git a/README.zh-TW.md b/README.zh-TW.md index f5c0129..8f15315 100644 --- a/README.zh-TW.md +++ b/README.zh-TW.md @@ -10,7 +10,7 @@ TCP / HTTP 伺服器執行的 JSON 驅動動作。內附 PySide6 GUI，每個功 - 本機檔案 / 目錄 / ZIP 操作，內建路徑穿越防護（`safe_join`） - 經 SSRF 驗證的 HTTP 下載，支援重試與大小 / 時間上限 - Google Drive CRUD（上傳、下載、搜尋、刪除、分享、資料夾） -- 一等公民的 S3、Azure Blob、Dropbox、SFTP 後端 — 預設安裝 +- S3、Azure Blob、Dropbox、SFTP 以及另外七種遠端後端，各自以對應的 extra 安裝（`pip install "automation_file[s3]"`，或以 `[all]` 一次安裝全部） - JSON 動作清單由共用的 `ActionExecutor` 執行 — 支援驗證、乾跑、平行 - Loopback 優先的 TCP **與** HTTP 伺服器，接受 JSON 指令批次並可選 shared-secret 驗證 - 可靠性原語：`retry_on_transient` 裝飾器、`Quota` 大小 / 時間預算 From 240de1517a3daac8b0ac7a753d3d508d866315b9 Mon Sep 17 00:00:00 2001 From: JeffreyChen Date: Thu, 8 Oct 2026 15:01:00 +0800 Subject: [PATCH 42/59] test: add metadata cases to the storage contract --- README.md | 4 +-- README.zh-CN.md | 4 +-- README.zh-TW.md | 4 +-- docs/source/Eng/usage/integration_tests.rst | 2 +- docs/source/Eng/usage/storage.rst | 8 +++-- docs/source/Zh-CN/usage/integration_tests.rst | 2 +- docs/source/Zh-CN/usage/storage.rst | 6 ++-- docs/source/Zh-TW/usage/integration_tests.rst | 2 +- docs/source/Zh-TW/usage/storage.rst | 6 ++-- docs/updates/2026-10.md | 14 ++++++++ docs/updates/README.md | 3 +- progress.md | 2 +- tests/storage_contract.py | 36 ++++++++++++++++++- 13 files changed, 74 insertions(+), 19 deletions(-) diff --git a/README.md b/README.md index 9a858a7..a59e3bd 100644 --- a/README.md +++ b/README.md @@ -48,7 +48,7 @@ facade. - **HTTP server observability** — `GET /healthz` / `GET /readyz` probes, `GET /openapi.json` spec, and `GET /progress` WebSocket stream of live transfer snapshots - **HTMX Web UI** — `start_web_ui()` serves a read-only dashboard (health, progress, registry) that polls HTML fragments; stdlib-only HTTP plus one CDN script with SRI - **MCP (Model Context Protocol) server** — `MCPServer` bridges the registry to any MCP host (Claude Desktop, MCP CLIs) over newline-delimited JSON-RPC 2.0 on stdio; every `FA_*` action becomes an MCP tool with an auto-generated input schema -- **Universal storage layer** — `File` / `Storage` address local and remote storage with one URI syntax (`local:///…`, `s3://…`, `azure://…`, `gdrive://…`, `sftp://…`, …), one `StorageBackend` contract and one error hierarchy; twelve backends are built in (local, in-memory, S3, Azure Blob, Google Drive, Dropbox, OneDrive, SFTP, FTP / FTPS, WebDAV, SMB, fsspec), and an 81-case contract suite checks any backend +- **Universal storage layer** — `File` / `Storage` address local and remote storage with one URI syntax (`local:///…`, `s3://…`, `azure://…`, `gdrive://…`, `sftp://…`, …), one `StorageBackend` contract and one error hierarchy; twelve backends are built in (local, in-memory, S3, Azure Blob, Google Drive, Dropbox, OneDrive, SFTP, FTP / FTPS, WebDAV, SMB, fsspec), and an 88-case contract suite checks any backend - **Event bus** — one `Event` model with ten core events (`pipeline.*`, `task.*`, `integrity.violation`, `storage.error`, `scheduler.error`, `system.error`), severities, correlation IDs and actors; subscribe on `event_bus` by class, type or prefix - **Notification router** — routes decide which sinks hear about which events (by type, source and minimum severity), with deduplication and rate limiting per route; declare them in code, in `automation_file.toml` or with `FA_notify_route_*` - **Audit trail** — `configure_audit(path)` records one row per event and per storage operation (actor, source, pipeline, task, action, resource, backend, status, duration, correlation ID), searchable with `audit_search` / `FA_audit_search` @@ -519,7 +519,7 @@ File("sandbox://jobs/42/out.csv").write(b"done") name the host the session is connected to, so a typo cannot write to another server. Box has no adapter and stays on its `FA_box_*` actions. Write your own by subclassing `StorageBackend` (`ObjectStorage` for an object store, `SessionStorage` for a login session) and check it with the - 81-case contract suite in `tests/storage_contract.py`. + 88-case contract suite in `tests/storage_contract.py`. - **Actions** — `FA_storage_exists`, `FA_storage_stat`, `FA_storage_list`, `FA_storage_mkdir`, `FA_storage_upload`, `FA_storage_download`, `FA_storage_delete`, `FA_storage_checksum`, diff --git a/README.zh-CN.md b/README.zh-CN.md index ef841e6..719652e 100644 --- a/README.zh-CN.md +++ b/README.zh-CN.md @@ -46,7 +46,7 @@ TCP / HTTP 服务器执行的 JSON 驱动动作。内附 PySide6 GUI，每个功 - **HTTP 服务器观测端点** — `GET /healthz` / `GET /readyz` 探针、`GET /openapi.json` 规格，以及 `GET /progress`（通过 WebSocket 推送实时传输快照） - **HTMX Web UI** — `start_web_ui()` 启动只读观测仪表板（health、progress、registry），通过 HTML 片段轮询；仅用标准库 HTTP，搭配一个带 SRI 的 CDN 脚本 - **MCP（Model Context Protocol）服务器** — `MCPServer` 通过 stdio 上的 JSON-RPC 2.0（换行分隔 JSON）将注册表桥接到任意 MCP 主机（Claude Desktop、MCP CLI）；每个 `FA_*` 动作都会自动生成输入 schema 并成为 MCP 工具 -- **通用存储层** — `File` / `Storage` 以同一套 URI 语法（`local:///…`、`s3://…`、`azure://…`、`gdrive://…`、`sftp://…`、…）、同一份 `StorageBackend` 契约与同一组异常层级访问本地与远端存储；内置十二种后端（本地、内存、S3、Azure Blob、Google Drive、Dropbox、OneDrive、SFTP、FTP / FTPS、WebDAV、SMB、fsspec），并附带 81 个用例的契约测试套件可检查任何后端 +- **通用存储层** — `File` / `Storage` 以同一套 URI 语法（`local:///…`、`s3://…`、`azure://…`、`gdrive://…`、`sftp://…`、…）、同一份 `StorageBackend` 契约与同一组异常层级访问本地与远端存储；内置十二种后端（本地、内存、S3、Azure Blob、Google Drive、Dropbox、OneDrive、SFTP、FTP / FTPS、WebDAV、SMB、fsspec），并附带 88 个用例的契约测试套件可检查任何后端 - **事件总线** — 单一 `Event` 模型与十种核心事件（`pipeline.*`、`task.*`、`integrity.violation`、`storage.error`、`scheduler.error`、`system.error`），具备严重程度、关联 ID 与 actor；可以在 `event_bus` 上按类、type 或前缀订阅 - **通知路由器** — 以路由决定哪些事件（按类型、来源与最低严重程度）发送到哪些 sink，每条路由各自去重与限流；可在代码、`automation_file.toml` 或通过 `FA_notify_route_*` 声明 - **审计轨迹** — `configure_audit(path)` 为每个事件与每次存储操作记录一条（actor、来源、pipeline、task、动作、资源、后端、状态、耗时、关联 ID），可用 `audit_search` / `FA_audit_search` 查询 @@ -513,7 +513,7 @@ File("sandbox://jobs/42/out.csv").write(b"done") extra（`pip install "automation_file[sftp]"`）。`sftp://` 或 `ftp://` URI 必须写出会话实际连接 的主机，打错字就不会写到另一台服务器。Box 没有适配器，仍使用它的 `FA_box_*` 动作。你可以继承 `StorageBackend`（对象存储继承 `ObjectStorage`，登录会话继承 `SessionStorage`）编写自己的后端， - 并用 `tests/storage_contract.py` 中 81 个用例的契约测试套件检查。 + 并用 `tests/storage_contract.py` 中 88 个用例的契约测试套件检查。 - **动作** — `FA_storage_exists`, `FA_storage_stat`, `FA_storage_list`, `FA_storage_mkdir`, `FA_storage_upload`, `FA_storage_download`, `FA_storage_delete`, `FA_storage_checksum`, diff --git a/README.zh-TW.md b/README.zh-TW.md index 8f15315..2b6a39d 100644 --- a/README.zh-TW.md +++ b/README.zh-TW.md @@ -46,7 +46,7 @@ TCP / HTTP 伺服器執行的 JSON 驅動動作。內附 PySide6 GUI，每個功 - **HTTP 伺服器觀測端點** — `GET /healthz` / `GET /readyz` 探針、`GET /openapi.json` 規格、以及 `GET /progress`（以 WebSocket 推送即時傳輸快照） - **HTMX Web UI** — `start_web_ui()` 啟動唯讀觀測儀表板（health、progress、registry），以 HTML 片段輪詢；僅用標準函式庫 HTTP，搭配一支帶 SRI 的 CDN 腳本 - **MCP（Model Context Protocol）伺服器** — `MCPServer` 透過 stdio 上的 JSON-RPC 2.0（行分隔 JSON）將登錄表橋接到任何 MCP 主機（Claude Desktop、MCP CLI）；每個 `FA_*` 動作都會自動生成輸入 schema 並成為 MCP 工具 -- **通用儲存層** — `File` / `Storage` 以同一套 URI 語法（`local:///…`、`s3://…`、`azure://…`、`gdrive://…`、`sftp://…`、…）、同一份 `StorageBackend` 契約與同一組例外階層存取本機與遠端儲存；內建十二種後端（本機、記憶體、S3、Azure Blob、Google Drive、Dropbox、OneDrive、SFTP、FTP / FTPS、WebDAV、SMB、fsspec），並附 81 個案例的契約測試套件可檢查任何後端 +- **通用儲存層** — `File` / `Storage` 以同一套 URI 語法（`local:///…`、`s3://…`、`azure://…`、`gdrive://…`、`sftp://…`、…）、同一份 `StorageBackend` 契約與同一組例外階層存取本機與遠端儲存；內建十二種後端（本機、記憶體、S3、Azure Blob、Google Drive、Dropbox、OneDrive、SFTP、FTP / FTPS、WebDAV、SMB、fsspec），並附 88 個案例的契約測試套件可檢查任何後端 - **事件匯流排** — 單一 `Event` 模型與十種核心事件（`pipeline.*`、`task.*`、`integrity.violation`、`storage.error`、`scheduler.error`、`system.error`），具備嚴重程度、關聯 ID 與 actor；可在 `event_bus` 上依類別、type 或前綴訂閱 - **通知路由器** — 以路由決定哪些事件（依類型、來源與最低嚴重程度）送到哪些 sink，每條路由各自去重與限流；可在程式、`automation_file.toml` 或以 `FA_notify_route_*` 宣告 - **稽核軌跡** — `configure_audit(path)` 為每個事件與每次儲存操作記錄一筆（actor、來源、pipeline、task、動作、資源、後端、狀態、耗時、關聯 ID），可用 `audit_search` / `FA_audit_search` 查詢 @@ -513,7 +513,7 @@ File("sandbox://jobs/42/out.csv").write(b"done") extra（`pip install "automation_file[sftp]"`）。`sftp://` 或 `ftp://` URI 必須寫出工作階段實際連線 的主機，打錯字就不會寫到另一台伺服器。Box 沒有轉接器，仍使用它的 `FA_box_*` 動作。你可以繼承 `StorageBackend`（物件儲存繼承 `ObjectStorage`，登入工作階段繼承 `SessionStorage`）撰寫自己的後端， - 並用 `tests/storage_contract.py` 中 81 個案例的契約測試套件檢查。 + 並用 `tests/storage_contract.py` 中 88 個案例的契約測試套件檢查。 - **動作** — `FA_storage_exists`, `FA_storage_stat`, `FA_storage_list`, `FA_storage_mkdir`, `FA_storage_upload`, `FA_storage_download`, `FA_storage_delete`, `FA_storage_checksum`, diff --git a/docs/source/Eng/usage/integration_tests.rst b/docs/source/Eng/usage/integration_tests.rst index efca2f3..158550c 100644 --- a/docs/source/Eng/usage/integration_tests.rst +++ b/docs/source/Eng/usage/integration_tests.rst @@ -2,7 +2,7 @@ Integration tests ================= The unit tests give every storage backend a stand-in for its service. The -integration tests run the same contract suite (``tests/storage_contract.py``, 81 +integration tests run the same contract suite (``tests/storage_contract.py``, 88 cases) against a real one: MinIO for S3, Azurite for Azure Blob, an OpenSSH server for SFTP, and an FTP, a WebDAV and a Samba server. They live in ``tests/integration/``. diff --git a/docs/source/Eng/usage/storage.rst b/docs/source/Eng/usage/storage.rst index 9e38a03..f1d73c0 100644 --- a/docs/source/Eng/usage/storage.rst +++ b/docs/source/Eng/usage/storage.rst @@ -705,10 +705,12 @@ implement ``_head``, ``_scan``, ``_put``, ``_get`` and ``_remove``. It supplies the directory behaviour described under `Built-in backends`_, and is what ``S3Storage`` and ``AzureStorage`` are built on. -Check it with the contract suite. ``tests/storage_contract.py`` holds 81 cases — +Check it with the contract suite. ``tests/storage_contract.py`` holds 88 cases — nested directories, empty and large files, Unicode paths, binary data, overwrite -and missing-path behaviour, path normalisation, streams, copy and move — and reads -``capabilities`` where backends legitimately differ: +and missing-path behaviour, path normalisation, streams, copy and move, and the +``FileInfo`` fields the backend declares — and reads ``capabilities`` where +backends legitimately differ. A field the backend does not declare must be +absent, so the flags cannot promise more or less than ``stat`` delivers: .. code-block:: python diff --git a/docs/source/Zh-CN/usage/integration_tests.rst b/docs/source/Zh-CN/usage/integration_tests.rst index 989a885..e4fe410 100644 --- a/docs/source/Zh-CN/usage/integration_tests.rst +++ b/docs/source/Zh-CN/usage/integration_tests.rst @@ -2,7 +2,7 @@ ======== 单元测试为每个存储后端准备了服务的替身。集成测试则把同一套契约测试 -（``tests/storage_contract.py``，81 个用例）拿去对真正的服务运行：S3 用 MinIO、 +（``tests/storage_contract.py``，88 个用例）拿去对真正的服务运行：S3 用 MinIO、 Azure Blob 用 Azurite、SFTP 用 OpenSSH 服务器，另外还有 FTP、WebDAV 与 Samba 服务器。 这些测试放在 ``tests/integration/``。 diff --git a/docs/source/Zh-CN/usage/storage.rst b/docs/source/Zh-CN/usage/storage.rst index 7c59dc5..0f3a161 100644 --- a/docs/source/Zh-CN/usage/storage.rst +++ b/docs/source/Zh-CN/usage/storage.rst @@ -644,9 +644,11 @@ scheme 或 authority，并且只接受其下的 URI。 ``_head``、``_scan``、``_put``、``_get`` 与 ``_remove``。它提供 `内置后端`_ 一节 所述的目录行为，``S3Storage`` 与 ``AzureStorage`` 都建立在它之上。 -请用契约测试套件检查。``tests/storage_contract.py`` 包含 81 个用例——嵌套目录、 +请用契约测试套件检查。``tests/storage_contract.py`` 包含 88 个用例——嵌套目录、 空文件与大文件、Unicode 路径、二进制数据、覆盖与路径不存在时的行为、路径规范化、 -流、复制与移动——并在后端确实有差异之处读取 ``capabilities``： +流、复制与移动，以及后端声明会填入的 ``FileInfo`` 字段——并在后端确实有差异之处读取 +``capabilities``。后端没有声明的字段必须不存在，因此这些标志所承诺的不会比 ``stat`` +实际给出的多，也不会少： .. code-block:: python diff --git a/docs/source/Zh-TW/usage/integration_tests.rst b/docs/source/Zh-TW/usage/integration_tests.rst index 82e5363..7a05353 100644 --- a/docs/source/Zh-TW/usage/integration_tests.rst +++ b/docs/source/Zh-TW/usage/integration_tests.rst @@ -2,7 +2,7 @@ ======== 單元測試為每個儲存後端準備了服務的替身。整合測試則把同一套契約測試 -（``tests/storage_contract.py``，81 個案例）拿去對真正的服務執行：S3 用 MinIO、 +（``tests/storage_contract.py``，88 個案例）拿去對真正的服務執行：S3 用 MinIO、 Azure Blob 用 Azurite、SFTP 用 OpenSSH 伺服器，另外還有 FTP、WebDAV 與 Samba 伺服器。 這些測試放在 ``tests/integration/``。 diff --git a/docs/source/Zh-TW/usage/storage.rst b/docs/source/Zh-TW/usage/storage.rst index d9f5336..340af95 100644 --- a/docs/source/Zh-TW/usage/storage.rst +++ b/docs/source/Zh-TW/usage/storage.rst @@ -644,9 +644,11 @@ scheme 或 authority，並且只接受其下的 URI。 ``_head``、``_scan``、``_put``、``_get`` 與 ``_remove``。它提供 `內建後端`_ 一節 所述的目錄行為，``S3Storage`` 與 ``AzureStorage`` 都建立在它之上。 -請用契約測試套件檢查。``tests/storage_contract.py`` 包含 81 個案例——巢狀目錄、 +請用契約測試套件檢查。``tests/storage_contract.py`` 包含 88 個案例——巢狀目錄、 空檔與大檔、Unicode 路徑、二進位資料、覆寫與路徑不存在時的行為、路徑正規化、 -串流、複製與搬移——並在後端確實有差異之處讀取 ``capabilities``： +串流、複製與搬移，以及後端宣告會填入的 ``FileInfo`` 欄位——並在後端確實有差異之處讀取 +``capabilities``。後端沒有宣告的欄位必須不存在，因此這些旗標所承諾的不會比 ``stat`` +實際給出的多，也不會少： .. code-block:: python diff --git a/docs/updates/2026-10.md b/docs/updates/2026-10.md index 96bdce3..2cac00b 100644 --- a/docs/updates/2026-10.md +++ b/docs/updates/2026-10.md @@ -558,3 +558,17 @@ Index and query commands: [README.md](README.md). New entries go at the end. - **Docs**: `CLAUDE.md` § Branching & CI (how a patch and how a MINOR or MAJOR release is made). - **Files**: `scripts/stable_release.py`, `tests/test_stable_release.py`, `.github/workflows/publish.yml`, `.github/workflows/ci-dev.yml`, `.github/workflows/ci-stable.yml`, `CLAUDE.md`, `progress.md`. - **Open items**: #26 (migration guide, documentation audit, the 1.0.0 release), #34. + +## U-20261008-25 · 2026-10-08 · Metadata cases in the storage contract · #storage #tests #done + +- **What**: seven cases added to `tests/storage_contract.py`, which now has 88. Closes `progress.md` #20. + - An etag, where the backend declares one, is present and changes when the content changes. + - A copy within a backend keeps the content type, where the backend declares content types. + - `FileInfo.metadata` is a mapping of strings, and empty for a backend that does not declare metadata. + - A field the backend does not declare (`modified_at`, `etag`, `version`, `content_type`) is `None`, so a capability flag cannot say less than `stat` delivers. The existing cases already held the flags to not saying more. +- **Result**: every contract class passes the new cases unchanged: 115 passed and 108 skipped by capability across the local, memory, S3, Azure, SFTP, FTP, Drive, OneDrive, Dropbox, WebDAV, SMB and fsspec classes. No adapter and no stand-in needed a change, so the flags were already honest. +- **What it does not cover**: user metadata written through the layer. `upload` and `write_bytes` take no metadata and no content type, so a backend's metadata can be read (`stat().metadata`) but not set through `File` / `Storage`; that is a feature the layer does not have, recorded as `progress.md` #36. +- **Result / numbers**: 4998 passed, 257 skipped, 0 failed with every extra; 3217 passed, 135 skipped with the base dependencies only. `ruff check` and `ruff format --check` pass. Python 3.14.7 on Windows. +- **Docs**: the case count and the description of the suite in the three `usage/storage.rst` pages, the count in the three READMEs and the three `usage/integration_tests.rst` pages. +- **Files**: `tests/storage_contract.py`, the documentation above, `progress.md`. +- **Open items**: #36. diff --git a/docs/updates/README.md b/docs/updates/README.md index eaa10e8..b41793b 100644 --- a/docs/updates/README.md +++ b/docs/updates/README.md @@ -58,6 +58,7 @@ In the same commit: delete the item from `progress.md`, add a `#done` entry here | ID | Date | Title | Tags | Batch | |---|---|---|---|---| +| U-20261008-25 | 2026-10-08 | Metadata cases in the storage contract | #storage #tests #done | [2026-10](2026-10.md) | | U-20261008-24 | 2026-10-08 | A release can raise MINOR or MAJOR | #release #ci #roadmap | [2026-10](2026-10.md) | | U-20261008-23 | 2026-10-08 | Integration tests and their workflow | #ci #tests #roadmap | [2026-10](2026-10.md) | | U-20261008-22 | 2026-10-08 | Public API and deprecation policy | #decision #docs #roadmap #done | [2026-10](2026-10.md) | @@ -119,5 +120,5 @@ In the same commit: delete the item from `progress.md`, add a `#done` entry here | File | Period | Entries | |---|---|---:| -| [2026-10.md](2026-10.md) | 2026-10 | 35 | +| [2026-10.md](2026-10.md) | 2026-10 | 36 | | [2026-09.md](2026-09.md) | 2026-09 | 21 | diff --git a/progress.md b/progress.md index 6c58aad..3df88c3 100644 --- a/progress.md +++ b/progress.md @@ -21,7 +21,7 @@ Items #10 to #26 are what is left of the 1.0 roadmap (`docs/FILEAUTOMATION-1.0-R ### Backend integration tests (roadmap M3) - **#19** [UNVERIFIED] `.github/workflows/integration.yml` and `tests/integration/` (U-20261008-23) have never run: there is no Docker on the development machine. The first run of the workflow decides whether each of the six service jobs (`s3`, `azure`, `sftp`, `ftp`, `webdav`, `smb`) and the two platform jobs (Linux, macOS) work. Expect to adjust the container options in `tests/integration/start_service.sh` (image tags are floating; pin them to digests once a run is green; the Samba share options and the FTP passive ports are the least certain) and to fix what a real service or another platform shows (#32, #18). Still missing: credential-gated jobs for Google Drive, OneDrive and Dropbox, which have no emulator, and FTPS. -- **#20** Metadata cases in the contract suite: user metadata and content type kept across an upload and a copy where `capabilities` says the backend supports them. The failure cases exist (U-20261008-10): a backend's contract class gets them by providing the `break_storage` fixture, as the local, S3 and Azure classes do. +- **#36** Writing metadata through the storage layer. `upload`, `write_bytes` and `open_write` take no user metadata and no content type, so `stat().metadata` can be read but a caller cannot set it, and the content type is always the one guessed from the name. The contract checks what is read (U-20261008-25). Adding it means a keyword on the write operations, a capability-conditional contract case, and deciding what a backend without metadata does with the argument (refuse, or ignore). ### Later milestones diff --git a/tests/storage_contract.py b/tests/storage_contract.py index ad3c26a..80be73e 100644 --- a/tests/storage_contract.py +++ b/tests/storage_contract.py @@ -23,7 +23,7 @@ def backend(self) -> StorageBackend: from __future__ import annotations import hashlib -from collections.abc import Callable +from collections.abc import Callable, Mapping from datetime import timedelta from pathlib import Path @@ -109,6 +109,40 @@ def test_stat_reports_a_content_type_where_supported(self, backend: StorageBacke pytest.skip("backend does not report content types") assert backend.write_bytes("notes.txt", b"x").content_type == "text/plain" + def test_stat_reports_an_etag_that_follows_the_content(self, backend: StorageBackend) -> None: + if not backend.capabilities.etag: + pytest.skip("backend does not report etags") + first = backend.write_bytes("a.txt", b"one").etag + second = backend.write_bytes("a.txt", b"two, and longer").etag + assert first + assert second + assert first != second + + def test_a_copy_keeps_the_content_type(self, backend: StorageBackend) -> None: + if not backend.capabilities.content_type: + pytest.skip("backend does not report content types") + original = backend.write_bytes("report.json", b"{}") + copied = backend.copy_from(backend, "report.json", "copies/report.json") + assert original.content_type == "application/json" + assert copied.content_type == original.content_type + + def test_metadata_is_a_mapping_of_strings(self, backend: StorageBackend) -> None: + metadata = backend.write_bytes("a.txt", b"x").metadata + assert isinstance(metadata, Mapping) + assert all( + isinstance(key, str) and isinstance(value, str) for key, value in metadata.items() + ) + if not backend.capabilities.metadata: + assert dict(metadata) == {} + + @pytest.mark.parametrize("field", ["modified_at", "etag", "version", "content_type"]) + def test_a_field_the_backend_does_not_declare_is_absent( + self, backend: StorageBackend, field: str + ) -> None: + if getattr(backend.capabilities, field): + pytest.skip(f"backend declares {field}") + assert getattr(backend.write_bytes("report.json", b"{}"), field) is None + def test_file_info_is_json_friendly(self, backend: StorageBackend) -> None: document = backend.write_bytes("a.txt", b"x").to_dict() assert document["path"] == "a.txt" From 089c7d7897886ce9babe80be8069262d14c1da81 Mon Sep 17 00:00:00 2001 From: JeffreyChen Date: Thu, 8 Oct 2026 15:05:01 +0800 Subject: [PATCH 43/59] docs: add the production deployment guide --- README.md | 10 + README.zh-CN.md | 8 + README.zh-TW.md | 8 + docs/source/Eng/eng_index.rst | 14 ++ docs/source/Eng/usage/deployment.rst | 260 +++++++++++++++++++++++++ docs/source/Zh-CN/usage/deployment.rst | 242 +++++++++++++++++++++++ docs/source/Zh-CN/zh_cn_index.rst | 14 ++ docs/source/Zh-TW/usage/deployment.rst | 243 +++++++++++++++++++++++ docs/source/Zh-TW/zh_tw_index.rst | 14 ++ docs/updates/2026-10.md | 15 ++ docs/updates/README.md | 3 +- 11 files changed, 830 insertions(+), 1 deletion(-) create mode 100644 docs/source/Eng/usage/deployment.rst create mode 100644 docs/source/Zh-CN/usage/deployment.rst create mode 100644 docs/source/Zh-TW/usage/deployment.rst diff --git a/README.md b/README.md index a59e3bd..d97a578 100644 --- a/README.md +++ b/README.md @@ -1240,6 +1240,16 @@ Each entry is either a bare command name, a `[name, kwargs]` pair, or a ] ``` +## Deployment + +The scheduler, the integrity monitors, the notification router, the audit trail and the servers are +threads of the process that starts them, so a production deployment is one script under your service +manager: load the configuration, point the audit trail and the pipeline run store at SQLite files, +initialise the backends, start what should run, and keep every server on the loopback interface behind +a shared secret and an action allow list. The manual chapter *Deploying to production* +(`docs/source/Eng/usage/deployment.rst`) has the script, a systemd unit, what to back up, what to +watch and how to upgrade. + ## Tests ```bash diff --git a/README.zh-CN.md b/README.zh-CN.md index 719652e..2f96297 100644 --- a/README.zh-CN.md +++ b/README.zh-CN.md @@ -1205,6 +1205,14 @@ python -m automation_file --create_project ./my_project ] ``` +## 部署 + +调度器、完整性监控、通知路由器、审计轨迹与各个服务器，都是启动它们的那个进程中的线程，因此 +生产环境的部署就是一个交给服务管理器运行的脚本：加载配置、把审计轨迹与流水线运行记录指向 SQLite +文件、初始化后端、启动该运行的部分，并让每个服务器都只绑定 loopback 接口、设有共享密钥与动作 +允许列表。手册的“部署到生产环境”一章（`docs/source/Zh-CN/usage/deployment.rst`）提供了这个 +脚本、systemd unit、该备份什么、该监控什么，以及如何升级。 + ## 测试 ```bash diff --git a/README.zh-TW.md b/README.zh-TW.md index 2b6a39d..771b078 100644 --- a/README.zh-TW.md +++ b/README.zh-TW.md @@ -1205,6 +1205,14 @@ python -m automation_file --create_project ./my_project ] ``` +## 部署 + +排程器、完整性監控、通知路由器、稽核軌跡與各個伺服器，都是啟動它們的那個行程中的執行緒，因此 +正式環境的部署就是一支交給服務管理員執行的腳本：載入設定、把稽核軌跡與管線執行紀錄指向 SQLite +檔案、初始化後端、啟動該執行的部分，並讓每個伺服器都只綁定 loopback 介面、設有共享密鑰與動作 +允許清單。手冊的「部署到正式環境」一章（`docs/source/Zh-TW/usage/deployment.rst`）提供了這支 +腳本、systemd unit、該備份什麼、該監看什麼，以及如何升級。 + ## 測試 ```bash diff --git a/docs/source/Eng/eng_index.rst b/docs/source/Eng/eng_index.rst index 66b756b..cd02f89 100644 --- a/docs/source/Eng/eng_index.rst +++ b/docs/source/Eng/eng_index.rst @@ -347,3 +347,17 @@ locally, the variables each module reads, and what CI runs. :caption: Integration Tests usage/integration_tests + +.. _eng-deployment: + +Chapter 23 — Deploying to Production +==================================== + +Running unattended: what to install, one long-lived process, the state on +disk, what to expose on the network, what to watch, and how to upgrade. + +.. toctree:: + :maxdepth: 2 + :caption: Deploying to Production + + usage/deployment diff --git a/docs/source/Eng/usage/deployment.rst b/docs/source/Eng/usage/deployment.rst new file mode 100644 index 0000000..e6500bb --- /dev/null +++ b/docs/source/Eng/usage/deployment.rst @@ -0,0 +1,260 @@ +Deploying to production +======================= + +This page is about running FileAutomation unattended: what to install, where its +state lives, how to start it as one long-lived process, what to expose on the +network and what to watch. Every part of it is ordinary Python; there is no +separate server product to operate. + +Install +------- + +Pin the release and name the extras you use. The base package carries no cloud +SDK and no GUI toolkit: + +.. code-block:: bash + + python -m venv /opt/fileautomation/venv + /opt/fileautomation/venv/bin/pip install "automation_file[s3,sftp]==1.0.0" + +Check what the installation can reach before anything depends on it:: + + /opt/fileautomation/venv/bin/python -m automation_file storage schemes + +A backend whose extra is missing fails at the first call with the command that +installs it (``pip install "automation_file[azure]"``), never at import. + +Run it under an account of its own. That account's rights on the filesystem and +the credentials you give it are the boundary of what an action can do. + +Configuration and secrets +------------------------- + +Keep settings in ``automation_file.toml`` and keep secrets out of it: + +.. code-block:: toml + + [secrets] + file_root = "/run/secrets" + + [[notify.sinks]] + type = "slack" + name = "team-alerts" + webhook_url = "${env:SLACK_WEBHOOK}" + + [[notify.routes]] + name = "failures" + sinks = ["team-alerts"] + types = ["pipeline.failed", "task.failed", "integrity.violation", "scheduler.error"] + min_severity = "error" + dedup_seconds = 600 + +``${env:NAME}`` and ``${file:name}`` are resolved when the file is loaded, and a +reference that cannot be resolved raises instead of becoming an empty string +(:doc:`config`). Credentials for the backends come from the environment or from +files the account can read, passed to each client's ``later_init``. Do not put a +secret in an action list, a pipeline definition or a command line: all three are +logged, stored or visible to other users of the machine. + +One process +----------- + +The scheduler, the integrity monitors, the notification router, the audit trail +and the servers are threads of the process that started them. A production +deployment is therefore one script that starts what it needs and then waits: + +.. code-block:: python + + # /opt/fileautomation/service.py + import os + import signal + import threading + + from automation_file import ( + ActionACL, AutomationConfig, IntegrityMonitor, SQLiteRunStore, configure_audit, + install_operational_metrics, notification_manager, notification_router, + s3_instance, start_http_action_server, start_metrics_server, + ) + from automation_file.pipeline import set_default_run_store + + STATE = "/var/lib/fileautomation" + + # 1. Settings, sinks and notification routes. + AutomationConfig.load("/etc/fileautomation/automation_file.toml").apply_to( + notification_manager, notification_router + ) + + # 2. State that must survive a restart. + configure_audit(f"{STATE}/audit.sqlite") + set_default_run_store(SQLiteRunStore(f"{STATE}/runs.sqlite")) + + # 3. Backends. + s3_instance.later_init(region_name=os.environ["AWS_REGION"]) + + # 4. What runs by itself. + monitor = IntegrityMonitor("s3://reports/2026", baseline=f"{STATE}/reports.baseline.json") + monitor.start() + + # 5. What listens: loopback only, a secret, and only the actions a client needs. + install_operational_metrics() + start_metrics_server(port=9945) + start_http_action_server( + port=9944, + shared_secret=os.environ["FA_SHARED_SECRET"], + action_acl=ActionACL.build(allowed=["FA_pipeline_run", "FA_storage_copy", "FA_storage_list"]), + ) + + # 6. Stay alive until asked to stop. + stop = threading.Event() + signal.signal(signal.SIGTERM, lambda *_: stop.set()) + signal.signal(signal.SIGINT, lambda *_: stop.set()) + stop.wait() + monitor.stop() + +Run it under your service manager. With systemd: + +.. code-block:: ini + + [Unit] + Description=FileAutomation + After=network-online.target + + [Service] + User=fileautomation + EnvironmentFile=/etc/fileautomation/environment + Environment=FILE_AUTOMATION_LOG_FILE=/var/log/fileautomation/FileAutomation.log + ExecStart=/opt/fileautomation/venv/bin/python /opt/fileautomation/service.py + Restart=on-failure + StateDirectory=fileautomation + LogsDirectory=fileautomation + + [Install] + WantedBy=multi-user.target + +On Windows the same script runs as a service through a wrapper such as NSSM, or +from Task Scheduler with "run whether the user is logged on or not". + +A job that only has to run once (a nightly pipeline from the system's own cron, +a verification in a CI step) does not need the service: the command line does it +and exits with a code you can act on (:doc:`cli`):: + + python -m automation_file pipeline --audit /var/lib/fileautomation/audit.sqlite \ + run /etc/fileautomation/daily.yaml --store /var/lib/fileautomation/runs.sqlite + +State on disk +------------- + +.. list-table:: + :header-rows: 1 + :widths: 26 74 + + * - What + - Where and how to treat it + * - Audit trail + - The SQLite file you gave ``configure_audit``. It is in WAL mode, so copy it + with ``sqlite3 audit.sqlite ".backup …"`` and not with ``cp`` while the + process runs. Keep it as long as your policy says, then + ``python -m automation_file audit purge --db … --older-than-days 365``. + * - Pipeline runs + - The SQLite file of the ``SQLiteRunStore``. It is what ``resume`` reads after + a crash; without it a run is forgotten when the process ends. + * - Integrity baselines + - One JSON file per monitored tree. Keep it outside the tree it describes + and, better, where the account that changes the tree cannot write: whoever + can rewrite the baseline can hide a change. + * - OAuth tokens + - The ``token_path`` you gave the Google Drive client. Readable by the + service account only. + * - Log + - ``~/.automation_file/logs/FileAutomation.log`` unless + ``FILE_AUTOMATION_LOG_FILE`` names another path. A file past 10 MB is moved + to ``.1`` when a process opens it; rotate it yourself if you need more. + * - Version snapshots, trash, the content store + - The directories you gave those features. They grow until you prune them. + +Network exposure +---------------- + +Every server binds the loopback interface unless you pass +``allow_non_loopback=True``, and that default is the recommendation: the action +servers run whatever is registered, so reaching one is equivalent to reaching a +Python prompt of the service account. + +* Clients on the same machine use the loopback address and the shared secret. +* A client elsewhere goes through something that terminates TLS and + authenticates it (a reverse proxy, an SSH tunnel, a service mesh). The servers + speak plain HTTP and plain TCP. +* Give each server an ``ActionACL`` with an allow list. The ACL also checks the + actions nested in the arguments of another action. It cannot see inside a file + an action is told to run, so do not allow ``FA_execute_files`` or a pipeline + given as a path to a client that must stay inside the list. +* The MCP server speaks over the standard streams of the process that starts it. + It needs no port; its allow list is :doc:`mcp`. + +Outbound requests to URLs supplied by a caller go through the SSRF guard: only +``http`` and ``https``, and no private, loopback or link-local address. That +guard is what makes it safe to accept a URL from a client; do not route around it. + +What to watch +------------- + +.. list-table:: + :header-rows: 1 + :widths: 26 74 + + * - Signal + - Where + * - Health + - ``GET /healthz`` and ``GET /readyz`` on the HTTP action server. + * - Metrics + - ``start_metrics_server()`` serves Prometheus text. ``automation_file_actions_total`` + and its duration histogram are always there; ``install_operational_metrics()`` + adds counters for events, notifications and storage operations. + * - Failures + - Events: ``pipeline.failed``, ``task.failed``, ``integrity.violation``, + ``storage.error``, ``scheduler.error``, ``system.error``. Route the ones + you want to be told about to a sink (:doc:`notifications`). + * - History + - The audit trail: ``python -m automation_file audit search --db … --status + error``, or ``--correlation-id `` for everything one run did. + * - Log + - INFO and above also go to standard error, which a service manager + captures. + +Surviving failure +----------------- + +* Give a pipeline task a ``RetryPolicy`` for the errors that pass (a dropped + connection, throttling) and a ``timeout`` for the ones that never return. +* Give a task that must not happen twice an ``idempotency_key``, and keep runs in + a ``SQLiteRunStore``: after a restart ``pipeline resume `` repeats only + what did not succeed. +* A scheduled job does not start while its previous run is still going, unless + you allowed the overlap. +* Remediation by the integrity monitor is off unless you configured it. Start + with alerts, and add quarantine or restore when you trust the baseline. + +Upgrading +--------- + +1. Read the release notes. A patch release only fixes; a minor release may + deprecate; only a major release removes (:doc:`api_policy`). +2. Run your own tests with ``-W error::DeprecationWarning`` against the new + version before it reaches production. +3. Back up the two SQLite files. A release that changes a stored format reads + the previous one and says how to convert. +4. Install into a fresh virtual environment next to the old one and switch the + service to it, so rolling back is switching again. + +Checklist +--------- + +* The service runs under an account of its own, and only that account reads the + secrets and the token files. +* The version and the extras are pinned. +* The audit trail and the run store are files on a disk that is backed up. +* Every server is on the loopback interface, has a shared secret and has an + allow list; anything remote goes through TLS. +* Failures reach a person: at least one notification route for errors. +* Baselines live where the monitored tree's writers cannot change them. +* Logs and the growing directories have a retention you chose. diff --git a/docs/source/Zh-CN/usage/deployment.rst b/docs/source/Zh-CN/usage/deployment.rst new file mode 100644 index 0000000..08b2ae2 --- /dev/null +++ b/docs/source/Zh-CN/usage/deployment.rst @@ -0,0 +1,242 @@ +部署到生产环境 +============== + +本页说明如何让 FileAutomation 在无人值守的情况下运行：要安装什么、它的状态存放在哪里、 +如何把它作为一个长时间运行的进程启动、在网络上要开放什么，以及该监控什么。其中每个 +部分都是普通的 Python；没有另外需要运维的服务器产品。 + +安装 +---- + +请固定版本，并写明你用到的 extra。基础包不含任何云端 SDK 与 GUI 工具包： + +.. code-block:: bash + + python -m venv /opt/fileautomation/venv + /opt/fileautomation/venv/bin/pip install "automation_file[s3,sftp]==1.0.0" + +在任何东西依赖它之前，先确认这份安装能连到哪些存储:: + + /opt/fileautomation/venv/bin/python -m automation_file storage schemes + +缺少 extra 的后端会在第一次调用时失败，并指出安装它的命令 +（``pip install "automation_file[azure]"``），而不是在导入时失败。 + +请用专属的账号运行。那个账号在文件系统上的权限，加上你交给它的凭据，就是一个动作 +所能做到的范围。 + +配置与机密 +---------- + +把配置放在 ``automation_file.toml``，并把机密留在文件之外： + +.. code-block:: toml + + [secrets] + file_root = "/run/secrets" + + [[notify.sinks]] + type = "slack" + name = "team-alerts" + webhook_url = "${env:SLACK_WEBHOOK}" + + [[notify.routes]] + name = "failures" + sinks = ["team-alerts"] + types = ["pipeline.failed", "task.failed", "integrity.violation", "scheduler.error"] + min_severity = "error" + dedup_seconds = 600 + +``${env:NAME}`` 与 ``${file:name}`` 会在加载文件时解析，无法解析的引用会抛出异常， +而不是变成空字符串（:doc:`config`）。各后端的凭据来自环境变量，或来自该账号读得到的 +文件，再传给各客户端的 ``later_init``。不要把机密写进动作列表、流水线定义或命令行： +这三者都会被记录、被存储，或被同一台机器上的其他用户看到。 + +单一进程 +-------- + +调度器、完整性监控、通知路由器、审计轨迹与各个服务器，都是启动它们的那个进程中的 +线程。因此生产环境的部署就是一个脚本：启动需要的东西，然后等待： + +.. code-block:: python + + # /opt/fileautomation/service.py + import os + import signal + import threading + + from automation_file import ( + ActionACL, AutomationConfig, IntegrityMonitor, SQLiteRunStore, configure_audit, + install_operational_metrics, notification_manager, notification_router, + s3_instance, start_http_action_server, start_metrics_server, + ) + from automation_file.pipeline import set_default_run_store + + STATE = "/var/lib/fileautomation" + + # 1. 配置、sink 与通知路由。 + AutomationConfig.load("/etc/fileautomation/automation_file.toml").apply_to( + notification_manager, notification_router + ) + + # 2. 重启之后必须还在的状态。 + configure_audit(f"{STATE}/audit.sqlite") + set_default_run_store(SQLiteRunStore(f"{STATE}/runs.sqlite")) + + # 3. 后端。 + s3_instance.later_init(region_name=os.environ["AWS_REGION"]) + + # 4. 会自行运行的部分。 + monitor = IntegrityMonitor("s3://reports/2026", baseline=f"{STATE}/reports.baseline.json") + monitor.start() + + # 5. 对外监听的部分：只绑定 loopback、需要密钥，而且只开放客户端需要的动作。 + install_operational_metrics() + start_metrics_server(port=9945) + start_http_action_server( + port=9944, + shared_secret=os.environ["FA_SHARED_SECRET"], + action_acl=ActionACL.build(allowed=["FA_pipeline_run", "FA_storage_copy", "FA_storage_list"]), + ) + + # 6. 持续存活，直到被要求停止。 + stop = threading.Event() + signal.signal(signal.SIGTERM, lambda *_: stop.set()) + signal.signal(signal.SIGINT, lambda *_: stop.set()) + stop.wait() + monitor.stop() + +请用你的服务管理器来运行它。以 systemd 为例： + +.. code-block:: ini + + [Unit] + Description=FileAutomation + After=network-online.target + + [Service] + User=fileautomation + EnvironmentFile=/etc/fileautomation/environment + Environment=FILE_AUTOMATION_LOG_FILE=/var/log/fileautomation/FileAutomation.log + ExecStart=/opt/fileautomation/venv/bin/python /opt/fileautomation/service.py + Restart=on-failure + StateDirectory=fileautomation + LogsDirectory=fileautomation + + [Install] + WantedBy=multi-user.target + +在 Windows 上，同一个脚本可以通过 NSSM 之类的包装程序作为服务运行，或由任务计划程序以 +“不论用户是否登录都运行”的方式启动。 + +只需要运行一次的作业（由系统本身的 cron 启动的夜间流水线、CI 步骤中的一次验证）不需要 +这个服务：命令行就能完成，并以你可以据此处理的退出码结束（:doc:`cli`）:: + + python -m automation_file pipeline --audit /var/lib/fileautomation/audit.sqlite \ + run /etc/fileautomation/daily.yaml --store /var/lib/fileautomation/runs.sqlite + +磁盘上的状态 +------------ + +.. list-table:: + :header-rows: 1 + :widths: 26 74 + + * - 项目 + - 位置与处理方式 + * - 审计轨迹 + - 你交给 ``configure_audit`` 的 SQLite 文件。它采用 WAL 模式，进程运行期间请用 + ``sqlite3 audit.sqlite ".backup …"`` 复制，不要用 ``cp``。按你的策略保留足够久， + 再运行 ``python -m automation_file audit purge --db … --older-than-days 365``。 + * - 流水线运行记录 + - ``SQLiteRunStore`` 的 SQLite 文件。崩溃之后 ``resume`` 读的就是它；没有它，进程 + 结束时运行记录就被遗忘了。 + * - 完整性基准 + - 每棵受监控的目录树一个 JSON 文件。请把它放在它所描述的目录树之外，最好放在能 + 修改该目录树的账号写不到的地方：谁能改写基准，谁就能掩盖变更。 + * - OAuth 令牌 + - 你交给 Google Drive 客户端的 ``token_path``。只允许服务账号读取。 + * - 日志 + - ``~/.automation_file/logs/FileAutomation.log``，除非 ``FILE_AUTOMATION_LOG_FILE`` + 指定了其他路径。超过 10 MB 的文件会在进程打开它时被移到 ``.1``；需要更多轮转 + 时请自行处理。 + * - 版本快照、回收站、内容存储库 + - 你交给这些功能的目录。在你清理之前，它们会持续增长。 + +网络暴露面 +---------- + +每个服务器都只绑定 loopback 接口，除非你传入 ``allow_non_loopback=True``；这个默认值 +也就是推荐做法：动作服务器会执行任何已注册的东西，所以连得到它，就等于连得到服务 +账号的 Python 提示符。 + +* 同一台机器上的客户端使用 loopback 地址与共享密钥。 +* 其他地方的客户端要经过某个负责终结 TLS 并验证身份的组件（反向代理、SSH 隧道、 + service mesh）。服务器本身使用的是明文 HTTP 与明文 TCP。 +* 为每个服务器设置带有允许列表的 ``ActionACL``。ACL 也会检查嵌套在另一个动作参数中 + 的动作。它看不到某个动作被指示去执行的文件内容，所以对必须留在列表之内的客户端， + 不要开放 ``FA_execute_files``，也不要开放以路径指定的流水线。 +* MCP 服务器通过启动它的进程的标准流通信，不需要任何端口；它的允许列表请见 + :doc:`mcp`。 + +对外连往调用方提供的 URL 的请求，会经过 SSRF 防护：只允许 ``http`` 与 ``https``， +而且不允许私有、loopback 或 link-local 地址。有了这道防护，才能安全地接受客户端提供 +的 URL；请不要绕过它。 + +该监控什么 +---------- + +.. list-table:: + :header-rows: 1 + :widths: 26 74 + + * - 信号 + - 位置 + * - 健康状态 + - HTTP 动作服务器的 ``GET /healthz`` 与 ``GET /readyz``。 + * - 指标 + - ``start_metrics_server()`` 提供 Prometheus 文本格式。``automation_file_actions_total`` + 及其耗时直方图一直都有；``install_operational_metrics()`` 会加上事件、通知与 + 存储操作的计数器。 + * - 失败 + - 事件：``pipeline.failed``、``task.failed``、``integrity.violation``、 + ``storage.error``、``scheduler.error``、``system.error``。把你想被告知的事件 + 路由到某个 sink（:doc:`notifications`）。 + * - 历史 + - 审计轨迹：``python -m automation_file audit search --db … --status error``，或用 + ``--correlation-id `` 查看一次运行所做的一切。 + * - 日志 + - INFO 以上的消息也会写到标准错误输出，由服务管理器收集。 + +承受失败 +-------- + +* 为流水线任务设置 ``RetryPolicy`` 来应对会自行消失的错误（连接中断、被限流），并设置 + ``timeout`` 来应对永远不返回的情况。 +* 为不可以发生两次的任务设置 ``idempotency_key``，并把运行记录保存在 + ``SQLiteRunStore``：重启之后，``pipeline resume `` 只会重做没有成功的部分。 +* 调度作业在前一次运行尚未结束时不会启动，除非你允许重叠。 +* 完整性监控的修复功能默认是关闭的，除非你配置了它。请先从告警开始，等你信任基准 + 之后，再加上隔离或还原。 + +升级 +---- + +1. 阅读版本说明。修订版只做修复；次版本可能会把东西标为弃用；只有主版本才会移除 + （:doc:`api_policy`）。 +2. 在新版本进入生产环境之前，先用 ``-W error::DeprecationWarning`` 对它运行你自己的 + 测试。 +3. 备份那两个 SQLite 文件。会改变存储格式的版本仍然读得懂前一种格式，并说明如何转换。 +4. 安装到旧环境旁边的一个全新虚拟环境，再把服务切换过去；这样要回滚时，再切换一次 + 即可。 + +检查清单 +-------- + +* 服务以专属账号运行，而且只有那个账号能读取机密与令牌文件。 +* 版本与 extra 都已固定。 +* 审计轨迹与运行记录存储库都是位于有备份的磁盘上的文件。 +* 每个服务器都绑定在 loopback 接口、设有共享密钥与允许列表；任何远程连接都经过 TLS。 +* 失败会传达给人：至少有一条针对错误的通知路由。 +* 基准存放在受监控目录树的写入者无法修改的位置。 +* 日志与持续增长的目录都有你自己决定的保留期限。 diff --git a/docs/source/Zh-CN/zh_cn_index.rst b/docs/source/Zh-CN/zh_cn_index.rst index dc8320e..e29455a 100644 --- a/docs/source/Zh-CN/zh_cn_index.rst +++ b/docs/source/Zh-CN/zh_cn_index.rst @@ -336,3 +336,17 @@ Slack、Email（SMTP）、Discord、Telegram、Microsoft Teams、PagerDuty :caption: 集成测试 usage/integration_tests + +.. _zh-cn-deployment: + +第 23 章 — 部署到生产环境 +========================= + +无人值守地运行：要安装什么、单一的长时间进程、磁盘上的状态、网络上要开放什么、 +该监控什么，以及如何升级。 + +.. toctree:: + :maxdepth: 2 + :caption: 部署到生产环境 + + usage/deployment diff --git a/docs/source/Zh-TW/usage/deployment.rst b/docs/source/Zh-TW/usage/deployment.rst new file mode 100644 index 0000000..4bd7e0a --- /dev/null +++ b/docs/source/Zh-TW/usage/deployment.rst @@ -0,0 +1,243 @@ +部署到正式環境 +============== + +本頁說明如何讓 FileAutomation 在無人看管的情況下運作：要安裝什麼、它的狀態存放在哪裡、 +如何把它當成一個長時間執行的行程啟動、在網路上要開放什麼，以及該監看什麼。其中每個 +部分都是一般的 Python；沒有另外需要維運的伺服器產品。 + +安裝 +---- + +請固定版本，並寫明你用到的 extra。基本套件不含任何雲端 SDK 與 GUI 工具組： + +.. code-block:: bash + + python -m venv /opt/fileautomation/venv + /opt/fileautomation/venv/bin/pip install "automation_file[s3,sftp]==1.0.0" + +在任何東西依賴它之前，先確認這份安裝能連到哪些儲存:: + + /opt/fileautomation/venv/bin/python -m automation_file storage schemes + +缺少 extra 的後端會在第一次呼叫時失敗，並指出安裝它的指令 +（``pip install "automation_file[azure]"``），而不是在匯入時失敗。 + +請用專屬的帳號執行。那個帳號在檔案系統上的權限，加上你交給它的憑證，就是一個動作 +所能做到的範圍。 + +設定與機密 +---------- + +把設定放在 ``automation_file.toml``，並把機密留在檔案之外： + +.. code-block:: toml + + [secrets] + file_root = "/run/secrets" + + [[notify.sinks]] + type = "slack" + name = "team-alerts" + webhook_url = "${env:SLACK_WEBHOOK}" + + [[notify.routes]] + name = "failures" + sinks = ["team-alerts"] + types = ["pipeline.failed", "task.failed", "integrity.violation", "scheduler.error"] + min_severity = "error" + dedup_seconds = 600 + +``${env:NAME}`` 與 ``${file:name}`` 會在載入檔案時解析，無法解析的參考會拋出例外， +而不是變成空字串（:doc:`config`）。各後端的憑證來自環境變數，或來自該帳號讀得到的 +檔案，再傳給各用戶端的 ``later_init``。不要把機密寫進動作清單、管線定義或命令列： +這三者都會被記錄、被儲存，或被同一台機器上的其他使用者看到。 + +單一行程 +-------- + +排程器、完整性監控、通知路由器、稽核軌跡與各個伺服器，都是啟動它們的那個行程中的 +執行緒。因此正式環境的部署就是一支腳本：啟動需要的東西，然後等待： + +.. code-block:: python + + # /opt/fileautomation/service.py + import os + import signal + import threading + + from automation_file import ( + ActionACL, AutomationConfig, IntegrityMonitor, SQLiteRunStore, configure_audit, + install_operational_metrics, notification_manager, notification_router, + s3_instance, start_http_action_server, start_metrics_server, + ) + from automation_file.pipeline import set_default_run_store + + STATE = "/var/lib/fileautomation" + + # 1. 設定、sink 與通知路由。 + AutomationConfig.load("/etc/fileautomation/automation_file.toml").apply_to( + notification_manager, notification_router + ) + + # 2. 重新啟動後必須還在的狀態。 + configure_audit(f"{STATE}/audit.sqlite") + set_default_run_store(SQLiteRunStore(f"{STATE}/runs.sqlite")) + + # 3. 後端。 + s3_instance.later_init(region_name=os.environ["AWS_REGION"]) + + # 4. 會自行運作的部分。 + monitor = IntegrityMonitor("s3://reports/2026", baseline=f"{STATE}/reports.baseline.json") + monitor.start() + + # 5. 對外監聽的部分：只綁定 loopback、需要密鑰，而且只開放用戶端需要的動作。 + install_operational_metrics() + start_metrics_server(port=9945) + start_http_action_server( + port=9944, + shared_secret=os.environ["FA_SHARED_SECRET"], + action_acl=ActionACL.build(allowed=["FA_pipeline_run", "FA_storage_copy", "FA_storage_list"]), + ) + + # 6. 持續存活，直到被要求停止。 + stop = threading.Event() + signal.signal(signal.SIGTERM, lambda *_: stop.set()) + signal.signal(signal.SIGINT, lambda *_: stop.set()) + stop.wait() + monitor.stop() + +請用你的服務管理員來執行它。以 systemd 為例： + +.. code-block:: ini + + [Unit] + Description=FileAutomation + After=network-online.target + + [Service] + User=fileautomation + EnvironmentFile=/etc/fileautomation/environment + Environment=FILE_AUTOMATION_LOG_FILE=/var/log/fileautomation/FileAutomation.log + ExecStart=/opt/fileautomation/venv/bin/python /opt/fileautomation/service.py + Restart=on-failure + StateDirectory=fileautomation + LogsDirectory=fileautomation + + [Install] + WantedBy=multi-user.target + +在 Windows 上，同一支腳本可以透過 NSSM 之類的包裝程式當成服務執行，或由工作排程器以 +「不論使用者是否登入都執行」的方式啟動。 + +只需要執行一次的工作（由系統本身的 cron 啟動的夜間管線、CI 步驟中的一次驗證）不需要 +這個服務：命令列就能完成，並以你可以據以處理的結束碼結束（:doc:`cli`）:: + + python -m automation_file pipeline --audit /var/lib/fileautomation/audit.sqlite \ + run /etc/fileautomation/daily.yaml --store /var/lib/fileautomation/runs.sqlite + +磁碟上的狀態 +------------ + +.. list-table:: + :header-rows: 1 + :widths: 26 74 + + * - 項目 + - 位置與處理方式 + * - 稽核軌跡 + - 你交給 ``configure_audit`` 的 SQLite 檔案。它採用 WAL 模式，行程執行期間請用 + ``sqlite3 audit.sqlite ".backup …"`` 複製，不要用 ``cp``。依你的政策保留足夠久， + 再執行 ``python -m automation_file audit purge --db … --older-than-days 365``。 + * - 管線執行紀錄 + - ``SQLiteRunStore`` 的 SQLite 檔案。當機之後 ``resume`` 讀的就是它；沒有它，行程 + 結束時執行紀錄就被遺忘了。 + * - 完整性基準 + - 每棵受監控的目錄樹一個 JSON 檔。請把它放在它所描述的目錄樹之外，最好放在能 + 修改該目錄樹的帳號寫不到的地方：誰能改寫基準，誰就能掩蓋變更。 + * - OAuth 權杖 + - 你交給 Google Drive 用戶端的 ``token_path``。只允許服務帳號讀取。 + * - 日誌 + - ``~/.automation_file/logs/FileAutomation.log``，除非 ``FILE_AUTOMATION_LOG_FILE`` + 指定了其他路徑。超過 10 MB 的檔案會在行程開啟它時被移到 ``.1``；需要更多輪替 + 時請自行處理。 + * - 版本快照、資源回收筒、內容儲存庫 + - 你交給這些功能的目錄。在你清理之前，它們會持續成長。 + +網路暴露面 +---------- + +每個伺服器都只綁定 loopback 介面，除非你傳入 ``allow_non_loopback=True``；這個預設值 +也就是建議做法：動作伺服器會執行任何已註冊的東西，所以連得到它，就等於連得到服務 +帳號的 Python 提示字元。 + +* 同一台機器上的用戶端使用 loopback 位址與共享密鑰。 +* 其他地方的用戶端要經過某個負責終結 TLS 並驗證身分的元件（反向代理、SSH 通道、 + service mesh）。伺服器本身使用的是明文 HTTP 與明文 TCP。 +* 為每個伺服器設定帶有允許清單的 ``ActionACL``。ACL 也會檢查巢狀在另一個動作引數中 + 的動作。它看不到某個動作被指示去執行的檔案內容，所以對必須留在清單之內的用戶端， + 不要開放 ``FA_execute_files``，也不要開放以路徑指定的管線。 +* MCP 伺服器透過啟動它的行程的標準串流通訊，不需要任何連接埠；它的允許清單請見 + :doc:`mcp`。 + +對外連往呼叫端提供的 URL 的請求，會經過 SSRF 防護：只允許 ``http`` 與 ``https``， +而且不允許私有、loopback 或 link-local 位址。有了這道防護，才能安全地接受用戶端提供 +的 URL；請不要繞過它。 + +該監看什麼 +---------- + +.. list-table:: + :header-rows: 1 + :widths: 26 74 + + * - 訊號 + - 位置 + * - 健康狀態 + - HTTP 動作伺服器的 ``GET /healthz`` 與 ``GET /readyz``。 + * - 指標 + - ``start_metrics_server()`` 提供 Prometheus 文字格式。``automation_file_actions_total`` + 及其耗時直方圖一直都有；``install_operational_metrics()`` 會加上事件、通知與 + 儲存操作的計數器。 + * - 失敗 + - 事件：``pipeline.failed``、``task.failed``、``integrity.violation``、 + ``storage.error``、``scheduler.error``、``system.error``。把你想被告知的事件 + 路由到某個 sink（:doc:`notifications`）。 + * - 歷史 + - 稽核軌跡：``python -m automation_file audit search --db … --status error``，或用 + ``--correlation-id `` 查看一次執行所做的一切。 + * - 日誌 + - INFO 以上的訊息也會寫到標準錯誤輸出，由服務管理員收集。 + +承受失敗 +-------- + +* 為管線任務設定 ``RetryPolicy`` 來應付會自行消失的錯誤（連線中斷、被限流），並設定 + ``timeout`` 來應付永遠不回傳的情況。 +* 為不可以發生兩次的任務設定 ``idempotency_key``，並把執行紀錄保存在 + ``SQLiteRunStore``：重新啟動之後，``pipeline resume `` 只會重做沒有成功的 + 部分。 +* 排程工作在前一次執行尚未結束時不會啟動，除非你允許重疊。 +* 完整性監控的修復功能預設是關閉的，除非你設定了它。請先從警示開始，等你信任基準 + 之後，再加上隔離或還原。 + +升級 +---- + +1. 閱讀版本說明。修訂版只做修正；次版本可能會把東西標為棄用；只有主版本才會移除 + （:doc:`api_policy`）。 +2. 在新版本進入正式環境之前，先用 ``-W error::DeprecationWarning`` 對它執行你自己的 + 測試。 +3. 備份那兩個 SQLite 檔案。會改變儲存格式的版本仍然讀得懂前一種格式，並說明如何轉換。 +4. 安裝到舊環境旁邊的一個全新虛擬環境，再把服務切換過去；這樣要回復時，再切換一次 + 即可。 + +檢查清單 +-------- + +* 服務以專屬帳號執行，而且只有那個帳號能讀取機密與權杖檔案。 +* 版本與 extra 都已固定。 +* 稽核軌跡與執行紀錄儲存庫都是位於有備份的磁碟上的檔案。 +* 每個伺服器都綁定在 loopback 介面、設有共享密鑰與允許清單；任何遠端連線都經過 TLS。 +* 失敗會傳達給人：至少有一條針對錯誤的通知路由。 +* 基準存放在受監控目錄樹的寫入者無法修改的位置。 +* 日誌與持續成長的目錄都有你自己決定的保留期限。 diff --git a/docs/source/Zh-TW/zh_tw_index.rst b/docs/source/Zh-TW/zh_tw_index.rst index 727367f..3a0651d 100644 --- a/docs/source/Zh-TW/zh_tw_index.rst +++ b/docs/source/Zh-TW/zh_tw_index.rst @@ -336,3 +336,17 @@ Slack、Email（SMTP）、Discord、Telegram、Microsoft Teams、PagerDuty :caption: 整合測試 usage/integration_tests + +.. _zh-tw-deployment: + +第 23 章 — 部署到正式環境 +========================= + +無人看管地運作：要安裝什麼、單一的長時間行程、磁碟上的狀態、網路上要開放什麼、 +該監看什麼，以及如何升級。 + +.. toctree:: + :maxdepth: 2 + :caption: 部署到正式環境 + + usage/deployment diff --git a/docs/updates/2026-10.md b/docs/updates/2026-10.md index 2cac00b..318d9a7 100644 --- a/docs/updates/2026-10.md +++ b/docs/updates/2026-10.md @@ -572,3 +572,18 @@ Index and query commands: [README.md](README.md). New entries go at the end. - **Docs**: the case count and the description of the suite in the three `usage/storage.rst` pages, the count in the three READMEs and the three `usage/integration_tests.rst` pages. - **Files**: `tests/storage_contract.py`, the documentation above, `progress.md`. - **Open items**: #36. + +## U-20261008-26 · 2026-10-08 · Production deployment guide · #docs #roadmap + +- **What**: a manual chapter, "Deploying to production", in the three languages (roadmap §14 item 15 and the definition of done: "documentation includes migration and production deployment guidance"; the migration half is still `progress.md` #26). + - Install: a pinned version with named extras, a check of what the installation reaches, a service account. + - Configuration and secrets: `automation_file.toml` with `${env:…}` / `${file:…}` references, and where a secret must not go. + - One process: the scheduler, the monitors, the router, the audit trail and the servers are threads, so a deployment is one script. The page has the script, a systemd unit, the Windows equivalents, and the one-shot alternative through the command line. + - State on disk: the audit trail, the run store, the baselines, the token files, the log, the growing directories, and how to back up a WAL database. + - Network exposure, what to watch (health, metrics, events, the audit trail, the log), surviving failure, upgrading, a checklist. +- **Verified**: the service script of the page was run in a scratch directory with local stand-ins for the S3 parts: the configuration applied and started the router, the audit trail recorded, the monitor started and stopped, `/healthz` answered 200, an allowed action ran with the shared secret, an action outside the allow list got 403, and the metrics endpoint served the operational counters. The names the examples import exist (checked by script). +- **Not verified**: the systemd unit and the Windows service wrappers; the Sphinx build of the pages (headings and markup were checked by script). +- **Also here**: the README bullet that still said the S3, Azure, Dropbox and SFTP backends are "installed by default" was corrected in the three READMEs in the commit before this one; it had been wrong since the extras were split (U-20261008-09). +- **Docs**: chapter 23 in the three manuals (`usage/deployment.rst`), the indexes, a "Deployment" section in the three READMEs. +- **Files**: the three `usage/deployment.rst` pages, the three indexes, the three READMEs. +- **Open items**: the scheduler, MCP and GUI specifics are added when that work lands (#26). diff --git a/docs/updates/README.md b/docs/updates/README.md index b41793b..6fececa 100644 --- a/docs/updates/README.md +++ b/docs/updates/README.md @@ -58,6 +58,7 @@ In the same commit: delete the item from `progress.md`, add a `#done` entry here | ID | Date | Title | Tags | Batch | |---|---|---|---|---| +| U-20261008-26 | 2026-10-08 | Production deployment guide | #docs #roadmap | [2026-10](2026-10.md) | | U-20261008-25 | 2026-10-08 | Metadata cases in the storage contract | #storage #tests #done | [2026-10](2026-10.md) | | U-20261008-24 | 2026-10-08 | A release can raise MINOR or MAJOR | #release #ci #roadmap | [2026-10](2026-10.md) | | U-20261008-23 | 2026-10-08 | Integration tests and their workflow | #ci #tests #roadmap | [2026-10](2026-10.md) | @@ -120,5 +121,5 @@ In the same commit: delete the item from `progress.md`, add a `#done` entry here | File | Period | Entries | |---|---|---:| -| [2026-10.md](2026-10.md) | 2026-10 | 36 | +| [2026-10.md](2026-10.md) | 2026-10 | 37 | | [2026-09.md](2026-09.md) | 2026-09 | 21 | From e998011f0002a6f6b6e4d72b72362bbe807304b1 Mon Sep 17 00:00:00 2001 From: JeffreyChen Date: Thu, 8 Oct 2026 15:13:40 +0800 Subject: [PATCH 44/59] feat: add semantic MCP tools with a permission model --- CLAUDE.md | 9 +- README.md | 84 +- README.zh-CN.md | 76 +- README.zh-TW.md | 76 +- architecture.md | 8 +- automation_file/__init__.py | 4 + automation_file/__main__.py | 10 + automation_file/server/mcp_file_tools.py | 414 ++++++++ .../server/mcp_pipeline_actions.py | 276 +++++ automation_file/server/mcp_pipeline_tools.py | 446 +++++++++ automation_file/server/mcp_policy.py | 448 +++++++++ automation_file/server/mcp_report_tools.py | 128 +++ automation_file/server/mcp_server.py | 213 +++- automation_file/server/mcp_storage_tools.py | 319 ++++++ automation_file/server/mcp_tool_model.py | 164 +++ automation_file/server/mcp_tools.py | 379 +++++++ docs/source/API/server.rst | 27 + docs/source/Eng/usage/cli.rst | 8 +- docs/source/Eng/usage/mcp.rst | 940 +++++++++++++++--- docs/source/Zh-CN/usage/cli.rst | 9 +- docs/source/Zh-CN/usage/mcp.rst | 843 +++++++++++++--- docs/source/Zh-TW/usage/cli.rst | 9 +- docs/source/Zh-TW/usage/mcp.rst | 855 +++++++++++++--- docs/updates/2026-10.md | 18 + docs/updates/README.md | 3 +- examples/mcp/README.md | 52 +- examples/mcp/claude_desktop_config.json | 8 + progress.md | 2 +- tests/mcp_support.py | 103 ++ tests/test_mcp_pipeline_tools.py | 729 ++++++++++++++ tests/test_mcp_policy.py | 495 +++++++++ tests/test_mcp_semantic_server.py | 433 ++++++++ tests/test_mcp_storage_tools.py | 318 ++++++ tests/test_mcp_tools.py | 684 +++++++++++++ 34 files changed, 8085 insertions(+), 505 deletions(-) create mode 100644 automation_file/server/mcp_file_tools.py create mode 100644 automation_file/server/mcp_pipeline_actions.py create mode 100644 automation_file/server/mcp_pipeline_tools.py create mode 100644 automation_file/server/mcp_policy.py create mode 100644 automation_file/server/mcp_report_tools.py create mode 100644 automation_file/server/mcp_storage_tools.py create mode 100644 automation_file/server/mcp_tool_model.py create mode 100644 automation_file/server/mcp_tools.py create mode 100644 tests/mcp_support.py create mode 100644 tests/test_mcp_pipeline_tools.py create mode 100644 tests/test_mcp_policy.py create mode 100644 tests/test_mcp_semantic_server.py create mode 100644 tests/test_mcp_storage_tools.py create mode 100644 tests/test_mcp_tools.py diff --git a/CLAUDE.md b/CLAUDE.md index b9cbe1b..978ef76 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -40,7 +40,8 @@ automation_file/ ├── integrity/ # IntegrityMonitor 2.0: target, hashing, snapshot, manifest (schema 2), baseline, │ # detector, report, alerts, remediation, watcher / local_watcher, legacy, monitor, │ # actions (FA_integrity_*); core/fim.py re-exports IntegrityMonitor -├── server/ # tcp_server, http_server, mcp_server (MCP over stdio), web_ui, metrics_server, +├── server/ # tcp_server, http_server, mcp_server (MCP over stdio), mcp_policy (MCPPolicy), +│ # mcp_tools + mcp_*_tools (the fourteen semantic tools), web_ui, metrics_server, │ # action_acl (ActionACL), network_guards (ensure_loopback) ├── client/ # HTTPActionClient for the HTTP action server ├── pipeline/ # Pipeline runtime: model (Task, RetryPolicy, PipelineRun, ...), graph, pipeline @@ -176,6 +177,12 @@ All code must follow secure-by-default principles. Review every change against t - `POST /actions` is the only endpoint that runs anything; the `GET` routes (`/healthz`, `/readyz`, `/openapi.json`, `/progress`) only report. Request body capped at 1 MB — do not raise without also switching to a streaming parser. - Responses are JSON. Auth failures return `401`; malformed JSON returns `400`; unknown paths return `404`. +### MCP server +- The semantic tools work only below the roots of the server's `MCPPolicy`, and with no root they refuse. A local root is enforced by `LocalStorage(root)` / `safe_join`, never by comparing strings; a remote root by scheme, exact authority and a path prefix that ends at a segment boundary. +- The default policy is read-only. Writing, overwriting and deleting are three separate permissions; never fold one into another, and never make a new changing tool available without `dry_run`. +- A pipeline made or run through MCP uses the guarded action set of `mcp_pipeline_actions.py`. An action added with `--pipeline-actions` runs unconfined, and the manual says so: keep it that explicit. +- A refused call is logged without argument values, and returned as a tool result with `isError`. + ### Path traversal - Any caller resolving a user-supplied path against a trusted root must go through `automation_file.local.safe_paths.safe_join` (raises `PathTraversalException`) or the `is_within` check. Never concatenate + `Path.resolve()` yourself and skip the containment check — symlinks and `..` segments bypass naive string checks. diff --git a/README.md b/README.md index d97a578..0d0e615 100644 --- a/README.md +++ b/README.md @@ -53,6 +53,7 @@ facade. - **Notification router** — routes decide which sinks hear about which events (by type, source and minimum severity), with deduplication and rate limiting per route; declare them in code, in `automation_file.toml` or with `FA_notify_route_*` - **Audit trail** — `configure_audit(path)` records one row per event and per storage operation (actor, source, pipeline, task, action, resource, backend, status, duration, correlation ID), searchable with `audit_search` / `FA_audit_search` - **Pipelines** — `Pipeline` runs tasks (callables or `FA_*` actions) in dependency order, independent ones in parallel, with retry, timeout, cancellation, conditions, idempotency keys, checkpoint and resume, a dry run and an execution history; definitions in Python, YAML or JSON +- **Semantic MCP tools** — fourteen tools with stable names (`file_read`, `file_copy`, `storage_list`, `pipeline_run`, `integrity_status`, `audit_search`, …) for AI hosts, confined to the roots you name, read-only until you allow writing, with a dry run for everything that changes something; the `FA_*` bridge stays available - PySide6 GUI (`python -m automation_file ui`) with a tab per backend, the JSON-action runner, and dedicated tabs for Triggers, Scheduler, and live Progress - Rich CLI with one-shot subcommands plus legacy JSON-batch flags - Project scaffolding (`ProjectBuilder`) for executor-based automations @@ -1090,33 +1091,74 @@ server = start_web_ui(host="127.0.0.1", port=9955, shared_secret="s3cr3t") ``` ### MCP (Model Context Protocol) server -Expose every registered `FA_*` action to an MCP host (Claude Desktop, MCP -CLIs) over JSON-RPC 2.0 on stdio: -```python -from automation_file import MCPServer - -MCPServer().serve_stdio() # reads JSON-RPC from stdin, writes to stdout -``` - -`pip install` exposes an `automation_file_mcp` console script (via -`[project.scripts]`) so MCP hosts can launch the bridge without any Python -glue. Three equivalent launch styles: +`MCPServer` speaks MCP over stdio (JSON-RPC 2.0), so an AI client such as Claude +Desktop or Claude Code can work with files through this library. It offers +fourteen **semantic tools** bound to a permission policy, and, for compatibility, +the **bridge** that exposes every registered `FA_*` action as a tool. ```bash -automation_file_mcp # installed console script -python -m automation_file mcp # CLI subcommand -python examples/mcp/run_mcp.py # standalone launcher +# Read-only access to one directory, semantic tools only +python -m automation_file mcp --root /srv/reports --no-bridge + +# Two locations, writing allowed, pipeline definitions kept on disk +python -m automation_file mcp --root s3://reports-export/daily --root /srv/outbox \ + --allow-write --pipeline-dir /var/lib/automation_file/pipelines --no-bridge ``` -All three accept `--name`, `--version`, and `--allowed-actions` (comma- -separated whitelist — strongly recommended since the default registry -includes high-privilege actions like `FA_run_shell`). See -[`examples/mcp/`](examples/mcp) for ready-to-copy Claude Desktop config. +```python +from automation_file import MCPServer +from automation_file.server.mcp_policy import MCPPolicy +from automation_file.server.mcp_tools import SemanticToolkit + +policy = MCPPolicy(roots=["s3://reports-export/daily", "sftp://sftp.example.com/inbound"], + allow_write=True, allow_delete=True) +MCPServer(policy=policy, bridge=False).serve_stdio() # blocks until stdin closes + +# The same tools without JSON-RPC, for tests and embedding +toolkit = SemanticToolkit(MCPPolicy(roots=["/srv/reports"], allow_write=True)) +outcome = toolkit.call( + "file_copy", + {"source": "/srv/reports/in/a.csv", "target": "/srv/reports/out/a.csv", "dry_run": True}, +) +outcome.is_error, outcome.payload["overwrites"], outcome.correlation_id +``` -Tool descriptors are generated on the fly by introspecting each action's -signature — parameter names and types become a JSON schema, so hosts can -render fields without any manual wiring. +- **Fourteen stable tools.** `file_read`, `file_write`, `file_copy`, `file_move`, + `file_search`, `file_checksum`, `file_verify`, `storage_list`, `storage_copy`, + `pipeline_create`, `pipeline_run`, `pipeline_status`, `integrity_status` and + `audit_search`. They take storage URIs, have hand-written input schemas, and + answer with one JSON document. +- **Safe defaults.** No location is allowed until `--root` names one; the server is + read-only until `--allow-write`; replacing a file and deleting (a move deletes + its source) need `--allow-overwrite` and `--allow-delete`. Reads, listings, + searches and written content are capped (`--max-read-bytes`, `--max-results`, + `--max-search-bytes`, `--max-write-bytes`). +- **Roots that hold.** A local root is served by a `LocalStorage` confined to it, + so a symbolic link or an absolute path that leaves it is refused by `safe_join`. + Other backends are compared by scheme, authority and whole path segments: + `s3://bucket/team` does not allow `s3://bucket/team-b`. +- **Dry run.** Every tool that changes something takes `dry_run` and returns what it + would do: source, target, sizes, whether something would be replaced. +- **Pipelines under the policy.** A pipeline created or run through MCP may call + only the `FA_storage_*` actions the permissions cover, in guarded versions that + check the roots when each task runs. `--pipeline-actions` lists other actions + explicitly; `FA_run_shell` is never available unless it is listed. +- **Traceable.** Each call runs under a correlation ID and an `mcp` actor and is + published as `mcp.tool.completed` or `mcp.tool.failed`, so `audit_search` returns + what a call did and a notification route can alert on refusals and failures. +- **The `FA_*` bridge.** On by default, as before, with `--allowed-actions` to + narrow it. The policy does not bind it: use `--no-bridge` for an AI client. + +`pip install` provides the `automation_file_mcp` console script, which takes the +same flags as `python -m automation_file mcp`. See the +[MCP manual](docs/source/Eng/usage/mcp.rst) for the permission model, the example +workflow (S3 to SFTP, verified, audited, alert on failure) and the security +guidance, and [`examples/mcp/`](examples/mcp) for host configurations. + +Suggested feature bullet: + +- **MCP (Model Context Protocol) server** — `MCPServer` serves fourteen semantic tools (`file_read`, `file_copy`, `pipeline_run`, `audit_search`, ...) bound to a permission policy with allowed roots, read-only defaults and dry run, next to the bridge that exposes every `FA_*` action, over newline-delimited JSON-RPC 2.0 on stdio ### DAG action executor Run actions in dependency order; independent branches fan out across a diff --git a/README.zh-CN.md b/README.zh-CN.md index 2f96297..ce959cc 100644 --- a/README.zh-CN.md +++ b/README.zh-CN.md @@ -51,6 +51,7 @@ TCP / HTTP 服务器执行的 JSON 驱动动作。内附 PySide6 GUI，每个功 - **通知路由器** — 以路由决定哪些事件（按类型、来源与最低严重程度）发送到哪些 sink，每条路由各自去重与限流；可在代码、`automation_file.toml` 或通过 `FA_notify_route_*` 声明 - **审计轨迹** — `configure_audit(path)` 为每个事件与每次存储操作记录一条（actor、来源、pipeline、task、动作、资源、后端、状态、耗时、关联 ID），可用 `audit_search` / `FA_audit_search` 查询 - **流水线（Pipeline）** — `Pipeline` 按依赖顺序执行任务（可调用对象或 `FA_*` 动作），互不依赖者并行执行，并支持重试、超时、取消、条件、幂等键、检查点与续跑、试运行以及执行历史；定义可以用 Python、YAML 或 JSON 编写 +- **语义化 MCP 工具** — 提供给 AI 宿主的十四个名称稳定的工具（`file_read`、`file_copy`、`storage_list`、`pipeline_run`、`integrity_status`、`audit_search` 等），仅限于你指定的根位置，在你允许写入之前均为只读，所有会变更内容的工具都支持试运行；`FA_*` 桥接仍然保留 - PySide6 GUI（`python -m automation_file ui`）每个后端一个页签，含 JSON 动作执行器，另有 Triggers、Scheduler、实时 Progress 专属页签 - 功能丰富的 CLI，包含一次性子命令与旧式 JSON 批量标志 - 项目脚手架（`ProjectBuilder`）协助构建以 executor 为核心的自动化项目 @@ -1061,30 +1062,69 @@ server = start_web_ui(host="127.0.0.1", port=9955, shared_secret="s3cr3t") ``` ### MCP（Model Context Protocol）服务器 -通过 stdio 上的 JSON-RPC 2.0 把每个已注册的 `FA_*` 动作暴露给 MCP 主机 -（Claude Desktop、MCP CLI）： -```python -from automation_file import MCPServer - -MCPServer().serve_stdio() # 从 stdin 读取 JSON-RPC，写入 stdout -``` - -`pip install` 后，`[project.scripts]` 会提供 `automation_file_mcp` console -script，MCP 主机无需编写 Python glue 即可启动桥接器。三种等价的启动方式： +`MCPServer` 通过 stdio 上的 JSON-RPC 2.0 提供 MCP，让 Claude Desktop 或 Claude Code +这类 AI 客户端可以通过本库处理文件。它提供十四个受权限策略约束的 **语义工具**， +并且为了兼容，保留把每个已注册的 `FA_*` 动作暴露为工具的 **桥接**。 ```bash -automation_file_mcp # 已安装的 console script -python -m automation_file mcp # CLI 子命令 -python examples/mcp/run_mcp.py # 独立启动脚本 +# 对单个目录的只读访问，只提供语义工具 +python -m automation_file mcp --root /srv/reports --no-bridge + +# 两个位置、允许写入、流水线定义存放在磁盘上 +python -m automation_file mcp --root s3://reports-export/daily --root /srv/outbox \ + --allow-write --pipeline-dir /var/lib/automation_file/pipelines --no-bridge ``` -三者都支持 `--name`、`--version`、`--allowed-actions`（逗号分隔白名单—— -强烈建议使用，因为默认注册表包含 `FA_run_shell` 等高权限动作）。可直接复制的 -Claude Desktop 示例配置请见 [`examples/mcp/`](examples/mcp)。 +```python +from automation_file import MCPServer +from automation_file.server.mcp_policy import MCPPolicy +from automation_file.server.mcp_tools import SemanticToolkit + +policy = MCPPolicy(roots=["s3://reports-export/daily", "sftp://sftp.example.com/inbound"], + allow_write=True, allow_delete=True) +MCPServer(policy=policy, bridge=False).serve_stdio() # 阻塞到 stdin 关闭为止 + +# 不经 JSON-RPC 使用同一组工具，适合测试与嵌入 +toolkit = SemanticToolkit(MCPPolicy(roots=["/srv/reports"], allow_write=True)) +outcome = toolkit.call( + "file_copy", + {"source": "/srv/reports/in/a.csv", "target": "/srv/reports/out/a.csv", "dry_run": True}, +) +outcome.is_error, outcome.payload["overwrites"], outcome.correlation_id +``` -工具描述符在运行时由动作签名自动生成——参数名称与类型会转换为 JSON schema， -主机无需任何手动配置即可渲染字段。 +- **十四个名称稳定的工具。** `file_read`、`file_write`、`file_copy`、`file_move`、 + `file_search`、`file_checksum`、`file_verify`、`storage_list`、`storage_copy`、 + `pipeline_create`、`pipeline_run`、`pipeline_status`、`integrity_status` 与 + `audit_search`。它们接受存储 URI，输入 schema 为手写，并以一份 JSON 文档响应。 +- **安全的默认值。** 在 `--root` 指定位置之前不允许任何位置；在 `--allow-write` 之前 + 服务器是只读的；替换文件与删除（移动会删除来源）分别需要 `--allow-overwrite` 与 + `--allow-delete`。读取、列表、搜索与写入的内容都有上限（`--max-read-bytes`、 + `--max-results`、`--max-search-bytes`、`--max-write-bytes`）。 +- **守得住的根位置。** 本地根位置由限制在其内的 `LocalStorage` 提供服务，所以离开它的 + 符号链接或绝对路径会被 `safe_join` 拒绝。其他后端按 scheme、authority 与完整的路径 + 段比较：`s3://bucket/team` 不会允许 `s3://bucket/team-b`。 +- **试运行。** 每个会改动东西的工具都接受 `dry_run`，并返回它将会做的事：来源、目标、 + 大小、是否会替换什么。 +- **受策略约束的流水线。** 通过 MCP 创建或运行的流水线只能调用权限所涵盖的 + `FA_storage_*` 动作，而且是受防护的版本，会在每个任务运行时检查根位置。 + `--pipeline-actions` 可以显式列出其他动作；`FA_run_shell` 除非被列出，否则永远 + 无法使用。 +- **可追溯。** 每次调用都带有关联 ID 与 `mcp` actor，并发布为 `mcp.tool.completed` 或 + `mcp.tool.failed`，所以 `audit_search` 能返回某次调用做了什么，通知路由也能在拒绝与 + 失败时发出告警。 +- **`FA_*` 桥接。** 和以前一样默认开启，可用 `--allowed-actions` 缩小范围。策略不约束 + 它：面对 AI 客户端请使用 `--no-bridge`。 + +`pip install` 会提供 `automation_file_mcp` 控制台脚本，它接受与 +`python -m automation_file mcp` 相同的命令行参数。权限模型、示例流程（S3 到 SFTP、 +验证、审计、失败时告警）与安全指引请见 [MCP 手册](docs/source/Zh-CN/usage/mcp.rst)， +宿主的配置示例请见 [`examples/mcp/`](examples/mcp)。 + +建议的功能条目： + +- **MCP（Model Context Protocol）服务器** — `MCPServer` 通过 stdio 上以换行分隔的 JSON-RPC 2.0，提供十四个受权限策略约束的语义工具（`file_read`、`file_copy`、`pipeline_run`、`audit_search` ……；允许的根位置、默认只读、试运行），以及把每个 `FA_*` 动作暴露为工具的桥接 ### DAG 动作执行器 按依赖顺序执行动作；独立分支通过线程池并行展开。每个节点的形式为 diff --git a/README.zh-TW.md b/README.zh-TW.md index 771b078..21b302e 100644 --- a/README.zh-TW.md +++ b/README.zh-TW.md @@ -51,6 +51,7 @@ TCP / HTTP 伺服器執行的 JSON 驅動動作。內附 PySide6 GUI，每個功 - **通知路由器** — 以路由決定哪些事件（依類型、來源與最低嚴重程度）送到哪些 sink，每條路由各自去重與限流；可在程式、`automation_file.toml` 或以 `FA_notify_route_*` 宣告 - **稽核軌跡** — `configure_audit(path)` 為每個事件與每次儲存操作記錄一筆（actor、來源、pipeline、task、動作、資源、後端、狀態、耗時、關聯 ID），可用 `audit_search` / `FA_audit_search` 查詢 - **管線（Pipeline）** — `Pipeline` 依相依順序執行任務（可呼叫物件或 `FA_*` 動作），互不相依者平行執行，並支援重試、逾時、取消、條件、冪等鍵、檢查點與續跑、試跑以及執行歷史；定義可用 Python、YAML 或 JSON 撰寫 +- **語意化 MCP 工具** — 提供給 AI 宿主的十四個名稱穩定的工具（`file_read`、`file_copy`、`storage_list`、`pipeline_run`、`integrity_status`、`audit_search` 等），僅限於你指定的根位置，在你允許寫入之前皆為唯讀，所有會變更內容的工具都支援試跑；`FA_*` 橋接仍然保留 - PySide6 GUI（`python -m automation_file ui`）每個後端一個分頁，含 JSON 動作執行器，另有 Triggers、Scheduler、即時 Progress 專屬分頁 - 功能豐富的 CLI，包含一次性子指令與舊式 JSON 批次旗標 - 專案鷹架（`ProjectBuilder`）協助建立以 executor 為核心的自動化專案 @@ -1061,30 +1062,69 @@ server = start_web_ui(host="127.0.0.1", port=9955, shared_secret="s3cr3t") ``` ### MCP（Model Context Protocol）伺服器 -透過 stdio 上的 JSON-RPC 2.0 將登錄的每個 `FA_*` 動作暴露給 MCP 主機 -（Claude Desktop、MCP CLI）： -```python -from automation_file import MCPServer - -MCPServer().serve_stdio() # 從 stdin 讀取 JSON-RPC，寫入 stdout -``` - -`pip install` 後，`[project.scripts]` 會提供 `automation_file_mcp` console -script，MCP 主機不需要寫任何 Python glue 也能啟動橋接器。三種等價的啟動方式： +`MCPServer` 透過 stdio 上的 JSON-RPC 2.0 提供 MCP，讓 Claude Desktop 或 Claude Code +這類 AI 用戶端可以透過本函式庫處理檔案。它提供十四個受權限政策約束的 **語意工具**， +並且為了相容，保留把每個已註冊的 `FA_*` 動作暴露為工具的 **橋接**。 ```bash -automation_file_mcp # 已安裝的 console script -python -m automation_file mcp # CLI 子指令 -python examples/mcp/run_mcp.py # 獨立啟動腳本 +# 對單一目錄的唯讀存取，只提供語意工具 +python -m automation_file mcp --root /srv/reports --no-bridge + +# 兩個位置、允許寫入、管線定義存放在磁碟上 +python -m automation_file mcp --root s3://reports-export/daily --root /srv/outbox \ + --allow-write --pipeline-dir /var/lib/automation_file/pipelines --no-bridge ``` -三者皆支援 `--name`、`--version`、`--allowed-actions`（逗號分隔白名單—— -強烈建議使用，因為預設登錄表包含 `FA_run_shell` 等高權限動作）。可直接複製的 -Claude Desktop 範例設定請見 [`examples/mcp/`](examples/mcp)。 +```python +from automation_file import MCPServer +from automation_file.server.mcp_policy import MCPPolicy +from automation_file.server.mcp_tools import SemanticToolkit + +policy = MCPPolicy(roots=["s3://reports-export/daily", "sftp://sftp.example.com/inbound"], + allow_write=True, allow_delete=True) +MCPServer(policy=policy, bridge=False).serve_stdio() # 阻塞到 stdin 關閉為止 + +# 不經 JSON-RPC 使用同一組工具，適合測試與內嵌 +toolkit = SemanticToolkit(MCPPolicy(roots=["/srv/reports"], allow_write=True)) +outcome = toolkit.call( + "file_copy", + {"source": "/srv/reports/in/a.csv", "target": "/srv/reports/out/a.csv", "dry_run": True}, +) +outcome.is_error, outcome.payload["overwrites"], outcome.correlation_id +``` -工具描述在執行時由動作簽章自動生成——參數名稱與型別會轉換為 JSON schema， -主機無需任何手動設定即可渲染欄位。 +- **十四個名稱穩定的工具。** `file_read`、`file_write`、`file_copy`、`file_move`、 + `file_search`、`file_checksum`、`file_verify`、`storage_list`、`storage_copy`、 + `pipeline_create`、`pipeline_run`、`pipeline_status`、`integrity_status` 與 + `audit_search`。它們接受儲存 URI，輸入 schema 為手寫，並以一份 JSON 文件回應。 +- **安全的預設值。** 在 `--root` 指定位置之前不允許任何位置；在 `--allow-write` 之前 + 伺服器是唯讀的；取代檔案與刪除（搬移會刪除來源）分別需要 `--allow-overwrite` 與 + `--allow-delete`。讀取、列表、搜尋與寫入的內容都有上限（`--max-read-bytes`、 + `--max-results`、`--max-search-bytes`、`--max-write-bytes`）。 +- **守得住的根位置。** 本機根位置由限制在其內的 `LocalStorage` 提供服務，所以離開它的 + 符號連結或絕對路徑會被 `safe_join` 拒絕。其他後端以 scheme、authority 與完整的路徑 + 區段比對：`s3://bucket/team` 不會允許 `s3://bucket/team-b`。 +- **試跑。** 每個會更動東西的工具都接受 `dry_run`，並回傳它將會做的事：來源、目標、 + 大小、是否會取代什麼。 +- **受政策約束的管線。** 透過 MCP 建立或執行的管線只能呼叫權限所涵蓋的 + `FA_storage_*` 動作，而且是受防護的版本，會在每個任務執行時檢查根位置。 + `--pipeline-actions` 可以明確列出其他動作；`FA_run_shell` 除非被列出，否則永遠 + 無法使用。 +- **可追溯。** 每次呼叫都帶有關聯 ID 與 `mcp` actor，並發布為 `mcp.tool.completed` 或 + `mcp.tool.failed`，所以 `audit_search` 能回傳某次呼叫做了什麼，通知路由也能在拒絕與 + 失敗時發出警示。 +- **`FA_*` 橋接。** 和以前一樣預設開啟，可用 `--allowed-actions` 縮小範圍。政策不約束 + 它：面對 AI 用戶端請使用 `--no-bridge`。 + +`pip install` 會提供 `automation_file_mcp` 主控台指令，它接受與 +`python -m automation_file mcp` 相同的旗標。權限模型、範例流程（S3 到 SFTP、驗證、 +稽核、失敗時警示）與安全指引請見 [MCP 手冊](docs/source/Zh-TW/usage/mcp.rst)，宿主的 +設定範例請見 [`examples/mcp/`](examples/mcp)。 + +建議的功能條目： + +- **MCP（Model Context Protocol）伺服器** — `MCPServer` 透過 stdio 上以換行分隔的 JSON-RPC 2.0，提供十四個受權限政策約束的語意工具（`file_read`、`file_copy`、`pipeline_run`、`audit_search` ……；允許的根位置、預設唯讀、試跑），以及把每個 `FA_*` 動作暴露為工具的橋接 ### DAG 動作執行器 依相依關係執行動作；獨立分支會透過執行緒池平行展開。每個節點形式為 diff --git a/architecture.md b/architecture.md index c124c32..e6d8a62 100644 --- a/architecture.md +++ b/architecture.md @@ -27,7 +27,7 @@ piece of the 1.0 roadmap (`docs/FILEAUTOMATION-1.0-ROADMAP.md`, PR #107); what i | `automation_file/storage/` | Universal storage layer. `uri.py` (`StorageURI`, `parse_storage_uri`, `normalize_path`), `types.py` (`FileInfo`, `Checksum`, `StorageCapabilities`), `backend.py` (`StorageBackend`: the public operations are template methods over the `_`-prefixed primitives a backend supplies), `local_storage.py` (`LocalStorage`, confined through `safe_join` when given a root), `memory_storage.py` (`MemoryStorage`), `object_storage.py` (`ObjectStorage`: directories as key prefixes over `_head`, `_scan`, `_put`, `_get`, `_remove`), `s3_storage.py` (`S3Storage`, over `s3_instance` or a given boto3 client), `azure_storage.py` (`AzureStorage`, over `azure_blob_instance` or a given `BlobServiceClient`), `session_storage.py` (`SessionStorage`: one login session, one operation at a time, and `require_session_host`), `sftp_storage.py` (`SFTPStorage`), `ftp_storage.py` (`FTPStorage`, for `ftp` and `ftps`), `gdrive_storage.py` (`GoogleDriveStorage`: paths resolved to file IDs, duplicate names refused), `onedrive_storage.py` (`OneDriveStorage`, Microsoft Graph), `dropbox_storage.py` (`DropboxStorage`), `webdav_storage.py` (`WebDAVStorage`), `smb_storage.py` (`SMBStorage`), `fsspec_storage.py` (`FsspecStorage`, any fsspec filesystem), `timestamps.py` (RFC 3339 parsing), `resolver.py` (`StorageResolver`, `default_resolver`: mounts first, then scheme factories), `file.py` (`File`), `storage.py` (`Storage`), `observe.py` (listeners for `upload`, `download`, `read`, `delete`, `mkdir`, `copy`, `move`), `streams.py` (staged file objects behind `open_read` / `open_write`), `tree.py` (`copy_tree`, `sync_tree`, `TreeResult`), `actions.py` (the `FA_storage_*` functions and `register_storage_ops`). At module level it imports only `exceptions`, `logging_config`, `core.checksum` and `local.safe_paths`: no registry, no GUI, no backend SDK. The adapters import their SDK's exceptions and the shared client inside the functions that use them | | `automation_file/events/` | The event model every component reports through. `model.py` (`Event`, `Severity`, the ten core events), `bus.py` (`EventBus`, the process-wide `event_bus`, `emit`), `context.py` (`correlation_scope`, `actor_scope`), `storage_bridge.py` (failed storage operations become `StorageError` events; installed when the package is imported). It imports only the standard library, `logging_config` and `storage.observe` | | `automation_file/integrity/` | IntegrityMonitor 2.0, on the storage layer and the event bus. `target.py` (`Target`: the monitored tree behind a storage URI), `hashing.py` (`HashEngine`; `md5` and `sha1` only with `allow_weak`), `snapshot.py` (`Snapshot`, `SnapshotEntry`, `build_snapshot`), `manifest.py` (schema version 2; the `write_manifest` format is read and converted), `baseline.py` (`BaselineManager`: an atomic write at any storage URI), `detector.py` (`Change`, `ChangeKind`, `detect_changes`: six kinds of change), `report.py` (`DriftReport`), `alerts.py` (`AlertEngine`, `AlertPolicy`: one `IntegrityViolation` per pass that finds drift), `remediation.py` (`RemediationPolicy`, `Remediator`: quarantine or restore, opt-in), `watcher.py` and `local_watcher.py` (polling, and watchdog events for a local target), `legacy.py` (the first monitor's summary, callback and notification), `monitor.py` (`IntegrityMonitor`), `actions.py` (`FA_integrity_*`). `core/fim.py` re-exports the class | -| `automation_file/server/` | `tcp_server.py`, `http_server.py`, `mcp_server.py`, `web_ui.py`, `metrics_server.py`, `action_acl.py` (`ActionACL`), `network_guards.py` (`ensure_loopback`) | +| `automation_file/server/` | `tcp_server.py`, `http_server.py`, `mcp_server.py` (JSON-RPC over stdio: the semantic tools first, then the `FA_*` bridge), `mcp_policy.py` (`MCPPolicy`, `StorageGuard`: roots, read-only by default, limits), `mcp_tools.py` and `mcp_*_tools.py` (`SemanticToolkit` and the fourteen tools), `mcp_pipeline_actions.py` (the guarded `FA_storage_*` set pipelines made through MCP run), `web_ui.py`, `metrics_server.py`, `action_acl.py` (`ActionACL`), `network_guards.py` (`ensure_loopback`) | | `automation_file/client/` | `HTTPActionClient` for the HTTP action server | | `automation_file/trigger/`, `scheduler/`, `notify/` | Watchdog file triggers, cron scheduler, notification sinks. Each registers its own `FA_*` ops. `notify/router.py` (`Route`, `NotificationRouter`, the process-wide `notification_router`) subscribes on the event bus and delivers events to named sinks by type, source and minimum severity, with deduplication and a rate limit per route and sink; a failing sink becomes a `system.error` event from the source `notify`, which is never routed | | `automation_file/pipeline/` | The pipeline runtime. `model.py` (`Task`, `TaskContext`, `RetryPolicy`, `Schedule`, `PipelineRun`, `TaskRun`, `RunStatus`, `TaskStatus`), `graph.py` (dependency order, cycles), `pipeline.py` (`Pipeline`: `task`, `run`, `start`, `resume`, `from_file` / `from_dict` / `to_dict`, `problems` / `validate`), `runner.py` and `worker.py` (one daemon thread per running task, capped at `max_workers`; retry, timeout, cancellation, conditions, idempotency), `substitution.py` (`${params.x}`, `${tasks.id.result}`), `store.py` (`RunStore`, `MemoryRunStore`, `SQLiteRunStore`: checkpoints and history), `definition.py` (`load_definition`, `validate_definition`, `PIPELINE_SCHEMA`), `reporting.py` (the `pipeline.*` and `task.*` events), `actions.py` (`FA_pipeline_*`). `core/dag_executor.py` is the older, unrecorded DAG helper and is unchanged | @@ -82,6 +82,12 @@ piece of the 1.0 roadmap (`docs/FILEAUTOMATION-1.0-ROADMAP.md`, PR #107); what i `STORAGE_OPERATION_DURATION`. Actions: `FA_notify_route_add` / `_remove` / `_list`, `FA_audit_configure` / `_search` / `_count` / `_purge`. Both are opt-in: the router delivers nothing until it has a route and is started, and the trail records nothing until it has a store. +- **Semantic MCP tools**: `file_read`, `file_write`, `file_copy`, `file_move`, `file_search`, + `file_checksum`, `file_verify`, `storage_list`, `storage_copy`, `pipeline_create`, `pipeline_run`, + `pipeline_status`, `integrity_status`, `audit_search`. Names, arguments and result shapes are public. + `MCPPolicy` and `SemanticToolkit` are on the facade; flags: `--root`, `--allow-write`, + `--allow-overwrite`, `--allow-delete`, `--max-read-bytes`, `--max-write-bytes`, `--max-results`, + `--max-search-bytes`, `--pipeline-dir`, `--pipeline-actions`, `--tools`, `--no-bridge`. - **Events** (same facade): `Event`, `Severity`, `EventBus`, `event_bus`, `emit`, `correlation_scope`, `actor_scope`, and the core events `PipelineStarted`, `PipelineCompleted`, `PipelineFailed`, `TaskStarted`, `TaskCompleted`, `TaskFailed`, `IntegrityViolation`, `StorageError`, `SchedulerError`, diff --git a/automation_file/__init__.py b/automation_file/__init__.py index 65618fe..907a035 100644 --- a/automation_file/__init__.py +++ b/automation_file/__init__.py @@ -328,7 +328,9 @@ ) from automation_file.server.action_acl import ActionACL, ActionNotPermittedException from automation_file.server.http_server import HTTPActionServer, start_http_action_server +from automation_file.server.mcp_policy import MCPPolicy from automation_file.server.mcp_server import MCPServer, tools_from_registry +from automation_file.server.mcp_tools import SemanticToolkit from automation_file.server.metrics_server import MetricsServer, start_metrics_server from automation_file.server.tcp_server import ( TCPActionServer, @@ -695,6 +697,8 @@ def __getattr__(name: str) -> Any: "WebUIServer", "start_web_ui", "MCPServer", + "MCPPolicy", + "SemanticToolkit", "tools_from_registry", # Triggers "FileWatcher", diff --git a/automation_file/__main__.py b/automation_file/__main__.py index 0d74585..79479ef 100644 --- a/automation_file/__main__.py +++ b/automation_file/__main__.py @@ -113,12 +113,21 @@ def _cmd_ui(_args: argparse.Namespace) -> int: return launch_ui() +def _add_mcp_policy_arguments(parser: argparse.ArgumentParser) -> None: + """Add the roots, permissions and limits of the semantic MCP tools.""" + from automation_file.server.mcp_server import add_semantic_arguments + + add_semantic_arguments(parser) + + def _cmd_mcp(args: argparse.Namespace) -> int: from automation_file.server.mcp_server import _cli as mcp_cli + from automation_file.server.mcp_server import semantic_argv forwarded: list[str] = ["--name", args.name, "--version", args.version] if args.allowed_actions: forwarded.extend(["--allowed-actions", args.allowed_actions]) + forwarded.extend(semantic_argv(args)) return mcp_cli(forwarded) @@ -203,6 +212,7 @@ def _add_integration_commands(subparsers: argparse._SubParsersAction) -> None: default=None, help="comma-separated allow list (default: expose every registered action)", ) + _add_mcp_policy_arguments(mcp_parser) mcp_parser.set_defaults(handler=_cmd_mcp) drive_parser = subparsers.add_parser("drive-upload", help="upload a file to Google Drive") diff --git a/automation_file/server/mcp_file_tools.py b/automation_file/server/mcp_file_tools.py new file mode 100644 index 0000000..1512e4f --- /dev/null +++ b/automation_file/server/mcp_file_tools.py @@ -0,0 +1,414 @@ +"""The semantic file tools: ``file_read``, ``file_write``, ``file_copy``, ``file_move``, +``file_checksum`` and ``file_verify``. + +Each handler is a plain function ``(session, arguments) -> dict``. It reaches +storage only through ``session.file``, so every location is checked by the +policy's guard, and it asks the policy before it writes, replaces or deletes. +A handler that changes something returns its plan and stops there when +``dry_run`` is set. +""" + +from __future__ import annotations + +import base64 +import binascii +import codecs +import hashlib +from dataclasses import dataclass +from typing import Any, BinaryIO + +from automation_file.exceptions import ( + StorageAlreadyExistsException, + StorageChecksumException, + StorageNotFoundException, + StoragePathTypeException, +) +from automation_file.server.mcp_policy import ( + LIMIT_EXCEEDED, + MCPPermissionException, + MCPToolException, +) +from automation_file.server.mcp_tool_model import ( + DRY_RUN_HELP, + OVERWRITE_HELP, + URI_HELP, + SemanticTool, + ToolSession, + arguments_schema, + flag, + text, + whole, +) +from automation_file.storage.backend import DEFAULT_CHECKSUM_ALGORITHM +from automation_file.storage.file import File +from automation_file.storage.types import Checksum, FileInfo + +BASE64 = "base64" +FILE_READ = "file_read" +FILE_WRITE = "file_write" +FILE_COPY = "file_copy" +FILE_MOVE = "file_move" +FILE_CHECKSUM = "file_checksum" +FILE_VERIFY = "file_verify" +_UTF8 = "utf-8" +_CHUNK = 1024 * 1024 +_ENCODING_HELP = ( + "A text encoding (utf-8, utf-16, big5, latin-1, ...) or base64 for content that is not text." +) + + +def described(uri: object, info: FileInfo) -> dict[str, Any]: + """Return one file or directory the way the tools report it.""" + return { + "uri": str(uri), + "path": info.path, + "name": info.name, + "is_dir": info.is_dir, + "size": info.size, + "modified_at": info.modified_at.isoformat() if info.modified_at else None, + } + + +def existing_file(source: File) -> FileInfo: + """Return the ``FileInfo`` of ``source``, which must be a file that exists.""" + info = source.stat() + if info.is_dir: + raise StoragePathTypeException(f"{source} is a directory, not a file") + return info + + +def replaceable(session: ToolSession, tool: str, target: File, overwrite: bool) -> FileInfo | None: + """Return the file now at ``target`` (``None`` when there is none) if it may be replaced. + + An existing file needs both the caller's ``overwrite`` and the policy's + permission to overwrite. + """ + try: + info = target.stat() + except StorageNotFoundException: + return None + if info.is_dir: + raise StoragePathTypeException(f"{target} is a directory, not a file") + if not overwrite: + hint = ( + "pass overwrite=true to replace it" + if session.policy.allow_overwrite + else "this server does not allow replacing files" + ) + raise StorageAlreadyExistsException(f"{target} already exists; {hint}") + session.policy.require_overwrite(tool) + return info + + +def read_window(stream: BinaryIO, offset: int, limit: int) -> bytes: + """Return at most ``limit`` bytes of ``stream``, starting ``offset`` bytes in.""" + if offset and stream.seekable(): + stream.seek(offset) + offset = 0 + chunks: list[bytes] = [] + remaining = limit + while offset > 0 or remaining > 0: + chunk = stream.read(min(offset or remaining, _CHUNK)) + if not chunk: + break + if offset > 0: + offset -= len(chunk) + else: + chunks.append(chunk) + remaining -= len(chunk) + return b"".join(chunks) + + +def _text_codec(encoding: str) -> str: + try: + "".encode(encoding) + except LookupError as error: + raise MCPToolException( + f"unknown text encoding {encoding[:40]!r}; use utf-8, another text encoding, or base64" + ) from error + return encoding + + +def _decoded(data: bytes, encoding: str, complete: bool) -> tuple[str, int]: + """Return ``data`` as text and the number of bytes that text stands for. + + When more of the file follows, a character cut in half at the end of ``data`` + is left for the next read. + """ + decoder = codecs.getincrementaldecoder(_text_codec(encoding))() + try: + content = decoder.decode(data, final=complete) + except UnicodeDecodeError as error: + raise MCPToolException( + f"the content is not {encoding} text from this offset; ask for encoding base64" + ) from error + return content, len(data) - len(decoder.getstate()[0]) + + +def _encoded(content: str, encoding: str) -> bytes: + if encoding == BASE64: + try: + return base64.b64decode(content, validate=True) + except (binascii.Error, ValueError) as error: + raise MCPToolException("content is not valid base64") from error + try: + return content.encode(_text_codec(encoding)) + except UnicodeEncodeError as error: + raise MCPToolException(f"content cannot be written as {encoding}") from error + + +def file_read(session: ToolSession, args: dict[str, Any]) -> dict[str, Any]: + """Return part of a file: at most ``max_bytes`` (and the policy's cap) from ``offset``.""" + source = session.file(args["uri"]) + info = existing_file(source) + offset, encoding = args["offset"], args["encoding"] + limit = min(args.get("max_bytes", session.policy.max_read_bytes), session.policy.max_read_bytes) + with source.open_read() as stream: + data = read_window(stream, offset, limit) + complete = len(data) < limit if info.size is None else offset + len(data) >= info.size + if encoding == BASE64: + content, used = base64.b64encode(data).decode("ascii"), len(data) + else: + content, used = _decoded(data, encoding, complete) + return { + "uri": str(source), + "size": info.size, + "offset": offset, + "bytes": used, + "truncated": not complete, + "next_offset": None if complete else offset + used, + "encoding": encoding, + "content": content, + } + + +def file_write(session: ToolSession, args: dict[str, Any]) -> dict[str, Any]: + """Store ``content`` as a file; the size is capped and an existing file needs overwrite.""" + policy = session.policy + policy.require_write(FILE_WRITE) + payload = _encoded(args["content"], args["encoding"]) + if len(payload) > policy.max_write_bytes: + raise MCPPermissionException( + f"the content is {len(payload)} bytes and this server writes at most " + f"{policy.max_write_bytes} per call (--max-write-bytes)", + LIMIT_EXCEEDED, + ) + target = session.file(args["uri"]) + replaced = replaceable(session, FILE_WRITE, target, args["overwrite"]) + body: dict[str, Any] = { + "uri": str(target), + "size": len(payload), + "sha256": hashlib.sha256(payload).hexdigest(), + "overwrites": replaced is not None, + "replaced_size": None if replaced is None else replaced.size, + "dry_run": args["dry_run"], + "written": False, + } + if not args["dry_run"]: + target.write(payload, overwrite=replaced is not None) + body["written"] = True + return body + + +@dataclass(frozen=True) +class _Transfer: + """One copy or move that passed every check.""" + + source: File + target: File + move: bool + overwrite: bool + verify: bool + + +def _carry_out(job: _Transfer) -> dict[str, Any]: + """Copy or move the file; with ``verify`` the source is deleted only after the digests match.""" + if not job.verify: + if job.move: + job.source.move_to(job.target, overwrite=job.overwrite) + else: + job.source.copy_to(job.target, overwrite=job.overwrite) + return {} + expected = job.source.checksum() + job.source.copy_to(job.target, overwrite=job.overwrite) + actual = job.target.checksum() + if not actual.matches(expected): + raise StorageChecksumException( + f"{job.target} does not have the SHA-256 of {job.source} after the copy; " + "the source was left in place" + ) + if job.move: + job.source.delete() + return {"sha256": actual.value, "verified": True} + + +def transfer(session: ToolSession, args: dict[str, Any], tool: str) -> dict[str, Any]: + """Copy (or, for ``file_move``, move) one file after the policy's checks.""" + move = tool == FILE_MOVE + if move: + session.policy.require_delete(tool) + else: + session.policy.require_write(tool) + source, target = session.file(args["source"]), session.file(args["target"]) + info = existing_file(source) + if source.uri == target.uri: + raise MCPToolException(f"{tool}: source and target are the same file") + replaced = replaceable(session, tool, target, args["overwrite"]) + body: dict[str, Any] = { + "source": str(source), + "target": str(target), + "size": info.size, + "overwrites": replaced is not None, + "replaced_size": None if replaced is None else replaced.size, + "deletes_source": move, + "dry_run": args["dry_run"], + "done": False, + } + if not args["dry_run"]: + job = _Transfer(source, target, move, replaced is not None, bool(args.get("verify"))) + body.update(_carry_out(job)) + body["done"] = True + return body + + +def file_copy(session: ToolSession, args: dict[str, Any]) -> dict[str, Any]: + """Copy one file to another location, in the same backend or another one.""" + return transfer(session, args, FILE_COPY) + + +def file_move(session: ToolSession, args: dict[str, Any]) -> dict[str, Any]: + """Move one file: copy it, then delete the source.""" + return transfer(session, args, FILE_MOVE) + + +def file_checksum(session: ToolSession, args: dict[str, Any]) -> dict[str, Any]: + """Return the digest of a file.""" + source = session.file(args["uri"]) + info = existing_file(source) + digest = source.checksum(args["algorithm"]) + return { + "uri": str(source), + "size": info.size, + "algorithm": digest.algorithm, + "value": digest.value, + } + + +def file_verify(session: ToolSession, args: dict[str, Any]) -> dict[str, Any]: + """Compare the digest of a file with ``expected`` and say whether they match.""" + source = session.file(args["uri"]) + expected, algorithm = args["expected"].strip(), args["algorithm"] + if ":" in expected: + try: + wanted = Checksum.parse(expected) + except ValueError as error: + raise MCPToolException( + "file_verify: expected must be a hex digest or ':'" + ) from error + algorithm, expected = wanted.algorithm, wanted.value + existing_file(source) + actual = source.checksum(algorithm) + return { + "uri": str(source), + "algorithm": actual.algorithm, + "expected": expected.lower(), + "actual": actual.value, + "match": actual.matches(expected), + } + + +_URI = {"uri": text(URI_HELP, minLength=1)} +_ALGORITHM = { + "algorithm": text( + "The hash: sha256 (default), sha512, sha1, md5 or another fixed-length hashlib name.", + default=DEFAULT_CHECKSUM_ALGORITHM, + ) +} +_TRANSFER_ARGUMENTS = { + "source": text(f"The file to read. {URI_HELP}", minLength=1), + "target": text(f"The file to create. {URI_HELP}", minLength=1), + "overwrite": flag(OVERWRITE_HELP), + "verify": flag( + "Compare the SHA-256 of the source and of the target after the copy and fail on a " + "mismatch. A move then deletes the source only once the digests match." + ), + "dry_run": flag(DRY_RUN_HELP), +} + +TOOLS: tuple[SemanticTool, ...] = ( + SemanticTool( + FILE_READ, + "Read a file from any storage backend. Returns the content as text, or as base64 " + "for a file that is not text, with its size. The server caps the bytes returned: " + "when 'truncated' is true, call again with offset set to 'next_offset'.", + arguments_schema( + { + **_URI, + "offset": whole("Byte to start at. Default 0.", minimum=0, default=0), + "max_bytes": whole("Return at most this many bytes. The server caps it."), + "encoding": text(_ENCODING_HELP, default=_UTF8, minLength=1), + }, + required=("uri",), + ), + file_read, + ), + SemanticTool( + FILE_WRITE, + "Create a file from the given content. Needs a server that allows writing. Refuses " + "to replace an existing file unless overwrite is true and the server allows " + "overwriting. Returns the size and SHA-256 of what was written.", + arguments_schema( + { + **_URI, + "content": text("The whole content of the file. The server caps its size."), + "encoding": text(_ENCODING_HELP, default=_UTF8, minLength=1), + "overwrite": flag(OVERWRITE_HELP), + "dry_run": flag(DRY_RUN_HELP), + }, + required=("uri", "content"), + ), + file_write, + changes=True, + ), + SemanticTool( + FILE_COPY, + "Copy one file to another location, in the same storage backend or across two " + "(for example S3 to SFTP). The source stays. Needs a server that allows writing. " + "Use storage_copy for a directory.", + arguments_schema(_TRANSFER_ARGUMENTS, required=("source", "target")), + file_copy, + changes=True, + ), + SemanticTool( + FILE_MOVE, + "Move one file to another location, in the same storage backend or across two. " + "The source is deleted, so the server must allow writing and deleting. Pass " + "verify=true to delete the source only after the SHA-256 of the copy matches.", + arguments_schema(_TRANSFER_ARGUMENTS, required=("source", "target")), + file_move, + changes=True, + ), + SemanticTool( + FILE_CHECKSUM, + "Compute the digest of a file (SHA-256 by default). Returns algorithm, value and size.", + arguments_schema({**_URI, **_ALGORITHM}, required=("uri",)), + file_checksum, + ), + SemanticTool( + FILE_VERIFY, + "Check that a file has an expected digest. Returns match (true or false) with the " + "expected and the actual digest; a mismatch is a result, not an error.", + arguments_schema( + { + **_URI, + "expected": text( + "The digest to expect: hex, or ':' such as 'sha256:9f86...'.", + minLength=1, + ), + **_ALGORITHM, + }, + required=("uri", "expected"), + ), + file_verify, + ), +) diff --git a/automation_file/server/mcp_pipeline_actions.py b/automation_file/server/mcp_pipeline_actions.py new file mode 100644 index 0000000..b232109 --- /dev/null +++ b/automation_file/server/mcp_pipeline_actions.py @@ -0,0 +1,276 @@ +"""The ``FA_storage_*`` actions as a pipeline run through MCP may call them. + +A pipeline task names an action and its arguments, and the arguments are only +known when the task runs (``${params.date}``, ``${tasks.fetch.result}``). The +policy therefore cannot be checked on the definition alone: these functions +take the place of the storage actions in the registry such a pipeline is given, +and each one checks the location and the permission at the moment it is called. + +They take the same arguments and return the same values as the functions of +:mod:`automation_file.storage.actions`, with three differences: + +* ``overwrite`` defaults to what the policy permits instead of ``True``, and an + explicit ``overwrite: true`` is refused where the policy forbids it; +* ``FA_storage_verify`` also takes the result of ``FA_storage_checksum`` as its + ``expected``, so ``"${tasks.digest.result}"`` compares two files; +* ``FA_storage_upload`` and ``FA_storage_download`` are missing: they take a + filesystem path, which is not a storage URI the policy could check. Copy to or + from a ``local://`` URI instead. +""" + +from __future__ import annotations + +from collections.abc import Callable, Mapping +from typing import Any + +from automation_file.exceptions import StorageChecksumException, StorageException +from automation_file.server.mcp_policy import ( + LIMIT_EXCEEDED, + OUTSIDE_ROOT, + MCPLocationException, + MCPPermissionException, + MCPPolicy, +) +from automation_file.server.mcp_tool_model import ToolSession +from automation_file.storage.backend import DEFAULT_CHECKSUM_ALGORITHM +from automation_file.storage.file import File +from automation_file.storage.storage import Storage +from automation_file.storage.types import Checksum, FileInfo + +_UTF8 = "utf-8" +_PREFIX = "FA_storage_" +EXISTS = f"{_PREFIX}exists" +STAT = f"{_PREFIX}stat" +LIST = f"{_PREFIX}list" +CHECKSUM = f"{_PREFIX}checksum" +VERIFY = f"{_PREFIX}verify" +READ_TEXT = f"{_PREFIX}read_text" +SCHEMES = f"{_PREFIX}schemes" +MKDIR = f"{_PREFIX}mkdir" +WRITE_TEXT = f"{_PREFIX}write_text" +COPY = f"{_PREFIX}copy" +COPY_TREE = f"{_PREFIX}copy_tree" +SYNC = f"{_PREFIX}sync" +MOVE = f"{_PREFIX}move" +DELETE = f"{_PREFIX}delete" + +#: The guarded actions by the permission each one needs. +READ_ACTIONS: frozenset[str] = frozenset({EXISTS, STAT, LIST, CHECKSUM, VERIFY, READ_TEXT, SCHEMES}) +WRITE_ACTIONS: frozenset[str] = frozenset({MKDIR, WRITE_TEXT, COPY, COPY_TREE}) +OVERWRITE_ACTIONS: frozenset[str] = frozenset({SYNC}) +DELETE_ACTIONS: frozenset[str] = frozenset({MOVE, DELETE}) +GUARDED_ACTIONS: frozenset[str] = READ_ACTIONS | WRITE_ACTIONS | OVERWRITE_ACTIONS | DELETE_ACTIONS + + +def default_pipeline_actions(policy: MCPPolicy) -> frozenset[str]: + """Return the actions a pipeline may call when the policy names none: what it permits.""" + allowed = set(READ_ACTIONS) + if policy.allow_write: + allowed |= WRITE_ACTIONS + if policy.allow_overwrite: + allowed |= OVERWRITE_ACTIONS + if policy.allow_delete: + allowed |= DELETE_ACTIONS + return frozenset(allowed) + + +def _described(info: FileInfo, uri: object) -> dict[str, Any]: + return {"uri": str(uri), **info.to_dict()} + + +def _expected_digest(expected: object) -> str | Checksum: + """Return what a file is to be compared with: digest text, or a checksum result.""" + if isinstance(expected, str): + return expected + if isinstance(expected, Mapping): + algorithm, value = expected.get("algorithm"), expected.get("value") + if isinstance(algorithm, str) and isinstance(value, str): + return Checksum(algorithm, value) + raise StorageException( + "expected must be a hex digest, ':' or the result of FA_storage_checksum" + ) + + +class GuardedStorageActions: + """The storage actions bound to one session's policy and guard.""" + + def __init__(self, session: ToolSession) -> None: + self._session = session + self._policy = session.policy + + def commands(self) -> dict[str, Callable[..., Any]]: + """Return every guarded action by its ``FA_storage_*`` name.""" + return { + EXISTS: self.exists, + STAT: self.stat, + LIST: self.list_dir, + CHECKSUM: self.checksum, + VERIFY: self.verify, + READ_TEXT: self.read_text, + SCHEMES: self.schemes, + MKDIR: self.mkdir, + WRITE_TEXT: self.write_text, + COPY: self.copy, + COPY_TREE: self.copy_tree, + SYNC: self.sync, + MOVE: self.move, + DELETE: self.delete, + } + + # ------------------------------------------------------------------ helpers + + def _file(self, uri: str) -> File: + return self._session.file(uri) + + def _storage(self, uri: str) -> Storage: + return self._session.storage(uri) + + def _replace(self, action: str, target: File, overwrite: bool | None) -> bool: + """Return the ``overwrite`` to pass on: the caller's wish, as far as the policy goes.""" + if overwrite is None: + return self._policy.allow_overwrite + if overwrite and not self._policy.allow_overwrite and target.exists(): + self._policy.require_overwrite(action) + return bool(overwrite) and self._policy.allow_overwrite + + # ------------------------------------------------------------------ reading + + def exists(self, uri: str) -> bool: + """Return whether a file or a directory is at ``uri``.""" + return self._file(uri).exists() + + def stat(self, uri: str) -> dict[str, Any]: + """Return the size, modification time and other metadata of ``uri``.""" + target = self._file(uri) + return _described(target.stat(), target) + + def list_dir(self, uri: str, recursive: bool = False) -> list[dict[str, Any]]: + """List the directory ``uri``; ``recursive`` adds every descendant.""" + directory = self._storage(uri) + return [ + _described(info, directory.uri.joinpath(info.path)) + for info in directory.list_dir(recursive=bool(recursive)) + ] + + def checksum(self, uri: str, algorithm: str = DEFAULT_CHECKSUM_ALGORITHM) -> dict[str, str]: + """Return ``{"algorithm": ..., "value": ...}`` for the content of ``uri``.""" + return self._file(uri).checksum(algorithm).to_dict() + + def verify( + self, + uri: str, + expected: str | Mapping[str, Any], + algorithm: str = DEFAULT_CHECKSUM_ALGORITHM, + strict: bool = False, + ) -> bool: + """Return whether ``uri`` has the digest ``expected``; ``strict`` raises on a mismatch. + + ``expected`` is a hex digest, ``":"``, or the mapping + ``FA_storage_checksum`` returns. + """ + target = self._file(uri) + if target.verify(_expected_digest(expected), algorithm=algorithm): + return True + if strict: + raise StorageChecksumException(f"{target} does not have the expected digest") + return False + + def read_text(self, uri: str, encoding: str = _UTF8) -> str: + """Return the content of ``uri`` as text, up to the policy's read limit.""" + source = self._file(uri) + size = source.stat().size + if size is None or size > self._policy.max_read_bytes: + raise MCPPermissionException( + f"{source} is larger than the {self._policy.max_read_bytes} bytes a task may " + "read into its result (--max-read-bytes)", + LIMIT_EXCEEDED, + ) + return source.read_text(encoding) + + def schemes(self) -> list[str]: + """Return the URI schemes a backend is registered or mounted for.""" + return self._session.guard.inner.schemes() + + # ------------------------------------------------------------------ writing + + def mkdir(self, uri: str, parents: bool = True, exist_ok: bool = True) -> bool: + """Create the directory ``uri``.""" + self._policy.require_write(MKDIR) + self._storage(uri).mkdir(parents=bool(parents), exist_ok=bool(exist_ok)) + return True + + def write_text( + self, uri: str, text: str, overwrite: bool | None = None, encoding: str = _UTF8 + ) -> dict[str, Any]: + """Store ``text`` as the content of ``uri``, up to the policy's write limit.""" + self._policy.require_write(WRITE_TEXT) + payload = str(text).encode(encoding) + if len(payload) > self._policy.max_write_bytes: + raise MCPPermissionException( + f"the text is {len(payload)} bytes and a task writes at most " + f"{self._policy.max_write_bytes} (--max-write-bytes)", + LIMIT_EXCEEDED, + ) + target = self._file(uri) + info = target.write(payload, overwrite=self._replace(WRITE_TEXT, target, overwrite)) + return _described(info, target) + + def copy(self, source: str, target: str, overwrite: bool | None = None) -> dict[str, Any]: + """Copy the file ``source`` to ``target``, in the same backend or another one.""" + self._policy.require_write(COPY) + origin, destination = self._file(source), self._file(target) + copied = origin.copy_to(destination, overwrite=self._replace(COPY, destination, overwrite)) + return _described(copied.stat(), copied) + + def copy_tree(self, source: str, target: str, overwrite: bool | None = None) -> dict[str, Any]: + """Copy every file below ``source`` to ``target``. + + A file the target already has is skipped unless overwriting is asked for + and permitted. + """ + self._policy.require_write(COPY_TREE) + if overwrite: + self._policy.require_overwrite(COPY_TREE) + replace = self._policy.allow_overwrite if overwrite is None else bool(overwrite) + return self._storage(source).copy_to(self._storage(target), overwrite=replace).to_dict() + + def sync( + self, + source: str, + target: str, + delete: bool = False, + checksum: bool = False, + dry_run: bool = False, + ) -> dict[str, Any]: + """Mirror ``source`` into ``target``. + + It replaces changed files, and ``delete`` removes what the source does not + have, so it needs those permissions. + """ + self._policy.require_overwrite(SYNC) + if delete: + self._policy.require_delete(SYNC) + result = self._storage(source).sync_to( + self._storage(target), delete=bool(delete), checksum=bool(checksum), dry_run=dry_run + ) + return result.to_dict() + + # ------------------------------------------------------------------ deleting + + def move(self, source: str, target: str, overwrite: bool | None = None) -> dict[str, Any]: + """Move the file ``source`` to ``target``; the source is deleted.""" + self._policy.require_delete(MOVE) + origin, destination = self._file(source), self._file(target) + moved = origin.move_to(destination, overwrite=self._replace(MOVE, destination, overwrite)) + return _described(moved.stat(), moved) + + def delete(self, uri: str, recursive: bool = False, missing_ok: bool = False) -> bool: + """Remove the file or directory at ``uri``. An allowed root itself is never removed.""" + self._policy.require_delete(DELETE) + target = self._storage(uri) + if self._policy.is_root(target.uri): + raise MCPLocationException( + f"{target} is an allowed location itself and is not deleted", OUTSIDE_ROOT + ) + target.delete("", recursive=bool(recursive), missing_ok=bool(missing_ok)) + return True diff --git a/automation_file/server/mcp_pipeline_tools.py b/automation_file/server/mcp_pipeline_tools.py new file mode 100644 index 0000000..3425265 --- /dev/null +++ b/automation_file/server/mcp_pipeline_tools.py @@ -0,0 +1,446 @@ +"""The semantic pipeline tools: ``pipeline_create``, ``pipeline_run`` and ``pipeline_status``. + +``pipeline_create`` validates a definition and stores it under a name; +``pipeline_run`` runs a stored definition, or plans it with ``dry_run``; +``pipeline_status`` reports recorded runs. + +Definitions are JSON files ``.json`` in the policy's ``pipeline_dir``, a +storage URI or a local directory. Without one they are kept in memory and are +gone when the server stops. + +What a definition may call is checked twice, when it is created and again +when it is run, because the file may have been written by something else in +between. A run gets a registry of its own that holds only the allowed actions, +with the ``FA_storage_*`` ones replaced by their guarded versions +(:mod:`automation_file.server.mcp_pipeline_actions`), so the locations a task +touches are checked when the task runs. +""" + +from __future__ import annotations + +import json +import re +from collections.abc import Mapping +from typing import Any + +from automation_file.core.action_executor import executor +from automation_file.core.action_registry import ActionRegistry +from automation_file.exceptions import MCPServerException, StorageNotFoundException +from automation_file.pipeline.definition import validate_definition +from automation_file.pipeline.errors import PipelineDefinitionException +from automation_file.pipeline.model import PipelineRun, RunStatus +from automation_file.pipeline.pipeline import Pipeline +from automation_file.pipeline.store import default_run_store +from automation_file.server.action_acl import nested_action_names +from automation_file.server.mcp_pipeline_actions import ( + GUARDED_ACTIONS, + GuardedStorageActions, + default_pipeline_actions, +) +from automation_file.server.mcp_policy import ( + ACTION_NOT_ALLOWED, + ALREADY_EXISTS, + LIMIT_EXCEEDED, + NOT_FOUND, + MCPPermissionException, + MCPToolException, +) +from automation_file.server.mcp_tool_model import ( + DRY_RUN_HELP, + SemanticTool, + ToolSession, + arguments_schema, + flag, + text, + whole, +) +from automation_file.storage.backend import StorageBackend, join_path +from automation_file.storage.local_storage import LocalStorage +from automation_file.storage.memory_storage import MemoryStorage + +PIPELINE_CREATE = "pipeline_create" +PIPELINE_RUN = "pipeline_run" +PIPELINE_STATUS = "pipeline_status" +_NAME = re.compile(r"[A-Za-z0-9][A-Za-z0-9._-]{0,99}") +_NAME_RULE = ( + "a pipeline name is 1 to 100 characters: letters, digits, '.', '_' and '-', " + "starting with a letter or a digit" +) +_SUFFIX = ".json" +_STATE_KEY = "pipeline_definitions" +_MEMORY_LOCATION = "memory (kept until the server stops)" +_DEFAULT_HISTORY = 5 +_LISTED_NAMES = 20 + + +def _unique_keys(pairs: list[tuple[str, Any]]) -> dict[str, Any]: + built: dict[str, Any] = {} + for key, value in pairs: + if key in built: + raise ValueError(f"duplicate key {key!r}") + built[key] = value + return built + + +class DefinitionStore: + """Named pipeline definitions, one JSON file each, in one storage directory.""" + + def __init__( + self, backend: StorageBackend, base: str = "", location: str | None = None + ) -> None: + self._backend = backend + self._base = base + self._location = location + + @classmethod + def for_session(cls, session: ToolSession) -> DefinitionStore: + """Return the store the policy describes; a local directory is confined to itself.""" + directory = session.policy.pipeline_dir + if directory is None: + return cls(MemoryStorage("mcp-pipelines")) + backend, path = session.guard.inner.resolve(directory) + if isinstance(backend, LocalStorage): + return cls(LocalStorage(backend.local_path(path)), "", str(directory)) + return cls(backend, path, str(directory)) + + @property + def location(self) -> str: + """Where the definitions are kept, for reports.""" + return _MEMORY_LOCATION if self._location is None else self._location + + @property + def persistent(self) -> bool: + return self._location is not None + + def _path(self, name: str) -> str: + return join_path(self._base, f"{name}{_SUFFIX}") + + def exists(self, name: str) -> bool: + return self._backend.exists(self._path(name)) + + def names(self) -> list[str]: + """Return the stored names, sorted.""" + try: + listing = self._backend.list_dir(self._base) + except StorageNotFoundException: + return [] + return sorted( + info.name[: -len(_SUFFIX)] + for info in listing + if not info.is_dir and info.name.endswith(_SUFFIX) + ) + + def save(self, name: str, data: bytes, *, overwrite: bool) -> None: + """Store the JSON text ``data`` as the definition ``name``.""" + self._backend.mkdir(self._base) + self._backend.write_bytes(self._path(name), data, overwrite=overwrite) + + def load(self, name: str, max_bytes: int) -> dict[str, Any]: + """Return the definition ``name`` as parsed, without validating it.""" + path = self._path(name) + try: + size = self._backend.stat(path).size + except StorageNotFoundException as error: + stored = ", ".join(self.names()[:_LISTED_NAMES]) or "none" + raise MCPToolException( + f"no pipeline named {name!r} is stored (stored: {stored})", NOT_FOUND + ) from error + if size is not None and size > max_bytes: + raise MCPPermissionException( + f"the definition of {name!r} is larger than {max_bytes} bytes (--max-write-bytes)", + LIMIT_EXCEEDED, + ) + try: + document = json.loads( + self._backend.read_bytes(path).decode("utf-8"), object_pairs_hook=_unique_keys + ) + except (ValueError, RecursionError) as error: + raise MCPToolException( + f"the stored definition of {name!r} is not valid JSON" + ) from error + if not isinstance(document, dict): + raise MCPToolException(f"the stored definition of {name!r} is not a mapping") + return document + + +def definitions(session: ToolSession) -> DefinitionStore: + """Return the session's definition store, creating it on first use.""" + store = session.state.get(_STATE_KEY) + if store is None: + store = session.state[_STATE_KEY] = DefinitionStore.for_session(session) + return store + + +# ---------------------------------------------------------------------- what a pipeline may call + + +def allowed_actions(session: ToolSession) -> frozenset[str]: + """Return the actions a pipeline may name: the policy's list, or its permissions' default.""" + explicit = session.policy.pipeline_actions + if explicit is None: + return default_pipeline_actions(session.policy) + return frozenset(explicit) + + +def check_pipeline_actions(session: ToolSession) -> None: + """Refuse a policy whose pipeline allow list names an action the server does not have.""" + missing = sorted( + name + for name in allowed_actions(session) + if name not in GUARDED_ACTIONS and session.registry.resolve(name) is None + ) + if missing: + raise MCPServerException( + "the pipeline allow list names action(s) this server does not expose: " + + ", ".join(missing) + ) + + +def pipeline_registry(session: ToolSession) -> ActionRegistry: + """Return the registry a pipeline runs with: the allowed actions and nothing else.""" + guarded = GuardedStorageActions(session).commands() + registry = ActionRegistry() + for name in sorted(allowed_actions(session)): + command = guarded.get(name) or session.registry.resolve(name) + if command is not None: + registry.register(name, command) + return registry + + +def action_refusals( + session: ToolSession, document: Mapping[str, Any], params: Mapping[str, Any] | None +) -> list[str]: + """Return what a valid definition (and the parameters of a run) may not call. + + An action named inside the arguments of another one would be run by the + shared executor, outside the policy. It is accepted only when the policy + lists it by name and it is not one of the guarded storage actions. + """ + allowed = allowed_actions(session) + nestable = frozenset(session.policy.pipeline_actions or ()) - GUARDED_ACTIONS + known = {*executor.registry.event_dict, *session.registry.event_dict, *GUARDED_ACTIONS} + refusals: list[str] = [] + for task_id, spec in document["tasks"].items(): + action = spec["action"] + if action[0] not in allowed: + refusals.append(f"tasks.{task_id}.action[0]: {action[0]} is not allowed in a pipeline") + refusals.extend( + f"tasks.{task_id}.action: its arguments name the action {name}" + for name in sorted(set(nested_action_names(action[1:], known)) - nestable) + ) + given = [document.get("params"), dict(params or {})] + refusals.extend( + f"params: a parameter names the action {name}" + for name in sorted(set(nested_action_names(given, known)) - nestable) + ) + return refusals + + +def _require_allowed( + session: ToolSession, document: Mapping[str, Any], params: Mapping[str, Any] | None +) -> None: + refusals = action_refusals(session, document, params) + if refusals: + allowed = ", ".join(sorted(allowed_actions(session))) or "none" + raise MCPPermissionException( + f"{'; '.join(refusals)}. A pipeline run through this server may call: {allowed}", + ACTION_NOT_ALLOWED, + ) + + +def _checked_name(name: str) -> str: + if _NAME.fullmatch(name) is None: + raise MCPToolException(_NAME_RULE) + return name + + +def _valid_document(document: Mapping[str, Any], name: str) -> None: + """Raise unless ``document`` is a valid definition that names itself ``name``.""" + problems = validate_definition(document) + if problems: + raise PipelineDefinitionException(problems) + if document["name"] != name: + raise MCPToolException( + f"the definition names itself {str(document['name'])[:100]!r}; its name must be " + f"{name!r}, the name it is stored under" + ) + + +def run_view(session: ToolSession, run: PipelineRun) -> dict[str, Any]: + """Return a run as JSON data, with a task result left out when it is too large to return.""" + limit = session.policy.max_read_bytes + document = run.to_dict() + for state in document["tasks"].values(): + size = len(json.dumps(state["result"], default=str)) + if size > limit: + state["result"] = None + state["result_omitted"] = f"{size} bytes, more than the {limit} this server returns" + return document + + +# ---------------------------------------------------------------------- the tools + + +def pipeline_create(session: ToolSession, args: dict[str, Any]) -> dict[str, Any]: + """Validate a definition and store it under ``name`` for ``pipeline_run``.""" + policy = session.policy + policy.require_write(PIPELINE_CREATE) + name = _checked_name(args["name"]) + document = {"name": name, **args["definition"]} + _valid_document(document, name) + if "schedule" in document: + raise MCPPermissionException( + "a pipeline created through MCP cannot carry a schedule: when a pipeline runs " + "by itself is decided by whoever operates the server", + ACTION_NOT_ALLOWED, + ) + _require_allowed(session, document, None) + data = json.dumps(document, indent=2, ensure_ascii=False).encode("utf-8") + if len(data) > policy.max_write_bytes: + raise MCPPermissionException( + f"the definition is {len(data)} bytes and this server stores at most " + f"{policy.max_write_bytes} (--max-write-bytes)", + LIMIT_EXCEEDED, + ) + store = definitions(session) + replaces = store.exists(name) + if replaces and not args["overwrite"]: + raise MCPToolException( + f"a pipeline named {name!r} is already stored; pass overwrite=true to replace it", + ALREADY_EXISTS, + ) + if replaces: + policy.require_overwrite(PIPELINE_CREATE) + if not args["dry_run"]: + store.save(name, data, overwrite=replaces) + return { + "name": name, + "location": store.location, + "persistent": store.persistent, + "tasks": list(document["tasks"]), + "actions": sorted({spec["action"][0] for spec in document["tasks"].values()}), + "overwrites": replaces, + "dry_run": args["dry_run"], + "stored": not args["dry_run"], + } + + +def pipeline_run(session: ToolSession, args: dict[str, Any]) -> dict[str, Any]: + """Run a stored definition; ``dry_run`` plans it, ``background`` returns at once.""" + policy = session.policy + policy.require_write(PIPELINE_RUN) + name = _checked_name(args["name"]) + params = args.get("params") or {} + document = definitions(session).load(name, policy.max_write_bytes) + _valid_document(document, name) + _require_allowed(session, document, params) + pipeline = Pipeline.from_dict(document, registry=pipeline_registry(session)) + if args["dry_run"]: + run = pipeline.run(params, dry_run=True) + elif args["background"]: + run = pipeline.start(params) + else: + run = pipeline.run(params) + return { + "name": name, + "run_id": run.run_id, + "status": run.status.value, + "dry_run": args["dry_run"], + "background": bool(args["background"]) and not args["dry_run"], + "ok": run.status in (RunStatus.SUCCEEDED, RunStatus.RUNNING), + "run": run_view(session, run), + } + + +def pipeline_status(session: ToolSession, args: dict[str, Any]) -> dict[str, Any]: + """Report one recorded run by its ID, or the latest runs of one pipeline or of all.""" + store = default_run_store() + run_id = args.get("run_id") + if run_id is None: + limit = session.policy.clamp_results(args.get("limit", _DEFAULT_HISTORY)) + runs = store.list_runs(args.get("name"), limit) + else: + found = store.get_run(run_id) + if found is None: + raise MCPToolException( + f"pipeline_status: no run {run_id[:64]!r} is recorded", NOT_FOUND + ) + runs = [found] + return {"runs": [run_view(session, run) for run in runs], "count": len(runs)} + + +_NAME_ARGUMENT = text(f"The name the definition is stored under: {_NAME_RULE}.", minLength=1) +_DEFINITION_HELP = ( + 'A pipeline definition, schema_version 1: {"schema_version": 1, "tasks": {"": ' + '{"action": ["FA_storage_copy", {"source": "s3://in/${params.date}.csv", "target": ' + '"sftp://host/in/${params.date}.csv"}], "depends_on": [""], "retry": ' + '{"max_attempts": 3, "backoff": 2}, "timeout": 300, "when": "on_success"}}}. ' + "Optional top-level keys: description, max_workers, params (defaults). An action is " + "[name], [name, {arguments}] or [name, [arguments]]; ${params.} and " + "${tasks..result} are filled in when the task runs. Only the actions this server " + "allows may be named, and the storage actions stay inside the allowed locations." +) + +TOOLS: tuple[SemanticTool, ...] = ( + SemanticTool( + PIPELINE_CREATE, + "Validate a pipeline definition and store it under a name, where pipeline_run " + "finds it. Nothing is run. Needs a server that allows writing. Every problem of " + "the definition is reported with the path of the entry it is about.", + arguments_schema( + { + "name": _NAME_ARGUMENT, + "definition": {"type": "object", "description": _DEFINITION_HELP}, + "overwrite": flag( + "Replace a definition already stored under this name. The server must " + "allow overwriting as well." + ), + "dry_run": flag("When true, validate and report, but store nothing."), + }, + required=("name", "definition"), + ), + pipeline_create, + changes=True, + ), + SemanticTool( + PIPELINE_RUN, + "Run a pipeline stored with pipeline_create and return the run: its run_id, its " + "status and every task with its result or error. With dry_run=true nothing is " + "executed and the tasks come back 'planned', in order. Needs a server that allows " + "writing. The run_id is also the correlation ID of everything the run does.", + arguments_schema( + { + "name": _NAME_ARGUMENT, + "params": { + "type": "object", + "description": "Parameters of this run; they fill ${params.} and " + "override the definition's defaults. Do not put secrets here: they are " + "recorded with the run.", + }, + "dry_run": flag(DRY_RUN_HELP), + "background": flag( + "Return at once with status 'running' and poll pipeline_status with the " + "run_id. Use it for a run that takes long." + ), + }, + required=("name",), + ), + pipeline_run, + changes=True, + ), + SemanticTool( + PIPELINE_STATUS, + "Report pipeline runs: one run by its run_id, or the latest recorded runs of one " + "pipeline (name) or of all. Each run has its status and the state of every task.", + arguments_schema( + { + "run_id": text("The run to report, as pipeline_run returned it.", minLength=1), + "name": text("Without run_id: only the runs of this pipeline.", minLength=1), + "limit": whole( + f"Without run_id: how many runs, newest first. Default {_DEFAULT_HISTORY}." + ), + } + ), + pipeline_status, + ), +) diff --git a/automation_file/server/mcp_policy.py b/automation_file/server/mcp_policy.py new file mode 100644 index 0000000..5088836 --- /dev/null +++ b/automation_file/server/mcp_policy.py @@ -0,0 +1,448 @@ +"""The permission model of the semantic MCP tools. + +An :class:`MCPPolicy` is built once, when the server starts, and never changes. +It says: + +* **where** a tool may work: the allowed roots, as storage URIs. Without a root + every tool that touches storage refuses. +* **what** it may do there: reading only, unless writing is switched on; + replacing an existing file and deleting need a permission of their own. +* **how much**: the bytes ``file_read`` returns, the entries a listing returns, + the size of a file written from a tool call, the bytes a content search reads. +* **which actions** a pipeline created or run through MCP may call, and which + of the fourteen tools are offered at all. + +:class:`StorageGuard` is the resolver the tools reach storage through. A +location inside a local root is served by a ``LocalStorage`` confined to that +root, so the storage layer's own ``safe_join`` check refuses a link or an +absolute path that leaves it. For every other backend the scheme, the authority +and the path are compared segment by segment: ``s3://bucket/team`` allows +``s3://bucket/team/a.csv`` and not ``s3://bucket/team-b/a.csv``. +""" + +from __future__ import annotations + +import os +import threading +from collections.abc import Collection, Iterable +from dataclasses import dataclass +from typing import Any + +from automation_file.exceptions import ( + MCPServerException, + PathTraversalException, + StoragePermissionException, + StorageURIException, +) +from automation_file.storage.backend import StorageBackend +from automation_file.storage.local_storage import LocalStorage +from automation_file.storage.resolver import StorageResolver, default_resolver +from automation_file.storage.uri import ( + LOCAL_SCHEME, + StorageURI, + URILike, + normalize_path, + parse_storage_uri, +) + +#: The fourteen semantic tools, in the order ``tools/list`` returns them. +SEMANTIC_TOOL_NAMES: tuple[str, ...] = ( + "file_read", + "file_write", + "file_copy", + "file_move", + "file_search", + "file_checksum", + "file_verify", + "storage_list", + "storage_copy", + "pipeline_create", + "pipeline_run", + "pipeline_status", + "integrity_status", + "audit_search", +) + +DEFAULT_MAX_READ_BYTES = 256 * 1024 +DEFAULT_MAX_WRITE_BYTES = 1024 * 1024 +DEFAULT_MAX_RESULTS = 200 +DEFAULT_MAX_SEARCH_BYTES = 8 * 1024 * 1024 +DEFAULT_ACTOR = "mcp" + +#: ``MCPPermissionException.code`` values: why a call was refused. +NO_ROOT = "no_root" +OUTSIDE_ROOT = "outside_root" +READ_ONLY = "read_only" +OVERWRITE_NOT_ALLOWED = "overwrite_not_allowed" +DELETE_NOT_ALLOWED = "delete_not_allowed" +TOOL_DISABLED = "tool_disabled" +ACTION_NOT_ALLOWED = "action_not_allowed" +LIMIT_EXCEEDED = "limit_exceeded" + +#: ``MCPToolException.kind`` values: why a call could not be carried out. +INVALID_ARGUMENTS = "invalid_arguments" +NOT_FOUND = "not_found" +ALREADY_EXISTS = "already_exists" +NOT_CONFIGURED = "not_configured" + +_WINDOWS = os.sep == "\\" +_DEVICE_NUMBERS = "123456789" +_WINDOWS_DEVICES = frozenset( + { + "CON", + "PRN", + "AUX", + "NUL", + "CONIN$", + "CONOUT$", + *(f"COM{number}" for number in _DEVICE_NUMBERS), + *(f"LPT{number}" for number in _DEVICE_NUMBERS), + } +) +_LIMITS = ("max_read_bytes", "max_write_bytes", "max_results", "max_search_bytes") +_HOW_TO_ADD_A_ROOT = ( + "start the server with --root (repeatable), " + "or pass MCPPolicy(roots=[...])" +) + + +class MCPPermissionException(MCPServerException): + """Raised when the policy refuses a call. ``code`` names the rule that refused it.""" + + def __init__(self, message: str, code: str) -> None: + super().__init__(message) + self.code = code + + +class MCPLocationException(MCPPermissionException, StoragePermissionException): + """Raised when a storage location is outside what the policy allows. + + It is a storage error as well, so a tree copy records it for the one file it + concerns and carries on with the others. + """ + + +class MCPToolException(MCPServerException): + """Raised when a semantic tool cannot do what it was asked. ``kind`` says why.""" + + def __init__(self, message: str, kind: str = INVALID_ARGUMENTS) -> None: + super().__init__(message) + self.kind = kind + + +def _root_uri(value: URILike) -> StorageURI: + try: + return parse_storage_uri(value) + except StorageURIException as error: + # The value is not repeated: a URI refused for carrying credentials must not be echoed. + raise MCPServerException(f"invalid storage location in the policy: {error}") from error + + +def _as_uris(values: Any) -> tuple[StorageURI, ...]: + if values is None: + return () + if isinstance(values, (str, os.PathLike, StorageURI)): + values = (values,) + return tuple(dict.fromkeys(_root_uri(value) for value in values)) + + +def _as_names(values: Any, what: str) -> frozenset[str] | None: + if values is None: + return None + names = (values,) if isinstance(values, str) else tuple(values) + wrong = [name for name in names if not (isinstance(name, str) and name.strip())] + if wrong: + raise MCPServerException(f"{what}: expected names, got {wrong!r}") + return frozenset(name.strip() for name in names) + + +def _positive(name: str, value: object) -> int: + if isinstance(value, bool) or not isinstance(value, int) or value < 1: + raise MCPServerException(f"{name} must be an integer of 1 or more, got {value!r}") + return value + + +def _segments(uri: StorageURI) -> tuple[str, ...]: + """Return the path of ``uri`` as segments; a Windows local path may use backslashes.""" + text = uri.path + if _WINDOWS and uri.scheme == LOCAL_SCHEME: + text = normalize_path(text.replace("\\", "/")) + return tuple(text.split("/")) if text else () + + +def _comparable(uri: StorageURI, segments: tuple[str, ...]) -> tuple[str, ...]: + """Fold the case of local segments the way the filesystem compares them.""" + if uri.scheme != LOCAL_SCHEME: + return segments + return tuple(os.path.normcase(segment) for segment in segments) + + +def names_windows_device(relative: str) -> bool: + """Return whether a path below a local root is a device or a data stream to Windows. + + ``reports/CON`` is the console and ``a.txt:hidden`` an alternate data stream, + wherever they are written: neither is a file in the directory it seems to be in. + """ + for segment in relative.split("/"): + stem = segment.partition(".")[0].rstrip(" ").upper() + if ":" in segment or stem in _WINDOWS_DEVICES: + return True + return False + + +def path_below(root: StorageURI, uri: StorageURI) -> str | None: + """Return the path of ``uri`` below ``root``, or ``None`` when it is not at or below it. + + The comparison is by whole segments, so ``team`` is not above ``team-b``. + """ + if (root.scheme, root.authority) != (uri.scheme, uri.authority): + return None + base, wanted = _segments(root), _segments(uri) + if _comparable(uri, wanted)[: len(base)] != _comparable(root, base): + return None + return "/".join(wanted[len(base) :]) + + +@dataclass(frozen=True) +class MCPPolicy: + """What the semantic MCP tools may do. Immutable; the defaults allow nothing to change. + + ``roots`` are storage URIs or local directories. ``pipeline_dir`` is where + ``pipeline_create`` keeps definitions; without one they live in memory until + the server stops. ``pipeline_actions`` replaces the default set of actions a + pipeline may call (the ``FA_storage_*`` actions the permissions cover), and + ``tools`` names the semantic tools to offer (all fourteen when ``None``). + """ + + roots: Collection[URILike] = () + allow_write: bool = False + allow_overwrite: bool = False + allow_delete: bool = False + max_read_bytes: int = DEFAULT_MAX_READ_BYTES + max_write_bytes: int = DEFAULT_MAX_WRITE_BYTES + max_results: int = DEFAULT_MAX_RESULTS + max_search_bytes: int = DEFAULT_MAX_SEARCH_BYTES + pipeline_dir: URILike | None = None + pipeline_actions: Collection[str] | None = None + tools: Collection[str] | None = None + actor: str = DEFAULT_ACTOR + + def __post_init__(self) -> None: + object.__setattr__(self, "roots", _as_uris(self.roots)) + for name in _LIMITS: + _positive(name, getattr(self, name)) + if (self.allow_overwrite or self.allow_delete) and not self.allow_write: + raise MCPServerException( + "allow_overwrite and allow_delete need allow_write (--allow-write): a " + "read-only server cannot replace or delete anything" + ) + if self.pipeline_dir is not None: + object.__setattr__(self, "pipeline_dir", _root_uri(self.pipeline_dir)) + object.__setattr__( + self, "pipeline_actions", _as_names(self.pipeline_actions, "pipeline_actions") + ) + tools = _as_names(self.tools, "tools") + unknown = sorted(set(tools or ()) - set(SEMANTIC_TOOL_NAMES)) + if unknown: + raise MCPServerException( + f"unknown semantic tool(s) {unknown}; known: {list(SEMANTIC_TOOL_NAMES)}" + ) + object.__setattr__(self, "tools", tools) + if not (isinstance(self.actor, str) and self.actor.strip()): + raise MCPServerException(f"actor must be a non-empty string, got {self.actor!r}") + + # ------------------------------------------------------------------ locations + + @property + def root_uris(self) -> tuple[StorageURI, ...]: + """The allowed roots as parsed storage URIs.""" + return tuple(parse_storage_uri(root) for root in self.roots) + + def candidates(self, uri: StorageURI) -> list[tuple[StorageURI, str]]: + """Return every allowed root at or above ``uri`` with the path below it, outermost first. + + Raises :class:`MCPLocationException` when no root is configured or none + contains ``uri``. + """ + roots = self.root_uris + if not roots: + raise MCPLocationException( + f"no storage location is allowed on this server: {_HOW_TO_ADD_A_ROOT}", NO_ROOT + ) + found: list[tuple[StorageURI, str]] = [] + for root in roots: + relative = path_below(root, uri) + if relative is not None: + found.append((root, relative)) + if not found: + allowed = ", ".join(str(root) for root in roots) + raise MCPLocationException( + f"{uri} is outside the allowed locations ({allowed})", OUTSIDE_ROOT + ) + return sorted(found, key=lambda entry: len(entry[0].path)) + + def is_root(self, uri: StorageURI) -> bool: + """Return whether ``uri`` is one of the allowed roots itself.""" + return any(path_below(root, uri) == "" for root in self.root_uris) + + # ------------------------------------------------------------------ permissions + + def enabled_tools(self) -> tuple[str, ...]: + """Return the semantic tools this policy offers, in catalogue order.""" + if self.tools is None: + return SEMANTIC_TOOL_NAMES + return tuple(name for name in SEMANTIC_TOOL_NAMES if name in self.tools) + + def require_tool(self, tool: str) -> None: + if tool not in self.enabled_tools(): + raise MCPPermissionException(f"{tool} is switched off on this server", TOOL_DISABLED) + + def require_write(self, tool: str) -> None: + if not self.allow_write: + raise MCPPermissionException( + f"{tool} changes something and this server is read-only: start it with " + "--allow-write, or pass MCPPolicy(allow_write=True)", + READ_ONLY, + ) + + def require_overwrite(self, tool: str) -> None: + self.require_write(tool) + if not self.allow_overwrite: + raise MCPPermissionException( + f"{tool} would replace an existing file and this server does not allow it: " + "start it with --allow-overwrite, or pass MCPPolicy(allow_overwrite=True)", + OVERWRITE_NOT_ALLOWED, + ) + + def require_delete(self, tool: str) -> None: + self.require_write(tool) + if not self.allow_delete: + raise MCPPermissionException( + f"{tool} deletes something and this server does not allow it: start it " + "with --allow-delete, or pass MCPPolicy(allow_delete=True)", + DELETE_NOT_ALLOWED, + ) + + def clamp_results(self, wanted: int | None) -> int: + """Return how many entries a call may return: ``wanted``, at most ``max_results``.""" + return self.max_results if wanted is None else min(wanted, self.max_results) + + # ------------------------------------------------------------------ description + + def describe(self) -> dict[str, Any]: + """Return the policy as a JSON-friendly mapping.""" + actions = self.pipeline_actions + return { + "roots": [str(root) for root in self.root_uris], + "allow_write": self.allow_write, + "allow_overwrite": self.allow_overwrite, + "allow_delete": self.allow_delete, + "max_read_bytes": self.max_read_bytes, + "max_write_bytes": self.max_write_bytes, + "max_results": self.max_results, + "max_search_bytes": self.max_search_bytes, + "pipeline_dir": None if self.pipeline_dir is None else str(self.pipeline_dir), + "pipeline_actions": None if actions is None else sorted(actions), + "tools": list(self.enabled_tools()), + "actor": self.actor, + } + + def summary(self) -> str: + """Return the policy in a few sentences, for the ``instructions`` of the handshake.""" + roots = ", ".join(str(root) for root in self.root_uris) + switches = ( + ("writing", self.allow_write), + ("replacing existing files", self.allow_overwrite), + ("deleting (a move deletes its source)", self.allow_delete), + ) + allowed = [name for name, enabled in switches if enabled] + refused = [name for name, enabled in switches if not enabled] + lines = [ + "Semantic tools take storage URIs (:///) or absolute " + "local paths.", + f"Allowed locations: {roots}." if roots else "No location is allowed yet.", + f"Allowed: reading{''.join(f', {name}' for name in allowed)}.", + ] + if refused: + lines.append(f"Refused: {', '.join(refused)}.") + lines.append( + "Tools that change something take dry_run=true to report what they would do. " + "Every result carries a correlation_id that audit_search accepts." + ) + return " ".join(lines) + + +class StorageGuard(StorageResolver): + """A resolver that serves only the locations a policy allows.""" + + def __init__(self, policy: MCPPolicy, inner: StorageResolver | None = None) -> None: + super().__init__(defaults=False) + self._policy = policy + self._inner = inner if inner is not None else default_resolver + self._confined: dict[StorageURI, LocalStorage] = {} + self._confined_lock = threading.Lock() + + @property + def policy(self) -> MCPPolicy: + return self._policy + + @property + def inner(self) -> StorageResolver: + """The resolver that knows the backends; the guard only decides what may be asked.""" + return self._inner + + def resolve(self, uri: URILike) -> tuple[StorageBackend, str]: + """Return the backend and path of ``uri``, or raise :class:`MCPLocationException`.""" + parsed = parse_storage_uri(uri) + candidates = self._policy.candidates(parsed) + backend, path = self._inner.resolve(parsed) + if not isinstance(backend, LocalStorage): + return backend, path + for root, relative in candidates: + try: + return self._confine(root, relative, backend, path) + except PathTraversalException: + continue + raise MCPLocationException( + f"{parsed} leaves the allowed location through a link or an absolute path", + OUTSIDE_ROOT, + ) + + def _confine( + self, root: StorageURI, relative: str, backend: LocalStorage, path: str + ) -> tuple[StorageBackend, str]: + """Serve a local location from a backend confined to ``root``. + + A mount of its own below the root, and a root that is the backend's own + root, are already confined by the backend that serves them. + """ + root_backend, root_path = self._inner.resolve(root) + if ( + root_path + and isinstance(root_backend, LocalStorage) + and root_backend.root == backend.root + ): + backend = self._confined_backend(root, root_backend.local_path(root_path)) + path = relative + if backend.root is None: + return backend, path # the root is the whole filesystem: nothing to confine to + if _WINDOWS and names_windows_device(path): + raise MCPLocationException( + f"{root.joinpath(relative)} names a Windows device or a data stream, not a " + "file in the allowed location", + OUTSIDE_ROOT, + ) + backend.local_path(path) + return backend, path + + def _confined_backend(self, root: StorageURI, directory: os.PathLike[str]) -> LocalStorage: + with self._confined_lock: + backend = self._confined.get(root) + if backend is None: + backend = self._confined[root] = LocalStorage(directory) + return backend + + +def names_from(values: Iterable[str]) -> frozenset[str]: + """Return the non-empty, stripped names of ``values`` (for comma-separated flags).""" + return frozenset(name.strip() for name in values if name.strip()) diff --git a/automation_file/server/mcp_report_tools.py b/automation_file/server/mcp_report_tools.py new file mode 100644 index 0000000..a7cd2fb --- /dev/null +++ b/automation_file/server/mcp_report_tools.py @@ -0,0 +1,128 @@ +"""The semantic reporting tools: ``integrity_status`` and ``audit_search``. + +Both read what the process already knows and change nothing: the named +integrity monitors with their last result, and the audit trail. Neither is +bound to the policy's roots, because neither touches a storage location; a +deployment whose client must not see them leaves them out of the policy's +``tools``. +""" + +from __future__ import annotations + +from typing import Any + +from automation_file.audit.trail import audit_trail +from automation_file.integrity.actions import integrity_status as monitor_statuses +from automation_file.integrity.errors import IntegrityException +from automation_file.server.mcp_policy import NOT_CONFIGURED, NOT_FOUND, MCPToolException +from automation_file.server.mcp_tool_model import ( + SemanticTool, + ToolSession, + arguments_schema, + text, + whole, +) + +INTEGRITY_STATUS = "integrity_status" +AUDIT_SEARCH = "audit_search" +_DEFAULT_AUDIT_LIMIT = 50 +_PAGING = ("limit", "offset") +_EXACT_FILTERS = { + "actor": "Who did it: a user, 'scheduler', or the 'mcp' actor of this server's own calls.", + "source": "The component that reported: storage, pipeline, mcp, integrity, scheduler, notify.", + "pipeline": "The pipeline name.", + "task": "The task ID inside a pipeline.", + "action": "What happened: a storage operation (upload, download, read, delete, mkdir, " + "copy, move) or an event type (pipeline.failed, task.failed, mcp.tool.completed).", + "backend": "The storage scheme: s3, sftp, local, ...", + "status": "The result: ok, warning, error, refused, or the word the emitter chose.", + "correlation_id": "Everything one call or one pipeline run did: the correlation_id of a " + "tool result, or the run_id of a pipeline run.", +} + + +def _trimmed(status: dict[str, Any], limit: int) -> dict[str, Any]: + """Cut the change list of a monitor's last report down to ``limit`` entries.""" + report = status.get("last_report") + if not report or len(report["changes"]) <= limit: + return status + shortened = {**report, "changes": report["changes"][:limit], "changes_truncated": True} + return {**status, "last_report": shortened} + + +def integrity_status(session: ToolSession, args: dict[str, Any]) -> dict[str, Any]: + """Report the named integrity monitors: running or not, the last run and its report.""" + limit = session.policy.max_results + try: + statuses = monitor_statuses(args.get("name")) + except IntegrityException as error: + raise MCPToolException(f"integrity_status: {error}", NOT_FOUND) from error + monitors = [_trimmed(status, limit) for status in statuses] + return {"monitors": monitors, "count": len(monitors)} + + +def audit_search(session: ToolSession, args: dict[str, Any]) -> dict[str, Any]: + """Search the audit trail, newest first, with the number of records capped.""" + if audit_trail.store is None: + raise MCPToolException( + "this server keeps no audit trail: whoever operates it has to call " + "configure_audit() before it starts serving", + NOT_CONFIGURED, + ) + limit = session.policy.clamp_results(args.get("limit", _DEFAULT_AUDIT_LIMIT)) + offset = args["offset"] + filters = {name: value for name, value in args.items() if name not in _PAGING} + records = audit_trail.search(**filters, limit=limit, offset=offset) + total = audit_trail.count(**filters) + return { + "records": [record.to_dict() for record in records], + "count": len(records), + "total": total, + "limit": limit, + "offset": offset, + "truncated": offset + len(records) < total, + } + + +TOOLS: tuple[SemanticTool, ...] = ( + SemanticTool( + INTEGRITY_STATUS, + "Report the integrity monitors of this server: what each one watches, whether it " + "is running, when it last ran and what it found (the changed files by kind). " + "Read-only; it starts no verification.", + arguments_schema({"name": text("Only this monitor. Default: every monitor.", minLength=1)}), + integrity_status, + ), + SemanticTool( + AUDIT_SEARCH, + "Search the audit trail: who did what, when, against which resource, with what " + "result. Records come newest first and their number is capped: page with offset " + "while 'truncated' is true. Pass the correlation_id of a tool result to see what " + "that call did. Fails when the server keeps no audit trail.", + arguments_schema( + { + **{ + name: text(description, minLength=1) + for name, description in _EXACT_FILTERS.items() + }, + "resource_prefix": text( + "Records whose resource (a storage URI) starts with this text.", minLength=1 + ), + "text": text( + "Records that contain this text in any field or in their metadata.", + minLength=1, + ), + "since": text( + "Earliest time, included: ISO 8601 with an offset, 2026-10-08T00:00:00+00:00.", + minLength=1, + ), + "until": text("Latest time, excluded: ISO 8601 with an offset.", minLength=1), + "limit": whole( + f"How many records. Default {_DEFAULT_AUDIT_LIMIT}; the server caps it." + ), + "offset": whole("How many records to skip. Default 0.", minimum=0, default=0), + } + ), + audit_search, + ), +) diff --git a/automation_file/server/mcp_server.py b/automation_file/server/mcp_server.py index d4951a2..2ee7870 100644 --- a/automation_file/server/mcp_server.py +++ b/automation_file/server/mcp_server.py @@ -1,20 +1,28 @@ -"""Model Context Protocol (MCP) server bridge. +"""Model Context Protocol (MCP) server. -Exposes every :class:`~automation_file.core.action_registry.ActionRegistry` -entry as an MCP tool over JSON-RPC 2.0. The default transport is stdio — -one JSON message per line — because that's what MCP host implementations -(Claude Desktop, MCP CLIs) consume today. +Speaks JSON-RPC 2.0 over stdio, one JSON message per line, which is what MCP +hosts (Claude Desktop, Claude Code, MCP CLIs) consume. It offers two sets of +tools: + +* the **semantic tools** (:mod:`automation_file.server.mcp_tools`): fourteen + stable, task-shaped names such as ``file_read`` and ``pipeline_run``, bound to + an :class:`~automation_file.server.mcp_policy.MCPPolicy` that says where they + may work and what they may change; +* the **bridge**: every entry of an + :class:`~automation_file.core.action_registry.ActionRegistry` as a tool of its + own (``FA_*``), with a schema derived from its signature. It is on by default + and ``bridge=False`` leaves only the semantic tools. Scope ----- * ``initialize`` — handshake, returns ``serverInfo`` + capabilities * ``notifications/initialized`` — acknowledged as a no-op -* ``tools/list`` — lists registered actions as MCP tools -* ``tools/call`` — dispatches through the action registry +* ``tools/list`` — the semantic tools, then the registered actions +* ``tools/call`` — runs a semantic tool, or dispatches through the registry -Errors surface as JSON-RPC error objects with a ``MCPServerException`` chain -in the data field, so hosts can render them without having to parse the -exception string. +A protocol error, an unknown tool and a bridge call that fails surface as +JSON-RPC error objects. A semantic tool that is refused or fails answers with a +result whose ``isError`` is true and whose text says why, so the model sees it. """ from __future__ import annotations @@ -28,9 +36,11 @@ from automation_file.core.action_executor import executor from automation_file.core.action_registry import ActionRegistry -from automation_file.exceptions import MCPServerException +from automation_file.exceptions import MCPServerException, StorageURIException from automation_file.logging_config import file_automation_logger from automation_file.server.action_acl import nested_action_names +from automation_file.server.mcp_policy import MCPPolicy, names_from +from automation_file.server.mcp_tools import SemanticToolkit _JSONRPC_VERSION = "2.0" _PROTOCOL_VERSION = "2024-11-05" @@ -41,9 +51,25 @@ _INVALID_PARAMS = -32602 _INTERNAL_ERROR = -32603 +_NO_TOOLS = "none" +#: CLI destination -> ``MCPPolicy`` field, for the flags that carry one value. +_POLICY_VALUES = ( + "max_read_bytes", + "max_write_bytes", + "max_results", + "max_search_bytes", + "pipeline_dir", +) +_POLICY_SWITCHES = ("allow_write", "allow_overwrite", "allow_delete") + class MCPServer: - """Bridge between an MCP host and an :class:`ActionRegistry`.""" + """An MCP server over an :class:`ActionRegistry` and an :class:`MCPPolicy`. + + ``policy`` governs the semantic tools; without one they are read-only and no + location is allowed. ``bridge=False`` stops offering the registry's actions + as tools. + """ def __init__( self, @@ -51,12 +77,21 @@ def __init__( *, name: str = "automation_file", version: str = "1.0.0", + policy: MCPPolicy | None = None, + bridge: bool = True, ) -> None: self._registry = registry if registry is not None else executor.registry self._name = name self._version = version + self._bridge = bridge + self._toolkit = SemanticToolkit(policy, self._registry) self._initialized = False + @property + def toolkit(self) -> SemanticToolkit: + """The semantic tools of this server, bound to its policy.""" + return self._toolkit + def handle_message(self, message: dict[str, Any]) -> dict[str, Any] | None: """Dispatch a single decoded JSON-RPC message. @@ -118,15 +153,26 @@ def serve_stdio( if response is not None: self._write(writer, response) - def _handle_initialize(self, _params: dict[str, Any]) -> dict[str, Any]: - return { + def _handle_initialize(self, params: object) -> dict[str, Any]: + client = params.get("clientInfo") if isinstance(params, dict) else None + self._toolkit.set_client(client.get("name") if isinstance(client, dict) else None) + result: dict[str, Any] = { "protocolVersion": _PROTOCOL_VERSION, "capabilities": {"tools": {"listChanged": False}}, "serverInfo": {"name": self._name, "version": self._version}, } + if self._toolkit.policy.enabled_tools(): + result["instructions"] = self._toolkit.instructions() + return result def _handle_tools_list(self) -> dict[str, Any]: - return {"tools": list(_catalogue(self._registry))} + tools = self._toolkit.descriptors() + if self._bridge: + # A registered action that took a semantic name would be unreachable: leave it out. + tools.extend( + tool for tool in _catalogue(self._registry) if not self._toolkit.owns(tool["name"]) + ) + return {"tools": tools} def _handle_tools_call(self, params: dict[str, Any]) -> dict[str, Any]: name = params.get("name") @@ -135,7 +181,9 @@ def _handle_tools_call(self, params: dict[str, Any]) -> dict[str, Any]: raise MCPServerException("tools/call requires a string 'name'") if not isinstance(arguments, dict): raise MCPServerException("'arguments' must be an object") - command = self._registry.resolve(name) + if self._toolkit.owns(name): + return self._toolkit.call(name, arguments).to_mcp() + command = self._registry.resolve(name) if self._bridge else None if command is None: raise MCPServerException(f"unknown tool: {name}") self._require_exposed(name, arguments) @@ -264,10 +312,76 @@ def _filtered_registry(source: ActionRegistry, allowed: Sequence[str]) -> Action return filtered +def add_semantic_arguments(parser: argparse.ArgumentParser) -> None: + """Add the flags of the semantic tools to ``parser``: the policy and ``--no-bridge``. + + A command line that wraps this one (``python -m automation_file mcp``) adds + them to its own parser with this function and passes them on with + :func:`semantic_argv`, so the two never differ. + """ + group = parser.add_argument_group( + "semantic tools", + "file_read, file_write, storage_copy, pipeline_run and the others. Without --root " + "they refuse every storage location, and without --allow-write they only read.", + ) + group.add_argument( + "--root", + action="append", + default=None, + metavar="URI", + help="a location the semantic tools may work in: a storage URI or a local " + "directory (repeatable)", + ) + group.add_argument("--allow-write", action="store_true", help="let the tools create files") + group.add_argument( + "--allow-overwrite", + action="store_true", + help="let them replace a file that exists (needs --allow-write)", + ) + group.add_argument( + "--allow-delete", + action="store_true", + help="let them delete; a move deletes its source (needs --allow-write)", + ) + for flag, meaning in ( + ("--max-read-bytes", "most bytes file_read returns in one call"), + ("--max-write-bytes", "largest content file_write and pipeline_create accept"), + ("--max-results", "most entries a listing, a search or audit_search returns"), + ("--max-search-bytes", "most bytes one content search reads"), + ): + group.add_argument(flag, type=int, default=None, metavar="N", help=meaning) + group.add_argument( + "--pipeline-dir", + default=None, + metavar="URI", + help="where pipeline_create keeps definitions: a storage URI or a local directory " + "(default: in memory, until the server stops)", + ) + group.add_argument( + "--pipeline-actions", + default=None, + metavar="NAMES", + help="comma-separated actions a pipeline run through MCP may call (default: the " + "FA_storage_* actions the permissions cover)", + ) + group.add_argument( + "--tools", + default=None, + metavar="NAMES", + help=f"comma-separated semantic tools to offer, or '{_NO_TOOLS}' (default: all fourteen)", + ) + group.add_argument( + "--no-bridge", + action="store_true", + help="do not offer the registered FA_* actions as tools; only the semantic tools", + ) + + def _build_cli_parser() -> argparse.ArgumentParser: parser = argparse.ArgumentParser( prog="automation_file_mcp", - description="Expose the automation_file action registry as an MCP server over stdio.", + description="Serve the semantic tools and the automation_file action registry as " + "an MCP server over stdio.", ) parser.add_argument( "--name", default="automation_file", help="serverInfo.name reported at handshake" @@ -283,22 +397,79 @@ def _build_cli_parser() -> argparse.ArgumentParser: "'FA_list_dir,FA_file_checksum'); defaults to every registered action" ), ) + add_semantic_arguments(parser) return parser +def semantic_argv(args: argparse.Namespace) -> list[str]: + """Return the flags of :func:`add_semantic_arguments` that ``args`` holds, as arguments again. + + A flag that was not given is left out, so a command line without any of them + gives an empty list. + """ + argv: list[str] = [] + for root in args.root or (): + argv.extend(["--root", root]) + argv.extend(_flag(name) for name in _POLICY_SWITCHES if getattr(args, name)) + for name in (*_POLICY_VALUES, "pipeline_actions", "tools"): + value = getattr(args, name) + if value is not None: + argv.extend([_flag(name), str(value)]) + if args.no_bridge: + argv.append("--no-bridge") + return argv + + +def _flag(destination: str) -> str: + return "--" + destination.replace("_", "-") + + +def _policy_options(args: argparse.Namespace) -> dict[str, Any]: + """Return the ``MCPPolicy`` fields the command line set; an absent flag is left out.""" + options: dict[str, Any] = {name: True for name in _POLICY_SWITCHES if getattr(args, name)} + for name in _POLICY_VALUES: + value = getattr(args, name) + if value is not None: + options[name] = value + if args.root: + options["roots"] = tuple(args.root) + if args.pipeline_actions is not None: + options["pipeline_actions"] = names_from(args.pipeline_actions.split(",")) + if args.tools is not None: + chosen = "" if args.tools.strip().lower() == _NO_TOOLS else args.tools + options["tools"] = names_from(chosen.split(",")) + return options + + +def _server_options(args: argparse.Namespace) -> dict[str, Any]: + """Return the keyword arguments of :class:`MCPServer` that differ from its defaults.""" + options: dict[str, Any] = {} + policy = _policy_options(args) + if policy: + options["policy"] = MCPPolicy(**policy) + if args.no_bridge: + options["bridge"] = False + return options + + def _cli(argv: Sequence[str] | None = None) -> int: """Console-script entry point for the MCP stdio server.""" - args = _build_cli_parser().parse_args(argv) + parser = _build_cli_parser() + args = parser.parse_args(argv) registry = executor.registry if args.allowed_actions: names = [name.strip() for name in args.allowed_actions.split(",") if name.strip()] registry = _filtered_registry(registry, names) - server = MCPServer(registry, name=args.name, version=args.version) + try: + server = MCPServer(registry, name=args.name, version=args.version, **_server_options(args)) + except (MCPServerException, StorageURIException) as error: + parser.error(str(error)) file_automation_logger.info( - "mcp_server: serving %d tools over stdio (name=%s version=%s)", - len(registry.event_dict), + "mcp_server: serving over stdio (name=%s version=%s bridge=%s actions=%d)", args.name, args.version, + "off" if args.no_bridge else "on", + len(registry.event_dict), ) server.serve_stdio() return 0 diff --git a/automation_file/server/mcp_storage_tools.py b/automation_file/server/mcp_storage_tools.py new file mode 100644 index 0000000..1aae961 --- /dev/null +++ b/automation_file/server/mcp_storage_tools.py @@ -0,0 +1,319 @@ +"""The semantic tools that work on a directory: ``storage_list``, ``storage_copy`` and +``file_search``. + +All three are bounded. A listing and a search return at most the policy's +``max_results`` entries and say when there were more. A content search reads at +most ``max_search_bytes`` in one call: a file that does not fit in what is left +of that budget is not opened at all and is reported under ``skipped``, so the +answer never claims a file was searched when only a part of it was. +""" + +from __future__ import annotations + +import fnmatch +from dataclasses import dataclass, field +from typing import Any + +from automation_file.exceptions import FileAutomationException, StorageNotFoundException +from automation_file.server.mcp_file_tools import described, read_window, transfer +from automation_file.server.mcp_policy import MCPToolException, path_below +from automation_file.server.mcp_tool_model import ( + DRY_RUN_HELP, + MAX_RESULTS_HELP, + OVERWRITE_HELP, + URI_HELP, + SemanticTool, + ToolSession, + arguments_schema, + flag, + text, + whole, +) +from automation_file.storage.storage import Storage +from automation_file.storage.tree import copy_tree +from automation_file.storage.types import FileInfo + +FILE_SEARCH = "file_search" +STORAGE_LIST = "storage_list" +STORAGE_COPY = "storage_copy" +_MAX_PATTERN = 256 +_MAX_NEEDLE = 1024 +_SNIPPET = 200 +_BINARY_PROBE = 8192 +_ANY_NAME = "*" + + +def _files_below(directory: Storage, *, missing_ok: bool = False) -> dict[str, FileInfo]: + """Return every file below ``directory`` by its relative path.""" + try: + listing = directory.list_dir(recursive=True) + except StorageNotFoundException: + if missing_ok: + return {} + raise + return {info.path: info for info in listing if not info.is_dir} + + +def storage_list(session: ToolSession, args: dict[str, Any]) -> dict[str, Any]: + """List a directory; ``recursive`` adds every descendant.""" + directory = session.storage(args["uri"]) + listing = directory.list_dir(recursive=args["recursive"]) + limit = session.policy.clamp_results(args.get("max_results")) + entries = [described(directory.uri.joinpath(info.path), info) for info in listing[:limit]] + return { + "uri": str(directory), + "recursive": args["recursive"], + "entries": entries, + "count": len(entries), + "total": len(listing), + "truncated": len(listing) > limit, + } + + +# ---------------------------------------------------------------------- storage_copy + + +def _copy_tree(session: ToolSession, args: dict[str, Any]) -> dict[str, Any]: + """Copy every file below a directory, or plan that copy when ``dry_run`` is set.""" + source, target = session.storage(args["source"]), session.storage(args["target"]) + if ( + path_below(source.uri, target.uri) is not None + or path_below(target.uri, source.uri) is not None + ): + raise MCPToolException( + "storage_copy: the target is inside the source, or the source inside it" + ) + files = _files_below(source) + existing = _files_below(target, missing_ok=True) + collisions = sorted(set(files) & set(existing)) + overwrite = bool(args["overwrite"]) + if overwrite and collisions: + session.policy.require_overwrite(STORAGE_COPY) + wanted = [path for path in sorted(files) if overwrite or path not in existing] + limit = session.policy.max_results + body: dict[str, Any] = { + "kind": "tree", + "source": str(source), + "target": str(target), + "dry_run": args["dry_run"], + "planned": { + "copy": len(wanted), + "overwrite": len(collisions) if overwrite else 0, + "skip": 0 if overwrite else len(collisions), + "bytes": sum(files[path].size or 0 for path in wanted), + }, + "paths": wanted[:limit], + "existing": collisions[:limit], + "truncated": len(wanted) > limit or len(collisions) > limit, + "done": False, + } + if args["dry_run"]: + return body + result = copy_tree(source, target, overwrite=overwrite) + errors = dict(list(result.errors.items())[:limit]) + body.update( + done=True, + ok=result.ok, + copied=len(result.copied), + skipped=len(result.skipped), + failed=len(result.errors), + errors=errors, + ) + return body + + +def storage_copy(session: ToolSession, args: dict[str, Any]) -> dict[str, Any]: + """Copy a file, or every file below a directory, to another location.""" + session.policy.require_write(STORAGE_COPY) + if session.file(args["source"]).stat().is_dir: + return _copy_tree(session, args) + return {"kind": "file", **transfer(session, args, STORAGE_COPY)} + + +# ---------------------------------------------------------------------- file_search + + +@dataclass +class _Scan: + """The budget of one content search and what it has used up.""" + + budget: int + needle: str + case_sensitive: bool + files: int = 0 + bytes_read: int = 0 + skipped: list[dict[str, str]] = field(default_factory=list) + + @property + def remaining(self) -> int: + return self.budget - self.bytes_read + + +def _folded(value: str, case_sensitive: bool) -> str: + return value if case_sensitive else value.casefold() + + +def _name_matches(info: FileInfo, pattern: str, case_sensitive: bool) -> bool: + """Match the name, or the whole relative path when the pattern holds a ``/``.""" + subject = info.path if "/" in pattern else info.name + return fnmatch.fnmatchcase(_folded(subject, case_sensitive), _folded(pattern, case_sensitive)) + + +def _first_hit(content: str, scan: _Scan) -> dict[str, Any] | None: + """Return the first line of ``content`` that holds the needle, and how many lines do.""" + needle = _folded(scan.needle, scan.case_sensitive) + first: tuple[int, str] | None = None + lines = 0 + for number, line in enumerate(content.splitlines(), start=1): + if needle in _folded(line, scan.case_sensitive): + lines += 1 + first = first or (number, line) + if first is None: + return None + return {"line": first[0], "snippet": first[1].strip()[:_SNIPPET], "matching_lines": lines} + + +def _searched(directory: Storage, info: FileInfo, scan: _Scan) -> dict[str, Any] | None: + """Search one file. A file that cannot be searched goes to ``scan.skipped`` instead.""" + uri = str(directory.uri.joinpath(info.path)) + if info.size is None or info.size > scan.remaining: + scan.skipped.append({"uri": uri, "reason": "larger than what is left of the read budget"}) + return None + try: + with directory.file(info.path).open_read() as stream: + data = read_window(stream, 0, scan.remaining) + except FileAutomationException as error: + scan.skipped.append({"uri": uri, "reason": type(error).__name__}) + return None + scan.files += 1 + scan.bytes_read += len(data) + if b"\x00" in data[:_BINARY_PROBE]: + scan.skipped.append({"uri": uri, "reason": "not a text file"}) + return None + hit = _first_hit(data.decode("utf-8", errors="replace"), scan) + return None if hit is None else {**described(uri, info), **hit} + + +def _content_matches( + directory: Storage, candidates: list[FileInfo], scan: _Scan, limit: int +) -> tuple[list[dict[str, Any]], bool]: + """Return the matching files and whether the search stopped at ``limit``.""" + matches: list[dict[str, Any]] = [] + for position, info in enumerate(candidates): + if len(matches) >= limit: + return matches, position < len(candidates) + found = _searched(directory, info, scan) + if found is not None: + matches.append(found) + return matches, False + + +def file_search(session: ToolSession, args: dict[str, Any]) -> dict[str, Any]: + """Find files below a location by name pattern and, optionally, by a text they contain.""" + policy = session.policy + directory = session.storage(args["uri"]) + pattern, needle = args["pattern"], args.get("content") + case_sensitive = args["case_sensitive"] + limit = policy.clamp_results(args.get("max_results")) + candidates = [ + info + for info in directory.list_dir(recursive=args["recursive"]) + if not info.is_dir and _name_matches(info, pattern, case_sensitive) + ] + body: dict[str, Any] = { + "uri": str(directory), + "pattern": pattern, + "candidates": len(candidates), + } + if needle is None: + matches = [ + described(directory.uri.joinpath(info.path), info) for info in candidates[:limit] + ] + body.update(matches=matches, count=len(matches), truncated=len(candidates) > limit) + return body + scan = _Scan(policy.max_search_bytes, needle, case_sensitive) + matches, stopped = _content_matches(directory, candidates, scan, limit) + body.update( + matches=matches, + count=len(matches), + truncated=stopped, + searched_files=scan.files, + searched_bytes=scan.bytes_read, + skipped=scan.skipped[:limit], + skipped_count=len(scan.skipped), + complete=not stopped and not scan.skipped, + ) + return body + + +_COPY_ARGUMENTS = { + "source": text(f"The file or directory to copy. {URI_HELP}", minLength=1), + "target": text( + f"Where the copy goes: a file for a file, a directory for a directory. {URI_HELP}", + minLength=1, + ), + "overwrite": flag( + f"{OVERWRITE_HELP} For a directory, files that exist at the target are skipped " + "when this is false." + ), + "verify": flag("For a single file: compare the SHA-256 of both sides after the copy."), + "dry_run": flag(DRY_RUN_HELP), +} + +TOOLS: tuple[SemanticTool, ...] = ( + SemanticTool( + FILE_SEARCH, + "Find files below a directory by name pattern and, optionally, by a text they " + "contain (a plain substring, not a regular expression). Results and the bytes " + "read are capped: check 'truncated', and for a content search 'skipped' and " + "'complete'.", + arguments_schema( + { + "uri": text(f"The directory to search. {URI_HELP}", minLength=1), + "pattern": text( + "Shell-style pattern for the file name: *.csv, report-2026-??.csv. With " + "a '/' it is matched against the path below the directory: 2026/*/*.csv. " + "Default: every file.", + default=_ANY_NAME, + minLength=1, + maxLength=_MAX_PATTERN, + ), + "content": text( + "Only return files that contain this text. The first matching line is " + "returned with its number.", + minLength=1, + maxLength=_MAX_NEEDLE, + ), + "recursive": flag("Search subdirectories too. Default true.", default=True), + "case_sensitive": flag("Match names and content case-sensitively."), + "max_results": whole(MAX_RESULTS_HELP), + }, + required=("uri",), + ), + file_search, + ), + SemanticTool( + STORAGE_LIST, + "List the files and directories at a location in any storage backend, with size " + "and modification time. Results are capped: check 'truncated' and 'total'.", + arguments_schema( + { + "uri": text(f"The directory to list. {URI_HELP}", minLength=1), + "recursive": flag("List every descendant, not only the direct children."), + "max_results": whole(MAX_RESULTS_HELP), + }, + required=("uri",), + ), + storage_list, + ), + SemanticTool( + STORAGE_COPY, + "Copy a file, or a whole directory tree, to another location, in the same storage " + "backend or across two. Needs a server that allows writing. For a tree the result " + "counts what was copied, skipped and failed; run it with dry_run=true first to see " + "the plan.", + arguments_schema(_COPY_ARGUMENTS, required=("source", "target")), + storage_copy, + changes=True, + ), +) diff --git a/automation_file/server/mcp_tool_model.py b/automation_file/server/mcp_tool_model.py new file mode 100644 index 0000000..8c59b56 --- /dev/null +++ b/automation_file/server/mcp_tool_model.py @@ -0,0 +1,164 @@ +"""What a semantic MCP tool is made of: a descriptor, checked arguments and a session. + +A :class:`SemanticTool` pairs a hand-written JSON input schema with a plain +function ``handler(session, arguments) -> dict``. The schema is what an MCP host +shows its model, so every description says what the argument means; the same +schema checks the arguments before the handler runs +(:func:`checked_arguments`). A :class:`ToolSession` carries what every handler +needs: the policy and the guarded way to storage. +""" + +from __future__ import annotations + +import copy +from collections.abc import Callable, Mapping +from dataclasses import dataclass, field +from typing import TYPE_CHECKING, Any + +from automation_file.server.mcp_policy import MCPPolicy, MCPToolException, StorageGuard +from automation_file.storage.file import File +from automation_file.storage.storage import Storage + +if TYPE_CHECKING: + from automation_file.core.action_registry import ActionRegistry + +URI_HELP = ( + "A storage URI, :///: local:///srv/reports/a.csv, " + "s3://bucket/2026/a.csv, sftp://host/inbox/a.csv, memory://name/a.csv. An absolute " + "local path works too. It must be inside a location this server allows." +) +DRY_RUN_HELP = "When true, change nothing and report what the call would do." +OVERWRITE_HELP = ( + "Replace a file that already exists at the target. Off by default; the server must " + "allow overwriting as well." +) +MAX_RESULTS_HELP = "Return at most this many entries. The server caps it." + +_JSON_TYPES: Mapping[str, tuple[type, ...]] = { + "string": (str,), + "integer": (int,), + "number": (int, float), + "boolean": (bool,), + "object": (dict,), + "array": (list,), +} +_BOOLEAN = "boolean" + + +def text(description: str, **extra: Any) -> dict[str, Any]: + """Return the schema of a string argument.""" + return {"type": "string", "description": description, **extra} + + +def flag(description: str, default: bool = False) -> dict[str, Any]: + """Return the schema of a boolean argument.""" + return {"type": _BOOLEAN, "description": description, "default": default} + + +def whole(description: str, minimum: int = 1, **extra: Any) -> dict[str, Any]: + """Return the schema of an integer argument.""" + return {"type": "integer", "description": description, "minimum": minimum, **extra} + + +def arguments_schema( + properties: Mapping[str, Mapping[str, Any]], required: tuple[str, ...] = () +) -> dict[str, Any]: + """Return the input schema of a tool: an object that takes exactly ``properties``.""" + schema: dict[str, Any] = { + "type": "object", + "properties": {name: dict(spec) for name, spec in properties.items()}, + "additionalProperties": False, + } + if required: + schema["required"] = list(required) + return schema + + +@dataclass +class ToolSession: + """What a tool handler works with. ``state`` holds what a tool module keeps per server.""" + + policy: MCPPolicy + guard: StorageGuard + registry: ActionRegistry + state: dict[str, Any] = field(default_factory=dict) + + def file(self, uri: str) -> File: + """Return the :class:`File` at ``uri``, reachable only inside the allowed locations.""" + return File(uri, resolver=self.guard) + + def storage(self, uri: str) -> Storage: + """Return the :class:`Storage` at ``uri``, reachable only inside the allowed locations.""" + return Storage(uri, resolver=self.guard) + + +ToolHandler = Callable[[ToolSession, dict[str, Any]], dict[str, Any]] + + +@dataclass(frozen=True) +class SemanticTool: + """One semantic tool. ``changes`` marks a tool that needs writing and takes ``dry_run``.""" + + name: str + description: str + input_schema: Mapping[str, Any] + handler: ToolHandler + changes: bool = False + + def descriptor(self) -> dict[str, Any]: + """Return the tool as ``tools/list`` describes it.""" + return { + "name": self.name, + "description": self.description, + "inputSchema": copy.deepcopy(dict(self.input_schema)), + } + + +def _type_problem(name: str, spec: Mapping[str, Any], value: object) -> str | None: + wanted = spec["type"] + is_boolean = isinstance(value, bool) + if is_boolean != (wanted == _BOOLEAN) or not isinstance(value, _JSON_TYPES[wanted]): + return f"'{name}' must be of type {wanted}" + return None + + +def _value_problem(name: str, spec: Mapping[str, Any], value: Any) -> str | None: + if "enum" in spec and value not in spec["enum"]: + return f"'{name}' must be one of {', '.join(map(str, spec['enum']))}" + if "minimum" in spec and value < spec["minimum"]: + return f"'{name}' must be {spec['minimum']} or more" + if "minLength" in spec and len(value) < spec["minLength"]: + return f"'{name}' must not be empty" + if "maxLength" in spec and len(value) > spec["maxLength"]: + return f"'{name}' is longer than {spec['maxLength']} characters" + return None + + +def checked_arguments(tool: SemanticTool, arguments: Mapping[str, Any]) -> dict[str, Any]: + """Return ``arguments`` checked against the tool's schema, with its defaults filled in. + + An argument given as ``null`` counts as left out. Raises + :class:`MCPToolException` naming every problem; argument values are never + part of the message. + """ + properties: Mapping[str, Mapping[str, Any]] = tool.input_schema["properties"] + given = {name: value for name, value in arguments.items() if value is not None} + problems = [ + f"'{name}' is not an argument of {tool.name}" for name in given if name not in properties + ] + problems.extend( + f"'{name}' is required" + for name in tool.input_schema.get("required", ()) + if name not in given + ) + for name, value in given.items(): + spec = properties.get(name) + if spec is None: + continue + problem = _type_problem(name, spec, value) or _value_problem(name, spec, value) + if problem is not None: + problems.append(problem) + if problems: + raise MCPToolException(f"{tool.name}: {'; '.join(problems)}") + defaults = {name: spec["default"] for name, spec in properties.items() if "default" in spec} + return {**defaults, **given} diff --git a/automation_file/server/mcp_tools.py b/automation_file/server/mcp_tools.py new file mode 100644 index 0000000..55ec285 --- /dev/null +++ b/automation_file/server/mcp_tools.py @@ -0,0 +1,379 @@ +"""Semantic MCP tools: stable, task-shaped names over the storage layer. + +The ``FA_*`` bridge exposes every registered action as a tool. The fourteen +tools here are what an AI client is meant to use instead: ``file_read``, +``file_write``, ``file_copy``, ``file_move``, ``file_search``, +``file_checksum``, ``file_verify``, ``storage_list``, ``storage_copy``, +``pipeline_create``, ``pipeline_run``, ``pipeline_status``, ``integrity_status`` +and ``audit_search``. They work on storage URIs, never on a backend SDK, and +every call passes an :class:`~automation_file.server.mcp_policy.MCPPolicy`. + +.. code-block:: python + + from automation_file.server.mcp_policy import MCPPolicy + from automation_file.server.mcp_tools import SemanticToolkit + + toolkit = SemanticToolkit(MCPPolicy(roots=["local:///srv/reports"], allow_write=True)) + outcome = toolkit.call("file_write", {"uri": "local:///srv/reports/a.txt", "content": "hi"}) + outcome.is_error # False + outcome.payload["sha256"] # the digest of what was written + outcome.correlation_id # ties the audit records of this call together + +:meth:`SemanticToolkit.call` never raises for something a client did: a refused +or failed call comes back as a :class:`ToolOutcome` with ``is_error`` set and an +``error`` that says why. Each call runs inside ``correlation_scope()`` and +``actor_scope(...)`` and is reported as one ``mcp.tool.completed`` or +``mcp.tool.failed`` event, so the audit trail, when it is configured, ties the +storage operations to the call. +""" + +from __future__ import annotations + +import json +import logging +import re +import time +from collections.abc import Mapping +from dataclasses import dataclass +from typing import Any, ClassVar + +from automation_file.core.action_executor import executor +from automation_file.core.action_registry import ActionRegistry +from automation_file.events import ( + Event, + Severity, + actor_scope, + correlation_scope, + current_correlation_id, + emit, +) +from automation_file.exceptions import ( + FileAutomationException, + PathTraversalException, + StorageAlreadyExistsException, + StorageChecksumException, + StorageNotEmptyException, + StorageNotFoundException, + StoragePathTypeException, + StorageUnsupportedException, + StorageURIException, +) +from automation_file.logging_config import file_automation_logger +from automation_file.pipeline.errors import PipelineDefinitionException +from automation_file.server import ( + mcp_file_tools, + mcp_pipeline_tools, + mcp_report_tools, + mcp_storage_tools, +) +from automation_file.server.mcp_policy import ( + ALREADY_EXISTS, + NOT_FOUND, + OUTSIDE_ROOT, + SEMANTIC_TOOL_NAMES, + MCPPermissionException, + MCPPolicy, + MCPToolException, + StorageGuard, +) +from automation_file.server.mcp_tool_model import SemanticTool, ToolSession, checked_arguments +from automation_file.storage.resolver import StorageResolver +from automation_file.storage.uri import parse_storage_uri + +EVENT_SOURCE = "mcp" +STATUS_OK = "ok" +STATUS_REFUSED = "refused" +STATUS_ERROR = "error" +PERMISSION_DENIED = "permission_denied" +INVALID_URI = "invalid_uri" +INVALID_DEFINITION = "invalid_definition" +CHECKSUM_MISMATCH = "checksum_mismatch" +FAILED = "failed" +INTERNAL_ERROR = "internal_error" +_MAX_CLIENT_NAME = 64 +_CLIENT_NAME = re.compile(r"[^A-Za-z0-9._-]+") +_PIPELINE_PREFIX = "pipeline_" +_LOG_LEVELS = { + Severity.INFO: logging.INFO, + Severity.WARNING: logging.WARNING, + Severity.ERROR: logging.ERROR, + Severity.CRITICAL: logging.ERROR, +} +_ERROR_TYPES: tuple[tuple[type[BaseException], str], ...] = ( + (StorageNotFoundException, NOT_FOUND), + (StorageAlreadyExistsException, ALREADY_EXISTS), + (StorageURIException, INVALID_URI), + (StorageChecksumException, CHECKSUM_MISMATCH), +) +#: Failures that say something about the request, not about the storage or the server. +_CALLER_MISTAKES: tuple[type[BaseException], ...] = ( + MCPToolException, + PipelineDefinitionException, + StorageNotFoundException, + StorageAlreadyExistsException, + StoragePathTypeException, + StorageNotEmptyException, + StorageURIException, + StorageUnsupportedException, +) + +_TOOLS_BY_NAME: dict[str, SemanticTool] = { + tool.name: tool + for module in (mcp_file_tools, mcp_storage_tools, mcp_pipeline_tools, mcp_report_tools) + for tool in module.TOOLS +} +#: The fourteen tools in catalogue order. +SEMANTIC_TOOLS: tuple[SemanticTool, ...] = tuple( + _TOOLS_BY_NAME[name] for name in SEMANTIC_TOOL_NAMES +) + + +@dataclass(frozen=True, kw_only=True) +class MCPToolCompleted(Event): + """A semantic tool call did what it was asked.""" + + type: ClassVar[str] = "mcp.tool.completed" + + +@dataclass(frozen=True, kw_only=True) +class MCPToolFailed(Event): + """A semantic tool call was refused by the policy or failed. ``status`` says which.""" + + type: ClassVar[str] = "mcp.tool.failed" + severity: Severity = Severity.WARNING + + +@dataclass(frozen=True) +class ToolOutcome: + """What one call returned: a JSON-friendly ``payload`` and whether it is an error.""" + + tool: str + payload: dict[str, Any] + is_error: bool = False + + @property + def correlation_id(self) -> str: + return str(self.payload["correlation_id"]) + + def to_mcp(self) -> dict[str, Any]: + """Return the ``tools/call`` result: the payload as one JSON text block.""" + encoded = json.dumps(self.payload, ensure_ascii=False, default=str) + return {"content": [{"type": "text", "text": encoded}], "isError": self.is_error} + + +@dataclass(frozen=True) +class _Failure: + """A call that did not succeed: what the client is told and what the event keeps.""" + + status: str + severity: Severity + error: dict[str, Any] + recorded: str + + +def _refusal(code: str, message: str) -> _Failure: + error = {"type": PERMISSION_DENIED, "code": code, "message": message} + return _Failure(STATUS_REFUSED, Severity.WARNING, error, f"{code}: {message}") + + +def _described(kind: str, error: Exception) -> dict[str, Any]: + return {"type": kind, "message": str(error), "exception": type(error).__name__} + + +def _severity_of(error: Exception) -> Severity: + """Say how much attention a failed call needs. + + A mistake of the caller (a missing file, an existing target, a wrong + argument) is information. A digest that does not match is an error. Anything + else the library reported is a warning. + """ + if isinstance(error, StorageChecksumException): + return Severity.ERROR + return Severity.INFO if isinstance(error, _CALLER_MISTAKES) else Severity.WARNING + + +def _failed(kind: str, error: Exception, **details: Any) -> _Failure: + described = {**_described(kind, error), **details} + recorded = f"{type(error).__name__}: {error}" + return _Failure(STATUS_ERROR, _severity_of(error), described, recorded) + + +def _failure(error: Exception) -> _Failure: + """Sort an exception into a refusal, a failed call or an internal error.""" + if isinstance(error, MCPPermissionException): + return _refusal(error.code, str(error)) + if isinstance(error, PathTraversalException): + return _refusal(OUTSIDE_ROOT, "the location leaves the allowed location through a link") + if isinstance(error, MCPToolException): + return _failed(error.kind, error) + if isinstance(error, PipelineDefinitionException): + return _failed(INVALID_DEFINITION, error, problems=list(error.problems)) + if isinstance(error, FileAutomationException): + kind = next((name for cls, name in _ERROR_TYPES if isinstance(error, cls)), FAILED) + return _failed(kind, error) + # Not an error this library raises on purpose. The client is told what it was; the event + # and the log keep only its type, because its text may quote what was being handled. + return _Failure( + STATUS_ERROR, Severity.ERROR, _described(INTERNAL_ERROR, error), type(error).__name__ + ) + + +_PARTIAL = _Failure( + STATUS_ERROR, + Severity.WARNING, + {"type": FAILED, "message": "the call finished with failures; the result says which"}, + "finished with failures", +) + + +def _shown_uri(value: object) -> str | None: + """Return a storage URI argument in its normal form, or ``None`` when it is not one.""" + if not isinstance(value, str): + return None + try: + return str(parse_storage_uri(value)) + except StorageURIException: + return None + + +def _subject_of( + tool: SemanticTool, arguments: Mapping[str, Any], body: Mapping[str, Any] +) -> dict[str, Any]: + """Return the payload keys that say what a call was about. Content is never among them.""" + details: dict[str, Any] = {} + resource = _shown_uri(arguments.get("target")) or _shown_uri(arguments.get("uri")) + if resource is not None: + details["resource"] = resource + source = _shown_uri(arguments.get("source")) + if source is not None: + details["source_uri"] = source + for key in ("run_id", "dry_run"): + if key in body: + details[key] = body[key] + if tool.name.startswith(_PIPELINE_PREFIX) and "name" in body: + details["pipeline"] = body["name"] + return details + + +class SemanticToolkit: + """The semantic tools of one server, bound to its policy. + + ``registry`` is where the non-storage actions of the policy's pipeline allow + list are looked up (default: the shared executor's registry). ``resolver`` + is the storage resolver behind the guard (default: the process-wide one). + """ + + def __init__( + self, + policy: MCPPolicy | None = None, + registry: ActionRegistry | None = None, + *, + resolver: StorageResolver | None = None, + ) -> None: + self._policy = policy if policy is not None else MCPPolicy() + self._session = ToolSession( + policy=self._policy, + guard=StorageGuard(self._policy, resolver), + registry=registry if registry is not None else executor.registry, + ) + self._client: str | None = None + mcp_pipeline_tools.check_pipeline_actions(self._session) + + @property + def policy(self) -> MCPPolicy: + return self._policy + + @property + def session(self) -> ToolSession: + """What the tool handlers work with; pass it to a handler to call one directly.""" + return self._session + + @property + def actor(self) -> str: + """Who the calls are made as: the policy's actor, with the client's name once known.""" + return f"{self._policy.actor}:{self._client}" if self._client else self._policy.actor + + def set_client(self, name: object) -> None: + """Remember the client's self-reported name. It labels the actor; it proves nothing.""" + cleaned = _CLIENT_NAME.sub("_", name).strip("_") if isinstance(name, str) else "" + self._client = cleaned[:_MAX_CLIENT_NAME] or None + + @staticmethod + def owns(name: str) -> bool: + """Return whether ``name`` is one of the fourteen semantic tools, offered or not.""" + return name in _TOOLS_BY_NAME + + def descriptors(self) -> list[dict[str, Any]]: + """Return the offered tools as ``tools/list`` describes them, in catalogue order.""" + return [_TOOLS_BY_NAME[name].descriptor() for name in self._policy.enabled_tools()] + + def instructions(self) -> str: + """Return what a client should know before its first call: the policy in words.""" + return self._policy.summary() + + def call(self, name: str, arguments: Mapping[str, Any] | None = None) -> ToolOutcome: + """Run the semantic tool ``name`` and return its outcome. + + Raises :class:`MCPToolException` only for a name that is not a semantic + tool. Everything else, a refusal by the policy included, is an outcome + with ``is_error`` set. + """ + tool = _TOOLS_BY_NAME.get(name) + if tool is None: + raise MCPToolException(f"unknown semantic tool: {name[:64]!r}", NOT_FOUND) + given = dict(arguments or {}) + with correlation_scope() as correlation_id, actor_scope(self.actor): + started = time.perf_counter() + body: dict[str, Any] = {} + failure: _Failure | None = None + try: + self._policy.require_tool(name) + body = tool.handler(self._session, checked_arguments(tool, given)) + except Exception as error: # pylint: disable=broad-exception-caught + # Dispatcher boundary: whatever a tool raises becomes the outcome of this call. + failure = _failure(error) + if failure is None and body.get("ok") is False: + failure = _PARTIAL + duration = round((time.perf_counter() - started) * 1000, 3) + self._report(tool, _subject_of(tool, given, body), failure, duration) + payload: dict[str, Any] = {"tool": name, "correlation_id": correlation_id, **body} + if failure is not None: + payload["error"] = failure.error + return ToolOutcome(name, payload, is_error=failure is not None) + + @staticmethod + def _report( + tool: SemanticTool, subject: Mapping[str, Any], failure: _Failure | None, duration: float + ) -> None: + """Publish the event of one call and log a refusal or a failure. + + The event holds the storage URIs the call was about and nothing else of + its arguments: no content, no parameters, no digest. The log line holds + the tool, the reason code and the correlation ID. + """ + status = STATUS_OK if failure is None else failure.status + title = f"{tool.name} {status}" + payload: dict[str, Any] = { + "action": tool.name, + "status": status, + "duration_ms": duration, + **subject, + } + if failure is None: + emit(MCPToolCompleted(source=EVENT_SOURCE, subject=title, payload=payload)) + return + code = failure.error.get("code", failure.error["type"]) + payload.update(error=failure.recorded, code=code) + file_automation_logger.log( + _LOG_LEVELS[failure.severity], + "mcp_tools: %s %s (%s) correlation_id=%s", + tool.name, + status, + code, + current_correlation_id(), + ) + emit( + MCPToolFailed( + source=EVENT_SOURCE, subject=title, severity=failure.severity, payload=payload + ) + ) diff --git a/docs/source/API/server.rst b/docs/source/API/server.rst index f0bec9e..06bb3b5 100644 --- a/docs/source/API/server.rst +++ b/docs/source/API/server.rst @@ -18,6 +18,33 @@ The HTTP server also exposes ``GET /healthz`` (liveness), ``GET /readyz`` .. automodule:: automation_file.server.mcp_server :members: +The semantic MCP tools: the policy, the toolkit, and the modules that hold the +tools. + +.. automodule:: automation_file.server.mcp_policy + :members: + +.. automodule:: automation_file.server.mcp_tools + :members: + +.. automodule:: automation_file.server.mcp_tool_model + :members: + +.. automodule:: automation_file.server.mcp_file_tools + :members: + +.. automodule:: automation_file.server.mcp_storage_tools + :members: + +.. automodule:: automation_file.server.mcp_pipeline_tools + :members: + +.. automodule:: automation_file.server.mcp_pipeline_actions + :members: + +.. automodule:: automation_file.server.mcp_report_tools + :members: + .. automodule:: automation_file.server.metrics_server :members: diff --git a/docs/source/Eng/usage/cli.rst b/docs/source/Eng/usage/cli.rst index c1f81dd..35efe9b 100644 --- a/docs/source/Eng/usage/cli.rst +++ b/docs/source/Eng/usage/cli.rst @@ -17,12 +17,16 @@ Subcommands for one-shot operations:: python -m automation_file create-file hello.txt --content "hi" python -m automation_file server --host 127.0.0.1 --port 9943 python -m automation_file http-server --host 127.0.0.1 --port 9944 + python -m automation_file mcp --root /srv/reports --no-bridge python -m automation_file mcp --allowed-actions FA_list_dir,FA_file_checksum python -m automation_file drive-upload my.txt --token token.json --credentials creds.json The ``mcp`` subcommand starts a Model Context Protocol server over stdio so -hosts such as Claude Desktop can call ``FA_*`` actions as MCP tools — see -:doc:`mcp` for the full integration guide. +hosts such as Claude Desktop can work with files: through the semantic tools +(``file_read``, ``storage_copy``, ``pipeline_run``, ...), which stay inside the +``--root`` locations and are read-only until ``--allow-write``, and through the +bridge that offers ``FA_*`` actions as MCP tools. See :doc:`mcp` for the flags, +the permission model and the full integration guide. Storage ------- diff --git a/docs/source/Eng/usage/mcp.rst b/docs/source/Eng/usage/mcp.rst index e7febe6..1685600 100644 --- a/docs/source/Eng/usage/mcp.rst +++ b/docs/source/Eng/usage/mcp.rst @@ -1,74 +1,28 @@ -MCP server (Claude Desktop / Claude Code) -========================================= +MCP server +========== -``automation_file`` ships a Model Context Protocol (MCP) server that -exposes every entry of the shared -:class:`~automation_file.core.action_registry.ActionRegistry` as an MCP -tool. Hosts such as **Claude Desktop**, **Claude Code**, and other -MCP-aware clients can then call ``FA_*`` actions exactly like any other -MCP tool — no plugin code, no extra packaging. +``automation_file`` ships a Model Context Protocol (MCP) server, so an AI client +such as **Claude Desktop** or **Claude Code** can work with files through it. The +transport is stdio: one JSON-RPC 2.0 message per line. -Transport is **stdio** (one JSON-RPC 2.0 message per line on -``stdin`` / ``stdout``), matching the protocol that current MCP hosts -consume. +The server offers two sets of tools: -What you get ------------- - -* ``initialize`` handshake reporting protocol version ``2024-11-05`` and - ``serverInfo.name`` / ``serverInfo.version``. -* ``tools/list`` returns one MCP tool per registered ``FA_*`` action, - with an auto-derived JSON Schema for its arguments (built from the - Python signature: ``str → "string"``, ``int → "integer"``, etc.). -* ``tools/call`` dispatches through the registry and returns the result - as a JSON-encoded text content block. -* ``--allowed-actions`` allow-list flag to surface only a subset of the - registry to the host. -* Every internal failure surfaces as a JSON-RPC error object — the host - can render it without parsing exception strings. - -Starting the server -------------------- - -CLI:: - - python -m automation_file mcp - python -m automation_file mcp --name automation_file --version 1.0.0 - python -m automation_file mcp --allowed-actions FA_list_dir,FA_file_checksum - -The process runs in the foreground, reading newline-delimited JSON from -``stdin`` and writing responses to ``stdout``. MCP hosts spawn this -process for you — you rarely run it by hand. - -From Python (e.g. when embedding into another stdio bridge): - -.. code-block:: python - - from automation_file import MCPServer - - server = MCPServer(name="automation_file", version="1.0.0") - server.serve_stdio() # blocks until stdin closes - -Filter to a smaller surface area by passing your own registry: - -.. code-block:: python - - from automation_file import MCPServer - from automation_file.core.action_registry import ActionRegistry - from automation_file import executor +* the **semantic tools**: fourteen stable, task-shaped tools (``file_read``, + ``file_copy``, ``pipeline_run``, ...) that work on :doc:`storage URIs ` + and are bound by a permission policy. This is the interface meant for AI + clients; +* the **bridge**: every registered ``FA_*`` action as a tool of its own, as in + earlier versions. It is on by default for compatibility and is **not** bound by + the policy. - safe = ActionRegistry() - for name in ("FA_list_dir", "FA_file_checksum", "FA_fast_find"): - safe.register(name, executor.registry.resolve(name)) +The defaults are safe for the semantic tools: no location is allowed, nothing can +be changed, and every answer is bounded in size. - MCPServer(safe).serve_stdio() +Minimal setup +------------- -Claude Desktop configuration ----------------------------- - -Add an entry under ``mcpServers`` in -``~/Library/Application Support/Claude/claude_desktop_config.json`` (macOS) or -``%APPDATA%\Claude\claude_desktop_config.json`` (Windows): +Give the server one directory to read. In ``claude_desktop_config.json`` (Claude +Desktop) or ``.mcp.json`` (Claude Code): .. code-block:: json @@ -76,129 +30,821 @@ Add an entry under ``mcpServers`` in "mcpServers": { "automation_file": { "command": "python", - "args": ["-m", "automation_file", "mcp"] + "args": ["-m", "automation_file", "mcp", "--root", "/srv/reports", "--no-bridge"] } } } -Restart Claude Desktop. The ``automation_file`` server appears in the -tools panel; every ``FA_*`` action is callable. +The client now sees the fourteen semantic tools. It can list, read, search and +checksum what is below ``/srv/reports``, and nothing else: writing is refused, +and so is every path outside that directory. On Windows write the path as +``"C:\\data\\reports"``. Use the interpreter of the environment the package is +installed in (``"command": "C:\\envs\\fa\\Scripts\\python.exe"``) when ``python`` +on the ``PATH`` is another one. + +With Claude Code the same server is added from a shell:: -Lock the surface area down to a curated allow-list — recommended for -hosts that operate on sensitive paths: + claude mcp add automation_file -- python -m automation_file mcp --root /srv/reports --no-bridge + +Production setup +---------------- + +Name every location, switch on only the permissions the work needs, keep the +definitions of pipelines on disk, and leave the bridge off: .. code-block:: json { "mcpServers": { "automation_file": { - "command": "python", + "command": "/opt/fa/bin/python", "args": [ "-m", "automation_file", "mcp", - "--allowed-actions", - "FA_list_dir,FA_fast_find,FA_file_checksum,FA_verify_checksum" + "--root", "/srv/reports/inbox", + "--root", "/srv/reports/outbox", + "--allow-write", + "--max-read-bytes", "65536", + "--max-results", "100", + "--pipeline-dir", "/var/lib/automation_file/pipelines", + "--tools", "file_read,file_write,file_copy,file_search,file_checksum,file_verify,storage_list,storage_copy", + "--no-bridge" ] } } } -Use a virtualenv interpreter explicitly (avoids picking up the system -Python): +A remote backend needs its client initialised, and an audit trail and +notification routes have to be set up, before the server starts. That takes a few +lines of Python, so start the server from a launcher script and point the host's +``command`` at it: + +.. code-block:: python + + # /opt/fa/mcp_server.py + import os + + from automation_file import ( + MCPServer, Route, Severity, SlackSink, configure_audit, + notification_manager, notification_router, s3_instance, sftp_instance, + ) + from automation_file.server.mcp_policy import MCPPolicy + + s3_instance.later_init(region_name="eu-west-1") # credentials: the AWS chain + sftp_instance.later_init(host="sftp.example.com", username="reports", + key_filename="/etc/fa/id_ed25519", + known_hosts="/etc/fa/known_hosts") + configure_audit("/var/lib/automation_file/audit.sqlite") # audit_search reads this + notification_manager.register(SlackSink(os.environ["SLACK_WEBHOOK"], name="ops")) + notification_router.add_route(Route( + "mcp-failures", sinks=("ops",), + types=("mcp.tool.failed", "pipeline.failed", "storage.error"), + min_severity=Severity.WARNING, + )) + notification_router.start() + + policy = MCPPolicy( + roots=["s3://reports-export/daily", "sftp://sftp.example.com/inbound/reports"], + allow_write=True, + allow_delete=True, # file_move deletes its source + max_read_bytes=64 * 1024, + pipeline_dir="/var/lib/automation_file/pipelines", + ) + MCPServer(policy=policy, bridge=False).serve_stdio() + +.. code-block:: json + + {"mcpServers": {"automation_file": {"command": "/opt/fa/bin/python", + "args": ["/opt/fa/mcp_server.py"]}}} + +Nothing may be written to ``stdout`` but the protocol: the library logs to +``stderr`` and to its log file, and a launcher must not ``print``. + +The semantic tools +------------------ + +Every location is a storage URI, ``:///`` +(``local:///srv/reports/a.csv``, ``s3://bucket/2026/a.csv``, +``sftp://host/inbox/a.csv``), or an absolute local path. "Needs" lists what the +policy must allow besides a root that contains every location of the call. + +.. list-table:: + :header-rows: 1 + :widths: 14 30 36 20 + + * - Tool + - Arguments + - Result + - Needs + * - ``file_read`` + - ``uri``, ``offset=0``, ``max_bytes``, ``encoding="utf-8"`` (a text + encoding, or ``base64``) + - ``content``, ``encoding``, ``size``, ``offset``, ``bytes``, + ``truncated``, ``next_offset`` + - — + * - ``file_write`` + - ``uri``, ``content``, ``encoding="utf-8"``, ``overwrite=false``, + ``dry_run=false`` + - ``size``, ``sha256``, ``overwrites``, ``replaced_size``, ``written`` + - write; overwrite to replace a file + * - ``file_copy`` + - ``source``, ``target``, ``overwrite=false``, ``verify=false``, + ``dry_run=false`` + - ``source``, ``target``, ``size``, ``overwrites``, ``replaced_size``, + ``deletes_source``, ``done``; with ``verify``: ``sha256``, ``verified`` + - write; overwrite to replace a file + * - ``file_move`` + - As ``file_copy`` + - As ``file_copy``; ``deletes_source`` is true + - write and delete; overwrite to replace a file + * - ``file_search`` + - ``uri``, ``pattern="*"``, ``content``, ``recursive=true``, + ``case_sensitive=false``, ``max_results`` + - ``matches`` (``uri``, ``path``, ``name``, ``size``, ``modified_at``; for a + content search also ``line``, ``snippet``, ``matching_lines``), ``count``, + ``candidates``, ``truncated``; for a content search also + ``searched_files``, ``searched_bytes``, ``skipped``, ``complete`` + - — + * - ``file_checksum`` + - ``uri``, ``algorithm="sha256"`` + - ``algorithm``, ``value``, ``size`` + - — + * - ``file_verify`` + - ``uri``, ``expected`` (hex, or ``sha256:``), ``algorithm="sha256"`` + - ``match``, ``expected``, ``actual``, ``algorithm`` + - — + * - ``storage_list`` + - ``uri``, ``recursive=false``, ``max_results`` + - ``entries`` (``uri``, ``path``, ``name``, ``is_dir``, ``size``, + ``modified_at``), ``count``, ``total``, ``truncated`` + - — + * - ``storage_copy`` + - ``source``, ``target``, ``overwrite=false``, ``verify=false``, + ``dry_run=false`` + - A file: ``kind="file"`` and the result of ``file_copy``. A directory: + ``kind="tree"``, ``planned`` (``copy``, ``overwrite``, ``skip``, + ``bytes``), ``paths``, ``existing``, ``truncated``, ``done``, and after + the copy ``copied``, ``skipped``, ``failed``, ``errors``, ``ok`` + - write; overwrite to replace files + * - ``pipeline_create`` + - ``name``, ``definition``, ``overwrite=false``, ``dry_run=false`` + - ``name``, ``location``, ``persistent``, ``tasks``, ``actions``, + ``overwrites``, ``stored`` + - write; overwrite to replace a definition + * - ``pipeline_run`` + - ``name``, ``params``, ``dry_run=false``, ``background=false`` + - ``run_id``, ``status``, ``ok``, ``run`` (the run with every task) + - write; each task needs what its action needs + * - ``pipeline_status`` + - ``run_id``, or ``name`` and ``limit=5`` + - ``runs``, ``count`` + - no root + * - ``integrity_status`` + - ``name`` + - ``monitors`` (``name``, ``target``, ``running``, ``last_run``, + ``last_error``, ``last_report``), ``count`` + - no root + * - ``audit_search`` + - ``actor``, ``source``, ``pipeline``, ``task``, ``action``, ``backend``, + ``status``, ``correlation_id``, ``resource_prefix``, ``text``, ``since``, + ``until``, ``limit=50``, ``offset=0`` + - ``records``, ``count``, ``total``, ``limit``, ``offset``, ``truncated`` + - no root; a configured audit trail + +Notes on single tools: + +``file_read`` + Returns at most ``--max-read-bytes`` per call. When ``truncated`` is true, + call again with ``offset`` set to ``next_offset``; a character is never cut + in half. Content that is not text in the chosen encoding is an error that + asks for ``encoding="base64"``. + +``file_copy``, ``file_move`` and ``verify`` + With ``verify=true`` the SHA-256 of the source is compared with that of the + copy. A move then copies, compares, and deletes the source only when the + digests match; on a mismatch the source stays and the call fails as + ``checksum_mismatch``. + +``file_verify`` + A mismatch is a result (``match`` is false), not an error. + +``file_search`` + ``pattern`` is a shell-style pattern for the file name (``*.csv``); with a + ``/`` in it, it is matched against the path below ``uri`` + (``2026/*/*.csv``). ``content`` is a plain substring, not a regular + expression. A content search reads at most ``--max-search-bytes`` in one + call. A file that does not fit in what is left of that budget is not opened + and is listed under ``skipped``, as are binary files and files that could + not be read; ``complete`` is true only when every candidate was searched. + +``storage_copy`` + Copies a file like ``file_copy``, or every file below a directory. For a + directory, files that exist at the target are skipped unless ``overwrite`` + is true. A file that fails is recorded under ``errors`` and the others are + still copied; the call is then an error whose result holds the counts. + +``pipeline_run`` + Runs in the calling request and returns the finished run. With + ``background=true`` it returns at once with ``status="running"``; poll + ``pipeline_status`` with the ``run_id``. A run that fails is an error whose + result still holds the run. + +Results and errors +~~~~~~~~~~~~~~~~~~ + +A semantic tool answers with one JSON document as the text of the result. It +always holds ``tool`` and ``correlation_id``. When the call was refused or +failed, ``isError`` is true and the document holds an ``error``: .. code-block:: json { - "mcpServers": { - "automation_file": { - "command": "C:\\envs\\fa\\Scripts\\python.exe", - "args": ["-m", "automation_file", "mcp"] - } + "tool": "file_write", + "correlation_id": "27acff55529c4317b74de6ea98759f42", + "error": { + "type": "permission_denied", + "code": "read_only", + "message": "file_write changes something and this server is read-only: start it with --allow-write, or pass MCPPolicy(allow_write=True)" } } -Claude Code configuration -------------------------- +.. list-table:: + :header-rows: 1 + :widths: 26 74 + + * - ``error.type`` + - Meaning + * - ``permission_denied`` + - The policy refused the call. ``code`` names the rule: ``no_root``, + ``outside_root``, ``read_only``, ``overwrite_not_allowed``, + ``delete_not_allowed``, ``tool_disabled``, ``action_not_allowed`` or + ``limit_exceeded``. + * - ``invalid_arguments`` + - An argument is missing, unknown, of the wrong type or out of range. Every + problem is listed. + * - ``invalid_uri`` + - The location is not a storage URI, or its path holds a ``..`` segment. + * - ``not_found``, ``already_exists`` + - There is no such file, pipeline, run or monitor; or the target exists and + ``overwrite`` was not given. + * - ``checksum_mismatch`` + - A ``verify`` found different digests on the two sides. + * - ``invalid_definition`` + - The pipeline definition is wrong; ``problems`` lists every finding with + its path. + * - ``not_configured`` + - The server keeps no audit trail. + * - ``failed`` + - Anything else the library reported; ``exception`` names the class. + * - ``internal_error`` + - An unexpected exception. Report it. + +The permission model +-------------------- + +An :class:`~automation_file.server.mcp_policy.MCPPolicy` is built when the server +starts and cannot be changed afterwards. Its text form is sent to the client in +the ``instructions`` of the handshake, so the model knows the boundaries before +its first call. + +.. list-table:: + :header-rows: 1 + :widths: 24 24 14 38 + + * - Field + - Flag + - Default + - Meaning + * - ``roots`` + - ``--root`` (repeatable) + - none + - The locations the tools may work in: storage URIs or local directories. + Without one, every tool that touches storage refuses and says how to add + a root. + * - ``allow_write`` + - ``--allow-write`` + - off + - ``file_write``, ``file_copy``, ``file_move``, ``storage_copy``, + ``pipeline_create`` and ``pipeline_run`` are refused without it, a dry + run included. + * - ``allow_overwrite`` + - ``--allow-overwrite`` + - off + - Replacing an existing file or definition. The call has to ask for it too + (``overwrite=true``). Needs ``allow_write``. + * - ``allow_delete`` + - ``--allow-delete`` + - off + - Deleting. ``file_move`` deletes its source, so it needs this. Needs + ``allow_write``. + * - ``max_read_bytes`` + - ``--max-read-bytes`` + - 262144 + - The most bytes ``file_read`` returns in one call, and the largest task + result a pipeline tool returns. + * - ``max_write_bytes`` + - ``--max-write-bytes`` + - 1048576 + - The largest content ``file_write`` accepts and the largest definition + ``pipeline_create`` stores. + * - ``max_results`` + - ``--max-results`` + - 200 + - The most entries a listing, a search, a tree plan or ``audit_search`` + returns. A larger ``max_results`` or ``limit`` in a call is lowered to it. + * - ``max_search_bytes`` + - ``--max-search-bytes`` + - 8388608 + - The most bytes one content search reads. + * - ``pipeline_dir`` + - ``--pipeline-dir`` + - memory + - Where ``pipeline_create`` keeps definitions: a storage URI or a local + directory, one ``.json`` per pipeline. Without it they are kept in + memory and are gone when the server stops. + * - ``pipeline_actions`` + - ``--pipeline-actions`` + - by permission + - The actions a pipeline created or run through MCP may call. See + `Pipelines`_. + * - ``tools`` + - ``--tools`` + - all fourteen + - The semantic tools to offer. A tool left out is not listed and a call to + it is refused. ``--tools none`` offers none. + * - ``actor`` + - (Python only) + - ``mcp`` + - The actor of every call in events and audit records. The client's name + from the handshake is appended: ``mcp:claude-desktop``. + +Locations +~~~~~~~~~ + +A call is allowed when every location it names is at or below a root. + +* **A local root** is enforced by the storage layer itself. The location is + served by a ``LocalStorage`` confined to the root, so every operation passes + ``safe_join``: a symbolic link (or a Windows junction) that leads out of the + root, and an absolute path smuggled below it, are refused as + ``outside_root``. A link that stays inside a root is followed. On Windows the + comparison ignores case and accepts backslashes, and a name that is a device + (``CON``, ``NUL``, ``COM1``) or an alternate data stream (``a.txt:stream``) is + refused too. +* **Any other backend** is compared by scheme, authority and whole path + segments. ``s3://bucket/team`` allows ``s3://bucket/team/2026/a.csv`` and + refuses ``s3://bucket/team-b/a.csv`` and ``s3://other/team/a.csv``. The + authority is compared exactly, letter case included. The root is a path prefix + and nothing more: a link that exists on the server, an SFTP symbolic link for + instance, is followed by the server. Give the backend an account that cannot + leave the tree. +* A path with a ``..`` segment is not a storage URI at all (``invalid_uri``). +* A backend mounted with ``Storage.mount`` is addressed through its mount URI, + and a root below a mounted ``LocalStorage`` is confined to that root. + +Write a location the way the root is written. A root given as +``/srv/reports`` does not match ``local:///mnt/disk2/reports`` even when both are +the same directory. + +Pipelines +~~~~~~~~~ + +``pipeline_create`` stores a definition (:doc:`pipeline`, ``schema_version: 1``) +and ``pipeline_run`` runs it. Both need ``allow_write``. + +What a definition may call is checked when it is created and again when it is +run, because the file may have been written by something else in between. By +default a pipeline may call the ``FA_storage_*`` actions the permissions cover: + +.. list-table:: + :header-rows: 1 + :widths: 30 70 + + * - Permission + - Actions + * - always + - ``FA_storage_exists``, ``FA_storage_stat``, ``FA_storage_list``, + ``FA_storage_checksum``, ``FA_storage_verify``, ``FA_storage_read_text``, + ``FA_storage_schemes`` + * - ``allow_write`` + - ``FA_storage_mkdir``, ``FA_storage_write_text``, ``FA_storage_copy``, + ``FA_storage_copy_tree`` + * - ``allow_overwrite`` + - ``FA_storage_sync`` (it replaces changed files) + * - ``allow_delete`` + - ``FA_storage_move``, ``FA_storage_delete`` + +Inside such a pipeline these names do not run the plain actions. They run +guarded versions that check the roots and the permissions at the moment the task +runs, after ``${params.}`` and ``${tasks..result}`` are filled in, so a +parameter cannot lead a task out of the roots. They behave like the plain +actions with these differences: + +* ``overwrite`` defaults to what the policy permits instead of ``true``; an + explicit ``overwrite: true`` is refused where overwriting is not allowed; +* ``FA_storage_verify`` also takes the result of ``FA_storage_checksum`` as its + ``expected``, so ``"${tasks..result}"`` compares two files; +* ``FA_storage_read_text`` and ``FA_storage_write_text`` keep the read and write + limits; +* ``FA_storage_sync`` needs ``allow_overwrite``, and ``delete: true`` needs + ``allow_delete``; +* ``FA_storage_delete`` never removes a root itself; +* ``FA_storage_upload`` and ``FA_storage_download`` are not available: they take + a filesystem path, not a storage URI. Copy to or from a ``local://`` URI. + +``--pipeline-actions a,b,c`` replaces the default list. The ``FA_storage_*`` +actions above stay guarded when they are listed. **Any other action runs as it +is, outside the roots and the permissions**: list only what you would let the +client call directly, and never an action that runs other actions +(``FA_execute_action``, ``FA_pipeline_run``, ``FA_run_shell``). Every name has to +be an action the server exposes, or the server does not start. An action named +inside the arguments or the parameters of another one is refused unless the list +names it, and a storage action is always refused there: nested, it would run +unguarded. + +A definition created through MCP cannot carry a ``schedule``: when a pipeline +runs by itself is the operator's decision. The definition's ``name`` is the name +it is stored under; it is filled in when left out. + +Runs are recorded in the default run store, where ``pipeline_status`` and +``FA_pipeline_status`` find them. That store is in memory unless a launcher calls +``set_default_run_store(SQLiteRunStore(path))``. + +Dry run +------- + +Every tool that changes something takes ``dry_run``. It then does every check +the real call would do, changes nothing, and returns the plan: -Claude Code consumes the same MCP definition. Register the server with -the ``claude mcp add`` CLI:: +.. code-block:: json + + { + "tool": "file_move", + "correlation_id": "bb62a807d4f64bf3b666d00d3b4f410f", + "source": "s3://reports-export/daily/2026-10-07.csv", + "target": "sftp://sftp.example.com/inbound/reports/2026-10-07.csv", + "size": 48211, + "overwrites": false, + "replaced_size": null, + "deletes_source": true, + "dry_run": true, + "done": false + } - claude mcp add automation_file -- python -m automation_file mcp +.. list-table:: + :header-rows: 1 + :widths: 24 76 + + * - Tool + - What the dry run reports + * - ``file_write`` + - The size and SHA-256 of the content, whether a file would be replaced and + how large it is. + * - ``file_copy``, ``file_move`` + - Source, target, size, whether the target would be replaced, whether the + source would be deleted. + * - ``storage_copy`` + - For a directory: how many files and bytes would be copied, replaced and + skipped, with the paths (capped by ``max_results``). + * - ``pipeline_create`` + - Whether the definition is valid and allowed, its tasks and actions, and + whether it would replace a stored one. + * - ``pipeline_run`` + - The tasks in dependency order as ``planned``, with an ``error`` on a task + that names an unknown action or a parameter the run was not given. + Nothing is executed or recorded. + +A dry run needs the same permissions as the real call: a read-only server +refuses it as well, so a plan is never offered for something the server would +not do. + +Traceability +------------ -Or commit a ``.mcp.json`` to the repo root: +Each semantic call runs inside ``correlation_scope()`` and ``actor_scope(...)``. +The result carries the ``correlation_id``, and so does everything the call does: +the storage operations and one event per call, ``mcp.tool.completed`` or +``mcp.tool.failed`` (source ``mcp``). + +.. list-table:: + :header-rows: 1 + :widths: 26 14 60 + + * - Outcome + - Severity + - Event + * - Done + - info + - ``mcp.tool.completed``, ``status="ok"`` + * - Refused by the policy + - warning + - ``mcp.tool.failed``, ``status="refused"``, ``code`` names the rule + * - A mistake in the request: a missing file, an existing target, a wrong + argument, an invalid definition + - info + - ``mcp.tool.failed``, ``status="error"``, ``code`` is the ``error.type`` + * - Failed: a backend error, a tree copy with failed files, a failed + pipeline run + - warning + - ``mcp.tool.failed``, ``status="error"``. A failing backend is also + reported as ``storage.error``, a failed run as ``pipeline.failed``. + * - A ``verify`` whose digests differ + - error + - ``mcp.tool.failed``, ``status="error"``, ``code="checksum_mismatch"`` + * - Unexpected exception + - error + - ``mcp.tool.failed``, ``status="error"``, ``code="internal_error"`` + +A notification route on ``mcp.tool.failed`` with ``min_severity=Severity.WARNING`` +therefore hears about refusals and real failures, and not about a model that +asked for a file that is not there. + +The payload names the tool (``action``), the storage URIs the call was about +(``resource``, ``source_uri``), ``duration_ms`` and, for the pipeline tools, +``pipeline`` and ``run_id``. It never holds content, parameters or digests. A +refused call is also logged as a warning with the tool, the rule and the +correlation ID, and no argument value. + +With an :doc:`audit trail ` configured, one search returns what a call +did: -.. code-block:: json +.. code-block:: python - { - "mcpServers": { - "automation_file": { - "command": "python", - "args": ["-m", "automation_file", "mcp"] + audit_search(correlation_id="7dc51e94bd3b494eae8e6b6f3f3b150b") + # [{"source": "mcp", "action": "mcp.tool.completed", "actor": "mcp:claude-desktop", ...}, + # {"source": "storage", "action": "copy", "resource": "sftp://...", "status": "ok", ...}] + +A pipeline run has a correlation ID of its own, its ``run_id``. The event of the +``pipeline_run`` call holds that ``run_id``, which ties the two together. + +Example: S3 to SFTP, verified, audited, with an alert on failure +---------------------------------------------------------------- + +The task: *move yesterday's CSV from S3 to the company SFTP server, verify its +SHA-256, audit the transfer, and notify Slack on failure*. The launcher of +`Production setup`_ provides what this needs: both backends initialised, two +roots, writing and deleting allowed, an audit trail, and a route that sends +failures to Slack. The client then makes these calls: + +.. code-block:: text + + 1. storage_list {"uri": "s3://reports-export/daily"} + -> the entries; the client picks 2026-10-07.csv + + 2. file_move {"source": "s3://reports-export/daily/2026-10-07.csv", + "target": "sftp://sftp.example.com/inbound/reports/2026-10-07.csv", + "verify": true, "dry_run": true} + -> the plan: size, overwrites=false, deletes_source=true + + 3. file_move the same arguments without dry_run + -> done=true, verified=true, sha256="9f86d0...", correlation_id="7dc5..." + The source was deleted only after the two SHA-256 digests matched. + + 4. file_verify {"uri": "sftp://sftp.example.com/inbound/reports/2026-10-07.csv", + "expected": "sha256:9f86d0..."} + -> match=true (an independent check, for the record) + + 5. audit_search {"correlation_id": "7dc5..."} + -> the mcp.tool.completed record of step 3 and the storage records + (copy, delete) with their resources, durations and status + +**On failure.** When step 3 fails, the client gets an ``error`` and the source +is still in S3. The server publishes ``mcp.tool.failed`` (an error for digests +that differ, a warning for a transfer that failed), plus ``storage.error`` when +a backend failed, and the route of the launcher delivers them to Slack. No tool +sends the notification, so the client cannot skip it. + +The same work as a pipeline that is created once and run daily: + +.. code-block:: text + + pipeline_create { + "name": "export-daily", + "definition": { + "schema_version": 1, + "description": "Move the daily export from S3 to SFTP and verify it", + "tasks": { + "digest": {"action": ["FA_storage_checksum", + {"uri": "s3://reports-export/daily/${params.date}.csv"}]}, + "copy": {"action": ["FA_storage_copy", + {"source": "s3://reports-export/daily/${params.date}.csv", + "target": "sftp://sftp.example.com/inbound/reports/${params.date}.csv"}], + "depends_on": ["digest"], + "retry": {"max_attempts": 3, "backoff": 5}, "timeout": 600}, + "verify": {"action": ["FA_storage_verify", + {"uri": "sftp://sftp.example.com/inbound/reports/${params.date}.csv", + "expected": "${tasks.digest.result}", "strict": true}], + "depends_on": ["copy"]}, + "remove": {"action": ["FA_storage_delete", + {"uri": "s3://reports-export/daily/${params.date}.csv"}], + "depends_on": ["verify"]} } } } + pipeline_run {"name": "export-daily", "params": {"date": "2026-10-07"}, "dry_run": true} + pipeline_run {"name": "export-daily", "params": {"date": "2026-10-07"}} + audit_search {"correlation_id": ""} -After it loads, ask Claude Code to use ``mcp__automation_file__FA_*`` -tools — for example ``"use FA_fast_find to locate every *.log under -./var"``. +``"${tasks.digest.result}"`` hands the checksum of the source to +``FA_storage_verify``. With ``strict: true`` a mismatch fails the task, so +``remove`` is skipped and the source stays. A failed run publishes +``pipeline.failed``, which the same route sends to Slack. -Inspecting the catalogue ------------------------- +The ``FA_*`` bridge +------------------- -Render the same descriptors a host would see — useful for tests, GUI -debugging, or generating documentation: +With the bridge on, ``tools/list`` returns the semantic tools first and then one +tool per registered action, sorted by name, with a JSON Schema derived from the +Python signature. ``tools/call`` on such a tool dispatches through the registry +and answers with the JSON-encoded return value; a failure is a JSON-RPC error. +This is what earlier versions did, and existing configurations keep working. -.. code-block:: python +.. code-block:: text - from automation_file import tools_from_registry, executor + python -m automation_file mcp # semantic tools + every FA_* action + python -m automation_file mcp --allowed-actions FA_list_dir,FA_file_checksum + python -m automation_file mcp --root /srv/reports --no-bridge # semantic tools only - for tool in tools_from_registry(executor.registry): - print(tool["name"], "->", tool["description"]) +``--allowed-actions`` narrows the bridge to the named actions. A call whose +arguments name an action outside that list is refused (``FA_execute_action`` +cannot be used to reach it). ``--no-bridge``, or ``MCPServer(bridge=False)``, +switches the bridge off; an ``FA_*`` name is then an unknown tool. -Each descriptor is shaped as:: +**The policy does not bind the bridge.** ``FA_storage_copy`` called through the +bridge reaches any location the process can, whatever ``--root`` says. For an AI +client use ``--no-bridge``; keep the bridge for tools you would also let the +client call without a policy. - { - "name": "FA_fast_find", - "description": "First docstring line of the underlying callable.", - "inputSchema": { - "type": "object", - "properties": {"root": {"type": "string"}, "pattern": {"type": "string"}}, - "required": ["root", "pattern"], - "additionalProperties": true, - }, - } +From Python: + +.. code-block:: python + + from automation_file import MCPServer, executor, tools_from_registry + from automation_file.server.mcp_policy import MCPPolicy + from automation_file.server.mcp_tools import SemanticToolkit -Manual smoke test + MCPServer().serve_stdio() # as before: bridge on + MCPServer(policy=MCPPolicy(roots=["/srv/reports"]), bridge=False).serve_stdio() + + for tool in tools_from_registry(executor.registry): # the bridge's catalogue + print(tool["name"], "->", tool["description"]) + + toolkit = SemanticToolkit(MCPPolicy(roots=["/srv/reports"])) # the tools without JSON-RPC + outcome = toolkit.call("file_checksum", {"uri": "/srv/reports/a.csv"}) + outcome.is_error, outcome.payload["value"], outcome.correlation_id + +Flags +----- + +``python -m automation_file mcp`` and the ``automation_file_mcp`` console script +take the same flags. + +.. list-table:: + :header-rows: 1 + :widths: 32 68 + + * - Flag + - Meaning + * - ``--name``, ``--version`` + - ``serverInfo`` of the handshake. Defaults: ``automation_file``, ``1.0.0``. + * - ``--allowed-actions a,b`` + - The registered actions the bridge offers. Default: all. + * - ``--no-bridge`` + - Offer only the semantic tools. + * - ``--root URI`` + - An allowed location; repeatable. + * - ``--allow-write``, ``--allow-overwrite``, ``--allow-delete`` + - The three permissions. The last two need the first. + * - ``--max-read-bytes N``, ``--max-write-bytes N``, ``--max-results N``, + ``--max-search-bytes N`` + - The limits. + * - ``--pipeline-dir URI`` + - Where pipeline definitions are kept. + * - ``--pipeline-actions a,b`` + - The actions a pipeline run through MCP may call. + * - ``--tools a,b`` + - The semantic tools to offer, or ``none``. + +A wrong flag (an unknown tool name, ``--allow-overwrite`` without +``--allow-write``, a root with credentials in it) ends the command with a usage +error before anything is served. + +Security guidance ----------------- -Pipe JSON-RPC frames straight at the server to confirm it loads:: - - printf '%s\n%s\n%s\n' \ - '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{}}' \ - '{"jsonrpc":"2.0","id":2,"method":"tools/list"}' \ - '{"jsonrpc":"2.0","id":3,"method":"tools/call","params":{"name":"FA_fast_find","arguments":{"root":".","pattern":"*.py","limit":3}}}' \ - | python -m automation_file mcp - -Each input line gets exactly one JSON-RPC reply on stdout (notifications -have no reply). - -Security considerations ------------------------ - -* The MCP server runs with **the same privileges as the Python process - that starts it.** Every tool call lands in your shell's user account. -* By default the server exposes **every** registered ``FA_*`` action, - including ones that delete files, write to the filesystem, or upload - to remote backends. Use ``--allowed-actions`` to whitelist a tight - surface for any host you don't fully trust. -* Do **not** call - :func:`~automation_file.PackageLoader.add_package_to_executor` (or - expose its action) before starting an MCP server intended for a - third-party host. That helper registers every top-level function / - class / builtin of an arbitrary package and is eval-grade power. -* The server logs at the ``INFO`` level via - ``file_automation_logger`` when it starts, including the number of - exposed tools and the configured server name. Tool-call payloads are - never logged. -* Outbound HTTP-bearing actions (``FA_download_file``, the cloud - backends) keep their SSRF guard; the MCP layer does not loosen any - per-action check. +* **The process's privileges are the outer limit.** The server runs as the user + who started it, with the credentials its backends were given. The policy + narrows that for the semantic tools; it does not replace least privilege on + the account, the bucket policy or the SFTP user. +* **Switch the bridge off for AI clients** (``--no-bridge``). The bridge offers + every registered action, ``FA_run_shell`` and ``FA_storage_delete`` included, + and the policy does not apply to it. +* **Keep roots narrow.** A root is a grant for everything below it. Do not use + ``local:///`` or a home directory. Put what a client may write in a directory + of its own. +* **Start read-only.** Add ``--allow-write`` when the work needs it, and + ``--allow-overwrite`` and ``--allow-delete`` only when it needs those. A + client that can write but not overwrite or delete cannot destroy what is + already there. +* **What a file contains is not an instruction.** A model reads file content + through ``file_read`` and ``file_search`` and may act on what it reads. The + policy bounds what such a hijacked session can do; so do the confirmation + prompts of the host. Ask for a ``dry_run`` first on anything that matters. +* **Pipelines.** The default actions stay inside the roots. Every action you add + with ``--pipeline-actions`` runs unconfined. The pipeline directory holds + definitions an AI client wrote: only ``pipeline_run`` applies the policy to + them, so do not run them with ``FA_pipeline_run`` or ``Pipeline.from_file``, + and do not point a scheduler at that directory. +* **The reporting tools are not bound by the roots.** ``audit_search``, + ``integrity_status`` and ``pipeline_status`` show resource names, errors and + run parameters from the whole process. Leave them out of ``--tools`` for a + client that must not see them. +* **The client's name proves nothing.** It labels the actor in the audit trail + and comes from the client itself. Stdio has no authentication: whoever can + start the process has its powers. +* **Secrets.** Credentials never belong in a storage URI (one that carries them + is refused) or in pipeline parameters, which are recorded with the run. The + log holds no argument values and no file content. +* **Network.** The semantic tools make no HTTP request of their own. The storage + backends keep their checks: TLS verification, SFTP host keys, and the SSRF + guard of the actions that fetch a URL. +* Do not call ``PackageLoader.add_package_to_executor`` in a process that serves + the bridge: it registers every member of a package as an action. + +When something goes wrong +------------------------- + +``no storage location is allowed on this server`` + No root is configured. Add ``--root ``, or pass + ``MCPPolicy(roots=[...])``. + +``... is outside the allowed locations (...)`` + The location is not at or below a root; the message lists the roots. Check + the spelling against the root: another bucket or host, a sibling directory + (``reports-old`` next to ``reports``), another letter case in the authority, + or another path to the same directory. + +``... leaves the allowed location through a link or an absolute path`` + A symbolic link or a junction below a local root points outside it. Add the + link's target as a root if the client should reach it. + +``this server is read-only``, ``does not allow it`` + The permission is off: ``--allow-write``, ``--allow-overwrite`` or + ``--allow-delete``. ``file_move`` needs writing and deleting. + +``already exists`` + Pass ``overwrite=true``; the server must allow overwriting as well. + +``file_read`` returns only part of a file + ``truncated`` is true: call again with ``offset=next_offset``, or raise + ``--max-read-bytes``. For a backend without ranged reads the whole file is + staged locally for each call, so page through a very large remote file + sparingly. + +``file_search`` misses a file + Look at ``complete`` and ``skipped``. A file larger than what is left of + ``--max-search-bytes`` is not searched; narrow ``pattern`` or raise the + budget. Content is matched as UTF-8 text. + +``... is not allowed in a pipeline`` + The definition names an action outside the allowed set; the message lists + that set. Use the storage actions, or add the action with + ``--pipeline-actions``. + +A pipeline task fails with ``MCPLocationException`` or ``MCPPermissionException`` + The task reached a location outside the roots, or needed a permission that + is off, when it ran. The definition was accepted because the location came + from a parameter. + +``no pipeline named ... is stored`` + The message lists the stored names. Without ``--pipeline-dir`` definitions + are gone when the server restarts. + +``pipeline_status`` does not know a run + Runs are kept in memory by default. Call + ``set_default_run_store(SQLiteRunStore(path))`` in a launcher. + +``this server keeps no audit trail`` + Call ``configure_audit(path)`` in a launcher before ``serve_stdio()``. + +An ``sftp://`` or ``s3://`` location fails although it is inside a root + The backend is not initialised in this process. Call its ``later_init`` in a + launcher; see `Production setup`_ and :doc:`storage`. + +The host lists hundreds of tools, or none of the semantic ones + The first is the bridge: add ``--no-bridge`` or ``--allowed-actions``. The + second is ``--tools none``, or a version of the package from before the + semantic tools. + +The host reports a broken connection at start + Something wrote to ``stdout``, or the command ended with a usage error. + Run the same command in a terminal: errors go to ``stderr``. To check a + server by hand:: + + printf '%s\n%s\n' \ + '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{}}' \ + '{"jsonrpc":"2.0","id":2,"method":"tools/list"}' \ + | python -m automation_file mcp --root /srv/reports --no-bridge + +Every exception of the semantic tools derives from ``MCPServerException``, and so +from ``FileAutomationException``: ``MCPPermissionException`` (with ``code``), +its subclass ``MCPLocationException``, and ``MCPToolException`` (with ``kind``). diff --git a/docs/source/Zh-CN/usage/cli.rst b/docs/source/Zh-CN/usage/cli.rst index e0dbd28..0ac4781 100644 --- a/docs/source/Zh-CN/usage/cli.rst +++ b/docs/source/Zh-CN/usage/cli.rst @@ -17,12 +17,15 @@ CLI python -m automation_file create-file hello.txt --content "hi" python -m automation_file server --host 127.0.0.1 --port 9943 python -m automation_file http-server --host 127.0.0.1 --port 9944 + python -m automation_file mcp --root /srv/reports --no-bridge python -m automation_file mcp --allowed-actions FA_list_dir,FA_file_checksum python -m automation_file drive-upload my.txt --token token.json --credentials creds.json -``mcp`` 子命令通过 stdio 启动 Model Context Protocol 服务器， -让 Claude Desktop 这类宿主可以把 ``FA_*`` 动作当作 MCP 工具调用——完整集成 -说明请见 :doc:`mcp`。 +``mcp`` 子命令通过 stdio 启动 Model Context Protocol 服务器，让 Claude Desktop 这类 +宿主可以处理文件：一是通过语义工具（``file_read``、``storage_copy``、 +``pipeline_run`` ……），它们只在 ``--root`` 指定的位置之内工作，而且在 +``--allow-write`` 之前都是只读的；二是通过把 ``FA_*`` 动作当作 MCP 工具提供的桥接。 +命令行参数、权限模型与完整集成说明请见 :doc:`mcp`。 存储 ---- diff --git a/docs/source/Zh-CN/usage/mcp.rst b/docs/source/Zh-CN/usage/mcp.rst index a6d6f93..9142da4 100644 --- a/docs/source/Zh-CN/usage/mcp.rst +++ b/docs/source/Zh-CN/usage/mcp.rst @@ -1,69 +1,26 @@ -MCP 服务器（Claude Desktop / Claude Code） -========================================== - -``automation_file`` 自带一个 Model Context Protocol（MCP）服务器， -将共享的 :class:`~automation_file.core.action_registry.ActionRegistry` -中的每一项暴露为 MCP 工具。**Claude Desktop**、**Claude Code** -以及其他 MCP 宿主，可以像调用普通 MCP 工具一样调用 ``FA_*`` 动作—— -无需自写插件，无需额外打包。 - -传输方式为 **stdio**（``stdin`` / ``stdout`` 上每行一条 JSON-RPC 2.0 消息）， -与目前 MCP 宿主使用的协议一致。 - -提供的能力 ----------- - -* ``initialize`` 握手，返回协议版本 ``2024-11-05``、 - ``serverInfo.name`` 与 ``serverInfo.version``。 -* ``tools/list`` 为每个已注册的 ``FA_*`` 动作返回一个 MCP 工具描述， - 其参数 JSON Schema 由 Python 函数签名自动派生 - （``str → "string"``、``int → "integer"`` 等）。 -* ``tools/call`` 通过注册表派发，并以 JSON 编码后的文本内容块返回结果。 -* ``--allowed-actions`` 允许列表参数，让你只把注册表的子集暴露给宿主。 -* 所有内部错误都会以 JSON-RPC 错误对象形式返回—— - 宿主无需解析异常字符串即可呈现错误。 - -启动服务器 ----------- - -CLI:: - - python -m automation_file mcp - python -m automation_file mcp --name automation_file --version 1.0.0 - python -m automation_file mcp --allowed-actions FA_list_dir,FA_file_checksum - -进程在前台运行，从 ``stdin`` 读取分行 JSON 并把响应写到 ``stdout``。 -通常宿主会替你启动该进程，不需要手工运行。 - -从 Python 启动（例如嵌入到自家 stdio 桥接器里）： - -.. code-block:: python - - from automation_file import MCPServer - - server = MCPServer(name="automation_file", version="1.0.0") - server.serve_stdio() # stdin 关闭前阻塞 - -通过自定义注册表把可调用动作面缩小： +MCP 服务器 +==================== -.. code-block:: python +``automation_file`` 内置一个 Model Context Protocol（MCP）服务器，让 **Claude Desktop** +或 **Claude Code** 这类 AI 客户端可以通过它处理文件。传输方式是 stdio：每行一条 +JSON-RPC 2.0 消息。 - from automation_file import MCPServer - from automation_file.core.action_registry import ActionRegistry - from automation_file import executor +服务器提供两组工具： - safe = ActionRegistry() - for name in ("FA_list_dir", "FA_file_checksum", "FA_fast_find"): - safe.register(name, executor.registry.resolve(name)) +* **语义工具**：十四个名称稳定、以任务为单位的工具（``file_read``、``file_copy``、 + ``pipeline_run`` ……），操作对象是 :doc:`存储 URI `，并受一份权限策略 + 约束。这是为 AI 客户端设计的接口； +* **桥接**：每个已注册的 ``FA_*`` 动作各自成为一个工具，与之前的版本相同。为了兼容， + 它默认开启，而且 **不受** 策略约束。 - MCPServer(safe).serve_stdio() +语义工具的默认值是安全的：不允许任何位置、不能改动任何东西，每个响应的大小也都有 +上限。 -Claude Desktop 配置 -------------------- +最小配置 +---------------- -在 ``~/Library/Application Support/Claude/claude_desktop_config.json`` -（macOS）或 ``%APPDATA%\Claude\claude_desktop_config.json`` -（Windows）的 ``mcpServers`` 下添加条目： +给服务器一个可以读取的目录。写在 ``claude_desktop_config.json``\ （Claude Desktop）或 +``.mcp.json``\ （Claude Code）里： .. code-block:: json @@ -71,119 +28,755 @@ Claude Desktop 配置 "mcpServers": { "automation_file": { "command": "python", - "args": ["-m", "automation_file", "mcp"] + "args": ["-m", "automation_file", "mcp", "--root", "/srv/reports", "--no-bridge"] } } } -重启 Claude Desktop。``automation_file`` 服务器会出现在工具面板中， -所有 ``FA_*`` 动作都可被调用。 +客户端现在会看到十四个语义工具。它可以列出、读取、搜索 ``/srv/reports`` 下面的 +内容并计算校验和，除此之外什么都不能做：写入会被拒绝，该目录以外的每个路径也一样。 +在 Windows 上路径要写成 ``"C:\\data\\reports"``。如果 ``PATH`` 上的 ``python`` 不是 +安装本包的那一个，请改用该环境的解释器 +（``"command": "C:\\envs\\fa\\Scripts\\python.exe"``）。 + +使用 Claude Code 时，同一个服务器可以从 shell 添加:: + + claude mcp add automation_file -- python -m automation_file mcp --root /srv/reports --no-bridge -对于操作敏感路径的宿主，建议把可用动作锁死为白名单： +生产环境配置 +------------------------ + +列出每一个位置、只开启工作需要的权限、把流水线的定义存在磁盘上，并且关闭桥接： .. code-block:: json { "mcpServers": { "automation_file": { - "command": "python", + "command": "/opt/fa/bin/python", "args": [ "-m", "automation_file", "mcp", - "--allowed-actions", - "FA_list_dir,FA_fast_find,FA_file_checksum,FA_verify_checksum" + "--root", "/srv/reports/inbox", + "--root", "/srv/reports/outbox", + "--allow-write", + "--max-read-bytes", "65536", + "--max-results", "100", + "--pipeline-dir", "/var/lib/automation_file/pipelines", + "--tools", "file_read,file_write,file_copy,file_search,file_checksum,file_verify,storage_list,storage_copy", + "--no-bridge" ] } } } -显式指定虚拟环境的解释器（避免误用系统 Python）： +远端后端必须先初始化客户端，审计轨迹与通知路由也要在服务器启动之前配置好。这需要 +几行 Python，所以请用一个启动脚本来启动服务器，再把宿主的 ``command`` 指向它： + +.. code-block:: python + + # /opt/fa/mcp_server.py + import os + + from automation_file import ( + MCPServer, Route, Severity, SlackSink, configure_audit, + notification_manager, notification_router, s3_instance, sftp_instance, + ) + from automation_file.server.mcp_policy import MCPPolicy + + s3_instance.later_init(region_name="eu-west-1") # 凭据：AWS 的默认来源链 + sftp_instance.later_init(host="sftp.example.com", username="reports", + key_filename="/etc/fa/id_ed25519", + known_hosts="/etc/fa/known_hosts") + configure_audit("/var/lib/automation_file/audit.sqlite") # audit_search 读的就是这里 + notification_manager.register(SlackSink(os.environ["SLACK_WEBHOOK"], name="ops")) + notification_router.add_route(Route( + "mcp-failures", sinks=("ops",), + types=("mcp.tool.failed", "pipeline.failed", "storage.error"), + min_severity=Severity.WARNING, + )) + notification_router.start() + + policy = MCPPolicy( + roots=["s3://reports-export/daily", "sftp://sftp.example.com/inbound/reports"], + allow_write=True, + allow_delete=True, # file_move 会删除来源 + max_read_bytes=64 * 1024, + pipeline_dir="/var/lib/automation_file/pipelines", + ) + MCPServer(policy=policy, bridge=False).serve_stdio() + +.. code-block:: json + + {"mcpServers": {"automation_file": {"command": "/opt/fa/bin/python", + "args": ["/opt/fa/mcp_server.py"]}}} + +``stdout`` 上除了协议之外什么都不能写：库的日志写到 ``stderr`` 与它的日志文件， +启动脚本也不可以 ``print``。 + +语义工具 +---------------- + +每个位置都是一个存储 URI，``:///`` +（``local:///srv/reports/a.csv``、``s3://bucket/2026/a.csv``、 +``sftp://host/inbox/a.csv``），或者是本地的绝对路径。“需要”一栏列出：除了要有一个 +涵盖这次调用所有位置的根位置之外，策略还必须允许什么。 + +.. list-table:: + :header-rows: 1 + :widths: 14 30 36 20 + + * - 工具 + - 参数 + - 结果 + - 需要 + * - ``file_read`` + - ``uri``、``offset=0``、``max_bytes``、``encoding="utf-8"``\ （文本编码，或 + ``base64``） + - ``content``、``encoding``、``size``、``offset``、``bytes``、 + ``truncated``、``next_offset`` + - — + * - ``file_write`` + - ``uri``、``content``、``encoding="utf-8"``、``overwrite=false``、 + ``dry_run=false`` + - ``size``、``sha256``、``overwrites``、``replaced_size``、``written`` + - 写入；要替换文件时还需要覆盖 + * - ``file_copy`` + - ``source``、``target``、``overwrite=false``、``verify=false``、 + ``dry_run=false`` + - ``source``、``target``、``size``、``overwrites``、``replaced_size``、 + ``deletes_source``、``done``；使用 ``verify`` 时另有 ``sha256``、 + ``verified`` + - 写入；要替换文件时还需要覆盖 + * - ``file_move`` + - 与 ``file_copy`` 相同 + - 与 ``file_copy`` 相同；``deletes_source`` 为 true + - 写入与删除；要替换文件时还需要覆盖 + * - ``file_search`` + - ``uri``、``pattern="*"``、``content``、``recursive=true``、 + ``case_sensitive=false``、``max_results`` + - ``matches``\ （``uri``、``path``、``name``、``size``、``modified_at``； + 内容搜索另有 ``line``、``snippet``、``matching_lines``）、``count``、 + ``candidates``、``truncated``；内容搜索另有 ``searched_files``、 + ``searched_bytes``、``skipped``、``complete`` + - — + * - ``file_checksum`` + - ``uri``、``algorithm="sha256"`` + - ``algorithm``、``value``、``size`` + - — + * - ``file_verify`` + - ``uri``、``expected``\ （十六进制，或 ``sha256:``）、 + ``algorithm="sha256"`` + - ``match``、``expected``、``actual``、``algorithm`` + - — + * - ``storage_list`` + - ``uri``、``recursive=false``、``max_results`` + - ``entries``\ （``uri``、``path``、``name``、``is_dir``、``size``、 + ``modified_at``）、``count``、``total``、``truncated`` + - — + * - ``storage_copy`` + - ``source``、``target``、``overwrite=false``、``verify=false``、 + ``dry_run=false`` + - 文件：``kind="file"`` 加上 ``file_copy`` 的结果。目录：``kind="tree"``、 + ``planned``\ （``copy``、``overwrite``、``skip``、``bytes``）、``paths``、 + ``existing``、``truncated``、``done``，复制之后另有 ``copied``、 + ``skipped``、``failed``、``errors``、``ok`` + - 写入；要替换文件时还需要覆盖 + * - ``pipeline_create`` + - ``name``、``definition``、``overwrite=false``、``dry_run=false`` + - ``name``、``location``、``persistent``、``tasks``、``actions``、 + ``overwrites``、``stored`` + - 写入；要替换定义时还需要覆盖 + * - ``pipeline_run`` + - ``name``、``params``、``dry_run=false``、``background=false`` + - ``run_id``、``status``、``ok``、``run``\ （这次运行与它的每个任务） + - 写入；每个任务需要它的动作所需要的权限 + * - ``pipeline_status`` + - ``run_id``，或 ``name`` 与 ``limit=5`` + - ``runs``、``count`` + - 不需要根位置 + * - ``integrity_status`` + - ``name`` + - ``monitors``\ （``name``、``target``、``running``、``last_run``、 + ``last_error``、``last_report``）、``count`` + - 不需要根位置 + * - ``audit_search`` + - ``actor``、``source``、``pipeline``、``task``、``action``、``backend``、 + ``status``、``correlation_id``、``resource_prefix``、``text``、``since``、 + ``until``、``limit=50``、``offset=0`` + - ``records``、``count``、``total``、``limit``、``offset``、``truncated`` + - 不需要根位置；需要已配置的审计轨迹 + +各个工具的说明： + +``file_read`` + 每次调用最多返回 ``--max-read-bytes`` 个字节。``truncated`` 为 true 时，把 + ``offset`` 设成 ``next_offset`` 再调用一次；字符绝不会被切成两半。内容不是所选 + 编码的文本时会报告错误，并要求改用 ``encoding="base64"``。 + +``file_copy``、``file_move`` 与 ``verify`` + ``verify=true`` 时会比较来源与副本的 SHA-256。此时移动会先复制、再比较，只有在 + 两个摘要一致时才删除来源；不一致时来源保留，调用以 ``checksum_mismatch`` 失败。 + +``file_verify`` + 不一致是一个结果（``match`` 为 false），不是错误。 + +``file_search`` + ``pattern`` 是匹配文件名的 shell 模式（``*.csv``）；模式里有 ``/`` 时，匹配的是 + ``uri`` 下面的相对路径（``2026/*/*.csv``）。``content`` 是普通的子串，不是正则 + 表达式。一次内容搜索最多读取 ``--max-search-bytes`` 个字节。超过剩余额度的 + 文件不会被打开，而是列在 ``skipped`` 里；二进制文件与读不到的文件也一样。只有在 + 每个候选文件都搜索过时，``complete`` 才是 true。 + +``storage_copy`` + 像 ``file_copy`` 一样复制一个文件，或复制某个目录下面的所有文件。对目录而言， + 除非 ``overwrite`` 为 true，否则目标已有的文件会被跳过。失败的文件记在 + ``errors`` 里，其他文件照样复制；这时调用是一个错误，但结果里仍有各项数量。 + +``pipeline_run`` + 在这次请求里运行，并返回已结束的运行。``background=true`` 时立刻返回 + ``status="running"``；请用 ``run_id`` 轮询 ``pipeline_status``。失败的运行是 + 一个错误，但结果里仍带有这次运行。 + +结果与错误 +~~~~~~~~~~~~~~~~~~~~ + +语义工具以一份 JSON 文档作为结果的文本响应。其中一定有 ``tool`` 与 +``correlation_id``。调用被拒绝或失败时，``isError`` 为 true，文档里会有 ``error``： .. code-block:: json { - "mcpServers": { - "automation_file": { - "command": "C:\\envs\\fa\\Scripts\\python.exe", - "args": ["-m", "automation_file", "mcp"] - } + "tool": "file_write", + "correlation_id": "27acff55529c4317b74de6ea98759f42", + "error": { + "type": "permission_denied", + "code": "read_only", + "message": "file_write changes something and this server is read-only: start it with --allow-write, or pass MCPPolicy(allow_write=True)" } } -Claude Code 配置 +.. list-table:: + :header-rows: 1 + :widths: 26 74 + + * - ``error.type`` + - 含义 + * - ``permission_denied`` + - 策略拒绝了这次调用。``code`` 指出是哪一条规则：``no_root``、 + ``outside_root``、``read_only``、``overwrite_not_allowed``、 + ``delete_not_allowed``、``tool_disabled``、``action_not_allowed`` 或 + ``limit_exceeded``。 + * - ``invalid_arguments`` + - 参数缺少、未知、类型错误或超出范围。每个问题都会列出。 + * - ``invalid_uri`` + - 位置不是存储 URI，或它的路径含有 ``..`` 段。 + * - ``not_found``、``already_exists`` + - 没有这个文件、流水线、运行或监控器；或目标已存在而没有给 ``overwrite``。 + * - ``checksum_mismatch`` + - ``verify`` 发现两边的摘要不同。 + * - ``invalid_definition`` + - 流水线定义有误；``problems`` 列出每一项发现与它的路径。 + * - ``not_configured`` + - 服务器没有保存审计轨迹。 + * - ``failed`` + - 库报告的其他错误；``exception`` 是异常类的名称。 + * - ``internal_error`` + - 非预期的异常。请报告。 + +权限模型 ---------------- -Claude Code 使用同一份 MCP 定义。可通过 ``claude mcp add`` CLI 注册:: - - claude mcp add automation_file -- python -m automation_file mcp +:class:`~automation_file.server.mcp_policy.MCPPolicy` 在服务器启动时创建，之后无法 +更改。它的文字描述会放在握手响应的 ``instructions`` 里发给客户端，所以模型在第一次 +调用之前就知道边界在哪里。 + +.. list-table:: + :header-rows: 1 + :widths: 24 24 14 38 + + * - 字段 + - 命令行参数 + - 默认值 + - 含义 + * - ``roots`` + - ``--root``\ （可重复） + - 无 + - 工具可以工作的位置：存储 URI 或本地目录。一个都没有时，每个会碰到存储的 + 工具都会拒绝，并说明如何添加根位置。 + * - ``allow_write`` + - ``--allow-write`` + - 关闭 + - 没有它，``file_write``、``file_copy``、``file_move``、``storage_copy``、 + ``pipeline_create`` 与 ``pipeline_run`` 都会被拒绝，试运行也一样。 + * - ``allow_overwrite`` + - ``--allow-overwrite`` + - 关闭 + - 替换已有的文件或定义。调用本身也必须提出要求（``overwrite=true``）。需要 + ``allow_write``。 + * - ``allow_delete`` + - ``--allow-delete`` + - 关闭 + - 删除。``file_move`` 会删除来源，所以需要它。需要 ``allow_write``。 + * - ``max_read_bytes`` + - ``--max-read-bytes`` + - 262144 + - ``file_read`` 一次调用最多返回的字节数，也是流水线工具返回的任务结果的 + 大小上限。 + * - ``max_write_bytes`` + - ``--max-write-bytes`` + - 1048576 + - ``file_write`` 接受的内容大小上限，以及 ``pipeline_create`` 存储的定义大小 + 上限。 + * - ``max_results`` + - ``--max-results`` + - 200 + - 列表、搜索、目录复制计划或 ``audit_search`` 最多返回的条目数。调用里较大的 + ``max_results`` 或 ``limit`` 会被降到这个值。 + * - ``max_search_bytes`` + - ``--max-search-bytes`` + - 8388608 + - 一次内容搜索最多读取的字节数。 + * - ``pipeline_dir`` + - ``--pipeline-dir`` + - 内存 + - ``pipeline_create`` 存放定义的地方：存储 URI 或本地目录，每条流水线一个 + ``.json``。没有配置时定义保存在内存里，服务器停止就消失。 + * - ``pipeline_actions`` + - ``--pipeline-actions`` + - 按权限而定 + - 通过 MCP 创建或运行的流水线可以调用的动作。见 `流水线`_。 + * - ``tools`` + - ``--tools`` + - 全部十四个 + - 要提供的语义工具。没有列入的工具不会出现在列表里，调用它也会被拒绝。 + ``--tools none`` 一个都不提供。 + * - ``actor`` + - （仅限 Python） + - ``mcp`` + - 事件与审计记录里每次调用的 actor。握手时客户端报告的名称会接在后面： + ``mcp:claude-desktop``。 + +位置 +~~~~~~~~ + +调用里提到的每个位置都在某个根位置之内（或就是根位置本身）时，调用才被允许。 + +* **本地根位置** 由存储层自己把关。该位置由一个限制在根位置内的 ``LocalStorage`` + 提供服务，所以每个操作都会经过 ``safe_join``：指向根位置之外的符号链接（或 + Windows 的 junction），以及夹带到根位置下面的绝对路径，都会以 ``outside_root`` + 被拒绝。留在根位置之内的链接则会被跟随。在 Windows 上比较不区分大小写，也接受 + 反斜杠；名称是设备（``CON``、``NUL``、``COM1``）或备用数据流 + （``a.txt:stream``）的路径同样会被拒绝。 +* **其他后端** 按 scheme、authority 与完整的路径段比较。``s3://bucket/team`` 允许 + ``s3://bucket/team/2026/a.csv``，并拒绝 ``s3://bucket/team-b/a.csv`` 与 + ``s3://other/team/a.csv``。authority 必须完全相同，包括大小写。根位置只是路径的 + 前缀，仅此而已：服务器上已有的链接（例如 SFTP 的符号链接）会由服务器跟随。请给 + 后端一个无法离开该目录树的账号。 +* 含有 ``..`` 段的路径根本不是存储 URI（``invalid_uri``）。 +* 用 ``Storage.mount`` 挂载的后端要通过它的挂载 URI 来指定；位于已挂载的 + ``LocalStorage`` 之下的根位置，会被限制在该根位置之内。 + +位置的写法要和根位置的写法一致。以 ``/srv/reports`` 给出的根位置不会匹配 +``local:///mnt/disk2/reports``，即使两者是同一个目录。 + +流水线 +~~~~~~~~~~~~ + +``pipeline_create`` 存储一份定义（:doc:`pipeline`，``schema_version: 1``）， +``pipeline_run`` 则运行它。两者都需要 ``allow_write``。 + +定义可以调用什么，在创建时检查一次，运行时再检查一次，因为这段时间文件可能被别的 +东西改写。默认情况下，流水线可以调用权限所涵盖的 ``FA_storage_*`` 动作： + +.. list-table:: + :header-rows: 1 + :widths: 30 70 + + * - 权限 + - 动作 + * - 始终可用 + - ``FA_storage_exists``、``FA_storage_stat``、``FA_storage_list``、 + ``FA_storage_checksum``、``FA_storage_verify``、``FA_storage_read_text``、 + ``FA_storage_schemes`` + * - ``allow_write`` + - ``FA_storage_mkdir``、``FA_storage_write_text``、``FA_storage_copy``、 + ``FA_storage_copy_tree`` + * - ``allow_overwrite`` + - ``FA_storage_sync``\ （它会替换有变动的文件） + * - ``allow_delete`` + - ``FA_storage_move``、``FA_storage_delete`` + +在这样的流水线里，这些名称运行的不是原本的动作，而是受防护的版本：它们在任务运行的 +当下检查根位置与权限，也就是在 ``${params.}`` 与 ``${tasks..result}`` +填入之后，所以参数无法把任务带到根位置之外。它们的行为与原本的动作相同，区别如下： + +* ``overwrite`` 的默认值是策略所允许的，而不是 ``true``；在不允许覆盖的地方，显式 + 写出的 ``overwrite: true`` 会被拒绝； +* ``FA_storage_verify`` 的 ``expected`` 也接受 ``FA_storage_checksum`` 的结果，所以 + ``"${tasks..result}"`` 可以用来比较两个文件； +* ``FA_storage_read_text`` 与 ``FA_storage_write_text`` 遵守读取与写入的上限； +* ``FA_storage_sync`` 需要 ``allow_overwrite``，``delete: true`` 还需要 + ``allow_delete``； +* ``FA_storage_delete`` 绝不会移除根位置本身； +* 没有 ``FA_storage_upload`` 与 ``FA_storage_download``：它们接受的是文件系统路径， + 不是存储 URI。请改为复制到 ``local://`` URI，或从它复制出来。 + +``--pipeline-actions a,b,c`` 会替换默认的列表。上面的 ``FA_storage_*`` 动作被列入时 +仍然受到防护。**其他动作则照原样运行，不受根位置与权限约束**：只列出你本来就愿意 +让客户端直接调用的动作，而且绝不要列出会运行其他动作的动作 +（``FA_execute_action``、``FA_pipeline_run``、``FA_run_shell``）。每个名称都必须是 +服务器提供的动作，否则服务器不会启动。出现在另一个动作的参数里、或运行参数里的动作 +会被拒绝，除非列表里有它；存储动作在那里则一律被拒绝，因为嵌套运行时它不受防护。 + +通过 MCP 创建的定义不能带有 ``schedule``：流水线何时自行运行，由运维服务器的人决定。 +定义的 ``name`` 就是它存储时使用的名称；省略时会自动填入。 + +运行记录存在默认的运行记录存储里，``pipeline_status`` 与 ``FA_pipeline_status`` 都 +从那里查找。除非启动脚本调用 ``set_default_run_store(SQLiteRunStore(path))``，否则 +它只存在内存里。 + +试运行 +------------ -或在仓库根目录提交 ``.mcp.json``： +每个会改动东西的工具都接受 ``dry_run``。这时它会做完真正调用要做的每一项检查， +不改动任何东西，并返回计划： .. code-block:: json { - "mcpServers": { - "automation_file": { - "command": "python", - "args": ["-m", "automation_file", "mcp"] + "tool": "file_move", + "correlation_id": "bb62a807d4f64bf3b666d00d3b4f410f", + "source": "s3://reports-export/daily/2026-10-07.csv", + "target": "sftp://sftp.example.com/inbound/reports/2026-10-07.csv", + "size": 48211, + "overwrites": false, + "replaced_size": null, + "deletes_source": true, + "dry_run": true, + "done": false + } + +.. list-table:: + :header-rows: 1 + :widths: 24 76 + + * - 工具 + - 试运行报告的内容 + * - ``file_write`` + - 内容的大小与 SHA-256、是否会替换某个文件，以及该文件的大小。 + * - ``file_copy``、``file_move`` + - 来源、目标、大小、目标是否会被替换、来源是否会被删除。 + * - ``storage_copy`` + - 对目录：会复制、替换与跳过多少文件与字节，以及路径（受 ``max_results`` + 限制）。 + * - ``pipeline_create`` + - 定义是否有效且被允许、它的任务与动作，以及是否会替换已存储的定义。 + * - ``pipeline_run`` + - 按依赖顺序排列、状态为 ``planned`` 的任务；如果任务指定了未知的动作，或用到 + 这次运行没有给的参数，会在该任务上附 ``error``。不会执行也不会记录任何 + 东西。 + +试运行需要的权限与真正的调用相同：只读的服务器同样会拒绝它，所以服务器不会做的事， +也不会给出计划。 + +可追溯性 +---------------- + +每次语义调用都在 ``correlation_scope()`` 与 ``actor_scope(...)`` 之内运行。结果带有 +``correlation_id``，这次调用所做的一切也都带有它：存储操作，以及每次调用一条的 +事件：``mcp.tool.completed`` 或 ``mcp.tool.failed``\ （source 为 ``mcp``）。 + +.. list-table:: + :header-rows: 1 + :widths: 26 14 60 + + * - 结果 + - 严重级别 + - 事件 + * - 完成 + - info + - ``mcp.tool.completed``，``status="ok"`` + * - 被策略拒绝 + - warning + - ``mcp.tool.failed``，``status="refused"``，``code`` 指出规则 + * - 请求本身有误：文件不存在、目标已存在、参数错误、定义无效 + - info + - ``mcp.tool.failed``，``status="error"``，``code`` 就是 ``error.type`` + * - 失败：后端错误、目录复制中有文件失败、流水线运行失败 + - warning + - ``mcp.tool.failed``，``status="error"``。失败的后端另外以 + ``storage.error`` 报告，失败的运行则以 ``pipeline.failed`` 报告。 + * - ``verify`` 发现摘要不同 + - error + - ``mcp.tool.failed``，``status="error"``，``code="checksum_mismatch"`` + * - 非预期的异常 + - error + - ``mcp.tool.failed``，``status="error"``，``code="internal_error"`` + +因此，一条针对 ``mcp.tool.failed``、``min_severity=Severity.WARNING`` 的通知路由会 +收到拒绝与真正的失败，而不会收到“模型要了一个不存在的文件”这种事。 + +事件的 payload 记有工具名称（``action``）、这次调用涉及的存储 URI（``resource``、 +``source_uri``）、``duration_ms``，流水线工具另有 ``pipeline`` 与 ``run_id``。它绝不 +包含内容、参数或摘要。被拒绝的调用也会以 warning 写进日志，内容是工具、规则与关联 +ID，不含任何参数值。 + +配置了 :doc:`审计轨迹 ` 之后，一次搜索就能看到某次调用做了什么： + +.. code-block:: python + + audit_search(correlation_id="7dc51e94bd3b494eae8e6b6f3f3b150b") + # [{"source": "mcp", "action": "mcp.tool.completed", "actor": "mcp:claude-desktop", ...}, + # {"source": "storage", "action": "copy", "resource": "sftp://...", "status": "ok", ...}] + +流水线的一次运行有它自己的关联 ID，也就是它的 ``run_id``。``pipeline_run`` 那次调用的 +事件里记有这个 ``run_id``，两者由此关联起来。 + +示例：从 S3 到 SFTP，经过验证与审计，失败时发出告警 +------------------------------------------------------------------------ + +任务内容：*把昨天的 CSV 从 S3 移到公司的 SFTP 服务器，验证它的 SHA-256，审计这次 +传输，失败时通知 Slack*。`生产环境配置`_ 的启动脚本已经准备好所需的一切：两个后端都 +已初始化、两个根位置、允许写入与删除、一份审计轨迹，以及一条把失败发到 Slack 的 +路由。接着客户端进行这些调用： + +.. code-block:: text + + 1. storage_list {"uri": "s3://reports-export/daily"} + -> 条目列表；客户端选出 2026-10-07.csv + + 2. file_move {"source": "s3://reports-export/daily/2026-10-07.csv", + "target": "sftp://sftp.example.com/inbound/reports/2026-10-07.csv", + "verify": true, "dry_run": true} + -> 计划：size、overwrites=false、deletes_source=true + + 3. file_move 同样的参数，去掉 dry_run + -> done=true、verified=true、sha256="9f86d0..."、correlation_id="7dc5..." + 两个 SHA-256 摘要一致之后，来源才被删除。 + + 4. file_verify {"uri": "sftp://sftp.example.com/inbound/reports/2026-10-07.csv", + "expected": "sha256:9f86d0..."} + -> match=true（独立的检查，留作记录） + + 5. audit_search {"correlation_id": "7dc5..."} + -> 第 3 步的 mcp.tool.completed 记录，以及存储记录（copy、delete）， + 附有资源、耗时与状态 + +**失败时。** 第 3 步失败时，客户端会收到 ``error``，来源仍然在 S3 里。服务器会发布 +``mcp.tool.failed``\ （摘要不同时是 error，传输失败时是 warning），后端失败时另外 +发布 ``storage.error``，启动脚本配置的路由再把它们发到 Slack。通知不是由任何工具 +发出的，所以客户端无法跳过它。 + +同一件工作也可以写成流水线，创建一次、每天运行： + +.. code-block:: text + + pipeline_create { + "name": "export-daily", + "definition": { + "schema_version": 1, + "description": "Move the daily export from S3 to SFTP and verify it", + "tasks": { + "digest": {"action": ["FA_storage_checksum", + {"uri": "s3://reports-export/daily/${params.date}.csv"}]}, + "copy": {"action": ["FA_storage_copy", + {"source": "s3://reports-export/daily/${params.date}.csv", + "target": "sftp://sftp.example.com/inbound/reports/${params.date}.csv"}], + "depends_on": ["digest"], + "retry": {"max_attempts": 3, "backoff": 5}, "timeout": 600}, + "verify": {"action": ["FA_storage_verify", + {"uri": "sftp://sftp.example.com/inbound/reports/${params.date}.csv", + "expected": "${tasks.digest.result}", "strict": true}], + "depends_on": ["copy"]}, + "remove": {"action": ["FA_storage_delete", + {"uri": "s3://reports-export/daily/${params.date}.csv"}], + "depends_on": ["verify"]} } } } + pipeline_run {"name": "export-daily", "params": {"date": "2026-10-07"}, "dry_run": true} + pipeline_run {"name": "export-daily", "params": {"date": "2026-10-07"}} + audit_search {"correlation_id": ""} -加载完成后，要求 Claude Code 使用 ``mcp__automation_file__FA_*`` 工具—— -例如「使用 FA_fast_find 找出 ./var 下所有 *.log」。 +``"${tasks.digest.result}"`` 把来源的校验和交给 ``FA_storage_verify``。加上 +``strict: true`` 之后，不一致会让任务失败，于是 ``remove`` 被跳过，来源保留。失败的 +运行会发布 ``pipeline.failed``，同一条路由会把它发到 Slack。 -查看工具目录 ------------- +``FA_*`` 桥接 +-------------------- -可渲染与宿主完全一致的描述符——便于测试、GUI 调试或生成文档： +桥接开启时，``tools/list`` 先返回语义工具，接着是每个已注册的动作各一个工具，按名称 +排序，其 JSON Schema 由 Python 签名派生而来。对这类工具调用 ``tools/call`` 会通过 +注册表派发，并以 JSON 编码的返回值响应；失败则是 JSON-RPC 错误。之前的版本就是这样 +工作的，已有的配置可以继续使用。 + +.. code-block:: text + + python -m automation_file mcp # 语义工具 + 每个 FA_* 动作 + python -m automation_file mcp --allowed-actions FA_list_dir,FA_file_checksum + python -m automation_file mcp --root /srv/reports --no-bridge # 只有语义工具 + +``--allowed-actions`` 把桥接缩小到指定的动作。参数里提到列表以外动作的调用会被拒绝 +（无法用 ``FA_execute_action`` 绕过去）。``--no-bridge`` 或 +``MCPServer(bridge=False)`` 会关闭桥接；此时 ``FA_*`` 名称是未知的工具。 + +**策略不约束桥接。** 通过桥接调用的 ``FA_storage_copy`` 可以到达进程能到的任何 +位置，不管 ``--root`` 怎么配置。面对 AI 客户端请使用 ``--no-bridge``；桥接只留给 +那些即使没有策略你也愿意让客户端调用的工具。 + +从 Python 使用： .. code-block:: python - from automation_file import tools_from_registry, executor + from automation_file import MCPServer, executor, tools_from_registry + from automation_file.server.mcp_policy import MCPPolicy + from automation_file.server.mcp_tools import SemanticToolkit - for tool in tools_from_registry(executor.registry): + MCPServer().serve_stdio() # 和以前一样：桥接开启 + MCPServer(policy=MCPPolicy(roots=["/srv/reports"]), bridge=False).serve_stdio() + + for tool in tools_from_registry(executor.registry): # 桥接的工具目录 print(tool["name"], "->", tool["description"]) -每个描述符的形状如下:: + toolkit = SemanticToolkit(MCPPolicy(roots=["/srv/reports"])) # 不经 JSON-RPC 使用工具 + outcome = toolkit.call("file_checksum", {"uri": "/srv/reports/a.csv"}) + outcome.is_error, outcome.payload["value"], outcome.correlation_id + +命令行参数 +-------------------- + +``python -m automation_file mcp`` 与 ``automation_file_mcp`` 控制台脚本接受相同的 +命令行参数。 + +.. list-table:: + :header-rows: 1 + :widths: 32 68 + + * - 命令行参数 + - 含义 + * - ``--name``、``--version`` + - 握手时的 ``serverInfo``。默认值：``automation_file``、``1.0.0``。 + * - ``--allowed-actions a,b`` + - 桥接提供的已注册动作。默认：全部。 + * - ``--no-bridge`` + - 只提供语义工具。 + * - ``--root URI`` + - 一个允许的位置；可重复。 + * - ``--allow-write``、``--allow-overwrite``、``--allow-delete`` + - 三项权限。后两项需要第一项。 + * - ``--max-read-bytes N``、``--max-write-bytes N``、``--max-results N``、 + ``--max-search-bytes N`` + - 各项上限。 + * - ``--pipeline-dir URI`` + - 流水线定义存放的地方。 + * - ``--pipeline-actions a,b`` + - 通过 MCP 运行的流水线可以调用的动作。 + * - ``--tools a,b`` + - 要提供的语义工具，或 ``none``。 + +错误的命令行参数（未知的工具名称、没有 ``--allow-write`` 的 ``--allow-overwrite``、 +带有凭据的根位置）会让命令在开始服务之前就以用法错误结束。 + +安全指引 +---------------- + +* **进程的权限就是外部边界。** 服务器以启动它的用户身份运行，使用后端被给予的 + 凭据。策略为语义工具缩小这个范围；它不能替代账号、bucket policy 或 SFTP 用户 + 上的最小权限。 +* **面对 AI 客户端请关闭桥接**\ （``--no-bridge``）。桥接提供每个已注册的动作， + 包括 ``FA_run_shell`` 与 ``FA_storage_delete``，而策略对它不适用。 +* **根位置要尽量小。** 根位置是对其下所有内容的授权。不要使用 ``local:///`` 或主 + 目录。客户端可以写入的东西，请放在它专属的目录里。 +* **从只读开始。** 工作需要时才加上 ``--allow-write``，确实需要时才加上 + ``--allow-overwrite`` 与 ``--allow-delete``。能写入但不能覆盖或删除的客户端， + 无法破坏原本就在那里的东西。 +* **文件的内容不是指令。** 模型通过 ``file_read`` 与 ``file_search`` 读到文件内容， + 并可能按照读到的东西行动。策略限制了这样被劫持的会话能做的事；宿主的确认提示 + 也是。对重要的操作，先要求一次 ``dry_run``。 +* **流水线。** 默认的动作留在根位置之内。你用 ``--pipeline-actions`` 添加的每个动作 + 都不受限制地运行。流水线目录里放的是 AI 客户端写下的定义：只有 ``pipeline_run`` + 会对它们应用策略，所以不要用 ``FA_pipeline_run`` 或 ``Pipeline.from_file`` 运行 + 它们，也不要让调度器指向那个目录。 +* **报告类工具不受根位置约束。** ``audit_search``、``integrity_status`` 与 + ``pipeline_status`` 会显示整个进程里的资源名称、错误与运行参数。对于不该看到这些 + 的客户端，请不要把它们列入 ``--tools``。 +* **客户端的名称不能证明任何事。** 它只是审计轨迹里 actor 的标签，而且由客户端 + 自己提供。stdio 没有认证机制：能启动这个进程的人，就拥有它的能力。 +* **机密。** 凭据绝不该放在存储 URI 里（带有凭据的 URI 会被拒绝），也不该放在流水线 + 参数里，因为参数会随运行一起被记录。日志不含参数值，也不含文件内容。 +* **网络。** 语义工具本身不发出任何 HTTP 请求。存储后端保有各自的检查：TLS 验证、 + SFTP 主机密钥，以及会抓取 URL 的动作所用的 SSRF 防护。 +* 不要在提供桥接的进程里调用 ``PackageLoader.add_package_to_executor``：它会把包的 + 每个成员注册成动作。 + +出问题时 +---------------- - { - "name": "FA_fast_find", - "description": "底层可调用对象的第一行 docstring。", - "inputSchema": { - "type": "object", - "properties": {"root": {"type": "string"}, "pattern": {"type": "string"}}, - "required": ["root", "pattern"], - "additionalProperties": true, - }, - } +``no storage location is allowed on this server`` + 没有配置任何根位置。加上 ``--root <存储 URI 或目录>``，或传入 + ``MCPPolicy(roots=[...])``。 -手工烟雾测试 ------------- +``... is outside the allowed locations (...)`` + 位置不在任何根位置之内；消息里会列出根位置。请对照根位置检查写法：另一个 + bucket 或主机、相邻的目录（``reports`` 旁边的 ``reports-old``）、authority 的 + 大小写不同，或通往同一个目录的另一条路径。 -直接把 JSON-RPC 帧喂给服务器以确认其能正常加载:: +``... leaves the allowed location through a link or an absolute path`` + 本地根位置下面的符号链接或 junction 指向根位置之外。如果客户端应该能到达那里， + 请把链接的目标加为根位置。 - printf '%s\n%s\n%s\n' \ - '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{}}' \ - '{"jsonrpc":"2.0","id":2,"method":"tools/list"}' \ - '{"jsonrpc":"2.0","id":3,"method":"tools/call","params":{"name":"FA_fast_find","arguments":{"root":".","pattern":"*.py","limit":3}}}' \ - | python -m automation_file mcp +``this server is read-only``、``does not allow it`` + 权限没有开启：``--allow-write``、``--allow-overwrite`` 或 ``--allow-delete``。 + ``file_move`` 需要写入与删除。 -每行输入在 stdout 上恰得到一条 JSON-RPC 响应（通知没有响应）。 +``already exists`` + 传入 ``overwrite=true``；服务器也必须允许覆盖。 -安全注意事项 ------------- +``file_read`` 只返回文件的一部分 + ``truncated`` 为 true：以 ``offset=next_offset`` 再调用一次，或调高 + ``--max-read-bytes``。对于不支持范围读取的后端，每次调用都会把整个文件暂存到 + 本地，所以翻阅很大的远端文件时请节制。 + +``file_search`` 漏掉某个文件 + 检查 ``complete`` 与 ``skipped``。超过 ``--max-search-bytes`` 剩余额度的文件 + 不会被搜索；请缩小 ``pattern`` 或调高额度。内容是按 UTF-8 文本匹配的。 + +``... is not allowed in a pipeline`` + 定义提到了允许范围以外的动作；消息里会列出允许的动作。请使用存储动作，或用 + ``--pipeline-actions`` 添加该动作。 + +流水线任务以 ``MCPLocationException`` 或 ``MCPPermissionException`` 失败 + 任务在运行时到达了根位置以外的位置，或需要一项没有开启的权限。定义之所以被 + 接受，是因为位置来自参数。 + +``no pipeline named ... is stored`` + 消息里会列出已存储的名称。没有 ``--pipeline-dir`` 时，服务器重新启动后定义就 + 不在了。 + +``pipeline_status`` 找不到某次运行 + 运行记录默认存在内存里。请在启动脚本里调用 + ``set_default_run_store(SQLiteRunStore(path))``。 + +``this server keeps no audit trail`` + 请在启动脚本里于 ``serve_stdio()`` 之前调用 ``configure_audit(path)``。 + +``sftp://`` 或 ``s3://`` 位置明明在根位置之内却失败 + 这个进程里的后端没有初始化。请在启动脚本里调用它的 ``later_init``；见 + `生产环境配置`_ 与 :doc:`storage`。 + +宿主列出数百个工具，或完全没有语义工具 + 前者是桥接：加上 ``--no-bridge`` 或 ``--allowed-actions``。后者是 + ``--tools none``，或包的版本早于语义工具。 + +宿主在启动时报告连接中断 + 有东西写到了 ``stdout``，或命令以用法错误结束。请在终端里运行同一条命令： + 错误会写到 ``stderr``。手动检查服务器的方法:: + + printf '%s\n%s\n' \ + '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{}}' \ + '{"jsonrpc":"2.0","id":2,"method":"tools/list"}' \ + | python -m automation_file mcp --root /srv/reports --no-bridge -* MCP 服务器以 **启动它的 Python 进程的权限** 运行。 - 每次工具调用都落在你 shell 用户账户里。 -* 默认情况下，服务器暴露 **全部** 已注册的 ``FA_*`` 动作， - 其中包含会删除文件、写文件系统、上传到远程后端的动作。 - 对任何不完全可信的宿主，请用 ``--allowed-actions`` 只白名单暴露 - 必要的动作。 -* 在打算给第三方宿主使用的 MCP 服务器启动前，**不要** 调用 - :func:`~automation_file.PackageLoader.add_package_to_executor` - （或暴露其动作）。该助手会注册任意包的全部顶层函数 / 类 / 内置， - 权限相当于 ``eval``。 -* 服务器启动时会通过 ``file_automation_logger`` 以 ``INFO`` 级别记录 - 暴露的工具数量与配置的服务器名称。工具调用负载从不被记录。 -* 涉及对外 HTTP 的动作（``FA_download_file``、各云后端）仍会通过 - SSRF 校验；MCP 层不会放宽任何单动作检查。 +语义工具的每个异常都派生自 ``MCPServerException``，因此也派生自 +``FileAutomationException``：``MCPPermissionException``\ （带有 ``code``）、它的子类 +``MCPLocationException``，以及 ``MCPToolException``\ （带有 ``kind``）。 diff --git a/docs/source/Zh-TW/usage/cli.rst b/docs/source/Zh-TW/usage/cli.rst index 1c2d43c..d4cfad7 100644 --- a/docs/source/Zh-TW/usage/cli.rst +++ b/docs/source/Zh-TW/usage/cli.rst @@ -17,12 +17,15 @@ CLI python -m automation_file create-file hello.txt --content "hi" python -m automation_file server --host 127.0.0.1 --port 9943 python -m automation_file http-server --host 127.0.0.1 --port 9944 + python -m automation_file mcp --root /srv/reports --no-bridge python -m automation_file mcp --allowed-actions FA_list_dir,FA_file_checksum python -m automation_file drive-upload my.txt --token token.json --credentials creds.json -``mcp`` 子指令以 stdio 啟動 Model Context Protocol 伺服器， -讓 Claude Desktop 之類的宿主可把 ``FA_*`` 動作當成 MCP 工具呼叫—— -完整整合說明請見 :doc:`mcp`。 +``mcp`` 子指令以 stdio 啟動 Model Context Protocol 伺服器，讓 Claude Desktop 之類的 +宿主可以處理檔案：一是透過語意工具（``file_read``、``storage_copy``、 +``pipeline_run`` ……），它們只在 ``--root`` 指定的位置之內作業，而且在 +``--allow-write`` 之前都是唯讀的；二是透過把 ``FA_*`` 動作當成 MCP 工具提供的橋接。 +旗標、權限模型與完整整合說明請見 :doc:`mcp`。 儲存 ---- diff --git a/docs/source/Zh-TW/usage/mcp.rst b/docs/source/Zh-TW/usage/mcp.rst index e87811b..9dbce2c 100644 --- a/docs/source/Zh-TW/usage/mcp.rst +++ b/docs/source/Zh-TW/usage/mcp.rst @@ -1,69 +1,26 @@ -MCP 伺服器（Claude Desktop / Claude Code） -========================================== - -``automation_file`` 內建一個 Model Context Protocol（MCP）伺服器， -把共用的 :class:`~automation_file.core.action_registry.ActionRegistry` -中每一項暴露為 MCP 工具。**Claude Desktop**、**Claude Code** -以及其他 MCP 宿主，可以像呼叫一般 MCP 工具一樣呼叫 ``FA_*`` 動作—— -無需自寫外掛、無需額外打包。 - -傳輸方式為 **stdio**（``stdin`` / ``stdout`` 上每行一條 JSON-RPC 2.0 訊息）， -與目前 MCP 宿主使用的協定一致。 - -提供的能力 ----------- - -* ``initialize`` 握手，回傳協定版本 ``2024-11-05``、 - ``serverInfo.name`` 與 ``serverInfo.version``。 -* ``tools/list`` 為每個已註冊的 ``FA_*`` 動作回傳一個 MCP 工具描述， - 其參數 JSON Schema 由 Python 函式簽名自動推導 - （``str → "string"``、``int → "integer"`` 等）。 -* ``tools/call`` 透過註冊表派送，並以 JSON 編碼的文字內容區塊回傳結果。 -* ``--allowed-actions`` 允許清單參數，讓你只把註冊表的子集暴露給宿主。 -* 所有內部錯誤都會以 JSON-RPC 錯誤物件形式回傳—— - 宿主不必解析例外字串即可呈現錯誤。 - -啟動伺服器 ----------- - -CLI:: - - python -m automation_file mcp - python -m automation_file mcp --name automation_file --version 1.0.0 - python -m automation_file mcp --allowed-actions FA_list_dir,FA_file_checksum - -行程在前景執行，從 ``stdin`` 讀取分行 JSON 並把回應寫到 ``stdout``。 -通常宿主會替你啟動該行程，不需要手動執行。 - -從 Python 啟動（例如嵌入到自家 stdio 橋接器中）： - -.. code-block:: python +MCP 伺服器 +==================== - from automation_file import MCPServer - - server = MCPServer(name="automation_file", version="1.0.0") - server.serve_stdio() # stdin 關閉前阻塞 - -以自訂註冊表把可呼叫動作面縮小： - -.. code-block:: python +``automation_file`` 內建一個 Model Context Protocol（MCP）伺服器，讓 **Claude Desktop** +或 **Claude Code** 這類 AI 用戶端可以透過它處理檔案。傳輸方式是 stdio：每行一則 +JSON-RPC 2.0 訊息。 - from automation_file import MCPServer - from automation_file.core.action_registry import ActionRegistry - from automation_file import executor +伺服器提供兩組工具： - safe = ActionRegistry() - for name in ("FA_list_dir", "FA_file_checksum", "FA_fast_find"): - safe.register(name, executor.registry.resolve(name)) +* **語意工具**：十四個名稱穩定、以工作為單位的工具（``file_read``、``file_copy``、 + ``pipeline_run`` ……），操作對象是 :doc:`儲存 URI `，並受一份權限政策 + 約束。這是為 AI 用戶端設計的介面； +* **橋接**：每個已註冊的 ``FA_*`` 動作各自成為一個工具，與先前的版本相同。為了相容， + 它預設開啟，而且 **不受** 政策約束。 - MCPServer(safe).serve_stdio() +語意工具的預設值是安全的：不允許任何位置、不能更動任何東西，每個回應的大小也都有 +上限。 -Claude Desktop 設定 -------------------- +最小設定 +---------------- -在 ``~/Library/Application Support/Claude/claude_desktop_config.json`` -（macOS）或 ``%APPDATA%\Claude\claude_desktop_config.json`` -（Windows）的 ``mcpServers`` 下新增條目： +給伺服器一個可以讀取的目錄。寫在 ``claude_desktop_config.json``\ （Claude Desktop）或 +``.mcp.json``\ （Claude Code）裡： .. code-block:: json @@ -71,119 +28,755 @@ Claude Desktop 設定 "mcpServers": { "automation_file": { "command": "python", - "args": ["-m", "automation_file", "mcp"] + "args": ["-m", "automation_file", "mcp", "--root", "/srv/reports", "--no-bridge"] } } } -重新啟動 Claude Desktop。``automation_file`` 伺服器會出現在工具面板， -所有 ``FA_*`` 動作皆可呼叫。 +用戶端現在會看到十四個語意工具。它可以列出、讀取、搜尋 ``/srv/reports`` 底下的 +內容並計算校驗碼，除此之外什麼都不能做：寫入會被拒絕，該目錄以外的每個路徑也一樣。 +在 Windows 上路徑要寫成 ``"C:\\data\\reports"``。如果 ``PATH`` 上的 ``python`` 不是 +安裝本套件的那一個，請改用該環境的直譯器 +（``"command": "C:\\envs\\fa\\Scripts\\python.exe"``）。 + +使用 Claude Code 時，同一個伺服器可以從 shell 加入:: + + claude mcp add automation_file -- python -m automation_file mcp --root /srv/reports --no-bridge + +正式環境設定 +------------------------ -對於會操作敏感路徑的宿主，建議把可用動作鎖死為白名單： +列出每一個位置、只開啟工作需要的權限、把管線的定義存在磁碟上，並且關閉橋接： .. code-block:: json { "mcpServers": { "automation_file": { - "command": "python", + "command": "/opt/fa/bin/python", "args": [ "-m", "automation_file", "mcp", - "--allowed-actions", - "FA_list_dir,FA_fast_find,FA_file_checksum,FA_verify_checksum" + "--root", "/srv/reports/inbox", + "--root", "/srv/reports/outbox", + "--allow-write", + "--max-read-bytes", "65536", + "--max-results", "100", + "--pipeline-dir", "/var/lib/automation_file/pipelines", + "--tools", "file_read,file_write,file_copy,file_search,file_checksum,file_verify,storage_list,storage_copy", + "--no-bridge" ] } } } -顯式指定虛擬環境的直譯器（避免誤用系統 Python）： +遠端後端必須先初始化用戶端，稽核軌跡與通知路由也要在伺服器啟動之前設定好。這需要 +幾行 Python，所以請用一支啟動腳本來啟動伺服器，再把宿主的 ``command`` 指向它： + +.. code-block:: python + + # /opt/fa/mcp_server.py + import os + + from automation_file import ( + MCPServer, Route, Severity, SlackSink, configure_audit, + notification_manager, notification_router, s3_instance, sftp_instance, + ) + from automation_file.server.mcp_policy import MCPPolicy + + s3_instance.later_init(region_name="eu-west-1") # 憑證：AWS 的預設來源鏈 + sftp_instance.later_init(host="sftp.example.com", username="reports", + key_filename="/etc/fa/id_ed25519", + known_hosts="/etc/fa/known_hosts") + configure_audit("/var/lib/automation_file/audit.sqlite") # audit_search 讀的就是這裡 + notification_manager.register(SlackSink(os.environ["SLACK_WEBHOOK"], name="ops")) + notification_router.add_route(Route( + "mcp-failures", sinks=("ops",), + types=("mcp.tool.failed", "pipeline.failed", "storage.error"), + min_severity=Severity.WARNING, + )) + notification_router.start() + + policy = MCPPolicy( + roots=["s3://reports-export/daily", "sftp://sftp.example.com/inbound/reports"], + allow_write=True, + allow_delete=True, # file_move 會刪除來源 + max_read_bytes=64 * 1024, + pipeline_dir="/var/lib/automation_file/pipelines", + ) + MCPServer(policy=policy, bridge=False).serve_stdio() + +.. code-block:: json + + {"mcpServers": {"automation_file": {"command": "/opt/fa/bin/python", + "args": ["/opt/fa/mcp_server.py"]}}} + +``stdout`` 上除了協定之外什麼都不能寫：函式庫的日誌寫到 ``stderr`` 與它的日誌檔， +啟動腳本也不可以 ``print``。 + +語意工具 +---------------- + +每個位置都是一個儲存 URI，``:///`` +（``local:///srv/reports/a.csv``、``s3://bucket/2026/a.csv``、 +``sftp://host/inbox/a.csv``），或是本機的絕對路徑。「需要」一欄列出：除了要有一個 +涵蓋這次呼叫所有位置的根位置之外，政策還必須允許什麼。 + +.. list-table:: + :header-rows: 1 + :widths: 14 30 36 20 + + * - 工具 + - 引數 + - 結果 + - 需要 + * - ``file_read`` + - ``uri``、``offset=0``、``max_bytes``、``encoding="utf-8"``\ （文字編碼，或 + ``base64``） + - ``content``、``encoding``、``size``、``offset``、``bytes``、 + ``truncated``、``next_offset`` + - — + * - ``file_write`` + - ``uri``、``content``、``encoding="utf-8"``、``overwrite=false``、 + ``dry_run=false`` + - ``size``、``sha256``、``overwrites``、``replaced_size``、``written`` + - 寫入；要取代檔案時還需要覆寫 + * - ``file_copy`` + - ``source``、``target``、``overwrite=false``、``verify=false``、 + ``dry_run=false`` + - ``source``、``target``、``size``、``overwrites``、``replaced_size``、 + ``deletes_source``、``done``；使用 ``verify`` 時另有 ``sha256``、 + ``verified`` + - 寫入；要取代檔案時還需要覆寫 + * - ``file_move`` + - 與 ``file_copy`` 相同 + - 與 ``file_copy`` 相同；``deletes_source`` 為 true + - 寫入與刪除；要取代檔案時還需要覆寫 + * - ``file_search`` + - ``uri``、``pattern="*"``、``content``、``recursive=true``、 + ``case_sensitive=false``、``max_results`` + - ``matches``\ （``uri``、``path``、``name``、``size``、``modified_at``； + 內容搜尋另有 ``line``、``snippet``、``matching_lines``）、``count``、 + ``candidates``、``truncated``；內容搜尋另有 ``searched_files``、 + ``searched_bytes``、``skipped``、``complete`` + - — + * - ``file_checksum`` + - ``uri``、``algorithm="sha256"`` + - ``algorithm``、``value``、``size`` + - — + * - ``file_verify`` + - ``uri``、``expected``\ （十六進位，或 ``sha256:``）、 + ``algorithm="sha256"`` + - ``match``、``expected``、``actual``、``algorithm`` + - — + * - ``storage_list`` + - ``uri``、``recursive=false``、``max_results`` + - ``entries``\ （``uri``、``path``、``name``、``is_dir``、``size``、 + ``modified_at``）、``count``、``total``、``truncated`` + - — + * - ``storage_copy`` + - ``source``、``target``、``overwrite=false``、``verify=false``、 + ``dry_run=false`` + - 檔案：``kind="file"`` 加上 ``file_copy`` 的結果。目錄：``kind="tree"``、 + ``planned``\ （``copy``、``overwrite``、``skip``、``bytes``）、``paths``、 + ``existing``、``truncated``、``done``，複製之後另有 ``copied``、 + ``skipped``、``failed``、``errors``、``ok`` + - 寫入；要取代檔案時還需要覆寫 + * - ``pipeline_create`` + - ``name``、``definition``、``overwrite=false``、``dry_run=false`` + - ``name``、``location``、``persistent``、``tasks``、``actions``、 + ``overwrites``、``stored`` + - 寫入；要取代定義時還需要覆寫 + * - ``pipeline_run`` + - ``name``、``params``、``dry_run=false``、``background=false`` + - ``run_id``、``status``、``ok``、``run``\ （這次執行與它的每個任務） + - 寫入；每個任務需要它的動作所需要的權限 + * - ``pipeline_status`` + - ``run_id``，或 ``name`` 與 ``limit=5`` + - ``runs``、``count`` + - 不需要根位置 + * - ``integrity_status`` + - ``name`` + - ``monitors``\ （``name``、``target``、``running``、``last_run``、 + ``last_error``、``last_report``）、``count`` + - 不需要根位置 + * - ``audit_search`` + - ``actor``、``source``、``pipeline``、``task``、``action``、``backend``、 + ``status``、``correlation_id``、``resource_prefix``、``text``、``since``、 + ``until``、``limit=50``、``offset=0`` + - ``records``、``count``、``total``、``limit``、``offset``、``truncated`` + - 不需要根位置；需要已設定的稽核軌跡 + +個別工具的說明： + +``file_read`` + 每次呼叫最多回傳 ``--max-read-bytes`` 個位元組。``truncated`` 為 true 時，把 + ``offset`` 設成 ``next_offset`` 再呼叫一次；字元絕不會被切成兩半。內容不是所選 + 編碼的文字時會回報錯誤，並要求改用 ``encoding="base64"``。 + +``file_copy``、``file_move`` 與 ``verify`` + ``verify=true`` 時會比對來源與副本的 SHA-256。此時搬移會先複製、再比對，只有在 + 兩個摘要相符時才刪除來源；不相符時來源保留，呼叫以 ``checksum_mismatch`` 失敗。 + +``file_verify`` + 不相符是一個結果（``match`` 為 false），不是錯誤。 + +``file_search`` + ``pattern`` 是比對檔名的 shell 樣式（``*.csv``）；樣式裡有 ``/`` 時，比對的是 + ``uri`` 底下的相對路徑（``2026/*/*.csv``）。``content`` 是單純的子字串，不是正規 + 表示式。一次內容搜尋最多讀取 ``--max-search-bytes`` 個位元組。超過剩餘額度的 + 檔案不會被開啟，而是列在 ``skipped`` 裡；二進位檔與讀不到的檔案也一樣。只有在 + 每個候選檔案都搜尋過時，``complete`` 才是 true。 + +``storage_copy`` + 像 ``file_copy`` 一樣複製一個檔案，或複製某個目錄底下的所有檔案。對目錄而言， + 除非 ``overwrite`` 為 true，否則目標已有的檔案會被略過。失敗的檔案記在 + ``errors`` 裡，其他檔案照樣複製；這時呼叫是一個錯誤，但結果裡仍有各項數量。 + +``pipeline_run`` + 在這次請求裡執行，並回傳已結束的執行。``background=true`` 時立刻回傳 + ``status="running"``；請用 ``run_id`` 輪詢 ``pipeline_status``。失敗的執行是 + 一個錯誤，但結果裡仍帶有這次執行。 + +結果與錯誤 +~~~~~~~~~~~~~~~~~~~~ + +語意工具以一份 JSON 文件作為結果的文字回應。其中一定有 ``tool`` 與 +``correlation_id``。呼叫被拒絕或失敗時，``isError`` 為 true，文件裡會有 ``error``： .. code-block:: json { - "mcpServers": { - "automation_file": { - "command": "C:\\envs\\fa\\Scripts\\python.exe", - "args": ["-m", "automation_file", "mcp"] - } + "tool": "file_write", + "correlation_id": "27acff55529c4317b74de6ea98759f42", + "error": { + "type": "permission_denied", + "code": "read_only", + "message": "file_write changes something and this server is read-only: start it with --allow-write, or pass MCPPolicy(allow_write=True)" } } -Claude Code 設定 +.. list-table:: + :header-rows: 1 + :widths: 26 74 + + * - ``error.type`` + - 意義 + * - ``permission_denied`` + - 政策拒絕了這次呼叫。``code`` 指出是哪一條規則：``no_root``、 + ``outside_root``、``read_only``、``overwrite_not_allowed``、 + ``delete_not_allowed``、``tool_disabled``、``action_not_allowed`` 或 + ``limit_exceeded``。 + * - ``invalid_arguments`` + - 引數缺少、未知、型別錯誤或超出範圍。每個問題都會列出。 + * - ``invalid_uri`` + - 位置不是儲存 URI，或它的路徑含有 ``..`` 區段。 + * - ``not_found``、``already_exists`` + - 沒有這個檔案、管線、執行或監控器；或目標已存在而沒有給 ``overwrite``。 + * - ``checksum_mismatch`` + - ``verify`` 發現兩邊的摘要不同。 + * - ``invalid_definition`` + - 管線定義有誤；``problems`` 列出每一項發現與它的路徑。 + * - ``not_configured`` + - 伺服器沒有保存稽核軌跡。 + * - ``failed`` + - 函式庫回報的其他錯誤；``exception`` 是例外類別的名稱。 + * - ``internal_error`` + - 非預期的例外。請回報。 + +權限模型 ---------------- -Claude Code 使用相同的 MCP 定義。可透過 ``claude mcp add`` CLI 註冊:: - - claude mcp add automation_file -- python -m automation_file mcp - -或在儲存庫根目錄提交 ``.mcp.json``： +:class:`~automation_file.server.mcp_policy.MCPPolicy` 在伺服器啟動時建立，之後無法 +更改。它的文字描述會放在握手回應的 ``instructions`` 裡送給用戶端，所以模型在第一次 +呼叫之前就知道界線在哪裡。 + +.. list-table:: + :header-rows: 1 + :widths: 24 24 14 38 + + * - 欄位 + - 旗標 + - 預設值 + - 意義 + * - ``roots`` + - ``--root``\ （可重複） + - 無 + - 工具可以作業的位置：儲存 URI 或本機目錄。一個都沒有時，每個會碰到儲存的 + 工具都會拒絕，並說明如何加入根位置。 + * - ``allow_write`` + - ``--allow-write`` + - 關閉 + - 沒有它，``file_write``、``file_copy``、``file_move``、``storage_copy``、 + ``pipeline_create`` 與 ``pipeline_run`` 都會被拒絕，試跑也一樣。 + * - ``allow_overwrite`` + - ``--allow-overwrite`` + - 關閉 + - 取代既有的檔案或定義。呼叫本身也必須提出要求（``overwrite=true``）。需要 + ``allow_write``。 + * - ``allow_delete`` + - ``--allow-delete`` + - 關閉 + - 刪除。``file_move`` 會刪除來源，所以需要它。需要 ``allow_write``。 + * - ``max_read_bytes`` + - ``--max-read-bytes`` + - 262144 + - ``file_read`` 一次呼叫最多回傳的位元組數，也是管線工具回傳的任務結果的 + 大小上限。 + * - ``max_write_bytes`` + - ``--max-write-bytes`` + - 1048576 + - ``file_write`` 接受的內容大小上限，以及 ``pipeline_create`` 儲存的定義大小 + 上限。 + * - ``max_results`` + - ``--max-results`` + - 200 + - 列表、搜尋、目錄複製計畫或 ``audit_search`` 最多回傳的項目數。呼叫裡較大的 + ``max_results`` 或 ``limit`` 會被降到這個值。 + * - ``max_search_bytes`` + - ``--max-search-bytes`` + - 8388608 + - 一次內容搜尋最多讀取的位元組數。 + * - ``pipeline_dir`` + - ``--pipeline-dir`` + - 記憶體 + - ``pipeline_create`` 存放定義的地方：儲存 URI 或本機目錄，每條管線一個 + ``.json``。沒有設定時定義保存在記憶體裡，伺服器停止就消失。 + * - ``pipeline_actions`` + - ``--pipeline-actions`` + - 依權限而定 + - 透過 MCP 建立或執行的管線可以呼叫的動作。見 `管線`_。 + * - ``tools`` + - ``--tools`` + - 全部十四個 + - 要提供的語意工具。沒有列入的工具不會出現在清單裡，呼叫它也會被拒絕。 + ``--tools none`` 一個都不提供。 + * - ``actor`` + - （僅限 Python） + - ``mcp`` + - 事件與稽核紀錄裡每次呼叫的 actor。握手時用戶端回報的名稱會接在後面： + ``mcp:claude-desktop``。 + +位置 +~~~~~~~~ + +呼叫裡提到的每個位置都在某個根位置之內（或就是根位置本身）時，呼叫才被允許。 + +* **本機根位置** 由儲存層自己把關。該位置由一個限制在根位置內的 ``LocalStorage`` + 提供服務，所以每個操作都會通過 ``safe_join``：指向根位置之外的符號連結（或 + Windows 的 junction），以及偷渡到根位置底下的絕對路徑，都會以 ``outside_root`` + 被拒絕。留在根位置之內的連結則會被跟隨。在 Windows 上比對不分大小寫，也接受 + 反斜線；名稱是裝置（``CON``、``NUL``、``COM1``）或替代資料流 + （``a.txt:stream``）的路徑同樣會被拒絕。 +* **其他後端** 以 scheme、authority 與完整的路徑區段比對。``s3://bucket/team`` 允許 + ``s3://bucket/team/2026/a.csv``，並拒絕 ``s3://bucket/team-b/a.csv`` 與 + ``s3://other/team/a.csv``。authority 必須完全相同，包含大小寫。根位置只是路徑的 + 前綴，僅此而已：伺服器上既有的連結（例如 SFTP 的符號連結）會由伺服器跟隨。請給 + 後端一個無法離開該目錄樹的帳號。 +* 含有 ``..`` 區段的路徑根本不是儲存 URI（``invalid_uri``）。 +* 以 ``Storage.mount`` 掛載的後端要透過它的掛載 URI 來指定；位於已掛載的 + ``LocalStorage`` 之下的根位置，會被限制在該根位置之內。 + +位置的寫法要和根位置的寫法一致。以 ``/srv/reports`` 給定的根位置不會符合 +``local:///mnt/disk2/reports``，即使兩者是同一個目錄。 + +管線 +~~~~~~~~ + +``pipeline_create`` 儲存一份定義（:doc:`pipeline`，``schema_version: 1``）， +``pipeline_run`` 則執行它。兩者都需要 ``allow_write``。 + +定義可以呼叫什麼，在建立時檢查一次，執行時再檢查一次，因為這段期間檔案可能被別的 +東西改寫。預設情況下，管線可以呼叫權限所涵蓋的 ``FA_storage_*`` 動作： + +.. list-table:: + :header-rows: 1 + :widths: 30 70 + + * - 權限 + - 動作 + * - 一律可用 + - ``FA_storage_exists``、``FA_storage_stat``、``FA_storage_list``、 + ``FA_storage_checksum``、``FA_storage_verify``、``FA_storage_read_text``、 + ``FA_storage_schemes`` + * - ``allow_write`` + - ``FA_storage_mkdir``、``FA_storage_write_text``、``FA_storage_copy``、 + ``FA_storage_copy_tree`` + * - ``allow_overwrite`` + - ``FA_storage_sync``\ （它會取代有變動的檔案） + * - ``allow_delete`` + - ``FA_storage_move``、``FA_storage_delete`` + +在這樣的管線裡，這些名稱執行的不是原本的動作，而是受防護的版本：它們在任務執行的 +當下檢查根位置與權限，也就是在 ``${params.}`` 與 ``${tasks..result}`` +填入之後，所以參數無法把任務帶到根位置之外。它們的行為與原本的動作相同，差別如下： + +* ``overwrite`` 的預設值是政策所允許的，而不是 ``true``；在不允許覆寫的地方，明確 + 寫出的 ``overwrite: true`` 會被拒絕； +* ``FA_storage_verify`` 的 ``expected`` 也接受 ``FA_storage_checksum`` 的結果，所以 + ``"${tasks..result}"`` 可以用來比對兩個檔案； +* ``FA_storage_read_text`` 與 ``FA_storage_write_text`` 遵守讀取與寫入的上限； +* ``FA_storage_sync`` 需要 ``allow_overwrite``，``delete: true`` 還需要 + ``allow_delete``； +* ``FA_storage_delete`` 絕不會移除根位置本身； +* 沒有 ``FA_storage_upload`` 與 ``FA_storage_download``：它們接受的是檔案系統路徑， + 不是儲存 URI。請改為複製到 ``local://`` URI，或從它複製出來。 + +``--pipeline-actions a,b,c`` 會取代預設的清單。上面的 ``FA_storage_*`` 動作被列入時 +仍然受到防護。**其他動作則照原樣執行，不受根位置與權限約束**：只列出你本來就願意 +讓用戶端直接呼叫的動作，而且絕不要列出會執行其他動作的動作 +（``FA_execute_action``、``FA_pipeline_run``、``FA_run_shell``）。每個名稱都必須是 +伺服器有提供的動作，否則伺服器不會啟動。出現在另一個動作的引數或參數裡的動作會被 +拒絕，除非清單裡有它；儲存動作在那裡則一律被拒絕，因為巢狀執行時它不受防護。 + +透過 MCP 建立的定義不能帶有 ``schedule``：管線何時自行執行，由維運伺服器的人決定。 +定義的 ``name`` 就是它儲存時使用的名稱；省略時會自動填入。 + +執行紀錄存在預設的執行紀錄儲存裡，``pipeline_status`` 與 ``FA_pipeline_status`` 都 +從那裡查找。除非啟動腳本呼叫 ``set_default_run_store(SQLiteRunStore(path))``，否則 +它只存在記憶體裡。 + +試跑 +-------- + +每個會更動東西的工具都接受 ``dry_run``。這時它會做完真正呼叫要做的每一項檢查， +不更動任何東西，並回傳計畫： .. code-block:: json { - "mcpServers": { - "automation_file": { - "command": "python", - "args": ["-m", "automation_file", "mcp"] + "tool": "file_move", + "correlation_id": "bb62a807d4f64bf3b666d00d3b4f410f", + "source": "s3://reports-export/daily/2026-10-07.csv", + "target": "sftp://sftp.example.com/inbound/reports/2026-10-07.csv", + "size": 48211, + "overwrites": false, + "replaced_size": null, + "deletes_source": true, + "dry_run": true, + "done": false + } + +.. list-table:: + :header-rows: 1 + :widths: 24 76 + + * - 工具 + - 試跑回報的內容 + * - ``file_write`` + - 內容的大小與 SHA-256、是否會取代某個檔案，以及該檔案的大小。 + * - ``file_copy``、``file_move`` + - 來源、目標、大小、目標是否會被取代、來源是否會被刪除。 + * - ``storage_copy`` + - 對目錄：會複製、取代與略過多少檔案與位元組，以及路徑（受 ``max_results`` + 限制）。 + * - ``pipeline_create`` + - 定義是否有效且被允許、它的任務與動作，以及是否會取代已儲存的定義。 + * - ``pipeline_run`` + - 依相依順序排列、狀態為 ``planned`` 的任務；若任務指定了未知的動作，或用到 + 這次執行沒有給的參數，會在該任務上附 ``error``。不會執行也不會記錄任何 + 東西。 + +試跑需要的權限與真正的呼叫相同：唯讀的伺服器同樣會拒絕它，所以伺服器不會做的事， +也不會給出計畫。 + +可追溯性 +---------------- + +每次語意呼叫都在 ``correlation_scope()`` 與 ``actor_scope(...)`` 之內執行。結果帶有 +``correlation_id``，這次呼叫所做的一切也都帶有它：儲存操作，以及每次呼叫一則的 +事件：``mcp.tool.completed`` 或 ``mcp.tool.failed``\ （source 為 ``mcp``）。 + +.. list-table:: + :header-rows: 1 + :widths: 26 14 60 + + * - 結果 + - 嚴重度 + - 事件 + * - 完成 + - info + - ``mcp.tool.completed``，``status="ok"`` + * - 被政策拒絕 + - warning + - ``mcp.tool.failed``，``status="refused"``，``code`` 指出規則 + * - 請求本身有誤：檔案不存在、目標已存在、引數錯誤、定義無效 + - info + - ``mcp.tool.failed``，``status="error"``，``code`` 就是 ``error.type`` + * - 失敗：後端錯誤、目錄複製中有檔案失敗、管線執行失敗 + - warning + - ``mcp.tool.failed``，``status="error"``。失敗的後端另外以 + ``storage.error`` 回報，失敗的執行則以 ``pipeline.failed`` 回報。 + * - ``verify`` 發現摘要不同 + - error + - ``mcp.tool.failed``，``status="error"``，``code="checksum_mismatch"`` + * - 非預期的例外 + - error + - ``mcp.tool.failed``，``status="error"``，``code="internal_error"`` + +因此，一條針對 ``mcp.tool.failed``、``min_severity=Severity.WARNING`` 的通知路由會 +收到拒絕與真正的失敗，而不會收到「模型要了一個不存在的檔案」這種事。 + +事件的 payload 記有工具名稱（``action``）、這次呼叫涉及的儲存 URI（``resource``、 +``source_uri``）、``duration_ms``，管線工具另有 ``pipeline`` 與 ``run_id``。它絕不 +包含內容、參數或摘要。被拒絕的呼叫也會以 warning 寫進日誌，內容是工具、規則與關聯 +ID，不含任何引數值。 + +設定了 :doc:`稽核軌跡 ` 之後，一次搜尋就能看到某次呼叫做了什麼： + +.. code-block:: python + + audit_search(correlation_id="7dc51e94bd3b494eae8e6b6f3f3b150b") + # [{"source": "mcp", "action": "mcp.tool.completed", "actor": "mcp:claude-desktop", ...}, + # {"source": "storage", "action": "copy", "resource": "sftp://...", "status": "ok", ...}] + +管線的一次執行有它自己的關聯 ID，也就是它的 ``run_id``。``pipeline_run`` 那次呼叫的 +事件裡記有這個 ``run_id``，兩者由此連結起來。 + +範例：從 S3 到 SFTP，經過驗證與稽核，失敗時發出警示 +------------------------------------------------------------------------ + +工作內容：*把昨天的 CSV 從 S3 搬到公司的 SFTP 伺服器，驗證它的 SHA-256，稽核這次 +傳輸，失敗時通知 Slack*。`正式環境設定`_ 的啟動腳本已經準備好所需的一切：兩個後端都 +已初始化、兩個根位置、允許寫入與刪除、一份稽核軌跡，以及一條把失敗送到 Slack 的 +路由。接著用戶端進行這些呼叫： + +.. code-block:: text + + 1. storage_list {"uri": "s3://reports-export/daily"} + -> 項目清單；用戶端選出 2026-10-07.csv + + 2. file_move {"source": "s3://reports-export/daily/2026-10-07.csv", + "target": "sftp://sftp.example.com/inbound/reports/2026-10-07.csv", + "verify": true, "dry_run": true} + -> 計畫：size、overwrites=false、deletes_source=true + + 3. file_move 同樣的引數，去掉 dry_run + -> done=true、verified=true、sha256="9f86d0..."、correlation_id="7dc5..." + 兩個 SHA-256 摘要相符之後，來源才被刪除。 + + 4. file_verify {"uri": "sftp://sftp.example.com/inbound/reports/2026-10-07.csv", + "expected": "sha256:9f86d0..."} + -> match=true（獨立的檢查，留作紀錄） + + 5. audit_search {"correlation_id": "7dc5..."} + -> 第 3 步的 mcp.tool.completed 紀錄，以及儲存紀錄（copy、delete）， + 附有資源、耗時與狀態 + +**失敗時。** 第 3 步失敗時，用戶端會收到 ``error``，來源仍然在 S3 裡。伺服器會發布 +``mcp.tool.failed``\ （摘要不同時是 error，傳輸失敗時是 warning），後端失敗時另外 +發布 ``storage.error``，啟動腳本設定的路由再把它們送到 Slack。通知不是由任何工具 +送出的，所以用戶端無法略過它。 + +同一件工作也可以寫成管線，建立一次、每天執行： + +.. code-block:: text + + pipeline_create { + "name": "export-daily", + "definition": { + "schema_version": 1, + "description": "Move the daily export from S3 to SFTP and verify it", + "tasks": { + "digest": {"action": ["FA_storage_checksum", + {"uri": "s3://reports-export/daily/${params.date}.csv"}]}, + "copy": {"action": ["FA_storage_copy", + {"source": "s3://reports-export/daily/${params.date}.csv", + "target": "sftp://sftp.example.com/inbound/reports/${params.date}.csv"}], + "depends_on": ["digest"], + "retry": {"max_attempts": 3, "backoff": 5}, "timeout": 600}, + "verify": {"action": ["FA_storage_verify", + {"uri": "sftp://sftp.example.com/inbound/reports/${params.date}.csv", + "expected": "${tasks.digest.result}", "strict": true}], + "depends_on": ["copy"]}, + "remove": {"action": ["FA_storage_delete", + {"uri": "s3://reports-export/daily/${params.date}.csv"}], + "depends_on": ["verify"]} } } } + pipeline_run {"name": "export-daily", "params": {"date": "2026-10-07"}, "dry_run": true} + pipeline_run {"name": "export-daily", "params": {"date": "2026-10-07"}} + audit_search {"correlation_id": ""} + +``"${tasks.digest.result}"`` 把來源的校驗碼交給 ``FA_storage_verify``。加上 +``strict: true`` 之後，不相符會讓任務失敗，於是 ``remove`` 被略過，來源保留。失敗的 +執行會發布 ``pipeline.failed``，同一條路由會把它送到 Slack。 -載入完成後，請 Claude Code 使用 ``mcp__automation_file__FA_*`` 工具—— -例如「使用 FA_fast_find 找出 ./var 下所有 *.log」。 +``FA_*`` 橋接 +-------------------- -檢視工具目錄 ------------- +橋接開啟時，``tools/list`` 先回傳語意工具，接著是每個已註冊的動作各一個工具，依名稱 +排序，其 JSON Schema 由 Python 簽名推導而來。對這類工具呼叫 ``tools/call`` 會透過 +註冊表派送，並以 JSON 編碼的回傳值回應；失敗則是 JSON-RPC 錯誤。先前的版本就是這樣 +運作的，既有的設定可以繼續使用。 -可呈現與宿主完全一致的描述符——便於測試、GUI 除錯或產生文件： +.. code-block:: text + + python -m automation_file mcp # 語意工具 + 每個 FA_* 動作 + python -m automation_file mcp --allowed-actions FA_list_dir,FA_file_checksum + python -m automation_file mcp --root /srv/reports --no-bridge # 只有語意工具 + +``--allowed-actions`` 把橋接縮小到指定的動作。引數裡提到清單以外動作的呼叫會被拒絕 +（無法用 ``FA_execute_action`` 繞過去）。``--no-bridge`` 或 +``MCPServer(bridge=False)`` 會關閉橋接；此時 ``FA_*`` 名稱是未知的工具。 + +**政策不約束橋接。** 透過橋接呼叫的 ``FA_storage_copy`` 可以到達行程能到的任何 +位置，不管 ``--root`` 怎麼設定。面對 AI 用戶端請使用 ``--no-bridge``；橋接只留給 +那些即使沒有政策你也願意讓用戶端呼叫的工具。 + +從 Python 使用： .. code-block:: python - from automation_file import tools_from_registry, executor + from automation_file import MCPServer, executor, tools_from_registry + from automation_file.server.mcp_policy import MCPPolicy + from automation_file.server.mcp_tools import SemanticToolkit + + MCPServer().serve_stdio() # 和以前一樣：橋接開啟 + MCPServer(policy=MCPPolicy(roots=["/srv/reports"]), bridge=False).serve_stdio() - for tool in tools_from_registry(executor.registry): + for tool in tools_from_registry(executor.registry): # 橋接的工具目錄 print(tool["name"], "->", tool["description"]) -每個描述符的形狀如下:: + toolkit = SemanticToolkit(MCPPolicy(roots=["/srv/reports"])) # 不經 JSON-RPC 使用工具 + outcome = toolkit.call("file_checksum", {"uri": "/srv/reports/a.csv"}) + outcome.is_error, outcome.payload["value"], outcome.correlation_id + +旗標 +-------- + +``python -m automation_file mcp`` 與 ``automation_file_mcp`` 主控台指令接受相同的 +旗標。 + +.. list-table:: + :header-rows: 1 + :widths: 32 68 + + * - 旗標 + - 意義 + * - ``--name``、``--version`` + - 握手時的 ``serverInfo``。預設值：``automation_file``、``1.0.0``。 + * - ``--allowed-actions a,b`` + - 橋接提供的已註冊動作。預設：全部。 + * - ``--no-bridge`` + - 只提供語意工具。 + * - ``--root URI`` + - 一個允許的位置；可重複。 + * - ``--allow-write``、``--allow-overwrite``、``--allow-delete`` + - 三項權限。後兩項需要第一項。 + * - ``--max-read-bytes N``、``--max-write-bytes N``、``--max-results N``、 + ``--max-search-bytes N`` + - 各項上限。 + * - ``--pipeline-dir URI`` + - 管線定義存放的地方。 + * - ``--pipeline-actions a,b`` + - 透過 MCP 執行的管線可以呼叫的動作。 + * - ``--tools a,b`` + - 要提供的語意工具，或 ``none``。 + +錯誤的旗標（未知的工具名稱、沒有 ``--allow-write`` 的 ``--allow-overwrite``、帶有 +憑證的根位置）會讓指令在開始服務之前就以用法錯誤結束。 + +安全指引 +---------------- - { - "name": "FA_fast_find", - "description": "底層可呼叫物件的第一行 docstring。", - "inputSchema": { - "type": "object", - "properties": {"root": {"type": "string"}, "pattern": {"type": "string"}}, - "required": ["root", "pattern"], - "additionalProperties": true, - }, - } +* **行程的權限就是外部界線。** 伺服器以啟動它的使用者身分執行，使用後端被給予的 + 憑證。政策為語意工具縮小這個範圍；它不能取代帳號、bucket policy 或 SFTP 使用者 + 上的最小權限。 +* **面對 AI 用戶端請關閉橋接**\ （``--no-bridge``）。橋接提供每個已註冊的動作， + 包含 ``FA_run_shell`` 與 ``FA_storage_delete``，而政策對它不適用。 +* **根位置要盡量小。** 根位置是對其下所有內容的授權。不要使用 ``local:///`` 或家 + 目錄。用戶端可以寫入的東西，請放在它專屬的目錄裡。 +* **從唯讀開始。** 工作需要時才加上 ``--allow-write``，真的需要時才加上 + ``--allow-overwrite`` 與 ``--allow-delete``。能寫入但不能覆寫或刪除的用戶端， + 無法破壞原本就在那裡的東西。 +* **檔案的內容不是指令。** 模型透過 ``file_read`` 與 ``file_search`` 讀到檔案內容， + 並可能依照讀到的東西行動。政策限制了這樣被挾持的工作階段能做的事；宿主的確認 + 提示也是。對重要的操作，先要求一次 ``dry_run``。 +* **管線。** 預設的動作留在根位置之內。你用 ``--pipeline-actions`` 加入的每個動作 + 都不受限制地執行。管線目錄裡放的是 AI 用戶端寫下的定義：只有 ``pipeline_run`` + 會對它們套用政策，所以不要用 ``FA_pipeline_run`` 或 ``Pipeline.from_file`` 執行 + 它們，也不要讓排程器指向那個目錄。 +* **回報類工具不受根位置約束。** ``audit_search``、``integrity_status`` 與 + ``pipeline_status`` 會顯示整個行程裡的資源名稱、錯誤與執行參數。對於不該看到這些 + 的用戶端，請不要把它們列入 ``--tools``。 +* **用戶端的名稱不能證明任何事。** 它只是稽核軌跡裡 actor 的標籤，而且由用戶端 + 自己提供。stdio 沒有驗證機制：能啟動這個行程的人，就擁有它的能力。 +* **機密。** 憑證絕不該放在儲存 URI 裡（帶有憑證的 URI 會被拒絕），也不該放在管線 + 參數裡，因為參數會隨執行一起被記錄。日誌不含引數值，也不含檔案內容。 +* **網路。** 語意工具本身不發出任何 HTTP 請求。儲存後端保有各自的檢查：TLS 驗證、 + SFTP 主機金鑰，以及會抓取 URL 的動作所用的 SSRF 防護。 +* 不要在提供橋接的行程裡呼叫 ``PackageLoader.add_package_to_executor``：它會把套件 + 的每個成員註冊成動作。 + +出問題時 +---------------- + +``no storage location is allowed on this server`` + 沒有設定任何根位置。加上 ``--root <儲存 URI 或目錄>``，或傳入 + ``MCPPolicy(roots=[...])``。 + +``... is outside the allowed locations (...)`` + 位置不在任何根位置之內；訊息裡會列出根位置。請對照根位置檢查寫法：另一個 + bucket 或主機、相鄰的目錄（``reports`` 旁邊的 ``reports-old``）、authority 的 + 大小寫不同，或通往同一個目錄的另一條路徑。 + +``... leaves the allowed location through a link or an absolute path`` + 本機根位置底下的符號連結或 junction 指向根位置之外。如果用戶端應該能到達那裡， + 請把連結的目標加為根位置。 + +``this server is read-only``、``does not allow it`` + 權限沒有開啟：``--allow-write``、``--allow-overwrite`` 或 ``--allow-delete``。 + ``file_move`` 需要寫入與刪除。 + +``already exists`` + 傳入 ``overwrite=true``；伺服器也必須允許覆寫。 + +``file_read`` 只回傳檔案的一部分 + ``truncated`` 為 true：以 ``offset=next_offset`` 再呼叫一次，或調高 + ``--max-read-bytes``。對於不支援範圍讀取的後端，每次呼叫都會把整個檔案暫存到 + 本機，所以翻閱很大的遠端檔案時請節制。 + +``file_search`` 漏掉某個檔案 + 檢查 ``complete`` 與 ``skipped``。超過 ``--max-search-bytes`` 剩餘額度的檔案 + 不會被搜尋；請縮小 ``pattern`` 或調高額度。內容是以 UTF-8 文字比對的。 + +``... is not allowed in a pipeline`` + 定義提到了允許範圍以外的動作；訊息裡會列出允許的動作。請使用儲存動作，或用 + ``--pipeline-actions`` 加入該動作。 + +管線任務以 ``MCPLocationException`` 或 ``MCPPermissionException`` 失敗 + 任務在執行時到達了根位置以外的位置，或需要一項沒有開啟的權限。定義之所以被 + 接受，是因為位置來自參數。 + +``no pipeline named ... is stored`` + 訊息裡會列出已儲存的名稱。沒有 ``--pipeline-dir`` 時，伺服器重新啟動後定義就 + 不在了。 + +``pipeline_status`` 找不到某次執行 + 執行紀錄預設存在記憶體裡。請在啟動腳本裡呼叫 + ``set_default_run_store(SQLiteRunStore(path))``。 + +``this server keeps no audit trail`` + 請在啟動腳本裡於 ``serve_stdio()`` 之前呼叫 ``configure_audit(path)``。 + +``sftp://`` 或 ``s3://`` 位置明明在根位置之內卻失敗 + 這個行程裡的後端沒有初始化。請在啟動腳本裡呼叫它的 ``later_init``；見 + `正式環境設定`_ 與 :doc:`storage`。 + +宿主列出數百個工具，或完全沒有語意工具 + 前者是橋接：加上 ``--no-bridge`` 或 ``--allowed-actions``。後者是 + ``--tools none``，或套件的版本早於語意工具。 + +宿主在啟動時回報連線中斷 + 有東西寫到了 ``stdout``，或指令以用法錯誤結束。請在終端機裡執行同一道指令： + 錯誤會寫到 ``stderr``。手動檢查伺服器的方法:: + + printf '%s\n%s\n' \ + '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{}}' \ + '{"jsonrpc":"2.0","id":2,"method":"tools/list"}' \ + | python -m automation_file mcp --root /srv/reports --no-bridge -手動煙霧測試 ------------- - -直接把 JSON-RPC frame 餵給伺服器以確認其能正常載入:: - - printf '%s\n%s\n%s\n' \ - '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{}}' \ - '{"jsonrpc":"2.0","id":2,"method":"tools/list"}' \ - '{"jsonrpc":"2.0","id":3,"method":"tools/call","params":{"name":"FA_fast_find","arguments":{"root":".","pattern":"*.py","limit":3}}}' \ - | python -m automation_file mcp - -每行輸入在 stdout 上恰得到一條 JSON-RPC 回應（通知沒有回應）。 - -安全注意事項 ------------- - -* MCP 伺服器以 **啟動它的 Python 行程的權限** 執行。 - 每次工具呼叫都落在你 shell 使用者帳號中。 -* 預設情況下，伺服器暴露 **全部** 已註冊的 ``FA_*`` 動作， - 其中包含會刪除檔案、寫檔案系統、上傳到遠端後端的動作。 - 對任何你並非完全信任的宿主，請用 ``--allowed-actions`` 只白名單暴露 - 必要動作。 -* 在打算給第三方宿主使用的 MCP 伺服器啟動前，**不要** 呼叫 - :func:`~automation_file.PackageLoader.add_package_to_executor` - （或暴露其動作）。該輔助函式會註冊任意套件的所有頂層函式 / 類別 / 內建， - 威力相當於 ``eval``。 -* 伺服器啟動時會透過 ``file_automation_logger`` 以 ``INFO`` 等級記錄 - 暴露的工具數量與設定的伺服器名稱。工具呼叫負載從不被記錄。 -* 涉及對外 HTTP 的動作（``FA_download_file``、各雲端後端）仍會通過 - SSRF 檢查；MCP 層不會放鬆任何單一動作的檢查。 +語意工具的每個例外都衍生自 ``MCPServerException``，因此也衍生自 +``FileAutomationException``：``MCPPermissionException``\ （帶有 ``code``）、它的子類別 +``MCPLocationException``，以及 ``MCPToolException``\ （帶有 ``kind``）。 diff --git a/docs/updates/2026-10.md b/docs/updates/2026-10.md index 318d9a7..530af2b 100644 --- a/docs/updates/2026-10.md +++ b/docs/updates/2026-10.md @@ -587,3 +587,21 @@ Index and query commands: [README.md](README.md). New entries go at the end. - **Docs**: chapter 23 in the three manuals (`usage/deployment.rst`), the indexes, a "Deployment" section in the three READMEs. - **Files**: the three `usage/deployment.rst` pages, the three indexes, the three READMEs. - **Open items**: the scheduler, MCP and GUI specifics are added when that work lands (#26). + +## U-20261008-27 · 2026-10-08 · Semantic MCP tools · #mcp #security #roadmap #done + +- **What** (roadmap §12, M8; closes `progress.md` #25): the MCP server offers fourteen tools with stable names next to the `FA_*` bridge: `file_read`, `file_write`, `file_copy`, `file_move`, `file_search`, `file_checksum`, `file_verify`, `storage_list`, `storage_copy`, `pipeline_create`, `pipeline_run`, `pipeline_status`, `integrity_status`, `audit_search`. Each has a hand-written input schema and a structured result, and works on storage URIs through `File` / `Storage`. + - **Permission model** (`MCPPolicy`, fixed when the server starts): the roots a tool may touch (`--root`, repeatable; with none, the storage tools refuse and say how to configure one); read-only by default, with `--allow-write`, `--allow-overwrite` and `--allow-delete` as three separate permissions; limits on bytes read, bytes written, results and bytes searched; a per-tool list (`--tools`). A local root is enforced by `LocalStorage(root)`, so `..` and links cannot leave it; a remote root by scheme, exact authority and a path prefix that ends at a segment boundary (`s3://bucket/team` does not cover `s3://bucket/team-b`). + - **Dry run**: every tool that changes something takes `dry_run` and reports what it would do. A read-only server refuses a changing tool even as a dry run. + - **Pipelines**: `pipeline_create` validates a definition and stores it as `.json` in `--pipeline-dir` (in memory without one), and refuses a `schedule`. A pipeline made or run through MCP gets a registry of its own with guarded `FA_storage_*` actions, so the roots are checked again when each task runs. `--pipeline-actions` replaces that set; an action listed there runs unconfined, which the manual says. + - **Traceability**: each call runs in a correlation scope as the actor `mcp`, returns the correlation ID and publishes one `mcp.tool.*` event; a refused call is logged without its argument values and returned as a result with `isError`. + - `tools/list` returns the semantic tools first, then the bridge; `--no-bridge` leaves only the semantic ones. `python -m automation_file mcp` forwards every flag. + - Beyond the roadmap: `verify` on a copy or a move (a move deletes its source only after the SHA-256 matches), `background` on `pipeline_run`, usage instructions in the handshake, Windows device and data-stream names refused below local roots. +- **Changed while integrating**: one test compared two audit records in a fixed order although both can carry the same timestamp; it failed once in the base-dependency run. It now compares them as a set. +- **Tests**: `tests/test_mcp_policy.py`, `test_mcp_tools.py`, `test_mcp_storage_tools.py`, `test_mcp_pipeline_tools.py`, `test_mcp_semantic_server.py` with `tests/mcp_support.py`; `tests/test_mcp_server.py` passes unchanged. +- **Result / numbers**: 5260 passed, 257 skipped, 0 failed with every extra; 3479 passed, 135 skipped with the base dependencies only. `ruff check`, `ruff format --check` and `mypy automation_file` (239 files) pass. Python 3.14.7 on Windows. +- **Not verified**: a real MCP host (a child process over stdio was used); real S3 or SFTP behind the tools; whether the link-escape tests ran through a symbolic link or the Windows junction fallback, and POSIX at all; the Sphinx build of the rewritten pages; Python 3.10 to 3.13. +- **Found and not changed** (`progress.md` #37): a `LocalStorage` listing reports a link's name and its target's metadata without a containment check (reading through it is refused); `StorageBackend` has no ranged read, so `file_read` on a remote file stages the whole file; `LocalStorage` on Windows opens device names such as `CON` and `NUL`; a link on an SFTP server cannot be confined by a path prefix. +- **Docs**: the three `usage/mcp.rst` pages rewritten (setup, the tool table, the permission model, dry run, the roadmap's S3-to-SFTP workflow, the bridge, security, diagnostics), the MCP part of the three `usage/cli.rst`, `docs/source/API/server.rst`, `examples/mcp/`, the feature list and the MCP section of the three READMEs, `architecture.md` §2 and §3, `CLAUDE.md` (package map, § Security › MCP server). +- **Files**: `automation_file/server/mcp_{policy,tools,tool_model,file_tools,storage_tools,pipeline_tools,pipeline_actions,report_tools}.py`, `automation_file/server/mcp_server.py`, `automation_file/__main__.py`, `automation_file/__init__.py`, the tests above, the documentation above. +- **Open items**: #37. diff --git a/docs/updates/README.md b/docs/updates/README.md index 6fececa..d3ce33a 100644 --- a/docs/updates/README.md +++ b/docs/updates/README.md @@ -58,6 +58,7 @@ In the same commit: delete the item from `progress.md`, add a `#done` entry here | ID | Date | Title | Tags | Batch | |---|---|---|---|---| +| U-20261008-27 | 2026-10-08 | Semantic MCP tools | #mcp #security #roadmap #done | [2026-10](2026-10.md) | | U-20261008-26 | 2026-10-08 | Production deployment guide | #docs #roadmap | [2026-10](2026-10.md) | | U-20261008-25 | 2026-10-08 | Metadata cases in the storage contract | #storage #tests #done | [2026-10](2026-10.md) | | U-20261008-24 | 2026-10-08 | A release can raise MINOR or MAJOR | #release #ci #roadmap | [2026-10](2026-10.md) | @@ -121,5 +122,5 @@ In the same commit: delete the item from `progress.md`, add a `#done` entry here | File | Period | Entries | |---|---|---:| -| [2026-10.md](2026-10.md) | 2026-10 | 37 | +| [2026-10.md](2026-10.md) | 2026-10 | 38 | | [2026-09.md](2026-09.md) | 2026-09 | 21 | diff --git a/examples/mcp/README.md b/examples/mcp/README.md index 00742f6..d512184 100644 --- a/examples/mcp/README.md +++ b/examples/mcp/README.md @@ -1,6 +1,6 @@ # automation_file MCP server -Three ways to launch the MCP stdio bridge, in order of preference: +Three ways to launch the MCP stdio server, in order of preference: ```bash automation_file_mcp # installed console script @@ -8,13 +8,35 @@ python -m automation_file mcp # CLI subcommand python examples/mcp/run_mcp.py # standalone launcher ``` -All three accept the same flags: +The server offers the fourteen **semantic tools** (`file_read`, `file_copy`, +`pipeline_run`, ...), which are bound to a permission policy, and the **bridge** +that exposes every registered `FA_*` action. For an AI client, give the semantic +tools a root and switch the bridge off: -| Flag | Default | Description | -|----------------------|--------------------|------------------------------------------------| -| `--name` | `automation_file` | `serverInfo.name` returned at handshake | -| `--version` | `1.0.0` | `serverInfo.version` returned at handshake | -| `--allowed-actions` | *(all)* | Comma-separated allow list (e.g. `FA_file_checksum,FA_fast_find`) | +```bash +automation_file_mcp --root /srv/reports --no-bridge # read-only +automation_file_mcp --root /srv/reports --allow-write --no-bridge # may create files +``` + +All three launch styles accept the same flags: + +| Flag | Default | Description | +|-----------------------|--------------------|------------------------------------------------| +| `--name` | `automation_file` | `serverInfo.name` returned at handshake | +| `--version` | `1.0.0` | `serverInfo.version` returned at handshake | +| `--allowed-actions` | *(all)* | Comma-separated allow list for the bridge (e.g. `FA_file_checksum,FA_fast_find`) | +| `--no-bridge` | *(bridge on)* | Offer only the semantic tools | +| `--root` | *(none)* | A location the semantic tools may work in: a storage URI or a local directory; repeatable | +| `--allow-write` | off | Let the semantic tools create files | +| `--allow-overwrite` | off | Let them replace an existing file (needs `--allow-write`) | +| `--allow-delete` | off | Let them delete; a move deletes its source (needs `--allow-write`) | +| `--max-read-bytes` | `262144` | Most bytes `file_read` returns in one call | +| `--max-write-bytes` | `1048576` | Largest content `file_write` accepts | +| `--max-results` | `200` | Most entries a listing or a search returns | +| `--max-search-bytes` | `8388608` | Most bytes one content search reads | +| `--pipeline-dir` | *(memory)* | Where `pipeline_create` keeps definitions | +| `--pipeline-actions` | *(by permission)* | Comma-separated actions a pipeline run through MCP may call | +| `--tools` | *(all fourteen)* | Comma-separated semantic tools to offer, or `none` | ## Claude Desktop @@ -23,8 +45,9 @@ Edit `claude_desktop_config.json`: - **Windows** — `%APPDATA%\Claude\claude_desktop_config.json` - **macOS** — `~/Library/Application Support/Claude/claude_desktop_config.json` -`claude_desktop_config.json` in this directory is a ready-to-copy sample -covering the three launch styles. Pick the one that matches your install. +`claude_desktop_config.json` in this directory is a ready-to-copy sample: the +semantic tools on one directory, the bridge narrowed to an allow list, the full +bridge, and the standalone launcher. Pick the one that matches your install. ## Manual smoke test @@ -36,6 +59,11 @@ A single line reply containing `serverInfo` means the server is healthy. ## Security -`--allowed-actions` is strongly recommended. The default registry includes -`FA_run_shell`, `FA_encrypt_file`, and other high-privilege actions that an -MCP host may invoke without prompting. +Use `--no-bridge` for an AI client. The semantic tools stay inside `--root` and +are read-only until `--allow-write`; the bridge is not bound by either. The +default registry includes `FA_run_shell`, `FA_encrypt_file`, and other +high-privilege actions that an MCP host may invoke without prompting, so when +the bridge stays on, narrow it with `--allowed-actions`. + +The full manual, with the permission model and an example workflow, is +`docs/source/Eng/usage/mcp.rst`. diff --git a/examples/mcp/claude_desktop_config.json b/examples/mcp/claude_desktop_config.json index fad87a5..847a94a 100644 --- a/examples/mcp/claude_desktop_config.json +++ b/examples/mcp/claude_desktop_config.json @@ -1,5 +1,13 @@ { "mcpServers": { + "automation_file_semantic": { + "command": "automation_file_mcp", + "args": [ + "--root", + "C:/absolute/path/to/reports", + "--no-bridge" + ] + }, "automation_file": { "command": "automation_file_mcp", "args": [ diff --git a/progress.md b/progress.md index 3df88c3..ad22578 100644 --- a/progress.md +++ b/progress.md @@ -27,7 +27,7 @@ Items #10 to #26 are what is left of the 1.0 roadmap (`docs/FILEAUTOMATION-1.0-R - **#23** Scheduler v2 (roadmap §8, the open half of M6): one scheduler with cron (time-zone aware), manual, file-event, webhook and pipeline-dependency triggers, run states and overlap protection, reading a pipeline's `schedule`. The event model, the `NotificationRouter` and audit schema v2 are done (U-20261008-05, U-20261008-17); the scheduler in `scheduler/` still dispatches action lists on its own cron loop. - **#24** UI 2.0 (roadmap §11, M7). Not before the APIs of #13 to #23 are stable (roadmap §20). -- **#25** Semantic MCP tools (roadmap §12, M8): `file_*`, `storage_*`, `pipeline_*`, `integrity_status`, `audit_search`, with a permission model and dry run, next to the existing `FA_*` bridge. +- **#37** Storage-layer gaps the MCP work found (U-20261008-27) and did not change: (a) a `LocalStorage(root)` listing reports a link's name and its target's metadata without a containment check, although reading through the link is refused; (b) `StorageBackend` has no ranged read, so reading the head of a large remote file stages all of it; (c) `LocalStorage` on Windows opens device names (`CON`, `NUL`) and alternate data streams, which the MCP tools refuse themselves; (d) a link on an SFTP or FTP server leads outside a root that is only a path prefix. - **#26** Release engineering and 1.0 (roadmap §13, M9). Done: semantic versioning with a way to release a MINOR or MAJOR (U-20261008-24), the public API policy (U-20261008-22), a package-build check and the integration workflow next to the PR checks. Open: the migration guide and the final documentation audit (after the scheduler, MCP and GUI work lands); making the integration jobs required once they are green (#19); the 1.0.0 release itself, which is the owner's call: write `1.0.0` in both TOMLs in the release pull request. - **#35** [UNVERIFIED] One full run of the suite on 2026-10-08 reported `1 failed, 4882 passed` and four runs around it passed. The machine was running three other test suites at the time, and the run was not started with `-rf`, so the test is not known. A test that depends on timing is the likely cause (the pipeline timeout and cancellation cases, the integrity watchers, the SFTP loopback, the scheduler). Run the suite with `-rf` under load, or read the first CI runs, to find it. - **#34** [BLOCKED] PyPI Trusted Publishing (roadmap §13). `publish.yml` and `publish-dev` still upload with the `PYPI_API_TOKEN` secret. Switching needs the owner to add a trusted publisher for each project on PyPI (`automation_file`: workflow `publish.yml`; `automation_file_dev`: workflow `ci-dev.yml`; an environment name if one is wanted) before the workflows can drop the token for `id-token: write` and `pypa/gh-action-pypi-publish`. Changing the workflows first would break both channels. diff --git a/tests/mcp_support.py b/tests/mcp_support.py new file mode 100644 index 0000000..eaaf064 --- /dev/null +++ b/tests/mcp_support.py @@ -0,0 +1,103 @@ +"""Helpers shared by the tests of the semantic MCP tools.""" + +from __future__ import annotations + +import contextlib +import os +from collections.abc import Iterator +from pathlib import Path +from typing import Any + +import pytest + +from automation_file.audit import MemoryAuditStore, audit_trail, configure_audit +from automation_file.integrity import actions as integrity_actions +from automation_file.pipeline import MemoryRunStore, set_default_run_store +from automation_file.server.mcp_policy import MCPPolicy +from automation_file.server.mcp_tools import SemanticToolkit, ToolOutcome +from automation_file.storage import clear_memory_stores + +BOX = "memory://box" +INBOX = f"{BOX}/in" + + +def link_directory(link: Path, target: Path) -> None: + """Make ``link`` lead to the directory ``target``, or skip the test when that is impossible. + + A symbolic link where the platform allows one; on Windows without that + privilege a junction, which needs none and is followed the same way. + """ + with contextlib.suppress(OSError, NotImplementedError): + link.symlink_to(target, target_is_directory=True) + return + if os.name != "nt": + pytest.skip("symbolic links cannot be created here") + import _winapi + + try: + _winapi.CreateJunction(str(target), str(link)) + except OSError: + pytest.skip("neither a symbolic link nor a junction can be created here") + + +def remove_link(link: Path) -> None: + """Remove what :func:`link_directory` made, leaving its target alone.""" + with contextlib.suppress(OSError): + if link.is_symlink(): + link.unlink() + else: + os.rmdir(link) + + +def toolkit(**policy: Any) -> SemanticToolkit: + """Return a toolkit whose policy allows the ``memory://box/in`` tree unless told otherwise.""" + policy.setdefault("roots", (INBOX,)) + return SemanticToolkit(MCPPolicy(**policy)) + + +def writable(**policy: Any) -> SemanticToolkit: + """Return a toolkit that may write below ``memory://box/in``.""" + return toolkit(allow_write=True, **policy) + + +def succeed(outcome: ToolOutcome) -> dict[str, Any]: + """Return the payload of an outcome that must not be an error.""" + assert outcome.is_error is False, outcome.payload + return outcome.payload + + +def refused(outcome: ToolOutcome, code: str) -> dict[str, Any]: + """Return the error of an outcome the policy must have refused with ``code``.""" + assert outcome.is_error is True, outcome.payload + error = outcome.payload["error"] + assert (error["type"], error["code"]) == ("permission_denied", code), error + return error + + +def failed(outcome: ToolOutcome, kind: str) -> dict[str, Any]: + """Return the error of an outcome that must have failed as ``kind``.""" + assert outcome.is_error is True, outcome.payload + error = outcome.payload["error"] + assert error["type"] == kind, error + return error + + +@contextlib.contextmanager +def clean_state() -> Iterator[None]: + """Give a test empty memory stores and a run store of its own, and take both away after.""" + clear_memory_stores() + previous = set_default_run_store(MemoryRunStore()) + try: + yield + finally: + set_default_run_store(previous) + integrity_actions.stop_all_monitors() + audit_trail.close() + clear_memory_stores() + + +def audited() -> MemoryAuditStore: + """Point the process-wide audit trail at a fresh in-memory store and return the store.""" + store = MemoryAuditStore() + configure_audit(store) + return store diff --git a/tests/test_mcp_pipeline_tools.py b/tests/test_mcp_pipeline_tools.py new file mode 100644 index 0000000..9471f3b --- /dev/null +++ b/tests/test_mcp_pipeline_tools.py @@ -0,0 +1,729 @@ +"""The semantic pipeline and reporting tools, and the guarded actions a pipeline runs with.""" + +from __future__ import annotations + +import json +import time +from collections.abc import Iterator +from pathlib import Path +from typing import Any + +import pytest + +from automation_file import PipelineStarted, correlation_scope, emit +from automation_file.core.action_registry import ActionRegistry +from automation_file.exceptions import ( + MCPServerException, + StorageAlreadyExistsException, + StorageChecksumException, + StorageException, +) +from automation_file.integrity import actions as integrity_actions +from automation_file.pipeline import default_run_store +from automation_file.server.mcp_pipeline_actions import ( + GUARDED_ACTIONS, + GuardedStorageActions, + default_pipeline_actions, +) +from automation_file.server.mcp_pipeline_tools import ( + action_refusals, + allowed_actions, + definitions, + pipeline_registry, +) +from automation_file.server.mcp_policy import ( + ACTION_NOT_ALLOWED, + DELETE_NOT_ALLOWED, + LIMIT_EXCEEDED, + OUTSIDE_ROOT, + OVERWRITE_NOT_ALLOWED, + READ_ONLY, + MCPLocationException, + MCPPermissionException, + MCPPolicy, +) +from automation_file.server.mcp_tools import SemanticToolkit +from automation_file.storage import File, Storage +from tests.mcp_support import ( + BOX, + INBOX, + audited, + clean_state, + failed, + refused, + succeed, + toolkit, + writable, +) + +SOURCE = f"{INBOX}/in.csv" +CALLS: list[dict[str, Any]] = [] + + +@pytest.fixture(autouse=True) +def _state() -> Iterator[None]: + with clean_state(): + File(SOURCE).write("id,amount\n1,10\n") + CALLS.clear() + yield + + +def _definition(**extra: Any) -> dict[str, Any]: + target = f"{INBOX}/out/${{params.date}}.csv" + return { + "schema_version": 1, + "params": {"date": "2026-10-07"}, + "tasks": { + "copy": {"action": ["FA_storage_copy", {"source": SOURCE, "target": target}]}, + "check": { + "action": ["FA_storage_checksum", {"uri": target}], + "depends_on": ["copy"], + }, + }, + **extra, + } + + +def _one_task(action: list[Any]) -> dict[str, Any]: + return {"schema_version": 1, "tasks": {"only": {"action": action}}} + + +def _create(kit: SemanticToolkit, name: str = "daily", **extra: Any): + return kit.call("pipeline_create", {"name": name, "definition": _definition(), **extra}) + + +def _notify(**arguments: Any) -> str: + CALLS.append(arguments) + return "sent" + + +def _with_notify(**policy: Any) -> SemanticToolkit: + """A writable toolkit whose registry has one action of its own, ``FA_test_notify``.""" + registry = ActionRegistry({"FA_test_notify": _notify, "FA_test_other": _notify}) + policy.setdefault("roots", (INBOX,)) + return SemanticToolkit(MCPPolicy(allow_write=True, **policy), registry) + + +# ---------------------------------------------------------------------- pipeline_create + + +def test_pipeline_create_validates_and_stores_the_definition() -> None: + kit = writable() + body = succeed(_create(kit)) + assert body["name"] == "daily" + assert body["tasks"] == ["copy", "check"] + assert body["actions"] == ["FA_storage_checksum", "FA_storage_copy"] + assert (body["stored"], body["overwrites"], body["dry_run"]) == (True, False, False) + assert (body["persistent"], body["location"]) == (False, "memory (kept until the server stops)") + stored = definitions(kit.session).load("daily", 10_000) + assert stored["name"] == "daily" + assert stored["tasks"].keys() == {"copy", "check"} + assert not File(f"{INBOX}/out/2026-10-07.csv").exists() + + +def test_pipeline_create_as_a_dry_run_stores_nothing() -> None: + kit = writable() + body = succeed(_create(kit, dry_run=True)) + assert (body["stored"], body["dry_run"]) == (False, True) + assert definitions(kit.session).names() == [] + + +def test_pipeline_create_keeps_definitions_in_the_pipeline_directory(tmp_path: Path) -> None: + directory = tmp_path / "pipelines" + kit = writable(pipeline_dir=directory) + body = succeed(_create(kit)) + assert body["persistent"] is True + assert body["location"].startswith("local:///") + document = json.loads((directory / "daily.json").read_text(encoding="utf-8")) + assert document["name"] == "daily" + again = writable(pipeline_dir=directory) + assert definitions(again.session).names() == ["daily"] + assert succeed(again.call("pipeline_run", {"name": "daily"}))["status"] == "succeeded" + + +def test_the_pipeline_directory_may_be_any_storage_uri() -> None: + kit = writable(pipeline_dir=f"{BOX}/definitions") + succeed(_create(kit)) + assert File(f"{BOX}/definitions/daily.json").exists() + assert succeed(kit.call("pipeline_run", {"name": "daily"}))["status"] == "succeeded" + + +def test_pipeline_create_reports_every_problem_of_a_definition() -> None: + broken = {"schema_version": 1, "max_workers": 0, "tasks": {"a": {"depends_on": ["x"]}}} + outcome = writable().call("pipeline_create", {"name": "broken", "definition": broken}) + error = failed(outcome, "invalid_definition") + assert "max_workers: expected an integer >= 1, got 0" in error["problems"] + assert "tasks.a.action: required" in error["problems"] + assert any("unknown task 'x'" in problem for problem in error["problems"]) + + +def test_pipeline_create_needs_a_schema_version() -> None: + outcome = writable().call("pipeline_create", {"name": "bare", "definition": {"tasks": {}}}) + assert failed(outcome, "invalid_definition")["problems"] == [ + "schema_version: required (supported: 1)" + ] + + +@pytest.mark.parametrize("name", ["../escape", "a/b", "", ".hidden", "x" * 101, "with space"]) +def test_a_pipeline_name_is_a_plain_file_name(name: str) -> None: + outcome = writable().call("pipeline_create", {"name": name, "definition": _definition()}) + assert failed(outcome, "invalid_arguments") + assert definitions(writable().session).names() == [] + + +def test_the_definition_must_name_itself_as_it_is_stored() -> None: + outcome = writable().call( + "pipeline_create", {"name": "daily", "definition": _definition(name="nightly")} + ) + assert "must be 'daily'" in failed(outcome, "invalid_arguments")["message"] + + +def test_a_definition_created_through_mcp_cannot_carry_a_schedule() -> None: + scheduled = _definition(schedule={"cron": "0 2 * * *"}) + outcome = writable().call("pipeline_create", {"name": "daily", "definition": scheduled}) + assert "schedule" in refused(outcome, ACTION_NOT_ALLOWED)["message"] + + +def test_replacing_a_definition_needs_the_argument_and_the_permission() -> None: + kit = writable() + succeed(_create(kit)) + assert "already stored" in failed(_create(kit), "already_exists")["message"] + refused(_create(kit, overwrite=True), OVERWRITE_NOT_ALLOWED) + allowed = writable(allow_overwrite=True) + succeed(_create(allowed)) + plan = succeed(_create(allowed, overwrite=True, dry_run=True)) + assert (plan["overwrites"], plan["stored"]) == (True, False) + assert succeed(_create(allowed, overwrite=True))["overwrites"] is True + + +def test_a_definition_larger_than_the_write_limit_is_refused() -> None: + kit = writable(max_write_bytes=64) + refused(_create(kit), LIMIT_EXCEEDED) + + +@pytest.mark.parametrize("tool", ["pipeline_create", "pipeline_run"]) +def test_pipelines_are_refused_on_a_read_only_server(tool: str) -> None: + arguments = {"name": "daily", "definition": _definition(), "dry_run": True} + if tool == "pipeline_run": + del arguments["definition"] + refused(toolkit().call(tool, arguments), READ_ONLY) + + +# ---------------------------------------------------------------------- which actions + + +def test_the_default_actions_follow_the_permissions() -> None: + read_only = default_pipeline_actions(MCPPolicy()) + assert "FA_storage_checksum" in read_only + assert "FA_storage_copy" not in read_only + writer = default_pipeline_actions(MCPPolicy(allow_write=True)) + assert {"FA_storage_copy", "FA_storage_write_text", "FA_storage_copy_tree"} <= writer + assert not {"FA_storage_sync", "FA_storage_move", "FA_storage_delete"} & writer + everything = default_pipeline_actions( + MCPPolicy(allow_write=True, allow_overwrite=True, allow_delete=True) + ) + assert everything == GUARDED_ACTIONS + assert all(name.startswith("FA_storage_") for name in everything) + assert not {"FA_storage_upload", "FA_storage_download"} & everything + + +@pytest.mark.parametrize( + "action", + [ + ["FA_run_shell", {"argv": ["echo", "hi"]}], + ["FA_storage_upload", {"local_path": "/etc/passwd", "uri": f"{INBOX}/p"}], + ["FA_storage_delete", {"uri": SOURCE}], + ["FA_execute_action", {"action_list": [["FA_storage_exists", {"uri": SOURCE}]]}], + ["FA_no_such_action"], + ], +) +def test_an_action_outside_the_allowed_set_is_refused_at_create(action: list[Any]) -> None: + outcome = writable().call("pipeline_create", {"name": "bad", "definition": _one_task(action)}) + error = refused(outcome, ACTION_NOT_ALLOWED) + assert f"tasks.only.action[0]: {action[0]} is not allowed" in error["message"] + assert "FA_storage_copy" in error["message"] + + +def test_an_action_hidden_in_the_arguments_of_another_one_is_refused() -> None: + hidden = ["FA_storage_exists", {"uri": [["FA_run_shell", {"argv": ["id"]}]]}] + outcome = writable().call("pipeline_create", {"name": "bad", "definition": _one_task(hidden)}) + assert ( + "its arguments name the action FA_run_shell" + in (refused(outcome, ACTION_NOT_ALLOWED)["message"]) + ) + by_default = _definition(params={"date": "x", "extra": ["FA_run_shell"]}) + outcome = writable().call("pipeline_create", {"name": "bad", "definition": by_default}) + assert ( + "a parameter names the action FA_run_shell" + in (refused(outcome, ACTION_NOT_ALLOWED)["message"]) + ) + + +def test_the_policy_may_list_the_actions_itself() -> None: + kit = _with_notify(pipeline_actions=["FA_storage_exists", "FA_test_notify"]) + assert allowed_actions(kit.session) == {"FA_storage_exists", "FA_test_notify"} + definition = _one_task(["FA_test_notify", {"subject": "done"}]) + succeed(kit.call("pipeline_create", {"name": "tell", "definition": definition})) + assert succeed(kit.call("pipeline_run", {"name": "tell"}))["status"] == "succeeded" + assert CALLS == [{"subject": "done"}] + copying = _one_task(["FA_storage_copy", {"source": SOURCE, "target": f"{INBOX}/x"}]) + refused( + kit.call("pipeline_create", {"name": "copy", "definition": copying}), (ACTION_NOT_ALLOWED) + ) + other = _one_task(["FA_test_other", {}]) + refused(kit.call("pipeline_create", {"name": "o", "definition": other}), ACTION_NOT_ALLOWED) + + +def test_a_listed_action_the_server_does_not_have_is_refused_at_start() -> None: + with pytest.raises(MCPServerException, match="FA_missing"): + _with_notify(pipeline_actions=["FA_test_notify", "FA_missing"]) + with pytest.raises(MCPServerException, match="does not expose"): + SemanticToolkit(MCPPolicy(pipeline_actions=["FA_run_shell"]), ActionRegistry()) + + +def test_a_storage_action_stays_guarded_when_the_policy_lists_it() -> None: + kit = _with_notify(pipeline_actions=["FA_storage_delete", "FA_storage_copy"]) + registry = pipeline_registry(kit.session) + assert set(registry.event_dict) == {"FA_storage_delete", "FA_storage_copy"} + with pytest.raises(MCPPermissionException) as caught: + registry.resolve("FA_storage_delete")(uri=SOURCE) + assert caught.value.code == DELETE_NOT_ALLOWED + assert File(SOURCE).exists() + + +def test_a_nested_storage_action_is_refused_even_when_it_is_listed() -> None: + # Nested, it would be run by the shared executor: unguarded. + kit = _with_notify(pipeline_actions=["FA_test_notify", "FA_storage_copy"]) + nested = _one_task(["FA_test_notify", {"then": [["FA_storage_copy", {"source": "a"}]]}]) + refusals = action_refusals(kit.session, nested, None) + assert refusals == ["tasks.only.action: its arguments name the action FA_storage_copy"] + plain = _one_task(["FA_test_notify", {"then": "FA_test_notify"}]) + assert action_refusals(kit.session, plain, {"also": ["FA_test_notify"]}) == [] + assert action_refusals(kit.session, plain, {"also": ["FA_test_other"]}) == [ + "params: a parameter names the action FA_test_other" + ] + + +# ---------------------------------------------------------------------- pipeline_run + + +def test_pipeline_run_as_a_dry_run_returns_the_plan_and_runs_nothing() -> None: + kit = writable() + succeed(_create(kit)) + body = succeed(kit.call("pipeline_run", {"name": "daily", "dry_run": True})) + assert (body["dry_run"], body["status"], body["background"]) == (True, "succeeded", False) + tasks = body["run"]["tasks"] + assert list(tasks) == ["copy", "check"] + assert [state["status"] for state in tasks.values()] == ["planned", "planned"] + assert [state["level"] for state in tasks.values()] == [0, 1] + assert not File(f"{INBOX}/out/2026-10-07.csv").exists() + assert default_run_store().get_run(body["run_id"]) is None + + +def test_pipeline_run_runs_the_stored_definition_with_its_parameters() -> None: + kit = writable() + succeed(_create(kit)) + body = succeed(kit.call("pipeline_run", {"name": "daily", "params": {"date": "2026-10-08"}})) + assert (body["status"], body["ok"], body["name"]) == ("succeeded", True, "daily") + assert File(f"{INBOX}/out/2026-10-08.csv").read() == File(SOURCE).read() + assert body["run"]["params"] == {"date": "2026-10-08"} + assert body["run"]["tasks"]["check"]["result"]["algorithm"] == "sha256" + assert default_run_store().get_run(body["run_id"]) is not None + + +def test_a_failed_run_is_an_error_outcome_that_still_carries_the_run() -> None: + kit = writable() + definition = _one_task(["FA_storage_copy", {"source": f"{INBOX}/absent", "target": SOURCE}]) + succeed(kit.call("pipeline_create", {"name": "fails", "definition": definition})) + outcome = kit.call("pipeline_run", {"name": "fails"}) + failed(outcome, "failed") + assert outcome.payload["status"] == "failed" + assert outcome.payload["ok"] is False + assert "StorageNotFoundException" in outcome.payload["run"]["tasks"]["only"]["error"] + + +def test_a_dry_run_reports_a_missing_parameter_as_an_error_outcome() -> None: + kit = writable() + definition = _one_task(["FA_storage_exists", {"uri": f"{INBOX}/${{params.missing}}"}]) + succeed(kit.call("pipeline_create", {"name": "needs", "definition": definition})) + outcome = kit.call("pipeline_run", {"name": "needs", "dry_run": True}) + failed(outcome, "failed") + assert "unknown parameter 'missing'" in outcome.payload["run"]["tasks"]["only"]["error"] + + +def test_pipeline_run_of_an_unknown_name_lists_what_is_stored() -> None: + kit = writable() + succeed(_create(kit)) + error = failed(kit.call("pipeline_run", {"name": "nightly"}), "not_found") + assert "stored: daily" in error["message"] + + +def test_a_task_cannot_leave_the_allowed_locations_at_run_time() -> None: + File(f"{BOX}/private/secret.csv").write("secret") + kit = writable() + definition = _one_task( + ["FA_storage_copy", {"source": "${params.source}", "target": f"{INBOX}/stolen.csv"}] + ) + succeed(kit.call("pipeline_create", {"name": "sneaky", "definition": definition})) + outcome = kit.call( + "pipeline_run", {"name": "sneaky", "params": {"source": f"{BOX}/private/secret.csv"}} + ) + assert outcome.is_error is True + assert "MCPLocationException" in outcome.payload["run"]["tasks"]["only"]["error"] + assert not File(f"{INBOX}/stolen.csv").exists() + + +def test_the_allowed_actions_are_checked_again_when_the_definition_is_run(tmp_path: Path) -> None: + directory = tmp_path / "pipelines" + directory.mkdir() + planted = {"name": "planted", **_one_task(["FA_run_shell", {"argv": ["echo", "hi"]}])} + (directory / "planted.json").write_text(json.dumps(planted), encoding="utf-8") + kit = writable(pipeline_dir=directory) + refused(kit.call("pipeline_run", {"name": "planted"}), ACTION_NOT_ALLOWED) + refused(kit.call("pipeline_run", {"name": "planted", "dry_run": True}), ACTION_NOT_ALLOWED) + + +def test_run_parameters_cannot_smuggle_an_action() -> None: + kit = writable() + succeed(_create(kit)) + outcome = kit.call("pipeline_run", {"name": "daily", "params": {"date": ["FA_run_shell"]}}) + assert ( + "a parameter names the action FA_run_shell" + in (refused(outcome, ACTION_NOT_ALLOWED)["message"]) + ) + + +def test_a_stored_file_that_is_not_a_valid_definition_is_reported(tmp_path: Path) -> None: + directory = tmp_path / "pipelines" + directory.mkdir() + (directory / "garbage.json").write_text("{not json", encoding="utf-8") + (directory / "twice.json").write_text('{"name": "a", "name": "b"}', encoding="utf-8") + (directory / "other.json").write_text( + json.dumps({"name": "different", **_one_task(["FA_storage_schemes"])}), encoding="utf-8" + ) + (directory / "huge.json").write_text(json.dumps(_definition()) + " " * 4096, encoding="utf-8") + kit = writable(pipeline_dir=directory, max_write_bytes=1024) + assert ( + "not valid JSON" + in failed(kit.call("pipeline_run", {"name": "garbage"}), ("invalid_arguments"))["message"] + ) + failed(kit.call("pipeline_run", {"name": "twice"}), "invalid_arguments") + assert ( + "names itself 'different'" + in failed(kit.call("pipeline_run", {"name": "other"}), ("invalid_arguments"))["message"] + ) + refused(kit.call("pipeline_run", {"name": "huge"}), LIMIT_EXCEEDED) + + +def test_a_background_run_returns_at_once_and_is_found_by_pipeline_status() -> None: + kit = writable() + succeed(_create(kit)) + started = succeed(kit.call("pipeline_run", {"name": "daily", "background": True})) + assert started["background"] is True + assert started["status"] in {"running", "succeeded"} + deadline = time.monotonic() + 10 + status = "running" + while status == "running" and time.monotonic() < deadline: + report = succeed(kit.call("pipeline_status", {"run_id": started["run_id"]})) + status = report["runs"][0]["status"] + time.sleep(0.02) + assert status == "succeeded" + + +def test_a_large_task_result_is_left_out_of_what_is_returned() -> None: + File(f"{INBOX}/big.txt").write("x" * 300) + kit = writable(max_read_bytes=400) + definition = { + "schema_version": 1, + "tasks": { + "read": {"action": ["FA_storage_read_text", {"uri": f"{INBOX}/big.txt"}]}, + "list": {"action": ["FA_storage_list", {"uri": INBOX, "recursive": True}]}, + }, + } + succeed(kit.call("pipeline_create", {"name": "big", "definition": definition})) + body = succeed(kit.call("pipeline_run", {"name": "big"})) + tasks = body["run"]["tasks"] + assert tasks["read"]["result"] == "x" * 300 + assert tasks["list"]["result"] is None + assert "more than the 400" in tasks["list"]["result_omitted"] + + +def _export_definition() -> dict[str, Any]: + """The pipeline of the manual: digest, copy, verify against the digest, remove the source.""" + source, target = f"{INBOX}/${{params.date}}.csv", f"{INBOX}/sent/${{params.date}}.csv" + return { + "schema_version": 1, + "tasks": { + "digest": {"action": ["FA_storage_checksum", {"uri": source}]}, + "copy": { + "action": ["FA_storage_copy", {"source": source, "target": target}], + "depends_on": ["digest"], + }, + "verify": { + "action": [ + "FA_storage_verify", + {"uri": target, "expected": "${tasks.digest.result}", "strict": True}, + ], + "depends_on": ["copy"], + }, + "remove": {"action": ["FA_storage_delete", {"uri": source}], "depends_on": ["verify"]}, + }, + } + + +def test_the_move_and_verify_pipeline_of_the_manual_runs() -> None: + File(f"{INBOX}/2026-10-07.csv").write("id,amount\n7,70\n") + kit = writable(allow_delete=True) + succeed(kit.call("pipeline_create", {"name": "export", "definition": _export_definition()})) + body = succeed(kit.call("pipeline_run", {"name": "export", "params": {"date": "2026-10-07"}})) + assert [state["status"] for state in body["run"]["tasks"].values()] == ["succeeded"] * 4 + assert body["run"]["tasks"]["verify"]["result"] is True + assert File(f"{INBOX}/sent/2026-10-07.csv").read() == b"id,amount\n7,70\n" + assert not File(f"{INBOX}/2026-10-07.csv").exists() + + +def test_the_source_stays_when_the_copy_does_not_verify(monkeypatch: pytest.MonkeyPatch) -> None: + File(f"{INBOX}/2026-10-07.csv").write("id,amount\n7,70\n") + real = File.copy_to + + def corrupting(self: File, target, *, overwrite: bool = True) -> File: + copied = real(self, target, overwrite=overwrite) + copied.write("damaged in transit") + return copied + + monkeypatch.setattr(File, "copy_to", corrupting) + kit = writable(allow_delete=True) + succeed(kit.call("pipeline_create", {"name": "export", "definition": _export_definition()})) + outcome = kit.call("pipeline_run", {"name": "export", "params": {"date": "2026-10-07"}}) + failed(outcome, "failed") + tasks = outcome.payload["run"]["tasks"] + assert "StorageChecksumException" in tasks["verify"]["error"] + assert (tasks["remove"]["status"], tasks["remove"]["reason"]) == ("skipped", "upstream_failed") + assert File(f"{INBOX}/2026-10-07.csv").exists() + + +# ---------------------------------------------------------------------- pipeline_status + + +def test_pipeline_status_reports_one_run_or_the_latest() -> None: + kit = writable() + succeed(_create(kit)) + first = succeed(kit.call("pipeline_run", {"name": "daily", "params": {"date": "a"}})) + second = succeed(kit.call("pipeline_run", {"name": "daily", "params": {"date": "b"}})) + one = succeed(toolkit().call("pipeline_status", {"run_id": first["run_id"]})) + assert (one["count"], one["runs"][0]["run_id"]) == (1, first["run_id"]) + assert one["runs"][0]["tasks"]["copy"]["status"] == "succeeded" + latest = succeed(kit.call("pipeline_status", {"name": "daily"})) + assert [run["run_id"] for run in latest["runs"]] == [second["run_id"], first["run_id"]] + newest = succeed(kit.call("pipeline_status", {"name": "daily", "limit": 1})) + assert [run["run_id"] for run in newest["runs"]] == [second["run_id"]] + assert succeed(kit.call("pipeline_status", {"name": "other"}))["runs"] == [] + assert succeed(kit.call("pipeline_status", {}))["count"] == 2 + + +def test_pipeline_status_of_an_unknown_run_fails() -> None: + error = failed(toolkit().call("pipeline_status", {"run_id": "no-such-run"}), "not_found") + assert "no run 'no-such-run' is recorded" in error["message"] + + +def test_a_run_is_tied_to_its_call_in_the_audit_trail() -> None: + store = audited() + kit = writable() + succeed(_create(kit)) + outcome = kit.call("pipeline_run", {"name": "daily"}) + run_id = succeed(outcome)["run_id"] + call = store.search(correlation_id=outcome.correlation_id) + # The call itself read the stored definition; what the run did is under the run ID. + assert [record.action for record in call] == ["mcp.tool.completed", "read"] + assert (call[0].pipeline, call[0].metadata["run_id"]) == ("daily", run_id) + run = store.search(correlation_id=run_id) + assert {record.actor for record in run} == {"mcp"} + assert "pipeline.completed" in [record.action for record in run] + assert ("storage", "copy") in [(record.source, record.action) for record in run] + + +# ---------------------------------------------------------------------- the guarded actions + + +def _actions(**policy: Any) -> GuardedStorageActions: + return GuardedStorageActions(writable(**policy).session) + + +def test_the_guarded_actions_answer_like_the_storage_actions() -> None: + actions = _actions() + assert set(actions.commands()) == GUARDED_ACTIONS + assert actions.exists(SOURCE) is True + assert actions.stat(SOURCE)["size"] == 15 + assert [entry["path"] for entry in actions.list_dir(INBOX)] == ["in.csv"] + assert actions.checksum(SOURCE)["algorithm"] == "sha256" + digest = actions.checksum(SOURCE)["value"] + assert actions.verify(SOURCE, digest) is True + assert actions.verify(SOURCE, "00") is False + assert actions.verify(SOURCE, actions.checksum(SOURCE)) is True + assert actions.verify(SOURCE, {"algorithm": "sha256", "value": "00"}) is False + with pytest.raises(StorageChecksumException): + actions.verify(SOURCE, "00", strict=True) + with pytest.raises(StorageException, match="expected must be"): + actions.verify(SOURCE, ["not", "a", "digest"]) # type: ignore[arg-type] + assert actions.read_text(SOURCE) == "id,amount\n1,10\n" + assert "memory" in actions.schemes() + assert actions.mkdir(f"{INBOX}/made") is True + assert actions.write_text(f"{INBOX}/made/a.txt", "abc")["size"] == 3 + assert actions.copy(SOURCE, f"{INBOX}/made/b.csv")["uri"] == f"{INBOX}/made/b.csv" + assert actions.copy_tree(f"{INBOX}/made", f"{INBOX}/again")["copied"] == ["a.txt", "b.csv"] + + +def test_the_guarded_actions_refuse_what_the_policy_does_not_allow() -> None: + read_only = GuardedStorageActions(toolkit().session) + for call in ( + lambda: read_only.mkdir(f"{INBOX}/made"), + lambda: read_only.write_text(f"{INBOX}/a.txt", "x"), + lambda: read_only.copy(SOURCE, f"{INBOX}/b.csv"), + lambda: read_only.copy_tree(INBOX, f"{INBOX}-b"), + ): + with pytest.raises(MCPPermissionException) as caught: + call() + assert caught.value.code == READ_ONLY + writer = _actions() + for call in ( + lambda: writer.move(SOURCE, f"{INBOX}/moved.csv"), + lambda: writer.delete(SOURCE), + ): + with pytest.raises(MCPPermissionException) as caught: + call() + assert caught.value.code == DELETE_NOT_ALLOWED + with pytest.raises(MCPPermissionException) as caught: + writer.sync(INBOX, f"{INBOX}/mirror") + assert caught.value.code == OVERWRITE_NOT_ALLOWED + for call in ( + lambda: writer.read_text(f"{BOX}/private/a.txt"), + lambda: writer.copy(SOURCE, f"{BOX}/private/a.txt"), + lambda: writer.list_dir(BOX), + ): + with pytest.raises(MCPLocationException) as caught: + call() + assert caught.value.code == OUTSIDE_ROOT + + +def test_overwrite_defaults_to_what_the_policy_permits() -> None: + target = f"{INBOX}/target.csv" + File(target).write("old") + writer = _actions() + with pytest.raises(StorageAlreadyExistsException): + writer.copy(SOURCE, target) + with pytest.raises(MCPPermissionException) as caught: + writer.copy(SOURCE, target, overwrite=True) + assert caught.value.code == OVERWRITE_NOT_ALLOWED + with pytest.raises(MCPPermissionException): + writer.copy_tree(INBOX, f"{INBOX}/tree", overwrite=True) + assert writer.copy(SOURCE, f"{INBOX}/fresh.csv", overwrite=True)["size"] == 15 + assert File(target).read() == b"old" + replacing = _actions(allow_overwrite=True) + replacing.copy(SOURCE, target) + assert File(target).read() == File(SOURCE).read() + with pytest.raises(StorageAlreadyExistsException): + replacing.write_text(target, "x", overwrite=False) + + +def test_the_guarded_actions_keep_the_size_limits() -> None: + actions = _actions(max_read_bytes=8, max_write_bytes=8) + with pytest.raises(MCPPermissionException) as caught: + actions.read_text(SOURCE) + assert caught.value.code == LIMIT_EXCEEDED + with pytest.raises(MCPPermissionException) as caught: + actions.write_text(f"{INBOX}/a.txt", "123456789") + assert caught.value.code == LIMIT_EXCEEDED + + +def test_delete_move_and_sync_work_with_their_permissions() -> None: + actions = _actions(allow_overwrite=True, allow_delete=True) + assert actions.move(SOURCE, f"{INBOX}/moved.csv")["uri"] == f"{INBOX}/moved.csv" + assert not File(SOURCE).exists() + mirrored = actions.sync(INBOX, f"{BOX}/in/mirror", delete=True, dry_run=True) + assert mirrored["dry_run"] is True + assert actions.delete(f"{INBOX}/moved.csv") is True + with pytest.raises(MCPLocationException, match="allowed location itself"): + actions.delete(INBOX, recursive=True) + assert Storage(INBOX).exists() + without_delete = _actions(allow_overwrite=True) + with pytest.raises(MCPPermissionException) as caught: + without_delete.sync(INBOX, f"{INBOX}/mirror", delete=True) + assert caught.value.code == DELETE_NOT_ALLOWED + + +# ---------------------------------------------------------------------- reporting tools + + +def test_integrity_status_reports_the_named_monitors() -> None: + kit = toolkit() + empty = succeed(kit.call("integrity_status", {})) + assert (empty["monitors"], empty["count"]) == ([], 0) + baseline = f"{BOX}/baselines/in.json" + integrity_actions.integrity_baseline(INBOX, baseline) + integrity_actions.integrity_watch_start("inbox", INBOX, baseline, interval=3600) + body = succeed(kit.call("integrity_status", {})) + assert body["count"] == 1 + monitor = body["monitors"][0] + assert (monitor["name"], monitor["target"], monitor["running"]) == ("inbox", INBOX, True) + assert succeed(kit.call("integrity_status", {"name": "inbox"}))["count"] == 1 + error = failed(kit.call("integrity_status", {"name": "absent"}), "not_found") + assert "no integrity monitor named 'absent'" in error["message"] + + +def test_integrity_status_caps_the_changes_of_the_last_report() -> None: + baseline = f"{BOX}/baselines/in.json" + integrity_actions.integrity_baseline(INBOX, baseline) + for index in range(5): + File(f"{INBOX}/new-{index}.txt").write("new") + integrity_actions.integrity_watch_start("inbox", INBOX, baseline, interval=0.05) + deadline = time.monotonic() + 20 + report = None + while report is None and time.monotonic() < deadline: + time.sleep(0.05) + report = integrity_actions.integrity_status("inbox")[0]["last_report"] + assert report is not None, "the monitor did not complete a pass" + body = succeed(toolkit(max_results=2).call("integrity_status", {})) + trimmed = body["monitors"][0]["last_report"] + assert len(trimmed["changes"]) == 2 + assert trimmed["changes_truncated"] is True + assert trimmed["counts"]["created"] == 5 + + +def test_audit_search_says_when_no_trail_is_kept() -> None: + error = failed(toolkit().call("audit_search", {}), "not_configured") + assert "configure_audit" in error["message"] + + +def test_audit_search_filters_and_caps_the_records() -> None: + audited() + with correlation_scope("run-1"): + for index in range(6): + emit(PipelineStarted(source="pipeline", subject=f"started {index}")) + kit = toolkit(max_results=4) + body = succeed(kit.call("audit_search", {"correlation_id": "run-1", "limit": 100})) + assert (body["count"], body["total"], body["limit"], body["truncated"]) == (4, 6, 4, True) + assert body["records"][0]["metadata"]["subject"] == "started 5" + page = succeed(kit.call("audit_search", {"correlation_id": "run-1", "offset": 4})) + assert (page["count"], page["offset"], page["truncated"]) == (2, 4, False) + assert ( + succeed(kit.call("audit_search", {"action": "pipeline.started", "limit": 1}))["count"] == 1 + ) + assert succeed(kit.call("audit_search", {"text": "started 3"}))["total"] == 1 + assert succeed(kit.call("audit_search", {"since": "2100-01-01T00:00:00+00:00"}))["total"] == 0 + failed(kit.call("audit_search", {"since": "yesterday"}), "failed") + failed(kit.call("audit_search", {"limit": 0}), "invalid_arguments") + failed(kit.call("audit_search", {"unknown_filter": "x"}), "invalid_arguments") + + +def test_audit_search_finds_what_a_call_did_by_its_correlation_id() -> None: + audited() + kit = writable() + written = kit.call("file_write", {"uri": f"{INBOX}/a.txt", "content": "abc"}) + body = succeed(kit.call("audit_search", {"correlation_id": written.correlation_id})) + # Two records of one call can carry the same timestamp, so their order is not fixed. + assert sorted((record["source"], record["action"]) for record in body["records"]) == [ + ("mcp", "mcp.tool.completed"), + ("storage", "upload"), + ] + assert {record["actor"] for record in body["records"]} == {"mcp"} + assert "abc" not in json.dumps(body["records"]) diff --git a/tests/test_mcp_policy.py b/tests/test_mcp_policy.py new file mode 100644 index 0000000..2012dc2 --- /dev/null +++ b/tests/test_mcp_policy.py @@ -0,0 +1,495 @@ +"""The permission model of the semantic MCP tools: ``MCPPolicy`` and ``StorageGuard``.""" + +from __future__ import annotations + +import dataclasses +import os +from collections.abc import Iterator +from pathlib import Path + +import pytest + +from automation_file.exceptions import ( + MCPServerException, + StoragePermissionException, + StorageURIException, +) +from automation_file.server.mcp_policy import ( + DEFAULT_MAX_READ_BYTES, + DEFAULT_MAX_RESULTS, + DELETE_NOT_ALLOWED, + NO_ROOT, + OUTSIDE_ROOT, + OVERWRITE_NOT_ALLOWED, + READ_ONLY, + SEMANTIC_TOOL_NAMES, + TOOL_DISABLED, + MCPLocationException, + MCPPermissionException, + MCPPolicy, + StorageGuard, + names_windows_device, + path_below, +) +from automation_file.storage import ( + File, + LocalStorage, + MemoryStorage, + StorageResolver, + clear_memory_stores, + parse_storage_uri, +) +from tests.mcp_support import link_directory, remove_link + +WINDOWS = os.sep == "\\" + + +@pytest.fixture(autouse=True) +def _memory() -> Iterator[None]: + clear_memory_stores() + yield + clear_memory_stores() + + +@pytest.fixture +def links() -> Iterator[list[Path]]: + """Collects the links a test made, so they are removed before the temporary tree is.""" + made: list[Path] = [] + yield made + for link in made: + remove_link(link) + + +def _uri(text: str): + return parse_storage_uri(text) + + +def _refusal(policy: MCPPolicy, text: str) -> MCPLocationException: + with pytest.raises(MCPLocationException) as caught: + policy.candidates(_uri(text)) + return caught.value + + +# ---------------------------------------------------------------------- the policy object + + +def test_the_default_policy_allows_no_location_and_no_change() -> None: + policy = MCPPolicy() + assert policy.roots == () + assert (policy.allow_write, policy.allow_overwrite, policy.allow_delete) == (False,) * 3 + assert policy.max_read_bytes == DEFAULT_MAX_READ_BYTES + assert policy.max_results == DEFAULT_MAX_RESULTS + assert policy.pipeline_dir is None + assert policy.pipeline_actions is None + assert policy.enabled_tools() == SEMANTIC_TOOL_NAMES + assert policy.actor == "mcp" + + +def test_there_are_fourteen_tools_with_the_roadmap_names() -> None: + assert SEMANTIC_TOOL_NAMES == ( + "file_read", + "file_write", + "file_copy", + "file_move", + "file_search", + "file_checksum", + "file_verify", + "storage_list", + "storage_copy", + "pipeline_create", + "pipeline_run", + "pipeline_status", + "integrity_status", + "audit_search", + ) + + +def test_roots_are_parsed_into_storage_uris(tmp_path: Path) -> None: + policy = MCPPolicy( + roots=[tmp_path, "s3://bucket/team/", "file:///srv/data", "s3://bucket/team"] + ) + schemes = [root.scheme for root in policy.root_uris] + assert schemes == ["local", "s3", "local"] + assert str(policy.root_uris[1]) == "s3://bucket/team" + assert policy.root_uris[0] == parse_storage_uri(tmp_path) + + +def test_one_root_may_be_given_without_a_list() -> None: + assert [str(root) for root in MCPPolicy(roots="memory://box/in").root_uris] == [ + "memory://box/in" + ] + + +def test_a_policy_cannot_be_changed() -> None: + policy = MCPPolicy(roots=["memory://box"]) + with pytest.raises(dataclasses.FrozenInstanceError): + policy.allow_write = True # type: ignore[misc] + assert isinstance(policy.roots, tuple) + assert hash(policy) == hash(MCPPolicy(roots=["memory://box"])) + + +def test_a_root_with_credentials_is_refused_without_repeating_them() -> None: + with pytest.raises(MCPServerException) as caught: + MCPPolicy(roots=["sftp://alice:hunter2@nas/data"]) + assert "hunter2" not in str(caught.value) + assert "alice" not in str(caught.value) + + +def test_overwrite_and_delete_need_write() -> None: + with pytest.raises(MCPServerException, match="need allow_write"): + MCPPolicy(allow_overwrite=True) + with pytest.raises(MCPServerException, match="need allow_write"): + MCPPolicy(allow_delete=True) + assert MCPPolicy(allow_write=True, allow_overwrite=True, allow_delete=True).allow_delete + + +@pytest.mark.parametrize("value", [0, -1, True, "5", 1.5]) +@pytest.mark.parametrize( + "limit", ["max_read_bytes", "max_write_bytes", "max_results", "max_search_bytes"] +) +def test_a_limit_is_a_positive_integer(limit: str, value: object) -> None: + with pytest.raises(MCPServerException, match=limit): + MCPPolicy(**{limit: value}) + + +def test_the_tool_list_takes_only_known_names_and_keeps_catalogue_order() -> None: + policy = MCPPolicy(tools=["storage_list", "file_read"]) + assert policy.enabled_tools() == ("file_read", "storage_list") + assert MCPPolicy(tools=[]).enabled_tools() == () + with pytest.raises(MCPServerException, match="file_delete"): + MCPPolicy(tools=["file_read", "file_delete"]) + + +def test_a_disabled_tool_is_refused_by_name() -> None: + policy = MCPPolicy(tools=["file_read"]) + policy.require_tool("file_read") + with pytest.raises(MCPPermissionException) as caught: + policy.require_tool("file_write") + assert caught.value.code == TOOL_DISABLED + + +def test_pipeline_actions_become_a_frozen_set_of_names() -> None: + policy = MCPPolicy(pipeline_actions=["FA_storage_copy", " FA_notify_send "]) + assert policy.pipeline_actions == frozenset({"FA_storage_copy", "FA_notify_send"}) + with pytest.raises(MCPServerException, match="pipeline_actions"): + MCPPolicy(pipeline_actions=["FA_storage_copy", 3]) + + +def test_an_empty_actor_is_refused() -> None: + with pytest.raises(MCPServerException, match="actor"): + MCPPolicy(actor=" ") + + +def test_the_permission_checks_name_the_flag_that_would_allow_the_call() -> None: + read_only = MCPPolicy() + with pytest.raises(MCPPermissionException, match="--allow-write") as caught: + read_only.require_write("file_write") + assert caught.value.code == READ_ONLY + writer = MCPPolicy(allow_write=True) + writer.require_write("file_write") + with pytest.raises(MCPPermissionException, match="--allow-overwrite") as caught: + writer.require_overwrite("file_write") + assert caught.value.code == OVERWRITE_NOT_ALLOWED + with pytest.raises(MCPPermissionException, match="--allow-delete") as caught: + writer.require_delete("file_move") + assert caught.value.code == DELETE_NOT_ALLOWED + with pytest.raises(MCPPermissionException) as caught: + read_only.require_delete("file_move") + assert caught.value.code == READ_ONLY + + +def test_results_are_clamped_to_the_policy() -> None: + policy = MCPPolicy(max_results=10) + assert policy.clamp_results(None) == 10 + assert policy.clamp_results(3) == 3 + assert policy.clamp_results(500) == 10 + + +def test_describe_and_summary_say_what_is_allowed() -> None: + policy = MCPPolicy(roots=["memory://box/in"], allow_write=True, tools=["file_read"]) + described = policy.describe() + assert described["roots"] == ["memory://box/in"] + assert described["allow_write"] is True + assert described["tools"] == ["file_read"] + assert described["pipeline_actions"] is None + summary = policy.summary() + assert "memory://box/in" in summary + assert "Allowed: reading, writing." in summary + assert "Refused: replacing existing files, deleting" in summary + assert "No location is allowed yet." in MCPPolicy().summary() + + +# ---------------------------------------------------------------------- which locations + + +def test_without_a_root_every_location_is_refused_with_how_to_add_one() -> None: + error = _refusal(MCPPolicy(), "memory://box/in/a.txt") + assert error.code == NO_ROOT + assert "--root" in str(error) + assert "MCPPolicy(roots=" in str(error) + + +def test_a_location_at_or_below_a_root_is_allowed() -> None: + policy = MCPPolicy(roots=["s3://bucket/team"]) + assert policy.candidates(_uri("s3://bucket/team")) == [(_uri("s3://bucket/team"), "")] + assert policy.candidates(_uri("s3://bucket/team/2026/a.csv"))[0][1] == "2026/a.csv" + + +@pytest.mark.parametrize( + "location", + [ + "s3://bucket/team-b/a.csv", + "s3://bucket/tea", + "s3://bucket/a.csv", + "s3://bucket", + "s3://bucket-2/team/a.csv", + "azure://bucket/team/a.csv", + "memory://bucket/team/a.csv", + ], +) +def test_a_sibling_prefix_another_authority_or_another_scheme_is_outside(location: str) -> None: + error = _refusal(MCPPolicy(roots=["s3://bucket/team"]), location) + assert error.code == OUTSIDE_ROOT + assert "s3://bucket/team" in str(error) + + +def test_the_authority_is_compared_exactly() -> None: + # memory://Box and memory://box are two different stores. + assert _refusal(MCPPolicy(roots=["memory://box/in"]), "memory://Box/in/a.txt").code == ( + OUTSIDE_ROOT + ) + + +def test_nested_roots_are_tried_outermost_first() -> None: + policy = MCPPolicy(roots=["s3://bucket/team/public", "s3://bucket/team"]) + found = policy.candidates(_uri("s3://bucket/team/public/a.csv")) + assert [str(root) for root, _ in found] == ["s3://bucket/team", "s3://bucket/team/public"] + assert [relative for _, relative in found] == ["public/a.csv", "a.csv"] + + +def test_is_root_recognises_only_the_roots_themselves() -> None: + policy = MCPPolicy(roots=["memory://box/in"]) + assert policy.is_root(_uri("memory://box/in")) is True + assert policy.is_root(_uri("memory://box/in/a")) is False + assert policy.is_root(_uri("memory://box")) is False + + +def test_path_below_compares_whole_segments() -> None: + assert path_below(_uri("s3://b/team"), _uri("s3://b/team/x/y")) == "x/y" + assert path_below(_uri("s3://b/team"), _uri("s3://b/team")) == "" + assert path_below(_uri("s3://b/team"), _uri("s3://b/team-b/x")) is None + assert path_below(_uri("s3://b"), _uri("s3://b/anything")) == "anything" + + +def test_a_location_refusal_is_a_permission_error_of_both_kinds() -> None: + error = _refusal(MCPPolicy(roots=["memory://box"]), "memory://other/a") + assert isinstance(error, MCPPermissionException) + assert isinstance(error, StoragePermissionException) + assert isinstance(error, MCPServerException) + + +# ---------------------------------------------------------------------- the guard + + +def test_the_guard_serves_a_memory_root_and_refuses_its_neighbour() -> None: + guard = StorageGuard(MCPPolicy(roots=["memory://box/in"])) + File("memory://box/in/a.txt").write("inside") + File("memory://box/in-b/a.txt").write("neighbour") + assert File("memory://box/in/a.txt", resolver=guard).read() == b"inside" + with pytest.raises(MCPLocationException) as caught: + File("memory://box/in-b/a.txt", resolver=guard).read() + assert caught.value.code == OUTSIDE_ROOT + + +def test_dot_dot_never_reaches_the_guard() -> None: + guard = StorageGuard(MCPPolicy(roots=["memory://box/in"])) + with pytest.raises(StorageURIException, match=r"\.\."): + File("memory://box/in/../secret.txt", resolver=guard) + + +def test_a_local_root_is_served_by_a_backend_confined_to_it(tmp_path: Path) -> None: + root = tmp_path / "root" + root.mkdir() + (root / "a.txt").write_text("inside", encoding="utf-8") + guard = StorageGuard(MCPPolicy(roots=[root])) + backend, path = guard.resolve(root / "a.txt") + assert isinstance(backend, LocalStorage) + assert backend.root == root.resolve() + assert path == "a.txt" + assert File(root / "a.txt", resolver=guard).read() == b"inside" + assert guard.resolve(root)[1] == "" + + +def test_a_local_file_next_to_the_root_is_outside(tmp_path: Path) -> None: + root = tmp_path / "root" + root.mkdir() + (tmp_path / "root-b").mkdir() + (tmp_path / "root-b" / "a.txt").write_text("neighbour", encoding="utf-8") + guard = StorageGuard(MCPPolicy(roots=[root])) + for outside in (tmp_path / "root-b" / "a.txt", tmp_path / "a.txt", tmp_path): + with pytest.raises(MCPLocationException) as caught: + guard.resolve(outside) + assert caught.value.code == OUTSIDE_ROOT + + +def test_a_link_out_of_a_local_root_is_refused(tmp_path: Path, links: list[Path]) -> None: + root, outside = tmp_path / "root", tmp_path / "outside" + root.mkdir() + outside.mkdir() + (outside / "secret.txt").write_text("secret", encoding="utf-8") + link_directory(root / "escape", outside) + links.append(root / "escape") + guard = StorageGuard(MCPPolicy(roots=[root])) + with pytest.raises(MCPLocationException) as caught: + File(root / "escape" / "secret.txt", resolver=guard).read() + assert caught.value.code == OUTSIDE_ROOT + assert "link" in str(caught.value) + with pytest.raises(MCPLocationException): + File(root / "escape" / "new.txt", resolver=guard).write("x") + assert not (outside / "new.txt").exists() + + +def test_a_link_that_stays_inside_a_local_root_is_followed( + tmp_path: Path, links: list[Path] +) -> None: + root = tmp_path / "root" + (root / "real").mkdir(parents=True) + (root / "real" / "a.txt").write_text("inside", encoding="utf-8") + link_directory(root / "alias", root / "real") + links.append(root / "alias") + guard = StorageGuard(MCPPolicy(roots=[root])) + assert File(root / "alias" / "a.txt", resolver=guard).read() == b"inside" + + +def test_a_link_between_two_roots_is_allowed_by_the_outer_one( + tmp_path: Path, links: list[Path] +) -> None: + outer = tmp_path / "outer" + (outer / "public").mkdir(parents=True) + (outer / "private").mkdir() + (outer / "private" / "a.txt").write_text("kept", encoding="utf-8") + link_directory(outer / "public" / "to-private", outer / "private") + links.append(outer / "public" / "to-private") + linked = outer / "public" / "to-private" / "a.txt" + only_public = StorageGuard(MCPPolicy(roots=[outer / "public"])) + with pytest.raises(MCPLocationException): + File(linked, resolver=only_public).read() + both = StorageGuard(MCPPolicy(roots=[outer / "public", outer])) + assert File(linked, resolver=both).read() == b"kept" + + +@pytest.mark.skipif(not WINDOWS, reason="drive letters and case folding are Windows behaviour") +def test_windows_paths_are_matched_without_regard_to_case_and_slashes(tmp_path: Path) -> None: + root = tmp_path / "Root" + root.mkdir() + (root / "a.txt").write_text("inside", encoding="utf-8") + guard = StorageGuard(MCPPolicy(roots=[root])) + shouted = f"local:///{str(root).upper()}/a.txt".replace("\\", "/") + assert File(shouted, resolver=guard).read() == b"inside" + backslashed = f"local:///{root}\\a.txt" + assert File(backslashed, resolver=guard).read() == b"inside" + with pytest.raises((MCPLocationException, StorageURIException)): + File(f"local:///{root}\\..\\outside.txt", resolver=guard).exists() + + +@pytest.mark.skipif(not WINDOWS, reason="an absolute path with a drive letter") +def test_an_absolute_path_smuggled_below_a_root_is_refused(tmp_path: Path) -> None: + root = tmp_path / "root" + root.mkdir() + guard = StorageGuard(MCPPolicy(roots=[root])) + smuggled = f"local:///{root.as_posix()}/C:/Windows/win.ini" + with pytest.raises(MCPLocationException): + File(smuggled, resolver=guard).exists() + + +@pytest.mark.parametrize( + ("relative", "expected"), + [ + ("reports/CON", True), + ("reports/nul.txt", True), + ("COM1", True), + ("a/lpt9.log/b.txt", True), + ("a/aux ", True), + ("a.txt:hidden", True), + ("C:/Windows/win.ini", True), + ("reports/a.csv", False), + ("console/nullable.txt", False), + ("com10", False), + ("", False), + ], +) +def test_windows_device_and_stream_names_are_recognised(relative: str, expected: bool) -> None: + assert names_windows_device(relative) is expected + + +@pytest.mark.skipif(not WINDOWS, reason="device names and data streams are Windows behaviour") +def test_a_windows_device_or_data_stream_below_a_root_is_refused(tmp_path: Path) -> None: + root = tmp_path / "root" + root.mkdir() + (root / "a.txt").write_text("inside", encoding="utf-8") + guard = StorageGuard(MCPPolicy(roots=[root])) + base = f"local:///{root.as_posix()}" + for name in ("NUL", "sub/CON", "com1.txt", "a.txt::$DATA", "b.txt:stream"): + with pytest.raises(MCPLocationException, match="Windows device or a data stream") as caught: + guard.resolve(f"{base}/{name}") + assert caught.value.code == OUTSIDE_ROOT + assert File(f"{base}/a.txt", resolver=guard).read() == b"inside" + + +def test_other_backends_are_resolved_by_the_inner_resolver() -> None: + inner = StorageResolver(defaults=False) + mounted = MemoryStorage("mounted") + inner.mount("vault://jobs", mounted) + guard = StorageGuard(MCPPolicy(roots=["vault://jobs/out"]), inner) + assert guard.inner is inner + assert guard.resolve("vault://jobs/out/a.txt") == (mounted, "out/a.txt") + with pytest.raises(MCPLocationException): + guard.resolve("vault://jobs/private/a.txt") + + +def test_a_root_below_a_mounted_local_backend_is_confined_to_the_root( + tmp_path: Path, links: list[Path] +) -> None: + jobs = tmp_path / "jobs" + (jobs / "out").mkdir(parents=True) + (jobs / "private").mkdir() + (jobs / "private" / "a.txt").write_text("private", encoding="utf-8") + link_directory(jobs / "out" / "sideways", jobs / "private") + links.append(jobs / "out" / "sideways") + inner = StorageResolver(defaults=False) + inner.mount("sandbox://jobs", LocalStorage(jobs)) + guard = StorageGuard(MCPPolicy(roots=["sandbox://jobs/out"]), inner) + backend, path = guard.resolve("sandbox://jobs/out/new.txt") + assert isinstance(backend, LocalStorage) + assert backend.root == (jobs / "out").resolve() + assert path == "new.txt" + # The mount itself would allow the link: it stays inside jobs/. The root does not. + with pytest.raises(MCPLocationException): + guard.resolve("sandbox://jobs/out/sideways/a.txt") + + +def test_a_root_that_is_a_mount_point_uses_the_mounted_backend( + tmp_path: Path, links: list[Path] +) -> None: + jobs, outside = tmp_path / "jobs", tmp_path / "outside" + jobs.mkdir() + outside.mkdir() + link_directory(jobs / "escape", outside) + links.append(jobs / "escape") + inner = StorageResolver(defaults=False) + mounted = LocalStorage(jobs) + inner.mount("sandbox://jobs", mounted) + guard = StorageGuard(MCPPolicy(roots=["sandbox://jobs"]), inner) + assert guard.resolve("sandbox://jobs/a/b.txt") == (mounted, "a/b.txt") + with pytest.raises(MCPLocationException) as caught: + guard.resolve("sandbox://jobs/escape/secret.txt") + assert caught.value.code == OUTSIDE_ROOT + + +def test_a_root_that_is_the_whole_filesystem_is_not_confined(tmp_path: Path) -> None: + (tmp_path / "a.txt").write_text("anywhere", encoding="utf-8") + guard = StorageGuard(MCPPolicy(roots=["local:///"])) + backend, _path = guard.resolve(tmp_path / "a.txt") + assert isinstance(backend, LocalStorage) + assert backend.root is None + assert File(tmp_path / "a.txt", resolver=guard).read() == b"anywhere" diff --git a/tests/test_mcp_semantic_server.py b/tests/test_mcp_semantic_server.py new file mode 100644 index 0000000..2919c7c --- /dev/null +++ b/tests/test_mcp_semantic_server.py @@ -0,0 +1,433 @@ +"""The semantic tools over JSON-RPC, next to the ``FA_*`` bridge, and the server's flags.""" + +from __future__ import annotations + +import argparse +import io +import json +from collections.abc import Iterator +from pathlib import Path +from typing import Any + +import pytest + +from automation_file.core.action_registry import ActionRegistry +from automation_file.exceptions import MCPServerException +from automation_file.server.mcp_policy import ( + DEFAULT_MAX_READ_BYTES, + SEMANTIC_TOOL_NAMES, + MCPPolicy, +) +from automation_file.server.mcp_server import ( + MCPServer, + _build_cli_parser, + _cli, + _server_options, + add_semantic_arguments, + semantic_argv, +) +from automation_file.storage import File, parse_storage_uri +from tests.mcp_support import INBOX, clean_state + +A = f"{INBOX}/a.txt" + + +@pytest.fixture(autouse=True) +def _state() -> Iterator[None]: + with clean_state(): + File(A).write("alpha") + yield + + +def _registry() -> ActionRegistry: + def echo(message: str, repeat: int = 1) -> str: + """Echo ``message`` back, optionally repeated.""" + return message * repeat + + def add(a: int, b: int) -> int: + return a + b + + return ActionRegistry({"echo": echo, "add": add}) + + +def _server(*, bridge: bool = True, **policy: Any) -> MCPServer: + policy.setdefault("roots", (INBOX,)) + return MCPServer(_registry(), policy=MCPPolicy(**policy), bridge=bridge) + + +def _request(server: MCPServer, method: str, params: dict | None = None) -> dict: + message: dict[str, Any] = {"jsonrpc": "2.0", "id": 7, "method": method} + if params is not None: + message["params"] = params + response = server.handle_message(message) + assert response is not None + assert response["id"] == 7 + return response + + +def _call(server: MCPServer, name: str, arguments: dict | None = None) -> dict: + params: dict[str, Any] = {"name": name} + if arguments is not None: + params["arguments"] = arguments + return _request(server, "tools/call", params) + + +def _payload(response: dict) -> dict: + """Return the JSON document a semantic tool answered with.""" + result = response["result"] + assert [block["type"] for block in result["content"]] == ["text"] + return json.loads(result["content"][0]["text"]) + + +def _names(server: MCPServer) -> list[str]: + return [tool["name"] for tool in _request(server, "tools/list")["result"]["tools"]] + + +# ---------------------------------------------------------------------- tools/list + + +def test_tools_list_returns_the_semantic_tools_first_and_then_the_bridge() -> None: + assert _names(_server()) == [*SEMANTIC_TOOL_NAMES, "add", "echo"] + + +def test_a_server_without_a_policy_still_lists_both_sets() -> None: + names = _names(MCPServer(_registry())) + assert names[:14] == list(SEMANTIC_TOOL_NAMES) + assert names[14:] == ["add", "echo"] + + +def test_the_default_server_lists_the_semantic_tools_before_every_registered_action() -> None: + names = _names(MCPServer()) + assert names[:14] == list(SEMANTIC_TOOL_NAMES) + assert "FA_storage_copy" in names[14:] + assert names[14:] == sorted(names[14:]) + assert len(names) == len(set(names)) + + +def test_the_listed_schemas_are_hand_written_and_closed() -> None: + tools = {tool["name"]: tool for tool in _request(_server(), "tools/list")["result"]["tools"]} + for name in SEMANTIC_TOOL_NAMES: + schema = tools[name]["inputSchema"] + assert schema["type"] == "object" + assert schema["additionalProperties"] is False + assert tools[name]["description"] + assert all("description" in spec for spec in schema["properties"].values()) + copy = tools["file_copy"]["inputSchema"] + assert copy["required"] == ["source", "target"] + assert set(copy["properties"]) == {"source", "target", "overwrite", "verify", "dry_run"} + assert copy["properties"]["dry_run"] == { + "type": "boolean", + "description": "When true, change nothing and report what the call would do.", + "default": False, + } + assert tools["file_read"]["inputSchema"]["properties"]["offset"]["minimum"] == 0 + assert tools["pipeline_create"]["inputSchema"]["properties"]["definition"]["type"] == "object" + assert "required" not in tools["audit_search"]["inputSchema"] + # The bridge keeps the schema it derives from a signature. + assert tools["echo"]["inputSchema"]["additionalProperties"] is True + assert tools["echo"]["inputSchema"]["required"] == ["message"] + + +def test_without_the_bridge_only_the_semantic_tools_are_listed() -> None: + assert _names(_server(bridge=False)) == list(SEMANTIC_TOOL_NAMES) + + +def test_the_tool_list_of_the_policy_narrows_what_is_listed() -> None: + server = _server(tools=["storage_list", "file_read"]) + assert _names(server) == ["file_read", "storage_list", "add", "echo"] + assert _names(_server(tools=[], bridge=False)) == [] + + +def test_a_registered_action_with_a_semantic_name_is_not_listed_twice() -> None: + registry = _registry() + registry.register("file_read", lambda uri: "from the registry") + server = MCPServer(registry, policy=MCPPolicy(roots=[INBOX])) + assert _names(server).count("file_read") == 1 + assert _payload(_call(server, "file_read", {"uri": A}))["content"] == "alpha" + + +# ---------------------------------------------------------------------- tools/call + + +def test_a_semantic_call_answers_with_a_json_document() -> None: + response = _call(_server(), "file_read", {"uri": A}) + assert response["result"]["isError"] is False + payload = _payload(response) + assert payload["tool"] == "file_read" + assert payload["content"] == "alpha" + assert len(payload["correlation_id"]) == 32 + + +def test_a_refused_call_is_a_tool_error_with_a_reason_not_a_protocol_error() -> None: + response = _call(_server(), "file_write", {"uri": f"{INBOX}/b.txt", "content": "x"}) + assert "error" not in response + assert response["result"]["isError"] is True + payload = _payload(response) + assert payload["error"]["type"] == "permission_denied" + assert payload["error"]["code"] == "read_only" + assert "--allow-write" in payload["error"]["message"] + assert len(payload["correlation_id"]) == 32 + assert not File(f"{INBOX}/b.txt").exists() + + +def test_a_failed_call_and_wrong_arguments_are_tool_errors_too() -> None: + server = _server() + missing = _payload(_call(server, "file_read", {"uri": f"{INBOX}/absent.txt"})) + assert missing["error"]["type"] == "not_found" + wrong = _call(server, "file_read", {"uri": A, "offset": "three"}) + assert wrong["result"]["isError"] is True + assert _payload(wrong)["error"]["type"] == "invalid_arguments" + none = _call(server, "file_read") + assert "'uri' is required" in _payload(none)["error"]["message"] + + +def test_a_location_outside_the_roots_is_refused_over_json_rpc() -> None: + File("memory://box/in-b/a.txt").write("neighbour") + payload = _payload(_call(_server(), "file_read", {"uri": "memory://box/in-b/a.txt"})) + assert payload["error"]["code"] == "outside_root" + assert "neighbour" not in json.dumps(payload) + + +def test_a_server_without_a_root_says_how_to_configure_one() -> None: + payload = _payload(_call(MCPServer(_registry()), "storage_list", {"uri": INBOX})) + assert payload["error"]["code"] == "no_root" + assert "--root" in payload["error"]["message"] + + +def test_a_disabled_semantic_tool_is_refused_and_never_falls_through_to_the_bridge() -> None: + registry = _registry() + registry.register("file_read", lambda uri: "from the registry") + server = MCPServer(registry, policy=MCPPolicy(roots=[INBOX], tools=["storage_list"])) + payload = _payload(_call(server, "file_read", {"uri": A})) + assert payload["error"]["code"] == "tool_disabled" + + +def test_arguments_that_are_not_an_object_are_a_protocol_error() -> None: + response = _call(_server(), "file_read", ["memory://box/in/a.txt"]) # type: ignore[arg-type] + assert response["error"]["code"] == -32602 + assert "'arguments' must be an object" in response["error"]["message"] + + +def test_the_bridge_still_dispatches_registered_actions() -> None: + server = _server() + response = _call(server, "add", {"a": 2, "b": 5}) + assert response["result"] == {"content": [{"type": "text", "text": "7"}], "isError": False} + assert _call(server, "add", {"a": 1})["error"]["code"] == -32602 + assert "unknown tool" in _call(server, "nope", {})["error"]["message"] + + +def test_without_the_bridge_a_registered_action_is_an_unknown_tool() -> None: + server = _server(bridge=False) + response = _call(server, "add", {"a": 2, "b": 5}) + assert response["error"] == {"code": -32602, "message": "unknown tool: add"} + assert _payload(_call(server, "file_read", {"uri": A}))["content"] == "alpha" + + +def test_a_whole_exchange_over_stdio() -> None: + server = _server(allow_write=True) + target = f"{INBOX}/b.txt" + frames = [ + {"jsonrpc": "2.0", "id": 1, "method": "initialize", "params": {}}, + {"jsonrpc": "2.0", "method": "notifications/initialized"}, + { + "jsonrpc": "2.0", + "id": 2, + "method": "tools/call", + "params": {"name": "file_copy", "arguments": {"source": A, "target": target}}, + }, + { + "jsonrpc": "2.0", + "id": 3, + "method": "tools/call", + "params": {"name": "echo", "arguments": {"message": "hi"}}, + }, + ] + stdout = io.StringIO() + server.serve_stdio(io.StringIO("".join(json.dumps(frame) + "\n" for frame in frames)), stdout) + replies = [json.loads(line) for line in stdout.getvalue().splitlines()] + assert [reply["id"] for reply in replies] == [1, 2, 3] + assert _payload(replies[1])["done"] is True + assert replies[2]["result"]["content"][0]["text"] == '"hi"' + assert File(target).read() == b"alpha" + + +# ---------------------------------------------------------------------- the handshake + + +def test_initialize_describes_the_policy_in_its_instructions() -> None: + result = _request(_server(), "initialize", {})["result"] + assert result["serverInfo"] == {"name": "automation_file", "version": "1.0.0"} + assert INBOX in result["instructions"] + assert "Refused: writing" in result["instructions"] + assert "instructions" not in _request(_server(tools=[]), "initialize", {})["result"] + + +def test_the_client_name_of_the_handshake_labels_the_actor() -> None: + server = _server() + assert server.toolkit.actor == "mcp" + _request(server, "initialize", {"clientInfo": {"name": "claude-desktop", "version": "1"}}) + assert server.toolkit.actor == "mcp:claude-desktop" + _request(server, "initialize", {"clientInfo": "not an object"}) + assert server.toolkit.actor == "mcp" + + +# ---------------------------------------------------------------------- pipelines and the registry + + +def test_a_pipeline_action_the_server_does_not_expose_is_refused_at_start() -> None: + with pytest.raises(MCPServerException, match="FA_run_shell"): + MCPServer(_registry(), policy=MCPPolicy(pipeline_actions=["FA_run_shell"])) + server = MCPServer(_registry(), policy=MCPPolicy(pipeline_actions=["echo"])) + assert server.toolkit.policy.pipeline_actions == frozenset({"echo"}) + + +def test_a_pipeline_runs_over_json_rpc_with_a_listed_action() -> None: + server = MCPServer( + _registry(), + policy=MCPPolicy( + roots=[INBOX], allow_write=True, pipeline_actions=["FA_storage_copy", "echo"] + ), + bridge=False, + ) + definition = { + "schema_version": 1, + "tasks": { + "copy": {"action": ["FA_storage_copy", {"source": A, "target": f"{INBOX}/c.txt"}]}, + "say": {"action": ["echo", {"message": "done"}], "depends_on": ["copy"]}, + }, + } + created = _payload(_call(server, "pipeline_create", {"name": "job", "definition": definition})) + assert created["stored"] is True + run = _payload(_call(server, "pipeline_run", {"name": "job"})) + assert run["status"] == "succeeded" + assert run["run"]["tasks"]["say"]["result"] == "done" + status = _payload(_call(server, "pipeline_status", {"run_id": run["run_id"]})) + assert status["runs"][0]["status"] == "succeeded" + assert File(f"{INBOX}/c.txt").read() == b"alpha" + + +# ---------------------------------------------------------------------- the command line + + +def _options(*argv: str) -> dict[str, Any]: + return _server_options(_build_cli_parser().parse_args(list(argv))) + + +def test_without_a_semantic_flag_the_server_gets_its_defaults() -> None: + assert _options() == {} + assert _options("--name", "x", "--allowed-actions", "echo") == {} + + +def test_the_flags_build_the_policy(tmp_path: Path) -> None: + options = _options( + "--root", + str(tmp_path), + "--root", + "s3://bucket/team", + "--allow-write", + "--allow-overwrite", + "--allow-delete", + "--max-read-bytes", + "1000", + "--max-write-bytes", + "2000", + "--max-results", + "30", + "--max-search-bytes", + "4000", + "--pipeline-dir", + str(tmp_path / "pipelines"), + "--pipeline-actions", + "FA_storage_copy, FA_storage_verify,", + "--tools", + "file_read,storage_list", + "--no-bridge", + ) + assert options["bridge"] is False + policy = options["policy"] + assert policy.root_uris == (parse_storage_uri(tmp_path), parse_storage_uri("s3://bucket/team")) + assert (policy.allow_write, policy.allow_overwrite, policy.allow_delete) == (True, True, True) + assert (policy.max_read_bytes, policy.max_write_bytes) == (1000, 2000) + assert (policy.max_results, policy.max_search_bytes) == (30, 4000) + assert policy.pipeline_dir == parse_storage_uri(tmp_path / "pipelines") + assert policy.pipeline_actions == frozenset({"FA_storage_copy", "FA_storage_verify"}) + assert policy.enabled_tools() == ("file_read", "storage_list") + + +def test_one_flag_leaves_the_other_defaults_alone() -> None: + policy = _options("--root", "memory://box/in")["policy"] + assert policy.allow_write is False + assert policy.max_read_bytes == DEFAULT_MAX_READ_BYTES + assert policy.pipeline_actions is None + assert policy.tools is None + assert "bridge" not in _options("--root", "memory://box/in") + assert _options("--no-bridge") == {"bridge": False} + + +def test_tools_none_switches_every_semantic_tool_off() -> None: + assert _options("--tools", "none")["policy"].enabled_tools() == () + assert _options("--tools", "")["policy"].enabled_tools() == () + assert _options("--pipeline-actions", "")["policy"].pipeline_actions == frozenset() + + +@pytest.mark.parametrize( + "argv", + [ + ["--allow-overwrite"], + ["--allow-delete"], + ["--tools", "file_read,file_delete"], + ["--max-results", "0"], + ["--root", "sftp://user:secret@host/data"], + ["--pipeline-actions", "FA_no_such_action"], + ], +) +def test_a_wrong_flag_ends_the_command_with_a_usage_error( + argv: list[str], capsys: pytest.CaptureFixture[str] +) -> None: + with pytest.raises(SystemExit) as caught: + _cli(argv) + assert caught.value.code == 2 + captured = capsys.readouterr() + assert "error:" in captured.err + assert "secret" not in captured.err + assert captured.out == "" + + +def test_a_wrapping_command_line_adds_and_forwards_the_same_flags(tmp_path: Path) -> None: + wrapper = argparse.ArgumentParser(prog="wrapper") + add_semantic_arguments(wrapper) + assert semantic_argv(wrapper.parse_args([])) == [] + argv = [ + "--no-bridge", + "--tools", + "file_read,storage_list", + "--allow-delete", + "--root", + "memory://box/in", + "--max-results", + "30", + "--pipeline-actions", + "FA_storage_copy,FA_storage_verify", + "--allow-write", + "--root", + str(tmp_path), + "--pipeline-dir", + "memory://box/definitions", + "--max-read-bytes", + "1000", + ] + forwarded = semantic_argv(wrapper.parse_args(argv)) + assert sorted(forwarded) == sorted(argv) + assert forwarded[:4] == ["--root", "memory://box/in", "--root", str(tmp_path)] + assert _options(*forwarded) == _options(*argv) + assert semantic_argv(wrapper.parse_args(["--tools", ""])) == ["--tools", ""] + + +def test_the_cli_starts_a_server_with_the_policy(monkeypatch: pytest.MonkeyPatch) -> None: + served: list[MCPServer] = [] + monkeypatch.setattr(MCPServer, "serve_stdio", lambda self: served.append(self)) + assert _cli(["--root", INBOX, "--allow-write", "--no-bridge", "--name", "files"]) == 0 + server = served[0] + assert server.toolkit.policy.allow_write is True + assert _names(server) == list(SEMANTIC_TOOL_NAMES) + assert _request(server, "initialize", {})["result"]["serverInfo"]["name"] == "files" diff --git a/tests/test_mcp_storage_tools.py b/tests/test_mcp_storage_tools.py new file mode 100644 index 0000000..c3982fa --- /dev/null +++ b/tests/test_mcp_storage_tools.py @@ -0,0 +1,318 @@ +"""The semantic tools that work on a directory: ``storage_list``, ``storage_copy``, ``file_search``.""" + +from __future__ import annotations + +from collections.abc import Iterator +from pathlib import Path + +import pytest + +from automation_file.exceptions import StorageTransientException +from automation_file.server.mcp_policy import ( + OUTSIDE_ROOT, + OVERWRITE_NOT_ALLOWED, + READ_ONLY, + MCPPolicy, +) +from automation_file.server.mcp_tools import SemanticToolkit +from automation_file.storage import File, Storage +from tests.mcp_support import ( + BOX, + INBOX, + clean_state, + failed, + link_directory, + refused, + remove_link, + succeed, + toolkit, + writable, +) + +TREE = f"{INBOX}/tree" +COPY = f"{INBOX}/copy" +FILES = { + "readme.txt": "A report about needles.\nNothing else.\n", + "2026/q1.csv": "id,amount\n1,10\n", + "2026/q2.csv": "id,amount\n2,20\nNEEDLE,30\n", + "2026/notes/todo.txt": "find the needle\nand another needle\n", +} + + +@pytest.fixture(autouse=True) +def _state() -> Iterator[None]: + with clean_state(): + for path, content in FILES.items(): + File(f"{TREE}/{path}").write(content) + yield + + +def _paths(body: dict, key: str = "matches") -> list[str]: + return [entry["path"] for entry in body[key]] + + +# ---------------------------------------------------------------------- storage_list + + +def test_storage_list_returns_the_direct_children() -> None: + body = succeed(toolkit().call("storage_list", {"uri": TREE})) + assert _paths(body, "entries") == ["2026", "readme.txt"] + folder, readme = body["entries"] + assert (folder["is_dir"], folder["size"], folder["uri"]) == (True, None, f"{TREE}/2026") + assert (readme["is_dir"], readme["size"], readme["name"]) == (False, 38, "readme.txt") + assert readme["modified_at"] is not None + assert (body["count"], body["total"], body["truncated"], body["recursive"]) == ( + 2, + 2, + False, + False, + ) + + +def test_storage_list_recursive_is_capped_by_the_argument_and_by_the_policy() -> None: + everything = succeed(toolkit().call("storage_list", {"uri": TREE, "recursive": True})) + assert _paths(everything, "entries") == [ + "2026", + "2026/notes", + "2026/notes/todo.txt", + "2026/q1.csv", + "2026/q2.csv", + "readme.txt", + ] + two = succeed( + toolkit().call("storage_list", {"uri": TREE, "recursive": True, "max_results": 2}) + ) + assert (two["count"], two["total"], two["truncated"]) == (2, 6, True) + capped = toolkit(max_results=3) + body = succeed(capped.call("storage_list", {"uri": TREE, "recursive": True, "max_results": 50})) + assert (body["count"], body["truncated"]) == (3, True) + + +def test_storage_list_of_a_file_or_of_nothing_fails() -> None: + kit = toolkit() + error = failed(kit.call("storage_list", {"uri": f"{TREE}/readme.txt"}), "failed") + assert "not a directory" in error["message"] + failed(kit.call("storage_list", {"uri": f"{INBOX}/absent"}), "not_found") + + +# ---------------------------------------------------------------------- file_search + + +def test_file_search_by_name_pattern() -> None: + kit = toolkit() + body = succeed(kit.call("file_search", {"uri": TREE, "pattern": "*.csv"})) + assert _paths(body) == ["2026/q1.csv", "2026/q2.csv"] + assert body["matches"][0]["uri"] == f"{TREE}/2026/q1.csv" + assert (body["count"], body["truncated"], body["candidates"]) == (2, False, 2) + assert _paths(succeed(kit.call("file_search", {"uri": TREE}))) == sorted(FILES) + assert _paths(succeed(kit.call("file_search", {"uri": TREE, "pattern": "Q?.CSV"}))) == [ + "2026/q1.csv", + "2026/q2.csv", + ] + sensitive = {"uri": TREE, "pattern": "Q?.CSV", "case_sensitive": True} + assert _paths(succeed(kit.call("file_search", sensitive))) == [] + + +def test_a_pattern_with_a_slash_is_matched_against_the_path() -> None: + kit = toolkit() + body = succeed(kit.call("file_search", {"uri": TREE, "pattern": "2026/*/*.txt"})) + assert _paths(body) == ["2026/notes/todo.txt"] + + +def test_file_search_without_recursion_stays_in_the_directory() -> None: + body = succeed(toolkit().call("file_search", {"uri": TREE, "recursive": False})) + assert _paths(body) == ["readme.txt"] + + +def test_file_search_by_content_returns_the_first_matching_line() -> None: + body = succeed(toolkit().call("file_search", {"uri": TREE, "content": "needle"})) + assert _paths(body) == ["2026/notes/todo.txt", "2026/q2.csv", "readme.txt"] + todo, q2, readme = body["matches"] + assert (todo["line"], todo["snippet"], todo["matching_lines"]) == (1, "find the needle", 2) + assert (q2["line"], q2["snippet"]) == (3, "NEEDLE,30") + assert readme["line"] == 1 + assert (body["searched_files"], body["complete"], body["skipped"]) == (4, True, []) + assert body["searched_bytes"] == sum(len(text) for text in FILES.values()) + sensitive = {"uri": TREE, "content": "needle", "case_sensitive": True} + assert _paths(succeed(toolkit().call("file_search", sensitive))) == [ + "2026/notes/todo.txt", + "readme.txt", + ] + + +def test_file_search_combines_the_name_and_the_content() -> None: + body = succeed( + toolkit().call("file_search", {"uri": TREE, "pattern": "*.csv", "content": "needle"}) + ) + assert _paths(body) == ["2026/q2.csv"] + assert body["searched_files"] == 2 + + +def test_file_search_stops_at_the_result_cap() -> None: + by_name = succeed(toolkit().call("file_search", {"uri": TREE, "max_results": 1})) + assert (by_name["count"], by_name["truncated"]) == (1, True) + by_content = succeed( + toolkit().call("file_search", {"uri": TREE, "content": "needle", "max_results": 1}) + ) + assert _paths(by_content) == ["2026/notes/todo.txt"] + assert (by_content["truncated"], by_content["complete"]) == (True, False) + assert by_content["searched_files"] == 1 + + +def test_a_content_search_reads_no_more_than_its_budget() -> None: + File(f"{TREE}/big.log").write("x" * 500 + " needle") + kit = toolkit(max_search_bytes=100) + body = succeed(kit.call("file_search", {"uri": TREE, "content": "needle"})) + assert body["searched_bytes"] <= 100 + assert body["complete"] is False + skipped = {entry["uri"]: entry["reason"] for entry in body["skipped"]} + assert "read budget" in skipped[f"{TREE}/big.log"] + assert f"{TREE}/big.log" not in [match["uri"] for match in body["matches"]] + assert body["skipped_count"] == len(skipped) + + +def test_a_content_search_skips_binary_files_and_files_it_cannot_read( + monkeypatch: pytest.MonkeyPatch, +) -> None: + File(f"{TREE}/blob.bin").write(b"\x00\x01needle\x00") + real = File.open_read + + def flaky(self: File): + if self.name == "q1.csv": + raise StorageTransientException("the backend timed out") + return real(self) + + monkeypatch.setattr(File, "open_read", flaky) + body = succeed(toolkit().call("file_search", {"uri": TREE, "content": "needle"})) + reasons = {entry["uri"]: entry["reason"] for entry in body["skipped"]} + assert reasons == { + f"{TREE}/blob.bin": "not a text file", + f"{TREE}/2026/q1.csv": "StorageTransientException", + } + assert body["complete"] is False + assert len(body["matches"]) == 3 + + +def test_a_pattern_or_a_needle_that_is_too_long_is_refused() -> None: + kit = toolkit() + failed(kit.call("file_search", {"uri": TREE, "pattern": "*" * 300}), "invalid_arguments") + failed(kit.call("file_search", {"uri": TREE, "content": "n" * 2000}), "invalid_arguments") + failed(kit.call("file_search", {"uri": TREE, "content": ""}), "invalid_arguments") + + +def test_file_search_does_not_follow_a_link_out_of_a_local_root(tmp_path: Path) -> None: + root, outside = tmp_path / "root", tmp_path / "outside" + root.mkdir() + outside.mkdir() + (root / "inside.txt").write_text("needle inside", encoding="utf-8") + (outside / "secret.txt").write_text("needle outside", encoding="utf-8") + link = root / "escape" + link_directory(link, outside) + try: + kit = SemanticToolkit(MCPPolicy(roots=[root])) + body = succeed(kit.call("file_search", {"uri": str(root), "content": "needle"})) + assert _paths(body) == ["inside.txt"] + refused(kit.call("file_search", {"uri": str(link), "content": "needle"}), OUTSIDE_ROOT) + finally: + remove_link(link) + + +# ---------------------------------------------------------------------- storage_copy + + +def test_storage_copy_of_a_file_behaves_like_file_copy() -> None: + source, target = f"{TREE}/readme.txt", f"{INBOX}/readme-copy.txt" + body = succeed(writable().call("storage_copy", {"source": source, "target": target})) + assert (body["kind"], body["done"], body["size"]) == ("file", True, 38) + assert File(target).read() == File(source).read() + verified = succeed( + writable().call( + "storage_copy", {"source": source, "target": f"{INBOX}/v.txt", "verify": True} + ) + ) + assert verified["verified"] is True + + +def test_storage_copy_of_a_tree_plans_first_and_then_copies() -> None: + kit = writable() + plan = succeed(kit.call("storage_copy", {"source": TREE, "target": COPY, "dry_run": True})) + assert (plan["kind"], plan["dry_run"], plan["done"]) == ("tree", True, False) + assert plan["planned"] == { + "copy": 4, + "overwrite": 0, + "skip": 0, + "bytes": sum(len(text) for text in FILES.values()), + } + assert plan["paths"] == sorted(FILES) + assert plan["existing"] == [] + assert not Storage(COPY).exists() + done = succeed(kit.call("storage_copy", {"source": TREE, "target": COPY})) + assert (done["done"], done["ok"], done["copied"], done["failed"]) == (True, True, 4, 0) + for path, content in FILES.items(): + assert File(f"{COPY}/{path}").read_text() == content + + +def test_a_tree_copy_skips_existing_files_unless_overwriting_is_asked_and_allowed() -> None: + File(f"{COPY}/readme.txt").write("kept") + kit = writable() + plan = succeed(kit.call("storage_copy", {"source": TREE, "target": COPY, "dry_run": True})) + assert plan["planned"]["copy"] == 3 + assert plan["planned"]["skip"] == 1 + assert plan["existing"] == ["readme.txt"] + arguments = {"source": TREE, "target": COPY, "overwrite": True} + refused(kit.call("storage_copy", {**arguments, "dry_run": True}), OVERWRITE_NOT_ALLOWED) + refused(kit.call("storage_copy", arguments), OVERWRITE_NOT_ALLOWED) + done = succeed(kit.call("storage_copy", {"source": TREE, "target": COPY})) + assert (done["copied"], done["skipped"]) == (3, 1) + assert File(f"{COPY}/readme.txt").read() == b"kept" + allowed = writable(allow_overwrite=True) + plan = succeed(allowed.call("storage_copy", {**arguments, "dry_run": True})) + assert plan["planned"]["overwrite"] == 4 + replaced = succeed(allowed.call("storage_copy", arguments)) + assert (replaced["copied"], replaced["skipped"]) == (4, 0) + assert File(f"{COPY}/readme.txt").read_text() == FILES["readme.txt"] + + +def test_a_tree_copy_reports_the_files_that_failed_as_an_error_outcome( + monkeypatch: pytest.MonkeyPatch, +) -> None: + real = File.copy_to + + def flaky(self: File, target, *, overwrite: bool = True) -> File: + if self.name == "q1.csv": + raise StorageTransientException("the backend timed out") + return real(self, target, overwrite=overwrite) + + monkeypatch.setattr(File, "copy_to", flaky) + outcome = writable().call("storage_copy", {"source": TREE, "target": COPY}) + error = failed(outcome, "failed") + assert "finished with failures" in error["message"] + body = outcome.payload + assert (body["ok"], body["copied"], body["failed"]) == (False, 3, 1) + assert "StorageTransientException" in body["errors"]["2026/q1.csv"] + + +def test_a_tree_cannot_be_copied_into_itself_or_out_of_the_roots() -> None: + kit = writable() + inside = kit.call("storage_copy", {"source": TREE, "target": f"{TREE}/2026/again"}) + assert "inside" in failed(inside, "invalid_arguments")["message"] + same = kit.call("storage_copy", {"source": TREE, "target": TREE}) + failed(same, "invalid_arguments") + refused(kit.call("storage_copy", {"source": TREE, "target": f"{BOX}/elsewhere"}), OUTSIDE_ROOT) + assert not Storage(f"{BOX}/elsewhere").exists() + + +def test_storage_copy_is_refused_on_a_read_only_server() -> None: + refused(toolkit().call("storage_copy", {"source": TREE, "target": COPY}), READ_ONLY) + failed( + writable().call("storage_copy", {"source": f"{INBOX}/absent", "target": COPY}), + ("not_found"), + ) + + +def test_the_lists_of_a_tree_plan_are_capped() -> None: + kit = writable(max_results=2) + plan = succeed(kit.call("storage_copy", {"source": TREE, "target": COPY, "dry_run": True})) + assert plan["planned"]["copy"] == 4 + assert len(plan["paths"]) == 2 + assert plan["truncated"] is True diff --git a/tests/test_mcp_tools.py b/tests/test_mcp_tools.py new file mode 100644 index 0000000..9513f2a --- /dev/null +++ b/tests/test_mcp_tools.py @@ -0,0 +1,684 @@ +"""The semantic file and storage tools, called without the JSON-RPC layer.""" + +from __future__ import annotations + +import base64 +import hashlib +import logging +from collections.abc import Iterator +from pathlib import Path + +import pytest + +from automation_file import Event, event_bus +from automation_file.exceptions import StorageTransientException +from automation_file.logging_config import file_automation_logger +from automation_file.server.mcp_policy import ( + DELETE_NOT_ALLOWED, + LIMIT_EXCEEDED, + NO_ROOT, + OUTSIDE_ROOT, + OVERWRITE_NOT_ALLOWED, + READ_ONLY, + SEMANTIC_TOOL_NAMES, + TOOL_DISABLED, + MCPPolicy, + MCPToolException, +) +from automation_file.server.mcp_tools import ( + SEMANTIC_TOOLS, + MCPToolCompleted, + MCPToolFailed, + SemanticToolkit, +) +from automation_file.storage import File, Storage +from tests.mcp_support import ( + BOX, + INBOX, + audited, + clean_state, + failed, + link_directory, + refused, + remove_link, + succeed, + toolkit, + writable, +) + +A = f"{INBOX}/a.txt" +B = f"{INBOX}/b.txt" +OUTSIDE = f"{BOX}/in-b/a.txt" +STORAGE_TOOLS = ( + ("file_read", {"uri": A}), + ("file_search", {"uri": INBOX}), + ("file_checksum", {"uri": A}), + ("file_verify", {"uri": A, "expected": "00"}), + ("storage_list", {"uri": INBOX}), +) +CHANGING_TOOLS = ( + ("file_write", {"uri": B, "content": "x"}), + ("file_copy", {"source": A, "target": B}), + ("file_move", {"source": A, "target": B}), + ("storage_copy", {"source": A, "target": B}), +) + + +@pytest.fixture(autouse=True) +def _state() -> Iterator[None]: + with clean_state(): + File(A).write("alpha\nbeta needle\ngamma\n") + yield + + +@pytest.fixture +def events() -> Iterator[list[Event]]: + seen: list[Event] = [] + subscription = event_bus.subscribe(seen.append, types="mcp.*") + yield seen + event_bus.unsubscribe(subscription) + + +@pytest.fixture +def logged() -> Iterator[list[logging.LogRecord]]: + """The records the library logs while the test runs (its logger does not propagate).""" + records: list[logging.LogRecord] = [] + handler = logging.Handler(level=logging.INFO) + handler.emit = records.append # type: ignore[method-assign] + file_automation_logger.addHandler(handler) + yield records + file_automation_logger.removeHandler(handler) + + +def _sha256(data: bytes) -> str: + return hashlib.sha256(data).hexdigest() + + +def _next_door(arguments: dict) -> dict: + """Return ``arguments`` pointing at the tree next to the allowed one.""" + swaps = {A: OUTSIDE, INBOX: f"{BOX}/in-b"} + return {key: swaps.get(value, value) for key, value in arguments.items()} + + +# ---------------------------------------------------------------------- the catalogue + + +def test_the_catalogue_holds_the_fourteen_tools_in_order() -> None: + assert tuple(tool.name for tool in SEMANTIC_TOOLS) == SEMANTIC_TOOL_NAMES + assert [tool["name"] for tool in toolkit().descriptors()] == list(SEMANTIC_TOOL_NAMES) + + +@pytest.mark.parametrize("tool", SEMANTIC_TOOLS, ids=lambda tool: tool.name) +def test_every_tool_has_a_closed_schema_with_described_arguments(tool) -> None: + schema = tool.input_schema + assert schema["type"] == "object" + assert schema["additionalProperties"] is False + assert len(tool.description) > 40 + for name, spec in schema["properties"].items(): + assert spec["type"] in {"string", "integer", "boolean", "object"}, name + assert len(spec["description"]) > 10, name + assert set(schema.get("required", ())) <= set(schema["properties"]) + + +def test_every_tool_that_changes_something_takes_dry_run() -> None: + changing = {tool.name for tool in SEMANTIC_TOOLS if tool.changes} + assert changing == { + "file_write", + "file_copy", + "file_move", + "storage_copy", + "pipeline_create", + "pipeline_run", + } + for tool in SEMANTIC_TOOLS: + assert ("dry_run" in tool.input_schema["properties"]) is tool.changes, tool.name + + +def test_a_descriptor_is_a_copy() -> None: + kit = toolkit() + kit.descriptors()[0]["inputSchema"]["properties"].clear() + assert "uri" in kit.descriptors()[0]["inputSchema"]["properties"] + + +def test_an_unknown_tool_name_is_not_an_outcome() -> None: + with pytest.raises(MCPToolException, match="unknown semantic tool"): + toolkit().call("file_delete", {}) + assert SemanticToolkit.owns("file_read") is True + assert SemanticToolkit.owns("FA_storage_copy") is False + + +# ---------------------------------------------------------------------- arguments + + +def test_wrong_arguments_are_reported_together_without_their_values() -> None: + outcome = toolkit().call( + "file_read", {"offset": "secret-value", "max_bytes": 0, "surprise": 1, "encoding": True} + ) + message = failed(outcome, "invalid_arguments")["message"] + assert "'uri' is required" in message + assert "'offset' must be of type integer" in message + assert "'max_bytes' must be 1 or more" in message + assert "'surprise' is not an argument of file_read" in message + assert "'encoding' must be of type string" in message + assert "secret-value" not in message + + +def test_a_boolean_is_not_an_integer_and_null_means_left_out() -> None: + kit = toolkit() + failed(kit.call("file_read", {"uri": A, "offset": True}), "invalid_arguments") + failed(kit.call("storage_list", {"uri": INBOX, "recursive": 1}), "invalid_arguments") + assert succeed(kit.call("file_read", {"uri": A, "max_bytes": None}))["bytes"] == 24 + + +# ---------------------------------------------------------------------- refusals by location + + +@pytest.mark.parametrize(("name", "arguments"), STORAGE_TOOLS + CHANGING_TOOLS) +def test_without_a_root_every_storage_tool_says_how_to_configure_one( + name: str, arguments: dict +) -> None: + kit = SemanticToolkit(MCPPolicy(allow_write=True, allow_delete=True)) + error = refused(kit.call(name, arguments), NO_ROOT) + assert "--root" in error["message"] + + +@pytest.mark.parametrize(("name", "arguments"), STORAGE_TOOLS) +def test_a_location_next_to_the_root_is_refused(name: str, arguments: dict) -> None: + File(OUTSIDE).write("neighbour") + refused(toolkit().call(name, _next_door(arguments)), OUTSIDE_ROOT) + + +@pytest.mark.parametrize("name", ["file_copy", "file_move", "storage_copy"]) +def test_both_ends_of_a_transfer_must_be_inside_a_root(name: str) -> None: + File(OUTSIDE).write("neighbour") + kit = writable(allow_delete=True) + refused(kit.call(name, {"source": OUTSIDE, "target": B}), OUTSIDE_ROOT) + refused(kit.call(name, {"source": A, "target": f"{BOX}/in-b/copy.txt"}), OUTSIDE_ROOT) + assert not File(B).exists() + assert not File(f"{BOX}/in-b/copy.txt").exists() + + +def test_dot_dot_is_an_invalid_uri() -> None: + error = failed(toolkit().call("file_read", {"uri": f"{INBOX}/../secret.txt"}), "invalid_uri") + assert ".." in error["message"] + + +def test_a_link_out_of_a_local_root_cannot_be_read_or_written(tmp_path: Path) -> None: + root, outside = tmp_path / "root", tmp_path / "outside" + root.mkdir() + outside.mkdir() + (outside / "secret.txt").write_text("secret", encoding="utf-8") + link = root / "escape" + link_directory(link, outside) + try: + kit = SemanticToolkit(MCPPolicy(roots=[root], allow_write=True)) + refused(kit.call("file_read", {"uri": str(link / "secret.txt")}), OUTSIDE_ROOT) + refused( + kit.call("file_write", {"uri": str(link / "new.txt"), "content": "x"}), OUTSIDE_ROOT + ) + assert not (outside / "new.txt").exists() + finally: + remove_link(link) + + +def test_a_local_root_works_end_to_end(tmp_path: Path) -> None: + kit = SemanticToolkit(MCPPolicy(roots=[tmp_path], allow_write=True)) + target = tmp_path / "reports" / "a.txt" + written = succeed(kit.call("file_write", {"uri": str(target), "content": "local"})) + assert target.read_text(encoding="utf-8") == "local" + assert written["uri"].startswith("local:///") + assert succeed(kit.call("file_read", {"uri": written["uri"]}))["content"] == "local" + listed = succeed(kit.call("storage_list", {"uri": str(tmp_path), "recursive": True})) + assert [entry["path"] for entry in listed["entries"]] == ["reports", "reports/a.txt"] + refused(kit.call("file_read", {"uri": str(tmp_path.parent / "elsewhere.txt")}), OUTSIDE_ROOT) + + +# ---------------------------------------------------------------------- read-only and disabled + + +@pytest.mark.parametrize(("name", "arguments"), CHANGING_TOOLS) +def test_a_read_only_server_refuses_every_change_even_as_a_dry_run( + name: str, arguments: dict +) -> None: + kit = toolkit() + error = refused(kit.call(name, arguments), READ_ONLY) + assert "--allow-write" in error["message"] + refused(kit.call(name, {**arguments, "dry_run": True}), READ_ONLY) + assert not File(B).exists() + assert File(A).exists() + + +def test_a_disabled_tool_is_refused_and_not_listed() -> None: + kit = toolkit(tools=["file_read"]) + assert [tool["name"] for tool in kit.descriptors()] == ["file_read"] + refused(kit.call("storage_list", {"uri": INBOX}), TOOL_DISABLED) + succeed(kit.call("file_read", {"uri": A})) + + +# ---------------------------------------------------------------------- file_read + + +def test_file_read_returns_text_with_its_size() -> None: + body = succeed(toolkit().call("file_read", {"uri": A})) + assert body["content"] == "alpha\nbeta needle\ngamma\n" + assert (body["size"], body["bytes"], body["offset"]) == (24, 24, 0) + assert body["truncated"] is False + assert body["next_offset"] is None + assert body["encoding"] == "utf-8" + assert body["uri"] == A + + +def test_file_read_stops_at_the_policy_cap_and_says_where_to_continue() -> None: + kit = toolkit(max_read_bytes=10) + first = succeed(kit.call("file_read", {"uri": A, "max_bytes": 1000})) + assert first["content"] == "alpha\nbeta" + assert first["truncated"] is True + assert first["next_offset"] == 10 + rest = succeed(kit.call("file_read", {"uri": A, "offset": 20})) + assert rest["content"] == "mma\n" + assert rest["truncated"] is False + + +def test_file_read_takes_a_smaller_window_than_the_cap() -> None: + body = succeed(toolkit().call("file_read", {"uri": A, "offset": 6, "max_bytes": 4})) + assert body["content"] == "beta" + assert body["next_offset"] == 10 + + +def test_file_read_does_not_cut_a_character_in_half() -> None: + File(f"{INBOX}/zh.txt").write("檔案自動化") + kit = toolkit(max_read_bytes=4) + first = succeed(kit.call("file_read", {"uri": f"{INBOX}/zh.txt"})) + assert first["content"] == "檔" + assert first["bytes"] == 3 + assert first["next_offset"] == 3 + second = succeed(kit.call("file_read", {"uri": f"{INBOX}/zh.txt", "offset": 3})) + assert second["content"] == "案" + + +def test_file_read_returns_binary_content_as_base64() -> None: + payload = bytes(range(256)) + File(f"{INBOX}/blob.bin").write(payload) + kit = toolkit() + failed(kit.call("file_read", {"uri": f"{INBOX}/blob.bin"}), "invalid_arguments") + body = succeed(kit.call("file_read", {"uri": f"{INBOX}/blob.bin", "encoding": "base64"})) + assert base64.b64decode(body["content"]) == payload + assert body["bytes"] == 256 + + +def test_file_read_refuses_a_codec_that_is_not_a_text_encoding() -> None: + error = failed(toolkit().call("file_read", {"uri": A, "encoding": "zlib"}), "invalid_arguments") + assert "unknown text encoding" in error["message"] + big5 = "繁體中文".encode("big5") + File(f"{INBOX}/big5.txt").write(big5) + body = succeed(toolkit().call("file_read", {"uri": f"{INBOX}/big5.txt", "encoding": "big5"})) + assert body["content"] == "繁體中文" + + +def test_file_read_of_a_missing_file_or_a_directory_fails() -> None: + kit = toolkit() + failed(kit.call("file_read", {"uri": f"{INBOX}/absent.txt"}), "not_found") + error = failed(kit.call("file_read", {"uri": INBOX}), "failed") + assert "is a directory" in error["message"] + + +# ---------------------------------------------------------------------- file_write + + +def test_file_write_creates_a_file_and_reports_its_digest() -> None: + body = succeed(writable().call("file_write", {"uri": B, "content": "héllo"})) + assert File(B).read() == "héllo".encode() + assert body["size"] == 6 + assert body["sha256"] == _sha256("héllo".encode()) + assert (body["overwrites"], body["written"], body["dry_run"]) == (False, True, False) + + +def test_file_write_takes_base64_and_other_encodings() -> None: + kit = writable() + payload = bytes(range(256)) + encoded = base64.b64encode(payload).decode("ascii") + succeed(kit.call("file_write", {"uri": B, "content": encoded, "encoding": "base64"})) + assert File(B).read() == payload + failed( + kit.call("file_write", {"uri": f"{INBOX}/c", "content": "***", "encoding": "base64"}), + "invalid_arguments", + ) + succeed(kit.call("file_write", {"uri": f"{INBOX}/c", "content": "中文", "encoding": "big5"})) + assert File(f"{INBOX}/c").read() == "中文".encode("big5") + error = failed( + kit.call("file_write", {"uri": f"{INBOX}/d", "content": "中文", "encoding": "ascii"}), + "invalid_arguments", + ) + assert "中文" not in error["message"] + + +def test_file_write_as_a_dry_run_changes_nothing() -> None: + body = succeed(writable().call("file_write", {"uri": B, "content": "abc", "dry_run": True})) + assert (body["dry_run"], body["written"], body["size"]) == (True, False, 3) + assert body["overwrites"] is False + assert not File(B).exists() + + +def test_file_write_refuses_content_above_the_size_limit() -> None: + kit = writable(max_write_bytes=4) + error = refused(kit.call("file_write", {"uri": B, "content": "12345"}), LIMIT_EXCEEDED) + assert "--max-write-bytes" in error["message"] + assert not File(B).exists() + succeed(kit.call("file_write", {"uri": B, "content": "1234"})) + + +def test_replacing_a_file_needs_the_argument_and_the_permission() -> None: + kit = writable() + error = failed(kit.call("file_write", {"uri": A, "content": "new"}), "already_exists") + assert "does not allow replacing" in error["message"] + refused( + kit.call("file_write", {"uri": A, "content": "new", "overwrite": True}), + OVERWRITE_NOT_ALLOWED, + ) + assert File(A).read().startswith(b"alpha") + allowed = writable(allow_overwrite=True) + error = failed(allowed.call("file_write", {"uri": A, "content": "new"}), "already_exists") + assert "pass overwrite=true" in error["message"] + plan = succeed( + allowed.call("file_write", {"uri": A, "content": "new", "overwrite": True, "dry_run": True}) + ) + assert (plan["overwrites"], plan["replaced_size"]) == (True, 24) + assert File(A).read().startswith(b"alpha") + succeed(allowed.call("file_write", {"uri": A, "content": "new", "overwrite": True})) + assert File(A).read() == b"new" + + +def test_overwrite_true_is_harmless_when_nothing_is_there() -> None: + body = succeed(writable().call("file_write", {"uri": B, "content": "x", "overwrite": True})) + assert body["overwrites"] is False + assert File(B).read() == b"x" + + +def test_file_write_onto_a_directory_fails() -> None: + Storage(INBOX).mkdir("sub") + kit = writable(allow_overwrite=True) + outcome = kit.call("file_write", {"uri": f"{INBOX}/sub", "content": "x", "overwrite": True}) + assert "is a directory" in failed(outcome, "failed")["message"] + + +# ---------------------------------------------------------------------- file_copy and file_move + + +def test_file_copy_keeps_the_source() -> None: + body = succeed(writable().call("file_copy", {"source": A, "target": B})) + assert File(B).read() == File(A).read() + assert (body["source"], body["target"], body["size"]) == (A, B, 24) + assert (body["done"], body["deletes_source"], body["overwrites"]) == (True, False, False) + + +def test_file_copy_with_verify_reports_the_digest() -> None: + body = succeed(writable().call("file_copy", {"source": A, "target": B, "verify": True})) + assert body["verified"] is True + assert body["sha256"] == _sha256(File(A).read()) + + +def test_file_copy_as_a_dry_run_reports_the_plan_and_copies_nothing() -> None: + File(B).write("old") + kit = writable(allow_overwrite=True) + arguments = {"source": A, "target": B, "overwrite": True, "dry_run": True} + plan = succeed(kit.call("file_copy", arguments)) + assert (plan["source"], plan["target"]) == (A, B) + assert (plan["size"], plan["overwrites"], plan["replaced_size"]) == (24, True, 3) + assert (plan["dry_run"], plan["done"]) == (True, False) + assert File(B).read() == b"old" + + +def test_file_copy_over_an_existing_file_needs_the_permission() -> None: + File(B).write("old") + kit = writable() + failed(kit.call("file_copy", {"source": A, "target": B}), "already_exists") + refused( + kit.call("file_copy", {"source": A, "target": B, "overwrite": True}), + (OVERWRITE_NOT_ALLOWED), + ) + assert File(B).read() == b"old" + allowed = writable(allow_overwrite=True) + succeed(allowed.call("file_copy", {"source": A, "target": B, "overwrite": True})) + assert File(B).read() == File(A).read() + + +def test_file_copy_onto_itself_or_from_nothing_fails() -> None: + kit = writable(allow_overwrite=True) + outcome = kit.call("file_copy", {"source": A, "target": A, "overwrite": True}) + assert "same file" in failed(outcome, "invalid_arguments")["message"] + failed(kit.call("file_copy", {"source": f"{INBOX}/absent", "target": B}), "not_found") + + +def test_file_move_needs_the_delete_permission() -> None: + error = refused(writable().call("file_move", {"source": A, "target": B}), DELETE_NOT_ALLOWED) + assert "--allow-delete" in error["message"] + assert File(A).exists() + assert not File(B).exists() + + +def test_file_move_deletes_the_source() -> None: + content = File(A).read() + kit = writable(allow_delete=True) + plan = succeed(kit.call("file_move", {"source": A, "target": B, "dry_run": True})) + assert (plan["deletes_source"], plan["done"]) == (True, False) + assert File(A).exists() + assert not File(B).exists() + body = succeed(kit.call("file_move", {"source": A, "target": B})) + assert body["done"] is True + assert File(B).read() == content + assert not File(A).exists() + + +def test_file_move_with_verify_deletes_only_after_the_digests_match() -> None: + content = File(A).read() + body = succeed( + writable(allow_delete=True).call("file_move", {"source": A, "target": B, "verify": True}) + ) + assert (body["verified"], body["sha256"]) == (True, _sha256(content)) + assert not File(A).exists() + + +def test_file_move_keeps_the_source_when_the_copy_does_not_match( + monkeypatch: pytest.MonkeyPatch, +) -> None: + real = File.copy_to + + def corrupting(self: File, target, *, overwrite: bool = True) -> File: + copied = real(self, target, overwrite=overwrite) + copied.write("damaged in transit") + return copied + + monkeypatch.setattr(File, "copy_to", corrupting) + outcome = writable(allow_delete=True).call( + "file_move", {"source": A, "target": B, "verify": True} + ) + error = failed(outcome, "checksum_mismatch") + assert "source was left in place" in error["message"] + assert File(A).read().startswith(b"alpha") + + +def test_a_transfer_between_a_local_root_and_a_memory_root(tmp_path: Path) -> None: + kit = SemanticToolkit(MCPPolicy(roots=[tmp_path, INBOX], allow_write=True, allow_delete=True)) + local = tmp_path / "out" / "a.txt" + succeed(kit.call("file_copy", {"source": A, "target": str(local), "verify": True})) + assert local.read_bytes() == File(A).read() + succeed(kit.call("file_move", {"source": str(local), "target": B})) + assert not local.exists() + assert File(B).read() == File(A).read() + + +# ---------------------------------------------------------------------- checksum and verify + + +def test_file_checksum_defaults_to_sha256() -> None: + kit = toolkit() + body = succeed(kit.call("file_checksum", {"uri": A})) + assert (body["algorithm"], body["value"], body["size"]) == ( + "sha256", + _sha256(File(A).read()), + 24, + ) + md5 = succeed(kit.call("file_checksum", {"uri": A, "algorithm": "MD5"})) + assert md5["value"] == hashlib.md5(File(A).read(), usedforsecurity=False).hexdigest() + failed(kit.call("file_checksum", {"uri": A, "algorithm": "crc-nope"}), "failed") + + +def test_file_verify_answers_match_or_mismatch_without_failing() -> None: + kit = toolkit() + digest = _sha256(File(A).read()) + for expected in (digest, digest.upper(), f"sha256:{digest}"): + body = succeed(kit.call("file_verify", {"uri": A, "expected": expected})) + assert (body["match"], body["actual"], body["algorithm"]) == (True, digest, "sha256") + wrong = succeed(kit.call("file_verify", {"uri": A, "expected": "00"})) + assert (wrong["match"], wrong["expected"]) == (False, "00") + sha1 = hashlib.sha1(File(A).read(), usedforsecurity=False).hexdigest() + prefixed = succeed(kit.call("file_verify", {"uri": A, "expected": f"sha1:{sha1}"})) + assert (prefixed["match"], prefixed["algorithm"]) == (True, "sha1") + failed(kit.call("file_verify", {"uri": A, "expected": "sha256:"}), "invalid_arguments") + failed(kit.call("file_verify", {"uri": f"{INBOX}/absent", "expected": digest}), "not_found") + + +# ---------------------------------------------------------------------- traceability + + +def test_every_outcome_carries_a_correlation_id_of_its_own() -> None: + kit = toolkit() + first = kit.call("file_read", {"uri": A}) + second = kit.call("file_read", {"uri": OUTSIDE}) + assert len(first.correlation_id) == 32 + assert first.payload["tool"] == "file_read" + assert second.is_error is True + assert len(second.correlation_id) == 32 + assert first.correlation_id != second.correlation_id + + +def test_a_call_is_reported_as_one_event_with_the_actor_and_the_correlation_id( + events: list[Event], +) -> None: + kit = writable() + kit.set_client("Claude Desktop/1.2") + outcome = kit.call("file_copy", {"source": A, "target": B}) + assert [type(event) for event in events] == [MCPToolCompleted] + event = events[0] + assert (event.type, event.source, event.actor) == ( + "mcp.tool.completed", + "mcp", + "mcp:Claude_Desktop_1.2", + ) + assert event.correlation_id == outcome.correlation_id + assert event.payload["action"] == "file_copy" + assert event.payload["status"] == "ok" + assert (event.payload["resource"], event.payload["source_uri"]) == (B, A) + assert event.payload["dry_run"] is False + + +def test_a_refused_call_is_a_warning_event_and_a_log_line_without_values( + events: list[Event], logged: list[logging.LogRecord] +) -> None: + outcome = toolkit().call("file_write", {"uri": B, "content": "TOP-SECRET-CONTENT"}) + refused(outcome, READ_ONLY) + event = events[0] + assert isinstance(event, MCPToolFailed) + assert (event.payload["status"], event.payload["code"]) == ("refused", READ_ONLY) + assert event.severity.value == "warning" + assert event.payload["resource"] == B + assert "TOP-SECRET-CONTENT" not in str(event.to_dict()) + assert [record.levelno for record in logged] == [logging.WARNING] + line = logged[0].getMessage() + assert f"mcp_tools: file_write refused ({READ_ONLY})" in line + assert outcome.correlation_id in line + assert "TOP-SECRET-CONTENT" not in line + assert B not in line + + +def test_a_failed_call_is_an_info_event(events: list[Event]) -> None: + failed(toolkit().call("file_read", {"uri": f"{INBOX}/absent.txt"}), "not_found") + event = events[0] + assert isinstance(event, MCPToolFailed) + assert (event.payload["status"], event.payload["code"]) == ("error", "not_found") + assert event.severity.value == "info" + assert "StorageNotFoundException" in event.payload["error"] + + +def test_a_digest_that_does_not_match_is_an_error_event( + events: list[Event], monkeypatch: pytest.MonkeyPatch +) -> None: + real = File.copy_to + + def corrupting(self: File, target, *, overwrite: bool = True) -> File: + copied = real(self, target, overwrite=overwrite) + copied.write("damaged in transit") + return copied + + monkeypatch.setattr(File, "copy_to", corrupting) + outcome = writable().call("file_copy", {"source": A, "target": B, "verify": True}) + failed(outcome, "checksum_mismatch") + assert (events[0].severity.value, events[0].payload["code"]) == ("error", "checksum_mismatch") + + +def test_a_backend_failure_and_a_partly_failed_copy_are_warning_events( + events: list[Event], monkeypatch: pytest.MonkeyPatch +) -> None: + def unavailable(self: File, target, *, overwrite: bool = True) -> File: + raise StorageTransientException("the backend timed out") + + File(f"{INBOX}/tree/one.txt").write("1") + monkeypatch.setattr(File, "copy_to", unavailable) + kit = writable() + failed(kit.call("file_copy", {"source": A, "target": B}), "failed") + partly = kit.call("storage_copy", {"source": f"{INBOX}/tree", "target": f"{INBOX}/copy"}) + failed(partly, "failed") + assert partly.payload["failed"] == 1 + assert [event.severity.value for event in events] == ["warning", "warning"] + + +def test_an_unexpected_exception_is_an_internal_error_whose_text_stays_out_of_the_event( + events: list[Event], monkeypatch: pytest.MonkeyPatch +) -> None: + def broken(self: File) -> None: + raise RuntimeError("holds file content") + + monkeypatch.setattr(File, "stat", broken) + outcome = toolkit().call("file_read", {"uri": A}) + error = failed(outcome, "internal_error") + assert error["exception"] == "RuntimeError" + assert events[0].severity.value == "error" + assert events[0].payload["error"] == "RuntimeError" + + +def test_the_audit_trail_ties_the_storage_operation_to_the_call() -> None: + store = audited() + outcome = writable().call("file_copy", {"source": A, "target": B}) + records = store.search(correlation_id=outcome.correlation_id) + assert [(record.source, record.action) for record in records] == [ + ("mcp", "mcp.tool.completed"), + ("storage", "copy"), + ] + assert {record.actor for record in records} == {"mcp"} + assert records[1].resource == B + assert records[0].metadata["action"] == "file_copy" + refusal = toolkit().call("file_read", {"uri": OUTSIDE}) + refused_records = store.search(correlation_id=refusal.correlation_id) + assert [(record.action, record.status) for record in refused_records] == [ + ("mcp.tool.failed", "refused") + ] + + +def test_the_client_name_is_cleaned_before_it_labels_the_actor() -> None: + kit = toolkit() + assert kit.actor == "mcp" + kit.set_client(" evil\nname with spaces " + "x" * 200) + assert kit.actor.startswith("mcp:evil_name_with_spaces_x") + assert "\n" not in kit.actor + assert len(kit.actor) <= len("mcp:") + 64 + kit.set_client(None) + assert kit.actor == "mcp" + assert toolkit(actor="mcp-finance").actor == "mcp-finance" + + +def test_to_mcp_wraps_the_payload_as_one_json_text_block() -> None: + outcome = toolkit().call("file_read", {"uri": f"{INBOX}/absent.txt"}) + wrapped = outcome.to_mcp() + assert wrapped["isError"] is True + assert [block["type"] for block in wrapped["content"]] == ["text"] + assert outcome.correlation_id in wrapped["content"][0]["text"] From 805dbf0f39d0a11b79e824f36a5fe844c7b36719 Mon Sep 17 00:00:00 2001 From: JeffreyChen Date: Thu, 8 Oct 2026 15:15:03 +0800 Subject: [PATCH 45/59] docs: state one positioning in the READMEs, the manuals and the metadata --- README.md | 20 +++++++++----- README.zh-CN.md | 16 ++++++++--- README.zh-TW.md | 16 ++++++++--- dev.toml | 8 +++++- docs/source/Eng/eng_index.rst | 5 ++++ docs/source/Eng/usage/quickstart.rst | 37 ++++++++++++++++++++++++++ docs/source/Zh-CN/usage/quickstart.rst | 36 +++++++++++++++++++++++++ docs/source/Zh-CN/zh_cn_index.rst | 4 +++ docs/source/Zh-TW/usage/quickstart.rst | 36 +++++++++++++++++++++++++ docs/source/Zh-TW/zh_tw_index.rst | 4 +++ docs/updates/2026-10.md | 13 +++++++++ docs/updates/README.md | 3 ++- stable.toml | 8 +++++- 13 files changed, 189 insertions(+), 17 deletions(-) diff --git a/README.md b/README.md index 0d0e615..aa4b8a8 100644 --- a/README.md +++ b/README.md @@ -2,12 +2,20 @@ **English** | [繁體中文](README.zh-TW.md) | [简体中文](README.zh-CN.md) -A modular automation framework for local file / directory / ZIP operations, -SSRF-validated HTTP downloads, remote storage (Google Drive, S3, Azure Blob, -Dropbox, SFTP), and JSON-driven action execution over embedded TCP / HTTP -servers. Ships with a PySide6 GUI that exposes every feature through tabs. -All public functionality is re-exported from the top-level `automation_file` -facade. +FileAutomation is a universal file layer and data-pipeline runtime: one API for local and +remote storage, file integrity monitoring, pipelines with retry and resume, scheduling, +event-driven notifications, an audit trail, and automation through JSON actions, embedded +TCP / HTTP servers and MCP. The object API (`File`, `Storage`, `Pipeline`, `IntegrityMonitor`) +and the `FA_*` JSON actions are two faces of the same operations, and everything public is +re-exported from the top-level `automation_file` facade. A desktop GUI and a read-only web UI +are included. + +```python +from automation_file import File, IntegrityMonitor, Pipeline, Storage + +File("s3://reports/2026/q1.csv").copy_to("sftp://nas.example/archive/q1.csv") +IntegrityMonitor("s3://reports/2026", baseline="reports.baseline.json").verify() +``` - Local file / directory / ZIP operations with path traversal guard (`safe_join`) - Validated HTTP downloads with SSRF protections, retry, and size / time caps diff --git a/README.zh-CN.md b/README.zh-CN.md index ce959cc..f1e9f60 100644 --- a/README.zh-CN.md +++ b/README.zh-CN.md @@ -2,10 +2,18 @@ [English](README.md) | [繁體中文](README.zh-TW.md) | **简体中文** -一套模块化的自动化框架，涵盖本地文件 / 目录 / ZIP 操作、经 SSRF 验证的 HTTP -下载、远程存储（Google Drive、S3、Azure Blob、Dropbox、SFTP），以及通过内嵌 -TCP / HTTP 服务器执行的 JSON 驱动动作。内附 PySide6 GUI，每个功能都有对应 -页签。所有公开 API 均由顶层 `automation_file` facade 统一导出。 +FileAutomation 是通用的文件层与数据流水线运行环境：以同一套 API 访问本地与远端存储，并提供 +文件完整性监控、具备重试与续跑能力的流水线、调度、事件驱动的通知、审计轨迹，以及通过 JSON +动作、内嵌 TCP / HTTP 服务器与 MCP 进行的自动化。对象 API（`File`、`Storage`、`Pipeline`、 +`IntegrityMonitor`）与 `FA_*` JSON 动作是同一组操作的两种面貌，所有公开名称均由顶层 +`automation_file` facade 统一导出。另附桌面 GUI 与只读的 Web UI。 + +```python +from automation_file import File, IntegrityMonitor, Pipeline, Storage + +File("s3://reports/2026/q1.csv").copy_to("sftp://nas.example/archive/q1.csv") +IntegrityMonitor("s3://reports/2026", baseline="reports.baseline.json").verify() +``` - 本地文件 / 目录 / ZIP 操作，内置路径穿越防护（`safe_join`） - 经 SSRF 验证的 HTTP 下载，支持重试与大小 / 时间上限 diff --git a/README.zh-TW.md b/README.zh-TW.md index 21b302e..4b250dc 100644 --- a/README.zh-TW.md +++ b/README.zh-TW.md @@ -2,10 +2,18 @@ [English](README.md) | **繁體中文** | [简体中文](README.zh-CN.md) -一套模組化的自動化框架，涵蓋本機檔案 / 目錄 / ZIP 操作、經 SSRF 驗證的 HTTP -下載、遠端儲存（Google Drive、S3、Azure Blob、Dropbox、SFTP），以及透過內建 -TCP / HTTP 伺服器執行的 JSON 驅動動作。內附 PySide6 GUI，每個功能都有對應 -分頁。所有公開 API 皆由頂層 `automation_file` facade 統一匯出。 +FileAutomation 是通用的檔案層與資料管線執行環境：以同一套 API 存取本機與遠端儲存，並提供 +檔案完整性監控、具備重試與續跑能力的管線、排程、事件驅動的通知、稽核軌跡，以及透過 JSON +動作、內建 TCP / HTTP 伺服器與 MCP 進行的自動化。物件 API（`File`、`Storage`、`Pipeline`、 +`IntegrityMonitor`）與 `FA_*` JSON 動作是同一組操作的兩種面貌，所有公開名稱皆由頂層 +`automation_file` facade 統一匯出。另附桌面 GUI 與唯讀的 Web UI。 + +```python +from automation_file import File, IntegrityMonitor, Pipeline, Storage + +File("s3://reports/2026/q1.csv").copy_to("sftp://nas.example/archive/q1.csv") +IntegrityMonitor("s3://reports/2026", baseline="reports.baseline.json").verify() +``` - 本機檔案 / 目錄 / ZIP 操作，內建路徑穿越防護（`safe_join`） - 經 SSRF 驗證的 HTTP 下載，支援重試與大小 / 時間上限 diff --git a/dev.toml b/dev.toml index 45e79f0..c3d541e 100644 --- a/dev.toml +++ b/dev.toml @@ -11,7 +11,11 @@ version = "0.0.33" authors = [ { name = "JE-Chen", email = "jechenmailman@gmail.com" }, ] -description = "JSON-driven file, Drive, and cloud automation framework (dev channel)." +description = "Universal file layer and data-pipeline runtime: one API for local and remote storage, integrity monitoring, pipelines, scheduling, notifications, audit and MCP automation (dev channel)." +keywords = [ + "file", "storage", "s3", "azure-blob", "sftp", "webdav", "pipeline", "dag", "scheduler", + "file-integrity", "audit", "automation", "mcp", +] readme = { file = "README.md", content-type = "text/markdown" } requires-python = ">=3.10" license = "MIT" @@ -102,6 +106,8 @@ automation_file_mcp = "automation_file.server.mcp_server:_cli" [project.urls] "Homepage" = "https://github.com/Integration-Automation/FileAutomation" +"Source" = "https://github.com/Integration-Automation/FileAutomation" +"Issues" = "https://github.com/Integration-Automation/FileAutomation/issues" [tool.setuptools.packages] # Only the library is installed. Without `include`, `tests` (it has an `__init__.py`) ships as a top-level package. diff --git a/docs/source/Eng/eng_index.rst b/docs/source/Eng/eng_index.rst index cd02f89..8481a74 100644 --- a/docs/source/Eng/eng_index.rst +++ b/docs/source/Eng/eng_index.rst @@ -2,6 +2,11 @@ automation_file English Manual ================================ +``automation_file`` is a universal file layer and data-pipeline runtime: one API +for local and remote storage, file integrity monitoring, pipelines, scheduling, +notifications, an audit trail, and automation through JSON actions, servers and +MCP. + The English manual is split into chapters that follow a typical reader journey: install → run JSON actions → drive locally → reach remote storage → expose servers → automate at scale. Use the table of contents on the diff --git a/docs/source/Eng/usage/quickstart.rst b/docs/source/Eng/usage/quickstart.rst index a176af5..7b573fa 100644 --- a/docs/source/Eng/usage/quickstart.rst +++ b/docs/source/Eng/usage/quickstart.rst @@ -1,6 +1,43 @@ Quickstart ========== +The object API +-------------- + +Four names carry most programs: ``File`` and ``Storage`` (:doc:`storage`), +``IntegrityMonitor`` (:doc:`integrity`) and ``Pipeline`` (:doc:`pipeline`). + +.. code-block:: python + + from automation_file import File, IntegrityMonitor, Pipeline, Storage + + # One API for every backend: a plain path or a storage URI. + report = File("memory://demo/reports/q1.csv") + report.write(b"region,total\nnorth,42\n") + report.copy_to("/srv/demo/q1.csv") # any backend to any other + [entry.path for entry in Storage("/srv/demo").list_dir()] # ['q1.csv'] + print(File("/srv/demo/q1.csv").checksum()) # sha256:8372… + + # Is the tree still what was approved? + monitor = IntegrityMonitor("/srv/demo", baseline="/srv/demo.baseline.json") + monitor.create_baseline() + monitor.verify().ok # True + + # Steps with dependencies, retries and a recorded history. + pipeline = Pipeline("publish") + pipeline.task("copy", ["FA_storage_copy", {"source": "memory://demo/reports/q1.csv", + "target": "memory://demo/published/q1.csv"}]) + pipeline.task("check", ["FA_storage_exists", {"uri": "memory://demo/published/q1.csv"}], + depends_on=["copy"]) + pipeline.run().status # RunStatus.SUCCEEDED + +Replace ``memory://`` and the local path with ``s3://bucket/key``, +``sftp://host/path`` or any other backend once its extra is installed +(``pip install "automation_file[s3]"``) and its client is initialised. + +The JSON action lists below are the same operations written as data. Use them +from configuration files, the action servers and the command line. + JSON action lists ----------------- diff --git a/docs/source/Zh-CN/usage/quickstart.rst b/docs/source/Zh-CN/usage/quickstart.rst index 357a435..a9521dc 100644 --- a/docs/source/Zh-CN/usage/quickstart.rst +++ b/docs/source/Zh-CN/usage/quickstart.rst @@ -1,6 +1,42 @@ 快速开始 ======== +对象 API +-------- + +大多数程序只需要四个名称：``File`` 与 ``Storage``（:doc:`storage`）、 +``IntegrityMonitor``（:doc:`integrity`）以及 ``Pipeline``（:doc:`pipeline`）。 + +.. code-block:: python + + from automation_file import File, IntegrityMonitor, Pipeline, Storage + + # 每一种后端都用同一套 API：普通路径或存储 URI。 + report = File("memory://demo/reports/q1.csv") + report.write(b"region,total\nnorth,42\n") + report.copy_to("/srv/demo/q1.csv") # 任何后端之间都能复制 + [entry.path for entry in Storage("/srv/demo").list_dir()] # ['q1.csv'] + print(File("/srv/demo/q1.csv").checksum()) # sha256:8372… + + # 这棵目录树是否仍与核准时相同？ + monitor = IntegrityMonitor("/srv/demo", baseline="/srv/demo.baseline.json") + monitor.create_baseline() + monitor.verify().ok # True + + # 具有依赖关系、重试与运行记录的步骤。 + pipeline = Pipeline("publish") + pipeline.task("copy", ["FA_storage_copy", {"source": "memory://demo/reports/q1.csv", + "target": "memory://demo/published/q1.csv"}]) + pipeline.task("check", ["FA_storage_exists", {"uri": "memory://demo/published/q1.csv"}], + depends_on=["copy"]) + pipeline.run().status # RunStatus.SUCCEEDED + +只要安装了对应的 extra（``pip install "automation_file[s3]"``）并初始化其客户端， +就可以把 ``memory://`` 与本地路径换成 ``s3://bucket/key``、``sftp://host/path`` +或任何其他后端。 + +下面的 JSON 动作列表是同一组操作的数据写法，可用于配置文件、动作服务器与命令行。 + JSON 动作列表 ------------- diff --git a/docs/source/Zh-CN/zh_cn_index.rst b/docs/source/Zh-CN/zh_cn_index.rst index e29455a..a058891 100644 --- a/docs/source/Zh-CN/zh_cn_index.rst +++ b/docs/source/Zh-CN/zh_cn_index.rst @@ -2,6 +2,10 @@ automation_file 简体中文文档 ============================ +``automation_file`` 是通用的文件层与数据流水线运行环境：以同一套 API 访问本地与远端 +存储，并提供文件完整性监控、流水线、调度、通知、审计轨迹，以及通过 JSON 动作、 +服务器与 MCP 进行的自动化。 + 简中手册按典型读者旅程拆分为章节：安装 → 执行 JSON 动作 → 操作本地文件 → 串接远端存储 → 对外开服务器 → 规模化自动化。可使用左侧目录，或直接 跳到下方任一章节。 diff --git a/docs/source/Zh-TW/usage/quickstart.rst b/docs/source/Zh-TW/usage/quickstart.rst index 66582bf..bd664da 100644 --- a/docs/source/Zh-TW/usage/quickstart.rst +++ b/docs/source/Zh-TW/usage/quickstart.rst @@ -1,6 +1,42 @@ 快速開始 ======== +物件 API +-------- + +大多數程式只需要四個名稱：``File`` 與 ``Storage``（:doc:`storage`）、 +``IntegrityMonitor``（:doc:`integrity`）以及 ``Pipeline``（:doc:`pipeline`）。 + +.. code-block:: python + + from automation_file import File, IntegrityMonitor, Pipeline, Storage + + # 每一種後端都用同一套 API：一般路徑或儲存 URI。 + report = File("memory://demo/reports/q1.csv") + report.write(b"region,total\nnorth,42\n") + report.copy_to("/srv/demo/q1.csv") # 任何後端之間都能複製 + [entry.path for entry in Storage("/srv/demo").list_dir()] # ['q1.csv'] + print(File("/srv/demo/q1.csv").checksum()) # sha256:8372… + + # 這棵目錄樹是否仍與核可時相同？ + monitor = IntegrityMonitor("/srv/demo", baseline="/srv/demo.baseline.json") + monitor.create_baseline() + monitor.verify().ok # True + + # 具有相依關係、重試與執行紀錄的步驟。 + pipeline = Pipeline("publish") + pipeline.task("copy", ["FA_storage_copy", {"source": "memory://demo/reports/q1.csv", + "target": "memory://demo/published/q1.csv"}]) + pipeline.task("check", ["FA_storage_exists", {"uri": "memory://demo/published/q1.csv"}], + depends_on=["copy"]) + pipeline.run().status # RunStatus.SUCCEEDED + +只要安裝了對應的 extra（``pip install "automation_file[s3]"``）並初始化其用戶端， +就可以把 ``memory://`` 與本機路徑換成 ``s3://bucket/key``、``sftp://host/path`` +或任何其他後端。 + +下面的 JSON 動作清單是同一組操作的資料寫法，可用於設定檔、動作伺服器與命令列。 + JSON 動作清單 ------------- diff --git a/docs/source/Zh-TW/zh_tw_index.rst b/docs/source/Zh-TW/zh_tw_index.rst index 3a0651d..26ca9ba 100644 --- a/docs/source/Zh-TW/zh_tw_index.rst +++ b/docs/source/Zh-TW/zh_tw_index.rst @@ -2,6 +2,10 @@ automation_file 繁體中文文件 ============================ +``automation_file`` 是通用的檔案層與資料管線執行環境：以同一套 API 存取本機與遠端 +儲存，並提供檔案完整性監控、管線、排程、通知、稽核軌跡，以及透過 JSON 動作、 +伺服器與 MCP 進行的自動化。 + 繁中手冊依典型讀者旅程拆分為章節：安裝 → 執行 JSON 動作 → 操作本地檔案 → 串接遠端儲存 → 對外開伺服器 → 規模化自動化。可使用左側目錄，或直接 跳到下方任一章節。 diff --git a/docs/updates/2026-10.md b/docs/updates/2026-10.md index 530af2b..c1d2abb 100644 --- a/docs/updates/2026-10.md +++ b/docs/updates/2026-10.md @@ -605,3 +605,16 @@ Index and query commands: [README.md](README.md). New entries go at the end. - **Docs**: the three `usage/mcp.rst` pages rewritten (setup, the tool table, the permission model, dry run, the roadmap's S3-to-SFTP workflow, the bridge, security, diagnostics), the MCP part of the three `usage/cli.rst`, `docs/source/API/server.rst`, `examples/mcp/`, the feature list and the MCP section of the three READMEs, `architecture.md` §2 and §3, `CLAUDE.md` (package map, § Security › MCP server). - **Files**: `automation_file/server/mcp_{policy,tools,tool_model,file_tools,storage_tools,pipeline_tools,pipeline_actions,report_tools}.py`, `automation_file/server/mcp_server.py`, `automation_file/__main__.py`, `automation_file/__init__.py`, the tests above, the documentation above. - **Open items**: #37. + +## U-20261008-28 · 2026-10-08 · One positioning in the READMEs, the manuals and the metadata · #docs #packaging #roadmap + +- **Why**: roadmap §19 asks that the README, the PyPI metadata and the documentation say the same thing about what the package is. They still described a "JSON-driven file, Drive, and cloud automation framework" with a GUI of tabs. +- **What**: + - The opening of the three READMEs: a universal file layer and data-pipeline runtime, the object API (`File`, `Storage`, `Pipeline`, `IntegrityMonitor`) and the `FA_*` actions as two faces of the same operations, and a two-line example. + - The three quick-start pages begin with "The object API": a file written, copied between backends, listed and hashed, a tree baselined and verified, and a two-task pipeline. The JSON action lists follow, introduced as the same operations written as data. + - The front page of the three manuals opens with the same sentence. + - `stable.toml` and `dev.toml`: the `description`, a `keywords` list, and `Source` and `Issues` project URLs. +- **Verified**: the quick-start example was run as written (with a scratch directory for `/srv/demo`) and printed what its comments say. The metadata parity and release tests pass. +- **Not changed**: the `Development Status :: 2 - Pre-Alpha` classifier. Raising it belongs to the 1.0 release, which is the owner's decision. +- **Files**: the three READMEs, the three `usage/quickstart.rst`, the three manual indexes, `stable.toml`, `dev.toml`. +- **Open items**: none. diff --git a/docs/updates/README.md b/docs/updates/README.md index d3ce33a..f27ec02 100644 --- a/docs/updates/README.md +++ b/docs/updates/README.md @@ -58,6 +58,7 @@ In the same commit: delete the item from `progress.md`, add a `#done` entry here | ID | Date | Title | Tags | Batch | |---|---|---|---|---| +| U-20261008-28 | 2026-10-08 | One positioning in the READMEs, the manuals and the metadata | #docs #packaging #roadmap | [2026-10](2026-10.md) | | U-20261008-27 | 2026-10-08 | Semantic MCP tools | #mcp #security #roadmap #done | [2026-10](2026-10.md) | | U-20261008-26 | 2026-10-08 | Production deployment guide | #docs #roadmap | [2026-10](2026-10.md) | | U-20261008-25 | 2026-10-08 | Metadata cases in the storage contract | #storage #tests #done | [2026-10](2026-10.md) | @@ -122,5 +123,5 @@ In the same commit: delete the item from `progress.md`, add a `#done` entry here | File | Period | Entries | |---|---|---:| -| [2026-10.md](2026-10.md) | 2026-10 | 38 | +| [2026-10.md](2026-10.md) | 2026-10 | 39 | | [2026-09.md](2026-09.md) | 2026-09 | 21 | diff --git a/stable.toml b/stable.toml index 7e3bf47..3424474 100644 --- a/stable.toml +++ b/stable.toml @@ -9,7 +9,11 @@ version = "0.0.31" authors = [ { name = "JE-Chen", email = "jechenmailman@gmail.com" }, ] -description = "JSON-driven file, Drive, and cloud automation framework." +description = "Universal file layer and data-pipeline runtime: one API for local and remote storage, integrity monitoring, pipelines, scheduling, notifications, audit and MCP automation." +keywords = [ + "file", "storage", "s3", "azure-blob", "sftp", "webdav", "pipeline", "dag", "scheduler", + "file-integrity", "audit", "automation", "mcp", +] readme = { file = "README.md", content-type = "text/markdown" } requires-python = ">=3.10" license = "MIT" @@ -100,6 +104,8 @@ automation_file_mcp = "automation_file.server.mcp_server:_cli" [project.urls] "Homepage" = "https://github.com/Integration-Automation/FileAutomation" +"Source" = "https://github.com/Integration-Automation/FileAutomation" +"Issues" = "https://github.com/Integration-Automation/FileAutomation/issues" [tool.setuptools.packages] # Only the library is installed. Without `include`, `tests` (it has an `__init__.py`) ships as a top-level package. From ee437d0855543109a895ac10b4c7a9f292057ae2 Mon Sep 17 00:00:00 2001 From: JeffreyChen Date: Thu, 8 Oct 2026 15:23:33 +0800 Subject: [PATCH 46/59] feat: add scheduler v2 Cron with a time zone, manual, file, event and pipeline triggers; action-list and pipeline targets; a record of every run with seven states; overlap protection, timeouts and cancellation. --- CLAUDE.md | 5 +- README.md | 63 +- README.zh-CN.md | 61 +- README.zh-TW.md | 61 +- architecture.md | 8 +- automation_file/__init__.py | 14 + automation_file/scheduler/__init__.py | 59 +- automation_file/scheduler/cron.py | 42 +- automation_file/scheduler/dispatch.py | 415 ++++++++++++ automation_file/scheduler/errors.py | 9 + automation_file/scheduler/job.py | 96 +++ automation_file/scheduler/manager.py | 658 +++++++++++++++---- automation_file/scheduler/runs.py | 211 ++++++ automation_file/scheduler/targets.py | 188 ++++++ automation_file/scheduler/triggers.py | 405 ++++++++++++ automation_file/trigger/manager.py | 31 +- dev.toml | 2 + docs/source/API/scheduler.rst | 56 +- docs/source/Eng/architecture.rst | 23 +- docs/source/Eng/eng_index.rst | 7 +- docs/source/Eng/usage/events.rst | 40 +- docs/source/Eng/usage/notifications.rst | 5 + docs/source/Eng/usage/pipeline.rst | 4 +- docs/source/Eng/usage/scheduler.rst | 753 ++++++++++++++++++++++ docs/source/Zh-CN/architecture.rst | 20 +- docs/source/Zh-CN/usage/events.rst | 31 +- docs/source/Zh-CN/usage/notifications.rst | 4 + docs/source/Zh-CN/usage/pipeline.rst | 3 +- docs/source/Zh-CN/usage/scheduler.rst | 703 ++++++++++++++++++++ docs/source/Zh-CN/zh_cn_index.rst | 1 + docs/source/Zh-TW/architecture.rst | 20 +- docs/source/Zh-TW/usage/events.rst | 31 +- docs/source/Zh-TW/usage/notifications.rst | 4 + docs/source/Zh-TW/usage/pipeline.rst | 3 +- docs/source/Zh-TW/usage/scheduler.rst | 703 ++++++++++++++++++++ docs/source/Zh-TW/zh_tw_index.rst | 1 + docs/updates/2026-10.md | 23 + docs/updates/README.md | 3 +- progress.md | 1 - stable.toml | 2 + tests/scheduler_kit.py | 133 ++++ tests/test_scheduler_actions.py | 403 ++++++++++++ tests/test_scheduler_lifecycle.py | 203 ++++++ tests/test_scheduler_pipeline.py | 469 ++++++++++++++ tests/test_scheduler_runs.py | 641 ++++++++++++++++++ tests/test_scheduler_triggers.py | 613 ++++++++++++++++++ 46 files changed, 6947 insertions(+), 284 deletions(-) create mode 100644 automation_file/scheduler/dispatch.py create mode 100644 automation_file/scheduler/errors.py create mode 100644 automation_file/scheduler/job.py create mode 100644 automation_file/scheduler/runs.py create mode 100644 automation_file/scheduler/targets.py create mode 100644 automation_file/scheduler/triggers.py create mode 100644 docs/source/Eng/usage/scheduler.rst create mode 100644 docs/source/Zh-CN/usage/scheduler.rst create mode 100644 docs/source/Zh-TW/usage/scheduler.rst create mode 100644 tests/scheduler_kit.py create mode 100644 tests/test_scheduler_actions.py create mode 100644 tests/test_scheduler_lifecycle.py create mode 100644 tests/test_scheduler_pipeline.py create mode 100644 tests/test_scheduler_runs.py create mode 100644 tests/test_scheduler_triggers.py diff --git a/CLAUDE.md b/CLAUDE.md index 978ef76..51602ae 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -50,7 +50,9 @@ automation_file/ ├── audit/ # Audit schema v2: record (AuditRecord), store (AuditStore, AuditQuery, │ # MemoryAuditStore), sqlite_store (SQLiteAuditStore), trail (AuditTrail, │ # audit_trail, configure_audit), actions (FA_audit_*) -├── trigger/, scheduler/, notify/ # watchdog file triggers, cron scheduler, notification sinks; +├── trigger/, scheduler/, notify/ # watchdog file triggers, the scheduler (cron with a time zone, +│ # manual, file, event and pipeline triggers; run records), +│ # notification sinks; │ # notify/router.py routes events to sinks (NotificationRouter); │ # each registers its own FA_* ops ├── project/ # ProjectBuilder, create_project_dir @@ -89,6 +91,7 @@ automation_file/ - `StorageBackend` — the contract a storage backend implements. The public operations (`exists`, `stat`, `list_dir`, `mkdir`, `upload`, `download`, `delete`, `checksum`, `read_bytes`, `write_bytes`, `copy_from`, `move_from`) are template methods; a backend supplies only the `_`-prefixed primitives. Twelve are built in: `LocalStorage`, `MemoryStorage`, `S3Storage` and `AzureStorage` (both on `ObjectStorage`), `SFTPStorage` and `FTPStorage` (both on `SessionStorage`), `GoogleDriveStorage`, `OneDriveStorage`, `DropboxStorage`, and the mounted `WebDAVStorage`, `SMBStorage` and `FsspecStorage`. Each uses its backend's shared client singleton unless given one, and reports a missing SDK with the extra to install. - `IntegrityMonitor` — compares a tree at any storage URI with an approved baseline (`create_baseline`, `verify`, `accept`, `watch`, `start` / `stop`, `snapshot`) and returns a `DriftReport`; drift is published as one `IntegrityViolation` per pass. It only reads unless a `RemediationPolicy` is passed. Its options are keyword arguments (`MonitorKeywords`). The first monitor's call and `check_once()` summary are kept, including the notification through `manager` or the process-wide `notification_manager`. - `Pipeline` — tasks (a callable taking a `TaskContext`, or an `FA_*` action) with `depends_on`, run in dependency order with `RetryPolicy`, a timeout, `when` conditions and idempotency keys. `run` executes in the calling thread, `start` in the background, `resume(run_id)` repeats only what did not succeed, `run(dry_run=True)` plans. Every transition is checkpointed in a `RunStore` (`MemoryRunStore`, `SQLiteRunStore`) and reported as a `pipeline.*` / `task.*` event. A task fails only by raising: an action that reports through its return value needs its raising form (`FA_storage_verify` with `strict=True`). +- `Scheduler` / `scheduler` — runs an action list or a pipeline when one of its triggers fires (`CronTrigger` with an IANA time zone, a manual `run_now`, `FileTrigger`, `EventTrigger`, `PipelineTrigger`). Every firing is a `JobRun` with one of seven states (`scheduled`, `started`, `completed`, `failed`, `skipped`, `timeout`, `cancelled`) in a bounded history. Overlap is refused unless `allow_overlap=True`. A failed or timed-out run publishes one `scheduler.error`. Tests drive it with `tick(now)` and an injected clock; never sleep through a minute. `Scheduler.add(name, cron, action_list, *, allow_overlap=False)` and the four original `FA_schedule_*` actions keep their shape. - `NotificationRouter` / `Route` / `notification_router` — delivers events to named sinks by type, source and minimum severity, with deduplication and a rate limit per route and sink. Opt-in: nothing is routed until a route exists and the router is started (`FA_notify_route_add` and `AutomationConfig.apply_to(manager, router)` start it). While it is active, `notify_on_failure` and the integrity monitor leave the direct notification to it, so nothing is announced twice. - `AuditTrail` / `audit_trail` / `configure_audit(path)` — audit schema v2: one `AuditRecord` per event and per storage operation in an `AuditStore` (`SQLiteAuditStore`, `MemoryAuditStore`), searched with `audit_search` / `FA_audit_search`. Records nothing until configured, and never raises into the code it audits. The v1 `AuditLog` is unchanged. - `Event` / `EventBus` / `event_bus` — every component reports through events (`PipelineFailed`, `TaskFailed`, `IntegrityViolation`, `StorageError`, ...) with a severity, a correlation ID and an actor; consumers subscribe on the bus by class, type name or prefix. New code that has something to report publishes an event; it does not call a notification sink or the audit log directly. diff --git a/README.md b/README.md index aa4b8a8..3f4e7c0 100644 --- a/README.md +++ b/README.md @@ -25,7 +25,7 @@ IntegrityMonitor("s3://reports/2026", baseline="reports.baseline.json").verify() - Loopback-first TCP **and** HTTP servers that accept JSON command batches with optional shared-secret auth - Reliability primitives: `retry_on_transient` decorator, `Quota` size / time budgets - **File-watcher triggers** — run an action list whenever a path changes (`FA_watch_*`) -- **Cron scheduler** — recurring action lists on a stdlib-only 5-field parser (`FA_schedule_*`) +- **Scheduler** — runs an action list or a pipeline when a trigger fires: cron with a time zone, a manual call, a file event, an event on the bus, or the end of another pipeline; every run is recorded with its state, overlap is refused by default, and a job can have a timeout and be cancelled (`FA_schedule_*`) - **Transfer progress + cancellation** — opt-in `progress_name` hook on HTTP and S3 transfers (`FA_progress_*`) - **Fast file search** — OS index fast path (`mdfind` / `locate` / `es.exe`) with a streaming `scandir` fallback (`FA_fast_find`) - **Checksums + integrity verification** — streaming `file_checksum` / `verify_checksum` with any `hashlib` algorithm; `download_file(expected_sha256=...)` verifies after transfer (`FA_file_checksum`, `FA_verify_checksum`) @@ -716,24 +716,59 @@ watch_stop("inbox-sweeper") `FA_watch_start` / `FA_watch_stop` / `FA_watch_stop_all` / `FA_watch_list` surface the same lifecycle to JSON action lists. -### Cron scheduler -Recurring action lists on a stdlib-only 5-field cron parser: +### Scheduler -```python -from automation_file import schedule_add +`automation_file.scheduler` runs an action list or a pipeline when something fires +it: a cron expression with a time zone, a file event, an event on the bus, the end +of another pipeline's run, or a call. Every firing leaves a run record. -schedule_add( - name="nightly-snapshot", - cron_expression="0 2 * * *", # every day at 02:00 local time - action_list=[["FA_zip_dir", {"dir_we_want_to_zip": "/data", - "zip_name": "/backup/data_nightly"}]], +```python +from automation_file.scheduler import PipelineTrigger, scheduler + +scheduler.add( + "nightly-snapshot", + "0 2 * * *", # every day at 02:00 ... + [["FA_zip_dir", {"dir_we_want_to_zip": "/data", + "zip_name": "/backup/data_nightly"}]], + timezone="Asia/Taipei", # ... in Taipei; local time without it + timeout=1800, ) + +# A pipeline that declares `schedule: {cron: "0 2 * * *", timezone: Asia/Taipei}` +scheduler.add_pipeline("pipelines/daily-report.yaml", + params={"date": "${date:%Y-%m-%d}"}, timeout=3600) +# ... and one that runs whenever daily-report has succeeded +scheduler.add_pipeline("pipelines/publish-summary.yaml", + triggers=PipelineTrigger("daily-report")) + +run = scheduler.run_now("nightly-snapshot") # fire by hand +run.wait(600) +scheduler.history(state="failed", limit=10) # the latest failed runs ``` -Supports `*`, exact values, `a-b` ranges, comma lists, and `*/n` step -syntax with `jan..dec` / `sun..sat` aliases. JSON actions: -`FA_schedule_add`, `FA_schedule_remove`, `FA_schedule_remove_all`, -`FA_schedule_list`. +- **Triggers.** `CronTrigger` (5 fields, optional IANA time zone), `FileTrigger` + (a watched path), `EventTrigger` (a type, a prefix or a source on the event bus, + which is also how a webhook that publishes an event fires a job), + `PipelineTrigger` (after another pipeline: `on_success`, `on_failure`, + `always`), and `run_now` for a job fired by hand. A job may have several. +- **Run records.** Every firing is a `JobRun` in one of seven states: + `scheduled`, `started`, `completed`, `failed`, `skipped`, `timeout`, + `cancelled`, with UTC times, the trigger, the error and a correlation ID. + `scheduler.history(job, state, limit)` returns the latest, newest first. +- **Overlap, timeout, cancellation.** A firing that meets a run still in progress + is recorded as `skipped` unless the job has `allow_overlap=True`. A run past its + `timeout` is recorded as `timeout` and told to stop; `scheduler.cancel(name)` + does the same on request. A pipeline stops through its cancellation token, an + action list before its next action. +- **Time zones.** Zone names come from `zoneinfo` (on Windows: `pip install + tzdata`; `UTC` needs nothing). A local time that does not exist on a + daylight-saving day is not fired, and one that occurs twice fires once. +- **Failures are events.** A run that fails or times out is published as + `scheduler.error`; route it to a sink with the notification router. +- **Actions.** `FA_schedule_add`, `FA_schedule_job`, `FA_schedule_pipeline`, + `FA_schedule_run`, `FA_schedule_cancel`, `FA_schedule_history`, + `FA_schedule_list`, `FA_schedule_remove` and `FA_schedule_remove_all` for JSON + action lists, the CLI, the action servers and MCP. ### Transfer progress + cancellation HTTP and S3 transfers accept an opt-in `progress_name` kwarg: diff --git a/README.zh-CN.md b/README.zh-CN.md index f1e9f60..78770d6 100644 --- a/README.zh-CN.md +++ b/README.zh-CN.md @@ -23,7 +23,7 @@ IntegrityMonitor("s3://reports/2026", baseline="reports.baseline.json").verify() - Loopback 优先的 TCP **与** HTTP 服务器，接受 JSON 指令批量并可选 shared-secret 验证 - 可靠性原语：`retry_on_transient` 装饰器、`Quota` 大小 / 时间预算 - **文件监听触发** — 当路径变动时执行动作清单（`FA_watch_*`） -- **Cron 调度器** — 仅用标准库的 5 字段解析器执行周期性动作清单（`FA_schedule_*`） +- **调度器** — 在触发条件成立时运行动作列表或流水线：带时区的 cron、手动调用、文件事件、总线上的事件，或另一条流水线结束；每次运行都会记录状态，默认拒绝重叠运行，作业可以设置超时并可取消（`FA_schedule_*`） - **传输进度 + 取消** — HTTP 与 S3 传输可选的 `progress_name` 钩子（`FA_progress_*`） - **快速文件搜索** — OS 索引快速路径（`mdfind` / `locate` / `es.exe`）搭配流式 `scandir` 回退（`FA_fast_find`） - **校验和 + 完整性验证** — 流式 `file_checksum` / `verify_checksum`，支持任何 `hashlib` 算法；`download_file(expected_sha256=...)` 在下载完成后立即验证（`FA_file_checksum`、`FA_verify_checksum`） @@ -700,23 +700,58 @@ watch_stop("inbox-sweeper") `FA_watch_start` / `FA_watch_stop` / `FA_watch_stop_all` / `FA_watch_list` 让 JSON 动作清单能使用相同的生命周期。 -### Cron 调度器 -以纯标准库的 5 字段 cron 解析器执行周期性动作清单： +### 调度器（Scheduler） -```python -from automation_file import schedule_add +`automation_file.scheduler` 会在某件事触发时执行一份动作列表或一条流水线：带时区的 +cron 表达式、文件事件、事件总线上的事件、另一条流水线的运行结束，或是一次调用。每一次 +触发都会留下一条运行记录。 -schedule_add( - name="nightly-snapshot", - cron_expression="0 2 * * *", # 每天本地时间 02:00 - action_list=[["FA_zip_dir", {"dir_we_want_to_zip": "/data", - "zip_name": "/backup/data_nightly"}]], +```python +from automation_file.scheduler import PipelineTrigger, scheduler + +scheduler.add( + "nightly-snapshot", + "0 2 * * *", # 每天 02:00 ... + [["FA_zip_dir", {"dir_we_want_to_zip": "/data", + "zip_name": "/backup/data_nightly"}]], + timezone="Asia/Taipei", # ... 台北时间；不给就是本地时间 + timeout=1800, ) + +# 声明了 `schedule: {cron: "0 2 * * *", timezone: Asia/Taipei}` 的流水线 +scheduler.add_pipeline("pipelines/daily-report.yaml", + params={"date": "${date:%Y-%m-%d}"}, timeout=3600) +# ... 以及每当 daily-report 成功就运行的流水线 +scheduler.add_pipeline("pipelines/publish-summary.yaml", + triggers=PipelineTrigger("daily-report")) + +run = scheduler.run_now("nightly-snapshot") # 手动触发 +run.wait(600) +scheduler.history(state="failed", limit=10) # 最近失败的运行 ``` -支持 `*`、确切值、`a-b` 范围、逗号列表、`*/n` 步进语法，以及 `jan..dec` / -`sun..sat` 别名。JSON 动作：`FA_schedule_add`、`FA_schedule_remove`、 -`FA_schedule_remove_all`、`FA_schedule_list`。 +- **触发器。** `CronTrigger`（5 个字段，可选的 IANA 时区）、`FileTrigger`（被监听的 + 路径）、`EventTrigger`（事件总线上的 type、前缀或来源；发布事件的 webhook 也是 + 这样触发作业的）、`PipelineTrigger`（在另一条流水线之后：`on_success`、 + `on_failure`、`always`），以及用来手动触发作业的 `run_now`。一个作业可以有好几个 + 触发器。 +- **运行记录。** 每一次触发都是一个 `JobRun`，状态是七种之一：`scheduled`、 + `started`、`completed`、`failed`、`skipped`、`timeout`、`cancelled`，并带有 UTC + 时间、触发器、错误与关联 ID。`scheduler.history(job, state, limit)` 返回最近的 + 记录，最新的在前。 +- **重叠、超时、取消。** 遇到仍在进行中的运行时，触发会记录为 `skipped`，除非作业 + 设置了 `allow_overlap=True`。超过 `timeout` 的运行会记录为 `timeout` 并被要求 + 停止；`scheduler.cancel(name)` 则是按要求这么做。流水线通过它的取消令牌停止，动作 + 列表则在下一个动作之前停止。 +- **时区。** 时区名称来自 `zoneinfo`（Windows 上请 `pip install tzdata`；`UTC` 不 + 需要任何东西）。在夏令时切换的日子，不存在的本地时间不会触发，出现两次的本地时间 + 只触发一次。 +- **失败就是事件。** 失败或超时的运行会发布成 `scheduler.error`；请用通知路由器把 + 它送到 sink。 +- **动作。** `FA_schedule_add`、`FA_schedule_job`、`FA_schedule_pipeline`、 + `FA_schedule_run`、`FA_schedule_cancel`、`FA_schedule_history`、 + `FA_schedule_list`、`FA_schedule_remove` 与 `FA_schedule_remove_all`，可用于 JSON + 动作列表、CLI、动作服务器与 MCP。 ### 传输进度 + 取消 HTTP 与 S3 传输支持可选的 `progress_name` 关键字参数： diff --git a/README.zh-TW.md b/README.zh-TW.md index 4b250dc..da6c8de 100644 --- a/README.zh-TW.md +++ b/README.zh-TW.md @@ -23,7 +23,7 @@ IntegrityMonitor("s3://reports/2026", baseline="reports.baseline.json").verify() - Loopback 優先的 TCP **與** HTTP 伺服器，接受 JSON 指令批次並可選 shared-secret 驗證 - 可靠性原語：`retry_on_transient` 裝飾器、`Quota` 大小 / 時間預算 - **檔案監看觸發** — 當路徑變動時執行動作清單（`FA_watch_*`） -- **Cron 排程器** — 僅用標準函式庫的 5 欄位解析器執行週期性動作清單（`FA_schedule_*`） +- **排程器** — 在觸發條件成立時執行動作清單或管線：帶時區的 cron、手動呼叫、檔案事件、匯流排上的事件，或另一條管線結束；每次執行都會記錄狀態，預設拒絕重疊執行，工作可設定逾時並可取消（`FA_schedule_*`） - **傳輸進度 + 取消** — HTTP 與 S3 傳輸可選的 `progress_name` 掛鉤（`FA_progress_*`） - **快速檔案搜尋** — OS 索引快速路徑（`mdfind` / `locate` / `es.exe`）搭配串流式 `scandir` 備援（`FA_fast_find`） - **檢查碼 + 完整性驗證** — 串流式 `file_checksum` / `verify_checksum`，支援任何 `hashlib` 演算法；`download_file(expected_sha256=...)` 於下載完成後立即驗證（`FA_file_checksum`、`FA_verify_checksum`） @@ -700,23 +700,58 @@ watch_stop("inbox-sweeper") `FA_watch_start` / `FA_watch_stop` / `FA_watch_stop_all` / `FA_watch_list` 讓 JSON 動作清單能使用相同的生命週期。 -### Cron 排程器 -以純標準函式庫的 5 欄位 cron 解析器執行週期性動作清單： +### 排程器（Scheduler） -```python -from automation_file import schedule_add +`automation_file.scheduler` 會在某件事觸發時執行一份動作清單或一條管線：帶時區的 cron +運算式、檔案事件、事件匯流排上的事件、另一條管線的執行結束，或是一次呼叫。每一次觸發 +都會留下一筆執行紀錄。 -schedule_add( - name="nightly-snapshot", - cron_expression="0 2 * * *", # 每天本地時間 02:00 - action_list=[["FA_zip_dir", {"dir_we_want_to_zip": "/data", - "zip_name": "/backup/data_nightly"}]], +```python +from automation_file.scheduler import PipelineTrigger, scheduler + +scheduler.add( + "nightly-snapshot", + "0 2 * * *", # 每天 02:00 ... + [["FA_zip_dir", {"dir_we_want_to_zip": "/data", + "zip_name": "/backup/data_nightly"}]], + timezone="Asia/Taipei", # ... 台北時間；不給就是本地時間 + timeout=1800, ) + +# 宣告了 `schedule: {cron: "0 2 * * *", timezone: Asia/Taipei}` 的管線 +scheduler.add_pipeline("pipelines/daily-report.yaml", + params={"date": "${date:%Y-%m-%d}"}, timeout=3600) +# ... 以及每當 daily-report 成功就執行的管線 +scheduler.add_pipeline("pipelines/publish-summary.yaml", + triggers=PipelineTrigger("daily-report")) + +run = scheduler.run_now("nightly-snapshot") # 手動觸發 +run.wait(600) +scheduler.history(state="failed", limit=10) # 最近失敗的執行 ``` -支援 `*`、確切值、`a-b` 範圍、逗號清單、`*/n` 步進語法，以及 `jan..dec` / -`sun..sat` 別名。JSON 動作：`FA_schedule_add`、`FA_schedule_remove`、 -`FA_schedule_remove_all`、`FA_schedule_list`。 +- **觸發器。** `CronTrigger`（5 個欄位，可選的 IANA 時區）、`FileTrigger`（被監看的 + 路徑）、`EventTrigger`（事件匯流排上的 type、前綴或來源；發布事件的 webhook 也是 + 這樣觸發工作的）、`PipelineTrigger`（在另一條管線之後：`on_success`、 + `on_failure`、`always`），以及用來手動觸發工作的 `run_now`。一個工作可以有好幾個 + 觸發器。 +- **執行紀錄。** 每一次觸發都是一個 `JobRun`，狀態是七種之一：`scheduled`、 + `started`、`completed`、`failed`、`skipped`、`timeout`、`cancelled`，並帶有 UTC + 時間、觸發器、錯誤與關聯 ID。`scheduler.history(job, state, limit)` 回傳最近的 + 紀錄，最新的在前。 +- **重疊、逾時、取消。** 遇到仍在進行中的執行時，觸發會記錄為 `skipped`，除非工作 + 設定了 `allow_overlap=True`。超過 `timeout` 的執行會記錄為 `timeout` 並被要求 + 停止；`scheduler.cancel(name)` 則是應要求這麼做。管線透過它的取消權杖停止，動作 + 清單則在下一個動作之前停止。 +- **時區。** 時區名稱來自 `zoneinfo`（Windows 上請 `pip install tzdata`；`UTC` 不 + 需要任何東西）。在日光節約時間切換的日子，不存在的本地時間不會觸發，出現兩次的 + 本地時間只觸發一次。 +- **失敗就是事件。** 失敗或逾時的執行會發布成 `scheduler.error`；請用通知路由器把 + 它送到 sink。 +- **動作。** `FA_schedule_add`、`FA_schedule_job`、`FA_schedule_pipeline`、 + `FA_schedule_run`、`FA_schedule_cancel`、`FA_schedule_history`、 + `FA_schedule_list`、`FA_schedule_remove` 與 `FA_schedule_remove_all`，可用於 JSON + 動作清單、CLI、動作伺服器與 MCP。 ### 傳輸進度 + 取消 HTTP 與 S3 傳輸支援可選的 `progress_name` 關鍵字參數： diff --git a/architecture.md b/architecture.md index e6d8a62..8151a88 100644 --- a/architecture.md +++ b/architecture.md @@ -29,7 +29,7 @@ piece of the 1.0 roadmap (`docs/FILEAUTOMATION-1.0-ROADMAP.md`, PR #107); what i | `automation_file/integrity/` | IntegrityMonitor 2.0, on the storage layer and the event bus. `target.py` (`Target`: the monitored tree behind a storage URI), `hashing.py` (`HashEngine`; `md5` and `sha1` only with `allow_weak`), `snapshot.py` (`Snapshot`, `SnapshotEntry`, `build_snapshot`), `manifest.py` (schema version 2; the `write_manifest` format is read and converted), `baseline.py` (`BaselineManager`: an atomic write at any storage URI), `detector.py` (`Change`, `ChangeKind`, `detect_changes`: six kinds of change), `report.py` (`DriftReport`), `alerts.py` (`AlertEngine`, `AlertPolicy`: one `IntegrityViolation` per pass that finds drift), `remediation.py` (`RemediationPolicy`, `Remediator`: quarantine or restore, opt-in), `watcher.py` and `local_watcher.py` (polling, and watchdog events for a local target), `legacy.py` (the first monitor's summary, callback and notification), `monitor.py` (`IntegrityMonitor`), `actions.py` (`FA_integrity_*`). `core/fim.py` re-exports the class | | `automation_file/server/` | `tcp_server.py`, `http_server.py`, `mcp_server.py` (JSON-RPC over stdio: the semantic tools first, then the `FA_*` bridge), `mcp_policy.py` (`MCPPolicy`, `StorageGuard`: roots, read-only by default, limits), `mcp_tools.py` and `mcp_*_tools.py` (`SemanticToolkit` and the fourteen tools), `mcp_pipeline_actions.py` (the guarded `FA_storage_*` set pipelines made through MCP run), `web_ui.py`, `metrics_server.py`, `action_acl.py` (`ActionACL`), `network_guards.py` (`ensure_loopback`) | | `automation_file/client/` | `HTTPActionClient` for the HTTP action server | -| `automation_file/trigger/`, `scheduler/`, `notify/` | Watchdog file triggers, cron scheduler, notification sinks. Each registers its own `FA_*` ops. `notify/router.py` (`Route`, `NotificationRouter`, the process-wide `notification_router`) subscribes on the event bus and delivers events to named sinks by type, source and minimum severity, with deduplication and a rate limit per route and sink; a failing sink becomes a `system.error` event from the source `notify`, which is never routed | +| `automation_file/trigger/`, `scheduler/`, `notify/` | Watchdog file triggers, the scheduler, notification sinks. Each registers its own `FA_*` ops. `scheduler/`: `cron.py` (`CronExpression`), `triggers.py` (`CronTrigger` with a time zone, `FileTrigger`, `EventTrigger`, `PipelineTrigger`), `job.py`, `targets.py` (an action list or a pipeline), `runs.py` (`JobRun`, `RunState`, a bounded `RunHistory`), `dispatch.py`, `manager.py` (`Scheduler`, the process-wide `scheduler`; `tick(now)` drives it in tests). `notify/router.py` (`Route`, `NotificationRouter`, the process-wide `notification_router`) subscribes on the event bus and delivers events to named sinks by type, source and minimum severity, with deduplication and a rate limit per route and sink; a failing sink becomes a `system.error` event from the source `notify`, which is never routed | | `automation_file/pipeline/` | The pipeline runtime. `model.py` (`Task`, `TaskContext`, `RetryPolicy`, `Schedule`, `PipelineRun`, `TaskRun`, `RunStatus`, `TaskStatus`), `graph.py` (dependency order, cycles), `pipeline.py` (`Pipeline`: `task`, `run`, `start`, `resume`, `from_file` / `from_dict` / `to_dict`, `problems` / `validate`), `runner.py` and `worker.py` (one daemon thread per running task, capped at `max_workers`; retry, timeout, cancellation, conditions, idempotency), `substitution.py` (`${params.x}`, `${tasks.id.result}`), `store.py` (`RunStore`, `MemoryRunStore`, `SQLiteRunStore`: checkpoints and history), `definition.py` (`load_definition`, `validate_definition`, `PIPELINE_SCHEMA`), `reporting.py` (the `pipeline.*` and `task.*` events), `actions.py` (`FA_pipeline_*`). `core/dag_executor.py` is the older, unrecorded DAG helper and is unchanged | | `automation_file/audit/` | Audit schema v2. `record.py` (`AuditRecord`, built from an event or from a storage operation), `store.py` (`AuditStore`, `AuditQuery`, `MemoryAuditStore`), `sqlite_store.py` (`SQLiteAuditStore`: parameterised SQL, a schema-version table, `import_v1`), `trail.py` (`AuditTrail`, the process-wide `audit_trail`, `configure_audit`), `actions.py` (`FA_audit_*`). The trail records nothing until it is configured; the v1 `core/audit.py` `AuditLog` is unchanged | | `automation_file/project/` | `ProjectBuilder`, `create_project_dir` | @@ -88,6 +88,12 @@ piece of the 1.0 roadmap (`docs/FILEAUTOMATION-1.0-ROADMAP.md`, PR #107); what i `MCPPolicy` and `SemanticToolkit` are on the facade; flags: `--root`, `--allow-write`, `--allow-overwrite`, `--allow-delete`, `--max-read-bytes`, `--max-write-bytes`, `--max-results`, `--max-search-bytes`, `--pipeline-dir`, `--pipeline-actions`, `--tools`, `--no-bridge`. +- **Scheduler** (same facade): `Scheduler`, `scheduler`, `CronExpression`, `CronTrigger`, `FileTrigger`, + `EventTrigger`, `PipelineTrigger`, `JobRun`, `RunState`, `ScheduledJob`, `SchedulerException`, and + `schedule_add` / `_remove` / `_remove_all` / `_list`. Actions: `FA_schedule_add`, `_remove`, + `_remove_all`, `_list` (unchanged), and `FA_schedule_job`, `FA_schedule_pipeline`, `FA_schedule_run`, + `FA_schedule_history`, `FA_schedule_cancel`. The keys of `ScheduledJob.as_dict()` are relied on by + the GUI and by action lists. - **Events** (same facade): `Event`, `Severity`, `EventBus`, `event_bus`, `emit`, `correlation_scope`, `actor_scope`, and the core events `PipelineStarted`, `PipelineCompleted`, `PipelineFailed`, `TaskStarted`, `TaskCompleted`, `TaskFailed`, `IntegrityViolation`, `StorageError`, `SchedulerError`, diff --git a/automation_file/__init__.py b/automation_file/__init__.py index 907a035..2a14ca8 100644 --- a/automation_file/__init__.py +++ b/automation_file/__init__.py @@ -317,8 +317,15 @@ from automation_file.remote.webdav import WebDAVClient, WebDAVEntry from automation_file.scheduler import ( CronExpression, + CronTrigger, + EventTrigger, + FileTrigger, + JobRun, + PipelineTrigger, + RunState, ScheduledJob, Scheduler, + SchedulerException, register_scheduler_ops, schedule_add, schedule_list, @@ -711,8 +718,15 @@ def __getattr__(name: str) -> Any: "watch_list", # Scheduler "CronExpression", + "CronTrigger", + "EventTrigger", + "FileTrigger", + "PipelineTrigger", + "JobRun", + "RunState", "ScheduledJob", "Scheduler", + "SchedulerException", "register_scheduler_ops", "schedule_add", "schedule_list", diff --git a/automation_file/scheduler/__init__.py b/automation_file/scheduler/__init__.py index ec2d481..886de2c 100644 --- a/automation_file/scheduler/__init__.py +++ b/automation_file/scheduler/__init__.py @@ -1,35 +1,78 @@ -"""Cron-style scheduler — run action lists on a recurring schedule. +"""The scheduler: run action lists and pipelines on a schedule, on an event, or by hand. -A :class:`ScheduledJob` pairs a 5-field cron expression (minute hour dom month -dow) with a JSON action list. The module-level :data:`scheduler` owns a -background thread that wakes every second, checks which jobs are due, and -dispatches their action lists through the shared -:class:`~automation_file.core.action_executor.ActionExecutor`. +A :class:`ScheduledJob` pairs a target (a JSON action list or a +:class:`~automation_file.pipeline.Pipeline`) with triggers: + +* :class:`CronTrigger`: a 5-field cron expression (minute hour dom month dow) + with an optional IANA time zone; +* :class:`FileTrigger`: a file event under a watched path; +* :class:`EventTrigger`: an event on the bus, which is also how a webhook that + publishes one fires a job; +* :class:`PipelineTrigger`: the end of another pipeline's run; +* no trigger at all: the job is fired by hand (``Scheduler.run_now``). + +Every firing leaves a :class:`JobRun` in one of the :class:`RunState` values +``scheduled``, ``started``, ``completed``, ``failed``, ``skipped``, ``timeout`` +and ``cancelled``. The module-level :data:`scheduler` owns a background thread +that wakes every second, fires the cron jobs that are due and ends the runs +whose timeout has passed. """ from __future__ import annotations -from automation_file.scheduler.cron import CronException, CronExpression +from automation_file.scheduler.cron import CronException, CronExpression, resolve_timezone +from automation_file.scheduler.errors import SchedulerException +from automation_file.scheduler.job import ScheduledJob from automation_file.scheduler.manager import ( - ScheduledJob, Scheduler, register_scheduler_ops, schedule_add, + schedule_cancel, + schedule_history, + schedule_job, schedule_list, + schedule_pipeline, schedule_remove, schedule_remove_all, + schedule_run, scheduler, ) +from automation_file.scheduler.runs import JobRun, RunHistory, RunState, TriggerKind +from automation_file.scheduler.triggers import ( + CronTrigger, + EventTrigger, + FileTrigger, + PipelineTrigger, + Trigger, + trigger_from_dict, +) __all__ = [ "CronException", "CronExpression", + "CronTrigger", + "EventTrigger", + "FileTrigger", + "JobRun", + "PipelineTrigger", + "RunHistory", + "RunState", "ScheduledJob", "Scheduler", + "SchedulerException", + "Trigger", + "TriggerKind", "register_scheduler_ops", + "resolve_timezone", "schedule_add", + "schedule_cancel", + "schedule_history", + "schedule_job", "schedule_list", + "schedule_pipeline", "schedule_remove", "schedule_remove_all", + "schedule_run", "scheduler", + "trigger_from_dict", ] diff --git a/automation_file/scheduler/cron.py b/automation_file/scheduler/cron.py index 08b0b5d..0fcb2d4 100644 --- a/automation_file/scheduler/cron.py +++ b/automation_file/scheduler/cron.py @@ -7,18 +7,53 @@ Explicitly *not* supported: ``@yearly`` / ``@reboot`` aliases, ``L``/``W`` modifiers, seconds. Callers needing that should use a dedicated cron library. + +An expression carries no time zone: :meth:`CronExpression.matches` compares the +fields of the moment it is given. :func:`resolve_timezone` turns an IANA name +into the ``tzinfo`` a caller converts the moment with. """ from __future__ import annotations import datetime as dt +import zoneinfo from dataclasses import dataclass from automation_file.exceptions import FileAutomationException +_UTC = "UTC" +_HOURS_PER_DAY = 24 + class CronException(FileAutomationException): - """Raised when a cron expression cannot be parsed.""" + """Raised when a cron expression or its time zone cannot be understood.""" + + +def resolve_timezone(name: str | None) -> dt.tzinfo | None: + """Return the ``tzinfo`` of the IANA time zone ``name``; ``None`` stays ``None``. + + ``"UTC"`` needs no zone data. Any other name is looked up with + :mod:`zoneinfo`, which on Windows reads the ``tzdata`` package: a name that + cannot be found raises :class:`CronException` saying so. + """ + if name is None: + return None + if not isinstance(name, str) or not name.strip(): + raise CronException( + f"cron: a time zone is an IANA name such as 'Asia/Taipei', got {name!r}" + ) + key = name.strip() + if key.upper() == _UTC: + return dt.timezone.utc + try: + return zoneinfo.ZoneInfo(key) + except zoneinfo.ZoneInfoNotFoundError as error: + raise CronException( + f"cron: unknown time zone {key!r}: no IANA zone data was found for it " + "(on Windows, install the 'tzdata' package)" + ) from error + except (ValueError, OSError) as error: + raise CronException(f"cron: {key!r} is not a time zone name") from error _FIELD_BOUNDS = ( @@ -140,6 +175,11 @@ def parse(cls, expression: str) -> CronExpression: weekdays = _parse_field(fields[4], 4) return cls(minutes, hours, days, months, weekdays, expression.strip()) + @property + def every_hour(self) -> bool: + """Whether the hour field leaves no hour of the day out (``*`` or its equal).""" + return len(self.hours) == _HOURS_PER_DAY + def matches(self, moment: dt.datetime) -> bool: """Return ``True`` when ``moment`` satisfies every field.""" weekday = moment.isoweekday() % 7 # Monday=1..Sunday=7 -> Monday=1..Sunday=0 diff --git a/automation_file/scheduler/dispatch.py b/automation_file/scheduler/dispatch.py new file mode 100644 index 0000000..7d49ec2 --- /dev/null +++ b/automation_file/scheduler/dispatch.py @@ -0,0 +1,415 @@ +"""Running jobs: admit a firing or skip it, follow the run to its end, record every step. + +A firing that is admitted gets a thread of its own. The record moves from +``scheduled`` to ``started`` and then to ``completed`` or ``failed``; a firing +that meets a run still in progress is recorded as ``skipped`` instead. + +Python cannot stop a thread. When a run's timeout passes or it is cancelled, the +record is closed at that moment (``timeout`` / ``cancelled``) and the run's +cancellation token is set: a pipeline stops through it, and an action list stops +before its next action. The job keeps counting as running until the thread has +really ended, so overlap protection holds in between. + +A run that fails or times out is published as a ``scheduler.error`` event. + +Runs may fire one another through events. Every run knows how many runs led to +it, and a firing at the end of a chain of :data:`MAX_CHAIN` runs is recorded as +``skipped``, so two jobs that fire each other cannot go on for ever. +""" + +from __future__ import annotations + +import threading +from collections.abc import Callable, Mapping +from contextlib import AbstractContextManager +from dataclasses import dataclass, field +from datetime import datetime, timedelta +from typing import Any + +from automation_file.core.progress import CancellationToken +from automation_file.events import ( + Event, + EventBus, + SchedulerError, + actor_scope, + correlation_scope, + new_correlation_id, +) +from automation_file.logging_config import file_automation_logger +from automation_file.scheduler.job import ScheduledJob +from automation_file.scheduler.runs import ( + REASON_CANCELLED, + REASON_CHAIN, + REASON_OVERLAP, + JobRun, + RunHistory, + RunState, + TriggerKind, + as_utc, +) +from automation_file.scheduler.targets import Outcome, describe, fill_params, run_pipeline + +ACTOR = "scheduler" +SOURCE = "scheduler" +#: How many runs may fire one another, each through an event of the one before. +MAX_CHAIN = 16 +_SUBJECTS = {RunState.FAILED: "failed", RunState.TIMEOUT: "timed out"} + +Clock = Callable[[], datetime] +ActionRunner = Callable[[str, list[list[Any]], CancellationToken], Outcome] + + +@dataclass(frozen=True) +class Firing: + """Why a job is fired: what fired it, the time it was due, and the details for the record. + + ``scheduled`` is the time the run was due (the minute, for cron); ``now`` is + when the firing was decided, which is where a timeout starts counting. + ``depth`` counts the runs that led to this firing through events: ``0`` for a + firing no run caused. + """ + + kind: TriggerKind + scheduled: datetime + now: datetime | None = None + detail: Mapping[str, Any] = field(default_factory=dict) + depth: int = 0 + + +@dataclass +class Flight: + """A run in the air: its job, its record, how to stop it, and when its time is up.""" + + job: ScheduledJob + record: JobRun + cancel: CancellationToken = field(default_factory=CancellationToken) + deadline: datetime | None = None + depth: int = 0 + + +@dataclass(frozen=True) +class Services: + """What the dispatcher works with. + + ``lock`` is the scheduler's own lock: the job counters, the flights and the + records change under it together. + """ + + lock: AbstractContextManager[Any] + bus: EventBus + clock: Clock + history: RunHistory + run_actions: ActionRunner + + +class Dispatcher: + """Starts runs, ends them, and keeps the overlap accounting of every job.""" + + def __init__(self, services: Services) -> None: + self._lock = services.lock + self._bus = services.bus + self._clock = services.clock + self._history = services.history + self._run_actions = services.run_actions + self._flights: dict[str, Flight] = {} + self._local = threading.local() + + def _now(self) -> datetime: + return as_utc(self._clock()) + + # ------------------------------------------------------------------ starting + + def dispatch(self, job: ScheduledJob, moment: datetime, firing: Firing) -> JobRun: + """Fire ``job``: start a run on a thread of its own, or record the firing as skipped. + + ``moment`` is the firing time as the job reads its clock; it becomes the + job's ``last_run``. + """ + record = JobRun( + run_id=new_correlation_id(), + job=job.name, + trigger=firing.kind, + scheduled_at=as_utc(firing.scheduled), + target=job.target, + pipeline=None if job.pipeline is None else job.pipeline.name, + detail=dict(firing.detail), + ) + flight = self._admit(job, record, moment, firing) + if flight is None: + return record + file_automation_logger.info( + "scheduler[%s]: firing at %s (run %s, trigger=%s)", + job.name, + moment.isoformat(), + record.run_id, + firing.kind.value, + ) + worker = threading.Thread( + target=self._fly, args=(flight,), name=f"fa-scheduler-{job.name}", daemon=True + ) + try: + worker.start() + except RuntimeError as error: + # No thread could be started: the run ends here, and the job is released. + self._land(flight, Outcome(RunState.FAILED, describe(error))) + return record + + def _admit( + self, job: ScheduledJob, record: JobRun, moment: datetime, firing: Firing + ) -> Flight | None: + """Count the firing as a run, or as skipped when it overlaps or ends a long chain.""" + flight: Flight | None = None + deadline = self._deadline(job, firing) + with self._lock: + self._history.add(record) + reason = self._refusal(job, firing) + if reason is None: + flight = Flight(job, record, deadline=deadline, depth=firing.depth) + self._flights[record.run_id] = flight + job.running = True + job.active += 1 + job.last_run = moment + job.runs += 1 + else: + job.skipped += 1 + record.close(RunState.SKIPPED, record.scheduled_at, reason=reason) + skipped = job.skipped + if reason == REASON_OVERLAP: + file_automation_logger.warning( + "scheduler[%s]: previous run still active — skipping (skipped=%d)", + job.name, + skipped, + ) + elif reason is not None: + file_automation_logger.warning( + "scheduler[%s]: %d runs have fired one another — skipping (skipped=%d)", + job.name, + firing.depth, + skipped, + ) + if flight is None: + record.settle() + return flight + + @staticmethod + def _refusal(job: ScheduledJob, firing: Firing) -> str | None: + """Return why the firing starts no run, ``None`` when it does.""" + if firing.depth >= MAX_CHAIN: + return REASON_CHAIN + if job.running and not job.allow_overlap: + return REASON_OVERLAP + return None + + @staticmethod + def _deadline(job: ScheduledJob, firing: Firing) -> datetime | None: + if job.timeout is None: + return None + return as_utc(firing.now or firing.scheduled) + timedelta(seconds=job.timeout) + + # ------------------------------------------------------------------ the run + + def _fly(self, flight: Flight) -> None: + """The run's thread: whatever ends it, the record is closed and the job released.""" + self._local.flight = flight + outcome = Outcome(RunState.FAILED, "the run's thread ended without a result") + try: + outcome = self._attempt(flight) + finally: + self._land(flight, outcome) + + def _take_off(self, flight: Flight) -> bool: + """Mark the run started; ``False`` when it was already cancelled or out of time.""" + with self._lock: + if flight.record.state.is_final: + return False + flight.record.started_at = self._now() + flight.record.state = RunState.STARTED + return True + + def _attempt(self, flight: Flight) -> Outcome: + job, record = flight.job, flight.record + if not self._take_off(flight): + return Outcome(record.state) + try: + with actor_scope(ACTOR), correlation_scope(record.run_id): + if job.pipeline is None: + return self._run_actions(job.name, job.action_list, flight.cancel) + params = fill_params(job.params, job.local(record.scheduled_at)) + return run_pipeline(job.pipeline, params, record, flight.cancel, self._bus) + except Exception as error: # pylint: disable=broad-except + # Boundary of the run's thread: whatever the target raised, the run + # is recorded and the job is released. + problem = describe(error) + file_automation_logger.error( + "scheduler[%s]: run %s raised %s", job.name, record.run_id, problem + ) + return Outcome(RunState.FAILED, problem) + + def _land(self, flight: Flight, outcome: Outcome) -> None: + """The run's thread has ended: close the record, report, release the job. + + The job is released before the failure is published, so a subscriber + may fire it again at once; the job is released whatever closing does. + """ + decided = True + try: + decided = self._conclude(flight, outcome) + finally: + self._release(flight) + try: + self._tell(flight, outcome, decided) + finally: + flight.record.settle() + + def _release(self, flight: Flight) -> None: + with self._lock: + self._flights.pop(flight.record.run_id, None) + flight.job.active = max(flight.job.active - 1, 0) + flight.job.running = flight.job.active > 0 + + def _conclude(self, flight: Flight, outcome: Outcome) -> bool: + """Close the record unless a timeout or a cancellation did; return whether one had.""" + with self._lock: + decided = flight.record.state.is_final + if not decided: + flight.record.close(outcome.state, self._now(), outcome.error) + flight.job.last_state = outcome.state.value + return decided + + def _tell(self, flight: Flight, outcome: Outcome, decided: bool) -> None: + job, record = flight.job, flight.record + if decided: + file_automation_logger.info( + "scheduler[%s]: run %s ended after it was recorded as %s", + job.name, + record.run_id, + record.state.value, + ) + else: + self._announce(record, outcome.reported) + + # ------------------------------------------------------------------ reporting + + def _announce(self, record: JobRun, reported: bool = False) -> None: + """Log how the run ended and publish ``scheduler.error`` for a failure or a timeout.""" + if record.state is RunState.COMPLETED: + file_automation_logger.info( + "scheduler[%s]: run %s completed", record.job, record.run_id + ) + return + file_automation_logger.warning( + "scheduler[%s]: run %s %s: %s", + record.job, + record.run_id, + record.state.value, + record.error or record.reason, + ) + subject = _SUBJECTS.get(record.state) + if subject is None or reported: + return + payload: dict[str, Any] = { + "job": record.job, + "trigger": record.trigger.value, + "status": record.state.value, + "error": record.error, + "target": record.target, + "scheduler_run_id": record.run_id, + } + if record.duration_ms is not None: + payload["duration_ms"] = record.duration_ms + if record.pipeline is not None: + payload["pipeline"] = record.pipeline + if record.correlation_id != record.run_id: + payload["run_id"] = record.correlation_id # the pipeline's run, once it had started + with actor_scope(ACTOR), correlation_scope(record.correlation_id): + self._bus.publish( + SchedulerError( + source=SOURCE, subject=f"scheduler[{record.job}] {subject}", payload=payload + ) + ) + + # ------------------------------------------------------------------ stopping + + def expire(self, instant: datetime) -> list[JobRun]: + """Close every run whose timeout has passed at ``instant`` and return those records.""" + with self._lock: + late = [ + flight + for flight in self._flights.values() + if flight.deadline is not None + and instant >= flight.deadline + and not flight.record.state.is_final + ] + for flight in late: + flight.record.close( + RunState.TIMEOUT, + instant, + f"TimeoutError: not finished within {flight.job.timeout:g} s", + ) + flight.job.last_state = RunState.TIMEOUT.value + for flight in late: + flight.cancel.cancel() + for flight in late: + # The report belongs to this run: a run it fires is the next link of its chain. + self._local.flight = flight + try: + self._announce(flight.record) + finally: + self._local.flight = None + flight.record.settle() + return [flight.record for flight in late] + + def cancel(self, name: str | None = None) -> list[JobRun]: + """Cancel the runs in progress of the job ``name`` (of every job when ``None``).""" + with self._lock: + chosen = [ + flight + for flight in self._flights.values() + if (name is None or flight.job.name == name) and not flight.record.state.is_final + ] + moment = self._now() + for flight in chosen: + flight.record.close(RunState.CANCELLED, moment, reason=REASON_CANCELLED) + flight.job.last_state = RunState.CANCELLED.value + for flight in chosen: + flight.cancel.cancel() + self._announce(flight.record) + flight.record.settle() + return [flight.record for flight in chosen] + + def is_own(self, name: str, event: Event) -> bool: + """Return whether ``event`` was published by a run of the job ``name`` itself. + + That is an event published on the thread of one of its runs, one that + carries the correlation ID of a run in progress, or a + ``scheduler.error`` about the job. + """ + if isinstance(event, SchedulerError) and event.payload.get("job") == name: + return True + return any(flight.job.name == name for flight in self._publishers(event)) + + def depth_after(self, event: Event) -> int: + """Return the chain depth of a run that ``event`` fires. + + That is one more than the depth of the run that published ``event``, and + ``0`` when no run in progress did. + """ + return max((flight.depth + 1 for flight in self._publishers(event)), default=0) + + def _publishers(self, event: Event) -> list[Flight]: + """Return the runs in progress that published ``event``. + + A run published it when it is published on the run's own thread, or + carries the run's correlation ID (the events of a pipeline's tasks, and + a timeout reported by the scheduler's thread). + """ + current: Flight | None = getattr(self._local, "flight", None) + with self._lock: + found = [ + flight + for flight in self._flights.values() + if event.correlation_id in (flight.record.run_id, flight.record.correlation_id) + ] + if current is not None and all(flight is not current for flight in found): + found.append(current) + return found diff --git a/automation_file/scheduler/errors.py b/automation_file/scheduler/errors.py new file mode 100644 index 0000000..499e26b --- /dev/null +++ b/automation_file/scheduler/errors.py @@ -0,0 +1,9 @@ +"""The scheduler's exception.""" + +from __future__ import annotations + +from automation_file.exceptions import FileAutomationException + + +class SchedulerException(FileAutomationException): + """Raised for duplicate / missing / invalid scheduled jobs.""" diff --git a/automation_file/scheduler/job.py b/automation_file/scheduler/job.py new file mode 100644 index 0000000..d883c0c --- /dev/null +++ b/automation_file/scheduler/job.py @@ -0,0 +1,96 @@ +"""A scheduled job: what fires it, what it runs, and its counters.""" + +from __future__ import annotations + +import datetime as dt +from dataclasses import dataclass, field +from typing import TYPE_CHECKING, Any + +from automation_file.scheduler.cron import CronExpression +from automation_file.scheduler.errors import SchedulerException +from automation_file.scheduler.runs import TARGET_ACTIONS, TARGET_PIPELINE +from automation_file.scheduler.triggers import CronTrigger, PipelineTrigger, Trigger + +if TYPE_CHECKING: + from automation_file.pipeline import Pipeline + + +@dataclass +class ScheduledJob: + """One named job: its triggers, its target and what happened to it so far. + + ``cron`` and ``timezone`` mirror the job's first cron trigger, and + ``action_list`` is empty for a pipeline target. ``running`` stays true until + the thread of the last run has really ended, also after a timeout or a + cancellation, so that overlap protection never lets two runs collide. + """ + + name: str + cron: CronExpression | None + action_list: list[list[Any]] + last_run: dt.datetime | None = field(default=None) + runs: int = field(default=0) + allow_overlap: bool = field(default=False) + running: bool = field(default=False) + skipped: int = field(default=0) + timezone: str | None = field(default=None) + triggers: tuple[Trigger, ...] = field(default=()) + pipeline: Pipeline | None = field(default=None) + params: dict[str, Any] = field(default_factory=dict) + timeout: float | None = field(default=None) + last_state: str | None = field(default=None) + active: int = field(default=0) + + def __post_init__(self) -> None: + self.triggers = tuple(self.triggers) + crons = [trigger for trigger in self.triggers if isinstance(trigger, CronTrigger)] + if self.cron is not None and not crons: + self.triggers = (CronTrigger(self.cron, self.timezone), *self.triggers) + elif crons: + self.cron = crons[0].expression + self.timezone = crons[0].timezone + if self.pipeline is None: + return + for trigger in self.triggers: + if isinstance(trigger, PipelineTrigger) and trigger.pipeline == self.pipeline.name: + raise SchedulerException( + f"job {self.name!r}: a pipeline cannot be fired by its own runs" + ) + + @property + def target(self) -> str: + """``"pipeline"`` or ``"actions"``: what the job runs.""" + return TARGET_ACTIONS if self.pipeline is None else TARGET_PIPELINE + + def due(self, instant: dt.datetime) -> CronTrigger | None: + """Return the first cron trigger that fires at ``instant``, ``None`` when none does.""" + for trigger in self.triggers: + if isinstance(trigger, CronTrigger) and trigger.due(instant): + return trigger + return None + + def local(self, instant: dt.datetime) -> dt.datetime: + """Return ``instant`` in the job's own time: its first cron trigger's, else local time.""" + for trigger in self.triggers: + if isinstance(trigger, CronTrigger): + return trigger.moment(instant) + return instant.astimezone().replace(tzinfo=None) + + def as_dict(self) -> dict[str, Any]: + """Return a JSON-serialisable snapshot of the job.""" + return { + "name": self.name, + "cron": "" if self.cron is None else self.cron.source, + "actions": len(self.action_list), + "last_run": self.last_run.isoformat() if self.last_run else None, + "runs": self.runs, + "allow_overlap": self.allow_overlap, + "running": self.running, + "skipped": self.skipped, + "timezone": self.timezone, + "triggers": [trigger.to_dict() for trigger in self.triggers], + "target": self.target, + "pipeline": None if self.pipeline is None else self.pipeline.name, + "timeout": self.timeout, + "last_state": self.last_state, + } diff --git a/automation_file/scheduler/manager.py b/automation_file/scheduler/manager.py index 9444d94..030bb00 100644 --- a/automation_file/scheduler/manager.py +++ b/automation_file/scheduler/manager.py @@ -1,122 +1,274 @@ -"""Background scheduler for cron-scheduled action lists. - -The scheduler thread wakes once a minute (aligned to wall-clock minute -boundaries), iterates registered jobs, and dispatches each matching job's -action list through the shared :class:`ActionExecutor`. Dispatch happens on a -short-lived worker thread so a long-running action cannot block subsequent -jobs — but callers are still responsible for keeping their action lists -reasonable in duration. +"""The scheduler: every job, whatever fires it, runs through here and leaves a record. + +A job pairs a target (an action list or a pipeline) with triggers: cron +expressions with an optional time zone, file events, events on the bus, the end +of another pipeline's run, or none at all for a job that is only fired by hand. +One background thread wakes every second: it fires the cron jobs due in the +current minute and closes the runs whose timeout has passed. Every run gets a +thread of its own, so a long job cannot hold up the others. + +Overlap protection is part of the contract: unless a job says +``allow_overlap``, a firing that arrives while the job is still running is +recorded as ``skipped`` and nothing is started. + +The module-level :data:`scheduler` is the process-wide instance behind the +``FA_schedule_*`` actions. """ from __future__ import annotations import datetime as dt +import math import threading -from dataclasses import dataclass, field +from collections.abc import Mapping +from functools import partial from typing import Any from automation_file.core.action_registry import ActionRegistry +from automation_file.core.progress import CancellationToken +from automation_file.events import Event, EventBus, event_bus from automation_file.exceptions import FileAutomationException from automation_file.logging_config import file_automation_logger -from automation_file.scheduler.cron import CronExpression - - -class SchedulerException(FileAutomationException): - """Raised for duplicate / missing / invalid scheduled jobs.""" - - -@dataclass -class ScheduledJob: - """One named cron expression paired with an action list.""" - - name: str - cron: CronExpression - action_list: list[list[Any]] - last_run: dt.datetime | None = field(default=None) - runs: int = field(default=0) - allow_overlap: bool = field(default=False) - running: bool = field(default=False) - skipped: int = field(default=0) - - def as_dict(self) -> dict[str, Any]: - return { - "name": self.name, - "cron": self.cron.source, - "actions": len(self.action_list), - "last_run": self.last_run.isoformat() if self.last_run else None, - "runs": self.runs, - "allow_overlap": self.allow_overlap, - "running": self.running, - "skipped": self.skipped, - } +from automation_file.scheduler.dispatch import Clock, Dispatcher, Firing, Services +from automation_file.scheduler.errors import SchedulerException +from automation_file.scheduler.job import ScheduledJob +from automation_file.scheduler.runs import ( + DEFAULT_HISTORY, + DEFAULT_QUERY_LIMIT, + JobRun, + RunHistory, + RunState, + TriggerKind, + as_utc, +) +from automation_file.scheduler.targets import ( + Outcome, + describe, + fill_params, + load_pipeline, + run_actions, +) +from automation_file.scheduler.triggers import ( + CronTrigger, + Disarm, + TriggerPort, + as_triggers, +) + +_LISTED_FAILURES = 5 +_MAX_TIMEOUT = 366 * 24 * 3600.0 + + +def _utc_now() -> dt.datetime: + return dt.datetime.now(dt.timezone.utc) + + +def _checked_name(name: Any) -> str: + if not isinstance(name, str) or not name.strip(): + raise SchedulerException(f"job name: expected a non-empty string, got {name!r}") + return name + + +def _checked_timeout(timeout: Any) -> float | None: + if timeout is None: + return None + if isinstance(timeout, bool) or not isinstance(timeout, (int, float)): + raise SchedulerException(f"timeout: expected seconds above 0, got {timeout!r}") + if not math.isfinite(timeout) or not 0 < timeout <= _MAX_TIMEOUT: + raise SchedulerException( + f"timeout: expected seconds above 0 and at most a year, got {timeout!r}" + ) + return float(timeout) + + +def _checked_params(params: Any) -> dict[str, Any]: + if params is None: + return {} + if not isinstance(params, Mapping): + raise SchedulerException(f"params: expected a mapping, got {type(params).__name__}") + try: + fill_params(params, _utc_now()) + except ValueError as error: + raise SchedulerException(f"params: a ${{date:...}} format is not valid: {error}") from error + return dict(params) + + +def _disarm(name: str, disarms: list[Disarm]) -> None: + """Stop the watchers and subscriptions of one job; one that fails does not keep the rest.""" + for disarm in disarms: + try: + disarm() + except (RuntimeError, OSError) as error: + file_automation_logger.error( + "scheduler[%s]: a trigger could not be stopped: %r", name, error + ) + + +def _safe_execute( + job_name: str, + action_list: list[list[Any]], + cancel: CancellationToken | None = None, +) -> Outcome: + """Run ``action_list`` and say how it went; never raises for a failing list. + + An action that raises fails the run and the list goes on. A list the + executor rejects is reported through ``notify_on_failure``, which publishes + the ``scheduler.error`` event itself. + """ + from automation_file.notify.manager import notify_on_failure + + try: + failures = run_actions(action_list, CancellationToken() if cancel is None else cancel) + except FileAutomationException as error: + file_automation_logger.warning("scheduler[%s]: dispatch failed: %r", job_name, error) + notify_on_failure(f"scheduler[{job_name}]", error) + return Outcome(RunState.FAILED, describe(error), reported=True) + if not failures: + return Outcome(RunState.COMPLETED) + listed = "; ".join(failures[:_LISTED_FAILURES]) + more = len(failures) - _LISTED_FAILURES + if more > 0: + listed = f"{listed}; and {more} more" + return Outcome( + RunState.FAILED, f"{len(failures)} of {len(action_list)} actions failed: {listed}" + ) class Scheduler: - """Process-wide scheduler — one background thread drives every job.""" + """Process-wide scheduler — one background thread drives every job. + + ``clock`` returns the current time as an aware ``datetime`` (UTC by default) + and ``bus`` is the event bus the scheduler listens and reports on (the + process-wide one by default). ``history_limit`` bounds the remembered runs. + With ``autostart=False`` no thread is started until :meth:`start` is called: + drive the scheduler with :meth:`tick` instead, which is what the tests do. + """ _TICK_SECONDS = 1.0 - def __init__(self) -> None: - self._lock = threading.Lock() + def __init__( + self, + *, + clock: Clock | None = None, + bus: EventBus | None = None, + history_limit: int = DEFAULT_HISTORY, + autostart: bool = True, + ) -> None: + self._lock = threading.RLock() self._jobs: dict[str, ScheduledJob] = {} + self._armed: dict[str, list[Disarm]] = {} + self._clock: Clock = _utc_now if clock is None else clock + self._bus = event_bus if bus is None else bus + self._history = RunHistory(history_limit) + self._runs = Dispatcher( + Services(self._lock, self._bus, self._clock, self._history, _safe_execute) + ) + self._autostart = autostart self._thread: threading.Thread | None = None self._stop = threading.Event() + self._last_minute: dt.datetime | None = None + + # ------------------------------------------------------------------ the clock def _ensure_running(self) -> None: - if self._thread is not None and self._thread.is_alive(): - return - self._stop.clear() - thread = threading.Thread(target=self._run, name="fa-scheduler", daemon=True) - thread.start() - self._thread = thread - - def _run(self) -> None: - last_minute: tuple[int, int, int, int, int] | None = None - while not self._stop.is_set(): - now = dt.datetime.now().replace(second=0, microsecond=0) - key = (now.year, now.month, now.day, now.hour, now.minute) - if key != last_minute: - last_minute = key - self._fire_due(now) - self._stop.wait(self._TICK_SECONDS) - - def _fire_due(self, moment: dt.datetime) -> None: - with self._lock: - due = [job for job in self._jobs.values() if job.cron.matches(moment)] - for job in due: - self._dispatch(job, moment) + if self._autostart: + self.start() - def _dispatch(self, job: ScheduledJob, moment: dt.datetime) -> None: + def start(self) -> None: + """Start the background thread and arm every trigger; safe to call again. + + Needed after :meth:`shutdown`, and for a scheduler made with + ``autostart=False``. Adding a job starts an ``autostart`` scheduler. + """ + waiting: list[tuple[ScheduledJob, list[Disarm]]] = [] with self._lock: - if job.running and not job.allow_overlap: - job.skipped += 1 - file_automation_logger.warning( - "scheduler[%s]: previous run still active — skipping (skipped=%d)", - job.name, - job.skipped, + if self._thread is None or not self._thread.is_alive(): + # A stop flag of its own: a thread that is slow to end still ends. + stop = threading.Event() + thread = threading.Thread( + target=self._run, args=(stop,), name="fa-scheduler", daemon=True + ) + thread.start() + self._stop, self._thread = stop, thread + for name, job in self._jobs.items(): + if name not in self._armed: + self._armed[name] = [] + waiting.append((job, self._armed[name])) + for job, slot in waiting: + try: + self._arm(job, slot) + except (FileAutomationException, OSError) as error: + file_automation_logger.error( + "scheduler[%s]: its triggers could not be armed: %r", job.name, error ) - return - job.running = True - job.last_run = moment - job.runs += 1 - run_no = job.runs - file_automation_logger.info( - "scheduler[%s]: firing at %s (run #%d)", job.name, moment.isoformat(), run_no - ) - worker = threading.Thread( - target=self._run_job, - args=(job,), - name=f"fa-scheduler-{job.name}", - daemon=True, - ) - worker.start() - def _run_job(self, job: ScheduledJob) -> None: - try: - _safe_execute(job.name, job.action_list) - finally: - with self._lock: - job.running = False + def _run(self, stop: threading.Event) -> None: + while not stop.is_set(): + try: + self.tick() + except Exception as error: # pylint: disable=broad-except + # Boundary of the scheduler's thread: one bad tick must not end scheduling. + file_automation_logger.error("scheduler: tick failed: %r", error) + stop.wait(self._TICK_SECONDS) + + def tick(self, now: dt.datetime | None = None) -> list[JobRun]: + """Bring the scheduler to ``now`` and return the records of what it fired. + + Runs whose timeout has passed are closed, and every cron job due in the + minute of ``now`` is fired; a minute is handled once, however often it is + ticked. ``now`` defaults to the scheduler's clock; a naive ``datetime`` + is taken as local time. The background thread calls this every second. + """ + instant = as_utc(self._clock() if now is None else now) + self._runs.expire(instant) + minute = instant.replace(second=0, microsecond=0) + with self._lock: + if minute == self._last_minute: + return [] + self._last_minute = minute + return self._fire_due(minute, instant) + + def _fire_due(self, minute: dt.datetime, now: dt.datetime | None = None) -> list[JobRun]: + with self._lock: + jobs = list(self._jobs.values()) + fired: list[JobRun] = [] + for job in jobs: + trigger = job.due(minute) + if trigger is None: + continue + detail = {"cron": trigger.cron, "timezone": trigger.timezone} + firing = Firing(TriggerKind.CRON, minute, now, detail) + fired.append(self._dispatch(job, trigger.moment(minute), firing)) + return fired + + def _dispatch( + self, job: ScheduledJob, moment: dt.datetime, firing: Firing | None = None + ) -> JobRun: + """Fire ``job`` at ``moment`` (the time as the job reads it); skip it if it overlaps.""" + chosen = Firing(TriggerKind.CRON, as_utc(moment)) if firing is None else firing + return self._runs.dispatch(job, moment, chosen) + + def _fire( + self, + name: str, + kind: TriggerKind, + detail: Mapping[str, Any], + cause: Event | None = None, + ) -> JobRun | None: + """A trigger's moment has come: fire the job ``name`` unless it is gone. + + ``cause`` is the event that fired it; a run of this scheduler that + published the event makes the new run one link longer in its chain. + """ + with self._lock: + job = self._jobs.get(name) + if job is None: + return None + now = as_utc(self._clock()) + depth = 0 if cause is None else self._runs.depth_after(cause) + return self._dispatch(job, job.local(now), Firing(kind, now, detail=detail, depth=depth)) + + # ------------------------------------------------------------------ jobs def add( self, @@ -125,68 +277,240 @@ def add( action_list: list[list[Any]], *, allow_overlap: bool = False, + timezone: str | None = None, + timeout: float | None = None, + ) -> dict[str, Any]: + """Register an action list on a cron expression and return the job's snapshot. + + ``timezone`` is an IANA name (``"Asia/Taipei"``) or ``"UTC"``; without it + the expression is read in local time. ``timeout`` is in seconds. + """ + trigger = CronTrigger(cron_expression, timezone) + job = ScheduledJob( + name=name, + cron=trigger.expression, + action_list=list(action_list), + allow_overlap=allow_overlap, + triggers=(trigger,), + timeout=_checked_timeout(timeout), + ) + return self._register(job) + + def add_job( + self, + name: str, + target: Any, + *, + triggers: Any = None, + allow_overlap: bool = False, + timeout: float | None = None, + params: Mapping[str, Any] | None = None, + ) -> dict[str, Any]: + """Register a job with any triggers and return its snapshot. + + ``target`` is an action list, or a pipeline: a ``Pipeline``, a definition + mapping or the path of a definition file. ``triggers`` is one trigger or + several (objects or their mappings); without any, the job runs only when + it is fired with :meth:`run_now`. ``params`` are the parameters of a + pipeline's runs; ``${date:FORMAT}`` in them is filled in at every firing. + A pipeline's own ``schedule`` is not read here: see :meth:`add_pipeline`. + """ + is_actions = isinstance(target, (list, tuple)) + if is_actions and params: + raise SchedulerException("params belong to a pipeline; an action list takes none") + job = ScheduledJob( + name=_checked_name(name), + cron=None, + action_list=list(target) if is_actions else [], + allow_overlap=bool(allow_overlap), + triggers=as_triggers(triggers), + pipeline=None if is_actions else load_pipeline(target), + params=_checked_params(params), + timeout=_checked_timeout(timeout), + ) + return self._register(job) + + def add_pipeline( + self, + pipeline: Any, + *, + name: str | None = None, + triggers: Any = None, + allow_overlap: bool = False, + timeout: float | None = None, + params: Mapping[str, Any] | None = None, ) -> dict[str, Any]: - cron = CronExpression.parse(cron_expression) + """Register a pipeline as a job, on the schedule it declares. + + ``pipeline`` is a ``Pipeline``, a definition mapping or the path of a + definition file. Its ``schedule`` becomes a cron trigger, time zone + included; ``triggers`` adds more. The job is named after the pipeline + unless ``name`` is given. A definition file is read once, now: remove the + job and register it again after changing the file. + """ + loaded = load_pipeline(pipeline) + chosen = as_triggers(triggers) + if loaded.schedule is not None: + schedule = CronTrigger(loaded.schedule.cron, loaded.schedule.timezone) + chosen = (schedule, *chosen) + return self.add_job( + loaded.name if name is None else name, + loaded, + triggers=chosen, + allow_overlap=allow_overlap, + timeout=timeout, + params=params, + ) + + def _register(self, job: ScheduledJob) -> dict[str, Any]: + slot: list[Disarm] = [] with self._lock: - if name in self._jobs: - raise SchedulerException(f"job already registered: {name}") - job = ScheduledJob( - name=name, - cron=cron, - action_list=list(action_list), - allow_overlap=allow_overlap, - ) - self._jobs[name] = job + if job.name in self._jobs: + raise SchedulerException(f"job already registered: {job.name}") + self._jobs[job.name] = job + self._armed[job.name] = slot snapshot = job.as_dict() + armed = False + try: + self._arm(job, slot) + armed = True + finally: + if not armed: + with self._lock: + if self._jobs.get(job.name) is job: + del self._jobs[job.name] self._ensure_running() file_automation_logger.info( - "scheduler: added job %r (cron=%r allow_overlap=%s)", - name, - cron.source, - allow_overlap, + "scheduler: added job %r (cron=%r target=%s allow_overlap=%s)", + job.name, + snapshot["cron"], + job.target, + job.allow_overlap, ) return snapshot + def _arm(self, job: ScheduledJob, slot: list[Disarm]) -> None: + """Start the job's watchers and subscriptions; on failure none of them is left. + + ``slot`` is the list registered for the job in ``_armed``. When the job + was removed, or the scheduler shut down, while its triggers were being + armed, the slot is no longer registered and what was armed is stopped. + """ + port = TriggerPort( + job=job.name, + bus=self._bus, + fire=partial(self._fire, job.name), + is_own=partial(self._runs.is_own, job.name), + ) + disarms: list[Disarm] = [] + armed = False + try: + for trigger in job.triggers: + disarm = trigger.arm(port) + if disarm is not None: + disarms.append(disarm) + armed = True + finally: + with self._lock: + registered = self._armed.get(job.name) is slot + if registered and armed: + slot.extend(disarms) + disarms = [] + elif registered: + del self._armed[job.name] + _disarm(job.name, disarms) + def remove(self, name: str) -> dict[str, Any]: + """Remove the job ``name`` and stop its triggers; a run in progress goes on.""" with self._lock: job = self._jobs.pop(name, None) + disarms = self._armed.pop(name, []) if job is None: raise SchedulerException(f"no such job: {name}") + _disarm(name, disarms) file_automation_logger.info("scheduler: removed job %r", name) return job.as_dict() def remove_all(self) -> list[dict[str, Any]]: + """Remove every job and stop every trigger; return the final snapshots.""" with self._lock: - snapshots = [job.as_dict() for job in self._jobs.values()] + jobs = list(self._jobs.values()) + armed = self._armed self._jobs.clear() - return snapshots + self._armed = {} + for name, disarms in armed.items(): + _disarm(name, disarms) + return [job.as_dict() for job in jobs] + + # ------------------------------------------------------------------ runs + + def run_now(self, name: str) -> JobRun: + """Fire the job ``name`` by hand and return the record of that firing. + + The record is ``skipped`` when the job is still running and does not + allow overlap. ``record.wait(timeout)`` blocks until the run has ended. + """ + with self._lock: + job = self._jobs.get(name) + if job is None: + raise SchedulerException(f"no such job: {name}") + now = as_utc(self._clock()) + return self._dispatch(job, job.local(now), Firing(TriggerKind.MANUAL, now)) + + def cancel(self, name: str) -> list[JobRun]: + """Cancel the runs in progress of the job ``name`` and return their records. + + The records become ``cancelled`` at once. A pipeline stops through its + cancellation token; an action list stops before its next action, since + the one in progress cannot be interrupted. The list is empty when the + job is not running. + """ + if not isinstance(name, str): + raise SchedulerException(f"no such job: {name!r}") + with self._lock: + known = name in self._jobs + cancelled = self._runs.cancel(name) + if not known and not cancelled: + raise SchedulerException(f"no such job: {name}") + return cancelled + + def history( + self, + job: str | None = None, + state: RunState | str | None = None, + limit: int = DEFAULT_QUERY_LIMIT, + ) -> list[JobRun]: + """Return up to ``limit`` run records, newest first, of one job and/or in one state.""" + return self._history.query(job, state, limit) def list(self) -> list[dict[str, Any]]: + """Return a snapshot of every registered job.""" with self._lock: return [job.as_dict() for job in self._jobs.values()] - def shutdown(self, timeout: float = 5.0) -> None: - self._stop.set() - thread = self._thread - self._thread = None - if thread is not None and thread.is_alive(): + def shutdown(self, timeout: float = 5.0, *, cancel_running: bool = False) -> None: + """Stop the background thread, every file watcher and every bus subscription. + + The jobs stay registered and :meth:`start` brings them back. Runs in + progress are left to finish unless ``cancel_running`` is set. + """ + with self._lock: + self._stop.set() + thread = self._thread + self._thread = None + armed = self._armed + self._armed = {} + if thread is not None and thread.is_alive() and thread is not threading.current_thread(): thread.join(timeout=timeout) + for name, disarms in armed.items(): + _disarm(name, disarms) + if cancel_running: + self._runs.cancel() def __contains__(self, name: object) -> bool: return isinstance(name, str) and name in self._jobs -def _safe_execute(job_name: str, action_list: list[list[Any]]) -> None: - from automation_file.core.action_executor import executor - from automation_file.notify.manager import notify_on_failure - - try: - executor.execute_action(action_list) - except FileAutomationException as error: - file_automation_logger.warning("scheduler[%s]: dispatch failed: %r", job_name, error) - notify_on_failure(f"scheduler[{job_name}]", error) - - scheduler: Scheduler = Scheduler() @@ -196,13 +520,24 @@ def schedule_add( action_list: list[list[Any]], *, allow_overlap: bool = False, + timezone: str | None = None, + timeout: float | None = None, ) -> dict[str, Any]: """Register a named job that fires ``action_list`` on ``cron_expression``. - When ``allow_overlap`` is False (the default), a tick that fires while a + When ``allow_overlap`` is False (the default), a firing that arrives while a previous run is still active is skipped and counted in ``skipped``. + ``timezone`` is an IANA name such as ``"Asia/Taipei"`` (local time without + it) and ``timeout`` the seconds a run may take. """ - return scheduler.add(name, cron_expression, action_list, allow_overlap=allow_overlap) + return scheduler.add( + name, + cron_expression, + action_list, + allow_overlap=allow_overlap, + timezone=timezone, + timeout=timeout, + ) def schedule_remove(name: str) -> dict[str, Any]: @@ -220,6 +555,70 @@ def schedule_list() -> list[dict[str, Any]]: return scheduler.list() +def schedule_job( + name: str, + action_list: list[list[Any]], + triggers: list[dict[str, Any]] | None = None, + allow_overlap: bool = False, + timeout: float | None = None, +) -> dict[str, Any]: + """Register an action list fired by any triggers, or only by hand when there are none. + + Each trigger is a mapping with its ``kind`` (``cron``, ``file``, ``event`` + or ``pipeline``) and that kind's arguments. + """ + if not isinstance(action_list, list): + raise SchedulerException( + f"action_list: expected a list of actions, got {type(action_list).__name__}" + ) + return scheduler.add_job( + name, action_list, triggers=triggers, allow_overlap=allow_overlap, timeout=timeout + ) + + +def schedule_pipeline( + definition: Any, + name: str | None = None, + triggers: list[dict[str, Any]] | None = None, + params: dict[str, Any] | None = None, + allow_overlap: bool = False, + timeout: float | None = None, +) -> dict[str, Any]: + """Register a pipeline definition (a mapping or a YAML/JSON file path) as a job. + + The definition's ``schedule`` becomes a cron trigger with its time zone; + ``triggers`` adds more. ``params`` are the parameters of every run, with + ``${date:FORMAT}`` filled in when the job fires. + """ + return scheduler.add_pipeline( + definition, + name=name, + triggers=triggers, + allow_overlap=allow_overlap, + timeout=timeout, + params=params, + ) + + +def schedule_run(name: str) -> dict[str, Any]: + """Fire the named job now and return the record of that firing.""" + return scheduler.run_now(name).to_dict() + + +def schedule_history( + job: str | None = None, + state: str | None = None, + limit: int = DEFAULT_QUERY_LIMIT, +) -> list[dict[str, Any]]: + """Return the latest run records, newest first, of one job and/or in one state.""" + return [run.to_dict() for run in scheduler.history(job, state, limit)] + + +def schedule_cancel(name: str) -> list[dict[str, Any]]: + """Cancel the runs in progress of the named job and return their records.""" + return [run.to_dict() for run in scheduler.cancel(name)] + + def register_scheduler_ops(registry: ActionRegistry) -> None: """Wire ``FA_schedule_*`` actions into a registry.""" registry.register_many( @@ -228,5 +627,10 @@ def register_scheduler_ops(registry: ActionRegistry) -> None: "FA_schedule_remove": schedule_remove, "FA_schedule_remove_all": schedule_remove_all, "FA_schedule_list": schedule_list, + "FA_schedule_job": schedule_job, + "FA_schedule_pipeline": schedule_pipeline, + "FA_schedule_run": schedule_run, + "FA_schedule_history": schedule_history, + "FA_schedule_cancel": schedule_cancel, } ) diff --git a/automation_file/scheduler/runs.py b/automation_file/scheduler/runs.py new file mode 100644 index 0000000..c1f467b --- /dev/null +++ b/automation_file/scheduler/runs.py @@ -0,0 +1,211 @@ +"""What the scheduler records about one firing of a job. + +Every firing leaves a :class:`JobRun`, whatever fired it and whether or not the +job ran: a firing that meets a run still in progress is recorded as ``skipped``. +A :class:`RunHistory` keeps the latest records of one scheduler in memory. +""" + +from __future__ import annotations + +import threading +from collections import deque +from dataclasses import dataclass, field +from datetime import datetime, timezone +from enum import Enum +from typing import Any + +from automation_file.scheduler.errors import SchedulerException + +DEFAULT_HISTORY = 1000 +DEFAULT_QUERY_LIMIT = 50 + +#: ``JobRun.target`` values. +TARGET_ACTIONS = "actions" +TARGET_PIPELINE = "pipeline" + +#: ``JobRun.reason``: skipped because the job was still running / skipped because +#: too many runs had fired one another / stopped by ``cancel``. +REASON_OVERLAP = "overlap" +REASON_CHAIN = "chain" +REASON_CANCELLED = "cancelled" + + +class RunState(str, Enum): + """Where one firing of a job stands.""" + + SCHEDULED = "scheduled" + STARTED = "started" + COMPLETED = "completed" + FAILED = "failed" + SKIPPED = "skipped" + TIMEOUT = "timeout" + CANCELLED = "cancelled" + + @property + def is_final(self) -> bool: + """Whether the record will not change state any more.""" + return self not in (RunState.SCHEDULED, RunState.STARTED) + + +class TriggerKind(str, Enum): + """What fired a run.""" + + CRON = "cron" + MANUAL = "manual" + FILE = "file" + EVENT = "event" + PIPELINE = "pipeline" + + +def as_utc(moment: datetime) -> datetime: + """Return ``moment`` as an aware UTC ``datetime``; a naive one is taken as local time.""" + return moment.astimezone(timezone.utc) + + +def _iso(moment: datetime | None) -> str | None: + return None if moment is None else moment.isoformat(timespec="microseconds") + + +@dataclass +class JobRun: + """One firing of a job: what fired it, how far it got, and how it ended. + + The times are timezone-aware UTC. ``correlation_id`` is the ID carried by + every event the run publishes: the run's own ID for an action list, and the + pipeline's run ID once a pipeline target has started (the ID + ``FA_pipeline_status`` takes). ``detail`` says what fired the run: the cron + expression, the file, the event, the pipeline. ``pipeline`` names the + pipeline of a pipeline target. ``reason`` says why a firing was skipped + (``overlap``, ``chain``) or that a run was ``cancelled``; ``error`` says + what went wrong in a ``failed`` or ``timeout`` run. + """ + + run_id: str + job: str + trigger: TriggerKind + scheduled_at: datetime + target: str = TARGET_ACTIONS + pipeline: str | None = None + state: RunState = RunState.SCHEDULED + started_at: datetime | None = None + finished_at: datetime | None = None + error: str | None = None + reason: str | None = None + correlation_id: str = "" + detail: dict[str, Any] = field(default_factory=dict) + _settled: threading.Event = field( + default_factory=threading.Event, init=False, repr=False, compare=False + ) + + def __post_init__(self) -> None: + if not self.correlation_id: + self.correlation_id = self.run_id + + @property + def done(self) -> bool: + """Whether the run has reached a final state.""" + return self.state.is_final + + @property + def duration_ms(self) -> float | None: + """Return the milliseconds between start and end, ``None`` while either is missing.""" + if self.started_at is None or self.finished_at is None: + return None + return round((self.finished_at - self.started_at).total_seconds() * 1000, 3) + + def wait(self, timeout: float | None = None) -> bool: + """Block until the scheduler is done with the run or ``timeout`` seconds passed. + + Returns whether the run has a final state. A run that ended by itself is + released once its job counts as not running any more; a run that timed + out or was cancelled is released at that moment, although its thread may + still be busy. + """ + self._settled.wait(timeout) + return self.done + + def close( + self, + state: RunState, + moment: datetime, + error: str | None = None, + reason: str | None = None, + ) -> None: + """Give the run its final state. The scheduler calls this; nothing else should.""" + self.finished_at = moment + self.error = error + self.reason = reason + self.state = state + + def settle(self) -> None: + """Release everything blocked in :meth:`wait`. The scheduler calls this.""" + self._settled.set() + + def to_dict(self) -> dict[str, Any]: + """Return a JSON-serialisable mapping of the record.""" + return { + "run_id": self.run_id, + "job": self.job, + "trigger": self.trigger.value, + "target": self.target, + "pipeline": self.pipeline, + "state": self.state.value, + "scheduled_at": _iso(self.scheduled_at), + "started_at": _iso(self.started_at), + "finished_at": _iso(self.finished_at), + "duration_ms": self.duration_ms, + "error": self.error, + "reason": self.reason, + "correlation_id": self.correlation_id, + "detail": dict(self.detail), + } + + +def as_run_state(state: RunState | str) -> RunState: + """Return ``state`` as a :class:`RunState`; an unknown word raises ``SchedulerException``.""" + try: + return RunState(state) + except ValueError as error: + known = ", ".join(item.value for item in RunState) + raise SchedulerException(f"unknown run state {state!r} (one of: {known})") from error + + +class RunHistory: + """The latest runs of one scheduler, oldest dropped first. Thread-safe.""" + + def __init__(self, limit: int = DEFAULT_HISTORY) -> None: + if isinstance(limit, bool) or not isinstance(limit, int) or limit < 1: + raise SchedulerException(f"history limit: expected an integer >= 1, got {limit!r}") + self._lock = threading.Lock() + self._runs: deque[JobRun] = deque(maxlen=limit) + + def __len__(self) -> int: + with self._lock: + return len(self._runs) + + def add(self, run: JobRun) -> None: + """Remember ``run``; the oldest record goes when the history is full.""" + with self._lock: + self._runs.append(run) + + def query( + self, + job: str | None = None, + state: RunState | str | None = None, + limit: int = DEFAULT_QUERY_LIMIT, + ) -> list[JobRun]: + """Return up to ``limit`` records, newest first, of one job and/or in one state.""" + wanted = None if state is None else as_run_state(state) + with self._lock: + runs = list(self._runs) + chosen = [ + run + for run in reversed(runs) + if (job is None or run.job == job) and (wanted is None or run.state is wanted) + ] + return chosen[: max(limit, 0)] + + def clear(self) -> None: + """Forget every record.""" + with self._lock: + self._runs.clear() diff --git a/automation_file/scheduler/targets.py b/automation_file/scheduler/targets.py new file mode 100644 index 0000000..0b663a6 --- /dev/null +++ b/automation_file/scheduler/targets.py @@ -0,0 +1,188 @@ +"""What a job runs: an action list or a pipeline. + +An action list goes through the shared executor one action at a time, so the +scheduler learns which action raised and can stop before the next one when the +run is cancelled or out of time. A pipeline runs in the calling thread with a +cancellation token the scheduler keeps. +""" + +from __future__ import annotations + +import os +import re +import threading +import time +from collections.abc import Mapping +from dataclasses import dataclass +from datetime import datetime +from typing import TYPE_CHECKING, Any + +from automation_file.core.progress import CancellationToken +from automation_file.events import Event, EventBus, PipelineStarted +from automation_file.exceptions import ExecuteActionException, FileAutomationException +from automation_file.logging_config import file_automation_logger +from automation_file.scheduler.errors import SchedulerException +from automation_file.scheduler.runs import JobRun, RunState + +if TYPE_CHECKING: + from automation_file.pipeline import Pipeline + +_DATE = re.compile(r"\$\{date(?::([^}]*))?\}", re.IGNORECASE) +_DEFAULT_DATE_FORMAT = "%Y-%m-%dT%H:%M:%S" +_UNKNOWN_ACTION = "unknown" + + +@dataclass(frozen=True) +class Outcome: + """How a target ended. + + ``reported`` is set when the failure has already been published (by + ``notify_on_failure``), so the scheduler does not publish it a second time. + """ + + state: RunState + error: str | None = None + reported: bool = False + + +def describe(error: BaseException) -> str: + """Return ``": "`` with every URL cut down to its host.""" + from automation_file.notify.manager import describe_error + + return describe_error(error) + + +def _action_name(action: Any) -> str: + if isinstance(action, list) and action and isinstance(action[0], str): + return action[0] + return _UNKNOWN_ACTION + + +def _run_action(action: Any) -> str | None: + """Run one action through the executor; return ``": "`` if it raised.""" + from automation_file.core.action_executor import executor + from automation_file.core.metrics import record_action + + name = _action_name(action) + started = time.monotonic() + try: + # pylint: disable-next=protected-access # the executor's single-action Template Method + executor._execute_event(action) + except Exception as error: # pylint: disable=broad-except + # Boundary: one failing action must not stop the list; it is recorded and reported. + record_action(name, time.monotonic() - started, False) + problem = describe(error) + file_automation_logger.error("scheduler: action %s failed: %s", name, problem) + return f"{name}: {problem}" + record_action(name, time.monotonic() - started, True) + file_automation_logger.info("scheduler: action %s done", name) + return None + + +def run_actions(action_list: Any, cancel: CancellationToken) -> list[str]: + """Run the actions of ``action_list`` in order; return one line per action that raised. + + A failing action does not stop the list, as in ``execute_action``. A + cancelled ``cancel`` stops it before the next action: the action in progress + cannot be interrupted. A list the executor does not accept (empty, not a + list) raises :class:`~automation_file.exceptions.ExecuteActionException` + before anything runs. + """ + from automation_file.core.action_executor import executor + + actions = executor.settings.rules.extract(action_list) + if actions is None: + raise ExecuteActionException("action_list is empty") + failures: list[str] = [] + for index, action in enumerate(actions): + if cancel.is_cancelled: + file_automation_logger.info( + "scheduler: action list stopped before action %d of %d", index + 1, len(actions) + ) + break + problem = _run_action(action) + if problem is not None: + failures.append(f"execute[{index}] {problem}") + return failures + + +def load_pipeline(source: Any) -> Pipeline: + """Return the pipeline ``source`` stands for, checked. + + ``source`` is a :class:`~automation_file.pipeline.Pipeline`, a definition + mapping, or the path of a ``.yaml`` / ``.yml`` / ``.json`` definition file. + A definition that is wrong raises ``PipelineDefinitionException`` here, when + the job is registered, not at its first firing. + """ + from automation_file.pipeline import Pipeline + + if isinstance(source, Pipeline): + pipeline = source + elif isinstance(source, Mapping): + pipeline = Pipeline.from_dict(source) + elif isinstance(source, (str, os.PathLike)): + pipeline = Pipeline.from_file(source) + else: + raise SchedulerException( + "pipeline: expected a Pipeline, a definition mapping or a file path, " + f"got {type(source).__name__}" + ) + pipeline.validate() + return pipeline + + +def run_pipeline( + pipeline: Pipeline, + params: Mapping[str, Any], + record: JobRun, + cancel: CancellationToken, + bus: EventBus, +) -> Outcome: + """Run ``pipeline`` in the calling thread and say how it ended. + + The pipeline's events carry its own run ID as their correlation ID, so + ``record.correlation_id`` is moved to that ID as soon as the run starts. + """ + from automation_file.pipeline import RunStatus + + worker = threading.get_ident() + + def adopt(event: Event) -> None: + if threading.get_ident() == worker: + record.correlation_id = str(event.payload.get("run_id") or record.correlation_id) + + subscription = bus.subscribe(adopt, types=PipelineStarted) + try: + run = pipeline.run(params=params, cancel=cancel, bus=bus) + except FileAutomationException as error: + return Outcome(RunState.FAILED, describe(error)) + finally: + bus.unsubscribe(subscription) + record.correlation_id = run.run_id + if run.status is RunStatus.SUCCEEDED: + return Outcome(RunState.COMPLETED) + if run.status is RunStatus.CANCELLED: + return Outcome(RunState.CANCELLED, run.error) + return Outcome(RunState.FAILED, run.error or f"pipeline {pipeline.name} did not succeed") + + +def _fill(value: Any, moment: datetime) -> Any: + if isinstance(value, str): + return _DATE.sub( + lambda match: moment.strftime(match.group(1) or _DEFAULT_DATE_FORMAT), value + ) + if isinstance(value, Mapping): + return {key: _fill(item, moment) for key, item in value.items()} + if isinstance(value, (list, tuple)): + return [_fill(item, moment) for item in value] + return value + + +def fill_params(params: Mapping[str, Any], moment: datetime) -> dict[str, Any]: + """Return ``params`` with ``${date}`` and ``${date:FORMAT}`` replaced by ``moment``. + + ``FORMAT`` is a ``strftime`` format; a bare ``${date}`` gives + ``2026-10-08T02:00:00``. Strings are looked at at every depth; anything else + is passed on unchanged. + """ + return {name: _fill(value, moment) for name, value in params.items()} diff --git a/automation_file/scheduler/triggers.py b/automation_file/scheduler/triggers.py new file mode 100644 index 0000000..077dcaa --- /dev/null +++ b/automation_file/scheduler/triggers.py @@ -0,0 +1,405 @@ +"""What fires a job. + +A job has any number of triggers; a job without one runs only when it is fired +by hand (:meth:`~automation_file.scheduler.manager.Scheduler.run_now`). + +* :class:`CronTrigger` is polled: the scheduler asks it once a minute. +* :class:`FileTrigger`, :class:`EventTrigger` and :class:`PipelineTrigger` are + *armed*: they start a file watcher or subscribe on the event bus, and call the + scheduler back through a :class:`TriggerPort` when their moment has come. + +Every trigger turns into a JSON-friendly mapping (``to_dict``) and back +(:func:`trigger_from_dict`), which is the form the ``FA_schedule_*`` actions take. +""" + +from __future__ import annotations + +import datetime as dt +import os +from abc import ABC, abstractmethod +from collections.abc import Callable, Iterable, Mapping +from dataclasses import dataclass, field +from functools import partial +from typing import Any, ClassVar + +from automation_file.events import ( + Event, + EventBus, + EventFilter, + PipelineCompleted, + PipelineFailed, + Severity, +) +from automation_file.pipeline.model import ALWAYS, ON_FAILURE, ON_SUCCESS, WHEN_CHOICES +from automation_file.scheduler.cron import CronException, CronExpression, resolve_timezone +from automation_file.scheduler.errors import SchedulerException +from automation_file.scheduler.runs import TriggerKind + +Disarm = Callable[[], object] +_KIND = "kind" +_DEFAULT_FILE_EVENTS = ("created", "modified") +_PIPELINE_EVENTS: dict[str, tuple[type[Event], ...]] = { + ON_SUCCESS: (PipelineCompleted,), + ON_FAILURE: (PipelineFailed,), + ALWAYS: (PipelineCompleted, PipelineFailed), +} + + +@dataclass(frozen=True) +class TriggerPort: + """What an armed trigger works with: the bus it listens on and how it fires its job. + + ``fire(kind, detail, cause)`` asks the scheduler to run the job; ``detail`` + ends up on the run record and ``cause`` is the event that did it, ``None`` + when there is none. ``is_own(event)`` tells whether an event was published + by a run of this very job, which must not fire the job again. + """ + + job: str + bus: EventBus + fire: Callable[[TriggerKind, Mapping[str, Any], Event | None], object] + is_own: Callable[[Event], bool] + + +class Trigger(ABC): + """Something that fires a job. A subclass says when.""" + + kind: ClassVar[TriggerKind] + + @abstractmethod + def to_dict(self) -> dict[str, Any]: + """Return the JSON-friendly form of the trigger, with its ``kind``.""" + + def arm(self, port: TriggerPort) -> Disarm | None: + """Start watching for the trigger's moment and return the call that stops it. + + The default starts nothing and returns ``None``: a cron trigger is + polled by the scheduler instead. + """ + return None + + +def _as_tuple(value: Any) -> tuple[Any, ...]: + if value is None: + return () + if isinstance(value, (str, type)): + return (value,) + if isinstance(value, Iterable): + return tuple(value) + return (value,) + + +def _is_text(value: Any) -> bool: + return isinstance(value, str) and bool(value.strip()) + + +@dataclass(frozen=True) +class CronTrigger(Trigger): + """Fires at the minutes a 5-field cron expression names. + + With ``timezone`` (an IANA name such as ``"Asia/Taipei"``, or ``"UTC"``) the + expression is read in that zone; without one it is read in the system's + local time, as the scheduler always did, and follows the system clock with + no special case: a repeated local hour fires twice. With a zone that has + daylight saving time, a wall-clock time that does not exist on the day the clocks go forward + is not fired, and one that occurs twice when they go back fires once, the + first time. An expression whose hour field is ``*`` runs every hour anyway + and keeps firing through the repeated hour. + """ + + cron: str | CronExpression + timezone: str | None = None + expression: CronExpression = field(init=False, repr=False, compare=False) + zone: dt.tzinfo | None = field(init=False, repr=False, compare=False) + + kind: ClassVar[TriggerKind] = TriggerKind.CRON + + def __post_init__(self) -> None: + expression: Any = self.cron + if not isinstance(expression, CronExpression): + if not isinstance(expression, str): + raise CronException( + f"cron: expected an expression, got {type(expression).__name__}" + ) + expression = CronExpression.parse(expression) + zone = resolve_timezone(self.timezone) + object.__setattr__(self, "cron", expression.source) + if self.timezone is not None: + object.__setattr__(self, "timezone", self.timezone.strip()) + object.__setattr__(self, "expression", expression) + object.__setattr__(self, "zone", zone) + + def moment(self, instant: dt.datetime) -> dt.datetime: + """Return ``instant`` as the wall-clock time the expression is compared with. + + That is an aware ``datetime`` in the trigger's zone, or a naive one in + local time when the trigger has no zone. + """ + if self.zone is None: + return instant.astimezone().replace(tzinfo=None) + return instant.astimezone(self.zone) + + def due(self, instant: dt.datetime) -> bool: + """Return whether the trigger fires at ``instant`` (an aware ``datetime``).""" + local = self.moment(instant) + if not self.expression.matches(local): + return False + # ``fold`` marks the second pass through an hour the clocks repeat. + return not local.fold or self.expression.every_hour + + def to_dict(self) -> dict[str, Any]: + return {_KIND: self.kind.value, "cron": self.cron, "timezone": self.timezone} + + +@dataclass(frozen=True) +class FileTrigger(Trigger): + """Fires when something happens to a file under ``path``. + + ``events`` holds any of ``created``, ``modified``, ``deleted`` and ``moved``. + The watching is done by :class:`~automation_file.trigger.FileWatcher`, the + one behind ``FA_watch_start``; the scheduler owns the watcher and stops it + when the job is removed. + """ + + path: str + events: tuple[str, ...] | str = _DEFAULT_FILE_EVENTS + recursive: bool = True + + kind: ClassVar[TriggerKind] = TriggerKind.FILE + + def __post_init__(self) -> None: + if not isinstance(self.path, (str, os.PathLike)) or not os.fspath(self.path): + raise SchedulerException(f"file trigger: expected a path, got {self.path!r}") + object.__setattr__(self, "path", os.fspath(self.path)) + object.__setattr__(self, "events", _as_tuple(self.events) or _DEFAULT_FILE_EVENTS) + object.__setattr__(self, "recursive", bool(self.recursive)) + + def arm(self, port: TriggerPort) -> Disarm | None: + from automation_file.trigger.manager import FileWatcher + + def on_event(kind: str, path: str) -> None: + port.fire(self.kind, {"path": path, "event": kind}, None) + + watcher = FileWatcher( + f"scheduler[{port.job}]", + self.path, + [], + events=_as_tuple(self.events), + recursive=self.recursive, + on_event=on_event, + ) + watcher.start() + return watcher.stop + + def to_dict(self) -> dict[str, Any]: + return { + _KIND: self.kind.value, + "path": self.path, + "events": list(_as_tuple(self.events)), + "recursive": self.recursive, + } + + +def _type_name(wanted: EventFilter) -> str: + return wanted if isinstance(wanted, str) else wanted.type + + +def _checked_filters(types: Any) -> tuple[EventFilter, ...]: + chosen = _as_tuple(types) + for wanted in chosen: + if not _is_text(wanted) and not (isinstance(wanted, type) and issubclass(wanted, Event)): + raise SchedulerException( + "event trigger: a type is a name ('task.failed'), a prefix ('pipeline.*') " + f"or an Event class, got {wanted!r}" + ) + return chosen + + +def _checked_sources(sources: Any) -> tuple[str, ...]: + chosen = _as_tuple(sources) + for source in chosen: + if not _is_text(source): + raise SchedulerException(f"event trigger: a source is a name, got {source!r}") + return chosen + + +def _checked_severity(severity: Any) -> Severity: + try: + return Severity(severity) + except ValueError as error: + known = ", ".join(item.value for item in Severity) + raise SchedulerException( + f"event trigger: unknown severity {severity!r} (one of: {known})" + ) from error + + +@dataclass(frozen=True) +class EventTrigger(Trigger): + """Fires when a matching event is published on the event bus. + + ``types`` takes what the bus takes: type names (``"task.failed"``), prefixes + (``"pipeline.*"``) and event classes. ``sources`` are exact ``event.source`` + values and ``min_severity`` is the lowest severity that counts. At least one + type or source is required, so that a job cannot fire on everything. This is + also how a webhook fires a job: the code that receives the request publishes + an event, and the trigger matches it. + + An event published by a run of the job itself does not fire the job again. + Jobs that fire one another through their events form a chain, and the + firing that would make a chain longer than 16 runs is recorded as + ``skipped`` with the reason ``chain``. + """ + + types: tuple[EventFilter, ...] | EventFilter = () + sources: tuple[str, ...] | str = () + min_severity: Severity | str = Severity.INFO + + kind: ClassVar[TriggerKind] = TriggerKind.EVENT + + def __post_init__(self) -> None: + types = _checked_filters(self.types) + sources = _checked_sources(self.sources) + if not types and not sources: + raise SchedulerException("event trigger: give at least one type or one source") + object.__setattr__(self, "types", types) + object.__setattr__(self, "sources", sources) + object.__setattr__(self, "min_severity", _checked_severity(self.min_severity)) + + def arm(self, port: TriggerPort) -> Disarm | None: + sources = _as_tuple(self.sources) + + def on_event(event: Event) -> None: + if (sources and event.source not in sources) or port.is_own(event): + return + port.fire( + self.kind, + { + "event_type": event.type, + "event_id": event.id, + "source": event.source, + "subject": event.subject, + "correlation_id": event.correlation_id, + }, + event, + ) + + subscription = port.bus.subscribe( + on_event, types=_as_tuple(self.types) or None, min_severity=Severity(self.min_severity) + ) + return partial(port.bus.unsubscribe, subscription) + + def to_dict(self) -> dict[str, Any]: + return { + _KIND: self.kind.value, + "types": [_type_name(wanted) for wanted in _as_tuple(self.types)], + "sources": list(_as_tuple(self.sources)), + "min_severity": Severity(self.min_severity).value, + } + + +@dataclass(frozen=True) +class PipelineTrigger(Trigger): + """Fires when a run of the pipeline named ``pipeline`` has ended. + + ``when`` uses the pipeline's own words: ``"on_success"`` (the default: the + run succeeded), ``"on_failure"`` (it failed or was cancelled) or + ``"always"``. The trigger listens for ``pipeline.completed`` and + ``pipeline.failed`` on the scheduler's bus, so it sees every run of that + pipeline published there, whoever started it. + """ + + pipeline: str + when: str = ON_SUCCESS + + kind: ClassVar[TriggerKind] = TriggerKind.PIPELINE + + def __post_init__(self) -> None: + if not _is_text(self.pipeline): + raise SchedulerException( + f"pipeline trigger: expected a pipeline name, got {self.pipeline!r}" + ) + if self.when not in WHEN_CHOICES: + raise SchedulerException( + f"pipeline trigger: when is one of {', '.join(WHEN_CHOICES)}, got {self.when!r}" + ) + + def arm(self, port: TriggerPort) -> Disarm | None: + def on_event(event: Event) -> None: + if event.payload.get("pipeline") != self.pipeline or port.is_own(event): + return + port.fire( + self.kind, + { + "pipeline": self.pipeline, + "run_id": event.payload.get("run_id"), + "status": event.payload.get("status"), + }, + event, + ) + + subscription = port.bus.subscribe(on_event, types=_PIPELINE_EVENTS[self.when]) + return partial(port.bus.unsubscribe, subscription) + + def to_dict(self) -> dict[str, Any]: + return {_KIND: self.kind.value, "pipeline": self.pipeline, "when": self.when} + + +@dataclass(frozen=True) +class _Shape: + """How one kind of trigger is written as a mapping.""" + + builder: Callable[..., Trigger] + keys: tuple[str, ...] + required: str | None = None + + +_SHAPES: dict[str, _Shape] = { + TriggerKind.CRON.value: _Shape(CronTrigger, ("cron", "timezone"), "cron"), + TriggerKind.FILE.value: _Shape(FileTrigger, ("path", "events", "recursive"), "path"), + TriggerKind.EVENT.value: _Shape(EventTrigger, ("types", "sources", "min_severity")), + TriggerKind.PIPELINE.value: _Shape(PipelineTrigger, ("pipeline", "when"), "pipeline"), +} + + +def trigger_from_dict(spec: Mapping[str, Any]) -> Trigger: + """Build a trigger from its mapping: ``{"kind": "cron", "cron": "0 2 * * *", ...}``. + + The keys besides ``kind`` are the trigger's arguments. An unknown kind, an + unknown key or a missing argument raises :class:`SchedulerException`. + """ + if not isinstance(spec, Mapping): + raise SchedulerException(f"trigger: expected a mapping, got {type(spec).__name__}") + kind = spec.get(_KIND) + if kind == TriggerKind.MANUAL.value: + raise SchedulerException( + "trigger: 'manual' needs no trigger; every job can be fired by hand " + "(Scheduler.run_now, FA_schedule_run)" + ) + if not isinstance(kind, str) or kind not in _SHAPES: + raise SchedulerException( + f"trigger: unknown kind {kind!r} (one of: {', '.join(sorted(_SHAPES))})" + ) + shape = _SHAPES[kind] + unknown = sorted(str(key) for key in spec if key != _KIND and key not in shape.keys) + if unknown: + raise SchedulerException( + f"{kind} trigger: unknown key {', '.join(unknown)} (known: {', '.join(shape.keys)})" + ) + if shape.required is not None and shape.required not in spec: + raise SchedulerException(f"{kind} trigger: {shape.required!r} is required") + return shape.builder(**{key: spec[key] for key in shape.keys if key in spec}) + + +def as_triggers(triggers: Any) -> tuple[Trigger, ...]: + """Return ``triggers`` as a tuple: ``None``, one trigger, one mapping, or several of either.""" + if triggers is None: + return () + if isinstance(triggers, (Trigger, Mapping)): + triggers = (triggers,) + if isinstance(triggers, (str, bytes)) or not isinstance(triggers, Iterable): + raise SchedulerException( + f"triggers: expected triggers or their mappings, got {type(triggers).__name__}" + ) + return tuple( + entry if isinstance(entry, Trigger) else trigger_from_dict(entry) for entry in triggers + ) diff --git a/automation_file/trigger/manager.py b/automation_file/trigger/manager.py index bb101f9..dc62ff3 100644 --- a/automation_file/trigger/manager.py +++ b/automation_file/trigger/manager.py @@ -10,12 +10,16 @@ same JSON action-list shape is used everywhere. Dispatch always happens on watchdog's dispatcher thread — the executor's per-action ``try/except`` prevents a bad action from killing the observer. + +A watcher given ``on_event`` calls it with the event kind and the path instead +of running an action list; the scheduler's file trigger watches that way. """ from __future__ import annotations +import os import threading -from collections.abc import Iterable +from collections.abc import Callable, Iterable from pathlib import Path from typing import Any @@ -28,6 +32,7 @@ from automation_file.logging_config import file_automation_logger _SUPPORTED_EVENTS = frozenset({"created", "modified", "deleted", "moved"}) +EventCallback = Callable[[str, str], object] class TriggerException(FileAutomationException): @@ -47,24 +52,29 @@ def _parse_events(events: Iterable[str] | str | None) -> frozenset[str]: class _DispatchingHandler(FileSystemEventHandler): - """Route watchdog events into an action list on the shared executor.""" + """Route watchdog events into an action list on the shared executor, or into a callback.""" def __init__( self, name: str, events: frozenset[str], action_list: list[list[Any]], + on_event: EventCallback | None = None, ) -> None: super().__init__() self._name = name self._events = events self._action_list = action_list + self._on_event = on_event def on_any_event(self, event: FileSystemEvent) -> None: kind = event.event_type if kind not in self._events: return file_automation_logger.info("trigger[%s]: %s %s", self._name, kind, event.src_path) + if self._on_event is not None: + self._call_back(self._on_event, kind, os.fsdecode(event.src_path)) + return from automation_file.core.action_executor import executor from automation_file.notify.manager import notify_on_failure @@ -76,9 +86,20 @@ def on_any_event(self, event: FileSystemEvent) -> None: ) notify_on_failure(f"trigger[{self._name}]", error) + def _call_back(self, on_event: EventCallback, kind: str, path: str) -> None: + try: + on_event(kind, path) + except FileAutomationException as error: + # A failing callback must not end the observer's thread. + file_automation_logger.warning("trigger[%s]: callback failed: %r", self._name, error) + class FileWatcher: - """One named watchdog observer tied to an action list.""" + """One named watchdog observer tied to an action list, or to a callback. + + With ``on_event`` the watcher runs no action list: every matching event + calls ``on_event(kind, path)`` on watchdog's dispatcher thread. + """ def __init__( self, @@ -88,6 +109,7 @@ def __init__( *, events: Iterable[str] | str | None = None, recursive: bool = True, + on_event: EventCallback | None = None, ) -> None: resolved = Path(path).expanduser().resolve() if not resolved.exists(): @@ -97,6 +119,7 @@ def __init__( self.recursive = bool(recursive) self.events = _parse_events(events) self.action_list: list[list[Any]] = list(action_list) + self._on_event = on_event self._observer: BaseObserver | None = None @property @@ -107,7 +130,7 @@ def is_running(self) -> bool: def start(self) -> None: if self.is_running: return - handler = _DispatchingHandler(self.name, self.events, self.action_list) + handler = _DispatchingHandler(self.name, self.events, self.action_list, self._on_event) observer = Observer() observer.schedule(handler, str(self.path), recursive=self.recursive) observer.daemon = True diff --git a/dev.toml b/dev.toml index c3d541e..f1bcbad 100644 --- a/dev.toml +++ b/dev.toml @@ -32,6 +32,8 @@ dependencies = [ "defusedxml>=0.7.1", "je_action_core>=0.0.2", "PyYAML>=6.0.3", + # zoneinfo has no time-zone database of its own on Windows; a scheduler cron with a zone needs one. + "tzdata>=2024.1; platform_system == 'Windows'", "opentelemetry-api>=1.44.0", "opentelemetry-sdk>=1.44.0", "tomli>=2.0.1; python_version<\"3.11\"" diff --git a/docs/source/API/scheduler.rst b/docs/source/API/scheduler.rst index 31cffb7..67fae31 100644 --- a/docs/source/API/scheduler.rst +++ b/docs/source/API/scheduler.rst @@ -1,17 +1,61 @@ Scheduler ========= -Cron-style scheduler for recurring action lists. The parser understands the -standard 5-field syntax (minute hour day-of-month month day-of-week) with -``*``, ranges, lists, and ``*/n`` steps plus month / day-of-week aliases. A -background thread wakes on minute boundaries and dispatches every matching -job through the shared :class:`ActionExecutor`. +The scheduler runs action lists and pipelines when something fires them: a cron +expression with an optional time zone, a file event, an event on the bus, the end +of another pipeline's run, or a call. Every firing leaves a run record in one of +seven states, a job does not overlap itself unless it allows it, and a run can be +given a timeout and can be cancelled. Usage is described in the manual chapter +*Scheduler*. .. automodule:: automation_file.scheduler :members: +Scheduler and actions +--------------------- + +.. automodule:: automation_file.scheduler.manager + :members: + +Jobs +---- + +.. automodule:: automation_file.scheduler.job + :members: + +Triggers +-------- + +.. automodule:: automation_file.scheduler.triggers + :members: + +Cron expressions and time zones +------------------------------- + +The parser understands the standard 5-field syntax (minute hour day-of-month +month day-of-week) with ``*``, ranges, lists, and ``*/n`` steps plus month / +day-of-week aliases. + .. automodule:: automation_file.scheduler.cron :members: -.. automodule:: automation_file.scheduler.manager +Run records +----------- + +.. automodule:: automation_file.scheduler.runs + :members: + +Runtime +------- + +.. automodule:: automation_file.scheduler.dispatch + :members: + +.. automodule:: automation_file.scheduler.targets + :members: + +Exceptions +---------- + +.. automodule:: automation_file.scheduler.errors :members: diff --git a/docs/source/Eng/architecture.rst b/docs/source/Eng/architecture.rst index 6334c92..33af81a 100644 --- a/docs/source/Eng/architecture.rst +++ b/docs/source/Eng/architecture.rst @@ -66,7 +66,7 @@ dispatchers. subgraph Events["event-driven"] Trigger["TriggerManager
watchdog file watcher"] - Sched["Scheduler
5-field cron + overlap guard"] + Sched["Scheduler
cron · file · event · pipeline triggers
run records + overlap guard"] end subgraph Servers["servers"] @@ -325,8 +325,13 @@ Module layout ├── trigger/ │ └── manager.py # FileWatcher + TriggerManager (watchdog-backed) ├── scheduler/ - │ ├── cron.py # 5-field cron expression parser - │ └── manager.py # Scheduler background thread + ScheduledJob + │ ├── cron.py # 5-field cron expression parser, time zones + │ ├── triggers.py # cron / file / event / pipeline triggers + │ ├── job.py # ScheduledJob + │ ├── runs.py # JobRun, RunState, RunHistory + │ ├── targets.py # running an action list or a pipeline + │ ├── dispatch.py # overlap, timeout, cancellation, scheduler.error + │ └── manager.py # Scheduler background thread + FA_schedule_* actions ├── notify/ │ ├── sinks.py # Webhook / Slack / Email sinks │ └── manager.py # NotificationManager (fanout + dedup + auto-notify hook) @@ -413,11 +418,13 @@ their own dispatch paths: events to an action list dispatched through the shared registry. :data:`~automation_file.trigger.trigger_manager` owns the name → watcher map so the GUI and JSON actions share one lifecycle. -* :mod:`automation_file.scheduler` runs one background thread that wakes on - minute boundaries, iterates registered - :class:`~automation_file.scheduler.ScheduledJob` instances, and dispatches - every matching job on a short-lived worker thread so a slow action can't - starve subsequent jobs. +* :mod:`automation_file.scheduler` runs one background thread that wakes + every second, fires the :class:`~automation_file.scheduler.ScheduledJob` + instances whose cron trigger is due in the current minute, and gives every + run a short-lived worker thread so a slow action can't starve subsequent + jobs. A job runs an action list or a pipeline, and can also be fired by a + file event, an event on the bus, the end of another pipeline's run, or by + hand; every firing leaves a run record. See :doc:`usage/scheduler`. Both dispatchers call :func:`automation_file.notify.manager.notify_on_failure` when an action diff --git a/docs/source/Eng/eng_index.rst b/docs/source/Eng/eng_index.rst index 8481a74..1d7b32e 100644 --- a/docs/source/Eng/eng_index.rst +++ b/docs/source/Eng/eng_index.rst @@ -179,14 +179,17 @@ Chapter 11 — Triggers and Scheduler =================================== File-watcher triggers (``FA_watch_*``) run an action list on a filesystem -event; the cron-style scheduler (``FA_schedule_*``) runs an action list on -a recurring schedule with overlap protection. +event. The scheduler (``FA_schedule_*``) runs an action list or a pipeline +when a trigger fires: a cron expression with a time zone, a manual call, a +file event, an event on the bus or the end of another pipeline. It records +every run and protects against overlap. .. toctree:: :maxdepth: 2 :caption: Triggers and Scheduler usage/events + usage/scheduler .. _eng-notifications: diff --git a/docs/source/Eng/usage/events.rst b/docs/source/Eng/usage/events.rst index 1e1dbf8..38d4851 100644 --- a/docs/source/Eng/usage/events.rst +++ b/docs/source/Eng/usage/events.rst @@ -28,31 +28,21 @@ lifecycle. Or drive it from a JSON action list with ``FA_watch_start`` / ``FA_watch_stop`` / ``FA_watch_stop_all`` / ``FA_watch_list``. -Cron scheduler --------------- - -Run an action list on a recurring schedule. The 5-field cron parser supports -``*``, exact values, ``a-b`` ranges, comma-separated lists, and ``*/n`` step -syntax with ``jan``..``dec`` / ``sun``..``sat`` aliases. - -.. code-block:: python - - from automation_file import schedule_add - - schedule_add( - name="nightly-snapshot", - cron_expression="0 2 * * *", # every day at 02:00 local time - action_list=[["FA_zip_dir", {"dir_we_want_to_zip": "/data", - "zip_name": "/backup/data_nightly"}]], - ) - -A background thread wakes on minute boundaries, so expressions with -sub-minute precision are not supported. JSON forms: ``FA_schedule_add`` / -``FA_schedule_remove`` / ``FA_schedule_remove_all`` / ``FA_schedule_list``. - -Both dispatchers call -:func:`~automation_file.notify.manager.notify_on_failure` when an action +Scheduler +--------- + +Running an action list or a pipeline on a cron expression with a time zone, on +a file event, on an event from the bus, after another pipeline or by hand is +described in :doc:`scheduler`, together with the run records, overlap +protection, timeouts and the ``FA_schedule_*`` actions. A job that should run on +a file event and leave a record of every run uses the scheduler's +``FileTrigger`` instead of ``FA_watch_start``. + +A watcher calls +:func:`~automation_file.notify.manager.notify_on_failure` when its action list raises :class:`~automation_file.exceptions.FileAutomationException`. The helper is a no-op when no sinks are registered, so auto-notification is an opt-in side effect of registering any -:class:`~automation_file.NotificationSink` — see :doc:`notifications`. +:class:`~automation_file.NotificationSink` — see :doc:`notifications`. The +scheduler does the same for an action list it cannot dispatch, and publishes +every other failed run as a ``scheduler.error`` event. diff --git a/docs/source/Eng/usage/notifications.rst b/docs/source/Eng/usage/notifications.rst index 57cd1ed..1dd4ad0 100644 --- a/docs/source/Eng/usage/notifications.rst +++ b/docs/source/Eng/usage/notifications.rst @@ -215,3 +215,8 @@ a ``SystemErrorEvent`` otherwise. Then: With the router active, the routes decide: a failure that no route matches is not delivered. A route such as ``Route("failures", types=("scheduler.error", "system.error"))`` keeps those alerts coming. + +The scheduler calls ``notify_on_failure`` for an action list it cannot dispatch. +A scheduled run that fails in any other way, or runs past its timeout, is +published by the scheduler itself as a ``scheduler.error`` event and reaches a +sink only through a route: see :doc:`scheduler`. diff --git a/docs/source/Eng/usage/pipeline.rst b/docs/source/Eng/usage/pipeline.rst index 3bbdcbf..2e32fb3 100644 --- a/docs/source/Eng/usage/pipeline.rst +++ b/docs/source/Eng/usage/pipeline.rst @@ -207,7 +207,9 @@ Options - Default parameters; ``run(params=...)`` adds to them and overrides them. * - ``schedule`` - A ``Schedule(cron, timezone=None)``. It is kept on ``pipeline.schedule`` - for the scheduler and not acted on by the pipeline. + for the scheduler and not acted on by the pipeline: + ``scheduler.add_pipeline(pipeline)`` turns it into a cron trigger + (:doc:`scheduler`). * - ``registry`` - Where action names are looked up. Default: the shared executor's registry. diff --git a/docs/source/Eng/usage/scheduler.rst b/docs/source/Eng/usage/scheduler.rst new file mode 100644 index 0000000..ebe2a95 --- /dev/null +++ b/docs/source/Eng/usage/scheduler.rst @@ -0,0 +1,753 @@ +Scheduler +========= + +``automation_file.scheduler`` runs an action list or a :doc:`pipeline ` +when something fires it: a cron expression with an optional time zone, a file +event, an event on the bus, the end of another pipeline's run, or a call. Cron is +one trigger among several, and every job goes through the same scheduler. + +Every firing leaves a record in one of seven states: ``scheduled``, ``started``, +``completed``, ``failed``, ``skipped``, ``timeout`` and ``cancelled``. A job does +not overlap itself unless it says so, a run can be given a timeout and can be +cancelled, and a run that fails or times out is published as a +``scheduler.error`` event (:doc:`event_bus`). + +Minimal example +--------------- + +.. code-block:: python + + from automation_file.scheduler import scheduler + + scheduler.add( + "nightly-snapshot", + "0 2 * * *", # every day at 02:00 ... + [["FA_zip_dir", {"dir_we_want_to_zip": "/data", + "zip_name": "/backup/data_nightly"}]], + timezone="Asia/Taipei", # ... in Taipei; local time without it + ) + + scheduler.history(job="nightly-snapshot", limit=5) # the latest runs, newest first + +``scheduler`` is the process-wide instance. Adding the first job starts its +background thread, which is a daemon thread: keep the process alive yourself. + +Production example +------------------ + +A pipeline every night at 02:00 Taipei time, a second pipeline that runs when the +first one has succeeded, and a message to the team when a run fails or takes too +long. + +.. code-block:: python + + import os + + from automation_file import Route, SlackSink, notification_manager, notification_router + from automation_file.pipeline import SQLiteRunStore, set_default_run_store + from automation_file.scheduler import PipelineTrigger, scheduler + + # Scheduled pipelines record their runs in the default run store. + set_default_run_store(SQLiteRunStore("/var/lib/automation/pipelines.db")) + + # A failed or timed-out run is a scheduler.error event; this route delivers it. + notification_manager.register(SlackSink(os.environ["SLACK_WEBHOOK"], name="team-alerts")) + notification_router.add_route( + Route("scheduler-failures", sinks=("team-alerts",), types=("scheduler.error",)) + ) + notification_router.start() + + # daily-report.yaml declares schedule: {cron: "0 2 * * *", timezone: Asia/Taipei} + scheduler.add_pipeline( + "pipelines/daily-report.yaml", + params={"date": "${date:%Y-%m-%d}"}, # the date in Taipei when the job fires + timeout=3600, # cancelled after an hour + ) + + # No schedule of its own: it runs whenever a run of daily-report has succeeded. + scheduler.add_pipeline( + "pipelines/publish-summary.yaml", + triggers=PipelineTrigger("daily-report"), + timeout=900, + ) + + for run in scheduler.history(state="failed", limit=10): + print(run.job, run.scheduled_at, run.error, run.correlation_id) + +``daily-report`` starts at 18:00 UTC, which is 02:00 in Taipei, with ``date`` set +to the Taipei date. When its run ends ``succeeded``, ``publish-summary`` is fired; +when it fails, ``publish-summary`` is not fired and the route sends +``[ERROR] scheduler.error: scheduler[daily-report] failed``. While a run of +``daily-report`` is still going, the next firing is recorded as ``skipped``. + +A pipeline that fails also publishes ``pipeline.failed`` and ``task.failed`` by +itself. Route either those or ``scheduler.error`` to a sink; a route with both +sends two messages for one failure. + +Jobs and targets +---------------- + +A job has a name, a target, and any number of triggers. + +**An action list** runs through the shared executor, one action after the other. +An action that raises fails the run, and the remaining actions still run, as in +``execute_action``. The values the actions return are not kept. + +**A pipeline** is a :class:`~automation_file.pipeline.Pipeline`, a definition +mapping, or the path of a ``.yaml`` / ``.yml`` / ``.json`` definition file. The +definition is checked when the job is registered, and a file is read once, at +that moment: remove the job and register it again after changing the file. Each +firing calls ``pipeline.run()`` with the default run store, so +``FA_pipeline_status`` and ``FA_pipeline_history`` see the run. + +``add_pipeline`` reads the pipeline's ``schedule`` and makes it a cron trigger, +time zone included; the job is named after the pipeline unless ``name`` is given. +``add_job`` takes a pipeline too and does not read its ``schedule``. + +``params`` are the parameters of every run of a pipeline job; they are added to +the pipeline's own defaults and override them. In a string, at any depth, +``${date:FORMAT}`` is replaced when the job fires (``FORMAT`` is a ``strftime`` +format; a bare ``${date}`` gives ``2026-10-08T02:00:00``). The time used is the +job's own: the zone of its first cron trigger, or local time when it has none. +Nothing else is replaced, and an action list takes no ``params``. + +Triggers +-------- + +.. list-table:: + :header-rows: 1 + :widths: 14 36 50 + + * - Kind + - In Python + - Fires + * - ``cron`` + - ``CronTrigger(cron, timezone=None)`` + - At the minutes the expression names. + * - ``manual`` + - ``scheduler.run_now(name)`` + - When it is called. Every job can be fired this way. + * - ``file`` + - ``FileTrigger(path, events=("created", "modified"), recursive=True)`` + - When a file under ``path`` is created, modified, deleted or moved. + * - ``event`` + - ``EventTrigger(types=(), sources=(), min_severity=Severity.INFO)`` + - When a matching event is published on the event bus. + * - ``pipeline`` + - ``PipelineTrigger(pipeline, when="on_success")`` + - When a run of the named pipeline has ended. + +.. code-block:: python + + from automation_file.scheduler import ( + CronTrigger, EventTrigger, FileTrigger, PipelineTrigger, scheduler, + ) + + scheduler.add_job( + "sweep-inbox", + [["FA_copy_all_file_to_dir", {"source_dir": "/data/inbox", + "target_dir": "/data/processed"}]], + triggers=[ + CronTrigger("*/30 * * * *", "UTC"), # every half hour + FileTrigger("/data/inbox", events=["created"]), # and on a new file + ], + ) + +A job may have several triggers, of any kinds, and overlap protection covers all +of them. A job without a trigger runs only when it is fired by hand. + +Cron +~~~~ + +Five fields: minute (0-59), hour (0-23), day of month (1-31), month (1-12) and +day of week (0-6, Sunday is 0 or 7). Each takes ``*``, a value, a range +``a-b``, a list ``a,b,c`` and a step ``*/n`` or ``a-b/n``; months and weekdays +also take ``jan``..``dec`` and ``sun``..``sat``. There are no seconds and no +``@daily`` aliases. + +The scheduler looks at the clock every second and handles each minute once. A +minute during which the process was not running, or the machine was asleep, is +not made up afterwards. See `Time zones and daylight saving time`_ for the zone. + +Manual +~~~~~~ + +.. code-block:: python + + run = scheduler.run_now("nightly-snapshot") # a JobRun + run.wait(600) # True once the run has ended + run.state # RunState.COMPLETED; equal to "completed" + +``run_now`` returns the record of the firing at once. The record is ``skipped`` +when the job is still running and does not allow overlap. From JSON, and so from +the HTTP action server, it is ``FA_schedule_run``. + +File events +~~~~~~~~~~~ + +``FileTrigger`` uses :class:`~automation_file.trigger.FileWatcher`, the watcher +behind ``FA_watch_start`` (:doc:`events`). The scheduler owns the watcher: it is +not listed by ``FA_watch_list`` and it stops when the job is removed. The record +of a run holds the file and the event in ``detail`` +(``{"path": "/data/inbox/a.csv", "event": "created"}``). + +Saving one file often produces several events. Each one is a firing; those that +arrive while the job is running are recorded as ``skipped``. + +Events and webhooks +~~~~~~~~~~~~~~~~~~~ + +``EventTrigger`` matches what the bus matches: type names (``"task.failed"``), +prefixes (``"pipeline.*"``) and event classes in ``types``, exact +``event.source`` values in ``sources``, and a lowest severity. At least one type +or one source is required. + +This is also how a webhook fires a job. The code that receives the request +publishes an event, and every job with a matching trigger runs: + +.. code-block:: python + + from dataclasses import dataclass + from typing import ClassVar + + from automation_file import Event, emit + from automation_file.scheduler import EventTrigger, scheduler + + @dataclass(frozen=True, kw_only=True) + class DeployFinished(Event): + type: ClassVar[str] = "deploy.finished" + + scheduler.add_job( + "smoke-test", + [["FA_execute_files", [["checks/smoke.json"]]]], + triggers=EventTrigger(types="deploy.finished", sources="webhook"), + ) + + # In the handler of your web framework, once the request has been verified: + emit(DeployFinished(source="webhook", subject="release 1.4 deployed")) + +A request to the HTTP action server (:doc:`servers`) can fire one job by name +instead: ``[["FA_schedule_run", {"name": "smoke-test"}]]``. That run is recorded +as ``manual``. + +The trigger's handler runs in the thread that published the event and only +starts the run. An event published by a run of the job itself does not fire the +job again, so a job that listens for ``scheduler.error`` does not loop on its +own failure. + +Jobs that fire one another through their events form a chain. The firing that +would make a chain longer than 16 runs starts nothing: it is recorded as +``skipped`` with the reason ``chain``, so two jobs that fire each other stop +instead of going on for ever. + +Pipeline dependency +~~~~~~~~~~~~~~~~~~~ + +``PipelineTrigger("daily-report")`` fires when a run of the pipeline named +``daily-report`` has ended. ``when`` uses the pipeline's own words: + +.. list-table:: + :header-rows: 1 + :widths: 22 78 + + * - ``when`` + - Fires when the run + * - ``"on_success"`` + - succeeded. This is the default. + * - ``"on_failure"`` + - failed or was cancelled. + * - ``"always"`` + - ended, however it ended. + +The trigger listens for ``pipeline.completed`` and ``pipeline.failed``, so it +sees every run of that pipeline on the scheduler's bus: one the scheduler +started, one started with ``pipeline.run()``, one started by ``FA_pipeline_run``. +The record's ``detail`` names the pipeline, the run ID and the status of the run +that fired it. A pipeline job cannot depend on its own pipeline, and two +pipelines that depend on each other are stopped by the chain limit above. + +Options +------- + +``scheduler.add(name, cron_expression, action_list, *, allow_overlap=False, timezone=None, timeout=None)`` + +``scheduler.add_job(name, target, *, triggers=None, allow_overlap=False, timeout=None, params=None)`` + +``scheduler.add_pipeline(pipeline, *, name=None, triggers=None, allow_overlap=False, timeout=None, params=None)`` + +.. list-table:: + :header-rows: 1 + :widths: 22 78 + + * - Option + - Meaning + * - ``name`` + - Identifies the job. A second job with the same name is refused. For + ``add_pipeline`` the default is the pipeline's name. + * - ``cron_expression`` + - The five fields of ``add``. + * - ``action_list`` / ``target`` / ``pipeline`` + - What the job runs. See `Jobs and targets`_. + * - ``triggers`` + - One trigger or several: objects, or their mappings (see `Actions`_). + * - ``allow_overlap`` + - Let a firing start while the job is still running. Default ``False``. + * - ``timezone`` + - The zone ``add`` reads its expression in. Default: local time. + * - ``timeout`` + - Seconds a run may take, above 0. Default: no limit. See `Timeout`_. + * - ``params`` + - Parameters of a pipeline job's runs. + +Each of the three returns the job's snapshot, the mapping ``scheduler.list()`` +holds for every job: + +.. list-table:: + :header-rows: 1 + :widths: 22 78 + + * - Key + - Value + * - ``name`` + - The job's name. + * - ``cron``, ``timezone`` + - The expression and the zone of the first cron trigger; ``""`` and + ``None`` when the job has none. + * - ``triggers`` + - Every trigger as a mapping. + * - ``target``, ``pipeline``, ``actions`` + - ``"actions"`` or ``"pipeline"``, the pipeline's name, and the number of + actions in the list. + * - ``allow_overlap``, ``timeout`` + - As given. + * - ``runs``, ``skipped`` + - How many firings were started, and how many were skipped. + * - ``running`` + - Whether a run's thread is still alive. See `Timeout`_. + * - ``last_run`` + - When the last run was fired, in the job's own time: with an offset when + the job has a time zone, in local time without one. + * - ``last_state`` + - The state the last run ended in. + +.. list-table:: + :header-rows: 1 + :widths: 34 66 + + * - Call + - Does + * - ``remove(name)``, ``remove_all()`` + - Remove jobs and stop their triggers. A run in progress goes on. + * - ``list()`` + - The snapshot of every job. + * - ``run_now(name)`` + - Fire a job by hand; returns the ``JobRun``. + * - ``cancel(name)`` + - Cancel the job's runs in progress; returns their records. + * - ``history(job=None, state=None, limit=50)`` + - The latest records, newest first. + * - ``start()``, ``shutdown(timeout=5.0, *, cancel_running=False)`` + - See `Starting and stopping`_. + * - ``tick(now=None)`` + - Handle the current minute and the timeouts once, for a scheduler that is + driven by hand. See `Starting and stopping`_. + +Run records and states +---------------------- + +``scheduler.history()`` returns :class:`~automation_file.scheduler.JobRun` +objects; ``run.to_dict()`` is JSON-serialisable. + +.. list-table:: + :header-rows: 1 + :widths: 22 78 + + * - Field + - Meaning + * - ``run_id`` + - The ID of this firing. + * - ``job`` + - The job's name. + * - ``trigger`` + - ``cron``, ``manual``, ``file``, ``event`` or ``pipeline``. + * - ``detail`` + - What fired the run: the expression and zone, the file and event, the + event's type, ID, source and subject, or the pipeline, run ID and status. + * - ``target``, ``pipeline`` + - ``"actions"`` or ``"pipeline"``, and the pipeline's name. + * - ``state`` + - A ``RunState``. It compares equal to its text. + * - ``scheduled_at`` + - When the run was due: the minute, for cron. + * - ``started_at``, ``finished_at`` + - When the target started and when the record was closed. + * - ``duration_ms`` + - The time between the two. + * - ``error`` + - What went wrong, for ``failed`` and ``timeout``. + * - ``reason`` + - ``overlap`` or ``chain`` for a skipped firing, ``cancelled`` for a + cancelled run. + * - ``correlation_id`` + - The ID every event of the run carries. For an action list it is + ``run_id``. For a pipeline it becomes the pipeline's run ID as soon as the + pipeline starts, which is the ID ``FA_pipeline_status`` takes. + +All times are timezone-aware UTC. The target runs inside +``correlation_scope(run_id)`` with the actor ``scheduler``, so storage errors +and audit records of the run carry the same ID. + +.. list-table:: + :header-rows: 1 + :widths: 18 82 + + * - State + - Meaning + * - ``scheduled`` + - The firing was accepted and its thread has not begun yet. + * - ``started`` + - The target is running. + * - ``completed`` + - Every action returned, or the pipeline's run succeeded. + * - ``failed`` + - At least one action raised, the pipeline's run did not succeed, or the + target could not be run at all. ``error`` says which. + * - ``skipped`` + - Nothing was started: the job was still running and does not allow overlap, + or the firing came at the end of a chain of 16 runs. + * - ``timeout`` + - The run did not end within its timeout. + * - ``cancelled`` + - ``cancel`` stopped the run. + +The history is kept in memory, per scheduler, and holds the latest 1000 records +(``Scheduler(history_limit=...)``); the oldest go first. It does not survive the +process. The runs of a scheduled pipeline are also in the pipeline's run store, +and every failure is an event that the :doc:`audit trail ` can keep. + +Overlap protection +------------------ + +Unless a job was registered with ``allow_overlap=True``, a firing that arrives +while the job is running starts nothing. It is recorded as ``skipped`` with the +reason ``overlap``, counted in the job's ``skipped``, and logged as a warning. +This holds for every kind of trigger, ``run_now`` included. + +A job counts as running until the thread of its run has really ended. After a +timeout or a cancellation that can be later than the record says. + +Timeout +------- + +``timeout`` is the number of seconds a run may take, counted from the moment it +was fired. The scheduler checks once a second. When the time is up, the record +becomes ``timeout``, a ``scheduler.error`` event is published, and the run is +told to stop: + +* a pipeline is cancelled through its cancellation token, exactly as + ``run.cancel()`` does: tasks that have not started become ``cancelled`` and + running tasks are told through ``ctx.cancel``; +* an action list stops before its next action. + +**A thread cannot be killed.** The action that is running, or a pipeline task +that does not look at its token, goes on until it returns. The job keeps +counting as running until then, so the next firing is skipped and two runs never +collide. What the thread does after the timeout does not change the record. + +Give a pipeline's tasks timeouts of their own as well (:doc:`pipeline`): a task +timeout fails one task and lets clean-up tasks run, while the job's timeout +stops the whole run. + +Cancellation +------------ + +.. code-block:: python + + cancelled = scheduler.cancel("daily-report") # the records, now "cancelled" + +``cancel(name)`` closes the records of the job's runs in progress as +``cancelled`` and stops the runs the same way a timeout does. It returns an empty +list when the job is not running, and raises ``SchedulerException`` for a name +that is neither registered nor running. A cancelled run publishes no +``scheduler.error``. Removing a job does not cancel a run in progress. + +Time zones and daylight saving time +----------------------------------- + +``timezone`` is an IANA name such as ``Asia/Taipei`` or ``America/New_York``, or +``UTC``. Zone data comes from the standard library's ``zoneinfo``, which reads +the system's database. Windows has none: install the ``tzdata`` package there +(``pip install tzdata``). ``UTC`` works without it. A name that cannot be found +raises ``CronException`` when the job is registered. + +A cron trigger without a time zone is read in the local time of the machine, as +the scheduler always did. In a container that is usually UTC: give production +jobs a zone. + +In a zone with daylight saving time: + +* **a local time that does not exist is not fired.** On the day the clocks go + from 02:00 to 03:00, ``30 2 * * *`` does not run; +* **a local time that occurs twice fires once**, the first time. On the day the + clocks go back from 02:00 to 01:00, ``30 1 * * *`` runs once; +* an expression whose hour field is ``*`` runs every hour anyway, so it keeps + firing through the repeated hour: ``*/15 * * * *`` runs every 15 minutes of + real time on both days. + +A job that must run at a fixed interval, or exactly once a day whatever the +clocks do, is best scheduled in ``UTC``. A trigger without a time zone follows +the system clock and gets none of this treatment: the repeated hour fires twice. + +Events +------ + +A run that ends ``failed`` or ``timeout`` publishes one ``SchedulerError`` +(``scheduler.error``, severity ``error``, source ``scheduler``) on the +scheduler's bus. Its subject is ``scheduler[] failed`` or +``scheduler[] timed out`` and its correlation ID is the record's +``correlation_id``. + +.. list-table:: + :header-rows: 1 + :widths: 24 76 + + * - Payload key + - Value + * - ``job`` + - The job's name. + * - ``trigger`` + - What fired the run. + * - ``status`` + - ``failed`` or ``timeout``. + * - ``error`` + - The record's ``error``. URLs in it are cut down to their host. + * - ``target`` + - ``actions`` or ``pipeline``. + * - ``scheduler_run_id`` + - The record's ``run_id``. + * - ``duration_ms`` + - Present when the target had started. + * - ``pipeline``, ``run_id`` + - For a pipeline target: its name, and the pipeline's run ID once the + pipeline had started. + +``completed``, ``skipped`` and ``cancelled`` publish nothing; they are in the +history and in the log. + +One failure is reported differently. An action list that the executor does not +accept at all (an empty list, something that is not a list) is reported through +:func:`~automation_file.notify.manager.notify_on_failure`, as before. It +publishes the ``scheduler.error`` event itself, with the payload ``job``, +``status`` (``error``), ``error`` and ``context``, on the process-wide bus; and +while the notification router is not active it also sends the message straight +to every registered sink (:doc:`notifications`). Every other failure is an event +only: add a route for ``scheduler.error`` to be told about it. + +Actions +------- + +.. list-table:: + :header-rows: 1 + :widths: 26 44 30 + + * - Action + - Parameters + - Returns + * - ``FA_schedule_add`` + - ``name, cron_expression, action_list, allow_overlap=False, timezone=None, + timeout=None`` + - The job's snapshot + * - ``FA_schedule_job`` + - ``name, action_list, triggers=None, allow_overlap=False, timeout=None`` + - The job's snapshot + * - ``FA_schedule_pipeline`` + - ``definition, name=None, triggers=None, params=None, allow_overlap=False, + timeout=None`` + - The job's snapshot + * - ``FA_schedule_run`` + - ``name`` + - The record of the firing + * - ``FA_schedule_cancel`` + - ``name`` + - The cancelled records + * - ``FA_schedule_history`` + - ``job=None, state=None, limit=50`` + - Records, newest first + * - ``FA_schedule_list`` + - (none) + - Every job's snapshot + * - ``FA_schedule_remove`` + - ``name`` + - The removed job's snapshot + * - ``FA_schedule_remove_all`` + - (none) + - The removed jobs' snapshots + +They act on the process-wide ``scheduler``. ``definition`` is a mapping or the +path of a definition file, and its ``schedule`` becomes a cron trigger. In +``FA_schedule_add``, ``allow_overlap``, ``timezone`` and ``timeout`` are passed by +name. A trigger is a mapping with its ``kind`` and that kind's arguments: + +.. list-table:: + :header-rows: 1 + :widths: 14 86 + + * - ``kind`` + - Keys + * - ``cron`` + - ``cron`` (required), ``timezone`` + * - ``file`` + - ``path`` (required), ``events``, ``recursive`` + * - ``event`` + - ``types``, ``sources``, ``min_severity``; types are names and prefixes + * - ``pipeline`` + - ``pipeline`` (required), ``when`` + +.. code-block:: json + + [ + ["FA_schedule_pipeline", {"definition": "pipelines/daily-report.yaml", + "params": {"date": "${date:%Y-%m-%d}"}, + "timeout": 3600}], + ["FA_schedule_pipeline", {"definition": "pipelines/publish-summary.yaml", + "triggers": [{"kind": "pipeline", + "pipeline": "daily-report"}]}], + ["FA_schedule_job", {"name": "sweep-inbox", + "action_list": [["FA_copy_all_file_to_dir", + {"source_dir": "/data/inbox", + "target_dir": "/data/processed"}]], + "triggers": [{"kind": "file", "path": "/data/inbox", + "events": ["created"]}]}], + ["FA_schedule_run", {"name": "daily-report"}], + ["FA_schedule_history", {"job": "daily-report", "limit": 5}] + ] + +``register_scheduler_ops(registry)`` adds the actions to a registry of your own. + +A job names the actions it will run later. As long as the action list or the +definition is part of the request, an :class:`~automation_file.ActionACL` on a +TCP or HTTP action server checks those names too. It cannot see inside a +definition given as a file path, and ``FA_schedule_run`` runs whatever a job that +is already registered holds: allow ``FA_schedule_pipeline`` and +``FA_schedule_run`` only for clients that may call every registered action. + +Starting and stopping +--------------------- + +.. code-block:: python + + from automation_file.scheduler import Scheduler + + scheduler = Scheduler(history_limit=5000) # a scheduler of your own + scheduler.shutdown() # stop the thread and every trigger + scheduler.start() # ... and bring them back + +``Scheduler(*, clock=None, bus=None, history_limit=1000, autostart=True)`` + +.. list-table:: + :header-rows: 1 + :widths: 22 78 + + * - Option + - Meaning + * - ``clock`` + - A callable returning the current time as an aware ``datetime``. Default: + the system clock in UTC. + * - ``bus`` + - The ``EventBus`` the scheduler listens and reports on, and the one its + pipelines publish on. Default: the process-wide bus. + * - ``history_limit`` + - How many records are kept. Default ``1000``. + * - ``autostart`` + - Start the background thread when a job is added. Default ``True``. + +``shutdown()`` stops the background thread, every file watcher and every bus +subscription the scheduler made. The jobs stay registered; ``start()``, or adding +another job, arms them again. Runs in progress are left to finish unless +``shutdown(cancel_running=True)`` is used. All the scheduler's threads are daemon +threads, so they never keep the interpreter alive. + +With ``autostart=False`` nothing happens by itself: ``tick(now)`` handles the +minute of ``now`` and the timeouts due at ``now``, and returns the records of +what it fired. That is how to test a schedule without waiting for it: + +.. code-block:: python + + from datetime import datetime, timezone + + from automation_file.scheduler import Scheduler + + engine = Scheduler(autostart=False) + engine.add("nightly", "0 2 * * *", [["FA_schedule_list"]], timezone="Asia/Taipei") + engine.tick(datetime(2026, 10, 7, 17, 59, tzinfo=timezone.utc)) # [] + (run,) = engine.tick(datetime(2026, 10, 7, 18, 0, tzinfo=timezone.utc)) + run.wait(10) + run.state # "completed" + +When something goes wrong +------------------------- + +A job did not run at its time + Look for a ``skipped`` record first: the previous run was still going. If + there is no record at all, the minute was not handled: the process was not + running or the machine was asleep (minutes are not made up), the local time + did not exist that day, or the expression is read in another zone than you + think. A trigger without ``timezone`` uses the machine's local time. + +``CronException: unknown time zone`` + The name is misspelt, or the machine has no zone data: on Windows, install + ``tzdata``. + +A run is ``failed`` although most of it worked + One action raised and the rest of the list ran. ``error`` lists the actions + that failed, by position and name. + +A run is ``completed`` although the work went wrong + A run fails only when an action raises or a pipeline's run does not succeed. + An action that reports through its return value completes: see the same + entry in :doc:`pipeline`. + +A run is ``timeout`` or ``cancelled`` but the job still shows ``running`` + Its thread could not be stopped and is still in the action or the task it + was in. The job stays busy, and its firings are skipped, until that returns. + Give long transfers a timeout of their own, and make long pipeline tasks + watch ``ctx.cancel``. + +A job fires more often than expected + One saved file is several file events; two triggers of one job both fire it; + a job with ``allow_overlap=True`` is not held back by a run in progress. + +A record is ``skipped`` with the reason ``chain`` + Sixteen runs fired one another in a row: two jobs listen for each other's + events, or two pipelines depend on each other. Break the cycle; a job that + has to repeat belongs on a cron trigger. + +A dependent pipeline never fires + The name in ``PipelineTrigger`` is not the upstream pipeline's ``name``, the + upstream run did not end the way ``when`` asks for, or it published on + another bus than the scheduler's. + +An event trigger does not fire + Check ``types``, ``sources`` and ``min_severity`` against + ``event_bus.recent()``. Events published by the job's own run are ignored on + purpose. + +No notification arrived + The router must be started and must have a route that matches + ``scheduler.error``. Without the router, only an action list the executor + rejects is sent straight to the sinks. + +The history is empty + It is kept in memory and ends with the process; a scheduler of your own has + a history of its own. Pipeline runs are in the run store. + +Nothing fires after ``shutdown()`` + The jobs are still registered but their triggers are stopped. Call + ``start()``. + +``SchedulerException`` is raised for a duplicate or unknown job, a wrong +trigger, a wrong timeout and a wrong history query, and ``CronException`` for an +expression or a time zone that cannot be understood. A definition that is wrong +raises ``PipelineDefinitionException`` and a watch path that does not exist +``TriggerException``, both when the job is registered. All of them derive from +``FileAutomationException``. diff --git a/docs/source/Zh-CN/architecture.rst b/docs/source/Zh-CN/architecture.rst index 5be3522..c6378e3 100644 --- a/docs/source/Zh-CN/architecture.rst +++ b/docs/source/Zh-CN/architecture.rst @@ -63,7 +63,7 @@ subgraph Events["事件驱动"] Trigger["TriggerManager
watchdog 文件监听"] - Sched["Scheduler
5-field cron + overlap guard"] + Sched["Scheduler
cron · file · event · pipeline triggers
run records + overlap guard"] end subgraph Servers["服务器"] @@ -318,8 +318,13 @@ ├── trigger/ │ └── manager.py # FileWatcher + TriggerManager（基于 watchdog） ├── scheduler/ - │ ├── cron.py # 5 字段 cron 表达式解析器 - │ └── manager.py # Scheduler 后台线程 + ScheduledJob + │ ├── cron.py # 5 字段 cron 表达式解析器、时区 + │ ├── triggers.py # cron / 文件 / 事件 / 流水线触发器 + │ ├── job.py # ScheduledJob + │ ├── runs.py # JobRun、RunState、RunHistory + │ ├── targets.py # 执行动作列表或流水线 + │ ├── dispatch.py # 重叠、超时、取消、scheduler.error + │ └── manager.py # Scheduler 后台线程 + FA_schedule_* 动作 ├── notify/ │ ├── sinks.py # Webhook / Slack / Email sink │ └── manager.py # NotificationManager（扇出 + 去重 + auto-notify hook） @@ -398,9 +403,12 @@ 转发给共享注册表调度的动作列表。 :data:`~automation_file.trigger.trigger_manager` 持有 name → watcher 映射，让 GUI 与 JSON 动作共享同一个生命周期。 -* :mod:`automation_file.scheduler` 运行一个后台线程，在分钟边界唤醒、 - 遍历已注册的 :class:`~automation_file.scheduler.ScheduledJob`，并在 - 短生命周期的工作线程上调度每个匹配的任务，避免慢动作拖累后续任务。 +* :mod:`automation_file.scheduler` 运行一个后台线程，每秒唤醒一次，触发 + cron 触发器在当前这一分钟到期的 + :class:`~automation_file.scheduler.ScheduledJob`，并让每次运行使用自己的 + 短生命周期工作线程，避免慢动作拖累后续任务。作业可以运行动作列表或流水线， + 也可以由文件事件、事件总线上的事件、另一条流水线的运行结束或手动触发；每次 + 触发都会留下一条运行记录。详见 :doc:`usage/scheduler`。 当动作列表抛出 :class:`~automation_file.exceptions.FileAutomationException` 时，两个调度器都会调用 diff --git a/docs/source/Zh-CN/usage/events.rst b/docs/source/Zh-CN/usage/events.rst index a5763bd..f4bc694 100644 --- a/docs/source/Zh-CN/usage/events.rst +++ b/docs/source/Zh-CN/usage/events.rst @@ -27,32 +27,19 @@ 也可以从 JSON 动作列表里调用 ``FA_watch_start`` / ``FA_watch_stop`` / ``FA_watch_stop_all`` / ``FA_watch_list``。 -Cron 调度器 ------------ +调度器 +------------ -按重复时刻执行动作列表。5 字段 cron 解析器支持 -``*``、精确值、``a-b`` 区间、逗号分隔列表与 ``*/n`` 步长， -也支持 ``jan``..``dec`` / ``sun``..``sat`` 别名。 - -.. code-block:: python - - from automation_file import schedule_add - - schedule_add( - name="nightly-snapshot", - cron_expression="0 2 * * *", # 每天本地时间 02:00 - action_list=[["FA_zip_dir", {"dir_we_want_to_zip": "/data", - "zip_name": "/backup/data_nightly"}]], - ) - -后台线程在每分钟边界唤醒，因此不支持小于一分钟的精度。 -JSON 形式：``FA_schedule_add`` / ``FA_schedule_remove`` / -``FA_schedule_remove_all`` / ``FA_schedule_list``。 +按带时区的 cron 表达式、文件事件、事件总线上的事件、另一条流水线结束之后，或以手动 +方式运行动作列表或流水线，都写在 :doc:`scheduler` 中，其中也说明了运行记录、重叠保护、 +超时与 ``FA_schedule_*`` 动作。需要在文件事件发生时运行并为每次运行留下记录的作业， +请使用调度器的 ``FileTrigger``，而不是 ``FA_watch_start``。 当动作列表抛出 :class:`~automation_file.exceptions.FileAutomationException` 时， -两个调度器都会调用 +监听器会调用 :func:`~automation_file.notify.manager.notify_on_failure`。 若未注册任何 sink，该助手是 no-op，因此自动通知是 注册 :class:`~automation_file.NotificationSink` 的可选副作用—— -详见 :doc:`notifications`。 +详见 :doc:`notifications`。调度器对它无法分派的动作列表也会这么做，其他每一次 +失败的运行则发布成 ``scheduler.error`` 事件。 diff --git a/docs/source/Zh-CN/usage/notifications.rst b/docs/source/Zh-CN/usage/notifications.rst index f33bd81..a48a50a 100644 --- a/docs/source/Zh-CN/usage/notifications.rst +++ b/docs/source/Zh-CN/usage/notifications.rst @@ -204,3 +204,7 @@ sink 收到的内容 路由器工作时由路由决定：没有任何路由匹配的失败不会被投递。像 ``Route("failures", types=("scheduler.error", "system.error"))`` 这样的路由可以 让这些告警持续送达。 + +调度器会为它无法分派的动作列表调用 ``notify_on_failure``。以其他任何方式失败、或 +超过超时的调度运行，则由调度器自己发布成 ``scheduler.error`` 事件，只有通过路由才会 +送到 sink：见 :doc:`scheduler`。 diff --git a/docs/source/Zh-CN/usage/pipeline.rst b/docs/source/Zh-CN/usage/pipeline.rst index ddd8fff..f6d5379 100644 --- a/docs/source/Zh-CN/usage/pipeline.rst +++ b/docs/source/Zh-CN/usage/pipeline.rst @@ -196,7 +196,8 @@ - 默认参数；``run(params=...)`` 会加入并覆盖它们。 * - ``schedule`` - ``Schedule(cron, timezone=None)``。保存在 ``pipeline.schedule`` 供调度器 - 使用，流水线本身不会据此行动。 + 使用，流水线本身不会据此行动：``scheduler.add_pipeline(pipeline)`` 会把它 + 变成 cron 触发器（见 :doc:`scheduler`）。 * - ``registry`` - 查找动作名称的地方。默认：共享执行器的注册表。 diff --git a/docs/source/Zh-CN/usage/scheduler.rst b/docs/source/Zh-CN/usage/scheduler.rst new file mode 100644 index 0000000..8b434cb --- /dev/null +++ b/docs/source/Zh-CN/usage/scheduler.rst @@ -0,0 +1,703 @@ +调度器（Scheduler） +======================== + +``automation_file.scheduler`` 会在某件事触发时执行一份动作列表或一条 +:doc:`流水线 `\ ：可以带时区的 cron 表达式、文件事件、事件总线上的事件、 +另一条流水线的运行结束，或是一次调用。cron 只是多种触发器之一，所有作业都经过同一个 +调度器。 + +每一次触发都会留下一条记录，状态是七种之一：``scheduled``、``started``、 +``completed``、``failed``、``skipped``、``timeout`` 与 ``cancelled``。除非作业自己 +允许，否则它不会与自己重叠；一次运行可以设置超时，也可以被取消；失败或超时的运行会 +发布成 ``scheduler.error`` 事件（见 :doc:`event_bus`）。 + +最小示例 +---------------- + +.. code-block:: python + + from automation_file.scheduler import scheduler + + scheduler.add( + "nightly-snapshot", + "0 2 * * *", # 每天 02:00 ... + [["FA_zip_dir", {"dir_we_want_to_zip": "/data", + "zip_name": "/backup/data_nightly"}]], + timezone="Asia/Taipei", # ... 台北时间；不给就是本地时间 + ) + + scheduler.history(job="nightly-snapshot", limit=5) # 最近的运行，最新的在前 + +``scheduler`` 是整个进程共用的实例。加入第一个作业时会启动它的后台线程；那是 +daemon 线程，进程要靠你自己保持存活。 + +生产环境示例 +------------------------ + +每晚台北时间 02:00 运行一条流水线，第一条成功之后运行第二条流水线，并在某次运行失败 +或跑得太久时通知团队。 + +.. code-block:: python + + import os + + from automation_file import Route, SlackSink, notification_manager, notification_router + from automation_file.pipeline import SQLiteRunStore, set_default_run_store + from automation_file.scheduler import PipelineTrigger, scheduler + + # 被调度的流水线把运行记录写进默认的运行记录存储。 + set_default_run_store(SQLiteRunStore("/var/lib/automation/pipelines.db")) + + # 失败或超时的运行是一个 scheduler.error 事件；这条路由负责投递它。 + notification_manager.register(SlackSink(os.environ["SLACK_WEBHOOK"], name="team-alerts")) + notification_router.add_route( + Route("scheduler-failures", sinks=("team-alerts",), types=("scheduler.error",)) + ) + notification_router.start() + + # daily-report.yaml 声明了 schedule: {cron: "0 2 * * *", timezone: Asia/Taipei} + scheduler.add_pipeline( + "pipelines/daily-report.yaml", + params={"date": "${date:%Y-%m-%d}"}, # 作业触发当时的台北日期 + timeout=3600, # 一小时后取消 + ) + + # 没有自己的调度：每当 daily-report 的一次运行成功，它就会运行。 + scheduler.add_pipeline( + "pipelines/publish-summary.yaml", + triggers=PipelineTrigger("daily-report"), + timeout=900, + ) + + for run in scheduler.history(state="failed", limit=10): + print(run.job, run.scheduled_at, run.error, run.correlation_id) + +``daily-report`` 在 UTC 18:00 启动，也就是台北的 02:00，``date`` 则是台北的日期。它的 +运行以 ``succeeded`` 结束时，``publish-summary`` 会被触发；它失败时，``publish-summary`` +不会被触发，路由则发出 ``[ERROR] scheduler.error: scheduler[daily-report] failed``。 +``daily-report`` 的一次运行还在进行时，下一次触发会记录为 ``skipped``。 + +失败的流水线自己也会发布 ``pipeline.failed`` 与 ``task.failed``。请把这两者或 +``scheduler.error`` 其中一边路由到 sink；同时涵盖两边的路由会为一次失败发出两条消息。 + +作业与目标 +-------------------- + +一个作业有名称、目标，以及任意数量的触发器。 + +**动作列表**\ 通过共享的执行器执行，一个动作接着一个动作。抛出异常的动作会让这次 +运行失败，其余的动作仍会执行，与 ``execute_action`` 相同。动作的返回值不会保留。 + +**流水线**\ 可以是 :class:`~automation_file.pipeline.Pipeline`、定义的映射，或是 +``.yaml`` / ``.yml`` / ``.json`` 定义文件的路径。定义在作业注册时检查，文件也只在那一刻 +读取一次：修改文件之后，请移除作业再重新注册。每一次触发都以默认的运行记录存储调用 +``pipeline.run()``，所以 ``FA_pipeline_status`` 与 ``FA_pipeline_history`` 看得到这次 +运行。 + +``add_pipeline`` 会读取流水线的 ``schedule``，把它变成 cron 触发器，时区也一并带入； +除非给了 ``name``，作业会以流水线的名称命名。``add_job`` 也接受流水线，但不会读取它的 +``schedule``。 + +``params`` 是流水线作业每一次运行的参数；它们会加到流水线自己的默认值上并覆盖同名的 +项目。在任何深度的字符串中，``${date:FORMAT}`` 会在作业触发时被替换（``FORMAT`` 是 +``strftime`` 格式；单独的 ``${date}`` 会得到 ``2026-10-08T02:00:00``）。使用的是作业 +自己的时间：第一个 cron 触发器的时区，没有 cron 触发器时则是本地时间。除此之外不会 +替换任何东西，动作列表也不接受 ``params``。 + +触发器 +------------ + +.. list-table:: + :header-rows: 1 + :widths: 14 36 50 + + * - 种类 + - Python 写法 + - 何时触发 + * - ``cron`` + - ``CronTrigger(cron, timezone=None)`` + - 在表达式指定的那些分钟。 + * - ``manual`` + - ``scheduler.run_now(name)`` + - 被调用时。每个作业都能这样触发。 + * - ``file`` + - ``FileTrigger(path, events=("created", "modified"), recursive=True)`` + - ``path`` 下面的文件被创建、修改、删除或移动时。 + * - ``event`` + - ``EventTrigger(types=(), sources=(), min_severity=Severity.INFO)`` + - 事件总线上发布了符合条件的事件时。 + * - ``pipeline`` + - ``PipelineTrigger(pipeline, when="on_success")`` + - 指定名称的流水线有一次运行结束时。 + +.. code-block:: python + + from automation_file.scheduler import ( + CronTrigger, EventTrigger, FileTrigger, PipelineTrigger, scheduler, + ) + + scheduler.add_job( + "sweep-inbox", + [["FA_copy_all_file_to_dir", {"source_dir": "/data/inbox", + "target_dir": "/data/processed"}]], + triggers=[ + CronTrigger("*/30 * * * *", "UTC"), # 每半小时 + FileTrigger("/data/inbox", events=["created"]), # 以及有新文件时 + ], + ) + +一个作业可以有好几个任何种类的触发器，重叠保护涵盖全部。没有触发器的作业只在手动 +触发时运行。 + +Cron +~~~~ + +五个字段：分（0-59）、时（0-23）、日（1-31）、月（1-12）与星期（0-6，星期日是 0 或 +7）。每个字段都接受 ``*``、单个值、区间 ``a-b``、列表 ``a,b,c``，以及步长 ``*/n`` 或 +``a-b/n``；月份与星期还接受 ``jan``..``dec`` 与 ``sun``..``sat``。没有秒，也没有 +``@daily`` 这类别名。 + +调度器每秒看一次时钟，每一分钟只处理一次。进程没在运行或机器在休眠的那些分钟，事后 +不会补跑。时区请见 `时区与夏令时`_。 + +手动 +~~~~~~~~ + +.. code-block:: python + + run = scheduler.run_now("nightly-snapshot") # 一个 JobRun + run.wait(600) # 运行结束后为 True + run.state # RunState.COMPLETED；等于 "completed" + +``run_now`` 会立刻返回这次触发的记录。作业还在运行且不允许重叠时，记录是 +``skipped``。在 JSON 中，也就是通过 HTTP 动作服务器时，它是 ``FA_schedule_run``。 + +文件事件 +~~~~~~~~~~~~~~~~ + +``FileTrigger`` 使用 :class:`~automation_file.trigger.FileWatcher`，也就是 +``FA_watch_start`` 背后的监听器（见 :doc:`events`）。监听器归调度器所有：它不会出现在 +``FA_watch_list`` 中，作业被移除时它就停止。运行记录的 ``detail`` 记着文件与事件 +（``{"path": "/data/inbox/a.csv", "event": "created"}``）。 + +保存一个文件常常会产生好几个事件。每一个都是一次触发；作业还在运行时到达的那些会 +记录为 ``skipped``。 + +事件与 webhook +~~~~~~~~~~~~~~~~~~~~~~~~ + +``EventTrigger`` 匹配的方式与总线相同：``types`` 中放 type 名称（``"task.failed"``）、 +前缀（``"pipeline.*"``）与事件类，``sources`` 中放完全相同的 ``event.source`` 值， +另外还有最低严重程度。至少要给一个 type 或一个来源。 + +webhook 也是这样触发作业的。收到请求的代码发布一个事件，每个触发器匹配的作业都会 +运行： + +.. code-block:: python + + from dataclasses import dataclass + from typing import ClassVar + + from automation_file import Event, emit + from automation_file.scheduler import EventTrigger, scheduler + + @dataclass(frozen=True, kw_only=True) + class DeployFinished(Event): + type: ClassVar[str] = "deploy.finished" + + scheduler.add_job( + "smoke-test", + [["FA_execute_files", [["checks/smoke.json"]]]], + triggers=EventTrigger(types="deploy.finished", sources="webhook"), + ) + + # 在你的 Web 框架的处理函数中，请求通过验证之后： + emit(DeployFinished(source="webhook", subject="release 1.4 deployed")) + +对 HTTP 动作服务器（见 :doc:`servers`）的请求也可以改用名称触发单个作业： +``[["FA_schedule_run", {"name": "smoke-test"}]]``。这样的运行会记录为 ``manual``。 + +触发器的处理函数在发布事件的线程中执行，而且只负责启动这次运行。由作业自己的运行 +所发布的事件不会再次触发该作业，所以监听 ``scheduler.error`` 的作业不会因为自己的 +失败而不断循环。 + +通过事件互相触发的作业会形成一条链。会让一条链超过 16 次运行的那次触发不会启动任何 +东西：它会记录为 ``skipped``，原因是 ``chain``，所以互相触发的两个作业会停下来，而不是 +永远持续下去。 + +流水线依赖 +~~~~~~~~~~~~~~~~~~~~ + +``PipelineTrigger("daily-report")`` 会在名为 ``daily-report`` 的流水线有一次运行结束时 +触发。``when`` 沿用流水线自己的用词： + +.. list-table:: + :header-rows: 1 + :widths: 22 78 + + * - ``when`` + - 在这次运行……时触发 + * - ``"on_success"`` + - 成功。这是默认值。 + * - ``"on_failure"`` + - 失败或被取消。 + * - ``"always"`` + - 结束，不论结果如何。 + +这个触发器监听 ``pipeline.completed`` 与 ``pipeline.failed``，所以它看得到调度器的 +总线上该流水线的每一次运行：调度器启动的、用 ``pipeline.run()`` 启动的，以及由 +``FA_pipeline_run`` 启动的。记录的 ``detail`` 记着触发它的那次运行的流水线、运行 ID 与 +状态。流水线作业不能依赖自己的流水线，而互相依赖的两条流水线会被上述的链长度上限挡下。 + +选项 +-------- + +``scheduler.add(name, cron_expression, action_list, *, allow_overlap=False, timezone=None, timeout=None)`` + +``scheduler.add_job(name, target, *, triggers=None, allow_overlap=False, timeout=None, params=None)`` + +``scheduler.add_pipeline(pipeline, *, name=None, triggers=None, allow_overlap=False, timeout=None, params=None)`` + +.. list-table:: + :header-rows: 1 + :widths: 22 78 + + * - 选项 + - 含义 + * - ``name`` + - 标识这个作业。第二个同名的作业会被拒绝。``add_pipeline`` 的默认值是流水线的 + 名称。 + * - ``cron_expression`` + - ``add`` 的五个字段。 + * - ``action_list`` / ``target`` / ``pipeline`` + - 作业要运行的东西。见 `作业与目标`_。 + * - ``triggers`` + - 一个或多个触发器：对象，或它们的映射（见 `动作`_）。 + * - ``allow_overlap`` + - 允许作业还在运行时就启动新的触发。默认 ``False``。 + * - ``timezone`` + - ``add`` 解读表达式所用的时区。默认：本地时间。 + * - ``timeout`` + - 一次运行可以花的秒数，必须大于 0。默认：不限制。见 `超时`_。 + * - ``params`` + - 流水线作业每次运行的参数。 + +这三个方法都返回作业的快照，也就是 ``scheduler.list()`` 为每个作业保存的映射： + +.. list-table:: + :header-rows: 1 + :widths: 22 78 + + * - 键 + - 值 + * - ``name`` + - 作业的名称。 + * - ``cron``、``timezone`` + - 第一个 cron 触发器的表达式与时区；作业没有 cron 触发器时是 ``""`` 与 + ``None``。 + * - ``triggers`` + - 以映射表示的每一个触发器。 + * - ``target``、``pipeline``、``actions`` + - ``"actions"`` 或 ``"pipeline"``、流水线的名称，以及列表中的动作数量。 + * - ``allow_overlap``、``timeout`` + - 与传入的值相同。 + * - ``runs``、``skipped`` + - 启动了几次触发，以及跳过了几次。 + * - ``running`` + - 是否还有运行的线程活着。见 `超时`_。 + * - ``last_run`` + - 最后一次运行被触发的时间，以作业自己的时间表示：作业有时区时带偏移量， + 没有时则是本地时间。 + * - ``last_state`` + - 最后一次运行结束时的状态。 + +.. list-table:: + :header-rows: 1 + :widths: 34 66 + + * - 调用 + - 作用 + * - ``remove(name)``、``remove_all()`` + - 移除作业并停止它们的触发器。进行中的运行会继续。 + * - ``list()`` + - 每个作业的快照。 + * - ``run_now(name)`` + - 手动触发一个作业；返回 ``JobRun``。 + * - ``cancel(name)`` + - 取消该作业进行中的运行；返回它们的记录。 + * - ``history(job=None, state=None, limit=50)`` + - 最近的记录，最新的在前。 + * - ``start()``、``shutdown(timeout=5.0, *, cancel_running=False)`` + - 见 `启动与停止`_。 + * - ``tick(now=None)`` + - 为手动驱动的调度器处理一次当前这一分钟与超时。见 `启动与停止`_。 + +运行记录与状态 +---------------------------- + +``scheduler.history()`` 返回 :class:`~automation_file.scheduler.JobRun` 对象； +``run.to_dict()`` 可以序列化成 JSON。 + +.. list-table:: + :header-rows: 1 + :widths: 22 78 + + * - 字段 + - 含义 + * - ``run_id`` + - 这次触发的 ID。 + * - ``job`` + - 作业的名称。 + * - ``trigger`` + - ``cron``、``manual``、``file``、``event`` 或 ``pipeline``。 + * - ``detail`` + - 是什么触发了这次运行：表达式与时区、文件与事件、事件的 type、ID、来源与 + 主题，或是流水线、运行 ID 与状态。 + * - ``target``、``pipeline`` + - ``"actions"`` 或 ``"pipeline"``，以及流水线的名称。 + * - ``state`` + - 一个 ``RunState``。它与自己的文本比较时相等。 + * - ``scheduled_at`` + - 这次运行应该开始的时间：对 cron 而言就是那一分钟。 + * - ``started_at``、``finished_at`` + - 目标开始执行的时间，以及记录关闭的时间。 + * - ``duration_ms`` + - 两者之间的时间。 + * - ``error`` + - 出了什么问题，用于 ``failed`` 与 ``timeout``。 + * - ``reason`` + - 被跳过的触发是 ``overlap`` 或 ``chain``，被取消的运行是 + ``cancelled``。 + * - ``correlation_id`` + - 这次运行的每个事件所带的 ID。对动作列表而言它就是 ``run_id``。对流水线而言， + 流水线一开始运行它就变成流水线的运行 ID，也就是 ``FA_pipeline_status`` 接受的 + 那个 ID。 + +所有时间都是带时区的 UTC。目标在 ``correlation_scope(run_id)`` 之内、以 actor +``scheduler`` 执行，所以这次运行的存储错误与审计记录都带着同一个 ID。 + +.. list-table:: + :header-rows: 1 + :widths: 18 82 + + * - 状态 + - 含义 + * - ``scheduled`` + - 触发已被接受，它的线程还没开始。 + * - ``started`` + - 目标正在执行。 + * - ``completed`` + - 每个动作都返回了，或流水线的运行成功了。 + * - ``failed`` + - 至少一个动作抛出异常、流水线的运行没有成功，或目标根本无法执行。``error`` + 会说明是哪一种。 + * - ``skipped`` + - 什么都没有启动：作业还在运行而且不允许重叠，或是这次触发落在一条 16 次运行的 + 链的末端。 + * - ``timeout`` + - 运行没有在超时时间内结束。 + * - ``cancelled`` + - ``cancel`` 停止了这次运行。 + +历史保存在内存中，每个调度器各有一份，保留最近的 1000 条记录 +（``Scheduler(history_limit=...)``）；最旧的先被丢弃。它不会在进程结束后留存。被调度 +流水线的运行同时也在流水线的运行记录存储中，而每一次失败都是事件，:doc:`审计轨迹 ` +可以把它保存下来。 + +重叠保护 +---------------- + +除非作业注册时给了 ``allow_overlap=True``，否则在作业运行期间到达的触发不会启动任何 +东西。它会记录为 ``skipped``，原因是 ``overlap``，计入作业的 ``skipped``，并以警告 +写入日志。每一种触发器都是如此，``run_now`` 也不例外。 + +在运行的线程真正结束之前，作业都算是运行中。超时或取消之后，这个时间点可能比记录上 +写的还晚。 + +超时 +-------- + +``timeout`` 是一次运行可以花的秒数，从它被触发的那一刻起算。调度器每秒检查一次。 +时间到了的时候，记录会变成 ``timeout``，发布一个 ``scheduler.error`` 事件，并要求这次 +运行停止： + +* 流水线会通过它的取消令牌被取消，与 ``run.cancel()`` 完全相同：尚未开始的任务变成 + ``cancelled``，运行中的任务则通过 ``ctx.cancel`` 得知； +* 动作列表会在下一个动作之前停止。 + +**线程无法被强制终止。**\ 正在执行的动作，或不检查令牌的流水线任务，会一直执行到它 +返回为止。在那之前作业都算是运行中，所以下一次触发会被跳过，两次运行绝不会相撞。 +线程在超时之后做的事不会改变记录。 + +也请为流水线的任务设置它们自己的超时（见 :doc:`pipeline`）：任务的超时只让一个任务 +失败，清理任务仍会执行；作业的超时则会停止整次运行。 + +取消 +-------- + +.. code-block:: python + + cancelled = scheduler.cancel("daily-report") # 这些记录，现在是 "cancelled" + +``cancel(name)`` 会把该作业进行中的运行的记录关闭为 ``cancelled``，并以与超时相同的 +方式停止这些运行。作业没有在运行时它返回空列表；名称既没有注册也没有在运行时则抛出 +``SchedulerException``。被取消的运行不会发布 ``scheduler.error``。移除作业不会取消 +进行中的运行。 + +时区与夏令时 +------------------------ + +``timezone`` 是 IANA 名称，例如 ``Asia/Taipei`` 或 ``America/New_York``，或是 +``UTC``。时区数据来自标准库的 ``zoneinfo``，它读取系统的数据库。Windows 没有这个 +数据库：请在那里安装 ``tzdata`` 包（``pip install tzdata``）。``UTC`` 不需要它就能 +使用。找不到的名称会在作业注册时抛出 ``CronException``。 + +没有时区的 cron 触发器以机器的本地时间解读，调度器一直以来都是如此。在容器中那通常 +是 UTC：请为生产环境的作业指定时区。 + +在有夏令时的时区中： + +* **不存在的本地时间不会触发。**\ 在时钟从 02:00 直接跳到 03:00 的那一天， + ``30 2 * * *`` 不会运行； +* **出现两次的本地时间只触发一次**\ ，在第一次出现时。在时钟从 02:00 拨回 01:00 的 + 那一天，``30 1 * * *`` 只运行一次； +* 小时字段是 ``*`` 的表达式本来就每小时运行，所以在重复的那一小时中仍会持续触发： + ``*/15 * * * *`` 在这两天都是每 15 分钟（实际经过的时间）运行一次。 + +必须以固定间隔运行，或不论时钟怎么变都必须每天刚好运行一次的作业，最好排在 ``UTC``。 +没有时区的触发器跟随系统时钟，不会得到上述任何处理：重复的那一小时会触发两次。 + +事件 +-------- + +以 ``failed`` 或 ``timeout`` 结束的运行会在调度器的总线上发布一个 +``SchedulerError``\ （``scheduler.error``，严重程度 ``error``，来源 ``scheduler``）。它的 +主题是 ``scheduler[] failed`` 或 ``scheduler[] timed out``，它的关联 ID 是 +记录的 ``correlation_id``。 + +.. list-table:: + :header-rows: 1 + :widths: 24 76 + + * - payload 的键 + - 值 + * - ``job`` + - 作业的名称。 + * - ``trigger`` + - 是什么触发了这次运行。 + * - ``status`` + - ``failed`` 或 ``timeout``。 + * - ``error`` + - 记录的 ``error``。其中的 URL 会被截到只剩主机。 + * - ``target`` + - ``actions`` 或 ``pipeline``。 + * - ``scheduler_run_id`` + - 记录的 ``run_id``。 + * - ``duration_ms`` + - 目标已经开始执行时才有。 + * - ``pipeline``、``run_id`` + - 目标是流水线时：它的名称，以及流水线开始运行之后的流水线运行 ID。 + +``completed``、``skipped`` 与 ``cancelled`` 不会发布任何事件；它们记在历史与日志中。 + +有一种失败的报告方式不同。执行器完全不接受的动作列表（空的列表、不是列表的东西） +仍像以前一样通过 :func:`~automation_file.notify.manager.notify_on_failure` 报告。它 +自己在整个进程共用的总线上发布 ``scheduler.error`` 事件，payload 是 ``job``、 +``status``\ （``error``）、``error`` 与 ``context``；而在通知路由器没有启用时，它还会把 +消息直接发到每一个已注册的 sink（见 :doc:`notifications`）。其他所有失败都只是事件： +想收到通知，请为 ``scheduler.error`` 加一条路由。 + +动作 +-------- + +.. list-table:: + :header-rows: 1 + :widths: 26 44 30 + + * - 动作 + - 参数 + - 返回 + * - ``FA_schedule_add`` + - ``name, cron_expression, action_list, allow_overlap=False, timezone=None, + timeout=None`` + - 作业的快照 + * - ``FA_schedule_job`` + - ``name, action_list, triggers=None, allow_overlap=False, timeout=None`` + - 作业的快照 + * - ``FA_schedule_pipeline`` + - ``definition, name=None, triggers=None, params=None, allow_overlap=False, + timeout=None`` + - 作业的快照 + * - ``FA_schedule_run`` + - ``name`` + - 这次触发的记录 + * - ``FA_schedule_cancel`` + - ``name`` + - 被取消的记录 + * - ``FA_schedule_history`` + - ``job=None, state=None, limit=50`` + - 记录，最新的在前 + * - ``FA_schedule_list`` + - （无） + - 每个作业的快照 + * - ``FA_schedule_remove`` + - ``name`` + - 被移除作业的快照 + * - ``FA_schedule_remove_all`` + - （无） + - 被移除的各个作业的快照 + +它们操作的是整个进程共用的 ``scheduler``。``definition`` 是映射或定义文件的路径，它的 +``schedule`` 会变成 cron 触发器。在 ``FA_schedule_add`` 中，``allow_overlap``、 +``timezone`` 与 ``timeout`` 要以名称传入。触发器是一个映射，内含它的 ``kind`` 与该 +种类的参数： + +.. list-table:: + :header-rows: 1 + :widths: 14 86 + + * - ``kind`` + - 键 + * - ``cron`` + - ``cron``\ （必填）、``timezone`` + * - ``file`` + - ``path``\ （必填）、``events``、``recursive`` + * - ``event`` + - ``types``、``sources``、``min_severity``；type 以名称与前缀表示 + * - ``pipeline`` + - ``pipeline``\ （必填）、``when`` + +.. code-block:: json + + [ + ["FA_schedule_pipeline", {"definition": "pipelines/daily-report.yaml", + "params": {"date": "${date:%Y-%m-%d}"}, + "timeout": 3600}], + ["FA_schedule_pipeline", {"definition": "pipelines/publish-summary.yaml", + "triggers": [{"kind": "pipeline", + "pipeline": "daily-report"}]}], + ["FA_schedule_job", {"name": "sweep-inbox", + "action_list": [["FA_copy_all_file_to_dir", + {"source_dir": "/data/inbox", + "target_dir": "/data/processed"}]], + "triggers": [{"kind": "file", "path": "/data/inbox", + "events": ["created"]}]}], + ["FA_schedule_run", {"name": "daily-report"}], + ["FA_schedule_history", {"job": "daily-report", "limit": 5}] + ] + +``register_scheduler_ops(registry)`` 可以把这些动作加进你自己的 registry。 + +作业会记下它之后要运行的动作名称。只要动作列表或定义是请求的一部分，TCP 或 HTTP +动作服务器上的 :class:`~automation_file.ActionACL` 也会检查这些名称。它看不到以文件 +路径指定的定义内部，而 ``FA_schedule_run`` 会运行已注册作业所持有的任何内容：请只对 +可以调用全部已注册动作的客户端开放 ``FA_schedule_pipeline`` 与 ``FA_schedule_run``。 + +启动与停止 +-------------------- + +.. code-block:: python + + from automation_file.scheduler import Scheduler + + scheduler = Scheduler(history_limit=5000) # 你自己的调度器 + scheduler.shutdown() # 停止线程与每一个触发器 + scheduler.start() # ... 再把它们带回来 + +``Scheduler(*, clock=None, bus=None, history_limit=1000, autostart=True)`` + +.. list-table:: + :header-rows: 1 + :widths: 22 78 + + * - 选项 + - 含义 + * - ``clock`` + - 返回当前时间（带时区的 ``datetime``）的可调用对象。默认：系统时钟的 UTC + 时间。 + * - ``bus`` + - 调度器监听与报告所用的 ``EventBus``，也是它的流水线发布事件的地方。默认： + 整个进程共用的总线。 + * - ``history_limit`` + - 保留几条记录。默认 ``1000``。 + * - ``autostart`` + - 加入作业时启动后台线程。默认 ``True``。 + +``shutdown()`` 会停止后台线程，以及调度器建立的每一个文件监听器与每一个总线订阅。 +作业仍保持注册；``start()`` 或再加入一个作业会让它们重新就绪。进行中的运行会跑完， +除非使用 ``shutdown(cancel_running=True)``。调度器的所有线程都是 daemon 线程，所以 +它们绝不会让解释器无法退出。 + +使用 ``autostart=False`` 时，什么都不会自己发生：``tick(now)`` 会处理 ``now`` 所在的 +那一分钟以及在 ``now`` 到期的超时，并返回它所触发的记录。不必等待就能测试调度的做法 +就是这样： + +.. code-block:: python + + from datetime import datetime, timezone + + from automation_file.scheduler import Scheduler + + engine = Scheduler(autostart=False) + engine.add("nightly", "0 2 * * *", [["FA_schedule_list"]], timezone="Asia/Taipei") + engine.tick(datetime(2026, 10, 7, 17, 59, tzinfo=timezone.utc)) # [] + (run,) = engine.tick(datetime(2026, 10, 7, 18, 0, tzinfo=timezone.utc)) + run.wait(10) + run.state # "completed" + +出问题时 +---------------- + +作业没有在它的时间运行 + 先找有没有 ``skipped`` 记录：那表示上一次运行还在进行。如果完全没有记录，表示 + 那一分钟没有被处理：进程没在运行或机器在休眠（错过的分钟不会补跑）、那一天不 + 存在那个本地时间，或是表达式其实以另一个时区解读。没有 ``timezone`` 的触发器使用 + 机器的本地时间。 + +``CronException: unknown time zone`` + 名称拼错了，或机器没有时区数据：在 Windows 上请安装 ``tzdata``。 + +运行是 ``failed``，但大部分都成功了 + 有一个动作抛出异常，列表的其余部分仍然执行了。``error`` 会按位置与名称列出失败 + 的动作。 + +运行是 ``completed``，工作却出了问题 + 只有动作抛出异常或流水线的运行没有成功时，运行才算失败。以返回值报告的动作会 + 完成：请见 :doc:`pipeline` 中的同一个条目。 + +运行已是 ``timeout`` 或 ``cancelled``，作业却仍显示 ``running`` + 它的线程无法被停止，还停在原本的动作或任务中。在那返回之前，作业都是忙碌的， + 它的触发也都会被跳过。请为长时间的传输设置它们自己的超时，并让长时间的流水线 + 任务检查 ``ctx.cancel``。 + +作业触发得比预期频繁 + 保存一个文件会产生好几个文件事件；同一个作业的两个触发器都会触发它；设置了 + ``allow_overlap=True`` 的作业不会被进行中的运行挡下。 + +记录是 ``skipped``，原因是 ``chain`` + 已经有十六次运行接连互相触发：两个作业监听彼此的事件，或两条流水线互相依赖。请 + 打破这个循环；需要重复运行的作业应该使用 cron 触发器。 + +依赖的流水线从不触发 + ``PipelineTrigger`` 中的名称不是上游流水线的 ``name``、上游的运行没有以 ``when`` + 要求的方式结束，或它发布事件的总线不是调度器的那一个。 + +事件触发器没有触发 + 请对照 ``event_bus.recent()`` 检查 ``types``、``sources`` 与 ``min_severity``。 + 由作业自己的运行所发布的事件是刻意被忽略的。 + +没有收到通知 + 路由器必须已经启动，而且必须有一条匹配 ``scheduler.error`` 的路由。没有路由器 + 时，只有执行器拒绝的动作列表会被直接发到 sink。 + +历史是空的 + 它保存在内存中，随进程结束而消失；你自己建立的调度器有它自己的历史。流水线的 + 运行则在运行记录存储中。 + +``shutdown()`` 之后什么都不触发 + 作业仍然注册着，但它们的触发器已经停止。请调用 ``start()``。 + +重复或未知的作业、错误的触发器、错误的超时与错误的历史查询会抛出 +``SchedulerException``；无法理解的表达式或时区则抛出 ``CronException``。错误的定义会 +抛出 ``PipelineDefinitionException``，不存在的监听路径则抛出 ``TriggerException``， +两者都发生在作业注册时。它们全都派生自 ``FileAutomationException``。 diff --git a/docs/source/Zh-CN/zh_cn_index.rst b/docs/source/Zh-CN/zh_cn_index.rst index a058891..8be4601 100644 --- a/docs/source/Zh-CN/zh_cn_index.rst +++ b/docs/source/Zh-CN/zh_cn_index.rst @@ -181,6 +181,7 @@ cron 风格调度器（``FA_schedule_*``）按调度周期性运行动作列表 :caption: 触发器与调度 usage/events + usage/scheduler .. _zh-cn-notifications: diff --git a/docs/source/Zh-TW/architecture.rst b/docs/source/Zh-TW/architecture.rst index 3ae73a0..d35bdd2 100644 --- a/docs/source/Zh-TW/architecture.rst +++ b/docs/source/Zh-TW/architecture.rst @@ -63,7 +63,7 @@ subgraph Events["事件驅動"] Trigger["TriggerManager
watchdog 檔案監聽"] - Sched["Scheduler
5-field cron + overlap guard"] + Sched["Scheduler
cron · file · event · pipeline triggers
run records + overlap guard"] end subgraph Servers["伺服器"] @@ -318,8 +318,13 @@ ├── trigger/ │ └── manager.py # FileWatcher + TriggerManager（以 watchdog 為底層） ├── scheduler/ - │ ├── cron.py # 5 欄位 cron 表達式解析器 - │ └── manager.py # Scheduler 背景執行緒 + ScheduledJob + │ ├── cron.py # 5 欄位 cron 表達式解析器、時區 + │ ├── triggers.py # cron / 檔案 / 事件 / 管線觸發器 + │ ├── job.py # ScheduledJob + │ ├── runs.py # JobRun、RunState、RunHistory + │ ├── targets.py # 執行動作清單或管線 + │ ├── dispatch.py # 重疊、逾時、取消、scheduler.error + │ └── manager.py # Scheduler 背景執行緒 + FA_schedule_* 動作 ├── notify/ │ ├── sinks.py # Webhook / Slack / Email sink │ └── manager.py # NotificationManager（扇出 + 去重 + auto-notify hook） @@ -398,9 +403,12 @@ 轉送給共享登錄表調度的動作清單。 :data:`~automation_file.trigger.trigger_manager` 擁有 name → watcher 對應表，讓 GUI 與 JSON 動作共享同一個生命週期。 -* :mod:`automation_file.scheduler` 執行一個背景執行緒，在分鐘邊界甦醒、 - 走訪已登錄的 :class:`~automation_file.scheduler.ScheduledJob`，並在 - 短生命週期的工作執行緒上調度每個相符的任務，避免慢動作拖累後續任務。 +* :mod:`automation_file.scheduler` 執行一個背景執行緒，每秒甦醒一次，觸發 + cron 觸發器在目前這一分鐘到期的 + :class:`~automation_file.scheduler.ScheduledJob`，並讓每次執行使用自己的 + 短生命週期工作執行緒，避免慢動作拖累後續任務。工作可以執行動作清單或管線， + 也可以由檔案事件、事件匯流排上的事件、另一條管線的執行結束或手動觸發；每次 + 觸發都會留下一筆執行紀錄。詳見 :doc:`usage/scheduler`。 當動作清單拋出 :class:`~automation_file.exceptions.FileAutomationException` 時，兩個調度器都會呼叫 diff --git a/docs/source/Zh-TW/usage/events.rst b/docs/source/Zh-TW/usage/events.rst index 07295c2..6121aab 100644 --- a/docs/source/Zh-TW/usage/events.rst +++ b/docs/source/Zh-TW/usage/events.rst @@ -27,32 +27,19 @@ 也可以在 JSON 動作清單中呼叫 ``FA_watch_start`` / ``FA_watch_stop`` / ``FA_watch_stop_all`` / ``FA_watch_list``。 -Cron 排程器 ------------ +排程器 +------------ -依重複時刻執行動作清單。5 欄位 cron 解析器支援 -``*``、精確值、``a-b`` 區間、逗號分隔清單與 ``*/n`` 步長， -也支援 ``jan``..``dec`` / ``sun``..``sat`` 別名。 - -.. code-block:: python - - from automation_file import schedule_add - - schedule_add( - name="nightly-snapshot", - cron_expression="0 2 * * *", # 每天本地時間 02:00 - action_list=[["FA_zip_dir", {"dir_we_want_to_zip": "/data", - "zip_name": "/backup/data_nightly"}]], - ) - -背景執行緒在每分鐘邊界喚醒，因此不支援小於一分鐘的精度。 -JSON 形式：``FA_schedule_add`` / ``FA_schedule_remove`` / -``FA_schedule_remove_all`` / ``FA_schedule_list``。 +依帶時區的 cron 運算式、檔案事件、事件匯流排上的事件、另一條管線結束之後，或以手動 +方式執行動作清單或管線，都寫在 :doc:`scheduler` 中，其中也說明了執行紀錄、重疊保護、 +逾時與 ``FA_schedule_*`` 動作。需要在檔案事件發生時執行並為每次執行留下紀錄的工作， +請使用排程器的 ``FileTrigger``，而不是 ``FA_watch_start``。 當動作清單擲出 :class:`~automation_file.exceptions.FileAutomationException` 時， -兩個排程器都會呼叫 +監看器會呼叫 :func:`~automation_file.notify.manager.notify_on_failure`。 若未註冊任何 sink，該輔助函式即為 no-op，因此自動通知是 註冊 :class:`~automation_file.NotificationSink` 的可選副作用—— -詳見 :doc:`notifications`。 +詳見 :doc:`notifications`。排程器對它無法分派的動作清單也會這麼做，其他每一次 +失敗的執行則發布成 ``scheduler.error`` 事件。 diff --git a/docs/source/Zh-TW/usage/notifications.rst b/docs/source/Zh-TW/usage/notifications.rst index e0f6739..a6fbadf 100644 --- a/docs/source/Zh-TW/usage/notifications.rst +++ b/docs/source/Zh-TW/usage/notifications.rst @@ -204,3 +204,7 @@ sink 收到的內容 路由器運作時由路由決定：沒有任何路由符合的失敗不會被投遞。像 ``Route("failures", types=("scheduler.error", "system.error"))`` 這樣的路由可以 讓這些告警持續送達。 + +排程器會為它無法分派的動作清單呼叫 ``notify_on_failure``。以其他任何方式失敗、或 +超過逾時的排程執行，則由排程器自己發布成 ``scheduler.error`` 事件，只有透過路由才會 +送到 sink：見 :doc:`scheduler`。 diff --git a/docs/source/Zh-TW/usage/pipeline.rst b/docs/source/Zh-TW/usage/pipeline.rst index 068d112..96a8366 100644 --- a/docs/source/Zh-TW/usage/pipeline.rst +++ b/docs/source/Zh-TW/usage/pipeline.rst @@ -196,7 +196,8 @@ - 預設參數；``run(params=...)`` 會加入並覆寫它們。 * - ``schedule`` - ``Schedule(cron, timezone=None)``。保存在 ``pipeline.schedule`` 供排程器 - 使用，管線本身不會據此行動。 + 使用，管線本身不會據此行動：``scheduler.add_pipeline(pipeline)`` 會把它 + 變成 cron 觸發器（見 :doc:`scheduler`）。 * - ``registry`` - 查找動作名稱的地方。預設：共用執行器的註冊表。 diff --git a/docs/source/Zh-TW/usage/scheduler.rst b/docs/source/Zh-TW/usage/scheduler.rst new file mode 100644 index 0000000..a755cb5 --- /dev/null +++ b/docs/source/Zh-TW/usage/scheduler.rst @@ -0,0 +1,703 @@ +排程器（Scheduler） +======================== + +``automation_file.scheduler`` 會在某件事觸發時執行一份動作清單或一條 +:doc:`管線 `\ ：可以帶時區的 cron 運算式、檔案事件、事件匯流排上的事件、 +另一條管線的執行結束，或是一次呼叫。cron 只是多種觸發器之一，所有工作都經過同一個 +排程器。 + +每一次觸發都會留下一筆紀錄，狀態是七種之一：``scheduled``、``started``、 +``completed``、``failed``、``skipped``、``timeout`` 與 ``cancelled``。除非工作自己 +允許，否則它不會與自己重疊；一次執行可以設定逾時，也可以被取消；失敗或逾時的執行會 +發布成 ``scheduler.error`` 事件（見 :doc:`event_bus`）。 + +最小範例 +---------------- + +.. code-block:: python + + from automation_file.scheduler import scheduler + + scheduler.add( + "nightly-snapshot", + "0 2 * * *", # 每天 02:00 ... + [["FA_zip_dir", {"dir_we_want_to_zip": "/data", + "zip_name": "/backup/data_nightly"}]], + timezone="Asia/Taipei", # ... 台北時間；不給就是本地時間 + ) + + scheduler.history(job="nightly-snapshot", limit=5) # 最近的執行，最新的在前 + +``scheduler`` 是整個行程共用的實例。加入第一個工作時會啟動它的背景執行緒；那是 +daemon 執行緒，行程要靠你自己保持存活。 + +正式環境範例 +------------------------ + +每晚台北時間 02:00 執行一條管線，第一條成功之後執行第二條管線，並在某次執行失敗或 +跑太久時通知團隊。 + +.. code-block:: python + + import os + + from automation_file import Route, SlackSink, notification_manager, notification_router + from automation_file.pipeline import SQLiteRunStore, set_default_run_store + from automation_file.scheduler import PipelineTrigger, scheduler + + # 排程的管線把執行紀錄寫進預設的執行紀錄儲存。 + set_default_run_store(SQLiteRunStore("/var/lib/automation/pipelines.db")) + + # 失敗或逾時的執行是一個 scheduler.error 事件；這條路由負責送出它。 + notification_manager.register(SlackSink(os.environ["SLACK_WEBHOOK"], name="team-alerts")) + notification_router.add_route( + Route("scheduler-failures", sinks=("team-alerts",), types=("scheduler.error",)) + ) + notification_router.start() + + # daily-report.yaml 宣告了 schedule: {cron: "0 2 * * *", timezone: Asia/Taipei} + scheduler.add_pipeline( + "pipelines/daily-report.yaml", + params={"date": "${date:%Y-%m-%d}"}, # 工作觸發當下的台北日期 + timeout=3600, # 一小時後取消 + ) + + # 沒有自己的排程：每當 daily-report 的一次執行成功，它就會執行。 + scheduler.add_pipeline( + "pipelines/publish-summary.yaml", + triggers=PipelineTrigger("daily-report"), + timeout=900, + ) + + for run in scheduler.history(state="failed", limit=10): + print(run.job, run.scheduled_at, run.error, run.correlation_id) + +``daily-report`` 在 UTC 18:00 啟動，也就是台北的 02:00，``date`` 則是台北的日期。它的 +執行以 ``succeeded`` 結束時，``publish-summary`` 會被觸發；它失敗時，``publish-summary`` +不會被觸發，路由則送出 ``[ERROR] scheduler.error: scheduler[daily-report] failed``。 +``daily-report`` 的一次執行還在進行時，下一次觸發會記錄為 ``skipped``。 + +失敗的管線自己也會發布 ``pipeline.failed`` 與 ``task.failed``。請把這兩者或 +``scheduler.error`` 其中一邊路由到 sink；同時涵蓋兩邊的路由會為一次失敗送出兩則訊息。 + +工作與目標 +-------------------- + +一個工作有名稱、目標，以及任意數量的觸發器。 + +**動作清單**\ 透過共用的執行器執行，一個動作接著一個動作。拋出例外的動作會讓這次 +執行失敗，其餘的動作仍會執行，與 ``execute_action`` 相同。動作的回傳值不會保留。 + +**管線**\ 可以是 :class:`~automation_file.pipeline.Pipeline`、定義的對應表，或是 +``.yaml`` / ``.yml`` / ``.json`` 定義檔的路徑。定義在工作註冊時檢查，檔案也只在那一刻 +讀取一次：修改檔案之後，請移除工作再重新註冊。每一次觸發都以預設的執行紀錄儲存呼叫 +``pipeline.run()``，所以 ``FA_pipeline_status`` 與 ``FA_pipeline_history`` 看得到這次 +執行。 + +``add_pipeline`` 會讀取管線的 ``schedule``，把它變成 cron 觸發器，時區也一併帶入； +除非給了 ``name``，工作會以管線的名稱命名。``add_job`` 也接受管線，但不會讀取它的 +``schedule``。 + +``params`` 是管線工作每一次執行的參數；它們會加到管線自己的預設值上並覆寫同名的 +項目。在任何深度的字串中，``${date:FORMAT}`` 會在工作觸發時被取代（``FORMAT`` 是 +``strftime`` 格式；單獨的 ``${date}`` 會得到 ``2026-10-08T02:00:00``）。使用的是工作 +自己的時間：第一個 cron 觸發器的時區，沒有 cron 觸發器時則是本地時間。除此之外不會 +取代任何東西，動作清單也不接受 ``params``。 + +觸發器 +------------ + +.. list-table:: + :header-rows: 1 + :widths: 14 36 50 + + * - 種類 + - Python 寫法 + - 何時觸發 + * - ``cron`` + - ``CronTrigger(cron, timezone=None)`` + - 在運算式指定的那些分鐘。 + * - ``manual`` + - ``scheduler.run_now(name)`` + - 被呼叫時。每個工作都能這樣觸發。 + * - ``file`` + - ``FileTrigger(path, events=("created", "modified"), recursive=True)`` + - ``path`` 底下的檔案被建立、修改、刪除或搬移時。 + * - ``event`` + - ``EventTrigger(types=(), sources=(), min_severity=Severity.INFO)`` + - 事件匯流排上發布了符合條件的事件時。 + * - ``pipeline`` + - ``PipelineTrigger(pipeline, when="on_success")`` + - 指定名稱的管線有一次執行結束時。 + +.. code-block:: python + + from automation_file.scheduler import ( + CronTrigger, EventTrigger, FileTrigger, PipelineTrigger, scheduler, + ) + + scheduler.add_job( + "sweep-inbox", + [["FA_copy_all_file_to_dir", {"source_dir": "/data/inbox", + "target_dir": "/data/processed"}]], + triggers=[ + CronTrigger("*/30 * * * *", "UTC"), # 每半小時 + FileTrigger("/data/inbox", events=["created"]), # 以及有新檔案時 + ], + ) + +一個工作可以有好幾個任何種類的觸發器，重疊保護涵蓋全部。沒有觸發器的工作只在手動 +觸發時執行。 + +Cron +~~~~ + +五個欄位：分（0-59）、時（0-23）、日（1-31）、月（1-12）與星期（0-6，星期日是 0 或 +7）。每個欄位都接受 ``*``、單一值、區間 ``a-b``、清單 ``a,b,c``，以及步長 ``*/n`` 或 +``a-b/n``；月份與星期還接受 ``jan``..``dec`` 與 ``sun``..``sat``。沒有秒，也沒有 +``@daily`` 這類別名。 + +排程器每秒看一次時鐘，每一分鐘只處理一次。行程沒在執行或機器在睡眠的那些分鐘，事後 +不會補跑。時區請見 `時區與日光節約時間`_。 + +手動 +~~~~~~~~ + +.. code-block:: python + + run = scheduler.run_now("nightly-snapshot") # 一個 JobRun + run.wait(600) # 執行結束後為 True + run.state # RunState.COMPLETED；等於 "completed" + +``run_now`` 會立刻回傳這次觸發的紀錄。工作還在執行且不允許重疊時，紀錄是 +``skipped``。在 JSON 中，也就是透過 HTTP 動作伺服器時，它是 ``FA_schedule_run``。 + +檔案事件 +~~~~~~~~~~~~~~~~ + +``FileTrigger`` 使用 :class:`~automation_file.trigger.FileWatcher`，也就是 +``FA_watch_start`` 背後的監看器（見 :doc:`events`）。監看器歸排程器所有：它不會出現在 +``FA_watch_list`` 中，工作被移除時它就停止。執行紀錄的 ``detail`` 記著檔案與事件 +（``{"path": "/data/inbox/a.csv", "event": "created"}``）。 + +儲存一個檔案常常會產生好幾個事件。每一個都是一次觸發；工作還在執行時到達的那些會 +記錄為 ``skipped``。 + +事件與 webhook +~~~~~~~~~~~~~~~~~~~~~~~~ + +``EventTrigger`` 比對的方式與匯流排相同：``types`` 中放 type 名稱（``"task.failed"``）、 +前綴（``"pipeline.*"``）與事件類別，``sources`` 中放完全相符的 ``event.source`` 值， +另外還有最低嚴重程度。至少要給一個 type 或一個來源。 + +webhook 也是這樣觸發工作的。收到請求的程式碼發布一個事件，每個觸發器符合的工作都會 +執行： + +.. code-block:: python + + from dataclasses import dataclass + from typing import ClassVar + + from automation_file import Event, emit + from automation_file.scheduler import EventTrigger, scheduler + + @dataclass(frozen=True, kw_only=True) + class DeployFinished(Event): + type: ClassVar[str] = "deploy.finished" + + scheduler.add_job( + "smoke-test", + [["FA_execute_files", [["checks/smoke.json"]]]], + triggers=EventTrigger(types="deploy.finished", sources="webhook"), + ) + + # 在你的網頁框架的處理函式中，請求通過驗證之後： + emit(DeployFinished(source="webhook", subject="release 1.4 deployed")) + +對 HTTP 動作伺服器（見 :doc:`servers`）的請求也可以改用名稱觸發單一工作： +``[["FA_schedule_run", {"name": "smoke-test"}]]``。這樣的執行會記錄為 ``manual``。 + +觸發器的處理函式在發布事件的執行緒中執行，而且只負責啟動這次執行。由工作自己的執行 +所發布的事件不會再次觸發該工作，所以監聽 ``scheduler.error`` 的工作不會因為自己的 +失敗而不斷循環。 + +透過事件互相觸發的工作會形成一條鏈。會讓一條鏈超過 16 次執行的那次觸發不會啟動任何 +東西：它會記錄為 ``skipped``，原因是 ``chain``，所以互相觸發的兩個工作會停下來，而不是 +永遠持續下去。 + +管線相依 +~~~~~~~~~~~~~~~~ + +``PipelineTrigger("daily-report")`` 會在名為 ``daily-report`` 的管線有一次執行結束時 +觸發。``when`` 沿用管線自己的用詞： + +.. list-table:: + :header-rows: 1 + :widths: 22 78 + + * - ``when`` + - 在這次執行……時觸發 + * - ``"on_success"`` + - 成功。這是預設值。 + * - ``"on_failure"`` + - 失敗或被取消。 + * - ``"always"`` + - 結束，不論結果如何。 + +這個觸發器監聽 ``pipeline.completed`` 與 ``pipeline.failed``，所以它看得到排程器的 +匯流排上該管線的每一次執行：排程器啟動的、用 ``pipeline.run()`` 啟動的，以及由 +``FA_pipeline_run`` 啟動的。紀錄的 ``detail`` 記著觸發它的那次執行的管線、執行 ID 與 +狀態。管線工作不能相依於自己的管線，而互相相依的兩條管線會被上述的鏈長度上限擋下。 + +選項 +-------- + +``scheduler.add(name, cron_expression, action_list, *, allow_overlap=False, timezone=None, timeout=None)`` + +``scheduler.add_job(name, target, *, triggers=None, allow_overlap=False, timeout=None, params=None)`` + +``scheduler.add_pipeline(pipeline, *, name=None, triggers=None, allow_overlap=False, timeout=None, params=None)`` + +.. list-table:: + :header-rows: 1 + :widths: 22 78 + + * - 選項 + - 意義 + * - ``name`` + - 識別這個工作。第二個同名的工作會被拒絕。``add_pipeline`` 的預設值是管線的 + 名稱。 + * - ``cron_expression`` + - ``add`` 的五個欄位。 + * - ``action_list`` / ``target`` / ``pipeline`` + - 工作要執行的東西。見 `工作與目標`_。 + * - ``triggers`` + - 一個或多個觸發器：物件，或它們的對應表（見 `動作`_）。 + * - ``allow_overlap`` + - 允許工作還在執行時就啟動新的觸發。預設 ``False``。 + * - ``timezone`` + - ``add`` 解讀運算式所用的時區。預設：本地時間。 + * - ``timeout`` + - 一次執行可以花的秒數，必須大於 0。預設：不限制。見 `逾時`_。 + * - ``params`` + - 管線工作每次執行的參數。 + +這三個方法都回傳工作的快照，也就是 ``scheduler.list()`` 為每個工作保存的對應表： + +.. list-table:: + :header-rows: 1 + :widths: 22 78 + + * - 鍵 + - 值 + * - ``name`` + - 工作的名稱。 + * - ``cron``、``timezone`` + - 第一個 cron 觸發器的運算式與時區；工作沒有 cron 觸發器時是 ``""`` 與 + ``None``。 + * - ``triggers`` + - 以對應表表示的每一個觸發器。 + * - ``target``、``pipeline``、``actions`` + - ``"actions"`` 或 ``"pipeline"``、管線的名稱，以及清單中的動作數量。 + * - ``allow_overlap``、``timeout`` + - 與傳入的值相同。 + * - ``runs``、``skipped`` + - 啟動了幾次觸發，以及略過了幾次。 + * - ``running`` + - 是否還有執行的執行緒活著。見 `逾時`_。 + * - ``last_run`` + - 最後一次執行被觸發的時間，以工作自己的時間表示：工作有時區時帶偏移量， + 沒有時則是本地時間。 + * - ``last_state`` + - 最後一次執行結束時的狀態。 + +.. list-table:: + :header-rows: 1 + :widths: 34 66 + + * - 呼叫 + - 作用 + * - ``remove(name)``、``remove_all()`` + - 移除工作並停止它們的觸發器。進行中的執行會繼續。 + * - ``list()`` + - 每個工作的快照。 + * - ``run_now(name)`` + - 手動觸發一個工作；回傳 ``JobRun``。 + * - ``cancel(name)`` + - 取消該工作進行中的執行；回傳它們的紀錄。 + * - ``history(job=None, state=None, limit=50)`` + - 最近的紀錄，最新的在前。 + * - ``start()``、``shutdown(timeout=5.0, *, cancel_running=False)`` + - 見 `啟動與停止`_。 + * - ``tick(now=None)`` + - 為手動驅動的排程器處理一次目前這一分鐘與逾時。見 `啟動與停止`_。 + +執行紀錄與狀態 +---------------------------- + +``scheduler.history()`` 回傳 :class:`~automation_file.scheduler.JobRun` 物件； +``run.to_dict()`` 可以序列化成 JSON。 + +.. list-table:: + :header-rows: 1 + :widths: 22 78 + + * - 欄位 + - 意義 + * - ``run_id`` + - 這次觸發的 ID。 + * - ``job`` + - 工作的名稱。 + * - ``trigger`` + - ``cron``、``manual``、``file``、``event`` 或 ``pipeline``。 + * - ``detail`` + - 是什麼觸發了這次執行：運算式與時區、檔案與事件、事件的 type、ID、來源與 + 主旨，或是管線、執行 ID 與狀態。 + * - ``target``、``pipeline`` + - ``"actions"`` 或 ``"pipeline"``，以及管線的名稱。 + * - ``state`` + - 一個 ``RunState``。它與自己的文字比較時相等。 + * - ``scheduled_at`` + - 這次執行應該開始的時間：對 cron 而言就是那一分鐘。 + * - ``started_at``、``finished_at`` + - 目標開始執行的時間，以及紀錄結案的時間。 + * - ``duration_ms`` + - 兩者之間的時間。 + * - ``error`` + - 出了什麼問題，用於 ``failed`` 與 ``timeout``。 + * - ``reason`` + - 被略過的觸發是 ``overlap`` 或 ``chain``，被取消的執行是 + ``cancelled``。 + * - ``correlation_id`` + - 這次執行的每個事件所帶的 ID。對動作清單而言它就是 ``run_id``。對管線而言， + 管線一開始執行它就變成管線的執行 ID，也就是 ``FA_pipeline_status`` 接受的 + 那個 ID。 + +所有時間都是帶時區的 UTC。目標在 ``correlation_scope(run_id)`` 之內、以 actor +``scheduler`` 執行，所以這次執行的儲存錯誤與稽核紀錄都帶著同一個 ID。 + +.. list-table:: + :header-rows: 1 + :widths: 18 82 + + * - 狀態 + - 意義 + * - ``scheduled`` + - 觸發已被接受，它的執行緒還沒開始。 + * - ``started`` + - 目標正在執行。 + * - ``completed`` + - 每個動作都回傳了，或管線的執行成功了。 + * - ``failed`` + - 至少一個動作拋出例外、管線的執行沒有成功，或目標根本無法執行。``error`` + 會說明是哪一種。 + * - ``skipped`` + - 什麼都沒有啟動：工作還在執行而且不允許重疊，或是這次觸發落在一條 16 次執行的 + 鏈的尾端。 + * - ``timeout`` + - 執行沒有在逾時時間內結束。 + * - ``cancelled`` + - ``cancel`` 停止了這次執行。 + +歷史保存在記憶體中，每個排程器各有一份，保留最近的 1000 筆紀錄 +（``Scheduler(history_limit=...)``）；最舊的先被丟掉。它不會在行程結束後留存。排程 +管線的執行同時也在管線的執行紀錄儲存中，而每一次失敗都是事件，:doc:`稽核軌跡 ` +可以把它保存下來。 + +重疊保護 +---------------- + +除非工作註冊時給了 ``allow_overlap=True``，否則在工作執行期間到達的觸發不會啟動任何 +東西。它會記錄為 ``skipped``，原因是 ``overlap``，計入工作的 ``skipped``，並以警告 +寫入日誌。每一種觸發器都是如此，``run_now`` 也不例外。 + +在執行的執行緒真正結束之前，工作都算是執行中。逾時或取消之後，這個時間點可能比紀錄 +上寫的還晚。 + +逾時 +-------- + +``timeout`` 是一次執行可以花的秒數，從它被觸發的那一刻起算。排程器每秒檢查一次。 +時間到了的時候，紀錄會變成 ``timeout``，發布一個 ``scheduler.error`` 事件，並要求這次 +執行停止： + +* 管線會透過它的取消權杖被取消，與 ``run.cancel()`` 完全相同：尚未開始的任務變成 + ``cancelled``，執行中的任務則透過 ``ctx.cancel`` 得知； +* 動作清單會在下一個動作之前停止。 + +**執行緒無法被強制終止。**\ 正在執行的動作，或不檢查權杖的管線任務，會一直執行到它 +回傳為止。在那之前工作都算是執行中，所以下一次觸發會被略過，兩次執行絕不會相撞。 +執行緒在逾時之後做的事不會改變紀錄。 + +也請為管線的任務設定它們自己的逾時（見 :doc:`pipeline`）：任務的逾時只讓一個任務 +失敗，清理任務仍會執行；工作的逾時則會停止整次執行。 + +取消 +-------- + +.. code-block:: python + + cancelled = scheduler.cancel("daily-report") # 這些紀錄，現在是 "cancelled" + +``cancel(name)`` 會把該工作進行中的執行的紀錄結案為 ``cancelled``，並以與逾時相同的 +方式停止這些執行。工作沒有在執行時它回傳空清單；名稱既沒有註冊也沒有在執行時則拋出 +``SchedulerException``。被取消的執行不會發布 ``scheduler.error``。移除工作不會取消 +進行中的執行。 + +時區與日光節約時間 +------------------------------------ + +``timezone`` 是 IANA 名稱，例如 ``Asia/Taipei`` 或 ``America/New_York``，或是 +``UTC``。時區資料來自標準函式庫的 ``zoneinfo``，它讀取系統的資料庫。Windows 沒有這個 +資料庫：請在那裡安裝 ``tzdata`` 套件（``pip install tzdata``）。``UTC`` 不需要它就能 +使用。找不到的名稱會在工作註冊時拋出 ``CronException``。 + +沒有時區的 cron 觸發器以機器的本地時間解讀，排程器一直以來都是如此。在容器中那通常 +是 UTC：請為正式環境的工作指定時區。 + +在有日光節約時間的時區中： + +* **不存在的本地時間不會觸發。**\ 在時鐘從 02:00 直接跳到 03:00 的那一天， + ``30 2 * * *`` 不會執行； +* **出現兩次的本地時間只觸發一次**\ ，在第一次出現時。在時鐘從 02:00 撥回 01:00 的 + 那一天，``30 1 * * *`` 只執行一次； +* 小時欄位是 ``*`` 的運算式本來就每小時執行，所以在重複的那一小時中仍會持續觸發： + ``*/15 * * * *`` 在這兩天都是每 15 分鐘（實際經過的時間）執行一次。 + +必須以固定間隔執行，或不論時鐘怎麼變都必須每天剛好執行一次的工作，最好排在 ``UTC``。 +沒有時區的觸發器跟隨系統時鐘，不會得到上述任何處理：重複的那一小時會觸發兩次。 + +事件 +-------- + +以 ``failed`` 或 ``timeout`` 結束的執行會在排程器的匯流排上發布一個 +``SchedulerError``\ （``scheduler.error``，嚴重程度 ``error``，來源 ``scheduler``）。它的 +主旨是 ``scheduler[] failed`` 或 ``scheduler[] timed out``，它的關聯 ID 是 +紀錄的 ``correlation_id``。 + +.. list-table:: + :header-rows: 1 + :widths: 24 76 + + * - payload 的鍵 + - 值 + * - ``job`` + - 工作的名稱。 + * - ``trigger`` + - 是什麼觸發了這次執行。 + * - ``status`` + - ``failed`` 或 ``timeout``。 + * - ``error`` + - 紀錄的 ``error``。其中的 URL 會被截到只剩主機。 + * - ``target`` + - ``actions`` 或 ``pipeline``。 + * - ``scheduler_run_id`` + - 紀錄的 ``run_id``。 + * - ``duration_ms`` + - 目標已經開始執行時才有。 + * - ``pipeline``、``run_id`` + - 目標是管線時：它的名稱，以及管線開始執行之後的管線執行 ID。 + +``completed``、``skipped`` 與 ``cancelled`` 不會發布任何事件；它們記在歷史與日誌中。 + +有一種失敗的回報方式不同。執行器完全不接受的動作清單（空的清單、不是清單的東西） +仍像以前一樣透過 :func:`~automation_file.notify.manager.notify_on_failure` 回報。它 +自己在整個行程共用的匯流排上發布 ``scheduler.error`` 事件，payload 是 ``job``、 +``status``\ （``error``）、``error`` 與 ``context``；而在通知路由器沒有啟用時，它還會把 +訊息直接送到每一個已註冊的 sink（見 :doc:`notifications`）。其他所有失敗都只是事件： +想收到通知，請為 ``scheduler.error`` 加一條路由。 + +動作 +-------- + +.. list-table:: + :header-rows: 1 + :widths: 26 44 30 + + * - 動作 + - 參數 + - 回傳 + * - ``FA_schedule_add`` + - ``name, cron_expression, action_list, allow_overlap=False, timezone=None, + timeout=None`` + - 工作的快照 + * - ``FA_schedule_job`` + - ``name, action_list, triggers=None, allow_overlap=False, timeout=None`` + - 工作的快照 + * - ``FA_schedule_pipeline`` + - ``definition, name=None, triggers=None, params=None, allow_overlap=False, + timeout=None`` + - 工作的快照 + * - ``FA_schedule_run`` + - ``name`` + - 這次觸發的紀錄 + * - ``FA_schedule_cancel`` + - ``name`` + - 被取消的紀錄 + * - ``FA_schedule_history`` + - ``job=None, state=None, limit=50`` + - 紀錄，最新的在前 + * - ``FA_schedule_list`` + - （無） + - 每個工作的快照 + * - ``FA_schedule_remove`` + - ``name`` + - 被移除工作的快照 + * - ``FA_schedule_remove_all`` + - （無） + - 被移除的各個工作的快照 + +它們操作的是整個行程共用的 ``scheduler``。``definition`` 是對應表或定義檔的路徑，它的 +``schedule`` 會變成 cron 觸發器。在 ``FA_schedule_add`` 中，``allow_overlap``、 +``timezone`` 與 ``timeout`` 要以名稱傳入。觸發器是一個對應表，內含它的 ``kind`` 與該 +種類的引數： + +.. list-table:: + :header-rows: 1 + :widths: 14 86 + + * - ``kind`` + - 鍵 + * - ``cron`` + - ``cron``\ （必填）、``timezone`` + * - ``file`` + - ``path``\ （必填）、``events``、``recursive`` + * - ``event`` + - ``types``、``sources``、``min_severity``；type 以名稱與前綴表示 + * - ``pipeline`` + - ``pipeline``\ （必填）、``when`` + +.. code-block:: json + + [ + ["FA_schedule_pipeline", {"definition": "pipelines/daily-report.yaml", + "params": {"date": "${date:%Y-%m-%d}"}, + "timeout": 3600}], + ["FA_schedule_pipeline", {"definition": "pipelines/publish-summary.yaml", + "triggers": [{"kind": "pipeline", + "pipeline": "daily-report"}]}], + ["FA_schedule_job", {"name": "sweep-inbox", + "action_list": [["FA_copy_all_file_to_dir", + {"source_dir": "/data/inbox", + "target_dir": "/data/processed"}]], + "triggers": [{"kind": "file", "path": "/data/inbox", + "events": ["created"]}]}], + ["FA_schedule_run", {"name": "daily-report"}], + ["FA_schedule_history", {"job": "daily-report", "limit": 5}] + ] + +``register_scheduler_ops(registry)`` 可以把這些動作加進你自己的 registry。 + +工作會記下它之後要執行的動作名稱。只要動作清單或定義是請求的一部分，TCP 或 HTTP +動作伺服器上的 :class:`~automation_file.ActionACL` 也會檢查這些名稱。它看不到以檔案 +路徑指定的定義內部，而 ``FA_schedule_run`` 會執行已註冊工作所持有的任何內容：請只對 +可以呼叫全部已註冊動作的用戶端開放 ``FA_schedule_pipeline`` 與 ``FA_schedule_run``。 + +啟動與停止 +-------------------- + +.. code-block:: python + + from automation_file.scheduler import Scheduler + + scheduler = Scheduler(history_limit=5000) # 你自己的排程器 + scheduler.shutdown() # 停止執行緒與每一個觸發器 + scheduler.start() # ... 再把它們帶回來 + +``Scheduler(*, clock=None, bus=None, history_limit=1000, autostart=True)`` + +.. list-table:: + :header-rows: 1 + :widths: 22 78 + + * - 選項 + - 意義 + * - ``clock`` + - 回傳目前時間（帶時區的 ``datetime``）的可呼叫物件。預設：系統時鐘的 UTC + 時間。 + * - ``bus`` + - 排程器監聽與回報所用的 ``EventBus``，也是它的管線發布事件的地方。預設： + 整個行程共用的匯流排。 + * - ``history_limit`` + - 保留幾筆紀錄。預設 ``1000``。 + * - ``autostart`` + - 加入工作時啟動背景執行緒。預設 ``True``。 + +``shutdown()`` 會停止背景執行緒，以及排程器建立的每一個檔案監看器與每一個匯流排 +訂閱。工作仍保持註冊；``start()`` 或再加入一個工作會讓它們重新就緒。進行中的執行會 +跑完，除非使用 ``shutdown(cancel_running=True)``。排程器的所有執行緒都是 daemon +執行緒，所以它們絕不會讓直譯器無法結束。 + +使用 ``autostart=False`` 時，什麼都不會自己發生：``tick(now)`` 會處理 ``now`` 所在的 +那一分鐘以及在 ``now`` 到期的逾時，並回傳它所觸發的紀錄。不必等待就能測試排程的做法 +就是這樣： + +.. code-block:: python + + from datetime import datetime, timezone + + from automation_file.scheduler import Scheduler + + engine = Scheduler(autostart=False) + engine.add("nightly", "0 2 * * *", [["FA_schedule_list"]], timezone="Asia/Taipei") + engine.tick(datetime(2026, 10, 7, 17, 59, tzinfo=timezone.utc)) # [] + (run,) = engine.tick(datetime(2026, 10, 7, 18, 0, tzinfo=timezone.utc)) + run.wait(10) + run.state # "completed" + +出問題時 +---------------- + +工作沒有在它的時間執行 + 先找有沒有 ``skipped`` 紀錄：那表示上一次執行還在進行。如果完全沒有紀錄，表示 + 那一分鐘沒有被處理：行程沒在執行或機器在睡眠（錯過的分鐘不會補跑）、那一天不 + 存在那個本地時間，或是運算式其實以另一個時區解讀。沒有 ``timezone`` 的觸發器使用 + 機器的本地時間。 + +``CronException: unknown time zone`` + 名稱拼錯了，或機器沒有時區資料：在 Windows 上請安裝 ``tzdata``。 + +執行是 ``failed``，但大部分都成功了 + 有一個動作拋出例外，清單的其餘部分仍然執行了。``error`` 會依位置與名稱列出失敗 + 的動作。 + +執行是 ``completed``，工作卻出了問題 + 只有動作拋出例外或管線的執行沒有成功時，執行才算失敗。以回傳值回報的動作會 + 完成：請見 :doc:`pipeline` 中的同一個條目。 + +執行已是 ``timeout`` 或 ``cancelled``，工作卻仍顯示 ``running`` + 它的執行緒無法被停止，還停在原本的動作或任務中。在那回傳之前，工作都是忙碌的， + 它的觸發也都會被略過。請為長時間的傳輸設定它們自己的逾時，並讓長時間的管線 + 任務檢查 ``ctx.cancel``。 + +工作觸發得比預期頻繁 + 儲存一個檔案會產生好幾個檔案事件；同一個工作的兩個觸發器都會觸發它；設定了 + ``allow_overlap=True`` 的工作不會被進行中的執行擋下。 + +紀錄是 ``skipped``，原因是 ``chain`` + 已經有十六次執行接連互相觸發：兩個工作監聽彼此的事件，或兩條管線互相相依。請 + 打破這個循環；需要重複執行的工作應該使用 cron 觸發器。 + +相依的管線從不觸發 + ``PipelineTrigger`` 中的名稱不是上游管線的 ``name``、上游的執行沒有以 ``when`` + 要求的方式結束，或它發布事件的匯流排不是排程器的那一個。 + +事件觸發器沒有觸發 + 請對照 ``event_bus.recent()`` 檢查 ``types``、``sources`` 與 ``min_severity``。 + 由工作自己的執行所發布的事件是刻意被忽略的。 + +沒有收到通知 + 路由器必須已經啟動，而且必須有一條符合 ``scheduler.error`` 的路由。沒有路由器 + 時，只有執行器拒絕的動作清單會被直接送到 sink。 + +歷史是空的 + 它保存在記憶體中，隨行程結束而消失；你自己建立的排程器有它自己的歷史。管線的 + 執行則在執行紀錄儲存中。 + +``shutdown()`` 之後什麼都不觸發 + 工作仍然註冊著，但它們的觸發器已經停止。請呼叫 ``start()``。 + +重複或未知的工作、錯誤的觸發器、錯誤的逾時與錯誤的歷史查詢會拋出 +``SchedulerException``；無法理解的運算式或時區則拋出 ``CronException``。錯誤的定義會 +拋出 ``PipelineDefinitionException``，不存在的監看路徑則拋出 ``TriggerException``， +兩者都發生在工作註冊時。它們全都衍生自 ``FileAutomationException``。 diff --git a/docs/source/Zh-TW/zh_tw_index.rst b/docs/source/Zh-TW/zh_tw_index.rst index 26ca9ba..36483a2 100644 --- a/docs/source/Zh-TW/zh_tw_index.rst +++ b/docs/source/Zh-TW/zh_tw_index.rst @@ -181,6 +181,7 @@ cron 風格排程器（``FA_schedule_*``）會依排程定期執行動作清單 :caption: 觸發器與排程 usage/events + usage/scheduler .. _zh-tw-notifications: diff --git a/docs/updates/2026-10.md b/docs/updates/2026-10.md index c1d2abb..1fc1cfb 100644 --- a/docs/updates/2026-10.md +++ b/docs/updates/2026-10.md @@ -618,3 +618,26 @@ Index and query commands: [README.md](README.md). New entries go at the end. - **Not changed**: the `Development Status :: 2 - Pre-Alpha` classifier. Raising it belongs to the 1.0 release, which is the owner's decision. - **Files**: the three READMEs, the three `usage/quickstart.rst`, the three manual indexes, `stable.toml`, `dev.toml`. - **Open items**: none. + +## U-20261008-29 · 2026-10-08 · Scheduler v2 · #scheduler #roadmap #done + +- **What** (roadmap §8, the open half of M6; closes `progress.md` #23): the scheduler runs an action list or a pipeline when one of its triggers fires. + - **Triggers**: `CronTrigger(cron, timezone=None)` with an IANA time zone, a manual `run_now(name)`, `FileTrigger` (on the existing watchdog machinery), `EventTrigger` (an event on the bus by type, source and minimum severity, which is how a webhook that publishes an event starts a job) and `PipelineTrigger` (the end of a named pipeline, by default only when it succeeded). + - **Targets**: an action list as before, or a pipeline given as an object, a definition or a file. `add_pipeline` gives a pipeline that declares `schedule` its cron trigger, zone included. + - **Run records**: every firing is a `JobRun` with one of seven states (`scheduled`, `started`, `completed`, `failed`, `skipped`, `timeout`, `cancelled`), UTC timestamps, the trigger kind, the error and a correlation ID, in a bounded history that `history(job, state, limit)` queries. For a pipeline the correlation ID is the pipeline's run ID. + - **Overlap, timeout, cancellation**: overlap is refused for every trigger kind unless `allow_overlap=True` and recorded as `skipped`; a job may have a timeout; `cancel(name)` stops a pipeline through its token and an action list before its next action. A chain of runs that fire one another stops after sixteen. + - **Events**: one `scheduler.error` per failed or timed-out run. + - **Time zones**: `zoneinfo`. A zone that cannot be found is a `CronException` that says to install `tzdata`; `tzdata` is now a base dependency on Windows, where the standard library has no zone database. With a zone, a local time that does not exist is not fired and one that occurs twice fires once (an expression whose hour field is `*` keeps firing through the repeated hour). Without a zone the behaviour is the old one: local time. + - **Actions**: `FA_schedule_add` (now also `timezone`, `timeout`), `FA_schedule_remove`, `_remove_all`, `_list` with their shapes kept, and `FA_schedule_job`, `FA_schedule_pipeline`, `FA_schedule_run`, `FA_schedule_history`, `FA_schedule_cancel`. + - The loop takes an injected clock and a `tick(now)`, so the tests do not wait for minutes. +- **What changed for existing callers**: + - An action list runs one action at a time. An action that raises now marks the run `failed` (the list still continues), where the run used to count as having run. + - Assigning `job.cron` after a job was added no longer changes when it fires; its triggers decide. + - `as_dict()["cron"]` is an empty string for a job without a cron trigger, and `last_run` carries a UTC offset when the job has a zone. + - `shutdown()` leaves runs in progress unless `cancel_running=True`. +- **Tests**: `tests/test_scheduler_{runs,triggers,pipeline,actions,lifecycle}.py` with `tests/scheduler_kit.py`; `tests/test_scheduler.py` passes unchanged. +- **Result / numbers**: 5459 passed, 257 skipped, 0 failed with every extra; 3678 passed, 135 skipped with the base dependencies only. `ruff check`, `ruff format --check` and `mypy automation_file` (245 files) pass. Python 3.14.7 on Windows. +- **Not verified**: real filesystem events behind `FileTrigger` (a stand-in observer and one real start and stop); a machine without zone data (simulated); any platform but Windows; the Sphinx build of the new pages. +- **Docs**: `usage/scheduler.rst` in the three manuals (in the "Triggers and Scheduler" chapter), `docs/source/API/scheduler.rst`, pointers in the three `usage/events.rst`, `usage/notifications.rst`, `usage/pipeline.rst` and `architecture.rst`, the feature bullet and the scheduler section of the three READMEs, `architecture.md` §2 and §3, `CLAUDE.md` (package map, key types). +- **Files**: `automation_file/scheduler/` (`errors`, `runs`, `triggers`, `job`, `targets`, `dispatch`, `cron`, `manager`, `__init__`), `automation_file/trigger/manager.py`, `automation_file/__init__.py`, `stable.toml`, `dev.toml`, the tests above, the documentation above. +- **Open items**: none for the scheduler. `automation_file.scheduler` as an attribute of the package is the `Scheduler` instance, as it was before; import the subpackage's names with `from automation_file.scheduler import ...`. diff --git a/docs/updates/README.md b/docs/updates/README.md index f27ec02..040202d 100644 --- a/docs/updates/README.md +++ b/docs/updates/README.md @@ -58,6 +58,7 @@ In the same commit: delete the item from `progress.md`, add a `#done` entry here | ID | Date | Title | Tags | Batch | |---|---|---|---|---| +| U-20261008-29 | 2026-10-08 | Scheduler v2 | #scheduler #roadmap #done | [2026-10](2026-10.md) | | U-20261008-28 | 2026-10-08 | One positioning in the READMEs, the manuals and the metadata | #docs #packaging #roadmap | [2026-10](2026-10.md) | | U-20261008-27 | 2026-10-08 | Semantic MCP tools | #mcp #security #roadmap #done | [2026-10](2026-10.md) | | U-20261008-26 | 2026-10-08 | Production deployment guide | #docs #roadmap | [2026-10](2026-10.md) | @@ -123,5 +124,5 @@ In the same commit: delete the item from `progress.md`, add a `#done` entry here | File | Period | Entries | |---|---|---:| -| [2026-10.md](2026-10.md) | 2026-10 | 39 | +| [2026-10.md](2026-10.md) | 2026-10 | 40 | | [2026-09.md](2026-09.md) | 2026-09 | 21 | diff --git a/progress.md b/progress.md index ad22578..51f7bab 100644 --- a/progress.md +++ b/progress.md @@ -25,7 +25,6 @@ Items #10 to #26 are what is left of the 1.0 roadmap (`docs/FILEAUTOMATION-1.0-R ### Later milestones -- **#23** Scheduler v2 (roadmap §8, the open half of M6): one scheduler with cron (time-zone aware), manual, file-event, webhook and pipeline-dependency triggers, run states and overlap protection, reading a pipeline's `schedule`. The event model, the `NotificationRouter` and audit schema v2 are done (U-20261008-05, U-20261008-17); the scheduler in `scheduler/` still dispatches action lists on its own cron loop. - **#24** UI 2.0 (roadmap §11, M7). Not before the APIs of #13 to #23 are stable (roadmap §20). - **#37** Storage-layer gaps the MCP work found (U-20261008-27) and did not change: (a) a `LocalStorage(root)` listing reports a link's name and its target's metadata without a containment check, although reading through the link is refused; (b) `StorageBackend` has no ranged read, so reading the head of a large remote file stages all of it; (c) `LocalStorage` on Windows opens device names (`CON`, `NUL`) and alternate data streams, which the MCP tools refuse themselves; (d) a link on an SFTP or FTP server leads outside a root that is only a path prefix. - **#26** Release engineering and 1.0 (roadmap §13, M9). Done: semantic versioning with a way to release a MINOR or MAJOR (U-20261008-24), the public API policy (U-20261008-22), a package-build check and the integration workflow next to the PR checks. Open: the migration guide and the final documentation audit (after the scheduler, MCP and GUI work lands); making the integration jobs required once they are green (#19); the 1.0.0 release itself, which is the owner's call: write `1.0.0` in both TOMLs in the release pull request. diff --git a/stable.toml b/stable.toml index 3424474..24e0038 100644 --- a/stable.toml +++ b/stable.toml @@ -30,6 +30,8 @@ dependencies = [ "defusedxml>=0.7.1", "je_action_core>=0.0.2", "PyYAML>=6.0.3", + # zoneinfo has no time-zone database of its own on Windows; a scheduler cron with a zone needs one. + "tzdata>=2024.1; platform_system == 'Windows'", "opentelemetry-api>=1.44.0", "opentelemetry-sdk>=1.44.0", "tomli>=2.0.1; python_version<\"3.11\"" diff --git a/tests/scheduler_kit.py b/tests/scheduler_kit.py new file mode 100644 index 0000000..d9328bf --- /dev/null +++ b/tests/scheduler_kit.py @@ -0,0 +1,133 @@ +"""What the scheduler tests share: a clock set by hand, a gate, and a scheduler driven by ``tick``. + +No test waits for a real minute. A :class:`Rig` owns a scheduler that has no +background thread, a clock the test moves, and a private event bus whose events +it collects. +""" + +from __future__ import annotations + +import threading +import time +from collections.abc import Callable +from datetime import datetime, timedelta, timezone +from types import TracebackType +from typing import Any + +from automation_file import executor +from automation_file.events import Event, EventBus +from automation_file.scheduler import JobRun, Scheduler + +START = datetime(2026, 10, 8, 2, 0, tzinfo=timezone.utc) +WAIT = 5.0 +_POLL = 0.005 +_PREFIX = "scheduler_test_" + + +def wait_until(predicate: Callable[[], object], timeout: float = WAIT) -> bool: + """Poll ``predicate`` until it is true or ``timeout`` seconds passed; return its last word.""" + deadline = time.monotonic() + timeout + while time.monotonic() < deadline: + if predicate(): + return True + time.sleep(_POLL) + return bool(predicate()) + + +class FakeClock: + """A clock that only moves when the test says so.""" + + def __init__(self, now: datetime = START) -> None: + self.now = now + + def __call__(self) -> datetime: + return self.now + + def advance(self, seconds: float = 0.0, minutes: float = 0.0) -> datetime: + self.now += timedelta(seconds=seconds, minutes=minutes) + return self.now + + +class Gate: + """An action that blocks until the test opens it.""" + + def __init__(self) -> None: + self.entered = threading.Event() + self.opened = threading.Event() + self.calls = 0 + + def __call__(self) -> str: + self.calls += 1 + self.entered.set() + self.opened.wait(WAIT) + return "through" + + def await_entry(self) -> None: + assert self.entered.wait(WAIT), "the gated action never started" + + def open(self) -> None: + self.opened.set() + + +class Rig: + """A scheduler driven by hand, with its clock, its bus and every event published on it.""" + + def __init__(self, **options: Any) -> None: + self.clock = FakeClock() + self.bus = EventBus() + self.events: list[Event] = [] + self.bus.subscribe(self.events.append) + options.setdefault("autostart", False) + self.engine = Scheduler(clock=self.clock, bus=self.bus, **options) + self._commands: list[str] = [] + self._gates: list[Gate] = [] + + def __enter__(self) -> Rig: + return self + + def __exit__( + self, + kind: type[BaseException] | None, + error: BaseException | None, + trace: TracebackType | None, + ) -> None: + for gate in self._gates: + gate.open() + self.engine.remove_all() + self.engine.shutdown(cancel_running=True) + for name in self._commands: + executor.registry.unregister(name) + + def command(self, name: str, function: Callable[..., Any]) -> str: + """Register ``function`` on the shared executor and return the action name to use.""" + registered = f"{_PREFIX}{name}" + executor.registry.register(registered, function) + self._commands.append(registered) + return registered + + def gate(self, name: str = "gate") -> tuple[str, Gate]: + """Register a blocking action; return its action name and the gate that releases it.""" + gate = Gate() + self._gates.append(gate) + return self.command(name, gate), gate + + def tick(self, seconds: float = 0.0, minutes: float = 0.0) -> list[JobRun]: + """Move the clock forward and tick the scheduler once.""" + self.clock.advance(seconds=seconds, minutes=minutes) + return self.engine.tick() + + def errors(self) -> list[Event]: + """The ``scheduler.error`` events published so far.""" + return [event for event in self.events if event.type == "scheduler.error"] + + def job(self, name: str) -> dict[str, Any]: + """The current snapshot of the job ``name``.""" + return next(job for job in self.engine.list() if job["name"] == name) + + def idle(self, name: str) -> bool: + """Wait until the job ``name`` no longer counts as running.""" + return wait_until(lambda: not self.job(name)["running"]) + + def runs(self, job: str) -> list[JobRun]: + """The records of ``job``, newest first.""" + return self.engine.history(job=job) diff --git a/tests/test_scheduler_actions.py b/tests/test_scheduler_actions.py new file mode 100644 index 0000000..ce4d056 --- /dev/null +++ b/tests/test_scheduler_actions.py @@ -0,0 +1,403 @@ +"""FA_schedule_* actions, the package's exports and what its modules may import.""" + +from __future__ import annotations + +import ast +import importlib +import json +import sys +import threading +from collections.abc import Iterator +from pathlib import Path +from typing import Any + +import pytest + +from automation_file import ActionRegistry, execute_action, executor +from automation_file.core.action_registry import build_default_registry +from automation_file.pipeline import MemoryRunStore, set_default_run_store +from automation_file.scheduler import ( + JobRun, + SchedulerException, + register_scheduler_ops, + schedule_add, + schedule_cancel, + schedule_history, + schedule_job, + schedule_list, + schedule_pipeline, + schedule_remove, + schedule_remove_all, + schedule_run, + scheduler, +) +from automation_file.server.action_acl import ActionACL, ActionNotPermittedException +from tests.scheduler_kit import WAIT, Gate, wait_until + +# ``automation_file.scheduler`` as an attribute is the facade's Scheduler instance, not the package. +scheduler_package = importlib.import_module("automation_file.scheduler") + +NAMES = [ + "FA_schedule_add", + "FA_schedule_cancel", + "FA_schedule_history", + "FA_schedule_job", + "FA_schedule_list", + "FA_schedule_pipeline", + "FA_schedule_remove", + "FA_schedule_remove_all", + "FA_schedule_run", +] +LEGACY_KEYS = ["name", "cron", "actions", "last_run", "runs", "allow_overlap", "running", "skipped"] +NEVER = "0 0 30 2 *" # 30 February: a valid expression that is never due +MARK = "scheduler_actions_mark" +HOLD = "scheduler_actions_hold" +PACKAGE = Path(scheduler_package.__file__).resolve().parent +MODULES = sorted(PACKAGE.glob("*.py")) +FORBIDDEN = ("automation_file.core.action_executor", "automation_file.ui", "automation_file.server") +PUBLIC_NAMES = [ + "CronException", + "CronExpression", + "CronTrigger", + "EventTrigger", + "FileTrigger", + "JobRun", + "PipelineTrigger", + "RunHistory", + "RunState", + "ScheduledJob", + "Scheduler", + "SchedulerException", + "Trigger", + "TriggerKind", + "register_scheduler_ops", + "resolve_timezone", + "schedule_add", + "schedule_cancel", + "schedule_history", + "schedule_job", + "schedule_list", + "schedule_pipeline", + "schedule_remove", + "schedule_remove_all", + "schedule_run", + "scheduler", + "trigger_from_dict", +] + + +@pytest.fixture +def marks() -> Iterator[list[Any]]: + """Clean the process-wide scheduler around a test and give it two actions to schedule.""" + seen: list[Any] = [] + gate = Gate() + previous = set_default_run_store(MemoryRunStore()) + executor.registry.register(MARK, seen.append) + executor.registry.register(HOLD, gate) + schedule_remove_all() + yield seen + gate.open() + schedule_remove_all() + scheduler.shutdown(cancel_running=True) + executor.registry.unregister(MARK) + executor.registry.unregister(HOLD) + set_default_run_store(previous) + + +def latest(job: str) -> JobRun: + return scheduler.history(job=job, limit=1)[0] + + +def definition(name: str) -> dict[str, Any]: + return { + "schema_version": 1, + "name": name, + "schedule": {"cron": NEVER, "timezone": "UTC"}, + "tasks": {"mark": {"action": [MARK, ["${params.label}"]]}}, + } + + +# ---------------------------------------------------------------------- the registry + + +def test_register_scheduler_ops_adds_every_action() -> None: + registry = ActionRegistry() + register_scheduler_ops(registry) + assert sorted(registry.event_dict) == NAMES + + +def test_the_default_registry_and_the_shared_executor_have_them() -> None: + registry = build_default_registry() + for name in NAMES: + assert name in registry + assert executor.registry.resolve(name) is not None + + +def test_every_action_says_what_it_does() -> None: + registry = ActionRegistry() + register_scheduler_ops(registry) + for name, command in registry.event_dict.items(): + assert (command.__doc__ or "").strip(), name + + +# ---------------------------------------------------------------------- the actions + + +def test_the_legacy_add_keeps_its_parameters_and_its_result(marks: list[Any]) -> None: + snapshot = schedule_add("legacy", NEVER, [[MARK, ["x"]]]) + assert list(snapshot)[: len(LEGACY_KEYS)] == LEGACY_KEYS + assert [snapshot[key] for key in LEGACY_KEYS] == ["legacy", NEVER, 1, None, 0, False, False, 0] + assert schedule_add("positional", NEVER, [[MARK, ["x"]]], allow_overlap=True)["allow_overlap"] + assert [job["name"] for job in schedule_list()] == ["legacy", "positional"] + assert schedule_remove("legacy")["name"] == "legacy" + assert [job["name"] for job in schedule_remove_all()] == ["positional"] + assert schedule_list() == [] + assert marks == [] + + +def test_add_takes_a_time_zone_and_a_timeout_through_the_executor(marks: list[Any]) -> None: + results = execute_action( + [ + [ + "FA_schedule_add", + { + "name": "zoned", + "cron_expression": NEVER, + "action_list": [[MARK, ["x"]]], + "timezone": "UTC", + "timeout": 120, + }, + ], + ["FA_schedule_add", ["by-position", NEVER, [[MARK, ["y"]]]]], + ["FA_schedule_list"], + ] + ) + added, positional, listed = results.values() + assert (added["timezone"], added["timeout"]) == ("UTC", 120.0) + assert (positional["timezone"], positional["timeout"]) == (None, None) + assert [job["name"] for job in listed] == ["zoned", "by-position"] + assert json.loads(json.dumps(listed)) == listed + assert marks == [] + + +def test_a_job_is_registered_fired_and_read_back_through_actions(marks: list[Any]) -> None: + results = execute_action( + [ + [ + "FA_schedule_job", + { + "name": "by-hand", + "action_list": [[MARK, ["fired"]]], + "triggers": [{"kind": "event", "types": ["deploy.finished"]}], + "timeout": 60, + }, + ], + ["FA_schedule_run", {"name": "by-hand"}], + ] + ) + job, fired = results.values() + assert job["triggers"] == [ + {"kind": "event", "types": ["deploy.finished"], "sources": [], "min_severity": "info"} + ] + assert (fired["job"], fired["trigger"]) == ("by-hand", "manual") + assert fired["state"] in ("scheduled", "started", "completed") + assert latest("by-hand").wait(WAIT) + assert marks == ["fired"] + (history,) = execute_action( + [["FA_schedule_history", {"job": "by-hand", "state": "completed", "limit": 5}]] + ).values() + assert [(run["run_id"], run["state"]) for run in history] == [(fired["run_id"], "completed")] + assert json.loads(json.dumps(history)) == history + assert schedule_history(job="by-hand", state="failed") == [] + + +def test_schedule_job_checks_its_action_list(marks: list[Any]) -> None: + with pytest.raises(SchedulerException, match="action_list: expected a list"): + schedule_job("job", {"schema_version": 1, "name": "sneaked-in", "tasks": {}}) + assert schedule_job("no-trigger", [[MARK, ["x"]]])["triggers"] == [] + assert marks == [] + + +def test_a_pipeline_is_registered_from_a_mapping_and_from_a_file( + marks: list[Any], tmp_path: Path +) -> None: + path = tmp_path / "from-file.json" + path.write_text(json.dumps(definition("from-file")), encoding="utf-8") + results = execute_action( + [ + [ + "FA_schedule_pipeline", + {"definition": definition("from-map"), "params": {"label": "a"}}, + ], + [ + "FA_schedule_pipeline", + { + "definition": str(path), + "name": "renamed", + "params": {"label": "${date:%Y}"}, + "triggers": [{"kind": "pipeline", "pipeline": "from-map"}], + "timeout": 600, + }, + ], + ] + ) + first, second = results.values() + assert (first["name"], first["cron"], first["timezone"]) == ("from-map", NEVER, "UTC") + assert (first["target"], first["pipeline"], first["actions"]) == ("pipeline", "from-map", 0) + assert (second["name"], second["pipeline"], second["timeout"]) == ( + "renamed", + "from-file", + 600.0, + ) + assert [trigger["kind"] for trigger in second["triggers"]] == ["cron", "pipeline"] + assert schedule_run("from-map")["target"] == "pipeline" + assert latest("from-map").wait(WAIT) + assert wait_until(lambda: len(scheduler.history(job="renamed")) == 1) + assert latest("renamed").wait(WAIT) + assert latest("renamed").trigger == "pipeline" + assert len(marks) == 2 and marks[0] == "a" and len(marks[1]) == 4 + + +def test_schedule_pipeline_takes_positional_arguments(marks: list[Any]) -> None: + assert schedule_pipeline(definition("positional"), "other-name")["name"] == "other-name" + assert marks == [] + + +def test_a_running_job_is_cancelled_through_an_action(marks: list[Any]) -> None: + schedule_job("slow", [[HOLD], [MARK, ["after"]]]) + fired = schedule_run("slow") + assert wait_until(lambda: latest("slow").state == "started") + skipped = schedule_run("slow") + assert (skipped["state"], skipped["reason"]) == ("skipped", "overlap") + (cancelled,) = execute_action([["FA_schedule_cancel", {"name": "slow"}]]).values() + assert [(run["run_id"], run["state"]) for run in cancelled] == [(fired["run_id"], "cancelled")] + assert schedule_cancel("slow") == [] + assert marks == [] + + +def test_an_action_that_fails_is_recorded_by_the_executor_as_usual(marks: list[Any]) -> None: + results = execute_action( + [ + ["FA_schedule_run", {"name": "nope"}], + ["FA_schedule_cancel", {"name": "nope"}], + ["FA_schedule_history", {"state": "done"}], + ["FA_schedule_pipeline", {"definition": {"name": "no-version"}}], + ] + ) + run, cancel, history, pipeline = results.values() + assert "SchedulerException" in run and "no such job: nope" in run + assert "SchedulerException" in cancel + assert "unknown run state" in history + assert "PipelineDefinitionException" in pipeline + assert marks == [] + + +def test_the_process_wide_scheduler_restarts_after_a_shutdown(marks: list[Any]) -> None: + def loops() -> set[threading.Thread]: + return {thread for thread in threading.enumerate() if thread.name == "fa-scheduler"} + + before = loops() + schedule_add("first", NEVER, [[MARK, ["x"]]]) + (first,) = loops() - before + scheduler.shutdown() + assert first.is_alive() is False + schedule_add("second", NEVER, [[MARK, ["y"]]]) + (second,) = loops() - before + assert second is not first and second.daemon is True + assert marks == [] + + +# ---------------------------------------------------------------------- the ACL + + +def test_an_acl_sees_the_actions_a_scheduled_job_would_run() -> None: + acl = ActionACL.build(denied=["FA_storage_delete"]) + nested = [["FA_storage_delete", {"uri": "local:///data"}]] + with pytest.raises(ActionNotPermittedException): + acl.enforce([["FA_schedule_job", {"name": "job", "action_list": nested}]]) + with pytest.raises(ActionNotPermittedException): + acl.enforce( + [ + [ + "FA_schedule_pipeline", + {"definition": {"tasks": {"wipe": {"action": nested[0]}}}}, + ] + ] + ) + acl.enforce([["FA_schedule_run", {"name": "job"}], ["FA_schedule_history"]]) + + +# ---------------------------------------------------------------------- the package + + +def _module_level_imports(path: Path) -> list[str]: + tree = ast.parse(path.read_text(encoding="utf-8"), filename=str(path)) + names: list[str] = [] + for node in tree.body: + if isinstance(node, ast.Import): + names.extend(alias.name for alias in node.names) + elif isinstance(node, ast.ImportFrom): + assert node.level == 0, f"{path.name}: relative import" + names.append(node.module or "") + names.extend(f"{node.module}.{alias.name}" for alias in node.names) + return names + + +def _is_allowed(name: str) -> bool: + top_level = name.partition(".")[0] + if top_level == "automation_file": + return not any(name == banned or name.startswith(f"{banned}.") for banned in FORBIDDEN) + return top_level in sys.stdlib_module_names or name == "__future__" + + +def test_the_package_has_its_modules() -> None: + assert {path.name for path in MODULES} == { + "__init__.py", + "cron.py", + "dispatch.py", + "errors.py", + "job.py", + "manager.py", + "runs.py", + "targets.py", + "triggers.py", + } + + +@pytest.mark.parametrize("path", MODULES, ids=lambda path: path.name) +def test_module_level_imports_are_safe_while_the_registry_is_built(path: Path) -> None: + """The registry is built while the executor is still being imported: no module may need it.""" + offending = [name for name in _module_level_imports(path) if not _is_allowed(name)] + assert offending == [] + + +@pytest.mark.parametrize("path", MODULES, ids=lambda path: path.name) +def test_every_module_starts_with_the_future_import(path: Path) -> None: + assert "from __future__ import annotations" in path.read_text(encoding="utf-8") + + +def test_the_package_exports_its_public_names() -> None: + assert sorted(scheduler_package.__all__) == sorted(PUBLIC_NAMES) + for name in PUBLIC_NAMES: + assert hasattr(scheduler_package, name), name + + +def test_the_names_the_old_scheduler_exported_are_still_there() -> None: + from automation_file.scheduler import manager + + for name in ( + "CronException", + "CronExpression", + "ScheduledJob", + "Scheduler", + "register_scheduler_ops", + "schedule_add", + "schedule_list", + "schedule_remove", + "schedule_remove_all", + "scheduler", + ): + assert name in scheduler_package.__all__ + for name in ("ScheduledJob", "Scheduler", "SchedulerException", "_safe_execute", "scheduler"): + assert hasattr(manager, name), name + assert manager.ScheduledJob is scheduler_package.ScheduledJob diff --git a/tests/test_scheduler_lifecycle.py b/tests/test_scheduler_lifecycle.py new file mode 100644 index 0000000..cbaad9b --- /dev/null +++ b/tests/test_scheduler_lifecycle.py @@ -0,0 +1,203 @@ +"""The scheduler's own thread: starting, ticking, shutting down, and arming triggers again.""" + +from __future__ import annotations + +import threading +from collections.abc import Iterator +from datetime import datetime +from pathlib import Path +from typing import Any + +import pytest + +from automation_file.events import Event +from automation_file.scheduler import ( + CronExpression, + EventTrigger, + FileTrigger, + RunState, + ScheduledJob, + Scheduler, + SchedulerException, + TriggerKind, +) +from tests.scheduler_kit import WAIT, Rig, wait_until + +EVERY_MINUTE = "* * * * *" +LOOP = "fa-scheduler" + + +class _Deploy(Event): + type = "deploy.finished" + + +@pytest.fixture +def fast(monkeypatch: pytest.MonkeyPatch) -> None: + """Let the background thread tick a hundred times a second.""" + monkeypatch.setattr(Scheduler, "_TICK_SECONDS", 0.01) + + +@pytest.fixture +def rig() -> Iterator[Rig]: + with Rig() as made: + yield made + + +def loops() -> set[threading.Thread]: + return {thread for thread in threading.enumerate() if thread.name == LOOP} + + +def test_the_background_thread_fires_what_is_due(fast: None) -> None: + before = loops() + with Rig(autostart=True) as rig: + calls: list[int] = [] + rig.engine.add( + "job", EVERY_MINUTE, [[rig.command("count", lambda: calls.append(1))]], timezone="UTC" + ) + (thread,) = loops() - before + assert thread.daemon is True + assert wait_until(lambda: calls == [1]) + rig.clock.advance(seconds=59) + assert not wait_until(lambda: len(calls) > 1, timeout=0.1) + rig.clock.advance(seconds=1) + assert wait_until(lambda: calls == [1, 1]) + rig.engine.shutdown() + assert thread.is_alive() is False + rig.clock.advance(minutes=1) + assert not wait_until(lambda: len(calls) > 2, timeout=0.1) + + +def test_the_background_thread_notices_a_timeout(fast: None) -> None: + with Rig(autostart=True) as rig: + action, gate = rig.gate() + rig.engine.add_job("job", [[action]], timeout=30) + run = rig.engine.run_now("job") + gate.await_entry() + rig.clock.advance(seconds=30) + assert run.wait(WAIT) + assert run.state is RunState.TIMEOUT + + +def test_a_tick_that_raises_does_not_end_the_thread( + fast: None, monkeypatch: pytest.MonkeyPatch +) -> None: + ticks: list[int] = [] + + def flaky(_self: Scheduler, _now: datetime | None = None) -> list[Any]: + ticks.append(1) + if len(ticks) == 1: + raise RuntimeError("one bad tick") + return [] + + monkeypatch.setattr(Scheduler, "tick", flaky) + engine = Scheduler() + try: + engine.start() + assert wait_until(lambda: len(ticks) >= 3) + finally: + engine.shutdown() + + +def test_a_scheduler_without_autostart_has_no_thread_until_it_is_started(rig: Rig) -> None: + before = loops() + rig.engine.add("job", EVERY_MINUTE, [[rig.command("ok", lambda: 1)]], timezone="UTC") + assert loops() == before + rig.engine.start() + rig.engine.start() + assert len(loops() - before) == 1 + rig.engine.shutdown() + assert loops() - before == set() + + +def test_adding_a_job_starts_the_thread_again_after_a_shutdown(fast: None) -> None: + before = loops() + with Rig(autostart=True) as rig: + action = rig.command("ok", lambda: 1) + rig.engine.add_job("first", [[action]]) + (first,) = loops() - before + rig.engine.shutdown() + assert first.is_alive() is False + rig.engine.add_job("second", [[action]]) + (second,) = loops() - before + assert second is not first and second.daemon is True + + +def test_shutdown_stops_the_bus_subscriptions_and_start_brings_them_back(rig: Rig) -> None: + rig.engine.add_job( + "job", [[rig.command("ok", lambda: 1)]], triggers=EventTrigger(types="deploy.finished") + ) + assert rig.bus.publish(_Deploy()) == 2 + assert rig.runs("job")[0].wait(WAIT) + rig.engine.shutdown() + assert rig.bus.publish(_Deploy()) == 1 # the rig's collector only + assert len(rig.runs("job")) == 1 + assert [job["name"] for job in rig.engine.list()] == ["job"] + rig.engine.start() + assert rig.bus.publish(_Deploy()) == 2 + assert wait_until(lambda: len(rig.runs("job")) == 2) + assert all(run.wait(WAIT) for run in rig.runs("job")) + rig.engine.start() + assert rig.bus.publish(_Deploy()) == 2 + + +def test_remove_all_stops_every_trigger(rig: Rig, tmp_path: Path) -> None: + before = set(threading.enumerate()) + rig.engine.add_job("a", [["FA_schedule_list"]], triggers=EventTrigger(types="deploy.*")) + rig.engine.add_job("b", [["FA_schedule_list"]], triggers=FileTrigger(str(tmp_path))) + watching = set(threading.enumerate()) - before + assert watching + assert [job["name"] for job in rig.engine.remove_all()] == ["a", "b"] + assert rig.bus.publish(_Deploy()) == 1 + assert wait_until(lambda: not any(thread.is_alive() for thread in watching)) + assert rig.engine.list() == [] + + +def test_a_trigger_that_cannot_be_armed_again_is_logged_and_the_rest_go_on( + rig: Rig, tmp_path: Path +) -> None: + watched = tmp_path / "inbox" + watched.mkdir() + rig.engine.add_job("files", [["FA_schedule_list"]], triggers=FileTrigger(str(watched))) + rig.engine.add_job( + "events", + [[rig.command("ok", lambda: 1)]], + triggers=EventTrigger(types="deploy.finished"), + ) + rig.engine.shutdown() + watched.rmdir() + rig.engine.start() + assert rig.bus.publish(_Deploy()) == 2 + assert wait_until(lambda: len(rig.runs("events")) == 1) + assert rig.runs("events")[0].wait(WAIT) + assert [job["name"] for job in rig.engine.list()] == ["files", "events"] + + +def test_a_removed_job_is_not_fired_by_a_trigger_that_is_still_in_flight(rig: Rig) -> None: + rig.engine.add_job("job", [["FA_schedule_list"]]) + rig.engine.remove("job") + # pylint: disable-next=protected-access # what a trigger calls when its moment has come + assert rig.engine._fire("job", TriggerKind.EVENT, {}) is None + with pytest.raises(SchedulerException, match="no such job: job"): + rig.engine.remove("job") + + +def test_dispatch_keeps_accepting_a_bare_job_and_a_naive_moment(rig: Rig) -> None: + job = ScheduledJob( + name="bare", + cron=CronExpression.parse(EVERY_MINUTE), + action_list=[[rig.command("ok", lambda: 1)]], + ) + assert [trigger.to_dict() for trigger in job.triggers] == [ + {"kind": "cron", "cron": EVERY_MINUTE, "timezone": None} + ] + odd = CronExpression( + frozenset({0}), frozenset({2}), frozenset({1}), frozenset({1}), frozenset({1}), "nightly" + ) + assert ScheduledJob(name="odd", cron=odd, action_list=[]).cron is odd + moment = datetime(2026, 4, 21, 12, 0) + # pylint: disable-next=protected-access # the entry point the old tests and callers use + run = rig.engine._dispatch(job, moment) + assert run.wait(WAIT) + assert (job.runs, job.last_run, job.running) == (1, moment, False) + assert run.scheduled_at == moment.astimezone(run.scheduled_at.tzinfo) + assert run.trigger == "cron" diff --git a/tests/test_scheduler_pipeline.py b/tests/test_scheduler_pipeline.py new file mode 100644 index 0000000..8e5e737 --- /dev/null +++ b/tests/test_scheduler_pipeline.py @@ -0,0 +1,469 @@ +"""Pipelines as scheduler targets: the declared schedule, parameters, dependencies, stopping.""" + +from __future__ import annotations + +import json +import time +import zoneinfo +from collections.abc import Iterator +from datetime import datetime, timezone +from pathlib import Path +from typing import Any + +import pytest + +from automation_file.pipeline import ( + MemoryRunStore, + Pipeline, + PipelineDefinitionException, + Schedule, + TaskContext, + set_default_run_store, +) +from automation_file.scheduler import ( + CronException, + CronTrigger, + PipelineTrigger, + RunState, + SchedulerException, + TriggerKind, +) +from automation_file.scheduler.targets import fill_params +from tests.scheduler_kit import START, WAIT, Rig, wait_until + +UTC = timezone.utc +NIGHTLY = Schedule("0 2 * * *", "UTC") + + +@pytest.fixture +def rig() -> Iterator[Rig]: + with Rig() as made: + yield made + + +@pytest.fixture(autouse=True) +def store() -> Iterator[MemoryRunStore]: + """A default run store of this test's own: scheduled pipelines record their runs there.""" + fresh = MemoryRunStore() + previous = set_default_run_store(fresh) + yield fresh + set_default_run_store(previous) + + +def simple(name: str, schedule: Schedule | None = None, result: Any = 1) -> Pipeline: + pipeline = Pipeline(name, schedule=schedule) + pipeline.task("load", lambda ctx: result) + return pipeline + + +def failing(name: str, schedule: Schedule | None = None) -> Pipeline: + def boom(_ctx: TaskContext) -> None: + raise ValueError("the report is empty") + + pipeline = Pipeline(name, schedule=schedule) + pipeline.task("load", boom) + return pipeline + + +def waiting(name: str, entered: list[str]) -> Pipeline: + """A pipeline whose only task goes on until its run is cancelled.""" + + def hold(ctx: TaskContext) -> str: + entered.append(ctx.run_id) + deadline = time.monotonic() + WAIT + while not ctx.cancel.is_cancelled and time.monotonic() < deadline: + time.sleep(0.005) + ctx.cancel.raise_if_cancelled() + return "never cancelled" + + pipeline = Pipeline(name) + pipeline.task("hold", hold) + return pipeline + + +def types(rig: Rig) -> list[str]: + return [event.type for event in rig.events] + + +# ---------------------------------------------------------------------- registration + + +def test_a_pipeline_with_a_schedule_is_registered_with_one_call(rig: Rig) -> None: + snapshot = rig.engine.add_pipeline(simple("daily-report", NIGHTLY)) + assert snapshot == { + "name": "daily-report", + "cron": "0 2 * * *", + "actions": 0, + "last_run": None, + "runs": 0, + "allow_overlap": False, + "running": False, + "skipped": 0, + "timezone": "UTC", + "triggers": [{"kind": "cron", "cron": "0 2 * * *", "timezone": "UTC"}], + "target": "pipeline", + "pipeline": "daily-report", + "timeout": None, + "last_state": None, + } + assert "daily-report" in rig.engine + + +def test_a_scheduled_pipeline_runs_at_its_minute(rig: Rig, store: MemoryRunStore) -> None: + rig.engine.add_pipeline(simple("daily-report", NIGHTLY)) + assert rig.engine.tick(datetime(2026, 10, 8, 1, 59, tzinfo=UTC)) == [] + (run,) = rig.engine.tick(datetime(2026, 10, 8, 2, 0, tzinfo=UTC)) + assert run.wait(WAIT) + assert (run.state, run.trigger, run.target, run.pipeline) == ( + RunState.COMPLETED, + TriggerKind.CRON, + "pipeline", + "daily-report", + ) + stored = store.get_run(run.correlation_id) + assert stored is not None and stored.status == "succeeded" + assert run.correlation_id != run.run_id + assert types(rig) == [ + "pipeline.started", + "task.started", + "task.completed", + "pipeline.completed", + ] + assert {event.correlation_id for event in rig.events} == {run.correlation_id} + assert {event.actor for event in rig.events} == {"scheduler"} + assert rig.job("daily-report")["last_state"] == "completed" + + +def test_the_schedule_of_a_pipeline_keeps_its_time_zone(rig: Rig) -> None: + try: + zoneinfo.ZoneInfo("Asia/Taipei") + except zoneinfo.ZoneInfoNotFoundError: + pytest.skip("no IANA zone data for Asia/Taipei (install tzdata)") + seen: list[Any] = [] + pipeline = Pipeline("daily-report", schedule=Schedule("0 2 * * *", "Asia/Taipei")) + pipeline.task("load", lambda ctx: seen.append(ctx.params["date"])) + snapshot = rig.engine.add_pipeline(pipeline, params={"date": "${date:%Y-%m-%d %H:%M}"}) + assert snapshot["timezone"] == "Asia/Taipei" + assert rig.engine.tick(datetime(2026, 10, 8, 2, 0, tzinfo=UTC)) == [] + (run,) = rig.engine.tick(datetime(2026, 10, 7, 18, 0, tzinfo=UTC)) + assert run.wait(WAIT) + # 18:00 UTC on the 7th is 02:00 on the 8th in Taipei: the date is the job's, not UTC's. + assert seen == ["2026-10-08 02:00"] + + +def test_a_schedule_in_an_unknown_zone_is_refused(rig: Rig) -> None: + with pytest.raises(CronException, match="unknown time zone"): + rig.engine.add_pipeline(simple("daily", Schedule("0 2 * * *", "Not/AZone"))) + assert rig.engine.list() == [] + + +def test_a_pipeline_without_a_schedule_is_fired_by_hand_or_by_other_triggers(rig: Rig) -> None: + snapshot = rig.engine.add_pipeline(simple("on-demand")) + assert (snapshot["cron"], snapshot["triggers"]) == ("", []) + assert rig.tick(minutes=1) == [] + run = rig.engine.run_now("on-demand") + assert run.wait(WAIT) + assert (run.state, run.trigger) == (RunState.COMPLETED, TriggerKind.MANUAL) + + +def test_extra_triggers_are_added_to_the_declared_schedule(rig: Rig) -> None: + snapshot = rig.engine.add_pipeline( + simple("daily", NIGHTLY), name="twice", triggers=CronTrigger("0 14 * * *", "UTC") + ) + assert snapshot["name"] == "twice" + assert [trigger["cron"] for trigger in snapshot["triggers"]] == ["0 2 * * *", "0 14 * * *"] + assert len(rig.engine.tick(datetime(2026, 10, 8, 14, 0, tzinfo=UTC))) == 1 + + +def test_add_job_takes_a_pipeline_and_does_not_read_its_schedule(rig: Rig) -> None: + snapshot = rig.engine.add_job("explicit", simple("daily", NIGHTLY)) + assert (snapshot["cron"], snapshot["triggers"], snapshot["target"]) == ("", [], "pipeline") + assert rig.engine.tick(datetime(2026, 10, 8, 2, 0, tzinfo=UTC)) == [] + + +def test_a_definition_mapping_and_a_definition_file_are_targets(rig: Rig, tmp_path: Path) -> None: + calls: list[str] = [] + definition = { + "schema_version": 1, + "name": "from-document", + "schedule": {"cron": "0 2 * * *", "timezone": "UTC"}, + "params": {"label": "default"}, + "tasks": {"mark": {"action": [rig.command("mark", calls.append), ["${params.label}"]]}}, + } + path = tmp_path / "from-file.json" + path.write_text(json.dumps({**definition, "name": "from-file"}), encoding="utf-8") + assert rig.engine.add_pipeline(definition)["name"] == "from-document" + assert rig.engine.add_pipeline(str(path), params={"label": "file"})["name"] == "from-file" + runs = rig.tick() + assert sorted(run.job for run in runs) == ["from-document", "from-file"] + assert all(run.wait(WAIT) for run in runs) + assert {run.state for run in runs} == {RunState.COMPLETED} + assert sorted(calls) == ["default", "file"] + + +def test_a_wrong_definition_is_refused_when_it_is_registered(rig: Rig, tmp_path: Path) -> None: + with pytest.raises(PipelineDefinitionException, match="at least one task"): + rig.engine.add_pipeline(Pipeline("empty")) + with pytest.raises(PipelineDefinitionException, match="schema_version"): + rig.engine.add_pipeline({"name": "no-version", "tasks": {}}) + with pytest.raises(PipelineDefinitionException): + rig.engine.add_pipeline(str(tmp_path / "missing.yaml")) + for wrong in (5, None): + with pytest.raises(SchedulerException, match="expected a Pipeline"): + rig.engine.add_job("job", wrong) + assert rig.engine.list() == [] + + +@pytest.mark.parametrize("name", ["", " ", None, 5]) +def test_a_job_needs_a_name(rig: Rig, name: Any) -> None: + with pytest.raises(SchedulerException, match="job name"): + rig.engine.add_job(name, simple("daily")) + + +def test_a_pipeline_cannot_depend_on_itself(rig: Rig) -> None: + with pytest.raises(SchedulerException, match="cannot be fired by its own runs"): + rig.engine.add_pipeline(simple("daily"), triggers=PipelineTrigger("daily")) + assert rig.engine.list() == [] + + +# ---------------------------------------------------------------------- parameters + + +def test_fill_params_replaces_the_date_placeholders() -> None: + moment = datetime(2026, 10, 8, 2, 0, tzinfo=UTC) + assert fill_params( + { + "date": "${date:%Y-%m-%d}", + "stamp": "${date}", + "path": "reports/${date:%Y}/${DATE:%m}.csv", + "limit": 20, + "nested": {"days": ["${date:%d}", 7]}, + "other": "${params.x} ${env:HOME}", + }, + moment, + ) == { + "date": "2026-10-08", + "stamp": "2026-10-08T02:00:00", + "path": "reports/2026/10.csv", + "limit": 20, + "nested": {"days": ["08", 7]}, + "other": "${params.x} ${env:HOME}", + } + + +def test_the_params_of_a_job_reach_every_run(rig: Rig) -> None: + seen: list[dict[str, Any]] = [] + pipeline = Pipeline("daily", params={"limit": 5, "date": "unset"}, schedule=NIGHTLY) + pipeline.task("load", lambda ctx: seen.append(dict(ctx.params))) + rig.engine.add_pipeline(pipeline, params={"date": "${date:%Y-%m-%d}"}) + (run,) = rig.tick() + assert run.wait(WAIT) + assert seen == [{"limit": 5, "date": "2026-10-08"}] + + +@pytest.mark.parametrize("params", [["date"], "date", 5]) +def test_params_are_a_mapping(rig: Rig, params: Any) -> None: + with pytest.raises(SchedulerException, match="params: expected a mapping"): + rig.engine.add_pipeline(simple("daily"), params=params) + + +def test_an_action_list_takes_no_params(rig: Rig) -> None: + with pytest.raises(SchedulerException, match="params belong to a pipeline"): + rig.engine.add_job("job", [["FA_schedule_list"]], params={"date": "x"}) + + +# ---------------------------------------------------------------------- failures + + +def test_a_pipeline_that_fails_is_recorded_and_published(rig: Rig) -> None: + rig.engine.add_pipeline(failing("daily", NIGHTLY)) + (run,) = rig.tick() + assert run.wait(WAIT) + assert run.state is RunState.FAILED + assert run.error == "did not succeed: load" + assert types(rig) == [ + "pipeline.started", + "task.started", + "task.failed", + "pipeline.failed", + "scheduler.error", + ] + assert {event.correlation_id for event in rig.events} == {run.correlation_id} + (event,) = rig.errors() + assert event.subject == "scheduler[daily] failed" + assert dict(event.payload) == { + "job": "daily", + "trigger": "cron", + "status": "failed", + "error": "did not succeed: load", + "target": "pipeline", + "scheduler_run_id": run.run_id, + "duration_ms": 0.0, + "pipeline": "daily", + "run_id": run.correlation_id, + } + assert rig.job("daily")["running"] is False + + +def test_a_run_that_cannot_start_is_recorded_as_failed(rig: Rig) -> None: + pipeline = Pipeline("daily") + pipeline.task("load", ["FA_schedule_list", {"unused": "${params.date}"}]) + rig.engine.add_pipeline(pipeline) + run = rig.engine.run_now("daily") + assert run.wait(WAIT) + assert run.state is RunState.FAILED + assert str(run.error).startswith("PipelineDefinitionException: ") + assert "date" in str(run.error) + assert run.correlation_id == run.run_id + (event,) = rig.errors() + assert event.correlation_id == run.run_id + assert "run_id" not in event.payload + assert event.payload["pipeline"] == "daily" + + +# ---------------------------------------------------------------------- dependencies + + +def test_a_dependent_pipeline_runs_after_the_other_succeeded( + rig: Rig, store: MemoryRunStore +) -> None: + rig.engine.add_pipeline(simple("daily-report", NIGHTLY)) + snapshot = rig.engine.add_pipeline( + simple("publish-summary"), triggers=PipelineTrigger("daily-report") + ) + assert snapshot["triggers"] == [ + {"kind": "pipeline", "pipeline": "daily-report", "when": "on_success"} + ] + (first,) = rig.tick() + assert first.wait(WAIT) + assert wait_until(lambda: len(rig.runs("publish-summary")) == 1) + (second,) = rig.runs("publish-summary") + assert second.wait(WAIT) + assert (second.state, second.trigger) == (RunState.COMPLETED, TriggerKind.PIPELINE) + assert second.detail == { + "pipeline": "daily-report", + "run_id": first.correlation_id, + "status": "succeeded", + } + assert [run.pipeline for run in store.list_runs(None, 10)] == [ + "publish-summary", + "daily-report", + ] + + +@pytest.mark.parametrize( + ("when", "upstream_fails", "fires"), + [ + ("on_success", False, True), + ("on_success", True, False), + ("on_failure", False, False), + ("on_failure", True, True), + ("always", False, True), + ("always", True, True), + ], +) +def test_a_dependency_looks_at_how_the_other_run_ended( + rig: Rig, when: str, upstream_fails: bool, fires: bool +) -> None: + upstream = failing("upstream") if upstream_fails else simple("upstream") + rig.engine.add_pipeline(upstream) + rig.engine.add_pipeline(simple("downstream"), triggers=PipelineTrigger("upstream", when)) + assert rig.engine.run_now("upstream").wait(WAIT) + assert len(rig.runs("downstream")) == int(fires) + assert all(run.wait(WAIT) for run in rig.runs("downstream")) + if fires: + status = "failed" if upstream_fails else "succeeded" + assert rig.runs("downstream")[0].detail["status"] == status + + +def test_a_dependency_also_sees_a_run_started_outside_the_scheduler(rig: Rig) -> None: + rig.engine.add_pipeline(simple("downstream"), triggers=PipelineTrigger("upstream")) + outside = simple("upstream").run(bus=rig.bus) + simple("unrelated").run(bus=rig.bus) + (run,) = rig.runs("downstream") + assert run.wait(WAIT) + assert run.detail["run_id"] == outside.run_id + + +def test_two_pipelines_that_depend_on_each_other_stop_after_sixteen_runs(rig: Rig) -> None: + rig.engine.add_pipeline(simple("ping"), triggers=PipelineTrigger("pong"), allow_overlap=True) + rig.engine.add_pipeline(simple("pong"), triggers=PipelineTrigger("ping"), allow_overlap=True) + rig.engine.run_now("ping") + assert wait_until(lambda: bool(rig.engine.history(state="skipped"))) + assert rig.idle("ping") and rig.idle("pong") + records = rig.engine.history(limit=100) + assert [run.state.value for run in records] == ["skipped"] + ["completed"] * 16 + assert (records[0].reason, records[0].trigger) == ("chain", TriggerKind.PIPELINE) + assert rig.errors() == [] + + +def test_an_action_list_can_depend_on_a_pipeline(rig: Rig) -> None: + calls: list[int] = [] + rig.engine.add_job( + "after-daily", + [[rig.command("count", lambda: calls.append(1))]], + triggers={"kind": "pipeline", "pipeline": "daily"}, + ) + simple("daily").run(bus=rig.bus) + (run,) = rig.runs("after-daily") + assert run.wait(WAIT) + assert calls == [1] + + +# ---------------------------------------------------------------------- stopping + + +def test_a_pipeline_past_its_timeout_is_cancelled_through_its_token( + rig: Rig, store: MemoryRunStore +) -> None: + entered: list[str] = [] + rig.engine.add_pipeline(waiting("slow", entered), timeout=60) + run = rig.engine.run_now("slow") + assert wait_until(lambda: bool(entered)) + assert run.correlation_id == entered[0] # the record follows the pipeline's run ID at once + assert rig.tick(seconds=59) == [] + assert run.state is RunState.STARTED + rig.tick(seconds=1) + assert run.state is RunState.TIMEOUT + assert run.error == "TimeoutError: not finished within 60 s" + assert rig.idle("slow") + stored = store.get_run(entered[0]) + assert stored is not None and stored.status == "cancelled" + (event,) = rig.errors() + assert (event.subject, event.payload["status"]) == ("scheduler[slow] timed out", "timeout") + assert (event.payload["pipeline"], event.payload["run_id"]) == ("slow", entered[0]) + assert event.correlation_id == entered[0] + assert run.state is RunState.TIMEOUT + + +def test_cancelling_a_scheduled_pipeline_stops_its_run(rig: Rig, store: MemoryRunStore) -> None: + entered: list[str] = [] + rig.engine.add_pipeline(waiting("slow", entered)) + run = rig.engine.run_now("slow") + assert wait_until(lambda: bool(entered)) + assert rig.engine.cancel("slow") == [run] + assert (run.state, run.reason) == (RunState.CANCELLED, "cancelled") + assert rig.idle("slow") + stored = store.get_run(entered[0]) + assert stored is not None and stored.status == "cancelled" + assert rig.errors() == [] + assert rig.engine.history(job="slow", state="cancelled") == [run] + + +def test_shutdown_can_cancel_what_is_running(store: MemoryRunStore) -> None: + entered: list[str] = [] + with Rig() as rig: + rig.engine.add_pipeline(waiting("slow", entered)) + run = rig.engine.run_now("slow") + assert wait_until(lambda: bool(entered)) + rig.engine.shutdown() + assert run.state is RunState.STARTED + rig.engine.shutdown(cancel_running=True) + assert run.state is RunState.CANCELLED + assert rig.idle("slow") + stored = store.get_run(entered[0]) + assert stored is not None and stored.status == "cancelled" + + +def test_the_start_time_of_the_fixture_is_the_nightly_minute() -> None: + assert CronTrigger(NIGHTLY.cron, NIGHTLY.timezone).due(START) is True diff --git a/tests/test_scheduler_runs.py b/tests/test_scheduler_runs.py new file mode 100644 index 0000000..ecdec5e --- /dev/null +++ b/tests/test_scheduler_runs.py @@ -0,0 +1,641 @@ +"""Run records of the scheduler: the seven states, overlap, timeout, cancellation, history.""" + +from __future__ import annotations + +import json +import threading +from collections.abc import Iterator +from datetime import timedelta, timezone +from typing import Any + +import pytest + +from automation_file.events import Event, SchedulerError, Severity, event_bus +from automation_file.exceptions import FileAutomationException +from automation_file.scheduler import ( + EventTrigger, + JobRun, + RunHistory, + RunState, + Scheduler, + SchedulerException, + TriggerKind, + dispatch, + manager, +) +from tests.scheduler_kit import START, WAIT, Rig, wait_until + +EVERY_MINUTE = "* * * * *" + + +@pytest.fixture +def rig() -> Iterator[Rig]: + with Rig() as made: + yield made + + +@pytest.fixture +def held(monkeypatch: pytest.MonkeyPatch) -> list[threading.Thread]: + """Threads that were asked to start and are kept back until the test starts them.""" + kept: list[threading.Thread] = [] + + def hold(thread: threading.Thread) -> None: + kept.append(thread) + + monkeypatch.setattr(threading.Thread, "start", hold) + return kept + + +def test_the_seven_states_and_which_of_them_are_final() -> None: + assert [state.value for state in RunState] == [ + "scheduled", + "started", + "completed", + "failed", + "skipped", + "timeout", + "cancelled", + ] + assert [state for state in RunState if not state.is_final] == [ + RunState.SCHEDULED, + RunState.STARTED, + ] + assert RunState.FAILED == "failed" + + +def test_a_run_is_scheduled_then_started_then_completed(rig: Rig) -> None: + action, gate = rig.gate() + rig.engine.add_job("job", [[action]]) + run = rig.engine.run_now("job") + gate.await_entry() + assert run.state is RunState.STARTED + assert run.done is False + assert run.started_at == START + assert rig.job("job")["running"] is True + rig.clock.advance(seconds=2) + gate.open() + assert run.wait(WAIT) is True + assert run.state is RunState.COMPLETED + assert (run.scheduled_at, run.started_at, run.finished_at) == ( + START, + START, + START + timedelta(seconds=2), + ) + assert run.duration_ms == 2000.0 + assert run.error is None + assert rig.job("job") == { + "name": "job", + "cron": "", + "actions": 1, + "last_run": START.astimezone().replace(tzinfo=None).isoformat(), + "runs": 1, + "allow_overlap": False, + "running": False, + "skipped": 0, + "timezone": None, + "triggers": [], + "target": "actions", + "pipeline": None, + "timeout": None, + "last_state": "completed", + } + + +def test_a_firing_is_recorded_as_scheduled_before_its_thread_runs( + rig: Rig, held: list[threading.Thread], monkeypatch: pytest.MonkeyPatch +) -> None: + rig.engine.add_job("job", [[rig.command("ok", lambda: 1)]]) + run = rig.engine.run_now("job") + assert run.state is RunState.SCHEDULED + assert (run.started_at, run.finished_at) == (None, None) + assert rig.job("job")["running"] is True + monkeypatch.undo() + held[0].start() + assert run.wait(WAIT) is True + assert run.state is RunState.COMPLETED + + +def test_the_times_of_a_record_are_aware_utc(rig: Rig) -> None: + rig.engine.add_job("job", [[rig.command("ok", lambda: 1)]]) + run = rig.engine.run_now("job") + assert run.wait(WAIT) + for moment in (run.scheduled_at, run.started_at, run.finished_at): + assert moment is not None and moment.utcoffset() == timedelta(0) + assert moment.tzinfo is timezone.utc + + +def test_a_record_is_json_friendly(rig: Rig) -> None: + rig.engine.add( + "job", EVERY_MINUTE, [[rig.command("ok", lambda: 1)]], timezone="UTC", timeout=30 + ) + (run,) = rig.engine.tick() + assert run.wait(WAIT) + document = json.loads(json.dumps(run.to_dict())) + assert document == { + "run_id": run.run_id, + "job": "job", + "trigger": "cron", + "target": "actions", + "pipeline": None, + "state": "completed", + "scheduled_at": "2026-10-08T02:00:00.000000+00:00", + "started_at": "2026-10-08T02:00:00.000000+00:00", + "finished_at": "2026-10-08T02:00:00.000000+00:00", + "duration_ms": 0.0, + "error": None, + "reason": None, + "correlation_id": run.run_id, + "detail": {"cron": EVERY_MINUTE, "timezone": "UTC"}, + } + assert len(run.run_id) == 32 + + +def test_the_target_runs_inside_the_runs_correlation_scope(rig: Rig) -> None: + from automation_file.events import current_actor, current_correlation_id + + seen: list[tuple[str | None, str]] = [] + rig.engine.add_job( + "job", + [[rig.command("look", lambda: seen.append((current_correlation_id(), current_actor())))]], + ) + run = rig.engine.run_now("job") + assert run.wait(WAIT) + assert seen == [(run.correlation_id, "scheduler")] + assert run.correlation_id == run.run_id + + +# ---------------------------------------------------------------------- overlap + + +def test_a_firing_that_overlaps_is_recorded_as_skipped(rig: Rig) -> None: + action, gate = rig.gate() + rig.engine.add_job("job", [[action]]) + first = rig.engine.run_now("job") + gate.await_entry() + second = rig.engine.run_now("job") + assert second.state is RunState.SKIPPED + assert second.reason == "overlap" + assert second.done and second.wait(0) is True + assert (second.started_at, second.finished_at) == (None, START) + assert rig.job("job")["skipped"] == 1 + assert rig.job("job")["runs"] == 1 + gate.open() + assert first.wait(WAIT) and first.state is RunState.COMPLETED + assert gate.calls == 1 + assert [run.state.value for run in rig.runs("job")] == ["skipped", "completed"] + assert rig.errors() == [] + + +def test_overlap_is_allowed_when_the_job_says_so(rig: Rig) -> None: + action, gate = rig.gate() + rig.engine.add_job("job", [[action]], allow_overlap=True) + first = rig.engine.run_now("job") + second = rig.engine.run_now("job") + assert wait_until(lambda: gate.calls == 2) + assert rig.job("job")["skipped"] == 0 + gate.open() + assert first.wait(WAIT) and second.wait(WAIT) + assert {first.state, second.state} == {RunState.COMPLETED} + assert rig.idle("job") + assert rig.job("job")["runs"] == 2 + + +def test_a_job_counts_as_running_until_its_last_overlapping_run_has_ended(rig: Rig) -> None: + slow_action, slow = rig.gate("slow") + rig.engine.add_job("job", [[slow_action]], allow_overlap=True) + first = rig.engine.run_now("job") + slow.await_entry() + second = rig.engine.run_now("job") + assert wait_until(lambda: slow.calls == 2) + assert rig.job("job")["running"] is True + slow.open() + assert first.wait(WAIT) and second.wait(WAIT) + assert rig.idle("job") + + +@pytest.mark.parametrize("kind", ["cron", "manual", "event"]) +def test_overlap_protection_covers_every_kind_of_trigger(rig: Rig, kind: str) -> None: + action, gate = rig.gate() + rig.engine.add_job( + "job", + [[action]], + triggers=[{"kind": "cron", "cron": EVERY_MINUTE}, EventTrigger(types="deploy.finished")], + ) + first = rig.engine.run_now("job") + gate.await_entry() + if kind == "cron": + rig.tick() + elif kind == "manual": + rig.engine.run_now("job") + else: + rig.bus.publish(_Deploy(source="webhook")) + skipped = rig.engine.history(job="job", state=RunState.SKIPPED) + assert [(run.trigger.value, run.reason) for run in skipped] == [(kind, "overlap")] + gate.open() + assert first.wait(WAIT) + assert gate.calls == 1 + + +# ---------------------------------------------------------------------- failures + + +def test_an_action_that_raises_fails_the_run_and_the_list_goes_on(rig: Rig) -> None: + def boom() -> None: + raise ValueError("the report is empty") + + later: list[int] = [] + rig.engine.add_job( + "job", [[rig.command("boom", boom)], [rig.command("later", lambda: later.append(1))]] + ) + run = rig.engine.run_now("job") + assert run.wait(WAIT) + assert run.state is RunState.FAILED + assert run.error == ( + "1 of 2 actions failed: execute[0] scheduler_test_boom: ValueError: the report is empty" + ) + assert later == [1] + assert rig.job("job")["last_state"] == "failed" + assert rig.job("job")["running"] is False + + +def test_a_failed_run_is_published_as_a_scheduler_error(rig: Rig) -> None: + def boom() -> None: + raise ValueError("the report is empty") + + rig.engine.add("job", EVERY_MINUTE, [[rig.command("boom", boom)]], timezone="UTC") + (run,) = rig.engine.tick() + assert run.wait(WAIT) + (event,) = rig.errors() + assert isinstance(event, SchedulerError) + assert (event.type, event.source, event.severity) == ( + "scheduler.error", + "scheduler", + Severity.ERROR, + ) + assert event.subject == "scheduler[job] failed" + assert event.correlation_id == run.correlation_id + assert event.actor == "scheduler" + assert dict(event.payload) == { + "job": "job", + "trigger": "cron", + "status": "failed", + "error": run.error, + "target": "actions", + "scheduler_run_id": run.run_id, + "duration_ms": 0.0, + } + + +def test_the_error_text_keeps_a_url_down_to_its_host(rig: Rig) -> None: + def boom() -> None: + raise ConnectionError("POST https://hooks.example.com/services/T0/B0/secret failed") + + rig.engine.add_job("job", [[rig.command("boom", boom)]]) + run = rig.engine.run_now("job") + assert run.wait(WAIT) + assert "secret" not in str(run.error) + assert "hooks.example.com" in str(run.error) + assert "secret" not in json.dumps(rig.errors()[0].to_dict()) + + +def test_many_failures_are_summarised(rig: Rig) -> None: + def boom() -> None: + raise ValueError("no") + + action = rig.command("boom", boom) + rig.engine.add_job("job", [[action]] * 7) + run = rig.engine.run_now("job") + assert run.wait(WAIT) + assert str(run.error).startswith("7 of 7 actions failed: execute[0] ") + assert str(run.error).endswith("; and 2 more") + assert str(run.error).count("execute[") == 5 + + +def test_a_list_the_executor_rejects_is_reported_through_notify_on_failure( + rig: Rig, monkeypatch: pytest.MonkeyPatch +) -> None: + from automation_file.notify import manager as notify_manager + + reported: list[tuple[str, BaseException]] = [] + monkeypatch.setattr( + notify_manager, + "notify_on_failure", + lambda context, error: reported.append((context, error)), + ) + rig.engine.add("job", EVERY_MINUTE, []) + (run,) = rig.engine.tick() + assert run.wait(WAIT) + assert run.state is RunState.FAILED + assert run.error == "ExecuteActionException: action_list is empty" + assert [(context, type(error).__name__) for context, error in reported] == [ + ("scheduler[job]", "ExecuteActionException") + ] + assert rig.errors() == [] + + +def test_a_rejected_list_gives_exactly_one_scheduler_error_on_the_process_bus() -> None: + seen: list[Any] = [] + subscription = event_bus.subscribe(seen.append, types="scheduler.error") + engine = Scheduler(autostart=False) + try: + engine.add_job("rejected-list", []) + run = engine.run_now("rejected-list") + assert run.wait(WAIT) + finally: + event_bus.unsubscribe(subscription) + engine.shutdown() + assert [(event.subject, event.correlation_id) for event in seen] == [ + ("scheduler[rejected-list] failed", run.correlation_id) + ] + assert seen[0].payload["job"] == "rejected-list" + + +def test_a_run_ended_by_a_base_exception_still_releases_the_job( + rig: Rig, held: list[threading.Thread], monkeypatch: pytest.MonkeyPatch +) -> None: + def leave() -> None: + raise SystemExit(3) # not an Exception: nothing in the run's thread catches it + + rig.engine.add_job("job", [[rig.command("leave", leave)]]) + run = rig.engine.run_now("job") + monkeypatch.undo() + with pytest.raises(SystemExit): + held[0].run() # the body of the run's thread, called here so the exit is seen + assert run.wait(0) is True + assert run.state is RunState.FAILED + assert run.error == "the run's thread ended without a result" + assert rig.job("job")["running"] is False + assert [event.payload["status"] for event in rig.errors()] == ["failed"] + + +def test_a_target_that_raises_unexpectedly_is_recorded_and_the_job_released( + monkeypatch: pytest.MonkeyPatch, +) -> None: + def broken(*_arguments: Any) -> None: + raise RuntimeError("the runner itself broke") + + monkeypatch.setattr(manager, "_safe_execute", broken) + with Rig() as rig: + rig.engine.add_job("job", [["FA_schedule_list"]]) + run = rig.engine.run_now("job") + assert run.wait(WAIT) + assert run.state is RunState.FAILED + assert run.error == "RuntimeError: the runner itself broke" + assert rig.job("job")["running"] is False + assert [event.payload["status"] for event in rig.errors()] == ["failed"] + + +def test_a_thread_that_cannot_start_fails_the_run_and_releases_the_job( + rig: Rig, monkeypatch: pytest.MonkeyPatch +) -> None: + def refuse(_thread: threading.Thread) -> None: + raise RuntimeError("can't start new thread") + + rig.engine.add_job("job", [[rig.command("ok", lambda: 1)]]) + monkeypatch.setattr(dispatch.threading.Thread, "start", refuse) + run = rig.engine.run_now("job") + monkeypatch.undo() + assert run.state is RunState.FAILED + assert run.error == "RuntimeError: can't start new thread" + assert rig.job("job")["running"] is False + assert rig.engine.run_now("job").wait(WAIT) + + +def test_safe_execute_reports_how_the_list_went(rig: Rig) -> None: + def boom() -> None: + raise ValueError("no") + + good = manager._safe_execute("unit", [[rig.command("ok", lambda: 1)]]) + assert (good.state, good.error, good.reported) == (RunState.COMPLETED, None, False) + bad = manager._safe_execute("unit", [[rig.command("boom", boom)]]) + assert (bad.state, bad.reported) == (RunState.FAILED, False) + unknown = manager._safe_execute("unit", [["FA_does_not_exist"]]) + assert unknown.state is RunState.FAILED + assert "FA_does_not_exist" in str(unknown.error) + + +# ---------------------------------------------------------------------- timeout + + +def test_a_run_past_its_timeout_is_recorded_as_timeout(rig: Rig) -> None: + action, gate = rig.gate() + after: list[int] = [] + rig.engine.add( + "job", + EVERY_MINUTE, + [[action], [rig.command("after", lambda: after.append(1))]], + timezone="UTC", + timeout=30, + ) + (run,) = rig.engine.tick() + gate.await_entry() + assert rig.tick(seconds=29) == [] + assert run.state is RunState.STARTED + rig.tick(seconds=1) + assert run.state is RunState.TIMEOUT + assert run.wait(0) is True + assert run.error == "TimeoutError: not finished within 30 s" + assert run.finished_at == START + timedelta(seconds=30) + (event,) = rig.errors() + assert event.subject == "scheduler[job] timed out" + assert (event.payload["status"], event.payload["duration_ms"]) == ("timeout", 30000.0) + assert event.correlation_id == run.correlation_id + # The thread cannot be killed: the job stays busy until the action returns. + assert rig.job("job")["running"] is True + assert rig.engine.run_now("job").state is RunState.SKIPPED + gate.open() + assert rig.idle("job") + assert after == [] + assert run.state is RunState.TIMEOUT + assert rig.job("job")["last_state"] == "timeout" + assert len(rig.errors()) == 1 + assert rig.engine.run_now("job").wait(WAIT) + + +def test_a_run_that_ends_in_time_is_not_touched_by_its_timeout(rig: Rig) -> None: + rig.engine.add_job("job", [[rig.command("ok", lambda: 1)]], timeout=30) + run = rig.engine.run_now("job") + assert run.wait(WAIT) + rig.tick(minutes=5) + assert run.state is RunState.COMPLETED + assert rig.errors() == [] + + +@pytest.mark.parametrize("timeout", [0, -1, True, "60", float("inf"), float("nan"), 1e12]) +def test_a_timeout_must_be_seconds_above_zero(rig: Rig, timeout: Any) -> None: + with pytest.raises(SchedulerException, match="timeout"): + rig.engine.add("job", EVERY_MINUTE, [["FA_schedule_list"]], timeout=timeout) + assert "job" not in rig.engine + + +# ---------------------------------------------------------------------- cancel + + +def test_cancelling_a_running_job_records_the_run_as_cancelled(rig: Rig) -> None: + action, gate = rig.gate() + after: list[int] = [] + rig.engine.add_job("job", [[action], [rig.command("after", lambda: after.append(1))]]) + run = rig.engine.run_now("job") + gate.await_entry() + rig.clock.advance(seconds=3) + assert rig.engine.cancel("job") == [run] + assert run.state is RunState.CANCELLED + assert (run.reason, run.error) == ("cancelled", None) + assert run.finished_at == START + timedelta(seconds=3) + assert run.wait(0) is True + assert rig.job("job")["running"] is True + gate.open() + assert rig.idle("job") + assert after == [] + assert rig.errors() == [] + assert rig.job("job")["last_state"] == "cancelled" + + +def test_cancel_returns_nothing_for_an_idle_job_and_refuses_an_unknown_one(rig: Rig) -> None: + rig.engine.add_job("job", [["FA_schedule_list"]]) + assert rig.engine.cancel("job") == [] + with pytest.raises(SchedulerException, match="no such job: nope"): + rig.engine.cancel("nope") + with pytest.raises(SchedulerException, match="no such job: None"): + rig.engine.cancel(None) + + +def test_a_subscriber_can_fire_the_failed_job_again_at_once(rig: Rig) -> None: + attempts: list[int] = [] + + def flaky() -> None: + attempts.append(1) + if len(attempts) == 1: + raise ValueError("first attempt") + + retried: list[JobRun] = [] + rig.bus.subscribe( + lambda event: retried.append(rig.engine.run_now(event.payload["job"])), + types="scheduler.error", + ) + rig.engine.add_job("job", [[rig.command("flaky", flaky)]]) + first = rig.engine.run_now("job") + assert first.wait(WAIT) and first.state is RunState.FAILED + (second,) = retried + assert second.wait(WAIT) + assert second.state is RunState.COMPLETED + assert rig.job("job")["skipped"] == 0 + + +def test_a_removed_job_that_is_still_running_can_be_cancelled(rig: Rig) -> None: + action, gate = rig.gate() + rig.engine.add_job("job", [[action]]) + run = rig.engine.run_now("job") + gate.await_entry() + rig.engine.remove("job") + assert run.state is RunState.STARTED + assert rig.engine.cancel("job") == [run] + assert run.state is RunState.CANCELLED + + +def test_a_run_cancelled_before_it_started_never_runs_its_target( + rig: Rig, held: list[threading.Thread], monkeypatch: pytest.MonkeyPatch +) -> None: + calls: list[int] = [] + rig.engine.add_job("job", [[rig.command("count", lambda: calls.append(1))]]) + run = rig.engine.run_now("job") + assert rig.engine.cancel("job") == [run] + monkeypatch.undo() + held[0].start() + assert rig.idle("job") + assert calls == [] + assert (run.state, run.started_at) == (RunState.CANCELLED, None) + + +# ---------------------------------------------------------------------- manual + + +def test_run_now_fires_a_job_that_has_no_trigger(rig: Rig) -> None: + calls: list[int] = [] + snapshot = rig.engine.add_job("by-hand", [[rig.command("count", lambda: calls.append(1))]]) + assert (snapshot["triggers"], snapshot["cron"]) == ([], "") + assert rig.tick(minutes=1) == [] + run = rig.engine.run_now("by-hand") + assert run.wait(WAIT) + assert (run.trigger, run.detail) == (TriggerKind.MANUAL, {}) + assert calls == [1] + + +def test_run_now_refuses_an_unknown_job(rig: Rig) -> None: + with pytest.raises(SchedulerException, match="no such job: nope"): + rig.engine.run_now("nope") + + +# ---------------------------------------------------------------------- history + + +def test_the_history_is_newest_first_and_can_be_filtered(rig: Rig) -> None: + def boom() -> None: + raise ValueError("no") + + rig.engine.add_job("good", [[rig.command("ok", lambda: 1)]]) + rig.engine.add_job("bad", [[rig.command("boom", boom)]]) + for name in ("good", "bad", "good"): + assert rig.engine.run_now(name).wait(WAIT) + assert [(run.job, run.state.value) for run in rig.engine.history()] == [ + ("good", "completed"), + ("bad", "failed"), + ("good", "completed"), + ] + assert [run.job for run in rig.engine.history(job="bad")] == ["bad"] + assert [run.job for run in rig.engine.history(state="completed")] == ["good", "good"] + assert [run.job for run in rig.engine.history(state=RunState.FAILED)] == ["bad"] + assert len(rig.engine.history(limit=2)) == 2 + assert rig.engine.history(limit=0) == [] + assert rig.engine.history(job="good", state="failed") == [] + with pytest.raises(SchedulerException, match="unknown run state 'done'"): + rig.engine.history(state="done") + + +def test_the_history_is_bounded() -> None: + with Rig(history_limit=3) as rig: + rig.engine.add_job("job", [[rig.command("ok", lambda: 1)]]) + runs = [rig.engine.run_now("job") for _ in range(5)] + assert all(run.wait(WAIT) for run in runs) + assert wait_until(lambda: not rig.job("job")["running"]) + kept = rig.engine.history(limit=100) + assert len(kept) == 3 + assert {run.job for run in kept} == {"job"} + + +def test_the_history_keeps_the_records_of_a_removed_job(rig: Rig) -> None: + rig.engine.add_job("job", [[rig.command("ok", lambda: 1)]]) + run = rig.engine.run_now("job") + assert run.wait(WAIT) + rig.engine.remove("job") + assert rig.engine.history(job="job") == [run] + + +@pytest.mark.parametrize("limit", [0, -1, True, "10"]) +def test_a_history_needs_room_for_one_record(limit: Any) -> None: + with pytest.raises(SchedulerException, match="history limit"): + RunHistory(limit) + with pytest.raises(SchedulerException, match="history limit"): + Scheduler(history_limit=limit, autostart=False) + + +def test_run_history_by_itself() -> None: + history = RunHistory(2) + runs = [JobRun(f"run-{index}", "job", TriggerKind.MANUAL, START) for index in range(3)] + for run in runs: + history.add(run) + assert len(history) == 2 + assert history.query() == [runs[2], runs[1]] + history.clear() + assert history.query() == [] + + +def test_the_scheduler_exception_is_a_file_automation_exception() -> None: + assert issubclass(SchedulerException, FileAutomationException) + assert manager.SchedulerException is SchedulerException + + +class _Deploy(Event): + """A stand-in for an event some webhook handler publishes.""" + + type = "deploy.finished" diff --git a/tests/test_scheduler_triggers.py b/tests/test_scheduler_triggers.py new file mode 100644 index 0000000..9ae3688 --- /dev/null +++ b/tests/test_scheduler_triggers.py @@ -0,0 +1,613 @@ +"""The scheduler's triggers: cron with a time zone, file events, events on the bus.""" + +from __future__ import annotations + +import zoneinfo +from collections.abc import Iterator +from datetime import datetime, timedelta, timezone +from pathlib import Path +from typing import Any + +import pytest +from watchdog.events import FileCreatedEvent, FileModifiedEvent, FileSystemEventHandler + +from automation_file.events import Event, PipelineFailed, Severity +from automation_file.scheduler import ( + CronException, + CronExpression, + CronTrigger, + EventTrigger, + FileTrigger, + PipelineTrigger, + RunState, + SchedulerException, + Trigger, + TriggerKind, + cron, + resolve_timezone, + trigger_from_dict, +) +from automation_file.scheduler.triggers import as_triggers +from automation_file.trigger import manager as watch_manager +from automation_file.trigger.manager import FileWatcher, TriggerException +from tests.scheduler_kit import START, WAIT, Rig, wait_until + +UTC = timezone.utc + + +@pytest.fixture +def rig() -> Iterator[Rig]: + with Rig() as made: + yield made + + +def zone(name: str) -> zoneinfo.ZoneInfo: + """Return the zone ``name``, or skip the test where no zone data is installed.""" + try: + return zoneinfo.ZoneInfo(name) + except zoneinfo.ZoneInfoNotFoundError: + pytest.skip(f"no IANA zone data for {name} (install tzdata)") + + +def minutes(start: datetime, count: int) -> list[datetime]: + return [start + timedelta(minutes=offset) for offset in range(count)] + + +def fired_at(trigger: CronTrigger, start: datetime, count: int) -> list[str]: + """The UTC minutes, as ``HH:MM``, at which ``trigger`` is due from ``start`` on.""" + return [moment.strftime("%H:%M") for moment in minutes(start, count) if trigger.due(moment)] + + +class _Deploy(Event): + type = "deploy.finished" + + +# ---------------------------------------------------------------------- time zones + + +def test_utc_needs_no_zone_data(monkeypatch: pytest.MonkeyPatch) -> None: + def missing(key: str) -> None: + raise zoneinfo.ZoneInfoNotFoundError(key) + + monkeypatch.setattr(cron.zoneinfo, "ZoneInfo", missing) + assert resolve_timezone("UTC") is UTC + assert resolve_timezone(" utc ") is UTC + assert CronTrigger("0 2 * * *", "UTC").zone is UTC + + +def test_a_zone_without_data_says_to_install_tzdata(monkeypatch: pytest.MonkeyPatch) -> None: + def missing(key: str) -> None: + raise zoneinfo.ZoneInfoNotFoundError(key) + + monkeypatch.setattr(cron.zoneinfo, "ZoneInfo", missing) + with pytest.raises(CronException, match=r"unknown time zone 'Asia/Taipei'.*tzdata"): + resolve_timezone("Asia/Taipei") + with pytest.raises(CronException, match="tzdata"): + CronTrigger("0 2 * * *", "Asia/Taipei") + + +def test_no_time_zone_stays_none() -> None: + assert resolve_timezone(None) is None + assert CronTrigger("0 2 * * *").zone is None + + +@pytest.mark.parametrize("name", ["", " ", 8, "Not/AZone", "../etc/passwd", "/etc/localtime"]) +def test_a_name_that_is_no_time_zone_is_refused(name: Any) -> None: + with pytest.raises(CronException): + resolve_timezone(name) + + +def test_a_cron_trigger_is_read_in_its_time_zone() -> None: + zone("Asia/Taipei") + trigger = CronTrigger("0 2 * * *", "Asia/Taipei") + assert trigger.due(datetime(2026, 10, 7, 18, 0, tzinfo=UTC)) is True + assert trigger.due(datetime(2026, 10, 8, 2, 0, tzinfo=UTC)) is False + moment = trigger.moment(datetime(2026, 10, 7, 18, 0, tzinfo=UTC)) + assert moment.isoformat() == "2026-10-08T02:00:00+08:00" + + +def test_a_cron_trigger_without_a_zone_is_read_in_local_time() -> None: + local = datetime(2026, 4, 21, 9, 30) + instant = local.astimezone(UTC) + assert CronTrigger("30 9 * * *").due(instant) is True + assert CronTrigger("31 9 * * *").due(instant) is False + assert CronTrigger("30 9 * * *").moment(instant) == local + + +def test_a_cron_trigger_keeps_its_expression_and_zone() -> None: + trigger = CronTrigger(" */5 * * * * ", " UTC ") + assert (trigger.cron, trigger.timezone) == ("*/5 * * * *", "UTC") + assert trigger.expression == CronExpression.parse("*/5 * * * *") + assert trigger.kind is TriggerKind.CRON + assert trigger == CronTrigger("*/5 * * * *", "UTC") + assert trigger.to_dict() == {"kind": "cron", "cron": "*/5 * * * *", "timezone": "UTC"} + + +@pytest.mark.parametrize("expression", ["not a cron", "", None, 5]) +def test_a_cron_trigger_refuses_a_bad_expression(expression: Any) -> None: + with pytest.raises(CronException): + CronTrigger(expression) + + +def test_every_hour_tells_a_wildcard_hour_field() -> None: + assert CronExpression.parse("30 * * * *").every_hour is True + assert CronExpression.parse("30 0-23 * * *").every_hour is True + assert CronExpression.parse("30 1 * * *").every_hour is False + assert CronExpression.parse("30 */2 * * *").every_hour is False + + +def test_a_local_time_that_does_not_exist_is_not_fired() -> None: + zone("America/New_York") + # 2026-03-08: the clocks go from 01:59 EST (06:59 UTC) straight to 03:00 EDT (07:00 UTC). + start = datetime(2026, 3, 8, 5, 0, tzinfo=UTC) + assert fired_at(CronTrigger("30 2 * * *", "America/New_York"), start, 240) == [] + assert fired_at(CronTrigger("30 1 * * *", "America/New_York"), start, 240) == ["06:30"] + assert fired_at(CronTrigger("30 3 * * *", "America/New_York"), start, 240) == ["07:30"] + + +def test_a_local_time_that_occurs_twice_fires_once() -> None: + zone("America/New_York") + # 2026-11-01: 01:30 happens at 05:30 UTC (EDT) and again at 06:30 UTC (EST). + start = datetime(2026, 11, 1, 4, 0, tzinfo=UTC) + assert fired_at(CronTrigger("30 1 * * *", "America/New_York"), start, 240) == ["05:30"] + assert fired_at(CronTrigger("30 0-2 * * *", "America/New_York"), start, 240) == [ + "04:30", + "05:30", + "07:30", + ] + + +def test_a_job_that_runs_every_hour_keeps_firing_through_the_repeated_hour() -> None: + zone("America/New_York") + start = datetime(2026, 11, 1, 4, 0, tzinfo=UTC) + assert fired_at(CronTrigger("30 * * * *", "America/New_York"), start, 240) == [ + "04:30", + "05:30", + "06:30", + "07:30", + ] + every_quarter = fired_at(CronTrigger("*/15 * * * *", "America/New_York"), start, 240) + assert len(every_quarter) == 16 + + +def test_the_scheduler_fires_a_zoned_job_at_its_local_minute(rig: Rig) -> None: + zone("Asia/Taipei") + calls: list[int] = [] + rig.engine.add( + "nightly", + "0 2 * * *", + [[rig.command("count", lambda: calls.append(1))]], + timezone="Asia/Taipei", + ) + assert rig.engine.tick(datetime(2026, 10, 7, 17, 59, tzinfo=UTC)) == [] + (run,) = rig.engine.tick(datetime(2026, 10, 7, 18, 0, 20, tzinfo=UTC)) + assert run.wait(WAIT) + assert (run.trigger, run.state) == (TriggerKind.CRON, RunState.COMPLETED) + assert run.scheduled_at == datetime(2026, 10, 7, 18, 0, tzinfo=UTC) + assert run.detail == {"cron": "0 2 * * *", "timezone": "Asia/Taipei"} + snapshot = rig.job("nightly") + assert snapshot["last_run"] == "2026-10-08T02:00:00+08:00" + assert (snapshot["cron"], snapshot["timezone"]) == ("0 2 * * *", "Asia/Taipei") + assert calls == [1] + + +def test_the_scheduler_refuses_a_job_in_an_unknown_zone(rig: Rig) -> None: + with pytest.raises(CronException): + rig.engine.add("job", "0 2 * * *", [["FA_schedule_list"]], timezone="Not/AZone") + assert rig.engine.list() == [] + + +# ---------------------------------------------------------------------- the tick + + +def test_a_minute_is_fired_once_however_often_it_is_ticked(rig: Rig) -> None: + calls: list[int] = [] + rig.engine.add( + "job", "* * * * *", [[rig.command("count", lambda: calls.append(1))]], timezone="UTC" + ) + (first,) = rig.tick() + assert first.wait(WAIT) + assert rig.tick(seconds=1) == [] + assert rig.tick(seconds=58) == [] + (second,) = rig.tick(seconds=1) + assert second.wait(WAIT) + assert second.scheduled_at == START + timedelta(minutes=1) + assert calls == [1, 1] + + +def test_a_job_without_a_zone_keeps_a_naive_local_last_run(rig: Rig) -> None: + rig.engine.add("job", "* * * * *", [[rig.command("ok", lambda: 1)]]) + (run,) = rig.tick(seconds=30) + assert run.wait(WAIT) + expected = START.astimezone().replace(tzinfo=None) + assert rig.job("job")["last_run"] == expected.isoformat() + assert run.detail == {"cron": "* * * * *", "timezone": None} + + +def test_tick_takes_a_naive_time_as_local_time(rig: Rig) -> None: + rig.engine.add("job", "30 9 * * *", [[rig.command("ok", lambda: 1)]]) + assert rig.engine.tick(datetime(2026, 4, 21, 9, 29)) == [] + (run,) = rig.engine.tick(datetime(2026, 4, 21, 9, 30)) + assert run.wait(WAIT) + assert run.scheduled_at == datetime(2026, 4, 21, 9, 30).astimezone(UTC) + + +def test_two_cron_triggers_due_in_one_minute_fire_the_job_once(rig: Rig) -> None: + rig.engine.add_job( + "job", + [[rig.command("ok", lambda: 1)]], + triggers=[CronTrigger("* * * * *", "UTC"), CronTrigger("0 2 * * *", "UTC")], + ) + assert len(rig.tick()) == 1 + + +# ---------------------------------------------------------------------- events + + +def test_an_event_trigger_fires_on_a_type_name(rig: Rig) -> None: + rig.engine.add_job( + "job", [[rig.command("ok", lambda: 1)]], triggers=EventTrigger(types="deploy.finished") + ) + event = _Deploy(source="webhook", subject="release 1.4") + rig.bus.publish(event) + (run,) = rig.runs("job") + assert run.wait(WAIT) + assert (run.trigger, run.state) == (TriggerKind.EVENT, RunState.COMPLETED) + assert run.detail == { + "event_type": "deploy.finished", + "event_id": event.id, + "source": "webhook", + "subject": "release 1.4", + "correlation_id": event.correlation_id, + } + assert run.correlation_id != event.correlation_id + + +@pytest.mark.parametrize( + ("trigger", "fires"), + [ + (EventTrigger(types="deploy.*"), True), + (EventTrigger(types=_Deploy), True), + (EventTrigger(types=["task.failed", "deploy.finished"]), True), + (EventTrigger(types="task.failed"), False), + (EventTrigger(sources="webhook"), True), + (EventTrigger(sources=["cli", "mcp"]), False), + (EventTrigger(types="deploy.finished", sources="webhook"), True), + (EventTrigger(types="deploy.finished", sources="cli"), False), + (EventTrigger(types="deploy.finished", min_severity="warning"), True), + (EventTrigger(types="deploy.finished", min_severity=Severity.ERROR), False), + ], +) +def test_an_event_trigger_matches_by_type_source_and_severity( + rig: Rig, trigger: EventTrigger, fires: bool +) -> None: + rig.engine.add_job("job", [[rig.command("ok", lambda: 1)]], triggers=trigger) + rig.bus.publish(_Deploy(source="webhook", severity=Severity.WARNING)) + assert len(rig.runs("job")) == int(fires) + assert all(run.wait(WAIT) for run in rig.runs("job")) + + +def test_an_event_trigger_stops_with_its_job(rig: Rig) -> None: + rig.engine.add_job("job", [["FA_schedule_list"]], triggers=EventTrigger(types="deploy.*")) + rig.engine.remove("job") + assert rig.bus.publish(_Deploy()) == 1 # only the rig's own collector is left + assert rig.runs("job") == [] + + +def test_a_job_does_not_fire_on_the_events_of_its_own_run(rig: Rig) -> None: + def boom() -> None: + raise ValueError("no") + + rig.engine.add_job( + "watcher", + [[rig.command("boom", boom)]], + triggers=EventTrigger(types="scheduler.error"), + allow_overlap=True, + ) + rig.engine.add_job("other", [[rig.command("boom2", boom)]]) + first = rig.engine.run_now("watcher") + assert first.wait(WAIT) + assert rig.idle("watcher") + assert [run.trigger.value for run in rig.runs("watcher")] == ["manual"] + # Another job's failure does fire it, and its own failure then stops there. + assert rig.engine.run_now("other").wait(WAIT) + assert wait_until(lambda: len(rig.runs("watcher")) == 2) + assert all(run.wait(WAIT) for run in rig.runs("watcher")) + assert rig.idle("watcher") + assert [run.trigger.value for run in rig.runs("watcher")] == ["event", "manual"] + assert len(rig.errors()) == 3 + + +def test_a_job_does_not_fire_on_its_own_timeout(rig: Rig) -> None: + action, gate = rig.gate() + rig.engine.add_job( + "watcher", + [[action]], + triggers=EventTrigger(types="scheduler.error"), + allow_overlap=True, + timeout=30, + ) + run = rig.engine.run_now("watcher") + gate.await_entry() + rig.tick(seconds=30) + assert run.state is RunState.TIMEOUT + assert len(rig.errors()) == 1 + assert rig.runs("watcher") == [run] + + +def test_two_jobs_that_fire_each_other_stop_after_a_chain_of_sixteen_runs(rig: Rig) -> None: + def boom() -> None: + raise ValueError("no") + + action = rig.command("boom", boom) + for name in ("ping", "pong"): + rig.engine.add_job( + name, [[action]], triggers=EventTrigger(types="scheduler.error"), allow_overlap=True + ) + rig.engine.run_now("ping") + assert wait_until(lambda: bool(rig.engine.history(state="skipped"))) + assert rig.idle("ping") and rig.idle("pong") + records = rig.engine.history(limit=100) + assert [run.state.value for run in records] == ["skipped"] + ["failed"] * 16 + assert [run.job for run in reversed(records)] == ["ping", "pong"] * 8 + ["ping"] + assert (records[0].reason, records[0].trigger) == ("chain", TriggerKind.EVENT) + assert len(rig.errors()) == 16 + assert rig.job("ping")["skipped"] == 1 + # A firing no run caused starts a chain of its own. + rig.engine.remove("pong") + assert rig.engine.run_now("ping").wait(WAIT) + assert len(rig.engine.history(limit=100)) == 18 + + +def test_an_action_that_publishes_an_event_does_not_fire_its_own_job(rig: Rig) -> None: + rig.engine.add_job( + "job", + [[rig.command("emit", lambda: rig.bus.publish(_Deploy(source="job")))]], + triggers=EventTrigger(types="deploy.finished"), + allow_overlap=True, + ) + run = rig.engine.run_now("job") + assert run.wait(WAIT) + assert rig.idle("job") + assert len(rig.runs("job")) == 1 + + +@pytest.mark.parametrize( + "arguments", + [ + {}, + {"types": ()}, + {"types": [""]}, + {"types": [5]}, + {"types": int}, + {"sources": [" "]}, + {"types": "task.failed", "min_severity": "fatal"}, + ], +) +def test_an_event_trigger_refuses_what_it_cannot_match(arguments: dict[str, Any]) -> None: + with pytest.raises(SchedulerException, match="event trigger"): + EventTrigger(**arguments) + + +def test_an_event_trigger_as_a_mapping() -> None: + trigger = EventTrigger( + types=[PipelineFailed, "task.*"], sources="pipeline", min_severity="error" + ) + assert trigger.to_dict() == { + "kind": "event", + "types": ["pipeline.failed", "task.*"], + "sources": ["pipeline"], + "min_severity": "error", + } + assert trigger.kind is TriggerKind.EVENT + assert trigger_from_dict(trigger.to_dict()) == EventTrigger( + types=("pipeline.failed", "task.*"), sources=("pipeline",), min_severity=Severity.ERROR + ) + + +# ---------------------------------------------------------------------- files + + +class _Observer: + """A stand-in for watchdog's observer: it records what it was told and fires nothing.""" + + def __init__(self, made: list[_Observer]) -> None: + self.handler: FileSystemEventHandler | None = None + self.path = "" + self.recursive = False + self.alive = False + self.daemon = False + made.append(self) + + def schedule(self, handler: FileSystemEventHandler, path: str, recursive: bool) -> None: + self.handler, self.path, self.recursive = handler, path, recursive + + def start(self) -> None: + self.alive = True + + def stop(self) -> None: + self.alive = False + + def join(self, timeout: float | None = None) -> None: + self.joined_with = timeout + + def is_alive(self) -> bool: + return self.alive + + def emit(self, event: Any) -> None: + assert self.handler is not None + self.handler.on_any_event(event) + + +@pytest.fixture +def observers(monkeypatch: pytest.MonkeyPatch) -> list[_Observer]: + """Every observer the watchers of this test made, in order.""" + made: list[_Observer] = [] + monkeypatch.setattr(watch_manager, "Observer", lambda: _Observer(made)) + return made + + +def test_a_file_trigger_fires_its_job_for_a_matching_event( + rig: Rig, observers: list[_Observer], tmp_path: Path +) -> None: + calls: list[int] = [] + snapshot = rig.engine.add_job( + "inbox", + [[rig.command("count", lambda: calls.append(1))]], + triggers=FileTrigger(str(tmp_path), events=["created"], recursive=False), + ) + assert snapshot["triggers"] == [ + {"kind": "file", "path": str(tmp_path), "events": ["created"], "recursive": False} + ] + (observer,) = observers + assert (observer.alive, observer.recursive) == (True, False) + assert Path(observer.path) == tmp_path.resolve() + target = str(tmp_path / "report.csv") + observer.emit(FileModifiedEvent(target)) + assert rig.runs("inbox") == [] + observer.emit(FileCreatedEvent(target)) + (run,) = rig.runs("inbox") + assert run.wait(WAIT) + assert (run.trigger, run.state) == (TriggerKind.FILE, RunState.COMPLETED) + assert run.detail == {"path": target, "event": "created"} + assert calls == [1] + + +def test_a_file_trigger_stops_watching_when_its_job_is_removed( + rig: Rig, observers: list[_Observer], tmp_path: Path +) -> None: + rig.engine.add_job("a", [["FA_schedule_list"]], triggers=FileTrigger(str(tmp_path))) + rig.engine.add_job("b", [["FA_schedule_list"]], triggers=FileTrigger(str(tmp_path))) + rig.engine.remove("a") + assert [observer.alive for observer in observers] == [False, True] + rig.engine.shutdown() + assert [observer.alive for observer in observers] == [False, False] + assert [job["name"] for job in rig.engine.list()] == ["b"] + + +def test_a_file_trigger_on_a_missing_path_registers_nothing(rig: Rig, tmp_path: Path) -> None: + with pytest.raises(TriggerException, match="watch path does not exist"): + rig.engine.add_job( + "job", [["FA_schedule_list"]], triggers=FileTrigger(str(tmp_path / "nope")) + ) + assert "job" not in rig.engine + assert rig.engine.list() == [] + + +def test_a_job_whose_second_trigger_fails_leaves_nothing_armed( + rig: Rig, observers: list[_Observer], tmp_path: Path +) -> None: + with pytest.raises(TriggerException, match="unsupported event types"): + rig.engine.add_job( + "job", + [["FA_schedule_list"]], + triggers=[ + EventTrigger(types="deploy.finished"), + FileTrigger(str(tmp_path)), + FileTrigger(str(tmp_path), events=["exploded"]), + ], + ) + assert "job" not in rig.engine + assert [observer.alive for observer in observers] == [False] + assert rig.bus.publish(_Deploy()) == 1 + rig.engine.add_job("job", [["FA_schedule_list"]]) + + +def test_a_file_trigger_with_a_real_observer_starts_and_stops(rig: Rig, tmp_path: Path) -> None: + rig.engine.add_job("job", [["FA_schedule_list"]], triggers=FileTrigger(tmp_path)) + assert rig.job("job")["triggers"][0]["events"] == ["created", "modified"] + rig.engine.remove("job") + + +def test_a_file_trigger_normalises_its_arguments(tmp_path: Path) -> None: + trigger = FileTrigger(tmp_path, events="deleted", recursive=0) + assert (trigger.path, trigger.events, trigger.recursive) == (str(tmp_path), ("deleted",), False) + assert FileTrigger(str(tmp_path), events=()).events == ("created", "modified") + assert trigger.kind is TriggerKind.FILE + with pytest.raises(SchedulerException, match="file trigger"): + FileTrigger("") + with pytest.raises(SchedulerException, match="file trigger"): + FileTrigger(None) + + +def test_a_watcher_with_a_callback_runs_no_action_list( + observers: list[_Observer], tmp_path: Path +) -> None: + seen: list[tuple[str, str]] = [] + watcher = FileWatcher( + "unit", str(tmp_path), [["FA_does_not_exist"]], on_event=lambda *call: seen.append(call) + ) + watcher.start() + (observer,) = observers + observer.emit(FileCreatedEvent(str(tmp_path / "a.txt"))) + observer.emit(FileCreatedEvent(bytes(tmp_path / "b.txt"))) + watcher.stop() + assert seen == [("created", str(tmp_path / "a.txt")), ("created", str(tmp_path / "b.txt"))] + + +def test_a_failing_watcher_callback_does_not_reach_the_observer( + observers: list[_Observer], tmp_path: Path +) -> None: + def refuse(_kind: str, _path: str) -> None: + raise SchedulerException("no such job") + + watcher = FileWatcher("unit", str(tmp_path), [], on_event=refuse) + watcher.start() + observers[0].emit(FileCreatedEvent(str(tmp_path / "a.txt"))) + watcher.stop() + + +# ---------------------------------------------------------------------- mappings + + +def test_every_trigger_turns_into_a_mapping_and_back(tmp_path: Path) -> None: + triggers: list[Trigger] = [ + CronTrigger("0 2 * * *", "UTC"), + CronTrigger("*/5 * * * *"), + FileTrigger(str(tmp_path), events=("created", "moved"), recursive=False), + EventTrigger(types=("task.failed",), sources=("pipeline",), min_severity=Severity.ERROR), + PipelineTrigger("daily-report", "always"), + ] + for trigger in triggers: + assert trigger_from_dict(trigger.to_dict()) == trigger + assert as_triggers([trigger.to_dict() for trigger in triggers]) == tuple(triggers) + + +def test_as_triggers_takes_one_or_many() -> None: + cron_trigger = CronTrigger("0 2 * * *") + assert as_triggers(None) == () + assert as_triggers(cron_trigger) == (cron_trigger,) + assert as_triggers({"kind": "cron", "cron": "0 2 * * *"}) == (cron_trigger,) + assert as_triggers((cron_trigger, {"kind": "pipeline", "pipeline": "daily"})) == ( + cron_trigger, + PipelineTrigger("daily"), + ) + for wrong in ("0 2 * * *", 5): + with pytest.raises(SchedulerException, match="triggers"): + as_triggers(wrong) + + +@pytest.mark.parametrize( + ("spec", "message"), + [ + ("cron", "expected a mapping"), + ({}, "unknown kind None"), + ({"kind": "hourly"}, "unknown kind 'hourly'"), + ({"kind": "manual"}, "'manual' needs no trigger"), + ({"kind": "cron"}, "'cron' is required"), + ({"kind": "file"}, "'path' is required"), + ({"kind": "pipeline"}, "'pipeline' is required"), + ({"kind": "cron", "cron": "0 2 * * *", "tz": "UTC"}, "unknown key tz"), + ({"kind": "event"}, "at least one type or one source"), + ({"kind": "pipeline", "pipeline": "daily", "when": "sometimes"}, "when is one of"), + ({"kind": "pipeline", "pipeline": " "}, "expected a pipeline name"), + ], +) +def test_a_trigger_mapping_that_is_wrong_is_refused(spec: Any, message: str) -> None: + with pytest.raises(SchedulerException, match=message): + trigger_from_dict(spec) + + +def test_a_bad_cron_in_a_mapping_is_a_cron_exception() -> None: + with pytest.raises(CronException): + trigger_from_dict({"kind": "cron", "cron": "61 * * * *"}) From 87d8908ac4bc8852fa4e3e6749ad87044bfa274e Mon Sep 17 00:00:00 2001 From: JeffreyChen Date: Thu, 8 Oct 2026 15:33:18 +0800 Subject: [PATCH 47/59] feat: add the application layer and UI 2.0 One plain-Python service per navigation entry, a sidebar GUI built on them with a pipeline editor, and a web UI that renders the same services. The older tabs stay under Advanced. --- CLAUDE.md | 11 +- README.md | 83 +- README.zh-CN.md | 75 +- README.zh-TW.md | 76 +- architecture.md | 4 + automation_file/__init__.py | 5 + automation_file/app/__init__.py | 102 ++ automation_file/app/arguments.py | 159 +++ automation_file/app/audit_service.py | 107 ++ automation_file/app/dashboard_service.py | 224 +++++ automation_file/app/errors.py | 15 + automation_file/app/file_service.py | 252 +++++ automation_file/app/integrity_service.py | 173 ++++ automation_file/app/masking.py | 98 ++ automation_file/app/notification_service.py | 132 +++ automation_file/app/pipeline_draft.py | 667 ++++++++++++ automation_file/app/pipeline_service.py | 456 +++++++++ automation_file/app/scheduler_service.py | 67 ++ automation_file/app/services.py | 136 +++ automation_file/app/settings_service.py | 182 ++++ automation_file/app/storage_service.py | 346 +++++++ automation_file/server/web_ui.py | 358 +++++-- automation_file/ui/__init__.py | 10 +- automation_file/ui/launcher.py | 13 +- automation_file/ui/main_window.py | 185 +++- automation_file/ui/pages/__init__.py | 47 + automation_file/ui/pages/advanced_page.py | 89 ++ automation_file/ui/pages/audit_page.py | 218 ++++ automation_file/ui/pages/base.py | 244 +++++ automation_file/ui/pages/dashboard_page.py | 235 +++++ automation_file/ui/pages/files_page.py | 277 +++++ automation_file/ui/pages/integrity_page.py | 248 +++++ .../ui/pages/notifications_page.py | 240 +++++ automation_file/ui/pages/pipeline_canvas.py | 395 ++++++++ automation_file/ui/pages/pipelines_page.py | 646 ++++++++++++ automation_file/ui/pages/run_panel.py | 211 ++++ automation_file/ui/pages/scheduler_page.py | 127 +++ automation_file/ui/pages/settings_page.py | 178 ++++ automation_file/ui/pages/storage_page.py | 167 +++ automation_file/ui/pages/task_form.py | 505 ++++++++++ docs/source/API/api_index.rst | 14 + docs/source/API/app.rst | 91 ++ docs/source/API/ui.rst | 69 ++ docs/source/Eng/architecture.rst | 24 +- docs/source/Eng/eng_index.rst | 6 +- docs/source/Eng/usage/app_layer.rst | 501 +++++++++ docs/source/Eng/usage/gui.rst | 498 ++++++++- docs/source/Eng/usage/pipeline.rst | 5 + docs/source/Eng/usage/servers.rst | 75 ++ docs/source/Zh-CN/architecture.rst | 24 +- docs/source/Zh-CN/usage/app_layer.rst | 475 +++++++++ docs/source/Zh-CN/usage/gui.rst | 444 +++++++- docs/source/Zh-CN/usage/pipeline.rst | 4 + docs/source/Zh-CN/usage/servers.rst | 70 ++ docs/source/Zh-CN/zh_cn_index.rst | 1 + docs/source/Zh-TW/architecture.rst | 24 +- docs/source/Zh-TW/usage/app_layer.rst | 475 +++++++++ docs/source/Zh-TW/usage/gui.rst | 444 +++++++- docs/source/Zh-TW/usage/pipeline.rst | 4 + docs/source/Zh-TW/usage/servers.rst | 70 ++ docs/source/Zh-TW/zh_tw_index.rst | 1 + docs/updates/2026-10.md | 17 + docs/updates/README.md | 3 +- progress.md | 4 +- tests/test_app_files.py | 361 +++++++ tests/test_app_masking.py | 185 ++++ tests/test_app_pipeline_draft.py | 472 +++++++++ tests/test_app_pipelines.py | 492 +++++++++ tests/test_app_services.py | 787 +++++++++++++++ tests/test_integrity_watch.py | 6 +- tests/test_ui_pages.py | 788 +++++++++++++++ tests/test_ui_pipeline_editor.py | 950 ++++++++++++++++++ tests/test_ui_smoke.py | 210 +++- tests/test_web_ui_app_layer.py | 260 +++++ tests/ui_stand_in.py | 39 + 75 files changed, 15442 insertions(+), 214 deletions(-) create mode 100644 automation_file/app/__init__.py create mode 100644 automation_file/app/arguments.py create mode 100644 automation_file/app/audit_service.py create mode 100644 automation_file/app/dashboard_service.py create mode 100644 automation_file/app/errors.py create mode 100644 automation_file/app/file_service.py create mode 100644 automation_file/app/integrity_service.py create mode 100644 automation_file/app/masking.py create mode 100644 automation_file/app/notification_service.py create mode 100644 automation_file/app/pipeline_draft.py create mode 100644 automation_file/app/pipeline_service.py create mode 100644 automation_file/app/scheduler_service.py create mode 100644 automation_file/app/services.py create mode 100644 automation_file/app/settings_service.py create mode 100644 automation_file/app/storage_service.py create mode 100644 automation_file/ui/pages/__init__.py create mode 100644 automation_file/ui/pages/advanced_page.py create mode 100644 automation_file/ui/pages/audit_page.py create mode 100644 automation_file/ui/pages/base.py create mode 100644 automation_file/ui/pages/dashboard_page.py create mode 100644 automation_file/ui/pages/files_page.py create mode 100644 automation_file/ui/pages/integrity_page.py create mode 100644 automation_file/ui/pages/notifications_page.py create mode 100644 automation_file/ui/pages/pipeline_canvas.py create mode 100644 automation_file/ui/pages/pipelines_page.py create mode 100644 automation_file/ui/pages/run_panel.py create mode 100644 automation_file/ui/pages/scheduler_page.py create mode 100644 automation_file/ui/pages/settings_page.py create mode 100644 automation_file/ui/pages/storage_page.py create mode 100644 automation_file/ui/pages/task_form.py create mode 100644 docs/source/API/app.rst create mode 100644 docs/source/Eng/usage/app_layer.rst create mode 100644 docs/source/Zh-CN/usage/app_layer.rst create mode 100644 docs/source/Zh-TW/usage/app_layer.rst create mode 100644 tests/test_app_files.py create mode 100644 tests/test_app_masking.py create mode 100644 tests/test_app_pipeline_draft.py create mode 100644 tests/test_app_pipelines.py create mode 100644 tests/test_app_services.py create mode 100644 tests/test_ui_pages.py create mode 100644 tests/test_ui_pipeline_editor.py create mode 100644 tests/test_web_ui_app_layer.py create mode 100644 tests/ui_stand_in.py diff --git a/CLAUDE.md b/CLAUDE.md index 51602ae..406c4ed 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -47,6 +47,8 @@ automation_file/ ├── pipeline/ # Pipeline runtime: model (Task, RetryPolicy, PipelineRun, ...), graph, pipeline │ # (Pipeline), runner + worker, substitution, store (RunStore, MemoryRunStore, │ # SQLiteRunStore), definition (YAML/JSON + PIPELINE_SCHEMA), reporting, actions +├── app/ # Application layer (no Qt, no SDK at import): services (AppServices), one +│ # *_service per navigation entry, pipeline_draft (PipelineDraft), masking ├── audit/ # Audit schema v2: record (AuditRecord), store (AuditStore, AuditQuery, │ # MemoryAuditStore), sqlite_store (SQLiteAuditStore), trail (AuditTrail, │ # audit_trail, configure_audit), actions (FA_audit_*) @@ -56,9 +58,10 @@ automation_file/ │ # notify/router.py routes events to sinks (NotificationRouter); │ # each registers its own FA_* ops ├── project/ # ProjectBuilder, create_project_dir -├── ui/ # PySide6 GUI: launcher.launch_ui, main_window.MainWindow, worker.ActionWorker, -│ # log_widget.LogPanel, tabs/ (home, local, http, JSON editor, servers, scheduler, -│ # trigger, progress; the cloud backends are panels grouped under transfer_tab) +├── ui/ # PySide6 GUI on the application layer: launcher.launch_ui, main_window.MainWindow +│ # (sidebar), pages/ (one per navigation entry, pipeline_canvas, task_form, run_panel, +│ # advanced_page), worker.ActionWorker, log_widget.LogPanel, tabs/ (the older tabs, +│ # shown under Advanced) └── utils/ # file discovery, fast find, grep, duplicate finder, backup rotation ``` @@ -80,7 +83,7 @@ automation_file/ - `PackageLoader` — imports a package by name and registers its top-level functions / classes / builtins as `_`. - `GoogleDriveClient` — wraps OAuth2 credential loading; exposes `service` lazily. `later_init(token_path, credentials_path)` bootstraps; `require_service()` raises if not initialised. - `S3Client` / `AzureBlobClient` / `DropboxClient` / `SFTPClient` — singleton wrappers around the SDKs of their extras. Each exposes `later_init(...)` plus `close()` where relevant. Their ops are auto-registered by `build_default_registry()`; `register__ops(registry)` is still exported so callers can populate custom registries. -- `MainWindow` — PySide6 tabbed control surface (`ui/main_window.py`). Nine tabs — Local, HTTP, Google Drive, S3, Azure Blob, Dropbox, SFTP, JSON actions, Servers — share a `LogPanel` and dispatch work through `ActionWorker(QRunnable)` on the global `QThreadPool`. +- `MainWindow` — PySide6 window with a sidebar (`ui/main_window.py`): Dashboard, Files, Storage, Pipelines (a canvas editor over `PipelineDraft`), Scheduler, Integrity, Audit, Notifications, Settings, and Advanced, which holds the older tabs (Local, Transfer, Progress, JSON actions, Triggers, Servers). Each page talks only to its service in `automation_file.app`; long work runs through `ActionWorker(QRunnable)` on the global `QThreadPool`. A new screen starts as a service in `app/`, tested without Qt, and a page is a thin view of it. The Web UI (`server/web_ui.py`) renders the same services, read-only. - `launch_ui(argv=None)` — boots / reuses a `QApplication`, shows `MainWindow`, and returns the exec code. Exposed lazily on the facade via `__getattr__` so the Qt runtime isn't paid for by non-UI importers. - `TCPActionServer` — threaded TCP server that deserialises a JSON action list per connection. Defaults to loopback; optional `shared_secret` enforces `AUTH \n` prefix. - `HTTPActionServer` — `ThreadingHTTPServer` exposing `POST /actions` plus `GET /healthz`, `/readyz`, `/openapi.json` and `/progress`. Defaults to loopback; optional `shared_secret` enforces `Authorization: Bearer `. diff --git a/README.md b/README.md index 3f4e7c0..c419221 100644 --- a/README.md +++ b/README.md @@ -7,8 +7,8 @@ remote storage, file integrity monitoring, pipelines with retry and resume, sche event-driven notifications, an audit trail, and automation through JSON actions, embedded TCP / HTTP servers and MCP. The object API (`File`, `Storage`, `Pipeline`, `IntegrityMonitor`) and the `FA_*` JSON actions are two faces of the same operations, and everything public is -re-exported from the top-level `automation_file` facade. A desktop GUI and a read-only web UI -are included. +re-exported from the top-level `automation_file` facade. A desktop GUI organised by workflow +and a read-only web UI are built on one application layer. ```python from automation_file import File, IntegrityMonitor, Pipeline, Storage @@ -54,7 +54,7 @@ IntegrityMonitor("s3://reports/2026", baseline="reports.baseline.json").verify() - **SMB / CIFS backend** — `SMBClient` over `smbprotocol`'s high-level `smbclient` API; UNC-based, encrypted sessions by default - **fsspec bridge** — drive any `fsspec`-backed filesystem (memory, local, s3, gcs, abfs, …) through the action registry with `get_fs` / `fsspec_upload` / `fsspec_download` / `fsspec_list_dir` etc. - **HTTP server observability** — `GET /healthz` / `GET /readyz` probes, `GET /openapi.json` spec, and `GET /progress` WebSocket stream of live transfer snapshots -- **HTMX Web UI** — `start_web_ui()` serves a read-only dashboard (health, progress, registry) that polls HTML fragments; stdlib-only HTTP plus one CDN script with SRI +- **HTMX Web UI** — `start_web_ui()` serves a read-only dashboard (health, pipeline runs, integrity, events, storage, audit, progress, registry) rendered from the application layer; stdlib-only HTTP plus one CDN script with SRI - **MCP (Model Context Protocol) server** — `MCPServer` bridges the registry to any MCP host (Claude Desktop, MCP CLIs) over newline-delimited JSON-RPC 2.0 on stdio; every `FA_*` action becomes an MCP tool with an auto-generated input schema - **Universal storage layer** — `File` / `Storage` address local and remote storage with one URI syntax (`local:///…`, `s3://…`, `azure://…`, `gdrive://…`, `sftp://…`, …), one `StorageBackend` contract and one error hierarchy; twelve backends are built in (local, in-memory, S3, Azure Blob, Google Drive, Dropbox, OneDrive, SFTP, FTP / FTPS, WebDAV, SMB, fsspec), and an 88-case contract suite checks any backend - **Event bus** — one `Event` model with ten core events (`pipeline.*`, `task.*`, `integrity.violation`, `storage.error`, `scheduler.error`, `system.error`), severities, correlation IDs and actors; subscribe on `event_bus` by class, type or prefix @@ -62,7 +62,8 @@ IntegrityMonitor("s3://reports/2026", baseline="reports.baseline.json").verify() - **Audit trail** — `configure_audit(path)` records one row per event and per storage operation (actor, source, pipeline, task, action, resource, backend, status, duration, correlation ID), searchable with `audit_search` / `FA_audit_search` - **Pipelines** — `Pipeline` runs tasks (callables or `FA_*` actions) in dependency order, independent ones in parallel, with retry, timeout, cancellation, conditions, idempotency keys, checkpoint and resume, a dry run and an execution history; definitions in Python, YAML or JSON - **Semantic MCP tools** — fourteen tools with stable names (`file_read`, `file_copy`, `storage_list`, `pipeline_run`, `integrity_status`, `audit_search`, …) for AI hosts, confined to the roots you name, read-only until you allow writing, with a dry run for everything that changes something; the `FA_*` bridge stays available -- PySide6 GUI (`python -m automation_file ui`) with a tab per backend, the JSON-action runner, and dedicated tabs for Triggers, Scheduler, and live Progress +- PySide6 GUI (`python -m automation_file ui`) organised by workflow — Dashboard, Files, Storage, Pipelines (a visual editor), Scheduler, Integrity, Audit, Notifications, Settings — with the per-backend tools under Advanced +- **Application layer** — `automation_file.app` has one plain-Python service per navigation entry; both user interfaces call it, and so can yours - Rich CLI with one-shot subcommands plus legacy JSON-batch flags - Project scaffolding (`ProjectBuilder`) for executor-based automations @@ -131,7 +132,8 @@ flowchart TD end subgraph UI["ui (PySide6)"] - MainWin["MainWindow
Home · Local · HTTP · Drive · S3 · Azure · Dropbox
SFTP · OneDrive · Box · JSON · Triggers · Scheduler
Progress · Transfer · Servers"] + MainWin["MainWindow
Dashboard · Files · Storage · Pipelines · Scheduler
Integrity · Audit · Notifications · Settings · Advanced"] + AppLayer["automation_file.app
one service per navigation entry"] Worker["ActionWorker
QRunnable on QThreadPool"] end @@ -187,6 +189,9 @@ flowchart TD Plugins ==> Loader MainWin ==> Worker + Worker ==> AppLayer + WebUI ==> AppLayer + AppLayer ==> PublicAPI Worker ==> PublicAPI PublicAPI ==> Executor @@ -313,7 +318,7 @@ flowchart TD class Secrets,Config,ConfW,Crypto,Check,SafeP,ACL sec; class Trigger,Sched event; class TCP,HTTPS,MCP,MetSrv,WebUI server; - class MainWin,Worker ui; + class MainWin,Worker,AppLayer ui; class FileOps,Archives,DataOps,TextOps,Misc localOps; class UrlVal,Http,Drive,S3M,Azure,Dropbox,SFTP,FTP,OneD,Box,WebDAV,SMB,Fsspec,Cross remote; class NM,Sinks notify; @@ -1123,14 +1128,17 @@ curl http://127.0.0.1:9944/openapi.json # OpenAPI 3.0 spec ### HTMX Web UI A read-only observability dashboard built on stdlib HTTP + HTMX (loaded from -a pinned CDN URL with SRI). Loopback-only by default; optional shared secret: +a pinned CDN URL with SRI) and rendered from the application layer, so it shows +what the desktop window shows. Loopback-only by default; optional shared secret: ```python from automation_file import start_web_ui server = start_web_ui(host="127.0.0.1", port=9955, shared_secret="s3cr3t") -# Browse http://127.0.0.1:9955/ — health, progress, and registry fragments -# auto-poll every few seconds. Write operations stay on the action servers. +# Browse http://127.0.0.1:9955/ — health, pipeline runs, integrity, recent +# events, storage, audit, progress and registry fragments poll every few +# seconds. Everything is escaped and secrets are masked. Write operations stay +# on the action servers. ``` ### MCP (Model Context Protocol) server @@ -1258,6 +1266,7 @@ break the library. ### GUI ```bash +pip install "automation_file[gui]" python -m automation_file ui # or: python main_ui.py ``` @@ -1266,8 +1275,60 @@ from automation_file import launch_ui launch_ui() ``` -Tabs: Home, Local, Transfer, Progress, JSON actions, Triggers, Scheduler, -Servers. A persistent log panel at the bottom streams every result and error. +The window is organised by workflow. The sidebar has nine pages, each a view +over one service of the application layer: + +| Page | What you do there | +|---|---| +| **Dashboard** | Health, running and recent pipeline runs, integrity drift, recent events, storage status | +| **Files** | Browse a storage URI, preview a file, copy, move, delete, create a directory | +| **Storage** | See which backends can be used (and the `pip install` command when an extra is missing); mount a local directory | +| **Pipelines** | Visual editor: drag actions onto a canvas, connect tasks, edit parameters, validate, dry-run, test one task, run, resume, retry, follow the run | +| **Scheduler** | List, add and remove cron jobs | +| **Integrity** | Baseline, verify and accept a tree; start and stop monitors | +| **Audit** | Point the audit trail at a database; search and count records | +| **Notifications** | Registered sinks, routes, a test message | +| **Settings** | Preview and apply `automation_file.toml`; installed extras; environment | + +**Advanced** keeps the earlier tabs unchanged: Local, Transfer (one panel per +backend, where a cloud client gets its credentials), Progress, JSON actions, +Triggers and Servers. + +In the pipeline editor, two tasks are connected by selecting the upstream one, +`Ctrl`-clicking the dependent one and pressing **Connect**; the order of the +selection is the direction of the arrow. Node positions are saved next to the +definition (`.layout.json`), never in it. + +A log panel at the bottom records every action and its outcome, and no page +shows a token, a password or a webhook URL. Background work runs on +`QThreadPool` through `ActionWorker`, so the window stays responsive. + +### Application layer +`automation_file.app` is what a user interface calls: one plain-Python service +per navigation entry, with no GUI toolkit and no backend SDK imported. The +PySide6 window and the Web UI are both built on it, so they show the same state, +and a third interface needs nothing else. + +```python +from automation_file.app import app_services + +services = app_services() +services.dashboard.summary().status # "ok" or "attention" +services.files.list_dir("s3://reports/2026") +services.storage.backends() # usable? missing extra? install hint + +draft = services.pipelines.new_draft("nightly") +draft.add_task("FA_storage_copy", "download", + arguments={"source": "s3://in/a.csv", "target": "local:///tmp/a.csv"}) +services.pipelines.validate(draft) # [] or problems, each with its path +run = services.pipelines.start(draft) # background; returns at once +services.pipelines.status(run["run_id"])["status"] +``` + +Services return dataclasses, dictionaries and lists that JSON can hold, mask +secrets in them, and raise `FileAutomationException` subclasses. +`build_services(ServiceOptions(...))` builds a private set on another run +store, event bus or resolver. ### Scaffold an executor-based project ```python diff --git a/README.zh-CN.md b/README.zh-CN.md index 78770d6..9a152ad 100644 --- a/README.zh-CN.md +++ b/README.zh-CN.md @@ -6,7 +6,7 @@ FileAutomation 是通用的文件层与数据流水线运行环境：以同一 文件完整性监控、具备重试与续跑能力的流水线、调度、事件驱动的通知、审计轨迹，以及通过 JSON 动作、内嵌 TCP / HTTP 服务器与 MCP 进行的自动化。对象 API（`File`、`Storage`、`Pipeline`、 `IntegrityMonitor`）与 `FA_*` JSON 动作是同一组操作的两种面貌，所有公开名称均由顶层 -`automation_file` facade 统一导出。另附桌面 GUI 与只读的 Web UI。 +`automation_file` facade 统一导出。另附按工作流组织的桌面 GUI 与只读的 Web UI，两者建立在同一个应用层之上。 ```python from automation_file import File, IntegrityMonitor, Pipeline, Storage @@ -52,7 +52,7 @@ IntegrityMonitor("s3://reports/2026", baseline="reports.baseline.json").verify() - **SMB / CIFS 后端** — `SMBClient` 基于 `smbprotocol` 的高阶 `smbclient` API；采用 UNC 路径，默认启用加密会话 - **fsspec 桥接** — 通过 `get_fs` / `fsspec_upload` / `fsspec_download` / `fsspec_list_dir` 等函数，驱动任何 `fsspec` 支持的文件系统（memory、local、s3、gcs、abfs、…） - **HTTP 服务器观测端点** — `GET /healthz` / `GET /readyz` 探针、`GET /openapi.json` 规格，以及 `GET /progress`（通过 WebSocket 推送实时传输快照） -- **HTMX Web UI** — `start_web_ui()` 启动只读观测仪表板（health、progress、registry），通过 HTML 片段轮询；仅用标准库 HTTP，搭配一个带 SRI 的 CDN 脚本 +- **HTMX Web UI** — `start_web_ui()` 启动只读仪表板（health、流水线运行、完整性、事件、存储、审计、progress、registry），由应用层渲染；仅用标准库 HTTP，搭配一个带 SRI 的 CDN 脚本 - **MCP（Model Context Protocol）服务器** — `MCPServer` 通过 stdio 上的 JSON-RPC 2.0（换行分隔 JSON）将注册表桥接到任意 MCP 主机（Claude Desktop、MCP CLI）；每个 `FA_*` 动作都会自动生成输入 schema 并成为 MCP 工具 - **通用存储层** — `File` / `Storage` 以同一套 URI 语法（`local:///…`、`s3://…`、`azure://…`、`gdrive://…`、`sftp://…`、…）、同一份 `StorageBackend` 契约与同一组异常层级访问本地与远端存储；内置十二种后端（本地、内存、S3、Azure Blob、Google Drive、Dropbox、OneDrive、SFTP、FTP / FTPS、WebDAV、SMB、fsspec），并附带 88 个用例的契约测试套件可检查任何后端 - **事件总线** — 单一 `Event` 模型与十种核心事件（`pipeline.*`、`task.*`、`integrity.violation`、`storage.error`、`scheduler.error`、`system.error`），具备严重程度、关联 ID 与 actor；可以在 `event_bus` 上按类、type 或前缀订阅 @@ -60,7 +60,8 @@ IntegrityMonitor("s3://reports/2026", baseline="reports.baseline.json").verify() - **审计轨迹** — `configure_audit(path)` 为每个事件与每次存储操作记录一条（actor、来源、pipeline、task、动作、资源、后端、状态、耗时、关联 ID），可用 `audit_search` / `FA_audit_search` 查询 - **流水线（Pipeline）** — `Pipeline` 按依赖顺序执行任务（可调用对象或 `FA_*` 动作），互不依赖者并行执行，并支持重试、超时、取消、条件、幂等键、检查点与续跑、试运行以及执行历史；定义可以用 Python、YAML 或 JSON 编写 - **语义化 MCP 工具** — 提供给 AI 宿主的十四个名称稳定的工具（`file_read`、`file_copy`、`storage_list`、`pipeline_run`、`integrity_status`、`audit_search` 等），仅限于你指定的根位置，在你允许写入之前均为只读，所有会变更内容的工具都支持试运行；`FA_*` 桥接仍然保留 -- PySide6 GUI（`python -m automation_file ui`）每个后端一个页签，含 JSON 动作执行器，另有 Triggers、Scheduler、实时 Progress 专属页签 +- PySide6 GUI（`python -m automation_file ui`）按工作流组织——Dashboard、Files、Storage、Pipelines（可视化编辑器）、Scheduler、Integrity、Audit、Notifications、Settings——各后端专属的工具放在 Advanced 之下 +- **应用层** — `automation_file.app` 为导航中的每个条目提供一个普通的 Python 服务；两种用户界面都调用它，你的界面也可以 - 功能丰富的 CLI，包含一次性子命令与旧式 JSON 批量标志 - 项目脚手架（`ProjectBuilder`）协助构建以 executor 为核心的自动化项目 @@ -129,7 +130,8 @@ flowchart TD end subgraph UI["ui (PySide6)"] - MainWin["MainWindow
Home · Local · HTTP · Drive · S3 · Azure · Dropbox
SFTP · OneDrive · Box · JSON · Triggers · Scheduler
Progress · Transfer · Servers"] + MainWin["MainWindow
Dashboard · Files · Storage · Pipelines · Scheduler
Integrity · Audit · Notifications · Settings · Advanced"] + AppLayer["automation_file.app
one service per navigation entry"] Worker["ActionWorker
QRunnable on QThreadPool"] end @@ -185,6 +187,9 @@ flowchart TD Plugins ==> Loader MainWin ==> Worker + Worker ==> AppLayer + WebUI ==> AppLayer + AppLayer ==> PublicAPI Worker ==> PublicAPI PublicAPI ==> Executor @@ -311,7 +316,7 @@ flowchart TD class Secrets,Config,ConfW,Crypto,Check,SafeP,ACL sec; class Trigger,Sched event; class TCP,HTTPS,MCP,MetSrv,WebUI server; - class MainWin,Worker ui; + class MainWin,Worker,AppLayer ui; class FileOps,Archives,DataOps,TextOps,Misc localOps; class UrlVal,Http,Drive,S3M,Azure,Dropbox,SFTP,FTP,OneD,Box,WebDAV,SMB,Fsspec,Cross remote; class NM,Sinks notify; @@ -1093,15 +1098,16 @@ curl http://127.0.0.1:9944/openapi.json # OpenAPI 3.0 规格 ``` ### HTMX Web UI -基于标准库 HTTP + HTMX（以带 SRI 的固定 CDN URL 加载）构建的只读观测仪表板。 -默认仅允许 loopback，可选 shared-secret： +基于标准库 HTTP + HTMX（以带 SRI 的固定 CDN URL 加载）构建的只读观测仪表板，由应用层 +渲染，所以它显示的就是桌面窗口显示的内容。默认仅允许 loopback，可选 shared-secret： ```python from automation_file import start_web_ui server = start_web_ui(host="127.0.0.1", port=9955, shared_secret="s3cr3t") -# 浏览 http://127.0.0.1:9955/ —— health、progress、registry 片段每几秒 -# 自动轮询一次；写入操作仍然保留在动作服务器。 +# 浏览 http://127.0.0.1:9955/ —— health、流水线运行、完整性、最近的事件、存储、 +# 审计、progress、registry 片段每几秒自动轮询一次。所有内容都经过转义，机密信息 +# 都已屏蔽；写入操作仍然保留在动作服务器。 ``` ### MCP（Model Context Protocol）服务器 @@ -1221,6 +1227,7 @@ execute_action([["FA_greet", {"name": "world"}]]) ### GUI ```bash +pip install "automation_file[gui]" python -m automation_file ui # 或：python main_ui.py ``` @@ -1229,8 +1236,54 @@ from automation_file import launch_ui launch_ui() ``` -页签：Home、Local、Transfer、Progress、JSON actions、Triggers、Scheduler、 -Servers。底部常驻的 log 面板实时流式输出每一笔结果与错误。 +窗口按工作流组织。侧边栏有九个页面，每个都是应用层某一个服务的视图： + +| 页面 | 在这里做什么 | +|---|---| +| **Dashboard** | 健康状态、运行中与最近的流水线运行、完整性漂移、最近的事件、存储状态 | +| **Files** | 浏览存储 URI、预览文件、复制、移动、删除、创建目录 | +| **Storage** | 查看哪些后端可用（缺少 extra 时会显示 `pip install` 命令）；挂载本地目录 | +| **Pipelines** | 可视化编辑器：把动作拖到画布上、连接任务、编辑参数、验证、试运行、测试单个任务、运行、续跑、重试、跟踪运行 | +| **Scheduler** | 列出、添加与移除 cron 作业 | +| **Integrity** | 为目录树建立基线、验证与接受；启动与停止监控器 | +| **Audit** | 把审计轨迹指向数据库；搜索并统计记录 | +| **Notifications** | 已注册的 sink、路由、测试消息 | +| **Settings** | 预览并应用 `automation_file.toml`；已安装的 extra；运行环境 | + +**Advanced** 原封不动地保留旧的页签：Local、Transfer（每个后端一个面板，云端 client +的凭据在这里提供）、Progress、JSON actions、Triggers 与 Servers。 + +在流水线编辑器中，要连接两个任务，请先选中上游任务，再按住 `Ctrl` 点依赖它的任务， +然后按 **Connect**；选中的顺序就是箭头的方向。节点位置存在定义旁边 +（`.layout.json`），绝不会存进定义里。 + +底部的 log 面板记录每个动作与它的结果，而且没有任何页面会显示 token、密码或 webhook +URL。后台工作通过 `ActionWorker` 在 `QThreadPool` 上运行，窗口始终保持响应。 + +### 应用层 +`automation_file.app` 是用户界面所调用的那一层：导航中的每个条目各有一个普通的 +Python 服务，不导入任何 GUI 工具包，也不导入任何后端 SDK。PySide6 窗口与 Web UI 都 +建立在它之上，所以两者显示相同的状态，第三种界面也不需要其他东西。 + +```python +from automation_file.app import app_services + +services = app_services() +services.dashboard.summary().status # "ok" 或 "attention" +services.files.list_dir("s3://reports/2026") +services.storage.backends() # 可用吗？缺 extra？安装提示 + +draft = services.pipelines.new_draft("nightly") +draft.add_task("FA_storage_copy", "download", + arguments={"source": "s3://in/a.csv", "target": "local:///tmp/a.csv"}) +services.pipelines.validate(draft) # [] 或问题列表，每项都附路径 +run = services.pipelines.start(draft) # 后台运行；立即返回 +services.pipelines.status(run["run_id"])["status"] +``` + +服务返回 JSON 能容纳的 dataclass、字典与列表，并屏蔽其中的机密信息，抛出的则是 +`FileAutomationException` 的子类。`build_services(ServiceOptions(...))` 可以在另一个 +run store、事件总线或 resolver 上构建私有的一组服务。 ### 以 executor 为核心构建项目脚手架 ```python diff --git a/README.zh-TW.md b/README.zh-TW.md index da6c8de..c5b21be 100644 --- a/README.zh-TW.md +++ b/README.zh-TW.md @@ -6,7 +6,7 @@ FileAutomation 是通用的檔案層與資料管線執行環境：以同一套 A 檔案完整性監控、具備重試與續跑能力的管線、排程、事件驅動的通知、稽核軌跡，以及透過 JSON 動作、內建 TCP / HTTP 伺服器與 MCP 進行的自動化。物件 API（`File`、`Storage`、`Pipeline`、 `IntegrityMonitor`）與 `FA_*` JSON 動作是同一組操作的兩種面貌，所有公開名稱皆由頂層 -`automation_file` facade 統一匯出。另附桌面 GUI 與唯讀的 Web UI。 +`automation_file` facade 統一匯出。另附依工作流程安排的桌面 GUI 與唯讀的 Web UI，兩者建立在同一個應用層之上。 ```python from automation_file import File, IntegrityMonitor, Pipeline, Storage @@ -52,7 +52,7 @@ IntegrityMonitor("s3://reports/2026", baseline="reports.baseline.json").verify() - **SMB / CIFS 後端** — `SMBClient` 建構於 `smbprotocol` 的高階 `smbclient` API；採用 UNC 路徑，預設啟用加密連線 - **fsspec 橋接** — 透過 `get_fs` / `fsspec_upload` / `fsspec_download` / `fsspec_list_dir` 等函式，驅動任何 `fsspec` 支援的檔案系統（memory、local、s3、gcs、abfs、…） - **HTTP 伺服器觀測端點** — `GET /healthz` / `GET /readyz` 探針、`GET /openapi.json` 規格、以及 `GET /progress`（以 WebSocket 推送即時傳輸快照） -- **HTMX Web UI** — `start_web_ui()` 啟動唯讀觀測儀表板（health、progress、registry），以 HTML 片段輪詢；僅用標準函式庫 HTTP，搭配一支帶 SRI 的 CDN 腳本 +- **HTMX Web UI** — `start_web_ui()` 啟動唯讀儀表板（health、管線執行、完整性、事件、儲存、稽核、progress、registry），由應用層繪製；僅用標準函式庫 HTTP，搭配一支帶 SRI 的 CDN 腳本 - **MCP（Model Context Protocol）伺服器** — `MCPServer` 透過 stdio 上的 JSON-RPC 2.0（行分隔 JSON）將登錄表橋接到任何 MCP 主機（Claude Desktop、MCP CLI）；每個 `FA_*` 動作都會自動生成輸入 schema 並成為 MCP 工具 - **通用儲存層** — `File` / `Storage` 以同一套 URI 語法（`local:///…`、`s3://…`、`azure://…`、`gdrive://…`、`sftp://…`、…）、同一份 `StorageBackend` 契約與同一組例外階層存取本機與遠端儲存；內建十二種後端（本機、記憶體、S3、Azure Blob、Google Drive、Dropbox、OneDrive、SFTP、FTP / FTPS、WebDAV、SMB、fsspec），並附 88 個案例的契約測試套件可檢查任何後端 - **事件匯流排** — 單一 `Event` 模型與十種核心事件（`pipeline.*`、`task.*`、`integrity.violation`、`storage.error`、`scheduler.error`、`system.error`），具備嚴重程度、關聯 ID 與 actor；可在 `event_bus` 上依類別、type 或前綴訂閱 @@ -60,7 +60,8 @@ IntegrityMonitor("s3://reports/2026", baseline="reports.baseline.json").verify() - **稽核軌跡** — `configure_audit(path)` 為每個事件與每次儲存操作記錄一筆（actor、來源、pipeline、task、動作、資源、後端、狀態、耗時、關聯 ID），可用 `audit_search` / `FA_audit_search` 查詢 - **管線（Pipeline）** — `Pipeline` 依相依順序執行任務（可呼叫物件或 `FA_*` 動作），互不相依者平行執行，並支援重試、逾時、取消、條件、冪等鍵、檢查點與續跑、試跑以及執行歷史；定義可用 Python、YAML 或 JSON 撰寫 - **語意化 MCP 工具** — 提供給 AI 宿主的十四個名稱穩定的工具（`file_read`、`file_copy`、`storage_list`、`pipeline_run`、`integrity_status`、`audit_search` 等），僅限於你指定的根位置，在你允許寫入之前皆為唯讀，所有會變更內容的工具都支援試跑；`FA_*` 橋接仍然保留 -- PySide6 GUI（`python -m automation_file ui`）每個後端一個分頁，含 JSON 動作執行器，另有 Triggers、Scheduler、即時 Progress 專屬分頁 +- PySide6 GUI（`python -m automation_file ui`）依工作流程安排——Dashboard、Files、Storage、Pipelines（視覺化編輯器）、Scheduler、Integrity、Audit、Notifications、Settings——各後端專屬的工具放在 Advanced 底下 +- **應用層** — `automation_file.app` 為導覽中的每個項目提供一個普通的 Python 服務；兩種使用者介面都呼叫它，你的介面也可以 - 功能豐富的 CLI，包含一次性子指令與舊式 JSON 批次旗標 - 專案鷹架（`ProjectBuilder`）協助建立以 executor 為核心的自動化專案 @@ -129,7 +130,8 @@ flowchart TD end subgraph UI["ui (PySide6)"] - MainWin["MainWindow
Home · Local · HTTP · Drive · S3 · Azure · Dropbox
SFTP · OneDrive · Box · JSON · Triggers · Scheduler
Progress · Transfer · Servers"] + MainWin["MainWindow
Dashboard · Files · Storage · Pipelines · Scheduler
Integrity · Audit · Notifications · Settings · Advanced"] + AppLayer["automation_file.app
one service per navigation entry"] Worker["ActionWorker
QRunnable on QThreadPool"] end @@ -185,6 +187,9 @@ flowchart TD Plugins ==> Loader MainWin ==> Worker + Worker ==> AppLayer + WebUI ==> AppLayer + AppLayer ==> PublicAPI Worker ==> PublicAPI PublicAPI ==> Executor @@ -311,7 +316,7 @@ flowchart TD class Secrets,Config,ConfW,Crypto,Check,SafeP,ACL sec; class Trigger,Sched event; class TCP,HTTPS,MCP,MetSrv,WebUI server; - class MainWin,Worker ui; + class MainWin,Worker,AppLayer ui; class FileOps,Archives,DataOps,TextOps,Misc localOps; class UrlVal,Http,Drive,S3M,Azure,Dropbox,SFTP,FTP,OneD,Box,WebDAV,SMB,Fsspec,Cross remote; class NM,Sinks notify; @@ -1093,15 +1098,17 @@ curl http://127.0.0.1:9944/openapi.json # OpenAPI 3.0 規格 ``` ### HTMX Web UI -建構於標準函式庫 HTTP + HTMX（以帶 SRI 的固定 CDN URL 載入）之上的唯讀觀測 -儀表板。預設僅允許 loopback，可選 shared-secret： +建構於標準函式庫 HTTP + HTMX（以帶 SRI 的固定 CDN URL 載入）之上的唯讀觀測儀表板， +由應用層繪製，所以它顯示的就是桌面視窗顯示的內容。預設僅允許 loopback，可選 +shared-secret： ```python from automation_file import start_web_ui server = start_web_ui(host="127.0.0.1", port=9955, shared_secret="s3cr3t") -# 瀏覽 http://127.0.0.1:9955/ —— health、progress、registry 片段每數秒 -# 自動輪詢；寫入操作仍保留在動作伺服器。 +# 瀏覽 http://127.0.0.1:9955/ —— health、管線執行、完整性、最近的事件、儲存、 +# 稽核、progress、registry 片段每數秒自動輪詢。所有內容都經過跳脫，機敏資訊 +# 都已遮蔽；寫入操作仍保留在動作伺服器。 ``` ### MCP（Model Context Protocol）伺服器 @@ -1221,6 +1228,7 @@ execute_action([["FA_greet", {"name": "world"}]]) ### GUI ```bash +pip install "automation_file[gui]" python -m automation_file ui # 或：python main_ui.py ``` @@ -1229,8 +1237,54 @@ from automation_file import launch_ui launch_ui() ``` -分頁：Home、Local、Transfer、Progress、JSON actions、Triggers、Scheduler、 -Servers。底部常駐的 log 面板即時串流每一筆結果與錯誤。 +視窗依工作流程安排。側邊欄有九個頁面，每個都是應用層某一個服務的檢視： + +| 頁面 | 在這裡做什麼 | +|---|---| +| **Dashboard** | 健康狀態、執行中與最近的管線執行、完整性漂移、最近的事件、儲存狀態 | +| **Files** | 瀏覽儲存 URI、預覽檔案、複製、搬移、刪除、建立目錄 | +| **Storage** | 查看哪些後端可用（缺少 extra 時會顯示 `pip install` 指令）；掛載本機目錄 | +| **Pipelines** | 視覺化編輯器：把動作拖到畫布上、連接任務、編輯參數、驗證、試跑、測試單一任務、執行、續跑、重試、追蹤執行 | +| **Scheduler** | 列出、新增與移除 cron 工作 | +| **Integrity** | 為目錄樹建立基準、驗證與接受；啟動與停止監控器 | +| **Audit** | 把稽核軌跡指向資料庫；搜尋並計算紀錄 | +| **Notifications** | 已註冊的 sink、路由、測試訊息 | +| **Settings** | 預覽並套用 `automation_file.toml`；已安裝的 extra；執行環境 | + +**Advanced** 原封不動地保留舊的分頁：Local、Transfer（每個後端一個面板，雲端 client +的憑證在這裡提供）、Progress、JSON actions、Triggers 與 Servers。 + +在管線編輯器中，要連接兩個任務，請先選取上游任務，再按住 `Ctrl` 點相依於它的任務， +然後按 **Connect**；選取的順序就是箭頭的方向。節點位置存在定義旁邊 +（`.layout.json`），絕不會存進定義裡。 + +底部的 log 面板記錄每個動作與它的結果，而且沒有任何頁面會顯示 token、密碼或 webhook +URL。背景工作透過 `ActionWorker` 在 `QThreadPool` 上執行，視窗始終保持回應。 + +### 應用層 +`automation_file.app` 是使用者介面所呼叫的那一層：導覽中的每個項目各有一個普通的 +Python 服務，不匯入任何 GUI 工具組，也不匯入任何後端 SDK。PySide6 視窗與 Web UI 都 +建立在它之上，所以兩者顯示相同的狀態，第三種介面也不需要其他東西。 + +```python +from automation_file.app import app_services + +services = app_services() +services.dashboard.summary().status # "ok" 或 "attention" +services.files.list_dir("s3://reports/2026") +services.storage.backends() # 可用嗎？缺 extra？安裝提示 + +draft = services.pipelines.new_draft("nightly") +draft.add_task("FA_storage_copy", "download", + arguments={"source": "s3://in/a.csv", "target": "local:///tmp/a.csv"}) +services.pipelines.validate(draft) # [] 或問題清單，每項都附路徑 +run = services.pipelines.start(draft) # 背景執行；立即回傳 +services.pipelines.status(run["run_id"])["status"] +``` + +服務回傳 JSON 能容納的 dataclass、字典與清單，並遮蔽其中的機敏資訊，拋出的則是 +`FileAutomationException` 的子類別。`build_services(ServiceOptions(...))` 可以在另一 +個 run store、事件匯流排或 resolver 上建立私有的一組服務。 ### 以 executor 為核心建立專案鷹架 ```python diff --git a/architecture.md b/architecture.md index 8151a88..1787854 100644 --- a/architecture.md +++ b/architecture.md @@ -27,6 +27,7 @@ piece of the 1.0 roadmap (`docs/FILEAUTOMATION-1.0-ROADMAP.md`, PR #107); what i | `automation_file/storage/` | Universal storage layer. `uri.py` (`StorageURI`, `parse_storage_uri`, `normalize_path`), `types.py` (`FileInfo`, `Checksum`, `StorageCapabilities`), `backend.py` (`StorageBackend`: the public operations are template methods over the `_`-prefixed primitives a backend supplies), `local_storage.py` (`LocalStorage`, confined through `safe_join` when given a root), `memory_storage.py` (`MemoryStorage`), `object_storage.py` (`ObjectStorage`: directories as key prefixes over `_head`, `_scan`, `_put`, `_get`, `_remove`), `s3_storage.py` (`S3Storage`, over `s3_instance` or a given boto3 client), `azure_storage.py` (`AzureStorage`, over `azure_blob_instance` or a given `BlobServiceClient`), `session_storage.py` (`SessionStorage`: one login session, one operation at a time, and `require_session_host`), `sftp_storage.py` (`SFTPStorage`), `ftp_storage.py` (`FTPStorage`, for `ftp` and `ftps`), `gdrive_storage.py` (`GoogleDriveStorage`: paths resolved to file IDs, duplicate names refused), `onedrive_storage.py` (`OneDriveStorage`, Microsoft Graph), `dropbox_storage.py` (`DropboxStorage`), `webdav_storage.py` (`WebDAVStorage`), `smb_storage.py` (`SMBStorage`), `fsspec_storage.py` (`FsspecStorage`, any fsspec filesystem), `timestamps.py` (RFC 3339 parsing), `resolver.py` (`StorageResolver`, `default_resolver`: mounts first, then scheme factories), `file.py` (`File`), `storage.py` (`Storage`), `observe.py` (listeners for `upload`, `download`, `read`, `delete`, `mkdir`, `copy`, `move`), `streams.py` (staged file objects behind `open_read` / `open_write`), `tree.py` (`copy_tree`, `sync_tree`, `TreeResult`), `actions.py` (the `FA_storage_*` functions and `register_storage_ops`). At module level it imports only `exceptions`, `logging_config`, `core.checksum` and `local.safe_paths`: no registry, no GUI, no backend SDK. The adapters import their SDK's exceptions and the shared client inside the functions that use them | | `automation_file/events/` | The event model every component reports through. `model.py` (`Event`, `Severity`, the ten core events), `bus.py` (`EventBus`, the process-wide `event_bus`, `emit`), `context.py` (`correlation_scope`, `actor_scope`), `storage_bridge.py` (failed storage operations become `StorageError` events; installed when the package is imported). It imports only the standard library, `logging_config` and `storage.observe` | | `automation_file/integrity/` | IntegrityMonitor 2.0, on the storage layer and the event bus. `target.py` (`Target`: the monitored tree behind a storage URI), `hashing.py` (`HashEngine`; `md5` and `sha1` only with `allow_weak`), `snapshot.py` (`Snapshot`, `SnapshotEntry`, `build_snapshot`), `manifest.py` (schema version 2; the `write_manifest` format is read and converted), `baseline.py` (`BaselineManager`: an atomic write at any storage URI), `detector.py` (`Change`, `ChangeKind`, `detect_changes`: six kinds of change), `report.py` (`DriftReport`), `alerts.py` (`AlertEngine`, `AlertPolicy`: one `IntegrityViolation` per pass that finds drift), `remediation.py` (`RemediationPolicy`, `Remediator`: quarantine or restore, opt-in), `watcher.py` and `local_watcher.py` (polling, and watchdog events for a local target), `legacy.py` (the first monitor's summary, callback and notification), `monitor.py` (`IntegrityMonitor`), `actions.py` (`FA_integrity_*`). `core/fim.py` re-exports the class | +| `automation_file/app/` | The application layer both user interfaces call: plain Python, no Qt and no SDK at import. `services.py` (`AppServices`, `app_services`, `build_services`, `NAVIGATION`), one `*_service.py` per navigation entry (dashboard, files, storage, pipelines, scheduler, integrity, audit, notifications, settings), `pipeline_draft.py` (`PipelineDraft`: the editable definition behind the pipeline editor; canvas positions go to a `.layout.json` sidecar), `arguments.py`, `masking.py` (secrets are masked before anything is shown), `errors.py` (`AppException`) | | `automation_file/server/` | `tcp_server.py`, `http_server.py`, `mcp_server.py` (JSON-RPC over stdio: the semantic tools first, then the `FA_*` bridge), `mcp_policy.py` (`MCPPolicy`, `StorageGuard`: roots, read-only by default, limits), `mcp_tools.py` and `mcp_*_tools.py` (`SemanticToolkit` and the fourteen tools), `mcp_pipeline_actions.py` (the guarded `FA_storage_*` set pipelines made through MCP run), `web_ui.py`, `metrics_server.py`, `action_acl.py` (`ActionACL`), `network_guards.py` (`ensure_loopback`) | | `automation_file/client/` | `HTTPActionClient` for the HTTP action server | | `automation_file/trigger/`, `scheduler/`, `notify/` | Watchdog file triggers, the scheduler, notification sinks. Each registers its own `FA_*` ops. `scheduler/`: `cron.py` (`CronExpression`), `triggers.py` (`CronTrigger` with a time zone, `FileTrigger`, `EventTrigger`, `PipelineTrigger`), `job.py`, `targets.py` (an action list or a pipeline), `runs.py` (`JobRun`, `RunState`, a bounded `RunHistory`), `dispatch.py`, `manager.py` (`Scheduler`, the process-wide `scheduler`; `tick(now)` drives it in tests). `notify/router.py` (`Route`, `NotificationRouter`, the process-wide `notification_router`) subscribes on the event bus and delivers events to named sinks by type, source and minimum severity, with deduplication and a rate limit per route and sink; a failing sink becomes a `system.error` event from the source `notify`, which is never routed | @@ -94,6 +95,9 @@ piece of the 1.0 roadmap (`docs/FILEAUTOMATION-1.0-ROADMAP.md`, PR #107); what i `_remove_all`, `_list` (unchanged), and `FA_schedule_job`, `FA_schedule_pipeline`, `FA_schedule_run`, `FA_schedule_history`, `FA_schedule_cancel`. The keys of `ScheduledJob.as_dict()` are relied on by the GUI and by action lists. +- **Application layer** (`automation_file.app`; `AppServices`, `PipelineDraft`, `app_services` and + `build_services` are also on the facade): the services the GUI pages and the Web UI fragments call. + A third interface builds on these, not on the Qt widgets. - **Events** (same facade): `Event`, `Severity`, `EventBus`, `event_bus`, `emit`, `correlation_scope`, `actor_scope`, and the core events `PipelineStarted`, `PipelineCompleted`, `PipelineFailed`, `TaskStarted`, `TaskCompleted`, `TaskFailed`, `IntegrityViolation`, `StorageError`, `SchedulerError`, diff --git a/automation_file/__init__.py b/automation_file/__init__.py index 2a14ca8..476f368 100644 --- a/automation_file/__init__.py +++ b/automation_file/__init__.py @@ -9,6 +9,7 @@ from typing import TYPE_CHECKING, Any +from automation_file.app import AppServices, PipelineDraft, app_services, build_services from automation_file.audit import ( AuditQuery, AuditRecord, @@ -705,6 +706,10 @@ def __getattr__(name: str) -> Any: "start_web_ui", "MCPServer", "MCPPolicy", + "AppServices", + "PipelineDraft", + "app_services", + "build_services", "SemanticToolkit", "tools_from_registry", # Triggers diff --git a/automation_file/app/__init__.py b/automation_file/app/__init__.py new file mode 100644 index 0000000..59696af --- /dev/null +++ b/automation_file/app/__init__.py @@ -0,0 +1,102 @@ +"""The application layer: what a user interface calls. + +One service per navigation entry -- Dashboard, Files, Storage, Pipelines, +Scheduler, Integrity, Audit, Notifications, Settings -- each a plain Python +object on top of the domain packages. The PySide6 window and the web page both +call these services and nothing below them, so they show the same state and a +third interface needs no knowledge of the domain packages either. + +.. code-block:: python + + from automation_file.app import app_services + + services = app_services() + services.dashboard.summary().status + services.files.list_dir("memory://demo") + draft = services.pipelines.new_draft("nightly") + +The layer imports no GUI toolkit and no backend SDK, returns dataclasses, +dictionaries and lists that JSON can hold, masks secrets in what it returns, +and raises :class:`~automation_file.exceptions.FileAutomationException` +subclasses. +""" + +from __future__ import annotations + +from automation_file.app.arguments import ( + ActionInfo, + ActionParameter, + describe_action, + format_argument_value, + parse_argument_text, + parse_json_text, + split_names, +) +from automation_file.app.audit_service import AuditService +from automation_file.app.dashboard_service import ( + DashboardService, + DashboardSources, + DashboardSummary, + brief_run, +) +from automation_file.app.errors import AppException +from automation_file.app.file_service import FileEntry, FilePreview, FileService +from automation_file.app.integrity_service import IntegrityService, MonitorDrift +from automation_file.app.masking import MASK, mask_secrets, mask_text, mask_url +from automation_file.app.notification_service import NotificationService +from automation_file.app.pipeline_draft import DraftTask, PipelineDraft, Problem +from automation_file.app.pipeline_service import PipelineService, layout_path +from automation_file.app.scheduler_service import SchedulerService +from automation_file.app.services import ( + NAVIGATION, + AppServices, + ServiceOptions, + app_services, + build_services, + reset_app_services, +) +from automation_file.app.settings_service import ExtraStatus, SettingsService +from automation_file.app.storage_service import BackendStatus, MountInfo, StorageService + +__all__ = [ + "MASK", + "NAVIGATION", + "ActionInfo", + "ActionParameter", + "AppException", + "AppServices", + "AuditService", + "BackendStatus", + "DashboardService", + "DashboardSources", + "DashboardSummary", + "DraftTask", + "ExtraStatus", + "FileEntry", + "FilePreview", + "FileService", + "IntegrityService", + "MonitorDrift", + "MountInfo", + "NotificationService", + "PipelineDraft", + "PipelineService", + "Problem", + "SchedulerService", + "ServiceOptions", + "SettingsService", + "StorageService", + "app_services", + "brief_run", + "build_services", + "describe_action", + "format_argument_value", + "layout_path", + "mask_secrets", + "mask_text", + "mask_url", + "parse_argument_text", + "parse_json_text", + "reset_app_services", + "split_names", +] diff --git a/automation_file/app/arguments.py b/automation_file/app/arguments.py new file mode 100644 index 0000000..224289d --- /dev/null +++ b/automation_file/app/arguments.py @@ -0,0 +1,159 @@ +"""Turn what a form holds into action arguments, and describe an action for a form. + +A user interface edits arguments as text. :func:`parse_argument_text` reads one +field: JSON when the text is JSON (``12``, ``true``, ``["a", "b"]``, ``"12"``), +otherwise the text itself, so a URI or a ``${params.date}`` placeholder needs +no quotes. :func:`format_argument_value` is the way back. :func:`describe_action` +lists the parameters of a registered action so a form can offer one row each. +""" + +from __future__ import annotations + +import inspect +import json +import re +from collections.abc import Callable +from dataclasses import asdict, dataclass +from typing import Any + +from automation_file.app.errors import AppException + +_VARIADIC = (inspect.Parameter.VAR_POSITIONAL, inspect.Parameter.VAR_KEYWORD) +_NAME_SEPARATORS = re.compile(r"[,\s]+") + + +@dataclass(frozen=True) +class ActionParameter: + """One parameter of an action. ``default`` is its default as form text.""" + + name: str + required: bool + default: str = "" + + def to_dict(self) -> dict[str, Any]: + """Return a JSON-serialisable mapping of the parameter.""" + return asdict(self) + + +@dataclass(frozen=True) +class ActionInfo: + """What a form needs to know about an action. + + ``known`` is false for a name the registry does not have. ``accepts_extra`` + says the action takes ``**kwargs``, so arguments beyond ``parameters`` are + allowed. ``signature`` is unavailable (empty) for a builtin without one. + """ + + name: str + known: bool = False + signature: str = "" + summary: str = "" + parameters: tuple[ActionParameter, ...] = () + accepts_extra: bool = False + + def to_dict(self) -> dict[str, Any]: + """Return a JSON-serialisable mapping of the description.""" + return asdict(self) + + +def _no_constant(name: str) -> Any: + raise ValueError(f"{name} is not a JSON value") + + +def _loads(text: str) -> Any: + """Parse strict JSON: ``NaN`` and ``Infinity`` are text, not numbers.""" + return json.loads(text, parse_constant=_no_constant) + + +def parse_argument_text(text: str) -> Any: + """Return the value one form field stands for: JSON when it parses, else the text.""" + stripped = text.strip() + try: + return _loads(stripped) + except ValueError: + return stripped + + +def format_argument_value(value: Any) -> str: + """Return ``value`` as form text that :func:`parse_argument_text` reads back unchanged.""" + if isinstance(value, str) and value and value == value.strip(): + try: + _loads(value) + except ValueError: + return value + try: + return json.dumps(value, ensure_ascii=False) + except (TypeError, ValueError): + return repr(value) + + +def split_names(value: Any) -> Any: + """Return the names in a form field: ``"a, b"`` becomes ``["a", "b"]``; other values pass.""" + if isinstance(value, str): + return [name for name in _NAME_SEPARATORS.split(value.strip()) if name] + return value + + +def parse_json_text(text: str, what: str, *, empty: Any = None) -> Any: + """Return the JSON document in ``text``, or ``empty`` when the text is blank. + + Raises :class:`AppException` naming ``what`` and the position of the mistake. + """ + if not text.strip(): + return empty + try: + return _loads(text) + except json.JSONDecodeError as error: + raise AppException( + f"{what} is not valid JSON: {error.msg} at line {error.lineno}, column {error.colno}" + ) from error + except ValueError as error: + raise AppException(f"{what} is not valid JSON: {error}") from error + + +def _summary(command: Callable[..., Any]) -> str: + lines = (inspect.getdoc(command) or "").strip().splitlines() + return lines[0] if lines else "" + + +def _without_annotations(signature: inspect.Signature) -> str: + """Return ``(name, other=default)``: the parameters as a caller writes them.""" + bare = [ + parameter.replace(annotation=inspect.Parameter.empty) + for parameter in signature.parameters.values() + ] + return str(signature.replace(parameters=bare, return_annotation=inspect.Signature.empty)) + + +def describe_action(name: str, command: Callable[..., Any] | None) -> ActionInfo: + """Describe the registered action ``name``; ``command`` is ``None`` when it is unknown.""" + if command is None: + return ActionInfo(name=name) + try: + signature = inspect.signature(command) + except (TypeError, ValueError): + return ActionInfo(name=name, known=True, summary=_summary(command), accepts_extra=True) + parameters = tuple( + ActionParameter( + name=parameter.name, + required=parameter.default is inspect.Parameter.empty, + default=( + "" + if parameter.default is inspect.Parameter.empty + else format_argument_value(parameter.default) + ), + ) + for parameter in signature.parameters.values() + if parameter.kind not in _VARIADIC + ) + return ActionInfo( + name=name, + known=True, + signature=f"{name}{_without_annotations(signature)}", + summary=_summary(command), + parameters=parameters, + accepts_extra=any( + parameter.kind is inspect.Parameter.VAR_KEYWORD + for parameter in signature.parameters.values() + ), + ) diff --git a/automation_file/app/audit_service.py b/automation_file/app/audit_service.py new file mode 100644 index 0000000..d6e48e6 --- /dev/null +++ b/automation_file/app/audit_service.py @@ -0,0 +1,107 @@ +"""The Audit service: point the audit trail at a store and search it. + +.. code-block:: python + + from automation_file.app import app_services + + audit = app_services().audit + audit.configure("/var/lib/automation_file/audit.sqlite") + audit.search(status="error", resource_prefix="s3://reports/", limit=20) + audit.count(actor="scheduler") + +The trail records nothing until it has a store, so :meth:`AuditService.recent` +returns an empty list instead of raising while audit is not configured: a +dashboard can call it without asking first. Records come back as dictionaries +with their secrets masked. +""" + +from __future__ import annotations + +import os +from dataclasses import fields +from typing import Any + +from automation_file.app.masking import mask_secrets +from automation_file.audit import ( + DEFAULT_LIMIT, + AuditQuery, + AuditStore, + AuditTrail, + SQLiteAuditStore, + audit_trail, +) + +_RECENT_LIMIT = 50 + + +def _given(filters: dict[str, Any]) -> dict[str, Any]: + """Drop the filters a form left empty.""" + return { + name: value + for name, value in filters.items() + if value is not None and not (isinstance(value, str) and not value.strip()) + } + + +class AuditService: + """Configure, search and count on one :class:`~automation_file.audit.AuditTrail`.""" + + def __init__(self, trail: AuditTrail | None = None) -> None: + self._trail = audit_trail if trail is None else trail + + def filter_names(self) -> tuple[str, ...]: + """Return the names :meth:`search` and :meth:`count` accept as filters.""" + return tuple(entry.name for entry in fields(AuditQuery)) + + def is_configured(self) -> bool: + """Return whether the trail has a store to record into.""" + return self._trail.store is not None + + def status(self) -> dict[str, Any]: + """Return whether audit is configured and recording, and where the records go.""" + store = self._trail.store + described: dict[str, Any] = { + "configured": store is not None, + "active": self._trail.active, + "store": None if store is None else type(store).__name__, + "db_path": None, + "schema_version": None, + } + if isinstance(store, SQLiteAuditStore): + described["db_path"] = str(store.path) + described["schema_version"] = store.schema_version + return described + + def configure(self, target: AuditStore | str | os.PathLike[str]) -> dict[str, Any]: + """Record into ``target`` from now on and return :meth:`status`. + + ``target`` is the path of a SQLite database (created when missing) or a + ready :class:`~automation_file.audit.AuditStore`. + """ + if isinstance(target, AuditStore): + self._trail.attach(target) + else: + self._trail.attach(SQLiteAuditStore(target), owned=True) + self._trail.start() + return self.status() + + def search(self, **filters: Any) -> list[dict[str, Any]]: + """Return the records that pass ``filters``, newest first. + + Filters: ``since``, ``until``, ``actor``, ``source``, ``pipeline``, + ``task``, ``action``, ``resource_prefix``, ``backend``, ``status``, + ``correlation_id``, ``text``, ``limit``, ``offset``. A filter left empty + does not restrict the search. Raises ``AuditException`` when audit is + not configured or a filter is not known. + """ + return [mask_secrets(entry.to_dict()) for entry in self._trail.search(**_given(filters))] + + def count(self, **filters: Any) -> int: + """Return how many records pass ``filters`` (paging is ignored).""" + return self._trail.count(**_given(filters)) + + def recent(self, limit: int = _RECENT_LIMIT) -> list[dict[str, Any]]: + """Return the latest records, or nothing while audit is not configured.""" + if not self.is_configured(): + return [] + return self.search(limit=max(min(limit, DEFAULT_LIMIT), 1)) diff --git a/automation_file/app/dashboard_service.py b/automation_file/app/dashboard_service.py new file mode 100644 index 0000000..f632535 --- /dev/null +++ b/automation_file/app/dashboard_service.py @@ -0,0 +1,224 @@ +"""The Dashboard service: one summary of how the installation is doing. + +.. code-block:: python + + from automation_file.app import app_services + + summary = app_services().dashboard.summary() + summary.status # "ok" or "attention" + summary.reasons # why it needs attention + summary.running_runs, summary.recent_runs, summary.run_counts + summary.integrity # what every monitor last found + summary.events # the latest events of the bus, newest first + summary.storage # every backend and whether it can be used + +The summary is built from the other services and from the event bus; it reads +state and starts nothing. ``summary.to_dict()`` is JSON-serialisable, so a web +page and a desktop page render the same data. +""" + +from __future__ import annotations + +from collections import Counter +from collections.abc import Callable +from dataclasses import dataclass, field +from datetime import datetime, timezone +from typing import Any + +from automation_file.app.audit_service import AuditService +from automation_file.app.integrity_service import IntegrityService +from automation_file.app.masking import mask_secrets +from automation_file.app.notification_service import NotificationService +from automation_file.app.pipeline_service import PipelineService +from automation_file.app.scheduler_service import SchedulerService +from automation_file.app.storage_service import StorageService +from automation_file.events import EventBus, Severity, event_bus +from automation_file.exceptions import FileAutomationException +from automation_file.logging_config import file_automation_logger + +STATUS_OK = "ok" +STATUS_ATTENTION = "attention" + +_DEFAULT_EVENTS = 20 +_DEFAULT_RUNS = 10 +_HISTORY_WINDOW = 50 +_RUN_STATUSES = ("running", "succeeded", "failed", "cancelled") +_FAILED = "failed" + + +@dataclass(frozen=True) +class DashboardSources: + """The services a dashboard reads.""" + + pipelines: PipelineService + integrity: IntegrityService + storage: StorageService + scheduler: SchedulerService + audit: AuditService + notifications: NotificationService + + +@dataclass(frozen=True) +class DashboardSummary: + """Everything a dashboard shows, read at one moment. + + ``status`` is ``"attention"`` when a recent run failed, a monitor found + drift or could not verify, or the bus holds a recent event of severity + ``error`` or worse; ``reasons`` says which, one sentence each. + """ + + generated_at: str + status: str = STATUS_OK + reasons: tuple[str, ...] = () + health: dict[str, Any] = field(default_factory=dict) + run_counts: dict[str, int] = field(default_factory=dict) + running_runs: list[dict[str, Any]] = field(default_factory=list) + recent_runs: list[dict[str, Any]] = field(default_factory=list) + integrity: list[dict[str, Any]] = field(default_factory=list) + events: list[dict[str, Any]] = field(default_factory=list) + storage: list[dict[str, Any]] = field(default_factory=list) + + def to_dict(self) -> dict[str, Any]: + """Return a JSON-serialisable mapping of the summary.""" + return { + "generated_at": self.generated_at, + "status": self.status, + "reasons": list(self.reasons), + "health": dict(self.health), + "run_counts": dict(self.run_counts), + "running_runs": list(self.running_runs), + "recent_runs": list(self.recent_runs), + "integrity": list(self.integrity), + "events": list(self.events), + "storage": list(self.storage), + } + + +def brief_run(view: dict[str, Any]) -> dict[str, Any]: + """Return a run without its task details: what a list of runs shows.""" + tasks = view.get("tasks") or {} + return { + "run_id": view.get("run_id"), + "pipeline": view.get("pipeline"), + "status": view.get("status"), + "active": bool(view.get("active")), + "started_at": view.get("started_at"), + "finished_at": view.get("finished_at"), + "error": view.get("error"), + "tasks": len(tasks), + "task_statuses": dict(Counter(str(state.get("status")) for state in tasks.values())), + } + + +def _now() -> str: + return datetime.now(timezone.utc).isoformat(timespec="seconds") + + +class DashboardService: + """Health, runs, integrity drift, recent events and storage status in one place.""" + + def __init__(self, sources: DashboardSources, *, bus: EventBus | None = None) -> None: + self._sources = sources + self._bus = event_bus if bus is None else bus + + def health(self) -> dict[str, Any]: + """Return what is alive and configured: counts and switches, no history.""" + sources = self._sources + return { + "process": "alive", + "time": _now(), + "registry_size": len(sources.pipelines.action_names()), + "running_runs": len(sources.pipelines.running()), + "scheduler_jobs": len(sources.scheduler.jobs()), + "integrity_monitors": len(sources.integrity.status()), + "notification_sinks": len(sources.notifications.sink_names()), + "notification_routes": len(sources.notifications.routes()), + "notification_router_active": sources.notifications.router_active(), + "audit": sources.audit.status(), + } + + def runs(self, limit: int = _DEFAULT_RUNS) -> dict[str, Any]: + """Return the running runs, the latest ended ones, and how the latest runs ended. + + ``counts`` covers the newest runs of the store (at most fifty), so it + says how things have been going lately, not since the beginning. + """ + pipelines = self._sources.pipelines + window = [brief_run(view) for view in pipelines.history(None, _HISTORY_WINDOW)] + counts = Counter(str(run["status"]) for run in window) + return { + "running": [brief_run(view) for view in pipelines.running()], + "recent": [run for run in window if run["status"] != "running"][: max(limit, 0)], + "counts": {status: counts.get(status, 0) for status in _RUN_STATUSES}, + "window": len(window), + } + + def integrity(self) -> list[dict[str, Any]]: + """Return what every named integrity monitor last found.""" + return [drift.to_dict() for drift in self._sources.integrity.drift()] + + def recent_events( + self, limit: int = _DEFAULT_EVENTS, min_severity: str = Severity.INFO.value + ) -> list[dict[str, Any]]: + """Return the latest events of the bus, newest first, with their secrets masked.""" + events = self._bus.recent(limit, min_severity=Severity(min_severity)) + return [mask_secrets(event.to_dict()) for event in events] + + def storage_status(self) -> list[dict[str, Any]]: + """Return every storage backend and whether it can be used.""" + return [status.to_dict() for status in self._sources.storage.backends()] + + def summary(self, events: int = _DEFAULT_EVENTS, runs: int = _DEFAULT_RUNS) -> DashboardSummary: + """Return the whole dashboard. A part that cannot be read is reported, not raised.""" + reasons: list[str] = [] + run_data = self._part("pipeline runs", lambda: self.runs(runs), reasons) or {} + integrity = self._part("integrity", self.integrity, reasons) or [] + recent = self._part("events", lambda: self.recent_events(events), reasons) or [] + storage = self._part("storage", self.storage_status, reasons) or [] + health = self._part("health", self.health, reasons) or {} + reasons.extend(_attention(run_data, integrity, recent)) + return DashboardSummary( + generated_at=_now(), + status=STATUS_ATTENTION if reasons else STATUS_OK, + reasons=tuple(reasons), + health=health, + run_counts=run_data.get("counts", {}), + running_runs=run_data.get("running", []), + recent_runs=run_data.get("recent", []), + integrity=integrity, + events=recent, + storage=storage, + ) + + @staticmethod + def _part(name: str, read: Callable[[], Any], reasons: list[str]) -> Any: + try: + return read() + except FileAutomationException as error: + file_automation_logger.warning("dashboard: cannot read %s: %r", name, error) + reasons.append(f"{name} cannot be read: {type(error).__name__}") + return None + + +def _attention( + run_data: dict[str, Any], integrity: list[dict[str, Any]], events: list[dict[str, Any]] +) -> list[str]: + reasons: list[str] = [] + failed = run_data.get("counts", {}).get(_FAILED, 0) + if failed: + reasons.append(f"{failed} of the last {run_data.get('window', 0)} pipeline runs failed") + for monitor in integrity: + if monitor.get("last_error"): + reasons.append(f"integrity monitor {monitor['name']!r} could not verify") + elif monitor.get("ok") is False: + reasons.append( + f"integrity monitor {monitor['name']!r} found {monitor.get('changes', 0)} change(s)" + ) + serious = [ + event + for event in events + if Severity(event.get("severity", Severity.INFO.value)).at_least(Severity.ERROR) + ] + if serious: + reasons.append(f"{len(serious)} recent event(s) of severity error or worse") + return reasons diff --git a/automation_file/app/errors.py b/automation_file/app/errors.py new file mode 100644 index 0000000..956e77c --- /dev/null +++ b/automation_file/app/errors.py @@ -0,0 +1,15 @@ +"""The exception of the application layer.""" + +from __future__ import annotations + +from automation_file.exceptions import FileAutomationException + + +class AppException(FileAutomationException): + """Raised when the application layer cannot carry out what a user interface asked for. + + The domain packages keep their own exceptions (``StorageException``, + ``PipelineException`` ...); this one covers what only the application layer + checks: an edit a draft refuses, a form value that cannot be read, a request + for something that is not there. + """ diff --git a/automation_file/app/file_service.py b/automation_file/app/file_service.py new file mode 100644 index 0000000..0573c16 --- /dev/null +++ b/automation_file/app/file_service.py @@ -0,0 +1,252 @@ +"""The Files service: browse and change what is behind a storage URI. + +.. code-block:: python + + from automation_file.app import app_services + + files = app_services().files + for entry in files.list_dir("s3://reports/2026"): + print(entry.name, entry.size) + files.preview("s3://reports/2026/q1.csv").text + files.copy("s3://reports/2026/q1.csv", "local:///backup/") + +Every path goes through the storage layer, so ``..`` is refused, credentials +in a URI are refused and a mounted directory cannot be left. A preview is +bounded twice: at most ``preview_bytes`` are returned, and a remote file larger +than ``fetch_limit`` is not fetched at all, because a backend without ranged +reads has to download a file before any of it can be read. +""" + +from __future__ import annotations + +import codecs +from dataclasses import asdict, dataclass +from typing import Any + +from automation_file.app.errors import AppException +from automation_file.storage import ( + File, + FileInfo, + LocalStorage, + MemoryStorage, + Storage, + StorageResolver, + StorageURI, + default_resolver, + parse_storage_uri, +) + +DEFAULT_PREVIEW_BYTES = 64 * 1024 +DEFAULT_FETCH_LIMIT = 16 * 1024 * 1024 +_HEX_PREVIEW_BYTES = 256 +_TEXT_ENCODING = "utf-8" +_NUL = b"\x00" + + +@dataclass(frozen=True) +class FileEntry: + """One file or directory, with the URI that addresses it.""" + + uri: str + path: str + name: str + is_dir: bool = False + size: int | None = None + modified_at: str | None = None + content_type: str | None = None + etag: str | None = None + + def to_dict(self) -> dict[str, Any]: + """Return a JSON-serialisable mapping of the entry.""" + return asdict(self) + + +@dataclass(frozen=True) +class FilePreview: + """The beginning of a file. + + ``text`` holds at most ``shown`` bytes of content, decoded as UTF-8; for + ``binary`` content it is a hexadecimal dump of the first bytes. ``truncated`` + says the file is longer than what is shown, and ``note`` says why nothing is + shown when the file was not read. + """ + + uri: str + size: int | None + shown: int = 0 + truncated: bool = False + binary: bool = False + text: str = "" + note: str = "" + + def to_dict(self) -> dict[str, Any]: + """Return a JSON-serialisable mapping of the preview.""" + return asdict(self) + + +def _entry(info: FileInfo, uri: StorageURI) -> FileEntry: + return FileEntry( + uri=str(uri), + path=info.path, + name=uri.name, + is_dir=info.is_dir, + size=info.size, + modified_at=info.modified_at.isoformat() if info.modified_at else None, + content_type=info.content_type, + etag=info.etag, + ) + + +def _sort_key(entry: FileEntry) -> tuple[bool, str]: + return (not entry.is_dir, entry.path.casefold()) + + +def _decoded(data: bytes) -> str | None: + """Return ``data`` as text, or ``None`` when it is not UTF-8 text.""" + if _NUL in data: + return None + try: + # A sample may end in the middle of a character: do not treat that as an error. + return codecs.getincrementaldecoder(_TEXT_ENCODING)().decode(data, final=False) + except UnicodeDecodeError: + return None + + +class FileService: + """List, inspect, preview, copy, move and delete through storage URIs.""" + + def __init__( + self, + resolver: StorageResolver | None = None, + *, + preview_bytes: int = DEFAULT_PREVIEW_BYTES, + fetch_limit: int = DEFAULT_FETCH_LIMIT, + ) -> None: + if preview_bytes < 1 or fetch_limit < 1: + raise AppException("preview_bytes and fetch_limit must be 1 or more") + self._resolver = default_resolver if resolver is None else resolver + self._preview_bytes = preview_bytes + self._fetch_limit = fetch_limit + + def normalize(self, uri: str) -> str: + """Return ``uri`` in its canonical form; a local path becomes a ``local://`` URI.""" + return str(parse_storage_uri(uri)) + + def parent(self, uri: str) -> str: + """Return the URI one level up; a root is its own parent.""" + return str(parse_storage_uri(uri).parent) + + def child(self, uri: str, name: str) -> str: + """Return the URI of ``name`` below the directory ``uri``.""" + return str(parse_storage_uri(uri).joinpath(name)) + + def exists(self, uri: str) -> bool: + """Return whether a file or a directory is at ``uri``.""" + return self._file(uri).exists() + + def stat(self, uri: str) -> FileEntry: + """Return the entry at ``uri``.""" + target = self._file(uri) + return _entry(target.stat(), target.uri) + + def list_dir(self, uri: str, recursive: bool = False) -> list[FileEntry]: + """Return the entries of the directory ``uri``: directories first, then by path.""" + directory = Storage(uri, resolver=self._resolver) + entries = [ + _entry(info, directory.uri.joinpath(info.path)) + for info in directory.list_dir(recursive=recursive) + ] + return sorted(entries, key=_sort_key) + + def preview(self, uri: str, max_bytes: int | None = None) -> FilePreview: + """Return the beginning of the file ``uri``, at most ``max_bytes`` of it.""" + limit = self._preview_bytes if max_bytes is None else max_bytes + if limit < 1: + raise AppException("a preview needs at least one byte") + target = self._file(uri) + info = target.stat() + location = str(target.uri) + if info.is_dir: + raise AppException(f"{location} is a directory; there is nothing to preview") + if not self._reads_in_place(target) and (info.size or 0) > self._fetch_limit: + return FilePreview( + uri=location, + size=info.size, + truncated=True, + note=( + f"not fetched: {info.size} bytes is more than the preview limit of " + f"{self._fetch_limit} bytes for a remote file" + ), + ) + with target.open_read() as stream: + data = stream.read(limit + 1) + truncated = len(data) > limit + sample = data[:limit] + text = _decoded(sample) + if text is None: + return FilePreview( + uri=location, + size=info.size, + shown=min(len(sample), _HEX_PREVIEW_BYTES), + truncated=truncated or len(sample) > _HEX_PREVIEW_BYTES, + binary=True, + text=sample[:_HEX_PREVIEW_BYTES].hex(" "), + note="binary content, shown as hexadecimal", + ) + return FilePreview( + uri=location, size=info.size, shown=len(sample), truncated=truncated, text=text + ) + + def copy(self, source: str, target: str, overwrite: bool = True) -> FileEntry: + """Copy ``source`` to ``target`` and return the new entry. + + A file copied onto an existing directory lands inside it under its own + name. A directory is copied with everything below it. + """ + origin = self._file(source) + if origin.stat().is_dir: + Storage(source, resolver=self._resolver).copy_to( + Storage(target, resolver=self._resolver), overwrite=overwrite + ) + return self.stat(target) + copied = origin.copy_to(self._destination(origin, target), overwrite=overwrite) + return _entry(copied.stat(), copied.uri) + + def move(self, source: str, target: str, overwrite: bool = True) -> FileEntry: + """Move the file ``source`` to ``target`` and return the new entry. + + A file moved onto an existing directory lands inside it. A directory is + refused: copy it and delete the original once the copy has been checked. + """ + origin = self._file(source) + if origin.stat().is_dir: + raise AppException( + f"{origin.uri} is a directory; copy it and delete the original instead of moving it" + ) + moved = origin.move_to(self._destination(origin, target), overwrite=overwrite) + return _entry(moved.stat(), moved.uri) + + def delete(self, uri: str, recursive: bool = False) -> bool: + """Remove the file or directory at ``uri``; a directory with entries needs ``recursive``.""" + Storage(uri, resolver=self._resolver).delete("", recursive=recursive) + return True + + def mkdir(self, uri: str) -> bool: + """Create the directory ``uri`` and its missing parents.""" + Storage(uri, resolver=self._resolver).mkdir() + return True + + def _file(self, uri: str) -> File: + return File(uri, resolver=self._resolver) + + def _destination(self, origin: File, target: str) -> File: + """Return where ``origin`` goes: ``target``, or inside it when it is a directory.""" + destination = self._file(target) + if destination.is_dir(): + return self._file(str(destination.uri.joinpath(origin.name))) + return destination + + def _reads_in_place(self, target: File) -> bool: + """Return whether the backend can read the start of a file without fetching all of it.""" + backend = self._resolver.resolve(target.uri)[0] + return isinstance(backend, (LocalStorage, MemoryStorage)) diff --git a/automation_file/app/integrity_service.py b/automation_file/app/integrity_service.py new file mode 100644 index 0000000..73dbc36 --- /dev/null +++ b/automation_file/app/integrity_service.py @@ -0,0 +1,173 @@ +"""The Integrity service: baselines, verification and the named monitors. + +.. code-block:: python + + from automation_file.app import app_services + + integrity = app_services().integrity + integrity.baseline("s3://reports/2026", "local:///var/lib/fa/reports.json") + report = integrity.verify("s3://reports/2026", "local:///var/lib/fa/reports.json") + report["ok"], report["counts"] + integrity.start_monitor("reports", "s3://reports/2026", + "local:///var/lib/fa/reports.json", interval=300) + integrity.drift() # one summary per monitor, for a dashboard + +Everything goes through the ``FA_integrity_*`` functions of +:mod:`automation_file.integrity.actions`, so a monitor started here is the same +named monitor the actions, the CLI and the servers see. Nothing here remediates. +""" + +from __future__ import annotations + +import threading +from dataclasses import asdict, dataclass, field +from typing import Any + +from automation_file.app.errors import AppException +from automation_file.integrity import DEFAULT_ALGORITHM, STRONG_ALGORITHMS, IntegrityException +from automation_file.integrity.actions import ( + integrity_accept, + integrity_baseline, + integrity_status, + integrity_verify, + integrity_watch_start, + integrity_watch_stop, +) +from automation_file.logging_config import file_automation_logger + +DEFAULT_INTERVAL = 60.0 + + +@dataclass(frozen=True) +class MonitorDrift: + """What one named monitor last found. + + ``ok`` is ``None`` until the monitor has verified once. ``changes`` is the + number of differences of the last report and ``counts`` splits it by kind. + """ + + name: str + target: str + baseline: str | None + running: bool + ok: bool | None = None + changes: int = 0 + counts: dict[str, int] = field(default_factory=dict) + last_run: str | None = None + last_error: str | None = None + + @property + def needs_attention(self) -> bool: + """Whether the monitor found drift or could not verify.""" + return self.ok is False or self.last_error is not None + + def to_dict(self) -> dict[str, Any]: + """Return a JSON-serialisable mapping of the summary.""" + return {**asdict(self), "needs_attention": self.needs_attention} + + +def _drift(status: dict[str, Any]) -> MonitorDrift: + report = status.get("last_report") or {} + counts = {kind: int(count) for kind, count in (report.get("counts") or {}).items()} + return MonitorDrift( + name=str(status.get("name", "")), + target=str(status.get("target", "")), + baseline=status.get("baseline"), + running=bool(status.get("running")), + ok=report.get("ok") if report else None, + changes=len(report.get("changes") or ()), + counts=counts, + last_run=status.get("last_run"), + last_error=status.get("last_error"), + ) + + +def _required(value: str, what: str) -> str: + if not isinstance(value, str) or not value.strip(): + raise AppException(f"{what} is required") + return value.strip() + + +class IntegrityService: + """Baseline, verify, accept, and start or stop a monitor.""" + + def __init__(self) -> None: + self._lock = threading.Lock() + self._started: list[str] = [] + + def algorithms(self) -> list[str]: + """Return the digest algorithms a baseline may use, the default first.""" + others = sorted(name for name in STRONG_ALGORITHMS if name != DEFAULT_ALGORITHM) + return [DEFAULT_ALGORITHM, *others] + + def baseline( + self, target: str, baseline: str, algorithm: str = DEFAULT_ALGORITHM + ) -> dict[str, Any]: + """Snapshot ``target`` and store it at ``baseline`` as the approved state.""" + return integrity_baseline( + _required(target, "the target"), _required(baseline, "the baseline"), algorithm + ) + + def verify(self, target: str, baseline: str, deep: bool = True) -> dict[str, Any]: + """Compare ``target`` with ``baseline`` and return the drift report. + + ``deep=False`` hashes only the files whose size, time or etag changed. + Drift is published as an ``integrity.violation`` event. + """ + return integrity_verify( + _required(target, "the target"), _required(baseline, "the baseline"), deep + ) + + def accept(self, target: str, baseline: str) -> dict[str, Any]: + """Approve the current state of ``target`` as the new baseline.""" + return integrity_accept( + _required(target, "the target"), _required(baseline, "the baseline") + ) + + def status(self, name: str | None = None) -> list[dict[str, Any]]: + """Return the status of the monitor ``name``, or of every named monitor.""" + return integrity_status(name) + + def drift(self) -> list[MonitorDrift]: + """Return what every named monitor last found.""" + return [_drift(status) for status in integrity_status()] + + def start_monitor( + self, name: str, target: str, baseline: str, interval: float = DEFAULT_INTERVAL + ) -> dict[str, Any]: + """Start the monitor ``name``: verify ``target`` every ``interval`` seconds.""" + if interval <= 0: + raise AppException(f"the interval must be more than 0 seconds, got {interval!r}") + chosen = _required(name, "the monitor name") + status = integrity_watch_start( + chosen, _required(target, "the target"), _required(baseline, "the baseline"), interval + ) + with self._lock: + self._started.append(chosen) + return status + + def stop_monitor(self, name: str) -> dict[str, Any]: + """Stop the monitor ``name`` and return its last status.""" + status = integrity_watch_stop(name) + with self._lock: + if name in self._started: + self._started.remove(name) + return status + + def stop_started(self) -> list[str]: + """Stop the monitors this service started and return their names. + + A user interface calls this when it closes, so the monitors it started + do not outlive it; monitors started elsewhere are left alone. + """ + with self._lock: + names, self._started = self._started, [] + stopped: list[str] = [] + for name in names: + try: + integrity_watch_stop(name) + except IntegrityException as error: + file_automation_logger.info("integrity: monitor %r was gone: %r", name, error) + else: + stopped.append(name) + return stopped diff --git a/automation_file/app/masking.py b/automation_file/app/masking.py new file mode 100644 index 0000000..6f65e21 --- /dev/null +++ b/automation_file/app/masking.py @@ -0,0 +1,98 @@ +"""Keep secrets out of every view. + +A user interface shows events, audit records, run parameters and configuration +summaries, and any of them may carry a token, a password or a webhook URL. +:func:`mask_secrets` returns a copy that is safe to render: + +* a value stored under a name that says it is a secret (``password``, + ``token``, ``api_key``, ``authorization`` ...) becomes :data:`MASK`; +* a value stored under a name that says it is a URL (``url``, ``webhook_url``) + keeps its scheme and host and loses the rest, because the path of a webhook + URL is the credential; +* in any other text, the user information of a URL (``user:password@``) and + the token after ``Bearer`` are removed. + +Storage URIs are left alone: they cannot carry credentials. Nothing here decides +what is logged; it is the last line of defence for what is displayed. +""" + +from __future__ import annotations + +import re +from collections.abc import Mapping +from typing import Any + +MASK = "********" + +_SECRET_WORDS = ( + "password", + "passwd", + "passphrase", + "secret", + "token", + "apikey", + "api_key", + "access_key", + "accesskey", + "private_key", + "credential", + "authorization", + "connection_string", + "signature", + "cookie", +) +_URL_WORDS = ("url", "webhook") +_CREDENTIALS = re.compile(r"(?i)\b([a-z][a-z0-9+.-]*://)[^/\s@'\"]+@") +_HTTP_URL = re.compile(r"(?i)\b(https?://)(?:[^/\s@'\"]*@)?([^/\s'\"?#]+)[^\s'\")]*") +_BEARER = re.compile(r"(?i)\b(bearer)\s+[A-Za-z0-9._~+/=-]+") + + +def _folded(name: object) -> str: + return name.strip().lower().replace("-", "_") if isinstance(name, str) else "" + + +def is_secret_name(name: object) -> bool: + """Return whether a field called ``name`` holds a secret.""" + folded = _folded(name) + return any(word in folded for word in _SECRET_WORDS) + + +def is_url_name(name: object) -> bool: + """Return whether a field called ``name`` holds a URL that may carry a credential.""" + parts = _folded(name).split("_") + return any(word in parts for word in _URL_WORDS) + + +def mask_url(url: str) -> str: + """Return ``url`` reduced to its scheme and host; text without an HTTP URL becomes the mask.""" + reduced, found = _HTTP_URL.subn(rf"\1\2/{MASK}", url) + return reduced if found else MASK + + +def mask_text(text: str) -> str: + """Return ``text`` without URL credentials and without bearer tokens.""" + cleaned = _CREDENTIALS.sub(rf"\1{MASK}@", text) + return _BEARER.sub(rf"\1 {MASK}", cleaned) + + +def _masked_value(name: object, value: Any) -> Any: + if is_secret_name(name): + return value if value in (None, "") else MASK + if is_url_name(name) and isinstance(value, str) and value: + return mask_url(value) + return mask_secrets(value) + + +def mask_secrets(value: Any) -> Any: + """Return a copy of ``value`` that is safe to show. + + Mappings, lists and tuples are walked; tuples come back as lists, so the + result is JSON-friendly. Any other value is returned as it is. + """ + if isinstance(value, str): + return mask_text(value) + if isinstance(value, Mapping): + return {key: _masked_value(key, item) for key, item in value.items()} + if isinstance(value, (list, tuple)): + return [mask_secrets(item) for item in value] + return value diff --git a/automation_file/app/notification_service.py b/automation_file/app/notification_service.py new file mode 100644 index 0000000..9c90751 --- /dev/null +++ b/automation_file/app/notification_service.py @@ -0,0 +1,132 @@ +"""The Notifications service: the registered sinks, the routes, and a test message. + +.. code-block:: python + + from automation_file.app import app_services + + notifications = app_services().notifications + notifications.sinks() # [{"name": "team-alerts", "type": "SlackSink", ...}] + notifications.add_route({"name": "failures", "sinks": ["team-alerts"], + "types": ["pipeline.failed", "task.failed"], + "min_severity": "error"}) + notifications.send_test("team-alerts") # {"team-alerts": "sent"} + +A sink is described by its name, its type and where it delivers, never by its +webhook URL, token or password. Sinks themselves are registered in code or from +the configuration file (see the Settings service). +""" + +from __future__ import annotations + +from collections.abc import Mapping +from typing import Any + +from automation_file.app.arguments import parse_argument_text, split_names +from automation_file.app.errors import AppException +from automation_file.app.masking import mask_secrets +from automation_file.events import Severity +from automation_file.notify import ( + NotificationManager, + NotificationRouter, + Route, + notification_manager, + notification_router, +) + +TEST_SUBJECT = "automation_file: test notification" +TEST_BODY = "This message was sent to check that the notification sink delivers." +_LIST_OPTIONS = ("sinks", "types", "sources") +_NUMBER_OPTIONS = ("dedup_seconds", "rate_limit", "rate_period") + + +def _option(key: str, value: Any) -> Any: + """Return a route option as the router expects it, read from form text when it is text.""" + if key in _LIST_OPTIONS: + return split_names(value) + if key in _NUMBER_OPTIONS and isinstance(value, str): + return parse_argument_text(value) + return value + + +class NotificationService: + """Sinks and routes of one notification manager and router.""" + + def __init__( + self, + manager: NotificationManager | None = None, + router: NotificationRouter | None = None, + ) -> None: + self._manager = notification_manager if manager is None else manager + self._router = notification_router if router is None else router + + def severities(self) -> list[str]: + """Return the severity names a route may use as its minimum, in rising order.""" + return [severity.value for severity in Severity] + + def sinks(self) -> list[dict[str, Any]]: + """Return a description of every registered sink, without its secrets.""" + return [mask_secrets(described) for described in self._manager.list()] + + def sink_names(self) -> list[str]: + """Return the names of the registered sinks, in registration order.""" + return list(self._manager.names()) + + def routes(self) -> list[dict[str, Any]]: + """Return every route, in the order they were added.""" + return [route.to_dict() for route in self._router.routes()] + + def router_active(self) -> bool: + """Return whether the router is delivering events.""" + return self._router.active + + def add_route(self, options: Mapping[str, Any]) -> dict[str, Any]: + """Add or replace a route and start routing; return the route as stored. + + ``options`` holds ``name`` and, optionally, ``sinks``, ``types``, + ``sources``, ``min_severity``, ``dedup_seconds``, ``rate_limit`` and + ``rate_period``. The three lists may be given as comma-separated text. + A sink that is not registered is refused, so a typing mistake does not + become a route that never delivers. + """ + given = { + key: _option(key, value) + for key, value in options.items() + if value is not None and value != "" + } + route = Route.from_mapping(given) + known = self._manager.names() + unknown = [sink for sink in route.sinks if sink not in known] + if unknown: + raise AppException( + f"route {route.name!r} names unknown sink(s) {unknown} (registered: {list(known)})" + ) + self._router.add_route(route) + self._router.start() + return route.to_dict() + + def remove_route(self, name: str) -> bool: + """Remove the route ``name``; stop routing when none is left. Return whether it existed.""" + removed = self._router.remove_route(name) + if removed and not self._router.routes(): + self._router.stop() + return removed + + def send_test( + self, + sink: str | None = None, + subject: str = TEST_SUBJECT, + body: str = TEST_BODY, + ) -> dict[str, str]: + """Send a test message to ``sink``, or to every sink, and return one outcome per sink. + + An outcome is ``"sent"`` or the error as ``": "`` + with its URLs reduced to the host. The deduplication window is not + applied: a test is sent every time. + """ + names = [sink] if sink else self.sink_names() + if not names: + raise AppException("no notification sink is registered; there is nothing to test") + return { + name: self._manager.send_to(name, subject or TEST_SUBJECT, body, "info") + for name in names + } diff --git a/automation_file/app/pipeline_draft.py b/automation_file/app/pipeline_draft.py new file mode 100644 index 0000000..be49d31 --- /dev/null +++ b/automation_file/app/pipeline_draft.py @@ -0,0 +1,667 @@ +"""An editable draft of a pipeline definition. + +A pipeline editor does not edit a :class:`~automation_file.pipeline.Pipeline`: +that object refuses anything invalid, and a definition under construction is +invalid most of the time. A :class:`PipelineDraft` holds whatever the user has +entered so far, answers :meth:`PipelineDraft.problems` with the path of every +finding, and becomes a definition document with :meth:`PipelineDraft.to_definition`. + +.. code-block:: python + + from automation_file.app import PipelineDraft + + draft = PipelineDraft("nightly") + draft.add_task("FA_storage_copy", "download", + arguments={"source": "s3://in/a.csv", "target": "local:///tmp/a.csv"}) + draft.add_task("FA_storage_delete", "tidy", arguments={"uri": "local:///tmp/a.csv"}) + draft.connect("download", "tidy") # tidy depends on download + draft.problems() # [] when the definition is valid + draft.to_definition() # what Pipeline.from_dict takes + +The position of every task on a canvas is editor metadata. It lives on the draft +(:meth:`PipelineDraft.set_position`, :meth:`PipelineDraft.layout`) and never +appears in :meth:`PipelineDraft.to_definition`. + +A draft is not thread-safe: edit it from one thread, and hand a worker the +document :meth:`PipelineDraft.to_definition` returns. +""" + +from __future__ import annotations + +import copy +from collections.abc import Callable, Iterator, Mapping +from contextlib import contextmanager +from dataclasses import dataclass, field +from typing import Any + +from automation_file.app.errors import AppException +from automation_file.pipeline import SCHEMA_VERSION, validate_definition +from automation_file.pipeline.model import ON_SUCCESS, WHEN_CHOICES +from automation_file.pipeline.substitution import NAME_RULE, is_name + +CHANGE_STRUCTURE = "structure" +CHANGE_TASK = "task" +CHANGE_HEADER = "header" +CHANGE_POSITION = "position" + +LAYOUT_VERSION = 1 +DEFAULT_NAME = "pipeline" +DEFAULT_MAX_WORKERS = 4 +DEFAULT_BACKOFF_CAP = 60.0 + +_TASKS_PREFIX = "tasks." +_ACTION_PREFIX = "FA_" +_COLUMN_WIDTH = 240.0 +_ROW_HEIGHT = 110.0 +_MARGIN = 40.0 +_PATH_SEPARATORS = (".", "[") +_SINGLE_ATTEMPT = {"max_attempts": 1} + +DraftListener = Callable[[str], None] +Arguments = dict[str, Any] | list[Any] | None + + +@dataclass(frozen=True) +class Problem: + """One finding of a validation: where it is and what is wrong. + + ``path`` is the path inside the definition (``tasks.verify.depends_on[0]``) + and ``task`` the ID of the task it belongs to, when it belongs to one. + """ + + path: str + message: str + task: str | None = None + + def __str__(self) -> str: + return f"{self.path}: {self.message}" if self.path else self.message + + def to_dict(self) -> dict[str, Any]: + """Return a JSON-serialisable mapping of the problem.""" + return {"path": self.path, "message": self.message, "task": self.task} + + @classmethod + def parse(cls, text: str, task_ids: tuple[str, ...] = ()) -> Problem: + """Build a problem from the ``": "`` text the pipeline package returns.""" + path, separator, message = text.partition(": ") + if not separator: + return cls(path="", message=text) + return cls(path=path, message=message, task=_task_of(path, task_ids)) + + +def _task_of(path: str, task_ids: tuple[str, ...]) -> str | None: + """Return the ID of the task ``path`` points into, the longest ID that fits.""" + if not path.startswith(_TASKS_PREFIX): + return None + rest = path[len(_TASKS_PREFIX) :] + fitting = [ + task_id + for task_id in task_ids + if rest == task_id + or rest.startswith(tuple(f"{task_id}{mark}" for mark in _PATH_SEPARATORS)) + ] + if fitting: + return max(fitting, key=len) + return rest.split(".", 1)[0] or None + + +@dataclass +class DraftTask: + """One task of a draft: an action, how it runs, and where it sits on the canvas. + + ``arguments`` is the action's keyword mapping, its positional list, or + ``None`` for an action without arguments. ``retry`` is the ``retry`` entry + of a definition (``max_attempts``, ``backoff``, ``backoff_cap``, ``on``) or + ``None`` for a single attempt. ``x`` and ``y`` are editor metadata. + """ + + task_id: str + action: str = "" + arguments: Arguments = None + depends_on: list[str] = field(default_factory=list) + retry: dict[str, Any] | None = None + timeout: float | None = None + when: str = ON_SUCCESS + idempotency_key: str | None = None + x: float = 0.0 + y: float = 0.0 + + def to_spec(self) -> dict[str, Any]: + """Return the task's entry in a definition; defaults and the position are left out.""" + action: list[Any] = [self.action] + if self.arguments is not None: + action.append(copy.deepcopy(self.arguments)) + spec: dict[str, Any] = {"action": action} + if self.depends_on: + spec["depends_on"] = list(self.depends_on) + if self.retry: + spec["retry"] = copy.deepcopy(self.retry) + if self.timeout is not None: + spec["timeout"] = self.timeout + if self.when != ON_SUCCESS: + spec["when"] = self.when + if self.idempotency_key: + spec["idempotency_key"] = self.idempotency_key + return spec + + def to_dict(self) -> dict[str, Any]: + """Return the task as a view needs it: its definition entry, its ID and its position.""" + return {"id": self.task_id, **self.to_spec(), "position": [self.x, self.y]} + + +def _renamed_references(value: Any, old: str, new: str) -> Any: + """Return ``value`` with every ``${tasks..result}`` placeholder pointing at ``new``.""" + if isinstance(value, str): + return value.replace(f"${{tasks.{old}.result}}", f"${{tasks.{new}.result}}") + if isinstance(value, Mapping): + return {key: _renamed_references(item, old, new) for key, item in value.items()} + if isinstance(value, list): + return [_renamed_references(item, old, new) for item in value] + return value + + +def _number(value: Any) -> float | None: + if isinstance(value, bool) or not isinstance(value, (int, float)): + return None + return float(value) + + +def _task_from_spec(task_id: str, spec: Mapping[str, Any]) -> DraftTask: + """Read what fits from a definition entry; validation reports what does not.""" + task = DraftTask(task_id=task_id) + action = spec.get("action") + if isinstance(action, list) and action: + task.action = action[0] if isinstance(action[0], str) else "" + if len(action) > 1 and isinstance(action[1], (dict, list)): + task.arguments = copy.deepcopy(action[1]) + wanted = spec.get("depends_on") + if isinstance(wanted, list): + task.depends_on = [item for item in dict.fromkeys(wanted) if isinstance(item, str)] + if isinstance(spec.get("retry"), Mapping): + task.retry = copy.deepcopy(dict(spec["retry"])) + task.timeout = _number(spec.get("timeout")) + if isinstance(spec.get("when"), str): + task.when = spec["when"] + if isinstance(spec.get("idempotency_key"), str): + task.idempotency_key = spec["idempotency_key"] + return task + + +class PipelineDraft: + """A pipeline definition being edited, with the canvas layout next to it.""" + + def __init__( + self, + name: str = DEFAULT_NAME, + description: str = "", + max_workers: int = DEFAULT_MAX_WORKERS, + ) -> None: + self._name = name + self._description = description + self._max_workers = max_workers + self._params: dict[str, Any] = {} + self._schedule: dict[str, Any] | None = None + self._tasks: dict[str, DraftTask] = {} + self._listeners: list[DraftListener] = [] + self._pending: list[str] | None = None + self._revision = 0 + self._saved_revision = 0 + #: What validation said about the document this draft was read from. + self.load_notes: tuple[str, ...] = () + + # ------------------------------------------------------------------ reading + + @property + def name(self) -> str: + return self._name + + @property + def description(self) -> str: + return self._description + + @property + def max_workers(self) -> int: + return self._max_workers + + @property + def params(self) -> dict[str, Any]: + """A copy of the default run parameters.""" + return copy.deepcopy(self._params) + + @property + def schedule(self) -> dict[str, Any] | None: + """A copy of the ``schedule`` entry (``cron``, ``timezone``), or ``None``.""" + return copy.deepcopy(self._schedule) + + @property + def tasks(self) -> tuple[DraftTask, ...]: + """The tasks in the order they were added.""" + return tuple(self._tasks.values()) + + @property + def revision(self) -> int: + """A number that grows with every change.""" + return self._revision + + @property + def dirty(self) -> bool: + """Whether the draft changed since :meth:`mark_saved`.""" + return self._revision != self._saved_revision + + def mark_saved(self) -> None: + """Remember the current state as the saved one.""" + self._saved_revision = self._revision + + def task_ids(self) -> tuple[str, ...]: + """Return the task IDs in the order the tasks were added.""" + return tuple(self._tasks) + + def has_task(self, task_id: str) -> bool: + return task_id in self._tasks + + def task(self, task_id: str) -> DraftTask: + """Return the task ``task_id``; an unknown ID raises :class:`AppException`.""" + try: + return self._tasks[task_id] + except KeyError: + raise AppException(f"the draft has no task {task_id!r}") from None + + def edges(self) -> list[tuple[str, str]]: + """Return ``(upstream, downstream)`` for every dependency between two existing tasks.""" + return [ + (dependency, task.task_id) + for task in self._tasks.values() + for dependency in task.depends_on + if dependency in self._tasks + ] + + # ------------------------------------------------------------------ listeners + + def add_listener(self, listener: DraftListener) -> None: + """Call ``listener(change)`` after every change; ``change`` is one of the ``CHANGE_*``.""" + if listener not in self._listeners: + self._listeners.append(listener) + + def remove_listener(self, listener: DraftListener) -> None: + if listener in self._listeners: + self._listeners.remove(listener) + + @contextmanager + def batch(self) -> Iterator[None]: + """Report the changes made inside the block once each, when the block ends.""" + if self._pending is not None: + yield + return + self._pending = [] + try: + yield + finally: + pending, self._pending = self._pending, None + for change in dict.fromkeys(pending): + self._emit(change) + + def _changed(self, change: str) -> None: + self._revision += 1 + if self._pending is not None: + self._pending.append(change) + else: + self._emit(change) + + def _emit(self, change: str) -> None: + for listener in list(self._listeners): + listener(change) + + # ------------------------------------------------------------------ the header + + def set_name(self, name: str) -> None: + self._name = name.strip() + self._changed(CHANGE_HEADER) + + def set_description(self, description: str) -> None: + self._description = description + self._changed(CHANGE_HEADER) + + def set_max_workers(self, max_workers: int) -> None: + if isinstance(max_workers, bool) or not isinstance(max_workers, int) or max_workers < 1: + raise AppException(f"max_workers must be an integer, 1 or more, got {max_workers!r}") + self._max_workers = max_workers + self._changed(CHANGE_HEADER) + + def set_params(self, params: Mapping[str, Any] | None) -> None: + """Replace the default run parameters.""" + if params is not None and not isinstance(params, Mapping): + raise AppException(f"params must be a mapping, got {type(params).__name__}") + self._params = copy.deepcopy(dict(params or {})) + self._changed(CHANGE_HEADER) + + def set_schedule(self, cron: str | None, timezone: str | None = None) -> None: + """Set the ``schedule`` entry, or remove it with ``cron=None``.""" + if cron is None or not cron.strip(): + self._schedule = None + else: + self._schedule = {"cron": cron.strip()} + if timezone and timezone.strip(): + self._schedule["timezone"] = timezone.strip() + self._changed(CHANGE_HEADER) + + # ------------------------------------------------------------------ tasks + + def add_task( + self, + action: str = "", + task_id: str | None = None, + *, + arguments: Arguments = None, + position: tuple[float, float] | None = None, + ) -> DraftTask: + """Add a task that calls ``action`` and return it. + + Without ``task_id`` an unused one is derived from the action name. + Without ``position`` the task is placed below the lowest one. + """ + chosen = self._unused_id(action) if task_id is None else self._checked_id(task_id) + x, y = self._free_position() if position is None else position + task = DraftTask( + task_id=chosen, + action=action, + arguments=_checked_arguments(arguments), + x=float(x), + y=float(y), + ) + self._tasks[chosen] = task + self._changed(CHANGE_STRUCTURE) + return task + + def remove_task(self, task_id: str) -> None: + """Remove a task and every dependency on it.""" + self.task(task_id) + del self._tasks[task_id] + for other in self._tasks.values(): + if task_id in other.depends_on: + other.depends_on.remove(task_id) + self._changed(CHANGE_STRUCTURE) + + def rename_task(self, task_id: str, new_id: str) -> DraftTask: + """Give a task another ID, keeping its place, its dependents and their placeholders.""" + task = self.task(task_id) + if new_id == task_id: + return task + self._checked_id(new_id) + task.task_id = new_id + self._tasks = { + (new_id if key == task_id else key): item for key, item in self._tasks.items() + } + for other in self._tasks.values(): + other.depends_on = [new_id if item == task_id else item for item in other.depends_on] + other.arguments = _renamed_references(other.arguments, task_id, new_id) + self._changed(CHANGE_STRUCTURE) + return task + + def set_action(self, task_id: str, action: str) -> None: + self.task(task_id).action = action.strip() + self._changed(CHANGE_TASK) + + def set_arguments(self, task_id: str, arguments: Arguments) -> None: + """Set the action's arguments: a keyword mapping, a positional list, or ``None``.""" + self.task(task_id).arguments = _checked_arguments(arguments) + self._changed(CHANGE_TASK) + + def set_retry( + self, + task_id: str, + max_attempts: int = 1, + backoff: float = 0.0, + backoff_cap: float = DEFAULT_BACKOFF_CAP, + on: list[str] | None = None, + ) -> None: + """Set how a task is retried; the defaults mean one attempt and remove the entry. + + ``on`` lists exception names (see + :data:`automation_file.pipeline.RETRYABLE_EXCEPTIONS`); ``None`` keeps + the transient kinds the runtime retries by default. + """ + task = self.task(task_id) + retry: dict[str, Any] = {"max_attempts": max_attempts} + if backoff: + retry["backoff"] = backoff + if backoff_cap != DEFAULT_BACKOFF_CAP: + retry["backoff_cap"] = backoff_cap + if on: + retry["on"] = list(on) + task.retry = None if retry == _SINGLE_ATTEMPT else retry + self._changed(CHANGE_TASK) + + def set_timeout(self, task_id: str, seconds: float | None) -> None: + """Set the task's budget in seconds, or remove it with ``None``.""" + self.task(task_id).timeout = seconds + self._changed(CHANGE_TASK) + + def set_condition(self, task_id: str, when: str) -> None: + """Set when the task runs: ``on_success``, ``on_failure`` or ``always``.""" + if when not in WHEN_CHOICES: + raise AppException(f"when must be one of {', '.join(WHEN_CHOICES)}, got {when!r}") + self.task(task_id).when = when + self._changed(CHANGE_TASK) + + def set_idempotency_key(self, task_id: str, key: str | None) -> None: + """Set the task's idempotency key, or remove it with ``None`` or an empty text.""" + self.task(task_id).idempotency_key = key or None + self._changed(CHANGE_TASK) + + # ------------------------------------------------------------------ dependency edges + + def connect(self, upstream: str, downstream: str) -> bool: + """Make ``downstream`` depend on ``upstream``; return whether an edge was added. + + An edge from a task to itself and an edge that would close a cycle are + refused with :class:`AppException`. + """ + self.task(upstream) + target = self.task(downstream) + if upstream == downstream: + raise AppException(f"task {upstream!r} cannot depend on itself") + if upstream in target.depends_on: + return False + if self._reaches(upstream, downstream): + raise AppException( + f"{upstream!r} already depends on {downstream!r}: " + "the edge would close a dependency cycle" + ) + target.depends_on.append(upstream) + self._changed(CHANGE_STRUCTURE) + return True + + def disconnect(self, upstream: str, downstream: str) -> bool: + """Remove the dependency of ``downstream`` on ``upstream``; return whether there was one.""" + target = self.task(downstream) + if upstream not in target.depends_on: + return False + target.depends_on.remove(upstream) + self._changed(CHANGE_STRUCTURE) + return True + + def set_dependencies(self, task_id: str, depends_on: list[str]) -> None: + """Make ``depends_on`` the complete list of what ``task_id`` depends on.""" + task = self.task(task_id) + wanted = list(dict.fromkeys(depends_on)) + with self.batch(): + for dependency in [item for item in task.depends_on if item not in wanted]: + self.disconnect(dependency, task_id) + for dependency in wanted: + self.connect(dependency, task_id) + + def _reaches(self, start: str, goal: str) -> bool: + """Return whether ``start`` depends on ``goal``, directly or through other tasks.""" + seen: set[str] = set() + stack = [start] + while stack: + current = stack.pop() + if current == goal: + return True + if current in seen or current not in self._tasks: + continue + seen.add(current) + stack.extend(self._tasks[current].depends_on) + return False + + # ------------------------------------------------------------------ canvas layout + + def set_position(self, task_id: str, x: float, y: float) -> None: + """Record where the task sits on the canvas.""" + task = self.task(task_id) + if (task.x, task.y) == (float(x), float(y)): + return + task.x, task.y = float(x), float(y) + self._changed(CHANGE_POSITION) + + def positions(self) -> dict[str, tuple[float, float]]: + """Return ``{task ID: (x, y)}``.""" + return {task.task_id: (task.x, task.y) for task in self._tasks.values()} + + def auto_layout(self) -> None: + """Place the tasks in columns by dependency depth, in the order they were added.""" + rows: dict[int, int] = {} + for task_id, level in self._levels().items(): + row = rows.get(level, 0) + rows[level] = row + 1 + task = self._tasks[task_id] + task.x = _MARGIN + level * _COLUMN_WIDTH + task.y = _MARGIN + row * _ROW_HEIGHT + self._changed(CHANGE_POSITION) + + def layout(self) -> dict[str, Any]: + """Return the editor metadata: the positions, kept apart from the definition.""" + return { + "layout_version": LAYOUT_VERSION, + "pipeline": self._name, + "positions": {task.task_id: [task.x, task.y] for task in self._tasks.values()}, + } + + def apply_layout(self, layout: Any) -> int: + """Take positions from what :meth:`layout` returned; return how many tasks were placed.""" + positions = layout.get("positions") if isinstance(layout, Mapping) else None + if not isinstance(positions, Mapping): + raise AppException("the layout has no 'positions' mapping") + placed = 0 + for task_id, point in positions.items(): + task = self._tasks.get(task_id) + if task is None or not isinstance(point, (list, tuple)) or len(point) != 2: + continue + x, y = _number(point[0]), _number(point[1]) + if x is None or y is None: + continue + task.x, task.y = x, y + placed += 1 + self._changed(CHANGE_POSITION) + return placed + + def _levels(self) -> dict[str, int]: + """Return the dependency depth of every task; a cycle cannot make it endless.""" + levels = dict.fromkeys(self._tasks, 0) + for _ in self._tasks: + moved = False + for task in self._tasks.values(): + depth = 1 + max( + (levels[item] for item in task.depends_on if item in levels), default=-1 + ) + depth = min(depth, len(self._tasks) - 1) + if depth != levels[task.task_id]: + levels[task.task_id] = depth + moved = True + if not moved: + break + return levels + + def _free_position(self) -> tuple[float, float]: + if not self._tasks: + return _MARGIN, _MARGIN + return _MARGIN, max(task.y for task in self._tasks.values()) + _ROW_HEIGHT + + # ------------------------------------------------------------------ documents + + def to_definition(self) -> dict[str, Any]: + """Return the definition document (``schema_version: 1``), without any editor metadata.""" + document: dict[str, Any] = {"schema_version": SCHEMA_VERSION, "name": self._name} + if self._description: + document["description"] = self._description + document["max_workers"] = self._max_workers + if self._schedule is not None: + document["schedule"] = copy.deepcopy(self._schedule) + if self._params: + document["params"] = copy.deepcopy(self._params) + document["tasks"] = {task.task_id: task.to_spec() for task in self._tasks.values()} + return document + + def problems(self) -> list[Problem]: + """Return what keeps the draft from being a valid definition, each with its path.""" + task_ids = self.task_ids() + return [Problem.parse(text, task_ids) for text in validate_definition(self.to_definition())] + + @classmethod + def from_definition(cls, document: Any, layout: Any = None) -> PipelineDraft: + """Build a draft from a definition document, valid or not. + + Whatever has the right shape is taken over, so a broken definition can + be opened and repaired; what validation says about the document as it + was is kept in :attr:`load_notes`. With ``layout`` the tasks get their + stored positions, otherwise they are laid out by dependency depth. + """ + if not isinstance(document, Mapping): + raise AppException(f"a pipeline definition is a mapping, got {type(document).__name__}") + draft = cls() + draft.load_notes = tuple(validate_definition(document)) + draft._read_header(document) + tasks = document.get("tasks") + for task_id, spec in tasks.items() if isinstance(tasks, Mapping) else (): + if is_name(task_id) and isinstance(spec, Mapping): + draft._tasks[task_id] = _task_from_spec(task_id, spec) + draft.auto_layout() + if layout is not None: + draft.apply_layout(layout) + draft._revision = 0 + return draft + + def _read_header(self, document: Mapping[str, Any]) -> None: + name = document.get("name") + self._name = name if isinstance(name, str) and name.strip() else DEFAULT_NAME + description = document.get("description") + self._description = description if isinstance(description, str) else "" + workers = document.get("max_workers") + if isinstance(workers, int) and not isinstance(workers, bool) and workers >= 1: + self._max_workers = workers + params = document.get("params") + self._params = copy.deepcopy(dict(params)) if isinstance(params, Mapping) else {} + schedule = document.get("schedule") + self._schedule = copy.deepcopy(dict(schedule)) if isinstance(schedule, Mapping) else None + + # ------------------------------------------------------------------ IDs + + def _checked_id(self, task_id: str) -> str: + if not is_name(task_id): + raise AppException(f"invalid task ID {task_id!r}: {NAME_RULE}") + if task_id in self._tasks: + raise AppException(f"the draft already has a task {task_id!r}") + return task_id + + def _unused_id(self, action: str) -> str: + base = action.strip().removeprefix(_ACTION_PREFIX) + if not is_name(base): + base = "task" + if base not in self._tasks: + return base + number = 2 + while f"{base}_{number}" in self._tasks: + number += 1 + return f"{base}_{number}" + + +def _checked_arguments(arguments: Any) -> Arguments: + if arguments is None: + return None + if isinstance(arguments, Mapping): + return copy.deepcopy(dict(arguments)) + if isinstance(arguments, list): + return copy.deepcopy(arguments) + raise AppException( + f"arguments must be a mapping, a list or nothing, got {type(arguments).__name__}" + ) diff --git a/automation_file/app/pipeline_service.py b/automation_file/app/pipeline_service.py new file mode 100644 index 0000000..c68ae2b --- /dev/null +++ b/automation_file/app/pipeline_service.py @@ -0,0 +1,456 @@ +"""The Pipelines service: check, run, follow and store pipeline definitions. + +.. code-block:: python + + from automation_file.app import app_services + + pipelines = app_services().pipelines + draft = pipelines.load("pipelines/daily-report.yaml") + pipelines.validate(draft) # [] or a list of Problem + plan = pipelines.dry_run(draft, {"date": "2026-10-08"}) + run = pipelines.start(draft, {"date": "2026-10-08"}) # returns at once + pipelines.status(run["run_id"])["status"] # "running", then "succeeded" ... + pipelines.cancel(run["run_id"]) + +Every method takes a :class:`~automation_file.app.pipeline_draft.PipelineDraft` +or a definition mapping, and returns plain dictionaries: a run is +``PipelineRun.to_dict()`` with its secrets masked and an ``active`` flag that +says whether this process is still executing it. + +A definition is saved as ``.yaml`` / ``.yml`` / ``.json``. The canvas layout goes +into a second file next to it (``.layout.json``), because a definition +refuses unknown keys and a layout is no part of what runs. +""" + +from __future__ import annotations + +import os +import threading +from collections.abc import Callable, Mapping +from dataclasses import dataclass +from pathlib import Path +from typing import TYPE_CHECKING, Any + +from automation_file.app.arguments import ActionInfo, describe_action +from automation_file.app.errors import AppException +from automation_file.app.masking import mask_secrets +from automation_file.app.pipeline_draft import DEFAULT_NAME, PipelineDraft, Problem +from automation_file.core.json_store import read_action_json, write_action_json +from automation_file.core.progress import CancellationToken +from automation_file.events import EventBus, event_bus +from automation_file.exceptions import FileAutomationException +from automation_file.logging_config import file_automation_logger +from automation_file.pipeline import ( + MemoryRunStore, + Pipeline, + PipelineDefinitionException, + PipelineException, + PipelineRun, + RunStatus, + RunStore, + default_run_store, + load_definition, + validate_definition, +) +from automation_file.pipeline.definition import retry_from_dict +from automation_file.pipeline.graph import upstream_tasks + +if TYPE_CHECKING: + from automation_file.core.action_registry import ActionRegistry + +Definition = PipelineDraft | Mapping[str, Any] + +LAYOUT_SUFFIX = ".layout.json" +_JSON_SUFFIX = ".json" +_YAML_SUFFIXES = (".yaml", ".yml") +_DEFAULT_HISTORY = 20 +_DEFAULT_EVENTS = 200 +_MAX_TRACKED = 200 +_SCAN_LIMIT = 200 + + +@dataclass +class _Tracked: + """A run this service started or resumed: how to stop it and how to tell it has ended.""" + + token: CancellationToken + run: PipelineRun | None = None + thread: threading.Thread | None = None + + @property + def alive(self) -> bool: + if self.run is not None: + return not self.run.done + return self.thread is not None and self.thread.is_alive() + + +def layout_path(path: str | os.PathLike[str]) -> Path: + """Return where the canvas layout of the definition file ``path`` is kept.""" + source = Path(path) + return source.with_name(source.name + LAYOUT_SUFFIX) + + +def _constant(value: Any) -> Callable[[Any], Any]: + return lambda _context: value + + +def _listed(spec: Any) -> list[str]: + wanted = spec.get("depends_on", []) if isinstance(spec, Mapping) else [] + return [item for item in wanted if isinstance(item, str)] if isinstance(wanted, list) else [] + + +class PipelineService: + """Validation, dry run, background runs, history and definition files. + + ``store`` is where runs are recorded; without one the default run store is + looked up on every call, so :func:`~automation_file.pipeline.set_default_run_store` + takes effect. ``registry`` is where action names are looked up (the shared + executor's by default) and ``bus`` receives the events of the runs. + """ + + def __init__( + self, + store: RunStore | None = None, + *, + registry: ActionRegistry | None = None, + bus: EventBus | None = None, + ) -> None: + self._store = store + self._registry = registry + self._bus = bus + self._lock = threading.Lock() + self._tracked: dict[str, _Tracked] = {} + + # ------------------------------------------------------------------ actions + + def action_names(self) -> list[str]: + """Return the name of every registered action, sorted.""" + return sorted(self._action_registry().event_dict) + + def describe_action(self, name: str) -> ActionInfo: + """Return the parameters and the summary of the action ``name``.""" + return describe_action(name, self._action_registry().resolve(name)) + + # ------------------------------------------------------------------ drafts and files + + def new_draft(self, name: str = DEFAULT_NAME) -> PipelineDraft: + """Return an empty draft.""" + return PipelineDraft(name) + + def load(self, path: str | os.PathLike[str]) -> PipelineDraft: + """Read a definition file into a draft, with its stored canvas layout when there is one. + + A file that cannot be read or parsed raises + :class:`~automation_file.pipeline.PipelineDefinitionException`. A file + that parses but is not a valid definition still opens: what is wrong + with it is in ``draft.load_notes`` and in :meth:`validate`. + """ + draft = PipelineDraft.from_definition(load_definition(path)) + sidecar = layout_path(path) + if sidecar.is_file(): + try: + draft.apply_layout(read_action_json(str(sidecar))) + except FileAutomationException as error: + file_automation_logger.warning( + "pipelines: ignoring the layout %s: %r", sidecar, error + ) + draft.mark_saved() + return draft + + def save(self, draft: PipelineDraft, path: str | os.PathLike[str]) -> str: + """Write the draft's definition to ``path`` and its layout next to it; return the path. + + The suffix picks the format: ``.yaml`` / ``.yml`` or ``.json``. A draft + that is not valid yet can be saved; validation is a separate step. + """ + target = Path(path) + suffix = target.suffix.lower() + definition = draft.to_definition() + if suffix == _JSON_SUFFIX: + write_action_json(str(target), definition) + elif suffix in _YAML_SUFFIXES: + _write_yaml(target, definition) + else: + raise AppException(f"{target}: save a definition as .yaml, .yml or .json") + write_action_json(str(layout_path(target)), draft.layout()) + draft.mark_saved() + file_automation_logger.info("pipelines: saved %s", target) + return str(target) + + # ------------------------------------------------------------------ checks + + def validate(self, definition: Definition) -> list[Problem]: + """Return every problem of the definition, each with the path of the entry it is about. + + Next to what :func:`~automation_file.pipeline.validate_definition` + finds, an action name the registry does not know is a problem too. + """ + document = _document(definition) + tasks = document.get("tasks") + task_ids = tuple(tasks) if isinstance(tasks, Mapping) else () + problems = [Problem.parse(text, task_ids) for text in validate_definition(document)] + registry = self._action_registry() + for task_id, spec in tasks.items() if isinstance(tasks, Mapping) else (): + action = spec.get("action") if isinstance(spec, Mapping) else None + if not (isinstance(action, list) and action and isinstance(action[0], str)): + continue + if action[0] and registry.resolve(action[0]) is None: + problems.append( + Problem( + path=f"tasks.{task_id}.action[0]", + message=f"unknown action {action[0]!r}", + task=task_id, + ) + ) + return problems + + def dry_run( + self, definition: Definition, params: Mapping[str, Any] | None = None + ) -> dict[str, Any]: + """Plan a run without executing, recording or publishing anything. + + Every task comes back ``planned`` in dependency order; a task that would + not run as planned says why in its ``error``. + """ + run = self._pipeline(definition).run(params=params, dry_run=True) + return self._view(run, active=False) + + def test_task( + self, + definition: Definition, + task_id: str, + params: Mapping[str, Any] | None = None, + results: Mapping[str, Any] | None = None, + ) -> dict[str, Any]: + """Execute one task alone and return ``{"run": ..., "events": [...]}``. + + The task's action really runs, with its retry policy and its timeout. + Its upstream tasks do not: each is replaced by a stand-in that returns + the value given for it in ``results`` (``None`` by default), so + ``${tasks..result}`` placeholders have something to read. The test + is recorded in no run store and published on no shared bus; its + condition and its idempotency key are ignored. + """ + document = _document(definition) + tasks = document.get("tasks") + if not isinstance(tasks, Mapping) or task_id not in tasks: + raise AppException(f"the definition has no task {task_id!r}") + own = [problem for problem in self.validate(document) if problem.task == task_id] + if own: + raise PipelineDefinitionException([str(problem) for problem in own]) + spec = tasks[task_id] + upstream = upstream_tasks({key: _listed(item) for key, item in tasks.items()})[task_id] + probe = Pipeline( + str(document.get("name") or DEFAULT_NAME), + params=document.get("params"), + registry=self._registry, + ) + given = dict(results or {}) + for other in upstream: + probe.task(other, _constant(given.get(other))) + probe.task( + task_id, + spec["action"], + depends_on=upstream, + retry=retry_from_dict(spec["retry"]) if "retry" in spec else None, + timeout=spec.get("timeout"), + ) + bus = EventBus() + run = probe.run(params=params, store=MemoryRunStore(), bus=bus) + view = self._view(run, active=False) + view["tasks"] = {task_id: view["tasks"][task_id]} + return {"run": view, "events": _chronological(bus, run.run_id, _DEFAULT_EVENTS)} + + # ------------------------------------------------------------------ runs + + def start( + self, definition: Definition, params: Mapping[str, Any] | None = None + ) -> dict[str, Any]: + """Start a run in the background and return it at once. + + A definition problem raises here, before anything runs. + """ + token = CancellationToken() + run = self._pipeline(definition).start( + params=params, store=self._run_store(), cancel=token, bus=self._bus + ) + self._track(run.run_id, _Tracked(token=token, run=run)) + file_automation_logger.info("pipelines: started %s run %s", run.pipeline, run.run_id) + return self._view(run, active=True) + + def resume(self, run_id: str, definition: Definition) -> dict[str, Any]: + """Continue a stored run in the background: keep what succeeded, run the rest. + + The run keeps its ID and its parameters. What would stop it at once is + raised here: an unknown run, a run of another pipeline, a run this + process is still executing, a definition that cannot run with the + stored parameters. A run that already succeeded is returned as stored. + """ + pipeline = self._pipeline(definition) + store = self._run_store() + earlier = store.get_run(run_id) + if earlier is None: + raise PipelineException(f"unknown run {run_id!r}") + if earlier.pipeline != pipeline.name: + raise PipelineException( + f"run {run_id!r} belongs to pipeline {earlier.pipeline!r}, not {pipeline.name!r}" + ) + if self._is_active(run_id): + raise AppException(f"run {run_id!r} is still executing; cancel it or wait for it") + if earlier.status is RunStatus.SUCCEEDED: + return self._view(earlier, active=False) + plan = pipeline.run(params=earlier.params, dry_run=True) + if plan.status is not RunStatus.SUCCEEDED: + notes = [state.error for state in plan.tasks.values() if state.error] + raise PipelineDefinitionException(notes or [plan.error or "the run cannot be resumed"]) + tracked = _Tracked(token=CancellationToken()) + tracked.thread = threading.Thread( + target=self._resume_in_background, + args=(pipeline, run_id, store, tracked.token), + name=f"pipeline-resume-{pipeline.name}", + ) + self._track(run_id, tracked) + tracked.thread.start() + file_automation_logger.info("pipelines: resuming %s run %s", pipeline.name, run_id) + return self._view(earlier, active=True) + + def retry(self, run_id: str, definition: Definition) -> dict[str, Any]: + """Start a new run with the parameters of the stored run ``run_id``.""" + earlier = self._run_store().get_run(run_id) + if earlier is None: + raise PipelineException(f"unknown run {run_id!r}") + return self.start(definition, earlier.params) + + def cancel(self, run_id: str) -> bool: + """Ask a run this service is executing to stop; return whether there was one.""" + with self._lock: + tracked = self._tracked.get(run_id) + if tracked is None or not tracked.alive: + return False + tracked.token.cancel() + file_automation_logger.info("pipelines: cancel requested for run %s", run_id) + return True + + def wait(self, run_id: str, timeout: float | None = None) -> bool: + """Block until a run this service is executing has ended; return whether it has.""" + with self._lock: + tracked = self._tracked.get(run_id) + if tracked is None: + return True + if tracked.run is not None: + return tracked.run.wait(timeout) + if tracked.thread is not None: + tracked.thread.join(timeout) + return not tracked.alive + + def status(self, run_id: str) -> dict[str, Any]: + """Return the state of the run ``run_id`` and of its tasks.""" + with self._lock: + tracked = self._tracked.get(run_id) + if tracked is not None and tracked.run is not None: + return self._view(tracked.run, active=tracked.alive) + run = self._run_store().get_run(run_id) + if run is None: + raise PipelineException(f"unknown run {run_id!r}") + return self._view(run, active=tracked is not None and tracked.alive) + + def history( + self, pipeline: str | None = None, limit: int = _DEFAULT_HISTORY + ) -> list[dict[str, Any]]: + """Return the latest recorded runs, newest first, of one pipeline or of all.""" + return [ + self._view(run, active=self._is_active(run.run_id)) + for run in self._run_store().list_runs(pipeline or None, limit) + ] + + def running(self) -> list[dict[str, Any]]: + """Return the runs that have not ended: those of this process and those the store says.""" + with self._lock: + tracked = {run_id: item for run_id, item in self._tracked.items() if item.alive} + views = { + run_id: self._view(item.run, active=True) + for run_id, item in tracked.items() + if item.run is not None + } + for run in self._run_store().list_runs(None, _SCAN_LIMIT): + if run.status is RunStatus.RUNNING or run.run_id in tracked: + views.setdefault(run.run_id, self._view(run, active=run.run_id in tracked)) + return sorted(views.values(), key=lambda view: view["started_at"] or "", reverse=True) + + def events(self, run_id: str, limit: int = _DEFAULT_EVENTS) -> list[dict[str, Any]]: + """Return the events of the run ``run_id`` the bus still remembers, oldest first.""" + return _chronological(event_bus if self._bus is None else self._bus, run_id, limit) + + def follow(self, run_id: str, limit: int = _DEFAULT_EVENTS) -> dict[str, Any]: + """Return ``{"run": status, "events": [...]}``: one call for a view that follows a run.""" + return {"run": self.status(run_id), "events": self.events(run_id, limit)} + + # ------------------------------------------------------------------ internals + + def _action_registry(self) -> ActionRegistry: + if self._registry is not None: + return self._registry + from automation_file.core.action_executor import executor + + return executor.registry + + def _run_store(self) -> RunStore: + return default_run_store() if self._store is None else self._store + + def _pipeline(self, definition: Definition) -> Pipeline: + return Pipeline.from_dict(_document(definition), registry=self._registry) + + def _is_active(self, run_id: str) -> bool: + with self._lock: + tracked = self._tracked.get(run_id) + return tracked is not None and tracked.alive + + def _track(self, run_id: str, tracked: _Tracked) -> None: + with self._lock: + self._tracked[run_id] = tracked + ended = [key for key, item in self._tracked.items() if not item.alive] + for key in ended[: max(len(self._tracked) - _MAX_TRACKED, 0)]: + del self._tracked[key] + + def _resume_in_background( + self, pipeline: Pipeline, run_id: str, store: RunStore, token: CancellationToken + ) -> None: + try: + pipeline.resume(run_id, store=store, cancel=token, bus=self._bus) + except Exception as error: # pylint: disable=broad-except + # Boundary of the background thread: there is no caller left to raise to. + file_automation_logger.error( + "pipelines: resuming %s run %s ended with %r", pipeline.name, run_id, error + ) + + @staticmethod + def _view(run: PipelineRun, *, active: bool) -> dict[str, Any]: + view: dict[str, Any] = mask_secrets(run.to_dict()) + view["active"] = active + return view + + +def _document(definition: Definition) -> Mapping[str, Any]: + if isinstance(definition, PipelineDraft): + return definition.to_definition() + if isinstance(definition, Mapping): + return definition + raise AppException( + f"expected a PipelineDraft or a definition mapping, got {type(definition).__name__}" + ) + + +def _chronological(bus: EventBus, run_id: str, limit: int) -> list[dict[str, Any]]: + recent = bus.recent(limit, correlation_id=run_id) + return [mask_secrets(event.to_dict()) for event in reversed(recent)] + + +def _write_yaml(target: Path, definition: Mapping[str, Any]) -> None: + import yaml + + try: + text = yaml.safe_dump(dict(definition), sort_keys=False, allow_unicode=True) + with open(target, "w", encoding="utf-8") as handle: + handle.write(text) + except (OSError, yaml.YAMLError) as error: + raise AppException(f"cannot write {target}: {error}") from error diff --git a/automation_file/app/scheduler_service.py b/automation_file/app/scheduler_service.py new file mode 100644 index 0000000..c12e30e --- /dev/null +++ b/automation_file/app/scheduler_service.py @@ -0,0 +1,67 @@ +"""The Scheduler service: list, add and remove cron jobs. + +.. code-block:: python + + from automation_file.app import app_services + + scheduler = app_services().scheduler + scheduler.add("tick", "*/5 * * * *", [["FA_create_file", {"file_path": "tick.txt"}]]) + scheduler.jobs() # [{"name": "tick", "cron": "*/5 * * * *", ...}] + scheduler.remove("tick") + +It calls the four scheduler functions that are public (``schedule_add``, +``schedule_remove``, ``schedule_remove_all``, ``schedule_list``) and nothing +else of the scheduler, so the scheduler's internals can change underneath it. +""" + +from __future__ import annotations + +from typing import Any + +from automation_file.app.arguments import parse_json_text +from automation_file.app.errors import AppException +from automation_file.scheduler import ( + schedule_add, + schedule_list, + schedule_remove, + schedule_remove_all, +) + + +def _action_list(actions: list[Any] | str) -> list[Any]: + """Return the action list, read from JSON text when a form supplied text.""" + parsed = parse_json_text(actions, "the action list") if isinstance(actions, str) else actions + if not isinstance(parsed, list) or not parsed: + raise AppException("the action list must be a non-empty JSON array of actions") + return parsed + + +class SchedulerService: + """Cron jobs of the process-wide scheduler.""" + + def jobs(self) -> list[dict[str, Any]]: + """Return a snapshot of every registered job.""" + return [dict(job) for job in schedule_list()] + + def add( + self, name: str, cron: str, actions: list[Any] | str, allow_overlap: bool = False + ) -> dict[str, Any]: + """Register the job ``name``: run ``actions`` whenever ``cron`` matches. + + ``actions`` is an action list or its JSON text. With ``allow_overlap`` + a tick fires even while the previous run of the job is still going. + """ + if not name.strip() or not cron.strip(): + raise AppException("a scheduled job needs a name and a cron expression") + action_list = _action_list(actions) + if allow_overlap: + return dict(schedule_add(name.strip(), cron.strip(), action_list, allow_overlap=True)) + return dict(schedule_add(name.strip(), cron.strip(), action_list)) + + def remove(self, name: str) -> dict[str, Any]: + """Remove the job ``name`` and return its last snapshot.""" + return dict(schedule_remove(name)) + + def remove_all(self) -> list[dict[str, Any]]: + """Remove every job and return their last snapshots.""" + return [dict(job) for job in schedule_remove_all()] diff --git a/automation_file/app/services.py b/automation_file/app/services.py new file mode 100644 index 0000000..56a0db2 --- /dev/null +++ b/automation_file/app/services.py @@ -0,0 +1,136 @@ +"""The set of services a user interface works with, and the navigation they stand for. + +.. code-block:: python + + from automation_file.app import NAVIGATION, app_services + + services = app_services() # the shared set, on the process-wide singletons + NAVIGATION # ("Dashboard", "Files", "Storage", ...) + services.dashboard.summary() + services.files.list_dir("local:///data") + +:func:`app_services` returns one set per process, so the desktop window and the +web page of one process show the same runs, monitors and routes. +:func:`build_services` makes a set of its own, on the stores and buses it is +given: tests and embedded uses want that. +""" + +from __future__ import annotations + +import threading +from dataclasses import dataclass +from typing import TYPE_CHECKING + +from automation_file.app.audit_service import AuditService +from automation_file.app.dashboard_service import DashboardService, DashboardSources +from automation_file.app.file_service import FileService +from automation_file.app.integrity_service import IntegrityService +from automation_file.app.notification_service import NotificationService +from automation_file.app.pipeline_service import PipelineService +from automation_file.app.scheduler_service import SchedulerService +from automation_file.app.settings_service import SettingsService +from automation_file.app.storage_service import StorageService +from automation_file.audit import AuditTrail +from automation_file.events import EventBus +from automation_file.notify import NotificationManager, NotificationRouter +from automation_file.pipeline import RunStore +from automation_file.storage import StorageResolver + +if TYPE_CHECKING: + from automation_file.core.action_registry import ActionRegistry + +#: The navigation entries of every user interface, in display order. +NAVIGATION: tuple[str, ...] = ( + "Dashboard", + "Files", + "Storage", + "Pipelines", + "Scheduler", + "Integrity", + "Audit", + "Notifications", + "Settings", +) + + +@dataclass(frozen=True) +class AppServices: + """One service per navigation entry.""" + + dashboard: DashboardService + files: FileService + storage: StorageService + pipelines: PipelineService + scheduler: SchedulerService + integrity: IntegrityService + audit: AuditService + notifications: NotificationService + settings: SettingsService + + +@dataclass(frozen=True) +class ServiceOptions: + """What a set of services is built on; ``None`` means the process-wide singleton.""" + + resolver: StorageResolver | None = None + run_store: RunStore | None = None + registry: ActionRegistry | None = None + bus: EventBus | None = None + audit_trail: AuditTrail | None = None + notification_manager: NotificationManager | None = None + notification_router: NotificationRouter | None = None + + +def build_services(options: ServiceOptions | None = None) -> AppServices: + """Build a set of services on ``options`` (the process-wide singletons by default).""" + chosen = ServiceOptions() if options is None else options + files = FileService(chosen.resolver) + storage = StorageService(chosen.resolver) + pipelines = PipelineService(chosen.run_store, registry=chosen.registry, bus=chosen.bus) + scheduler = SchedulerService() + integrity = IntegrityService() + audit = AuditService(chosen.audit_trail) + notifications = NotificationService(chosen.notification_manager, chosen.notification_router) + settings = SettingsService(chosen.notification_manager, chosen.notification_router) + dashboard = DashboardService( + DashboardSources( + pipelines=pipelines, + integrity=integrity, + storage=storage, + scheduler=scheduler, + audit=audit, + notifications=notifications, + ), + bus=chosen.bus, + ) + return AppServices( + dashboard=dashboard, + files=files, + storage=storage, + pipelines=pipelines, + scheduler=scheduler, + integrity=integrity, + audit=audit, + notifications=notifications, + settings=settings, + ) + + +_shared: dict[str, AppServices] = {} +_shared_lock = threading.Lock() +_SHARED_KEY = "services" + + +def app_services() -> AppServices: + """Return the process-wide set of services, built on first use.""" + with _shared_lock: + services = _shared.get(_SHARED_KEY) + if services is None: + services = _shared[_SHARED_KEY] = build_services() + return services + + +def reset_app_services() -> None: + """Forget the process-wide set, so the next :func:`app_services` builds a new one.""" + with _shared_lock: + _shared.clear() diff --git a/automation_file/app/settings_service.py b/automation_file/app/settings_service.py new file mode 100644 index 0000000..f23b7eb --- /dev/null +++ b/automation_file/app/settings_service.py @@ -0,0 +1,182 @@ +"""The Settings service: the configuration file, the extras, the environment. + +.. code-block:: python + + from automation_file.app import app_services + + settings = app_services().settings + settings.load("automation_file.toml") # a summary, secrets masked; nothing changes + settings.apply("automation_file.toml") # registers its sinks and routes + for extra in settings.extras(): + print(extra.name, extra.installed, extra.install_hint) + +``load`` shows what a file would do; ``apply`` does it. Both resolve the +``${env:...}`` and ``${file:...}`` references of the file, and neither returns a +resolved secret: the summary is masked before it leaves the service. +""" + +from __future__ import annotations + +import importlib.metadata +import os +import platform +from dataclasses import asdict, dataclass +from pathlib import Path +from typing import Any + +from automation_file.app.masking import mask_secrets +from automation_file.app.storage_service import is_installed +from automation_file.core.config import AutomationConfig +from automation_file.core.optional import EXTRAS, install_hint +from automation_file.logging_config import default_log_file +from automation_file.notify import ( + NotificationManager, + NotificationRouter, + notification_manager, + notification_router, +) + +#: Extra name -> the modules it installs; an extra that needs nothing has none. +EXTRA_MODULES: dict[str, tuple[str, ...]] = { + "s3": ("boto3",), + "azure": ("azure.storage.blob",), + "gdrive": ("googleapiclient", "google_auth_oauthlib", "google_auth_httplib2"), + "dropbox": ("dropbox",), + "sftp": ("paramiko",), + "ftp": (), + "webdav": (), + "smb": ("smbclient",), + "fsspec": ("fsspec",), + "onedrive": ("msal",), + "box": ("box_sdk_gen",), + "parquet": ("pyarrow",), + "gui": ("PySide6",), +} +_DISTRIBUTIONS = ("automation_file", "automation_file_dev") +_UNKNOWN = "unknown" + + +@dataclass(frozen=True) +class ExtraStatus: + """Whether one optional extra is installed. + + ``installed`` is ``None`` for an extra this table does not know the modules + of. ``install_hint`` is the ``pip install`` command, set when it is missing. + """ + + name: str + feature: str + installed: bool | None + modules: tuple[str, ...] = () + missing: tuple[str, ...] = () + install_hint: str | None = None + + def to_dict(self) -> dict[str, Any]: + """Return a JSON-serialisable mapping of the status.""" + described = asdict(self) + described["modules"] = list(self.modules) + described["missing"] = list(self.missing) + return described + + +def _extra_status(name: str, feature: str) -> ExtraStatus: + modules = EXTRA_MODULES.get(name) + if modules is None: + return ExtraStatus(name=name, feature=feature, installed=None) + missing = tuple(module for module in modules if not is_installed(module)) + return ExtraStatus( + name=name, + feature=feature, + installed=not missing, + modules=modules, + missing=missing, + install_hint=install_hint(name) if missing else None, + ) + + +def _package_version() -> str: + for distribution in _DISTRIBUTIONS: + try: + return importlib.metadata.version(distribution) + except importlib.metadata.PackageNotFoundError: + continue + return _UNKNOWN + + +def _sink_tables(config: AutomationConfig) -> list[dict[str, Any]]: + entries = config.section("notify").get("sinks") or [] + tables = ( + [entry for entry in entries if isinstance(entry, dict)] if isinstance(entries, list) else [] + ) + return [ + {"name": str(entry.get("name") or entry.get("type") or ""), "type": entry.get("type")} + for entry in tables + ] + + +class SettingsService: + """Load and apply ``automation_file.toml``; report the extras and the environment.""" + + def __init__( + self, + manager: NotificationManager | None = None, + router: NotificationRouter | None = None, + ) -> None: + self._manager = notification_manager if manager is None else manager + self._router = notification_router if router is None else router + self._applied: dict[str, Any] | None = None + + def extras(self) -> list[ExtraStatus]: + """Return the status of every optional extra, in the order the package lists them.""" + return [_extra_status(name, feature) for name, feature in EXTRAS.items()] + + def environment(self) -> dict[str, Any]: + """Return the versions and locations a user needs when reporting a problem.""" + return { + "version": _package_version(), + "python": platform.python_version(), + "platform": platform.platform(), + "log_file": str(default_log_file()), + "applied_config": None if self._applied is None else self._applied["source"], + } + + def load(self, path: str | os.PathLike[str]) -> dict[str, Any]: + """Read a configuration file and return its summary. Nothing is registered. + + The summary names the file, its sections, the sinks and the routes it + declares, and holds the document with every secret masked. A file that + is missing, malformed or names an unknown sink raises ``ConfigException``. + """ + return self._summary(AutomationConfig.load(Path(path)), applied=False) + + def apply(self, path: str | os.PathLike[str]) -> dict[str, Any]: + """Read a configuration file and register its sinks and its routes. + + Sinks already registered stay; one with the same name is replaced. The + routes the file declared before and no longer does are removed. Returns + the summary of :meth:`load` with ``applied`` set and the number of + sinks registered. + """ + config = AutomationConfig.load(Path(path)) + registered = config.apply_to(self._manager, self._router) + summary = self._summary(config, applied=True) + summary["registered_sinks"] = registered + self._applied = summary + return summary + + def applied(self) -> dict[str, Any] | None: + """Return the summary of the configuration applied last, or ``None``.""" + return None if self._applied is None else dict(self._applied) + + def _summary(self, config: AutomationConfig, *, applied: bool) -> dict[str, Any]: + document = config.raw + routes = config.notification_routes(known_sinks=self._manager.names()) + return { + "source": None if config.source is None else str(config.source), + "applied": applied, + "sections": sorted(document), + "sinks": _sink_tables(config), + "routes": [route.to_dict() for route in routes], + "defaults": mask_secrets(config.section("defaults")), + "document": mask_secrets(document), + } diff --git a/automation_file/app/storage_service.py b/automation_file/app/storage_service.py new file mode 100644 index 0000000..99521f0 --- /dev/null +++ b/automation_file/app/storage_service.py @@ -0,0 +1,346 @@ +"""The Storage service: which backends are there, and whether each one can be used. + +.. code-block:: python + + from automation_file.app import app_services + + storage = app_services().storage + for backend in storage.backends(): + print(backend.name, backend.usable, backend.detail) + storage.mount_local("sandbox://jobs", "/srv/jobs") + +A backend is *usable* when its client is ready: the local and in-memory +backends always are, a cloud backend once its shared client has been +initialised. When the package its extra installs is missing, the status says +so and carries the ``pip install`` command. Nothing here opens a connection. +""" + +from __future__ import annotations + +import importlib +import importlib.util +import os +from collections.abc import Callable +from dataclasses import asdict, dataclass +from typing import Any + +from automation_file.app.errors import AppException +from automation_file.core.optional import install_hint +from automation_file.exceptions import FileAutomationException +from automation_file.storage import ( + LocalStorage, + StorageBackend, + StorageResolver, + StorageURI, + default_resolver, + parse_storage_uri, +) + +KIND_SCHEME = "scheme" +KIND_MOUNT = "mount" +KIND_CLIENT = "client" + +_READY = "ready" +_MOUNTED = "mounted" +_NO_FACTORY_INFO = "registered by the application" + + +@dataclass(frozen=True) +class _Client: + """One shared client singleton: where it lives and how to ask whether it is ready.""" + + label: str + module: str + attribute: str + probe: str + extra: str | None + sdk: str | None + init_hint: str + + +_CLIENTS: dict[str, _Client] = { + "s3": _Client( + "Amazon S3", + "automation_file.remote.s3.client", + "s3_instance", + "require_client", + "s3", + "boto3", + "call s3_instance.later_init() or FA_s3_later_init", + ), + "azure": _Client( + "Azure Blob", + "automation_file.remote.azure_blob.client", + "azure_blob_instance", + "require_service", + "azure", + "azure.storage.blob", + "call azure_blob_instance.later_init() or FA_azure_blob_later_init", + ), + "gdrive": _Client( + "Google Drive", + "automation_file.remote.google_drive.client", + "driver_instance", + "require_service", + "gdrive", + "googleapiclient", + "call driver_instance.later_init(token_path, credentials_path)", + ), + "dropbox": _Client( + "Dropbox", + "automation_file.remote.dropbox_api.client", + "dropbox_instance", + "require_client", + "dropbox", + "dropbox", + "call dropbox_instance.later_init(token) or FA_dropbox_later_init", + ), + "onedrive": _Client( + "OneDrive", + "automation_file.remote.onedrive.client", + "onedrive_instance", + "require_session", + "onedrive", + "msal", + "call onedrive_instance.later_init(access_token) or device_code_login()", + ), + "sftp": _Client( + "SFTP", + "automation_file.remote.sftp.client", + "sftp_instance", + "require_sftp", + "sftp", + "paramiko", + "call sftp_instance.later_init(...) or FA_sftp_later_init", + ), + "ftp": _Client( + "FTP / FTPS", + "automation_file.remote.ftp.client", + "ftp_instance", + "require_ftp", + "ftp", + None, + "call ftp_instance.later_init(...) or FA_ftp_later_init", + ), + "box": _Client( + "Box", + "automation_file.remote.box.client", + "box_instance", + "require_client", + "box", + "box_sdk_gen", + "call box_instance.later_init(...) or FA_box_later_init", + ), +} +#: Storage scheme -> the client that serves it. ``box`` has actions but no storage scheme. +_SCHEME_CLIENTS = { + "s3": "s3", + "azure": "azure", + "gdrive": "gdrive", + "dropbox": "dropbox", + "onedrive": "onedrive", + "sftp": "sftp", + "ftp": "ftp", + "ftps": "ftp", +} +_ALWAYS_READY = {"local": "Local filesystem", "memory": "In-memory store"} +_CLIENTS_WITHOUT_SCHEME = tuple( + name for name in _CLIENTS if name not in frozenset(_SCHEME_CLIENTS.values()) +) + + +@dataclass(frozen=True) +class BackendStatus: + """Whether one backend can be used, and what is missing when it cannot. + + ``kind`` is ``"scheme"`` for a URI scheme with a factory, ``"mount"`` for a + backend mounted at a URI and ``"client"`` for a shared client that has + ``FA_*`` actions but no storage scheme. ``install_hint`` is set only when + the package of the backend's extra is not installed. + """ + + name: str + kind: str + label: str + backend: str = "" + extra: str | None = None + installed: bool = True + ready: bool = True + usable: bool = True + detail: str = _READY + install_hint: str | None = None + + def to_dict(self) -> dict[str, Any]: + """Return a JSON-serialisable mapping of the status.""" + return asdict(self) + + +@dataclass(frozen=True) +class MountInfo: + """One mount of the resolver: the URI it serves and the backend behind it.""" + + uri: str + scheme: str + backend: str + + def to_dict(self) -> dict[str, str]: + """Return a JSON-serialisable mapping of the mount.""" + return asdict(self) + + +def is_installed(module: str | None) -> bool: + """Return whether ``module`` can be imported, without importing it.""" + if module is None: + return True + try: + return importlib.util.find_spec(module) is not None + except (ImportError, ValueError): + # A missing parent package, or a module that is half-imported. + return False + + +def _client_ready(client: _Client) -> bool: + """Ask the shared client whether it has been initialised. Never opens a connection.""" + try: + instance = getattr(importlib.import_module(client.module), client.attribute) + probe: Callable[[], Any] = getattr(instance, client.probe) + probe() + except (FileAutomationException, RuntimeError, ImportError, AttributeError): + return False + return True + + +def _client_status(name: str, kind: str, client: _Client) -> BackendStatus: + installed = is_installed(client.sdk) + ready = _client_ready(client) + hint = None if installed or client.extra is None else install_hint(client.extra) + if ready: + detail = _READY + elif hint is not None: + detail = f"{client.sdk} is not installed: {hint}" + else: + detail = f"not initialised: {client.init_hint}" + return BackendStatus( + name=name, + kind=kind, + label=client.label, + extra=client.extra, + installed=installed, + ready=ready, + usable=ready, + detail=detail, + install_hint=hint, + ) + + +def _mount_uri(key: tuple[str, str, str]) -> str: + scheme, authority, path = key + return str(StorageURI(scheme, authority, path)) + + +class StorageService: + """Schemes, mounts and backend status of one :class:`StorageResolver`.""" + + def __init__(self, resolver: StorageResolver | None = None) -> None: + self._resolver = default_resolver if resolver is None else resolver + + @property + def resolver(self) -> StorageResolver: + """The resolver this service reads and changes.""" + return self._resolver + + def schemes(self) -> list[str]: + """Return every scheme that has a factory or a mount, sorted.""" + return self._resolver.schemes() + + def mounts(self) -> list[MountInfo]: + """Return the mounts, sorted by URI.""" + return sorted( + ( + MountInfo(uri=_mount_uri(key), scheme=key[0], backend=type(backend).__name__) + for key, backend in self._mount_table().items() + ), + key=lambda mount: mount.uri, + ) + + def backends(self) -> list[BackendStatus]: + """Return one status per scheme, per mount and per shared client without a scheme. + + A scheme that exists only because something is mounted under it is + listed through its mounts. + """ + mounted = self.mounts() + factories = self._factory_schemes() + statuses = [self._scheme_status(scheme) for scheme in self.schemes() if scheme in factories] + statuses.extend( + BackendStatus( + name=mount.uri, + kind=KIND_MOUNT, + label=f"Mount of {mount.backend}", + backend=mount.backend, + detail=_MOUNTED, + ) + for mount in mounted + ) + statuses.extend( + _client_status(name, KIND_CLIENT, _CLIENTS[name]) for name in _CLIENTS_WITHOUT_SCHEME + ) + return statuses + + def capabilities(self, uri: str) -> dict[str, bool]: + """Return what the backend serving ``uri`` provides beyond the mandatory contract.""" + return self._resolver.capabilities(uri).to_dict() + + def mount_local(self, uri: str, root: str | os.PathLike[str]) -> MountInfo: + """Serve ``uri`` from the local directory ``root``, confined to that directory. + + This is how paths that come from outside the process get a sandbox: + ``mount_local("sandbox://jobs", "/srv/jobs")``. The directory must exist. + """ + if not os.path.isdir(root): + raise AppException(f"cannot mount {os.fspath(root)!r}: it is not a directory") + parsed = parse_storage_uri(uri) + self._resolver.mount(parsed, LocalStorage(root)) + return MountInfo(uri=str(parsed), scheme=parsed.scheme, backend=LocalStorage.__name__) + + def mount(self, uri: str, backend: StorageBackend) -> MountInfo: + """Serve ``uri`` and everything below it from ``backend``.""" + parsed = parse_storage_uri(uri) + try: + self._resolver.mount(parsed, backend) + except TypeError as error: + raise AppException(str(error)) from error + return MountInfo(uri=str(parsed), scheme=parsed.scheme, backend=type(backend).__name__) + + def unmount(self, uri: str) -> bool: + """Remove the mount at exactly ``uri``; return whether there was one.""" + return self._resolver.unmount(uri) + + def _table(self, name: str) -> dict[Any, Any] | None: + # StorageResolver lists its schemes but neither its mounts nor which schemes have a + # factory, so its two tables are read here, in one place, under its own lock. + table = getattr(self._resolver, name, None) + lock = getattr(self._resolver, "_lock", None) + if not isinstance(table, dict) or lock is None: + return None + with lock: + return dict(table) + + def _mount_table(self) -> dict[tuple[str, str, str], StorageBackend]: + return self._table("_mounts") or {} + + def _factory_schemes(self) -> frozenset[str]: + """Return the schemes served by a factory; every scheme when that cannot be told.""" + factories = self._table("_factories") + return frozenset(self.schemes() if factories is None else factories) + + @staticmethod + def _scheme_status(scheme: str) -> BackendStatus: + if scheme in _ALWAYS_READY: + return BackendStatus(name=scheme, kind=KIND_SCHEME, label=_ALWAYS_READY[scheme]) + client = _CLIENTS.get(_SCHEME_CLIENTS.get(scheme, "")) + if client is None: + return BackendStatus( + name=scheme, kind=KIND_SCHEME, label=scheme, detail=_NO_FACTORY_INFO + ) + return _client_status(scheme, KIND_SCHEME, client) diff --git a/automation_file/server/web_ui.py b/automation_file/server/web_ui.py index 806ae0f..2ba8243 100644 --- a/automation_file/server/web_ui.py +++ b/automation_file/server/web_ui.py @@ -1,15 +1,20 @@ """Read-only observability Web UI (stdlib + HTMX). -Serves a single HTML page that polls three HTML fragments — registered -actions, live progress, and health summary — using HTMX (loaded from a -pinned CDN URL). Write operations are deliberately out of scope; trigger -actions through :mod:`http_server` / :mod:`tcp_server` with their auth -story intact. +Serves a single HTML page that polls HTML fragments using HTMX (loaded from a +pinned CDN URL). Every fragment but the transfer progress is rendered from the +application layer (:mod:`automation_file.app`), the same services the PySide6 +window calls, so the two show the same health, runs, integrity drift, events, +storage status and audit records. Write operations are deliberately out of +scope; trigger actions through :mod:`http_server` / :mod:`tcp_server` with +their auth story intact. Loopback-only by default; ``allow_non_loopback=True`` is required to bind elsewhere. When ``shared_secret`` is supplied every request must carry ``Authorization: Bearer `` — the rendered HTML includes a ``hx-headers`` attribute so HTMX's polled requests carry the token. + +Everything a fragment shows is escaped, and the application layer has already +masked tokens, passwords and webhook URLs in it. """ from __future__ import annotations @@ -18,12 +23,14 @@ import html as html_lib import json import threading -import time +from collections.abc import Callable, Iterable, Sequence from http import HTTPStatus from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer +from typing import Any -from automation_file.core.action_executor import executor +from automation_file.app import NAVIGATION, AppServices, app_services from automation_file.core.progress import progress_registry +from automation_file.exceptions import FileAutomationException from automation_file.logging_config import file_automation_logger from automation_file.server.network_guards import ensure_loopback @@ -31,6 +38,12 @@ _DEFAULT_PORT = 9955 _HTMX_CDN = "https://unpkg.com/htmx.org@1.9.12/dist/htmx.min.js" _HTMX_SRI = "sha384-ujb1lZYygJmzgSwoxRggbCHcjc0rB2XoQrxeTUQyRjrOnlCoYta87iKBWq3EsdM2" +_EVENT_LIMIT = 30 +_RUN_LIMIT = 15 +_AUDIT_LIMIT = 30 +_RUN_ID_LENGTH = 8 +_AUDIT_KEYS = ("timestamp", "actor", "source", "action", "resource", "status", "error") +_EMPTY = "—" _INDEX_TEMPLATE = """ @@ -46,18 +59,48 @@ th, td {{ padding: 0.35rem 0.6rem; border-bottom: 1px solid #eee; text-align: left; }} th {{ background: #f5f5f5; }} .muted {{ color: #888; }} + .ok {{ color: #2f8f3f; font-weight: bold; }} + .attention {{ color: #b3261e; font-weight: bold; }} + nav {{ font-size: 0.9rem; margin: 0.6rem 0 0; }} + nav span {{ margin-right: 0.9rem; color: #555; }} code {{ background: #f3f3f3; padding: 0.1rem 0.3rem; border-radius: 3px; }}

automation_file

Read-only dashboard. Write operations live on the action server.

+

Health

loading…
+

Pipeline runs

+
+ loading… +
+ +

Integrity

+
+ loading… +
+ +

Recent events

+
+ loading… +
+ +

Storage

+
+ loading… +
+ +

Audit

+
+ loading… +
+

Progress

loading… @@ -72,8 +115,217 @@ """ +def _text(value: object) -> str: + """Return ``value`` as escaped HTML text; nothing becomes a dash.""" + if value is None or value == "": + return _EMPTY + if isinstance(value, bool): + return "yes" if value else "no" + if isinstance(value, (list, tuple)): + return html_lib.escape(", ".join(str(item) for item in value), quote=True) or _EMPTY + return html_lib.escape(str(value), quote=True) + + +def _muted(message: str) -> str: + return f"

{html_lib.escape(message, quote=True)}

" + + +def _table(headers: Sequence[str], rows: Iterable[Sequence[object]]) -> str: + """Render a table; every header and every cell is escaped here.""" + head = "".join(f"{_text(header)}" for header in headers) + body = "".join( + "" + "".join(f"{_text(cell)}" for cell in row) + "" for row in rows + ) + return f"{head}{body}
" + + +def _pairs(rows: Iterable[tuple[str, object]]) -> str: + """Render a two-column table of names and values.""" + body = "".join( + f"{_text(name)}{_text(value)}" for name, value in rows + ) + return f"{body}
" + + +def _audit_state(audit: dict[str, Any]) -> str: + if not audit.get("configured"): + return "not configured" + return "recording" if audit.get("active") else "configured, not recording" + + +def _render_health(services: AppServices) -> str: + summary = services.dashboard.summary(events=_EVENT_LIMIT, runs=_RUN_LIMIT) + health = summary.health + status_class = "ok" if summary.status == "ok" else "attention" + rows: list[tuple[str, object]] = [ + ("process", health.get("process", "alive")), + ("registry size", health.get("registry_size")), + ("running runs", health.get("running_runs")), + ("scheduled jobs", health.get("scheduler_jobs")), + ("integrity monitors", health.get("integrity_monitors")), + ("notification sinks", health.get("notification_sinks")), + ("notification routes", health.get("notification_routes")), + ("audit", _audit_state(health.get("audit") or {})), + ("time", health.get("time")), + ] + reasons = "".join(f"
  • {_text(reason)}
    • " for reason in summary.reasons) + return ( + f"

    status: {_text(summary.status)}

    " + + (f"
      {reasons}
    " if reasons else "") + + _pairs(rows) + ) + + +def _run_row(run: dict[str, Any]) -> list[object]: + statuses = run.get("task_statuses") or {} + return [ + str(run.get("run_id") or "")[:_RUN_ID_LENGTH], + run.get("pipeline"), + run.get("status"), + run.get("started_at"), + run.get("finished_at"), + ", ".join(f"{count} {status}" for status, count in statuses.items()), + run.get("error"), + ] + + +def _render_runs(services: AppServices) -> str: + data = services.dashboard.runs(_RUN_LIMIT) + runs = [*data["running"], *data["recent"]] + counts = ", ".join(f"{count} {status}" for status, count in data["counts"].items()) + if not runs: + return _muted("no pipeline runs recorded") + headers = ("run", "pipeline", "status", "started", "finished", "tasks", "error") + return f"

    {_text(counts)}

    " + _table(headers, map(_run_row, runs)) + + +def _drift(monitor: dict[str, Any]) -> str: + if monitor.get("ok") is None: + return "not verified yet" + return "none" if monitor["ok"] else f"{monitor.get('changes', 0)} change(s)" + + +def _render_integrity(services: AppServices) -> str: + monitors = services.dashboard.integrity() + if not monitors: + return _muted("no integrity monitor is running") + headers = ("monitor", "target", "running", "drift", "last run", "error") + return _table( + headers, + ( + [ + monitor.get("name"), + monitor.get("target"), + bool(monitor.get("running")), + _drift(monitor), + monitor.get("last_run"), + monitor.get("last_error"), + ] + for monitor in monitors + ), + ) + + +def _render_events(services: AppServices) -> str: + events = services.dashboard.recent_events(_EVENT_LIMIT) + if not events: + return _muted("no events yet") + headers = ("time", "severity", "type", "source", "subject", "error") + return _table( + headers, + ( + [ + event.get("timestamp"), + event.get("severity"), + event.get("type"), + event.get("source"), + event.get("subject"), + (event.get("payload") or {}).get("error"), + ] + for event in events + ), + ) + + +def _render_storage(services: AppServices) -> str: + headers = ("backend", "kind", "usable", "detail") + return _table( + headers, + ( + [ + backend.get("name"), + backend.get("kind"), + bool(backend.get("usable")), + backend.get("detail"), + ] + for backend in services.dashboard.storage_status() + ), + ) + + +def _render_audit(services: AppServices) -> str: + if not services.audit.is_configured(): + return _muted("audit is not configured; call configure_audit to start recording") + records = services.audit.recent(_AUDIT_LIMIT) + if not records: + return _muted("no audit records yet") + headers = ("time", "actor", "source", "action", "resource", "status", "error") + return _table(headers, ([record.get(key) for key in _AUDIT_KEYS] for record in records)) + + +def _render_progress(_services: AppServices) -> str: + snapshots = progress_registry.list() + if not snapshots: + return "

    no active transfers

    " + rows = [] + for item in snapshots: + name = html_lib.escape(str(item.get("name", "")), quote=True) + status = html_lib.escape(str(item.get("status", "")), quote=True) + transferred = int(item.get("transferred", 0) or 0) + total = item.get("total") + total_cell = "—" if total in (None, 0) else html_lib.escape(str(total), quote=True) + pct = "" + if isinstance(total, int) and total > 0: + pct = f" ({(transferred / total) * 100:.1f}%)" + rows.append( + "" + f"{name}" + f"{status}" + f"{transferred}{pct}" + f"{total_cell}" + "" + ) + return ( + "" + "" + + "".join(rows) + + "
    namestatustransferredtotal
    " + ) + + +def _render_registry(services: AppServices) -> str: + names = services.pipelines.action_names() + if not names: + return "

    registry empty

    " + items = "".join(f"
  • {html_lib.escape(name, quote=True)}
    • " for name in names) + return f"
      {items}
    " + + +#: Fragment path -> the function that renders it from the application services. +_FRAGMENTS: dict[str, Callable[[AppServices], str]] = { + "/ui/health": _render_health, + "/ui/runs": _render_runs, + "/ui/integrity": _render_integrity, + "/ui/events": _render_events, + "/ui/storage": _render_storage, + "/ui/audit": _render_audit, + "/ui/progress": _render_progress, + "/ui/registry": _render_registry, +} + + class _WebUIHandler(BaseHTTPRequestHandler): - """Serves the dashboard page plus its three HTMX fragment endpoints.""" + """Serves the dashboard page plus its HTMX fragment endpoints.""" def log_message( # pylint: disable=arguments-differ self, format_str: str, *args: object @@ -88,16 +340,20 @@ def do_GET(self) -> None: # pylint: disable=invalid-name if path in ("/", "/index.html"): self._send_html(HTTPStatus.OK, self._render_index()) return - if path == "/ui/health": - self._send_html(HTTPStatus.OK, _render_health()) + render = _FRAGMENTS.get(path) + if render is None: + self._send_html(HTTPStatus.NOT_FOUND, "

    not found

    ") return - if path == "/ui/progress": - self._send_html(HTTPStatus.OK, _render_progress()) - return - if path == "/ui/registry": - self._send_html(HTTPStatus.OK, _render_registry()) - return - self._send_html(HTTPStatus.NOT_FOUND, "

    not found

    ") + self._send_html(HTTPStatus.OK, self._fragment(path, render)) + + def _fragment(self, path: str, render: Callable[[AppServices], str]) -> str: + """Render one fragment; a service that cannot answer becomes a note, not a broken page.""" + services: AppServices = getattr(self.server, "services", None) or app_services() + try: + return render(services) + except FileAutomationException as error: + file_automation_logger.warning("web_ui: %s cannot be rendered: %r", path, error) + return _muted(f"unavailable: {type(error).__name__}") def _authorized(self) -> bool: secret: str | None = getattr(self.server, "shared_secret", None) @@ -121,73 +377,32 @@ def _render_index(self) -> str: secret: str | None = getattr(self.server, "shared_secret", None) auth_headers_obj = {"Authorization": f"Bearer {secret}"} if secret else {} auth_headers = html_lib.escape(json.dumps(auth_headers_obj), quote=True) + navigation = "".join(f"{_text(name)}" for name in NAVIGATION) return _INDEX_TEMPLATE.format( htmx_src=_HTMX_CDN, htmx_sri=_HTMX_SRI, auth_headers=auth_headers, + navigation=navigation, ) -def _render_health() -> str: - names = list(executor.registry.event_dict.keys()) - return ( - "" - "" - f"" - f"" - "
    processalive
    registry size{len(names)}
    time{html_lib.escape(time.strftime('%Y-%m-%d %H:%M:%S'))}
    " - ) - - -def _render_progress() -> str: - snapshots = progress_registry.list() - if not snapshots: - return "

    no active transfers

    " - rows = [] - for item in snapshots: - name = html_lib.escape(str(item.get("name", ""))) - status = html_lib.escape(str(item.get("status", ""))) - transferred = int(item.get("transferred", 0) or 0) - total = item.get("total") - total_cell = "—" if total in (None, 0) else str(total) - pct = "" - if isinstance(total, int) and total > 0: - pct = f" ({(transferred / total) * 100:.1f}%)" - rows.append( - "" - f"{name}" - f"{status}" - f"{transferred}{pct}" - f"{total_cell}" - "" - ) - return ( - "" - "" - + "".join(rows) - + "
    namestatustransferredtotal
    " - ) - - -def _render_registry() -> str: - names = sorted(executor.registry.event_dict.keys()) - if not names: - return "

    registry empty

    " - items = "".join(f"
  • {html_lib.escape(name)}
    • " for name in names) - return f"
      {items}
    " - - class WebUIServer(ThreadingHTTPServer): - """Threaded HTTP server for the HTMX dashboard.""" + """Threaded HTTP server for the HTMX dashboard. + + ``services`` is the set of application services the fragments read; the + process-wide set by default, the one the desktop window uses too. + """ def __init__( self, server_address: tuple[str, int], handler_class: type = _WebUIHandler, shared_secret: str | None = None, + services: AppServices | None = None, ) -> None: super().__init__(server_address, handler_class) self.shared_secret: str | None = shared_secret + self.services: AppServices = app_services() if services is None else services def start_web_ui( @@ -195,15 +410,20 @@ def start_web_ui( port: int = _DEFAULT_PORT, allow_non_loopback: bool = False, shared_secret: str | None = None, + services: AppServices | None = None, ) -> WebUIServer: - """Start the Web UI server on a background thread.""" + """Start the Web UI server on a background thread. + + ``services`` replaces the process-wide application services, for a server + that should show another run store, bus or resolver. + """ if not allow_non_loopback: ensure_loopback(host) if allow_non_loopback and not shared_secret: file_automation_logger.warning( "web_ui: non-loopback bind without shared_secret is insecure", ) - server = WebUIServer((host, port), shared_secret=shared_secret) + server = WebUIServer((host, port), shared_secret=shared_secret, services=services) thread = threading.Thread(target=server.serve_forever, daemon=True) thread.start() file_automation_logger.info( diff --git a/automation_file/ui/__init__.py b/automation_file/ui/__init__.py index 33fee27..52e4274 100644 --- a/automation_file/ui/__init__.py +++ b/automation_file/ui/__init__.py @@ -1,9 +1,11 @@ """PySide6 GUI for automation_file. -Exposes every registered ``FA_*`` action through a tabbed main window so users -can drive local file ops, HTTP downloads, Google Drive, S3, Azure Blob, -Dropbox, SFTP, JSON action lists, and the TCP / HTTP action servers without -writing any code. +The main window is organised by workflow: Dashboard, Files, Storage, Pipelines, +Scheduler, Integrity, Audit, Notifications and Settings, each a page over one +service of :mod:`automation_file.app`. An Advanced entry keeps the tools that +address a single action or backend: local file ops, the per-backend transfer +panels, transfer progress, JSON action lists, file triggers and the TCP / HTTP +action servers. The entry point is :func:`launch_ui` (also mirrored as the ``ui`` subcommand of ``python -m automation_file``). diff --git a/automation_file/ui/launcher.py b/automation_file/ui/launcher.py index 0166902..9de4a2d 100644 --- a/automation_file/ui/launcher.py +++ b/automation_file/ui/launcher.py @@ -9,17 +9,24 @@ import sys from collections.abc import Sequence +from automation_file.core.optional import require_module from automation_file.logging_config import file_automation_logger +_GUI_EXTRA = "gui" + def launch_ui(argv: Sequence[str] | None = None) -> int: - """Launch the automation_file GUI. Blocks on the Qt event loop.""" - from PySide6.QtWidgets import QApplication + """Launch the automation_file GUI. Blocks on the Qt event loop. + + Raises :class:`~automation_file.exceptions.OptionalDependencyException`, + naming the ``gui`` extra, when PySide6 is not installed. + """ + widgets = require_module("PySide6.QtWidgets", extra=_GUI_EXTRA) from automation_file.ui.main_window import MainWindow args = list(argv) if argv is not None else sys.argv - app = QApplication.instance() or QApplication(args) + app = widgets.QApplication.instance() or widgets.QApplication(args) window = MainWindow() window.show() file_automation_logger.info("ui: launched main window") diff --git a/automation_file/ui/main_window.py b/automation_file/ui/main_window.py index 3e1e3e9..b27523f 100644 --- a/automation_file/ui/main_window.py +++ b/automation_file/ui/main_window.py @@ -1,89 +1,180 @@ -"""Main window — tabbed interface over every built-in feature.""" +"""Main window: a sidebar of operational workflows over the application layer. + +The navigation is the one :data:`automation_file.app.NAVIGATION` names -- +Dashboard, Files, Storage, Pipelines, Scheduler, Integrity, Audit, +Notifications, Settings -- followed by Advanced, which keeps the tools that +address a single action or backend. Each of the nine pages talks only to its +service of :mod:`automation_file.app`. +""" from __future__ import annotations -from PySide6.QtCore import Qt, QThreadPool +from PySide6.QtCore import Qt, QThreadPool, QTimer from PySide6.QtGui import QKeySequence, QShortcut -from PySide6.QtWidgets import QMainWindow, QSplitter, QTabWidget, QVBoxLayout, QWidget +from PySide6.QtWidgets import ( + QHBoxLayout, + QListWidget, + QMainWindow, + QSplitter, + QStackedWidget, + QVBoxLayout, + QWidget, +) +from automation_file.app import NAVIGATION, AppServices, app_services from automation_file.logging_config import file_automation_logger from automation_file.ui.log_widget import LogPanel -from automation_file.ui.tabs import ( - HomeTab, - JSONEditorTab, - LocalOpsTab, - ProgressTab, - SchedulerTab, - ServerTab, - TransferTab, - TriggerTab, +from automation_file.ui.pages import ( + AdvancedPage, + AuditPage, + BasePage, + DashboardPage, + FilesPage, + IntegrityPage, + NotificationsPage, + PipelinesPage, + SchedulerPage, + SettingsPage, + StoragePage, ) +ADVANCED = "Advanced" _WINDOW_TITLE = "automation_file" -_DEFAULT_SIZE = (1100, 780) +_DEFAULT_SIZE = (1280, 860) _STATUS_DEFAULT = "Ready" +_SIDEBAR_WIDTH = 170 +_STATUS_MESSAGE_MS = 5000 +_SHORTCUT_COUNT = 9 +_CLOSE_WAIT_MS = 2000 class MainWindow(QMainWindow): - """Tab-based control surface for every registered FA_* feature.""" + """Sidebar navigation over the nine workflow pages and the Advanced tools. + + ``services`` is the set of application services the pages use; the + process-wide set by default, so the window shows what the rest of the + process does. + """ - def __init__(self) -> None: + def __init__(self, services: AppServices | None = None) -> None: super().__init__() self.setWindowTitle(_WINDOW_TITLE) self.resize(*_DEFAULT_SIZE) + self._services = app_services() if services is None else services self._pool = QThreadPool.globalInstance() self._log = LogPanel() self._log.message_appended.connect(self._on_log_message) + self._advanced = AdvancedPage(self._log, self._pool) + self._pages: dict[str, BasePage] = {**self._workflow_pages(), ADVANCED: self._advanced} + + self._sidebar = QListWidget() + self._sidebar.setFixedWidth(_SIDEBAR_WIDTH) + self._stack = QStackedWidget() + for name, page in self._pages.items(): + self._sidebar.addItem(name) + self._stack.addWidget(page) + self._sidebar.setCurrentRow(0) + self._sidebar.currentRowChanged.connect(self._on_page_changed) + + self.setCentralWidget(self._central_widget()) + self._register_shortcuts() + self.statusBar().showMessage(_STATUS_DEFAULT) + # The first page reads its data once the event loop runs, not while the window is built. + self._startup = QTimer(self) + self._startup.setSingleShot(True) + self._startup.timeout.connect(self._show_current_page) + self._startup.start(0) + file_automation_logger.info("ui: main window constructed") - self._tabs = QTabWidget() - self._home_tab = HomeTab(self._log, self._pool) - self._home_tab.navigate_to_tab.connect(self._focus_tab_by_name) - self._tabs.addTab(self._home_tab, "Home") - self._tabs.addTab(LocalOpsTab(self._log, self._pool), "Local") - self._tabs.addTab(TransferTab(self._log, self._pool), "Transfer") - self._tabs.addTab(ProgressTab(self._log, self._pool), "Progress") - self._tabs.addTab(JSONEditorTab(self._log, self._pool), "JSON actions") - self._trigger_tab = TriggerTab(self._log, self._pool) - self._tabs.addTab(self._trigger_tab, "Triggers") - self._scheduler_tab = SchedulerTab(self._log, self._pool) - self._tabs.addTab(self._scheduler_tab, "Scheduler") - self._server_tab = ServerTab(self._log, self._pool) - self._tabs.addTab(self._server_tab, "Servers") + def _workflow_pages(self) -> dict[str, BasePage]: + services, log, pool = self._services, self._log, self._pool + pages: tuple[BasePage, ...] = ( + DashboardPage(services.dashboard, log, pool), + FilesPage(services.files, log, pool), + StoragePage(services.storage, log, pool), + PipelinesPage(services.pipelines, log, pool), + SchedulerPage(services.scheduler, log, pool), + IntegrityPage(services.integrity, log, pool), + AuditPage(services.audit, log, pool), + NotificationsPage(services.notifications, log, pool), + SettingsPage(services.settings, log, pool), + ) + return dict(zip(NAVIGATION, pages, strict=True)) + + def _central_widget(self) -> QWidget: + navigation = QWidget() + row = QHBoxLayout(navigation) + row.setContentsMargins(0, 0, 0, 0) + row.setSpacing(8) + row.addWidget(self._sidebar) + row.addWidget(self._stack, 1) splitter = QSplitter() splitter.setOrientation(Qt.Orientation.Vertical) - splitter.addWidget(self._tabs) + splitter.addWidget(navigation) splitter.addWidget(self._log) - splitter.setStretchFactor(0, 4) + splitter.setStretchFactor(0, 5) splitter.setStretchFactor(1, 1) container = QWidget() layout = QVBoxLayout(container) layout.setContentsMargins(8, 8, 8, 8) layout.addWidget(splitter) - self.setCentralWidget(container) - - self._register_shortcuts() - self.statusBar().showMessage(_STATUS_DEFAULT) - file_automation_logger.info("ui: main window constructed") + return container def _register_shortcuts(self) -> None: - for index in range(self._tabs.count()): + for index in range(min(self._stack.count(), _SHORTCUT_COUNT)): shortcut = QShortcut(QKeySequence(f"Ctrl+{index + 1}"), self) - shortcut.activated.connect(lambda i=index: self._tabs.setCurrentIndex(i)) + shortcut.activated.connect(lambda i=index: self._sidebar.setCurrentRow(i)) + advanced = QShortcut(QKeySequence("Ctrl+0"), self) + advanced.activated.connect(lambda: self.navigate(ADVANCED)) + + # ------------------------------------------------------------------ navigation + + def page_names(self) -> list[str]: + """Return the navigation entries, in display order.""" + return list(self._pages) + + def page(self, name: str) -> BasePage: + """Return the page of the navigation entry ``name``.""" + return self._pages[name] + + def current_page_name(self) -> str: + """Return the navigation entry that is shown.""" + return self.page_names()[self._stack.currentIndex()] + + def navigate(self, name: str) -> bool: + """Show the page ``name``; return whether there is one.""" + names = self.page_names() + if name not in names: + return False + self._sidebar.setCurrentRow(names.index(name)) + return True + + def open_tool(self, name: str) -> bool: + """Show the Advanced tool ``name`` (Local, Transfer, ...); return whether there is one.""" + return self._advanced.open_tool(name) and self.navigate(ADVANCED) + + def _on_page_changed(self, row: int) -> None: + if 0 <= row < self._stack.count(): + self._stack.setCurrentIndex(row) + self._show_current_page() - def _focus_tab_by_name(self, name: str) -> None: - for index in range(self._tabs.count()): - if self._tabs.tabText(index) == name: - self._tabs.setCurrentIndex(index) - return + def _show_current_page(self) -> None: + self._pages[self.current_page_name()].on_shown() def _on_log_message(self, message: str) -> None: - self.statusBar().showMessage(message, 5000) + self.statusBar().showMessage(message, _STATUS_MESSAGE_MS) def closeEvent(self, event) -> None: # noqa: N802 # pylint: disable=invalid-name — Qt override - self._server_tab.closeEvent(event) - self._trigger_tab.closeEvent(event) - self._scheduler_tab.closeEvent(event) + self._startup.stop() + self._advanced.close_tools(event) + for page in self._pages.values(): + page.shutdown() + # A refresh that is still reading must not outlive the objects its signals belong to. + if not self._pool.waitForDone(_CLOSE_WAIT_MS): + file_automation_logger.warning( + "ui: background work was still running when the window closed" + ) super().closeEvent(event) diff --git a/automation_file/ui/pages/__init__.py b/automation_file/ui/pages/__init__.py new file mode 100644 index 0000000..ceae4dd --- /dev/null +++ b/automation_file/ui/pages/__init__.py @@ -0,0 +1,47 @@ +"""Pages of the main window, one per navigation entry of the application layer.""" + +from __future__ import annotations + +from automation_file.ui.pages.advanced_page import ADVANCED_TOOLS, AdvancedPage +from automation_file.ui.pages.audit_page import AuditPage +from automation_file.ui.pages.base import BasePage +from automation_file.ui.pages.dashboard_page import DashboardPage +from automation_file.ui.pages.files_page import FilesPage +from automation_file.ui.pages.integrity_page import IntegrityPage +from automation_file.ui.pages.notifications_page import NotificationsPage +from automation_file.ui.pages.pipeline_canvas import ( + ACTION_MIME, + ActionPalette, + EdgeItem, + PipelineCanvas, + TaskNode, +) +from automation_file.ui.pages.pipelines_page import PipelinesPage +from automation_file.ui.pages.run_panel import RunPanel +from automation_file.ui.pages.scheduler_page import SchedulerPage +from automation_file.ui.pages.settings_page import SettingsPage +from automation_file.ui.pages.storage_page import StoragePage +from automation_file.ui.pages.task_form import ArgumentsEditor, TaskForm + +__all__ = [ + "ACTION_MIME", + "ADVANCED_TOOLS", + "ActionPalette", + "AdvancedPage", + "ArgumentsEditor", + "AuditPage", + "BasePage", + "DashboardPage", + "EdgeItem", + "FilesPage", + "IntegrityPage", + "NotificationsPage", + "PipelineCanvas", + "PipelinesPage", + "RunPanel", + "SchedulerPage", + "SettingsPage", + "StoragePage", + "TaskForm", + "TaskNode", +] diff --git a/automation_file/ui/pages/advanced_page.py b/automation_file/ui/pages/advanced_page.py new file mode 100644 index 0000000..72f8112 --- /dev/null +++ b/automation_file/ui/pages/advanced_page.py @@ -0,0 +1,89 @@ +"""Advanced page: the tools that address one action or one backend directly. + +These are the tabs the main window had before it was organised by workflow. +They are reused as they are, so everything they could do is still here: local +file, directory and ZIP actions, the per-backend transfer panels (where a cloud +client is given its credentials), live transfer progress, the JSON action +editor, file triggers and the TCP / HTTP action servers. +""" + +from __future__ import annotations + +from typing import Any + +from PySide6.QtCore import QThreadPool +from PySide6.QtWidgets import QTabWidget, QVBoxLayout, QWidget + +from automation_file.ui.log_widget import LogPanel +from automation_file.ui.pages.base import BasePage +from automation_file.ui.tabs import ( + JSONEditorTab, + LocalOpsTab, + ProgressTab, + ServerTab, + TransferTab, + TriggerTab, +) + +#: The tools of the page, in display order. +ADVANCED_TOOLS: tuple[str, ...] = ( + "Local", + "Transfer", + "Progress", + "JSON actions", + "Triggers", + "Servers", +) + + +class AdvancedPage(BasePage): + """Local, Transfer, Progress, JSON actions, Triggers and Servers, each in a tab.""" + + title = "Advanced" + + def __init__(self, log: LogPanel, pool: QThreadPool) -> None: + super().__init__(log, pool) + self._trigger_tab = TriggerTab(log, pool) + self._server_tab = ServerTab(log, pool) + tools: tuple[QWidget, ...] = ( + LocalOpsTab(log, pool), + TransferTab(log, pool), + ProgressTab(log, pool), + JSONEditorTab(log, pool), + self._trigger_tab, + self._server_tab, + ) + self._tabs = QTabWidget() + for name, tool in zip(ADVANCED_TOOLS, tools, strict=True): + self._tabs.addTab(tool, name) + layout = QVBoxLayout(self) + layout.setContentsMargins(8, 8, 8, 8) + layout.addWidget(self._tabs) + + def tool_names(self) -> list[str]: + """Return the names of the tools, in display order.""" + return [self._tabs.tabText(index) for index in range(self._tabs.count())] + + def current_tool(self) -> str: + """Return the name of the tool that is shown.""" + return self._tabs.tabText(self._tabs.currentIndex()) + + def tool(self, name: str) -> QWidget | None: + """Return the widget of the tool called ``name``, or ``None``.""" + for index in range(self._tabs.count()): + if self._tabs.tabText(index) == name: + return self._tabs.widget(index) + return None + + def open_tool(self, name: str) -> bool: + """Show the tool called ``name``; return whether there is one.""" + for index in range(self._tabs.count()): + if self._tabs.tabText(index) == name: + self._tabs.setCurrentIndex(index) + return True + return False + + def close_tools(self, event: Any) -> None: + """Stop what the tools started: the action servers and the file triggers.""" + self._server_tab.closeEvent(event) + self._trigger_tab.closeEvent(event) diff --git a/automation_file/ui/pages/audit_page.py b/automation_file/ui/pages/audit_page.py new file mode 100644 index 0000000..dc52b50 --- /dev/null +++ b/automation_file/ui/pages/audit_page.py @@ -0,0 +1,218 @@ +"""Audit page: point the audit trail at a database, then search and count its records.""" + +from __future__ import annotations + +import json +from typing import Any + +from PySide6.QtCore import QThreadPool +from PySide6.QtWidgets import ( + QFormLayout, + QGridLayout, + QGroupBox, + QHBoxLayout, + QLabel, + QLineEdit, + QPlainTextEdit, + QSpinBox, + QSplitter, + QVBoxLayout, +) + +from automation_file.app import AuditService +from automation_file.ui.log_widget import LogPanel +from automation_file.ui.pages.base import BasePage, fill_table, make_table + +_COLUMNS = ("Time", "Actor", "Source", "Action", "Resource", "Status", "Duration (ms)", "Error") +_KEYS = ("timestamp", "actor", "source", "action", "resource", "status", "duration_ms", "error") +#: Filter name -> (label, placeholder); the order is the order of the form. +_FILTERS: dict[str, tuple[str, str]] = { + "text": ("Contains", "text in the action, resource, error or metadata"), + "status": ("Status", "ok, warning, error ..."), + "action": ("Action", "pipeline.failed, upload ..."), + "actor": ("Actor", ""), + "source": ("Source", "pipeline, storage, scheduler ..."), + "pipeline": ("Pipeline", ""), + "task": ("Task", ""), + "backend": ("Backend", "s3, local ..."), + "resource_prefix": ("Resource starts with", "s3://reports/"), + "correlation_id": ("Run / correlation ID", ""), + "since": ("Since", "2026-10-01T00:00:00+00:00"), + "until": ("Until", "2026-10-08T00:00:00+00:00"), +} +_COLUMNS_PER_ROW = 3 +_DEFAULT_LIMIT = 100 +_MAX_LIMIT = 10_000 + + +class AuditPage(BasePage): + """Who did what, when, against which resource, with what result.""" + + title = "Audit" + + def __init__(self, service: AuditService, log: LogPanel, pool: QThreadPool) -> None: + super().__init__(log, pool) + self._service = service + self._records: list[dict[str, Any]] = [] + self._fields: dict[str, QLineEdit] = {} + + root = QVBoxLayout(self) + root.setContentsMargins(12, 12, 12, 12) + root.setSpacing(10) + root.addWidget(self._configure_group()) + root.addWidget(self._filter_group()) + splitter = QSplitter() + splitter.addWidget(self._results_group()) + splitter.addWidget(self._detail_group()) + splitter.setStretchFactor(0, 3) + splitter.setStretchFactor(1, 2) + root.addWidget(splitter, 1) + root.addWidget(self.status_label()) + + # ------------------------------------------------------------------ layout + + def _configure_group(self) -> QGroupBox: + box = QGroupBox("Audit trail") + form = QFormLayout(box) + self._state = QLabel("Not read yet") + self._db_path = QLineEdit() + self._db_path.setPlaceholderText("SQLite database of the audit trail; created when missing") + row = QHBoxLayout() + row.addWidget(self._db_path, 1) + row.addWidget(self.make_button("Browse…", self._on_browse)) + row.addWidget(self.make_button("Configure", self.configure)) + form.addRow("State", self._state) + form.addRow("Database", row) + return box + + def _filter_group(self) -> QGroupBox: + box = QGroupBox("Search") + layout = QVBoxLayout(box) + grid = QGridLayout() + for index, (name, (label, placeholder)) in enumerate(_FILTERS.items()): + field = QLineEdit() + field.setPlaceholderText(placeholder) + field.returnPressed.connect(self.search) + self._fields[name] = field + row, column = divmod(index, _COLUMNS_PER_ROW) + grid.addWidget(QLabel(label), row, column * 2) + grid.addWidget(field, row, column * 2 + 1) + layout.addLayout(grid) + buttons = QHBoxLayout() + self._limit = QSpinBox() + self._limit.setRange(1, _MAX_LIMIT) + self._limit.setValue(_DEFAULT_LIMIT) + buttons.addWidget(QLabel("Limit")) + buttons.addWidget(self._limit) + buttons.addWidget(self.make_button("Search", self.search)) + buttons.addWidget(self.make_button("Count", self.count)) + buttons.addWidget(self.make_button("Clear filters", self.clear_filters)) + buttons.addStretch() + layout.addLayout(buttons) + return box + + def _results_group(self) -> QGroupBox: + box = QGroupBox("Records (newest first)") + layout = QVBoxLayout(box) + self._table = make_table(_COLUMNS) + self._table.itemSelectionChanged.connect(self._on_selection) + layout.addWidget(self._table) + return box + + def _detail_group(self) -> QGroupBox: + box = QGroupBox("Selected record") + layout = QVBoxLayout(box) + self._detail = QPlainTextEdit() + self._detail.setReadOnly(True) + layout.addWidget(self._detail) + return box + + # ------------------------------------------------------------------ configuration + + def refresh(self) -> None: + self.run_async( + self._service.status, "read state", self._show_state, key="state", quiet=True + ) + + def _show_state(self, state: dict[str, Any]) -> None: + if not state.get("configured"): + self._state.setText("Not configured: nothing is recorded yet") + return + where = state.get("db_path") or state.get("store") + recording = "recording" if state.get("active") else "configured, not recording" + self._state.setText(f"{recording} into {where}") + if state.get("db_path") and not self._db_path.text(): + self._db_path.setText(str(state["db_path"])) + + def _on_browse(self) -> None: + chosen = self.pick_save_file( + "Audit database", "SQLite database (*.sqlite *.db);;All files (*)" + ) + if chosen: + self._db_path.setText(chosen) + + def configure(self) -> None: + path = self._db_path.text().strip() + if not path: + self.report_error("enter the path of the audit database first") + return + self.run_async(lambda: self._service.configure(path), "configure", self._configured) + + def _configured(self, state: dict[str, Any]) -> None: + self._show_state(state) + self.report(f"audit records go to {state.get('db_path') or state.get('store')}") + + # ------------------------------------------------------------------ search + + def set_filter(self, name: str, value: str) -> None: + """Fill in one filter field.""" + self._fields[name].setText(value) + + def filters(self) -> dict[str, Any]: + """Return the filters as the form holds them; an empty field is left out.""" + return { + name: field.text().strip() + for name, field in self._fields.items() + if field.text().strip() + } + + def clear_filters(self) -> None: + for field in self._fields.values(): + field.clear() + + def records(self) -> list[dict[str, Any]]: + """Return the records the table shows.""" + return list(self._records) + + def search(self) -> None: + filters = {**self.filters(), "limit": self._limit.value()} + self.run_async(lambda: self._service.search(**filters), "search", self._show_records) + + def count(self) -> None: + filters = self.filters() + self.run_async( + lambda: self._service.count(**filters), + "count", + lambda found: self.report(f"{found} record(s) match"), + ) + + def _show_records(self, records: list[dict[str, Any]]) -> None: + self._records = records + fill_table( + self._table, + [[record.get(key) for key in _KEYS] for record in records], + keep_selection=False, + ) + self._detail.clear() + self.report(f"{len(records)} record(s) shown") + + def _on_selection(self) -> None: + row = self._table.currentRow() + if 0 <= row < len(self._records): + self._detail.setPlainText( + json.dumps(self._records[row], indent=2, ensure_ascii=False, default=str) + ) + + def detail_text(self) -> str: + """Return what the detail pane shows.""" + return self._detail.toPlainText() diff --git a/automation_file/ui/pages/base.py b/automation_file/ui/pages/base.py new file mode 100644 index 0000000..1c8233f --- /dev/null +++ b/automation_file/ui/pages/base.py @@ -0,0 +1,244 @@ +"""Shared base class and helpers for the pages of the main window. + +A page is a thin view over one service of :mod:`automation_file.app`. It reads +its widgets, calls the service off the UI thread through +:meth:`BasePage.run_async`, and shows what comes back. It imports nothing below +the application layer. +""" + +from __future__ import annotations + +from collections.abc import Callable, Sequence +from typing import Any + +from PySide6.QtCore import QThreadPool +from PySide6.QtWidgets import ( + QAbstractItemView, + QFileDialog, + QHeaderView, + QLabel, + QMessageBox, + QPushButton, + QTableWidget, + QTableWidgetItem, + QWidget, +) + +from automation_file.app import mask_text +from automation_file.ui.log_widget import LogPanel +from automation_file.ui.worker import ActionWorker + +EMPTY_CELL = "—" +_OK_STYLE = "color: #2f8f3f;" +_ERROR_STYLE = "color: #b3261e;" +_MUTED_STYLE = "color: #777;" + + +def cell_text(value: object) -> str: + """Return what a table cell shows for ``value``.""" + if value is None or value == "": + return EMPTY_CELL + if isinstance(value, bool): + return "yes" if value else "no" + if isinstance(value, (list, tuple)): + return ", ".join(str(item) for item in value) or EMPTY_CELL + return str(value) + + +def make_table(columns: Sequence[str]) -> QTableWidget: + """Build a read-only table that selects whole rows.""" + table = QTableWidget(0, len(columns)) + table.setHorizontalHeaderLabels(list(columns)) + table.horizontalHeader().setSectionResizeMode(QHeaderView.ResizeMode.Stretch) + table.verticalHeader().setVisible(False) + table.setSelectionBehavior(QAbstractItemView.SelectionBehavior.SelectRows) + table.setSelectionMode(QAbstractItemView.SelectionMode.SingleSelection) + table.setEditTriggers(QAbstractItemView.EditTrigger.NoEditTriggers) + return table + + +def fill_table( + table: QTableWidget, rows: Sequence[Sequence[object]], *, keep_selection: bool = True +) -> None: + """Replace the rows of ``table``; every value is shown through :func:`cell_text`. + + The selection follows the entry, not the row number: the row whose first + cell reads the same as before stays selected, and when there is none (or + ``keep_selection`` is false) nothing is selected. A button that acts on the + selection can therefore never hit an entry that merely moved into its row. + """ + selected = selected_cell(table) if keep_selection else None + table.setRowCount(len(rows)) + for row, values in enumerate(rows): + for column, value in enumerate(values): + table.setItem(row, column, QTableWidgetItem(cell_text(value))) + table.clearSelection() + table.setCurrentCell(-1, -1) + if selected is None: + return + for row, values in enumerate(rows): + if values and cell_text(values[0]) == selected: + table.selectRow(row) + return + + +def selected_cell(table: QTableWidget, column: int = 0) -> str | None: + """Return the text of ``column`` in the selected row, or ``None`` without a selection.""" + row = table.currentRow() + item = table.item(row, column) if row >= 0 else None + return None if item is None else item.text() + + +class BasePage(QWidget): + """Common behaviour of every page: background calls, a status line, the activity log.""" + + #: The navigation entry of the page; it prefixes the page's log lines. + title = "" + + def __init__(self, log: LogPanel, pool: QThreadPool) -> None: + super().__init__() + self._log = log + self._pool = pool + self._workers: set[ActionWorker] = set() + self._in_flight: set[str] = set() + self._closed = False + self._status = QLabel("") + self._status.setWordWrap(True) + + # ------------------------------------------------------------------ lifecycle + + def refresh(self) -> None: + """Read the page's data again. The default page has none.""" + + def on_shown(self) -> None: + """Called by the main window when the page becomes the visible one.""" + self.refresh() + + def shutdown(self) -> None: + """Called by the main window when it closes: results that arrive later are dropped.""" + self._closed = True + + # ------------------------------------------------------------------ background work + + def run_async( + self, + target: Callable[..., Any], + label: str, + on_done: Callable[[Any], None] | None = None, + *, + key: str | None = None, + quiet: bool = False, + on_error: Callable[[], None] | None = None, + ) -> bool: + """Call ``target()`` on the thread pool and hand its result to ``on_done`` on the UI thread. + + With ``key``, a call is skipped (and ``False`` returned) while an earlier + call with the same key is still running: a timer cannot pile up requests. + ``quiet`` keeps the start of the call out of the activity log; a failure + is always logged and shown on the status line, and then ``on_error`` is + called, for a page that has to stop something when a call fails. + """ + if key is not None: + if key in self._in_flight: + return False + self._in_flight.add(key) + worker = ActionWorker(target, label=f"{self.title}: {label}") + self._workers.add(worker) + if not quiet: + self._log.append_line(f"{self.title}: {label} ...") + worker.signals.finished.connect( + lambda result: self._on_finished(worker, key, on_done, result) + ) + worker.signals.failed.connect( + lambda error: self._on_failed(worker, key, f"{label} failed: {error}", on_error) + ) + self._pool.start(worker) + return True + + def _release(self, worker: ActionWorker, key: str | None) -> None: + self._workers.discard(worker) + if key is not None: + self._in_flight.discard(key) + + def _on_finished( + self, + worker: ActionWorker, + key: str | None, + on_done: Callable[[Any], None] | None, + result: Any, + ) -> None: + self._release(worker, key) + if not self._closed and on_done is not None: + on_done(result) + + def _on_failed( + self, + worker: ActionWorker, + key: str | None, + message: str, + on_error: Callable[[], None] | None, + ) -> None: + self._release(worker, key) + if self._closed: + return + self.report_error(message) + if on_error is not None: + on_error() + + # ------------------------------------------------------------------ feedback + + def status_label(self) -> QLabel: + """Return the page's status line, for the page to place in its layout.""" + return self._status + + def status_text(self) -> str: + """Return what the status line currently says.""" + return self._status.text() + + def report(self, message: str) -> None: + """Show ``message`` on the status line and append it to the activity log.""" + self._set_status(message, _OK_STYLE) + + def report_error(self, message: str) -> None: + """Show ``message`` as a failure on the status line and in the activity log.""" + self._set_status(message, _ERROR_STYLE) + + def _set_status(self, message: str, style: str) -> None: + shown = mask_text(message) + self._status.setStyleSheet(style) + self._status.setText(shown) + self._log.append_line(f"{self.title}: {shown}") + + def confirm(self, question: str) -> bool: + """Ask the user to confirm something that cannot be undone.""" + answer = QMessageBox.question(self, self.title, question) + return answer == QMessageBox.StandardButton.Yes + + # ------------------------------------------------------------------ small widgets + + @staticmethod + def make_button(label: str, handler: Callable[[], Any]) -> QPushButton: + """Build a ``QPushButton`` that calls ``handler`` when clicked.""" + button = QPushButton(label) + button.clicked.connect(lambda _checked=False: handler()) + return button + + @staticmethod + def muted_label(text: str) -> QLabel: + """Build a word-wrapped explanatory label.""" + label = QLabel(text) + label.setWordWrap(True) + label.setStyleSheet(_MUTED_STYLE) + return label + + def pick_directory(self) -> str | None: + """Let the user choose a directory; ``None`` when the dialog is cancelled.""" + return QFileDialog.getExistingDirectory(self, "Select directory") or None + + def pick_open_file(self, caption: str, name_filter: str = "") -> str | None: + """Let the user choose an existing file; ``None`` when the dialog is cancelled.""" + return QFileDialog.getOpenFileName(self, caption, "", name_filter)[0] or None + + def pick_save_file(self, caption: str, name_filter: str = "") -> str | None: + """Let the user choose where to save; ``None`` when the dialog is cancelled.""" + return QFileDialog.getSaveFileName(self, caption, "", name_filter)[0] or None diff --git a/automation_file/ui/pages/dashboard_page.py b/automation_file/ui/pages/dashboard_page.py new file mode 100644 index 0000000..eacc227 --- /dev/null +++ b/automation_file/ui/pages/dashboard_page.py @@ -0,0 +1,235 @@ +"""Dashboard page: health, runs, integrity drift, recent events and storage status.""" + +from __future__ import annotations + +from typing import Any + +from PySide6.QtCore import QThreadPool, QTimer +from PySide6.QtWidgets import ( + QCheckBox, + QFormLayout, + QGridLayout, + QGroupBox, + QHBoxLayout, + QLabel, + QVBoxLayout, +) + +from automation_file.app import DashboardService, DashboardSummary +from automation_file.ui.log_widget import LogPanel +from automation_file.ui.pages.base import BasePage, cell_text, fill_table, make_table + +_REFRESH_INTERVAL_MS = 5000 +_HEADLINES = {"ok": "All clear", "attention": "Needs attention"} +_HEADLINE_STYLES = { + "ok": "font-size: 18px; font-weight: bold; color: #2f8f3f;", + "attention": "font-size: 18px; font-weight: bold; color: #b3261e;", +} +_HEALTH_ROWS = ( + ("registry_size", "Registered actions"), + ("running_runs", "Running pipeline runs"), + ("scheduler_jobs", "Scheduled jobs"), + ("integrity_monitors", "Integrity monitors"), + ("notification_sinks", "Notification sinks"), + ("notification_routes", "Notification routes"), + ("notification_router_active", "Notification router active"), +) +_RUN_COLUMNS = ("Run", "Pipeline", "Status", "Started", "Tasks", "Error") +_INTEGRITY_COLUMNS = ("Monitor", "Target", "Running", "Drift", "Last run", "Error") +_STORAGE_COLUMNS = ("Backend", "Kind", "Usable", "Detail") +_EVENT_COLUMNS = ("Time", "Severity", "Type", "Source", "Subject") +_RUN_ID_LENGTH = 8 + + +def _run_row(run: dict[str, Any]) -> list[object]: + statuses = run.get("task_statuses") or {} + tasks = ", ".join(f"{count} {status}" for status, count in statuses.items()) + return [ + str(run.get("run_id") or "")[:_RUN_ID_LENGTH], + run.get("pipeline"), + run.get("status"), + run.get("started_at"), + tasks, + run.get("error"), + ] + + +def _drift_text(monitor: dict[str, Any]) -> str: + if monitor.get("ok") is None: + return "not verified yet" + return "none" if monitor["ok"] else f"{monitor.get('changes', 0)} change(s)" + + +class DashboardPage(BasePage): + """One look at how the installation is doing; it reads and starts nothing.""" + + title = "Dashboard" + + def __init__(self, service: DashboardService, log: LogPanel, pool: QThreadPool) -> None: + super().__init__(log, pool) + self._service = service + self._summary: DashboardSummary | None = None + self._health_labels: dict[str, QLabel] = {} + + root = QVBoxLayout(self) + root.setContentsMargins(12, 12, 12, 12) + root.setSpacing(10) + root.addLayout(self._header()) + grid = QGridLayout() + grid.setSpacing(10) + grid.addWidget(self._health_group(), 0, 0) + grid.addWidget(self._runs_group(), 0, 1) + grid.addWidget(self._integrity_group(), 1, 0) + grid.addWidget(self._storage_group(), 1, 1) + grid.addWidget(self._events_group(), 2, 0, 1, 2) + grid.setColumnStretch(0, 1) + grid.setColumnStretch(1, 2) + root.addLayout(grid, 1) + root.addWidget(self.status_label()) + + self._timer = QTimer(self) + self._timer.setInterval(_REFRESH_INTERVAL_MS) + self._timer.timeout.connect(self._on_tick) + self._timer.start() + + # ------------------------------------------------------------------ layout + + def _header(self) -> QHBoxLayout: + row = QHBoxLayout() + self._headline = QLabel("Not read yet") + self._headline.setStyleSheet("font-size: 18px; font-weight: bold; color: #777;") + self._reasons = QLabel("") + self._reasons.setWordWrap(True) + self._auto = QCheckBox("Refresh every 5 s") + self._auto.setChecked(True) + row.addWidget(self._headline) + row.addWidget(self._reasons, 1) + row.addWidget(self._auto) + row.addWidget(self.make_button("Refresh", self.refresh)) + return row + + def _health_group(self) -> QGroupBox: + box = QGroupBox("Health") + form = QFormLayout(box) + for key, label in (*_HEALTH_ROWS, ("audit", "Audit trail"), ("time", "Read at")): + value = QLabel(cell_text(None)) + self._health_labels[key] = value + form.addRow(label, value) + return box + + def _runs_group(self) -> QGroupBox: + box = QGroupBox("Pipeline runs") + layout = QVBoxLayout(box) + self._run_counts = QLabel("No runs recorded") + self._running_table = make_table(_RUN_COLUMNS) + self._recent_table = make_table(_RUN_COLUMNS) + layout.addWidget(self._run_counts) + layout.addWidget(QLabel("Running")) + layout.addWidget(self._running_table) + layout.addWidget(QLabel("Latest results")) + layout.addWidget(self._recent_table) + return box + + def _integrity_group(self) -> QGroupBox: + box = QGroupBox("Integrity drift") + layout = QVBoxLayout(box) + self._integrity_table = make_table(_INTEGRITY_COLUMNS) + layout.addWidget(self._integrity_table) + return box + + def _storage_group(self) -> QGroupBox: + box = QGroupBox("Storage status") + layout = QVBoxLayout(box) + self._storage_table = make_table(_STORAGE_COLUMNS) + layout.addWidget(self._storage_table) + return box + + def _events_group(self) -> QGroupBox: + box = QGroupBox("Recent events") + layout = QVBoxLayout(box) + self._events_table = make_table(_EVENT_COLUMNS) + layout.addWidget(self._events_table) + return box + + # ------------------------------------------------------------------ data + + def refresh(self) -> None: + self.run_async( + self._service.summary, "read summary", self.show_summary, key="refresh", quiet=True + ) + + def shutdown(self) -> None: + self._timer.stop() + super().shutdown() + + def _on_tick(self) -> None: + if self._auto.isChecked() and self.isVisible(): + self.refresh() + + def summary(self) -> DashboardSummary | None: + """Return the summary the page currently shows.""" + return self._summary + + def show_summary(self, summary: DashboardSummary) -> None: + """Render ``summary``: the whole page is a function of it.""" + self._summary = summary + self._headline.setText(_HEADLINES.get(summary.status, summary.status)) + self._headline.setStyleSheet(_HEADLINE_STYLES.get(summary.status, "")) + self._reasons.setText("; ".join(summary.reasons)) + self._show_health(summary.health) + counts = summary.run_counts + self._run_counts.setText( + ", ".join(f"{count} {status}" for status, count in counts.items()) or "No runs recorded" + ) + fill_table(self._running_table, [_run_row(run) for run in summary.running_runs]) + fill_table(self._recent_table, [_run_row(run) for run in summary.recent_runs]) + fill_table( + self._integrity_table, + [ + [ + monitor.get("name"), + monitor.get("target"), + bool(monitor.get("running")), + _drift_text(monitor), + monitor.get("last_run"), + monitor.get("last_error"), + ] + for monitor in summary.integrity + ], + ) + fill_table( + self._storage_table, + [ + [ + backend.get("name"), + backend.get("kind"), + bool(backend.get("usable")), + backend.get("detail"), + ] + for backend in summary.storage + ], + ) + fill_table( + self._events_table, + [ + [ + event.get("timestamp"), + event.get("severity"), + event.get("type"), + event.get("source"), + event.get("subject"), + ] + for event in summary.events + ], + ) + + def _show_health(self, health: dict[str, Any]) -> None: + for key, _label in _HEALTH_ROWS: + self._health_labels[key].setText(cell_text(health.get(key))) + audit = health.get("audit") or {} + if not audit.get("configured"): + described = "not configured" + else: + described = "recording" if audit.get("active") else "configured, not recording" + self._health_labels["audit"].setText(described) + self._health_labels["time"].setText(cell_text(health.get("time"))) diff --git a/automation_file/ui/pages/files_page.py b/automation_file/ui/pages/files_page.py new file mode 100644 index 0000000..149ab4a --- /dev/null +++ b/automation_file/ui/pages/files_page.py @@ -0,0 +1,277 @@ +"""Files page: browse a storage URI, preview a file, copy, move, delete, make a directory.""" + +from __future__ import annotations + +from collections.abc import Callable + +from PySide6.QtCore import QThreadPool +from PySide6.QtWidgets import ( + QCheckBox, + QFormLayout, + QGroupBox, + QHBoxLayout, + QLineEdit, + QPlainTextEdit, + QSplitter, + QVBoxLayout, + QWidget, +) + +from automation_file.app import FileEntry, FilePreview, FileService +from automation_file.ui.log_widget import LogPanel +from automation_file.ui.pages.base import BasePage, fill_table, make_table + +_COLUMNS = ("Name", "Type", "Size", "Modified") +_DIRECTORY = "directory" +_FILE = "file" + + +class FilesPage(BasePage): + """A file manager over storage URIs; every operation goes through the Files service.""" + + title = "Files" + + def __init__(self, service: FileService, log: LogPanel, pool: QThreadPool) -> None: + super().__init__(log, pool) + self._service = service + self._entries: list[FileEntry] = [] + self._current = "" + self._note = "" + + root = QVBoxLayout(self) + root.setContentsMargins(12, 12, 12, 12) + root.setSpacing(10) + root.addLayout(self._location_row()) + splitter = QSplitter() + splitter.addWidget(self._listing_group()) + splitter.addWidget(self._preview_group()) + splitter.setStretchFactor(0, 3) + splitter.setStretchFactor(1, 2) + root.addWidget(splitter, 1) + root.addWidget(self._operations_group()) + root.addWidget(self.status_label()) + + # ------------------------------------------------------------------ layout + + def _location_row(self) -> QHBoxLayout: + row = QHBoxLayout() + self._uri = QLineEdit() + self._uri.setPlaceholderText( + "storage URI or local path, e.g. local:///data, s3://bucket/reports, memory://demo" + ) + self._uri.returnPressed.connect(self.open_location) + row.addWidget(self._uri, 1) + row.addWidget(self.make_button("Open", self.open_location)) + row.addWidget(self.make_button("Up", self.go_up)) + row.addWidget(self.make_button("Refresh", self.refresh)) + row.addWidget(self.make_button("Browse local…", self._on_browse)) + return row + + def _listing_group(self) -> QGroupBox: + box = QGroupBox("Entries") + layout = QVBoxLayout(box) + self._table = make_table(_COLUMNS) + self._table.cellDoubleClicked.connect(lambda _row, _column: self.open_selected()) + layout.addWidget(self._table) + return box + + def _preview_group(self) -> QGroupBox: + box = QGroupBox("Preview") + layout = QVBoxLayout(box) + self._preview = QPlainTextEdit() + self._preview.setReadOnly(True) + self._preview.setPlaceholderText("Double-click a file, or select it and press Preview.") + layout.addWidget(self._preview) + layout.addWidget(self.make_button("Preview selected", self.preview_selected)) + return box + + def _operations_group(self) -> QWidget: + box = QGroupBox("Operations on the selected entry") + form = QFormLayout(box) + self._target = QLineEdit() + self._target.setPlaceholderText("target URI; an existing directory receives the file") + self._overwrite = QCheckBox("Overwrite an existing target") + self._overwrite.setChecked(True) + transfer = QHBoxLayout() + transfer.addWidget(self._target, 1) + transfer.addWidget(self._overwrite) + transfer.addWidget(self.make_button("Copy", self.copy_selected)) + transfer.addWidget(self.make_button("Move", self.move_selected)) + form.addRow("Copy / move to", transfer) + + self._folder = QLineEdit() + self._folder.setPlaceholderText("name of a new directory below the current location") + self._recursive = QCheckBox("Delete a directory with its contents") + manage = QHBoxLayout() + manage.addWidget(self._folder, 1) + manage.addWidget(self.make_button("Create directory", self.create_directory)) + manage.addWidget(self._recursive) + manage.addWidget(self.make_button("Delete selected", self.delete_selected)) + form.addRow("Manage", manage) + return box + + # ------------------------------------------------------------------ navigation + + def location(self) -> str: + """Return the URI the listing shows.""" + return self._current + + def entries(self) -> list[FileEntry]: + """Return the entries the listing shows.""" + return list(self._entries) + + def refresh(self) -> None: + if self._current: + self._list(self._current) + + def open_location(self, uri: str | None = None) -> None: + """List the URI in the location field, or ``uri`` when one is given.""" + wanted = (self._uri.text() if uri is None else uri).strip() + if not wanted: + self.report_error("enter a storage URI or a local path first") + return + self._list(wanted) + + def go_up(self) -> None: + if self._current: + self._list(self._service.parent(self._current)) + + def open_selected(self) -> None: + """Enter the selected directory, or preview the selected file.""" + entry = self.selected_entry() + if entry is None: + return + if entry.is_dir: + self._list(entry.uri) + else: + self._load_preview(entry.uri) + + def selected_entry(self) -> FileEntry | None: + row = self._table.currentRow() + return self._entries[row] if 0 <= row < len(self._entries) else None + + def select_entry(self, name: str) -> bool: + """Select the entry called ``name``; return whether it is listed.""" + for row, entry in enumerate(self._entries): + if entry.name == name: + self._table.selectRow(row) + return True + return False + + def _on_browse(self) -> None: + chosen = self.pick_directory() + if chosen: + self._list(chosen) + + def _list(self, uri: str) -> None: + self.run_async( + lambda: (self._service.normalize(uri), self._service.list_dir(uri)), + f"list {uri}", + self._show_listing, + quiet=True, + ) + + def _show_listing(self, result: tuple[str, list[FileEntry]]) -> None: + self._current, self._entries = result + self._uri.setText(self._current) + fill_table( + self._table, + [ + [ + entry.name, + _DIRECTORY if entry.is_dir else _FILE, + None if entry.is_dir else entry.size, + entry.modified_at, + ] + for entry in self._entries + ], + ) + note, self._note = self._note, "" + self.report(note or f"{len(self._entries)} entries in {self._current}") + + # ------------------------------------------------------------------ preview + + def preview_selected(self) -> None: + entry = self.selected_entry() + if entry is None: + self.report_error("select a file to preview") + elif entry.is_dir: + self.report_error(f"{entry.name} is a directory; open it instead") + else: + self._load_preview(entry.uri) + + def preview_text(self) -> str: + """Return what the preview pane shows.""" + return self._preview.toPlainText() + + def _load_preview(self, uri: str) -> None: + self.run_async( + lambda: self._service.preview(uri), f"preview {uri}", self._show_preview, quiet=True + ) + + def _show_preview(self, preview: FilePreview) -> None: + self._preview.setPlainText(preview.text) + parts = [f"{preview.shown} of {preview.size} bytes of {preview.uri}"] + if preview.truncated: + parts.append("truncated") + if preview.note: + parts.append(preview.note) + self.report("; ".join(parts)) + + # ------------------------------------------------------------------ operations + + def copy_selected(self) -> None: + self._transfer("copy", self._service.copy) + + def move_selected(self) -> None: + self._transfer("move", self._service.move) + + def _transfer(self, verb: str, operation: Callable[[str, str, bool], FileEntry]) -> None: + entry = self.selected_entry() + target = self._target.text().strip() + if entry is None or not target: + self.report_error(f"select an entry and enter a target URI to {verb} it") + return + overwrite = self._overwrite.isChecked() + self.run_async( + lambda: operation(entry.uri, target, overwrite), + f"{verb} {entry.uri} to {target}", + lambda done: self._after_change(f"{verb}: {entry.uri} -> {done.uri}"), + ) + + def create_directory(self) -> None: + name = self._folder.text().strip() + if not self._current or not name: + self.report_error("open a location and enter a directory name first") + return + self.run_async( + lambda: self._make_directory(name), + f"create directory {name}", + lambda uri: self._after_change(f"created {uri}"), + ) + + def _make_directory(self, name: str) -> str: + uri = self._service.child(self._current, name) + self._service.mkdir(uri) + return uri + + def delete_selected(self) -> None: + entry = self.selected_entry() + if entry is None: + self.report_error("select an entry to delete") + return + if not self.confirm(f"Delete {entry.uri}? This cannot be undone."): + return + recursive = self._recursive.isChecked() + self.run_async( + lambda: self._service.delete(entry.uri, recursive), + f"delete {entry.uri}", + lambda _done: self._after_change(f"deleted {entry.uri}"), + ) + + def _after_change(self, message: str) -> None: + if not self._current: + self.report(message) + return + self._note = message + self.refresh() diff --git a/automation_file/ui/pages/integrity_page.py b/automation_file/ui/pages/integrity_page.py new file mode 100644 index 0000000..2c05b6e --- /dev/null +++ b/automation_file/ui/pages/integrity_page.py @@ -0,0 +1,248 @@ +"""Integrity page: baseline, verify, accept, and the named monitors.""" + +from __future__ import annotations + +from typing import Any + +from PySide6.QtCore import QThreadPool +from PySide6.QtWidgets import ( + QCheckBox, + QComboBox, + QDoubleSpinBox, + QFormLayout, + QGroupBox, + QHBoxLayout, + QLabel, + QLineEdit, + QVBoxLayout, +) + +from automation_file.app import IntegrityService, MonitorDrift +from automation_file.ui.log_widget import LogPanel +from automation_file.ui.pages.base import BasePage, fill_table, make_table, selected_cell + +_CHANGE_COLUMNS = ("Kind", "Path", "Previous path", "Fields", "Note") +_MONITOR_COLUMNS = ("Monitor", "Target", "Baseline", "Running", "Drift", "Last run", "Error") +_MAX_INTERVAL = 7 * 24 * 3600.0 +_DEFAULT_INTERVAL = 60.0 + + +def _drift_text(monitor: MonitorDrift) -> str: + if monitor.ok is None: + return "not verified yet" + return "none" if monitor.ok else f"{monitor.changes} change(s)" + + +class IntegrityPage(BasePage): + """Approve the state of a tree, compare it with what was approved, keep watching it.""" + + title = "Integrity" + + def __init__(self, service: IntegrityService, log: LogPanel, pool: QThreadPool) -> None: + super().__init__(log, pool) + self._service = service + self._report: dict[str, Any] | None = None + self._monitors: list[MonitorDrift] = [] + + root = QVBoxLayout(self) + root.setContentsMargins(12, 12, 12, 12) + root.setSpacing(10) + root.addWidget(self._target_group()) + root.addWidget(self._report_group(), 1) + root.addWidget(self._monitor_group(), 1) + root.addWidget(self.status_label()) + + # ------------------------------------------------------------------ layout + + def _target_group(self) -> QGroupBox: + box = QGroupBox("Target and baseline") + form = QFormLayout(box) + self._target = QLineEdit() + self._target.setPlaceholderText("the tree to check, e.g. s3://reports/2026 or a local path") + self._baseline = QLineEdit() + self._baseline.setPlaceholderText( + "where the approved state is kept, e.g. local:///var/lib/fa/reports.json" + ) + self._algorithm = QComboBox() + self._algorithm.addItems(self._service.algorithms()) + self._deep = QCheckBox("Deep: hash every file (off: only files whose size or time changed)") + self._deep.setChecked(True) + form.addRow("Target", self._target) + form.addRow("Baseline", self._baseline) + form.addRow("Algorithm", self._algorithm) + form.addRow(self._deep) + row = QHBoxLayout() + row.addWidget(self.make_button("Create baseline", self.create_baseline)) + row.addWidget(self.make_button("Verify", self.verify)) + row.addWidget(self.make_button("Accept current state", self.accept)) + row.addStretch() + form.addRow(row) + return box + + def _report_group(self) -> QGroupBox: + box = QGroupBox("Last verification") + layout = QVBoxLayout(box) + self._summary = QLabel("Nothing verified yet.") + self._summary.setWordWrap(True) + self._changes = make_table(_CHANGE_COLUMNS) + layout.addWidget(self._summary) + layout.addWidget(self._changes) + return box + + def _monitor_group(self) -> QGroupBox: + box = QGroupBox("Monitors") + layout = QVBoxLayout(box) + row = QHBoxLayout() + self._name = QLineEdit() + self._name.setPlaceholderText("monitor name") + self._interval = QDoubleSpinBox() + self._interval.setRange(1.0, _MAX_INTERVAL) + self._interval.setDecimals(0) + self._interval.setValue(_DEFAULT_INTERVAL) + self._interval.setSuffix(" s") + row.addWidget(self._name, 1) + row.addWidget(QLabel("every")) + row.addWidget(self._interval) + row.addWidget(self.make_button("Start monitor", self.start_monitor)) + row.addWidget(self.make_button("Stop selected", self.stop_selected)) + row.addWidget(self.make_button("Refresh", self.refresh)) + layout.addLayout(row) + self._monitor_table = make_table(_MONITOR_COLUMNS) + layout.addWidget(self._monitor_table) + return box + + # ------------------------------------------------------------------ baseline and verify + + def set_location(self, target: str, baseline: str) -> None: + """Fill in the target and the baseline fields.""" + self._target.setText(target) + self._baseline.setText(baseline) + + def last_report(self) -> dict[str, Any] | None: + """Return the drift report the page shows.""" + return self._report + + def create_baseline(self) -> None: + target, baseline = self._target.text(), self._baseline.text() + algorithm = self._algorithm.currentText() + self.run_async( + lambda: self._service.baseline(target, baseline, algorithm), + "create baseline", + lambda stored: self.report( + f"baseline of {stored['entries']} file(s) stored at {stored['baseline']}" + ), + ) + + def verify(self) -> None: + target, baseline = self._target.text(), self._baseline.text() + deep = self._deep.isChecked() + self.run_async( + lambda: self._service.verify(target, baseline, deep), "verify", self._show_report + ) + + def accept(self) -> None: + target, baseline = self._target.text(), self._baseline.text() + if not self.confirm( + "Approve the current state as the new baseline? Drift found so far will no longer be reported." + ): + return + self.run_async( + lambda: self._service.accept(target, baseline), + "accept current state", + lambda stored: self.report( + f"accepted {stored['entries']} file(s) as the baseline at {stored['baseline']}" + ), + ) + + def _show_report(self, report: dict[str, Any]) -> None: + self._report = report + changes = report.get("changes") or [] + fill_table( + self._changes, + [ + [ + change.get("kind"), + change.get("path"), + change.get("previous_path"), + change.get("fields"), + change.get("note"), + ] + for change in changes + ], + ) + counts = ", ".join( + f"{count} {kind}" for kind, count in (report.get("counts") or {}).items() if count + ) + verdict = "no drift" if report.get("ok") else f"drift: {counts or len(changes)}" + notes = " ".join(report.get("notes") or ()) + text = ( + f"{report.get('target')}: {verdict}; {report.get('hashed')} of " + f"{report.get('checked')} file(s) hashed. {notes}" + ).strip() + self._summary.setText(text) + if report.get("ok"): + self.report(text) + else: + self.report_error(text) + + # ------------------------------------------------------------------ monitors + + def monitors(self) -> list[MonitorDrift]: + """Return the monitors the table shows.""" + return list(self._monitors) + + def refresh(self) -> None: + self.run_async( + self._service.drift, "read monitors", self._show_monitors, key="refresh", quiet=True + ) + + def _show_monitors(self, monitors: list[MonitorDrift]) -> None: + self._monitors = monitors + fill_table( + self._monitor_table, + [ + [ + monitor.name, + monitor.target, + monitor.baseline, + monitor.running, + _drift_text(monitor), + monitor.last_run, + monitor.last_error, + ] + for monitor in monitors + ], + ) + + def start_monitor(self) -> None: + name, interval = self._name.text(), float(self._interval.value()) + target, baseline = self._target.text(), self._baseline.text() + self.run_async( + lambda: self._service.start_monitor(name, target, baseline, interval), + f"start monitor {name.strip()}", + lambda status: self._after_change( + f"monitor {status['name']} verifies {status['target']} every {interval:.0f} s" + ), + ) + + def stop_selected(self) -> None: + name = selected_cell(self._monitor_table) + if name is None: + self.report_error("select a monitor to stop") + return + self.run_async( + lambda: self._service.stop_monitor(name), + f"stop monitor {name}", + lambda _status: self._after_change(f"monitor {name} stopped"), + ) + + def _after_change(self, message: str) -> None: + self.report(message) + self.refresh() + + def shutdown(self) -> None: + """Stop the monitors this window started, so they do not outlive it.""" + stopped = self._service.stop_started() + if stopped: + self._log.append_line(f"{self.title}: stopped monitor(s) {', '.join(stopped)}") + super().shutdown() diff --git a/automation_file/ui/pages/notifications_page.py b/automation_file/ui/pages/notifications_page.py new file mode 100644 index 0000000..83257bf --- /dev/null +++ b/automation_file/ui/pages/notifications_page.py @@ -0,0 +1,240 @@ +"""Notifications page: the registered sinks, the routes, and a test message.""" + +from __future__ import annotations + +from typing import Any + +from PySide6.QtCore import QThreadPool +from PySide6.QtWidgets import ( + QComboBox, + QFormLayout, + QGroupBox, + QHBoxLayout, + QLabel, + QLineEdit, + QVBoxLayout, +) + +from automation_file.app import NotificationService +from automation_file.ui.log_widget import LogPanel +from automation_file.ui.pages.base import BasePage, fill_table, make_table, selected_cell + +_SINK_COLUMNS = ("Name", "Type", "Delivers to") +_ROUTE_COLUMNS = ("Route", "Sinks", "Types", "Sources", "Min severity", "Dedup (s)", "Rate") +_ALL_SINKS = "All sinks" +_SINK_IDENTITY = ("name", "type") +_DEFAULT_SEVERITY = "warning" + + +def _delivers_to(sink: dict[str, Any]) -> str: + """Describe where a sink delivers, from what its description holds beyond name and type.""" + details = {key: value for key, value in sink.items() if key not in _SINK_IDENTITY} + return ", ".join(f"{key}: {value}" for key, value in details.items()) + + +def _rate(route: dict[str, Any]) -> str: + if not route.get("rate_limit"): + return "unlimited" + return f"{route['rate_limit']} per {route.get('rate_period')} s" + + +class NotificationsPage(BasePage): + """Shows sinks without their secrets, edits routes, sends a test message.""" + + title = "Notifications" + + def __init__(self, service: NotificationService, log: LogPanel, pool: QThreadPool) -> None: + super().__init__(log, pool) + self._service = service + self._sinks: list[dict[str, Any]] = [] + self._routes: list[dict[str, Any]] = [] + + root = QVBoxLayout(self) + root.setContentsMargins(12, 12, 12, 12) + root.setSpacing(10) + root.addWidget(self._sinks_group(), 1) + root.addWidget(self._routes_group(), 1) + root.addWidget(self._route_form_group()) + root.addWidget(self.status_label()) + + # ------------------------------------------------------------------ layout + + def _sinks_group(self) -> QGroupBox: + box = QGroupBox("Sinks") + layout = QVBoxLayout(box) + layout.addWidget( + self.muted_label( + "Sinks are registered in code or from the configuration file (Settings). " + "A webhook URL, a token or a password is never shown." + ) + ) + self._sink_table = make_table(_SINK_COLUMNS) + layout.addWidget(self._sink_table) + row = QHBoxLayout() + self._test_sink = QComboBox() + self._test_sink.addItem(_ALL_SINKS) + self._test_subject = QLineEdit() + self._test_subject.setPlaceholderText("subject of the test message (optional)") + row.addWidget(QLabel("Test")) + row.addWidget(self._test_sink) + row.addWidget(self._test_subject, 1) + row.addWidget(self.make_button("Send test message", self.send_test)) + row.addWidget(self.make_button("Refresh", self.refresh)) + layout.addLayout(row) + return box + + def _routes_group(self) -> QGroupBox: + box = QGroupBox("Routes") + layout = QVBoxLayout(box) + self._router_state = QLabel("") + self._route_table = make_table(_ROUTE_COLUMNS) + layout.addWidget(self._router_state) + layout.addWidget(self._route_table) + layout.addWidget(self.make_button("Remove selected route", self.remove_selected)) + return box + + def _route_form_group(self) -> QGroupBox: + box = QGroupBox("Add or replace a route") + form = QFormLayout(box) + self._name = QLineEdit() + self._name.setPlaceholderText("unique route name; an existing name is replaced") + self._route_sinks = QLineEdit() + self._route_sinks.setPlaceholderText("sink names, comma-separated; empty means every sink") + self._types = QLineEdit() + self._types.setPlaceholderText( + "pipeline.*, task.failed, integrity.violation; empty means all" + ) + self._sources = QLineEdit() + self._sources.setPlaceholderText("pipeline, scheduler, storage; empty means all") + self._severity = QComboBox() + self._severity.addItems(self._service.severities()) + self._severity.setCurrentText(_DEFAULT_SEVERITY) + self._dedup = QLineEdit() + self._dedup.setPlaceholderText("seconds a repeat is dropped; empty: 300, 0: off") + self._rate_limit = QLineEdit() + self._rate_limit.setPlaceholderText("messages per period; empty or 0: unlimited") + self._rate_period = QLineEdit() + self._rate_period.setPlaceholderText("seconds; empty: 60") + form.addRow("Name", self._name) + form.addRow("Sinks", self._route_sinks) + form.addRow("Event types", self._types) + form.addRow("Sources", self._sources) + form.addRow("Minimum severity", self._severity) + throttle = QHBoxLayout() + throttle.addWidget(self._dedup) + throttle.addWidget(self._rate_limit) + throttle.addWidget(self._rate_period) + form.addRow("Dedup / rate / period", throttle) + form.addRow(self.make_button("Add route", self.add_route)) + return box + + # ------------------------------------------------------------------ data + + def sinks(self) -> list[dict[str, Any]]: + """Return the sinks the table shows.""" + return list(self._sinks) + + def routes(self) -> list[dict[str, Any]]: + """Return the routes the table shows.""" + return list(self._routes) + + def refresh(self) -> None: + self.run_async(self._read, "read sinks and routes", self._show, key="refresh", quiet=True) + + def _read(self) -> dict[str, Any]: + return { + "sinks": self._service.sinks(), + "routes": self._service.routes(), + "active": self._service.router_active(), + } + + def _show(self, data: dict[str, Any]) -> None: + self._sinks, self._routes = data["sinks"], data["routes"] + fill_table( + self._sink_table, + [[sink.get("name"), sink.get("type"), _delivers_to(sink)] for sink in self._sinks], + ) + fill_table( + self._route_table, + [ + [ + route.get("name"), + route.get("sinks") or "every sink", + route.get("types") or "every type", + route.get("sources") or "every source", + route.get("min_severity"), + route.get("dedup_seconds"), + _rate(route), + ] + for route in self._routes + ], + ) + self._router_state.setText( + "The router is delivering events." + if data["active"] + else "The router is not running: it starts with the first route." + ) + chosen = self._test_sink.currentText() + self._test_sink.clear() + self._test_sink.addItem(_ALL_SINKS) + self._test_sink.addItems([str(sink.get("name")) for sink in self._sinks]) + self._test_sink.setCurrentText(chosen) + + # ------------------------------------------------------------------ routes + + def route_options(self) -> dict[str, Any]: + """Return the route the form describes, as the service takes it.""" + return { + "name": self._name.text().strip(), + "sinks": self._route_sinks.text(), + "types": self._types.text(), + "sources": self._sources.text(), + "min_severity": self._severity.currentText(), + "dedup_seconds": self._dedup.text().strip(), + "rate_limit": self._rate_limit.text().strip(), + "rate_period": self._rate_period.text().strip(), + } + + def add_route(self) -> None: + options = self.route_options() + self.run_async( + lambda: self._service.add_route(options), + f"add route {options['name']}", + lambda route: self._after_change(f"route {route['name']} added"), + ) + + def remove_selected(self) -> None: + name = selected_cell(self._route_table) + if name is None: + self.report_error("select a route to remove") + return + self.run_async( + lambda: self._service.remove_route(name), + f"remove route {name}", + lambda removed: self._after_change( + f"route {name} removed" if removed else f"there was no route {name}" + ), + ) + + def _after_change(self, message: str) -> None: + self.report(message) + self.refresh() + + # ------------------------------------------------------------------ test message + + def send_test(self) -> None: + chosen = self._test_sink.currentText() + sink = None if chosen == _ALL_SINKS else chosen + subject = self._test_subject.text().strip() + self.run_async( + lambda: self._service.send_test(sink, subject), + f"send test message to {chosen}", + self._show_outcomes, + ) + + def _show_outcomes(self, outcomes: dict[str, str]) -> None: + text = "; ".join(f"{name}: {outcome}" for name, outcome in outcomes.items()) + if all(outcome == "sent" for outcome in outcomes.values()): + self.report(f"test message: {text}") + else: + self.report_error(f"test message: {text}") diff --git a/automation_file/ui/pages/pipeline_canvas.py b/automation_file/ui/pages/pipeline_canvas.py new file mode 100644 index 0000000..f813d57 --- /dev/null +++ b/automation_file/ui/pages/pipeline_canvas.py @@ -0,0 +1,395 @@ +"""Pipeline canvas: tasks as nodes that can be dragged, dependencies as arrows between them. + +The canvas is a view over a :class:`~automation_file.app.PipelineDraft` and +keeps no state of its own beyond the selection: every node and every arrow is +rebuilt from the draft when the draft says it changed, and dragging a node +writes its position back with ``draft.set_position``. + +Two tasks are connected by selecting them one after the other; the order of the +selection is the direction of the arrow, and :meth:`PipelineCanvas.selected_tasks` +returns the tasks in that order. +""" + +from __future__ import annotations + +from collections.abc import Callable, Sequence +from typing import Any + +from PySide6.QtCore import QMimeData, QPointF, QRectF, Qt, Signal +from PySide6.QtGui import ( + QBrush, + QColor, + QPainter, + QPainterPath, + QPainterPathStroker, + QPen, +) +from PySide6.QtWidgets import ( + QAbstractItemView, + QGraphicsItem, + QGraphicsPathItem, + QGraphicsRectItem, + QGraphicsScene, + QGraphicsView, + QListWidget, + QListWidgetItem, +) + +from automation_file.app import PipelineDraft +from automation_file.app.pipeline_draft import CHANGE_HEADER, CHANGE_POSITION + +ACTION_MIME = "application/x-automation-file-action" +NODE_WIDTH = 190.0 +NODE_HEIGHT = 62.0 + +_CORNER = 8.0 +_PADDING = 8.0 +_ARROW_LENGTH = 11.0 +_ARROW_HALF_WIDTH = 5.0 +_MIN_BEND = 40.0 +_EDGE_HIT_WIDTH = 12.0 +_NO_ACTION = "(no action)" +_NODE_FILL = "#f4f6f8" +_NODE_BORDER = "#5f6b7a" +_SELECTED = "#1a73e8" +_TEXT = "#1d1f21" +_MUTED_TEXT = "#5f6b7a" +_EDGE = "#5f6b7a" +#: Task status -> the colour of its node while a run is followed. +STATUS_COLOURS: dict[str, str] = { + "pending": "#eceff1", + "planned": "#e3f2fd", + "running": "#bbdefb", + "succeeded": "#c8e6c9", + "failed": "#ffcdd2", + "timeout": "#ffcdd2", + "cancelled": "#ffe0b2", + "skipped": "#fff9c4", +} + +MoveHandler = Callable[[str, float, float], None] + + +class TaskNode(QGraphicsRectItem): + """One task on the canvas: its ID, its action and, during a run, its status.""" + + def __init__(self, task_id: str, action: str, on_moved: MoveHandler) -> None: + super().__init__(0.0, 0.0, NODE_WIDTH, NODE_HEIGHT) + self.task_id = task_id + self.action = action + self.status = "" + self._on_moved = on_moved + self.setFlag(QGraphicsItem.GraphicsItemFlag.ItemIsMovable, True) + self.setFlag(QGraphicsItem.GraphicsItemFlag.ItemIsSelectable, True) + self.setFlag(QGraphicsItem.GraphicsItemFlag.ItemSendsGeometryChanges, True) + self.setZValue(1.0) + self.setToolTip(f"{task_id}\n{action or _NO_ACTION}") + + def output_point(self) -> QPointF: + """Where an arrow leaves the node: the middle of its right side.""" + return self.pos() + QPointF(NODE_WIDTH, NODE_HEIGHT / 2) + + def input_point(self) -> QPointF: + """Where an arrow reaches the node: the middle of its left side.""" + return self.pos() + QPointF(0.0, NODE_HEIGHT / 2) + + def itemChange(self, change: Any, value: Any) -> Any: # noqa: N802 # pylint: disable=invalid-name — Qt override + if change == QGraphicsItem.GraphicsItemChange.ItemPositionHasChanged: + self._on_moved(self.task_id, self.pos().x(), self.pos().y()) + return super().itemChange(change, value) + + def paint(self, painter: QPainter, _option: Any, _widget: Any = None) -> None: + selected = self.isSelected() + painter.setRenderHint(QPainter.RenderHint.Antialiasing) + painter.setPen( + QPen(QColor(_SELECTED if selected else _NODE_BORDER), 2.5 if selected else 1.2) + ) + painter.setBrush(QBrush(QColor(STATUS_COLOURS.get(self.status, _NODE_FILL)))) + painter.drawRoundedRect(self.rect(), _CORNER, _CORNER) + half = NODE_HEIGHT / 2 + text_width = NODE_WIDTH - 2 * _PADDING + font = painter.font() + font.setBold(True) + painter.setFont(font) + painter.setPen(QColor(_TEXT)) + metrics = painter.fontMetrics() + painter.drawText( + QRectF(_PADDING, 4.0, text_width, half - 4.0), + int(Qt.AlignmentFlag.AlignLeft | Qt.AlignmentFlag.AlignVCenter), + metrics.elidedText(self.task_id, Qt.TextElideMode.ElideRight, int(text_width)), + ) + font.setBold(False) + painter.setFont(font) + painter.setPen(QColor(_MUTED_TEXT)) + caption = self.action or _NO_ACTION + if self.status: + caption = f"{caption} [{self.status}]" + painter.drawText( + QRectF(_PADDING, half, text_width, half - 4.0), + int(Qt.AlignmentFlag.AlignLeft | Qt.AlignmentFlag.AlignVCenter), + painter.fontMetrics().elidedText( + caption, Qt.TextElideMode.ElideMiddle, int(text_width) + ), + ) + + +class EdgeItem(QGraphicsPathItem): + """A dependency: an arrow from the upstream task to the task that depends on it.""" + + def __init__(self, upstream: TaskNode, downstream: TaskNode) -> None: + super().__init__() + self.upstream_id = upstream.task_id + self.downstream_id = downstream.task_id + self._upstream = upstream + self._downstream = downstream + self.setFlag(QGraphicsItem.GraphicsItemFlag.ItemIsSelectable, True) + self.setZValue(0.0) + self.setPen(QPen(QColor(_EDGE), 1.6)) + self.setToolTip(f"{self.downstream_id} depends on {self.upstream_id}") + self.refresh() + + def refresh(self) -> None: + """Redraw the arrow between the current positions of its two nodes.""" + start, end = self._upstream.output_point(), self._downstream.input_point() + bend = max(abs(end.x() - start.x()) / 2, _MIN_BEND) + path = QPainterPath(start) + path.cubicTo(start + QPointF(bend, 0.0), end - QPointF(bend, 0.0), end) + path.moveTo(end) + path.lineTo(end + QPointF(-_ARROW_LENGTH, -_ARROW_HALF_WIDTH)) + path.moveTo(end) + path.lineTo(end + QPointF(-_ARROW_LENGTH, _ARROW_HALF_WIDTH)) + self.setPath(path) + + def shape(self) -> QPainterPath: + """Make the thin curve easy to click: a band around it counts as the arrow.""" + stroker = QPainterPathStroker() + stroker.setWidth(_EDGE_HIT_WIDTH) + return stroker.createStroke(self.path()) + + def itemChange(self, change: Any, value: Any) -> Any: # noqa: N802 # pylint: disable=invalid-name — Qt override + if change == QGraphicsItem.GraphicsItemChange.ItemSelectedHasChanged: + self.setPen(QPen(QColor(_SELECTED if value else _EDGE), 2.6 if value else 1.6)) + return super().itemChange(change, value) + + +class ActionPalette(QListWidget): + """The registered actions; an entry can be dragged onto the canvas to add a task.""" + + def __init__(self) -> None: + super().__init__() + self.setDragEnabled(True) + self.setSelectionMode(QAbstractItemView.SelectionMode.SingleSelection) + + def set_actions(self, names: list[str]) -> None: + """Replace the entries.""" + self.clear() + self.addItems(names) + + def filter_actions(self, text: str) -> None: + """Show only the entries that contain ``text``, whatever its case.""" + wanted = text.strip().casefold() + for row in range(self.count()): + entry = self.item(row) + entry.setHidden(bool(wanted) and wanted not in entry.text().casefold()) + + def selected_action(self) -> str: + """Return the selected action name, or an empty text.""" + entry = self.currentItem() + return "" if entry is None or entry.isHidden() else entry.text() + + def mimeData(self, items: Sequence[QListWidgetItem]) -> QMimeData: # noqa: N802 # pylint: disable=invalid-name — Qt override + data = QMimeData() + if items: + data.setData(ACTION_MIME, items[0].text().encode("utf-8")) + data.setText(items[0].text()) + return data + + +class PipelineCanvas(QGraphicsView): + """Shows a draft as a graph and lets the user arrange and select its tasks.""" + + #: The selected task IDs, in the order they were selected. + selection_changed = Signal(list) + #: An action name dropped from the palette, and where: ``(name, x, y)`` in scene coordinates. + action_dropped = Signal(str, float, float) + + def __init__(self) -> None: + super().__init__() + self._scene = QGraphicsScene(self) + self.setScene(self._scene) + self.setRenderHint(QPainter.RenderHint.Antialiasing) + self.setDragMode(QGraphicsView.DragMode.RubberBandDrag) + self.setAcceptDrops(True) + self._draft: PipelineDraft | None = None + self._nodes: dict[str, TaskNode] = {} + self._edges: list[EdgeItem] = [] + self._order: list[str] = [] + self._statuses: dict[str, str] = {} + self._syncing = False + self._scene.selectionChanged.connect(self._on_selection_changed) + + # ------------------------------------------------------------------ the draft + + def set_draft(self, draft: PipelineDraft | None) -> None: + """Show ``draft`` from now on and follow its changes.""" + if self._draft is not None: + self._draft.remove_listener(self._on_draft_changed) + self._draft = draft + self._statuses = {} + self._order = [] + if draft is not None: + draft.add_listener(self._on_draft_changed) + self.sync() + + def sync(self) -> None: + """Rebuild every node and every arrow from the draft, keeping the selection.""" + keep = list(self._order) + self._syncing = True + try: + self._scene.clear() + self._nodes, self._edges = {}, [] + self._build() + self._order = [task_id for task_id in keep if task_id in self._nodes] + for task_id in self._order: + self._nodes[task_id].setSelected(True) + finally: + self._syncing = False + self.selection_changed.emit(list(self._order)) + + def _build(self) -> None: + if self._draft is None: + return + for task in self._draft.tasks: + node = TaskNode(task.task_id, task.action, self._node_moved) + node.status = self._statuses.get(task.task_id, "") + node.setPos(task.x, task.y) + self._scene.addItem(node) + self._nodes[task.task_id] = node + for upstream, downstream in self._draft.edges(): + edge = EdgeItem(self._nodes[upstream], self._nodes[downstream]) + self._scene.addItem(edge) + self._edges.append(edge) + + def _on_draft_changed(self, change: str) -> None: + if change == CHANGE_POSITION: + self._place_nodes() + elif change != CHANGE_HEADER: + self.sync() + + def _place_nodes(self) -> None: + """Move the nodes to where the draft says they are (after an automatic layout).""" + if self._draft is None or self._syncing: + return + self._syncing = True + try: + for task_id, (x, y) in self._draft.positions().items(): + node = self._nodes.get(task_id) + if node is not None and (node.pos().x(), node.pos().y()) != (x, y): + node.setPos(x, y) + for edge in self._edges: + edge.refresh() + finally: + self._syncing = False + + def _node_moved(self, task_id: str, x: float, y: float) -> None: + if self._syncing or self._draft is None: + return + self._syncing = True + try: + self._draft.set_position(task_id, x, y) + finally: + self._syncing = False + for edge in self._edges: + if task_id in (edge.upstream_id, edge.downstream_id): + edge.refresh() + + # ------------------------------------------------------------------ what it shows + + def node(self, task_id: str) -> TaskNode | None: + """Return the node of the task ``task_id``, or ``None``.""" + return self._nodes.get(task_id) + + def node_ids(self) -> list[str]: + """Return the IDs of the tasks on the canvas, in the draft's order.""" + return list(self._nodes) + + def edge_pairs(self) -> list[tuple[str, str]]: + """Return ``(upstream, downstream)`` for every arrow on the canvas.""" + return [(edge.upstream_id, edge.downstream_id) for edge in self._edges] + + def set_statuses(self, statuses: dict[str, str]) -> None: + """Colour the nodes by task status; an empty mapping clears the colours.""" + self._statuses = dict(statuses) + for task_id, node in self._nodes.items(): + node.status = self._statuses.get(task_id, "") + node.update() + + # ------------------------------------------------------------------ selection + + def selected_tasks(self) -> list[str]: + """Return the selected task IDs, in the order they were selected.""" + return list(self._order) + + def selected_edges(self) -> list[tuple[str, str]]: + """Return ``(upstream, downstream)`` for every selected arrow.""" + return [ + (item.upstream_id, item.downstream_id) + for item in self._scene.selectedItems() + if isinstance(item, EdgeItem) + ] + + def select_tasks(self, task_ids: list[str]) -> None: + """Select exactly ``task_ids``, in that order, and report the selection once.""" + chosen = [task_id for task_id in dict.fromkeys(task_ids) if task_id in self._nodes] + self._syncing = True + try: + self._scene.clearSelection() + for task_id in chosen: + self._nodes[task_id].setSelected(True) + finally: + self._syncing = False + self._order = chosen + self.selection_changed.emit(list(chosen)) + + def select_edge(self, upstream: str, downstream: str) -> bool: + """Select the arrow from ``upstream`` to ``downstream``; return whether there is one.""" + for edge in self._edges: + if (edge.upstream_id, edge.downstream_id) == (upstream, downstream): + self._scene.clearSelection() + edge.setSelected(True) + return True + return False + + def _on_selection_changed(self) -> None: + if self._syncing: + return + current = [ + item.task_id for item in self._scene.selectedItems() if isinstance(item, TaskNode) + ] + kept = [task_id for task_id in self._order if task_id in current] + self._order = kept + [task_id for task_id in current if task_id not in kept] + self.selection_changed.emit(list(self._order)) + + # ------------------------------------------------------------------ drop from the palette + + def dragEnterEvent(self, event: Any) -> None: # noqa: N802 # pylint: disable=invalid-name — Qt override + if event.mimeData().hasFormat(ACTION_MIME): + event.acceptProposedAction() + else: + super().dragEnterEvent(event) + + def dragMoveEvent(self, event: Any) -> None: # noqa: N802 # pylint: disable=invalid-name — Qt override + if event.mimeData().hasFormat(ACTION_MIME): + event.acceptProposedAction() + else: + super().dragMoveEvent(event) + + def dropEvent(self, event: Any) -> None: # noqa: N802 # pylint: disable=invalid-name — Qt override + data = event.mimeData() + if not data.hasFormat(ACTION_MIME): + super().dropEvent(event) + return + action = bytes(data.data(ACTION_MIME).data()).decode("utf-8") + point = self.mapToScene(event.position().toPoint()) + event.acceptProposedAction() + self.action_dropped.emit(action, point.x(), point.y()) diff --git a/automation_file/ui/pages/pipelines_page.py b/automation_file/ui/pages/pipelines_page.py new file mode 100644 index 0000000..dded663 --- /dev/null +++ b/automation_file/ui/pages/pipelines_page.py @@ -0,0 +1,646 @@ +"""Pipelines page: the visual editor of a pipeline definition, and its runs. + +The page owns one :class:`~automation_file.app.PipelineDraft`. The canvas and +the task form are views over it; every button calls the draft or the Pipelines +service and then shows what they return. Nothing about pipelines is decided +here. +""" + +from __future__ import annotations + +import json +from collections.abc import Callable +from typing import Any + +from PySide6.QtCore import Qt, QThreadPool, QTimer +from PySide6.QtWidgets import ( + QGroupBox, + QHBoxLayout, + QLabel, + QLineEdit, + QPushButton, + QScrollArea, + QSpinBox, + QSplitter, + QVBoxLayout, + QWidget, +) + +from automation_file.app import ( + AppException, + PipelineDraft, + PipelineService, + Problem, + parse_json_text, +) +from automation_file.exceptions import FileAutomationException +from automation_file.ui.log_widget import LogPanel +from automation_file.ui.pages.base import BasePage +from automation_file.ui.pages.pipeline_canvas import ( + NODE_HEIGHT, + NODE_WIDTH, + ActionPalette, + PipelineCanvas, +) +from automation_file.ui.pages.run_panel import RunPanel +from automation_file.ui.pages.task_form import TaskForm + +_FOLLOW_INTERVAL_MS = 500 +_HISTORY_LIMIT = 50 +_MAX_WORKERS = 64 +_FILE_FILTER = "Pipeline definitions (*.yaml *.yml *.json);;All files (*)" +_UNSAVED = "not saved yet" +_CONNECT_HINT = "Connect (select two tasks)" +_RUN_PARAMETERS = "the run parameters" +_DEFAULT_PARAMETERS = "the default parameters" +_PAIR = 2 +_FINISHED_OK = "succeeded" +#: Starting widths of the action list, the canvas and the task form. +_EDITOR_WIDTHS = (220, 560, 420) + + +class PipelinesPage(BasePage): + """Build a pipeline on a canvas, check it, run it and follow the run.""" + + title = "Pipelines" + + def __init__(self, service: PipelineService, log: LogPanel, pool: QThreadPool) -> None: + super().__init__(log, pool) + self._service = service + self._draft = service.new_draft() + self._path: str | None = None + self._run_id: str | None = None + self._following = False + self._loading = False + + self._canvas = PipelineCanvas() + self._palette = ActionPalette() + self._form = TaskForm(service) + self._panel = RunPanel() + self._file_label = QLabel(_UNSAVED) + self._connect_button = QPushButton(_CONNECT_HINT) + self._connect_button.clicked.connect(lambda _checked=False: self.connect_selected()) + + editor = QSplitter(Qt.Orientation.Horizontal) + editor.addWidget(self._palette_group()) + editor.addWidget(self._canvas_group()) + editor.addWidget(self._form_group()) + editor.setStretchFactor(0, 1) + editor.setStretchFactor(1, 3) + editor.setStretchFactor(2, 2) + editor.setSizes(list(_EDITOR_WIDTHS)) + lower = QWidget() + lower_layout = QVBoxLayout(lower) + lower_layout.setContentsMargins(0, 0, 0, 0) + lower_layout.addLayout(self._run_bar()) + lower_layout.addWidget(self._panel, 1) + body = QSplitter(Qt.Orientation.Vertical) + body.addWidget(editor) + body.addWidget(lower) + body.setStretchFactor(0, 3) + body.setStretchFactor(1, 2) + + root = QVBoxLayout(self) + root.setContentsMargins(12, 12, 12, 12) + root.setSpacing(8) + root.addLayout(self._tool_bar()) + root.addLayout(self._header_bar()) + root.addWidget(body, 1) + root.addWidget(self.status_label()) + + self._timer = QTimer(self) + self._timer.setInterval(_FOLLOW_INTERVAL_MS) + self._timer.timeout.connect(self.poll) + self._canvas.selection_changed.connect(self._on_selection) + self._canvas.action_dropped.connect(self._on_action_dropped) + self._form.applied.connect(self._on_task_applied) + self._panel.task_chosen.connect(lambda task_id: self._canvas.select_tasks([task_id])) + self._panel.run_chosen.connect(self.follow_run) + self._set_draft(self._draft, None) + + # ------------------------------------------------------------------ layout + + def _tool_bar(self) -> QHBoxLayout: + row = QHBoxLayout() + row.addWidget(self.make_button("New", self.new_pipeline)) + row.addWidget(self.make_button("Open…", self.open_pipeline)) + row.addWidget(self.make_button("Save", self.save_pipeline)) + row.addWidget(self.make_button("Save as…", self.save_pipeline_as)) + row.addWidget(self.make_button("Auto layout", self.auto_layout)) + row.addWidget(self._file_label, 1) + return row + + def _header_bar(self) -> QHBoxLayout: + row = QHBoxLayout() + self._name = QLineEdit() + self._description = QLineEdit() + self._description.setPlaceholderText("description") + self._max_workers = QSpinBox() + self._max_workers.setRange(1, _MAX_WORKERS) + self._defaults = QLineEdit() + self._defaults.setPlaceholderText('default parameters (JSON), e.g. {"date": "2026-10-08"}') + self._cron = QLineEdit() + self._cron.setPlaceholderText("schedule (cron), kept for the scheduler") + for field in (self._name, self._description, self._defaults, self._cron): + field.editingFinished.connect(self._apply_header) + self._max_workers.valueChanged.connect(lambda _value: self._apply_header()) + row.addWidget(QLabel("Name")) + row.addWidget(self._name, 2) + row.addWidget(QLabel("Max workers")) + row.addWidget(self._max_workers) + row.addWidget(self._description, 3) + row.addWidget(self._defaults, 3) + row.addWidget(self._cron, 2) + return row + + def _palette_group(self) -> QGroupBox: + box = QGroupBox("Actions") + layout = QVBoxLayout(box) + self._filter = QLineEdit() + self._filter.setPlaceholderText("filter, e.g. storage") + self._filter.textChanged.connect(self._palette.filter_actions) + self._palette.itemDoubleClicked.connect(lambda item: self.add_task(item.text())) + layout.addWidget(self._filter) + layout.addWidget(self._palette, 1) + layout.addWidget(self.muted_label("Drag an action onto the canvas, or double-click it.")) + layout.addWidget(self.make_button("Add task", self.add_task)) + return box + + def _canvas_group(self) -> QGroupBox: + box = QGroupBox("Tasks and dependencies") + layout = QVBoxLayout(box) + row = QHBoxLayout() + row.addWidget(self._connect_button) + row.addWidget(self.make_button("Disconnect", self.disconnect_selected)) + row.addWidget(self.make_button("Remove selected", self.remove_selected)) + row.addStretch() + layout.addLayout(row) + layout.addWidget(self._canvas, 1) + layout.addWidget( + self.muted_label( + "Select the upstream task, then Ctrl-click the task that depends on it, and " + "press Connect. An arrow points from a task to the one that waits for it." + ) + ) + return box + + def _form_group(self) -> QGroupBox: + box = QGroupBox("Selected task") + layout = QVBoxLayout(box) + scroll = QScrollArea() + scroll.setWidgetResizable(True) + scroll.setWidget(self._form) + layout.addWidget(scroll) + return box + + def _run_bar(self) -> QVBoxLayout: + self._run_params = QLineEdit() + self._run_params.setPlaceholderText('a JSON object, e.g. {"date": "2026-10-09"}') + params = QHBoxLayout() + params.addWidget(QLabel("Run parameters")) + params.addWidget(self._run_params, 1) + row = QHBoxLayout() + buttons: tuple[tuple[str, Callable[[], Any]], ...] = ( + ("Validate", self.validate), + ("Dry run", self.dry_run), + ("Test task", self.test_task), + ("Run", self.run), + ("Resume", self.resume), + ("Retry", self.retry), + ("Cancel", self.cancel), + ("Refresh history", self.refresh_history), + ("Follow selected run", self.follow_selected), + ) + for label, handler in buttons: + row.addWidget(self.make_button(label, handler)) + row.addStretch() + bar = QVBoxLayout() + bar.addLayout(params) + bar.addLayout(row) + return bar + + # ------------------------------------------------------------------ parts, for callers + + def draft(self) -> PipelineDraft: + """Return the draft the page edits.""" + return self._draft + + def canvas(self) -> PipelineCanvas: + return self._canvas + + def form(self) -> TaskForm: + return self._form + + def panel(self) -> RunPanel: + return self._panel + + def action_palette(self) -> ActionPalette: + return self._palette + + def current_run(self) -> str | None: + """Return the ID of the run the page follows.""" + return self._run_id + + def set_run_params(self, text: str) -> None: + """Fill in the run parameters field.""" + self._run_params.setText(text) + + # ------------------------------------------------------------------ the draft + + def _set_draft(self, draft: PipelineDraft, path: str | None) -> None: + self._draft.remove_listener(self._on_draft_changed) + self._draft, self._path = draft, path + self._run_id, self._following = None, False + self._timer.stop() + draft.add_listener(self._on_draft_changed) + self._canvas.set_draft(draft) + self._form.set_draft(draft) + self._load_header() + self._panel.show_run(None) + self._panel.show_events([]) + self._panel.clear_problems() + self._show_file() + + def _load_header(self) -> None: + self._loading = True + try: + self._name.setText(self._draft.name) + self._description.setText(self._draft.description) + self._max_workers.setValue(self._draft.max_workers) + params = self._draft.params + self._defaults.setText(_compact_json(params) if params else "") + self._cron.setText((self._draft.schedule or {}).get("cron", "")) + finally: + self._loading = False + + def _apply_header(self) -> None: + if self._loading: + return + try: + params = parse_json_text(self._defaults.text(), _DEFAULT_PARAMETERS, empty={}) + self._write_header(params) + except AppException as error: + self.report_error(str(error)) + + def _write_header(self, params: Any) -> None: + """Write the header fields that differ from the draft, so reading them changes nothing.""" + draft = self._draft + schedule = draft.schedule or {} + with draft.batch(): + if self._name.text().strip() != draft.name: + draft.set_name(self._name.text()) + if self._description.text() != draft.description: + draft.set_description(self._description.text()) + if self._max_workers.value() != draft.max_workers: + draft.set_max_workers(self._max_workers.value()) + if params != draft.params: + draft.set_params(params) + if self._cron.text().strip() != schedule.get("cron", ""): + draft.set_schedule(self._cron.text(), schedule.get("timezone")) + + def _on_draft_changed(self, _change: str) -> None: + self._show_file() + + def _show_file(self) -> None: + marker = " (modified)" if self._draft.dirty else "" + self._file_label.setText(f"{self._path or _UNSAVED}{marker}") + + def new_pipeline(self) -> None: + """Start an empty draft.""" + if self._keep_unsaved(): + return + self._set_draft(self._service.new_draft(), None) + self.report("new pipeline") + + def open_pipeline(self, path: str | None = None) -> None: + """Open a definition file; without ``path`` a file dialog asks for one.""" + if self._keep_unsaved(): + return + chosen = path or self.pick_open_file("Open pipeline definition", _FILE_FILTER) + if not chosen: + return + self.run_async( + lambda: self._service.load(chosen), + f"open {chosen}", + lambda draft: self._opened(draft, chosen), + ) + + def _opened(self, draft: PipelineDraft, path: str) -> None: + self._set_draft(draft, path) + if draft.load_notes: + # What was wrong with the file as it was read, including what could not be shown. + task_ids = draft.task_ids() + self._panel.show_problems([Problem.parse(note, task_ids) for note in draft.load_notes]) + self.report_error(f"opened {path} with {len(draft.load_notes)} problem(s) to fix") + else: + self.report(f"opened {path}: {len(draft.tasks)} task(s)") + + def save_pipeline(self) -> None: + """Save to the file the draft was opened from or last saved to.""" + self.save_pipeline_as(self._path) + + def save_pipeline_as(self, path: str | None = None) -> None: + """Save the definition and, next to it, the canvas layout.""" + self._apply_header() + chosen = path or self.pick_save_file("Save pipeline definition", _FILE_FILTER) + if not chosen: + return + try: + self._path = self._service.save(self._draft, chosen) + except FileAutomationException as error: + self.report_error(f"cannot save {chosen}: {error}") + return + self._show_file() + self.report(f"saved {self._path}") + + def _keep_unsaved(self) -> bool: + """Return whether the user wants to keep the unsaved draft instead of replacing it.""" + return self._draft.dirty and not self.confirm("Discard the changes that are not saved?") + + # ------------------------------------------------------------------ tasks and edges + + def add_task( + self, action: str | None = None, position: tuple[float, float] | None = None + ) -> str: + """Add a task for ``action`` (the palette's selection by default); return its ID.""" + chosen = self._palette.selected_action() if action is None else action + task = self._draft.add_task(chosen, position=position) + self._canvas.select_tasks([task.task_id]) + self.report(f"added task {task.task_id}") + return task.task_id + + def _on_action_dropped(self, action: str, x: float, y: float) -> None: + self.add_task(action, (x - NODE_WIDTH / 2, y - NODE_HEIGHT / 2)) + + def remove_selected(self) -> None: + """Remove the selected arrows, or else the selected tasks.""" + edges, tasks = self._canvas.selected_edges(), self._canvas.selected_tasks() + if not edges and not tasks: + self.report_error("select a task or an arrow to remove") + return + with self._draft.batch(): + for upstream, downstream in edges: + self._draft.disconnect(upstream, downstream) + for task_id in [] if edges else tasks: + self._draft.remove_task(task_id) + removed = f"{len(edges)} arrow(s)" if edges else f"task(s) {', '.join(tasks)}" + self.report(f"removed {removed}") + + def connect_selected(self) -> None: + """Make the task selected second depend on the task selected first.""" + selected = self._canvas.selected_tasks() + if len(selected) != _PAIR: + self.report_error( + "select exactly two tasks: first the upstream one, then its dependent" + ) + return + upstream, downstream = selected + try: + added = self._draft.connect(upstream, downstream) + except AppException as error: + self.report_error(str(error)) + return + self.report( + f"{downstream} now depends on {upstream}" + if added + else f"{downstream} already depends on {upstream}" + ) + + def disconnect_selected(self) -> None: + """Remove the selected arrows, or the dependency between the two selected tasks.""" + pairs = self._canvas.selected_edges() + selected = self._canvas.selected_tasks() + if not pairs and len(selected) == _PAIR: + pairs = [(selected[0], selected[1]), (selected[1], selected[0])] + with self._draft.batch(): + removed = [pair for pair in pairs if self._draft.disconnect(*pair)] + if removed: + self.report("removed " + ", ".join(f"{up} -> {down}" for up, down in removed)) + else: + self.report_error("select an arrow, or two connected tasks, to disconnect") + + def auto_layout(self) -> None: + self._draft.auto_layout() + self.report("tasks laid out by dependency depth") + + def _on_selection(self, selected: list[str]) -> None: + self._form.show_task(selected[-1] if selected else None) + if len(selected) == _PAIR: + self._connect_button.setText(f"Connect {selected[0]} -> {selected[1]}") + else: + self._connect_button.setText(_CONNECT_HINT) + + def _on_task_applied(self, task_id: str) -> None: + self._canvas.select_tasks([task_id]) + self.report(f"task {task_id} updated") + + # ------------------------------------------------------------------ checks + + def validate(self) -> bool: + """List every problem of the draft; return whether there is none.""" + problems = self._service.validate(self._draft.to_definition()) + self._panel.show_problems(problems) + if problems: + self.report_error(f"{len(problems)} problem(s): see the Problems tab") + else: + self.report("the definition is valid") + return not problems + + def _inputs(self) -> tuple[dict[str, Any], dict[str, Any] | None] | None: + """Return the definition and the run parameters, or ``None`` when they cannot be used.""" + self._apply_header() + try: + params = parse_json_text(self._run_params.text(), _RUN_PARAMETERS) + except AppException as error: + self.report_error(str(error)) + return None + if params is not None and not isinstance(params, dict): + self.report_error(f"{_RUN_PARAMETERS} must be a JSON object") + return None + if not self.validate(): + return None + return self._draft.to_definition(), params + + def dry_run(self) -> None: + """Plan a run without executing anything.""" + inputs = self._inputs() + if inputs is not None: + definition, params = inputs + self.run_async( + lambda: self._service.dry_run(definition, params), "dry run", self._show_plan + ) + + def _show_plan(self, plan: dict[str, Any]) -> None: + self._show_run(plan, "dry run") + self._panel.show_tasks_tab() + if plan.get("status") == _FINISHED_OK: + self.report(f"dry run: {len(plan.get('tasks') or {})} task(s) would run as planned") + else: + notes = [str(state["error"]) for state in plan["tasks"].values() if state.get("error")] + self.report_error(f"dry run: {'; '.join(notes) or plan.get('error')}") + + def test_task(self) -> None: + """Execute the selected task alone; its upstream tasks are stand-ins that return nothing.""" + task_id = self._form.current_task() + if task_id is None: + self.report_error("select the task to test") + return + inputs = self._inputs() + if inputs is not None: + definition, params = inputs + self.run_async( + lambda: self._service.test_task(definition, task_id, params), + f"test task {task_id}", + lambda outcome: self._show_test(task_id, outcome), + ) + + def _show_test(self, task_id: str, outcome: dict[str, Any]) -> None: + run = outcome["run"] + self._show_run(run, f"test of {task_id}") + self._panel.show_tasks_tab() + self._panel.show_events(outcome["events"]) + state = run["tasks"][task_id] + if state.get("status") == _FINISHED_OK: + self.report(f"test of {task_id} succeeded in {state.get('attempts')} attempt(s)") + else: + self.report_error(f"test of {task_id}: {state.get('status')}: {state.get('error')}") + + def _show_run(self, run: dict[str, Any], headline: str = "") -> None: + """Show a run in the task table and on the canvas, leaving the visible tab alone.""" + self._panel.show_run(run, headline) + self._canvas.set_statuses( + {task_id: str(state.get("status")) for task_id, state in run["tasks"].items()} + ) + + # ------------------------------------------------------------------ runs + + def run(self) -> None: + """Start a run in the background and follow it.""" + inputs = self._inputs() + if inputs is not None: + definition, params = inputs + self.run_async( + lambda: self._service.start(definition, params), "start run", self._on_started + ) + + def resume(self) -> None: + """Continue the followed run: keep what succeeded, run the rest.""" + self._again("resume", self._service.resume) + + def retry(self) -> None: + """Start a new run with the parameters of the followed run.""" + self._again("retry", self._service.retry) + + def _again(self, verb: str, call: Callable[[str, dict[str, Any]], dict[str, Any]]) -> None: + run_id = self._run_id or self._panel.selected_run() + if run_id is None: + self.report_error(f"there is no run to {verb}: run the pipeline or pick one in History") + return + if not self.validate(): + return + definition = self._draft.to_definition() + self.run_async(lambda: call(run_id, definition), f"{verb} run {run_id}", self._on_started) + + def cancel(self) -> None: + """Ask the followed run to stop.""" + if self._run_id is None: + self.report_error("there is no run to cancel") + elif self._service.cancel(self._run_id): + self.report(f"cancel requested for run {self._run_id}") + else: + self.report_error(f"run {self._run_id} is not running in this process") + + def _on_started(self, run: dict[str, Any]) -> None: + self._run_id, self._following = run["run_id"], True + self._show_run(run) + self._panel.show_tasks_tab() + self._panel.show_events([]) + self.report(f"run {run['run_id']} is {run.get('status')}") + self._timer.start() + self.poll() + + def follow_selected(self) -> None: + run_id = self._panel.selected_run() + if run_id is None: + self.report_error("select a run in the History tab first") + else: + self.follow_run(run_id) + + def follow_run(self, run_id: str) -> None: + """Show the run ``run_id`` and keep following it until it has ended.""" + self._run_id, self._following = run_id, True + self._panel.show_tasks_tab() + self._timer.start() + self.poll() + + def _stop_following(self) -> None: + """Stop asking about a run that cannot be read; the failure is already on the status line.""" + self._following = False + self._timer.stop() + + def poll(self) -> None: + """Read the followed run once; the timer calls this until the run has ended.""" + run_id = self._run_id + if run_id is None: + self._timer.stop() + return + self.run_async( + lambda: self._service.follow(run_id), + "follow run", + self._show_followed, + key="follow", + quiet=True, + on_error=self._stop_following, + ) + + def _show_followed(self, data: dict[str, Any]) -> None: + run = data["run"] + if run.get("run_id") != self._run_id: + return + self._show_run(run) + self._panel.show_events(data["events"]) + if run.get("active") or not self._following: + return + self._following = False + self._timer.stop() + if run.get("status") == _FINISHED_OK: + self.report(f"run {run['run_id']} succeeded") + else: + self.report_error(f"run {run['run_id']} ended {run.get('status')}: {run.get('error')}") + self.refresh_history() + + def refresh_history(self) -> None: + self.run_async( + lambda: self._service.history(None, _HISTORY_LIMIT), + "read history", + self._panel.show_history, + key="history", + quiet=True, + ) + + # ------------------------------------------------------------------ lifecycle + + def refresh(self) -> None: + self.run_async( + self._service.action_names, + "read actions", + self._show_actions, + key="actions", + quiet=True, + ) + self.refresh_history() + + def _show_actions(self, names: list[str]) -> None: + self._palette.set_actions(names) + self._palette.filter_actions(self._filter.text()) + self._form.set_actions(names) + + def shutdown(self) -> None: + self._timer.stop() + self._draft.remove_listener(self._on_draft_changed) + self._canvas.set_draft(None) + super().shutdown() + + +def _compact_json(value: Any) -> str: + return json.dumps(value, ensure_ascii=False, default=repr) diff --git a/automation_file/ui/pages/run_panel.py b/automation_file/ui/pages/run_panel.py new file mode 100644 index 0000000..96c4772 --- /dev/null +++ b/automation_file/ui/pages/run_panel.py @@ -0,0 +1,211 @@ +"""What a pipeline editor shows about checks and runs: problems, task statuses, log, history.""" + +from __future__ import annotations + +from typing import Any + +from PySide6.QtCore import Signal +from PySide6.QtWidgets import ( + QLabel, + QListWidget, + QListWidgetItem, + QPlainTextEdit, + QTabWidget, + QVBoxLayout, + QWidget, +) + +from automation_file.app import Problem +from automation_file.ui.pages.base import fill_table, make_table, selected_cell + +_TASK_COLUMNS = ("Task", "Status", "Level", "Attempts", "Duration (ms)", "Error / reason", "Result") +_HISTORY_COLUMNS = ("Run", "Pipeline", "Status", "Active", "Started", "Finished", "Error") +_NO_PROBLEMS = "No problems found." +_NOT_VALIDATED = "Not validated yet: press Validate." +_TIME_START, _TIME_END = 11, 19 +_MAX_RESULT = 120 +_PROBLEMS_TAB, _TASKS_TAB, _HISTORY_TAB = 0, 1, 3 + + +def _short(value: object) -> str | None: + if value is None: + return None + text = str(value) + return text if len(text) <= _MAX_RESULT else f"{text[: _MAX_RESULT - 1]}…" + + +def event_line(event: dict[str, Any]) -> str: + """Return one log line for an event: time, severity, type, subject and error.""" + stamp = str(event.get("timestamp") or "")[_TIME_START:_TIME_END] + line = f"{stamp} [{event.get('severity')}] {event.get('type')}: {event.get('subject')}" + error = (event.get("payload") or {}).get("error") + return f"{line} -- {error}" if error else line + + +class RunPanel(QTabWidget): + """Four tabs under the canvas: Problems, Tasks, Log and History.""" + + #: A task ID the user picked in the problem list or in the task table. + task_chosen = Signal(str) + #: A run ID the user picked in the history. + run_chosen = Signal(str) + + def __init__(self) -> None: + super().__init__() + self._problems: list[Problem] = [] + self._problem_list = QListWidget() + self._problem_list.itemClicked.connect(self._on_problem_clicked) + self._run_label = QLabel("No run yet.") + self._run_label.setWordWrap(True) + self._task_table = make_table(_TASK_COLUMNS) + self._task_table.cellClicked.connect(lambda _row, _column: self._on_task_clicked()) + self._log = QPlainTextEdit() + self._log.setReadOnly(True) + self._log.setPlaceholderText("The events of the run appear here.") + self._history = make_table(_HISTORY_COLUMNS) + self._history.cellDoubleClicked.connect(lambda _row, _column: self._on_history_chosen()) + self._history_ids: list[str] = [] + + tasks = QWidget() + layout = QVBoxLayout(tasks) + layout.setContentsMargins(4, 4, 4, 4) + layout.addWidget(self._run_label) + layout.addWidget(self._task_table) + self.addTab(self._problem_list, "Problems") + self.addTab(tasks, "Tasks") + self.addTab(self._log, "Log") + self.addTab(self._history, "History") + + # ------------------------------------------------------------------ problems + + def show_problems(self, problems: list[Problem]) -> None: + """List ``problems``, each with its path, and bring the tab forward.""" + self._problems = list(problems) + self._problem_list.clear() + for problem in problems: + self._problem_list.addItem(QListWidgetItem(str(problem))) + if not problems: + self._problem_list.addItem(QListWidgetItem(_NO_PROBLEMS)) + self.setTabText(_PROBLEMS_TAB, f"Problems ({len(problems)})") + self.setCurrentIndex(_PROBLEMS_TAB) + + def clear_problems(self) -> None: + """Forget the listed problems: the definition has not been checked.""" + self._problems = [] + self._problem_list.clear() + self._problem_list.addItem(QListWidgetItem(_NOT_VALIDATED)) + self.setTabText(_PROBLEMS_TAB, "Problems") + + def problem_texts(self) -> list[str]: + """Return the listed problems as text.""" + return [str(problem) for problem in self._problems] + + def _on_problem_clicked(self, item: QListWidgetItem) -> None: + row = self._problem_list.row(item) + if 0 <= row < len(self._problems) and self._problems[row].task: + self.task_chosen.emit(str(self._problems[row].task)) + + # ------------------------------------------------------------------ the run + + def show_run(self, run: dict[str, Any] | None, headline: str = "") -> None: + """Show the state of ``run`` and of its tasks; ``None`` clears the table.""" + if run is None: + self._run_label.setText("No run yet.") + fill_table(self._task_table, []) + return + kind = "dry run" if run.get("dry_run") else "run" + described = headline or f"{kind} {run.get('run_id')}" + error = f" -- {run.get('error')}" if run.get("error") else "" + self._run_label.setText(f"{described}: {run.get('status')}{error}") + fill_table( + self._task_table, + [ + [ + task_id, + state.get("status"), + state.get("level"), + state.get("attempts"), + state.get("duration_ms"), + state.get("error") or state.get("reason"), + _short(state.get("result")), + ] + for task_id, state in (run.get("tasks") or {}).items() + ], + ) + + def task_rows(self) -> list[tuple[str, str]]: + """Return ``(task, status)`` for every row of the task table.""" + rows: list[tuple[str, str]] = [] + for row in range(self._task_table.rowCount()): + task, status = self._task_table.item(row, 0), self._task_table.item(row, 1) + if task is not None and status is not None: + rows.append((task.text(), status.text())) + return rows + + def run_text(self) -> str: + """Return the line above the task table.""" + return self._run_label.text() + + def show_tasks_tab(self) -> None: + self.setCurrentIndex(_TASKS_TAB) + + def _on_task_clicked(self) -> None: + task_id = selected_cell(self._task_table) + if task_id: + self.task_chosen.emit(task_id) + + # ------------------------------------------------------------------ the log + + def show_events(self, events: list[dict[str, Any]]) -> None: + """Replace the log with one line per event, oldest first.""" + self._log.setPlainText("\n".join(event_line(event) for event in events)) + self._log.verticalScrollBar().setValue(self._log.verticalScrollBar().maximum()) + + def log_text(self) -> str: + """Return what the log shows.""" + return self._log.toPlainText() + + # ------------------------------------------------------------------ history + + def show_history(self, runs: list[dict[str, Any]]) -> None: + """List ``runs``, newest first.""" + self._history_ids = [str(run.get("run_id")) for run in runs] + fill_table( + self._history, + [ + [ + run.get("run_id"), + run.get("pipeline"), + run.get("status"), + bool(run.get("active")), + run.get("started_at"), + run.get("finished_at"), + run.get("error"), + ] + for run in runs + ], + ) + + def history_ids(self) -> list[str]: + """Return the run IDs of the history, in the order shown.""" + return list(self._history_ids) + + def selected_run(self) -> str | None: + """Return the run ID selected in the history, or ``None``.""" + row = self._history.currentRow() + return self._history_ids[row] if 0 <= row < len(self._history_ids) else None + + def select_run(self, run_id: str) -> bool: + """Select ``run_id`` in the history; return whether it is listed.""" + if run_id not in self._history_ids: + return False + self._history.selectRow(self._history_ids.index(run_id)) + return True + + def show_history_tab(self) -> None: + self.setCurrentIndex(_HISTORY_TAB) + + def _on_history_chosen(self) -> None: + run_id = self.selected_run() + if run_id is not None: + self.run_chosen.emit(run_id) diff --git a/automation_file/ui/pages/scheduler_page.py b/automation_file/ui/pages/scheduler_page.py new file mode 100644 index 0000000..49dd63c --- /dev/null +++ b/automation_file/ui/pages/scheduler_page.py @@ -0,0 +1,127 @@ +"""Scheduler page: list, add and remove cron jobs.""" + +from __future__ import annotations + +from typing import Any + +from PySide6.QtCore import QThreadPool +from PySide6.QtWidgets import ( + QCheckBox, + QFormLayout, + QGroupBox, + QHBoxLayout, + QLineEdit, + QPlainTextEdit, + QVBoxLayout, +) + +from automation_file.app import SchedulerService +from automation_file.exceptions import FileAutomationException +from automation_file.ui.log_widget import LogPanel +from automation_file.ui.pages.base import BasePage, fill_table, make_table, selected_cell + +_COLUMNS = ("Name", "Cron", "Actions", "Runs", "Last run", "Running", "Skipped") +_KEYS = ("name", "cron", "actions", "runs", "last_run", "running", "skipped") + + +class SchedulerPage(BasePage): + """Cron jobs of the process-wide scheduler, through the Scheduler service.""" + + title = "Scheduler" + + def __init__(self, service: SchedulerService, log: LogPanel, pool: QThreadPool) -> None: + super().__init__(log, pool) + self._service = service + self._jobs: list[dict[str, Any]] = [] + + root = QVBoxLayout(self) + root.setContentsMargins(12, 12, 12, 12) + root.setSpacing(10) + root.addWidget(self._add_group()) + root.addWidget(self._list_group(), 1) + root.addWidget(self.status_label()) + + def _add_group(self) -> QGroupBox: + box = QGroupBox("Schedule a job") + form = QFormLayout(box) + self._name = QLineEdit() + self._name.setPlaceholderText("unique job name") + self._cron = QLineEdit("*/5 * * * *") + self._cron.setPlaceholderText("minute hour day-of-month month day-of-week") + self._actions = QPlainTextEdit() + self._actions.setPlaceholderText( + '[["FA_pipeline_run", {"definition": "pipelines/daily-report.yaml"}]]' + ) + self._actions.setMinimumHeight(100) + self._overlap = QCheckBox("Start a run even while the previous one is still going") + form.addRow("Name", self._name) + form.addRow("Cron", self._cron) + form.addRow("Actions (JSON)", self._actions) + form.addRow(self._overlap) + form.addRow(self.make_button("Add job", self.add_job)) + return box + + def _list_group(self) -> QGroupBox: + box = QGroupBox("Jobs") + layout = QVBoxLayout(box) + self._table = make_table(_COLUMNS) + layout.addWidget(self._table) + row = QHBoxLayout() + row.addWidget(self.make_button("Refresh", self.refresh)) + row.addWidget(self.make_button("Remove selected", self.remove_selected)) + row.addWidget(self.make_button("Remove all", self.remove_all)) + row.addStretch() + layout.addLayout(row) + return box + + def jobs(self) -> list[dict[str, Any]]: + """Return the jobs the table shows.""" + return list(self._jobs) + + def refresh(self) -> None: + self.run_async(self._service.jobs, "read jobs", self._show, key="refresh", quiet=True) + + def _show(self, jobs: list[dict[str, Any]]) -> None: + self._jobs = jobs + fill_table(self._table, [[job.get(key) for key in _KEYS] for job in jobs]) + + def add_job(self) -> None: + name = self._name.text() + cron = self._cron.text() + actions = self._actions.toPlainText() + overlap = self._overlap.isChecked() + self.run_async( + lambda: self._service.add(name, cron, actions, overlap), + f"add job {name.strip()}", + lambda job: self._after_change(f"added job {job.get('name')} ({job.get('cron')})"), + ) + + def remove_selected(self) -> None: + name = selected_cell(self._table) + if name is None: + self.report_error("select a job to remove") + return + self.run_async( + lambda: self._service.remove(name), + f"remove job {name}", + lambda _job: self._after_change(f"removed job {name}"), + ) + + def remove_all(self) -> None: + self.run_async( + self._service.remove_all, + "remove all jobs", + lambda jobs: self._after_change(f"removed {len(jobs)} job(s)"), + ) + + def _after_change(self, message: str) -> None: + self.report(message) + self.refresh() + + def shutdown(self) -> None: + """Remove the jobs when the window closes, as the scheduler tab always did.""" + try: + self._service.remove_all() + except FileAutomationException as error: + self._log.append_line(f"{self.title}: could not remove the jobs: {error}") + super().shutdown() diff --git a/automation_file/ui/pages/settings_page.py b/automation_file/ui/pages/settings_page.py new file mode 100644 index 0000000..5d83f45 --- /dev/null +++ b/automation_file/ui/pages/settings_page.py @@ -0,0 +1,178 @@ +"""Settings page: the configuration file, the optional extras, the environment.""" + +from __future__ import annotations + +import json +from collections.abc import Callable +from typing import Any + +from PySide6.QtCore import QThreadPool +from PySide6.QtWidgets import ( + QFormLayout, + QGroupBox, + QHBoxLayout, + QLabel, + QLineEdit, + QPlainTextEdit, + QVBoxLayout, +) + +from automation_file.app import ExtraStatus, SettingsService +from automation_file.ui.log_widget import LogPanel +from automation_file.ui.pages.base import BasePage, fill_table, make_table + +_EXTRA_COLUMNS = ("Extra", "Enables", "Installed", "Install command") +_ENVIRONMENT_ROWS = ( + ("version", "automation_file"), + ("python", "Python"), + ("platform", "Platform"), + ("log_file", "Log file"), + ("applied_config", "Applied configuration"), +) +_TOML_FILTER = "TOML files (*.toml);;All files (*)" + + +def _installed_text(extra: ExtraStatus) -> str: + if extra.installed is None: + return "unknown" + return "yes" if extra.installed else f"no ({', '.join(extra.missing)} missing)" + + +class SettingsPage(BasePage): + """Preview and apply ``automation_file.toml``; see which extras are installed.""" + + title = "Settings" + + def __init__(self, service: SettingsService, log: LogPanel, pool: QThreadPool) -> None: + super().__init__(log, pool) + self._service = service + self._summary: dict[str, Any] | None = None + self._extras: list[ExtraStatus] = [] + self._environment_labels: dict[str, QLabel] = {} + + root = QVBoxLayout(self) + root.setContentsMargins(12, 12, 12, 12) + root.setSpacing(10) + root.addWidget(self._config_group(), 2) + row = QHBoxLayout() + row.addWidget(self._extras_group(), 3) + row.addWidget(self._environment_group(), 2) + root.addLayout(row, 2) + root.addWidget(self.status_label()) + + # ------------------------------------------------------------------ layout + + def _config_group(self) -> QGroupBox: + box = QGroupBox("Configuration file") + layout = QVBoxLayout(box) + row = QHBoxLayout() + self._path = QLineEdit() + self._path.setPlaceholderText("automation_file.toml") + row.addWidget(self._path, 1) + row.addWidget(self.make_button("Browse…", self._on_browse)) + row.addWidget(self.make_button("Preview", self.preview)) + row.addWidget(self.make_button("Apply", self.apply)) + layout.addLayout(row) + layout.addWidget( + self.muted_label( + "Preview reads the file and changes nothing. Apply registers its notification " + "sinks and routes. Secrets are resolved from ${env:...} and ${file:...} and are " + "shown masked." + ) + ) + self._document = QPlainTextEdit() + self._document.setReadOnly(True) + self._document.setPlaceholderText("The configuration summary appears here.") + layout.addWidget(self._document) + return box + + def _extras_group(self) -> QGroupBox: + box = QGroupBox("Optional extras") + layout = QVBoxLayout(box) + self._extras_table = make_table(_EXTRA_COLUMNS) + layout.addWidget(self._extras_table) + layout.addWidget(self.make_button("Refresh", self.refresh)) + return box + + def _environment_group(self) -> QGroupBox: + box = QGroupBox("Environment") + form = QFormLayout(box) + for key, label in _ENVIRONMENT_ROWS: + value = QLabel("") + value.setWordWrap(True) + self._environment_labels[key] = value + form.addRow(label, value) + return box + + # ------------------------------------------------------------------ extras and environment + + def extras(self) -> list[ExtraStatus]: + """Return the extras the table shows.""" + return list(self._extras) + + def refresh(self) -> None: + self.run_async( + lambda: (self._service.extras(), self._service.environment()), + "read extras", + self._show_environment, + key="refresh", + quiet=True, + ) + + def _show_environment(self, result: tuple[list[ExtraStatus], dict[str, Any]]) -> None: + self._extras, environment = result + fill_table( + self._extras_table, + [ + [extra.name, extra.feature, _installed_text(extra), extra.install_hint] + for extra in self._extras + ], + ) + for key, _label in _ENVIRONMENT_ROWS: + value = environment.get(key) + self._environment_labels[key].setText("none" if value is None else str(value)) + + # ------------------------------------------------------------------ configuration + + def set_path(self, path: str) -> None: + """Fill in the configuration file field.""" + self._path.setText(path) + + def summary(self) -> dict[str, Any] | None: + """Return the configuration summary the page shows.""" + return self._summary + + def document_text(self) -> str: + """Return what the summary pane shows.""" + return self._document.toPlainText() + + def _on_browse(self) -> None: + chosen = self.pick_open_file("Configuration file", _TOML_FILTER) + if chosen: + self._path.setText(chosen) + + def preview(self) -> None: + self._read("preview", self._service.load) + + def apply(self) -> None: + self._read("apply", self._service.apply) + + def _read(self, verb: str, read: Callable[[str], dict[str, Any]]) -> None: + path = self._path.text().strip() + if not path: + self.report_error("enter the path of the configuration file first") + return + self.run_async(lambda: read(path), f"{verb} {path}", self._show_summary) + + def _show_summary(self, summary: dict[str, Any]) -> None: + self._summary = summary + self._document.setPlainText(json.dumps(summary, indent=2, ensure_ascii=False, default=str)) + sinks, routes = len(summary.get("sinks") or ()), len(summary.get("routes") or ()) + if summary.get("applied"): + self.report(f"applied {summary.get('source')}: {sinks} sink(s), {routes} route(s)") + self.refresh() + else: + self.report( + f"{summary.get('source')} declares {sinks} sink(s) and {routes} route(s); " + "nothing was changed" + ) diff --git a/automation_file/ui/pages/storage_page.py b/automation_file/ui/pages/storage_page.py new file mode 100644 index 0000000..5aae986 --- /dev/null +++ b/automation_file/ui/pages/storage_page.py @@ -0,0 +1,167 @@ +"""Storage page: the backends, whether each can be used, and the mounts.""" + +from __future__ import annotations + +from PySide6.QtCore import QThreadPool +from PySide6.QtWidgets import ( + QFormLayout, + QGroupBox, + QHBoxLayout, + QLabel, + QLineEdit, + QVBoxLayout, +) + +from automation_file.app import BackendStatus, StorageService +from automation_file.exceptions import FileAutomationException +from automation_file.ui.log_widget import LogPanel +from automation_file.ui.pages.base import BasePage, fill_table, make_table, selected_cell + +_COLUMNS = ("Backend", "Kind", "Label", "Extra", "Installed", "Usable", "Detail") +_MOUNT_KIND = "mount" +#: Schemes whose root can be resolved without a bucket, a container or a session. +_ROOT_URIS = {"local": "local:///", "memory": "memory:///"} +_NAME_COLUMN = 0 +_KIND_COLUMN = 1 + + +class StoragePage(BasePage): + """Shows every scheme, mount and shared client, and mounts a local directory.""" + + title = "Storage" + + def __init__(self, service: StorageService, log: LogPanel, pool: QThreadPool) -> None: + super().__init__(log, pool) + self._service = service + self._backends: list[BackendStatus] = [] + + root = QVBoxLayout(self) + root.setContentsMargins(12, 12, 12, 12) + root.setSpacing(10) + root.addWidget(self._backends_group(), 1) + root.addWidget(self._mount_group()) + root.addWidget(self.status_label()) + + def _backends_group(self) -> QGroupBox: + box = QGroupBox("Backends") + layout = QVBoxLayout(box) + layout.addWidget( + self.muted_label( + "A cloud backend becomes usable once its client has been initialised: use the " + "Credentials panel under Advanced, Transfer, or its FA_*_later_init action. " + "When the package of its extra is missing, Detail shows the install command." + ) + ) + self._table = make_table(_COLUMNS) + self._table.itemSelectionChanged.connect(self._on_selection) + layout.addWidget(self._table) + self._capabilities = QLabel("Select a backend to see what it provides.") + self._capabilities.setWordWrap(True) + layout.addWidget(self._capabilities) + row = QHBoxLayout() + row.addWidget(self.make_button("Refresh", self.refresh)) + row.addWidget(self.make_button("Unmount selected", self.unmount_selected)) + row.addStretch() + layout.addLayout(row) + return box + + def _mount_group(self) -> QGroupBox: + box = QGroupBox("Mount a local directory") + form = QFormLayout(box) + self._mount_uri = QLineEdit() + self._mount_uri.setPlaceholderText("sandbox://jobs") + self._mount_root = QLineEdit() + self._mount_root.setPlaceholderText("an existing directory; the mount cannot leave it") + root_row = QHBoxLayout() + root_row.addWidget(self._mount_root, 1) + root_row.addWidget(self.make_button("Browse…", self._on_browse)) + form.addRow("URI", self._mount_uri) + form.addRow("Directory", root_row) + form.addRow(self.make_button("Mount", self.mount_local)) + return box + + # ------------------------------------------------------------------ data + + def backends(self) -> list[BackendStatus]: + """Return the statuses the table shows.""" + return list(self._backends) + + def refresh(self) -> None: + self.run_async( + self._service.backends, "read backends", self._show, key="refresh", quiet=True + ) + + def _show(self, backends: list[BackendStatus]) -> None: + self._backends = backends + fill_table( + self._table, + [ + [ + status.name, + status.kind, + status.label, + status.extra, + status.installed, + status.usable, + status.detail, + ] + for status in backends + ], + ) + + def _on_selection(self) -> None: + name = selected_cell(self._table, _NAME_COLUMN) + kind = selected_cell(self._table, _KIND_COLUMN) + if name is None: + return + probe = name if kind == _MOUNT_KIND else _ROOT_URIS.get(name) + if probe is None: + self._capabilities.setText( + f"{name}: what it provides depends on the bucket, container or session; " + "open one of its URIs on the Files page." + ) + return + try: + # Resolving a mount or the local root touches no network and no disk. + found = self._service.capabilities(probe) + except FileAutomationException as error: + self._capabilities.setText(f"{name}: {error}") + return + provided = ", ".join(f"{key}: {'yes' if value else 'no'}" for key, value in found.items()) + self._capabilities.setText(f"{name} provides {provided}") + + # ------------------------------------------------------------------ mounts + + def _on_browse(self) -> None: + chosen = self.pick_directory() + if chosen: + self._mount_root.setText(chosen) + + def mount_local(self) -> None: + uri = self._mount_uri.text().strip() + root = self._mount_root.text().strip() + if not uri or not root: + self.report_error("enter the URI to serve and the directory to serve it from") + return + self.run_async( + lambda: self._service.mount_local(uri, root), + f"mount {uri}", + lambda mount: self._after_change(f"mounted {mount.uri} on {root}"), + ) + + def unmount_selected(self) -> None: + name = selected_cell(self._table, _NAME_COLUMN) + if name is None or selected_cell(self._table, _KIND_COLUMN) != _MOUNT_KIND: + self.report_error("select a mount to unmount") + return + self.run_async( + lambda: self._service.unmount(name), + f"unmount {name}", + lambda removed: self._after_change( + f"unmounted {name}" if removed else f"{name} was not mounted" + ), + ) + + def _after_change(self, message: str) -> None: + self.report(message) + self.refresh() diff --git a/automation_file/ui/pages/task_form.py b/automation_file/ui/pages/task_form.py new file mode 100644 index 0000000..76f6051 --- /dev/null +++ b/automation_file/ui/pages/task_form.py @@ -0,0 +1,505 @@ +"""The form of one pipeline task: its action, its arguments and how it runs. + +:class:`TaskForm` shows the selected task of a +:class:`~automation_file.app.PipelineDraft` and writes the fields back through +the draft's own methods when the user presses Apply. It holds no copy of the +task: after every apply, and whenever the selection changes, it is filled from +the draft again. +""" + +from __future__ import annotations + +import json +from typing import Any + +from PySide6.QtCore import Qt, Signal +from PySide6.QtWidgets import ( + QCheckBox, + QComboBox, + QDoubleSpinBox, + QFormLayout, + QHBoxLayout, + QHeaderView, + QLabel, + QLineEdit, + QListWidget, + QListWidgetItem, + QPlainTextEdit, + QPushButton, + QSpinBox, + QStackedWidget, + QTableWidget, + QTableWidgetItem, + QVBoxLayout, + QWidget, +) + +from automation_file.app import ( + ActionInfo, + AppException, + PipelineDraft, + PipelineService, + format_argument_value, + parse_argument_text, + parse_json_text, + split_names, +) + +_ARGUMENT_COLUMNS = ("Argument", "Value (JSON or text)", "Default") +_NAME_COLUMN, _VALUE_COLUMN, _DEFAULT_COLUMN = 0, 1, 2 +_TABLE_PAGE, _JSON_PAGE = 0, 1 +_REQUIRED = "required" +_WHEN_CHOICES = ("on_success", "on_failure", "always") +_MAX_ATTEMPTS = 100 +_MAX_SECONDS = 86_400.0 +_DEFAULT_BACKOFF_CAP = 60.0 +_ERROR_STYLE = "color: #b3261e;" +_OK_STYLE = "color: #2f8f3f;" +_NO_TASK = "Select a task on the canvas to edit it." +_ARGUMENTS = "the arguments" + + +def _read_only(text: str) -> QTableWidgetItem: + item = QTableWidgetItem(text) + item.setFlags(item.flags() & ~Qt.ItemFlag.ItemIsEditable) + return item + + +def _as_json(value: Any) -> str: + return json.dumps(value, indent=2, ensure_ascii=False, default=repr) + + +class ArgumentsEditor(QWidget): + """The arguments of an action: one row per parameter, or the JSON itself. + + A keyword mapping is edited in a table whose rows come from the action's + signature; a value is JSON when it parses as JSON and text otherwise, and a + row left empty is not passed, so the action's default applies. A positional + list can only be edited as JSON. + """ + + def __init__(self) -> None: + super().__init__() + self._table = QTableWidget(0, len(_ARGUMENT_COLUMNS)) + self._table.setHorizontalHeaderLabels(list(_ARGUMENT_COLUMNS)) + self._table.horizontalHeader().setSectionResizeMode(QHeaderView.ResizeMode.Stretch) + self._table.verticalHeader().setVisible(False) + self._raw = QPlainTextEdit() + self._raw.setPlaceholderText('{"source": "s3://in/a.csv"} or ["s3://in/a.csv", true]') + self._stack = QStackedWidget() + self._stack.addWidget(self._table) + self._stack.addWidget(self._raw) + self._as_json = QCheckBox("Edit as JSON") + self._as_json.toggled.connect(self._on_mode_toggled) + add_row = QPushButton("Add argument") + add_row.clicked.connect(lambda _checked=False: self.add_row()) + + buttons = QHBoxLayout() + buttons.addWidget(self._as_json) + buttons.addWidget(add_row) + buttons.addStretch() + layout = QVBoxLayout(self) + layout.setContentsMargins(0, 0, 0, 0) + layout.addWidget(self._stack) + layout.addLayout(buttons) + + def is_json_mode(self) -> bool: + """Return whether the arguments are edited as JSON text.""" + return self._as_json.isChecked() + + def set_json_mode(self, enabled: bool) -> None: + """Switch between the table and the JSON text, carrying the arguments over.""" + self._as_json.setChecked(enabled) + + def set_arguments(self, arguments: Any, info: ActionInfo | None = None) -> None: + """Show ``arguments``; ``info`` supplies the parameter rows of the action.""" + positional = isinstance(arguments, list) + self._fill_table({} if positional or arguments is None else dict(arguments), info) + self._raw.setPlainText("" if arguments is None else _as_json(arguments)) + self._set_mode(positional) + + def arguments(self) -> dict[str, Any] | list[Any] | None: + """Return the arguments the editor holds; unreadable JSON raises :class:`AppException`.""" + if not self._as_json.isChecked(): + return self._table_arguments() or None + parsed = parse_json_text(self._raw.toPlainText(), _ARGUMENTS) + if parsed is not None and not isinstance(parsed, (dict, list)): + raise AppException(f"{_ARGUMENTS} must be a JSON object, a JSON array or empty") + return parsed + + def add_row(self, name: str = "", value: str = "") -> None: + """Append a row for an argument the signature does not list.""" + row = self._table.rowCount() + self._table.insertRow(row) + self._table.setItem(row, _NAME_COLUMN, QTableWidgetItem(name)) + self._table.setItem(row, _VALUE_COLUMN, QTableWidgetItem(value)) + self._table.setItem(row, _DEFAULT_COLUMN, _read_only("")) + + def set_value(self, name: str, text: str) -> bool: + """Type ``text`` into the row of the argument ``name``; return whether there is one.""" + for row in range(self._table.rowCount()): + if self._cell(row, _NAME_COLUMN) == name: + self._table.setItem(row, _VALUE_COLUMN, QTableWidgetItem(text)) + return True + return False + + def set_json_text(self, text: str) -> None: + """Replace the JSON text (the editor must be in JSON mode for it to count).""" + self._raw.setPlainText(text) + + def row_names(self) -> list[str]: + """Return the argument names of the table rows, in order.""" + return [self._cell(row, _NAME_COLUMN) for row in range(self._table.rowCount())] + + def _cell(self, row: int, column: int) -> str: + item = self._table.item(row, column) + return "" if item is None else item.text().strip() + + def _set_mode(self, as_json: bool) -> None: + self._as_json.blockSignals(True) + self._as_json.setChecked(as_json) + self._as_json.blockSignals(False) + self._stack.setCurrentIndex(_JSON_PAGE if as_json else _TABLE_PAGE) + + def _fill_table(self, values: dict[str, Any], info: ActionInfo | None) -> None: + self._table.setRowCount(0) + listed: list[str] = [] + for parameter in info.parameters if info is not None else (): + listed.append(parameter.name) + row = self._table.rowCount() + self._table.insertRow(row) + self._table.setItem(row, _NAME_COLUMN, _read_only(parameter.name)) + shown = ( + format_argument_value(values[parameter.name]) if parameter.name in values else "" + ) + self._table.setItem(row, _VALUE_COLUMN, QTableWidgetItem(shown)) + hint = _REQUIRED if parameter.required else parameter.default + self._table.setItem(row, _DEFAULT_COLUMN, _read_only(hint)) + for name, value in values.items(): + if name not in listed: + self.add_row(str(name), format_argument_value(value)) + + def _table_arguments(self) -> dict[str, Any]: + found: dict[str, Any] = {} + for row in range(self._table.rowCount()): + name, text = self._cell(row, _NAME_COLUMN), self._cell(row, _VALUE_COLUMN) + if name and text: + found[name] = parse_argument_text(text) + return found + + def _on_mode_toggled(self, as_json: bool) -> None: + if as_json: + values = self._table_arguments() + self._raw.setPlainText(_as_json(values) if values else "") + self._stack.setCurrentIndex(_JSON_PAGE) + return + try: + parsed = parse_json_text(self._raw.toPlainText(), _ARGUMENTS) + except AppException: + # Text that is not JSON yet has no rows to become: stay with the text. + self._set_mode(True) + return + if parsed is not None and not isinstance(parsed, dict): + # A positional list has no argument names: it can only be edited as JSON. + self._set_mode(True) + return + for name, value in (parsed or {}).items(): + if not self.set_value(name, format_argument_value(value)): + self.add_row(name, format_argument_value(value)) + self._stack.setCurrentIndex(_TABLE_PAGE) + + +class TaskForm(QWidget): + """Edits the selected task of a draft; nothing changes until Apply is pressed.""" + + #: Emitted after the fields were written to the draft, with the task's (possibly new) ID. + applied = Signal(str) + + def __init__(self, service: PipelineService) -> None: + super().__init__() + self._service = service + self._draft: PipelineDraft | None = None + self._task: str | None = None + self._loading = False + self._applying = False + + self._task_id = QLineEdit() + self._action = QComboBox() + self._action.setEditable(True) + self._action.currentTextChanged.connect(self._on_action_changed) + self._signature = QLabel("") + self._signature.setWordWrap(True) + self._arguments = ArgumentsEditor() + self._depends = QListWidget() + self._depends.setMaximumHeight(110) + self._attempts = QSpinBox() + self._attempts.setRange(1, _MAX_ATTEMPTS) + self._backoff = self._seconds_box(0.0) + self._backoff_cap = self._seconds_box(_DEFAULT_BACKOFF_CAP) + self._retry_on = QLineEdit() + self._retry_on.setPlaceholderText( + "exception names, comma-separated; empty: transient errors" + ) + self._timeout = QLineEdit() + self._timeout.setPlaceholderText("seconds for the whole task; empty: no limit") + self._when = QComboBox() + self._when.addItems(_WHEN_CHOICES) + self._key = QLineEdit() + self._key.setPlaceholderText("e.g. publish-${params.date}; empty: none") + self._message = QLabel(_NO_TASK) + self._message.setWordWrap(True) + self._lines = { + "task_id": self._task_id, + "retry_on": self._retry_on, + "timeout": self._timeout, + "key": self._key, + } + self._build_layout() + self.setEnabled(False) + + @staticmethod + def _seconds_box(value: float) -> QDoubleSpinBox: + box = QDoubleSpinBox() + box.setRange(0.0, _MAX_SECONDS) + box.setDecimals(1) + box.setSuffix(" s") + box.setValue(value) + return box + + def _build_layout(self) -> None: + form = QFormLayout() + # Labels above their fields: the form lives in a side panel, where width is scarce. + form.setRowWrapPolicy(QFormLayout.RowWrapPolicy.WrapAllRows) + form.addRow("Task ID", self._task_id) + form.addRow("Action", self._action) + form.addRow(self._signature) + form.addRow("Arguments", self._arguments) + form.addRow("Depends on", self._depends) + form.addRow("Attempts", self._attempts) + form.addRow("Back-off", self._backoff) + form.addRow("Back-off cap", self._backoff_cap) + form.addRow("Retry on", self._retry_on) + form.addRow("Timeout", self._timeout) + form.addRow("Run when", self._when) + form.addRow("Idempotency key", self._key) + apply_button = QPushButton("Apply changes") + apply_button.clicked.connect(lambda _checked=False: self.apply()) + revert_button = QPushButton("Revert") + revert_button.clicked.connect(lambda _checked=False: self.revert()) + buttons = QHBoxLayout() + buttons.addWidget(apply_button) + buttons.addWidget(revert_button) + buttons.addStretch() + layout = QVBoxLayout(self) + layout.setContentsMargins(0, 0, 0, 0) + layout.addLayout(form) + layout.addLayout(buttons) + layout.addWidget(self._message) + + # ------------------------------------------------------------------ what it edits + + def set_actions(self, names: list[str]) -> None: + """Offer ``names`` in the action field.""" + self._loading = True + try: + current = self._action.currentText() + self._action.clear() + self._action.addItems(names) + self._action.setEditText(current) + finally: + self._loading = False + + def set_draft(self, draft: PipelineDraft | None) -> None: + """Edit tasks of ``draft`` from now on; the form is emptied.""" + self._draft = draft + self._task = None + self.show_task(None) + + def current_task(self) -> str | None: + """Return the ID of the task the form shows.""" + return self._task + + def message(self) -> str: + """Return what the form last said: the outcome of an apply, or what to do.""" + return self._message.text() + + def arguments_editor(self) -> ArgumentsEditor: + """Return the editor of the action's arguments.""" + return self._arguments + + def revert(self) -> None: + """Throw away what was typed and show the task as the draft has it.""" + task_id, self._task = self._task, None + self.show_task(task_id) + + def show_task(self, task_id: str | None) -> None: + """Fill the form from the draft's task ``task_id``; ``None`` empties and disables it. + + Showing the task the form already shows keeps what was typed and only + brings the dependency list up to date: the canvas reports the same + selection again after every change of the draft, and an arrow drawn + there must not be undone by the next Apply. :meth:`revert` reloads + every field. + """ + if self._applying: + return + draft = self._draft + if draft is None or task_id is None or not draft.has_task(task_id): + self._task = None + self.setEnabled(False) + self._say(_NO_TASK, "") + return + if task_id == self._task: + self._fill_dependencies(draft, task_id) + return + self._task = task_id + self.setEnabled(True) + self._loading = True + try: + self._load(draft, task_id) + finally: + self._loading = False + self._say(f"Editing {task_id}.", "") + + def _load(self, draft: PipelineDraft, task_id: str) -> None: + task = draft.task(task_id) + self._task_id.setText(task.task_id) + self._action.setEditText(task.action) + info = self._service.describe_action(task.action) + self._show_signature(info) + self._arguments.set_arguments(task.arguments, info) + self._fill_dependencies(draft, task_id) + retry = task.retry or {} + self._attempts.setValue(int(retry.get("max_attempts", 1))) + self._backoff.setValue(float(retry.get("backoff", 0.0))) + self._backoff_cap.setValue(float(retry.get("backoff_cap", _DEFAULT_BACKOFF_CAP))) + self._retry_on.setText(", ".join(retry.get("on") or ())) + self._timeout.setText("" if task.timeout is None else f"{task.timeout:g}") + self._when.setCurrentText(task.when) + self._key.setText(task.idempotency_key or "") + + def _fill_dependencies(self, draft: PipelineDraft, task_id: str) -> None: + self._depends.clear() + wanted = draft.task(task_id).depends_on + for other in draft.task_ids(): + if other == task_id: + continue + entry = QListWidgetItem(other) + entry.setFlags(entry.flags() | Qt.ItemFlag.ItemIsUserCheckable) + entry.setCheckState( + Qt.CheckState.Checked if other in wanted else Qt.CheckState.Unchecked + ) + self._depends.addItem(entry) + + def _show_signature(self, info: ActionInfo) -> None: + if not info.name: + self._signature.setText("Choose an action.") + elif not info.known: + self._signature.setText(f"{info.name} is not a registered action.") + else: + self._signature.setText(f"{info.signature or info.name}\n{info.summary}".strip()) + + def _on_action_changed(self, name: str) -> None: + if self._loading or self._task is None: + return + info = self._service.describe_action(name.strip()) + self._show_signature(info) + if self._arguments.is_json_mode(): + return + self._arguments.set_arguments(self._arguments.arguments(), info) + + # ------------------------------------------------------------------ setters for callers + + def set_field(self, name: str, text: str) -> None: + """Type ``text`` into a line field: ``task_id``, ``retry_on``, ``timeout`` or ``key``.""" + self._lines[name].setText(text) + + def set_action(self, name: str) -> None: + """Type ``name`` into the action field.""" + self._action.setEditText(name) + + def set_retry(self, attempts: int, backoff: float = 0.0) -> None: + """Set the number of attempts and the first back-off.""" + self._attempts.setValue(attempts) + self._backoff.setValue(backoff) + + def set_condition(self, when: str) -> None: + """Choose when the task runs.""" + self._when.setCurrentText(when) + + def set_dependency(self, task_id: str, checked: bool) -> bool: + """Tick or clear the dependency on ``task_id``; return whether it is listed.""" + for row in range(self._depends.count()): + entry = self._depends.item(row) + if entry.text() == task_id: + entry.setCheckState(Qt.CheckState.Checked if checked else Qt.CheckState.Unchecked) + return True + return False + + def checked_dependencies(self) -> list[str]: + """Return the task IDs ticked in the dependency list.""" + return [ + self._depends.item(row).text() + for row in range(self._depends.count()) + if self._depends.item(row).checkState() == Qt.CheckState.Checked + ] + + # ------------------------------------------------------------------ apply + + def apply(self) -> bool: + """Write the fields to the draft; return whether everything was accepted. + + The rename comes last, so a field the draft refuses leaves the task + under the ID the canvas has selected, with what was typed still there. + """ + draft, task_id = self._draft, self._task + if draft is None or task_id is None: + return False + self._applying = True + try: + task_id = self._write(draft, task_id) + except AppException as error: + self._say(str(error), _ERROR_STYLE) + return False + finally: + self._applying = False + self._task = None + self.show_task(task_id) + self._say(f"Applied to {task_id}.", _OK_STYLE) + self.applied.emit(task_id) + return True + + def _write(self, draft: PipelineDraft, task_id: str) -> str: + arguments = self._arguments.arguments() + timeout = self._timeout_value() + wanted_id = self._task_id.text().strip() + with draft.batch(): + draft.set_action(task_id, self._action.currentText()) + draft.set_arguments(task_id, arguments) + draft.set_dependencies(task_id, self.checked_dependencies()) + draft.set_retry( + task_id, + self._attempts.value(), + self._backoff.value(), + self._backoff_cap.value(), + split_names(self._retry_on.text()) or None, + ) + draft.set_timeout(task_id, timeout) + draft.set_condition(task_id, self._when.currentText()) + draft.set_idempotency_key(task_id, self._key.text().strip() or None) + if wanted_id != task_id: + draft.rename_task(task_id, wanted_id) + return wanted_id + + def _timeout_value(self) -> float | None: + text = self._timeout.text().strip() + if not text: + return None + try: + return float(text) + except ValueError: + raise AppException(f"the timeout must be a number of seconds, got {text!r}") from None + + def _say(self, text: str, style: str) -> None: + self._message.setStyleSheet(style) + self._message.setText(text) diff --git a/docs/source/API/api_index.rst b/docs/source/API/api_index.rst index 834eff8..1555583 100644 --- a/docs/source/API/api_index.rst +++ b/docs/source/API/api_index.rst @@ -243,3 +243,17 @@ and its schema, and the ``FA_pipeline_*`` actions. :caption: Pipelines pipeline + +.. _api-app: + +Chapter R — Application Layer +============================= + +``automation_file.app``: one plain-Python service per navigation entry, the +pipeline draft the editor works on, and the masking of secrets. + +.. toctree:: + :maxdepth: 2 + :caption: Application Layer + + app diff --git a/docs/source/API/app.rst b/docs/source/API/app.rst new file mode 100644 index 0000000..b6cca9c --- /dev/null +++ b/docs/source/API/app.rst @@ -0,0 +1,91 @@ +Application layer +================= + +What a user interface calls: one service per navigation entry, on top of the +domain packages. The PySide6 window and the Web UI are both built on it. It +imports no GUI toolkit and no backend SDK, returns JSON-friendly data and masks +secrets in what it returns. Usage is described in the manual chapter +*Application layer*. + +The set of services +------------------- + +.. automodule:: automation_file.app + +.. automodule:: automation_file.app.services + :members: + +Dashboard +--------- + +.. automodule:: automation_file.app.dashboard_service + :members: + +Files +----- + +.. automodule:: automation_file.app.file_service + :members: + +Storage +------- + +.. automodule:: automation_file.app.storage_service + :members: + +Pipelines +--------- + +.. automodule:: automation_file.app.pipeline_draft + :members: + +.. automodule:: automation_file.app.pipeline_service + :members: + +Scheduler +--------- + +.. automodule:: automation_file.app.scheduler_service + :members: + +Integrity +--------- + +.. automodule:: automation_file.app.integrity_service + :members: + +Audit +----- + +.. automodule:: automation_file.app.audit_service + :members: + +Notifications +------------- + +.. automodule:: automation_file.app.notification_service + :members: + +Settings +-------- + +.. automodule:: automation_file.app.settings_service + :members: + +Form helpers +------------ + +.. automodule:: automation_file.app.arguments + :members: + +Secrets +------- + +.. automodule:: automation_file.app.masking + :members: + +Exceptions +---------- + +.. automodule:: automation_file.app.errors + :members: diff --git a/docs/source/API/ui.rst b/docs/source/API/ui.rst index f61e181..93d47a0 100644 --- a/docs/source/API/ui.rst +++ b/docs/source/API/ui.rst @@ -5,6 +5,12 @@ PySide6 front-end. Importing ``automation_file.ui`` loads Qt eagerly; the facade ``automation_file.launch_ui`` attribute is lazy (only pulls Qt when accessed) so non-UI workloads keep their import cost low. +The main window is a sidebar over nine workflow pages -- Dashboard, Files, +Storage, Pipelines, Scheduler, Integrity, Audit, Notifications, Settings -- +and an Advanced page that keeps the earlier tabs. Every workflow page is a +view over one service of :mod:`automation_file.app` and imports nothing below +that layer. Usage is described in the manual chapter *GUI*. + Launcher -------- @@ -29,9 +35,66 @@ Log panel .. automodule:: automation_file.ui.log_widget :members: +Pages +----- + +.. automodule:: automation_file.ui.pages + :members: + +.. automodule:: automation_file.ui.pages.base + :members: + +.. automodule:: automation_file.ui.pages.dashboard_page + :members: + +.. automodule:: automation_file.ui.pages.files_page + :members: + +.. automodule:: automation_file.ui.pages.storage_page + :members: + +.. automodule:: automation_file.ui.pages.scheduler_page + :members: + +.. automodule:: automation_file.ui.pages.integrity_page + :members: + +.. automodule:: automation_file.ui.pages.audit_page + :members: + +.. automodule:: automation_file.ui.pages.notifications_page + :members: + +.. automodule:: automation_file.ui.pages.settings_page + :members: + +.. automodule:: automation_file.ui.pages.advanced_page + :members: + +Pipeline editor +--------------- + +.. automodule:: automation_file.ui.pages.pipelines_page + :members: + +.. automodule:: automation_file.ui.pages.pipeline_canvas + :members: + +.. automodule:: automation_file.ui.pages.task_form + :members: + +.. automodule:: automation_file.ui.pages.run_panel + :members: + Tabs ---- +The tabs of the earlier window. ``LocalOpsTab``, ``TransferTab`` (with one +panel per backend), ``ProgressTab``, ``JSONEditorTab``, ``TriggerTab`` and +``ServerTab`` are shown by the Advanced page. ``HomeTab`` and ``SchedulerTab`` +are no longer part of the main window (the Dashboard and Scheduler pages took +their place); they remain importable widgets. + .. automodule:: automation_file.ui.tabs :members: @@ -62,6 +125,12 @@ Tabs .. automodule:: automation_file.ui.tabs.sftp_tab :members: +.. automodule:: automation_file.ui.tabs.onedrive_tab + :members: + +.. automodule:: automation_file.ui.tabs.box_tab + :members: + .. automodule:: automation_file.ui.tabs.transfer_tab :members: diff --git a/docs/source/Eng/architecture.rst b/docs/source/Eng/architecture.rst index 33af81a..e10d4ea 100644 --- a/docs/source/Eng/architecture.rst +++ b/docs/source/Eng/architecture.rst @@ -78,7 +78,8 @@ dispatchers. end subgraph UI["ui (PySide6)"] - MainWin["MainWindow
    Home · Local · HTTP · Drive · S3 · Azure · Dropbox
    SFTP · OneDrive · Box · JSON · Triggers · Scheduler
    Progress · Transfer · Servers"] + MainWin["MainWindow
    Dashboard · Files · Storage · Pipelines · Scheduler
    Integrity · Audit · Notifications · Settings
    Advanced: Local · Transfer · Progress · JSON · Triggers · Servers"] + AppLayer["automation_file.app
    one service per navigation entry"] Worker["ActionWorker
    QRunnable on QThreadPool"] end @@ -128,6 +129,7 @@ dispatchers. Plugins ==> Loader MainWin ==> Worker + Worker ==> AppLayer Worker ==> PublicAPI PublicAPI ==> Executor @@ -143,6 +145,8 @@ dispatchers. HTTPS ==> Executor MCP ==> Registry MetSrv ==> Metrics + WebUI ==> AppLayer + AppLayer ==> PublicAPI WebUI ==> Registry ACL ==> TCP ACL ==> HTTPS @@ -238,7 +242,7 @@ dispatchers. class Secrets,Config,ConfW,Crypto,Check,SafeP,ACL sec; class Trigger,Sched event; class TCP,HTTPS,MCP,MetSrv,WebUI server; - class MainWin,Worker ui; + class MainWin,Worker,AppLayer ui; class FileOps,Archives,DataOps,TextOps,Misc localOps; class UrlVal,Http,Drive,S3M,Azure,Dropbox,SFTP,FTP,OneD,Box,WebDAV,SMB,Fsspec,Cross remote; class NM,Sinks notify; @@ -338,14 +342,20 @@ Module layout ├── project/ │ ├── project_builder.py │ └── templates.py - ├── ui/ # PySide6 GUI + ├── app/ # application layer: what a user interface calls + │ ├── services.py # AppServices, app_services(), NAVIGATION + │ ├── pipeline_draft.py # PipelineDraft: the editable definition + │ └── *_service.py # one service per navigation entry + ├── ui/ # PySide6 GUI, built on app/ │ ├── launcher.py # launch_ui(argv) - │ ├── main_window.py # tabbed MainWindow (Home, Local, Transfer, - │ │ # Progress, JSON actions, Triggers, - │ │ # Scheduler, Servers) + │ ├── main_window.py # sidebar MainWindow (Dashboard, Files, Storage, + │ │ # Pipelines, Scheduler, Integrity, Audit, + │ │ # Notifications, Settings, Advanced) │ ├── worker.py # ActionWorker (QRunnable) │ ├── log_widget.py # LogPanel - │ └── tabs/ # one tab per backend + JSON runner + servers + │ ├── pages/ # one page per navigation entry + pipeline editor + │ └── tabs/ # the tools under Advanced: one tab per backend, + │ # JSON runner, triggers, servers └── utils/ ├── file_discovery.py ├── fast_find.py # OS-index (mdfind/locate/es) + scandir fallback diff --git a/docs/source/Eng/eng_index.rst b/docs/source/Eng/eng_index.rst index 1d7b32e..8868099 100644 --- a/docs/source/Eng/eng_index.rst +++ b/docs/source/Eng/eng_index.rst @@ -146,14 +146,16 @@ Chapter 8 — MCP Server Chapter 9 — GUI =============== -The PySide6 desktop control surface — tabbed layout, log panel, and -``ActionWorker`` thread-pool model. +The PySide6 desktop interface, organised by workflow (Dashboard, Files, Storage, +Pipelines with a visual editor, Scheduler, Integrity, Audit, Notifications, +Settings), and the application layer both user interfaces are built on. .. toctree:: :maxdepth: 2 :caption: GUI usage/gui + usage/app_layer .. _eng-reliability: diff --git a/docs/source/Eng/usage/app_layer.rst b/docs/source/Eng/usage/app_layer.rst new file mode 100644 index 0000000..5219ece --- /dev/null +++ b/docs/source/Eng/usage/app_layer.rst @@ -0,0 +1,501 @@ +Application layer +================= + +``automation_file.app`` is what a user interface calls. It has one service per +navigation entry -- Dashboard, Files, Storage, Pipelines, Scheduler, Integrity, +Audit, Notifications, Settings -- each a plain Python object on top of the +domain packages (:doc:`storage`, :doc:`pipeline`, :doc:`events`, +:doc:`integrity`, :doc:`audit`, :doc:`notifications`, :doc:`event_bus`, +:doc:`config`). + +The PySide6 window (:doc:`gui`) and the Web UI (:doc:`servers`) call these +services and nothing below them. That is why they show the same state, and why +a third interface -- a terminal UI, a web application, a chat bot -- needs no +knowledge of the domain packages either. + +The layer keeps four promises: + +* It imports no GUI toolkit and no backend SDK. ``import automation_file.app`` + works on the base install. +* It returns dataclasses, dictionaries and lists that JSON can hold. +* It masks tokens, passwords and webhook URLs in what it returns. +* It raises ``FileAutomationException`` subclasses: the domain's own + (``StorageException``, ``PipelineException`` ...) and ``AppException`` for + what only this layer checks. + +Minimal example +--------------- + +.. code-block:: python + + from automation_file.app import app_services + + services = app_services() # one set per process + + summary = services.dashboard.summary() + print(summary.status, summary.reasons) # "ok" () or "attention" (...) + + for entry in services.files.list_dir("local:///data"): + print(entry.name, entry.size) + + draft = services.pipelines.new_draft("nightly") + draft.add_task("FA_storage_copy", "download", + arguments={"source": "s3://in/a.csv", "target": "local:///tmp/a.csv"}) + run = services.pipelines.start(draft) # returns at once + services.pipelines.wait(run["run_id"], timeout=60) + print(services.pipelines.status(run["run_id"])["status"]) + +The services +------------ + +``automation_file.app.NAVIGATION`` is the tuple of the nine entries in display +order; ``AppServices`` has one attribute per entry, named in lower case. + +.. list-table:: + :header-rows: 1 + :widths: 18 24 58 + + * - Entry + - Attribute and class + - What it does + * - Dashboard + - ``dashboard``, ``DashboardService`` + - One summary: health, runs, integrity drift, recent events, storage + status. + * - Files + - ``files``, ``FileService`` + - List, stat, preview, copy, move, delete, mkdir on storage URIs. + * - Storage + - ``storage``, ``StorageService`` + - Schemes, mounts, and whether each backend can be used. + * - Pipelines + - ``pipelines``, ``PipelineService`` + - Drafts, validation, dry run, background runs, status, history, resume, + cancel, definition files. + * - Scheduler + - ``scheduler``, ``SchedulerService`` + - List, add and remove cron jobs. + * - Integrity + - ``integrity``, ``IntegrityService`` + - Baseline, verify, accept, status, start and stop a monitor. + * - Audit + - ``audit``, ``AuditService`` + - Configure, search and count audit records. + * - Notifications + - ``notifications``, ``NotificationService`` + - Registered sinks, routes, a test message. + * - Settings + - ``settings``, ``SettingsService`` + - Load and apply a configuration file; which extras are installed. + +Dashboard +--------- + +.. code-block:: python + + summary = services.dashboard.summary() + summary.to_dict() # JSON-serialisable + +.. list-table:: + :header-rows: 1 + :widths: 22 78 + + * - Field + - Meaning + * - ``status``, ``reasons`` + - ``"ok"``, or ``"attention"`` with one sentence per reason: a recent run + failed, a monitor found drift or could not verify, the bus holds a + recent event of severity ``error`` or worse. + * - ``health`` + - Counts and switches: registered actions, running runs, scheduled jobs, + monitors, sinks, routes, router active, the state of the audit trail. + * - ``run_counts`` + - How the newest fifty runs ended: ``running``, ``succeeded``, + ``failed``, ``cancelled``. + * - ``running_runs``, ``recent_runs`` + - Runs without their task details, newest first. + * - ``integrity`` + - What every named monitor last found. + * - ``events`` + - The latest events of the bus, newest first, masked. + * - ``storage`` + - Every backend with ``usable``, ``detail`` and, when a package is + missing, ``install_hint``. + +The parts are also available one by one: ``health()``, ``runs()``, +``integrity()``, ``recent_events()``, ``storage_status()``. ``summary()`` never +raises for a part that cannot be read; it names the part among the reasons. + +Files +----- + +.. code-block:: python + + files = services.files + files.list_dir("s3://reports/2026") # directories first, then by path + files.stat("s3://reports/2026/q1.csv").size + preview = files.preview("s3://reports/2026/q1.csv", max_bytes=4096) + preview.text, preview.truncated, preview.binary + files.copy("s3://reports/2026/q1.csv", "local:///backup/") # into the directory + files.move("local:///inbox/a.csv", "local:///done/a.csv") + files.mkdir("local:///backup/2026") + files.delete("local:///backup/2026", recursive=True) + +A preview is bounded twice. At most ``preview_bytes`` (64 KiB) are returned, and +a remote file larger than ``fetch_limit`` (16 MiB) is not fetched: a backend +without ranged reads has to download a file before any of it can be read. Both +limits are arguments of ``FileService``. Binary content comes back as a +hexadecimal dump with ``binary`` set. + +A directory is copied with everything below it and cannot be moved. Every URI +goes through the storage layer, so ``..`` and credentials in a URI are refused. + +Storage +------- + +.. code-block:: python + + storage = services.storage + for backend in storage.backends(): + print(backend.name, backend.kind, backend.usable, backend.detail) + storage.mount_local("sandbox://jobs", "/srv/jobs") # confined to that directory + storage.mounts() + storage.capabilities("sandbox://jobs") + storage.unmount("sandbox://jobs") + +``backends()`` returns one ``BackendStatus`` per scheme, per mount, and per +shared client without a scheme. ``usable`` is true for the local and in-memory +backends, for a mount, and for a cloud backend whose client was initialised. +Otherwise ``detail`` says how to initialise it or, when the package of its +extra is missing, carries the ``pip install`` command (also in +``install_hint``). Nothing here opens a connection. + +Pipelines +--------- + +The draft +~~~~~~~~~ + +A pipeline editor cannot edit a ``Pipeline``: that object refuses anything +invalid, and a definition under construction is invalid most of the time. A +``PipelineDraft`` holds whatever was entered so far. + +.. code-block:: python + + from automation_file.app import PipelineDraft + + draft = PipelineDraft("daily-report") + draft.set_params({"date": "2026-10-08"}) + draft.add_task("FA_storage_copy", "download", + arguments={"source": "s3://in/${params.date}.csv", + "target": "local:///tmp/report.csv"}) + draft.add_task("FA_storage_delete", "tidy", arguments={"uri": "local:///tmp/report.csv"}) + draft.connect("download", "tidy") # tidy depends on download + draft.set_retry("download", max_attempts=5, backoff=2.0, on=["ConnectionError"]) + draft.set_timeout("download", 300) + draft.set_condition("tidy", "always") + draft.rename_task("tidy", "clean-up") # edges and placeholders follow + + draft.problems() # [] or Problem(path, message, task) + draft.to_definition() # what Pipeline.from_dict takes + +.. list-table:: + :header-rows: 1 + :widths: 36 64 + + * - Method + - Effect + * - ``add_task``, ``remove_task``, ``rename_task`` + - Change the set of tasks. An ID is derived from the action name when + none is given. + * - ``set_action``, ``set_arguments`` + - The action and its keyword mapping, positional list or ``None``. + * - ``set_retry``, ``set_timeout``, ``set_condition``, ``set_idempotency_key`` + - How the task runs. + * - ``connect``, ``disconnect``, ``set_dependencies``, ``edges`` + - Dependency edges. An edge to itself or one that closes a cycle is + refused with ``AppException``. + * - ``set_position``, ``positions``, ``auto_layout``, ``layout``, ``apply_layout`` + - Where each task sits on a canvas. This is editor metadata: it never + appears in ``to_definition()``. + * - ``set_name``, ``set_description``, ``set_max_workers``, ``set_params``, ``set_schedule`` + - The header of the definition. + * - ``add_listener``, ``batch`` + - ``listener(change)`` is called after every change with ``"structure"``, + ``"task"``, ``"header"`` or ``"position"``; inside ``with draft.batch():`` + each kind is reported once, at the end. + * - ``problems``, ``to_definition``, ``from_definition`` + - Validation with the path of every problem, and the definition document. + +A value the runtime would refuse (a negative timeout, an unknown action name) +is kept and reported by ``problems()``; only an edit that would leave the draft +inconsistent (a duplicate ID, a cycle) raises at once. A draft is not +thread-safe: edit it from one thread and hand a worker ``draft.to_definition()``. + +Checking and running +~~~~~~~~~~~~~~~~~~~~ + +Every method takes a draft or a definition mapping and returns plain +dictionaries. A run is ``PipelineRun.to_dict()`` with its secrets masked and an +``active`` flag that says whether this process is still executing it. + +.. code-block:: python + + pipelines = services.pipelines + + pipelines.action_names() # for a palette + pipelines.describe_action("FA_storage_copy") # parameters, defaults, summary + + pipelines.validate(draft) # also: an unknown action name + plan = pipelines.dry_run(draft, {"date": "2026-10-09"}) + outcome = pipelines.test_task(draft, "download", {"date": "2026-10-09"}) + + run = pipelines.start(draft, {"date": "2026-10-09"}) # background + followed = pipelines.follow(run["run_id"]) # {"run": ..., "events": [...]} + pipelines.cancel(run["run_id"]) + pipelines.wait(run["run_id"], timeout=30) + + pipelines.resume(run["run_id"], draft) # keep what succeeded, run the rest + pipelines.retry(run["run_id"], draft) # a new run, same parameters + pipelines.history("daily-report", limit=10) + pipelines.running() + +``start`` and ``resume`` return at once; a definition problem raises before +anything runs. ``test_task`` executes one task alone: its action really runs, +its upstream tasks are replaced by stand-ins that return the values given in +``results=`` (``None`` by default), and the test is recorded in no run store and +published on no shared bus. + +Definition files and layout +~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. code-block:: python + + draft = pipelines.load("pipelines/daily-report.yaml") + draft.load_notes # what was wrong with the file, if anything + pipelines.save(draft, "pipelines/daily-report.yaml") + +``save`` writes the definition as ``.yaml``, ``.yml`` or ``.json`` and the canvas +layout into ``.layout.json`` next to it. A definition refuses unknown +keys, and a layout is no part of what runs, so the two never share a file. +``load`` opens a file that parses but is not valid, so it can be repaired. + +Scheduler +--------- + +.. code-block:: python + + scheduler = services.scheduler + scheduler.add("nightly", "0 2 * * *", + [["FA_pipeline_run", {"definition": "pipelines/daily-report.yaml"}]]) + scheduler.jobs() + scheduler.remove("nightly") + scheduler.remove_all() + +``add`` also takes the action list as JSON text, as a form supplies it. The +service calls only ``schedule_add``, ``schedule_remove``, ``schedule_remove_all`` +and ``schedule_list`` (:doc:`events`). + +Integrity +--------- + +.. code-block:: python + + integrity = services.integrity + integrity.baseline("s3://reports/2026", "local:///var/lib/fa/reports.json") + report = integrity.verify("s3://reports/2026", "local:///var/lib/fa/reports.json") + report["ok"], report["counts"], report["changes"] + integrity.accept("s3://reports/2026", "local:///var/lib/fa/reports.json") + + integrity.start_monitor("reports", "s3://reports/2026", + "local:///var/lib/fa/reports.json", interval=300) + integrity.drift() # MonitorDrift per monitor, for a dashboard + integrity.stop_monitor("reports") + integrity.stop_started() # the monitors this service started + +A monitor started here is the named monitor the ``FA_integrity_*`` actions see. + +Audit +----- + +.. code-block:: python + + audit = services.audit + audit.status() # {"configured": False, ...} until it has a store + audit.configure("/var/lib/automation_file/audit.sqlite") + audit.search(status="error", resource_prefix="s3://reports/", limit=20) + audit.count(actor="scheduler") + audit.recent(limit=30) # [] while audit is not configured + +A filter given as ``None`` or as an empty text does not restrict the search, so +the values of a form can be passed as they are. + +Notifications +------------- + +.. code-block:: python + + notifications = services.notifications + notifications.sinks() # name, type, destination; never a secret + notifications.add_route({"name": "failures", "sinks": "team-alerts, ops-mail", + "types": "pipeline.failed, task.failed", + "min_severity": "error", "dedup_seconds": "600"}) + notifications.routes() + notifications.remove_route("failures") + notifications.send_test("team-alerts") # {"team-alerts": "sent"} + +Lists may be comma-separated text and numbers may be text. A route that names a +sink that is not registered is refused. ``send_test`` returns one outcome per +sink: ``"sent"`` or the error, with its URLs reduced to the host. + +Settings +-------- + +.. code-block:: python + + settings = services.settings + settings.load("automation_file.toml") # a summary; nothing changes + settings.apply("automation_file.toml") # registers sinks and routes + for extra in settings.extras(): + print(extra.name, extra.installed, extra.install_hint) + settings.environment() # versions, platform, log file + +The summary holds the file's sections, sinks and routes, and the document with +every secret masked. + +Form helpers +------------ + +.. list-table:: + :header-rows: 1 + :widths: 32 68 + + * - Function + - What it does + * - ``parse_argument_text(text)`` + - One form field to a value: JSON when the text is JSON, else the text. + * - ``format_argument_value(value)`` + - The way back: text that reads back as the same value. + * - ``parse_json_text(text, what)`` + - A JSON document from a text field, or ``AppException`` naming ``what`` + and the position of the mistake. + * - ``split_names(text)`` + - ``"a, b"`` to ``["a", "b"]``. + * - ``describe_action(name, command)`` + - The parameters, the defaults and the summary of an action. + +Secrets +------- + +``mask_secrets(value)`` returns a copy that is safe to show, and every service +applies it to what it returns: + +* a value stored under a name that says it is a secret (``password``, + ``token``, ``api_key``, ``authorization`` ...) becomes ``********``; +* a value stored under ``url`` or ``..._url`` keeps its scheme and host; +* in any other text, the user information of a URL and the token after + ``Bearer`` are removed. + +A storage URI is left alone: it cannot carry credentials. A task's result is +returned as it is, so keep secrets out of results. + +Shared and private services +--------------------------- + +``app_services()`` returns one set per process, built on the process-wide +singletons (the default resolver, the default run store, the event bus, the +audit trail, the notification manager and router). The window and the Web UI of +one process therefore show the same runs, monitors and routes. + +``build_services`` makes a set of its own: + +.. code-block:: python + + from automation_file.app import ServiceOptions, build_services + from automation_file.events import EventBus + from automation_file.pipeline import SQLiteRunStore + + services = build_services(ServiceOptions( + run_store=SQLiteRunStore("/var/lib/automation/pipelines.db"), + bus=EventBus(), + )) + +``ServiceOptions`` takes ``resolver``, ``run_store``, ``registry``, ``bus``, +``audit_trail``, ``notification_manager`` and ``notification_router``. The +scheduler and the integrity monitors are process-wide whatever the options. + +Writing another user interface +------------------------------ + +1. Get the services: ``app_services()``, or ``build_services(...)``. +2. Build the navigation from ``NAVIGATION``. +3. For each view, call one service method and render what it returns. Catch + ``FileAutomationException`` and show its text. +4. Call from a worker thread whatever touches storage or the network + (``files.*``, ``integrity.verify``, ``notifications.send_test``, + ``settings.apply``). ``pipelines.start`` and ``pipelines.resume`` already + return at once. +5. For a pipeline editor, keep one ``PipelineDraft``, change it only through + its methods, and redraw from it in a listener. + +A complete, if small, terminal interface: + +.. code-block:: python + + from automation_file.app import NAVIGATION, app_services + from automation_file.exceptions import FileAutomationException + + services = app_services() + views = { + "Dashboard": lambda: services.dashboard.summary().to_dict(), + "Storage": lambda: [backend.to_dict() for backend in services.storage.backends()], + "Pipelines": lambda: services.pipelines.history(limit=10), + "Scheduler": services.scheduler.jobs, + "Integrity": lambda: [drift.to_dict() for drift in services.integrity.drift()], + "Audit": services.audit.recent, + "Notifications": services.notifications.routes, + "Settings": lambda: [extra.to_dict() for extra in services.settings.extras()], + } + for number, name in enumerate(NAVIGATION, start=1): + print(number, name) + chosen = NAVIGATION[int(input("> ")) - 1] + try: + print(views.get(chosen, lambda: "use services.files for Files")()) + except FileAutomationException as error: + print("failed:", error) + +When something goes wrong +------------------------- + +``AppException`` + The layer refused the request: an edit the draft cannot take, form text + that is not JSON, a missing name. The message says what to change. + +``PipelineDefinitionException`` from ``dry_run``, ``start`` or ``resume`` + The definition is not valid. ``error.problems`` holds every finding; + ``validate`` returns the same as ``Problem`` objects without raising. + +``status`` raises "unknown run" + The run is neither tracked by this service nor in its run store. Runs are + kept in memory unless a ``SQLiteRunStore`` is the default store or was + passed to ``build_services``. + +``cancel`` returns ``False`` + The run is not executing in this process: it has ended, or another process + started it. + +A backend is not ``usable`` + Read ``detail``. Initialise the client, or install the extra named by + ``install_hint``. + +``audit.search`` raises "audit is not configured" + Call ``audit.configure(path)`` first. ``audit.recent()`` returns an empty + list instead. + +A secret shows up in a view + It was stored under a name that does not say it is a secret, or it is part + of a task's result. Rename the field, or keep it out of the result. + +Two interfaces disagree + They use different service sets. ``app_services()`` is shared; + ``build_services()`` is not. diff --git a/docs/source/Eng/usage/gui.rst b/docs/source/Eng/usage/gui.rst index 9e684e3..cffdb63 100644 --- a/docs/source/Eng/usage/gui.rst +++ b/docs/source/Eng/usage/gui.rst @@ -1,10 +1,17 @@ GUI (PySide6) ============= -A tabbed control surface wraps every feature: +The desktop window is organised by what you want to get done, not by backend: +a sidebar with nine workflow pages, and an **Advanced** entry that keeps the +tools addressing a single action or a single backend. + +Every page is a thin view over one service of the application layer +(:doc:`app_layer`). The window holds no logic of its own, so what it shows is +what the Web UI (:doc:`servers`), the CLI and your own Python code see. .. code-block:: bash + pip install "automation_file[gui]" # PySide6 is an extra, not a base dependency python -m automation_file ui # or from the repo root during development: python main_ui.py @@ -15,11 +22,486 @@ A tabbed control surface wraps every feature: launch_ui() -Tabs: Home, Local, Transfer, Progress, JSON actions, Triggers, Scheduler, -Servers. A persistent log panel below the tabs streams every call's result -or error. Background work runs on ``QThreadPool`` via ``ActionWorker`` so -the UI stays responsive. +Without PySide6, ``launch_ui`` raises ``OptionalDependencyException`` with the +``pip install`` command above. + +Navigation +---------- + +.. list-table:: + :header-rows: 1 + :widths: 18 12 70 + + * - Entry + - Shortcut + - What it is for + * - Dashboard + - ``Ctrl+1`` + - Health, running and recent pipeline runs, integrity drift, recent + events and storage status at one glance. + * - Files + - ``Ctrl+2`` + - Browse a storage URI, preview a file, copy, move, delete, create a + directory. + * - Storage + - ``Ctrl+3`` + - Which backends exist, whether each can be used, and the mounts. + * - Pipelines + - ``Ctrl+4`` + - The visual pipeline editor: build, validate, dry-run, test, run, resume. + * - Scheduler + - ``Ctrl+5`` + - Cron jobs: list, add, remove. + * - Integrity + - ``Ctrl+6`` + - Baseline, verify and accept a tree; start and stop monitors. + * - Audit + - ``Ctrl+7`` + - Point the audit trail at a database, search and count its records. + * - Notifications + - ``Ctrl+8`` + - Registered sinks, routes, and a test message. + * - Settings + - ``Ctrl+9`` + - The configuration file, the optional extras, the environment. + * - Advanced + - ``Ctrl+0`` + - Local, Transfer, Progress, JSON actions, Triggers and Servers: the tabs + of the earlier window. See `Where the old tabs went`_. + +The window +---------- + +The sidebar is on the left and the selected page on the right. Below both is +the **activity log**, shared by every page: each action writes a line when it +starts and another with its outcome, prefixed with the page name. The newest +line also appears in the status bar for a few seconds. + +Every page has a **status line** at its bottom edge. It holds the outcome of +the last thing you did on that page: green when it worked, red when it did not. + +Nothing blocks the window. A page hands every call to a thread pool +(``ActionWorker`` on ``QThreadPool``) and shows the result when it arrives. A +page reads its data when you open it and again when you press its **Refresh** +button. Two views refresh by themselves: the Dashboard every five seconds while +it is visible, and a followed pipeline run twice a second until it has ended. + +The window shares the process-wide singletons with the rest of the library. A +sink registered from Python, a pipeline started through the HTTP action server +or a monitor started by a JSON action shows up after the next refresh. + +Dashboard +--------- + +**All clear** or **Needs attention**, with the reasons next to it. The +dashboard asks for attention when one of the newest fifty runs failed, a +monitor found drift or could not verify, or the event bus holds a recent event +of severity ``error`` or worse. + +* **Health**: registered actions, running runs, scheduled jobs, integrity + monitors, notification sinks and routes, whether the router is active, and + whether the audit trail is recording. +* **Pipeline runs**: how the newest runs ended, the runs still going, and the + latest results with their error. +* **Integrity drift**: each named monitor, its target, and what it last found. +* **Storage status**: each backend and whether it can be used. +* **Recent events**: the twenty latest events of the bus, newest first. + +**Refresh every 5 s** turns the timer off and on; **Refresh** reads at once. +The dashboard only reads: it starts nothing and changes nothing. + +Files +----- + +Type a storage URI or a local path (``s3://reports/2026``, ``local:///data``, +``C:\data``, ``memory://demo``) and press **Open**. **Up** goes to the parent, +**Browse local…** picks a directory. Double-click a directory to enter it and a +file to preview it. + +A preview is bounded: at most 64 KiB is shown, and a file on a remote backend +that is larger than 16 MiB is not fetched at all (the status line says so). +Binary content is shown as a hexadecimal dump of its first 256 bytes. + +The operations act on the selected entry: + +* **Copy** / **Move** to the target URI, in the same backend or another one. An + existing directory as the target receives the file under its own name. A + directory is copied with everything below it; a directory cannot be moved + (copy it, check the copy, delete the original). +* **Create directory** below the current location. +* **Delete selected** asks first. A directory with entries needs **Delete a + directory with its contents**. + +Every path goes through the storage layer (:doc:`storage`): ``..`` is refused, +credentials in a URI are refused, and a mounted directory cannot be left. + +Storage +------- + +One row per backend: + +.. list-table:: + :header-rows: 1 + :widths: 14 86 + + * - Kind + - Meaning + * - ``scheme`` + - A URI scheme with a factory: ``local``, ``memory``, ``s3``, ``azure``, + ``gdrive``, ``dropbox``, ``onedrive``, ``sftp``, ``ftp``, ``ftps``. + * - ``mount`` + - A backend mounted at a URI. Its name is that URI. + * - ``client`` + - A shared client that has ``FA_*`` actions but no storage scheme (Box). + +**Usable** is ``yes`` for the local and in-memory backends, for a mount, and +for a cloud backend whose client has been initialised. **Detail** says what is +missing: how to initialise the client, or, when the package of its extra is not +installed, the ``pip install`` command. + +**Mount a local directory** serves a URI of your choice from a directory and +confines it to that directory (``sandbox://jobs`` on ``/srv/jobs``). Use it for +paths that come from outside the process. **Unmount selected** removes a mount. +Selecting a mount, ``local`` or ``memory`` shows what the backend provides +(directories, modification times, ETags ...). + +Pipelines +--------- + +The page is an editor for a pipeline definition (:doc:`pipeline`) and a +console for its runs. + +.. list-table:: + :header-rows: 1 + :widths: 24 76 + + * - Area + - What it holds + * - Top row + - **New**, **Open…**, **Save**, **Save as…**, **Auto layout**, and the file + name (with ``(modified)`` while there are unsaved changes). + * - Second row + - The pipeline's name, **Max workers**, description, default parameters + (JSON) and schedule (a cron expression, kept for the scheduler). + * - Actions (left) + - Every registered action, with a filter. + * - Canvas (centre) + - One node per task and one arrow per dependency, with **Connect**, + **Disconnect** and **Remove selected** above it. + * - Selected task (right) + - The form of the task selected on the canvas. + * - Run bar + - The run parameters (JSON) and **Validate**, **Dry run**, **Test task**, + **Run**, **Resume**, **Retry**, **Cancel**, **Refresh history**, + **Follow selected run**. + * - Tabs at the bottom + - **Problems**, **Tasks**, **Log**, **History**. + +The pipeline editor step by step +-------------------------------- + +1. **Start.** Press **New** for an empty definition or **Open…** for a + ``.yaml`` / ``.yml`` / ``.json`` file. A file that is not a valid definition + still opens: whatever has the right shape is shown, and the **Problems** tab + lists what is wrong with it. + +2. **Name it.** Fill in the name, the number of workers and, if you like, the + description, the default parameters as a JSON object and the schedule. + +3. **Add tasks.** Drag an action from the **Actions** list onto the canvas: the + task appears where you drop it. Double-clicking an action, or selecting it + and pressing **Add task**, adds it below the lowest task. The filter narrows + the list (``storage`` shows the ``FA_storage_*`` actions). The task's ID is + derived from the action name; change it in the form. + +4. **Arrange.** Drag the nodes where you want them. **Auto layout** puts each + task in a column by its dependency depth. Positions are editor metadata: + they are saved next to the definition, never in it. + +5. **Connect.** Click the upstream task, ``Ctrl``-click the task that depends + on it, and press **Connect**. The button shows the direction it will use + (``Connect download -> publish``): the order in which you selected the two + tasks is the direction of the arrow. An arrow that would close a dependency + cycle is refused. You can also tick the upstream task under **Depends on** + in the task form. + + To remove a dependency, click its arrow (or select its two tasks) and press + **Disconnect**. **Remove selected** removes the selected arrows, or, when no + arrow is selected, the selected tasks with their arrows. + +6. **Edit the selected task.** The form shows the task selected last: + + .. list-table:: + :header-rows: 1 + :widths: 24 76 + + * - Field + - Meaning + * - Task ID + - Unique in the pipeline. Renaming keeps the arrows and rewrites the + ``${tasks..result}`` placeholders that point at the task. + * - Action + - A registered action. Its signature and summary appear below it. + * - Arguments + - One row per parameter of the action, with its default. A value is + JSON when it parses as JSON (``12``, ``true``, ``["a", "b"]``, + ``"12"``) and text otherwise, so a URI or ``${params.date}`` needs no + quotes. A row left empty is not passed, so the default applies. + **Add argument** adds a row for an action that takes ``**kwargs``. + **Edit as JSON** shows the arguments as one JSON document; positional + arguments (a JSON array) can only be edited that way. + * - Depends on + - Tick the tasks that must end first. + * - Attempts, Back-off, Back-off cap, Retry on + - Attempts in total, the first back-off and its cap, and the exception + names worth another attempt (empty: the transient ones). + * - Timeout + - Seconds for the whole task; empty for no limit. + * - Run when + - ``on_success``, ``on_failure`` or ``always``. + * - Idempotency key + - Text with ``${params.}`` placeholders; empty for none. + + Nothing changes until you press **Apply changes**. **Revert** throws away + what you typed, and so does selecting another task. When the draft refuses a + value, the form says why and keeps what you typed. + +7. **Validate.** The **Problems** tab lists every finding with the path of the + entry it is about (``tasks.publish.depends_on[0]: unknown task 'x'``), + including an action name the registry does not know. Click a problem to + select its task. **Dry run**, **Test task**, **Run**, **Resume** and + **Retry** validate first and stop when there is a problem. + +8. **Dry run.** Enter the run parameters as a JSON object and press + **Dry run**. Nothing is executed or recorded. The **Tasks** tab shows every + task as ``planned`` with its level, and a task that would not run as planned + says why (a parameter the run was not given, for instance). + +9. **Test one task.** Select a task and press **Test task**. Its action really + runs, with its retry policy and its timeout. Its upstream tasks do not: each + is replaced by a stand-in that returns nothing, so a + ``${tasks..result}`` placeholder receives ``null``. A test is recorded in + no run store and published on no shared event bus. + +10. **Run.** **Run** starts the pipeline in the background and follows it: the + **Tasks** tab shows status, attempts, duration and error of every task, the + **Log** tab the events of the run, and each node takes the colour of its + status (grey pending, blue running, green succeeded, red failed or timed + out, orange cancelled, yellow skipped). **Cancel** asks the run to stop. + +11. **Resume or retry.** After a failure, fix the cause. **Resume** continues + the same run: tasks that succeeded are kept, the rest run again, with the + same run ID and parameters. **Retry** starts a new run with the parameters + of the followed one. + +12. **Look back.** The **History** tab lists the recorded runs, newest first. + Double-click one, or select it and press **Follow selected run**, to see + its tasks and events again; **Resume** and **Retry** then act on it. + +13. **Save.** **Save as…** writes the definition as ``.yaml``, ``.yml`` or + ``.json``, and the canvas layout into ``.layout.json`` next to it. The + definition file is exactly what ``Pipeline.from_file`` and + ``FA_pipeline_run`` take. + +Scheduler +--------- + +**Schedule a job** takes a unique name, a five-field cron expression and the +action list as JSON, and **Add job** registers it. Tick the checkbox to let a +run start while the previous one is still going; otherwise that tick is skipped +and counted under **Skipped**. The table shows every job with its runs and its +last run; **Remove selected** and **Remove all** remove jobs. To run a pipeline +on a schedule, schedule ``FA_pipeline_run`` with the path of its definition. + +The jobs are removed when the window closes. + +Integrity +--------- + +Enter the **Target** (the tree to check) and the **Baseline** (where the +approved state is kept), both storage URIs or local paths. + +* **Create baseline** approves what is there now. +* **Verify** compares the tree with the baseline and lists every change with + its kind and path. With **Deep** off, only files whose size or time changed + are hashed, and the summary says it was a quick pass. +* **Accept current state** asks first, then stores the current tree as the new + baseline. + +Under **Monitors**, give a name and an interval and press **Start monitor** to +verify the target on a thread; the table shows what each monitor last found. +Monitors started from the window stop when it closes. See :doc:`integrity`. + +Audit +----- + +The audit trail records nothing until it has a store. Enter the path of a +SQLite database (it is created when missing) and press **Configure**; **State** +then says where the records go. + +Fill in any of the filters and press **Search** (newest first, up to +**Limit**) or **Count**. A filter left empty does not restrict the search. +**Since** and **Until** take an ISO 8601 time with an offset. Select a record +to see all of it, as JSON. See :doc:`audit`. + +Notifications +------------- + +**Sinks** lists the registered sinks by name, type and destination. Sinks are +registered in code or from the configuration file (Settings); this page does +not create them. Choose a sink (or **All sinks**), optionally a subject, and +press **Send test message**; the status line shows one outcome per sink. + +**Routes** lists which events reach which sinks. **Add or replace a route** +takes a name, sink names, event types (``pipeline.*``, ``task.failed``), +sources, the minimum severity, and the throttling values; lists are separated +by commas. A route that names a sink that is not registered is refused. The +router starts with the first route and stops when the last one is removed. See +:doc:`notifications`. + +Settings +-------- + +Enter the path of ``automation_file.toml`` (:doc:`config`). + +* **Preview** reads the file and shows its summary. Nothing changes. +* **Apply** registers its notification sinks and routes. + +**Optional extras** lists every extra with what it enables, whether it is +installed, and the ``pip install`` command when it is not. **Environment** +shows the package and Python versions, the platform, the log file and the +configuration applied last. + +Where the old tabs went +----------------------- + +Nothing was removed. The **Advanced** entry holds the earlier tabs, unchanged: + +.. list-table:: + :header-rows: 1 + :widths: 28 72 + + * - Earlier tab + - Now + * - Home + - **Dashboard** (backend readiness is under *Storage status*, and on the + **Storage** page). + * - Local + - **Advanced** → *Local*. Browsing and copying are also on **Files**. + * - Transfer (HTTP, Google Drive, S3, Azure Blob, Dropbox, SFTP, OneDrive, + Box) + - **Advanced** → *Transfer*. This is where a cloud client is given its + credentials. + * - Progress + - **Advanced** → *Progress*. + * - JSON actions + - **Advanced** → *JSON actions*. + * - Triggers + - **Advanced** → *Triggers*. + * - Scheduler + - **Scheduler**. + * - Servers + - **Advanced** → *Servers*. + +Secrets +------- + +No page shows a secret. Sinks are described by name, type and destination; a +configuration summary has its passwords and tokens replaced by ``********`` and +its webhook URLs reduced to the host; events, audit records and run parameters +are masked the same way before they are displayed; credentials inside a URL and +bearer tokens are removed from the messages a page shows. + +A value is masked when the *name* it is stored under says it is a secret +(``password``, ``token``, ``api_key``, ``authorization`` ...). A task's result +is shown as the task returned it. So give a secret parameter such a name, and +keep secrets out of results. + +Running without a display +------------------------- + +Qt needs a display to open a window. On a server without one: + +* Use the Web UI (:doc:`servers`) for the read-only views; it is rendered from + the same application layer. +* Use the application layer itself from Python (:doc:`app_layer`), or the + ``FA_*`` actions through the CLI and the action servers. Everything the + window does is a call to one of them. +* To construct the window without showing it (tests, screenshots in CI), set + ``QT_QPA_PLATFORM=offscreen`` before Qt is imported: + + .. code-block:: python + + import os + os.environ.setdefault("QT_QPA_PLATFORM", "offscreen") + + from PySide6.QtWidgets import QApplication + from automation_file.app import build_services + from automation_file.ui.main_window import MainWindow + + app = QApplication([]) + window = MainWindow(build_services()) # or MainWindow() for the shared services + window.navigate("Pipelines") + window.close() + +``import automation_file`` and ``import automation_file.app`` never import +PySide6; only ``automation_file.ui`` does. + +When something goes wrong +------------------------- + +A button seems to do nothing + Read the status line at the bottom of the page and the activity log. Every + refusal and every failure is reported there, with the reason. + +``launch_ui`` raises ``OptionalDependencyException`` + PySide6 is not installed: ``pip install "automation_file[gui]"``. + +The window does not open: "could not load the Qt platform plugin" + There is no display. See `Running without a display`_. + +A cloud backend is "not initialised" + Its shared client has no credentials yet. Open **Advanced** → *Transfer*, + choose the backend and fill in *Credentials*, or run its + ``FA_*_later_init`` action. + +Detail says a package "is not installed" + Install the extra the row names; **Settings** lists every extra with its + command. + +**Connect** is refused + The arrow would close a dependency cycle, or fewer or more than two tasks + are selected. + +**Run** stops at "problem(s): see the Problems tab" + The definition is not valid. Each problem starts with the path of the entry + it is about; click it to select the task. + +A run stays ``running`` after **Cancel** + A running action cannot be interrupted. Tasks that had not started are + cancelled at once; the run ends when the running ones return. + +**Resume** is refused + The run is still executing, belongs to a pipeline with another name, or the + definition now needs a parameter the stored run does not have. + +The history is empty after a restart + Runs are kept in memory by default. Call + ``set_default_run_store(SQLiteRunStore(path))`` before ``launch_ui()`` to + keep them in a file. + +**Search** says audit is not configured + Give the audit trail a database first, on the same page. + +A test message fails + The status line shows the error of each sink, with URLs reduced to their + host. The sink itself is described in :doc:`notifications`. + +The page shows old data + Press **Refresh**. Only the Dashboard and a followed run refresh by + themselves. -The GUI shares the same singletons as the rest of the library — registering -a sink, custom command, or trigger from Python takes effect immediately in -the running window. +Something kept running after the window closed + Closing the window removes the scheduled jobs, stops the monitors, action + servers and triggers it started, and leaves pipeline runs alone: a run + that was started goes on to its end, and the process does not exit before. diff --git a/docs/source/Eng/usage/pipeline.rst b/docs/source/Eng/usage/pipeline.rst index 2e32fb3..91a759f 100644 --- a/docs/source/Eng/usage/pipeline.rst +++ b/docs/source/Eng/usage/pipeline.rst @@ -14,6 +14,11 @@ bus (:doc:`event_bus`); it calls no notification sink and writes no audit row. list of actions once and returns their results; use a pipeline when the run has to be recorded, retried, resumed or observed. +A definition can also be built, checked and run without writing it by hand: the +Pipelines page of the desktop window is a visual editor for it (:doc:`gui`), +and ``automation_file.app`` offers the same operations to any other interface +(:doc:`app_layer`). + Minimal example --------------- diff --git a/docs/source/Eng/usage/servers.rst b/docs/source/Eng/usage/servers.rst index 70d190b..cf9cb2f 100644 --- a/docs/source/Eng/usage/servers.rst +++ b/docs/source/Eng/usage/servers.rst @@ -46,3 +46,78 @@ bind elsewhere. The shared secret comparison uses :func:`hmac.compare_digest` (constant time). Never log the secret or the raw payload. + +Web UI +------ + +A read-only dashboard in the browser, served with the standard library and +HTMX (one script from a pinned CDN URL, with an SRI hash). + +.. code-block:: python + + from automation_file import start_web_ui + + server = start_web_ui(host="127.0.0.1", port=9955, shared_secret="optional-secret") + # Browse http://127.0.0.1:9955/ + # later: + server.shutdown() + server.server_close() + +The page polls one HTML fragment per section. Every fragment but the transfer +progress is rendered from the application layer (:doc:`app_layer`), the same +services the desktop window (:doc:`gui`) calls, so the two show the same state. + +.. list-table:: + :header-rows: 1 + :widths: 22 14 64 + + * - Fragment + - Polled + - Shows + * - ``GET /ui/health`` + - 5 s + - ``ok`` or ``attention`` with the reasons; registered actions, running + runs, scheduled jobs, monitors, sinks and routes, the audit trail. + * - ``GET /ui/runs`` + - 3 s + - Running and recent pipeline runs, and how the newest runs ended. + * - ``GET /ui/integrity`` + - 10 s + - Each named integrity monitor and the drift it last found. + * - ``GET /ui/events`` + - 5 s + - The latest events of the bus, newest first. + * - ``GET /ui/storage`` + - 30 s + - Each storage backend and whether it can be used. + * - ``GET /ui/audit`` + - 10 s + - The latest audit records, once ``configure_audit`` gave the trail a + store. + * - ``GET /ui/progress`` + - 2 s + - Live transfers of the progress registry. + * - ``GET /ui/registry`` + - 30 s + - The name of every registered action. + +``GET /`` and ``GET /index.html`` serve the page; any other path returns +``404``. There is no route that changes anything: run actions through the +action servers above, with their own authentication. + +* **Loopback only by default.** ``allow_non_loopback=True`` is required to bind + elsewhere, and doing so without a ``shared_secret`` logs a warning. +* **Shared secret.** With ``shared_secret``, every request needs + ``Authorization: Bearer `` and gets ``401`` without it. The page + carries the header in ``hx-headers`` so its own polling is authorised; anyone + who can read the page can therefore read the secret, so serve it over + loopback or behind TLS. +* **Escaped and masked.** Everything a fragment shows is HTML-escaped, and the + application layer has already masked tokens, passwords and webhook URLs in + it. +* **A fragment never breaks the page.** When a service cannot answer, its + fragment says ``unavailable: `` and the others keep working. + +``start_web_ui(services=...)`` takes a set built with +``automation_file.app.build_services`` to show another run store, event bus or +resolver than the process-wide ones. diff --git a/docs/source/Zh-CN/architecture.rst b/docs/source/Zh-CN/architecture.rst index c6378e3..14ed5a9 100644 --- a/docs/source/Zh-CN/architecture.rst +++ b/docs/source/Zh-CN/architecture.rst @@ -75,7 +75,8 @@ end subgraph UI["ui (PySide6)"] - MainWin["MainWindow
    Home · Local · HTTP · Drive · S3 · Azure · Dropbox
    SFTP · OneDrive · Box · JSON · Triggers · Scheduler
    Progress · Transfer · Servers"] + MainWin["MainWindow
    Dashboard · Files · Storage · Pipelines · Scheduler
    Integrity · Audit · Notifications · Settings
    Advanced: Local · Transfer · Progress · JSON · Triggers · Servers"] + AppLayer["automation_file.app
    one service per navigation entry"] Worker["ActionWorker
    QRunnable on QThreadPool"] end @@ -125,6 +126,7 @@ Plugins ==> Loader MainWin ==> Worker + Worker ==> AppLayer Worker ==> PublicAPI PublicAPI ==> Executor @@ -140,6 +142,8 @@ HTTPS ==> Executor MCP ==> Registry MetSrv ==> Metrics + WebUI ==> AppLayer + AppLayer ==> PublicAPI WebUI ==> Registry ACL ==> TCP ACL ==> HTTPS @@ -235,7 +239,7 @@ class Secrets,Config,ConfW,Crypto,Check,SafeP,ACL sec; class Trigger,Sched event; class TCP,HTTPS,MCP,MetSrv,WebUI server; - class MainWin,Worker ui; + class MainWin,Worker,AppLayer ui; class FileOps,Archives,DataOps,TextOps,Misc localOps; class UrlVal,Http,Drive,S3M,Azure,Dropbox,SFTP,FTP,OneD,Box,WebDAV,SMB,Fsspec,Cross remote; class NM,Sinks notify; @@ -331,14 +335,20 @@ ├── project/ │ ├── project_builder.py │ └── templates.py - ├── ui/ # PySide6 GUI + ├── app/ # 应用层：用户界面所调用的那一层 + │ ├── services.py # AppServices、app_services()、NAVIGATION + │ ├── pipeline_draft.py # PipelineDraft：可编辑的定义 + │ └── *_service.py # 每个导航条目一个服务 + ├── ui/ # PySide6 GUI，建立在 app/ 之上 │ ├── launcher.py # launch_ui(argv) - │ ├── main_window.py # 标签式 MainWindow（Home、Local、Transfer、 - │ │ # Progress、JSON actions、Triggers、 - │ │ # Scheduler、Servers） + │ ├── main_window.py # 侧边栏式 MainWindow（Dashboard、Files、Storage、 + │ │ # Pipelines、Scheduler、Integrity、Audit、 + │ │ # Notifications、Settings、Advanced） │ ├── worker.py # ActionWorker（QRunnable） │ ├── log_widget.py # LogPanel - │ └── tabs/ # 每个后端一个标签 + JSON runner + servers + │ ├── pages/ # 每个导航条目一个页面 + 流水线编辑器 + │ └── tabs/ # Advanced 之下的工具：每个后端一个标签、 + │ # JSON runner、triggers、servers └── utils/ ├── file_discovery.py ├── fast_find.py # OS 索引（mdfind/locate/es）+ scandir 兜底 diff --git a/docs/source/Zh-CN/usage/app_layer.rst b/docs/source/Zh-CN/usage/app_layer.rst new file mode 100644 index 0000000..cae6371 --- /dev/null +++ b/docs/source/Zh-CN/usage/app_layer.rst @@ -0,0 +1,475 @@ +应用层 +====== + +``automation_file.app`` 是用户界面所调用的那一层。导航中的每个条目各有一个服务—— +Dashboard、Files、Storage、Pipelines、Scheduler、Integrity、Audit、Notifications、 +Settings——每个都是建立在领域包（:doc:`storage`、:doc:`pipeline`、:doc:`events`、 +:doc:`integrity`、:doc:`audit`、:doc:`notifications`、:doc:`event_bus`、 +:doc:`config`）之上的普通 Python 对象。 + +PySide6 窗口（:doc:`gui`）与 Web UI（:doc:`servers`）只调用这些服务，不碰它们之下的 +任何东西。这就是两者显示相同状态的原因，也是第三种界面——终端 UI、Web 应用、聊天 +机器人——同样不需要了解领域包的原因。 + +这一层遵守四个承诺： + +* 不导入任何 GUI 工具包，也不导入任何后端 SDK。``import automation_file.app`` 在 + 基础安装上就能工作。 +* 返回 JSON 能容纳的 dataclass、字典与列表。 +* 在返回的内容中屏蔽 token、密码与 webhook URL。 +* 抛出 ``FileAutomationException`` 的子类：领域自己的（``StorageException``、 + ``PipelineException``……），以及只有这一层才检查的情况所用的 ``AppException``。 + +最小示例 +---------------- + +.. code-block:: python + + from automation_file.app import app_services + + services = app_services() # 每个进程一组 + + summary = services.dashboard.summary() + print(summary.status, summary.reasons) # "ok" () 或 "attention" (...) + + for entry in services.files.list_dir("local:///data"): + print(entry.name, entry.size) + + draft = services.pipelines.new_draft("nightly") + draft.add_task("FA_storage_copy", "download", + arguments={"source": "s3://in/a.csv", "target": "local:///tmp/a.csv"}) + run = services.pipelines.start(draft) # 立即返回 + services.pipelines.wait(run["run_id"], timeout=60) + print(services.pipelines.status(run["run_id"])["status"]) + +服务一览 +---------------- + +``automation_file.app.NAVIGATION`` 是九个条目按显示顺序排列的 tuple； +``AppServices`` 为每个条目各有一个属性，名称为小写。 + +.. list-table:: + :header-rows: 1 + :widths: 18 24 58 + + * - 条目 + - 属性与类 + - 用途 + * - Dashboard + - ``dashboard``、``DashboardService`` + - 一份摘要：健康状态、运行、完整性漂移、最近的事件、存储状态。 + * - Files + - ``files``、``FileService`` + - 对存储 URI 进行列出、stat、预览、复制、移动、删除、mkdir。 + * - Storage + - ``storage``、``StorageService`` + - Scheme、挂载点，以及每个后端能不能用。 + * - Pipelines + - ``pipelines``、``PipelineService`` + - 草稿、验证、试运行、后台运行、状态、历史、续跑、取消、定义文件。 + * - Scheduler + - ``scheduler``、``SchedulerService`` + - 列出、添加与移除 cron 作业。 + * - Integrity + - ``integrity``、``IntegrityService`` + - 基线、验证、接受、状态、启动与停止监控器。 + * - Audit + - ``audit``、``AuditService`` + - 配置、搜索与统计审计记录。 + * - Notifications + - ``notifications``、``NotificationService`` + - 已注册的 sink、路由、测试消息。 + * - Settings + - ``settings``、``SettingsService`` + - 加载并应用配置文件；哪些 extra 已安装。 + +Dashboard +--------- + +.. code-block:: python + + summary = services.dashboard.summary() + summary.to_dict() # 可序列化成 JSON + +.. list-table:: + :header-rows: 1 + :widths: 22 78 + + * - 字段 + - 含义 + * - ``status``、``reasons`` + - ``"ok"``，或是 ``"attention"`` 并为每个原因附上一句话：最近有运行失败、某个 + 监控器发现漂移或无法验证、总线上有严重程度为 ``error`` 及以上的近期事件。 + * - ``health`` + - 计数与开关：已注册的动作、运行中的运行、调度作业、监控器、sink、路由、路由器 + 是否启用、审计轨迹的状态。 + * - ``run_counts`` + - 最新五十次运行的结局：``running``、``succeeded``、``failed``、 + ``cancelled``。 + * - ``running_runs``、``recent_runs`` + - 不含任务细节的运行，新的在前。 + * - ``integrity`` + - 每个具名监控器上次发现了什么。 + * - ``events`` + - 总线上最新的事件，新的在前，已屏蔽。 + * - ``storage`` + - 每个后端，附 ``usable``、``detail``，以及缺少包时的 ``install_hint``。 + +各部分也可以分开获取：``health()``、``runs()``、``integrity()``、 +``recent_events()``、``storage_status()``。``summary()`` 绝不会因为某个部分读不到而 +抛出异常；它会把那个部分列在原因里。 + +Files +----- + +.. code-block:: python + + files = services.files + files.list_dir("s3://reports/2026") # 目录在前，其次按路径 + files.stat("s3://reports/2026/q1.csv").size + preview = files.preview("s3://reports/2026/q1.csv", max_bytes=4096) + preview.text, preview.truncated, preview.binary + files.copy("s3://reports/2026/q1.csv", "local:///backup/") # 放进该目录 + files.move("local:///inbox/a.csv", "local:///done/a.csv") + files.mkdir("local:///backup/2026") + files.delete("local:///backup/2026", recursive=True) + +预览有两道上限。最多返回 ``preview_bytes``\ （64 KiB），而大于 ``fetch_limit``\ +（16 MiB）的远程文件不会被获取：不支持范围读取的后端必须先把整个文件下载下来，才读得 +到其中任何一部分。两个上限都是 ``FileService`` 的参数。二进制内容会以十六进制转储 +返回，并设置 ``binary``。 + +目录会连同其下的一切一起复制，而且不能移动。每个 URI 都经过存储层，所以 ``..`` 与 +URI 中的凭据都会被拒绝。 + +Storage +------- + +.. code-block:: python + + storage = services.storage + for backend in storage.backends(): + print(backend.name, backend.kind, backend.usable, backend.detail) + storage.mount_local("sandbox://jobs", "/srv/jobs") # 限制在那个目录内 + storage.mounts() + storage.capabilities("sandbox://jobs") + storage.unmount("sandbox://jobs") + +``backends()`` 为每个 scheme、每个挂载点，以及每个没有 scheme 的共享 client 各返回 +一个 ``BackendStatus``。本地与内存后端、挂载点，以及 client 已初始化的云端后端， +``usable`` 为真。否则 ``detail`` 会说明如何初始化；若该 extra 的包没有安装，则带有 +``pip install`` 命令（也在 ``install_hint`` 中）。这里不会打开任何连接。 + +Pipelines +--------- + +草稿 +~~~~ + +流水线编辑器无法编辑 ``Pipeline``：那个对象拒绝任何无效的内容，而构建中的定义大部分 +时间都是无效的。``PipelineDraft`` 保存到目前为止输入的一切。 + +.. code-block:: python + + from automation_file.app import PipelineDraft + + draft = PipelineDraft("daily-report") + draft.set_params({"date": "2026-10-08"}) + draft.add_task("FA_storage_copy", "download", + arguments={"source": "s3://in/${params.date}.csv", + "target": "local:///tmp/report.csv"}) + draft.add_task("FA_storage_delete", "tidy", arguments={"uri": "local:///tmp/report.csv"}) + draft.connect("download", "tidy") # tidy 依赖 download + draft.set_retry("download", max_attempts=5, backoff=2.0, on=["ConnectionError"]) + draft.set_timeout("download", 300) + draft.set_condition("tidy", "always") + draft.rename_task("tidy", "clean-up") # 边与占位符会跟着改 + + draft.problems() # [] 或 Problem(path, message, task) + draft.to_definition() # Pipeline.from_dict 所接受的文档 + +.. list-table:: + :header-rows: 1 + :widths: 36 64 + + * - 方法 + - 效果 + * - ``add_task``、``remove_task``、``rename_task`` + - 改变任务集合。没有指定 ID 时，会由动作名称推导。 + * - ``set_action``、``set_arguments`` + - 动作，以及它的关键字映射、位置列表或 ``None``。 + * - ``set_retry``、``set_timeout``、``set_condition``、``set_idempotency_key`` + - 任务如何运行。 + * - ``connect``、``disconnect``、``set_dependencies``、``edges`` + - 依赖边。指向自己的边，或会形成环的边，会以 ``AppException`` 拒绝。 + * - ``set_position``、``positions``、``auto_layout``、``layout``、``apply_layout`` + - 每个任务在画布上的位置。这是编辑器的元数据：绝不会出现在 + ``to_definition()`` 中。 + * - ``set_name``、``set_description``、``set_max_workers``、``set_params``、``set_schedule`` + - 定义的头部。 + * - ``add_listener``、``batch`` + - 每次变更后都会调用 ``listener(change)``，其值为 ``"structure"``、 + ``"task"``、``"header"`` 或 ``"position"``；在 ``with draft.batch():`` + 之内，每一种只在结束时报告一次。 + * - ``problems``、``to_definition``、``from_definition`` + - 附上每个问题路径的验证，以及定义文档。 + +运行时会拒绝的值（负的超时、未知的动作名称）会被保留下来，并由 ``problems()`` 报告； +只有会让草稿不一致的编辑（重复的 ID、环）才会立刻抛出异常。草稿不是线程安全的：请只 +从一个线程编辑它，交给 worker 的则是 ``draft.to_definition()``。 + +检查与运行 +~~~~~~~~~~~~ + +每个方法都接受草稿或定义映射，并返回普通的字典。一次运行就是 +``PipelineRun.to_dict()``，其中的机密信息已屏蔽，另外加上 ``active`` 标志，说明这个 +进程是否仍在执行它。 + +.. code-block:: python + + pipelines = services.pipelines + + pipelines.action_names() # 给动作列表用 + pipelines.describe_action("FA_storage_copy") # 参数、默认值、摘要 + + pipelines.validate(draft) # 也包括：未知的动作名称 + plan = pipelines.dry_run(draft, {"date": "2026-10-09"}) + outcome = pipelines.test_task(draft, "download", {"date": "2026-10-09"}) + + run = pipelines.start(draft, {"date": "2026-10-09"}) # 后台运行 + followed = pipelines.follow(run["run_id"]) # {"run": ..., "events": [...]} + pipelines.cancel(run["run_id"]) + pipelines.wait(run["run_id"], timeout=30) + + pipelines.resume(run["run_id"], draft) # 保留已成功的，运行其余的 + pipelines.retry(run["run_id"], draft) # 新的一次运行，参数相同 + pipelines.history("daily-report", limit=10) + pipelines.running() + +``start`` 与 ``resume`` 立即返回；定义有问题时，会在任何东西运行之前抛出异常。 +``test_task`` 单独执行一个任务：它的动作会真的执行，它的上游任务则换成替身，返回 +``results=`` 中给的值（默认为 ``None``），而且这次测试不会记录到任何 run store，也 +不会发布到共享的总线。 + +定义文件与布局 +~~~~~~~~~~~~~~ + +.. code-block:: python + + draft = pipelines.load("pipelines/daily-report.yaml") + draft.load_notes # 文件有什么问题（如果有的话） + pipelines.save(draft, "pipelines/daily-report.yaml") + +``save`` 把定义写成 ``.yaml``、``.yml`` 或 ``.json``，并把画布布局写进旁边的 +``.layout.json``。定义会拒绝未知的键，而布局不属于会被运行的内容，所以两者绝 +不共用一个文件。``load`` 会打开能解析但无效的文件，让它可以被修好。 + +Scheduler +--------- + +.. code-block:: python + + scheduler = services.scheduler + scheduler.add("nightly", "0 2 * * *", + [["FA_pipeline_run", {"definition": "pipelines/daily-report.yaml"}]]) + scheduler.jobs() + scheduler.remove("nightly") + scheduler.remove_all() + +``add`` 也接受 JSON 文本形式的动作列表，也就是表单提供的形式。这个服务只调用 +``schedule_add``、``schedule_remove``、``schedule_remove_all`` 与 +``schedule_list``\ （:doc:`events`）。 + +Integrity +--------- + +.. code-block:: python + + integrity = services.integrity + integrity.baseline("s3://reports/2026", "local:///var/lib/fa/reports.json") + report = integrity.verify("s3://reports/2026", "local:///var/lib/fa/reports.json") + report["ok"], report["counts"], report["changes"] + integrity.accept("s3://reports/2026", "local:///var/lib/fa/reports.json") + + integrity.start_monitor("reports", "s3://reports/2026", + "local:///var/lib/fa/reports.json", interval=300) + integrity.drift() # 每个监控器一个 MonitorDrift，给仪表板用 + integrity.stop_monitor("reports") + integrity.stop_started() # 这个服务启动的监控器 + +在这里启动的监控器，就是 ``FA_integrity_*`` 动作看到的那个具名监控器。 + +Audit +----- + +.. code-block:: python + + audit = services.audit + audit.status() # 取得 store 之前为 {"configured": False, ...} + audit.configure("/var/lib/automation_file/audit.sqlite") + audit.search(status="error", resource_prefix="s3://reports/", limit=20) + audit.count(actor="scheduler") + audit.recent(limit=30) # 尚未配置审计时为 [] + +以 ``None`` 或空字符串给定的筛选条件不会限制搜索，所以表单的值可以原样传入。 + +Notifications +------------- + +.. code-block:: python + + notifications = services.notifications + notifications.sinks() # 名称、类型、送达位置；绝不含机密信息 + notifications.add_route({"name": "failures", "sinks": "team-alerts, ops-mail", + "types": "pipeline.failed, task.failed", + "min_severity": "error", "dedup_seconds": "600"}) + notifications.routes() + notifications.remove_route("failures") + notifications.send_test("team-alerts") # {"team-alerts": "sent"} + +列表可以是以逗号分隔的文本，数字也可以是文本。路由若指名未注册的 sink 会被拒绝。 +``send_test`` 为每个 sink 返回一个结果：``"sent"`` 或错误，其中的 URL 只留下主机。 + +Settings +-------- + +.. code-block:: python + + settings = services.settings + settings.load("automation_file.toml") # 一份摘要；不改变任何东西 + settings.apply("automation_file.toml") # 注册 sink 与路由 + for extra in settings.extras(): + print(extra.name, extra.installed, extra.install_hint) + settings.environment() # 版本、平台、日志文件 + +摘要包含文件的区段、sink 与路由，以及所有机密信息都已屏蔽的文档。 + +表单辅助函数 +------------------------ + +.. list-table:: + :header-rows: 1 + :widths: 32 68 + + * - 函数 + - 用途 + * - ``parse_argument_text(text)`` + - 把一个表单字段变成值：文本是 JSON 就当 JSON，否则就是文本本身。 + * - ``format_argument_value(value)`` + - 反方向：生成读回来会是同一个值的文本。 + * - ``parse_json_text(text, what)`` + - 从文本字段取得 JSON 文档，否则抛出 ``AppException``，指出 ``what`` 与错误的 + 位置。 + * - ``split_names(text)`` + - 把 ``"a, b"`` 变成 ``["a", "b"]``。 + * - ``describe_action(name, command)`` + - 动作的参数、默认值与摘要。 + +机密信息 +---------------- + +``mask_secrets(value)`` 返回可以安全显示的副本，每个服务都会把它应用在返回的内容上： + +* 存放在表明自己是机密信息的名称（``password``、``token``、``api_key``、 + ``authorization``……）之下的值，会变成 ``********``； +* 存放在 ``url`` 或 ``..._url`` 之下的值，只保留 scheme 与主机； +* 在其他任何文本中，URL 的用户信息与 ``Bearer`` 后面的 token 会被移除。 + +存储 URI 不会被改动：它不可能带有凭据。任务的结果会原样返回，所以不要把机密信息放进 +结果里。 + +共享与私有的服务 +-------------------------------- + +``app_services()`` 为每个进程返回一组服务，建立在进程内的单例之上（默认的 resolver、 +默认的 run store、事件总线、审计轨迹、通知管理器与路由器）。因此同一个进程中的窗口 +与 Web UI 会显示相同的运行、监控器与路由。 + +``build_services`` 则构建自己的一组： + +.. code-block:: python + + from automation_file.app import ServiceOptions, build_services + from automation_file.events import EventBus + from automation_file.pipeline import SQLiteRunStore + + services = build_services(ServiceOptions( + run_store=SQLiteRunStore("/var/lib/automation/pipelines.db"), + bus=EventBus(), + )) + +``ServiceOptions`` 接受 ``resolver``、``run_store``、``registry``、``bus``、 +``audit_trail``、``notification_manager`` 与 ``notification_router``。不论选项如何， +调度器与完整性监控器都是整个进程共用的。 + +编写另一种用户界面 +------------------------------------ + +1. 取得服务：``app_services()``，或 ``build_services(...)``。 +2. 由 ``NAVIGATION`` 构建导航。 +3. 每个视图调用一个服务方法，并渲染它返回的内容。捕获 + ``FileAutomationException`` 并显示它的文本。 +4. 会碰到存储或网络的调用（``files.*``、``integrity.verify``、 + ``notifications.send_test``、``settings.apply``）请从 worker 线程进行。 + ``pipelines.start`` 与 ``pipelines.resume`` 本来就立即返回。 +5. 流水线编辑器请保留一个 ``PipelineDraft``，只通过它的方法修改它，并在 listener 中 + 按它重绘。 + +一个虽小但完整的终端界面： + +.. code-block:: python + + from automation_file.app import NAVIGATION, app_services + from automation_file.exceptions import FileAutomationException + + services = app_services() + views = { + "Dashboard": lambda: services.dashboard.summary().to_dict(), + "Storage": lambda: [backend.to_dict() for backend in services.storage.backends()], + "Pipelines": lambda: services.pipelines.history(limit=10), + "Scheduler": services.scheduler.jobs, + "Integrity": lambda: [drift.to_dict() for drift in services.integrity.drift()], + "Audit": services.audit.recent, + "Notifications": services.notifications.routes, + "Settings": lambda: [extra.to_dict() for extra in services.settings.extras()], + } + for number, name in enumerate(NAVIGATION, start=1): + print(number, name) + chosen = NAVIGATION[int(input("> ")) - 1] + try: + print(views.get(chosen, lambda: "use services.files for Files")()) + except FileAutomationException as error: + print("failed:", error) + +出问题时 +---------------- + +``AppException`` + 这一层拒绝了请求：草稿无法接受的编辑、不是 JSON 的表单文本、缺少名称。消息会说明 + 该改什么。 + +``dry_run``、``start`` 或 ``resume`` 抛出 ``PipelineDefinitionException`` + 定义无效。``error.problems`` 保存每一项发现；``validate`` 会以 ``Problem`` 对象 + 返回相同的内容而不抛出异常。 + +``status`` 抛出“unknown run” + 这次运行既不在这个服务的跟踪之中，也不在它的 run store 里。除非默认 store 是 + ``SQLiteRunStore``，或曾把它传给 ``build_services``，否则运行记录只保存在 + 内存中。 + +``cancel`` 返回 ``False`` + 这次运行不在这个进程中进行：它已经结束，或是由另一个进程启动的。 + +后端不是 ``usable`` + 请读 ``detail``。初始化 client，或安装 ``install_hint`` 指名的 extra。 + +``audit.search`` 抛出“audit is not configured” + 请先调用 ``audit.configure(path)``。``audit.recent()`` 则会返回空列表。 + +机密信息出现在视图中 + 它存放在没有表明自己是机密信息的名称之下，或者它是任务结果的一部分。请改掉字段 + 名称，或不要把它放进结果。 + +两个界面显示的内容不一致 + 它们用的是不同的服务组。``app_services()`` 是共享的；``build_services()`` + 不是。 diff --git a/docs/source/Zh-CN/usage/gui.rst b/docs/source/Zh-CN/usage/gui.rst index ef8f650..b28509b 100644 --- a/docs/source/Zh-CN/usage/gui.rst +++ b/docs/source/Zh-CN/usage/gui.rst @@ -1,10 +1,16 @@ GUI（PySide6） ============== -分页式控制面板封装了所有功能： +桌面窗口按“你想完成什么事”来组织，而不是按后端：侧边栏有九个工作流页面，另有一个 +**Advanced** 条目，保留那些直接操作单个动作或单个后端的工具。 + +每个页面都只是应用层（:doc:`app_layer`）某一个服务的薄薄一层视图。窗口本身不带任何 +逻辑，所以它显示的内容，就是 Web UI（:doc:`servers`）、CLI 与你自己的 Python 代码 +看到的内容。 .. code-block:: bash + pip install "automation_file[gui]" # PySide6 是 extra，不是基础依赖 python -m automation_file ui # 或在仓库根目录开发时： python main_ui.py @@ -15,10 +21,434 @@ GUI（PySide6） launch_ui() -标签页：Home、Local、Transfer、Progress、JSON actions、Triggers、 -Scheduler、Servers。所有标签页下方共用一个常驻日志面板， -逐条输出每次调用的结果或错误。后台任务通过 ``ActionWorker`` 在 -``QThreadPool`` 上执行，UI 始终保持响应。 +没有安装 PySide6 时，``launch_ui`` 会抛出 ``OptionalDependencyException``，消息中 +带有上面那行 ``pip install`` 命令。 + +导航 +-------- + +.. list-table:: + :header-rows: 1 + :widths: 18 12 70 + + * - 条目 + - 快捷键 + - 用途 + * - Dashboard + - ``Ctrl+1`` + - 一眼看完健康状态、运行中与最近的流水线运行、完整性漂移、最近的事件与存储状态。 + * - Files + - ``Ctrl+2`` + - 浏览存储 URI、预览文件、复制、移动、删除、创建目录。 + * - Storage + - ``Ctrl+3`` + - 有哪些后端、每个后端能不能用，以及挂载点。 + * - Pipelines + - ``Ctrl+4`` + - 可视化流水线编辑器：构建、验证、试运行、测试、运行、续跑。 + * - Scheduler + - ``Ctrl+5`` + - Cron 作业：列出、添加、移除。 + * - Integrity + - ``Ctrl+6`` + - 为目录树建立基线、验证与接受；启动与停止监控器。 + * - Audit + - ``Ctrl+7`` + - 把审计轨迹指向数据库，搜索并统计其中的记录。 + * - Notifications + - ``Ctrl+8`` + - 已注册的 sink、路由，以及测试消息。 + * - Settings + - ``Ctrl+9`` + - 配置文件、可选的 extra、运行环境。 + * - Advanced + - ``Ctrl+0`` + - Local、Transfer、Progress、JSON actions、Triggers 与 Servers：旧版窗口的 + 页签。见 `旧页签去了哪里`_。 + +窗口 +-------- + +侧边栏在左，选中的页面在右。两者下方是所有页面共用的 **活动日志**：每个动作开始时写 +一行、得到结果时再写一行，并以页面名称开头。最新的一行也会在状态栏显示几秒钟。 + +每个页面的底部都有一行 **状态文字**，显示你在这个页面上最后做的那件事的结果：成功是 +绿色，失败是红色。 + +没有任何操作会卡住窗口。页面把每一次调用交给线程池（``QThreadPool`` 上的 +``ActionWorker``），结果回来时才显示。页面在你打开它时读取数据，按下它的 **Refresh** +按钮时再读一次。有两个视图会自己刷新：Dashboard 在可见时每五秒刷新一次；被跟踪的 +流水线运行每秒刷新两次，直到它结束。 + +窗口与库的其余部分共用进程内的单例。从 Python 注册的 sink、通过 HTTP 动作服务器启动 +的流水线，或由 JSON 动作启动的监控器，都会在下一次刷新后出现。 + +Dashboard +--------- + +**All clear** 或 **Needs attention**，旁边列出原因。只要最新的五十次运行中有一次 +失败、某个监控器发现漂移或无法验证，或事件总线上有严重程度为 ``error`` 及以上的近期 +事件，仪表板就会要求关注。 + +* **Health**：已注册的动作、运行中的运行、调度作业、完整性监控器、通知 sink 与路由、 + 路由器是否启用，以及审计轨迹是否正在记录。 +* **Pipeline runs**：最新几次运行的结局、仍在进行的运行，以及最近的结果与错误。 +* **Integrity drift**：每个具名监控器、它的目标，以及它上次发现了什么。 +* **Storage status**：每个后端以及能不能用。 +* **Recent events**：总线上最新的二十个事件，新的在前。 + +**Refresh every 5 s** 开关定时器；**Refresh** 立即读取。仪表板只读取：不启动任何 +东西，也不改变任何东西。 + +Files +----- + +输入存储 URI 或本地路径（``s3://reports/2026``、``local:///data``、``C:\data``、 +``memory://demo``）后按 **Open**。**Up** 回到上一级，**Browse local…** 选择目录。 +在目录上双击会进入该目录，在文件上双击会预览它。 + +预览是有上限的：最多显示 64 KiB，而远程后端上超过 16 MiB 的文件根本不会被获取（状态 +文字会说明）。二进制内容以前 256 个字节的十六进制转储显示。 + +各项操作都作用在选中的条目上： + +* **Copy** / **Move** 到目标 URI，可以在同一个后端或另一个后端。目标若是已有的目录， + 文件会以原来的名称放进去。目录会连同其下的一切一起复制；目录不能移动（请先复制、 + 检查副本，再删除原来的）。 +* **Create directory** 在当前位置下创建目录。 +* **Delete selected** 会先询问。有内容的目录需要勾选 **Delete a directory with its + contents**。 + +每个路径都经过存储层（:doc:`storage`）：``..`` 会被拒绝，URI 中的凭据会被拒绝，挂载 +的目录无法被跳出。 + +Storage +------- + +每个后端一行： + +.. list-table:: + :header-rows: 1 + :widths: 14 86 + + * - Kind + - 含义 + * - ``scheme`` + - 有工厂函数的 URI scheme：``local``、``memory``、``s3``、``azure``、 + ``gdrive``、``dropbox``、``onedrive``、``sftp``、``ftp``、``ftps``。 + * - ``mount`` + - 挂载在某个 URI 上的后端。它的名称就是那个 URI。 + * - ``client`` + - 有 ``FA_*`` 动作但没有存储 scheme 的共享 client（Box）。 + +本地与内存后端、挂载点，以及 client 已初始化的云端后端，**Usable** 为 ``yes``。 +**Detail** 说明缺了什么：如何初始化 client；若该 extra 的包没有安装，则是 +``pip install`` 命令。 + +**Mount a local directory** 以你选的 URI 提供某个目录，并把它限制在那个目录内 +（把 ``sandbox://jobs`` 挂在 ``/srv/jobs``）。来自进程外部的路径请用这种方式处理。 +**Unmount selected** 移除挂载点。选中挂载点、``local`` 或 ``memory`` 时，会显示该 +后端提供哪些能力（目录、修改时间、ETag……）。 + +Pipelines +--------- + +这个页面是流水线定义（:doc:`pipeline`）的编辑器，也是它的运行控制台。 + +.. list-table:: + :header-rows: 1 + :widths: 24 76 + + * - 区域 + - 内容 + * - 最上面一行 + - **New**、**Open…**、**Save**、**Save as…**、**Auto layout**，以及文件名 + （有未保存的更改时会加上 ``(modified)``）。 + * - 第二行 + - 流水线名称、**Max workers**、描述、默认参数（JSON）与调度（cron 表达式，留给 + 调度器使用）。 + * - Actions（左） + - 所有已注册的动作，附筛选框。 + * - 画布（中） + - 每个任务一个节点、每个依赖一条箭头，上方有 **Connect**、**Disconnect** 与 + **Remove selected**。 + * - Selected task（右） + - 画布上选中的任务的表单。 + * - 运行栏 + - 运行参数（JSON）以及 **Validate**、**Dry run**、**Test task**、**Run**、 + **Resume**、**Retry**、**Cancel**、**Refresh history**、 + **Follow selected run**。 + * - 底部的页签 + - **Problems**、**Tasks**、**Log**、**History**。 + +流水线编辑器分步说明 +---------------------------------------- + +1. **开始。** 按 **New** 创建空白定义，或按 **Open…** 打开 ``.yaml`` / ``.yml`` / + ``.json`` 文件。不是有效定义的文件仍然打得开：形状正确的部分会显示出来， + **Problems** 页签则列出它哪里有问题。 + +2. **命名。** 填入名称、worker 数量，需要的话再填描述、JSON 对象形式的默认参数与 + 调度。 + +3. **添加任务。** 把 **Actions** 列表中的动作拖到画布上：任务会出现在你松开的位置。 + 在动作上双击，或选中它再按 **Add task**，会把它加在最下面的任务之下。筛选框可以 + 缩小列表（输入 ``storage`` 会显示 ``FA_storage_*`` 动作）。任务的 ID 由动作名称 + 推导而来；可以在表单中修改。 + +4. **排列。** 把节点拖到你想要的位置。**Auto layout** 按依赖深度把每个任务放进对应的 + 列。位置是编辑器的元数据：它存在定义旁边，绝不会存进定义里。 + +5. **连接。** 先点上游任务，再按住 ``Ctrl`` 点依赖它的任务，然后按 **Connect**。 + 按钮会显示它将采用的方向（``Connect download -> publish``）：你选中这两个任务的 + 顺序，就是箭头的方向。会形成依赖环的箭头会被拒绝。你也可以在任务表单的 + **Depends on** 中勾选上游任务。 + + 要移除依赖，请点它的箭头（或选中它两端的任务），再按 **Disconnect**。 + **Remove selected** 会移除选中的箭头；没有选中箭头时，则移除选中的任务与它们的 + 箭头。 + +6. **编辑选中的任务。** 表单显示最后选中的任务： + + .. list-table:: + :header-rows: 1 + :widths: 24 76 + + * - 字段 + - 含义 + * - Task ID + - 在流水线中唯一。改名会保留箭头，并改写指向该任务的 + ``${tasks..result}`` 占位符。 + * - Action + - 已注册的动作。它的签名与摘要显示在下方。 + * - Arguments + - 动作的每个参数一行，并附上默认值。值若能解析成 JSON 就是 JSON + （``12``、``true``、``["a", "b"]``、``"12"``），否则就是文本，所以 URI 或 + ``${params.date}`` 不需要加引号。留空的行不会被传入，因此使用默认值。 + **Add argument** 为接受 ``**kwargs`` 的动作添加一行。**Edit as JSON** + 把调用参数显示成一份 JSON 文档；位置参数（JSON 数组）只能用这种方式编辑。 + * - Depends on + - 勾选必须先结束的任务。 + * - Attempts、Back-off、Back-off cap、Retry on + - 总尝试次数、第一次退避时间与其上限，以及值得再试一次的异常名称（留空： + 瞬时性的那几种）。 + * - Timeout + - 整个任务的秒数；留空表示不限。 + * - Run when + - ``on_success``、``on_failure`` 或 ``always``。 + * - Idempotency key + - 带有 ``${params.}`` 占位符的文本；留空表示没有。 + + 在你按下 **Apply changes** 之前，什么都不会改变。**Revert** 会丢掉你输入的内容， + 选中另一个任务也一样。草稿拒绝某个值时，表单会说明原因，并保留你输入的内容。 + +7. **验证。** **Problems** 页签列出每一项发现，并附上所指条目的路径 + （``tasks.publish.depends_on[0]: unknown task 'x'``），包括注册表不认识的动作 + 名称。点某个问题会选中它的任务。**Dry run**、**Test task**、**Run**、**Resume** + 与 **Retry** 都会先验证，有问题就停下来。 + +8. **试运行。** 以 JSON 对象输入运行参数，然后按 **Dry run**。不会执行也不会记录任何 + 东西。**Tasks** 页签把每个任务显示为 ``planned`` 并附上层级；无法按计划运行的任务 + 会说明原因（例如这次运行没有提供某个参数）。 + +9. **测试单个任务。** 选中一个任务再按 **Test task**。它的动作会真的执行，并应用它的 + 重试策略与超时。它的上游任务则不会执行：每一个都换成不返回任何东西的替身，所以 + ``${tasks..result}`` 占位符会得到 ``null``。测试不会记录到任何 run store， + 也不会发布到共享的事件总线。 + +10. **运行。** **Run** 在后台启动流水线并跟踪它：**Tasks** 页签显示每个任务的状态、 + 尝试次数、耗时与错误，**Log** 页签显示这次运行的事件，每个节点则换成它的状态 + 颜色（灰色 pending、蓝色 running、绿色 succeeded、红色 failed 或超时、橙色 + cancelled、黄色 skipped）。**Cancel** 要求这次运行停下来。 + +11. **续跑或重试。** 失败之后，先排除原因。**Resume** 接续同一次运行：已成功的任务 + 会保留，其余的重新运行，运行 ID 与参数都不变。**Retry** 以被跟踪的那次运行的 + 参数启动一次新的运行。 + +12. **回顾。** **History** 页签列出已记录的运行，新的在前。在某一条上双击，或选中 + 它再按 **Follow selected run**，就能再看到它的任务与事件；之后 **Resume** 与 + **Retry** 会作用在它身上。 + +13. **保存。** **Save as…** 把定义写成 ``.yaml``、``.yml`` 或 ``.json``，并把画布 + 布局写进旁边的 ``.layout.json``。定义文件正是 ``Pipeline.from_file`` 与 + ``FA_pipeline_run`` 所接受的文件。 + +Scheduler +--------- + +**Schedule a job** 需要唯一的名称、五个字段的 cron 表达式，以及 JSON 形式的动作 +列表，按 **Add job** 注册。勾选复选框可以让新的一次运行在前一次还没结束时启动；否则 +那次触发会被跳过，并计入 **Skipped**。表格显示每个作业的运行次数与上次运行时间； +**Remove selected** 与 **Remove all** 移除作业。要让流水线按调度运行，请调度 +``FA_pipeline_run`` 并给它定义文件的路径。 + +窗口关闭时，这些作业会被移除。 + +Integrity +--------- + +输入 **Target**（要检查的目录树）与 **Baseline**（已核准状态存放的位置），两者都是 +存储 URI 或本地路径。 + +* **Create baseline** 核准当前的内容。 +* **Verify** 把目录树与基线比较，并列出每一项变更的种类与路径。关闭 **Deep** 时，只有 + 大小或时间改变的文件会被哈希，摘要会注明这是快速检查。 +* **Accept current state** 会先询问，然后把当前的目录树存成新的基线。 + +在 **Monitors** 下，给一个名称与间隔，按 **Start monitor** 就会在线程上持续验证 +目标；表格显示每个监控器上次发现了什么。从窗口启动的监控器会在窗口关闭时停止。见 +:doc:`integrity`。 + +Audit +----- + +审计轨迹在取得 store 之前什么都不记录。输入 SQLite 数据库的路径（不存在时会创建）， +按 **Configure**；之后 **State** 会显示记录写到哪里。 + +填入任何筛选条件，按 **Search**（新的在前，最多 **Limit** 条）或 **Count**。留空的 +筛选条件不会限制搜索。**Since** 与 **Until** 接受带时区偏移的 ISO 8601 时间。选中 +一条记录可以看到它的全部内容（JSON）。见 :doc:`audit`。 + +Notifications +------------- + +**Sinks** 按名称、类型与送达位置列出已注册的 sink。Sink 是在代码中或从配置文件 +（Settings）注册的；这个页面不会创建它们。选一个 sink（或 **All sinks**），需要的话 +填入主题，按 **Send test message**；状态文字会显示每个 sink 的结果。 + +**Routes** 列出哪些事件送到哪些 sink。**Add or replace a route** 需要名称、sink +名称、事件类型（``pipeline.*``、``task.failed``）、来源、最低严重程度与节流数值；列表 +以逗号分隔。路由若指名未注册的 sink 会被拒绝。路由器随第一条路由启动，最后一条路由 +被移除时停止。见 :doc:`notifications`。 + +Settings +-------- + +输入 ``automation_file.toml`` 的路径（:doc:`config`）。 + +* **Preview** 读取文件并显示摘要。不会改变任何东西。 +* **Apply** 注册它的通知 sink 与路由。 + +**Optional extras** 列出每个 extra、它启用的功能、是否已安装，以及未安装时的 +``pip install`` 命令。**Environment** 显示包与 Python 版本、平台、日志文件，以及 +最后应用的配置文件。 + +旧页签去了哪里 +---------------------------- + +什么都没有移除。**Advanced** 条目保留了旧的页签，原封不动： + +.. list-table:: + :header-rows: 1 + :widths: 28 72 + + * - 旧页签 + - 现在 + * - Home + - **Dashboard**（后端是否就绪在 *Storage status* 下，也在 **Storage** 页面）。 + * - Local + - **Advanced** → *Local*。浏览与复制也可以在 **Files** 进行。 + * - Transfer（HTTP、Google Drive、S3、Azure Blob、Dropbox、SFTP、OneDrive、 + Box） + - **Advanced** → *Transfer*。云端 client 的凭据就是在这里提供。 + * - Progress + - **Advanced** → *Progress*。 + * - JSON actions + - **Advanced** → *JSON actions*。 + * - Triggers + - **Advanced** → *Triggers*。 + * - Scheduler + - **Scheduler**。 + * - Servers + - **Advanced** → *Servers*。 + +机密信息 +---------------- + +没有任何页面会显示机密信息。Sink 以名称、类型与送达位置描述；配置文件摘要中的密码与 +token 会换成 ``********``，webhook URL 只留下主机；事件、审计记录与运行参数在显示前 +也以同样方式屏蔽；URL 中的凭据与 bearer token 会从页面显示的消息中移除。 + +值是否被屏蔽，取决于存放它的 *名称* 是否表明它是机密信息（``password``、``token``、 +``api_key``、``authorization``……）。任务的结果则按任务返回的样子显示。所以请给机密 +参数取这样的名称，并且不要把机密信息放进结果里。 + +在没有显示器的环境运行 +------------------------------------------ + +Qt 需要显示器才能打开窗口。在没有显示器的服务器上： + +* 只读视图请用 Web UI（:doc:`servers`）；它由同一个应用层渲染。 +* 从 Python 直接使用应用层（:doc:`app_layer`），或通过 CLI 与动作服务器使用 + ``FA_*`` 动作。窗口做的每一件事，都是对其中之一的调用。 +* 要构造窗口但不显示它（测试、CI 中的截图），请在导入 Qt 之前设置 + ``QT_QPA_PLATFORM=offscreen``： + + .. code-block:: python + + import os + os.environ.setdefault("QT_QPA_PLATFORM", "offscreen") + + from PySide6.QtWidgets import QApplication + from automation_file.app import build_services + from automation_file.ui.main_window import MainWindow + + app = QApplication([]) + window = MainWindow(build_services()) # 或以 MainWindow() 使用共享的服务 + window.navigate("Pipelines") + window.close() + +``import automation_file`` 与 ``import automation_file.app`` 绝不会导入 PySide6； +只有 ``automation_file.ui`` 会。 + +出问题时 +---------------- + +按钮好像没有反应 + 请看页面底部的状态文字与活动日志。每一次拒绝与每一次失败都会在那里报告，并附上 + 原因。 + +``launch_ui`` 抛出 ``OptionalDependencyException`` + 没有安装 PySide6：``pip install "automation_file[gui]"``。 + +窗口打不开：“could not load the Qt platform plugin” + 没有显示器。见 `在没有显示器的环境运行`_。 + +云端后端显示“not initialised” + 它的共享 client 还没有凭据。打开 **Advanced** → *Transfer*，选择该后端并填写 + *Credentials*，或运行它的 ``FA_*_later_init`` 动作。 + +Detail 说某个包“is not installed” + 安装该行指名的 extra；**Settings** 列出每个 extra 与它的命令。 + +**Connect** 被拒绝 + 这条箭头会形成依赖环，或是选中的任务不是恰好两个。 + +**Run** 停在“problem(s): see the Problems tab” + 定义无效。每个问题都以所指条目的路径开头；点它就会选中该任务。 + +按了 **Cancel** 之后运行仍是 ``running`` + 运行中的动作无法被中断。尚未开始的任务会立刻被取消；运行中的任务返回后，这次运行 + 才会结束。 + +**Resume** 被拒绝 + 这次运行仍在进行、属于另一个名称的流水线，或是定义现在需要已存储的运行所没有的 + 参数。 + +重启后历史是空的 + 运行记录默认保存在内存中。在 ``launch_ui()`` 之前调用 + ``set_default_run_store(SQLiteRunStore(path))``，就能把它们保存在文件里。 + +**Search** 说 audit is not configured + 请先在同一个页面为审计轨迹指定数据库。 + +测试消息失败 + 状态文字会显示每个 sink 的错误，URL 只留下主机。Sink 本身的说明见 + :doc:`notifications`。 + +页面显示的是旧数据 + 按 **Refresh**。只有 Dashboard 与被跟踪的运行会自己刷新。 -GUI 与库其余部分共用同一组单例——从 Python 注册的 sink、 -自定义动作、触发器都会立即在运行中的窗口生效。 +窗口关闭后还有东西在运行 + 关闭窗口会移除调度作业，并停止它启动的监控器、动作服务器与触发器，但不会动流水线 + 运行：已启动的运行会跑到结束，进程在那之前不会退出。 diff --git a/docs/source/Zh-CN/usage/pipeline.rst b/docs/source/Zh-CN/usage/pipeline.rst index f6d5379..a2216fc 100644 --- a/docs/source/Zh-CN/usage/pipeline.rst +++ b/docs/source/Zh-CN/usage/pipeline.rst @@ -12,6 +12,10 @@ :func:`~automation_file.execute_action_dag`\ （见 :doc:`dag`）保持不变。它把一份动作 列表执行一次并返回结果；当运行需要被记录、重试、续跑或观察时，请改用流水线。 +定义也可以不必手写就构建、检查与运行：桌面窗口的 Pipelines 页面是它的可视化编辑器 +（:doc:`gui`），而 ``automation_file.app`` 则把同样的操作提供给任何其他界面 +（:doc:`app_layer`）。 + 最小示例 ---------------- diff --git a/docs/source/Zh-CN/usage/servers.rst b/docs/source/Zh-CN/usage/servers.rst index 85006c5..f7c72d1 100644 --- a/docs/source/Zh-CN/usage/servers.rst +++ b/docs/source/Zh-CN/usage/servers.rst @@ -44,3 +44,73 @@ HTTP 响应均为 JSON。鉴权失败返回 ``401``；非法 JSON 返回 ``400`` 共享密钥比较使用 :func:`hmac.compare_digest`（常数时间）。 切勿记录密钥或原始负载。 + +Web UI +------ + +浏览器中的只读仪表板，以标准库与 HTMX 提供（从固定的 CDN URL 加载一个脚本，并附 SRI +哈希）。 + +.. code-block:: python + + from automation_file import start_web_ui + + server = start_web_ui(host="127.0.0.1", port=9955, shared_secret="optional-secret") + # 浏览 http://127.0.0.1:9955/ + # 稍后： + server.shutdown() + server.server_close() + +页面为每个区段轮询一个 HTML 片段。除了传输进度以外，每个片段都由应用层 +（:doc:`app_layer`）渲染，也就是桌面窗口（:doc:`gui`）所调用的同一组服务，所以两者 +显示相同的状态。 + +.. list-table:: + :header-rows: 1 + :widths: 22 14 64 + + * - 片段 + - 轮询间隔 + - 显示内容 + * - ``GET /ui/health`` + - 5 秒 + - ``ok`` 或 ``attention`` 与其原因；已注册的动作、运行中的运行、调度作业、 + 监控器、sink 与路由、审计轨迹。 + * - ``GET /ui/runs`` + - 3 秒 + - 运行中与最近的流水线运行，以及最新几次运行的结局。 + * - ``GET /ui/integrity`` + - 10 秒 + - 每个具名完整性监控器，以及它上次发现的漂移。 + * - ``GET /ui/events`` + - 5 秒 + - 总线上最新的事件，新的在前。 + * - ``GET /ui/storage`` + - 30 秒 + - 每个存储后端以及能不能用。 + * - ``GET /ui/audit`` + - 10 秒 + - 最新的审计记录；要先以 ``configure_audit`` 给审计轨迹一个 store。 + * - ``GET /ui/progress`` + - 2 秒 + - 进度注册表中进行中的传输。 + * - ``GET /ui/registry`` + - 30 秒 + - 每个已注册动作的名称。 + +``GET /`` 与 ``GET /index.html`` 提供页面；其他路径一律返回 ``404``。没有任何路由会 +改变东西：要执行动作，请通过上面的动作服务器，并使用它们自己的鉴权机制。 + +* **默认仅绑定 loopback。** 要绑定其他地址必须传 ``allow_non_loopback=True``；这么做 + 却没有 ``shared_secret`` 时会记录一条警告。 +* **共享密钥。** 设置 ``shared_secret`` 后，每个请求都需要 + ``Authorization: Bearer ``，否则返回 ``401``。页面把这个请求头放在 + ``hx-headers`` 中，让它自己的轮询得到授权；因此能读到页面的人就能读到密钥，请通过 + loopback 或在 TLS 之后提供它。 +* **已转义且已屏蔽。** 片段显示的一切都经过 HTML 转义，而且应用层已经先屏蔽其中的 + token、密码与 webhook URL。 +* **片段绝不会弄坏页面。** 某个服务无法响应时，它的片段会显示 + ``unavailable: ``，其他片段照常工作。 + +``start_web_ui(services=...)`` 接受以 ``automation_file.app.build_services`` 构建的 +一组服务，用来显示进程共用的那一组以外的 run store、事件总线或 resolver。 diff --git a/docs/source/Zh-CN/zh_cn_index.rst b/docs/source/Zh-CN/zh_cn_index.rst index 8be4601..098a3be 100644 --- a/docs/source/Zh-CN/zh_cn_index.rst +++ b/docs/source/Zh-CN/zh_cn_index.rst @@ -150,6 +150,7 @@ PySide6 桌面控制界面——分页布局、日志面板，以及 ``ActionWor :caption: 图形界面 usage/gui + usage/app_layer .. _zh-cn-reliability: diff --git a/docs/source/Zh-TW/architecture.rst b/docs/source/Zh-TW/architecture.rst index d35bdd2..aba5130 100644 --- a/docs/source/Zh-TW/architecture.rst +++ b/docs/source/Zh-TW/architecture.rst @@ -75,7 +75,8 @@ end subgraph UI["ui (PySide6)"] - MainWin["MainWindow
    Home · Local · HTTP · Drive · S3 · Azure · Dropbox
    SFTP · OneDrive · Box · JSON · Triggers · Scheduler
    Progress · Transfer · Servers"] + MainWin["MainWindow
    Dashboard · Files · Storage · Pipelines · Scheduler
    Integrity · Audit · Notifications · Settings
    Advanced: Local · Transfer · Progress · JSON · Triggers · Servers"] + AppLayer["automation_file.app
    one service per navigation entry"] Worker["ActionWorker
    QRunnable on QThreadPool"] end @@ -125,6 +126,7 @@ Plugins ==> Loader MainWin ==> Worker + Worker ==> AppLayer Worker ==> PublicAPI PublicAPI ==> Executor @@ -140,6 +142,8 @@ HTTPS ==> Executor MCP ==> Registry MetSrv ==> Metrics + WebUI ==> AppLayer + AppLayer ==> PublicAPI WebUI ==> Registry ACL ==> TCP ACL ==> HTTPS @@ -235,7 +239,7 @@ class Secrets,Config,ConfW,Crypto,Check,SafeP,ACL sec; class Trigger,Sched event; class TCP,HTTPS,MCP,MetSrv,WebUI server; - class MainWin,Worker ui; + class MainWin,Worker,AppLayer ui; class FileOps,Archives,DataOps,TextOps,Misc localOps; class UrlVal,Http,Drive,S3M,Azure,Dropbox,SFTP,FTP,OneD,Box,WebDAV,SMB,Fsspec,Cross remote; class NM,Sinks notify; @@ -331,14 +335,20 @@ ├── project/ │ ├── project_builder.py │ └── templates.py - ├── ui/ # PySide6 GUI + ├── app/ # 應用層：使用者介面所呼叫的那一層 + │ ├── services.py # AppServices、app_services()、NAVIGATION + │ ├── pipeline_draft.py # PipelineDraft：可編輯的定義 + │ └── *_service.py # 每個導覽項目一個服務 + ├── ui/ # PySide6 GUI，建立在 app/ 之上 │ ├── launcher.py # launch_ui(argv) - │ ├── main_window.py # 分頁式 MainWindow（Home、Local、Transfer、 - │ │ # Progress、JSON actions、Triggers、 - │ │ # Scheduler、Servers） + │ ├── main_window.py # 側邊欄式 MainWindow（Dashboard、Files、Storage、 + │ │ # Pipelines、Scheduler、Integrity、Audit、 + │ │ # Notifications、Settings、Advanced） │ ├── worker.py # ActionWorker（QRunnable） │ ├── log_widget.py # LogPanel - │ └── tabs/ # 每個後端一個分頁 + JSON runner + servers + │ ├── pages/ # 每個導覽項目一個頁面 + 管線編輯器 + │ └── tabs/ # Advanced 底下的工具：每個後端一個分頁、 + │ # JSON runner、triggers、servers └── utils/ ├── file_discovery.py ├── fast_find.py # OS 索引（mdfind/locate/es）+ scandir 後備 diff --git a/docs/source/Zh-TW/usage/app_layer.rst b/docs/source/Zh-TW/usage/app_layer.rst new file mode 100644 index 0000000..9718f04 --- /dev/null +++ b/docs/source/Zh-TW/usage/app_layer.rst @@ -0,0 +1,475 @@ +應用層 +====== + +``automation_file.app`` 是使用者介面所呼叫的那一層。導覽中的每個項目各有一個服務—— +Dashboard、Files、Storage、Pipelines、Scheduler、Integrity、Audit、Notifications、 +Settings——每個都是建立在領域套件（:doc:`storage`、:doc:`pipeline`、:doc:`events`、 +:doc:`integrity`、:doc:`audit`、:doc:`notifications`、:doc:`event_bus`、 +:doc:`config`）之上的普通 Python 物件。 + +PySide6 視窗（:doc:`gui`）與 Web UI（:doc:`servers`）只呼叫這些服務，不碰它們底下的 +任何東西。這就是兩者顯示相同狀態的原因，也是第三種介面——終端機 UI、Web 應用程式、 +聊天機器人——同樣不需要認識領域套件的原因。 + +這一層遵守四個承諾： + +* 不匯入任何 GUI 工具組，也不匯入任何後端 SDK。``import automation_file.app`` 在 + 基礎安裝上就能運作。 +* 回傳 JSON 能容納的 dataclass、字典與清單。 +* 在回傳的內容中遮蔽 token、密碼與 webhook URL。 +* 拋出 ``FileAutomationException`` 的子類別：領域自己的（``StorageException``、 + ``PipelineException``……），以及只有這一層才檢查的情況所用的 ``AppException``。 + +最小範例 +---------------- + +.. code-block:: python + + from automation_file.app import app_services + + services = app_services() # 每個行程一組 + + summary = services.dashboard.summary() + print(summary.status, summary.reasons) # "ok" () 或 "attention" (...) + + for entry in services.files.list_dir("local:///data"): + print(entry.name, entry.size) + + draft = services.pipelines.new_draft("nightly") + draft.add_task("FA_storage_copy", "download", + arguments={"source": "s3://in/a.csv", "target": "local:///tmp/a.csv"}) + run = services.pipelines.start(draft) # 立即回傳 + services.pipelines.wait(run["run_id"], timeout=60) + print(services.pipelines.status(run["run_id"])["status"]) + +服務一覽 +---------------- + +``automation_file.app.NAVIGATION`` 是九個項目依顯示順序排列的 tuple； +``AppServices`` 為每個項目各有一個屬性，名稱為小寫。 + +.. list-table:: + :header-rows: 1 + :widths: 18 24 58 + + * - 項目 + - 屬性與類別 + - 用途 + * - Dashboard + - ``dashboard``、``DashboardService`` + - 一份摘要：健康狀態、執行、完整性漂移、最近的事件、儲存狀態。 + * - Files + - ``files``、``FileService`` + - 對儲存 URI 進行列出、stat、預覽、複製、搬移、刪除、mkdir。 + * - Storage + - ``storage``、``StorageService`` + - Scheme、掛載點，以及每個後端能不能用。 + * - Pipelines + - ``pipelines``、``PipelineService`` + - 草稿、驗證、試跑、背景執行、狀態、歷史、續跑、取消、定義檔。 + * - Scheduler + - ``scheduler``、``SchedulerService`` + - 列出、新增與移除 cron 工作。 + * - Integrity + - ``integrity``、``IntegrityService`` + - 基準、驗證、接受、狀態、啟動與停止監控器。 + * - Audit + - ``audit``、``AuditService`` + - 設定、搜尋與計算稽核紀錄。 + * - Notifications + - ``notifications``、``NotificationService`` + - 已註冊的 sink、路由、測試訊息。 + * - Settings + - ``settings``、``SettingsService`` + - 載入並套用設定檔；哪些 extra 已安裝。 + +Dashboard +--------- + +.. code-block:: python + + summary = services.dashboard.summary() + summary.to_dict() # 可序列化成 JSON + +.. list-table:: + :header-rows: 1 + :widths: 22 78 + + * - 欄位 + - 意義 + * - ``status``、``reasons`` + - ``"ok"``，或是 ``"attention"`` 並為每個原因附上一句話：最近有執行失敗、某個 + 監控器發現漂移或無法驗證、匯流排上有嚴重程度為 ``error`` 以上的近期事件。 + * - ``health`` + - 計數與開關：已註冊的動作、執行中的執行、排程工作、監控器、sink、路由、路由器 + 是否啟用、稽核軌跡的狀態。 + * - ``run_counts`` + - 最新五十次執行的結局：``running``、``succeeded``、``failed``、 + ``cancelled``。 + * - ``running_runs``、``recent_runs`` + - 不含任務細節的執行，新的在前。 + * - ``integrity`` + - 每個具名監控器上次發現了什麼。 + * - ``events`` + - 匯流排上最新的事件，新的在前，已遮蔽。 + * - ``storage`` + - 每個後端，附 ``usable``、``detail``，以及缺少套件時的 ``install_hint``。 + +各部分也可以分開取得：``health()``、``runs()``、``integrity()``、 +``recent_events()``、``storage_status()``。``summary()`` 絕不會因為某個部分讀不到而 +拋出例外；它會把那個部分列在原因裡。 + +Files +----- + +.. code-block:: python + + files = services.files + files.list_dir("s3://reports/2026") # 目錄在前，其次依路徑 + files.stat("s3://reports/2026/q1.csv").size + preview = files.preview("s3://reports/2026/q1.csv", max_bytes=4096) + preview.text, preview.truncated, preview.binary + files.copy("s3://reports/2026/q1.csv", "local:///backup/") # 放進該目錄 + files.move("local:///inbox/a.csv", "local:///done/a.csv") + files.mkdir("local:///backup/2026") + files.delete("local:///backup/2026", recursive=True) + +預覽有兩道上限。最多回傳 ``preview_bytes``\ （64 KiB），而大於 ``fetch_limit``\ +（16 MiB）的遠端檔案不會被抓取：不支援範圍讀取的後端必須先把整個檔案下載下來，才讀得 +到其中任何一部分。兩個上限都是 ``FileService`` 的引數。二進位內容會以十六進位傾印 +回傳，並設定 ``binary``。 + +目錄會連同底下的一切一起複製，而且不能搬移。每個 URI 都經過儲存層，所以 ``..`` 與 +URI 中的憑證都會被拒絕。 + +Storage +------- + +.. code-block:: python + + storage = services.storage + for backend in storage.backends(): + print(backend.name, backend.kind, backend.usable, backend.detail) + storage.mount_local("sandbox://jobs", "/srv/jobs") # 限制在那個目錄內 + storage.mounts() + storage.capabilities("sandbox://jobs") + storage.unmount("sandbox://jobs") + +``backends()`` 為每個 scheme、每個掛載點，以及每個沒有 scheme 的共用 client 各回傳 +一個 ``BackendStatus``。本機與記憶體後端、掛載點，以及 client 已初始化的雲端後端， +``usable`` 為真。否則 ``detail`` 會說明如何初始化；若該 extra 的套件沒有安裝，則帶有 +``pip install`` 指令（也在 ``install_hint`` 中）。這裡不會開啟任何連線。 + +Pipelines +--------- + +草稿 +~~~~ + +管線編輯器無法編輯 ``Pipeline``：那個物件拒絕任何無效的內容，而建構中的定義大部分 +時間都是無效的。``PipelineDraft`` 保存目前為止輸入的一切。 + +.. code-block:: python + + from automation_file.app import PipelineDraft + + draft = PipelineDraft("daily-report") + draft.set_params({"date": "2026-10-08"}) + draft.add_task("FA_storage_copy", "download", + arguments={"source": "s3://in/${params.date}.csv", + "target": "local:///tmp/report.csv"}) + draft.add_task("FA_storage_delete", "tidy", arguments={"uri": "local:///tmp/report.csv"}) + draft.connect("download", "tidy") # tidy 相依於 download + draft.set_retry("download", max_attempts=5, backoff=2.0, on=["ConnectionError"]) + draft.set_timeout("download", 300) + draft.set_condition("tidy", "always") + draft.rename_task("tidy", "clean-up") # 邊與占位符會跟著改 + + draft.problems() # [] 或 Problem(path, message, task) + draft.to_definition() # Pipeline.from_dict 所接受的文件 + +.. list-table:: + :header-rows: 1 + :widths: 36 64 + + * - 方法 + - 效果 + * - ``add_task``、``remove_task``、``rename_task`` + - 改變任務集合。沒有指定 ID 時，會由動作名稱推導。 + * - ``set_action``、``set_arguments`` + - 動作，以及它的關鍵字對應、位置清單或 ``None``。 + * - ``set_retry``、``set_timeout``、``set_condition``、``set_idempotency_key`` + - 任務如何執行。 + * - ``connect``、``disconnect``、``set_dependencies``、``edges`` + - 相依邊。指向自己的邊，或會形成循環的邊，會以 ``AppException`` 拒絕。 + * - ``set_position``、``positions``、``auto_layout``、``layout``、``apply_layout`` + - 每個任務在畫布上的位置。這是編輯器的中繼資料：絕不會出現在 + ``to_definition()`` 中。 + * - ``set_name``、``set_description``、``set_max_workers``、``set_params``、``set_schedule`` + - 定義的表頭。 + * - ``add_listener``、``batch`` + - 每次變更後都會呼叫 ``listener(change)``，其值為 ``"structure"``、 + ``"task"``、``"header"`` 或 ``"position"``；在 ``with draft.batch():`` + 之內，每一種只在結束時回報一次。 + * - ``problems``、``to_definition``、``from_definition`` + - 附上每個問題路徑的驗證，以及定義文件。 + +執行期會拒絕的值（負的逾時、未知的動作名稱）會被保留下來，並由 ``problems()`` 回報； +只有會讓草稿不一致的編輯（重複的 ID、循環）才會立刻拋出例外。草稿不是執行緒安全的： +請只從一個執行緒編輯它，交給 worker 的則是 ``draft.to_definition()``。 + +檢查與執行 +~~~~~~~~~~~~ + +每個方法都接受草稿或定義對應，並回傳普通的字典。一次執行就是 +``PipelineRun.to_dict()``，其中的機敏資訊已遮蔽，另外加上 ``active`` 旗標，說明這個 +行程是否仍在執行它。 + +.. code-block:: python + + pipelines = services.pipelines + + pipelines.action_names() # 給動作清單用 + pipelines.describe_action("FA_storage_copy") # 參數、預設值、摘要 + + pipelines.validate(draft) # 也包含：未知的動作名稱 + plan = pipelines.dry_run(draft, {"date": "2026-10-09"}) + outcome = pipelines.test_task(draft, "download", {"date": "2026-10-09"}) + + run = pipelines.start(draft, {"date": "2026-10-09"}) # 背景執行 + followed = pipelines.follow(run["run_id"]) # {"run": ..., "events": [...]} + pipelines.cancel(run["run_id"]) + pipelines.wait(run["run_id"], timeout=30) + + pipelines.resume(run["run_id"], draft) # 保留已成功的，執行其餘的 + pipelines.retry(run["run_id"], draft) # 新的一次執行，參數相同 + pipelines.history("daily-report", limit=10) + pipelines.running() + +``start`` 與 ``resume`` 立即回傳；定義有問題時，會在任何東西執行之前拋出例外。 +``test_task`` 單獨執行一個任務：它的動作會真的執行，它的上游任務則換成替身，回傳 +``results=`` 中給的值（預設為 ``None``），而且這次測試不會記錄到任何 run store，也 +不會發布到共用的匯流排。 + +定義檔與配置 +~~~~~~~~~~~~ + +.. code-block:: python + + draft = pipelines.load("pipelines/daily-report.yaml") + draft.load_notes # 檔案有什麼問題（如果有的話） + pipelines.save(draft, "pipelines/daily-report.yaml") + +``save`` 把定義寫成 ``.yaml``、``.yml`` 或 ``.json``，並把畫布配置寫進旁邊的 +``.layout.json``。定義會拒絕未知的鍵，而配置不屬於會被執行的內容，所以兩者絕 +不共用一個檔案。``load`` 會開啟能解析但無效的檔案，讓它可以被修好。 + +Scheduler +--------- + +.. code-block:: python + + scheduler = services.scheduler + scheduler.add("nightly", "0 2 * * *", + [["FA_pipeline_run", {"definition": "pipelines/daily-report.yaml"}]]) + scheduler.jobs() + scheduler.remove("nightly") + scheduler.remove_all() + +``add`` 也接受 JSON 文字形式的動作清單，也就是表單提供的形式。這個服務只呼叫 +``schedule_add``、``schedule_remove``、``schedule_remove_all`` 與 +``schedule_list``\ （:doc:`events`）。 + +Integrity +--------- + +.. code-block:: python + + integrity = services.integrity + integrity.baseline("s3://reports/2026", "local:///var/lib/fa/reports.json") + report = integrity.verify("s3://reports/2026", "local:///var/lib/fa/reports.json") + report["ok"], report["counts"], report["changes"] + integrity.accept("s3://reports/2026", "local:///var/lib/fa/reports.json") + + integrity.start_monitor("reports", "s3://reports/2026", + "local:///var/lib/fa/reports.json", interval=300) + integrity.drift() # 每個監控器一個 MonitorDrift，給儀表板用 + integrity.stop_monitor("reports") + integrity.stop_started() # 這個服務啟動的監控器 + +在這裡啟動的監控器，就是 ``FA_integrity_*`` 動作看到的那個具名監控器。 + +Audit +----- + +.. code-block:: python + + audit = services.audit + audit.status() # 取得 store 之前為 {"configured": False, ...} + audit.configure("/var/lib/automation_file/audit.sqlite") + audit.search(status="error", resource_prefix="s3://reports/", limit=20) + audit.count(actor="scheduler") + audit.recent(limit=30) # 尚未設定稽核時為 [] + +以 ``None`` 或空字串給定的篩選條件不會限制搜尋，所以表單的值可以原樣傳入。 + +Notifications +------------- + +.. code-block:: python + + notifications = services.notifications + notifications.sinks() # 名稱、型別、送達位置；絕不含機敏資訊 + notifications.add_route({"name": "failures", "sinks": "team-alerts, ops-mail", + "types": "pipeline.failed, task.failed", + "min_severity": "error", "dedup_seconds": "600"}) + notifications.routes() + notifications.remove_route("failures") + notifications.send_test("team-alerts") # {"team-alerts": "sent"} + +清單可以是以逗號分隔的文字，數字也可以是文字。路由若指名未註冊的 sink 會被拒絕。 +``send_test`` 為每個 sink 回傳一個結果：``"sent"`` 或錯誤，其中的 URL 只留下主機。 + +Settings +-------- + +.. code-block:: python + + settings = services.settings + settings.load("automation_file.toml") # 一份摘要；不改變任何東西 + settings.apply("automation_file.toml") # 註冊 sink 與路由 + for extra in settings.extras(): + print(extra.name, extra.installed, extra.install_hint) + settings.environment() # 版本、平台、日誌檔 + +摘要包含檔案的區段、sink 與路由，以及所有機敏資訊都已遮蔽的文件。 + +表單輔助函式 +------------------------ + +.. list-table:: + :header-rows: 1 + :widths: 32 68 + + * - 函式 + - 用途 + * - ``parse_argument_text(text)`` + - 把一個表單欄位變成值：文字是 JSON 就當 JSON，否則就是文字本身。 + * - ``format_argument_value(value)`` + - 反方向：產生讀回來會是同一個值的文字。 + * - ``parse_json_text(text, what)`` + - 從文字欄位取得 JSON 文件，否則拋出 ``AppException``，指出 ``what`` 與錯誤的 + 位置。 + * - ``split_names(text)`` + - 把 ``"a, b"`` 變成 ``["a", "b"]``。 + * - ``describe_action(name, command)`` + - 動作的參數、預設值與摘要。 + +機敏資訊 +---------------- + +``mask_secrets(value)`` 回傳可以安全顯示的副本，每個服務都會把它套用在回傳的內容上： + +* 存放在表明自己是機敏資訊的名稱（``password``、``token``、``api_key``、 + ``authorization``……）底下的值，會變成 ``********``； +* 存放在 ``url`` 或 ``..._url`` 底下的值，只保留 scheme 與主機； +* 在其他任何文字中，URL 的使用者資訊與 ``Bearer`` 後面的 token 會被移除。 + +儲存 URI 不會被更動：它不可能帶有憑證。任務的結果會原樣回傳，所以不要把機敏資訊放進 +結果裡。 + +共用與私有的服務 +-------------------------------- + +``app_services()`` 為每個行程回傳一組服務，建立在行程內的單例之上（預設的 resolver、 +預設的 run store、事件匯流排、稽核軌跡、通知管理器與路由器）。因此同一個行程中的視窗 +與 Web UI 會顯示相同的執行、監控器與路由。 + +``build_services`` 則建立自己的一組： + +.. code-block:: python + + from automation_file.app import ServiceOptions, build_services + from automation_file.events import EventBus + from automation_file.pipeline import SQLiteRunStore + + services = build_services(ServiceOptions( + run_store=SQLiteRunStore("/var/lib/automation/pipelines.db"), + bus=EventBus(), + )) + +``ServiceOptions`` 接受 ``resolver``、``run_store``、``registry``、``bus``、 +``audit_trail``、``notification_manager`` 與 ``notification_router``。不論選項為何， +排程器與完整性監控器都是整個行程共用的。 + +撰寫另一種使用者介面 +---------------------------------------- + +1. 取得服務：``app_services()``，或 ``build_services(...)``。 +2. 由 ``NAVIGATION`` 建立導覽。 +3. 每個檢視呼叫一個服務方法，並繪製它回傳的內容。攔截 + ``FileAutomationException`` 並顯示它的文字。 +4. 會碰到儲存或網路的呼叫（``files.*``、``integrity.verify``、 + ``notifications.send_test``、``settings.apply``）請從 worker 執行緒進行。 + ``pipelines.start`` 與 ``pipelines.resume`` 本來就立即回傳。 +5. 管線編輯器請保留一個 ``PipelineDraft``，只透過它的方法修改它，並在 listener 中 + 依它重繪。 + +一個雖小但完整的終端機介面： + +.. code-block:: python + + from automation_file.app import NAVIGATION, app_services + from automation_file.exceptions import FileAutomationException + + services = app_services() + views = { + "Dashboard": lambda: services.dashboard.summary().to_dict(), + "Storage": lambda: [backend.to_dict() for backend in services.storage.backends()], + "Pipelines": lambda: services.pipelines.history(limit=10), + "Scheduler": services.scheduler.jobs, + "Integrity": lambda: [drift.to_dict() for drift in services.integrity.drift()], + "Audit": services.audit.recent, + "Notifications": services.notifications.routes, + "Settings": lambda: [extra.to_dict() for extra in services.settings.extras()], + } + for number, name in enumerate(NAVIGATION, start=1): + print(number, name) + chosen = NAVIGATION[int(input("> ")) - 1] + try: + print(views.get(chosen, lambda: "use services.files for Files")()) + except FileAutomationException as error: + print("failed:", error) + +出問題時 +---------------- + +``AppException`` + 這一層拒絕了請求：草稿無法接受的編輯、不是 JSON 的表單文字、缺少名稱。訊息會說明 + 該改什麼。 + +``dry_run``、``start`` 或 ``resume`` 拋出 ``PipelineDefinitionException`` + 定義無效。``error.problems`` 保存每一項發現；``validate`` 會以 ``Problem`` 物件 + 回傳相同的內容而不拋出例外。 + +``status`` 拋出「unknown run」 + 這次執行既不在這個服務的追蹤之中，也不在它的 run store 裡。除非預設 store 是 + ``SQLiteRunStore``，或曾把它傳給 ``build_services``，否則執行紀錄只保存在 + 記憶體中。 + +``cancel`` 回傳 ``False`` + 這次執行不在這個行程中進行：它已經結束，或是由另一個行程啟動的。 + +後端不是 ``usable`` + 請讀 ``detail``。初始化 client，或安裝 ``install_hint`` 指名的 extra。 + +``audit.search`` 拋出「audit is not configured」 + 請先呼叫 ``audit.configure(path)``。``audit.recent()`` 則會回傳空清單。 + +機敏資訊出現在檢視中 + 它存放在沒有表明自己是機敏資訊的名稱底下，或者它是任務結果的一部分。請改掉欄位 + 名稱，或不要把它放進結果。 + +兩個介面顯示的內容不一致 + 它們用的是不同的服務組。``app_services()`` 是共用的；``build_services()`` + 不是。 diff --git a/docs/source/Zh-TW/usage/gui.rst b/docs/source/Zh-TW/usage/gui.rst index 6aa7717..005dd3b 100644 --- a/docs/source/Zh-TW/usage/gui.rst +++ b/docs/source/Zh-TW/usage/gui.rst @@ -1,10 +1,16 @@ GUI（PySide6） ============== -分頁式控制面板封裝了所有功能： +桌面視窗依「你想完成什麼事」來安排，而不是依後端：側邊欄有九個工作流程頁面，另有一個 +**Advanced** 項目，保留那些直接操作單一動作或單一後端的工具。 + +每個頁面都只是應用層（:doc:`app_layer`）某一個服務的薄薄一層檢視。視窗本身不帶任何 +邏輯，所以它顯示的內容，就是 Web UI（:doc:`servers`）、CLI 與你自己的 Python 程式 +看到的內容。 .. code-block:: bash + pip install "automation_file[gui]" # PySide6 是 extra，不是基礎相依套件 python -m automation_file ui # 或在儲存庫根目錄開發時： python main_ui.py @@ -15,10 +21,434 @@ GUI（PySide6） launch_ui() -分頁：Home、Local、Transfer、Progress、JSON actions、Triggers、 -Scheduler、Servers。所有分頁下方共用一個常駐日誌面板， -逐條輸出每次呼叫的結果或錯誤。背景工作透過 ``ActionWorker`` 在 -``QThreadPool`` 上執行，UI 始終保持回應。 +沒有安裝 PySide6 時，``launch_ui`` 會拋出 ``OptionalDependencyException``，訊息中 +帶有上面那行 ``pip install`` 指令。 + +導覽 +-------- + +.. list-table:: + :header-rows: 1 + :widths: 18 12 70 + + * - 項目 + - 快速鍵 + - 用途 + * - Dashboard + - ``Ctrl+1`` + - 一眼看完健康狀態、執行中與最近的管線執行、完整性漂移、最近的事件與儲存狀態。 + * - Files + - ``Ctrl+2`` + - 瀏覽儲存 URI、預覽檔案、複製、搬移、刪除、建立目錄。 + * - Storage + - ``Ctrl+3`` + - 有哪些後端、每個後端能不能用，以及掛載點。 + * - Pipelines + - ``Ctrl+4`` + - 視覺化管線編輯器：建立、驗證、試跑、測試、執行、續跑。 + * - Scheduler + - ``Ctrl+5`` + - Cron 工作：列出、新增、移除。 + * - Integrity + - ``Ctrl+6`` + - 為目錄樹建立基準、驗證與接受；啟動與停止監控器。 + * - Audit + - ``Ctrl+7`` + - 把稽核軌跡指向資料庫，搜尋並計算其中的紀錄。 + * - Notifications + - ``Ctrl+8`` + - 已註冊的 sink、路由，以及測試訊息。 + * - Settings + - ``Ctrl+9`` + - 設定檔、選用的 extra、執行環境。 + * - Advanced + - ``Ctrl+0`` + - Local、Transfer、Progress、JSON actions、Triggers 與 Servers：舊版視窗的 + 分頁。見 `舊分頁去了哪裡`_。 + +視窗 +-------- + +側邊欄在左，選取的頁面在右。兩者下方是所有頁面共用的 **活動日誌**：每個動作開始時寫 +一行、得到結果時再寫一行，並以頁面名稱開頭。最新的一行也會在狀態列顯示幾秒鐘。 + +每個頁面的底部都有一行 **狀態列文字**，顯示你在這個頁面上最後做的那件事的結果：成功 +是綠色，失敗是紅色。 + +沒有任何操作會卡住視窗。頁面把每一次呼叫交給執行緒池（``QThreadPool`` 上的 +``ActionWorker``），結果回來時才顯示。頁面在你開啟它時讀取資料，按下它的 **Refresh** +按鈕時再讀一次。有兩個檢視會自己更新：Dashboard 在可見時每五秒更新一次；被追蹤的管線 +執行每秒更新兩次，直到它結束。 + +視窗與函式庫其餘部分共用行程內的單例。從 Python 註冊的 sink、透過 HTTP 動作伺服器 +啟動的管線，或由 JSON 動作啟動的監控器，都會在下一次更新後出現。 + +Dashboard +--------- + +**All clear** 或 **Needs attention**，旁邊列出原因。只要最新的五十次執行中有一次 +失敗、某個監控器發現漂移或無法驗證，或事件匯流排上有嚴重程度為 ``error`` 以上的近期 +事件，儀表板就會要求注意。 + +* **Health**：已註冊的動作、執行中的執行、排程工作、完整性監控器、通知 sink 與路由、 + 路由器是否啟用，以及稽核軌跡是否正在記錄。 +* **Pipeline runs**：最新幾次執行的結局、仍在進行的執行，以及最近的結果與錯誤。 +* **Integrity drift**：每個具名監控器、它的目標，以及它上次發現了什麼。 +* **Storage status**：每個後端以及能不能用。 +* **Recent events**：匯流排上最新的二十個事件，新的在前。 + +**Refresh every 5 s** 開關計時器；**Refresh** 立即讀取。儀表板只讀取：不啟動任何 +東西，也不改變任何東西。 + +Files +----- + +輸入儲存 URI 或本機路徑（``s3://reports/2026``、``local:///data``、``C:\data``、 +``memory://demo``）後按 **Open**。**Up** 回到上一層，**Browse local…** 選取目錄。 +在目錄上按兩下會進入該目錄，在檔案上按兩下會預覽它。 + +預覽是有上限的：最多顯示 64 KiB，而遠端後端上超過 16 MiB 的檔案根本不會被抓取（狀態 +列文字會說明）。二進位內容以前 256 個位元組的十六進位傾印顯示。 + +各項操作都作用在選取的項目上： + +* **Copy** / **Move** 到目標 URI，可以在同一個後端或另一個後端。目標若是既有的目錄， + 檔案會以原本的名稱放進去。目錄會連同底下的一切一起複製；目錄不能搬移（請先複製、 + 檢查副本，再刪除原本的）。 +* **Create directory** 在目前位置底下建立目錄。 +* **Delete selected** 會先詢問。有內容的目錄需要勾選 **Delete a directory with its + contents**。 + +每個路徑都經過儲存層（:doc:`storage`）：``..`` 會被拒絕，URI 中的憑證會被拒絕，掛載 +的目錄無法被跳出。 + +Storage +------- + +每個後端一列： + +.. list-table:: + :header-rows: 1 + :widths: 14 86 + + * - Kind + - 意義 + * - ``scheme`` + - 有工廠函式的 URI scheme：``local``、``memory``、``s3``、``azure``、 + ``gdrive``、``dropbox``、``onedrive``、``sftp``、``ftp``、``ftps``。 + * - ``mount`` + - 掛載在某個 URI 上的後端。它的名稱就是那個 URI。 + * - ``client`` + - 有 ``FA_*`` 動作但沒有儲存 scheme 的共用 client（Box）。 + +本機與記憶體後端、掛載點，以及 client 已初始化的雲端後端，**Usable** 為 ``yes``。 +**Detail** 說明缺了什麼：如何初始化 client；若該 extra 的套件沒有安裝，則是 +``pip install`` 指令。 + +**Mount a local directory** 以你選的 URI 提供某個目錄，並把它限制在那個目錄內 +（把 ``sandbox://jobs`` 掛在 ``/srv/jobs``）。來自行程外部的路徑請用這個方式處理。 +**Unmount selected** 移除掛載點。選取掛載點、``local`` 或 ``memory`` 時，會顯示該 +後端提供哪些能力（目錄、修改時間、ETag……）。 + +Pipelines +--------- + +這個頁面是管線定義（:doc:`pipeline`）的編輯器，也是它的執行主控台。 + +.. list-table:: + :header-rows: 1 + :widths: 24 76 + + * - 區域 + - 內容 + * - 最上面一列 + - **New**、**Open…**、**Save**、**Save as…**、**Auto layout**，以及檔案名稱 + （有未儲存的變更時會加上 ``(modified)``）。 + * - 第二列 + - 管線名稱、**Max workers**、描述、預設參數（JSON）與排程（cron 運算式，留給 + 排程器使用）。 + * - Actions（左） + - 所有已註冊的動作，附篩選欄。 + * - 畫布（中） + - 每個任務一個節點、每個相依一條箭頭，上方有 **Connect**、**Disconnect** 與 + **Remove selected**。 + * - Selected task（右） + - 畫布上選取的任務的表單。 + * - 執行列 + - 執行參數（JSON）以及 **Validate**、**Dry run**、**Test task**、**Run**、 + **Resume**、**Retry**、**Cancel**、**Refresh history**、 + **Follow selected run**。 + * - 底部的分頁 + - **Problems**、**Tasks**、**Log**、**History**。 + +管線編輯器逐步說明 +------------------------------------ + +1. **開始。** 按 **New** 建立空白定義，或按 **Open…** 開啟 ``.yaml`` / ``.yml`` / + ``.json`` 檔案。不是有效定義的檔案仍然打得開：形狀正確的部分會顯示出來， + **Problems** 分頁則列出它哪裡有問題。 + +2. **命名。** 填入名稱、worker 數量，需要的話再填描述、JSON 物件形式的預設參數與 + 排程。 + +3. **新增任務。** 把 **Actions** 清單中的動作拖到畫布上：任務會出現在你放開的位置。 + 在動作上按兩下，或選取它再按 **Add task**，會把它加在最下面的任務之下。篩選欄可以 + 縮小清單（輸入 ``storage`` 會顯示 ``FA_storage_*`` 動作）。任務的 ID 由動作名稱 + 推導而來；可以在表單中修改。 + +4. **排列。** 把節點拖到你想要的位置。**Auto layout** 依相依深度把每個任務放進對應的 + 欄。位置是編輯器的中繼資料：它存在定義旁邊，絕不會存進定義裡。 + +5. **連接。** 先點上游任務，再按住 ``Ctrl`` 點相依於它的任務，然後按 **Connect**。 + 按鈕會顯示它將採用的方向（``Connect download -> publish``）：你選取這兩個任務的 + 順序，就是箭頭的方向。會形成相依循環的箭頭會被拒絕。你也可以在任務表單的 + **Depends on** 中勾選上游任務。 + + 要移除相依，請點它的箭頭（或選取它兩端的任務），再按 **Disconnect**。 + **Remove selected** 會移除選取的箭頭；沒有選取箭頭時，則移除選取的任務與它們的 + 箭頭。 + +6. **編輯選取的任務。** 表單顯示最後選取的任務： + + .. list-table:: + :header-rows: 1 + :widths: 24 76 + + * - 欄位 + - 意義 + * - Task ID + - 在管線中唯一。改名會保留箭頭，並改寫指向該任務的 + ``${tasks..result}`` 占位符。 + * - Action + - 已註冊的動作。它的簽章與摘要顯示在下方。 + * - Arguments + - 動作的每個參數一列，並附上預設值。值若能解析成 JSON 就是 JSON + （``12``、``true``、``["a", "b"]``、``"12"``），否則就是文字，所以 URI 或 + ``${params.date}`` 不需要加引號。留空的列不會被傳入，因此套用預設值。 + **Add argument** 為接受 ``**kwargs`` 的動作新增一列。**Edit as JSON** + 把引數顯示成一份 JSON 文件；位置引數（JSON 陣列）只能用這種方式編輯。 + * - Depends on + - 勾選必須先結束的任務。 + * - Attempts、Back-off、Back-off cap、Retry on + - 總嘗試次數、第一次退避時間與其上限，以及值得再試一次的例外名稱（留空： + 暫時性的那幾種）。 + * - Timeout + - 整個任務的秒數；留空表示不限。 + * - Run when + - ``on_success``、``on_failure`` 或 ``always``。 + * - Idempotency key + - 帶有 ``${params.}`` 占位符的文字；留空表示沒有。 + + 在你按下 **Apply changes** 之前，什麼都不會改變。**Revert** 會丟掉你輸入的內容， + 選取另一個任務也一樣。草稿拒絕某個值時，表單會說明原因，並保留你輸入的內容。 + +7. **驗證。** **Problems** 分頁列出每一項發現，並附上所指項目的路徑 + （``tasks.publish.depends_on[0]: unknown task 'x'``），包含註冊表不認得的動作 + 名稱。點某個問題會選取它的任務。**Dry run**、**Test task**、**Run**、**Resume** + 與 **Retry** 都會先驗證，有問題就停下來。 + +8. **試跑。** 以 JSON 物件輸入執行參數，然後按 **Dry run**。不會執行也不會記錄任何 + 東西。**Tasks** 分頁把每個任務顯示為 ``planned`` 並附上層級；無法照計畫執行的任務 + 會說明原因（例如這次執行沒有提供某個參數）。 + +9. **測試單一任務。** 選取一個任務再按 **Test task**。它的動作會真的執行，並套用它的 + 重試策略與逾時。它的上游任務則不會執行：每一個都換成不回傳任何東西的替身，所以 + ``${tasks..result}`` 占位符會得到 ``null``。測試不會記錄到任何 run store， + 也不會發布到共用的事件匯流排。 + +10. **執行。** **Run** 在背景啟動管線並追蹤它：**Tasks** 分頁顯示每個任務的狀態、 + 嘗試次數、耗時與錯誤，**Log** 分頁顯示這次執行的事件，每個節點則換成它的狀態 + 顏色（灰色 pending、藍色 running、綠色 succeeded、紅色 failed 或逾時、橘色 + cancelled、黃色 skipped）。**Cancel** 要求這次執行停下來。 + +11. **續跑或重試。** 失敗之後，先排除原因。**Resume** 接續同一次執行：已成功的任務 + 會保留，其餘的重新執行，執行 ID 與參數都不變。**Retry** 以被追蹤的那次執行的 + 參數啟動一次新的執行。 + +12. **回顧。** **History** 分頁列出已記錄的執行，新的在前。在某一筆上按兩下，或選取 + 它再按 **Follow selected run**，就能再看到它的任務與事件；之後 **Resume** 與 + **Retry** 會作用在它身上。 + +13. **儲存。** **Save as…** 把定義寫成 ``.yaml``、``.yml`` 或 ``.json``，並把畫布 + 配置寫進旁邊的 ``.layout.json``。定義檔正是 ``Pipeline.from_file`` 與 + ``FA_pipeline_run`` 所接受的檔案。 + +Scheduler +--------- + +**Schedule a job** 需要唯一的名稱、五個欄位的 cron 運算式，以及 JSON 形式的動作 +清單，按 **Add job** 註冊。勾選核取方塊可以讓新的一次執行在前一次還沒結束時啟動；否則 +那次觸發會被略過，並計入 **Skipped**。表格顯示每個工作的執行次數與上次執行時間； +**Remove selected** 與 **Remove all** 移除工作。要讓管線依排程執行，請排程 +``FA_pipeline_run`` 並給它定義檔的路徑。 + +視窗關閉時，這些工作會被移除。 + +Integrity +--------- + +輸入 **Target**（要檢查的目錄樹）與 **Baseline**（已核可狀態存放的位置），兩者都是 +儲存 URI 或本機路徑。 + +* **Create baseline** 核可目前的內容。 +* **Verify** 把目錄樹與基準比較，並列出每一項變更的種類與路徑。關閉 **Deep** 時，只有 + 大小或時間改變的檔案會被雜湊，摘要會註明這是快速檢查。 +* **Accept current state** 會先詢問，然後把目前的目錄樹存成新的基準。 + +在 **Monitors** 底下，給一個名稱與間隔，按 **Start monitor** 就會在執行緒上持續驗證 +目標；表格顯示每個監控器上次發現了什麼。從視窗啟動的監控器會在視窗關閉時停止。見 +:doc:`integrity`。 + +Audit +----- + +稽核軌跡在取得 store 之前什麼都不記錄。輸入 SQLite 資料庫的路徑（不存在時會建立）， +按 **Configure**；之後 **State** 會顯示紀錄寫到哪裡。 + +填入任何篩選條件，按 **Search**（新的在前，最多 **Limit** 筆）或 **Count**。留空的 +篩選條件不會限制搜尋。**Since** 與 **Until** 接受帶時區偏移的 ISO 8601 時間。選取 +一筆紀錄可以看到它的全部內容（JSON）。見 :doc:`audit`。 + +Notifications +------------- + +**Sinks** 依名稱、型別與送達位置列出已註冊的 sink。Sink 是在程式中或從設定檔 +（Settings）註冊的；這個頁面不會建立它們。選一個 sink（或 **All sinks**），需要的話 +填入主旨，按 **Send test message**；狀態列文字會顯示每個 sink 的結果。 + +**Routes** 列出哪些事件送到哪些 sink。**Add or replace a route** 需要名稱、sink +名稱、事件類型（``pipeline.*``、``task.failed``）、來源、最低嚴重程度與節流數值；清單 +以逗號分隔。路由若指名未註冊的 sink 會被拒絕。路由器隨第一條路由啟動，最後一條路由 +被移除時停止。見 :doc:`notifications`。 + +Settings +-------- + +輸入 ``automation_file.toml`` 的路徑（:doc:`config`）。 + +* **Preview** 讀取檔案並顯示摘要。不會改變任何東西。 +* **Apply** 註冊它的通知 sink 與路由。 + +**Optional extras** 列出每個 extra、它啟用的功能、是否已安裝，以及未安裝時的 +``pip install`` 指令。**Environment** 顯示套件與 Python 版本、平台、日誌檔，以及 +最後套用的設定檔。 + +舊分頁去了哪裡 +---------------------------- + +什麼都沒有移除。**Advanced** 項目保留了舊的分頁，原封不動： + +.. list-table:: + :header-rows: 1 + :widths: 28 72 + + * - 舊分頁 + - 現在 + * - Home + - **Dashboard**（後端是否就緒在 *Storage status* 底下，也在 **Storage** 頁面）。 + * - Local + - **Advanced** → *Local*。瀏覽與複製也可以在 **Files** 進行。 + * - Transfer（HTTP、Google Drive、S3、Azure Blob、Dropbox、SFTP、OneDrive、 + Box） + - **Advanced** → *Transfer*。雲端 client 的憑證就是在這裡提供。 + * - Progress + - **Advanced** → *Progress*。 + * - JSON actions + - **Advanced** → *JSON actions*。 + * - Triggers + - **Advanced** → *Triggers*。 + * - Scheduler + - **Scheduler**。 + * - Servers + - **Advanced** → *Servers*。 + +機敏資訊 +---------------- + +沒有任何頁面會顯示機敏資訊。Sink 以名稱、型別與送達位置描述；設定檔摘要中的密碼與 +token 會換成 ``********``，webhook URL 只留下主機；事件、稽核紀錄與執行參數在顯示前 +也以同樣方式遮蔽；URL 中的憑證與 bearer token 會從頁面顯示的訊息中移除。 + +值是否被遮蔽，取決於存放它的 *名稱* 是否表明它是機敏資訊（``password``、``token``、 +``api_key``、``authorization``……）。任務的結果則照任務回傳的樣子顯示。所以請給機敏 +參數取這樣的名稱，並且不要把機敏資訊放進結果裡。 + +在沒有顯示器的環境執行 +------------------------------------------ + +Qt 需要顯示器才能開啟視窗。在沒有顯示器的伺服器上： + +* 唯讀檢視請用 Web UI（:doc:`servers`）；它由同一個應用層繪製。 +* 從 Python 直接使用應用層（:doc:`app_layer`），或透過 CLI 與動作伺服器使用 + ``FA_*`` 動作。視窗做的每一件事，都是對其中之一的呼叫。 +* 要建構視窗但不顯示它（測試、CI 中的螢幕截圖），請在匯入 Qt 之前設定 + ``QT_QPA_PLATFORM=offscreen``： + + .. code-block:: python + + import os + os.environ.setdefault("QT_QPA_PLATFORM", "offscreen") + + from PySide6.QtWidgets import QApplication + from automation_file.app import build_services + from automation_file.ui.main_window import MainWindow + + app = QApplication([]) + window = MainWindow(build_services()) # 或以 MainWindow() 使用共用的服務 + window.navigate("Pipelines") + window.close() + +``import automation_file`` 與 ``import automation_file.app`` 絕不會匯入 PySide6； +只有 ``automation_file.ui`` 會。 + +出問題時 +---------------- + +按鈕好像沒有反應 + 請看頁面底部的狀態列文字與活動日誌。每一次拒絕與每一次失敗都會在那裡回報，並附上 + 原因。 + +``launch_ui`` 拋出 ``OptionalDependencyException`` + 沒有安裝 PySide6：``pip install "automation_file[gui]"``。 + +視窗打不開：「could not load the Qt platform plugin」 + 沒有顯示器。見 `在沒有顯示器的環境執行`_。 + +雲端後端顯示「not initialised」 + 它的共用 client 還沒有憑證。開啟 **Advanced** → *Transfer*，選擇該後端並填寫 + *Credentials*，或執行它的 ``FA_*_later_init`` 動作。 + +Detail 說某個套件「is not installed」 + 安裝該列指名的 extra；**Settings** 列出每個 extra 與它的指令。 + +**Connect** 被拒絕 + 這條箭頭會形成相依循環，或是選取的任務不是剛好兩個。 + +**Run** 停在「problem(s): see the Problems tab」 + 定義無效。每個問題都以所指項目的路徑開頭；點它就會選取該任務。 + +按了 **Cancel** 之後執行仍是 ``running`` + 執行中的動作無法被中斷。尚未開始的任務會立刻被取消；執行中的任務回傳後，這次執行 + 才會結束。 + +**Resume** 被拒絕 + 這次執行仍在進行、屬於另一個名稱的管線，或是定義現在需要已儲存的執行所沒有的 + 參數。 + +重新啟動後歷史是空的 + 執行紀錄預設保存在記憶體中。在 ``launch_ui()`` 之前呼叫 + ``set_default_run_store(SQLiteRunStore(path))``，就能把它們保存在檔案裡。 + +**Search** 說 audit is not configured + 請先在同一個頁面為稽核軌跡指定資料庫。 + +測試訊息失敗 + 狀態列文字會顯示每個 sink 的錯誤，URL 只留下主機。Sink 本身的說明見 + :doc:`notifications`。 + +頁面顯示的是舊資料 + 按 **Refresh**。只有 Dashboard 與被追蹤的執行會自己更新。 -GUI 與函式庫其餘部分共用同一組單例——從 Python 註冊的 sink、 -自訂動作、觸發器都會立即在執行中的視窗生效。 +視窗關閉後還有東西在執行 + 關閉視窗會移除排程工作，並停止它啟動的監控器、動作伺服器與觸發器，但不會動管線 + 執行：已啟動的執行會跑到結束，行程在那之前不會結束。 diff --git a/docs/source/Zh-TW/usage/pipeline.rst b/docs/source/Zh-TW/usage/pipeline.rst index 96a8366..cf8bd99 100644 --- a/docs/source/Zh-TW/usage/pipeline.rst +++ b/docs/source/Zh-TW/usage/pipeline.rst @@ -12,6 +12,10 @@ :func:`~automation_file.execute_action_dag`\ （見 :doc:`dag`）維持不變。它把一份動作 清單執行一次並回傳結果；當執行需要被記錄、重試、續跑或觀察時，請改用管線。 +定義也可以不必手寫就建立、檢查與執行：桌面視窗的 Pipelines 頁面是它的視覺化編輯器 +（:doc:`gui`），而 ``automation_file.app`` 則把同樣的操作提供給任何其他介面 +（:doc:`app_layer`）。 + 最小範例 ---------------- diff --git a/docs/source/Zh-TW/usage/servers.rst b/docs/source/Zh-TW/usage/servers.rst index 78c894e..f3dd0da 100644 --- a/docs/source/Zh-TW/usage/servers.rst +++ b/docs/source/Zh-TW/usage/servers.rst @@ -44,3 +44,73 @@ HTTP 回應皆為 JSON。授權失敗回 ``401``；JSON 異常回 ``400``； 共享密鑰比較使用 :func:`hmac.compare_digest`（常數時間）。 切勿記錄密鑰或原始負載。 + +Web UI +------ + +瀏覽器中的唯讀儀表板，以標準函式庫與 HTMX 提供（從固定的 CDN URL 載入一支腳本，並 +附 SRI 雜湊）。 + +.. code-block:: python + + from automation_file import start_web_ui + + server = start_web_ui(host="127.0.0.1", port=9955, shared_secret="optional-secret") + # 瀏覽 http://127.0.0.1:9955/ + # 稍後： + server.shutdown() + server.server_close() + +頁面為每個區段輪詢一個 HTML 片段。除了傳輸進度以外，每個片段都由應用層 +（:doc:`app_layer`）繪製，也就是桌面視窗（:doc:`gui`）所呼叫的同一組服務，所以兩者 +顯示相同的狀態。 + +.. list-table:: + :header-rows: 1 + :widths: 22 14 64 + + * - 片段 + - 輪詢間隔 + - 顯示內容 + * - ``GET /ui/health`` + - 5 秒 + - ``ok`` 或 ``attention`` 與其原因；已註冊的動作、執行中的執行、排程工作、 + 監控器、sink 與路由、稽核軌跡。 + * - ``GET /ui/runs`` + - 3 秒 + - 執行中與最近的管線執行，以及最新幾次執行的結局。 + * - ``GET /ui/integrity`` + - 10 秒 + - 每個具名完整性監控器，以及它上次發現的漂移。 + * - ``GET /ui/events`` + - 5 秒 + - 匯流排上最新的事件，新的在前。 + * - ``GET /ui/storage`` + - 30 秒 + - 每個儲存後端以及能不能用。 + * - ``GET /ui/audit`` + - 10 秒 + - 最新的稽核紀錄；要先以 ``configure_audit`` 給稽核軌跡一個 store。 + * - ``GET /ui/progress`` + - 2 秒 + - 進度註冊表中進行中的傳輸。 + * - ``GET /ui/registry`` + - 30 秒 + - 每個已註冊動作的名稱。 + +``GET /`` 與 ``GET /index.html`` 提供頁面；其他路徑一律回 ``404``。沒有任何路由會 +改變東西：要執行動作，請透過上面的動作伺服器，並使用它們自己的驗證機制。 + +* **預設只綁定 loopback。** 要綁定其他位址必須傳 ``allow_non_loopback=True``；這麼做 + 卻沒有 ``shared_secret`` 時會記錄一則警告。 +* **共享密鑰。** 設定 ``shared_secret`` 後，每個請求都需要 + ``Authorization: Bearer ``，否則回 ``401``。頁面把這個標頭放在 + ``hx-headers`` 中，讓它自己的輪詢得到授權；因此能讀到頁面的人就能讀到密鑰，請透過 + loopback 或在 TLS 之後提供它。 +* **已跳脫且已遮蔽。** 片段顯示的一切都經過 HTML 跳脫，而且應用層已經先遮蔽其中的 + token、密碼與 webhook URL。 +* **片段絕不會弄壞頁面。** 某個服務無法回應時，它的片段會顯示 + ``unavailable: ``，其他片段照常運作。 + +``start_web_ui(services=...)`` 接受以 ``automation_file.app.build_services`` 建立的 +一組服務，用來顯示行程共用的那一組以外的 run store、事件匯流排或 resolver。 diff --git a/docs/source/Zh-TW/zh_tw_index.rst b/docs/source/Zh-TW/zh_tw_index.rst index 36483a2..28eb02f 100644 --- a/docs/source/Zh-TW/zh_tw_index.rst +++ b/docs/source/Zh-TW/zh_tw_index.rst @@ -150,6 +150,7 @@ PySide6 桌面控制介面——分頁佈局、日誌面板，以及 ``ActionWor :caption: 圖形介面 usage/gui + usage/app_layer .. _zh-tw-reliability: diff --git a/docs/updates/2026-10.md b/docs/updates/2026-10.md index 1fc1cfb..f2f89cb 100644 --- a/docs/updates/2026-10.md +++ b/docs/updates/2026-10.md @@ -641,3 +641,20 @@ Index and query commands: [README.md](README.md). New entries go at the end. - **Docs**: `usage/scheduler.rst` in the three manuals (in the "Triggers and Scheduler" chapter), `docs/source/API/scheduler.rst`, pointers in the three `usage/events.rst`, `usage/notifications.rst`, `usage/pipeline.rst` and `architecture.rst`, the feature bullet and the scheduler section of the three READMEs, `architecture.md` §2 and §3, `CLAUDE.md` (package map, key types). - **Files**: `automation_file/scheduler/` (`errors`, `runs`, `triggers`, `job`, `targets`, `dispatch`, `cron`, `manager`, `__init__`), `automation_file/trigger/manager.py`, `automation_file/__init__.py`, `stable.toml`, `dev.toml`, the tests above, the documentation above. - **Open items**: none for the scheduler. `automation_file.scheduler` as an attribute of the package is the `Scheduler` instance, as it was before; import the subpackage's names with `from automation_file.scheduler import ...`. + +## U-20261008-30 · 2026-10-08 · UI 2.0 and the application layer · #ui #roadmap #done + +- **What** (roadmap §11, M7; closes `progress.md` #24): + - **Application layer**, `automation_file/app/`: plain Python with no Qt and no SDK at import, returning JSON-friendly values. One service per navigation entry: dashboard (running pipelines, recent successes and failures, integrity drift, recent events, storage status with the install hint of a missing extra), files, storage, pipelines, scheduler, integrity, audit, notifications, settings. `PipelineDraft` is the editable definition behind the editor. Secrets are masked before anything is shown (`masking.py`). + - **GUI**: `MainWindow` has a sidebar with Dashboard, Files, Storage, Pipelines, Scheduler, Integrity, Audit, Notifications and Settings; each page talks only to its service. The older tabs (Local, Transfer, Progress, JSON actions, Triggers, Servers) are under Advanced. `HomeTab` and `SchedulerTab` stay importable but are no longer mounted: the Dashboard and Scheduler pages replace them. + - **Pipeline editor**: tasks as draggable nodes on a canvas with dependency edges; a task is added from the registered actions; two tasks are connected by selecting both and pressing Connect (the first selected is upstream) or through the "Depends on" list; a form edits the task; Validate, Dry run, Test task, Run, Resume, Retry and Cancel; a status table and a log of the run's events. Canvas positions are kept in a `.layout.json` sidecar, since a definition refuses unknown keys. Dragging from one node to another to connect them was not built. + - **Web UI**: `start_web_ui(services=...)` renders the same services, still read-only: fragments for runs, integrity, events, storage and audit next to the existing ones. + - `launch_ui` raises `OptionalDependencyException` naming the `gui` extra when PySide6 is missing. +- **Changed while integrating**: `tests/test_integrity_watch.py::test_the_interval_runner_ticks_until_stopped_and_can_start_again` failed once in a full run. It waited on a semaphore that could still hold a tick from the runner's first start, then stopped the runner before a new tick. It now waits for the count to rise. This is very likely the unidentified failure of `progress.md` #35, which is closed with it. +- **Tests**: `tests/test_app_{masking,files,pipeline_draft,pipelines,services}.py` without Qt; `tests/test_ui_pages.py`, `tests/test_ui_pipeline_editor.py` and the updated `tests/test_ui_smoke.py` offscreen; `tests/test_web_ui_app_layer.py`. +- **Result / numbers**: 5763 passed, 257 skipped, 0 failed with every extra; 3879 passed, 137 skipped with the base dependencies only. `ruff check`, `ruff format --check` and `mypy automation_file` (275 files) pass. Importing `automation_file` loads no Qt module. Python 3.14.7 on Windows. +- **Not verified, and it matters**: nothing ran on a real display. All Qt ran on the offscreen platform, which has no fonts, so layout was checked and text fit was not. A real drag from the palette, Ctrl-click selection, the file dialogs, the confirmation boxes and the keyboard shortcuts were not exercised. No real cloud backend stood behind any page. Recorded as `progress.md` #38. +- **Known shortcuts**: `StorageService` reads the resolver's private mount and factory tables, and the app layer uses four pipeline helpers that are not in the package's `__all__` (`progress.md` #39). +- **Docs**: `usage/gui.rst` rewritten and `usage/app_layer.rst` added in the three manuals, a Web UI section in the three `usage/servers.rst`, `docs/source/API/app.rst` and `API/ui.rst`, the indexes, the GUI, application-layer and Web UI sections, three bullets and the diagram of the three READMEs, `architecture.md` §2 and §3, `CLAUDE.md` (package map, `MainWindow`). +- **Files**: `automation_file/app/` (15 modules), `automation_file/ui/pages/` (15 modules), `automation_file/ui/{main_window,launcher,__init__}.py`, `automation_file/server/web_ui.py`, `automation_file/__init__.py`, the tests above, the documentation above. +- **Open items**: #38, #39. diff --git a/docs/updates/README.md b/docs/updates/README.md index 040202d..d89f38a 100644 --- a/docs/updates/README.md +++ b/docs/updates/README.md @@ -58,6 +58,7 @@ In the same commit: delete the item from `progress.md`, add a `#done` entry here | ID | Date | Title | Tags | Batch | |---|---|---|---|---| +| U-20261008-30 | 2026-10-08 | UI 2.0 and the application layer | #ui #roadmap #done | [2026-10](2026-10.md) | | U-20261008-29 | 2026-10-08 | Scheduler v2 | #scheduler #roadmap #done | [2026-10](2026-10.md) | | U-20261008-28 | 2026-10-08 | One positioning in the READMEs, the manuals and the metadata | #docs #packaging #roadmap | [2026-10](2026-10.md) | | U-20261008-27 | 2026-10-08 | Semantic MCP tools | #mcp #security #roadmap #done | [2026-10](2026-10.md) | @@ -124,5 +125,5 @@ In the same commit: delete the item from `progress.md`, add a `#done` entry here | File | Period | Entries | |---|---|---:| -| [2026-10.md](2026-10.md) | 2026-10 | 40 | +| [2026-10.md](2026-10.md) | 2026-10 | 41 | | [2026-09.md](2026-09.md) | 2026-09 | 21 | diff --git a/progress.md b/progress.md index 51f7bab..1440d0c 100644 --- a/progress.md +++ b/progress.md @@ -25,10 +25,10 @@ Items #10 to #26 are what is left of the 1.0 roadmap (`docs/FILEAUTOMATION-1.0-R ### Later milestones -- **#24** UI 2.0 (roadmap §11, M7). Not before the APIs of #13 to #23 are stable (roadmap §20). +- **#38** [UNVERIFIED] UI 2.0 (U-20261008-30) has only run on Qt's offscreen platform. Open it on a real display (`python -m automation_file ui`) and check: text fit and layout of the ten pages, dragging a node and adding a task on the pipeline canvas, selecting two tasks and connecting them, the file dialogs, the confirmation boxes, the keyboard shortcuts, and one page against a real backend. Dragging from node to node to connect two tasks is not built. +- **#39** The application layer reaches into two private places: `StorageService` reads `StorageResolver._mounts` / `_factories` (give the resolver a public listing of its mounts), and `app/` uses `pipeline.definition.retry_from_dict`, `graph.upstream_tasks`, `substitution.is_name` / `NAME_RULE` and `model.ON_SUCCESS` / `WHEN_CHOICES`, which are not in `automation_file.pipeline.__all__` (export them, or give the pipeline package the functions the editor needs). - **#37** Storage-layer gaps the MCP work found (U-20261008-27) and did not change: (a) a `LocalStorage(root)` listing reports a link's name and its target's metadata without a containment check, although reading through the link is refused; (b) `StorageBackend` has no ranged read, so reading the head of a large remote file stages all of it; (c) `LocalStorage` on Windows opens device names (`CON`, `NUL`) and alternate data streams, which the MCP tools refuse themselves; (d) a link on an SFTP or FTP server leads outside a root that is only a path prefix. - **#26** Release engineering and 1.0 (roadmap §13, M9). Done: semantic versioning with a way to release a MINOR or MAJOR (U-20261008-24), the public API policy (U-20261008-22), a package-build check and the integration workflow next to the PR checks. Open: the migration guide and the final documentation audit (after the scheduler, MCP and GUI work lands); making the integration jobs required once they are green (#19); the 1.0.0 release itself, which is the owner's call: write `1.0.0` in both TOMLs in the release pull request. -- **#35** [UNVERIFIED] One full run of the suite on 2026-10-08 reported `1 failed, 4882 passed` and four runs around it passed. The machine was running three other test suites at the time, and the run was not started with `-rf`, so the test is not known. A test that depends on timing is the likely cause (the pipeline timeout and cancellation cases, the integrity watchers, the SFTP loopback, the scheduler). Run the suite with `-rf` under load, or read the first CI runs, to find it. - **#34** [BLOCKED] PyPI Trusted Publishing (roadmap §13). `publish.yml` and `publish-dev` still upload with the `PYPI_API_TOKEN` secret. Switching needs the owner to add a trusted publisher for each project on PyPI (`automation_file`: workflow `publish.yml`; `automation_file_dev`: workflow `ci-dev.yml`; an environment name if one is wanted) before the workflows can drop the token for `id-token: write` and `pypa/gh-action-pypi-publish`. Changing the workflows first would break both channels. ### Packaging follow-ups diff --git a/tests/test_app_files.py b/tests/test_app_files.py new file mode 100644 index 0000000..e9cd0e6 --- /dev/null +++ b/tests/test_app_files.py @@ -0,0 +1,361 @@ +"""The Files and Storage services of the application layer, on private resolvers.""" + +from __future__ import annotations + +from collections.abc import Iterable +from pathlib import Path + +import pytest + +from automation_file.app import ( + AppException, + BackendStatus, + FileService, + StorageService, +) +from automation_file.app import storage_service as storage_module +from automation_file.core.optional import install_hint +from automation_file.exceptions import ( + PathTraversalException, + StorageNotEmptyException, + StorageNotFoundException, + StorageURIException, +) +from automation_file.storage import ( + File, + FileInfo, + LocalStorage, + MemoryStorage, + StorageBackend, + StorageResolver, +) + +ROOT = "memory://files" + + +class _Remote(StorageBackend): + """A backend that is neither local nor in-memory to the service: files are fetched whole.""" + + scheme = "remote" + + def __init__(self) -> None: + self._inner = MemoryStorage() + + def _stat(self, path: str) -> FileInfo | None: + return self._inner._stat(path) + + def _list_dir(self, path: str) -> Iterable[FileInfo]: + return self._inner._list_dir(path) + + def _upload(self, source: Path, path: str) -> None: + self._inner._upload(source, path) + + def _download(self, path: str, target: Path) -> None: + self._inner._download(path, target) + + def _delete_file(self, path: str) -> None: + self._inner._delete_file(path) + + def _mkdir(self, path: str) -> None: + self._inner._mkdir(path) + + def _rmdir(self, path: str) -> None: + self._inner._rmdir(path) + + +@pytest.fixture(name="resolver") +def _resolver() -> StorageResolver: + resolver = StorageResolver(defaults=False) + resolver.mount(ROOT, MemoryStorage()) + resolver.mount("remote://far", _Remote()) + return resolver + + +@pytest.fixture(name="files") +def _files(resolver: StorageResolver) -> FileService: + return FileService(resolver) + + +def _write(resolver: StorageResolver, uri: str, data: bytes | str) -> None: + File(uri, resolver=resolver).write(data) + + +# ---------------------------------------------------------------------- files + + +def test_listing_puts_directories_first_and_carries_the_uri( + files: FileService, resolver: StorageResolver +) -> None: + _write(resolver, f"{ROOT}/b.txt", "bb") + _write(resolver, f"{ROOT}/A.txt", "a") + files.mkdir(f"{ROOT}/zeta") + entries = files.list_dir(ROOT) + assert [(entry.name, entry.is_dir) for entry in entries] == [ + ("zeta", True), + ("A.txt", False), + ("b.txt", False), + ] + assert entries[1].uri == f"{ROOT}/A.txt" + assert entries[2].size == 2 + assert entries[1].modified_at is not None + assert entries[0].to_dict()["is_dir"] is True + + +def test_a_recursive_listing_reports_paths_below_the_directory( + files: FileService, resolver: StorageResolver +) -> None: + _write(resolver, f"{ROOT}/dir/inner/a.txt", "a") + paths = [entry.path for entry in files.list_dir(ROOT, recursive=True)] + assert paths == ["dir", "dir/inner", "dir/inner/a.txt"] + assert files.list_dir(f"{ROOT}/dir", recursive=True)[-1].uri == f"{ROOT}/dir/inner/a.txt" + + +def test_stat_exists_and_the_uri_helpers(files: FileService, resolver: StorageResolver) -> None: + _write(resolver, f"{ROOT}/dir/a.txt", "hello") + entry = files.stat(f"{ROOT}/dir/a.txt") + assert (entry.name, entry.size, entry.is_dir) == ("a.txt", 5, False) + assert files.exists(f"{ROOT}/dir") is True + assert files.exists(f"{ROOT}/missing") is False + assert files.parent(f"{ROOT}/dir/a.txt") == f"{ROOT}/dir" + assert files.parent(ROOT) == ROOT + assert files.child(f"{ROOT}/dir", "b.txt") == f"{ROOT}/dir/b.txt" + assert files.normalize("MEMORY://files//dir/./a.txt") == f"{ROOT}/dir/a.txt" + with pytest.raises(StorageNotFoundException): + files.stat(f"{ROOT}/missing") + + +def test_a_path_cannot_climb_out_and_a_uri_cannot_carry_credentials(files: FileService) -> None: + with pytest.raises(StorageURIException): + files.list_dir(f"{ROOT}/../other") + with pytest.raises(StorageURIException): + files.child(ROOT, "../x") + with pytest.raises(StorageURIException) as caught: + files.stat("memory://user:hunter2@files/a.txt") + assert "hunter2" not in str(caught.value) + + +def test_a_preview_is_bounded(files: FileService, resolver: StorageResolver) -> None: + _write(resolver, f"{ROOT}/big.txt", "x" * 500) + preview = files.preview(f"{ROOT}/big.txt", max_bytes=100) + assert (preview.shown, preview.truncated, preview.binary) == (100, True, False) + assert preview.text == "x" * 100 + assert preview.size == 500 + whole = files.preview(f"{ROOT}/big.txt") + assert (whole.shown, whole.truncated) == (500, False) + assert whole.to_dict()["uri"] == f"{ROOT}/big.txt" + + +def test_a_preview_cut_inside_a_character_is_still_text( + files: FileService, resolver: StorageResolver +) -> None: + _write(resolver, f"{ROOT}/utf8.txt", "é" * 10) + preview = files.preview(f"{ROOT}/utf8.txt", max_bytes=5) + assert preview.binary is False + assert preview.text == "éé" + + +def test_binary_content_is_shown_as_hexadecimal( + files: FileService, resolver: StorageResolver +) -> None: + _write(resolver, f"{ROOT}/blob.bin", bytes(range(256)) * 4) + preview = files.preview(f"{ROOT}/blob.bin") + assert preview.binary is True + assert preview.text.startswith("00 01 02 03") + assert (preview.shown, preview.truncated) == (256, True) + assert "hexadecimal" in preview.note + + +def test_a_large_remote_file_is_not_fetched_for_a_preview(resolver: StorageResolver) -> None: + files = FileService(resolver, preview_bytes=10, fetch_limit=100) + _write(resolver, "remote://far/huge.txt", "y" * 101) + _write(resolver, "remote://far/small.txt", "z" * 50) + _write(resolver, f"{ROOT}/huge.txt", "y" * 101) + refused = files.preview("remote://far/huge.txt") + assert (refused.text, refused.shown, refused.truncated) == ("", 0, True) + assert "not fetched" in refused.note + assert files.preview("remote://far/small.txt").text == "z" * 10 + assert files.preview(f"{ROOT}/huge.txt").text == "y" * 10 + + +def test_a_directory_has_no_preview_and_the_limits_must_be_positive( + files: FileService, resolver: StorageResolver +) -> None: + files.mkdir(f"{ROOT}/dir") + with pytest.raises(AppException, match="is a directory"): + files.preview(f"{ROOT}/dir") + with pytest.raises(AppException, match="at least one byte"): + files.preview(f"{ROOT}/dir", max_bytes=0) + with pytest.raises(AppException): + FileService(resolver, preview_bytes=0) + + +def test_copy_to_a_file_to_a_directory_and_of_a_tree( + files: FileService, resolver: StorageResolver +) -> None: + _write(resolver, f"{ROOT}/src/a.txt", "a") + _write(resolver, f"{ROOT}/src/sub/b.txt", "b") + assert files.copy(f"{ROOT}/src/a.txt", f"{ROOT}/copy.txt").uri == f"{ROOT}/copy.txt" + files.mkdir(f"{ROOT}/inbox") + assert files.copy(f"{ROOT}/src/a.txt", f"{ROOT}/inbox").uri == f"{ROOT}/inbox/a.txt" + tree = files.copy(f"{ROOT}/src", "remote://far/mirror") + assert tree.is_dir is True + assert File("remote://far/mirror/sub/b.txt", resolver=resolver).read_text() == "b" + assert files.exists(f"{ROOT}/src/a.txt") is True + + +def test_move_a_file_and_refuse_a_directory(files: FileService, resolver: StorageResolver) -> None: + _write(resolver, f"{ROOT}/a.txt", "a") + files.mkdir(f"{ROOT}/done") + moved = files.move(f"{ROOT}/a.txt", f"{ROOT}/done") + assert moved.uri == f"{ROOT}/done/a.txt" + assert files.exists(f"{ROOT}/a.txt") is False + with pytest.raises(AppException, match="copy it and delete the original"): + files.move(f"{ROOT}/done", f"{ROOT}/elsewhere") + + +def test_delete_needs_recursive_for_a_directory_with_entries( + files: FileService, resolver: StorageResolver +) -> None: + _write(resolver, f"{ROOT}/dir/a.txt", "a") + with pytest.raises(StorageNotEmptyException): + files.delete(f"{ROOT}/dir") + assert files.delete(f"{ROOT}/dir", recursive=True) is True + assert files.exists(f"{ROOT}/dir") is False + with pytest.raises(StorageNotFoundException): + files.delete(f"{ROOT}/dir") + + +def test_a_mounted_local_directory_cannot_be_left(tmp_path: Path) -> None: + resolver = StorageResolver(defaults=False) + inside = tmp_path / "inside" + inside.mkdir() + (tmp_path / "secret.txt").write_text("outside", encoding="utf-8") + (inside / "a.txt").write_text("in", encoding="utf-8") + StorageService(resolver).mount_local("sandbox://jobs", inside) + files = FileService(resolver) + assert files.preview("sandbox://jobs/a.txt").text == "in" + with pytest.raises(StorageURIException): + files.preview("sandbox://jobs/../secret.txt") + assert [entry.name for entry in files.list_dir("sandbox://jobs")] == ["a.txt"] + + +# ---------------------------------------------------------------------- storage + + +def test_schemes_and_mounts_of_a_resolver(resolver: StorageResolver) -> None: + storage = StorageService(resolver) + assert storage.resolver is resolver + assert storage.schemes() == ["memory", "remote"] + assert [mount.to_dict() for mount in storage.mounts()] == [ + {"uri": "memory://files", "scheme": "memory", "backend": "MemoryStorage"}, + {"uri": "remote://far", "scheme": "remote", "backend": "_Remote"}, + ] + assert storage.capabilities(ROOT) == { + "directories": True, + "modified_at": True, + "etag": False, + "version": False, + "content_type": False, + "metadata": False, + } + + +def test_mounting_and_unmounting(tmp_path: Path) -> None: + storage = StorageService(StorageResolver(defaults=False)) + mount = storage.mount_local("sandbox://jobs/in", tmp_path) + assert (mount.uri, mount.backend) == ("sandbox://jobs/in", "LocalStorage") + assert storage.mount("memory://scratch", MemoryStorage()).scheme == "memory" + assert [status.name for status in storage.backends() if status.kind == "mount"] == [ + "memory://scratch", + "sandbox://jobs/in", + ] + assert storage.unmount("sandbox://jobs/in") is True + assert storage.unmount("sandbox://jobs/in") is False + with pytest.raises(AppException, match="not a directory"): + storage.mount_local("sandbox://jobs", tmp_path / "missing") + with pytest.raises(AppException, match="not a StorageBackend"): + storage.mount("memory://bad", object()) # type: ignore[arg-type] + + +def test_a_local_mount_is_confined_to_its_directory(tmp_path: Path) -> None: + resolver = StorageResolver(defaults=False) + StorageService(resolver).mount_local("sandbox://jobs", tmp_path) + backend, _path = resolver.resolve("sandbox://jobs/a.txt") + assert isinstance(backend, LocalStorage) + with pytest.raises((PathTraversalException, StorageURIException)): + backend.stat("../outside.txt") + + +def _status(statuses: list[BackendStatus], name: str) -> BackendStatus: + return next(status for status in statuses if status.name == name) + + +def test_the_default_resolver_lists_every_built_in_scheme_and_the_box_client() -> None: + statuses = StorageService().backends() + names = [status.name for status in statuses] + for scheme in ("local", "memory", "s3", "azure", "gdrive", "dropbox", "onedrive", "sftp"): + assert scheme in names + assert {"ftp", "ftps", "box"} <= set(names) + local = _status(statuses, "local") + assert (local.kind, local.usable, local.detail, local.install_hint) == ( + "scheme", + True, + "ready", + None, + ) + assert _status(statuses, "memory").usable is True + box = _status(statuses, "box") + assert (box.kind, box.extra) == ("client", "box") + + +def test_an_uninitialised_client_says_how_to_initialise_it( + monkeypatch: pytest.MonkeyPatch, +) -> None: + monkeypatch.setattr(storage_module, "is_installed", lambda _module: True) + s3 = _status(StorageService().backends(), "s3") + assert (s3.installed, s3.ready, s3.usable) == (True, False, False) + assert s3.install_hint is None + assert "not initialised" in s3.detail + assert "s3_instance.later_init" in s3.detail + assert s3.label == "Amazon S3" + + +def test_a_missing_extra_is_reported_with_its_install_command( + monkeypatch: pytest.MonkeyPatch, +) -> None: + monkeypatch.setattr(storage_module, "is_installed", lambda module: module is None) + statuses = StorageService().backends() + s3 = _status(statuses, "s3") + assert (s3.installed, s3.usable) == (False, False) + assert s3.install_hint == install_hint("s3") + assert install_hint("s3") in s3.detail + assert "boto3 is not installed" in s3.detail + ftp = _status(statuses, "ftp") + assert (ftp.installed, ftp.install_hint) == (True, None) + assert _status(statuses, "local").usable is True + assert s3.to_dict()["install_hint"] == 'pip install "automation_file[s3]"' + + +def test_a_client_that_is_ready_makes_its_scheme_usable(monkeypatch: pytest.MonkeyPatch) -> None: + from automation_file.remote.s3.client import s3_instance + + monkeypatch.setattr(s3_instance, "client", object()) + s3 = _status(StorageService().backends(), "s3") + assert (s3.ready, s3.usable, s3.detail) == (True, True, "ready") + + +def test_is_installed_does_not_import_and_tolerates_a_missing_parent() -> None: + assert storage_module.is_installed(None) is True + assert storage_module.is_installed("json") is True + assert storage_module.is_installed("no_such_package_for_fa.sub.module") is False + assert storage_module.is_installed("no_such_package_for_fa") is False + + +def test_a_scheme_registered_by_the_application_is_listed_as_such() -> None: + resolver = StorageResolver(defaults=False) + store = MemoryStorage() + resolver.register_scheme("vault", lambda uri: (store, uri.path)) + statuses = StorageService(resolver).backends() + vault = _status(statuses, "vault") + assert (vault.kind, vault.usable) == ("scheme", True) + assert "application" in vault.detail diff --git a/tests/test_app_masking.py b/tests/test_app_masking.py new file mode 100644 index 0000000..2f7b52f --- /dev/null +++ b/tests/test_app_masking.py @@ -0,0 +1,185 @@ +"""The application layer keeps secrets out of what it returns, and reads form text.""" + +from __future__ import annotations + +import pytest + +from automation_file.app import ( + MASK, + AppException, + describe_action, + format_argument_value, + mask_secrets, + mask_text, + mask_url, + parse_argument_text, + parse_json_text, +) +from automation_file.app.masking import is_secret_name, is_url_name +from automation_file.exceptions import FileAutomationException + + +@pytest.mark.parametrize( + "name", + [ + "password", + "PASSWORD", + "smtp_password", + "token", + "access_token", + "token_path", + "api_key", + "X-Api-Key", + "Authorization", + "client_secret", + "connection_string", + "aws_secret_access_key", + "private_key", + ], +) +def test_names_that_hold_a_secret(name: str) -> None: + assert is_secret_name(name) is True + + +@pytest.mark.parametrize( + "name", ["name", "uri", "target", "idempotency_key", "status", "task", 3, None] +) +def test_names_that_do_not_hold_a_secret(name: object) -> None: + assert is_secret_name(name) is False + + +def test_url_names_are_told_from_storage_uris() -> None: + assert is_url_name("url") is True + assert is_url_name("webhook_url") is True + assert is_url_name("webhook") is True + assert is_url_name("uri") is False + assert is_url_name("curl_options") is False + assert is_url_name(None) is False + + +def test_a_secret_value_is_replaced_whatever_its_type() -> None: + masked = mask_secrets( + {"password": "hunter2", "token": {"value": "abc"}, "api_key": 12, "name": "ops"} + ) + assert masked == {"password": MASK, "token": MASK, "api_key": MASK, "name": "ops"} + + +def test_an_empty_secret_stays_empty_so_a_view_can_tell_it_is_unset() -> None: + assert mask_secrets({"password": "", "token": None}) == {"password": "", "token": None} + + +def test_a_webhook_url_keeps_only_its_host() -> None: + masked = mask_secrets({"webhook_url": "https://hooks.example.com/services/T0/B0/s3cr3t"}) + assert masked == {"webhook_url": f"https://hooks.example.com/{MASK}"} + assert "s3cr3t" not in str(masked) + + +def test_a_url_field_without_an_http_url_is_masked_whole() -> None: + assert mask_url("not a url") == MASK + assert mask_secrets({"url": "token-only"}) == {"url": MASK} + + +def test_nested_values_are_walked_and_tuples_become_lists() -> None: + masked = mask_secrets({"sinks": ({"name": "a", "password": "x"}, {"name": "b"})}) + assert masked == {"sinks": [{"name": "a", "password": MASK}, {"name": "b"}]} + + +def test_credentials_inside_ordinary_text_are_removed() -> None: + text = mask_text("fetch https://user:pw@example.com/x with Bearer abc.def-123") + assert "user:pw" not in text + assert "abc.def-123" not in text + assert f"https://{MASK}@example.com/x" in text + assert f"Bearer {MASK}" in text + + +def test_a_storage_uri_is_left_alone() -> None: + record = {"resource": "s3://reports/2026/q1.csv", "uri": "local:///C:/data/a.csv"} + assert mask_secrets(record) == record + + +def test_the_input_is_not_changed() -> None: + original = {"password": "hunter2", "items": [{"token": "t"}]} + mask_secrets(original) + assert original == {"password": "hunter2", "items": [{"token": "t"}]} + + +def test_values_that_are_not_containers_pass_through() -> None: + assert mask_secrets(12) == 12 + assert mask_secrets(None) is None + + +@pytest.mark.parametrize( + ("text", "value"), + [ + ("12", 12), + ("1.5", 1.5), + ("true", True), + ("null", None), + ('"12"', "12"), + ('["a", 1]', ["a", 1]), + ('{"a": 1}', {"a": 1}), + ("s3://bucket/key", "s3://bucket/key"), + ("${params.date}", "${params.date}"), + (" padded ", "padded"), + ("NaN", "NaN"), + ("", ""), + ], +) +def test_form_text_is_json_when_it_parses_and_text_otherwise(text: str, value: object) -> None: + assert parse_argument_text(text) == value + + +@pytest.mark.parametrize( + "value", + [12, 1.5, True, None, "12", "true", "plain text", "s3://b/k", "", " padded ", ["a"], {"a": 1}], +) +def test_a_formatted_value_reads_back_unchanged(value: object) -> None: + assert parse_argument_text(format_argument_value(value)) == value + + +def test_a_plain_string_is_shown_without_quotes() -> None: + assert format_argument_value("s3://bucket/key") == "s3://bucket/key" + assert format_argument_value("12") == '"12"' + + +def test_a_value_json_cannot_hold_is_shown_as_its_repr() -> None: + assert format_argument_value(set) == repr(set) + + +def test_json_text_of_a_form() -> None: + assert parse_json_text('{"a": 1}', "params") == {"a": 1} + assert parse_json_text(" ", "params", empty={}) == {} + with pytest.raises(AppException, match="params is not valid JSON") as caught: + parse_json_text("{oops", "params") + assert "line 1" in str(caught.value) + assert isinstance(caught.value, FileAutomationException) + with pytest.raises(AppException, match="not valid JSON"): + parse_json_text("NaN", "params") + + +def _sample(source: str, target: str, overwrite: bool = True, *extra: str, **options: int) -> None: + """Copy something. + + More text. + """ + + +def test_an_action_is_described_for_a_form() -> None: + info = describe_action("FA_sample", _sample) + assert info.known is True + assert info.signature == "FA_sample(source, target, overwrite=True, *extra, **options)" + assert info.summary == "Copy something." + assert [(entry.name, entry.required, entry.default) for entry in info.parameters] == [ + ("source", True, ""), + ("target", True, ""), + ("overwrite", False, "true"), + ] + assert info.accepts_extra is True + assert info.to_dict()["parameters"][0] == {"name": "source", "required": True, "default": ""} + + +def test_an_unknown_action_and_a_builtin_are_described_without_parameters() -> None: + unknown = describe_action("FA_missing", None) + assert (unknown.known, unknown.parameters, unknown.signature) == (False, (), "") + builtin = describe_action("FA_dict", dict) + assert builtin.known is True diff --git a/tests/test_app_pipeline_draft.py b/tests/test_app_pipeline_draft.py new file mode 100644 index 0000000..1cd94aa --- /dev/null +++ b/tests/test_app_pipeline_draft.py @@ -0,0 +1,472 @@ +"""The editable pipeline draft of the application layer.""" + +from __future__ import annotations + +import json + +import pytest + +from automation_file.app import AppException, DraftTask, PipelineDraft, Problem +from automation_file.app.pipeline_draft import ( + CHANGE_HEADER, + CHANGE_POSITION, + CHANGE_STRUCTURE, + CHANGE_TASK, +) +from automation_file.pipeline import Pipeline, validate_definition + +COPY = "FA_storage_copy" +DELETE = "FA_storage_delete" + + +def _three_tasks() -> PipelineDraft: + draft = PipelineDraft("nightly") + draft.add_task(COPY, "download", arguments={"source": "memory://a/x", "target": "memory://b/x"}) + draft.add_task(COPY, "publish", arguments={"source": "memory://b/x", "target": "memory://c/x"}) + draft.add_task(DELETE, "tidy", arguments={"uri": "memory://b/x"}) + draft.connect("download", "publish") + draft.connect("publish", "tidy") + return draft + + +# ---------------------------------------------------------------------- tasks + + +def test_a_new_draft_is_empty_and_says_what_is_missing() -> None: + draft = PipelineDraft() + assert (draft.name, draft.max_workers, draft.tasks, draft.dirty) == ("pipeline", 4, (), False) + assert [str(problem) for problem in draft.problems()] == [ + "tasks: at least one task is required" + ] + + +def test_adding_a_task_derives_an_unused_id_from_the_action() -> None: + draft = PipelineDraft() + first = draft.add_task(COPY) + second = draft.add_task(COPY) + third = draft.add_task(COPY) + blank = draft.add_task() + assert [first.task_id, second.task_id, third.task_id, blank.task_id] == [ + "storage_copy", + "storage_copy_2", + "storage_copy_3", + "task", + ] + assert draft.task_ids() == ("storage_copy", "storage_copy_2", "storage_copy_3", "task") + assert draft.has_task("task") is True + assert isinstance(draft.task("task"), DraftTask) + + +def test_a_task_id_must_be_valid_and_unused() -> None: + draft = PipelineDraft() + draft.add_task(COPY, "a") + with pytest.raises(AppException, match="already has a task 'a'"): + draft.add_task(COPY, "a") + with pytest.raises(AppException, match="invalid task ID"): + draft.add_task(COPY, "has space") + with pytest.raises(AppException, match="invalid task ID"): + draft.add_task(COPY, "a.b") + with pytest.raises(AppException, match="no task 'missing'"): + draft.task("missing") + + +def test_a_new_task_lands_below_the_lowest_one_unless_a_position_is_given() -> None: + draft = PipelineDraft() + first = draft.add_task(COPY, "a") + second = draft.add_task(COPY, "b") + third = draft.add_task(COPY, "c", position=(300, 12.5)) + assert (first.x, first.y) == (40.0, 40.0) + assert (second.x, second.y) == (40.0, 150.0) + assert (third.x, third.y) == (300.0, 12.5) + + +def test_removing_a_task_removes_the_edges_to_it() -> None: + draft = _three_tasks() + draft.remove_task("publish") + assert draft.task_ids() == ("download", "tidy") + assert draft.task("tidy").depends_on == [] + assert draft.edges() == [] + with pytest.raises(AppException): + draft.remove_task("publish") + + +def test_renaming_keeps_the_order_the_edges_and_the_placeholders() -> None: + draft = _three_tasks() + draft.set_arguments( + "tidy", + { + "uri": "${tasks.publish.result}", + "note": ["${tasks.publish.result}", {"deep": "${tasks.publish.result}"}], + "other": "${tasks.download.result}", + }, + ) + renamed = draft.rename_task("publish", "upload") + assert renamed.task_id == "upload" + assert draft.task_ids() == ("download", "upload", "tidy") + assert draft.edges() == [("download", "upload"), ("upload", "tidy")] + assert draft.task("tidy").arguments == { + "uri": "${tasks.upload.result}", + "note": ["${tasks.upload.result}", {"deep": "${tasks.upload.result}"}], + "other": "${tasks.download.result}", + } + assert draft.problems() == [] + assert draft.rename_task("upload", "upload").task_id == "upload" + with pytest.raises(AppException, match="already has a task"): + draft.rename_task("upload", "tidy") + with pytest.raises(AppException, match="invalid task ID"): + draft.rename_task("upload", "") + + +def test_the_action_and_its_arguments() -> None: + draft = PipelineDraft() + draft.add_task(COPY, "a") + assert draft.task("a").to_spec() == {"action": [COPY]} + draft.set_action("a", f" {DELETE} ") + draft.set_arguments("a", {"uri": "memory://x/a"}) + assert draft.task("a").to_spec() == {"action": [DELETE, {"uri": "memory://x/a"}]} + draft.set_arguments("a", ["memory://x/a", True]) + assert draft.task("a").to_spec() == {"action": [DELETE, ["memory://x/a", True]]} + draft.set_arguments("a", None) + assert draft.task("a").arguments is None + with pytest.raises(AppException, match="mapping, a list or nothing"): + draft.set_arguments("a", "text") # type: ignore[arg-type] + + +def test_arguments_are_copied_in_and_out() -> None: + draft = PipelineDraft() + given = {"paths": ["a"]} + draft.add_task(COPY, "a", arguments=given) + given["paths"].append("b") + assert draft.task("a").arguments == {"paths": ["a"]} + spec = draft.to_definition()["tasks"]["a"] + spec["action"][1]["paths"].append("c") + assert draft.task("a").arguments == {"paths": ["a"]} + + +def test_retry_timeout_condition_and_idempotency_key() -> None: + draft = PipelineDraft() + draft.add_task(COPY, "a") + draft.set_retry("a", max_attempts=3, backoff=1.5, on=["ConnectionError"]) + draft.set_timeout("a", 30) + draft.set_condition("a", "always") + draft.set_idempotency_key("a", "copy-${params.date}") + assert draft.task("a").to_spec() == { + "action": [COPY], + "retry": {"max_attempts": 3, "backoff": 1.5, "on": ["ConnectionError"]}, + "timeout": 30, + "when": "always", + "idempotency_key": "copy-${params.date}", + } + draft.set_retry("a", backoff_cap=10) + assert draft.task("a").retry == {"max_attempts": 1, "backoff_cap": 10} + draft.set_retry("a") + draft.set_timeout("a", None) + draft.set_condition("a", "on_success") + draft.set_idempotency_key("a", "") + assert draft.task("a").to_spec() == {"action": [COPY]} + with pytest.raises(AppException, match="when must be one of"): + draft.set_condition("a", "sometimes") + + +def test_a_wrong_value_is_kept_and_reported_with_its_path() -> None: + draft = PipelineDraft() + draft.add_task(COPY, "a") + draft.set_timeout("a", -1) + draft.set_retry("a", max_attempts=0) + found = {problem.path: problem for problem in draft.problems()} + assert found["tasks.a.timeout"].task == "a" + assert "seconds > 0" in found["tasks.a.timeout"].message + assert found["tasks.a.retry.max_attempts"].task == "a" + + +# ---------------------------------------------------------------------- edges + + +def test_connecting_and_disconnecting() -> None: + draft = _three_tasks() + assert draft.edges() == [("download", "publish"), ("publish", "tidy")] + assert draft.connect("download", "tidy") is True + assert draft.connect("download", "tidy") is False + assert draft.task("tidy").depends_on == ["publish", "download"] + assert draft.disconnect("download", "tidy") is True + assert draft.disconnect("download", "tidy") is False + assert draft.to_definition()["tasks"]["tidy"]["depends_on"] == ["publish"] + + +def test_an_edge_to_itself_a_cycle_and_an_unknown_task_are_refused() -> None: + draft = _three_tasks() + with pytest.raises(AppException, match="cannot depend on itself"): + draft.connect("tidy", "tidy") + with pytest.raises(AppException, match="dependency cycle"): + draft.connect("tidy", "download") + with pytest.raises(AppException, match="no task 'ghost'"): + draft.connect("ghost", "tidy") + with pytest.raises(AppException, match="no task 'ghost'"): + draft.connect("tidy", "ghost") + assert draft.edges() == [("download", "publish"), ("publish", "tidy")] + + +def test_the_whole_dependency_list_of_a_task_can_be_replaced() -> None: + draft = _three_tasks() + changes: list[str] = [] + draft.add_listener(changes.append) + draft.set_dependencies("tidy", ["download", "download"]) + assert draft.task("tidy").depends_on == ["download"] + assert changes == [CHANGE_STRUCTURE] + with pytest.raises(AppException, match="dependency cycle"): + draft.set_dependencies("download", ["tidy"]) + + +def test_an_edge_to_a_missing_task_is_not_an_edge_but_is_a_problem() -> None: + draft = PipelineDraft.from_definition( + { + "schema_version": 1, + "name": "p", + "tasks": {"a": {"action": [COPY], "depends_on": ["ghost"]}}, + } + ) + assert draft.edges() == [] + assert [str(problem) for problem in draft.problems()] == [ + "tasks.a.depends_on[0]: unknown task 'ghost'" + ] + + +# ---------------------------------------------------------------------- layout + + +def test_positions_are_editor_metadata_and_never_reach_the_definition() -> None: + draft = _three_tasks() + draft.set_position("download", 512.5, 77) + assert draft.positions()["download"] == (512.5, 77.0) + definition = draft.to_definition() + text = json.dumps(definition) + assert "512.5" not in text + assert "position" not in text + assert "layout" not in text + assert validate_definition(definition) == [] + assert Pipeline.from_dict(definition).name == "nightly" + assert draft.task("download").to_dict()["position"] == [512.5, 77.0] + + +def test_the_layout_round_trips_apart_from_the_definition() -> None: + draft = _three_tasks() + draft.set_position("tidy", 9, 8) + layout = draft.layout() + assert layout["layout_version"] == 1 + assert layout["pipeline"] == "nightly" + assert layout["positions"]["tidy"] == [9.0, 8.0] + again = PipelineDraft.from_definition(draft.to_definition(), json.loads(json.dumps(layout))) + assert again.positions() == draft.positions() + + +def test_a_layout_with_wrong_entries_places_what_it_can() -> None: + draft = _three_tasks() + placed = draft.apply_layout( + {"positions": {"tidy": [1, 2], "ghost": [3, 4], "publish": "no", "download": [True, 1]}} + ) + assert placed == 1 + assert draft.positions()["tidy"] == (1.0, 2.0) + with pytest.raises(AppException, match="no 'positions' mapping"): + draft.apply_layout({"positions": []}) + with pytest.raises(AppException, match="no 'positions' mapping"): + draft.apply_layout("nonsense") + + +def test_the_automatic_layout_puts_each_dependency_depth_in_a_column() -> None: + draft = _three_tasks() + draft.add_task(DELETE, "side") + draft.auto_layout() + positions = draft.positions() + assert positions["download"] == (40.0, 40.0) + assert positions["side"] == (40.0, 150.0) + assert positions["publish"] == (280.0, 40.0) + assert positions["tidy"] == (520.0, 40.0) + + +def test_the_automatic_layout_survives_a_cycle() -> None: + draft = PipelineDraft.from_definition( + { + "schema_version": 1, + "name": "loop", + "tasks": { + "a": {"action": [COPY], "depends_on": ["b"]}, + "b": {"action": [COPY], "depends_on": ["a"]}, + }, + } + ) + assert set(draft.positions()) == {"a", "b"} + assert any("dependency cycle" in str(problem) for problem in draft.problems()) + + +# ---------------------------------------------------------------------- header, listeners + + +def test_the_header_fields() -> None: + draft = PipelineDraft() + draft.set_name(" daily-report ") + draft.set_description("Fetch and publish") + draft.set_max_workers(2) + draft.set_params({"date": "2026-10-08"}) + draft.set_schedule("0 2 * * *", "Asia/Taipei") + draft.add_task(COPY, "a") + assert draft.to_definition() == { + "schema_version": 1, + "name": "daily-report", + "description": "Fetch and publish", + "max_workers": 2, + "schedule": {"cron": "0 2 * * *", "timezone": "Asia/Taipei"}, + "params": {"date": "2026-10-08"}, + "tasks": {"a": {"action": [COPY]}}, + } + assert draft.problems() == [] + draft.set_schedule(None) + draft.set_params(None) + assert draft.schedule is None + assert draft.params == {} + with pytest.raises(AppException, match="max_workers"): + draft.set_max_workers(0) + with pytest.raises(AppException, match="params must be a mapping"): + draft.set_params(["a"]) # type: ignore[arg-type] + + +def test_params_and_schedule_are_handed_out_as_copies() -> None: + draft = PipelineDraft() + draft.set_params({"list": [1]}) + draft.set_schedule("* * * * *") + draft.params["list"].append(2) + schedule = draft.schedule + assert schedule is not None + schedule["cron"] = "changed" + assert draft.params == {"list": [1]} + assert draft.schedule == {"cron": "* * * * *"} + + +def test_listeners_hear_every_change_with_its_kind() -> None: + draft = PipelineDraft() + changes: list[str] = [] + draft.add_listener(changes.append) + draft.add_listener(changes.append) + draft.add_task(COPY, "a") + draft.add_task(COPY, "b") + draft.connect("a", "b") + draft.set_action("a", DELETE) + draft.set_position("a", 5, 5) + draft.set_position("a", 5, 5) + draft.set_name("x") + assert changes == [ + CHANGE_STRUCTURE, + CHANGE_STRUCTURE, + CHANGE_STRUCTURE, + CHANGE_TASK, + CHANGE_POSITION, + CHANGE_HEADER, + ] + draft.remove_listener(changes.append) + draft.set_name("y") + assert len(changes) == 6 + + +def test_a_batch_reports_each_kind_of_change_once_at_its_end() -> None: + draft = PipelineDraft() + draft.add_task(COPY, "a") + changes: list[str] = [] + draft.add_listener(changes.append) + with draft.batch(): + draft.set_action("a", DELETE) + draft.set_timeout("a", 5) + with draft.batch(): + draft.set_name("x") + assert changes == [] + assert changes == [CHANGE_TASK, CHANGE_HEADER] + + +def test_the_revision_grows_and_dirty_follows_the_saved_state() -> None: + draft = PipelineDraft() + assert draft.revision == 0 + draft.add_task(COPY, "a") + assert (draft.revision, draft.dirty) == (1, True) + draft.mark_saved() + assert draft.dirty is False + draft.set_position("a", 1, 1) + assert draft.dirty is True + + +# ---------------------------------------------------------------------- documents + + +def test_a_definition_round_trips_through_a_draft() -> None: + definition = { + "schema_version": 1, + "name": "daily-report", + "description": "Fetch and publish", + "max_workers": 3, + "schedule": {"cron": "0 2 * * *"}, + "params": {"date": "2026-10-08"}, + "tasks": { + "download": { + "action": [COPY, {"source": "s3://in/${params.date}.csv", "target": "local:///x"}], + "retry": {"max_attempts": 5, "backoff": 2, "on": ["ConnectionError"]}, + "timeout": 300, + }, + "withdraw": { + "action": [DELETE, ["azure://reports/x", False, True]], + "depends_on": ["download"], + "when": "on_failure", + "idempotency_key": "withdraw-${params.date}", + }, + "bare": {"action": ["FA_storage_schemes"]}, + }, + } + draft = PipelineDraft.from_definition(definition) + assert draft.load_notes == () + assert draft.dirty is False + assert draft.to_definition() == definition + assert draft.task("bare").arguments is None + + +def test_a_broken_definition_still_opens_and_keeps_what_validation_said() -> None: + draft = PipelineDraft.from_definition( + { + "schema_version": 1, + "name": "", + "max_workers": 0, + "params": "nope", + "tasks": { + "ok": {"action": [COPY, "not arguments"], "depends_on": "ok", "timeout": "soon"}, + "bad id": {"action": [COPY]}, + "no-mapping": 3, + "empty": {"action": []}, + }, + } + ) + assert draft.name == "pipeline" + assert draft.max_workers == 4 + assert draft.task_ids() == ("ok", "empty") + assert draft.task("ok").arguments is None + assert draft.task("ok").timeout is None + assert draft.task("empty").action == "" + assert any(note.startswith("max_workers:") for note in draft.load_notes) + assert any("invalid task ID" in note for note in draft.load_notes) + assert [problem.path for problem in draft.problems()] == ["tasks.empty.action[0]"] + + +def test_a_definition_must_be_a_mapping() -> None: + with pytest.raises(AppException, match="is a mapping, got list"): + PipelineDraft.from_definition([]) + + +def test_a_problem_knows_its_task_and_its_text() -> None: + ids = ("a", "a[0]", "verify") + assert Problem.parse("tasks.verify.depends_on[0]: unknown task 'x'", ids) == Problem( + "tasks.verify.depends_on[0]", "unknown task 'x'", "verify" + ) + assert Problem.parse("tasks.a[0].action: wrong", ids).task == "a[0]" + assert Problem.parse("tasks.a: duplicate task ID", ids).task == "a" + assert Problem.parse("tasks.gone.action: wrong", ids).task == "gone" + assert Problem.parse("tasks: dependency cycle: a -> b -> a", ids).task is None + assert Problem.parse("max_workers: expected an integer", ids).task is None + plain = Problem.parse("no path at all") + assert (plain.path, plain.message, str(plain)) == ("", "no path at all", "no path at all") + problem = Problem.parse("tasks.a.timeout: wrong", ids) + assert str(problem) == "tasks.a.timeout: wrong" + assert problem.to_dict() == {"path": "tasks.a.timeout", "message": "wrong", "task": "a"} diff --git a/tests/test_app_pipelines.py b/tests/test_app_pipelines.py new file mode 100644 index 0000000..a6fc447 --- /dev/null +++ b/tests/test_app_pipelines.py @@ -0,0 +1,492 @@ +"""The Pipelines service of the application layer, on a private store, registry and bus.""" + +from __future__ import annotations + +import json +import threading +from pathlib import Path +from typing import Any + +import pytest + +from automation_file.app import ( + MASK, + AppException, + PipelineDraft, + PipelineService, + layout_path, +) +from automation_file.core.action_registry import ActionRegistry +from automation_file.events import EventBus +from automation_file.pipeline import ( + MemoryRunStore, + Pipeline, + PipelineDefinitionException, + PipelineException, + SQLiteRunStore, + default_run_store, + set_default_run_store, +) + +WAIT = 10.0 + + +class _Workshop: + """The actions the pipelines of these tests call.""" + + def __init__(self) -> None: + self.gate = threading.Event() + self.entered = threading.Event() + self.calls: list[str] = [] + self.broken = True + + def echo(self, value: Any = None) -> Any: + self.calls.append(f"echo:{value!r}") + return value + + def fail(self) -> None: + self.calls.append("fail") + raise ValueError("it broke") + + def flaky(self) -> str: + self.calls.append("flaky") + if self.broken: + raise ConnectionError("not yet") + return "repaired" + + def hold(self) -> str: + self.entered.set() + self.gate.wait(WAIT) + return "released" + + def registry(self) -> ActionRegistry: + return ActionRegistry( + {"T_echo": self.echo, "T_fail": self.fail, "T_flaky": self.flaky, "T_hold": self.hold} + ) + + +@pytest.fixture(name="workshop") +def _workshop() -> _Workshop: + return _Workshop() + + +@pytest.fixture(name="bus") +def _bus() -> EventBus: + return EventBus() + + +@pytest.fixture(name="service") +def _service(workshop: _Workshop, bus: EventBus) -> PipelineService: + return PipelineService(MemoryRunStore(), registry=workshop.registry(), bus=bus) + + +def _draft(*tasks: tuple[str, str, Any]) -> PipelineDraft: + """Build a chain: each ``(task ID, action, arguments)`` depends on the one before it.""" + draft = PipelineDraft("chain") + previous: str | None = None + for task_id, action, arguments in tasks: + draft.add_task(action, task_id, arguments=arguments) + if previous is not None: + draft.connect(previous, task_id) + previous = task_id + return draft + + +def _finished(service: PipelineService, run_id: str) -> dict[str, Any]: + assert service.wait(run_id, WAIT) is True + return service.status(run_id) + + +# ---------------------------------------------------------------------- actions + + +def test_the_registered_actions_are_listed_and_described(service: PipelineService) -> None: + assert service.action_names() == ["T_echo", "T_fail", "T_flaky", "T_hold"] + info = service.describe_action("T_echo") + assert (info.known, info.signature) == (True, "T_echo(value=None)") + assert service.describe_action("T_missing").known is False + + +def test_the_shared_registry_is_the_default() -> None: + names = PipelineService(MemoryRunStore()).action_names() + assert "FA_storage_copy" in names + assert names == sorted(names) + + +# ---------------------------------------------------------------------- validation, dry run + + +def test_validation_returns_every_problem_with_its_path(service: PipelineService) -> None: + draft = _draft(("a", "T_echo", {"value": 1}), ("b", "T_missing", None)) + draft.set_timeout("a", 0) + found = {str(problem): problem.task for problem in service.validate(draft)} + assert found == { + "tasks.a.timeout: expected a number of seconds > 0, got 0": "a", + "tasks.b.action[0]: unknown action 'T_missing'": "b", + } + draft.set_timeout("a", None) + draft.set_action("b", "T_echo") + assert service.validate(draft) == [] + assert service.validate(draft.to_definition()) == [] + + +def test_validation_takes_a_document_that_is_not_a_definition(service: PipelineService) -> None: + assert [str(problem) for problem in service.validate({"name": "x"})] == [ + "schema_version: required (supported: 1)" + ] + with pytest.raises(AppException, match="PipelineDraft or a definition mapping"): + service.validate("text") # type: ignore[arg-type] + + +def test_a_dry_run_plans_without_running(service: PipelineService, workshop: _Workshop) -> None: + draft = _draft(("a", "T_echo", {"value": "${params.word}"}), ("b", "T_echo", None)) + plan = service.dry_run(draft, {"word": "hi"}) + assert (plan["status"], plan["dry_run"], plan["active"]) == ("succeeded", True, False) + assert [(task, state["status"], state["level"]) for task, state in plan["tasks"].items()] == [ + ("a", "planned", 0), + ("b", "planned", 1), + ] + missing = service.dry_run(draft) + assert missing["status"] == "failed" + assert "unknown parameter 'word'" in missing["tasks"]["a"]["error"] + assert workshop.calls == [] + assert service.history() == [] + + +def test_a_dry_run_of_an_invalid_definition_raises_with_the_problems( + service: PipelineService, +) -> None: + with pytest.raises(PipelineDefinitionException) as caught: + service.dry_run(PipelineDraft("empty")) + assert caught.value.problems == ("tasks: at least one task is required",) + + +# ---------------------------------------------------------------------- runs + + +def test_a_run_starts_in_the_background_and_is_followed( + service: PipelineService, workshop: _Workshop, bus: EventBus +) -> None: + draft = _draft(("hold", "T_hold", None), ("after", "T_echo", {"value": "${params.word}"})) + started = service.start(draft, {"word": "hi"}) + run_id = started["run_id"] + assert started["active"] is True + assert workshop.entered.wait(WAIT) + live = service.status(run_id) + assert (live["status"], live["active"]) == ("running", True) + assert live["tasks"]["hold"]["status"] == "running" + assert [run["run_id"] for run in service.running()] == [run_id] + workshop.gate.set() + done = _finished(service, run_id) + assert (done["status"], done["active"]) == ("succeeded", False) + assert done["tasks"]["after"]["result"] == "hi" + assert service.running() == [] + followed = service.follow(run_id) + assert followed["run"]["status"] == "succeeded" + assert [event["type"] for event in followed["events"]] == [ + "pipeline.started", + "task.started", + "task.completed", + "task.started", + "task.completed", + "pipeline.completed", + ] + assert {event["correlation_id"] for event in followed["events"]} == {run_id} + assert len(bus.recent(100)) == 6 + + +def test_a_failed_task_does_not_raise_and_shows_in_the_status(service: PipelineService) -> None: + draft = _draft(("bad", "T_fail", None), ("after", "T_echo", None)) + done = _finished(service, service.start(draft)["run_id"]) + assert done["status"] == "failed" + assert done["tasks"]["bad"]["error"] == "ValueError: it broke" + assert (done["tasks"]["after"]["status"], done["tasks"]["after"]["reason"]) == ( + "skipped", + "upstream_failed", + ) + + +def test_starting_an_invalid_definition_raises_before_anything_runs( + service: PipelineService, workshop: _Workshop +) -> None: + with pytest.raises(PipelineDefinitionException): + service.start(_draft(("a", "T_echo", {"value": "${params.absent}"}))) + assert workshop.calls == [] + assert service.history() == [] + + +def test_a_running_run_can_be_cancelled(service: PipelineService, workshop: _Workshop) -> None: + draft = _draft(("hold", "T_hold", None), ("after", "T_echo", None)) + run_id = service.start(draft)["run_id"] + assert workshop.entered.wait(WAIT) + assert service.cancel(run_id) is True + workshop.gate.set() + done = _finished(service, run_id) + assert done["status"] == "cancelled" + assert done["tasks"]["after"]["status"] == "cancelled" + assert service.cancel(run_id) is False + assert service.cancel("no-such-run") is False + assert service.wait("no-such-run", 0.01) is True + + +def test_history_is_newest_first_and_filters_by_pipeline(service: PipelineService) -> None: + first = _draft(("a", "T_echo", None)) + second = _draft(("a", "T_echo", None)) + second.set_name("other") + ids = [ + _finished(service, service.start(first)["run_id"])["run_id"], + _finished(service, service.start(second)["run_id"])["run_id"], + _finished(service, service.start(first)["run_id"])["run_id"], + ] + assert [run["run_id"] for run in service.history()] == list(reversed(ids)) + assert [run["run_id"] for run in service.history("chain")] == [ids[2], ids[0]] + assert [run["run_id"] for run in service.history(limit=1)] == [ids[2]] + assert all(run["active"] is False for run in service.history()) + + +def test_the_status_of_an_unknown_run_raises(service: PipelineService) -> None: + with pytest.raises(PipelineException, match="unknown run 'nope'"): + service.status("nope") + + +def test_a_run_recorded_by_someone_else_is_read_from_the_store(bus: EventBus) -> None: + store = MemoryRunStore() + pipeline = Pipeline("external") + pipeline.task("only", lambda _context: "done") + run = pipeline.run(store=store, bus=bus) + service = PipelineService(store, bus=bus) + status = service.status(run.run_id) + assert (status["status"], status["active"]) == ("succeeded", False) + assert [entry["run_id"] for entry in service.history("external")] == [run.run_id] + assert service.running() == [] + + +def test_a_run_the_store_says_is_running_counts_as_running(bus: EventBus) -> None: + store = MemoryRunStore() + pipeline = Pipeline("interrupted") + pipeline.task("only", lambda _context: None) + run = pipeline.run(dry_run=True) + run.status = type(run.status)("running") + store.save_run(run) + listed = PipelineService(store, bus=bus).running() + assert [(entry["run_id"], entry["active"]) for entry in listed] == [(run.run_id, False)] + + +def test_secrets_in_the_parameters_are_masked_in_every_view(service: PipelineService) -> None: + draft = _draft(("a", "T_echo", {"value": "${params.word}"})) + params = {"word": "hi", "password": "hunter2"} + started = service.start(draft, params) + assert started["params"] == {"word": "hi", "password": MASK} + done = _finished(service, started["run_id"]) + assert done["params"] == {"word": "hi", "password": MASK} + assert "hunter2" not in json.dumps(service.history()) + assert "hunter2" not in json.dumps(service.follow(started["run_id"])) + + +# ---------------------------------------------------------------------- resume and retry + + +def test_resume_keeps_what_succeeded_and_runs_the_rest( + service: PipelineService, workshop: _Workshop +) -> None: + draft = _draft(("first", "T_echo", {"value": "kept"}), ("second", "T_flaky", None)) + failed = _finished(service, service.start(draft)["run_id"]) + assert failed["status"] == "failed" + workshop.broken = False + resumed = service.resume(failed["run_id"], draft) + assert (resumed["run_id"], resumed["active"]) == (failed["run_id"], True) + done = _finished(service, failed["run_id"]) + assert (done["status"], done["active"]) == ("succeeded", False) + assert done["tasks"]["second"]["result"] == "repaired" + assert workshop.calls.count("echo:'kept'") == 1 + assert workshop.calls.count("flaky") == 2 + again = service.resume(failed["run_id"], draft) + assert (again["status"], again["active"]) == ("succeeded", False) + + +def test_resume_refuses_what_cannot_be_resumed( + service: PipelineService, workshop: _Workshop +) -> None: + draft = _draft(("bad", "T_fail", None)) + failed = _finished(service, service.start(draft)["run_id"]) + with pytest.raises(PipelineException, match="unknown run"): + service.resume("nope", draft) + other = _draft(("bad", "T_fail", None)) + other.set_name("other") + with pytest.raises(PipelineException, match="belongs to pipeline 'chain'"): + service.resume(failed["run_id"], other) + needs_param = _draft(("bad", "T_echo", {"value": "${params.absent}"})) + with pytest.raises(PipelineDefinitionException, match="unknown parameter 'absent'"): + service.resume(failed["run_id"], needs_param) + with pytest.raises(PipelineDefinitionException): + service.resume(failed["run_id"], PipelineDraft("chain")) + assert workshop.calls == ["fail"] + + +def test_a_run_that_is_still_executing_cannot_be_resumed( + service: PipelineService, workshop: _Workshop +) -> None: + draft = _draft(("hold", "T_hold", None)) + run_id = service.start(draft)["run_id"] + assert workshop.entered.wait(WAIT) + with pytest.raises(AppException, match="still executing"): + service.resume(run_id, draft) + workshop.gate.set() + assert _finished(service, run_id)["status"] == "succeeded" + + +def test_retry_starts_a_new_run_with_the_same_parameters(service: PipelineService) -> None: + draft = _draft(("a", "T_echo", {"value": "${params.word}"})) + first = _finished(service, service.start(draft, {"word": "again"})["run_id"]) + second = service.retry(first["run_id"], draft) + assert second["run_id"] != first["run_id"] + assert _finished(service, second["run_id"])["tasks"]["a"]["result"] == "again" + with pytest.raises(PipelineException, match="unknown run"): + service.retry("nope", draft) + + +# ---------------------------------------------------------------------- testing one task + + +def test_one_task_is_tested_alone_with_stand_ins_for_its_upstream( + service: PipelineService, workshop: _Workshop, bus: EventBus +) -> None: + draft = _draft( + ("first", "T_fail", None), + ("second", "T_echo", {"value": "${tasks.first.result}"}), + ("third", "T_fail", None), + ) + outcome = service.test_task(draft, "second", results={"first": {"rows": 3}}) + run = outcome["run"] + assert run["status"] == "succeeded" + assert list(run["tasks"]) == ["second"] + assert run["tasks"]["second"]["result"] == {"rows": 3} + assert [event["type"] for event in outcome["events"]][-1] == "pipeline.completed" + assert workshop.calls == ["echo:{'rows': 3}"] + assert service.history() == [] + assert bus.recent(10) == [] + assert service.test_task(draft, "second")["run"]["tasks"]["second"]["result"] is None + + +def test_a_tested_task_that_fails_reports_its_error(service: PipelineService) -> None: + draft = _draft(("bad", "T_fail", None)) + run = service.test_task(draft, "bad")["run"] + assert run["status"] == "failed" + assert run["tasks"]["bad"]["error"] == "ValueError: it broke" + + +def test_a_tested_task_uses_its_retry_policy_and_the_given_parameters( + service: PipelineService, workshop: _Workshop +) -> None: + draft = _draft(("flaky", "T_flaky", None), ("say", "T_echo", {"value": "${params.word}"})) + draft.set_retry("flaky", max_attempts=2, on=["ConnectionError"]) + assert service.test_task(draft, "flaky")["run"]["tasks"]["flaky"]["attempts"] == 2 + assert service.test_task(draft, "say", {"word": "hi"})["run"]["tasks"]["say"]["result"] == "hi" + assert workshop.calls.count("flaky") == 2 + + +def test_a_task_with_problems_or_an_unknown_task_cannot_be_tested( + service: PipelineService, +) -> None: + draft = _draft(("a", "T_missing", None)) + with pytest.raises(PipelineDefinitionException, match="unknown action 'T_missing'"): + service.test_task(draft, "a") + with pytest.raises(AppException, match="no task 'ghost'"): + service.test_task(draft, "ghost") + + +# ---------------------------------------------------------------------- files + + +@pytest.mark.parametrize("suffix", [".yaml", ".yml", ".json"]) +def test_a_draft_is_saved_and_loaded_with_its_layout_next_to_it( + service: PipelineService, tmp_path: Path, suffix: str +) -> None: + draft = _draft(("a", "T_echo", {"value": "${params.word}"}), ("b", "T_flaky", None)) + draft.set_retry("b", max_attempts=3, on=["ConnectionError"]) + draft.set_params({"word": "hi"}) + draft.set_position("b", 321.0, 123.0) + path = tmp_path / f"chain{suffix}" + assert service.save(draft, path) == str(path) + assert draft.dirty is False + sidecar = layout_path(path) + assert sidecar.name == f"chain{suffix}.layout.json" + assert json.loads(sidecar.read_text(encoding="utf-8"))["positions"]["b"] == [321.0, 123.0] + assert "321" not in path.read_text(encoding="utf-8") + assert Pipeline.from_file(path).name == "chain" + loaded = service.load(path) + assert loaded.to_definition() == draft.to_definition() + assert loaded.positions() == draft.positions() + assert (loaded.dirty, loaded.load_notes) == (False, ()) + + +def test_a_definition_without_a_layout_is_laid_out_and_a_broken_layout_is_ignored( + service: PipelineService, tmp_path: Path +) -> None: + path = tmp_path / "plain.json" + path.write_text( + json.dumps( + { + "schema_version": 1, + "name": "plain", + "tasks": { + "a": {"action": ["T_echo"]}, + "b": {"action": ["T_echo"], "depends_on": ["a"]}, + }, + } + ), + encoding="utf-8", + ) + expected = {"a": (40.0, 40.0), "b": (280.0, 40.0)} + assert service.load(path).positions() == expected + layout_path(path).write_text("{not json", encoding="utf-8") + assert service.load(path).positions() == expected + layout_path(path).write_text("[]", encoding="utf-8") + assert service.load(path).positions() == expected + + +def test_saving_and_loading_report_what_they_cannot_do( + service: PipelineService, tmp_path: Path +) -> None: + draft = _draft(("a", "T_echo", None)) + with pytest.raises(AppException, match=r"\.yaml, \.yml or \.json"): + service.save(draft, tmp_path / "chain.txt") + with pytest.raises(PipelineDefinitionException, match="cannot read the definition"): + service.load(tmp_path / "missing.yaml") + broken = tmp_path / "broken.yaml" + broken.write_text("tasks: [unclosed", encoding="utf-8") + with pytest.raises(PipelineDefinitionException, match="invalid YAML"): + service.load(broken) + + +def test_an_invalid_definition_file_opens_with_its_notes( + service: PipelineService, tmp_path: Path +) -> None: + path = tmp_path / "invalid.yaml" + path.write_text("schema_version: 1\nname: x\ntasks:\n a: {action: []}\n", encoding="utf-8") + draft = service.load(path) + assert draft.task_ids() == ("a",) + assert any("tasks.a.action" in note for note in draft.load_notes) + + +def test_a_new_draft_carries_the_given_name(service: PipelineService) -> None: + assert service.new_draft("nightly").name == "nightly" + assert service.new_draft().name == "pipeline" + + +# ---------------------------------------------------------------------- the default store + + +def test_without_a_store_the_default_run_store_is_used_at_every_call( + workshop: _Workshop, bus: EventBus, tmp_path: Path +) -> None: + service = PipelineService(registry=workshop.registry(), bus=bus) + previous = default_run_store() + replacement = SQLiteRunStore(tmp_path / "runs.db") + set_default_run_store(replacement) + try: + done = _finished(service, service.start(_draft(("a", "T_echo", None)))["run_id"]) + assert replacement.get_run(done["run_id"]) is not None + assert previous.get_run(done["run_id"]) is None + finally: + set_default_run_store(previous) diff --git a/tests/test_app_services.py b/tests/test_app_services.py new file mode 100644 index 0000000..b0d0f27 --- /dev/null +++ b/tests/test_app_services.py @@ -0,0 +1,787 @@ +"""The scheduler, integrity, audit, notification, settings and dashboard services.""" + +from __future__ import annotations + +import ast +import json +import subprocess +import sys +from collections.abc import Iterator +from pathlib import Path +from typing import Any + +import pytest + +from automation_file.app import ( + MASK, + NAVIGATION, + AppException, + AppServices, + AuditService, + IntegrityService, + NotificationService, + SchedulerService, + ServiceOptions, + SettingsService, + app_services, + brief_run, + build_services, + reset_app_services, +) +from automation_file.app import settings_service as settings_module +from automation_file.app.settings_service import EXTRA_MODULES +from automation_file.audit import AuditException, AuditTrail, MemoryAuditStore +from automation_file.core.action_registry import ActionRegistry +from automation_file.core.config import ConfigException +from automation_file.core.optional import EXTRAS, install_hint +from automation_file.events import Event, EventBus, PipelineFailed, Severity, SystemErrorEvent +from automation_file.exceptions import FileAutomationException +from automation_file.integrity import IntegrityException +from automation_file.integrity import actions as integrity_actions +from automation_file.notify import ( + NotificationException, + NotificationManager, + NotificationRouter, + NotificationSink, +) +from automation_file.pipeline import MemoryRunStore +from automation_file.scheduler import schedule_list, schedule_remove_all +from automation_file.storage import File, clear_memory_stores + +REPO_ROOT = Path(__file__).resolve().parents[1] +APP_PACKAGE = REPO_ROOT / "automation_file" / "app" +TREE = "memory://app-tree/data" +BASELINE = "memory://app-state/data.json" +NEVER = "0 0 29 2 *" +WAIT = 10.0 + + +class _Recorder(NotificationSink): + def __init__(self, name: str) -> None: + self.name = name + self.sent: list[tuple[str, str, str]] = [] + + def send(self, subject: str, body: str, level: str = "info") -> None: + self.sent.append((subject, body, level)) + + +class _Broken(NotificationSink): + name = "broken" + + def send(self, subject: str, body: str, level: str = "info") -> None: + raise NotificationException("POST https://hooks.example.com/services/s3cr3t failed") + + +@pytest.fixture(autouse=True) +def _clean_global_state() -> Iterator[None]: + clear_memory_stores() + yield + integrity_actions.stop_all_monitors() + schedule_remove_all() + clear_memory_stores() + reset_app_services() + + +@pytest.fixture(name="bus") +def _bus() -> EventBus: + return EventBus() + + +@pytest.fixture(name="manager") +def _manager() -> NotificationManager: + return NotificationManager() + + +@pytest.fixture(name="router") +def _router(manager: NotificationManager, bus: EventBus) -> Iterator[NotificationRouter]: + router = NotificationRouter(manager, bus) + yield router + router.stop() + + +@pytest.fixture(name="trail") +def _trail(bus: EventBus) -> Iterator[AuditTrail]: + trail = AuditTrail(bus=bus) + yield trail + trail.close() + + +# ---------------------------------------------------------------------- scheduler + + +def test_jobs_are_added_listed_and_removed() -> None: + scheduler = SchedulerService() + added = scheduler.add("app-job", NEVER, [["FA_storage_schemes"]]) + assert (added["name"], added["cron"]) == ("app-job", NEVER) + assert [job["name"] for job in scheduler.jobs()] == ["app-job"] + scheduler.add(" second ", NEVER, '[["FA_storage_schemes"]]', allow_overlap=True) + assert [job["name"] for job in schedule_list()] == ["app-job", "second"] + assert scheduler.remove("app-job")["name"] == "app-job" + assert [job["name"] for job in scheduler.remove_all()] == ["second"] + assert scheduler.jobs() == [] + + +def test_a_job_needs_a_name_a_cron_expression_and_an_action_list() -> None: + scheduler = SchedulerService() + with pytest.raises(AppException, match="a name and a cron expression"): + scheduler.add("", NEVER, [["FA_storage_schemes"]]) + with pytest.raises(AppException, match="a name and a cron expression"): + scheduler.add("job", " ", [["FA_storage_schemes"]]) + with pytest.raises(AppException, match="not valid JSON"): + scheduler.add("job", NEVER, "[[oops") + for wrong in ("{}", "[]", ""): + with pytest.raises(AppException, match="non-empty JSON array"): + scheduler.add("job", NEVER, wrong) + with pytest.raises(FileAutomationException): + scheduler.add("job", "not a cron", [["FA_storage_schemes"]]) + with pytest.raises(FileAutomationException): + scheduler.remove("missing") + assert scheduler.jobs() == [] + + +# ---------------------------------------------------------------------- integrity + + +def _tree() -> None: + File(f"{TREE}/a.txt").write(b"alpha") + File(f"{TREE}/sub/b.txt").write(b"bravo") + + +def test_baseline_verify_and_accept() -> None: + _tree() + integrity = IntegrityService() + stored = integrity.baseline(TREE, BASELINE) + assert (stored["entries"], stored["algorithm"]) == (2, "sha256") + assert integrity.verify(TREE, BASELINE)["ok"] is True + File(f"{TREE}/a.txt").write(b"changed") + File(f"{TREE}/new.txt").write(b"new") + report = integrity.verify(f" {TREE} ", BASELINE, deep=True) + assert report["ok"] is False + assert (report["counts"]["modified"], report["counts"]["created"]) == (1, 1) + assert integrity.accept(TREE, BASELINE)["entries"] == 3 + assert integrity.verify(TREE, BASELINE)["ok"] is True + + +def test_the_algorithms_offered_start_with_the_default() -> None: + assert IntegrityService().algorithms() == ["sha256", "blake2b", "sha512"] + + +def test_a_target_and_a_baseline_are_required() -> None: + integrity = IntegrityService() + with pytest.raises(AppException, match="the target is required"): + integrity.verify("", BASELINE) + with pytest.raises(AppException, match="the baseline is required"): + integrity.baseline(TREE, " ") + with pytest.raises(AppException, match="the monitor name is required"): + integrity.start_monitor(" ", TREE, BASELINE) + with pytest.raises(AppException, match="more than 0 seconds"): + integrity.start_monitor("m", TREE, BASELINE, interval=0) + + +def test_a_monitor_is_started_reported_and_stopped() -> None: + _tree() + integrity = IntegrityService() + integrity.baseline(TREE, BASELINE) + started = integrity.start_monitor("app-monitor", TREE, BASELINE, interval=3600) + assert (started["name"], started["running"]) == ("app-monitor", True) + assert [status["name"] for status in integrity.status()] == ["app-monitor"] + assert integrity.status("app-monitor")[0]["target"] == TREE + drift = integrity.drift()[0] + assert (drift.name, drift.running, drift.ok, drift.needs_attention) == ( + "app-monitor", + True, + None, + False, + ) + with pytest.raises(IntegrityException, match="already running"): + integrity.start_monitor("app-monitor", TREE, BASELINE) + assert integrity.stop_monitor("app-monitor")["running"] is False + assert integrity.status() == [] + with pytest.raises(IntegrityException): + integrity.stop_monitor("app-monitor") + + +def test_a_monitor_needs_a_stored_baseline() -> None: + _tree() + with pytest.raises(IntegrityException, match="no baseline"): + IntegrityService().start_monitor("app-monitor", TREE, BASELINE) + + +def test_only_the_monitors_this_service_started_are_stopped_with_it() -> None: + _tree() + integrity = IntegrityService() + integrity.baseline(TREE, BASELINE) + integrity.start_monitor("mine", TREE, BASELINE, interval=3600) + integrity.start_monitor("gone", TREE, BASELINE, interval=3600) + integrity_actions.integrity_watch_start("theirs", TREE, BASELINE, 3600) + integrity_actions.integrity_watch_stop("gone") + assert integrity.stop_started() == ["mine"] + assert [status["name"] for status in integrity.status()] == ["theirs"] + assert integrity.stop_started() == [] + + +def test_drift_summarises_the_last_report_of_each_monitor( + monkeypatch: pytest.MonkeyPatch, +) -> None: + statuses = [ + { + "name": "drifted", + "target": "memory://t/a", + "baseline": "memory://s/a.json", + "running": True, + "last_run": "2026-10-08T00:00:00+00:00", + "last_error": None, + "last_report": { + "ok": False, + "changes": [{"kind": "modified"}, {"kind": "deleted"}], + "counts": {"modified": 1, "deleted": 1, "created": 0}, + }, + }, + {"name": "failing", "target": "memory://t/b", "running": True, "last_error": "boom"}, + {"name": "clean", "target": "memory://t/c", "last_report": {"ok": True, "changes": []}}, + ] + monkeypatch.setattr( + "automation_file.app.integrity_service.integrity_status", lambda name=None: statuses + ) + drifted, failing, clean = IntegrityService().drift() + assert (drifted.ok, drifted.changes, drifted.needs_attention) == (False, 2, True) + assert drifted.counts == {"modified": 1, "deleted": 1, "created": 0} + assert (failing.ok, failing.last_error, failing.needs_attention) == (None, "boom", True) + assert (clean.ok, clean.running, clean.needs_attention) == (True, False, False) + assert drifted.to_dict()["needs_attention"] is True + + +# ---------------------------------------------------------------------- audit + + +def test_audit_is_not_configured_until_it_has_a_store(trail: AuditTrail) -> None: + audit = AuditService(trail) + assert audit.is_configured() is False + assert audit.status() == { + "configured": False, + "active": False, + "store": None, + "db_path": None, + "schema_version": None, + } + assert audit.recent() == [] + with pytest.raises(AuditException, match="not configured"): + audit.search() + with pytest.raises(AuditException, match="not configured"): + audit.count() + + +def test_audit_records_are_searched_counted_and_masked(trail: AuditTrail, bus: EventBus) -> None: + audit = AuditService(trail) + status = audit.configure(MemoryAuditStore()) + assert (status["configured"], status["active"], status["store"]) == ( + True, + True, + "MemoryAuditStore", + ) + bus.publish( + PipelineFailed( + source="pipeline", + subject="nightly failed", + payload={"pipeline": "nightly", "status": "failed", "password": "hunter2"}, + ) + ) + trail.record("manual.note", resource="s3://reports/a.csv", status="ok", actor="ops") + assert audit.count() == 2 + assert audit.count(status="failed") == 1 + assert audit.count(status="", actor=None, pipeline=" ") == 2 + found = audit.search(pipeline="nightly") + assert [record["action"] for record in found] == ["pipeline.failed"] + assert found[0]["metadata"]["password"] == MASK + assert "hunter2" not in json.dumps(audit.search()) + assert [record["action"] for record in audit.search(resource_prefix="s3://reports/")] == [ + "manual.note" + ] + assert [record["action"] for record in audit.recent(limit=1)] == ["manual.note"] + with pytest.raises(AuditException, match="unknown audit filter"): + audit.search(colour="red") + assert "since" in audit.filter_names() + assert "limit" in audit.filter_names() + + +def test_audit_is_configured_with_a_sqlite_path(trail: AuditTrail, tmp_path: Path) -> None: + audit = AuditService(trail) + path = tmp_path / "audit.sqlite" + status = audit.configure(path) + assert status["store"] == "SQLiteAuditStore" + assert status["db_path"] == str(path) + assert status["schema_version"] == 2 + assert audit.search() == [] + + +# ---------------------------------------------------------------------- notifications + + +def test_sinks_are_described_without_their_secrets( + manager: NotificationManager, router: NotificationRouter +) -> None: + from automation_file.notify import EmailSink + + manager.register(_Recorder("team")) + manager.register( + EmailSink( + host="smtp.example.com", + port=587, + sender="bot@example.com", + recipients=["ops@example.com"], + username="bot", + password="hunter2", + name="mail", + ) + ) + notifications = NotificationService(manager, router) + sinks = notifications.sinks() + assert [sink["name"] for sink in sinks] == ["team", "mail"] + assert sinks[1]["type"] == "EmailSink" + assert "hunter2" not in json.dumps(sinks) + assert notifications.sink_names() == ["team", "mail"] + assert notifications.severities() == ["info", "warning", "error", "critical"] + + +def test_routes_are_added_from_form_text_listed_and_removed( + manager: NotificationManager, router: NotificationRouter, bus: EventBus +) -> None: + recorder = _Recorder("team") + manager.register(recorder) + notifications = NotificationService(manager, router) + assert notifications.router_active() is False + route = notifications.add_route( + { + "name": "failures", + "sinks": "team", + "types": "pipeline.failed, task.failed", + "sources": "", + "min_severity": "error", + "dedup_seconds": "0", + "rate_limit": "5", + "rate_period": None, + } + ) + assert route == { + "name": "failures", + "sinks": ["team"], + "types": ["pipeline.failed", "task.failed"], + "sources": [], + "min_severity": "error", + "dedup_seconds": 0.0, + "rate_limit": 5, + "rate_period": 60.0, + } + assert notifications.router_active() is True + assert notifications.routes() == [route] + bus.publish(PipelineFailed(source="pipeline", subject="nightly failed")) + assert len(recorder.sent) == 1 + assert notifications.remove_route("failures") is True + assert notifications.remove_route("failures") is False + assert (notifications.routes(), notifications.router_active()) == ([], False) + + +def test_a_route_to_an_unknown_sink_or_with_a_wrong_option_is_refused( + manager: NotificationManager, router: NotificationRouter +) -> None: + manager.register(_Recorder("team")) + notifications = NotificationService(manager, router) + with pytest.raises(AppException, match=r"unknown sink\(s\) \['tema'\]"): + notifications.add_route({"name": "typo", "sinks": ["tema"]}) + with pytest.raises(NotificationException, match="needs a 'name'"): + notifications.add_route({"sinks": "team"}) + with pytest.raises(NotificationException, match="min_severity"): + notifications.add_route({"name": "x", "min_severity": "loud"}) + with pytest.raises(NotificationException, match="rate_limit"): + notifications.add_route({"name": "x", "rate_limit": "many"}) + with pytest.raises(NotificationException, match="unknown route option"): + notifications.add_route({"name": "x", "colour": "red"}) + assert notifications.routes() == [] + + +def test_a_test_message_goes_to_one_sink_or_to_all( + manager: NotificationManager, router: NotificationRouter +) -> None: + first, second = _Recorder("first"), _Recorder("second") + notifications = NotificationService(manager, router) + with pytest.raises(AppException, match="nothing to test"): + notifications.send_test() + manager.register(first) + manager.register(second) + manager.register(_Broken()) + assert notifications.send_test("first") == {"first": "sent"} + assert notifications.send_test("first", subject="again") == {"first": "sent"} + assert [message[0] for message in first.sent] == [ + "automation_file: test notification", + "again", + ] + outcomes = notifications.send_test() + assert (outcomes["first"], outcomes["second"]) == ("sent", "sent") + assert outcomes["broken"].startswith("NotificationException:") + assert "s3cr3t" not in outcomes["broken"] + assert len(second.sent) == 1 + with pytest.raises(NotificationException, match="no notification sink"): + notifications.send_test("missing") + + +# ---------------------------------------------------------------------- settings + +_CONFIG = """ +[[notify.sinks]] +type = "email" +name = "ops-mail" +host = "smtp.example.com" +port = 587 +sender = "bot@example.com" +recipients = ["ops@example.com"] +username = "bot" +password = "${env:FA_APP_TEST_SMTP}" + +[[notify.routes]] +name = "failures" +sinks = ["ops-mail"] +types = ["pipeline.failed"] +min_severity = "error" + +[defaults] +dedup_seconds = 120 +""" + + +@pytest.fixture(name="config_path") +def _config_path(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> Path: + monkeypatch.setenv("FA_APP_TEST_SMTP", "hunter2") + path = tmp_path / "automation_file.toml" + path.write_text(_CONFIG, encoding="utf-8") + return path + + +def test_loading_a_configuration_shows_it_masked_and_changes_nothing( + manager: NotificationManager, router: NotificationRouter, config_path: Path +) -> None: + settings = SettingsService(manager, router) + summary = settings.load(config_path) + assert summary["source"] == str(config_path) + assert summary["applied"] is False + assert summary["sections"] == ["defaults", "notify"] + assert summary["sinks"] == [{"name": "ops-mail", "type": "email"}] + assert [route["name"] for route in summary["routes"]] == ["failures"] + assert summary["defaults"] == {"dedup_seconds": 120} + assert summary["document"]["notify"]["sinks"][0]["password"] == MASK + assert "hunter2" not in json.dumps(summary) + assert manager.names() == () + assert router.routes() == [] + assert settings.applied() is None + assert settings.environment()["applied_config"] is None + + +def test_a_webhook_url_in_a_configuration_keeps_only_its_host( + manager: NotificationManager, router: NotificationRouter, tmp_path: Path +) -> None: + path = tmp_path / "hooks.toml" + path.write_text( + '[[notify.sinks]]\ntype = "slack"\nname = "team"\n' + 'webhook_url = "https://hooks.example.com/services/T0/B0/s3cr3t"\n', + encoding="utf-8", + ) + summary = SettingsService(manager, router).load(path) + assert summary["document"]["notify"]["sinks"][0]["webhook_url"] == ( + f"https://hooks.example.com/{MASK}" + ) + assert "s3cr3t" not in json.dumps(summary) + + +def test_applying_a_configuration_registers_its_sinks_and_routes( + manager: NotificationManager, router: NotificationRouter, config_path: Path +) -> None: + settings = SettingsService(manager, router) + summary = settings.apply(config_path) + assert (summary["applied"], summary["registered_sinks"]) == (True, 1) + assert manager.names() == ("ops-mail",) + assert manager.dedup_seconds == 120.0 + assert [route.name for route in router.routes()] == ["failures"] + assert router.active is True + assert settings.applied() == summary + assert settings.environment()["applied_config"] == str(config_path) + assert "hunter2" not in json.dumps(settings.applied()) + + +def test_a_configuration_that_cannot_be_used_raises( + manager: NotificationManager, router: NotificationRouter, tmp_path: Path +) -> None: + settings = SettingsService(manager, router) + with pytest.raises(ConfigException, match="not found"): + settings.load(tmp_path / "missing.toml") + bad = tmp_path / "bad.toml" + bad.write_text('[[notify.routes]]\nname = "r"\nsinks = ["ghost"]\n', encoding="utf-8") + with pytest.raises(ConfigException, match="unknown sink"): + settings.load(bad) + with pytest.raises(ConfigException, match="unknown sink"): + settings.apply(bad) + assert manager.names() == () + + +def test_every_extra_the_package_names_has_a_status() -> None: + assert set(EXTRA_MODULES) == set(EXTRAS) + extras = SettingsService().extras() + assert [extra.name for extra in extras] == list(EXTRAS) + by_name = {extra.name: extra for extra in extras} + assert (by_name["ftp"].installed, by_name["ftp"].install_hint) == (True, None) + assert by_name["webdav"].installed is True + assert by_name["s3"].feature == EXTRAS["s3"] + assert by_name["s3"].to_dict()["modules"] == ["boto3"] + + +def test_a_missing_extra_carries_its_install_command(monkeypatch: pytest.MonkeyPatch) -> None: + monkeypatch.setattr(settings_module, "is_installed", lambda module: module != "msal") + monkeypatch.setitem(EXTRAS, "future", "a feature this table does not know") + by_name = {extra.name: extra for extra in SettingsService().extras()} + assert by_name["onedrive"].installed is False + assert by_name["onedrive"].missing == ("msal",) + assert by_name["onedrive"].install_hint == install_hint("onedrive") + assert (by_name["s3"].installed, by_name["s3"].install_hint) == (True, None) + assert by_name["future"].installed is None + + +def test_the_environment_names_the_versions_and_the_log_file() -> None: + environment = SettingsService().environment() + assert environment["python"].count(".") == 2 + assert environment["log_file"].endswith(".log") + assert set(environment) == {"version", "python", "platform", "log_file", "applied_config"} + + +# ---------------------------------------------------------------------- dashboard + + +def _echo(value: Any = None) -> Any: + return value + + +def _fail() -> None: + raise ValueError("it broke") + + +@pytest.fixture(name="services") +def _services( + bus: EventBus, manager: NotificationManager, router: NotificationRouter, trail: AuditTrail +) -> AppServices: + return build_services( + ServiceOptions( + run_store=MemoryRunStore(), + registry=ActionRegistry({"T_echo": _echo, "T_fail": _fail}), + bus=bus, + audit_trail=trail, + notification_manager=manager, + notification_router=router, + ) + ) + + +def _run(services: AppServices, action: str) -> dict[str, Any]: + draft = services.pipelines.new_draft("dash") + draft.add_task(action, "only") + started = services.pipelines.start(draft) + assert services.pipelines.wait(started["run_id"], WAIT) + return services.pipelines.status(started["run_id"]) + + +def test_a_quiet_installation_is_ok(services: AppServices) -> None: + summary = services.dashboard.summary() + assert (summary.status, summary.reasons) == ("ok", ()) + assert summary.run_counts == {"running": 0, "succeeded": 0, "failed": 0, "cancelled": 0} + assert (summary.running_runs, summary.recent_runs, summary.integrity, summary.events) == ( + [], + [], + [], + [], + ) + health = summary.health + assert (health["process"], health["registry_size"], health["scheduler_jobs"]) == ("alive", 2, 0) + assert health["audit"]["configured"] is False + assert health["notification_router_active"] is False + assert {"local", "memory"} <= {backend["name"] for backend in summary.storage} + assert json.loads(json.dumps(summary.to_dict()))["status"] == "ok" + + +def test_the_dashboard_counts_runs_and_asks_for_attention_after_a_failure( + services: AppServices, +) -> None: + good = _run(services, "T_echo") + assert services.dashboard.summary().status == "ok" + bad = _run(services, "T_fail") + summary = services.dashboard.summary() + assert summary.status == "attention" + assert summary.run_counts == {"running": 0, "succeeded": 1, "failed": 1, "cancelled": 0} + assert [run["run_id"] for run in summary.recent_runs] == [bad["run_id"], good["run_id"]] + assert summary.recent_runs[0]["task_statuses"] == {"failed": 1} + assert "1 of the last 2 pipeline runs failed" in summary.reasons + assert any("severity error or worse" in reason for reason in summary.reasons) + assert summary.events[0]["type"] == "pipeline.failed" + assert services.dashboard.runs(limit=1)["recent"][0]["run_id"] == bad["run_id"] + + +def test_recent_events_are_newest_first_filtered_and_masked( + services: AppServices, bus: EventBus +) -> None: + bus.publish(Event(source="test", subject="first", payload={"token": "abc"})) + bus.publish(SystemErrorEvent(source="test", subject="second")) + events = services.dashboard.recent_events() + assert [event["subject"] for event in events] == ["second", "first"] + assert events[1]["payload"] == {"token": MASK} + serious = services.dashboard.recent_events(min_severity=Severity.ERROR.value) + assert [event["subject"] for event in serious] == ["second"] + assert len(services.dashboard.recent_events(limit=1)) == 1 + + +def test_integrity_drift_reaches_the_dashboard(services: AppServices) -> None: + _tree() + services.integrity.baseline(TREE, BASELINE) + services.integrity.start_monitor("dash-monitor", TREE, BASELINE, interval=3600) + summary = services.dashboard.summary() + assert [monitor["name"] for monitor in summary.integrity] == ["dash-monitor"] + assert summary.health["integrity_monitors"] == 1 + assert summary.status == "ok" + + +def test_a_monitor_that_found_drift_asks_for_attention( + services: AppServices, monkeypatch: pytest.MonkeyPatch +) -> None: + statuses = [ + {"name": "a", "target": "memory://t/a", "last_report": {"ok": False, "changes": [{}]}}, + {"name": "b", "target": "memory://t/b", "last_error": "boom"}, + ] + monkeypatch.setattr( + "automation_file.app.integrity_service.integrity_status", lambda name=None: statuses + ) + summary = services.dashboard.summary() + assert summary.status == "attention" + assert summary.reasons == ( + "integrity monitor 'a' found 1 change(s)", + "integrity monitor 'b' could not verify", + ) + + +def test_a_part_that_cannot_be_read_is_reported_not_raised( + services: AppServices, monkeypatch: pytest.MonkeyPatch +) -> None: + def broken() -> list[Any]: + raise IntegrityException("the store is gone") + + monkeypatch.setattr(services.integrity, "drift", broken) + summary = services.dashboard.summary() + assert summary.status == "attention" + assert summary.reasons == ("integrity cannot be read: IntegrityException",) + assert summary.integrity == [] + assert summary.health["process"] == "alive" + + +def test_a_brief_run_drops_the_task_details() -> None: + brief = brief_run( + { + "run_id": "r1", + "pipeline": "p", + "status": "failed", + "active": False, + "started_at": "t0", + "finished_at": "t1", + "error": "bad", + "params": {"a": 1}, + "tasks": {"a": {"status": "succeeded"}, "b": {"status": "failed"}}, + } + ) + assert brief == { + "run_id": "r1", + "pipeline": "p", + "status": "failed", + "active": False, + "started_at": "t0", + "finished_at": "t1", + "error": "bad", + "tasks": 2, + "task_statuses": {"succeeded": 1, "failed": 1}, + } + + +# ---------------------------------------------------------------------- the set of services + + +def test_the_navigation_names_one_service_each_in_order() -> None: + assert NAVIGATION == ( + "Dashboard", + "Files", + "Storage", + "Pipelines", + "Scheduler", + "Integrity", + "Audit", + "Notifications", + "Settings", + ) + services = build_services() + assert [name.lower() for name in NAVIGATION] == list(AppServices.__dataclass_fields__) + for name in NAVIGATION: + assert getattr(services, name.lower()) is not None + + +def test_the_shared_set_is_built_once_and_can_be_forgotten() -> None: + first = app_services() + assert app_services() is first + reset_app_services() + assert app_services() is not first + + +def _module_level_imports(path: Path) -> list[str]: + tree = ast.parse(path.read_text(encoding="utf-8"), filename=str(path)) + names: list[str] = [] + for node in tree.body: + if isinstance(node, ast.Import): + names.extend(alias.name for alias in node.names) + elif isinstance(node, ast.ImportFrom): + assert node.level == 0, f"{path.name}: relative import" + names.append(node.module or "") + return names + + +_FORBIDDEN_ROOTS = ( + "PySide6", + "boto3", + "botocore", + "azure", + "dropbox", + "paramiko", + "googleapiclient", + "msal", + "box_sdk_gen", + "pyarrow", + "fsspec", + "smbclient", +) + + +@pytest.mark.parametrize("path", sorted(APP_PACKAGE.glob("*.py")), ids=lambda path: path.name) +def test_no_module_of_the_layer_imports_a_gui_toolkit_or_an_sdk(path: Path) -> None: + imported = _module_level_imports(path) + assert [name for name in imported if name.partition(".")[0] in _FORBIDDEN_ROOTS] == [] + assert [name for name in imported if name.startswith("automation_file.ui")] == [] + assert [name for name in imported if name == "automation_file"] == [] + + +def test_importing_and_using_the_layer_loads_no_gui_toolkit() -> None: + probe = ( + "import sys\n" + "import automation_file.app as app\n" + "services = app.app_services()\n" + "services.dashboard.summary()\n" + "services.settings.extras()\n" + "print('PySide6' in sys.modules)\n" + ) + result = subprocess.run( # nosec B603 - fixed argv: this interpreter and the script above + [sys.executable, "-c", probe], + cwd=REPO_ROOT, + capture_output=True, + text=True, + timeout=120, + check=False, + ) + assert result.returncode == 0, result.stderr[-2000:] + assert result.stdout.strip().splitlines()[-1] == "False" diff --git a/tests/test_integrity_watch.py b/tests/test_integrity_watch.py index ab718b1..6756869 100644 --- a/tests/test_integrity_watch.py +++ b/tests/test_integrity_watch.py @@ -9,6 +9,7 @@ from __future__ import annotations import threading +import time from collections.abc import Iterator from pathlib import Path @@ -150,7 +151,10 @@ def test_the_interval_runner_ticks_until_stopped_and_can_start_again() -> None: assert runner.is_running is False count = len(ticks.items) runner.start() - assert ticks.wait() + # The first run may have left ticks nobody waited for, so wait for a new one by count. + deadline = time.monotonic() + WAIT + while len(ticks.items) <= count and time.monotonic() < deadline: + time.sleep(0.005) runner.stop() assert len(ticks.items) > count with pytest.raises(IntegrityException, match="interval must be positive"): diff --git a/tests/test_ui_pages.py b/tests/test_ui_pages.py new file mode 100644 index 0000000..21d24bb --- /dev/null +++ b/tests/test_ui_pages.py @@ -0,0 +1,788 @@ +"""The pages of the main window, fed by their services through a pool that runs at once.""" + +from __future__ import annotations + +import ast +import json +import os +import threading +import time +from collections.abc import Iterator +from pathlib import Path +from typing import Any + +import pytest + +pytest.importorskip("PySide6") + +os.environ.setdefault("QT_QPA_PLATFORM", "offscreen") + +from automation_file.app import ( + MASK, + AppServices, + ServiceOptions, + build_services, +) +from automation_file.audit import AuditTrail, MemoryAuditStore +from automation_file.core.action_registry import ActionRegistry +from automation_file.events import EventBus, PipelineFailed, SystemErrorEvent +from automation_file.integrity import actions as integrity_actions +from automation_file.notify import ( + NotificationException, + NotificationManager, + NotificationRouter, + NotificationSink, +) +from automation_file.pipeline import MemoryRunStore +from automation_file.scheduler import schedule_list, schedule_remove_all +from automation_file.storage import ( + File, + MemoryStorage, + StorageResolver, + clear_memory_stores, +) +from tests.ui_stand_in import HeldPool, SyncPool + +ROOT = "memory://pages" +TREE = "memory://pages-tree/data" +BASELINE = "memory://pages-state/data.json" +NEVER = "0 0 29 2 *" +WAIT = 10.0 + + +class _Recorder(NotificationSink): + def __init__(self, name: str) -> None: + self.name = name + self.sent: list[tuple[str, str, str]] = [] + + def send(self, subject: str, body: str, level: str = "info") -> None: + self.sent.append((subject, body, level)) + + +class _Broken(NotificationSink): + name = "broken" + + def send(self, subject: str, body: str, level: str = "info") -> None: + raise NotificationException("POST https://hooks.example.com/services/s3cr3t failed") + + +@pytest.fixture(name="qt_app", scope="module") +def _qt_app(): + from PySide6.QtWidgets import QApplication + + app = QApplication.instance() or QApplication([]) + yield app + + +@pytest.fixture(autouse=True) +def _clean_global_state() -> Iterator[None]: + clear_memory_stores() + yield + integrity_actions.stop_all_monitors() + schedule_remove_all() + clear_memory_stores() + + +@pytest.fixture(name="bus") +def _bus() -> EventBus: + return EventBus() + + +@pytest.fixture(name="manager") +def _manager() -> NotificationManager: + return NotificationManager() + + +@pytest.fixture(name="resolver") +def _resolver() -> StorageResolver: + resolver = StorageResolver(defaults=False) + resolver.mount(ROOT, MemoryStorage()) + return resolver + + +@pytest.fixture(name="services") +def _services( + bus: EventBus, manager: NotificationManager, resolver: StorageResolver +) -> Iterator[AppServices]: + router = NotificationRouter(manager, bus) + trail = AuditTrail(bus=bus) + yield build_services( + ServiceOptions( + resolver=resolver, + run_store=MemoryRunStore(), + registry=ActionRegistry({"T_echo": lambda value=None: value, "T_fail": _fail}), + bus=bus, + audit_trail=trail, + notification_manager=manager, + notification_router=router, + ) + ) + router.stop() + trail.close() + + +@pytest.fixture(name="log") +def _log(qt_app) -> Iterator[Any]: + from automation_file.ui.log_widget import LogPanel + + assert qt_app is not None + panel = LogPanel() + yield panel + panel.deleteLater() + + +def _fail() -> None: + raise ValueError("it broke") + + +def _page(page_class: str, service: Any, log: Any, pool: Any = None) -> Any: + from automation_file.ui import pages + + return getattr(pages, page_class)(service, log, SyncPool() if pool is None else pool) + + +def _column(table: Any, column: int) -> list[str]: + return [table.item(row, column).text() for row in range(table.rowCount())] + + +def _select(table: Any, text: str, column: int = 0) -> None: + table.selectRow(_column(table, column).index(text)) + + +# ---------------------------------------------------------------------- the base page + + +def test_a_call_with_a_key_is_skipped_while_the_earlier_one_runs( + services: AppServices, log: Any +) -> None: + pool = HeldPool() + page = _page("StoragePage", services.storage, log, pool) + seen: list[Any] = [] + assert page.run_async(lambda: 1, "first", seen.append, key="k") is True + assert page.run_async(lambda: 2, "second", seen.append, key="k") is False + assert page.run_async(lambda: 3, "other key", seen.append, key="other") is True + assert page.run_async(lambda: 4, "no key", seen.append) is True + assert pool.release() == 3 + assert seen == [1, 3, 4] + assert page.run_async(lambda: 5, "again", seen.append, key="k") is True + pool.release() + assert seen[-1] == 5 + + +def test_a_failure_is_shown_masked_on_the_status_line_and_in_the_log( + services: AppServices, log: Any +) -> None: + page = _page("StoragePage", services.storage, log) + + def fails() -> None: + raise ValueError("cannot reach https://user:hunter2@example.com/x") + + done: list[Any] = [] + page.run_async(fails, "read", done.append, key="k") + assert done == [] + assert page.status_text().startswith("read failed:") + assert "hunter2" not in page.status_text() + assert "hunter2" not in log.toPlainText() + assert "Storage: read failed" in log.toPlainText() + assert page.run_async(lambda: 1, "after", done.append, key="k") is True + assert done == [1] + + +def test_a_quiet_call_leaves_no_line_and_a_normal_one_announces_itself( + services: AppServices, log: Any +) -> None: + page = _page("StoragePage", services.storage, log) + page.run_async(lambda: 1, "quiet work", quiet=True) + assert log.toPlainText() == "" + page.run_async(lambda: 1, "loud work") + assert "Storage: loud work" in log.toPlainText() + + +def test_results_that_arrive_after_shutdown_are_dropped(services: AppServices, log: Any) -> None: + pool = HeldPool() + page = _page("StoragePage", services.storage, log, pool) + seen: list[Any] = [] + page.run_async(lambda: 1, "late", seen.append) + page.run_async(_fail, "late failure") + page.shutdown() + pool.release() + assert seen == [] + assert page.status_text() == "" + + +def test_with_a_real_thread_pool_the_work_leaves_the_ui_thread_and_the_result_returns_to_it( + qt_app, services: AppServices, log: Any +) -> None: + from PySide6.QtCore import QThreadPool + + pool = QThreadPool() + page = _page("StoragePage", services.storage, log, pool) + main = threading.main_thread() + seen: list[tuple[str, bool]] = [] + + def work() -> int: + seen.append(("work", threading.current_thread() is main)) + return 42 + + def done(result: int) -> None: + seen.append((f"done {result}", threading.current_thread() is main)) + + page.run_async(work, "probe", done) + page.run_async(_fail, "broken probe") + assert pool.waitForDone(int(WAIT * 1000)) + assert seen == [("work", False)] + assert page.status_text() == "" + deadline = time.monotonic() + WAIT + while time.monotonic() < deadline and (len(seen) < 2 or not page.status_text()): + qt_app.processEvents() + assert seen == [("work", False), ("done 42", True)] + assert page.status_text() == "broken probe failed: it broke" + page.shutdown() + + +def test_a_refilled_table_keeps_the_selected_entry_not_the_selected_row(qt_app) -> None: + from automation_file.ui.pages.base import fill_table, make_table, selected_cell + + assert qt_app is not None + table = make_table(("Name", "Value")) + fill_table(table, [["a", 1], ["b", 2], ["c", 3]]) + assert selected_cell(table) is None + table.selectRow(1) + assert (selected_cell(table), selected_cell(table, 1)) == ("b", "2") + fill_table(table, [["new", 0], ["a", 1], ["b", 2]]) + assert (table.currentRow(), selected_cell(table)) == (2, "b") + fill_table(table, [["new", 0], ["a", 1]]) + assert (table.currentRow(), selected_cell(table)) == (-1, None) + table.selectRow(0) + fill_table(table, [["new", 0]], keep_selection=False) + assert selected_cell(table) is None + table.deleteLater() + + +_PAGE_MODULES = sorted( + path + for path in (Path(__file__).resolve().parents[1] / "automation_file" / "ui" / "pages").glob( + "*.py" + ) + if path.name != "advanced_page.py" +) +_ALLOWED_FIRST_PARTY = ( + "automation_file.app", + "automation_file.exceptions", + "automation_file.logging_config", + "automation_file.ui.log_widget", + "automation_file.ui.worker", + "automation_file.ui.pages", +) + + +@pytest.mark.parametrize("path", _PAGE_MODULES, ids=lambda path: path.name) +def test_a_workflow_page_imports_nothing_below_the_application_layer(path: Path) -> None: + tree = ast.parse(path.read_text(encoding="utf-8"), filename=str(path)) + imported = [ + node.module or "" + for node in ast.walk(tree) + if isinstance(node, ast.ImportFrom) and (node.module or "").startswith("automation_file") + ] + imported.extend( + alias.name + for node in ast.walk(tree) + if isinstance(node, ast.Import) + for alias in node.names + if alias.name.startswith("automation_file") + ) + assert len(_PAGE_MODULES) >= 14 + outside = [ + name + for name in imported + if not any( + name == allowed or name.startswith(f"{allowed}.") for allowed in _ALLOWED_FIRST_PARTY + ) + ] + assert outside == [] + + +def test_cell_text_of_the_values_a_table_shows() -> None: + from automation_file.ui.pages.base import cell_text + + assert cell_text(None) == "—" + assert cell_text("") == "—" + assert cell_text(True) == "yes" + assert cell_text(False) == "no" + assert cell_text(["a", "b"]) == "a, b" + assert cell_text([]) == "—" + assert cell_text(0) == "0" + + +# ---------------------------------------------------------------------- dashboard + + +def test_the_dashboard_renders_the_summary_of_its_service( + services: AppServices, log: Any, bus: EventBus +) -> None: + page = _page("DashboardPage", services.dashboard, log) + assert page.summary() is None + page.refresh() + summary = page.summary() + assert summary is not None + assert summary.status == "ok" + assert page._headline.text() == "All clear" + assert page._health_labels["registry_size"].text() == "2" + assert page._health_labels["audit"].text() == "not configured" + assert _column(page._storage_table, 0) == [ROOT, "box"] + assert page._events_table.rowCount() == 0 + + draft = services.pipelines.new_draft("dash") + draft.add_task("T_fail", "only") + run_id = services.pipelines.start(draft)["run_id"] + assert services.pipelines.wait(run_id, WAIT) + bus.publish(SystemErrorEvent(source="test", subject="disk full", payload={"token": "abc"})) + page.refresh() + assert page._headline.text() == "Needs attention" + assert "1 of the last 1 pipeline runs failed" in page._reasons.text() + assert _column(page._recent_table, 2) == ["failed"] + assert _column(page._recent_table, 0) == [run_id[:8]] + assert "disk full" in _column(page._events_table, 4) + assert page._run_counts.text() == "0 running, 0 succeeded, 1 failed, 0 cancelled" + page.shutdown() + + +def test_the_dashboard_timer_refreshes_only_a_visible_page_with_auto_refresh_on( + services: AppServices, log: Any +) -> None: + pool = SyncPool() + page = _page("DashboardPage", services.dashboard, log, pool) + page._on_tick() + assert pool.started == 0 + page.show() + page._on_tick() + assert pool.started == 1 + page._auto.setChecked(False) + page._on_tick() + assert pool.started == 1 + page.hide() + page.shutdown() + assert page._timer.isActive() is False + + +# ---------------------------------------------------------------------- files + + +def test_the_files_page_lists_previews_and_navigates( + services: AppServices, log: Any, resolver: StorageResolver +) -> None: + File(f"{ROOT}/dir/a.txt", resolver=resolver).write("alpha") + File(f"{ROOT}/b.txt", resolver=resolver).write("bravo") + page = _page("FilesPage", services.files, log) + page.open_location(ROOT) + assert page.location() == ROOT + assert [entry.name for entry in page.entries()] == ["dir", "b.txt"] + assert _column(page._table, 1) == ["directory", "file"] + assert page.status_text() == f"2 entries in {ROOT}" + + assert page.select_entry("b.txt") is True + page.preview_selected() + assert page.preview_text() == "bravo" + assert "5 of 5 bytes" in page.status_text() + + assert page.select_entry("dir") is True + page.preview_selected() + assert "is a directory" in page.status_text() + page.open_selected() + assert page.location() == f"{ROOT}/dir" + assert [entry.name for entry in page.entries()] == ["a.txt"] + page.select_entry("a.txt") + page.open_selected() + assert page.preview_text() == "alpha" + page.go_up() + assert page.location() == ROOT + assert page.select_entry("missing") is False + + +def test_the_files_page_copies_moves_creates_and_deletes( + services: AppServices, log: Any, resolver: StorageResolver, monkeypatch: pytest.MonkeyPatch +) -> None: + File(f"{ROOT}/a.txt", resolver=resolver).write("alpha") + page = _page("FilesPage", services.files, log) + page.open_location(ROOT) + + page._folder.setText("inbox") + page.create_directory() + assert page.status_text() == f"created {ROOT}/inbox" + assert [entry.name for entry in page.entries()] == ["inbox", "a.txt"] + + page.select_entry("a.txt") + page._target.setText(f"{ROOT}/inbox") + page.copy_selected() + assert page.status_text() == f"copy: {ROOT}/a.txt -> {ROOT}/inbox/a.txt" + page.select_entry("a.txt") + page._target.setText(f"{ROOT}/renamed.txt") + page.move_selected() + assert [entry.name for entry in page.entries()] == ["inbox", "renamed.txt"] + + answers = iter([False, True, True]) + monkeypatch.setattr(page, "confirm", lambda _question: next(answers)) + page.select_entry("renamed.txt") + page.delete_selected() + assert [entry.name for entry in page.entries()] == ["inbox", "renamed.txt"] + page.delete_selected() + assert [entry.name for entry in page.entries()] == ["inbox"] + page.select_entry("inbox") + page.delete_selected() + assert "delete" in page.status_text() and "failed" in page.status_text() + page._recursive.setChecked(True) + monkeypatch.setattr(page, "confirm", lambda _question: True) + page.select_entry("inbox") + page.delete_selected() + assert page.entries() == [] + + +def test_the_files_page_says_what_is_missing(services: AppServices, log: Any) -> None: + page = _page("FilesPage", services.files, log) + page.open_location() + assert "enter a storage URI" in page.status_text() + page.copy_selected() + assert "select an entry" in page.status_text() + page.create_directory() + assert "open a location" in page.status_text() + page.delete_selected() + assert "select an entry to delete" in page.status_text() + page.preview_selected() + assert "select a file to preview" in page.status_text() + page.refresh() + page.go_up() + page.open_location("memory://pages/../escape") + assert "failed" in page.status_text() + assert page.location() == "" + + +# ---------------------------------------------------------------------- storage + + +def test_the_storage_page_shows_backends_and_mounts_a_directory( + services: AppServices, log: Any, tmp_path: Path +) -> None: + page = _page("StoragePage", services.storage, log) + page.refresh() + assert [status.name for status in page.backends()] == [ROOT, "box"] + _select(page._table, ROOT) + assert "directories: yes" in page._capabilities.text() + _select(page._table, "box") + assert "depends on" in page._capabilities.text() + + page.mount_local() + assert "enter the URI" in page.status_text() + page._mount_uri.setText("sandbox://jobs") + page._mount_root.setText(str(tmp_path)) + page.mount_local() + assert page.status_text() == f"mounted sandbox://jobs on {tmp_path}" + assert "sandbox://jobs" in _column(page._table, 0) + + page.unmount_selected() + assert "select a mount" in page.status_text() + _select(page._table, "sandbox://jobs") + page.unmount_selected() + assert page.status_text() == "unmounted sandbox://jobs" + assert "sandbox://jobs" not in _column(page._table, 0) + + page._mount_root.setText(str(tmp_path / "missing")) + page.mount_local() + assert "not a directory" in page.status_text() + + +def test_the_storage_page_lists_the_default_backends_with_their_detail(log: Any) -> None: + page = _page("StoragePage", build_services().storage, log) + page.refresh() + names = _column(page._table, 0) + assert {"local", "memory", "s3", "box"} <= set(names) + assert _column(page._table, 5)[names.index("local")] == "yes" + _select(page._table, "local") + assert page._capabilities.text().startswith("local provides") + + +# ---------------------------------------------------------------------- scheduler + + +def test_the_scheduler_page_adds_lists_and_removes_jobs(services: AppServices, log: Any) -> None: + page = _page("SchedulerPage", services.scheduler, log) + page.add_job() + assert "add job" in page.status_text() and "failed" in page.status_text() + page._name.setText("nightly") + page._cron.setText(NEVER) + page._actions.setPlainText('[["FA_storage_schemes"]]') + page.add_job() + assert page.status_text() == f"added job nightly ({NEVER})" + assert [job["name"] for job in page.jobs()] == ["nightly"] + assert _column(page._table, 1) == [NEVER] + + page._name.setText("second") + page._actions.setPlainText("[[oops") + page.add_job() + assert "not valid JSON" in page.status_text() + page._actions.setPlainText('[["FA_storage_schemes"]]') + page.add_job() + page.remove_selected() + assert "select a job" in page.status_text() + _select(page._table, "nightly") + page.remove_selected() + assert page.status_text() == "removed job nightly" + assert _column(page._table, 0) == ["second"] + page.remove_all() + assert page.status_text() == "removed 1 job(s)" + assert page.jobs() == [] + + +def test_closing_the_scheduler_page_removes_the_jobs(services: AppServices, log: Any) -> None: + page = _page("SchedulerPage", services.scheduler, log) + services.scheduler.add("left-over", NEVER, [["FA_storage_schemes"]]) + page.shutdown() + assert schedule_list() == [] + + +# ---------------------------------------------------------------------- integrity + + +def test_the_integrity_page_baselines_verifies_and_accepts( + services: AppServices, log: Any, monkeypatch: pytest.MonkeyPatch +) -> None: + File(f"{TREE}/a.txt").write(b"alpha") + page = _page("IntegrityPage", services.integrity, log) + page.verify() + assert "the target is required" in page.status_text() + page.set_location(TREE, BASELINE) + page.create_baseline() + assert page.status_text() == f"baseline of 1 file(s) stored at {BASELINE}" + page.verify() + assert "no drift" in page.status_text() + assert page.last_report()["ok"] is True + + File(f"{TREE}/a.txt").write(b"changed") + File(f"{TREE}/new.txt").write(b"new") + page.verify() + assert "drift: 1 created, 1 modified" in page.status_text() + assert sorted(_column(page._changes, 0)) == ["created", "modified"] + assert sorted(_column(page._changes, 1)) == ["a.txt", "new.txt"] + + monkeypatch.setattr(page, "confirm", lambda _question: False) + page.accept() + page.verify() + assert page.last_report()["ok"] is False + monkeypatch.setattr(page, "confirm", lambda _question: True) + page.accept() + assert page.status_text() == f"accepted 2 file(s) as the baseline at {BASELINE}" + page.verify() + assert page.last_report()["ok"] is True + + +def test_the_integrity_page_starts_and_stops_a_monitor(services: AppServices, log: Any) -> None: + File(f"{TREE}/a.txt").write(b"alpha") + page = _page("IntegrityPage", services.integrity, log) + page.set_location(TREE, BASELINE) + page._name.setText("watch") + page._interval.setValue(3600) + page.start_monitor() + assert "no baseline" in page.status_text() + page.create_baseline() + page.start_monitor() + assert page.status_text() == f"monitor watch verifies {TREE} every 3600 s" + assert [monitor.name for monitor in page.monitors()] == ["watch"] + assert _column(page._monitor_table, 4) == ["not verified yet"] + page.stop_selected() + assert "select a monitor" in page.status_text() + _select(page._monitor_table, "watch") + page.stop_selected() + assert page.status_text() == "monitor watch stopped" + assert page.monitors() == [] + + page.start_monitor() + page.shutdown() + assert services.integrity.status() == [] + assert "stopped monitor(s) watch" in log.toPlainText() + + +# ---------------------------------------------------------------------- audit + + +def test_the_audit_page_configures_searches_and_shows_a_record( + services: AppServices, log: Any, bus: EventBus, tmp_path: Path +) -> None: + page = _page("AuditPage", services.audit, log) + page.refresh() + assert page._state.text().startswith("Not configured") + page.search() + assert "not configured" in page.status_text() + page.configure() + assert "enter the path" in page.status_text() + + database = tmp_path / "audit.sqlite" + page._db_path.setText(str(database)) + page.configure() + assert page._state.text() == f"recording into {database}" + bus.publish( + PipelineFailed( + source="pipeline", + subject="nightly failed", + payload={"pipeline": "nightly", "status": "failed", "password": "hunter2"}, + ) + ) + bus.publish(SystemErrorEvent(source="system", subject="disk full")) + page.search() + assert page.status_text() == "2 record(s) shown" + assert _column(page._table, 3) == ["system.error", "pipeline.failed"] + + page.set_filter("pipeline", "nightly") + assert page.filters() == {"pipeline": "nightly"} + page.search() + assert _column(page._table, 3) == ["pipeline.failed"] + page._table.selectRow(0) + detail = json.loads(page.detail_text()) + assert detail["metadata"]["password"] == MASK + assert "hunter2" not in page.detail_text() + page.count() + assert page.status_text() == "1 record(s) match" + page.clear_filters() + assert page.filters() == {} + page.set_filter("since", "yesterday") + page.search() + assert "search failed" in page.status_text() + assert [record["action"] for record in page.records()] == ["pipeline.failed"] + + +def test_the_audit_page_shows_a_store_that_was_configured_in_code( + services: AppServices, log: Any +) -> None: + services.audit.configure(MemoryAuditStore()) + page = _page("AuditPage", services.audit, log) + page.refresh() + assert page._state.text() == "recording into MemoryAuditStore" + + +# ---------------------------------------------------------------------- notifications + + +def test_the_notifications_page_shows_sinks_and_edits_routes( + services: AppServices, log: Any, manager: NotificationManager, bus: EventBus +) -> None: + recorder = _Recorder("team") + manager.register(recorder) + page = _page("NotificationsPage", services.notifications, log) + page.refresh() + assert [sink["name"] for sink in page.sinks()] == ["team"] + assert _column(page._sink_table, 1) == ["_Recorder"] + assert "not running" in page._router_state.text() + assert [page._test_sink.itemText(i) for i in range(page._test_sink.count())] == [ + "All sinks", + "team", + ] + + page.add_route() + assert "add route" in page.status_text() and "failed" in page.status_text() + page._name.setText("failures") + page._route_sinks.setText("team") + page._types.setText("pipeline.failed, task.failed") + page._severity.setCurrentText("error") + page._dedup.setText("0") + assert page.route_options()["types"] == "pipeline.failed, task.failed" + page.add_route() + assert page.status_text() == "route failures added" + assert [route["name"] for route in page.routes()] == ["failures"] + assert _column(page._route_table, 2) == ["pipeline.failed, task.failed"] + assert _column(page._route_table, 6) == ["unlimited"] + assert "delivering" in page._router_state.text() + bus.publish(PipelineFailed(source="pipeline", subject="nightly failed")) + assert len(recorder.sent) == 1 + + page._route_sinks.setText("tema") + page.add_route() + assert "unknown sink" in page.status_text() + page.remove_selected() + assert "select a route" in page.status_text() + _select(page._route_table, "failures") + page.remove_selected() + assert page.status_text() == "route failures removed" + assert page.routes() == [] + + +def test_the_notifications_page_sends_a_test_message( + services: AppServices, log: Any, manager: NotificationManager +) -> None: + page = _page("NotificationsPage", services.notifications, log) + page.refresh() + page.send_test() + assert "nothing to test" in page.status_text() + recorder = _Recorder("team") + manager.register(recorder) + manager.register(_Broken()) + page.refresh() + page._test_sink.setCurrentText("team") + page._test_subject.setText("hello") + page.send_test() + assert page.status_text() == "test message: team: sent" + assert recorder.sent[0][0] == "hello" + page._test_sink.setCurrentText("All sinks") + page._test_subject.clear() + page.send_test() + assert "team: sent" in page.status_text() + assert "broken: NotificationException" in page.status_text() + assert "s3cr3t" not in page.status_text() + assert "s3cr3t" not in log.toPlainText() + assert recorder.sent[1][0] == "automation_file: test notification" + + +# ---------------------------------------------------------------------- settings + + +def test_the_settings_page_previews_and_applies_a_configuration( + services: AppServices, + log: Any, + manager: NotificationManager, + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + monkeypatch.setenv("FA_PAGES_TEST_SMTP", "hunter2") + config = tmp_path / "automation_file.toml" + config.write_text( + '[[notify.sinks]]\ntype = "email"\nname = "ops-mail"\nhost = "smtp.example.com"\n' + 'port = 587\nsender = "bot@example.com"\nrecipients = ["ops@example.com"]\n' + 'password = "${env:FA_PAGES_TEST_SMTP}"\n', + encoding="utf-8", + ) + page = _page("SettingsPage", services.settings, log) + page.preview() + assert "enter the path" in page.status_text() + page.set_path(str(config)) + page.preview() + assert "declares 1 sink(s) and 0 route(s); nothing was changed" in page.status_text() + assert page.summary()["applied"] is False + assert MASK in page.document_text() + assert "hunter2" not in page.document_text() + assert manager.names() == () + + page.apply() + assert page.status_text() == f"applied {config}: 1 sink(s), 0 route(s)" + assert manager.names() == ("ops-mail",) + assert page._environment_labels["applied_config"].text() == str(config) + assert "hunter2" not in page.document_text() + + page.set_path(str(tmp_path / "missing.toml")) + page.apply() + assert "not found" in page.status_text() + + +def test_the_settings_page_lists_the_extras_and_the_environment( + services: AppServices, log: Any, monkeypatch: pytest.MonkeyPatch +) -> None: + from automation_file.app import settings_service + from automation_file.core.optional import EXTRAS, install_hint + + monkeypatch.setattr(settings_service, "is_installed", lambda module: module != "boto3") + page = _page("SettingsPage", services.settings, log) + page.refresh() + assert [extra.name for extra in page.extras()] == list(EXTRAS) + assert _column(page._extras_table, 0) == list(EXTRAS) + row = list(EXTRAS).index("s3") + assert _column(page._extras_table, 2)[row] == "no (boto3 missing)" + assert _column(page._extras_table, 3)[row] == install_hint("s3") + assert _column(page._extras_table, 2)[list(EXTRAS).index("ftp")] == "yes" + assert page._environment_labels["python"].text().count(".") == 2 + assert page._environment_labels["applied_config"].text() == "none" diff --git a/tests/test_ui_pipeline_editor.py b/tests/test_ui_pipeline_editor.py new file mode 100644 index 0000000..95a0f9f --- /dev/null +++ b/tests/test_ui_pipeline_editor.py @@ -0,0 +1,950 @@ +"""The pipeline editor: canvas, task form and page, as thin views over a draft.""" + +from __future__ import annotations + +import itertools +import json +import os +import threading +from collections.abc import Iterator +from pathlib import Path +from typing import Any + +import pytest + +pytest.importorskip("PySide6") + +os.environ.setdefault("QT_QPA_PLATFORM", "offscreen") + +from automation_file.app import ( + PipelineDraft, + PipelineService, + layout_path, +) +from automation_file.core.action_registry import ActionRegistry +from automation_file.events import EventBus +from automation_file.pipeline import MemoryRunStore, Pipeline +from tests.ui_stand_in import SyncPool + +WAIT = 10.0 + + +class _Workshop: + """The actions the pipelines of these tests call.""" + + def __init__(self) -> None: + self.gate = threading.Event() + self.entered = threading.Event() + self.broken = True + self.calls: list[str] = [] + + def echo(self, value: Any = None, times: int = 1) -> Any: + self.calls.append(f"echo:{value!r}") + return value + + def fail(self) -> None: + raise ValueError("it broke") + + def flaky(self) -> str: + if self.broken: + raise ConnectionError("not yet") + return "repaired" + + def hold(self) -> str: + self.entered.set() + self.gate.wait(WAIT) + return "released" + + def positional(self, *values: Any) -> list[Any]: + return list(values) + + def registry(self) -> ActionRegistry: + return ActionRegistry( + { + "T_echo": self.echo, + "T_fail": self.fail, + "T_flaky": self.flaky, + "T_hold": self.hold, + "T_positional": self.positional, + } + ) + + +@pytest.fixture(name="qt_app", scope="module") +def _qt_app(): + from PySide6.QtWidgets import QApplication + + app = QApplication.instance() or QApplication([]) + yield app + + +@pytest.fixture(name="workshop") +def _workshop() -> Iterator[_Workshop]: + workshop = _Workshop() + yield workshop + workshop.gate.set() + + +@pytest.fixture(name="service") +def _service(workshop: _Workshop) -> PipelineService: + return PipelineService(MemoryRunStore(), registry=workshop.registry(), bus=EventBus()) + + +@pytest.fixture(name="page") +def _page( + qt_app, service: PipelineService, workshop: _Workshop, monkeypatch: pytest.MonkeyPatch +) -> Iterator[Any]: + from automation_file.ui.log_widget import LogPanel + from automation_file.ui.pages import PipelinesPage + + assert qt_app is not None + log = LogPanel() + page = PipelinesPage(service, log, SyncPool()) + monkeypatch.setattr(page, "confirm", lambda _question: True) + page.refresh() + yield page + # Let a run that is still held finish before the test ends, so its thread logs nothing later. + workshop.gate.set() + if page.current_run() is not None: + service.wait(page.current_run(), WAIT) + page.shutdown() + page.deleteLater() + log.deleteLater() + + +def _chain(page: Any, *actions: str) -> list[str]: + """Add one task per action and connect each to the one before it.""" + ids = [page.add_task(action) for action in actions] + for upstream, downstream in itertools.pairwise(ids): + page.canvas().select_tasks([upstream, downstream]) + page.connect_selected() + return ids + + +def _finish(page: Any, service: PipelineService) -> None: + assert service.wait(page.current_run(), WAIT) is True + page.poll() + + +# ---------------------------------------------------------------------- canvas + + +def test_the_canvas_shows_the_tasks_and_edges_of_the_draft(qt_app) -> None: + from automation_file.ui.pages import EdgeItem, PipelineCanvas, TaskNode + + assert qt_app is not None + draft = PipelineDraft("p") + draft.add_task("T_echo", "a", position=(10, 20)) + draft.add_task("T_echo", "b", position=(300, 40)) + draft.connect("a", "b") + canvas = PipelineCanvas() + canvas.set_draft(draft) + assert canvas.node_ids() == ["a", "b"] + assert canvas.edge_pairs() == [("a", "b")] + node = canvas.node("a") + assert isinstance(node, TaskNode) + assert (node.pos().x(), node.pos().y(), node.action) == (10.0, 20.0, "T_echo") + kinds = [type(item) for item in canvas.scene().items()] + assert kinds.count(TaskNode) == 2 + assert kinds.count(EdgeItem) == 1 + + draft.add_task("T_fail", "c") + draft.connect("b", "c") + assert canvas.node_ids() == ["a", "b", "c"] + assert canvas.edge_pairs() == [("a", "b"), ("b", "c")] + draft.rename_task("c", "last") + draft.remove_task("a") + assert canvas.node_ids() == ["b", "last"] + assert canvas.edge_pairs() == [("b", "last")] + canvas.set_draft(None) + assert canvas.node_ids() == [] + draft.add_task("T_echo", "ignored") + assert canvas.node_ids() == [] + + +def test_dragging_a_node_writes_its_position_to_the_draft_and_moves_its_arrows(qt_app) -> None: + from automation_file.ui.pages import PipelineCanvas + + assert qt_app is not None + draft = PipelineDraft("p") + draft.add_task("T_echo", "a", position=(0, 0)) + draft.add_task("T_echo", "b", position=(300, 0)) + draft.connect("a", "b") + canvas = PipelineCanvas() + canvas.set_draft(draft) + edge = next(item for item in canvas.scene().items() if hasattr(item, "upstream_id")) + before = edge.path().boundingRect() + revision = draft.revision + canvas.node("b").setPos(420.0, 180.0) + assert draft.positions()["b"] == (420.0, 180.0) + assert draft.revision == revision + 1 + assert canvas.node_ids() == ["a", "b"] + assert edge.path().boundingRect() != before + assert "420" not in json.dumps(draft.to_definition()) + + +def test_a_real_mouse_drag_moves_the_node(qt_app) -> None: + from PySide6.QtCore import QEvent, QPoint, QPointF, Qt + from PySide6.QtGui import QMouseEvent + + from automation_file.ui.pages import PipelineCanvas + + draft = PipelineDraft("p") + draft.add_task("T_echo", "a", position=(50, 50)) + canvas = PipelineCanvas() + canvas.resize(600, 400) + canvas.set_draft(draft) + canvas.show() + qt_app.processEvents() + start = canvas.mapFromScene(QPointF(60.0, 60.0)) + end = start + QPoint(120, 70) + viewport = canvas.viewport() + + def send(kind: QEvent.Type, point: QPoint, button: Any, buttons: Any) -> None: + event = QMouseEvent( + kind, + QPointF(point), + QPointF(viewport.mapToGlobal(point)), + button, + buttons, + Qt.KeyboardModifier.NoModifier, + ) + qt_app.sendEvent(viewport, event) + + left, none = Qt.MouseButton.LeftButton, Qt.MouseButton.NoButton + send(QEvent.Type.MouseButtonPress, start, left, left) + send(QEvent.Type.MouseMove, start + QPoint(60, 35), none, left) + send(QEvent.Type.MouseMove, end, none, left) + send(QEvent.Type.MouseButtonRelease, end, left, none) + canvas.hide() + x, y = draft.positions()["a"] + assert (round(x), round(y)) == (170, 120) + assert canvas.selected_tasks() == ["a"] + + +def test_positions_set_on_the_draft_move_the_nodes(qt_app) -> None: + from automation_file.ui.pages import PipelineCanvas + + assert qt_app is not None + draft = PipelineDraft("p") + draft.add_task("T_echo", "a", position=(500, 500)) + draft.add_task("T_echo", "b", position=(700, 700)) + draft.connect("a", "b") + canvas = PipelineCanvas() + canvas.set_draft(draft) + node_a, node_b = canvas.node("a"), canvas.node("b") + draft.auto_layout() + assert canvas.node("a") is node_a + assert (node_a.pos().x(), node_a.pos().y()) == (40.0, 40.0) + assert (node_b.pos().x(), node_b.pos().y()) == (280.0, 40.0) + draft.set_position("a", 5, 6) + assert (node_a.pos().x(), node_a.pos().y()) == (5.0, 6.0) + + +def test_the_selection_keeps_the_order_in_which_tasks_were_selected(qt_app) -> None: + from automation_file.ui.pages import PipelineCanvas + + assert qt_app is not None + draft = PipelineDraft("p") + for task_id in ("a", "b", "c"): + draft.add_task("T_echo", task_id) + canvas = PipelineCanvas() + canvas.set_draft(draft) + heard: list[list[str]] = [] + canvas.selection_changed.connect(heard.append) + canvas.node("c").setSelected(True) + canvas.node("a").setSelected(True) + assert canvas.selected_tasks() == ["c", "a"] + assert heard[-1] == ["c", "a"] + canvas.node("c").setSelected(False) + canvas.node("b").setSelected(True) + assert canvas.selected_tasks() == ["a", "b"] + canvas.select_tasks(["b", "c", "missing"]) + assert canvas.selected_tasks() == ["b", "c"] + draft.set_action("a", "T_fail") + assert canvas.selected_tasks() == ["b", "c"] + assert canvas.node("b").isSelected() is True + draft.remove_task("b") + assert canvas.selected_tasks() == ["c"] + + +def test_an_arrow_can_be_selected(qt_app) -> None: + from automation_file.ui.pages import PipelineCanvas + + assert qt_app is not None + draft = PipelineDraft("p") + draft.add_task("T_echo", "a") + draft.add_task("T_echo", "b", position=(300, 0)) + draft.connect("a", "b") + canvas = PipelineCanvas() + canvas.set_draft(draft) + assert canvas.selected_edges() == [] + assert canvas.select_edge("a", "b") is True + assert canvas.selected_edges() == [("a", "b")] + assert canvas.selected_tasks() == [] + assert canvas.select_edge("b", "a") is False + + +def test_statuses_colour_the_nodes_and_the_scene_paints(qt_app) -> None: + from PySide6.QtGui import QColor, QImage, QPainter + + from automation_file.ui.pages import PipelineCanvas + from automation_file.ui.pages.pipeline_canvas import STATUS_COLOURS + + assert qt_app is not None + draft = PipelineDraft("p") + draft.add_task("T_echo", "a", position=(0, 0)) + draft.add_task("", "b", position=(300, 0)) + draft.connect("a", "b") + canvas = PipelineCanvas() + canvas.set_draft(draft) + canvas.set_statuses({"a": "succeeded", "b": "failed"}) + assert (canvas.node("a").status, canvas.node("b").status) == ("succeeded", "failed") + canvas.node("a").setSelected(True) + canvas.select_edge("a", "b") + image = QImage(600, 200, QImage.Format.Format_ARGB32) + image.fill(QColor("white")) + painter = QPainter(image) + canvas.scene().render(painter) + painter.end() + colours = {image.pixelColor(x, y).name() for x in range(0, 600, 4) for y in range(0, 200, 4)} + assert STATUS_COLOURS["succeeded"] in colours + assert STATUS_COLOURS["failed"] in colours + draft.add_task("T_echo", "c") + assert canvas.node("a").status == "succeeded" + canvas.set_statuses({}) + assert canvas.node("a").status == "" + + +def test_the_palette_filters_and_offers_an_action_for_dragging(qt_app) -> None: + from automation_file.ui.pages import ACTION_MIME, ActionPalette + + assert qt_app is not None + palette = ActionPalette() + palette.set_actions(["FA_storage_copy", "FA_storage_delete", "FA_zip_dir"]) + palette.filter_actions("STORAGE") + assert [palette.item(row).isHidden() for row in range(3)] == [False, False, True] + palette.setCurrentRow(1) + assert palette.selected_action() == "FA_storage_delete" + palette.filter_actions("zip") + assert palette.selected_action() == "" + palette.filter_actions("") + data = palette.mimeData([palette.item(2)]) + assert bytes(data.data(ACTION_MIME).data()).decode("utf-8") == "FA_zip_dir" + assert data.text() == "FA_zip_dir" + assert palette.mimeData([]).hasFormat(ACTION_MIME) is False + + +def test_dropping_an_action_on_the_canvas_adds_a_task_where_it_was_dropped(page: Any) -> None: + from PySide6.QtCore import QMimeData, QPointF, Qt + from PySide6.QtGui import QDropEvent + + from automation_file.ui.pages import ACTION_MIME + from automation_file.ui.pages.pipeline_canvas import NODE_HEIGHT, NODE_WIDTH + + canvas = page.canvas() + canvas.resize(600, 400) + data = QMimeData() + data.setData(ACTION_MIME, b"T_echo") + point = QPointF(200.0, 150.0) + event = QDropEvent( + point, + Qt.DropAction.CopyAction, + data, + Qt.MouseButton.LeftButton, + Qt.KeyboardModifier.NoModifier, + ) + canvas.dropEvent(event) + assert page.draft().task_ids() == ("T_echo",) + scene_point = canvas.mapToScene(point.toPoint()) + x, y = page.draft().positions()["T_echo"] + assert (x, y) == (scene_point.x() - NODE_WIDTH / 2, scene_point.y() - NODE_HEIGHT / 2) + assert canvas.selected_tasks() == ["T_echo"] + + other = QMimeData() + other.setText("not an action") + canvas.dropEvent( + QDropEvent( + point, + Qt.DropAction.CopyAction, + other, + Qt.MouseButton.LeftButton, + Qt.KeyboardModifier.NoModifier, + ) + ) + assert page.draft().task_ids() == ("T_echo",) + + +# ---------------------------------------------------------------------- tasks and edges on the page + + +def test_the_palette_lists_the_registered_actions(page: Any) -> None: + palette = page.action_palette() + assert [palette.item(row).text() for row in range(palette.count())] == [ + "T_echo", + "T_fail", + "T_flaky", + "T_hold", + "T_positional", + ] + page._filter.setText("fl") + assert [palette.item(row).isHidden() for row in range(palette.count())] == [ + True, + True, + False, + True, + True, + ] + + +def test_adding_a_task_uses_the_selected_action_and_selects_the_new_node(page: Any) -> None: + page.action_palette().setCurrentRow(1) + first = page.add_task() + assert first == "T_fail" + assert page.draft().task("T_fail").action == "T_fail" + assert page.canvas().selected_tasks() == ["T_fail"] + assert page.form().current_task() == "T_fail" + page.action_palette().itemDoubleClicked.emit(page.action_palette().item(0)) + assert page.draft().task_ids() == ("T_fail", "T_echo") + assert page.status_text() == "added task T_echo" + + +def test_two_selected_tasks_are_connected_in_the_order_they_were_selected(page: Any) -> None: + first, second = page.add_task("T_echo"), page.add_task("T_echo") + assert page._connect_button.text() == "Connect (select two tasks)" + page.connect_selected() + assert "select exactly two tasks" in page.status_text() + page.canvas().select_tasks([second, first]) + assert page._connect_button.text() == f"Connect {second} -> {first}" + page.connect_selected() + assert page.draft().edges() == [(second, first)] + assert page.canvas().edge_pairs() == [(second, first)] + assert page.status_text() == f"{first} now depends on {second}" + page.connect_selected() + assert page.status_text() == f"{first} already depends on {second}" + page.canvas().select_tasks([first, second]) + page.connect_selected() + assert "dependency cycle" in page.status_text() + assert page.draft().edges() == [(second, first)] + + +def test_disconnecting_by_arrow_and_by_two_tasks(page: Any) -> None: + a, b, c = _chain(page, "T_echo", "T_echo", "T_echo") + assert page.draft().edges() == [(a, b), (b, c)] + page.canvas().select_tasks([]) + page.disconnect_selected() + assert "select an arrow" in page.status_text() + page.canvas().select_edge(a, b) + page.disconnect_selected() + assert page.draft().edges() == [(b, c)] + assert page.status_text() == f"removed {a} -> {b}" + page.canvas().select_tasks([c, b]) + page.disconnect_selected() + assert page.draft().edges() == [] + + +def test_remove_takes_the_selected_arrows_first_and_tasks_otherwise(page: Any) -> None: + a, b, c = _chain(page, "T_echo", "T_echo", "T_echo") + page.canvas().select_tasks([]) + page.remove_selected() + assert "select a task or an arrow" in page.status_text() + page.canvas().select_edge(b, c) + page.remove_selected() + assert page.draft().task_ids() == (a, b, c) + assert page.draft().edges() == [(a, b)] + page.canvas().select_tasks([a, c]) + page.remove_selected() + assert page.draft().task_ids() == (b,) + assert page.canvas().node_ids() == [b] + assert page.form().current_task() is None + + +def test_auto_layout_arranges_the_nodes_by_dependency_depth(page: Any) -> None: + a, b = _chain(page, "T_echo", "T_echo") + page.canvas().node(b).setPos(900.0, 900.0) + page.auto_layout() + assert page.draft().positions() == {a: (40.0, 40.0), b: (280.0, 40.0)} + assert page.canvas().node(b).pos().x() == 280.0 + + +# ---------------------------------------------------------------------- the task form + + +def test_the_form_shows_the_selected_task_and_its_parameters(page: Any) -> None: + form = page.form() + assert form.isEnabled() is False + task_id = page.add_task("T_echo") + assert form.isEnabled() is True + assert form._task_id.text() == task_id + assert form._action.currentText() == "T_echo" + assert form._signature.text().startswith("T_echo(value=None, times=1)") + editor = form.arguments_editor() + assert editor.row_names() == ["value", "times"] + assert editor._table.item(0, 2).text() == "null" + assert editor._table.item(1, 2).text() == "1" + page.canvas().select_tasks([]) + assert form.isEnabled() is False + assert form.current_task() is None + assert "Select a task" in form.message() + + +def test_applying_the_form_writes_every_field_through_the_draft(page: Any) -> None: + first, second = page.add_task("T_echo"), page.add_task("T_echo") + form = page.form() + assert form.current_task() == second + editor = form.arguments_editor() + editor.set_value("value", "${tasks." + first + ".result}") + editor.set_value("times", "3") + editor.add_row("extra", '{"deep": [1, 2]}') + assert form.set_dependency(first, True) is True + assert form.set_dependency("missing", True) is False + form.set_retry(4, 1.5) + form.set_field("retry_on", "ConnectionError, TimeoutError") + form.set_field("timeout", "30") + form.set_condition("always") + form.set_field("key", "copy-${params.date}") + assert form.apply() is True + assert page.draft().task(second).to_spec() == { + "action": [ + "T_echo", + {"value": "${tasks." + first + ".result}", "times": 3, "extra": {"deep": [1, 2]}}, + ], + "depends_on": [first], + "retry": {"max_attempts": 4, "backoff": 1.5, "on": ["ConnectionError", "TimeoutError"]}, + "timeout": 30.0, + "when": "always", + "idempotency_key": "copy-${params.date}", + } + assert page.canvas().edge_pairs() == [(first, second)] + assert form.checked_dependencies() == [first] + assert page.status_text() == f"task {second} updated" + assert editor.row_names() == ["value", "times", "extra"] + + +def test_renaming_through_the_form_keeps_the_task_selected(page: Any) -> None: + first, second = _chain(page, "T_echo", "T_echo") + page.canvas().select_tasks([first]) + form = page.form() + form.set_field("task_id", "download") + assert form.apply() is True + assert page.draft().task_ids() == ("download", second) + assert page.draft().edges() == [("download", second)] + assert page.canvas().selected_tasks() == ["download"] + assert form.current_task() == "download" + form.set_field("task_id", second) + assert form.apply() is False + assert "already has a task" in form.message() + assert form.current_task() == "download" + assert form._task_id.text() == second + form.revert() + assert form._task_id.text() == "download" + + +def test_what_was_typed_survives_an_arrow_drawn_on_the_canvas(page: Any) -> None: + first, second = page.add_task("T_echo"), page.add_task("T_echo") + page.canvas().select_tasks([first, second]) + form = page.form() + assert form.current_task() == second + form.arguments_editor().set_value("value", "typed") + form.set_field("timeout", "12") + assert form.checked_dependencies() == [] + page.connect_selected() + assert form.current_task() == second + assert form._timeout.text() == "12" + assert form.arguments_editor().arguments() == {"value": "typed"} + assert form.checked_dependencies() == [first] + assert form.apply() is True + assert page.draft().edges() == [(first, second)] + assert page.draft().task(second).timeout == 12.0 + assert page.draft().task(second).arguments == {"value": "typed"} + + +def test_a_field_the_draft_refuses_is_reported_and_what_was_typed_stays(page: Any) -> None: + first, second = _chain(page, "T_echo", "T_echo") + page.canvas().select_tasks([first]) + form = page.form() + form.set_field("timeout", "soon") + assert form.apply() is False + assert "timeout must be a number" in form.message() + assert form._timeout.text() == "soon" + form.set_field("timeout", "") + form.set_dependency(second, True) + assert form.apply() is False + assert "dependency cycle" in form.message() + assert page.draft().edges() == [(first, second)] + assert page.canvas().selected_tasks() == [first] + + +def test_changing_the_action_rebuilds_the_argument_rows_and_keeps_typed_values(page: Any) -> None: + page.add_task("T_echo") + form, editor = page.form(), page.form().arguments_editor() + editor.set_value("value", "kept") + form.set_action("T_missing") + assert "not a registered action" in form._signature.text() + assert editor.row_names() == ["value"] + assert editor.arguments() == {"value": "kept"} + form.set_action("T_echo") + assert editor.row_names() == ["value", "times"] + assert editor.arguments() == {"value": "kept"} + form.set_action("") + assert form._signature.text() == "Choose an action." + + +def test_positional_arguments_are_edited_as_json(page: Any) -> None: + task_id = page.add_task("T_positional") + form, editor = page.form(), page.form().arguments_editor() + page.draft().set_arguments(task_id, ["a", 2]) + assert editor.is_json_mode() is False + form.revert() + assert editor.is_json_mode() is True + assert json.loads(editor._raw.toPlainText()) == ["a", 2] + editor.set_json_mode(False) + assert editor.is_json_mode() is True + editor.set_json_text('["b", 3, true]') + assert form.apply() is True + assert page.draft().task(task_id).arguments == ["b", 3, True] + editor.set_json_text("[oops") + assert form.apply() is False + assert "not valid JSON" in form.message() + editor.set_json_mode(False) + assert editor.is_json_mode() is True + editor.set_json_text('"just text"') + assert form.apply() is False + assert "JSON object, a JSON array or empty" in form.message() + editor.set_json_text("") + assert form.apply() is True + assert page.draft().task(task_id).arguments is None + + +def test_keyword_arguments_move_between_the_table_and_json(page: Any) -> None: + page.add_task("T_echo") + editor = page.form().arguments_editor() + editor.set_value("value", "hello") + editor.set_json_mode(True) + assert json.loads(editor._raw.toPlainText()) == {"value": "hello"} + editor.set_json_text('{"value": "changed", "added": 5}') + editor.set_json_mode(False) + assert editor.is_json_mode() is False + assert editor.row_names() == ["value", "times", "added"] + assert editor.arguments() == {"value": "changed", "added": 5} + + +# ---------------------------------------------------------------------- header and files + + +def test_the_header_fields_are_written_to_the_draft_only_when_they_change(page: Any) -> None: + draft = page.draft() + page._name.editingFinished.emit() + assert draft.dirty is False + page._name.setText("nightly") + page._name.editingFinished.emit() + page._description.setText("Fetch and publish") + page._description.editingFinished.emit() + page._max_workers.setValue(2) + page._defaults.setText('{"date": "2026-10-08"}') + page._defaults.editingFinished.emit() + page._cron.setText("0 2 * * *") + page._cron.editingFinished.emit() + assert draft.to_definition() == { + "schema_version": 1, + "name": "nightly", + "description": "Fetch and publish", + "max_workers": 2, + "schedule": {"cron": "0 2 * * *"}, + "params": {"date": "2026-10-08"}, + "tasks": {}, + } + assert page._file_label.text() == "not saved yet (modified)" + page._defaults.setText("{oops") + page._defaults.editingFinished.emit() + assert "default parameters is not valid JSON" in page.status_text() + assert draft.params == {"date": "2026-10-08"} + + +def test_a_pipeline_is_saved_and_opened_with_its_layout(page: Any, tmp_path: Path) -> None: + first, second = _chain(page, "T_echo", "T_fail") + page.canvas().node(second).setPos(432.0, 234.0) + page._name.setText("saved") + path = tmp_path / "saved.yaml" + page.save_pipeline_as(str(path)) + assert page.status_text() == f"saved {path}" + assert page._file_label.text() == str(path) + assert Pipeline.from_file(path).name == "saved" + assert json.loads(layout_path(path).read_text(encoding="utf-8"))["positions"][second] == [ + 432.0, + 234.0, + ] + + page.new_pipeline() + assert page.draft().tasks == () + assert page.canvas().node_ids() == [] + assert page._file_label.text() == "not saved yet" + page.open_pipeline(str(path)) + assert page.status_text() == f"opened {path}: 2 task(s)" + assert page.draft().task_ids() == (first, second) + assert page.canvas().edge_pairs() == [(first, second)] + assert page.canvas().node(second).pos().x() == 432.0 + assert page._name.text() == "saved" + + page.canvas().node(first).setPos(1.0, 2.0) + assert page._file_label.text() == f"{path} (modified)" + page.save_pipeline() + assert page._file_label.text() == str(path) + assert json.loads(layout_path(path).read_text(encoding="utf-8"))["positions"][first] == [ + 1.0, + 2.0, + ] + page.save_pipeline_as(str(tmp_path / "saved.txt")) + assert "cannot save" in page.status_text() + + +def test_unsaved_changes_are_kept_when_the_user_says_so( + page: Any, tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + page.add_task("T_echo") + monkeypatch.setattr(page, "confirm", lambda _question: False) + page.new_pipeline() + page.open_pipeline(str(tmp_path / "missing.yaml")) + assert page.draft().task_ids() == ("T_echo",) + monkeypatch.setattr(page, "confirm", lambda _question: True) + page.open_pipeline(str(tmp_path / "missing.yaml")) + assert "open" in page.status_text() and "failed" in page.status_text() + assert page.draft().task_ids() == ("T_echo",) + + +def test_a_definition_with_problems_opens_and_lists_them(page: Any, tmp_path: Path) -> None: + path = tmp_path / "broken.json" + path.write_text( + json.dumps( + { + "schema_version": 1, + "name": "broken", + "tasks": {"a": {"action": ["T_echo"], "depends_on": ["ghost"]}}, + } + ), + encoding="utf-8", + ) + page.open_pipeline(str(path)) + assert "1 problem(s) to fix" in page.status_text() + assert page.panel().problem_texts() == ["tasks.a.depends_on[0]: unknown task 'ghost'"] + assert page.canvas().node_ids() == ["a"] + + +# ---------------------------------------------------------------------- validate, dry run, test + + +def test_validate_lists_every_problem_and_a_problem_leads_to_its_task(page: Any) -> None: + panel = page.panel() + assert panel.problem_texts() == [] + assert panel.tabText(0) == "Problems" + assert page.validate() is False + assert panel.problem_texts() == ["tasks: at least one task is required"] + good, bad = page.add_task("T_echo"), page.add_task("T_missing") + page.draft().set_timeout(good, -1) + assert page.validate() is False + assert panel.problem_texts() == [ + f"tasks.{good}.timeout: expected a number of seconds > 0, got -1", + f"tasks.{bad}.action[0]: unknown action 'T_missing'", + ] + assert panel.tabText(0) == "Problems (2)" + assert page.status_text() == "2 problem(s): see the Problems tab" + page.canvas().select_tasks([]) + panel._problem_list.itemClicked.emit(panel._problem_list.item(1)) + assert page.canvas().selected_tasks() == [bad] + page.draft().set_timeout(good, None) + page.draft().set_action(bad, "T_echo") + assert page.validate() is True + assert page.status_text() == "the definition is valid" + + +def test_a_dry_run_shows_the_plan_on_the_canvas_and_in_the_table( + page: Any, workshop: _Workshop +) -> None: + first, second = _chain(page, "T_echo", "T_echo") + page.draft().set_arguments(first, {"value": "${params.word}"}) + page.dry_run() + assert "unknown parameter 'word'" in page.status_text() + page.set_run_params('{"word": "hi"}') + page.dry_run() + assert page.status_text() == "dry run: 2 task(s) would run as planned" + assert page.panel().task_rows() == [(first, "planned"), (second, "planned")] + assert page.panel().run_text() == "dry run: succeeded" + assert page.canvas().node(first).status == "planned" + assert workshop.calls == [] + assert page.current_run() is None + + +def test_the_run_parameters_must_be_a_json_object(page: Any, workshop: _Workshop) -> None: + page.add_task("T_echo") + page.set_run_params("{oops") + page.run() + assert "run parameters is not valid JSON" in page.status_text() + page.set_run_params("[1, 2]") + page.dry_run() + assert "must be a JSON object" in page.status_text() + assert workshop.calls == [] + + +def test_nothing_runs_while_the_definition_has_problems(page: Any, workshop: _Workshop) -> None: + page.add_task("T_missing") + for attempt in (page.dry_run, page.run, page.test_task): + attempt() + assert page.status_text() == "1 problem(s): see the Problems tab" + assert page.current_run() is None + assert workshop.calls == [] + + +def test_one_task_is_tested_alone(page: Any, workshop: _Workshop, service: Any) -> None: + first, second = _chain(page, "T_fail", "T_echo") + page.canvas().select_tasks([]) + page.test_task() + assert page.status_text() == "select the task to test" + page.canvas().select_tasks([second]) + page.draft().set_arguments(second, {"value": "probe"}) + page.test_task() + assert page.status_text() == f"test of {second} succeeded in 1 attempt(s)" + assert page.panel().task_rows() == [(second, "succeeded")] + assert "task.completed" in page.panel().log_text() + assert workshop.calls == ["echo:'probe'"] + assert service.history() == [] + page.canvas().select_tasks([first]) + page.test_task() + assert page.status_text() == f"test of {first}: failed: ValueError: it broke" + + +# ---------------------------------------------------------------------- runs + + +def test_a_run_is_followed_until_it_ends(page: Any, workshop: _Workshop, service: Any) -> None: + first, second = _chain(page, "T_hold", "T_echo") + page.run() + run_id = page.current_run() + assert run_id is not None + assert page._timer.isActive() is True + assert workshop.entered.wait(WAIT) + page.poll() + assert page.panel().task_rows() == [(first, "running"), (second, "pending")] + assert page.canvas().node(first).status == "running" + assert "task.started" in page.panel().log_text() + workshop.gate.set() + _finish(page, service) + assert page.status_text() == f"run {run_id} succeeded" + assert page.panel().task_rows() == [(first, "succeeded"), (second, "succeeded")] + assert page.canvas().node(second).status == "succeeded" + assert "pipeline.completed" in page.panel().log_text() + assert page._timer.isActive() is False + assert page.panel().history_ids() == [run_id] + + +def test_a_failed_run_is_resumed_and_retried(page: Any, workshop: _Workshop, service: Any) -> None: + first, second = _chain(page, "T_echo", "T_flaky") + page.resume() + assert "there is no run to resume" in page.status_text() + page.run() + failed = page.current_run() + _finish(page, service) + assert page.status_text().startswith(f"run {failed} ended failed") + assert page.panel().task_rows() == [(first, "succeeded"), (second, "failed")] + assert page.canvas().node(second).status == "failed" + + workshop.broken = False + page.resume() + assert page.current_run() == failed + _finish(page, service) + assert page.status_text() == f"run {failed} succeeded" + assert workshop.calls.count("echo:None") == 1 + + page.retry() + again = page.current_run() + assert again != failed + _finish(page, service) + assert page.status_text() == f"run {again} succeeded" + assert workshop.calls.count("echo:None") == 2 + assert page.panel().history_ids() == [again, failed] + + +def test_a_running_run_is_cancelled(page: Any, workshop: _Workshop, service: Any) -> None: + page.cancel() + assert page.status_text() == "there is no run to cancel" + first, second = _chain(page, "T_hold", "T_echo") + page.run() + run_id = page.current_run() + assert workshop.entered.wait(WAIT) + page.cancel() + assert page.status_text() == f"cancel requested for run {run_id}" + workshop.gate.set() + _finish(page, service) + assert page.status_text().startswith(f"run {run_id} ended cancelled") + assert page.panel().task_rows() == [(first, "succeeded"), (second, "cancelled")] + page.cancel() + assert "is not running in this process" in page.status_text() + + +def test_a_run_of_the_history_can_be_followed_again(page: Any, service: Any) -> None: + _chain(page, "T_echo") + page.run() + first_run = page.current_run() + _finish(page, service) + page.run() + second_run = page.current_run() + _finish(page, service) + assert page.panel().history_ids() == [second_run, first_run] + page.follow_selected() + assert "select a run" in page.status_text() + assert page.panel().select_run(first_run) is True + assert page.panel().select_run("unknown") is False + page.follow_selected() + assert page.current_run() == first_run + assert page.panel().run_text() == f"run {first_run}: succeeded" + page.panel().run_chosen.emit(second_run) + assert page.current_run() == second_run + + +def test_following_leaves_the_visible_tab_alone_and_stops_when_the_run_cannot_be_read( + page: Any, workshop: _Workshop, service: Any +) -> None: + page.add_task("T_hold") + page.run() + panel = page.panel() + assert panel.tabText(panel.currentIndex()) == "Tasks" + panel.setCurrentIndex(2) + assert workshop.entered.wait(WAIT) + page.poll() + assert panel.tabText(panel.currentIndex()) == "Log" + workshop.gate.set() + _finish(page, service) + assert panel.tabText(panel.currentIndex()) == "Log" + + page.follow_run("no-such-run") + assert page.status_text() == "follow run failed: unknown run 'no-such-run'" + assert page._timer.isActive() is False + page.poll() + assert page._timer.isActive() is False + + +def test_a_new_draft_forgets_the_followed_run(page: Any, service: Any) -> None: + task_id = page.add_task("T_echo") + page.run() + _finish(page, service) + assert page.canvas().node(task_id).status == "succeeded" + page.new_pipeline() + assert page.current_run() is None + assert page.panel().task_rows() == [] + assert page.panel().log_text() == "" + page.poll() + assert page._timer.isActive() is False + + +def test_shutdown_stops_following_and_lets_go_of_the_draft( + page: Any, workshop: _Workshop, service: Any +) -> None: + page.add_task("T_hold") + page.run() + assert page._timer.isActive() is True + draft = page.draft() + page.shutdown() + assert page._timer.isActive() is False + draft.add_task("T_echo", "later") + assert page.canvas().node_ids() == [] + workshop.gate.set() + assert service.wait(page.current_run(), WAIT) is True diff --git a/tests/test_ui_smoke.py b/tests/test_ui_smoke.py index bf5164e..e01b994 100644 --- a/tests/test_ui_smoke.py +++ b/tests/test_ui_smoke.py @@ -1,13 +1,15 @@ -"""UI smoke tests — construct every tab with the offscreen Qt platform. +"""UI smoke tests — construct the main window, every page and every tab offscreen. These tests don't exercise the event loop; they just confirm the widget tree builds without raising, which catches import errors, bad signal wiring, and -drift between ops-module signatures and tab form fields. +drift between ops-module signatures and tab form fields. What the pages do with +their services is in ``test_ui_pages.py`` and ``test_ui_pipeline_editor.py``. """ from __future__ import annotations import os +from collections.abc import Iterator import pytest @@ -15,6 +17,30 @@ os.environ.setdefault("QT_QPA_PLATFORM", "offscreen") +NAVIGATION = ( + "Dashboard", + "Files", + "Storage", + "Pipelines", + "Scheduler", + "Integrity", + "Audit", + "Notifications", + "Settings", +) +ADVANCED_TOOLS = ("Local", "Transfer", "Progress", "JSON actions", "Triggers", "Servers") +PAGES = ( + ("Dashboard", "DashboardPage", "dashboard"), + ("Files", "FilesPage", "files"), + ("Storage", "StoragePage", "storage"), + ("Pipelines", "PipelinesPage", "pipelines"), + ("Scheduler", "SchedulerPage", "scheduler"), + ("Integrity", "IntegrityPage", "integrity"), + ("Audit", "AuditPage", "audit"), + ("Notifications", "NotificationsPage", "notifications"), + ("Settings", "SettingsPage", "settings"), +) + @pytest.fixture(name="qt_app", scope="module") def _qt_app(): @@ -24,6 +50,18 @@ def _qt_app(): yield app +@pytest.fixture(name="window") +def _window(qt_app) -> Iterator: + from automation_file.ui.main_window import MainWindow + + assert qt_app is not None + window = MainWindow() + try: + yield window + finally: + window.close() + + def test_launch_ui_is_lazy_facade_attr() -> None: import automation_file @@ -31,15 +69,177 @@ def test_launch_ui_is_lazy_facade_attr() -> None: assert callable(launcher) -def test_main_window_constructs(qt_app) -> None: +def test_launch_ui_shows_the_main_window_and_returns_the_exit_code( + qt_app, monkeypatch: pytest.MonkeyPatch +) -> None: + from PySide6.QtWidgets import QApplication + + from automation_file.ui import launcher + from automation_file.ui.main_window import MainWindow + + shown: list[MainWindow] = [] + monkeypatch.setattr(MainWindow, "show", lambda self: shown.append(self)) + monkeypatch.setattr(QApplication, "exec", lambda *_args: 7) + assert qt_app is not None + try: + assert launcher.launch_ui([]) == 7 + assert len(shown) == 1 + assert shown[0].current_page_name() == "Dashboard" + finally: + for window in shown: + window.close() + + +def test_launch_ui_names_the_gui_extra_when_pyside6_is_missing( + monkeypatch: pytest.MonkeyPatch, +) -> None: + from automation_file.core.optional import install_hint + from automation_file.exceptions import OptionalDependencyException + from automation_file.ui import launcher + + asked: list[tuple[str, str]] = [] + + def missing(name: str, *, extra: str) -> None: + asked.append((name, extra)) + raise OptionalDependencyException(f"PySide6 is not installed: {install_hint(extra)}") + + monkeypatch.setattr(launcher, "require_module", missing) + with pytest.raises(OptionalDependencyException, match=r"automation_file\[gui\]"): + launcher.launch_ui([]) + assert asked == [("PySide6.QtWidgets", "gui")] + + +def test_main_window_constructs(window) -> None: + assert window.windowTitle() == "automation_file" + + +def test_closing_the_window_shuts_every_page_down(qt_app) -> None: from automation_file.ui.main_window import MainWindow assert qt_app is not None window = MainWindow() + closed: list[str] = [] + for name in window.page_names(): + page = window.page(name) + original = page.shutdown + + def shutdown(original=original, name=name) -> None: + closed.append(name) + original() + + page.shutdown = shutdown # type: ignore[method-assign] + window.close() + assert closed == [*NAVIGATION, "Advanced"] + assert window.page("Dashboard")._timer.isActive() is False + + +def test_the_sidebar_lists_the_nine_workflows_and_then_advanced(window) -> None: + from automation_file import app + + assert app.NAVIGATION == NAVIGATION + assert window.page_names() == [*NAVIGATION, "Advanced"] + assert window.current_page_name() == "Dashboard" + + +@pytest.mark.parametrize(("name", "page_class", "_service"), PAGES) +def test_every_navigation_entry_has_its_page(window, name: str, page_class: str, _service) -> None: + from automation_file.ui import pages + + assert isinstance(window.page(name), getattr(pages, page_class)) + assert window.page(name).title == name + assert window.navigate(name) is True + assert window.current_page_name() == name + + +def test_the_old_tabs_are_reachable_under_advanced(window) -> None: + from automation_file.ui import tabs + from automation_file.ui.pages import AdvancedPage + + advanced = window.page("Advanced") + assert isinstance(advanced, AdvancedPage) + assert advanced.tool_names() == list(ADVANCED_TOOLS) + expected = { + "Local": tabs.LocalOpsTab, + "Transfer": tabs.TransferTab, + "Progress": tabs.ProgressTab, + "JSON actions": tabs.JSONEditorTab, + "Triggers": tabs.TriggerTab, + "Servers": tabs.ServerTab, + } + for name, tab_class in expected.items(): + assert isinstance(advanced.tool(name), tab_class) + assert window.open_tool(name) is True + assert window.current_page_name() == "Advanced" + assert advanced.current_tool() == name + assert window.open_tool("No such tool") is False + assert advanced.tool("No such tool") is None + + +def test_every_cloud_backend_panel_is_still_under_transfer(window) -> None: + from automation_file.ui import tabs + + transfer = window.page("Advanced").tool("Transfer") + panels = { + "HTTP download": tabs.HTTPDownloadTab, + "Google Drive": tabs.GoogleDriveTab, + "Amazon S3": tabs.S3Tab, + "Azure Blob": tabs.AzureBlobTab, + "Dropbox": tabs.DropboxTab, + "SFTP": tabs.SFTPTab, + "OneDrive": tabs.OneDriveTab, + "Box": tabs.BoxTab, + } + for label, panel_class in panels.items(): + assert isinstance(transfer.inner_widget(label), panel_class) + assert transfer.select_backend(label) is True + + +def test_navigating_to_an_unknown_page_changes_nothing(window) -> None: + assert window.navigate("Files") is True + assert window.navigate("Nowhere") is False + assert window.current_page_name() == "Files" + + +@pytest.mark.parametrize(("name", "page_class", "service"), PAGES) +def test_each_page_constructs(qt_app, name: str, page_class: str, service: str) -> None: + from PySide6.QtCore import QThreadPool + + from automation_file.app import build_services + from automation_file.ui import pages + from automation_file.ui.log_widget import LogPanel + + assert qt_app is not None + log = LogPanel() + page = getattr(pages, page_class)( + getattr(build_services(), service), log, QThreadPool.globalInstance() + ) try: - assert window.windowTitle() == "automation_file" + assert page.title == name + assert page.status_text() == "" finally: - window.close() + page.shutdown() + page.deleteLater() + log.deleteLater() + + +def test_the_advanced_page_constructs_on_its_own(qt_app) -> None: + from PySide6.QtCore import QThreadPool + from PySide6.QtGui import QCloseEvent + + from automation_file.ui import pages + from automation_file.ui.log_widget import LogPanel + from automation_file.ui.pages import AdvancedPage + + assert qt_app is not None + log = LogPanel() + page = AdvancedPage(log, QThreadPool.globalInstance()) + try: + assert pages.ADVANCED_TOOLS == ADVANCED_TOOLS + assert page.tool_names() == list(ADVANCED_TOOLS) + finally: + page.close_tools(QCloseEvent()) + page.deleteLater() + log.deleteLater() @pytest.mark.parametrize( diff --git a/tests/test_web_ui_app_layer.py b/tests/test_web_ui_app_layer.py new file mode 100644 index 0000000..73f32c0 --- /dev/null +++ b/tests/test_web_ui_app_layer.py @@ -0,0 +1,260 @@ +"""The Web UI renders its fragments from the application layer, escaped and masked.""" +# pylint: disable=cyclic-import + +from __future__ import annotations + +import urllib.error +import urllib.request +from collections.abc import Iterator +from typing import Any + +import pytest + +from automation_file.app import MASK, NAVIGATION, AppServices, ServiceOptions, build_services +from automation_file.audit import AuditTrail, MemoryAuditStore +from automation_file.core.action_registry import ActionRegistry +from automation_file.events import Event, EventBus, PipelineFailed +from automation_file.integrity import IntegrityException +from automation_file.integrity import actions as integrity_actions +from automation_file.notify import NotificationManager, NotificationRouter +from automation_file.pipeline import MemoryRunStore +from automation_file.server.web_ui import WebUIServer, start_web_ui +from automation_file.storage import File, MemoryStorage, StorageResolver, clear_memory_stores +from tests._insecure_fixtures import insecure_url + +TREE = "memory://webui-tree/data" +BASELINE = "memory://webui-state/data.json" +SCRIPT = "" +WAIT = 10.0 + + +def _fail() -> None: + raise ValueError(f"it broke {SCRIPT}") + + +@pytest.fixture(autouse=True) +def _clean_global_state() -> Iterator[None]: + clear_memory_stores() + yield + integrity_actions.stop_all_monitors() + clear_memory_stores() + + +@pytest.fixture(name="bus") +def _bus() -> EventBus: + return EventBus() + + +@pytest.fixture(name="services") +def _services(bus: EventBus) -> Iterator[AppServices]: + manager = NotificationManager() + router = NotificationRouter(manager, bus) + trail = AuditTrail(bus=bus) + resolver = StorageResolver(defaults=False) + resolver.mount("memory://webui", MemoryStorage()) + yield build_services( + ServiceOptions( + resolver=resolver, + run_store=MemoryRunStore(), + registry=ActionRegistry({"T_echo": lambda value=None: value, "T_fail": _fail}), + bus=bus, + audit_trail=trail, + notification_manager=manager, + notification_router=router, + ) + ) + router.stop() + trail.close() + + +@pytest.fixture(name="server") +def _server(services: AppServices) -> Iterator[WebUIServer]: + server = start_web_ui(host="127.0.0.1", port=0, services=services) + yield server + server.shutdown() + server.server_close() + + +def _get(server: WebUIServer, path: str, headers: dict[str, str] | None = None) -> tuple[int, str]: + host, port = server.server_address[:2] + url = insecure_url("http", f"{host}:{port}{path}") + request = urllib.request.Request(url, headers=headers or {}, method="GET") + try: + with urllib.request.urlopen(request, timeout=5) as resp: # nosec B310 + return resp.status, resp.read().decode("utf-8") + except urllib.error.HTTPError as error: + return error.code, error.read().decode("utf-8") + + +def _run(services: AppServices, action: str, name: str = "web") -> str: + draft = services.pipelines.new_draft(name) + draft.add_task(action, "only") + run_id = services.pipelines.start(draft)["run_id"] + assert services.pipelines.wait(run_id, WAIT) + return run_id + + +def test_the_index_polls_every_fragment_and_names_the_navigation(server: WebUIServer) -> None: + status, body = _get(server, "/") + assert status == 200 + for fragment in ( + "health", + "runs", + "integrity", + "events", + "storage", + "audit", + "progress", + "registry", + ): + assert f'hx-get="/ui/{fragment}"' in body + for name in NAVIGATION: + assert f"{name}" in body + assert _get(server, "/index.html")[0] == 200 + + +def test_the_server_uses_the_services_it_was_given( + server: WebUIServer, services: AppServices +) -> None: + assert server.services is services + status, body = _get(server, "/ui/registry") + assert status == 200 + assert "
  • T_echo
    • " in body + assert "FA_storage_copy" not in body + + +def test_the_default_services_are_the_process_wide_ones() -> None: + from automation_file.app import app_services + + server = WebUIServer(("127.0.0.1", 0)) + try: + assert server.services is app_services() + finally: + server.server_close() + + +def test_health_comes_from_the_dashboard_and_says_why_it_needs_attention( + server: WebUIServer, services: AppServices +) -> None: + status, body = _get(server, "/ui/health") + assert status == 200 + assert "

    status: ok

    " in body + assert "registry size2" in body + assert "processalive" in body + assert "auditnot configured" in body + _run(services, "T_fail") + body = _get(server, "/ui/health")[1] + assert "

    status: attention

    " in body + assert "
  • 1 of the last 1 pipeline runs failed
    • " in body + + +def test_runs_are_listed_newest_first_with_their_counts( + server: WebUIServer, services: AppServices +) -> None: + assert "no pipeline runs recorded" in _get(server, "/ui/runs")[1] + good = _run(services, "T_echo") + bad = _run(services, "T_fail") + body = _get(server, "/ui/runs")[1] + assert "0 running, 1 succeeded, 1 failed, 0 cancelled" in body + assert body.index(bad[:8]) < body.index(good[:8]) + assert "1 failed" in body + assert "web" in body + + +def test_everything_rendered_is_escaped( + server: WebUIServer, services: AppServices, bus: EventBus +) -> None: + _run(services, "T_fail", name=SCRIPT) + bus.publish(Event(source=SCRIPT, subject=SCRIPT, payload={"error": SCRIPT})) + services.audit.configure(MemoryAuditStore()) + bus.publish(Event(source="test", subject="after", payload={"resource": SCRIPT})) + for path in ("/ui/health", "/ui/runs", "/ui/events", "/ui/audit", "/ui/storage"): + status, body = _get(server, path) + assert status == 200 + assert " None: + services.audit.configure(MemoryAuditStore()) + bus.publish( + PipelineFailed( + source="pipeline", + subject="fetch https://user:hunter2@example.com/x failed", + payload={"error": "Bearer abc.def-123 was refused", "token": "t0ps3cret"}, + ) + ) + for path in ("/ui/events", "/ui/audit", "/ui/health"): + body = _get(server, path)[1] + assert "hunter2" not in body, path + assert "abc.def-123" not in body, path + assert "t0ps3cret" not in body, path + assert MASK in _get(server, "/ui/events")[1] + + +def test_integrity_monitors_are_listed(server: WebUIServer, services: AppServices) -> None: + assert "no integrity monitor is running" in _get(server, "/ui/integrity")[1] + File(f"{TREE}/a.txt").write(b"alpha") + services.integrity.baseline(TREE, BASELINE) + services.integrity.start_monitor("webui-monitor", TREE, BASELINE, interval=3600) + body = _get(server, "/ui/integrity")[1] + assert "webui-monitor" in body + assert f"{TREE}" in body + assert "not verified yet" in body + + +def test_storage_status_lists_the_backends(server: WebUIServer) -> None: + body = _get(server, "/ui/storage")[1] + assert "memory://webuimountyesmounted" in body + assert "boxclient" in body + + +def test_audit_entries_appear_once_audit_is_configured( + server: WebUIServer, services: AppServices, bus: EventBus +) -> None: + assert "audit is not configured" in _get(server, "/ui/audit")[1] + services.audit.configure(MemoryAuditStore()) + assert "no audit records yet" in _get(server, "/ui/audit")[1] + bus.publish(PipelineFailed(source="pipeline", subject="nightly failed")) + body = _get(server, "/ui/audit")[1] + assert "pipeline.failed" in body + assert "error" in body + + +def test_a_fragment_whose_service_fails_says_so_instead_of_breaking( + server: WebUIServer, services: AppServices, monkeypatch: pytest.MonkeyPatch +) -> None: + def broken() -> list[Any]: + raise IntegrityException(f"the store is gone {SCRIPT}") + + monkeypatch.setattr(services.dashboard, "integrity", broken) + status, body = _get(server, "/ui/integrity") + assert status == 200 + assert body == "

    unavailable: IntegrityException

    " + assert "attention" in _get(server, "/ui/health")[1] + + +def test_the_fragments_stay_read_only_and_behind_the_secret(services: AppServices) -> None: + server = start_web_ui(host="127.0.0.1", port=0, shared_secret="s3cr3t", services=services) + try: + for path in ("/ui/runs", "/ui/events", "/ui/audit", "/ui/integrity", "/ui/storage"): + assert _get(server, path)[0] == 401 + assert _get(server, path, {"Authorization": "Bearer wrong"})[0] == 401 + assert _get(server, path, {"Authorization": "Bearer s3cr3t"})[0] == 200 + host, port = server.server_address[:2] + request = urllib.request.Request( + insecure_url("http", f"{host}:{port}/ui/runs"), + data=b"{}", + headers={"Authorization": "Bearer s3cr3t"}, + method="POST", + ) + with pytest.raises(urllib.error.HTTPError) as caught: + urllib.request.urlopen(request, timeout=5) # nosec B310 + assert caught.value.code == 501 + finally: + server.shutdown() + server.server_close() diff --git a/tests/ui_stand_in.py b/tests/ui_stand_in.py new file mode 100644 index 0000000..c351265 --- /dev/null +++ b/tests/ui_stand_in.py @@ -0,0 +1,39 @@ +"""Stand-ins for the GUI tests: a thread pool that runs its work at once. + +A page hands every service call to a thread pool through ``ActionWorker``. With +:class:`SyncPool` the worker runs in the calling thread, so its ``finished`` and +``failed`` signals are delivered before ``start`` returns and a test can look at +the page right after triggering an action, without an event loop. +""" + +from __future__ import annotations + +from typing import Any + + +class SyncPool: + """Runs each worker immediately, in the thread that starts it.""" + + def __init__(self) -> None: + self.started = 0 + + def start(self, worker: Any) -> None: + self.started += 1 + worker.run() + + +class HeldPool: + """Keeps the workers it is given until :meth:`release`, to test what a page does meanwhile.""" + + def __init__(self) -> None: + self.held: list[Any] = [] + + def start(self, worker: Any) -> None: + self.held.append(worker) + + def release(self) -> int: + """Run every held worker; return how many there were.""" + held, self.held = self.held, [] + for worker in held: + worker.run() + return len(held) From f446ef97005a538c62547ae231bad1e069569208 Mon Sep 17 00:00:00 2001 From: JeffreyChen Date: Thu, 8 Oct 2026 15:38:49 +0800 Subject: [PATCH 48/59] docs: add the migration guide --- README.md | 4 +- README.zh-CN.md | 4 +- README.zh-TW.md | 4 +- docs/source/Eng/eng_index.rst | 1 + docs/source/Eng/usage/migration.rst | 128 ++++++++++++++++++++++++++ docs/source/Zh-CN/usage/migration.rst | 108 ++++++++++++++++++++++ docs/source/Zh-CN/zh_cn_index.rst | 1 + docs/source/Zh-TW/usage/migration.rst | 108 ++++++++++++++++++++++ docs/source/Zh-TW/zh_tw_index.rst | 1 + docs/updates/2026-10.md | 13 +++ docs/updates/README.md | 3 +- progress.md | 2 +- 12 files changed, 372 insertions(+), 5 deletions(-) create mode 100644 docs/source/Eng/usage/migration.rst create mode 100644 docs/source/Zh-CN/usage/migration.rst create mode 100644 docs/source/Zh-TW/usage/migration.rst diff --git a/README.md b/README.md index c419221..38dcfaf 100644 --- a/README.md +++ b/README.md @@ -1419,7 +1419,9 @@ storage layer, the event bus, pipelines, the integrity monitor, the audit trail, router and the semantic MCP tools are provisional: they may still change in a minor release, and the release notes say how. A deprecated name keeps working for at least two minor releases, warns with its replacement, and is removed only in a major release. The full policy is in the manual: -*Public API and compatibility* (`docs/source/Eng/usage/api_policy.rst`). +*Public API and compatibility* (`docs/source/Eng/usage/api_policy.rst`). Coming from 0.0.x: nothing was +removed, the cloud SDKs moved into extras (`pip install "automation_file[all]"`), and *Migrating to 1.0* +(`docs/source/Eng/usage/migration.rst`) lists the few behaviours that changed. ## Documentation diff --git a/README.zh-CN.md b/README.zh-CN.md index 9a152ad..344ca72 100644 --- a/README.zh-CN.md +++ b/README.zh-CN.md @@ -1369,7 +1369,9 @@ python -m pytest tests/integration/test_s3_minio.py 在 1.0 之前，存储层、事件总线、流水线、完整性监控、审计轨迹、通知路由器与语义化 MCP 工具属于 暂定功能：仍可能在次版本中变动，版本说明会交代如何应对。被弃用的名称至少会保留两个次版本、 发出附带替代方案的警告，并且只会在主版本中移除。完整的政策请见手册的“公开 API 与兼容性” -（`docs/source/Zh-CN/usage/api_policy.rst`）。 +（`docs/source/Zh-CN/usage/api_policy.rst`）。从 0.0.x 升级时：没有任何东西被移除，云端 SDK 改放到 +extra（`pip install "automation_file[all]"`），少数有所改变的行为列在“迁移到 1.0” +（`docs/source/Zh-CN/usage/migration.rst`）。 ## 文档 diff --git a/README.zh-TW.md b/README.zh-TW.md index c5b21be..896d2b7 100644 --- a/README.zh-TW.md +++ b/README.zh-TW.md @@ -1370,7 +1370,9 @@ python -m pytest tests/integration/test_s3_minio.py 在 1.0 之前，儲存層、事件匯流排、管線、完整性監控、稽核軌跡、通知路由器與語意化 MCP 工具屬於 暫定功能：仍可能在次版本中變動，版本說明會交代如何因應。被棄用的名稱至少會保留兩個次版本、 發出附帶替代方案的警告，並且只會在主版本中移除。完整的政策請見手冊的「公開 API 與相容性」 -（`docs/source/Zh-TW/usage/api_policy.rst`）。 +（`docs/source/Zh-TW/usage/api_policy.rst`）。從 0.0.x 升級時：沒有任何東西被移除，雲端 SDK 改放到 +extra（`pip install "automation_file[all]"`），少數有所改變的行為列在「遷移到 1.0」 +（`docs/source/Zh-TW/usage/migration.rst`）。 ## 文件 diff --git a/docs/source/Eng/eng_index.rst b/docs/source/Eng/eng_index.rst index 8868099..ca08f73 100644 --- a/docs/source/Eng/eng_index.rst +++ b/docs/source/Eng/eng_index.rst @@ -343,6 +343,7 @@ promises, and how a name is deprecated and removed. :caption: Public API and Compatibility usage/api_policy + usage/migration .. _eng-integration-tests: diff --git a/docs/source/Eng/usage/migration.rst b/docs/source/Eng/usage/migration.rst new file mode 100644 index 0000000..96b35b2 --- /dev/null +++ b/docs/source/Eng/usage/migration.rst @@ -0,0 +1,128 @@ +Migrating to 1.0 +================ + +Code written for 0.0.x keeps running: no ``FA_*`` action, facade name or command +line flag was removed. This page lists the few things that behave differently, +and, for each older interface, the newer one and when to prefer it. + +What you have to do +------------------- + +.. list-table:: + :header-rows: 1 + :widths: 34 66 + + * - If you + - Then + * - install the package and use a cloud backend, Parquet or the GUI + - Name the extra. The base install no longer brings the SDKs: + ``pip install "automation_file[s3,sftp]"``, or ``[all]`` for what 0.0.x + installed. A missing SDK fails at the first call with the command to run. + * - use time zones in a schedule on Windows + - Nothing: ``tzdata`` is installed with the package there. + * - call ``copy_between`` with ``sftp://host/path`` or ``ftp://host/path`` + - Two slashes now mean a host and an absolute path, and the host must be the + one the session is connected to. The documented one-slash form + (``sftp:/path``, relative to the login directory) is unchanged. + * - rely on ``copy_between`` raising ``RuntimeError`` for a backend that was + not initialised + - Catch ``StorageUnavailableException`` (a ``FileAutomationException``). + * - pass a path to ``WebDAVClient`` that is a full URL on another host, or + rely on it following redirects to another host + - That is refused now. Build a client for that host. + * - read ``job.cron`` or assign to it on a scheduled job + - Reading still works. Assigning no longer reschedules the job: remove it + and add it again. + * - count on a scheduled action list being recorded as run when one of its + actions raised + - The run is now ``failed`` and publishes ``scheduler.error``. The rest of + the list still runs. + * - give an action server an ``ActionACL`` + - Check your allow list: an action nested in the arguments of another + (``FA_execute_action``, a pipeline definition, a scheduled list) is now + checked too, and must be allowed as well. + * - narrowed the MCP server with ``--allowed-actions`` + - The same applies: a tool can no longer run an action the server does not + expose. + * - compared the text of a notification sink's error + - URLs in it are now cut to the host, so a token in a path is not shown. + * - import ``HomeTab`` or ``SchedulerTab`` to embed them + - They still import, but the window no longer mounts them; the Dashboard + and Scheduler pages replaced them. + +Everything else in this release is an addition. + +Older and newer interfaces +-------------------------- + +Both columns are supported. Nothing on the left is deprecated. + +.. list-table:: + :header-rows: 1 + :widths: 30 34 36 + + * - Older + - Newer + - Prefer the newer one when + * - ``FA_s3_upload_file``, ``FA_sftp_download_file`` and the other + per-backend actions + - ``File`` / ``Storage`` and ``FA_storage_*`` (:doc:`storage`) + - the same code should work on more than one backend, or you want typed + errors, the audit trail and the events + * - ``copy_between`` / ``FA_copy_between`` + - ``File(source).copy_to(target)``, ``FA_storage_copy`` + - you want an error instead of ``False``, and the result described + * - ``write_manifest`` / ``verify_manifest`` + - ``IntegrityMonitor`` (:doc:`integrity`) + - the tree is not local, or you need renames, metadata changes, watching or + an approved baseline + * - ``IntegrityMonitor(root, manifest_path, …)`` and ``check_once()`` + - ``IntegrityMonitor(target, baseline=…)`` with ``verify()``, ``accept()`` + - you want a ``DriftReport`` and not a summary dictionary. A baseline that + ``accept()`` or ``create_baseline()`` wrote is no longer readable by + ``verify_manifest`` + * - ``execute_action_dag`` + - ``Pipeline`` (:doc:`pipeline`) + - you need retries, timeouts, resume after a crash or a history of runs + * - ``AuditLog`` + - ``configure_audit`` and ``audit_search`` (:doc:`audit`) + - you want every event and storage operation recorded without calling + ``record`` yourself. ``SQLiteAuditStore.import_v1()`` copies the old rows + * - ``notification_manager.notify`` and ``notify_on_failure`` + - notification routes (:doc:`notifications`) + - different events should reach different sinks, with their own + deduplication and rate limit + * - ``FA_schedule_add`` with a cron expression + - ``FA_schedule_job``, ``FA_schedule_pipeline`` and triggers + (:doc:`scheduler`) + - a job should run in a time zone, on an event, after another pipeline, or + run a pipeline + * - the ``FA_*`` tools of the MCP server + - the semantic tools (:doc:`mcp`) + - an AI host should be confined to named locations and start read-only + * - the GUI's per-backend tabs + - the workflow pages; the tabs are under Advanced (:doc:`gui`) + - always, unless you need a backend-specific operation + +A notification that used to arrive +---------------------------------- + +Two components notified the process-wide ``notification_manager`` directly and +still do: the integrity monitor on drift, and ``notify_on_failure`` for a failed +trigger or schedule. Once you start the notification router, it delivers those +events through its routes and the direct notification stops, so nothing is +announced twice. If you start the router, add a route for the events you were +being told about (``integrity.violation``, ``scheduler.error``, +``system.error``); otherwise they are published and reach no sink. + +Before you upgrade production +----------------------------- + +1. Install the new version with the extras you need in a fresh environment. +2. Run your tests with ``-W error::DeprecationWarning``. +3. Run ``python -m automation_file storage schemes`` and one real transfer per + backend you use. +4. If an action server has an ``ActionACL``, send it one request of each kind + your clients send. +5. Follow :doc:`deployment` for the state that should now live in files (the + audit trail, the pipeline run store). diff --git a/docs/source/Zh-CN/usage/migration.rst b/docs/source/Zh-CN/usage/migration.rst new file mode 100644 index 0000000..5791cc6 --- /dev/null +++ b/docs/source/Zh-CN/usage/migration.rst @@ -0,0 +1,108 @@ +迁移到 1.0 +========== + +为 0.0.x 写的代码仍然可以运行：没有任何 ``FA_*`` 动作、facade 名称或命令行选项被移除。 +本页列出少数行为有所不同之处，并针对每一个旧接口，说明对应的新接口以及何时该优先采用。 + +你必须做的事 +------------ + +.. list-table:: + :header-rows: 1 + :widths: 34 66 + + * - 如果你 + - 那么 + * - 安装本包并使用云端后端、Parquet 或 GUI + - 请写明 extra。基础安装不再附带各家 SDK： + ``pip install "automation_file[s3,sftp]"``，或用 ``[all]`` 获得 0.0.x 所安装的 + 全部内容。缺少 SDK 时，会在第一次调用时失败并指出要运行的命令。 + * - 在 Windows 上的调度使用时区 + - 不必做任何事：在 Windows 上 ``tzdata`` 会随包一起安装。 + * - 以 ``sftp://host/path`` 或 ``ftp://host/path`` 调用 ``copy_between`` + - 两个斜线现在代表主机与绝对路径，而且主机必须是会话实际连接的那一台。 + 文档记载的单斜线写法（``sftp:/path``，相对于登录目录）没有改变。 + * - 依赖 ``copy_between`` 在后端尚未初始化时抛出 ``RuntimeError`` + - 请改为捕获 ``StorageUnavailableException``（属于 ``FileAutomationException``）。 + * - 把指向其他主机的完整 URL 当成路径传给 ``WebDAVClient``，或依赖它跟随重定向到 + 其他主机 + - 现在会被拒绝。请为那台主机另外建立客户端。 + * - 读取或赋值调度作业的 ``job.cron`` + - 读取仍然可行。赋值不再会重新调度：请先移除作业，再重新加入。 + * - 依赖调度的动作列表在其中一个动作抛出异常时仍被记为已运行 + - 该次运行现在是 ``failed``，并发布 ``scheduler.error``。列表的其余部分仍会运行。 + * - 为动作服务器设置了 ``ActionACL`` + - 请检查你的允许列表：嵌套在另一个动作参数中的动作（``FA_execute_action``、 + 流水线定义、调度的列表）现在也会被检查，因此同样必须被允许。 + * - 以 ``--allowed-actions`` 限缩了 MCP 服务器 + - 同样适用：工具不能再运行服务器没有开放的动作。 + * - 比对通知 sink 的错误文字 + - 其中的 URL 现在只保留主机部分，路径中的令牌不会显示出来。 + * - 导入 ``HomeTab`` 或 ``SchedulerTab`` 来嵌入它们 + - 它们仍然可以导入，但窗口不再挂载它们；Dashboard 与 Scheduler 页面取代了它们。 + +本次发布的其余内容都是新增的功能。 + +旧接口与新接口 +-------------- + +两栏都受到支持。左栏没有任何东西被弃用。 + +.. list-table:: + :header-rows: 1 + :widths: 30 34 36 + + * - 旧接口 + - 新接口 + - 何时优先采用新接口 + * - ``FA_s3_upload_file``、``FA_sftp_download_file`` 以及其他各后端专属的动作 + - ``File`` / ``Storage`` 与 ``FA_storage_*``（:doc:`storage`） + - 同一份代码要能用在不止一种后端，或者你想要有类型的错误、审计轨迹与事件 + * - ``copy_between`` / ``FA_copy_between`` + - ``File(source).copy_to(target)``、``FA_storage_copy`` + - 你想要的是异常而不是 ``False``，并且想获得结果的描述 + * - ``write_manifest`` / ``verify_manifest`` + - ``IntegrityMonitor``（:doc:`integrity`） + - 目录树不在本地，或者你需要检测重命名、元数据变更、监听，或经过核准的基准 + * - ``IntegrityMonitor(root, manifest_path, …)`` 与 ``check_once()`` + - ``IntegrityMonitor(target, baseline=…)`` 搭配 ``verify()``、``accept()`` + - 你想要 ``DriftReport`` 而不是摘要字典。由 ``accept()`` 或 ``create_baseline()`` + 写出的基准，``verify_manifest`` 无法再读取 + * - ``execute_action_dag`` + - ``Pipeline``（:doc:`pipeline`） + - 你需要重试、超时、崩溃后续跑，或运行历史 + * - ``AuditLog`` + - ``configure_audit`` 与 ``audit_search``（:doc:`audit`） + - 你希望每个事件与每次存储操作都被记录，而不必自己调用 ``record``。 + ``SQLiteAuditStore.import_v1()`` 可以复制旧的行 + * - ``notification_manager.notify`` 与 ``notify_on_failure`` + - 通知路由（:doc:`notifications`） + - 不同的事件要送到不同的 sink，并各有自己的去重与限流 + * - 以 cron 表达式调用 ``FA_schedule_add`` + - ``FA_schedule_job``、``FA_schedule_pipeline`` 与各种触发条件（:doc:`scheduler`） + - 作业要按某个时区运行、由事件触发、接在另一条流水线之后，或要运行流水线 + * - MCP 服务器的 ``FA_*`` 工具 + - 语义化工具（:doc:`mcp`） + - AI 宿主应该被限制在指定的位置之内，并从只读开始 + * - GUI 中各后端专属的页签 + - 按工作流组织的页面；原本的页签放在 Advanced 之下（:doc:`gui`） + - 一律优先，除非你需要某个后端专属的操作 + +原本会收到的通知 +---------------- + +有两个组件过去会直接通知整个进程共用的 ``notification_manager``，现在仍然如此：完整性 +监控在检测到偏移时，以及 ``notify_on_failure`` 在触发或调度失败时。一旦你启动了通知 +路由器，这些事件就改由它的路由送达，直接通知随之停止，因此同一件事不会被通知两次。 +如果你启动了路由器，请为你原本会被告知的事件（``integrity.violation``、 +``scheduler.error``、``system.error``）加上路由；否则它们会被发布，却送不到任何 sink。 + +升级生产环境之前 +---------------- + +1. 在全新的环境中安装新版本以及你需要的 extra。 +2. 以 ``-W error::DeprecationWarning`` 运行你的测试。 +3. 运行 ``python -m automation_file storage schemes``，并对你使用的每一种后端做一次真实的 + 传输。 +4. 如果动作服务器设有 ``ActionACL``，请把你的客户端会发送的每一种请求各发送一次。 +5. 按照 :doc:`deployment` 安排现在应该保存在文件中的状态（审计轨迹、流水线运行记录）。 diff --git a/docs/source/Zh-CN/zh_cn_index.rst b/docs/source/Zh-CN/zh_cn_index.rst index 098a3be..788cbcb 100644 --- a/docs/source/Zh-CN/zh_cn_index.rst +++ b/docs/source/Zh-CN/zh_cn_index.rst @@ -328,6 +328,7 @@ Slack、Email（SMTP）、Discord、Telegram、Microsoft Teams、PagerDuty :caption: 公开 API 与兼容性 usage/api_policy + usage/migration .. _zh-cn-integration-tests: diff --git a/docs/source/Zh-TW/usage/migration.rst b/docs/source/Zh-TW/usage/migration.rst new file mode 100644 index 0000000..1b3be13 --- /dev/null +++ b/docs/source/Zh-TW/usage/migration.rst @@ -0,0 +1,108 @@ +遷移到 1.0 +========== + +為 0.0.x 寫的程式仍然可以執行：沒有任何 ``FA_*`` 動作、facade 名稱或命令列旗標被移除。 +本頁列出少數行為有所不同之處，並針對每一個舊介面，說明對應的新介面以及何時該優先採用。 + +你必須做的事 +------------ + +.. list-table:: + :header-rows: 1 + :widths: 34 66 + + * - 如果你 + - 那麼 + * - 安裝本套件並使用雲端後端、Parquet 或 GUI + - 請寫明 extra。基本安裝不再附帶各家 SDK： + ``pip install "automation_file[s3,sftp]"``，或用 ``[all]`` 取得 0.0.x 所安裝的 + 全部內容。缺少 SDK 時，會在第一次呼叫時失敗並指出要執行的指令。 + * - 在 Windows 上的排程使用時區 + - 不必做任何事：在 Windows 上 ``tzdata`` 會隨套件一併安裝。 + * - 以 ``sftp://host/path`` 或 ``ftp://host/path`` 呼叫 ``copy_between`` + - 兩個斜線現在代表主機與絕對路徑，而且主機必須是工作階段實際連線的那一台。 + 文件記載的單斜線寫法（``sftp:/path``，相對於登入目錄）沒有改變。 + * - 依賴 ``copy_between`` 在後端尚未初始化時拋出 ``RuntimeError`` + - 請改為攔截 ``StorageUnavailableException``（屬於 ``FileAutomationException``）。 + * - 把指向其他主機的完整 URL 當成路徑傳給 ``WebDAVClient``，或依賴它跟隨轉址到 + 其他主機 + - 現在會被拒絕。請為那台主機另外建立用戶端。 + * - 讀取或指定排程工作的 ``job.cron`` + - 讀取仍然可行。指定不再會重新排程：請先移除工作，再重新加入。 + * - 依賴排程的動作清單在其中一個動作拋出例外時仍被記為已執行 + - 該次執行現在是 ``failed``，並發布 ``scheduler.error``。清單的其餘部分仍會執行。 + * - 為動作伺服器設定了 ``ActionACL`` + - 請檢查你的允許清單：巢狀在另一個動作引數中的動作（``FA_execute_action``、 + 管線定義、排程的清單）現在也會被檢查，因此同樣必須被允許。 + * - 以 ``--allowed-actions`` 限縮了 MCP 伺服器 + - 同樣適用：工具不能再執行伺服器沒有開放的動作。 + * - 比對通知 sink 的錯誤文字 + - 其中的 URL 現在只保留主機部分，路徑中的權杖不會顯示出來。 + * - 匯入 ``HomeTab`` 或 ``SchedulerTab`` 來嵌入它們 + - 它們仍然可以匯入，但視窗不再掛載它們；Dashboard 與 Scheduler 頁面取代了它們。 + +本次發行的其餘內容都是新增的功能。 + +舊介面與新介面 +-------------- + +兩欄都受到支援。左欄沒有任何東西被棄用。 + +.. list-table:: + :header-rows: 1 + :widths: 30 34 36 + + * - 舊介面 + - 新介面 + - 何時優先採用新介面 + * - ``FA_s3_upload_file``、``FA_sftp_download_file`` 以及其他各後端專屬的動作 + - ``File`` / ``Storage`` 與 ``FA_storage_*``（:doc:`storage`） + - 同一份程式要能用在不只一種後端，或者你想要有型別的錯誤、稽核軌跡與事件 + * - ``copy_between`` / ``FA_copy_between`` + - ``File(source).copy_to(target)``、``FA_storage_copy`` + - 你想要的是例外而不是 ``False``，並且想取得結果的描述 + * - ``write_manifest`` / ``verify_manifest`` + - ``IntegrityMonitor``（:doc:`integrity`） + - 目錄樹不在本機，或者你需要偵測重新命名、中繼資料變更、監看，或經過核可的基準 + * - ``IntegrityMonitor(root, manifest_path, …)`` 與 ``check_once()`` + - ``IntegrityMonitor(target, baseline=…)`` 搭配 ``verify()``、``accept()`` + - 你想要 ``DriftReport`` 而不是摘要字典。由 ``accept()`` 或 ``create_baseline()`` + 寫出的基準，``verify_manifest`` 無法再讀取 + * - ``execute_action_dag`` + - ``Pipeline``（:doc:`pipeline`） + - 你需要重試、逾時、當機後續跑，或執行歷史 + * - ``AuditLog`` + - ``configure_audit`` 與 ``audit_search``（:doc:`audit`） + - 你希望每個事件與每次儲存操作都被記錄，而不必自己呼叫 ``record``。 + ``SQLiteAuditStore.import_v1()`` 可以複製舊的資料列 + * - ``notification_manager.notify`` 與 ``notify_on_failure`` + - 通知路由（:doc:`notifications`） + - 不同的事件要送到不同的 sink，並各有自己的去重與限流 + * - 以 cron 運算式呼叫 ``FA_schedule_add`` + - ``FA_schedule_job``、``FA_schedule_pipeline`` 與各種觸發條件（:doc:`scheduler`） + - 工作要依某個時區執行、由事件觸發、接在另一條管線之後，或要執行管線 + * - MCP 伺服器的 ``FA_*`` 工具 + - 語意化工具（:doc:`mcp`） + - AI 宿主應該被限制在指定的位置之內，並從唯讀開始 + * - GUI 中各後端專屬的分頁 + - 依工作流程安排的頁面；原本的分頁放在 Advanced 底下（:doc:`gui`） + - 一律優先，除非你需要某個後端專屬的操作 + +原本會收到的通知 +---------------- + +有兩個元件過去會直接通知整個行程共用的 ``notification_manager``，現在仍然如此：完整性 +監控在偵測到偏移時，以及 ``notify_on_failure`` 在觸發或排程失敗時。一旦你啟動了通知 +路由器，這些事件就改由它的路由送達，直接通知隨之停止，因此同一件事不會被通知兩次。 +如果你啟動了路由器，請為你原本會被告知的事件（``integrity.violation``、 +``scheduler.error``、``system.error``）加上路由；否則它們會被發布，卻送不到任何 sink。 + +升級正式環境之前 +---------------- + +1. 在全新的環境中安裝新版本以及你需要的 extra。 +2. 以 ``-W error::DeprecationWarning`` 執行你的測試。 +3. 執行 ``python -m automation_file storage schemes``，並對你使用的每一種後端做一次真實的 + 傳輸。 +4. 如果動作伺服器設有 ``ActionACL``，請把你的用戶端會送出的每一種請求各送一次。 +5. 依照 :doc:`deployment` 安排現在應該保存在檔案中的狀態（稽核軌跡、管線執行紀錄）。 diff --git a/docs/source/Zh-TW/zh_tw_index.rst b/docs/source/Zh-TW/zh_tw_index.rst index 28eb02f..ea117c1 100644 --- a/docs/source/Zh-TW/zh_tw_index.rst +++ b/docs/source/Zh-TW/zh_tw_index.rst @@ -328,6 +328,7 @@ Slack、Email（SMTP）、Discord、Telegram、Microsoft Teams、PagerDuty :caption: 公開 API 與相容性 usage/api_policy + usage/migration .. _zh-tw-integration-tests: diff --git a/docs/updates/2026-10.md b/docs/updates/2026-10.md index f2f89cb..c08cd3a 100644 --- a/docs/updates/2026-10.md +++ b/docs/updates/2026-10.md @@ -658,3 +658,16 @@ Index and query commands: [README.md](README.md). New entries go at the end. - **Docs**: `usage/gui.rst` rewritten and `usage/app_layer.rst` added in the three manuals, a Web UI section in the three `usage/servers.rst`, `docs/source/API/app.rst` and `API/ui.rst`, the indexes, the GUI, application-layer and Web UI sections, three bullets and the diagram of the three READMEs, `architecture.md` §2 and §3, `CLAUDE.md` (package map, `MainWindow`). - **Files**: `automation_file/app/` (15 modules), `automation_file/ui/pages/` (15 modules), `automation_file/ui/{main_window,launcher,__init__}.py`, `automation_file/server/web_ui.py`, `automation_file/__init__.py`, the tests above, the documentation above. - **Open items**: #38, #39. + +## U-20261008-31 · 2026-10-08 · Migration guide · #docs #migration #roadmap + +- **What**: a manual page, "Migrating to 1.0", in the three languages (roadmap M9; the definition of done asks for migration guidance). + - **What you have to do**: the eleven places where 0.0.x code meets a different behaviour, each with what to change: the extras, `copy_between` with two slashes, the exception for an uninitialised backend, `WebDAVClient` and other hosts, `job.cron`, a scheduled list whose action raises, nested actions under an `ActionACL` and under `--allowed-actions`, sink error text, and the two GUI tabs that are no longer mounted. + - **Older and newer interfaces**: ten pairs (per-backend actions and the storage layer, `copy_between` and `File.copy_to`, manifests and `IntegrityMonitor`, `execute_action_dag` and `Pipeline`, `AuditLog` and the audit trail, direct notifications and routes, cron jobs and triggers, the MCP bridge and the semantic tools, the tabs and the pages), all supported, with when to prefer the newer one. + - **A notification that used to arrive**: what starting the notification router changes for the integrity monitor and `notify_on_failure`, and the route to add. + - A five-step check before upgrading production. +- **Source**: the "what changed for callers" parts of U-20261008-09, -14, -16, -17, -19, -21, -29 and -30. Nothing was removed in any of them. +- **Not verified**: the Sphinx build of the pages (headings and markup were checked by script). +- **Docs**: `usage/migration.rst` in the three manuals, next to the API policy in chapter 21; one sentence in the "Compatibility" section of the three READMEs. +- **Files**: the three `usage/migration.rst` pages, the three indexes, the three READMEs, `progress.md`. +- **Open items**: `progress.md` #26 now holds only what is the owner's to decide. diff --git a/docs/updates/README.md b/docs/updates/README.md index d89f38a..3c523d9 100644 --- a/docs/updates/README.md +++ b/docs/updates/README.md @@ -58,6 +58,7 @@ In the same commit: delete the item from `progress.md`, add a `#done` entry here | ID | Date | Title | Tags | Batch | |---|---|---|---|---| +| U-20261008-31 | 2026-10-08 | Migration guide | #docs #migration #roadmap | [2026-10](2026-10.md) | | U-20261008-30 | 2026-10-08 | UI 2.0 and the application layer | #ui #roadmap #done | [2026-10](2026-10.md) | | U-20261008-29 | 2026-10-08 | Scheduler v2 | #scheduler #roadmap #done | [2026-10](2026-10.md) | | U-20261008-28 | 2026-10-08 | One positioning in the READMEs, the manuals and the metadata | #docs #packaging #roadmap | [2026-10](2026-10.md) | @@ -125,5 +126,5 @@ In the same commit: delete the item from `progress.md`, add a `#done` entry here | File | Period | Entries | |---|---|---:| -| [2026-10.md](2026-10.md) | 2026-10 | 41 | +| [2026-10.md](2026-10.md) | 2026-10 | 42 | | [2026-09.md](2026-09.md) | 2026-09 | 21 | diff --git a/progress.md b/progress.md index 1440d0c..69613b1 100644 --- a/progress.md +++ b/progress.md @@ -28,7 +28,7 @@ Items #10 to #26 are what is left of the 1.0 roadmap (`docs/FILEAUTOMATION-1.0-R - **#38** [UNVERIFIED] UI 2.0 (U-20261008-30) has only run on Qt's offscreen platform. Open it on a real display (`python -m automation_file ui`) and check: text fit and layout of the ten pages, dragging a node and adding a task on the pipeline canvas, selecting two tasks and connecting them, the file dialogs, the confirmation boxes, the keyboard shortcuts, and one page against a real backend. Dragging from node to node to connect two tasks is not built. - **#39** The application layer reaches into two private places: `StorageService` reads `StorageResolver._mounts` / `_factories` (give the resolver a public listing of its mounts), and `app/` uses `pipeline.definition.retry_from_dict`, `graph.upstream_tasks`, `substitution.is_name` / `NAME_RULE` and `model.ON_SUCCESS` / `WHEN_CHOICES`, which are not in `automation_file.pipeline.__all__` (export them, or give the pipeline package the functions the editor needs). - **#37** Storage-layer gaps the MCP work found (U-20261008-27) and did not change: (a) a `LocalStorage(root)` listing reports a link's name and its target's metadata without a containment check, although reading through the link is refused; (b) `StorageBackend` has no ranged read, so reading the head of a large remote file stages all of it; (c) `LocalStorage` on Windows opens device names (`CON`, `NUL`) and alternate data streams, which the MCP tools refuse themselves; (d) a link on an SFTP or FTP server leads outside a root that is only a path prefix. -- **#26** Release engineering and 1.0 (roadmap §13, M9). Done: semantic versioning with a way to release a MINOR or MAJOR (U-20261008-24), the public API policy (U-20261008-22), a package-build check and the integration workflow next to the PR checks. Open: the migration guide and the final documentation audit (after the scheduler, MCP and GUI work lands); making the integration jobs required once they are green (#19); the 1.0.0 release itself, which is the owner's call: write `1.0.0` in both TOMLs in the release pull request. +- **#26** The 1.0.0 release (roadmap M9). Everything the roadmap lists is on the branch `feat/universal-storage-layer`; what is left is the owner's: review and merge the pull request to `dev`, read the first CI and integration runs (#19, #32, #18, #38), decide the 1.0 date, then write `1.0.0` in `stable.toml` and `dev.toml` in the release pull request to `main` and raise the `Development Status` classifier. Not written: a separate security page in the manual (the deployment chapter and `CLAUDE.md` § Security carry that guidance today). - **#34** [BLOCKED] PyPI Trusted Publishing (roadmap §13). `publish.yml` and `publish-dev` still upload with the `PYPI_API_TOKEN` secret. Switching needs the owner to add a trusted publisher for each project on PyPI (`automation_file`: workflow `publish.yml`; `automation_file_dev`: workflow `ci-dev.yml`; an environment name if one is wanted) before the workflows can drop the token for `id-token: write` and `pypa/gh-action-pypi-publish`. Changing the workflows first would break both channels. ### Packaging follow-ups From 075799537e8c273280247d2fe13c1ea6a03f0504 Mon Sep 17 00:00:00 2001 From: JeffreyChen Date: Thu, 8 Oct 2026 15:53:02 +0800 Subject: [PATCH 49/59] test: stop an audit test from matching its content in random ids --- docs/updates/2026-10.md | 13 ++++++++++++- docs/updates/README.md | 3 ++- tests/test_mcp_pipeline_tools.py | 6 ++++-- 3 files changed, 18 insertions(+), 4 deletions(-) diff --git a/docs/updates/2026-10.md b/docs/updates/2026-10.md index c08cd3a..a1bff6b 100644 --- a/docs/updates/2026-10.md +++ b/docs/updates/2026-10.md @@ -597,7 +597,7 @@ Index and query commands: [README.md](README.md). New entries go at the end. - **Traceability**: each call runs in a correlation scope as the actor `mcp`, returns the correlation ID and publishes one `mcp.tool.*` event; a refused call is logged without its argument values and returned as a result with `isError`. - `tools/list` returns the semantic tools first, then the bridge; `--no-bridge` leaves only the semantic ones. `python -m automation_file mcp` forwards every flag. - Beyond the roadmap: `verify` on a copy or a move (a move deletes its source only after the SHA-256 matches), `background` on `pipeline_run`, usage instructions in the handshake, Windows device and data-stream names refused below local roots. -- **Changed while integrating**: one test compared two audit records in a fixed order although both can carry the same timestamp; it failed once in the base-dependency run. It now compares them as a set. +- **Changed while integrating**: one test compared two audit records in a fixed order although both can carry the same timestamp; it failed once in the base-dependency run. It now compares them as a set. → corrected in U-20261008-32: that was not the cause. - **Tests**: `tests/test_mcp_policy.py`, `test_mcp_tools.py`, `test_mcp_storage_tools.py`, `test_mcp_pipeline_tools.py`, `test_mcp_semantic_server.py` with `tests/mcp_support.py`; `tests/test_mcp_server.py` passes unchanged. - **Result / numbers**: 5260 passed, 257 skipped, 0 failed with every extra; 3479 passed, 135 skipped with the base dependencies only. `ruff check`, `ruff format --check` and `mypy automation_file` (239 files) pass. Python 3.14.7 on Windows. - **Not verified**: a real MCP host (a child process over stdio was used); real S3 or SFTP behind the tools; whether the link-escape tests ran through a symbolic link or the Windows junction fallback, and POSIX at all; the Sphinx build of the rewritten pages; Python 3.10 to 3.13. @@ -671,3 +671,14 @@ Index and query commands: [README.md](README.md). New entries go at the end. - **Docs**: `usage/migration.rst` in the three manuals, next to the API policy in chapter 21; one sentence in the "Compatibility" section of the three READMEs. - **Files**: the three `usage/migration.rst` pages, the three indexes, the three READMEs, `progress.md`. - **Open items**: `progress.md` #26 now holds only what is the owner's to decide. + +## U-20261008-32 · 2026-10-08 · An intermittent test failure, and its wrong first diagnosis · #tests #incident + +- **What happened**: `tests/test_mcp_pipeline_tools.py::test_audit_search_finds_what_a_call_did_by_its_correlation_id` failed in about one full run in five. U-20261008-27 blamed the order of two audit records with equal timestamps and changed the comparison to a sorted one. That was a guess, and it was wrong: the test failed again after it. +- **Cause**: the test writes a file whose content is `abc` and then asserts that `abc` does not appear in the audit records, to show that file content is not recorded. Record IDs and correlation IDs are random hexadecimal strings, and `abc` is a hexadecimal substring. A record whose ID happened to contain it (`…082abc4d16…` in the captured run) failed the assertion. +- **Fix**: the content is now `payload-kept-out-of-the-trail`, which no hexadecimal string contains. The other tests that assert a secret is absent from JSON use `hunter2`, `s3cr3t` and `neighbour`, none of which is hexadecimal. +- **The sorted comparison of U-20261008-27 stays**: the store orders by timestamp and then by sequence, so it is harmless, but it was not the fix. +- **Lesson kept in the test**: a comment on the content says why it is not a short word. +- **Result / numbers**: after the fix, 5763 passed, 257 skipped, 0 failed with every extra in three consecutive runs; 3879 passed, 137 skipped with the base dependencies only. +- **Files**: `tests/test_mcp_pipeline_tools.py`. +- **Open items**: none. diff --git a/docs/updates/README.md b/docs/updates/README.md index 3c523d9..5988c95 100644 --- a/docs/updates/README.md +++ b/docs/updates/README.md @@ -58,6 +58,7 @@ In the same commit: delete the item from `progress.md`, add a `#done` entry here | ID | Date | Title | Tags | Batch | |---|---|---|---|---| +| U-20261008-32 | 2026-10-08 | An intermittent test failure, and its wrong first diagnosis | #tests #incident | [2026-10](2026-10.md) | | U-20261008-31 | 2026-10-08 | Migration guide | #docs #migration #roadmap | [2026-10](2026-10.md) | | U-20261008-30 | 2026-10-08 | UI 2.0 and the application layer | #ui #roadmap #done | [2026-10](2026-10.md) | | U-20261008-29 | 2026-10-08 | Scheduler v2 | #scheduler #roadmap #done | [2026-10](2026-10.md) | @@ -126,5 +127,5 @@ In the same commit: delete the item from `progress.md`, add a `#done` entry here | File | Period | Entries | |---|---|---:| -| [2026-10.md](2026-10.md) | 2026-10 | 42 | +| [2026-10.md](2026-10.md) | 2026-10 | 43 | | [2026-09.md](2026-09.md) | 2026-09 | 21 | diff --git a/tests/test_mcp_pipeline_tools.py b/tests/test_mcp_pipeline_tools.py index 9471f3b..5012384 100644 --- a/tests/test_mcp_pipeline_tools.py +++ b/tests/test_mcp_pipeline_tools.py @@ -718,7 +718,9 @@ def test_audit_search_filters_and_caps_the_records() -> None: def test_audit_search_finds_what_a_call_did_by_its_correlation_id() -> None: audited() kit = writable() - written = kit.call("file_write", {"uri": f"{INBOX}/a.txt", "content": "abc"}) + # Not a word a random hexadecimal ID can contain. + content = "payload-kept-out-of-the-trail" + written = kit.call("file_write", {"uri": f"{INBOX}/a.txt", "content": content}) body = succeed(kit.call("audit_search", {"correlation_id": written.correlation_id})) # Two records of one call can carry the same timestamp, so their order is not fixed. assert sorted((record["source"], record["action"]) for record in body["records"]) == [ @@ -726,4 +728,4 @@ def test_audit_search_finds_what_a_call_did_by_its_correlation_id() -> None: ("storage", "upload"), ] assert {record["actor"] for record in body["records"]} == {"mcp"} - assert "abc" not in json.dumps(body["records"]) + assert content not in json.dumps(body["records"]) From d0f945a2c7b0ce18c07f45755d9d12154138f45c Mon Sep 17 00:00:00 2001 From: JeffreyChen Date: Thu, 8 Oct 2026 16:04:04 +0800 Subject: [PATCH 50/59] fix: correct what the first run against real services and macOS showed MinIO from quay.io, a writable FTP root, Samba on port 445 and a replace that removes the target first, a 400 from Apache for a path below a file, a versioning name too long to ask about on macOS, the package job on the locked tools, and equality tests written so that a scanner does not read them as self-comparisons. --- .github/workflows/ci-dev.yml | 5 +++-- .github/workflows/ci-stable.yml | 5 +++-- automation_file/local/versioning.py | 11 +++++++++-- automation_file/storage/smb_storage.py | 14 ++++++++++++-- automation_file/storage/webdav_storage.py | 6 ++++++ tests/integration/start_service.sh | 11 +++++++---- tests/test_audit_v2.py | 2 +- tests/test_events.py | 2 +- tests/test_storage_azure.py | 6 +++++- tests/test_storage_dropbox.py | 6 ++++-- tests/test_storage_fsspec.py | 6 +++--- tests/test_storage_ftp.py | 3 ++- tests/test_storage_onedrive.py | 3 ++- tests/test_storage_s3.py | 3 ++- tests/test_storage_sftp.py | 3 ++- tests/test_storage_smb.py | 3 ++- tests/test_storage_webdav.py | 15 ++++++++++++++- 17 files changed, 78 insertions(+), 26 deletions(-) diff --git a/.github/workflows/ci-dev.yml b/.github/workflows/ci-dev.yml index 0809a88..f338b1d 100644 --- a/.github/workflows/ci-dev.yml +++ b/.github/workflows/ci-dev.yml @@ -133,9 +133,10 @@ jobs: cache: pip - name: Build the distribution and check its metadata run: | - python -m pip install --upgrade pip build twine + # The locked, wheels-only tools the publish jobs build with. + python -m pip install --require-hashes --only-binary :all: -r .github/requirements/publish.txt cp dev.toml pyproject.toml - python -m build + python -m build --no-isolation twine check dist/* publish-dev: diff --git a/.github/workflows/ci-stable.yml b/.github/workflows/ci-stable.yml index 38aec62..9db360c 100644 --- a/.github/workflows/ci-stable.yml +++ b/.github/workflows/ci-stable.yml @@ -133,7 +133,8 @@ jobs: cache: pip - name: Build the distribution and check its metadata run: | - python -m pip install --upgrade pip build twine + # The locked, wheels-only tools the publish jobs build with. + python -m pip install --require-hashes --only-binary :all: -r .github/requirements/publish.txt cp stable.toml pyproject.toml - python -m build + python -m build --no-isolation twine check dist/* diff --git a/automation_file/local/versioning.py b/automation_file/local/versioning.py index 3b1a4f8..167f45e 100644 --- a/automation_file/local/versioning.py +++ b/automation_file/local/versioning.py @@ -16,6 +16,7 @@ from pathlib import Path from automation_file.exceptions import VersioningException +from automation_file.logging_config import file_automation_logger _VERSION_RE = re.compile(r"^v(\d+)__(\d+)$") # A flattened source path longer than this is shortened to its tail plus a digest: @@ -109,8 +110,14 @@ def _bucket_for(self, src: Path) -> Path: if len(safe) <= _MAX_BUCKET_NAME: return self._root / safe legacy = self._root / safe - if legacy.is_dir(): - return legacy + try: + if legacy.is_dir(): + return legacy + except OSError: + # A name past the filesystem's limit cannot be asked about, so it is not there. + file_automation_logger.debug( + "versioning: %d-character name is too long to exist", len(safe) + ) return self._root / _shortened(safe) def _next_version(self, bucket: Path) -> int: diff --git a/automation_file/storage/smb_storage.py b/automation_file/storage/smb_storage.py index 9f27238..e017cc0 100644 --- a/automation_file/storage/smb_storage.py +++ b/automation_file/storage/smb_storage.py @@ -208,8 +208,18 @@ def _rmdir(self, path: str) -> None: def _move_from(self, source: StorageBackend, source_path: str, path: str) -> bool: if not isinstance(source, SMBStorage) or source._client is not self._client: return False - with _smb_errors(self.uri_for(path)): - self._client.rename(source._remote(source_path), self._remote(path), overwrite=True) + origin, target = source._remote(source_path), self._remote(path) + try: + with _smb_errors(self.uri_for(path)): + self._client.rename(origin, target, overwrite=True) + except StoragePermissionException: + # Samba refuses to rename onto an existing file. Remove it and rename again: + # not atomic, and the one way that server replaces a file. + if self._stat(path) is None: + raise + with _smb_errors(self.uri_for(path)): + self._client.delete(target) + self._client.rename(origin, target, overwrite=False) return True def __eq__(self, other: object) -> bool: diff --git a/automation_file/storage/webdav_storage.py b/automation_file/storage/webdav_storage.py index 2ff3331..4ab1b5f 100644 --- a/automation_file/storage/webdav_storage.py +++ b/automation_file/storage/webdav_storage.py @@ -48,6 +48,7 @@ WEBDAV_SCHEME = "webdav" _MISSING_STATUS = 404 +_BELOW_A_FILE_STATUS = 400 _DENIED_STATUS = frozenset({401, 403}) _TRANSIENT_STATUS = frozenset({408, 429}) _SERVER_ERROR = 500 @@ -190,6 +191,11 @@ def _stat(self, path: str) -> FileInfo | None: entry = self._client.stat(self._remote(path)) except StorageNotFoundException: return None + except StorageException as error: + # Apache answers 400, not 404, for a path below a file: nothing can be there. + if _status_of(error) != _BELOW_A_FILE_STATUS: + raise + return None return _file_info(path, entry) def _list_dir(self, path: str) -> Iterable[FileInfo]: diff --git a/tests/integration/start_service.sh b/tests/integration/start_service.sh index 38760e8..ff68b4a 100755 --- a/tests/integration/start_service.sh +++ b/tests/integration/start_service.sh @@ -42,7 +42,7 @@ case "$service" in s3) docker run -d --name fa-it-s3 -p 9000:9000 \ -e "MINIO_ROOT_USER=$user-integration" -e "MINIO_ROOT_PASSWORD=$secret" \ - minio/minio server /data >&2 + quay.io/minio/minio server /data >&2 wait_for_port 9000 emit FA_IT_S3_ENDPOINT "http://127.0.0.1:9000" emit FA_IT_S3_ACCESS_KEY "$user-integration" @@ -86,6 +86,8 @@ case "$service" in emit FA_IT_FTP_PORT 2121 emit FA_IT_FTP_USER "$user" emit FA_IT_FTP_PASSWORD "$secret" + # The login lands in the filesystem root, which is read-only; the user's own directory is not. + emit FA_IT_FTP_ROOT "/ftp/$user" ;; webdav) docker run -d --name fa-it-webdav -p 8080:80 \ @@ -97,11 +99,12 @@ case "$service" in emit FA_IT_WEBDAV_PASSWORD "$secret" ;; smb) - docker run -d --name fa-it-smb -p 4450:445 \ + # On port 445 itself: smbprotocol drops a non-default port in some of its calls. + docker run -d --name fa-it-smb -p 445:445 \ dperson/samba -p -u "$user;$secret" -s "share;/share;yes;no;no;$user" >&2 - wait_for_port 4450 + wait_for_port 445 emit FA_IT_SMB_SERVER 127.0.0.1 - emit FA_IT_SMB_PORT 4450 + emit FA_IT_SMB_PORT 445 emit FA_IT_SMB_SHARE share emit FA_IT_SMB_USER "$user" emit FA_IT_SMB_PASSWORD "$secret" diff --git a/tests/test_audit_v2.py b/tests/test_audit_v2.py index bac4ad9..9585a34 100644 --- a/tests/test_audit_v2.py +++ b/tests/test_audit_v2.py @@ -139,7 +139,7 @@ def test_a_record_is_frozen_and_keyword_only() -> None: record.status = "error" with pytest.raises(TypeError): AuditRecord("upload") - assert hash(record) == hash(record) + assert isinstance(hash(record), int) def test_a_record_turns_into_json_and_back() -> None: diff --git a/tests/test_events.py b/tests/test_events.py index 13a240d..1fc258c 100644 --- a/tests/test_events.py +++ b/tests/test_events.py @@ -94,7 +94,7 @@ def test_an_event_is_frozen_and_keyword_only() -> None: event.subject = "changed" # type: ignore[misc] with pytest.raises(TypeError): TaskFailed("positional") # type: ignore[misc] - assert hash(event) == hash(event) + assert isinstance(hash(event), int) def test_to_dict_is_json_serialisable() -> None: diff --git a/tests/test_storage_azure.py b/tests/test_storage_azure.py index da98798..37933aa 100644 --- a/tests/test_storage_azure.py +++ b/tests/test_storage_azure.py @@ -300,7 +300,11 @@ def test_a_container_name_is_required() -> None: def test_equality_and_repr(service: FakeBlobService) -> None: - assert AzureStorage("container", service=service) == AzureStorage("container", service=service) + first, second = ( + AzureStorage("container", service=service), + AzureStorage("container", service=service), + ) + assert first == second assert AzureStorage("container", service=service) != AzureStorage("archive", service=service) assert AzureStorage("container", service=service) != AzureStorage( "container", service=FakeBlobService() diff --git a/tests/test_storage_dropbox.py b/tests/test_storage_dropbox.py index 71e6aaa..5475f63 100644 --- a/tests/test_storage_dropbox.py +++ b/tests/test_storage_dropbox.py @@ -638,10 +638,12 @@ def test_uri_equality_and_repr(client: FakeDropbox) -> None: assert DropboxStorage(client).uri_for("/a//b.txt") == "dropbox:///a/b.txt" assert DropboxStorage(client, root="team").uri_for("a.txt") == "dropbox:///team/a.txt" assert DropboxStorage(client, root="team").uri_for() == "dropbox:///team" - assert DropboxStorage(client) == DropboxStorage(client) + first, second = DropboxStorage(client), DropboxStorage(client) + assert first == second assert DropboxStorage(client) != DropboxStorage(client, root="team") assert DropboxStorage(client) != DropboxStorage(FakeDropbox()) - assert DropboxStorage() == DropboxStorage() + first, second = DropboxStorage(), DropboxStorage() + assert first == second assert len({DropboxStorage(client), DropboxStorage(client)}) == 1 assert repr(DropboxStorage(client, root="team/a")) == "DropboxStorage(root='team/a')" assert DropboxStorage.scheme == DROPBOX_SCHEME == "dropbox" diff --git a/tests/test_storage_fsspec.py b/tests/test_storage_fsspec.py index 343c1af..0f6a358 100644 --- a/tests/test_storage_fsspec.py +++ b/tests/test_storage_fsspec.py @@ -552,9 +552,9 @@ def test_uri_equality_and_repr(tmp_path: Path) -> None: assert storage != FsspecStorage(KeyValueFileSystem(), root="bucket/tenant") assert len({storage, FsspecStorage(filesystem, root="bucket/tenant")}) == 1 # fsspec hands out one object per filesystem configuration, so these two are one storage. - assert FsspecStorage(LocalFileSystem(), root=str(tmp_path)) == FsspecStorage( - LocalFileSystem(), root=str(tmp_path) - ) + first = FsspecStorage(LocalFileSystem(), root=str(tmp_path)) + second = FsspecStorage(LocalFileSystem(), root=str(tmp_path)) + assert first == second # ---------------------------------------------------------------------- the real fsspec diff --git a/tests/test_storage_ftp.py b/tests/test_storage_ftp.py index d8f8f8a..6e510ae 100644 --- a/tests/test_storage_ftp.py +++ b/tests/test_storage_ftp.py @@ -341,7 +341,8 @@ def test_uri_for_names_the_host_of_the_session(server: FakeFTPServer) -> None: def test_equality_and_repr(server: FakeFTPServer) -> None: client = connected(FakeFTP(server)) - assert FTPStorage(client) == FTPStorage(client) + first, second = FTPStorage(client), FTPStorage(client) + assert first == second assert FTPStorage(client) != FTPStorage(client, root="/srv") assert FTPStorage(client) != FTPStorage(connected(FakeFTP(server))) assert FTPStorage() == FTPStorage(ftp_instance) diff --git a/tests/test_storage_onedrive.py b/tests/test_storage_onedrive.py index ff2eb9f..6fd4b24 100644 --- a/tests/test_storage_onedrive.py +++ b/tests/test_storage_onedrive.py @@ -618,7 +618,8 @@ def test_equality_and_repr(client: OneDriveClient, graph: FakeGraph) -> None: assert OneDriveStorage(client) != OneDriveStorage(client, root="a") assert OneDriveStorage(client) != OneDriveStorage(_client(graph)) assert OneDriveStorage(client) != OneDriveStorage() - assert OneDriveStorage() == OneDriveStorage() + first, second = OneDriveStorage(), OneDriveStorage() + assert first == second assert len({OneDriveStorage(client), OneDriveStorage(client)}) == 1 assert repr(OneDriveStorage(client, root="a/b")) == "OneDriveStorage(root='a/b')" assert OneDriveStorage.scheme == ONEDRIVE_SCHEME == "onedrive" diff --git a/tests/test_storage_s3.py b/tests/test_storage_s3.py index 32a2910..04712c3 100644 --- a/tests/test_storage_s3.py +++ b/tests/test_storage_s3.py @@ -318,7 +318,8 @@ def test_a_bucket_name_is_required() -> None: def test_equality_and_repr(client: FakeS3Client) -> None: - assert S3Storage("bucket", client=client) == S3Storage("bucket", client=client) + first, second = S3Storage("bucket", client=client), S3Storage("bucket", client=client) + assert first == second assert S3Storage("bucket", client=client) != S3Storage("archive", client=client) assert S3Storage("bucket", client=client) != S3Storage("bucket", client=client, prefix="a") assert S3Storage("bucket", client=client) != S3Storage("bucket", client=FakeS3Client()) diff --git a/tests/test_storage_sftp.py b/tests/test_storage_sftp.py index ffb4480..fdeb0b0 100644 --- a/tests/test_storage_sftp.py +++ b/tests/test_storage_sftp.py @@ -402,7 +402,8 @@ def test_uri_for_names_the_host_of_the_session(session: FakeSFTP) -> None: def test_equality_and_repr(session: FakeSFTP) -> None: client = connected(session) - assert SFTPStorage(client) == SFTPStorage(client) + first, second = SFTPStorage(client), SFTPStorage(client) + assert first == second assert SFTPStorage(client) != SFTPStorage(client, root="/srv") assert SFTPStorage(client) != SFTPStorage(connected(session)) assert SFTPStorage() == SFTPStorage(sftp_instance) diff --git a/tests/test_storage_smb.py b/tests/test_storage_smb.py index a9b9b92..180d4b5 100644 --- a/tests/test_storage_smb.py +++ b/tests/test_storage_smb.py @@ -481,7 +481,8 @@ def test_uri_equality_and_repr(client: SMBClient) -> None: assert rooted.root == "team/a" assert rooted.uri_for("b.txt") == f"smb://{SERVER}/{SHARE}/team/a/b.txt" assert repr(rooted) == f"SMBStorage('smb://{SERVER}/{SHARE}/team/a')" - assert SMBStorage(client) == SMBStorage(client) + first, second = SMBStorage(client), SMBStorage(client) + assert first == second assert SMBStorage(client) != rooted assert SMBStorage(client) != SMBStorage(SMBClient(SERVER, SHARE)) assert len({SMBStorage(client), SMBStorage(client)}) == 1 diff --git a/tests/test_storage_webdav.py b/tests/test_storage_webdav.py index 5869be1..7f891ab 100644 --- a/tests/test_storage_webdav.py +++ b/tests/test_storage_webdav.py @@ -571,7 +571,8 @@ def test_uri_equality_and_repr(client: WebDAVClient) -> None: assert rooted.root == "team/a" assert rooted.uri_for("b.txt") == f"webdav://{HOST}{DAV_ROOT}/team/a/b.txt" assert repr(rooted) == f"WebDAVStorage('webdav://{HOST}{DAV_ROOT}/team/a')" - assert WebDAVStorage(client) == WebDAVStorage(client) + first, second = WebDAVStorage(client), WebDAVStorage(client) + assert first == second assert WebDAVStorage(client) != rooted assert WebDAVStorage(client) != WebDAVStorage(WebDAVClient(BASE_URL)) assert len({WebDAVStorage(client), WebDAVStorage(client)}) == 1 @@ -606,3 +607,15 @@ def test_two_roots_of_one_server_do_not_lose_a_file_to_itself(client: WebDAVClie with pytest.raises(StorageException, match="same file"): inner.move_from(whole, "team/a/docs/a.txt", "docs/a.txt") assert whole.read_bytes("team/a/docs/a.txt") == b"payload" + + +def test_a_400_to_a_stat_means_nothing_is_there( + server: FakeDavServer, client: WebDAVClient +) -> None: + # Apache answers 400 to PROPFIND for a path below a file, where others answer 404. + storage = WebDAVStorage(client) + server.status_for["PROPFIND"] = 400 + assert storage.exists("a.txt/child.txt") is False + server.status_for["PROPFIND"] = 500 + with pytest.raises(StorageException): + storage.exists("a.txt/child.txt") From d578a0b605ecd9c7268beb173f5b19eb6b9bbd86 Mon Sep 17 00:00:00 2001 From: JeffreyChen Date: Thu, 8 Oct 2026 16:19:57 +0800 Subject: [PATCH 51/59] chore: answer the static-analysis findings of the pull request File-level pylint disables in the test modules for the rules pytest idioms trip, each with its reason; markers with reasons where a scanner reads a test digest or a constant SQL statement as a risk; nine long lines wrapped; one variable renamed. The S3 integration job uses S3Mock, since MinIO's image can no longer be pulled. --- automation_file/app/pipeline_draft.py | 1 + automation_file/app/pipeline_service.py | 1 + automation_file/app/storage_service.py | 1 + automation_file/audit/sqlite_store.py | 6 ++++-- automation_file/audit/store.py | 3 ++- automation_file/core/optional.py | 1 + automation_file/events/model.py | 3 ++- automation_file/integrity/target.py | 5 ++++- automation_file/pipeline/store.py | 1 + automation_file/scheduler/manager.py | 1 + automation_file/scheduler/triggers.py | 1 + automation_file/storage/backend.py | 11 +++++++++-- automation_file/storage/dropbox_storage.py | 2 ++ automation_file/storage/file.py | 2 ++ automation_file/storage/fsspec_storage.py | 2 ++ automation_file/storage/gdrive_storage.py | 2 ++ automation_file/storage/local_storage.py | 2 ++ automation_file/storage/onedrive_storage.py | 4 +++- automation_file/storage/resolver.py | 1 + automation_file/storage/s3_storage.py | 2 ++ automation_file/storage/session_storage.py | 2 ++ automation_file/storage/sftp_storage.py | 5 ++++- automation_file/storage/smb_storage.py | 2 ++ automation_file/storage/streams.py | 1 + automation_file/storage/webdav_storage.py | 2 ++ automation_file/ui/pages/integrity_page.py | 3 ++- automation_file/ui/pages/pipelines_page.py | 11 ++++++----- automation_file/ui/pages/task_form.py | 1 + scripts/stable_release.py | 2 +- tests/drive_stand_in.py | 11 ++++++++--- tests/ftp_stand_in.py | 4 +++- tests/graph_stand_in.py | 5 +++++ tests/integration/service_env.py | 2 ++ tests/integration/start_service.sh | 6 +++--- tests/integration/test_azure_azurite.py | 4 ++++ tests/integration/test_ftp_server.py | 3 +++ tests/integration/test_s3_minio.py | 4 ++++ tests/integration/test_sftp_openssh.py | 4 ++++ tests/integration/test_smb_samba.py | 4 ++++ tests/integration/test_webdav_server.py | 3 +++ tests/storage_contract.py | 5 ++++- tests/test_app_files.py | 2 ++ tests/test_app_masking.py | 13 ++++++++----- tests/test_app_pipeline_draft.py | 2 ++ tests/test_app_pipelines.py | 2 +- tests/test_app_services.py | 9 ++++++--- tests/test_audit_v2.py | 6 +++++- tests/test_backends.py | 4 +++- tests/test_cli_operations.py | 2 ++ tests/test_config.py | 2 ++ tests/test_cross_backend_storage.py | 3 +++ tests/test_events.py | 4 ++++ tests/test_ftp_ops.py | 5 ++++- tests/test_integrity_actions.py | 3 +++ tests/test_integrity_legacy.py | 3 +++ tests/test_integrity_monitor.py | 5 ++++- tests/test_integrity_object_store.py | 8 ++++++-- tests/test_integrity_remediation.py | 4 ++++ tests/test_integrity_snapshot.py | 2 +- tests/test_integrity_watch.py | 4 ++++ tests/test_mcp_pipeline_tools.py | 2 ++ tests/test_mcp_policy.py | 3 +++ tests/test_mcp_semantic_server.py | 3 +++ tests/test_mcp_storage_tools.py | 2 ++ tests/test_mcp_tools.py | 6 ++++-- tests/test_notify.py | 2 ++ tests/test_notify_router.py | 4 ++++ tests/test_operational_metrics.py | 2 ++ tests/test_optional_dependencies.py | 4 +++- tests/test_pipeline_definition.py | 7 +++++-- tests/test_pipeline_events.py | 2 ++ tests/test_pipeline_rejected.py | 2 ++ tests/test_pipeline_run.py | 3 +++ tests/test_pipeline_store.py | 6 +++++- tests/test_scheduler_actions.py | 2 ++ tests/test_scheduler_lifecycle.py | 3 +++ tests/test_scheduler_pipeline.py | 2 ++ tests/test_scheduler_runs.py | 4 ++++ tests/test_scheduler_triggers.py | 6 +++++- tests/test_storage_actions.py | 2 +- tests/test_storage_azure.py | 8 +++++++- tests/test_storage_dropbox.py | 6 ++++++ tests/test_storage_file.py | 7 +++++-- tests/test_storage_fsspec.py | 7 +++++++ tests/test_storage_ftp.py | 5 +++++ tests/test_storage_gdrive.py | 8 +++++++- tests/test_storage_local.py | 3 +++ tests/test_storage_observe.py | 2 ++ tests/test_storage_onedrive.py | 4 ++++ tests/test_storage_resolver.py | 2 ++ tests/test_storage_s3.py | 12 ++++++++++-- tests/test_storage_sftp.py | 5 +++++ tests/test_storage_sftp_loopback.py | 7 +++++++ tests/test_storage_smb.py | 4 ++++ tests/test_storage_tree.py | 2 ++ tests/test_storage_types.py | 2 ++ tests/test_storage_webdav.py | 14 +++++++++++--- tests/test_ui_pages.py | 8 ++++++-- tests/test_ui_pipeline_editor.py | 6 ++++++ tests/test_ui_smoke.py | 3 +++ tests/test_web_ui_app_layer.py | 4 +++- 101 files changed, 345 insertions(+), 60 deletions(-) diff --git a/automation_file/app/pipeline_draft.py b/automation_file/app/pipeline_draft.py index be49d31..743d8b2 100644 --- a/automation_file/app/pipeline_draft.py +++ b/automation_file/app/pipeline_draft.py @@ -187,6 +187,7 @@ def _task_from_spec(task_id: str, spec: Mapping[str, Any]) -> DraftTask: return task +# pylint: disable-next=too-many-public-methods # one method per edit the editor offers class PipelineDraft: """A pipeline definition being edited, with the canvas layout next to it.""" diff --git a/automation_file/app/pipeline_service.py b/automation_file/app/pipeline_service.py index c68ae2b..a3aa67a 100644 --- a/automation_file/app/pipeline_service.py +++ b/automation_file/app/pipeline_service.py @@ -215,6 +215,7 @@ def dry_run( run = self._pipeline(definition).run(params=params, dry_run=True) return self._view(run, active=False) + # pylint: disable-next=too-many-locals # one task run from start to result, told in order def test_task( self, definition: Definition, diff --git a/automation_file/app/storage_service.py b/automation_file/app/storage_service.py index 99521f0..28db440 100644 --- a/automation_file/app/storage_service.py +++ b/automation_file/app/storage_service.py @@ -202,6 +202,7 @@ def is_installed(module: str | None) -> bool: def _client_ready(client: _Client) -> bool: """Ask the shared client whether it has been initialised. Never opens a connection.""" try: + # nosemgrep # the module name is one of this package's own client modules, from a fixed table instance = getattr(importlib.import_module(client.module), client.attribute) probe: Callable[[], Any] = getattr(instance, client.probe) probe() diff --git a/automation_file/audit/sqlite_store.py b/automation_file/audit/sqlite_store.py index d271031..0058ffe 100644 --- a/automation_file/audit/sqlite_store.py +++ b/automation_file/audit/sqlite_store.py @@ -88,9 +88,10 @@ "duration_ms, error, metadata, correlation_id" ) _PLACEHOLDERS = "?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?" -_INSERT = f"INSERT INTO audit_records ({_COLUMNS}) VALUES ({_PLACEHOLDERS})" +# The two fragments are constants of this module; every value goes in as a parameter. +_INSERT = f"INSERT INTO audit_records ({_COLUMNS}) VALUES ({_PLACEHOLDERS})" # nosec B608 _INSERT_NEW = f"INSERT OR IGNORE INTO audit_records ({_COLUMNS}) VALUES ({_PLACEHOLDERS})" -_SELECT = f"SELECT {_COLUMNS} FROM audit_records" +_SELECT = f"SELECT {_COLUMNS} FROM audit_records" # nosec B608 _COUNT = "SELECT COUNT(*) FROM audit_records" _NEWEST_FIRST = " ORDER BY ts_us DESC, seq DESC LIMIT ? OFFSET ?" _PURGE = "DELETE FROM audit_records WHERE ts_us < ?" @@ -228,6 +229,7 @@ def append(self, record: AuditRecord) -> None: check_record(record) try: with self._lock, self._conn: + # nosemgrep # a constant statement with bound parameters self._conn.execute(_INSERT, _row(record)) except sqlite3.IntegrityError as err: raise AuditException(f"audit record {record.id} is already stored") from err diff --git a/automation_file/audit/store.py b/automation_file/audit/store.py index 630876b..9fb8696 100644 --- a/automation_file/audit/store.py +++ b/automation_file/audit/store.py @@ -9,7 +9,8 @@ ``since`` / ``until`` The time range, ``since`` included and ``until`` excluded. An aware ``datetime``, an ISO 8601 string with an offset, or seconds since the epoch. -``actor``, ``source``, ``pipeline``, ``task``, ``action``, ``backend``, ``status``, ``correlation_id`` +``actor``, ``source``, ``pipeline``, ``task``, ``action``, ``backend``, ``status``, +``correlation_id`` Exact matches. ``resource_prefix`` Records whose resource starts with the text. diff --git a/automation_file/core/optional.py b/automation_file/core/optional.py index dabe28a..effaea2 100644 --- a/automation_file/core/optional.py +++ b/automation_file/core/optional.py @@ -44,6 +44,7 @@ def require_module(name: str, *, extra: str) -> ModuleType: cannot be imported. """ try: + # nosemgrep # callers pass the literal name of an optional SDK, never input from outside return importlib.import_module(name) except ImportError as error: feature = EXTRAS.get(extra, f"the {extra} feature") diff --git a/automation_file/events/model.py b/automation_file/events/model.py index dab948a..ba01a26 100644 --- a/automation_file/events/model.py +++ b/automation_file/events/model.py @@ -6,7 +6,8 @@ details in ``payload`` under the keys listed in :data:`PAYLOAD_KEYS`. The ten core events are subclasses that fix the ``type`` and the default -severity (which a caller may still override). ``SystemErrorEvent`` is the roadmap's "SystemError"; the shorter name +severity (which a caller may still override). ``SystemErrorEvent`` is the roadmap's +"SystemError"; the shorter name would shadow Python's builtin exception. """ diff --git a/automation_file/integrity/target.py b/automation_file/integrity/target.py index 315b1a3..5cc7e2b 100644 --- a/automation_file/integrity/target.py +++ b/automation_file/integrity/target.py @@ -77,7 +77,10 @@ def fold(self, path: str) -> str: return path.casefold() if self._case_blind else path def relative(self, other: URILike) -> str | None: - """Return the path of ``other`` inside this tree: ``""`` for the tree itself, ``None`` outside.""" + """Return the path of ``other`` inside this tree. + + ``""`` is the tree itself and ``None`` means outside it. + """ candidate = parse_storage_uri(other) if candidate.scheme != self._uri.scheme: return None diff --git a/automation_file/pipeline/store.py b/automation_file/pipeline/store.py index 1887b3d..aa9f9dd 100644 --- a/automation_file/pipeline/store.py +++ b/automation_file/pipeline/store.py @@ -237,6 +237,7 @@ def path(self) -> Path: @contextmanager def _session(self) -> Iterator[sqlite3.Connection]: try: + # pylint: disable-next=confusing-with-statement # three managers, entered in this order with ( self._lock, closing(sqlite3.connect(self._path, timeout=_BUSY_SECONDS)) as connection, diff --git a/automation_file/scheduler/manager.py b/automation_file/scheduler/manager.py index 030bb00..08a3241 100644 --- a/automation_file/scheduler/manager.py +++ b/automation_file/scheduler/manager.py @@ -576,6 +576,7 @@ def schedule_job( ) +# pylint: disable-next=too-many-positional-arguments # a JSON action: by name or by position def schedule_pipeline( definition: Any, name: str | None = None, diff --git a/automation_file/scheduler/triggers.py b/automation_file/scheduler/triggers.py index 077dcaa..b1a305b 100644 --- a/automation_file/scheduler/triggers.py +++ b/automation_file/scheduler/triggers.py @@ -70,6 +70,7 @@ class Trigger(ABC): def to_dict(self) -> dict[str, Any]: """Return the JSON-friendly form of the trigger, with its ``kind``.""" + # pylint: disable-next=unused-argument # the default trigger needs nothing from the port def arm(self, port: TriggerPort) -> Disarm | None: """Start watching for the trigger's moment and return the call that stops it. diff --git a/automation_file/storage/backend.py b/automation_file/storage/backend.py index e13dca7..2d77097 100644 --- a/automation_file/storage/backend.py +++ b/automation_file/storage/backend.py @@ -14,6 +14,8 @@ The root itself is the empty string. """ +# pylint: disable=protected-access # a backend reads the private parts of another instance of its own kind + from __future__ import annotations import contextlib @@ -163,10 +165,12 @@ def _walk(self, path: str) -> Iterable[FileInfo]: pending.append(info.path) return found + # pylint: disable-next=unused-argument # a hook: the default declines, a backend overrides it def _copy_from(self, source: StorageBackend, source_path: str, path: str) -> bool: """Copy a file from ``source`` without a local staging copy; ``False`` if unable.""" return False + # pylint: disable-next=unused-argument # a hook: the default declines, a backend overrides it def _move_from(self, source: StorageBackend, source_path: str, path: str) -> bool: """Move a file from ``source`` natively (a rename); ``False`` if unable.""" return False @@ -243,7 +247,7 @@ def exists(self, path: str) -> bool: return self._stat(self._normalize(path)) is not None def stat(self, path: str) -> FileInfo: - """Return the :class:`FileInfo` of ``path``; raise ``StorageNotFoundException`` if absent.""" + """Return the :class:`FileInfo` of ``path``, or raise ``StorageNotFoundException``.""" clean = self._normalize(path) info = self._stat(clean) if info is None: @@ -393,7 +397,10 @@ def copy_from( def move_from( self, source: StorageBackend, source_path: str, path: str, *, overwrite: bool = True ) -> FileInfo: - """Move the file ``source_path`` of ``source`` to ``path``: a rename, or copy then delete.""" + """Move the file ``source_path`` of ``source`` to ``path``. + + A rename where the backends can, a copy followed by a delete otherwise. + """ with self._observing("move", path, (source, source_path)), observe.suppressed(): origin, target = self._transfer_paths(source, source_path, path, overwrite) if not self._move_from(source, origin, target): diff --git a/automation_file/storage/dropbox_storage.py b/automation_file/storage/dropbox_storage.py index 467df0b..0b46387 100644 --- a/automation_file/storage/dropbox_storage.py +++ b/automation_file/storage/dropbox_storage.py @@ -16,6 +16,8 @@ deleted first, then the copy or move runs. """ +# pylint: disable=protected-access # a backend reads the private parts of another instance of its own kind + from __future__ import annotations import contextlib diff --git a/automation_file/storage/file.py b/automation_file/storage/file.py index 0280d7f..a58522a 100644 --- a/automation_file/storage/file.py +++ b/automation_file/storage/file.py @@ -15,6 +15,8 @@ backend is initialised or mounted. """ +# pylint: disable=protected-access # File reads the private locator of another File + from __future__ import annotations import os diff --git a/automation_file/storage/fsspec_storage.py b/automation_file/storage/fsspec_storage.py index 05eab0b..defc043 100644 --- a/automation_file/storage/fsspec_storage.py +++ b/automation_file/storage/fsspec_storage.py @@ -18,6 +18,8 @@ paths without ``*``, ``?`` or ``[``; other moves are a copy followed by a delete. """ +# pylint: disable=protected-access # a backend reads the private parts of another instance of its own kind + from __future__ import annotations import contextlib diff --git a/automation_file/storage/gdrive_storage.py b/automation_file/storage/gdrive_storage.py index 93ffbd4..8f88942 100644 --- a/automation_file/storage/gdrive_storage.py +++ b/automation_file/storage/gdrive_storage.py @@ -29,6 +29,8 @@ raise :class:`~automation_file.exceptions.StorageUnsupportedException`. """ +# pylint: disable=protected-access # a backend reads the private parts of another instance of its own kind + from __future__ import annotations import contextlib diff --git a/automation_file/storage/local_storage.py b/automation_file/storage/local_storage.py index 62f6679..4d41f7f 100644 --- a/automation_file/storage/local_storage.py +++ b/automation_file/storage/local_storage.py @@ -13,6 +13,8 @@ them: the link is removed and its target is left alone. """ +# pylint: disable=protected-access # a backend reads the private parts of another instance of its own kind + from __future__ import annotations import contextlib diff --git a/automation_file/storage/onedrive_storage.py b/automation_file/storage/onedrive_storage.py index aa04bc4..838d822 100644 --- a/automation_file/storage/onedrive_storage.py +++ b/automation_file/storage/onedrive_storage.py @@ -22,6 +22,8 @@ ``stat`` reports the size, modification time, ETag and MIME type of a file. """ +# pylint: disable=protected-access # a backend reads the private parts of another instance of its own kind + from __future__ import annotations import contextlib @@ -436,7 +438,7 @@ def _delete_directory(self, path: str, recursive: bool) -> None: raise not_empty_error(location) self._send(_DELETE, self._item_url(path), location) - # ------------------------------------------------------------------ copy and move inside OneDrive + # ------------------------------------------------------------------ copy and move in OneDrive def _copy_from(self, source: StorageBackend, source_path: str, path: str) -> bool: if isinstance(source, OneDriveStorage): diff --git a/automation_file/storage/resolver.py b/automation_file/storage/resolver.py index 7173544..227765f 100644 --- a/automation_file/storage/resolver.py +++ b/automation_file/storage/resolver.py @@ -130,6 +130,7 @@ def _find_mount(self, uri: StorageURI) -> tuple[StorageBackend, str] | None: if scheme != uri.scheme or mounted_authority != authority: continue relative = _below(uri.path, prefix) + # pylint: disable-next=unsubscriptable-object # best is a tuple once it is not None if relative is not None and (best is None or len(prefix) > best[0]): best = (len(prefix), backend, relative) return None if best is None else (best[1], best[2]) diff --git a/automation_file/storage/s3_storage.py b/automation_file/storage/s3_storage.py index c84ba65..850c564 100644 --- a/automation_file/storage/s3_storage.py +++ b/automation_file/storage/s3_storage.py @@ -12,6 +12,8 @@ not a digest of a multipart upload, so it is never used as one. """ +# pylint: disable=protected-access # a backend reads the private parts of another instance of its own kind + from __future__ import annotations import contextlib diff --git a/automation_file/storage/session_storage.py b/automation_file/storage/session_storage.py index e151532..841425d 100644 --- a/automation_file/storage/session_storage.py +++ b/automation_file/storage/session_storage.py @@ -15,6 +15,8 @@ share: a URI names no host, or the host of the open session. """ +# pylint: disable=protected-access # a backend reads the private parts of another instance of its own kind + from __future__ import annotations import contextlib diff --git a/automation_file/storage/sftp_storage.py b/automation_file/storage/sftp_storage.py index ce729d3..ad37de3 100644 --- a/automation_file/storage/sftp_storage.py +++ b/automation_file/storage/sftp_storage.py @@ -48,7 +48,10 @@ def _closed(sftp: Any) -> bool: - """Say whether the channel under ``sftp`` is closed; using it then raises a plain ``OSError``.""" + """Say whether the channel under ``sftp`` is closed. + + Using a closed channel raises a plain ``OSError``. + """ channel = sftp.get_channel() return channel is None or bool(channel.closed) diff --git a/automation_file/storage/smb_storage.py b/automation_file/storage/smb_storage.py index e017cc0..925fcb0 100644 --- a/automation_file/storage/smb_storage.py +++ b/automation_file/storage/smb_storage.py @@ -14,6 +14,8 @@ local staging file. """ +# pylint: disable=protected-access # a backend reads the private parts of another instance of its own kind + from __future__ import annotations import contextlib diff --git a/automation_file/storage/streams.py b/automation_file/storage/streams.py index 4863c0c..1962375 100644 --- a/automation_file/storage/streams.py +++ b/automation_file/storage/streams.py @@ -56,6 +56,7 @@ def discard(self) -> None: self.close() def close(self) -> None: + # pylint: disable-next=using-constant-test # closed is a property of the stream if self.closed: return commit, self._commit = self._commit, None diff --git a/automation_file/storage/webdav_storage.py b/automation_file/storage/webdav_storage.py index 4ab1b5f..80af1bb 100644 --- a/automation_file/storage/webdav_storage.py +++ b/automation_file/storage/webdav_storage.py @@ -15,6 +15,8 @@ ``MOVE``), and deleting a directory is one ``DELETE``. """ +# pylint: disable=protected-access # a backend reads the private parts of another instance of its own kind + from __future__ import annotations import contextlib diff --git a/automation_file/ui/pages/integrity_page.py b/automation_file/ui/pages/integrity_page.py index 2c05b6e..3c22159 100644 --- a/automation_file/ui/pages/integrity_page.py +++ b/automation_file/ui/pages/integrity_page.py @@ -143,7 +143,8 @@ def verify(self) -> None: def accept(self) -> None: target, baseline = self._target.text(), self._baseline.text() if not self.confirm( - "Approve the current state as the new baseline? Drift found so far will no longer be reported." + "Approve the current state as the new baseline? " + "Drift found so far will no longer be reported." ): return self.run_async( diff --git a/automation_file/ui/pages/pipelines_page.py b/automation_file/ui/pages/pipelines_page.py index dded663..fdb2501 100644 --- a/automation_file/ui/pages/pipelines_page.py +++ b/automation_file/ui/pages/pipelines_page.py @@ -59,6 +59,7 @@ _EDITOR_WIDTHS = (220, 560, 420) +# pylint: disable-next=too-many-instance-attributes,too-many-public-methods # one editor: widgets and slots class PipelinesPage(BasePage): """Build a pipeline on a canvas, check it, run it and follow the run.""" @@ -214,10 +215,10 @@ def _run_bar(self) -> QVBoxLayout: for label, handler in buttons: row.addWidget(self.make_button(label, handler)) row.addStretch() - bar = QVBoxLayout() - bar.addLayout(params) - bar.addLayout(row) - return bar + controls = QVBoxLayout() + controls.addLayout(params) + controls.addLayout(row) + return controls # ------------------------------------------------------------------ parts, for callers @@ -574,7 +575,7 @@ def follow_run(self, run_id: str) -> None: self.poll() def _stop_following(self) -> None: - """Stop asking about a run that cannot be read; the failure is already on the status line.""" + """Stop asking about a run that cannot be read; the status line already says why.""" self._following = False self._timer.stop() diff --git a/automation_file/ui/pages/task_form.py b/automation_file/ui/pages/task_form.py index 76f6051..53bc009 100644 --- a/automation_file/ui/pages/task_form.py +++ b/automation_file/ui/pages/task_form.py @@ -209,6 +209,7 @@ def _on_mode_toggled(self, as_json: bool) -> None: self._stack.setCurrentIndex(_TABLE_PAGE) +# pylint: disable-next=too-many-instance-attributes # one input widget per field of a task class TaskForm(QWidget): """Edits the selected task of a draft; nothing changes until Apply is pressed.""" diff --git a/scripts/stable_release.py b/scripts/stable_release.py index d664444..a47074f 100644 --- a/scripts/stable_release.py +++ b/scripts/stable_release.py @@ -21,7 +21,7 @@ import os import re -import subprocess +import subprocess # nosec B404 # one fixed git command, no shell import sys from collections.abc import Iterable from pathlib import Path diff --git a/tests/drive_stand_in.py b/tests/drive_stand_in.py index bee4930..6819039 100644 --- a/tests/drive_stand_in.py +++ b/tests/drive_stand_in.py @@ -19,6 +19,11 @@ says so. """ +# pylint: disable=line-too-long # an expected value is kept on one line +# pylint: disable=raising-bad-type # a stand-in raises what the test hands it +# pylint: disable=too-many-positional-arguments # a stand-in keeps the real signature +# pylint: disable=unsupported-membership-test # the value is a container at run time + from __future__ import annotations import hashlib @@ -116,8 +121,8 @@ def resource(self, *, digests: bool = True) -> dict[str, Any]: if self.data is not None: resource["size"] = str(len(self.data)) if digests: - resource["md5Checksum"] = hashlib.md5(self.data, usedforsecurity=False).hexdigest() - resource["sha1Checksum"] = hashlib.sha1( + resource["md5Checksum"] = hashlib.md5(self.data, usedforsecurity=False).hexdigest() # nosec B324 # nosemgrep # the digest under test, not a security use + resource["sha1Checksum"] = hashlib.sha1( # nosec B324 # nosemgrep # the digest under test, not a security use self.data, usedforsecurity=False ).hexdigest() resource["sha256Checksum"] = hashlib.sha256(self.data).hexdigest() @@ -208,7 +213,7 @@ def request( ): if isinstance(self.fail_with, Exception): raise self.fail_with - return error_answer(*self.fail_with) + return error_answer(*self.fail_with) # pylint: disable=not-an-iterable # a tuple here lowered = {key.lower(): value for key, value in (headers or {}).items()} try: if uri.startswith(UPLOAD_URL): diff --git a/tests/ftp_stand_in.py b/tests/ftp_stand_in.py index e8a9cd0..da9ea8e 100644 --- a/tests/ftp_stand_in.py +++ b/tests/ftp_stand_in.py @@ -8,6 +8,8 @@ answered with 550, and a rename that will not replace an existing file. """ +# pylint: disable=unused-argument # a fixture is requested for its effect; a stand-in keeps the real signature + from __future__ import annotations import ftplib # nosec B402 - the sessions built here lead to an in-memory server @@ -67,7 +69,7 @@ def __exit__(self, *exc_info: object) -> None: self.close() -class FakeFTPServer: +class FakeFTPServer: # pylint: disable=too-many-instance-attributes # the state of one in-memory server """An FTP server in memory: a tree of entries and the far end of one control connection.""" def __init__( diff --git a/tests/graph_stand_in.py b/tests/graph_stand_in.py index 839794a..e0ecf9f 100644 --- a/tests/graph_stand_in.py +++ b/tests/graph_stand_in.py @@ -15,6 +15,11 @@ deletes what is in it. """ +# pylint: disable=raising-bad-type # a stand-in raises what the test hands it +# pylint: disable=too-many-locals # one scenario told in order +# pylint: disable=too-many-positional-arguments # a stand-in keeps the real signature +# pylint: disable=unsupported-membership-test # the value is a container at run time + from __future__ import annotations import io diff --git a/tests/integration/service_env.py b/tests/integration/service_env.py index 37e9571..c518e16 100644 --- a/tests/integration/service_env.py +++ b/tests/integration/service_env.py @@ -6,6 +6,8 @@ would look green without having touched the service. """ +# pylint: disable=inconsistent-return-statements # the other branch raises, or no test reaches it + from __future__ import annotations import os diff --git a/tests/integration/start_service.sh b/tests/integration/start_service.sh index ff68b4a..7d32246 100755 --- a/tests/integration/start_service.sh +++ b/tests/integration/start_service.sh @@ -40,9 +40,9 @@ wait_for_port() { case "$service" in s3) - docker run -d --name fa-it-s3 -p 9000:9000 \ - -e "MINIO_ROOT_USER=$user-integration" -e "MINIO_ROOT_PASSWORD=$secret" \ - quay.io/minio/minio server /data >&2 + # MinIO no longer publishes an image that can be pulled without an account. S3Mock + # answers the S3 API on port 9090 and accepts any access key. + docker run -d --name fa-it-s3 -p 9000:9090 adobe/s3mock >&2 wait_for_port 9000 emit FA_IT_S3_ENDPOINT "http://127.0.0.1:9000" emit FA_IT_S3_ACCESS_KEY "$user-integration" diff --git a/tests/integration/test_azure_azurite.py b/tests/integration/test_azure_azurite.py index 67a19d1..166af24 100644 --- a/tests/integration/test_azure_azurite.py +++ b/tests/integration/test_azure_azurite.py @@ -3,6 +3,10 @@ Environment: ``FA_IT_AZURE_CONNECTION_STRING``. """ +# pylint: disable=arguments-differ # a fixture or a stand-in takes other arguments than the one it replaces +# pylint: disable=redefined-outer-name # pytest passes fixtures by matching name +# pylint: disable=ungrouped-imports # imports follow pytest.importorskip + from __future__ import annotations from collections.abc import Iterator diff --git a/tests/integration/test_ftp_server.py b/tests/integration/test_ftp_server.py index 0ece239..c95dda6 100644 --- a/tests/integration/test_ftp_server.py +++ b/tests/integration/test_ftp_server.py @@ -5,6 +5,9 @@ directory the user may write to) and ``FA_IT_FTP_TLS`` (``1`` for FTPS). """ +# pylint: disable=arguments-differ # a fixture or a stand-in takes other arguments than the one it replaces +# pylint: disable=redefined-outer-name # pytest passes fixtures by matching name + from __future__ import annotations from collections.abc import Iterator diff --git a/tests/integration/test_s3_minio.py b/tests/integration/test_s3_minio.py index 6057e43..3842190 100644 --- a/tests/integration/test_s3_minio.py +++ b/tests/integration/test_s3_minio.py @@ -4,6 +4,10 @@ ``FA_IT_S3_ACCESS_KEY``, ``FA_IT_S3_SECRET_KEY`` and optionally ``FA_IT_S3_REGION``. """ +# pylint: disable=arguments-differ # a fixture or a stand-in takes other arguments than the one it replaces +# pylint: disable=redefined-outer-name # pytest passes fixtures by matching name +# pylint: disable=ungrouped-imports # imports follow pytest.importorskip + from __future__ import annotations from collections.abc import Iterator diff --git a/tests/integration/test_sftp_openssh.py b/tests/integration/test_sftp_openssh.py index c6d5ce1..145f8d2 100644 --- a/tests/integration/test_sftp_openssh.py +++ b/tests/integration/test_sftp_openssh.py @@ -6,6 +6,10 @@ (``/upload``, an absolute directory the user may write to). """ +# pylint: disable=arguments-differ # a fixture or a stand-in takes other arguments than the one it replaces +# pylint: disable=redefined-outer-name # pytest passes fixtures by matching name +# pylint: disable=ungrouped-imports # imports follow pytest.importorskip + from __future__ import annotations from collections.abc import Iterator diff --git a/tests/integration/test_smb_samba.py b/tests/integration/test_smb_samba.py index 5207dc2..051dd4c 100644 --- a/tests/integration/test_smb_samba.py +++ b/tests/integration/test_smb_samba.py @@ -5,6 +5,10 @@ ``FA_IT_SMB_ENCRYPT`` (``1``; ``0`` for a server without SMB3 encryption). """ +# pylint: disable=arguments-differ # a fixture or a stand-in takes other arguments than the one it replaces +# pylint: disable=redefined-outer-name # pytest passes fixtures by matching name +# pylint: disable=ungrouped-imports # imports follow pytest.importorskip + from __future__ import annotations from collections.abc import Iterator diff --git a/tests/integration/test_webdav_server.py b/tests/integration/test_webdav_server.py index 02eeb76..d3703cb 100644 --- a/tests/integration/test_webdav_server.py +++ b/tests/integration/test_webdav_server.py @@ -4,6 +4,9 @@ ``FA_IT_WEBDAV_USER`` and ``FA_IT_WEBDAV_PASSWORD``. """ +# pylint: disable=arguments-differ # a fixture or a stand-in takes other arguments than the one it replaces +# pylint: disable=redefined-outer-name # pytest passes fixtures by matching name + from __future__ import annotations from collections.abc import Iterator diff --git a/tests/storage_contract.py b/tests/storage_contract.py index 80be73e..d3c85f3 100644 --- a/tests/storage_contract.py +++ b/tests/storage_contract.py @@ -20,6 +20,9 @@ def backend(self) -> StorageBackend: those cases skip. """ +# pylint: disable=unused-argument # a fixture is requested for its effect; a stand-in keeps the real signature +# pylint: disable=use-implicit-booleaness-not-comparison # an exact empty value is what is asserted + from __future__ import annotations import hashlib @@ -61,7 +64,7 @@ def _local_file(directory: Path, data: bytes, name: str = "source.bin") -> Path: return path -class StorageContract: +class StorageContract: # pylint: disable=too-many-public-methods # one method per contract case """Behaviour shared by every backend. Not collected on its own.""" @pytest.fixture diff --git a/tests/test_app_files.py b/tests/test_app_files.py index e9cd0e6..0546f92 100644 --- a/tests/test_app_files.py +++ b/tests/test_app_files.py @@ -1,5 +1,7 @@ """The Files and Storage services of the application layer, on private resolvers.""" +# pylint: disable=protected-access # the tests look at private state on purpose + from __future__ import annotations from collections.abc import Iterable diff --git a/tests/test_app_masking.py b/tests/test_app_masking.py index 2f7b52f..33c9388 100644 --- a/tests/test_app_masking.py +++ b/tests/test_app_masking.py @@ -1,5 +1,8 @@ """The application layer keeps secrets out of what it returns, and reads form text.""" +# pylint: disable=keyword-arg-before-vararg # a stand-in keeps the real signature +# pylint: disable=unused-argument # a fixture is requested for its effect; a stand-in keeps the real signature + from __future__ import annotations import pytest @@ -59,13 +62,13 @@ def test_url_names_are_told_from_storage_uris() -> None: def test_a_secret_value_is_replaced_whatever_its_type() -> None: masked = mask_secrets( - {"password": "hunter2", "token": {"value": "abc"}, "api_key": 12, "name": "ops"} + {"password": "hunter2", "token": {"value": "abc"}, "api_key": 12, "name": "ops"} # nosec B105 # a made-up value for a stand-in, not a credential ) assert masked == {"password": MASK, "token": MASK, "api_key": MASK, "name": "ops"} def test_an_empty_secret_stays_empty_so_a_view_can_tell_it_is_unset() -> None: - assert mask_secrets({"password": "", "token": None}) == {"password": "", "token": None} + assert mask_secrets({"password": "", "token": None}) == {"password": "", "token": None} # nosec B105 # a made-up value for a stand-in, not a credential def test_a_webhook_url_keeps_only_its_host() -> None: @@ -80,7 +83,7 @@ def test_a_url_field_without_an_http_url_is_masked_whole() -> None: def test_nested_values_are_walked_and_tuples_become_lists() -> None: - masked = mask_secrets({"sinks": ({"name": "a", "password": "x"}, {"name": "b"})}) + masked = mask_secrets({"sinks": ({"name": "a", "password": "x"}, {"name": "b"})}) # nosec B105 # a made-up value for a stand-in, not a credential assert masked == {"sinks": [{"name": "a", "password": MASK}, {"name": "b"}]} @@ -98,9 +101,9 @@ def test_a_storage_uri_is_left_alone() -> None: def test_the_input_is_not_changed() -> None: - original = {"password": "hunter2", "items": [{"token": "t"}]} + original = {"password": "hunter2", "items": [{"token": "t"}]} # nosec B105 # a made-up value for a stand-in, not a credential mask_secrets(original) - assert original == {"password": "hunter2", "items": [{"token": "t"}]} + assert original == {"password": "hunter2", "items": [{"token": "t"}]} # nosec B105 # a made-up value for a stand-in, not a credential def test_values_that_are_not_containers_pass_through() -> None: diff --git a/tests/test_app_pipeline_draft.py b/tests/test_app_pipeline_draft.py index 1cd94aa..1e858a1 100644 --- a/tests/test_app_pipeline_draft.py +++ b/tests/test_app_pipeline_draft.py @@ -1,5 +1,7 @@ """The editable pipeline draft of the application layer.""" +# pylint: disable=use-implicit-booleaness-not-comparison # an exact empty value is what is asserted + from __future__ import annotations import json diff --git a/tests/test_app_pipelines.py b/tests/test_app_pipelines.py index a6fc447..2d5f9ae 100644 --- a/tests/test_app_pipelines.py +++ b/tests/test_app_pipelines.py @@ -274,7 +274,7 @@ def test_a_run_the_store_says_is_running_counts_as_running(bus: EventBus) -> Non def test_secrets_in_the_parameters_are_masked_in_every_view(service: PipelineService) -> None: draft = _draft(("a", "T_echo", {"value": "${params.word}"})) - params = {"word": "hi", "password": "hunter2"} + params = {"word": "hi", "password": "hunter2"} # nosec B105 # a made-up value for a stand-in, not a credential started = service.start(draft, params) assert started["params"] == {"word": "hi", "password": MASK} done = _finished(service, started["run_id"]) diff --git a/tests/test_app_services.py b/tests/test_app_services.py index b0d0f27..e1f60d7 100644 --- a/tests/test_app_services.py +++ b/tests/test_app_services.py @@ -1,10 +1,13 @@ """The scheduler, integrity, audit, notification, settings and dashboard services.""" +# pylint: disable=no-member # the member exists on the object the fixture builds +# pylint: disable=use-implicit-booleaness-not-comparison # an exact empty value is what is asserted + from __future__ import annotations import ast import json -import subprocess +import subprocess # nosec B404 # the test starts this interpreter with a fixed argument list import sys from collections.abc import Iterator from pathlib import Path @@ -283,7 +286,7 @@ def test_audit_records_are_searched_counted_and_masked(trail: AuditTrail, bus: E PipelineFailed( source="pipeline", subject="nightly failed", - payload={"pipeline": "nightly", "status": "failed", "password": "hunter2"}, + payload={"pipeline": "nightly", "status": "failed", "password": "hunter2"}, # nosec B105 # a made-up value for a stand-in, not a credential ) ) trail.record("manual.note", resource="s3://reports/a.csv", status="ok", actor="ops") @@ -623,7 +626,7 @@ def test_the_dashboard_counts_runs_and_asks_for_attention_after_a_failure( def test_recent_events_are_newest_first_filtered_and_masked( services: AppServices, bus: EventBus ) -> None: - bus.publish(Event(source="test", subject="first", payload={"token": "abc"})) + bus.publish(Event(source="test", subject="first", payload={"token": "abc"})) # nosec B105 # a made-up value for a stand-in, not a credential bus.publish(SystemErrorEvent(source="test", subject="second")) events = services.dashboard.recent_events() assert [event["subject"] for event in events] == ["second", "first"] diff --git a/tests/test_audit_v2.py b/tests/test_audit_v2.py index 9585a34..fcf495b 100644 --- a/tests/test_audit_v2.py +++ b/tests/test_audit_v2.py @@ -1,5 +1,9 @@ """Audit schema v2: the record, the stores, the trail, the v1 import and the actions.""" +# pylint: disable=redefined-outer-name # pytest passes fixtures by matching name +# pylint: disable=too-many-function-args # the call is expected to be refused +# pylint: disable=use-implicit-booleaness-not-comparison # an exact empty value is what is asserted + from __future__ import annotations import dataclasses @@ -486,7 +490,7 @@ def test_the_database_states_its_schema_version(sqlite_store: SQLiteAuditStore) def test_the_database_has_its_indexes(sqlite_store: SQLiteAuditStore) -> None: with closing(sqlite3.connect(sqlite_store.path)) as conn: indexed = { - conn.execute(f"PRAGMA index_info({name})").fetchall()[0][2] + conn.execute(f"PRAGMA index_info({name})").fetchall()[0][2] # nosec B608 # nosemgrep # the name comes from sqlite_master for name in _names(sqlite_store.path, "index") if name.startswith("idx_audit_records_") } diff --git a/tests/test_backends.py b/tests/test_backends.py index 24dc28f..9a7177f 100644 --- a/tests/test_backends.py +++ b/tests/test_backends.py @@ -7,6 +7,8 @@ real cloud backend lives outside CI. """ +# pylint: disable=unused-argument # a fixture is requested for its effect; a stand-in keeps the real signature + from __future__ import annotations import importlib @@ -110,7 +112,7 @@ def set_missing_host_key_policy(self, policy: object) -> None: def connect(self, **options: object) -> None: self.connected_to = (str(options["hostname"]), int(str(options["port"]))) - def open_sftp(self) -> _StubSSH: + def open_sftp(self) -> _StubSSH: # pylint: disable=undefined-variable # a forward reference, read lazily return self def close(self) -> None: diff --git a/tests/test_cli_operations.py b/tests/test_cli_operations.py index 4397dd6..7dfbb54 100644 --- a/tests/test_cli_operations.py +++ b/tests/test_cli_operations.py @@ -1,5 +1,7 @@ """The ``integrity``, ``pipeline`` and ``audit`` subcommands: JSON out, exit codes that mean something.""" +# pylint: disable=line-too-long # an expected value is kept on one line + from __future__ import annotations import json diff --git a/tests/test_config.py b/tests/test_config.py index a709c1a..b59cef0 100644 --- a/tests/test_config.py +++ b/tests/test_config.py @@ -1,5 +1,7 @@ """Tests for automation_file.core.config.""" +# pylint: disable=use-implicit-booleaness-not-comparison # an exact empty value is what is asserted + from __future__ import annotations import time diff --git a/tests/test_cross_backend_storage.py b/tests/test_cross_backend_storage.py index 59443af..a834f1c 100644 --- a/tests/test_cross_backend_storage.py +++ b/tests/test_cross_backend_storage.py @@ -1,5 +1,8 @@ """``copy_between`` on the storage layer: storage URIs, the older spellings, and what fails how.""" +# pylint: disable=protected-access # the tests look at private state on purpose +# pylint: disable=unused-argument # a fixture is requested for its effect; a stand-in keeps the real signature + from __future__ import annotations from collections.abc import Iterator diff --git a/tests/test_events.py b/tests/test_events.py index 1fc258c..bc9e4c2 100644 --- a/tests/test_events.py +++ b/tests/test_events.py @@ -1,5 +1,9 @@ """The event model, the bus, the scopes, and the storage-error bridge.""" +# pylint: disable=redefined-outer-name # pytest passes fixtures by matching name +# pylint: disable=too-many-function-args # the call is expected to be refused +# pylint: disable=use-implicit-booleaness-not-comparison # an exact empty value is what is asserted + from __future__ import annotations import dataclasses diff --git a/tests/test_ftp_ops.py b/tests/test_ftp_ops.py index 5ac8482..72d790b 100644 --- a/tests/test_ftp_ops.py +++ b/tests/test_ftp_ops.py @@ -4,6 +4,9 @@ facade exports, guard clauses, and offline error-path behaviour. """ +# pylint: disable=no-member # the member exists on the object the fixture builds +# pylint: disable=unused-argument # a fixture is requested for its effect; a stand-in keeps the real signature + from __future__ import annotations from pathlib import Path @@ -150,7 +153,7 @@ def connect(self, host: str, port: int, timeout: float | None = None) -> None: def auth(self) -> None: self.steps.append("auth") - def login(self, user: str = "", passwd: str = "") -> None: + def login(self, user: str = "", passwd: str = "") -> None: # nosec B107 # ftplib's own default self.steps.append("login") def prot_p(self) -> None: diff --git a/tests/test_integrity_actions.py b/tests/test_integrity_actions.py index 63fb526..ccd0101 100644 --- a/tests/test_integrity_actions.py +++ b/tests/test_integrity_actions.py @@ -1,5 +1,8 @@ """FA_integrity_* actions: the monitor through the registry, the executor and MCP.""" +# pylint: disable=redefined-outer-name # pytest passes fixtures by matching name +# pylint: disable=unused-argument # a fixture is requested for its effect; a stand-in keeps the real signature + from __future__ import annotations import hashlib diff --git a/tests/test_integrity_legacy.py b/tests/test_integrity_legacy.py index 14063a4..ad763e7 100644 --- a/tests/test_integrity_legacy.py +++ b/tests/test_integrity_legacy.py @@ -6,6 +6,9 @@ change kinds the first monitor did not know. """ +# pylint: disable=redefined-outer-name # pytest passes fixtures by matching name +# pylint: disable=use-implicit-booleaness-not-comparison # an exact empty value is what is asserted + from __future__ import annotations from pathlib import Path diff --git a/tests/test_integrity_monitor.py b/tests/test_integrity_monitor.py index 64b67de..09033ff 100644 --- a/tests/test_integrity_monitor.py +++ b/tests/test_integrity_monitor.py @@ -1,5 +1,8 @@ """IntegrityMonitor: snapshot, baseline, verify, accept, alerts and continuous mode.""" +# pylint: disable=redefined-outer-name # pytest passes fixtures by matching name +# pylint: disable=unused-argument # a fixture is requested for its effect; a stand-in keeps the real signature + from __future__ import annotations import hashlib @@ -487,7 +490,7 @@ def test_a_monitor_refuses_a_weak_algorithm_unless_told_otherwise(tree: Storage) allowed = IntegrityMonitor(TREE, baseline=BASELINE, algorithm="md5", allow_weak=True) entry = allowed.snapshot().get("a.txt") assert entry is not None - assert entry.checksum == hashlib.md5(b"alpha", usedforsecurity=False).hexdigest() + assert entry.checksum == hashlib.md5(b"alpha", usedforsecurity=False).hexdigest() # nosec B324 # nosemgrep # the digest under test, not a security use def test_a_weak_baseline_is_not_verified_without_allow_weak(tree: Storage, bus: EventBus) -> None: diff --git a/tests/test_integrity_object_store.py b/tests/test_integrity_object_store.py index aa72116..94ff431 100644 --- a/tests/test_integrity_object_store.py +++ b/tests/test_integrity_object_store.py @@ -6,6 +6,10 @@ quick pass from a deep one. """ +# pylint: disable=consider-using-with # the handle is closed by the code under test +# pylint: disable=redefined-outer-name # pytest passes fixtures by matching name +# pylint: disable=unused-argument # a fixture is requested for its effect; a stand-in keeps the real signature + from __future__ import annotations import hashlib @@ -69,7 +73,7 @@ def _head(self, key: str) -> FileInfo | None: size=len(item.data), # An HTTP Last-Modified header carries whole seconds; the listing has the stored time. modified_at=item.modified.replace(microsecond=0), - etag=hashlib.md5(item.data, usedforsecurity=False).hexdigest(), + etag=hashlib.md5(item.data, usedforsecurity=False).hexdigest(), # nosec B324 # nosemgrep # the digest under test, not a security use version=item.version, content_type="text/csv" if key.endswith(".csv") else "application/octet-stream", ) @@ -165,7 +169,7 @@ def test_a_snapshot_takes_the_etag_from_the_listing_and_the_rest_from_a_head( entry = snapshot.get("q1.csv") assert entry is not None assert entry.checksum == hashlib.sha256(b"region,total\nEMEA,42\n").hexdigest() - assert entry.etag == hashlib.md5(b"region,total\nEMEA,42\n", usedforsecurity=False).hexdigest() + assert entry.etag == hashlib.md5(b"region,total\nEMEA,42\n", usedforsecurity=False).hexdigest() # nosec B324 # nosemgrep # the digest under test, not a security use assert (entry.version, entry.content_type, entry.backend, entry.mode) == ( "v1", "text/csv", diff --git a/tests/test_integrity_remediation.py b/tests/test_integrity_remediation.py index bba8425..aa29471 100644 --- a/tests/test_integrity_remediation.py +++ b/tests/test_integrity_remediation.py @@ -1,5 +1,9 @@ """Remediation: off by default, quarantine, restore with checksum verification.""" +# pylint: disable=redefined-outer-name # pytest passes fixtures by matching name +# pylint: disable=unused-argument # a fixture is requested for its effect; a stand-in keeps the real signature +# pylint: disable=use-implicit-booleaness-not-comparison # an exact empty value is what is asserted + from __future__ import annotations from collections.abc import Iterator diff --git a/tests/test_integrity_snapshot.py b/tests/test_integrity_snapshot.py index 6e865dc..2b9bf52 100644 --- a/tests/test_integrity_snapshot.py +++ b/tests/test_integrity_snapshot.py @@ -394,7 +394,7 @@ def test_a_weak_algorithm_works_when_explicitly_allowed() -> None: engine = HashEngine("md5", allow_weak=True) assert engine.algorithm == "md5" assert engine.hash_file(storage, "a.txt") == ( - hashlib.md5(b"alpha", usedforsecurity=False).hexdigest() + hashlib.md5(b"alpha", usedforsecurity=False).hexdigest() # nosec B324 # nosemgrep # the digest under test, not a security use ) diff --git a/tests/test_integrity_watch.py b/tests/test_integrity_watch.py index 6756869..98967ae 100644 --- a/tests/test_integrity_watch.py +++ b/tests/test_integrity_watch.py @@ -6,6 +6,10 @@ writes to a watched directory for real and is skipped where no event arrives. """ +# pylint: disable=redefined-outer-name # pytest passes fixtures by matching name +# pylint: disable=unused-argument # a fixture is requested for its effect; a stand-in keeps the real signature +# pylint: disable=use-implicit-booleaness-not-comparison # an exact empty value is what is asserted + from __future__ import annotations import threading diff --git a/tests/test_mcp_pipeline_tools.py b/tests/test_mcp_pipeline_tools.py index 5012384..7003e69 100644 --- a/tests/test_mcp_pipeline_tools.py +++ b/tests/test_mcp_pipeline_tools.py @@ -1,5 +1,7 @@ """The semantic pipeline and reporting tools, and the guarded actions a pipeline runs with.""" +# pylint: disable=use-implicit-booleaness-not-comparison # an exact empty value is what is asserted + from __future__ import annotations import json diff --git a/tests/test_mcp_policy.py b/tests/test_mcp_policy.py index 2012dc2..6a3bdf7 100644 --- a/tests/test_mcp_policy.py +++ b/tests/test_mcp_policy.py @@ -1,5 +1,8 @@ """The permission model of the semantic MCP tools: ``MCPPolicy`` and ``StorageGuard``.""" +# pylint: disable=redefined-outer-name # pytest passes fixtures by matching name +# pylint: disable=use-implicit-booleaness-not-comparison # an exact empty value is what is asserted + from __future__ import annotations import dataclasses diff --git a/tests/test_mcp_semantic_server.py b/tests/test_mcp_semantic_server.py index 2919c7c..a326b8d 100644 --- a/tests/test_mcp_semantic_server.py +++ b/tests/test_mcp_semantic_server.py @@ -1,5 +1,8 @@ """The semantic tools over JSON-RPC, next to the ``FA_*`` bridge, and the server's flags.""" +# pylint: disable=unnecessary-lambda # the lambda is looked up late, when it is called +# pylint: disable=use-implicit-booleaness-not-comparison # an exact empty value is what is asserted + from __future__ import annotations import argparse diff --git a/tests/test_mcp_storage_tools.py b/tests/test_mcp_storage_tools.py index c3982fa..dc67acb 100644 --- a/tests/test_mcp_storage_tools.py +++ b/tests/test_mcp_storage_tools.py @@ -1,5 +1,7 @@ """The semantic tools that work on a directory: ``storage_list``, ``storage_copy``, ``file_search``.""" +# pylint: disable=line-too-long # an expected value is kept on one line + from __future__ import annotations from collections.abc import Iterator diff --git a/tests/test_mcp_tools.py b/tests/test_mcp_tools.py index 9513f2a..c346aca 100644 --- a/tests/test_mcp_tools.py +++ b/tests/test_mcp_tools.py @@ -1,5 +1,7 @@ """The semantic file and storage tools, called without the JSON-RPC layer.""" +# pylint: disable=redefined-outer-name # pytest passes fixtures by matching name + from __future__ import annotations import base64 @@ -519,7 +521,7 @@ def test_file_checksum_defaults_to_sha256() -> None: 24, ) md5 = succeed(kit.call("file_checksum", {"uri": A, "algorithm": "MD5"})) - assert md5["value"] == hashlib.md5(File(A).read(), usedforsecurity=False).hexdigest() + assert md5["value"] == hashlib.md5(File(A).read(), usedforsecurity=False).hexdigest() # nosec B324 # nosemgrep # the digest under test, not a security use failed(kit.call("file_checksum", {"uri": A, "algorithm": "crc-nope"}), "failed") @@ -531,7 +533,7 @@ def test_file_verify_answers_match_or_mismatch_without_failing() -> None: assert (body["match"], body["actual"], body["algorithm"]) == (True, digest, "sha256") wrong = succeed(kit.call("file_verify", {"uri": A, "expected": "00"})) assert (wrong["match"], wrong["expected"]) == (False, "00") - sha1 = hashlib.sha1(File(A).read(), usedforsecurity=False).hexdigest() + sha1 = hashlib.sha1(File(A).read(), usedforsecurity=False).hexdigest() # nosec B324 # nosemgrep # the digest under test, not a security use prefixed = succeed(kit.call("file_verify", {"uri": A, "expected": f"sha1:{sha1}"})) assert (prefixed["match"], prefixed["algorithm"]) == (True, "sha1") failed(kit.call("file_verify", {"uri": A, "expected": "sha256:"}), "invalid_arguments") diff --git a/tests/test_notify.py b/tests/test_notify.py index 1730bce..776eb19 100644 --- a/tests/test_notify.py +++ b/tests/test_notify.py @@ -1,5 +1,7 @@ """Tests for automation_file.notify.""" +# pylint: disable=use-implicit-booleaness-not-comparison # an exact empty value is what is asserted + from __future__ import annotations from collections.abc import Iterator diff --git a/tests/test_notify_router.py b/tests/test_notify_router.py index b87d3d1..6591229 100644 --- a/tests/test_notify_router.py +++ b/tests/test_notify_router.py @@ -1,5 +1,9 @@ """The notification router: routes, deduplication, rate limits, failures, notify_on_failure.""" +# pylint: disable=redefined-outer-name # pytest passes fixtures by matching name +# pylint: disable=unused-argument # a fixture is requested for its effect; a stand-in keeps the real signature +# pylint: disable=use-implicit-booleaness-not-comparison # an exact empty value is what is asserted + from __future__ import annotations import dataclasses diff --git a/tests/test_operational_metrics.py b/tests/test_operational_metrics.py index 2aac2f2..0f48452 100644 --- a/tests/test_operational_metrics.py +++ b/tests/test_operational_metrics.py @@ -1,5 +1,7 @@ """Operational metrics: events, notifications and storage operations.""" +# pylint: disable=redefined-outer-name # pytest passes fixtures by matching name + from __future__ import annotations from collections.abc import Iterator diff --git a/tests/test_optional_dependencies.py b/tests/test_optional_dependencies.py index fb5ac68..f7a77b1 100644 --- a/tests/test_optional_dependencies.py +++ b/tests/test_optional_dependencies.py @@ -9,11 +9,13 @@ optional package sits among the base dependencies. """ +# pylint: disable=redefined-outer-name # pytest passes fixtures by matching name + from __future__ import annotations import json import re -import subprocess +import subprocess # nosec B404 # the test starts this interpreter with a fixed argument list import sys from pathlib import Path diff --git a/tests/test_pipeline_definition.py b/tests/test_pipeline_definition.py index 633e317..8542c9b 100644 --- a/tests/test_pipeline_definition.py +++ b/tests/test_pipeline_definition.py @@ -1,5 +1,8 @@ """Pipeline definitions: validation with paths, the schema, round trips, YAML and JSON files.""" +# pylint: disable=line-too-long # an expected value is kept on one line +# pylint: disable=use-implicit-booleaness-not-comparison # an exact empty value is what is asserted + from __future__ import annotations import copy @@ -792,7 +795,7 @@ def fetch(source: str) -> dict[str, str]: calls.append(source) if len(calls) < 3: raise StorageTransientException("throttled") - return {"path": f"/tmp/{source.rsplit('/', 1)[-1]}"} + return {"path": f"/tmp/{source.rsplit('/', 1)[-1]}"} # nosec B108 # a path that is never opened registry = ActionRegistry() registry.register("T_fetch", fetch) @@ -819,7 +822,7 @@ def fetch(source: str) -> dict[str, str]: assert run.status is RunStatus.SUCCEEDED assert calls == ["s3://in/2026-10-08.csv"] * 3 assert run.tasks["fetch"].attempts == 3 - assert run.tasks["report"].result == "/tmp/2026-10-08.csv for 2026-10-08" + assert run.tasks["report"].result == "/tmp/2026-10-08.csv for 2026-10-08" # nosec B108 # a path that is never opened # ---------------------------------------------------------------------- substitution diff --git a/tests/test_pipeline_events.py b/tests/test_pipeline_events.py index 93c72df..f067929 100644 --- a/tests/test_pipeline_events.py +++ b/tests/test_pipeline_events.py @@ -1,5 +1,7 @@ """What a pipeline run publishes: the events, their order, payload and correlation ID.""" +# pylint: disable=redefined-outer-name # pytest passes fixtures by matching name + from __future__ import annotations import threading diff --git a/tests/test_pipeline_rejected.py b/tests/test_pipeline_rejected.py index 6f107e1..626b825 100644 --- a/tests/test_pipeline_rejected.py +++ b/tests/test_pipeline_rejected.py @@ -1,5 +1,7 @@ """The pipeline runtime: what is rejected before anything runs.""" +# pylint: disable=use-implicit-booleaness-not-comparison # an exact empty value is what is asserted + from __future__ import annotations import re diff --git a/tests/test_pipeline_run.py b/tests/test_pipeline_run.py index 7167f9e..a77c2c3 100644 --- a/tests/test_pipeline_run.py +++ b/tests/test_pipeline_run.py @@ -1,5 +1,8 @@ """The pipeline runtime: ordering, fan-out, statuses, retry, timeout, cancellation, conditions.""" +# pylint: disable=redefined-outer-name # pytest passes fixtures by matching name +# pylint: disable=use-implicit-booleaness-not-comparison # an exact empty value is what is asserted + from __future__ import annotations import threading diff --git a/tests/test_pipeline_store.py b/tests/test_pipeline_store.py index e3d68e8..8e2b992 100644 --- a/tests/test_pipeline_store.py +++ b/tests/test_pipeline_store.py @@ -1,5 +1,9 @@ """Run stores, checkpoints, resume, idempotency and the execution history.""" +# pylint: disable=broad-exception-caught # the test records whatever was raised +# pylint: disable=redefined-outer-name # pytest passes fixtures by matching name +# pylint: disable=use-implicit-booleaness-not-comparison # an exact empty value is what is asserted + from __future__ import annotations import inspect @@ -264,7 +268,7 @@ def write(worker_index: int) -> None: def test_run_store_is_abstract() -> None: with pytest.raises(TypeError): - RunStore() # type: ignore[abstract] + RunStore() # type: ignore[abstract] # pylint: disable=abstract-class-instantiated # the refusal is the test # ---------------------------------------------------------------------- memory diff --git a/tests/test_scheduler_actions.py b/tests/test_scheduler_actions.py index ce4d056..1ab6dde 100644 --- a/tests/test_scheduler_actions.py +++ b/tests/test_scheduler_actions.py @@ -1,5 +1,7 @@ """FA_schedule_* actions, the package's exports and what its modules may import.""" +# pylint: disable=redefined-outer-name # pytest passes fixtures by matching name + from __future__ import annotations import ast diff --git a/tests/test_scheduler_lifecycle.py b/tests/test_scheduler_lifecycle.py index cbaad9b..a0c48e4 100644 --- a/tests/test_scheduler_lifecycle.py +++ b/tests/test_scheduler_lifecycle.py @@ -1,5 +1,8 @@ """The scheduler's own thread: starting, ticking, shutting down, and arming triggers again.""" +# pylint: disable=redefined-outer-name # pytest passes fixtures by matching name +# pylint: disable=unused-argument # a fixture is requested for its effect; a stand-in keeps the real signature + from __future__ import annotations import threading diff --git a/tests/test_scheduler_pipeline.py b/tests/test_scheduler_pipeline.py index 8e5e737..0264946 100644 --- a/tests/test_scheduler_pipeline.py +++ b/tests/test_scheduler_pipeline.py @@ -1,5 +1,7 @@ """Pipelines as scheduler targets: the declared schedule, parameters, dependencies, stopping.""" +# pylint: disable=redefined-outer-name # pytest passes fixtures by matching name + from __future__ import annotations import json diff --git a/tests/test_scheduler_runs.py b/tests/test_scheduler_runs.py index ecdec5e..dfffc08 100644 --- a/tests/test_scheduler_runs.py +++ b/tests/test_scheduler_runs.py @@ -1,5 +1,9 @@ """Run records of the scheduler: the seven states, overlap, timeout, cancellation, history.""" +# pylint: disable=protected-access # the tests look at private state on purpose +# pylint: disable=redefined-outer-name # pytest passes fixtures by matching name +# pylint: disable=use-implicit-booleaness-not-comparison # an exact empty value is what is asserted + from __future__ import annotations import json diff --git a/tests/test_scheduler_triggers.py b/tests/test_scheduler_triggers.py index 9ae3688..fd8dda6 100644 --- a/tests/test_scheduler_triggers.py +++ b/tests/test_scheduler_triggers.py @@ -1,5 +1,9 @@ """The scheduler's triggers: cron with a time zone, file events, events on the bus.""" +# pylint: disable=inconsistent-return-statements # the other branch raises, or no test reaches it +# pylint: disable=redefined-outer-name # pytest passes fixtures by matching name +# pylint: disable=use-implicit-booleaness-not-comparison # an exact empty value is what is asserted + from __future__ import annotations import zoneinfo @@ -429,7 +433,7 @@ def stop(self) -> None: self.alive = False def join(self, timeout: float | None = None) -> None: - self.joined_with = timeout + self.joined_with = timeout # pylint: disable=attribute-defined-outside-init # kept for the assertion def is_alive(self) -> bool: return self.alive diff --git a/tests/test_storage_actions.py b/tests/test_storage_actions.py index d36edf1..6400b8b 100644 --- a/tests/test_storage_actions.py +++ b/tests/test_storage_actions.py @@ -138,7 +138,7 @@ def test_checksum_and_verify() -> None: "algorithm": "sha256", "value": SHA256_HELLO, } - md5 = hashlib.md5(b"hello", usedforsecurity=False).hexdigest() + md5 = hashlib.md5(b"hello", usedforsecurity=False).hexdigest() # nosec B324 # nosemgrep # the digest under test, not a security use assert actions.storage_checksum("memory://scratch/a.txt", "md5")["value"] == md5 assert actions.storage_verify("memory://scratch/a.txt", SHA256_HELLO) is True assert actions.storage_verify("memory://scratch/a.txt", f"md5:{md5}") is True diff --git a/tests/test_storage_azure.py b/tests/test_storage_azure.py index 37933aa..a07872d 100644 --- a/tests/test_storage_azure.py +++ b/tests/test_storage_azure.py @@ -7,6 +7,12 @@ leaves the process. """ +# pylint: disable=protected-access # the tests look at private state on purpose +# pylint: disable=raising-bad-type # a stand-in raises what the test hands it +# pylint: disable=redefined-outer-name # pytest passes fixtures by matching name +# pylint: disable=unidiomatic-typecheck # the exact class is what is asserted +# pylint: disable=use-implicit-booleaness-not-comparison # an exact empty value is what is asserted + from __future__ import annotations import hashlib @@ -74,7 +80,7 @@ def properties(self, name: str) -> SimpleNamespace: name=name, size=len(self.data), last_modified=self.modified, - etag=f'"0x{hashlib.md5(self.data, usedforsecurity=False).hexdigest()[:16].upper()}"', + etag=f'"0x{hashlib.md5(self.data, usedforsecurity=False).hexdigest()[:16].upper()}"', # nosec B324 # nosemgrep # the digest under test, not a security use content_settings=SimpleNamespace(content_type=self.content_type), metadata={}, version_id=None, diff --git a/tests/test_storage_dropbox.py b/tests/test_storage_dropbox.py index 5475f63..0dd6ad5 100644 --- a/tests/test_storage_dropbox.py +++ b/tests/test_storage_dropbox.py @@ -7,6 +7,12 @@ raises the SDK's own exceptions. No request leaves the process. """ +# pylint: disable=arguments-differ # a fixture or a stand-in takes other arguments than the one it replaces +# pylint: disable=protected-access # the tests look at private state on purpose +# pylint: disable=redefined-outer-name # pytest passes fixtures by matching name +# pylint: disable=unidiomatic-typecheck # the exact class is what is asserted +# pylint: disable=use-implicit-booleaness-not-comparison # an exact empty value is what is asserted + from __future__ import annotations import hashlib diff --git a/tests/test_storage_file.py b/tests/test_storage_file.py index 30b3745..a7cda49 100644 --- a/tests/test_storage_file.py +++ b/tests/test_storage_file.py @@ -1,5 +1,8 @@ """File and Storage: the object API over the resolver, including cross-backend transfers.""" +# pylint: disable=redefined-outer-name # pytest passes fixtures by matching name +# pylint: disable=use-implicit-booleaness-not-comparison # an exact empty value is what is asserted + from __future__ import annotations import hashlib @@ -156,13 +159,13 @@ def test_checksum_and_verify(resolver: StorageResolver) -> None: file = File("memory://scratch/a.bin", resolver=resolver) file.write(PAYLOAD) assert file.checksum() == Checksum("sha256", SHA256) - assert file.checksum("md5").value == hashlib.md5(PAYLOAD, usedforsecurity=False).hexdigest() + assert file.checksum("md5").value == hashlib.md5(PAYLOAD, usedforsecurity=False).hexdigest() # nosec B324 # nosemgrep # the digest under test, not a security use assert file.verify(SHA256) is True assert file.verify(SHA256.upper()) is True assert file.verify(f"sha256:{SHA256}") is True assert file.verify(Checksum("sha256", SHA256)) is True assert file.verify("0" * 64) is False - md5 = hashlib.md5(PAYLOAD, usedforsecurity=False).hexdigest() + md5 = hashlib.md5(PAYLOAD, usedforsecurity=False).hexdigest() # nosec B324 # nosemgrep # the digest under test, not a security use assert file.verify(md5, algorithm="md5") is True assert file.verify(f"md5:{md5}") is True assert file.verify(md5) is False diff --git a/tests/test_storage_fsspec.py b/tests/test_storage_fsspec.py index 0f6a358..7d6a7b3 100644 --- a/tests/test_storage_fsspec.py +++ b/tests/test_storage_fsspec.py @@ -5,6 +5,13 @@ an object store does, for the ``directories=False`` mode. """ +# pylint: disable=arguments-differ # a fixture or a stand-in takes other arguments than the one it replaces +# pylint: disable=protected-access # the tests look at private state on purpose +# pylint: disable=redefined-outer-name # pytest passes fixtures by matching name +# pylint: disable=unidiomatic-typecheck # the exact class is what is asserted +# pylint: disable=unused-argument # a fixture is requested for its effect; a stand-in keeps the real signature +# pylint: disable=use-implicit-booleaness-not-comparison # an exact empty value is what is asserted + from __future__ import annotations import inspect diff --git a/tests/test_storage_ftp.py b/tests/test_storage_ftp.py index 6e510ae..3fa954a 100644 --- a/tests/test_storage_ftp.py +++ b/tests/test_storage_ftp.py @@ -7,6 +7,11 @@ ``CWD``, ``SIZE``, ``MDTM`` and ``NLST``. """ +# pylint: disable=protected-access # the tests look at private state on purpose +# pylint: disable=redefined-outer-name # pytest passes fixtures by matching name +# pylint: disable=unidiomatic-typecheck # the exact class is what is asserted +# pylint: disable=unused-argument # a fixture is requested for its effect; a stand-in keeps the real signature + from __future__ import annotations import errno diff --git a/tests/test_storage_gdrive.py b/tests/test_storage_gdrive.py index f63c8d2..78ddac6 100644 --- a/tests/test_storage_gdrive.py +++ b/tests/test_storage_gdrive.py @@ -6,6 +6,12 @@ one, and the last section names what that pins and what it leaves open. """ +# pylint: disable=line-too-long # an expected value is kept on one line +# pylint: disable=protected-access # the tests look at private state on purpose +# pylint: disable=redefined-outer-name # pytest passes fixtures by matching name +# pylint: disable=unidiomatic-typecheck # the exact class is what is asserted +# pylint: disable=use-implicit-booleaness-not-comparison # an exact empty value is what is asserted + from __future__ import annotations import hashlib @@ -109,7 +115,7 @@ def test_stat_reports_what_drive_holds(storage: GoogleDriveStorage, drive: FakeD info = storage.write_bytes("reports/q1.json", b"{}") assert info.path == "reports/q1.json" assert info.size == 2 - assert info.etag == hashlib.md5(b"{}", usedforsecurity=False).hexdigest() + assert info.etag == hashlib.md5(b"{}", usedforsecurity=False).hexdigest() # nosec B324 # nosemgrep # the digest under test, not a security use assert info.version == "1" assert info.content_type == "application/json" assert info.modified_at is not None diff --git a/tests/test_storage_local.py b/tests/test_storage_local.py index 3c07c5d..0cfa20f 100644 --- a/tests/test_storage_local.py +++ b/tests/test_storage_local.py @@ -1,5 +1,8 @@ """LocalStorage: the storage contract plus what is specific to a filesystem.""" +# pylint: disable=arguments-differ # a fixture or a stand-in takes other arguments than the one it replaces +# pylint: disable=redefined-outer-name # pytest passes fixtures by matching name + from __future__ import annotations import os diff --git a/tests/test_storage_observe.py b/tests/test_storage_observe.py index f326d6e..3c91898 100644 --- a/tests/test_storage_observe.py +++ b/tests/test_storage_observe.py @@ -1,5 +1,7 @@ """Storage observers: what each backend operation reports.""" +# pylint: disable=redefined-outer-name # pytest passes fixtures by matching name + from __future__ import annotations from collections.abc import Iterator diff --git a/tests/test_storage_onedrive.py b/tests/test_storage_onedrive.py index 6fd4b24..5e1144e 100644 --- a/tests/test_storage_onedrive.py +++ b/tests/test_storage_onedrive.py @@ -6,6 +6,10 @@ ``driveItem`` documentation: nothing installed describes that API. """ +# pylint: disable=redefined-outer-name # pytest passes fixtures by matching name +# pylint: disable=unidiomatic-typecheck # the exact class is what is asserted +# pylint: disable=use-implicit-booleaness-not-comparison # an exact empty value is what is asserted + # pylint: disable=protected-access # the shared client's session is swapped for the stand-in's from __future__ import annotations diff --git a/tests/test_storage_resolver.py b/tests/test_storage_resolver.py index c893c4c..9069775 100644 --- a/tests/test_storage_resolver.py +++ b/tests/test_storage_resolver.py @@ -1,5 +1,7 @@ """StorageResolver: mounts, scheme factories, and the default table.""" +# pylint: disable=redefined-outer-name # pytest passes fixtures by matching name + from __future__ import annotations import os diff --git a/tests/test_storage_s3.py b/tests/test_storage_s3.py index 04712c3..e51bb7b 100644 --- a/tests/test_storage_s3.py +++ b/tests/test_storage_s3.py @@ -6,6 +6,14 @@ real ``botocore`` exceptions. No request leaves the process. """ +# pylint: disable=protected-access # the tests look at private state on purpose +# pylint: disable=raising-bad-type # a stand-in raises what the test hands it +# pylint: disable=redefined-outer-name # pytest passes fixtures by matching name +# pylint: disable=too-many-locals # one scenario told in order +# pylint: disable=unidiomatic-typecheck # the exact class is what is asserted +# pylint: disable=unused-argument # a fixture is requested for its effect; a stand-in keeps the real signature +# pylint: disable=use-implicit-booleaness-not-comparison # an exact empty value is what is asserted + from __future__ import annotations import hashlib @@ -57,7 +65,7 @@ class _Object: @property def etag(self) -> str: - return f'"{hashlib.md5(self.data, usedforsecurity=False).hexdigest()}"' + return f'"{hashlib.md5(self.data, usedforsecurity=False).hexdigest()}"' # nosec B324 # nosemgrep # the digest under test, not a security use class _Paginator: @@ -212,7 +220,7 @@ def test_stat_reports_what_head_object_returns(storage: S3Storage) -> None: info = storage.write_bytes("reports/q1.json", b"{}") assert info.path == "reports/q1.json" assert info.size == 2 - assert info.etag == hashlib.md5(b"{}", usedforsecurity=False).hexdigest() + assert info.etag == hashlib.md5(b"{}", usedforsecurity=False).hexdigest() # nosec B324 # nosemgrep # the digest under test, not a security use assert info.content_type == "application/json" assert info.version is None assert dict(info.metadata) == {} diff --git a/tests/test_storage_sftp.py b/tests/test_storage_sftp.py index fdeb0b0..ece274c 100644 --- a/tests/test_storage_sftp.py +++ b/tests/test_storage_sftp.py @@ -7,6 +7,11 @@ status into. No connection is opened. """ +# pylint: disable=protected-access # the tests look at private state on purpose +# pylint: disable=redefined-outer-name # pytest passes fixtures by matching name +# pylint: disable=unidiomatic-typecheck # the exact class is what is asserted +# pylint: disable=unused-argument # a fixture is requested for its effect; a stand-in keeps the real signature + from __future__ import annotations import errno diff --git a/tests/test_storage_sftp_loopback.py b/tests/test_storage_sftp_loopback.py index 02832ee..86708bd 100644 --- a/tests/test_storage_sftp_loopback.py +++ b/tests/test_storage_sftp_loopback.py @@ -10,6 +10,13 @@ that the stand-in answers the way paramiko does. """ +# pylint: disable=arguments-differ # a fixture or a stand-in takes other arguments than the one it replaces +# pylint: disable=protected-access # the tests look at private state on purpose +# pylint: disable=redefined-outer-name # pytest passes fixtures by matching name +# pylint: disable=unidiomatic-typecheck # the exact class is what is asserted +# pylint: disable=unused-argument # a fixture is requested for its effect; a stand-in keeps the real signature +# pylint: disable=use-implicit-booleaness-not-comparison # an exact empty value is what is asserted + from __future__ import annotations import contextlib diff --git a/tests/test_storage_smb.py b/tests/test_storage_smb.py index 180d4b5..333606e 100644 --- a/tests/test_storage_smb.py +++ b/tests/test_storage_smb.py @@ -8,6 +8,10 @@ and an NTSTATUS code. The real ``SMBClient`` runs on top of it. """ +# pylint: disable=arguments-differ # a fixture or a stand-in takes other arguments than the one it replaces +# pylint: disable=redefined-outer-name # pytest passes fixtures by matching name +# pylint: disable=unidiomatic-typecheck # the exact class is what is asserted + from __future__ import annotations import errno diff --git a/tests/test_storage_tree.py b/tests/test_storage_tree.py index d572f05..5e792b9 100644 --- a/tests/test_storage_tree.py +++ b/tests/test_storage_tree.py @@ -1,5 +1,7 @@ """copy_tree and sync_tree: directory trees between any two backends.""" +# pylint: disable=redefined-outer-name # pytest passes fixtures by matching name + from __future__ import annotations import os diff --git a/tests/test_storage_types.py b/tests/test_storage_types.py index 6b5ddf2..4f449e5 100644 --- a/tests/test_storage_types.py +++ b/tests/test_storage_types.py @@ -1,5 +1,7 @@ """Checksum, FileInfo and StorageCapabilities value types.""" +# pylint: disable=use-implicit-booleaness-not-comparison # an exact empty value is what is asserted + from __future__ import annotations import dataclasses diff --git a/tests/test_storage_webdav.py b/tests/test_storage_webdav.py index 7f891ab..2a830ad 100644 --- a/tests/test_storage_webdav.py +++ b/tests/test_storage_webdav.py @@ -6,6 +6,12 @@ process. """ +# pylint: disable=arguments-differ # a fixture or a stand-in takes other arguments than the one it replaces +# pylint: disable=protected-access # the tests look at private state on purpose +# pylint: disable=redefined-outer-name # pytest passes fixtures by matching name +# pylint: disable=unidiomatic-typecheck # the exact class is what is asserted +# pylint: disable=use-implicit-booleaness-not-comparison # an exact empty value is what is asserted + from __future__ import annotations import hashlib @@ -20,7 +26,9 @@ from pathlib import Path from typing import Any from urllib.parse import quote, unquote, urlsplit -from xml.sax.saxutils import escape +from xml.sax.saxutils import ( + escape, # nosec B406 # nosemgrep # escapes what the fake server writes; parses nothing +) import pytest import requests @@ -63,7 +71,7 @@ class _Resource: @property def etag(self) -> str: - return f'"{hashlib.md5(self.data, usedforsecurity=False).hexdigest()}"' + return f'"{hashlib.md5(self.data, usedforsecurity=False).hexdigest()}"' # nosec B324 # nosemgrep # the digest under test, not a security use class _Response: @@ -288,7 +296,7 @@ def test_stat_reports_what_propfind_returns(storage: WebDAVStorage, server: Fake assert info.size == 2 assert info.modified_at == stored.modified assert info.modified_at.utcoffset() == timedelta(0) - assert info.etag == hashlib.md5(b"{}", usedforsecurity=False).hexdigest() + assert info.etag == hashlib.md5(b"{}", usedforsecurity=False).hexdigest() # nosec B324 # nosemgrep # the digest under test, not a security use assert info.content_type == "application/json" assert info.version is None folder = storage.stat("reports") diff --git a/tests/test_ui_pages.py b/tests/test_ui_pages.py index 21d24bb..751affe 100644 --- a/tests/test_ui_pages.py +++ b/tests/test_ui_pages.py @@ -1,5 +1,9 @@ """The pages of the main window, fed by their services through a pool that runs at once.""" +# pylint: disable=protected-access # the tests look at private state on purpose +# pylint: disable=use-implicit-booleaness-not-comparison # an exact empty value is what is asserted +# pylint: disable=wrong-import-position # imports follow pytest.importorskip + from __future__ import annotations import ast @@ -336,7 +340,7 @@ def test_the_dashboard_renders_the_summary_of_its_service( draft.add_task("T_fail", "only") run_id = services.pipelines.start(draft)["run_id"] assert services.pipelines.wait(run_id, WAIT) - bus.publish(SystemErrorEvent(source="test", subject="disk full", payload={"token": "abc"})) + bus.publish(SystemErrorEvent(source="test", subject="disk full", payload={"token": "abc"})) # nosec B105 # a made-up value for a stand-in, not a credential page.refresh() assert page._headline.text() == "Needs attention" assert "1 of the last 1 pipeline runs failed" in page._reasons.text() @@ -622,7 +626,7 @@ def test_the_audit_page_configures_searches_and_shows_a_record( PipelineFailed( source="pipeline", subject="nightly failed", - payload={"pipeline": "nightly", "status": "failed", "password": "hunter2"}, + payload={"pipeline": "nightly", "status": "failed", "password": "hunter2"}, # nosec B105 # a made-up value for a stand-in, not a credential ) ) bus.publish(SystemErrorEvent(source="system", subject="disk full")) diff --git a/tests/test_ui_pipeline_editor.py b/tests/test_ui_pipeline_editor.py index 95a0f9f..c8de231 100644 --- a/tests/test_ui_pipeline_editor.py +++ b/tests/test_ui_pipeline_editor.py @@ -1,5 +1,11 @@ """The pipeline editor: canvas, task form and page, as thin views over a draft.""" +# pylint: disable=protected-access # the tests look at private state on purpose +# pylint: disable=too-many-locals # one scenario told in order +# pylint: disable=unused-argument # a fixture is requested for its effect; a stand-in keeps the real signature +# pylint: disable=use-implicit-booleaness-not-comparison # an exact empty value is what is asserted +# pylint: disable=wrong-import-position # imports follow pytest.importorskip + from __future__ import annotations import itertools diff --git a/tests/test_ui_smoke.py b/tests/test_ui_smoke.py index e01b994..3bcc342 100644 --- a/tests/test_ui_smoke.py +++ b/tests/test_ui_smoke.py @@ -6,6 +6,9 @@ their services is in ``test_ui_pages.py`` and ``test_ui_pipeline_editor.py``. """ +# pylint: disable=protected-access # the tests look at private state on purpose +# pylint: disable=unnecessary-lambda # the lambda is looked up late, when it is called + from __future__ import annotations import os diff --git a/tests/test_web_ui_app_layer.py b/tests/test_web_ui_app_layer.py index 73f32c0..4ec2af6 100644 --- a/tests/test_web_ui_app_layer.py +++ b/tests/test_web_ui_app_layer.py @@ -1,4 +1,6 @@ """The Web UI renders its fragments from the application layer, escaped and masked.""" + +# pylint: disable=consider-using-with # the handle is closed by the code under test # pylint: disable=cyclic-import from __future__ import annotations @@ -185,7 +187,7 @@ def test_secrets_are_masked_before_they_are_rendered( PipelineFailed( source="pipeline", subject="fetch https://user:hunter2@example.com/x failed", - payload={"error": "Bearer abc.def-123 was refused", "token": "t0ps3cret"}, + payload={"error": "Bearer abc.def-123 was refused", "token": "t0ps3cret"}, # nosec B105 # a made-up value for a stand-in, not a credential ) ) for path in ("/ui/events", "/ui/audit", "/ui/health"): From 5bff680359d7b6b6352185f48ca8277f535eb252 Mon Sep 17 00:00:00 2001 From: JeffreyChen Date: Thu, 8 Oct 2026 16:31:27 +0800 Subject: [PATCH 52/59] chore: shorten the scanner markers and name the S3 test server that is used --- CLAUDE.md | 2 +- automation_file/app/storage_service.py | 2 +- docs/source/Eng/usage/integration_tests.rst | 2 +- docs/source/Zh-CN/usage/integration_tests.rst | 2 +- docs/source/Zh-TW/usage/integration_tests.rst | 2 +- tests/drive_stand_in.py | 8 ++++++-- tests/graph_stand_in.py | 6 +++--- tests/integration/test_s3_minio.py | 2 +- tests/test_app_masking.py | 14 +++++++++----- tests/test_app_pipelines.py | 5 ++++- tests/test_app_services.py | 11 ++++++++--- tests/test_audit_v2.py | 5 ++++- tests/test_cli_operations.py | 2 +- tests/test_integrity_monitor.py | 6 +++++- tests/test_integrity_object_store.py | 10 +++++++--- tests/test_integrity_snapshot.py | 5 ++++- tests/test_mcp_storage_tools.py | 2 +- tests/test_mcp_tools.py | 8 ++++++-- tests/test_optional_dependencies.py | 3 ++- tests/test_storage_actions.py | 5 ++++- tests/test_storage_azure.py | 6 +++++- tests/test_storage_file.py | 8 ++++++-- tests/test_storage_gdrive.py | 6 +++++- tests/test_storage_onedrive.py | 16 +++++++++------- tests/test_storage_s3.py | 8 ++++++-- tests/test_storage_webdav.py | 15 ++++++++++----- tests/test_ui_pages.py | 8 ++++++-- tests/test_web_ui_app_layer.py | 5 ++++- 28 files changed, 121 insertions(+), 53 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 406c4ed..9ee92f3 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -131,7 +131,7 @@ python -m automation_file --help - Unit tests live under `tests/` (pytest). Fixtures in `tests/conftest.py` (`sample_file`, `sample_dir`). - Tests cover every module in `core/`, `local/`, `remote/url_validator`, `project/`, `server/`, `utils/`, plus a facade smoke test, retry/quota/safe_paths, HTTP+TCP auth, and optional-backend registration. - Google Drive / HTTP-download / S3 / Azure / Dropbox / SFTP code paths that require real credentials or network access are **not** exercised in CI — only their URL-validation, auth, and guard-clause behaviour are. -- `tests/integration/` runs the storage contract suite against real services (MinIO, Azurite, OpenSSH, FTP, WebDAV, Samba). Each module is skipped unless its `FA_IT_*` variables are set; `tests/integration/start_service.sh ` starts the container and exports them, and `.github/workflows/integration.yml` runs one service per job with `FA_IT_REQUIRED=1`, which turns a skip into a failure. The same workflow runs the unit tests on Linux and macOS. It does not gate publishing. A backend whose service can run in a container gets a module there, an entry in the script and one in the workflow matrix. +- `tests/integration/` runs the storage contract suite against real services (S3Mock, Azurite, OpenSSH, FTP, WebDAV, Samba). Each module is skipped unless its `FA_IT_*` variables are set; `tests/integration/start_service.sh ` starts the container and exports them, and `.github/workflows/integration.yml` runs one service per job with `FA_IT_REQUIRED=1`, which turns a skip into a failure. The same workflow runs the unit tests on Linux and macOS. It does not gate publishing. A backend whose service can run in a container gets a module there, an entry in the script and one in the workflow matrix. - Run all tests before submitting changes: `python -m pytest tests/ -v`. ## Conventions diff --git a/automation_file/app/storage_service.py b/automation_file/app/storage_service.py index 28db440..aea0295 100644 --- a/automation_file/app/storage_service.py +++ b/automation_file/app/storage_service.py @@ -202,7 +202,7 @@ def is_installed(module: str | None) -> bool: def _client_ready(client: _Client) -> bool: """Ask the shared client whether it has been initialised. Never opens a connection.""" try: - # nosemgrep # the module name is one of this package's own client modules, from a fixed table + # nosemgrep # the module name comes from a fixed table of this package's clients instance = getattr(importlib.import_module(client.module), client.attribute) probe: Callable[[], Any] = getattr(instance, client.probe) probe() diff --git a/docs/source/Eng/usage/integration_tests.rst b/docs/source/Eng/usage/integration_tests.rst index 158550c..3ac536b 100644 --- a/docs/source/Eng/usage/integration_tests.rst +++ b/docs/source/Eng/usage/integration_tests.rst @@ -3,7 +3,7 @@ Integration tests The unit tests give every storage backend a stand-in for its service. The integration tests run the same contract suite (``tests/storage_contract.py``, 88 -cases) against a real one: MinIO for S3, Azurite for Azure Blob, an OpenSSH server +cases) against a real one: S3Mock for S3, Azurite for Azure Blob, an OpenSSH server for SFTP, and an FTP, a WebDAV and a Samba server. They live in ``tests/integration/``. diff --git a/docs/source/Zh-CN/usage/integration_tests.rst b/docs/source/Zh-CN/usage/integration_tests.rst index e4fe410..cd00af1 100644 --- a/docs/source/Zh-CN/usage/integration_tests.rst +++ b/docs/source/Zh-CN/usage/integration_tests.rst @@ -2,7 +2,7 @@ ======== 单元测试为每个存储后端准备了服务的替身。集成测试则把同一套契约测试 -（``tests/storage_contract.py``，88 个用例）拿去对真正的服务运行：S3 用 MinIO、 +（``tests/storage_contract.py``，88 个用例）拿去对真正的服务运行：S3 用 S3Mock、 Azure Blob 用 Azurite、SFTP 用 OpenSSH 服务器，另外还有 FTP、WebDAV 与 Samba 服务器。 这些测试放在 ``tests/integration/``。 diff --git a/docs/source/Zh-TW/usage/integration_tests.rst b/docs/source/Zh-TW/usage/integration_tests.rst index 7a05353..95ef0ef 100644 --- a/docs/source/Zh-TW/usage/integration_tests.rst +++ b/docs/source/Zh-TW/usage/integration_tests.rst @@ -2,7 +2,7 @@ ======== 單元測試為每個儲存後端準備了服務的替身。整合測試則把同一套契約測試 -（``tests/storage_contract.py``，88 個案例）拿去對真正的服務執行：S3 用 MinIO、 +（``tests/storage_contract.py``，88 個案例）拿去對真正的服務執行：S3 用 S3Mock、 Azure Blob 用 Azurite、SFTP 用 OpenSSH 伺服器，另外還有 FTP、WebDAV 與 Samba 伺服器。 這些測試放在 ``tests/integration/``。 diff --git a/tests/drive_stand_in.py b/tests/drive_stand_in.py index 6819039..4ec05fb 100644 --- a/tests/drive_stand_in.py +++ b/tests/drive_stand_in.py @@ -19,6 +19,10 @@ says so. """ +# The nosec / nosemgrep markers below sit on made-up values and on digests that are the thing +# under test; none is a credential or a security use of a hash. +# pylint: disable=line-too-long # a marker has to follow the value it is about + # pylint: disable=line-too-long # an expected value is kept on one line # pylint: disable=raising-bad-type # a stand-in raises what the test hands it # pylint: disable=too-many-positional-arguments # a stand-in keeps the real signature @@ -121,8 +125,8 @@ def resource(self, *, digests: bool = True) -> dict[str, Any]: if self.data is not None: resource["size"] = str(len(self.data)) if digests: - resource["md5Checksum"] = hashlib.md5(self.data, usedforsecurity=False).hexdigest() # nosec B324 # nosemgrep # the digest under test, not a security use - resource["sha1Checksum"] = hashlib.sha1( # nosec B324 # nosemgrep # the digest under test, not a security use + resource["md5Checksum"] = hashlib.md5(self.data, usedforsecurity=False).hexdigest() # nosec B324 # nosemgrep + resource["sha1Checksum"] = hashlib.sha1( # nosec B324 # nosemgrep self.data, usedforsecurity=False ).hexdigest() resource["sha256Checksum"] = hashlib.sha256(self.data).hexdigest() diff --git a/tests/graph_stand_in.py b/tests/graph_stand_in.py index e0ecf9f..52951fb 100644 --- a/tests/graph_stand_in.py +++ b/tests/graph_stand_in.py @@ -41,7 +41,7 @@ DOWNLOAD_ORIGIN = "https://download.onedrive.invalid" # Not credentials: markers the stand-in hands out and the tests look for in messages. FAKE_TOKEN = "fake-token" # nosec B105 -URL_SECRET = "tempauth=fake-url-secret" # nosec B105 +URL_SIGNATURE = "tempauth=fake-url-secret" # nosec B105 ROOT_ID = "ROOT" DRIVE_ID = "fake-drive" LIST_PAGE = 2 @@ -356,7 +356,7 @@ def _on_content(self, request: Any, path: str, params: dict[str, str], body: byt data = self._existing(path).data if data is None: raise _Failure(400) - ticket = f"{DOWNLOAD_ORIGIN}/content/{next(self._ids)}?{URL_SECRET}" + ticket = f"{DOWNLOAD_ORIGIN}/content/{next(self._ids)}?{URL_SIGNATURE}" self._downloads[ticket] = data return graph_answer(request, 302, location=ticket) @@ -370,7 +370,7 @@ def _on_session(self, request: Any, path: str, params: dict[str, str], body: byt existing = self.find(path) if existing is not None and existing.data is None: raise _Failure(409) - upload_url = f"{UPLOAD_ORIGIN}/session/{next(self._ids)}?{URL_SECRET}" + upload_url = f"{UPLOAD_ORIGIN}/session/{next(self._ids)}?{URL_SIGNATURE}" self.sessions[upload_url] = (path, bytearray()) return graph_answer(request, 200, {"uploadUrl": upload_url, "nextExpectedRanges": ["0-"]}) diff --git a/tests/integration/test_s3_minio.py b/tests/integration/test_s3_minio.py index 3842190..4d62b49 100644 --- a/tests/integration/test_s3_minio.py +++ b/tests/integration/test_s3_minio.py @@ -1,4 +1,4 @@ -"""S3Storage against an S3-compatible service (MinIO in CI). +"""S3Storage against an S3-compatible service (S3Mock in CI). Environment: ``FA_IT_S3_ENDPOINT`` (for example ``http://127.0.0.1:9000``), ``FA_IT_S3_ACCESS_KEY``, ``FA_IT_S3_SECRET_KEY`` and optionally ``FA_IT_S3_REGION``. diff --git a/tests/test_app_masking.py b/tests/test_app_masking.py index 33c9388..47abe65 100644 --- a/tests/test_app_masking.py +++ b/tests/test_app_masking.py @@ -1,5 +1,9 @@ """The application layer keeps secrets out of what it returns, and reads form text.""" +# The nosec / nosemgrep markers below sit on made-up values and on digests that are the thing +# under test; none is a credential or a security use of a hash. +# pylint: disable=line-too-long # a marker has to follow the value it is about + # pylint: disable=keyword-arg-before-vararg # a stand-in keeps the real signature # pylint: disable=unused-argument # a fixture is requested for its effect; a stand-in keeps the real signature @@ -62,13 +66,13 @@ def test_url_names_are_told_from_storage_uris() -> None: def test_a_secret_value_is_replaced_whatever_its_type() -> None: masked = mask_secrets( - {"password": "hunter2", "token": {"value": "abc"}, "api_key": 12, "name": "ops"} # nosec B105 # a made-up value for a stand-in, not a credential + {"password": "hunter2", "token": {"value": "abc"}, "api_key": 12, "name": "ops"} # nosec B105 ) assert masked == {"password": MASK, "token": MASK, "api_key": MASK, "name": "ops"} def test_an_empty_secret_stays_empty_so_a_view_can_tell_it_is_unset() -> None: - assert mask_secrets({"password": "", "token": None}) == {"password": "", "token": None} # nosec B105 # a made-up value for a stand-in, not a credential + assert mask_secrets({"password": "", "token": None}) == {"password": "", "token": None} # nosec B105 def test_a_webhook_url_keeps_only_its_host() -> None: @@ -83,7 +87,7 @@ def test_a_url_field_without_an_http_url_is_masked_whole() -> None: def test_nested_values_are_walked_and_tuples_become_lists() -> None: - masked = mask_secrets({"sinks": ({"name": "a", "password": "x"}, {"name": "b"})}) # nosec B105 # a made-up value for a stand-in, not a credential + masked = mask_secrets({"sinks": ({"name": "a", "password": "x"}, {"name": "b"})}) # nosec B105 assert masked == {"sinks": [{"name": "a", "password": MASK}, {"name": "b"}]} @@ -101,9 +105,9 @@ def test_a_storage_uri_is_left_alone() -> None: def test_the_input_is_not_changed() -> None: - original = {"password": "hunter2", "items": [{"token": "t"}]} # nosec B105 # a made-up value for a stand-in, not a credential + original = {"password": "hunter2", "items": [{"token": "t"}]} # nosec B105 mask_secrets(original) - assert original == {"password": "hunter2", "items": [{"token": "t"}]} # nosec B105 # a made-up value for a stand-in, not a credential + assert original == {"password": "hunter2", "items": [{"token": "t"}]} # nosec B105 def test_values_that_are_not_containers_pass_through() -> None: diff --git a/tests/test_app_pipelines.py b/tests/test_app_pipelines.py index 2d5f9ae..e36c545 100644 --- a/tests/test_app_pipelines.py +++ b/tests/test_app_pipelines.py @@ -1,5 +1,8 @@ """The Pipelines service of the application layer, on a private store, registry and bus.""" +# The nosec / nosemgrep markers below sit on made-up values and on digests that are the thing +# under test; none is a credential or a security use of a hash. + from __future__ import annotations import json @@ -274,7 +277,7 @@ def test_a_run_the_store_says_is_running_counts_as_running(bus: EventBus) -> Non def test_secrets_in_the_parameters_are_masked_in_every_view(service: PipelineService) -> None: draft = _draft(("a", "T_echo", {"value": "${params.word}"})) - params = {"word": "hi", "password": "hunter2"} # nosec B105 # a made-up value for a stand-in, not a credential + params = {"word": "hi", "password": "hunter2"} # nosec B105 started = service.start(draft, params) assert started["params"] == {"word": "hi", "password": MASK} done = _finished(service, started["run_id"]) diff --git a/tests/test_app_services.py b/tests/test_app_services.py index e1f60d7..cf85218 100644 --- a/tests/test_app_services.py +++ b/tests/test_app_services.py @@ -1,5 +1,9 @@ """The scheduler, integrity, audit, notification, settings and dashboard services.""" +# The nosec / nosemgrep markers below sit on made-up values and on digests that are the thing +# under test; none is a credential or a security use of a hash. +# pylint: disable=line-too-long # a marker has to follow the value it is about + # pylint: disable=no-member # the member exists on the object the fixture builds # pylint: disable=use-implicit-booleaness-not-comparison # an exact empty value is what is asserted @@ -286,7 +290,7 @@ def test_audit_records_are_searched_counted_and_masked(trail: AuditTrail, bus: E PipelineFailed( source="pipeline", subject="nightly failed", - payload={"pipeline": "nightly", "status": "failed", "password": "hunter2"}, # nosec B105 # a made-up value for a stand-in, not a credential + payload={"pipeline": "nightly", "status": "failed", "password": "hunter2"}, # nosec B105 ) ) trail.record("manual.note", resource="s3://reports/a.csv", status="ok", actor="ops") @@ -626,7 +630,7 @@ def test_the_dashboard_counts_runs_and_asks_for_attention_after_a_failure( def test_recent_events_are_newest_first_filtered_and_masked( services: AppServices, bus: EventBus ) -> None: - bus.publish(Event(source="test", subject="first", payload={"token": "abc"})) # nosec B105 # a made-up value for a stand-in, not a credential + bus.publish(Event(source="test", subject="first", payload={"token": "abc"})) # nosec B105 bus.publish(SystemErrorEvent(source="test", subject="second")) events = services.dashboard.recent_events() assert [event["subject"] for event in events] == ["second", "first"] @@ -778,7 +782,8 @@ def test_importing_and_using_the_layer_loads_no_gui_toolkit() -> None: "services.settings.extras()\n" "print('PySide6' in sys.modules)\n" ) - result = subprocess.run( # nosec B603 - fixed argv: this interpreter and the script above + # nosemgrep # a fixed argument list: this interpreter and a script of this test + result = subprocess.run( # nosec B603 [sys.executable, "-c", probe], cwd=REPO_ROOT, capture_output=True, diff --git a/tests/test_audit_v2.py b/tests/test_audit_v2.py index fcf495b..37c898b 100644 --- a/tests/test_audit_v2.py +++ b/tests/test_audit_v2.py @@ -1,5 +1,8 @@ """Audit schema v2: the record, the stores, the trail, the v1 import and the actions.""" +# The nosec / nosemgrep markers below sit on made-up values and on digests that are the thing +# under test; none is a credential or a security use of a hash. + # pylint: disable=redefined-outer-name # pytest passes fixtures by matching name # pylint: disable=too-many-function-args # the call is expected to be refused # pylint: disable=use-implicit-booleaness-not-comparison # an exact empty value is what is asserted @@ -490,7 +493,7 @@ def test_the_database_states_its_schema_version(sqlite_store: SQLiteAuditStore) def test_the_database_has_its_indexes(sqlite_store: SQLiteAuditStore) -> None: with closing(sqlite3.connect(sqlite_store.path)) as conn: indexed = { - conn.execute(f"PRAGMA index_info({name})").fetchall()[0][2] # nosec B608 # nosemgrep # the name comes from sqlite_master + conn.execute(f"PRAGMA index_info({name})").fetchall()[0][2] # nosec B608 # nosemgrep for name in _names(sqlite_store.path, "index") if name.startswith("idx_audit_records_") } diff --git a/tests/test_cli_operations.py b/tests/test_cli_operations.py index 7dfbb54..7d4068c 100644 --- a/tests/test_cli_operations.py +++ b/tests/test_cli_operations.py @@ -1,4 +1,4 @@ -"""The ``integrity``, ``pipeline`` and ``audit`` subcommands: JSON out, exit codes that mean something.""" +"""The ``integrity``, ``pipeline`` and ``audit`` subcommands: JSON out, meaningful exit codes.""" # pylint: disable=line-too-long # an expected value is kept on one line diff --git a/tests/test_integrity_monitor.py b/tests/test_integrity_monitor.py index 09033ff..551fbf0 100644 --- a/tests/test_integrity_monitor.py +++ b/tests/test_integrity_monitor.py @@ -1,5 +1,9 @@ """IntegrityMonitor: snapshot, baseline, verify, accept, alerts and continuous mode.""" +# The nosec / nosemgrep markers below sit on made-up values and on digests that are the thing +# under test; none is a credential or a security use of a hash. +# pylint: disable=line-too-long # a marker has to follow the value it is about + # pylint: disable=redefined-outer-name # pytest passes fixtures by matching name # pylint: disable=unused-argument # a fixture is requested for its effect; a stand-in keeps the real signature @@ -490,7 +494,7 @@ def test_a_monitor_refuses_a_weak_algorithm_unless_told_otherwise(tree: Storage) allowed = IntegrityMonitor(TREE, baseline=BASELINE, algorithm="md5", allow_weak=True) entry = allowed.snapshot().get("a.txt") assert entry is not None - assert entry.checksum == hashlib.md5(b"alpha", usedforsecurity=False).hexdigest() # nosec B324 # nosemgrep # the digest under test, not a security use + assert entry.checksum == hashlib.md5(b"alpha", usedforsecurity=False).hexdigest() # nosec B324 # nosemgrep def test_a_weak_baseline_is_not_verified_without_allow_weak(tree: Storage, bus: EventBus) -> None: diff --git a/tests/test_integrity_object_store.py b/tests/test_integrity_object_store.py index 94ff431..4673b1b 100644 --- a/tests/test_integrity_object_store.py +++ b/tests/test_integrity_object_store.py @@ -6,6 +6,10 @@ quick pass from a deep one. """ +# The nosec / nosemgrep markers below sit on made-up values and on digests that are the thing +# under test; none is a credential or a security use of a hash. +# pylint: disable=line-too-long # a marker has to follow the value it is about + # pylint: disable=consider-using-with # the handle is closed by the code under test # pylint: disable=redefined-outer-name # pytest passes fixtures by matching name # pylint: disable=unused-argument # a fixture is requested for its effect; a stand-in keeps the real signature @@ -73,7 +77,7 @@ def _head(self, key: str) -> FileInfo | None: size=len(item.data), # An HTTP Last-Modified header carries whole seconds; the listing has the stored time. modified_at=item.modified.replace(microsecond=0), - etag=hashlib.md5(item.data, usedforsecurity=False).hexdigest(), # nosec B324 # nosemgrep # the digest under test, not a security use + etag=hashlib.md5(item.data, usedforsecurity=False).hexdigest(), # nosec B324 # nosemgrep version=item.version, content_type="text/csv" if key.endswith(".csv") else "application/octet-stream", ) @@ -95,7 +99,7 @@ def _scan(self, key_prefix: str, *, shallow: bool) -> Iterable[FileInfo]: path=key, size=len(item.data), modified_at=item.modified, - etag=hashlib.md5(item.data, usedforsecurity=False).hexdigest(), + etag=hashlib.md5(item.data, usedforsecurity=False).hexdigest(), # nosec B324 # nosemgrep ) ) if not shallow: @@ -169,7 +173,7 @@ def test_a_snapshot_takes_the_etag_from_the_listing_and_the_rest_from_a_head( entry = snapshot.get("q1.csv") assert entry is not None assert entry.checksum == hashlib.sha256(b"region,total\nEMEA,42\n").hexdigest() - assert entry.etag == hashlib.md5(b"region,total\nEMEA,42\n", usedforsecurity=False).hexdigest() # nosec B324 # nosemgrep # the digest under test, not a security use + assert entry.etag == hashlib.md5(b"region,total\nEMEA,42\n", usedforsecurity=False).hexdigest() # nosec B324 # nosemgrep assert (entry.version, entry.content_type, entry.backend, entry.mode) == ( "v1", "text/csv", diff --git a/tests/test_integrity_snapshot.py b/tests/test_integrity_snapshot.py index 2b9bf52..c024d42 100644 --- a/tests/test_integrity_snapshot.py +++ b/tests/test_integrity_snapshot.py @@ -1,5 +1,8 @@ """Snapshots, the manifest schema and the hash engine.""" +# The nosec / nosemgrep markers below sit on made-up values and on digests that are the thing +# under test; none is a credential or a security use of a hash. + from __future__ import annotations import dataclasses @@ -394,7 +397,7 @@ def test_a_weak_algorithm_works_when_explicitly_allowed() -> None: engine = HashEngine("md5", allow_weak=True) assert engine.algorithm == "md5" assert engine.hash_file(storage, "a.txt") == ( - hashlib.md5(b"alpha", usedforsecurity=False).hexdigest() # nosec B324 # nosemgrep # the digest under test, not a security use + hashlib.md5(b"alpha", usedforsecurity=False).hexdigest() # nosec B324 # nosemgrep ) diff --git a/tests/test_mcp_storage_tools.py b/tests/test_mcp_storage_tools.py index dc67acb..e18c5d4 100644 --- a/tests/test_mcp_storage_tools.py +++ b/tests/test_mcp_storage_tools.py @@ -1,4 +1,4 @@ -"""The semantic tools that work on a directory: ``storage_list``, ``storage_copy``, ``file_search``.""" +"""The semantic tools for a directory: ``storage_list``, ``storage_copy``, ``file_search``.""" # pylint: disable=line-too-long # an expected value is kept on one line diff --git a/tests/test_mcp_tools.py b/tests/test_mcp_tools.py index c346aca..5578078 100644 --- a/tests/test_mcp_tools.py +++ b/tests/test_mcp_tools.py @@ -1,5 +1,9 @@ """The semantic file and storage tools, called without the JSON-RPC layer.""" +# The nosec / nosemgrep markers below sit on made-up values and on digests that are the thing +# under test; none is a credential or a security use of a hash. +# pylint: disable=line-too-long # a marker has to follow the value it is about + # pylint: disable=redefined-outer-name # pytest passes fixtures by matching name from __future__ import annotations @@ -521,7 +525,7 @@ def test_file_checksum_defaults_to_sha256() -> None: 24, ) md5 = succeed(kit.call("file_checksum", {"uri": A, "algorithm": "MD5"})) - assert md5["value"] == hashlib.md5(File(A).read(), usedforsecurity=False).hexdigest() # nosec B324 # nosemgrep # the digest under test, not a security use + assert md5["value"] == hashlib.md5(File(A).read(), usedforsecurity=False).hexdigest() # nosec B324 # nosemgrep failed(kit.call("file_checksum", {"uri": A, "algorithm": "crc-nope"}), "failed") @@ -533,7 +537,7 @@ def test_file_verify_answers_match_or_mismatch_without_failing() -> None: assert (body["match"], body["actual"], body["algorithm"]) == (True, digest, "sha256") wrong = succeed(kit.call("file_verify", {"uri": A, "expected": "00"})) assert (wrong["match"], wrong["expected"]) == (False, "00") - sha1 = hashlib.sha1(File(A).read(), usedforsecurity=False).hexdigest() # nosec B324 # nosemgrep # the digest under test, not a security use + sha1 = hashlib.sha1(File(A).read(), usedforsecurity=False).hexdigest() # nosec B324 # nosemgrep prefixed = succeed(kit.call("file_verify", {"uri": A, "expected": f"sha1:{sha1}"})) assert (prefixed["match"], prefixed["algorithm"]) == (True, "sha1") failed(kit.call("file_verify", {"uri": A, "expected": "sha256:"}), "invalid_arguments") diff --git a/tests/test_optional_dependencies.py b/tests/test_optional_dependencies.py index f7a77b1..9b311ef 100644 --- a/tests/test_optional_dependencies.py +++ b/tests/test_optional_dependencies.py @@ -123,7 +123,8 @@ def client(backend): def _probe(blocked: tuple[str, ...]) -> dict: - result = subprocess.run( # nosec B603 - fixed argv: this interpreter and a script defined above + # nosemgrep # a fixed argument list: this interpreter and a script of this test + result = subprocess.run( # nosec B603 [sys.executable, "-c", _PROBE, json.dumps(blocked), json.dumps(OPTIONAL_ROOTS)], cwd=REPO_ROOT, capture_output=True, diff --git a/tests/test_storage_actions.py b/tests/test_storage_actions.py index 6400b8b..71542a3 100644 --- a/tests/test_storage_actions.py +++ b/tests/test_storage_actions.py @@ -1,5 +1,8 @@ """FA_storage_* actions: the storage layer through the registry, the executor and MCP.""" +# The nosec / nosemgrep markers below sit on made-up values and on digests that are the thing +# under test; none is a credential or a security use of a hash. + from __future__ import annotations import hashlib @@ -138,7 +141,7 @@ def test_checksum_and_verify() -> None: "algorithm": "sha256", "value": SHA256_HELLO, } - md5 = hashlib.md5(b"hello", usedforsecurity=False).hexdigest() # nosec B324 # nosemgrep # the digest under test, not a security use + md5 = hashlib.md5(b"hello", usedforsecurity=False).hexdigest() # nosec B324 # nosemgrep assert actions.storage_checksum("memory://scratch/a.txt", "md5")["value"] == md5 assert actions.storage_verify("memory://scratch/a.txt", SHA256_HELLO) is True assert actions.storage_verify("memory://scratch/a.txt", f"md5:{md5}") is True diff --git a/tests/test_storage_azure.py b/tests/test_storage_azure.py index a07872d..9c8b794 100644 --- a/tests/test_storage_azure.py +++ b/tests/test_storage_azure.py @@ -7,6 +7,10 @@ leaves the process. """ +# The nosec / nosemgrep markers below sit on made-up values and on digests that are the thing +# under test; none is a credential or a security use of a hash. +# pylint: disable=line-too-long # a marker has to follow the value it is about + # pylint: disable=protected-access # the tests look at private state on purpose # pylint: disable=raising-bad-type # a stand-in raises what the test hands it # pylint: disable=redefined-outer-name # pytest passes fixtures by matching name @@ -80,7 +84,7 @@ def properties(self, name: str) -> SimpleNamespace: name=name, size=len(self.data), last_modified=self.modified, - etag=f'"0x{hashlib.md5(self.data, usedforsecurity=False).hexdigest()[:16].upper()}"', # nosec B324 # nosemgrep # the digest under test, not a security use + etag=f'"0x{hashlib.md5(self.data, usedforsecurity=False).hexdigest()[:16].upper()}"', # nosec B324 # nosemgrep content_settings=SimpleNamespace(content_type=self.content_type), metadata={}, version_id=None, diff --git a/tests/test_storage_file.py b/tests/test_storage_file.py index a7cda49..2d40330 100644 --- a/tests/test_storage_file.py +++ b/tests/test_storage_file.py @@ -1,5 +1,9 @@ """File and Storage: the object API over the resolver, including cross-backend transfers.""" +# The nosec / nosemgrep markers below sit on made-up values and on digests that are the thing +# under test; none is a credential or a security use of a hash. +# pylint: disable=line-too-long # a marker has to follow the value it is about + # pylint: disable=redefined-outer-name # pytest passes fixtures by matching name # pylint: disable=use-implicit-booleaness-not-comparison # an exact empty value is what is asserted @@ -159,13 +163,13 @@ def test_checksum_and_verify(resolver: StorageResolver) -> None: file = File("memory://scratch/a.bin", resolver=resolver) file.write(PAYLOAD) assert file.checksum() == Checksum("sha256", SHA256) - assert file.checksum("md5").value == hashlib.md5(PAYLOAD, usedforsecurity=False).hexdigest() # nosec B324 # nosemgrep # the digest under test, not a security use + assert file.checksum("md5").value == hashlib.md5(PAYLOAD, usedforsecurity=False).hexdigest() # nosec B324 # nosemgrep assert file.verify(SHA256) is True assert file.verify(SHA256.upper()) is True assert file.verify(f"sha256:{SHA256}") is True assert file.verify(Checksum("sha256", SHA256)) is True assert file.verify("0" * 64) is False - md5 = hashlib.md5(PAYLOAD, usedforsecurity=False).hexdigest() # nosec B324 # nosemgrep # the digest under test, not a security use + md5 = hashlib.md5(PAYLOAD, usedforsecurity=False).hexdigest() # nosec B324 # nosemgrep assert file.verify(md5, algorithm="md5") is True assert file.verify(f"md5:{md5}") is True assert file.verify(md5) is False diff --git a/tests/test_storage_gdrive.py b/tests/test_storage_gdrive.py index 78ddac6..e91779a 100644 --- a/tests/test_storage_gdrive.py +++ b/tests/test_storage_gdrive.py @@ -6,6 +6,10 @@ one, and the last section names what that pins and what it leaves open. """ +# The nosec / nosemgrep markers below sit on made-up values and on digests that are the thing +# under test; none is a credential or a security use of a hash. +# pylint: disable=line-too-long # a marker has to follow the value it is about + # pylint: disable=line-too-long # an expected value is kept on one line # pylint: disable=protected-access # the tests look at private state on purpose # pylint: disable=redefined-outer-name # pytest passes fixtures by matching name @@ -115,7 +119,7 @@ def test_stat_reports_what_drive_holds(storage: GoogleDriveStorage, drive: FakeD info = storage.write_bytes("reports/q1.json", b"{}") assert info.path == "reports/q1.json" assert info.size == 2 - assert info.etag == hashlib.md5(b"{}", usedforsecurity=False).hexdigest() # nosec B324 # nosemgrep # the digest under test, not a security use + assert info.etag == hashlib.md5(b"{}", usedforsecurity=False).hexdigest() # nosec B324 # nosemgrep assert info.version == "1" assert info.content_type == "application/json" assert info.modified_at is not None diff --git a/tests/test_storage_onedrive.py b/tests/test_storage_onedrive.py index 5e1144e..23b2f43 100644 --- a/tests/test_storage_onedrive.py +++ b/tests/test_storage_onedrive.py @@ -55,7 +55,7 @@ KIB, SIMPLE_UPLOAD_LIMIT, UPLOAD_ORIGIN, - URL_SECRET, + URL_SIGNATURE, FakeGraph, graph_answer, ) @@ -272,7 +272,7 @@ def test_the_default_fragment_size_is_a_multiple_of_320_kib() -> None: @pytest.mark.parametrize( "failure", - [503, requests.ConnectionError(f"Max retries exceeded with url: /session/1?{URL_SECRET}")], + [503, requests.ConnectionError(f"Max retries exceeded with url: /session/1?{URL_SIGNATURE}")], ) def test_a_failed_fragment_cancels_the_session_and_keeps_its_url_secret( storage: OneDriveStorage, @@ -290,8 +290,8 @@ def test_a_failed_fragment_cancels_the_session_and_keeps_its_url_secret( assert storage.exists("big.bin") is False assert caught.value.__cause__ is None assert "onedrive:///big.bin" in str(caught.value) - assert URL_SECRET not in _printed(caught.value) - assert not any(URL_SECRET in message or FAKE_TOKEN in message for message in logged) + assert URL_SIGNATURE not in _printed(caught.value) + assert not any(URL_SIGNATURE in message or FAKE_TOKEN in message for message in logged) def test_a_session_that_cannot_be_cancelled_is_logged_and_the_first_error_wins( @@ -306,7 +306,7 @@ def test_a_session_that_cannot_be_cancelled_is_logged_and_the_first_error_wins( storage.write_bytes("big.bin", b"x" * 100) (warning,) = [message for message in logged if "could not be cancelled" in message] assert "onedrive:///big.bin" in warning - assert URL_SECRET not in warning + assert URL_SIGNATURE not in warning def test_an_upload_session_without_an_https_url_is_refused( @@ -351,12 +351,14 @@ def test_a_failed_download_keeps_the_download_url_secret_and_leaves_no_file( target = tmp_path / "out" / "a.txt" target.parent.mkdir() target.write_bytes(b"local") - graph.fail_with = requests.ConnectionError(f"Max retries exceeded with url: /c/1?{URL_SECRET}") + graph.fail_with = requests.ConnectionError( + f"Max retries exceeded with url: /c/1?{URL_SIGNATURE}" + ) graph.fail_calls = frozenset({"download"}) with pytest.raises(StorageTransientException) as caught: storage.download("a.txt", target) assert caught.value.__cause__ is None - assert URL_SECRET not in _printed(caught.value) + assert URL_SIGNATURE not in _printed(caught.value) assert target.read_bytes() == b"local" assert [entry.name for entry in target.parent.iterdir()] == ["a.txt"] diff --git a/tests/test_storage_s3.py b/tests/test_storage_s3.py index e51bb7b..a540927 100644 --- a/tests/test_storage_s3.py +++ b/tests/test_storage_s3.py @@ -6,6 +6,10 @@ real ``botocore`` exceptions. No request leaves the process. """ +# The nosec / nosemgrep markers below sit on made-up values and on digests that are the thing +# under test; none is a credential or a security use of a hash. +# pylint: disable=line-too-long # a marker has to follow the value it is about + # pylint: disable=protected-access # the tests look at private state on purpose # pylint: disable=raising-bad-type # a stand-in raises what the test hands it # pylint: disable=redefined-outer-name # pytest passes fixtures by matching name @@ -65,7 +69,7 @@ class _Object: @property def etag(self) -> str: - return f'"{hashlib.md5(self.data, usedforsecurity=False).hexdigest()}"' # nosec B324 # nosemgrep # the digest under test, not a security use + return f'"{hashlib.md5(self.data, usedforsecurity=False).hexdigest()}"' # nosec B324 # nosemgrep class _Paginator: @@ -220,7 +224,7 @@ def test_stat_reports_what_head_object_returns(storage: S3Storage) -> None: info = storage.write_bytes("reports/q1.json", b"{}") assert info.path == "reports/q1.json" assert info.size == 2 - assert info.etag == hashlib.md5(b"{}", usedforsecurity=False).hexdigest() # nosec B324 # nosemgrep # the digest under test, not a security use + assert info.etag == hashlib.md5(b"{}", usedforsecurity=False).hexdigest() # nosec B324 # nosemgrep assert info.content_type == "application/json" assert info.version is None assert dict(info.metadata) == {} diff --git a/tests/test_storage_webdav.py b/tests/test_storage_webdav.py index 2a830ad..233dc7c 100644 --- a/tests/test_storage_webdav.py +++ b/tests/test_storage_webdav.py @@ -6,6 +6,10 @@ process. """ +# The nosec / nosemgrep markers below sit on made-up values and on digests that are the thing +# under test; none is a credential or a security use of a hash. +# pylint: disable=line-too-long # a marker has to follow the value it is about + # pylint: disable=arguments-differ # a fixture or a stand-in takes other arguments than the one it replaces # pylint: disable=protected-access # the tests look at private state on purpose # pylint: disable=redefined-outer-name # pytest passes fixtures by matching name @@ -26,9 +30,10 @@ from pathlib import Path from typing import Any from urllib.parse import quote, unquote, urlsplit -from xml.sax.saxutils import ( - escape, # nosec B406 # nosemgrep # escapes what the fake server writes; parses nothing -) + +# The fake server escapes the text it writes; nothing is parsed with this module. +# nosemgrep +from xml.sax.saxutils import escape # nosec B406 import pytest import requests @@ -71,7 +76,7 @@ class _Resource: @property def etag(self) -> str: - return f'"{hashlib.md5(self.data, usedforsecurity=False).hexdigest()}"' # nosec B324 # nosemgrep # the digest under test, not a security use + return f'"{hashlib.md5(self.data, usedforsecurity=False).hexdigest()}"' # nosec B324 # nosemgrep class _Response: @@ -296,7 +301,7 @@ def test_stat_reports_what_propfind_returns(storage: WebDAVStorage, server: Fake assert info.size == 2 assert info.modified_at == stored.modified assert info.modified_at.utcoffset() == timedelta(0) - assert info.etag == hashlib.md5(b"{}", usedforsecurity=False).hexdigest() # nosec B324 # nosemgrep # the digest under test, not a security use + assert info.etag == hashlib.md5(b"{}", usedforsecurity=False).hexdigest() # nosec B324 # nosemgrep assert info.content_type == "application/json" assert info.version is None folder = storage.stat("reports") diff --git a/tests/test_ui_pages.py b/tests/test_ui_pages.py index 751affe..806326a 100644 --- a/tests/test_ui_pages.py +++ b/tests/test_ui_pages.py @@ -1,5 +1,9 @@ """The pages of the main window, fed by their services through a pool that runs at once.""" +# The nosec / nosemgrep markers below sit on made-up values and on digests that are the thing +# under test; none is a credential or a security use of a hash. +# pylint: disable=line-too-long # a marker has to follow the value it is about + # pylint: disable=protected-access # the tests look at private state on purpose # pylint: disable=use-implicit-booleaness-not-comparison # an exact empty value is what is asserted # pylint: disable=wrong-import-position # imports follow pytest.importorskip @@ -340,7 +344,7 @@ def test_the_dashboard_renders_the_summary_of_its_service( draft.add_task("T_fail", "only") run_id = services.pipelines.start(draft)["run_id"] assert services.pipelines.wait(run_id, WAIT) - bus.publish(SystemErrorEvent(source="test", subject="disk full", payload={"token": "abc"})) # nosec B105 # a made-up value for a stand-in, not a credential + bus.publish(SystemErrorEvent(source="test", subject="disk full", payload={"token": "abc"})) # nosec B105 page.refresh() assert page._headline.text() == "Needs attention" assert "1 of the last 1 pipeline runs failed" in page._reasons.text() @@ -626,7 +630,7 @@ def test_the_audit_page_configures_searches_and_shows_a_record( PipelineFailed( source="pipeline", subject="nightly failed", - payload={"pipeline": "nightly", "status": "failed", "password": "hunter2"}, # nosec B105 # a made-up value for a stand-in, not a credential + payload={"pipeline": "nightly", "status": "failed", "password": "hunter2"}, # nosec B105 ) ) bus.publish(SystemErrorEvent(source="system", subject="disk full")) diff --git a/tests/test_web_ui_app_layer.py b/tests/test_web_ui_app_layer.py index 4ec2af6..b0d4a82 100644 --- a/tests/test_web_ui_app_layer.py +++ b/tests/test_web_ui_app_layer.py @@ -1,5 +1,8 @@ """The Web UI renders its fragments from the application layer, escaped and masked.""" +# The nosec / nosemgrep markers below sit on made-up values and on digests that are the thing +# under test; none is a credential or a security use of a hash. + # pylint: disable=consider-using-with # the handle is closed by the code under test # pylint: disable=cyclic-import @@ -187,7 +190,7 @@ def test_secrets_are_masked_before_they_are_rendered( PipelineFailed( source="pipeline", subject="fetch https://user:hunter2@example.com/x failed", - payload={"error": "Bearer abc.def-123 was refused", "token": "t0ps3cret"}, # nosec B105 # a made-up value for a stand-in, not a credential + payload={"error": "Bearer abc.def-123 was refused", "token": "t0ps3cret"}, # nosec B105 ) ) for path in ("/ui/events", "/ui/audit", "/ui/health"): From b2794926edc11597e4cff9bc1558cbd69d4c177d Mon Sep 17 00:00:00 2001 From: JeffreyChen Date: Thu, 8 Oct 2026 16:39:53 +0800 Subject: [PATCH 53/59] docs: record the first CI runs of the pull request and what is left --- docs/updates/2026-10.md | 19 +++++++++++++++++++ docs/updates/README.md | 3 ++- progress.md | 6 +++--- 3 files changed, 24 insertions(+), 4 deletions(-) diff --git a/docs/updates/2026-10.md b/docs/updates/2026-10.md index a1bff6b..fdcf568 100644 --- a/docs/updates/2026-10.md +++ b/docs/updates/2026-10.md @@ -682,3 +682,22 @@ Index and query commands: [README.md](README.md). New entries go at the end. - **Result / numbers**: after the fix, 5763 passed, 257 skipped, 0 failed with every extra in three consecutive runs; 3879 passed, 137 skipped with the base dependencies only. - **Files**: `tests/test_mcp_pipeline_tools.py`. - **Open items**: none. + +## U-20261008-33 · 2026-10-08 · First CI runs of pull request #109, and what they showed · #ci #storage #incident + +- **What**: pull request #109 (`feat/universal-storage-layer` into `dev`) was opened and its checks were followed through four pushes. This was the first run of everything recorded as unverified in U-20261008-12, -23 and -24. +- **Passed without a change**: lint; the unit tests on Windows for Python 3.10 to 3.14; the base-dependency job; each extra on its own; the unit tests on Linux, where the symbolic-link tests of `LocalStorage` ran for the first time (`progress.md` #18, closed); the package build; the SFTP adapter against OpenSSH and the Azure adapter against Azurite, all 88 contract cases each. +- **Found by a real server or another platform, and fixed**: + - **WebDAV (Apache)** answers `400` to a `PROPFIND` for a path below a file, where the stand-in answered `404`. `WebDAVStorage._stat` now reads that as "nothing there", so writing below a file is refused as a path-type error and not as an unknown server error. + - **SMB (Samba)** refuses to rename onto an existing file (`STATUS_ACCESS_DENIED`). `SMBStorage` now removes the target and renames again in that case; that replace is not atomic. + - **SMB with a port other than 445**: listing and directory creation tried port 445 anyway. smbprotocol appears to drop the `port` argument in some calls. The CI server now listens on 445; the limit is recorded in `progress.md` #32. + - **macOS**: asking whether a directory with a name past the filesystem's limit exists raises `OSError` there (Windows answers "no"). `local/versioning.py` now treats that as absent. Two tests had failed. + - **The integration script**: MinIO's image can no longer be pulled from Docker Hub or quay.io without an account, so the S3 job uses S3Mock; the FTP login lands in a read-only root, so the tests use the user's own directory. +- **Static analysis**: + - **SonarCloud** reported twelve "bugs", all tests of `__eq__` written as `assert X(a) == X(a)`, which the rule reads as a self-comparison. They compare two named instances now. It also flagged the unlocked `pip install` of the new `package` job, which now installs the hash-locked tools the publish jobs use. + - **Codacy** counted 1070 new issues against a threshold of zero. About 960 were pylint rules that pytest idioms trip in test modules (fixtures by name, private state, stand-in signatures); each test module now disables, at file level and with a reason, only the rules it trips. 24 were a backend reading the private parts of another instance of its own class, disabled per module with that reason. Nine long lines were wrapped and one variable renamed. Bandit and Semgrep findings on digests under test, made-up credentials of stand-ins and two constant SQL statements carry a marker and a reason. Codacy passes. +- **Result**: every check passes except SonarCloud. Its one failing condition is the security rating of new code: eight findings on the `pip install` steps of `integration.yml`, the same kind of install the existing test jobs use. That is `progress.md` #40, for the owner to decide. +- **Result / numbers**: 5764 passed, 257 skipped locally with every extra; 3880 passed, 137 skipped with the base dependencies only. In CI: six service jobs, two platform jobs, five Python versions, thirteen extras, the base install, lint and the package build pass. +- **Still not verified**: Google Drive, OneDrive and Dropbox against their services; S3 against AWS (S3Mock was used); FTPS; UI 2.0 on a real display. +- **Files**: `automation_file/storage/{webdav_storage,smb_storage}.py`, `automation_file/local/versioning.py`, `tests/integration/start_service.sh`, `.github/workflows/ci-dev.yml`, `.github/workflows/ci-stable.yml`, the test modules and the library modules that carry the markers, the three `usage/integration_tests.rst`, `CLAUDE.md`, `progress.md`. +- **Open items**: #40 (SonarCloud), #19, #32. diff --git a/docs/updates/README.md b/docs/updates/README.md index 5988c95..47a67bb 100644 --- a/docs/updates/README.md +++ b/docs/updates/README.md @@ -58,6 +58,7 @@ In the same commit: delete the item from `progress.md`, add a `#done` entry here | ID | Date | Title | Tags | Batch | |---|---|---|---|---| +| U-20261008-33 | 2026-10-08 | First CI runs of pull request #109, and what they showed | #ci #storage #incident | [2026-10](2026-10.md) | | U-20261008-32 | 2026-10-08 | An intermittent test failure, and its wrong first diagnosis | #tests #incident | [2026-10](2026-10.md) | | U-20261008-31 | 2026-10-08 | Migration guide | #docs #migration #roadmap | [2026-10](2026-10.md) | | U-20261008-30 | 2026-10-08 | UI 2.0 and the application layer | #ui #roadmap #done | [2026-10](2026-10.md) | @@ -127,5 +128,5 @@ In the same commit: delete the item from `progress.md`, add a `#done` entry here | File | Period | Entries | |---|---|---:| -| [2026-10.md](2026-10.md) | 2026-10 | 43 | +| [2026-10.md](2026-10.md) | 2026-10 | 44 | | [2026-09.md](2026-09.md) | 2026-09 | 21 | diff --git a/progress.md b/progress.md index 69613b1..a64cf1a 100644 --- a/progress.md +++ b/progress.md @@ -15,12 +15,11 @@ Items #10 to #26 are what is left of the 1.0 roadmap (`docs/FILEAUTOMATION-1.0-R - **#17** Native streams and checksums for the remote backends. `open_read` / `open_write`, `copy_tree` and `sync_tree` exist (U-20261008-04), but outside `LocalStorage` a stream is a staged local copy and the default `checksum` downloads the file: S3 could read `get_object()["Body"]`, and a backend with a server-side digest could answer `checksum` from it. - **#31** `FsspecStorage(directories=False)`: a `/` placeholder key that another tool wrote survives `delete(dir, recursive=True)`, so the directory still exists afterwards, and an empty directory that only a placeholder holds cannot be deleted at all. `ObjectStorage` removes placeholders because it lists raw keys; through fsspec the key has to be addressed with the filesystem's own call, and `_strip_protocol` of s3fs / gcsfs removes a trailing slash, so `rm_file(path + "/")` is probably wrong. Needs a real s3fs or gcsfs (MinIO under #19) before it is written. -- **#32** [UNVERIFIED] No storage adapter has met a real service: FTP ran against an in-memory model of RFC 959 / 3659 and FTPS data channels did not run at all; SFTP met paramiko's own server, not OpenSSH; Microsoft Graph (`OneDriveClient.graph_send`), Drive (the discovery document as a transport), Dropbox, WebDAV and SMB met stand-ins, and the `smbprotocol` error shapes were written from its documentation. The hand-written redirect handling of `WebDAVClient` (U-20261008-14) has not met a server that redirects. Run each against the service (#19) and fix what differs. -- **#18** [UNVERIFIED] The five symbolic-link tests of `tests/test_storage_local.py` have not run anywhere: the development machine may not create links (they skip there). The Linux and macOS jobs of `integration.yml` can; read their first run and fix `LocalStorage` if one fails. +- **#32** [UNVERIFIED] Adapters that have still not met their real service: Google Drive, OneDrive (Microsoft Graph, `OneDriveClient.graph_send`) and Dropbox met stand-ins only; S3 met S3Mock, not AWS; FTPS data channels have not run. SFTP (OpenSSH), FTP (vsftpd), WebDAV (Apache), SMB (Samba) and Azure Blob (Azurite) pass the contract in CI. `SMBClient` with a port other than 445 fails in listing and directory creation: smbprotocol drops the `port` argument in some of its calls, so the CI server listens on 445; confirm against smbprotocol and either pass the port another way or document that only 445 works. ### Backend integration tests (roadmap M3) -- **#19** [UNVERIFIED] `.github/workflows/integration.yml` and `tests/integration/` (U-20261008-23) have never run: there is no Docker on the development machine. The first run of the workflow decides whether each of the six service jobs (`s3`, `azure`, `sftp`, `ftp`, `webdav`, `smb`) and the two platform jobs (Linux, macOS) work. Expect to adjust the container options in `tests/integration/start_service.sh` (image tags are floating; pin them to digests once a run is green; the Samba share options and the FTP passive ports are the least certain) and to fix what a real service or another platform shows (#32, #18). Still missing: credential-gated jobs for Google Drive, OneDrive and Dropbox, which have no emulator, and FTPS. +- **#19** Integration workflow, what is left after its first green run (U-20261008-33): pin the six container images to digests (they are pulled by floating tag); credential-gated jobs for Google Drive, OneDrive and Dropbox, which have no emulator; an FTPS run; and the real S3 API, since the S3 job runs against S3Mock (MinIO's image can no longer be pulled without an account). - **#36** Writing metadata through the storage layer. `upload`, `write_bytes` and `open_write` take no user metadata and no content type, so `stat().metadata` can be read but a caller cannot set it, and the content type is always the one guessed from the name. The contract checks what is read (U-20261008-25). Adding it means a keyword on the write operations, a capability-conditional contract case, and deciding what a backend without metadata does with the argument (refuse, or ignore). ### Later milestones @@ -29,6 +28,7 @@ Items #10 to #26 are what is left of the 1.0 roadmap (`docs/FILEAUTOMATION-1.0-R - **#39** The application layer reaches into two private places: `StorageService` reads `StorageResolver._mounts` / `_factories` (give the resolver a public listing of its mounts), and `app/` uses `pipeline.definition.retry_from_dict`, `graph.upstream_tasks`, `substitution.is_name` / `NAME_RULE` and `model.ON_SUCCESS` / `WHEN_CHOICES`, which are not in `automation_file.pipeline.__all__` (export them, or give the pipeline package the functions the editor needs). - **#37** Storage-layer gaps the MCP work found (U-20261008-27) and did not change: (a) a `LocalStorage(root)` listing reports a link's name and its target's metadata without a containment check, although reading through the link is refused; (b) `StorageBackend` has no ranged read, so reading the head of a large remote file stages all of it; (c) `LocalStorage` on Windows opens device names (`CON`, `NUL`) and alternate data streams, which the MCP tools refuse themselves; (d) a link on an SFTP or FTP server leads outside a root that is only a path prefix. - **#26** The 1.0.0 release (roadmap M9). Everything the roadmap lists is on the branch `feat/universal-storage-layer`; what is left is the owner's: review and merge the pull request to `dev`, read the first CI and integration runs (#19, #32, #18, #38), decide the 1.0 date, then write `1.0.0` in `stable.toml` and `dev.toml` in the release pull request to `main` and raise the `Development Status` classifier. Not written: a separate security page in the manual (the deployment chapter and `CLAUDE.md` § Security carry that guidance today). +- **#40** [DECIDE] SonarCloud fails pull request #109 on one condition, the security rating of new code: eight findings, all on the two `pip install` steps of the two jobs in `.github/workflows/integration.yml` (`S8541` no `--only-binary :all:`, `S8544` versions not locked). The `pytest`, `minimal` and `extras` jobs of `ci-dev.yml` install the same way and are not flagged only because their lines are not new. Either accept the findings in SonarCloud, or lock the test dependencies for every job: a `test.in` / `test.txt` pair next to `.github/requirements/publish.in`, generated with `uv pip compile` for the platforms the jobs run on, and `pip install --require-hashes --only-binary :all:` in each job. The lock could not be generated on the development machine (no `uv`, and a network too slow to resolve every extra). - **#34** [BLOCKED] PyPI Trusted Publishing (roadmap §13). `publish.yml` and `publish-dev` still upload with the `PYPI_API_TOKEN` secret. Switching needs the owner to add a trusted publisher for each project on PyPI (`automation_file`: workflow `publish.yml`; `automation_file_dev`: workflow `ci-dev.yml`; an environment name if one is wanted) before the workflows can drop the token for `id-token: write` and `pypa/gh-action-pypi-publish`. Changing the workflows first would break both channels. ### Packaging follow-ups From 7ffd7ce792cafd0edcbbc70b5ef25ed7bb27e875 Mon Sep 17 00:00:00 2001 From: JeffreyChen Date: Thu, 8 Oct 2026 17:15:50 +0800 Subject: [PATCH 54/59] Lock lint and integration dependencies and resolve main review findings --- .github/requirements/integration.in | 33 + .github/requirements/integration.txt | 1152 +++++++++++++++++++++++ .github/requirements/lint.in | 6 + .github/requirements/lint.txt | 290 ++++++ .github/workflows/ci-dev.yml | 5 +- .github/workflows/ci-stable.yml | 5 +- .github/workflows/integration.yml | 8 +- architecture.md | 1 + automation_file/core/action_executor.py | 5 +- automation_file/core/package_loader.py | 2 +- docs/updates/2026-10.md | 7 + docs/updates/README.md | 1 + progress.md | 2 +- tests/test_dev_toml_parity.py | 2 +- tests/test_tcp_wire.py | 3 +- tests/test_workflow_actions.py | 39 +- 16 files changed, 1545 insertions(+), 16 deletions(-) create mode 100644 .github/requirements/integration.in create mode 100644 .github/requirements/integration.txt create mode 100644 .github/requirements/lint.in create mode 100644 .github/requirements/lint.txt diff --git a/.github/requirements/integration.in b/.github/requirements/integration.in new file mode 100644 index 0000000..204c1a7 --- /dev/null +++ b/.github/requirements/integration.in @@ -0,0 +1,33 @@ +# Python 3.12 integration dependencies, locked for all runner platforms. +# Regenerate: uv pip compile --universal --python-version 3.12 --generate-hashes +# --only-binary :all: --exclude-newer 2026-10-02T00:00:00Z +# .github/requirements/integration.in -o .github/requirements/integration.txt +setuptools>=77 +wheel +requests>=2.31.0 +tqdm>=4.66.0 +watchdog>=4.0.0 +cryptography>=50.0.0 +prometheus_client>=0.26.0 +defusedxml>=0.7.1 +je_action_core>=0.0.2 +PyYAML>=6.0.3 +tzdata>=2024.1; platform_system == 'Windows' +opentelemetry-api>=1.44.0 +opentelemetry-sdk>=1.44.0 +tomli>=2.0.1; python_version<"3.11" +boto3>=1.34.0 +azure-storage-blob>=12.19.0 +google-api-python-client>=2.100.0 +google-auth-httplib2>=0.2.0 +google-auth-oauthlib>=1.2.0 +dropbox>=11.36.2 +paramiko>=3.4.0 +smbprotocol>=1.13.0 +fsspec>=2024.2.0 +msal>=1.39.0 +boxsdk>=10.15.0,<11 +pyarrow>=25.0.1 +PySide6>=6.6.0 +pytest>=8.0.0 +pytest-cov>=5.0.0 diff --git a/.github/requirements/integration.txt b/.github/requirements/integration.txt new file mode 100644 index 0000000..5559258 --- /dev/null +++ b/.github/requirements/integration.txt @@ -0,0 +1,1152 @@ +# This file was autogenerated by uv via the following command: +# uv pip compile FileAutomation/.github/requirements/integration.in --universal --generate-hashes --python-version 3.12 --only-binary :all: --exclude-newer 2026-10-02T00:00:00Z -o FileAutomation/.github/requirements/integration.txt +azure-core==1.41.0 \ + --hash=sha256:522b4011e8180b1a3dcd2024396a4e7fe9ac37fb8597db47163d230b5efe892d \ + --hash=sha256:f46ff5dfcd230f25cf1c19e8a34b8dc08a337b2503e268bb600a16c00db8ad5a + # via azure-storage-blob +azure-storage-blob==12.31.0 \ + --hash=sha256:0c0cb601d3462491d09ea96023cd791bb9dd4b173bf950daf3cff34ff47ba5b5 \ + --hash=sha256:997b393cfcbdc4b186d5911790d91f80387f7edc12c4d73eab963a2d26e5b2a9 + # via -r FileAutomation/.github/requirements/integration.in +bcrypt==5.0.0 \ + --hash=sha256:046ad6db88edb3c5ece4369af997938fb1c19d6a699b9c1b27b0db432faae4c4 \ + --hash=sha256:0c418ca99fd47e9c59a301744d63328f17798b5947b0f791e9af3c1c499c2d0a \ + --hash=sha256:0c8e093ea2532601a6f686edbc2c6b2ec24131ff5c52f7610dd64fa4553b5464 \ + --hash=sha256:0cae4cb350934dfd74c020525eeae0a5f79257e8a201c0c176f4b84fdbf2a4b4 \ + --hash=sha256:137c5156524328a24b9fac1cb5db0ba618bc97d11970b39184c1d87dc4bf1746 \ + --hash=sha256:200af71bc25f22006f4069060c88ed36f8aa4ff7f53e67ff04d2ab3f1e79a5b2 \ + --hash=sha256:212139484ab3207b1f0c00633d3be92fef3c5f0af17cad155679d03ff2ee1e41 \ + --hash=sha256:2b732e7d388fa22d48920baa267ba5d97cca38070b69c0e2d37087b381c681fd \ + --hash=sha256:35a77ec55b541e5e583eb3436ffbbf53b0ffa1fa16ca6782279daf95d146dcd9 \ + --hash=sha256:38cac74101777a6a7d3b3e3cfefa57089b5ada650dce2baf0cbdd9d65db22a9e \ + --hash=sha256:3abeb543874b2c0524ff40c57a4e14e5d3a66ff33fb423529c88f180fd756538 \ + --hash=sha256:3ca8a166b1140436e058298a34d88032ab62f15aae1c598580333dc21d27ef10 \ + --hash=sha256:3cf67a804fc66fc217e6914a5635000259fbbbb12e78a99488e4d5ba445a71eb \ + --hash=sha256:4870a52610537037adb382444fefd3706d96d663ac44cbb2f37e3919dca3d7ef \ + --hash=sha256:48f753100931605686f74e27a7b49238122aa761a9aefe9373265b8b7aa43ea4 \ + --hash=sha256:4bfd2a34de661f34d0bda43c3e4e79df586e4716ef401fe31ea39d69d581ef23 \ + --hash=sha256:560ddb6ec730386e7b3b26b8b4c88197aaed924430e7b74666a586ac997249ef \ + --hash=sha256:5b1589f4839a0899c146e8892efe320c0fa096568abd9b95593efac50a87cb75 \ + --hash=sha256:5feebf85a9cefda32966d8171f5db7e3ba964b77fdfe31919622256f80f9cf42 \ + --hash=sha256:611f0a17aa4a25a69362dcc299fda5c8a3d4f160e2abb3831041feb77393a14a \ + --hash=sha256:61afc381250c3182d9078551e3ac3a41da14154fbff647ddf52a769f588c4172 \ + --hash=sha256:64d7ce196203e468c457c37ec22390f1a61c85c6f0b8160fd752940ccfb3a683 \ + --hash=sha256:64ee8434b0da054d830fa8e89e1c8bf30061d539044a39524ff7dec90481e5c2 \ + --hash=sha256:6b8f520b61e8781efee73cba14e3e8c9556ccfb375623f4f97429544734545b4 \ + --hash=sha256:741449132f64b3524e95cd30e5cd3343006ce146088f074f31ab26b94e6c75ba \ + --hash=sha256:744d3c6b164caa658adcb72cb8cc9ad9b4b75c7db507ab4bc2480474a51989da \ + --hash=sha256:79cfa161eda8d2ddf29acad370356b47f02387153b11d46042e93a0a95127493 \ + --hash=sha256:7aeef54b60ceddb6f30ee3db090351ecf0d40ec6e2abf41430997407a46d2254 \ + --hash=sha256:7edda91d5ab52b15636d9c30da87d2cc84f426c72b9dba7a9b4fe142ba11f534 \ + --hash=sha256:7f277a4b3390ab4bebe597800a90da0edae882c6196d3038a73adf446c4f969f \ + --hash=sha256:7f4c94dec1b5ab5d522750cb059bb9409ea8872d4494fd152b53cca99f1ddd8c \ + --hash=sha256:801cad5ccb6b87d1b430f183269b94c24f248dddbbc5c1f78b6ed231743e001c \ + --hash=sha256:83e787d7a84dbbfba6f250dd7a5efd689e935f03dd83b0f919d39349e1f23f83 \ + --hash=sha256:89042e61b5e808b67daf24a434d89bab164d4de1746b37a8d173b6b14f3db9ff \ + --hash=sha256:92864f54fb48b4c718fc92a32825d0e42265a627f956bc0361fe869f1adc3e7d \ + --hash=sha256:9d52ed507c2488eddd6a95bccee4e808d3234fa78dd370e24bac65a21212b861 \ + --hash=sha256:9fffdb387abe6aa775af36ef16f55e318dcda4194ddbf82007a6f21da29de8f5 \ + --hash=sha256:a28bc05039bdf3289d757f49d616ab3efe8cf40d8e8001ccdd621cd4f98f4fc9 \ + --hash=sha256:a5393eae5722bcef046a990b84dff02b954904c36a194f6cfc817d7dca6c6f0b \ + --hash=sha256:a71f70ee269671460b37a449f5ff26982a6f2ba493b3eabdd687b4bf35f875ac \ + --hash=sha256:b17366316c654e1ad0306a6858e189fc835eca39f7eb2cafd6aaca8ce0c40a2e \ + --hash=sha256:baade0a5657654c2984468efb7d6c110db87ea63ef5a4b54732e7e337253e44f \ + --hash=sha256:c2388ca94ffee269b6038d48747f4ce8df0ffbea43f31abfa18ac72f0218effb \ + --hash=sha256:c58b56cdfb03202b3bcc9fd8daee8e8e9b6d7e3163aa97c631dfcfcc24d36c86 \ + --hash=sha256:cde08734f12c6a4e28dc6755cd11d3bdfea608d93d958fffbe95a7026ebe4980 \ + --hash=sha256:d79e5c65dcc9af213594d6f7f1fa2c98ad3fc10431e7aa53c176b441943efbdd \ + --hash=sha256:d8d65b564ec849643d9f7ea05c6d9f0cd7ca23bdd4ac0c2dbef1104ab504543d \ + --hash=sha256:db99dca3b1fdc3db87d7c57eac0c82281242d1eabf19dcb8a6b10eb29a2e72d1 \ + --hash=sha256:dcd58e2b3a908b5ecc9b9df2f0085592506ac2d5110786018ee5e160f28e0911 \ + --hash=sha256:dd19cf5184a90c873009244586396a6a884d591a5323f0e8a5922560718d4993 \ + --hash=sha256:ddb4e1500f6efdd402218ffe34d040a1196c072e07929b9820f363a1fd1f4191 \ + --hash=sha256:e3cf5b2560c7b5a142286f69bde914494b6d8f901aaa71e453078388a50881c4 \ + --hash=sha256:ed2e1365e31fc73f1825fa830f1c8f8917ca1b3ca6185773b349c20fd606cec2 \ + --hash=sha256:edfcdcedd0d0f05850c52ba3127b1fce70b9f89e0fe5ff16517df7e81fa3cbb8 \ + --hash=sha256:f0ce778135f60799d89c9693b9b398819d15f1921ba15fe719acb3178215a7db \ + --hash=sha256:f2347d3534e76bf50bca5500989d6c1d05ed64b440408057a37673282c654927 \ + --hash=sha256:f3c08197f3039bec79cee59a606d62b96b16669cff3949f21e74796b6e3cd2be \ + --hash=sha256:f632fd56fc4e61564f78b46a2269153122db34988e78b6be8b32d28507b7eaeb \ + --hash=sha256:f6984a24db30548fd39a44360532898c33528b74aedf81c26cf29c51ee47057e \ + --hash=sha256:f70aadb7a809305226daedf75d90379c397b094755a710d7014b8b117df1ebbf \ + --hash=sha256:f748f7c2d6fd375cc93d3fba7ef4a9e3a092421b8dbf34d8d4dc06be9492dfdd \ + --hash=sha256:f8429e1c410b4073944f03bd778a9e066e7fad723564a52ff91841d278dfc822 \ + --hash=sha256:fc746432b951e92b58317af8e0ca746efe93e66555f1b40888865ef5bf56446b + # via paramiko +boto3==1.43.107 \ + --hash=sha256:632a4f8298725b94144c8cd383cbdacb34624daa73221e954c3d33ac6e2528d6 \ + --hash=sha256:c4e0f1a0295cbb7103f2950128cf88463c076d220080a7d0f127cf834969fc3a + # via -r FileAutomation/.github/requirements/integration.in +botocore==1.43.107 \ + --hash=sha256:23cbe854e815dbaccf097f7fd32b461c9e1d2ed7e0c7dcc5658218704509d840 \ + --hash=sha256:4a37fa072a00280c746313532d19b00e2dc53f1993222601df104d71d548d5b6 + # via + # boto3 + # s3transfer +boxsdk==10.17.0 \ + --hash=sha256:86c9a1f665c8879af501c02ec51bbe02247b24f96c6bc39f6890912e70091840 \ + --hash=sha256:fe42658185901b80437a0d5b393c91b4a732f3b78d1be11d881aa7484807c9d5 + # via -r FileAutomation/.github/requirements/integration.in +certifi==2026.7.22 \ + --hash=sha256:62f22742b58a1a33014a2b6b706588a8d7e2a88ae7bd1a6ebe8c992928483775 \ + --hash=sha256:741e2c3b351ddf169a738da9f2c048608ff7f2c5cc02f1ebc6b118bb090d5d55 + # via requests +cffi==2.1.1 ; platform_python_implementation != 'PyPy' \ + --hash=sha256:046bfc24911b37851ee1b51aab8bffe713d89c68c6a057b09484ce9fd5f69b4e \ + --hash=sha256:06c72bb76605a4b0cd0aad6930b69d4baf7dd5d806cfc409b824191099700e66 \ + --hash=sha256:0beceaabe56af686895136a2de78db54ecd8e4046b236b8fd6d6cb61389e9bf2 \ + --hash=sha256:154852545011f779917b11c78db2358d095da62a9a172b78ad0a583ee5adc0d0 \ + --hash=sha256:194cffa889098ced9976c3fc6340305e43f6303657d298da55366907c05c22d6 \ + --hash=sha256:19ee6127ee34de7d83ce3d371ebc5ed91addbdcc39f9ab15ce4eb35a4e534971 \ + --hash=sha256:1a18a57b58cfb21fc28d72e876acf10eaed67a1ed96226f92af4df681d571c4c \ + --hash=sha256:1aa5645c30469b09530c4ebca77ebf8f17618293c58f8549cb1a543a50236e7d \ + --hash=sha256:1dea0e4d7d4f11f619fe8c1d76caf49e24405b4b5743c0e3be16a500ecd930c9 \ + --hash=sha256:208f941bb9d18e768138677f0a6d2ce01f590df56043dda1df1535ac57c88517 \ + --hash=sha256:210019b6c7cf07f081b4c54635c8cf744377001350e29cc0f81c4377b4797735 \ + --hash=sha256:246fa40ce8645a614ff682e0b70f37134e460eaf93a775e0cbe3cca585a67a80 \ + --hash=sha256:25792eac27877609e7bb06d42ff88278a6624fff2ba9bbb523c09616b117e80f \ + --hash=sha256:27350daa11d4f10c540e6e89dada4c54feb7256ad03e9a4dc075ebad7ba360d1 \ + --hash=sha256:28907ab9bfb6aa13184cfc17c6b8e1023c5ab6fd7076d8c20a35e59fe04f8f29 \ + --hash=sha256:2ae64be792b8966f2c69538199728b290e34726562896df1e5dc8ffd8d8188e8 \ + --hash=sha256:31348097ff5bbe827ccc41795d4dd099d9f0625e7def00ee653c137a490c2a6c \ + --hash=sha256:3143d81e29e1e20a9ce10901ec369012947876596f75a222235965f2b7ae832e \ + --hash=sha256:3222ba5d678f80a030e6afbcc33dc1ae5cb45facabb61cee2c7016b8432fde48 \ + --hash=sha256:3311ed60d36f83378794e1009ac6258bafbf81f7888b4caa7b35a521e3f95813 \ + --hash=sha256:334644fbac4eff73d985a17a91226df55d0f394160c4cfb880e084c8f7161cac \ + --hash=sha256:34e261f78cb6ceaaa36f42f2613f4380d94d9c759a9c73c769ee6e0247364632 \ + --hash=sha256:363e05fa78e15116c3c32c210ee36884fd6b9afa6d440e47112c3bd511d64cb6 \ + --hash=sha256:398aff33cee2767e3e781d2554c54bd0dff386bb437581e0d8011fde1a942ec1 \ + --hash=sha256:3d22a20b1fb1632cc72c22f95f7b0d2961c3e1c235f245ba4c606c4771035659 \ + --hash=sha256:42a494cee34437f05546455144f2b5d9ac09b1face62bcfce597d2e521066688 \ + --hash=sha256:42e2f76b9455f5a9a844f770bf3e200ed3da0e15f5df3db9c31fe80b04b3d004 \ + --hash=sha256:42f6930c31dc7f50732c9ae793c2786c7b6b044195967bbdde40bb9be81c4cc0 \ + --hash=sha256:456a61fa52d579ebf9df2e9552ead5129855dbaff6c1e5a9b1bc408809bdc062 \ + --hash=sha256:471cee653ae88de62096552e6d24ccb4a5adb8c8c9f10b5054d0122c15bf2779 \ + --hash=sha256:49cbc70e6542d4ccccb936558d1064a8012541e78f821f955cff24e357776c94 \ + --hash=sha256:4a7c934f7360e8cd64fe9efadcbd10c7c6364f531e432b9a4bf5ccbc9e0e8b50 \ + --hash=sha256:4be96343e422f2dfcd12ab5c9f5aebe03f82f737c6bffeca6830b3875cb44aab \ + --hash=sha256:4f42141fc14250de6dde5ee7ea4432be017252d91f19c5ad043c084cea629cac \ + --hash=sha256:507a24c282e0f42f8ed737cf048572cbf580468da5555764a8331735e9c736b6 \ + --hash=sha256:51b31d1c98274844cfd7838ce00bfc27c7423a4dc00fc0772fc3331c2cc90676 \ + --hash=sha256:58acb8ab8e295e6c5ea12f888cbb13cf21511ef2a3303a23f4325c29d17fe5c1 \ + --hash=sha256:5a59cc1c4442bc3d5c703bf720b51138d0bfc173618807c9ee2490a7541dd3d9 \ + --hash=sha256:5bb4e7ea95dcd6a014a6fef62e62467d67d8e582326443f3d68e71d6320a9fcf \ + --hash=sha256:5c58fe613dc5e5336357eff555824a314d8e43282600435c8d1cb6a7a2fedd13 \ + --hash=sha256:5e7cecbaadb83884793e05828cee59b210b24583b9c7425d0ba6a754fe22eb4e \ + --hash=sha256:616f097f2fe415bc92a247f02e11f634e1f9e9a83d327e3c915c15089c87869e \ + --hash=sha256:63bbfd5ded17c4840ac07cd8f1c21ba9d9708141f840b324f422f41b207e3973 \ + --hash=sha256:64faea20f4e2613363a1a9b9c7dd73058f3ecd00133a511e72ad7c511658f527 \ + --hash=sha256:661c298b4821edebead0c91edd2b00374d67ad7c5a1f7a91d4442633b79d6a72 \ + --hash=sha256:68e62fe11f30d5ca8289242866f0a5291402d8529ca2178ab8afc5c9694ae890 \ + --hash=sha256:6a8dddef476fab96d066d578fc88526767b836ab5ab21754e1d5bf3879c31c7c \ + --hash=sha256:6e192623c49c94421616a5778fba35cf0d5a8d000650c1967ef4448ee5cdd990 \ + --hash=sha256:7225e4514edb64eb6740324353e0da0711954fd8d7da4576755b1c6e09b697cd \ + --hash=sha256:75f80557d1389eddbd0de2681f6a390a0c5338c31ddaa821381c203fc3fd50d9 \ + --hash=sha256:770de9db11e84213beec501cfcaa013b019820ca881e03344dea5844f7876d94 \ + --hash=sha256:7750c6449dff7864bb9bb27ddfb0267756189201a3afc911d82b3caacd70dfc3 \ + --hash=sha256:7bde5e4cc5c10140859842b9d383af292b22639a4dffb725314baf45968cef80 \ + --hash=sha256:7ce713ace7c0e4520535b42b77eaa742c16dab813978064913e5a3cf82973b41 \ + --hash=sha256:7da0c5eff80f0197f3b3d1232ec5a682a9325f4ae9016a78f5f5ca35f9ced1f5 \ + --hash=sha256:7dbb61fe3a7699468030f71bbe5f8a0e326a151daa91beb11a6fc1f980c55e1c \ + --hash=sha256:811bd1e21d32de12efca32393a0ab3f5133b54fce9bd44b8bd77ab07da14bf6a \ + --hash=sha256:8ef53b2de9bcb9197d31854256575d59dbac0cba72ac627bb291ef5eceb74be4 \ + --hash=sha256:937c0052c05a31ca1daf18de3158eed4dbfcb9cc107adbea227728d647be701e \ + --hash=sha256:9d2055050ea716bd38b7f7f1579c275386646b4894c155a3e2f3cd62ed41b7c6 \ + --hash=sha256:9f8d177621de5cb38ee3e731eda45d421db093ec0739f46a5594babda7987a98 \ + --hash=sha256:a2d7755bef5a12ed488f4ef1f1b69ee9191d7396083b755a5d2295f6edb4768b \ + --hash=sha256:a48d62ab9d6f4f98c983223a547af44be6ca3691074c31cecced6facd3ba2dc1 \ + --hash=sha256:a4f00aa42f75d6e4595e8866e748cc1705adc0cddfeb2ca86d0d03993d63ba03 \ + --hash=sha256:a6e721d4b0e45d5b65e87534470e67b18dcd092c83f68fba09f152b9cbc061af \ + --hash=sha256:a730a083190634c65cca36ba5f489531576ebd79bcd5c8e172130f6453127231 \ + --hash=sha256:a931079504ecc49efed7744c476a5c343a92fabf66dec2db95edb1b2fdc770e2 \ + --hash=sha256:aa9511c62d14da7aacc9b4bf51f3f697a621e83b2d6919008243c3aad168eea3 \ + --hash=sha256:ab36d55f9ed2d067327667c2fea18dda018eb628dd6347aa01dda6cf1f5d3836 \ + --hash=sha256:ad2c86c495b899d862ea0f4b42891b8713a3bd45dd4105c7fd51c2a72f39f3a5 \ + --hash=sha256:aeae0e330c9f6acd681f647d46cefd30c29f93e3392882e792e82080c9691399 \ + --hash=sha256:b0431303acaea1089ad4b3e9ce4e6518193def1118d4073ca848635ee4ea2e96 \ + --hash=sha256:b5bdfd1c873d4e093aabc0ca84c4ca6dbc4f752afb5c86f146d9742580c9da2e \ + --hash=sha256:baed1e86cc735622097354b9d1281406caf42ff42a886d29faa8e8d1630333be \ + --hash=sha256:c1453022f490d2459a11819d83ad1d586e9ff65a12ac3e705ffebd46d3685dcf \ + --hash=sha256:c26608d2222fb1e94487e4a387d85f13eb55d5ed725cb25a0c589ac4ee60e7bc \ + --hash=sha256:c7659f22557c5a0bc4855cd635f55edec690cc008a40768527762cb9fb263455 \ + --hash=sha256:c8c69575568085ba0b1b10c0249d779a214aea6f6522e949a0fc9fb0fcb449d0 \ + --hash=sha256:c8d2c9fd1f2d16f780d15127abb050d13d1a76c03a4bd87d7e4980e45e511e12 \ + --hash=sha256:ca82be1a1d406ecfe1d25dc16cb33488e5a16bf4438c9fb590484ea29d92478b \ + --hash=sha256:cc572dace3f60ef98d7b12ff411d20f5362feb31a0439eab0085bbfd349982d7 \ + --hash=sha256:d18e5ac0f2f03f4f518d3e23db0f0cad7faa1da8620e9c09461d443bbf6e6692 \ + --hash=sha256:d28630f5854ab07ab1fd4aba756de52326c82e6be15d414b12793f1975048b54 \ + --hash=sha256:d9c275eaacd24aa73f94ffd6de08fc3f932424d8b6c376f4bed7cde376fe7bc3 \ + --hash=sha256:da0e573f9f97159390c89d9f1a9e41908b66d408cc5b58d08cf3847d844c531b \ + --hash=sha256:dd31f52ea1086513bb9df30f8fcee9b8918323ae067a3d5b78bc826a000712be \ + --hash=sha256:dddad92b554513a31f272570678ba307fb9f618f05e3d4a5eacafff9eae03e1d \ + --hash=sha256:df423d40ee8654634421812bc3b196da3f9bd7d32929da813f8394c4348a5358 \ + --hash=sha256:df913725b79db7bcf03448f36b7bf8815363417d5b58deecf9305e3e30f0f21a \ + --hash=sha256:e0bcb7e0f677f543555d2adff3bf19c05f66cdb4796e5ff602442ab2fe3c4ef7 \ + --hash=sha256:e2d65b31f36619cda3999b78b2aa9632e76b78448e7a56fc4240824200e7c4fc \ + --hash=sha256:e6e8cff14d6fb0be70a09c0bdc58096f501952d04624ebf867e0e56da2df8960 \ + --hash=sha256:f16c709686a78c727bbbf059f92b0bf41c6fc60deec706d2dc19f529175a6125 \ + --hash=sha256:f24fb43132a4c6b4cb4eb029492919b2db645be6808d738f244fd146c03c32cb \ + --hash=sha256:f53e442b08449d42821fa4a4fba000095af9f62742a500f978a9f557ec44339a \ + --hash=sha256:f5cfbc5fe74540d335175b656c725d74d90e3730c626d92575eea35029d9afaa \ + --hash=sha256:f81b3b8f3d4e343550fa4baa0e479bba9f2d29ce9c2e9b51d1ce1718d7442fcf \ + --hash=sha256:f8ec5e643a9a937f64e1999eb9f75d072263751912dc5cd06d3c85f8f44be7c3 \ + --hash=sha256:fb92203a88b3d3053034db775110081c49d28be6551923805e039924093761e4 \ + --hash=sha256:fcd22650c908d7b7da162bbfaab594a1227a15d1643a98c68b122ac642fa2264 + # via + # cryptography + # pynacl +charset-normalizer==3.5.2 \ + --hash=sha256:01077390b03f7988f11d700a2194e69b119741a86b1a638b1db88891e3eced8e \ + --hash=sha256:01b0c0d2262a9e28e8484a278c7e1b5d650e3ac8cf2683d2967e25899f208bdf \ + --hash=sha256:04851f73ae72b8413dddadb16a49dfee95263553741fd42d546f7d66907e6be5 \ + --hash=sha256:0521c5665880b33d603717defa76c094048900010897909952397feb3039da56 \ + --hash=sha256:0774bf9bf620249fee3e0b8b9fd3065de213be30f3aa94ce2494b3b638949e26 \ + --hash=sha256:0891b9d3903c5571c03771ca669a4b0ec5618ca722a5c957d3d29cd4e5062848 \ + --hash=sha256:0c951d5e6dd9c2ff60609476752bee49da4206adde960ebc247766937f72e718 \ + --hash=sha256:0fed1d06615f022ee3b13caf5e8b180cfea32bb2c5aded8a9d44277afc040f93 \ + --hash=sha256:114e4d0c92d618409ed82a99e22b5c5e768fe995f2973f78265f4524f49d4640 \ + --hash=sha256:11912e4bb14baae7c5d8791aa55ba0a3a03ec6729073307b0f57270abaa713d3 \ + --hash=sha256:11a4d68a6ecda3292cb1e50239e111543ba5d709bb62a6b4ea1afcfa729d8875 \ + --hash=sha256:124fbf1a8ff966d87ae05bb8bd45a71f966055ed8bba320d0c7cf450bc5f4d0e \ + --hash=sha256:1461ac396c4fdb983a675f20aa555624f0ee18ac83d832b9244ffff3d8055275 \ + --hash=sha256:1503bccbeb36d5527790c3930327704c39af22de3112f1b1666a9f3ce15ee204 \ + --hash=sha256:15bb4005af6320d259dc7593ca84a38d7fe06a421dbcf7b910ae23979101e787 \ + --hash=sha256:15c44f7edfd477b06f517a5cc317fc1707edb9de2c865f43d4b6513907473234 \ + --hash=sha256:16fa0eccf81304b79c5cd87f9271c3b85dd9dd99245e4422ae9c0dd45e0f99d3 \ + --hash=sha256:183b88127acdb4fabe59d951ab424faf1af7b63cdbb5f776186c1ea2ffcaed98 \ + --hash=sha256:195c26fb65950f8fce54e26349852b7bdd7c5f120aeefbcc440b8a20faaed4a3 \ + --hash=sha256:1afb975bd5d68d5ce9f6b6d44fdf2f7e34b895a35e95708a7a91b20a3b51d187 \ + --hash=sha256:1b4cbc7c3491ccb4aa17fcd8165649d01cf39f76de1696da8631b5f71b85401d \ + --hash=sha256:1bc0baf5ef96b6ede57d47f4b8fe4d9d84019c3bfcbeb20a41edc6a6ee341f1f \ + --hash=sha256:1c50fe28bbc2ced33386f298650d91218076c05420e6cbd790b913adc41659e7 \ + --hash=sha256:1db38f4c5496827c1a501846d64d14c3b80c7e6714e406cd7dc36a9899fa1011 \ + --hash=sha256:211d5a3eb6af8f513b8d4ca19a8c1b7accab1b5f0d3175f9826b03c1a920dc1f \ + --hash=sha256:23851fb4e1b85ed3f6c2a27b777cdfe2e19fb5b38429a8faf38c7542b7665869 \ + --hash=sha256:254eb48b9fa5ee9898a3c445825a1f340fe53712a098904b39b0bddba8ea3cb1 \ + --hash=sha256:2625388c6c754520c37abaf3b41eb34d1cc4a373f457898f08606c8e362b891d \ + --hash=sha256:281cb91036248400f4cc957495cccd44c275c2e0c5854f7e45ac5cf7dc193847 \ + --hash=sha256:28a15fdad492a99b6eccfaaed66ef3f74050680545ea61ec8b2f4c538f1f1320 \ + --hash=sha256:28b4f0d66fb834ff90f28209ac7bce77868c45d8c93e26f906709d9b7c2e1af9 \ + --hash=sha256:2a925889534b3748302dae5dead07cc13480de1dac3aea80a941b729b471ef93 \ + --hash=sha256:2b7b3bbfb4fe8ef40600792d762fbaa9057559f9d3fad209525b7a22b99e91fd \ + --hash=sha256:2c9ad19a6cfcd5ea5c0d41161d22f9df1dcc277e9bef2751391334546a314c00 \ + --hash=sha256:2cc961b171b3f3440f410489ab3573e86aea8736134ebbb40ea1338b7f0831bc \ + --hash=sha256:2ce45c6627b22c47e390bc91a41c3d13032192e699fa0bea96e9671b373d69b0 \ + --hash=sha256:2e06a3a98f916dd41d27f3105e02e7a40181c98c94b9158733d03a6f80506c09 \ + --hash=sha256:304d5463e65a35d7bb0850550e0780395395f6fcf452f04db7d5ca7cecc425ac \ + --hash=sha256:304d8e4d493af723536393eee0c689eb7813f4a474c8b479dee63f1fdd98f621 \ + --hash=sha256:30fcd120b732aa79317f08dee04d7de0847822e4cf7ee0e9f445bb958832252c \ + --hash=sha256:31f3930700408d211f13378ccbe1c40845d8da54bd0681fac3a9b5aae81c7aa8 \ + --hash=sha256:34276fd796040bf0993ab33a369aa572e6979c7aab225a88893667ad8eac8f7a \ + --hash=sha256:355ad8011081dec5412240c087a9a0c9d4d5039f3ed11a3f13e18c2b29b56c51 \ + --hash=sha256:38a873987f3be698494da8b2e3085e29da02da7b633dce73e79c699a113d7bf0 \ + --hash=sha256:39de2a259fc954455c57274dc94c79d5842774e1247a016aff30bc0efed0f4ef \ + --hash=sha256:3d14b50de6bf4d0edf857a9386836846f982b8f524e188e2e68b96d702bcf4aa \ + --hash=sha256:3d21b8b13c7592db2ac5e544a6d83187b995257472b0c9e8351b6d507ae37ed6 \ + --hash=sha256:3d31298449090ab8d47b7b1b2a555ff73cac7ed438a08b7ac160980c7ebed649 \ + --hash=sha256:3ddacd27458c45bdacd6bd6db644bfb730efbf9e830310186e3045c9c5be8fb2 \ + --hash=sha256:3df041de8887954562c9b261cba85ca0e9ded74048daf125f45edcfaa4832229 \ + --hash=sha256:40ab6bffa02ae10a0581e6c198be7d2d8ca5c2a0c64e4ed3465d766df457573e \ + --hash=sha256:4275811936e2f06feff5e598fb42a1b7ae852da8e39605211892b56b81a34efd \ + --hash=sha256:443eae2bf318abeaf6f15d785138f71fd6de770e99a92158b8b814265e079115 \ + --hash=sha256:447441e76ec720b15e64418d32e092297340387053047c7c694f579efb0ee1d9 \ + --hash=sha256:4495c5002a7b28557e7e222e77e0b661183e432b7d6d2e788101e3f240e05b8c \ + --hash=sha256:44bd4fbb29dfbeba60e7d2bd000c59e4b21ddb3cc53912b14048d37092706d7c \ + --hash=sha256:4685902cf26edf013ed7a3da0f426ebba7a00ebb9541386d835afbf002c11cab \ + --hash=sha256:498dc3188ca05a68231ac3fdbfc7f57eb67e1343c30e0fea17f8218c1599b253 \ + --hash=sha256:4c2b5031f63e331e3839b40aed2dd6f191e9c07edbde303e7876846ea1946995 \ + --hash=sha256:4d48f2d08b9de5864e2c8744d4461b862fb149a18274abc8b698c45975573438 \ + --hash=sha256:4f87960d57feabfb618e4e0af6e7371645fa26a277860739d6e5d6e0012c92f0 \ + --hash=sha256:50e3adfb96fc189eb27b1cf62d3b598b89b4bb0420d93a3d3e42e137409011be \ + --hash=sha256:51cf45226a9b588d0d2b4880c62d686934b63ab0bd79ca23ab0e9762eb27441b \ + --hash=sha256:52aa6992700996af31f375de0c6bacd402b0097fe40b53c426b9f51a90ebabc7 \ + --hash=sha256:55ea99acb17b9325618de155a0cd6a2e8f5d10be008113e1d433bbb58db543b2 \ + --hash=sha256:56bc200a365efb37383b7852e4cc5898d3b2da5987289b543956cf8cad71018a \ + --hash=sha256:588461c2e8384d309bd63e5826019b6977bc66d629b99ac8737bb795d7b2cb5a \ + --hash=sha256:58ca3755ee7ff7f59b57789ec9833c9de9ea275405cdd240eda1f193112e398a \ + --hash=sha256:58f361dcbab699cf8f42db3f47c8e7fd1036f138c23a5d08de9fde5f425a730c \ + --hash=sha256:598a11a2c7ebaa5334bf698bf29568c9c390abac6a154d8170fedecd1cea38c5 \ + --hash=sha256:59f63901b0031c3136cf64704dcb21de0bbae62ce2c9529bc39d27665463de37 \ + --hash=sha256:5cde776b7cc66e4f6c99612cea4aa7269aa65863f7a15841b2c264f103822f4e \ + --hash=sha256:5e2b6b57e9733d39f0c9fd3185efa6b8e29652c4cd8fe94180272cf6ed9a78c4 \ + --hash=sha256:5fb29fb8cd1a46c27a1bf9613ad5ec2599310d46b4025d9556404a6b6a292800 \ + --hash=sha256:6045373d5a89a5ec71afde535db987ca28e76dfa276c2d4c818265b375d4b055 \ + --hash=sha256:619799369eeef6366ed3e8755a5670f4f2f0fb6b30a0fd7264dc0fdc2357058e \ + --hash=sha256:62588a277bfb59def052abd940703fa35107152bf479781a878617d60faf8fb5 \ + --hash=sha256:62603db9a7caa0802eaa28c1c46fecd7b3a263a774069c24c3c28c302448721c \ + --hash=sha256:65cd72beeeca9d3aaea1201e5923859f308f952f9c71de93f06063c79f0f7a3b \ + --hash=sha256:68eb192d85ab8e5f6ec69c2bc6ac0179fbf04a5ac1569d12fbef74883fe102d0 \ + --hash=sha256:6bd128f206a7752ae1f2ab6c61bf8a24ba28913a10df8b14c2637b973ff97a80 \ + --hash=sha256:6be488a102b8cf28d0391d8c4ba7748938ae28b78ad901f8585520fca33ead1a \ + --hash=sha256:7218e8f32b0956cfcd048fd42d9d5779809745ca1d86113ca56f66e7ae1549c4 \ + --hash=sha256:7441d755b7ab94f8d4eb3e43ec05482d760842fd263d003a99102d742cd835e2 \ + --hash=sha256:749e97e1b32313717a565abbe321bc2190bc8b35f1a67e4cdbc7c56c8d8ffe58 \ + --hash=sha256:75a3ceed0724d625d64b86ca20aba182e4df462e04c2414fc941c0f523f06aac \ + --hash=sha256:780fbe7cab297b81dad9fb8dc5eb003c0468ffb0d9e5f65068c53a34661a96bc \ + --hash=sha256:78456a747de8dc58360ffa581f30a002baf5aa28cb262536545e91f113ed7639 \ + --hash=sha256:7967d08cf06dee78443b874f98c98036f624f3a4e73e11f9f64f5be4d25393cf \ + --hash=sha256:7a881931aa470808df94a8c380eed2bbbc76cd9dc622310f99665658c821eb6d \ + --hash=sha256:7dcd882da75ef9adf94903b1e3b9419e8aa8fb4c7396822b834b9ef7fb96954f \ + --hash=sha256:7e841fb9010836c992c9f12fcbd43a831de93a5f726fc1ccd8ca1d0268c5014c \ + --hash=sha256:7fdde2c9fd9e3eca40631e024664cf2584272cc8f96308cbe5fdfc930f51d8bc \ + --hash=sha256:8024d00c3faf3fc0c16e07a69f4405e8eac7cc0ab15f65fe6cf43827c4cf72b4 \ + --hash=sha256:80d02b6f04e92601a081dd97b23d3128033098bff5d35d392ddcc0476ea11253 \ + --hash=sha256:838dcc90063569a0448120554591a1d6c4a4ffe11babf048908793154ab86ade \ + --hash=sha256:849df64e889b2e17230d58410a03dba311a65b163508fd33679b2b737d4b7858 \ + --hash=sha256:87475fabc8d9996fd9c27debb395e642e8c838d78a00b6e932227a0e06b81e26 \ + --hash=sha256:87e50a3e7cb90af586b6c5faf23e302a970415ac73bd7bd90a515a04b427ef96 \ + --hash=sha256:89b53f3cda69831909888e0494f4fa0bcd3537e3e138dabeb620bd6ad946bae8 \ + --hash=sha256:8a893cc101149f80a653f82062ebc95b34525a2614382e1da5458fe7c6997249 \ + --hash=sha256:8b2bfab86aa71ae13aa41a6a26aab338e0db2b8bc75434b05aea89e011ff35a4 \ + --hash=sha256:8d86d6fc60743dc916eb79e2eb1ec4818e21e427731543af40a3021851174a13 \ + --hash=sha256:915563965d418f986e7e145accc592eae9e1a1be3566ff98a05d7a9ec42a76e1 \ + --hash=sha256:92888bb3187c5ba50500b00b3b310c9f2c651709d28036077680cb5255450a03 \ + --hash=sha256:93223adc95033dd47133a46ccfc316a0139176fd79085762e27202ec56018f03 \ + --hash=sha256:9373ad13ef0d2c0fb761e04e55bfdee5a08b52cef2c882c8fbe9935b1517152e \ + --hash=sha256:9409a8bf35cf78353942504b24a57de3d75b708997a1e4bd8db71ac8633ce364 \ + --hash=sha256:9b7f416ff0978e2f2249330527f0ad6fa02f4932e6199692d3b52da2048c19e4 \ + --hash=sha256:9bde855991b7e362c146535e3136a50bfaffc0487d38b33ca7e5edefc6e23849 \ + --hash=sha256:9cae88599c7219005d879f98e5ed53341e9a122af585e1091200358a3003d2a0 \ + --hash=sha256:9cf9b1a857e25c4baceeb3624e92a56df3668f398c4acba74e174d81fb4d1d3a \ + --hash=sha256:9f56f72050826f63dcee7a7f55b0a77168cb3bfc553fd405e7f8f9ece75a4036 \ + --hash=sha256:a090bb2c68df85450502e3e20d665e3a5af9c65a84d6508ed477badd49166fd3 \ + --hash=sha256:a192e2c40070d92c3ccf777e3a5c4ff515573cd2bb7ed0c537fdadbbec5bbf21 \ + --hash=sha256:a19a731138fc27d5682277d3b9df22855cea1239bce7fcec5f78f42ef2d1f3c3 \ + --hash=sha256:a66c3bc5ab1f0ff2164fc9965ddd611ff0802173f4b9d24554c563f6ab7e1d6e \ + --hash=sha256:a815775b6c38d4e0ff7bcffbeba67feded90202bb6a226b8dd35f1c855217413 \ + --hash=sha256:a89012d6d5476ee112d20d998570ed58df2260a852afb1758809cd6900411d21 \ + --hash=sha256:ae4f5fea5b8b8ccff88238cc8569303e5ee95efae67fa62922a311397a71f346 \ + --hash=sha256:b6856554c4f44d79fc2307d5768854310a8f0096e501c75637542c82292b0429 \ + --hash=sha256:b6b751274acb69d77b3323d6b7dbaa3c7fdfc1eb829b7eb61d262f32e1af9685 \ + --hash=sha256:b736353c0a625bbd5fcec108576e2385db3496f4f771f785ff32e108d3c3bc45 \ + --hash=sha256:b7fd005a73d9e657273b7a10dc71a9e03c8fb9ee6999798d6918ce095b81ac7f \ + --hash=sha256:b91363207bd9dc966a691e959bb47f64b30f7ac4b072be9968b366982f7db77c \ + --hash=sha256:ba0b1d2620edf869789c3879223f52bf2afc5d31b3cb47cc57b3a12c05e2aa9d \ + --hash=sha256:bbbfc8e28816f19d7c0f1816664980c0a9875d01b27cdf8eedddb639d9e108ad \ + --hash=sha256:bd16aabe4a02a297c23417aa17ac6299dbd8c49f673bcd645b4929b11f5a4400 \ + --hash=sha256:c0afc6800ba57ccc350374c5bd6150419915d95ce93cdbab2d783d75eaf30ecb \ + --hash=sha256:c6708715abcf3c73b99508253e961a9967f02fe536532834149574eda6de0d1c \ + --hash=sha256:c7c9ab723cde841fefb34efbad91e87f00a674b1fe1cd0784fde742bf2c154dc \ + --hash=sha256:c8f3d67aeaf55f017982b73683f0e7342ba2f6635a78f69ce89ebb26aa411e5c \ + --hash=sha256:c9790464842f85f437dbbb54417eda1e0e6bfc52dd8d22d6fd1c994b73b2dc74 \ + --hash=sha256:ca403d7e4798f525fdfc78e258820419cbbd0f0ecbab9de7840e3c017cf6b8cf \ + --hash=sha256:d008d90a7f2471519aef0c90dfbe73b3e6e4d5e66ac48e19154c17e89e98b604 \ + --hash=sha256:d19fbd981a488e22cd04883659ca6b08f50b5974f9fd7c95655ef6a043e5893f \ + --hash=sha256:d1befeed746d247c81127bb14de9dc3d30edb6e5976d34f83f86ed262b1d9105 \ + --hash=sha256:d2374b62878abb00cd8309b32af6c0b715cd02dec0ca74ef12e5069bdc64144a \ + --hash=sha256:d376bbd28b3a8999db1a103b3b388aee6f1ddeb3e51bc2172993efdcd86e064d \ + --hash=sha256:d4a7319f304a774bed22115bc891618e45f85065ab44ea6acd07d274e750519a \ + --hash=sha256:d6734d2ef8a50fbf8445c139477da401f50d62a0606bf00e20ec6d87773fefb1 \ + --hash=sha256:d760fe2a4d7c3b226cb9026d6a842868d52a7901bd98420e1baf14e80da85cf5 \ + --hash=sha256:d913de495d90407cd859d263bee2e5d1a4ed3eb6573c04e70d9ec619a7cbed7f \ + --hash=sha256:db19d07e2e0129e974a0e65d0064fc222a446cd5122c2fd4184d2af9fc734a9e \ + --hash=sha256:dca9ab98072a5a54ebacebdc45f53e645336b320c667410b061be1ca588ae709 \ + --hash=sha256:ddc7dacc8ece3a182e7f15cb862d1fd616b46d076cb1ae9dd232b2c38b655874 \ + --hash=sha256:ddf19c062bea7a0cc80f519243d2c01dd091be0cf952a0750d4ad576709559f5 \ + --hash=sha256:def79fa35ef0cef8d2accec024f4fdc7ead3012ff02f5215c783f39f03ef8cfc \ + --hash=sha256:df29a0a7107f7011e77f4eebdddec4c7331e24d787a0b21a46d63bdf7445da95 \ + --hash=sha256:e09a3942ecbdee5cce73ea9d42da82b81b72ac1bf031ce069b93b5adf4eac8cd \ + --hash=sha256:e242bb1c5e76e97dfa9e7f209a71e93a01d7f19ffdd5cfbb2e2d55b4f08f8ab0 \ + --hash=sha256:e243bd13217235fc7290c621941c3f5cc8b66e4872495be821d7436ba2fb838d \ + --hash=sha256:e2af3aad578aa6bd1384bcf4750fc285e5a9de53f40b7d41e5a0bf748edeb2b3 \ + --hash=sha256:e4e81e09c1578b8df602e3db08b0b3ea0a6947ad612f52bf8dc5ea8d47691f0c \ + --hash=sha256:e54da4baf05720032d527874d40b65fa4d7e5c6c6a43d0c3adbeffcaf275a2b3 \ + --hash=sha256:e80e6c2f55656b4824d72065abb4ddd6a525c74bd78a0aab5d9fc2cf4fb5af50 \ + --hash=sha256:ed2a239c0ea213acc1908150a3037257083c7c083128f1a4cec2ec4b97dca491 \ + --hash=sha256:ed905975ab14056a2e5eb1c376cb2e1ebc5396baf84163939c518556fccde9f5 \ + --hash=sha256:ee21e28f0430bd6dc9086c6e525d5e818a44a5ad19720c8a0ef766792f3eb5e5 \ + --hash=sha256:ee43c17b173d46a3212baa6ead3ae258eeabdae48c263a01ccf0218c366dd655 \ + --hash=sha256:ef4fcbf3327382cd4c9f540babd61248208af7b93eec4de397b4d5f58a09e288 \ + --hash=sha256:eff0ac9dbe711a4aee69bf04a83896aa9b85f19641264053a9f6d48573abb7dd \ + --hash=sha256:f0aa869112ef88429ae17820d99c3dd9504c9e9c671d3c246f3d7442cb051084 \ + --hash=sha256:f3c96f633825733f735c5a9cf21d21a257d8e1edf0b1cee0a064b9c424ca0f7d \ + --hash=sha256:f5833ad231be5eb6553de524a70f48d71b2c8563101750531e0b80184e175cd4 \ + --hash=sha256:f5ec61164adcec446f8969a3358ec3f9b26bbda3b9213e5586d219afa8df2915 \ + --hash=sha256:f7d486c83842422badd511868fd8a9a20e9407ace71564b6af47ce7e60a336c1 \ + --hash=sha256:fb9e68df06293761f9fe66ade60a9bc6d0f5e42b8acf2939a9158af86ab0e5bd \ + --hash=sha256:fc14a032f813bf5fe624d991960ea83e9715adc27e4c1830a2361eb1d02ac341 \ + --hash=sha256:fcff63213e8e6e47770541a4607175404f47cbb3ebea7b6058cc82d524a0e424 \ + --hash=sha256:fd1fbe0f116b6e55da77aca2c6ddcddcfac2186cbf78bdebf40fc156efca389d \ + --hash=sha256:fe9753dfee015c570d73df76f899f18444d41388bffcde097deba51c4fadbb9f + # via requests +colorama==0.4.6 ; sys_platform == 'win32' \ + --hash=sha256:08695f5cb7ed6e0531a20572697297273c47b8cae5a63ffc6d6ed5c201be6e44 \ + --hash=sha256:4f1d9991f5acc0ca119f9d443620b77f9d6b33703e51011c16baf57afb285fc6 + # via + # pytest + # tqdm +coverage==7.16.2 \ + --hash=sha256:00d3eb96e9988c45f50cccd1f1496571ac5c1f91386ac02c4d55516eeda19a24 \ + --hash=sha256:01c6908bc613b420c26c818fe948e1b97dfd041a53c98b01c63bd8321f5c9aae \ + --hash=sha256:066429634299e14dd2d511e1e85f8f9cecc500781f6b41907c0dd6f1baea7e63 \ + --hash=sha256:0993d0e90858c03943d3cb152e068a20dd4707924deec84dd2230261baae3b1b \ + --hash=sha256:0dcbcfcc059117284c603ff8cb61a65872512882f84a8cf0339241f7f7c2f148 \ + --hash=sha256:0fd7a86fdda7cb6d616d178654bd0ad6bc0f3f33c2e478aa598500a1a9e34eda \ + --hash=sha256:11d28e9123a9156cb405d8d27b44256c9a58fb5decc2073a8f17862057e3aa0f \ + --hash=sha256:11e597173af1dc33d5f8a7332ada544199269a223af1ee1770ddd5e245ad0fe8 \ + --hash=sha256:126d1af8804d7224421fe991ff65d3ce649081560df7a98b1a5ffff07f9923bd \ + --hash=sha256:14253fc7bb15749b849795a06f5d3b6d8bc3fb8a4b5ddc341faf7a89dce205fc \ + --hash=sha256:152877cdc8a07264882cfcd503ba56a3ef6cba56a70e8c70f6eb8ffd7384789a \ + --hash=sha256:17228fbca0f22976f797be94e975dcd237799c657d49551c7de1e0654d1202e9 \ + --hash=sha256:191803c4996b499fcd78c2ad5e5f767dcc53cb4dc6de6d6a741b443a1821ef02 \ + --hash=sha256:1a37c6e478cf687e1aa30a593d19c92c02fad9d122b51ab73f51b8dc7a0c0fc9 \ + --hash=sha256:1c569a9fd25505f1cd6bea90588818f90373ce90e2632e2cacf19ddbd6e14fdb \ + --hash=sha256:1d56e4d21c56d2046447733f8b118409597db48c01efe898ee9ac24e858ec2d6 \ + --hash=sha256:1d5d0e3b660506fb84f995814e3118a21efdc0c8eb80127da1be627d90093c17 \ + --hash=sha256:1f15254427c9b33eedac4f198eaf9e356eb4f6214551afb43da6194a2c088ad7 \ + --hash=sha256:218d742afca2b5ad5ca759e93eddedfbcc6eadf8322f080dcefc40b7bd4e2d48 \ + --hash=sha256:22957cef43ce038641de78ba995de7568d2d6a37c6ddbf7fa0fd7d1ae2344d91 \ + --hash=sha256:23219888477edd736b6fcaec1272d47d93b926e999641ffea7e53a1738e70b2b \ + --hash=sha256:251aed777c47c77aba047096d4542889db089227655711dfc2b9c54ef0e15e35 \ + --hash=sha256:28ff850182a67d117990fa2ce5ea1032836d8c9630dae867e8bdd3bff4533b79 \ + --hash=sha256:29309ccc86b7f33df7db12813c299f215bbbc470ed6292d0bedd63ffae1ebf64 \ + --hash=sha256:2aca0bdfa9e91621d5b09d815357bf63def4fc0e9cb66da67bf2cf93f3b1a6f5 \ + --hash=sha256:30c1b65d529e46569899fadca59e4a87c1faf2886923f1307ba61e654d4f3c20 \ + --hash=sha256:35f37886699cb9abd29958247d718628d5bc6f39e623dff66a09e546c42a7e03 \ + --hash=sha256:382d3346d56b0eec1b793d53a4c88799c8053f516aa3a8d7c44315696954bacf \ + --hash=sha256:396bb16e04ce04efbb3df91456ae4e3da918e69ecdf67fb711b0a0fdf35ccce0 \ + --hash=sha256:3e7f99698ba3a7d13988bdd984b7ebf13af4dbe2166dc8502eef90d77603b0a4 \ + --hash=sha256:3e861f1071dcc2fec1e88bef0920f6b1eaa66a143555b4f8ab79ba2b0f30ef55 \ + --hash=sha256:3f43bac1856ba269b905302778d4df433d6006489a192174ad77ac528e395032 \ + --hash=sha256:40c0f00899fe6181ae7f434ceb200e51f5ee4b8ed10e3b5f0b605f0cae15da87 \ + --hash=sha256:414c26dfdb96aac2d570a54e03008f001e32eb2d413705365503648c6bd361d8 \ + --hash=sha256:4358b9c8c0125b460407f3017c6cce8156e904b32772c5630d27112f52bdbfe5 \ + --hash=sha256:444889f7f66b74e4455c0a97e0e166dd41177f1dca8c0239a47cff25e05ba7e1 \ + --hash=sha256:44f21e407b278efdfc1ee5e481e00518bd1d500310a30a5fbf2bcbedfef4aaf0 \ + --hash=sha256:4cc4f73aa3fabc36e32046d6cd2971405948d8a903636508a3d3b2f9128b3a95 \ + --hash=sha256:4dbbd1155ca46e6e0b6b89d204428c56ef6a459af21333f365d135a2820e5a09 \ + --hash=sha256:4ee546b9e4872ffa194bf07ac87bfa1202ebb824d0795dc1ef22f175545ca90a \ + --hash=sha256:5139009b5efd2194fc168ee9362f0e191ba612ef5d29242f9269c22f9b8f80c7 \ + --hash=sha256:5375ebd99038021b35e99dc88255022912c06565d316212f4a576e4b08d30f5d \ + --hash=sha256:5397e21a90dde0e9c6896b77ded8f0be26b66f8b22b33aed41f6043ed95d55e6 \ + --hash=sha256:57ff3783f99d75a1e81dd56a9737eb5665e6736a5d93258ba596b6dcad8fd05b \ + --hash=sha256:58d4a54c6ea672afef66d49be922a2c69826c5ae1a42a9cd94f0c9c2bacdf800 \ + --hash=sha256:59c3926585e1cd1f2190f4b2ac9014de1bbeaf0d5d0587b0dc6b0aa90d17896a \ + --hash=sha256:5a27b731c171e43dc8b5f32b76a5051dde2ec9b9366c87028f08a7088ebc2c7b \ + --hash=sha256:5b3146d2317c75f70df2509066d979dadd941f7021cdf9b5db4bcd8568258e25 \ + --hash=sha256:5dca0bb66b4c3d624ba047887bf70270030c150692d543cb501293dc38a9f4b5 \ + --hash=sha256:611a44e5229a59d7483ce830160e1a0e85f700562c7a5651c7c63fb8f4eb528c \ + --hash=sha256:648352b94507179d82637292e7ae8802508d95f78e2f00a705a50b6c48011681 \ + --hash=sha256:6a75180829efb8ae62b4aded25be6ddca1c888d138d2d82e21d93bfbd88f41cb \ + --hash=sha256:705e5af11d34647efdc170c7840b6857c81cf74be96419a553f237e68e62cb72 \ + --hash=sha256:723dcdab91357159b722935b500ee8abc0a66c8c432e1e9fabf4cc7598952de8 \ + --hash=sha256:724bd0f1e81856b35e59fc98cf7b4e544a3cb662e4e0864dca73d4326ee9d808 \ + --hash=sha256:732d950e51f3ba4fb6209c73250f3e8924fefca42953ee04a9e65d8c02414d7d \ + --hash=sha256:736fde09ea39646d11f8e3b76bd3425c075aa4dd45f24891970bb77c14ff20f5 \ + --hash=sha256:7a076277ca9f5750cc230f0f578ebd2620cec60255b25707361699fef6fb465c \ + --hash=sha256:7b3bce4a0d05401d70b7d0d5ca783e686bc9d30e81dbd7d980d532609bf809e4 \ + --hash=sha256:7b451c68218c150f616bc9649783ec8de76a59792c759b43aa0c9c0466a465e4 \ + --hash=sha256:7d0732c83746bc24123c581a85d9dd96b70ddb538c9076020aa1a041790361e9 \ + --hash=sha256:7ed238d227e23cc300c3d464babdaf9f6ddc740aa1b15a77ae96136e6a7c4516 \ + --hash=sha256:80d3f7b48d43ee8fc5e8707a8adb43d743a5a1a85256c25a24f9d6d0e2238fa6 \ + --hash=sha256:80e9fdb4c3d926b6ba721d4bf7435bdb869c3527ae7803290361d0ab73db13b6 \ + --hash=sha256:848893e1d361448c113dc2f0913503522a6f7be231d0e38333d2a22d9698a011 \ + --hash=sha256:893ea9cf86cb8d2546812ac93d973aaf2ee1fb45110a873b014214fd23e3725e \ + --hash=sha256:8afd9bf35cc6a1f22eb3634808fa8e0b91902459c5721ef2e4461dfe771d7f08 \ + --hash=sha256:8be099e979fc42559328a21828281b4578304191ae46ed4e80a407048a82eee6 \ + --hash=sha256:8e209591f7c41ae4a9171335cf6156afda0b21de73b02f73f5aa95b2d5fbb08d \ + --hash=sha256:8fc15cc8d0d06e873c00ef18e1372d605f9aaf3de27d8c24e50782e75bc8b843 \ + --hash=sha256:9174f0af24e5eff248b9dbfe76ec5275a3d19d37edbc2810543f12cf97347a34 \ + --hash=sha256:921415102a90637fcc2e3f169f61dad7699ecf690e8639fc21b813acbedc0967 \ + --hash=sha256:967d72c835d7a8cf0af99ec813a2d06e3db6df706402f1fe85b31b437645f495 \ + --hash=sha256:98d9c97f51b334b0adce7b964442a9af33c1a00c6ac856984cc5dc8d18f81c75 \ + --hash=sha256:99704f73721e23859112072d522076e11c31744fc96b5652e5dd2018aa4359f7 \ + --hash=sha256:9a75a4704ff640e46170042eec1f984385a121227c505d5a16ad8e495f452541 \ + --hash=sha256:9acc7f7ec4a1b5f89bd929fde5b8a714f6fafdc6cc18725413d510aa082b47ad \ + --hash=sha256:9c6afdd69218202bc1758c9a14b86b8cf1084f37ed2ca143e567a103772b16d1 \ + --hash=sha256:9cdf19874e0d247f32f03609200370343c3c7aa260b191d8c2bb251d36198283 \ + --hash=sha256:9e1d0ced76318bab499693ff25f64faa343415187cb2e4d7befdfdd391a1cf6a \ + --hash=sha256:9fd670ac43b709c575aefc25bf52d8a598a3bc5017bddfd0a179152ab06a2deb \ + --hash=sha256:a0f2285329dac10ab08f79cb11f5692c497018e6c7c511f95e6fd63a70b8f831 \ + --hash=sha256:a2fac6895eb299a2e52d7bbb8fb3903502b9da8d3f5309ceb16ec40c646b58ee \ + --hash=sha256:a336eec40e3520d369b8a6cdabb4f596e69a8b42927ca074aa1452fed943238a \ + --hash=sha256:a4624f80732f6b427ac58f1f59c577a0994a12e8174b5af6a027b4b58795d4c3 \ + --hash=sha256:a56ac4fa5a75c7e182e8f62600cfb4aff43c5ed7356a034f3557659c3bec1d90 \ + --hash=sha256:a678c0b6b22086ec2427359d22e37445d4a792f5fdbbc744112c7dade65cad02 \ + --hash=sha256:a740ea6f083c6db7b926534d159508f80ba275ab35e722522de0d18d0f56e55f \ + --hash=sha256:a90700f743e29aa3d75a6ff5f01953176a889c00e526194bc4d281731b88d99d \ + --hash=sha256:a9a638be322a8d76a41cdb17781c7f82aaee6a66493d8ffb7e2c09ee22423d99 \ + --hash=sha256:a9cd3de0a5bfe7b0e21ee10e1a14e3d61bf52efc88217ab1d95d6ace6970bd46 \ + --hash=sha256:aa62c85046473959c13ba9edca9dc90a77d5c1095b1ba313556314d77fe5b036 \ + --hash=sha256:aba5c63b7afdc749cc9eae943d5b868cba2b261a176378fa1c5a30bc8bc89982 \ + --hash=sha256:ac0f3b379c94acc2f7dce5f5f0b24d44fa1cc6a509717ef83dfee07450c2117c \ + --hash=sha256:af2a2a8c7c74de0559e0c368d94c8def9e16c58faaee33a0bf081057c4227e3b \ + --hash=sha256:af98ad5ed9d6daaca956201e00bb429a7eb2b080426686f70a20353e0f9839f5 \ + --hash=sha256:afdf43b72ef3876c1fe66423b91466e37877c9e81e8cec70542b7e8525b9d1b7 \ + --hash=sha256:b88841e654f09732804809e435b3e005a929ffd9998b872b7b213957b8759cb8 \ + --hash=sha256:bb2fc905bbf4e6b7f40806ea79e31515abf6349594cdf0adf27c4215f0463204 \ + --hash=sha256:bb4ffe96aa663cee727659db5a2afeb38c95f8677b747d447b90d6d4874ea2c5 \ + --hash=sha256:bc0b0ac781d489304b741269857f1f8338b7a26b1b89c06c0344658001ec0035 \ + --hash=sha256:bf1bd822ec4e387ed245bed0d71151582cf7be9e5309bc4145eefe36083d5878 \ + --hash=sha256:c19cd6d025c1673f22afcd22c7df8a662d779e05d8e3fa6820c22afb895b0206 \ + --hash=sha256:c3305c38a2fa21a4254f2ace7dd9ef5fc569c9a558b66e7017650b3d637fb95e \ + --hash=sha256:c85d54e7e8a2ca932fe8399301af9b8d5907ea2a455ffaff6e7d1208db83b943 \ + --hash=sha256:ca64d9f1f384f151b9511bec01126072acd2f313439f8ed015a22d8790aab6fa \ + --hash=sha256:cce2bc991293f15cc4084ca116827b5900c5f34e1a54dfe83f10ab5c43162eb7 \ + --hash=sha256:d6276d78f6fca7d0ac066d5da4165c5acd07829e8305c2cb900b738fb3a75a72 \ + --hash=sha256:d93db87adb6b1c1b408dce4763314b55d76a9f589e96783a84ac9e7689e48bdf \ + --hash=sha256:db5f8394e17f877a625b257f2ba0ce8e728a499c2c1579ad66220272cd3df510 \ + --hash=sha256:db76506aa5416081f3e8974ae0f7965c58ada0bb0ef7339ac86099588dbb20d3 \ + --hash=sha256:dba2edfb054f6d4a08df9d1637c39a5aa3865bca6617c13c86be21e45658a59c \ + --hash=sha256:dcf4bc2aab4e16b1c4c0c2005918f23a7dd5d7821ddae82caed9e3342dc2fcce \ + --hash=sha256:e1fa594c887365b69745f25a416806e61085dd07b94c9eae68a6e20730629b23 \ + --hash=sha256:e6c52d3307824ff93b39efd99e4185d557db40bd841452abfb32e5d9151ca162 \ + --hash=sha256:eb57acff4a74246ae513c142d4b36e18c389c3aed8661914a53f7cd0071031b2 \ + --hash=sha256:f80bd9f9633eafc73d0a913ba2645c96ba58bba1befc30590f7c0fbfde59d865 \ + --hash=sha256:f8475460aa33ee28ac896ab1156d0bb3b6c639f7f8383c2677d3359eb35f8205 \ + --hash=sha256:fb2bde05838fffae1a1bf75e5d411a6cac3e4e9bb97e6640fed8cd47888b33f0 \ + --hash=sha256:fb9d92ecfe2d5b494367c67f7446f8b75b68d8d0c8cf3bc3e6997478be25d9e2 \ + --hash=sha256:fd3d72233eb8b48acc94fa57d44e2d32ce8e7abed02882ccb6d855ccc4ed33ec + # via pytest-cov +cryptography==50.0.2 \ + --hash=sha256:0ddc924c04591c2811ca024d62ecad4f7f6f08af8939c211438f48a16bd23602 \ + --hash=sha256:0ec5f09541743261e66e291b4a0cbf0fb2997aeaab6d9e9c740b9dba1b58d1c2 \ + --hash=sha256:0ecbc5652bdb6fc9eaf89a7d196e20941adfe812f43bc4ca05d9150496821047 \ + --hash=sha256:1981f1db4630889b9ef7803fadef12b056f428cb6b85c27ba57b774793b6093c \ + --hash=sha256:1ba34f04897fcdaa73f74145c25f3ec146fbd56593853e88adc2e811303c5f42 \ + --hash=sha256:241449bf940a5d27309bd317e6f9a2af6932113818bb2b8f5c59ddc7ef16da18 \ + --hash=sha256:25784ce8b9621c90c643efb9e1e2162ab3b0224cae446ad5e70e7fcb1ce18b51 \ + --hash=sha256:3dc4fd8058cea1644971207d530e1a03a184a805ffc8ebdddf0599d78a331b81 \ + --hash=sha256:4061c0079120205fb760c58acab6443e217307dcf05e3702cf970e0689972856 \ + --hash=sha256:4a20ce1e5cb4284a86692fdcba7cb8754185c6b2e5c56fcef3751cf451d3cdc2 \ + --hash=sha256:4e81d95e5bafc2d6e34e4bed780e53e4d5b9a2f928573428aa4d35fbec1eb0de \ + --hash=sha256:58a0c478eeca76fe5e07993c5a0703def34a6dc6a0cda4f5564639b33112ffe7 \ + --hash=sha256:58ddb5a8e3179d12f19e4ea34d2d32e9d63a4baa142c875c1eb59f41b7243acd \ + --hash=sha256:630ebfea3bf689d075f82316324ff7433dc447fe6bc1bfc76524b74b4a9567d2 \ + --hash=sha256:6f8700550aa1474a91e5dc07049c46f98b423b5b1ddd0483e0b51362eeeaf5be \ + --hash=sha256:78198641e5be9521beea5aa782bb551a58068d10e6eb04c9c680c1b69f2e7d45 \ + --hash=sha256:79def8d059362e7831389ed3be0ecdf58a89386e1271e35dd9f5af84e81bffd0 \ + --hash=sha256:7a8701d6b584d76e909e3d305b7d126b41439876a5aaf76cddc67fc230eafa2e \ + --hash=sha256:7afa5a6602a9f29af1f3a2965f831bae7c9d5d597b7cbb716d41ab3b7d89879c \ + --hash=sha256:7b46165bb56eb4704e2eaaf86f3c940d19154535d9b0ca7d6d590b04060e00d5 \ + --hash=sha256:7b75de3c8b3be1cdb1052747c929440c3eea46c1bc2cb8a6e3a48388e9b7b452 \ + --hash=sha256:7c6d0330c472d96f6a6afe24d80dfdf15176c33096f0a4397ae4c60f3dd3be48 \ + --hash=sha256:828d49b0ff5a0e3975865571c5d91dbbdd0d38d8289b249a163e9425413a5e05 \ + --hash=sha256:84f964e537f916e2cc85199e5a88742e964939b575ac8598b3f9d6cc416cdaf1 \ + --hash=sha256:85d0d9a31b9098e98534226d5686b47264b95e62ce459dc2e62fdfc809f9fe93 \ + --hash=sha256:87e9ce85beb6b328ba370cc6e6aea483c92617b4c95b1d33a49297eb662bfb04 \ + --hash=sha256:8c71ba2cd31fc93748c38e1b613200ff1c2665cbfd5341fe3a61cfde35a1430e \ + --hash=sha256:92e665960f25fcdc73725b9cec7a3824f279ba97a98653afe9ffac2e43668f67 \ + --hash=sha256:94e5e9f108ee10471288214d3d233fbfbb492840a8457eb85178d643ddeb32c7 \ + --hash=sha256:9c8402a82ea0dc4ceeab793db05f0fafa8ca139ca34fcde5df0f596103c74107 \ + --hash=sha256:9dab55f57c74c3cad24c323bacbbd04be4705ba6eb0d92e920b1fc4837ed5079 \ + --hash=sha256:a582ab2ae1d34f67112cadc86702774c9ea4374df6bca6afe672817203c99134 \ + --hash=sha256:a6557e5f38e065ca9fbdaf7cfc7435ecb1d113aa81a022d1b51921ee7432e227 \ + --hash=sha256:a9f7355e6fab51f6c369b86fb7571cffa05edee2c2121e0380a37fb9ac1cd5c1 \ + --hash=sha256:ab50ee449bf968271e820086f10a33d101dd060370abc10bcd22279be2656539 \ + --hash=sha256:ac9ed99d81760c62fe89d5f0815cdfa1ba9a35141cf30f1c2d044f04b4803d2e \ + --hash=sha256:b13478603dcd0a2479ff8e87e2c19a7d525734686fe3c49542472293a204212d \ + --hash=sha256:c423ab384a46c4dff7217b2ea5ba2e11cffdeab6441acd04cf65a369caf0366c \ + --hash=sha256:c5e67125c7dca78d199ec4e116aa93dbb83494808ecbb8211a2cb09b1bf41dbd \ + --hash=sha256:c71be1cbfa5cd9a41ee452acf1eccd82b2c05950358b106ec8ceb83411d1a020 \ + --hash=sha256:cbc8738fd8526d80f35cb3a40d41f41a2e7030bb3b18b09a6778ef63d291c2fd \ + --hash=sha256:ce47f66801c20ec6c6632453bb5960fe38939e9306970b48b3a5a26de7745d94 \ + --hash=sha256:d370b8d1dfcdf7130178137f6fbee6140774a1acc6cacefc4b42643ec11d0a3a \ + --hash=sha256:d38cdff612d06fa6a32840d5e1b1f7a27cee4a349aa9085d94a67789d6bfd408 \ + --hash=sha256:d8947001be83df1394050758ce0e745dd74fb134eef0a4b5124208dfc3a68c37 \ + --hash=sha256:deb9fde5c60e437ee4821bc9bc39ff31b42135c27e1dc61ef0a629389c1de62e \ + --hash=sha256:dfe9763530994147d9af1def057a5b9658b00e8f8fe8743d144d1e0911c2e454 \ + --hash=sha256:e105ab60406787da31fccc883fc0f733af1efd78f0136a4599692c4083a73d0c \ + --hash=sha256:e275096ea1e60cc595cda2836fd4a6c725d1125108b868be17f53684d164e2cc \ + --hash=sha256:edc3342adf8f697fc5f59c887a304356f147b397809440ed64e2fa6af2f50f37 \ + --hash=sha256:ee247f5c245c9a2fe7c8e2214e295918838e44e00a45a6718451e4004219e767 \ + --hash=sha256:eef4c2f3423810b3070ab391f85436d2f8bbfcb286ac15cbc73190b3563b1f1a \ + --hash=sha256:f21e8a22c8605750c7af886bab299a363721264061b4ac0a30efb73cfd58efc5 \ + --hash=sha256:f265528741e048bce55c3463ed721fb0aa45a5888d8add8cfeccb3035451bbdc \ + --hash=sha256:f2f9bd7f90c64fe89253f0a2c05e3c4856072660429ce8831b4235bf29403a67 \ + --hash=sha256:f785f6161f202ab04d8ca194158968798e480ca058943907972da5f12e2881e8 \ + --hash=sha256:f9f6143a8c75945eb960d9eb98905a441394abfa24afaae239d514ffb2586480 \ + --hash=sha256:fa8f5efb344d6908a1ce62f4a24e2e5780f825d6f53f5f50ec5ffacac72936cb \ + --hash=sha256:fdd28f912fccfec1846a94e2e1e8f9b0012f557f0c46fe4f3eb0d7a87afcf90b + # via + # -r FileAutomation/.github/requirements/integration.in + # azure-storage-blob + # google-auth + # msal + # paramiko + # pyjwt + # pyspnego + # smbprotocol +defusedxml==0.7.1 \ + --hash=sha256:1bb3032db185915b62d7c6209c5a8792be6a32ab2fedacc84e01b52c51aa3e69 \ + --hash=sha256:a352e7e428770286cc899e2542b6cdaedb2b4953ff269a210103ec58f6198a61 + # via -r FileAutomation/.github/requirements/integration.in +dropbox==12.2.2 \ + --hash=sha256:044d963cdee83a5149e103c792b1a0284f681bb1b6984259cda01244cc3c1a6e \ + --hash=sha256:9abc636a57788165c1e1f263971163230ccc26fd448df643c1005e0dc1abd1a3 + # via -r FileAutomation/.github/requirements/integration.in +fsspec==2026.9.0 \ + --hash=sha256:0f08147951c8cb31d844c3547d631053b127863b60be04cf06e121333ee0e2fe \ + --hash=sha256:8dd6e646e99ea382bd85f97a45e6b526a442d79423a7dc673f1e2756d05fcb5f + # via -r FileAutomation/.github/requirements/integration.in +google-api-core==2.40.0 \ + --hash=sha256:4b9e0a80024c269ae173136d5439f4ed284651d6e5e2773ba5c694f481f9c0f4 \ + --hash=sha256:ebee7d1b138b5362beecec260e6e8988ac97346562c7382ccb6f0ad8435c599f + # via google-api-python-client +google-api-python-client==2.201.0 \ + --hash=sha256:2d9bf1ba3f12eee8ed3d0f1791ce0605d163432f496baa72d3677faa2cf097d6 \ + --hash=sha256:d5691982abd7287f53cb0b0e0c6a9984d4103cf864ea0a88cb6e4347bbaf70de + # via -r FileAutomation/.github/requirements/integration.in +google-auth==2.59.1 \ + --hash=sha256:89c3f931683a482ac97e61df7eb9da5e08a91703f3c752b2377d72cfb7d69e6a \ + --hash=sha256:ce50fc533ac02f489a2b183a0c156672c376ecb2091b1127bc7efba2975fff27 + # via + # google-api-core + # google-api-python-client + # google-auth-httplib2 + # google-auth-oauthlib +google-auth-httplib2==0.4.4 \ + --hash=sha256:b931de392c20cfaa351cd789274922bd8cdc001e0e9e96de31b39d71347f8e16 \ + --hash=sha256:bbe5d7b2401bb3a4017f4720e1e91bd273ab9a2bb60b84e65edbc0de127852da + # via + # -r FileAutomation/.github/requirements/integration.in + # google-api-python-client +google-auth-oauthlib==1.5.0 \ + --hash=sha256:71625fdea21c6c03217eb9ff13741c3096e6d257ebca0ad084dbd6e9f4aa4ef4 \ + --hash=sha256:b351107c7dd9017f426cbb0272ea1bc04f469020fd18f4443c7d40362e0b1510 + # via -r FileAutomation/.github/requirements/integration.in +googleapis-common-protos==1.75.5 \ + --hash=sha256:c7a866fc34ed29a3b10af627a4b9b1dc2433313ca6e959f0ae4feb132047ed72 \ + --hash=sha256:d7285525c23039db98f2463e6d5a4f9b958b94d497f03a844ece3259c4e72d5d + # via google-api-core +httplib2==0.32.0 \ + --hash=sha256:48a0ef30a42db65d8f3399045e1d09ab0ba66e3b9efc360d07f80ea55d286025 \ + --hash=sha256:dc6705cacdf3fb0a2aba7629fa33c90fd93e30035db0c157325826be177e4816 + # via + # google-api-python-client + # google-auth-httplib2 +idna==3.20 \ + --hash=sha256:a7db850025b95ded1eae8a46181a1a6c56c92c96f0e2b005d9ff8dc0210cab44 \ + --hash=sha256:ab7ae7122974553370f0bdb919e1a960b2cd1bc1ef0276416d896db81c14582c + # via requests +iniconfig==2.3.0 \ + --hash=sha256:c76315c77db068650d49c5b56314774a7804df16fee4402c1f19d6d15d8c4730 \ + --hash=sha256:f631c04d2c48c52b84d0d0549c99ff3859c98df65b3101406327ecc7d53fbf12 + # via pytest +invoke==3.0.3 \ + --hash=sha256:437b6a622223824380bfb4e64f612711a6b648c795f565efc8625af66fb57f0c \ + --hash=sha256:f11327165e5cbb89b2ad1d88d3292b5113332c43b8553b494da435d6ec6f5053 + # via paramiko +isodate==0.7.2 \ + --hash=sha256:28009937d8031054830160fce6d409ed342816b543597cece116d966c6d99e15 \ + --hash=sha256:4cd1aa0f43ca76f4a6c6c0292a85f40b35ec2e43e315b59f06e6d32171a953e6 + # via azure-storage-blob +je-action-core==0.0.3 \ + --hash=sha256:395890665483e58f17fede17aa809bf6e2762ba8f0bd6751126fe46efabb8c85 \ + --hash=sha256:7b73ab0172a3c90c91b0a81b8a662350199a35842f529acf7a54706052fb38ee + # via -r FileAutomation/.github/requirements/integration.in +jinja2==3.1.6 \ + --hash=sha256:0137fb05990d35f1275a587e9aee6d56da821fc83491a0fb838183be43f66d6d \ + --hash=sha256:85ece4451f492d0c13c5dd7c13a64681a86afae63a5f347908daf103ce6d2f67 + # via stone +jmespath==1.1.0 \ + --hash=sha256:472c87d80f36026ae83c6ddd0f1d05d4e510134ed462851fd5f754c8c3cbb88d \ + --hash=sha256:a5663118de4908c91729bea0acadca56526eb2698e83de10cd116ae0f4e97c64 + # via + # boto3 + # botocore +markupsafe==3.0.3 \ + --hash=sha256:0303439a41979d9e74d18ff5e2dd8c43ed6c6001fd40e5bf2e43f7bd9bbc523f \ + --hash=sha256:068f375c472b3e7acbe2d5318dea141359e6900156b5b2ba06a30b169086b91a \ + --hash=sha256:0bf2a864d67e76e5c9a34dc26ec616a66b9888e25e7b9460e1c76d3293bd9dbf \ + --hash=sha256:0db14f5dafddbb6d9208827849fad01f1a2609380add406671a26386cdf15a19 \ + --hash=sha256:0eb9ff8191e8498cca014656ae6b8d61f39da5f95b488805da4bb029cccbfbaf \ + --hash=sha256:0f4b68347f8c5eab4a13419215bdfd7f8c9b19f2b25520968adfad23eb0ce60c \ + --hash=sha256:1085e7fbddd3be5f89cc898938f42c0b3c711fdcb37d75221de2666af647c175 \ + --hash=sha256:116bb52f642a37c115f517494ea5feb03889e04df47eeff5b130b1808ce7c219 \ + --hash=sha256:12c63dfb4a98206f045aa9563db46507995f7ef6d83b2f68eda65c307c6829eb \ + --hash=sha256:133a43e73a802c5562be9bbcd03d090aa5a1fe899db609c29e8c8d815c5f6de6 \ + --hash=sha256:1353ef0c1b138e1907ae78e2f6c63ff67501122006b0f9abad68fda5f4ffc6ab \ + --hash=sha256:15d939a21d546304880945ca1ecb8a039db6b4dc49b2c5a400387cdae6a62e26 \ + --hash=sha256:177b5253b2834fe3678cb4a5f0059808258584c559193998be2601324fdeafb1 \ + --hash=sha256:1872df69a4de6aead3491198eaf13810b565bdbeec3ae2dc8780f14458ec73ce \ + --hash=sha256:1b4b79e8ebf6b55351f0d91fe80f893b4743f104bff22e90697db1590e47a218 \ + --hash=sha256:1b52b4fb9df4eb9ae465f8d0c228a00624de2334f216f178a995ccdcf82c4634 \ + --hash=sha256:1ba88449deb3de88bd40044603fafffb7bc2b055d626a330323a9ed736661695 \ + --hash=sha256:1cc7ea17a6824959616c525620e387f6dd30fec8cb44f649e31712db02123dad \ + --hash=sha256:218551f6df4868a8d527e3062d0fb968682fe92054e89978594c28e642c43a73 \ + --hash=sha256:26a5784ded40c9e318cfc2bdb30fe164bdb8665ded9cd64d500a34fb42067b1c \ + --hash=sha256:2713baf880df847f2bece4230d4d094280f4e67b1e813eec43b4c0e144a34ffe \ + --hash=sha256:2a15a08b17dd94c53a1da0438822d70ebcd13f8c3a95abe3a9ef9f11a94830aa \ + --hash=sha256:2f981d352f04553a7171b8e44369f2af4055f888dfb147d55e42d29e29e74559 \ + --hash=sha256:32001d6a8fc98c8cb5c947787c5d08b0a50663d139f1305bac5885d98d9b40fa \ + --hash=sha256:3524b778fe5cfb3452a09d31e7b5adefeea8c5be1d43c4f810ba09f2ceb29d37 \ + --hash=sha256:3537e01efc9d4dccdf77221fb1cb3b8e1a38d5428920e0657ce299b20324d758 \ + --hash=sha256:35add3b638a5d900e807944a078b51922212fb3dedb01633a8defc4b01a3c85f \ + --hash=sha256:38664109c14ffc9e7437e86b4dceb442b0096dfe3541d7864d9cbe1da4cf36c8 \ + --hash=sha256:3a7e8ae81ae39e62a41ec302f972ba6ae23a5c5396c8e60113e9066ef893da0d \ + --hash=sha256:3b562dd9e9ea93f13d53989d23a7e775fdfd1066c33494ff43f5418bc8c58a5c \ + --hash=sha256:457a69a9577064c05a97c41f4e65148652db078a3a509039e64d3467b9e7ef97 \ + --hash=sha256:4bd4cd07944443f5a265608cc6aab442e4f74dff8088b0dfc8238647b8f6ae9a \ + --hash=sha256:4e885a3d1efa2eadc93c894a21770e4bc67899e3543680313b09f139e149ab19 \ + --hash=sha256:4faffd047e07c38848ce017e8725090413cd80cbc23d86e55c587bf979e579c9 \ + --hash=sha256:509fa21c6deb7a7a273d629cf5ec029bc209d1a51178615ddf718f5918992ab9 \ + --hash=sha256:5678211cb9333a6468fb8d8be0305520aa073f50d17f089b5b4b477ea6e67fdc \ + --hash=sha256:591ae9f2a647529ca990bc681daebdd52c8791ff06c2bfa05b65163e28102ef2 \ + --hash=sha256:5a7d5dc5140555cf21a6fefbdbf8723f06fcd2f63ef108f2854de715e4422cb4 \ + --hash=sha256:69c0b73548bc525c8cb9a251cddf1931d1db4d2258e9599c28c07ef3580ef354 \ + --hash=sha256:6b5420a1d9450023228968e7e6a9ce57f65d148ab56d2313fcd589eee96a7a50 \ + --hash=sha256:722695808f4b6457b320fdc131280796bdceb04ab50fe1795cd540799ebe1698 \ + --hash=sha256:729586769a26dbceff69f7a7dbbf59ab6572b99d94576a5592625d5b411576b9 \ + --hash=sha256:77f0643abe7495da77fb436f50f8dab76dbc6e5fd25d39589a0f1fe6548bfa2b \ + --hash=sha256:795e7751525cae078558e679d646ae45574b47ed6e7771863fcc079a6171a0fc \ + --hash=sha256:7be7b61bb172e1ed687f1754f8e7484f1c8019780f6f6b0786e76bb01c2ae115 \ + --hash=sha256:7c3fb7d25180895632e5d3148dbdc29ea38ccb7fd210aa27acbd1201a1902c6e \ + --hash=sha256:7e68f88e5b8799aa49c85cd116c932a1ac15caaa3f5db09087854d218359e485 \ + --hash=sha256:83891d0e9fb81a825d9a6d61e3f07550ca70a076484292a70fde82c4b807286f \ + --hash=sha256:8485f406a96febb5140bfeca44a73e3ce5116b2501ac54fe953e488fb1d03b12 \ + --hash=sha256:8709b08f4a89aa7586de0aadc8da56180242ee0ada3999749b183aa23df95025 \ + --hash=sha256:8f71bc33915be5186016f675cd83a1e08523649b0e33efdb898db577ef5bb009 \ + --hash=sha256:915c04ba3851909ce68ccc2b8e2cd691618c4dc4c4232fb7982bca3f41fd8c3d \ + --hash=sha256:949b8d66bc381ee8b007cd945914c721d9aba8e27f71959d750a46f7c282b20b \ + --hash=sha256:94c6f0bb423f739146aec64595853541634bde58b2135f27f61c1ffd1cd4d16a \ + --hash=sha256:9a1abfdc021a164803f4d485104931fb8f8c1efd55bc6b748d2f5774e78b62c5 \ + --hash=sha256:9b79b7a16f7fedff2495d684f2b59b0457c3b493778c9eed31111be64d58279f \ + --hash=sha256:a320721ab5a1aba0a233739394eb907f8c8da5c98c9181d1161e77a0c8e36f2d \ + --hash=sha256:a4afe79fb3de0b7097d81da19090f4df4f8d3a2b3adaa8764138aac2e44f3af1 \ + --hash=sha256:ad2cf8aa28b8c020ab2fc8287b0f823d0a7d8630784c31e9ee5edea20f406287 \ + --hash=sha256:b8512a91625c9b3da6f127803b166b629725e68af71f8184ae7e7d54686a56d6 \ + --hash=sha256:bc51efed119bc9cfdf792cdeaa4d67e8f6fcccab66ed4bfdd6bde3e59bfcbb2f \ + --hash=sha256:bdc919ead48f234740ad807933cdf545180bfbe9342c2bb451556db2ed958581 \ + --hash=sha256:bdd37121970bfd8be76c5fb069c7751683bdf373db1ed6c010162b2a130248ed \ + --hash=sha256:be8813b57049a7dc738189df53d69395eba14fb99345e0a5994914a3864c8a4b \ + --hash=sha256:c0c0b3ade1c0b13b936d7970b1d37a57acde9199dc2aecc4c336773e1d86049c \ + --hash=sha256:c47a551199eb8eb2121d4f0f15ae0f923d31350ab9280078d1e5f12b249e0026 \ + --hash=sha256:c4ffb7ebf07cfe8931028e3e4c85f0357459a3f9f9490886198848f4fa002ec8 \ + --hash=sha256:ccfcd093f13f0f0b7fdd0f198b90053bf7b2f02a3927a30e63f3ccc9df56b676 \ + --hash=sha256:d2ee202e79d8ed691ceebae8e0486bd9a2cd4794cec4824e1c99b6f5009502f6 \ + --hash=sha256:d53197da72cc091b024dd97249dfc7794d6a56530370992a5e1a08983ad9230e \ + --hash=sha256:d6dd0be5b5b189d31db7cda48b91d7e0a9795f31430b7f271219ab30f1d3ac9d \ + --hash=sha256:d88b440e37a16e651bda4c7c2b930eb586fd15ca7406cb39e211fcff3bf3017d \ + --hash=sha256:de8a88e63464af587c950061a5e6a67d3632e36df62b986892331d4620a35c01 \ + --hash=sha256:df2449253ef108a379b8b5d6b43f4b1a8e81a061d6537becd5582fba5f9196d7 \ + --hash=sha256:e1c1493fb6e50ab01d20a22826e57520f1284df32f2d8601fdd90b6304601419 \ + --hash=sha256:e1cf1972137e83c5d4c136c43ced9ac51d0e124706ee1c8aa8532c1287fa8795 \ + --hash=sha256:e2103a929dfa2fcaf9bb4e7c091983a49c9ac3b19c9061b6d5427dd7d14d81a1 \ + --hash=sha256:e56b7d45a839a697b5eb268c82a71bd8c7f6c94d6fd50c3d577fa39a9f1409f5 \ + --hash=sha256:e8afc3f2ccfa24215f8cb28dcf43f0113ac3c37c2f0f0806d8c70e4228c5cf4d \ + --hash=sha256:e8fc20152abba6b83724d7ff268c249fa196d8259ff481f3b1476383f8f24e42 \ + --hash=sha256:eaa9599de571d72e2daf60164784109f19978b327a3910d3e9de8c97b5b70cfe \ + --hash=sha256:ec15a59cf5af7be74194f7ab02d0f59a62bdcf1a537677ce67a2537c9b87fcda \ + --hash=sha256:f190daf01f13c72eac4efd5c430a8de82489d9cff23c364c3ea822545032993e \ + --hash=sha256:f34c41761022dd093b4b6896d4810782ffbabe30f2d443ff5f083e0cbbb8c737 \ + --hash=sha256:f3e98bb3798ead92273dc0e5fd0f31ade220f59a266ffd8a4f6065e0a3ce0523 \ + --hash=sha256:f42d0984e947b8adf7dd6dde396e720934d12c506ce84eea8476409563607591 \ + --hash=sha256:f71a396b3bf33ecaa1626c255855702aca4d3d9fea5e051b41ac59a9c1c41edc \ + --hash=sha256:f9e130248f4462aaa8e2552d547f36ddadbeaa573879158d721bbd33dfe4743a \ + --hash=sha256:fed51ac40f757d41b7c48425901843666a6677e3e8eb0abcff09e4ba6e664f50 + # via jinja2 +msal==1.39.0 \ + --hash=sha256:2d2577886906cd7293850dffa2da29119966c213bfc6ec0cecf8bf7621e1ca77 \ + --hash=sha256:6ab7de335e6d7f5717e2c7e1dbf86e4dda2f6acf3c56773b78dc53ebc6395b5f + # via -r FileAutomation/.github/requirements/integration.in +oauthlib==4.0.0 \ + --hash=sha256:624c28c13a0a59cabf9747dfa52af63be3e512a7f2714df16e91b5b3a145e6cd \ + --hash=sha256:efb274799819440f95b4ab3b818869f1ce9ae26c5beacba0201d1a1b76b54f86 + # via requests-oauthlib +opentelemetry-api==1.45.0 \ + --hash=sha256:711ede81773c8025c2c03dac0450bc89f3d30aea6eabcc815c570d4e35a963f7 \ + --hash=sha256:80e068aba7cd56c8b58512d6a36f8d25cb1dfaa0c0a4cc1c938ccf9f362d9cb3 + # via + # -r FileAutomation/.github/requirements/integration.in + # google-api-core + # opentelemetry-sdk + # opentelemetry-semantic-conventions +opentelemetry-sdk==1.45.0 \ + --hash=sha256:20caa5130505e386c67c3da1c76e446c842698ced54c76c6148679539aa97972 \ + --hash=sha256:5dc634c946546f61b757c5b1781f9fd1a10e96ffaf7357c797e508fe9c57e75e + # via -r FileAutomation/.github/requirements/integration.in +opentelemetry-semantic-conventions==0.66b0 \ + --hash=sha256:175b19dd98c4473f4f43a2b1df59186fd7b4a48cd3f77cbe03438b6d6fda230a \ + --hash=sha256:97a77dce484c54861e7eeff7651fd8a806dd3c30e501dc316730215ec36890e6 + # via opentelemetry-sdk +packaging==26.3 \ + --hash=sha256:94edc256424af38762eb31306eed28beb9f0efc50a8837492c9d6fd6004aed79 \ + --hash=sha256:d7193f7c8e4e93f444fde0262bf90af30e16fa0ad0ad44cb553c87339b23cd1c + # via + # pytest + # stone + # wheel +paramiko==5.0.0 \ + --hash=sha256:36763b5b95c2a0dcfdf1abc48e48156ee425b21efe2f0e787c2dd5a95c0e5e79 \ + --hash=sha256:b7044611c30140d9a75261653210e2002977b71a0497ff3ba0d98d7edbf62f7c + # via -r FileAutomation/.github/requirements/integration.in +pluggy==1.6.0 \ + --hash=sha256:7dcc130b76258d33b90f61b658791dede3486c3e6bfb003ee5c9bfb396dd22f3 \ + --hash=sha256:e920276dd6813095e9377c0bc5566d94c932c33b27a3e3945d8389c374dd4746 + # via + # pytest + # pytest-cov +prometheus-client==0.26.0 \ + --hash=sha256:04a91bcf94e2cf74a44a1a874d651a2e853ed354b6e822f3b7487751465d5c2b \ + --hash=sha256:fa93d06737aa02bacd05794768508bb97d2fbee28cb3bca04eaae92f0ca953d6 + # via -r FileAutomation/.github/requirements/integration.in +proto-plus==1.29.0 \ + --hash=sha256:8acd070469a7aaf43f440b022ef9757c8cac1a9f866e933f59ae98669ddc6c8b \ + --hash=sha256:cfb4e62ad7e13dd18f346cabbda00cab39930d36a05791fd81ddb074d6ee884f + # via google-api-core +protobuf==7.36.2 \ + --hash=sha256:497d0463ff3316681da6c0b9e8d06cb465d61abce00b613ab42226175644d1bb \ + --hash=sha256:89f23aa53c24553a2416fd4fd1ec06f74fa42b14b546d8883128813f775bbfd2 \ + --hash=sha256:912c1221170e16c08d1f086762f563dd61ff83c18b5fa6652952dfaded66f728 \ + --hash=sha256:a300819d441e078a5608c0d3c709796bb548136058fda017ae51d425b44fd353 \ + --hash=sha256:bdb3a345d48db958e6ce1f18e508beb0cc981d64f24088427549c866cd039f1e \ + --hash=sha256:cbc70b17ee27e28894c7fee8bb04be1abead49e936bc70eb60052531eee2079e \ + --hash=sha256:e11e1f0180583a2af89db6a2ecd9e8dc40aa6d2988ca175bfd0e6d12ea72d74e \ + --hash=sha256:f4fee11ec330d238b34a05c9b675f693c20415d1c5bd7d5320cc2f8a798eb9cf + # via + # google-api-core + # googleapis-common-protos + # proto-plus +pyarrow==25.0.1 \ + --hash=sha256:0b1edbb2f385a6a65e9711b62ba86ac54a7816a3f8d17bb3e8a5929d65fb2485 \ + --hash=sha256:0b726ad7e7b669be982b0c71c07fe4b037d654354130da79a7902a669e93a66b \ + --hash=sha256:0befcf816e45a1af33ac775a9970b749e4868a230c7372f0ae5e932bee27039f \ + --hash=sha256:0fe7c8b6c03969b49c8c66182e4a18e3819ab92d07cfab5d8370c531b9369ef0 \ + --hash=sha256:119297a6dc197e45d9c6d4415f7814a67ffa36c180d26f68c154c58067ae782d \ + --hash=sha256:169d3429d5be7c752125890620f75a60776d38b0035eddae939651640822332e \ + --hash=sha256:25f8720bf6387d5dc2ebd2622112de630760419e4b66134405dd24110d15f37e \ + --hash=sha256:31e49a7888fcdf3a835da33ae777f6bb9a866334e5a789282fc26dcf426f7f15 \ + --hash=sha256:35935cd5de130aa5cf4dea052a63e6bf2e17006c35c3a468194242b9b2bf5956 \ + --hash=sha256:38a9a4b4b9613380e200641891495a56c3d5a98a092db4a870af9975e220471d \ + --hash=sha256:3f89685964f46e4216103c75483aac0c0692a5f72212d7ca835adba5ede56ce3 \ + --hash=sha256:4288f27577352d608ca08553b0865e4a9b3aa14820c5d95b53337218d609835b \ + --hash=sha256:4340f0ba6c1d2e13f21658de1d7c662ca2545018568d0030a1e9afca159d87e3 \ + --hash=sha256:44a9120ce5bd81936b8ab9a88076e3fd47c2c6838e0e43630fed83626aca81d9 \ + --hash=sha256:4facd65742a024a4a366328a1d2292062d72d6e023c1b7dda8d4c37544933a25 \ + --hash=sha256:51093dd9e10325fbdb3c10a2ae7c4806e5c822d94e74ae4938b26524a3323fee \ + --hash=sha256:514ddb60285631af068875550c90eddc181db3e8e63a032b1559be189e82f056 \ + --hash=sha256:5389cdf79447ed1515c9e31620e6e1e2302249564d603f2ad727d4f6d313e4c3 \ + --hash=sha256:59a2de54c0cbd954da861eee4d1d330f8e909c45b53455baef696380f2c55033 \ + --hash=sha256:60e89d8f13861a1f7f8d950fa54aebb8023b30734d0ac51ffa80beabe2df4bba \ + --hash=sha256:6109c94d8b9f3b17a041daca16cacb2f651ad8f1ef70a4232c2c0f37a23da2a8 \ + --hash=sha256:62cd0d785b8aa6675ee355f9fc02252a340f4441257c42674937826fd7594325 \ + --hash=sha256:6943e2fe7954d29d84de45d29d34c8dc36ce96570e67d89aa9976e650a4a9138 \ + --hash=sha256:6a1fdfc6659b6b19022f2e50627fb5cf7156a66c46bf4299379955cbe742382a \ + --hash=sha256:880523be3d29efcf83d3998835d206118ccf35e3871dbd2fb60408cf6b007a80 \ + --hash=sha256:8858d7bfc22e3f51529aeaa4077225029724623e4595dc9eff8c793935c34140 \ + --hash=sha256:9150a83248bfed9813ea3c3af74c3856c1984d444aa28e58bf7733b9750ddf6a \ + --hash=sha256:9171748cdf796972d85a4b60157c279913e242992e350c90c7450182a9838b2a \ + --hash=sha256:a4d6d5e9a3d1879a97c08ded0c797579b7965eafd0f0c26c30b45ccc06db939b \ + --hash=sha256:a4dd8bf99a8fac133efc0ed6a92f5fddbe2adba0d0f6dd720e39ba9855cea85c \ + --hash=sha256:aa0559502e1cd6254d6814614085dd9c5a3dd0419362978a936a3f68a9e5c3df \ + --hash=sha256:b7a296aac7a71fa0886c08e155ddb6c636a50013f801f6178daafa0f9e726188 \ + --hash=sha256:bddd0c4f7630c2a3ddf6347c1bdaa79d97bcf6bd445f9e60c816b7d77c85a5ae \ + --hash=sha256:bf0b672390cdcb640d7288f96b826d71ff4e9abb254a86c89890baf51a29cee6 \ + --hash=sha256:c7c534ec03c358a76ea3e505e74c1b6aef290af90c444dfd092dbfe23e755b85 \ + --hash=sha256:cab40b1edfef0262e0e5251aa2c58d75630f24d06dd7794480243acc001a1d7d \ + --hash=sha256:cc4aa407fde9fc660be3939e49ea31f50f3e9fec17c0ec63159f7711edd3efc9 \ + --hash=sha256:d51592cb7561e87877c506113e7adbf1342ab579e6c21f0ef44b8ba41cb74c80 \ + --hash=sha256:dda9470024204d7bbf2042b47c6e8a0e47a3eeb8e34405882dfaea6577e0c153 \ + --hash=sha256:df961f2e7ae9cf496459259d798652c70625f6c080650d6952f8c04053c58ee9 \ + --hash=sha256:eb6203482ff3746a5632303a7279ae0b5a304c46985b49ed1378cb350ea6728d \ + --hash=sha256:f3831aaa25c67a99f99dc8b05873cb9d64560390372e2aa197ce9dd4a3f06a44 \ + --hash=sha256:f729cfdbd36fd99d543b67a914d2de044c84ebe45be8b34902b299b608c15c8f + # via -r FileAutomation/.github/requirements/integration.in +pyasn1==0.6.4 \ + --hash=sha256:9c447d8431c947fe4c8febc4ed9e760bc29011a5b01e5c74b67025bd9fb8ce81 \ + --hash=sha256:deda9277cfd454080ec40b207fb6df82206a3a2688735233cdcd8d3d565f088b + # via pyasn1-modules +pyasn1-modules==0.4.2 \ + --hash=sha256:29253a9207ce32b64c3ac6600edc75368f98473906e8fd1043bd6b5b1de2c14a \ + --hash=sha256:677091de870a80aae844b1ca6134f54652fa2c8c5a52aa396440ac3106e941e6 + # via google-auth +pycparser==3.0 ; implementation_name != 'PyPy' and platform_python_implementation != 'PyPy' \ + --hash=sha256:600f49d217304a5902ac3c37e1281c9fe94e4d0489de643a9504c5cdfdfc6b29 \ + --hash=sha256:b727414169a36b7d524c1c3e31839a521725078d7b2ff038656844266160a992 + # via cffi +pygments==2.21.0 \ + --hash=sha256:2363c69b61c4a97c838da3b130dcd6468f4848992b21a82f2a63ec34377137d9 \ + --hash=sha256:610ca751c9bc2492b38eb9a38a7fbc93edbbb2d7182edaf34e66ae493dee5c8c + # via pytest +pyjwt==2.15.1 \ + --hash=sha256:42d59d631f7768a1028a64c7ff581a9bf7519804daf91fc5b6c56e30eec5e193 \ + --hash=sha256:4f259e80cdfb6b3fc18a7de51fd1ef9ec79652f25019bae68975ca2468a34df8 + # via msal +pynacl==1.6.2 \ + --hash=sha256:018494d6d696ae03c7e656e5e74cdfd8ea1326962cc401bcf018f1ed8436811c \ + --hash=sha256:04316d1fc625d860b6c162fff704eb8426b1a8bcd3abacea11142cbd99a6b574 \ + --hash=sha256:22de65bb9010a725b0dac248f353bb072969c94fa8d6b1f34b87d7953cf7bbe4 \ + --hash=sha256:26bfcd00dcf2cf160f122186af731ae30ab120c18e8375684ec2670dccd28130 \ + --hash=sha256:2fef529ef3ee487ad8113d287a593fa26f48ee3620d92ecc6f1d09ea38e0709b \ + --hash=sha256:320ef68a41c87547c91a8b58903c9caa641ab01e8512ce291085b5fe2fcb7590 \ + --hash=sha256:3bffb6d0f6becacb6526f8f42adfb5efb26337056ee0831fb9a7044d1a964444 \ + --hash=sha256:44081faff368d6c5553ccf55322ef2819abb40e25afaec7e740f159f74813634 \ + --hash=sha256:46065496ab748469cdd999246d17e301b2c24ae2fdf739132e580a0e94c94a87 \ + --hash=sha256:5811c72b473b2f38f7e2a3dc4f8642e3a3e9b5e7317266e4ced1fba85cae41aa \ + --hash=sha256:622d7b07cc5c02c666795792931b50c91f3ce3c2649762efb1ef0d5684c81594 \ + --hash=sha256:62985f233210dee6548c223301b6c25440852e13d59a8b81490203c3227c5ba0 \ + --hash=sha256:68be3a09455743ff9505491220b64440ced8973fe930f270c8e07ccfa25b1f9e \ + --hash=sha256:834a43af110f743a754448463e8fd61259cd4ab5bbedcf70f9dabad1d28a394c \ + --hash=sha256:8845c0631c0be43abdd865511c41eab235e0be69c81dc66a50911594198679b0 \ + --hash=sha256:8a66d6fb6ae7661c58995f9c6435bda2b1e68b54b598a6a10247bfcdadac996c \ + --hash=sha256:8b097553b380236d51ed11356c953bf8ce36a29a3e596e934ecabe76c985a577 \ + --hash=sha256:a84bf1c20339d06dc0c85d9aea9637a24f718f375d861b2668b2f9f96fa51145 \ + --hash=sha256:a9f9932d8d2811ce1a8ffa79dcbdf3970e7355b5c8eb0c1a881a57e7f7d96e88 \ + --hash=sha256:bc4a36b28dd72fb4845e5d8f9760610588a96d5a51f01d84d8c6ff9849968c14 \ + --hash=sha256:c8a231e36ec2cab018c4ad4358c386e36eede0319a0c41fed24f840b1dac59f6 \ + --hash=sha256:c949ea47e4206af7c8f604b8278093b674f7c79ed0d4719cc836902bf4517465 \ + --hash=sha256:d071c6a9a4c94d79eb665db4ce5cedc537faf74f2355e4d502591d850d3913c0 \ + --hash=sha256:d29bfe37e20e015a7d8b23cfc8bd6aa7909c92a1b8f41ee416bbb3e79ef182b2 \ + --hash=sha256:fe9847ca47d287af41e82be1dd5e23023d3c31a951da134121ab02e42ac218c9 + # via paramiko +pyparsing==3.3.3 \ + --hash=sha256:928ae7e20211f3b6f3915a72f06a0cfd29ab9d24279dd6346b6b1a7146397d36 \ + --hash=sha256:ece8c00a69cf01b45d0b1dedabb469c90d8caf996d4fda40f147627a122849a4 + # via httplib2 +pyside6==6.11.2 \ + --hash=sha256:0444ac71d0791a19bded35f6d9a94515941f23a6eb7098ca45a1b64a338be697 \ + --hash=sha256:13a3c79816879d9743672a669f1988f74459a78f89a97a2624fae4fe059ffa3a \ + --hash=sha256:3201d67e3c10be2eaedd3910ff0f02351eca7e88c95a291cde5e7f2f55ef207f \ + --hash=sha256:57fe867a6c93821a085d74e8a26f863fc2363516848181c466032595b33b1729 \ + --hash=sha256:dc6d03990489a5085842770718392ef5de36352d8f608d9a9fe03e60af4d7b66 + # via -r FileAutomation/.github/requirements/integration.in +pyside6-addons==6.11.2 \ + --hash=sha256:354c574a839f7d0751960ba03f33bec87c95e0f2796dca3287f54d03abe97dac \ + --hash=sha256:a2ca3c73e5f060d4aca49451abd9fba68dd0e1c66ff40cea3a1fe0415def579a \ + --hash=sha256:a8b00956925fbdeffc7052cf933506391289f35ac4f3f71a0a167ec195cd47b4 \ + --hash=sha256:d5606af369484b862b4d5741cfc179e19d2b8819c3e04d210b8188e0c9f85988 \ + --hash=sha256:f449ea4431da20e7b86752cca8d166f93434516fe417f981c27e5f8e1b554407 + # via pyside6 +pyside6-essentials==6.11.2 \ + --hash=sha256:77795c145202e65a78d88f7cd409d186e3ba23d159bdb3ba2dcd159ae5e5f0d9 \ + --hash=sha256:aaf9f25f0f324874085fa5b26a610318db8a8e243cf85bb3e5400595191c7778 \ + --hash=sha256:c8a29def77032773a30879f7f24415b5395ad08592d147c170824ef4c735dfc1 \ + --hash=sha256:d3ec6e1885c46e57f16364f38ea8040ffc5dc0b18341058f608fede3ba930567 \ + --hash=sha256:fadd75c5c20800d64dd0a586ea8cb337c5e630aa2246df0e5105738afa46e02c + # via + # pyside6 + # pyside6-addons +pyspnego==0.12.3 \ + --hash=sha256:39b87aa00491e554cef05441bf2f7e52e85b09b5229ff62a89c35b59548794be \ + --hash=sha256:c4982c9f92e6aa5979c9d9a142a21339ff82ebcfbdb4e0649320cb050961a3cd + # via smbprotocol +pytest==9.1.1 \ + --hash=sha256:1088fbde8f2b49d95a549a195707afa7a76a3ce9bcadc26b6d71f0ffda5fe313 \ + --hash=sha256:37a86b45efb9a47a61a36449063e8e18d0cab3161329fc099eb21783169c4f0c + # via + # -r FileAutomation/.github/requirements/integration.in + # pytest-cov +pytest-cov==7.1.0 \ + --hash=sha256:30674f2b5f6351aa09702a9c8c364f6a01c27aae0c1366ae8016160d1efc56b2 \ + --hash=sha256:a0461110b7865f9a271aa1b51e516c9a95de9d696734a2f71e3e78f46e1d4678 + # via -r FileAutomation/.github/requirements/integration.in +python-dateutil==2.9.0.post0 \ + --hash=sha256:37dd54208da7e1cd875388217d5e00ebd4179249f90fb72437e91a35459a0ad3 \ + --hash=sha256:a8b2bc7bffae282281c8140a97d3aa9c14da0b136dfe83f850eea9a5f7470427 + # via botocore +pyyaml==6.0.3 \ + --hash=sha256:00c4bdeba853cc34e7dd471f16b4114f4162dc03e6b7afcc2128711f0eca823c \ + --hash=sha256:0150219816b6a1fa26fb4699fb7daa9caf09eb1999f3b70fb6e786805e80375a \ + --hash=sha256:02893d100e99e03eda1c8fd5c441d8c60103fd175728e23e431db1b589cf5ab3 \ + --hash=sha256:02ea2dfa234451bbb8772601d7b8e426c2bfa197136796224e50e35a78777956 \ + --hash=sha256:0f29edc409a6392443abf94b9cf89ce99889a1dd5376d94316ae5145dfedd5d6 \ + --hash=sha256:10892704fc220243f5305762e276552a0395f7beb4dbf9b14ec8fd43b57f126c \ + --hash=sha256:16249ee61e95f858e83976573de0f5b2893b3677ba71c9dd36b9cf8be9ac6d65 \ + --hash=sha256:1d37d57ad971609cf3c53ba6a7e365e40660e3be0e5175fa9f2365a379d6095a \ + --hash=sha256:1ebe39cb5fc479422b83de611d14e2c0d3bb2a18bbcb01f229ab3cfbd8fee7a0 \ + --hash=sha256:214ed4befebe12df36bcc8bc2b64b396ca31be9304b8f59e25c11cf94a4c033b \ + --hash=sha256:2283a07e2c21a2aa78d9c4442724ec1eb15f5e42a723b99cb3d822d48f5f7ad1 \ + --hash=sha256:22ba7cfcad58ef3ecddc7ed1db3409af68d023b7f940da23c6c2a1890976eda6 \ + --hash=sha256:27c0abcb4a5dac13684a37f76e701e054692a9b2d3064b70f5e4eb54810553d7 \ + --hash=sha256:28c8d926f98f432f88adc23edf2e6d4921ac26fb084b028c733d01868d19007e \ + --hash=sha256:2e71d11abed7344e42a8849600193d15b6def118602c4c176f748e4583246007 \ + --hash=sha256:34d5fcd24b8445fadc33f9cf348c1047101756fd760b4dacb5c3e99755703310 \ + --hash=sha256:37503bfbfc9d2c40b344d06b2199cf0e96e97957ab1c1b546fd4f87e53e5d3e4 \ + --hash=sha256:3c5677e12444c15717b902a5798264fa7909e41153cdf9ef7ad571b704a63dd9 \ + --hash=sha256:3ff07ec89bae51176c0549bc4c63aa6202991da2d9a6129d7aef7f1407d3f295 \ + --hash=sha256:41715c910c881bc081f1e8872880d3c650acf13dfa8214bad49ed4cede7c34ea \ + --hash=sha256:418cf3f2111bc80e0933b2cd8cd04f286338bb88bdc7bc8e6dd775ebde60b5e0 \ + --hash=sha256:44edc647873928551a01e7a563d7452ccdebee747728c1080d881d68af7b997e \ + --hash=sha256:4a2e8cebe2ff6ab7d1050ecd59c25d4c8bd7e6f400f5f82b96557ac0abafd0ac \ + --hash=sha256:4ad1906908f2f5ae4e5a8ddfce73c320c2a1429ec52eafd27138b7f1cbe341c9 \ + --hash=sha256:501a031947e3a9025ed4405a168e6ef5ae3126c59f90ce0cd6f2bfc477be31b7 \ + --hash=sha256:5190d403f121660ce8d1d2c1bb2ef1bd05b5f68533fc5c2ea899bd15f4399b35 \ + --hash=sha256:5498cd1645aa724a7c71c8f378eb29ebe23da2fc0d7a08071d89469bf1d2defb \ + --hash=sha256:5cf4e27da7e3fbed4d6c3d8e797387aaad68102272f8f9752883bc32d61cb87b \ + --hash=sha256:5e0b74767e5f8c593e8c9b5912019159ed0533c70051e9cce3e8b6aa699fcd69 \ + --hash=sha256:5ed875a24292240029e4483f9d4a4b8a1ae08843b9c54f43fcc11e404532a8a5 \ + --hash=sha256:5fcd34e47f6e0b794d17de1b4ff496c00986e1c83f7ab2fb8fcfe9616ff7477b \ + --hash=sha256:5fdec68f91a0c6739b380c83b951e2c72ac0197ace422360e6d5a959d8d97b2c \ + --hash=sha256:6344df0d5755a2c9a276d4473ae6b90647e216ab4757f8426893b5dd2ac3f369 \ + --hash=sha256:64386e5e707d03a7e172c0701abfb7e10f0fb753ee1d773128192742712a98fd \ + --hash=sha256:652cb6edd41e718550aad172851962662ff2681490a8a711af6a4d288dd96824 \ + --hash=sha256:66291b10affd76d76f54fad28e22e51719ef9ba22b29e1d7d03d6777a9174198 \ + --hash=sha256:66e1674c3ef6f541c35191caae2d429b967b99e02040f5ba928632d9a7f0f065 \ + --hash=sha256:6adc77889b628398debc7b65c073bcb99c4a0237b248cacaf3fe8a557563ef6c \ + --hash=sha256:79005a0d97d5ddabfeeea4cf676af11e647e41d81c9a7722a193022accdb6b7c \ + --hash=sha256:7c6610def4f163542a622a73fb39f534f8c101d690126992300bf3207eab9764 \ + --hash=sha256:7f047e29dcae44602496db43be01ad42fc6f1cc0d8cd6c83d342306c32270196 \ + --hash=sha256:8098f252adfa6c80ab48096053f512f2321f0b998f98150cea9bd23d83e1467b \ + --hash=sha256:850774a7879607d3a6f50d36d04f00ee69e7fc816450e5f7e58d7f17f1ae5c00 \ + --hash=sha256:8d1fab6bb153a416f9aeb4b8763bc0f22a5586065f86f7664fc23339fc1c1fac \ + --hash=sha256:8da9669d359f02c0b91ccc01cac4a67f16afec0dac22c2ad09f46bee0697eba8 \ + --hash=sha256:8dc52c23056b9ddd46818a57b78404882310fb473d63f17b07d5c40421e47f8e \ + --hash=sha256:9149cad251584d5fb4981be1ecde53a1ca46c891a79788c0df828d2f166bda28 \ + --hash=sha256:93dda82c9c22deb0a405ea4dc5f2d0cda384168e466364dec6255b293923b2f3 \ + --hash=sha256:96b533f0e99f6579b3d4d4995707cf36df9100d67e0c8303a0c55b27b5f99bc5 \ + --hash=sha256:9c57bb8c96f6d1808c030b1687b9b5fb476abaa47f0db9c0101f5e9f394e97f4 \ + --hash=sha256:9c7708761fccb9397fe64bbc0395abcae8c4bf7b0eac081e12b809bf47700d0b \ + --hash=sha256:9f3bfb4965eb874431221a3ff3fdcddc7e74e3b07799e0e84ca4a0f867d449bf \ + --hash=sha256:a33284e20b78bd4a18c8c2282d549d10bc8408a2a7ff57653c0cf0b9be0afce5 \ + --hash=sha256:a80cb027f6b349846a3bf6d73b5e95e782175e52f22108cfa17876aaeff93702 \ + --hash=sha256:b30236e45cf30d2b8e7b3e85881719e98507abed1011bf463a8fa23e9c3e98a8 \ + --hash=sha256:b3bc83488de33889877a0f2543ade9f70c67d66d9ebb4ac959502e12de895788 \ + --hash=sha256:b865addae83924361678b652338317d1bd7e79b1f4596f96b96c77a5a34b34da \ + --hash=sha256:b8bb0864c5a28024fac8a632c443c87c5aa6f215c0b126c449ae1a150412f31d \ + --hash=sha256:ba1cc08a7ccde2d2ec775841541641e4548226580ab850948cbfda66a1befcdc \ + --hash=sha256:bdb2c67c6c1390b63c6ff89f210c8fd09d9a1217a465701eac7316313c915e4c \ + --hash=sha256:c1ff362665ae507275af2853520967820d9124984e0f7466736aea23d8611fba \ + --hash=sha256:c2514fceb77bc5e7a2f7adfaa1feb2fb311607c9cb518dbc378688ec73d8292f \ + --hash=sha256:c3355370a2c156cffb25e876646f149d5d68f5e0a3ce86a5084dd0b64a994917 \ + --hash=sha256:c458b6d084f9b935061bc36216e8a69a7e293a2f1e68bf956dcd9e6cbcd143f5 \ + --hash=sha256:d0eae10f8159e8fdad514efdc92d74fd8d682c933a6dd088030f3834bc8e6b26 \ + --hash=sha256:d76623373421df22fb4cf8817020cbb7ef15c725b9d5e45f17e189bfc384190f \ + --hash=sha256:ebc55a14a21cb14062aa4162f906cd962b28e2e9ea38f9b4391244cd8de4ae0b \ + --hash=sha256:eda16858a3cab07b80edaf74336ece1f986ba330fdb8ee0d6c0d68fe82bc96be \ + --hash=sha256:ee2922902c45ae8ccada2c5b501ab86c36525b883eff4255313a253a3160861c \ + --hash=sha256:efd7b85f94a6f21e4932043973a7ba2613b059c4a000551892ac9f1d11f5baf3 \ + --hash=sha256:f7057c9a337546edc7973c0d3ba84ddcdf0daa14533c2065749c9075001090e6 \ + --hash=sha256:fa160448684b4e94d80416c0fa4aac48967a969efe22931448d853ada8baf926 \ + --hash=sha256:fc09d0aa354569bc501d4e787133afc08552722d3ab34836a80547331bb5d4a0 + # via -r FileAutomation/.github/requirements/integration.in +requests==2.34.2 \ + --hash=sha256:2a0d60c172f83ac6ab31e4554906c0f3b3588d37b5cb939b1c061f4907e278e0 \ + --hash=sha256:f288924cae4e29463698d6d60bc6a4da69c89185ad1e0bcc4104f584e960b9ed + # via + # -r FileAutomation/.github/requirements/integration.in + # azure-core + # boxsdk + # dropbox + # google-api-core + # msal + # requests-oauthlib +requests-oauthlib==2.0.0 \ + --hash=sha256:7dd8a5c40426b779b0868c404bdef9768deccf22749cde15852df527e6269b36 \ + --hash=sha256:b3dffaebd884d8cd778494369603a9e7b58d29111bf6b41bdc2dcd87203af4e9 + # via google-auth-oauthlib +s3transfer==0.19.2 \ + --hash=sha256:ba0309fd86be3c27dbf78cdd813c13c5e1df16e5874b99d2535ebbdfb9892993 \ + --hash=sha256:d8168eccca828cbb2cd573675333f3bddd254313a9c42494b84c76b539e8ba25 + # via boto3 +setuptools==84.0.0 \ + --hash=sha256:51a52592b3b99e102b609654876bd65f19f999935166d1352678931132b0c670 \ + --hash=sha256:f4695c21257f0d9b537ec2692c941d02ee143b7cc1276941349a546573b2ef73 + # via -r FileAutomation/.github/requirements/integration.in +shiboken6==6.11.2 \ + --hash=sha256:4bbbd6fa4d7cff5ec5e12bc4c10e1d845fa30c89457ecb13cc64f7bddb77f6f9 \ + --hash=sha256:53659683b1f7a08e9f87eff9b1065f1ceb7110cd7a4bc09fdf5efe43d286604d \ + --hash=sha256:6ab0eba1c904455df621f9a6df3ca2bb896bab8670572d2bc4e37804ae91f19a \ + --hash=sha256:7a7a0a72a9ed26c9bf77d42246b1c736486befb8f31aa2fb29957ea4cdd1c1c2 \ + --hash=sha256:97c49432488df958f0308736f3d15b6915d8826fc380af17bb4b9ec5c4a5e81b + # via + # pyside6 + # pyside6-addons + # pyside6-essentials +six==1.17.0 \ + --hash=sha256:4721f391ed90541fddacab5acf947aa0d3dc7d27b2e1e8eda2be8970586c3274 \ + --hash=sha256:ff70335d468e7eb6ec65b95b99d3a2836546063f63acc5171de367e834932a81 + # via python-dateutil +smbprotocol==1.17.0 \ + --hash=sha256:bcc27edfff7d727a7eb30424138766e1ee216ae2ffc1af03dab0448f29c4731c \ + --hash=sha256:bd1abff5417f5af83ca516a64ab8e5acece3dbdcf58d4e5e23e47f5165a77349 + # via -r FileAutomation/.github/requirements/integration.in +sspilib==0.6.0 ; sys_platform == 'win32' \ + --hash=sha256:03335e14f8563e350506b47a15887a99f08102a80cfb6bd715ad83ab1c2c89ae \ + --hash=sha256:0482631c67a57d8710ecbba3bbe49a9fb60e153dfcfee6d1f9b8a8ff6e419548 \ + --hash=sha256:05883cd90cb2c67f18fd2418ab0f959549fbe6a9a4fc8013679411af23226d6e \ + --hash=sha256:071e4e2a617a39a33e31de2c1f313427533879acabe475fedea31f129f7970bb \ + --hash=sha256:2758aac58b6ca0aa5b726bdc76cea7d020820ac9831f4082b5b6fceda2cb48d0 \ + --hash=sha256:3b24b6c19657f7b8b74a92e6cadb3ae5e25602ee348a7142cebb5194a0eed01b \ + --hash=sha256:5937dbe41c34cbbfabbafb84238a06aa8912210cae8a4b909fa20f965868cc89 \ + --hash=sha256:68a34e702fd81de3d74ea8caf6bc6a1218af551b75d0573a3b0607124a54e5f2 \ + --hash=sha256:6ad61668c4eea0fdeefd1f596c4fb7b3f45237b1a5989c12d4412729bded7231 \ + --hash=sha256:70223bdfefde9656c48dd17bb58fd0fb7e879e5b08ab028267d585206f393886 \ + --hash=sha256:73343ba40b04de0486ab92f60b729eb0768657f71e4c279555d3273f323c0ce4 \ + --hash=sha256:737adc3b14c2079480bcd10139fbab0b2ad683a24a6046ef69d9c769f4672c0a \ + --hash=sha256:76a289f415f9beb6a2f3f0cb37b655863616c303b2a06754e1bb84ad86348aab \ + --hash=sha256:7ea194eeb68d8848fde7125a7ac67fbd4a9052442b7ce69ce6c0c8115227ec67 \ + --hash=sha256:83f99ed351d3091434f182579d20a419fa7baf03ec154632b5f7f3fbc6aa0788 \ + --hash=sha256:8957c47c6d4e145a9e111a4c2a2b357ea15cb69719ff99fc0088fbd612618226 \ + --hash=sha256:8b4b911f59f03601c23357146814e784c9a7612f34f5b933bdb1c50591b173d0 \ + --hash=sha256:9073afb8f30dc0e7f3a40b39fef0fc29e4bda6a3d4e77936f3c496cedc4beade \ + --hash=sha256:97bb2e9d4d916d4d6f566888a81997bb3ab0e05ac41549fa0fbebabac66de0e8 \ + --hash=sha256:aa97562ca3d27a0ba5c830c56bff0fb6d3c8257bd16589b63869a212de6c92bd \ + --hash=sha256:b4d4e3402230f14cd93999f3953619163075bcaca3e13dcf2ab5942c269627f4 \ + --hash=sha256:b65fa6f90e523daa6eb95c0aa707cd6c04705bdfab3e42c00723d23141bc74ad \ + --hash=sha256:b79f8088f8e2826586e5eb14b68f03f443637df178e95c190765c8065fc57a37 \ + --hash=sha256:b841fabe630101d9e2be6ba56407a2a539087ee85d6ca1477748c21135c9f636 \ + --hash=sha256:be2b500e7b745569794e8cfa557510f6c18e5402ad8f1fa1abe7adf53b3f3964 \ + --hash=sha256:bf125782912e4a9012822619eb41e2c9c0e32cb9148efb649e608804b4b4e313 \ + --hash=sha256:cbc8cf653e71b32ab7cf11d1630866cf12133fb235422f3f9bccb60ba62290e4 \ + --hash=sha256:d239d78f9619d4fc76a9d57eb929258de50024c56097e73737d46ad58be0acaa \ + --hash=sha256:fe8c7c9f749ffd302336a7820889ad9189c237d1f6d24e1232d7e2c653433d5b + # via pyspnego +stone==3.5.5 \ + --hash=sha256:998e2d0909c859065547f5f54c7d11b66d019882bb4f773e78ec9f2023ca3694 \ + --hash=sha256:ff04dcfe37efef6bfe744f59f4df1dd1a43716f47e28d7cf1165eb984a0b2c33 + # via dropbox +tqdm==4.70.1 \ + --hash=sha256:c293e525e6fef9c20e8728fd4612df02a0aa31bb5fe91ecd93e123b1b7bffa73 \ + --hash=sha256:cefd0eca11b2a37a3aee776544d4f4ae913f02688135b5556b8788dfa474afc4 + # via -r FileAutomation/.github/requirements/integration.in +typing-extensions==4.16.0 \ + --hash=sha256:481caa481374e813c1b176ada14e97f1f67a4539ce9cfeb3f350d78d6370c2e8 \ + --hash=sha256:dc983d19a509c94dba722ee6abd33940f7c05a89e243c47e907eb4db6f1a43e5 + # via + # azure-core + # azure-storage-blob + # opentelemetry-api + # opentelemetry-sdk + # opentelemetry-semantic-conventions +tzdata==2026.4 ; sys_platform == 'win32' \ + --hash=sha256:c2169a8b0a7a5e9674da5a135ccdfb2b3e671b333ed9fed17b41f73c34476e81 \ + --hash=sha256:f1b8bd365d8d210c55353f4d7f8d6d8561c0ba50d704b700d195a9424bba0d79 + # via -r FileAutomation/.github/requirements/integration.in +uritemplate==4.2.0 \ + --hash=sha256:480c2ed180878955863323eea31b0ede668795de182617fef9c6ca09e6ec9d0e \ + --hash=sha256:962201ba1c4edcab02e60f9a0d3821e82dfc5d2d6662a21abd533879bdb8a686 + # via google-api-python-client +urllib3==2.8.0 \ + --hash=sha256:0cf3cae568d36aa9576b28dfb35f11328f1cb974ca7647d9475ebb86c75ac6e3 \ + --hash=sha256:63bf2ead4c879426ebf22ef2a781eeb4aa3b4ae798a0435506f8687fd5bb9b63 + # via + # botocore + # boxsdk + # requests +watchdog==6.0.0 \ + --hash=sha256:07df1fdd701c5d4c8e55ef6cf55b8f0120fe1aef7ef39a1c6fc6bc2e606d517a \ + --hash=sha256:20ffe5b202af80ab4266dcd3e91aae72bf2da48c0d33bdb15c66658e685e94e2 \ + --hash=sha256:212ac9b8bf1161dc91bd09c048048a95ca3a4c4f5e5d4a7d1b1a7d5752a7f96f \ + --hash=sha256:2cce7cfc2008eb51feb6aab51251fd79b85d9894e98ba847408f662b3395ca3c \ + --hash=sha256:490ab2ef84f11129844c23fb14ecf30ef3d8a6abafd3754a6f75ca1e6654136c \ + --hash=sha256:6eb11feb5a0d452ee41f824e271ca311a09e250441c262ca2fd7ebcf2461a06c \ + --hash=sha256:6f10cb2d5902447c7d0da897e2c6768bca89174d0c6e1e30abec5421af97a5b0 \ + --hash=sha256:7607498efa04a3542ae3e05e64da8202e58159aa1fa4acddf7678d34a35d4f13 \ + --hash=sha256:76aae96b00ae814b181bb25b1b98076d5fc84e8a53cd8885a318b42b6d3a5134 \ + --hash=sha256:7a0e56874cfbc4b9b05c60c8a1926fedf56324bb08cfbc188969777940aef3aa \ + --hash=sha256:82dc3e3143c7e38ec49d61af98d6558288c415eac98486a5c581726e0737c00e \ + --hash=sha256:9041567ee8953024c83343288ccc458fd0a2d811d6a0fd68c4c22609e3490379 \ + --hash=sha256:90c8e78f3b94014f7aaae121e6b909674df5b46ec24d6bebc45c44c56729af2a \ + --hash=sha256:9513f27a1a582d9808cf21a07dae516f0fab1cf2d7683a742c498b93eedabb11 \ + --hash=sha256:9ddf7c82fda3ae8e24decda1338ede66e1c99883db93711d8fb941eaa2d8c282 \ + --hash=sha256:a175f755fc2279e0b7312c0035d52e27211a5bc39719dd529625b1930917345b \ + --hash=sha256:a1914259fa9e1454315171103c6a30961236f508b9b623eae470268bbcc6a22f \ + --hash=sha256:afd0fe1b2270917c5e23c2a65ce50c2a4abb63daafb0d419fde368e272a76b7c \ + --hash=sha256:bc64ab3bdb6a04d69d4023b29422170b74681784ffb9463ed4870cf2f3e66112 \ + --hash=sha256:bdd4e6f14b8b18c334febb9c4425a878a2ac20efd1e0b231978e7b150f92a948 \ + --hash=sha256:c7ac31a19f4545dd92fc25d200694098f42c9a8e391bc00bdd362c5736dbf881 \ + --hash=sha256:c7c15dda13c4eb00d6fb6fc508b3c0ed88b9d5d374056b239c4ad1611125c860 \ + --hash=sha256:c897ac1b55c5a1461e16dae288d22bb2e412ba9807df8397a635d88f671d36c3 \ + --hash=sha256:cbafb470cf848d93b5d013e2ecb245d4aa1c8fd0504e863ccefa32445359d680 \ + --hash=sha256:d1cdb490583ebd691c012b3d6dae011000fe42edb7a82ece80965b42abd61f26 \ + --hash=sha256:e3df4cbb9a450c6d49318f6d14f4bbc80d763fa587ba46ec86f99f9e6876bb26 \ + --hash=sha256:e6439e374fc012255b4ec786ae3c4bc838cd7309a540e5fe0952d03687d8804e \ + --hash=sha256:e6f0e77c9417e7cd62af82529b10563db3423625c5fce018430b249bf977f9e8 \ + --hash=sha256:e7631a77ffb1f7d2eefa4445ebbee491c720a5661ddf6df3498ebecae5ed375c \ + --hash=sha256:ef810fbf7b781a5a593894e4f439773830bdecb885e6880d957d5b9382a960d2 + # via -r FileAutomation/.github/requirements/integration.in +wheel==0.48.0 \ + --hash=sha256:3217dcc807155e45db462d7ef2431f5ddda0d7273b700d05a67b271ceb1287ab \ + --hash=sha256:94800765601e9171bf5d58d066e640662842bcedcbab982b2c90787a2c987322 + # via -r FileAutomation/.github/requirements/integration.in diff --git a/.github/requirements/lint.in b/.github/requirements/lint.in new file mode 100644 index 0000000..71deae3 --- /dev/null +++ b/.github/requirements/lint.in @@ -0,0 +1,6 @@ +# Regenerate with uv pip compile --generate-hashes --python-version 3.12 +# --only-binary :all: --exclude-newer 2026-10-02T00:00:00Z +# .github/requirements/lint.in -o .github/requirements/lint.txt +ruff +mypy +je_action_core>=0.0.2 diff --git a/.github/requirements/lint.txt b/.github/requirements/lint.txt new file mode 100644 index 0000000..1c8d58f --- /dev/null +++ b/.github/requirements/lint.txt @@ -0,0 +1,290 @@ +# This file was autogenerated by uv via the following command: +# uv pip compile FileAutomation/.github/requirements/lint.in --generate-hashes --python-version 3.12 --only-binary :all: --exclude-newer 2026-10-02T00:00:00Z -o FileAutomation/.github/requirements/lint.txt +ast-serialize==0.11.2 \ + --hash=sha256:00119a8fb8c1dc0f1fab023f4d8071fa49e3b0208ee54d589fd463c16ab0124e \ + --hash=sha256:00bbf1f6669f813b48925b759f7ae4591067d456d443924055cab386e7e0a719 \ + --hash=sha256:08eda88a0f290a36c38cab33df8bf7e35eb95bc802ca5beb2c8fcda471a7d10c \ + --hash=sha256:0d01f61352c96370febf6c0dbd488dee9183a731fb2702170da9163ae317cded \ + --hash=sha256:0de02520c11391a026e62987a9aa2c3c2ff01545155059ddf0c4bdf2c5ecbe9f \ + --hash=sha256:13b13afe32e845c86a573497729e1b7ddeb26c572c78bf50ece51da23b8fad5e \ + --hash=sha256:1844ed9a487fb3de7325c52ddb33f2918b66b65cd54d3f8d83d23785ffe99fa4 \ + --hash=sha256:1d6ad94edbe93bf1dabc06c9f37d55b898fdabc456aa6d7ced5e23c14f795f32 \ + --hash=sha256:2fa3be25f7f5351b1b39c9f8a52779b2dbf21199efbae564b4746422e8edca4e \ + --hash=sha256:2fdf31a0bb85ea2575cc91669f005e6647d2efed491231c4dc1497bc9a5b3aa6 \ + --hash=sha256:3b78e6fdef3b06c86ed263e1962fee5a7b9d2d158e738b212d13b2c605ee12f5 \ + --hash=sha256:40b2801cf2221bd922d9f69d2f0ebc373c3db47207315d525b2d87fa161a2af4 \ + --hash=sha256:43b51e6ebe6549bf21416c3c78ee886147b80875a87cc6f69e303dde0d75be0b \ + --hash=sha256:554d117cb916d8032d85007c654d179efbbfd446174c048062778136a922944f \ + --hash=sha256:57c0f5cb0021a5beb1e5e4d6e840ae2f23a28909703ef4d256a144cc1ad3d437 \ + --hash=sha256:59c25f47524efa052971b860e128b1add0c94ede7dd16b2962952c85c3582365 \ + --hash=sha256:6061a54f39e82a9f2cbcb9c268fc441890e4818a6636473caa4f4063254e0750 \ + --hash=sha256:75a1c7f46b9c19fc0ae01ca6fd076301628faa2ed7a8edbd55c6353c483946a3 \ + --hash=sha256:76cc294246e60a914326b4ca88c6a5ea89c064906614aaf1537ce82f09e9449f \ + --hash=sha256:7aaaffc32905159774a107d3cf33dad59bd41b7a0d1bc9885532186753ee7439 \ + --hash=sha256:7f1823275b246f9c7d373be6879e4eec09686948895d4ad083f4b27fd7e4da70 \ + --hash=sha256:8532f20916fa3189d4d785ef2a62d93c4d651ec9c5bffda66d2fc36898351f34 \ + --hash=sha256:85fbb01e83967a126d71f679f2b9528ef0912cb0854aa1a4657314c34e255b57 \ + --hash=sha256:89499a439955931281986e97ca4dd3c064bf0d2e0027c0017344eb86667733a1 \ + --hash=sha256:8a5ffa70e76191dcf240d3c43e20c93b3bfd26f54d89148c762d57837f5bcd2c \ + --hash=sha256:8d62a47714c8bc432b9fabcc29989c815c5da17327d35151f2fd0d85c2a7a5ff \ + --hash=sha256:8df32ad4ff7843734a6c2f067ee974f6d3109ee5a2c3e1a9d2f79347bd282a9a \ + --hash=sha256:976a5bd75845d22f4b52905ddf53ab669ef1b14dba7735f5512841a2ef2b5450 \ + --hash=sha256:9d80a81ec84660422579bdb8e789f656a794b48c7a1ae1261f6bd8bc1897d17d \ + --hash=sha256:a0fd40c668b0fa19b8fdb61d9e63d547e2e19cfbfe053a51ef0b6c37070298a8 \ + --hash=sha256:a586be418eb70a9f1396cea29ddac8f4b9bf277fb73ea2340db31e218bc00f32 \ + --hash=sha256:a7004ba572f09be34342ccb98dcd4bad5707d3d81adc8cb4c3f685d2a2c51bbc \ + --hash=sha256:a9ffa8a197a721f07a352d0be6185f5b3e6f9aaebfdb66169ed652108531ae3b \ + --hash=sha256:ab924ba260efd7509492f272d4e236d24564033f20c005d7c63a107c6a76fc85 \ + --hash=sha256:abdb3e49ba053c3486ac1263bee9f16cc9a4a8abd9f8c90bfc21e3669f3ad9d1 \ + --hash=sha256:af8c003ce721b0099dd55cef4ba733500fc3054ea0cc8565d8957aaf7cccdeb4 \ + --hash=sha256:b17869f4ba261a5fa468a753328a548f4dbaf74b4eadae9e28aff66df7f1425b \ + --hash=sha256:b4e4558956b6a0fb35e18fba58f7d1810b1f2c0e6b52352572cd5dfb6b4ef33a \ + --hash=sha256:b9065dd23131a23b41f5bab3bf4e9b3c350a3fe8e36e8200eded9b729fcea484 \ + --hash=sha256:bfbe47a3a7c368f28836e78b2440a3643ac0ec4c67d9fe53588e1448f0a3d35d \ + --hash=sha256:c58bb119b73657fdc5569692f316e1e25ca114bd62f7782eb527c6be438ba3a9 \ + --hash=sha256:cae5addfbb54cc1d47fe947ef9138e9d83849ed1cbc72b819cf36d96a2315b07 \ + --hash=sha256:cb073bfa15742699d408ac50f60878383b5665ae1791d1b6799ea6f08633cd77 \ + --hash=sha256:cd320a5c4f1f2742af97eea22954f776379175c5ef2504801e9a155f2ff9a4d7 \ + --hash=sha256:d60515335750d431e462af6e722bb55720a5e7827192777bddfd9c4376065a4d \ + --hash=sha256:d70556a2f9230a44c99a655774cde823f056efc34466eabfb4085f0cb1ea9f99 \ + --hash=sha256:daadf1c3e0224621607ffe16f1379e4bd372271ed2e1db8a67878f0bab3ef7e4 \ + --hash=sha256:dab599cbdcb7b45b18c41fad746645580b3a24357082b7f0e8921cd373804f27 \ + --hash=sha256:ec1c20f89c3e0d83576e3c06f79375ce936266591fe0d5fd969914af3185cbaa \ + --hash=sha256:ee732ae167e686d1d3c00f98d7d82b23138304694f0441b14d7ddf9c0f8a921c \ + --hash=sha256:efa819d7c14c8e4153dcd84671826331538be7cbe460383fc6386f5eea5bd234 \ + --hash=sha256:f3109fe4805384effc8d0f8e41fbf875aa8f389af91b4348c1cfb60ea6e4cb82 \ + --hash=sha256:f3a367e0e05ed2d1b747ceb07aa728a8c204cc008b589127e9bd4f40053d7575 \ + --hash=sha256:f6a8dfc5ab204a706f6e5d39c6f77c18c27ef084fa2081803a64a9160ce89277 \ + --hash=sha256:f739e0b601be7300c5697a2573d9200bd1db74b34ab111ef9537b9d5dcd7f106 \ + --hash=sha256:fd666cebd6ab3b3c0fd348a6202c26e18a401ee34293c3804d3472266bc146f6 \ + --hash=sha256:feb16d9c2a720e0120c58dd5d6e7b3c7c86b43249b60a3bc212bcb8fa031e2dd + # via mypy +je-action-core==0.0.3 \ + --hash=sha256:395890665483e58f17fede17aa809bf6e2762ba8f0bd6751126fe46efabb8c85 \ + --hash=sha256:7b73ab0172a3c90c91b0a81b8a662350199a35842f529acf7a54706052fb38ee + # via -r FileAutomation/.github/requirements/lint.in +librt==0.16.0 \ + --hash=sha256:001bfd59a7d45b17e3e75f2a8c6405280b35e7b84471792778e718c4f368950e \ + --hash=sha256:0058f9d68721094105917254c72ac0569117bb7b13b9769cf45d26d89f9d21cd \ + --hash=sha256:02118f56a9c36ddd07dfd9b919d9ecc117ba20a90987d56aa4c429fa34509188 \ + --hash=sha256:0253721561787b8df8443eb347b7a6461015354e5bdd37ee38a41fef220d2bb0 \ + --hash=sha256:02d89c813d5ff74b17df72d3a34819d132cd168e56b81bf755b809bd9e46b8c4 \ + --hash=sha256:0314058469f4d2fd279ce7c62ac274ac82c3918ef7db62ef0697c4c359370155 \ + --hash=sha256:0dbe4096a7ecc00fa835d24510ad8545a4efef738dac96e0e63516783ccde905 \ + --hash=sha256:0ead24d2562a49473dddd9efef8581f020007eb0054389c3ee3ffad38b1ca4c9 \ + --hash=sha256:13b4e8aba90b0b1c82474e9844aa9ffe7ad3faa484350e1da64cb8188d903134 \ + --hash=sha256:14ed6ebe3e4f85f326d7920011ad30ff49ed9334e62cf88caef9ba973d9e3a92 \ + --hash=sha256:17bac7f7a16b328fff77e440287693eb017abde913595b5827ebccbc21ecd8a6 \ + --hash=sha256:1b384b90ab79a7bc30b566895809a636e0666f21f3cf12b54823d025b7e83839 \ + --hash=sha256:1bc17e54e5305f8d40b7ca203671ff5a9e59c1d0f8ea0f625dcca53a3984de11 \ + --hash=sha256:1d28ae980ae2218f9c5b95d191e947296f918c9bf0b400d467a9430275bbe678 \ + --hash=sha256:1e511762a074005bb0aa569166779834e75e438370226930d0ce1866d4b6a33b \ + --hash=sha256:20fe0bf9053885e21c62b3e091fb5e73e1c54d1daf2e70eb388d70763bcd4220 \ + --hash=sha256:242e00b3d4fa37c3d3c1ca5f5c9adb7d909ddb1eac9c41f2787320d00caa0af2 \ + --hash=sha256:25a58a19ea8d83b68209f04912df765e9260635ef77646542ed4b4abe6bc7940 \ + --hash=sha256:273d00be33792a15189331df10f1f1331621b043881e66c6c4377f7f776e1291 \ + --hash=sha256:28e038895b998d7a0c7798922ce8a1dc157675df5cf1c9ef0aca809ed804b7a1 \ + --hash=sha256:2bec3818c7da7c96ceae0ef5915a3d16c52dd08f3ea913bf1fe8568c447c7978 \ + --hash=sha256:2c4aa329c17bd1aaea4f6e89335d8ccd494b3a5830b6654462273e50e11023f0 \ + --hash=sha256:300c3ffdc459f4a779a8411ecb188e3ac0b1ff3a3a7b099642555dedae06c69b \ + --hash=sha256:30b7beaf3f4487b7d8adef1f158b49067cb4d5a19fa7a3bf31a4e7a820e435c5 \ + --hash=sha256:314e703f0c19320dc8094e7a784b9cf29e1b67402515abf580069a6363c0b4f1 \ + --hash=sha256:33f41443a1f4e1f099331b3d8120e409fbff84b9760bc1cc9ea496f37ddaa5cc \ + --hash=sha256:349c0bcb87ebd07481b6ff781e25cdc699723dbe2212e57dabb27f7a13b7b87d \ + --hash=sha256:36e53948e99bbe3ffea257124cfcae1cfb01831555c9a9c903c9f9a72db7fd07 \ + --hash=sha256:375bfe6b572a8f6cfc398709356046173bf27e64c4c5edaf5f7062f051fb4bf9 \ + --hash=sha256:378dfaffb38e59c24a87cde5713cd865d51ff7383fa12947f3907f306ea1ca55 \ + --hash=sha256:3931f7a3db322e7f44e02a280e3949326ce9579ad388ee8d691dc7c76da9fb70 \ + --hash=sha256:39ca4f2f2fe05de8e63493da592d84311adabe5bef52b193851981da9816b302 \ + --hash=sha256:39ec1d5a14e37baf1450a6cabf03fe552340808bf1ad9d71824ab90117716459 \ + --hash=sha256:3ddeb3c9dedb461bb457c6c7d9aa7fbf35329da313d1a7543d00c8d0f3473c96 \ + --hash=sha256:3e0c39bdc85370422e8b637be76eb1fd07d30967551b03e62267dd156f553152 \ + --hash=sha256:3e483a8d69ede8067db70c0e83007423b6925de6fd53afed01d66160f2e9398c \ + --hash=sha256:3f0b8114c44b2ac06ff5dacd08e07e8e807ff4f46083f2a1602685122559be41 \ + --hash=sha256:3ff4b2367926b69c6215635902cccb04048e73094e9862900d27cb2c6bbff143 \ + --hash=sha256:4323193ac0cd025f85af531df8ba91bf24d1973b401697347a6282e8fd3fcf5e \ + --hash=sha256:468df902df016a06eb0e40b0747dc8d14e47d7a38b18b63b1fb167d85cb94d63 \ + --hash=sha256:473eebc7866bb0a0c8849a292b5e7157c1aba5d14d0f0f610c52158d6d964262 \ + --hash=sha256:47ada6ea32636492c61aa8ad27ae3b9404bfe7a97e3ba946d1984236cc741da0 \ + --hash=sha256:4b6183e2e2e0ee00aac2ec07c7f7d151c97e85666b304f574b71cff0f9fccc4e \ + --hash=sha256:4e29522c62e28595ff7e324c6834ade51127707f0e255b18d1c1cf03d39c1048 \ + --hash=sha256:4eb1313a19847089ee81e88742abedf285c60538816640b99742d8534b81d26a \ + --hash=sha256:52327da75a94012e7f932f913d20d3876bed3c102be00e6c3e8600ff7bdd58a7 \ + --hash=sha256:54d11f726aae9df5a6ffbbf0a03a52449bbac84a53ef03669cb41cdfd4ae41bf \ + --hash=sha256:5696d7f52e7b37217cb3a8f92c744fe835942602fdd4c1a8bc4741d3bfdce15e \ + --hash=sha256:5750a105b42a416f930edc59054927a406effb2550cd5bab92ad7a5842ed5d05 \ + --hash=sha256:5810ba811297fdf37a1531a57667cb8ace0842013ca8606bf9eb7c24cf4be154 \ + --hash=sha256:5981c011b306781ce561e18e14230a14524a3d8109b97553666c942c18f31a96 \ + --hash=sha256:5a269c46ae327d8e6f8c1f85f7516cb52c0fa48127565a1105a4f4a05ff2a0b4 \ + --hash=sha256:5b976054553670829985ed767feb78fb6bcede0175327c4844dd5c281c1be659 \ + --hash=sha256:5bcc2c4726ced915b00de0c9856a4eeabfb3fddb93e10e0b8f735b7709358b6d \ + --hash=sha256:5cd5b092441053364af968ea12084692cb9d4a22f3ce9524e377880bf028761e \ + --hash=sha256:5f49cff01bd608ef7d97104cb035c75455e79c2d70bf4a506cf773338ac1860d \ + --hash=sha256:6072e92dd876ff6ceeb6cf371e35e51f479349837391341f479b08df4564242b \ + --hash=sha256:64c79520414a3fdfc6aabd7593e6169afa14d5f8d9908d4b498db068868b08dd \ + --hash=sha256:67e718c7a43f8db325abbbf1404e2d535f12f8f7a1a82259568385cc5274b82a \ + --hash=sha256:69ba927445cfaaffb4081003ef5224c55a5c2ab67ef956f416ef744916e44121 \ + --hash=sha256:6a63610fa76524edfa605b5b259a603915c7a6e10e54f003e5503030506e81de \ + --hash=sha256:6c5da27e8056439f927ea896735da60e616c477a8293feaa3233d4e7781a6726 \ + --hash=sha256:6c8893eae2fd13c5488d94056f3e6e5cf3142bfb1c4acaf136cb33d760c5964b \ + --hash=sha256:6d4a64283ee61824b5790de882bc68e2d9d7a5143537cb7a966f7354f71646d4 \ + --hash=sha256:6fe436af2eaf630474f491af5d032cbe45f93fcff5c3b9fe4ab194a7255b20ff \ + --hash=sha256:71b93b42784e25b975079573c642a8fedb049a7bb31d70a51721b1666b3b2ced \ + --hash=sha256:7393c9a48dcce4817dbd4b0d8ff6237efe9b0a0609f5b0adaef315f8541726b5 \ + --hash=sha256:77c7a2b4fe2c1369e0d5aa1cade26740a7b14be32fbc9a5535d617d20065c39d \ + --hash=sha256:7a1d272724b581bb6bc769dfdafed6da2ecc9886ba2450311de55a4ac2e1e9cd \ + --hash=sha256:7cc365f006891afb006b52d5ee5ee74c09306ffa20e2f8705a32a4450af2f3ba \ + --hash=sha256:7e510b7770bee609617a3374a96548eb114cae048023e3f049ee449e7ff2db32 \ + --hash=sha256:80039ba9b6a7d5f1a0175a4cca6bbefead87bd854c80abad1cb30afe47a830db \ + --hash=sha256:83d4041a3d9b2fd053a8a4e1f22878b3e5833e2712956382d5c048d791454e91 \ + --hash=sha256:845a511b60ca43b9880dcc84a9784c891d6a2098c829130b320846c69c9c0c68 \ + --hash=sha256:877698bf6bca5721d8be345f2fe09778e40ecadea8b58c73075f2b1a53666bf2 \ + --hash=sha256:8caf96a4ef8fb27d0ac0d1ad8337d26a240acd4a02fe4345d0a8f264753e8f99 \ + --hash=sha256:8ceafb70f2a4f0826f11031942e59c0728fd98da112dc346d4352bde1e486866 \ + --hash=sha256:8f36c58e33b304b525c6c9c5076399c6ebf1109e17b9051a05a407b091b9215b \ + --hash=sha256:8ff5d26c529336be9bd7ae04483235d77778ee7d6444a95353102b542601ce81 \ + --hash=sha256:909d8e3c1faee44cb762b1c519ff8613dcc5ceae5c99987a00917b5a31fd1d6a \ + --hash=sha256:92caf82ebef5e12d21c72242b70d1e92536f1711cf2a727a4c276de4b4469087 \ + --hash=sha256:931a0bb0fcac88f263e269e46eb30ba8e21402cd3c62ca40cb97034c0693fab1 \ + --hash=sha256:943c6bbecbdf7fa575a4f2952fcfd848c88ef95507c3fca411e89d4ac3ff8143 \ + --hash=sha256:94aed6a8308818b91677957d1bd03188869cd7aeb23c5dba7912a6c0402f7602 \ + --hash=sha256:94be5cb7bca4df6201f4183e9e4fa2086c655283d20b38cd84500a69057575a7 \ + --hash=sha256:953107e2f68d0f3512c48f898b0dbf0ce5cc52bba0f318d847c985dc555ee4cc \ + --hash=sha256:96f576f2711f8519152ec76d0e599243555c1f07679fa73606ca8c8c868c0be6 \ + --hash=sha256:a33e0dae1f8592146a4764d54ce842b278732d21a84e17c3bbe6b1bc158a2248 \ + --hash=sha256:a4aaefb4ba6c07e1aeebb2795c8958148f1d6f9af3b555b53d23d766edb6d67a \ + --hash=sha256:a8afb6557920860b7a3a596eb804cf37e09e7cf8a803db2478c202acc72d8c2e \ + --hash=sha256:aa9357a1b4d4fc787bb718a59cb1112c28c8a976d6bfa268b71cc0a4ab8f3a94 \ + --hash=sha256:ac38d6d8d66bf3d744148dbbc0b8e193e195a51e364ed55e224631f5721891fc \ + --hash=sha256:ad37d5b9abd49c9a655dcda7ea52a8a752884062ef1ee71ae17c2f2a0f81fe6a \ + --hash=sha256:aea7b1f2b125dad5de85f049136651bff256c883c65e6b9209b2da0a1ac3cdef \ + --hash=sha256:afced3dfc17cd805ecf7a3d77996a71cf5f2c75aa66eb0c21a9930f4fc992f86 \ + --hash=sha256:b0e3e721c75d2e79a76d4422c79d7ba705fe1bbafec907037fe7a657a480a0e3 \ + --hash=sha256:b6d085d70bce51d43c5c7c36d63490770180d8779e71c49305c87b4213918de7 \ + --hash=sha256:b95d5d92ab83d39e760a52091bb1baba664f3a2351e39b1e16801e5747c2f0e9 \ + --hash=sha256:b9d6d4b14e92d876f8026b54c20c445f36425214c1081dc76f74e40db386b82b \ + --hash=sha256:bc02954b1295de798bbdb0b4e2d8a28c2117de8b5c73dcbeb27dc32572dfb971 \ + --hash=sha256:bd3150023d3dc2bc70f3784e59ffa1140d56ddba3d8125b3d6f9f85221279bfc \ + --hash=sha256:be56ba9c884143495b517f23fe794ae367d58cd89ea0fdd6d437e3c024a87f9f \ + --hash=sha256:c17194318e4c0c0348b36f36c2ec7534436fe0a4c15582403162a4f08c80797a \ + --hash=sha256:c3d1bb7841a816ace6449bb26d3f9560dbfa20e71c568d23f0f62bf1e68f50b1 \ + --hash=sha256:c43bd6e642d8a248c114327f98dd25ac5a7cb5aa168ef02f0559b91874df16b8 \ + --hash=sha256:c5db585d43449a5f54303d4b2774e45e1babd975cfe1630a3d708c0b80c3e560 \ + --hash=sha256:c5e6144e68b577f157519f2ba88ca20e3ed61c29b00e5cdfa76cd2d45acf059a \ + --hash=sha256:c6f1b27bf1632a7e016af9f145f82be95e1edd7721a646505c21059257cb5a04 \ + --hash=sha256:c71d1b76210a36729fedfc5115069b50a3d8619054745f8758fe5d6f19e86671 \ + --hash=sha256:c72c5295a84bd249526da9bdca38f2e176d15c31c13bb0063c5053f4ca023421 \ + --hash=sha256:ca8052401c55d7511dda6760719fda7618067e83535d7d0010096d216c34b667 \ + --hash=sha256:d1aabe3925cbb4a08d15b7b20ba4011b53019da0c4173a25155139b7b1baed65 \ + --hash=sha256:d3c94211ee0c4f8d649ec06b7c115c0ec4eadb873a0e3154ca15cef3f814b071 \ + --hash=sha256:d46ca272b251d033dd4527b0dec5f261a28a52bd5fa0f99c117b0a1f8588cc2d \ + --hash=sha256:d608f0bf3b8cbddd0067fe02cb8cab7d13e8eb9386b23a1843d4363044fd9e22 \ + --hash=sha256:d6a365f2ab45a984d0e00eee0dd17f599ceab8cadab6ea07b6111c8132fc0e42 \ + --hash=sha256:d92db7a0f6aee44f1baee94750457e8d2d1c6ccea41842de6268d34e8dc7eddd \ + --hash=sha256:df183721229ae51eef90108c115b43e98cb169b6155d34480338e5fc6616df00 \ + --hash=sha256:e05108e0849966f53a8d2d3112a7af881d0efaa479bc735bba91108f9f2350a7 \ + --hash=sha256:e1967e36ac4cae0c7e9615ad32e1a513cdacff79f9e8afb28bedc91caf48b4f3 \ + --hash=sha256:e42f8e098b9c5396fefa05fb1cc7e33b0e08fc51da106b5de4a45fd22aac6743 \ + --hash=sha256:e56aaf8c167548dc8e5d6f3bd0f48dcdd299a23c73be3f744aab79d99e9c7f5d \ + --hash=sha256:e9ce0bc440e7fd09b5f51f372f0f6640658f854b8fb05260f819cc669c93c42d \ + --hash=sha256:ebefd60b42e2a82b32d136bb5f7c94eadfcd29f772b547df6f3291d1ed855a1c \ + --hash=sha256:ef46c1a29ffb8c72e882e22618ec618778eacd0578fb22c6e7cf9c11d15f357b \ + --hash=sha256:efc49c462d4516b8a58b00b490078fa64689fd1fe66970cc190131d7afb8027e \ + --hash=sha256:f01f3805f2dae4781c0c34b440e31740d082950bdaf89a6f601ad589a28af57a \ + --hash=sha256:f06c689cb14afd9b612727553a5ec5a40febf113ca41c4413a2b0b334285884b \ + --hash=sha256:f1e8591bd8a5a628cd7f07954c6a1592359a878bf032957a8e9057a41d644311 \ + --hash=sha256:f4462528b6000afe8f16907b5c7c2553abf1df005ba5140e6eb394541c3624c3 \ + --hash=sha256:f7be7cf555bc30ec12622e9447299cc4a9b8ff307548b634794353db0c2065dc \ + --hash=sha256:f81b5b19ce748ef68d4746656b7929762eb2fe99269b266e4be07e2ee4de7144 \ + --hash=sha256:f9807485a908f00355820f18e91e045ffdcdc5adb68aaec40a1e2b88c5f7bba1 \ + --hash=sha256:fbe4fb8c5445f7496d7f7f6bb0807875d09d47e6771ffa175fb2df2895fb86ba \ + --hash=sha256:fe4372c52d4849096c6cc1cda2817d293ec51440c890474ed59ef38d46556f18 \ + --hash=sha256:fe52bf4641069e7978a14253b036cb9002def1926317e710f2e249f8a8c47742 \ + --hash=sha256:ff7baa55f8e7c69851419e50a666015d02a74198716fd45c0125a2112e0a389f + # via mypy +mypy==2.4.0 \ + --hash=sha256:058165f564ccf559c68c70fec2091fca5891110480210c22594635e3f6683437 \ + --hash=sha256:0bb95cf34899e4619c61ab0a8667804e139e580b30d5df12af2102dfe44d0c97 \ + --hash=sha256:13fa24f439c0e48a290a3922fa14ccd22f2762bae99d2142931b3e40a9055080 \ + --hash=sha256:172e30b8fea631fe310f0c665477f52d9ea40bb4e99e0c81dc30118563b13710 \ + --hash=sha256:1dc0f64b0a92ae27a49d2175f0bacfa56e15bdc5b420cf92e0c1f79292219cbb \ + --hash=sha256:20e9a5cd875837520c43db98dea0b6d0c2197833d95c30127d8f570fb9b1f00b \ + --hash=sha256:2106b55105ba5ea9be4f53a24517fc5fa927ff1585edc9bc1a975abb72caef89 \ + --hash=sha256:236e0d68f6941992b0811128e652590f590db444ab29ad8f1324765b9298b946 \ + --hash=sha256:29243242cf72582b65f9582ad9e56e8cb281566ed3519f4cd70bb8b9f2977e90 \ + --hash=sha256:295ecf2e57542cd836ca537486951289678c8c7d1ee6ad74ebe29b2168a003cf \ + --hash=sha256:29eb0b9427a6b11b992e452f6cceb8af724f4dceb47e779d0b35e405e996ea5e \ + --hash=sha256:3011537be6cf1de4511c0255a324362a812b58184bbe61e15f59c8b31033bd74 \ + --hash=sha256:3adef556a19eb3b630bf86a79c29d7da3d61e541472d0d897ca01b171c0abf8f \ + --hash=sha256:3bd0e340f0ebe65c548210f53be3fd8192e83964760caf0c28bef368e68b0d37 \ + --hash=sha256:4209da39d85cf240f762af622d8180fcdfcb4727d021f44ade62d613a1a43324 \ + --hash=sha256:4a378fc15fb33e321f04652c166ce73eeb8833a97c3d218132844e938cd93220 \ + --hash=sha256:502b94b0b331f7dafe32fd6b151797ddbb4f32385b362e722c783a025e5954a3 \ + --hash=sha256:528c8744b8b5e3ecb8774f86af38d2376216816e9908317ad055f3c9c2d74799 \ + --hash=sha256:5786ef987b3767e51aaa53f20aec104c0252b42ecda7aef8e8b4cbae279b05c5 \ + --hash=sha256:5d20e6c7c35fcbf2a0ebdd0eaeacfbc243009dfd33ab7822d54e213912e6dbbd \ + --hash=sha256:615b03922d40e186fd1df73156473db0ba525c639bb4fd1cf28e1694879c9b23 \ + --hash=sha256:6306086b87cf7f8a29aa618d9fd9bffb56c59247166b9660fdb54d86d7714ecd \ + --hash=sha256:6be721bd4bd57576193653b75b4af3461c9d0bf7dd8b528f782e9be210dc75bb \ + --hash=sha256:720434d48542ecfe84d32d287b727569d3fc8f5769acd39051130e490a5c295c \ + --hash=sha256:77bdaebd452f43fcfc4cc3ba94352a3ea537cd01e3f2d0879f48673d2ec00d6e \ + --hash=sha256:7c4f8f8d1d1c0e2832d8ee7113dd08f6df6c7aad9e863fcbed9f25832be0b8c4 \ + --hash=sha256:7da85fbcff6dac1abcc636707bed38b45598131fb7a605d9719c70b5cc733af8 \ + --hash=sha256:7f38f57d344f8b6accb40e01c3d83cfc590498231724d16c07ffb7940f157818 \ + --hash=sha256:82d0f94c8587ccb472622ee7795280aaa38a06640d5f45b3f16909d6dd86a989 \ + --hash=sha256:86d616fe84c6eab8026f8c50ab5bcb90db780d2ccd233d971e34e92bede9b359 \ + --hash=sha256:9279488933040b638c0ab739084c0ca100efeea6db581bf5d7628d8e89de53fe \ + --hash=sha256:970b221ed5842213d98e3c480c08f795ace4b1f81fb21e1b126bd0476bce1c34 \ + --hash=sha256:9f03a7828cca2b0adcd6662aee8f2711ff8830e1027641fdea3ab0b787483566 \ + --hash=sha256:9f459f0b4f0596d9d51fe7716b404b35287b99e77da98a7af90a65dd5fd61141 \ + --hash=sha256:9fa247e02b505a45a2775f69df38d360d197e3790bc60f717595db9eda358b6e \ + --hash=sha256:a3f86fd1313dd69d013e265f1fdcd12ea7a9d606f9875b2a3db946cd334555f3 \ + --hash=sha256:a6e851b82c0661f69f1630fc16172c68787a6a9cf0991e7c6437d60976cdcd76 \ + --hash=sha256:a96b07a49b7b1d025ce59c1b3acbcf24bead9a83da4523c4a6bde1bb94e7a0e1 \ + --hash=sha256:afa89837d9be67e0cadfa33bca3bb7efdda98c3b07e74dc3b635ebfb1c8a926a \ + --hash=sha256:ba05652540bf12828e52abae807b024b09ca144ff4f75e2450a81d69c376425b \ + --hash=sha256:bc378bdad4e9f12b5bd96466083d1e71acf00594ec9c7b2bdb5e02816f77f303 \ + --hash=sha256:c9de622fd397495695d0598ddc789222bfcfec9d7c9ec3a1e385c855e3bc5e01 \ + --hash=sha256:cb734b2668c1f40d07ce093bbeb4407e9527c67901627b0e1679825be3f09975 \ + --hash=sha256:d01c5d26a352acc6d5cf3128225477e1e8465e8d3029d4c345807fbf7f3cf093 \ + --hash=sha256:e05ff2925d8b37ad26c80c1b9dc43ae5d455da2df1e23c24c095a6425917c57e \ + --hash=sha256:e1fde197ae65be856a034a91b70ed747a16562ca69577785f06c661548424bf1 \ + --hash=sha256:e3ebe2f72a2a1156065a9851570ffbf50c0a93cdccadef9c6e05c508a4fd10b1 \ + --hash=sha256:e76172710bd4e5eeae061abfd68347e5264632e02778be61784671ae3a2132f5 \ + --hash=sha256:f83353e47ab520bf6fd4df8f5897d9fe081211f2fbc4b7d37736a3e3c166cbcf \ + --hash=sha256:f9b028548b3af480e2b1ed8df14ccaac86f99c9f600d1580770ab7ba3dcd40f0 \ + --hash=sha256:fb443e81057896132d3642d6be219e6efd158691ac7883e3ba8fcb469865f05d \ + --hash=sha256:ffda5244fd1ad71a1e54405e35f50d09b80c3978efa120f58bd1252ae32c62d2 + # via -r FileAutomation/.github/requirements/lint.in +mypy-extensions==1.1.0 \ + --hash=sha256:1be4cccdb0f2482337c4743e60421de3a356cd97508abadd57d47403e94f5505 \ + --hash=sha256:52e68efc3284861e772bbcd66823fde5ae21fd2fdb51c62a211403730b916558 + # via mypy +pathspec==1.1.1 \ + --hash=sha256:17db5ecd524104a120e173814c90367a96a98d07c45b2e10c2f3919fff91bf5a \ + --hash=sha256:a00ce642f577bf7f473932318056212bc4f8bfdf53128c78bbd5af0b9b20b189 + # via mypy +ruff==0.16.10 \ + --hash=sha256:1dfc6f0088149fb6a362c1c446bcbb3fd2157b3852fe2fa68409276eab9ad9b3 \ + --hash=sha256:25a65fe998c4e6861ec079ada5826a2fc605e6cbccbe9dcd7fac1f54e791621b \ + --hash=sha256:2a12e01cb9156c10c466f63b46eaae5ecea28dfbd21b5836353ae498e7d1349a \ + --hash=sha256:3031a4a2e8e7b8a46f70be45f198c35a11ece509a94b80334d8d397a33c67550 \ + --hash=sha256:3e70175e29cc94c26ea296c80e470180b744b7419026898e58f520c6ab32578e \ + --hash=sha256:488b0fe3f3574210e5cf80d9f59b9e3ab17a127a8155de3f307b392589cfb511 \ + --hash=sha256:494401c86df4c4c25f69b9419605d944467ee98c42fb6ad405ef4fa40b8fb67d \ + --hash=sha256:6553498afc35f580f036030795810b9e6bcea31604b0fd9e8d352795473042e3 \ + --hash=sha256:7ae7375f803b5520dc9f546bed7e3a0acb70b91812e9bb4b19927de22f25b77d \ + --hash=sha256:92e59a70bcbd9d3a5483656da906ec28edfdacfce00afd99edb8b4e9d15644be \ + --hash=sha256:97f2015c92aa97105b0eab19eb5d224884399281cfc5da86a92db4ab5e7fb2ca \ + --hash=sha256:a3b8471dea115d37f123882be852bed13403746d3a76c11de4a19ec5f9ff5a03 \ + --hash=sha256:bc2610fb269fa56dd8a68669ae470fa6272902668c0fc2ebc3aa112b2633d5b8 \ + --hash=sha256:bd83d1235a5258d318477bc5b576303974cbdef5df0c01a1bff14efcc23bd12a \ + --hash=sha256:d203abc0ff2b773ee33d00ab8df0bb67046fbc7c07b119332f08b9b345cf8221 \ + --hash=sha256:e748ff95c934c4e978783b8e687bc174e7bd84e8ad24e3243e1ecfcda5e0282d \ + --hash=sha256:eff4728c4eaae93f0955cd264d24b2ab348e74bf59986ccf282ba6dc16b3b017 \ + --hash=sha256:f33f43a864a8483eebd160e713336c8bab02c934feaff0a33cf5ccb41546d09a + # via -r FileAutomation/.github/requirements/lint.in +typing-extensions==4.16.0 \ + --hash=sha256:481caa481374e813c1b176ada14e97f1f67a4539ce9cfeb3f350d78d6370c2e8 \ + --hash=sha256:dc983d19a509c94dba722ee6abd33940f7c05a89e243c47e907eb4db6f1a43e5 + # via mypy diff --git a/.github/workflows/ci-dev.yml b/.github/workflows/ci-dev.yml index f338b1d..6b748d9 100644 --- a/.github/workflows/ci-dev.yml +++ b/.github/workflows/ci-dev.yml @@ -25,9 +25,8 @@ jobs: cache: pip - name: Install tooling run: | - python -m pip install --upgrade pip - # mypy reads je_action_core's types (py.typed) for the classes built on it - pip install ruff mypy "je_action_core>=0.0.2" + # mypy reads the action engine types from the hash-locked tooling. + python -m pip install --require-hashes --only-binary :all: -r .github/requirements/lint.txt - name: Ruff check run: ruff check automation_file tests - name: Ruff format check diff --git a/.github/workflows/ci-stable.yml b/.github/workflows/ci-stable.yml index 9db360c..3277866 100644 --- a/.github/workflows/ci-stable.yml +++ b/.github/workflows/ci-stable.yml @@ -25,9 +25,8 @@ jobs: cache: pip - name: Install tooling run: | - python -m pip install --upgrade pip - # mypy reads je_action_core's types (py.typed) for the classes built on it - pip install ruff mypy "je_action_core>=0.0.2" + # mypy reads the action engine types from the hash-locked tooling. + python -m pip install --require-hashes --only-binary :all: -r .github/requirements/lint.txt - name: Ruff check run: ruff check automation_file tests - name: Ruff format check diff --git a/.github/workflows/integration.yml b/.github/workflows/integration.yml index e72c17f..39eee64 100644 --- a/.github/workflows/integration.yml +++ b/.github/workflows/integration.yml @@ -43,9 +43,9 @@ jobs: cache: pip - name: Install the package with every extra run: | - python -m pip install --upgrade pip wheel + python -m pip install --require-hashes --only-binary :all: -r .github/requirements/integration.txt cp dev.toml pyproject.toml - pip install -e ".[all,test]" + python -m pip install --no-deps --no-build-isolation --only-binary :all: -e . - name: Start the service run: bash tests/integration/start_service.sh "$SERVICE" - name: Run the contract suite against it @@ -80,8 +80,8 @@ jobs: sudo apt-get install -y --no-install-recommends libegl1 libgl1 libxkbcommon0 libdbus-1-3 libfontconfig1 - name: Install the package with every extra run: | - python -m pip install --upgrade pip wheel + python -m pip install --require-hashes --only-binary :all: -r .github/requirements/integration.txt cp dev.toml pyproject.toml - pip install -e ".[all,test]" + python -m pip install --no-deps --no-build-isolation --only-binary :all: -e . - name: Run pytest run: python -m pytest tests/ -v --tb=short diff --git a/architecture.md b/architecture.md index 1787854..560e693 100644 --- a/architecture.md +++ b/architecture.md @@ -41,6 +41,7 @@ piece of the 1.0 roadmap (`docs/FILEAUTOMATION-1.0-ROADMAP.md`, PR #107); what i | `MANIFEST.in` | Keeps `tests/` out of both source distributions (`tests/test_sdist_manifest.py`); package discovery in the TOMLs already keeps it out of the wheels | | `scripts/dev_release.py` | Release helper for the dev channel (standard library only): picks the next `automation_file_dev` version from PyPI and tells whether the built wheel differs from the newest published one | | `.github/requirements/publish.in`, `publish.txt` | The tools of the two publish jobs (`build`, `twine`, and the build backend `setuptools`) and their hash-locked resolution for Python 3.12 on Linux. `publish.in` holds the `uv pip compile` command that regenerates `publish.txt`; Dependabot reads the directory | +| `.github/requirements/lint.in`, `lint.txt`, `integration.in`, `integration.txt` | Hash-locked wheel dependencies for lint and Python 3.12 integration jobs. The editable package uses the locked build backend without dependency resolution or build isolation. | | `main_ui.py` | Development shortcut for `launch_ui()` | | `tests/`, `docs/`, `examples/mcp/` | pytest suite (fixtures in `tests/conftest.py`); Sphinx docs; MCP host configuration example | diff --git a/automation_file/core/action_executor.py b/automation_file/core/action_executor.py index 460be8a..fac3101 100644 --- a/automation_file/core/action_executor.py +++ b/automation_file/core/action_executor.py @@ -56,7 +56,10 @@ class ActionExecutor(_CoreActionExecutor): - """Execute named actions resolved through an :class:`ActionRegistry` (je_action_core's executor).""" + """Execute named actions resolved through an :class:`ActionRegistry`. + + Uses the shared action executor's implementation. + """ registry: ActionRegistry diff --git a/automation_file/core/package_loader.py b/automation_file/core/package_loader.py index 137fd0b..0306933 100644 --- a/automation_file/core/package_loader.py +++ b/automation_file/core/package_loader.py @@ -16,7 +16,7 @@ _SETTINGS = PackageManagerSettings( import_errors=(ImportError,), - # No FA_* command reaches the loader (it is Python-only, see CLAUDE.md, Plugin / package loading), + # No FA_* command reaches the loader (Python-only, see CLAUDE.md, Plugin / package loading), # so je_action_core's package gate would only warn the host itself. tests/test_package_loader.py # fails if a command that loads packages is added; switch the gate on then (workspace X-12). gate=PackageGate.OFF, diff --git a/docs/updates/2026-10.md b/docs/updates/2026-10.md index fdcf568..63f856b 100644 --- a/docs/updates/2026-10.md +++ b/docs/updates/2026-10.md @@ -701,3 +701,10 @@ Index and query commands: [README.md](README.md). New entries go at the end. - **Still not verified**: Google Drive, OneDrive and Dropbox against their services; S3 against AWS (S3Mock was used); FTPS; UI 2.0 on a real display. - **Files**: `automation_file/storage/{webdav_storage,smb_storage}.py`, `automation_file/local/versioning.py`, `tests/integration/start_service.sh`, `.github/workflows/ci-dev.yml`, `.github/workflows/ci-stable.yml`, the test modules and the library modules that carry the markers, the three `usage/integration_tests.rst`, `CLAUDE.md`, `progress.md`. - **Open items**: #40 (SonarCloud), #19, #32. + +## U-20261008-34 · 2026-10-08 · Lock lint and integration dependencies for the main pull request · #ci #security + +- **What**: lint and integration jobs install hash-locked wheels. The local editable package uses `--no-deps --no-build-isolation`, so installation cannot resolve additional dependencies. Integration dependencies cover the base package, every storage extra, tests and the build backend. Added guards for installation and metadata parity. +- **Other fixes**: TCP wire tests generate their authentication key per run, and four overlong comments or docstrings were wrapped. +- **Files**: `.github/requirements/{lint,integration}.{in,txt}`, three workflows, `tests/test_workflow_actions.py`, `tests/test_tcp_wire.py` and wrapped comments. +- **Validation**: CI results are tracked by PR #110. diff --git a/docs/updates/README.md b/docs/updates/README.md index 47a67bb..9a40102 100644 --- a/docs/updates/README.md +++ b/docs/updates/README.md @@ -58,6 +58,7 @@ In the same commit: delete the item from `progress.md`, add a `#done` entry here | ID | Date | Title | Tags | Batch | |---|---|---|---|---| +| U-20261008-34 | 2026-10-08 | Lock lint and integration dependencies for the main pull request | #ci #security | [2026-10](2026-10.md) | | U-20261008-33 | 2026-10-08 | First CI runs of pull request #109, and what they showed | #ci #storage #incident | [2026-10](2026-10.md) | | U-20261008-32 | 2026-10-08 | An intermittent test failure, and its wrong first diagnosis | #tests #incident | [2026-10](2026-10.md) | | U-20261008-31 | 2026-10-08 | Migration guide | #docs #migration #roadmap | [2026-10](2026-10.md) | diff --git a/progress.md b/progress.md index a64cf1a..fcdcb32 100644 --- a/progress.md +++ b/progress.md @@ -28,7 +28,7 @@ Items #10 to #26 are what is left of the 1.0 roadmap (`docs/FILEAUTOMATION-1.0-R - **#39** The application layer reaches into two private places: `StorageService` reads `StorageResolver._mounts` / `_factories` (give the resolver a public listing of its mounts), and `app/` uses `pipeline.definition.retry_from_dict`, `graph.upstream_tasks`, `substitution.is_name` / `NAME_RULE` and `model.ON_SUCCESS` / `WHEN_CHOICES`, which are not in `automation_file.pipeline.__all__` (export them, or give the pipeline package the functions the editor needs). - **#37** Storage-layer gaps the MCP work found (U-20261008-27) and did not change: (a) a `LocalStorage(root)` listing reports a link's name and its target's metadata without a containment check, although reading through the link is refused; (b) `StorageBackend` has no ranged read, so reading the head of a large remote file stages all of it; (c) `LocalStorage` on Windows opens device names (`CON`, `NUL`) and alternate data streams, which the MCP tools refuse themselves; (d) a link on an SFTP or FTP server leads outside a root that is only a path prefix. - **#26** The 1.0.0 release (roadmap M9). Everything the roadmap lists is on the branch `feat/universal-storage-layer`; what is left is the owner's: review and merge the pull request to `dev`, read the first CI and integration runs (#19, #32, #18, #38), decide the 1.0 date, then write `1.0.0` in `stable.toml` and `dev.toml` in the release pull request to `main` and raise the `Development Status` classifier. Not written: a separate security page in the manual (the deployment chapter and `CLAUDE.md` § Security carry that guidance today). -- **#40** [DECIDE] SonarCloud fails pull request #109 on one condition, the security rating of new code: eight findings, all on the two `pip install` steps of the two jobs in `.github/workflows/integration.yml` (`S8541` no `--only-binary :all:`, `S8544` versions not locked). The `pytest`, `minimal` and `extras` jobs of `ci-dev.yml` install the same way and are not flagged only because their lines are not new. Either accept the findings in SonarCloud, or lock the test dependencies for every job: a `test.in` / `test.txt` pair next to `.github/requirements/publish.in`, generated with `uv pip compile` for the platforms the jobs run on, and `pip install --require-hashes --only-binary :all:` in each job. The lock could not be generated on the development machine (no `uv`, and a network too slow to resolve every extra). +- **#40** [VERIFY] PR #110 replaces unlocked lint and integration installs with hash-locked wheel dependencies. Confirm SonarCloud and all platform integration jobs pass before main is merged. - **#34** [BLOCKED] PyPI Trusted Publishing (roadmap §13). `publish.yml` and `publish-dev` still upload with the `PYPI_API_TOKEN` secret. Switching needs the owner to add a trusted publisher for each project on PyPI (`automation_file`: workflow `publish.yml`; `automation_file_dev`: workflow `ci-dev.yml`; an environment name if one is wanted) before the workflows can drop the token for `id-token: write` and `pypa/gh-action-pypi-publish`. Changing the workflows first would break both channels. ### Packaging follow-ups diff --git a/tests/test_dev_toml_parity.py b/tests/test_dev_toml_parity.py index e8b1121..d662417 100644 --- a/tests/test_dev_toml_parity.py +++ b/tests/test_dev_toml_parity.py @@ -60,7 +60,7 @@ def test_shipped_files_match(): @pytest.mark.parametrize("metadata", [STABLE_FILE, DEV_FILE], ids=["stable.toml", "dev.toml"]) def test_only_the_library_is_packaged(metadata): - # ``tests`` has an ``__init__.py``, so discovery without ``include`` installs the test suite as a + # ``tests`` has an ``__init__.py``; discovery without ``include`` installs it as a # top-level package next to ``automation_file``. find = metadata["tool"]["setuptools"]["packages"]["find"] assert find["include"] == ["automation_file", "automation_file.*"] diff --git a/tests/test_tcp_wire.py b/tests/test_tcp_wire.py index 0f364fb..cea079c 100644 --- a/tests/test_tcp_wire.py +++ b/tests/test_tcp_wire.py @@ -3,6 +3,7 @@ from __future__ import annotations import json +import secrets import socket import time @@ -13,7 +14,7 @@ from automation_file.server.tcp_server import start_autocontrol_socket_server END = b"Return_Data_Over_JE\n" -SECRET = "s3cr3t" +SECRET = secrets.token_hex(16) def _wire_echo(value: str) -> str: diff --git a/tests/test_workflow_actions.py b/tests/test_workflow_actions.py index 4e7a321..4213f30 100644 --- a/tests/test_workflow_actions.py +++ b/tests/test_workflow_actions.py @@ -202,6 +202,43 @@ def _is_locked(requirement: Requirement, pins: dict[str, str]) -> bool: return version is not None and requirement.specifier.contains(version) +@pytest.mark.parametrize("workflow", _WORKFLOWS, ids=lambda path: path.name) +def test_lint_and_integration_jobs_cannot_resolve_unlocked_dependencies(workflow): + jobs = dict(_jobs(workflow)) + for name in ("lint", "services", "platforms"): + if name not in jobs: + continue + commands = _commands(jobs[name], _PIP_INSTALL) + assert commands + for command in commands: + if "-e ." in command: + assert "--no-deps" in command + assert "--no-build-isolation" in command + else: + assert "--require-hashes" in command + assert "--only-binary :all:" in command + + +def test_integration_lock_covers_the_package_and_its_build_backend(): + text = (_REQUIREMENTS / "integration.txt").read_text(encoding="utf-8") + pins = {canonicalize_name(name): version for name, version in _PIN.findall(text)} + with (_ROOT / "dev.toml").open("rb") as handle: + metadata = tomllib.load(handle) + project = metadata["project"] + requirements = ( + project["dependencies"] + + project["optional-dependencies"]["all"] + + project["optional-dependencies"]["test"] + + metadata["build-system"]["requires"] + ) + for item in requirements: + requirement = Requirement(item) + # The integration runners all use Python 3.12. + if requirement.marker and 'python_version < "3.11"' in str(requirement.marker): + continue + assert _is_locked(requirement, pins), item + + def test_the_publish_jobs_are_the_two_known_ones(): # A new job that is given the token has to be looked at against the rule below. assert [name for name, _body in _publish_jobs()] == [ @@ -221,7 +258,7 @@ def test_publish_jobs_install_only_hash_locked_tools(job): @pytest.mark.parametrize("job", _publish_jobs(), ids=lambda job: job[0]) def test_publish_jobs_build_with_the_locked_backend(job): - # An isolated build downloads the newest setuptools of that minute, outside publish.txt, and runs + # An isolated build downloads the newest setuptools outside publish.txt and runs # it beside the token. --no-isolation builds with the backend the locked install put in the job. _name, body = job builds = _commands(body, _BUILD) From 1f416ad938045997b8e6096d1f93f72defba8013 Mon Sep 17 00:00:00 2001 From: JeffreyChen Date: Thu, 8 Oct 2026 17:24:38 +0800 Subject: [PATCH 55/59] Freeze the limiter clock while testing concurrent burst capacity --- tests/test_rate_limit.py | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/tests/test_rate_limit.py b/tests/test_rate_limit.py index c33a0c2..3bd986d 100644 --- a/tests/test_rate_limit.py +++ b/tests/test_rate_limit.py @@ -4,9 +4,11 @@ import threading import time +from types import SimpleNamespace import pytest +from automation_file.core import rate_limit from automation_file.core.rate_limit import RateLimiter from automation_file.exceptions import RateLimitExceededException @@ -59,7 +61,10 @@ def step(x: int) -> int: assert calls == [0, 1, 2] -def test_concurrent_acquires_serialize() -> None: +def test_concurrent_acquires_serialize(monkeypatch) -> None: + # A busy runner may take longer than one refill interval to start the threads. + # Freeze only the limiter's clock, preserving real time for thread scheduling. + monkeypatch.setattr(rate_limit, "time", SimpleNamespace(monotonic=lambda: 0.0)) limiter = RateLimiter(rate=20, burst=2) results: list[bool] = [] lock = threading.Lock() From 1a2718f6d4d8b678f54b7c7da5af0b365fda5007 Mon Sep 17 00:00:00 2001 From: JeffreyChen Date: Thu, 8 Oct 2026 17:55:34 +0800 Subject: [PATCH 56/59] Close the shared Qt application before interpreter shutdown --- architecture.md | 2 +- docs/updates/2026-10.md | 6 ++++++ docs/updates/README.md | 1 + tests/conftest.py | 14 ++++++++++++++ tests/test_ui_pages.py | 8 -------- tests/test_ui_pipeline_editor.py | 8 -------- tests/test_ui_smoke.py | 8 -------- 7 files changed, 22 insertions(+), 25 deletions(-) diff --git a/architecture.md b/architecture.md index 560e693..785fa4a 100644 --- a/architecture.md +++ b/architecture.md @@ -43,7 +43,7 @@ piece of the 1.0 roadmap (`docs/FILEAUTOMATION-1.0-ROADMAP.md`, PR #107); what i | `.github/requirements/publish.in`, `publish.txt` | The tools of the two publish jobs (`build`, `twine`, and the build backend `setuptools`) and their hash-locked resolution for Python 3.12 on Linux. `publish.in` holds the `uv pip compile` command that regenerates `publish.txt`; Dependabot reads the directory | | `.github/requirements/lint.in`, `lint.txt`, `integration.in`, `integration.txt` | Hash-locked wheel dependencies for lint and Python 3.12 integration jobs. The editable package uses the locked build backend without dependency resolution or build isolation. | | `main_ui.py` | Development shortcut for `launch_ui()` | -| `tests/`, `docs/`, `examples/mcp/` | pytest suite (fixtures in `tests/conftest.py`); Sphinx docs; MCP host configuration example | +| `tests/`, `docs/`, `examples/mcp/` | pytest suite (fixtures, including the shared Qt application lifecycle, in `tests/conftest.py`); Sphinx docs; MCP host configuration example | ## 3. Entry points and public interfaces diff --git a/docs/updates/2026-10.md b/docs/updates/2026-10.md index 63f856b..f828526 100644 --- a/docs/updates/2026-10.md +++ b/docs/updates/2026-10.md @@ -708,3 +708,9 @@ Index and query commands: [README.md](README.md). New entries go at the end. - **Other fixes**: TCP wire tests generate their authentication key per run, and four overlong comments or docstrings were wrapped. - **Files**: `.github/requirements/{lint,integration}.{in,txt}`, three workflows, `tests/test_workflow_actions.py`, `tests/test_tcp_wire.py` and wrapped comments. - **Validation**: CI results are tracked by PR #110. + +## U-20261008-35 · 2026-10-08 · Share one Qt application and shut it down after GUI tests · #done #tests #gui + +- **What**: the three GUI modules use one session fixture. After widgets are released, it closes windows, processes pending events and explicitly shuts down the application. Previously each module kept its own fixture and application cleanup ran during interpreter shutdown. +- **Validation**: 121 GUI tests pass; Ruff and formatting checks pass. PR #110 verifies the Windows process exit on all supported Python versions. +- **Files**: `tests/conftest.py`, `tests/test_ui_smoke.py`, `tests/test_ui_pages.py`, `tests/test_ui_pipeline_editor.py`, `architecture.md`. diff --git a/docs/updates/README.md b/docs/updates/README.md index 9a40102..247b41f 100644 --- a/docs/updates/README.md +++ b/docs/updates/README.md @@ -58,6 +58,7 @@ In the same commit: delete the item from `progress.md`, add a `#done` entry here | ID | Date | Title | Tags | Batch | |---|---|---|---|---| +| U-20261008-35 | 2026-10-08 | Share one Qt application and shut it down after GUI tests | #done #tests #gui | [2026-10](2026-10.md) | | U-20261008-34 | 2026-10-08 | Lock lint and integration dependencies for the main pull request | #ci #security | [2026-10](2026-10.md) | | U-20261008-33 | 2026-10-08 | First CI runs of pull request #109, and what they showed | #ci #storage #incident | [2026-10](2026-10.md) | | U-20261008-32 | 2026-10-08 | An intermittent test failure, and its wrong first diagnosis | #tests #incident | [2026-10](2026-10.md) | diff --git a/tests/conftest.py b/tests/conftest.py index 235e08e..c93ab97 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -27,3 +27,17 @@ def sample_dir(tmp_path: Path) -> Path: nested.mkdir() (nested / "d.txt").write_text("d", encoding="utf-8") return root + + +@pytest.fixture(scope="session") +def qt_app(): + """Keep one application alive until every GUI module has released its widgets.""" + import os + + os.environ.setdefault("QT_QPA_PLATFORM", "offscreen") + widgets = pytest.importorskip("PySide6.QtWidgets") + app = widgets.QApplication.instance() or widgets.QApplication([]) + yield app + app.closeAllWindows() + app.processEvents() + app.shutdown() diff --git a/tests/test_ui_pages.py b/tests/test_ui_pages.py index 806326a..446e168 100644 --- a/tests/test_ui_pages.py +++ b/tests/test_ui_pages.py @@ -74,14 +74,6 @@ def send(self, subject: str, body: str, level: str = "info") -> None: raise NotificationException("POST https://hooks.example.com/services/s3cr3t failed") -@pytest.fixture(name="qt_app", scope="module") -def _qt_app(): - from PySide6.QtWidgets import QApplication - - app = QApplication.instance() or QApplication([]) - yield app - - @pytest.fixture(autouse=True) def _clean_global_state() -> Iterator[None]: clear_memory_stores() diff --git a/tests/test_ui_pipeline_editor.py b/tests/test_ui_pipeline_editor.py index c8de231..1ab2562 100644 --- a/tests/test_ui_pipeline_editor.py +++ b/tests/test_ui_pipeline_editor.py @@ -76,14 +76,6 @@ def registry(self) -> ActionRegistry: ) -@pytest.fixture(name="qt_app", scope="module") -def _qt_app(): - from PySide6.QtWidgets import QApplication - - app = QApplication.instance() or QApplication([]) - yield app - - @pytest.fixture(name="workshop") def _workshop() -> Iterator[_Workshop]: workshop = _Workshop() diff --git a/tests/test_ui_smoke.py b/tests/test_ui_smoke.py index 3bcc342..801fa1d 100644 --- a/tests/test_ui_smoke.py +++ b/tests/test_ui_smoke.py @@ -45,14 +45,6 @@ ) -@pytest.fixture(name="qt_app", scope="module") -def _qt_app(): - from PySide6.QtWidgets import QApplication - - app = QApplication.instance() or QApplication([]) - yield app - - @pytest.fixture(name="window") def _window(qt_app) -> Iterator: from automation_file.ui.main_window import MainWindow From 58b6bc8c13d61fbe6ddc07e70e4e20995e65f4ec Mon Sep 17 00:00:00 2001 From: JeffreyChen Date: Thu, 8 Oct 2026 18:04:20 +0800 Subject: [PATCH 57/59] Keep GUI installs on the verified Qt 6.11 series --- .github/requirements/integration.in | 2 +- .github/requirements/integration.txt | 58 ++++++++++++++-------------- dev.toml | 4 +- docs/updates/2026-10.md | 6 +++ docs/updates/README.md | 1 + requirements.txt | 2 +- stable.toml | 4 +- 7 files changed, 42 insertions(+), 35 deletions(-) diff --git a/.github/requirements/integration.in b/.github/requirements/integration.in index 204c1a7..4a66d5c 100644 --- a/.github/requirements/integration.in +++ b/.github/requirements/integration.in @@ -28,6 +28,6 @@ fsspec>=2024.2.0 msal>=1.39.0 boxsdk>=10.15.0,<11 pyarrow>=25.0.1 -PySide6>=6.6.0 +PySide6>=6.11.2,<6.12 pytest>=8.0.0 pytest-cov>=5.0.0 diff --git a/.github/requirements/integration.txt b/.github/requirements/integration.txt index 5559258..709249b 100644 --- a/.github/requirements/integration.txt +++ b/.github/requirements/integration.txt @@ -1,5 +1,5 @@ # This file was autogenerated by uv via the following command: -# uv pip compile FileAutomation/.github/requirements/integration.in --universal --generate-hashes --python-version 3.12 --only-binary :all: --exclude-newer 2026-10-02T00:00:00Z -o FileAutomation/.github/requirements/integration.txt +# uv pip compile .github/requirements/integration.in --python-version 3.12 --universal --generate-hashes --only-binary=:all: --exclude-newer 2026-10-02 -o .github/requirements/integration.txt azure-core==1.41.0 \ --hash=sha256:522b4011e8180b1a3dcd2024396a4e7fe9ac37fb8597db47163d230b5efe892d \ --hash=sha256:f46ff5dfcd230f25cf1c19e8a34b8dc08a337b2503e268bb600a16c00db8ad5a @@ -7,7 +7,7 @@ azure-core==1.41.0 \ azure-storage-blob==12.31.0 \ --hash=sha256:0c0cb601d3462491d09ea96023cd791bb9dd4b173bf950daf3cff34ff47ba5b5 \ --hash=sha256:997b393cfcbdc4b186d5911790d91f80387f7edc12c4d73eab963a2d26e5b2a9 - # via -r FileAutomation/.github/requirements/integration.in + # via -r .github/requirements/integration.in bcrypt==5.0.0 \ --hash=sha256:046ad6db88edb3c5ece4369af997938fb1c19d6a699b9c1b27b0db432faae4c4 \ --hash=sha256:0c418ca99fd47e9c59a301744d63328f17798b5947b0f791e9af3c1c499c2d0a \ @@ -76,7 +76,7 @@ bcrypt==5.0.0 \ boto3==1.43.107 \ --hash=sha256:632a4f8298725b94144c8cd383cbdacb34624daa73221e954c3d33ac6e2528d6 \ --hash=sha256:c4e0f1a0295cbb7103f2950128cf88463c076d220080a7d0f127cf834969fc3a - # via -r FileAutomation/.github/requirements/integration.in + # via -r .github/requirements/integration.in botocore==1.43.107 \ --hash=sha256:23cbe854e815dbaccf097f7fd32b461c9e1d2ed7e0c7dcc5658218704509d840 \ --hash=sha256:4a37fa072a00280c746313532d19b00e2dc53f1993222601df104d71d548d5b6 @@ -86,7 +86,7 @@ botocore==1.43.107 \ boxsdk==10.17.0 \ --hash=sha256:86c9a1f665c8879af501c02ec51bbe02247b24f96c6bc39f6890912e70091840 \ --hash=sha256:fe42658185901b80437a0d5b393c91b4a732f3b78d1be11d881aa7484807c9d5 - # via -r FileAutomation/.github/requirements/integration.in + # via -r .github/requirements/integration.in certifi==2026.7.22 \ --hash=sha256:62f22742b58a1a33014a2b6b706588a8d7e2a88ae7bd1a6ebe8c992928483775 \ --hash=sha256:741e2c3b351ddf169a738da9f2c048608ff7f2c5cc02f1ebc6b118bb090d5d55 @@ -559,7 +559,7 @@ cryptography==50.0.2 \ --hash=sha256:fa8f5efb344d6908a1ce62f4a24e2e5780f825d6f53f5f50ec5ffacac72936cb \ --hash=sha256:fdd28f912fccfec1846a94e2e1e8f9b0012f557f0c46fe4f3eb0d7a87afcf90b # via - # -r FileAutomation/.github/requirements/integration.in + # -r .github/requirements/integration.in # azure-storage-blob # google-auth # msal @@ -570,15 +570,15 @@ cryptography==50.0.2 \ defusedxml==0.7.1 \ --hash=sha256:1bb3032db185915b62d7c6209c5a8792be6a32ab2fedacc84e01b52c51aa3e69 \ --hash=sha256:a352e7e428770286cc899e2542b6cdaedb2b4953ff269a210103ec58f6198a61 - # via -r FileAutomation/.github/requirements/integration.in + # via -r .github/requirements/integration.in dropbox==12.2.2 \ --hash=sha256:044d963cdee83a5149e103c792b1a0284f681bb1b6984259cda01244cc3c1a6e \ --hash=sha256:9abc636a57788165c1e1f263971163230ccc26fd448df643c1005e0dc1abd1a3 - # via -r FileAutomation/.github/requirements/integration.in + # via -r .github/requirements/integration.in fsspec==2026.9.0 \ --hash=sha256:0f08147951c8cb31d844c3547d631053b127863b60be04cf06e121333ee0e2fe \ --hash=sha256:8dd6e646e99ea382bd85f97a45e6b526a442d79423a7dc673f1e2756d05fcb5f - # via -r FileAutomation/.github/requirements/integration.in + # via -r .github/requirements/integration.in google-api-core==2.40.0 \ --hash=sha256:4b9e0a80024c269ae173136d5439f4ed284651d6e5e2773ba5c694f481f9c0f4 \ --hash=sha256:ebee7d1b138b5362beecec260e6e8988ac97346562c7382ccb6f0ad8435c599f @@ -586,7 +586,7 @@ google-api-core==2.40.0 \ google-api-python-client==2.201.0 \ --hash=sha256:2d9bf1ba3f12eee8ed3d0f1791ce0605d163432f496baa72d3677faa2cf097d6 \ --hash=sha256:d5691982abd7287f53cb0b0e0c6a9984d4103cf864ea0a88cb6e4347bbaf70de - # via -r FileAutomation/.github/requirements/integration.in + # via -r .github/requirements/integration.in google-auth==2.59.1 \ --hash=sha256:89c3f931683a482ac97e61df7eb9da5e08a91703f3c752b2377d72cfb7d69e6a \ --hash=sha256:ce50fc533ac02f489a2b183a0c156672c376ecb2091b1127bc7efba2975fff27 @@ -599,12 +599,12 @@ google-auth-httplib2==0.4.4 \ --hash=sha256:b931de392c20cfaa351cd789274922bd8cdc001e0e9e96de31b39d71347f8e16 \ --hash=sha256:bbe5d7b2401bb3a4017f4720e1e91bd273ab9a2bb60b84e65edbc0de127852da # via - # -r FileAutomation/.github/requirements/integration.in + # -r .github/requirements/integration.in # google-api-python-client google-auth-oauthlib==1.5.0 \ --hash=sha256:71625fdea21c6c03217eb9ff13741c3096e6d257ebca0ad084dbd6e9f4aa4ef4 \ --hash=sha256:b351107c7dd9017f426cbb0272ea1bc04f469020fd18f4443c7d40362e0b1510 - # via -r FileAutomation/.github/requirements/integration.in + # via -r .github/requirements/integration.in googleapis-common-protos==1.75.5 \ --hash=sha256:c7a866fc34ed29a3b10af627a4b9b1dc2433313ca6e959f0ae4feb132047ed72 \ --hash=sha256:d7285525c23039db98f2463e6d5a4f9b958b94d497f03a844ece3259c4e72d5d @@ -634,7 +634,7 @@ isodate==0.7.2 \ je-action-core==0.0.3 \ --hash=sha256:395890665483e58f17fede17aa809bf6e2762ba8f0bd6751126fe46efabb8c85 \ --hash=sha256:7b73ab0172a3c90c91b0a81b8a662350199a35842f529acf7a54706052fb38ee - # via -r FileAutomation/.github/requirements/integration.in + # via -r .github/requirements/integration.in jinja2==3.1.6 \ --hash=sha256:0137fb05990d35f1275a587e9aee6d56da821fc83491a0fb838183be43f66d6d \ --hash=sha256:85ece4451f492d0c13c5dd7c13a64681a86afae63a5f347908daf103ce6d2f67 @@ -739,7 +739,7 @@ markupsafe==3.0.3 \ msal==1.39.0 \ --hash=sha256:2d2577886906cd7293850dffa2da29119966c213bfc6ec0cecf8bf7621e1ca77 \ --hash=sha256:6ab7de335e6d7f5717e2c7e1dbf86e4dda2f6acf3c56773b78dc53ebc6395b5f - # via -r FileAutomation/.github/requirements/integration.in + # via -r .github/requirements/integration.in oauthlib==4.0.0 \ --hash=sha256:624c28c13a0a59cabf9747dfa52af63be3e512a7f2714df16e91b5b3a145e6cd \ --hash=sha256:efb274799819440f95b4ab3b818869f1ce9ae26c5beacba0201d1a1b76b54f86 @@ -748,14 +748,14 @@ opentelemetry-api==1.45.0 \ --hash=sha256:711ede81773c8025c2c03dac0450bc89f3d30aea6eabcc815c570d4e35a963f7 \ --hash=sha256:80e068aba7cd56c8b58512d6a36f8d25cb1dfaa0c0a4cc1c938ccf9f362d9cb3 # via - # -r FileAutomation/.github/requirements/integration.in + # -r .github/requirements/integration.in # google-api-core # opentelemetry-sdk # opentelemetry-semantic-conventions opentelemetry-sdk==1.45.0 \ --hash=sha256:20caa5130505e386c67c3da1c76e446c842698ced54c76c6148679539aa97972 \ --hash=sha256:5dc634c946546f61b757c5b1781f9fd1a10e96ffaf7357c797e508fe9c57e75e - # via -r FileAutomation/.github/requirements/integration.in + # via -r .github/requirements/integration.in opentelemetry-semantic-conventions==0.66b0 \ --hash=sha256:175b19dd98c4473f4f43a2b1df59186fd7b4a48cd3f77cbe03438b6d6fda230a \ --hash=sha256:97a77dce484c54861e7eeff7651fd8a806dd3c30e501dc316730215ec36890e6 @@ -770,7 +770,7 @@ packaging==26.3 \ paramiko==5.0.0 \ --hash=sha256:36763b5b95c2a0dcfdf1abc48e48156ee425b21efe2f0e787c2dd5a95c0e5e79 \ --hash=sha256:b7044611c30140d9a75261653210e2002977b71a0497ff3ba0d98d7edbf62f7c - # via -r FileAutomation/.github/requirements/integration.in + # via -r .github/requirements/integration.in pluggy==1.6.0 \ --hash=sha256:7dcc130b76258d33b90f61b658791dede3486c3e6bfb003ee5c9bfb396dd22f3 \ --hash=sha256:e920276dd6813095e9377c0bc5566d94c932c33b27a3e3945d8389c374dd4746 @@ -780,7 +780,7 @@ pluggy==1.6.0 \ prometheus-client==0.26.0 \ --hash=sha256:04a91bcf94e2cf74a44a1a874d651a2e853ed354b6e822f3b7487751465d5c2b \ --hash=sha256:fa93d06737aa02bacd05794768508bb97d2fbee28cb3bca04eaae92f0ca953d6 - # via -r FileAutomation/.github/requirements/integration.in + # via -r .github/requirements/integration.in proto-plus==1.29.0 \ --hash=sha256:8acd070469a7aaf43f440b022ef9757c8cac1a9f866e933f59ae98669ddc6c8b \ --hash=sha256:cfb4e62ad7e13dd18f346cabbda00cab39930d36a05791fd81ddb074d6ee884f @@ -842,7 +842,7 @@ pyarrow==25.0.1 \ --hash=sha256:eb6203482ff3746a5632303a7279ae0b5a304c46985b49ed1378cb350ea6728d \ --hash=sha256:f3831aaa25c67a99f99dc8b05873cb9d64560390372e2aa197ce9dd4a3f06a44 \ --hash=sha256:f729cfdbd36fd99d543b67a914d2de044c84ebe45be8b34902b299b608c15c8f - # via -r FileAutomation/.github/requirements/integration.in + # via -r .github/requirements/integration.in pyasn1==0.6.4 \ --hash=sha256:9c447d8431c947fe4c8febc4ed9e760bc29011a5b01e5c74b67025bd9fb8ce81 \ --hash=sha256:deda9277cfd454080ec40b207fb6df82206a3a2688735233cdcd8d3d565f088b @@ -900,7 +900,7 @@ pyside6==6.11.2 \ --hash=sha256:3201d67e3c10be2eaedd3910ff0f02351eca7e88c95a291cde5e7f2f55ef207f \ --hash=sha256:57fe867a6c93821a085d74e8a26f863fc2363516848181c466032595b33b1729 \ --hash=sha256:dc6d03990489a5085842770718392ef5de36352d8f608d9a9fe03e60af4d7b66 - # via -r FileAutomation/.github/requirements/integration.in + # via -r .github/requirements/integration.in pyside6-addons==6.11.2 \ --hash=sha256:354c574a839f7d0751960ba03f33bec87c95e0f2796dca3287f54d03abe97dac \ --hash=sha256:a2ca3c73e5f060d4aca49451abd9fba68dd0e1c66ff40cea3a1fe0415def579a \ @@ -925,12 +925,12 @@ pytest==9.1.1 \ --hash=sha256:1088fbde8f2b49d95a549a195707afa7a76a3ce9bcadc26b6d71f0ffda5fe313 \ --hash=sha256:37a86b45efb9a47a61a36449063e8e18d0cab3161329fc099eb21783169c4f0c # via - # -r FileAutomation/.github/requirements/integration.in + # -r .github/requirements/integration.in # pytest-cov pytest-cov==7.1.0 \ --hash=sha256:30674f2b5f6351aa09702a9c8c364f6a01c27aae0c1366ae8016160d1efc56b2 \ --hash=sha256:a0461110b7865f9a271aa1b51e516c9a95de9d696734a2f71e3e78f46e1d4678 - # via -r FileAutomation/.github/requirements/integration.in + # via -r .github/requirements/integration.in python-dateutil==2.9.0.post0 \ --hash=sha256:37dd54208da7e1cd875388217d5e00ebd4179249f90fb72437e91a35459a0ad3 \ --hash=sha256:a8b2bc7bffae282281c8140a97d3aa9c14da0b136dfe83f850eea9a5f7470427 @@ -1009,12 +1009,12 @@ pyyaml==6.0.3 \ --hash=sha256:f7057c9a337546edc7973c0d3ba84ddcdf0daa14533c2065749c9075001090e6 \ --hash=sha256:fa160448684b4e94d80416c0fa4aac48967a969efe22931448d853ada8baf926 \ --hash=sha256:fc09d0aa354569bc501d4e787133afc08552722d3ab34836a80547331bb5d4a0 - # via -r FileAutomation/.github/requirements/integration.in + # via -r .github/requirements/integration.in requests==2.34.2 \ --hash=sha256:2a0d60c172f83ac6ab31e4554906c0f3b3588d37b5cb939b1c061f4907e278e0 \ --hash=sha256:f288924cae4e29463698d6d60bc6a4da69c89185ad1e0bcc4104f584e960b9ed # via - # -r FileAutomation/.github/requirements/integration.in + # -r .github/requirements/integration.in # azure-core # boxsdk # dropbox @@ -1032,7 +1032,7 @@ s3transfer==0.19.2 \ setuptools==84.0.0 \ --hash=sha256:51a52592b3b99e102b609654876bd65f19f999935166d1352678931132b0c670 \ --hash=sha256:f4695c21257f0d9b537ec2692c941d02ee143b7cc1276941349a546573b2ef73 - # via -r FileAutomation/.github/requirements/integration.in + # via -r .github/requirements/integration.in shiboken6==6.11.2 \ --hash=sha256:4bbbd6fa4d7cff5ec5e12bc4c10e1d845fa30c89457ecb13cc64f7bddb77f6f9 \ --hash=sha256:53659683b1f7a08e9f87eff9b1065f1ceb7110cd7a4bc09fdf5efe43d286604d \ @@ -1050,7 +1050,7 @@ six==1.17.0 \ smbprotocol==1.17.0 \ --hash=sha256:bcc27edfff7d727a7eb30424138766e1ee216ae2ffc1af03dab0448f29c4731c \ --hash=sha256:bd1abff5417f5af83ca516a64ab8e5acece3dbdcf58d4e5e23e47f5165a77349 - # via -r FileAutomation/.github/requirements/integration.in + # via -r .github/requirements/integration.in sspilib==0.6.0 ; sys_platform == 'win32' \ --hash=sha256:03335e14f8563e350506b47a15887a99f08102a80cfb6bd715ad83ab1c2c89ae \ --hash=sha256:0482631c67a57d8710ecbba3bbe49a9fb60e153dfcfee6d1f9b8a8ff6e419548 \ @@ -1089,7 +1089,7 @@ stone==3.5.5 \ tqdm==4.70.1 \ --hash=sha256:c293e525e6fef9c20e8728fd4612df02a0aa31bb5fe91ecd93e123b1b7bffa73 \ --hash=sha256:cefd0eca11b2a37a3aee776544d4f4ae913f02688135b5556b8788dfa474afc4 - # via -r FileAutomation/.github/requirements/integration.in + # via -r .github/requirements/integration.in typing-extensions==4.16.0 \ --hash=sha256:481caa481374e813c1b176ada14e97f1f67a4539ce9cfeb3f350d78d6370c2e8 \ --hash=sha256:dc983d19a509c94dba722ee6abd33940f7c05a89e243c47e907eb4db6f1a43e5 @@ -1102,7 +1102,7 @@ typing-extensions==4.16.0 \ tzdata==2026.4 ; sys_platform == 'win32' \ --hash=sha256:c2169a8b0a7a5e9674da5a135ccdfb2b3e671b333ed9fed17b41f73c34476e81 \ --hash=sha256:f1b8bd365d8d210c55353f4d7f8d6d8561c0ba50d704b700d195a9424bba0d79 - # via -r FileAutomation/.github/requirements/integration.in + # via -r .github/requirements/integration.in uritemplate==4.2.0 \ --hash=sha256:480c2ed180878955863323eea31b0ede668795de182617fef9c6ca09e6ec9d0e \ --hash=sha256:962201ba1c4edcab02e60f9a0d3821e82dfc5d2d6662a21abd533879bdb8a686 @@ -1145,8 +1145,8 @@ watchdog==6.0.0 \ --hash=sha256:e6f0e77c9417e7cd62af82529b10563db3423625c5fce018430b249bf977f9e8 \ --hash=sha256:e7631a77ffb1f7d2eefa4445ebbee491c720a5661ddf6df3498ebecae5ed375c \ --hash=sha256:ef810fbf7b781a5a593894e4f439773830bdecb885e6880d957d5b9382a960d2 - # via -r FileAutomation/.github/requirements/integration.in + # via -r .github/requirements/integration.in wheel==0.48.0 \ --hash=sha256:3217dcc807155e45db462d7ef2431f5ddda0d7273b700d05a67b271ceb1287ab \ --hash=sha256:94800765601e9171bf5d58d066e640662842bcedcbab982b2c90787a2c987322 - # via -r FileAutomation/.github/requirements/integration.in + # via -r .github/requirements/integration.in diff --git a/dev.toml b/dev.toml index f1bcbad..82ba666 100644 --- a/dev.toml +++ b/dev.toml @@ -71,7 +71,7 @@ fsspec = ["fsspec>=2024.2.0"] onedrive = ["msal>=1.39.0"] box = ["boxsdk>=10.15.0,<11"] parquet = ["pyarrow>=25.0.1"] -gui = ["PySide6>=6.6.0"] +gui = ["PySide6>=6.11.2,<6.12"] # Everything above. The list is written out because the two channels have different package names, # so neither can refer to its own extras by name and stay identical to the other. all = [ @@ -87,7 +87,7 @@ all = [ "msal>=1.39.0", "boxsdk>=10.15.0,<11", "pyarrow>=25.0.1", - "PySide6>=6.6.0" + "PySide6>=6.11.2,<6.12" ] test = [ "pytest>=8.0.0", diff --git a/docs/updates/2026-10.md b/docs/updates/2026-10.md index f828526..a477c25 100644 --- a/docs/updates/2026-10.md +++ b/docs/updates/2026-10.md @@ -714,3 +714,9 @@ Index and query commands: [README.md](README.md). New entries go at the end. - **What**: the three GUI modules use one session fixture. After widgets are released, it closes windows, processes pending events and explicitly shuts down the application. Previously each module kept its own fixture and application cleanup ran during interpreter shutdown. - **Validation**: 121 GUI tests pass; Ruff and formatting checks pass. PR #110 verifies the Windows process exit on all supported Python versions. - **Files**: `tests/conftest.py`, `tests/test_ui_smoke.py`, `tests/test_ui_pages.py`, `tests/test_ui_pipeline_editor.py`, `architecture.md`. + +## U-20261008-36 · 2026-10-08 · Keep the GUI on the tested Qt 6.11 series · #incident #gui #ci + +- **Finding**: Windows CI with PySide6 6.12.0 passes GUI assertions but crashes during QApplication shutdown. Moving cleanup into the fixture exposes the access violation at `tests/conftest.py`, instead of an unexplained nonzero interpreter exit. +- **Fix**: both package metadata variants, the development requirements and integration input require `PySide6>=6.11.2,<6.12`; the integration lock is regenerated. This keeps fresh GUI installs on the version series that passed the full local suite (5,769 passed, 257 skipped). +- **Files**: `dev.toml`, `stable.toml`, `requirements.txt`, `.github/requirements/integration.in`, `.github/requirements/integration.txt`. diff --git a/docs/updates/README.md b/docs/updates/README.md index 247b41f..73bb44d 100644 --- a/docs/updates/README.md +++ b/docs/updates/README.md @@ -58,6 +58,7 @@ In the same commit: delete the item from `progress.md`, add a `#done` entry here | ID | Date | Title | Tags | Batch | |---|---|---|---|---| +| U-20261008-36 | 2026-10-08 | Keep the GUI on the tested Qt 6.11 series | #incident #gui #ci | [2026-10](2026-10.md) | | U-20261008-35 | 2026-10-08 | Share one Qt application and shut it down after GUI tests | #done #tests #gui | [2026-10](2026-10.md) | | U-20261008-34 | 2026-10-08 | Lock lint and integration dependencies for the main pull request | #ci #security | [2026-10](2026-10.md) | | U-20261008-33 | 2026-10-08 | First CI runs of pull request #109, and what they showed | #ci #storage #incident | [2026-10](2026-10.md) | diff --git a/requirements.txt b/requirements.txt index 2c0f480..abb6ac9 100644 --- a/requirements.txt +++ b/requirements.txt @@ -2,7 +2,7 @@ automation_file[all] google-api-python-client google-auth-httplib2 google-auth-oauthlib -PySide6 +PySide6>=6.11.2,<6.12 requests protobuf tqdm diff --git a/stable.toml b/stable.toml index 24e0038..d01eff1 100644 --- a/stable.toml +++ b/stable.toml @@ -69,7 +69,7 @@ fsspec = ["fsspec>=2024.2.0"] onedrive = ["msal>=1.39.0"] box = ["boxsdk>=10.15.0,<11"] parquet = ["pyarrow>=25.0.1"] -gui = ["PySide6>=6.6.0"] +gui = ["PySide6>=6.11.2,<6.12"] # Everything above. The list is written out because the two channels have different package names, # so neither can refer to its own extras by name and stay identical to the other. all = [ @@ -85,7 +85,7 @@ all = [ "msal>=1.39.0", "boxsdk>=10.15.0,<11", "pyarrow>=25.0.1", - "PySide6>=6.6.0" + "PySide6>=6.11.2,<6.12" ] test = [ "pytest>=8.0.0", From ef8c06f16d934fc70e5d867b4ebf3eaa59686a51 Mon Sep 17 00:00:00 2001 From: JeffreyChen Date: Thu, 8 Oct 2026 18:17:31 +0800 Subject: [PATCH 58/59] Archive the completed main pull request validation --- docs/updates/2026-10.md | 7 +++++++ docs/updates/README.md | 1 + progress.md | 1 - 3 files changed, 8 insertions(+), 1 deletion(-) diff --git a/docs/updates/2026-10.md b/docs/updates/2026-10.md index a477c25..632f40d 100644 --- a/docs/updates/2026-10.md +++ b/docs/updates/2026-10.md @@ -720,3 +720,10 @@ Index and query commands: [README.md](README.md). New entries go at the end. - **Finding**: Windows CI with PySide6 6.12.0 passes GUI assertions but crashes during QApplication shutdown. Moving cleanup into the fixture exposes the access violation at `tests/conftest.py`, instead of an unexplained nonzero interpreter exit. - **Fix**: both package metadata variants, the development requirements and integration input require `PySide6>=6.11.2,<6.12`; the integration lock is regenerated. This keeps fresh GUI installs on the version series that passed the full local suite (5,769 passed, 257 skipped). - **Files**: `dev.toml`, `stable.toml`, `requirements.txt`, `.github/requirements/integration.in`, `.github/requirements/integration.txt`. + +## U-20261008-37 · 2026-10-08 · Main PR validation passes the full CI and analysis gates · #done #ci #security + +- **Completed**: `progress.md` #40. PR #110 at `58b6bc8` passes all 52 checks, including SonarCloud, Codacy, five Python versions, both platforms, all six storage-service integrations and every extra. The PR-only publisher is skipped as intended. +- **Local validation**: 5,769 passed, 257 skipped; metadata, workflow and GUI focused checks pass 166 tests. +- **Files**: `progress.md` and the update-log index. +- **Evidence**: https://github.com/Integration-Automation/FileAutomation/pull/110 diff --git a/docs/updates/README.md b/docs/updates/README.md index 73bb44d..6df3570 100644 --- a/docs/updates/README.md +++ b/docs/updates/README.md @@ -58,6 +58,7 @@ In the same commit: delete the item from `progress.md`, add a `#done` entry here | ID | Date | Title | Tags | Batch | |---|---|---|---|---| +| U-20261008-37 | 2026-10-08 | Main PR validation passes the full CI and analysis gates | #done #ci #security | [2026-10](2026-10.md) | | U-20261008-36 | 2026-10-08 | Keep the GUI on the tested Qt 6.11 series | #incident #gui #ci | [2026-10](2026-10.md) | | U-20261008-35 | 2026-10-08 | Share one Qt application and shut it down after GUI tests | #done #tests #gui | [2026-10](2026-10.md) | | U-20261008-34 | 2026-10-08 | Lock lint and integration dependencies for the main pull request | #ci #security | [2026-10](2026-10.md) | diff --git a/progress.md b/progress.md index fcdcb32..8c2d5ec 100644 --- a/progress.md +++ b/progress.md @@ -28,7 +28,6 @@ Items #10 to #26 are what is left of the 1.0 roadmap (`docs/FILEAUTOMATION-1.0-R - **#39** The application layer reaches into two private places: `StorageService` reads `StorageResolver._mounts` / `_factories` (give the resolver a public listing of its mounts), and `app/` uses `pipeline.definition.retry_from_dict`, `graph.upstream_tasks`, `substitution.is_name` / `NAME_RULE` and `model.ON_SUCCESS` / `WHEN_CHOICES`, which are not in `automation_file.pipeline.__all__` (export them, or give the pipeline package the functions the editor needs). - **#37** Storage-layer gaps the MCP work found (U-20261008-27) and did not change: (a) a `LocalStorage(root)` listing reports a link's name and its target's metadata without a containment check, although reading through the link is refused; (b) `StorageBackend` has no ranged read, so reading the head of a large remote file stages all of it; (c) `LocalStorage` on Windows opens device names (`CON`, `NUL`) and alternate data streams, which the MCP tools refuse themselves; (d) a link on an SFTP or FTP server leads outside a root that is only a path prefix. - **#26** The 1.0.0 release (roadmap M9). Everything the roadmap lists is on the branch `feat/universal-storage-layer`; what is left is the owner's: review and merge the pull request to `dev`, read the first CI and integration runs (#19, #32, #18, #38), decide the 1.0 date, then write `1.0.0` in `stable.toml` and `dev.toml` in the release pull request to `main` and raise the `Development Status` classifier. Not written: a separate security page in the manual (the deployment chapter and `CLAUDE.md` § Security carry that guidance today). -- **#40** [VERIFY] PR #110 replaces unlocked lint and integration installs with hash-locked wheel dependencies. Confirm SonarCloud and all platform integration jobs pass before main is merged. - **#34** [BLOCKED] PyPI Trusted Publishing (roadmap §13). `publish.yml` and `publish-dev` still upload with the `PYPI_API_TOKEN` secret. Switching needs the owner to add a trusted publisher for each project on PyPI (`automation_file`: workflow `publish.yml`; `automation_file_dev`: workflow `ci-dev.yml`; an environment name if one is wanted) before the workflows can drop the token for `id-token: write` and `pypa/gh-action-pypi-publish`. Changing the workflows first would break both channels. ### Packaging follow-ups From 4d51cb9e04a0acedb3c8c27047b591157107177b Mon Sep 17 00:00:00 2001 From: JeffreyChen Date: Thu, 8 Oct 2026 18:30:16 +0800 Subject: [PATCH 59/59] Record the integrated PyBreeze dependency compatibility fix --- docs/updates/2026-10.md | 7 +++++++ docs/updates/README.md | 1 + progress.md | 1 - 3 files changed, 8 insertions(+), 1 deletion(-) diff --git a/docs/updates/2026-10.md b/docs/updates/2026-10.md index 632f40d..7c342dc 100644 --- a/docs/updates/2026-10.md +++ b/docs/updates/2026-10.md @@ -727,3 +727,10 @@ Index and query commands: [README.md](README.md). New entries go at the end. - **Local validation**: 5,769 passed, 257 skipped; metadata, workflow and GUI focused checks pass 166 tests. - **Files**: `progress.md` and the update-log index. - **Evidence**: https://github.com/Integration-Automation/FileAutomation/pull/110 + +## U-20261008-38 · 2026-10-08 · Integrate the PyBreeze all-extras dependency before stable release · #done #dependencies + +- **Completed**: `progress.md` #29. The isolated `0a6d08c` dependency change is integrated into PyBreeze main PR #142, pushed and documented. Both metadata variants and requirements declare `automation-file[all]`; metadata parity and framework metadata checks pass 30 tests. +- **Merge order**: PyBreeze #142 must pass and merge before FileAutomation #110 publishes the stable extras split. +- **Files**: `progress.md` and the update-log index. +- **Evidence**: https://github.com/Integration-Automation/PyBreeze/pull/142 diff --git a/docs/updates/README.md b/docs/updates/README.md index 6df3570..254bad6 100644 --- a/docs/updates/README.md +++ b/docs/updates/README.md @@ -58,6 +58,7 @@ In the same commit: delete the item from `progress.md`, add a `#done` entry here | ID | Date | Title | Tags | Batch | |---|---|---|---|---| +| U-20261008-38 | 2026-10-08 | Integrate the PyBreeze all-extras dependency before stable release | #done #dependencies | [2026-10](2026-10.md) | | U-20261008-37 | 2026-10-08 | Main PR validation passes the full CI and analysis gates | #done #ci #security | [2026-10](2026-10.md) | | U-20261008-36 | 2026-10-08 | Keep the GUI on the tested Qt 6.11 series | #incident #gui #ci | [2026-10](2026-10.md) | | U-20261008-35 | 2026-10-08 | Share one Qt application and shut it down after GUI tests | #done #tests #gui | [2026-10](2026-10.md) | diff --git a/progress.md b/progress.md index 8c2d5ec..eb580cf 100644 --- a/progress.md +++ b/progress.md @@ -32,4 +32,3 @@ Items #10 to #26 are what is left of the 1.0 roadmap (`docs/FILEAUTOMATION-1.0-R ### Packaging follow-ups -- **#29** [BLOCKED] PyBreeze has to declare `automation-file[all]` before the stable release that splits the extras reaches users, or it installs without the SDKs it relied on. The change exists on PyBreeze's local branch `deps/automation-file-all-extra` (one commit on its `origin/dev`: `dev.toml`, `pyproject.toml`, `requirements.txt`), not pushed: it waits for someone to open the PR there and for PyBreeze's own update log. PyBreeze's checkout was on `docs/tutorials` with other work, so nothing else was touched.