diff --git a/.agents/skills/babysit/SKILL.md b/.agents/skills/babysit/SKILL.md new file mode 100644 index 00000000000..6e5b883ee31 --- /dev/null +++ b/.agents/skills/babysit/SKILL.md @@ -0,0 +1,137 @@ +--- +name: babysit +description: Drive a PR to a clean review (Greptile 5/5, zero open threads) — ships if needed, triggers Greptile/Cursor Bugbot, fixes real findings, replies to and resolves every thread, and loops until clean +--- + +# Babysit PRs + +Owns a PR end-to-end through review: ship it, wait for the automatic review round, and if it +isn't already clean, drive fix → reply → resolve → re-review cycles until Greptile reports 5/5 +and there are zero open comment threads. Designed to be run under `/loop` (no fixed interval — +let it self-pace on review latency) so it survives across multiple wakeups in the same session. + +## When to use + +- The user says "babysit this PR", "keep working the reviews until it's clean", or similar +- As the natural follow-up to `/ship` when the user wants the review loop automated rather than + manually re-triggering reviews and answering comments themselves + +## Inputs + +Needs a PR number. If none is given and there's no open PR for the current branch, run `/ship` +first (which includes the `origin/staging` sync check — see `.agents/skills/ship/SKILL.md`) to +create one. + +## Definition of "clean" + +Both must hold: +1. The latest Greptile summary comment reports **Confidence Score: 5/5** +2. `reviewThreads` (GraphQL, see below) has **zero threads with `isResolved: false`** + +Do not stop early on "no new comments this round" alone — a thread can be open from an earlier +round. Always check both conditions freshly after every push. + +## Loop + +1. **Check current state** before doing anything: + ```bash + gh pr view --json comments -q '[.comments[] | select(.author.login=="greptile-apps")] | last | .body' + gh api graphql -f query=' + query { repository(owner: "", name: "") { pullRequest(number: ) { + reviewThreads(first: 50) { pageInfo { hasNextPage endCursor } nodes { id isResolved path line + comments(first: 5) { nodes { id databaseId author { login } body } } } } } } }' + ``` + `[.comments[]] | last | .body`, not `... | .body | tail -1` — the latter pipes every matching + comment's full multi-line body through the pipeline and keeps only the final *line* of that + combined output (usually the "Reviews (n): Last reviewed commit..." footer), not the last + *comment*, so it silently misses the actual "Confidence Score: X/5" line. + `reviewThreads(first: 50)` is a single page — check `pageInfo.hasNextPage`. If `true`, don't + stop yet: re-run the same query with `after: ""` and keep paging until + `hasNextPage` is `false` before evaluating "clean." A PR with more than 50 threads is rare but + stopping on a partial page would silently miss unresolved ones past the cutoff. + If Greptile is 5/5 and every thread across all pages has `isResolved: true`, stop — report the + outcome (see "Reporting" below) and skip the rest of this list. + +2. **If no review has run yet** (fresh PR, no Greptile/Cursor comments): they usually run + automatically on PR open — confirm via `gh pr checks ` (look for `Cursor Bugbot` / + `Greptile Review`) and wait for that first round before doing anything else. + +3. **If a review round has landed and it isn't clean**: for every thread where + `isResolved: false`, triage the finding on its own merits — this is the part that requires + judgment, not a mechanical loop: + - **Real bug**: fix it in the cleanest way available. Match the codebase's existing + conventions for that kind of problem before inventing a new one (e.g. an SSRF-prone + user-supplied-host fetch should use whatever `validateUrlWithDNS`/`secureFetchWithPinnedIP` + pattern the rest of the codebase already uses for that exact situation — grep for a sibling + integration solving the same problem first). Never patch around a finding with a + workaround, a broad try/catch, or a suppression comment — fix the actual cause. + - **False positive**: don't change code. Reply with the specific reason it doesn't apply + (cite the type definition, the established pattern it matches, or the doc it follows) so + the reviewer bot and a human skimming later both understand why it was left as-is. + - **Already fixed by an earlier finding in the same round**: note that and resolve without a + duplicate code change. + +4. **Reply to every thread individually** before resolving it — never resolve silently: + ```bash + gh api repos///pulls//comments//replies -f body="" + ``` + Then resolve via GraphQL (needs the thread `id` from step 1, not the comment id): + ```bash + gh api graphql -f query='mutation { resolveReviewThread(input: {threadId: ""}) { thread { isResolved } } }' + ``` + +5. **Before pushing, re-run the full sync check from `/ship` step 2** — not just the log command, + the whole check-and-recover flow (stash WIP if needed, rebase, verify the rebase didn't just + cleanly replay stray commits, cherry-pick rebuild if it did or if it conflicted). A babysit + loop spanning a long session is exactly the scenario where a branch can drift, and pushing + review fixes on top of undetected drift is how an oversized PR happens even after the branch + was fixed once. Then run the repo's pre-ship checks the same way `/ship` does before + committing — not just lint/typecheck/boundary-validation, but also the conditional `/cleanup` + (if this round's fix touched UI code) and `/db-migrate` (if it touched schema/migrations) + gates from `/ship` steps 4 and 5. A review-fix round is still a code change and can trip + either gate just as easily as the original commit did. + +6. **Commit and push** the round's fixes as one commit — `--force-with-lease` whenever step 5's + sync check rewrote history, which includes a plain `git rebase origin/staging` that completed + with no conflicts, not only the cherry-pick rebuild path; both rewrite commits already + published to the remote, so a plain `git push` can be rejected either way — then run `/ship` + step 9's post-push verify — not just before the first push, every push in the loop: + ```bash + git fetch origin staging && git log --oneline --reverse origin/staging..HEAD + gh pr view --json commits -q '.commits[].messageHeadline' + ``` + `--reverse` matches `git log`'s newest-first default to the PR commit list's oldest-first + order — without it a positional comparison can spuriously fail on any multi-commit branch. + These two lists must describe the same commits. A review loop runs many pushes across many + rounds; checking sync only before the push (step 5) and never after is how a bad push or a + PR whose commit history quietly went stale between rounds goes unnoticed. + +7. **Re-trigger review** by posting `@greptile` and `@cursor review` as **two separate PR + comments** — never combine them into one comment, each bot only responds to its own mention: + ```bash + gh pr comment --body "@greptile" + gh pr comment --body "@cursor review" + ``` + +8. **Wait for the new round**, then go back to step 1. Pace the wait with `ScheduleWakeup` using + a fallback delay of ~250–300s (Greptile/Cursor typically take 1–3 minutes) — never busy-poll + in a sleep loop. Pass the same `/loop babysit PR ` prompt on each wakeup so the loop + resumes correctly. + +9. **Stop conditions**: clean state reached (see above), or the same unresolved finding survives + two consecutive rounds with no new information (surface it to the user instead of looping + forever), or the user interrupts. + +## Reporting + +When the loop ends, summarize: how many rounds it took, what was actually fixed (one line each), +what was pushed back on as a false positive and why, and the final Greptile score / thread count. + +## Hard rules + +- Never post the two re-review mentions as a single combined comment. +- Never resolve a thread without replying to it first. +- Never fix a finding with a hacky workaround — if the clean fix isn't obvious, find the sibling + pattern elsewhere in the codebase solving the same class of problem and match it. +- Never silently drop a finding — every thread gets either a code fix or a reasoned reply. +- Always re-run the `/ship`-style sync check before every push in the loop, not just the first. diff --git a/.agents/skills/ship/SKILL.md b/.agents/skills/ship/SKILL.md index db61568d6c0..43024747fe4 100644 --- a/.agents/skills/ship/SKILL.md +++ b/.agents/skills/ship/SKILL.md @@ -12,21 +12,48 @@ You help ship code by creating commits, pushing to the remote branch, and creati When the user runs `/ship`: 1. **Check git status** - See what files have changed -2. **Generate a commit message** following this format: `type(scope): description` +2. **Sync check**: `git fetch origin staging && git log --oneline origin/staging..HEAD`. Read the actual commit list, not just how many there are — it must show ONLY commits you can attribute to this session (recognizable subjects/SHAs). A worktree/branch can silently be cut from a stale local `staging`, dragging in unrelated commits; a corrupted branch's inflated commit *count* can coincidentally match a later check even when the *commits* are wrong, so always compare content, never just a number. + - If it shows commits you don't recognize, fix it now, **before** staging/committing any new work (step 7 hasn't run yet): + - If the working tree has uncommitted changes, stash them first — `git stash push -u -m ship-sync-fix` — so the rebase below isn't blocked by dirty state. Restore with `git stash pop` once the branch is fixed. + - Try `git rebase origin/staging` first. + - **A rebase finishing without conflicts does NOT by itself mean the branch is clean** — it can replay stray commits onto the new base with no conflict at all. After the rebase (clean or not), re-run `git log --oneline origin/staging..HEAD` and re-check the commit list against what you recognize. + - If the rebase conflicted on commits you don't recognize, OR it finished cleanly but the re-checked log still shows commits you don't recognize, abandon that result (`git rebase --abort` if still mid-rebase) and rebuild instead, in this exact order: + 1. **While still on ``**, identify the SHA(s) to preserve — **not** the whole range. `git log --oneline --reverse origin/staging..` lists everything ahead of `origin/staging`, but in exactly this scenario that range also contains the unrecognized/stray commits you're trying to leave behind — blindly cherry-picking the full range recreates the same polluted branch. Read the list and write down only the SHA(s) you recognize as your own session's work (e.g. `abc1234 def5678`); do this *before* touching any temp branch, since once you check out `ship-sync-tmp` at `origin/staging` in step 4, `HEAD` no longer contains these commits and the same lookup at that point returns nothing. + 2. `git checkout ` — harmless no-op if you're already there, but required if an earlier interrupted attempt left you sitting on `ship-sync-tmp`: git refuses to delete the branch you're currently on, so deleting it before switching away silently fails and blocks the rest of the rebuild. + 3. Delete any leftover from an earlier attempt: `git branch -D ship-sync-tmp 2>/dev/null || true` — always succeeds, including when there's nothing to delete (a first attempt), so it never blocks the rest of the rebuild on its own exit code. + 4. `git checkout -b ship-sync-tmp origin/staging`. + 5. `git cherry-pick` the SHAs captured in step 1, **in that oldest-first order** — cherry-picking more than one session commit out of order can fail or produce the wrong history. Resolve conflicts. + 6. `git branch -f HEAD`, `git checkout `, and delete `ship-sync-tmp` (`git branch -D ship-sync-tmp`). + - Re-verify with `git log --oneline origin/staging..HEAD` — it must list only commits you recognize before you proceed to committing new work. +3. **Generate a commit message** following this format: `type(scope): description` - Types: `fix`, `feat`, `improvement`, `chore` - Scope: short identifier (e.g., `undo-redo`, `api`, `ui`) - Keep it concise -3. **Run the cleanup pass** — only if the diff modifies UI code (any `.tsx` file, or anything under `apps/sim/components/`, `apps/sim/hooks/`, or `apps/sim/stores/`): `/cleanup` +4. **Run the cleanup pass** — only if the diff modifies UI code (any `.tsx` file, or anything under `apps/sim/components/`, `apps/sim/hooks/`, or `apps/sim/stores/`): `/cleanup` - The six code-quality skills (effects, memo, callbacks, state, React Query, emcn) only apply to React code, so skip this step entirely when no UI was touched. When it runs, it applies fixes so they land in this commit. -4. **Run migration safety** — only if the diff touches `packages/db/migrations/**` or `packages/db/schema.ts`: +5. **Run migration safety** — only if the diff touches `packages/db/migrations/**` or `packages/db/schema.ts`: - Run `/db-migrate` to review the migration for zero-downtime safety (expand/contract phasing, backward-compatibility with the deployed app version). - `bun run check:migrations origin/staging` must pass (staging is the PR base). Do not silence a flagged statement with a `-- migration-safe:` annotation unless `/db-migrate` confirmed the old code no longer depends on it; otherwise split the destructive change into a later deploy. -5. **Run pre-ship checks** from the repo root before staging: +6. **Run pre-ship checks** from the repo root before staging: - `bun run lint` to fix formatting issues - `bun run check:api-validation:strict` to catch boundary contract failures before CI -6. **Stage and commit** the changes with the generated message -7. **Push to origin** using the current branch name -8. **Create a PR** to staging with a description in the user's voice +7. **Stage and commit** the changes with the generated message +8. **Push to origin** using the current branch name — `--force-with-lease` if step 2's sync + check did any history rewrite (a clean rebase or a cherry-pick rebuild) on a branch that had + already been pushed once; a plain push would be rejected in exactly the polluted-remote case + step 2 exists to fix +9. **Create a PR** to staging with a description in the user's voice, then do a final content check — not a count check — comparing what actually landed: + ```bash + git fetch origin staging && git log --oneline --reverse origin/staging..HEAD + gh pr view --json commits -q '.commits[].messageHeadline' + ``` + Re-fetch first — comparing against a stale local `origin/staging` ref can mask real drift or + flag a false mismatch even when the branch and push are correct. `--reverse` makes the git log + oldest-first, matching the PR commit list's order — plain `git log` is newest-first, and a + positional/line-by-line comparison against the PR's oldest-first list can spuriously fail on + any multi-commit branch. These two lists must describe the same commits in the same order + (same subjects, the last one being the commit from step 7). If they don't match, the branch + still has a problem — redo step 2's fix and `git push --force-with-lease`. ## Commit Message Format diff --git a/.cursor/commands/babysit.md b/.cursor/commands/babysit.md new file mode 100644 index 00000000000..3935f6b5e30 --- /dev/null +++ b/.cursor/commands/babysit.md @@ -0,0 +1,131 @@ +# Babysit Command + +Owns a PR end-to-end through review: ship it, wait for the automatic review round, and if it +isn't already clean, drive fix → reply → resolve → re-review cycles until Greptile reports 5/5 +and there are zero open comment threads. + +## When to use + +- The user says "babysit this PR", "keep working the reviews until it's clean", or similar +- As the natural follow-up to `/ship` when the user wants the review loop automated rather than + manually re-triggering reviews and answering comments themselves + +## Inputs + +Needs a PR number. If none is given and there's no open PR for the current branch, run `/ship` +first (which includes the `origin/staging` sync check — see `.cursor/commands/ship.md`) to +create one. + +## Definition of "clean" + +Both must hold: +1. The latest Greptile summary comment reports **Confidence Score: 5/5** +2. `reviewThreads` (GraphQL, see below) has **zero threads with `isResolved: false`** + +Do not stop early on "no new comments this round" alone — a thread can be open from an earlier +round. Always check both conditions freshly after every push. + +## Loop + +1. **Check current state** before doing anything: + ```bash + gh pr view --json comments -q '[.comments[] | select(.author.login=="greptile-apps")] | last | .body' + gh api graphql -f query=' + query { repository(owner: "", name: "") { pullRequest(number: ) { + reviewThreads(first: 50) { pageInfo { hasNextPage endCursor } nodes { id isResolved path line + comments(first: 5) { nodes { id databaseId author { login } body } } } } } } }' + ``` + `[.comments[]] | last | .body`, not `... | .body | tail -1` — the latter pipes every matching + comment's full multi-line body through the pipeline and keeps only the final *line* of that + combined output (usually the "Reviews (n): Last reviewed commit..." footer), not the last + *comment*, so it silently misses the actual "Confidence Score: X/5" line. + `reviewThreads(first: 50)` is a single page — check `pageInfo.hasNextPage`. If `true`, don't + stop yet: re-run the same query with `after: ""` and keep paging until + `hasNextPage` is `false` before evaluating "clean." A PR with more than 50 threads is rare but + stopping on a partial page would silently miss unresolved ones past the cutoff. + If Greptile is 5/5 and every thread across all pages has `isResolved: true`, stop — report the + outcome (see "Reporting" below) and skip the rest of this list. + +2. **If no review has run yet** (fresh PR, no Greptile/Cursor comments): they usually run + automatically on PR open — confirm via `gh pr checks ` (look for `Cursor Bugbot` / + `Greptile Review`) and wait for that first round before doing anything else. + +3. **If a review round has landed and it isn't clean**: for every thread where + `isResolved: false`, triage the finding on its own merits — this is the part that requires + judgment, not a mechanical loop: + - **Real bug**: fix it in the cleanest way available. Match the codebase's existing + conventions for that kind of problem before inventing a new one (e.g. an SSRF-prone + user-supplied-host fetch should use whatever `validateUrlWithDNS`/`secureFetchWithPinnedIP` + pattern the rest of the codebase already uses for that exact situation — grep for a sibling + integration solving the same problem first). Never patch around a finding with a + workaround, a broad try/catch, or a suppression comment — fix the actual cause. + - **False positive**: don't change code. Reply with the specific reason it doesn't apply + (cite the type definition, the established pattern it matches, or the doc it follows) so + the reviewer bot and a human skimming later both understand why it was left as-is. + - **Already fixed by an earlier finding in the same round**: note that and resolve without a + duplicate code change. + +4. **Reply to every thread individually** before resolving it — never resolve silently: + ```bash + gh api repos///pulls//comments//replies -f body="" + ``` + Then resolve via GraphQL (needs the thread `id` from step 1, not the comment id): + ```bash + gh api graphql -f query='mutation { resolveReviewThread(input: {threadId: ""}) { thread { isResolved } } }' + ``` + +5. **Before pushing, re-run the full sync check from `/ship` step 2** — not just the log command, + the whole check-and-recover flow (stash WIP if needed, rebase, verify the rebase didn't just + cleanly replay stray commits, cherry-pick rebuild if it did or if it conflicted). A babysit + loop spanning a long session is exactly the scenario where a branch can drift, and pushing + review fixes on top of undetected drift is how an oversized PR happens even after the branch + was fixed once. Then run the repo's pre-ship checks — not just lint/typecheck/boundary- + validation, but also the conditional `/cleanup` (UI changes) and `/db-migrate` + (schema/migration changes) gates from `.agents/skills/ship/SKILL.md` steps 4 and 5 (Cursor's + own 7-step `ship.md` doesn't carry those steps, but a review-fix round is still a code change + and can trip either gate just as easily as the original commit did). + +6. **Commit and push** the round's fixes as one commit — `--force-with-lease` whenever step 5's + sync check rewrote history, which includes a plain `git rebase origin/staging` that completed + with no conflicts, not only the cherry-pick rebuild path; both rewrite commits already + published to the remote, so a plain `git push` can be rejected either way — then run `/ship` + step 7's post-push verify — not just before + the first push, every push in the loop (the Cursor `/ship` has 7 steps; the Claude Code skill + version's equivalent is step 9 — see `.agents/skills/babysit/SKILL.md` if working from that copy): + ```bash + git fetch origin staging && git log --oneline --reverse origin/staging..HEAD + gh pr view --json commits -q '.commits[].messageHeadline' + ``` + `--reverse` matches `git log`'s newest-first default to the PR commit list's oldest-first + order — without it a positional comparison can spuriously fail on any multi-commit branch. + These two lists must describe the same commits. A review loop runs many pushes across many + rounds; checking sync only before the push (step 5) and never after is how a bad push or a + PR whose commit history quietly went stale between rounds goes unnoticed. + +7. **Re-trigger review** by posting `@greptile` and `@cursor review` as **two separate PR + comments** — never combine them into one comment, each bot only responds to its own mention: + ```bash + gh pr comment --body "@greptile" + gh pr comment --body "@cursor review" + ``` + +8. **Wait for the new round**, then go back to step 1. Never busy-poll in a sleep loop — pace the + wait to roughly how long Greptile/Cursor take (1–3 minutes). + +9. **Stop conditions**: clean state reached (see above), or the same unresolved finding survives + two consecutive rounds with no new information (surface it to the user instead of looping + forever), or the user interrupts. + +## Reporting + +When the loop ends, summarize: how many rounds it took, what was actually fixed (one line each), +what was pushed back on as a false positive and why, and the final Greptile score / thread count. + +## Hard rules + +- Never post the two re-review mentions as a single combined comment. +- Never resolve a thread without replying to it first. +- Never fix a finding with a hacky workaround — if the clean fix isn't obvious, find the sibling + pattern elsewhere in the codebase solving the same class of problem and match it. +- Never silently drop a finding — every thread gets either a code fix or a reasoned reply. +- Always re-run the `/ship`-style sync check before every push in the loop, not just the first. diff --git a/.cursor/commands/ship.md b/.cursor/commands/ship.md index 2530e463ee3..b08075ceb3c 100644 --- a/.cursor/commands/ship.md +++ b/.cursor/commands/ship.md @@ -7,20 +7,47 @@ You help ship code by creating commits, pushing to the remote branch, and creati When the user runs `/ship`: 1. **Check git status** - See what files have changed -2. **Generate a commit message** following this format: `type(scope): description` +2. **Sync check**: `git fetch origin staging && git log --oneline origin/staging..HEAD`. Read the actual commit list, not just how many there are — it must show ONLY commits you can attribute to this session (recognizable subjects/SHAs). A worktree/branch can silently be cut from a stale local `staging`, dragging in unrelated commits; a corrupted branch's inflated commit *count* can coincidentally match a later check even when the *commits* are wrong, so always compare content, never just a number. + - If it shows commits you don't recognize, fix it now, **before** staging/committing any new work (step 5 hasn't run yet): + - If the working tree has uncommitted changes, stash them first — `git stash push -u -m ship-sync-fix` — so the rebase below isn't blocked by dirty state. Restore with `git stash pop` once the branch is fixed. + - Try `git rebase origin/staging` first. + - **A rebase finishing without conflicts does NOT by itself mean the branch is clean** — it can replay stray commits onto the new base with no conflict at all. After the rebase (clean or not), re-run `git log --oneline origin/staging..HEAD` and re-check the commit list against what you recognize. + - If the rebase conflicted on commits you don't recognize, OR it finished cleanly but the re-checked log still shows commits you don't recognize, abandon that result (`git rebase --abort` if still mid-rebase) and rebuild instead, in this exact order: + 1. **While still on ``**, identify the SHA(s) to preserve — **not** the whole range. `git log --oneline --reverse origin/staging..` lists everything ahead of `origin/staging`, but in exactly this scenario that range also contains the unrecognized/stray commits you're trying to leave behind — blindly cherry-picking the full range recreates the same polluted branch. Read the list and write down only the SHA(s) you recognize as your own session's work (e.g. `abc1234 def5678`); do this *before* touching any temp branch, since once you check out `ship-sync-tmp` at `origin/staging` in step 4, `HEAD` no longer contains these commits and the same lookup at that point returns nothing. + 2. `git checkout ` — harmless no-op if you're already there, but required if an earlier interrupted attempt left you sitting on `ship-sync-tmp`: git refuses to delete the branch you're currently on, so deleting it before switching away silently fails and blocks the rest of the rebuild. + 3. Delete any leftover from an earlier attempt: `git branch -D ship-sync-tmp 2>/dev/null || true` — always succeeds, including when there's nothing to delete (a first attempt), so it never blocks the rest of the rebuild on its own exit code. + 4. `git checkout -b ship-sync-tmp origin/staging`. + 5. `git cherry-pick` the SHAs captured in step 1, **in that oldest-first order** — cherry-picking more than one session commit out of order can fail or produce the wrong history. Resolve conflicts. + 6. `git branch -f HEAD`, `git checkout `, and delete `ship-sync-tmp` (`git branch -D ship-sync-tmp`). + - Re-verify with `git log --oneline origin/staging..HEAD` — it must list only commits you recognize before you proceed to committing new work. +3. **Generate a commit message** following this format: `type(scope): description` - Types: `fix`, `feat`, `improvement`, `chore` - Scope: short identifier (e.g., `undo-redo`, `api`, `ui`) - Keep it concise -3. **Run pre-ship checks** from the repo root before staging: +4. **Run pre-ship checks** from the repo root before staging: - `bun run lint` to fix formatting issues - `bun run check:api-validation:strict` to catch boundary contract failures before CI -4. **Stage and commit** the changes with the generated message - -5. **Push to origin** using the current branch name - -6. **Create a PR** to staging with a description in the user's voice +5. **Stage and commit** the changes with the generated message + +6. **Push to origin** using the current branch name — `--force-with-lease` if step 2's sync + check did any history rewrite (a clean rebase or a cherry-pick rebuild) on a branch that had + already been pushed once; a plain push would be rejected in exactly the polluted-remote case + step 2 exists to fix + +7. **Create a PR** to staging with a description in the user's voice, then do a final content check — not a count check — comparing what actually landed: + ```bash + git fetch origin staging && git log --oneline --reverse origin/staging..HEAD + gh pr view --json commits -q '.commits[].messageHeadline' + ``` + Re-fetch first — comparing against a stale local `origin/staging` ref can mask real drift or + flag a false mismatch even when the branch and push are correct. `--reverse` makes the git log + oldest-first, matching the PR commit list's order — plain `git log` is newest-first, and a + positional/line-by-line comparison against the PR's oldest-first list can spuriously fail on + any multi-commit branch. These two lists must describe the same commits in the same order + (same subjects, the last one being the commit from step 5). If they don't match, the branch + still has a problem — redo step 2's fix and `git push --force-with-lease`. ## Commit Message Format diff --git a/.github/workflows/test-build.yml b/.github/workflows/test-build.yml index 53a18cf16fd..5d906dc709a 100644 --- a/.github/workflows/test-build.yml +++ b/.github/workflows/test-build.yml @@ -116,6 +116,9 @@ jobs: - name: API contract boundary audit run: bun run check:api-validation:strict + - name: Shared utils enforcement audit + run: bun run check:utils + - name: Zustand v5 selector audit run: bun run check:zustand-v5 diff --git a/.gitignore b/.gitignore index a700a66602a..a8a0d8e2fde 100644 --- a/.gitignore +++ b/.gitignore @@ -96,3 +96,7 @@ i18n.cache # Personal Cursor Skills .cursor/skills/ask-sim/ + +# Python (apps/pii tests/tooling) +__pycache__/ +.pytest_cache/ diff --git a/apps/docs/components/icons.tsx b/apps/docs/components/icons.tsx index ea1eeea8a10..c213134b0e1 100644 --- a/apps/docs/components/icons.tsx +++ b/apps/docs/components/icons.tsx @@ -8412,3 +8412,72 @@ export function FlowiseIcon(props: SVGProps) { ) } + +export function JupyterIcon(props: SVGProps) { + return ( + + + + + + + + + + + + + + + + + + + + + + + + + + + + + ) +} diff --git a/apps/docs/components/ui/icon-mapping.ts b/apps/docs/components/ui/icon-mapping.ts index f5f3a71a108..38effba830e 100644 --- a/apps/docs/components/ui/icon-mapping.ts +++ b/apps/docs/components/ui/icon-mapping.ts @@ -115,6 +115,7 @@ import { JinaAIIcon, JiraIcon, JiraServiceManagementIcon, + JupyterIcon, KalshiIcon, KetchIcon, LangsmithIcon, @@ -364,6 +365,7 @@ export const blockTypeToIconMap: Record = { jina: JinaAIIcon, jira: JiraIcon, jira_service_management: JiraServiceManagementIcon, + jupyter: JupyterIcon, kalshi: KalshiIcon, kalshi_v2: KalshiIcon, ketch: KetchIcon, diff --git a/apps/docs/content/docs/en/integrations/jupyter.mdx b/apps/docs/content/docs/en/integrations/jupyter.mdx new file mode 100644 index 00000000000..81c333a73f2 --- /dev/null +++ b/apps/docs/content/docs/en/integrations/jupyter.mdx @@ -0,0 +1,383 @@ +--- +title: Jupyter +description: Manage files, notebooks, kernels, and sessions on a Jupyter server +--- + +import { BlockInfoCard } from "@/components/ui/block-info-card" + + + +## Usage Instructions + +Integrate a self-hosted Jupyter server into the workflow. Browse, read, create, upload, rename, copy, and delete files and notebooks; start, stop, restart, and interrupt kernels; and manage sessions that bind notebooks to kernels. + + + +## Actions + +### `jupyter_list_contents` + +List files, notebooks, and subdirectories at a path on a Jupyter server + +#### Input + +| Parameter | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `serverUrl` | string | Yes | Base URL of the Jupyter server \(e.g. http://localhost:8888\) | +| `token` | string | Yes | Jupyter server authentication token | +| `path` | string | No | Directory path to list, relative to the server root. Leave blank for root. | + +#### Output + +| Parameter | Type | Description | +| --------- | ---- | ----------- | +| `items` | array | Directory entries at the requested path | +| ↳ `name` | string | Entry name | +| ↳ `path` | string | Entry path relative to server root | +| ↳ `type` | string | directory, file, or notebook | +| ↳ `writable` | boolean | Whether the entry is writable | +| ↳ `created` | string | Creation timestamp | +| ↳ `lastModified` | string | Last modified timestamp | +| ↳ `size` | number | Size in bytes | +| ↳ `mimetype` | string | MIME type \(files only\) | +| ↳ `format` | string | json, text, or base64 | +| `path` | string | The listed directory path | + +### `jupyter_get_content` + +Read a file or notebook from a Jupyter server + +#### Input + +| Parameter | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `serverUrl` | string | Yes | Base URL of the Jupyter server \(e.g. http://localhost:8888\) | +| `token` | string | Yes | Jupyter server authentication token | +| `path` | string | Yes | Path of the file or notebook to read, relative to the server root | + +#### Output + +| Parameter | Type | Description | +| --------- | ---- | ----------- | +| `name` | string | File or notebook name | +| `path` | string | Path relative to the server root | +| `mimetype` | string | MIME type of the content | +| `text` | string | Text content, for text files and notebooks \(JSON-stringified\) | +| `file` | file | Binary content stored as a file, for base64-format content | + +### `jupyter_create_file` + +Create a file, notebook, or directory on a Jupyter server + +#### Input + +| Parameter | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `serverUrl` | string | Yes | Base URL of the Jupyter server \(e.g. http://localhost:8888\) | +| `token` | string | Yes | Jupyter server authentication token | +| `path` | string | Yes | Path to create, relative to the server root | +| `type` | string | Yes | Type of entry to create: file, notebook, or directory | +| `content` | string | No | Content to write. For a file, plain text. For a notebook, a JSON-stringified nbformat document \(defaults to an empty notebook\). Ignored for directories. | + +#### Output + +| Parameter | Type | Description | +| --------- | ---- | ----------- | +| `name` | string | Created entry name | +| `path` | string | Created entry path | +| `type` | string | directory, file, or notebook | +| `createdAt` | string | Creation timestamp | +| `lastModified` | string | Last modified timestamp | + +### `jupyter_upload_file` + +Upload a file to a Jupyter server + +#### Input + +| Parameter | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `serverUrl` | string | Yes | Base URL of the Jupyter server \(e.g. http://localhost:8888\) | +| `token` | string | Yes | Jupyter server authentication token | +| `directory` | string | No | Destination directory, relative to the server root. Leave blank to upload to the root directory. | +| `file` | file | No | The file to upload \(UserFile object\) | +| `fileContent` | string | No | Legacy: base64 encoded file content | +| `fileName` | string | No | Optional filename override | + +#### Output + +| Parameter | Type | Description | +| --------- | ---- | ----------- | +| `name` | string | Uploaded file name | +| `path` | string | Uploaded file path | +| `size` | number | File size in bytes | +| `lastModified` | string | Last modified timestamp | + +### `jupyter_rename_content` + +Rename or move a file, notebook, or directory on a Jupyter server + +#### Input + +| Parameter | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `serverUrl` | string | Yes | Base URL of the Jupyter server \(e.g. http://localhost:8888\) | +| `token` | string | Yes | Jupyter server authentication token | +| `path` | string | Yes | Current path of the entry, relative to the server root | +| `newPath` | string | Yes | New path for the entry, relative to the server root | + +#### Output + +| Parameter | Type | Description | +| --------- | ---- | ----------- | +| `name` | string | New entry name | +| `path` | string | New entry path | +| `lastModified` | string | Last modified timestamp | + +### `jupyter_delete_content` + +Delete a file, notebook, or directory on a Jupyter server + +#### Input + +| Parameter | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `serverUrl` | string | Yes | Base URL of the Jupyter server \(e.g. http://localhost:8888\) | +| `token` | string | Yes | Jupyter server authentication token | +| `path` | string | Yes | Path of the entry to delete, relative to the server root | + +#### Output + +| Parameter | Type | Description | +| --------- | ---- | ----------- | +| `success` | boolean | Whether the entry was deleted | +| `path` | string | Deleted entry path | + +### `jupyter_copy_content` + +Duplicate a file or notebook into a directory on a Jupyter server + +#### Input + +| Parameter | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `serverUrl` | string | Yes | Base URL of the Jupyter server \(e.g. http://localhost:8888\) | +| `token` | string | Yes | Jupyter server authentication token | +| `path` | string | Yes | Destination directory path, relative to the server root | +| `copyFromPath` | string | Yes | Path of the file or notebook to copy, relative to the server root | + +#### Output + +| Parameter | Type | Description | +| --------- | ---- | ----------- | +| `name` | string | Name of the copied entry | +| `path` | string | Path of the copied entry | +| `createdAt` | string | Creation timestamp | + +### `jupyter_list_kernels` + +List running kernels on a Jupyter server + +#### Input + +| Parameter | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `serverUrl` | string | Yes | Base URL of the Jupyter server \(e.g. http://localhost:8888\) | +| `token` | string | Yes | Jupyter server authentication token | + +#### Output + +| Parameter | Type | Description | +| --------- | ---- | ----------- | +| `kernels` | array | Running kernels | +| ↳ `id` | string | Kernel ID | +| ↳ `name` | string | Kernel spec name | +| ↳ `lastActivity` | string | Last activity timestamp | +| ↳ `executionState` | string | Kernel execution state | +| ↳ `connections` | number | Active connection count | + +### `jupyter_start_kernel` + +Start a new kernel on a Jupyter server + +#### Input + +| Parameter | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `serverUrl` | string | Yes | Base URL of the Jupyter server \(e.g. http://localhost:8888\) | +| `token` | string | Yes | Jupyter server authentication token | +| `kernelName` | string | No | Kernel spec name to start \(e.g. python3\). Defaults to the server default. | + +#### Output + +| Parameter | Type | Description | +| --------- | ---- | ----------- | +| `id` | string | Kernel ID | +| `name` | string | Kernel spec name | +| `lastActivity` | string | Last activity timestamp | +| `executionState` | string | Kernel execution state | +| `connections` | number | Active connection count | + +### `jupyter_stop_kernel` + +Shut down a running kernel on a Jupyter server + +#### Input + +| Parameter | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `serverUrl` | string | Yes | Base URL of the Jupyter server \(e.g. http://localhost:8888\) | +| `token` | string | Yes | Jupyter server authentication token | +| `kernelId` | string | Yes | ID of the kernel to shut down | + +#### Output + +| Parameter | Type | Description | +| --------- | ---- | ----------- | +| `success` | boolean | Whether the kernel was shut down | +| `kernelId` | string | Shut down kernel ID | + +### `jupyter_restart_kernel` + +Restart a running kernel on a Jupyter server + +#### Input + +| Parameter | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `serverUrl` | string | Yes | Base URL of the Jupyter server \(e.g. http://localhost:8888\) | +| `token` | string | Yes | Jupyter server authentication token | +| `kernelId` | string | Yes | ID of the kernel to restart | + +#### Output + +| Parameter | Type | Description | +| --------- | ---- | ----------- | +| `id` | string | Kernel ID | +| `name` | string | Kernel spec name | +| `lastActivity` | string | Last activity timestamp | +| `executionState` | string | Kernel execution state | +| `connections` | number | Active connection count | + +### `jupyter_interrupt_kernel` + +Interrupt a running kernel on a Jupyter server + +#### Input + +| Parameter | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `serverUrl` | string | Yes | Base URL of the Jupyter server \(e.g. http://localhost:8888\) | +| `token` | string | Yes | Jupyter server authentication token | +| `kernelId` | string | Yes | ID of the kernel to interrupt | + +#### Output + +| Parameter | Type | Description | +| --------- | ---- | ----------- | +| `success` | boolean | Whether the interrupt was sent | +| `kernelId` | string | Interrupted kernel ID | + +### `jupyter_list_kernelspecs` + +List available kernel specs (languages/runtimes) on a Jupyter server + +#### Input + +| Parameter | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `serverUrl` | string | Yes | Base URL of the Jupyter server \(e.g. http://localhost:8888\) | +| `token` | string | Yes | Jupyter server authentication token | + +#### Output + +| Parameter | Type | Description | +| --------- | ---- | ----------- | +| `defaultKernelName` | string | Default kernel spec name | +| `kernelspecs` | array | Available kernel specs | +| ↳ `name` | string | Kernel spec name | +| ↳ `displayName` | string | Human-readable display name | +| ↳ `language` | string | Kernel language | +| ↳ `argv` | array | Launch command arguments | +| ↳ `interruptMode` | string | Interrupt mode | + +### `jupyter_list_sessions` + +List active sessions (notebook-to-kernel bindings) on a Jupyter server + +#### Input + +| Parameter | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `serverUrl` | string | Yes | Base URL of the Jupyter server \(e.g. http://localhost:8888\) | +| `token` | string | Yes | Jupyter server authentication token | + +#### Output + +| Parameter | Type | Description | +| --------- | ---- | ----------- | +| `sessions` | array | Active sessions | +| ↳ `id` | string | Session ID | +| ↳ `path` | string | Notebook path bound to this session | +| ↳ `name` | string | Session name | +| ↳ `type` | string | Session type | +| ↳ `kernel` | object | Kernel bound to this session | +| ↳ `id` | string | Kernel ID | +| ↳ `name` | string | Kernel spec name | +| ↳ `lastActivity` | string | Last activity timestamp | +| ↳ `executionState` | string | Kernel execution state | +| ↳ `connections` | number | Active connection count | + +### `jupyter_create_session` + +Create a session that binds a notebook path to a (new or existing) kernel + +#### Input + +| Parameter | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `serverUrl` | string | Yes | Base URL of the Jupyter server \(e.g. http://localhost:8888\) | +| `token` | string | Yes | Jupyter server authentication token | +| `path` | string | Yes | Notebook path to bind the session to, relative to the server root | +| `kernelName` | string | No | Kernel spec name to start for this session \(e.g. python3\) | +| `name` | string | No | Optional session name | +| `type` | string | No | Session type, defaults to 'notebook' | + +#### Output + +| Parameter | Type | Description | +| --------- | ---- | ----------- | +| `id` | string | Session ID | +| `path` | string | Notebook path bound to this session | +| `name` | string | Session name | +| `type` | string | Session type | +| `kernel` | object | Kernel bound to this session | +| ↳ `id` | string | Kernel ID | +| ↳ `name` | string | Kernel spec name | +| ↳ `lastActivity` | string | Last activity timestamp | +| ↳ `executionState` | string | Kernel execution state | +| ↳ `connections` | number | Active connection count | + +### `jupyter_delete_session` + +Delete a session on a Jupyter server (does not shut down its kernel) + +#### Input + +| Parameter | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `serverUrl` | string | Yes | Base URL of the Jupyter server \(e.g. http://localhost:8888\) | +| `token` | string | Yes | Jupyter server authentication token | +| `sessionId` | string | Yes | ID of the session to delete | + +#### Output + +| Parameter | Type | Description | +| --------- | ---- | ----------- | +| `success` | boolean | Whether the session was deleted | +| `sessionId` | string | Deleted session ID | + + diff --git a/apps/docs/content/docs/en/integrations/meta.json b/apps/docs/content/docs/en/integrations/meta.json index ae36529830d..85ab2e676bc 100644 --- a/apps/docs/content/docs/en/integrations/meta.json +++ b/apps/docs/content/docs/en/integrations/meta.json @@ -116,6 +116,7 @@ "jina", "jira", "jira_service_management", + "jupyter", "kalshi", "ketch", "knowledge", diff --git a/apps/docs/content/docs/en/platform/enterprise/access-control.mdx b/apps/docs/content/docs/en/platform/enterprise/access-control.mdx index f71ad748aff..dcc896d9fc3 100644 --- a/apps/docs/content/docs/en/platform/enterprise/access-control.mdx +++ b/apps/docs/content/docs/en/platform/enterprise/access-control.mdx @@ -77,7 +77,7 @@ Controls which workflow blocks members can place and execute. Controls visibility of platform features and modules. -Platform tab showing feature toggles grouped by category: Sidebar (Knowledge Base, Tables, Templates), Workflow Panel (Copilot), Settings Tabs, Tools, Deploy Tabs, Features, Logs, and Collaboration Each checkbox maps to a specific feature; checking it hides or disables that feature for group members. +Platform tab showing feature toggles grouped by category: Sidebar (Knowledge Base, Tables), Workflow Panel (Copilot), Settings Tabs, Tools, Deploy Tabs, Features, Logs, and Collaboration Each checkbox maps to a specific feature; checking it hides or disables that feature for group members. **Sidebar** @@ -85,7 +85,6 @@ Controls visibility of platform features and modules. |---------|-------------------| | Knowledge Base | Hides the Knowledge Base section from the sidebar | | Tables | Hides the Tables section from the sidebar | -| Templates | Hides the Templates section from the sidebar | **Workflow Panel** diff --git a/apps/docs/content/docs/en/platform/enterprise/custom-blocks.mdx b/apps/docs/content/docs/en/platform/enterprise/custom-blocks.mdx new file mode 100644 index 00000000000..57fe1bc6ce4 --- /dev/null +++ b/apps/docs/content/docs/en/platform/enterprise/custom-blocks.mdx @@ -0,0 +1,136 @@ +--- +title: Custom Blocks +description: Publish a deployed workflow as a reusable block for your organization +--- + +import { FAQ } from '@/components/ui/faq' +import { Image } from '@/components/ui/image' + +Custom blocks let you package a deployed workflow as a reusable block for your whole organization. Once published, the block appears in the workflow editor's block toolbar alongside built-in blocks like Agent and Function. Anyone in your organization can drop it into a workflow, fill in its inputs, and use its outputs — without needing access to the workflow behind it. + +A custom block always runs the **latest deployed version** of its source workflow, so improving the block is as simple as redeploying that workflow. + +--- + +## Common uses + +Custom blocks turn a workflow one team owns into infrastructure the whole organization can safely reuse. Credentials and complexity stay with the block's author; everyone else gets a clean block that's always up to date. A few patterns: + +- **Internal API gateway.** Wrap an authenticated internal or partner endpoint — "Create Ticket", "Charge Account", "Provision User" — behind a block that takes only the business inputs. Teammates call it without the base URL, API key, or auth headers, and when the endpoint changes you update one workflow instead of every consumer's. +- **Blessed knowledge lookup.** Package a vetted retrieval pipeline — chunking, filters, reranking — as "Search Company Docs" with a single query input, so teams reuse the approved retrieval instead of each rebuilding it. +- **Governed LLM step.** Share a prompt, model, and guardrail combination like "Summarize (house style)". You control the model and prompt org-wide; changing them is a redeploy, not a hunt through everyone's workflows. +- **Data enrichment.** Expose "Enrich Company" or "Lookup Customer by Email" that hides the provider keys and returns clean fields. +- **Compliance gate.** Offer "Redact PII" or "Policy Check" as a standardized step teams drop into their flows, with the rules maintained in one place. + +--- + +## Before you start + +- **Deploy the workflow first.** Only deployed workflows can be published as a block. If the workflow isn't deployed, deploy it, then come back. +- **Be a workspace admin.** Publishing and managing custom blocks requires workspace admin access. + +--- + +## Publishing a block + +### 1. Open Custom blocks settings + +Go to **Settings → Enterprise → Custom blocks** and click **Create block**. + +Custom blocks settings page listing a published block with its icon, name, and description, with a Create block button in the header + + +### 2. Choose the source workflow + +| Field | Description | +|-------|-------------| +| **Workspace** | The workspace that holds the workflow you want to publish. Only workspaces in your organization are listed. | +| **Workflow** | The workflow to publish. Only deployed workflows appear. If a workflow isn't deployed, deploy it first, then return here. | + +The block runs the source workflow's latest deployment. You publish one block per workflow, and the source workflow can't be changed later — to point at a different workflow, delete the block and publish a new one. + +### 3. Name and describe the block + +| Field | Description | +|-------|-------------| +| **Icon** | Optional. A square image (PNG, JPEG, SVG, or WebP) shown on the block. Falls back to your organization's logo, then a default glyph. | +| **Name** | The block's display name in the toolbar (e.g. `Invoice Parser`). Max 60 characters. | +| **Description** | A short summary of what the block does. Max 280 characters. | + +### 4. Configure inputs + +Every input on the source workflow's **Start** block is exposed as a field on the custom block. You don't choose which inputs to include — they all carry over, along with each input's name, type, and description. + +For each input you can add an optional **placeholder** (max 200 characters) — the hint text shown in the empty field to tell consumers what to enter. + +Because inputs are read live from the deployed workflow, renaming or adding a Start input and redeploying updates the block's fields automatically. + +### 5. Choose outputs + +Pick which of the workflow's outputs consumers can use, and give each one a name. Outputs are grouped by the block they come from, so you can expose exactly the values that matter and hide everything internal. + +- **At least one output is required.** +- Each exposed output needs a unique **name** (max 60 characters) — that's the name consumers reference in their workflows. + +Create block form filled in: Workspace and Workflow selectors, an uploaded icon, Name and Description fields, an expanded input with a placeholder, and two selected outputs each given a name + +### 6. Save + +Click **Save changes**. The block is published immediately and becomes available to everyone in your organization in the workflow editor's block toolbar. + +--- + +## Using a custom block + +In the workflow editor, open the block toolbar. Published custom blocks appear under a **Custom blocks** section. Drag one onto the canvas like any other block, fill in its inputs (using the placeholders as a guide), and reference its outputs in downstream blocks. + +Workflow editor block toolbar with a Custom Blocks section listing two published blocks below Core Blocks + +Consumers don't need any access to the source workflow. The block runs on its own, using only the inputs provided, and returns only the outputs you exposed. Internal steps, models, and intermediate values of the source workflow are never visible. + +A custom block connected to a Start block on the workflow canvas, with its query input filled in and the run output showing the returned fields + +--- + +## Managing blocks + +Open a block from **Settings → Enterprise → Custom blocks** to edit or delete it. + +- **Editing** changes only the block's presentation and interface — name, description, icon, input placeholders, and exposed outputs. The source workflow can't be re-pointed. +- **Changing what the block does** is done by editing and **redeploying the source workflow**. The block picks up the new deployment automatically; there's nothing to republish. +- **Deleting** a block is permanent. Workflows already using it will have that block removed, so replace it before deleting if it's in active use. + +--- + +## Good to know + +- **Always current.** A custom block runs the source workflow's latest deployment. There are no versions to pin or manage. +- **Access control.** Custom blocks appear in [Access Control](/platform/enterprise/access-control) alongside built-in blocks, so you can allowlist or restrict them per permission group. +- **Disabled blocks.** A disabled block can't be added to new workflows, but existing placements keep working. +- **Removed blocks.** Deleting a block (or deleting its source workflow) removes it from workflows that used it. +- **Not deployed.** If the source workflow is undeployed, the block shows a clear error when run. Redeploy the workflow to fix it. + +--- + + diff --git a/apps/docs/content/docs/en/platform/enterprise/forks.mdx b/apps/docs/content/docs/en/platform/enterprise/forks.mdx new file mode 100644 index 00000000000..08f7e3bfd3a --- /dev/null +++ b/apps/docs/content/docs/en/platform/enterprise/forks.mdx @@ -0,0 +1,361 @@ +--- +title: Workspace Forks +description: Clone a workspace, keep it linked to its parent, and push or pull deployed workflow changes +--- + +import { Callout } from 'fumadocs-ui/components/callout' +import { FAQ } from '@/components/ui/faq' +import { Image } from '@/components/ui/image' + +Workspace Forks let you clone a workspace into a child, keep the two linked, and move **deployed** workflow changes between them later. Think of **Deploy** as a git commit: only what you have deployed is what Forks can copy or sync. **Sync** is always a force push or force pull — the target side is overwritten for the workflows in the sync. There is no merge UI. + +One workspace has at most one parent. A parent can have many children. You can only sync along a **direct** parent↔child edge — not with a grandparent or sibling. + +--- + +## Who can use Forks + +| Requirement | Detail | +|-------------|--------| +| **Plan** | Enterprise on Sim Cloud | +| **Role** | Workspace **admin** — non-admins never see the Forks tab | +| **Self-hosted** | Set `FORKING_ENABLED` / `NEXT_PUBLIC_FORKING_ENABLED` (see [Self-hosted setup](#self-hosted-setup)) | + +On Sim Cloud, your organization may also need the feature turned on for your account before the tab appears. + +--- + +## Setup + +### 1. Open Forks + +Go to **Settings → Enterprise → Workspace Forks** in the workspace you want to fork from (or manage). + +Workspace Forks settings page showing Parent and Forks sections with Docs, See activity, and Create fork actions + +You will see: + +- **Parent** — if this workspace was forked from another +- **Forks** — children of this workspace +- **See activity** — history of forks, syncs, and rollbacks +- **Create fork** — start a new child + +### 2. Create a fork + +Click **Create fork**. Name the child (defaults to `{workspace} (fork)`), then review **Copy resources**. + +Fork workspace modal with name field and Copy resources list fully selected + +Everything under **Copy resources** starts **selected**. That is usually what you want: tables, knowledge bases, files, custom tools, skills, and MCP servers the child will need. + + + If you deselect a resource, references to it in the forked workflows are **cleared** in the child. You will see a warning before you confirm. + + +Fork workspace modal showing a warning that deselected resources will clear references in the fork + +Click **Fork**. The child workspace is created immediately. Deployed workflows land as **drafts** in the child. Large content (table rows, knowledge base files, file blobs) may finish copying in the background — watch **Activity** on the source workspace. + + + Only **deployed** workflows are forked. Drafts and undeployed work stay in the parent. If the parent has nothing deployed, the child starts with a blank starter workflow. + + +### 3. Open the parent edge (from the child) + +Open the **child** workspace → **Settings → Enterprise → Workspace Forks**. On the **Parent** row, open the menu and choose **Edit mappings**. + +Child rows (when you are on the parent) only offer **Open workspace** and **Disconnect** — mapping and sync are owned by the child configuring how it relates to its parent. + +**Open workspace** is disabled with a tooltip if you do not have access to the other workspace. **Disconnect** stays available so you can still sever the link. + +### 4. Understand mappings + +A **mapping** means: “this resource in the source is the same logical thing as that resource in the target.” When you sync, workflow fields that pointed at the source resource are rewritten to the target resource so blocks keep working. + +| Approach | When to use it | +|----------|----------------| +| **Map** | Both sides already have (or should keep) their own resource — e.g. each workspace’s Slack credential, Gmail account, or secret value | +| **Copy** | The target should receive a **new** clone of the source resource — a new table, knowledge base, file, or MCP server config | + +**Credentials** and **Secrets** are **map-only**. They are never copied. Secret *values* never leave their workspace; only names like `{{API_KEY}}` appear in workflow text. + +Mappings are saved on the fork relationship. You can click **Save** without syncing. Sync always uses the saved mappings. + +After you map or copy a parent resource (credential, knowledge base, table, …), **dependent fields** — Gmail labels, Slack channels, knowledge base documents, and similar — often need a fresh pick in the target. Required dependents block **Sync** until they are filled. + +### 5. Map, reconfigure, then Sync + +On the sync page you will see direction (**Push** / **Pull**), deployed workflow changes, **Mappings**, optional **Copy resources**, and any **Blocking sync** items. + +Sync detail page with Push and Pull direction, deployed workflows, and mapping status badges + +- **Push** — overwrite the **other** workspace with this workspace’s deployed workflows +- **Pull** — overwrite **this** workspace with the other’s deployed workflows + +Both are force operations. Confirm carefully. + +Copy resources section showing resources used by workflows and a Not used by any workflow group + +Resources referenced by the workflows in the sync default to selected for copy. Unused ones sit under **Not used by any workflow** (off by default). If you **map** a resource, it leaves the copy list — maps win. + +Dependent field reconfigure card under a mapped resource with a required field marked + +**Sync** stays disabled until: + +- Every blocking reference is mapped, copied, or fixed in the source +- Required credentials and secrets are mapped +- Required dependent fields are filled +- Sync details have finished loading + +### 6. Confirm and run + +Click **Sync**. You will get an overwrite confirmation. + +Overwrite target workspace confirmation dialog warning that syncing overwrites changes on the target + +On success you will see a toast such as **Pushed to "…"** or **Pulled from "…"**. If some workflows fail to redeploy, you get a warning to open and redeploy them manually. + +--- + +## Activity + +**See activity** (or the Activity view from the Forks header) lists forks, pushes, pulls, and rollbacks that involve this workspace — including events recorded on the other side of the edge. + +Activity view showing Fork and Push events with expandable detail rows + +Expand a row for names of workflows and resources that were created, updated, or archived, and any warnings (for example failed background copies or deploy failures). + +--- + +## Rollback vs Disconnect + +| Action | What it does | Keep in mind | +|--------|--------------|--------------| +| **Rollback** | Undoes the **last sync into this workspace** — restores each affected workflow to its prior deployed version and removes workflows that sync created | Copied resources from past syncs **may remain**. Rollback does not delete tables, KBs, or files that were copied in. | +| **Disconnect** | Permanently removes the fork relationship | Both workspaces stay. Saved mappings and sync history for the pair are deleted. Forking again creates a **new** child, not a reconnect. | + +--- + +## Permissions + +| Action | Who | +|--------|-----| +| See Forks / create a fork | Admin on this workspace (+ feature available) | +| Sync / edit mappings | Admin on **both** sides of the edge | +| Rollback | Admin on the workspace the sync landed in | +| Disconnect | Admin on **this** side only (you can disconnect even without access to the other workspace) | +| Open the other workspace | You must be a member of that workspace | + +--- + +## Resource reference + +How each resource behaves at **fork** time vs **sync** time. Use this when you are deciding what to select under **Copy resources**, or why Sync is asking you to map something. + +### At a glance + +| Resource | Fork | Sync | +|----------|------|------| +| Deployed workflows | Always copied as drafts | Updated / created / archived (force overwrite) | +| Undeployed workflows | Not copied | Not synced | +| Files | Optional copy (default on) | Map or copy | +| Tables | Optional copy (default on) | Map or copy | +| Knowledge bases (+ documents) | Optional copy; referenced docs come with the KB | Map or copy; documents follow the KB | +| Custom tools | Optional copy (default on) | Map or copy | +| Skills | Optional copy (default on) | Map or copy | +| External MCP servers | Optional copy (config only; sign-in cleared) | Map or copy (config only; sign-in cleared) | +| Workflow MCP servers | Optional copy (server shells + attachments when workflows copy) | Kept in sync automatically with the workflows | +| Deployed chats | Carried with a **new** chat URL | Created on the target if it has no chat yet | +| Credentials | Never — fields cleared | Map only | +| Secrets | Values never; `{{KEY}}` names kept | Map key names only | +| Public API | Child starts private | Flag follows the workflow | +| Schedules / webhooks / triggers | Not live until you deploy in the child | Follow whatever you deploy on the target | +| History, API keys, memory, scheduled jobs | Never | Never | + +--- + +### Workflows + +Only **deployed** workflows move. Deploy is the commit; sync is the force push/pull of those commits. + +| | Behavior | +|---|----------| +| **Fork** | Each deployed workflow becomes a **draft** in the child. Run history is not copied. Only folders that contain a copied workflow are kept. | +| **Sync** | The change list shows what will be updated, created, or archived. The target is overwritten for those workflows. | + +**Example:** Parent has `Support triage` deployed and `WIP experiment` as a draft. The fork gets only `Support triage` as a draft. A later push updates the child from the parent’s latest deploy of `Support triage`. + +--- + +### Files + +| | Behavior | +|---|----------| +| **Fork** | Listed under **Copy resources** (default on). Copies land at the child’s **file root** (original file folders are not rebuilt). Deselect → file fields in workflows clear. | +| **Sync** | Map to a file that already exists on the target, or copy. Files used by the sync’s workflows default to selected. | + +**Example:** A workflow attaches `brand-guide.pdf`. Fork with Files selected → the child has its own copy and the block still points at it. + +--- + +### Tables + +| | Behavior | +|---|----------| +| **Fork** | Optional copy (default on). Rows may finish copying in the background after the fork is created. Deselect → table fields clear. | +| **Sync** | Map if both sides should keep pointing at “the same” logical table through the mapping, or copy for an independent dataset on the target. | + +**Example:** An enrichment workflow uses a “Leads” table. Copy on fork so the child can experiment without touching production data. On sync, map if you intentionally want both sides aligned to paired tables, or copy a new table when the target should get a fresh clone. + +--- + +### Knowledge bases and documents + +| | Behavior | +|---|----------| +| **Fork** | Optional copy (default on). Tag definitions come with the knowledge base. Documents that the forked workflows actually reference are included. Deselect → knowledge base / document fields clear. | +| **Sync** | Map or copy the knowledge base. Documents are not mapped by themselves — they follow the knowledge base (copied with it, or re-picked when you map to an existing one). | + +**Example:** An agent searches knowledge base “Product docs.” Fork with that knowledge base selected → the child gets the base, tags, and the documents the agent used. On sync, mapping to the child’s existing “Product docs” means re-picking which document the tool should use. + +--- + +### Custom tools + +| | Behavior | +|---|----------| +| **Fork** | Optional copy (default on). The tool definition comes along so agent / tool picks keep working. | +| **Sync** | Map when both sides already maintain the same tool; copy when the target should receive the source’s definition as a new tool. | + +**Example:** A `lookup_customer` custom tool used by an agent. Fork with Custom tools selected → the child’s agent still has the tool. On push, a brand-new tool on the child can be copied into the parent if you leave it selected under **Copy resources**. + +--- + +### Skills + +| | Behavior | +|---|----------| +| **Fork** | Optional copy (default on). Skill content comes along. Links inside the skill to files or other Sim resources are updated when those resources were also copied. Deselect → skill fields on agents clear. | +| **Sync** | Map or copy, same idea as custom tools. | + +**Example:** A “Support tone” skill on an agent. Deselecting it on fork clears the skill on the child’s agent until you attach one again. + +--- + +### External MCP servers + +Servers you connect to an external MCP endpoint (not “publish this workflow as MCP”). + +| | Behavior | +|---|----------| +| **Fork** | Optional copy (default on). **Connection settings** (URL, headers, transport) copy. **Sign-in / OAuth is not copied** — OAuth servers show up disconnected until someone signs in again in the child. Tool picks on MCP and agent blocks follow the new server. | +| **Sync** | Map or copy under the same rules. Mapping is typical when each workspace has its own server aimed at the same upstream system. | + +**Example:** An MCP server for an internal API. After fork, open MCP settings in the child and complete OAuth (or confirm API headers) before those tools will run. On sync, map child ↔ parent servers so tool selections survive push and pull. + +--- + +### Workflow MCP servers + +Servers that **publish workflows as MCP tools**. + +| | Behavior | +|---|----------| +| **Fork** | Optional under **Copy resources**. You get matching server shells in the child. When the workflow was also forked, its tool attachment on that server is carried over. | +| **Sync** | These are not shown in the mapping list. Attachments stay aligned as you sync — tools are added, updated, or removed to match the source. | + +**Example:** Parent exposes `Support triage` on a workflow MCP server. Fork with Workflow MCP servers selected → the child gets a matching server and the attachment to the child’s copy of the workflow. + +--- + +### Deployed chats + +| | Behavior | +|---|----------| +| **Fork** | Live chat deployments on copied workflows come along with a **new chat URL**. Conversation history is not copied — only the chat setup (including which blocks feed the chat). | +| **Sync** | If the target workflow has **no** chat yet, the source’s live chat is created there (again with a new URL). Existing chats on the target are left alone. Workflows that are not ready to deploy do not get a chat. | + +**Example:** Parent has a public chat on `Support triage`. The fork gets its own chat URL right away. A later push onto a parent workflow that never had a chat can create one there. + +--- + +### Credentials + +| | Behavior | +|---|----------| +| **Fork** | **Never copied.** Credential fields in workflows are cleared. Connect or pick credentials in the child. | +| **Sync** | **Map only.** Every credential used by the synced workflows must be mapped to a credential on the target (same kind of integration). Sync stays blocked until that is done. Then re-pick dependents (labels, calendars, channels, …). | + +**Example:** A Gmail block using “Support inbox.” Fork clears it. Before the first sync, map it to the child’s “Support inbox” credential and re-pick the label. + +--- + +### Secrets (environment variables) + +| | Behavior | +|---|----------| +| **Fork** | **Values never leave the source.** Workflow text still contains `{{KEY}}` names. Create matching secrets (or the names you will map to) under the child’s **Secrets**. | +| **Sync** | Map source key names to target key names. Values stay in each workspace. Unmapped required secrets block Sync. | + +**Example:** Workflows use `{{OPENAI_API_KEY}}`. After fork, add that secret in the child (or map `OPENAI_API_KEY` to whatever name the child uses) before runs and syncs succeed. + +--- + +### Public API + +| | Behavior | +|---|----------| +| **Fork** | Child workflows start **private** — not exposed as public API endpoints, even if the parent was. | +| **Sync** | The source workflow’s public API setting is carried. Push a public endpoint and the target becomes public too. | + +--- + +### Not carried + +These do not move at fork or sync time: + +- Undeployed workflows and local drafts +- Execution / run history +- Sim API keys +- Memory stores and scheduled jobs +- BYOK provider keys +- Permission groups (the child inherits the organization, but groups are not specially applied by forking) + +Schedules, webhooks, and triggers are not live in the child until you **deploy** there — same “deploy = commit” rule. + +--- + +## Edge cases to keep in mind + +- **Force overwrite** — Sync does not merge canvas changes. Anything only on the target that conflicts with the source’s deployed workflows can be lost. Use the confirm dialog’s archive list carefully. +- **Deselect on fork** — Cleared references are intentional. Prefer copying the resource, or plan to reconnect it in the child. +- **Background copy** — Right after fork, large tables / knowledge bases / files may still be copying. Check **Activity** if something looks empty. +- **OAuth MCP** — Expect a reconnect in the child (and after a copied server on sync). +- **Rollback ≠ undo copies** — Workflow versions roll back; copied resources can remain as orphans. +- **Disconnect is permanent** — You cannot “reconnect” the same edge; you would fork again into a new workspace. +- **No grandparent sync** — Only the direct parent↔child pair. + +--- + + + +--- + +## Self-hosted setup + +Self-hosted deployments turn Forks on with an environment variable instead of the Enterprise plan. + +| Variable | Description | +|----------|-------------| +| `FORKING_ENABLED`, `NEXT_PUBLIC_FORKING_ENABLED` | Enables workspace forking when billing is not used as the entitlement gate | + +Once enabled, use the same **Settings → Enterprise → Workspace Forks** UI as Sim Cloud. Only workspace admins can manage forks. diff --git a/apps/docs/content/docs/en/platform/enterprise/index.mdx b/apps/docs/content/docs/en/platform/enterprise/index.mdx index fef18101d99..6aee826a8af 100644 --- a/apps/docs/content/docs/en/platform/enterprise/index.mdx +++ b/apps/docs/content/docs/en/platform/enterprise/index.mdx @@ -5,7 +5,7 @@ description: Enterprise features for business organizations import { FAQ } from '@/components/ui/faq' -Sim Enterprise adds fine-grained access control, SSO, audit logging, and compliance features on top of Team plans. +Sim Enterprise adds fine-grained access control, SSO, audit logging, compliance features, and workspace forking on top of Team plans. --- @@ -65,8 +65,14 @@ Continuously export workflow logs, audit logs, and Mothership data to a customer --- +## Workspace Forks + +Clone a workspace into a linked child, then push or pull **deployed** workflow changes between them. Deploy is like a commit; sync is a force push or force pull. See the [workspace forks guide](/platform/enterprise/forks). + +--- + @@ -86,6 +92,7 @@ Self-hosted deployments enable enterprise features via environment variables ins | `AUDIT_LOGS_ENABLED`, `NEXT_PUBLIC_AUDIT_LOGS_ENABLED` | Audit logging | | `NEXT_PUBLIC_DATA_RETENTION_ENABLED` | Data retention configuration | | `DATA_DRAINS_ENABLED`, `NEXT_PUBLIC_DATA_DRAINS_ENABLED` | Data drains | +| `FORKING_ENABLED`, `NEXT_PUBLIC_FORKING_ENABLED` | Workspace forking | | `INBOX_ENABLED`, `NEXT_PUBLIC_INBOX_ENABLED` | Sim Mailer inbox | | `DISABLE_INVITATIONS`, `NEXT_PUBLIC_DISABLE_INVITATIONS` | Disable invitations; manage membership via Admin API | diff --git a/apps/docs/content/docs/en/platform/enterprise/meta.json b/apps/docs/content/docs/en/platform/enterprise/meta.json index 8ac0662175e..c8a33a7e287 100644 --- a/apps/docs/content/docs/en/platform/enterprise/meta.json +++ b/apps/docs/content/docs/en/platform/enterprise/meta.json @@ -4,10 +4,12 @@ "index", "sso", "access-control", + "custom-blocks", "whitelabeling", "audit-logs", "data-retention", - "data-drains" + "data-drains", + "forks" ], "defaultOpen": false } diff --git a/apps/docs/package.json b/apps/docs/package.json index 9f8d622f6cc..6c9e64cf5bb 100644 --- a/apps/docs/package.json +++ b/apps/docs/package.json @@ -50,8 +50,9 @@ "@types/node": "^22.14.1", "@types/react": "^19.1.2", "@types/react-dom": "^19.0.4", + "@typescript/native-preview": "7.0.0-dev.20260707.2", "postcss": "^8.5.3", "tailwindcss": "^4.0.12", - "typescript": "^5.8.2" + "typescript": "^7.0.2" } } diff --git a/apps/docs/public/static/enterprise/custom-blocks-canvas.png b/apps/docs/public/static/enterprise/custom-blocks-canvas.png new file mode 100644 index 00000000000..e952b15bb1b Binary files /dev/null and b/apps/docs/public/static/enterprise/custom-blocks-canvas.png differ diff --git a/apps/docs/public/static/enterprise/custom-blocks-form.png b/apps/docs/public/static/enterprise/custom-blocks-form.png new file mode 100644 index 00000000000..ed68e0e9ce5 Binary files /dev/null and b/apps/docs/public/static/enterprise/custom-blocks-form.png differ diff --git a/apps/docs/public/static/enterprise/custom-blocks-list.png b/apps/docs/public/static/enterprise/custom-blocks-list.png new file mode 100644 index 00000000000..18671499275 Binary files /dev/null and b/apps/docs/public/static/enterprise/custom-blocks-list.png differ diff --git a/apps/docs/public/static/enterprise/custom-blocks-toolbar.png b/apps/docs/public/static/enterprise/custom-blocks-toolbar.png new file mode 100644 index 00000000000..cd8fc3b8d92 Binary files /dev/null and b/apps/docs/public/static/enterprise/custom-blocks-toolbar.png differ diff --git a/apps/docs/public/static/enterprise/forks-activity.png b/apps/docs/public/static/enterprise/forks-activity.png new file mode 100644 index 00000000000..dd9398b34a2 Binary files /dev/null and b/apps/docs/public/static/enterprise/forks-activity.png differ diff --git a/apps/docs/public/static/enterprise/forks-copy-resources.png b/apps/docs/public/static/enterprise/forks-copy-resources.png new file mode 100644 index 00000000000..616b8204a1f Binary files /dev/null and b/apps/docs/public/static/enterprise/forks-copy-resources.png differ diff --git a/apps/docs/public/static/enterprise/forks-create-modal.png b/apps/docs/public/static/enterprise/forks-create-modal.png new file mode 100644 index 00000000000..ff20e2c78e7 Binary files /dev/null and b/apps/docs/public/static/enterprise/forks-create-modal.png differ diff --git a/apps/docs/public/static/enterprise/forks-create-warning.png b/apps/docs/public/static/enterprise/forks-create-warning.png new file mode 100644 index 00000000000..14e120bdc42 Binary files /dev/null and b/apps/docs/public/static/enterprise/forks-create-warning.png differ diff --git a/apps/docs/public/static/enterprise/forks-list.png b/apps/docs/public/static/enterprise/forks-list.png new file mode 100644 index 00000000000..5d31ccef839 Binary files /dev/null and b/apps/docs/public/static/enterprise/forks-list.png differ diff --git a/apps/docs/public/static/enterprise/forks-reconfigure.png b/apps/docs/public/static/enterprise/forks-reconfigure.png new file mode 100644 index 00000000000..32ea2605de1 Binary files /dev/null and b/apps/docs/public/static/enterprise/forks-reconfigure.png differ diff --git a/apps/docs/public/static/enterprise/forks-sync-confirm.png b/apps/docs/public/static/enterprise/forks-sync-confirm.png new file mode 100644 index 00000000000..74d5ea07dd2 Binary files /dev/null and b/apps/docs/public/static/enterprise/forks-sync-confirm.png differ diff --git a/apps/docs/public/static/enterprise/forks-sync-overview.png b/apps/docs/public/static/enterprise/forks-sync-overview.png new file mode 100644 index 00000000000..3ae743c1571 Binary files /dev/null and b/apps/docs/public/static/enterprise/forks-sync-overview.png differ diff --git a/apps/docs/tsconfig.json b/apps/docs/tsconfig.json index 1a45ee64711..857bd1f7da6 100644 --- a/apps/docs/tsconfig.json +++ b/apps/docs/tsconfig.json @@ -1,7 +1,6 @@ { "extends": "@sim/tsconfig/nextjs.json", "compilerOptions": { - "baseUrl": ".", "paths": { "@/.source/*": ["./.source/*"], "@/*": ["./*"] diff --git a/apps/pii/engines.py b/apps/pii/engines.py new file mode 100644 index 00000000000..c8edf982ff1 --- /dev/null +++ b/apps/pii/engines.py @@ -0,0 +1,267 @@ +"""Analyzer engine builders for the PII service. + +Two NER engines share one recognizer surface: + +- spacy (default): the 5 large spaCy models do NER (PERSON/LOCATION/NRP/ + DATE_TIME) and tokenization. +- gliner (opt-in): one multilingual GLiNER model does NER on CPU or GPU; + small spaCy models remain only for tokenization + lemmas. + +Both engines register the identical regex/checksum recognizer set (Presidio +defaults, EXTRA_RECOGNIZERS, VIN) — only the source of the 4 NER entity types +differs. Side-effect free: importing this module loads no models. +""" + +import importlib.util + +import spacy.util +from presidio_analyzer import AnalyzerEngine, Pattern, PatternRecognizer +from presidio_analyzer.nlp_engine import NlpEngineProvider +from presidio_analyzer.predefined_recognizers import ( + AuAbnRecognizer, + AuAcnRecognizer, + AuMedicareRecognizer, + AuTfnRecognizer, + EsNieRecognizer, + EsNifRecognizer, + FiPersonalIdentityCodeRecognizer, + GLiNERRecognizer, + InAadhaarRecognizer, + InPanRecognizer, + InPassportRecognizer, + InVehicleRegistrationRecognizer, + InVoterRecognizer, + ItDriverLicenseRecognizer, + ItFiscalCodeRecognizer, + ItIdentityCardRecognizer, + ItPassportRecognizer, + ItVatCodeRecognizer, + PlPeselRecognizer, + SgFinRecognizer, + SgUenRecognizer, + UkNinoRecognizer, +) + +# Languages served. Each needs its spaCy model installed in the image; the +# es/it/pl/fi predefined recognizers (ES_NIF, IT_FISCAL_CODE, PL_PESEL, ...) +# auto-load once their NLP engine is present. +NLP_CONFIGURATION = { + "nlp_engine_name": "spacy", + "models": [ + {"lang_code": "en", "model_name": "en_core_web_lg"}, + {"lang_code": "es", "model_name": "es_core_news_lg"}, + {"lang_code": "it", "model_name": "it_core_news_lg"}, + {"lang_code": "pl", "model_name": "pl_core_news_lg"}, + {"lang_code": "fi", "model_name": "fi_core_news_lg"}, + ], +} +SUPPORTED_LANGUAGES = [m["lang_code"] for m in NLP_CONFIGURATION["models"]] + +# The gliner engine still needs a spaCy pipeline per language: the regex +# recognizers consume NlpArtifacts and the LemmaContextAwareEnhancer boosts +# scores from surrounding lemmas. The small models (~12-40MB each vs ~400MB +# large) keep tokenization + lemmas intact while GLiNER owns NER. Blank +# pipelines ("blank:xx") are not an option: Presidio's SpacyNlpEngine treats +# unknown model names as pip packages and tries to download them. +# labels_to_ignore strips the small models' NER output from NlpArtifacts — +# correctness comes from removing SpacyRecognizer in build_gliner_analyzer; +# this only silences unmapped-label noise. +GLINER_NLP_CONFIGURATION = { + "nlp_engine_name": "spacy", + "models": [ + {"lang_code": "en", "model_name": "en_core_web_sm"}, + {"lang_code": "es", "model_name": "es_core_news_sm"}, + {"lang_code": "it", "model_name": "it_core_news_sm"}, + {"lang_code": "pl", "model_name": "pl_core_news_sm"}, + {"lang_code": "fi", "model_name": "fi_core_news_sm"}, + ], + "ner_model_configuration": { + "labels_to_ignore": [ + "CARDINAL", "DATE", "EVENT", "FAC", "GPE", "LANGUAGE", "LAW", + "LOC", "MISC", "MONEY", "NORP", "ORDINAL", "ORG", "PER", + "PERCENT", "PERSON", "PRODUCT", "QUANTITY", "TIME", "WORK_OF_ART", + ], + }, +} + +# Zero-shot label prompts -> the 4 Presidio NER entities GLiNER owns. Multiple +# prompts per entity trade a little inference cost for recall; tune against +# scripts/bench_engines.py output. +GLINER_ENTITY_MAPPING = { + "person": "PERSON", + "name": "PERSON", + "location": "LOCATION", + "address": "LOCATION", + "date": "DATE_TIME", + "time": "DATE_TIME", + "nationality": "NRP", + "religious group": "NRP", + "political group": "NRP", + "ethnic group": "NRP", +} + +# Predefined recognizers Presidio ships but does NOT load into the default +# registry — they must be added explicitly. Each carries its own +# supported_language, so it fires under that language once its NLP model is +# loaded. en: UK/AU/IN/SG locale ids; es/it/pl/fi: national ids. +EXTRA_RECOGNIZERS = [ + UkNinoRecognizer, + AuAbnRecognizer, + AuAcnRecognizer, + AuTfnRecognizer, + AuMedicareRecognizer, + InPanRecognizer, + InAadhaarRecognizer, + InVehicleRegistrationRecognizer, + InVoterRecognizer, + InPassportRecognizer, + SgFinRecognizer, + SgUenRecognizer, + EsNifRecognizer, + EsNieRecognizer, + ItFiscalCodeRecognizer, + ItDriverLicenseRecognizer, + ItVatCodeRecognizer, + ItPassportRecognizer, + ItIdentityCardRecognizer, + PlPeselRecognizer, + FiPersonalIdentityCodeRecognizer, +] + + +class VinRecognizer(PatternRecognizer): + """VIN (17 chars, A-Z/0-9 excluding I/O/Q) with ISO 3779 check-digit + validation (position 9). Validation makes accidental matches on arbitrary + 17-char codes (request ids, SKUs, tokens) extremely unlikely. Some + non-North-American VINs omit the check digit and are skipped — an + intentional bias toward precision. + """ + + _TRANSLIT = { + **{str(d): d for d in range(10)}, + "A": 1, "B": 2, "C": 3, "D": 4, "E": 5, "F": 6, "G": 7, "H": 8, + "J": 1, "K": 2, "L": 3, "M": 4, "N": 5, "P": 7, "R": 9, + "S": 2, "T": 3, "U": 4, "V": 5, "W": 6, "X": 7, "Y": 8, "Z": 9, + } + _WEIGHTS = [8, 7, 6, 5, 4, 3, 2, 10, 0, 9, 8, 7, 6, 5, 4, 3, 2] + + def validate_result(self, pattern_text: str): + vin = pattern_text.upper() + if len(vin) != 17: + return False + try: + total = sum(self._TRANSLIT[c] * w for c, w in zip(vin, self._WEIGHTS)) + except KeyError: + return False + check = total % 11 + expected = "X" if check == 10 else str(check) + return vin[8] == expected + + +class SharedModelGLiNERRecognizer(GLiNERRecognizer): + """Per-language GLiNER recognizer sharing ONE loaded model. + + Presidio routes recognizers by supported_language, so the registry holds + one instance per served language — but each instance's load() would pull + its own ~1.2GB model copy. The first instance loads (an ImportError from + a missing gliner package propagates — fail fast in the lean image); the + rest reuse the cached model. + """ + + _shared_models: dict = {} + + def load(self) -> None: + key = (self.model_name, self.map_location) + cached = self._shared_models.get(key) + if cached is None: + super().load() + self._shared_models[key] = self.gliner + else: + self.gliner = cached + + def analyze(self, text, entities, nlp_artifacts=None): + """GLiNERRecognizer appends any requested entity it doesn't know as an + ad-hoc zero-shot label and returns its hits. The analyzer passes ALL + supported entities (~40) when a request doesn't narrow them, which + would prompt GLiNER for CREDIT_CARD/VIN/ES_NIF/... — wrong scope, and + inference cost scales with label count. Restrict to the NER entities + this recognizer owns.""" + requested = [e for e in (entities or self.supported_entities) if e in self.supported_entities] + if not requested: + return [] + return super().analyze(text, requested, nlp_artifacts) + + +def _register_common_recognizers(analyzer: AnalyzerEngine) -> None: + """Regex/checksum recognizers shared by both engines.""" + # VIN is language-agnostic, so register it under every served language — + # a recognizer only fires for the language the caller routes to. + vin_pattern = Pattern(name="vin", regex=r"\b[A-HJ-NPR-Z0-9]{17}\b", score=0.7) + for language in SUPPORTED_LANGUAGES: + analyzer.registry.add_recognizer( + VinRecognizer( + supported_entity="VIN", + patterns=[vin_pattern], + context=["vin", "vehicle", "chassis"], + supported_language=language, + ) + ) + for recognizer_cls in EXTRA_RECOGNIZERS: + analyzer.registry.add_recognizer(recognizer_cls()) + + +def build_spacy_analyzer() -> AnalyzerEngine: + nlp_engine = NlpEngineProvider(nlp_configuration=NLP_CONFIGURATION).create_engine() + analyzer = AnalyzerEngine(nlp_engine=nlp_engine, supported_languages=SUPPORTED_LANGUAGES) + _register_common_recognizers(analyzer) + return analyzer + + +def build_gliner_analyzer(model_name: str, device: str | None) -> AnalyzerEngine: + """GLiNER engine: one multilingual zero-shot model replaces spaCy NER for + PERSON/LOCATION/NRP/DATE_TIME; everything else is unchanged. + + :param model_name: HuggingFace id of the GLiNER model. + :param device: torch device ("cpu", "cuda", "cuda:0"); None auto-detects + via Presidio's device_detector (cuda when available, else cpu). + """ + # Fail fast with an actionable message when gliner deps are missing (e.g. + # a custom-built image without them). Without these checks Presidio would + # try to pip-download the missing spaCy models at startup (a silent + # network fallback that dies with an unrelated pip permission error), and + # the gliner ImportError would surface only later. + if importlib.util.find_spec("gliner") is None: + raise RuntimeError( + "PII_ENGINE=gliner but the gliner package is not installed; " + "use the stock pii image (docker/pii.Dockerfile ships torch + gliner)" + ) + missing = [ + m["model_name"] + for m in GLINER_NLP_CONFIGURATION["models"] + if not spacy.util.is_package(m["model_name"]) + ] + if missing: + raise RuntimeError( + f"PII_ENGINE=gliner needs spaCy models {missing}; " + "use the stock pii image (docker/pii.Dockerfile ships them)" + ) + nlp_engine = NlpEngineProvider(nlp_configuration=GLINER_NLP_CONFIGURATION).create_engine() + analyzer = AnalyzerEngine(nlp_engine=nlp_engine, supported_languages=SUPPORTED_LANGUAGES) + # The default registry wires SpacyRecognizer per language; with GLiNER + # owning the NER entities it would emit duplicate/competing spans from the + # small models' ner pipe. remove_recognizer only logs when nothing matched, + # so assert the removal actually happened. + analyzer.registry.remove_recognizer("SpacyRecognizer") + if any(r.name == "SpacyRecognizer" for r in analyzer.registry.recognizers): + raise RuntimeError("SpacyRecognizer removal failed; Presidio registry layout changed") + for language in SUPPORTED_LANGUAGES: + analyzer.registry.add_recognizer( + SharedModelGLiNERRecognizer( + entity_mapping=GLINER_ENTITY_MAPPING, + model_name=model_name, + map_location=device, + supported_language=language, + ) + ) + _register_common_recognizers(analyzer) + return analyzer diff --git a/apps/pii/requirements-dev.txt b/apps/pii/requirements-dev.txt new file mode 100644 index 00000000000..4c330d999ae --- /dev/null +++ b/apps/pii/requirements-dev.txt @@ -0,0 +1,5 @@ +# Test-only deps. Unit tests need requirements.txt + this file (no models); +# integration tests additionally need the models baked into the docker images +# (see tests/test_integration.py). +pytest==8.4.1 +httpx==0.28.1 diff --git a/apps/pii/requirements-gliner.txt b/apps/pii/requirements-gliner.txt new file mode 100644 index 00000000000..d620c5f1825 --- /dev/null +++ b/apps/pii/requirements-gliner.txt @@ -0,0 +1,10 @@ +# Extras for the opt-in GLiNER engine — installed ONLY in the `gliner` +# Dockerfile target, on top of requirements.txt. Pinned for reproducible image +# builds; bump deliberately. presidio-analyzer 2.2.362 requires +# gliner >=0.2.13,<1.0.0. +# +# torch is pinned in the Dockerfile instead: the CPU and CUDA targets install +# the same version from different wheel indexes. +gliner==0.2.27 +transformers==4.56.2 +huggingface_hub==0.35.3 diff --git a/apps/pii/scripts/bench_engines.py b/apps/pii/scripts/bench_engines.py new file mode 100644 index 00000000000..b7ea70ca764 --- /dev/null +++ b/apps/pii/scripts/bench_engines.py @@ -0,0 +1,206 @@ +"""Benchmark + parity harness for the spacy vs gliner NER engines. + +Runs the same payload through both engines and reports per-engine throughput +(batch analyze, the production /redact_batch path) and per-text latency, plus +an accuracy diff over the 4 NER entity types (PERSON/LOCATION/NRP/DATE_TIME). +Non-NER (regex/checksum) results must be identical between engines — both +register the same recognizers — so any mismatch there is a wiring bug and the +script exits non-zero. + +Meant to run inside the pii image (both engines ship in it): + + docker run --rm python scripts/bench_engines.py + docker run --rm -v $PWD/texts.json:/data.json \\ + python scripts/bench_engines.py --payload /data.json + +Payload format: JSON list of {"text": str, "language": str} objects. +This doubles as the tuning harness for GLINER_ENTITY_MAPPING label prompts. +""" + +import argparse +import json +import statistics +import sys +import time +from collections import defaultdict +from pathlib import Path + +sys.path.insert(0, str(Path(__file__).resolve().parent.parent)) + +import engines # noqa: E402 + +# Entities sourced from the NER models rather than regex/checksum patterns. +# ORGANIZATION is emitted by the spacy engine's NER on unfiltered requests but +# is not in the app's supported set and has no GLiNER mapping — it shows up in +# the NER diff (spacy-only) rather than failing the regex-parity gate. +NER_ENTITIES = {"PERSON", "LOCATION", "NRP", "DATE_TIME", "ORGANIZATION"} +DEFAULT_PAYLOAD = Path(__file__).resolve().parent / "bench_payload.json" + + +def parse_args(): + parser = argparse.ArgumentParser(description=__doc__.splitlines()[0]) + parser.add_argument("--payload", type=Path, default=DEFAULT_PAYLOAD) + parser.add_argument("--engines", default="spacy,gliner") + parser.add_argument("--runs", type=int, default=3) + parser.add_argument("--warmup", type=int, default=1) + parser.add_argument("--device", default=None, help="torch device for gliner (default: auto)") + parser.add_argument("--gliner-model", default="urchade/gliner_multi_pii-v1") + parser.add_argument("--max-examples", type=int, default=10) + parser.add_argument("--json", action="store_true", help="emit machine-readable JSON") + return parser.parse_args() + + +def build(engine: str, args) -> tuple: + started = time.perf_counter() + if engine == "spacy": + analyzer = engines.build_spacy_analyzer() + elif engine == "gliner": + analyzer = engines.build_gliner_analyzer(model_name=args.gliner_model, device=args.device) + else: + raise ValueError(f"Unknown engine {engine!r}") + return analyzer, time.perf_counter() - started + + +def analyze_all(analyzer, items) -> list[list]: + """One analyze() call per text, in payload order.""" + return [analyzer.analyze(text=item["text"], language=item["language"]) for item in items] + + +def bench(analyzer, items, runs: int, warmup: int) -> dict: + for _ in range(warmup): + analyze_all(analyzer, items) + run_times = [] + latencies = [] + for _ in range(runs): + run_started = time.perf_counter() + for item in items: + text_started = time.perf_counter() + analyzer.analyze(text=item["text"], language=item["language"]) + latencies.append(time.perf_counter() - text_started) + run_times.append(time.perf_counter() - run_started) + total_chars = sum(len(item["text"]) for item in items) + avg_run = statistics.mean(run_times) + return { + "texts_per_sec": len(items) / avg_run, + "chars_per_sec": total_chars / avg_run, + "latency_p50_ms": statistics.median(latencies) * 1000, + "latency_p95_ms": statistics.quantiles(latencies, n=20)[18] * 1000, + } + + +def spans(results, keep_ner: bool) -> set: + return { + (r.entity_type, r.start, r.end) + for r in results + if (r.entity_type in NER_ENTITIES) == keep_ner + } + + +def iou(a: tuple, b: tuple) -> float: + inter = max(0, min(a[2], b[2]) - max(a[1], b[1])) + union = max(a[2], b[2]) - min(a[1], b[1]) + return inter / union if union else 0.0 + + +def diff_ner(items, results_a, results_b, max_examples: int) -> dict: + """Per-entity-type agreement between two engines (span IoU >= 0.5).""" + per_type = defaultdict(lambda: {"a_total": 0, "b_total": 0, "matched": 0}) + examples = [] + for item, res_a, res_b in zip(items, results_a, results_b): + a = sorted(spans(res_a, keep_ner=True)) + b = sorted(spans(res_b, keep_ner=True)) + unmatched_b = set(b) + for span_a in a: + per_type[span_a[0]]["a_total"] += 1 + match = next( + (s for s in unmatched_b if s[0] == span_a[0] and iou(span_a, s) >= 0.5), None + ) + if match: + per_type[span_a[0]]["matched"] += 1 + unmatched_b.discard(match) + for span_b in b: + per_type[span_b[0]]["b_total"] += 1 + only_a = [s for s in a if not any(s[0] == t[0] and iou(s, t) >= 0.5 for t in b)] + only_b = sorted(unmatched_b) + if (only_a or only_b) and len(examples) < max_examples: + examples.append( + { + "text": item["text"], + "language": item["language"], + "only_a": [f"{t}[{s}:{e}]={item['text'][s:e]!r}" for t, s, e in only_a], + "only_b": [f"{t}[{s}:{e}]={item['text'][s:e]!r}" for t, s, e in only_b], + } + ) + return {"per_type": dict(per_type), "examples": examples} + + +def diff_regex(items, results_a, results_b) -> list: + """Non-NER results must be identical: same recognizers on both engines.""" + mismatches = [] + for item, res_a, res_b in zip(items, results_a, results_b): + a = spans(res_a, keep_ner=False) + b = spans(res_b, keep_ner=False) + if a != b: + mismatches.append({"text": item["text"], "only_a": sorted(a - b), "only_b": sorted(b - a)}) + return mismatches + + +def main() -> int: + args = parse_args() + items = json.loads(args.payload.read_text()) + engine_names = [e.strip() for e in args.engines.split(",") if e.strip()] + + report = {"payload": str(args.payload), "texts": len(items), "engines": {}} + results_by_engine = {} + for name in engine_names: + analyzer, build_secs = build(name, args) + stats = bench(analyzer, items, runs=args.runs, warmup=args.warmup) + stats["build_secs"] = build_secs + report["engines"][name] = stats + results_by_engine[name] = analyze_all(analyzer, items) + + exit_code = 0 + if set(engine_names) >= {"spacy", "gliner"}: + report["ner_diff"] = diff_ner( + items, results_by_engine["spacy"], results_by_engine["gliner"], args.max_examples + ) + regex_mismatches = diff_regex( + items, results_by_engine["spacy"], results_by_engine["gliner"] + ) + report["regex_mismatches"] = regex_mismatches + if regex_mismatches: + exit_code = 1 + + if args.json: + print(json.dumps(report, indent=2, default=str)) + return exit_code + + for name, stats in report["engines"].items(): + print(f"\n== {name} ==") + print(f" build: {stats['build_secs']:.1f}s") + print(f" throughput: {stats['texts_per_sec']:.2f} texts/s ({stats['chars_per_sec']:.0f} chars/s)") + print(f" latency: p50 {stats['latency_p50_ms']:.1f}ms p95 {stats['latency_p95_ms']:.1f}ms") + if "ner_diff" in report: + print("\n== NER parity (spacy=a vs gliner=b, span IoU>=0.5) ==") + for entity, counts in sorted(report["ner_diff"]["per_type"].items()): + print( + f" {entity:<10} spacy={counts['a_total']:<4} gliner={counts['b_total']:<4} " + f"matched={counts['matched']}" + ) + for example in report["ner_diff"]["examples"]: + print(f"\n [{example['language']}] {example['text']}") + if example["only_a"]: + print(f" spacy only: {', '.join(example['only_a'])}") + if example["only_b"]: + print(f" gliner only: {', '.join(example['only_b'])}") + if report["regex_mismatches"]: + print("\n!! REGEX MISMATCHES (wiring bug — engines must agree on non-NER):") + for mismatch in report["regex_mismatches"]: + print(f" {mismatch}") + else: + print("\n regex/checksum entities: identical across engines ✓") + return exit_code + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/apps/pii/scripts/bench_payload.json b/apps/pii/scripts/bench_payload.json new file mode 100644 index 00000000000..fd0f207164d --- /dev/null +++ b/apps/pii/scripts/bench_payload.json @@ -0,0 +1,97 @@ +[ + { "text": "My name is John Smith and I live in Paris with my wife Marie.", "language": "en" }, + { + "text": "Dr. Angela Rodriguez will see you on March 14, 2026 at 3:30 PM in the Boston clinic.", + "language": "en" + }, + { + "text": "Contact Sarah O'Connor at sarah.oconnor@example.com or call (212) 555-0123.", + "language": "en" + }, + { "text": "The package ships to 45 Queen Street, Toronto, next Tuesday.", "language": "en" }, + { + "text": "Ahmed is a practicing Muslim from Egypt who moved to Berlin in 2019.", + "language": "en" + }, + { "text": "She is a Catholic Norwegian citizen born on 12/05/1988.", "language": "en" }, + { + "text": "Payment with card 4111111111111111 was declined yesterday at 10am.", + "language": "en" + }, + { + "text": "The vehicle VIN 1HGCM82633A004352 was registered to James Wilson in Ohio.", + "language": "en" + }, + { + "text": "His National Insurance number is AB123456C and he lives in Manchester.", + "language": "en" + }, + { + "text": "Meeting rescheduled: Friday, 9 January 2026, 14:00, with Priya Natarajan from Mumbai.", + "language": "en" + }, + { "text": "Send the invoice to accounts@acme.io before the end of Q1 2026.", "language": "en" }, + { + "text": "Klaus, a German engineer, and his Buddhist colleague Mei flew from Munich to Osaka.", + "language": "en" + }, + { + "text": "The Democrats and Republicans debated in Washington on election night.", + "language": "en" + }, + { + "text": "No PII here: the quarterly revenue grew 14% and margins held steady.", + "language": "en" + }, + { + "text": "Server request id a7f3k2m9x1q8w5z2b is not a VIN and not a person.", + "language": "en" + }, + { + "text": "Me llamo María García y vivo en Madrid desde el 3 de mayo de 2020.", + "language": "es" + }, + { "text": "Mi NIF es 12345678Z y mi correo es maria.garcia@ejemplo.es.", "language": "es" }, + { + "text": "El señor Javier Morales, ciudadano mexicano, llegó a Barcelona el lunes.", + "language": "es" + }, + { "text": "La reunión con Carmen será el 15 de junio de 2026 en Sevilla.", "language": "es" }, + { + "text": "Los musulmanes y los católicos convivieron durante siglos en Córdoba.", + "language": "es" + }, + { "text": "Mi chiamo Marco Rossi e abito a Roma vicino al Colosseo.", "language": "it" }, + { + "text": "Il codice fiscale di Maria Rossi è RSSMRA85T10A562S, nata il 10 dicembre 1985.", + "language": "it" + }, + { + "text": "Giulia Bianchi, cittadina italiana, si trasferì a Milano nel gennaio 2021.", + "language": "it" + }, + { + "text": "L'appuntamento con il dottor Ferrari è fissato per il 20 marzo 2026 a Torino.", + "language": "it" + }, + { "text": "Nazywam się Jan Kowalski i mieszkam w Warszawie od 2015 roku.", "language": "pl" }, + { + "text": "Mój numer PESEL to 44051401359, urodziłem się 14 maja 1944 w Krakowie.", + "language": "pl" + }, + { + "text": "Anna Nowak, obywatelka polska, spotka się z nami we wtorek 12 maja 2026.", + "language": "pl" + }, + { "text": "Katolicy i protestanci wspólnie świętowali w Gdańsku.", "language": "pl" }, + { + "text": "Nimeni on Matti Virtanen ja asun Helsingissä Töölön kaupunginosassa.", + "language": "fi" + }, + { + "text": "Henkilötunnukseni on 131052-308T ja synnyin Tampereella lokakuussa 1952.", + "language": "fi" + }, + { "text": "Liisa Korhonen muutti Ouluun maanantaina 5. tammikuuta 2026.", "language": "fi" }, + { "text": "Suomalaiset ja ruotsalaiset kilpailevat jääkiekossa joka vuosi.", "language": "fi" } +] diff --git a/apps/pii/server.py b/apps/pii/server.py index e2f7ad706d3..3f59cc540aa 100644 --- a/apps/pii/server.py +++ b/apps/pii/server.py @@ -3,148 +3,52 @@ Constructs one warm AnalyzerEngine (multi-language NLP + a native check-digit VIN recognizer) and one AnonymizerEngine at startup, exposing stock-compatible endpoints so a single PRESIDIO_URL serves both. + +NER engine selection (see engines.py): +- PII_ENGINE=spacy (default): the 5 large spaCy models, unchanged behavior. +- PII_ENGINE=gliner: one multilingual GLiNER model for PERSON/LOCATION/NRP/ + DATE_TIME. The stock image ships both engines, so this is a pure env flip. + PII_DEVICE picks cpu/cuda (unset = auto-detect), PII_GLINER_MODEL overrides + the model id. The same code runs on CPU and GPU. Each uvicorn worker + (PII_WORKERS) loads its own GLiNER model copy — into GPU memory when on + cuda — so GPU deployments generally want PII_WORKERS=1 per GPU, unlike the + CPU/spacy path where workers scale with vCPUs. """ import logging +import os import time from typing import Any +from engines import build_gliner_analyzer, build_spacy_analyzer from fastapi import FastAPI -from presidio_analyzer import ( - AnalyzerEngine, - BatchAnalyzerEngine, - Pattern, - PatternRecognizer, - RecognizerResult, -) -from presidio_analyzer.nlp_engine import NlpEngineProvider -from presidio_analyzer.predefined_recognizers import ( - AuAbnRecognizer, - AuAcnRecognizer, - AuMedicareRecognizer, - AuTfnRecognizer, - EsNieRecognizer, - EsNifRecognizer, - FiPersonalIdentityCodeRecognizer, - InAadhaarRecognizer, - InPanRecognizer, - InPassportRecognizer, - InVehicleRegistrationRecognizer, - InVoterRecognizer, - ItDriverLicenseRecognizer, - ItFiscalCodeRecognizer, - ItIdentityCardRecognizer, - ItPassportRecognizer, - ItVatCodeRecognizer, - PlPeselRecognizer, - SgFinRecognizer, - SgUenRecognizer, - UkNinoRecognizer, -) +from presidio_analyzer import AnalyzerEngine, BatchAnalyzerEngine, RecognizerResult from presidio_anonymizer import AnonymizerEngine from presidio_anonymizer.entities import OperatorConfig from pydantic import BaseModel -# Languages served. Each needs its spaCy model installed in the image; the -# es/it/pl/fi predefined recognizers (ES_NIF, IT_FISCAL_CODE, PL_PESEL, ...) -# auto-load once their NLP engine is present. -NLP_CONFIGURATION = { - "nlp_engine_name": "spacy", - "models": [ - {"lang_code": "en", "model_name": "en_core_web_lg"}, - {"lang_code": "es", "model_name": "es_core_news_lg"}, - {"lang_code": "it", "model_name": "it_core_news_lg"}, - {"lang_code": "pl", "model_name": "pl_core_news_lg"}, - {"lang_code": "fi", "model_name": "fi_core_news_lg"}, - ], -} -SUPPORTED_LANGUAGES = [m["lang_code"] for m in NLP_CONFIGURATION["models"]] - -# Predefined recognizers Presidio ships but does NOT load into the default -# registry — they must be added explicitly. Each carries its own -# supported_language, so it fires under that language once its NLP model is -# loaded. en: UK/AU/IN/SG locale ids; es/it/pl/fi: national ids. -EXTRA_RECOGNIZERS = [ - UkNinoRecognizer, - AuAbnRecognizer, - AuAcnRecognizer, - AuTfnRecognizer, - AuMedicareRecognizer, - InPanRecognizer, - InAadhaarRecognizer, - InVehicleRegistrationRecognizer, - InVoterRecognizer, - InPassportRecognizer, - SgFinRecognizer, - SgUenRecognizer, - EsNifRecognizer, - EsNieRecognizer, - ItFiscalCodeRecognizer, - ItDriverLicenseRecognizer, - ItVatCodeRecognizer, - ItPassportRecognizer, - ItIdentityCardRecognizer, - PlPeselRecognizer, - FiPersonalIdentityCodeRecognizer, -] - - -class VinRecognizer(PatternRecognizer): - """VIN (17 chars, A-Z/0-9 excluding I/O/Q) with ISO 3779 check-digit - validation (position 9). Validation makes accidental matches on arbitrary - 17-char codes (request ids, SKUs, tokens) extremely unlikely. Some - non-North-American VINs omit the check digit and are skipped — an - intentional bias toward precision. - """ - - _TRANSLIT = { - **{str(d): d for d in range(10)}, - "A": 1, "B": 2, "C": 3, "D": 4, "E": 5, "F": 6, "G": 7, "H": 8, - "J": 1, "K": 2, "L": 3, "M": 4, "N": 5, "P": 7, "R": 9, - "S": 2, "T": 3, "U": 4, "V": 5, "W": 6, "X": 7, "Y": 8, "Z": 9, - } - _WEIGHTS = [8, 7, 6, 5, 4, 3, 2, 10, 0, 9, 8, 7, 6, 5, 4, 3, 2] +PII_ENGINE = os.environ.get("PII_ENGINE", "spacy") +if PII_ENGINE not in ("spacy", "gliner"): + raise ValueError(f"Invalid PII_ENGINE={PII_ENGINE!r}; expected 'spacy' or 'gliner'") +# Empty/unset -> None -> auto-detect (cuda when torch sees a GPU, else cpu). +PII_DEVICE = os.environ.get("PII_DEVICE") or None +PII_GLINER_MODEL = os.environ.get("PII_GLINER_MODEL", "urchade/gliner_multi_pii-v1") - def validate_result(self, pattern_text: str): - vin = pattern_text.upper() - if len(vin) != 17: - return False - try: - total = sum(self._TRANSLIT[c] * w for c, w in zip(vin, self._WEIGHTS)) - except KeyError: - return False - check = total % 11 - expected = "X" if check == 10 else str(check) - return vin[8] == expected +# Propagates to uvicorn's root handler, so timing lands in the container log stream. +logger = logging.getLogger("sim.pii") def build_analyzer() -> AnalyzerEngine: - nlp_engine = NlpEngineProvider(nlp_configuration=NLP_CONFIGURATION).create_engine() - analyzer = AnalyzerEngine(nlp_engine=nlp_engine, supported_languages=SUPPORTED_LANGUAGES) - # VIN is language-agnostic, so register it under every served language — - # a recognizer only fires for the language the caller routes to. - vin_pattern = Pattern(name="vin", regex=r"\b[A-HJ-NPR-Z0-9]{17}\b", score=0.7) - for language in SUPPORTED_LANGUAGES: - analyzer.registry.add_recognizer( - VinRecognizer( - supported_entity="VIN", - patterns=[vin_pattern], - context=["vin", "vehicle", "chassis"], - supported_language=language, - ) - ) - for recognizer_cls in EXTRA_RECOGNIZERS: - analyzer.registry.add_recognizer(recognizer_cls()) - return analyzer + if PII_ENGINE == "gliner": + return build_gliner_analyzer(model_name=PII_GLINER_MODEL, device=PII_DEVICE) + return build_spacy_analyzer() +logger.info("building analyzer engine=%s device=%s", PII_ENGINE, PII_DEVICE or "auto") analyzer = build_analyzer() batch_analyzer = BatchAnalyzerEngine(analyzer_engine=analyzer) anonymizer = AnonymizerEngine() -# Propagates to uvicorn's root handler, so timing lands in the container log stream. -logger = logging.getLogger("sim.pii") - app = FastAPI(title="Sim Presidio", docs_url=None, redoc_url=None) diff --git a/apps/pii/tests/conftest.py b/apps/pii/tests/conftest.py new file mode 100644 index 00000000000..c9787a309d9 --- /dev/null +++ b/apps/pii/tests/conftest.py @@ -0,0 +1,5 @@ +import sys +from pathlib import Path + +# server.py / engines.py live one level up (repo: apps/pii, image: /app). +sys.path.insert(0, str(Path(__file__).resolve().parent.parent)) diff --git a/apps/pii/tests/test_engines.py b/apps/pii/tests/test_engines.py new file mode 100644 index 00000000000..1e6c1b8840c --- /dev/null +++ b/apps/pii/tests/test_engines.py @@ -0,0 +1,105 @@ +"""Unit tests for engines.py — no models, no downloads, no network. + +Run: pip install -r requirements.txt -r requirements-dev.txt && python -m pytest tests +""" + +import importlib.util +import os +import subprocess +import sys +from pathlib import Path + +import pytest +from presidio_analyzer.predefined_recognizers.ner import gliner_recognizer + +import engines + +PII_DIR = Path(__file__).resolve().parent.parent + + +class FakeModel: + def __init__(self): + self.seen_labels: list[list[str]] = [] + + def predict_entities(self, text, labels, flat_ner=True, threshold=0.3, multi_label=False): + self.seen_labels.append(list(labels)) + return [{"label": "person", "score": 0.92, "start": 0, "end": 4, "text": text[0:4]}] + + +class FakeGLiNER: + calls = 0 + + @classmethod + def from_pretrained(cls, model_name, **kwargs): + cls.calls += 1 + return FakeModel() + + +@pytest.fixture +def fake_gliner(monkeypatch): + monkeypatch.setattr(gliner_recognizer, "GLiNER", FakeGLiNER) + engines.SharedModelGLiNERRecognizer._shared_models.clear() + FakeGLiNER.calls = 0 + yield FakeGLiNER + engines.SharedModelGLiNERRecognizer._shared_models.clear() + + +def make_recognizer(language: str): + return engines.SharedModelGLiNERRecognizer( + entity_mapping=engines.GLINER_ENTITY_MAPPING, + model_name="fake/model", + map_location="cpu", + supported_language=language, + ) + + +def test_invalid_pii_engine_fails_import(): + result = subprocess.run( + [sys.executable, "-c", "import server"], + cwd=PII_DIR, + env={**os.environ, "PII_ENGINE": "bogus"}, + capture_output=True, + text=True, + ) + assert result.returncode != 0 + assert "Invalid PII_ENGINE" in result.stderr + + +@pytest.mark.skipif( + importlib.util.find_spec("gliner") is not None, + reason="fail-fast path only exists when gliner is not installed", +) +def test_build_gliner_analyzer_fails_fast_without_gliner(): + with pytest.raises(RuntimeError, match="gliner package is not installed"): + engines.build_gliner_analyzer(model_name="fake/model", device="cpu") + + +def test_shared_model_loads_once_across_languages(fake_gliner): + first = make_recognizer("en") + second = make_recognizer("es") + assert fake_gliner.calls == 1 + assert first.gliner is second.gliner + + +def test_analyze_never_prompts_gliner_with_foreign_entities(fake_gliner): + recognizer = make_recognizer("en") + all_supported = ["PERSON", "LOCATION", "NRP", "DATE_TIME", "CREDIT_CARD", "VIN", "ES_NIF"] + results = recognizer.analyze("John went home", entities=all_supported) + for labels in recognizer.gliner.seen_labels: + assert set(labels) <= set(engines.GLINER_ENTITY_MAPPING) + assert results and results[0].entity_type == "PERSON" + + +def test_analyze_skips_inference_when_no_owned_entity_requested(fake_gliner): + recognizer = make_recognizer("en") + assert recognizer.analyze("4111111111111111", entities=["CREDIT_CARD"]) == [] + assert recognizer.gliner.seen_labels == [] + + +def test_entity_mapping_targets_exactly_the_ner_entities(): + assert set(engines.GLINER_ENTITY_MAPPING.values()) == { + "PERSON", + "LOCATION", + "NRP", + "DATE_TIME", + } diff --git a/apps/pii/tests/test_integration.py b/apps/pii/tests/test_integration.py new file mode 100644 index 00000000000..40c97975754 --- /dev/null +++ b/apps/pii/tests/test_integration.py @@ -0,0 +1,102 @@ +"""Integration tests — exercise the real engines end-to-end via the FastAPI app. + +Requires the models present, so run inside the built images (gated behind +RUN_PII_INTEGRATION to keep plain `pytest` runs model-free): + + # spacy regression (default engine) + docker run --rm -e RUN_PII_INTEGRATION=1 python -m pytest tests + + # gliner engine + docker run --rm -e RUN_PII_INTEGRATION=1 -e PII_ENGINE=gliner \ + python -m pytest tests/test_integration.py + +The suite adapts to PII_ENGINE: shared assertions always run, engine-specific +ones only for the active engine. +""" + +import os + +import pytest + +if not os.environ.get("RUN_PII_INTEGRATION"): + pytest.skip( + "integration tests need the built image (RUN_PII_INTEGRATION=1)", + allow_module_level=True, + ) + +from fastapi.testclient import TestClient + +import server + +ENGINE = server.PII_ENGINE +client = TestClient(server.app) + + +def redact_batch(texts, language="en"): + response = client.post("/redact_batch", json={"texts": texts, "language": language}) + assert response.status_code == 200 + return response.json()["texts"] + + +def test_health(): + assert client.get("/health").json() == {"status": "ok"} + + +def test_masks_person_and_email(): + [masked] = redact_batch(["My name is John Smith, email john.smith@example.com."]) + assert "" in masked + assert "" in masked + assert "John Smith" not in masked + assert "john.smith@example.com" not in masked + + +def test_masks_location_and_phone(): + [masked] = redact_batch(["I live in Paris, call me at (212) 555-0123."]) + assert "" in masked + assert "" in masked + assert "Paris" not in masked + + +def test_regex_recognizers_fire_in_non_english_languages(): + [masked] = redact_batch(["Mi NIF es 12345678Z."], language="es") + assert "" in masked + # On the spacy engine the it_core_news_lg NER tags the fiscal code as + # ORGANIZATION and outscores the pattern recognizer, so only assert the + # value is masked; the exact label is checked on the gliner engine where + # spaCy NER can't compete. + [masked] = redact_batch(["Il codice fiscale è RSSMRA85T10A562S."], language="it") + assert "RSSMRA85T10A562S" not in masked + if ENGINE == "gliner": + assert "" in masked + + +def test_vin_checksum_recognizer_fires(): + [masked] = redact_batch(["The car VIN is 1HGCM82633A004352."]) + assert "" in masked + + +def test_no_pii_passes_through_unchanged(): + # NB: "Quarterly" would be tagged DATE_TIME by the spacy engine — keep + # this text free of anything either engine considers an entity. + text = "Revenue grew and margins held steady." + assert redact_batch([text]) == [text] + + +@pytest.mark.skipif(ENGINE != "gliner", reason="gliner-only wiring assertions") +def test_gliner_registry_has_no_spacy_recognizer(): + names = {r.name for r in server.analyzer.registry.recognizers} + assert "SpacyRecognizer" not in names + assert "GLiNERRecognizer" in names + + +@pytest.mark.skipif(ENGINE != "gliner", reason="gliner-only wiring assertions") +def test_gliner_supported_entities_keep_ner_types(): + supported = set(server.analyzer.get_supported_entities("en")) + assert {"PERSON", "LOCATION", "NRP", "DATE_TIME"} <= supported + + +@pytest.mark.skipif(ENGINE != "spacy", reason="spacy-only wiring assertions") +def test_spacy_registry_unchanged(): + names = {r.name for r in server.analyzer.registry.recognizers} + assert "SpacyRecognizer" in names + assert "GLiNERRecognizer" not in names diff --git a/apps/realtime/package.json b/apps/realtime/package.json index 83ca341b44d..633ffb60827 100644 --- a/apps/realtime/package.json +++ b/apps/realtime/package.json @@ -43,7 +43,7 @@ "@sim/tsconfig": "workspace:*", "@types/node": "24.2.1", "socket.io-client": "4.8.1", - "typescript": "^5.7.3", + "typescript": "^7.0.2", "vitest": "^4.1.0" } } diff --git a/apps/realtime/src/database/operations.ts b/apps/realtime/src/database/operations.ts index 4c2735a2087..7f514bda77f 100644 --- a/apps/realtime/src/database/operations.ts +++ b/apps/realtime/src/database/operations.ts @@ -576,6 +576,39 @@ async function handleBlockOperationTx( break } + case BLOCK_OPERATIONS.REPLACE_CANONICAL_MODES: { + if (!payload.id || !payload.data?.canonicalModes) { + throw new Error('Missing required fields for replace canonical modes operation') + } + + const existingBlock = await tx + .select({ data: workflowBlocks.data }) + .from(workflowBlocks) + .where(and(eq(workflowBlocks.id, payload.id), eq(workflowBlocks.workflowId, workflowId))) + .limit(1) + + const currentData = (existingBlock?.[0]?.data as Record) || {} + + const updateResult = await tx + .update(workflowBlocks) + .set({ + data: { + ...currentData, + canonicalModes: payload.data.canonicalModes, + }, + updatedAt: new Date(), + }) + .where(and(eq(workflowBlocks.id, payload.id), eq(workflowBlocks.workflowId, workflowId))) + .returning({ id: workflowBlocks.id }) + + if (updateResult.length === 0) { + throw new Error(`Block ${payload.id} not found in workflow ${workflowId}`) + } + + logger.debug(`Replaced block canonical modes: ${payload.id}`) + break + } + case BLOCK_OPERATIONS.TOGGLE_HANDLES: { if (!payload.id || payload.horizontalHandles === undefined) { throw new Error('Missing required fields for toggle handles operation') diff --git a/apps/realtime/src/handlers/variables.ts b/apps/realtime/src/handlers/variables.ts index 98dc3a5b7af..7a4303e70b3 100644 --- a/apps/realtime/src/handlers/variables.ts +++ b/apps/realtime/src/handlers/variables.ts @@ -1,3 +1,4 @@ +import { AuditAction, AuditResourceType, recordAudit } from '@sim/audit' import { db } from '@sim/db' import { workflow } from '@sim/db/schema' import { createLogger } from '@sim/logger' @@ -18,6 +19,9 @@ type PendingVariable = { latest: { variableId: string; field: string; value: any; timestamp: number } timeout: NodeJS.Timeout opToSocket: Map + /** Most recent writer, used as the audit actor when the coalesced update is persisted. */ + actorId: string + actorName?: string } // Keyed by `${workflowId}:${variableId}:${field}` @@ -177,6 +181,8 @@ export function setupVariablesHandlers(socket: AuthenticatedSocket, roomManager: if (existing) { clearTimeout(existing.timeout) existing.latest = { variableId, field, value, timestamp } + existing.actorId = session.userId + existing.actorName = session.userName if (operationId) existing.opToSocket.set(operationId, socket.id) existing.timeout = setTimeout(async () => { await flushVariableUpdate(workflowId, existing, roomManager) @@ -196,6 +202,8 @@ export function setupVariablesHandlers(socket: AuthenticatedSocket, roomManager: latest: { variableId, field, value, timestamp }, timeout, opToSocket, + actorId: session.userId, + actorName: session.userName, }) } } catch (error) { @@ -231,7 +239,11 @@ async function flushVariableUpdate( try { const workflowExists = await db - .select({ id: workflow.id }) + .select({ + id: workflow.id, + name: workflow.name, + workspaceId: workflow.workspaceId, + }) .from(workflow) .where(eq(workflow.id, workflowId)) .limit(1) @@ -294,6 +306,19 @@ async function flushVariableUpdate( }) if (updateSuccessful) { + const workflowRow = workflowExists[0] + recordAudit({ + workspaceId: workflowRow.workspaceId ?? null, + actorId: pending.actorId, + actorName: pending.actorName, + action: AuditAction.WORKFLOW_VARIABLES_UPDATED, + resourceType: AuditResourceType.WORKFLOW, + resourceId: workflowId, + resourceName: workflowRow.name ?? undefined, + description: `Updated workflow variables`, + metadata: { variableId, field }, + }) + // Broadcast to room excluding all senders (works cross-pod via Redis adapter) const senderSocketIds = [...pending.opToSocket.values()] const broadcastPayload = { diff --git a/apps/realtime/src/middleware/permissions.ts b/apps/realtime/src/middleware/permissions.ts index 00fc5c9580f..4f4a4296c16 100644 --- a/apps/realtime/src/middleware/permissions.ts +++ b/apps/realtime/src/middleware/permissions.ts @@ -28,6 +28,7 @@ const WRITE_OPERATIONS: string[] = [ BLOCK_OPERATIONS.UPDATE_PARENT, BLOCK_OPERATIONS.UPDATE_ADVANCED_MODE, BLOCK_OPERATIONS.UPDATE_CANONICAL_MODE, + BLOCK_OPERATIONS.REPLACE_CANONICAL_MODES, BLOCK_OPERATIONS.TOGGLE_HANDLES, // Batch block operations BLOCKS_OPERATIONS.BATCH_UPDATE_POSITIONS, diff --git a/apps/realtime/tsconfig.json b/apps/realtime/tsconfig.json index cce62771d4d..00b8ae857fc 100644 --- a/apps/realtime/tsconfig.json +++ b/apps/realtime/tsconfig.json @@ -1,9 +1,9 @@ { "extends": "@sim/tsconfig/base.json", "compilerOptions": { - "baseUrl": ".", + "lib": ["ES2022", "DOM"], "paths": { - "@/*": ["src/*"] + "@/*": ["./src/*"] } }, "include": ["src/**/*"], diff --git a/apps/sim/app/(auth)/login/login-form.tsx b/apps/sim/app/(auth)/login/login-form.tsx index d0f1fa59a29..9fd5adb148c 100644 --- a/apps/sim/app/(auth)/login/login-form.tsx +++ b/apps/sim/app/(auth)/login/login-form.tsx @@ -11,6 +11,7 @@ import { } from '@sim/emcn' import { createLogger } from '@sim/logger' import { getErrorMessage } from '@sim/utils/errors' +import { normalizeEmail } from '@sim/utils/string' import { useRouter, useSearchParams } from 'next/navigation' import { requestJson } from '@/lib/api/client/request' import { forgetPasswordContract } from '@/lib/api/contracts' @@ -45,7 +46,7 @@ const validateEmailField = (emailValue: string): string[] => { return errors } - const validation = quickValidateEmail(emailValue.trim().toLowerCase()) + const validation = quickValidateEmail(normalizeEmail(emailValue)) if (!validation.isValid) { errors.push(validation.reason || 'Please enter a valid email address.') } @@ -159,7 +160,7 @@ export default function LoginPage({ const formData = new FormData(e.currentTarget) const emailRaw = formData.get('email') as string - const email = emailRaw.trim().toLowerCase() + const email = normalizeEmail(emailRaw) const emailValidationErrors = validateEmailField(email) setEmailErrors(emailValidationErrors) @@ -277,7 +278,7 @@ export default function LoginPage({ return } - const emailValidation = quickValidateEmail(forgotPasswordEmail.trim().toLowerCase()) + const emailValidation = quickValidateEmail(normalizeEmail(forgotPasswordEmail)) if (!emailValidation.isValid) { setResetStatus({ type: 'error', diff --git a/apps/sim/app/(interfaces)/resume/[workflowId]/[executionId]/resume-page-client.tsx b/apps/sim/app/(interfaces)/resume/[workflowId]/[executionId]/resume-page-client.tsx index e6bf318c428..88a7cf3edae 100644 --- a/apps/sim/app/(interfaces)/resume/[workflowId]/[executionId]/resume-page-client.tsx +++ b/apps/sim/app/(interfaces)/resume/[workflowId]/[executionId]/resume-page-client.tsx @@ -18,6 +18,7 @@ import { TableRow, Tooltip, } from '@sim/emcn' +import { formatDateTime } from '@sim/utils/formatting' import { useQueryClient } from '@tanstack/react-query' import { RefreshCw } from 'lucide-react' import { useRouter } from 'next/navigation' @@ -68,7 +69,7 @@ const STATUS_BADGE_VARIANT: Record` prefetches its target route's JS once it's in the viewport - a navbar/hero CTA to `/signup` or `/login` is always in view, so it downloads that route's bundle on every pageview. Pass `prefetch={false}` there. Leave the default on CTAs that only enter the viewport on scroll (prefetch-on-approach is the desired behavior there). ## SEO @@ -70,6 +75,7 @@ Follow `.claude/rules/constitution.md` exactly: Sim is "the open-source AI works ├── page.tsx # route entry: metadata + ├── landing.tsx # root composition:
section order ├── workflows/ # a platform route: page.tsx (metadata) + workflows.tsx (config + shell) +├── hooks/ # cross-page client hooks (useLazyMount, …) - bare files, no folder/barrel └── components/ ├── index.ts # top barrel ├── navbar/{navbar.tsx, index.ts, components//…} #