Skip to content

fix(harbor-dev): gate teardown on deletionPolicy, verify the right Kustomization (PLT-1216) - #423

Open
bdchatham wants to merge 18 commits into
mainfrom
feat/plt-1216-harbor-dev-teardown-v2
Open

bdchatham wants to merge 18 commits into
mainfrom
feat/plt-1216-harbor-dev-teardown-v2

Conversation

@bdchatham

@bdchatham bdchatham commented Sep 9, 2026

Copy link
Copy Markdown
Collaborator

The ticket premise is false. PLT-1216 says teardown leaks disks because a prune option is missing. Prune is already enabled — platform/clusters/harbor/engineers/base/sync.yaml:14, in the shared base that renders all nine engineer reconcilers. It works correctly for everything Flux owns. This PR changes no Flux YAML, no prune setting, and nothing in the platform repo.

What actually leaks the disks: SeiNetwork.spec.deletionPolicy defaults to Retain. On deletion the reconciler deliberately strips the owner reference from the child SeiNodes rather than deleting them (sei-k8s-controller internal/controller/seinetwork/controller.go:135-143). The orphaned validators keep running, each still holding its PVC and its EBS volume. Flux prune cannot reach them — the controller created them, so they were never in Flux's inventory — and garbage collection has no owner reference left to follow. Storage-class reclaim is irrelevant, because reclaim only fires on a PVC that actually gets deleted. gp3-10k-750 and gp3 are already reclaimPolicy: Delete; nothing needs changing there.

This PR does not stop the leak. It is docs/skills only. The load-bearing fix is benchmark SeiNetworks carrying deletionPolicy: Delete, tracked separately. What this PR does is stop the harbor-dev skill from reporting a successful teardown while disks are still running, and give the operator the procedure that prevents the orphaning in the first place.

What this fixes

1. Wrong reconcile target. Teardown verification reconciled flux-system and read its lastAppliedRevision. That root Kustomization tracks sei-protocol/platform at clusters/harbor — not harbor-engineering-workspace. So teardown verified a reconciler that never touched the engineer's change and reported success regardless. Workspace merges now target Flux Kustomization <alias> in namespace eng-<alias> (verified in-repo: references/onboarding-pr.md:110, references/cluster-inspection-recipes.md recipe #8). flux-system stays correct for platform-repo merges (onboarding), and the split is now a table rather than a single command.

2. No resource-disappearance check. A successful reconcile only means Flux issued the deletes. Added an explicit poll of the SeiNetwork/SeiNodes/PVCs on a bounded budget, with the caveat stated: deletion is asynchronous and sei.io/seinode-finalizer holds a SeiNode until the controller releases its PVC. An object still Terminating past the budget is a reportable finding — stripping the finalizer to make the check pass is exactly how a stuck teardown becomes a leaked disk.

3. The deletionPolicy: Retain trap was undocumented. Now a guardrail, a procedure step, a halt condition, and a section of the new reference. deletionPolicy is mutable (no CEL XValidation, no webhook — SeiNetworkSpec carries exactly three immutability rules: spec.genesis, spec.replicas, spec.dataVolume), so an operator can patch a live SeiNetwork to Delete. The ordering is the load-bearing part: the policy change must land in git and reconcile before the removal merges. A bare kubectl patch on a Flux-owned SeiNetwork is drift — the rendered manifest normally carries the server-defaulted Retain, so Flux owns the field and reverts the patch on its next pass, typically while the removal PR is in review. And once a Retain deletion has stripped the owner references and removed the parent, nothing restores the cascade at all.

4. No "destroy my namespace" procedure. Added, and honest about the boundary: a workspace PR removes the engineer's workloads only. The Namespace object, the three ServiceAccounts, the RBAC, and the Flux Kustomization itself are platform-owned and stay. So does anything created outside git — an escape-hatch apply, or an orphan from an earlier Retain teardown.

5. Teardown skipped the PR contract. Spinup goes through a PR; teardown now does too — branch, commit, gh pr create, merge, reconcile, verify.

6. No cleanup path for already-leaked resources. Added. Orphan signature: sei.io/role=validator and no ownerReferences and no live parent. Absence of owner references alone is not the signal — a follower from seictl node apply is a top-level object and legitimately has none. Cleanup is kubectl delete seinode on the confirmed orphan: its finalizer deletes the PVC and the Delete reclaim policy releases the volume. Disk signals are candidates, never verdicts — EC2 available means unattached, not unowned, and a volume backing a live Bound PVC reads available the moment its workload stops. Ownership resolves by walking volume ID → PV (spec.csi.volumeHandle) → claimRef → PVC → workload; any hop that is Forbidden, errors, or finds nothing escalates as UNRESOLVED, never as confirmed-safe. This skill never deletes an EBS volume.

Explicitly not done

  • No Flux YAML, no prune setting, no platform-repo change. Prune is correct.
  • No delete on persistentvolumeclaims added to the per-engineer Role. That restriction is deliberate: the controller's finalizer owns PVC lifecycle, PVCs are not in the workspace repo so prune never targets them, and widening the Role would be a security regression that does not fix this bug. The new reference says so in place, so the next reader does not re-propose it.
  • No controller Go changes. No storage-class changes.

Files

File Change
references/teardown.md New. The canonical teardown reference: the Retain trap and patch-before-delete ordering, the PR-based procedure, the reconcile target + disappearance poll, the namespace procedures, the leaked-resource sweep, and seven halt conditions.
SKILL.md Post-merge reconciliation retargeted with a repo→Kustomization table; Guardrail #10; Procedure: tear down (PR-based); three halt conditions; teardown + namespace intent rows; reference-index entry; step-11/step-16 pointers.
references/cluster-inspection-recipes.md Recipes #9 (disappearance poll, stuck-Terminating read) and #10 (orphan detection); recipe #8 states it is the target for workspace merges; bench:teardown reconciles <alias> and confirms the objects went away.
references/seinetwork-crd.md deletionPolicy documented as the disk-leak field: the orphan mechanism, mutability, the closing window, and what the Delete cascade does end to end.
references/ephemeral-chain-flow.md Step 14 gates on deletionPolicy before the git rm.
references/comparative-bench.md Step 16, the single-teardown bullet, and the half-comparison halt: both SeiNetworks need the gate.
references/preflight.md Gate 2b's rationale names the right target per repo.
references/troubleshooting-seinode.md The finalizer-strip override says what it costs; the Retain note distinguishes forensics from a leak.
README.md Teardown is PR-based, and the skill gates on the field that leaks disks.
evals/evals.json Two evals: the Retain halt (empty ≠ safe; forbids the prune/reclaim/Role "fixes") and a clean teardown happy path.

Verification

Docs/skills only — no build, no module compile.

  • The teardown verifier distinguishes API failure from verified absence. It reports exactly one of GONE / PRESENT / UNVERIFIED, capturing kubectl's exit status separately rather than counting lines from a pipeline. A Forbidden or dropped connection is UNVERIFIED, never a pass. The poll deadline is arithmetic on date +%s; Bash's SECONDS is unset under sh, where the comparison fails with Illegal number and the loop never runs. Both behaviours were reproduced under this host's /bin/sh before and after the fix.
  • writing/scripts/lint.sh (Vale 3.17.1, after vale sync): 0 errors on every added line (530 added lines across 9 changed .md files). Warnings remain (sentence length, passive voice) at the same density as the surrounding file; CI's fail_level is error.
  • xreview/scripts/skill-package-checks.sh --skill-dir .claude/skills/harbor-dev: no block-severity failure. Same two pre-existing non-block failures as main (T2 info: missing state/.gitkeep; A1 warn: time-sensitive content in SKILL.md/preflight.md/seictl-cli.md). B1 passes — SKILL.md is 396 lines against the 500 limit.
  • writing/scripts/check-{verifiers,artifact-length,coverage,contract-anchors,anchor-authorities,generated-rules,admission,consumer-scoping,template-deltas}.sh: all pass.
  • scripts/verify-references.sh: no error-class finding. scripts/sync-skills.sh --verify: catalog coverage OK. scripts/tests/skill-package-checks.test.sh: 47 passed, 0 failed.
  • evals.json parses; the diff is 49 insertions, 0 deletions.

No command in this PR was run against a live cluster — none was available. Every documented invocation is derived from the repo's existing conventions.

Inferred rather than verified

  • --with-source on the workspace Kustomization. The GitRepository it references comes from the platform-repo base (clusters/harbor/engineers/base/sync.yaml), which is not in this repo, so its namespace is unverified. If it sits outside eng-<alias>, the engineer's namespace-scoped Role returns Forbidden. The docs state that failure mode and the fallback (reconcile the Kustomization alone) rather than asserting the flag always works.
  • EBS volume tag keys. kubernetes.io/created-for/pvc/namespace and .../pvc/name are the EBS CSI driver's own convention, not something these repos set. The doc tells the operator to confirm the keys against a known-live volume before trusting an empty filter result.
  • The offboarding file list. Derived by mirroring the verified onboarding four-file shape in references/onboarding-pr.md; no worked offboarding PR exists to check it against. Marked as such in place, and routed to the platform team rather than presented as a self-service runbook.

The Kustomization name for fix 1 was verified, not inferred<alias> in eng-<alias>, from references/onboarding-pr.md:110 and recipe #8.

🤖 Generated with Claude Code


Fix round — cross-review findings addressed

All three P1s, the P2, and the three non-blocking items are fixed on this branch. Commits 85cca82, f470156, a65e1b3, 8b9ffea.

# Finding Fix
P1-1 A failed API read reported success — kubectl | wc -l takes wc's exit status, so a Forbidden yielded zero lines and printed "all objects gone" The poll captures kubectl's status separately and reports one of GONE / PRESENT / UNVERIFIED, with distinct exit codes 0/1/2. UNVERIFIED is a halt condition. Reproduced the false pass under /bin/sh, then reproduced all three outcomes correctly after the fix.
P1-2 The live-patch fast path is revertible — Flux reconciles Retain from git back over the patch, typically mid-review Git-first is now the documented default: a policy PR, merged and reconciled, verified on both the committed file and the live object before the removal PR merges. The live patch survives only as a repair for a SeiNetwork no reconcile owns, and carries the re-verify-immediately-before-merge requirement. Propagated to SKILL.md, seinetwork-crd.md, ephemeral-chain-flow.md, comparative-bench.md, and both evals.
P1-3 available / Used By: <none> treated as proof of garbage, routing live disks into a deletion escalation Both demoted to candidate signals. Added an ownership walk (volume ID → PV spec.csi.volumeHandleclaimRef → PVC → workload) and a verdict table whose every failed hop lands on UNRESOLVED. kubectl get pv is cluster-scoped, so Forbidden is called out as the normal engineer case and an unresolved result. This skill now never deletes an EBS volume.
P2-4 Bash SECONDS in an sh fence skips the poll; PVC and bench checks were single reads; PVC expectation contradicted the imported-PVC exception Portable date +%s deadline. poll_gone in recipe #9 is reused for CRs, PVCs, pods, and bench Jobs/ConfigMaps — bench now polls pods too, since a Job can be gone while its pod is Terminating. The PVC expectation is no longer zero: controller-managed PVCs must go, imported PVCs must survive, and inventory step 2 now records which nodes carry spec.import because nothing says so after they are deleted.
NB Unsupported namespace-cascade claim Restated as an operational preference. The namespace controller does remove namespaced resources; emptying first is about keeping the gate, the poll, and the sweep usable while the objects still exist.
NB Non-Git resources acknowledged but given no cleanup path Added a table covering direct-applied CRs, SeiNodeTaskWorkflow, hand-applied bench Jobs, controller-managed PVCs, and S3 artifacts — each with an escalation or a guarded path. Anything not in it escalates as unresolved.
NB Forbidden on --with-source stated as diagnostic Now "may indicate" the cross-namespace GitRepository, alongside expired session, missing access entry, or a Role without the Flux verbs. Read the message first.
NB "run without --filters" would enumerate other tenants' volumes Replaced with --volume-ids <known-live-id>, scoped to one volume the operator already knows.

Also corrected, and it was wrong in the original body as well as the doc: spec.resources was listed among the immutable SeiNetworkSpec fields. There are exactly three immutability rules — spec.genesis, spec.replicas, spec.dataVolume — and SeiNetworkSpec has no resources field. Removed from both.

Untouched by design, per the review: the reconcile target (A), PR parity (E), the orphan-detection triple, the legitimate flux-system command in preflight.md, and the three inference caveats.

Gates after the fix round

  • Vale: 0 errors on all 530 added lines across 9 changed .md files.
  • skill-package-checks.sh: 20 pass / 2 fail / 4 skipped — the same two non-block failures as main (T2 info, A1 warn), no block failure. B1 passes at 396/500 lines.
  • All 9 writing/scripts/check-*.sh: pass. verify-references.sh, sync-skills.sh --verify, skill-package-checks.test.sh (47/0): pass.
  • evals.json parses; internal anchors resolve.

Round 3 — cross-review findings addressed

All nine fixed. Commits dfc6d90, 4b2e6ab, 95b3f7c, 45a623c. No finding was disputed; findings 1, 2 and 4 were reproduced or re-derived locally before fixing.

# Finding Fix
1 (block) teardown.md re-implemented the poll twice instead of calling the shared poll_gone, and the copies had already diverged — the PVC loop emitted no verdict at all, just counts until its deadline Both call sites now invoke the shared functions. Recipe #9 is the single implementation and says so.
2 (block) UNVERIFIED still exited 0 — break succeeds, so the completed while returned 0, and no caller checked poll_gone's return code either Added record/VERDICT aggregation where the worst outcome wins (0 GONE < 1 PRESENT < 2 UNVERIFIED) and every call site records. Confirmed under dash: break-after-UNVERIFIED does exit 0; after the fix, UNVERIFIED then GONE aggregates to 2.
3 (block) The "engineer insists" exception let a pre-merge re-read stand in for a committed policy Exception deleted for Flux-owned networks, with the four-step losing sequence written out — a pre-merge read narrows the window but does not order against Flux's reconcile. Path B is now a stacked draft PR; a merged PR cannot carry a later removal commit.
4 (block) .spec.import does not exist — the field is .spec.dataVolume.import.pvcName, so the query matched nothing and every imported claim was reclassified as one that must disappear Inventory rewritten: real node→claim identities from the pods' own claimName, imported vs controller-managed classification, abort on any API or parse failure. Poll named managed claims; assert imported ones with expect_present. Field path caveated as repo-main API inspection, not the deployed CRD. Also corrected the three pre-existing spec.import prose mentions elsewhere in the skill — see "scope call" below.
5 EBS walk: no PVC→node relationship; kubectl-into-jq hid API failures; missing-claim rows read as plain candidates Hop 3 resolves the claim to the mounting pod and its owner reference; an empty result is unresolved, not unowned. Every hop separates the API call from the parse and checks both. Missing-claim and no-PV rows are now explicitly unresolved candidates requiring platform review.
6 Scope guards: cluster-wide claimRef followed without a namespace check; deletes without namespace/context; a failed grep read as "no manifest" Added the claim_ns != eng-<alias> escalation gate; every delete names --context harbor and -n eng-<alias>; the authorizing search runs against a freshly fetched clone and branches on grep's exit status — only 1 (no match) authorizes a delete, 2+ is unresolved. Verified the three exit codes.
7 sed -n '/Used By/,+3p' is a GNU extension; blanket POSIX claim That parse is gone (replaced by the hop-3 jq). The claim now reads "portable shell (dash, ash, bash)" and names date +%s as a deliberate near-universal extension rather than POSIX.
8 Evals preserved the unsafe exception and let the happy path proceed on live Delete alone Both scenarios now require committed Delete for a Flux-owned object; the live-patch forbidden signal no longer has an escape clause; the happy path requires named-claim polling with aggregation.
9 --with-source fallback claimed to help "whatever the cause" Now: it helps only the cross-namespace-source case, repairs nothing for expired credentials or a missing permission, and the fallback itself must succeed or you have no reconcile at all.

One extra defect found while fixing 4. jq's unique sorts by codepoint; comm assumes its input is sorted the way the locale collates, and a locale that ignores punctuation orders hyphenated claim names differently. The inventory script now pins LC_ALL=C.

Scope call to flag. Finding 4 was scoped to teardown.md, but the same wrong path appeared in three pre-existing prose lines in troubleshooting-seinode.md and one in seinetwork-crd.md. Leaving them would have made the skill contradict itself on an API path, so they were corrected to spec.dataVolume.import — prose only, no query. Revert if you would rather keep that out of this PR.

Gates after round 3

  • Vale: 0 errors on all 669 added lines across 9 changed .md files (one write-good.ThereIs introduced and fixed mid-round).
  • skill-package-checks.sh: 20 pass / 2 fail / 4 skipped — the same two non-block failures as main (T2 info, A1 warn). B1 passes at 396/500.
  • All 9 writing/scripts/check-*.sh pass. verify-references.sh, sync-skills.sh --verify, skill-package-checks.test.sh (47/0) pass. evals.json parses; internal anchors resolve.
  • Not executed: the jq filters. This host has no jq, and installing one was out of budget — they are reviewed by inspection only. The shell control flow around them (aggregation, expect_present, set -eu abort on a redirected failure, grep exit codes, comm) was executed under /bin/sh.

bdchatham and others added 3 commits September 9, 2026 21:37
Documents the deletionPolicy: Retain trap that leaks EBS disks, the
patch-before-delete ordering, the workspace-Kustomization reconcile
target, the resource-disappearance poll, the namespace procedures, and
the cleanup path for already-orphaned SeiNodes and volumes.

Co-authored-by: omnigent <noreply@omnigent.ai>
… on deletionPolicy

Post-merge reconciliation targeted `flux-system`, which tracks the
platform repo — a workspace-repo merge was verified against a
reconciler that never applied it. Route workspace merges to
`Kustomization <alias>` in `eng-<alias>` and keep `flux-system` for
platform merges.

Add Guardrail #10, a PR-based teardown procedure with the
patch-before-delete ordering and a resource-disappearance poll, three
halt conditions, and the `teardown.md` reference-index entry.

Co-authored-by: omnigent <noreply@omnigent.ai>
Recipes #9 (disappearance poll) and #10 (orphaned-SeiNode detection);
`bench:teardown` now reconciles the engineer's Kustomization and
confirms the objects went away. The SeiNetwork CRD reference documents
deletionPolicy as the disk-leak field, its mutability, and the closing
window. Chain-flow, comparative-bench, preflight, troubleshooting, and
the package README point at the gated procedure. Two evals cover the
Retain halt and the clean teardown.

Co-authored-by: omnigent <noreply@omnigent.ai>
@cursor

cursor Bot commented Sep 9, 2026

Copy link
Copy Markdown

PR Summary

Low Risk
Changes are limited to Claude skill documentation and evals; no production cluster, Flux, or controller behavior is modified.

Overview
Docs-only update to the harbor-dev skill so agents stop treating a workspace teardown as done while validator disks are still running. No Flux, platform, or controller code changes.

The skill now documents that default SeiNetwork.spec.deletionPolicy: Retain orphans generated validators (and their PVCs/EBS volumes) when manifests are removed, and adds Guardrail #10 plus a full Procedure: tear down backed by new references/teardown.md.

Post-merge behavior is corrected: workspace-repo merges must reconcile Flux Kustomization <alias> in eng-<alias>, not flux-system, then poll until resources are gone with GONE / PRESENT / UNVERIFIED outcomes (API failures must not read as success). Shared shell helpers live in cluster-inspection-recipes.md recipe #9 (verify_teardown, poll_gone, sweep_residual); recipe #10 covers orphan detection.

Teardown is explicitly git-first: set deletionPolicy: Delete in the manifest and reconcile before the removal PR (not a bare kubectl patch on Flux-owned objects). Comparative bench, ephemeral-chain, and bench teardown docs point at the same procedure; evals/evals.json adds halt and happy-path teardown scenarios. Minor fixes align imported-PVC field paths to spec.dataVolume.import in troubleshooting prose.

Reviewed by Cursor Bugbot for commit 2218e68. Bugbot is set up for automated code reviews on this repo. Configure here.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remaining comments which cannot be posted as a review comment to avoid GitHub Rate Limit

vale

⚠️ [vale] <AgenticWriting.STE-SentenceLength-Description> reported by reviewdog 🐶
Descriptive sentence has 31 words. ASD-STE100 allows 25. Split it.

Keep `Retain` only to preserve a validator's disk for forensics after the network goes away, and say so where the choice is made — a retained disk is a cost somebody chose.


⚠️ [vale] <AgenticWriting.STE-Passive> reported by reviewdog 🐶
Passive voice: 'is made'. ASD-STE100 asks for the active voice.

Keep `Retain` only to preserve a validator's disk for forensics after the network goes away, and say so where the choice is made — a retained disk is a cost somebody chose.


⚠️ [vale] <AgenticWriting.STE-SentenceLength-Description> reported by reviewdog 🐶
Descriptive sentence has 29 words. ASD-STE100 allows 25. Split it.

Teardown removes an engineer's workloads from `eng-<alias>` through the same PR contract that created them. One ordering rule governs the whole file: **patch `spec.deletionPolicy` to `Delete` on every SeiNetwork you are about to remove, and land that patch before the removal merges.** A SeiNetwork deleted under the default `Retain` orphans its generated validator SeiNodes, and each orphan keeps its PVC and its EBS disk running with nothing left to clean it up.


⚠️ [vale] <AgenticWriting.STE-SentenceLength-Description> reported by reviewdog 🐶
Descriptive sentence has 30 words. ASD-STE100 allows 25. Split it.

That window closes at deletion. Once a `Retain` deletion has stripped the owner references and removed the parent SeiNetwork, no patch brings the cascade back — the parent is gone and the children are top-level objects. The leftover SeiNodes and PVCs then need the manual cleanup in [Find and clean up already-leaked resources](#find-and-clean-up-already-leaked-resources).


⚠️ [vale] <AgenticWriting.STE-Passive> reported by reviewdog 🐶
Passive voice: 'is gone'. ASD-STE100 asks for the active voice.

That window closes at deletion. Once a `Retain` deletion has stripped the owner references and removed the parent SeiNetwork, no patch brings the cascade back — the parent is gone and the children are top-level objects. The leftover SeiNodes and PVCs then need the manual cleanup in [Find and clean up already-leaked resources](#find-and-clean-up-already-leaked-resources).


⚠️ [vale] <AgenticWriting.STE-Passive> reported by reviewdog 🐶
Passive voice: 'be deleted'. ASD-STE100 asks for the active voice.

The patch mutates a live object outside git. That is acceptable here only because the object is about to be deleted, and only after the verify read prints `Delete`. Say in the removal PR body that the patch ran, so the reviewer sees the whole teardown.


⚠️ [vale] <AgenticWriting.STE-SentenceLength-Description> reported by reviewdog 🐶
Descriptive sentence has 35 words. ASD-STE100 allows 25. Split it.

`--dry-run` runs server-side apply against the apiserver, so a wrong path fails the render rather than the teardown. Keep `Retain` only when the engineer wants a validator's disk preserved for forensics after the network goes away, and say so in the PR body — a retained disk is a cost the engineer is choosing.


⚠️ [vale] <AgenticWriting.STE-SentenceLength-Description> reported by reviewdog 🐶
Descriptive sentence has 37 words. ASD-STE100 allows 25. Split it.

With `deletionPolicy: Delete` the chain runs end to end: SeiNetwork deleted → generated validator SeiNodes deleted through their owner references → each SeiNode's finalizer (`sei.io/seinode-finalizer`) deletes the node's data PVC → the storage class's `Delete` reclaim policy releases the EBS volume.


⚠️ [vale] <AgenticWriting.STE-Passive> reported by reviewdog 🐶
Passive voice: 'is preserved'. ASD-STE100 asks for the active voice.

The finalizer **skips an imported PVC** (`spec.import` set on the SeiNode). An imported PVC is preserved by design; its disk is not a leak.


⚠️ [vale] <AgenticWriting.STE-Passive> reported by reviewdog 🐶
Passive voice: 'are required'. ASD-STE100 asks for the active voice.

5. **Remove the manifests**`git rm -r engineers/<alias>/<task>/` **and** remove the `<task>` entry from `engineers/<alias>/kustomization.yaml`'s `resources:` list. Both edits are required: Kustomize fails to render with a missing-resource entry, and Flux then applies nothing at all.


⚠️ [vale] <AgenticWriting.STE-SentenceLength-Description> reported by reviewdog 🐶
Descriptive sentence has 27 words. ASD-STE100 allows 25. Split it.

7. **Open the PR** — title `feat(eng/<alias>): tear down <task>`. The body names the chain-id, every CR that goes away, the `deletionPolicy` value the SeiNetwork now carries, and the patch path (A or B) that set it. `gh pr create --repo sei-protocol/harbor-engineering-workspace --base main`.


⚠️ [vale] <AgenticWriting.STE-Passive> reported by reviewdog 🐶
Passive voice: 'is burned'. ASD-STE100 asks for the active voice.

9. **Report what survives** — the chain-id's S3 genesis artifacts are **not** purged by teardown, so the chain-id is burned. A later respin uses a fresh chain-id or purges the `<chain-id>/` prefix in `harbor-sei-k8s-genesis-artifacts` first.


⚠️ [vale] <AgenticWriting.STE-Passive> reported by reviewdog 🐶
Passive voice: 'are applied'. ASD-STE100 asks for the active voice.

The engineer's manifests are applied by the Flux `Kustomization <alias>` in namespace `eng-<alias>`, which watches `harbor-engineering-workspace` at `./engineers/<alias>` and reconciles every 5 minutes. The root `flux-system` Kustomization tracks `sei-protocol/platform` at `clusters/harbor`. Reconciling `flux-system` after a workspace-repo merge reconciles a different repository and reports success without applying the engineer's change.


⚠️ [vale] <AgenticWriting.STE-Passive> reported by reviewdog 🐶
Passive voice: 'are gone'. ASD-STE100 asks for the active voice.

A successful reconcile says Flux applied the change. It does not say the objects are gone. Deletion is asynchronous and finalizers hold objects in `Terminating` while the controller releases their PVCs, so poll instead of asserting once:


⚠️ [vale] <AgenticWriting.STE-Passive> reported by reviewdog 🐶
Passive voice: 'be gone'. ASD-STE100 asks for the active voice.

Every PVC belonging to the torn-down chain must be gone. A `Bound` PVC that outlives its SeiNode is a held disk.


⚠️ [vale] <AgenticWriting.STE-Passive> reported by reviewdog 🐶
Passive voice: 'are held'. ASD-STE100 asks for the active voice.

If the poll runs out with objects still present, do not report the teardown as done and do not force the objects away. Read why they are held:


⚠️ [vale] <write-good.Weasel> reported by reviewdog 🐶
'very' is a weasel word!

"Destroy my namespace" means one of two very different things. Ask which before acting.


⚠️ [vale] <AgenticWriting.STE-SentenceLength-Description> reported by reviewdog 🐶
Descriptive sentence has 30 words. ASD-STE100 allows 25. Split it.

Empty the namespace first, through the steps above. Deleting the `Namespace` object while SeiNetworks still live in it starts a namespace-wide cascade that races the controller's finalizers and can strand PVCs with no owning CR to inspect.


⚠️ [vale] <AgenticWriting.STE-Passive> reported by reviewdog 🐶
Passive voice: 'is gone'. ASD-STE100 asks for the active voice.

Each line is a validator still running with nothing that will ever delete it. Confirm the parent is gone before treating one as an orphan:


⚠️ [vale] <AgenticWriting.STE-Passive> reported by reviewdog 🐶
Passive voice: 'is attached'. ASD-STE100 asks for the active voice.

An orphaned SeiNode still shows a **`Bound`** PVC — the disk is attached and billing, not free-floating. A disk whose PVC has already gone shows up on the AWS side as **`available`**. Check both.


⚠️ [vale] <AgenticWriting.STE-Passive> reported by reviewdog 🐶
Passive voice: 'was provisioned'. ASD-STE100 asks for the active voice.

On the AWS side, the EBS CSI driver tags each volume with the PVC it was provisioned for:


⚠️ [vale] <AgenticWriting.STE-SentenceLength-Description> reported by reviewdog 🐶
Descriptive sentence has 42 words. ASD-STE100 allows 25. Split it.

`state: in-use` with an orphaned SeiNode above it is a running leak. `state: available` is a disk nothing references at all. Those tag keys are the EBS CSI driver's own convention rather than something this skill's repos set — run the command once without `--filters` against a volume you know is live to confirm the keys are present before trusting an empty result.


⚠️ [vale] <AgenticWriting.STE-Passive> reported by reviewdog 🐶
Passive voice: 'is ******'. ASD-STE100 asks for the active voice.

The node's finalizer deletes its data PVC, and the `Delete` reclaim policy on `gp3-10k-750` (validators) and `gp3` (default) releases the EBS volume. `gp3-archive` is `Retain` by design — a volume on that class stays after its PVC goes, and its removal is an AWS-side decision, not a mistake to correct here.


⚠️ [vale] <AgenticWriting.STE-SentenceLength-Description> reported by reviewdog 🐶
Descriptive sentence has 27 words. ASD-STE100 allows 25. Split it.

The node's finalizer deletes its data PVC, and the `Delete` reclaim policy on `gp3-10k-750` (validators) and `gp3` (default) releases the EBS volume. `gp3-archive` is `Retain` by design — a volume on that class stays after its PVC goes, and its removal is an AWS-side decision, not a mistake to correct here.


⚠️ [vale] <AgenticWriting.STE-Passive> reported by reviewdog 🐶
Passive voice: 'is gone'. ASD-STE100 asks for the active voice.

- **The removal PR merged while a SeiNetwork still read `Retain`.** The cascade is gone and no patch restores it. Do not re-apply the SeiNetwork to "reattach" the children — a fresh network under the same chain-id wedges at height 0 on the burned genesis artifacts. Go straight to [Find and clean up already-leaked resources](#find-and-clean-up-already-leaked-resources).


⚠️ [vale] <AgenticWriting.STE-SentenceLength-Description> reported by reviewdog 🐶
Descriptive sentence has 27 words. ASD-STE100 allows 25. Split it.

- **The removal PR merged while a SeiNetwork still read `Retain`.** The cascade is gone and no patch restores it. Do not re-apply the SeiNetwork to "reattach" the children — a fresh network under the same chain-id wedges at height 0 on the burned genesis artifacts. Go straight to [Find and clean up already-leaked resources](#find-and-clean-up-already-leaked-resources).


⚠️ [vale] <AgenticWriting.STE-Passive> reported by reviewdog 🐶
Passive voice: 'is ********'. ASD-STE100 asks for the active voice.

- **`kustomization <alias>` is `NotFound` in `eng-<alias>`.** The engineer's Flux wiring is missing, so no workspace-repo merge reconciles at all. Surface to the platform team; do not create the Kustomization.


⚠️ [vale] <AgenticWriting.STE-SentenceLength-Description> reported by reviewdog 🐶
Descriptive sentence has 27 words. ASD-STE100 allows 25. Split it.

- **`lastAppliedRevision` does not reach the merge commit within two reconcile intervals (~10 min).** Read the Ready condition's message (`cluster-inspection-recipes.md` recipe #8). A render error in `engineers/<alias>/kustomization.yaml` — most often a `resources:` entry pointing at the dir that was just removed — blocks every later apply in the namespace, not only this teardown.


⚠️ [vale] <AgenticWriting.STE-SentenceLength-Description> reported by reviewdog 🐶
Descriptive sentence has 27 words. ASD-STE100 allows 25. Split it.

This abandons the PVC and its EBS disk — the finalizer is what deletes the PVC, so removing it is how a stuck teardown becomes a leaked disk. Take it only with the engineer's explicit acceptance, record the PVC name, and follow up with the sweep in `teardown.md`*find and clean up already-leaked resources*.


⚠️ [vale] <AgenticWriting.STE-Passive> reported by reviewdog 🐶
Passive voice: 'is governed'. ASD-STE100 asks for the active voice.

For a SeiNode, whether its PVC survives deletion is governed by `spec.import` (imported PVC = preserved) vs controller-managed (wiped on teardown) — documented under **Phase: Failed** above. A `SeiNetwork`'s `spec.deletionPolicy` (defaults `Retain`) governs whether the controller orphans its generated validator SeiNodes (and thus their PVCs) when the network is deleted. Forensics is the one case where `Retain` is the right answer; on an ordinary teardown it is a disk leak, because the orphaned validators keep running with no owner left to delete them (see `teardown.md`). The hardlink trick above is for **live debugging** while the node continues running. The two are complementary, not redundant.


⚠️ [vale] <AgenticWriting.STE-Passive> reported by reviewdog 🐶
Passive voice: 'is deleted'. ASD-STE100 asks for the active voice.

For a SeiNode, whether its PVC survives deletion is governed by `spec.import` (imported PVC = preserved) vs controller-managed (wiped on teardown) — documented under **Phase: Failed** above. A `SeiNetwork`'s `spec.deletionPolicy` (defaults `Retain`) governs whether the controller orphans its generated validator SeiNodes (and thus their PVCs) when the network is deleted. Forensics is the one case where `Retain` is the right answer; on an ordinary teardown it is a disk leak, because the orphaned validators keep running with no owner left to delete them (see `teardown.md`). The hardlink trick above is for **live debugging** while the node continues running. The two are complementary, not redundant.


⚠️ [vale] <AgenticWriting.STE-SentenceLength-Description> reported by reviewdog 🐶
Descriptive sentence has 33 words. ASD-STE100 allows 25. Split it.

For a SeiNode, whether its PVC survives deletion is governed by `spec.import` (imported PVC = preserved) vs controller-managed (wiped on teardown) — documented under **Phase: Failed** above. A `SeiNetwork`'s `spec.deletionPolicy` (defaults `Retain`) governs whether the controller orphans its generated validator SeiNodes (and thus their PVCs) when the network is deleted. Forensics is the one case where `Retain` is the right answer; on an ordinary teardown it is a disk leak, because the orphaned validators keep running with no owner left to delete them (see `teardown.md`). The hardlink trick above is for **live debugging** while the node continues running. The two are complementary, not redundant.

- Defaults to GitOps: renders CRs via `--dry-run`, writes them under `engineers/<alias>/<task>/`, opens a PR, and lets Flux apply on merge — direct apply is a rare, double-confirmed escape hatch.
- Covers the full daily-driver surface: onboarding, chain spinup, RPC fleets, single and comparative benches, status reads, and `git rm`-based teardown.
- Covers the full daily-driver surface: onboarding, chain spinup, RPC fleets, single and comparative benches, status reads, and PR-based teardown.
- Gates teardown on the field that leaks disks. A `SeiNetwork` deleted under its default `spec.deletionPolicy: Retain` orphans the validator SeiNodes it generated — they keep running, keep their PVCs, and keep their EBS volumes, and nothing in git or in Flux will ever remove them. The skill reads the policy, patches it to `Delete` before the removal merges, verifies against the engineer's own Flux Kustomization rather than `flux-system`, and polls the resources to gone.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ [vale] <AgenticWriting.STE-SentenceLength-Description> reported by reviewdog 🐶
Descriptive sentence has 33 words. ASD-STE100 allows 25. Split it.

- Defaults to GitOps: renders CRs via `--dry-run`, writes them under `engineers/<alias>/<task>/`, opens a PR, and lets Flux apply on merge — direct apply is a rare, double-confirmed escape hatch.
- Covers the full daily-driver surface: onboarding, chain spinup, RPC fleets, single and comparative benches, status reads, and `git rm`-based teardown.
- Covers the full daily-driver surface: onboarding, chain spinup, RPC fleets, single and comparative benches, status reads, and PR-based teardown.
- Gates teardown on the field that leaks disks. A `SeiNetwork` deleted under its default `spec.deletionPolicy: Retain` orphans the validator SeiNodes it generated — they keep running, keep their PVCs, and keep their EBS volumes, and nothing in git or in Flux will ever remove them. The skill reads the policy, patches it to `Delete` before the removal merges, verifies against the engineer's own Flux Kustomization rather than `flux-system`, and polls the resources to gone.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ [vale] <AgenticWriting.STE-SentenceLength-Description> reported by reviewdog 🐶
Descriptive sentence has 28 words. ASD-STE100 allows 25. Split it.

| "What's running in my namespace" / "what chains do I have" | `seictl network list -n eng-<alias>` for the chains + `seictl node list -n eng-<alias>` for the followers (yaml default; `-o name` for short, `-o jsonpath=...` for one-shot field reads). |
| "Show me chain X" / "what's the status of X" | `seictl network get <name> -n eng-<alias>` for the network (`.status.phase`); `seictl node get <name>-rpc-<k> -n eng-<alias>` for a follower (`.status.phase`, `.status.endpoint`). |
| "Tear down chain X" | `git rm -r engineers/<alias>/<task>/` **and** remove `<task>` from `engineers/<alias>/kustomization.yaml`'s `resources:` list (Kustomize fails to render with a missing-resource entry). Commit → push → merge. Flux prunes the SeiNetwork + SeiNodes on next reconcile, which cascades to pods / PVCs per k8s deletion propagation. **Teardown does NOT purge the chain-id's S3 genesis artifacts — the chain-id is burned**; a later respin must use a fresh chain-id (see the naming step) or purge the `<chain-id>/` genesis-bucket prefix. See `bench:teardown` recipe in `references/cluster-inspection-recipes.md` for the bench-specific variant. |
| "Tear down chain X" / "delete my bench" | **PR-based** (see Procedure: tear down). Read `spec.deletionPolicy` on every SeiNetwork in the task dir **first** — on `Retain` (the default) the teardown orphans the validators and leaks their disks. Then `git rm -r engineers/<alias>/<task>/` **and** remove `<task>` from `engineers/<alias>/kustomization.yaml`'s `resources:` list (Kustomize fails to render with a missing-resource entry). Commit → push → merge → reconcile `<alias>` → poll until the objects are gone. **Teardown does NOT purge the chain-id's S3 genesis artifacts — the chain-id is burned**; a later respin must use a fresh chain-id (see the naming step) or purge the `<chain-id>/` genesis-bucket prefix. |

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ [vale] <AgenticWriting.STE-Passive> reported by reviewdog 🐶
Passive voice: 'are gone'. ASD-STE100 asks for the active voice.

| "What's running in my namespace" / "what chains do I have" | `seictl network list -n eng-<alias>` for the chains + `seictl node list -n eng-<alias>` for the followers (yaml default; `-o name` for short, `-o jsonpath=...` for one-shot field reads). |
| "Show me chain X" / "what's the status of X" | `seictl network get <name> -n eng-<alias>` for the network (`.status.phase`); `seictl node get <name>-rpc-<k> -n eng-<alias>` for a follower (`.status.phase`, `.status.endpoint`). |
| "Tear down chain X" | `git rm -r engineers/<alias>/<task>/` **and** remove `<task>` from `engineers/<alias>/kustomization.yaml`'s `resources:` list (Kustomize fails to render with a missing-resource entry). Commit → push → merge. Flux prunes the SeiNetwork + SeiNodes on next reconcile, which cascades to pods / PVCs per k8s deletion propagation. **Teardown does NOT purge the chain-id's S3 genesis artifacts — the chain-id is burned**; a later respin must use a fresh chain-id (see the naming step) or purge the `<chain-id>/` genesis-bucket prefix. See `bench:teardown` recipe in `references/cluster-inspection-recipes.md` for the bench-specific variant. |
| "Tear down chain X" / "delete my bench" | **PR-based** (see Procedure: tear down). Read `spec.deletionPolicy` on every SeiNetwork in the task dir **first** — on `Retain` (the default) the teardown orphans the validators and leaks their disks. Then `git rm -r engineers/<alias>/<task>/` **and** remove `<task>` from `engineers/<alias>/kustomization.yaml`'s `resources:` list (Kustomize fails to render with a missing-resource entry). Commit → push → merge → reconcile `<alias>` → poll until the objects are gone. **Teardown does NOT purge the chain-id's S3 genesis artifacts — the chain-id is burned**; a later respin must use a fresh chain-id (see the naming step) or purge the `<chain-id>/` genesis-bucket prefix. |

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ [vale] <AgenticWriting.STE-SentenceLength-Description> reported by reviewdog 🐶
Descriptive sentence has 35 words. ASD-STE100 allows 25. Split it.

| "What's running in my namespace" / "what chains do I have" | `seictl network list -n eng-<alias>` for the chains + `seictl node list -n eng-<alias>` for the followers (yaml default; `-o name` for short, `-o jsonpath=...` for one-shot field reads). |
| "Show me chain X" / "what's the status of X" | `seictl network get <name> -n eng-<alias>` for the network (`.status.phase`); `seictl node get <name>-rpc-<k> -n eng-<alias>` for a follower (`.status.phase`, `.status.endpoint`). |
| "Tear down chain X" | `git rm -r engineers/<alias>/<task>/` **and** remove `<task>` from `engineers/<alias>/kustomization.yaml`'s `resources:` list (Kustomize fails to render with a missing-resource entry). Commit → push → merge. Flux prunes the SeiNetwork + SeiNodes on next reconcile, which cascades to pods / PVCs per k8s deletion propagation. **Teardown does NOT purge the chain-id's S3 genesis artifacts — the chain-id is burned**; a later respin must use a fresh chain-id (see the naming step) or purge the `<chain-id>/` genesis-bucket prefix. See `bench:teardown` recipe in `references/cluster-inspection-recipes.md` for the bench-specific variant. |
| "Tear down chain X" / "delete my bench" | **PR-based** (see Procedure: tear down). Read `spec.deletionPolicy` on every SeiNetwork in the task dir **first** — on `Retain` (the default) the teardown orphans the validators and leaks their disks. Then `git rm -r engineers/<alias>/<task>/` **and** remove `<task>` from `engineers/<alias>/kustomization.yaml`'s `resources:` list (Kustomize fails to render with a missing-resource entry). Commit → push → merge → reconcile `<alias>` → poll until the objects are gone. **Teardown does NOT purge the chain-id's S3 genesis artifacts — the chain-id is burned**; a later respin must use a fresh chain-id (see the naming step) or purge the `<chain-id>/` genesis-bucket prefix. |

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ [vale] <AgenticWriting.STE-Passive> reported by reviewdog 🐶
Passive voice: 'is burned'. ASD-STE100 asks for the active voice.


### 9. Did the teardown actually remove the resources?

A Flux reconcile reports success once it issues the deletes. Deletion is asynchronous and finalizers hold objects in `Terminating` while the controller releases their PVCs, so poll rather than assert once.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📝 [vale] <AgenticWriting.STE-NounCluster> reported by reviewdog 🐶
Noun cluster of more than three nouns: 'Flux reconcile reports success'. Break it with 'of' or 'for'.

14. **Poll bench Jobs to terminal** — both `seiload-<COMPARE_RUN_ID>-a` and `-b` to `Complete` or `Failed`. Deadline `<DURATION> * 60 + 660` seconds.
15. **Fetch + render** — `aws s3 cp` both reports; extract metrics; render the side-by-side table. On any extraction gap, fall back to the raw-tail format with both S3 paths surfaced.
16. **Teardown guidance** — `git rm -r engineers/<alias>/compare-<COMPARE_RUN_ID>/` and remove the entry from `engineers/<alias>/kustomization.yaml` `resources:`. Flux prunes both SeiNetworks, all follower SeiNodes, and both Jobs; child pods/PVCs cascade.
16. **Teardown guidance** — run the procedure in `teardown.md` against `engineers/<alias>/compare-<COMPARE_RUN_ID>/`. **Both** SeiNetworks need the `deletionPolicy` gate before the removal PR opens: a comparison under the default `Retain` orphans two validator pools and leaks both sets of EBS disks. After that gate, `git rm -r` the dir, remove the entry from `engineers/<alias>/kustomization.yaml` `resources:`, merge, reconcile `kustomization <alias>` in `eng-<alias>`, and poll both chain-ids to gone.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ [vale] <AgenticWriting.STE-SentenceLength-Description> reported by reviewdog 🐶
Descriptive sentence has 26 words. ASD-STE100 allows 25. Split it.

```

**Why:** the post-merge reconcile pattern (`flux reconcile kustomization flux-system --with-source -n flux-system`) is the fast path from "PR merged" to "manifests applied in cluster." Without `flux`, the fallback is `kubectl annotate kustomization flux-system reconcile.fluxcd.io/requestedAt=$(date +%s) --overwrite -n flux-system`, which works but doesn't fetch the latest source revision in the same call.
**Why:** the post-merge reconcile pattern is the fast path from "PR merged" to "manifests applied in cluster." The target depends on which repo merged: a **workspace**-repo merge (every chain, bench, and teardown) goes to `flux reconcile kustomization <alias> -n eng-<alias> --with-source`, and only a **platform**-repo merge (onboarding) goes to `flux reconcile kustomization flux-system --with-source -n flux-system`. Reconciling `flux-system` for a workspace merge reconciles the platform repo and reports success without applying the engineer's change. Without `flux`, the fallback is `kubectl annotate kustomization <name> reconcile.fluxcd.io/requestedAt=$(date +%s) --overwrite -n <namespace>`, which works but does not fetch the latest source revision in the same call.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ [vale] <AgenticWriting.STE-SentenceLength-Description> reported by reviewdog 🐶
Descriptive sentence has 27 words. ASD-STE100 allows 25. Split it.

## Deletion — `deletionPolicy` is the disk-leak field

`spec.deletionPolicy` defaults to `Retain` — it governs whether the controller orphans its generated validator SeiNodes on delete. This is orthogonal to the client-side `--cascade` propagation policy on `seictl network delete`; both apply.
`spec.deletionPolicy` defaults to **`Retain`**. Under `Retain` the controller does not delete the generated validator SeiNodes on deletion — it **strips their owner reference** and leaves them running. Each orphan keeps its PVC and its EBS disk, garbage collection has no owner reference left to follow, and Flux prune never reaches them because the controller created them and Flux never held them in its inventory. The teardown looks clean and the spend continues.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ [vale] <AgenticWriting.STE-SentenceLength-Description> reported by reviewdog 🐶
Descriptive sentence has 37 words. ASD-STE100 allows 25. Split it.

`spec.deletionPolicy` defaults to `Retain` — it governs whether the controller orphans its generated validator SeiNodes on delete. This is orthogonal to the client-side `--cascade` propagation policy on `seictl network delete`; both apply.
`spec.deletionPolicy` defaults to **`Retain`**. Under `Retain` the controller does not delete the generated validator SeiNodes on deletion — it **strips their owner reference** and leaves them running. Each orphan keeps its PVC and its EBS disk, garbage collection has no owner reference left to follow, and Flux prune never reaches them because the controller created them and Flux never held them in its inventory. The teardown looks clean and the spend continues.

The storage class is not the lever. A `Delete` reclaim policy releases a disk only when the PVC is deleted, and an orphaned SeiNode never releases its PVC.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ [vale] <AgenticWriting.STE-Passive> reported by reviewdog 🐶
Passive voice: 'is deleted'. ASD-STE100 asks for the active voice.

…ore removal

A kubectl patch against a Flux-owned SeiNetwork is drift. The rendered
manifest normally carries the server-defaulted deletionPolicy: Retain,
so Flux owns the field and reverts the patch on its next reconcile —
typically while the removal PR is still in review. The engineer then
merges a teardown they believe is safe and it orphans the validators.

Document a policy-PR-then-removal-PR ordering with a reconcile and a
read-back of both the live object and the committed file. The live
patch stays only for a SeiNetwork no reconcile owns, and carries the
re-verify-immediately-before-merge requirement.

Also drop spec.resources from the immutable-field list: SeiNetworkSpec
carries exactly three immutability rules — genesis, replicas,
dataVolume — and has no resources field.

Co-authored-by: omnigent <noreply@omnigent.ai>

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remaining comments which cannot be posted as a review comment to avoid GitHub Rate Limit

vale

⚠️ [vale] <AgenticWriting.STE-Passive> reported by reviewdog 🐶
Passive voice: 'is ********'. ASD-STE100 asks for the active voice.

- **`kustomization <alias>` is `NotFound` in `eng-<alias>`.** The engineer's Flux wiring is missing, so no workspace-repo merge reconciles at all. Surface to the platform team; do not create the Kustomization.


⚠️ [vale] <AgenticWriting.STE-SentenceLength-Description> reported by reviewdog 🐶
Descriptive sentence has 27 words. ASD-STE100 allows 25. Split it.

- **`lastAppliedRevision` does not reach the merge commit within two reconcile intervals (~10 min).** Read the Ready condition's message (`cluster-inspection-recipes.md` recipe #8). A render error in `engineers/<alias>/kustomization.yaml` — most often a `resources:` entry pointing at the dir that was just removed — blocks every later apply in the namespace, not only this teardown.


⚠️ [vale] <AgenticWriting.STE-SentenceLength-Description> reported by reviewdog 🐶
Descriptive sentence has 27 words. ASD-STE100 allows 25. Split it.

This abandons the PVC and its EBS disk — the finalizer is what deletes the PVC, so removing it is how a stuck teardown becomes a leaked disk. Take it only with the engineer's explicit acceptance, record the PVC name, and follow up with the sweep in `teardown.md`*find and clean up already-leaked resources*.


⚠️ [vale] <AgenticWriting.STE-Passive> reported by reviewdog 🐶
Passive voice: 'is governed'. ASD-STE100 asks for the active voice.

For a SeiNode, whether its PVC survives deletion is governed by `spec.import` (imported PVC = preserved) vs controller-managed (wiped on teardown) — documented under **Phase: Failed** above. A `SeiNetwork`'s `spec.deletionPolicy` (defaults `Retain`) governs whether the controller orphans its generated validator SeiNodes (and thus their PVCs) when the network is deleted. Forensics is the one case where `Retain` is the right answer; on an ordinary teardown it is a disk leak, because the orphaned validators keep running with no owner left to delete them (see `teardown.md`). The hardlink trick above is for **live debugging** while the node continues running. The two are complementary, not redundant.


⚠️ [vale] <AgenticWriting.STE-Passive> reported by reviewdog 🐶
Passive voice: 'is deleted'. ASD-STE100 asks for the active voice.

For a SeiNode, whether its PVC survives deletion is governed by `spec.import` (imported PVC = preserved) vs controller-managed (wiped on teardown) — documented under **Phase: Failed** above. A `SeiNetwork`'s `spec.deletionPolicy` (defaults `Retain`) governs whether the controller orphans its generated validator SeiNodes (and thus their PVCs) when the network is deleted. Forensics is the one case where `Retain` is the right answer; on an ordinary teardown it is a disk leak, because the orphaned validators keep running with no owner left to delete them (see `teardown.md`). The hardlink trick above is for **live debugging** while the node continues running. The two are complementary, not redundant.


⚠️ [vale] <AgenticWriting.STE-SentenceLength-Description> reported by reviewdog 🐶
Descriptive sentence has 33 words. ASD-STE100 allows 25. Split it.

For a SeiNode, whether its PVC survives deletion is governed by `spec.import` (imported PVC = preserved) vs controller-managed (wiped on teardown) — documented under **Phase: Failed** above. A `SeiNetwork`'s `spec.deletionPolicy` (defaults `Retain`) governs whether the controller orphans its generated validator SeiNodes (and thus their PVCs) when the network is deleted. Forensics is the one case where `Retain` is the right answer; on an ordinary teardown it is a disk leak, because the orphaned validators keep running with no owner left to delete them (see `teardown.md`). The hardlink trick above is for **live debugging** while the node continues running. The two are complementary, not redundant.


Under `Delete` the chain runs end to end: SeiNetwork deleted → validators deleted through their owner references → each SeiNode's finalizer deletes its data PVC → the storage class's `Delete` reclaim policy releases the EBS volume. The finalizer skips an **imported** PVC (`spec.import` on the SeiNode) by design.

Keep `Retain` only to preserve a validator's disk for forensics after the network goes away, and say so where the choice is made — a retained disk is a cost somebody chose.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ [vale] <AgenticWriting.STE-SentenceLength-Description> reported by reviewdog 🐶
Descriptive sentence has 31 words. ASD-STE100 allows 25. Split it.


Under `Delete` the chain runs end to end: SeiNetwork deleted → validators deleted through their owner references → each SeiNode's finalizer deletes its data PVC → the storage class's `Delete` reclaim policy releases the EBS volume. The finalizer skips an **imported** PVC (`spec.import` on the SeiNode) by design.

Keep `Retain` only to preserve a validator's disk for forensics after the network goes away, and say so where the choice is made — a retained disk is a cost somebody chose.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ [vale] <AgenticWriting.STE-Passive> reported by reviewdog 🐶
Passive voice: 'is made'. ASD-STE100 asks for the active voice.

@@ -0,0 +1,290 @@
# Teardown (chain, bench, namespace)

Teardown removes an engineer's workloads from `eng-<alias>` through the same PR contract that created them. One ordering rule governs the whole file: **patch `spec.deletionPolicy` to `Delete` on every SeiNetwork you are about to remove, and land that patch before the removal merges.** A SeiNetwork deleted under the default `Retain` orphans its generated validator SeiNodes, and each orphan keeps its PVC and its EBS disk running with nothing left to clean it up.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ [vale] <AgenticWriting.STE-SentenceLength-Description> reported by reviewdog 🐶
Descriptive sentence has 29 words. ASD-STE100 allows 25. Split it.


`spec.deletionPolicy` is **mutable** — no CEL validation rule and no webhook makes it immutable. `SeiNetworkSpec` carries exactly three immutability rules, on `spec.genesis`, `spec.replicas`, and `spec.dataVolume`. An operator can therefore move a SeiNetwork from `Retain` to `Delete`.

That window closes at deletion. Once a `Retain` deletion has stripped the owner references and removed the parent SeiNetwork, no patch brings the cascade back — the parent is gone and the children are top-level objects. The leftover SeiNodes and PVCs then need the manual cleanup in [Find and clean up already-leaked resources](#find-and-clean-up-already-leaked-resources).

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ [vale] <AgenticWriting.STE-SentenceLength-Description> reported by reviewdog 🐶
Descriptive sentence has 30 words. ASD-STE100 allows 25. Split it.


`spec.deletionPolicy` is **mutable** — no CEL validation rule and no webhook makes it immutable. `SeiNetworkSpec` carries exactly three immutability rules, on `spec.genesis`, `spec.replicas`, and `spec.dataVolume`. An operator can therefore move a SeiNetwork from `Retain` to `Delete`.

That window closes at deletion. Once a `Retain` deletion has stripped the owner references and removed the parent SeiNetwork, no patch brings the cascade back — the parent is gone and the children are top-level objects. The leftover SeiNodes and PVCs then need the manual cleanup in [Find and clean up already-leaked resources](#find-and-clean-up-already-leaked-resources).

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ [vale] <AgenticWriting.STE-Passive> reported by reviewdog 🐶
Passive voice: 'is gone'. ASD-STE100 asks for the active voice.

The disappearance poll piped kubectl into wc -l, so a Forbidden or a
dropped connection yielded zero lines and printed success — the worst
possible failure for a verifier, reporting the disks gone exactly when
it cannot see them. Capture kubectl's exit status separately and print
one of GONE / PRESENT / UNVERIFIED.

The loop also used Bash's SECONDS inside an sh fence, where it is unset
and the comparison dies with `Illegal number`, skipping the poll
outright. Use arithmetic on `date +%s`.

Poll the PVCs rather than reading them once, and expect the imported
ones to survive: the SeiNode finalizer skips spec.import by design, so
zero PVCs is the wrong end state. Inventory now records which nodes
carry spec.import, since nothing says so once they are deleted.

Co-authored-by: omnigent <noreply@omnigent.ai>

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remaining comments which cannot be posted as a review comment to avoid GitHub Rate Limit

vale

⚠️ [vale] <AgenticWriting.STE-SentenceLength-Description> reported by reviewdog 🐶
Descriptive sentence has 33 words. ASD-STE100 allows 25. Split it.

For a SeiNode, whether its PVC survives deletion is governed by `spec.import` (imported PVC = preserved) vs controller-managed (wiped on teardown) — documented under **Phase: Failed** above. A `SeiNetwork`'s `spec.deletionPolicy` (defaults `Retain`) governs whether the controller orphans its generated validator SeiNodes (and thus their PVCs) when the network is deleted. Forensics is the one case where `Retain` is the right answer; on an ordinary teardown it is a disk leak, because the orphaned validators keep running with no owner left to delete them (see `teardown.md`). The hardlink trick above is for **live debugging** while the node continues running. The two are complementary, not redundant.

kubectl patch seinode <name> -p '{"metadata":{"finalizers":[]}}' --type=merge
```

This abandons the PVC and its EBS disk — the finalizer is what deletes the PVC, so removing it is how a stuck teardown becomes a leaked disk. Take it only with the engineer's explicit acceptance, record the PVC name, and follow up with the sweep in `teardown.md` → *find and clean up already-leaked resources*.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ [vale] <AgenticWriting.STE-SentenceLength-Description> reported by reviewdog 🐶
Descriptive sentence has 27 words. ASD-STE100 allows 25. Split it.

bdchatham and others added 2 commits September 9, 2026 21:54
… ownership walk

EC2 `available` means unattached, not unowned — a volume backing a live
Bound PVC reads available the moment its workload stops. `Used By` is
current attachment, not ownership. As written the sweep routed
legitimate disks into a deletion escalation framed as confirmed
garbage.

Both signals are now candidates. Ownership resolves by walking volume
ID to PV via spec.csi.volumeHandle, PV to PVC via claimRef, then PVC to
its workload, with a verdict table that sends any failed hop to
UNRESOLVED rather than to confirmed-safe. The tag-confirmation command
takes --volume-ids instead of dropping --filters, which would have
enumerated other tenants' volumes.

Also: no EBS delete from this skill at all; an escalation path for
every non-Git resource a workspace PR leaves behind; and the namespace
ordering is stated as a preference rather than asserting a cascade
failure mode the namespace controller does not have.

Co-authored-by: omnigent <noreply@omnigent.ai>
…on git-first

Recipe #9 becomes a reusable poll_gone with the exit-status handling and
the portable deadline; bench:teardown polls Jobs, ConfigMaps AND pods
instead of reading once, since a Job can be gone while its pod is still
Terminating.

SKILL.md, the SeiNetwork CRD reference, chain-flow, comparative-bench,
and the two evals now say the policy lands in git rather than through a
live patch, and the teardown step reports GONE / PRESENT / UNVERIFIED.

Co-authored-by: omnigent <noreply@omnigent.ai>
9. **Two paths wipe a node's chain data — gate both.** `seictl workflow state-sync` is the destructive **paved road**; a mutating `seictl task submit` is the destructive **escape hatch**. Neither is ever the default, and the agent volunteers neither.
- **`seictl workflow state-sync`** re-bootstraps an existing node by wiping its local chain state (an `rm -rf` on that node's data), optionally with an irreversible `--migration GigaStore --backend <pebbledb|rocksdb>` store change — both tokens are required together, never `--migration` alone. Require explicit engineer sign-off before the non-dry-run apply, verify the target node against the live cluster first, `--dry-run` to inspect, and escalate to the owner — never wipe on agent initiative — for any shared or long-lived `pacific-1`/`atlantic-2` follower. Never commit a workflow CR to the Flux workspace repo (a one-shot, spec-immutable request object; force-delete recovery fights Flux). Full gate in `references/seictl-cli.md` → `seictl workflow state-sync`.
- **`seictl task submit`** POSTs a raw task straight to one pod's sidecar, and the accepted types include `reset-data`. Submitted that way the wipe runs with **none** of the recipe's protections — no `mark-not-ready` hold, no `stop-seid` first, no ordering, and no adoption pointer telling the controller the node is occupied — so it is strictly more dangerous than the paved road, not a lighter-weight version of it. Prefer `workflow state-sync` for anything the recipe covers; a mutating `task submit` requires explicit sign-off naming node, namespace, and task type. `task get` / `task list` are reads and safe. Full gate in `references/seictl-cli.md` → `seictl task`.
10. **Never tear down a SeiNetwork before reading its `spec.deletionPolicy`.** It defaults to `Retain`, and a `Retain` deletion strips the owner reference from every generated validator SeiNode instead of deleting it. The orphans keep running, keep their PVCs, and keep their EBS disks, with nothing left in-cluster that will ever remove them — the teardown reports success while the spend continues. Land `deletionPolicy: Delete` **in git** and confirm it reconciled onto the live object **before** the removal merges — a bare `kubectl patch` on a Flux-owned SeiNetwork is drift the next reconcile reverts, typically while the removal PR is still in review. After the parent SeiNetwork is gone, nothing restores the cascade and the cleanup is manual. Full procedure and the leaked-resource sweep: `references/teardown.md`.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ [vale] <AgenticWriting.STE-SentenceLength-Description> reported by reviewdog 🐶
Descriptive sentence has 30 words. ASD-STE100 allows 25. Split it.

9. **Two paths wipe a node's chain data — gate both.** `seictl workflow state-sync` is the destructive **paved road**; a mutating `seictl task submit` is the destructive **escape hatch**. Neither is ever the default, and the agent volunteers neither.
- **`seictl workflow state-sync`** re-bootstraps an existing node by wiping its local chain state (an `rm -rf` on that node's data), optionally with an irreversible `--migration GigaStore --backend <pebbledb|rocksdb>` store change — both tokens are required together, never `--migration` alone. Require explicit engineer sign-off before the non-dry-run apply, verify the target node against the live cluster first, `--dry-run` to inspect, and escalate to the owner — never wipe on agent initiative — for any shared or long-lived `pacific-1`/`atlantic-2` follower. Never commit a workflow CR to the Flux workspace repo (a one-shot, spec-immutable request object; force-delete recovery fights Flux). Full gate in `references/seictl-cli.md` → `seictl workflow state-sync`.
- **`seictl task submit`** POSTs a raw task straight to one pod's sidecar, and the accepted types include `reset-data`. Submitted that way the wipe runs with **none** of the recipe's protections — no `mark-not-ready` hold, no `stop-seid` first, no ordering, and no adoption pointer telling the controller the node is occupied — so it is strictly more dangerous than the paved road, not a lighter-weight version of it. Prefer `workflow state-sync` for anything the recipe covers; a mutating `task submit` requires explicit sign-off naming node, namespace, and task type. `task get` / `task list` are reads and safe. Full gate in `references/seictl-cli.md` → `seictl task`.
10. **Never tear down a SeiNetwork before reading its `spec.deletionPolicy`.** It defaults to `Retain`, and a `Retain` deletion strips the owner reference from every generated validator SeiNode instead of deleting it. The orphans keep running, keep their PVCs, and keep their EBS disks, with nothing left in-cluster that will ever remove them — the teardown reports success while the spend continues. Land `deletionPolicy: Delete` **in git** and confirm it reconciled onto the live object **before** the removal merges — a bare `kubectl patch` on a Flux-owned SeiNetwork is drift the next reconcile reverts, typically while the removal PR is still in review. After the parent SeiNetwork is gone, nothing restores the cascade and the cleanup is manual. Full procedure and the leaked-resource sweep: `references/teardown.md`.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ [vale] <AgenticWriting.STE-SentenceLength-Description> reported by reviewdog 🐶
Descriptive sentence has 37 words. ASD-STE100 allows 25. Split it.

9. **Two paths wipe a node's chain data — gate both.** `seictl workflow state-sync` is the destructive **paved road**; a mutating `seictl task submit` is the destructive **escape hatch**. Neither is ever the default, and the agent volunteers neither.
- **`seictl workflow state-sync`** re-bootstraps an existing node by wiping its local chain state (an `rm -rf` on that node's data), optionally with an irreversible `--migration GigaStore --backend <pebbledb|rocksdb>` store change — both tokens are required together, never `--migration` alone. Require explicit engineer sign-off before the non-dry-run apply, verify the target node against the live cluster first, `--dry-run` to inspect, and escalate to the owner — never wipe on agent initiative — for any shared or long-lived `pacific-1`/`atlantic-2` follower. Never commit a workflow CR to the Flux workspace repo (a one-shot, spec-immutable request object; force-delete recovery fights Flux). Full gate in `references/seictl-cli.md` → `seictl workflow state-sync`.
- **`seictl task submit`** POSTs a raw task straight to one pod's sidecar, and the accepted types include `reset-data`. Submitted that way the wipe runs with **none** of the recipe's protections — no `mark-not-ready` hold, no `stop-seid` first, no ordering, and no adoption pointer telling the controller the node is occupied — so it is strictly more dangerous than the paved road, not a lighter-weight version of it. Prefer `workflow state-sync` for anything the recipe covers; a mutating `task submit` requires explicit sign-off naming node, namespace, and task type. `task get` / `task list` are reads and safe. Full gate in `references/seictl-cli.md` → `seictl task`.
10. **Never tear down a SeiNetwork before reading its `spec.deletionPolicy`.** It defaults to `Retain`, and a `Retain` deletion strips the owner reference from every generated validator SeiNode instead of deleting it. The orphans keep running, keep their PVCs, and keep their EBS disks, with nothing left in-cluster that will ever remove them — the teardown reports success while the spend continues. Land `deletionPolicy: Delete` **in git** and confirm it reconciled onto the live object **before** the removal merges — a bare `kubectl patch` on a Flux-owned SeiNetwork is drift the next reconcile reverts, typically while the removal PR is still in review. After the parent SeiNetwork is gone, nothing restores the cascade and the cleanup is manual. Full procedure and the leaked-resource sweep: `references/teardown.md`.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ [vale] <AgenticWriting.STE-Passive> reported by reviewdog 🐶
Passive voice: 'is gone'. ASD-STE100 asks for the active voice.

1. **Pre-flight** — five gates. Halt on first failure.
2. **Inventory what goes away** — `kubectl get seinetwork,seinode -n eng-<alias> -l sei.io/seinetwork=<chain-id>` plus `kubectl get pvc -n eng-<alias>`. Show the engineer the list before touching anything.
3. **Gate on `deletionPolicy`** — `kubectl get seinetwork <chain-id> -n eng-<alias> -o jsonpath='{.spec.deletionPolicy}'` for every SeiNetwork in the task dir. `Retain` or empty means halt: removing the manifest orphans the generated validators and leaks their EBS disks (Guardrail #10).
4. **Set it to `Delete` in git, then confirm both** — a policy PR that sets `deletionPolicy: Delete` on the SeiNetwork manifest, merged and reconciled, before the removal PR merges. Both the committed file and the live object must read `Delete`. **A bare `kubectl patch` is not enough on a Flux-owned SeiNetwork**: git still declares `Retain`, so the next reconcile reverts the patch, often mid-review, and the removal then merges under `Retain` anyway. The patch is a repair for a SeiNetwork no reconcile owns. Full ordering in `references/teardown.md`.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ [vale] <AgenticWriting.STE-SentenceLength-Description> reported by reviewdog 🐶
Descriptive sentence has 30 words. ASD-STE100 allows 25. Split it.

7. **Open the PR** — title `feat(eng/<alias>): tear down <task>`. Body names the chain-id, every CR that goes away, and the `deletionPolicy` value the SeiNetwork now carries. Surface the URL and halt for the merge.
8. **After merge — reconcile the workspace Kustomization** — `flux --context harbor reconcile kustomization <alias> -n eng-<alias> --with-source`, then compare `.status.lastAppliedRevision` to the merge SHA. Reconciling `flux-system` here verifies the wrong repo (see Post-merge reconciliation).
9. **Poll until the resources disappear, and distinguish three outcomes** — a reconcile only says Flux issued the deletes. Deletion is asynchronous and finalizers hold objects in `Terminating` while the controller releases PVCs, so poll on a budget (~5 min) rather than asserting once. Report exactly one of **`GONE`** (the API answered and matched nothing), **`PRESENT`** (objects remain at the deadline), or **`UNVERIFIED`** (the API call failed). **A failed read is never a pass** — a `Forbidden` or dropped connection returns zero lines, so a check that counts lines without reading `kubectl`'s exit status claims success precisely when it cannot see the cluster. Use `poll_gone` from `references/cluster-inspection-recipes.md` recipe #9; it captures the exit status separately and uses a `date +%s` deadline (Bash's `SECONDS` is unset under `sh`, where the loop silently never runs). Poll the PVCs too — but expect the **imported** ones to survive, since the SeiNode finalizer skips `spec.import` by design.
10. **Report** — what is gone, what remains, and the burned chain-id. An object still `Terminating` past the budget is a real finding: surface the finalizer and the controller log line; never strip a finalizer to make the check pass. An `UNVERIFIED` result is reported as unknown, never as done.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ [vale] <AgenticWriting.STE-Passive> reported by reviewdog 🐶
Passive voice: 'is gone'. ASD-STE100 asks for the active voice.

- `.status.endpoint.tendermintRest` — Tendermint REST URL
- For pod-targeted connectivity (seiload's WebSocket block collector, etc.), pick one follower — its `.status.endpoint` is already its stable per-node URL.
14. **Report teardown** — `git rm -r engineers/<alias>/<task>/` **and** remove the `<task>` entry from `engineers/<alias>/kustomization.yaml`'s `resources:` list (Kustomize fails to render with an orphan reference). Commit → push → merge. Flux prunes the SeiNetwork + SeiNodes on next reconcile, cascading to pods/PVCs per k8s deletion propagation.
14. **Report teardown** — point at `teardown.md`; do not restate it. The load-bearing step comes *before* the `git rm`: read `spec.deletionPolicy` on the SeiNetwork and, if it reads `Retain` (the default), land `Delete` **in the manifest** and let it reconcile first — a live `kubectl patch` is drift Flux reverts on its next pass. A `Retain` teardown orphans the generated validator SeiNodes and leaks their EBS disks, and nothing after the fact undoes that. Then `git rm -r engineers/<alias>/<task>/`, remove the `<task>` entry from `engineers/<alias>/kustomization.yaml`'s `resources:` list (Kustomize fails to render with an orphan reference), merge, reconcile `kustomization <alias>` in `eng-<alias>`, and poll until the CRs and their PVCs are gone.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ [vale] <AgenticWriting.STE-SentenceLength-Description> reported by reviewdog 🐶
Descriptive sentence has 28 words. ASD-STE100 allows 25. Split it.

- `.status.endpoint.tendermintRest` — Tendermint REST URL
- For pod-targeted connectivity (seiload's WebSocket block collector, etc.), pick one follower — its `.status.endpoint` is already its stable per-node URL.
14. **Report teardown** — `git rm -r engineers/<alias>/<task>/` **and** remove the `<task>` entry from `engineers/<alias>/kustomization.yaml`'s `resources:` list (Kustomize fails to render with an orphan reference). Commit → push → merge. Flux prunes the SeiNetwork + SeiNodes on next reconcile, cascading to pods/PVCs per k8s deletion propagation.
14. **Report teardown** — point at `teardown.md`; do not restate it. The load-bearing step comes *before* the `git rm`: read `spec.deletionPolicy` on the SeiNetwork and, if it reads `Retain` (the default), land `Delete` **in the manifest** and let it reconcile first — a live `kubectl patch` is drift Flux reverts on its next pass. A `Retain` teardown orphans the generated validator SeiNodes and leaks their EBS disks, and nothing after the fact undoes that. Then `git rm -r engineers/<alias>/<task>/`, remove the `<task>` entry from `engineers/<alias>/kustomization.yaml`'s `resources:` list (Kustomize fails to render with an orphan reference), merge, reconcile `kustomization <alias>` in `eng-<alias>`, and poll until the CRs and their PVCs are gone.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ [vale] <AgenticWriting.STE-Passive> reported by reviewdog 🐶
Passive voice: 'are gone'. ASD-STE100 asks for the active voice.

kubectl get seinetwork <id> -n eng-<alias> -o jsonpath='{.spec.deletionPolicy}' # empty means Retain
```

**For a Flux-owned SeiNetwork the change goes through git, not `kubectl patch`.** The manifest rendered at spin-up normally carries the server-defaulted `Retain`, so Flux owns the field and reverts a live patch on its next reconcile — often while a removal PR is still in review, which is exactly when the revert does the damage. Set `deletionPolicy: Delete` in `engineers/<alias>/<task>/seinetwork-<id>.yaml`, merge, reconcile, and read back both git and the live object. `kubectl patch seinetwork <id> -n eng-<alias> --type=merge -p '{"spec":{"deletionPolicy":"Delete"}}'` is correct only where no reconcile owns the object.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ [vale] <AgenticWriting.STE-SentenceLength-Description> reported by reviewdog 🐶
Descriptive sentence has 43 words. ASD-STE100 allows 25. Split it.


### Set it to `Delete`

**The change must land in git. A `kubectl patch` alone does not survive to merge time.** Flux reconciles `engineers/<alias>/` every 5 minutes against what git declares. The manifest that spun the chain up was rendered from `seictl network apply --dry-run`, which captures the server-defaulted CR, so `deletionPolicy: Retain` is normally written out in the committed file. Flux owns that field, and the next reconcile reverts the patch — typically while the removal PR sits in review. The engineer then merges a teardown they believe is safe, and it orphans the validators anyway. Do not rely on server-side-apply field ownership to keep a patch alive across a reconcile, even where git happens to omit the field.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📝 [vale] <AgenticWriting.STE-ParagraphLength> reported by reviewdog 🐶
Paragraph has more than 6 sentences. ASD-STE100 allows 6.


### Set it to `Delete`

**The change must land in git. A `kubectl patch` alone does not survive to merge time.** Flux reconciles `engineers/<alias>/` every 5 minutes against what git declares. The manifest that spun the chain up was rendered from `seictl network apply --dry-run`, which captures the server-defaulted CR, so `deletionPolicy: Retain` is normally written out in the committed file. Flux owns that field, and the next reconcile reverts the patch — typically while the removal PR sits in review. The engineer then merges a teardown they believe is safe, and it orphans the validators anyway. Do not rely on server-side-apply field ownership to keep a patch alive across a reconcile, even where git happens to omit the field.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ [vale] <AgenticWriting.STE-Passive> reported by reviewdog 🐶
Passive voice: 'was rendered'. ASD-STE100 asks for the active voice.

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

Comment thread .claude/skills/harbor-dev/references/teardown.md Outdated
Comment thread .claude/skills/harbor-dev/references/cluster-inspection-recipes.md Outdated
bdchatham and others added 2 commits September 9, 2026 22:15
…be overwritten

poll_gone returned the right code but no caller checked it, so an
UNVERIFIED first call followed by a clean last call left the block
exiting 0 — the original "reports gone when it could not look" bug
surviving on the exit-code path. Add a record/VERDICT aggregator where
the worst outcome wins, and make every call site use it.

Poll PVCs by name rather than by namespace sweep: a sweep matches
imported claims and other chains' claims, so a correct teardown reported
PRESENT. Explicit names need --ignore-not-found, or the success
condition (NotFound) reads as UNVERIFIED. Add expect_present as the
mirror assertion for imported claims the controller preserves by design.

Recipe #9 is now the single implementation, and the POSIX claim is
softened: date +%s is a near-universal extension, not a specified format.

Verified under dash: UNVERIFIED-then-GONE aggregates to 2; expect_present
returns 0 on all-present and 1 on a missing claim.

Co-authored-by: omnigent <noreply@omnigent.ai>
…VC inventory path

teardown.md had re-implemented the poll twice instead of calling
poll_gone, and the copies had already diverged: the PVC loop emitted no
GONE/PRESENT/UNVERIFIED verdict at all and just spun to its deadline
printing counts. Both loops also ended in `break`, which succeeds, so
the completed while exited 0 even after printing UNVERIFIED. Both call
sites now use the shared functions and aggregate through record.

The imported-PVC query selected .spec.import, which does not exist —
the field is .spec.dataVolume.import.pvcName, so the query matched
nothing and every imported claim was silently reclassified as one that
must disappear. The inventory now records real node-to-claim identities
from the pods' own claimName, classifies imported against
controller-managed, and aborts on any API or parse failure. The path is
caveated as repo-main API inspection, not the deployed CRD.

Close the live-patch exception for Flux-owned networks: a pre-merge
re-read narrows the window but does not order against Flux's reconcile,
and the losing sequence needs no unusual timing. Path B is now a stacked
draft PR, since a merged PR cannot carry a later removal commit.

Co-authored-by: omnigent <noreply@omnigent.ai>

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remaining comments which cannot be posted as a review comment to avoid GitHub Rate Limit

vale

⚠️ [vale] <AgenticWriting.STE-Passive> reported by reviewdog 🐶
Passive voice: 'are gone'. ASD-STE100 asks for the active voice.

A successful reconcile says Flux applied the change. It does not say the objects are gone. Deletion is asynchronous and finalizers hold objects in `Terminating` while the controller releases their PVCs, so poll instead of asserting once.


⚠️ [vale] <AgenticWriting.STE-SentenceLength-Description> reported by reviewdog 🐶
Descriptive sentence has 28 words. ASD-STE100 allows 25. Split it.

**A failed API read is never a pass.** A `Forbidden` or a dropped connection returns zero lines, and a check that counts lines without reading the exit status prints "gone" precisely when it cannot see the cluster.


⚠️ [vale] <AgenticWriting.STE-SentenceLength-Description> reported by reviewdog 🐶
Descriptive sentence has 27 words. ASD-STE100 allows 25. Split it.

**And a later success must never overwrite an earlier failure.** Printing `UNVERIFIED` is not enough on its own: a `break` out of a loop, or a bare call whose return code nobody reads, still leaves the block exiting 0. A human sees the warning; a wrapper script or an agent reading `$?` sees success. Every check records its outcome into a running verdict, and the worst one wins.


⚠️ [vale] <AgenticWriting.STE-Passive> reported by reviewdog 🐶
Passive voice: 'are held'. ASD-STE100 asks for the active voice.

If the poll runs out with objects still present, do not report the teardown as done and do not force the objects away. Read why they are held:


⚠️ [vale] <write-good.Weasel> reported by reviewdog 🐶
'very' is a weasel word!

"Destroy my namespace" means one of two very different things. Ask which before acting.


⚠️ [vale] <AgenticWriting.STE-SentenceLength-Description> reported by reviewdog 🐶
Descriptive sentence has 26 words. ASD-STE100 allows 25. Split it.

**Empty the namespace first, through the steps above.** This is an operational preference, not a claim about a failure mode: the Kubernetes namespace controller does remove namespaced resources on its own. Emptying first keeps the `deletionPolicy` gate, the disappearance poll, and the leak sweep available while the objects are still there to inspect. Once the namespace is going away, a `Retain` SeiNetwork's orphans are much harder to reason about, and there is no inventory left to check them against.


⚠️ [vale] <AgenticWriting.STE-Passive> reported by reviewdog 🐶
Passive voice: 'is gone'. ASD-STE100 asks for the active voice.

Each line is a validator still running with nothing that will ever delete it. Confirm the parent is gone before treating one as an orphan:


⚠️ [vale] <AgenticWriting.STE-SentenceLength-Description> reported by reviewdog 🐶
Descriptive sentence has 28 words. ASD-STE100 allows 25. Split it.

- **EC2 `state: available` does not mean unowned.** It means unattached. A volume backing a live PV whose PVC is `Bound` reads `available` the moment its workload stops — a scaled-to-zero StatefulSet, a pod stuck `Pending`, a node drained mid-reschedule. Deleting on that signal destroys a disk somebody is coming back to.


⚠️ [vale] <AgenticWriting.STE-Passive> reported by reviewdog 🐶
Passive voice: 'is unwanted'. ASD-STE100 asks for the active voice.

- **`Used By: <pod>` does not prove the pod belongs to an orphan**, and `Used By: <none>` does not prove the PVC is unwanted. `describe pvc` reports current pod attachment, not ownership.


⚠️ [vale] <AgenticWriting.STE-SentenceLength-Description> reported by reviewdog 🐶
Descriptive sentence has 34 words. ASD-STE100 allows 25. Split it.

Those tag keys are the EBS CSI driver's own convention rather than something this skill's repos set. To confirm they are present before trusting an empty result, describe **one volume you already know is live, by ID** — never re-run without `--filters`, which enumerates every volume in the account including other tenants':


⚠️ [vale] <AgenticWriting.STE-Passive> reported by reviewdog 🐶
Passive voice: 'is namespaced'. ASD-STE100 asks for the active voice.

`kubectl get pv` is cluster-scoped, and the per-engineer Role is namespaced. Expect `Forbidden` here as the normal case for an engineer — that is an **unresolved** result, not a clean one. Hand the volume IDs to the platform team and let them walk the chain.


⚠️ [vale] <AgenticWriting.STE-Passive> reported by reviewdog 🐶
Passive voice: 'was stopped'. ASD-STE100 asks for the active voice.

| Volume → PV → PVC → a live, wanted workload | **Not garbage.** Leave it. `available` only meant the workload was stopped. |


⚠️ [vale] <AgenticWriting.STE-Passive> reported by reviewdog 🐶
Passive voice: 'is gone'. ASD-STE100 asks for the active voice.

| Volume → PV → PVC whose claim is gone, PV `Released` | Candidate for platform-team deletion. Report the PV, PVC name, and reclaim policy. |


⚠️ [vale] <AgenticWriting.STE-Passive> reported by reviewdog 🐶
Passive voice: 'is ******'. ASD-STE100 asks for the active voice.

The node's finalizer deletes its data PVC, and the `Delete` reclaim policy on `gp3-10k-750` (validators) and `gp3` (default) releases the EBS volume. `gp3-archive` is `Retain` by design — a volume on that class stays after its PVC goes, and its removal is an AWS-side decision, not a mistake to correct here.


⚠️ [vale] <AgenticWriting.STE-SentenceLength-Description> reported by reviewdog 🐶
Descriptive sentence has 27 words. ASD-STE100 allows 25. Split it.

The node's finalizer deletes its data PVC, and the `Delete` reclaim policy on `gp3-10k-750` (validators) and `gp3` (default) releases the EBS volume. `gp3-archive` is `Retain` by design — a volume on that class stays after its PVC goes, and its removal is an AWS-side decision, not a mistake to correct here.


⚠️ [vale] <AgenticWriting.STE-SentenceLength-Description> reported by reviewdog 🐶
Descriptive sentence has 33 words. ASD-STE100 allows 25. Split it.

**Never delete an EBS volume from this skill.** Even a volume the ownership walk resolved to a dead PVC goes to the platform team: `ec2:DeleteVolume` is outside the engineer's policy, the walk can be wrong, and an EBS delete is unrecoverable. Hand over the volume IDs, sizes, creation times, and the walk's verdict per volume — including every `UNRESOLVED` one, labelled as unresolved. Escalate through `#harbor-onboarding`. Do not report the cleanup as complete while any ID is outstanding.


⚠️ [vale] <AgenticWriting.STE-SentenceLength-Description> reported by reviewdog 🐶
Descriptive sentence has 30 words. ASD-STE100 allows 25. Split it.

Deleting an orphaned SeiNode is the one cleanup with a paved road. The rest of what a workspace PR leaves behind needs its own handling, so nothing in the [what a workspace-repo PR removes](#what-a-workspace-repo-pr-removes) list is left with no next step:


⚠️ [vale] <AgenticWriting.STE-Passive> reported by reviewdog 🐶
Passive voice: 'is left'. ASD-STE100 asks for the active voice.

Deleting an orphaned SeiNode is the one cleanup with a paved road. The rest of what a workspace PR leaves behind needs its own handling, so nothing in the [what a workspace-repo PR removes](#what-a-workspace-repo-pr-removes) list is left with no next step:


⚠️ [vale] <AgenticWriting.STE-Passive> reported by reviewdog 🐶
Passive voice: 'is gone'. ASD-STE100 asks for the active voice.

- **The removal PR merged while a SeiNetwork still read `Retain`.** The cascade is gone and no patch restores it. Do not re-apply the SeiNetwork to "reattach" the children — a fresh network under the same chain-id wedges at height 0 on the burned genesis artifacts. Go straight to [Find and clean up already-leaked resources](#find-and-clean-up-already-leaked-resources).


⚠️ [vale] <AgenticWriting.STE-SentenceLength-Description> reported by reviewdog 🐶
Descriptive sentence has 27 words. ASD-STE100 allows 25. Split it.

- **The removal PR merged while a SeiNetwork still read `Retain`.** The cascade is gone and no patch restores it. Do not re-apply the SeiNetwork to "reattach" the children — a fresh network under the same chain-id wedges at height 0 on the burned genesis artifacts. Go straight to [Find and clean up already-leaked resources](#find-and-clean-up-already-leaked-resources).


⚠️ [vale] <AgenticWriting.STE-Passive> reported by reviewdog 🐶
Passive voice: 'is ********'. ASD-STE100 asks for the active voice.

- **`kustomization <alias>` is `NotFound` in `eng-<alias>`.** The engineer's Flux wiring is missing, so no workspace-repo merge reconciles at all. Surface to the platform team; do not create the Kustomization.


⚠️ [vale] <AgenticWriting.STE-SentenceLength-Description> reported by reviewdog 🐶
Descriptive sentence has 27 words. ASD-STE100 allows 25. Split it.

- **`lastAppliedRevision` does not reach the merge commit within two reconcile intervals (~10 min).** Read the Ready condition's message (`cluster-inspection-recipes.md` recipe #8). A render error in `engineers/<alias>/kustomization.yaml` — most often a `resources:` entry pointing at the dir that was just removed — blocks every later apply in the namespace, not only this teardown.


⚠️ [vale] <AgenticWriting.STE-Passive> reported by reviewdog 🐶
Passive voice: 'is fixed'. ASD-STE100 asks for the active voice.

- **A verification read returned `UNVERIFIED`.** The API call failed, so the teardown state is unknown. Report it as unknown — never as verified-gone, and never as still-present. Re-run once the access problem is fixed; a teardown with an unverified check is not a finished teardown.


⚠️ [vale] <AgenticWriting.STE-Passive> reported by reviewdog 🐶
Passive voice: 'is preserved'. ASD-STE100 asks for the active voice.

- For **imported** PVCs (`spec.dataVolume.import` set on the SeiNode): the PVC is preserved; the recreated SeiNode reuses existing data.


⚠️ [vale] <AgenticWriting.STE-Passive> reported by reviewdog 🐶
Passive voice: 'being recreated'. ASD-STE100 asks for the active voice.

- For **controller-managed** PVCs (no `spec.dataVolume.import`): the controller's `handleNodeDeletion` path deletes the PVC during teardown. Delete-and-recreate **wipes data**. Safe for ephemeral chains being recreated from genesis; not safe for archive nodes or any chain with state worth preserving.


⚠️ [vale] <AgenticWriting.STE-Passive> reported by reviewdog 🐶
Passive voice: 'is governed'. ASD-STE100 asks for the active voice.

For a SeiNode, whether its PVC survives deletion is governed by `spec.dataVolume.import` (imported PVC = preserved) vs controller-managed (wiped on teardown) — documented under **Phase: Failed** above. A `SeiNetwork`'s `spec.deletionPolicy` (defaults `Retain`) governs whether the controller orphans its generated validator SeiNodes (and thus their PVCs) when the network is deleted. Forensics is the one case where `Retain` is the right answer; on an ordinary teardown it is a disk leak, because the orphaned validators keep running with no owner left to delete them (see `teardown.md`). The hardlink trick above is for **live debugging** while the node continues running. The two are complementary, not redundant.


⚠️ [vale] <AgenticWriting.STE-Passive> reported by reviewdog 🐶
Passive voice: 'is deleted'. ASD-STE100 asks for the active voice.

For a SeiNode, whether its PVC survives deletion is governed by `spec.dataVolume.import` (imported PVC = preserved) vs controller-managed (wiped on teardown) — documented under **Phase: Failed** above. A `SeiNetwork`'s `spec.deletionPolicy` (defaults `Retain`) governs whether the controller orphans its generated validator SeiNodes (and thus their PVCs) when the network is deleted. Forensics is the one case where `Retain` is the right answer; on an ordinary teardown it is a disk leak, because the orphaned validators keep running with no owner left to delete them (see `teardown.md`). The hardlink trick above is for **live debugging** while the node continues running. The two are complementary, not redundant.


⚠️ [vale] <AgenticWriting.STE-SentenceLength-Description> reported by reviewdog 🐶
Descriptive sentence has 33 words. ASD-STE100 allows 25. Split it.

For a SeiNode, whether its PVC survives deletion is governed by `spec.dataVolume.import` (imported PVC = preserved) vs controller-managed (wiped on teardown) — documented under **Phase: Failed** above. A `SeiNetwork`'s `spec.deletionPolicy` (defaults `Retain`) governs whether the controller orphans its generated validator SeiNodes (and thus their PVCs) when the network is deleted. Forensics is the one case where `Retain` is the right answer; on an ordinary teardown it is a disk leak, because the orphaned validators keep running with no owner left to delete them (see `teardown.md`). The hardlink trick above is for **live debugging** while the node continues running. The two are complementary, not redundant.


**Either way it works only before deletion.** Once a `Retain` deletion has stripped the owner references and removed the parent, nothing restores the cascade — the leftover SeiNodes and PVCs need manual cleanup (`teardown.md`).

Under `Delete` the chain runs end to end: SeiNetwork deleted → validators deleted through their owner references → each SeiNode's finalizer deletes its data PVC → the storage class's `Delete` reclaim policy releases the EBS volume. The finalizer skips an **imported** PVC (`spec.dataVolume.import` on the SeiNode) by design.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ [vale] <AgenticWriting.STE-SentenceLength-Description> reported by reviewdog 🐶
Descriptive sentence has 33 words. ASD-STE100 allows 25. Split it.

Both reads must agree on `Delete`. A live object reading `Delete` while git still declares `Retain` is the drift this path exists to close.
3. **Removal PR.** Only now `git rm` the task dir, per the procedure below.

**Path B — stack the removal PR on the policy PR.** Where preparing two changes serially is too slow, write both up front: branch the removal from the policy branch and open its PR as a **draft**. Then merge the policy PR, run the step-2 reads, and only mark the removal ready once both reads say `Delete`. This is still two separately landed changes — the saving is in preparation, not in the ordering. A single PR carrying both the policy edit and the `git rm` is not this path: it merges as one revision, so the policy and the removal reach the cluster in the same reconcile and the ordering never exists.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ [vale] <AgenticWriting.STE-SentenceLength-Description> reported by reviewdog 🐶
Descriptive sentence has 26 words. ASD-STE100 allows 25. Split it.

Both reads must agree on `Delete`. A live object reading `Delete` while git still declares `Retain` is the drift this path exists to close.
3. **Removal PR.** Only now `git rm` the task dir, per the procedure below.

**Path B — stack the removal PR on the policy PR.** Where preparing two changes serially is too slow, write both up front: branch the removal from the policy branch and open its PR as a **draft**. Then merge the policy PR, run the step-2 reads, and only mark the removal ready once both reads say `Delete`. This is still two separately landed changes — the saving is in preparation, not in the ordering. A single PR carrying both the policy edit and the `git rm` is not this path: it merges as one revision, so the policy and the removal reach the cluster in the same reconcile and the ordering never exists.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ [vale] <AgenticWriting.STE-SentenceLength-Description> reported by reviewdog 🐶
Descriptive sentence has 37 words. ASD-STE100 allows 25. Split it.


**Path B — stack the removal PR on the policy PR.** Where preparing two changes serially is too slow, write both up front: branch the removal from the policy branch and open its PR as a **draft**. Then merge the policy PR, run the step-2 reads, and only mark the removal ready once both reads say `Delete`. This is still two separately landed changes — the saving is in preparation, not in the ordering. A single PR carrying both the policy edit and the `git rm` is not this path: it merges as one revision, so the policy and the removal reach the cluster in the same reconcile and the ordering never exists.

**The live patch is a repair, and it is not available for a Flux-owned network.** `kubectl patch seinetwork <chain-id> -n eng-<alias> --type=merge -p '{"spec":{"deletionPolicy":"Delete"}}'` is correct in exactly one situation: the SeiNetwork is **not** in the workspace repo at all — an escape-hatch direct apply, or an object already orphaned from an earlier teardown — so no reconcile will revert it. Confirm that with the workspace search in [the other resources git never owned](#the-other-resources-git-never-owned) before relying on it.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ [vale] <AgenticWriting.STE-SentenceLength-Description> reported by reviewdog 🐶
Descriptive sentence has 36 words. ASD-STE100 allows 25. Split it.

1. Git declares `Retain`. The engineer patches the live object to `Delete`.
2. The pre-merge read returns `Delete`. It is true, and it is already stale.
3. Flux reconciles the existing revision — the one that still declares `Retain` — and restores `Retain`.
4. The removal merges. The SeiNetwork is deleted under `Retain`. The validators are orphaned.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ [vale] <AgenticWriting.STE-Passive> reported by reviewdog 🐶
Passive voice: 'is deleted'. ASD-STE100 asks for the active voice.


With `deletionPolicy: Delete` the chain runs end to end: SeiNetwork deleted → generated validator SeiNodes deleted through their owner references → each SeiNode's finalizer (`sei.io/seinode-finalizer`) deletes the node's data PVC → the storage class's `Delete` reclaim policy releases the EBS volume.

The finalizer **skips an imported PVC** (`spec.dataVolume.import` set on the SeiNode, naming the claim in `.pvcName`). An imported PVC is preserved by design; its disk is not a leak. See the field-path caveat under inventory step 2 before relying on that path in a query.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ [vale] <AgenticWriting.STE-Passive> reported by reviewdog 🐶
Passive voice: 'is preserved'. ASD-STE100 asks for the active voice.

Teardown follows the same PR contract as spinup: render the change, open a PR, let the engineer merge, verify what Flux did. Never `kubectl delete` a Flux-owned CR — the next reconcile re-applies it and the removal PR never lands.

1. **Pre-flight** — the five gates. Halt on first failure.
2. **Inventory what goes away and what stays — by name, before anything is deleted.** The verification in step 8 polls *named* claims, so this step produces those names. It has to run first: once the SeiNodes are gone, nothing in the cluster still records which claims were imported and which the controller managed.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ [vale] <AgenticWriting.STE-Passive> reported by reviewdog 🐶
Passive voice: 'is deleted'. ASD-STE100 asks for the active voice.

Teardown follows the same PR contract as spinup: render the change, open a PR, let the engineer merge, verify what Flux did. Never `kubectl delete` a Flux-owned CR — the next reconcile re-applies it and the removal PR never lands.

1. **Pre-flight** — the five gates. Halt on first failure.
2. **Inventory what goes away and what stays — by name, before anything is deleted.** The verification in step 8 polls *named* claims, so this step produces those names. It has to run first: once the SeiNodes are gone, nothing in the cluster still records which claims were imported and which the controller managed.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ [vale] <AgenticWriting.STE-Passive> reported by reviewdog 🐶
Passive voice: 'are gone'. ASD-STE100 asks for the active voice.

Teardown follows the same PR contract as spinup: render the change, open a PR, let the engineer merge, verify what Flux did. Never `kubectl delete` a Flux-owned CR — the next reconcile re-applies it and the removal PR never lands.

1. **Pre-flight** — the five gates. Halt on first failure.
2. **Inventory what goes away and what stays — by name, before anything is deleted.** The verification in step 8 polls *named* claims, so this step produces those names. It has to run first: once the SeiNodes are gone, nothing in the cluster still records which claims were imported and which the controller managed.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ [vale] <AgenticWriting.STE-Passive> reported by reviewdog 🐶
Passive voice: 'were imported'. ASD-STE100 asks for the active voice.

1. **Pre-flight** — the five gates. Halt on first failure.
2. **Inventory what goes away and what stays — by name, before anything is deleted.** The verification in step 8 polls *named* claims, so this step produces those names. It has to run first: once the SeiNodes are gone, nothing in the cluster still records which claims were imported and which the controller managed.

Save this as `inventory.sh` and run it with `sh inventory.sh`. It aborts on the first API or parse failure, because a partial inventory under-reports what must disappear and then reads as a clean teardown later.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ [vale] <AgenticWriting.STE-SentenceLength-Description> reported by reviewdog 🐶
Descriptive sentence has 26 words. ASD-STE100 allows 25. Split it.

… delete scope

The ownership walk listed every SeiNode in the namespace without
establishing which one referenced the candidate claim, then offered a
"confirmed orphan" verdict off that. Hop 3 now resolves the claim to the
pod that mounts it and that pod's owner reference, and an empty result
is unresolved — a stopped workload is exactly what made the volume read
available in the first place.

Each hop separates the API call from the parse and checks both: a
kubectl-into-jq pipeline exits with jq's status, so a Forbidden read as
"no match", which on this walk is the difference between unowned and
could-not-look. Missing-claim and no-PV rows are now explicitly
unresolved candidates requiring platform review, not bare candidates.

Add the tenant guard: claimRef comes from a cluster-wide PV list and can
name any namespace, so anything other than eng-<alias> escalates instead
of being inspected. Name the namespace and context on every delete, since
both CLIs fall back to the current context otherwise. Replace the
authorizing grep with one that distinguishes no-match (exit 1, the only
status that authorizes a delete) from search error (exit 2+), against a
freshly fetched clone.

Drop the GNU-only `sed -n '/x/,+3p'`.

Co-authored-by: omnigent <noreply@omnigent.ai>

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remaining comments which cannot be posted as a review comment to avoid GitHub Rate Limit

vale

⚠️ [vale] <AgenticWriting.STE-Passive> reported by reviewdog 🐶
Passive voice: 'is preserved'. ASD-STE100 asks for the active voice.

- For **imported** PVCs (`spec.dataVolume.import` set on the SeiNode): the PVC is preserved; the recreated SeiNode reuses existing data.


⚠️ [vale] <AgenticWriting.STE-Passive> reported by reviewdog 🐶
Passive voice: 'being recreated'. ASD-STE100 asks for the active voice.

- For **controller-managed** PVCs (no `spec.dataVolume.import`): the controller's `handleNodeDeletion` path deletes the PVC during teardown. Delete-and-recreate **wipes data**. Safe for ephemeral chains being recreated from genesis; not safe for archive nodes or any chain with state worth preserving.


⚠️ [vale] <AgenticWriting.STE-Passive> reported by reviewdog 🐶
Passive voice: 'is governed'. ASD-STE100 asks for the active voice.

For a SeiNode, whether its PVC survives deletion is governed by `spec.dataVolume.import` (imported PVC = preserved) vs controller-managed (wiped on teardown) — documented under **Phase: Failed** above. A `SeiNetwork`'s `spec.deletionPolicy` (defaults `Retain`) governs whether the controller orphans its generated validator SeiNodes (and thus their PVCs) when the network is deleted. Forensics is the one case where `Retain` is the right answer; on an ordinary teardown it is a disk leak, because the orphaned validators keep running with no owner left to delete them (see `teardown.md`). The hardlink trick above is for **live debugging** while the node continues running. The two are complementary, not redundant.


⚠️ [vale] <AgenticWriting.STE-Passive> reported by reviewdog 🐶
Passive voice: 'is deleted'. ASD-STE100 asks for the active voice.

For a SeiNode, whether its PVC survives deletion is governed by `spec.dataVolume.import` (imported PVC = preserved) vs controller-managed (wiped on teardown) — documented under **Phase: Failed** above. A `SeiNetwork`'s `spec.deletionPolicy` (defaults `Retain`) governs whether the controller orphans its generated validator SeiNodes (and thus their PVCs) when the network is deleted. Forensics is the one case where `Retain` is the right answer; on an ordinary teardown it is a disk leak, because the orphaned validators keep running with no owner left to delete them (see `teardown.md`). The hardlink trick above is for **live debugging** while the node continues running. The two are complementary, not redundant.


⚠️ [vale] <AgenticWriting.STE-SentenceLength-Description> reported by reviewdog 🐶
Descriptive sentence has 33 words. ASD-STE100 allows 25. Split it.

For a SeiNode, whether its PVC survives deletion is governed by `spec.dataVolume.import` (imported PVC = preserved) vs controller-managed (wiped on teardown) — documented under **Phase: Failed** above. A `SeiNetwork`'s `spec.deletionPolicy` (defaults `Retain`) governs whether the controller orphans its generated validator SeiNodes (and thus their PVCs) when the network is deleted. Forensics is the one case where `Retain` is the right answer; on an ordinary teardown it is a disk leak, because the orphaned validators keep running with no owner left to delete them (see `teardown.md`). The hardlink trick above is for **live debugging** while the node continues running. The two are complementary, not redundant.

The Retain eval forbade live patching only when there was no git change
AND no pre-merge re-verification, which preserved the exact unsafe
exception the docs now close; the happy path let the agent proceed on a
live Delete without requiring the committed manifest to agree. Both now
require git verification for a Flux-owned object, and the happy path
requires named-claim polling with outcome aggregation rather than a
namespace PVC sweep.

Co-authored-by: omnigent <noreply@omnigent.ai>
6. **Commit + push** — branch `feat/eng-<alias>-teardown-<task>`. Message: `feat(eng/<alias>): tear down <task> — chain-id=<chain-id>`.
7. **Open the PR** — title `feat(eng/<alias>): tear down <task>`. Body names the chain-id, every CR that goes away, and the `deletionPolicy` value the SeiNetwork now carries. Surface the URL and halt for the merge.
8. **After merge — reconcile the workspace Kustomization** — `flux --context harbor reconcile kustomization <alias> -n eng-<alias> --with-source`, then compare `.status.lastAppliedRevision` to the merge SHA. Reconciling `flux-system` here verifies the wrong repo (see Post-merge reconciliation).
9. **Poll until the resources disappear, and distinguish three outcomes** — a reconcile only says Flux issued the deletes. Deletion is asynchronous and finalizers hold objects in `Terminating` while the controller releases PVCs, so poll on a budget (~5 min) rather than asserting once. Report exactly one of **`GONE`** (the API answered and matched nothing), **`PRESENT`** (objects remain at the deadline), or **`UNVERIFIED`** (the API call failed). **A failed read is never a pass** — a `Forbidden` or dropped connection returns zero lines, so a check that counts lines without reading `kubectl`'s exit status claims success precisely when it cannot see the cluster. Use `poll_gone` / `expect_present` / `record` from `references/cluster-inspection-recipes.md` recipe #9 — one implementation, called, never re-typed. **Aggregate the outcomes**: a bare call discards its return code, so an early `UNVERIFIED` followed by a clean later read leaves the whole check exiting 0, which is the original bug on the exit-code path. Poll the chain's controller-managed claims **by name** from the step-2 inventory, and assert the imported ones separately — a namespace-wide PVC sweep also matches imported claims and other chains', so a correct teardown reports `PRESENT`.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ [vale] <AgenticWriting.STE-SentenceLength-Description> reported by reviewdog 🐶
Descriptive sentence has 37 words. ASD-STE100 allows 25. Split it.

6. **Commit + push** — branch `feat/eng-<alias>-teardown-<task>`. Message: `feat(eng/<alias>): tear down <task> — chain-id=<chain-id>`.
7. **Open the PR** — title `feat(eng/<alias>): tear down <task>`. Body names the chain-id, every CR that goes away, and the `deletionPolicy` value the SeiNetwork now carries. Surface the URL and halt for the merge.
8. **After merge — reconcile the workspace Kustomization** — `flux --context harbor reconcile kustomization <alias> -n eng-<alias> --with-source`, then compare `.status.lastAppliedRevision` to the merge SHA. Reconciling `flux-system` here verifies the wrong repo (see Post-merge reconciliation).
9. **Poll until the resources disappear, and distinguish three outcomes** — a reconcile only says Flux issued the deletes. Deletion is asynchronous and finalizers hold objects in `Terminating` while the controller releases PVCs, so poll on a budget (~5 min) rather than asserting once. Report exactly one of **`GONE`** (the API answered and matched nothing), **`PRESENT`** (objects remain at the deadline), or **`UNVERIFIED`** (the API call failed). **A failed read is never a pass** — a `Forbidden` or dropped connection returns zero lines, so a check that counts lines without reading `kubectl`'s exit status claims success precisely when it cannot see the cluster. Use `poll_gone` / `expect_present` / `record` from `references/cluster-inspection-recipes.md` recipe #9 — one implementation, called, never re-typed. **Aggregate the outcomes**: a bare call discards its return code, so an early `UNVERIFIED` followed by a clean later read leaves the whole check exiting 0, which is the original bug on the exit-code path. Poll the chain's controller-managed claims **by name** from the step-2 inventory, and assert the imported ones separately — a namespace-wide PVC sweep also matches imported claims and other chains', so a correct teardown reports `PRESENT`.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📝 [vale] <AgenticWriting.STE-ParagraphLength> reported by reviewdog 🐶
Paragraph has more than 6 sentences. ASD-STE100 allows 6.

6. **Commit + push** — branch `feat/eng-<alias>-teardown-<task>`. Message: `feat(eng/<alias>): tear down <task> — chain-id=<chain-id>`.
7. **Open the PR** — title `feat(eng/<alias>): tear down <task>`. Body names the chain-id, every CR that goes away, and the `deletionPolicy` value the SeiNetwork now carries. Surface the URL and halt for the merge.
8. **After merge — reconcile the workspace Kustomization** — `flux --context harbor reconcile kustomization <alias> -n eng-<alias> --with-source`, then compare `.status.lastAppliedRevision` to the merge SHA. Reconciling `flux-system` here verifies the wrong repo (see Post-merge reconciliation).
9. **Poll until the resources disappear, and distinguish three outcomes** — a reconcile only says Flux issued the deletes. Deletion is asynchronous and finalizers hold objects in `Terminating` while the controller releases PVCs, so poll on a budget (~5 min) rather than asserting once. Report exactly one of **`GONE`** (the API answered and matched nothing), **`PRESENT`** (objects remain at the deadline), or **`UNVERIFIED`** (the API call failed). **A failed read is never a pass** — a `Forbidden` or dropped connection returns zero lines, so a check that counts lines without reading `kubectl`'s exit status claims success precisely when it cannot see the cluster. Use `poll_gone` / `expect_present` / `record` from `references/cluster-inspection-recipes.md` recipe #9 — one implementation, called, never re-typed. **Aggregate the outcomes**: a bare call discards its return code, so an early `UNVERIFIED` followed by a clean later read leaves the whole check exiting 0, which is the original bug on the exit-code path. Poll the chain's controller-managed claims **by name** from the step-2 inventory, and assert the imported ones separately — a namespace-wide PVC sweep also matches imported claims and other chains', so a correct teardown reports `PRESENT`.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ [vale] <AgenticWriting.STE-SentenceLength-Description> reported by reviewdog 🐶
Descriptive sentence has 34 words. ASD-STE100 allows 25. Split it.

6. **Commit + push** — branch `feat/eng-<alias>-teardown-<task>`. Message: `feat(eng/<alias>): tear down <task> — chain-id=<chain-id>`.
7. **Open the PR** — title `feat(eng/<alias>): tear down <task>`. Body names the chain-id, every CR that goes away, and the `deletionPolicy` value the SeiNetwork now carries. Surface the URL and halt for the merge.
8. **After merge — reconcile the workspace Kustomization** — `flux --context harbor reconcile kustomization <alias> -n eng-<alias> --with-source`, then compare `.status.lastAppliedRevision` to the merge SHA. Reconciling `flux-system` here verifies the wrong repo (see Post-merge reconciliation).
9. **Poll until the resources disappear, and distinguish three outcomes** — a reconcile only says Flux issued the deletes. Deletion is asynchronous and finalizers hold objects in `Terminating` while the controller releases PVCs, so poll on a budget (~5 min) rather than asserting once. Report exactly one of **`GONE`** (the API answered and matched nothing), **`PRESENT`** (objects remain at the deadline), or **`UNVERIFIED`** (the API call failed). **A failed read is never a pass** — a `Forbidden` or dropped connection returns zero lines, so a check that counts lines without reading `kubectl`'s exit status claims success precisely when it cannot see the cluster. Use `poll_gone` / `expect_present` / `record` from `references/cluster-inspection-recipes.md` recipe #9 — one implementation, called, never re-typed. **Aggregate the outcomes**: a bare call discards its return code, so an early `UNVERIFIED` followed by a clean later read leaves the whole check exiting 0, which is the original bug on the exit-code path. Poll the chain's controller-managed claims **by name** from the step-2 inventory, and assert the imported ones separately — a namespace-wide PVC sweep also matches imported claims and other chains', so a correct teardown reports `PRESENT`.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ [vale] <AgenticWriting.STE-SentenceLength-Description> reported by reviewdog 🐶
Descriptive sentence has 35 words. ASD-STE100 allows 25. Split it.

**PVC behavior** — verify before deleting on stateful nodes:
- For **imported** PVCs (`spec.import` set on the SeiNode): the PVC is preserved; the recreated SeiNode reuses existing data.
- For **controller-managed** PVCs (no `spec.import`): the controller's `handleNodeDeletion` path deletes the PVC during teardown. Delete-and-recreate **wipes data**. Safe for ephemeral chains being recreated from genesis; not safe for archive nodes or any chain with state worth preserving.
- For **imported** PVCs (`spec.dataVolume.import` set on the SeiNode): the PVC is preserved; the recreated SeiNode reuses existing data.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ [vale] <AgenticWriting.STE-Passive> reported by reviewdog 🐶
Passive voice: 'is preserved'. ASD-STE100 asks for the active voice.

- For **imported** PVCs (`spec.import` set on the SeiNode): the PVC is preserved; the recreated SeiNode reuses existing data.
- For **controller-managed** PVCs (no `spec.import`): the controller's `handleNodeDeletion` path deletes the PVC during teardown. Delete-and-recreate **wipes data**. Safe for ephemeral chains being recreated from genesis; not safe for archive nodes or any chain with state worth preserving.
- For **imported** PVCs (`spec.dataVolume.import` set on the SeiNode): the PVC is preserved; the recreated SeiNode reuses existing data.
- For **controller-managed** PVCs (no `spec.dataVolume.import`): the controller's `handleNodeDeletion` path deletes the PVC during teardown. Delete-and-recreate **wipes data**. Safe for ephemeral chains being recreated from genesis; not safe for archive nodes or any chain with state worth preserving.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ [vale] <AgenticWriting.STE-Passive> reported by reviewdog 🐶
Passive voice: 'being recreated'. ASD-STE100 asks for the active voice.

### vs. retained data on delete

For a SeiNode, whether its PVC survives deletion is governed by `spec.import` (imported PVC = preserved) vs controller-managed (wiped on teardown) — documented under **Phase: Failed** above. A `SeiNetwork`'s `spec.deletionPolicy` (defaults `Retain`) governs whether the controller orphans its generated validator SeiNodes (and thus their PVCs) when the network is deleted — useful when tearing down a network but keeping a validator's disk for forensics. The hardlink trick above is for **live debugging** while the node continues running. They're complementary, not redundant.
For a SeiNode, whether its PVC survives deletion is governed by `spec.dataVolume.import` (imported PVC = preserved) vs controller-managed (wiped on teardown) — documented under **Phase: Failed** above. A `SeiNetwork`'s `spec.deletionPolicy` (defaults `Retain`) governs whether the controller orphans its generated validator SeiNodes (and thus their PVCs) when the network is deleted. Forensics is the one case where `Retain` is the right answer; on an ordinary teardown it is a disk leak, because the orphaned validators keep running with no owner left to delete them (see `teardown.md`). The hardlink trick above is for **live debugging** while the node continues running. The two are complementary, not redundant.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ [vale] <AgenticWriting.STE-Passive> reported by reviewdog 🐶
Passive voice: 'is governed'. ASD-STE100 asks for the active voice.

### vs. retained data on delete

For a SeiNode, whether its PVC survives deletion is governed by `spec.import` (imported PVC = preserved) vs controller-managed (wiped on teardown) — documented under **Phase: Failed** above. A `SeiNetwork`'s `spec.deletionPolicy` (defaults `Retain`) governs whether the controller orphans its generated validator SeiNodes (and thus their PVCs) when the network is deleted — useful when tearing down a network but keeping a validator's disk for forensics. The hardlink trick above is for **live debugging** while the node continues running. They're complementary, not redundant.
For a SeiNode, whether its PVC survives deletion is governed by `spec.dataVolume.import` (imported PVC = preserved) vs controller-managed (wiped on teardown) — documented under **Phase: Failed** above. A `SeiNetwork`'s `spec.deletionPolicy` (defaults `Retain`) governs whether the controller orphans its generated validator SeiNodes (and thus their PVCs) when the network is deleted. Forensics is the one case where `Retain` is the right answer; on an ordinary teardown it is a disk leak, because the orphaned validators keep running with no owner left to delete them (see `teardown.md`). The hardlink trick above is for **live debugging** while the node continues running. The two are complementary, not redundant.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ [vale] <AgenticWriting.STE-Passive> reported by reviewdog 🐶
Passive voice: 'is deleted'. ASD-STE100 asks for the active voice.

### vs. retained data on delete

For a SeiNode, whether its PVC survives deletion is governed by `spec.import` (imported PVC = preserved) vs controller-managed (wiped on teardown) — documented under **Phase: Failed** above. A `SeiNetwork`'s `spec.deletionPolicy` (defaults `Retain`) governs whether the controller orphans its generated validator SeiNodes (and thus their PVCs) when the network is deleted — useful when tearing down a network but keeping a validator's disk for forensics. The hardlink trick above is for **live debugging** while the node continues running. They're complementary, not redundant.
For a SeiNode, whether its PVC survives deletion is governed by `spec.dataVolume.import` (imported PVC = preserved) vs controller-managed (wiped on teardown) — documented under **Phase: Failed** above. A `SeiNetwork`'s `spec.deletionPolicy` (defaults `Retain`) governs whether the controller orphans its generated validator SeiNodes (and thus their PVCs) when the network is deleted. Forensics is the one case where `Retain` is the right answer; on an ordinary teardown it is a disk leak, because the orphaned validators keep running with no owner left to delete them (see `teardown.md`). The hardlink trick above is for **live debugging** while the node continues running. The two are complementary, not redundant.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ [vale] <AgenticWriting.STE-SentenceLength-Description> reported by reviewdog 🐶
Descriptive sentence has 33 words. ASD-STE100 allows 25. Split it.

@seidroid seidroid Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Findings on the changed lines. The verdict and the summary are in this tool's comment on this pull request.

```
2. For every SeiNetwork in the inventory, run the `deletionPolicy` gate in [The `deletionPolicy: Retain` trap](#the-deletionpolicy-retain-trap). One `Retain` network is enough to leak a set of disks.
3. `git rm -r` every task dir under `engineers/<alias>/`, and reduce `engineers/<alias>/kustomization.yaml` to `resources: []`. Keep that file: deleting it makes the Flux Kustomization fail reconcile with `path not found`, which is the same breakage the onboarding scaffolding PR exists to prevent.
4. Open the PR, merge, then run [Verify the teardown](#verify-the-teardown) with no `-l` selector, so the poll covers the whole namespace.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

blocker — This step now contradicts the section it links to. Verify the teardown polls PVCs by name from ./teardown-inventory/managed-claims.txt and imported-claims.txt — its PVC calls carry no -l to drop — and both line 245 here and cluster-inspection-recipes.md:249 state in bold that a namespace sweep is the wrong check, because imported claims and other chains' claims make it report PRESENT after a correct teardown. Step 1 of this procedure is a display-only kubectl get seinetwork,seinode,job,pvc, so it never generates those name lists: an operator following this step either runs the sweep the same file argues trains the reader to ignore the check, or has no inputs for the verification at all. It is also worst in the case that most needs it — a namespace holding orphans from an earlier Retain teardown, where a sweep can never reach GONE. Point step 1 at the inventory script from Procedure step 2, run over every chain in the namespace, and have step 4 use the same by-name calls.

res=$1; shift
deadline=$(( $(date +%s) + 300 ))
while : ; do
out=$(kubectl --context harbor get "$res" -n eng-<alias> "$@" -o name 2>&1); rc=$?

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

suggestion2>&1 merges stderr into out, and left=$(printf '%s' "$out" | grep -c .) counts every line in it as a remaining object. Any stderr line emitted alongside a successful read — an apiserver deprecation or warning header, a stale-cache notice — inflates the count, so the loop runs to the deadline and reports PRESENT for a namespace that is actually empty; expect_present at line 224 has the same construction and would report MISSING. The direction is fail-safe, but a false alarm is exactly what line 249 says trains the reader to ignore the check. Capture stderr into a separate variable or file so the count sees only stdout, and keep it for the UNVERIFIED message. (Correcting my earlier reviews: I attributed this to kubectl's "No resources found" notice; on the -o name path that is unverified, and the warning-header case is what holds.)

- **`<chain-tag>` exceeds the 22-char budget** when the `-{a,b}-rpc-<k>` suffix is added. Surface the overflow and ask the engineer to pick a shorter tag.
- **CR name collision on either side.** Halt before render; surface the existing object's age + labels.
- **One network reaches `Ready` while the other reaches `Failed`.** The comparison is invalid. Surface the failed side's `.status.plan.failedTaskDetail.error`. The half-teardown is two coordinated edits, **both required** — Flux refuses to apply a kustomization with a missing resource:
- **First**, land `deletionPolicy: Delete` in the surviving side's SeiNetwork manifest if it reads `Retain`, and let it reconcile before the removal merges (`teardown.md` — a live patch gets reverted). The failed side needs the same read: a network that never reached `Ready` may still have generated validators to orphan.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

suggestion — The gate names the wrong SeiNetwork for the steps it prefixes. This bullet lands deletionPolicy: Delete on the surviving side, but the two steps below git rm the failed side's chain-<a-or-b>/ and the closing line says Flux prunes the SeiNetwork on the failed side. The network whose policy decides whether validators are orphaned is the one being removed, so the gate belongs there; "the failed side needs the same read" leaves the load-bearing case as a secondary check. The verb is right now — swap which side it applies to.

**Read `references/teardown.md` first.** It carries the `deletionPolicy` trap in full, the namespace procedures, and the sweep for resources that already leaked.

1. **Pre-flight** — five gates. Halt on first failure.
2. **Inventory what goes away** — `kubectl get seinetwork,seinode -n eng-<alias> -l sei.io/seinetwork=<chain-id>` plus `kubectl get pvc -n eng-<alias>`. Show the engineer the list before touching anything.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

suggestion — This inventory is a bare kubectl get pvc -n eng-<alias> and step 9 says only "poll the PVCs too", so the always-loaded procedure produces neither the managed nor the imported claim-name list that recipe #9 now requires, and never mentions the record/VERDICT aggregation that keeps an early UNVERIFIED from being masked. An agent working from SKILL.md alone lands on the namespace sweep that teardown.md:245 and recipe #9 both call the wrong check. Point this step at the inventory script in references/teardown.md procedure step 2, and have step 9 name the by-name calls and the verdict aggregation.

seidroid[bot]
seidroid Bot previously requested changes Sep 9, 2026

@seidroid seidroid Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

review found something blocking. The findings are on the lines they are about, and the summary is in this tool's comment on this pull request.

@seidroid

seidroid Bot commented Sep 9, 2026

Copy link
Copy Markdown

What changed

The pull request moved again (merge commit 965ea6d, diff now 907 lines). references/teardown.md grew from 381 to 406 lines, cluster-inspection-recipes.md recipe #9 was substantially rewritten, and troubleshooting-seinode.md gained a hunk correcting spec.import to spec.dataVolume.import.

Three of my four blockers are addressed. Path B is now executable — it stacks a draft removal PR on the policy PR and states plainly that a single PR carrying both is not that path (teardown.md:65). The imported-PVC inventory is rebuilt: the field path is now .spec.dataVolume.import.pvcName, matching the only path this repo records, carrying an explicit caveat that it comes from controller main rather than the deployed CRD and that the CRD wins; and the inventory now emits real claim names, read from the pods' own spec.volumes[].persistentVolumeClaim.claimName rather than a guessed naming rule, split into managed and imported lists. The unselected poll_gone pvc is gone — PVCs are polled by name with --ignore-not-found, with a new expect_present asserting the imported claims survived, and a record/VERDICT aggregation so a later success cannot mask an earlier UNVERIFIED. The inventory script also avoids the kubectl | jq pipeline whose exit status belongs to jq, and pins LC_ALL=C for the comm against jq's codepoint sort. One blocker remains, and it is one this revision created.

A correction to my last two reviews: I said 2>&1 makes GONE unreachable because a namespaced kubectl get prints "No resources found" to stderr. With -o name, kubectl takes its non-human-readable print path, and I do not have a cluster here to confirm the notice is emitted on that path. Treat that claim as unverified — the narrower defect below is what holds.

1. Blocking

The namespace-emptying path instructs the check the verification section forbids, using files that path never creates. references/teardown.md:270 says to run Verify the teardown "with no -l selector, so the poll covers the whole namespace." The section it links to no longer works that way: its PVC calls take names from ./teardown-inventory/managed-claims.txt and imported-claims.txt, they carry no -l to drop, and teardown.md:245 and cluster-inspection-recipes.md:249 both state in bold that a namespace sweep is the wrong check because imported claims and other chains' claims make it report PRESENT after a correct teardown. The empty-the-namespace procedure's own inventory (teardown.md:264) is a display-only kubectl get seinetwork,seinode,job,pvc, so it never produces those name lists. An operator following this step either performs the sweep the same file argues trains readers to ignore the check, or has no inputs for the verification at all — and in the case the sweep is most likely to be run, a namespace holding orphans from an earlier Retain teardown, the poll cannot reach GONE at all.

2. Non-blocking

  • references/cluster-inspection-recipes.md:202 (and :224) merge stderr into the counted output with 2>&1, and grep -c . counts every line in it; any stderr line emitted alongside a successful read — an apiserver deprecation warning, a stale-cache notice — inflates left/got and yields a false PRESENT or MISSING. Fail-safe in direction, but a false alarm is what the file at :249 says trains the reader to ignore the check; capture stderr separately and count only stdout.
  • references/comparative-bench.md:429 names the wrong SeiNetwork: the bullet gates the surviving side, while the two steps it prefixes git rm the failed side's chain-<a-or-b>/ and the closing line says Flux prunes the failed side — the network being removed is the one whose policy decides whether validators are orphaned. (The verb was correctly updated to "land in the manifest" this revision.)
  • SKILL.md:265 inventories with a bare kubectl get pvc -n eng-<alias>, and SKILL.md:272 says only "poll the PVCs too"; neither produces the named managed/imported lists that recipe chore(agents): add Output Discipline section to all agents #9 now requires, nor mentions the record/verdict aggregation, so the always-loaded procedure still leads to the sweep the reference forbids.
  • references/teardown.md:3 states the file's governing rule as "patch spec.deletionPolicy to Delete … and land that patch before the removal merges", which the file's own :67 now closes off for any Flux-owned SeiNetwork.
  • evals/evals.json:242 still says spec.import where the rest of the change moved to spec.dataVolume.import.
  • references/teardown.md:26 asserts SeiNetworkSpec carries "exactly three" immutability rules, while the PR body says four and seinetwork-crd.md:38 names two.
  • references/teardown.md:150 tells the reader to "source" poll_gone, expect_present, and record, but they live in a fenced block in a markdown reference with no file to source.
  • references/teardown.md:342 uses sed -n '/Used By/,+3p'; the addr,+N form is GNU-only and fails on the BSD sed macOS ships, which is the laptop this skill's pre-flight assumes.
  • "Render new chains with Delete from the start" (references/teardown.md:81-90) is still the only preventive step and still sits in the teardown-time file; the spinup procedure is untouched, so every new chain renders Retain.
  • The Delete enum literal remains uncited in this repo — the /kubernetes kit corroborates only the Retain branch — and the PR body's "inferred rather than verified" list still omits it, unlike the field path and tag keys it now marks in place.
  • references/teardown.md:367 asserts gp3-10k-750, gp3, and gp3-archive and their reclaim policies; those names appear nowhere else in the repo and carry none of the unverified marking the same file now gives the CRD field path.
  • The escape-hatch row at references/teardown.md:385 directs seictl network|node delete without restating that SKILL.md:381 requires explicit per-invocation confirmation for that verb.
  • The orphan jq (references/teardown.md:293, cluster-inspection-recipes.md:331) emits - when the sei.io/seinetwork label is absent, and the follow-up kubectl get seinetwork <seinetwork-label-value> then runs against -.

3. Summary

This revision closed Path B, the imported-PVC field path and identifier, and the unselected PVC poll — the three blockers I carried in, one of which was codex's from an earlier dispatch — and the replacements are stronger than what I asked for: claim names read from the pods rather than guessed, a MISSING assertion for preserved claims, verdict aggregation so no later success masks an earlier UNVERIFIED, and a field-path caveat that defers to the deployed CRD. What blocks now is a seam the rewrite opened: the empty-the-namespace step still calls for a namespace-wide poll that the verification section it links to has just finished arguing is the wrong check, and it depends on inventory files that path never generates. I also retract, above, my earlier claim that the 2>&1 capture makes GONE unreachable — the mechanism I named is unverified for -o name, and the remaining defect is narrower and non-blocking. Codex contributed no reading this dispatch — it reported reading no diff — so nothing was carried over or dropped from it here. The repository has no REVIEW.md on main.

seidroid review · decision request_changes · session 9e5542c4a82248afa23681c6eda59b44 · turn resp_claude_d4d3a5917b087706481b35377ecaeb84 · item 1c5c78d4c2045a9aaf3f5902ae91ab0f

Findings: 1 blocking | 14 non-blocking | 4 posted inline | 1 pre-existing

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

Comment thread .claude/skills/harbor-dev/references/teardown.md Outdated
6. **Commit + push** — branch `feat/eng-<alias>-teardown-<task>`. Message: `feat(eng/<alias>): tear down <task> — chain-id=<chain-id>`.
7. **Open the PR** — title `feat(eng/<alias>): tear down <task>`. Body names the chain-id, every CR that goes away, and the `deletionPolicy` value the SeiNetwork now carries. Surface the URL and halt for the merge.
8. **After merge — reconcile the workspace Kustomization** — `flux --context harbor reconcile kustomization <alias> -n eng-<alias> --with-source`, then compare `.status.lastAppliedRevision` to the merge SHA. Reconciling `flux-system` here verifies the wrong repo (see Post-merge reconciliation).
9. **Poll until the resources disappear, and distinguish three outcomes** — a reconcile only says Flux issued the deletes. Deletion is asynchronous and finalizers hold objects in `Terminating` while the controller releases PVCs, so poll on a budget (~5 min) rather than asserting once. Report exactly one of **`GONE`** (the API answered and matched nothing), **`PRESENT`** (objects remain at the deadline), or **`UNVERIFIED`** (the API call failed). **A failed read is never a pass** — a `Forbidden` or dropped connection returns zero lines, so a check that counts lines without reading `kubectl`'s exit status claims success precisely when it cannot see the cluster. Use `poll_gone` / `expect_present` / `record` from `references/cluster-inspection-recipes.md` recipe #9 — one implementation, called, never re-typed. **Aggregate the outcomes**: a bare call discards its return code, so an early `UNVERIFIED` followed by a clean later read leaves the whole check exiting 0, which is the original bug on the exit-code path. Poll the chain's controller-managed claims **by name** from the step-2 inventory, and assert the imported ones separately — a namespace-wide PVC sweep also matches imported claims and other chains', so a correct teardown reports `PRESENT`.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SKILL inventory omits claim classification

Medium Severity

Step 9 now polls controller-managed PVCs by name from the step-2 inventory and forbids a namespace-wide sweep, and the happy-path eval requires capturing imported versus managed claims. Step 2 still runs kubectl get pvc -n eng-<alias> and never classifies those claims, so an agent following SKILL.md inventories the sweep that step 9 says makes a correct teardown report PRESENT.

Additional Locations (2)
Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit 45a623c. Configure here.

… reads a checked step

expect_present merged stderr into stdout and counted non-empty lines, so
one API deprecation warning satisfied want=1 and an absent claim reported
PRESERVED. poll_gone had the same mixing, where a warning could make an
empty result report PRESENT. Both now keep stderr in a separate sink,
count only lines shaped like a resource id, and expect_present matches
the returned identities against the requested names rather than counting.

Inventory name lists were inlined as $(cat file) into the helper argument
list, so a missing or unreadable file collapsed to an empty argument list
and the helpers succeeded against an empty namespace — losing the
inventory read as a clean teardown, and $? was the helper's status, never
cat's. read_inventory is now its own checked step with three distinct
states: has-entries, legitimately-empty, and missing-or-unreadable. The
empty and unreadable branches are implemented, not described.

Every helper runs its API call inside a condition and every caller uses
an OR-list, so set -e can no longer terminate inside an assignment before
classification or before record runs.

Verified under dash and bash, with and without set -eu: stderr warning
with an absent claim reports MISSING; warning with empty stdout reports
GONE; missing inventory reports UNVERIFIED; empty inventory takes the
NOTE branch and leaves the verdict alone; UNVERIFIED then GONE still
aggregates to 2.

Co-authored-by: omnigent <noreply@omnigent.ai>

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

Comment thread .claude/skills/harbor-dev/references/cluster-inspection-recipes.md Outdated
Comment thread .claude/skills/harbor-dev/references/cluster-inspection-recipes.md Outdated
…elete context

Managed claims came only from pods that currently exist, so a node whose
pod was absent contributed nothing — and a provisioned PVC with no pod is
exactly the leaked disk this document exists to catch. The inventory now
attributes claims per node, computes the nodes that resolve to no storage
at all, writes them to unresolved-nodes.txt, exits non-zero, and leaves a
status file reading UNRESOLVED that forces the verifier to UNVERIFIED.
Pod phase is deliberately not consulted: a Pending pod still declares its
volumes. Namespace PVCs no chain attributes are surfaced separately for
the leak sweep rather than folded into the managed list.

The namespace-emptying path had only a display command and generated none
of the named-claim files the verifier consumes; it now runs inventory.sh
once per chain-id, each with its own INV directory.

Replace the unenforceable "(harbor context)" parenthetical on the direct
delete: seictl's documented flags are --kubeconfig and -n only, with no
--context, so the direct deletes use kubectl --context harbor, which pins
cluster and namespace on the line that deletes. A guarded seictl form is
documented for workflows that need it, with its check-to-call window
stated.

Also: the hop-3 pod parse now has the explicit UNRESOLVED branch the
other hops have, and the ownership search exits 0/1/2 so a scripted
caller can tell "not in git" from "the search failed".

Verified under dash and bash: a node with no pod lands unresolved and
stops the run; the happy path flips status to OK; the search returns
three distinct exit codes under set -e.

Co-authored-by: omnigent <noreply@omnigent.ai>

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remaining comments which cannot be posted as a review comment to avoid GitHub Rate Limit

vale

⚠️ [vale] <AgenticWriting.STE-Passive> reported by reviewdog 🐶
Passive voice: 'is gone'. ASD-STE100 asks for the active voice.

| Volume → PV → PVC whose claim is gone, PV `Released` | **Unresolved candidate.** Platform review required. Report the PV, PVC name, and reclaim policy; do not act on it here. |


⚠️ [vale] <AgenticWriting.STE-Passive> reported by reviewdog 🐶
Passive voice: 'is told'. ASD-STE100 asks for the active voice.

Only the first two rows are verdicts. Every other row is an escalation, and the platform team is told which row it came from.


⚠️ [vale] <AgenticWriting.STE-SentenceLength-Description> reported by reviewdog 🐶
Descriptive sentence has 28 words. ASD-STE100 allows 25. Split it.

`kubectl get pv` is cluster-scoped and the per-engineer Role is namespaced, so `Forbidden` at hop 1 is the **normal** case for an engineer — an unresolved result, not a clean one. When it happens, hand the volume IDs to the platform team and let them walk the chain; do not substitute the tag data for the walk.


⚠️ [vale] <AgenticWriting.STE-Passive> reported by reviewdog 🐶
Passive voice: 'is namespaced'. ASD-STE100 asks for the active voice.

`kubectl get pv` is cluster-scoped and the per-engineer Role is namespaced, so `Forbidden` at hop 1 is the **normal** case for an engineer — an unresolved result, not a clean one. When it happens, hand the volume IDs to the platform team and let them walk the chain; do not substitute the tag data for the walk.


⚠️ [vale] <AgenticWriting.STE-SentenceLength-Description> reported by reviewdog 🐶
Descriptive sentence has 26 words. ASD-STE100 allows 25. Split it.

`kubectl get pv` is cluster-scoped and the per-engineer Role is namespaced, so `Forbidden` at hop 1 is the **normal** case for an engineer — an unresolved result, not a clean one. When it happens, hand the volume IDs to the platform team and let them walk the chain; do not substitute the tag data for the walk.


⚠️ [vale] <AgenticWriting.STE-Passive> reported by reviewdog 🐶
Passive voice: 'is ******'. ASD-STE100 asks for the active voice.

The node's finalizer deletes its data PVC, and the `Delete` reclaim policy on `gp3-10k-750` (validators) and `gp3` (default) releases the EBS volume. `gp3-archive` is `Retain` by design — a volume on that class stays after its PVC goes, and its removal is an AWS-side decision, not a mistake to correct here.


⚠️ [vale] <AgenticWriting.STE-SentenceLength-Description> reported by reviewdog 🐶
Descriptive sentence has 27 words. ASD-STE100 allows 25. Split it.

The node's finalizer deletes its data PVC, and the `Delete` reclaim policy on `gp3-10k-750` (validators) and `gp3` (default) releases the EBS volume. `gp3-archive` is `Retain` by design — a volume on that class stays after its PVC goes, and its removal is an AWS-side decision, not a mistake to correct here.


⚠️ [vale] <AgenticWriting.STE-SentenceLength-Description> reported by reviewdog 🐶
Descriptive sentence has 33 words. ASD-STE100 allows 25. Split it.

**Never delete an EBS volume from this skill.** Even a volume the ownership walk resolved to a dead PVC goes to the platform team: `ec2:DeleteVolume` is outside the engineer's policy, the walk can be wrong, and an EBS delete is unrecoverable. Hand over the volume IDs, sizes, creation times, and the walk's verdict per volume — including every `UNRESOLVED` one, labelled as unresolved. Escalate through `#harbor-onboarding`. Do not report the cleanup as complete while any ID is outstanding.


⚠️ [vale] <AgenticWriting.STE-SentenceLength-Description> reported by reviewdog 🐶
Descriptive sentence has 30 words. ASD-STE100 allows 25. Split it.

Deleting an orphaned SeiNode is the one cleanup with a paved road. The rest of what a workspace PR leaves behind needs its own handling, so nothing in the [what a workspace-repo PR removes](#what-a-workspace-repo-pr-removes) list is left with no next step:


⚠️ [vale] <AgenticWriting.STE-Passive> reported by reviewdog 🐶
Passive voice: 'is left'. ASD-STE100 asks for the active voice.

Deleting an orphaned SeiNode is the one cleanup with a paved road. The rest of what a workspace PR leaves behind needs its own handling, so nothing in the [what a workspace-repo PR removes](#what-a-workspace-repo-pr-removes) list is left with no next step:


⚠️ [vale] <AgenticWriting.STE-SentenceLength-Description> reported by reviewdog 🐶
Descriptive sentence has 26 words. ASD-STE100 allows 25. Split it.

**Every namespace-scoped command above names its namespace, and every destructive one names its context.** An unqualified `delete` deletes wherever the shell happens to point, and that is not a typo you can retry — it is a delete in the wrong place.


⚠️ [vale] <AgenticWriting.STE-SentenceLength-Description> reported by reviewdog 🐶
Descriptive sentence has 32 words. ASD-STE100 allows 25. Split it.

If a workflow genuinely needs `seictl` for a destructive verb, pin the cluster out of band and prove it immediately before, in the same command list — a guard that runs, not a parenthetical:


⚠️ [vale] <AgenticWriting.STE-SentenceLength-Description> reported by reviewdog 🐶
Descriptive sentence has 39 words. ASD-STE100 allows 25. Split it.

Before deleting anything imperatively, prove the object is **not** in the workspace repo. A search that fails must never read as "no manifest found" — `grep` exits 1 for no match and 2 or more for an error, and an unreadable or stale clone produces the same empty output as a genuinely absent manifest.


⚠️ [vale] <AgenticWriting.STE-SentenceLength-Description> reported by reviewdog 🐶
Descriptive sentence has 47 words. ASD-STE100 allows 25. Split it.

**Only exit status 0 from this block authorizes a direct delete.** Every branch used to end in a successful `echo`, so the block's own status was 0 whatever it found — a scripted caller could not tell "not in git" from "the search failed", which is the same class of defect as counting lines without reading an exit status. `1` routes to the PR path; `2` means the question was never answered.


⚠️ [vale] <AgenticWriting.STE-Passive> reported by reviewdog 🐶
Passive voice: 'is gone'. ASD-STE100 asks for the active voice.

- **The removal PR merged while a SeiNetwork still read `Retain`.** The cascade is gone and no patch restores it. Do not re-apply the SeiNetwork to "reattach" the children — a fresh network under the same chain-id wedges at height 0 on the burned genesis artifacts. Go straight to [Find and clean up already-leaked resources](#find-and-clean-up-already-leaked-resources).


⚠️ [vale] <AgenticWriting.STE-SentenceLength-Description> reported by reviewdog 🐶
Descriptive sentence has 27 words. ASD-STE100 allows 25. Split it.

- **The removal PR merged while a SeiNetwork still read `Retain`.** The cascade is gone and no patch restores it. Do not re-apply the SeiNetwork to "reattach" the children — a fresh network under the same chain-id wedges at height 0 on the burned genesis artifacts. Go straight to [Find and clean up already-leaked resources](#find-and-clean-up-already-leaked-resources).


⚠️ [vale] <AgenticWriting.STE-Passive> reported by reviewdog 🐶
Passive voice: 'is ********'. ASD-STE100 asks for the active voice.

- **`kustomization <alias>` is `NotFound` in `eng-<alias>`.** The engineer's Flux wiring is missing, so no workspace-repo merge reconciles at all. Surface to the platform team; do not create the Kustomization.


⚠️ [vale] <AgenticWriting.STE-SentenceLength-Description> reported by reviewdog 🐶
Descriptive sentence has 27 words. ASD-STE100 allows 25. Split it.

- **`lastAppliedRevision` does not reach the merge commit within two reconcile intervals (~10 min).** Read the Ready condition's message (`cluster-inspection-recipes.md` recipe #8). A render error in `engineers/<alias>/kustomization.yaml` — most often a `resources:` entry pointing at the dir that was just removed — blocks every later apply in the namespace, not only this teardown.


⚠️ [vale] <AgenticWriting.STE-Passive> reported by reviewdog 🐶
Passive voice: 'is fixed'. ASD-STE100 asks for the active voice.

- **A verification read returned `UNVERIFIED`.** The API call failed, so the teardown state is unknown. Report it as unknown — never as verified-gone, and never as still-present. Re-run once the access problem is fixed; a teardown with an unverified check is not a finished teardown.

@seidroid seidroid Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Findings on the changed lines. The verdict and the summary are in this tool's comment on this pull request.


**And a later success must never overwrite an earlier failure.** Printing `UNVERIFIED` is not enough on its own: a `break` out of a loop, or a bare call whose return code nobody reads, still leaves the block exiting 0. A human sees the warning; a wrapper script or an agent reading `$?` sees success. Every check records its outcome into a running verdict, and the worst one wins.

**Use `poll_gone`, `expect_present`, and `record` from `cluster-inspection-recipes.md` recipe #9 — do not re-implement them here.** One implementation, one place to fix. Source them, then:

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

blocker — This is the block an operator following teardown.md runs, and it is a generation behind the recipe it cites. It never reads $INV/status, it inlines $(cat ./teardown-inventory/managed-claims.txt) into the arguments, and it uses ; record $? — the two patterns recipe #9 names as defects, one in the words 'losing the inventory reads as a clean teardown'. The new inventory makes the consequence concrete: inventory.sh writes managed-claims.txt before completeness check 1, so an aborted run leaves that file present but missing the unresolved node's claim while status reads UNRESOLVED. This block polls the truncated list, finds nothing, and prints TEARDOWN VERIFIED — precisely the node whose disk may be leaking, reported as verified. Two smaller faults in the same block: it hardcodes ./teardown-inventory while the namespace procedure now directs per-chain INV=./teardown-inventory-<chain-id> directories, and a missing file collapses to an empty argument list that turns poll_gone pvc --ignore-not-found into the namespace sweep this page forbids. Line 256 already says not to re-implement the helpers: replace the block with recipe #9's read_inventory/OR-list version, parameterised on INV, or drop it and link there.

- **`<chain-tag>` exceeds the 22-char budget** when the `-{a,b}-rpc-<k>` suffix is added. Surface the overflow and ask the engineer to pick a shorter tag.
- **CR name collision on either side.** Halt before render; surface the existing object's age + labels.
- **One network reaches `Ready` while the other reaches `Failed`.** The comparison is invalid. Surface the failed side's `.status.plan.failedTaskDetail.error`. The half-teardown is two coordinated edits, **both required** — Flux refuses to apply a kustomization with a missing resource:
- **First**, land `deletionPolicy: Delete` in the surviving side's SeiNetwork manifest if it reads `Retain`, and let it reconcile before the removal merges (`teardown.md` — a live patch gets reverted). The failed side needs the same read: a network that never reached `Ready` may still have generated validators to orphan.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

suggestion — The gate still names the wrong side. It lands deletionPolicy: Delete in the surviving side's manifest, but the two steps below git rm the failed side's chain-<a-or-b>/ and the closing line says Flux prunes the SeiNetwork on the failed side. The network whose policy decides whether its validators are orphaned is the one being removed, so the gate belongs there; 'the failed side needs the same read' leaves the load-bearing case as a secondary check. The verb is correct now — only the side needs swapping.

**Read `references/teardown.md` first.** It carries the `deletionPolicy` trap in full, the namespace procedures, and the sweep for resources that already leaked.

1. **Pre-flight** — five gates. Halt on first failure.
2. **Inventory what goes away** — `kubectl get seinetwork,seinode -n eng-<alias> -l sei.io/seinetwork=<chain-id>` plus `kubectl get pvc -n eng-<alias>`. Show the engineer the list before touching anything.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

suggestion — This step is still a bare kubectl get seinetwork,seinode plus kubectl get pvc -n eng-<alias>, which captures no imported-versus-managed split and produces none of the named-claim files. Step 9 now correctly tells the agent to poll 'by name from the step-2 inventory' and to aggregate outcomes, so the always-loaded procedure asks its own step 2 for something it does not produce — and an agent working from SKILL.md alone lands on the namespace sweep that teardown.md and recipe #9 both reject. Point this step at inventory.sh in references/teardown.md procedure step 2, and name the status/managed-claims.txt/imported-claims.txt outputs step 9 consumes.

seidroid[bot]
seidroid Bot previously requested changes Sep 9, 2026

@seidroid seidroid Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

review found something blocking. The findings are on the lines they are about, and the summary is in this tool's comment on this pull request.

@seidroid

seidroid Bot commented Sep 9, 2026

Copy link
Copy Markdown

What changed

The pull request moved again (merge commit daf1fb5, diff now 1116 lines). Only references/teardown.md changed, 459 → 542 lines; every other file is byte-identical to the last revision.

Two of my three blockers are closed. The inventory script now writes $INV/statusUNRESOLVED up front, flipped to OK only after both completeness checks pass — and produces the unresolved-nodes.txt that recipe #9's gate names, so the verification block can reach VERIFIED on a clean teardown instead of always reporting UNVERIFIED. The empty-the-namespace path no longer calls for the sweep the verification forbids: it enumerates the namespace's chain-ids, runs inventory.sh once per chain into its own INV directory, and states that the CR and pod polls may drop -l while the claim checks may not. Beyond what I raised, the inventory now attributes claims node-by-node and refuses to pass when a SeiNode resolves to no storage — closing the "node whose pod is absent goes unverified" gap I had listed as non-blocking — surfaces unattributed namespace claims, and marks the pod-to-node attribution as a convention rather than a contract. The ownership walk's hop 3 now separates parse failure from "no pod mounts it", and the ownership-search block returns distinct exit statuses instead of always exiting 0. The switch of direct deletes from seictl to kubectl --context harbor checks out against seictl-cli.md:31-34, which lists only --kubeconfig and -n/--namespace as common flags and carries no --context anywhere in the file. One blocker is untouched, and the hardening around it has made it sharper.

1. Blocking

The verify block in teardown.md is still the pre-rewrite version, and it now silently discards the safety the inventory just gained. references/teardown.md:256-278 inlines $(cat ./teardown-inventory/managed-claims.txt) into the helpers' arguments and uses ; record $? — the two patterns recipe #9 explicitly names as defects, one of them in the words "losing the inventory reads as a clean teardown" — and it never reads $INV/status. The concrete consequence is new this revision: when inventory.sh aborts at completeness check 1, it has already written managed-claims.txt, so that file exists but omits the unresolved node's claim while status reads UNRESOLVED. This block polls the truncated list, finds it clean, and prints TEARDOWN VERIFIED — a node's disk never checked, reported as verified. It also hardcodes ./teardown-inventory while the namespace path four sections down now directs per-chain INV=./teardown-inventory-<chain-id> directories, and a missing file yields an empty argument list that turns poll_gone pvc --ignore-not-found into the namespace sweep the same page forbids. Line 256 tells the reader not to re-implement these helpers while the block underneath does exactly that; recipe #9's read_inventory/OR-list version already exists to be pointed at.

2. Non-blocking

  • The whole inventory now hinges on a pod's first owner reference name equalling the SeiNode name (references/teardown.md:130-139); the file marks it a convention and fails safe, but if the controller's StatefulSet naming differs at all, every node lands unresolved and inventory.sh exits 2 on every chain, so the documented teardown can never complete until someone verifies the convention.
  • references/comparative-bench.md:429 gates the surviving side, while the two steps it prefixes git rm the failed side's chain-<a-or-b>/ and the closing line says Flux prunes the failed side — the network being removed is the one whose policy decides whether validators are orphaned.
  • SKILL.md:265 inventories with a bare kubectl get pvc -n eng-<alias> and captures no imported-versus-managed split, while step 9 tells the agent to poll "by name from the step-2 inventory"; the always-loaded procedure's step 2 still does not produce those names or mention inventory.sh.
  • references/cluster-inspection-recipes.md:293 hardcodes INV=./teardown-inventory, which the new per-chain namespace procedure contradicts.
  • references/teardown.md:3 still states the file's governing rule as "patch spec.deletionPolicy to Delete … and land that patch before the removal merges", which :67 closes off for any Flux-owned SeiNetwork.
  • references/teardown.md:482 now prescribes kubectl --context harbor delete seinetwork|seinode for a confirmed non-Flux object, but SKILL.md:381's confirmation taxonomy gates only seictl network|node delete, so the newly recommended command sits outside the documented approval gate.
  • references/teardown.md:26 asserts SeiNetworkSpec carries "exactly three" immutability rules, while the PR body says four and seinetwork-crd.md:38 names two.
  • references/cluster-inspection-recipes.md:198 leaves $ERRF in ${TMPDIR:-/tmp} with no cleanup.
  • references/teardown.md:256 says to "source" the four helpers, but they live in a fenced block inside a markdown reference with no file to source.
  • "Render new chains with Delete from the start" (references/teardown.md:81-90) is still the only preventive step and still sits in the teardown-time file; the spinup procedure is untouched, so every new chain renders Retain.
  • The Delete enum literal remains uncited in this repo — the /kubernetes kit corroborates only the Retain branch — and the PR body's "inferred rather than verified" list still omits it, unlike the field path, attribution convention, and tag keys the file marks in place.
  • references/teardown.md:467 asserts gp3-10k-750, gp3, and gp3-archive and their reclaim policies; those names appear nowhere else in the repo and carry none of the unverified marking the same file gives its other inferences.
  • The orphan check (references/teardown.md:359, cluster-inspection-recipes.md:353) substitutes - when the sei.io/seinetwork label is absent, so the parent lookup runs against -.

3. Summary

This revision closed the status-gate mismatch and the namespace-sweep contradiction, and went further than I asked on both: the inventory now refuses to declare itself complete when a SeiNode resolves to no storage, which is the leak case, and the namespace path runs a real per-chain inventory instead of a display read. What still blocks is the one seam that has survived three revisions — teardown.md's own verify block is the copy an operator runs, and it is a generation behind the recipe it tells the reader to use: no status check, inlined $(cat …), ; record $?, a hardcoded inventory directory. Because the inventory now writes a truncated managed-claims.txt alongside an UNRESOLVED status when it aborts, that stale block will report TEARDOWN VERIFIED in exactly the case the new checks were added to catch. The fix is deletion and a pointer, not new material. Codex contributed no reading this dispatch — it reported reading no diff — so nothing was carried over or dropped from it here. The repository has no REVIEW.md on main.

seidroid review · decision request_changes · session 9e5542c4a82248afa23681c6eda59b44 · turn resp_claude_044fb88b1c6d2107a3b5aafe4d31de55 · item 629b1292390452ccb59fc6aef1c70d54

Findings: 1 blocking | 13 non-blocking | 3 posted inline | 1 pre-existing

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

Comment thread .claude/skills/harbor-dev/references/teardown.md Outdated
Comment thread .claude/skills/harbor-dev/references/teardown.md Outdated
Comment thread .claude/skills/harbor-dev/references/teardown.md Outdated
…tion

The canonical caller in teardown.md re-implemented the whole
orchestration under a paragraph telling the reader not to, and never
called read_inventory, never read the completeness certificate, and
never consumed unresolved-nodes.txt. Everything built in the library was
dead code on the path that matters. That is the same defect relocating
one layer up for the fourth time: duplicated poll bodies, then the caller
chain, then the caller arguments, then a caller that bypassed the fix.

Recipe #9 now holds one parameterized verify_teardown carrying the
certificate gate, the checked list reads, the empty-list branches as
code, the polls and the aggregation. teardown.md contains no
verification shell at all — only a call. Single-chain, namespace, and
bench teardown all go through that one function; a bench passes `-` for
the inventory dir because it owns no claims.

The certificate now names its target, so a complete inventory for another
chain cannot authorize this one, and it is written only after every check
passes. The inventory takes a fresh directory, so a failed refresh cannot
leave a stale OK beside half-written lists. Each transformation is its
own command: `cut … | sort -u` exits with sort's status, so a failed cut
produced a successful empty claim list and the inventory certified itself
complete while omitting every managed claim.

Namespace teardown checks chain discovery's own status, aggregates every
chain into one verdict, and sweeps for unlabelled leftovers.

Remove the guarded seictl delete: a current-context check reads mutable
state rather than pinning the config the delete consumes, and the
explicitly scoped kubectl path already exists.

Verified under dash and bash, with and without set -eu: 11 verify_teardown
scenarios including a stale cross-chain certificate, an unresolved node, a
missing imported claim behind a stderr warning, bench mode, the unlabelled
sweep, and multi-chain aggregation where a clean second chain must not
cover an unverified first. Plus: a stale certificate does not survive a
failed refresh, and a failed cut aborts before certifying.

Co-authored-by: omnigent <noreply@omnigent.ai>

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remaining comments which cannot be posted as a review comment to avoid GitHub Rate Limit

vale

⚠️ [vale] <AgenticWriting.STE-Passive> reported by reviewdog 🐶
Passive voice: 'are held'. ASD-STE100 asks for the active voice.

If the poll runs out with objects still present, do not report the teardown as done and do not force the objects away. Read why they are held:


⚠️ [vale] <write-good.Weasel> reported by reviewdog 🐶
'very' is a weasel word!

"Destroy my namespace" means one of two very different things. Ask which before acting.


⚠️ [vale] <AgenticWriting.STE-SentenceLength-Description> reported by reviewdog 🐶
Descriptive sentence has 34 words. ASD-STE100 allows 25. Split it.

1. **List what is there, then inventory each chain properly.** The display read below is an overview, not an inventory — it produces none of the named-claim files the verifier consumes, so it cannot stand in for step 2 of the per-chain procedure:


⚠️ [vale] <AgenticWriting.STE-SentenceLength-Description> reported by reviewdog 🐶
Descriptive sentence has 35 words. ASD-STE100 allows 25. Split it.

Then discover the chain-ids **with the discovery's own status checked**. Piping `kubectl` into `sort` exits with sort's status, so a `Forbidden` becomes a successful empty list — and "no chains found" then reads as "nothing to do", which is the whole defect class this document exists to close:


⚠️ [vale] <write-good.Weasel> reported by reviewdog 🐶
'usually' is a weasel word!

Run `inventory.sh` **once per chain-id**, each writing its own `./teardown-inventory-<chain-id>`. A namespace usually holds more than one chain, and a single sweep cannot tell one chain's controller-managed claim from another's. Any chain whose `inventory.sh` exits non-zero writes no certificate, and its verification then reports `UNVERIFIED` — emptying a namespace on an incomplete inventory is how a leak becomes invisible. Claims no chain attributes land in each run's `unattributed-claims.txt`; take those to the leak sweep in step 5, not to a delete.


⚠️ [vale] <AgenticWriting.STE-SentenceLength-Description> reported by reviewdog 🐶
Descriptive sentence has 27 words. ASD-STE100 allows 25. Split it.

Run `inventory.sh` **once per chain-id**, each writing its own `./teardown-inventory-<chain-id>`. A namespace usually holds more than one chain, and a single sweep cannot tell one chain's controller-managed claim from another's. Any chain whose `inventory.sh` exits non-zero writes no certificate, and its verification then reports `UNVERIFIED` — emptying a namespace on an incomplete inventory is how a leak becomes invisible. Claims no chain attributes land in each run's `unattributed-claims.txt`; take those to the leak sweep in step 5, not to a delete.


⚠️ [vale] <AgenticWriting.STE-SentenceLength-Description> reported by reviewdog 🐶
Descriptive sentence has 26 words. ASD-STE100 allows 25. Split it.

**Empty the namespace first, through the steps above.** This is an operational preference, not a claim about a failure mode: the Kubernetes namespace controller does remove namespaced resources on its own. Emptying first keeps the `deletionPolicy` gate, the disappearance poll, and the leak sweep available while the objects are still there to inspect. Once the namespace is going away, a `Retain` SeiNetwork's orphans are much harder to reason about, and there is no inventory left to check them against.


⚠️ [vale] <AgenticWriting.STE-Passive> reported by reviewdog 🐶
Passive voice: 'is gone'. ASD-STE100 asks for the active voice.

Each line is a validator still running with nothing that will ever delete it. Confirm the parent is gone before treating one as an orphan:


⚠️ [vale] <AgenticWriting.STE-SentenceLength-Description> reported by reviewdog 🐶
Descriptive sentence has 28 words. ASD-STE100 allows 25. Split it.

- **EC2 `state: available` does not mean unowned.** It means unattached. A volume backing a live PV whose PVC is `Bound` reads `available` the moment its workload stops — a scaled-to-zero StatefulSet, a pod stuck `Pending`, a node drained mid-reschedule. Deleting on that signal destroys a disk somebody is coming back to.


⚠️ [vale] <AgenticWriting.STE-Passive> reported by reviewdog 🐶
Passive voice: 'is unwanted'. ASD-STE100 asks for the active voice.

- **`Used By: <pod>` does not prove the pod belongs to an orphan**, and `Used By: <none>` does not prove the PVC is unwanted. `describe pvc` reports current pod attachment, not ownership.


⚠️ [vale] <AgenticWriting.STE-SentenceLength-Description> reported by reviewdog 🐶
Descriptive sentence has 34 words. ASD-STE100 allows 25. Split it.

Those tag keys are the EBS CSI driver's own convention rather than something this skill's repos set. To confirm they are present before trusting an empty result, describe **one volume you already know is live, by ID** — never re-run without `--filters`, which enumerates every volume in the account including other tenants':


⚠️ [vale] <AgenticWriting.STE-SentenceLength-Description> reported by reviewdog 🐶
Descriptive sentence has 26 words. ASD-STE100 allows 25. Split it.

**Hop 2 is a scope gate, not just a lookup.** `kubectl get pv` is cluster-scoped, so the `claimRef` it returns can name *any* namespace. Assert it is this tenant's before inspecting further — a claim in another namespace is another tenant's disk, and this skill does not investigate those.


⚠️ [vale] <AgenticWriting.STE-Passive> reported by reviewdog 🐶
Passive voice: 'is unresolved'. ASD-STE100 asks for the active voice.

The owner reference names the StatefulSet the controller created for the node, not the SeiNode directly. Map it back to a SeiNode by name and confirm that node is a **confirmed orphan** by the signature in [Orphaned SeiNodes](#orphaned-seinodes). If you cannot make that link, the hop is unresolved.


⚠️ [vale] <AgenticWriting.STE-Passive> reported by reviewdog 🐶
Passive voice: 'was stopped'. ASD-STE100 asks for the active voice.

| Volume → PV → PVC → pod → a live, wanted workload | **Not garbage.** Leave it. `available` only meant the workload was stopped. |


⚠️ [vale] <AgenticWriting.STE-Passive> reported by reviewdog 🐶
Passive voice: 'is gone'. ASD-STE100 asks for the active voice.

| Volume → PV → PVC whose claim is gone, PV `Released` | **Unresolved candidate.** Platform review required. Report the PV, PVC name, and reclaim policy; do not act on it here. |


⚠️ [vale] <AgenticWriting.STE-Passive> reported by reviewdog 🐶
Passive voice: 'is told'. ASD-STE100 asks for the active voice.

Only the first two rows are verdicts. Every other row is an escalation, and the platform team is told which row it came from.


⚠️ [vale] <AgenticWriting.STE-SentenceLength-Description> reported by reviewdog 🐶
Descriptive sentence has 28 words. ASD-STE100 allows 25. Split it.

`kubectl get pv` is cluster-scoped and the per-engineer Role is namespaced, so `Forbidden` at hop 1 is the **normal** case for an engineer — an unresolved result, not a clean one. When it happens, hand the volume IDs to the platform team and let them walk the chain; do not substitute the tag data for the walk.


⚠️ [vale] <AgenticWriting.STE-Passive> reported by reviewdog 🐶
Passive voice: 'is namespaced'. ASD-STE100 asks for the active voice.

`kubectl get pv` is cluster-scoped and the per-engineer Role is namespaced, so `Forbidden` at hop 1 is the **normal** case for an engineer — an unresolved result, not a clean one. When it happens, hand the volume IDs to the platform team and let them walk the chain; do not substitute the tag data for the walk.


⚠️ [vale] <AgenticWriting.STE-SentenceLength-Description> reported by reviewdog 🐶
Descriptive sentence has 26 words. ASD-STE100 allows 25. Split it.

`kubectl get pv` is cluster-scoped and the per-engineer Role is namespaced, so `Forbidden` at hop 1 is the **normal** case for an engineer — an unresolved result, not a clean one. When it happens, hand the volume IDs to the platform team and let them walk the chain; do not substitute the tag data for the walk.


⚠️ [vale] <AgenticWriting.STE-Passive> reported by reviewdog 🐶
Passive voice: 'is ******'. ASD-STE100 asks for the active voice.

The node's finalizer deletes its data PVC, and the `Delete` reclaim policy on `gp3-10k-750` (validators) and `gp3` (default) releases the EBS volume. `gp3-archive` is `Retain` by design — a volume on that class stays after its PVC goes, and its removal is an AWS-side decision, not a mistake to correct here.


⚠️ [vale] <AgenticWriting.STE-SentenceLength-Description> reported by reviewdog 🐶
Descriptive sentence has 27 words. ASD-STE100 allows 25. Split it.

The node's finalizer deletes its data PVC, and the `Delete` reclaim policy on `gp3-10k-750` (validators) and `gp3` (default) releases the EBS volume. `gp3-archive` is `Retain` by design — a volume on that class stays after its PVC goes, and its removal is an AWS-side decision, not a mistake to correct here.


⚠️ [vale] <AgenticWriting.STE-SentenceLength-Description> reported by reviewdog 🐶
Descriptive sentence has 33 words. ASD-STE100 allows 25. Split it.

**Never delete an EBS volume from this skill.** Even a volume the ownership walk resolved to a dead PVC goes to the platform team: `ec2:DeleteVolume` is outside the engineer's policy, the walk can be wrong, and an EBS delete is unrecoverable. Hand over the volume IDs, sizes, creation times, and the walk's verdict per volume — including every `UNRESOLVED` one, labelled as unresolved. Escalate through `#harbor-onboarding`. Do not report the cleanup as complete while any ID is outstanding.


⚠️ [vale] <AgenticWriting.STE-SentenceLength-Description> reported by reviewdog 🐶
Descriptive sentence has 30 words. ASD-STE100 allows 25. Split it.

Deleting an orphaned SeiNode is the one cleanup with a paved road. The rest of what a workspace PR leaves behind needs its own handling, so nothing in the [what a workspace-repo PR removes](#what-a-workspace-repo-pr-removes) list is left with no next step:


⚠️ [vale] <AgenticWriting.STE-Passive> reported by reviewdog 🐶
Passive voice: 'is left'. ASD-STE100 asks for the active voice.

Deleting an orphaned SeiNode is the one cleanup with a paved road. The rest of what a workspace PR leaves behind needs its own handling, so nothing in the [what a workspace-repo PR removes](#what-a-workspace-repo-pr-removes) list is left with no next step:


⚠️ [vale] <AgenticWriting.STE-SentenceLength-Description> reported by reviewdog 🐶
Descriptive sentence has 26 words. ASD-STE100 allows 25. Split it.

**Every namespace-scoped command above names its namespace, and every destructive one names its context.** An unqualified `delete` deletes wherever the shell happens to point, and that is not a typo you can retry — it is a delete in the wrong place.


🚫 [vale] <write-good.ThereIs> reported by reviewdog 🐶
Don't start a sentence with 'There is'.

**There is no `seictl` alternative for network or node deletion here, guarded or otherwise.** A `kubectl config current-context` check reads mutable state rather than pinning the config the delete then consumes, so it leaves a window between the check and the call — and it buys nothing, because `kubectl --context harbor delete` does the same deletion against the same CR with no window at all.


⚠️ [vale] <AgenticWriting.STE-SentenceLength-Description> reported by reviewdog 🐶
Descriptive sentence has 43 words. ASD-STE100 allows 25. Split it.

**There is no `seictl` alternative for network or node deletion here, guarded or otherwise.** A `kubectl config current-context` check reads mutable state rather than pinning the config the delete then consumes, so it leaves a window between the check and the call — and it buys nothing, because `kubectl --context harbor delete` does the same deletion against the same CR with no window at all.


⚠️ [vale] <AgenticWriting.STE-SentenceLength-Description> reported by reviewdog 🐶
Descriptive sentence has 31 words. ASD-STE100 allows 25. Split it.

Should some genuinely `seictl`-only destructive verb ever need documenting, the pin belongs on the invocation: hand it a kubeconfig that contains the harbor cluster and nothing else, via `--kubeconfig <harbor-only-file>` (a documented `seictl` flag). A file that cannot name another cluster cannot select one. Do not substitute a current-context check.


⚠️ [vale] <AgenticWriting.STE-SentenceLength-Description> reported by reviewdog 🐶
Descriptive sentence has 39 words. ASD-STE100 allows 25. Split it.

Before deleting anything imperatively, prove the object is **not** in the workspace repo. A search that fails must never read as "no manifest found" — `grep` exits 1 for no match and 2 or more for an error, and an unreadable or stale clone produces the same empty output as a genuinely absent manifest.


⚠️ [vale] <AgenticWriting.STE-SentenceLength-Description> reported by reviewdog 🐶
Descriptive sentence has 47 words. ASD-STE100 allows 25. Split it.

**Only exit status 0 from this block authorizes a direct delete.** Every branch used to end in a successful `echo`, so the block's own status was 0 whatever it found — a scripted caller could not tell "not in git" from "the search failed", which is the same class of defect as counting lines without reading an exit status. `1` routes to the PR path; `2` means the question was never answered.


⚠️ [vale] <AgenticWriting.STE-Passive> reported by reviewdog 🐶
Passive voice: 'is gone'. ASD-STE100 asks for the active voice.

- **The removal PR merged while a SeiNetwork still read `Retain`.** The cascade is gone and no patch restores it. Do not re-apply the SeiNetwork to "reattach" the children — a fresh network under the same chain-id wedges at height 0 on the burned genesis artifacts. Go straight to [Find and clean up already-leaked resources](#find-and-clean-up-already-leaked-resources).


⚠️ [vale] <AgenticWriting.STE-SentenceLength-Description> reported by reviewdog 🐶
Descriptive sentence has 27 words. ASD-STE100 allows 25. Split it.

- **The removal PR merged while a SeiNetwork still read `Retain`.** The cascade is gone and no patch restores it. Do not re-apply the SeiNetwork to "reattach" the children — a fresh network under the same chain-id wedges at height 0 on the burned genesis artifacts. Go straight to [Find and clean up already-leaked resources](#find-and-clean-up-already-leaked-resources).


⚠️ [vale] <AgenticWriting.STE-Passive> reported by reviewdog 🐶
Passive voice: 'is ********'. ASD-STE100 asks for the active voice.

- **`kustomization <alias>` is `NotFound` in `eng-<alias>`.** The engineer's Flux wiring is missing, so no workspace-repo merge reconciles at all. Surface to the platform team; do not create the Kustomization.


⚠️ [vale] <AgenticWriting.STE-SentenceLength-Description> reported by reviewdog 🐶
Descriptive sentence has 27 words. ASD-STE100 allows 25. Split it.

- **`lastAppliedRevision` does not reach the merge commit within two reconcile intervals (~10 min).** Read the Ready condition's message (`cluster-inspection-recipes.md` recipe #8). A render error in `engineers/<alias>/kustomization.yaml` — most often a `resources:` entry pointing at the dir that was just removed — blocks every later apply in the namespace, not only this teardown.


⚠️ [vale] <AgenticWriting.STE-Passive> reported by reviewdog 🐶
Passive voice: 'is fixed'. ASD-STE100 asks for the active voice.

- **A verification read returned `UNVERIFIED`.** The API call failed, so the teardown state is unknown. Report it as unknown — never as verified-gone, and never as still-present. Re-run once the access problem is fixed; a teardown with an unverified check is not a finished teardown.


A Flux reconcile reports success once it issues the deletes. Deletion is asynchronous and finalizers hold objects in `Terminating` while the controller releases their PVCs, so poll rather than assert once.

**This block is the only verification code in this skill.** `verify_teardown` is the single entry point: every teardown — one chain, a whole namespace, a bench — calls it and reads its return value. Nothing re-implements the orchestration, because the one defect this whole procedure exists to prevent ("report success when the check could not actually look") has repeatedly survived by reappearing in a second copy of the orchestration one layer up. One copy is the control for that.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ [vale] <AgenticWriting.STE-SentenceLength-Description> reported by reviewdog 🐶
Descriptive sentence has 39 words. ASD-STE100 allows 25. Split it.


Written for a portable shell (`dash`, `ash`, `bash`). Three deliberate non-POSIX dependencies, all near-universal: `date +%s`, `mktemp -d`, and `kubectl`'s own flags. Bash's `SECONDS` is *not* usable — it is unset under `sh`, where the comparison dies with `Illegal number` and the loop never runs.

**Three rules hold everywhere below.** Each corresponds to a defect found while reviewing this document, and fixed before it merged. None of them ever ran against a cluster. They are recorded because each would have shipped a verifier that passes when it cannot see the cluster, and because the same defect class kept reappearing until the rule was written down:

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ [vale] <AgenticWriting.STE-SentenceLength-Description> reported by reviewdog 🐶
Descriptive sentence has 32 words. ASD-STE100 allows 25. Split it.


Written for a portable shell (`dash`, `ash`, `bash`). Three deliberate non-POSIX dependencies, all near-universal: `date +%s`, `mktemp -d`, and `kubectl`'s own flags. Bash's `SECONDS` is *not* usable — it is unset under `sh`, where the comparison dies with `Illegal number` and the loop never runs.

**Three rules hold everywhere below.** Each corresponds to a defect found while reviewing this document, and fixed before it merged. None of them ever ran against a cluster. They are recorded because each would have shipped a verifier that passes when it cannot see the cluster, and because the same defect class kept reappearing until the rule was written down:

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ [vale] <AgenticWriting.STE-Passive> reported by reviewdog 🐶
Passive voice: 'are recorded'. ASD-STE100 asks for the active voice.


Written for a portable shell (`dash`, `ash`, `bash`). Three deliberate non-POSIX dependencies, all near-universal: `date +%s`, `mktemp -d`, and `kubectl`'s own flags. Bash's `SECONDS` is *not* usable — it is unset under `sh`, where the comparison dies with `Illegal number` and the loop never runs.

**Three rules hold everywhere below.** Each corresponds to a defect found while reviewing this document, and fixed before it merged. None of them ever ran against a cluster. They are recorded because each would have shipped a verifier that passes when it cannot see the cluster, and because the same defect class kept reappearing until the rule was written down:

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ [vale] <AgenticWriting.STE-Passive> reported by reviewdog 🐶
Passive voice: 'was written'. ASD-STE100 asks for the active voice.

**Three rules hold everywhere below.** Each corresponds to a defect found while reviewing this document, and fixed before it merged. None of them ever ran against a cluster. They are recorded because each would have shipped a verifier that passes when it cannot see the cluster, and because the same defect class kept reappearing until the rule was written down:

1. **stderr never mixes with resource output.** `2>&1` merges API deprecation warnings into the result, and "count the non-empty lines" then treats one warning as one resource.
2. **Identities are matched, not counted.** A count says how many lines came back, not whether the resources you asked about are the ones that came back.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ [vale] <AgenticWriting.STE-Passive> reported by reviewdog 🐶
Passive voice: 'are matched'. ASD-STE100 asks for the active voice.


Compare that revision to the merge commit SHA. A stale revision means Flux has not applied the removal yet, so any disappearance check below is premature.

A `Forbidden` on `--with-source` **may** mean the `GitRepository` the Kustomization references sits outside `eng-<alias>`, beyond the engineer's namespace-scoped Role. It may equally be an expired session, a missing EKS access entry, or a Role that never carried the Flux verbs. Read the message before concluding which. Dropping `--with-source` helps only the first cause: reconciling the Kustomization alone applies the revision the source has already fetched, and the source polls on its own schedule. It repairs nothing for an expired session, a missing access entry, or a Role without the Flux verbs — those fail the same way with or without the flag. **The fallback has to succeed on its own terms.** If the reconcile without `--with-source` also fails, you have no reconcile at all: stop, fix the access problem, and do not proceed to the disappearance check, whose result would be `UNVERIFIED` anyway.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ [vale] <AgenticWriting.STE-SentenceLength-Description> reported by reviewdog 🐶
Descriptive sentence has 28 words. ASD-STE100 allows 25. Split it.


Compare that revision to the merge commit SHA. A stale revision means Flux has not applied the removal yet, so any disappearance check below is premature.

A `Forbidden` on `--with-source` **may** mean the `GitRepository` the Kustomization references sits outside `eng-<alias>`, beyond the engineer's namespace-scoped Role. It may equally be an expired session, a missing EKS access entry, or a Role that never carried the Flux verbs. Read the message before concluding which. Dropping `--with-source` helps only the first cause: reconciling the Kustomization alone applies the revision the source has already fetched, and the source polls on its own schedule. It repairs nothing for an expired session, a missing access entry, or a Role without the Flux verbs — those fail the same way with or without the flag. **The fallback has to succeed on its own terms.** If the reconcile without `--with-source` also fails, you have no reconcile at all: stop, fix the access problem, and do not proceed to the disappearance check, whose result would be `UNVERIFIED` anyway.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ [vale] <AgenticWriting.STE-SentenceLength-Description> reported by reviewdog 🐶
Descriptive sentence has 30 words. ASD-STE100 allows 25. Split it.


### Confirm the resources disappeared

A successful reconcile says Flux applied the change. It does not say the objects are gone. Deletion is asynchronous and finalizers hold objects in `Terminating` while the controller releases their PVCs, so poll instead of asserting once.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ [vale] <AgenticWriting.STE-Passive> reported by reviewdog 🐶
Passive voice: 'are gone'. ASD-STE100 asks for the active voice.

| `PRESENT` | The API answered and objects remain at the deadline. | Not torn down. Read the finalizers below. |
| `UNVERIFIED` | The API call failed — `Forbidden`, expired credential, connection error. | **Teardown not confirmed.** Say the check could not run. |

**A failed API read is never a pass.** A `Forbidden` or a dropped connection returns zero lines, and a check that counts lines without reading the exit status prints "gone" precisely when it cannot see the cluster.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ [vale] <AgenticWriting.STE-SentenceLength-Description> reported by reviewdog 🐶
Descriptive sentence has 28 words. ASD-STE100 allows 25. Split it.


**A failed API read is never a pass.** A `Forbidden` or a dropped connection returns zero lines, and a check that counts lines without reading the exit status prints "gone" precisely when it cannot see the cluster.

**And a later success must never overwrite an earlier failure.** Printing `UNVERIFIED` is not enough on its own: a `break` out of a loop, or a bare call whose return code nobody reads, still leaves the block exiting 0. A human sees the warning; a wrapper script or an agent reading `$?` sees success. Every check records its outcome into a running verdict, and the worst one wins.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ [vale] <AgenticWriting.STE-SentenceLength-Description> reported by reviewdog 🐶
Descriptive sentence has 27 words. ASD-STE100 allows 25. Split it.

`storage:\n` matches skill-package-checks A2's drive-letter pattern as
`e:\n`. Same false positive as round 4, same fix.

Co-authored-by: omnigent <noreply@omnigent.ai>

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remaining comments which cannot be posted as a review comment to avoid GitHub Rate Limit

vale

⚠️ [vale] <AgenticWriting.STE-SentenceLength-Description> reported by reviewdog 🐶
Descriptive sentence has 27 words. ASD-STE100 allows 25. Split it.

- **The removal PR merged while a SeiNetwork still read `Retain`.** The cascade is gone and no patch restores it. Do not re-apply the SeiNetwork to "reattach" the children — a fresh network under the same chain-id wedges at height 0 on the burned genesis artifacts. Go straight to [Find and clean up already-leaked resources](#find-and-clean-up-already-leaked-resources).


⚠️ [vale] <AgenticWriting.STE-Passive> reported by reviewdog 🐶
Passive voice: 'is ********'. ASD-STE100 asks for the active voice.

- **`kustomization <alias>` is `NotFound` in `eng-<alias>`.** The engineer's Flux wiring is missing, so no workspace-repo merge reconciles at all. Surface to the platform team; do not create the Kustomization.


⚠️ [vale] <AgenticWriting.STE-SentenceLength-Description> reported by reviewdog 🐶
Descriptive sentence has 27 words. ASD-STE100 allows 25. Split it.

- **`lastAppliedRevision` does not reach the merge commit within two reconcile intervals (~10 min).** Read the Ready condition's message (`cluster-inspection-recipes.md` recipe #8). A render error in `engineers/<alias>/kustomization.yaml` — most often a `resources:` entry pointing at the dir that was just removed — blocks every later apply in the namespace, not only this teardown.


⚠️ [vale] <AgenticWriting.STE-Passive> reported by reviewdog 🐶
Passive voice: 'is fixed'. ASD-STE100 asks for the active voice.

- **A verification read returned `UNVERIFIED`.** The API call failed, so the teardown state is unknown. Report it as unknown — never as verified-gone, and never as still-present. Re-run once the access problem is fixed; a teardown with an unverified check is not a finished teardown.

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

Comment thread .claude/skills/harbor-dev/references/teardown.md

```sh
# ============ harbor teardown verification library ========================
# Source this, then call verify_teardown. Do not copy pieces of it.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SKILL.md still calls old helpers

Medium Severity

Recipe #9 now says verify_teardown is the only entry point and changes poll_gone / expect_present to take the namespace first. SKILL.md step 9 still tells the agent to call poll_gone / expect_present / record directly. That contradicts the skill’s first surface: an agent following SKILL.md skips the certificate and named-claim gates, and the old helper signature binds kinds as the namespace so the check cannot see the cluster.

Additional Locations (1)
Fix in Cursor Fix in Web

Triggered by learned rule: Skill cross-file claim precision: taxonomy and authority statements must match everywhere

Reviewed by Cursor Bugbot for commit ba2c5a0. Configure here.

…ore the verdict, isolate traps

Chain discovery checked its API read but not its transformations: `| sort -u`
takes sort's status and `|| true` swallowed the rest, so a failed grep or sort
produced a successful EMPTY chain list, the per-chain loop ran zero times, and
every certificate and claim check was skipped at VERDICT 0. Filter and sort are
now separate checked steps where only grep's exit 1 counts as legitimate
emptiness. The prose that argued this could not happen carried the same gap —
it reasoned only about an API failure — and now names both.

Both orphan sweeps piped an unchecked kubectl into jq. A failed read gives jq no
input, and jq exits 0 with no output, so an unreadable sweep read as "no
orphans". Read and parse are separated and both checked.

The namespace path printed NAMESPACE EMPTIED and exited before step 5, so a
namespace whose only leftover was a leaked PVC passed: clean discovery, clean
CR/pod poll, exit 0. The `-` call is now documented and used as a CR/pod
disappearance check only; a new sweep_residual runs first, excludes each chain's
imported claims BY NAME as expected survivors, and its result is aggregated
before anything is printed or exited.

Sourcing the library no longer installs a trap or creates a temp dir: verify_
teardown and sweep_residual are subshell functions owning their own temp dir and
traps, so a caller's EXIT handler survives, nothing leaks on return, and a later
check cannot inherit a deleted stderr dir. An interrupt exits 130/143, which is
above every verdict value, so both verdict switches gained an explicit catch-all
rather than ending silently.

Also: the inventory abort note said status "stays UNRESOLVED"; the implementation
writes no certificate at all.

Verified under dash and bash, with and without set -eu: 11 prior scenarios still
pass, plus 11 new — grep/sort failures in discovery, all-empty labels as
legitimate, residual sweep with a leaked PVC vs an imported survivor, forbidden
residual read, unreadable survivors list, and the signature case where the CR/pod
check alone verdicts 0 while the residual sweep correctly verdicts 1.

Co-authored-by: omnigent <noreply@omnigent.ai>

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remaining comments which cannot be posted as a review comment to avoid GitHub Rate Limit

vale

⚠️ [vale] <AgenticWriting.STE-SentenceLength-Description> reported by reviewdog 🐶
Descriptive sentence has 30 words. ASD-STE100 allows 25. Split it.

Deleting an orphaned SeiNode is the one cleanup with a paved road. The rest of what a workspace PR leaves behind needs its own handling, so nothing in the [what a workspace-repo PR removes](#what-a-workspace-repo-pr-removes) list is left with no next step:


⚠️ [vale] <AgenticWriting.STE-Passive> reported by reviewdog 🐶
Passive voice: 'is left'. ASD-STE100 asks for the active voice.

Deleting an orphaned SeiNode is the one cleanup with a paved road. The rest of what a workspace PR leaves behind needs its own handling, so nothing in the [what a workspace-repo PR removes](#what-a-workspace-repo-pr-removes) list is left with no next step:


⚠️ [vale] <AgenticWriting.STE-SentenceLength-Description> reported by reviewdog 🐶
Descriptive sentence has 26 words. ASD-STE100 allows 25. Split it.

**Every namespace-scoped command above names its namespace, and every destructive one names its context.** An unqualified `delete` deletes wherever the shell happens to point, and that is not a typo you can retry — it is a delete in the wrong place.


⚠️ [vale] <AgenticWriting.STE-SentenceLength-Description> reported by reviewdog 🐶
Descriptive sentence has 43 words. ASD-STE100 allows 25. Split it.

**No `seictl` alternative for network or node deletion belongs here, guarded or otherwise.** A `kubectl config current-context` check reads mutable state rather than pinning the config the delete then consumes, so it leaves a window between the check and the call — and it buys nothing, because `kubectl --context harbor delete` does the same deletion against the same CR with no window at all.


⚠️ [vale] <AgenticWriting.STE-SentenceLength-Description> reported by reviewdog 🐶
Descriptive sentence has 31 words. ASD-STE100 allows 25. Split it.

Should some genuinely `seictl`-only destructive verb ever need documenting, the pin belongs on the invocation: hand it a kubeconfig that contains the harbor cluster and nothing else, via `--kubeconfig <harbor-only-file>` (a documented `seictl` flag). A file that cannot name another cluster cannot select one. Do not substitute a current-context check.


⚠️ [vale] <AgenticWriting.STE-SentenceLength-Description> reported by reviewdog 🐶
Descriptive sentence has 39 words. ASD-STE100 allows 25. Split it.

Before deleting anything imperatively, prove the object is **not** in the workspace repo. A search that fails must never read as "no manifest found" — `grep` exits 1 for no match and 2 or more for an error, and an unreadable or stale clone produces the same empty output as a genuinely absent manifest.


⚠️ [vale] <AgenticWriting.STE-SentenceLength-Description> reported by reviewdog 🐶
Descriptive sentence has 47 words. ASD-STE100 allows 25. Split it.

**Only exit status 0 from this block authorizes a direct delete.** Every branch used to end in a successful `echo`, so the block's own status was 0 whatever it found — a scripted caller could not tell "not in git" from "the search failed", which is the same class of defect as counting lines without reading an exit status. `1` routes to the PR path; `2` means the question was never answered.


⚠️ [vale] <AgenticWriting.STE-Passive> reported by reviewdog 🐶
Passive voice: 'is gone'. ASD-STE100 asks for the active voice.

- **The removal PR merged while a SeiNetwork still read `Retain`.** The cascade is gone and no patch restores it. Do not re-apply the SeiNetwork to "reattach" the children — a fresh network under the same chain-id wedges at height 0 on the burned genesis artifacts. Go straight to [Find and clean up already-leaked resources](#find-and-clean-up-already-leaked-resources).


⚠️ [vale] <AgenticWriting.STE-SentenceLength-Description> reported by reviewdog 🐶
Descriptive sentence has 27 words. ASD-STE100 allows 25. Split it.

- **The removal PR merged while a SeiNetwork still read `Retain`.** The cascade is gone and no patch restores it. Do not re-apply the SeiNetwork to "reattach" the children — a fresh network under the same chain-id wedges at height 0 on the burned genesis artifacts. Go straight to [Find and clean up already-leaked resources](#find-and-clean-up-already-leaked-resources).


⚠️ [vale] <AgenticWriting.STE-Passive> reported by reviewdog 🐶
Passive voice: 'is ********'. ASD-STE100 asks for the active voice.

- **`kustomization <alias>` is `NotFound` in `eng-<alias>`.** The engineer's Flux wiring is missing, so no workspace-repo merge reconciles at all. Surface to the platform team; do not create the Kustomization.


⚠️ [vale] <AgenticWriting.STE-SentenceLength-Description> reported by reviewdog 🐶
Descriptive sentence has 27 words. ASD-STE100 allows 25. Split it.

- **`lastAppliedRevision` does not reach the merge commit within two reconcile intervals (~10 min).** Read the Ready condition's message (`cluster-inspection-recipes.md` recipe #8). A render error in `engineers/<alias>/kustomization.yaml` — most often a `resources:` entry pointing at the dir that was just removed — blocks every later apply in the namespace, not only this teardown.


⚠️ [vale] <AgenticWriting.STE-Passive> reported by reviewdog 🐶
Passive voice: 'is fixed'. ASD-STE100 asks for the active voice.

- **A verification read returned `UNVERIFIED`.** The API call failed, so the teardown state is unknown. Report it as unknown — never as verified-gone, and never as still-present. Re-run once the access problem is fixed; a teardown with an unverified check is not a finished teardown.


Claim names come from the **pods' own `spec.volumes[].persistentVolumeClaim.claimName`**, not from a guessed naming rule — the controller owns how it names a generated claim, and a rule inferred here would desync the moment it changes.

**A node with no pod resolves to nothing, and that is the leak case, not a nuisance.** The controller reconciles each SeiNode into a StatefulSet (`seinode-crd.md`), so a node whose StatefulSet has no pod — scaled down, unschedulable, evicted — still has its PVC and its EBS volume. The old version of this inventory dropped that node's claim silently and the teardown then verified clean. Check 1 makes the gap executable: the node lands in `unresolved-nodes.txt`, the script exits non-zero, and **no certificate is written at all** — the `status` file simply does not exist, which `read_inventory` reports as `UNVERIFIED`. The verifier reads that file, and also reads `unresolved-nodes.txt` directly, so either one alone is enough to fail the run.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ [vale] <AgenticWriting.STE-SentenceLength-Description> reported by reviewdog 🐶
Descriptive sentence has 28 words. ASD-STE100 allows 25. Split it.


Claim names come from the **pods' own `spec.volumes[].persistentVolumeClaim.claimName`**, not from a guessed naming rule — the controller owns how it names a generated claim, and a rule inferred here would desync the moment it changes.

**A node with no pod resolves to nothing, and that is the leak case, not a nuisance.** The controller reconciles each SeiNode into a StatefulSet (`seinode-crd.md`), so a node whose StatefulSet has no pod — scaled down, unschedulable, evicted — still has its PVC and its EBS volume. The old version of this inventory dropped that node's claim silently and the teardown then verified clean. Check 1 makes the gap executable: the node lands in `unresolved-nodes.txt`, the script exits non-zero, and **no certificate is written at all** — the `status` file simply does not exist, which `read_inventory` reports as `UNVERIFIED`. The verifier reads that file, and also reads `unresolved-nodes.txt` directly, so either one alone is enough to fail the run.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ [vale] <AgenticWriting.STE-SentenceLength-Description> reported by reviewdog 🐶
Descriptive sentence has 31 words. ASD-STE100 allows 25. Split it.


Claim names come from the **pods' own `spec.volumes[].persistentVolumeClaim.claimName`**, not from a guessed naming rule — the controller owns how it names a generated claim, and a rule inferred here would desync the moment it changes.

**A node with no pod resolves to nothing, and that is the leak case, not a nuisance.** The controller reconciles each SeiNode into a StatefulSet (`seinode-crd.md`), so a node whose StatefulSet has no pod — scaled down, unschedulable, evicted — still has its PVC and its EBS volume. The old version of this inventory dropped that node's claim silently and the teardown then verified clean. Check 1 makes the gap executable: the node lands in `unresolved-nodes.txt`, the script exits non-zero, and **no certificate is written at all** — the `status` file simply does not exist, which `read_inventory` reports as `UNVERIFIED`. The verifier reads that file, and also reads `unresolved-nodes.txt` directly, so either one alone is enough to fail the run.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ [vale] <AgenticWriting.STE-Passive> reported by reviewdog 🐶
Passive voice: 'is written'. ASD-STE100 asks for the active voice.

@seidroid seidroid Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Findings on the changed lines. The verdict and the summary are in this tool's comment on this pull request.

4. Open the PR, merge, then verify **every chain, retaining the worst result**. One `VERDICT` spans the whole namespace, so a clean second chain cannot cover an unverified first one:

```sh
# Same shell as step 1 — the library is already sourced and $VERDICT already

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

suggestion — Steps 4-5 still inherit $chains and $VERDICT from step 1 across opening a PR, review, and a merge, so a fresh shell is the ordinary case. sweep_residual now keeps that from hiding a leak — a leftover claim or a lingering SeiNode still fails the run — so this is no longer blocking. What survives is narrower: with an empty $chains the per-chain loop runs zero times, expect_present never asserts the imported claims survived, and the summary at line 417 still prints 'every chain verified'. Re-derive $chains at the top of step 4 from the ./teardown-inventory-* directories on disk (treating an empty result as UNVERIFIED unless a discovery that succeeded in this shell says otherwise), and the assumption goes away.

6. **Commit + push** — branch `feat/eng-<alias>-teardown-<task>`. Message: `feat(eng/<alias>): tear down <task> — chain-id=<chain-id>`.
7. **Open the PR** — title `feat(eng/<alias>): tear down <task>`. Body names the chain-id, every CR that goes away, and the `deletionPolicy` value the SeiNetwork now carries. Surface the URL and halt for the merge.
8. **After merge — reconcile the workspace Kustomization** — `flux --context harbor reconcile kustomization <alias> -n eng-<alias> --with-source`, then compare `.status.lastAppliedRevision` to the merge SHA. Reconciling `flux-system` here verifies the wrong repo (see Post-merge reconciliation).
9. **Poll until the resources disappear, and distinguish three outcomes** — a reconcile only says Flux issued the deletes. Deletion is asynchronous and finalizers hold objects in `Terminating` while the controller releases PVCs, so poll on a budget (~5 min) rather than asserting once. Report exactly one of **`GONE`** (the API answered and matched nothing), **`PRESENT`** (objects remain at the deadline), or **`UNVERIFIED`** (the API call failed). **A failed read is never a pass** — a `Forbidden` or dropped connection returns zero lines, so a check that counts lines without reading `kubectl`'s exit status claims success precisely when it cannot see the cluster. Use `poll_gone` / `expect_present` / `record` from `references/cluster-inspection-recipes.md` recipe #9 — one implementation, called, never re-typed. **Aggregate the outcomes**: a bare call discards its return code, so an early `UNVERIFIED` followed by a clean later read leaves the whole check exiting 0, which is the original bug on the exit-code path. Poll the chain's controller-managed claims **by name** from the step-2 inventory, and assert the imported ones separately — a namespace-wide PVC sweep also matches imported claims and other chains', so a correct teardown reports `PRESENT`.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

suggestion — This still points the agent at poll_gone / expect_present / record, but recipe #9 now makes verify_teardown and sweep_residual the entry points, with 'Do not copy pieces' written into the block. The always-loaded file names the parts the library asks callers not to assemble themselves. Point step 9 at verify_teardown, and mention sweep_residual for the namespace case.

- **`<chain-tag>` exceeds the 22-char budget** when the `-{a,b}-rpc-<k>` suffix is added. Surface the overflow and ask the engineer to pick a shorter tag.
- **CR name collision on either side.** Halt before render; surface the existing object's age + labels.
- **One network reaches `Ready` while the other reaches `Failed`.** The comparison is invalid. Surface the failed side's `.status.plan.failedTaskDetail.error`. The half-teardown is two coordinated edits, **both required** — Flux refuses to apply a kustomization with a missing resource:
- **First**, land `deletionPolicy: Delete` in the surviving side's SeiNetwork manifest if it reads `Retain`, and let it reconcile before the removal merges (`teardown.md` — a live patch gets reverted). The failed side needs the same read: a network that never reached `Ready` may still have generated validators to orphan.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

suggestion — The gate names the wrong side. It lands deletionPolicy: Delete in the surviving side's manifest, but the two steps below git rm the failed side's chain-<a-or-b>/ and the closing line says Flux prunes the SeiNetwork on the failed side. The network whose policy decides whether its validators are orphaned is the one being removed, so the gate belongs there; 'the failed side needs the same read' leaves the load-bearing case as a secondary check. The verb is correct — only the side needs swapping.

@seidroid seidroid Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

review found nothing blocking.

@seidroid
seidroid Bot dismissed stale reviews from themself September 9, 2026 23:12

Superseded: the latest review found nothing blocking in this change.

@seidroid

seidroid Bot commented Sep 9, 2026

Copy link
Copy Markdown

What changed

The pull request moved to merge commit bc0c799; the diff is 1355 lines, and the delta from the revision I reviewed last touches only cluster-inspection-recipes.md and teardown.md (+184/−40).

My blocker is closed, though not by the fix I proposed. The namespace procedure still inherits $chains and $VERDICT across the merge wait, but a new sweep_residual now runs at the end of every namespace teardown: it lists PVCs, Jobs and ConfigMaps in the namespace, excludes the imported claims by name, and returns 1 for anything left over. That closes the path I described — a fresh shell with an empty $chains now still runs the unlabelled CR/pod check and the residual sweep, so a leaked claim or a lingering SeiNode fails the run instead of passing it, and a legitimately imported claim produces a false alarm rather than a false clear. The discovery step's transformations are now checked too (grep exit 1 distinguished from a real failure, sort status read), so the || true that could have manufactured an empty chain list is gone. Two smaller hardenings came with it: the library no longer installs a source-time EXIT trap or a shared temp dir — verify_teardown and sweep_residual are subshell functions owning their own mktemp -d and traps, with INT/TERM re-raising 130/143 and a catch-all verdict branch so an interrupt cannot end the run silently — and the orphan sweep separates the API read from the jq parse, so a failed read no longer reads as "no orphans". The remaining consequence of the shell assumption is narrower than blocking and is listed below.

Nothing new blocks.

1. Blocking

None.

2. Non-blocking

  • references/teardown.md:380 still assumes steps 4-5 run in the same shell as step 1, across a PR merge wait. In a fresh shell $chains is empty, so the per-chain loop runs zero times and the summary at :417 prints "every chain verified" having verified none; the residual sweep keeps that from hiding a leak, but expect_present is skipped, so an imported claim that was wrongly deleted goes unreported, and the message overstates what ran. Re-deriving $chains from the ./teardown-inventory-* directories at the top of step 4 would remove the assumption.
  • SKILL.md:272 still names poll_gone / expect_present / record while recipe chore(agents): add Output Discipline section to all agents #9 now makes verify_teardown and sweep_residual the entry points and says not to copy pieces of it.
  • references/teardown.md:284 and :344 instruct . ./verify-lib.sh, which does not exist in references/; the reader must create it from the fenced block, which is the copy step the consolidation exists to remove.
  • SKILL.md:265 inventories with a bare kubectl get pvc -n eng-<alias> and produces none of the named-claim files step 9 consumes.
  • sweep_residual treats every ConfigMap except kube-root-ca.crt as a residual, so any other platform- or mesh-injected ConfigMap in eng-<alias> reports as leftover.
  • references/teardown.md:409 truncates ./expected-survivors.txt with : > without checking that write, though the later appends into it are checked.
  • The inventory's node-to-claim attribution still depends on a pod's first owner reference name equalling the SeiNode name; it fails safe, but if the controller's StatefulSet naming differs, inventory.sh exits 2 for every chain and the documented teardown cannot complete.
  • references/comparative-bench.md:429 gates the surviving side while the steps it prefixes remove the failed side, which is the side whose policy decides whether validators are orphaned.
  • references/teardown.md:3 still states the governing rule as "patch spec.deletionPolicy to Delete", which :67 closes off for any Flux-owned SeiNetwork.
  • references/teardown.md:593 prescribes kubectl --context harbor delete seinetwork|seinode, but SKILL.md:381's confirmation taxonomy gates only seictl network|node delete.
  • references/teardown.md:26 asserts "exactly three" immutability rules while the PR body says four and seinetwork-crd.md:38 names two.
  • "Render new chains with Delete from the start" (references/teardown.md:81-90) is still the only preventive step and still sits in the teardown-time file; the spinup procedure is untouched.
  • The Delete enum literal remains uncited in this repo, and the PR body's "inferred rather than verified" list still omits it.
  • references/teardown.md:578 asserts gp3-10k-750, gp3, and gp3-archive and their reclaim policies with none of the unverified marking the file gives its other inferences.
  • The orphan check's follow-up at references/teardown.md:470 still runs kubectl get seinetwork <seinetwork-label-value> against the - placeholder the jq emits when the sei.io/seinetwork label is absent.

3. Summary

The residual sweep is the right answer to the blocker I raised: rather than patch the shell-continuity assumption, it added the check that can contradict "the namespace is empty" — leftover claims, Jobs and ConfigMaps, with imported claims excluded by name — so the verdict no longer rests on a per-chain loop having run. With the discovery transformations now status-checked, the library's traps confined to subshells, and the orphan sweep's read separated from its parse, I have no blocking findings left; what remains are accuracy and consistency notes, chiefly that the namespace summary line still claims every chain was verified when the loop may have run zero times, and that SKILL.md still points at the sub-helpers rather than the entry points. Approving with those. Codex contributed no reading this dispatch — it reported reading no diff — so nothing was carried over or dropped from it here. The repository has no REVIEW.md on main.

seidroid review · decision approve · session 9e5542c4a82248afa23681c6eda59b44 · turn resp_claude_0ec6a439d7a236daa6e32b76554b3ac4 · item 5b65dfec44c950c5a2f1f719a2d02716

Findings: 0 blocking | 15 non-blocking | 3 posted inline | 1 pre-existing

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

There are 4 total unresolved issues (including 3 from previous reviews).

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 604b772. Configure here.

# An interrupt exits 130/143, which is above every verdict value. Without
# this branch the case matches nothing and the run ends silently.
*) printf 'ABORTED %s %s — interrupted or unexpected status %s; treat as unverified\n' \
"$vt_ns" "$vt_sel" "$VT_WORST" ;;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

EXIT trap zeros verifier status

High Severity

Converting verify_teardown and sweep_residual to subshell functions means return now exits a subshell and runs trap 'rm -rf …' EXIT. On dash (the documented target), that cleanup’s status replaces the function’s, so PRESENT or UNVERIFIED becomes 0. Callers then record a pass and report the teardown verified while objects or a failed read remain.

Additional Locations (1)
Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit 604b772. Configure here.

…stale survivor list

sweep_residual compared only the bare name against the survivors list, with no
kind check, so an imported PVC named `data` also exempted job/data and
configmap/data — all three kinds reported CLEAR at verdict 0. That is the same
full-identity defect already fixed in expect_present, in code that should have
reused the hardened matching rather than re-inventing it. Every exemption is now
kind-scoped and compares the full `<kind>/<name>` identity that `-o name`
prints, matching the shape the kube-root-ca.crt exemption already had.

The namespace step built its survivors list with an unchecked `: > file`. In
bash without set -e a failed redirection on that builtin leaves the OLD file
intact and execution continues, so a previous run's list could authorise this
run's exemptions: stale list readable, zero labelled chains, concat loop runs
zero times, and a leaked PVC named `data` is exempted while the namespace
reports success. The list is now built in a fresh per-namespace directory with
every step checked; on failure the run records UNVERIFIED and passes `-` so no
exemptions apply, rather than falling back to whatever was on disk. A missing
per-chain imported-claims file is now its own explicit UNVERIFIED branch.

Narrow the completion message to the kinds actually swept, and extend the sweep
to `service` and `cronjob`: a leftover type=LoadBalancer Service bills with no
pod running and the per-tenant ResourceQuota caps load balancers. Platform-owned
objects stay out of scope — a workspace PR never owned them.

Also add the ABORTED catch-all to the single-chain caller, so an interrupt that
exits through the trap does not end nonzero in silence.

Verified under dash and bash, with and without set -eu — including bash WITHOUT
errexit, where the stale-list defect lives: imported pvc `data` alongside
job/data and configmap/data now verdicts 1; a leftover Service verdicts 1; a
stale list with an unwritable target verdicts 2 and grants no exemption; zero
chains with a fresh empty list leaves a pvc unexempted; a missing imported-claims
file verdicts 2; a real imported list still exempts correctly. All 22 prior
scenarios still pass.

Co-authored-by: omnigent <noreply@omnigent.ai>

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remaining comments which cannot be posted as a review comment to avoid GitHub Rate Limit

vale

⚠️ [vale] <AgenticWriting.STE-Passive> reported by reviewdog 🐶
Passive voice: 'is gone'. ASD-STE100 asks for the active voice.

| Volume → PV → PVC whose claim is gone, PV `Released` | **Unresolved candidate.** Platform review required. Report the PV, PVC name, and reclaim policy; do not act on it here. |


⚠️ [vale] <AgenticWriting.STE-Passive> reported by reviewdog 🐶
Passive voice: 'is told'. ASD-STE100 asks for the active voice.

Only the first two rows are verdicts. Every other row is an escalation, and the platform team is told which row it came from.


⚠️ [vale] <AgenticWriting.STE-SentenceLength-Description> reported by reviewdog 🐶
Descriptive sentence has 28 words. ASD-STE100 allows 25. Split it.

`kubectl get pv` is cluster-scoped and the per-engineer Role is namespaced, so `Forbidden` at hop 1 is the **normal** case for an engineer — an unresolved result, not a clean one. When it happens, hand the volume IDs to the platform team and let them walk the chain; do not substitute the tag data for the walk.


⚠️ [vale] <AgenticWriting.STE-Passive> reported by reviewdog 🐶
Passive voice: 'is namespaced'. ASD-STE100 asks for the active voice.

`kubectl get pv` is cluster-scoped and the per-engineer Role is namespaced, so `Forbidden` at hop 1 is the **normal** case for an engineer — an unresolved result, not a clean one. When it happens, hand the volume IDs to the platform team and let them walk the chain; do not substitute the tag data for the walk.


⚠️ [vale] <AgenticWriting.STE-SentenceLength-Description> reported by reviewdog 🐶
Descriptive sentence has 26 words. ASD-STE100 allows 25. Split it.

`kubectl get pv` is cluster-scoped and the per-engineer Role is namespaced, so `Forbidden` at hop 1 is the **normal** case for an engineer — an unresolved result, not a clean one. When it happens, hand the volume IDs to the platform team and let them walk the chain; do not substitute the tag data for the walk.


⚠️ [vale] <AgenticWriting.STE-Passive> reported by reviewdog 🐶
Passive voice: 'is ******'. ASD-STE100 asks for the active voice.

The node's finalizer deletes its data PVC, and the `Delete` reclaim policy on `gp3-10k-750` (validators) and `gp3` (default) releases the EBS volume. `gp3-archive` is `Retain` by design — a volume on that class stays after its PVC goes, and its removal is an AWS-side decision, not a mistake to correct here.


⚠️ [vale] <AgenticWriting.STE-SentenceLength-Description> reported by reviewdog 🐶
Descriptive sentence has 27 words. ASD-STE100 allows 25. Split it.

The node's finalizer deletes its data PVC, and the `Delete` reclaim policy on `gp3-10k-750` (validators) and `gp3` (default) releases the EBS volume. `gp3-archive` is `Retain` by design — a volume on that class stays after its PVC goes, and its removal is an AWS-side decision, not a mistake to correct here.


⚠️ [vale] <AgenticWriting.STE-SentenceLength-Description> reported by reviewdog 🐶
Descriptive sentence has 33 words. ASD-STE100 allows 25. Split it.

**Never delete an EBS volume from this skill.** Even a volume the ownership walk resolved to a dead PVC goes to the platform team: `ec2:DeleteVolume` is outside the engineer's policy, the walk can be wrong, and an EBS delete is unrecoverable. Hand over the volume IDs, sizes, creation times, and the walk's verdict per volume — including every `UNRESOLVED` one, labelled as unresolved. Escalate through `#harbor-onboarding`. Do not report the cleanup as complete while any ID is outstanding.


⚠️ [vale] <AgenticWriting.STE-SentenceLength-Description> reported by reviewdog 🐶
Descriptive sentence has 30 words. ASD-STE100 allows 25. Split it.

Deleting an orphaned SeiNode is the one cleanup with a paved road. The rest of what a workspace PR leaves behind needs its own handling, so nothing in the [what a workspace-repo PR removes](#what-a-workspace-repo-pr-removes) list is left with no next step:


⚠️ [vale] <AgenticWriting.STE-Passive> reported by reviewdog 🐶
Passive voice: 'is left'. ASD-STE100 asks for the active voice.

Deleting an orphaned SeiNode is the one cleanup with a paved road. The rest of what a workspace PR leaves behind needs its own handling, so nothing in the [what a workspace-repo PR removes](#what-a-workspace-repo-pr-removes) list is left with no next step:


⚠️ [vale] <AgenticWriting.STE-SentenceLength-Description> reported by reviewdog 🐶
Descriptive sentence has 26 words. ASD-STE100 allows 25. Split it.

**Every namespace-scoped command above names its namespace, and every destructive one names its context.** An unqualified `delete` deletes wherever the shell happens to point, and that is not a typo you can retry — it is a delete in the wrong place.


⚠️ [vale] <AgenticWriting.STE-SentenceLength-Description> reported by reviewdog 🐶
Descriptive sentence has 43 words. ASD-STE100 allows 25. Split it.

**No `seictl` alternative for network or node deletion belongs here, guarded or otherwise.** A `kubectl config current-context` check reads mutable state rather than pinning the config the delete then consumes, so it leaves a window between the check and the call — and it buys nothing, because `kubectl --context harbor delete` does the same deletion against the same CR with no window at all.


⚠️ [vale] <AgenticWriting.STE-SentenceLength-Description> reported by reviewdog 🐶
Descriptive sentence has 31 words. ASD-STE100 allows 25. Split it.

Should some genuinely `seictl`-only destructive verb ever need documenting, the pin belongs on the invocation: hand it a kubeconfig that contains the harbor cluster and nothing else, via `--kubeconfig <harbor-only-file>` (a documented `seictl` flag). A file that cannot name another cluster cannot select one. Do not substitute a current-context check.


⚠️ [vale] <AgenticWriting.STE-SentenceLength-Description> reported by reviewdog 🐶
Descriptive sentence has 39 words. ASD-STE100 allows 25. Split it.

Before deleting anything imperatively, prove the object is **not** in the workspace repo. A search that fails must never read as "no manifest found" — `grep` exits 1 for no match and 2 or more for an error, and an unreadable or stale clone produces the same empty output as a genuinely absent manifest.


⚠️ [vale] <AgenticWriting.STE-SentenceLength-Description> reported by reviewdog 🐶
Descriptive sentence has 47 words. ASD-STE100 allows 25. Split it.

**Only exit status 0 from this block authorizes a direct delete.** Every branch used to end in a successful `echo`, so the block's own status was 0 whatever it found — a scripted caller could not tell "not in git" from "the search failed", which is the same class of defect as counting lines without reading an exit status. `1` routes to the PR path; `2` means the question was never answered.


⚠️ [vale] <AgenticWriting.STE-Passive> reported by reviewdog 🐶
Passive voice: 'is gone'. ASD-STE100 asks for the active voice.

- **The removal PR merged while a SeiNetwork still read `Retain`.** The cascade is gone and no patch restores it. Do not re-apply the SeiNetwork to "reattach" the children — a fresh network under the same chain-id wedges at height 0 on the burned genesis artifacts. Go straight to [Find and clean up already-leaked resources](#find-and-clean-up-already-leaked-resources).


⚠️ [vale] <AgenticWriting.STE-SentenceLength-Description> reported by reviewdog 🐶
Descriptive sentence has 27 words. ASD-STE100 allows 25. Split it.

- **The removal PR merged while a SeiNetwork still read `Retain`.** The cascade is gone and no patch restores it. Do not re-apply the SeiNetwork to "reattach" the children — a fresh network under the same chain-id wedges at height 0 on the burned genesis artifacts. Go straight to [Find and clean up already-leaked resources](#find-and-clean-up-already-leaked-resources).


⚠️ [vale] <AgenticWriting.STE-Passive> reported by reviewdog 🐶
Passive voice: 'is ********'. ASD-STE100 asks for the active voice.

- **`kustomization <alias>` is `NotFound` in `eng-<alias>`.** The engineer's Flux wiring is missing, so no workspace-repo merge reconciles at all. Surface to the platform team; do not create the Kustomization.


⚠️ [vale] <AgenticWriting.STE-SentenceLength-Description> reported by reviewdog 🐶
Descriptive sentence has 27 words. ASD-STE100 allows 25. Split it.

- **`lastAppliedRevision` does not reach the merge commit within two reconcile intervals (~10 min).** Read the Ready condition's message (`cluster-inspection-recipes.md` recipe #8). A render error in `engineers/<alias>/kustomization.yaml` — most often a `resources:` entry pointing at the dir that was just removed — blocks every later apply in the namespace, not only this teardown.


⚠️ [vale] <AgenticWriting.STE-Passive> reported by reviewdog 🐶
Passive voice: 'is fixed'. ASD-STE100 asks for the active voice.

- **A verification read returned `UNVERIFIED`.** The API call failed, so the teardown state is unknown. Report it as unknown — never as verified-gone, and never as still-present. Re-run once the access problem is fixed; a teardown with an unverified check is not a finished teardown.

exit "$VERDICT"
```

That is the whole verification step. **If you find yourself writing a `poll_gone` line in this file, stop** — a second copy of the orchestration is how this exact defect survived four review rounds, reappearing one layer up each time: duplicated poll bodies, then the caller chain, then the arguments feeding the callers, then a canonical caller that bypassed the fixed library entirely while the paragraph above it said not to re-implement.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ [vale] <AgenticWriting.STE-SentenceLength-Description> reported by reviewdog 🐶
Descriptive sentence has 65 words. ASD-STE100 allows 25. Split it.


That is the whole verification step. **If you find yourself writing a `poll_gone` line in this file, stop** — a second copy of the orchestration is how this exact defect survived four review rounds, reappearing one layer up each time: duplicated poll bodies, then the caller chain, then the arguments feeding the callers, then a canonical caller that bypassed the fixed library entirely while the paragraph above it said not to re-implement.

The empty-list cases are handled inside the function, as code rather than as advice here: an empty `managed-claims.txt` takes a `NOTE` branch instead of calling `poll_gone` with no names, because `kubectl get persistentvolumeclaim` with no arguments lists the whole namespace — the sweep this design exists to avoid.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ [vale] <AgenticWriting.STE-SentenceLength-Description> reported by reviewdog 🐶
Descriptive sentence has 42 words. ASD-STE100 allows 25. Split it.


That is the whole verification step. **If you find yourself writing a `poll_gone` line in this file, stop** — a second copy of the orchestration is how this exact defect survived four review rounds, reappearing one layer up each time: duplicated poll bodies, then the caller chain, then the arguments feeding the callers, then a canonical caller that bypassed the fixed library entirely while the paragraph above it said not to re-implement.

The empty-list cases are handled inside the function, as code rather than as advice here: an empty `managed-claims.txt` takes a `NOTE` branch instead of calling `poll_gone` with no names, because `kubectl get persistentvolumeclaim` with no arguments lists the whole namespace — the sweep this design exists to avoid.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ [vale] <AgenticWriting.STE-Passive> reported by reviewdog 🐶
Passive voice: 'are handled'. ASD-STE100 asks for the active voice.


### A stuck `Terminating` object is a real signal

If the poll runs out with objects still present, do not report the teardown as done and do not force the objects away. Read why they are held:

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ [vale] <AgenticWriting.STE-Passive> reported by reviewdog 🐶
Passive voice: 'are held'. ASD-STE100 asks for the active voice.


## Procedure: empty or remove my namespace

"Destroy my namespace" means one of two very different things. Ask which before acting.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ [vale] <write-good.Weasel> reported by reviewdog 🐶
'very' is a weasel word!

rm -f "$derr"
```

Run `inventory.sh` **once per chain-id**, each writing its own `./teardown-inventory-<chain-id>`. A namespace usually holds more than one chain, and a single sweep cannot tell one chain's controller-managed claim from another's. Any chain whose `inventory.sh` exits non-zero writes no certificate, and its verification then reports `UNVERIFIED` — emptying a namespace on an incomplete inventory is how a leak becomes invisible. Claims no chain attributes land in each run's `unattributed-claims.txt`; take those to the leak sweep in step 5, not to a delete.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ [vale] <write-good.Weasel> reported by reviewdog 🐶
'usually' is a weasel word!

rm -f "$derr"
```

Run `inventory.sh` **once per chain-id**, each writing its own `./teardown-inventory-<chain-id>`. A namespace usually holds more than one chain, and a single sweep cannot tell one chain's controller-managed claim from another's. Any chain whose `inventory.sh` exits non-zero writes no certificate, and its verification then reports `UNVERIFIED` — emptying a namespace on an incomplete inventory is how a leak becomes invisible. Claims no chain attributes land in each run's `unattributed-claims.txt`; take those to the leak sweep in step 5, not to a delete.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ [vale] <AgenticWriting.STE-SentenceLength-Description> reported by reviewdog 🐶
Descriptive sentence has 27 words. ASD-STE100 allows 25. Split it.

record "$rc"
```

**Do not print a verdict yet.** The check that can contradict "the namespace is empty" has not run: a namespace whose only leftover is a leaked PVC passes everything above — successful discovery, a clean CR/pod poll, exit 0.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ [vale] <AgenticWriting.STE-SentenceLength-Description> reported by reviewdog 🐶
Descriptive sentence has 33 words. ASD-STE100 allows 25. Split it.

```

**Do not print a verdict yet.** The check that can contradict "the namespace is empty" has not run: a namespace whose only leftover is a leaked PVC passes everything above — successful discovery, a clean CR/pod poll, exit 0.
5. **Sweep the residuals, then decide.** Build the expected-survivors list from every chain's imported claims, sweep what is left, and only then print and exit:

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ [vale] <AgenticWriting.STE-Passive> reported by reviewdog 🐶
Passive voice: 'is left'. ASD-STE100 asks for the active voice.

exit "$VERDICT"
```

With `surv='-'` the sweep runs with **no** exemptions, so genuinely imported claims are reported as residuals. That is the safe direction and it is not the verdict: `record 2` already fired, and `2` dominates the `1` a residual would raise.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ [vale] <AgenticWriting.STE-Passive> reported by reviewdog 🐶
Passive voice: 'are reported'. ASD-STE100 asks for the active voice.

…tovers

The sweep filters by Kind, not by ownership: its only ownership-aware
exemptions are the imported claims and kube-root-ca.crt, so a
platform-owned or controller-created Service, CronJob or ConfigMap is
reported too. Saying it "deliberately leaves platform-owned objects
alone" claimed more certainty than the implementation delivers — the
same species of defect this PR exists to correct. Residuals are now
candidates requiring ownership resolution, and the unswept kinds are
described as unswept rather than as recognised and excluded.

Caveat the ownership evidence for Service and CronJob: the platform base
listing neither is what the repository declares, not what the cluster
holds.

Add the ABORTED catch-all to the recipe's single-chain caller, matching
both callers in teardown.md.

Record one line for the next editor beside the shared helpers: reuse the
matching and status handling next door rather than re-deriving it. That
was this file's recurring defect, not any single bug.

No behaviour change — the verification library is byte-identical to the
reviewed revision.

Co-authored-by: omnigent <noreply@omnigent.ai>

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remaining comments which cannot be posted as a review comment to avoid GitHub Rate Limit

vale

⚠️ [vale] <AgenticWriting.STE-Passive> reported by reviewdog 🐶
Passive voice: 'is ******'. ASD-STE100 asks for the active voice.

The node's finalizer deletes its data PVC, and the `Delete` reclaim policy on `gp3-10k-750` (validators) and `gp3` (default) releases the EBS volume. `gp3-archive` is `Retain` by design — a volume on that class stays after its PVC goes, and its removal is an AWS-side decision, not a mistake to correct here.


⚠️ [vale] <AgenticWriting.STE-SentenceLength-Description> reported by reviewdog 🐶
Descriptive sentence has 27 words. ASD-STE100 allows 25. Split it.

The node's finalizer deletes its data PVC, and the `Delete` reclaim policy on `gp3-10k-750` (validators) and `gp3` (default) releases the EBS volume. `gp3-archive` is `Retain` by design — a volume on that class stays after its PVC goes, and its removal is an AWS-side decision, not a mistake to correct here.


⚠️ [vale] <AgenticWriting.STE-SentenceLength-Description> reported by reviewdog 🐶
Descriptive sentence has 33 words. ASD-STE100 allows 25. Split it.

**Never delete an EBS volume from this skill.** Even a volume the ownership walk resolved to a dead PVC goes to the platform team: `ec2:DeleteVolume` is outside the engineer's policy, the walk can be wrong, and an EBS delete is unrecoverable. Hand over the volume IDs, sizes, creation times, and the walk's verdict per volume — including every `UNRESOLVED` one, labelled as unresolved. Escalate through `#harbor-onboarding`. Do not report the cleanup as complete while any ID is outstanding.


⚠️ [vale] <AgenticWriting.STE-SentenceLength-Description> reported by reviewdog 🐶
Descriptive sentence has 30 words. ASD-STE100 allows 25. Split it.

Deleting an orphaned SeiNode is the one cleanup with a paved road. The rest of what a workspace PR leaves behind needs its own handling, so nothing in the [what a workspace-repo PR removes](#what-a-workspace-repo-pr-removes) list is left with no next step:


⚠️ [vale] <AgenticWriting.STE-Passive> reported by reviewdog 🐶
Passive voice: 'is left'. ASD-STE100 asks for the active voice.

Deleting an orphaned SeiNode is the one cleanup with a paved road. The rest of what a workspace PR leaves behind needs its own handling, so nothing in the [what a workspace-repo PR removes](#what-a-workspace-repo-pr-removes) list is left with no next step:


⚠️ [vale] <AgenticWriting.STE-SentenceLength-Description> reported by reviewdog 🐶
Descriptive sentence has 26 words. ASD-STE100 allows 25. Split it.

**Every namespace-scoped command above names its namespace, and every destructive one names its context.** An unqualified `delete` deletes wherever the shell happens to point, and that is not a typo you can retry — it is a delete in the wrong place.


⚠️ [vale] <AgenticWriting.STE-SentenceLength-Description> reported by reviewdog 🐶
Descriptive sentence has 43 words. ASD-STE100 allows 25. Split it.

**No `seictl` alternative for network or node deletion belongs here, guarded or otherwise.** A `kubectl config current-context` check reads mutable state rather than pinning the config the delete then consumes, so it leaves a window between the check and the call — and it buys nothing, because `kubectl --context harbor delete` does the same deletion against the same CR with no window at all.


⚠️ [vale] <AgenticWriting.STE-SentenceLength-Description> reported by reviewdog 🐶
Descriptive sentence has 31 words. ASD-STE100 allows 25. Split it.

Should some genuinely `seictl`-only destructive verb ever need documenting, the pin belongs on the invocation: hand it a kubeconfig that contains the harbor cluster and nothing else, via `--kubeconfig <harbor-only-file>` (a documented `seictl` flag). A file that cannot name another cluster cannot select one. Do not substitute a current-context check.


⚠️ [vale] <AgenticWriting.STE-SentenceLength-Description> reported by reviewdog 🐶
Descriptive sentence has 39 words. ASD-STE100 allows 25. Split it.

Before deleting anything imperatively, prove the object is **not** in the workspace repo. A search that fails must never read as "no manifest found" — `grep` exits 1 for no match and 2 or more for an error, and an unreadable or stale clone produces the same empty output as a genuinely absent manifest.


⚠️ [vale] <AgenticWriting.STE-SentenceLength-Description> reported by reviewdog 🐶
Descriptive sentence has 47 words. ASD-STE100 allows 25. Split it.

**Only exit status 0 from this block authorizes a direct delete.** Every branch used to end in a successful `echo`, so the block's own status was 0 whatever it found — a scripted caller could not tell "not in git" from "the search failed", which is the same class of defect as counting lines without reading an exit status. `1` routes to the PR path; `2` means the question was never answered.


⚠️ [vale] <AgenticWriting.STE-Passive> reported by reviewdog 🐶
Passive voice: 'is gone'. ASD-STE100 asks for the active voice.

- **The removal PR merged while a SeiNetwork still read `Retain`.** The cascade is gone and no patch restores it. Do not re-apply the SeiNetwork to "reattach" the children — a fresh network under the same chain-id wedges at height 0 on the burned genesis artifacts. Go straight to [Find and clean up already-leaked resources](#find-and-clean-up-already-leaked-resources).


⚠️ [vale] <AgenticWriting.STE-SentenceLength-Description> reported by reviewdog 🐶
Descriptive sentence has 27 words. ASD-STE100 allows 25. Split it.

- **The removal PR merged while a SeiNetwork still read `Retain`.** The cascade is gone and no patch restores it. Do not re-apply the SeiNetwork to "reattach" the children — a fresh network under the same chain-id wedges at height 0 on the burned genesis artifacts. Go straight to [Find and clean up already-leaked resources](#find-and-clean-up-already-leaked-resources).


⚠️ [vale] <AgenticWriting.STE-Passive> reported by reviewdog 🐶
Passive voice: 'is ********'. ASD-STE100 asks for the active voice.

- **`kustomization <alias>` is `NotFound` in `eng-<alias>`.** The engineer's Flux wiring is missing, so no workspace-repo merge reconciles at all. Surface to the platform team; do not create the Kustomization.


⚠️ [vale] <AgenticWriting.STE-SentenceLength-Description> reported by reviewdog 🐶
Descriptive sentence has 27 words. ASD-STE100 allows 25. Split it.

- **`lastAppliedRevision` does not reach the merge commit within two reconcile intervals (~10 min).** Read the Ready condition's message (`cluster-inspection-recipes.md` recipe #8). A render error in `engineers/<alias>/kustomization.yaml` — most often a `resources:` entry pointing at the dir that was just removed — blocks every later apply in the namespace, not only this teardown.


⚠️ [vale] <AgenticWriting.STE-Passive> reported by reviewdog 🐶
Passive voice: 'is fixed'. ASD-STE100 asks for the active voice.

- **A verification read returned `UNVERIFIED`.** The API call failed, so the teardown state is unknown. Report it as unknown — never as verified-gone, and never as still-present. Re-run once the access problem is fixed; a teardown with an unverified check is not a finished teardown.

2. **Identities are matched, not counted.** A count says how many lines came back, not whether the resources you asked about are the ones that came back.
3. **Every command that can fail runs inside a condition.** Under `set -e` a bare `out=$(kubectl …)` terminates the shell at the assignment — before classification, and before the caller records anything.

**Adding a helper here? Reuse the matching and status handling beside it rather than re-deriving them.** Through this file's review history the recurring defect was never one bug — it was new code re-deriving logic already hardened next door, and arriving without the fix. `sweep_residual` was written comparing bare names in the same review round that `expect_present` was corrected to compare full identities, one function away.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ [vale] <AgenticWriting.STE-SentenceLength-Description> reported by reviewdog 🐶
Descriptive sentence has 29 words. ASD-STE100 allows 25. Split it.

2. **Identities are matched, not counted.** A count says how many lines came back, not whether the resources you asked about are the ones that came back.
3. **Every command that can fail runs inside a condition.** Under `set -e` a bare `out=$(kubectl …)` terminates the shell at the assignment — before classification, and before the caller records anything.

**Adding a helper here? Reuse the matching and status handling beside it rather than re-deriving them.** Through this file's review history the recurring defect was never one bug — it was new code re-deriving logic already hardened next door, and arriving without the fix. `sweep_residual` was written comparing bare names in the same review round that `expect_present` was corrected to compare full identities, one function away.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ [vale] <AgenticWriting.STE-Passive> reported by reviewdog 🐶
Passive voice: 'was written'. ASD-STE100 asks for the active voice.

2. **Identities are matched, not counted.** A count says how many lines came back, not whether the resources you asked about are the ones that came back.
3. **Every command that can fail runs inside a condition.** Under `set -e` a bare `out=$(kubectl …)` terminates the shell at the assignment — before classification, and before the caller records anything.

**Adding a helper here? Reuse the matching and status handling beside it rather than re-deriving them.** Through this file's review history the recurring defect was never one bug — it was new code re-deriving logic already hardened next door, and arriving without the fix. `sweep_residual` was written comparing bare names in the same review round that `expect_present` was corrected to compare full identities, one function away.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ [vale] <AgenticWriting.STE-Passive> reported by reviewdog 🐶
Passive voice: 'was corrected'. ASD-STE100 asks for the active voice.

-o custom-columns='NAME:.metadata.name,PHASE:.status.phase,DELETED:.metadata.deletionTimestamp,FINALIZERS:.metadata.finalizers'
```

**Zero PVCs is the wrong expectation, and a namespace-wide PVC poll is the wrong check.** The SeiNode finalizer deliberately skips an imported PVC, so imported claims survive by design and other chains' claims are none of this teardown's business. Both make a namespace sweep report `PRESENT` after a correct teardown. `verify_teardown` therefore polls the target's controller-managed claims by name and asserts the imported ones separately, from the lists `teardown.md` inventory step 2 captured **before** the SeiNodes were deleted — afterwards nothing in the cluster still says which claims were which.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ [vale] <AgenticWriting.STE-SentenceLength-Description> reported by reviewdog 🐶
Descriptive sentence has 39 words. ASD-STE100 allows 25. Split it.

-o custom-columns='NAME:.metadata.name,PHASE:.status.phase,DELETED:.metadata.deletionTimestamp,FINALIZERS:.metadata.finalizers'
```

**Zero PVCs is the wrong expectation, and a namespace-wide PVC poll is the wrong check.** The SeiNode finalizer deliberately skips an imported PVC, so imported claims survive by design and other chains' claims are none of this teardown's business. Both make a namespace sweep report `PRESENT` after a correct teardown. `verify_teardown` therefore polls the target's controller-managed claims by name and asserts the imported ones separately, from the lists `teardown.md` inventory step 2 captured **before** the SeiNodes were deleted — afterwards nothing in the cluster still says which claims were which.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ [vale] <AgenticWriting.STE-Passive> reported by reviewdog 🐶
Passive voice: 'were deleted'. ASD-STE100 asks for the active voice.

|---|---|
| Volume → PV → PVC → pod → a SeiNode confirmed orphaned by the signature | Reclaimable. Delete the **SeiNode**, not the volume — see below. |
| Volume → PV → PVC → pod → a live, wanted workload | **Not garbage.** Leave it. `available` only meant the workload was stopped. |
| Volume → PV → PVC whose claim is gone, PV `Released` | **Unresolved candidate.** Platform review required. Report the PV, PVC name, and reclaim policy; do not act on it here. |

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ [vale] <AgenticWriting.STE-Passive> reported by reviewdog 🐶
Passive voice: 'is gone'. ASD-STE100 asks for the active voice.

| `claimRef` names a namespace other than `eng-<alias>` | **Out of scope.** Another tenant's disk. Escalate; do not inspect. |
| Any hop returned `Forbidden`, errored, or found nothing | **UNRESOLVED.** Escalate as unresolved. Never as confirmed-safe. |

Only the first two rows are verdicts. Every other row is an escalation, and the platform team is told which row it came from.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ [vale] <AgenticWriting.STE-Passive> reported by reviewdog 🐶
Passive voice: 'is told'. ASD-STE100 asks for the active voice.


Only the first two rows are verdicts. Every other row is an escalation, and the platform team is told which row it came from.

`kubectl get pv` is cluster-scoped and the per-engineer Role is namespaced, so `Forbidden` at hop 1 is the **normal** case for an engineer — an unresolved result, not a clean one. When it happens, hand the volume IDs to the platform team and let them walk the chain; do not substitute the tag data for the walk.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ [vale] <AgenticWriting.STE-SentenceLength-Description> reported by reviewdog 🐶
Descriptive sentence has 28 words. ASD-STE100 allows 25. Split it.


Only the first two rows are verdicts. Every other row is an escalation, and the platform team is told which row it came from.

`kubectl get pv` is cluster-scoped and the per-engineer Role is namespaced, so `Forbidden` at hop 1 is the **normal** case for an engineer — an unresolved result, not a clean one. When it happens, hand the volume IDs to the platform team and let them walk the chain; do not substitute the tag data for the walk.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ [vale] <AgenticWriting.STE-Passive> reported by reviewdog 🐶
Passive voice: 'is namespaced'. ASD-STE100 asks for the active voice.


Only the first two rows are verdicts. Every other row is an escalation, and the platform team is told which row it came from.

`kubectl get pv` is cluster-scoped and the per-engineer Role is namespaced, so `Forbidden` at hop 1 is the **normal** case for an engineer — an unresolved result, not a clean one. When it happens, hand the volume IDs to the platform team and let them walk the chain; do not substitute the tag data for the walk.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ [vale] <AgenticWriting.STE-SentenceLength-Description> reported by reviewdog 🐶
Descriptive sentence has 26 words. ASD-STE100 allows 25. Split it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant