diff --git a/docs/suite-environment-retest.md b/docs/suite-environment-retest.md
new file mode 100644
index 0000000..c0f2c09
--- /dev/null
+++ b/docs/suite-environment-retest.md
@@ -0,0 +1,45 @@
+# Retest after test-suite and environment fixes
+
+This change is the test-side attribution for [backlog #3173](https://github.com/rustfs/backlog/issues/3173), based on the merged auto-testing #144. It does not mean the server 503, the namespace lock timeout, or the performance anomaly has been fixed.
+
+| Original failure | Fix behavior | Retest acceptance |
+|---|---|---|
+| TBL-101 import failed | Prepare the whole `scripts/table-catalog` from the RustFS checkout that matches the candidate. Record the source SHA and file hashes. Check the local modules first, then create the venv | End-to-end reads and writes actually succeed. A missing module must be reported as a tool-preparation error |
+| M-1x2EC1 started a non-participating node | Pass the participating node count explicitly. Submit a non-blocking start only for those nodes and wait until they are active | Keep every topology, EC, and fault probe. An unexecuted probe cannot stand in for a pass |
+| Rebalance activity criterion timed out | Increase the Rebalance activity fixture on its own and check capacity first. Keep the `Started` assertion and the non-zero progress assertion. An early completion explicitly reports that the activity window was insufficient | stop/resume, live I/O, and operation guards must run inside the activity window. The completed state still requires non-zero migration progress and a data check |
+| Pool-expansion load filled a single disk | Sample capacity every 2 seconds. Protect single-disk utilization and remaining bytes. Reserve in-flight objects and the observed growth, and shorten the stop wait | Pass only when the target is reached. Stopping early because of capacity protection is an environment-capacity failure, and MPU cleanup still runs |
+| A performance round contained service errors but was counted in the summary | Error logs fail that round. The raw metrics are retained for diagnosis. Each round records whether the Report appeared, the exit code, and the error-line count | `rounds.tsv` and the Markdown include every selected round. Timeouts and rounds that did not run do not count as a pass |
+| AppleDouble input and a gap in the syntax report | Exclude `._*`. Run `bash -n` independently on every real script and save the per-file result | If any real script fails, the overall result is non-zero. An empty selection does not pass either |
+
+## Preparing Table source
+
+```bash
+python3 scripts/prepare_table_smoke.py prepare \
+ --source-root /path/to/rustfs-checkout \
+ --destination /path/to/new-run/table-catalog \
+ --expected-sha FULL_CANDIDATE_SOURCE_SHA
+```
+
+`--destination` must be a new directory, so old-version modules are not mixed in. A chained workflow uses the source SHA from the manifest. A standalone workflow can pass `source_ref` to name the source that matches the artifact. When it is omitted, the release tag or that checkout's main is used, and `source-manifest.json` retains the actual SHA.
+
+The Table workflow uploads `table-catalog/source-manifest.json` together with the report, the case table, and the logs. It retains the actual smoke source SHA and file hashes so they can still be traced after the runner's temporary directory is cleaned up.
+
+## Capacity configuration
+
+The Rebalance source case explicitly requires non-zero migration progress during the activity phase. The previous run, at 32 MiB per object, finished before the first cycle saved progress, so it cannot pass. The dedicated activity fixture now defaults to 128 MiB per object (128 objects by default, of which 3 boundary objects are smaller). It is configured with `RUSTFS_ACTIVE_REBALANCE_FIXTURE_BYTES` / `--active-rebalance-fixture-bytes` and recorded in the run manifest. Before writing, capacity is checked against the upper bound of EC usage and at least 1 GiB free on each disk. Other Decommission activity fixtures keep their original size. A faster machine may still need a larger fixture. When the window is insufficient, the case fails rather than relaxing the assertion.
+
+Capacity sampling is separate from the Rebalance / Decommission `--poll-interval`. The default sample interval is 2 seconds, and a single direct API request is capped at 3 seconds. There is no long retry or proxy probe during the activity write. The proxy is still probed before and after the write, and errors are still retained.
+
+The reserve is the minimum free bytes per disk (512 MiB by default) plus `object size × concurrency`, plus the observed cluster growth rate multiplied by the sampling, request, and stop windows. Cluster growth is a conservative upper bound on single-disk growth. Uniform data placement is not assumed. An empty, invalid, or changed capacity inventory stops further writes.
+
+`RUSTFS_CAPACITY_POLL_INTERVAL`, `RUSTFS_CAPACITY_SAMPLE_TIMEOUT`, `RUSTFS_CAPACITY_MIN_FREE_BYTES`, `RUSTFS_CAPACITY_MAX_DISK_PERCENT`, `RUSTFS_WARP_STOP_GRACE`, and `RUSTFS_WARP_OBJ_SIZE` are configurable. A small-capacity environment should first lower concurrency or the target utilization according to free disk space. The sidecar `.capacity.jsonl` retains the inputs, the margin, and the stop reason for every decision. This is a load protection. It cannot guarantee that capacity is never exhausted under an unpredictable write burst.
+
+An MPU list error retains the HTTP status, `x-rustfs-error-cause`, the request ID, and retry-after when the server returns it. The full AWS debug output and response cookies are not collected.
+
+## Result boundaries
+
+The overall performance-round timeout still covers preparation, load, and teardown. `timeout_before_report` / `timeout_after_report` only say whether a Warp Report appeared. They cannot infer which internal Warp stage was stuck. Do not use a disk-full event during pool expansion to explain an earlier performance timeout.
+
+`--parse-only` writes separate `rounds.reparsed.tsv`, `summary.reparsed.tsv`, `summary.reparsed.md`, and `master.reparsed.log` files, and keeps the original execution ledger, summary, and logs. Reparsing cannot recover elapsed time or the original run configuration. The `NOT_RUN / aborted` rows and the elapsed time in the original `rounds.tsv` are the execution evidence. The performance workflow uploads both the original execution ledger and the reparsed artifacts. When step 6 has no original summary, it shows the reparsed summary. A reparse failure still returns non-zero.
+
+A retest must retain the product source SHA, the CI artifact identity, the binary SHA-256, the test commit SHA, the actual configuration, and the complete per-item results. VM writes and the final recovery use the same shared lock. After recovery, check the four nodes, the 24 data mounts, and the object verification.
diff --git a/docs/suite-environment-retest.zh-CN.md b/docs/suite-environment-retest.zh-CN.md
deleted file mode 100644
index 0540362..0000000
--- a/docs/suite-environment-retest.zh-CN.md
+++ /dev/null
@@ -1,45 +0,0 @@
-# 测试套件与环境问题修复后的复测
-
-本次改动对应 [backlog #3173](https://github.com/rustfs/backlog/issues/3173) 的测试侧归因,基于已合并的 auto-testing #144。它不代表修复了服务端 503、命名空间锁超时或性能异常。
-
-| 原失败 | 修复行为 | 复测验收 |
-|---|---|---|
-| TBL-101 导入失败 | 从候选对应的 RustFS checkout 准备整个 `scripts/table-catalog`,记录源码 SHA 与文件哈希;先检查本地模块,再创建 venv | 端到端实际读写成功;缺模块必须报告工具准备错误 |
-| M-1x2EC1 启动非参与节点 | 显式传递参与节点数量,只对这些节点提交非阻塞启动并等待 active | 完整保留各拓扑 / EC / 故障探针,不能用未执行代替通过 |
-| Rebalance 活动判据超时 | 单独增大 Rebalance 活动夹具并先校验容量,保留 `Started` 与非零进度断言;提前完成明确报告活动窗口不足 | stop/resume、live I/O、operation guards 必须在活动窗口执行;完成态仍要求非零迁移进度及数据校验 |
-| 扩池负载写满单盘 | 2 秒容量采样,单盘利用率与剩余字节保护;预留在途对象和观测到的增长量,缩短停止等待 | 达到目标才通过;容量保护提前停止是环境容量失败,仍执行 MPU 清理 |
-| 性能轮次含服务错误却计入摘要 | 错误日志使该轮失败,原始指标保留作诊断;逐轮记录 Report 是否出现、退出码、错误行数 | `rounds.tsv` 和 Markdown 包含全部选定轮次;超时与未执行都不算通过 |
-| AppleDouble 输入 / 语法报告缺口 | 排除 `._*`,对每个真实脚本独立执行 `bash -n`,保存逐文件结果 | 任一真实脚本失败,整体返回非零;空选择也不通过 |
-
-## Table 源码准备
-
-```bash
-python3 scripts/prepare_table_smoke.py prepare \
- --source-root /path/to/rustfs-checkout \
- --destination /path/to/new-run/table-catalog \
- --expected-sha FULL_CANDIDATE_SOURCE_SHA
-```
-
-`--destination` 必须是新目录,避免混入旧版本模块。链式 workflow 使用 manifest 的 source SHA;独立 workflow 可通过 `source_ref` 指定与产物匹配的源码。未指定时使用 release tag 或当次 checkout 的 main,并在 `source-manifest.json` 保留实际 SHA。
-
-Table workflow 将 `table-catalog/source-manifest.json` 与报告、用例表和日志一起上传,保留实际 smoke 源码 SHA 与文件哈希,供 runner 临时目录清理后追溯。
-
-## 容量配置
-
-Rebalance 源用例明确要求活动阶段已有非零迁移进度。上一轮每对象 32 MiB 的任务在首次周期保存进度前完成,因此不能通过。专用活动夹具默认改为每对象 128 MiB(默认 128 对象,其中 3 个边界对象更小),通过 `RUSTFS_ACTIVE_REBALANCE_FIXTURE_BYTES` / `--active-rebalance-fixture-bytes` 配置,并记录在 run manifest。写入前按 EC 占用上界与每盘至少 1 GiB 的剩余余量检查容量。其他 Decommission 活动夹具保留原大小;更快的机器仍可能需要增加夹具,窗口不足时用例失败而不是放松断言。
-
-容量采样与 Rebalance / Decommission 的 `--poll-interval` 分开。默认每 2 秒采样,一次直接 API 请求上限 3 秒,活动写入期间不做长重试或代理探测。代理仍在写入前后探测,错误继续保留。
-
-预留量为每盘最少剩余字节(默认 512 MiB)加 `对象大小 × 并发数`,再加观测到的集群增长速率乘以采样、请求与停止窗口。集群增长作为单盘增长的保守上界,不假设数据均匀。容量库存为空、无效或发生变化会停止继续写入。
-
-可配置 `RUSTFS_CAPACITY_POLL_INTERVAL`、`RUSTFS_CAPACITY_SAMPLE_TIMEOUT`、`RUSTFS_CAPACITY_MIN_FREE_BYTES`、`RUSTFS_CAPACITY_MAX_DISK_PERCENT`、`RUSTFS_WARP_STOP_GRACE`、`RUSTFS_WARP_OBJ_SIZE`。小容量环境应先按磁盘余量降低并发或目标利用率。旁路 `.capacity.jsonl` 保留每次判断的输入、余量与停止原因;这是一项负载保护,无法保证在不可预测的突发写入下绝不耗尽容量。
-
-MPU 列表错误会保留 HTTP status、`x-rustfs-error-cause`、request ID 与 retry-after(服务端返回时);不采集完整 AWS debug 输出或响应 Cookie。
-
-## 结果边界
-
-性能轮次总超时仍覆盖准备、负载和收尾。`timeout_before_report` / `timeout_after_report` 仅说明是否出现 Warp Report,不能推断 Warp 内部卡在具体阶段。不要把扩池发生的磁盘满用来解释此前的性能超时。
-
-`--parse-only` 输出独立的 `rounds.reparsed.tsv`、`summary.reparsed.tsv`、`summary.reparsed.md` 与 `master.reparsed.log`,保留原执行台账、摘要及日志。重解析无法恢复耗时和原始运行配置;应以原 `rounds.tsv` 中的 `NOT_RUN / aborted` 及耗时为执行证据。性能 workflow 同时上传原执行台账与重解析产物。步骤 6 缺少原摘要时展示重解析摘要,重解析失败仍返回非零。
-
-复测需保留产品源码 SHA、CI 产物身份、二进制 SHA-256、测试提交 SHA、实际配置和完整逐条结果。VM 写操作及最终恢复使用同一个共享锁;恢复后核对四节点、24 个数据挂载与对象校验。
diff --git a/docs/validation-heal-cases.md b/docs/validation-heal-cases.md
index 72b29d7..1d184cd 100644
--- a/docs/validation-heal-cases.md
+++ b/docs/validation-heal-cases.md
@@ -9,20 +9,20 @@ Use [the operations reference](validation-heal-operations.md) before injecting a
Topologies: H1,H2. Branches: normal, deep.
-1. 用 DB 夹具生成完整 manifest;先对每个对象保存 S3 基线和四盘分片摘要,关闭 scanner/MRF/auto-heal。
-2. 依次执行 START object(scanMode=1、2),WAIT 原 token 直至终态;再 STATUS 同 token 两次。
-3. 对每个对象的 outcome 区分 Normal 的 metadata_healthy 与 Deep 的 verified_healthy;两者均要求 healed=0、unchanged>0,Normal 不得冒充 payload 完整性证明。比较目标/控制对象物理摘要和 S3 清单,重复请求不得新建版本。
+1. Generate a complete manifest with the DB fixture; first, for each object, save the S3 baseline and the four-disk shard digests, and disable scanner/MRF/auto-heal.
+2. In sequence, run START object (scanMode=1 and 2), WAIT the original token until the terminal state; then STATUS the same token twice.
+3. For each object's outcome, distinguish Normal's metadata_healthy from Deep's verified_healthy; both require healed=0 and unchanged>0, and Normal must not be passed off as proof of payload integrity. Compare the physical digests of the target and control objects with the S3 inventory, and a repeated request must not create a new version.
Operations: [SEED_DB](validation-heal-operations.md#SEED_DB) → [BASELINE](validation-heal-operations.md#BASELINE) → [START](validation-heal-operations.md#START) → [WAIT](validation-heal-operations.md#WAIT) → [STATUS](validation-heal-operations.md#STATUS) → [VERIFY](validation-heal-operations.md#VERIFY) → [PHYSICAL](validation-heal-operations.md#PHYSICAL)
Source assertions:
-- 对健康对象分别执行 normal/deep,再重复查询
-- 不修改用户数据或版本;Normal 只接受 `metadata_healthy`,Deep 才要求 `verified_healthy`,healthy 与 repaired 分开计数,不能把检查盘数当修复盘数。
+- On healthy objects, run normal and deep separately, then query again.
+- Do not modify user data or versions. Normal accepts only `metadata_healthy`, and only Deep requires `verified_healthy`. Count healthy and repaired separately, and do not treat the number of disks checked as the number of disks repaired.
-先保存失败现场及物理/状态证据,停止本例写入与在途任务,再按 RECOVER 撤销本例故障。并发版本已变化时禁止盲目恢复旧 xl.meta;恢复不能充作产品修复。未恢复环境标 DIRTY,保留本例备份和原失败结果。
+First save the failure scene and the physical and state evidence, stop this case's writes and in-flight tasks, then undo this case's fault per RECOVER. When a concurrent version has already changed, do not blindly restore the old xl.meta; recovery must not serve as a product fix. Mark an unrestored environment DIRTY, and keep this case's backup and the original failure result.
-逐项满足设计公共契约、该分支步骤及附加断言后人工评审。缺资源为 BLOCKED,窗口未命中为 NOT_OBSERVED,产品错误为 FAIL,工具/恢复错误为 INFRA_ERROR;未运行不记 PASS。
+Manually review after each item satisfies the design's common contract, that branch's steps, and the additional assertions. Missing resources are BLOCKED, a window miss is NOT_OBSERVED, a product error is FAIL, and a tool or recovery error is INFRA_ERROR; a case that did not run is not recorded as PASS.
@@ -30,21 +30,21 @@ Source assertions:
Topologies: H1,H2. Branches: root, bucket, prefix, object, pool-set.
-1. 用 DS 在两个桶的 a/、ab/、b/ 写入不同内容;另在未选 set 放控制对象;逐对象预先记录实际 pool/set。
-2. 分别隔离选中对象、相邻前缀及另一桶/未选 set 的一个 part。每个 selector 使用新夹具和故障 ID。
-3. 对象请求路径到完整 key;prefix 路径到 a/ 且 recursive=true;bucket 路径只含桶;root 为 /;pool/set 写入 body。START 后 WAIT。
-4. 先 PHYSICAL:选择范围内全部重建、范围外控制故障仍缺失;然后 GET 验证,避免验证请求提前修好控制故障。root 必须包含整个专用服务清单。
+1. Use DS to write different content at a/, ab/, and b/ in two buckets; also place a control object in an unselected set; record the actual pool/set of each object in advance.
+2. Separately isolate one part of the selected object, of an adjacent prefix, and of another bucket or an unselected set. Each selector uses a new fixture and a new fault ID.
+3. The object request path is the full key; the prefix path is a/ with recursive=true; the bucket path contains only the bucket; root is /; pool/set is written in the body. WAIT after START.
+4. First PHYSICAL: everything inside the selected range is rebuilt, and control faults outside the range are still missing; then verify with GET, so the verification request does not repair the control fault early. root must include the entire dedicated-service manifest.
Operations: [SEED_DS](validation-heal-operations.md#SEED_DS) → [BASELINE](validation-heal-operations.md#BASELINE) → [FAULT](validation-heal-operations.md#FAULT) → [START](validation-heal-operations.md#START) → [WAIT](validation-heal-operations.md#WAIT) → [PHYSICAL](validation-heal-operations.md#PHYSICAL) → [VERIFY](validation-heal-operations.md#VERIFY)
Source assertions:
-- 分别以 root、bucket、prefix、object、pool/set 选择器启动
-- 目标范围实际遍历;相邻前缀、其他桶及未选 set 的故障保持原状;按对象清单而非目录数量验收。
+- Start separately with the root, bucket, prefix, object, and pool/set selectors.
+- The target range is actually traversed. Faults on an adjacent prefix, on other buckets, and on an unselected set stay as they were. Accept by the object manifest rather than by the directory count.
-先保存失败现场及物理/状态证据,停止本例写入与在途任务,再按 RECOVER 撤销本例故障。并发版本已变化时禁止盲目恢复旧 xl.meta;恢复不能充作产品修复。未恢复环境标 DIRTY,保留本例备份和原失败结果。
+First save the failure scene and the physical and state evidence, stop this case's writes and in-flight tasks, then undo this case's fault per RECOVER. When a concurrent version has already changed, do not blindly restore the old xl.meta; recovery must not serve as a product fix. Mark an unrestored environment DIRTY, and keep this case's backup and the original failure result.
-逐项满足设计公共契约、该分支步骤及附加断言后人工评审。缺资源为 BLOCKED,窗口未命中为 NOT_OBSERVED,产品错误为 FAIL,工具/恢复错误为 INFRA_ERROR;未运行不记 PASS。
+Manually review after each item satisfies the design's common contract, that branch's steps, and the additional assertions. Missing resources are BLOCKED, a window miss is NOT_OBSERVED, a product error is FAIL, and a tool or recovery error is INFRA_ERROR; a case that did not run is not recorded as PASS.
@@ -52,21 +52,21 @@ Source assertions:
Topologies: H2. Branches: coordinator-to-peer, coordinator-to-stable-endpoint.
-1. 在 H2 同一跨节点 set 写入 DB;保存协调者、目标、稳定入口分别对应的端点,不把同一节点的两个别名当跨节点。
-2. 隔离目标节点 part;从 n0 START,记录原始 clientToken;通过 n1 和 stable 对同一路径、同 token 执行 STATUS。
-3. 按同一任务身份对齐三端状态,最多等候冻结的状态收敛预算;终态及核心计数必须一致,404/unknown 不能视为结束。
-4. WAIT 后先物理证明目标重建,再从所有 S3 入口 VERIFY;保存每个请求实际连接的节点日志。
+1. Write DB on the same cross-node set in H2; save the endpoints for the coordinator, the target, and the stable entry respectively, and do not treat two aliases of the same node as cross-node.
+2. Isolate the target node's part; START from n0 and record the original clientToken; run STATUS through n1 and stable on the same path with the same token.
+3. Align the three endpoints' state by the same task identity, waiting at most the frozen state-convergence budget; the terminal state and the core counts must match, and 404/unknown cannot be treated as finished.
+4. After WAIT, first physically prove that the target was rebuilt, then VERIFY from every S3 entry; save the log of the node each request actually connected to.
Operations: [SEED_DB](validation-heal-operations.md#SEED_DB) → [FAULT](validation-heal-operations.md#FAULT) → [START](validation-heal-operations.md#START) → [STATUS](validation-heal-operations.md#STATUS) → [WAIT](validation-heal-operations.md#WAIT) → [PHYSICAL](validation-heal-operations.md#PHYSICAL) → [VERIFY](validation-heal-operations.md#VERIFY)
Source assertions:
-- 从节点甲启动,经乙及稳定入口按原 token 查询
-- 任务身份、scope、终态及核心计数有界收敛;状态缺失/未知不能解释为完成。
+- Start from node A, and query with the original token through node B and the stable entry.
+- Task identity, scope, terminal state, and core counts converge within bounds. A missing or unknown state must not be interpreted as completion.
-先保存失败现场及物理/状态证据,停止本例写入与在途任务,再按 RECOVER 撤销本例故障。并发版本已变化时禁止盲目恢复旧 xl.meta;恢复不能充作产品修复。未恢复环境标 DIRTY,保留本例备份和原失败结果。
+First save the failure scene and the physical and state evidence, stop this case's writes and in-flight tasks, then undo this case's fault per RECOVER. When a concurrent version has already changed, do not blindly restore the old xl.meta; recovery must not serve as a product fix. Mark an unrestored environment DIRTY, and keep this case's backup and the original failure result.
-逐项满足设计公共契约、该分支步骤及附加断言后人工评审。缺资源为 BLOCKED,窗口未命中为 NOT_OBSERVED,产品错误为 FAIL,工具/恢复错误为 INFRA_ERROR;未运行不记 PASS。
+Manually review after each item satisfies the design's common contract, that branch's steps, and the additional assertions. Missing resources are BLOCKED, a window miss is NOT_OBSERVED, a product error is FAIL, and a tool or recovery error is INFRA_ERROR; a case that did not run is not recorded as PASS.
@@ -74,21 +74,21 @@ Source assertions:
Topologies: H1,H2. Branches: missing-remove0-recreate0, missing-remove1-recreate0, bitflip-remove0-recreate1, bitflip-remove1-recreate1, versioned-missing, versioned-bitflip.
-1. 分别用 DB 与 DV 建夹具,冻结全部 versionId/marker;关闭后台来源,不在注入后 GET。
-2. 在原始 part 上做 missing/bitflip,每组独立新目录;保存故障后的 PHYSICAL 快照作为 dry-run 前基线。
-3. START body 同时传 dryRun=true、scanMode=2 和该实例的 remove/recreate;WAIT 仅接受 dry_run_observed,不能要求 dry-run 持有 incarnation。
-4. 前后所有分片/元数据/marker 字节与版本清单相同;没有 repaired ACK。完成非变更证明后恢复故障,再 VERIFY。
+1. Build fixtures with DB and DV respectively, and freeze every versionId/marker; disable background sources, and do not GET after injection.
+2. Apply missing/bitflip on the original part, with an independent new directory for each group; save the post-fault PHYSICAL snapshot as the baseline before the dry-run.
+3. The START body passes dryRun=true, scanMode=2, and that instance's remove/recreate together; WAIT accepts only dry_run_observed, and must not require the dry-run to hold an incarnation.
+4. All shard, metadata, and marker bytes and the version manifest are the same before and after; there is no repaired ACK. After the no-change proof is complete, clear the fault, then VERIFY.
Operations: [SEED_DB](validation-heal-operations.md#SEED_DB) → [SEED_DV](validation-heal-operations.md#SEED_DV) → [BASELINE](validation-heal-operations.md#BASELINE) → [FAULT](validation-heal-operations.md#FAULT) → [PHYSICAL](validation-heal-operations.md#PHYSICAL) → [START](validation-heal-operations.md#START) → [WAIT](validation-heal-operations.md#WAIT) → [RECOVER](validation-heal-operations.md#RECOVER) → [VERIFY](validation-heal-operations.md#VERIFY)
Source assertions:
-- 对缺片与损坏对象执行 dryRun,参数化 remove/recreate
-- 有明确 dry_run_observed 诊断;用户对象分片、xl.meta、版本与删除标记不变;后台修复干扰须排除。
+- Run dryRun on objects with missing shards and on corrupt objects, with remove/recreate parameterized.
+- There is an explicit dry_run_observed diagnosis. The user object's shards, xl.meta, versions, and delete markers stay unchanged. Interference from background repair must be ruled out.
-先保存失败现场及物理/状态证据,停止本例写入与在途任务,再按 RECOVER 撤销本例故障。并发版本已变化时禁止盲目恢复旧 xl.meta;恢复不能充作产品修复。未恢复环境标 DIRTY,保留本例备份和原失败结果。
+First save the failure scene and the physical and state evidence, stop this case's writes and in-flight tasks, then undo this case's fault per RECOVER. When a concurrent version has already changed, do not blindly restore the old xl.meta; recovery must not serve as a product fix. Mark an unrestored environment DIRTY, and keep this case's backup and the original failure result.
-逐项满足设计公共契约、该分支步骤及附加断言后人工评审。缺资源为 BLOCKED,窗口未命中为 NOT_OBSERVED,产品错误为 FAIL,工具/恢复错误为 INFRA_ERROR;未运行不记 PASS。
+Manually review after each item satisfies the design's common contract, that branch's steps, and the additional assertions. Missing resources are BLOCKED, a window miss is NOT_OBSERVED, a product error is FAIL, and a tool or recovery error is INFRA_ERROR; a case that did not run is not recorded as PASS.
@@ -96,21 +96,21 @@ Source assertions:
Topologies: H1,H2. Branches: same-scope, parent-child, disjoint, force-start.
-1. 用 DX 超过 128 项的外部分片夹具注入多对象缺片;预先从固定候选记录 overlap policy(复用 token 或冲突错误),禁止接受任意结果。
-2. START 递归范围,WINDOW 证明 0
@@ -118,21 +118,21 @@ Source assertions:
Topologies: H1,H2. Branches: stop-active, repeat-stop, old-status, old-stop-new-task.
-1. DX 批量缺片后 START;WINDOW 观察非零且未完成进度,再 STOP 原 token。
-2. 状态显示 aborted/cancelled 后仍观察物理写入静默窗口;停止受理成功不能替代在途退出。
-3. 新 START 得到不同 token,再对旧 token STATUS/STOP;新任务不应被旧请求终止,重复 stop 按冻结的 HTTP/错误码验收。
-4. 新任务完成后完整 VERIFY;记录取消项不能算 repaired,所有暂态错误保留原始 body。
+1. After the DX batch of missing shards, START; use WINDOW to observe nonzero, incomplete progress, then STOP the original token.
+2. After the status shows aborted/cancelled, still observe a quiescence window for physical writes; successful acceptance of the stop cannot substitute for in-flight exit.
+3. A new START yields a different token, then STATUS/STOP the old token; the new task must not be terminated by the old request, and a repeated stop is checked against the frozen HTTP status and error code.
+4. After the new task completes, run a full VERIFY; items recorded as cancelled must not count as repaired, and every transient error keeps its original body.
Operations: [SEED_DX](validation-heal-operations.md#SEED_DX) → [FAULT](validation-heal-operations.md#FAULT) → [START](validation-heal-operations.md#START) → [WINDOW](validation-heal-operations.md#WINDOW) → [STOP](validation-heal-operations.md#STOP) → [STATUS](validation-heal-operations.md#STATUS) → [WAIT](validation-heal-operations.md#WAIT) → [VERIFY](validation-heal-operations.md#VERIFY) → [PHYSICAL](validation-heal-operations.md#PHYSICAL)
Source assertions:
-- 非零修复进度时 stop,稳定后新建任务,再发旧 token 的查询/停止
-- 停止需等在途写结束并稳定;新任务身份独立,旧请求不得终止新任务;重复 stop 按公开契约处理。
+- Stop while repair progress is non-zero, create a new task after the state is stable, then send a query and a stop for the old token.
+- A stop must wait until in-flight writes finish and the state is stable. The new task's identity is independent, and an old request must not terminate the new task. A repeated stop is handled according to the public contract.
-先保存失败现场及物理/状态证据,停止本例写入与在途任务,再按 RECOVER 撤销本例故障。并发版本已变化时禁止盲目恢复旧 xl.meta;恢复不能充作产品修复。未恢复环境标 DIRTY,保留本例备份和原失败结果。
+First save the failure scene and the physical and state evidence, stop this case's writes and in-flight tasks, then undo this case's fault per RECOVER. When a concurrent version has already changed, do not blindly restore the old xl.meta; recovery must not serve as a product fix. Mark an unrestored environment DIRTY, and keep this case's backup and the original failure result.
-逐项满足设计公共契约、该分支步骤及附加断言后人工评审。缺资源为 BLOCKED,窗口未命中为 NOT_OBSERVED,产品错误为 FAIL,工具/恢复错误为 INFRA_ERROR;未运行不记 PASS。
+Manually review after each item satisfies the design's common contract, that branch's steps, and the additional assertions. Missing resources are BLOCKED, a window miss is NOT_OBSERVED, a product error is FAIL, and a tool or recovery error is INFRA_ERROR; a case that did not run is not recorded as PASS.
@@ -140,21 +140,21 @@ Source assertions:
Topologies: H1,H2. Branches: duplicate-query, invalid-bool, invalid-scan, conflicting-flags, invalid-selector, oversize-body.
-1. 建立健康 DB 和 PHYSICAL 基线;保存后台/任务状态及对象版本集合,用于检查负请求未意外启动任务。
-2. 逐个发送服务端负例:重复 clientToken;forceStart=wat;scanMode=99;forceStart=true&forceStop=true;pool=-1/set 越界;超过候选限制的 body。每个例固定预期 HTTP 及结构化错误码。
-3. 使用 ADMIN_RAW 的重复 --query 和原始 --body-file,不能只调用 rc 的参数校验;不使用超出专用 Heal 路由的请求。
-4. 全部响应无有效新任务受理,服务仍可用;PHYSICAL、版本、后台责任无改变;若返回 2xx 或错误无明确原因保留 FAIL。
+1. Establish a healthy DB and PHYSICAL baseline; save the background and task state and the object version set, to check that a negative request did not unexpectedly start a task.
+2. Send the server-side negative cases one by one: a repeated clientToken; forceStart=wat; scanMode=99; forceStart=true&forceStop=true; pool=-1 or set out of range; a body that exceeds the candidate limit. Each case has a fixed expected HTTP status and structured error code.
+3. Use ADMIN_RAW's repeated --query and the raw --body-file, and do not rely only on rc's parameter validation; do not use a request outside the dedicated Heal route.
+4. No response accepts a valid new task, and the service remains available; PHYSICAL, versions, and background responsibility are unchanged; if a 2xx is returned or an error has no clear cause, keep FAIL.
Operations: [SEED_DB](validation-heal-operations.md#SEED_DB) → [BASELINE](validation-heal-operations.md#BASELINE) → [ADMIN_RAW](validation-heal-operations.md#ADMIN_RAW) → [STATUS](validation-heal-operations.md#STATUS) → [PHYSICAL](validation-heal-operations.md#PHYSICAL) → [VERIFY](validation-heal-operations.md#VERIFY)
Source assertions:
-- 发送非法 selector、重复 query、非法布尔/scanMode、冲突 flags、超长 body
-- 服务端返回已核实的结构化错误;无误启动、范围扩大或持久化副作用;CLI 参数错误不能替代服务端负例。
+- Send an illegal selector, a duplicate query, an illegal boolean or scanMode, conflicting flags, and an oversized body.
+- The server returns a verified structured error. There is no mistaken start, no widened range, and no persisted side effect. A CLI parameter error cannot take the place of a server-side negative case.
-先保存失败现场及物理/状态证据,停止本例写入与在途任务,再按 RECOVER 撤销本例故障。并发版本已变化时禁止盲目恢复旧 xl.meta;恢复不能充作产品修复。未恢复环境标 DIRTY,保留本例备份和原失败结果。
+First save the failure scene and the physical and state evidence, stop this case's writes and in-flight tasks, then undo this case's fault per RECOVER. When a concurrent version has already changed, do not blindly restore the old xl.meta; recovery must not serve as a product fix. Mark an unrestored environment DIRTY, and keep this case's backup and the original failure result.
-逐项满足设计公共契约、该分支步骤及附加断言后人工评审。缺资源为 BLOCKED,窗口未命中为 NOT_OBSERVED,产品错误为 FAIL,工具/恢复错误为 INFRA_ERROR;未运行不记 PASS。
+Manually review after each item satisfies the design's common contract, that branch's steps, and the additional assertions. Missing resources are BLOCKED, a window miss is NOT_OBSERVED, a product error is FAIL, and a tool or recovery error is INFRA_ERROR; a case that did not run is not recorded as PASS.
@@ -162,21 +162,21 @@ Source assertions:
Topologies: H1,H2. Branches: anonymous, bad-signature, expired-signature, denied-user, least-privilege-user.
-1. 通过 IAM 手册在专用服务创建两个临时用户,分别没有和仅有 HealAdminAction;用受控凭据环境运行,先证明限制用户不能读未授权对象。
-2. 管理员创建待查询/停止任务;分别对 start/status/stop 发匿名、坏签名、有效签名但时间超期、无授权用户请求;过期请求由隔离客户端签名时间设置产生,不改服务器时间。
-3. 检查冻结的 AccessDenied/签名/时间错误;响应不暴露对象结果。最小授权用户完成受理/查询/停止,证明接口本身可工作。
-4. 移除临时用户和策略并验证凭据失效;所有原始证据先脱敏再归档,不把管理员 token 暴露给无关主体。
+1. Per the IAM manual, create two temporary users on the dedicated service, one with no HealAdminAction and one with only HealAdminAction; run in a controlled credential environment, and first prove that the restricted user cannot read an unauthorized object.
+2. The administrator creates a task to be queried and stopped; for start/status/stop, send an anonymous request, a bad signature, a valid signature whose time has expired, and an unauthorized-user request; the expired request is produced by the isolated client's signature-time setting, without changing the server time.
+3. Check the frozen AccessDenied, signature, and time errors; the response does not expose object results. The least-privilege user completes acceptance, query, and stop, proving the interface itself works.
+4. Remove the temporary users and policies and verify that the credentials are invalid; redact all raw evidence before archiving, and do not expose the administrator token to an unrelated principal.
Operations: [SEED_DB](validation-heal-operations.md#SEED_DB) → [IAM](validation-heal-operations.md#IAM) → [START](validation-heal-operations.md#START) → [ADMIN_RAW](validation-heal-operations.md#ADMIN_RAW) → [STATUS](validation-heal-operations.md#STATUS) → [STOP](validation-heal-operations.md#STOP) → [VERIFY](validation-heal-operations.md#VERIFY) → [RECOVER](validation-heal-operations.md#RECOVER)
Source assertions:
-- 匿名、坏签名、过期签名、无 HealAdminAction 用户与最小授权用户分别请求
-- 未授权 start/status/stop 均不能越权或泄漏对象信息;授权用户可完成规定操作,凭据不得写入证据。
+- Send the request separately as anonymous, with a bad signature, with an expired signature, as a user without HealAdminAction, and as a minimally authorized user.
+- Unauthorized start, status, and stop must not escalate privilege or leak object information. An authorized user can complete the specified operations. Credentials must not be written into the evidence.
-先保存失败现场及物理/状态证据,停止本例写入与在途任务,再按 RECOVER 撤销本例故障。并发版本已变化时禁止盲目恢复旧 xl.meta;恢复不能充作产品修复。未恢复环境标 DIRTY,保留本例备份和原失败结果。
+First save the failure scene and the physical and state evidence, stop this case's writes and in-flight tasks, then undo this case's fault per RECOVER. When a concurrent version has already changed, do not blindly restore the old xl.meta; recovery must not serve as a product fix. Mark an unrestored environment DIRTY, and keep this case's backup and the original failure result.
-逐项满足设计公共契约、该分支步骤及附加断言后人工评审。缺资源为 BLOCKED,窗口未命中为 NOT_OBSERVED,产品错误为 FAIL,工具/恢复错误为 INFRA_ERROR;未运行不记 PASS。
+Manually review after each item satisfies the design's common contract, that branch's steps, and the additional assertions. Missing resources are BLOCKED, a window miss is NOT_OBSERVED, a product error is FAIL, and a tool or recovery error is INFRA_ERROR; a case that did not run is not recorded as PASS.
@@ -184,21 +184,21 @@ Source assertions:
Topologies: H1,H2. Branches: overwrite-nolock, delete-nolock, native-lock-fence.
-1. DB/DV 准备同一目标并缺片;两客户端使用同 key,冻结每个 PUT/DELETE 的唯一 nonce 和 ACK ledger。
-2. 一端 START body nolock=true,另一端按 WORKLOAD 提交覆盖或精确 versionId 删除;记录确实交叠的运行区间。
-3. 按 ACK 因果顺序重建期望清单并 VERIFY;最后成功覆盖不回退、已删除版本不复活。
-4. 另外运行 NATIVE 的锁/提交屏障回归并保存精确测试名、源码 SHA、命中日志;黑盒竞态与 native-lock-fence 单独结果,不以偶然重叠证明强制锁时点。
+1. Use DB/DV to prepare the same target with a missing shard; the two clients use the same key, and freeze a unique nonce and the ACK ledger for each PUT/DELETE.
+2. One side sends a START body with nolock=true, and the other side submits an overwrite or an exact versionId delete per WORKLOAD; record the run interval that actually overlapped.
+3. Rebuild the expected manifest in ACK causal order and VERIFY; the last successful overwrite does not roll back, and a deleted version does not come back.
+4. Also run the NATIVE lock/commit-fence regression and save the exact test name, source SHA, and hit log; the black-box race and native-lock-fence are separate results, and an accidental overlap does not prove the time at which the lock is mandatory.
Operations: [SEED_DV](validation-heal-operations.md#SEED_DV) → [FAULT](validation-heal-operations.md#FAULT) → [WORKLOAD](validation-heal-operations.md#WORKLOAD) → [START](validation-heal-operations.md#START) → [WINDOW](validation-heal-operations.md#WINDOW) → [VERIFY](validation-heal-operations.md#VERIFY) → [NATIVE](validation-heal-operations.md#NATIVE)
Source assertions:
-- 通过签名 API 传 nolock=true,并与同对象提交制造确定性争用
-- 兼容字段不得跳过存储层对象锁;最后成功写入和已确认删除保持正确,原生屏障与真机竞争证据分开。
+- Pass nolock=true through the signed API, and create deterministic contention with a commit of the same object.
+- The compatibility field must not skip the storage-layer object lock. The last successful write and the confirmed deletes stay correct. Keep native-barrier evidence separate from live contention evidence.
-先保存失败现场及物理/状态证据,停止本例写入与在途任务,再按 RECOVER 撤销本例故障。并发版本已变化时禁止盲目恢复旧 xl.meta;恢复不能充作产品修复。未恢复环境标 DIRTY,保留本例备份和原失败结果。
+First save the failure scene and the physical and state evidence, stop this case's writes and in-flight tasks, then undo this case's fault per RECOVER. When a concurrent version has already changed, do not blindly restore the old xl.meta; recovery must not serve as a product fix. Mark an unrestored environment DIRTY, and keep this case's backup and the original failure result.
-逐项满足设计公共契约、该分支步骤及附加断言后人工评审。缺资源为 BLOCKED,窗口未命中为 NOT_OBSERVED,产品错误为 FAIL,工具/恢复错误为 INFRA_ERROR;未运行不记 PASS。
+Manually review after each item satisfies the design's common contract, that branch's steps, and the additional assertions. Missing resources are BLOCKED, a window miss is NOT_OBSERVED, a product error is FAIL, and a tool or recovery error is INFRA_ERROR; a case that did not run is not recorded as PASS.
@@ -206,10 +206,10 @@ Source assertions:
Topologies: H1,H2. Branches: recursive-false, recursive-true, recreate, remove-false, remove-true, update-parity, read-repair, selector-valid, selector-invalid.
-1. 将候选 HealOpts 支持表和危险参数副作用写入本批 contract.json;不根据 CLI help 推导 API 未暴露字段。
-2. 为每个 variant 使用新 DB/DV:固定范围、足够权威和控制对象;remove=true 只对经 C10 证明无活引用的本次孤儿执行。
-3. 逐一 START 并对比只改变一个字段的基线;recursive 检查子前缀,pool/set 检查实际目标身份,非法组合要求结构化拒绝且无副作用。
-4. WAIT、PHYSICAL、VERIFY 三者验收;updateParity/readRepair 若候选不支持必须显示明确拒绝,不可悄悄按默认修复算覆盖。
+1. Write the candidate HealOpts support table and the side effects of dangerous parameters into this batch's contract.json; do not infer unexposed API fields from CLI help.
+2. Use a new DB/DV for each variant: a fixed range, sufficient authority, and control objects; remove=true is applied only to this run's orphans that C10 has proved have no live reference.
+3. START each one and compare it with the baseline that changes only one field; recursive checks child prefixes, pool/set checks the actual target identity, and an illegal combination requires a structured rejection with no side effects.
+4. Accept on all three of WAIT, PHYSICAL, and VERIFY; if the candidate does not support updateParity/readRepair, it must show an explicit rejection, and a default repair must not quietly be counted as coverage.
Source implementation detail: frozen `heal_h1_options_batch.py` and
`heal_h1_options_node.py` at `524eb5b32c3236497fdd8a9271df0494fa3d9974`
@@ -233,12 +233,12 @@ Operations: [SEED_DB](validation-heal-operations.md#SEED_DB) → [SEED_DV](valid
Source assertions:
-- 参数化 recursive、recreate、remove、updateParity、readRepair 与合法/非法 pool/set 组合
-- 仅测试已核实的候选语义;normal/deep 与危险 remove 的副作用明确;API 独有选项不得伪称 rc 已覆盖。
+- Parameterize recursive, recreate, remove, updateParity, and readRepair, together with legal and illegal pool/set combinations.
+- Test only candidate semantics that have been verified. The side effects of normal/deep and of a dangerous remove are explicit. An option that exists only in the API must not be presented as though rc already covered it.
-先保存失败现场及物理/状态证据,停止本例写入与在途任务,再按 RECOVER 撤销本例故障。并发版本已变化时禁止盲目恢复旧 xl.meta;恢复不能充作产品修复。未恢复环境标 DIRTY,保留本例备份和原失败结果。
+First save the failure scene and the physical and state evidence, stop this case's writes and in-flight tasks, then undo this case's fault per RECOVER. When a concurrent version has already changed, do not blindly restore the old xl.meta; recovery must not serve as a product fix. Mark an unrestored environment DIRTY, and keep this case's backup and the original failure result.
-逐项满足设计公共契约、该分支步骤及附加断言后人工评审。缺资源为 BLOCKED,窗口未命中为 NOT_OBSERVED,产品错误为 FAIL,工具/恢复错误为 INFRA_ERROR;未运行不记 PASS。
+Manually review after each item satisfies the design's common contract, that branch's steps, and the additional assertions. Missing resources are BLOCKED, a window miss is NOT_OBSERVED, a product error is FAIL, and a tool or recovery error is INFRA_ERROR; a case that did not run is not recorded as PASS.
@@ -246,21 +246,21 @@ Source assertions:
Topologies: H1,H2. Branches: original-drive-return, offline-ack-repair.
-1. 由 layout.json 选一成员盘,记录同版本 N/R/W;使用 DRIVE_GATE 仅让本服务看不到该逻辑盘,不卸载共享 XFS。
-2. 故障窗口经测试进程 mount namespace 的读失败与 Admin offline 状态双重见证;在 quorum 允许范围执行 GET 和唯一 key PUT,逐次记 ACK。
-3. DRIVE_GATE 恢复原盘身份后,等待所选手工/自动来源处理离线期间责任;不得复制备份补盘。
-4. 验证所有故障前对象及成功 ACK 对象、目标盘实际新增分片;节点 ready/GET 成功不是补片证据。
+1. Select one member disk from layout.json and record N/R/W for the same version; use DRIVE_GATE so that only this service cannot see that logical disk, without unmounting the shared XFS.
+2. Witness the fault window both by a read failure in the test process's mount namespace and by the Admin offline state; within the range quorum allows, run GET and a unique-key PUT, and record an ACK each time.
+3. After DRIVE_GATE restores the original disk identity, wait for the selected manual or automatic source to handle responsibility from the offline period; do not fill the disk by copying a backup.
+4. Verify every pre-fault object and every successfully ACKed object, and that the target disk actually gained new shards; node ready or a successful GET is not evidence of shard repair.
Operations: [SEED_DB](validation-heal-operations.md#SEED_DB) → [BASELINE](validation-heal-operations.md#BASELINE) → [DRIVE_GATE](validation-heal-operations.md#DRIVE_GATE) → [WORKLOAD](validation-heal-operations.md#WORKLOAD) → [BACKGROUND](validation-heal-operations.md#BACKGROUND) → [START](validation-heal-operations.md#START) → [WAIT](validation-heal-operations.md#WAIT) → [PHYSICAL](validation-heal-operations.md#PHYSICAL) → [VERIFY](validation-heal-operations.md#VERIFY)
Source assertions:
-- 让一个已确认 set 成员盘对 heal 服务不可用,持续读取,恢复原盘
-- 记录真实故障窗口和源/目标盘身份;在可恢复条件下最终补齐目标分片及完整 manifest,不只验证降级 GET。
+- Make one confirmed set-member disk unavailable to the heal service, keep reading, and restore the original disk.
+- Record the real fault window and the source and target disk identities. When the fault is recoverable, finally reconstruct the target shards and the complete manifest; verifying a degraded GET alone is not enough.
-先保存失败现场及物理/状态证据,停止本例写入与在途任务,再按 RECOVER 撤销本例故障。并发版本已变化时禁止盲目恢复旧 xl.meta;恢复不能充作产品修复。未恢复环境标 DIRTY,保留本例备份和原失败结果。
+First save the failure scene and the physical and state evidence, stop this case's writes and in-flight tasks, then undo this case's fault per RECOVER. When a concurrent version has already changed, do not blindly restore the old xl.meta; recovery must not serve as a product fix. Mark an unrestored environment DIRTY, and keep this case's backup and the original failure result.
-逐项满足设计公共契约、该分支步骤及附加断言后人工评审。缺资源为 BLOCKED,窗口未命中为 NOT_OBSERVED,产品错误为 FAIL,工具/恢复错误为 INFRA_ERROR;未运行不记 PASS。
+Manually review after each item satisfies the design's common contract, that branch's steps, and the additional assertions. Missing resources are BLOCKED, a window miss is NOT_OBSERVED, a product error is FAIL, and a tool or recovery error is INFRA_ERROR; a case that did not run is not recorded as PASS.
@@ -268,23 +268,23 @@ Source assertions:
Topologies: H1,H2. Branches: read-only, permission-denied.
-1. 在专用测试服务的 mount namespace 设置目标根只读;permission 实例用专用非 root 服务 UID 和独立目录 ACL,不能对 root 仅 chmod 后假称生效。
-2. 在服务相同身份/namespace 执行有界写探针,要求 EROFS/EACCES;先保存并隔离目标 part,再触发 deep heal。
-3. 目标不可写期间必须保留失败/待修责任,不能 reported repaired;PHYSICAL 确認没有写到父目录/系统盘。
-4. 解除只读或恢复原 ACL,重新修复并 VERIFY;保留故障期间错误和恢复后物理补片证据。
+1. In the dedicated test service's mount namespace, set the target root read-only; the permission instance uses a dedicated non-root service UID and an independent directory ACL, and must not merely chmod as root and then claim it took effect.
+2. Under the same identity and namespace as the service, run a bounded write probe and require EROFS/EACCES; first save and isolate the target part, then trigger deep heal.
+3. While the target is not writable, the failed or pending-repair responsibility must be kept, and it must not be reported repaired; PHYSICAL confirms that nothing was written to the parent directory or the system disk.
+4. Clear the read-only state or restore the original ACL, repair again, and VERIFY; keep the errors from the fault period and the physical shard-repair evidence after recovery.
-permission-denied 分支释放 ACL gate 后,保持同一候选进程运行,并用进程 birth、候选摘要及释放后的物理快照证明状态。释放后、提交新的唯一 Admin token 前,目标分片必须仍保持故障态;若自动恢复先改变分片,该分支记为 `NOT_OBSERVED`,不能将恢复归因给新的手工任务。新的修复 token 终态后再停止服务并验证原始分片摘要。read-only 分支按其 unit mount gate 的生命周期清理 gate,再执行独立修复任务。
+After the permission-denied branch releases the ACL gate, keep the same candidate process running and prove the state with the process birth, the candidate digest, and the physical snapshot taken after release. After release, and before a new unique Admin token is submitted, the target shard must still be faulty. If automatic recovery changes the shard first, record the branch as `NOT_OBSERVED`; the recovery must not be attributed to the new manual task. After the new repair token reaches a terminal state, stop the service and verify the original shard digest. The read-only branch clears the gate according to its unit mount-gate lifecycle, then runs an independent repair task.
Operations: [SEED_DB](validation-heal-operations.md#SEED_DB) → [FAULT](validation-heal-operations.md#FAULT) → [DRIVE_GATE](validation-heal-operations.md#DRIVE_GATE) → [START](validation-heal-operations.md#START) → [WAIT](validation-heal-operations.md#WAIT) → [PHYSICAL](validation-heal-operations.md#PHYSICAL) → [RECOVER](validation-heal-operations.md#RECOVER) → [VERIFY](validation-heal-operations.md#VERIFY)
Source assertions:
-- 仅对独占测试卷制造只读/拒绝访问,恢复后重试
-- 不可写目标不得 reported repaired;错误及待修责任可见;恢复后真实写入目标且不落到系统盘或其他目录。
+- Make only the exclusive test volume read-only or access-denied, and retry after it is restored.
+- An unwritable target must not be reported repaired. The error and the responsibility still awaiting repair are visible. After recovery, the write actually goes to the target and does not land on the system disk or in another directory.
-先保存失败现场及物理/状态证据,停止本例写入与在途任务,再按 RECOVER 撤销本例故障。并发版本已变化时禁止盲目恢复旧 xl.meta;恢复不能充作产品修复。未恢复环境标 DIRTY,保留本例备份和原失败结果。
+First save the failure scene and the physical and state evidence, stop this case's writes and in-flight tasks, then undo this case's fault per RECOVER. When a concurrent version has already changed, do not blindly restore the old xl.meta; recovery must not serve as a product fix. Mark an unrestored environment DIRTY, and keep this case's backup and the original failure result.
-逐项满足设计公共契约、该分支步骤及附加断言后人工评审。缺资源为 BLOCKED,窗口未命中为 NOT_OBSERVED,产品错误为 FAIL,工具/恢复错误为 INFRA_ERROR;未运行不记 PASS。
+Manually review after each item satisfies the design's common contract, that branch's steps, and the additional assertions. Missing resources are BLOCKED, a window miss is NOT_OBSERVED, a product error is FAIL, and a tool or recovery error is INFRA_ERROR; a case that did not run is not recorded as PASS.
@@ -292,21 +292,21 @@ Source assertions:
Topologies: H1,H2. Branches: new-disk-data, new-disk-versioned.
-1. 完整基线后停止本例服务,将目标逻辑盘 data 整体隔离到本 run 的 quarantine;使用 REPLACE 建空的新 data 根。
-2. 按候选初始化契约启动,读取 format.json 证明新 this UUID 属于原 pool/set/slot;其他盘格式不得变成新集群。
-3. 显式 START root/deep 或本实例指定重建来源,旧盘备份不可读给测试进程;WAIT 全部责任。
-4. 逐对象/版本验证目标盘分片与同角色基线摘要;确认不是从 quarantine 拷贝回去;恢复操作只在取证后做并标为 rescue。
+1. After a complete baseline, stop this case's service and isolate the target logical disk's data as a whole into this run's quarantine; use REPLACE to create an empty new data root.
+2. Start according to the candidate initialization contract, and read format.json to prove the new this UUID belongs to the original pool/set/slot; other disks' formats must not become a new cluster.
+3. Explicitly START root/deep or the rebuild source this instance specifies, and the old-disk backup must not be readable by the test process; WAIT for all responsibility.
+4. For each object and version, verify that the target disk's shards match the same-role baseline digest; confirm they were not copied back from quarantine; do recovery only after evidence collection and mark it as rescue.
Operations: [SEED_DB](validation-heal-operations.md#SEED_DB) → [SEED_DV](validation-heal-operations.md#SEED_DV) → [BASELINE](validation-heal-operations.md#BASELINE) → [REPLACE](validation-heal-operations.md#REPLACE) → [FORMAT](validation-heal-operations.md#FORMAT) → [START](validation-heal-operations.md#START) → [WAIT](validation-heal-operations.md#WAIT) → [PHYSICAL](validation-heal-operations.md#PHYSICAL) → [VERIFY](validation-heal-operations.md#VERIFY)
Source assertions:
-- 换入空的新逻辑盘,确认新 disk UUID/正确格式后触发重建
-- 新盘与旧盘身份可区分且属于正确 pool/set/slot;逐版本重建完成,不能从隔离备份复制回来冒充 heal。
+- Swap in an empty new logical disk, and after confirming the new disk UUID and a correct format, trigger rebuild.
+- The new disk is distinguishable from the old disk and belongs to the correct pool/set/slot. Rebuild completes for every version. Copying the data back from an isolated backup must not be passed off as heal.
-先保存失败现场及物理/状态证据,停止本例写入与在途任务,再按 RECOVER 撤销本例故障。并发版本已变化时禁止盲目恢复旧 xl.meta;恢复不能充作产品修复。未恢复环境标 DIRTY,保留本例备份和原失败结果。
+First save the failure scene and the physical and state evidence, stop this case's writes and in-flight tasks, then undo this case's fault per RECOVER. When a concurrent version has already changed, do not blindly restore the old xl.meta; recovery must not serve as a product fix. Mark an unrestored environment DIRTY, and keep this case's backup and the original failure result.
-逐项满足设计公共契约、该分支步骤及附加断言后人工评审。缺资源为 BLOCKED,窗口未命中为 NOT_OBSERVED,产品错误为 FAIL,工具/恢复错误为 INFRA_ERROR;未运行不记 PASS。
+Manually review after each item satisfies the design's common contract, that branch's steps, and the additional assertions. Missing resources are BLOCKED, a window miss is NOT_OBSERVED, a product error is FAIL, and a tool or recovery error is INFRA_ERROR; a case that did not run is not recorded as PASS.
@@ -320,21 +320,21 @@ If exact endpoint mounts or the reviewed mount-switch adapter are unavailable,
the case is recorded as pre-deployment `BLOCKED` without creating either
the blank or foreign-format fixture.
-1. REPLACE 建没有 format 的空逻辑盘;记录空目录证据,启动并观察候选支持的安全初始化或明确拒绝。
-2. 第二实例使用同 run 的第二个隔离小集群生成外来 format,将其按 FORMAT_BAD 手册装入替换根,保留原始副本和所有 UUID。
-3. 外来身份不能被静默覆盖或挂到错误 set;若拒绝,等待有界 deadline 并确认其他正常盘数据不变。
-4. 停止本服务、撤回替换根、恢复原身份后 VERIFY;空盘正常接纳实例还须逐对象重建证明。
+1. Use REPLACE to create an empty logical disk with no format; record the empty-directory evidence, start, and observe the safe initialization the candidate supports or an explicit rejection.
+2. The second instance uses this run's second isolated small cluster to generate a foreign format, installs it into the replacement root per the FORMAT_BAD manual, and keeps the original copy and every UUID.
+3. A foreign identity must not be silently overwritten or attached to the wrong set; if it is rejected, wait for a bounded deadline and confirm that data on the other healthy disks is unchanged.
+4. Stop this service, withdraw the replacement root, and restore the original identity, then VERIFY; an instance that normally accepts an empty disk must also prove the rebuild object by object.
Operations: [SEED_DB](validation-heal-operations.md#SEED_DB) → [REPLACE](validation-heal-operations.md#REPLACE) → [FORMAT_BAD](validation-heal-operations.md#FORMAT_BAD) → [BACKGROUND](validation-heal-operations.md#BACKGROUND) → [PHYSICAL](validation-heal-operations.md#PHYSICAL) → [RECOVER](validation-heal-operations.md#RECOVER) → [VERIFY](validation-heal-operations.md#VERIFY)
Source assertions:
-- 引入空未格式化测试盘,并与带异集群格式的盘分别比较
-- 按候选支持的初始化契约安全接纳或明确拒绝;不得覆盖外来身份或将未就绪盘标为可写健康。
+- Introduce an empty unformatted test disk, and compare that case separately with a disk that carries a foreign cluster's format.
+- Per the initialization contract the candidate supports, accept it safely or reject it explicitly. Do not overwrite a foreign identity, and do not mark a disk that is not ready as writable and healthy.
-先保存失败现场及物理/状态证据,停止本例写入与在途任务,再按 RECOVER 撤销本例故障。并发版本已变化时禁止盲目恢复旧 xl.meta;恢复不能充作产品修复。未恢复环境标 DIRTY,保留本例备份和原失败结果。
+First save the failure scene and the physical and state evidence, stop this case's writes and in-flight tasks, then undo this case's fault per RECOVER. When a concurrent version has already changed, do not blindly restore the old xl.meta; recovery must not serve as a product fix. Mark an unrestored environment DIRTY, and keep this case's backup and the original failure result.
-逐项满足设计公共契约、该分支步骤及附加断言后人工评审。缺资源为 BLOCKED,窗口未命中为 NOT_OBSERVED,产品错误为 FAIL,工具/恢复错误为 INFRA_ERROR;未运行不记 PASS。
+Manually review after each item satisfies the design's common contract, that branch's steps, and the additional assertions. Missing resources are BLOCKED, a window miss is NOT_OBSERVED, a product error is FAIL, and a tool or recovery error is INFRA_ERROR; a case that did not run is not recorded as PASS.
@@ -351,21 +351,21 @@ members expire as `NOT_OBSERVED` without retrying the Heal task. The original
`38200772548` unknown-version receipts remain a retained failure. This startup
check alone does not establish its underlying cause or a new live PASS.
-1. 从候选/实际元数据确定 M=N-R,枚举 1..M 的合法缺片数及 data/parity 组合,写入本批展开参数表;不得只运行 loss=1。
-2. 每实例用相同版本的新对象,将指定成员上的 part 分别 FAULT missing;多份元数据也要证明指向同 version/data_dir。
-3. 在满足修复提交条件时 START deep/WAIT;先检查全部目标片与角色基线相同,再 VERIFY。
-4. 每次恢复隔离备份后换新对象再做下一组合;低于 R 的组合另由 B07 计负例,不合并成绩。
+1. From the candidate and the actual metadata, determine M=N-R, enumerate the legal missing-shard counts 1..M and the data/parity combinations, and write them into this batch's expanded parameter table; do not run only loss=1.
+2. Each instance uses a new object of the same version and applies FAULT missing, separately, to the part on each specified member; multiple metadata copies must also be shown to point at the same version/data_dir.
+3. When the conditions for committing a repair are met, START deep/WAIT; first check that every target shard matches the role baseline, then VERIFY.
+4. After each restore of the isolated backup, switch to a new object before the next combination; combinations below R are counted as negative cases by B07 on their own, and the results are not merged.
Operations: [SEED_DB](validation-heal-operations.md#SEED_DB) → [SEED_DV](validation-heal-operations.md#SEED_DV) → [FORMAT](validation-heal-operations.md#FORMAT) → [BASELINE](validation-heal-operations.md#BASELINE) → [FAULT](validation-heal-operations.md#FAULT) → [START](validation-heal-operations.md#START) → [WAIT](validation-heal-operations.md#WAIT) → [PHYSICAL](validation-heal-operations.md#PHYSICAL) → [VERIFY](validation-heal-operations.md#VERIFY)
Source assertions:
-- 按实际同版本 EC 布局依次缺失 1..M 个 shard,覆盖 data/parity 角色组合
-- 每个参数实例单独满足读取与写入 quorum;满足修复条件时全部目标重建,不把跨版本或跨 set 的 shard 合并。
+- Following the actual same-version EC layout, drop 1..M shards in turn, covering the data and parity role combinations.
+- Each parameter instance on its own meets read quorum and write quorum. When the repair conditions are met, rebuild every target, and do not merge shards across versions or across sets.
-先保存失败现场及物理/状态证据,停止本例写入与在途任务,再按 RECOVER 撤销本例故障。并发版本已变化时禁止盲目恢复旧 xl.meta;恢复不能充作产品修复。未恢复环境标 DIRTY,保留本例备份和原失败结果。
+First save the failure scene and the physical and state evidence, stop this case's writes and in-flight tasks, then undo this case's fault per RECOVER. When a concurrent version has already changed, do not blindly restore the old xl.meta; recovery must not serve as a product fix. Mark an unrestored environment DIRTY, and keep this case's backup and the original failure result.
-逐项满足设计公共契约、该分支步骤及附加断言后人工评审。缺资源为 BLOCKED,窗口未命中为 NOT_OBSERVED,产品错误为 FAIL,工具/恢复错误为 INFRA_ERROR;未运行不记 PASS。
+Manually review after each item satisfies the design's common contract, that branch's steps, and the additional assertions. Missing resources are BLOCKED, a window miss is NOT_OBSERVED, a product error is FAIL, and a tool or recovery error is INFRA_ERROR; a case that did not run is not recorded as PASS.
@@ -373,23 +373,23 @@ Source assertions:
Topologies: H1,H2. Branches: exact-r-read, exact-r-heal-write-limited, restored-w.
-执行器仅接受经逐文件和锁策略复核的 `072b57b37559fbe626aeda8abf61617fdcbdd926` 与 GET quorum 修复候选 `1c7fc818dd858247c45b39542abd8291b1e6e31b`。记录所选候选的精确锁契约;inspector 必须同时匹配候选及当前 helper 源码摘要,未知候选在部署前拒绝。below-R GET 仍必须为候选冻结的 `503/SlowDownRead`,旧 `500/InternalError` 失败记录保留;支持新候选不等于新的 live 验收。
+The executor accepts only `072b57b37559fbe626aeda8abf61617fdcbdd926`, reviewed file by file and against the lock policy, and the GET-quorum repair candidate `1c7fc818dd858247c45b39542abd8291b1e6e31b`. Record the exact lock contract of the selected candidate. The inspector must match both the candidate and the current helper source digest; an unknown candidate is rejected before deployment. A below-R GET must still be the candidate-frozen `503/SlowDownRead`. The historical `500/InternalError` failure record is retained. Supporting a new candidate is not a new live acceptance.
-1. 记录读取 R、修复写 W、锁 quorum 的来源;选择同一版本保留恰好 R 个权威片,其他片隔离。
-2. 第一次请求 GET 验证读取边界并记录 read-repair 干扰;用第二个相同形态新对象执行只手工 deep heal。
-3. 如果只有 R 但写/锁 quorum 不足,要求拒绝持久化成功;若拓扑中该组合不可实现,在运行前声明精确原因并选满足条件的拓扑,不伪造离线数。
-4. 恢复到 W 后重新启动修复并保存独立 token,比较精确目标片与全部内容;成功读不能充修复成功。
+1. Record the source of read R, of repair-write W, and of the lock quorum; for the same version, keep exactly R authoritative shards and isolate the others.
+2. The first request uses GET to verify the read boundary and records read-repair interference; with a second new object of the same shape, run a manual-only deep heal.
+3. If only R are present but the write or lock quorum is insufficient, require that a persisted success be rejected; if that combination cannot be realized in the topology, state the exact reason before the run and choose a topology that meets the conditions, without fabricating an offline count.
+4. After restoring to W, start the repair again and save an independent token, and compare the exact target shards with the full content; a successful read cannot count as a successful repair.
Operations: [SEED_DB](validation-heal-operations.md#SEED_DB) → [FORMAT](validation-heal-operations.md#FORMAT) → [FAULT](validation-heal-operations.md#FAULT) → [GET](validation-heal-operations.md#GET) → [START](validation-heal-operations.md#START) → [WAIT](validation-heal-operations.md#WAIT) → [RECOVER](validation-heal-operations.md#RECOVER) → [PHYSICAL](validation-heal-operations.md#PHYSICAL) → [VERIFY](validation-heal-operations.md#VERIFY)
Source assertions:
-- 保留恰好 R 个同版本权威 shard,测试读取及修复提交,再恢复写 quorum
-- 读取边界和修复写入边界分别判定;不能因还能 GET 就要求在写 quorum 不足时提交成功。
+- Keep exactly R authority shards of the same version, test the read and the repair commit, then restore write quorum.
+- Judge the read boundary and the repair-write boundary separately. Do not require a successful commit when write quorum is short merely because GET still succeeds.
-先保存失败现场及物理/状态证据,停止本例写入与在途任务,再按 RECOVER 撤销本例故障。并发版本已变化时禁止盲目恢复旧 xl.meta;恢复不能充作产品修复。未恢复环境标 DIRTY,保留本例备份和原失败结果。
+First save the failure scene and the physical and state evidence, stop this case's writes and in-flight tasks, then undo this case's fault per RECOVER. When a concurrent version has already changed, do not blindly restore the old xl.meta; recovery must not serve as a product fix. Mark an unrestored environment DIRTY, and keep this case's backup and the original failure result.
-逐项满足设计公共契约、该分支步骤及附加断言后人工评审。缺资源为 BLOCKED,窗口未命中为 NOT_OBSERVED,产品错误为 FAIL,工具/恢复错误为 INFRA_ERROR;未运行不记 PASS。
+Manually review after each item satisfies the design's common contract, that branch's steps, and the additional assertions. Missing resources are BLOCKED, a window miss is NOT_OBSERVED, a product error is FAIL, and a tool or recovery error is INFRA_ERROR; a case that did not run is not recorded as PASS.
@@ -397,21 +397,21 @@ Source assertions:
Topologies: H1,H2. Branches: r-minus-1-unversioned, r-minus-1-historical.
-1. 选定精确 versionId 保留 R-1 份同版本权威;隔离其余 part,确保没有其他 pool 或旧版本兜底被当作同版本来源。
-2. GET 必须返回冻结的不可恢复错误,不能返回 200 错内容或空对象;deep heal 的 execution/disposition 必须如实失败/未解决。
-3. 在恢复前快照所有好片和 xl.meta,证实负例没有扩大损坏或执行 remove;保留失败任务原 token。
-4. 仅在负例取证完成后 RECOVER 原始片,再显式 heal 和 VERIFY;恢复后的绿不能覆盖故障期间的错误。
+1. For the chosen exact versionId, keep R-1 authoritative copies of the same version; isolate the remaining parts, and ensure that no other pool or old-version fallback is treated as a same-version source.
+2. GET must return the frozen unrecoverable error, and must not return 200 with the wrong content or an empty object; deep heal's execution/disposition must truthfully be failed or unresolved.
+3. Before recovery, snapshot every good shard and xl.meta, and confirm that the negative case did not spread the damage or execute remove; keep the failed task's original token.
+4. Only after negative-case evidence collection is complete, RECOVER the original shards, then explicitly heal and VERIFY; a green result after recovery must not cover the errors from the fault period.
Operations: [SEED_DB](validation-heal-operations.md#SEED_DB) → [SEED_DV](validation-heal-operations.md#SEED_DV) → [FAULT](validation-heal-operations.md#FAULT) → [GET](validation-heal-operations.md#GET) → [START](validation-heal-operations.md#START) → [WAIT](validation-heal-operations.md#WAIT) → [PHYSICAL](validation-heal-operations.md#PHYSICAL) → [RECOVER](validation-heal-operations.md#RECOVER) → [VERIFY](validation-heal-operations.md#VERIFY)
Source assertions:
-- 使权威来源小于 R,执行 GET 与 deep heal,再恢复被隔离源
-- 故障期间不得返回伪造的成功内容、空对象或误删记录;明确 unresolved/失败;恢复后原对象和版本完整。
+- Reduce the authority sources to fewer than R, run GET and deep heal, then restore the isolated sources.
+- During the fault, do not return forged successful content, an empty object, or a mistaken delete record. State unresolved or failure explicitly. After recovery, the original object and its versions are complete.
-先保存失败现场及物理/状态证据,停止本例写入与在途任务,再按 RECOVER 撤销本例故障。并发版本已变化时禁止盲目恢复旧 xl.meta;恢复不能充作产品修复。未恢复环境标 DIRTY,保留本例备份和原失败结果。
+First save the failure scene and the physical and state evidence, stop this case's writes and in-flight tasks, then undo this case's fault per RECOVER. When a concurrent version has already changed, do not blindly restore the old xl.meta; recovery must not serve as a product fix. Mark an unrestored environment DIRTY, and keep this case's backup and the original failure result.
-逐项满足设计公共契约、该分支步骤及附加断言后人工评审。缺资源为 BLOCKED,窗口未命中为 NOT_OBSERVED,产品错误为 FAIL,工具/恢复错误为 INFRA_ERROR;未运行不记 PASS。
+Manually review after each item satisfies the design's common contract, that branch's steps, and the additional assertions. Missing resources are BLOCKED, a window miss is NOT_OBSERVED, a product error is FAIL, and a tool or recovery error is INFRA_ERROR; a case that did not run is not recorded as PASS.
@@ -419,21 +419,21 @@ Source assertions:
Topologies: H1,H2. Branches: read-present-write-absent, lock-quorum-absent.
-1. 用 layout.json 分别计算对象写和锁 quorum;DRIVE_GATE/NETWORK 构造仍可读但不能安全提交的测试服务可见性。
-2. 执行唯一 key PUT 与已有缺片对象 heal;保存每个 ACK/HTTP/body,超时写必须查询唯一 key 解析结果,禁止直接重试换 key。
-3. 期间不能 reported repaired 或返回无法兑现的成功写 ACK;从各 peer 收集 quorum 与锁失败日志。
-4. 解除故障后按全部已确认/已解析写入重建 manifest,实际补齐目标并 VERIFY;普通低 quorum 错误不是基础设施错误。
+1. Use layout.json to compute the object-write quorum and the lock quorum separately; use DRIVE_GATE/NETWORK to construct a test-service visibility that can still read but cannot commit safely.
+2. Run a unique-key PUT and a heal of an existing object that is missing a shard; save every ACK/HTTP/body, a timed-out write must query that unique key's resolution, and a direct retry that changes the key is forbidden.
+3. During this period it must not be reported repaired or return a successful write ACK that cannot be honored; collect quorum and lock-failure logs from every peer.
+4. After the fault is cleared, rebuild the manifest from every confirmed or resolved write, actually complete the target shards, and VERIFY; an ordinary low-quorum error is not an infrastructure error.
Operations: [SEED_DB](validation-heal-operations.md#SEED_DB) → [FORMAT](validation-heal-operations.md#FORMAT) → [DRIVE_GATE](validation-heal-operations.md#DRIVE_GATE) → [NETWORK](validation-heal-operations.md#NETWORK) → [WORKLOAD](validation-heal-operations.md#WORKLOAD) → [START](validation-heal-operations.md#START) → [PHYSICAL](validation-heal-operations.md#PHYSICAL) → [RECOVER](validation-heal-operations.md#RECOVER) → [VERIFY](validation-heal-operations.md#VERIFY)
Source assertions:
-- 构造读 quorum 尚存而写/锁 quorum 不足,执行 heal 与新 PUT
-- 不能虚报持久化修复或成功写入;分别记录对象、元数据、分布式锁 quorum;恢复后核对所有已确认写入。
+- Construct a case where read quorum still exists but write quorum and lock quorum are insufficient, and run heal and a new PUT.
+- Do not falsely report a persisted repair or a successful write. Record the object, metadata, and distributed-lock quorums separately. After recovery, check every confirmed write.
-先保存失败现场及物理/状态证据,停止本例写入与在途任务,再按 RECOVER 撤销本例故障。并发版本已变化时禁止盲目恢复旧 xl.meta;恢复不能充作产品修复。未恢复环境标 DIRTY,保留本例备份和原失败结果。
+First save the failure scene and the physical and state evidence, stop this case's writes and in-flight tasks, then undo this case's fault per RECOVER. When a concurrent version has already changed, do not blindly restore the old xl.meta; recovery must not serve as a product fix. Mark an unrestored environment DIRTY, and keep this case's backup and the original failure result.
-逐项满足设计公共契约、该分支步骤及附加断言后人工评审。缺资源为 BLOCKED,窗口未命中为 NOT_OBSERVED,产品错误为 FAIL,工具/恢复错误为 INFRA_ERROR;未运行不记 PASS。
+Manually review after each item satisfies the design's common contract, that branch's steps, and the additional assertions. Missing resources are BLOCKED, a window miss is NOT_OBSERVED, a product error is FAIL, and a tool or recovery error is INFRA_ERROR; a case that did not run is not recorded as PASS.
@@ -441,21 +441,21 @@ Source assertions:
Topologies: H2. Branches: one-node-stop, rejoin-with-new-versions.
-1. 确认 H2 目标 set 跨节点,统计停一节点实际损失的盘数,不能使用每台独立 pool 的 H5 替代。
-2. PROCESS stop 精确测试 unit,见证 MainPID 消失及端点不可达;保持其他节点服务。
-3. 按计算允许的 quorum 执行降级 PUT/DELETE/GET,保存版本化 ACK;重入同一节点同一数据根。
-4. 观察 MRF/修复责任直到补齐离线期间成功对象;从所有节点 VERIFY 和目标 PHYSICAL,不能只看进程 ready。
+1. Confirm that the H2 target set is cross-node, count the disks actually lost by stopping one node, and do not substitute H5, which gives each machine its own pool.
+2. PROCESS stop the exact test unit, and witness that MainPID is gone and the endpoint is unreachable; keep the services on the other nodes running.
+3. Within the quorum the calculation allows, run degraded PUT/DELETE/GET and save the versioned ACKs; re-enter the same node and the same data root.
+4. Observe MRF and repair responsibility until the objects that succeeded during the offline period are fully repaired; VERIFY from every node and PHYSICAL the target, and do not look only at process ready.
Operations: [SEED_DB](validation-heal-operations.md#SEED_DB) → [SEED_DV](validation-heal-operations.md#SEED_DV) → [FORMAT](validation-heal-operations.md#FORMAT) → [PROCESS](validation-heal-operations.md#PROCESS) → [WORKLOAD](validation-heal-operations.md#WORKLOAD) → [BACKGROUND](validation-heal-operations.md#BACKGROUND) → [PHYSICAL](validation-heal-operations.md#PHYSICAL) → [VERIFY](validation-heal-operations.md#VERIFY)
Source assertions:
-- 停止同一跨节点 set 的一个节点,执行允许的降级 I/O,再重入
-- 按实际丢失盘数判定可用性;缺失期间成功提交的对象补齐;节点重新 ready 不等于所有对象已重建。
+- Stop one node of the same cross-node set, perform the degraded I/O that is allowed, then rejoin.
+- Judge availability by the number of disks actually lost. Reconstruct the objects that were successfully committed during the outage. A node becoming ready again does not mean every object has been rebuilt.
-先保存失败现场及物理/状态证据,停止本例写入与在途任务,再按 RECOVER 撤销本例故障。并发版本已变化时禁止盲目恢复旧 xl.meta;恢复不能充作产品修复。未恢复环境标 DIRTY,保留本例备份和原失败结果。
+First save the failure scene and the physical and state evidence, stop this case's writes and in-flight tasks, then undo this case's fault per RECOVER. When a concurrent version has already changed, do not blindly restore the old xl.meta; recovery must not serve as a product fix. Mark an unrestored environment DIRTY, and keep this case's backup and the original failure result.
-逐项满足设计公共契约、该分支步骤及附加断言后人工评审。缺资源为 BLOCKED,窗口未命中为 NOT_OBSERVED,产品错误为 FAIL,工具/恢复错误为 INFRA_ERROR;未运行不记 PASS。
+Manually review after each item satisfies the design's common contract, that branch's steps, and the additional assertions. Missing resources are BLOCKED, a window miss is NOT_OBSERVED, a product error is FAIL, and a tool or recovery error is INFRA_ERROR; a case that did not run is not recorded as PASS.
@@ -463,21 +463,21 @@ Source assertions:
Topologies: H2. Branches: drive-flap-3, node-flap-3, interrupt-rebuild-again.
-1. 预注册三个故障周期、每次最长窗口和总 deadline;准备带 nonce 的 ACK ledger,设置只属于本服务的 drive gate 或 unit 操作。
-2. 每轮离线/在线都先见证实际状态;第三轮在 WINDOW 证明部分重建时再次离线,不以固定 sleep 替代。
-3. 记录每轮责任/队列、任务身份、错误率及 CPU/FD;超过预注册重试/资源预算即失败,不继续循环等绿。
-4. 彻底解除故障后验证全部已确认对象、目标分片和持久队列收敛;活动窗口没命中记 NOT_OBSERVED。
+1. Pre-register three fault cycles, the maximum window of each, and the total deadline; prepare an ACK ledger with nonces, and set a drive gate or unit operation that belongs only to this service.
+2. On each offline and online round, witness the actual state first; on the third round, go offline again when WINDOW proves a partial rebuild, and do not substitute a fixed sleep.
+3. Record each round's responsibility and queue, task identity, error rate, and CPU/FD; exceeding the pre-registered retry or resource budget is a failure, and do not keep looping to wait for green.
+4. After the fault is fully cleared, verify that every confirmed object, the target shards, and the persistent queue have converged; if the active window was not hit, record NOT_OBSERVED.
Operations: [SEED_DB](validation-heal-operations.md#SEED_DB) → [DRIVE_GATE](validation-heal-operations.md#DRIVE_GATE) → [PROCESS](validation-heal-operations.md#PROCESS) → [WORKLOAD](validation-heal-operations.md#WORKLOAD) → [WINDOW](validation-heal-operations.md#WINDOW) → [BACKGROUND](validation-heal-operations.md#BACKGROUND) → [METRICS](validation-heal-operations.md#METRICS) → [PHYSICAL](validation-heal-operations.md#PHYSICAL) → [VERIFY](validation-heal-operations.md#VERIFY)
Source assertions:
-- 重复短暂离线/在线并在重建途中再次失联
-- 重试与队列有界、身份稳定,无重复任务风暴;最终故障解除后收敛,所有已确认对象保持正确。
+- Go briefly offline and online repeatedly, and drop offline again while a rebuild is in progress.
+- Retries and the queue stay bounded, and identity stays stable, with no storm of duplicate tasks. After the fault is finally cleared, converge, and every confirmed object stays correct.
-先保存失败现场及物理/状态证据,停止本例写入与在途任务,再按 RECOVER 撤销本例故障。并发版本已变化时禁止盲目恢复旧 xl.meta;恢复不能充作产品修复。未恢复环境标 DIRTY,保留本例备份和原失败结果。
+First save the failure scene and the physical and state evidence, stop this case's writes and in-flight tasks, then undo this case's fault per RECOVER. When a concurrent version has already changed, do not blindly restore the old xl.meta; recovery must not serve as a product fix. Mark an unrestored environment DIRTY, and keep this case's backup and the original failure result.
-逐项满足设计公共契约、该分支步骤及附加断言后人工评审。缺资源为 BLOCKED,窗口未命中为 NOT_OBSERVED,产品错误为 FAIL,工具/恢复错误为 INFRA_ERROR;未运行不记 PASS。
+Manually review after each item satisfies the design's common contract, that branch's steps, and the additional assertions. Missing resources are BLOCKED, a window miss is NOT_OBSERVED, a product error is FAIL, and a tool or recovery error is INFRA_ERROR; a case that did not run is not recorded as PASS.
@@ -485,21 +485,21 @@ Source assertions:
Topologies: H1,H2. Branches: data-single-part, parity-single-part, multipart-first, multipart-middle, multipart-last.
-1. DB/DM 写入后用 FORMAT/PHYSICAL 将每个 version/part 映射到真实盘角色;每个 variant 选不同目标,不按盘序号猜 data/parity。
-2. 停住注入目标写入后 FAULT missing 只移除已冻结摘要的 part,xl.meta 和 format 保持;恢复运行后禁止目标 GET。
-3. START deep 并 WAIT;首先验证该角色目标 part 恢复到故障前完整 SHA-256/长度,再 GET/Range 和版本清单。
-4. 任何仅 xl.meta 恢复、重建错误 part 或 control 变更都失败;清理时才允许使用原始备份。
+1. After the DB/DM writes, use FORMAT/PHYSICAL to map each version/part to its real disk role; each variant selects a different target, and does not guess data/parity by disk index.
+2. After writes to the injection target are stopped, FAULT missing removes only the part whose digest was frozen, while xl.meta and format stay; once it is running again, GET of the target is forbidden.
+3. START deep and WAIT; first verify that the target part of that role is restored to the full pre-fault SHA-256 and length, then check GET/Range and the version manifest.
+4. Restoring only xl.meta, rebuilding the wrong part, or changing the control is a failure; the original backup may be used only at cleanup.
Operations: [SEED_DB](validation-heal-operations.md#SEED_DB) → [SEED_DM](validation-heal-operations.md#SEED_DM) → [FORMAT](validation-heal-operations.md#FORMAT) → [BASELINE](validation-heal-operations.md#BASELINE) → [FAULT](validation-heal-operations.md#FAULT) → [START](validation-heal-operations.md#START) → [WAIT](validation-heal-operations.md#WAIT) → [PHYSICAL](validation-heal-operations.md#PHYSICAL) → [VERIFY](validation-heal-operations.md#VERIFY)
Source assertions:
-- 仅隔离指定对象某个 part/shard,保留 format 与其他元数据,执行 deep heal
-- 对应数据或校验分片重新生成,part/版本身份准确;不得只看到 xl.meta 重现就通过。
+- Isolate only one part/shard of the specified object, keep format and the other metadata, and run deep heal
+- The corresponding data or parity shard is regenerated, and the part/version identity is accurate; do not pass merely because xl.meta reappears.
-先保存失败现场及物理/状态证据,停止本例写入与在途任务,再按 RECOVER 撤销本例故障。并发版本已变化时禁止盲目恢复旧 xl.meta;恢复不能充作产品修复。未恢复环境标 DIRTY,保留本例备份和原失败结果。
+First save the failure scene and the physical and state evidence, stop this case's writes and in-flight tasks, then undo this case's fault per RECOVER. When a concurrent version has already changed, do not blindly restore the old xl.meta; recovery must not serve as a product fix. Mark an unrestored environment DIRTY, and keep this case's backup and the original failure result.
-逐项满足设计公共契约、该分支步骤及附加断言后人工评审。缺资源为 BLOCKED,窗口未命中为 NOT_OBSERVED,产品错误为 FAIL,工具/恢复错误为 INFRA_ERROR;未运行不记 PASS。
+Manually review after each item satisfies the design's common contract, that branch's steps, and the additional assertions. Missing resources are BLOCKED, a window miss is NOT_OBSERVED, a product error is FAIL, and a tool or recovery error is INFRA_ERROR; a case that did not run is not recorded as PASS.
@@ -507,21 +507,21 @@ Source assertions:
Topologies: H1,H2. Branches: data-first-byte, data-middle-byte, data-last-byte, multipart-data.
-1. 记录候选解析出的 data shard;用不重复 SHAKE 内容避免数据片/校验片摘要相同,验证健康基线。
-2. FAULT bitflip 在冻结的首/中/尾字节偏移各翻转一位;保存偏移、注入前后 hash 及目标物理角色。
-3. 保持足够正确来源,分别执行本批预注册 deep/read-repair 触发;手工实例不得预读目标。
-4. 目标恢复完整原始片摘要,解码后全文和 Range 正确;只返回正确 GET 但坏片仍在必须失败。
+1. Record the data shard the candidate resolved; use non-repeating SHAKE content so the data-shard and parity-shard digests are not the same, and verify the healthy baseline.
+2. FAULT bitflip flips one bit at each of the frozen first, middle, and last byte offsets; save the offsets, the hashes before and after injection, and the target physical role.
+3. Keep enough correct sources, and run this batch's pre-registered deep and read-repair triggers separately; the manual instance must not pre-read the target.
+4. The target is restored to the complete original shard digest, and the decoded full content and Range are correct; a correct GET alone, while the bad shard remains, must fail.
Operations: [SEED_DB](validation-heal-operations.md#SEED_DB) → [SEED_DM](validation-heal-operations.md#SEED_DM) → [FORMAT](validation-heal-operations.md#FORMAT) → [FAULT](validation-heal-operations.md#FAULT) → [START](validation-heal-operations.md#START) → [WAIT](validation-heal-operations.md#WAIT) → [PHYSICAL](validation-heal-operations.md#PHYSICAL) → [VERIFY](validation-heal-operations.md#VERIFY)
Source assertions:
-- 在指定 data shard 的已知偏移翻转字节,包含首/中/尾位置
-- 故障前后摘要证明注入生效;deep 或 read-repair 发现损坏并修复;独立 bitrot/EC 及 S3 全内容验证通过。
+- Flip bytes at known offsets of the specified data shard, including the head, middle, and tail positions
+- Digests before and after the fault prove the injection took effect; deep or read-repair finds the corruption and repairs it; independent bitrot/EC and full-content S3 verification pass.
-先保存失败现场及物理/状态证据,停止本例写入与在途任务,再按 RECOVER 撤销本例故障。并发版本已变化时禁止盲目恢复旧 xl.meta;恢复不能充作产品修复。未恢复环境标 DIRTY,保留本例备份和原失败结果。
+First save the failure scene and the physical and state evidence, stop this case's writes and in-flight tasks, then undo this case's fault per RECOVER. When a concurrent version has already changed, do not blindly restore the old xl.meta; recovery must not serve as a product fix. Mark an unrestored environment DIRTY, and keep this case's backup and the original failure result.
-逐项满足设计公共契约、该分支步骤及附加断言后人工评审。缺资源为 BLOCKED,窗口未命中为 NOT_OBSERVED,产品错误为 FAIL,工具/恢复错误为 INFRA_ERROR;未运行不记 PASS。
+Manually review after each item satisfies the design's common contract, that branch's steps, and the additional assertions. Missing resources are BLOCKED, a window miss is NOT_OBSERVED, a product error is FAIL, and a tool or recovery error is INFRA_ERROR; a case that did not run is not recorded as PASS.
@@ -529,21 +529,21 @@ Source assertions:
Topologies: H1,H2. Branches: parity-first-byte, parity-middle-byte, parity-last-byte.
-1. 选择经 FORMAT 证明的 parity 片,保留足够 data;普通读取是否需要该片另作控制实例,不能触发后再声称纯 deep。
-2. FAULT bitflip 覆盖首/中/尾位置;注入后仅 PHYSICAL,不 GET/HEAD。
-3. START deep/WAIT,要求 repaired 责任对应正确对象/版本/pool/set;与修复前同角色 SHA-256 比较。
-4. 全文 GET 及版本/metadata 对照通过且错误 parity 已消失才通过;后台提前补片则该触发实例未观测。
+1. Select a parity shard proved by FORMAT, and keep enough data; whether an ordinary read needs that shard is a separate control instance, and it must not be triggered and then claimed to be pure deep.
+2. FAULT bitflip covers the first, middle, and last positions; after injection, run PHYSICAL only, with no GET or HEAD.
+3. START deep/WAIT, and require the repaired responsibility to correspond to the correct object, version, pool, and set; compare it with the same-role SHA-256 from before the repair.
+4. It passes only when the full GET and the version/metadata comparison pass and the bad parity is gone; if the background repairs the shard early, that trigger instance was not observed.
Operations: [SEED_DB](validation-heal-operations.md#SEED_DB) → [FORMAT](validation-heal-operations.md#FORMAT) → [FAULT](validation-heal-operations.md#FAULT) → [START](validation-heal-operations.md#START) → [WAIT](validation-heal-operations.md#WAIT) → [PHYSICAL](validation-heal-operations.md#PHYSICAL) → [VERIFY](validation-heal-operations.md#VERIFY)
Source assertions:
-- 损坏 parity shard,使普通 GET 仍可能成功,再执行 deep heal
-- 必须识别并重建损坏 parity;只有 GET 正确、无目标分片修复不能通过。
+- Corrupt a parity shard so that an ordinary GET may still succeed, then run deep heal
+- The corrupted parity must be identified and reconstructed; a correct GET alone, with no repair of the target shard, cannot pass.
-先保存失败现场及物理/状态证据,停止本例写入与在途任务,再按 RECOVER 撤销本例故障。并发版本已变化时禁止盲目恢复旧 xl.meta;恢复不能充作产品修复。未恢复环境标 DIRTY,保留本例备份和原失败结果。
+First save the failure scene and the physical and state evidence, stop this case's writes and in-flight tasks, then undo this case's fault per RECOVER. When a concurrent version has already changed, do not blindly restore the old xl.meta; recovery must not serve as a product fix. Mark an unrestored environment DIRTY, and keep this case's backup and the original failure result.
-逐项满足设计公共契约、该分支步骤及附加断言后人工评审。缺资源为 BLOCKED,窗口未命中为 NOT_OBSERVED,产品错误为 FAIL,工具/恢复错误为 INFRA_ERROR;未运行不记 PASS。
+Manually review after each item satisfies the design's common contract, that branch's steps, and the additional assertions. Missing resources are BLOCKED, a window miss is NOT_OBSERVED, a product error is FAIL, and a tool or recovery error is INFRA_ERROR; a case that did not run is not recorded as PASS.
@@ -551,21 +551,21 @@ Source assertions:
Topologies: H1,H2. Branches: truncate-half, truncate-near-tail, zero-all, checksum-header.
-1. 保存外部 part 完整备份及正确 metadata quorum;每种损坏使用新的独立对象。
-2. FAULT truncate 默认保留一半;near-tail 按 FAULT_FORMAT 手册只截去最后一字节并记录原始长度;zero 清零全部;header 翻转前 32 字节。
-3. 见证实际长度/摘要差异后 START deep,捕获短读/校验诊断与责任结果。
-4. 目标原始长度和完整片摘要恢复,GET 无截断/补零,控制对象不变;拒绝把一份可解析 metadata 当内容健康。
+1. Save a complete backup of the external part and a correct metadata quorum; each kind of corruption uses a new independent object.
+2. FAULT truncate keeps half by default; near-tail, per the FAULT_FORMAT manual, removes only the last byte and records the original length; zero clears everything to zero; header flips the first 32 bytes.
+3. After witnessing the actual length and digest difference, START deep, and capture the short-read and checksum diagnostics and the responsibility result.
+4. The target's original length and complete shard digest are restored, GET has no truncation or zero-fill, and the control object is unchanged; do not treat one parseable metadata copy as healthy content.
Operations: [SEED_DB](validation-heal-operations.md#SEED_DB) → [BASELINE](validation-heal-operations.md#BASELINE) → [FAULT](validation-heal-operations.md#FAULT) → [FAULT_FORMAT](validation-heal-operations.md#FAULT_FORMAT) → [START](validation-heal-operations.md#START) → [WAIT](validation-heal-operations.md#WAIT) → [PHYSICAL](validation-heal-operations.md#PHYSICAL) → [VERIFY](validation-heal-operations.md#VERIFY)
Source assertions:
-- 分别截断 shard、置零和破坏校验头,保留正确元数据仲裁来源
-- 短读、全零和校验损坏被准确识别;修复不能接受截断内容或错误长度。
+- Separately truncate the shard, zero it, and corrupt the checksum header, keeping a correct metadata-arbitration source
+- Short reads, all-zero content, and checksum corruption are identified accurately; repair must not accept truncated content or a wrong length.
-先保存失败现场及物理/状态证据,停止本例写入与在途任务,再按 RECOVER 撤销本例故障。并发版本已变化时禁止盲目恢复旧 xl.meta;恢复不能充作产品修复。未恢复环境标 DIRTY,保留本例备份和原失败结果。
+First save the failure scene and the physical and state evidence, stop this case's writes and in-flight tasks, then undo this case's fault per RECOVER. When a concurrent version has already changed, do not blindly restore the old xl.meta; recovery must not serve as a product fix. Mark an unrestored environment DIRTY, and keep this case's backup and the original failure result.
-逐项满足设计公共契约、该分支步骤及附加断言后人工评审。缺资源为 BLOCKED,窗口未命中为 NOT_OBSERVED,产品错误为 FAIL,工具/恢复错误为 INFRA_ERROR;未运行不记 PASS。
+Manually review after each item satisfies the design's common contract, that branch's steps, and the additional assertions. Missing resources are BLOCKED, a window miss is NOT_OBSERVED, a product error is FAIL, and a tool or recovery error is INFRA_ERROR; a case that did not run is not recorded as PASS.
@@ -573,21 +573,21 @@ Source assertions:
Topologies: H1,H2. Branches: missing-xlmeta, truncated-xlmeta, corrupt-xlmeta, inline-payload, versioned-xlmeta.
-1. DB 小对象与 DV 历史版本先全量读取,FORMAT 解析每盘 xl.meta 的版本/inline/data_dir 关联并保存解析器源码 SHA。
-2. 在少数成员上对精确对象 xl.meta 做 missing/truncate/header;inline 实例仅由匹配候选的解析器定位 payload 后翻位,不能随便偏移后称 inline。
-3. 保留同版本正确来源,START deep/WAIT;恢复后重新解析 xl.meta,核对所有版本、marker 与 data 引用。
-4. 每个 versionId 全 GET 和版本集合一致;可以有合法重编码,但不能新建空 metadata、遗漏历史版本或移动删除语义。
+1. First read the DB small objects and the DV historical versions in full; FORMAT parses the version/inline/data_dir linkage in each disk's xl.meta and saves the parser source SHA.
+2. On a minority of members, apply missing/truncate/header to the exact object's xl.meta; the inline instance flips a bit only after the parser that matches the candidate has located the payload, and must not pick an arbitrary offset and call it inline.
+3. Keep correct sources of the same version, and START deep/WAIT; after recovery, parse xl.meta again and check every version, marker, and data reference.
+4. A full GET of every versionId agrees with the version set; legal re-encoding is allowed, but it must not create empty metadata, omit a historical version, or move delete semantics.
Operations: [SEED_DB](validation-heal-operations.md#SEED_DB) → [SEED_DV](validation-heal-operations.md#SEED_DV) → [FORMAT](validation-heal-operations.md#FORMAT) → [FAULT](validation-heal-operations.md#FAULT) → [FAULT_FORMAT](validation-heal-operations.md#FAULT_FORMAT) → [START](validation-heal-operations.md#START) → [WAIT](validation-heal-operations.md#WAIT) → [PHYSICAL](validation-heal-operations.md#PHYSICAL) → [VERIFY](validation-heal-operations.md#VERIFY)
Source assertions:
-- 在少数成员上移走或损坏 xl.meta/inline 内容,保留足够权威来源
-- 恢复同一对象的全部应存版本、数据引用与删除标记;不以新建空元数据替代真正修复。
+- On a minority of members, remove or corrupt xl.meta/inline content, keeping enough authority sources
+- Restore every version that should exist for the same object, together with its data references and delete markers; do not substitute newly created empty metadata for a real repair.
-先保存失败现场及物理/状态证据,停止本例写入与在途任务,再按 RECOVER 撤销本例故障。并发版本已变化时禁止盲目恢复旧 xl.meta;恢复不能充作产品修复。未恢复环境标 DIRTY,保留本例备份和原失败结果。
+First save the failure scene and the physical and state evidence, stop this case's writes and in-flight tasks, then undo this case's fault per RECOVER. When a concurrent version has already changed, do not blindly restore the old xl.meta; recovery must not serve as a product fix. Mark an unrestored environment DIRTY, and keep this case's backup and the original failure result.
-逐项满足设计公共契约、该分支步骤及附加断言后人工评审。缺资源为 BLOCKED,窗口未命中为 NOT_OBSERVED,产品错误为 FAIL,工具/恢复错误为 INFRA_ERROR;未运行不记 PASS。
+Manually review after each item satisfies the design's common contract, that branch's steps, and the additional assertions. Missing resources are BLOCKED, a window miss is NOT_OBSERVED, a product error is FAIL, and a tool or recovery error is INFRA_ERROR; a case that did not run is not recorded as PASS.
@@ -595,21 +595,21 @@ Source assertions:
Topologies: H1,H2. Branches: unversioned-overwrite, delete-historical-version, delete-marker.
-1. 保存 DV/旧代次,DRIVE_GATE 让一个成员离线且其旧数据原样保留。
-2. 在线 quorum 执行新内容覆盖、精确旧 versionId 删除或创建 marker;串行 ACK 确定因果顺序,并在在线端确认提交。
-3. 恢复携带旧数据的同一盘,核对原 disk UUID 后观察手工/自动修复,并保存旧/新代次权威选择日志。
-4. 最终最新值与全部历史/marker 严格匹配 ACK 清单;已删除版本不复活;目标旧盘同一 key 的 metadata 与内容纠正。
+1. Save the DV and the old generation; DRIVE_GATE takes one member offline while its old data is kept as-is.
+2. The online quorum performs a new-content overwrite, a delete of the exact old versionId, or marker creation; serial ACKs determine the causal order, and the commit is confirmed on the online side.
+3. Restore the same disk that carries the old data, check the original disk UUID, then observe manual or automatic repair, and save the authority-selection log for the old and new generations.
+4. The final latest value and the full history and markers strictly match the ACK manifest; a deleted version does not come back; metadata and content for the same key on the target old disk are corrected.
Operations: [SEED_DV](validation-heal-operations.md#SEED_DV) → [DRIVE_GATE](validation-heal-operations.md#DRIVE_GATE) → [WORKLOAD](validation-heal-operations.md#WORKLOAD) → [BACKGROUND](validation-heal-operations.md#BACKGROUND) → [START](validation-heal-operations.md#START) → [PHYSICAL](validation-heal-operations.md#PHYSICAL) → [VERIFY](validation-heal-operations.md#VERIFY)
Source assertions:
-- 盘离线后执行覆盖/版本删除,再让携带旧元数据的盘重入
-- 以已确认的新代次/版本为准;旧当前值与已删版本不复活,旧盘被纠正而非成为新的错误权威。
+- After a disk goes offline, perform an overwrite or a version delete, then let the disk that carries the old metadata rejoin
+- The confirmed new generation/version is authoritative; the old current value and deleted versions are not resurrected, and the old disk is corrected rather than becoming a new incorrect authority.
-先保存失败现场及物理/状态证据,停止本例写入与在途任务,再按 RECOVER 撤销本例故障。并发版本已变化时禁止盲目恢复旧 xl.meta;恢复不能充作产品修复。未恢复环境标 DIRTY,保留本例备份和原失败结果。
+First save the failure scene and the physical and state evidence, stop this case's writes and in-flight tasks, then undo this case's fault per RECOVER. When a concurrent version has already changed, do not blindly restore the old xl.meta; recovery must not serve as a product fix. Mark an unrestored environment DIRTY, and keep this case's backup and the original failure result.
-逐项满足设计公共契约、该分支步骤及附加断言后人工评审。缺资源为 BLOCKED,窗口未命中为 NOT_OBSERVED,产品错误为 FAIL,工具/恢复错误为 INFRA_ERROR;未运行不记 PASS。
+Manually review after each item satisfies the design's common contract, that branch's steps, and the additional assertions. Missing resources are BLOCKED, a window miss is NOT_OBSERVED, a product error is FAIL, and a tool or recovery error is INFRA_ERROR; a case that did not run is not recorded as PASS.
@@ -617,21 +617,21 @@ Source assertions:
Topologies: H1,H2. Branches: disjoint-version-quorums, disjoint-incarnations, native-authority-fence.
-1. 同一个本例 key 串行写两个不同版本/代次,保存各自成员副本;按 FORMAT_BAD 在静止服务下组合为两个各小于 R 的集合,原片全部保留。
-2. 解析每个成员版本身份,生成按 version/incarnation 分组的计数表;即使总数达到 R,也证明每一组都没有 quorum。
-3. GET/deep heal 必须安全拒绝,不以较新 mtime 或跨版本片并集提交;检查原本仍可恢复的证据没有被 remove。
-4. 运行匹配源代码的 NATIVE 权威仲裁回归;在真机解除混合现场后恢复原夹具并 VERIFY,两类证据分开计数。
+1. Serially write two different versions or generations to this case's same key, and save each member's copy; per FORMAT_BAD, with the service at rest, combine them into two sets each smaller than R, keeping every original shard.
+2. Parse each member's version identity and produce a count table grouped by version/incarnation; even if the total reaches R, prove that every group has no quorum.
+3. GET and deep heal must reject safely, and must not commit from a newer mtime or from the union of shards across versions; check that evidence which was still recoverable has not had remove applied.
+4. Run the NATIVE authority-arbitration regression that matches the source code; on the fleet, after the mixed scene is cleared, restore the original fixture and VERIFY, and count the two kinds of evidence separately.
Operations: [SEED_DV](validation-heal-operations.md#SEED_DV) → [FORMAT](validation-heal-operations.md#FORMAT) → [FORMAT_BAD](validation-heal-operations.md#FORMAT_BAD) → [GET](validation-heal-operations.md#GET) → [START](validation-heal-operations.md#START) → [WAIT](validation-heal-operations.md#WAIT) → [PHYSICAL](validation-heal-operations.md#PHYSICAL) → [RECOVER](validation-heal-operations.md#RECOVER) → [VERIFY](validation-heal-operations.md#VERIFY) → [NATIVE](validation-heal-operations.md#NATIVE)
Source assertions:
-- 用受控原生夹具构造互不构成 quorum 的版本/代次集合,再做对应真机反例
-- 禁止把不同版本的 shard 数量并集当成权威;无法证明时安全拒绝且保留可恢复证据,不猜测较新时间戳。
+- Use a controlled native fixture to construct version/generation sets that do not form a quorum with each other, then run the corresponding live counterexample on the fleet
+- Do not treat the union of shard counts from different versions as authority; when it cannot be proven, reject safely and retain recoverable evidence, and do not guess a newer timestamp.
-先保存失败现场及物理/状态证据,停止本例写入与在途任务,再按 RECOVER 撤销本例故障。并发版本已变化时禁止盲目恢复旧 xl.meta;恢复不能充作产品修复。未恢复环境标 DIRTY,保留本例备份和原失败结果。
+First save the failure scene and the physical and state evidence, stop this case's writes and in-flight tasks, then undo this case's fault per RECOVER. When a concurrent version has already changed, do not blindly restore the old xl.meta; recovery must not serve as a product fix. Mark an unrestored environment DIRTY, and keep this case's backup and the original failure result.
-逐项满足设计公共契约、该分支步骤及附加断言后人工评审。缺资源为 BLOCKED,窗口未命中为 NOT_OBSERVED,产品错误为 FAIL,工具/恢复错误为 INFRA_ERROR;未运行不记 PASS。
+Manually review after each item satisfies the design's common contract, that branch's steps, and the additional assertions. Missing resources are BLOCKED, a window miss is NOT_OBSERVED, a product error is FAIL, and a tool or recovery error is INFRA_ERROR; a case that did not run is not recorded as PASS.
@@ -639,21 +639,21 @@ Source assertions:
Topologies: H1,H2. Branches: length-mismatch, payload-mismatch, insufficient-authority.
-1. 先保存版本化外部 part 的全套 metadata 与 hash;从本例另一种长度/内容的版本取得替换 payload。
-2. 停服务按 FAULT_FORMAT 只更换少数 part 的内容或截断,保留原 xl.meta 声称的长度/摘要,不改健康成员。
-3. deep heal 对比 metadata 与实际内容;充分权威实例应重建,不充分权威实例应拒绝且保留好的副本。
-4. 取证后恢复,验证精确版本无跨代内容;保留实际解析结果,不把任意无效 header 注入当作长度字段覆盖。
+1. First save the full metadata and the hash of the versioned external part; take the replacement payload from another version of this case with a different length or content.
+2. With the service stopped, per FAULT_FORMAT, replace the content of or truncate only a minority of parts, keep the length and digest claimed by the original xl.meta, and do not change healthy members.
+3. deep heal compares the metadata with the actual content; an instance with sufficient authority should rebuild the shard, and an instance with insufficient authority should reject and keep the good copies.
+4. After evidence collection, recover, and verify that the exact version has no cross-generation content; keep the actual parse result, and do not treat an arbitrary invalid-header injection as an overwrite of the length field.
Operations: [SEED_DB](validation-heal-operations.md#SEED_DB) → [SEED_DV](validation-heal-operations.md#SEED_DV) → [FORMAT](validation-heal-operations.md#FORMAT) → [FAULT_FORMAT](validation-heal-operations.md#FAULT_FORMAT) → [START](validation-heal-operations.md#START) → [WAIT](validation-heal-operations.md#WAIT) → [PHYSICAL](validation-heal-operations.md#PHYSICAL) → [RECOVER](validation-heal-operations.md#RECOVER) → [VERIFY](validation-heal-operations.md#VERIFY)
Source assertions:
-- 构造元数据声称某长度/摘要但实际 part 不匹配的少数副本
-- 元数据与内容联合判断;不把可解析的 xl.meta 当健康;权威不足时拒绝覆盖正确副本。
+- Construct a minority of replicas whose metadata claims a length or digest that the actual part does not match
+- Judge metadata and content together; do not treat a parseable xl.meta as healthy; when authority is insufficient, refuse to overwrite a correct replica.
-先保存失败现场及物理/状态证据,停止本例写入与在途任务,再按 RECOVER 撤销本例故障。并发版本已变化时禁止盲目恢复旧 xl.meta;恢复不能充作产品修复。未恢复环境标 DIRTY,保留本例备份和原失败结果。
+First save the failure scene and the physical and state evidence, stop this case's writes and in-flight tasks, then undo this case's fault per RECOVER. When a concurrent version has already changed, do not blindly restore the old xl.meta; recovery must not serve as a product fix. Mark an unrestored environment DIRTY, and keep this case's backup and the original failure result.
-逐项满足设计公共契约、该分支步骤及附加断言后人工评审。缺资源为 BLOCKED,窗口未命中为 NOT_OBSERVED,产品错误为 FAIL,工具/恢复错误为 INFRA_ERROR;未运行不记 PASS。
+Manually review after each item satisfies the design's common contract, that branch's steps, and the additional assertions. Missing resources are BLOCKED, a window miss is NOT_OBSERVED, a product error is FAIL, and a tool or recovery error is INFRA_ERROR; a case that did not run is not recorded as PASS.
@@ -661,21 +661,21 @@ Source assertions:
Topologies: H1,H2. Branches: foreign-disk, foreign-set, future-format, future-object-schema.
-1. 用第二个隔离测试集群生成真实外来 disk/set format;当前实例只有一个替换逻辑根可被修改。
-2. FORMAT_BAD 在服务停止时分别装入外来 format 或按候选解析器支持的版本头写入未来 schema;记录原始 bytes/digest,不碰共享盘 format。
-3. 启动、GET/deep heal,核对明确隔离/unsupported,不得把未知格式当 missing 初始化或写成新集群。
-4. 停止服务收集原记录仍存在的物理证据;撤回坏根、恢复原身份后完整 VERIFY;无法确认格式字段不能冒充已注入。
+1. Use a second isolated test cluster to generate a real foreign disk/set format; the current instance has only one replacement logical root that may be modified.
+2. While the service is stopped, FORMAT_BAD separately installs a foreign format or writes a future schema using the version header the candidate parser supports; record the original bytes/digest, and do not touch the shared disk's format.
+3. Start, then GET and deep heal, and check for explicit isolation or unsupported; an unknown format must not be initialized as missing or written out as a new cluster.
+4. Stop the service and collect physical evidence that the original records are still present; withdraw the bad root, restore the original identity, then run a full VERIFY; a format field that cannot be confirmed must not be passed off as injected.
Operations: [SEED_DB](validation-heal-operations.md#SEED_DB) → [FORMAT](validation-heal-operations.md#FORMAT) → [REPLACE](validation-heal-operations.md#REPLACE) → [FORMAT_BAD](validation-heal-operations.md#FORMAT_BAD) → [PROCESS](validation-heal-operations.md#PROCESS) → [START](validation-heal-operations.md#START) → [PHYSICAL](validation-heal-operations.md#PHYSICAL) → [RECOVER](validation-heal-operations.md#RECOVER) → [VERIFY](validation-heal-operations.md#VERIFY)
Source assertions:
-- 提供外来 disk/set 身份、未来 schema 或不支持的记录格式
-- 明确隔离/拒绝且不破坏原记录,不把未知当 missing 后直接重建;解除故障后原数据可用。
+- Provide a foreign disk/set identity, a future schema, or an unsupported record format
+- Explicitly isolate or reject without damaging the original record, and do not treat unknown as missing and then rebuild immediately; after the fault is cleared, the original data is usable.
-先保存失败现场及物理/状态证据,停止本例写入与在途任务,再按 RECOVER 撤销本例故障。并发版本已变化时禁止盲目恢复旧 xl.meta;恢复不能充作产品修复。未恢复环境标 DIRTY,保留本例备份和原失败结果。
+First save the failure scene and the physical and state evidence, stop this case's writes and in-flight tasks, then undo this case's fault per RECOVER. When a concurrent version has already changed, do not blindly restore the old xl.meta; recovery must not serve as a product fix. Mark an unrestored environment DIRTY, and keep this case's backup and the original failure result.
-逐项满足设计公共契约、该分支步骤及附加断言后人工评审。缺资源为 BLOCKED,窗口未命中为 NOT_OBSERVED,产品错误为 FAIL,工具/恢复错误为 INFRA_ERROR;未运行不记 PASS。
+Manually review after each item satisfies the design's common contract, that branch's steps, and the additional assertions. Missing resources are BLOCKED, a window miss is NOT_OBSERVED, a product error is FAIL, and a tool or recovery error is INFRA_ERROR; a case that did not run is not recorded as PASS.
@@ -683,21 +683,21 @@ Source assertions:
Topologies: H1,H2. Branches: remove-false, remove-true-before-grace, remove-true-after-grace, shared-version-reference, pending-mpu.
-1. 创建 DV/DM 并合法覆盖/删除形成候选孤儿;以匹配格式解析器列出所有 version→data_dir 引用和 pending UploadId。
-2. 只在没有活引用的本次测试目录形成 dangling;保存 orphan 成因与真实年龄,宽限期实例等待配置窗口,不改全局时间。
-3. 分别 START remove=false/true;宽限期前、共享历史引用、pending MPU 必须保留,过期且有充分删除证明的孤儿才允许清理。
-4. 比较物理引用集合,所有应存版本及未完成 upload list_parts 完整;remove 误删任何共享引用即失败,救援只在取证后。
+1. Create DV/DM and form candidate orphans by a legal overwrite or delete; with the matching format parser, list every version→data_dir reference and pending UploadId.
+2. Form dangling only in this test's directories that have no live reference; save the orphan's cause and its real age, and have the grace-period instance wait for the configured window without changing the global time.
+3. START remove=false and remove=true separately; items still within the grace period, shared historical references, and a pending MPU must be kept, and only an orphan that has expired and has sufficient delete proof may be cleaned up.
+4. Compare the physical reference set: every version that should remain, and list_parts of incomplete uploads, are complete; if remove deletes any shared reference by mistake, it fails, and rescue happens only after evidence collection.
Operations: [SEED_DV](validation-heal-operations.md#SEED_DV) → [SEED_DM](validation-heal-operations.md#SEED_DM) → [FORMAT](validation-heal-operations.md#FORMAT) → [FAULT_FORMAT](validation-heal-operations.md#FAULT_FORMAT) → [START](validation-heal-operations.md#START) → [WAIT](validation-heal-operations.md#WAIT) → [PHYSICAL](validation-heal-operations.md#PHYSICAL) → [VERIFY](validation-heal-operations.md#VERIFY)
Source assertions:
-- 构造 dangling part、旧 data_dir 与共享历史引用,分别 remove=false/true、宽限期前后检查
-- 默认不得破坏仍可恢复数据;显式 remove 只清除具有充分删除证明且过期的本次孤儿,不删共享版本或 MPU。
+- Construct a dangling part, an old data_dir, and a shared historical reference, and check remove=false and remove=true separately, before and after the grace period
+- By default, do not destroy data that can still be recovered; an explicit remove clears only this orphan, and only when it has sufficient delete proof and has expired, and does not delete a shared version or an MPU.
-先保存失败现场及物理/状态证据,停止本例写入与在途任务,再按 RECOVER 撤销本例故障。并发版本已变化时禁止盲目恢复旧 xl.meta;恢复不能充作产品修复。未恢复环境标 DIRTY,保留本例备份和原失败结果。
+First save the failure scene and the physical and state evidence, stop this case's writes and in-flight tasks, then undo this case's fault per RECOVER. When a concurrent version has already changed, do not blindly restore the old xl.meta; recovery must not serve as a product fix. Mark an unrestored environment DIRTY, and keep this case's backup and the original failure result.
-逐项满足设计公共契约、该分支步骤及附加断言后人工评审。缺资源为 BLOCKED,窗口未命中为 NOT_OBSERVED,产品错误为 FAIL,工具/恢复错误为 INFRA_ERROR;未运行不记 PASS。
+Manually review after each item satisfies the design's common contract, that branch's steps, and the additional assertions. Missing resources are BLOCKED, a window miss is NOT_OBSERVED, a product error is FAIL, and a tool or recovery error is INFRA_ERROR; a case that did not run is not recorded as PASS.
@@ -707,23 +707,23 @@ Topologies: H1,H2. Branches: old-metadata-rejoin, old-checkpoint-replay, old-mrf
Execution policy: `evidence-blocked`; disabled by default: `false`.
-Blocked reason (`EVIDENCE_INCARNATION_LIVE_ORCHESTRATION`): 已实现 H1/H2 旧元数据重返、真实 checkpoint/resume 回放、ACK 绑定的 committed MRF 回放和精确 native 证据门禁;尚缺全部现场分支的独立审计,诊断执行保持 FAIL/BLOCKED,原生前置通过不能替代 canonical 验收。
+Blocked reason (`EVIDENCE_INCARNATION_LIVE_ORCHESTRATION`): H1/H2 old-metadata rejoin, real checkpoint/resume replay, ACK-bound committed MRF replay, and the precise native evidence gate are implemented; an independent audit of every live branch is still missing, diagnostic execution stays FAIL/BLOCKED, and a passing native prerequisite cannot replace canonical acceptance.
-1. 建旧 DV 桶并让一盘离线;按 S3 正常列举/删除全部版本和 marker,确认 delete_bucket 成功,禁止直接内部清空绕过删除。
-2. 同名 create_bucket 后写新内容,保存新 incarnation;恢复旧盘并让旧任务/持久 MRF 重放。
-3. 对比任务身份和代次守卫,旧责任不得删除/覆盖新对象,不得把旧 marker 或数据复活到新桶。
-4. 按 NATIVE 验证精确 incarnation 提交屏障;真机清单及新桶配置始终匹配成功 ACK,旧/新物理范围独立留证。
+1. Create an old DV bucket and take one disk offline; list and delete every version and marker through normal S3, confirm that delete_bucket succeeds, and do not wipe internally to bypass deletion.
+2. After create_bucket with the same name, write new content and save the new incarnation; restore the old disk and let the old task and the persistent MRF replay.
+3. Compare the task identity and the generation guard: old responsibility must not delete or overwrite the new object, and must not resurrect an old marker or old data into the new bucket.
+4. Per NATIVE, verify the commit fence of the exact incarnation; on the fleet, the manifest and the new bucket configuration always match the successful ACKs, and the old and new physical ranges are kept as separate evidence.
Operations: [SEED_DV](validation-heal-operations.md#SEED_DV) → [DRIVE_GATE](validation-heal-operations.md#DRIVE_GATE) → [S3](validation-heal-operations.md#S3) → [BACKGROUND](validation-heal-operations.md#BACKGROUND) → [START](validation-heal-operations.md#START) → [FORMAT](validation-heal-operations.md#FORMAT) → [PHYSICAL](validation-heal-operations.md#PHYSICAL) → [VERIFY](validation-heal-operations.md#VERIFY) → [NATIVE](validation-heal-operations.md#NATIVE)
Source assertions:
-- 盘离线时完成旧桶合法清空/删除,再同名建桶写入新对象,恢复旧盘与旧任务
-- 旧 bucket incarnation 的 checkpoint/MRF/元数据不得污染新桶;旧任务不得删除或恢复新 incarnation 的对象。
+- While a disk is offline, complete a legitimate empty or delete of the old bucket, then create a bucket of the same name and write new objects, and bring the old disk and the old tasks back
+- The old bucket incarnation's checkpoint/MRF/metadata must not contaminate the new bucket; old tasks must not delete or restore objects of the new incarnation.
-先保存失败现场及物理/状态证据,停止本例写入与在途任务,再按 RECOVER 撤销本例故障。并发版本已变化时禁止盲目恢复旧 xl.meta;恢复不能充作产品修复。未恢复环境标 DIRTY,保留本例备份和原失败结果。
+First save the failure scene and the physical and state evidence, stop this case's writes and in-flight tasks, then undo this case's fault per RECOVER. When a concurrent version has already changed, do not blindly restore the old xl.meta; recovery must not serve as a product fix. Mark an unrestored environment DIRTY, and keep this case's backup and the original failure result.
-逐项满足设计公共契约、该分支步骤及附加断言后人工评审。缺资源为 BLOCKED,窗口未命中为 NOT_OBSERVED,产品错误为 FAIL,工具/恢复错误为 INFRA_ERROR;未运行不记 PASS。
+Manually review after each item satisfies the design's common contract, that branch's steps, and the additional assertions. Missing resources are BLOCKED, a window miss is NOT_OBSERVED, a product error is FAIL, and a tool or recovery error is INFRA_ERROR; a case that did not run is not recorded as PASS.
@@ -731,21 +731,21 @@ Source assertions:
Topologies: H1,H2. Branches: zero, one, 4kib-minus-exact-plus, inline-minus-exact-plus, stripe-minus-exact-plus, large-external.
-1. SEED_DB 显式包含 0、1、4095、4096、4097;从固定候选配置/FORMAT 获得 inline 与 EC block 阈值,新增阈值±1 和一个多 block 外部对象。
-2. 每个大小先按物理记录确认实际形态;零字节/inline 没有外部 part 时改用合法 metadata/inline 故障,不能虚构 part。
-3. data/parity 上依次 missing 与 bitflip,START deep/WAIT;每个形态独立 variant 计数。
-4. 核对全体长度/摘要/ETag/HEAD/LIST;跨边界及末字节 Range 为正确 206,空对象范围按冻结契约拒绝。
+1. SEED_DB explicitly includes 0, 1, 4095, 4096, and 4097; obtain the inline and EC block thresholds from the fixed candidate configuration and FORMAT, and add threshold±1 and one multi-block external object.
+2. For each size, first confirm the actual form from the physical record; when a zero-byte or inline object has no external part, switch to a legal metadata or inline fault, and do not invent a part.
+3. On data and on parity, apply missing and then bitflip, and START deep/WAIT; count each form as an independent variant.
+4. Check every length, digest, ETag, HEAD, and LIST; a Range that crosses a boundary and a Range of the last byte are a correct 206, and a range on an empty object is rejected per the frozen contract.
Operations: [SEED_DB](validation-heal-operations.md#SEED_DB) → [FORMAT](validation-heal-operations.md#FORMAT) → [FAULT](validation-heal-operations.md#FAULT) → [FAULT_FORMAT](validation-heal-operations.md#FAULT_FORMAT) → [START](validation-heal-operations.md#START) → [WAIT](validation-heal-operations.md#WAIT) → [RANGE](validation-heal-operations.md#RANGE) → [VERIFY](validation-heal-operations.md#VERIFY)
Source assertions:
-- 覆盖 0/1 字节、页/inline/EC block 边界±1以及大对象
-- 每个实际存储形态均确认后注入;长度、SHA-256、ETag、HEAD/LIST 与边界 Range 正确,无假设阈值。
+- Cover 0-byte and 1-byte objects, page/inline/EC block boundaries ±1, and large objects
+- Inject only after each actual storage form is confirmed; length, SHA-256, ETag, HEAD/LIST, and boundary Range are correct, with no assumed threshold.
-先保存失败现场及物理/状态证据,停止本例写入与在途任务,再按 RECOVER 撤销本例故障。并发版本已变化时禁止盲目恢复旧 xl.meta;恢复不能充作产品修复。未恢复环境标 DIRTY,保留本例备份和原失败结果。
+First save the failure scene and the physical and state evidence, stop this case's writes and in-flight tasks, then undo this case's fault per RECOVER. When a concurrent version has already changed, do not blindly restore the old xl.meta; recovery must not serve as a product fix. Mark an unrestored environment DIRTY, and keep this case's backup and the original failure result.
-逐项满足设计公共契约、该分支步骤及附加断言后人工评审。缺资源为 BLOCKED,窗口未命中为 NOT_OBSERVED,产品错误为 FAIL,工具/恢复错误为 INFRA_ERROR;未运行不记 PASS。
+Manually review after each item satisfies the design's common contract, that branch's steps, and the additional assertions. Missing resources are BLOCKED, a window miss is NOT_OBSERVED, a product error is FAIL, and a tool or recovery error is INFRA_ERROR; a case that did not run is not recorded as PASS.
@@ -753,21 +753,21 @@ Source assertions:
Topologies: H1,H2. Branches: first-part, middle-part, last-part, cross-part-range.
-1. SEED_DM 建至少三 part 且非末 part≥5 MiB,非等长内容和独立 nonce;保存 UploadId、每个 part ETag、Complete ACK/最终版本。
-2. 每实例损坏第一、中间、末尾 part 对应的一个物理片;按 FORMAT 映射实际 part,不依赖文件名猜逻辑位置。
-3. START deep/WAIT 后先验证对应目标片,再完整 GET;最终 ETag 应与基线一致,但不用 multipart ETag 计算整对象 MD5。
-4. RANGE 覆盖两个 part 边界及尾部,比较逐字节/摘要、metadata 和版本;其他未完成 MPU 仍可 list_parts。
+1. SEED_DM creates at least three parts, with every non-final part ≥5 MiB, unequal content lengths, and independent nonces; save the UploadId, each part's ETag, and the Complete ACK and final version.
+2. Each instance corrupts one physical shard corresponding to the first, the middle, or the last part; map the actual part per FORMAT, and do not guess the logical position from the file name.
+3. After START deep/WAIT, first verify the corresponding target shard, then do a full GET; the final ETag should match the baseline, but do not compute the whole-object MD5 from the multipart ETag.
+4. RANGE covers two part boundaries and the tail; compare byte-by-byte/digest, metadata, and version; other incomplete MPUs can still list_parts.
Operations: [SEED_DM](validation-heal-operations.md#SEED_DM) → [FORMAT](validation-heal-operations.md#FORMAT) → [FAULT](validation-heal-operations.md#FAULT) → [START](validation-heal-operations.md#START) → [WAIT](validation-heal-operations.md#WAIT) → [PHYSICAL](validation-heal-operations.md#PHYSICAL) → [RANGE](validation-heal-operations.md#RANGE) → [VERIFY](validation-heal-operations.md#VERIFY)
Source assertions:
-- 修复已完成的不等长多 part 对象,分别损坏首/中/末 part
-- 完整内容、part 顺序、原版本/ETag/元数据和跨 part Range 不变;multipart ETag 不能当整对象 MD5。
+- Repair a completed multipart object with unequal part lengths, corrupting the first, middle, and last part separately
+- Full content, part order, the original version/ETag/metadata, and cross-part Range stay unchanged; a multipart ETag must not be treated as the whole-object MD5.
-先保存失败现场及物理/状态证据,停止本例写入与在途任务,再按 RECOVER 撤销本例故障。并发版本已变化时禁止盲目恢复旧 xl.meta;恢复不能充作产品修复。未恢复环境标 DIRTY,保留本例备份和原失败结果。
+First save the failure scene and the physical and state evidence, stop this case's writes and in-flight tasks, then undo this case's fault per RECOVER. When a concurrent version has already changed, do not blindly restore the old xl.meta; recovery must not serve as a product fix. Mark an unrestored environment DIRTY, and keep this case's backup and the original failure result.
-逐项满足设计公共契约、该分支步骤及附加断言后人工评审。缺资源为 BLOCKED,窗口未命中为 NOT_OBSERVED,产品错误为 FAIL,工具/恢复错误为 INFRA_ERROR;未运行不记 PASS。
+Manually review after each item satisfies the design's common contract, that branch's steps, and the additional assertions. Missing resources are BLOCKED, a window miss is NOT_OBSERVED, a product error is FAIL, and a tool or recovery error is INFRA_ERROR; a case that did not run is not recorded as PASS.
@@ -775,21 +775,21 @@ Source assertions:
Topologies: H1,H2. Branches: old-version-part, current-version-part, recursive-all-versions, multiple-version-targets.
-1. SEED_DV 每 key 三个不同 payload 版本,逐一保存 versionId/ETag;可额外写 marker,清单不按当前 GET 代替历史读取。
-2. 为不同历史/当前版本选不同盘和 part,metadata 故障实例保留同版本 quorum。
-3. 对 object 和 recursive bucket 分别 START/WAIT;物理解析必须显示所有应存 version/data_dir 而不是只最新对象。
-4. 对每个原 versionId 全 GET,LIST versions 无新增/遗漏,isLatest 身份与 marker 顺序保持。
+1. SEED_DV writes three different payload versions per key and saves each versionId/ETag; a marker may also be written. The manifest must not substitute the current GET for historical reads.
+2. Select different disks and parts for different historical/current versions; the metadata-fault instance retains a same-version quorum.
+3. START/WAIT separately for the object and the recursive bucket; physical parsing must show every version/data_dir that should be stored, not only the latest object.
+4. Full GET of every original versionId; LIST versions has no additions or omissions; isLatest identity and marker order are preserved.
Operations: [SEED_DV](validation-heal-operations.md#SEED_DV) → [FORMAT](validation-heal-operations.md#FORMAT) → [FAULT](validation-heal-operations.md#FAULT) → [START](validation-heal-operations.md#START) → [WAIT](validation-heal-operations.md#WAIT) → [PHYSICAL](validation-heal-operations.md#PHYSICAL) → [VERIFY](validation-heal-operations.md#VERIFY)
Source assertions:
-- 不同版本分布不同损坏位置,按对象和递归范围修复
-- 所有原 versionId 可寻址且内容正确,最新版本身份不漂移,无多余版本或遗漏历史版本。
+- Place different corruption locations on different versions, and repair by object and by recursive scope
+- Every original versionId is addressable and its content is correct, the latest version identity does not drift, and there are no extra versions or omitted historical versions.
-先保存失败现场及物理/状态证据,停止本例写入与在途任务,再按 RECOVER 撤销本例故障。并发版本已变化时禁止盲目恢复旧 xl.meta;恢复不能充作产品修复。未恢复环境标 DIRTY,保留本例备份和原失败结果。
+First save the failure scene and the physical and state evidence, stop this case's writes and in-flight tasks, then undo this case's fault per RECOVER. When a concurrent version has already changed, do not blindly restore the old xl.meta; recovery must not serve as a product fix. Mark an unrestored environment DIRTY, and keep this case's backup and the original failure result.
-逐项满足设计公共契约、该分支步骤及附加断言后人工评审。缺资源为 BLOCKED,窗口未命中为 NOT_OBSERVED,产品错误为 FAIL,工具/恢复错误为 INFRA_ERROR;未运行不记 PASS。
+Manually review after each item satisfies the design's common contract, that branch's steps, and the additional assertions. Missing resources are BLOCKED, a window miss is NOT_OBSERVED, a product error is FAIL, and a tool or recovery error is INFRA_ERROR; a case that did not run is not recorded as PASS.
@@ -797,21 +797,21 @@ Source assertions:
Topologies: H1,H2. Branches: current-marker, historical-marker, missing-marker-replica, stale-disk-marker.
-1. 创建三个数据版本与当前/历史 marker,记录 delete_object 返回的 VersionId/DeleteMarker。
-2. 缺少 marker 的实例使用 xl.meta 少数副本缺失;陈旧实例通过成员离线后合法创建/删除 marker 形成,不能直接改任意 metadata。
-3. heal 后 current GET/HEAD 对当前 marker 应为 404 且正确 marker 身份头;指定 marker versionId 的 GET 按 S3 契约为 405 并保留版本头。
-4. LIST versions 精确集合与最新标记一致;显式删除当前 marker 后露出的应是预期历史对象,不能产生额外数据版本。
+1. Create three data versions and current/historical markers, and record the VersionId/DeleteMarker returned by delete_object.
+2. The instance missing a marker uses a minority of xl.meta replicas missing; the stale instance is formed by legitimately creating/deleting a marker after a member goes offline, and arbitrary metadata must not be edited directly.
+3. After heal, current GET/HEAD of the current marker should be 404 with the correct marker identity headers; GET of a specified marker versionId is 405 per the S3 contract and retains the version headers.
+4. The exact LIST versions set matches the latest marker; after the current marker is explicitly deleted, what is exposed should be the expected historical object, and no extra data version may be produced.
Operations: [SEED_DV](validation-heal-operations.md#SEED_DV) → [S3](validation-heal-operations.md#S3) → [FAULT](validation-heal-operations.md#FAULT) → [DRIVE_GATE](validation-heal-operations.md#DRIVE_GATE) → [START](validation-heal-operations.md#START) → [WAIT](validation-heal-operations.md#WAIT) → [VERIFY](validation-heal-operations.md#VERIFY)
Source assertions:
-- 缺失/陈旧 marker 副本后 heal,并按当前值与精确 marker versionId 查询
-- 当前 GET/HEAD 与指定 marker 的状态码及身份头符合契约;marker 不复活数据,LIST versions 无重复。
+- After marker replicas are missing or stale, heal, and query by the current value and the exact marker versionId
+- The status codes and identity headers of the current GET/HEAD and of the specified marker match the contract; the marker does not resurrect data, and LIST versions has no duplicates.
-先保存失败现场及物理/状态证据,停止本例写入与在途任务,再按 RECOVER 撤销本例故障。并发版本已变化时禁止盲目恢复旧 xl.meta;恢复不能充作产品修复。未恢复环境标 DIRTY,保留本例备份和原失败结果。
+First save the failure scene and the physical and state evidence, stop this case's writes and in-flight tasks, then undo this case's fault per RECOVER. When a concurrent version has already changed, do not blindly restore the old xl.meta; recovery must not serve as a product fix. Mark an unrestored environment DIRTY, and keep this case's backup and the original failure result.
-逐项满足设计公共契约、该分支步骤及附加断言后人工评审。缺资源为 BLOCKED,窗口未命中为 NOT_OBSERVED,产品错误为 FAIL,工具/恢复错误为 INFRA_ERROR;未运行不记 PASS。
+Manually review after each item satisfies the design's common contract, that branch's steps, and the additional assertions. Missing resources are BLOCKED, a window miss is NOT_OBSERVED, a product error is FAIL, and a tool or recovery error is INFRA_ERROR; a case that did not run is not recorded as PASS.
@@ -819,21 +819,21 @@ Source assertions:
Topologies: H1,H2. Branches: enabled-to-suspended, null-overwrite, null-delete-marker.
-1. 启用版本写三代,再 put_bucket_versioning Status=Suspended,连续写两次 null 版本并创建/删除 null marker。
-2. 基线同时保存非 null 历史和当前 null 身份;对目标外部 part 或 xl.meta 制造少数缺失。
-3. START deep/WAIT;记录 null 不能被转换成随机 versionId,也不能与启用期历史版本混片。
-4. VERIFY 全部原 versionId、null 内容和 marker;再次开启版本后写新对象,验证旧历史仍可寻址。
+1. Enable versioning and write three generations, then put_bucket_versioning with Status=Suspended, write the null version twice in succession, and create/delete a null marker.
+2. The baseline saves both the non-null history and the current null identity; induce a minority missing on the target external part or xl.meta.
+3. START deep/WAIT; record that null must not be converted into a random versionId, and must not have its shards mixed with historical versions from the versioning-enabled period.
+4. VERIFY every original versionId, the null content, and the marker; after versioning is enabled again, write a new object and verify the old history is still addressable.
Operations: [SEED_DV](validation-heal-operations.md#SEED_DV) → [S3](validation-heal-operations.md#S3) → [FORMAT](validation-heal-operations.md#FORMAT) → [FAULT](validation-heal-operations.md#FAULT) → [START](validation-heal-operations.md#START) → [WAIT](validation-heal-operations.md#WAIT) → [VERIFY](validation-heal-operations.md#VERIFY)
Source assertions:
-- 在版本开启、暂停、null 版本覆盖之间建夹具再修复
-- null 和非 null 版本不混淆,不新增假版本;暂停前历史对象仍准确可读。
+- Build the fixture across versioning enabled, suspended, and null-version overwrite, then repair
+- Null and non-null versions are not confused, and no fake version is added; historical objects from before suspension remain accurately readable.
-先保存失败现场及物理/状态证据,停止本例写入与在途任务,再按 RECOVER 撤销本例故障。并发版本已变化时禁止盲目恢复旧 xl.meta;恢复不能充作产品修复。未恢复环境标 DIRTY,保留本例备份和原失败结果。
+First save the failure scene and the physical and state evidence, stop this case's writes and in-flight tasks, then undo this case's fault per RECOVER. When a concurrent version has already changed, do not blindly restore the old xl.meta; recovery must not serve as a product fix. Mark an unrestored environment DIRTY, and keep this case's backup and the original failure result.
-逐项满足设计公共契约、该分支步骤及附加断言后人工评审。缺资源为 BLOCKED,窗口未命中为 NOT_OBSERVED,产品错误为 FAIL,工具/恢复错误为 INFRA_ERROR;未运行不记 PASS。
+Manually review after each item satisfies the design's common contract, that branch's steps, and the additional assertions. Missing resources are BLOCKED, a window miss is NOT_OBSERVED, a product error is FAIL, and a tool or recovery error is INFRA_ERROR; a case that did not run is not recorded as PASS.
@@ -841,21 +841,21 @@ Source assertions:
Topologies: H1,H2. Branches: user-metadata, content-headers, versioned-tags.
-1. 用 DS 设置中文值可编码的 metadata、Content-Type、Content-Encoding、Content-Disposition、Cache-Control 与两组 tags;内容本身使用已声明编码。
-2. 先 GET/HEAD/get_object_tagging 精确记录;对外部 part 和少数 xl.meta 分别故障。
-3. START deep/WAIT,比较所有对外属性和逐版本 tags,修复不新增版本、不丢用户 key 大小写规范化结果。
-4. 允许内部 metadata 合法重写,但必须通过 FORMAT 引用和物理内容证明;不把 Content-Encoding 当服务端压缩已启用。
+1. Use DS to set metadata, Content-Type, Content-Encoding, Content-Disposition, Cache-Control, and two tag sets to encodable Chinese values; the content itself uses the declared encoding.
+2. First record GET/HEAD/get_object_tagging exactly; fault the external part and a minority of xl.meta separately.
+3. START deep/WAIT, and compare every externally visible attribute and the per-version tags; repair adds no version and does not drop the case-normalization result of user keys.
+4. Legitimate rewrite of internal metadata is allowed, but it must be proven by FORMAT references and physical content; do not treat Content-Encoding as server-side compression being enabled.
Operations: [SEED_DS](validation-heal-operations.md#SEED_DS) → [BASELINE](validation-heal-operations.md#BASELINE) → [FAULT](validation-heal-operations.md#FAULT) → [START](validation-heal-operations.md#START) → [WAIT](validation-heal-operations.md#WAIT) → [PHYSICAL](validation-heal-operations.md#PHYSICAL) → [VERIFY](validation-heal-operations.md#VERIFY)
Source assertions:
-- 包含用户 metadata、Content-Type/Encoding/Disposition、tags 的对象修复
-- 对外 metadata、tags 和相关响应头逐项相同;不要求无关内部元数据字节完全一致。
+- Repair objects that include user metadata, Content-Type/Encoding/Disposition, and tags
+- External metadata, tags, and the related response headers match item by item; unrelated internal metadata bytes are not required to be identical.
-先保存失败现场及物理/状态证据,停止本例写入与在途任务,再按 RECOVER 撤销本例故障。并发版本已变化时禁止盲目恢复旧 xl.meta;恢复不能充作产品修复。未恢复环境标 DIRTY,保留本例备份和原失败结果。
+First save the failure scene and the physical and state evidence, stop this case's writes and in-flight tasks, then undo this case's fault per RECOVER. When a concurrent version has already changed, do not blindly restore the old xl.meta; recovery must not serve as a product fix. Mark an unrestored environment DIRTY, and keep this case's backup and the original failure result.
-逐项满足设计公共契约、该分支步骤及附加断言后人工评审。缺资源为 BLOCKED,窗口未命中为 NOT_OBSERVED,产品错误为 FAIL,工具/恢复错误为 INFRA_ERROR;未运行不记 PASS。
+Manually review after each item satisfies the design's common contract, that branch's steps, and the additional assertions. Missing resources are BLOCKED, a window miss is NOT_OBSERVED, a product error is FAIL, and a tool or recovery error is INFRA_ERROR; a case that did not run is not recorded as PASS.
@@ -863,21 +863,21 @@ Source assertions:
Topologies: H1,H2. Branches: head, range-valid, range-outside, if-match, if-none-match, if-modified-since, if-unmodified-since.
-1. DB/DM 基线保存 ETag/LastModified 和跨 part 内容;先执行所有条件请求,冻结预期 200/206/304/412/416。
-2. 对精确 part 制造缺失/坏校验并 repair;同一组 HEAD、Range、IfMatch/IfNoneMatch、时间条件再次请求。
-3. 304/412/416 验证响应无错误成功内容;合法范围比较准确长度、ContentRange 和原片段摘要。
-4. 完整 VERIFY 版本和 metadata;条件请求必须用 API 参数传递,不由本地短路模拟状态码。
+1. The DB/DM baseline saves ETag/LastModified and cross-part content; first issue all conditional requests and freeze the expected 200/206/304/412/416.
+2. Induce missing/bad checksum on the exact part and repair; issue the same set of HEAD, Range, IfMatch/IfNoneMatch, and time-condition requests again.
+3. For 304/412/416, verify the response contains no erroneous success content; for a valid range, compare the exact length, ContentRange, and the original fragment digest.
+4. Fully VERIFY the version and metadata; conditional requests must be passed as API parameters, and status codes must not be simulated by a local short-circuit.
Operations: [SEED_DB](validation-heal-operations.md#SEED_DB) → [SEED_DM](validation-heal-operations.md#SEED_DM) → [RANGE](validation-heal-operations.md#RANGE) → [FAULT](validation-heal-operations.md#FAULT) → [START](validation-heal-operations.md#START) → [WAIT](validation-heal-operations.md#WAIT) → [RANGE](validation-heal-operations.md#RANGE) → [VERIFY](validation-heal-operations.md#VERIFY)
Source assertions:
-- 修复前后验证 HEAD、合法/越界 Range、If-Match/If-None-Match 及时间条件
-- 200/206/304/412/416 等按逐请求契约判断,绝不返回旧内容、错误范围或错误 ETag。
+- Before and after repair, verify HEAD, valid and out-of-range Range, If-Match/If-None-Match, and time conditionals
+- Judge 200/206/304/412/416 and similar codes against the per-request contract, and never return old content, a wrong range, or a wrong ETag.
-先保存失败现场及物理/状态证据,停止本例写入与在途任务,再按 RECOVER 撤销本例故障。并发版本已变化时禁止盲目恢复旧 xl.meta;恢复不能充作产品修复。未恢复环境标 DIRTY,保留本例备份和原失败结果。
+First save the failure scene and the physical and state evidence, stop this case's writes and in-flight tasks, then undo this case's fault per RECOVER. When a concurrent version has already changed, do not blindly restore the old xl.meta; recovery must not serve as a product fix. Mark an unrestored environment DIRTY, and keep this case's backup and the original failure result.
-逐项满足设计公共契约、该分支步骤及附加断言后人工评审。缺资源为 BLOCKED,窗口未命中为 NOT_OBSERVED,产品错误为 FAIL,工具/恢复错误为 INFRA_ERROR;未运行不记 PASS。
+Manually review after each item satisfies the design's common contract, that branch's steps, and the additional assertions. Missing resources are BLOCKED, a window miss is NOT_OBSERVED, a product error is FAIL, and a tool or recovery error is INFRA_ERROR; a case that did not run is not recorded as PASS.
@@ -885,21 +885,21 @@ Source assertions:
Topologies: H1,H2. Branches: sse-s3, sse-kms, sse-c, encrypted-versioned.
-1. DEPENDENCY 先验证候选支持的每种 SSE 与专用 KMS/key 就绪;不支持的产品能力按明确错误契约另记,不能删除该参数实例。
-2. 用 S3 ServerSideEncryption/SSEKMSKeyId 或 SSECustomerAlgorithm/SSECustomerKey 创建对象;密钥从环境注入到内存,不放命令行/清单。
-3. 在密文物理层选择 part 做 missing/bitflip;deep heal 后比较同角色原密文片或匹配候选的重编码验证。
-4. 用正确授权逐版本解密 GET 校验原文,用缺少/错误密钥请求验证拒绝;不得写出明文替代或把密钥错误判成不存在。
+1. DEPENDENCY first verifies that every SSE the candidate supports and the dedicated KMS/key are ready; unsupported product capabilities are recorded separately under an explicit error contract, and that parameter instance must not be deleted.
+2. Create the object with S3 ServerSideEncryption/SSEKMSKeyId or SSECustomerAlgorithm/SSECustomerKey; inject the key from the environment into memory, and do not place it on the command line or in the manifest.
+3. At the ciphertext physical layer, select a part for missing/bitflip; after deep heal, compare the original ciphertext shard of the same role, or verify by re-encoding that matches the candidate.
+4. With correct authorization, decrypt-GET each version to check the plaintext; with a missing/wrong key, verify the request is rejected. Do not write a plaintext substitute, and do not classify a key error as not-found.
Operations: [DEPENDENCY](validation-heal-operations.md#DEPENDENCY) → [SEED_DE](validation-heal-operations.md#SEED_DE) → [FORMAT](validation-heal-operations.md#FORMAT) → [FAULT](validation-heal-operations.md#FAULT) → [START](validation-heal-operations.md#START) → [WAIT](validation-heal-operations.md#WAIT) → [PHYSICAL](validation-heal-operations.md#PHYSICAL) → [S3](validation-heal-operations.md#S3) → [VERIFY](validation-heal-operations.md#VERIFY)
Source assertions:
-- 按候选支持的 SSE-S3/SSE-KMS/SSE-C 建加密对象并修复
-- 授权解密后原内容及版本一致,物理验证使用正确编码/密文层;密钥不可用时不误删、不降级明文,证据脱敏。
+- Create encrypted objects with the SSE-S3/SSE-KMS/SSE-C modes the candidate supports, and repair them
+- After authorized decryption, the original content and version match, and physical verification uses the correct encoding/ciphertext layer; when the key is unavailable, do not delete by mistake or downgrade to plaintext, and redact the evidence.
-先保存失败现场及物理/状态证据,停止本例写入与在途任务,再按 RECOVER 撤销本例故障。并发版本已变化时禁止盲目恢复旧 xl.meta;恢复不能充作产品修复。未恢复环境标 DIRTY,保留本例备份和原失败结果。
+First save the failure scene and the physical and state evidence, stop this case's writes and in-flight tasks, then undo this case's fault per RECOVER. When a concurrent version has already changed, do not blindly restore the old xl.meta; recovery must not serve as a product fix. Mark an unrestored environment DIRTY, and keep this case's backup and the original failure result.
-逐项满足设计公共契约、该分支步骤及附加断言后人工评审。缺资源为 BLOCKED,窗口未命中为 NOT_OBSERVED,产品错误为 FAIL,工具/恢复错误为 INFRA_ERROR;未运行不记 PASS。
+Manually review after each item satisfies the design's common contract, that branch's steps, and the additional assertions. Missing resources are BLOCKED, a window miss is NOT_OBSERVED, a product error is FAIL, and a tool or recovery error is INFRA_ERROR; a case that did not run is not recorded as PASS.
@@ -907,21 +907,21 @@ Source assertions:
Topologies: H1,H2. Branches: governance-retention, compliance-retention, legal-hold, historical-protected-version.
-1. 创建启用 ObjectLock 的专用桶,SEED_DO 使用短且真实未到期的 retention 或 ON legal hold;记录 UTC 截止和独占资源到期安排。
-2. 在无 bypass 权限角色下对受保护 versionId DELETE 必须 AccessDenied,确认保护本来有效。
-3. 对受保护版本缺片/xl.meta 缺失后 repair,核对内容及 get_object_retention/legal_hold;保留期内再次删除仍拒绝。
-4. legal hold 仅正常 API 解除;Compliance 等到真实到期后清理,禁止改时钟或内部删文件;未到期现场标 RETAINED,不能交还空环境。
+1. Create a dedicated bucket with ObjectLock enabled. SEED_DO uses a short retention that has genuinely not expired, or legal hold ON; record the UTC deadline and the exclusive-resource expiry schedule.
+2. DELETE of a protected versionId by a role without bypass permission must be AccessDenied, confirming the protection was effective to begin with.
+3. After a missing shard or missing xl.meta on a protected version, repair and check the content and get_object_retention/legal_hold; a further delete within the retention period is still rejected.
+4. Legal hold is released only through the normal API; Compliance is cleaned up only after genuine expiry. Changing the clock or deleting files internally is forbidden. An unexpired scene is marked RETAINED and must not be handed back as an empty environment.
Operations: [SEED_DO](validation-heal-operations.md#SEED_DO) → [S3](validation-heal-operations.md#S3) → [FAULT](validation-heal-operations.md#FAULT) → [START](validation-heal-operations.md#START) → [WAIT](validation-heal-operations.md#WAIT) → [PHYSICAL](validation-heal-operations.md#PHYSICAL) → [VERIFY](validation-heal-operations.md#VERIFY) → [RECOVER](validation-heal-operations.md#RECOVER)
Source assertions:
-- 对开启 retention/legal hold 的受保护版本修复,期间测试非法删除
-- heal 不绕过保留/hold,也不丢失其元数据;拒绝删除与合法修复均有明确证据。
+- Repair a protected version with retention/legal hold enabled, and test an illegal delete during that interval
+- Heal does not bypass retention/hold and does not lose their metadata; both the rejected delete and the legitimate repair have explicit evidence.
-先保存失败现场及物理/状态证据,停止本例写入与在途任务,再按 RECOVER 撤销本例故障。并发版本已变化时禁止盲目恢复旧 xl.meta;恢复不能充作产品修复。未恢复环境标 DIRTY,保留本例备份和原失败结果。
+First save the failure scene and the physical and state evidence, stop this case's writes and in-flight tasks, then undo this case's fault per RECOVER. When a concurrent version has already changed, do not blindly restore the old xl.meta; recovery must not serve as a product fix. Mark an unrestored environment DIRTY, and keep this case's backup and the original failure result.
-逐项满足设计公共契约、该分支步骤及附加断言后人工评审。缺资源为 BLOCKED,窗口未命中为 NOT_OBSERVED,产品错误为 FAIL,工具/恢复错误为 INFRA_ERROR;未运行不记 PASS。
+Manually review after each item satisfies the design's common contract, that branch's steps, and the additional assertions. Missing resources are BLOCKED, a window miss is NOT_OBSERVED, a product error is FAIL, and a tool or recovery error is INFRA_ERROR; a case that did not run is not recorded as PASS.
@@ -929,21 +929,21 @@ Source assertions:
Topologies: H1,H2. Branches: unicode-space-percent, encoded-slash, adjacent-prefixes, multi-page-list.
-1. DS 创建中文、空格、%2F 字面量、真实 / 和多层前缀键;至少 257 个对象并设置 List MaxKeys=17,确保跨页。
-2. FORMAT 建逻辑 key→物理路径映射,确认 %2F 与 / 不混淆;文件故障只用受限相对路径工具,不拼 shell。
-3. 对选定 key 分别 missing part/xl.meta,使用 URL percent-encode 后的 Admin scope,deep heal。
-4. 分页 LIST/versions 与 manifest 无遗漏/重复;控制前缀不变;对每个特殊 key 精确 GET 和 metadata 校验。
+1. DS creates keys with Chinese, spaces, a literal %2F, a real /, and multi-level prefixes; at least 257 objects, with List MaxKeys=17, to ensure the listing crosses pages.
+2. FORMAT builds the logical key→physical path mapping and confirms %2F is not confused with /; file faults use only a constrained relative-path tool, and do not concatenate a shell command.
+3. For the selected keys, apply missing part/xl.meta separately, using the URL percent-encoded Admin scope, then deep heal.
+4. Paginated LIST/versions and the manifest have no omissions or duplicates; the control prefix is unchanged; exact GET and metadata checks for every special key.
Operations: [SEED_DS](validation-heal-operations.md#SEED_DS) → [FORMAT](validation-heal-operations.md#FORMAT) → [FAULT](validation-heal-operations.md#FAULT) → [START](validation-heal-operations.md#START) → [WAIT](validation-heal-operations.md#WAIT) → [S3](validation-heal-operations.md#S3) → [VERIFY](validation-heal-operations.md#VERIFY)
Source assertions:
-- 覆盖中文、空格、百分号、编码斜线、多层前缀和多页对象列表
-- 逻辑 key 与物理路径映射准确,不跨桶/前缀;分页无遗漏重复,不能把用户 key 当可执行 shell 或裸文件路径。
+- Cover Chinese, spaces, percent signs, encoded slashes, multi-level prefixes, and multi-page object listings
+- The logical key maps accurately to the physical path and does not cross buckets or prefixes; pagination has no omissions or duplicates, and a user key must not be treated as an executable shell command or a bare file path.
-先保存失败现场及物理/状态证据,停止本例写入与在途任务,再按 RECOVER 撤销本例故障。并发版本已变化时禁止盲目恢复旧 xl.meta;恢复不能充作产品修复。未恢复环境标 DIRTY,保留本例备份和原失败结果。
+First save the failure scene and the physical and state evidence, stop this case's writes and in-flight tasks, then undo this case's fault per RECOVER. When a concurrent version has already changed, do not blindly restore the old xl.meta; recovery must not serve as a product fix. Mark an unrestored environment DIRTY, and keep this case's backup and the original failure result.
-逐项满足设计公共契约、该分支步骤及附加断言后人工评审。缺资源为 BLOCKED,窗口未命中为 NOT_OBSERVED,产品错误为 FAIL,工具/恢复错误为 INFRA_ERROR;未运行不记 PASS。
+Manually review after each item satisfies the design's common contract, that branch's steps, and the additional assertions. Missing resources are BLOCKED, a window miss is NOT_OBSERVED, a product error is FAIL, and a tool or recovery error is INFRA_ERROR; a case that did not run is not recorded as PASS.
@@ -951,21 +951,21 @@ Source assertions:
Topologies: H1,H2. Branches: versioning-config, bucket-policy-tags, lock-config, unknown-record-negative.
-1. DS/DV 创建隔离桶配置:版本、只授权本桶的策略、bucket tags、可到期锁配置;通过 S3 GET 配置存基线。
-2. FORMAT_BAD 白名单仅包含该桶对应的系统 bucket metadata 文件;在匹配格式/可恢复契约已核实后少数副本缺失/截断。
-3. 不能使用通用 heal_fault 修改 .rustfs.sys;按专用系统记录手册在停止服务、精确备份和审批清单下操作,未知格式实例只要求安全拒绝。
-4. repair 后逐 API 读取配置比对,业务对象版本完整;确认 IAM/凭据配置及其他桶系统记录完全未变。
+1. DS/DV creates an isolated bucket configuration: versioning, a policy that authorizes only this bucket, bucket tags, and an expirable lock configuration; save the baseline by S3 GET of the configuration.
+2. The FORMAT_BAD allowlist contains only the system bucket metadata files for that bucket; after the matching format/recoverable contract has been verified, a minority of replicas are missing/truncated.
+3. Do not use the generic heal_fault to modify .rustfs.sys; operate per the dedicated system-record runbook with the service stopped, an exact backup, and an approved manifest. Unknown-format instances require only a safe rejection.
+4. After repair, read and compare the configuration through each API; business-object versions are complete. Confirm that the IAM/credential configuration and the system records of other buckets are completely unchanged.
Operations: [SEED_DS](validation-heal-operations.md#SEED_DS) → [SEED_DV](validation-heal-operations.md#SEED_DV) → [S3](validation-heal-operations.md#S3) → [FORMAT](validation-heal-operations.md#FORMAT) → [FAULT_FORMAT](validation-heal-operations.md#FAULT_FORMAT) → [START](validation-heal-operations.md#START) → [WAIT](validation-heal-operations.md#WAIT) → [VERIFY](validation-heal-operations.md#VERIFY) → [PHYSICAL](validation-heal-operations.md#PHYSICAL)
Source assertions:
-- 在隔离桶白名单内测试可支持的桶元数据恢复,并核对外部 bucket 配置
-- 版本设置、策略/标签/保留配置按契约保持;不得遍历凭据配置或把未知内部记录当可重建数据。
+- Within the isolated-bucket allowlist, test supported bucket-metadata recovery, and check the external bucket configuration
+- Versioning settings and policy/tag/retention configuration are kept per the contract; do not walk credential configuration or treat an unknown internal record as rebuildable data.
-先保存失败现场及物理/状态证据,停止本例写入与在途任务,再按 RECOVER 撤销本例故障。并发版本已变化时禁止盲目恢复旧 xl.meta;恢复不能充作产品修复。未恢复环境标 DIRTY,保留本例备份和原失败结果。
+First save the failure scene and the physical and state evidence, stop this case's writes and in-flight tasks, then undo this case's fault per RECOVER. When a concurrent version has already changed, do not blindly restore the old xl.meta; recovery must not serve as a product fix. Mark an unrestored environment DIRTY, and keep this case's backup and the original failure result.
-逐项满足设计公共契约、该分支步骤及附加断言后人工评审。缺资源为 BLOCKED,窗口未命中为 NOT_OBSERVED,产品错误为 FAIL,工具/恢复错误为 INFRA_ERROR;未运行不记 PASS。
+Manually review after each item satisfies the design's common contract, that branch's steps, and the additional assertions. Missing resources are BLOCKED, a window miss is NOT_OBSERVED, a product error is FAIL, and a tool or recovery error is INFRA_ERROR; a case that did not run is not recorded as PASS.
@@ -973,21 +973,21 @@ Source assertions:
Topologies: H1,H2. Branches: normal-missing-data, normal-missing-parity.
-1. 部署 manual 配置:关闭 scanner、auto-heal、MRF;保留 format,记录服务实际环境与 PID,不只保存拟用 env 文件。
-2. FAULT missing 后不能 GET/HEAD 目标;先确认缺失仍在,再 START scanMode=1。
-3. WAIT 精确 token/outcome,先 PHYSICAL 证明目标新生成,再 VERIFY;源归属需结合配置、时间线和任务对象。
-4. 正常扫描未发现静默 bitrot 不能因此判 E01 失败;那是 E02/C02/C03 的 deep 实例。
+1. Deploy the manual configuration: turn off scanner, auto-heal, and MRF; keep format, and record the service's actual environment and PID, not only the intended env file.
+2. After FAULT missing, the target cannot be GET/HEAD; first confirm the missing state is still present, then START scanMode=1.
+3. WAIT for the exact token/outcome; PHYSICAL first proves the target was newly generated, then VERIFY. Source attribution needs to combine the configuration, the timeline, and the task object.
+4. A normal scan that does not find silent bitrot must not fail E01 for that reason; that is the deep instance of E02/C02/C03.
Operations: [SEED_DB](validation-heal-operations.md#SEED_DB) → [CONFIG](validation-heal-operations.md#CONFIG) → [FAULT](validation-heal-operations.md#FAULT) → [START](validation-heal-operations.md#START) → [WAIT](validation-heal-operations.md#WAIT) → [PHYSICAL](validation-heal-operations.md#PHYSICAL) → [VERIFY](validation-heal-operations.md#VERIFY)
Source assertions:
-- 禁止预读目标,保留 format,隔离背景干扰后显式 normal heal
-- 证明本次 Admin token 实际覆盖目标并修复可由 normal 检出的缺失;不要求 normal 发现所有静默 bitrot。
+- Do not pre-read the target, keep format, isolate background interference, then explicitly run normal heal
+- Prove that this Admin token actually covers the target and repairs the missing data that normal can detect; normal is not required to find all silent bitrot.
-先保存失败现场及物理/状态证据,停止本例写入与在途任务,再按 RECOVER 撤销本例故障。并发版本已变化时禁止盲目恢复旧 xl.meta;恢复不能充作产品修复。未恢复环境标 DIRTY,保留本例备份和原失败结果。
+First save the failure scene and the physical and state evidence, stop this case's writes and in-flight tasks, then undo this case's fault per RECOVER. When a concurrent version has already changed, do not blindly restore the old xl.meta; recovery must not serve as a product fix. Mark an unrestored environment DIRTY, and keep this case's backup and the original failure result.
-逐项满足设计公共契约、该分支步骤及附加断言后人工评审。缺资源为 BLOCKED,窗口未命中为 NOT_OBSERVED,产品错误为 FAIL,工具/恢复错误为 INFRA_ERROR;未运行不记 PASS。
+Manually review after each item satisfies the design's common contract, that branch's steps, and the additional assertions. Missing resources are BLOCKED, a window miss is NOT_OBSERVED, a product error is FAIL, and a tool or recovery error is INFRA_ERROR; a case that did not run is not recorded as PASS.
@@ -995,21 +995,21 @@ Source assertions:
Topologies: H1,H2. Branches: deep-parity-bitrot, deep-data-bitrot.
-1. manual 配置排除其他后台来源;FORMAT 确认分片角色,注入 bitflip 并保存坏片仍存在证据。
-2. 不先读目标,START scanMode=2;保留请求 body、clientToken、incarnation/pool/set 与最终对象 disposition。
-3. 先验证坏片完整恢复,再读取原文;如果任务开始前已有恢复,记录 ATTRIBUTION_NOT_OBSERVED。
-4. 检查健康 control 没被意外修改,source 不能仅由最后一次发出的命令推测。
+1. The manual configuration excludes other background sources; FORMAT confirms the shard role, injects a bitflip, and saves evidence that the bad shard still exists.
+2. Do not read the target first; START scanMode=2; retain the request body, clientToken, incarnation/pool/set, and the final object disposition.
+3. First verify the bad shard is fully restored, then read the original content; if restoration already occurred before the task started, record ATTRIBUTION_NOT_OBSERVED.
+4. Check that the healthy control was not modified unexpectedly; source must not be inferred only from the last command issued.
Operations: [SEED_DB](validation-heal-operations.md#SEED_DB) → [CONFIG](validation-heal-operations.md#CONFIG) → [FORMAT](validation-heal-operations.md#FORMAT) → [FAULT](validation-heal-operations.md#FAULT) → [START](validation-heal-operations.md#START) → [WAIT](validation-heal-operations.md#WAIT) → [PHYSICAL](validation-heal-operations.md#PHYSICAL) → [VERIFY](validation-heal-operations.md#VERIFY)
Source assertions:
-- 在普通读取仍可成功的 parity 损坏上显式 deep heal
-- 按 token/对象/版本及物理变更证明 deep 检查和修复,不把后台提前修复算给手工任务。
+- On parity corruption where an ordinary read can still succeed, explicitly run deep heal
+- Prove the deep check and the repair by token, object, version, and physical change, and do not credit a background repair that finished early to the manual task.
-先保存失败现场及物理/状态证据,停止本例写入与在途任务,再按 RECOVER 撤销本例故障。并发版本已变化时禁止盲目恢复旧 xl.meta;恢复不能充作产品修复。未恢复环境标 DIRTY,保留本例备份和原失败结果。
+First save the failure scene and the physical and state evidence, stop this case's writes and in-flight tasks, then undo this case's fault per RECOVER. When a concurrent version has already changed, do not blindly restore the old xl.meta; recovery must not serve as a product fix. Mark an unrestored environment DIRTY, and keep this case's backup and the original failure result.
-逐项满足设计公共契约、该分支步骤及附加断言后人工评审。缺资源为 BLOCKED,窗口未命中为 NOT_OBSERVED,产品错误为 FAIL,工具/恢复错误为 INFRA_ERROR;未运行不记 PASS。
+Manually review after each item satisfies the design's common contract, that branch's steps, and the additional assertions. Missing resources are BLOCKED, a window miss is NOT_OBSERVED, a product error is FAIL, and a tool or recovery error is INFRA_ERROR; a case that did not run is not recorded as PASS.
@@ -1017,21 +1017,21 @@ Source assertions:
Topologies: H1,H2. Branches: get-missing, range-bitrot, historical-version-read.
-1. 关闭 scanner/auto-heal,按候选 read-repair 实现配置其所需 MRF 队列,记录为何开启;不发送 Admin heal。
-2. FAULT 后物理证明损坏尚存;只对一个精确 key/version 发一次带请求 ID 的 GET 或 Range。
-3. GET 内容先验正确还不够:只观察 PHYSICAL/后台状态/日志,直到该片重建;异步路径需关联 read-repair 入队和责任完成。
-4. 完成补片后才全量 VERIFY;如果只能看到读取成功但找不到 source 证据,该路径仍 NOT_OBSERVED,不能计 PASS。
+1. Turn off scanner/auto-heal; per the candidate's read-repair implementation, configure the MRF queue it requires and record why it is enabled; do not send an Admin heal.
+2. After FAULT, physically prove the corruption still exists; send only one GET or Range, with a request ID, for one exact key/version.
+3. It is not enough that the GET content is already correct: observe only PHYSICAL/background state/logs until that shard is rebuilt; the async path needs to correlate read-repair enqueue and completion of the responsibility.
+4. Run a full VERIFY only after shard repair completes; if only a successful read is visible and source evidence cannot be found, that path remains NOT_OBSERVED and must not count as PASS.
Operations: [SEED_DB](validation-heal-operations.md#SEED_DB) → [SEED_DV](validation-heal-operations.md#SEED_DV) → [CONFIG](validation-heal-operations.md#CONFIG) → [FAULT](validation-heal-operations.md#FAULT) → [GET](validation-heal-operations.md#GET) → [RANGE](validation-heal-operations.md#RANGE) → [BACKGROUND](validation-heal-operations.md#BACKGROUND) → [LOGS](validation-heal-operations.md#LOGS) → [PHYSICAL](validation-heal-operations.md#PHYSICAL) → [VERIFY](validation-heal-operations.md#VERIFY)
Source assertions:
-- 不发 Admin heal,对已确认的特定损坏对象发一次可追踪 GET/Range
-- 读取正确且后续真实补片;若走异步队列需证实 source=read-repair,不能只有 GET 成功。
+- Do not issue an Admin heal; issue one traceable GET/Range against the confirmed, specifically corrupted object
+- The read is correct and a real on-disk shard reconstruction follows; if it goes through the async queue, source=read-repair must be proven, and a successful GET alone is not enough.
-先保存失败现场及物理/状态证据,停止本例写入与在途任务,再按 RECOVER 撤销本例故障。并发版本已变化时禁止盲目恢复旧 xl.meta;恢复不能充作产品修复。未恢复环境标 DIRTY,保留本例备份和原失败结果。
+First save the failure scene and the physical and state evidence, stop this case's writes and in-flight tasks, then undo this case's fault per RECOVER. When a concurrent version has already changed, do not blindly restore the old xl.meta; recovery must not serve as a product fix. Mark an unrestored environment DIRTY, and keep this case's backup and the original failure result.
-逐项满足设计公共契约、该分支步骤及附加断言后人工评审。缺资源为 BLOCKED,窗口未命中为 NOT_OBSERVED,产品错误为 FAIL,工具/恢复错误为 INFRA_ERROR;未运行不记 PASS。
+Manually review after each item satisfies the design's common contract, that branch's steps, and the additional assertions. Missing resources are BLOCKED, a window miss is NOT_OBSERVED, a product error is FAIL, and a tool or recovery error is INFRA_ERROR; a case that did not run is not recorded as PASS.
@@ -1039,21 +1039,21 @@ Source assertions:
Topologies: H1,H2. Branches: cold-missing, cold-bitrot, scanner-cycle-boundary.
-1. 启用 scanner 和所需自动修复,预注册 cycle/start delay/bitrot cycle 与预算;先用健康冷对象证明扫描器实际巡检。
-2. 注入时用 PROCESS 短停以保持静止窗口;重启后不发目标 GET/HEAD、不发 Admin heal。
-3. 只用磁盘快照、scanner 遍历日志和后台责任观察,证明当前周期检查目标,记录发现→入队→补片。
-4. PHYSICAL 完成后再全量 VERIFY;单纯周期计数增长、全局 idle 或其他自动 replacement 先修不能记 scanner 通过。
+1. Enable the scanner and the required automatic repair; pre-register cycle/start delay/bitrot cycle and the budget; first use a healthy cold object to prove the scanner actually inspects.
+2. During injection, use PROCESS for a short stop to hold a quiescent window; after restart, do not send a target GET/HEAD and do not send an Admin heal.
+3. Use only disk snapshots, scanner traversal logs, and background responsibility observation to prove the current cycle inspects the target, and record discovery→enqueue→shard repair.
+4. Run a full VERIFY only after PHYSICAL completes; a mere cycle-count increase, global idle, or another automatic replacement repairing first must not be recorded as the scanner passing.
Operations: [SEED_DB](validation-heal-operations.md#SEED_DB) → [CONFIG](validation-heal-operations.md#CONFIG) → [PROCESS](validation-heal-operations.md#PROCESS) → [FAULT](validation-heal-operations.md#FAULT) → [BACKGROUND](validation-heal-operations.md#BACKGROUND) → [LOGS](validation-heal-operations.md#LOGS) → [PHYSICAL](validation-heal-operations.md#PHYSICAL) → [VERIFY](validation-heal-operations.md#VERIFY)
Source assertions:
-- 不 GET/HEAD 损坏目标、不发手工 heal,仅等待已启用的 scanner 路径
-- 冷对象被发现并修复,来源和遍历窗口明确;scanner 周期结束但目标没检查不能通过。
+- Do not GET/HEAD the corrupted target and do not issue a manual heal; wait only for the scanner path that is already enabled
+- The cold object is discovered and repaired, with the source and the traversal window made explicit; it cannot pass if the scanner cycle ends without the target being checked.
-先保存失败现场及物理/状态证据,停止本例写入与在途任务,再按 RECOVER 撤销本例故障。并发版本已变化时禁止盲目恢复旧 xl.meta;恢复不能充作产品修复。未恢复环境标 DIRTY,保留本例备份和原失败结果。
+First save the failure scene and the physical and state evidence, stop this case's writes and in-flight tasks, then undo this case's fault per RECOVER. When a concurrent version has already changed, do not blindly restore the old xl.meta; recovery must not serve as a product fix. Mark an unrestored environment DIRTY, and keep this case's backup and the original failure result.
-逐项满足设计公共契约、该分支步骤及附加断言后人工评审。缺资源为 BLOCKED,窗口未命中为 NOT_OBSERVED,产品错误为 FAIL,工具/恢复错误为 INFRA_ERROR;未运行不记 PASS。
+Manually review after each item satisfies the design's common contract, that branch's steps, and the additional assertions. Missing resources are BLOCKED, a window miss is NOT_OBSERVED, a product error is FAIL, and a tool or recovery error is INFRA_ERROR; a case that did not run is not recorded as PASS.
@@ -1061,21 +1061,21 @@ Source assertions:
Topologies: H2. Branches: offline-put, offline-version-delete, offline-marker.
-1. 在 H2 开启 durable MRF,先验证目标节点离线时剩余同 set 满足提交条件;DRIVE_GATE 或 PROCESS 只影响目标成员。
-2. 在线节点执行唯一 key PUT、精确版本 DELETE、marker 操作;逐个保存 ACK 和责任 identity。
-3. 在目标仍离线时采集 journal/状态,证明责任已持久化;恢复同一盘身份,不发手工 heal。
-4. 观察 MRF 重放和退休理由;只有实际 repaired/verified_healthy/authoritatively_absent 后退休;VERIFY 全部 ACK 和目标片,不能以队列空独自证明。
+1. On H2, enable durable MRF; first verify that, when the target node is offline, the remaining members of the same set satisfy the commit condition; DRIVE_GATE or PROCESS affects only the target member.
+2. Online nodes perform a unique-key PUT, an exact-version DELETE, and marker operations; save each ACK and responsibility identity.
+3. While the target is still offline, collect the journal/state to prove the responsibility has been persisted; restore the same disk identity, and do not issue a manual heal.
+4. Observe MRF replay and the retire reason; retire only after the result is actually repaired/verified_healthy/authoritatively_absent; VERIFY every ACK and the target shard, and do not use an empty queue as the sole proof.
Operations: [SEED_DV](validation-heal-operations.md#SEED_DV) → [CONFIG](validation-heal-operations.md#CONFIG) → [DRIVE_GATE](validation-heal-operations.md#DRIVE_GATE) → [WORKLOAD](validation-heal-operations.md#WORKLOAD) → [CHECKPOINT](validation-heal-operations.md#CHECKPOINT) → [BACKGROUND](validation-heal-operations.md#BACKGROUND) → [PHYSICAL](validation-heal-operations.md#PHYSICAL) → [VERIFY](validation-heal-operations.md#VERIFY)
Source assertions:
-- 成员离线时完成有 quorum 的 PUT/DELETE,确认待修责任后重入
-- durable MRF 责任持久且身份精确;只在 repaired/verified healthy/authoritatively absent 等已证明结果后释放,成功提交不丢失。
+- While a member is offline, complete a quorum PUT/DELETE, confirm the pending-repair responsibility, then rejoin
+- The durable MRF responsibility persists and its identity is exact; release it only after a proven result such as repaired/verified healthy/authoritatively absent, and a successful commit is not lost.
-先保存失败现场及物理/状态证据,停止本例写入与在途任务,再按 RECOVER 撤销本例故障。并发版本已变化时禁止盲目恢复旧 xl.meta;恢复不能充作产品修复。未恢复环境标 DIRTY,保留本例备份和原失败结果。
+First save the failure scene and the physical and state evidence, stop this case's writes and in-flight tasks, then undo this case's fault per RECOVER. When a concurrent version has already changed, do not blindly restore the old xl.meta; recovery must not serve as a product fix. Mark an unrestored environment DIRTY, and keep this case's backup and the original failure result.
-逐项满足设计公共契约、该分支步骤及附加断言后人工评审。缺资源为 BLOCKED,窗口未命中为 NOT_OBSERVED,产品错误为 FAIL,工具/恢复错误为 INFRA_ERROR;未运行不记 PASS。
+Manually review after each item satisfies the design's common contract, that branch's steps, and the additional assertions. Missing resources are BLOCKED, a window miss is NOT_OBSERVED, a product error is FAIL, and a tool or recovery error is INFRA_ERROR; a case that did not run is not recorded as PASS.
@@ -1083,21 +1083,21 @@ Source assertions:
Topologies: H1,H2. Branches: replacement-unversioned, replacement-versioned-marker.
-1. 启用 replacement auto-heal;REPLACE 注入空逻辑盘,旧盘备份对服务不可达;保存新 UUID/slot。
-2. 重启/接纳后不发 Admin heal、不读目标;仅查询 BACKGROUND/REPLACEMENT、日志及 PHYSICAL。
-3. 从 replacement generation、ready 与责任完成状态关联所有 manifest 对象/版本的补片时间线。
-4. 全部目标分片实际重建后 VERIFY;服务 ready、计数重置或队列空都不能提前结束。
+1. Enable replacement auto-heal; REPLACE injects an empty logical disk, and the old disk's backup is unreachable to the service; save the new UUID/slot.
+2. After restart/acceptance, do not send an Admin heal and do not read the target; query only BACKGROUND/REPLACEMENT, logs, and PHYSICAL.
+3. From the replacement generation, ready, and responsibility-completion state, correlate the shard-repair timeline of every manifest object/version.
+4. VERIFY only after every target shard has actually been rebuilt; service ready, a counter reset, or an empty queue must not end the run early.
Operations: [SEED_DB](validation-heal-operations.md#SEED_DB) → [SEED_DV](validation-heal-operations.md#SEED_DV) → [CONFIG](validation-heal-operations.md#CONFIG) → [REPLACE](validation-heal-operations.md#REPLACE) → [FORMAT](validation-heal-operations.md#FORMAT) → [REPLACEMENT](validation-heal-operations.md#REPLACEMENT) → [BACKGROUND](validation-heal-operations.md#BACKGROUND) → [PHYSICAL](validation-heal-operations.md#PHYSICAL) → [VERIFY](validation-heal-operations.md#VERIFY)
Source assertions:
-- 替换盘接纳后不发 Admin/目标 GET,观察自动 replacement heal
-- 替换代次、ready/completion 与盘上全部目标版本一致;不能用服务 ready、队列空或计数重置代替完成。
+- After the replacement disk is accepted, do not issue an Admin or target GET; observe automatic replacement heal
+- The replacement generation, ready/completion, and every target version on disk agree; service ready, an empty queue, or a counter reset cannot substitute for completion.
-先保存失败现场及物理/状态证据,停止本例写入与在途任务,再按 RECOVER 撤销本例故障。并发版本已变化时禁止盲目恢复旧 xl.meta;恢复不能充作产品修复。未恢复环境标 DIRTY,保留本例备份和原失败结果。
+First save the failure scene and the physical and state evidence, stop this case's writes and in-flight tasks, then undo this case's fault per RECOVER. When a concurrent version has already changed, do not blindly restore the old xl.meta; recovery must not serve as a product fix. Mark an unrestored environment DIRTY, and keep this case's backup and the original failure result.
-逐项满足设计公共契约、该分支步骤及附加断言后人工评审。缺资源为 BLOCKED,窗口未命中为 NOT_OBSERVED,产品错误为 FAIL,工具/恢复错误为 INFRA_ERROR;未运行不记 PASS。
+Manually review after each item satisfies the design's common contract, that branch's steps, and the additional assertions. Missing resources are BLOCKED, a window miss is NOT_OBSERVED, a product error is FAIL, and a tool or recovery error is INFRA_ERROR; a case that did not run is not recorded as PASS.
@@ -1105,21 +1105,21 @@ Source assertions:
Topologies: H1,H2. Branches: root-partial-graceful, same-record-disk-resume.
-1. DX root heal 启动前保存原协调者、拓扑、记录盘和预算;所有 root checkpoint 通过 CHECKPOINT 只读白名单采集。
-2. WINDOW 证明 0<已修目标数<总数 且任务 running,再 PROCESS graceful-stop 原协调者;记录退出信号和持久记录。
-3. 用原 unit/env/数据根/候选重启原协调者,不换记录盘;观察原任务 ID/剩余责任恢复,不能将新扫描当原任务续跑。
-4. 最终全部对象/片正确,root 活动记录退休,重复重启无重复任务;未观察部分进度保留 NOT_OBSERVED。
+1. Before DX root heal starts, save the original coordinator, topology, record disk, and budget; collect every root checkpoint through the CHECKPOINT read-only allowlist.
+2. WINDOW proves 0 < repaired target count < total and that the task is running, then PROCESS graceful-stop of the original coordinator; record the exit signal and the persistent record.
+3. Restart the original coordinator with the original unit/env/data root/candidate, without replacing the record disk; observe recovery of the original task ID/remaining responsibility, and do not treat a new scan as the original task continuing.
+4. In the end every object/shard is correct, the root activity record is retired, and repeated restarts produce no duplicate task; unobserved partial progress remains NOT_OBSERVED.
Operations: [SEED_DX](validation-heal-operations.md#SEED_DX) → [FAULT](validation-heal-operations.md#FAULT) → [START](validation-heal-operations.md#START) → [WINDOW](validation-heal-operations.md#WINDOW) → [CHECKPOINT](validation-heal-operations.md#CHECKPOINT) → [PROCESS](validation-heal-operations.md#PROCESS) → [STATUS](validation-heal-operations.md#STATUS) → [PHYSICAL](validation-heal-operations.md#PHYSICAL) → [VERIFY](validation-heal-operations.md#VERIFY)
Source assertions:
-- root heal 非零进度时优雅重启原协调者,保留原拓扑与记录盘
-- 在明确支持的边界内恢复原 task ID 和剩余责任/预算,完成后无遗留活动记录;不扩张为协调盘丢失后的故障转移。
+- While root heal has non-zero progress, gracefully restart the original coordinator, keeping the original topology and the record disk
+- Within the explicitly supported boundary, restore the original task ID and the remaining responsibility/budget, and leave no active record after completion; do not expand this into failover after the coordinator disk is lost.
-先保存失败现场及物理/状态证据,停止本例写入与在途任务,再按 RECOVER 撤销本例故障。并发版本已变化时禁止盲目恢复旧 xl.meta;恢复不能充作产品修复。未恢复环境标 DIRTY,保留本例备份和原失败结果。
+First save the failure scene and the physical and state evidence, stop this case's writes and in-flight tasks, then undo this case's fault per RECOVER. When a concurrent version has already changed, do not blindly restore the old xl.meta; recovery must not serve as a product fix. Mark an unrestored environment DIRTY, and keep this case's backup and the original failure result.
-逐项满足设计公共契约、该分支步骤及附加断言后人工评审。缺资源为 BLOCKED,窗口未命中为 NOT_OBSERVED,产品错误为 FAIL,工具/恢复错误为 INFRA_ERROR;未运行不记 PASS。
+Manually review after each item satisfies the design's common contract, that branch's steps, and the additional assertions. Missing resources are BLOCKED, a window miss is NOT_OBSERVED, a product error is FAIL, and a tool or recovery error is INFRA_ERROR; a case that did not run is not recorded as PASS.
@@ -1127,21 +1127,21 @@ Source assertions:
Topologies: H2. Branches: target-partial-graceful, versioned-target-partial.
-1. REPLACE 目标逻辑盘并启用自动重建,使用 DX/DV 使目标总量可观测;协调者不是重启目标。
-2. PHYSICAL 与 REPLACEMENT 双证实目标只完成部分,再 PROCESS 对目标 unit 优雅停止;保存 MainPID、退出码、前后 boot_id。
-3. 保留同一新 disk UUID/代次,正常启动原目标服务;不能换新目录重测。
-4. 观察余下目标继续构建且任务责任一致,全部版本/marker/片完整;备份不得参与重建。
+1. REPLACE the target logical disk and enable automatic rebuild, using DX/DV so the target total is observable; the coordinator is not the restart target.
+2. PHYSICAL and REPLACEMENT both confirm the target has completed only part, then PROCESS gracefully stops the target unit; save MainPID, the exit code, and boot_id before and after.
+3. Keep the same new disk UUID/generation and start the original target service normally; do not switch to a new directory and retest.
+4. Observe that the remaining targets continue to build with consistent task responsibility, and that every version/marker/shard is complete; the backup must not participate in the rebuild.
Operations: [SEED_DX](validation-heal-operations.md#SEED_DX) → [SEED_DV](validation-heal-operations.md#SEED_DV) → [REPLACE](validation-heal-operations.md#REPLACE) → [REPLACEMENT](validation-heal-operations.md#REPLACEMENT) → [WINDOW](validation-heal-operations.md#WINDOW) → [PROCESS](validation-heal-operations.md#PROCESS) → [PHYSICAL](validation-heal-operations.md#PHYSICAL) → [VERIFY](validation-heal-operations.md#VERIFY)
Source assertions:
-- 替换目标仅完成部分重建时优雅重启目标节点
-- 故障确实发生于部分完成窗口;重启后续建同一目标盘/代次,最终 manifest 与全部目标分片正确。
+- When the replacement target has completed only part of the reconstruction, gracefully restart the target node
+- The fault actually occurs inside the partial-completion window; after the restart, reconstruction continues on the same target disk/generation, and the final manifest and all target shards are correct.
-先保存失败现场及物理/状态证据,停止本例写入与在途任务,再按 RECOVER 撤销本例故障。并发版本已变化时禁止盲目恢复旧 xl.meta;恢复不能充作产品修复。未恢复环境标 DIRTY,保留本例备份和原失败结果。
+First save the failure scene and the physical and state evidence, stop this case's writes and in-flight tasks, then undo this case's fault per RECOVER. When a concurrent version has already changed, do not blindly restore the old xl.meta; recovery must not serve as a product fix. Mark an unrestored environment DIRTY, and keep this case's backup and the original failure result.
-逐项满足设计公共契约、该分支步骤及附加断言后人工评审。缺资源为 BLOCKED,窗口未命中为 NOT_OBSERVED,产品错误为 FAIL,工具/恢复错误为 INFRA_ERROR;未运行不记 PASS。
+Manually review after each item satisfies the design's common contract, that branch's steps, and the additional assertions. Missing resources are BLOCKED, a window miss is NOT_OBSERVED, a product error is FAIL, and a tool or recovery error is INFRA_ERROR; a case that did not run is not recorded as PASS.
@@ -1149,21 +1149,21 @@ Source assertions:
Topologies: H2. Branches: target-partial-sigkill, target-partial-sigkill-versioned.
-1. 与 F02 同样准备替换盘和可观察部分进度,先冻结将被杀掉的 unit MainPID/进程启动时间。
-2. WINDOW 内 PROCESS kill,精确 systemctl kill --signal=SIGKILL --kill-whom=main;记录 signal=9 与非优雅退出,明确不是物理断电。
-3. 正常启动同 unit/根/新 disk UUID;记录恢复 responsibility/代次,不允许丢弃尚未完成对象。
-4. 全量 VERIFY/PHYSICAL,检查临时文件和持久责任无泄漏;最终通过不抹掉这次真实中断的证据。
+1. Prepare the replacement disk and observable partial progress the same way as F02, and first freeze the MainPID/process start time of the unit that will be killed.
+2. Within the WINDOW, PROCESS kill, exactly systemctl kill --signal=SIGKILL --kill-whom=main; record signal=9 and a non-graceful exit, and state explicitly that this is not a physical power loss.
+3. Start normally with the same unit/root/new disk UUID; record the recovered responsibility/generation, and do not allow objects that are not yet complete to be discarded.
+4. Full VERIFY/PHYSICAL; check that temporary files and persistent responsibilities are not leaked. A final pass does not erase the evidence of this real interruption.
Operations: [SEED_DX](validation-heal-operations.md#SEED_DX) → [SEED_DV](validation-heal-operations.md#SEED_DV) → [REPLACE](validation-heal-operations.md#REPLACE) → [WINDOW](validation-heal-operations.md#WINDOW) → [PROCESS](validation-heal-operations.md#PROCESS) → [CHECKPOINT](validation-heal-operations.md#CHECKPOINT) → [PHYSICAL](validation-heal-operations.md#PHYSICAL) → [VERIFY](validation-heal-operations.md#VERIFY)
Source assertions:
-- 部分重建时 SIGKILL 测试进程,再正常启动
-- unclean shutdown 与修复进度有真实证据;已提交对象不丢失,未完成目标不会被当完整;明确不是断电测试。
+- During partial reconstruction, SIGKILL the test process, then start it normally
+- There is real evidence of the unclean shutdown and of the repair progress; committed objects are not lost, and an unfinished target is not treated as complete; this is explicitly not a power-loss test.
-先保存失败现场及物理/状态证据,停止本例写入与在途任务,再按 RECOVER 撤销本例故障。并发版本已变化时禁止盲目恢复旧 xl.meta;恢复不能充作产品修复。未恢复环境标 DIRTY,保留本例备份和原失败结果。
+First save the failure scene and the physical and state evidence, stop this case's writes and in-flight tasks, then undo this case's fault per RECOVER. When a concurrent version has already changed, do not blindly restore the old xl.meta; recovery must not serve as a product fix. Mark an unrestored environment DIRTY, and keep this case's backup and the original failure result.
-逐项满足设计公共契约、该分支步骤及附加断言后人工评审。缺资源为 BLOCKED,窗口未命中为 NOT_OBSERVED,产品错误为 FAIL,工具/恢复错误为 INFRA_ERROR;未运行不记 PASS。
+Manually review after each item satisfies the design's common contract, that branch's steps, and the additional assertions. Missing resources are BLOCKED, a window miss is NOT_OBSERVED, a product error is FAIL, and a tool or recovery error is INFRA_ERROR; a case that did not run is not recorded as PASS.
@@ -1171,21 +1171,21 @@ Source assertions:
Topologies: H2. Branches: root-coordinator-kill, mrf-coordinator-kill, replacement-coordinator-kill.
-1. 三个机制使用独立场景和冻结的持久化支持表;在协调者非零且未完成进度时采集 CHECKPOINT。
-2. PROCESS SIGKILL 协调者,目标节点保持;恢复原根/原拓扑。
-3. 分别核对机制承诺:持久任务必须续责;易失 token 丢失不能算已完成,应显示未完成并允许明确的新扫描补偿。
-4. 原任务与补偿任务分别保存;最终 VERIFY 所有 ACK、版本和片,不能把安全重扫成功写成原 token 自动恢复。
+1. The three mechanisms use independent scenarios and a frozen persistence-support table; collect CHECKPOINT while the coordinator has nonzero, incomplete progress.
+2. PROCESS SIGKILL the coordinator; the target node stays up. Restore the original root/original topology.
+3. Check each mechanism's promise separately: a durable task must continue its responsibility; loss of a volatile token must not count as completed. It should show incomplete and allow an explicit new-scan compensation.
+4. Save the original task and the compensation task separately; finally VERIFY all ACKs, versions, and shards. A successful safe rescan must not be recorded as automatic recovery of the original token.
Operations: [SEED_DX](validation-heal-operations.md#SEED_DX) → [SEED_DB](validation-heal-operations.md#SEED_DB) → [START](validation-heal-operations.md#START) → [WINDOW](validation-heal-operations.md#WINDOW) → [CHECKPOINT](validation-heal-operations.md#CHECKPOINT) → [PROCESS](validation-heal-operations.md#PROCESS) → [BACKGROUND](validation-heal-operations.md#BACKGROUND) → [PHYSICAL](validation-heal-operations.md#PHYSICAL) → [VERIFY](validation-heal-operations.md#VERIFY)
Source assertions:
-- 协调者非优雅退出并恢复,分别观察 root、MRF、replacement
-- 按各机制已核实的持久化边界判断;不得把丢失的易失 token 当成功,需显式记录剩余责任并可安全重扫恢复。
+- The coordinator exits ungracefully and then recovers; observe root, MRF, and replacement separately
+- Judge by each mechanism's verified persistence boundary; a lost volatile token must not be treated as success, remaining responsibility must be recorded explicitly, and recovery by a safe rescan must be possible.
-先保存失败现场及物理/状态证据,停止本例写入与在途任务,再按 RECOVER 撤销本例故障。并发版本已变化时禁止盲目恢复旧 xl.meta;恢复不能充作产品修复。未恢复环境标 DIRTY,保留本例备份和原失败结果。
+First save the failure scene and the physical and state evidence, stop this case's writes and in-flight tasks, then undo this case's fault per RECOVER. When a concurrent version has already changed, do not blindly restore the old xl.meta; recovery must not serve as a product fix. Mark an unrestored environment DIRTY, and keep this case's backup and the original failure result.
-逐项满足设计公共契约、该分支步骤及附加断言后人工评审。缺资源为 BLOCKED,窗口未命中为 NOT_OBSERVED,产品错误为 FAIL,工具/恢复错误为 INFRA_ERROR;未运行不记 PASS。
+Manually review after each item satisfies the design's common contract, that branch's steps, and the additional assertions. Missing resources are BLOCKED, a window miss is NOT_OBSERVED, a product error is FAIL, and a tool or recovery error is INFRA_ERROR; a case that did not run is not recorded as PASS.
@@ -1193,21 +1193,21 @@ Source assertions:
Topologies: H2. Branches: all-graceful-restart, all-restart-with-mpu, repeat-full-restart.
-1. 准备 DB/DV/DM、pending UploadId 与未完成持久修复责任;冻结全部节点/盘/slot。
-2. WINDOW 命中后按固定顺序只停止本例全部 units;保存每节点退出码、checkpoint/MRF 与完整时序。
-3. 按原拓扑同时或预注册顺序重启,等待全部 peer,不能用自动重新格式化或新数据根恢复。
-4. 完成后全版本/marker/MPU 和物理片校验,再重复一次重启验证幂等,任何丢失的已承诺责任计失败。
+1. Prepare DB/DV/DM, a pending UploadId, and an incomplete durable repair responsibility; freeze every node/disk/slot.
+2. After the WINDOW is hit, stop only all units of this case, in a fixed order; save each node's exit code, checkpoint/MRF, and the full timeline.
+3. Restart on the original topology, simultaneously or in the pre-registered order, and wait for every peer; do not recover by automatic reformat or a new data root.
+4. After completion, verify every version/marker/MPU and physical shard, then repeat one restart to verify idempotence; any lost already-promised responsibility counts as a failure.
Operations: [SEED_DB](validation-heal-operations.md#SEED_DB) → [SEED_DV](validation-heal-operations.md#SEED_DV) → [SEED_DM](validation-heal-operations.md#SEED_DM) → [FAULT](validation-heal-operations.md#FAULT) → [START](validation-heal-operations.md#START) → [WINDOW](validation-heal-operations.md#WINDOW) → [PROCESS](validation-heal-operations.md#PROCESS) → [CHECKPOINT](validation-heal-operations.md#CHECKPOINT) → [PHYSICAL](validation-heal-operations.md#PHYSICAL) → [VERIFY](validation-heal-operations.md#VERIFY)
Source assertions:
-- 有未完成修复责任时重启全部测试节点,固定原拓扑/磁盘身份
-- 有持久化保证的责任不遗失;全部端点恢复一致,旧版本/marker/MPU 与目标分片完整,重复重启不扩散错误。
+- While unfinished repair responsibility remains, restart all test nodes, with the original topology and disk identity fixed
+- Responsibility that has a persistence guarantee is not lost; all endpoints recover consistently, old versions/markers/MPUs and the target shards are complete, and repeated restarts do not spread the error.
-先保存失败现场及物理/状态证据,停止本例写入与在途任务,再按 RECOVER 撤销本例故障。并发版本已变化时禁止盲目恢复旧 xl.meta;恢复不能充作产品修复。未恢复环境标 DIRTY,保留本例备份和原失败结果。
+First save the failure scene and the physical and state evidence, stop this case's writes and in-flight tasks, then undo this case's fault per RECOVER. When a concurrent version has already changed, do not blindly restore the old xl.meta; recovery must not serve as a product fix. Mark an unrestored environment DIRTY, and keep this case's backup and the original failure result.
-逐项满足设计公共契约、该分支步骤及附加断言后人工评审。缺资源为 BLOCKED,窗口未命中为 NOT_OBSERVED,产品错误为 FAIL,工具/恢复错误为 INFRA_ERROR;未运行不记 PASS。
+Manually review after each item satisfies the design's common contract, that branch's steps, and the additional assertions. Missing resources are BLOCKED, a window miss is NOT_OBSERVED, a product error is FAIL, and a tool or recovery error is INFRA_ERROR; a case that did not run is not recorded as PASS.
@@ -1215,21 +1215,21 @@ Source assertions:
Topologies: H1,H2. Branches: cancel-restart, old-token-after-restart, force-new-after-cancel.
-1. DX 缺片任务在 WINDOW 中 STOP;STATUS 与磁盘静默窗口证明取消和在途退出完成。
-2. 保存旧 token/checkpoint,PROCESS 优雅重启;确认已取消任务不会自行复活或报 repaired。
-3. 新 START/forceStart 获取独立 token,向旧 token 发 stop/status;不得影响新任务。
-4. 新任务完成后 VERIFY/PHYSICAL;取消记录/新任务责任不串代。
+1. STOP the DX missing-shard task during the WINDOW; STATUS and the disk quiescence window prove that cancellation and in-flight exit have completed.
+2. Save the old token/checkpoint and PROCESS a graceful restart; confirm the cancelled task does not revive on its own or report repaired.
+3. A new START/forceStart obtains an independent token; send stop/status to the old token. It must not affect the new task.
+4. After the new task completes, VERIFY/PHYSICAL; the cancellation record and the new task's responsibility must not cross generations.
Operations: [SEED_DX](validation-heal-operations.md#SEED_DX) → [FAULT](validation-heal-operations.md#FAULT) → [START](validation-heal-operations.md#START) → [WINDOW](validation-heal-operations.md#WINDOW) → [STOP](validation-heal-operations.md#STOP) → [CHECKPOINT](validation-heal-operations.md#CHECKPOINT) → [PROCESS](validation-heal-operations.md#PROCESS) → [STATUS](validation-heal-operations.md#STATUS) → [VERIFY](validation-heal-operations.md#VERIFY) → [PHYSICAL](validation-heal-operations.md#PHYSICAL)
Source assertions:
-- 完成取消及在途退出后重启,随后 forceStart/新任务正常修复
-- 已取消的任务不自动复活;旧 checkpoint/token 不能取消或覆盖新任务,旧责任不会误标为修复完成。
+- Restart after cancellation and in-flight exit have finished, then forceStart or a new task repairs normally
+- A cancelled task does not revive on its own; an old checkpoint/token cannot cancel or overwrite a new task, and old responsibility is not marked repair-complete by mistake.
-先保存失败现场及物理/状态证据,停止本例写入与在途任务,再按 RECOVER 撤销本例故障。并发版本已变化时禁止盲目恢复旧 xl.meta;恢复不能充作产品修复。未恢复环境标 DIRTY,保留本例备份和原失败结果。
+First save the failure scene and the physical and state evidence, stop this case's writes and in-flight tasks, then undo this case's fault per RECOVER. When a concurrent version has already changed, do not blindly restore the old xl.meta; recovery must not serve as a product fix. Mark an unrestored environment DIRTY, and keep this case's backup and the original failure result.
-逐项满足设计公共契约、该分支步骤及附加断言后人工评审。缺资源为 BLOCKED,窗口未命中为 NOT_OBSERVED,产品错误为 FAIL,工具/恢复错误为 INFRA_ERROR;未运行不记 PASS。
+Manually review after each item satisfies the design's common contract, that branch's steps, and the additional assertions. Missing resources are BLOCKED, a window miss is NOT_OBSERVED, a product error is FAIL, and a tool or recovery error is INFRA_ERROR; a case that did not run is not recorded as PASS.
@@ -1237,11 +1237,11 @@ Source assertions:
Topologies: H1,H2. Branches: checkpoint-truncate, mrf-checksum, future-schema, stale-generation, journal-write-failure, native-persist-fence.
-实时读取遇到原子替换后链接数为零的旧文件句柄时,先核对文件类型、大小与挂载边界,再拒绝旧句柄字节并返回读取变化。只有原有有界观察循环可以重试;停止后的读取和故障注入仍直接失败,硬链接、非普通文件、超限和挂载逃逸仍被拒绝。原 pidfd、60 秒 advance 预算和完整 checkpoint/resume/root 校验不变。Linux 原子替换组件复现仅证明读取竞争,不证明旧 live 失败的唯一原因,也不替代 RustFS 持久化屏障证据。
+When a live read meets an old file handle whose link count is zero after an atomic replace, first check the file type, size, and mount boundary, then reject the old handle's bytes and report that the read changed. Only the original bounded observation loop may retry. Reads and fault injection after stop still fail immediately. Hard links, non-regular files, over-limit files, and mount escapes are still rejected. The original pidfd, the 60-second advance budget, and the complete checkpoint/resume/root checks are unchanged. Reproducing the Linux atomic-replace component only proves a read race. It does not prove the sole cause of a previous live failure, and it does not replace RustFS persistence-fence evidence.
Execution policy: `evidence-blocked`; disabled by default: `false`.
-Blocked reason (`EVIDENCE_DURABLE_JOURNAL_ORCHESTRATION`): 已实现 H1/H2 checkpoint 截断、未来 schema、真实较早代次回放、committed MRF CRC 故障、私有限额卷的真实 journal ENOSPC 和独立安全重扫;10 项原生回归仅为前置证据。仍缺全部所需 fsync/commit 临界点审核和完整现场分支的独立 live 审计,诊断执行不能提升 canonical 验收。
+Blocked reason (`EVIDENCE_DURABLE_JOURNAL_ORCHESTRATION`): Implemented H1/H2 checkpoint truncation, future schema, replay of a real earlier generation, a committed MRF CRC fault, real journal ENOSPC on a private quota volume, and an independent safe rescan; the 10 native regressions are only prerequisite evidence. Still missing are review of every required fsync/commit boundary and an independent live audit of the complete live branches. Diagnostic execution cannot raise canonical acceptance.
Journal ENOSPC recovery preserves the first failed checkpoint/terminal publication
attempt before restarting the original durable owner after filler release. A new
@@ -1253,21 +1253,21 @@ GET; the later distinct rescan must report zero repairs and preserve original
completion bytes. No forceStart, owner deletion or old-record restoration is
used to bypass overlap admission. This flow still requires fresh live evidence.
-1. 先制造并证明真实 durable root/MRF 责任,CHECKPOINT 定位精确本 run task/record,其他系统记录禁止触碰。
-2. 停止测试服务,按 FAULT_FORMAT 对这些已审核记录分别截断、坏校验、未来 schema、旧代次;写失败实例用独占限额故障层,不塞满真实共享盘。
-3. 重启后核对解析/错误状态:未知或损坏记录保留、可诊断,不能静默删除并 ACK;责任仍能安全重扫处理。
-4. 精确 fsync/commit 崩溃窗口执行 NATIVE 回归,保存屏障命中证据;真机记录损坏与原生临界点不要混算。
+1. First create and prove a real durable root/MRF responsibility; CHECKPOINT locates the exact task/record of this run, and other system records must not be touched.
+2. Stop the test service and, per FAULT_FORMAT, apply truncation, a bad checksum, a future schema, and an older generation separately to these reviewed records; the write-failure instance uses an exclusive quota fault layer and must not fill a real shared disk.
+3. After restart, check the parse/error state: unknown or damaged records are retained and diagnosable, and must not be silently deleted and ACKed; the responsibility can still be handled by a safe rescan.
+4. Run the NATIVE regression in the exact fsync/commit crash window and save evidence that the barrier was hit; do not count corruption recorded on the fleet together with the native boundary.
Operations: [SEED_DV](validation-heal-operations.md#SEED_DV) → [START](validation-heal-operations.md#START) → [CHECKPOINT](validation-heal-operations.md#CHECKPOINT) → [PROCESS](validation-heal-operations.md#PROCESS) → [FAULT_FORMAT](validation-heal-operations.md#FAULT_FORMAT) → [IO_FAULT](validation-heal-operations.md#IO_FAULT) → [BACKGROUND](validation-heal-operations.md#BACKGROUND) → [PHYSICAL](validation-heal-operations.md#PHYSICAL) → [VERIFY](validation-heal-operations.md#VERIFY) → [NATIVE](validation-heal-operations.md#NATIVE)
Source assertions:
-- 受控构造 checkpoint/MRF 截断、坏校验、未来 schema、陈旧代次及写入失败
-- 已确认持久化责任不得静默删除或伪 ACK;未知记录保留且可诊断;精确崩溃窗口由原生屏障证明,不伪称真机命中。
+- Under control, construct checkpoint/MRF truncation, a bad checksum, a future schema, a stale generation, and a write failure
+- Confirmed persisted responsibility must not be silently deleted or given a false ACK; unknown records are retained and diagnosable; the exact crash window is proven by a native barrier and is not falsely claimed as a hit on the live fleet.
-先保存失败现场及物理/状态证据,停止本例写入与在途任务,再按 RECOVER 撤销本例故障。并发版本已变化时禁止盲目恢复旧 xl.meta;恢复不能充作产品修复。未恢复环境标 DIRTY,保留本例备份和原失败结果。
+First save the failure scene and the physical and state evidence, stop this case's writes and in-flight tasks, then undo this case's fault per RECOVER. When a concurrent version has already changed, do not blindly restore the old xl.meta; recovery must not serve as a product fix. Mark an unrestored environment DIRTY, and keep this case's backup and the original failure result.
-逐项满足设计公共契约、该分支步骤及附加断言后人工评审。缺资源为 BLOCKED,窗口未命中为 NOT_OBSERVED,产品错误为 FAIL,工具/恢复错误为 INFRA_ERROR;未运行不记 PASS。
+Manually review after each item satisfies the design's common contract, that branch's steps, and the additional assertions. Missing resources are BLOCKED, a window miss is NOT_OBSERVED, a product error is FAIL, and a tool or recovery error is INFRA_ERROR; a case that did not run is not recorded as PASS.
@@ -1277,23 +1277,23 @@ Topologies: H2. Branches: rolling-old-new, mixed-peer-status, rollback-new-old,
Execution policy: `environment-gated`; disabled by default: `true`.
-Blocked reason (`ENV_IMMUTABLE_RELEASE_PAIR`): 需要两个已固定且相互兼容的 RustFS release、混合版本 peer 状态和经过审核的回滚兼容矩阵;当前 auto-testing Heal 节点没有这组不可变双版本资源。
+Blocked reason (`ENV_IMMUTABLE_RELEASE_PAIR`): Requires two pinned, mutually compatible RustFS releases, mixed-version peer state, and a reviewed rollback-compatibility matrix; the current auto-testing Heal node does not have this immutable dual-version resource set.
-1. 固定两个 release optimized 包的 commit/tree/binary/格式兼容说明;UPGRADE 先验证支持回滚的记录版本,不能使用任意旧包。
-2. 有持久修复责任时逐节点升级,保留数据根和 unit;混合阶段通过各 endpoint STATUS/BACKGROUND 并持续唯一版本写入。
-3. 缺少能力的 peer 必须显示 partial/unsupported,不能 aggregate 成 complete;新记录不被老节点误退休。
-4. 按兼容方案回滚并重启,验证所有 ACK 和责任;不兼容格式只测明确拒绝/保护,禁止冒险降级覆盖。
+1. Pin the commit/tree/binary/format-compatibility notes of two release optimized packages; UPGRADE first verifies a record version that supports rollback, and must not use an arbitrary old package.
+2. When a durable repair responsibility exists, upgrade node by node, keeping the data root and unit; during the mixed phase, pass STATUS/BACKGROUND on each endpoint and continue unique-version writes.
+3. A peer that lacks the capability must show partial/unsupported and must not be aggregated into complete; new records must not be retired by mistake by an old node.
+4. Roll back and restart according to the compatibility plan, and verify all ACKs and responsibilities; incompatible formats are tested only for explicit rejection/protection. A risky downgrade overwrite is forbidden.
Operations: [SEED_DV](validation-heal-operations.md#SEED_DV) → [START](validation-heal-operations.md#START) → [CHECKPOINT](validation-heal-operations.md#CHECKPOINT) → [UPGRADE](validation-heal-operations.md#UPGRADE) → [WORKLOAD](validation-heal-operations.md#WORKLOAD) → [STATUS](validation-heal-operations.md#STATUS) → [BACKGROUND](validation-heal-operations.md#BACKGROUND) → [PHYSICAL](validation-heal-operations.md#PHYSICAL) → [VERIFY](validation-heal-operations.md#VERIFY)
Source assertions:
-- 固定前后两个明确兼容候选,覆盖滚动升级、混合 peer、回滚与重启
-- 能力不足必须显示 partial/unsupported 而非完整成功;按格式兼容契约保留责任,旧版本不能误退休新版本记录。
+- Pin two explicitly compatible candidates, one before and one after, covering rolling upgrade, mixed peers, rollback, and restart
+- Insufficient capability must be shown as partial/unsupported rather than complete success; retain responsibility under the format-compatibility contract, and an older version must not mistakenly retire a newer version's records.
-先保存失败现场及物理/状态证据,停止本例写入与在途任务,再按 RECOVER 撤销本例故障。并发版本已变化时禁止盲目恢复旧 xl.meta;恢复不能充作产品修复。未恢复环境标 DIRTY,保留本例备份和原失败结果。
+First save the failure scene and the physical and state evidence, stop this case's writes and in-flight tasks, then undo this case's fault per RECOVER. When a concurrent version has already changed, do not blindly restore the old xl.meta; recovery must not serve as a product fix. Mark an unrestored environment DIRTY, and keep this case's backup and the original failure result.
-逐项满足设计公共契约、该分支步骤及附加断言后人工评审。缺资源为 BLOCKED,窗口未命中为 NOT_OBSERVED,产品错误为 FAIL,工具/恢复错误为 INFRA_ERROR;未运行不记 PASS。
+Manually review after each item satisfies the design's common contract, that branch's steps, and the additional assertions. Missing resources are BLOCKED, a window miss is NOT_OBSERVED, a product error is FAIL, and a tool or recovery error is INFRA_ERROR; a case that did not run is not recorded as PASS.
@@ -1303,23 +1303,23 @@ Topologies: H6. Branches: dedicated-vm-forced-stop, dedicated-host-power-cycle.
Execution policy: `environment-gated`; disabled by default: `true`.
-Blocked reason (`ENV_DEDICATED_H6_HOST`): 需要 H6 专用主机、平台强制停启权限、带外恢复和 boot_id/平台事件证据;共享 Heal 节点不能证明真实电源循环。
+Blocked reason (`ENV_DEDICATED_H6_HOST`): Requires a dedicated H6 host, platform force stop/start permission, out-of-band recovery, and boot_id/platform-event evidence; a shared Heal node cannot prove a real power cycle.
-1. 先取得 H6 专项资源 ID、独占证明和带外恢复批准;普通 fixed fleet 禁止执行本例。
-2. 持续限量写入并在 replacement 部分重建窗口,记录已成功 ACK/结果未知写集合与最近持久化证据。
-3. HOST_POWER 使用批准的平台命令强制停启,保留平台活动 ID、时间线和 boot_id;VM stop 与真实电源断电分别命名实例。
-4. 恢复后按原盘/根重启并解析未知写;全版本/片和持久责任验收,不把 SIGKILL 或软件 reboot 当作此项证据。
+1. First obtain the H6 dedicated resource ID, proof of exclusive use, and out-of-band recovery approval; an ordinary fixed fleet must not run this case.
+2. Continue limited-volume writes and, in the replacement partial-rebuild window, record the set of writes with a successful ACK or an unknown result, and the latest persistence evidence.
+3. HOST_POWER uses an approved platform command to force stop and start, retaining the platform activity ID, timeline, and boot_id; VM stop and a real power loss are named as separate instances.
+4. After recovery, restart on the original disk/root and resolve the unknown writes; accept every version/shard and durable responsibility. Do not treat SIGKILL or a software reboot as evidence for this item.
Operations: [SEED_DX](validation-heal-operations.md#SEED_DX) → [SEED_DV](validation-heal-operations.md#SEED_DV) → [WORKLOAD](validation-heal-operations.md#WORKLOAD) → [REPLACE](validation-heal-operations.md#REPLACE) → [WINDOW](validation-heal-operations.md#WINDOW) → [HOST_POWER](validation-heal-operations.md#HOST_POWER) → [CHECKPOINT](validation-heal-operations.md#CHECKPOINT) → [PHYSICAL](validation-heal-operations.md#PHYSICAL) → [VERIFY](validation-heal-operations.md#VERIFY)
Source assertions:
-- 经专项授权在部分修复/持续写入时强制重启独占测试 VM/主机
-- 保留平台事件与 ACK ledger;验证落盘和恢复,不把 SIGKILL 当断电;VM 强制停止也不等同于物理控制器断电。
+- Under specific authorization, force-restart the exclusive test VM/host during partial repair or continuous writes
+- Retain platform events and the ACK ledger; verify the on-disk write and recovery, and do not treat SIGKILL as power loss; a forced VM stop is not equivalent to power loss of the physical controller either.
-先保存失败现场及物理/状态证据,停止本例写入与在途任务,再按 RECOVER 撤销本例故障。并发版本已变化时禁止盲目恢复旧 xl.meta;恢复不能充作产品修复。未恢复环境标 DIRTY,保留本例备份和原失败结果。
+First save the failure scene and the physical and state evidence, stop this case's writes and in-flight tasks, then undo this case's fault per RECOVER. When a concurrent version has already changed, do not blindly restore the old xl.meta; recovery must not serve as a product fix. Mark an unrestored environment DIRTY, and keep this case's backup and the original failure result.
-逐项满足设计公共契约、该分支步骤及附加断言后人工评审。缺资源为 BLOCKED,窗口未命中为 NOT_OBSERVED,产品错误为 FAIL,工具/恢复错误为 INFRA_ERROR;未运行不记 PASS。
+Manually review after each item satisfies the design's common contract, that branch's steps, and the additional assertions. Missing resources are BLOCKED, a window miss is NOT_OBSERVED, a product error is FAIL, and a tool or recovery error is INFRA_ERROR; a case that did not run is not recorded as PASS.
@@ -1327,21 +1327,21 @@ Source assertions:
Topologies: H1,H2. Branches: overwrite-vs-deep, overwrite-vs-read-repair, native-commit-cleanup.
-1. 同 key 串行 nonce 的覆盖客户端与 heal 客户端独立;WORKLOAD 每次记录 request-start/end、ACK、payload hash,不凭跨机墙钟判最后版本。
-2. 旧对象缺片/bitflip 后两端同时运行,WINDOW 证明实际交叠;每个 worker 的同 key PUT 必须串行,避免期望值本身无序。
-3. 停止写入后解析全部未知 ACK,按最后可证明提交核对最新 GET 与 data_dir;不能回滚旧内容。
-4. 运行 NATIVE commit/cleanup 屏障实例;真实竞争没命中精确窗口只算黑盒实例,记录 no-lock/锁重验日志。
+1. The overwrite client, with a per-key serial nonce, is independent of the heal client; WORKLOAD records request-start/end, ACK, and payload hash every time, and does not decide the last version from cross-machine wall clocks.
+2. After the old object has a missing shard/bitflip, run both sides at the same time; WINDOW proves actual overlap. Same-key PUTs of each worker must be serial, so the expected value is not itself unordered.
+3. After writes stop, resolve every unknown ACK and check the latest GET and data_dir against the last provable commit; do not roll back to the old content.
+4. Run the NATIVE commit/cleanup barrier instance; a real race that misses the exact window counts only as a black-box instance. Record the no-lock/lock-revalidation logs.
Operations: [SEED_DB](validation-heal-operations.md#SEED_DB) → [FAULT](validation-heal-operations.md#FAULT) → [WORKLOAD](validation-heal-operations.md#WORKLOAD) → [START](validation-heal-operations.md#START) → [WINDOW](validation-heal-operations.md#WINDOW) → [VERIFY](validation-heal-operations.md#VERIFY) → [PHYSICAL](validation-heal-operations.md#PHYSICAL) → [NATIVE](validation-heal-operations.md#NATIVE)
Source assertions:
-- 同 key 连续覆盖与 heal 同时推进,包含提交尾部清理窗口
-- 按可证明的提交顺序验证最终最后一次成功写入;不能回滚到旧 data_dir,未确认超时写入单独解析。
+- Successive overwrites of the same key proceed concurrently with heal, including the commit-tail cleanup window
+- In a provable commit order, verify that the final result is the last successful write; do not roll back to the old data_dir, and resolve an unconfirmed timed-out write separately.
-先保存失败现场及物理/状态证据,停止本例写入与在途任务,再按 RECOVER 撤销本例故障。并发版本已变化时禁止盲目恢复旧 xl.meta;恢复不能充作产品修复。未恢复环境标 DIRTY,保留本例备份和原失败结果。
+First save the failure scene and the physical and state evidence, stop this case's writes and in-flight tasks, then undo this case's fault per RECOVER. When a concurrent version has already changed, do not blindly restore the old xl.meta; recovery must not serve as a product fix. Mark an unrestored environment DIRTY, and keep this case's backup and the original failure result.
-逐项满足设计公共契约、该分支步骤及附加断言后人工评审。缺资源为 BLOCKED,窗口未命中为 NOT_OBSERVED,产品错误为 FAIL,工具/恢复错误为 INFRA_ERROR;未运行不记 PASS。
+Manually review after each item satisfies the design's common contract, that branch's steps, and the additional assertions. Missing resources are BLOCKED, a window miss is NOT_OBSERVED, a product error is FAIL, and a tool or recovery error is INFRA_ERROR; a case that did not run is not recorded as PASS.
@@ -1349,21 +1349,21 @@ Source assertions:
Topologies: H1,H2. Branches: new-version-vs-historical-heal, read-all-versions-during-heal, native-version-fence.
-1. DV 历史版本缺片时,前台串行写入独立 payload 的新版本,保存每个成功 versionId。
-2. START deep 与 WORKLOAD 交叠;读取端始终指定 versionId 验证新旧内容,禁止用 current GET 代替。
-3. 静默后按完整 ACK ledger 重建期望版本集合,最新值根据串行提交顺序判断;全量 PHYSICAL 检查不同 data_dir 无混片。
-4. 精确版本提交屏障运行 NATIVE;任何未解析超时写/缺失 ACK 阻止最终通过。
+1. When a DV historical version is missing a shard, the foreground serially writes a new version with an independent payload and saves every successful versionId.
+2. START deep overlaps WORKLOAD; the reader always specifies versionId to verify old and new content, and must not substitute a current GET.
+3. After quiescence, rebuild the expected version set from the complete ACK ledger; the latest value is determined by serial commit order. A full PHYSICAL checks that different data_dirs have no mixed shards.
+4. Run NATIVE on the exact-version commit barrier; any unresolved timed-out write or missing ACK blocks the final pass.
Operations: [SEED_DV](validation-heal-operations.md#SEED_DV) → [FAULT](validation-heal-operations.md#FAULT) → [WORKLOAD](validation-heal-operations.md#WORKLOAD) → [START](validation-heal-operations.md#START) → [WINDOW](validation-heal-operations.md#WINDOW) → [VERIFY](validation-heal-operations.md#VERIFY) → [PHYSICAL](validation-heal-operations.md#PHYSICAL) → [NATIVE](validation-heal-operations.md#NATIVE)
Source assertions:
-- 修复历史版本时创建新版本并重复读取各 versionId
-- 新旧版本不混片、不串内容;已返回成功的每个版本都保留,最新值按确定的先后关系验收。
+- While repairing a historical version, create a new version and read each versionId repeatedly
+- Old and new versions do not mix shards or cross content; every version that has already returned success is retained, and the latest value is accepted according to a determined order.
-先保存失败现场及物理/状态证据,停止本例写入与在途任务,再按 RECOVER 撤销本例故障。并发版本已变化时禁止盲目恢复旧 xl.meta;恢复不能充作产品修复。未恢复环境标 DIRTY,保留本例备份和原失败结果。
+First save the failure scene and the physical and state evidence, stop this case's writes and in-flight tasks, then undo this case's fault per RECOVER. When a concurrent version has already changed, do not blindly restore the old xl.meta; recovery must not serve as a product fix. Mark an unrestored environment DIRTY, and keep this case's backup and the original failure result.
-逐项满足设计公共契约、该分支步骤及附加断言后人工评审。缺资源为 BLOCKED,窗口未命中为 NOT_OBSERVED,产品错误为 FAIL,工具/恢复错误为 INFRA_ERROR;未运行不记 PASS。
+Manually review after each item satisfies the design's common contract, that branch's steps, and the additional assertions. Missing resources are BLOCKED, a window miss is NOT_OBSERVED, a product error is FAIL, and a tool or recovery error is INFRA_ERROR; a case that did not run is not recorded as PASS.
@@ -1371,21 +1371,21 @@ Source assertions:
Topologies: H1,H2. Branches: delete-old-version-vs-deep, delete-current-version-vs-deep, native-delete-fence.
-1. 选 DV 的精确 versionId 制造可恢复缺片;前台删除请求明确带该 ID,不是只创建 marker。
-2. WINDOW 内并发 delete_object 与 deep heal,保存 DELETE 成功回执和 heal scope/identity。
-3. 终态按精确 version GET 应不存在,LIST versions 不得复活;其他所有版本内容保持。
-4. 运行 NATIVE 删除/修复提交交错点;清理旧片仅作为最终物理证据,不能先用备份复活已删版本。
+1. Select an exact versionId from DV and induce a recoverable missing shard; the foreground delete request explicitly carries that ID, and does not only create a marker.
+2. Within the WINDOW, run delete_object concurrently with deep heal, and save the DELETE success receipt and the heal scope/identity.
+3. In the terminal state, GET of the exact version should find it absent, and LIST versions must not resurrect it; the content of every other version is preserved.
+4. Run NATIVE at the delete/repair commit interleaving point; cleanup of the old shard is only final physical evidence, and a backup must not be used first to resurrect a deleted version.
Operations: [SEED_DV](validation-heal-operations.md#SEED_DV) → [FAULT](validation-heal-operations.md#FAULT) → [START](validation-heal-operations.md#START) → [WORKLOAD](validation-heal-operations.md#WORKLOAD) → [WINDOW](validation-heal-operations.md#WINDOW) → [S3](validation-heal-operations.md#S3) → [VERIFY](validation-heal-operations.md#VERIFY) → [PHYSICAL](validation-heal-operations.md#PHYSICAL) → [NATIVE](validation-heal-operations.md#NATIVE)
Source assertions:
-- 对损坏的精确历史 versionId 发 DELETE,与 deep heal 竞争
-- 已确认删除的版本不能被复活,其他历史版本不受影响;原生临界点测试与真机竞争分别留证。
+- Issue DELETE for the exact corrupted historical versionId, racing deep heal
+- A version whose deletion is confirmed cannot be resurrected, and other historical versions are unaffected; the native boundary test and the live race on the fleet each keep their own evidence.
-先保存失败现场及物理/状态证据,停止本例写入与在途任务,再按 RECOVER 撤销本例故障。并发版本已变化时禁止盲目恢复旧 xl.meta;恢复不能充作产品修复。未恢复环境标 DIRTY,保留本例备份和原失败结果。
+First save the failure scene and the physical and state evidence, stop this case's writes and in-flight tasks, then undo this case's fault per RECOVER. When a concurrent version has already changed, do not blindly restore the old xl.meta; recovery must not serve as a product fix. Mark an unrestored environment DIRTY, and keep this case's backup and the original failure result.
-逐项满足设计公共契约、该分支步骤及附加断言后人工评审。缺资源为 BLOCKED,窗口未命中为 NOT_OBSERVED,产品错误为 FAIL,工具/恢复错误为 INFRA_ERROR;未运行不记 PASS。
+Manually review after each item satisfies the design's common contract, that branch's steps, and the additional assertions. Missing resources are BLOCKED, a window miss is NOT_OBSERVED, a product error is FAIL, and a tool or recovery error is INFRA_ERROR; a case that did not run is not recorded as PASS.
@@ -1393,21 +1393,21 @@ Source assertions:
Topologies: H1,H2. Branches: create-marker, delete-marker, batch-version-delete, unversioned-delete, native-marker-fence.
-1. 分别建立版本化和非版本化桶;marker创建/删除、批量带versionId删除和无版本DELETE分开运行,不混淆接口语义。
-2. 在 xl.meta 少数缺失下启动 heal,同时提交冻结的删除列表;delete_objects 必须逐项检查 Deleted/Errors,HTTP200不代表全成功。
-3. 按逐项 ACK 计算最终数据版本、marker 和最新值集合;精确 GET验证不存在项和存留项。
-4. native-marker-fence 独立取证;恢复故障不得把已确认删除的旧 metadata 覆盖回来。
+1. Create a versioned bucket and an unversioned bucket separately; run marker create/delete, batch delete with versionId, and versionless DELETE separately, without mixing the API semantics.
+2. Start heal with a minority of xl.meta missing, and submit the frozen delete list at the same time; delete_objects must check Deleted/Errors item by item. HTTP 200 does not mean every item succeeded.
+3. From per-item ACKs, compute the final sets of data versions, markers, and latest values; exact GET verifies absent items and retained items.
+4. native-marker-fence is evidenced independently; recovering the fault must not overwrite back old metadata that was already confirmed deleted.
Operations: [SEED_DV](validation-heal-operations.md#SEED_DV) → [SEED_DB](validation-heal-operations.md#SEED_DB) → [FAULT](validation-heal-operations.md#FAULT) → [START](validation-heal-operations.md#START) → [WORKLOAD](validation-heal-operations.md#WORKLOAD) → [S3](validation-heal-operations.md#S3) → [VERIFY](validation-heal-operations.md#VERIFY) → [PHYSICAL](validation-heal-operations.md#PHYSICAL) → [NATIVE](validation-heal-operations.md#NATIVE)
Source assertions:
-- 与创建/删除 marker、批量版本删除和无版本删除分别交错
-- 按逐项 DELETE 回执核对版本/marker 集合,无复活、误删或遗漏批量错误。
+- Interleave separately with marker create/delete, batch version delete, and unversioned delete
+- Check the version/marker set against each DELETE receipt, with no resurrection, no mistaken deletion, and no omitted batch error.
-先保存失败现场及物理/状态证据,停止本例写入与在途任务,再按 RECOVER 撤销本例故障。并发版本已变化时禁止盲目恢复旧 xl.meta;恢复不能充作产品修复。未恢复环境标 DIRTY,保留本例备份和原失败结果。
+First save the failure scene and the physical and state evidence, stop this case's writes and in-flight tasks, then undo this case's fault per RECOVER. When a concurrent version has already changed, do not blindly restore the old xl.meta; recovery must not serve as a product fix. Mark an unrestored environment DIRTY, and keep this case's backup and the original failure result.
-逐项满足设计公共契约、该分支步骤及附加断言后人工评审。缺资源为 BLOCKED,窗口未命中为 NOT_OBSERVED,产品错误为 FAIL,工具/恢复错误为 INFRA_ERROR;未运行不记 PASS。
+Manually review after each item satisfies the design's common contract, that branch's steps, and the additional assertions. Missing resources are BLOCKED, a window miss is NOT_OBSERVED, a product error is FAIL, and a tool or recovery error is INFRA_ERROR; a case that did not run is not recorded as PASS.
@@ -1415,21 +1415,21 @@ Source assertions:
Topologies: H1,H2. Branches: upload-continue, complete, abort, native-mpu-fence.
-1. DM 同时准备已提交对象与两个可识别的 pending UploadId,各自 part/ETag 独立。
-2. 已提交对象缺片并 heal;并发给 upload-A 续传并 Complete,upload-B Abort,WINDOW 保存实际交叠。
-3. Complete 返回版本只出现一次,完整内容为按PartNumber排序的ACK part;Abort 的 UploadId 不再可list_parts,不能误伤另一个upload或已提交对象。
-4. 核对所有 MPU 列表和物理目录;原生提交/取消屏障另用 NATIVE,不允许仅发出并发请求即声称命中。
+1. DM prepares, at the same time, a committed object and two identifiable pending UploadIds, each with an independent part/ETag.
+2. The committed object is missing a shard and is healed; concurrently resume upload-A and Complete it, and Abort upload-B. WINDOW saves the actual overlap.
+3. The version returned by Complete appears only once, and the full content is the ACKed parts sorted by PartNumber; the Aborted UploadId can no longer list_parts, and must not harm the other upload or the committed object.
+4. Check every MPU listing and physical directory; the native commit/cancel barrier uses NATIVE separately. Issuing concurrent requests alone must not be claimed as a hit.
Operations: [SEED_DM](validation-heal-operations.md#SEED_DM) → [FAULT](validation-heal-operations.md#FAULT) → [START](validation-heal-operations.md#START) → [WORKLOAD](validation-heal-operations.md#WORKLOAD) → [WINDOW](validation-heal-operations.md#WINDOW) → [S3](validation-heal-operations.md#S3) → [VERIFY](validation-heal-operations.md#VERIFY) → [PHYSICAL](validation-heal-operations.md#PHYSICAL) → [NATIVE](validation-heal-operations.md#NATIVE)
Source assertions:
-- heal 期间对不同已知 UploadId 分别续传、Complete、Abort
-- 成功 Complete 内容只出现一次;Abort 不破坏已提交对象或别的 upload;未完成 part 不被伪造为完整对象。
+- During heal, separately resume, Complete, and Abort different known UploadIds
+- Content from a successful Complete appears only once; Abort does not damage a committed object or another upload; an unfinished part is not forged into a complete object.
-先保存失败现场及物理/状态证据,停止本例写入与在途任务,再按 RECOVER 撤销本例故障。并发版本已变化时禁止盲目恢复旧 xl.meta;恢复不能充作产品修复。未恢复环境标 DIRTY,保留本例备份和原失败结果。
+First save the failure scene and the physical and state evidence, stop this case's writes and in-flight tasks, then undo this case's fault per RECOVER. When a concurrent version has already changed, do not blindly restore the old xl.meta; recovery must not serve as a product fix. Mark an unrestored environment DIRTY, and keep this case's backup and the original failure result.
-逐项满足设计公共契约、该分支步骤及附加断言后人工评审。缺资源为 BLOCKED,窗口未命中为 NOT_OBSERVED,产品错误为 FAIL,工具/恢复错误为 INFRA_ERROR;未运行不记 PASS。
+Manually review after each item satisfies the design's common contract, that branch's steps, and the additional assertions. Missing resources are BLOCKED, a window miss is NOT_OBSERVED, a product error is FAIL, and a tool or recovery error is INFRA_ERROR; a case that did not run is not recorded as PASS.
@@ -1437,21 +1437,21 @@ Source assertions:
Topologies: H1,H2. Branches: copy-source, copy-destination, replace-metadata, update-tags, native-copy-fence.
-1. DS 源/目标使用不同内容和 metadata,copy 请求 CopySource 使用结构化 Bucket/Key/VersionId,不拼路径字符串。
-2. 源或目标缺片并 heal,同窗口 COPY、MetadataDirective=REPLACE 的自拷贝、put_object_tagging 分别运行。
-3. 保存对应提交ETag/versionId及更新顺序,验证最终内容与该提交 metadata/tags 组合一致;旧 read-repair 必须重验代次。
-4. 检查源目标和control均无交叉覆盖;NATIVE 记录锁升级/陈旧提交拒绝的精确窗口。
+1. The DS source and target use different content and metadata; the copy request's CopySource uses structured Bucket/Key/VersionId and does not concatenate a path string.
+2. The source or the target is missing a shard and is healed; in the same window, run COPY, a self-copy with MetadataDirective=REPLACE, and put_object_tagging separately.
+3. Save the corresponding commit ETag/versionId and the update order, and verify the final content matches that commit's metadata/tags combination; an old read-repair must revalidate the generation.
+4. Check that the source, the target, and the control have no cross-overwrite; NATIVE records the exact window of lock upgrade/stale-commit rejection.
Operations: [SEED_DS](validation-heal-operations.md#SEED_DS) → [FAULT](validation-heal-operations.md#FAULT) → [START](validation-heal-operations.md#START) → [WORKLOAD](validation-heal-operations.md#WORKLOAD) → [WINDOW](validation-heal-operations.md#WINDOW) → [S3](validation-heal-operations.md#S3) → [VERIFY](validation-heal-operations.md#VERIFY) → [PHYSICAL](validation-heal-operations.md#PHYSICAL) → [NATIVE](validation-heal-operations.md#NATIVE)
Source assertions:
-- COPY 源/目标和标签/元数据更新与修复交错
-- 最终对象内容与对应提交的 metadata 一致,读修复发现新代次应重验/放弃陈旧提交而非覆盖。
+- Interleave COPY of the source/target and tag/metadata updates with repair
+- The final object content matches the metadata of the corresponding commit; when read-repair finds a new generation, it re-verifies or abandons the stale commit instead of overwriting.
-先保存失败现场及物理/状态证据,停止本例写入与在途任务,再按 RECOVER 撤销本例故障。并发版本已变化时禁止盲目恢复旧 xl.meta;恢复不能充作产品修复。未恢复环境标 DIRTY,保留本例备份和原失败结果。
+First save the failure scene and the physical and state evidence, stop this case's writes and in-flight tasks, then undo this case's fault per RECOVER. When a concurrent version has already changed, do not blindly restore the old xl.meta; recovery must not serve as a product fix. Mark an unrestored environment DIRTY, and keep this case's backup and the original failure result.
-逐项满足设计公共契约、该分支步骤及附加断言后人工评审。缺资源为 BLOCKED,窗口未命中为 NOT_OBSERVED,产品错误为 FAIL,工具/恢复错误为 INFRA_ERROR;未运行不记 PASS。
+Manually review after each item satisfies the design's common contract, that branch's steps, and the additional assertions. Missing resources are BLOCKED, a window miss is NOT_OBSERVED, a product error is FAIL, and a tool or recovery error is INFRA_ERROR; a case that did not run is not recorded as PASS.
@@ -1461,23 +1461,23 @@ Topologies: H2. Branches: current-expiry, noncurrent-expiry, transition, restore
Execution policy: `environment-gated`; disabled by default: `true`.
-Blocked reason (`ENV_DEDICATED_ILM_TIER_WINDOW`): 需要隔离 ILM/tier 租户、真实 Expire/Transition/Restore 窗口和远端版本映射;共享 ILM 环境不能安全制造交叠窗口。
+Blocked reason (`ENV_DEDICATED_ILM_TIER_WINDOW`): Requires an isolated ILM/tier tenant, a real Expire/Transition/Restore window, and remote version mapping; a shared ILM environment cannot safely create an overlap window.
-1. 用现有 ILM 专用远端 tier 能力建立本例桶和规则;记录真实到期窗口/远端version映射,不在共享桶上改规则。
-2. 在实际出现 Expire/Transition/Restore 进度时启动缺片 heal,WINDOW 必须从日志/状态证明交叠;提前完成只能记 NOT_OBSERVED。
-3. 过期 ACK/ILM 删除证明决定哪些版本应不存在,禁止 heal 复活;迁移对象通过真实远端 GET/restore 验证原版本内容。
-4. 本地物理与远端引用双重校验;源端 GET 成功不足,清理只删除本例规则/远端数据。
+1. Use the existing ILM dedicated remote-tier capability to create this case's bucket and rules; record the real expiry window/remote version mapping, and do not change rules on a shared bucket.
+2. Start the missing-shard heal when Expire/Transition/Restore progress actually appears; WINDOW must prove overlap from logs/state. Finishing early can only be recorded as NOT_OBSERVED.
+3. Expiry ACKs/ILM delete proofs decide which versions should be absent; heal must not resurrect them. Migrated objects verify the original version content through a real remote GET/restore.
+4. Double-check local physical data and the remote reference; a successful source-side GET is not sufficient. Cleanup deletes only this case's rules/remote data.
Operations: [SEED_DV](validation-heal-operations.md#SEED_DV) → [SEED_DE](validation-heal-operations.md#SEED_DE) → [DEPENDENCY](validation-heal-operations.md#DEPENDENCY) → [ILM](validation-heal-operations.md#ILM) → [FAULT](validation-heal-operations.md#FAULT) → [START](validation-heal-operations.md#START) → [WINDOW](validation-heal-operations.md#WINDOW) → [S3](validation-heal-operations.md#S3) → [PHYSICAL](validation-heal-operations.md#PHYSICAL) → [VERIFY](validation-heal-operations.md#VERIFY) → [NATIVE](validation-heal-operations.md#NATIVE)
Source assertions:
-- 分别与当前/历史版本过期、远端 transition/restore 同窗口执行
-- 已授权过期不被恢复;迁移状态与远端版本引用正确;只有实际观察到交叠才计并发覆盖,tier 未配置需明确阻断对应实例。
+- Run in the same window as expiry of the current or historical version, and as remote transition/restore, respectively
+- An authorized expiry is not restored; migration state and remote version references are correct; concurrent coverage counts only when overlap is actually observed, and an unconfigured tier must explicitly block the corresponding instance.
-先保存失败现场及物理/状态证据,停止本例写入与在途任务,再按 RECOVER 撤销本例故障。并发版本已变化时禁止盲目恢复旧 xl.meta;恢复不能充作产品修复。未恢复环境标 DIRTY,保留本例备份和原失败结果。
+First save the failure scene and the physical and state evidence, stop this case's writes and in-flight tasks, then undo this case's fault per RECOVER. When a concurrent version has already changed, do not blindly restore the old xl.meta; recovery must not serve as a product fix. Mark an unrestored environment DIRTY, and keep this case's backup and the original failure result.
-逐项满足设计公共契约、该分支步骤及附加断言后人工评审。缺资源为 BLOCKED,窗口未命中为 NOT_OBSERVED,产品错误为 FAIL,工具/恢复错误为 INFRA_ERROR;未运行不记 PASS。
+Manually review after each item satisfies the design's common contract, that branch's steps, and the additional assertions. Missing resources are BLOCKED, a window miss is NOT_OBSERVED, a product error is FAIL, and a tool or recovery error is INFRA_ERROR; a case that did not run is not recorded as PASS.
@@ -1487,23 +1487,23 @@ Topologies: H2. Branches: replicate-put, replicate-delete, resync, native-replic
Execution policy: `environment-gated`; disabled by default: `true`.
-Blocked reason (`ENV_ISOLATED_REPLICATION_PAIR`): 需要隔离的复制两端、resync 控制、跨端版本映射和 Heal/resync 实际交叠窗口;当前固定 fleet 没有独占复制对。
+Blocked reason (`ENV_ISOLATED_REPLICATION_PAIR`): Requires isolated replication endpoints, resync control, cross-endpoint version mapping, and an actual Heal/resync overlap window; the current fixed fleet has no exclusive replication pair.
-1. 准备两个隔离复制端点/本例桶、明确源目标版本映射和同步规则;先跑一份健康复制控制。
-2. 源端缺片 heal 同时提交 PUT/DELETE 并启动指定 resync;保存两侧日志与跨端版本映射,证明操作时间交叠。
-3. 分别从两端全量列举版本、marker、内容并比对复制契约;源端正确但远端缺失/无限重试必须失败。
-4. 恢复临时规则/凭据,确认无无限循环/重复传播;精确交错仅由匹配 NATIVE 证据补充。
+1. Prepare two isolated replication endpoints/this case's buckets, an explicit source-target version mapping, and sync rules; first run one healthy replication control.
+2. While the source heals a missing shard, concurrently submit PUT/DELETE and start the specified resync; save logs on both sides and the cross-endpoint version mapping to prove the operations overlap in time.
+3. Fully list versions, markers, and content from both ends and compare them with the replication contract; if the source is correct but the remote side is missing or retries forever, the case must fail.
+4. Restore the temporary rules and credentials, and confirm there is no infinite loop or repeated propagation; precise interleaving may be supplied only by matching NATIVE evidence.
Operations: [SEED_DV](validation-heal-operations.md#SEED_DV) → [DEPENDENCY](validation-heal-operations.md#DEPENDENCY) → [REPLICATION](validation-heal-operations.md#REPLICATION) → [FAULT](validation-heal-operations.md#FAULT) → [START](validation-heal-operations.md#START) → [WORKLOAD](validation-heal-operations.md#WORKLOAD) → [WINDOW](validation-heal-operations.md#WINDOW) → [VERIFY](validation-heal-operations.md#VERIFY) → [LOGS](validation-heal-operations.md#LOGS) → [NATIVE](validation-heal-operations.md#NATIVE)
Source assertions:
-- 在隔离复制两端与 heal 同时运行新增/删除/重同步
-- 对端版本、删除语义及源数据最终一致,无无限回环或重复传播;不得依赖仅源端 GET 宣称复制正确。
+- On both isolated replication ends, run create, delete, and resync concurrently with heal
+- Peer versions, delete semantics, and source data end up consistent, with no infinite loop-back and no duplicate propagation; do not claim replication is correct from a source-side GET alone.
-先保存失败现场及物理/状态证据,停止本例写入与在途任务,再按 RECOVER 撤销本例故障。并发版本已变化时禁止盲目恢复旧 xl.meta;恢复不能充作产品修复。未恢复环境标 DIRTY,保留本例备份和原失败结果。
+First save the failure scene and the physical and state evidence, stop this case's writes and in-flight tasks, then undo this case's fault per RECOVER. When a concurrent version has already changed, do not blindly restore the old xl.meta; recovery must not serve as a product fix. Mark an unrestored environment DIRTY, and keep this case's backup and the original failure result.
-逐项满足设计公共契约、该分支步骤及附加断言后人工评审。缺资源为 BLOCKED,窗口未命中为 NOT_OBSERVED,产品错误为 FAIL,工具/恢复错误为 INFRA_ERROR;未运行不记 PASS。
+Manually review after each item satisfies the design's common contract, that branch's steps, and the additional assertions. Missing resources are BLOCKED, a window miss is NOT_OBSERVED, a product error is FAIL, and a tool or recovery error is INFRA_ERROR; a case that did not run is not recorded as PASS.
@@ -1513,23 +1513,23 @@ Topologies: H5. Branches: expand-while-data-heal, expand-while-parity-heal, vers
Execution policy: `environment-gated`; disabled by default: `true`.
-Blocked reason (`ENV_H5_EXPANSION_REBALANCE`): 需要 H5 pool 扩容、rebalance 和 Heal 同时活动的完整拓扑,并逐对象保存 pool/set 迁移证据;当前 auto-testing 未绑定扩容前后布局与专用操作窗口。
+Blocked reason (`ENV_H5_EXPANSION_REBALANCE`): Requires a complete topology in which H5 pool expansion, rebalance, and Heal are active at the same time, with pool/set migration evidence saved per object; current auto-testing has not bound the layout before and after expansion or a dedicated operation window.
-1. 使用已批准 H5,先在源 pool 写 DB/DV 并以 FORMAT/PHYSICAL 证明位置;新 pool 为空且未纳入旧选择器。
-2. 目标 data/parity 缺失或bitflip后按 POOL 手册扩容重启,确认新拓扑;启动 rebalance 与 heal。
-3. WINDOW 保存两任务活动证据,逐对象跟踪源/目标 pool/set/版本身份,不能只看全局数量。
-4. 两任务均正常完成后物理证明目标权威副本,VERIFY全版本,保留scanner backlog的持久提交/退休证据。
+1. Using the approved H5, first write DB/DV in the source pool and prove placement with FORMAT/PHYSICAL; the new pool is empty and is not included in the old selector.
+2. After the target data/parity is missing or bitflipped, expand and restart per the POOL manual, confirm the new topology, and start rebalance and heal.
+3. WINDOW saves evidence that both tasks are active and tracks source/target pool/set/version identity per object; looking only at global counts is not enough.
+4. After both tasks complete normally, physically prove the target authority replicas, VERIFY every version, and retain the scanner backlog's persisted commit and retirement evidence.
Operations: [SEED_DB](validation-heal-operations.md#SEED_DB) → [SEED_DV](validation-heal-operations.md#SEED_DV) → [FORMAT](validation-heal-operations.md#FORMAT) → [FAULT](validation-heal-operations.md#FAULT) → [POOL](validation-heal-operations.md#POOL) → [START](validation-heal-operations.md#START) → [WINDOW](validation-heal-operations.md#WINDOW) → [BACKGROUND](validation-heal-operations.md#BACKGROUND) → [PHYSICAL](validation-heal-operations.md#PHYSICAL) → [VERIFY](validation-heal-operations.md#VERIFY)
Source assertions:
-- 先证明源 pool 持有夹具,再扩容并与修复同时 rebalance
-- 修复目标不得指向错误 pool/set;新旧数据完整、移动终态正确,保留 scanner backlog 提交权威检查。
+- First prove the source pool holds the fixture, then expand and rebalance concurrently with repair
+- The repair target must not point at the wrong pool/set; old and new data are complete and the move terminal state is correct, and the scanner-backlog commit authority check is retained.
-先保存失败现场及物理/状态证据,停止本例写入与在途任务,再按 RECOVER 撤销本例故障。并发版本已变化时禁止盲目恢复旧 xl.meta;恢复不能充作产品修复。未恢复环境标 DIRTY,保留本例备份和原失败结果。
+First save the failure scene and the physical and state evidence, stop this case's writes and in-flight tasks, then undo this case's fault per RECOVER. When a concurrent version has already changed, do not blindly restore the old xl.meta; recovery must not serve as a product fix. Mark an unrestored environment DIRTY, and keep this case's backup and the original failure result.
-逐项满足设计公共契约、该分支步骤及附加断言后人工评审。缺资源为 BLOCKED,窗口未命中为 NOT_OBSERVED,产品错误为 FAIL,工具/恢复错误为 INFRA_ERROR;未运行不记 PASS。
+Manually review after each item satisfies the design's common contract, that branch's steps, and the additional assertions. Missing resources are BLOCKED, a window miss is NOT_OBSERVED, a product error is FAIL, and a tool or recovery error is INFRA_ERROR; a case that did not run is not recorded as PASS.
@@ -1539,23 +1539,23 @@ Topologies: H5. Branches: rebalance-stop, rebalance-resume, rebalance-service-re
Execution policy: `environment-gated`; disabled by default: `true`.
-Blocked reason (`ENV_H5_REBALANCE_LIFECYCLE`): 需要可观测的 H5 rebalance、停止/恢复/服务重启控制及独立 task 身份;当前 auto-testing 未绑定这两类任务的独立身份和操作窗口。
+Blocked reason (`ENV_H5_REBALANCE_LIFECYCLE`): Requires observable H5 rebalance, stop/resume/service-restart control, and independent task identities; current auto-testing has not bound the independent identities and operation windows of these two task classes.
-1. H5 中使用足够对象使移动可观测,缺片 heal 与 rebalance 同时活动;记录两类独立 task ID。
-2. 只停止 rebalance,观察 heal 不被误取消;然后恢复 rebalance,重启实例仅重启本例 unit。
-3. 持续唯一 key 写入并保存 ACK;停止/恢复前后物理和任务状态要连续,不用新数据根替代恢复。
-4. 最终两类任务各自满足完成契约、对象/版本及移动目标正确,重复停止不影响新代次。
+1. On H5, use enough objects for movement to be observable, with missing-shard heal and rebalance active at the same time; record an independent task ID for each of the two classes.
+2. Stop only rebalance, and observe that heal is not cancelled by mistake; then resume rebalance. The restart instance restarts only this case's unit.
+3. Keep writing unique keys and save the ACKs; physical state and task state must stay continuous across stop and resume, and a new data root must not stand in for recovery.
+4. In the end, each of the two task classes meets its completion contract, and the objects, versions, and move targets are correct; stopping again does not affect the new generation.
Operations: [SEED_DB](validation-heal-operations.md#SEED_DB) → [POOL](validation-heal-operations.md#POOL) → [FAULT](validation-heal-operations.md#FAULT) → [START](validation-heal-operations.md#START) → [WINDOW](validation-heal-operations.md#WINDOW) → [PROCESS](validation-heal-operations.md#PROCESS) → [WORKLOAD](validation-heal-operations.md#WORKLOAD) → [PHYSICAL](validation-heal-operations.md#PHYSICAL) → [VERIFY](validation-heal-operations.md#VERIFY)
Source assertions:
-- heal 活动期间 stop/restart/resume rebalance
-- 两类任务状态和身份独立;停止/恢复移动不误终止修复或丢失对象,恢复后两者均满足各自契约。
+- While heal is active, stop/restart/resume rebalance
+- The two kinds of tasks have independent state and identity; stopping or resuming the move does not terminate repair by mistake or lose objects, and after resume both satisfy their own contracts.
-先保存失败现场及物理/状态证据,停止本例写入与在途任务,再按 RECOVER 撤销本例故障。并发版本已变化时禁止盲目恢复旧 xl.meta;恢复不能充作产品修复。未恢复环境标 DIRTY,保留本例备份和原失败结果。
+First save the failure scene and the physical and state evidence, stop this case's writes and in-flight tasks, then undo this case's fault per RECOVER. When a concurrent version has already changed, do not blindly restore the old xl.meta; recovery must not serve as a product fix. Mark an unrestored environment DIRTY, and keep this case's backup and the original failure result.
-逐项满足设计公共契约、该分支步骤及附加断言后人工评审。缺资源为 BLOCKED,窗口未命中为 NOT_OBSERVED,产品错误为 FAIL,工具/恢复错误为 INFRA_ERROR;未运行不记 PASS。
+Manually review after each item satisfies the design's common contract, that branch's steps, and the additional assertions. Missing resources are BLOCKED, a window miss is NOT_OBSERVED, a product error is FAIL, and a tool or recovery error is INFRA_ERROR; a case that did not run is not recorded as PASS.
@@ -1565,23 +1565,23 @@ Topologies: H5. Branches: heal-decommission-source, heal-survivor-target, versio
Execution policy: `environment-gated`; disabled by default: `true`.
-Blocked reason (`ENV_H5_DECOMMISSION`): 需要 H5 pool decommission、survivor 目标、写入限制和移动 ACK/物理权威证据;当前 auto-testing 未绑定本例退役前后拓扑和恢复方案。
+Blocked reason (`ENV_H5_DECOMMISSION`): Requires H5 pool decommission, a survivor target, write restrictions, and move-ACK and physical-authority evidence; current auto-testing has not bound this case's topology before and after decommission, or its recovery plan.
-1. 先证实夹具位于待退役源,另写 survivor control;分别在源和目标制造在quorum内可恢复故障。
-2. 按既有 Pool 行为启动退役与 heal;退役期间按当前与 MinIO 一致的写入限制检查拒绝,不能把拒绝写当数据丢失。
-3. 源数据只能在目标提交权威后删除;记录移动 ACK 与 target physical,而不是临时副本数量。
-4. 退役完成后停止/移除源,只访问 survivor 全版本/MPU/内容;若还依赖源兜底必失败。
+1. First confirm that the fixture sits on the source pending decommission, and write a separate survivor control; on the source and on the target, inject recoverable faults that remain within quorum.
+2. Start decommission and heal according to existing Pool behavior; during decommission, check rejections against the current write restrictions that match MinIO, and do not treat a rejected write as data loss.
+3. Source data may be deleted only after the target commits authority; record the move ACK and target physical, not the number of temporary replicas.
+4. After decommission completes, stop and remove the source, and read only the survivor for every version, MPU, and the content; if the result still depends on the source as a fallback, the case must fail.
Operations: [SEED_DB](validation-heal-operations.md#SEED_DB) → [SEED_DV](validation-heal-operations.md#SEED_DV) → [SEED_DM](validation-heal-operations.md#SEED_DM) → [POOL](validation-heal-operations.md#POOL) → [FAULT](validation-heal-operations.md#FAULT) → [START](validation-heal-operations.md#START) → [WINDOW](validation-heal-operations.md#WINDOW) → [PHYSICAL](validation-heal-operations.md#PHYSICAL) → [VERIFY](validation-heal-operations.md#VERIFY)
Source assertions:
-- 分别在待退役源和存活目标制造可恢复缺片,再执行退役
-- 不得提前删源或以临时副本冒充迁移成功;按 Pool 既定写入限制验收;只剩 survivor 后仍完整可读且有物理证明。
+- On the source pending decommission and on the surviving target, separately create recoverable missing shards, then run decommission
+- Do not delete the source early or pass a temporary replica off as a successful migration; accept according to the Pool's established write restrictions; after only the survivor remains, the data is still fully readable and there is physical proof.
-先保存失败现场及物理/状态证据,停止本例写入与在途任务,再按 RECOVER 撤销本例故障。并发版本已变化时禁止盲目恢复旧 xl.meta;恢复不能充作产品修复。未恢复环境标 DIRTY,保留本例备份和原失败结果。
+First save the failure scene and the physical and state evidence, stop this case's writes and in-flight tasks, then undo this case's fault per RECOVER. When a concurrent version has already changed, do not blindly restore the old xl.meta; recovery must not serve as a product fix. Mark an unrestored environment DIRTY, and keep this case's backup and the original failure result.
-逐项满足设计公共契约、该分支步骤及附加断言后人工评审。缺资源为 BLOCKED,窗口未命中为 NOT_OBSERVED,产品错误为 FAIL,工具/恢复错误为 INFRA_ERROR;未运行不记 PASS。
+Manually review after each item satisfies the design's common contract, that branch's steps, and the additional assertions. Missing resources are BLOCKED, a window miss is NOT_OBSERVED, a product error is FAIL, and a tool or recovery error is INFRA_ERROR; a case that did not run is not recorded as PASS.
@@ -1591,23 +1591,23 @@ Topologies: H5. Branches: cancel-during-heal, clear-after-inflight-exit, resume-
Execution policy: `environment-gated`; disabled by default: `true`.
-Blocked reason (`ENV_H5_DECOMMISSION_CANCEL`): 需要退役取消/清理/新代次恢复以及 #2423 所需容量锁重试窗口;普通取消结果不能替代这个临界点。
+Blocked reason (`ENV_H5_DECOMMISSION_CANCEL`): Requires decommission cancellation, cleanup, and new-generation recovery, plus the capacity-lock retry window required by #2423; an ordinary cancellation result cannot take the place of this boundary.
-1. 在移动和 heal 都有部分进度时取消退役;保存取消持久化记录、通知和在途退出顺序。
-2. 确认已退出后 Clear,再恢复/新建退役;旧 token/代次不得误取消新任务,持续 ACK ledger 不能丢。
-3. 容量锁超时实例需要 LOCK_WINDOW 或匹配 NATIVE 屏障,实际见证 #2423 所涉重试路径;普通取消成功不能代替。
-4. 两流程收敛后 VERIFY/PHYSICAL;没有命中锁窗口单列 NOT_OBSERVED,不声称覆盖 #2423 的临界点。
+1. Cancel decommission while both the move and heal have partial progress; save the persisted cancellation record, the notifications, and the in-flight exit order.
+2. After confirming that it has exited, Clear, then resume or create a new decommission; an old token or generation must not cancel the new task by mistake, and the continuous ACK ledger must not be lost.
+3. The capacity-lock timeout instance requires a LOCK_WINDOW or a matching NATIVE barrier, and must actually witness the retry path involved in #2423; an ordinary successful cancellation cannot take its place.
+4. After both flows converge, run VERIFY/PHYSICAL; if the lock window was not hit, record NOT_OBSERVED on its own, and do not claim coverage of the #2423 boundary.
Operations: [SEED_DB](validation-heal-operations.md#SEED_DB) → [POOL](validation-heal-operations.md#POOL) → [FAULT](validation-heal-operations.md#FAULT) → [START](validation-heal-operations.md#START) → [WINDOW](validation-heal-operations.md#WINDOW) → [WORKLOAD](validation-heal-operations.md#WORKLOAD) → [LOCK_WINDOW](validation-heal-operations.md#LOCK_WINDOW) → [CHECKPOINT](validation-heal-operations.md#CHECKPOINT) → [PHYSICAL](validation-heal-operations.md#PHYSICAL) → [VERIFY](validation-heal-operations.md#VERIFY) → [NATIVE](validation-heal-operations.md#NATIVE)
Source assertions:
-- heal/移动争用中取消退役、Clear、恢复,并覆盖容量锁超时恢复
-- 保持持久化后通知及在途退出顺序;不误取消新代次;对 #2423 的重试窗口单独注明真机是否实际触发,不只看最终绿色。
+- During heal/move contention, cancel decommission, Clear, and resume, and cover recovery from capacity-lock timeout
+- Keep the order of notify-after-persist and in-flight exit; do not cancel a new generation by mistake; for the #2423 retry window, separately note whether the live run on the fleet actually triggered it, and do not look only at the final green result.
-先保存失败现场及物理/状态证据,停止本例写入与在途任务,再按 RECOVER 撤销本例故障。并发版本已变化时禁止盲目恢复旧 xl.meta;恢复不能充作产品修复。未恢复环境标 DIRTY,保留本例备份和原失败结果。
+First save the failure scene and the physical and state evidence, stop this case's writes and in-flight tasks, then undo this case's fault per RECOVER. When a concurrent version has already changed, do not blindly restore the old xl.meta; recovery must not serve as a product fix. Mark an unrestored environment DIRTY, and keep this case's backup and the original failure result.
-逐项满足设计公共契约、该分支步骤及附加断言后人工评审。缺资源为 BLOCKED,窗口未命中为 NOT_OBSERVED,产品错误为 FAIL,工具/恢复错误为 INFRA_ERROR;未运行不记 PASS。
+Manually review after each item satisfies the design's common contract, that branch's steps, and the additional assertions. Missing resources are BLOCKED, a window miss is NOT_OBSERVED, a product error is FAIL, and a tool or recovery error is INFRA_ERROR; a case that did not run is not recorded as PASS.
@@ -1617,23 +1617,23 @@ Topologies: H5. Branches: survivor-missing-part, survivor-missing-metadata, reti
Execution policy: `environment-gated`; disabled by default: `true`.
-Blocked reason (`ENV_H5_RETIRED_POOL_REMOVAL`): 需要完整迁移后移除 retired pool、源保持离线和 survivor-only 访问;当前 auto-testing 未提供本例源持续离线和 survivor-only 的绑定证据。
+Blocked reason (`ENV_H5_RETIRED_POOL_REMOVAL`): Requires the retired pool to be removed after a complete migration, the source to stay offline, and survivor-only access; current auto-testing has not provided bound evidence for this case that the source stays offline and that access is survivor-only.
-1. 依既有 Pool 全量迁移并完成退休配置移除,停止源服务,证明入口仅绑定 survivor;保留源不可达证据。
-2. 只在 survivor 的选定对象/版本制造缺片或metadata缺失;START root或正确pool/set deep。
-3. 记录 source authority 选择,不能向已退休 pool 写入,也不能无权威覆盖/错误回退(#2422)。
-4. 检查survivor实际补片、全部版本/marker和源根未变化;在源持续停机情况下全量 VERIFY。
+1. Following existing Pool behavior, finish a full migration and removal of the retirement configuration, stop the source service, and prove the entry point is bound only to the survivor; retain evidence that the source is unreachable.
+2. Inject missing shards or missing metadata only on selected objects or versions on the survivor; START root, or deep on the correct pool/set.
+3. Record the source authority selection. Do not write to a pool that has already been retired, and do not overwrite without authority or fall back incorrectly (#2422).
+4. Check that the survivor actually reconstructed the on-disk shards, and that every version and marker and the source root are unchanged; with the source kept down, VERIFY in full.
Operations: [SEED_DV](validation-heal-operations.md#SEED_DV) → [POOL](validation-heal-operations.md#POOL) → [PROCESS](validation-heal-operations.md#PROCESS) → [FAULT](validation-heal-operations.md#FAULT) → [START](validation-heal-operations.md#START) → [WAIT](validation-heal-operations.md#WAIT) → [PHYSICAL](validation-heal-operations.md#PHYSICAL) → [VERIFY](validation-heal-operations.md#VERIFY) → [LOGS](validation-heal-operations.md#LOGS)
Source assertions:
-- 源 pool 完全退役移除并停机后,只在 survivor 上触发修复
-- 不再向已移除源写入或依赖其兜底;来源权威和版本正确;#2422 路径不能出现无权威覆盖或错误回退。
+- After the source pool is fully decommissioned, removed, and stopped, trigger repair only on the survivor
+- Do not write to the removed source or rely on it as a fallback; source authority and version are correct; the #2422 path must not show an overwrite without authority or an incorrect rollback.
-先保存失败现场及物理/状态证据,停止本例写入与在途任务,再按 RECOVER 撤销本例故障。并发版本已变化时禁止盲目恢复旧 xl.meta;恢复不能充作产品修复。未恢复环境标 DIRTY,保留本例备份和原失败结果。
+First save the failure scene and the physical and state evidence, stop this case's writes and in-flight tasks, then undo this case's fault per RECOVER. When a concurrent version has already changed, do not blindly restore the old xl.meta; recovery must not serve as a product fix. Mark an unrestored environment DIRTY, and keep this case's backup and the original failure result.
-逐项满足设计公共契约、该分支步骤及附加断言后人工评审。缺资源为 BLOCKED,窗口未命中为 NOT_OBSERVED,产品错误为 FAIL,工具/恢复错误为 INFRA_ERROR;未运行不记 PASS。
+Manually review after each item satisfies the design's common contract, that branch's steps, and the additional assertions. Missing resources are BLOCKED, a window miss is NOT_OBSERVED, a product error is FAIL, and a tool or recovery error is INFRA_ERROR; a case that did not run is not recorded as PASS.
@@ -1641,21 +1641,21 @@ Source assertions:
Topologies: H5. Branches: selected-pool, selected-set, unselected-control-fault.
-1. 在两个以上 pool/set分别放不同对象,用实际layout确认归属;健康control与控制故障分开登记。
-2. 只选一个pool/set START;其他pool保留缺片控制,不先GET它们。
-3. PHYSICAL先确认选中目标补齐,未选故障原样保留;任务、容量、配额与drive身份都指向正确目标。
-4. 取证后才恢复控制故障和VERIFY;对象在其他pool可读不能证明所选盘完成修复。
+1. Place different objects on two or more pool/sets, and confirm ownership from the actual layout; register the healthy control and the control fault separately.
+2. START only one pool/set; leave the missing-shard controls on the other pools, and do not GET them first.
+3. PHYSICAL first confirms that the selected target is fully reconstructed and that unselected faults remain as they were; the task, capacity, quota, and drive identity all point to the correct target.
+4. Restore the control fault and run VERIFY only after the evidence has been collected; an object that is readable on another pool does not prove that the selected disk finished repair.
Operations: [SEED_DB](validation-heal-operations.md#SEED_DB) → [SEED_DV](validation-heal-operations.md#SEED_DV) → [FORMAT](validation-heal-operations.md#FORMAT) → [FAULT](validation-heal-operations.md#FAULT) → [START](validation-heal-operations.md#START) → [WAIT](validation-heal-operations.md#WAIT) → [PHYSICAL](validation-heal-operations.md#PHYSICAL) → [RECOVER](validation-heal-operations.md#RECOVER) → [VERIFY](validation-heal-operations.md#VERIFY)
Source assertions:
-- 仅破坏选定 pool/set,并在另一 pool 保留不同对象与控制故障
-- 选择器、任务、容量/配额和目标 drive 一致;未选 pool 不受错误修复,跨 pool 存在不等于该目标已修复。
+- Corrupt only the selected pool/set, and in another pool keep different objects and a control fault
+- The selector, the task, capacity/quota, and the target drive agree; an unselected pool is not affected by an incorrect repair, and presence across pools does not mean that target has been repaired.
-先保存失败现场及物理/状态证据,停止本例写入与在途任务,再按 RECOVER 撤销本例故障。并发版本已变化时禁止盲目恢复旧 xl.meta;恢复不能充作产品修复。未恢复环境标 DIRTY,保留本例备份和原失败结果。
+First save the failure scene and the physical and state evidence, stop this case's writes and in-flight tasks, then undo this case's fault per RECOVER. When a concurrent version has already changed, do not blindly restore the old xl.meta; recovery must not serve as a product fix. Mark an unrestored environment DIRTY, and keep this case's backup and the original failure result.
-逐项满足设计公共契约、该分支步骤及附加断言后人工评审。缺资源为 BLOCKED,窗口未命中为 NOT_OBSERVED,产品错误为 FAIL,工具/恢复错误为 INFRA_ERROR;未运行不记 PASS。
+Manually review after each item satisfies the design's common contract, that branch's steps, and the additional assertions. Missing resources are BLOCKED, a window miss is NOT_OBSERVED, a product error is FAIL, and a tool or recovery error is INFRA_ERROR; a case that did not run is not recorded as PASS.
@@ -1665,23 +1665,23 @@ Topologies: H3,H4. Branches: ec84-one-set-restart, ec84-one-set-crash, ec84-two-
Execution policy: `environment-gated`; disabled by default: `true`.
-Blocked reason (`ENV_EC84_MULTI_SET_TOPOLOGY`): 需要 H3 的 12 个独立盘和 H4 的至少 24 个独立 endpoint,并按 set 分别记录重启/崩溃窗口;当前 H4 资源不可用。
+Blocked reason (`ENV_EC84_MULTI_SET_TOPOLOGY`): Requires 12 independent disks for H3 and at least 24 independent endpoints for H4, with the restart and crash windows recorded per set; H4 resources are currently unavailable.
-1. 分别绑定 H3 12盘和 H4 至少24盘,FORMAT必须证实每set为8data+4parity;同一物理设备多个目录不能算独立故障盘。
-2. 每个set都准备DB/DV/DX并证明对象归属,REPLACE选定目标,在目标仅部分重建时优雅重启或SIGKILL。
-3. 保存实际中断窗口、盘代次、每set已修/未修对象,不能跨set合并片数。
-4. 每个set单独完成物理与全版本校验;四盘pilot不进入该项分子,native EC8+4回归作为独立补证。
+1. Bind 12 disks for H3 and at least 24 disks for H4. FORMAT must prove that every set is 8data+4parity; multiple directories on the same physical device do not count as independent fault disks.
+2. Prepare DB/DV/DX on every set and prove object ownership, REPLACE the selected target, and gracefully restart or send SIGKILL while that target is only partly rebuilt.
+3. Save the actual interruption window, the disk generation, and the repaired and unrepaired objects of each set; do not merge shard counts across sets.
+4. Each set completes its own physical check and full-version check; the four-disk pilot does not enter this item's numerator, and the native EC8+4 regression is independent supplementary evidence.
Operations: [SEED_DX](validation-heal-operations.md#SEED_DX) → [SEED_DV](validation-heal-operations.md#SEED_DV) → [FORMAT](validation-heal-operations.md#FORMAT) → [REPLACE](validation-heal-operations.md#REPLACE) → [WINDOW](validation-heal-operations.md#WINDOW) → [PROCESS](validation-heal-operations.md#PROCESS) → [PHYSICAL](validation-heal-operations.md#PHYSICAL) → [VERIFY](validation-heal-operations.md#VERIFY) → [NATIVE](validation-heal-operations.md#NATIVE)
Source assertions:
-- 固定 EC8+4 单 set 与至少两 set,部分重建时重启/崩溃目标
-- 每个 set 的故障覆盖和重建分别证明,不跨 set 拼 shard;必须满足真实盘数及映射,不能用 4 盘用例替代 EC8+4。
+- Pin EC8+4 for a single set and for at least two sets, and restart or crash the target during partial reconstruction
+- Prove fault coverage and reconstruction separately for each set, and do not assemble shards across sets; the real disk count and mapping must be met, and a 4-disk case cannot substitute for EC8+4.
-先保存失败现场及物理/状态证据,停止本例写入与在途任务,再按 RECOVER 撤销本例故障。并发版本已变化时禁止盲目恢复旧 xl.meta;恢复不能充作产品修复。未恢复环境标 DIRTY,保留本例备份和原失败结果。
+First save the failure scene and the physical and state evidence, stop this case's writes and in-flight tasks, then undo this case's fault per RECOVER. When a concurrent version has already changed, do not blindly restore the old xl.meta; recovery must not serve as a product fix. Mark an unrestored environment DIRTY, and keep this case's backup and the original failure result.
-逐项满足设计公共契约、该分支步骤及附加断言后人工评审。缺资源为 BLOCKED,窗口未命中为 NOT_OBSERVED,产品错误为 FAIL,工具/恢复错误为 INFRA_ERROR;未运行不记 PASS。
+Manually review after each item satisfies the design's common contract, that branch's steps, and the additional assertions. Missing resources are BLOCKED, a window miss is NOT_OBSERVED, a product error is FAIL, and a tool or recovery error is INFRA_ERROR; a case that did not run is not recorded as PASS.
@@ -1691,23 +1691,23 @@ Topologies: H1,H2. Branches: bounded-latency, short-read, eio.
Execution policy: `environment-gated`; disabled by default: `true`.
-Blocked reason (`ENV_DEDICATED_FAULT_IO`): 需要专用 faultfs 或故障设备、可恢复的 slow/short-read/EIO 注入和 namespace 隔离;共享盘模拟会污染其他运行。
+Blocked reason (`ENV_DEDICATED_FAULT_IO`): Requires a dedicated faultfs or fault device, recoverable slow/short-read/EIO injection, and namespace isolation; simulating on a shared disk would contaminate other runs.
-1. IO_FAULT 只在预分配专用故障设备或受限faultfs,冻结延迟/错误率、IO范围、最长窗口和原状恢复命令。
-2. 在服务相同namespace/UID先证明迟延/短读/EIO真实生效,再启动目标缺片heal;不得让dd填满共享盘。
-3. 错误窗口记录重试、CPU、队列和物理片,不能提前reported healthy;暂态与永久错误分开记录。
-4. 撤销故障后重新观察责任完成并VERIFY;故障层证据缺失算INFRA_ERROR,不把模拟EIO说成真实介质损坏。
+1. Apply IO_FAULT only on a preallocated dedicated fault device or a restricted faultfs, and freeze the latency and error rate, the IO range, the maximum window, and the command that restores the original state.
+2. Under the same namespace and UID as the service, first prove that the delay, short read, and EIO really take effect, then start the target missing-shard heal; do not let dd fill the shared disk.
+3. In the error window, record retries, CPU, the queue, and the physical shards, and do not let the result be reported healthy early. Record transient errors and permanent errors separately.
+4. After the fault is removed, observe completion of responsibility again and VERIFY; missing fault-layer evidence counts as INFRA_ERROR, and a simulated EIO must not be described as real media damage.
Operations: [SEED_DB](validation-heal-operations.md#SEED_DB) → [SEED_DX](validation-heal-operations.md#SEED_DX) → [IO_FAULT](validation-heal-operations.md#IO_FAULT) → [FAULT](validation-heal-operations.md#FAULT) → [START](validation-heal-operations.md#START) → [BACKGROUND](validation-heal-operations.md#BACKGROUND) → [METRICS](validation-heal-operations.md#METRICS) → [RECOVER](validation-heal-operations.md#RECOVER) → [PHYSICAL](validation-heal-operations.md#PHYSICAL) → [VERIFY](validation-heal-operations.md#VERIFY)
Source assertions:
-- 在独占故障层注入有界延迟、短读、EIO,解除后观察恢复
-- 暂态与永久错误分开,重试有界且无忙循环;解除故障后实际修复,期间不得提前标健康。
+- At the exclusive fault layer, inject bounded delay, short reads, and EIO, then observe recovery after they are cleared
+- Keep transient and permanent errors separate; retries are bounded and there is no busy loop; after the fault is cleared, repair actually occurs, and health must not be marked early while the fault is in effect.
-先保存失败现场及物理/状态证据,停止本例写入与在途任务,再按 RECOVER 撤销本例故障。并发版本已变化时禁止盲目恢复旧 xl.meta;恢复不能充作产品修复。未恢复环境标 DIRTY,保留本例备份和原失败结果。
+First save the failure scene and the physical and state evidence, stop this case's writes and in-flight tasks, then undo this case's fault per RECOVER. When a concurrent version has already changed, do not blindly restore the old xl.meta; recovery must not serve as a product fix. Mark an unrestored environment DIRTY, and keep this case's backup and the original failure result.
-逐项满足设计公共契约、该分支步骤及附加断言后人工评审。缺资源为 BLOCKED,窗口未命中为 NOT_OBSERVED,产品错误为 FAIL,工具/恢复错误为 INFRA_ERROR;未运行不记 PASS。
+Manually review after each item satisfies the design's common contract, that branch's steps, and the additional assertions. Missing resources are BLOCKED, a window miss is NOT_OBSERVED, a product error is FAIL, and a tool or recovery error is INFRA_ERROR; a case that did not run is not recorded as PASS.
@@ -1715,21 +1715,21 @@ Source assertions:
Topologies: H1,H2. Branches: bounded-volume-enospc, bounded-inode-exhaustion, bucket-quota.
-1. IO_FAULT使用有容量上限的独立loop/fault卷,预注册字节/inode限制及本例桶配额,不改变共享XFS全局配额。
-2. 先证明服务实际写返回ENOSPC或quota错误;准备缺片目标和DM写入,再触发heal/PUT,逐次保留ACK和错误。
-3. 不可写期间不得损坏健康来源、虚报补片或释放仍需容量;元数据/part双重快照。
-4. 释放本例占用或提高已批准限额后续修,VERIFY全内容和目标片;共享盘可用空间保持安全余量。
+1. IO_FAULT uses an independent loop/fault volume with a capacity limit, preregisters the byte and inode limits and this case's bucket quota, and does not change the shared XFS global quota.
+2. First prove that a real service write returns ENOSPC or a quota error; prepare a missing-shard target and a DM write, then trigger heal/PUT, and keep the ACK and the error of every attempt.
+3. During the unwritable period, do not damage healthy sources, falsely report shard reconstruction, or release capacity that is still required; snapshot both metadata and parts.
+4. Release this case's usage or raise the approved limit, then continue the repair, and VERIFY the full content and the target shards; keep a safe margin of free space on the shared disk.
Operations: [SEED_DB](validation-heal-operations.md#SEED_DB) → [SEED_DM](validation-heal-operations.md#SEED_DM) → [IO_FAULT](validation-heal-operations.md#IO_FAULT) → [QUOTA](validation-heal-operations.md#QUOTA) → [FAULT](validation-heal-operations.md#FAULT) → [START](validation-heal-operations.md#START) → [WORKLOAD](validation-heal-operations.md#WORKLOAD) → [PHYSICAL](validation-heal-operations.md#PHYSICAL) → [RECOVER](validation-heal-operations.md#RECOVER) → [VERIFY](validation-heal-operations.md#VERIFY)
Source assertions:
-- 限制专用测试卷空间/inode,并参数化本次桶配额
-- 仅填专用受限卷、不耗尽共享真实盘;失败不破坏好副本或释放尚需容量,释放空间后正确续修。
+- Limit space and inodes on the dedicated test volume, and parameterize this run's bucket quota
+- Fill only the dedicated constrained volume, and do not exhaust the shared real disk; on failure, do not damage good replicas or release capacity that is still needed, and continue the repair correctly after space is freed.
-先保存失败现场及物理/状态证据,停止本例写入与在途任务,再按 RECOVER 撤销本例故障。并发版本已变化时禁止盲目恢复旧 xl.meta;恢复不能充作产品修复。未恢复环境标 DIRTY,保留本例备份和原失败结果。
+First save the failure scene and the physical and state evidence, stop this case's writes and in-flight tasks, then undo this case's fault per RECOVER. When a concurrent version has already changed, do not blindly restore the old xl.meta; recovery must not serve as a product fix. Mark an unrestored environment DIRTY, and keep this case's backup and the original failure result.
-逐项满足设计公共契约、该分支步骤及附加断言后人工评审。缺资源为 BLOCKED,窗口未命中为 NOT_OBSERVED,产品错误为 FAIL,工具/恢复错误为 INFRA_ERROR;未运行不记 PASS。
+Manually review after each item satisfies the design's common contract, that branch's steps, and the additional assertions. Missing resources are BLOCKED, a window miss is NOT_OBSERVED, a product error is FAIL, and a tool or recovery error is INFRA_ERROR; a case that did not run is not recorded as PASS.
@@ -1737,25 +1737,25 @@ Source assertions:
Topologies: H2. Branches: minority-partition, read-boundary, write-boundary, lock-boundary, rejoin.
-候选支持与 B06 相同,但 storageclass 与锁策略分别按精确候选绑定。`1c7fc81` 的锁客户端范围、quorum 阈值及 EC:8 配置已复核;其 RPC 控制通道工厂有变化,必须重新取得真实双向网络探针、admission、partial coverage 和 rejoin 证据,不能由源码等价或 B07 通过推定本例通过。B08 的候选范围保持独立。
+Candidate support matches B06, but the storage class and the lock policy are bound separately to the exact candidate. The lock-client scope, quorum thresholds, and EC:8 configuration of `1c7fc81` have been reviewed. Its RPC control-channel factory has changed, so real bidirectional network probes, admission, partial coverage, and rejoin evidence must be collected again. Source equivalence, or a passing B07, must not be taken to mean this case passed. The candidate scope for B08 stays independent.
The adapter additionally binds exact source `62ff96774fa4139d45f14d70b70519aae8070ac0` and its typed RPC `Unavailable` response mapping to `503/ServiceUnavailable`. Every referenced lock, EC, multipart and delete-recovery source file remains independently checked against that committed source; neighboring or abbreviated revisions are rejected. This source requires its matching immutable package and source-bound inspector before live execution. All five original branches, complete DB/DV/DL fixtures, refused-mutation accounting, physical-before-GET proofs, task responsibility and cleanup requirements remain mandatory; the source admission alone does not establish live acceptance.
-1. H2 先冻结实际节点/slot/对象R/W和锁quorum,NETWORK规则只匹配本例服务端口并带run ID、超时恢复。
-2. 构造少数派及边界分区,双端探针证实隔离;SSH、跳板机和其他服务端口保持连通。
-3. 双方执行带nonce的PUT/GET/heal,少数派不能承诺不安全提交;BACKGROUND必须暴露missing peer/partial coverage。
-4. 解除规则并检查规则计数归零,按ACK因果对账全版本/内容及目标片,无分叉、旧代覆盖或丢失责任。
+1. On H2, first freeze the actual node/slot/object R/W quorum and the lock quorum. NETWORK rules match only this case's service port and carry the run ID and timeout recovery.
+2. Construct a minority partition and a boundary partition, and confirm the isolation with probes from both ends; SSH, the jump host, and the other service ports stay connected.
+3. Both sides run PUT/GET/heal with a nonce. The minority must not promise an unsafe commit; BACKGROUND must expose missing peer/partial coverage.
+4. Remove the rules and check that the rule count returns to zero. Reconcile every version, the content, and the target shards by ACK causality, with no fork, no old-generation overwrite, and no lost responsibility.
Operations: [SEED_DV](validation-heal-operations.md#SEED_DV) → [FORMAT](validation-heal-operations.md#FORMAT) → [NETWORK](validation-heal-operations.md#NETWORK) → [WORKLOAD](validation-heal-operations.md#WORKLOAD) → [START](validation-heal-operations.md#START) → [BACKGROUND](validation-heal-operations.md#BACKGROUND) → [METRICS](validation-heal-operations.md#METRICS) → [RECOVER](validation-heal-operations.md#RECOVER) → [PHYSICAL](validation-heal-operations.md#PHYSICAL) → [VERIFY](validation-heal-operations.md#VERIFY)
Source assertions:
-- 仅隔离测试服务节点间流量,覆盖少数派、读/写/锁 quorum 边界及恢复
-- 少数派不得承诺无安全性的修复或写入;cluster coverage 不完整不能显示全局健康;恢复无分叉、跨代覆盖。
+- Isolate only traffic among the test service nodes, covering the minority, the read/write/lock quorum boundaries, and recovery
+- A minority must not promise a repair or write that lacks safety; incomplete cluster coverage must not be shown as global health; recovery has no fork and no cross-generation overwrite.
-先保存失败现场及物理/状态证据,停止本例写入与在途任务,再按 RECOVER 撤销本例故障。并发版本已变化时禁止盲目恢复旧 xl.meta;恢复不能充作产品修复。未恢复环境标 DIRTY,保留本例备份和原失败结果。
+First save the failure scene and the physical and state evidence, stop this case's writes and in-flight tasks, then undo this case's fault per RECOVER. When a concurrent version has already changed, do not blindly restore the old xl.meta; recovery must not serve as a product fix. Mark an unrestored environment DIRTY, and keep this case's backup and the original failure result.
-逐项满足设计公共契约、该分支步骤及附加断言后人工评审。缺资源为 BLOCKED,窗口未命中为 NOT_OBSERVED,产品错误为 FAIL,工具/恢复错误为 INFRA_ERROR;未运行不记 PASS。
+Manually review after each item satisfies the design's common contract, that branch's steps, and the additional assertions. Missing resources are BLOCKED, a window miss is NOT_OBSERVED, a product error is FAIL, and a tool or recovery error is INFRA_ERROR; a case that did not run is not recorded as PASS.
@@ -1763,21 +1763,21 @@ Source assertions:
Topologies: H1,H2. Branches: full-queue, dropped-admission, merged-duplicates, cross-bucket-fairness.
-1. 专用服务冻结队列容量、并发、最大请求数和时限;DX与多桶夹具通过WORKLOAD施加有上限的Admin/read-repair请求。
-2. 分别记录Full/Dropped/Merged的真实返回及任务ID,重复token/合并不按独立修复数计分。
-3. 持久责任不能因入队失败被退休;记录journal、队列深度、等待时延和资源曲线,超过预算立即停注入。
-4. 停止负载后在deadline内收敛,所有可修目标真实补齐;不能只以CLI exit0或全局idle通过。
+1. The dedicated service freezes queue capacity, concurrency, the maximum request count, and the time limit; the DX fixture and the multi-bucket fixtures apply a bounded number of Admin/read-repair requests through WORKLOAD.
+2. Record the actual return and the task ID of Full, Dropped, and Merged separately; a repeated token or a merge is not scored as an independent repair.
+3. Persistent responsibility must not be retired because enqueue failed. Record the journal, queue depth, wait latency, and resource curves, and stop injection as soon as the budget is exceeded.
+4. After the load stops, converge within the deadline, and actually reconstruct every repairable target; the case must not pass on CLI exit0 or global idle alone.
Operations: [SEED_DX](validation-heal-operations.md#SEED_DX) → [CONFIG](validation-heal-operations.md#CONFIG) → [WORKLOAD](validation-heal-operations.md#WORKLOAD) → [START](validation-heal-operations.md#START) → [BACKGROUND](validation-heal-operations.md#BACKGROUND) → [CHECKPOINT](validation-heal-operations.md#CHECKPOINT) → [METRICS](validation-heal-operations.md#METRICS) → [PHYSICAL](validation-heal-operations.md#PHYSICAL) → [VERIFY](validation-heal-operations.md#VERIFY)
Source assertions:
-- 多来源请求冲击有限队列,混合重复任务、不同优先级和多个桶
-- Full/Dropped/Merged 语义与实际任务一致;丢弃不能丢 durable 责任;在预先声明负载窗口内无无限堆积或健康误报。
+- Requests from multiple sources stress the bounded queue, mixing duplicate tasks, different priorities, and multiple buckets
+- Full/Dropped/Merged semantics match the actual tasks; a drop must not drop durable responsibility; within the predeclared load window there is no unbounded buildup and no false health report.
-先保存失败现场及物理/状态证据,停止本例写入与在途任务,再按 RECOVER 撤销本例故障。并发版本已变化时禁止盲目恢复旧 xl.meta;恢复不能充作产品修复。未恢复环境标 DIRTY,保留本例备份和原失败结果。
+First save the failure scene and the physical and state evidence, stop this case's writes and in-flight tasks, then undo this case's fault per RECOVER. When a concurrent version has already changed, do not blindly restore the old xl.meta; recovery must not serve as a product fix. Mark an unrestored environment DIRTY, and keep this case's backup and the original failure result.
-逐项满足设计公共契约、该分支步骤及附加断言后人工评审。缺资源为 BLOCKED,窗口未命中为 NOT_OBSERVED,产品错误为 FAIL,工具/恢复错误为 INFRA_ERROR;未运行不记 PASS。
+Manually review after each item satisfies the design's common contract, that branch's steps, and the additional assertions. Missing resources are BLOCKED, a window miss is NOT_OBSERVED, a product error is FAIL, and a tool or recovery error is INFRA_ERROR; a case that did not run is not recorded as PASS.
@@ -1787,23 +1787,23 @@ Topologies: H1,H2. Branches: object-lock-timeout, caller-disconnect, stale-read-
Execution policy: `evidence-blocked`; disabled by default: `false`.
-Blocked reason (`NATIVE_LOCK_OWNER_ABORT_BARRIER`): 已有同候选源码的 lock owner、取消物理 unlink 与 stale read-repair 原生子集;H1/H2 timeout 与 queued-successor 诊断均已通过独立 live 审计(39767336741/1、39817606025/1),包括原始 ACK、历史代次、真实锁事件、各 peer inventory、停机后新分片完整性和有界 FD 收尾。仍缺实际断连边界的精确所有权、完整 native timeout/stale 屏障、stale-generation controller,以及 H1/H2 DL/DV 完整对象/ACK/物理/锁/FD 闭环。精确源码 cfg(test) 原生证据独立保留;诊断子集和原生子集不能充作完整 canonical PASS。
+Blocked reason (`NATIVE_LOCK_OWNER_ABORT_BARRIER`): A native subset from the same candidate source already exists for lock owner, cancelled physical unlink, and stale read-repair. The H1/H2 timeout and queued-successor diagnostics have both passed an independent live audit (39767336741/1, 39817606025/1), including the original ACKs, historical generations, real lock events, each peer inventory, new-shard integrity after shutdown, and bounded FD cleanup. Still missing are precise ownership at the actual disconnect boundary, a complete native timeout/stale barrier, the stale-generation controller, and the complete H1/H2 DL/DV object/ACK/physical/lock/FD closed loop. Precise source cfg(test) native evidence is retained on its own; the diagnostic subset and the native subset cannot stand in for a complete canonical PASS.
-1. LOCK_WINDOW用本例慢IO/受控原生锁屏障持有目标对象锁,先证明真实争用超过预注册预算。
-2. 同时heal/覆盖或删除;调用方断开实例仅断本次TCP连接,不杀服务器或关闭跳板机;记录请求开始/断开/服务在途结束。
-3. 锁释放后观察取消/超时不提前释放所有权、不重复提交,陈旧read-repair必须重验版本。
-4. 核对所有ACK和最终代次,检查锁/FD无泄漏;黑盒不能证明锁所有权时以NATIVE窗口独立补证,不能猜命中。
+1. LOCK_WINDOW holds the target object lock with this case's slow IO or a controlled native lock barrier, and first proves that the real contention exceeds the preregistered budget.
+2. At the same time, heal, overwrite, or delete. The caller-disconnect instance drops only this TCP connection and does not kill the server or shut down the jump host; record request start, disconnect, and the end of in-flight work on the service.
+3. After the lock is released, observe that cancel and timeout do not release ownership early and do not commit a second time, and that a stale read-repair revalidates the version.
+4. Check every ACK and the final generation, and check that locks and FDs do not leak. When a black box cannot prove lock ownership, supply independent evidence with a NATIVE window; do not guess that the window was hit.
Operations: [SEED_DV](validation-heal-operations.md#SEED_DV) → [IO_FAULT](validation-heal-operations.md#IO_FAULT) → [LOCK_WINDOW](validation-heal-operations.md#LOCK_WINDOW) → [WORKLOAD](validation-heal-operations.md#WORKLOAD) → [START](validation-heal-operations.md#START) → [WINDOW](validation-heal-operations.md#WINDOW) → [METRICS](validation-heal-operations.md#METRICS) → [PHYSICAL](validation-heal-operations.md#PHYSICAL) → [VERIFY](validation-heal-operations.md#VERIFY) → [NATIVE](validation-heal-operations.md#NATIVE)
Source assertions:
-- 真实对象锁争用超过预算及调用方断开,再释放锁恢复
-- 取消/超时不得提前释放在途所有权或重复提交;陈旧 read-repair 必须重验代次;恢复后无泄漏和错误覆盖。
+- Real object-lock contention exceeds the budget and the caller disconnects, then release the lock and recover
+- Cancel or timeout must not release in-flight ownership early or commit twice; a stale read-repair must re-verify the generation; after recovery there is no leak and no incorrect overwrite.
-先保存失败现场及物理/状态证据,停止本例写入与在途任务,再按 RECOVER 撤销本例故障。并发版本已变化时禁止盲目恢复旧 xl.meta;恢复不能充作产品修复。未恢复环境标 DIRTY,保留本例备份和原失败结果。
+First save the failure scene and the physical and state evidence, stop this case's writes and in-flight tasks, then undo this case's fault per RECOVER. When a concurrent version has already changed, do not blindly restore the old xl.meta; recovery must not serve as a product fix. Mark an unrestored environment DIRTY, and keep this case's backup and the original failure result.
-逐项满足设计公共契约、该分支步骤及附加断言后人工评审。缺资源为 BLOCKED,窗口未命中为 NOT_OBSERVED,产品错误为 FAIL,工具/恢复错误为 INFRA_ERROR;未运行不记 PASS。
+Manually review after each item satisfies the design's common contract, that branch's steps, and the additional assertions. Missing resources are BLOCKED, a window miss is NOT_OBSERVED, a product error is FAIL, and a tool or recovery error is INFRA_ERROR; a case that did not run is not recorded as PASS.
@@ -1813,23 +1813,23 @@ Topologies: H2. Branches: kms-unavailable, kms-denied, tier-unavailable, tier-ve
Execution policy: `environment-gated`; disabled by default: `true`.
-Blocked reason (`ENV_DEDICATED_KMS_TIER_TENANT`): 需要专用 KMS/tier 租户、依赖网络隔离和安全的 key/tier 版本证据;不能在共享依赖上全局断网。
+Blocked reason (`ENV_DEDICATED_KMS_TIER_TENANT`): Requires a dedicated KMS/tier tenant, network isolation of the dependency, and safe key/tier version evidence; do not cut the network globally on a shared dependency.
-1. 先在专用KMS/tier租户验证DE/DV夹具可正常解密/restore,保存非敏感key/tier版本标识。
-2. DEPENDENCY仅断本测试客户端/服务到该依赖的专用namespace路由,或撤销本例租户权限;不全局停共享KMS。
-3. 发GET/restore/deep heal并观察责任,依赖错误不能判对象不存在、可remove、已修好;密钥不进入错误日志。
-4. 恢复依赖和权限后原版本可解密/恢复,物理/责任完整;远端对象不存在与远端暂不可达分别判定。
+1. On a dedicated KMS/tier tenant, first verify that the DE/DV fixtures decrypt and restore normally, and save the non-sensitive key/tier version identifiers.
+2. DEPENDENCY cuts only this test client's or service's dedicated namespace route to that dependency, or revokes this case's tenant permission; do not stop the shared KMS globally.
+3. Issue GET, restore, and deep heal, and observe responsibility. A dependency error must not be judged as the object not existing, as remove being allowed, or as already repaired; keys must not enter error logs.
+4. After the dependency and the permissions are restored, the original version can be decrypted and restored, and the physical state and responsibility are complete; judge a remote object that does not exist separately from a remote that is temporarily unreachable.
Operations: [SEED_DE](validation-heal-operations.md#SEED_DE) → [SEED_DV](validation-heal-operations.md#SEED_DV) → [DEPENDENCY](validation-heal-operations.md#DEPENDENCY) → [NETWORK](validation-heal-operations.md#NETWORK) → [S3](validation-heal-operations.md#S3) → [START](validation-heal-operations.md#START) → [BACKGROUND](validation-heal-operations.md#BACKGROUND) → [PHYSICAL](validation-heal-operations.md#PHYSICAL) → [VERIFY](validation-heal-operations.md#VERIFY)
Source assertions:
-- 仅让已分配测试 KMS/远端 tier 暂不可用,持续查询与修复
-- 依赖故障不得解释为对象不存在/可删/已修复,不暴露密钥;恢复后原版本可解密或恢复,责任未丢失。
+- Make only the assigned test KMS/remote tier temporarily unavailable, and keep querying and repairing
+- A dependency failure must not be interpreted as the object being absent, deletable, or already repaired, and must not expose the key; after recovery the original version can be decrypted or restored, and responsibility is not lost.
-先保存失败现场及物理/状态证据,停止本例写入与在途任务,再按 RECOVER 撤销本例故障。并发版本已变化时禁止盲目恢复旧 xl.meta;恢复不能充作产品修复。未恢复环境标 DIRTY,保留本例备份和原失败结果。
+First save the failure scene and the physical and state evidence, stop this case's writes and in-flight tasks, then undo this case's fault per RECOVER. When a concurrent version has already changed, do not blindly restore the old xl.meta; recovery must not serve as a product fix. Mark an unrestored environment DIRTY, and keep this case's backup and the original failure result.
-逐项满足设计公共契约、该分支步骤及附加断言后人工评审。缺资源为 BLOCKED,窗口未命中为 NOT_OBSERVED,产品错误为 FAIL,工具/恢复错误为 INFRA_ERROR;未运行不记 PASS。
+Manually review after each item satisfies the design's common contract, that branch's steps, and the additional assertions. Missing resources are BLOCKED, a window miss is NOT_OBSERVED, a product error is FAIL, and a tool or recovery error is INFRA_ERROR; a case that did not run is not recorded as PASS.
@@ -1837,21 +1837,21 @@ Source assertions:
Topologies: H1,H2. Branches: enabled-idle, disabled, uninitialized, missing-peer, recovered-complete.
-1. 在独立服务配置分别观察 enabled无任务、HEAL禁用、初始化未完成,保留实际启动配置及时间线。
-2. H2 PROCESS停一个peer并跨节点查询BACKGROUND/REPLACEMENT;完整coverage必须含expected/responded/unknown明细。
-3. disabled/uninitialized/partial/degraded不能被rc或聚合器解释成全局完成;收集rc原输出与raw Admin逐字段对照。
-4. 恢复所有peer后才允许完整idle,依然用PHYSICAL/manifest核对无遗留责任;未观测初始化窗口单独记NOT_OBSERVED。
+1. On an independent service configuration, observe separately enabled with no task, HEAL disabled, and initialization not finished, and keep the actual startup configuration and timeline.
+2. On H2, PROCESS stops one peer and queries BACKGROUND/REPLACEMENT across nodes; complete coverage must include the expected/responded/unknown detail.
+3. disabled/uninitialized/partial/degraded must not be interpreted by rc or by an aggregator as global completion; collect the original rc output and compare it field by field with raw Admin.
+4. A complete idle is allowed only after every peer is restored, and PHYSICAL/manifest is still used to check that no responsibility remains; an unobserved initialization window is recorded on its own as NOT_OBSERVED.
Operations: [SEED_DB](validation-heal-operations.md#SEED_DB) → [CONFIG](validation-heal-operations.md#CONFIG) → [PROCESS](validation-heal-operations.md#PROCESS) → [BACKGROUND](validation-heal-operations.md#BACKGROUND) → [REPLACEMENT](validation-heal-operations.md#REPLACEMENT) → [RC_STATUS](validation-heal-operations.md#RC_STATUS) → [PHYSICAL](validation-heal-operations.md#PHYSICAL) → [VERIFY](validation-heal-operations.md#VERIFY)
Source assertions:
-- 分别检查 enabled/disabled/uninitialized、全体可达及缺失 peer 的状态
-- idle 只在覆盖完整且真实无活动时成立;partial/unknown/degraded/disabled 不被客户端转换为全部完成。
+- Separately check the status for enabled/disabled/uninitialized, for all peers reachable, and for a missing peer
+- idle holds only when coverage is complete and there is truly no activity; partial/unknown/degraded/disabled are not converted by the client into all-complete.
-先保存失败现场及物理/状态证据,停止本例写入与在途任务,再按 RECOVER 撤销本例故障。并发版本已变化时禁止盲目恢复旧 xl.meta;恢复不能充作产品修复。未恢复环境标 DIRTY,保留本例备份和原失败结果。
+First save the failure scene and the physical and state evidence, stop this case's writes and in-flight tasks, then undo this case's fault per RECOVER. When a concurrent version has already changed, do not blindly restore the old xl.meta; recovery must not serve as a product fix. Mark an unrestored environment DIRTY, and keep this case's backup and the original failure result.
-逐项满足设计公共契约、该分支步骤及附加断言后人工评审。缺资源为 BLOCKED,窗口未命中为 NOT_OBSERVED,产品错误为 FAIL,工具/恢复错误为 INFRA_ERROR;未运行不记 PASS。
+Manually review after each item satisfies the design's common contract, that branch's steps, and the additional assertions. Missing resources are BLOCKED, a window miss is NOT_OBSERVED, a product error is FAIL, and a tool or recovery error is INFRA_ERROR; a case that did not run is not recorded as PASS.
@@ -1861,13 +1861,13 @@ Topologies: H1,H2. Branches: mixed-healthy-repaired-failed, dry-run, deferred, r
Execution policy: `evidence-blocked`; disabled by default: `false`.
-Blocked reason (`EVIDENCE_MIXED_OUTCOME_ORCHESTRATION`): 已有 mixed healthy/repaired/failed、独立 dry-run、依赖 deferral、真实 SDK legacy Unknown、原始全副本损坏元数据 Unknown、只读永久失败/新 token 补偿及版本级 grace Deferred/新 token 清理执行流程,均按逐对象 manifest/attemptFailures 对账;仍需完整串行 live 证据及独立验收,故保持禁止默认 dispatch 和 canonical admission;本地检查不能作为完整 PASS。
+Blocked reason (`EVIDENCE_MIXED_OUTCOME_ORCHESTRATION`): Execution flows already exist for mixed healthy/repaired/failed, an independent dry-run, dependency deferral, a real SDK legacy Unknown, an original all-replica corrupt-metadata Unknown, read-only permanent failure with new-token compensation, and version-level grace Deferred with new-token cleanup, all reconciled per object against the manifest and attemptFailures. Complete serial live evidence and independent acceptance are still required, so default dispatch and canonical admission stay prohibited; a local check cannot serve as a complete PASS.
-1. 同批独立key建立健康、可恢复缺片、低于R的不可恢复对象;deferred由冻结的锁/依赖故障产生,dry-run另建任务。
-2. 保存逐对象物理基线和期望disposition,对结果的execution/coverage/counters/object identities做逐项对账。
-3. unknown/skipped/completed_with_errors不算修复成功,legacy objectsHealed与canonical healed分开解释;attemptFailures不得被隐藏。
-4. 使用独立完整manifest弥补有界outcome窗口,保持不可修对象的正确失败记录;恢复后另任务修复不覆盖原结果。
-5. grace 分支先用 SDK PUT 建立两个真实版本并验证完整分片;保存一个成员的原对象树后,SDK DELETE 精确历史版本,逐盘验证仅当前版本存在且其逻辑元数据、数据片及控制对象未变。只向原成员回放原对象树;实际候选进程固定 grace=120 秒,不修改元数据年龄或时钟。原任务须在窗口内完成,精确报告旧版 deferred、当前版 verified_healthy 及 retryNotBefore;自然等待过期后原 token 仍不变,新 token 才能清理旧版。清理后先核对全部物理字节等于 DELETE 后权威现场,再跨节点 GET/404 和复查旧 token,最后才做 fixture finalization。错过窗口保留 NOT_OBSERVED;异常保留现场并标 DIRTY;救援仅恢复 DELETE 后权威树,不复活已删除版本。
+1. In the same batch of independent keys, create healthy objects, objects with recoverable missing shards, and unrecoverable objects below R; deferred is produced by a frozen lock or dependency fault, and dry-run is a separate task.
+2. Save the per-object physical baseline and the expected disposition, and reconcile the result's execution/coverage/counters/object identities item by item.
+3. unknown/skipped/completed_with_errors do not count as a successful repair. Explain legacy objectsHealed separately from canonical healed; attemptFailures must not be hidden.
+4. Use an independent complete manifest to make up for the bounded outcome window, and keep the correct failure record of each unrepairable object; a repair by another task after recovery must not overwrite the original result.
+5. The grace branch first uses SDK PUT to create two real versions and verifies that the shards are complete. After saving one member's original object tree, SDK DELETE the exact historical version, and verify disk by disk that only the current version exists and that its logical metadata, data shards, and control objects are unchanged. Replay the original object tree only onto the original member. The actual candidate process fixes grace=120 seconds and does not change metadata age or the clock. The original task must finish inside the window and report exactly the old version as deferred, the current version as verified_healthy, and retryNotBefore. After naturally waiting out the expiry, the original token is still unchanged, and only a new token may clean up the old version. After cleanup, first check that every physical byte equals the authoritative scene after DELETE, then run cross-node GET/404 and recheck the old token, and only then perform fixture finalization. If the window is missed, retain NOT_OBSERVED. On an exception, preserve the scene and mark DIRTY. Rescue restores only the authoritative tree from after DELETE and does not resurrect the deleted version.
6. Manual diagnostic entry: explicitly select heal.outcome-counters-and-error-truth through Functional Chain with the frozen candidate and candidate-bound manager/inspector tools. The default catalog policy stays evidence-blocked; even all passing diagnostic branches retain complete=false, canonicalCasePassClaimed=false and a blocking gate verdict until independent acceptance.
7. For a pre-replay public-manifest refusal, recover only after the complete SDK request/ACK ledger proves all writes acknowledged and no DELETE or replay occurred. Bind all four original physical members and the reserved owner, stop the exact candidate, then preserve all roots and state using durable same-filesystem non-overwriting rename. Partial archive retries require the same durable inventory; missing roots or drift retain DIRTY. Isolation is not product repair or restoration and never rewrites the failed case.
8. Manual J02 candidate admission is limited to the original 072b57b37559fbe626aeda8abf61617fdcbdd926 and corrected 1c7fc818dd858247c45b39542abd8291b1e6e31b sources. Each execution requires the matching immutable package and independently audited inspector/manager bundle. A corrected-candidate run is new evidence; it cannot replace an original failure. The full exact-version, physical, timing, retry and responsibility oracles remain unchanged, and all passing diagnostics still require independent canonical acceptance.
@@ -1877,12 +1877,12 @@ Operations: [SEED_DB](validation-heal-operations.md#SEED_DB) → [SEED_DV](valid
Source assertions:
-- 同批次混合健康、可修、不可修、deferred、dry-run 对象
-- execution/coverage/disposition/计数和逐对象物理结果相符;unknown、skipped 或 completed_with_errors 不能算修复成功。
+- Mix healthy, repairable, unrepairable, deferred, and dry-run objects in the same batch
+- execution/coverage/disposition, the counts, and the per-object physical results agree; unknown, skipped, or completed_with_errors cannot count as a successful repair.
-先保存失败现场及物理/状态证据,停止本例写入与在途任务,再按 RECOVER 撤销本例故障。并发版本已变化时禁止盲目恢复旧 xl.meta;恢复不能充作产品修复。未恢复环境标 DIRTY,保留本例备份和原失败结果。
+First save the failure scene and the physical and state evidence, stop this case's writes and in-flight tasks, then undo this case's fault per RECOVER. When a concurrent version has already changed, do not blindly restore the old xl.meta; recovery must not serve as a product fix. Mark an unrestored environment DIRTY, and keep this case's backup and the original failure result.
-逐项满足设计公共契约、该分支步骤及附加断言后人工评审。缺资源为 BLOCKED,窗口未命中为 NOT_OBSERVED,产品错误为 FAIL,工具/恢复错误为 INFRA_ERROR;未运行不记 PASS。
+Manually review after each item satisfies the design's common contract, that branch's steps, and the additional assertions. Missing resources are BLOCKED, a window miss is NOT_OBSERVED, a product error is FAIL, and a tool or recovery error is INFRA_ERROR; a case that did not run is not recorded as PASS.
@@ -1890,21 +1890,21 @@ Source assertions:
Topologies: H1,H2. Branches: within-window-cursor, past-retention-gap, cross-node-cursor, terminal-truncation.
-1. DX至少257个目标,超过128诊断窗口;边启动recursive heal边以sinceSeq查询,每次请求/nextSeq/minSeq/items全部保存。
-2. 单端连续读取应无重复倒退;故意使用过旧cursor应明示truncated/gap;跨节点查询按任务/epoch而非全局数字比较。
-3. 任务终态也读取旧cursor,验证有界列表不被当完整清单;所有错误与缺口原样进入报告。
-4. 最终用全量独立S3/physical manifest与canonical counters验收覆盖;若任务太快仍可做终态截断,但不能声称观察了运行中游标。
+1. DX has at least 257 targets, past the 128 diagnostic window. While starting recursive heal, query with sinceSeq, and save every request, nextSeq, minSeq, and items.
+2. Continuous reads from a single end must not repeat or move backward. Deliberately using an overly old cursor must explicitly indicate truncated/gap. Compare cross-node queries by task/epoch rather than by a global number.
+3. At the task's terminal state, also read the old cursor, and verify that a bounded list is not treated as a complete manifest; every error and gap enters the report unchanged.
+4. Finally accept coverage with a full independent S3/physical manifest and the canonical counters. If the task is too fast, truncation at the terminal state is still allowed, but it must not be claimed as an observation of a running cursor.
Operations: [SEED_DX](validation-heal-operations.md#SEED_DX) → [FAULT](validation-heal-operations.md#FAULT) → [START](validation-heal-operations.md#START) → [CURSOR](validation-heal-operations.md#CURSOR) → [STATUS](validation-heal-operations.md#STATUS) → [PHYSICAL](validation-heal-operations.md#PHYSICAL) → [VERIFY](validation-heal-operations.md#VERIFY)
Source assertions:
-- 生成超过结果保留上限的任务,分页轮询 sinceSeq,并跨节点/终态读取
-- nextSeq/minSeq 单调且 gap/truncation 明示;有界 items 不是完整责任清单,最终须用独立 manifest/计数补证,不静默丢错误。
+- Generate tasks past the result-retention limit, poll sinceSeq with pagination, and read across nodes and terminal states
+- nextSeq/minSeq are monotonic and gap/truncation is explicit; bounded items are not the complete responsibility manifest, the final result must be corroborated by an independent manifest and counts, and errors are not dropped silently.
-先保存失败现场及物理/状态证据,停止本例写入与在途任务,再按 RECOVER 撤销本例故障。并发版本已变化时禁止盲目恢复旧 xl.meta;恢复不能充作产品修复。未恢复环境标 DIRTY,保留本例备份和原失败结果。
+First save the failure scene and the physical and state evidence, stop this case's writes and in-flight tasks, then undo this case's fault per RECOVER. When a concurrent version has already changed, do not blindly restore the old xl.meta; recovery must not serve as a product fix. Mark an unrestored environment DIRTY, and keep this case's backup and the original failure result.
-逐项满足设计公共契约、该分支步骤及附加断言后人工评审。缺资源为 BLOCKED,窗口未命中为 NOT_OBSERVED,产品错误为 FAIL,工具/恢复错误为 INFRA_ERROR;未运行不记 PASS。
+Manually review after each item satisfies the design's common contract, that branch's steps, and the additional assertions. Missing resources are BLOCKED, a window miss is NOT_OBSERVED, a product error is FAIL, and a tool or recovery error is INFRA_ERROR; a case that did not run is not recorded as PASS.
@@ -1912,21 +1912,21 @@ Source assertions:
Topologies: H1,H2. Branches: normal-repeat, deep-repeat, repeat-after-restart, versioned-repeat.
-1. DB/DV缺片先修复;versioned 的整份 xl.meta 缺失允许所有 identity repaired,或至少一个 repaired 加其余正健康态;marker 只能为 repaired/metadata_healthy,counters.healed 与 repaired 数量一致、counters.unchanged 与其余数量一致,unknown/skipped/failed=0。完整保存 post-heal S3/physical 作为第二轮基线,不使用最初故障前 xl.meta 要求内部字节完全相同。
-2. 再次 normal/deep:normal 只接受 metadata_healthy,deep 才要求 verified_healthy;两者均 healed=0、unknown/skipped/failed=0,不得新增 repaired,所有原版本/marker 身份与数量一致。
-3. PROCESS原unit重启后再重复;检查新任务有独立token但不产生新对象版本、无限责任或重复修复计数。
-4. 允许经格式证据证明的内部维护记录变化,用户内容/引用不变,目标片摘要稳定。
+1. Repair missing DB/DV shards first. When an entire versioned xl.meta is missing, every identity may be repaired, or at least one repaired with the rest in a positive healthy state. A marker may only be repaired/metadata_healthy. counters.healed matches the repaired count, counters.unchanged matches the count of the rest, and unknown/skipped/failed=0. Save the complete post-heal S3/physical result as the second-round baseline, and do not use the original pre-fault xl.meta to require identical internal bytes.
+2. Run normal/deep again: normal accepts only metadata_healthy, and only deep requires verified_healthy. Both have healed=0 and unknown/skipped/failed=0, must not add any repaired, and every original version and marker keeps the same identity and count.
+3. After PROCESS restarts the original unit, repeat. Check that the new task has an independent token and does not create a new object version, unbounded responsibility, or a duplicate repair count.
+4. Changes to internal maintenance records are allowed when format evidence proves them. User content and references stay unchanged, and the target-shard digests stay stable.
Operations: [SEED_DB](validation-heal-operations.md#SEED_DB) → [SEED_DV](validation-heal-operations.md#SEED_DV) → [FAULT](validation-heal-operations.md#FAULT) → [START](validation-heal-operations.md#START) → [WAIT](validation-heal-operations.md#WAIT) → [PHYSICAL](validation-heal-operations.md#PHYSICAL) → [VERIFY](validation-heal-operations.md#VERIFY) → [PROCESS](validation-heal-operations.md#PROCESS)
Source assertions:
-- 第一次修复后重复 normal/deep、重启再检查
-- versioned 首次允许所有 identity repaired 或至少一个 repaired 加其余正健康态,marker 仅 repaired/metadata_healthy,计数必须与 receipt 集合一致;normal repeat 接受 `metadata_healthy`,deep repeat 要求 `verified_healthy`,repeat marker 为 `metadata_healthy` 且 healed=0;unknown/skipped/failed 不得成为 PASS。
+- After the first repair, repeat normal/deep, restart, and check again
+- For the first versioned run, allow all identities repaired, or at least one repaired plus the rest in a positive healthy state; a marker is only repaired/metadata_healthy, and the counts must match the receipt set; a normal repeat accepts `metadata_healthy`, a deep repeat requires `verified_healthy`, a repeat marker is `metadata_healthy` with healed=0; unknown/skipped/failed must not become PASS.
-先保存失败现场及物理/状态证据,停止本例写入与在途任务,再按 RECOVER 撤销本例故障。并发版本已变化时禁止盲目恢复旧 xl.meta;恢复不能充作产品修复。未恢复环境标 DIRTY,保留本例备份和原失败结果。
+First save the failure scene and the physical and state evidence, stop this case's writes and in-flight tasks, then undo this case's fault per RECOVER. When a concurrent version has already changed, do not blindly restore the old xl.meta; recovery must not serve as a product fix. Mark an unrestored environment DIRTY, and keep this case's backup and the original failure result.
-逐项满足设计公共契约、该分支步骤及附加断言后人工评审。缺资源为 BLOCKED,窗口未命中为 NOT_OBSERVED,产品错误为 FAIL,工具/恢复错误为 INFRA_ERROR;未运行不记 PASS。
+Manually review after each item satisfies the design's common contract, that branch's steps, and the additional assertions. Missing resources are BLOCKED, a window miss is NOT_OBSERVED, a product error is FAIL, and a tool or recovery error is INFRA_ERROR; a case that did not run is not recorded as PASS.
@@ -1936,23 +1936,23 @@ Topologies: H2,H5. Branches: 24h, 72h.
Execution policy: `environment-gated`; disabled by default: `true`.
-Blocked reason (`ENV_DEDICATED_LONG_SOAK_LANE`): 必须在独占 H2/H5 lane 真实运行 24 小时和 72 小时,包含限速、监控、到期回收和完整资源预算;短跑不能替代 soak 证据。
+Blocked reason (`ENV_DEDICATED_LONG_SOAK_LANE`): Must actually run for 24 hours and for 72 hours on an exclusive H2/H5 lane, including rate limiting, monitoring, expiry reclamation, and a complete resource budget. A short run cannot substitute for soak evidence.
-1. 只在专门分配的H2/H5独占lane,冻结完整24h或72h墙钟窗口、循环故障日程、并发/限速/存储上限和到期回收安排。
-2. WORKLOAD持续记录逐请求ACK,周期性故障仅使用本run DRIVE_GATE/PROCESS/依赖规则;每轮见证与恢复,不无限加数据。
-3. 持续METRICS、日志、队列/MRF/checkpoint取样;资源增长、panic、未解释责任达到预注册阈值立即保留失败证据。
-4. 到期停止写入/注入,解除全部故障,解析未知ACK后全量VERIFY和PHYSICAL;短跑不能充24h/72h结果。
+1. Only on a specially assigned exclusive H2/H5 lane, freeze the full 24h or 72h wall-clock window, the cyclic fault schedule, the concurrency, rate-limit, and storage caps, and the expiry-reclamation plan.
+2. WORKLOAD continuously records the ACK of every request. Periodic faults use only this run's DRIVE_GATE/PROCESS/dependency rules; witness and recover every round, and do not add data without a bound.
+3. Continuously sample METRICS, logs, and queue/MRF/checkpoint. As soon as resource growth, a panic, or unexplained responsibility reaches a preregistered threshold, retain the failure evidence.
+4. When the window expires, stop writes and injection, clear every fault, and after unknown ACKs are resolved run a full VERIFY and PHYSICAL. A short run cannot stand in for a 24h or 72h result.
Operations: [SEED_DX](validation-heal-operations.md#SEED_DX) → [SEED_DV](validation-heal-operations.md#SEED_DV) → [WORKLOAD](validation-heal-operations.md#WORKLOAD) → [DRIVE_GATE](validation-heal-operations.md#DRIVE_GATE) → [PROCESS](validation-heal-operations.md#PROCESS) → [BACKGROUND](validation-heal-operations.md#BACKGROUND) → [CHECKPOINT](validation-heal-operations.md#CHECKPOINT) → [METRICS](validation-heal-operations.md#METRICS) → [LOGS](validation-heal-operations.md#LOGS) → [RECOVER](validation-heal-operations.md#RECOVER) → [PHYSICAL](validation-heal-operations.md#PHYSICAL) → [VERIFY](validation-heal-operations.md#VERIFY)
Source assertions:
-- 独占环境运行预注册 24h/72h 窗口,混合修复、掉线、写入和版本删除
-- 每次 ACK 可追踪;最终全量 manifest 正确,队列/内存/FD/临时空间无无界增长,无后台 panic 或未解释持久责任。
+- In an exclusive environment, run a preregistered 24h/72h window, mixing repair, offline events, writes, and version deletes
+- Every ACK is traceable; the final full manifest is correct; queue, memory, FDs, and temporary space do not grow without bound; and there is no background panic or unexplained persisted responsibility.
-先保存失败现场及物理/状态证据,停止本例写入与在途任务,再按 RECOVER 撤销本例故障。并发版本已变化时禁止盲目恢复旧 xl.meta;恢复不能充作产品修复。未恢复环境标 DIRTY,保留本例备份和原失败结果。
+First save the failure scene and the physical and state evidence, stop this case's writes and in-flight tasks, then undo this case's fault per RECOVER. When a concurrent version has already changed, do not blindly restore the old xl.meta; recovery must not serve as a product fix. Mark an unrestored environment DIRTY, and keep this case's backup and the original failure result.
-逐项满足设计公共契约、该分支步骤及附加断言后人工评审。缺资源为 BLOCKED,窗口未命中为 NOT_OBSERVED,产品错误为 FAIL,工具/恢复错误为 INFRA_ERROR;未运行不记 PASS。
+Manually review after each item satisfies the design's common contract, that branch's steps, and the additional assertions. Missing resources are BLOCKED, a window miss is NOT_OBSERVED, a product error is FAIL, and a tool or recovery error is INFRA_ERROR; a case that did not run is not recorded as PASS.
@@ -1960,18 +1960,18 @@ Source assertions:
Topologies: H2. Branches: abba-repeat-1, abba-repeat-2, abba-repeat-3.
-1. 同硬件/候选/rc预注册预热、每轮时长、并发、读写比例、数据量/缓存策略、P95/P99与错误率阈值;禁止结果出来再定门槛。
-2. 按A无heal、B有heal、B有heal、A无heal顺序,每轮从相同逻辑内容和队列基线开始;完成至少三组独立ABBA。
-3. WORKLOAD记录每请求延迟/成功/错误/超时,METRICS记录CPU/IO/FD及实际修复字节;错误不从总请求分母删除。
-4. 报告每轮及配对差异、修复吞吐和GET/PUT各自P95/P99;功能VERIFY必须通过,性能阈值也必须独立满足。
+1. On the same hardware, candidate, and rc, preregister the warmup, each round's duration, concurrency, the read/write ratio, the data volume and cache policy, and the P95/P99 and error-rate thresholds. Do not set the threshold after the results are in.
+2. Follow the order A without heal, B with heal, B with heal, and A without heal. Each round starts from the same logical content and the same queue baseline. Complete at least three independent ABBA groups.
+3. WORKLOAD records latency, success, error, and timeout for every request. METRICS records CPU, IO, FD, and the bytes actually repaired. Do not remove errors from the total-request denominator.
+4. Report each round and the paired difference, repair throughput, and the P95/P99 of GET and of PUT separately. Functional VERIFY must pass, and the performance thresholds must also be met on their own.
Operations: [SEED_DX](validation-heal-operations.md#SEED_DX) → [PERFORMANCE](validation-heal-operations.md#PERFORMANCE) → [WORKLOAD](validation-heal-operations.md#WORKLOAD) → [FAULT](validation-heal-operations.md#FAULT) → [START](validation-heal-operations.md#START) → [METRICS](validation-heal-operations.md#METRICS) → [LOGS](validation-heal-operations.md#LOGS) → [PHYSICAL](validation-heal-operations.md#PHYSICAL) → [VERIFY](validation-heal-operations.md#VERIFY)
Source assertions:
-- 同硬件同候选按 ABBA 运行无 heal 基线和有 heal 负载,重复并固定限速
-- 分别报告修复吞吐、GET/PUT P95/P99、错误率和 CPU/IO;阈值事先冻结,不能事后挑轮次或把功能 PASS 充性能验收。
+- On the same hardware and the same candidate, run the no-heal baseline and the with-heal load in ABBA order, repeat, and keep the rate limit fixed
+- Report repair throughput, GET/PUT P95/P99, error rate, and CPU/IO separately; freeze the thresholds in advance, and do not cherry-pick rounds afterward or treat a functional PASS as performance acceptance.
-先保存失败现场及物理/状态证据,停止本例写入与在途任务,再按 RECOVER 撤销本例故障。并发版本已变化时禁止盲目恢复旧 xl.meta;恢复不能充作产品修复。未恢复环境标 DIRTY,保留本例备份和原失败结果。
+First save the failure scene and the physical and state evidence, stop this case's writes and in-flight tasks, then undo this case's fault per RECOVER. When a concurrent version has already changed, do not blindly restore the old xl.meta; recovery must not serve as a product fix. Mark an unrestored environment DIRTY, and keep this case's backup and the original failure result.
-逐项满足设计公共契约、该分支步骤及附加断言后人工评审。缺资源为 BLOCKED,窗口未命中为 NOT_OBSERVED,产品错误为 FAIL,工具/恢复错误为 INFRA_ERROR;未运行不记 PASS。
+Manually review after each item satisfies the design's common contract, that branch's steps, and the additional assertions. Missing resources are BLOCKED, a window miss is NOT_OBSERVED, a product error is FAIL, and a tool or recovery error is INFRA_ERROR; a case that did not run is not recorded as PASS.
diff --git a/docs/validation-heal-contract.md b/docs/validation-heal-contract.md
index 63b216c..d4877b2 100644
--- a/docs/validation-heal-contract.md
+++ b/docs/validation-heal-contract.md
@@ -1,309 +1,309 @@
-# Heal 真机验证设计与验收契约
+# Heal live validation design and acceptance contract
-## 状态与范围
+## Status and scope
-本规范保留源仓库的 84 个 Heal canonical 用例、302 个操作分支及 534 个适用拓扑分支模板。资源要求、公共通过判据和逐项附加断言必须一起满足。
+This specification retains the source repository's 84 Heal canonical cases, 302 operation branches, and 534 applicable topology-branch templates. Resource requirements, the common pass criteria, and the per-item additional assertions must be satisfied together.
-入口为 `scripts/validation_heal/matrix.py`、[逐例操作卡](validation-heal-cases.md) 和 [执行手册](validation-heal-operations.md)。运行环境是当前 auto-testing 的 Heal 节点;候选及 rc 使用当前 validation lane 的不可变包和实际二进制摘要。源仓库历史 pilot 结果不能计入本次通过数量。
+The entry points are `scripts/validation_heal/matrix.py`, the [per-case procedure cards](validation-heal-cases.md), and the [execution manual](validation-heal-operations.md). The runtime environment is the Heal nodes of the current auto-testing setup; the candidate and rc use the immutable packages and the actual binary digests of the current validation lane. Historical pilot results from the source repository cannot be counted toward this run's pass total.
-全部 84 个 Heal ID 已有调度路由,包括 67 个常规入口、13 个默认关闭的环境场景入口,以及 C11/F07/I05/J02 的四个诊断入口。测试实现完整性与产品 live 结果独立统计:C11/F07/I05/J02 的源分支已有执行流程;F07 组合相同候选的 manager 中断与目录 fsync 证据,I05 总控串行执行 H1/H2 超时和断连控制器,并独立绑定完整内容/物理代次及 owner 原生证明,见[实现审查](validation-migration-implementation-review.md)。四个诊断入口的 canonical 验收门禁继续关闭;注册入口或 native 子集通过均不能充作完整 PASS。A09/G01–G06 的精确 native 与候选准入按各自执行器核对,G06 已有完整独立 live 验收。产品 FAIL 不自动等于测试未迁移。分片执行器核对实际盘角色、停止服务后的文件故障、原始同角色分片摘要与逐块 HighwayHash。A03 通过原路径/token 比较 H2 协调者、独立 peer 和固定节点入口的任务身份、终态及完整 outcome;候选每响应生成的 startTime 不作为任务 epoch。A04 保留四个 DB 参数组合及两个 DV 分支,要求完整 dry_run_observed 集合且对象没有变化。A05 使用独立部署执行全部八个 H1/H2 分支,冻结实际 merge 配置、真实非零 running 窗口、精确冲突与 token 策略及旧/新任务责任,完成后在服务运行状态下证明所有 peer 退出和物理静默。A06/F06 证明取消后的物理静默、旧 token 隔离;F06 还保留取消任务原始持久化记录,重启后不得复活或覆盖新任务。A08 保留五种主体各自 start/status/stop 操作、真实过期签名、仅授权 admin:Heal 的临时用户及清理后凭据失效。B09 使用明确启用 MRF 的专用服务,记录节点离线时的降级 ACK,并先验逐盘补片再执行目标 S3 请求。J03 必须实际超过 1024 项保留窗口,截断的诊断清单仍需独立全量物理/S3 补证。13 个环境场景保留源操作分支、未满足条件和显式启用入口;其完整 live 验证仍需资源适配器与源断言审核。每项实际状态见 [迁移映射](validation-migration-map.md);本地检查、操作卡和归档不能代替 live PASS。
+All 84 Heal IDs already have dispatch routes, including 67 regular entry points, 13 environment-scenario entry points that are off by default, and four diagnostic entry points for C11/F07/I05/J02. Test-implementation completeness and product live results are counted independently: the source branches of C11/F07/I05/J02 already have execution flows; F07 combines manager interruption and directory fsync evidence for the same candidate, and the I05 controller serially executes the H1/H2 timeout and disconnect controllers and independently binds the full content/physical generation and the owner's native proof, as described in the [implementation review](validation-migration-implementation-review.md). The canonical acceptance gates of the four diagnostic entry points remain closed; a passing registered entry point or a passing native subset cannot stand in for a full PASS. Exact native checks and candidate admission for A09/G01–G06 are verified by their respective executors, and G06 already has a complete, independent live acceptance. A product FAIL does not automatically mean the test was not migrated. The shard executor checks the actual disk role, the file fault after the service is stopped, the raw digests of same-role shards, and per-block HighwayHash. A03 compares, via the original path/token, the task identity, terminal state, and full outcome of the H2 coordinator, an independent peer, and the fixed node entry; the startTime generated by the candidate on each response is not used as the task epoch. A04 retains four DB parameter combinations and two DV branches, requires the complete dry_run_observed set, and requires that objects are unchanged. A05 uses an independent deployment to execute all eight H1/H2 branches, freezes the actual merge configuration, a real non-zero running window, the exact conflict and token policy, and old/new task responsibility, and, after completion, proves while the service is running that every peer has exited and that the system is physically quiescent. A06/F06 prove physical quiescence after cancellation and isolation of the old token; F06 also retains the cancelled task's original persisted record, which must not be revived or overwrite a new task after restart. A08 retains the start/status/stop operations of each of the five principals, a genuinely expired signature, a temporary user authorized only for admin:Heal, and credential invalidation after cleanup. B09 uses a dedicated service with MRF explicitly enabled, records the degraded ACK while a node is offline, and validates the per-disk shard repair before executing the target S3 request. J03 must actually exceed the 1024-item retention window; a truncated diagnostic manifest still requires independent full physical/S3 corroboration. The 13 environment scenarios retain the source operation branches, the unmet conditions, and the explicit enablement entry points; their complete live validation still requires review of the resource adapter and the source assertions. The actual status of each item is in the [migration map](validation-migration-map.md); local checks, procedure cards, and archives cannot replace a live PASS.
-B03/B04 replacement runner 先执行只读 mount admission preflight,再允许停止服务、下载/安装候选包或创建远端 run namespace;阻断结果明确记录 `execution_stage=pre-deployment`、候选/rc 来源和 `binary_sha256=null`,表示尚未观察到安装候选二进制。通过预检后,部署阶段仍会在 reservation 前再次读取同一组 mount/设备身份,避免预检与实际变更之间的环境漂移。
+The B03/B04 replacement runner first executes a read-only mount admission preflight before it is allowed to stop the service, download or install the candidate package, or create the remote run namespace. A blocking result explicitly records `execution_stage=pre-deployment`, the candidate/rc source, and `binary_sha256=null`, indicating that an installed candidate binary has not yet been observed. After the preflight passes, the deployment stage still reads the same set of mount and device identities again before reservation, so that the environment does not drift between the preflight and the actual change.
-H07 的单 set 两个分支只属于 H3,双 set 两个分支只属于 H4;四个不匹配组合记为 `nonApplicableCombinations`。缺少机器属于资源阻塞。
+Of H07, the two single-set branches belong only to H3, and the two dual-set branches belong only to H4; the four mismatched combinations are recorded as `nonApplicableCombinations`. A missing machine is a resource block.
-目标不是只证明“坏盘后仍能 GET”,而是分别证明:
+The goal is not only to prove that "GET still works after a disk goes bad", but to prove each of the following separately:
-1. 故障真实发生在指定对象、版本、set 和目标盘。
-2. 客户可见控制接口及期望触发源处理了这份修复责任。
-3. 可恢复时确实修好了目标分片;不可恢复时明确失败且不扩大损坏。
-4. 对象、历史版本、删除语义、并发成功写入和持久化责任正确。
-5. 原始证据完整,临时故障被解除,环境可安全交还。
+1. The fault actually occurred on the specified object, version, set, and target disk.
+2. The customer-visible control interface and the expected trigger source handled this repair responsibility.
+3. When recovery is possible, the target shard was actually repaired; when it is not, the failure is explicit and the damage is not widened.
+4. Objects, historical versions, delete semantics, concurrent successful writes, and persisted responsibility are correct.
+5. The raw evidence is complete, temporary faults have been cleared, and the environment can be handed back safely.
-## 版本与已核实的产品边界
+## Versions and verified product boundaries
-接口参考来自源设计记录的 RustFS 与 rc 提交,保留下面的源码链接用于追溯。每批执行仍以当前 candidate manifest 的精确源码、包、已安装二进制摘要和 auto-testing/rc 摘要为准,不由历史版本的验证结果推断本次候选已通过。
+Interface references come from the RustFS and rc commits recorded in the source design, and the source links below are retained for traceability. Each execution batch is still governed by the exact source, package, installed-binary digest, and auto-testing/rc digest in the current candidate manifest. A pass for this candidate is not inferred from the validation results of a historical version.
-| 已核实的接口/行为 | 对设计的影响 |
+| Verified interface/behavior | Impact on the design |
| --- | --- |
-| 签名 Admin POST `/rustfs/admin/v3/heal/`、`/{bucket}`、`/{bucket}/{prefix}`,token 查询与 forceStop/forceStart | 启动回执仅表示受理;必须查询任务并结合内容与物理证据验收。 |
-| POST `/rustfs/admin/v3/background-heal/status`,GET `/rustfs/admin/v4/heal/replacement-recovery` | 后台状态、完整 peer coverage、replacement readiness 与具体任务终态分开判断。 |
-| outcome 含 execution、coverage、counters、逐对象 disposition;objects/items 是有界诊断窗口 | finished、队列为 0、HTTP 200 都不是“所有对象已修好”的充分条件。 |
-| query 支持 sinceSeq;结果含 nextSeq/minSeq/truncated | 需要游标、截断和缺口负例;被截断的 items 不能用作完整清单。 |
-| Admin 保留 nolock wire 字段,但实际强制走正常对象锁路径 | A09 验证其不能绕过锁;不沿用历史架构文档中允许 Admin 绕锁的旧描述。 |
-| root heal 优雅重启恢复限于原协调者、原拓扑且记录盘仍可读 | 不预设跨协调者故障转移、记录盘丢失恢复或任意降级兼容;MRF/replacement/root 各按各自责任契约验收。 |
-| 参考 rc 提供 heal start/status/stop 及 bucket/prefix/token、normal/deep、dry-run、remove/recreate | pool/set、forceStart、sinceSeq 等未在该 CLI 版本暴露的功能通过签名 Admin 测,不虚构 rc 参数;新批次先检查实际 help/schema。 |
+| Signed Admin POST `/rustfs/admin/v3/heal/`, `/{bucket}`, `/{bucket}/{prefix}`, token query, and forceStop/forceStart | A start receipt only indicates that the request was accepted; the task must be queried, and acceptance must combine content evidence with physical evidence. |
+| POST `/rustfs/admin/v3/background-heal/status`, GET `/rustfs/admin/v4/heal/replacement-recovery` | Background status, full peer coverage, replacement readiness, and the terminal state of the specific task are judged separately. |
+| An outcome contains execution, coverage, counters, and per-object disposition; objects/items are a bounded diagnostic window | finished, a queue of 0, and HTTP 200 are not sufficient conditions for "all objects have been repaired". |
+| query supports sinceSeq; the result contains nextSeq/minSeq/truncated | Cursor, truncation, and gap negative cases are required; truncated items cannot be used as a complete manifest. |
+| Admin retains the nolock wire field, but is in practice forced onto the normal object-lock path | A09 verifies that it cannot bypass the lock; the old description in historical architecture documents, which allowed Admin to bypass the lock, is not carried forward. |
+| Recovery of root heal across a graceful restart is limited to the original coordinator, the original topology, and a record disk that is still readable | Cross-coordinator failover, recovery from a lost record disk, and arbitrary degraded compatibility are not assumed; MRF, replacement, and root are each accepted under their own responsibility contract. |
+| The reference rc provides heal start/status/stop, plus bucket/prefix/token, normal/deep, dry-run, and remove/recreate | Features not exposed in that CLI version, such as pool/set, forceStart, and sinceSeq, are tested through signed Admin, and rc parameters are not invented; a new batch first checks the actual help/schema. |
-源码:[Admin heal](https://github.com/rustfs/rustfs/blob/940e221988694da33fe005054bb86086bae88cd6/rustfs/src/admin/handlers/heal.rs)、[HealOpts](https://github.com/rustfs/rustfs/blob/940e221988694da33fe005054bb86086bae88cd6/crates/heal-contracts/src/heal_channel.rs)、[outcome/责任回执](https://github.com/rustfs/rustfs/blob/940e221988694da33fe005054bb86086bae88cd6/crates/heal/src/heal/outcome.rs)、[rc 命令](https://github.com/rustfs/cli/blob/204eb5ad85b46a007afd4da538cbeb1cadd867b6/crates/cli/src/commands/admin/heal.rs)。
+Source: [Admin heal](https://github.com/rustfs/rustfs/blob/940e221988694da33fe005054bb86086bae88cd6/rustfs/src/admin/handlers/heal.rs), [HealOpts](https://github.com/rustfs/rustfs/blob/940e221988694da33fe005054bb86086bae88cd6/crates/heal-contracts/src/heal_channel.rs), [outcome/responsibility receipt](https://github.com/rustfs/rustfs/blob/940e221988694da33fe005054bb86086bae88cd6/crates/heal/src/heal/outcome.rs), [rc command](https://github.com/rustfs/cli/blob/204eb5ad85b46a007afd4da538cbeb1cadd867b6/crates/cli/src/commands/admin/heal.rs).
-## 拓扑与故障域
+## Topology and fault domains
-下表是**需求模型,不是已经分配的可用机器清单**。所有真机动作必须经跳板机和固定 fleet lease,不能私自创建/删除 Azure VM、磁盘或网络。已有 example profile 不是运行许可;实现时必须绑定节点、端口、服务、mount UUID、盘数、pool/set/slot 与独占范围,读回实际 EC 元数据确认布局。
+The table below is a **requirements model, not an inventory of machines that have already been allocated**. Every live action must go through the jump host and a fixed fleet lease. Azure VMs, disks, and networks must not be created or deleted privately. An existing example profile is not permission to run. The implementation must bind the node, port, service, mount UUID, disk count, pool/set/slot, and exclusive range, and must read back the actual EC metadata to confirm the layout.
-| 编号 | 最低形状/用途 | 当前边界 |
+| ID | Minimum shape / purpose | Current boundary |
| --- | --- | --- |
-| H1 | 单节点四独立盘、一个 EC set;基础盘修复 | 可参考 single-4disk;使用独立 heal 服务和每盘 heal-validation 根。需要先绑定空闲资源,不复用现有 Pool 数据目录。 |
-| H2 | 四节点各四盘、同一分布式 pool,明确固定 set 划分;跨节点故障/锁/状态 | 可参考 cluster-1pool;必须证明目标 set 跨节点,不能只看全局节点总数。 |
-| H3 | 精确 EC8+4 单 set,参考主仓库的 3 节点×4 盘测试形状 | 是测试参数,不是生产部署推荐;旧两节点 Pool 环境不满足此拓扑。 |
-| H4 | 精确 EC8+4 至少两个 set,参考 3 节点×8 盘、同 pool | 至少 24 个独立盘端点及正确 set 映射;不得用 4 盘或一个 set 替代。固定资源未分配时为阻塞,不是通过。 |
-| H5 | 两个以上 pool;分开登记两节点各自四盘的生命周期形状和跨节点 pool 形状 | 现 lifecycle-a 每台一个独立 pool,无节点级冗余;可验证盘级修复及移动交互,不能证明节点故障容错。对齐主仓库多 pool EC8+4 基准时,另登记其三单机×12 盘/三个 pool 布局,不混用计数。 |
-| H6 | 独占 VM/主机与专用故障盘,可做强制重启 | 需要额外运维授权、带外恢复和事件证据;普通故障 lane 不允许自动进入。 |
+| H1 | One node, four independent disks, one EC set; basic disk repair | single-4disk may be used as a reference; use an independent heal service and a heal-validation root on each disk. Idle resources must be bound first, and existing Pool data directories are not reused. |
+| H2 | Four nodes with four disks each, the same distributed pool, and an explicit fixed set division; cross-node faults, locks, and state | cluster-1pool may be used as a reference; the target set must be proven to span nodes, and the global node count alone is not sufficient. |
+| H3 | Exact EC8+4, a single set, referencing the main repository's test shape of 3 nodes × 4 disks | This is a test parameter, not a production deployment recommendation; an old two-node Pool environment does not satisfy this topology. |
+| H4 | Exact EC8+4 with at least two sets, referencing 3 nodes × 8 disks in the same pool | At least 24 independent disk endpoints and a correct set mapping; 4 disks or a single set must not be substituted. When the fixed resources have not been allocated, the result is blocked, not a pass. |
+| H5 | Two or more pools; separately register the lifecycle shape of two nodes with four disks each, and the cross-node pool shape | The current lifecycle-a has one independent pool per machine and no node-level redundancy; it can validate disk-level repair and move interaction, and it cannot prove tolerance of a node failure. When aligning with the main repository's multi-pool EC8+4 baseline, separately register its layout of three single machines × 12 disks / three pools, and do not mix the counts. |
+| H6 | An exclusive VM/host and a dedicated fault disk, able to perform a forced restart | Additional operations authorization, out-of-band recovery, and event evidence are required; a normal fault lane is not allowed to enter automatically. |
-生产拓扑取舍可参考 [RustFS Linux 部署说明](https://docs.rustfs.com/en/installation/linux):单机多盘的盘冗余不等于整机故障冗余。本设计的结论只对实际验证的拓扑成立。
+Production topology trade-offs may refer to the [RustFS Linux deployment guide](https://docs.rustfs.com/en/installation/linux): disk redundancy from multiple disks on a single machine is not whole-machine failure redundancy. Conclusions of this design hold only for the topologies that were actually validated.
-每个对象/版本预先记录 N、data/parity、分片索引与 set/盘映射,再从候选实现与元数据确定读取 quorum R、修复写 quorum W 和锁 quorum。测试 R-1/R/R+1 与 W-1/W/W+1 的**合法组合**,不以全局在线盘数或固定“坏一半仍能写”代替计算。跨版本、跨 set 的片不能相加;R 与 W 不一定相同。N=1 无 EC 的能力/不可恢复负例属于额外参数实例,不得宣传可自动找回完全丢失的数据。
+For each object/version, record N, data/parity, the shard index, and the set/disk mapping in advance, then determine read quorum R, repair-write quorum W, and lock quorum from the candidate implementation and the metadata. Test the **legal combinations** of R-1/R/R+1 and W-1/W/W+1. Do not replace that calculation with the global number of online disks, or with a fixed assumption that "writes still succeed with half the disks bad". Shards across versions or across sets must not be added together; R and W are not necessarily the same. The N=1 no-EC capability and unrecoverable negative cases are additional parameter instances, and it must not be advertised that completely lost data can be recovered automatically.
-## 故障后端与安全门禁
+## Fault backends and safety gates
-所有注入必须有 machine-readable fault-plan 和 fault-receipt;不存在“命令成功即注入成功”。默认每次只隔离本用例一个已解析目标,再按预注册组合扩展。使用以下分类,报告必须记录实际 faultBackend,不能互相冒充。
+Every injection must have a machine-readable fault-plan and fault-receipt. There is no rule that "a successful command means a successful injection". By default, each run isolates only one resolved target of this case, then expands according to the pre-registered combinations. Use the classes below. The report must record the actual faultBackend, and the classes must not impersonate one another.
-| 编号 | 故障类别 | 注入有效性与限制 |
+| ID | Fault class | Injection validity and limits |
| --- | --- | --- |
-| F0 | 无故障、纯接口请求 | 记录稳定基线;仅用于无副作用/负请求验证。 |
-| F1 | 指定 part/shard 缺失 | 先校验精确路径和摘要,以同盘隔离备份/原子移位保存,再证明目标缺失;不是物理掉盘。 |
-| F2 | 指定字节翻转、截断、校验头损坏 | 在隔离测试对象副本上操作,保存偏移、长度、前后摘要及原始备份;必须有足够未损坏权威。 |
-| F3 | 指定 xl.meta/inline/身份/桶元数据缺失或损坏 | 需要与候选格式匹配的只读解析器和允许列表;不得操作 IAM/凭据配置、共享 format 或无关对象。 |
-| F4 | 原逻辑盘/服务卷短暂不可达、只读 | 采用仅影响测试服务的受限挂载命名空间或故障层,并实际验证访问失败。不得卸载含其他服务数据的共享 XFS mount;chmod 不能未经验证当作有效故障。 |
-| F5 | 慢 I/O、EIO、短读、ENOSPC/inode 故障 | 只在预分配独占测试设备或有容量上限的故障层;不得塞满共享数据盘。明确模拟故障与真实设备故障的区别。 |
-| F6 | 测试进程优雅停止、SIGKILL、重启 | 限定已验证服务和 MainPID,证明在部分进度窗口发生;记录 boot/process 身份。SIGKILL 不是断电。 |
-| F7 | 节点间测试端口网络故障 | 仅对本服务地址/端口、受控 namespace 或独占环境,保护 SSH/跳板机/管理平面;规则带唯一 ID、超时和幂等恢复。 |
-| F8 | 空替换逻辑盘、新 disk UUID | 验证原/新盘和 pool/set/slot;不把移动回备份当重建。真实设备 detach/attach 另需独占资源授权,不修改共享盘。 |
-| F9 | checkpoint/commit/锁窗口的确定性故障 | 优先原生 runtime-fence/E2E 屏障;Release 黑盒无法证明精确窗口时,单独列原生证据与真机覆盖缺口,禁止给优化包塞入隐藏测试 hook 冒充同一候选。 |
-| F10 | 独占 VM/主机强制重启 | 单独审批的破坏性 lane;有带外恢复和平台时间线。VM forced stop 不证明物理控制器缓存断电语义。 |
-| F11 | 队列/请求压力、测试 KMS/tier/配额依赖故障 | 只作用于已分配测试租户/进程,限量限时,禁止全局断开共享 KMS 或远端服务。 |
-
-fault-plan 至少包含 run/case/variant ID、拓扑指纹、对象/版本/incarnation、pool/set/slot、盘 UUID、精确文件允许列表、设备/根目录身份、注入前条件、变更预算、备份摘要、恢复动作和自动解除故障截止时间。路径按目录描述符解析,拒绝链接逃逸、设备漂移、未知文件和自由 shell 插值。
-
-故障解除与数据清理是两件事:网络/只读/进程等临时环境损伤要安全解除;失败数据、分片、checkpoint 和证据应保留,环境标记 DIRTY/QUARANTINED,不能继续下一例。捕获证据后可按已批准流程恢复业务可达性,但不覆盖失败记录或把恢复重跑算首次通过。备份只可用于失败后的救援/重置,不参与修复成功判定。
-
-## 数据夹具与参数维度
-
-| 编号 | 内容 |
+| F0 | No fault; a pure interface request | Record a stable baseline; used only for no-side-effect and negative-request validation. |
+| F1 | A specified part/shard is missing | First verify the exact path and digest, preserve it with an isolated same-disk backup or an atomic move, then prove that the target is missing; this is not a physical disk loss. |
+| F2 | Specified bit flips, truncation, or checksum-header corruption | Operate on an isolated copy of the test object, and save the offset, length, before and after digests, and the original backup; enough undamaged authority must remain. |
+| F3 | Specified xl.meta, inline data, identity, or bucket metadata is missing or corrupt | Requires a read-only parser that matches the candidate format, and an allowlist; IAM and credential configuration, the shared format, and unrelated objects must not be operated on. |
+| F4 | The original logical disk or service volume is briefly unreachable or read-only | Use a restricted mount namespace or fault layer that affects only the test service, and actually verify that access fails. A shared XFS mount that contains data of other services must not be unmounted; chmod must not be treated as a valid fault until it has been verified. |
+| F5 | Slow I/O, EIO, short reads, or ENOSPC/inode faults | Only on a pre-allocated exclusive test device or a fault layer with a capacity cap; a shared data disk must not be filled up. The difference between a simulated fault and a real device fault must be explicit. |
+| F6 | Graceful stop, SIGKILL, and restart of the test process | Limited to the verified service and MainPID, and proven to occur inside a partial-progress window; record the boot and process identity. SIGKILL is not a power loss. |
+| F7 | A network fault on inter-node test ports | Only for this service's address and port, a controlled namespace, or an exclusive environment, while protecting SSH, the jump host, and the management plane; rules carry a unique ID, a timeout, and idempotent recovery. |
+| F8 | An empty replacement logical disk, with a new disk UUID | Verify the original and new disks and pool/set/slot; moving data back from a backup is not a rebuild. Real-device detach/attach additionally requires exclusive-resource authorization, and shared disks are not modified. |
+| F9 | A deterministic fault in the checkpoint, commit, or lock window | Prefer a native runtime-fence or E2E barrier; when a Release black box cannot prove the exact window, list the native evidence and the live coverage gap separately. Do not insert a hidden test hook into an optimized package to impersonate the same candidate. |
+| F10 | Forced restart of an exclusive VM/host | A separately approved destructive lane, with out-of-band recovery and a platform timeline. A VM forced stop does not prove the power-loss semantics of a physical controller cache. |
+| F11 | Queue or request pressure, and faults in test KMS, tier, or quota dependencies | Acts only on the allocated test tenant or process, with limits on quantity and duration. A shared KMS or a remote service must not be disconnected globally. |
+
+A fault-plan contains at least the run/case/variant ID, the topology fingerprint, the object/version/incarnation, pool/set/slot, the disk UUID, the exact file allowlist, the device and root-directory identity, the pre-injection conditions, the change budget, the backup digest, the recovery action, and the deadline for automatic fault clearance. Paths are resolved through directory descriptors. Link escape, device drift, unknown files, and free shell interpolation are rejected.
+
+Clearing a fault and cleaning up data are two separate matters. Temporary environmental damage, such as network, read-only, or process faults, must be cleared safely. Failed data, shards, checkpoints, and evidence should be retained, the environment marked DIRTY/QUARANTINED, and the next case must not proceed. After evidence has been captured, business reachability may be restored by an approved procedure, but failure records must not be overwritten, and a recovery rerun must not be counted as the first pass. A backup may be used only for rescue or reset after a failure, and it does not take part in the repair-success verdict.
+
+## Data fixtures and parameter dimensions
+
+| ID | Content |
| --- | --- |
-| DB | 确定性基础 manifest:默认 128 个对象,包含 0/1、4095/4096/4097 字节以及实际 inline、EC stripe/block、multipart 边界±1;按磁盘证据确认形态,不硬编码内部阈值。 |
-| DM | 已完成的多 part 对象、非等长 part、跨 part Range;另外创建可追踪 UploadId 的 pending/Complete/Abort 夹具,遵守 S3 最小非末 part 大小限制。 |
-| DV | 每 key 至少三个不同内容版本、当前/历史 delete marker、versioning suspended/null、明确删除回执;记录 versionId、Last-Modified、ETag 和版本列表。 |
-| DS | 多桶、相邻前缀、多页 key、中文/空格/百分号等编码,metadata/tags/content headers 与隔离桶配置。 |
-| DE | 候选明确支持且依赖就绪的 SSE-S3/KMS/C、远端 tier 版本;密钥只由受控凭据渠道提供,禁止进入日志/manifest。 |
-| DO | object-lock、retention 和 legal hold 的受保护版本;使用可到期测试策略,防止成功用例因无法清理永久占用环境。 |
-| DL | 并发 PUT/GET/HEAD/LIST/COPY/DELETE、版本及 MPU 操作的逐请求 ACK ledger;每 key 用递增 nonce 与确定性内容,保存请求/响应和不确定结果。 |
-| DX | 保证可观察部分进度、大于状态窗口(>128 项)的夹具;默认上限在运行计划中冻结,长稳/性能单独配额,不靠无限加数据等到偶然命中。 |
+| DB | Deterministic baseline manifest: 128 objects by default, including 0/1 and 4095/4096/4097 bytes, plus the actual inline, EC stripe/block, and multipart boundaries ±1; confirm the shape from disk evidence, and do not hard-code internal thresholds. |
+| DM | Completed multipart objects, parts of unequal length, and cross-part Ranges; also create pending/Complete/Abort fixtures with a traceable UploadId, observing the S3 minimum size for a non-final part. |
+| DV | At least three versions of different content per key, current and historical delete markers, versioning suspended/null, and an explicit delete receipt; record versionId, Last-Modified, ETag, and the version list. |
+| DS | Multiple buckets, adjacent prefixes, multi-page keys, encodings such as Chinese, spaces, and percent signs, metadata/tags/content headers, and isolated bucket configuration. |
+| DE | SSE-S3/KMS/C and remote tier versions that the candidate explicitly supports and whose dependencies are ready; keys are supplied only through a controlled credential channel and must not enter logs or the manifest. |
+| DO | Protected versions under object-lock, retention, and legal hold; use an expirable test policy so that a successful case does not permanently occupy the environment because it cannot be cleaned up. |
+| DL | A per-request ACK ledger of concurrent PUT/GET/HEAD/LIST/COPY/DELETE, version, and MPU operations; each key uses an incrementing nonce and deterministic content, and the request, the response, and indeterminate results are saved. |
+| DX | Fixtures that guarantee observable partial progress and exceed the status window (>128 items); the default upper bound is frozen in the run plan, soak and performance have a separate quota, and the test does not add data without limit while waiting for a chance hit. |
-按 case/variant 参数化:normal/deep、data/parity shard、inline/外置 part、单片/多片、原盘/新盘、对象/前缀/root/set、版本/加密、协调者/目标节点、单节点/多节点/多 set/pool。使用预先冻结的 pairwise 组合控制规模;损坏+quorum、旧代次+并发删除、目标重建+进程中断等高风险组合必须显式三维覆盖,不由 pairwise 替代。
+Parameterize by case/variant: normal/deep, data/parity shard, inline/external part, single shard/multiple shards, original disk/new disk, object/prefix/root/set, version/encryption, coordinator/target node, and single-node/multi-node/multi-set/pool. Use pairwise combinations frozen in advance to control scale. High-risk combinations, such as corruption plus quorum, an old generation plus concurrent delete, and target rebuild plus process interruption, must have explicit three-dimensional coverage, and pairwise does not replace them.
-候选声明支持的非默认 parity/storage class、服务端压缩 on/off 也应加入 DB/DM/DV 参数;Content-Encoding 响应头不是服务端压缩已被执行的证据。每个实例都从实际元数据确认编码形态。Object Lock 的拒绝删除必须发生于真实保留期内;清理不能绕过 Compliance、修改主机时间或以内部删文件规避限制。Legal hold 仅由本次授权测试角色正常解除;保留期超出 lane 窗口的实例必须预先分配独占 stateful 环境及到期处理方案,未清理环境不能交还 READY。
+Non-default parity and storage class, and server-side compression on/off, that the candidate claims to support should also be added to the DB/DM/DV parameters. A Content-Encoding response header is not evidence that server-side compression was executed. Every instance confirms the encoded form from the actual metadata. An Object Lock rejection of delete must occur during a real retention period. Cleanup must not bypass Compliance, change the host time, or evade the restriction by deleting files internally. A legal hold is released only in the normal way by the test role authorized for this run. An instance whose retention exceeds the lane window must be allocated, in advance, an exclusive stateful environment and an expiry-handling plan. An environment that has not been cleaned up cannot be returned as READY.
-重试不能隐藏 ACK:PUT/Complete 超时属于结果未知,必须按唯一 key/version/UploadId 查询解析;不能把未收到成功的一次写当丢失,也不能把重试新版本当同一次成功。并发时只根据可证明的 happens-before/提交回执判断“最后一次”,不凭跨节点墙钟排序。
+A retry must not hide an ACK. A PUT or Complete timeout is an unknown result and must be resolved by querying the unique key, version, or UploadId. A write for which success was not received must not be treated as lost, and a new version produced by a retry must not be treated as the same success. Under concurrency, "the last one" is judged only from a provable happens-before or commit receipt, and not by ordering wall clocks across nodes.
-## 公共通过判据(每例均适用)
+## Common pass criteria (apply to every case)
-每行用例的完整规范 = 本节 + 对应拓扑/夹具/故障契约 + 行内操作与附加断言。参数实例独立留证、计分。
+The complete specification of each case row = this section + the corresponding topology, fixture, and fault contract + the in-row operation and additional assertions. Parameter instances are evidenced and scored independently.
-1. **预检和基线。** 全部预期 peer/磁盘、candidate/rc/harness 身份一致;预先 GET/HEAD/LIST 完整基线并保存必要的盘上记录。记录扫描/自动 heal 配置与任务归属。
-2. **注入和见证。** 以只读物理证据确认故障确实发生,未影响其他对象/服务;注入后不要先 GET 目标而意外触发 read-repair。受理过快或已被其他来源修好,不能冒充本例触发成功。
-3. **控制面真相。** 收集每次 rc exitCode、签名 Admin HTTP/body、token、source、scope、outcome 和 peer coverage。区分 repaired、verified_healthy、metadata_healthy、authoritatively_absent、deferred、failed、unknown、cancelled、dry_run_observed;deferred、failed、unknown、cancelled、dry_run_observed 不能作为“修复完成”。`metadata_healthy` 只证明 metadata/identity 健康,不证明 payload integrity;Normal scan 不得冒充 `verified_healthy`,deep payload 证明继续要求 `verified_healthy`。实际 wire 枚举以固定候选为准。
-4. **对象正确。** 所有必须存在的对象/版本全量 GET 验证长度/SHA-256/ETag/metadata/tags;HEAD、分页 LIST、历史版本与 marker 语义、已确认删除和并发 ACK 全部核对。允许修复产生合法内部元数据变化,不要求整个 xl.meta 字节不变。
-5. **实际补片。** 在静默校验窗口解析目标物理数据与元数据,核对版本/代次/data_dir、part 长度、bitrot 校验与 EC 一致性。对 data/parity 各有独立重建或重新编码对照,或用包含目标片的固定合法解码集合佐证;不能只统计文件、看 mtime 或依赖 GET 自动纠删码兜底。密文/压缩存储层不能直接拿明文摘要比较。格式不支持/HighwayHash 未验证时必须标明缺口,不能声称已完成深度物理验收。
-6. **持久化与幂等。** 需重启的用例验证重启后责任和数据;重复 heal 不引入新版本、复活删除或无限增加任务。状态计数按进程/任务/epoch 区分,不能把重置后的全局计数简单判为回退。
-7. **完成条件。** 正向修复要有相关责任完成、范围 coverage 完整、对象与物理证据一致。无关的全局 idle、列表项被截断、缺失 peer、unknown/残留待修工作都不能代替证明。预期不可恢复/权限/取消/dry-run 用例按预注册负向契约通过,明确它不是修复成功。
-8. **恢复和证据。** 解除临时故障,保存 before/fault/after/restart 四阶段证据,校验全局结果/JUnit/计划/对象数量/证据引用及清理。任何取证缺失、目标误命中、恢复失败或结果结构损坏均阻断该 lane。
+1. **Preflight and baseline.** All expected peers and disks, and the candidate, rc, and harness identities, are consistent. GET, HEAD, and LIST the complete baseline in advance, and save the necessary on-disk records. Record the scan and automatic-heal configuration and the task ownership.
+2. **Injection and witness.** Confirm, with read-only physical evidence, that the fault actually occurred and did not affect other objects or services. After injection, do not GET the target first and accidentally trigger read-repair. Acceptance that is too fast, or a repair already completed by another source, cannot impersonate a successful trigger of this case.
+3. **Control-plane truth.** Collect every rc exitCode, the signed Admin HTTP status and body, and the token, source, scope, outcome, and peer coverage. Distinguish repaired, verified_healthy, metadata_healthy, authoritatively_absent, deferred, failed, unknown, cancelled, and dry_run_observed. deferred, failed, unknown, cancelled, and dry_run_observed cannot stand for "repair complete". `metadata_healthy` proves only that metadata and identity are healthy, and does not prove payload integrity. A Normal scan must not impersonate `verified_healthy`, and proof of a deep payload continues to require `verified_healthy`. The actual wire enumeration is taken from the pinned candidate.
+4. **Object correctness.** Full GET of every object and version that must exist, verifying length, SHA-256, ETag, metadata, and tags. HEAD, paginated LIST, historical-version and marker semantics, confirmed deletes, and concurrent ACKs are all checked. A repair may produce legal internal metadata changes; the whole xl.meta is not required to stay byte-for-byte unchanged.
+5. **Actual on-disk shard repair.** In the quiescent verification window, parse the target physical data and metadata, and check version, generation, data_dir, part length, bitrot checksums, and EC consistency. Data and parity each have an independent rebuild or re-encode comparison, or are corroborated by a fixed legal decoding set that includes the target shard. Counting files, looking at mtime, or relying on GET's automatic erasure-code fallback is not sufficient. A ciphertext or compressed storage layer cannot be compared directly with a plaintext digest. When the format is unsupported, or HighwayHash has not been verified, the gap must be stated, and deep physical acceptance must not be claimed as complete.
+6. **Persistence and idempotency.** Cases that require a restart verify responsibility and data after the restart. A repeated heal does not introduce a new version, revive a delete, or increase tasks without bound. Status counts are distinguished by process, task, and epoch; a global count after a reset must not be judged as a regression on that basis alone.
+7. **Completion conditions.** A positive repair requires the relevant responsibility to be complete, scope coverage to be complete, and the object evidence to agree with the physical evidence. An unrelated global idle, truncated list items, a missing peer, or unknown or leftover work still awaiting repair cannot substitute for that proof. Cases expected to be unrecoverable, denied by permission, cancelled, or dry-run pass against the pre-registered negative contract, and it is explicit that this is not a successful repair.
+8. **Recovery and evidence.** Clear the temporary fault, save the four-phase before/fault/after/restart evidence, and check the global result, JUnit, plan, object counts, evidence references, and cleanup. Any missing evidence capture, a hit on the wrong target, a recovery failure, or a damaged result structure blocks that lane.
-针对精确时间窗口,必须观察活动/部分进度或受控屏障,再注入故障;普通重建允许快速完成但必须有实际补片证据。禁止用固定 sleep 或“重跑直到绿”代替见证。建议起始测试预算:接口就绪 90 秒、普通修复 15 分钟、重启恢复 30 分钟、同 epoch 静默窗口 5 秒;这些是待校准的**测试预算,不是产品 SLO**,所有最终预算须在运行前冻结。预期暂态错误只按明确 code 有界重试,每次都留证。
+For an exact time window, activity, partial progress, or a controlled barrier must be observed before the fault is injected. An ordinary rebuild may finish quickly, but it must have evidence of actual on-disk shard repair. A fixed sleep, or "rerun until green", must not replace a witness. Suggested starting test budgets: 90 seconds for interface readiness, 15 minutes for ordinary repair, 30 minutes for restart recovery, and 5 seconds for the same-epoch quiescent window. These are **test budgets pending calibration, not a product SLO**, and every final budget must be frozen before the run. An expected transient error is retried only within bounds and only for an explicit code, and every attempt is evidenced.
-判定类型:产品安全/内容/终态错误为 fail;环境注入/取证工具失败为 infra_error;缺少尚未分配拓扑或辅助能力在部署前阻断。不得把产品 SlowDown 或低 quorum 下的预期错误泛化成 infra_error。必选能力缺失、未知选项、缺失参数实例不能用 unsupported/skipped 充 PASS;不适用项只能在运行前的 applicability manifest 中说明,环境尚未准备好不是“不适用”。
+Verdict types: a product safety, content, or terminal-state error is fail; a failure of environment injection or of an evidence-capture tool is infra_error; absence of a topology or auxiliary capability that has not yet been allocated blocks the run before deployment. A product SlowDown, or an expected error under low quorum, must not be generalized into infra_error. A missing required capability, an unknown option, or a missing parameter instance cannot be counted as PASS by way of unsupported or skipped. An inapplicable item may be stated only in the applicability manifest before the run. An environment that is not yet ready is not "not applicable".
-## 84 个基础用例
+## 84 base cases
-编号用于设计评审;`heal.*` 为稳定 catalog ID。topology、fixture、fault 中逗号表示必须展开记录的适用实例/后端集合,不表示同时向全部对象注入所有故障。对应的组合、先后和最小 quorum 必须在编译后的 variant plan 中明确,未证明的一项不得计为通过。
+IDs are for design review; `heal.*` is the stable catalog ID. A comma in topology, fixture, or fault denotes the set of applicable instances or backends that must be expanded and recorded. It does not mean that every fault is injected into every object at the same time. The corresponding combination, order, and minimum quorum must be explicit in the compiled variant plan, and an item that has not been proven must not be counted as a pass.
-| 分类 | 设计条目数 |
+| Category | Design item count |
| --- | ---: |
-| A 控制面、任务与权限契约 | 10 |
-| B 磁盘故障、替换与 quorum 边界 | 10 |
-| C 分片损坏、元数据与权威选择 | 11 |
-| D 对象类型与 S3 语义 | 11 |
-| E 触发路径与修复责任 | 6 |
-| F 中断、持久化、恢复与升级 | 9 |
-| G 与前台和后台操作并发 | 8 |
-| H 多 Pool、移动与多 set 隔离 | 7 |
-| I 资源、依赖故障与调度 | 6 |
-| J 可观测性、幂等、长稳与性能 | 6 |
-| 合计 | 84 |
+| A Control plane, tasks, and authorization contract | 10 |
+| B Disk faults, replacement, and quorum boundaries | 10 |
+| C Shard corruption, metadata, and authority selection | 11 |
+| D Object types and S3 semantics | 11 |
+| E Trigger paths and repair responsibility | 6 |
+| F Interruption, persistence, recovery, and upgrade | 9 |
+| G Concurrency with foreground and background operations | 8 |
+| H Multiple pools, movement, and multi-set isolation | 7 |
+| I Resources, dependency faults, and scheduling | 6 |
+| J Observability, idempotency, soak, and performance | 6 |
+| Total | 84 |
-### A. 控制面、任务与权限契约
+### A. Control plane, tasks, and authorization contract
-| 编号 / 稳定 ID | P | 拓扑 / 夹具 / 故障 | 操作与触发 | 附加通过判据 |
+| ID / stable ID | P | Topology / fixture / fault | Operation and trigger | Additional pass criteria |
| --- | --- | --- | --- | --- |
-| A01 · `heal.admin-healthy-noop` | P0 | H1,H2 / DB / F0 | 对健康对象分别执行 normal/deep,再重复查询 | 不修改用户数据或版本;Normal 只接受 `metadata_healthy`,Deep 才要求 `verified_healthy`,healthy 与 repaired 分开计数,不能把检查盘数当修复盘数。 |
-| A02 · `heal.admin-scope-isolation` | P0 | H1,H2 / DB,DS / F1 | 分别以 root、bucket、prefix、object、pool/set 选择器启动 | 目标范围实际遍历;相邻前缀、其他桶及未选 set 的故障保持原状;按对象清单而非目录数量验收。 |
-| A03 · `heal.admin-token-cross-node-status` | P0 | H2 / DB / F1 | 从节点甲启动,经乙及稳定入口按原 token 查询 | 任务身份、scope、终态及核心计数有界收敛;状态缺失/未知不能解释为完成。 |
-| A04 · `heal.admin-dry-run` | P0 | H1,H2 / DB,DV / F1,F2 | 对缺片与损坏对象执行 dryRun,参数化 remove/recreate | 有明确 dry_run_observed 诊断;用户对象分片、xl.meta、版本与删除标记不变;后台修复干扰须排除。 |
-| A05 · `heal.admin-overlap-and-force-start` | P1 | H1,H2 / DB,DS / F1 | 活动中重发同范围、父子前缀和不相交请求,再 forceStart | 按已固定 overlap policy 验证合并返回原 token 或明确冲突;forceStart 不能泄漏旧任务责任或并发破坏对象。 |
-| A06 · `heal.admin-stop-and-stale-token` | P0 | H1,H2 / DB / F1 | 非零修复进度时 stop,稳定后新建任务,再发旧 token 的查询/停止 | 停止需等在途写结束并稳定;新任务身份独立,旧请求不得终止新任务;重复 stop 按公开契约处理。 |
-| A07 · `heal.admin-invalid-request` | P0 | H1,H2 / DB / F0 | 发送非法 selector、重复 query、非法布尔/scanMode、冲突 flags、超长 body | 服务端返回已核实的结构化错误;无误启动、范围扩大或持久化副作用;CLI 参数错误不能替代服务端负例。 |
-| A08 · `heal.admin-authorization` | P0 | H1,H2 / DB / F0 | 匿名、坏签名、过期签名、无 HealAdminAction 用户与最小授权用户分别请求 | 未授权 start/status/stop 均不能越权或泄漏对象信息;授权用户可完成规定操作,凭据不得写入证据。 |
-| A09 · `heal.admin-nolock-safety` | P0 | H1,H2 / DL,DV / F1,F9 | 通过签名 API 传 nolock=true,并与同对象提交制造确定性争用 | 兼容字段不得跳过存储层对象锁;最后成功写入和已确认删除保持正确,原生屏障与真机竞争证据分开。 |
-| A10 · `heal.admin-option-contract` | P1 | H1,H2 / DB,DV / F1,F2 | 参数化 recursive、recreate、remove、updateParity、readRepair 与合法/非法 pool/set 组合 | 仅测试已核实的候选语义;normal/deep 与危险 remove 的副作用明确;API 独有选项不得伪称 rc 已覆盖。 |
+| A01 · `heal.admin-healthy-noop` | P0 | H1,H2 / DB / F0 | Run normal and deep separately on a healthy object, then query repeatedly | Do not modify user data or versions; Normal accepts only `metadata_healthy`, and only Deep requires `verified_healthy`. healthy and repaired are counted separately, and the number of disks checked must not be treated as the number of disks repaired. |
+| A02 · `heal.admin-scope-isolation` | P0 | H1,H2 / DB,DS / F1 | Start separately with the root, bucket, prefix, object, and pool/set selectors | The target scope is actually traversed; faults on adjacent prefixes, other buckets, and unselected sets remain as they were; acceptance is by the object manifest, not by directory count. |
+| A03 · `heal.admin-token-cross-node-status` | P0 | H2 / DB / F1 | Start from node A, and query by the original token via node B and the stable entry | Task identity, scope, terminal state, and core counters converge within bounds; a missing or unknown status cannot be interpreted as complete. |
+| A04 · `heal.admin-dry-run` | P0 | H1,H2 / DB,DV / F1,F2 | Run dryRun on objects with a missing shard and on corrupt objects, parameterized by remove/recreate | There is an explicit dry_run_observed diagnosis; user-object shards, xl.meta, versions, and delete markers are unchanged; interference from background repair must be excluded. |
+| A05 · `heal.admin-overlap-and-force-start` | P1 | H1,H2 / DB,DS / F1 | While a task is active, resubmit the same scope, parent and child prefixes, and disjoint requests, then forceStart | Per the pinned overlap policy, verify that a merge returns the original token or an explicit conflict; forceStart must not leak the old task's responsibility or damage objects concurrently. |
+| A06 · `heal.admin-stop-and-stale-token` | P0 | H1,H2 / DB / F1 | Stop at non-zero repair progress, create a new task after the system stabilizes, then query and stop with the old token | A stop must wait for in-flight writes to finish and for the system to stabilize; the new task identity is independent, and the old request must not terminate the new task; a repeated stop is handled according to the public contract. |
+| A07 · `heal.admin-invalid-request` | P0 | H1,H2 / DB / F0 | Send an illegal selector, a duplicate query, an illegal boolean or scanMode, conflicting flags, and an oversized body | The server returns a verified structured error; there is no mistaken start, no scope expansion, and no persisted side effect; a CLI parameter error cannot substitute for a server-side negative case. |
+| A08 · `heal.admin-authorization` | P0 | H1,H2 / DB / F0 | Request separately as anonymous, with a bad signature, with an expired signature, as a user without HealAdminAction, and as a minimally authorized user | Unauthorized start, status, and stop must neither exceed privilege nor leak object information; an authorized user can complete the prescribed operations, and credentials must not be written into the evidence. |
+| A09 · `heal.admin-nolock-safety` | P0 | H1,H2 / DL,DV / F1,F9 | Pass nolock=true through the signed API, and create deterministic contention with a commit of the same object | The compatibility field must not skip the storage-layer object lock; the last successful write and confirmed deletes remain correct, and native-barrier evidence is kept separate from live contention evidence. |
+| A10 · `heal.admin-option-contract` | P1 | H1,H2 / DB,DV / F1,F2 | Parameterize recursive, recreate, remove, updateParity, readRepair, and legal and illegal pool/set combinations | Test only candidate semantics that have been verified; the side effects of normal/deep and of the dangerous remove are explicit; an option that exists only on the API must not be falsely claimed as already covered by rc. |
-### B. 磁盘故障、替换与 quorum 边界
+### B. Disk faults, replacement, and quorum boundaries
-| 编号 / 稳定 ID | P | 拓扑 / 夹具 / 故障 | 操作与触发 | 附加通过判据 |
+| ID / stable ID | P | Topology / fixture / fault | Operation and trigger | Additional pass criteria |
| --- | --- | --- | --- | --- |
-| B01 · `heal.disk-loss-and-repair` | P0 | H1,H2 / DB / F4 | 让一个已确认 set 成员盘对 heal 服务不可用,持续读取,恢复原盘 | 记录真实故障窗口和源/目标盘身份;在可恢复条件下最终补齐目标分片及完整 manifest,不只验证降级 GET。 |
-| B02 · `heal.disk-readonly-and-permission` | P1 | H1,H2 / DB / F4,F5 | 仅对独占测试卷制造只读/拒绝访问,恢复后重试 | 不可写目标不得 reported repaired;错误及待修责任可见;恢复后真实写入目标且不落到系统盘或其他目录。 |
-| B03 · `heal.disk-replacement-formatted` | P0 | H1,H2 / DB,DV / F8 | 换入空的新逻辑盘,确认新 disk UUID/正确格式后触发重建 | 新盘与旧盘身份可区分且属于正确 pool/set/slot;逐版本重建完成,不能从隔离备份复制回来冒充 heal。 |
-| B04 · `heal.disk-replacement-unformatted` | P1 | H1,H2 / DB / F8 | 引入空未格式化测试盘,并与带异集群格式的盘分别比较 | 按候选支持的初始化契约安全接纳或明确拒绝;不得覆盖外来身份或将未就绪盘标为可写健康。 |
-| B05 · `heal.multiple-shard-loss` | P0 | H1,H2 / DB,DV / F1 | 按实际同版本 EC 布局依次缺失 1..M 个 shard,覆盖 data/parity 角色组合 | 每个参数实例单独满足读取与写入 quorum;满足修复条件时全部目标重建,不把跨版本或跨 set 的 shard 合并。 |
-| B06 · `heal.read-quorum-boundary` | P0 | H1,H2 / DB / F1 | 保留恰好 R 个同版本权威 shard,测试读取及修复提交,再恢复写 quorum | 读取边界和修复写入边界分别判定;不能因还能 GET 就要求在写 quorum 不足时提交成功。 |
-| B07 · `heal.below-read-quorum` | P0 | H1,H2 / DB,DV / F1 | 使权威来源小于 R,执行 GET 与 deep heal,再恢复被隔离源 | 故障期间不得返回伪造的成功内容、空对象或误删记录;明确 unresolved/失败;恢复后原对象和版本完整。 |
-| B08 · `heal.below-write-quorum` | P0 | H1,H2 / DB,DL / F4 | 构造读 quorum 尚存而写/锁 quorum 不足,执行 heal 与新 PUT | 不能虚报持久化修复或成功写入;分别记录对象、元数据、分布式锁 quorum;恢复后核对所有已确认写入。 |
-| B09 · `heal.node-outage-rejoin` | P0 | H2 / DB,DV,DL / F6 | 停止同一跨节点 set 的一个节点,执行允许的降级 I/O,再重入 | 按实际丢失盘数判定可用性;缺失期间成功提交的对象补齐;节点重新 ready 不等于所有对象已重建。 |
-| B10 · `heal.disk-and-node-flapping` | P1 | H2 / DB,DL / F4,F6 | 重复短暂离线/在线并在重建途中再次失联 | 重试与队列有界、身份稳定,无重复任务风暴;最终故障解除后收敛,所有已确认对象保持正确。 |
+| B01 · `heal.disk-loss-and-repair` | P0 | H1,H2 / DB / F4 | Make one confirmed set-member disk unavailable to the heal service, keep reading, and restore the original disk | Record the real fault window and the source and target disk identities; under recoverable conditions, eventually fill in the target shards and the complete manifest, and do not verify only a degraded GET. |
+| B02 · `heal.disk-readonly-and-permission` | P1 | H1,H2 / DB / F4,F5 | Make only the exclusive test volume read-only or access-denied, and retry after recovery | An unwritable target must not be reported repaired; the error and the responsibility still awaiting repair are visible; after recovery, the target is actually written, and the write does not land on the system disk or in another directory. |
+| B03 · `heal.disk-replacement-formatted` | P0 | H1,H2 / DB,DV / F8 | Swap in an empty new logical disk, and trigger rebuild after confirming the new disk UUID and the correct format | The new disk and the old disk are distinguishable and belong to the correct pool/set/slot; rebuild completes version by version, and copying data back from the isolated backup cannot impersonate heal. |
+| B04 · `heal.disk-replacement-unformatted` | P1 | H1,H2 / DB / F8 | Introduce an empty unformatted test disk, and compare it separately with a disk that carries a foreign-cluster format | Safely admit or explicitly reject according to the initialization contract the candidate supports; do not overwrite a foreign identity, and do not mark a disk that is not ready as writable and healthy. |
+| B05 · `heal.multiple-shard-loss` | P0 | H1,H2 / DB,DV / F1 | Following the actual same-version EC layout, drop 1..M shards in turn, covering data and parity role combinations | Each parameter instance independently satisfies the read quorum and the write quorum; when the repair conditions are met, every target is rebuilt, and shards across versions or across sets are not merged. |
+| B06 · `heal.read-quorum-boundary` | P0 | H1,H2 / DB / F1 | Keep exactly R same-version authority shards, test the read and the repair commit, then restore write quorum | The read boundary and the repair-write boundary are judged separately; the fact that GET still succeeds must not be used to require a successful commit when write quorum is insufficient. |
+| B07 · `heal.below-read-quorum` | P0 | H1,H2 / DB,DV / F1 | Make the authority sources fewer than R, run GET and deep heal, then restore the isolated sources | During the fault, forged successful content, an empty object, or a mistakenly deleted record must not be returned; unresolved or failure is explicit; after recovery, the original object and versions are complete. |
+| B08 · `heal.below-write-quorum` | P0 | H1,H2 / DB,DL / F4 | Construct a state in which read quorum still exists but write and lock quorum are insufficient, and run heal and a new PUT | A persisted repair or a successful write must not be reported falsely; record the object, metadata, and distributed-lock quorums separately; after recovery, check every confirmed write. |
+| B09 · `heal.node-outage-rejoin` | P0 | H2 / DB,DV,DL / F6 | Stop one node of the same cross-node set, perform the allowed degraded I/O, then rejoin | Judge availability by the actual number of lost disks; objects successfully committed during the absence are filled in; a node becoming ready again does not mean every object has been rebuilt. |
+| B10 · `heal.disk-and-node-flapping` | P1 | H2 / DB,DL / F4,F6 | Repeatedly go briefly offline and online, and lose contact again during an in-flight rebuild | Retries and the queue are bounded, identity is stable, and there is no storm of duplicate tasks; after the fault is finally cleared, the system converges, and every confirmed object remains correct. |
-### C. 分片损坏、元数据与权威选择
+### C. Shard corruption, metadata, and authority selection
-| 编号 / 稳定 ID | P | 拓扑 / 夹具 / 故障 | 操作与触发 | 附加通过判据 |
+| ID / stable ID | P | Topology / fixture / fault | Operation and trigger | Additional pass criteria |
| --- | --- | --- | --- | --- |
-| C01 · `heal.missing-part-file` | P0 | H1,H2 / DB,DM / F1 | 仅隔离指定对象某个 part/shard,保留 format 与其他元数据,执行 deep heal | 对应数据或校验分片重新生成,part/版本身份准确;不得只看到 xl.meta 重现就通过。 |
-| C02 · `heal.data-shard-bitrot` | P0 | H1,H2 / DB,DM / F2 | 在指定 data shard 的已知偏移翻转字节,包含首/中/尾位置 | 故障前后摘要证明注入生效;deep 或 read-repair 发现损坏并修复;独立 bitrot/EC 及 S3 全内容验证通过。 |
-| C03 · `heal.parity-shard-bitrot` | P0 | H1,H2 / DB,DM / F2 | 损坏 parity shard,使普通 GET 仍可能成功,再执行 deep heal | 必须识别并重建损坏 parity;只有 GET 正确、无目标分片修复不能通过。 |
-| C04 · `heal.truncated-and-zeroed-shard` | P0 | H1,H2 / DB,DM / F2 | 分别截断 shard、置零和破坏校验头,保留正确元数据仲裁来源 | 短读、全零和校验损坏被准确识别;修复不能接受截断内容或错误长度。 |
-| C05 · `heal.missing-or-corrupt-xlmeta` | P0 | H1,H2 / DB,DV / F3 | 在少数成员上移走或损坏 xl.meta/inline 内容,保留足够权威来源 | 恢复同一对象的全部应存版本、数据引用与删除标记;不以新建空元数据替代真正修复。 |
-| C06 · `heal.stale-version-rejoin` | P0 | H1,H2 / DV,DL / F4 | 盘离线后执行覆盖/版本删除,再让携带旧元数据的盘重入 | 以已确认的新代次/版本为准;旧当前值与已删版本不复活,旧盘被纠正而非成为新的错误权威。 |
-| C07 · `heal.conflicting-subquorum-authority` | P0 | H1,H2 / DV / F3,F9 | 用受控原生夹具构造互不构成 quorum 的版本/代次集合,再做对应真机反例 | 禁止把不同版本的 shard 数量并集当成权威;无法证明时安全拒绝且保留可恢复证据,不猜测较新时间戳。 |
-| C08 · `heal.metadata-payload-mismatch` | P1 | H1,H2 / DB,DV / F2,F3 | 构造元数据声称某长度/摘要但实际 part 不匹配的少数副本 | 元数据与内容联合判断;不把可解析的 xl.meta 当健康;权威不足时拒绝覆盖正确副本。 |
-| C09 · `heal.foreign-format-and-unknown-schema` | P0 | H1,H2 / DB / F3,F8 | 提供外来 disk/set 身份、未来 schema 或不支持的记录格式 | 明确隔离/拒绝且不破坏原记录,不把未知当 missing 后直接重建;解除故障后原数据可用。 |
-| C10 · `heal.dangling-and-orphan-safety` | P1 | H1,H2 / DV,DM / F1,F3 | 构造 dangling part、旧 data_dir 与共享历史引用,分别 remove=false/true、宽限期前后检查 | 默认不得破坏仍可恢复数据;显式 remove 只清除具有充分删除证明且过期的本次孤儿,不删共享版本或 MPU。 |
-| C11 · `heal.bucket-incarnation-recreation` | P0 | H1,H2 / DV / F4,F9 | 盘离线时完成旧桶合法清空/删除,再同名建桶写入新对象,恢复旧盘与旧任务 | 旧 bucket incarnation 的 checkpoint/MRF/元数据不得污染新桶;旧任务不得删除或恢复新 incarnation 的对象。 |
+| C01 · `heal.missing-part-file` | P0 | H1,H2 / DB,DM / F1 | Isolate only one part or shard of the specified object, keep format and the other metadata, and run deep heal | The corresponding data or parity shard is regenerated, and the part and version identity are accurate; the case must not pass merely because xl.meta reappears. |
+| C02 · `heal.data-shard-bitrot` | P0 | H1,H2 / DB,DM / F2 | Flip bytes at a known offset of the specified data shard, including the head, middle, and tail | Digests before and after the fault prove that the injection took effect; deep or read-repair detects the corruption and repairs it; independent bitrot and EC verification, and full-content S3 verification, pass. |
+| C03 · `heal.parity-shard-bitrot` | P0 | H1,H2 / DB,DM / F2 | Corrupt a parity shard so that an ordinary GET may still succeed, then run deep heal | The corrupt parity must be identified and rebuilt; a correct GET with no repair of the target shard cannot pass. |
+| C04 · `heal.truncated-and-zeroed-shard` | P0 | H1,H2 / DB,DM / F2 | Separately truncate the shard, zero it, and corrupt the checksum header, while keeping a correct metadata arbitration source | Short reads, all-zero content, and checksum corruption are identified accurately; the repair must not accept truncated content or a wrong length. |
+| C05 · `heal.missing-or-corrupt-xlmeta` | P0 | H1,H2 / DB,DV / F3 | On a minority of members, remove or corrupt xl.meta or inline content, keeping enough authority sources | Restore every version of the same object that should be stored, together with its data references and delete markers; newly created empty metadata must not substitute for a real repair. |
+| C06 · `heal.stale-version-rejoin` | P0 | H1,H2 / DV,DL / F4 | After the disk goes offline, perform an overwrite or a version delete, then rejoin the disk that still carries the old metadata | The confirmed new generation or version is the authority; the old current value and deleted versions are not revived, and the old disk is corrected rather than becoming a new incorrect authority. |
+| C07 · `heal.conflicting-subquorum-authority` | P0 | H1,H2 / DV / F3,F9 | Use a controlled native fixture to construct version and generation sets that do not form a quorum with one another, then run the corresponding live negative case | The union of shard counts from different versions must not be treated as authority; when it cannot be proven, reject safely and retain recoverable evidence, and do not guess a newer timestamp. |
+| C08 · `heal.metadata-payload-mismatch` | P1 | H1,H2 / DB,DV / F2,F3 | Construct a minority of replicas whose metadata claims a length or digest that the actual part does not match | Judge metadata and content together; do not treat a parseable xl.meta as healthy; when authority is insufficient, refuse to overwrite a correct replica. |
+| C09 · `heal.foreign-format-and-unknown-schema` | P0 | H1,H2 / DB / F3,F8 | Provide a foreign disk or set identity, a future schema, or an unsupported record format | Isolate or reject explicitly, and do not damage the original record; do not treat unknown as missing and then rebuild directly; after the fault is cleared, the original data is usable. |
+| C10 · `heal.dangling-and-orphan-safety` | P1 | H1,H2 / DV,DM / F1,F3 | Construct a dangling part, an old data_dir, and a shared historical reference, and check remove=false/true, before and after the grace period | By default, data that can still be recovered must not be damaged; an explicit remove clears only this run's orphans that have sufficient delete proof and have expired, and does not delete a shared version or an MPU. |
+| C11 · `heal.bucket-incarnation-recreation` | P0 | H1,H2 / DV / F4,F9 | While the disk is offline, legally empty or delete the old bucket, then create a bucket of the same name and write new objects, and restore the old disk and the old task | The checkpoint, MRF, and metadata of the old bucket incarnation must not contaminate the new bucket; the old task must not delete or restore objects of the new incarnation. |
-### D. 对象类型与 S3 语义
+### D. Object types and S3 semantics
-| 编号 / 稳定 ID | P | 拓扑 / 夹具 / 故障 | 操作与触发 | 附加通过判据 |
+| ID / stable ID | P | Topology / fixture / fault | Operation and trigger | Additional pass criteria |
| --- | --- | --- | --- | --- |
-| D01 · `heal.object-size-boundaries` | P0 | H1,H2 / DB / F1,F2 | 覆盖 0/1 字节、页/inline/EC block 边界±1以及大对象 | 每个实际存储形态均确认后注入;长度、SHA-256、ETag、HEAD/LIST 与边界 Range 正确,无假设阈值。 |
-| D02 · `heal.completed-multipart-object` | P0 | H1,H2 / DM / F1,F2 | 修复已完成的不等长多 part 对象,分别损坏首/中/末 part | 完整内容、part 顺序、原版本/ETag/元数据和跨 part Range 不变;multipart ETag 不能当整对象 MD5。 |
-| D03 · `heal.historical-versions` | P0 | H1,H2 / DV / F1,F3 | 不同版本分布不同损坏位置,按对象和递归范围修复 | 所有原 versionId 可寻址且内容正确,最新版本身份不漂移,无多余版本或遗漏历史版本。 |
-| D04 · `heal.delete-marker-semantics` | P0 | H1,H2 / DV / F3,F4 | 缺失/陈旧 marker 副本后 heal,并按当前值与精确 marker versionId 查询 | 当前 GET/HEAD 与指定 marker 的状态码及身份头符合契约;marker 不复活数据,LIST versions 无重复。 |
-| D05 · `heal.versioning-suspended-null` | P1 | H1,H2 / DV / F1,F3 | 在版本开启、暂停、null 版本覆盖之间建夹具再修复 | null 和非 null 版本不混淆,不新增假版本;暂停前历史对象仍准确可读。 |
-| D06 · `heal.object-metadata-and-tags` | P1 | H1,H2 / DS / F1,F3 | 包含用户 metadata、Content-Type/Encoding/Disposition、tags 的对象修复 | 对外 metadata、tags 和相关响应头逐项相同;不要求无关内部元数据字节完全一致。 |
-| D07 · `heal.range-head-and-conditions` | P1 | H1,H2 / DB,DM / F1,F2 | 修复前后验证 HEAD、合法/越界 Range、If-Match/If-None-Match 及时间条件 | 200/206/304/412/416 等按逐请求契约判断,绝不返回旧内容、错误范围或错误 ETag。 |
-| D08 · `heal.encrypted-objects` | P0 | H1,H2 / DE / F1,F2 | 按候选支持的 SSE-S3/SSE-KMS/SSE-C 建加密对象并修复 | 授权解密后原内容及版本一致,物理验证使用正确编码/密文层;密钥不可用时不误删、不降级明文,证据脱敏。 |
-| D09 · `heal.object-lock-retention` | P0 | H1,H2 / DO,DV / F1,F3 | 对开启 retention/legal hold 的受保护版本修复,期间测试非法删除 | heal 不绕过保留/hold,也不丢失其元数据;拒绝删除与合法修复均有明确证据。 |
-| D10 · `heal.special-keys-and-pagination` | P1 | H1,H2 / DS / F1,F3 | 覆盖中文、空格、百分号、编码斜线、多层前缀和多页对象列表 | 逻辑 key 与物理路径映射准确,不跨桶/前缀;分页无遗漏重复,不能把用户 key 当可执行 shell 或裸文件路径。 |
-| D11 · `heal.bucket-metadata-integrity` | P0 | H1,H2 / DS,DV / F3 | 在隔离桶白名单内测试可支持的桶元数据恢复,并核对外部 bucket 配置 | 版本设置、策略/标签/保留配置按契约保持;不得遍历凭据配置或把未知内部记录当可重建数据。 |
+| D01 · `heal.object-size-boundaries` | P0 | H1,H2 / DB / F1,F2 | Cover 0/1 bytes, page, inline, and EC block boundaries ±1, and large objects | Inject only after each actual storage form has been confirmed; length, SHA-256, ETag, HEAD, LIST, and boundary Ranges are correct, with no assumed threshold. |
+| D02 · `heal.completed-multipart-object` | P0 | H1,H2 / DM / F1,F2 | Repair a completed multipart object with unequal part lengths, corrupting the first, middle, and last part separately | Full content, part order, the original version, ETag, metadata, and cross-part Ranges are unchanged; a multipart ETag must not be treated as the whole-object MD5. |
+| D03 · `heal.historical-versions` | P0 | H1,H2 / DV / F1,F3 | Place corruption at different positions across versions, and repair by object and by recursive scope | Every original versionId is addressable and its content is correct, the latest version identity does not drift, and there is no extra version and no omitted historical version. |
+| D04 · `heal.delete-marker-semantics` | P0 | H1,H2 / DV / F3,F4 | After marker replicas are missing or stale, heal, and query by the current value and by the exact marker versionId | The status codes and identity headers of the current GET and HEAD, and of the specified marker, conform to the contract; the marker does not revive data, and LIST versions has no duplicates. |
+| D05 · `heal.versioning-suspended-null` | P1 | H1,H2 / DV / F1,F3 | Build fixtures across versioning enabled, suspended, and null-version overwrite, then repair | null and non-null versions are not confused, and no spurious version is added; historical objects from before suspension remain accurately readable. |
+| D06 · `heal.object-metadata-and-tags` | P1 | H1,H2 / DS / F1,F3 | Repair objects that contain user metadata, Content-Type/Encoding/Disposition, and tags | External metadata, tags, and the related response headers match item by item; unrelated internal metadata is not required to be byte-for-byte identical. |
+| D07 · `heal.range-head-and-conditions` | P1 | H1,H2 / DB,DM / F1,F2 | Before and after repair, verify HEAD, in-range and out-of-range Range, If-Match/If-None-Match, and time conditions | 200/206/304/412/416 and similar codes are judged by the per-request contract, and old content, a wrong range, or a wrong ETag is never returned. |
+| D08 · `heal.encrypted-objects` | P0 | H1,H2 / DE / F1,F2 | Create encrypted objects for the SSE-S3/SSE-KMS/SSE-C modes the candidate supports, and repair them | After authorized decryption, the original content and version match, and physical verification uses the correct encoding and ciphertext layer; when the key is unavailable, do not delete by mistake and do not fall back to plaintext, and redact the evidence. |
+| D09 · `heal.object-lock-retention` | P0 | H1,H2 / DO,DV / F1,F3 | Repair protected versions that have retention or legal hold enabled, and test illegal deletion during that time | heal does not bypass retention or hold, and does not lose their metadata; both the rejected deletion and the legal repair have explicit evidence. |
+| D10 · `heal.special-keys-and-pagination` | P1 | H1,H2 / DS / F1,F3 | Cover Chinese, spaces, percent signs, encoded slashes, multi-level prefixes, and multi-page object listings | The mapping from logical key to physical path is accurate and does not cross buckets or prefixes; pagination has no omissions and no duplicates, and a user key must not be treated as executable shell or as a bare file path. |
+| D11 · `heal.bucket-metadata-integrity` | P0 | H1,H2 / DS,DV / F3 | Within the isolated-bucket allowlist, test recovery of supported bucket metadata, and check the external bucket configuration | Versioning settings and policy, tag, and retention configuration are kept according to the contract; credential configuration must not be walked, and an unknown internal record must not be treated as data that can be rebuilt. |
-### E. 触发路径与修复责任
+### E. Trigger paths and repair responsibility
-| 编号 / 稳定 ID | P | 拓扑 / 夹具 / 故障 | 操作与触发 | 附加通过判据 |
+| ID / stable ID | P | Topology / fixture / fault | Operation and trigger | Additional pass criteria |
| --- | --- | --- | --- | --- |
-| E01 · `heal.manual-normal-attribution` | P1 | H1,H2 / DB / F1 | 禁止预读目标,保留 format,隔离背景干扰后显式 normal heal | 证明本次 Admin token 实际覆盖目标并修复可由 normal 检出的缺失;不要求 normal 发现所有静默 bitrot。 |
-| E02 · `heal.manual-deep-attribution` | P0 | H1,H2 / DB / F2 | 在普通读取仍可成功的 parity 损坏上显式 deep heal | 按 token/对象/版本及物理变更证明 deep 检查和修复,不把后台提前修复算给手工任务。 |
-| E03 · `heal.read-repair-attribution` | P0 | H1,H2 / DB,DV / F1,F2 | 不发 Admin heal,对已确认的特定损坏对象发一次可追踪 GET/Range | 读取正确且后续真实补片;若走异步队列需证实 source=read-repair,不能只有 GET 成功。 |
-| E04 · `heal.scanner-cold-object-repair` | P0 | H1,H2 / DB / F1,F2 | 不 GET/HEAD 损坏目标、不发手工 heal,仅等待已启用的 scanner 路径 | 冷对象被发现并修复,来源和遍历窗口明确;scanner 周期结束但目标没检查不能通过。 |
-| E05 · `heal.mrf-offline-write-replay` | P0 | H2 / DL,DV / F4 | 成员离线时完成有 quorum 的 PUT/DELETE,确认待修责任后重入 | durable MRF 责任持久且身份精确;只在 repaired/verified healthy/authoritatively absent 等已证明结果后释放,成功提交不丢失。 |
-| E06 · `heal.replacement-auto-rebuild` | P0 | H1,H2 / DB,DV / F8 | 替换盘接纳后不发 Admin/目标 GET,观察自动 replacement heal | 替换代次、ready/completion 与盘上全部目标版本一致;不能用服务 ready、队列空或计数重置代替完成。 |
+| E01 · `heal.manual-normal-attribution` | P1 | H1,H2 / DB / F1 | Forbid pre-reading the target, keep format, isolate background interference, then run an explicit normal heal | Prove that this Admin token actually covered the target and repaired the absence that normal can detect; normal is not required to discover every silent bitrot. |
+| E02 · `heal.manual-deep-attribution` | P0 | H1,H2 / DB / F2 | Run an explicit deep heal on parity corruption for which an ordinary read can still succeed | Prove the deep check and the repair by token, object, version, and the physical change, and do not attribute a background repair that finished earlier to the manual task. |
+| E03 · `heal.read-repair-attribution` | P0 | H1,H2 / DB,DV / F1,F2 | Do not send an Admin heal; send one traceable GET or Range to a confirmed, specifically corrupt object | The read is correct and a real on-disk shard repair follows; if it goes through an asynchronous queue, source=read-repair must be confirmed, and a successful GET alone is not enough. |
+| E04 · `heal.scanner-cold-object-repair` | P0 | H1,H2 / DB / F1,F2 | Do not GET or HEAD the corrupt target, and do not send a manual heal; only wait for the enabled scanner path | The cold object is discovered and repaired, and the source and the traversal window are explicit; a scanner cycle that ends without the target having been checked cannot pass. |
+| E05 · `heal.mrf-offline-write-replay` | P0 | H2 / DL,DV / F4 | While a member is offline, complete a PUT or DELETE that has quorum, confirm the responsibility still awaiting repair, then rejoin | The durable MRF responsibility is persistent and its identity is exact; it is released only after a proven result such as repaired/verified healthy/authoritatively absent, and a successful commit is not lost. |
+| E06 · `heal.replacement-auto-rebuild` | P0 | H1,H2 / DB,DV / F8 | After the replacement disk is admitted, do not send an Admin request or a target GET; observe automatic replacement heal | The replacement generation, ready and completion, and every target version on disk agree; service ready, an empty queue, or a counter reset cannot substitute for completion. |
-### F. 中断、持久化、恢复与升级
+### F. Interruption, persistence, recovery, and upgrade
-| 编号 / 稳定 ID | P | 拓扑 / 夹具 / 故障 | 操作与触发 | 附加通过判据 |
+| ID / stable ID | P | Topology / fixture / fault | Operation and trigger | Additional pass criteria |
| --- | --- | --- | --- | --- |
-| F01 · `heal.root-graceful-coordinator-restart` | P0 | H1,H2 / DX / F6 | root heal 非零进度时优雅重启原协调者,保留原拓扑与记录盘 | 在明确支持的边界内恢复原 task ID 和剩余责任/预算,完成后无遗留活动记录;不扩张为协调盘丢失后的故障转移。 |
-| F02 · `heal.target-graceful-restart` | P0 | H2 / DB,DV,DX / F6,F8 | 替换目标仅完成部分重建时优雅重启目标节点 | 故障确实发生于部分完成窗口;重启后续建同一目标盘/代次,最终 manifest 与全部目标分片正确。 |
-| F03 · `heal.target-process-crash` | P0 | H2 / DB,DV,DX / F6,F8 | 部分重建时 SIGKILL 测试进程,再正常启动 | unclean shutdown 与修复进度有真实证据;已提交对象不丢失,未完成目标不会被当完整;明确不是断电测试。 |
-| F04 · `heal.coordinator-process-crash` | P0 | H2 / DB,DX / F6 | 协调者非优雅退出并恢复,分别观察 root、MRF、replacement | 按各机制已核实的持久化边界判断;不得把丢失的易失 token 当成功,需显式记录剩余责任并可安全重扫恢复。 |
-| F05 · `heal.full-cluster-restart` | P0 | H2 / DB,DV,DM / F6 | 有未完成修复责任时重启全部测试节点,固定原拓扑/磁盘身份 | 有持久化保证的责任不遗失;全部端点恢复一致,旧版本/marker/MPU 与目标分片完整,重复重启不扩散错误。 |
-| F06 · `heal.cancelled-task-restart` | P0 | H1,H2 / DB,DX / F6 | 完成取消及在途退出后重启,随后 forceStart/新任务正常修复 | 已取消的任务不自动复活;旧 checkpoint/token 不能取消或覆盖新任务,旧责任不会误标为修复完成。 |
-| F07 · `heal.checkpoint-and-mrf-corruption` | P0 | H1,H2 / DB,DV / F9 | 受控构造 checkpoint/MRF 截断、坏校验、未来 schema、陈旧代次及写入失败 | 已确认持久化责任不得静默删除或伪 ACK;未知记录保留且可诊断;精确崩溃窗口由原生屏障证明,不伪称真机命中。 |
-| F08 · `heal.mixed-version-upgrade-rollback` | P1 | H2 / DB,DV / F6,F9 | 固定前后两个明确兼容候选,覆盖滚动升级、混合 peer、回滚与重启 | 能力不足必须显示 partial/unsupported 而非完整成功;按格式兼容契约保留责任,旧版本不能误退休新版本记录。 |
-| F09 · `heal.dedicated-host-forced-restart` | P1 | H6 / DB,DV,DX / F10 | 经专项授权在部分修复/持续写入时强制重启独占测试 VM/主机 | 保留平台事件与 ACK ledger;验证落盘和恢复,不把 SIGKILL 当断电;VM 强制停止也不等同于物理控制器断电。 |
+| F01 · `heal.root-graceful-coordinator-restart` | P0 | H1,H2 / DX / F6 | At non-zero progress of a root heal, gracefully restart the original coordinator, keeping the original topology and the record disk | Within the explicitly supported boundary, restore the original task ID and the remaining responsibility and budget, and leave no leftover active record after completion; do not expand this into failover after the coordinator disk is lost. |
+| F02 · `heal.target-graceful-restart` | P0 | H2 / DB,DV,DX / F6,F8 | When the replacement target has completed only a partial rebuild, gracefully restart the target node | The fault actually occurs inside the partially complete window; after the restart, building continues for the same target disk and generation, and the final manifest and all target shards are correct. |
+| F03 · `heal.target-process-crash` | P0 | H2 / DB,DV,DX / F6,F8 | SIGKILL the test process during a partial rebuild, then start it normally | There is real evidence of the unclean shutdown and of the repair progress; committed objects are not lost, and an unfinished target is not treated as complete; this is explicitly not a power-loss test. |
+| F04 · `heal.coordinator-process-crash` | P0 | H2 / DB,DX / F6 | The coordinator exits ungracefully and then recovers; observe root, MRF, and replacement separately | Judge by each mechanism's verified persistence boundary; a lost volatile token must not be treated as success, the remaining responsibility must be recorded explicitly, and recovery by a safe rescan must be possible. |
+| F05 · `heal.full-cluster-restart` | P0 | H2 / DB,DV,DM / F6 | With unfinished repair responsibility, restart every test node, holding the original topology and disk identity fixed | Responsibility that has a persistence guarantee is not lost; every endpoint recovers to a consistent state, old versions, markers, and MPUs and the target shards are complete, and a repeated restart does not spread errors. |
+| F06 · `heal.cancelled-task-restart` | P0 | H1,H2 / DB,DX / F6 | Restart after cancellation and in-flight exit are complete, then let forceStart or a new task repair normally | A cancelled task does not revive automatically; an old checkpoint or token cannot cancel or overwrite the new task, and old responsibility is not mislabeled as repair complete. |
+| F07 · `heal.checkpoint-and-mrf-corruption` | P0 | H1,H2 / DB,DV / F9 | Under control, construct checkpoint and MRF truncation, a bad checksum, a future schema, a stale generation, and a write failure | Confirmed persisted responsibility must not be silently deleted or falsely ACKed; unknown records are retained and can be diagnosed; the exact crash window is proven by a native barrier, and is not falsely claimed as a live hit. |
+| F08 · `heal.mixed-version-upgrade-rollback` | P1 | H2 / DB,DV / F6,F9 | Pin two explicitly compatible candidates, before and after, covering rolling upgrade, mixed peers, rollback, and restart | Insufficient capability must be shown as partial/unsupported rather than as full success; responsibility is retained according to the format-compatibility contract, and an older version must not mistakenly retire a newer version's records. |
+| F09 · `heal.dedicated-host-forced-restart` | P1 | H6 / DB,DV,DX / F10 | Under dedicated authorization, force-restart the exclusive test VM or host during a partial repair or continuous writes | Retain platform events and the ACK ledger; verify data persisted to disk and the recovery, and do not treat SIGKILL as a power loss; a forced VM stop is also not equivalent to a physical controller power loss. |
-### G. 与前台和后台操作并发
+### G. Concurrency with foreground and background operations
-| 编号 / 稳定 ID | P | 拓扑 / 夹具 / 故障 | 操作与触发 | 附加通过判据 |
+| ID / stable ID | P | Topology / fixture / fault | Operation and trigger | Additional pass criteria |
| --- | --- | --- | --- | --- |
-| G01 · `heal.race-unversioned-overwrite` | P0 | H1,H2 / DL / F1,F2,F9 | 同 key 连续覆盖与 heal 同时推进,包含提交尾部清理窗口 | 按可证明的提交顺序验证最终最后一次成功写入;不能回滚到旧 data_dir,未确认超时写入单独解析。 |
-| G02 · `heal.race-versioned-put` | P0 | H1,H2 / DV,DL / F1,F9 | 修复历史版本时创建新版本并重复读取各 versionId | 新旧版本不混片、不串内容;已返回成功的每个版本都保留,最新值按确定的先后关系验收。 |
-| G03 · `heal.race-version-delete` | P0 | H1,H2 / DV / F1,F2,F9 | 对损坏的精确历史 versionId 发 DELETE,与 deep heal 竞争 | 已确认删除的版本不能被复活,其他历史版本不受影响;原生临界点测试与真机竞争分别留证。 |
-| G04 · `heal.race-markers-and-batch-delete` | P0 | H1,H2 / DV / F3,F9 | 与创建/删除 marker、批量版本删除和无版本删除分别交错 | 按逐项 DELETE 回执核对版本/marker 集合,无复活、误删或遗漏批量错误。 |
-| G05 · `heal.race-multipart-complete-abort` | P0 | H1,H2 / DM,DL / F1,F9 | heal 期间对不同已知 UploadId 分别续传、Complete、Abort | 成功 Complete 内容只出现一次;Abort 不破坏已提交对象或别的 upload;未完成 part 不被伪造为完整对象。 |
-| G06 · `heal.race-copy-and-metadata-update` | P1 | H1,H2 / DS,DL / F1,F9 | COPY 源/目标和标签/元数据更新与修复交错 | 最终对象内容与对应提交的 metadata 一致,读修复发现新代次应重验/放弃陈旧提交而非覆盖。 |
-| G07 · `heal.race-ilm-expiry-and-transition` | P0 | H2 / DV,DE / F1,F9,F11 | 分别与当前/历史版本过期、远端 transition/restore 同窗口执行 | 已授权过期不被恢复;迁移状态与远端版本引用正确;只有实际观察到交叠才计并发覆盖,tier 未配置需明确阻断对应实例。 |
-| G08 · `heal.race-replication-resync` | P1 | H2 / DV,DL / F1,F9,F11 | 在隔离复制两端与 heal 同时运行新增/删除/重同步 | 对端版本、删除语义及源数据最终一致,无无限回环或重复传播;不得依赖仅源端 GET 宣称复制正确。 |
+| G01 · `heal.race-unversioned-overwrite` | P0 | H1,H2 / DL / F1,F2,F9 | Continuous overwrites of the same key proceed together with heal, including the commit-tail cleanup window | Verify the final last successful write by a provable commit order; do not roll back to an old data_dir, and resolve unconfirmed timed-out writes separately. |
+| G02 · `heal.race-versioned-put` | P0 | H1,H2 / DV,DL / F1,F9 | While repairing a historical version, create a new version and repeatedly read each versionId | Old and new versions do not mix shards or cross their content; every version that returned success is retained, and the latest value is accepted according to a determined order. |
+| G03 · `heal.race-version-delete` | P0 | H1,H2 / DV / F1,F2,F9 | DELETE a corrupt exact historical versionId, racing with deep heal | A version whose deletion has been confirmed cannot be revived, and other historical versions are unaffected; native critical-point tests and live races are evidenced separately. |
+| G04 · `heal.race-markers-and-batch-delete` | P0 | H1,H2 / DV / F3,F9 | Interleave separately with creating and deleting a marker, batch version delete, and unversioned delete | Check the version and marker set against per-item DELETE receipts, with no revival, no mistaken deletion, and no omitted batch error. |
+| G05 · `heal.race-multipart-complete-abort` | P0 | H1,H2 / DM,DL / F1,F9 | During heal, resume, Complete, and Abort different known UploadIds separately | The content of a successful Complete appears only once; Abort does not damage a committed object or another upload; an unfinished part is not forged into a complete object. |
+| G06 · `heal.race-copy-and-metadata-update` | P1 | H1,H2 / DS,DL / F1,F9 | Interleave COPY of source and target, and tag and metadata updates, with repair | The final object content matches the metadata of the corresponding commit; when read-repair discovers a new generation, it re-verifies or abandons the stale commit rather than overwriting it. |
+| G07 · `heal.race-ilm-expiry-and-transition` | P0 | H2 / DV,DE / F1,F9,F11 | Run, in the same window and separately, with current and historical version expiry and with remote transition and restore | An expiry that was authorized is not restored; transition state and remote version references are correct; concurrent coverage is counted only when overlap is actually observed, and an unconfigured tier must explicitly block the corresponding instance. |
+| G08 · `heal.race-replication-resync` | P1 | H2 / DV,DL / F1,F9,F11 | On the two isolated replication ends, run create, delete, and resync at the same time as heal | The peer's versions, delete semantics, and the source data are eventually consistent, with no infinite loop and no duplicate propagation; replication must not be declared correct on the basis of a source-side GET alone. |
-### H. 多 Pool、移动与多 set 隔离
+### H. Multiple pools, movement, and multi-set isolation
-| 编号 / 稳定 ID | P | 拓扑 / 夹具 / 故障 | 操作与触发 | 附加通过判据 |
+| ID / stable ID | P | Topology / fixture / fault | Operation and trigger | Additional pass criteria |
| --- | --- | --- | --- | --- |
-| H01 · `heal.with-expansion-rebalance` | P0 | H5 / DB,DV / F1,F2 | 先证明源 pool 持有夹具,再扩容并与修复同时 rebalance | 修复目标不得指向错误 pool/set;新旧数据完整、移动终态正确,保留 scanner backlog 提交权威检查。 |
-| H02 · `heal.with-rebalance-stop-resume` | P1 | H5 / DB,DL / F1,F6 | heal 活动期间 stop/restart/resume rebalance | 两类任务状态和身份独立;停止/恢复移动不误终止修复或丢失对象,恢复后两者均满足各自契约。 |
-| H03 · `heal.with-decommission` | P0 | H5 / DB,DV,DM / F1,F2 | 分别在待退役源和存活目标制造可恢复缺片,再执行退役 | 不得提前删源或以临时副本冒充迁移成功;按 Pool 既定写入限制验收;只剩 survivor 后仍完整可读且有物理证明。 |
-| H04 · `heal.with-decommission-cancel-clear` | P0 | H5 / DB,DL / F1,F9 | heal/移动争用中取消退役、Clear、恢复,并覆盖容量锁超时恢复 | 保持持久化后通知及在途退出顺序;不误取消新代次;对 #2423 的重试窗口单独注明真机是否实际触发,不只看最终绿色。 |
-| H05 · `heal.after-retired-pool-removal` | P0 | H5 / DB,DV / F1,F3 | 源 pool 完全退役移除并停机后,只在 survivor 上触发修复 | 不再向已移除源写入或依赖其兜底;来源权威和版本正确;#2422 路径不能出现无权威覆盖或错误回退。 |
-| H06 · `heal.multi-pool-target-isolation` | P0 | H5 / DB,DV / F1,F4 | 仅破坏选定 pool/set,并在另一 pool 保留不同对象与控制故障 | 选择器、任务、容量/配额和目标 drive 一致;未选 pool 不受错误修复,跨 pool 存在不等于该目标已修复。 |
-| H07 · `heal.ec84-multi-set-restart` | P0 | H3,H4 / DB,DV,DX / F6,F8 | 固定 EC8+4 单 set 与至少两 set,部分重建时重启/崩溃目标 | 每个 set 的故障覆盖和重建分别证明,不跨 set 拼 shard;必须满足真实盘数及映射,不能用 4 盘用例替代 EC8+4。 |
+| H01 · `heal.with-expansion-rebalance` | P0 | H5 / DB,DV / F1,F2 | First prove that the source pool holds the fixture, then perform expansion and rebalance at the same time as repair | The repair target must not point at the wrong pool or set; old and new data are complete and the move terminal state is correct, and the scanner-backlog commit-authority check is retained. |
+| H02 · `heal.with-rebalance-stop-resume` | P1 | H5 / DB,DL / F1,F6 | stop, restart, and resume rebalance while heal is active | The two kinds of tasks have independent state and identity; stopping or resuming movement does not mistakenly terminate repair or lose objects, and after recovery both satisfy their own contracts. |
+| H03 · `heal.with-decommission` | P0 | H5 / DB,DV,DM / F1,F2 | Separately create a recoverable missing shard on the source pending decommission and on the surviving target, then decommission | Do not delete the source early, and do not impersonate a successful migration with a temporary replica; accept against the Pool's established write limits; after only the survivor remains, the data is still fully readable and has physical proof. |
+| H04 · `heal.with-decommission-cancel-clear` | P0 | H5 / DB,DL / F1,F9 | During heal and move contention, cancel decommission, Clear, and recover, and cover capacity-lock timeout recovery | Preserve the order of notification after persistence and of in-flight exit; do not mistakenly cancel a new generation; for the #2423 retry window, note separately whether it was actually triggered live, and do not look only at a final green. |
+| H05 · `heal.after-retired-pool-removal` | P0 | H5 / DB,DV / F1,F3 | After the source pool has been fully decommissioned, removed, and shut down, trigger repair only on the survivor | Do not write to the removed source again, and do not rely on it as a fallback; source authority and versions are correct; the #2422 path must not exhibit an overwrite without authority or an incorrect fallback. |
+| H06 · `heal.multi-pool-target-isolation` | P0 | H5 / DB,DV / F1,F4 | Corrupt only the selected pool or set, and keep different objects and a control fault in another pool | The selector, the task, capacity and quota, and the target drive agree; an unselected pool is not repaired incorrectly, and the existence of another pool does not mean that this target has been repaired. |
+| H07 · `heal.ec84-multi-set-restart` | P0 | H3,H4 / DB,DV,DX / F6,F8 | Pin EC8+4 as a single set and as at least two sets, and restart or crash the target during a partial rebuild | Fault coverage and rebuild are proven separately for each set, and shards are not assembled across sets; the real disk count and mapping must be satisfied, and a 4-disk case cannot substitute for EC8+4. |
-### I. 资源、依赖故障与调度
+### I. Resources, dependency faults, and scheduling
-| 编号 / 稳定 ID | P | 拓扑 / 夹具 / 故障 | 操作与触发 | 附加通过判据 |
+| ID / stable ID | P | Topology / fixture / fault | Operation and trigger | Additional pass criteria |
| --- | --- | --- | --- | --- |
-| I01 · `heal.slow-io-and-eio` | P1 | H1,H2 / DB,DX / F5 | 在独占故障层注入有界延迟、短读、EIO,解除后观察恢复 | 暂态与永久错误分开,重试有界且无忙循环;解除故障后实际修复,期间不得提前标健康。 |
-| I02 · `heal.enospc-inodes-and-quota` | P0 | H1,H2 / DB,DM / F5,F11 | 限制专用测试卷空间/inode,并参数化本次桶配额 | 仅填专用受限卷、不耗尽共享真实盘;失败不破坏好副本或释放尚需容量,释放空间后正确续修。 |
-| I03 · `heal.network-partition-and-lock-quorum` | P0 | H2 / DB,DV,DL / F7 | 仅隔离测试服务节点间流量,覆盖少数派、读/写/锁 quorum 边界及恢复 | 少数派不得承诺无安全性的修复或写入;cluster coverage 不完整不能显示全局健康;恢复无分叉、跨代覆盖。 |
-| I04 · `heal.queue-pressure-and-admission` | P1 | H1,H2 / DX / F11 | 多来源请求冲击有限队列,混合重复任务、不同优先级和多个桶 | Full/Dropped/Merged 语义与实际任务一致;丢弃不能丢 durable 责任;在预先声明负载窗口内无无限堆积或健康误报。 |
-| I05 · `heal.lock-timeout-and-caller-abort` | P0 | H1,H2 / DL,DV / F5,F9 | 真实对象锁争用超过预算及调用方断开,再释放锁恢复 | 取消/超时不得提前释放在途所有权或重复提交;陈旧 read-repair 必须重验代次;恢复后无泄漏和错误覆盖。 |
-| I06 · `heal.kms-and-tier-outage` | P1 | H2 / DE,DV / F11 | 仅让已分配测试 KMS/远端 tier 暂不可用,持续查询与修复 | 依赖故障不得解释为对象不存在/可删/已修复,不暴露密钥;恢复后原版本可解密或恢复,责任未丢失。 |
+| I01 · `heal.slow-io-and-eio` | P1 | H1,H2 / DB,DX / F5 | Inject bounded delay, short reads, and EIO in the exclusive fault layer, and observe recovery after the fault is cleared | Transient and permanent errors are separated, retries are bounded, and there is no busy loop; after the fault is cleared there is an actual repair, and health must not be marked early while the fault is in effect. |
+| I02 · `heal.enospc-inodes-and-quota` | P0 | H1,H2 / DB,DM / F5,F11 | Limit space and inodes of the dedicated test volume, and parameterize this run's bucket quota | Fill only the dedicated restricted volume, and do not exhaust the shared real disk; a failure does not damage a good replica or release capacity that is still needed, and repair continues correctly after space is released. |
+| I03 · `heal.network-partition-and-lock-quorum` | P0 | H2 / DB,DV,DL / F7 | Isolate only inter-node traffic of the test service, covering the minority, the read, write, and lock quorum boundaries, and recovery | A minority must not promise a repair or a write that lacks safety; incomplete cluster coverage cannot be displayed as global health; recovery has no split and no cross-generation overwrite. |
+| I04 · `heal.queue-pressure-and-admission` | P1 | H1,H2 / DX / F11 | Requests from multiple sources hit a limited queue, mixing duplicate tasks, different priorities, and multiple buckets | Full/Dropped/Merged semantics agree with the actual tasks; a drop cannot drop durable responsibility; within the pre-declared load window there is no unbounded accumulation and no false report of health. |
+| I05 · `heal.lock-timeout-and-caller-abort` | P0 | H1,H2 / DL,DV / F5,F9 | Real object-lock contention exceeds the budget and the caller disconnects, then the lock is released and the system recovers | Cancel or timeout must not release in-flight ownership early or commit twice; a stale read-repair must re-verify the generation; after recovery there is no leak and no incorrect overwrite. |
+| I06 · `heal.kms-and-tier-outage` | P1 | H2 / DE,DV / F11 | Make only the allocated test KMS or remote tier temporarily unavailable, and keep querying and repairing | A dependency fault must not be interpreted as the object not existing, being deletable, or already repaired, and keys are not exposed; after recovery the original version can be decrypted or restored, and responsibility has not been lost. |
-### J. 可观测性、幂等、长稳与性能
+### J. Observability, idempotency, soak, and performance
-| 编号 / 稳定 ID | P | 拓扑 / 夹具 / 故障 | 操作与触发 | 附加通过判据 |
+| ID / stable ID | P | Topology / fixture / fault | Operation and trigger | Additional pass criteria |
| --- | --- | --- | --- | --- |
-| J01 · `heal.status-partial-disabled-unknown` | P0 | H1,H2 / DB / F0,F6 | 分别检查 enabled/disabled/uninitialized、全体可达及缺失 peer 的状态 | idle 只在覆盖完整且真实无活动时成立;partial/unknown/degraded/disabled 不被客户端转换为全部完成。 |
-| J02 · `heal.outcome-counters-and-error-truth` | P0 | H1,H2 / DB,DV / F1,F2 | 同批次混合健康、可修、不可修、deferred、dry-run 对象 | execution/coverage/disposition/计数和逐对象物理结果相符;unknown、skipped 或 completed_with_errors 不能算修复成功。 |
-| J03 · `heal.incremental-cursor-and-truncation` | P0 | H1,H2 / DS,DX / F1 | 生成超过结果保留上限的任务,分页轮询 sinceSeq,并跨节点/终态读取 | nextSeq/minSeq 单调且 gap/truncation 明示;有界 items 不是完整责任清单,最终须用独立 manifest/计数补证,不静默丢错误。 |
-| J04 · `heal.repeat-idempotency` | P1 | H1,H2 / DB,DV / F1 | 第一次修复后重复 normal/deep、重启再检查 | versioned 首次允许所有 identity repaired 或至少一个 repaired 加其余正健康态,marker 仅 repaired/metadata_healthy,计数必须与 receipt 集合一致;normal repeat 接受 `metadata_healthy`,deep repeat 要求 `verified_healthy`,repeat marker 为 `metadata_healthy` 且 healed=0;unknown/skipped/failed 不得成为 PASS。 |
-| J05 · `heal.long-soak` | P2 | H2,H5 / DX,DL,DV / F4,F6,F11 | 独占环境运行预注册 24h/72h 窗口,混合修复、掉线、写入和版本删除 | 每次 ACK 可追踪;最终全量 manifest 正确,队列/内存/FD/临时空间无无界增长,无后台 panic 或未解释持久责任。 |
-| J06 · `heal.performance-abba` | P2 | H2 / DX,DL / F1,F11 | 同硬件同候选按 ABBA 运行无 heal 基线和有 heal 负载,重复并固定限速 | 分别报告修复吞吐、GET/PUT P95/P99、错误率和 CPU/IO;阈值事先冻结,不能事后挑轮次或把功能 PASS 充性能验收。 |
+| J01 · `heal.status-partial-disabled-unknown` | P0 | H1,H2 / DB / F0,F6 | Separately check the status of enabled/disabled/uninitialized, of every peer reachable, and of a missing peer | idle holds only when coverage is complete and there is truly no activity; partial/unknown/degraded/disabled are not converted by the client into fully complete. |
+| J02 · `heal.outcome-counters-and-error-truth` | P0 | H1,H2 / DB,DV / F1,F2 | In the same batch, mix healthy, repairable, unrepairable, deferred, and dry-run objects | execution, coverage, disposition, and the counters agree with the per-object physical results; unknown, skipped, or completed_with_errors cannot count as a successful repair. |
+| J03 · `heal.incremental-cursor-and-truncation` | P0 | H1,H2 / DS,DX / F1 | Generate a task that exceeds the result retention limit, poll sinceSeq page by page, and read across nodes and terminal states | nextSeq and minSeq are monotonic, and gap and truncation are stated explicitly; bounded items are not a complete responsibility manifest, the end must be corroborated by an independent manifest and count, and errors are not dropped silently. |
+| J04 · `heal.repeat-idempotency` | P1 | H1,H2 / DB,DV / F1 | After the first repair, repeat normal and deep, restart, and check again | The first versioned run allows every identity repaired, or at least one repaired plus the rest in positive healthy states; a marker is only repaired/metadata_healthy; counts must agree with the receipt set; a normal repeat accepts `metadata_healthy`, a deep repeat requires `verified_healthy`, and a repeat marker is `metadata_healthy` with healed=0; unknown/skipped/failed must not become PASS. |
+| J05 · `heal.long-soak` | P2 | H2,H5 / DX,DL,DV / F4,F6,F11 | On an exclusive environment, run a pre-registered 24h/72h window, mixing repair, dropouts, writes, and version deletes | Every ACK is traceable; the final full manifest is correct, queue, memory, FDs, and temporary space do not grow without bound, and there is no background panic and no unexplained persisted responsibility. |
+| J06 · `heal.performance-abba` | P2 | H2 / DX,DL / F1,F11 | On the same hardware and the same candidate, run ABBA with a no-heal baseline and a with-heal load, repeated and with a fixed rate limit | Report repair throughput, GET and PUT P95/P99, error rate, and CPU/IO separately; thresholds are frozen beforehand, rounds must not be cherry-picked afterward, and a functional PASS must not stand in for performance acceptance. |
-## 与主仓库既有验证的复用关系
+## Reuse relationship with existing main-repository validation
-以相同源 SHA 的实现为准,先读脚本的 scope、环境要求和故障 oracle,再适配到固定 fleet;**不直接在共享机器运行原 E2E 的清盘/建进程脚本**。已有覆盖只是复用线索,不能直接标记本目录 planned 用例为 implemented。
+Take the implementation at the same source SHA as the authority. First read the script's scope, environment requirements, and fault oracle, then adapt them to the fixed fleet. **Do not run the original E2E disk-wipe and process-creation scripts directly on a shared machine.** Existing coverage is only a clue for reuse, and it cannot directly mark a planned case in this directory as implemented.
-| 已有资产 | 可复用内容 / 本矩阵对应 | 不能混淆的边界 |
+| Existing asset | Reusable content / correspondence in this matrix | Boundaries that must not be confused |
| --- | --- | --- |
-| `crates/e2e_test/src/heal_erasure_disk_rebuild_test.rs` | runtime wiped disk、Admin deep、remote replacement,B/C/E/F | 进程 E2E 与固定真机、同节点目录盘与独立 XFS 物理盘需分别标记;检查其 skip 分支,必选故障能力缺失不能通过。 |
-| registry 的 background-target-restart / crash | F02/F03、目标部分进度与 replacement shard 证据 | SIGKILL 不是断电;原夹具无版本覆盖不能自动扩张为 DV 已覆盖。 |
-| registry 的 EC8+4 single-set / multi-set / multi-pool 项 | H07/H06 与相应 F02/F03 参数 | 精确盘数、set/pool 分布与同一故障窗口要匹配,4 盘 PASS 不能覆盖 G14。 |
-| W13 durable MRF 与 W16 recovery-intent/quota 证据 | E05、F07、I02、J02 | 责任 ACK、恢复意图、配额与终态内容都需要原始实测输入,不只运行 JSON assembler。 |
-| G05/G06 status/outcome、G02 checkpoint、G09 upgrade 证据生产脚本 | A03、J01-J03、F07/F08 | schema 自检、descriptor 组装和真实故障执行是三种不同证据,不可互相代替。 |
-| ECStore/heal 原生 runtime-fence、版本删除/覆盖竞争回归 | A09、C07、G01-G06、I05 | 原生可控屏障覆盖精确窗口;没有在 fleet 触发该窗口时只记录功能覆盖,不把两类日志混为真机因果证明。 |
+| `crates/e2e_test/src/heal_erasure_disk_rebuild_test.rs` | runtime wiped disk, Admin deep, remote replacement, B/C/E/F | Process E2E and the fixed live fleet, and same-node directory disks and independent XFS physical disks, must be marked separately; check its skip branches, and a missing required fault capability cannot pass. |
+| registry background-target-restart / crash | F02/F03, target partial progress, and replacement shard evidence | SIGKILL is not a power loss; an original fixture with no version coverage cannot be automatically expanded into DV already covered. |
+| registry EC8+4 single-set / multi-set / multi-pool items | H07/H06 and the corresponding F02/F03 parameters | Exact disk count, set and pool distribution, and the same fault window must match; a 4-disk PASS cannot cover G14. |
+| W13 durable MRF and W16 recovery-intent/quota evidence | E05, F07, I02, J02 | Responsibility ACKs, recovery intent, quota, and terminal-state content all require raw measured inputs, not only running a JSON assembler. |
+| G05/G06 status/outcome, G02 checkpoint, and G09 upgrade evidence-production scripts | A03, J01-J03, F07/F08 | Schema self-check, descriptor assembly, and real fault execution are three different kinds of evidence, and they cannot substitute for one another. |
+| ECStore/heal native runtime-fence, and version-delete and overwrite race regressions | A09, C07, G01-G06, I05 | A native controllable barrier covers the exact window; when that window was not triggered on the fleet, record only functional coverage, and do not mix the two kinds of logs into a live causal proof. |
-来源:[必选 E2E registry](https://github.com/rustfs/rustfs/blob/940e221988694da33fe005054bb86086bae88cd6/.config/scanner-heal-required-tests.json)、[磁盘重建 E2E](https://github.com/rustfs/rustfs/blob/940e221988694da33fe005054bb86086bae88cd6/crates/e2e_test/src/heal_erasure_disk_rebuild_test.rs)、[Linux evidence plan](https://github.com/rustfs/rustfs/blob/940e221988694da33fe005054bb86086bae88cd6/scripts/run_scanner_heal_linux_evidence_plan.py)、[脚本分类](https://github.com/rustfs/rustfs/blob/940e221988694da33fe005054bb86086bae88cd6/scripts/README.md)。其中 plan/preflight 不产生 measured evidence,assembler 不替代采集。
+Sources: [required E2E registry](https://github.com/rustfs/rustfs/blob/940e221988694da33fe005054bb86086bae88cd6/.config/scanner-heal-required-tests.json), [disk-rebuild E2E](https://github.com/rustfs/rustfs/blob/940e221988694da33fe005054bb86086bae88cd6/crates/e2e_test/src/heal_erasure_disk_rebuild_test.rs), [Linux evidence plan](https://github.com/rustfs/rustfs/blob/940e221988694da33fe005054bb86086bae88cd6/scripts/run_scanner_heal_linux_evidence_plan.py), [script classification](https://github.com/rustfs/rustfs/blob/940e221988694da33fe005054bb86086bae88cd6/scripts/README.md). Of these, plan/preflight does not produce measured evidence, and the assembler does not replace collection.
-## 当前框架的执行方式
+## Execution method of the current framework
-Functional Chain 在共享节点并发锁内调用 auto-testing 的 canonical dispatcher。普通 live 执行消费候选 deb 和工具产物,无需 RustFS checkout;只读 metadata inspector 和原生屏障测试须在构建阶段绑定精确候选源码,单独留证。
+Functional Chain calls auto-testing's canonical dispatcher inside the shared-node concurrency lock. Ordinary live execution consumes the candidate deb and the tool artifacts and does not require a RustFS checkout. The read-only metadata inspector and the native barrier tests must bind the exact candidate source at build time and be evidenced separately.
-尚未自动化的分支由 `matrix.py plan` 固定全部适用拓扑、源断言、操作顺序、参数绑定与恢复步骤。`matrix.py audit` 校验九类证据的摘要和分支完整性,结果仍须逐项审阅,绝不自行生成 canonical PASS。
+Branches that are not yet automated are pinned by `matrix.py plan` for every applicable topology, source assertion, operation order, parameter binding, and recovery step. `matrix.py audit` checks the digests and branch completeness of the nine evidence classes. The results must still be reviewed item by item, and a canonical PASS is never generated on its own.
-正向 H1/H2 夹具显式启用 `RUSTFS_SHARD_INTEGRITY_WRITE=true` 和 `RUSTFS_SHARD_INTEGRITY_FLEET_CONFIRMED=true`。该确认只针对本例私有 endpoint 集合:所有参与节点先安装并校验同一候选二进制,再启动专用服务;配置从真实进程环境读回。候选默认值为 false,默认 legacy 对象的 unknown 回执已单独保留为失败证据,不能改写为 repaired,也不从保护模式推断默认部署已满足这些正向证明契约。
+Positive H1/H2 fixtures explicitly enable `RUSTFS_SHARD_INTEGRITY_WRITE=true` and `RUSTFS_SHARD_INTEGRITY_FLEET_CONFIRMED=true`. That confirmation applies only to this case's private endpoint set: every participating node first installs and verifies the same candidate binary, then starts the dedicated service; configuration is read back from the real process environment. The candidate default is false. The unknown receipt of a default legacy object is retained separately as failure evidence, must not be rewritten as repaired, and protection mode must not be used to infer that the default deployment already satisfies these positive proof contracts.
diff --git a/docs/validation-heal-environment-gates.md b/docs/validation-heal-environment-gates.md
index 6486284..5934377 100644
--- a/docs/validation-heal-environment-gates.md
+++ b/docs/validation-heal-environment-gates.md
@@ -1,4 +1,4 @@
-# Heal 环境门控与原生阻断
+# Heal environment gates and native blocks
## Read-only fleet inspection
@@ -8,49 +8,49 @@ These observations do not certify a clean environment. Independently audit retai
When the shared runner cannot start `environment.inspect`, the separate `validation_runner_inspect_only=true` entry inspects GitHub registration and queue metadata on a hosted runner without fleet access. Inspection [40256930895/1](https://github.com/rustfs/rustfs/actions/runs/40256930895) observed `rustfs-smoke-testing` (ID 139653) offline at `2026-10-26T02:00:24Z`. This is not a diagnosis of host or disk health. API failures remain incomplete; restoring registration does not certify retained-scene cleanup or release the existing live admission gates.
-第一批 13 个用例保留全部 recipe、分支和 source assertions,并新增 `environment_cases.py` 的控制流程与断言,接入 validation 调度。它们默认关闭,只产生 `BLOCKED` 结果与未满足原因;默认路径不会连接或修改 fleet。真实执行沿用 auto-testing 的 S3/Admin collector、ACK ledger、候选包绑定、物理故障 oracle 和恢复模式,专用资源操作由部署环境的适配器提供。当前没有把适配器协议或未运行代码计为 canonical PASS。
+The first batch of 13 cases keeps every recipe, branch, and source assertion, and adds the control flow and assertions in `environment_cases.py`, wired into validation dispatch. They are disabled by default and produce only a `BLOCKED` result and the unmet reason. The default path does not connect to or modify the fleet. A real run reuses auto-testing's S3/Admin collector, ACK ledger, candidate-package binding, physical-fault oracle, and recovery mode. Dedicated resource operations are provided by an adapter in the deployment environment. The adapter protocol, and code that has not been run, are not counted as a canonical PASS.
-设置 `AUTO_TESTING_ENABLE_ENV_GATED_HEAL=1` 并通过 `VALIDATION_HEAL_ENVIRONMENT_CAPABILITIES` 指向 JSON 能力清单后,门控会检查全部能力。缺项仍返回 `BLOCKED`。能力齐全后还要用 `VALIDATION_HEAL_ENVIRONMENT_RUNTIME` 指向 runtime JSON;只有候选/checkout/client 摘要、全部拓扑、当前 Heal SSH 节点和适配器摘要校验通过后才会运行分支。未绑定适配器仍为 `BLOCKED`;执行结果需要 source assertion 审核,当前实现不会自动生成 canonical PASS。
+After `AUTO_TESTING_ENABLE_ENV_GATED_HEAL=1` is set and `VALIDATION_HEAL_ENVIRONMENT_CAPABILITIES` points at a JSON capability manifest, the gate checks every capability. A missing item still returns `BLOCKED`. Once the capabilities are complete, `VALIDATION_HEAL_ENVIRONMENT_RUNTIME` must also point at the runtime JSON. Branches run only after the candidate/checkout/client digests, every topology, the current Heal SSH nodes, and the adapter digest have been checked. An unbound adapter remains `BLOCKED`. Execution results require source-assertion review. The current implementation does not automatically produce a canonical PASS.
-能力清单示例:
+Example capability manifest:
```json
{"capabilities": ["dedicated_faultfs_or_device", "slow_io_short_read_eio", "namespace_isolation", "fault_layer_restore"]}
```
-| Code | 用例 | 当前未满足原因 | 启用条件 |
+| Code | Case | Currently unmet reason | Enablement condition |
| --- | --- | --- | --- |
-| F08 | `heal.mixed-version-upgrade-rollback` | 没有经过审核的不可变双版本组合 | 两个兼容 release、混合 peer 状态、滚动升级/回滚和格式兼容矩阵 |
-| F09 | `heal.dedicated-host-forced-restart` | 共享 Heal 节点不具备 H6 专用主机及平台断电能力 | 专用 H6、平台 power-cycle、带外恢复、boot_id/平台事件证据 |
-| G07 | `heal.race-ilm-expiry-and-transition` | 缺隔离 ILM/tier 与真实操作交叠窗口 | 专用租户、Expire/Transition/Restore 窗口、远端版本映射、规则清理 |
-| G08 | `heal.race-replication-resync` | 缺独占复制两端与 resync 控制 | 隔离复制对、resync、跨端版本映射、复制/Heal 交叠证据 |
-| H01 | `heal.with-expansion-rebalance` | 缺完整 H5 扩容/rebalance 拓扑 | H5 扩容、rebalance task、逐对象 pool/set 映射、scanner 退休证据 |
-| H02 | `heal.with-rebalance-stop-resume` | 缺独立 rebalance 停止/恢复/重启控制 | H5 rebalance、独立 task、stop/resume 和 service restart |
-| H03 | `heal.with-decommission` | 缺可回滚的退役拓扑 | H5 decommission、survivor pool、写入限制、移动 ACK 与物理权威证据 |
-| H04 | `heal.with-decommission-cancel-clear` | 缺退役新代次与容量锁重试窗口 | cancel/clear/resume、#2423 锁窗口、代次绑定 ACK ledger |
-| H05 | `heal.after-retired-pool-removal` | 缺 retired pool 移除后的 survivor-only 拓扑 | retired pool 配置移除、源持续离线、survivor 权威、旧 pool 写保护 |
-| H07 | `heal.ec84-multi-set-restart` | H4 的至少 24 个独立 endpoint 不可用 | H3 12 独立盘、H4 24 endpoint、逐 set 身份与重启/崩溃证据 |
-| I01 | `heal.slow-io-and-eio` | 没有专用 faultfs/故障设备 | 专用故障层、slow/short-read/EIO、namespace 隔离、原状恢复 |
-| I06 | `heal.kms-and-tier-outage` | 没有专用 KMS/tier 租户与依赖隔离 | 独占租户、依赖网络隔离、key 脱敏、tier 版本映射 |
-| J05 | `heal.long-soak` | 缺可持续 24h/72h 的独占 lane 与资源预算 | 真实墙钟窗口、限速和监控、循环故障、到期回收;短跑不能替代 |
-
-`matrix.py list` 同时输出 `automaticExecutor`、`dispatchable` 和 `executionPolicy`:67 个 Heal 自动执行器、13 个环境门控、4 个证据/执行器阻断。调度覆盖共 80 个 Heal,连同 74 个非 Heal 为 154/158;这不代表 154 个 live PASS。
-
-第二批 4 个用例仍未进入 canonical 调度。C11/F07/J02 已有 candidate-bound 原生回归和显式诊断入口;诊断分支通过仍需完整矩阵和独立审计后才能解除门控。I05 仍缺精确服务端屏障,增加节点不能解决:
-
-| Code | 用例 | Block 原因 | 解锁所需证据 |
+| F08 | `heal.mixed-version-upgrade-rollback` | No reviewed immutable dual-version pair | Two compatible releases, mixed peer state, rolling upgrade/rollback, and a format-compatibility matrix |
+| F09 | `heal.dedicated-host-forced-restart` | The shared Heal nodes do not have a dedicated H6 host or platform power-off capability | Dedicated H6, platform power-cycle, out-of-band recovery, and boot_id/platform-event evidence |
+| G07 | `heal.race-ilm-expiry-and-transition` | Missing an isolated ILM/tier tenant and a real overlapping operation window | Dedicated tenant, Expire/Transition/Restore window, remote version mapping, and rule cleanup |
+| G08 | `heal.race-replication-resync` | Missing exclusive replication endpoints and resync control | Isolated replication pair, resync, cross-endpoint version mapping, and replication/Heal overlap evidence |
+| H01 | `heal.with-expansion-rebalance` | Missing a complete H5 expansion/rebalance topology | H5 expansion, rebalance task, per-object pool/set mapping, and retired-scanner evidence |
+| H02 | `heal.with-rebalance-stop-resume` | Missing independent rebalance stop/resume/restart control | H5 rebalance, an independent task, stop/resume, and service restart |
+| H03 | `heal.with-decommission` | Missing a rollback-capable decommission topology | H5 decommission, survivor pool, write restriction, and movement ACK plus physical-authority evidence |
+| H04 | `heal.with-decommission-cancel-clear` | Missing a new decommission generation and the capacity-lock retry window | cancel/clear/resume, the #2423 lock window, and a generation-bound ACK ledger |
+| H05 | `heal.after-retired-pool-removal` | Missing a survivor-only topology after retired-pool removal | Retired-pool configuration removal, source kept offline, survivor authority, and write protection for the old pool |
+| H07 | `heal.ec84-multi-set-restart` | H4's minimum of 24 independent endpoints is unavailable | H3 with 12 independent disks, H4 with 24 endpoints, per-set identity, and restart/crash evidence |
+| I01 | `heal.slow-io-and-eio` | No dedicated faultfs or fault device | Dedicated fault layer, slow/short-read/EIO, namespace isolation, and restoration to the original state |
+| I06 | `heal.kms-and-tier-outage` | No dedicated KMS/tier tenant or dependency isolation | Exclusive tenant, dependency network isolation, key redaction, and tier version mapping |
+| J05 | `heal.long-soak` | Missing an exclusive lane and resource budget that can sustain 24h/72h | A real wall-clock window, rate limits and monitoring, cyclic faults, and expiry reclamation; a short run cannot stand in |
+
+`matrix.py list` also reports `automaticExecutor`, `dispatchable`, and `executionPolicy`: 67 Heal automatic executors, 13 environment gates, and 4 evidence/executor blocks. Dispatch covers 80 Heal cases, and 154/158 together with the 74 non-Heal cases. That is not 154 live PASSes.
+
+The second batch of 4 cases is still outside canonical dispatch. C11/F07/J02 already have candidate-bound native regressions and explicit diagnostic entry points. A passing diagnostic branch still needs the full matrix and an independent audit before the gate can be lifted. I05 still lacks an exact server-side fence, and adding nodes does not close it:
+
+| Code | Case | Block reason | Evidence required to unlock |
| --- | --- | --- | --- |
-| C11 | `heal.bucket-incarnation-recreation` | H1/H2 旧元数据盘重返、原始 checkpoint/resume 与 committed MRF 重放编排已实现;精确候选的 9 项存储原生回归和 3 项 manager incarnation 前置证据已独立核验。完整现场分支尚无独立审计结果 | 旧/新 incarnation、正常 S3 删除与新 ACK、物理范围和原生 artifact 精确绑定;单分支不提升整例 |
-| F07 | `heal.checkpoint-and-mrf-corruption` | 已实现 H1/H2 checkpoint 截断、未来 schema、committed MRF CRC 故障及独立安全重扫;10 项原生回归作为前置证据。真实较早代次回放与私有限额卷的 journal ENOSPC 已实现;仍缺完整 native cut 审核,完整 live 尚未审计 | 真实 durable 记录故障与 publication 证据绑定;现有 interruption hooks 不等于每个 fsync 或断电 |
-| I05 | `heal.lock-timeout-and-caller-abort` | 同源码 native 子集已保留;H1/H2 timeout 与 queued-successor 诊断已通过独立 live 审计(39767336741/1、39817606025/1),完整用例仍 BLOCKED | 实际断连边界所有权、完整 native timeout/stale 屏障、stale-generation controller 和 H1/H2 DL/DV 完整对象/ACK/物理/锁/FD 闭环 |
-| J02 | `heal.outcome-counters-and-error-truth` | 全部夹具与 manifest 对账均已实现。候选 `7951bb629821246868ee3761b321dd07c06db45c` 修复整桶重试覆盖 repaired 记录,210 项 manager 与 5 项独立进程集成测试通过。新实机 `38874900846/1` 完成 20 个分支:16 个报告通过,4 个原始全元数据损坏探针返回永久 Failed,保留 NOT_OBSERVED;四组混合任务与全部 50 条节点清理记录已独立审计。真实 legacy Unknown 已观察,不能代替原始探针 | 原始元数据损坏分支的 Unknown 仍未观察;完整 source-procedure 验收尚未闭合。原始 Failed、NOT_OBSERVED 和完整用例门控保持不变 |
+| C11 | `heal.bucket-incarnation-recreation` | Orchestration for H1/H2 old-metadata disk rejoin, original checkpoint/resume, and committed MRF replay is implemented. Nine storage native regressions and three manager-incarnation prerequisites for the exact candidate have been independently checked. The complete live branches still have no independent audit result | Exact binding of old/new incarnation, a normal S3 delete and the new ACK, the physical range, and the native artifact. One branch does not promote the whole case |
+| F07 | `heal.checkpoint-and-mrf-corruption` | H1/H2 checkpoint truncation, a future schema, committed MRF CRC faults, and an independent safe rescan are implemented. Ten native regressions are prerequisite evidence. Real earlier-generation replay and journal ENOSPC on a private quota volume are implemented. A complete native-cut review is still missing, and the complete live run has not been audited | Binding of a real durable-record fault to publication evidence. Existing interruption hooks are not every fsync or a power loss |
+| I05 | `heal.lock-timeout-and-caller-abort` | The same-source native subset is retained. H1/H2 timeout and queued-successor diagnostics have passed independent live audit (39767336741/1, 39817606025/1). The complete case remains BLOCKED | Ownership at the real disconnect boundary, the complete native timeout/stale fence, a stale-generation controller, and the full H1/H2 DL/DV object/ACK/physical/lock/FD closure |
+| J02 | `heal.outcome-counters-and-error-truth` | Every fixture and manifest reconciliation is implemented. Candidate `7951bb629821246868ee3761b321dd07c06db45c` fixes whole-bucket retry overwriting repaired records. 210 manager tests and 5 independent-process integration tests passed. The new live run `38874900846/1` completed 20 branches: 16 reported pass, and 4 probes of originally corrupt metadata on every replica returned permanent Failed and retained NOT_OBSERVED. Four mixed-task groups and all 50 node-cleanup records were independently audited. A real legacy Unknown has been observed and cannot stand in for the original probe | Unknown on the originally corrupt-metadata branch is still unobserved. Complete source-procedure acceptance is not closed. The original Failed, NOT_OBSERVED, and the full-case gate stay unchanged |
-I05 与 A09、G01–G06 的精确屏障证据允许使用同一不可变候选源码单独构建的 `cfg(test)` 原生测试执行文件。不得给优化服务候选加入隐藏 hook 冒充原包,也不能把缺少 release hook 本身归为外部环境阻塞。原生子集、完整服务调用和现场最终 ACK/物理检查分别验收;已验收 native 证据不能代替尚缺的 controller 或完整 live 结果。第一批 G07/G08 的 native 分支与 H04 容量锁窗口保留各自未完成条件;外部请求重试或最终对象正确均不能证明精确锁或提交窗口。
+Exact fence evidence for I05 and for A09 and G01–G06 may use a `cfg(test)` native test executable built separately from the same immutable candidate source. Do not add a hidden hook to an optimized service candidate and present it as the original package, and do not classify the absence of a release hook itself as an external environment block. The native subset, the full service invocation, and the final live ACK/physical check are accepted separately. Accepted native evidence cannot stand in for a controller that is still missing, or for a complete live result. The native branches of first-batch G07/G08, and the H04 capacity-lock window, each keep their unfinished conditions. An external request retry, or a finally correct object, does not prove the exact lock or commit window.
-2026-10-26 已独立复核 hosted-only tools [40086026324/1](https://github.com/rustfs/rustfs/actions/runs/40086026324):候选 `757b751482bb970bc321e92059690a7698554358` 上 10 项精确原生测试和 104 条有序 milestone 全部通过,覆盖 A09、G01–G06 与 I05 的原生子集。固定 helper `002473b` 的 runtime admission 在该候选上仅开放 A09/G05/G06;G01–G04 仍须选择各自审核过的候选及匹配工具,或先完成新的 runtime 源码审核。完整 live 用例和 I05 剩余控制流程均未因此通过,canonical 计数保持 114/158。
+On 2026-10-26 the hosted-only tools run [40086026324/1](https://github.com/rustfs/rustfs/actions/runs/40086026324) was independently reviewed: on candidate `757b751482bb970bc321e92059690a7698554358`, 10 exact native tests and 104 ordered milestones all passed, covering the native subsets of A09, G01–G06, and I05. Runtime admission for the pinned helper `002473b` opens only A09/G05/G06 on that candidate. G01–G04 must still select their own reviewed candidate and matching tools, or complete a new runtime source review first. The complete live cases and the remaining I05 control flow did not pass because of this, and the canonical count stays 114/158.
-资源适配器协议使用 stdin/stdout JSON,不调用 shell。runtime 中的 `adapter` 包含绝对可执行路径、SHA-256 和操作 allowlist;`environment_cases.required_operations(case_id)` 给出所需操作。每个请求携带 run ID、原候选 commit、sequence、拓扑/分支 scope、bucket prefix、租约和参数。响应必须回显上述身份、按 `heal_probe.canonical(request)` 的 UTF-8 SHA-256 绑定原请求,并把真实观察写在 `observation`,禁止返回 verdict。资源适配器需独立实现平台 power control、故障层、pool lifecycle 或 KMS/tier 隔离;本 PR 未向当前共享环境部署适配器。
+The resource-adapter protocol uses stdin/stdout JSON and does not invoke a shell. The runtime `adapter` contains an absolute executable path, a SHA-256, and an operation allowlist. `environment_cases.required_operations(case_id)` lists the required operations. Each request carries the run ID, the original candidate commit, a sequence, the topology/branch scope, the bucket prefix, the lease, and the parameters. The response must echo that identity, bind the original request by the UTF-8 SHA-256 from `heal_probe.canonical(request)`, and write real observations in `observation`. It must not return a verdict. A resource adapter has to implement platform power control, the fault layer, pool lifecycle, or KMS/tier isolation on its own. This change does not deploy an adapter into the current shared environment.
-runtime 的 `binding` 使用现有 `heal_probe.validate_binding` 契约,另含 `schemaVersion: 1`、`provider: auto-testing`、`sshNodes`、`rcBinary`、`topologies`、`capabilities`、`adapter` 和对应资源字段。F08 需要 `releasePair`,F09 需要 `hostResourceId`,I01 需要 `faultDeviceId/ioLatencyMs`,G07/I06 需要 `kmsKeyId/tierEndpoint/tenantId/dependencyNamespaceId`,G08 需要 `replicaEndpoint`,J05 需要 `soakLimits`。H6 专用主机应由适配器按 `hostResourceId` 独占绑定;SSH fleet 仍记录当前 auto-testing Heal 节点。secret 通过环境传入,适配器诊断会脱敏,`redactionEnvironmentVariables` 可指定额外 secret 环境变量名。
+The runtime `binding` uses the existing `heal_probe.validate_binding` contract and also contains `schemaVersion: 1`, `provider: auto-testing`, `sshNodes`, `rcBinary`, `topologies`, `capabilities`, `adapter`, and the corresponding resource fields. F08 requires `releasePair`, F09 requires `hostResourceId`, I01 requires `faultDeviceId/ioLatencyMs`, G07/I06 require `kmsKeyId/tierEndpoint/tenantId/dependencyNamespaceId`, G08 requires `replicaEndpoint`, and J05 requires `soakLimits`. A dedicated H6 host must be exclusively bound by the adapter using `hostResourceId`. The SSH fleet still records the current auto-testing Heal nodes. Secrets are passed through the environment. Adapter diagnostics are redacted, and `redactionEnvironmentVariables` can name additional secret environment variables.
-J05 的实际控制循环保留固定 86400/259200 秒窗口、精确版本写入/删除 ACK、三类循环故障、恢复、资源阈值和到期全量校验。单元测试用可注入的时钟验证窗口控制,不修改生产时长。G07/G08 精确 native 分支不调用资源适配器。清理失败保留原失败与 DIRTY 原因,停止后续分支。
+J05's real control loop keeps the fixed 86400/259200-second windows, exact version write/delete ACKs, three kinds of cyclic faults, recovery, resource thresholds, and a full check at expiry. Unit tests verify window control with an injectable clock and do not change the production duration. The exact native branches of G07/G08 do not call the resource adapter. A cleanup failure retains the original failure and the DIRTY reason and stops later branches.
diff --git a/docs/validation-heal-operations.md b/docs/validation-heal-operations.md
index 71247e0..99c44e9 100644
--- a/docs/validation-heal-operations.md
+++ b/docs/validation-heal-operations.md
@@ -1,10 +1,10 @@
-# Heal 真机执行手册:84 项操作卡的公共命令
+# Heal Live Execution Manual: Common Commands for the 84 Procedure Cards
-此手册和 [84 项逐例操作卡](validation-heal-cases.md) 一起构成可执行的**人工操作方案 + 脚本辅助**。不是 84 个无人值守自动测试,也不是通过报告。操作卡列出 302 个分支模板;真实盘角色、阈值、损坏数量和拓扑仍需在每批 `parameterBindings` 展开,不能把模板数量当实测次数。
+This manual, together with the [84 per-case procedure cards](validation-heal-cases.md), forms an executable **manual procedure plus script assistance**. It is not 84 unattended automated tests, and it is not a pass report. The procedure cards list 302 branch templates; actual disk roles, thresholds, damage counts, and topologies still have to be expanded in each batch's `parameterBindings`. The template count must not be treated as the number of live measurements.
-公共验收契约见 [Heal 规范](validation-heal-contract.md)。源仓库历史 pilot 结果不计入当前 canonical 分母。自动入口 `scripts/run_validation_case.sh CASE_ID CANDIDATE_MANIFEST TESTING_COMMIT ARTIFACTS_DIR` 已接入 67 个 Heal 用例,完整清单通过 `matrix.py list` 查看,以 `automaticExecutor` 为准。入口使用当前 Heal 节点、候选 deb 和独立 H1/H2 服务;其余操作卡仍需按本手册操作留证。
+The common acceptance contract is in the [Heal specification](validation-heal-contract.md). Historical pilot results from the source repository are not counted in the current canonical denominator. The automatic entry point `scripts/run_validation_case.sh CASE_ID CANDIDATE_MANIFEST TESTING_COMMIT ARTIFACTS_DIR` already covers 67 Heal cases. The complete list is viewed with `matrix.py list`, and `automaticExecutor` is authoritative. The entry point uses the current Heal nodes, the candidate deb, and independent H1/H2 services; the remaining procedure cards still have to be carried out, with evidence retained, according to this manual.
-## 1. 使用入口与交付物
+## 1. Entry points and deliverables
```bash
python scripts/validation_heal/matrix.py list
@@ -16,61 +16,61 @@ python scripts/validation_heal/matrix.py plan --selection A01,A04,C01,C02,C04,J0
--topologies H1 --run-id manual-heal-20260910-b --out reports/heal-h1-plan.json
```
-`plan` 只冻结选择/分支和对应操作,不部署、不评分。`show` 同时给出步骤、命令入口、验收和恢复要求。不同拓扑是不同实例;未选拓扑会保留 `scopeExclusions`,不能据局部结果声称全矩阵通过。
+`plan` only freezes the selection, the branches, and the corresponding operations. It does not deploy, and it does not score. `show` also gives the steps, the command entry points, the acceptance criteria, and the recovery requirements. Different topologies are different instances; an unselected topology retains `scopeExclusions`. A partial result must not be claimed as a full-matrix pass.
-自动用例通过 RustFS Functional Chain 的 `validation_case` 和精确 `auto_testing_sha` 分发。先用同一不可变候选的 `build_run_id`/`build_run_attempt`、`validation_tools_only=true` 构建只读 metadata inspector;live 分发提供该成功构建的 `validation_tools_run_id` 与 `validation_tools_run_attempt`。工具的 candidate/helper SHA、Linux 二进制 SHA 和独立 Cargo.lock 摘要均须通过检查,再复制到本例节点。普通 live 执行消费构建产物,不需要 RustFS checkout。
+Automatic cases are dispatched through the RustFS Functional Chain `validation_case` and the exact `auto_testing_sha`. First build the read-only metadata inspector from the same immutable candidate's `build_run_id`/`build_run_attempt` with `validation_tools_only=true`. Live dispatch supplies that successful build's `validation_tools_run_id` and `validation_tools_run_attempt`. The tool's candidate/helper SHA, Linux binary SHA, and independent Cargo.lock digest must all pass their checks before they are copied onto this case's nodes. Ordinary live execution consumes the build artifacts and does not need a RustFS checkout.
-| 工具 | 已实现的职责 | 不代表什么 |
+| Tool | Implemented responsibility | What it does not represent |
| --- | --- | --- |
-| `matrix.py` | 84 项/302 分支索引、逐例操作卡、冻结计划、证据完整性审计 | 不自动分配拓扑,不根据文件存在就判 PASS |
-| `heal_probe.py` | 带签名/匿名/坏签名的原始 Admin;受限 S3 操作;全版本/标签/内容摘要清单及比较 | HTTP 检查通过不代表 heal 成功;inventory 的 GET 会触发读修复 |
-| `heal_workload.py` | DB/DV/DM/DS/DX 真实夹具;有界并发、同key前台修改串行、请求前及ACK后持久落盘 | 不替用户推导竞态后的最终版本;超时写必须单独解析 |
-| `heal_fault.py` | Linux 描述符相对寻址、mount ID 保护、单文件 missing/bitflip/truncate/zero/header、快照和恢复 | 文件故障不是物理掉盘;不修改系统 metadata;不盲目恢复新代次 |
-| `prepare_heal_meta_inspector.py` | 为精确候选源码准备只读 Rust metadata 解析器工程 | 需在批准的构建机编译;解析 metadata 不是验证 bitrot/EC |
-| `shards.py` | C01–C04、D02、D07、E01、E02 全 24 分支以及 A03 的两个 H2 跨节点状态分支、A04 的六个 DB/DV dry-run 分支;静默故障、真实角色、同角色原片与 HighwayHash、先物理后 GET/Range、MPU control | 有自动入口不等于候选通过;任务 unknown、目标只恢复 metadata 或救援备份均不能记 PASS |
-| `check_bitrot_tool.py` | 在固定 MinIO/历史 RustFS 产物上检查 24 个健康片和 12 个坏片负例 | 工具验证不等于 live 用例通过;不调用被测 RustFS 服务 |
+| `matrix.py` | Index of the 84 items and 302 branches, per-case procedure cards, frozen plans, and evidence-completeness audits | It does not assign topologies automatically, and it does not judge PASS merely because a file exists |
+| `heal_probe.py` | Raw Admin with a signature, anonymous access, or a bad signature; restricted S3 operations; full-version, tag, and content-digest inventories and comparison | An HTTP check passing does not mean heal succeeded; an inventory GET triggers read-repair |
+| `heal_workload.py` | Real DB/DV/DM/DS/DX fixtures; bounded concurrency, serial foreground mutation of the same key, and durable persistence before the request and after the ACK | It does not derive the post-race final version for the user; a timed-out write must be resolved separately |
+| `heal_fault.py` | Linux descriptor-relative addressing, mount-ID protection, single-file missing/bitflip/truncate/zero/header, snapshots, and recovery | A file fault is not a physical disk loss; it does not modify system metadata; it does not blindly restore a new generation |
+| `prepare_heal_meta_inspector.py` | Prepares a read-only Rust metadata parser project for the exact candidate source | It must be compiled on an approved build machine; parsing metadata is not verification of bitrot or EC |
+| `shards.py` | All 24 branches of C01–C04, D02, D07, E01, and E02, plus A03's two H2 cross-node state branches and A04's six DB/DV dry-run branches; quiescent faults, real roles, same-role original shards and HighwayHash, physical checks before GET/Range, and MPU control | An automatic entry point does not mean the candidate passed; a task that is unknown, a target that only restores metadata, or a rescued backup cannot be recorded as PASS |
+| `check_bitrot_tool.py` | Checks 24 healthy shards and 12 bad-shard negative cases against fixed MinIO and historical RustFS artifacts | Tool verification is not a live case pass; it does not call the RustFS service under test |
-## 2. 先绑定环境,禁止直接猜节点
+## 2. Bind the environment first; do not guess nodes directly
-live 命令使用当前 auto-testing 的 `smoke-testing` runner、`RUSTFS_NODES` 和 `RUSTFS_SSH_USER`,复用 `rustfs_heal_test.sh` 的 Heal 节点。运行前取得与 Functional Chain 一致的共享节点并发锁。H1–H6 是测试拓扑要求,不是另一组 SSH 主机;不得按源仓库历史主机名或 IP 选择机器。
+Live commands use the current auto-testing `smoke-testing` runner, `RUSTFS_NODES`, and `RUSTFS_SSH_USER`, and they reuse the Heal nodes of `rustfs_heal_test.sh`. Before a run, acquire the shared node concurrency lock that matches the Functional Chain. H1–H6 are test-topology requirements, not another set of SSH hosts. Machines must not be chosen by historical hostnames or IP addresses from the source repository.
-### Runner 无法接单时的只读诊断
+### Read-only diagnosis when the runner cannot accept work
-RustFS Functional Chain 提供 `validation_runner_inspect_only=true` 入口。它在 `ubuntu-latest` 上读取 GitHub 的 runner 注册信息与活动任务,使用独立并发组,不占用或替换测试 fleet 队列。一次请求只能选择该模式,不同时传入 candidate、tools、case 或 recovery 参数;使用包含此入口的已审核 workflow ref:
+The RustFS Functional Chain provides the `validation_runner_inspect_only=true` entry point. On `ubuntu-latest` it reads GitHub runner registration information and active jobs, uses an independent concurrency group, and neither occupies nor replaces the test fleet queue. A single request may select only this mode, and must not also pass candidate, tools, case, or recovery parameters. Use a reviewed workflow ref that contains this entry point:
```bash
gh workflow run rustfs-functional-chain.yml --repo rustfs/rustfs \
--ref "$VALIDATION_DRIVER_REF" -f validation_runner_inspect_only=true
```
-下载该次 `validation-runner-inspection--` artifact,核对 `workflowCommit`、`runId`、`runAttempt`、`observedAt`、`complete` 和 `runnerAvailability`。`offline`、`online-busy`、`online-idle`、`unregistered` 是 GitHub 注册状态的不同观察;API 拒绝、分页超过上限或其他错误保留 `complete=false` 与失败阶段,不能解释为空闲。报告不连接节点、不重启服务、不清理文件、不取消 workflow,也不证明环境已恢复或共享 workflow 全局空闲。runner 服务恢复后,仍需重新执行资源检查及原有现场/并发门控。
+Download that run's `validation-runner-inspection--` artifact and check `workflowCommit`, `runId`, `runAttempt`, `observedAt`, `complete`, and `runnerAvailability`. `offline`, `online-busy`, `online-idle`, and `unregistered` are distinct observations of GitHub registration state. An API rejection, pagination past the limit, or any other error retains `complete=false` and the failure stage, and must not be interpreted as idle. The report does not connect to nodes, does not restart services, does not clean up files, and does not cancel workflows, and it does not prove that the environment has recovered or that the shared workflow is globally idle. After the runner service recovers, the resource checks and the original scene and concurrency gates still have to be executed again.
-用 `matrix.py plan` 冻结当前 SSH 节点与所有适用操作分支,再根据本次只读 preflight 填写私有 `lab.json`。计划中的 `verified:false` 不可直接改成资源验证通过。H4 需要至少 24 个已分配独立盘端点;H6 需要独占主机及平台事件/恢复绑定。缺少资源的分支保留为 BLOCKED。
+Use `matrix.py plan` to freeze the current SSH nodes and every applicable operation branch, then fill in the private `lab.json` from this read-only preflight. `verified:false` in the plan must not be rewritten directly as a passed resource verification. H4 requires at least 24 allocated independent disk endpoints. H6 requires an exclusive host and bindings for platform events and recovery. A branch that lacks resources remains BLOCKED.
-填写内容须包含:
+The filled-in contents must include:
-- 本批 run ID、截止时间、独占服务/端口、jump route、实际主机名、每个磁盘的 mount UUID/major:minor、服务 unit/原 PID/数据根;
-- 与当前 validation lane 相同的不可变 candidate manifest、源码 commit、包与已安装二进制 SHA-256;当前 rc 路径与二进制 SHA-256;auto-testing commit;保存真实构建清单,不只填摘要字符串;
-- H1–H6 实際形状、每个对象 N/data/parity/R/W/lock quorum、pool/set/slot/盘 UUID;H4 至少 24 个实际盘端点;
-- 最长修复/重启/故障时限、对象/字节/请求/并发预算、所有危险动作的精确目标及恢复人;
-- 包含平台操作、独占故障设备、旧版本、KMS/tier 或原生屏障的实例,先提供能力及授权证据。
+- This batch's run ID, deadline, exclusive service and port, jump route, actual hostnames, each disk's mount UUID and major:minor, and the service unit, original PID, and data root;
+- The immutable candidate manifest, source commit, and package and installed-binary SHA-256 that match the current validation lane; the current rc path and binary SHA-256; the auto-testing commit; save the real build inventory, and do not fill in only a digest string;
+- The actual shape of H1–H6, each object's N/data/parity/R/W/lock quorum, and pool/set/slot/disk UUID; H4 has at least 24 actual disk endpoints;
+- The maximum repair, restart, and fault time limits, the object, byte, request, and concurrency budgets, and the exact target and recovery owner of every dangerous action;
+- For an instance that includes platform operations, an exclusive fault device, an old version, KMS/tier, or a native barrier, first provide capability and authorization evidence.
-`binding-check` 只是语法/租约检查,不是资源或包身份的远程证明:
+`binding-check` is only a syntax and lease check. It is not a remote proof of resource or package identity:
```bash
python scripts/validation_heal/heal_probe.py --binding lab.json --out reports/binding-check.json binding-check
```
-先在跳板只读核验 `hostname`、`findmnt -J`、`lsblk -J`、`ss -lnt`、二进制 `sha256sum`/`--version`、实际运行配置与 PID。保存输出为 preflight。租约没有发放、拓扑不符、端口占用、磁盘 UUID 漂移时在这里 BLOCKED,不先注入后跳过。
+First, on the jump host, read-only verify `hostname`, `findmnt -J`, `lsblk -J`, `ss -lnt`, the binary `sha256sum` and `--version`, the actually running configuration, and the PID. Save the output as the preflight. When the lease has not been issued, the topology does not match, a port is occupied, or a disk UUID has drifted, record BLOCKED here. Do not inject first and skip afterward.
-### 独立实验服务的部署
+### Deployment of the independent experiment service
-H1 使用当前 Heal 节点经核验的四个独立 mount,**另建** `/data/rustfsN/heal-validation//data` 和同级 `quarantine`,root-owned 0700;每个 `` 根写入包含 `runId` 的 root-owned `owner.json`。普通对象故障可以使用这些独立父盘下的 run 私有目录。B03 和 B04 还要求替换 target 和同组兄弟 endpoint 都能取得 candidate 认可的精确 mount lease;替换过程中必须证明旧 mount 与 quarantine 对服务不可读、新空设备挂到原 endpoint 后 candidate 取得新的 mount lease,并前后记录 mount ID、物理设备 ID 和 filesystem identity。独立文件系统或隔离 mount namespace 中的 bind mount 都可能作为实现方式,但仅创建 bind mount 不足以满足要求:adapter 还必须安全切换被替换 endpoint,且通过 candidate 对 root 和 sibling 物理设备身份的检查。Functional Chain 会先在 `stop_existing` 和候选 deb 下载/安装之前调用只读 `replacement-admission-preflight`,覆盖 H1/H2 endpoints;未通过时以 `BLOCKED` 保留 package/client provenance 和完整现场读数,不停服务、不碰已安装 package,也不创建远端 run namespace。`Fleet.deploy()` 在真正 reservation 前再重复该检查。缺少真实 endpoint mount 或已审核 mount-switch adapter 时应 BLOCKED;当前 runner 只把 run 私有子目录作为 endpoint,尚无此 mount-switch adapter,因此它现有的四个独立父盘不能证明 B03/B04 满足资源条件。不得延长等待时间或复制 quarantine 数据。先检查每个根不存在;已存在的根不能覆盖。复用 `rustfs_heal_test.sh` 的 candidate deb 下载、SHA 校验和安装流程,把已安装且摘要已核验的 `/usr/bin/rustfs` 放在本批 state 下;执行用例不需要 RustFS 源码 checkout。凭据文件为 0600。以下专用 unit 仍运行在当前 Heal 节点,使用已分配的 9000 测试端口:
+H1 uses the four verified independent mounts on the current Heal nodes, and **additionally creates** `/data/rustfsN/heal-validation//data` and a sibling `quarantine`, root-owned mode 0700. Each `` root is written with a root-owned `owner.json` that contains `runId`. Ordinary object faults may use the run-private directories under these independent parent disks. B03 and B04 also require that both the replacement target and the sibling endpoints in the same group can obtain an exact mount lease recognized by the candidate. During replacement it must be proved that the old mount and the quarantine are unreadable to the service, that after the new empty device is mounted at the original endpoint the candidate obtains a new mount lease, and that the mount ID, physical device ID, and filesystem identity are recorded before and after. A bind mount on an independent filesystem, or in an isolated mount namespace, may be used as an implementation, but creating a bind mount alone does not meet the requirement: the adapter must also switch the replaced endpoint safely, and it must pass the candidate's checks of the root and sibling physical-device identities. Before `stop_existing` and before the candidate deb is downloaded or installed, the Functional Chain calls the read-only `replacement-admission-preflight`, covering the H1/H2 endpoints. On failure it retains the package and client provenance and the complete scene readings as `BLOCKED`. It does not stop services, does not touch the installed package, and does not create a remote run namespace. `Fleet.deploy()` repeats that check again before the real reservation. When a real endpoint mount or a reviewed mount-switch adapter is missing, the result should be BLOCKED. The current runner uses only a run-private subdirectory as the endpoint and does not yet have this mount-switch adapter, so its existing four independent parent disks cannot prove that B03 and B04 meet the resource conditions. Do not extend the wait, and do not copy quarantine data. First check that each root does not exist. An existing root must not be overwritten. Reuse the candidate deb download, SHA verification, and install flow from `rustfs_heal_test.sh`, and place the installed, digest-verified `/usr/bin/rustfs` under this batch's state. Executing a case does not require a RustFS source checkout. Credential files are mode 0600. The dedicated unit below still runs on the current Heal nodes and uses the allocated test port 9000:
-如果首次 gate 通过后、`Fleet.deploy()` 的 reservation-time 复检发现 mount 漂移,仍记录 `BLOCKED` 并保留已安装候选的 binary provenance;不得把该结果改写成产品 `FAIL`,也不得把它伪装成 pre-deployment 的 `NOT_REQUIRED_NO_MUTATION`。两阶段检查都必须跳过 reservation 和 fixture seed,并完成 cleanup。
+If the first gate passes and the reservation-time recheck in `Fleet.deploy()` then finds mount drift, still record `BLOCKED` and retain the installed candidate's binary provenance. That result must not be rewritten as a product `FAIL`, and it must not be disguised as the pre-deployment `NOT_REQUIRED_NO_MUTATION`. Both stages of the check must skip reservation and fixture seeding, and must complete cleanup.
```bash
-# 在目标节点;所有变量来自已审核的 lab binding,不接受任意在线输入。
+# On the target node; every variable comes from the reviewed lab binding. Arbitrary online input is not accepted.
sudo systemd-run --unit="$HEAL_UNIT" --property=Type=exec \
--property=PrivateMounts=yes --property=KillMode=control-group \
--property=TimeoutStopSec=30 --property=RuntimeMaxSec=3600 \
@@ -80,11 +80,11 @@ sudo systemd-run --unit="$HEAL_UNIT" --property=Type=exec \
"$HEAL_STATE/rustfs" "${HEAL_DATA_ROOTS[@]}"
```
-需重启的场景保留 unit 定义、env、二进制及原始 endpoint 参数,不使用 `--collect`;RuntimeMaxSec 按该批预算冻结,J05 必须单独分配长时服务。`HEAL_DATA_ROOTS` 是已审核的 Bash 数组,不从未审核字符串 eval。H2/H3/H4/H5 用各节点一致的 endpoint 序列/显式 pool 分组配置;启动后从每盘 `format.json` 读回,不能从启动参数猜拓扑成功。不得直接运行会清理旧 Pool 根的部署/reset 命令。
+A scenario that needs a restart retains the unit definition, the env, the binary, and the original endpoint arguments, and does not use `--collect`. RuntimeMaxSec is frozen to that batch's budget. J05 must be assigned its own long-running service. `HEAL_DATA_ROOTS` is a reviewed Bash array and is not eval'd from an unreviewed string. H2/H3/H4/H5 use an endpoint sequence, or an explicit pool-grouping configuration, that is consistent on every node. After startup, read the result back from each disk's `format.json`. Topology success must not be guessed from the startup arguments. Do not directly run a deploy or reset command that would clean old Pool roots.
-### 命令约定
+### Command conventions
-下列函数在**跳板机**执行。`HEAL_BINDING`、`HEAL_REPORT` 是该实例私有绝对路径;`B` 是绑定 prefix 下的一个桶,`K` 为逻辑 key,`V` 为先前 ACK 的精确版本。每条命令的 label 唯一,已有证据拒绝覆盖。凭据只通过受控环境注入 `HEAL_ACCESS_KEY`/`HEAL_SECRET_KEY`/可选 session token,禁用 `set -x`。
+The following functions run on the **jump host**. `HEAL_BINDING` and `HEAL_REPORT` are private absolute paths for this instance. `B` is a bucket under the binding prefix, `K` is the logical key, and `V` is the exact version of a previous ACK. Each command's label is unique, and existing evidence refuses overwrite. Credentials are injected only through the controlled environment, as `HEAL_ACCESS_KEY`, `HEAL_SECRET_KEY`, and an optional session token. `set -x` is disabled.
```bash
set -euo pipefail
@@ -105,16 +105,16 @@ B=$(jq -er .bucketPrefix "$HEAL_BINDING")
RUN=$(jq -er .runId "$HEAL_BINDING")
```
-特权故障命令在**经跳板连接的目标节点**执行。`ROOT` 是本例目标盘 `` 根;`NODE` 是已绑定 SSH alias;把该 SHA 的 `heal_fault.py` 放在 root-owned 的本批 state 下并核对摘要。不得从逻辑 key 拼 SSH shell。以下 `hf` 仅在该目标节点的交互 shell 定义:
+Privileged fault commands run on the **target node reached through the jump host**. `ROOT` is this case's `` root on the target disk. `NODE` is the bound SSH alias. Place that SHA's `heal_fault.py` under this batch's root-owned state and check the digest. Do not assemble an SSH shell from a logical key. The following `hf` is defined only in an interactive shell on that target node:
```bash
hf() { sudo python3 "$HEAL_STATE/heal_fault.py" --root "$ROOT" --run-id "$RUN" "$@"; }
hf init --bucket-prefix "$B" --lease-expires-at "$LEASE_EXPIRES_AT"
```
-初始化只增加故障所有权标记,不创建/格式化数据盘。每盘只 init 一次。系统重启导致 mount ID 变化时暂停注入;先重新核对 UUID/major:minor/所有权并人工重新绑定,不绕过检查。
+Initialization only adds the fault-ownership marker. It does not create or format a data disk. Each disk is initialized only once. When a system restart changes the mount ID, pause injection. First recheck the UUID, major:minor, and ownership, and rebind manually. Do not bypass the check.
-## 3. 数据与客户端命令
+## 3. Data and client commands
@@ -122,9 +122,9 @@ hf init --bucket-prefix "$B" --lease-expires-at "$LEASE_EXPIRES_AT"
-### 确定性夹具
+### Deterministic fixtures
-每个命令使用新桶,禁止在现有夹具桶上重复 seed 后称同一轮。
+Each command uses a new bucket. Do not seed an existing fixture bucket again and then call it the same round.
```bash
hw seed-db seed --fixture DB --bucket "$B-db"
@@ -132,23 +132,23 @@ hw seed-dv seed --fixture DV --bucket "$B-dv"
hw seed-dm seed --fixture DM --bucket "$B-dm"
hw seed-ds seed --fixture DS --bucket "$B-ds"
hw seed-dx seed --fixture DX --bucket "$B-dx"
-# 替换为候选确认的真实 inline/block 边界;可另外生成一桶:
+# Replace with the real inline/block boundary confirmed by the candidate; an additional bucket may be generated:
hw seed-boundary seed --fixture DB --bucket "$B-boundary" \
--count 128 --sizes "$BOUNDARY_SIZES"
```
-DB 默认 128 项,包含 0/1/4095/4096/4097、1MiB±1、多MiB,非重复 SHAKE 数据;1MiB 不是预设产品阈值。DV 默认 4 key,每 key 三个版本,交替创建当前 marker。DM 有已完成三 part(前两 part 分别 5MiB+17、5MiB+93,末 part 1MiB+47)及两个独立 pending upload。DS 默认 257 特殊/多前缀 key;DX 默认 257 外部对象。自动 seed 不包括 suspended/null、特殊压缩、SSE/retention 等额外分支,按下列 S3 命令补充并保存 ACK。
+DB defaults to 128 items, including 0/1/4095/4096/4097, 1MiB±1, and multi-MiB sizes, of non-repeating SHAKE data. 1MiB is not a preset product threshold. DV defaults to 4 keys, with three versions per key, alternately creating the current marker. DM has one completed three-part object (the first two parts are 5MiB+17 and 5MiB+93, and the last part is 1MiB+47) and two independent pending uploads. DS defaults to 257 special/multi-prefix keys. DX defaults to 257 external objects. Automatic seeding does not include extra branches such as suspended/null, special compression, or SSE/retention. Supplement those with the S3 commands below and save the ACKs.
-从 ACK 文件取得真正的 UploadId/VersionId/part ETag;不要在多个上传之间复用猜测的 ID。总字节预算包含所有版本/part,不足时先扩展经批准的预算,不自动跳过对象。
+Take the real UploadId, VersionId, and part ETag from the ACK file. Do not reuse a guessed ID across uploads. The total byte budget includes every version and part. When it is insufficient, first extend the approved budget. Do not skip objects automatically.
-后续命令的 `B` 必须改为该实例实际桶,例如 `B="$(jq -er .bucketPrefix "$HEAL_BINDING")-db"`,不要对没有创建的裸 prefix 做 inventory。根目录 owner 仍绑定原始 run prefix,以允许本run的各后缀桶。
+In later commands, `B` must be changed to this instance's actual bucket, for example `B="$(jq -er .bucketPrefix "$HEAL_BINDING")-db"`. Do not inventory a bare prefix that was never created. The root-directory owner stays bound to the original run prefix, so that each suffixed bucket of this run is allowed.
-seed 完成后自动以完整 GET/版本清单对账确定性原文与创建 ACK,保存 `expected-fixture.json`、`baseline.json`;没有通过基线校验就不准进入注入。这个 GET 发生在故障前,不可拿 seed 命令监测 scanner/read-repair 故障窗口。
+After seeding completes, the deterministic original content and the creation ACKs are reconciled automatically against a complete GET and version inventory, and `expected-fixture.json` and `baseline.json` are saved. Injection is not allowed until the baseline check has passed. This GET happens before the fault. The seed command must not be used to observe the scanner or read-repair fault window.
-### 单个 S3 操作与精确版本
+### Individual S3 operations and exact versions
```bash
hp put s3 --operation put_object --size 5243003 --seed "$RUN/new/1" \
@@ -161,30 +161,30 @@ hp delete-version s3 --operation delete_object --expect-http 204 \
--params "$(jq -nc --arg b "$B" --arg k "$K" --arg v "$V" '{Bucket:$b,Key:$k,VersionId:$v}')"
```
-请求复杂时 `--params-file` 使用 JSON。接口参数为 boto3 snake_case 操作名、原生 CamelCase 参数;只允许本 run 桶、受限操作,copy 源也必须在本 run。GET 只保存完整内容的 hash/长度,不把对象内容写入日志。PUT/Complete 超时记 resultUnknown,下一步用相同唯一 key/version/UploadId 查询解析,不静默重试。
+When a request is complex, `--params-file` uses JSON. Interface parameters are boto3 snake_case operation names and native CamelCase parameters. Only this run's buckets and restricted operations are allowed, and a copy source must also be in this run. GET saves only the hash and length of the complete content, and does not write object content into the log. A PUT or Complete timeout is recorded as resultUnknown. The next step resolves it by querying the same unique key/version/UploadId, with no silent retry.
-`--expect-http` 必须使用该请求预注册的精确值。例如 create/put/list 常见 200,delete/abort 常见 204,Range 206;负例按候选契约分别 304/403/404/405/412/416。CLI 返回 0 只说明该次预期 HTTP 匹配,永不宣称整个 case PASS。SSE-C 参数值可使用字面量 `$HEAL_SSE_C_KEY`,脚本只在内存替换,输出脱敏。
+`--expect-http` must use the exact value pre-registered for that request. For example, create, put, and list are commonly 200, delete and abort are commonly 204, and Range is 206. Negative cases are 304, 403, 404, 405, 412, and 416 respectively, according to the candidate contract. A CLI exit of 0 only means that this expected HTTP status matched. It never declares the whole case PASS. An SSE-C parameter value may use the literal `$HEAL_SSE_C_KEY`. The script substitutes it only in memory, and the output is redacted.
-### 全版本基线和最终核对
+### Full-version baseline and final check
```bash
hp baseline inventory --bucket "$B"
-# 先完成触发归属和 PHYSICAL,再调用下面 GET 型检查:
+# Finish trigger attribution and PHYSICAL first, then call the GET-type checks below:
hp post-heal inventory --bucket "$B"
hp s3-comparison compare --before "$HEAL_REPORT/baseline.json" \
--after "$HEAL_REPORT/post-heal.json"
```
-inventory 分页遍历全部版本/marker,按每个 versionId GET 全内容、tags 和公开 metadata,再用分页 current LIST 交叉核对;对象上限 10,000、单对象 64MiB,批次字节预算来自绑定文件。更大对象场景需单独预注册流式工具和预算,不能截断后称全量。
+The inventory pages through every version and marker, GETs the full content, tags, and public metadata for each versionId, and then cross-checks them with a paged current LIST. The object limit is 10,000 and a single object is limited to 64MiB. The batch byte budget comes from the binding file. A larger-object scenario must pre-register a streaming tool and a budget of its own. A truncated result must not be called a complete inventory.
-并发、生命周期、合法删除场景的预期清单会变化:人工先按逐请求 ACK 因果关系生成 `expected-post.json`,记录每项增删对应的 ACK 文件;再比较实际清单,不能直接拿静态基线判失败或拿实际输出复制成期望。未知写入未解析则不可通过。加密/归档对象先用授权 GET/restore 参数逐版本取证;默认 inventory 不会自动解锁、传 SSE-C 或 restore。恢复型 GET 必须保存密文/明文两层对应关系。
+For concurrency, lifecycle, and legitimate-deletion scenarios, the expected inventory changes. A person first generates `expected-post.json` from the causal relationship of the per-request ACKs, recording the ACK file that corresponds to each addition or deletion, and then compares the actual inventory. Failure must not be judged directly from the static baseline, and the actual output must not be copied into the expectation. An unresolved unknown write cannot pass. Encrypted or archived objects are evidenced version by version first, using authorized GET or restore parameters. The default inventory does not automatically unlock an object, supply SSE-C, or restore. A restore-type GET must save the correspondence between the ciphertext and plaintext layers.
-### Range、HEAD、条件请求
+### Range, HEAD, and conditional requests
```bash
hp head s3 --operation head_object --params-file request-head.json
@@ -194,13 +194,13 @@ hp if-match-reject s3 --operation get_object --expect-http 412 --params-file req
hp range-outside s3 --operation get_object --expect-http 416 --params-file request-range-outside.json
```
-JSON 都包含本例 Bucket/Key/可选 VersionId;另外分别添加 `Range:"bytes=5242879-5242882"`、`IfNoneMatch:<原ETag>`、`IfMatch:<错误但语法合法ETag>`、超出对象长度的 Range。时间条件用 `IfModifiedSince`/`IfUnmodifiedSince` 的 ISO 日期;从 HEAD 的真实 LastModified 推导早/晚两个控制值。验证 ContentRange、响应长度和从确定性原文计算的片段 hash,不能只验状态码。
+Every JSON object contains this case's Bucket, Key, and optional VersionId. In addition, add respectively `Range:"bytes=5242879-5242882"`, `IfNoneMatch:`, `IfMatch:`, and a Range past the object length. Time conditions use ISO dates for `IfModifiedSince` and `IfUnmodifiedSince`. Derive one earlier and one later control value from the real LastModified returned by HEAD. Verify ContentRange, the response length, and the fragment hash computed from the deterministic original content. Do not verify only the status code.
-### SSE、Object Lock 和外部依赖
+### SSE, Object Lock, and external dependencies
```bash
hp sse-s3-put s3 --operation put_object --size 5243003 --seed "$RUN/sse" --params-file sse-s3-put.json
@@ -213,17 +213,17 @@ hp hold s3 --operation get_object_legal_hold --params-file protected-version.jso
hp reject-delete s3 --operation delete_object --expect-http 403 --params-file protected-version.json
```
-SSE JSON 分别加 `ServerSideEncryption:"AES256"`、`ServerSideEncryption:"aws:kms",SSEKMSKeyId:<本例专用key>`、`SSECustomerAlgorithm:"AES256",SSECustomerKey:"$HEAL_SSE_C_KEY"`。SSE-C GET 也提供同一密钥,先用原请求证明可解密。所有外部 key/endpoint 是经过审批的测试租户,不复用生产依赖。
+The SSE JSON adds, respectively, `ServerSideEncryption:"AES256"`, `ServerSideEncryption:"aws:kms",SSEKMSKeyId:`, and `SSECustomerAlgorithm:"AES256",SSECustomerKey:"$HEAL_SSE_C_KEY"`. An SSE-C GET also supplies the same key. First prove that the object can be decrypted with the original request. Every external key and endpoint is an approved test tenant. Do not reuse a production dependency.
-锁桶 JSON 为 `{Bucket:...,ObjectLockEnabledForBucket:true}`。受保护 PUT 使用 `ObjectLockMode:"GOVERNANCE"` 或 `"COMPLIANCE"`、实际未来 UTC `ObjectLockRetainUntilDate`,或 `ObjectLockLegalHoldStatus:"ON"`。必须验证无 bypass 角色在保留期间真的不能删除;脚本拒绝 BypassGovernanceRetention=true。Legal hold 解除使用 `put_object_legal_hold` 的 `LegalHold:{Status:"OFF"}`,仅作用于本例版本。Compliance 等真实到期,不改主机时间、不内部删文件。
+The lock-bucket JSON is `{Bucket:...,ObjectLockEnabledForBucket:true}`. A protected PUT uses `ObjectLockMode:"GOVERNANCE"` or `"COMPLIANCE"`, an actual future UTC `ObjectLockRetainUntilDate`, or `ObjectLockLegalHoldStatus:"ON"`. It must be verified that a role without bypass really cannot delete during the retention period. The script rejects BypassGovernanceRetention=true. A legal hold is removed with `LegalHold:{Status:"OFF"}` of `put_object_legal_hold`, and it applies only to this case's version. Compliance waits for real expiry. Do not change the host clock, and do not delete files internally.
-KMS/tier outage 使用 NETWORK 在本例专用 net namespace 内仅阻断对应依赖的 IP/port,或仅撤销本例租户凭据;恢复原规则/权限后重试原 version。tier 缺失一个远端版本可通过远端本例桶 API 删除并保留ACK/原备份,但这是“远端对象丢失”负例,不等同网络中断。
+A KMS/tier outage uses NETWORK to block only the corresponding dependency's IP and port inside this case's dedicated net namespace, or revokes only this case's tenant credentials. After the original rules and permissions are restored, retry the original version. One missing remote version on a tier may be deleted through this case's remote bucket API, with the ACK and the original backup retained, but that is a "remote object lost" negative case and is not the same as a network interruption.
-## 4. Admin 触发、状态与窗口
+## 4. Admin triggers, status, and windows
-实际服务配置要取证,但**不要转储包含凭据的全环境**。manual 模式记录以下非敏感字段:
+The actual service configuration must be captured as evidence, but **do not dump the full environment when it contains credentials**. Manual mode records the following non-sensitive fields:
```text
RUSTFS_HEAL_ENABLED=true
@@ -233,7 +233,7 @@ RUSTFS_HEAL_MRF_ENABLE=false
RUSTFS_PUT_RENAME_EARLY_ACK_ENABLE=false
```
-scanner/自动/MRF 场景按操作卡逐项开启相应字段,并冻结 scanner cycle/start delay/bitrot cycle。配置变更只改专用 service.env,重启本例 unit 后证明实际生效。read-repair 所需队列依赖按候选源码核实,不能为了“排除后台”关掉被测路径。
+Scanner, automatic, and MRF scenarios enable the corresponding fields item by item according to the procedure card, and freeze the scanner cycle, start delay, and bitrot cycle. A configuration change modifies only the dedicated service.env. After this case's unit is restarted, prove that the change actually took effect. Queue dependencies required by read-repair are verified against the candidate source. The path under test must not be turned off in order to "exclude the background".
@@ -241,9 +241,9 @@ scanner/自动/MRF 场景按操作卡逐项开启相应字段,并冻结 scanne
-### 原始 Admin 命令
+### Raw Admin commands
-`SCOPE_PATH` 为编码后的 `/rustfs/admin/v3/heal//`;key 的每个字节按 URL 编码,root 为 `/rustfs/admin/v3/heal/`。JSON 中 pool/set 是实际零基索引,不能在 H5 固定填 0。例:
+`SCOPE_PATH` is the encoded `/rustfs/admin/v3/heal//`. Every byte of the key is URL-encoded, and root is `/rustfs/admin/v3/heal/`. pool and set in the JSON are the actual zero-based indexes, and they must not be fixed at 0 on H5. Example:
```bash
hp start admin --path "$SCOPE_PATH" \
@@ -252,12 +252,12 @@ TOKEN=$(jq -er '.observation.response.clientToken | select(type=="string" and le
hp status-001 admin --path "$SCOPE_PATH" --query "clientToken=$TOKEN"
hp stop admin --path "$SCOPE_PATH" --query "clientToken=$TOKEN" --query forceStop=true
hp force-start admin --path "$SCOPE_PATH" --query forceStart=true --body-file heal-options.json
-# 保留重复 query,真正送到服务端验证:
+# Keep the duplicate query and actually send it to the server for verification:
hp invalid-duplicate admin --path "$SCOPE_PATH" --query clientToken=a \
--query clientToken=b --expect-http 400
```
-WAIT 用有界循环保存每一响应,不仅保存最后绿色状态;普通 repair 900 秒、重启后 1800 秒是起始测试预算,须预注册,不是产品 SLO:
+WAIT uses a bounded loop to save every response, not only the last green status. 900 seconds for an ordinary repair and 1800 seconds after a restart are starting test budgets. They must be pre-registered, and they are not a product SLO:
```bash
for i in $(seq 1 900); do
@@ -269,9 +269,9 @@ for i in $(seq 1 900); do
done
```
-循环退出后人工/候选 adapter 检查:execution completed、coverage complete;failed/unknown/attemptFailures=0;不溢出;逐对象 disposition 与设计匹配;非 dry-run 的 incarnation/版本/pool/set 正确。dry-run 只允许 observed,healed=0,随后无物理变更。负例以预注册错误/未完成契约判,不套正向 completed。
+After the loop exits, a person or the candidate adapter checks: execution completed and coverage complete; failed/unknown/attemptFailures=0; no overflow; per-object disposition matches the design; and, for a non-dry-run, incarnation, version, pool, and set are correct. A dry-run allows only observed, with healed=0 and no subsequent physical change. A negative case is judged by the pre-registered error or incomplete contract. The positive completed outcome is not applied to it.
-`summary=finished` 或 HTTP200 单独不够;`truncated=false` 可以省略,null/true 不算 false;status.settings 是查询默认值,不代表原任务实际 scanMode。outcome 的 128 项窗口不是完整对象清单。当前 rc 会简化部分字段,以 raw Admin + 独立 manifest 为准。
+`summary=finished` or HTTP 200 alone is not sufficient. `truncated=false` may be omitted, and null or true does not count as false. status.settings is the query default and does not represent the original task's actual scanMode. The 128-item window of an outcome is not a complete object inventory. The current rc simplifies some fields. Raw Admin plus an independent manifest is authoritative.
@@ -283,12 +283,12 @@ hp replacement admin --method GET --path /rustfs/admin/v4/heal/replacement-recov
"$RC_BIN" --json --no-color admin heal status "$RC_ALIAS"
```
-rc 必须是本批冻结二进制,私有 RC_CONFIG_DIR/alias 只指向本例服务,stdout/stderr/exitCode 都留存。查询其他节点:直接运行 heal_probe 的全局 `--endpoint n1`/`--endpoint stable`;保存实际连接 peer 身份。健康状态需要 expected/responded/unknown 与所有 peer 覆盖,不允许缺 peer 时 idle 假绿。
+The rc must be this batch's frozen binary. The private RC_CONFIG_DIR and alias point only at this case's service, and stdout, stderr, and exitCode are all retained. To query another node, run heal_probe's global `--endpoint n1` or `--endpoint stable` directly, and save the identity of the peer that was actually connected. A healthy status requires expected, responded, and unknown, together with coverage of every peer. An idle false green when a peer is missing is not allowed.
-### 部分进度与增量游标
+### Partial progress and incremental cursors
```bash
hp cursor-001 admin --path "$SCOPE_PATH" --query "clientToken=$TOKEN" --query sinceSeq=0
@@ -296,209 +296,209 @@ NEXT=$(jq -er '.observation.response.nextSeq - 1' "$HEAL_REPORT/cursor-001.json"
hp cursor-002 admin --path "$SCOPE_PATH" --query "clientToken=$TOKEN" --query "sinceSeq=$NEXT"
```
-WINDOW 的必要证据是 running 且 `0`, combined with a snapshot of the target shards proving that files have indeed not yet been created. Save the observation response, the trigger command, and the target PID and generation. When processed alone is insufficient, an explicit target completed/expected set is required. Status and sampling latency must be shorter than the fault window. If it exceeds that window, judge NOT_OBSERVED. Do not unconditionally kill the process after a fixed sleep, and do not rerun without bound until a hit happens to occur.
-CURSOR 逐响应保存 nextSeq/minSeq/truncated/items;过旧 cursor 的 gap 是预期观测项,不可静默丢弃。跨节点和重启后按 task/epoch分组,有界窗口之外用完整manifest补证。DX 257 是起始数据量;若仍过快,下一批预注册增加负载或受限故障层,而不是本批临时改分母。
+CURSOR saves nextSeq, minSeq, truncated, and items for every response. A gap from a cursor that is too old is an expected observation and must not be silently discarded. After cross-node operation and after a restart, group by task and epoch, and supplement evidence outside the bounded window with the complete manifest. DX 257 is the starting data volume. If it is still too fast, the next batch pre-registers additional load or a restricted fault layer, instead of temporarily changing the denominator in this batch.
-固定候选的 `sinceSeq` 是排他游标,`nextSeq` 是下一待分配序号,因此完整响应之后传 `nextSeq - 1`。上例只适用于本次 items 完整返回;若 payload 本身裁剪了 items,只推进到 `max(sinceSeq + 1, minSeq) + 实际收到项数 - 1`,继续读取未返回后缀。不要把响应裁剪或保留缺口隐藏在 nextSeq 跳跃中。J03 自动执行器预注册 1025 个 1MiB+73 字节的外部对象,实际越过该候选 1024 项的 item retention;128 项 outcome 诊断窗口另行验证。每节点物理清单显式限制 16384 文件,每批独立 streaming 校验最多 128 个 part。未观察到两个有效 running 增量窗口时记录 NOT_OBSERVED,完整用例不能 PASS。H1 的跨节点分支明确不适用,H2 查询所有四个实际节点。
+For a fixed candidate, `sinceSeq` is an exclusive cursor and `nextSeq` is the next sequence number to be assigned, so after a complete response pass `nextSeq - 1`. The example above applies only when this response's items were returned in full. If the payload itself trimmed the items, advance only to `max(sinceSeq + 1, minSeq) + actual number of items received - 1`, and continue reading the suffix that was not returned. Do not hide response trimming, or a retained gap, inside a nextSeq jump. The J03 automatic executor pre-registers 1025 external objects of 1MiB+73 bytes, which actually exceeds that candidate's item retention of 1024 items. The 128-item outcome diagnostic window is verified separately. Each node's physical inventory is explicitly limited to 16384 files, and each batch's independent streaming verification covers at most 128 parts. When two valid running incremental windows are not observed, record NOT_OBSERVED, and the complete case cannot PASS. H1's cross-node branch is explicitly inapplicable. H2 queries all four actual nodes.
-### 权限和过期签名
+### Permissions and expired signatures
-在专用服务先保存当前 rc 的 `admin user --help`、`admin policy --help`、`admin heal --help`。使用候选 IAM 支持的 Admin Heal action 创建最小策略(从候选 `HealAdminAction` 的实际字符串导出,不猜拼写);另建明确不含该 action 的角色。示例用户/策略动作按本批 rc help 的参数顺序执行并保存原始回执,凭据用私有输入通道,不开 shell trace。
+On the dedicated service, first save the current rc's `admin user --help`, `admin policy --help`, and `admin heal --help`. Create a minimal policy that uses the Admin Heal action supported by the candidate IAM, exported from the actual string of the candidate `HealAdminAction` without guessing the spelling, and separately create a role that explicitly does not contain that action. Example user and policy actions are executed in the parameter order given by this batch's rc help, and the raw receipts are saved. Credentials use a private input channel, and shell trace is not enabled.
-已核对参考候选 action 为 `admin:Heal`,最小策略主体为 `{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Action":["admin:Heal"],"Resource":["arn:aws:s3:::*"]}]}`;RustFS policy parser 要求带类型的 ARN,裸 `"*"` 会被拒绝。将这个权限仅赋予本例临时身份,不改共享管理员策略。新候选仍需核对 action 和 rc attach 语法。
+The reference candidate action that has been checked is `admin:Heal`, and the minimal policy body is `{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Action":["admin:Heal"],"Resource":["arn:aws:s3:::*"]}]}`. The RustFS policy parser requires a typed ARN, and a bare `"*"` is rejected. Grant this permission only to this case's temporary identity. Do not change the shared administrator policy. A new candidate still needs its action and the rc attach syntax checked.
```bash
hp anonymous-start admin --path "$SCOPE_PATH" --auth anonymous --expect-http 403 --body-file heal-options.json
hp bad-signature admin --path "$SCOPE_PATH" --auth bad-signature --expect-http 403 --body-file heal-options.json
-# 无权限/最小权限实例:更换本次调用环境中的 HEAL_ACCESS_KEY/HEAL_SECRET_KEY,执行相同请求。
+# No-permission or minimal-permission instance: replace HEAL_ACCESS_KEY and HEAL_SECRET_KEY in this invocation's environment, and execute the same request.
```
-过期签名:保存一次已签请求,在签名有效期外重放到**相同本例地址/路径/请求体**,仅等待客户端签名超期,不改变服务器时钟。先用新签名同请求做控制。时限按候选签名校验窗口预注册;签名本身含敏感身份,原请求存私有0600文件,只归档脱敏结果和请求摘要。对 start/status/stop 都测试;token 不能替代权限。结束时通过正常 IAM API 删除本例用户/策略并验证旧凭据失效。
+Expired signatures: save one signed request and, outside the signature validity period, replay it to the **same address, path, and request body of this case**. Only wait for the client signature to expire. Do not change the server clock. First run the same request with a new signature as a control. The time limit is pre-registered from the candidate's signature-verification window. The signature itself contains sensitive identity. Store the original request in a private mode-0600 file, and archive only the redacted result and the request digest. Test start, status, and stop. A token cannot stand in for a permission. At the end, delete this case's user and policy through the normal IAM API and verify that the old credentials are no longer valid.
-## 5. 物理证据与故障后端
+## 5. Physical evidence and fault backends
-### 真实布局、角色和物理快照
+### Real layout, roles, and physical snapshots
-在目标节点逐盘读取本例 `data/.rustfs.sys/format.json`(只读)及 mount UUID/major:minor,保存 pool/set/this 映射。不能把同一设备的多个目录作为独立盘。逐对象 xl.meta 使用与候选相同源码的只读解析器:
+On the target node, read this case's `data/.rustfs.sys/format.json` (read-only) and the mount UUID and major:minor on each disk, and save the pool/set/this mapping. Multiple directories on the same device must not be treated as independent disks. Per-object xl.meta uses a read-only parser built from the same source as the candidate:
```bash
python scripts/validation_heal/prepare_heal_meta_inspector.py --source "$CANDIDATE_SOURCE" \
--commit "$CANDIDATE_COMMIT" --out "$INSPECTOR_BUILD"
-# 仅批准的构建机;生成独立 helper 的锁文件后冻结,和服务构建清单分开。
+# Approved build machine only; after the independent helper's lockfile is generated, freeze it separately from the service build inventory.
cargo build --release --manifest-path "$INSPECTOR_BUILD/Cargo.toml"
sha256sum "$INSPECTOR_BUILD/Cargo.lock" "$INSPECTOR_BUILD/target/release/heal-meta-inspect"
"$INSPECTOR_BUILD/target/release/heal-meta-inspect" "$XL_META_EVIDENCE_COPY" "$B" "$K"
```
-解析器不输出用户metadata或密钥,只输出版本、data_dir、parts、EC参数/分片角色。首先对同批健康样本和损坏样本分别证明成功/拒绝;不支持的版本不得换别的版本解析器后宣称同候选验证。源 checkout 必须为固定 SHA 且无已跟踪修改。
+The parser does not emit user metadata or keys. It emits only versions, data_dir, parts, and EC parameters and shard roles. First prove success on the healthy samples from the same batch, and refusal on the damaged samples. An unsupported version must not be declared verified for the same candidate after switching to a parser for another version. The source checkout must be a fixed SHA with no tracked modifications.
```bash
-# 目标节点。所有输出重定向到本例证据文件,不能覆盖已有阶段。
+# Target node. Redirect all output to this case's evidence files. An existing stage must not be overwritten.
hf inspect --path "$TARGET_RELATIVE_PATH"
hf snapshot --bucket "$B"
```
-快照对整桶每个文件保存长度/hash,目录描述符逐层拒绝 symlink/hardlink/不同mount ID(含同设备bind mount);最多10,000文件/8GiB,每文件64MiB。四阶段 before/fault/after/restart 每盘各一份。外部part与**同一个角色、同版本的健康原片**字节完全相同是独立重建对照;数据应不重复且四片摘要不同。不得读取 quarantine 来辅助服务器重建。
+A snapshot saves the length and hash of every file in the whole bucket. Directory descriptors reject, level by level, a symlink, a hardlink, or a different mount ID, including a bind mount of the same device. The limit is 10,000 files and 8GiB, with 64MiB per file. Each of the four stages before, fault, after, and restart has one copy per disk. An external part that is byte-for-byte identical to the **healthy original shard of the same role and the same version** is the independent rebuild control. The data should not be duplicated, and the four shards' digests should differ. Do not read quarantine to assist the server rebuild.
-xl.meta 可合法重写;比较解析后的版本/引用,而非要求所有内部字节不变。允许的 xl.meta.bkp 必须属于同目标data_dir且摘要等于原xl.meta,不能泛化豁免任意额外文件。普通解析模式不验证 payload;`--verify-part XL_META BUCKET KEY PART_FILE VERSION_ID PART_NUMBER` 按候选 metadata 的实际版本、part 和 legacy/current framing 逐块验证 HighwayHash,同时拒绝短读、尾随字节和不支持的格式。分片执行器还逐项要求所有同版本片与故障前原始同角色长度/hash 完全一致,并确认基线全文、真实 set slot 及不重复分片摘要。它不支持加密/压缩、inline 或改变编码形态的物理验收;这些用例仍保留各自工具要求。
+xl.meta may be legally rewritten. Compare the parsed versions and references, rather than requiring every internal byte to stay unchanged. An allowed xl.meta.bkp must belong to the same target data_dir, and its digest must equal the original xl.meta. That allowance must not be generalized to exempt an arbitrary extra file. Ordinary parse mode does not verify the payload. `--verify-part XL_META BUCKET KEY PART_FILE VERSION_ID PART_NUMBER` verifies HighwayHash block by block according to the actual version, part, and legacy/current framing in the candidate metadata, and it also rejects a short read, trailing bytes, and an unsupported format. The shard executor further requires, item by item, that every same-version shard exactly match the pre-fault original length and hash of the same role, and it confirms the full baseline content, the real set slot, and non-duplicate shard digests. It does not support physical acceptance of encryption, compression, inline data, or a changed encoding shape. Those cases still keep their own tool requirements.
-### 单文件故障(已有脚本)
+### Single-file faults (existing script)
-先停止本例写入并让测试进程静止;格式映射已确认后选择精确物理路径,不能只找第一个part。注入/恢复时 `--service-quiescent` 是操作者确认,工具不会替你停止共享服务。
+First stop this case's writes and make the test processes quiescent. After the format mapping has been confirmed, select the exact physical path. Do not just find the first part. On inject and on recovery, `--service-quiescent` is the operator's confirmation. The tool will not stop the shared service for you.
```bash
hf apply --path "$TARGET_RELATIVE_PATH" --fault-id C02-data-middle \
--kind bitflip --offset "$BYTE_OFFSET" --expected-sha256 "$BASELINE_SHA256" --service-quiescent
-# 其他实例各用新故障ID:--kind missing / truncate / zero / header
+# Each other instance uses a new fault ID: --kind missing / truncate / zero / header
hf snapshot --bucket "$B"
```
-`truncate` 保留一半;`header` 翻转前32字节;bitflip 可指定任意有效offset;zero 若不会改变字节则拒绝。先持久化intent/原件,再变更,再见证;重复apply拒绝覆盖。备份在同run隔离目录,绝不在RustFS数据根中。
+`truncate` keeps half; `header` flips the first 32 bytes; bitflip may specify any valid offset; zero is rejected if it would not change any byte. First persist the intent and the original, then apply the change, then witness it. A repeated apply refuses overwrite. The backup is in the same run's quarantine directory, and never inside a RustFS data root.
-文件故障为 F1/F2/对象级F3,不是整盘故障。helper 明确禁止 `.rustfs.sys`、format、IAM、任意shell或设备操作。用例需要系统记录/特殊格式时走下面人工白名单后端。
+File faults are F1, F2, and object-level F3, not whole-disk faults. The helper explicitly forbids `.rustfs.sys`, format, IAM, and any shell or device operation. When a case needs a system record or a special format, use the manual allowlist backend below.
-### 格式/系统记录故障(人工白名单后端)
+### Format and system-record faults (manual allowlist backend)
-先 `PROCESS stop`,从候选源码/真实task定位**本例**精确记录,生成 `record-plan.json`:绝对路径、run/task/bucket/incarnation、盘UUID、原sha/size、变更offset或替换源sha、允许新增/删除字段、备份和undo。第二人/批准的操作人逐项审核;禁止递归 `.rustfs.sys` 修改。
+First `PROCESS stop`. Locate **this case's** exact record from the candidate source or the real task, and generate `record-plan.json`: the absolute path, run/task/bucket/incarnation, disk UUID, original sha and size, the change offset or the replacement-source sha, the fields allowed to be added or deleted, the backup, and the undo. A second person, or an approved operator, reviews each item. Recursive modification of `.rustfs.sys` is forbidden.
-可操作的配方:
+Operable recipes:
-1. **near-tail/长度不符**:复制目标part到本例quarantine并校验sha;在该目标副本上 `truncate -s "$NEW_SIZE" "$TARGET"`,其中 NEW_SIZE=原size−1或指定长度。核对新size/hash和原metadata未变。停止服务期间执行,目标路径来自已审核清单。
-2. **外来format/set**:同run的第二个隔离小集群正常启动产生format,停止两服务。仅把其format安装到本例空替换根,不改健康盘;记录原/外来id、this、sets。启动后期待安全拒绝,原外来format不能被改写。
-3. **未来format版本**:在本例替换根的format JSON副本,用 `jq` 只更改已核实的版本字段为不支持值;原件与变更diff归档。未来对象/checkpoint schema 按候选编解码结构更改对应版本头,不能随机翻位后称“未来schema”。如果存在校验,必须重算后证明结构可解析且schema未知;否则只计坏校验实例。
-4. **冲突代次/quorum**:串行合法写V1/V2或两个incarnation,分别保存全套本例副本;停服务后按预注册盘分配表安装各自不足R的集合。重启前用FORMAT逐盘核对,证明没有任一同版本quorum。
-5. **checkpoint/MRF坏记录**:只读保存 `.rustfs.sys/root-heal-.json` 与候选实际journal路径;确认属于本run。截断、替换为从本run早期阶段保留的陈旧代次、或单独更改校验字段。不得凭目录名删除整个queue。写失败通过IO_FAULT构造,不改权限后未经见证就称失败。
-6. **bucket metadata**:只列入对应本例桶的记录;建立同run健康备份,分别缺失/截断后重启/heal。原策略、tags、版本/lock配置通过S3核对,其他系统文件摘要不变。
-7. **inline偏移**:用匹配候选的codec在证据副本中定位目标version的inline map/value;保存解析offset/长度及反向映射,之后只修改该value。不会解析时不能任意翻xl.meta头冒充inline损坏;改用已明确定义的“坏metadata”实例,原inline实例仍未覆盖。
-8. **orphan/dangling**:通过正常完成/覆盖/删除形成历史目录,FORMAT证明所有活version/MPU引用后再测试remove。宽限期按真实时间/本例服务配置等待,不改主机时间。共享引用控制必须保留。
+1. **near-tail / length mismatch**: copy the target part into this case's quarantine and verify the sha. On that target copy, run `truncate -s "$NEW_SIZE" "$TARGET"`, where NEW_SIZE is the original size−1 or a specified length. Check the new size and hash, and check that the original metadata is unchanged. Execute this while the service is stopped. The target path comes from the reviewed inventory.
+2. **Foreign format/set**: a second isolated small cluster of the same run starts normally and produces a format, and then both services are stopped. Install only that format onto this case's empty replacement root. Do not change a healthy disk. Record the original and foreign ids, this, and sets. After startup, expect a safe refusal. The original foreign format must not be rewritten.
+3. **Future format version**: on the format JSON copy in this case's replacement root, use `jq` to change only the verified version field to an unsupported value. Archive the original and the change diff. A future object or checkpoint schema changes the corresponding version header according to the candidate's codec structure. Do not flip bits at random and then call the result a "future schema". If a checksum exists, it must be recomputed and then used to prove that the structure is parseable and that the schema is unknown; otherwise count only the bad-checksum instance.
+4. **Conflicting generations/quorum**: serially perform valid writes of V1 and V2, or of two incarnations, and save a complete set of this case's copies for each. After the service is stopped, install each set that is short of R according to the pre-registered disk assignment table. Before restart, check disk by disk with FORMAT and prove that no same-version quorum exists.
+5. **Bad checkpoint/MRF records**: read-only save `.rustfs.sys/root-heal-.json` and the candidate's actual journal path, and confirm that they belong to this run. Truncate them, replace them with a stale generation retained from an early stage of this run, or change only the checksum field. Do not delete an entire queue on the basis of a directory name. A write failure is constructed through IO_FAULT. Do not change permissions and then call it a failure without a witness.
+6. **bucket metadata**: include only the records that correspond to this case's bucket. Establish a healthy backup of the same run, then restart and heal after a missing case and after a truncated case respectively. Check the original policy, tags, and version and lock configuration through S3. The digests of the other system files stay unchanged.
+7. **inline offset**: use a codec that matches the candidate to locate the target version's inline map and value in the evidence copy. Save the parsed offset and length and the reverse mapping, and then modify only that value. If it cannot be parsed, do not arbitrarily flip the xl.meta header and present that as inline damage. Use the already defined "bad metadata" instance instead. The original inline instance remains uncovered.
+8. **orphan/dangling**: form historical directories through normal completion, overwrite, and deletion. After FORMAT proves every live version and MPU reference, test remove. The grace period waits according to real time and this case's service configuration. Do not change the host clock. The shared-reference control must be retained.
-上述人工操作每一步均需精确原件备份、变更前后hash、操作返回值与复查;不能直接 `rm` 记录。用 `mv -n` 同盘移到隔离路径可保留原件,但必须核实源消失/目标摘要相同,因为 `mv -n` 跳过未必报错。恢复时先保留失败后新记录,若有新ACK/代次,禁止把旧metadata拷回去。
+Every step of the manual operations above requires an exact backup of the original, hashes before and after the change, the operation's return value, and a recheck. Do not `rm` a record directly. Moving with `mv -n`, on the same disk, to a quarantine path can retain the original, but the disappearance of the source and the identical target digest must be verified, because an `mv -n` skip does not necessarily report an error. On recovery, first retain the new records produced after the failure. If there is a new ACK or generation, copying the old metadata back is forbidden.
-### 逻辑盘不可达、只读、权限
+### Logical disk unreachable, read-only, and permissions
-仅在已设置 PrivateMounts 的**本例服务 mount namespace**使用;先证明目标PID仍属于本例unit、`/proc/$PID/ns/mnt` 与host不同,且目标挂载非shared。不能进入host namespace卸载 `/data/rustfsN`。原理参考 [Linux nsenter](https://www.man7.org/linux/man-pages/man1/nsenter.1.html)。
+Use this only inside **this case's service mount namespace** where PrivateMounts is already set. First prove that the target PID still belongs to this case's unit, that `/proc/$PID/ns/mnt` differs from the host, and that the target mount is not shared. Do not enter the host namespace to unmount `/data/rustfsN`. For the principle, see [Linux nsenter](https://www.man7.org/linux/man-pages/man1/nsenter.1.html).
-只读配方(在目标节点,DATA为本例完整数据根):
+Read-only recipe (on the target node; DATA is this case's complete data root):
```bash
sudo nsenter -t "$HEAL_PID" -m -- mount --bind "$DATA" "$DATA"
sudo nsenter -t "$HEAL_PID" -m -- mount -o remount,bind,ro "$DATA"
sudo nsenter -t "$HEAL_PID" -m -- findmnt -T "$DATA" -o TARGET,SOURCE,OPTIONS
-# 以服务实际UID在该namespace做本例临时文件写探针,必须观察EROFS。
-# 解除仅此额外的bind层:
+# Using the service's actual UID, write-probe a temporary file of this case in that namespace; EROFS must be observed.
+# Remove only this extra bind layer:
sudo nsenter -t "$HEAL_PID" -m -- umount "$DATA"
```
-不可达用经批准的该服务 mount namespace 故障层覆盖此数据根(返回EIO的专用device-mapper设备,见IO_FAULT);不能把空目录覆盖当作掉盘,因为可能触发新盘初始化。权限实例可使用本例非root UID +精确目录ACL拒绝,在相同UID实测EACCES;root服务单纯chmod不算有效注入。恢复后重新验证mount/磁盘身份和读写可达;在 namespace 内 helper 的跨mount拒绝是正确保护,不应关闭它。
+For unreachable, use an approved fault layer in that service's mount namespace to cover this data root (a dedicated device-mapper device that returns EIO; see IO_FAULT). Do not treat covering the root with an empty directory as a lost disk, because that may trigger new-disk initialization. A permission instance may use this case's non-root UID plus an exact directory ACL denial, and EACCES is measured under the same UID. A bare chmod against a root service does not count as a valid injection. After recovery, re-verify the mount and disk identity and that read and write are reachable. The helper's cross-mount refusal inside the namespace is correct protection, and it should not be turned off.
-### 新逻辑盘替换
+### New logical-disk replacement
-停止本例unit及写入,校验 ROOT/data 与owner/run/mount;将**精确 data 根**同盘移至本例 quarantine 的唯一 old-disk 目录,新建同位置空data根。其他盘不动。重启后读取format,确认新disk UUID与slot;格式初始化失败要如实记录,不手动补出一个UUID。
+Stop this case's unit and its writes, and check ROOT/data against the owner, the run, and the mount. Move the **exact data root**, on the same disk, into the unique old-disk directory of this case's quarantine, and create a new empty data root in the same location. Leave the other disks untouched. After restart, read format and confirm the new disk UUID and slot. If format initialization fails, record it as it happened. Do not manually supply a UUID.
```bash
-# 仅已审核的本例 ROOT;先验证备份目标不存在,不递归删除任何目录。
+# Only this case's reviewed ROOT; first verify that the backup target does not exist. Do not recursively delete any directory.
sudo mv -n -- "$ROOT/data" "$ROOT/quarantine/$FAULT_ID-old-disk"
sudo install -d -m 0700 -- "$ROOT/data"
```
-检查旧根确实移走、摘要清单完整、服务无法访问quarantine;再正常启动。rebuild期间不得从old-disk复制part。失败救援时先停止unit并把新失败data根移至另一个保留目录,再恢复旧根;该恢复明确为rescue,不是heal结果。真实设备detach/attach仅属于另行批准的H6,不由此命令模拟。
+Check that the old root has really been moved away, that the digest inventory is complete, and that the service cannot access quarantine; then start normally. During rebuild, do not copy parts from old-disk. For a failure rescue, first stop the unit and move the new failed data root to another retained directory, then restore the old root. That recovery is explicitly a rescue, not a heal result. Real device detach and attach belong only to a separately approved H6, and they are not simulated by this command.
-### 进程停止、崩溃、重启、升级
+### Process stop, crash, restart, and upgrade
```bash
sudo systemctl show "$HEAL_UNIT" -p MainPID -p ExecMainStartTimestampMonotonic -p FragmentPath -p ActiveState
sudo systemctl stop "$HEAL_UNIT"
sudo systemctl start "$HEAL_UNIT"
-# 独立 crash 实例,必须先 WINDOW、记录被杀的实际MainPID:
+# Independent crash instance; WINDOW must come first, and the actual MainPID that is killed must be recorded:
sudo systemctl kill --kill-whom=main --signal=SIGKILL "$HEAL_UNIT"
sudo systemctl show "$HEAL_UNIT" -p ExecMainCode -p ExecMainStatus -p Result
sudo systemctl reset-failed "$HEAL_UNIT"
sudo systemctl start "$HEAL_UNIT"
```
-每次操作前核对 MainPID/exe 摘要/starttime,拒绝0或已替换PID,不 `pkill rustfs`。保存 `/proc/sys/kernel/random/boot_id`,SIGKILL前后boot_id相同,不能说断电。
+Before every operation, check the MainPID, the exe digest, and starttime. Refuse 0 or a PID that has already been replaced. Do not `pkill rustfs`. Save `/proc/sys/kernel/random/boot_id`. The boot_id is the same before and after SIGKILL, so this must not be called a power loss.
-UPGRADE 对本例节点逐一stop,用已经验证的第二个optimized release二进制替换该unit的exec路径(保留旧二进制),原env/endpoint/data不变,start后核对每个peer实际版本。旧新两份构建清单、包sha、库格式兼容表都须预注册;未知格式禁止直接回滚,改测安全拒绝。精确升级期间责任按CHECKPOINT验证,不依赖版本字符串判断兼容。
+UPGRADE stops this case's nodes one by one and replaces that unit's exec path with a second, already verified, optimized release binary, retaining the old binary. The original env, endpoint, and data stay unchanged. After start, check each peer's actual version. Both the old and the new build inventories, the package shas, and the library-format compatibility table must be pre-registered. An unknown format forbids a direct rollback; test safe refusal instead. Responsibility during the exact upgrade is verified according to CHECKPOINT. Compatibility is not judged from a version string.
-### 测试端口分区 / 依赖故障
+### Test-port partition / dependency fault
-优先预分配专用net namespace,证明它与host不同;需要业务节点连接的路由已配置,SSH仍走host。只对本例API/RPC端口或本例KMS/tier endpoint加规则。先保存规则/路由,设置独立超时恢复任务。例:
+Prefer a pre-allocated dedicated net namespace, and prove that it differs from the host. Routes required for business-node connectivity are already configured, and SSH still goes through the host. Add rules only for this case's API/RPC port, or for this case's KMS/tier endpoint. First save the rules and routes, and set an independent timeout-recovery task. Example:
```bash
sudo nsenter -t "$HEAL_PID" -n -- iptables -w -I OUTPUT 1 \
-d "$TEST_PEER_IP" -p tcp --dport "$TEST_SERVICE_PORT" \
-m comment --comment "$FAULT_ID" -j DROP
sudo nsenter -t "$HEAL_PID" -n -- iptables -w -nvL OUTPUT --line-numbers
-# 相同namespace、完整相同规则删除,不flush任何链:
+# In the same namespace, delete the fully identical rule. Do not flush any chain:
sudo nsenter -t "$HEAL_PID" -n -- iptables -w -D OUTPUT \
-d "$TEST_PEER_IP" -p tcp --dport "$TEST_SERVICE_PORT" \
-m comment --comment "$FAULT_ID" -j DROP
```
-双向分区分别在绑定双方namespace应用,不能扩大到22、管理平面或host OUTPUT。记录丢包计数与双端探针。无独立net namespace或专项host规则批准时该实例BLOCKED,不能全局断网凑结果。
+A bidirectional partition is applied in the namespaces of both bound parties, and it must not be expanded to port 22, the management plane, or host OUTPUT. Record the drop counters and the probes at both ends. When there is no independent net namespace, or no approval for a dedicated host rule, that instance is BLOCKED. Do not disconnect the network globally in order to force a result.
-### 慢IO/EIO/容量与锁窗口
+### Slow IO, EIO, capacity, and the lock window
-此组**只在预分配独占故障盘/有硬上限的专用loop设备**操作,绝不对 `/dev/nvme*` 共享生产设备运行。操作前记录backing file、loop/device-mapper名字、major:minor、mount UUID、上限和唯一FAULT_ID。设备创建/format需要本批单独的资源批准,不是case默认授权。
+This group operates **only on a pre-allocated exclusive fault disk, or on a dedicated loop device with a hard limit**, and it never runs against a shared production device such as `/dev/nvme*`. Before operating, record the backing file, the loop or device-mapper name, major:minor, the mount UUID, the limit, and the unique FAULT_ID. Creating or formatting a device requires a separate resource approval for this batch. It is not a default authorization of the case.
-已分配专用 `HEAL_DM_NAME` / `HEAL_BACKING_DEV` 的device-mapper配方:
+device-mapper recipe for an already allocated dedicated `HEAL_DM_NAME` / `HEAL_BACKING_DEV`:
```bash
sudo dmsetup table "$HEAL_DM_NAME"
sudo dmsetup suspend "$HEAL_DM_NAME"
sudo dmsetup reload "$HEAL_DM_NAME" --table "0 $HEAL_SECTORS delay $HEAL_BACKING_DEV 0 200"
sudo dmsetup resume "$HEAL_DM_NAME"
-# EIO实例用完全返回error的表;只用于专用故障层:
+# The EIO instance uses a table that returns error entirely; use it only for the dedicated fault layer:
sudo dmsetup suspend "$HEAL_DM_NAME"
sudo dmsetup reload "$HEAL_DM_NAME" --table "0 $HEAL_SECTORS error"
sudo dmsetup resume "$HEAL_DM_NAME"
```
-恢复reload预先保存的精确原table,不猜linear偏移;在同服务namespace做定长读写见证(记录耗时/errno),并验证源设备可用。块设备EIO不等同应用层短读,短读实例使用FAULT truncate和真实读错误证据;若需“无内容变化的短读”必须有已分配的faultfs后端,另记能力条件。
+Recovery reloads the exact original table that was saved beforehand, and does not guess a linear offset. In the same service namespace, witness fixed-length reads and writes, recording the elapsed time and errno, and verify that the source device is usable. Block-device EIO is not the same as an application-layer short read. A short-read instance uses FAULT truncate together with real read-error evidence. If a "short read with no content change" is required, an allocated faultfs backend must exist, and that capability condition is recorded separately.
-ENOSPC仅写本例**硬上限卷**中的占位文件,先 `findmnt`/`df -B1` 和 inode计数确认不会耗尽外层XFS;按上限分块填充至返回ENOSPC并记录实际服务写失败。inode实例用创建空文件耗尽该卷的inode预算,不能在共享根创建无限文件。释放只删除已列举的本例占位文件或回收专用卷,不使用宽目录递归删除。
+ENOSPC writes placeholder files only into this case's **hard-limit volume**. First use `findmnt`, `df -B1`, and the inode count to confirm that the outer XFS will not be exhausted. Fill in chunks up to the limit until ENOSPC is returned, and record the actual service write failure. An inode instance exhausts that volume's inode budget by creating empty files. Do not create an unlimited number of files on a shared root. Release deletes only this case's enumerated placeholder files, or reclaims the dedicated volume. Do not use a broad recursive directory deletion.
-桶配额使用冻结rc的 `quota --help`,对本例桶 `quota set`/`quota info`;前后查询证明限额真实生效,结束恢复原配置。禁止把客户端本地限速当服务器quota。
+Bucket quota uses the frozen rc's `quota --help`, and runs `quota set` and `quota info` on this case's bucket. Queries before and after prove that the limit really took effect. At the end, restore the original configuration. Do not treat client-local rate limiting as a server quota.
```bash
"$RC_BIN" --json --no-color quota info "$RC_ALIAS/$B"
"$RC_BIN" --json --no-color quota set "$RC_ALIAS/$B" 64M
-# 只在原配置没有quota且本例结束时清除;否则恢复原限额:
+# Clear only when the original configuration had no quota and this case is ending; otherwise restore the original limit:
"$RC_BIN" --json --no-color quota clear "$RC_ALIAS/$B"
```
-LOCK_WINDOW:慢IO层作用于一个实际持有对象写锁的请求,另发同key heal/PUT。用日志/原生runtime fence证明锁获取、超时、调用方退出、在途提交退出顺序;没有可观察锁窗口则黑盒实例为NOT_OBSERVED。客户端断开可由有界 `timeout` 结束**本次请求进程**,不杀服务;之后继续采集服务端在途状态。精确所有权证明依赖NATIVE,不能从一次超时推导锁已释放。
+LOCK_WINDOW: the slow-IO layer acts on one request that actually holds the object write lock, and a same-key heal or PUT is issued separately. Use logs, or a native runtime fence, to prove the order of lock acquisition, timeout, caller exit, and in-flight commit exit. If there is no observable lock window, the black-box instance is NOT_OBSERVED. A client disconnect may end **this request process** with a bounded `timeout`, without killing the service. Afterward, continue collecting the server-side in-flight state. Exact ownership proof depends on NATIVE. Do not infer from a single timeout that the lock has been released.
-### 强制停机
+### Forced stop
-只对H6已批准的精确Azure资源ID执行,先核验该VM独占、没有其他服务、磁盘持久化策略及带外恢复;记录当前boot_id和平台状态:
+Execute only against the exact Azure resource ID already approved for H6. First verify that the VM is exclusive, that no other service is present, the disk persistence policy, and out-of-band recovery. Record the current boot_id and the platform status:
```bash
az vm get-instance-view --ids "$APPROVED_EXCLUSIVE_VM_ID"
@@ -508,18 +508,18 @@ az monitor activity-log list --resource-id "$APPROVED_EXCLUSIVE_VM_ID" \
--start-time "$RUN_STARTED_AT" --end-time "$RUN_FINISHED_AT"
```
-这是真实VM强制停止,不证明物理控制器掉电。物理主机power-cycle必须使用该独占机器已批准的带外控制台,记录BMC事件和电源周期;不能给未知机型编造通用命令。无授权不执行,也不把SIGKILL替代成通过。
+This is a real forced stop of the VM. It does not prove that a physical controller lost power. A physical-host power cycle must use the out-of-band console already approved for that exclusive machine, and it must record the BMC events and the power cycle. Do not invent a generic command for an unknown machine model. Without authorization, do not execute it, and do not substitute SIGKILL for a pass.
-## 6. 并发、Pool、ILM、复制及原生证据
+## 6. Concurrency, Pool, ILM, replication, and native evidence
-### 有界负载与观测
+### Bounded load and observation
-`trace.json` 是按序的S3操作数组,例如:
+`trace.json` is an ordered array of S3 operations, for example:
```json
[
@@ -532,27 +532,27 @@ az monitor activity-log list --resource-id "$APPROVED_EXCLUSIVE_VM_ID" \
hw writer run --trace trace.json --rounds 100 --workers 2 --interval 0.1 --duration 300
```
-同worker串行;不同worker用不同key可明确先后,测试真正同key竞争时人工按ACK/不确定结果建立期望。`${round}`、`${worker}` 只做JSON字符串替换,不eval。上传内容由run/worker/round/step确定,大小/摘要写入ACK。每次ACK立即独占写文件;意外错误停止后续入场并保留在途结果。不因服务返回错误而无限重试。
+The same worker is serial. Different workers that use different keys can make the order explicit. When testing a real same-key race, a person builds the expectation from the ACKs and from uncertain results. `${round}` and `${worker}` are only JSON string substitution, not eval. Upload content is determined by run, worker, round, and step, and the size and digest are written into the ACK. Each ACK is immediately written to its own exclusive file. An unexpected error stops further admission and retains the in-flight results. Do not retry without bound because the service returned an error.
-长稳分支增加已批准 `--duration` 至86400/259200、足够rounds及限速,并核对实际elapsed,提前耗尽rounds/字节预算不算跑满窗口;每批最多1,000,000请求/16worker/64GiB写入预算,可按预注册多个分段运行并保留连续时序。不可通过事后拼接零散短跑冒充连续24h。
+A long-stability branch raises the approved `--duration` to 86400 or 259200, with enough rounds and rate limiting, and it checks the actual elapsed time. Exhausting the rounds or the byte budget early does not count as having filled the window. Each batch has a write budget of at most 1,000,000 requests, 16 workers, and 64GiB. It may run as several pre-registered segments, and a continuous timeline must be retained. Scattered short runs must not be spliced together afterward and presented as a continuous 24h.
-Admin队列压力使用同样的限量并发方式调用hp(例如shell后台最多4个固定任务),先冻结请求集合,独立label保存每个返回;不能把S3负载当Admin admission覆盖。
+Admin queue pressure calls hp with the same bounded concurrency, for example at most four fixed tasks in the shell background. First freeze the request set, and save each return under an independent label. An S3 load must not be treated as coverage of Admin admission.
```bash
-# 在目标节点按固定频率采样,时间窗口与本例一致;不要采集全环境凭据。
+# Sample on the target node at a fixed rate, with a time window that matches this case; do not collect credentials from the full environment.
sudo journalctl -u "$HEAL_UNIT" --since "$RUN_STARTED_AT" --until "$RUN_FINISHED_AT" --no-pager
sudo systemctl show "$HEAL_UNIT" -p MemoryCurrent -p TasksCurrent -p CPUUsageNSec
pidstat -dru -p "$HEAL_PID" 1 60
iostat -dx 1 60
```
-同时记录FD数量、目标IO、队列/持久记录及临时空间;工具未安装先预检阻断所需性能证据,不跑完才省略。性能ABBA每组四轮、至少三组:warmup、并发、内容、缓存/初始队列、负载窗口和阈值固定。分别统计GET/PUT P95/P99、错误率(所有失败/超时都在总分母)、修复吞吐、CPU/IO。没有预注册门限只能出测量报告,不能记性能通过。
+Also record the FD count, the target IO, the queue and persisted records, and temporary space. If a tool is not installed, preflight first blocks the performance evidence that is required. Do not finish the run and only then omit that evidence. Performance ABBA uses four rounds per group and at least three groups. Warmup, concurrency, content, the cache and the initial queue, the load window, and the thresholds are fixed. Separately compute GET and PUT P95 and P99, the error rate (every failure and timeout is in the total denominator), repair throughput, and CPU and IO. Without a pre-registered threshold, only a measurement report can be produced. A performance pass must not be recorded.
-### Pool交互
+### Pool interaction
-使用**本例H5服务**的rc alias,已有13项Pool回归结果仅为前置控制,不直接调用其会reset旧fleet的部署脚本。扩容/移除拓扑复用已审阅的endpoint分组方案,但根目录替换为本例heal命名空间。所有poolId从实际status和format映射获得。
+Use the rc alias of **this case's H5 service**. The existing 13 Pool regression results are only a precondition control. Do not directly invoke the deploy script that would reset the old fleet. Scale-out and removal topologies reuse the reviewed endpoint-grouping plan, but the root directory is replaced by this case's heal namespace. Every poolId is taken from the actual status and from the format mapping.
```bash
"$RC_BIN" --json --no-color admin rebalance start "$RC_ALIAS"
@@ -564,14 +564,14 @@ iostat -dx 1 60
"$RC_BIN" --json --no-color admin decommission clear "$RC_ALIAS" "$POOL_ID" --by-id
```
-不是把以上所有操作依次跑完:按H01–H06操作卡在相应窗口执行。每次保存rc stdout/stderr/exitCode、相关Admin状态及前后manifest。退役写入限制沿用既定产品契约;仅survivor验证时源必须真实停机且配置移除。#2422/#2423的临界点是否实际触发单独登记。
+Do not run every operation above through to completion in sequence. Execute them in the corresponding window according to the H01–H06 procedure cards. Each time, save the rc stdout, stderr, and exitCode, the related Admin status, and the manifests before and after. Decommission write restrictions follow the established product contract. When only the survivor is verified, the source must actually be stopped and its configuration removed. Whether the critical points of #2422 and #2423 actually fire is registered separately.
-### ILM、远端tier、复制
+### ILM, remote tier, and replication
-使用本仓库已实现lane作为**控制实验**核对候选功能,并在本例独立服务重新建立专用桶/远端租户。配置JSON只含本例资源,API操作如下:
+Use the lane already implemented in this repository as a **control experiment** to check the candidate's functionality, and re-establish a dedicated bucket and remote tenant on this case's independent service. The configuration JSON contains only this case's resources. The API operations follow:
```bash
hp ilm-put s3 --operation put_bucket_lifecycle_configuration --params-file lifecycle.json
@@ -581,11 +581,11 @@ hp replication-put s3 --operation put_bucket_replication --params-file replicati
hp replication-get s3 --operation get_bucket_replication --params-file bucket.json
```
-`lifecycle.json` 使用已验证候选的真实期限;规则ID含run,Filter仅本例prefix,分别配置Expiration/NoncurrentVersionExpiration/Transition。不要用未合并的分钟级测试hook冒充release。远端tier的初始化依照当前rc `ilm tier add --help` 和已有 `scripts/validation_ilm/ilm_behavior.py` 的实际调用,填写本例endpoint/bucket/region与受控凭据;原始凭据不归档。
+`lifecycle.json` uses the real durations of the candidate that has been verified. The rule ID contains the run, Filter is only this case's prefix, and Expiration, NoncurrentVersionExpiration, and Transition are configured respectively. Do not use an unmerged minute-level test hook and present it as a release. Remote-tier initialization follows the actual invocation of the current rc `ilm tier add --help` and of the existing `scripts/validation_ilm/ilm_behavior.py`, filling in this case's endpoint, bucket, and region and the controlled credentials. Raw credentials are not archived.
-复制配置含本例role/destination ARN、Filter和DeleteMarkerReplication;先证明对端版本映射/删除语义。resync使用本批rc `replicate resync --help` 所支持的实际命令,保存启动ID和两端状态。观察真实重叠窗口;没有tier/第二端,相关分支BLOCKED而非删除该项。结束通过API删除本例lifecycle/replication配置,远端本例数据按版本清单清理,不改共享服务规则。
+The replication configuration contains this case's role and destination ARN, Filter, and DeleteMarkerReplication. First prove the peer's version mapping and deletion semantics. resync uses the actual command supported by this batch's rc `replicate resync --help`, and saves the start ID and the status at both ends. Observe the real overlap window. Without a tier or a second endpoint, the related branch is BLOCKED rather than that item being deleted. At the end, delete this case's lifecycle and replication configuration through the API, and clean this case's remote data according to the version inventory. Do not change the rules of a shared service.
-已核对参考rc的resync操作(仍须新批help核验):
+Resync operations of the reference rc that have been checked (a new batch must still verify them against help):
```bash
"$RC_BIN" --json --no-color replicate resync start "$RC_ALIAS/$B" \
@@ -595,69 +595,69 @@ hp replication-get s3 --operation get_bucket_replication --params-file bucket.js
-### 精确屏障与原生回归补证
+### Exact barriers and supplemental native-regression evidence
-F9类窗口不能靠普通release黑盒保证命中。使用同源码SHA的**独立native回归二进制**,绝不换掉被测optimized release服务。先列出真实测试并固定完整测试名、crate、源码位置和预期屏障事件:
+An F9-class window cannot rely on an ordinary release black box to guarantee a hit. Use an **independent native regression binary** built from the same source SHA, and never replace the optimized release service under test. First list the real tests and freeze the complete test name, the crate, the source location, and the expected barrier events:
```bash
cargo test -p rustfs-heal -- --list
cargo test -p rustfs-ecstore -- --list
cargo test -p e2e_test -- --list
-# NATIVE_TEST必须逐字来自上面列表,日志必须显示恰好1项且屏障命中:
+# NATIVE_TEST must come verbatim from the list above, and the log must show exactly one item and a barrier hit:
cargo test -p "$NATIVE_CRATE" "$NATIVE_TEST" -- --exact --nocapture
```
-已有replacement回归函数包含 `test_cluster_root_heal_recovers_remote_shards_after_background_target_restart`、`..._crash`、EC8+4和multi-set版本,完整限定名从该commit --list取得。它们默认创建本地E2E临时目录,**不是远端真实独立盘部署器**;不通过环境变量把这些会清理临时目录的测试指到现有fleet。原生锁/代次/checkpoint测试按所选操作卡在同commit源码中找到精确回归和屏障标记后运行,不存在对应屏障的分支要保留“需源侧增加确定性测试”的能力缺口,不能给计划生成器喂手写JSON声称measured。
+Existing replacement regression functions include `test_cluster_root_heal_recovers_remote_shards_after_background_target_restart`, `..._crash`, and the EC8+4 and multi-set versions. The fully qualified names are taken from that commit's --list. By default they create local E2E temporary directories, and they are **not a remote deployer of real independent disks**. Do not point these tests, which clean temporary directories, at the existing fleet through environment variables. Native lock, generation, and checkpoint tests are run only after the exact regression and its barrier marker have been found, in the source of the same commit, according to the selected procedure card. A branch with no corresponding barrier retains the capability gap "deterministic tests need to be added on the source side". Do not feed handwritten JSON to the plan generator and claim that the result was measured.
-归档 candidate SHA、native测试源码SHA、完整命令、测试数量、exitCode、运行时间及屏障事件。native与fleet分别计分;缺失精确native证据不影响已经完成的黑盒分支真实性,但不能宣布该case所有分支齐全。
+Archive the candidate SHA, the native-test source SHA, the complete command, the test count, the exitCode, the runtime, and the barrier events. Native and fleet are scored separately. Missing exact native evidence does not affect the truthfulness of black-box branches that have already been completed, but it cannot be announced that every branch of that case is complete.
-## 7. 恢复、报告和安全交还
+## 7. Recovery, reporting, and safe handover
-先保存出错时的状态、日志和PHYSICAL,停止本例负载/任务/进程;再按fault receipt撤销规则/只读层/替换根。对已失效lease允许受限恢复,不允许继续注入。单文件恢复:
+First save the state, the logs, and PHYSICAL from the time of the error, and stop this case's load, tasks, and processes. Then, according to the fault receipt, revoke the rules, the read-only layer, and the replacement root. A restricted recovery is allowed for a lease that is already invalid. Continued injection is not allowed. Single-file recovery:
```bash
hf recover --fault-id C02-data-middle --service-quiescent
```
-只恢复同run的备份,原件摘要不符则拒绝;目标如果已经是原始正确字节不重写;如果变成既不是原始也不是注入后字节(可能有新ACK或合法metadata重写),拒绝盲目覆盖,交人工保留现场。恢复工具输出 `physicalRepairPassAllowed=false`;任何 `restoredFromBackup=true` 都不能作为产品补片证据。
+Restore only a backup from the same run. Refuse the restore if the original's digest does not match. If the target is already the original correct bytes, do not rewrite it. If it has become bytes that are neither the original nor the post-injection bytes (there may be a new ACK, or a legal metadata rewrite), refuse a blind overwrite and leave the scene for a person to retain. The recovery tool outputs `physicalRepairPassAllowed=false`. Any `restoredFromBackup=true` cannot be used as evidence that the product repaired or rebuilt the shard.
For an externally cancelled Functional Chain, `validation_cleanup_run_id` accepts only a completed run on the same workflow branch. F07 recovery supports the bounded H2/DV stale-generation boundary after four missing-part injections and before Admin admission: all 18 earlier branches must have complete cleanup receipts, and all 282 fixture requests must have matching successful SDK acknowledgements. Any additional action, unknown write, lost root or physical drift fails closed. Recovery stops the owned services, restores only the four original part backups, and verifies the complete original business files and disk formats. It then retains every data root and private state directory by the existing non-overwriting, same-filesystem archive protocol, with a durable inventory bound to each node's original business-scene digest. Partial restoration and partial archive moves are retryable. `ISOLATED` releases the active namespace while preserving backups; it never rewrites the original cancelled/RUNNING result, restores old system records, or claims canonical PASS.
-强制故障设置第二条恢复通道/超时撤销;本地 finally 不是网络或进程被杀时可靠的唯一恢复办法。退出前枚举本例网络规则、bind层、dm表、测试unit、临时IAM和未完成MPU,逐条撤销并复核。保留失败数据/backup,retention未到期则保留专用资源;记录剩余路径/到期安排。未恢复为 DIRTY/QUARANTINED,不继续下一用例。
+A forced fault sets up a second recovery channel and a timeout revocation. A local finally is not a reliable sole recovery method when the network or the process is killed. Before exit, enumerate this case's network rules, bind layers, dm tables, test units, temporary IAM, and incomplete MPUs, and revoke and recheck each one. Retain the failed data and the backup. If retention has not expired, retain the dedicated resources. Record the remaining paths and the expiry schedule. Anything not yet recovered is DIRTY or QUARANTINED, and the next case does not proceed.
-每个**实际参数实例**都填写人工结论和证据索引,不只填一个基础case总绿:
+Every **actual parameter instance** is filled in with a manual conclusion and an evidence index, not only one overall green result for the base case:
```text
-preflight/ 基础设施/包/rc/harness/lease/拓扑/预注册参数
-baseline/ S3版本清单、物理基线、ACK、桶配置
-fault/ 精确plan、intent、原件摘要、故障有效性见证
-trigger/ 原始Admin/GET/后台来源及活动窗口
-control/ token/epoch/coverage/counter/负向错误/游标原始响应
-physical/ after/restart分片、metadata映射、独立重建对照
-s3/ 所有应存/应删版本与ACK解析、完整内容和配置
-recovery/ 故障解除、停止服务、retained路径与安全交还
-logs/ 全部相关节点的有界日志/资源/平台或native事件
+preflight/ infrastructure/package/rc/harness/lease/topology/pre-registered parameters
+baseline/ S3 version inventory, physical baseline, ACKs, bucket configuration
+fault/ exact plan, intent, original digest, fault-effectiveness witness
+trigger/ raw Admin/GET/background source and the active window
+control/ token/epoch/coverage/counter/negative errors/raw cursor responses
+physical/ after/restart shards, metadata mapping, independent rebuild control
+s3/ every version to retain or delete, ACK resolution, complete content and configuration
+recovery/ fault removal, service stop, retained paths, and safe handover
+logs/ bounded logs/resources/platform or native events from every related node
```
-人工结论为 MANUAL_PASS/FAIL/INFRA_ERROR/BLOCKED/NOT_OBSERVED,列评审人、每条断言及引用。`evidence.json`包含runId、instanceId、九类evidence的数组,每个引用为相对于该实例目录的`path`和`sha256`。全部分支完成才能评审基础case;缺资源/窗口未命中/证据不全都不是PASS。
+The manual conclusion is MANUAL_PASS, FAIL, INFRA_ERROR, BLOCKED, or NOT_OBSERVED, and it lists the reviewer, each assertion, and its citation. `evidence.json` contains runId, instanceId, and an array for each of the nine evidence classes. Each citation is a `path` relative to that instance's directory and a `sha256`. The base case can be reviewed only after every branch is complete. Missing resources, a window that was not hit, and incomplete evidence are not a PASS.
```bash
python scripts/validation_heal/matrix.py audit --plan reports/validation-heal-84-plan.json \
--evidence-dir reports/heal-84 --out reports/heal-84-evidence-audit.json
```
-audit检查完整性/路径/摘要,输出`EVIDENCE_COMPLETE_REVIEW_REQUIRED`仍不是产品通过报告。原始失败批次永久保留,修复后新批次另记。本文提供全84项的可操作方案,但实际机器、专用依赖和native屏障的可用性必须单独核验,不能把“方案就绪”改写成“环境就绪”或“已实测”。
+audit checks completeness, paths, and digests. An output of `EVIDENCE_COMPLETE_REVIEW_REQUIRED` is still not a product pass report. The original failed batch is retained permanently, and a new batch after the fix is recorded separately. This document provides an operable plan for all 84 items, but the availability of the actual machines, the dedicated dependencies, and the native barriers must be verified separately. "Plan ready" must not be rewritten as "environment ready" or "already measured live".
-自动正向夹具固定使用独立分片承诺配置;`integrity-fleet-attestation` 记录本例 endpoint/节点/候选摘要,启动前逐节点 reserve 校验候选,启动后实际进程环境读回两项开关。此配置只作用于 run-owned 服务。默认 legacy 配置的失败批次独立保留;保护模式不代表旧版本混部或默认环境能力。D07 的条件请求以服务器 HEAD 的 LastModified 和基线 ETag 冻结预期,不改主机时间;304/412/416 不能包含成功 payload 或未知传输结果。
+Automatic positive fixtures are fixed to the independent shard-commitment configuration. `integrity-fleet-attestation` records this case's endpoint, node, and candidate digests, reserve-checks the candidate on each node before startup, and reads the two switches back from the actual process environment after startup. This configuration applies only to run-owned services. Failed batches of the default legacy configuration are retained separately. Protected mode does not represent mixed deployment with an old version, or default-environment capability. D07's conditional requests freeze the expectation from the server HEAD LastModified and the baseline ETag, and they do not change the host clock. 304, 412, and 416 must not contain a successful payload or an unknown transfer result.
-J06 的自动入口为 `heal.performance-abba`。执行器在部署前冻结三组 ABBA、每阶段 30 秒、2 并发、4 请求/秒、GET/PUT 各半与完整阈值;每阶段重启同一服务并按固定次数预热。每个 B 阶段的真实部分修复窗口必须包围 GET 和 PUT,否则整例不能 PASS。A 阶段持续检查完整 idle 与物理不变。所有排定请求(包括失败、未知和错过 admission 的请求)均计入分母;GET/PUT 独立报告 P95/P99,三组配对差异、实际修复分片吞吐与服务 CPU/IO/RSS/FD 分别留证。阈值保存在 `scripts/validation_heal/performance.py` 的 `PROTOCOL`,修改阈值必须产生新的 harness commit 和新验证记录;此入口不缩短 J05 的 24h/72h 要求。
+J06's automatic entry point is `heal.performance-abba`. Before deployment, the executor freezes three ABBA groups, 30 seconds per phase, a concurrency of 2, 4 requests per second, a half-and-half split of GET and PUT, and the complete thresholds. Each phase restarts the same service and warms it up a fixed number of times. The real partial-repair window of each phase B must enclose both GET and PUT; otherwise the whole case cannot PASS. Phase A continuously checks that idle is complete and that the physical state is unchanged. Every scheduled request, including failures, unknowns, and requests that missed admission, is counted in the denominator. GET and PUT report P95 and P99 independently. The paired differences of the three groups, the actual shard-repair throughput, and the service CPU, IO, RSS, and FD are each retained as evidence. Thresholds are kept in `PROTOCOL` of `scripts/validation_heal/performance.py`. Changing a threshold must produce a new harness commit and a new verification record. This entry point does not shorten J05's 24h/72h requirement.
-A02 的自动入口为 `heal.admin-scope-isolation`。H2 的专用 `scope-isolation` runtime 使用当前四节点/十六盘,按盘序交织节点并设置 `RUSTFS_ERASURE_SET_DRIVE_COUNT=8`,要求真实两个 set 各跨全部四个节点。普通执行器仍只接受原单 set。两个桶的完整 ACK/实际物理 set 清单冻结后,root/bucket/prefix/object/pool-set 各自新部署、注入选中与控制故障。完整物理和 streaming 证明必须先于 GET 和控制备份恢复。H1 保留原四盘单 set,明确记录不存在未选 set 的适用边界;不能拿同盘目录冒充第二个 set。未限制选择器的原始 receipt 在该固定候选中允许 null,单独保存实际物理映射,显式 pool/set 则必须严格匹配原始字段。
+A02's automatic entry point is `heal.admin-scope-isolation`. H2's dedicated `scope-isolation` runtime uses the current four nodes and sixteen disks, interleaves nodes in disk order, and sets `RUSTFS_ERASURE_SET_DRIVE_COUNT=8`, requiring two real sets that each span all four nodes. The ordinary executor still accepts only the original single set. After the complete ACKs and the actual physical set inventories of the two buckets are frozen, root, bucket, prefix, object, and pool-set are each deployed anew, and the selected fault and the control fault are injected. Complete physical and streaming proof must precede GET and recovery of the control backup. H1 retains the original four-disk single set, and it explicitly records the applicability boundary that no unselected set exists. Directories on the same disk must not be presented as a second set. On this fixed candidate, the raw receipt of an unrestricted selector is allowed to be null, and the actual physical mapping is saved separately. An explicit pool or set must match the original fields exactly.
### I04 durable responsibility during queue pressure
diff --git a/docs/validation-migration-implementation-review.md b/docs/validation-migration-implementation-review.md
index 5c62603..e62e14c 100644
--- a/docs/validation-migration-implementation-review.md
+++ b/docs/validation-migration-implementation-review.md
@@ -1,61 +1,61 @@
-# 用例迁移实现审查
+# Case-migration implementation review
-审查日期:2026-10-28。冻结源 catalog 提交为
-`524eb5b32c3236497fdd8a9271df0494fa3d9974`,SHA-256 为
-`463fdd02dee43e42fef4f3aacf7506709ce87af535e1a53d91c521bc2e361cad`。
-本次交付范围是 auto-testing 中的测试流程、原断言和证据采集。产品执行结果独立保留。
+Review date: 2026-10-28. The frozen source catalog commit is
+`524eb5b32c3236497fdd8a9271df0494fa3d9974`, and its SHA-256 is
+`463fdd02dee43e42fef4f3aacf7506709ce87af535e1a53d91c521bc2e361cad`.
+This delivery covers the test flow, the original assertions, and evidence collection in auto-testing. Product execution results are retained separately.
-## 实现和执行结果
+## Implementation and execution results
-| 测试实现状态 | canonical 用例数 | 含义 |
+| Test implementation status | Canonical cases | Meaning |
| --- | ---: | --- |
-| 已实现 | 145 | 各源分支已有执行路径、原断言及证据采集;其中 115 项已有完整 live 审计,30 项运行失败或尚未完整验收。 |
-| 环境条件限制,默认关闭 | 13 | 操作流程、能力检查、原断言和未满足原因已保留;执行需要明确的资源及经过审核的资源适配器。 |
-| 实现部分完成 | 0 | 已补齐 F07、I05 的测试接线和源流程总控。 |
-| 合计 | 158 | 保留全部源 canonical ID。 |
+| Implemented | 145 | Every source branch already has an execution path, the original assertions, and evidence collection. 115 of them have a complete live audit. 30 failed at runtime or have not yet been fully accepted. |
+| Limited by environment conditions, disabled by default | 13 | The operation flow, capability checks, original assertions, and unmet reasons are retained. Execution needs an explicit resource and a reviewed resource adapter. |
+| Implementation partially complete | 0 | Test wiring and source-flow control for F07 and I05 have been filled in. |
+| Total | 158 | Every source canonical ID is retained. |
-独立 live 状态仍为 115 项 accepted、22 项 retained-failure、4 项 pending、
-4 项 blocked、13 项 disabled。共 43 项尚无完整 live 验收,其中 13 项按约定默认关闭。
-这些结果跨多个不可变候选和历史批次,不构成某个候选通过全部 158 项的产品认证。
-22 项失败需要逐例归因,不能统一宣称为产品缺陷,也不能直接据此认定测试没有迁移。
+The independent live status is still 115 accepted, 22 retained-failure, 4 pending,
+4 blocked, and 13 disabled. 43 items do not yet have a complete live acceptance, of which 13 are disabled by default as agreed.
+These results span multiple immutable candidates and historical batches. They are not a product certification that one candidate passed all 158 items.
+The 22 failures need per-case attribution. They cannot be announced together as product defects, and they cannot by themselves be used to conclude that the tests were not migrated.
-`validation-migration-review.json` 列出全部 158 个 ID、实现状态、代码/测试引用、
-源分支、缺口及独立 live 状态。`reviewed_files` 固定引用文件摘要,变化后须重新审查。
-文件存在和本地测试通过不代替源语义审查或 live 审计。本文件不覆盖原始结果。
+`validation-migration-review.json` lists all 158 IDs, the implementation status, code/test references,
+source branches, gaps, and the independent live status. `reviewed_files` pins file digests. After a change, the review must be repeated.
+File existence and a passing local test do not replace source-semantic review or a live audit. This file does not overwrite the original results.
```bash
python3 -B scripts/validation_migration_review.py --require-complete
```
-此门禁仅因测试实现缺口失败,产品 FAIL 不影响实现计数。13 项环境流程另行输出所需
-适配器清单。产品门禁 `validation_report.py` 仍要求源断言、清理、来源和完整 live PASS。
+This gate fails only because of a test-implementation gap. A product FAIL does not affect the implementation count. The 13 environment flows separately report the adapter manifest they require.
+The product gate `validation_report.py` still requires the source assertions, cleanup, provenance, and a complete live PASS.
-## F07:持久化临界点
+## F07: persistence boundary
-冻结源 `docs/heal-cases.md#F07` / `scenarios/heal-recipes.json` 要求精确本 run 的
-checkpoint/MRF 故障、持久化责任保留、损坏记录保留,以及独立 native fsync/commit
-窗口补证。入口 `persistence.py` 组合以下已存在的能力:
+The frozen source `docs/heal-cases.md#F07` / `scenarios/heal-recipes.json` requires an exact checkpoint/MRF fault for this run,
+retention of persisted responsibility, retention of the corrupt record, and supplementary evidence for the independent native fsync/commit
+window. The entry point `persistence.py` combines the following existing capabilities:
-| 能力 | 证明范围 |
+| Capability | What it proves |
| --- | --- |
-| `persistence.py` / `persistence_mrf.py` | 五种 live 字节损坏、写失败及重扫分支,保留原责任、失败和恢复证据。 |
-| manager checkpoint-page-order / checkpoint-stale-manager | 正常发布和 reload 次序;旧 manager 不能覆盖新 cursor。正常 reload 不冒称提交中断。 |
-| manager mrf-migration-boundaries | 实际 AfterPayload、BeforeManifest、AfterManifest、AfterReadback 四个中断窗口及保留/重试断言。 |
-| manager root-uncertain-publication | 实际终态写入后注入故障,保留 pending owner,重启隔离旧任务。 |
-| `persistence_native.py` | 六个 ECStore CAS/MRF 目录 fsync 故障回归,覆盖更新、首次创建、删除和恢复锚点。 |
+| `persistence.py` / `persistence_mrf.py` | Five live byte-corruption, write-failure, and rescan branches, retaining the original responsibility, failure, and recovery evidence. |
+| manager checkpoint-page-order / checkpoint-stale-manager | Normal publication and reload order. An old manager cannot overwrite a new cursor. A normal reload is not presented as a commit interruption. |
+| manager mrf-migration-boundaries | The four real interruption windows AfterPayload, BeforeManifest, AfterManifest, and AfterReadback, plus the retention and retry assertions. |
+| manager root-uncertain-publication | A fault injected after the real terminal write, retaining the pending owner and isolating the old task across restart. |
+| `persistence_native.py` | Six ECStore CAS/MRF directory fsync-fault regressions, covering update, first creation, deletion, and the recovery anchor. |
-checkpoint 通过 `EcstoreDiskAPI::compare_and_update_file` 落盘。manager 的发布次序、
-中断/重启责任与 ECStore CAS 落盘失败分别补证;没有为每个 fsync 发明额外测试要求。
-审查源码为允许的不可变候选 `072b57b37559fbe626aeda8abf61617fdcbdd926` 和
-`1aa9be87d18cf941e5020f19e8622c26ba4b10ac`;完整目录 fsync 组合只允许后者。
+Checkpoints are persisted through `EcstoreDiskAPI::compare_and_update_file`. The manager's publication order,
+interruption/restart responsibility, and ECStore CAS persistence failures are evidenced separately. No extra test requirement was invented for every fsync.
+The reviewed source is the allowed immutable candidates `072b57b37559fbe626aeda8abf61617fdcbdd926` and
+`1aa9be87d18cf941e5020f19e8622c26ba4b10ac`. The complete directory-fsync combination is allowed only on the latter.
-`collect_native_persistence` 在执行前核对 native manifest 与 fleet 候选,执行六项回归
-后独立验证并再次核对候选;`native_branch_result` 再核对 manager/native 同候选。
-成功和失败分别写入 `independent-native-persistence.json` 与
-`independent-native-persistence-failure.json`,原始 native 日志保留在 `native-persistence/`。
-缺少 artifact 时只允许原诊断子集,不能宣称完整 native coverage。
+`collect_native_persistence` checks the native manifest against the fleet candidate before execution, runs the six regressions,
+verifies them independently, and checks the candidate again. `native_branch_result` then checks that the manager and native builds are the same candidate.
+Success and failure are written to `independent-native-persistence.json` and
+`independent-native-persistence-failure.json` respectively. The raw native logs are retained under `native-persistence/`.
+When an artifact is missing, only the original diagnostic subset is allowed. Complete native coverage must not be claimed.
-使用当前 auto-testing 环境时,由现有 workflow 传入以下目录:
+When the current auto-testing environment is used, the existing workflow passes in the following directories:
```text
VALIDATION_HEAL_MANAGER_DIRECTORY/
@@ -68,32 +68,32 @@ VALIDATION_HEAL_NATIVE_DIRECTORY/
native-evidence/
```
-伴随 workflow 在精确目录 fsync 候选上下载已有 native artifact,并将目录传给入口。
-同一候选的原生结果仍是独立 prerequisite;不代表 H1/H2 fleet 命中了内部窗口,不声称
-物理断电或每个 file fsync hook 已执行。F07 的完整独立 live 审计待完成,canonical 门禁保留。
+The companion workflow downloads the existing native artifact on the exact directory-fsync candidate and passes the directory to the entry point.
+A native result for the same candidate is still an independent prerequisite. It does not mean the H1/H2 fleet hit the internal window, and it does not claim
+that a physical power loss or every file fsync hook was executed. F07's complete independent live audit is still pending, and the canonical gate remains.
-## I05:源流程总控和代次证明
+## I05: source-flow control and generation proof
-`lock_source.py` 复用 `lock_timeout_runner.py`、`lock_timeout.py`、`lock_successor.py`
-及 `lock_observer.py`。默认 `source-plan` 顺序执行 timeout/H1、timeout/H2、
-queued-successor/H1、queued-successor/H2,每次使用独立 run ID、bucket 和证据目录。
-任何分支、证据链或清理失败立即停止后续部署,保留原请求、ACK、日志及停机现场。
+`lock_source.py` reuses `lock_timeout_runner.py`, `lock_timeout.py`, `lock_successor.py`,
+and `lock_observer.py`. The default `source-plan` runs timeout/H1, timeout/H2,
+queued-successor/H1, and queued-successor/H2 in order, each with its own run ID, bucket, and evidence directory.
+Any branch, evidence-chain, or cleanup failure stops later deployment immediately and retains the original request, ACK, logs, and the stopped scene.
-部署前只读验证同一成功 hosted artifact 中的三个精确源码 native 子集:
+Before deployment, read-only validation checks three exact-source native subsets in the same successful hosted artifact:
-| 子集 | 证明范围 |
+| Subset | What it proves |
| --- | --- |
-| I05 owner/unlink/stale | 历史 read-repair owner、取消 physical unlink owner 和原有 stale final rename 回归。旧 stale 测试只检查 ETag。 |
-| G02 explicit-version append | 真实缺片重建在 final rename 屏障暂停;新版本 ACK 和物理 tail drain 后释放;检查类型化结果、全部存活 body、版本、dataDir 和逐盘 shard。 |
-| G03 explicit-version delete | historical/current 删除与最终提交竞态,保留上述完整内容和物理代次断言。 |
+| I05 owner/unlink/stale | The historical read-repair owner, the cancelled physical-unlink owner, and the existing stale final-rename regression. The old stale test checks only the ETag. |
+| G02 explicit-version append | Real missing-shard reconstruction pauses at the final-rename fence. It is released after the new-version ACK and the physical tail drain. The typed result, every surviving body, the version, dataDir, and the per-disk shard are checked. |
+| G03 explicit-version delete | The race between historical/current delete and the final commit, retaining the same complete content and physical-generation assertions. |
-G02/G03 的较强证明来自已有原生测试,不给旧 ETag 测试补写能力。两组覆盖 4/2 与
-16/4,分别保留 6 与 12 条有序屏障里程碑。append 可能安全修复原选定版本并保留新版本,
-不把所有类型化结果统一描述为 stale 拒绝。候选固定为
-`757b751482bb970bc321e92059690a7698554358`;三个 proof 的 package/source tree/
-Cargo.lock/二进制/工作流 attempt/helper/contract 身份必须一致。
+The stronger proof for G02/G03 comes from existing native tests. Capability was not added to the old ETag test. The two groups cover 4/2 and
+16/4, retaining 6 and 12 ordered fence milestones respectively. Append may safely repair the originally selected version and keep the new version.
+Not every typed result is described uniformly as a stale rejection. The candidate is pinned to
+`757b751482bb970bc321e92059690a7698554358`. The package, source tree,
+Cargo.lock, binary, workflow attempt, helper, and contract identity of the three proofs must match.
-原生前提可单独检查,命令不执行下载的二进制:
+The native prerequisite can be checked on its own. The command does not execute the downloaded binary:
```bash
python3 -B scripts/validation_heal/lock_source.py \
@@ -101,23 +101,23 @@ python3 -B scripts/validation_heal/lock_source.py \
--candidate candidate.json --testing-commit "$AUTO_TESTING_SHA"
```
-fleet 控制器保留 DV 历史版本和 DL 单次并发请求/ACK ledger,继续核对所有 peer 的
-完整对象/版本、停机物理分片、锁事件和有界 FD 收尾。DL 是工作负载语义,不是新存储格式。
-新证据链关联请求写出、超时/排队、TCP 断开、原 guard drop、successor 获取和观察结束;
-对原始 bytes 逐一固定 SHA-256,核对请求/lock owner/binding 和同一协调者时钟顺序。
-不会比较不同主机 monotonic 时钟,也不把本地 guard drop 当成分布式 owner 释放。
+The fleet controller retains the DV historical versions and the DL single-concurrency request/ACK ledger, and continues to check every peer's
+complete object/version, the stopped physical shards, lock events, and bounded FD cleanup. DL is a workload semantic, not a new storage format.
+The new evidence chain associates request write, timeout/queue, TCP disconnect, the original guard drop, successor acquisition, and the end of observation.
+It pins a SHA-256 of each original byte sequence and checks the request, lock owner, binding, and order on the same coordinator clock.
+Monotonic clocks on different hosts are not compared, and a local guard drop is not treated as release of the distributed owner.
-`source-plan-summary.json` 记录四个源分支的 fleet/native 证明范围;
-`sourceProceduresComplete` 只有 H1/H2 四个实例完整且清理通过才为 true。单独选择 timeout
-或 queued-successor 仍是诊断子集。全部控制器完成也不自动增加 canonical PASS;完整
-source-plan 的独立 live 审计仍待执行。
+`source-plan-summary.json` records the fleet/native proof scope of the four source branches.
+`sourceProceduresComplete` is true only when all four H1/H2 instances are complete and cleanup passed. Selecting only timeout
+or queued-successor is still a diagnostic subset. Completing every controller does not automatically add a canonical PASS. Independent live audit of the complete
+source-plan is still pending.
-## 失败归因和后续验收
+## Failure attribution and later acceptance
-C11 的旧 metadata、checkpoint、MRF replay 和 native incarnation 均有执行路径;
-J02 的混合 outcome、dry-run、deferred 和 unknown-negative 均有控制器。其历史产品
-结果和证据限制保留,尤其不能修改 J02 的 Failed/Unknown oracle 来获取全绿。
+C11's old metadata, checkpoint, MRF replay, and native incarnation each have an execution path.
+J02's mixed outcome, dry-run, deferred, and unknown-negative paths each have a controller. Their historical product
+results and evidence limits are retained. In particular, J02's Failed/Unknown oracle must not be changed in order to get an all-green result.
-G01/G02 的失败 PUT 不重试,未知 mutation 不算成功。B03/B04 仍缺精确 mount/切换
-适配器,在部署前阻断。今后按产品缺陷、环境前提、证据缺失和测试实现问题逐项归因;
-有明确测试实现问题才重开实现缺口。实现交付完成不代替候选包的后续回归与产品验收。
+A failed PUT in G01/G02 is not retried. An unknown mutation does not count as success. B03/B04 still lack an exact mount/switch
+adapter and are blocked before deployment. From here on, attribute each item as a product defect, an environment prerequisite, missing evidence, or a test-implementation problem.
+Reopen an implementation gap only when there is a definite test-implementation problem. Completing the implementation delivery does not replace later regression of the candidate package or product acceptance.
diff --git a/rustfs_heal_test.md b/rustfs_heal_test.md
index 4e489ab..3ac64ce 100644
--- a/rustfs_heal_test.md
+++ b/rustfs_heal_test.md
@@ -2,7 +2,7 @@
Node-outage heal test driven by
[`rustfs_heal_test.sh`](rustfs_heal_test.sh), based on the
-Obsidian note "RustFS Heal 测试步骤". Uses the same 3-node test environment as
+Obsidian note "RustFS Heal test procedure". Uses the same 3-node test environment as
the pool expansion test (`vm000 vm001 vm002`).
All status checks talk to the RustFS admin API directly (SigV4-signed,
diff --git a/scripts/validation_heal/environment_gate.py b/scripts/validation_heal/environment_gate.py
index e80685c..2a3bcc5 100644
--- a/scripts/validation_heal/environment_gate.py
+++ b/scripts/validation_heal/environment_gate.py
@@ -65,8 +65,8 @@ def decision(case_id: str, *, environ: Mapping[str, str] | None = None,
"enabled": False,
"missingCapabilities": list(policy["requiredCapabilities"]),
"reason": (
- f"{policy['reason']} 默认关闭;不会启动真实测试。只有设置 "
- f"{ENABLE_ENV}=1 并提供 {CAPABILITIES_ENV},且清单满足全部能力后才允许进入准备阶段。"
+ f"{policy['reason']} Disabled by default; a real test is not started. Preparation is allowed only after "
+ f"{ENABLE_ENV}=1 is set and {CAPABILITIES_ENV} is provided, and only once the manifest satisfies every capability."
),
}
missing = sorted(set(policy["requiredCapabilities"]) - requested)
@@ -77,8 +77,8 @@ def decision(case_id: str, *, environ: Mapping[str, str] | None = None,
"enabled": True,
"missingCapabilities": missing,
"reason": (
- f"已显式开启 {ENABLE_ENV}=1,但 auto-testing Heal 环境能力清单仍缺少:"
- f"{', '.join(missing)}。原始阻断原因:{policy['reason']}"
+ f"{ENABLE_ENV}=1 is explicitly enabled, but the auto-testing Heal environment capability manifest is still missing: "
+ f"{', '.join(missing)}. Original block reason: {policy['reason']}"
),
}
return {
@@ -87,8 +87,9 @@ def decision(case_id: str, *, environ: Mapping[str, str] | None = None,
"enabled": True,
"missingCapabilities": [],
"reason": (
- "已显式开启且环境能力清单满足;还需 VALIDATION_HEAL_ENVIRONMENT_RUNTIME 指定候选绑定的资源与控制适配器。"
- "缺少实际资源绑定仍为 BLOCKED,能力声明本身不记 PASS。"
+ "Explicitly enabled, and the environment capability manifest is satisfied. "
+ "VALIDATION_HEAL_ENVIRONMENT_RUNTIME must still name the candidate-bound resources and control adapter. "
+ "A missing real resource binding remains BLOCKED. A capability declaration itself is not recorded as PASS."
),
}
@@ -122,7 +123,7 @@ def write_blocked(case_id: str, artifacts: Path, *, candidate: Path | None,
result_decision = decision(case_id, environ=values, capabilities=capabilities)
if capabilities_error:
result_decision.update(state="blocked", reasonCode="ENV_CAPABILITY_MANIFEST_INVALID",
- reason=f"能力清单无效:{capabilities_error};原始阻断原因:{policy['reason']}")
+ reason=f"Capability manifest is invalid: {capabilities_error}; original block reason: {policy['reason']}")
if override:
result_decision.update(state="blocked", reasonCode=override[0], reason=override[1])
diff --git a/scripts/validation_heal/execution_policy.py b/scripts/validation_heal/execution_policy.py
index f007b36..3adbfcb 100644
--- a/scripts/validation_heal/execution_policy.py
+++ b/scripts/validation_heal/execution_policy.py
@@ -58,67 +58,67 @@ def _native(code: str, reason_code: str, reason: str, capabilities: tuple[str, .
ENVIRONMENT_GATED_CASES = {
"heal.mixed-version-upgrade-rollback": _environment(
"F08", "ENV_IMMUTABLE_RELEASE_PAIR",
- "需要两个已固定且相互兼容的 RustFS release、混合版本 peer 状态和经过审核的回滚兼容矩阵;当前 auto-testing Heal 节点没有这组不可变双版本资源。",
+ "Requires two pinned, mutually compatible RustFS releases, mixed-version peer state, and a reviewed rollback-compatibility matrix; the current auto-testing Heal node does not have this immutable dual-version resource set.",
("immutable_release_pair", "mixed_version_peer_status", "rolling_upgrade_rollback", "rollback_compatibility_matrix"),
),
"heal.dedicated-host-forced-restart": _environment(
"F09", "ENV_DEDICATED_H6_HOST",
- "需要 H6 专用主机、平台强制停启权限、带外恢复和 boot_id/平台事件证据;共享 Heal 节点不能证明真实电源循环。",
+ "Requires a dedicated H6 host, platform force stop/start permission, out-of-band recovery, and boot_id/platform-event evidence; a shared Heal node cannot prove a real power cycle.",
("dedicated_h6_host", "platform_power_cycle", "out_of_band_recovery", "boot_id_evidence"),
),
"heal.race-ilm-expiry-and-transition": _environment(
"G07", "ENV_DEDICATED_ILM_TIER_WINDOW",
- "需要隔离 ILM/tier 租户、真实 Expire/Transition/Restore 窗口和远端版本映射;共享 ILM 环境不能安全制造交叠窗口。",
+ "Requires an isolated ILM/tier tenant, a real Expire/Transition/Restore window, and remote version mapping; a shared ILM environment cannot safely create an overlap window.",
("dedicated_ilm_tier", "real_expire_transition_restore_window", "remote_version_mapping", "isolated_rule_cleanup"),
),
"heal.race-replication-resync": _environment(
"G08", "ENV_ISOLATED_REPLICATION_PAIR",
- "需要隔离的复制两端、resync 控制、跨端版本映射和 Heal/resync 实际交叠窗口;当前固定 fleet 没有独占复制对。",
+ "Requires isolated replication endpoints, resync control, cross-endpoint version mapping, and an actual Heal/resync overlap window; the current fixed fleet has no exclusive replication pair.",
("isolated_replication_pair", "resync_control", "cross_endpoint_version_mapping", "replication_overlap_window"),
),
"heal.with-expansion-rebalance": _environment(
"H01", "ENV_H5_EXPANSION_REBALANCE",
- "需要 H5 pool 扩容、rebalance 和 Heal 同时活动的完整拓扑,并逐对象保存 pool/set 迁移证据;当前 auto-testing 未绑定扩容前后布局与专用操作窗口。",
+ "Requires a complete topology in which H5 pool expansion, rebalance, and Heal are active at the same time, with pool/set migration evidence saved per object; current auto-testing has not bound the layout before and after expansion or a dedicated operation window.",
("h5_pool_expansion", "rebalance_task", "per_object_pool_set_mapping", "retired_scanner_evidence"),
),
"heal.with-rebalance-stop-resume": _environment(
"H02", "ENV_H5_REBALANCE_LIFECYCLE",
- "需要可观测的 H5 rebalance、停止/恢复/服务重启控制及独立 task 身份;当前 auto-testing 未绑定这两类任务的独立身份和操作窗口。",
+ "Requires observable H5 rebalance, stop/resume/service-restart control, and independent task identities; current auto-testing has not bound the independent identities and operation windows of these two task classes.",
("h5_rebalance", "rebalance_stop_resume", "independent_task_identity", "service_restart_control"),
),
"heal.with-decommission": _environment(
"H03", "ENV_H5_DECOMMISSION",
- "需要 H5 pool decommission、survivor 目标、写入限制和移动 ACK/物理权威证据;当前 auto-testing 未绑定本例退役前后拓扑和恢复方案。",
+ "Requires H5 pool decommission, a survivor target, write restrictions, and move-ACK and physical-authority evidence; current auto-testing has not bound this case's topology before and after decommission, or its recovery plan.",
("h5_decommission", "survivor_pool", "decommission_write_guard", "movement_ack_physical_proof"),
),
"heal.with-decommission-cancel-clear": _environment(
"H04", "ENV_H5_DECOMMISSION_CANCEL",
- "需要退役取消/清理/新代次恢复以及 #2423 所需容量锁重试窗口;普通取消结果不能替代这个临界点。",
+ "Requires decommission cancellation, cleanup, and new-generation recovery, plus the capacity-lock retry window required by #2423; an ordinary cancellation result cannot take the place of this boundary.",
("h5_decommission", "cancel_clear_resume", "capacity_lock_retry_window", "generation_bound_ack_ledger"),
),
"heal.after-retired-pool-removal": _environment(
"H05", "ENV_H5_RETIRED_POOL_REMOVAL",
- "需要完整迁移后移除 retired pool、源保持离线和 survivor-only 访问;当前 auto-testing 未提供本例源持续离线和 survivor-only 的绑定证据。",
+ "Requires the retired pool to be removed after a complete migration, the source to stay offline, and survivor-only access; current auto-testing has not provided bound evidence for this case that the source stays offline and that access is survivor-only.",
("h5_retired_pool_removal", "source_offline_proof", "survivor_authority", "retired_pool_write_guard"),
),
"heal.ec84-multi-set-restart": _environment(
"H07", "ENV_EC84_MULTI_SET_TOPOLOGY",
- "需要 H3 的 12 个独立盘和 H4 的至少 24 个独立 endpoint,并按 set 分别记录重启/崩溃窗口;当前 H4 资源不可用。",
+ "Requires 12 independent disks for H3 and at least 24 independent endpoints for H4, with the restart and crash windows recorded per set; H4 resources are currently unavailable.",
("h3_ec84_12_independent_disks", "h4_ec84_24_independent_endpoints", "per_set_identity", "per_set_restart_crash_evidence"),
),
"heal.slow-io-and-eio": _environment(
"I01", "ENV_DEDICATED_FAULT_IO",
- "需要专用 faultfs 或故障设备、可恢复的 slow/short-read/EIO 注入和 namespace 隔离;共享盘模拟会污染其他运行。",
+ "Requires a dedicated faultfs or fault device, recoverable slow/short-read/EIO injection, and namespace isolation; simulating on a shared disk would contaminate other runs.",
("dedicated_faultfs_or_device", "slow_io_short_read_eio", "namespace_isolation", "fault_layer_restore"),
),
"heal.kms-and-tier-outage": _environment(
"I06", "ENV_DEDICATED_KMS_TIER_TENANT",
- "需要专用 KMS/tier 租户、依赖网络隔离和安全的 key/tier 版本证据;不能在共享依赖上全局断网。",
+ "Requires a dedicated KMS/tier tenant, network isolation of the dependency, and safe key/tier version evidence; do not cut the network globally on a shared dependency.",
("dedicated_kms_tier_tenant", "dependency_network_isolation", "key_redaction", "tier_version_mapping"),
),
"heal.long-soak": _environment(
"J05", "ENV_DEDICATED_LONG_SOAK_LANE",
- "必须在独占 H2/H5 lane 真实运行 24 小时和 72 小时,包含限速、监控、到期回收和完整资源预算;短跑不能替代 soak 证据。",
+ "Must actually run for 24 hours and for 72 hours on an exclusive H2/H5 lane, including rate limiting, monitoring, expiry reclamation, and a complete resource budget. A short run cannot substitute for soak evidence.",
("dedicated_soak_lane", "wall_clock_24h_72h", "resource_budget_monitoring", "scheduled_fault_recovery"),
),
}
@@ -129,22 +129,22 @@ def _native(code: str, reason_code: str, reason: str, capabilities: tuple[str, .
EVIDENCE_BLOCKED_CASES = {
"heal.bucket-incarnation-recreation": _native(
"C11", "EVIDENCE_INCARNATION_LIVE_ORCHESTRATION",
- "已实现 H1/H2 旧元数据重返、真实 checkpoint/resume 回放、ACK 绑定的 committed MRF 回放和精确 native 证据门禁;尚缺全部现场分支的独立审计,诊断执行保持 FAIL/BLOCKED,原生前置通过不能替代 canonical 验收。",
+ "H1/H2 old-metadata rejoin, real checkpoint/resume replay, ACK-bound committed MRF replay, and the precise native evidence gate are implemented; an independent audit of every live branch is still missing, diagnostic execution stays FAIL/BLOCKED, and a passing native prerequisite cannot replace canonical acceptance.",
("bucket_incarnation_isolation", "checkpoint_mrf_replay_guard", "incarnation_native_evidence_binding", "complete_live_orchestrator"),
),
"heal.checkpoint-and-mrf-corruption": _native(
"F07", "EVIDENCE_DURABLE_JOURNAL_ORCHESTRATION",
- "已实现 H1/H2 checkpoint 截断、未来 schema、真实较早代次回放、committed MRF CRC 故障、私有限额卷的真实 journal ENOSPC 和独立安全重扫;10 项原生回归仅为前置证据。仍缺全部所需 fsync/commit 临界点审核和完整现场分支的独立 live 审计,诊断执行不能提升 canonical 验收。",
+ "Implemented H1/H2 checkpoint truncation, future schema, replay of a real earlier generation, a committed MRF CRC fault, real journal ENOSPC on a private quota volume, and an independent safe rescan; the 10 native regressions are only prerequisite evidence. Still missing are review of every required fsync/commit boundary and an independent live audit of the complete live branches. Diagnostic execution cannot raise canonical acceptance.",
("durable_checkpoint_mrf_fault", "write_failure_retention", "publication_boundary_evidence_binding", "complete_live_orchestrator"),
),
"heal.lock-timeout-and-caller-abort": _native(
"I05", "NATIVE_LOCK_OWNER_ABORT_BARRIER",
- "已有同候选源码的 lock owner、取消物理 unlink 与 stale read-repair 原生子集;H1/H2 timeout 与 queued-successor 诊断均已通过独立 live 审计(39767336741/1、39817606025/1),包括原始 ACK、历史代次、真实锁事件、各 peer inventory、停机后新分片完整性和有界 FD 收尾。仍缺实际断连边界的精确所有权、完整 native timeout/stale 屏障、stale-generation controller,以及 H1/H2 DL/DV 完整对象/ACK/物理/锁/FD 闭环。精确源码 cfg(test) 原生证据独立保留;诊断子集和原生子集不能充作完整 canonical PASS。",
+ "A native subset from the same candidate source already exists for lock owner, cancelled physical unlink, and stale read-repair. The H1/H2 timeout and queued-successor diagnostics have both passed an independent live audit (39767336741/1, 39817606025/1), including the original ACKs, historical generations, real lock events, each peer inventory, new-shard integrity after shutdown, and bounded FD cleanup. Still missing are precise ownership at the actual disconnect boundary, a complete native timeout/stale barrier, the stale-generation controller, and the complete H1/H2 DL/DV object/ACK/physical/lock/FD closed loop. Precise source cfg(test) native evidence is retained on its own; the diagnostic subset and the native subset cannot stand in for a complete canonical PASS.",
("server_lock_owner", "lock_timeout_fence", "caller_abort_sequence", "stale_read_repair_fence", "exact_source_native_evidence", "complete_live_orchestrator", "final_object_lock_fd_closure"),
),
"heal.outcome-counters-and-error-truth": _native(
"J02", "EVIDENCE_MIXED_OUTCOME_ORCHESTRATION",
- "已有 mixed healthy/repaired/failed、独立 dry-run、依赖 deferral、真实 SDK legacy Unknown、原始全副本损坏元数据 Unknown、只读永久失败/新 token 补偿及版本级 grace Deferred/新 token 清理执行流程,均按逐对象 manifest/attemptFailures 对账;仍需完整串行 live 证据及独立验收,故保持禁止默认 dispatch 和 canonical admission;本地检查不能作为完整 PASS。",
+ "Execution flows already exist for mixed healthy/repaired/failed, an independent dry-run, dependency deferral, a real SDK legacy Unknown, an original all-replica corrupt-metadata Unknown, read-only permanent failure with new-token compensation, and version-level grace Deferred with new-token cleanup, all reconciled per object against the manifest and attemptFailures. Complete serial live evidence and independent acceptance are still required, so default dispatch and canonical admission stay prohibited; a local check cannot serve as a complete PASS.",
("mixed_outcome_fixture", "per_object_canonical_receipts", "complete_live_orchestrator", "attempt_failure_visibility"),
),
}
diff --git a/scripts/validation_heal/recipes.json b/scripts/validation_heal/recipes.json
index a693522..f66c061 100644
--- a/scripts/validation_heal/recipes.json
+++ b/scripts/validation_heal/recipes.json
@@ -11,9 +11,9 @@
"deep"
],
"steps": [
- "用 DB 夹具生成完整 manifest;先对每个对象保存 S3 基线和四盘分片摘要,关闭 scanner/MRF/auto-heal。",
- "依次执行 START object(scanMode=1、2),WAIT 原 token 直至终态;再 STATUS 同 token 两次。",
- "对每个对象的 outcome 区分 Normal 的 metadata_healthy 与 Deep 的 verified_healthy;两者均要求 healed=0、unchanged>0,Normal 不得冒充 payload 完整性证明。比较目标/控制对象物理摘要和 S3 清单,重复请求不得新建版本。"
+ "Generate a complete manifest with the DB fixture; first, for each object, save the S3 baseline and the four-disk shard digests, and disable scanner/MRF/auto-heal.",
+ "In sequence, run START object (scanMode=1 and 2), WAIT the original token until the terminal state; then STATUS the same token twice.",
+ "For each object's outcome, distinguish Normal's metadata_healthy from Deep's verified_healthy; both require healed=0 and unchanged>0, and Normal must not be passed off as proof of payload integrity. Compare the physical digests of the target and control objects with the S3 inventory, and a repeated request must not create a new version."
],
"commands": [
"SEED_DB",
@@ -35,8 +35,8 @@
"recovery",
"logs"
],
- "recovery": "先保存失败现场及物理/状态证据,停止本例写入与在途任务,再按 RECOVER 撤销本例故障。并发版本已变化时禁止盲目恢复旧 xl.meta;恢复不能充作产品修复。未恢复环境标 DIRTY,保留本例备份和原失败结果。",
- "verdict": "逐项满足设计公共契约、该分支步骤及附加断言后人工评审。缺资源为 BLOCKED,窗口未命中为 NOT_OBSERVED,产品错误为 FAIL,工具/恢复错误为 INFRA_ERROR;未运行不记 PASS。"
+ "recovery": "First save the failure scene and the physical and state evidence, stop this case's writes and in-flight tasks, then undo this case's fault per RECOVER. When a concurrent version has already changed, do not blindly restore the old xl.meta; recovery must not serve as a product fix. Mark an unrestored environment DIRTY, and keep this case's backup and the original failure result.",
+ "verdict": "Manually review after each item satisfies the design's common contract, that branch's steps, and the additional assertions. Missing resources are BLOCKED, a window miss is NOT_OBSERVED, a product error is FAIL, and a tool or recovery error is INFRA_ERROR; a case that did not run is not recorded as PASS."
},
{
"code": "A02",
@@ -49,10 +49,10 @@
"pool-set"
],
"steps": [
- "用 DS 在两个桶的 a/、ab/、b/ 写入不同内容;另在未选 set 放控制对象;逐对象预先记录实际 pool/set。",
- "分别隔离选中对象、相邻前缀及另一桶/未选 set 的一个 part。每个 selector 使用新夹具和故障 ID。",
- "对象请求路径到完整 key;prefix 路径到 a/ 且 recursive=true;bucket 路径只含桶;root 为 /;pool/set 写入 body。START 后 WAIT。",
- "先 PHYSICAL:选择范围内全部重建、范围外控制故障仍缺失;然后 GET 验证,避免验证请求提前修好控制故障。root 必须包含整个专用服务清单。"
+ "Use DS to write different content at a/, ab/, and b/ in two buckets; also place a control object in an unselected set; record the actual pool/set of each object in advance.",
+ "Separately isolate one part of the selected object, of an adjacent prefix, and of another bucket or an unselected set. Each selector uses a new fixture and a new fault ID.",
+ "The object request path is the full key; the prefix path is a/ with recursive=true; the bucket path contains only the bucket; root is /; pool/set is written in the body. WAIT after START.",
+ "First PHYSICAL: everything inside the selected range is rebuilt, and control faults outside the range are still missing; then verify with GET, so the verification request does not repair the control fault early. root must include the entire dedicated-service manifest."
],
"commands": [
"SEED_DS",
@@ -74,8 +74,8 @@
"recovery",
"logs"
],
- "recovery": "先保存失败现场及物理/状态证据,停止本例写入与在途任务,再按 RECOVER 撤销本例故障。并发版本已变化时禁止盲目恢复旧 xl.meta;恢复不能充作产品修复。未恢复环境标 DIRTY,保留本例备份和原失败结果。",
- "verdict": "逐项满足设计公共契约、该分支步骤及附加断言后人工评审。缺资源为 BLOCKED,窗口未命中为 NOT_OBSERVED,产品错误为 FAIL,工具/恢复错误为 INFRA_ERROR;未运行不记 PASS。"
+ "recovery": "First save the failure scene and the physical and state evidence, stop this case's writes and in-flight tasks, then undo this case's fault per RECOVER. When a concurrent version has already changed, do not blindly restore the old xl.meta; recovery must not serve as a product fix. Mark an unrestored environment DIRTY, and keep this case's backup and the original failure result.",
+ "verdict": "Manually review after each item satisfies the design's common contract, that branch's steps, and the additional assertions. Missing resources are BLOCKED, a window miss is NOT_OBSERVED, a product error is FAIL, and a tool or recovery error is INFRA_ERROR; a case that did not run is not recorded as PASS."
},
{
"code": "A03",
@@ -85,10 +85,10 @@
"coordinator-to-stable-endpoint"
],
"steps": [
- "在 H2 同一跨节点 set 写入 DB;保存协调者、目标、稳定入口分别对应的端点,不把同一节点的两个别名当跨节点。",
- "隔离目标节点 part;从 n0 START,记录原始 clientToken;通过 n1 和 stable 对同一路径、同 token 执行 STATUS。",
- "按同一任务身份对齐三端状态,最多等候冻结的状态收敛预算;终态及核心计数必须一致,404/unknown 不能视为结束。",
- "WAIT 后先物理证明目标重建,再从所有 S3 入口 VERIFY;保存每个请求实际连接的节点日志。"
+ "Write DB on the same cross-node set in H2; save the endpoints for the coordinator, the target, and the stable entry respectively, and do not treat two aliases of the same node as cross-node.",
+ "Isolate the target node's part; START from n0 and record the original clientToken; run STATUS through n1 and stable on the same path with the same token.",
+ "Align the three endpoints' state by the same task identity, waiting at most the frozen state-convergence budget; the terminal state and the core counts must match, and 404/unknown cannot be treated as finished.",
+ "After WAIT, first physically prove that the target was rebuilt, then VERIFY from every S3 entry; save the log of the node each request actually connected to."
],
"commands": [
"SEED_DB",
@@ -110,8 +110,8 @@
"recovery",
"logs"
],
- "recovery": "先保存失败现场及物理/状态证据,停止本例写入与在途任务,再按 RECOVER 撤销本例故障。并发版本已变化时禁止盲目恢复旧 xl.meta;恢复不能充作产品修复。未恢复环境标 DIRTY,保留本例备份和原失败结果。",
- "verdict": "逐项满足设计公共契约、该分支步骤及附加断言后人工评审。缺资源为 BLOCKED,窗口未命中为 NOT_OBSERVED,产品错误为 FAIL,工具/恢复错误为 INFRA_ERROR;未运行不记 PASS。"
+ "recovery": "First save the failure scene and the physical and state evidence, stop this case's writes and in-flight tasks, then undo this case's fault per RECOVER. When a concurrent version has already changed, do not blindly restore the old xl.meta; recovery must not serve as a product fix. Mark an unrestored environment DIRTY, and keep this case's backup and the original failure result.",
+ "verdict": "Manually review after each item satisfies the design's common contract, that branch's steps, and the additional assertions. Missing resources are BLOCKED, a window miss is NOT_OBSERVED, a product error is FAIL, and a tool or recovery error is INFRA_ERROR; a case that did not run is not recorded as PASS."
},
{
"code": "A04",
@@ -125,10 +125,10 @@
"versioned-bitflip"
],
"steps": [
- "分别用 DB 与 DV 建夹具,冻结全部 versionId/marker;关闭后台来源,不在注入后 GET。",
- "在原始 part 上做 missing/bitflip,每组独立新目录;保存故障后的 PHYSICAL 快照作为 dry-run 前基线。",
- "START body 同时传 dryRun=true、scanMode=2 和该实例的 remove/recreate;WAIT 仅接受 dry_run_observed,不能要求 dry-run 持有 incarnation。",
- "前后所有分片/元数据/marker 字节与版本清单相同;没有 repaired ACK。完成非变更证明后恢复故障,再 VERIFY。"
+ "Build fixtures with DB and DV respectively, and freeze every versionId/marker; disable background sources, and do not GET after injection.",
+ "Apply missing/bitflip on the original part, with an independent new directory for each group; save the post-fault PHYSICAL snapshot as the baseline before the dry-run.",
+ "The START body passes dryRun=true, scanMode=2, and that instance's remove/recreate together; WAIT accepts only dry_run_observed, and must not require the dry-run to hold an incarnation.",
+ "All shard, metadata, and marker bytes and the version manifest are the same before and after; there is no repaired ACK. After the no-change proof is complete, clear the fault, then VERIFY."
],
"commands": [
"SEED_DB",
@@ -152,8 +152,8 @@
"recovery",
"logs"
],
- "recovery": "先保存失败现场及物理/状态证据,停止本例写入与在途任务,再按 RECOVER 撤销本例故障。并发版本已变化时禁止盲目恢复旧 xl.meta;恢复不能充作产品修复。未恢复环境标 DIRTY,保留本例备份和原失败结果。",
- "verdict": "逐项满足设计公共契约、该分支步骤及附加断言后人工评审。缺资源为 BLOCKED,窗口未命中为 NOT_OBSERVED,产品错误为 FAIL,工具/恢复错误为 INFRA_ERROR;未运行不记 PASS。"
+ "recovery": "First save the failure scene and the physical and state evidence, stop this case's writes and in-flight tasks, then undo this case's fault per RECOVER. When a concurrent version has already changed, do not blindly restore the old xl.meta; recovery must not serve as a product fix. Mark an unrestored environment DIRTY, and keep this case's backup and the original failure result.",
+ "verdict": "Manually review after each item satisfies the design's common contract, that branch's steps, and the additional assertions. Missing resources are BLOCKED, a window miss is NOT_OBSERVED, a product error is FAIL, and a tool or recovery error is INFRA_ERROR; a case that did not run is not recorded as PASS."
},
{
"code": "A05",
@@ -165,10 +165,10 @@
"force-start"
],
"steps": [
- "用 DX 超过 128 项的外部分片夹具注入多对象缺片;预先从固定候选记录 overlap policy(复用 token 或冲突错误),禁止接受任意结果。",
- "START 递归范围,WINDOW 证明 0