Skip to content

Commit 9fd3228

Browse files
committed
perf(test): speed up offline command E2E execution
- Run offline validation and dry-run cases in process - Serialize captures and restore process state after each run - Preserve subprocess coverage for live, I/O, preload and proxy contracts - Load only command modules required by topic routes - Isolate the harness npm update lookup while preserving business fetches - Add execution parity and isolation tests - Stabilize preload checks across cache modes - Document offline E2E execution boundaries
1 parent 8d9ed0b commit 9fd3228

29 files changed

Lines changed: 967 additions & 379 deletions

‎docs/agents/cli-e2e-tests.md‎

Lines changed: 13 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -2,14 +2,14 @@
22

33
## 架构分层
44

5-
| 层级 | 路径 | 测什么 |
6-
| --------------- | ----------------------------------------------------- | --------------------------------------------------------------------------------------- |
7-
| **共享基建** | `packages/e2e` | gating、子进程 runner、registry help 捕获、output、globalSetup(`private`,不发布) |
8-
| **命令 E2E** | `packages/commands/tests/e2e` | 进程内 help、子进程缺参/dry-run/live(gated);每用例最小路由 |
9-
| **Journey E2E** | `packages/commands/tests/e2e/knowledge/journeys` | 用户旅程全链路(跨命令回路 + 标记词召回闭环),全部 live gated;见 `journeys/README.md` |
10-
| **bl smoke** | `packages/cli/tests/e2e/registry.smoke.e2e.test.ts` | 产品 map 全部 path/分组的进程内 help;根 help、鉴权域等代表性子进程冒烟 |
11-
| **kscli smoke** | `packages/kscli/tests/e2e/registry.smoke.e2e.test.ts` | map 全部 path/分组的进程内 help;`--version`、`search --help` 等代表性子进程冒烟 |
12-
| **runtime** | `packages/runtime/tests` | `proxy.e2e`、console 跨域 flag 拒绝 |
5+
| 层级 | 路径 | 测什么 |
6+
| --------------- | ----------------------------------------------------- | -------------------------------------------------------------------------------------------- |
7+
| **共享基建** | `packages/e2e` | gating、子进程 / 进程内 runner、registry help 捕获、output、globalSetup(`private`,不发布) |
8+
| **命令 E2E** | `packages/commands/tests/e2e` | 进程内 help / 离线校验 / dry-run、代表性子进程、live(gated);每用例最小路由 |
9+
| **Journey E2E** | `packages/commands/tests/e2e/knowledge/journeys` | 用户旅程全链路(跨命令回路 + 标记词召回闭环),全部 live gated;见 `journeys/README.md` |
10+
| **bl smoke** | `packages/cli/tests/e2e/registry.smoke.e2e.test.ts` | 产品 map 全部 path/分组的进程内 help;根 help、鉴权域等代表性子进程冒烟 |
11+
| **kscli smoke** | `packages/kscli/tests/e2e/registry.smoke.e2e.test.ts` | map 全部 path/分组的进程内 help;`--version`、`search --help` 等代表性子进程冒烟 |
12+
| **runtime** | `packages/runtime/tests` | `proxy.e2e`、console 跨域 flag 拒绝 |
1313

1414
**依赖边界**:`e2e` → `core`;`commands/tests` → `e2e` + `commands/src`;产品 tests → `e2e` + 各自 `src` + `runtime` 公共 API。**禁止**产品 import `commands/tests/**`(子进程 spawn harness 路径除外)。
1515

@@ -30,7 +30,10 @@
3030

3131
- 路径:`packages/commands/tests/e2e/<kebab-topic>.e2e.test.ts`;knowledge 领域集中在 `packages/commands/tests/e2e/knowledge/` 子目录(新增 knowledge 命令测试放这里)
3232
- help:`runCommandHelp(routes, [...path, "--help"])` from `./helpers.ts`(当前 Vitest worker 内用真实 command + `CommandRegistry` 渲染,不启动子进程)
33-
- 子进程:缺参、dry-run、live 使用 `runCommandE2e(routes, args)`(spawn `harness/main.ts`,`routes` 为本 topic 最小 path → export 映射)
33+
- 离线校验 / dry-run:确认用例不依赖进程初始化、真实请求或持久化后,可显式使用 `runCommandInProcess(routes, args, envOverrides)`;走真实 `createCli`、middleware 和 command,捕获 stdout / stderr / 退出码,执行后恢复进程环境。worker 内串行捕获,禁止与其他操作全局状态的并发用例混用
34+
- 子进程:`runCommandE2e(routes, args)` 保留给 live、实际写盘、进程隔离及 shell / stdio / env 契约;每个 topic 保留代表性子进程用例。`NODE_OPTIONS` / HTTP(S) 代理配置存在时,`runCommandInProcess` 自动回退子进程
35+
- 路由加载:harness 按 command 库公开导出名加载 topic 所需模块,不整体求值命令 barrel;新增导出形式需扩展 loader 并保留与公开导出对象一致性的测试
36+
- 更新检查:未发布的 harness 包 `latest` 查询固定返回本地 404,避免重复访问 npm;只拦截这个精确 URL,业务 fetch 和生产 updater 测试仍走原逻辑
3437
- fixtures:`packages/commands/tests/e2e/fixtures/`
3538
- 路由常量:`topic-routes.ts`(按 topic 维护,**非**全量产品 map)
3639

@@ -44,7 +47,7 @@
4447

4548
### 共享
4649

47-
- gating / output / runner:`e2e/gating`、`e2e/output`、`e2e/runner`;runner 使用 `node --import tsx` 执行 TypeScript 入口,不启动 tsx CLI IPC server
50+
- gating / output / runner:`e2e/gating`、`e2e/output`、`e2e/runner`、`e2e/in-process`;runner 使用 `node --import tsx` 执行 TypeScript 入口,不启动 tsx CLI IPC server
4851
- globalSetup:根 `vite.config.ts` → `packages/e2e/src/global-setup.ts`
4952
- 解析 JSON stdout:`parseStdoutJson`;输出目录:`makeE2eOutputDir(e2eLabelFromMetaUrl(import.meta.url))`
5053
- 长任务:`cliTimeoutPrefix()`;视频用例加 `test(..., 3_600_000)` 等显式超时

‎packages/commands/tests/e2e/alert.e2e.test.ts‎

Lines changed: 8 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
import { describe, expect, test } from "vite-plus/test";
2-
import { parseStdoutJson, runCommandE2e } from "./helpers.ts";
2+
import { parseStdoutJson, runCommandE2e, runCommandInProcess } from "./helpers.ts";
33
import { ALERT_ROUTES } from "./topic-routes.ts";
44

55
// 只覆盖 help / 参数校验 / dry-run;真实调用依赖 console 凭证与已开通的 CMS 服务。
@@ -74,7 +74,7 @@ describe("e2e: alert", () => {
7474
});
7575

7676
test("alert create --dry-run 输出完整通知配置", async () => {
77-
const { stdout, stderr, exitCode } = await runCommandE2e(ALERT_ROUTES, [
77+
const { stdout, stderr, exitCode } = await runCommandInProcess(ALERT_ROUTES, [
7878
"alert",
7979
"create",
8080
"--name",
@@ -146,7 +146,7 @@ describe("e2e: alert", () => {
146146
});
147147

148148
test("alert delete --dry-run 支持多 ID", async () => {
149-
const { stdout, stderr, exitCode } = await runCommandE2e(ALERT_ROUTES, [
149+
const { stdout, stderr, exitCode } = await runCommandInProcess(ALERT_ROUTES, [
150150
"alert",
151151
"delete",
152152
"--rule-id",
@@ -187,7 +187,7 @@ describe("e2e: alert", () => {
187187
["enable", "zeldaEasy.bailian-telemetry.alertRule.enableAlertRule"],
188188
["disable", "zeldaEasy.bailian-telemetry.alertRule.disableAlertRule"],
189189
] as const) {
190-
const { stdout, stderr, exitCode } = await runCommandE2e(ALERT_ROUTES, [
190+
const { stdout, stderr, exitCode } = await runCommandInProcess(ALERT_ROUTES, [
191191
"alert",
192192
verb,
193193
"--rule-id",
@@ -246,7 +246,7 @@ describe("e2e: alert", () => {
246246
});
247247

248248
test("alert template create --dry-run 输出条件结构", async () => {
249-
const { stdout, stderr, exitCode } = await runCommandE2e(ALERT_ROUTES, [
249+
const { stdout, stderr, exitCode } = await runCommandInProcess(ALERT_ROUTES, [
250250
"alert",
251251
"template",
252252
"create",
@@ -295,7 +295,7 @@ describe("e2e: alert", () => {
295295
});
296296

297297
test("alert template update --dry-run 输出更新请求", async () => {
298-
const { stdout, stderr, exitCode } = await runCommandE2e(ALERT_ROUTES, [
298+
const { stdout, stderr, exitCode } = await runCommandInProcess(ALERT_ROUTES, [
299299
"alert",
300300
"template",
301301
"update",
@@ -319,7 +319,7 @@ describe("e2e: alert", () => {
319319
});
320320

321321
test("alert template delete --dry-run 支持多 ID", async () => {
322-
const { stdout, stderr, exitCode } = await runCommandE2e(ALERT_ROUTES, [
322+
const { stdout, stderr, exitCode } = await runCommandInProcess(ALERT_ROUTES, [
323323
"alert",
324324
"template",
325325
"delete",
@@ -362,7 +362,7 @@ describe("e2e: alert", () => {
362362
});
363363

364364
test("alert history --dry-run 输出时间范围映射", async () => {
365-
const { stdout, stderr, exitCode } = await runCommandE2e(ALERT_ROUTES, [
365+
const { stdout, stderr, exitCode } = await runCommandInProcess(ALERT_ROUTES, [
366366
"alert",
367367
"history",
368368
"--rule-id",
Lines changed: 125 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,125 @@
1+
import { mkdtempSync, rmSync, writeFileSync } from "node:fs";
2+
import { tmpdir } from "node:os";
3+
import { join } from "node:path";
4+
import { expect, test } from "vite-plus/test";
5+
import { runCommandE2e, runCommandInProcess } from "./helpers.ts";
6+
import {
7+
CONFIG_ROUTES,
8+
KNOWLEDGE_CHUNK_CATEGORY_FILE_ROUTES,
9+
QUOTA_ROUTES,
10+
} from "./topic-routes.ts";
11+
12+
const captureEnvironment = {
13+
NODE_OPTIONS: undefined,
14+
HTTP_PROXY: undefined,
15+
HTTPS_PROXY: undefined,
16+
http_proxy: undefined,
17+
https_proxy: undefined,
18+
};
19+
20+
const cases = [
21+
{ name: "group help", routes: QUOTA_ROUTES, args: ["quota", "--help"] },
22+
{
23+
name: "supplied flag usage error",
24+
routes: QUOTA_ROUTES,
25+
args: ["quota", "list", "--unsupported", "--output", "json"],
26+
},
27+
{
28+
name: "required flag usage error",
29+
routes: CONFIG_ROUTES,
30+
args: ["config", "set", "--quiet", "--output", "json"],
31+
},
32+
{
33+
name: "config dry-run",
34+
routes: CONFIG_ROUTES,
35+
args: [
36+
"config",
37+
"set",
38+
"--key",
39+
"watermark",
40+
"--value",
41+
"false",
42+
"--dry-run",
43+
"--output",
44+
"json",
45+
],
46+
},
47+
{
48+
name: "command validation",
49+
routes: KNOWLEDGE_CHUNK_CATEGORY_FILE_ROUTES,
50+
args: [
51+
"knowledge",
52+
"chunk",
53+
"add",
54+
"--index-id",
55+
"fixture",
56+
"--content",
57+
"hello",
58+
"--field",
59+
"column=1",
60+
"--workspace-id",
61+
"fixture",
62+
"--dry-run",
63+
"--output",
64+
"json",
65+
],
66+
},
67+
];
68+
69+
test.each(cases)(
70+
"in-process $name matches the real CLI subprocess exactly",
71+
async ({ routes, args }) => {
72+
const directory = mkdtempSync(join(tmpdir(), "bl-command-parity-"));
73+
try {
74+
const environment = {
75+
...captureEnvironment,
76+
BAILIAN_CONFIG_DIR: directory,
77+
DASHSCOPE_API_KEY: "",
78+
BAILIAN_E2E: "0",
79+
};
80+
const subprocess = await runCommandE2e(routes, args, environment);
81+
const inProcess = await runCommandInProcess(routes, args, environment);
82+
expect(inProcess).toEqual(subprocess);
83+
} finally {
84+
rmSync(directory, { recursive: true, force: true });
85+
}
86+
},
87+
);
88+
89+
test("profile-language errors match in-process and subprocess output", async () => {
90+
const directory = mkdtempSync(join(tmpdir(), "bl-command-locale-parity-"));
91+
try {
92+
writeFileSync(
93+
join(directory, "config.json"),
94+
JSON.stringify({ language: "en-US", work: { language: "zh-CN" } }),
95+
);
96+
const args = ["--config", "work", "quota", "list", "--unsupported", "--output", "json"];
97+
const environment = { ...captureEnvironment, BAILIAN_CONFIG_DIR: directory };
98+
const result = await runCommandInProcess(QUOTA_ROUTES, args, environment);
99+
expect(result).toEqual(await runCommandE2e(QUOTA_ROUTES, args, environment));
100+
expect(result.stderr).toContain("未知参数");
101+
} finally {
102+
rmSync(directory, { recursive: true, force: true });
103+
}
104+
});
105+
106+
test("Node preloads retain the real subprocess contract", async () => {
107+
const directory = mkdtempSync(join(tmpdir(), "bl-command-preload-"));
108+
// esbuild workers also inherit NODE_OPTIONS on cache misses; only the CLI main thread marks startup.
109+
const preload =
110+
'import { isMainThread } from "node:worker_threads"; if (isMainThread) process.stdout.write("preload marker\\n");';
111+
try {
112+
for (const disableCache of [undefined, "1"]) {
113+
const result = await runCommandInProcess(QUOTA_ROUTES, ["quota", "--help"], {
114+
BAILIAN_CONFIG_DIR: directory,
115+
TSX_DISABLE_CACHE: disableCache,
116+
NODE_OPTIONS: `--import=data:text/javascript,${encodeURIComponent(preload)}`,
117+
});
118+
expect(result.exitCode).toBe(0);
119+
expect(result.stdout).toBe("preload marker\n");
120+
expect(result.stderr).toContain("Usage: bl quota");
121+
}
122+
} finally {
123+
rmSync(directory, { recursive: true, force: true });
124+
}
125+
});
Lines changed: 45 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,45 @@
1+
import { expect, test } from "vite-plus/test";
2+
import { loadRouteCommands } from "./harness/routes.ts";
3+
import * as topicRoutes from "./topic-routes.ts";
4+
import * as commandExports from "../../src/index.ts";
5+
6+
const fixtureIndex = new URL("./fixtures/route-loader/index.ts", import.meta.url);
7+
8+
test("loads a declared command without evaluating an unrelated broken export", async () => {
9+
const commands = await loadRouteCommands({ "probe run": "probe" }, fixtureIndex);
10+
expect(commands["probe run"].description).toBe("fixture probe");
11+
expect(await commands["probe run"].run({} as never)).toBe("probe result");
12+
});
13+
14+
test("aliases retain the same command instance", async () => {
15+
const commands = await loadRouteCommands(
16+
{ "probe run": "probe", "probe alias": "probeAlias" },
17+
fixtureIndex,
18+
);
19+
expect(commands["probe run"]).toBe(commands["probe alias"]);
20+
});
21+
22+
test("rejects unknown exports and declared non-command defaults", async () => {
23+
await expect(loadRouteCommands({ invalid: "missing" }, fixtureIndex)).rejects.toThrow(
24+
"Unknown bailian-cli-commands export: missing",
25+
);
26+
await expect(loadRouteCommands({ invalid: "notACommand" }, fixtureIndex)).rejects.toThrow(
27+
"Unknown bailian-cli-commands export: notACommand",
28+
);
29+
});
30+
31+
test("every topic route resolves to the same public command exported by the library", async () => {
32+
const library = commandExports as Record<string, unknown>;
33+
for (const routes of Object.values(topicRoutes)) {
34+
const commands = await loadRouteCommands(routes);
35+
for (const [path, exportName] of Object.entries(routes))
36+
expect(commands[path]).toBe(library[exportName]);
37+
}
38+
});
39+
40+
test("supports named aliases and preserves route declaration order", async () => {
41+
const routes = { "named first": "namedProbe", "default second": "probe" };
42+
const commands = await loadRouteCommands(routes, fixtureIndex);
43+
expect(Object.keys(commands)).toEqual(Object.keys(routes));
44+
expect(await commands["named first"].run({} as never)).toBe("named result");
45+
});

0 commit comments

Comments
 (0)