diff --git a/.agents/docs/2026-08-27-issue516-windows-acp-glob-walk-fix.md b/.agents/docs/2026-08-27-issue516-windows-acp-glob-walk-fix.md new file mode 100644 index 00000000..7070b0a2 --- /dev/null +++ b/.agents/docs/2026-08-27-issue516-windows-acp-glob-walk-fix.md @@ -0,0 +1,482 @@ +# #516 解决方案:glob walk 在 Windows 上撞到 ANSI 代码页拼不出的目录名就崩 + +日期:2026-08-27 · 基线:`e187d3f`(origin/main,2026.8.27.1)· 发布版本:**2026.8.27.2** · 状态:**已决策,已实现** + +> **决策记录(2026-08-27,来自 review)** +> 1. 口径 **B** —— 跳过要报出来,不静默(§2) +> 2. **P0 + P1 同一个 PR** 全部实现,不拆 +> 3. §6.2 的独立 issue **保留**(P2 清单 + 项目根含非 ACP 的静默空构建) +> 4. **不做 httplib 端到端**,单元测试覆盖即可(§5.4 相应作废) +> +> 实现过程中有三处偏离本文初稿,都是测量推翻的,记在 §10。 + +> Issue: mcpp-community/mcpp#516 · 复现:mcpplibs/mcpp-index PR #260 +> 完整取证过程见 openxlings/xlings 仓 `.agents/docs/2026-08-27-mcpp516-windows-acp-path-analysis.md` +> xlings 侧的同类隐患(与本 issue 无关,独立修):openxlings/xlings#571 + +--- + +## 0. 三行结论 + +1. **不是解压问题。** 解压是成功的,文件以正确的 UTF-16 名字落盘; + 抛异常的是 **mcpp 自己**,在 `src/modgraph/scanner.cppm:238` 的 `dir.filename().string()`。 +2. **这是 #230 的同一处漏网。** #231 的 "never-throw on unnarrowable names" 加固了 + `path_matches_glob` / `rewrite_rel_copy` / `local_include_dirs_for` 三处, + **漏掉了同一个 walk 循环里比它们早一行执行的 `is_excluded_walk_dir`**。 + #516 看到的这条整洁报错,正是 #231 顺手加的顶层 catch 在正常工作。 +3. **爆炸半径不是一个包。** 在 mcpplibs/mcpp-index `891b2f7` 上量:130 个 recipe 里 + **103 个**含以 `*` 开头的 glob, + 即"从解压根无界递归遍历"。任何上游 tarball 哪天多一个非 ASCII 文件名,当天 Windows 就红。 + +--- + +## 1. 缺陷 + +### 1.1 抛出点 + +```cpp +// src/modgraph/scanner.cppm:236-239 +bool is_excluded_walk_dir(const std::filesystem::path& dir, + const std::filesystem::path& root) { + auto name = dir.filename().string(); // ← 抛 + if (name == ".mcpp" || name == ".git" || name == "target") return true; +``` + +MSVC STL 的 `path::string()` → `__std_fs_convert_wide_to_narrow(ACP, …)`; +对非 UTF-8 代码页它传 `lpUsedDefaultChar`,一旦用了替换字符就返回 +`__std_win_error::_No_unicode_translation`(Win32 1113), +`_Throw_system_error_from_std_win_error` 抛 `std::system_error`。 +`system_error(ec)` 不带 what_arg 时 `what()` 就是 `ec.message()`,即 + +> `No mapping for the Unicode character exists in the target multi-byte code page.` + +与 issue 里的报错**逐字一致**。 + +该函数是 walk 循环体的**第一行**(`scanner.cppm:388` 和 `:483` 两处调用), +**每个目录条目调用一次** —— 所以它比后面所有已加固的站点都先执行, +加固后面三处对目录名根本无效。 + +### 1.2 怎么走到那棵树的 + +`compat.httplib.lua`(Form B)带 `include_dirs = { "*" }`, +经 `manifest/xpkg.cppm:1242` 落到 `buildConfig.includeDirs`, +由 `scanner.cppm:788` `local_include_dirs_for()` → `expand_dir_glob(verRoot, "*")`: + +``` +"*" 含通配符 → 不走字面量快路径 +glob_literal_prefix("*") → 通配符在第 0 位 → 前缀为空 +start = root → recursive_directory_iterator 遍历整棵 cpp-httplib 源码树 + → test/www/日本語Dir/ 必然被访问 +``` + +`scanner.cppm:796` 自己的注释把 `"*"` 称作 +"the `*` extracted-tarball-root glob convention" —— 这是**约定**,不是 httplib 的怪癖。 + +### 1.3 异常一路无人接手 + +`expand_dir_glob` → `local_include_dirs_for` → scanner → prepare → `cli::run` +全链路无 catch,最终由 `src/main.cpp:36` 的顶层 catch 接住并 exit 70。 + +### 1.4 归属反证(值得记住的一条) + +有人会怀疑是 xlings 解压时按 ANSI 写坏了名字。**恰恰相反**: +`ERROR_NO_UNICODE_TRANSLATION` 的前提是**宽名里存在 ACP 拼不出的字符**。 +若盘上是 mojibake(UTF-8 字节被 CP1252 逐字节打散),那些字符**逐个都在 CP1252 里**, +mcpp 反而不会抛。**mcpp 抛了,正好证明 xlings 写对了。** + +--- + +## 2. 必须先定的口径(A / B) + +收敛成唯一入口这件事,**必然要一次性定下"转不出来时怎么办"**,否则每个调用方还是各答各的。 + +**现状**(#231 留下的):`path_matches_glob` 的 try/catch 返回 `false` +→ 一个 CJK 命名的源文件**既不报错也不编译,它就是不存在**。 + +| | 行为 | 代价 | +|---|---|---| +| **A** | 保持静默丢弃 | 用户加了个 `模块.cppm`,构建"成功"、符号找不到、零提示 | +| **B** | 丢弃并 `ui::warn` 一次,指名 | 要给这条 warning 找一个不刷屏的去重点 | + +**推荐 B。** 理由不是洁癖:这条路径上"没发生"和"成功了"输出完全一样, +而 #230 → #516 之间隔了整整一个大版本才被再次发现,正是因为中间那段时间它只是**静默少编译**。 + +**本方案按 B 写。** 若定 A,§4.2 的 `emit_unnarrowable_warning()` 整块删掉即可,其余不变 —— +**P0 与口径无关**,可以先合。 + +--- + +## 3. P0 — 修掉抛出点 + +### 3.1 补丁 + +`src/modgraph/scanner.cppm:236-245`: + +```cpp +bool is_excluded_walk_dir(const std::filesystem::path& dir, + const std::filesystem::path& root) { + // 按 path 比较,不要窄化。 + // + // `filename().string()` 在 MSVC 上走宽→ANSI 转换,对当前代码页拼不出的名字 + // 抛 std::system_error(#230 是同一个转换,#516 是同一个转换的另一处站点)。 + // 而这个函数是 walk 循环体的第一行 —— 每个目录条目都经过它,所以它比 + // path_matches_glob 的 try/catch 更早执行,加固那里对目录名无效。 + // + // 三个字面量都是 ASCII,转成 native(Windows 上是宽)无损; + // path::operator== 比较的是 native 串,大小写敏感,行为与原来的窄串比较逐字一致。 + // 用静态常量而非每次构造临时 path,保住 #225 的 per-entry 开销预算。 + static const std::filesystem::path kMcpp{".mcpp"}; + static const std::filesystem::path kGit{".git"}; + static const std::filesystem::path kTarget{"target"}; + const auto name = dir.filename(); + if (name == kMcpp || name == kGit || name == kTarget) return true; + + auto const& submodules = submodule_paths(root); + if (submodules.empty()) return false; + std::error_code ec; + auto c = std::filesystem::canonical(dir, ec); + return submodules.contains(ec ? dir : c); +} +``` + +### 3.2 为什么这一处就够(下游已核过) + +修好之后,那条 walk 上**不再有**会碰到坏名字的窄化点: + +| 后续步骤 | 是否窄化 | +|---|---| +| `it.depth()` / `chain.resize` | 否 | +| `fs::canonical(e.path(), eec)` | `error_code` 重载,不抛 | +| `std::find(chain…)` / `std::set` | `path` 比较,原生串 | +| `path_matches_glob` | **已加固**(`glob.cppm:47-57`)→ 返回 `false` | +| `local_include_dirs_for` 收集结果 | 坏目录已被上一行判为不匹配,不在结果里 | + +即:坏名字被**挡在窄化世界之外**,后面拿不到它。 + +### 3.3 修好之后的行为 + +`日本語Dir` **不会**成为 include dir(`path_matches_glob` 判不匹配)。 +这是**可接受的既有语义**:一个 ACP 拼不出的名字,也没法写进 glob 或编译命令行。 +对 cpp-httplib 而言那是 `test/www/` 下的测试数据,不影响头文件消费。 + +--- + +## 4. P1 — 收敛(真正的修复) + +P0 只是第四个点补丁。再打点补丁,下次会有第五个站点。 + +### 4.1 两条规则,不是一条 + +- **规则 1:只与 ASCII 字面量比较的谓词,根本不该窄化** —— 按 `path` 比。 + (`is_excluded_walk_dir` 属于这条,所以 P0 里没用到 `try_narrow`。) +- **规则 2:必须产出窄串的站点,走唯一入口。** + +### 4.2 唯一入口 + +放进 `mcpp.modgraph.glob`(已有的"唯一 glob 匹配器"模块,`glob.cppm` 开头那段 +"两个匹配器'应该'一致是这个代码库反复付账的形状,所以只有一个" 的理由同样适用): + +```cpp +// 走查得到的路径 → 窄串。当前 ACP 拼不出该名字时返回 nullopt,**永不抛**。 +// +// 任何来自 directory_iterator 的 path 都必须经过这里,不得直接 .string()。 +// 非 Windows 上这是一次拷贝,永远有值。 +std::optional try_narrow(const std::filesystem::path& p) { + try { return p.generic_string(); } + catch (const std::exception&) { return std::nullopt; } +} +``` + +口径 B 的 warning(定 A 则删掉此块): + +```cpp +// 每个进程每个"不可拼写的父目录"只报一次,避免一棵树刷屏。 +void emit_unnarrowable_warning(const std::filesystem::path& p); +``` + +warning 文案必须**只用可拼写的部分**定位,否则打印它本身又会抛: + +``` +warning: skipped a path whose name cannot be represented in the active code page (1252) + under: + hint: 这些文件不会参与构建。`chcp` 改的是控制台代码页,不是进程 ACP。 +``` + +> 拼装这条消息时**不要**去窄化那个坏名字。向上找到第一个 `try_narrow` 成功的祖先, +> 报到那一级为止。这条不是洁癖 —— 是本类缺陷最容易复发的地方 +> (诊断代码自己碰同一个转换)。 + +### 4.3 审计范围(逐个追过输入来源) + +初稿我按文件名 grep 出一张表就交了,那不是测量。逐个追输入来源后砍掉两个误报: + +| 站点 | 输入来源 | 结论 | +|---|---|---| +| `modgraph/scanner.cppm:238` | 任意包/项目树的目录名 | **P0,确认触发** | +| `pack/digest.cppm:48` | `pack/prebuilt.cppm:144` 的递归走查 | **真** —— 打包路径不经 glob 过滤 | +| `scaffold/template.cppm:265` | 模板目录名 | **真**,仅当第三方模板目录含非 ACP 名 | +| ~~`build/resources.cppm:66`~~ | `RcTool::path.filename()` | **误报**:那是 `windres`/`rc.exe`/`llvm-rc`,payload 相对定位,全 ASCII | +| ~~`modgraph/p1689.cppm:339`~~ | `source.filename()` | **误报**:CJK 文件名到不了这里(`path_matches_glob` 已先判不匹配) | + +`p1689` 那处值得单说,因为它顺带答了一个会被问到的问题: +**"用户自己的 CJK 源文件今天会怎样?"** —— 不是崩,是**静默不编译**。 +真正能让 p1689 抛的是同一行的 `source.string()`(**整条绝对路径**,第 341 行): +当**项目根目录自身**含非 ACP 字符时(英文版 Windows + 中文项目路径)。 +但那种情况下 `path_matches_glob` 会先把所有源文件判为不匹配, +构建会先以"没有源文件"失败 —— 又一个"没发生和成功了输出一样"。 +**这一条不在本 PR 范围**,建议单独立 issue(见 §6.2)。 + +### 4.4 不变式与 CI 门 + +写进 `.agents/skills/mcpp-contributing/SKILL.md`(**初稿写的是 `AGENTS.md` —— mcpp 没有 +这个文件,那是 xlings 的习惯**;贡献规范在 skill 里): + +> **任何来自 `directory_iterator` 的 `path` 都不得直接 `.string()` / `.generic_string()`。** +> 只与 ASCII 字面量比较时按 `path` 比;必须产出窄串时走 `mcpp::modgraph::try_narrow()`。 + +CI 门(只覆盖真正会走查任意树的目录,并**明说它的边界**): + +```bash +# .github/tools/check_narrow_conversions.sh —— 作用域收窄的理由见脚本头。 +# 它挡的是"新写的代码直接窄化",挡不住"经由别处传进来的 path"—— +# 后者靠 §4.2 的唯一入口约定,不靠这个 grep。 +if grep -rnE '\.(filename|stem|extension)\(\)\.(generic_)?string\(\)' src/modgraph/ \ + | grep -v 'NARROW-OK:'; then + echo "FAIL: 直接窄化走查路径。用 try_narrow(),或加 // NARROW-OK: <理由>"; exit 1 +fi +``` + +> 这个门**不是**判据本身,只是让复发变贵一点。判据是 §5 的 Windows 测试。 +> 别把它写进验收标准里当"已覆盖"用。 + +--- + +## 5. 回归测试(这一节比补丁重要) + +### 5.1 一个必须先说清的前提 + +**在 Linux/macOS 上写的任何测试都无法证伪这个缺陷。** +那两个平台上 `path::string()` 只是拷贝 native 串,不做编码转换,任何文件名都"能表示"。 +所以:**测试必须跑在 Windows CI 上,且 runner 的 ACP ≠ 65001。** + +`ci-windows.yml` 的 `build + test + package (windows x64, self-host)` job 会跑 +`mcpp test`,`tests/unit/test_modgraph.cpp` 在其中 —— 这就是落点。 +`windows-latest` 默认 ACP=1252,满足。 + +**但不能默认它永远满足**:哪天 runner 镜像默认 UTF-8 ACP,这个 case 会**静默变成永远绿的装饰品**。 +所以测试自己必须探测,并在探测不到条件时 `GTEST_SKIP()` 且**说明原因**。 + +### 5.2 名字选择(一个容易踩的坑) + +用 `日本語Dir` 忠实于 issue,但它在 **CP932(日)/CP936(简中)/CP950(繁中)上是可拼写的** —— +在这些机器上测试会 skip,等于本地没有覆盖。 + +**天城文(Devanagari)不在任何 Windows ANSI 代码页里**,因此在**所有**非 UTF-8 ACP 下都不可拼写。 +两个都造:天城文保证覆盖,日文保证与 issue 同形。 + +用 `\uXXXX` 转义而非直接写字面量 —— 不依赖源文件编码设置(MSVC 需要 `/utf-8`,clang 默认 UTF-8; +这个测试不该因为哪天换了编译器就变味)。 + +### 5.3 测试代码 + +追加到 `tests/unit/test_modgraph.cpp`: + +```cpp +namespace { +// 当前 ACP 能否拼出这个宽名?能 → 说明 ACP 是 UTF-8(或该名字恰好在表内), +// 本用例的前提不成立。用 path 自己的转换来问,问的就是被测代码走的那条路。 +bool acp_can_spell(const std::wstring& w) { + try { (void)std::filesystem::path(w).string(); return true; } + catch (const std::exception&) { return false; } +} +} // namespace + +// #516 / #230:走查任意包树时,一个当前 ANSI 代码页拼不出的目录名 +// 曾让 is_excluded_walk_dir 的 .filename().string() 抛 std::system_error, +// 一路逃到 main() 变成 `error: internal: unhandled exception:` + exit 70。 +// +// 断言的是「不抛」且「walk 没被截断」,不是「能匹配到那个名字」—— +// 一个 ACP 拼不出的名字也没法写进 glob 或编译命令,判它不匹配是既有语义。 +TEST(Scanner, GlobWalkSurvivesUnnarrowableNames) { + // 用 \u 转义,不写字面量:这个测试不该因为哪天换了编译器或改了源文件编码 + // 设置(MSVC 要 /utf-8,clang 默认 UTF-8)就悄悄变味。 + const std::wstring devanagari = L"\u0915\u0916\u0917Dir"; // कखगDir：不在任何 ANSI 代码页 + const std::wstring japanese = L"\u65e5\u672c\u8a9eDir"; // 日本語Dir：与 #516 同形 + + if (acp_can_spell(devanagari)) { + GTEST_SKIP() << "active code page can spell any name (UTF-8 ACP or non-Windows); " + "this defect is unreachable here"; + } + + auto dir = make_tempdir("mcpp-scanner-acp"); + std::filesystem::create_directories(dir / std::filesystem::path(devanagari)); + std::filesystem::create_directories(dir / std::filesystem::path(japanese)); + // 好邻居:证明 walk 没有在坏条目处提前结束。 + write(dir / "zzz_ascii" / "x.h", "#pragma once\n"); + + // include-dir 通道(#516 的实际路径:include_dirs = { "*" }) + std::vector dirs; + ASSERT_NO_THROW({ dirs = expand_dir_glob(dir, "*"); }); + EXPECT_NE(std::find(dirs.begin(), dirs.end(), dir / "zzz_ascii"), dirs.end()) + << "walk 在不可拼写的条目处被截断了"; + + // 文件通道(installedLayoutMatchesIndex 走的那条:前缀为空 → 全树遍历) + std::vector files; + ASSERT_NO_THROW({ files = expand_glob(dir, "**/*.h"); }); + EXPECT_NE(std::find(files.begin(), files.end(), dir / "zzz_ascii" / "x.h"), files.end()); + + std::filesystem::remove_all(dir); +} +``` + +### 5.4 端到端那一层 + +单测锁住机制,但锁不住"真实上游 tarball + `include_dirs = {"*"}` 约定"这个组合。 +唯一能覆盖它的是 **mcpp-index 的 windows 矩阵跑 httplib 三个 example**(PR #260 本身)。 +建议:#260 合入后,把 `httplib` / `httplib-tls` / `httplib-zstd` 留在 windows 矩阵里, +**不要**以"windows 上不消费该 feature"为由把它们门控掉 —— 那会把这个覆盖点关掉。 + +--- + +## 6. 不做什么 + +### 6.1 不在 mcpp 侧"宽容处理"文件名 + +不要试图把不可拼写的名字转义/哈希后当成可用路径。它们最终要交给 +编译器驱动、ninja、link 命令行,而那些都是**独立进程、按各自的 ACP 解释字节**。 +在 mcpp 内部造一个"看起来能用"的名字,只会把失败推到更远、更难归因的地方。 + +### 6.2 P2(嵌 `activeCodePage=UTF-8` 清单)不进本 PR + +给 Windows 可执行文件嵌带 `UTF-8` 的清单, +`__std_fs_code_page()` 返回 CP_UTF8,`path::string()` 从此返回 UTF-8 且**永不抛**, +上面所有站点一次性全对。要求 Windows 10 1903+(CI runner 满足)。 + +**它不危险**:今天能跑通的路径全是 ASCII,而 ASCII 在 UTF-8 和 CP1252/936 下**字节完全相同** —— +build.ninja、CDB、命令行一个字节不变。 + +**真正的未知是"够不够"**:mcpp 写出 build.ninja 之后,是**另一个进程** +(xlings store 里的 `ninja.exe`,`config.cppm:192`)按**它自己的** ACP 去读。 +所以嵌清单只解决 mcpp 这一半;非 ASCII 路径能否端到端跑通,取决于 ninja 和编译器驱动那一侧。 + +**建议独立立 issue,验收写成"mcpp 不再抛这个异常",不要写成"支持 CJK 路径"。** +后者要单独测量,不能顺带宣称。 + +同一个 issue 里一并收 §4.3 提到的"项目根目录含非 ACP 字符 → 先以『没有源文件』失败"。 + +--- + +## 7. 落地顺序与验收 + +| 步 | 内容 | 验收 | +|---|---|---| +| 1 | P0 补丁(§3.1) | `ci-windows` 绿 | +| 2 | §5.3 单测 | **打补丁前必须红**,补丁后绿。CI 日志里能看到它**没有** skip | +| 3 | P1 收敛(§4.2/4.3)+ 贡献规范不变式 + CI 门 | 门在故意加一行裸 `.filename().string()` 时会红 | +| 4 | 真实复现验证 | mcpp-index PR #260 的 windows 矩阵三个 httplib 用例转绿 | + +**第 2 步的"打补丁前必须红"不能省。** 一个从来没红过的测试不证明任何事 +—— 先在打补丁前跑一次 `ci-windows`(或 workflow_dispatch),把那条 +`error: internal: unhandled exception: No mapping…` 留在日志里,再合补丁。 + +**第 4 步要注意归因**:mcpp-index 的 CI 会随索引发布漂移。 +验证时**重跑同一个未变更的 run id**,不要用一次新的 push 去比对 —— +否则"绿了"可能是索引变了,不是补丁起作用了。 + +### 不算验收的东西 + +- Linux/macOS 全绿:**与本缺陷无关**,那两个平台上它不可能发生。 +- CI 门通过:它只挡新写的直接窄化,不是判据(§4.4)。 +- 单测 skip 掉也算绿:**必须检查它真的跑了**。 + +--- + +## 8. 风险与回滚 + +- **P0 行为变化**:`path::operator==` 大小写敏感,与原来的 `std::string` 比较**逐字一致**; + 三个字面量全 ASCII,native 转换无损。**没有行为变化**,只是不再抛。 +- **性能**:静态 `path` 常量 + 三次 native 串比较,比原来"每条目一次 `std::string` 构造 + 三次比较" + **更便宜**。#225 的 per-entry 预算不受影响。 +- **回滚**:P0 是单函数改动,回滚即恢复原状(重新可崩)。 + P1 若引入问题,可先只回滚 §4.2 的 warning(口径 B → A),保留唯一入口。 + +--- + +## 9. 多角度拆分、依赖关系与跨仓协作 + +### 9.1 各角度落在哪一处改动上 + +| 角度 | 这次的具体决定 | +|---|---| +| **架构** | `src/modgraph/` 与 `src/manifest/` 是 leaf 层 —— **全仓 31 个模块 import `mcpp.ui`,这两层一个都没有**,`mcpp.diag` 亦然。所以 glob 层**记录**(`note_unnarrowable_path`),CLI 层**排空上报**。不为了少写十行而捅穿这条边界。 | +| **稳定性** | 抛出点消失,而不是被 catch 住。`try_narrow` 本身 `noexcept` 语义(内部 try),调用方拿 `nullopt`。 | +| **优雅简洁** | 不是第四个 try/catch,而是**按用途分三档的一条规则**(§4.1)。`is_excluded_walk_dir` 的正解是**根本不窄化**,比加保护更短也更快。 | +| **用户体验** | 口径 B:跳过按目录报告一次,走 `diag::degraded` 并给出 `impact`。`hint` 里点名 `chcp` 改的是控制台代码页 —— 那是用户第一个会试的东西。 | +| **兼容性** | `path::operator==` 与原来的窄串比较**逐字同解**;`interface_set_digest` 改用 `u8string()` 对纯 ASCII 名字**字节不变**,已发布包摘要不变。 | +| **跨平台** | 非 Windows 上 `try_narrow` 永不失败 → 一条记录都不会产生 → 行为与今天完全一致。`digest` 那处顺带修掉了一个**真实的跨平台不一致**(Linux 打包 / Windows 校验对非 ASCII 名字给出不同摘要)。 | +| **一致性** | 窄化只有一个答案者;报告只有一个 sink(`mcpp.diag`,它本就是全仓唯一的用户可见告警通道);排空只有一个点(`cli::run` 的 scope guard,覆盖全部五条 return)。 | +| **无感升级** | 没有配置项、没有新 flag、没有行为开关。ASCII 项目(几乎全部)的输出一个字节都不变。 | + +### 9.2 任务依赖 + +``` +T1 try_narrow + 记录/排空 API (glob.cppm) + ├─→ T2 P0: is_excluded_walk_dir 停止窄化 ← #516 的实际崩溃点 + ├─→ T3 其余站点归并 (path_matches_glob / scan_file 诊断 / digest / template) + ├─→ T4 单元测试 (依赖 T1 的 API 才能断言"记录到了") + └─→ T5 CI 门 (依赖 T2/T3 才可能绿) +T6 文档与规范 (docs/05 EN+zh 双份 —— check_docs_style 强制标题结构对齐) +T7 版本号 2026.8.27.1 → 2026.8.27.2 + CHANGELOG +T8 xlings pin 2026.8.17.2 → 2026.8.27.4 ← 跨仓,见 9.3;放在最后一个 commit +T2..T8 → T9 CI 全绿 → T10 自我 review → T11 release → T12 生态验证 +``` + +T1 是唯一的串行瓶颈;T2/T3/T4/T6/T7 之后可并行。 + +### 9.3 跨仓协作边界 + +| 仓 | 这次做什么 | 依赖 | +|---|---|---| +| `mcpp-community/mcpp` | 全部代码改动 + 发布 2026.8.27.2 | — | +| `openxlings/xlings` | **不改**。#516 不是 xlings 的缺陷(§1.4)。它自己的同类隐患另立 **openxlings/xlings#571** | 独立 | +| `mcpplibs/mcpp-index` | **不改**。按 review 决定不做 httplib 端到端(§5.4 作废),PR #260 在 mcpp 2026.8.27.2 发布后自然转绿 | 依赖 mcpp 发布 | + +**唯一一条真正的跨仓阻塞**:`kXlingsVersion` → `2026.8.27.4` 要求该版本**已发布**, +否则 CI 会去下载一个不存在的 `xlings-2026.8.27.4-` tarball。 +`.github/tools/check_version_pins.sh` 会强制 `.github/` 下全部 pin 点与常量一致, +所以这是一次**原子的多点改动**,放在最后一个 commit,发布确认之后再推。 + +### 9.4 "打补丁前必须红"怎么落地 + +分两个 commit 推,第一个**故意不含 P0**: + +- `ci-windows` 应当在 `Scanner.GlobWalkSurvivesNamesTheCodePageCannotSpell` 失败 + → 证明这个测试**能**失败(而不是恰好通过); +- `ci-linux` 应当在 `check_narrow_conversions.sh` 失败并点名 `scanner.cppm` 那一行 + → 证明这道门**能**抓到真实缺陷,而不只是抓到我构造的负例。 + +第二个 commit 打上 P0,两条腿转绿。一次推送拿到两份证据。 + +--- + +## 10. 实现推翻了初稿的三处 + +1. **`build/resources.cppm` 与 `modgraph/p1689.cppm` 是误报。** 初稿那张审计表是按文件名 + grep 出来的,不算测量。逐个追输入来源后:前者narrow 的是 `windres`/`rc.exe` 的文件名 + (全 ASCII),后者拿到的路径必然已过 glob 过滤。**真正漏网的是 `pack/digest.cppm`** + —— 它吃的是对已发布包的**未经过滤**的 `recursive_directory_iterator`,初稿没提。 +2. **CI 门的作用域必须收窄。** 第一版覆盖 `src/modgraph src/manifest src/pack src/scaffold`, + 一开就是 **22 个命中,其中约 20 个是假阳性** —— `src/pack` 窄化的是 mcpp 自己产生的 + 名字(staging root、built binary、strip artifact),`src/manifest` 只窄化 `.extension()`。 + 一道有二十个假阳性的门,一个月内必然被绕过,而那条豁免会变成"曾经有过规则"的唯一记录。 + 收窄到 `src/modgraph src/scaffold`,并在脚本头写明**通过 ≠ 已审计**。 +3. **口径 B 的实现不能用 `mcpp.log`。** 它的默认 level 是 `off`,`log::warn` 默认什么都不打印 + —— 那样口径 B 会退化成口径 A,而且看起来像做了。正确通道是 `mcpp.diag`, + 它的模块注释本就写着"`log::debug` 和 `log::verbose` 不算用户可见"。 + +### 一个已知的、没有测试覆盖的环节 + +`cli.cppm` 里那个 scope guard 的**接线**只由阅读保证:记录逻辑有单测 +(`Glob.UnnarrowablePathsDedupToTheirSpellableAncestor`),Windows 用例断言了"确实记录到了", +但"CLI 真的把它打出来了"没有自动化验证 —— 那需要一个 Windows e2e,而本次 review +明确把范围限定在单元测试。写在这里,而不是假装它被覆盖了。 diff --git a/.agents/skills/mcpp-contributing/SKILL.md b/.agents/skills/mcpp-contributing/SKILL.md index 9ab7c9d2..017978f7 100644 --- a/.agents/skills/mcpp-contributing/SKILL.md +++ b/.agents/skills/mcpp-contributing/SKILL.md @@ -278,6 +278,40 @@ docs/ ← 用户文档 .agents/skills/ ← Agent 技能文档 ``` +## 路径窄化不变式（走查得到的 path 不得直接 `.string()`） + +Windows 上 `std::filesystem::path::string()` 会把 native（宽）名经**进程 ANSI 代码页** +转换，遇到该代码页拼不出的字符就抛 `std::system_error`。**非 Windows 上同一个调用只是 +一次拷贝，永不失败**——所以这个隐患在 Linux/macOS 上（包括它们的测试里）完全不可见。 + +它已经付过两次代价，每次戴着不同的面具：#230 抛出后逃到 `std::terminate`，git-bash +显示为**裸 exit 127**（看起来像"命令找不到"）；#516 逃到 `main()` 的 catch，显示为 +`internal: unhandled exception`（看起来像下载器的**解压/编码缺陷**）。#231 加固了三处 +调用点，漏掉了同一个 walk 循环里**早一行**执行的第四处。 + +规则（按用途选，不是三选一的风格问题）： + +| 用途 | 写法 | +|---|---| +| 与 ASCII 字面量比较 | **按 `path` 比**，根本不窄化 | +| 需要稳定身份（hash / key / digest） | `p.u8string()` —— 各平台都是 UTF-8，不碰代码页 | +| 需要交给编译器 / ninja / CDB | `mcpp::modgraph::try_narrow(p)`，并处理 `nullopt` | + +`try_narrow` 返回 `nullopt` 表示"这个文件没法出现在任何交给工具链的字符串里"。 +**跳过它，并且必须报出来**——`mcpp.diag` 的批次不变式对此已有规定：因为前提不满足而 +少做事，必须走 `diag::degraded()` 并给出 `impact`。静默丢弃是这类缺陷藏身的地方。 + +`src/modgraph/` 与 `src/manifest/` 是 leaf 层（全仓没有一条到 `mcpp.ui` / `mcpp.diag` +的 import 边），所以它们**记录**（`note_unnarrowable_path`），由 CLI 层排空上报。 + +`.github/tools/check_narrow_conversions.sh` 是硬门，但它只扫 `src/modgraph`、 +`src/scaffold`——**通过不等于已审计**。确有把握的站点用 `// NARROW-OK: <理由>` 标注， +理由必须写出"为什么这个输入不可能带这种名字"。 + +**测试只有跑在 Windows CI 上才有意义**，且必须自己检查 `GetACP()`：runner 镜像哪天默认 +UTF-8 ACP（65001），这类用例会静默变成永远绿的装饰品。参见 +`tests/unit/test_modgraph.cpp` 的 `Scanner.GlobWalkSurvivesNamesTheCodePageCannotSpell`。 + ## 注意事项 - C++23 模块项目，修改模块时注意 import 依赖顺序 diff --git a/.github/actions/bootstrap-mcpp/action.yml b/.github/actions/bootstrap-mcpp/action.yml index 6b50301b..4b498c4c 100644 --- a/.github/actions/bootstrap-mcpp/action.yml +++ b/.github/actions/bootstrap-mcpp/action.yml @@ -25,7 +25,7 @@ inputs: # `package.name`, so one of the two was simply unreachable — and which one # depended on the machine, which is why CI failed on `compat:lua` on # Windows and `mcpplibs.capi:lua` on Linux. Never pin below that. - default: '2026.8.17.2' + default: '2026.8.27.4' cache-target: description: also restore/save target/ (build artifacts + BMIs) required: false diff --git a/.github/actions/setup-macos-llvm/action.yml b/.github/actions/setup-macos-llvm/action.yml index 3cfdd438..f9aa8a4b 100644 --- a/.github/actions/setup-macos-llvm/action.yml +++ b/.github/actions/setup-macos-llvm/action.yml @@ -15,7 +15,7 @@ inputs: # Floor imposed by the index, not a routine bump — see # .github/actions/bootstrap-mcpp/action.yml for why 0.4.69 is required # (two packages named `lua` in one repo need openxlings/xlings#381). - default: '2026.8.17.2' + default: '2026.8.27.4' runs: using: composite diff --git a/.github/tools/check_narrow_conversions.sh b/.github/tools/check_narrow_conversions.sh new file mode 100755 index 00000000..6f4fd034 --- /dev/null +++ b/.github/tools/check_narrow_conversions.sh @@ -0,0 +1,125 @@ +#!/usr/bin/env bash +# +# Guard: a path that came out of a directory walk is never narrowed directly. +# +# WHY +# +# On Windows `std::filesystem::path::string()` converts the native (wide) name +# through the process ANSI code page and THROWS std::system_error when a +# character has no spelling there — "No mapping for the Unicode character +# exists in the target multi-byte code page". Off Windows the same call is a +# copy that cannot fail, so nothing on Linux or macOS — including their tests — +# can see the hazard. +# +# It has cost two incidents, each wearing a different mask: +# +# #230 a walked index tree held a CJK-named issue template; the throw +# escaped to std::terminate → __fastfail → git-bash reported a bare +# exit 127, which reads as "command not found". +# #516 cpp-httplib ships test/www/Dir/ and the `include_dirs = { "*" }` +# convention walks the whole extracted tarball; the throw escaped to +# main()'s catch as `internal: unhandled exception`, which reads as an +# extraction/encoding bug in the downloader. +# +# #231 hardened three call sites and missed a fourth — `is_excluded_walk_dir`, +# which runs ONE LINE EARLIER in the same walk loop. A fifth site is what this +# script exists to make expensive. +# +# THE RULE +# +# * Comparing against ASCII literals? Compare as `path`. Do not narrow. +# * Need a stable identity (hash, key)? `u8string()` — UTF-8 everywhere, +# never touches the code page. +# * Need a build-facing string (compiler +# argument, ninja file, CDB)? `mcpp::modgraph::try_narrow()`, +# and handle the nullopt. +# +# WHAT THIS DOES AND DOES NOT CATCH +# +# It greps the leaf layers that walk trees mcpp does not control. It catches a +# NEW direct narrowing written there. It does NOT catch a path narrowed after +# being passed out to another layer — that is what the try_narrow convention is +# for, and no grep can enforce it. Do not read a pass here as "audited". +# +# `.extension()` is deliberately NOT matched: an extension is ASCII in every +# case that reaches these predicates, so matching it would produce only noise — +# and noise is how a gate gets suppressed. +# +# Escape hatch: `// NARROW-OK: ` on the line itself or within the two +# lines above it. Use it when the input provably cannot carry an unspellable +# name, and say why — a bare marker with no argument is worse than no gate, +# because it reads as "someone checked". +# +# Usage: bash .github/tools/check_narrow_conversions.sh [repo_dir] + +set -uo pipefail + +REPO_DIR="${1:-$(pwd)}" +cd "$REPO_DIR" || { echo "FAIL: cannot cd to $REPO_DIR" >&2; exit 1; } + +# SCOPE, and why it is this narrow. +# +# The hazard needs a path from a tree MCPP DOES NOT CONTROL. Two directories +# qualify: src/modgraph walks arbitrary package and project trees, and +# src/scaffold enumerates third-party template providers. +# +# The first draft of this guard also covered src/pack and src/manifest and +# produced 22 hits, ~20 of them false: src/pack narrows names MCPP ITSELF +# produced (staging roots, built binaries, strip artifacts — all derived from +# validated ASCII package/target names), and src/manifest only ever narrows an +# `.extension()`. A gate with twenty false positives is a gate that gets +# suppressed within a month, and the suppression then becomes the only record +# that a rule existed. The real hazards in those two directories were fixed by +# hand instead (pack/digest.cppm, which feeds on an unfiltered +# recursive_directory_iterator over a published package). +# +# So: a pass here does NOT mean "the tree is audited". It means no NEW direct +# narrowing was written where this class originates. +SCAN_DIRS="src/modgraph src/scaffold" + +PATTERN='\.(filename|stem)\(\)\.(generic_)?string\(\)' + +fail=0 +found=0 + +for dir in $SCAN_DIRS; do + [ -d "$dir" ] || { echo "FAIL: $dir does not exist — this guard has gone stale" >&2; exit 1; } + while IFS= read -r file; do + # Strip // line comments before matching: several of these files DESCRIBE + # the forbidden call in prose (that is the point of the comments), and a + # guard that trips on its own documentation gets deleted. + while IFS=: read -r lineno text; do + [ -n "${lineno:-}" ] || continue + found=1 + # NARROW-OK on the line itself, or on either of the two lines above it. + ctx=$(sed -n "$(( lineno > 2 ? lineno - 2 : 1 )),${lineno}p" "$file") + case "$ctx" in + *NARROW-OK:*) continue ;; + esac + echo "FAIL: $file:$lineno narrows a path directly:" >&2 + echo " ${text# }" >&2 + fail=1 + done < <(sed 's://.*::' "$file" | grep -nE "$PATTERN") + done < <(find "$dir" -type f \( -name '*.cppm' -o -name '*.cpp' -o -name '*.hpp' \) | sort) +done + +if [ "$fail" = 1 ]; then + cat >&2 <<'EOF' + + Use one of: + - compare as std::filesystem::path (ASCII literals; no narrowing) + - p.u8string() (stable identity: hashes, keys) + - mcpp::modgraph::try_narrow(p) (build-facing; handle nullopt) + or annotate with `// NARROW-OK: `. + + Background: mcpp#516, mcpp#230, src/modgraph/glob.cppm. +EOF + exit 1 +fi + +if [ "$found" = 0 ]; then + echo "ok: no direct path narrowing in $SCAN_DIRS" +else + echo "ok: every direct narrowing in $SCAN_DIRS carries a NARROW-OK rationale" +fi +exit 0 diff --git a/.github/workflows/bootstrap-macos.yml b/.github/workflows/bootstrap-macos.yml index ecbfb38e..c725a236 100644 --- a/.github/workflows/bootstrap-macos.yml +++ b/.github/workflows/bootstrap-macos.yml @@ -17,7 +17,7 @@ jobs: # Dormant (workflow_dispatch only), but kept in step with the rest — # check_version_pins.sh holds it there. Floor: 0.4.69, below which the # index cannot resolve two packages that share a short name. - XLINGS_VERSION: '2026.8.17.2' + XLINGS_VERSION: '2026.8.27.4' steps: - uses: actions/checkout@v4 diff --git a/.github/workflows/ci-fresh-install.yml b/.github/workflows/ci-fresh-install.yml index 2995291c..fd00913c 100644 --- a/.github/workflows/ci-fresh-install.yml +++ b/.github/workflows/ci-fresh-install.yml @@ -152,7 +152,7 @@ jobs: env: XLINGS_NON_INTERACTIVE: '1' run: | - curl -fsSL https://raw.githubusercontent.com/openxlings/xlings/main/tools/other/quick_install.sh | bash -s v2026.8.17.2 + curl -fsSL https://raw.githubusercontent.com/openxlings/xlings/main/tools/other/quick_install.sh | bash -s v2026.8.27.4 echo "$HOME/.xlings/subos/current/bin" >> "$GITHUB_PATH" - name: Install mcpp and config mirror @@ -293,7 +293,7 @@ jobs: - name: Install xlings + mcpp run: | - curl -fsSL https://raw.githubusercontent.com/openxlings/xlings/main/tools/other/quick_install.sh | bash -s v2026.8.17.2 + curl -fsSL https://raw.githubusercontent.com/openxlings/xlings/main/tools/other/quick_install.sh | bash -s v2026.8.27.4 # Deliberately NOT writing to $GITHUB_PATH here. On container # images that declare no PATH in their config (opensuse/ # tumbleweed), appending a single dir to GITHUB_PATH makes the @@ -364,7 +364,7 @@ jobs: # (older ones carry minos=15 and refuse to start). # v0.4.51+: in-process sha256 — this image has no sha256sum # binary, so pinned fetches failed before it. - curl -fsSL https://raw.githubusercontent.com/openxlings/xlings/main/tools/other/quick_install.sh | bash -s v2026.8.17.2 + curl -fsSL https://raw.githubusercontent.com/openxlings/xlings/main/tools/other/quick_install.sh | bash -s v2026.8.27.4 echo "$HOME/.xlings/subos/current/bin" >> "$GITHUB_PATH" - name: Install mcpp and config mirror diff --git a/.github/workflows/ci-linux-e2e.yml b/.github/workflows/ci-linux-e2e.yml index 6b443920..248279a8 100644 --- a/.github/workflows/ci-linux-e2e.yml +++ b/.github/workflows/ci-linux-e2e.yml @@ -237,7 +237,7 @@ jobs: - name: Bootstrap xlings + released mcpp run: | - curl -fsSL https://raw.githubusercontent.com/openxlings/xlings/main/tools/other/quick_install.sh | bash -s v2026.8.17.2 + curl -fsSL https://raw.githubusercontent.com/openxlings/xlings/main/tools/other/quick_install.sh | bash -s v2026.8.27.4 export PATH="$HOME/.xlings/subos/current/bin:$PATH" xlings update xlings install mcpp -y -g diff --git a/.github/workflows/ci-linux.yml b/.github/workflows/ci-linux.yml index 9a9d2d02..cbf6fb39 100644 --- a/.github/workflows/ci-linux.yml +++ b/.github/workflows/ci-linux.yml @@ -55,6 +55,21 @@ jobs: - name: Check version / xlings pin consistency run: bash .github/tools/check_version_pins.sh + # Same placement, same reason: pure text, no toolchain, under a second. + # + # This one is a HARD gate (unlike lint-ci-assertions.sh below) because it + # has no false positives left — its scope was cut to the two directories + # that walk trees mcpp does not control, and the one legitimate site + # carries a NARROW-OK rationale. See the script's header for why the + # scope is that narrow, and mcpp#516 for what it costs when it is missed. + # + # It runs on LINUX on purpose even though the bug it guards is + # Windows-only: it is text analysis, and putting it where the fast leg is + # means a violation is reported in seconds rather than after a Windows + # bootstrap. + - name: Check no walk-derived path is narrowed directly + run: bash .github/tools/check_narrow_conversions.sh + # Same placement, same reason: pure text, no toolchain. # # ⚠️ IT PRINTS AND DOES NOT FAIL, DELIBERATELY. The three rules it carries diff --git a/.github/workflows/cross-build-test.yml b/.github/workflows/cross-build-test.yml index 8c2e1517..95bdb29f 100644 --- a/.github/workflows/cross-build-test.yml +++ b/.github/workflows/cross-build-test.yml @@ -122,7 +122,7 @@ jobs: # release assets were uploaded in a broken state (records present, # blobs missing → 404 on GET); re-uploaded clean. The stale-INDEX # half is handled by the marker-clear below. - XLINGS_VERSION: '2026.8.17.2' + XLINGS_VERSION: '2026.8.27.4' run: | tarball="xlings-${XLINGS_VERSION}-linux-x86_64.tar.gz" bash "$GITHUB_WORKSPACE/.github/tools/fetch_release.sh" \ @@ -263,7 +263,7 @@ jobs: - name: Bootstrap mcpp via xlings env: XLINGS_NON_INTERACTIVE: '1' - XLINGS_VERSION: '2026.8.17.2' + XLINGS_VERSION: '2026.8.27.4' run: | tarball="xlings-${XLINGS_VERSION}-linux-x86_64.tar.gz" bash "$GITHUB_WORKSPACE/.github/tools/fetch_release.sh" \ diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index 780c1ccb..20b493a6 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -96,7 +96,7 @@ jobs: # Pin xlings to a known-good version. The upstream install # script always grabs `latest` (no version override), so we # download + self-install manually to avoid broken releases. - XLINGS_VERSION: '2026.8.17.2' + XLINGS_VERSION: '2026.8.27.4' run: | if [ ! -x "$HOME/.xlings/subos/default/bin/xlings" ]; then tarball="xlings-${XLINGS_VERSION}-linux-x86_64.tar.gz" @@ -289,7 +289,7 @@ jobs: - name: Bootstrap mcpp via xlings env: XLINGS_NON_INTERACTIVE: '1' - XLINGS_VERSION: '2026.8.17.2' + XLINGS_VERSION: '2026.8.27.4' run: | tarball="xlings-${XLINGS_VERSION}-linux-x86_64.tar.gz" bash "$GITHUB_WORKSPACE/.github/tools/fetch_release.sh" \ @@ -360,7 +360,7 @@ jobs: # below are pinned to the same version as XLINGS_VERSION; they are # NOT interpolated from it, so check_version_pins.sh scans for them # explicitly (they were absent from the old lock-step comment). - XLA="xlings-2026.8.17.2-linux-aarch64.tar.gz" + XLA="xlings-2026.8.27.4-linux-aarch64.tar.gz" # NOT fetch_release.sh: this asset is OPTIONAL and the `if` is the # point — an arch with no prebuilt xlings must fall through quietly, # while the helper retries a 404 five times before giving up. The one @@ -369,9 +369,9 @@ jobs: # cover it. if curl -fsSL --retry 3 --retry-delay 2 --retry-all-errors \ --connect-timeout 20 --max-time 600 -o "/tmp/$XLA" \ - "https://github.com/openxlings/xlings/releases/download/v2026.8.17.2/$XLA"; then + "https://github.com/openxlings/xlings/releases/download/v2026.8.27.4/$XLA"; then tar -xzf "/tmp/$XLA" -C /tmp - XLBIN=$(find /tmp/xlings-2026.8.17.2-linux-aarch64 -path '*/bin/xlings' -type f | head -1) + XLBIN=$(find /tmp/xlings-2026.8.27.4-linux-aarch64 -path '*/bin/xlings' -type f | head -1) if [ -n "$XLBIN" ]; then mkdir -p "$STAGING/$WRAPPER/registry/bin" cp "$XLBIN" "$STAGING/$WRAPPER/registry/bin/xlings" @@ -449,7 +449,7 @@ jobs: - name: Bootstrap mcpp via xlings env: XLINGS_NON_INTERACTIVE: '1' - XLINGS_VERSION: '2026.8.17.2' + XLINGS_VERSION: '2026.8.27.4' run: | if [ ! -x "$HOME/.xlings/subos/default/bin/xlings" ]; then WORK=$(mktemp -d) @@ -632,7 +632,7 @@ jobs: shell: bash env: XLINGS_NON_INTERACTIVE: '1' - XLINGS_VERSION: '2026.8.17.2' + XLINGS_VERSION: '2026.8.27.4' run: | # Captured before the `cd` below, in POSIX form: this step never # returns to the workspace, and GITHUB_WORKSPACE is a backslash diff --git a/CHANGELOG.md b/CHANGELOG.md index 4819456e..5795ba6e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -3,6 +3,74 @@ > 本文件追踪 `mcpp-community/mcpp` 公开仓的版本演进。 > 格式参考 [Keep a Changelog](https://keepachangelog.com/zh-CN/1.1.0/)。 +## [2026.8.27.2] — 2026-08-27 + +一个文件名把整个 Windows 构建打断了,而报错说的是别的事。完整分析见 +[`.agents/docs/2026-08-27-issue516-windows-acp-glob-walk-fix.md`](.agents/docs/2026-08-27-issue516-windows-acp-glob-walk-fix.md)。 + +⭐ **这是 `#230` 的同一处漏网,不是新缺陷。** `#231` 加固了三个窄化站点, +漏掉了同一个 walk 循环里**早一行**执行的第四处。 + +### 修复 + +- **⭐⭐ 一个当前代码页拼不出的目录名,会让 `mcpp` 在 Windows 上以内部错误退出。**(#516) + + `src/modgraph/scanner.cppm` 的 `is_excluded_walk_dir()` 用 + `dir.filename().string()` 取目录名。MSVC 的 `path::string()` 走 + `WideCharToMultiByte(ACP)`,遇到该代码页拼不出的字符就抛 `std::system_error`: + + ``` + error: internal: unhandled exception: No mapping for the Unicode character + exists in the target multi-byte code page. + ``` + + 该函数是 walk 循环体的**第一行**,每个目录条目过一次 —— 所以它比 `#231` + 加固过的 `path_matches_glob` **更早**执行,加固那里对目录名从来无效。 + + 触发条件比看上去宽:`include_dirs = { "*" }` 这类以 `*` 开头的 glob, + 字面前缀为空,会从解压根**无界递归遍历整棵上游源码树**。 + 在 mcpplibs/mcpp-index `891b2f7` 上量:130 个 recipe 里有 **103 个**至少含一条 + 这样的 glob(口径:`.lua` 里出现以 `*` 开头的字符串字面量;抽查其分布为 + `*/include` / `*` / `*/src` / `*/mcpp.toml` 等,全部是 glob,无假阳性)。 + 这个比例会随索引增长而变,写下的是当天那个 commit 的数。 + cpp-httplib 带了 `test/www/日本語Dir/`,于是 `httplib` / `httplib-tls` / + `httplib-zstd` 三个测试在 Windows 上一起挂 —— 而 Linux/macOS 全绿,因为那两个 + 平台上 `path::string()` 不做任何编码转换。 + + **报错指向的方向是错的**:它看起来像下载器的解压/编码缺陷。实际上解压是**对的** + —— `ERROR_NO_UNICODE_TRANSLATION` 的前提正是宽名里有 ACP 拼不出的字符; + 若真落成了 mojibake,反而不会抛。 + +- **窄化收敛成一条规则,而不是第四个 try/catch。**(#516) + + `mcpp::modgraph::try_narrow()` 是走查路径变成窄串的唯一入口。按用途分三档: + 与 ASCII 字面量比较 → **按 `path` 比,不窄化**;需要稳定身份(hash/digest)→ + `u8string()`;需要交给编译器/ninja/CDB → `try_narrow()` 并处理 `nullopt`。 + `.github/tools/check_narrow_conversions.sh` 是硬门(作用域被刻意收窄到 + `src/modgraph`、`src/scaffold` 两处 —— 第一版覆盖四个目录、22 个命中里 20 个是 + 假阳性,那样的门一个月内就会被绕过)。 + +- **跳过的文件不再是静默的。** 一个无法命名的条目会按目录报告一次,走 + `mcpp.diag` 的 `degraded` 通道(它的批次不变式本就要求"因前提不满足而少做事 + 必须给出 `impact`"): + + ``` + warning: '' contains names this system's active code page cannot represent + impact: those files take no part in the build + ``` + +- **`interface_set_digest` 不再依赖宿主代码页。** + + 它用 `.string()` 折入文件名,而输入是对已发布包 interface 目录的**未经过滤**的 + `recursive_directory_iterator` 走查。除了会抛,它还让**在 Linux 上打包、在 + Windows 上校验**的同一棵树对非 ASCII 名字给出不同摘要 —— 表现为 + "does not match what was packaged",本文件能产生的最吓人的诊断。 + 改用 `u8string()`(各平台同一串字节);纯 ASCII 名字字节不变,已发布包的摘要不变。 + +### 其他 + +- 内部依赖的 xlings pin 升至 `2026.8.27.4`。 + ## [2026.8.27.1] — 2026-08-27 目标侧被解析出来了,只发给了一个编译单元。完整分析见 diff --git a/docs/05-mcpp-toml.md b/docs/05-mcpp-toml.md index ff02455a..7c76e886 100644 --- a/docs/05-mcpp-toml.md +++ b/docs/05-mcpp-toml.md @@ -621,6 +621,34 @@ skipped). Limits: `.asm` targets x86 only (hard error elsewhere — gate the files off other targets), `.S` is unavailable on the MSVC toolchain, and `.asm` means NASM syntax (MASM sources should be `!`-excluded). +### File names outside the host code page + +Globs are narrow strings, and so are compile commands and `build.ninja`. On +Windows those strings are produced in the process's **ANSI code page**, so a +file whose name has no spelling in that code page cannot be matched by a glob, +named on a compile command, or written into a build file. + +Such entries are skipped, and the skip is reported once per directory: + +```text +warning: 'C:/.../pkg/test/www' contains names this system's active code page cannot represent + impact: those files take no part in the build + hint: Windows only: this is the process ANSI code page, which `chcp` does not change. ... +``` + +The reported path is the nearest ancestor whose name the code page *can* spell, +in generic (`/`) spelling. The offending name itself is never printed: rendering +it would throw the same exception the message is reporting. + +`chcp` sets the *console* code page and has no effect here. Names that are only +test data or documentation are harmless — an upstream tarball carrying a +Japanese-named fixture directory builds fine on an en-US host. Sources are not: +they need renaming, or a host whose code page covers them. + +Linux and macOS perform no such conversion, so nothing is skipped there. A +package that builds on one and not the other, with an +`internal: unhandled exception` from a code-page message, was mcpp#516. + ### 2.4 `[lib]` — Library Root Module Convention ```toml diff --git a/docs/zh/05-mcpp-toml.md b/docs/zh/05-mcpp-toml.md index cbf38566..44579cd0 100644 --- a/docs/zh/05-mcpp-toml.md +++ b/docs/zh/05-mcpp-toml.md @@ -545,6 +545,30 @@ nasm 则**硬失败**(汇编绝不静默跳过)。限制:`.asm` 仅限 x86 目 报错——用条件 sources 门控)、MSVC 工具链不支持 `.S`、`.asm` 即 NASM 语法 (MASM 源请用 `!` 排除)。 +### 宿主代码页之外的文件名 + +glob 是窄字符串,编译命令和 `build.ninja` 也是。在 Windows 上这些字符串由进程的 +**ANSI 代码页**产生,因此一个名字在该代码页里无法拼写的文件,既匹配不了 glob,也 +写不进编译命令或构建文件。 + +这类条目会被跳过,并按目录报告一次: + +```text +warning: 'C:/.../pkg/test/www' contains names this system's active code page cannot represent + impact: those files take no part in the build + hint: Windows only: this is the process ANSI code page, which `chcp` does not change. ... +``` + +报告里给的是**最近一个代码页拼得出的祖先目录**,用通用(`/`)写法。拼不出的那个名字本身 +永远不会被打印:渲染它会抛出这条消息正在报告的同一个异常。 + +`chcp` 改的是**控制台**代码页,对此无效。若这些名字只是测试数据或文档,跳过是无害 +的——上游 tarball 里带一个日文夹具目录,在 en-US 宿主上照样构建。源文件则不然:需要 +改名,或换一台代码页覆盖得了的机器。 + +Linux 与 macOS 不做这种转换,因此那里不会跳过任何名字。一个包在一边能构建、在另一 +边报 `internal: unhandled exception` 并指向代码页,就是 mcpp#516。 + ### 2.4 `[lib]` — 库根模块约定 ```toml diff --git a/mcpp.toml b/mcpp.toml index a592ecf8..edb63bf4 100644 --- a/mcpp.toml +++ b/mcpp.toml @@ -1,6 +1,6 @@ [package] name = "mcpp" -version = "2026.8.27.1" +version = "2026.8.27.2" description = "Modern C++ build & package management tool" license = "Apache-2.0" authors = ["mcpp-community"] diff --git a/src/cli.cppm b/src/cli.cppm index 2ad20c05..95a9214c 100644 --- a/src/cli.cppm +++ b/src/cli.cppm @@ -33,6 +33,8 @@ import mcpp.platform.env; // --offline → MCPP_OFFLINE import mcpp.platform.runtime_search; // linker-wrapper path-injection opt-out import mcpp.ui; import mcpp.log; +import mcpp.diag; // the single sink for the report below +import mcpp.modgraph.glob; // take_unnarrowable_paths() export namespace mcpp::cli { @@ -95,8 +97,51 @@ void print_usage() { std::println("Docs: https://github.com/mcpp-community/mcpp/tree/main/docs"); } +// The ONE place this run's "could not be named in the active code page" +// records are reported. +// +// `src/modgraph/` and `src/manifest/` are leaf layers — not one module in +// either imports `mcpp.ui` or `mcpp.diag` — so the glob walk RECORDS +// (mcpp::modgraph::note_unnarrowable_path) and the CLI reports. A scope guard +// rather than a call before `return` because run() has several exits — an +// unknown command, a parse error, `--help`, the dispatched action — and the +// one added next year would silently drop the report, which is precisely the +// failure shape this whole change is about. +// +// Reported as `degraded`, not `warning`: mcpp.diag's batch invariant is that a +// branch doing LESS because a precondition was not met owes the user an +// `impact` sentence. Skipping files is doing less. +// +// Known boundary: both `diag::flush(strict)` call sites live inside the build +// path and run before this guard fires, so `--strict` does not promote these +// to errors. Deliberate — the alternative is a second drain point, i.e. a +// second answerer for the same question. +struct ReportUnnarrowablePaths { + // A destructor is implicitly noexcept, so anything escaping this body is + // std::terminate — and run() can be left by an exception (main() catches + // one), which is exactly when this runs during unwinding. A change whose + // entire subject is "an uncaught exception must not end the build" does + // not get to introduce a second one in its own reporting path. + ~ReportUnnarrowablePaths() try { + for (auto const& anchor : mcpp::modgraph::take_unnarrowable_paths()) { + mcpp::diag::degraded( + "path/codepage", + std::format("'{}' contains names this system's active code " + "page cannot represent", anchor), + "those files take no part in the build", + "Windows only: this is the process ANSI code page, which " + "`chcp` does not change. Harmless when the names are test " + "data or docs; if they are sources, rename them or build on a " + "system whose code page covers them."); + } + } catch (...) { + // Losing the report is bad; terminating instead of it is worse. + } +}; + int run(int argc, char** argv) { namespace cl = mcpplibs::cmdline; + ReportUnnarrowablePaths reportUnnarrowable_; // ─── --quiet / --no-color: pre-scan ───────────────────────────────── // The cmdline lib propagates global options into nested subcommand diff --git a/src/modgraph/glob.cppm b/src/modgraph/glob.cppm index 7d929c54..ca50bae0 100644 --- a/src/modgraph/glob.cppm +++ b/src/modgraph/glob.cppm @@ -32,6 +32,73 @@ std::filesystem::path native_path_from_generic(std::string_view s) { return p; } +// ─── narrowing a walk-derived path ──────────────────────────────────────── +// +// THE ONE PLACE a path that came out of a directory walk becomes a narrow +// string. A direct `.string()` / `.generic_string()` on such a path needs a +// written reason (`// NARROW-OK: …`) and there is exactly one of those today, +// in p1689.cppm. This is a convention with a gate behind it, not a guarantee +// the compiler enforces — see the gate's header for what it does and does not +// cover. +// +// Why it needs to exist at all: on Windows `path::string()` converts the +// native (wide) name through the process's ANSI code page and THROWS +// `std::system_error` when a character has no spelling there — +// "No mapping for the Unicode character exists in the target multi-byte code +// page". Off Windows the same call is a copy that cannot fail, so this whole +// hazard is invisible on Linux and macOS — including to their tests. +// +// It has cost two incidents, wearing a different mask each time: +// +// #230 a walked index tree held `bug-report---问题反馈.md`; the throw +// escaped to std::terminate → `__fastfail(0xC0000409)` → git-bash +// reported a bare **exit 127**, which reads as "command not found". +// #516 cpp-httplib ships `test/www/日本語Dir/`, and the `include_dirs = +// { "*" }` convention walks the whole extracted tarball; the throw +// escaped to main()'s catch as `internal: unhandled exception`, +// which reads as an **extraction/encoding bug in the downloader**. +// +// #231 hardened three sites against it and missed a fourth +// (`is_excluded_walk_dir`, which runs one line EARLIER in the same walk) — +// which is why this is now a single function with a CI gate behind it +// (.github/tools/check_narrow_conversions.sh) rather than a fourth try/catch. +// +// nullopt means: this path cannot be named in any string we hand to a +// compiler, a build file, or a glob. Skip it — and record it, because +// "silently not built" is exactly where this class of bug hides. +std::optional try_narrow(const std::filesystem::path& p) { + try { + return p.generic_string(); + } catch (const std::exception&) { + return std::nullopt; + } +} + +// Record that something had to be skipped because `try_narrow` could not name +// it. Deduplicated to the nearest ANCESTOR that CAN be named: one unreadable +// subtree produces one record, not one per file. +// +// What is stored is that ancestor — never the offending name, which by +// definition cannot be put into a message without throwing the very exception +// this module exists to avoid. (Diagnostic code walking into its own trap is +// the most likely way this regresses.) +// +// The stored spelling is GENERIC (`/`), because that is what try_narrow +// produces and there is no second narrowing here to disagree with it. On +// Windows the reported path therefore reads `C:/pkg/test/www`, not +// `C:\pkg\test\www`; docs/05-mcpp-toml.md shows it that way too. +void note_unnarrowable_path(const std::filesystem::path& p); + +// Take and clear this run's records. +// +// modgraph is a leaf layer — no module under `src/modgraph/` or +// `src/manifest/` imports `mcpp.ui` or `mcpp.diag` — so it RECORDS and the +// CLI reports. Drained in exactly one place (`cli::run`'s scope guard), which +// is what keeps "recorded but never shown" from becoming the next silent +// failure. The rule is written up in .agents/skills/mcpp-contributing/SKILL.md +// ("路径窄化不变式") and the user-facing behaviour in docs/05-mcpp-toml.md. +std::vector take_unnarrowable_paths(); + // Does `candidate` match `glob`, interpreted relative to `root`? // // Supports "**" (any number of directory levels) and "*" (within one segment). @@ -43,16 +110,15 @@ bool path_matches_glob(const std::filesystem::path& candidate, const std::filesystem::path& root, std::string_view glob) { - std::string rel; - try { - rel = candidate.lexically_relative(root).generic_string(); - } catch (const std::exception&) { - // MSVC's narrow conversion throws std::system_error when the native - // (wide) name has no spelling in the ANSI codepage (e.g. a CJK - // filename on an en-US host — mcpp#230 hit this on an issue template - // inside a walked index tree). Such a name can never be spelled in a - // glob or a compile command either: not a match, and never a reason to - // tear down the whole build. + // lexically_relative is pure path arithmetic and cannot throw; the + // narrowing is the part that can, so it is the part that goes through + // try_narrow. + auto rel = try_narrow(candidate.lexically_relative(root)); + if (!rel) { + // A name the code page cannot spell can never match a glob (a glob is + // a narrow string) and could never reach a compile command either. + // Not a match — and not a reason to tear down the whole build. + note_unnarrowable_path(candidate); return false; } @@ -91,7 +157,46 @@ bool path_matches_glob(const std::filesystem::path& candidate, }; return rec(0, 0); }; - return match(rel, glob); + return match(*rel, glob); } } // namespace mcpp::modgraph + +// ── implementation ────────────────────────────────────────────────────────── + +namespace mcpp::modgraph { +namespace { + +// A run's worth of unnarrowable subtrees, keyed by their nearest spellable +// ancestor. `std::set` so the report comes out in a stable order regardless of +// directory-enumeration order, which the standard leaves unspecified. +std::mutex g_unnarrowableMu; +std::set g_unnarrowable; + +} // namespace + +void note_unnarrowable_path(const std::filesystem::path& p) { + // Climb to the first ancestor this code page CAN spell. `p` itself fails + // by construction; usually exactly one component is at fault, so the + // parent already succeeds. + std::string anchor; + for (auto dir = p.parent_path();; dir = dir.parent_path()) { + if (auto s = try_narrow(dir)) { anchor = std::move(*s); break; } + if (dir.parent_path() == dir) break; // reached the root, still unspellable + } + // Every component was unspellable (or `p` was a bare relative name). Say so + // rather than reporting an empty path, which reads as a bug in the report. + if (anchor.empty()) anchor = "(a path this code page cannot spell)"; + + std::lock_guard lk(g_unnarrowableMu); + g_unnarrowable.insert(std::move(anchor)); +} + +std::vector take_unnarrowable_paths() { + std::lock_guard lk(g_unnarrowableMu); + std::vector out(g_unnarrowable.begin(), g_unnarrowable.end()); + g_unnarrowable.clear(); + return out; +} + +} // namespace mcpp::modgraph diff --git a/src/modgraph/p1689.cppm b/src/modgraph/p1689.cppm index 028a4595..a591c641 100644 --- a/src/modgraph/p1689.cppm +++ b/src/modgraph/p1689.cppm @@ -336,7 +336,14 @@ scan_file(const std::filesystem::path& source, // One unique name per source — flat layout in tmpDir is fine since // the caller gives us a fresh dir per build. - auto stem = source.filename().string(); + // `source` is always a glob-filtered path. path_matches_glob drops any + // candidate try_narrow() cannot spell, so a name this code page has no + // spelling for never reaches a scan — and a PROJECT ROOT that cannot be + // spelled makes every candidate fail that same filter, so the build stops + // at "no sources" well before here. What this feeds is a compiler command + // line anyway: a path that cannot be narrowed cannot be handed to a + // compiler at all. (mcpp#516 audit) + auto stem = source.filename().string(); // NARROW-OK: glob-filtered, see above auto base = tmpDir / std::format("{}_{}", std::hash{}(source.string()) % 1000000, stem); auto ddi = base; ddi += ".ddi"; diff --git a/src/modgraph/scanner.cppm b/src/modgraph/scanner.cppm index 9a983373..c4635a27 100644 --- a/src/modgraph/scanner.cppm +++ b/src/modgraph/scanner.cppm @@ -235,8 +235,25 @@ submodule_paths(const std::filesystem::path& root) { // package's source glob. bool is_excluded_walk_dir(const std::filesystem::path& dir, const std::filesystem::path& root) { - auto name = dir.filename().string(); - if (name == ".mcpp" || name == ".git" || name == "target") return true; + // Compare as paths. Do NOT narrow. + // + // #516: `dir.filename().string()` went through MSVC's wide→ANSI + // conversion and threw std::system_error for any directory name the + // active code page cannot spell (`test/www/Dir/` in cpp-httplib). + // This function is the FIRST line of the walk loop and runs once per + // directory entry, so it fires before `path_matches_glob`'s guard — + // hardening that one (#231) could never cover a directory name. + // + // The three literals are ASCII, so their conversion to the native + // representation is lossless, and `path::operator==` compares native + // strings case-sensitively — byte-for-byte the same decision the narrow + // comparison made. Static constants rather than temporaries per entry: + // #225 bounded this walk for a reason, and this is on its hot path. + static const std::filesystem::path kMcppDir{".mcpp"}; + static const std::filesystem::path kGitDir{".git"}; + static const std::filesystem::path kTargetDir{"target"}; + const auto name = dir.filename(); + if (name == kMcppDir || name == kGitDir || name == kTargetDir) return true; auto const& submodules = submodule_paths(root); if (submodules.empty()) return false; std::error_code ec; @@ -597,6 +614,14 @@ std::expected scan_file(const std::filesystem::path& file // have meant" a module interface would be a second answer to the same // question — the very thing that produced this defect. if (u.kind == mcpp::SourceKind::Other) { + // Narrow through try_narrow even here. Everything that reaches + // scan_file today came through a glob filter, so the conversion cannot + // actually fail — but a DIAGNOSTIC that throws the exception it is + // describing is the single most likely way this class of bug comes + // back, and it costs one `value_or` to make that impossible. + auto nameNarrow = try_narrow(file.filename()).value_or( + "(a name this code page cannot spell)"); + auto extNarrow = try_narrow(file.extension()).value_or(""); return std::unexpected(ScanError{ file, 0, std::format( "'{}' is listed in [build] sources, and mcpp has no role for the " "extension '{}'.\n" @@ -608,9 +633,9 @@ std::expected scan_file(const std::filesystem::path& file " Otherwise remove it from `sources` — headers belong in " "`include_dirs`, and\n" " Windows resource scripts in `[resources]`.", - file.filename().string(), - file.extension().string().empty() ? "(none)" : file.extension().string(), - file.extension().string().empty() ? ".ixx" : file.extension().string()) }); + nameNarrow, + extNarrow.empty() ? "(none)" : extNarrow, + extNarrow.empty() ? ".ixx" : extNarrow) }); } // C-like files are not C++ modules: they cannot legally contain `module` / `import` @@ -1012,7 +1037,10 @@ void scan_one_into(ScanResult& result, if (ext.empty()) continue; bool seen = false; for (auto const& f : all_files) - if (f.extension().string() == ext) { seen = true; break; } + // Compare as paths — `ext` is ASCII, so it converts to the native + // representation losslessly, and no walk-derived name is narrowed + // (see mcpp::modgraph::try_narrow). + if (f.extension() == ext) { seen = true; break; } if (!seen) { result.warnings.push_back(ScanError{root, 0, std::format( "[build] module_extensions declares '{}' but no source file " diff --git a/src/pack/digest.cppm b/src/pack/digest.cppm index 60917d04..13e0c566 100644 --- a/src/pack/digest.cppm +++ b/src/pack/digest.cppm @@ -43,9 +43,38 @@ std::string file_digest(const std::filesystem::path& p) { return "fnv1a:" + mcpp::toolchain::hash_file(p); } +namespace { + +// The file NAME as platform-independent UTF-8 bytes. +// +// NOT `.string()`, for two independent reasons: +// +// 1. On Windows `.string()` converts through the process ANSI code page and +// THROWS std::system_error for a name that code page cannot spell. This +// input is an UNFILTERED `recursive_directory_iterator` walk of a +// published package's interface directory (prebuilt.cppm), so a package +// shipping one non-ASCII filename would abort the build with +// "No mapping for the Unicode character exists…" — mcpp#516's failure +// class, at a site that never went through a glob filter. +// 2. It makes the digest DEPEND ON THE HOST. A package digested on Linux +// (UTF-8 bytes) and verified on Windows (ANSI bytes) would disagree about +// a non-ASCII name while every byte on disk was identical — a false +// "does not match what was packaged", which is the most alarming +// diagnostic this file can produce. +// +// `u8string()` is UTF-8 on every platform and never touches the code page. For +// ASCII names — every package published to date — the bytes are identical, so +// no existing digest changes. +std::string name_utf8_(const std::filesystem::path& p) { + auto u8 = p.filename().u8string(); + return std::string(reinterpret_cast(u8.data()), u8.size()); +} + +} // namespace + std::string interface_set_digest(const std::vector& files) { std::vector> byName; - for (auto const& f : files) byName.emplace_back(f.filename().string(), f); + for (auto const& f : files) byName.emplace_back(name_utf8_(f), f); std::ranges::sort(byName, {}, &std::pair::first); std::string acc; diff --git a/src/platform/xlings/xlings.cppm b/src/platform/xlings/xlings.cppm index 0f8a5e84..df085e38 100644 --- a/src/platform/xlings/xlings.cppm +++ b/src/platform/xlings/xlings.cppm @@ -45,7 +45,7 @@ namespace pinned { // in lock-step by hand; that list was already missing both composite // actions, which is how CI's sandbox sat on 0.4.30 unnoticed while // everything else had moved on. Don't reintroduce a hand-maintained list. - inline constexpr std::string_view kXlingsVersion = "2026.8.17.2"; + inline constexpr std::string_view kXlingsVersion = "2026.8.27.4"; inline constexpr std::string_view kNasmVersion = "3.02"; } diff --git a/src/scaffold/template.cppm b/src/scaffold/template.cppm index 09bfda2e..51853759 100644 --- a/src/scaffold/template.cppm +++ b/src/scaffold/template.cppm @@ -137,16 +137,25 @@ struct TemplateMeta { std::expected load_meta(const std::filesystem::path& templateDir) { + // u8string(), not string(): `templateDir` comes from a directory walk of a + // third-party template provider, and on Windows `.string()` throws for a + // name the active code page cannot spell (mcpp#516). A diagnostic that + // throws the exception it is trying to report is the worst version of this + // bug, and these two are diagnostics. + auto dirNameU8 = templateDir.filename().u8string(); + auto dirName = std::string(reinterpret_cast(dirNameU8.data()), + dirNameU8.size()); + auto metaPath = templateDir / "template.toml"; if (!std::filesystem::exists(metaPath)) { return std::unexpected(std::format( - "template '{}' has no template.toml", templateDir.filename().string())); + "template '{}' has no template.toml", dirName)); } auto doc = mcpp::libs::toml::parse_file(metaPath); if (!doc) { return std::unexpected(std::format( "template '{}': bad template.toml: {}", - templateDir.filename().string(), doc.error().message)); + dirName, doc.error().message)); } TemplateMeta meta; @@ -262,7 +271,15 @@ list_templates(const std::filesystem::path& packageRoot) { } continue; } - auto templateName = e.path().filename().string(); + // u8string(), not string(): this is an unfiltered directory walk of a + // third-party template provider, and on Windows `.string()` throws for + // any directory name the active code page cannot spell (mcpp#516). + // The name is validated as "one ASCII atom" three lines down, so a + // non-ASCII name still gets refused — it now gets refused with the + // message that explains it, instead of aborting the whole command. + auto nameU8 = e.path().filename().u8string(); + auto templateName = std::string(reinterpret_cast(nameU8.data()), + nameU8.size()); auto parsedName = mcpp::pm::parse_package_selector(templateName); if (!parsedName || parsedName->namespace_) { return std::unexpected(std::format( diff --git a/src/version.cppm b/src/version.cppm index bc5da918..dfc26637 100644 --- a/src/version.cppm +++ b/src/version.cppm @@ -31,6 +31,6 @@ import std; export namespace mcpp { -inline constexpr std::string_view MCPP_VERSION = "2026.8.27.1"; +inline constexpr std::string_view MCPP_VERSION = "2026.8.27.2"; } // namespace mcpp diff --git a/tests/unit/test_modgraph.cpp b/tests/unit/test_modgraph.cpp index 25d36678..23bae543 100644 --- a/tests/unit/test_modgraph.cpp +++ b/tests/unit/test_modgraph.cpp @@ -883,3 +883,137 @@ TEST(Scanner, RelativeIncludeFlagsAbsolutized) { std::filesystem::remove_all(dir); } + +// ─── mcpp#516 / #230: names the active code page cannot spell ────────────── +// +// On Windows `path::string()` converts the native (wide) name through the +// process ANSI code page and throws std::system_error when a character has no +// spelling there. OFF Windows the same call is a copy that cannot fail — so +// the defect is NOT FALSIFIABLE on Linux or macOS, and the Windows leg of CI +// is the only place the last test below means anything. It says so out loud +// (GTEST_SKIP with a reason) rather than passing vacuously. + +namespace { + +// Can the active code page spell this name? Asked through path's own +// conversion, i.e. the exact call the code under test makes. Doubles as the +// guard against a future runner image whose default ACP is UTF-8 (65001), +// which would otherwise quietly turn the regression test into decoration. +[[maybe_unused]] bool acp_can_spell(const std::wstring& w) { + try { (void)std::filesystem::path(w).string(); return true; } + catch (const std::exception&) { return false; } +} + +} // namespace + +// The recorder: one unreadable subtree produces ONE record, keyed by the +// nearest ancestor that CAN be named — never by the offending name, which +// cannot be put into a message without throwing the very exception being +// reported. Platform-independent logic, so this runs everywhere. +TEST(Glob, UnnarrowablePathsDedupToTheirSpellableAncestor) { + (void)take_unnarrowable_paths(); // other tests in this binary walk too + + const std::filesystem::path base = "/pkg/test/www"; + note_unnarrowable_path(base / "bad" / "a.txt"); + note_unnarrowable_path(base / "bad" / "b.txt"); + note_unnarrowable_path(base / "other.txt"); + + auto notes = take_unnarrowable_paths(); + ASSERT_EQ(notes.size(), 2u); // ".../www/bad" and ".../www" — not three files + EXPECT_NE(std::find(notes.begin(), notes.end(), (base / "bad").generic_string()), + notes.end()); + EXPECT_NE(std::find(notes.begin(), notes.end(), base.generic_string()), + notes.end()); + + // take() clears: the next command reports afresh rather than replaying. + EXPECT_TRUE(take_unnarrowable_paths().empty()); +} + +// A non-ASCII directory name must not disturb the walk on ANY platform. This +// one is reproducible everywhere (on Windows the UTF-8 bytes land as whatever +// the ACP makes of them, which still exercises the walk). +TEST(Scanner, GlobWalkHandlesNonAsciiNames) { + auto dir = make_tempdir("mcpp-scanner-nonascii"); + // U+65E5 U+672C U+8A9E in UTF-8, spelled as bytes so the test does not + // depend on the source file's encoding. Split before "Dir" because a C++ + // hex escape is greedy — "\x9ED" would be one (out-of-range) escape. + std::filesystem::create_directories( + dir / "\xE6\x97\xA5\xE6\x9C\xAC\xE8\xAA\x9E" "Dir"); + write(dir / "zzz_ascii" / "x.h", "#pragma once\n"); + + std::vector dirs; + ASSERT_NO_THROW({ dirs = expand_dir_glob(dir, "*"); }); + EXPECT_NE(std::find(dirs.begin(), dirs.end(), dir / "zzz_ascii"), dirs.end()); + + std::vector files; + ASSERT_NO_THROW({ files = expand_glob(dir, "**/*.h"); }); + EXPECT_NE(std::find(files.begin(), files.end(), dir / "zzz_ascii" / "x.h"), + files.end()); + + std::filesystem::remove_all(dir); +} + +// mcpp#516 proper: a directory name the ACTIVE CODE PAGE cannot spell used to +// throw out of `is_excluded_walk_dir`'s `.filename().string()` — the first +// line of the walk loop, which is why #231 hardening `path_matches_glob` (one +// line later) could never cover it. The throw escaped to main() as +// error: internal: unhandled exception: No mapping for the Unicode +// character exists in the target multi-byte code page. +// and exit 70. +// +// Asserts (a) the walk does not throw, (b) it is not TRUNCATED at the bad +// entry, and (c) the skip was RECORDED — silently dropping the file is the +// half-fix this whole change exists to avoid. +TEST(Scanner, GlobWalkSurvivesNamesTheCodePageCannotSpell) { +#ifndef _WIN32 + GTEST_SKIP() << "path::string() performs no encoding conversion off Windows; " + "mcpp#516 is not reproducible here"; +#else + // Devanagari is in NO Windows ANSI code page, so this case stays live on + // every non-UTF-8 ACP. The Japanese name mirrors mcpp#516's real input + // (cpp-httplib's test/www/Dir), but CP932/936/950 CAN spell it, so on + // Japanese and Chinese hosts it alone would silently prove nothing. + // + // Built from explicit code units rather than a literal: the test must not + // depend on the source file's encoding, nor on /utf-8 reaching the + // compiler, nor on a checkout preserving the bytes. + // U+0915 U+0916 U+0917 = DEVANAGARI KA KHA GA + const std::wstring unspellable{wchar_t(0x0915), wchar_t(0x0916), wchar_t(0x0917), + L'D', L'i', L'r'}; + // U+65E5 U+672C U+8A9E = the three han characters in mcpp#516's directory + const std::wstring japanese{wchar_t(0x65E5), wchar_t(0x672C), wchar_t(0x8A9E), + L'D', L'i', L'r'}; + + if (acp_can_spell(unspellable)) { + GTEST_SKIP() << "the active code page can spell any name (UTF-8 ACP); " + "mcpp#516 is unreachable on this host"; + } + + auto dir = make_tempdir("mcpp-scanner-acp"); + std::filesystem::create_directories(dir / std::filesystem::path(unspellable)); + std::filesystem::create_directories(dir / std::filesystem::path(japanese)); + // A good neighbour, to prove the walk continued past the bad entry. + write(dir / "zzz_ascii" / "x.h", "#pragma once\n"); + + (void)take_unnarrowable_paths(); + + // The include-dir channel — mcpp#516's actual path (`include_dirs = {"*"}`, + // whose literal prefix is empty, so the walk starts at the package root). + std::vector dirs; + ASSERT_NO_THROW({ dirs = expand_dir_glob(dir, "*"); }); + EXPECT_NE(std::find(dirs.begin(), dirs.end(), dir / "zzz_ascii"), dirs.end()) + << "the walk was truncated at the unspellable entry"; + + // The file channel — installedLayoutMatchesIndex's + // `expand_glob(verRoot, "mcpp.toml")` walks the whole tree the same way. + std::vector files; + ASSERT_NO_THROW({ files = expand_glob(dir, "**/*.h"); }); + EXPECT_NE(std::find(files.begin(), files.end(), dir / "zzz_ascii" / "x.h"), + files.end()); + + EXPECT_FALSE(take_unnarrowable_paths().empty()) + << "the skipped entries were not recorded, so nothing would be reported"; + + std::filesystem::remove_all(dir); +#endif +}