Skip to content

Sync upstream 0.42.24 into v3 performance integration #87

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Ivanbeethoven merged 1 commit into from
Oct 7, 2026
Merged

Sync upstream 0.42.24 into v3 performance integration #87

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion Cargo.lock
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion Cargo.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
[package]
name = "mega2"
version = "0.42.23"
version = "0.42.24"
edition = "2024"
license = "MIT OR Apache-2.0"
description = "Monorepo engine and Git hosting server for AI agents: trunk-based storage, Git Smart HTTP, Git LFS, OCI registry, and Agent Session Capture"
Expand Down
56 changes: 33 additions & 23 deletions docs/plan/plan-20261002.md
View file
Open in desktop

Large diffs are not rendered by default.

20 changes: 19 additions & 1 deletion docs/refactoring/history-projection.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1365,7 +1365,7 @@ P1 的清扫周期，以及闲置阈值 `idle_recycle_secs`（下界为 2T = 720
- 批量写入的块大小为 `BATCH_CHUNK_SIZE = 1000`，带死锁重试（`base_storage.rs`）。
- **Josh【代码】：** `walk2` 用 `known()` 剪枝；`history.rs` 中有一个单槽的"最近父提交树"缓存，专门优化线性历史。

### 7.2 规模估算（纯算术推导，未实测）
### 7.2 规模估算（纯算术推导，P0 实测见本节末条）
- **根链表。** H 行，每行约 8 + 41 + 41 B，加上 Postgres 元组头、行指针和对齐约 30 B；再算上主键与 `commit_id` 唯一索引，约 250 B/行。H=10⁶ 时约 250 MB。回走暂存表只在扫描期间有行：冷启动时峰值约 H 行，与根链表同一量级，接入后删除。
- **commit_map。** 每个视图的行数 = 该视图的 NEW 提交数 ≤ H。按 hex 文本 id 计，堆约 136 B/行；两个索引合计约 140 B（叶页填充率按 70%）。合计约 280–300 B/行。
- **mega_view_object + mega_view_object_ref。** 每个 NEW 视图提交对应：
Expand All @@ -1376,6 +1376,24 @@ P1 的清扫周期，以及闲置阈值 `idle_recycle_secs`（下界为 2T = 720
- **示例**（H=10⁶，V=100，每个视图的 NEW 提交平均占 5%，即 5×10⁶ 个 NEW 提交）：派生表合计约 4.7–8.3 GB，主要来自对象表。最坏情况是每个视图都覆盖几乎全部改动，此时 NEW 提交达 10⁸ 个，约 95–165 GB。如果采用 3.5 的"不持久化视图提交字节"优化，提交对象这部分可以基本消除。
- **单个新根提交对一个 Subdir 视图的开销。** 按层批量读 d 次 tree（d 为路径深度），最多新增一行映射。
- **冷启动。** 共 H/B 批，每批 O(1) 次提交查询，加 O(深度) 次批量 tree 读取。吞吐在 P0 基准中实测。
- **P0 基准实测（HP-24）。** 运行 `source .env.test && cargo run --release --example view_bench -- --params examples/view_bench.params.json`；参数冻结于 `7acc2b569475c137f905496438027fa174b002f0`，数据是 git.git v0.99.5 的 `3857284f7b892f855edffc5b9c196a0dd74b1b7d` 原始 pack（SHA-256 `812723cd7e7c540101e3cb297a7dc4f14f18f407cd68d76452e97321e2992003`）。计入判定的运行是 `target/tmp/view_bench/run-20261007T213147Z.json`，被测 HEAD `d2ac0beb0effa5fa1744d0c4cf1d8865ad39a5a7`，接缝与 example 的 SHA-256 见该证据 `env.code`。环境：Apple M4 Max 16 核、128 GiB、USB SSD、macOS 27.0.1、Postgres 18.6。H 三档为 307 / 615 / 1231，B=1000；每组 5 次预热、50 次样本；以下时间单位为 µs，大小单位为字节。
| H | 根链冷启动耗时 / SQL | `:/Documentation` 冷启动耗时 / SQL / commits/s | `:exclude[::t/]` 冷启动耗时 / SQL / commits/s | 根链 / commit_map / object / object_ref 大小 |
|---:|---:|---:|---:|---:|
| 307 | 60082 / 19 | 34300 / 14 / 8950 | 53589 / 17 / 5729 | 139264 / 172032 / 262144 / 147456 |
| 615 | 41728 / 19 | 31397 / 17 / 19588 | 75839 / 17 / 8109 | 221184 / 253952 / 1466368 / 294912 |
| 1231 | 77556 / 27 | 93874 / 31 / 13113 | 217212 / 34 / 5667 | 368640 / 442368 / 5545984 / 704512 |

单提交增量每格为 `p50 / p95 / SQL`，SQL 是该组 50 个样本共同的语句数。视图 0 为 `:/Documentation`，视图 1 为 `:exclude[::t/]`。
| H | 视图 | in_view | outside | t_only |
|---:|---:|---:|---:|---:|
| 307 | 0 | 23443 / 31431 / 23 | 22242 / 43777 / 19 | 21036 / 37593 / 19 |
| 307 | 1 | 24617 / 36936 / 23 | 21687 / 42749 / 19 | 20525 / 38388 / 19 |
| 615 | 0 | 22700 / 38126 / 23 | 21902 / 39527 / 19 | 21402 / 41113 / 19 |
| 615 | 1 | 22974 / 40108 / 23 | 22496 / 44565 / 19 | 20740 / 41269 / 19 |
| 1231 | 0 | 24739 / 39020 / 23 | 26077 / 41710 / 19 | 24760 / 38769 / 19 |
| 1231 | 1 | 24171 / 38809 / 23 | 27341 / 39856 / 19 | 23692 / 42684 / 19 |

判据 (a) PASS（跨档逐样本 SQL 相等）；(b) PASS（最大 p95 比 1.24145 ≤ 1.5）；(c) PASS（最大投影批均耗时比 0.88864 ≤ 1）；(d) PASS（每视图最多 61 ≤ 100 条 SQL）。顶格过滤器上限 Compose 64、Exclude 256、k=64；冷启动 950462 µs / 24 SQL / 0 个脊柱 tree；增量 p50=23893 µs、p95=27295 µs ≤ 1000000 µs，样本均为 25 SQL、1 个脊柱 tree。H=307 时 `Documentation/` 尚未出现，顶格源路径在该历史中也未命中；这些数据只支持冻结夹具下的 P0 判据，不外推至更长历史或真实顶格命中负载。

### 7.3 风险
| 风险 | 说明 | 缓解 |
Expand Down
Loading
Loading