feat(runtime): reuse codec allocations and KVSpace addresses

bench/experiments/20261008-current-python/README.md

@@ -0,0 +1,114 @@
+# Current interpreter versus Python
+
+Nine alternating runs per variant on each repository benchmark's largest frozen
+scale. All 630 samples have identical functional output within each case.
+Results are medians of the fixtures' `__bench_ns` interval in milliseconds;
+process startup, layout loading and store setup are excluded. `samples.csv`
+also records wall time. Each interpreter sample starts with a fresh SHM store.
+
+Environment: Intel Xeon Gold 6418H, CPU 24 affinity, GCC 14.3.0,
+`-O3 -g -DNDEBUG`, CPython 3.10.12, `LOG_LEVEL=warn`.
+
+## Isolated variants
+
+The runtime baseline is kvlang `774e5315eff9ec39caa1cd6e9791620811845a9d`.
+The backend baseline is kvspace-c `0e5660c37d5584fe774b1c75d71eca3777115671`.
+Both interpreters use the same CLI, layout library and kvspace v0.3.0 frontend.
+Blockmalloc is v0.1.4. The allocator baseline is slotsboxmalloc v0.1.5
+(header identical to main `e2c4c6bf67648a7c200d0354ff25e5f65c417a00`).
+
+| Variant | Runtime | SHM write path | Allocator |
+|---|---|---|---|
+| main | baseline | baseline | v0.1.5 |
+| runtime | this patch | baseline | v0.1.5 |
+| backend | baseline | existing-value write patch | v0.1.5 |
+| combined | this patch | existing-value write patch | v0.1.5 |
+| allocator | baseline | baseline | bitmap allocation patch |
+| all | this patch | existing-value write patch | bitmap allocation patch |
+| python | stock case | — | — |
+
+The bitmap patch replaces the reverse slot scan with word operations and rejects
+requests exceeding a leaf's `max_obj_cap`. It retains the v0.1.5 pool format and
+slot selection. The separate, previously submitted slotsboxmalloc PR #9 capacity
+initialization fix is absent from every measured variant. The earlier kvspace-c
+direct-child listing PR #39 is also absent.
+
+## Results
+
+These combined results require all three source patches; they are not the gain
+from the runtime patch alone. Runtime-only medians range from 0.97x to 1.07x
+baseline speed, including a 3.1% slower iops median. The larger improvements
+come from the separately isolated write and allocator patches.
+
+| Case | Scale | Main ms | All patches ms | Speedup | Python ms |
+|---|---:|---:|---:|---:|---:|
+| iops | 2000 | 33.231 | 26.754 | 1.24x | 0.094775 |
+| prime_sieve | 100 | 43.125 | 33.189 | 1.30x | 0.078565 |
+| fib | 10 | 116.614 | 39.435 | 2.96x | 0.019457 |
+| nqueens | 6 | 319.365 | 103.438 | 3.09x | 0.080029 |
+| quicksort | 128 | 50.026 | 38.346 | 1.30x | 0.089687 |
+| binary_search | 64 | 141.625 | 53.834 | 2.63x | 0.048057 |
+| binary_trees | 6 | 27.805 | 16.576 | 1.68x | 0.036597 |
+| hash_table | 200 | 21.035 | 13.794 | 1.52x | 0.090475 |
+| matmul | 8 | 21.447 | 16.185 | 1.33x | 0.092590 |
+| k_nucleotide | 8 | 33.396 | 27.585 | 1.21x | 0.071613 |
+
+The optimized interpreter remains 152–2027x slower than Python at these scales.
+These are SHM results, not FS/Redis performance measurements. The dependency
+manifest still pins slotsboxmalloc v0.1.5: consuming the bitmap gain requires
+building the backend with the patched header, then a dependency release/update.
+
+## Reproduce
+
+Build each backend with the same flags and blockmalloc header. Put the allocator
+include path first for the `allocator` and `all` variants:
+
+```sh
+cmake -S "$BACKEND_SOURCE" -B "$BACKEND_BUILD" \
+  -DCMAKE_BUILD_TYPE=RelWithDebInfo -DKVSPACE_BUILD_TESTS=ON \
+  -DCMAKE_C_FLAGS_RELWITHDEBINFO='-O3 -g -DNDEBUG' \
+  -DCMAKE_C_FLAGS="-I$ALLOCATOR_SOURCE/include -I$ABI_PREFIX/include"
+cmake --build "$BACKEND_BUILD" -j
+
+cmake -S "$KVLANG_SOURCE/runtime" -B "$RUNTIME_BUILD" \
+  -DCMAKE_BUILD_TYPE=RelWithDebInfo \
+  -DCMAKE_C_FLAGS_RELWITHDEBINFO='-O3 -g -DNDEBUG' \
+  -DKVSPACE_LIB_DIR="$ABI_PREFIX/lib/kvspace"
+cmake --build "$RUNTIME_BUILD" -j
+```
+
+The checked snapshot used these paths (their actual loaded libraries and hashes
+are recorded in `results/metadata.json`):
+
+```sh
+python3 bench/experiments/20261008-current-python/run.py \
+  --baseline /tmp/kvlang-perf-current-baseline/bin/kvlang \
+  --candidate /tmp/kvlang-perf-current/bin/kvlang \
+  --baseline-backend /tmp/kvlang-perf-experiment/backend-main \
+  --candidate-backend /tmp/kvspace-c-perf-write-build \
+  --allocator-backend /tmp/kvspace-c-perf-leaf-build \
+  --all-backend /tmp/kvspace-c-perf-total-build \
+  --cpu 24 --pairs 9 --output results
+```
+
+`run.py` reuses the stock sweep and parser, substitutes only `__SCALE__`, rejects
+failed or missing-time samples, checks every functional output against the other
+variants and Python, and records source and loaded-library hashes.
+
+## Validation
+
+- Runtime CTest: 9/9 with the pinned backend and with all patches.
+- SHM backend CTest: 6/6 for write-only, allocator-only and all patches.
+- Candidate tutorials: SHM and FS each 222 passed, 0 failed, 1 skipped.
+- Independent execution probes: external PC/value/status changes on cache hits,
+  key fallback and write failures, worker exit/resume, codec bytes and ownership.
+- All 30 frozen stock scale points pass with both pinned and combined SHM backends.
+- Independent allocator oracle: 15,219,555 bitmap cases, 240,000 fragmented
+  alloc/free operations, cross-address v0.1.5 pool handover, concurrent data
+  integrity and persisted backend reopen checks.
+
+Runtime's normal-length probes and all nine CTests pass ASan/UBSan. A separate
+700-byte-ID probe reproduces an existing frame-key fallback leak on baseline
+and candidate. Backend sanitizer comparison disables inherited unaligned ART
+access checks; its independent write harness passes, while the full suite retains
+the baseline `art_scan` leak. These limits are not full sanitizer-clean claims.

bench/experiments/20261008-current-python/results/medians-ms.json

@@ -0,0 +1,92 @@
+{
+  "iops": {
+    "main": 33.230698,
+    "runtime": 34.249536,
+    "backend": 29.299886,
+    "combined": 27.683865,
+    "allocator": 33.217073,
+    "all": 26.754023,
+    "python": 0.094775
+  },
+  "prime_sieve": {
+    "main": 43.124912,
+    "runtime": 40.354541,
+    "backend": 34.861587,
+    "combined": 34.631411,
+    "allocator": 41.596825,
+    "all": 33.188722,
+    "python": 0.078565
+  },
+  "fib": {
+    "main": 116.614486,
+    "runtime": 117.200918,
+    "backend": 116.315874,
+    "combined": 115.574115,
+    "allocator": 41.433727,
+    "all": 39.435195,
+    "python": 0.019457
+  },
+  "nqueens": {
+    "main": 319.365155,
+    "runtime": 316.669465,
+    "backend": 314.260717,
+    "combined": 313.492908,
+    "allocator": 108.160385,
+    "all": 103.437525,
+    "python": 0.080029
+  },
+  "quicksort": {
+    "main": 50.026257,
+    "runtime": 48.435898,
+    "backend": 42.536906,
+    "combined": 41.607775,
+    "allocator": 47.376373,
+    "all": 38.34567,
+    "python": 0.089687
+  },
+  "binary_search": {
+    "main": 141.625351,
+    "runtime": 140.764034,
+    "backend": 137.777669,
+    "combined": 136.834907,
+    "allocator": 59.21245,
+    "all": 53.83428,
+    "python": 0.048057
+  },
+  "binary_trees": {
+    "main": 27.805124,
+    "runtime": 27.389447,
+    "backend": 24.894097,
+    "combined": 24.337998,
+    "allocator": 20.466159,
+    "all": 16.575519,
+    "python": 0.036597
+  },
+  "hash_table": {
+    "main": 21.035234,
+    "runtime": 20.501664,
+    "backend": 18.208816,
+    "combined": 17.612647,
+    "allocator": 17.221449,
+    "all": 13.793723,
+    "python": 0.090475
+  },
+  "matmul": {
+    "main": 21.446574,
+    "runtime": 20.88364,
+    "backend": 18.069819,
+    "combined": 17.603285,
+    "allocator": 20.361865,
+    "all": 16.184711,
+    "python": 0.09259
+  },
+  "k_nucleotide": {
+    "main": 33.396298,
+    "runtime": 33.060279,
+    "backend": 28.868662,
+    "combined": 28.413805,
+    "allocator": 32.910521,
+    "all": 27.585147,
+    "python": 0.071613
+  }
+}

bench/experiments/20261008-current-python/results/metadata.json

@@ -0,0 +1,144 @@
+{
+  "cpu": 24,
+  "pairs": 9,
+  "python": "3.10.12",
+  "cpu_model": "Intel(R) Xeon(R) Gold 6418H",
+  "variants": {
+    "main": {
+      "binary": "/tmp/kvlang-perf-current-baseline/bin/kvlang",
+      "backend_path": "/tmp/kvlang-perf-experiment/backend-main",
+      "binary_sha256": "931157271cc79507b6a33c0e9c3fc72013ceb65ff5520c046b887955bce49d7a",
+      "libraries": {
+        "/tmp/kvlang-perf-deps/prefix/lib/kvspace/libkvspace.so.1": "b6230a008bfbc2326797c361f589f35b8129f32522bc6fe55a1808fbae42aa73",
+        "/tmp/kvlang-perf-current-baseline/bin/libkvlanglayout.so": "30af52956c24a10bf98ee6b5dc1507003993e79606a40aef43457516012599ef",
+        "/tmp/kvlang-perf-current-baseline/bin/libkvlang_runtime.so": "abea1bd6c985a70b648ce55c04de6fc1c3d7220b40c7e7bf19c5730e6eea3faa",
+        "/tmp/kvlang-perf-experiment/backend-main/libkvspace-c.so.1": "a6a8d894675173fc6c005bd29b45e857d0da16f4a40c50cd669f4e39416e91bf"
+      }
+    },
+    "runtime": {
+      "binary": "/tmp/kvlang-perf-current/bin/kvlang",
+      "backend_path": "/tmp/kvlang-perf-experiment/backend-main",
+      "binary_sha256": "931157271cc79507b6a33c0e9c3fc72013ceb65ff5520c046b887955bce49d7a",
+      "libraries": {
+        "/tmp/kvlang-perf-deps/prefix/lib/kvspace/libkvspace.so.1": "b6230a008bfbc2326797c361f589f35b8129f32522bc6fe55a1808fbae42aa73",
+        "/tmp/kvlang-perf-current/bin/libkvlanglayout.so": "30af52956c24a10bf98ee6b5dc1507003993e79606a40aef43457516012599ef",
+        "/tmp/kvlang-perf-current/bin/libkvlang_runtime.so": "77134092e1713357b88aa04db2d5a4b8470ae0cbdc79e7ae39f909b2869d73c9",
+        "/tmp/kvlang-perf-experiment/backend-main/libkvspace-c.so.1": "a6a8d894675173fc6c005bd29b45e857d0da16f4a40c50cd669f4e39416e91bf"
+      }
+    },
+    "backend": {
+      "binary": "/tmp/kvlang-perf-current-baseline/bin/kvlang",
+      "backend_path": "/tmp/kvspace-c-perf-write-build",
+      "binary_sha256": "931157271cc79507b6a33c0e9c3fc72013ceb65ff5520c046b887955bce49d7a",
+      "libraries": {
+        "/tmp/kvlang-perf-deps/prefix/lib/kvspace/libkvspace.so.1": "b6230a008bfbc2326797c361f589f35b8129f32522bc6fe55a1808fbae42aa73",
+        "/tmp/kvlang-perf-current-baseline/bin/libkvlanglayout.so": "30af52956c24a10bf98ee6b5dc1507003993e79606a40aef43457516012599ef",
+        "/tmp/kvlang-perf-current-baseline/bin/libkvlang_runtime.so": "abea1bd6c985a70b648ce55c04de6fc1c3d7220b40c7e7bf19c5730e6eea3faa",
+        "/tmp/kvspace-c-perf-write-build/libkvspace-c.so.1": "dfa5fd3b3ef9a8393fb7bb142767e592c979276c23a2cf7ce65bc1dee5abf65a"
+      }
+    },
+    "combined": {
+      "binary": "/tmp/kvlang-perf-current/bin/kvlang",
+      "backend_path": "/tmp/kvspace-c-perf-write-build",
+      "binary_sha256": "931157271cc79507b6a33c0e9c3fc72013ceb65ff5520c046b887955bce49d7a",
+      "libraries": {
+        "/tmp/kvlang-perf-deps/prefix/lib/kvspace/libkvspace.so.1": "b6230a008bfbc2326797c361f589f35b8129f32522bc6fe55a1808fbae42aa73",
+        "/tmp/kvlang-perf-current/bin/libkvlanglayout.so": "30af52956c24a10bf98ee6b5dc1507003993e79606a40aef43457516012599ef",
+        "/tmp/kvlang-perf-current/bin/libkvlang_runtime.so": "77134092e1713357b88aa04db2d5a4b8470ae0cbdc79e7ae39f909b2869d73c9",
+        "/tmp/kvspace-c-perf-write-build/libkvspace-c.so.1": "dfa5fd3b3ef9a8393fb7bb142767e592c979276c23a2cf7ce65bc1dee5abf65a"
+      }
+    },
+    "allocator": {
+      "binary": "/tmp/kvlang-perf-current-baseline/bin/kvlang",
+      "backend_path": "/tmp/kvspace-c-perf-leaf-build",
+      "binary_sha256": "931157271cc79507b6a33c0e9c3fc72013ceb65ff5520c046b887955bce49d7a",
+      "libraries": {
+        "/tmp/kvlang-perf-deps/prefix/lib/kvspace/libkvspace.so.1": "b6230a008bfbc2326797c361f589f35b8129f32522bc6fe55a1808fbae42aa73",
+        "/tmp/kvlang-perf-current-baseline/bin/libkvlanglayout.so": "30af52956c24a10bf98ee6b5dc1507003993e79606a40aef43457516012599ef",
+        "/tmp/kvlang-perf-current-baseline/bin/libkvlang_runtime.so": "abea1bd6c985a70b648ce55c04de6fc1c3d7220b40c7e7bf19c5730e6eea3faa",
+        "/tmp/kvspace-c-perf-leaf-build/libkvspace-c.so.1": "9f3b419a89a5021c8a2a01dfc27ba3f6c792e58940b0e79272278d77fa8a0869"
+      }
+    },
+    "all": {
+      "binary": "/tmp/kvlang-perf-current/bin/kvlang",
+      "backend_path": "/tmp/kvspace-c-perf-total-build",
+      "binary_sha256": "931157271cc79507b6a33c0e9c3fc72013ceb65ff5520c046b887955bce49d7a",
+      "libraries": {
+        "/tmp/kvlang-perf-deps/prefix/lib/kvspace/libkvspace.so.1": "b6230a008bfbc2326797c361f589f35b8129f32522bc6fe55a1808fbae42aa73",
+        "/tmp/kvlang-perf-current/bin/libkvlanglayout.so": "30af52956c24a10bf98ee6b5dc1507003993e79606a40aef43457516012599ef",
+        "/tmp/kvlang-perf-current/bin/libkvlang_runtime.so": "77134092e1713357b88aa04db2d5a4b8470ae0cbdc79e7ae39f909b2869d73c9",
+        "/tmp/kvspace-c-perf-total-build/libkvspace-c.so.1": "6b143b678a0d1be7761401809b08c6d9093f9501ec261da3aaa2bcbd96b433c0"
+      }
+    }
+  },
+  "sources": {
+    "iops": {
+      "scale": 2000,
+      "kv_sha256": "360d3a2b7bd4d014255ba34cd86f6c72ee72a46b4787ed248ee5a6db96887dd7",
+      "python_sha256": "e53ef1d628cf1b39be8e2ee2e7ee04a9e0517403226e5f8f18ada9a44ce12d04"
+    },
+    "prime_sieve": {
+      "scale": 100,
+      "kv_sha256": "e0dd66398f3027ccd02ea9615089e57693840c037e4674f492f46c4cd6aae346",
+      "python_sha256": "7365d00a931b3332c68021f14646d651cabbb208c0a2dd6cdd5635bed9475af9"
+    },
+    "fib": {
+      "scale": 10,
+      "kv_sha256": "f9f308e2939ecd712442586ee87b8340f55331e75bcd6a6c6ffc12aea3a7bab4",
+      "python_sha256": "63a329cd3bbc9271f9142f9fa762b3389c17525679763a14e21a7138e12987fa"
+    },
+    "nqueens": {
+      "scale": 6,
+      "kv_sha256": "e4ba6073d1cd9e76949e3b81b46604e28b97e712afd6f6158db811f7e97c169b",
+      "python_sha256": "61ebaeccf79bf06a85e213180930552748cca126ec60b8636a7ed7849765a8af"
+    },
+    "quicksort": {
+      "scale": 128,
+      "kv_sha256": "7e50702ad83e4a9e4e9b9627a92f2a6d22abe3219f830bb17320a6bc400f7f16",
+      "python_sha256": "048c1991bb5e6f528fbacfa8167abe5d860a71117142e736613ced91b8a1a92d"
+    },
+    "binary_search": {
+      "scale": 64,
+      "kv_sha256": "42e05e05ccbbdb4af0e5b175ad4f87a367d00a97ddcd4ae0ff4d25edfc7ac153",
+      "python_sha256": "64e43b7a2a8b0fd928bb93cc8b57cd3f5fd45daf97f46ea269090a395ac6e6dc"
+    },
+    "binary_trees": {
+      "scale": 6,
+      "kv_sha256": "39add5228fab5953224b0959b0b8b176586a866179608909390108189f6b46d8",
+      "python_sha256": "454f14208f826ef084a593953b6780ab29f94f5d565e8b55d5a322f037323b13"
+    },
+    "hash_table": {
+      "scale": 200,
+      "kv_sha256": "606feac126238d1908a787ba46f1a7a8d4fd2858cf6a69e327c2acc2f6605541",
+      "python_sha256": "b19fbe72320375471e91ad31093b37b385f8ef5be46a29465381bac05391ced0"
+    },
+    "matmul": {
+      "scale": 8,
+      "kv_sha256": "ba5437bb1dc4f702073fc8c9c8519b28ee502cb27b51d1e5ba09b69262d1ac89",
+      "python_sha256": "103b272337bff44df02001a52dea61d6c9db8e843245dbc869afe673203764f8"
+    },
+    "k_nucleotide": {
+      "scale": 8,
+      "kv_sha256": "9d4cc69ee5455b0902f47b7b6a2e60bbd6d3432084b6e15c252ecbbfbcd5a694",
+      "python_sha256": "c35d8d9fe4cb2be7065befed69a1c763a7d3ab6e5c569857249b21bbf8a8a6ec"
+    }
+  },
+  "build": {
+    "compiler": "GCC 14.3.0",
+    "c_flags": "-O3 -g -DNDEBUG",
+    "runtime_base": "774e5315eff9ec39caa1cd6e9791620811845a9d",
+    "backend_base": "0e5660c37d5584fe774b1c75d71eca3777115671",
+    "allocator_base": "e2c4c6bf67648a7c200d0354ff25e5f65c417a00",
+    "allocator_baseline_version": "v0.1.5",
+    "blockmalloc": "v0.1.4",
+    "frontend": "kvspace v0.3.0 release",
+    "candidate_source_sha256": {
+      "runtime/src/xvalue.c": "d2f24c94c99b1a317657ac43c619e4aad5b0e39741cd05b97f331e4ad7b14ff1",
+      "runtime/src/kvcpu.c": "2fb04432b3e393ef0de94188a0fcf829dc9c4c1c4ae4802a4bbb7a31094b8222",
+      "runtime/src/vthread.c": "e39840c88d879db821815430e01ccaac9d74788f146419169e5827808b3a3dc2",
+      "runtime/src/runtime_internal.h": "90095626f698d221e1cf6b2ec9e771417d1e4a28caf4814965c8473017bc1ae0"
+    },
+    "backend_candidate_source_sha256": "6c6d6b93a5c5856616460280d0d8f63dad9539e68d5387ad769aaf2a758e73ab",
+    "allocator_candidate_header_sha256": "d424a34e0ef92d13d063d6631df7174bb37d6cfd129e3423b97241d0e3f7b14b"
+  }
+}