Repository navigation
Fix/2955 cache path diagnostic - #2965
Bowen-Xue-ac2 wants to merge 13 commits into
Conversation
|
Thanks for opening this — it has been seen, and it is queued. This note is automated, but it is not a brush-off: it exists so you know where your PR stands instead of having to guess from silence. Current review status: working through a backlog. What that means for this PR, concretely:
Things that will genuinely speed it up whenever review does happen:
If this fixes a bug, a reproduction we can run is worth more than a description of the symptom. Thanks for contributing, and sorry in advance for the wait. |
e9a6661 to
f27c820
Compare
Add an authenticated status-details probe that returns the daemon's own resolved cache directory while preserving the legacy status response. Report escaped paths after verifying the conflict identity, and explain when diagnostics fall back to fingerprints. Cover long paths, Unicode, control characters and unavailable status details. Fixes DeusData#2955 Signed-off-by: BowenX <318502935+Bowen-Xue-ac2@users.noreply.github.com>
Honor an explicit CBM_RUNTIME_DIR before creating the default runtime so Windows long-path cases reach the binary without extra setup failures. Match UTF-8 cache paths without JSON ASCII escaping in the smoke check. Signed-off-by: BowenX <318502935+Bowen-Xue-ac2@users.noreply.github.com>
f27c820 to
034df5e
Compare
DeusData
left a comment
There was a problem hiding this comment.
Hi @Bowen-Xue-ac2, thank you for this, and welcome! This is a really careful first contribution. You took the reporter's plan and both of their follow-up suggestions on board, kept the existing 121-byte STATUS reply untouched so older clients keep working, re-check both paths against the recorded fingerprints before naming them (so a daemon that turned over in between can never be mis-attributed), escape the paths, and keep the durable conflict log fingerprint-only. Folding the two duplicated cohort-failure blocks in main.c into one helper is the right call as well: it guarantees the CLI and the MCP entry point report the conflict the same way. We are happy with the new STATUS_DETAILS operation as you designed it.
We built your head and ran it against a real conflict on macOS (daemon on one cache root, client on another):
- before: "... uses a different cache directory (active cache 9277...; requested cache 704c...)"
- after, for both
cli list_projectsand the MCP entry point:
requested cache: "/private/tmp/.../cache B "
daemon cache: "/private/tmp/.../cacheA"
Wall time is unchanged, the fallback ("Cache paths unavailable or unverified.") shows up when the daemon does not answer, and quotes, newlines and control characters come out escaped as intended. The new daemon_runtime tests pass under ASan/UBSan here too.
Four things before we can merge:
-
The red
test / test-windows-guardscheck (which makesci-okred). As the reporter already worked out in #2955, at the 259-character cache root CBM stops at "could not load saved runtime settings" before the conflict check runs. That message comes fromsrc/main.cwhencbm_runtime_settings_init()(src/cli/runtime_settings.c) fails for the resolved cache directory, most likely because the settings file path under such a root no longer fits Windows' 260-character MAX_PATH. It is an existing limitation, not something your change introduced, but since this PR is about making long and unusual cache roots work in the conflict message, we would like it fixed here: please make the settings load work for long cache roots on Windows and keep your 259/260/261/1024 lengths incheck_cache_conflict_paths, so the Windows leg proves it. If you would rather not dig into the Windows path handling, just say so: maintainer edits are enabled, and we can push that part to your branch ourselves and explain it here. -
Please leave
runtime_control_request_sendas it is on main, and apply the single time budget only insidecbm_daemon_runtime_request_status_details(the legacy fallback gets whatever the first attempt left over). Why: the shared deadline also changes the existing STATUS and STOP calls (daemon status,daemon stop, the start confirmation, the CLI guards). On a healthy daemon it makes no measurable difference, but when connect has to wait (on Windows that happens when all pipe instances are busy), the reply then only gets the remainder of the budget, anddaemon statuswould describe a busy but healthy daemon as an unresponsive endpoint holder. That behaviour change is outside #2955, so it is better left out of this PR. -
Please drop the end-to-end fallback guard in
tests/windows/test_daemon_lifecycle.pythat polls for the cohort owner's lock record (the 20 ms loop with the 15 s deadline). This project does not let tests poll or sleep while waiting for state, and a timeout must never be what decides a test; your C test of the IPC fallback (unsupported and malformed replies) already covers that path deterministically on every platform, including Windows. -
In the same file, the
if os.name != "nt"branch with the control-character path never runs: that file is only executed on the Windows leg (via scripts/test-windows.ps1). Please drop it so the test does not suggest coverage it does not provide. We checked the escaping by hand on macOS instead: a cache root containing a quote, a newline and an ESC byte comes out as cache"q...\nnl\u001besc, exactly as intended.
Two small FYIs, no action needed: tests/test_daemon_smoke.py is not part of the CI matrix (it runs via make test-daemon-smoke), so your new assertion there is not gated, but it matches what we saw locally. And test-impact-shadow is red for an unrelated reason (its indexer ran over its memory budget); it is not a required check.
Once these are in, the checks should go green and we can take it from there. Thanks again for the thorough work, and for working so closely with the reporter on the design!
|
Roger 👍 I will solve these later |
…eep/wall-clock faces across tests/ (evidence: test_run_tree_timeout_kills_whole_windows_tree flaked once in each of the last two full quick runs under machine load, no traceback retained; discipline reference: PR DeusData/codebase-memory-mcp#2965 review - a timeout must never be what decides a test) - census work/probe_rw_census.py four faces: A deadline-decided poll loop (test_procs._wait_dead, 1 site), B wall-clock upper-bound asserts (test_s0_timeout <4.0 margin 3s RISKY; test_s7_enforce/test_s7_enforce_rs <10.0 vs ms-scale ReDoS-guard ops = 1000x margin SAFE keep), C injected sleep= dependencies (deterministic, no real clock, fine), D real time.sleep call sites (deep/conftest rmtree backoff: bounded 6 attempts, deadline does not decide -> allowlist; test_procs poll-loop sleep -> removed), E victim-script sleeps in string literals (correct fixture usage, margins 30x); side-finding: UTF-8 BOM in tests/deep/test_deep_cargo_mutants.py stripped (PS 5.1 Set-Content default artifact); synthetic-load repro attempt work/probe_rw_stress.py: pure-CPU 10 busy loops x 12 iterations 12/12 PASS on old code (real flake needs disk/AV/memory pressure stacking - honest record: mechanisms pinned by analysis, repro not achieved) - fixes: test_procs _pid_alive switched from tasklist /FI spawn-per-probe (each spawn >1s under load eats deadline budget) to ctypes OpenProcess+GetExitCodeProcess probe (zero spawn, us-scale, load-immune; old naive substring match also false-positived on pid 0 - caught by red-face), _wait_dead rewritten as sleep-free spin (by-construction: run_tree timeout branch returns only after pipe EOF => all write-handle holders dead => grandchild already dead, R-L orphan-holds-pipe evidence; spin only tolerates kernel handle-teardown lag, 2s cap; green path first-check-returns, red path loud bounded failure), parent.py victim gains Popen retry loop (transient spawn failure under memory pressure used to crash parent early -> timed_out=False false red; parent can now only be killed or run to completion), run_tree timeout 2s->6s (loaded interpreter boot could exceed 2s killing parent before gc.pid write -> FileNotFoundError false red; tree-kill semantics are timeout-precision-independent, precision pinned separately by test_s0_timeout); test_s0_timeout wall bound 4.0->10.0 with victim sleep 5->30 (no-kill regression still red: 30s > 10s; load margin 3s->9s, both axes improved); guards +4 tests/test_timing_discipline.py: test_no_real_sleep_call_sites_in_tests (AST scan of tests/, allowlist deep/conftest.py only) + test_no_bom_in_test_sources (BOM recurrence guard) + test_pid_alive_ctypes_probe (alive/invalid-pid/exited behavior pin) + test_wait_dead_red_path_bounded (red path bounded honest False); red-face self-proof: git stash the 3 modified files -> 3/4 guards precisely red (sleep site + BOM + tasklist pid-0 false-positive) + red-path pin green on old code (both impls return False on red path, regression-guard type) -> pop -> 13/13 green; stress rerun with fixed code 12/12 pass under load (~6.4s/iter, 6s timeout dominated, ctypes first-check-returns confirming EOF-by-construction); quick full 1270 passed (1266 + 4 new guards) 366.70s zero red zero flake
…- deadline/sleep/wall-clock faces; evidence: test_run_tree_timeout_kills_whole_windows_tree flaked once in each of the last two full quick runs) - commit b76b66c recorded (4 files: test_procs ctypes liveness probe + sleep-free teardown-lag spin + Popen-retry victim + timeout 2s->6s, test_s0_timeout wall bound 4.0->10.0 with victim 5->30, new tests/test_timing_discipline.py +4 guards (no-real-sleep AST scan with deep/conftest.py sole allowlist, no-BOM, ctypes probe behavior pin, red-path bounded pin), BOM strip in test_deep_cargo_mutants), census work/probe_rw_census.py four faces (A deadline-decided poll 1 site / B wall bounds 3 sites with s7 ReDoS guards 1000x-margin safe / C injected sleeps fine / D real sleep sites: rmtree backoff allowlisted, poll sleep removed / E victim literal sleeps 30x margins), discipline reference PR DeusData/codebase-memory-mcp#2965 review (a timeout must never decide a test), red-face 3/4 precisely red on stashed old code (sleep site + BOM + tasklist pid-0 substring false-positive) + red-path pin green on both impls (regression-guard type), honest repro record: pure-CPU 10-busy-loop x 12-iteration stress 12/12 pass on OLD code (real flake needs disk/AV/memory pressure stacking; mechanisms pinned by analysis), post-fix stress 12/12 pass under same load (~6.4s/iter), quick full 1270 passed 366.70s zero red zero flake
Signed-off-by: BowenX <318502935+Bowen-Xue-ac2@users.noreply.github.com>
Signed-off-by: BowenX <318502935+Bowen-Xue-ac2@users.noreply.github.com>
Rename the internal status-details transport callback to ipc_connect so static network auditing does not mistake it for a raw connect syscall. Keep the production IPC implementation, test seam, and network policy unchanged. Signed-off-by: BowenX <318502935+Bowen-Xue-ac2@users.noreply.github.com>
Measure the same root and descendant before and after the controlled 128 MiB allocation. The descendant publishes an idle marker, waits for the parent to measure it, then touches the allocation and publishes the loaded marker. Assert that both markers identify the same descendant. Keep the 96 MiB growth floor, member-fault coverage, and terminal-empty checks. Do not change production RSS measurement or weaken CI checks. Signed-off-by: BowenX <318502935+Bowen-Xue-ac2@users.noreply.github.com>
|
The fixes requested in the earlier review have been implemented, including Windows long-path handling, isolating the STATUS_DETAILS timeout budget, and replacing timing-dependent test coverage with deterministic tests. The Windows test suite now passes. I have also prepared a local fix for the separate --skip-binary issue. The targeted regression test fails against the old code and passes with the fix, but the patch has not yet been pushed, and platform-specific validation is still pending. On the latest Ubuntu CI run, two daemon runtime tests failed and the CLI suite timed out. I’m investigating the daemon failures and working to validate them in an Ubuntu-capable environment. The root cause is not yet confirmed, working :) |
Keep config-only installs from re-enabling binary replacement, and make macOS signing respect the same skip-binary policy. Cover existing and absent targets, content and metadata preservation, agent configuration updates, and approved replacement controls. Add a real-IPC existing-target no-drain case alongside the absent-target case. Signed-off-by: BowenX <318502935+Bowen-Xue-ac2@users.noreply.github.com>
|
Thank you for this round, Bowen, and for the clear status update! One thing to save you time: the two On We are reviewing the rest of this round now. |
DeusData
left a comment
There was a problem hiding this comment.
Hi @Bowen-Xue-ac2, thank you for this round, and for turning it around so quickly! You took every point from the last review on board, and in a couple of places you went further than we asked. In particular, moving the single time budget into a STATUS_DETAILS-only transport and testing it against a scripted clock (daemon_runtime_status_details_uses_one_budget_across_legacy_fallback and ..._stops_when_budget_expires) is exactly the kind of test we want: no scheduler delay can consume the budget or decide the result. Thank you also for fixing the Windows long cache root yourself rather than leaving it to us.
We built 18cfd55 with ASan/UBSan on macOS and ran the affected suites (daemon_runtime, daemon_frontend, daemon_bootstrap, cli, subprocess, runtime_settings): 592 passed and 3 were skipped as Windows-only. The one failure is specific to our test machine, also fails on main there, and has nothing to do with your change. We also repeated last round's real conflict on macOS: a live session on one cache root, then cli list_projects and the MCP entry point on another. Both still name the requested cache and the daemon cache, both for a root with a space and non-ASCII characters and for one with a quote, a newline and an ESC byte (printed as cache"q\nnl\u001besc). The active session was undisturbed, and the conflict log still holds only the two fingerprints.
Where the four asks stand:
-
Windows long cache roots: done. The settings database now opens through SQLite's long-path VFS (
cbm_runtime_settings_open_db), the private-directory walk insrc/daemon/ipc.cadds the\\?\prefix from 240 characters on while still rejecting.and..components itself, andcbm_config_openno longer truncates silently. Your 259/260/261/1024 lengths stayed incheck_cache_conflict_paths, andtest-windows-guardsis green on your ce65401 run, so the Windows leg now proves it. One optional suggestion, not a blocker:cbm_path_for_file_api()insrc/foundation/compat.calready produces the prefixed UTF-8 spelling into a caller buffer, socbm_runtime_settings_open_dbcould call it and pass the result tosqlite3_open_v2(..., "win32-longpath")instead of repeating the wide round trip. Same behaviour, one less copy of the conversion to keep in sync. -
runtime_control_request_send: done. It is byte-for-byte what main has, sodaemon status,daemon stop, the start confirmation and the CLI guards keep their own budgets, and onlycbm_daemon_runtime_request_status_detailsshares one deadline across the STATUS_DETAILS attempt and the legacy retry. -
The polling fallback guard in
tests/windows/test_daemon_lifecycle.py: done, thank you. One leftover:import time(line 28) was only used by that loop and is now unused, so please drop it too. -
The non-Windows control-character branch: done.
Two commits in this round are outside #2955, and we would like this PR to stay about the cache-path diagnostic only. Keeping each PR to the one change it names makes it easier to review, to revert if something goes wrong, and to find later with git bisect.
-
ce65401 (
test(subprocess): measure RSS growth within one process tree). We can see why you wrote it: the Windows leg on your 889170c run failedsubprocess_tree_rss_measures_contained_descendantswith idle 167 MiB against loaded 245 MiB, below the 96 MiB floor. That failure does not come from your change, because nothing in this PR touches the process-tree probe. The test compares two independently spawned process trees, and on Windows their baselines can differ. It is a real problem in that test, and thank you for digging into it, but it belongs in its own PR so we can review it on its own. If you open one, we would like to look at the handshake there: the descendant now polls for the load marker every 10 ms, and this project avoids adding new poll loops to tests. A blocking handshake, for example the descendant blocking on a pipe read until the parent has taken its idle sample, would avoid the loop. Please take ce65401 out of this branch (a revert commit is fine). If the Windows leg trips over that test again afterwards, we will treat it as our problem, not yours. -
af0205d (
fix(cli): honor skip-binary for existing targets).
You found a real bug here: under --yes, the "a different binary already exists" prompt turned the copy back on even though --skip-binary promised to leave the target alone, and on macOS an existing target was re-signed and republished anyway. You published your fix before we opened #2990 for the same bug, so we would like yours to land under your name, and we will close #2990 in its favour. Because it is separate from #2955, please move af0205d into its own PR: a branch from main with that commit cherry-picked onto it is enough. Your tests already cover more than ours did (the --force combination, the absent target, config and hooks still written, native file identity on Windows, the real-IPC no-drain check and the --yes replacement control). Two small additions would make it complete. First, a case where install infers skip-binary because the running binary is managed by mise, Homebrew or nix (#1566/#2722). Your production change already fixes that path, and the test just pins it. Second, a check that nothing is left staged next to the target (the bin directory holds exactly one entry afterwards). cli_install_managed_binary_keeps_a_different_existing_binary_issue2722 in #2990 does both, and you are welcome to copy it as it is. If you would rather we add it, just say so and we will push it to your new branch.
Once ce65401 and af0205d are out of this branch and the stray import is gone, the remaining red on the Ubuntu x86 GCC leg is the runner issue we are fixing in #2987, not your code, and we can take it from there. Thanks again for the careful work and the quick turnaround!
|
A quick follow-up on the RSS test, Bowen: since that Windows test was tripping other PRs as well, we went ahead and rebuilt it ourselves in #2992, so there is no need for you to open a separate PR for it. Your diagnosis was the key: the two independently spawned process trees are exactly why it flaked, and #2992 credits you for it. It now samples one tree before and after loading, with a blocking pipe handshake instead of the marker poll. For this PR that leaves only dropping ce65401 (a revert commit is fine), the unused |
Fixes #2955
What does this PR do?
Problem
When a client requests a different cache directory from the active daemon, the error shows SHA-256 fingerprints instead of the actual paths. This makes it difficult to identify which directories are in conflict.
How this fixes it
Cache paths unavailable or unverified.Tests and follow-up fixes
ensure_ascii=False, so their expectations match UTF-8 diagnostic output.AI disclosure
OpenAI Codex assisted with the fix.
Checklist
git commit -s) — required, CI rejectsunsigned commits (DCO, see CONTRIBUTING.md)
make -f Makefile.cbm test)make -f Makefile.cbm lint-ci)