Skip to content

Fix/2955 cache path diagnostic - #2965

Open
Bowen-Xue-ac2 wants to merge 13 commits into
DeusData:mainfrom
Bowen-Xue-ac2:fix/2955-cache-path-diagnostic
Open

Bowen-Xue-ac2 wants to merge 13 commits into
DeusData:mainfrom
Bowen-Xue-ac2:fix/2955-cache-path-diagnostic

Conversation

@Bowen-Xue-ac2

@Bowen-Xue-ac2 Bowen-Xue-ac2 commented Nov 12, 2026 •

Copy link
Copy Markdown

Fixes #2955

What does this PR do?

Problem

When a client requests a different cache directory from the active daemon, the error shows SHA-256 fingerprints instead of the actual paths. This makes it difficult to identify which directories are in conflict.

How this fixes it

  • Adds an authenticated status-details request that obtains the daemon’s own resolved cache directory. The existing status response remains unchanged for compatibility with older daemons.
  • Verifies the daemon’s build and cache fingerprint against the recorded conflict before displaying the requested and daemon paths. Special characters are escaped in the diagnostic.
  • Keeps the fingerprint-based message when paths are unavailable or cannot be verified, and adds the hint: Cache paths unavailable or unverified.
  • Keeps persistent conflict logs fingerprint-only.

Tests and follow-up fixes

  • Adds coverage for Unicode, spaces, long paths around the Windows 260-character boundary, longer paths, escaped characters, and fingerprint-only fallback in both CLI and MCP.
  • Updates the Windows lifecycle helper to honor a supplied runtime directory before creating a default one, avoiding an unnecessary unprefixed directory creation for long-path cases.
  • Updates both smoke-test path assertions to use ensure_ascii=False, so their expectations match UTF-8 diagnostic output.

AI disclosure

OpenAI Codex assisted with the fix.

Checklist

  • Every commit is signed off (git commit -s) — required, CI rejects
    unsigned commits (DCO, see CONTRIBUTING.md)
  • Tests pass locally (make -f Makefile.cbm test)
  • Lint passes (make -f Makefile.cbm lint-ci)
  • New behavior is covered by a test (reproduce-first for bug fixes)

@github-actions

Copy link
Copy Markdown

Thanks for opening this — it has been seen, and it is queued.

This note is automated, but it is not a brush-off: it exists so you know where your PR stands instead of having to guess from silence.

Current review status: working through a backlog. 0.9.1-rc.1 is out, so the release freeze that held reviews is over — but it left a large queue of open pull requests behind it, and we are reading through them oldest-first. The background is in discussion #1144.

What that means for this PR, concretely:

  • It will not be closed for inactivity. No stale bot touches pull requests here.
  • It may still sit a while before a human reads it. That is on us, not on you.
  • Older PRs are read first, so a recent one is not being skipped — it is behind a queue.

Things that will genuinely speed it up whenever review does happen:

  • Keep it rebased on main — the tree is moving quickly right now, and a conflicting branch cannot be reviewed as the diff you intended.
  • Get CI green, or say which failures you believe are pre-existing.
  • Keep the change to one claim. Bundled features and refactors get split before they get merged, which costs you a round trip.
  • Every commit needs a sign-off (git commit -s) — CI enforces DCO.

If this fixes a bug, a reproduction we can run is worth more than a description of the symptom.

Thanks for contributing, and sorry in advance for the wait.

@Bowen-Xue-ac2
Bowen-Xue-ac2 force-pushed the fix/2955-cache-path-diagnostic branch from e9a6661 to f27c820 Compare November 12, 2026 12:41
Add an authenticated status-details probe that returns the daemon's own
resolved cache directory while preserving the legacy status response.
Report escaped paths after verifying the conflict identity, and explain
when diagnostics fall back to fingerprints.

Cover long paths, Unicode, control characters and unavailable status details.

Fixes DeusData#2955

Signed-off-by: BowenX <318502935+Bowen-Xue-ac2@users.noreply.github.com>
Honor an explicit CBM_RUNTIME_DIR before creating the default runtime
so Windows long-path cases reach the binary without extra setup failures.
Match UTF-8 cache paths without JSON ASCII escaping in the smoke check.

Signed-off-by: BowenX <318502935+Bowen-Xue-ac2@users.noreply.github.com>
@Bowen-Xue-ac2
Bowen-Xue-ac2 force-pushed the fix/2955-cache-path-diagnostic branch from f27c820 to 034df5e Compare November 12, 2026 14:28
@Bowen-Xue-ac2
Bowen-Xue-ac2 marked this pull request as ready for review November 12, 2026 14:51

@DeusData DeusData left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @Bowen-Xue-ac2, thank you for this, and welcome! This is a really careful first contribution. You took the reporter's plan and both of their follow-up suggestions on board, kept the existing 121-byte STATUS reply untouched so older clients keep working, re-check both paths against the recorded fingerprints before naming them (so a daemon that turned over in between can never be mis-attributed), escape the paths, and keep the durable conflict log fingerprint-only. Folding the two duplicated cohort-failure blocks in main.c into one helper is the right call as well: it guarantees the CLI and the MCP entry point report the conflict the same way. We are happy with the new STATUS_DETAILS operation as you designed it.

We built your head and ran it against a real conflict on macOS (daemon on one cache root, client on another):

  • before: "... uses a different cache directory (active cache 9277...; requested cache 704c...)"
  • after, for both cli list_projects and the MCP entry point:
    requested cache: "/private/tmp/.../cache B "
    daemon cache: "/private/tmp/.../cacheA"

Wall time is unchanged, the fallback ("Cache paths unavailable or unverified.") shows up when the daemon does not answer, and quotes, newlines and control characters come out escaped as intended. The new daemon_runtime tests pass under ASan/UBSan here too.

Four things before we can merge:

  1. The red test / test-windows-guards check (which makes ci-ok red). As the reporter already worked out in #2955, at the 259-character cache root CBM stops at "could not load saved runtime settings" before the conflict check runs. That message comes from src/main.c when cbm_runtime_settings_init() (src/cli/runtime_settings.c) fails for the resolved cache directory, most likely because the settings file path under such a root no longer fits Windows' 260-character MAX_PATH. It is an existing limitation, not something your change introduced, but since this PR is about making long and unusual cache roots work in the conflict message, we would like it fixed here: please make the settings load work for long cache roots on Windows and keep your 259/260/261/1024 lengths in check_cache_conflict_paths, so the Windows leg proves it. If you would rather not dig into the Windows path handling, just say so: maintainer edits are enabled, and we can push that part to your branch ourselves and explain it here.

  2. Please leave runtime_control_request_send as it is on main, and apply the single time budget only inside cbm_daemon_runtime_request_status_details (the legacy fallback gets whatever the first attempt left over). Why: the shared deadline also changes the existing STATUS and STOP calls (daemon status, daemon stop, the start confirmation, the CLI guards). On a healthy daemon it makes no measurable difference, but when connect has to wait (on Windows that happens when all pipe instances are busy), the reply then only gets the remainder of the budget, and daemon status would describe a busy but healthy daemon as an unresponsive endpoint holder. That behaviour change is outside #2955, so it is better left out of this PR.

  3. Please drop the end-to-end fallback guard in tests/windows/test_daemon_lifecycle.py that polls for the cohort owner's lock record (the 20 ms loop with the 15 s deadline). This project does not let tests poll or sleep while waiting for state, and a timeout must never be what decides a test; your C test of the IPC fallback (unsupported and malformed replies) already covers that path deterministically on every platform, including Windows.

  4. In the same file, the if os.name != "nt" branch with the control-character path never runs: that file is only executed on the Windows leg (via scripts/test-windows.ps1). Please drop it so the test does not suggest coverage it does not provide. We checked the escaping by hand on macOS instead: a cache root containing a quote, a newline and an ESC byte comes out as cache"q...\nnl\u001besc, exactly as intended.

Two small FYIs, no action needed: tests/test_daemon_smoke.py is not part of the CI matrix (it runs via make test-daemon-smoke), so your new assertion there is not gated, but it matches what we saw locally. And test-impact-shadow is red for an unrelated reason (its indexer ran over its memory budget); it is not a required check.

Once these are in, the checks should go green and we can take it from there. Thanks again for the thorough work, and for working so closely with the reporter on the design!

@Bowen-Xue-ac2

Copy link
Copy Markdown
Author

Roger 👍 I will solve these later

AH23333 added a commit to AH23333/VRP that referenced this pull request Nov 13, 2026
…eep/wall-clock faces across tests/ (evidence: test_run_tree_timeout_kills_whole_windows_tree flaked once in each of the last two full quick runs under machine load, no traceback retained; discipline reference: PR DeusData/codebase-memory-mcp#2965 review - a timeout must never be what decides a test) - census work/probe_rw_census.py four faces: A deadline-decided poll loop (test_procs._wait_dead, 1 site), B wall-clock upper-bound asserts (test_s0_timeout <4.0 margin 3s RISKY; test_s7_enforce/test_s7_enforce_rs <10.0 vs ms-scale ReDoS-guard ops = 1000x margin SAFE keep), C injected sleep= dependencies (deterministic, no real clock, fine), D real time.sleep call sites (deep/conftest rmtree backoff: bounded 6 attempts, deadline does not decide -> allowlist; test_procs poll-loop sleep -> removed), E victim-script sleeps in string literals (correct fixture usage, margins 30x); side-finding: UTF-8 BOM in tests/deep/test_deep_cargo_mutants.py stripped (PS 5.1 Set-Content default artifact); synthetic-load repro attempt work/probe_rw_stress.py: pure-CPU 10 busy loops x 12 iterations 12/12 PASS on old code (real flake needs disk/AV/memory pressure stacking - honest record: mechanisms pinned by analysis, repro not achieved) - fixes: test_procs _pid_alive switched from tasklist /FI spawn-per-probe (each spawn >1s under load eats deadline budget) to ctypes OpenProcess+GetExitCodeProcess probe (zero spawn, us-scale, load-immune; old naive substring match also false-positived on pid 0 - caught by red-face), _wait_dead rewritten as sleep-free spin (by-construction: run_tree timeout branch returns only after pipe EOF => all write-handle holders dead => grandchild already dead, R-L orphan-holds-pipe evidence; spin only tolerates kernel handle-teardown lag, 2s cap; green path first-check-returns, red path loud bounded failure), parent.py victim gains Popen retry loop (transient spawn failure under memory pressure used to crash parent early -> timed_out=False false red; parent can now only be killed or run to completion), run_tree timeout 2s->6s (loaded interpreter boot could exceed 2s killing parent before gc.pid write -> FileNotFoundError false red; tree-kill semantics are timeout-precision-independent, precision pinned separately by test_s0_timeout); test_s0_timeout wall bound 4.0->10.0 with victim sleep 5->30 (no-kill regression still red: 30s > 10s; load margin 3s->9s, both axes improved); guards +4 tests/test_timing_discipline.py: test_no_real_sleep_call_sites_in_tests (AST scan of tests/, allowlist deep/conftest.py only) + test_no_bom_in_test_sources (BOM recurrence guard) + test_pid_alive_ctypes_probe (alive/invalid-pid/exited behavior pin) + test_wait_dead_red_path_bounded (red path bounded honest False); red-face self-proof: git stash the 3 modified files -> 3/4 guards precisely red (sleep site + BOM + tasklist pid-0 false-positive) + red-path pin green on old code (both impls return False on red path, regression-guard type) -> pop -> 13/13 green; stress rerun with fixed code 12/12 pass under load (~6.4s/iter, 6s timeout dominated, ctypes first-check-returns confirming EOF-by-construction); quick full 1270 passed (1266 + 4 new guards) 366.70s zero red zero flake
AH23333 added a commit to AH23333/VRP that referenced this pull request Nov 13, 2026
…- deadline/sleep/wall-clock faces; evidence: test_run_tree_timeout_kills_whole_windows_tree flaked once in each of the last two full quick runs) - commit b76b66c recorded (4 files: test_procs ctypes liveness probe + sleep-free teardown-lag spin + Popen-retry victim + timeout 2s->6s, test_s0_timeout wall bound 4.0->10.0 with victim 5->30, new tests/test_timing_discipline.py +4 guards (no-real-sleep AST scan with deep/conftest.py sole allowlist, no-BOM, ctypes probe behavior pin, red-path bounded pin), BOM strip in test_deep_cargo_mutants), census work/probe_rw_census.py four faces (A deadline-decided poll 1 site / B wall bounds 3 sites with s7 ReDoS guards 1000x-margin safe / C injected sleeps fine / D real sleep sites: rmtree backoff allowlisted, poll sleep removed / E victim literal sleeps 30x margins), discipline reference PR DeusData/codebase-memory-mcp#2965 review (a timeout must never decide a test), red-face 3/4 precisely red on stashed old code (sleep site + BOM + tasklist pid-0 substring false-positive) + red-path pin green on both impls (regression-guard type), honest repro record: pure-CPU 10-busy-loop x 12-iteration stress 12/12 pass on OLD code (real flake needs disk/AV/memory pressure stacking; mechanisms pinned by analysis), post-fix stress 12/12 pass under same load (~6.4s/iter), quick full 1270 passed 366.70s zero red zero flake
Signed-off-by: BowenX <318502935+Bowen-Xue-ac2@users.noreply.github.com>
Signed-off-by: BowenX <318502935+Bowen-Xue-ac2@users.noreply.github.com>
Rename the internal status-details transport callback to ipc_connect so
static network auditing does not mistake it for a raw connect syscall.
Keep the production IPC implementation, test seam, and network policy
unchanged.

Signed-off-by: BowenX <318502935+Bowen-Xue-ac2@users.noreply.github.com>
Measure the same root and descendant before and after the controlled
128 MiB allocation. The descendant publishes an idle marker, waits for
the parent to measure it, then touches the allocation and publishes the
loaded marker. Assert that both markers identify the same descendant.

Keep the 96 MiB growth floor, member-fault coverage, and terminal-empty
checks. Do not change production RSS measurement or weaken CI checks.

Signed-off-by: BowenX <318502935+Bowen-Xue-ac2@users.noreply.github.com>
@Bowen-Xue-ac2

Bowen-Xue-ac2 commented Nov 14, 2026 •

Copy link
Copy Markdown
Author

The fixes requested in the earlier review have been implemented, including Windows long-path handling, isolating the STATUS_DETAILS timeout budget, and replacing timing-dependent test coverage with deterministic tests. The Windows test suite now passes.

I have also prepared a local fix for the separate --skip-binary issue. The targeted regression test fails against the old code and passes with the fix, but the patch has not yet been pushed, and platform-specific validation is still pending.

On the latest Ubuntu CI run, two daemon runtime tests failed and the CLI suite timed out. I’m investigating the daemon failures and working to validate them in an Ubuntu-capable environment. The root cause is not yet confirmed, working :)

Keep config-only installs from re-enabling binary replacement, and make
macOS signing respect the same skip-binary policy.

Cover existing and absent targets, content and metadata preservation,
agent configuration updates, and approved replacement controls. Add a
real-IPC existing-target no-drain case alongside the absent-target case.

Signed-off-by: BowenX <318502935+Bowen-Xue-ac2@users.noreply.github.com>
@DeusData

Copy link
Copy Markdown
Owner

Thank you for this round, Bowen, and for the clear status update!

One thing to save you time: the two daemon_runtime failures and the cli timeout on the Ubuntu x86 GCC leg do not come from your change. On hosted x86 runners without SHA hardware instructions, the sanitizer build hashes the ~785 MB test binary slowly enough to blow a 60 s budget in those copied-image tests (and the cli suite does dozens of those hashes). We are fixing that on our side in #2987, so there is no need to chase it.

On --skip-binary: you found the same bug we did (the "different binary" prompt turns the copy back on under --yes). Since it is separate from #2955, we opened #2990 for it before we saw your commit. Thanks for catching it too; we will compare both fixes while reviewing your new commits and come back to you on which one lands and how we keep this PR to the #2955 change.

We are reviewing the rest of this round now.

@DeusData DeusData left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @Bowen-Xue-ac2, thank you for this round, and for turning it around so quickly! You took every point from the last review on board, and in a couple of places you went further than we asked. In particular, moving the single time budget into a STATUS_DETAILS-only transport and testing it against a scripted clock (daemon_runtime_status_details_uses_one_budget_across_legacy_fallback and ..._stops_when_budget_expires) is exactly the kind of test we want: no scheduler delay can consume the budget or decide the result. Thank you also for fixing the Windows long cache root yourself rather than leaving it to us.

We built 18cfd55 with ASan/UBSan on macOS and ran the affected suites (daemon_runtime, daemon_frontend, daemon_bootstrap, cli, subprocess, runtime_settings): 592 passed and 3 were skipped as Windows-only. The one failure is specific to our test machine, also fails on main there, and has nothing to do with your change. We also repeated last round's real conflict on macOS: a live session on one cache root, then cli list_projects and the MCP entry point on another. Both still name the requested cache and the daemon cache, both for a root with a space and non-ASCII characters and for one with a quote, a newline and an ESC byte (printed as cache"q\nnl\u001besc). The active session was undisturbed, and the conflict log still holds only the two fingerprints.

Where the four asks stand:

  1. Windows long cache roots: done. The settings database now opens through SQLite's long-path VFS (cbm_runtime_settings_open_db), the private-directory walk in src/daemon/ipc.c adds the \\?\ prefix from 240 characters on while still rejecting . and .. components itself, and cbm_config_open no longer truncates silently. Your 259/260/261/1024 lengths stayed in check_cache_conflict_paths, and test-windows-guards is green on your ce65401 run, so the Windows leg now proves it. One optional suggestion, not a blocker: cbm_path_for_file_api() in src/foundation/compat.c already produces the prefixed UTF-8 spelling into a caller buffer, so cbm_runtime_settings_open_db could call it and pass the result to sqlite3_open_v2(..., "win32-longpath") instead of repeating the wide round trip. Same behaviour, one less copy of the conversion to keep in sync.

  2. runtime_control_request_send: done. It is byte-for-byte what main has, so daemon status, daemon stop, the start confirmation and the CLI guards keep their own budgets, and only cbm_daemon_runtime_request_status_details shares one deadline across the STATUS_DETAILS attempt and the legacy retry.

  3. The polling fallback guard in tests/windows/test_daemon_lifecycle.py: done, thank you. One leftover: import time (line 28) was only used by that loop and is now unused, so please drop it too.

  4. The non-Windows control-character branch: done.

Two commits in this round are outside #2955, and we would like this PR to stay about the cache-path diagnostic only. Keeping each PR to the one change it names makes it easier to review, to revert if something goes wrong, and to find later with git bisect.

  • ce65401 (test(subprocess): measure RSS growth within one process tree). We can see why you wrote it: the Windows leg on your 889170c run failed subprocess_tree_rss_measures_contained_descendants with idle 167 MiB against loaded 245 MiB, below the 96 MiB floor. That failure does not come from your change, because nothing in this PR touches the process-tree probe. The test compares two independently spawned process trees, and on Windows their baselines can differ. It is a real problem in that test, and thank you for digging into it, but it belongs in its own PR so we can review it on its own. If you open one, we would like to look at the handshake there: the descendant now polls for the load marker every 10 ms, and this project avoids adding new poll loops to tests. A blocking handshake, for example the descendant blocking on a pipe read until the parent has taken its idle sample, would avoid the loop. Please take ce65401 out of this branch (a revert commit is fine). If the Windows leg trips over that test again afterwards, we will treat it as our problem, not yours.

  • af0205d (fix(cli): honor skip-binary for existing targets).

You found a real bug here: under --yes, the "a different binary already exists" prompt turned the copy back on even though --skip-binary promised to leave the target alone, and on macOS an existing target was re-signed and republished anyway. You published your fix before we opened #2990 for the same bug, so we would like yours to land under your name, and we will close #2990 in its favour. Because it is separate from #2955, please move af0205d into its own PR: a branch from main with that commit cherry-picked onto it is enough. Your tests already cover more than ours did (the --force combination, the absent target, config and hooks still written, native file identity on Windows, the real-IPC no-drain check and the --yes replacement control). Two small additions would make it complete. First, a case where install infers skip-binary because the running binary is managed by mise, Homebrew or nix (#1566/#2722). Your production change already fixes that path, and the test just pins it. Second, a check that nothing is left staged next to the target (the bin directory holds exactly one entry afterwards). cli_install_managed_binary_keeps_a_different_existing_binary_issue2722 in #2990 does both, and you are welcome to copy it as it is. If you would rather we add it, just say so and we will push it to your new branch.

Once ce65401 and af0205d are out of this branch and the stray import is gone, the remaining red on the Ubuntu x86 GCC leg is the runner issue we are fixing in #2987, not your code, and we can take it from there. Thanks again for the careful work and the quick turnaround!

This reverts commit af0205d.

Signed-off-by: BowenX <318502935+Bowen-Xue-ac2@users.noreply.github.com>
This reverts commit ce65401.

Signed-off-by: BowenX <318502935+Bowen-Xue-ac2@users.noreply.github.com>
Signed-off-by: BowenX <318502935+Bowen-Xue-ac2@users.noreply.github.com>
@DeusData

Copy link
Copy Markdown
Owner

A quick follow-up on the RSS test, Bowen: since that Windows test was tripping other PRs as well, we went ahead and rebuilt it ourselves in #2992, so there is no need for you to open a separate PR for it. Your diagnosis was the key: the two independently spawned process trees are exactly why it flaked, and #2992 credits you for it. It now samples one tree before and after loading, with a blocking pipe handshake instead of the marker poll.

For this PR that leaves only dropping ce65401 (a revert commit is fine), the unused import time, and moving af0205d into its own PR as described above. Thanks again!

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Daemon cache-directory mismatch error shows SHA-256 fingerprints instead of the actual cache paths

2 participants