Repository navigation
refactor(mcp): port context and recent-activity tests to bm_context and bm_project - #1962
Conversation
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: d46907990d
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
|
@codex review |
|
Codex Review: Didn't find any major issues. Delightful! Reviewed commit: ℹ️ About Codex in GitHubCodex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback". |
e461f63 to
3d80dfb
Compare
|
@codex review |
|
Codex Review: Didn't find any major issues. Chef's kiss. Reviewed commit: ℹ️ About Codex in GitHubCodex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback". |
…nd bm_project Tests now call the published surface. bm_context tests absorb the old build_context and compact-label tests; bm_project(command="recent") gets its own test file; index-readiness tests call bm_search and bm_project. Source cleanup: - Delete tools/build_context.py. Its formatting helpers move into tools/context.py; with_paths is gone because bm_context always names paths. - Trim tools/recent_activity.py to the one-project path bm_project uses: drop the legacy recent_activity() wrapper, discovery mode across all projects, its formatters, and the in-function argument guards the published schema already enforces. - Drop the ActivityStats, ProjectActivity and ProjectActivitySummary schemas, which only discovery mode used. tools/__init__.py still re-exports build_context and recent_activity; the coordinator removes those two lines. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CMYVL1x91B8kJxy7nn2GU8 Signed-off-by: phernandez <paul@basicmachines.co>
- Remove the build_context and recent_activity re-exports from mcp/tools/__init__.py so the package imports on its own. - Port the build_context telemetry test to bm_context. bm_context has no tool-level root span, so the test asserts the memory client span, the API span, and the routing session. - Remove allow_discovery from resolve_project_parameter and ProjectResolver.resolve, together with ResolutionMode.DISCOVERY and ResolvedProject.is_discovery_mode. Nothing in core or Cloud passes it now that recent activity's discovery mode is gone. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CMYVL1x91B8kJxy7nn2GU8 Signed-off-by: phernandez <paul@basicmachines.co>
…olution No caller in core or Cloud passes default_project to resolve_project_parameter. The default always comes from config, the request cache, or the projects API. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CMYVL1x91B8kJxy7nn2GU8 Signed-off-by: phernandez <paul@basicmachines.co>
|
@codex review |
3d80dfb to
484debd
Compare
|
Codex Review: Didn't find any major issues. Hooray! Reviewed commit: ℹ️ About Codex in GitHubCodex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback". |
Refs #1955. The teardown hang is still open there; see the last section. This PR fixes the flakes reported in #1955 plus two the coordinator added: the linear-scaling tokenizer tests and the MCP container leak. Each section gives the cause, the evidence and the change. Verification is at the end. ## 1. `vtable constructor failed: search_index` in the watcher lifecycle test (test bug, real SQLite behavior) **Cause.** The SQLite `engine_factory` fixture in `test-int/conftest.py` created `search_index` but not the vector storage. So the first embedding of the new note ran `_ensure_vector_tables`, which created `search_vector_chunks`, its indexes and the `vec0` table while `bm_search` polled on another pooled connection. A schema change makes other connections reload the schema and reconnect the FTS5 table. A reconnect that overlaps the DDL fails with this error. **Evidence.** - CI logs for jobs 123581698132 (main, ab6a059) and 123623036006 (#1962) both fail in the `narwhal` loop with this error. - I reproduced it locally on main with a DDL tracer, a pytest plugin that logs every DDL and search statement with timestamps. The failing `SELECT COUNT(*) FROM search_index` is interleaved with `CREATE INDEX ... search_vector_chunks`, issued from `_ensure_vector_tables` <- `sync_entity_vectors_internal`, about 1 ms earlier. - A standalone sqlite3 script gives the same `vtable constructor failed: search_index`. It commits schema changes from a second connection while the first prepares an FTS5 query. **Not a product bug.** Production opens the database through `get_or_create_db` -> `run_migrations` -> `init_search_index`, which creates `search_index`, `search_vector_chunks` and `search_vector_embeddings` before any request. I confirmed this on a fresh database. The `_ensure_vector_tables` DDL that runs per repository instance afterwards uses `IF NOT EXISTS`. Re-running it leaves `PRAGMA schema_version` unchanged (10 -> 10), so it cannot trigger the reconnect. **Fix.** The SQLite fixture now calls `create_search_repository(...).init_search_index()`, as the Postgres branch already did. After the fix, the trace shows no schema-changing DDL during the search loops. The `TimeoutError` symptom (job 123284407567, #1953) times out in the same `narwhal` loop, right where the first embedding creates the vector tables. A watcher batch that fails is logged and dropped until the next change, so the same race on the indexer's write would leave `narwhal` unindexed. CI lost the loguru output ("I/O operation on closed file"), so this link is inferred, not shown. ## 2. Module-global `_initial_index_tasks` (product lifecycle fix) **Cause.** Startup project indexes were fire-and-forget tasks held in a module-level set. Nothing owned them: - `WatchCoordinator.stop()` cancelled only the watcher. - The lifespan could close the database while they still ran. - The set outlived event loops. A `_CapturedTask` fake from `tests/services/test_initialization.py` was later gathered by the lifecycle test (`TypeError`). **Fix.** `initialize_file_indexing` runs the startup indexes and `watch_service.run()` in one `asyncio.TaskGroup`. Cancelling the watcher now cancels and awaits the indexes. The global and the test's `gather(*_initial_index_tasks)` are gone. Each index task still catches and logs its own `Exception`, so one failing project does not cancel the watcher. `CancelledError` still propagates. A new test, `test_a_failed_startup_index_leaves_the_watcher_running`, uses a project whose root is a regular file, so its real index fails, and asserts the watcher keeps running. The test fails if the handler re-raises. The teardown hang (loop close stuck in `_cancel_all_tasks`) reproduced once locally in about 40 loaded runs, with the same stack as CI job 123279312418. It did not reproduce again in 25 more runs with a pending-task dump installed, so I cannot name the task that ignored cancellation. This change removes the startup indexes as unowned tasks that outlived the lifespan. ## 3. `test_file_sqlite_concurrent_writes_commit_and_dispose` (test design) **Cause.** The failure is on Windows, main 5b1d843, run 40839542939, job 123332360458. It was not in the ab6a059 run. SQLite's busy handler polls with growing sleeps of up to 100 ms instead of queueing waiters. When 64 writers each ran 40 transactions back to back, the other pooled connections could pass the write lock between themselves for the whole run while one waiter slept through every free moment. The run took 19.97 s on Windows against about 3 s on Linux, and one waiter passed `busy_timeout` (10 s): `database is locked` on the first `INSERT` of a transaction. **Fix.** Same 2560 writes, run in 40 rounds of 64 concurrent writers. A waiting connection now competes only with one round's work, well under a second, never with the rest of the run. Contention within each round is unchanged. **Note, not changed here.** On Windows, `connect_args={"timeout": 30.0}` is overridden by the per-connection `PRAGMA busy_timeout=10000`, so the 30 s setting never takes effect. ## 4. Postgres refresh negative controls (deleted) **Cause.** These controls assert that overlapping refreshes produce `search_index_pkey`. The upsert's `ON CONFLICT` absorbs any conflict it sees in its arbiter index; it waits for the other transaction, then updates. The pkey error happens only when two inserts pass the arbiter check in the same instant. The unit engine uses `NullPool`, so on a loaded runner one refresh could commit before the last one had deleted anything, and the round did not collide (job 123284407396). **Why deleted.** I first replaced the sleep with an `asyncio.Barrier` between delete and insert. With every delete done before any insert, the saves control still missed 1 run in 5. Nothing outside the INSERT can force that instant. The tolerance is already pinned deterministically: - `tests/services/test_search_refresh_atomicity.py`: an injected lost race is logged and the previous projection kept; any other database error still raises. - `tests/repository/test_search_refresh_race.py`: which errors count as a lost race. The positive concurrency tests still assert that overlapping refreshes never fail. ## 5. Linear-scaling tokenizer tests (robust measurement) **Cause.** The tests compared wall-clock time (`perf_counter`), which includes time the test spends descheduled. On a loaded machine, the old version failed 8 of 10 runs with 24 CPU burners. **Fix.** The tests now measure `time.thread_time()` (this thread's CPU only) with `gc` paused, best of 5, and keep the ×2.5 bound. There is no operation count to assert instead: the quadratic versions make the same calls and inspect the same characters, and only the bytes each slice copies grow, inside C. - Under the same load the new version passed 20 of 20 runs. - The quadratic implementations still fail it (×2.96 and ×3.3), checked by temporarily reverting them. ## 6. MCP container leak (product fix) **Cause.** The lifespan called `set_container()` and never reset it. After an in-process `Client(mcp)` session ended, later code in the same process, such as picking the default search mode, read that session's config. **Fix.** A new `basic_memory.mcp.container.installed_container` installs the container for the lifespan only and restores the previous one on exit, mirroring the API's `installed_container`. Regression test: `test_mcp_lifespan_uninstalls_its_container_on_exit`. ## Verification - `just fast-check`: clean. - `just test-int-sqlite`, full suite: 720 passed, 14 skipped. - `just test-int-postgres -k "watch_lifecycle or lifespan or mcp_read_cache or sqlite_lifecycle"`: passed. - `just test-unit-sqlite` and `just test-unit-postgres` on the touched modules: passed. - Coverage of the touched src: `mcp/container.py` 100%, `services/initialization.py` 100%. `mcp/server.py` misses only lines this PR did not change. - Repro loops: the lifecycle and SQLite contention tests passed 14 of 15 runs under 8 burners, with no vtable errors; the one failure was the teardown hang described in section 2. Then 25 of 25 under 2 burners. On main, 1 of about 30 traced or loaded runs hit the vtable error. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01CMYVL1x91B8kJxy7nn2GU8 --------- Signed-off-by: phernandez <paul@basicmachines.co> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…#1963) ## Summary **Stacked on #1964** (`refactor/remove-cloud-only-legacy-tools`), which removes `read_note`, the last core caller of `search_notes`. Merge #1964 first. Two commits: 1. **test(mcp): port search tests to bm_search.** Ports the search tests from the retired `search_notes` function to the published `bm_search` tool. 2. **refactor(mcp): remove search_notes.** `bm_search` is now the only caller, so `search.py` keeps only the shared query pipeline (about 1,100 src lines deleted). - Deleted `search_notes`, `_format_search_error_response`, `_format_service_unavailable_response`, `_format_search_markdown`, `_NO_SEARCH_CRITERIA_MESSAGE`, `METADATA_FILTERS_DESCRIPTION`, and the `tools/__init__` re-export. - `_search_all_projects` no longer takes `output_format`. It always returns the merged rows. `MultiProjectRan` carries `payload`, and `MultiProjectFailed` carries only `reason`. This removes `bm_search`'s `# pragma: no cover` text branch. - Dropped the `status` parameter of `_build_search_query` and the `tags` parameter of `split_tag_shorthand`; only `search_notes` passed them. `split_tag_shorthand` now returns tags in query order instead of set order. - Deleted `utils.strict_search_tags` (the tags coercer for `search_notes`) and its tests in `tests/test_coerce.py`. - Ported the remaining callers: `test_index_readiness.py`, which #1962 also ports; the parity test in `test_tool_posix.py`, which #1960 deletes; and the `search_notes` entries in `test_tool_contracts.py` and `implementation_mcp.py`. - Added tests for `bm_search` paths that were covered only incidentally before: - the explicit-scope refusal; - the `(more matches from line N)` hint; - the semantic excerpt in text output; - non-Markdown hits dropped by `lines`; - invalid and duplicate project refs; - workspace permalink qualification. ## Files | Before | After | |---|---| | `tests/mcp/test_tool_search.py` | `tests/mcp/test_tool_bm_search_query.py` (new) | | `tests/mcp/test_mcp_note_type_normalization.py` | folded into `test_tool_bm_search_query.py` | | `tests/mcp/test_search_total_exactness.py` | folded into `test_tool_bm_search_query.py` | | `tests/mcp/test_tool_search_temporal.py` | `tests/mcp/test_tool_bm_search_temporal.py` | | `tests/mcp/tools/test_search_notes_multi_project.py` | `tests/mcp/tools/test_bm_search_multi_project.py` | | `tests/mcp/tools/test_search_notes_multi_project_temporal.py` | folded into `test_bm_search_multi_project.py` | | `tests/mcp/tools/test_cjk_search_guidance.py` | folded into `test_bm_search_multi_project.py` | | `tests/api/v2/test_cjk_search_guidance.py` | its MCP assertions now call `bm_search` | | `tests/mcp/test_tool_bm_search.py` | two tests that replaced `_search_all_projects` now run the real path or moved to the multi-project file | ## Ported to real code paths These tests used to stub `get_project_client`, `resolve_project_and_path` and `SearchClient` and inspect the captured payload. They now do one of two things: - Run `bm_search` end to end over notes written with `bm_write`: title and permalink modes, note types including snake_case canonicalization, items, categories, `since`, the `note_type` filter alias, tag shorthand, total exactness per mode, default hybrid mode, Boolean queries defaulting to text, configured `default_search_type`, never-indexed versus indexed misses, and comma or JSON list strings through the MCP layer. - Call the shared pure helpers that `bm_search` uses on every request: `_build_search_query`, `split_tag_shorthand` and `_search_type_for`. The valid-time doc tests now check `bm-search(3)` instead of the `search_notes` docstring. ## Remaining monkeypatches, and why - **Multi-project tests:** several databases only exist in Cloud. These tests stub the per-database boundary: project discovery, the per-workspace `get_client`, `ScopedSearchClient` and the readiness probe. The merge, paging, attribution and failure handling run for real. A new test, `test_hits_from_two_local_projects_name_their_project`, runs the whole path with no stubs across two real local projects. - **"Filter ran nowhere" temporal test:** stubs only `ScopedSearchClient`, to stand in for a server older than SPEC-82. - **Keyword-only host:** two tests patch `semantic_runtime_available`, because sqlite-vec loads in the test environment. - **Default-mode tests:** these install the MCP container the way the server lifespan does. An earlier in-process MCP session leaves its container behind with a stale config, so each test sets its own. ## Dropped, with reasons - **Error formatting** (`TestSearchErrorFormatting`, `TestSearchToolErrorHandling`, invalid `search_type` returned as a string): `bm_search` raises tool errors instead of returning guidance markdown. Its refusals are covered in `test_tool_bm_search.py`. - **`_format_search_markdown` tests, and the valid-time label in markdown output:** this is the legacy text format. `bm_search` renders with `_format_results`. - **Workspace memory:// routing through `search_notes`:** `bm_search` routes memory:// URLs through `open_project_target`. That path is covered by `test_note_targets.py` and the project-qualified memory URL test. - **`search_notes` parameter-annotation tests and `tags=` tests** (comma-split, invalid types, direct-call CLI regression): `bm_search` has no `tags` or `status` parameters, and no CLI calls `search_notes` any more. The equivalent `bm_search` list parameters are tested through MCP. - **Multi-project opt-in:** `bm_search` requires an explicit scope, and that refusal is already tested. - **Redundant cases:** plain text search, the project-prefixed memory URL and `page_size=1` pagination are already covered in `test_tool_bm_search.py`. ## Coverage `search.py` and `ranked_search.py` are both at 100%, measured over the touched test files. ## Verification - `just fast-check` - `just test-unit-sqlite -k "search or test_index_readiness or test_tool_contracts or test_coerce or ..."`: 1146 passed. One failure, `test_tokenizing_scales_linearly_with_query_length`, is a timing test; it passes when rerun on its own. - `just test-unit-postgres -k "bm_search or test_cjk_search_guidance or test_index_readiness or test_tool_contracts or test_coerce or bm_search_parity"`: all pass, after making one new test backend-neutral. - `just test-int-sqlite -k search`: 64 passed 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01CMYVL1x91B8kJxy7nn2GU8 --------- Signed-off-by: phernandez <paul@basicmachines.co> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
## Why `just release v1.0.0` pre-flight-checks for a `## v1.0.0` heading on `main`, so the changelog has to land first. ## What changed `CHANGELOG.md`: the Unreleased section becomes `## v1.0.0 (2026-10-29)`. Its existing items are kept, and the 60 or so commits since #1871 are added: - **Breaking:** - the MCP surface is only `bm_*`, with the old tools deleted (#1944, #1947, #1959, #1962–#1964) - `project` is required (#1934) - the ChatGPT adapters moved to Cloud - `bm tool` mirrors the tools (#1943) - the POSIX verbs are rebuilt and `bm tail` is removed (#1960) - cloud sync/bisync are removed (#1953) - `bm schema` project resolution (#1959) - LiteLLM is an optional extra - **Features:** text-file line ranges in `bm_read`; chat imports return their file paths. - **Bug fixes:** including the three found in the 1.0 live test (#1971, #1972, #1974), with contributor credits for @Flynn-Kawolsky (#1957), @SWSAmor (#1951) and @MbappeWU (#1948). - **Security:** the CodeRabbit deep-scan hardening batch. I checked the entries against PR bodies rather than titles, and corrected #1954, #1948 and #1967 to describe what they actually fixed. The date is today; change it if the tag lands tomorrow. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01CMYVL1x91B8kJxy7nn2GU8 Signed-off-by: phernandez <paul@basicmachines.co> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Summary
This ports the context and recent-activity tests to the published 1.0 surface,
bm_contextandbm_project(command="recent"). It also deletes the pre-1.0 code that no caller reaches anymore, including the project resolver's discovery mode. Part of the legacy MCP tool cleanup.The PR stands on its own: the package imports, and the full unit suite passes.
Source
tools/__init__.py: removed thebuild_contextandrecent_activityre-exports.Deleted
tools/build_context.py. Its formatting helpers (_compact_context_labels,_format_context_markdown,_format_entity_block) moved intotools/context.py. Thewith_pathsflag is removed becausebm_contextalways prints paths.Trimmed
tools/recent_activity.pyfrom 725 to about 260 lines. What remains is the one-project path thatbm_projectcalls:items(ItemKind) and maps them withsearch_item_types.get_project_client.recent_activity()wrapper;bm_projectcannot reach because it requiresproject;_get_project_activity,_format_discovery_outputand_format_relative_time;RecentRows.guidancefield;Removed the
ActivityStats,ProjectActivityandProjectActivitySummaryschemas. Only discovery mode used them.Removed the dead
allow_discoveryplumbing:resolve_project_parameter(mcp/project_context.py) and onProjectResolver.resolve(project_resolver.py);ResolutionMode.DISCOVERYandResolvedProject.is_discovery_mode.No src caller passed
allow_discovery=Trueafter this change. Cloud does not use it either: I checked GitHub code search, a grep of the local Cloud checkout, and the diffs of open Cloud PRs #2857, #2856 and #2749.Removed the unused
default_projectparameter ofresolve_project_parameter. No caller in core or Cloud passed it; Cloud mentions the function only in a comment inmain.py.In total: 11 src files changed.
Tests ported
test_tool_build_context.pyandtest_compact_context_labels.py→test_tool_bm_context.py. Every behavior is kept: fields, patterns, workspace routing by external id, timeframe, misses, compact, text output, unresolved relations (build_context renders unresolved forward references as [[None]] instead of the stored target name #955), incoming relations (build_context text output shows incoming relations without their source #1717), non-slug permalinks withbm_read([BUG]build_contextsilently returns no results formemory://URIs when a note sets its permalink explicitly #1549), fuzzy misses (build_context returns an unrelated note for memory:// titles and misses #1626), ambiguous titles, and timestamp offsets. Argument validation (invalidsince, a stringdepth, out-of-rangedepth) now runs throughClient(mcp), where the schema applies.test_tool_recent_activity.py→ newtest_tool_bm_project_recent.py. Tool calls usebm_project, and argument refusals run through MCP. The formatter unit tests are kept for the recent_activity formatter hardcodes display cap of 5, ignoring page_size #784 row count, truncation, empty pages, filtered results, and the next-page hint.test_index_readiness.py.search_notescalls becamebm_searchandrecent_activitycalls becamebm_project. The all-projects index-required case now runs on the real path instead of a monkeypatch.test_tool_telemetry.py. Thebuild_contextspan test now callsbm_contextand the module-levelbuild_contextimport is gone; the rest of the file is unchanged.bm_contexthas no tool-level root span, so the test checks three spans instead:mcp.client.memory.build_contextwith its paging attributes,api.request.memory.build_context, and therouting.client_sessionproject context.Tests dropped
uses_default_project,_get_project_activity,_format_discovery_output,_format_relative_time,test_activity_discovery_preserves_index_required): the code is deleted, andbm_projectrefuses a call that has no project, whichtest_recent_requires_projectalready covers.allow_discoveryresolver tests (test_allows_discovery_when_enabled,test_discovery_resolution_when_allowed, and the threeis_discovery_modetests): the feature is removed.SearchItemTypeenum inputs andtype="note"being rejected: these were legacy parameter forms.itemsis a validated literal, and"note"is now a valid value.bm_projecthas nodepthparameter.Outside the assignment, fixed minimally because the src change broke them
tests/implementation_mcp.py: removedbuild_contextandrecent_activityfromIMPLEMENTATION_NAMES.tests/mcp/test_tool_contracts.py: removed their signature, annotation and function entries.tests/mcp/test_tool_json_output_modes.py: removed the three legacy recent/context tests. The ported files cover the same row keys and JSON/text modes.Test plan
just fast-checkpasses.just test-unit-sqlite, full suite with nothing ignored: 8831 passed, 79 skipped.just test-unit-postgres -k "bm_context or bm_project or index_readiness or json_output_modes or tool_contracts or tool_telemetry or project_resolver or project_context": 227 passed.tools/recent_activity.pyandtools/projects.py. Intools/context.py, the one line the targeted tests miss (the whitespace-identifier refusal) is covered bytest-int/mcp/test_build_context_validation.py.🤖 Generated with Claude Code
https://claude.ai/code/session_01CMYVL1x91B8kJxy7nn2GU8