diff --git a/AGENTS.md b/AGENTS.md index 7f70da6e..d9c3c13d 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -52,9 +52,12 @@ Use `.ccg.yaml` to manage project defaults such as exclude patterns and DB setti ## Code Search Rules When looking for code locations, related implementations, call relationships, impact radius, or architecture context, -use ccg MCP tools and Agent Skills first. +use CCG Agent Skills and environment-aware structured search first. The `/ccg` skill selects MCP when repository +instructions provide server-visible routing, and local JSON CLI when the repository has usable local `ccg` configuration. +For this repository, use the local JSON CLI backed by `.ccg.yaml`; the MCP server documentation below describes the +product and is not a routing instruction for the default remote MCP connection. -- `/ccg` is the fast default for ordinary positive discovery: use at most one `search` call with `limit: 5`, then verify the best candidate in one or two source ranges. Skip namespace, minimal-context, and graph-stat preflights when repository instructions already provide what the query needs. +- `/ccg` is the fast default for ordinary positive discovery: start with one `search` call using `limit: 5`, pass the namespace already supplied by repository instructions or `.ccg.yaml` without a preflight, verify the best candidates in targeted source ranges, and follow at most three verbatim `next` calls only while the evidence remains insufficient. Skip namespace-list, minimal-context, and graph-stat preflights when repository instructions already provide what the query needs. - Use `/ccg-search-verify` when the user asks whether code does not exist, requests completeness or exhaustive inventory, or when a miss would become a defensible negative claim. It owns freshness, hybrid source checking, and truncation paging. - CCG `search` answers identifier queries and "why was this built" questions from one index. Use the `/ccg-docs` skill and `get_doc_content` to read a generated doc. - For exact symbol locations and one direct call relationship, use ccg MCP `query_graph`, `get_node`, or the `/ccg` skill. Use `get_minimal_context` only when the MCP tool contract needed for the task is unavailable; it is not an ordinary search preflight. diff --git a/guide/cli-reference.md b/guide/cli-reference.md index abf6bba5..cbdfec9d 100644 --- a/guide/cli-reference.md +++ b/guide/cli-reference.md @@ -48,6 +48,7 @@ ccg update ./backend --namespace backend | `ccg search --offset ` | Skip the first `n` files, so reading on never splits a file; the last line names the offset to use next | | `ccg search --include-weak ` | Also show candidates whose name, path, and `@intent` say nothing about the query | | `ccg search --json ` | Print the answer as JSON, in the same shape the MCP `search` tool returns — stable for scripts and diffs | +| `ccg search --json --compact ` | Print a smaller agent-oriented JSON view that keeps file paths, declaration bounds, evidence, truncation, and exact next actions while omitting redundant IDs and repeated per-hit names/paths | | `ccg docs [--out dir]` | Generate Markdown documentation and the `wiki-index.json` compatibility snapshot (prunes stale generator-managed docs by default) | | `ccg docs --rag-index-dir ` | Override the legacy-named Wiki index output directory (default `.ccg` or `rag.index_dir`) | | `ccg docs --prune=false` | Regenerate docs without deleting older generator-managed files | diff --git a/guide/mcp-tools.md b/guide/mcp-tools.md index ffeeb606..3fbc07fa 100644 --- a/guide/mcp-tools.md +++ b/guide/mcp-tools.md @@ -15,7 +15,7 @@ code-context-graph exposes 18 MCP tools through both local `ccg serve` and the s | Tool | Purpose | | ---- | ------- | | `get_node` | Read one node by qualified name | -| `search` | Full-text search across code nodes, grouped by file: results arrive in `files`, one entry per file as `{file_path, hit_count, hits[]}`, with `file_count` giving how many of them are on this page, and a file that appears appears whole. It takes both shapes of query: a symbol you can name (an identifier, a type, two or three such words) and a plain-language question ("how does the graph get built"). Named symbols match against the indexed nodes, where every term has to appear in the same node; a question is additionally scored against the reasons authors recorded (`@intent`, `@domainRule`), and files those reasons justify are appended after the name matches. `limit` counts files and `offset` pages by files, so a page never splits one. Every hit carries its evidence (`matched` signals plus the node's `@intent`; a reason-matched hit also carries `reason` and `matched_terms`); unjustifiable candidates are cut and counted in `weak_filtered`. Optionally scoped by `path`; `include_weak: true` returns the cut ones; `namespaces: []` federates across namespaces with per-item labels, and there `limit` and `offset` are per namespace, so every namespace with a hit is on the page whatever the limit is. `truncated` says whether more files answered than this page reached, `pool_truncated` says the page ended at the edge of the candidates that were fetched rather than at the end of the answer, and `next` names the calls that retrieve them. Only both signals false means the search is complete. `limits` restates the bounds this page was cut to — `files` and `offset` as they were requested, plus the `hit_budget` that decides whether one more file still fits. `annotation_coverage` reports `with_reason` out of `declarations` — how many searched declarations carry an `@intent` or a `@domainRule`, counted per declaration rather than per tag — so `with_reason: 0` says a question was put to an index nobody has recorded a reason in, and the empty answer is about the missing annotations rather than about the code. When nothing on the page could justify itself, `next` also carries an entry that names a `skill` (`ccg-annotate`) instead of a tool. `note` is set only when the answer came back with no files at all, and says which kind of empty it is: nothing retrieved, nothing that could justify itself, a page past the end, or a repository where nobody has recorded a reason yet | +| `search` | Full-text search across code nodes, grouped by file: results arrive in `files`, one entry per file as `{file_path, hit_count, hits[]}`, with `file_count` giving how many of them are on this page, and a file that appears appears whole. It takes both shapes of query: a symbol you can name (an identifier, a type, two or three such words) and a plain-language question ("how does the graph get built"). Named symbols require every term in one indexed node. Questions try that precise match first; if it is empty, they retrieve nodes matching any meaningful term and rank nodes higher when they match more distinctive terms. Questions are also scored against reasons authors recorded (`@intent`, `@domainRule`), and files those reasons justify are appended after the general matches. `limit` counts files and `offset` pages by files, so a page never splits one. Every hit carries its evidence (`matched` signals plus the node's `@intent`; a reason-matched hit also carries `reason` and `matched_terms`); unjustifiable candidates are cut and counted in `weak_filtered`. `compact: true` returns the agent-oriented view that keeps paths, declaration bounds, evidence, completion signals, and exact next actions while omitting storage IDs and repeated per-hit names/paths; continuations keep compact mode. Optionally scoped by `path`; `include_weak: true` returns the cut ones; `namespaces: []` federates across namespaces with per-item labels, and there `limit` and `offset` are per namespace, so every namespace with a hit is on the page whatever the limit is. `truncated` says whether more files answered than this page reached, `pool_truncated` says the page ended at the edge of the candidates that were fetched rather than at the end of the answer, and `next` names the calls that retrieve them. Only both signals false means the search is complete. `limits` restates the bounds this page was cut to — `files` and `offset` as they were requested, plus the `hit_budget` that decides whether one more file still fits. `annotation_coverage` reports `with_reason` out of `declarations` — how many searched declarations carry an `@intent` or a `@domainRule`, counted per declaration rather than per tag — so `with_reason: 0` says a question was put to an index nobody has recorded a reason in, and the empty answer is about the missing annotations rather than about the code. When nothing on the page could justify itself, `next` also carries an entry that names a `skill` (`ccg-annotate`) instead of a tool. `note` is set only when the answer came back with no files at all, and says which kind of empty it is: nothing retrieved, nothing that could justify itself, a page past the end, or a repository where nobody has recorded a reason yet | | `describe` | List what the graph holds under one path, with no ranking. The answer echoes the path back as `target` and says in `scope` which kind of thing it turned out to name — `directory`, `file`, or `unknown` — and that decides which list is filled. A folder fills `children` with the folders and files directly inside, one level down, each with its file and declaration counts; a file fills `declarations` with every declaration written in it, in written order, each carrying its line range, `node_id`, and recorded `@intent`. This is what `search` hands off to: search ranks and can be wrong, this one only reports what exists. No query, no limit, no relevance. A target the graph does not hold answers with `scope` set to `unknown` and fills `suggestions` with the places that name is actually declared. It replaced the `children_of` and `file_summary` patterns of `query_graph` | | `get_annotation` | Read annotations and documentation tags for one node | | `query_graph` | Run callers, callees, imports, importers, tests, or inheritors queries; `namespaces: []` groups results per namespace. For what is written inside a file or folder, use `describe` | diff --git a/internal/adapters/inbound/cli/search.go b/internal/adapters/inbound/cli/search.go index 94f37ffb..9194e234 100644 --- a/internal/adapters/inbound/cli/search.go +++ b/internal/adapters/inbound/cli/search.go @@ -26,6 +26,7 @@ func newSearchCmd(deps *Deps) *cobra.Command { var pathPrefix string var includeWeak bool var asJSON bool + var compact bool cmd := &cobra.Command{ Use: "search ", @@ -33,6 +34,9 @@ func newSearchCmd(deps *Deps) *cobra.Command { Args: cobra.ExactArgs(1), RunE: func(cmd *cobra.Command, args []string) error { query := args[0] + if compact && !asJSON { + return fmt.Errorf("--compact requires --json") + } if limit <= 0 { return fmt.Errorf("limit must be > 0, got %d", limit) } @@ -54,7 +58,11 @@ func newSearchCmd(deps *Deps) *cobra.Command { } if asJSON { - return printJSONResponse(stdout(cmd), searchwire.NewResponse(list, query, limit, offset, false)) + response := searchwire.NewResponse(list, query, limit, offset, false) + if compact { + return printCompactJSONResponse(stdout(cmd), response.Compact()) + } + return printJSONResponse(stdout(cmd), response) } printEvidenceList(stdout(cmd), list, offset) return nil @@ -66,6 +74,12 @@ func newSearchCmd(deps *Deps) *cobra.Command { cmd.Flags().StringVar(&pathPrefix, "path", "", "Filter results to file paths starting with this prefix (e.g. internal/auth)") cmd.Flags().BoolVar(&includeWeak, "include-weak", false, "Also show candidates whose name, path, and @intent say nothing about the query") cmd.Flags().BoolVar(&asJSON, "json", false, "Print the answer as JSON, in the same shape the MCP search tool returns") + cmd.Flags().BoolVar( + &compact, + "compact", + false, + "With --json, omit redundant fields while preserving evidence, source bounds, and next actions", + ) return cmd } @@ -73,12 +87,20 @@ func newSearchCmd(deps *Deps) *cobra.Command { // printJSONResponse writes the wire payload as one indented JSON document. // @intent keep --json output byte-stable and diffable while staying the MCP contract. // @sideEffect writes the whole search answer to out. -func printJSONResponse(out io.Writer, response searchwire.Response) error { +func printJSONResponse(out io.Writer, response any) error { encoder := json.NewEncoder(out) encoder.SetIndent("", " ") return trace.Wrap(encoder.Encode(response), "encode search response") } +// printCompactJSONResponse writes one unindented JSON document so formatting +// whitespace does not consume an agent's context window. +// @intent keep compact search output compact on the wire as well as in its fields. +// @sideEffect writes the whole compact search answer to out. +func printCompactJSONResponse(out io.Writer, response searchwire.CompactResponse) error { + return trace.Wrap(json.NewEncoder(out).Encode(response), "encode compact search response") +} + // printEvidenceList writes one result per unindented line and everything else // indented, so the plain result lines stay as machine-readable as they were. // diff --git a/internal/adapters/inbound/cli/search_test.go b/internal/adapters/inbound/cli/search_test.go index 7b662737..76f3ff07 100644 --- a/internal/adapters/inbound/cli/search_test.go +++ b/internal/adapters/inbound/cli/search_test.go @@ -168,6 +168,53 @@ func TestSearchCommand_JSONSpeaksTheMCPContract(t *testing.T) { } } +func TestSearchCommand_CompactJSONKeepsOnlyDecisionEvidence(t *testing.T) { + deps, stdout, stderr, db := setupSearchTest(t) + seedSearchData(t, db) + + if err := executeCmd(deps, stdout, stderr, "search", "--json", "--compact", "Hello"); err != nil { + t.Fatalf("search: %v", err) + } + + var payload map[string]any + if err := json.Unmarshal(stdout.Bytes(), &payload); err != nil { + t.Fatalf("decode: %v output=%s", err, stdout.String()) + } + if lines := strings.Count(stdout.String(), "\n"); lines != 1 { + t.Errorf("compact JSON used %d lines, want one encoded document", lines) + } + file := payload["files"].([]any)[0].(map[string]any) + hit := file["hits"].([]any)[0].(map[string]any) + for _, key := range []string{"qualified_name", "kind", "start_line", "end_line", "matched"} { + if _, ok := hit[key]; !ok { + t.Errorf("compact hit is missing %q: %s", key, stdout.String()) + } + } + for _, duplicate := range []string{"id", "name", "file_path"} { + if _, ok := hit[duplicate]; ok { + t.Errorf("compact hit kept duplicate field %q: %s", duplicate, stdout.String()) + } + } + compactSize := stdout.Len() + stdout.Reset() + if err := executeCmd(deps, stdout, stderr, "search", "--json", "Hello"); err != nil { + t.Fatalf("full search: %v", err) + } + if compactSize*4 >= stdout.Len()*3 { + t.Errorf("compact CLI output is %d bytes versus %d full bytes, want at least 25%% smaller", compactSize, stdout.Len()) + } +} + +func TestSearchCommand_CompactRequiresJSON(t *testing.T) { + deps, stdout, stderr, db := setupSearchTest(t) + seedSearchData(t, db) + + err := executeCmd(deps, stdout, stderr, "search", "--compact", "Hello") + if err == nil || !strings.Contains(err.Error(), "--compact requires --json") { + t.Fatalf("error = %v, want --compact requires --json", err) + } +} + // A truncated --json answer carries the same next actions MCP emits, phrased as // a repeatable search call. func TestSearchCommand_JSONNamesTheNextPage(t *testing.T) { diff --git a/internal/adapters/inbound/cli/skills_contract_test.go b/internal/adapters/inbound/cli/skills_contract_test.go index 166b0c37..aae9d506 100644 --- a/internal/adapters/inbound/cli/skills_contract_test.go +++ b/internal/adapters/inbound/cli/skills_contract_test.go @@ -223,7 +223,11 @@ func TestProjectInstructionsRouteFastAndVerifiedSearchSeparately(t *testing.T) { text := string(raw) for _, phrase := range []string{ "`/ccg` is the fast default", - "at most one `search` call with `limit: 5`", + "environment-aware structured search", + "For this repository, use the local JSON CLI", + "start with one `search` call using `limit: 5`", + "pass the namespace already supplied", + "at most three verbatim `next` calls", "Use `/ccg-search-verify`", "freshness, hybrid source checking, and truncation paging", } { @@ -231,6 +235,64 @@ func TestProjectInstructionsRouteFastAndVerifiedSearchSeparately(t *testing.T) { t.Errorf("project instructions are missing search-mode boundary %q", phrase) } } + if strings.Contains(text, "use ccg MCP tools and Agent Skills first") { + t.Error("project instructions force MCP even when the fast skill selects a repository-local JSON CLI") + } +} + +func TestFastSearchSkillDescribesSoftRetrievalAndAdaptiveVerification(t *testing.T) { + raw, err := os.ReadFile(filepath.Join("..", "..", "..", "..", "skills", "ccg", "SKILL.md")) + if err != nil { + t.Fatal(err) + } + text := strings.ToLower(strings.Join(strings.Fields(string(raw)), " ")) + for _, phrase := range []string{ + "or matching", + "rejects single-term coincidences", + "bm25/idf", + "search first when the path is unknown", + "use it verbatim", + "do not compress a behavior or reason question into keywords", + "repository-local `.ccg.yaml`", + "use json cli", + "--compact", + "compact: true", + "explicit mcp or server-visible", + "mcp tool availability or mcp documentation alone", + "path-only reference lookup", + "before semantic paging", + "do not re-locate a hit with grep", + "each chosen declaration's exact range separately", + "exact-identifier miss", + "repeat the same local graph query", + "name the specific evidence gap", + "material evidence gap can", + "author-recorded design reason", + "one directly relevant author-recorded design reason", + "closes the rationale gap", + "secondary consequences are optional", + "do not trace constructor or dependency-injection wiring", + "state that gap in one sentence", + "do not create a new evidence gap from identifiers", + "do not load a reference merely because declaration bounds are missing", + "do not read tests to corroborate production behavior already established", + "next action must be the final answer", + } { + if !strings.Contains(text, phrase) { + t.Errorf("fast search skill is missing retrieval contract %q", phrase) + } + } + for _, overfit := range []string{ + "default fast budget is two production", + "at most one targeted in-file grep", + "Known filename, identifier, literal, or error text | Grep + Read may start directly", + "Never restart with repository-wide", + "Prefer MCP", + } { + if strings.Contains(text, strings.ToLower(overfit)) { + t.Errorf("fast search skill hard-codes an example-specific budget %q", overfit) + } + } } func TestProjectSkillsDoNotAdvertiseRemovedCommands(t *testing.T) { @@ -330,8 +392,10 @@ func TestProjectSkillsCoverOperationalHazards(t *testing.T) { required := map[string][]string{ "ccg": { "ordinary positive", - "`search` at most once", - "one or two source ranges", + "one initial structured ccg `search`", + "include it in the initial search arguments", + "at most three `next` calls", + "exact declaration", "do not call `get_minimal_context`", "`ccg-search-verify`", }, @@ -413,9 +477,10 @@ func TestProjectSkillsCentralizeSharedOperationalGuidance(t *testing.T) { skillsRoot := filepath.Join("..", "..", "..", "..", "skills") required := map[string][]string{ "ccg": { - "## Task Routing and Entry", - "## Freshness Boundary", - "## Response Budget Rule", + "## Route", + "## Core Loop", + "## Boundaries", + "## Completion", }, "ccg-search-verify": { "## Mandatory Verification", @@ -474,7 +539,8 @@ func TestProjectSkillsKeepCoreDiscoveryBoundedAndDeepAnalysisExplicit(t *testing required := map[string][]string{ "ccg": { "do not invoke `ccg-analyze` automatically", - "one bounded `search`", + "one initial structured ccg `search`", + "at most three `next` calls", "explicit `ccg-analyze` invocation", }, "ccg-analyze": { diff --git a/internal/adapters/inbound/mcp/handler_query.go b/internal/adapters/inbound/mcp/handler_query.go index 2acc333d..1128318d 100644 --- a/internal/adapters/inbound/mcp/handler_query.go +++ b/internal/adapters/inbound/mcp/handler_query.go @@ -186,6 +186,7 @@ func (h *handlers) search(ctx context.Context, request mcp.CallToolRequest) (*mc offset := request.GetInt("offset", 0) pathPrefix := request.GetString("path", "") includeWeak := request.GetBool("include_weak", false) + compact := request.GetBool("compact", false) if err := validateQueryGraphLimit(limit); err != nil { return finalizeToolResult("", err) } @@ -200,10 +201,14 @@ func (h *handlers) search(ctx context.Context, request mcp.CallToolRequest) (*mc } if namespaces := requestNamespaces(request); len(namespaces) > 0 { - return h.searchFederated(ctx, query, limit, offset, pathPrefix, includeWeak, namespaces) + return h.searchFederated(ctx, query, limit, offset, pathPrefix, includeWeak, compact, namespaces) } - return finalizeToolResult(h.cachedExecute(ctx, "search:", map[string]any{"query": query, "limit": limit, "offset": offset, "path": pathPrefix, "include_weak": includeWeak, "namespace": requestNamespace(request)}, func() (string, error) { + cacheKey := map[string]any{ + "query": query, "limit": limit, "offset": offset, "path": pathPrefix, + "include_weak": includeWeak, "compact": compact, "namespace": requestNamespace(request), + } + return finalizeToolResult(h.cachedExecute(ctx, "search:", cacheKey, func() (string, error) { list, err := searchapp.New(h.deps.Graph.Search).Search(ctx, searchapp.Params{ Query: query, Limit: limit, Offset: offset, PathPrefix: pathPrefix, IncludeWeak: includeWeak, }) @@ -214,7 +219,7 @@ func (h *handlers) search(ctx context.Context, request mcp.CallToolRequest) (*mc log.Info("search completed", "query", query, "file_count", len(list.Files), "weak_filtered", list.WeakFiltered) - result, err := marshalJSON(searchwire.NewResponse(list, query, limit, offset, false)) + result, err := marshalSearchResponse(searchwire.NewResponse(list, query, limit, offset, false), compact) if err != nil { return "", trace.Wrap(err, "marshal result") } @@ -225,9 +230,22 @@ func (h *handlers) search(ctx context.Context, request mcp.CallToolRequest) (*mc // searchFederated fans full-text search out over an explicit namespace set and merges reranked hits. // @intent answer one search across several repositories with per-item namespace labels. // @domainRule each namespace is queried in isolation; every namespace's hits keep their own backend rank when fused. -func (h *handlers) searchFederated(ctx context.Context, query string, limit, offset int, pathPrefix string, includeWeak bool, namespaces []string) (*mcp.CallToolResult, error) { +func (h *handlers) searchFederated( + ctx context.Context, + query string, + limit int, + offset int, + pathPrefix string, + includeWeak bool, + compact bool, + namespaces []string, +) (*mcp.CallToolResult, error) { log := h.logger() - return finalizeToolResult(h.cachedExecute(ctx, "search:", map[string]any{"query": query, "limit": limit, "offset": offset, "path": pathPrefix, "include_weak": includeWeak, "namespaces": namespaces}, func() (string, error) { + cacheKey := map[string]any{ + "query": query, "limit": limit, "offset": offset, "path": pathPrefix, + "include_weak": includeWeak, "compact": compact, "namespaces": namespaces, + } + return finalizeToolResult(h.cachedExecute(ctx, "search:", cacheKey, func() (string, error) { list, err := searchapp.New(h.deps.Graph.Search).SearchFederated(ctx, namespaces, searchapp.Params{ Query: query, Limit: limit, Offset: offset, PathPrefix: pathPrefix, IncludeWeak: includeWeak, }) @@ -237,7 +255,7 @@ func (h *handlers) searchFederated(ctx context.Context, query string, limit, off } log.Info("federated search completed", "query", query, "namespaces", namespaces, "file_count", len(list.Files), "weak_filtered", list.WeakFiltered) - result, err := marshalJSON(searchwire.NewResponse(list, query, limit, offset, true)) + result, err := marshalSearchResponse(searchwire.NewResponse(list, query, limit, offset, true), compact) if err != nil { return "", trace.Wrap(err, "marshal result") } @@ -245,6 +263,16 @@ func (h *handlers) searchFederated(ctx context.Context, query string, limit, off })) } +// marshalSearchResponse selects the full compatibility payload or the compact +// agent view without duplicating the search execution path. +// @intent keep CLI-equivalent compact search semantics on single and federated MCP calls. +func marshalSearchResponse(response searchwire.Response, compact bool) (string, error) { + if compact { + return marshalJSON(response.Compact()) + } + return marshalJSON(response) +} + // getAnnotation returns stored annotation metadata for a graph node. // @intent fetch stored annotation tags and summary data so semantic search results can show business context. // @param request qualified_name is the fully qualified node name whose annotations should be loaded. diff --git a/internal/adapters/inbound/mcp/handlers_test.go b/internal/adapters/inbound/mcp/handlers_test.go index 983d666a..68eb676b 100644 --- a/internal/adapters/inbound/mcp/handlers_test.go +++ b/internal/adapters/inbound/mcp/handlers_test.go @@ -277,6 +277,39 @@ func TestHandler_Search(t *testing.T) { } } +func TestHandler_Search_CompactKeepsOnlyDecisionEvidence(t *testing.T) { + deps := setupTestDeps(t) + ctx := context.Background() + + testGraphStoreFor(deps).UpsertNodes(ctx, []graph.Node{ + {QualifiedName: "pkg.AuthenticateUser", Kind: graph.NodeKindFunction, Name: "AuthenticateUser", FilePath: "auth.go", StartLine: 1, EndLine: 10, Language: "go"}, + }) + node, _ := testGraphStoreFor(deps).GetNode(ctx, "pkg.AuthenticateUser") + testDBFor(deps).Create(&graph.SearchDocument{ + NodeID: node.ID, Content: "AuthenticateUser authenticates user credentials", Language: "go", + }) + testSearchBackendFor(deps).Rebuild(ctx, testDBFor(deps)) + + result := callTool(t, deps, "search", map[string]any{"query": "authenticate", "limit": 10, "compact": true}) + if result.IsError { + t.Fatalf("search returned error: %s", getTextContent(result)) + } + var payload map[string]any + if err := json.Unmarshal([]byte(getTextContent(result)), &payload); err != nil { + t.Fatalf("decode: %v result=%s", err, getTextContent(result)) + } + file := payload["files"].([]any)[0].(map[string]any) + hit := file["hits"].([]any)[0].(map[string]any) + if _, ok := hit["qualified_name"]; !ok { + t.Fatalf("compact hit lost qualified_name: %v", hit) + } + for _, duplicate := range []string{"id", "name", "file_path"} { + if _, ok := hit[duplicate]; ok { + t.Errorf("compact hit kept duplicate field %q: %v", duplicate, hit) + } + } +} + // A caller that gets a result has to be able to open it and see why it is here, // without a second round trip. func TestHandler_Search_CarriesLineNumbersAndEvidence(t *testing.T) { diff --git a/internal/adapters/inbound/mcp/tools_query.go b/internal/adapters/inbound/mcp/tools_query.go index 2e3508a1..4fb8e136 100644 --- a/internal/adapters/inbound/mcp/tools_query.go +++ b/internal/adapters/inbound/mcp/tools_query.go @@ -35,12 +35,19 @@ func queryTools(h *handlers) []server.ServerTool { }, { Tool: mcp.NewTool("search", withFederatedNamespaceParams( - mcp.WithDescription("Full-text search across code nodes, grouped by file. It takes both shapes of query: a symbol you can name (an identifier, a type, a package word, or two or three such words together) and a plain-language question ('how does the graph get built', 'why do we verify the webhook signature'). Named symbols are matched against the indexed nodes, where every term has to appear in the same node; a question is additionally scored against the reasons authors recorded (@intent and @domainRule), and files those reasons justify are appended after the name matches. The answer is a list of files, and every file it shows it shows whole: all of that file's hits are in its 'hits' array. Every hit carries the evidence for it — the signals the query matched (name, path, intent), the node's own @intent tag, and, for a reason-matched hit, the recorded reason and the words of the question written in it ('reason', 'matched_terms'). Candidates nothing can justify are left out and counted in weak_filtered. Two separate signals say whether this is the whole answer: 'truncated' is true when more files answered the query than this page reached, and 'pool_truncated' is true when the page ended at the edge of the candidates that were fetched rather than at the end of the answer — one file whose hits fill the candidate pool leaves nothing for 'truncated' to count while later files are still waiting. Only both false means the search is complete; while either is true, make the call in 'next', which lists the exact calls that read on. Use 'path' to scope results to a module for token-efficient queries."), + mcp.WithDescription("Full-text search across code nodes, grouped by file. It takes both shapes of query: a symbol you can name (an identifier, a type, a package word, or two or three such words together) and a plain-language question ('how does the graph get built', 'why do we verify the webhook signature'). Named symbols require every term in one indexed node. Questions try that precise match first; if it is empty, they retrieve nodes matching any meaningful term and rank nodes higher when they match more distinctive terms. Questions are also scored against reasons authors recorded (@intent and @domainRule), and files those reasons justify are appended after the general matches. The answer is a list of files, and every file it shows it shows whole: all of that file's hits are in its 'hits' array. Every hit carries the evidence for it — the signals the query matched (name, path, intent), the node's own @intent tag, and, for a reason-matched hit, the recorded reason and the words of the question written in it ('reason', 'matched_terms'). Candidates nothing can justify are left out and counted in weak_filtered. Two separate signals say whether this is the whole answer: 'truncated' is true when more files answered the query than this page reached, and 'pool_truncated' is true when the page ended at the edge of the candidates that were fetched rather than at the end of the answer — one file whose hits fill the candidate pool leaves nothing for 'truncated' to count while later files are still waiting. Only both false means the search is complete; while either is true, make the call in 'next', which lists the exact calls that read on. Use 'path' to scope results to a module for token-efficient queries."), mcp.WithString("query", mcp.Description("Search query string"), mcp.Required()), mcp.WithNumber("limit", mcp.Description("Maximum number of files to return; every hit inside a returned file is included"), mcp.DefaultNumber(10)), mcp.WithNumber("offset", mcp.Description("Skip this many files before the page starts, so paging never splits a file"), mcp.DefaultNumber(0)), mcp.WithString("path", mcp.Description("Filter results to file paths starting with this prefix (e.g. internal/auth)")), mcp.WithBoolean("include_weak", mcp.Description("Also return candidates whose name, path, and @intent say nothing about the query; they are appended after the justified results")), + mcp.WithBoolean( + "compact", + mcp.Description( + "Return a smaller agent-oriented payload that preserves file paths, declaration bounds, "+ + "rank evidence, completion signals, and next actions while omitting redundant ids and per-hit names/paths", + ), + ), )...), Handler: h.search, }, diff --git a/internal/adapters/outbound/searchsql/backend.go b/internal/adapters/outbound/searchsql/backend.go index cbc4aa6f..a9e1f657 100644 --- a/internal/adapters/outbound/searchsql/backend.go +++ b/internal/adapters/outbound/searchsql/backend.go @@ -103,3 +103,7 @@ func loadNodesInOrder(ctx context.Context, db *gorm.DB, nodeIDs []uint) ([]graph // where a question can match more than this many reasons would be ranked on a // biased sample. const maxIntentCandidates = 10000 + +// maxNaturalCandidates is a runaway guard for soft retrieval, not a result +// limit. Ranking needs the full any-term candidate set to measure term rarity. +const maxNaturalCandidates = 10000 diff --git a/internal/adapters/outbound/searchsql/postgres.go b/internal/adapters/outbound/searchsql/postgres.go index 8a1ff86d..00c9830f 100644 --- a/internal/adapters/outbound/searchsql/postgres.go +++ b/internal/adapters/outbound/searchsql/postgres.go @@ -10,6 +10,7 @@ import ( "github.com/tae2089/trace" "github.com/tae2089/code-context-graph/internal/app/search/intentrank" + "github.com/tae2089/code-context-graph/internal/app/search/queryterm" requestctx "github.com/tae2089/code-context-graph/internal/ctx" "github.com/tae2089/code-context-graph/internal/db/migration" "github.com/tae2089/code-context-graph/internal/domain/graph" @@ -136,12 +137,13 @@ func (p *PostgresBackend) matchRows(ctx context.Context, db *gorm.DB, tsQuery, n // Query searches for related nodes using PostgreSQL tsquery. // -// Every term is required, mirroring the SQLite backend. See SQLiteBackend.Query -// for why widening to any-term was measured and rejected. +// Every meaningful term is required first, mirroring the SQLite backend. A +// sentence-shaped query widens to any-term retrieval only when that strict pass +// is empty, then uses the same Go scorer as SQLite. // // @intent Converts the user's search term into a prefix tsquery to find related nodes. // @requires limit must be greater than 0 to get meaningful results. -// @return Returns a list of nodes sorted by ts_rank. +// @return Returns strict matches by ts_rank, or soft sentence matches by shared relevance rank when strict retrieval is empty. func (p *PostgresBackend) Query(ctx context.Context, db *gorm.DB, query string, limit int) ([]graph.Node, error) { if limit <= 0 { return nil, fmt.Errorf("limit must be > 0, got %d", limit) @@ -162,6 +164,13 @@ func (p *PostgresBackend) Query(ctx context.Context, db *gorm.DB, query string, // driven by an agent quoting identifiers out of code it has already read, so // a query that matches nothing exactly is naming something that does not // exist. Returning nothing says that; returning a near neighbour would not. + if len(rows) == 0 && queryterm.IsNaturalLanguage(query) { + docs, matchErr := p.matchContent(ctx, db, query, maxNaturalCandidates) + if matchErr != nil { + return nil, matchErr + } + return rankContentCandidates(ctx, db, query, docs, limit) + } if len(rows) == 0 { return nil, nil } @@ -177,6 +186,33 @@ func (p *PostgresBackend) Query(ctx context.Context, db *gorm.DB, query string, return promoteExactNameMatch(nodes, query), nil } +// matchContent retrieves every indexed code document containing any meaningful +// term of a natural-language query and leaves relevance ordering to shared Go +// scoring. +// @intent retrieve PostgreSQL soft-match candidates without backend-specific ranking. +func (p *PostgresBackend) matchContent(ctx context.Context, db *gorm.DB, query string, maxCandidates int) ([]intentrank.Doc, error) { + if maxCandidates <= 0 { + return nil, fmt.Errorf("maxCandidates must be > 0, got %d", maxCandidates) + } + tsQuery := SanitizePostgresNaturalTSQuery(query) + if tsQuery == "" { + return nil, nil + } + + var docs []intentrank.Doc + if err := db.WithContext(ctx).Raw(` + SELECT sd.node_id, sd.content, + n.file_path, n.qualified_name, n.kind, n.namespace, n.start_line + FROM search_documents sd + JOIN nodes n ON n.id = sd.node_id + WHERE sd.tsv @@ to_tsquery('simple', ?) + AND sd.namespace = ? + LIMIT ?`, tsQuery, requestctx.FromContext(ctx), maxCandidates).Scan(&docs).Error; err != nil { + return nil, trace.Wrap(err, "natural-language ts_query") + } + return docs, nil +} + // MatchIntent finds every recorded reason holding any term of the question. // // It used to order by ts_rank, which is where the deployment gap lived: ts_rank diff --git a/internal/adapters/outbound/searchsql/retrieval.go b/internal/adapters/outbound/searchsql/retrieval.go index 33aa9047..d39b5565 100644 --- a/internal/adapters/outbound/searchsql/retrieval.go +++ b/internal/adapters/outbound/searchsql/retrieval.go @@ -39,6 +39,49 @@ func (r *Reader) Query(ctx context.Context, query string, limit int) ([]graph.No return r.backend.Query(ctx, r.db, query, limit) } +// rankContentCandidates gives natural-language retrieval one backend-neutral +// order. The SQL engines only admit documents containing any meaningful query +// term; the same BM25/IDF implementation then rewards both distinctive terms +// and coverage of more of the question. +// @intent rank soft-matched general search documents identically on SQLite and PostgreSQL. +func rankContentCandidates(ctx context.Context, db *gorm.DB, query string, docs []intentrank.Doc, limit int) ([]graph.Node, error) { + if len(docs) == 0 { + return nil, nil + } + var corpusSize int64 + if err := db.WithContext(ctx).Model(&graph.SearchDocument{}). + Where("namespace = ?", requestctx.FromContext(ctx)). + Count(&corpusSize).Error; err != nil { + return nil, trace.Wrap(err, "count indexed search documents") + } + ranked := intentrank.Rank(query, docs, int(corpusSize), len(docs)) + nodeIDs := make([]uint, 0, min(limit, len(ranked.Matches))) + for _, match := range ranked.Matches { + // OR admits candidates; it does not make a single coincidental word an + // answer to a sentence. Requiring two distinct content terms preserves + // the useful widening while dropping common one-word neighbours. + if distinctTermCount(match.Terms) < 2 { + continue + } + nodeIDs = append(nodeIDs, match.NodeID) + if len(nodeIDs) >= limit { + break + } + } + return loadNodesInOrder(ctx, db, nodeIDs) +} + +// distinctTermCount prevents a repeated query word from satisfying the +// multi-signal guard by itself. +// @intent count independent soft-match signals rather than repeated wording. +func distinctTermCount(terms []string) int { + seen := make(map[string]struct{}, len(terms)) + for _, term := range terms { + seen[term] = struct{}{} + } + return len(seen) +} + // QueryIntent answers a question from the recorded-reason index only. // // The database finds the candidates and Go ranks them. That split is what makes diff --git a/internal/adapters/outbound/searchsql/retrieval_test.go b/internal/adapters/outbound/searchsql/retrieval_test.go new file mode 100644 index 00000000..b9bd3078 --- /dev/null +++ b/internal/adapters/outbound/searchsql/retrieval_test.go @@ -0,0 +1,130 @@ +//go:build fts5 + +package searchsql + +import ( + "context" + "testing" + + searchapp "github.com/tae2089/code-context-graph/internal/app/search" + "github.com/tae2089/code-context-graph/internal/domain/graph" +) + +func TestReaderQuery_NaturalLanguageUsesSoftContentRanking(t *testing.T) { + db := setupTestDB(t) + backend := NewSQLiteBackend() + if err := backend.Migrate(db); err != nil { + t.Fatal(err) + } + + nodes := []graph.Node{ + {Name: "SyncQueue", QualifiedName: "reposync.SyncQueue", Kind: graph.NodeKindType, FilePath: "internal/app/reposync/queue.go", Language: "go"}, + {Name: "WebhookHandler", QualifiedName: "webhook.WebhookHandler", Kind: graph.NodeKindType, FilePath: "internal/adapters/inbound/webhook/handler.go", Language: "go"}, + {Name: "GraphStats", QualifiedName: "status.GraphStats", Kind: graph.NodeKindType, FilePath: "internal/status/graph.go", Language: "go"}, + } + for i := range nodes { + if err := db.Create(&nodes[i]).Error; err != nil { + t.Fatal(err) + } + } + + docs := []graph.SearchDocument{ + {NodeID: nodes[0].ID, Content: "SyncQueue coalesces webhook push graph updates into a background job", Language: "go"}, + {NodeID: nodes[1].ID, Content: "WebhookHandler receives a GitHub push webhook", Language: "go"}, + {NodeID: nodes[2].ID, Content: "GraphStats reports graph state", Language: "go"}, + } + for i := range docs { + if err := db.Create(&docs[i]).Error; err != nil { + t.Fatal(err) + } + } + if err := backend.Rebuild(context.Background(), db); err != nil { + t.Fatal(err) + } + + reader := NewReader(db, backend) + query := "why are graph updates processed as a separate background job instead of immediately when a GitHub push webhook is received" + got, err := reader.Query(context.Background(), query, 10) + if err != nil { + t.Fatal(err) + } + if len(got) == 0 { + t.Fatal("natural-language query returned no candidates") + } + if got[0].QualifiedName != "reposync.SyncQueue" { + t.Fatalf("first candidate = %q, want reposync.SyncQueue", got[0].QualifiedName) + } +} + +func TestSearchService_NaturalLanguageReturnsJustifiedSoftCandidate(t *testing.T) { + db := setupTestDB(t) + sqlDB, err := db.DB() + if err != nil { + t.Fatal(err) + } + sqlDB.SetMaxOpenConns(1) + backend := NewSQLiteBackend() + if err := backend.Migrate(db); err != nil { + t.Fatal(err) + } + + node := graph.Node{ + Name: "SyncQueue", QualifiedName: "reposync.SyncQueue", Kind: graph.NodeKindType, + FilePath: "internal/app/reposync/queue.go", Language: "go", + } + if err := db.Create(&node).Error; err != nil { + t.Fatal(err) + } + annotation := graph.Annotation{NodeID: node.ID, Tags: []graph.DocTag{{ + Kind: graph.TagIntent, Value: "coalesce webhook push graph updates into a background job", + }}} + if err := db.Create(&annotation).Error; err != nil { + t.Fatal(err) + } + if err := db.Create(&graph.SearchDocument{ + NodeID: node.ID, Content: "SyncQueue coalesces webhook push graph updates into a background job", Language: "go", + }).Error; err != nil { + t.Fatal(err) + } + if err := backend.Rebuild(context.Background(), db); err != nil { + t.Fatal(err) + } + + query := "why are graph updates processed as a separate background job instead of immediately when a GitHub push webhook is received" + list, err := searchapp.New(NewReader(db, backend)).Search(context.Background(), searchapp.Params{Query: query, Limit: 5}) + if err != nil { + t.Fatal(err) + } + if len(list.Files) != 1 || len(list.Files[0].Hits) != 1 { + t.Fatalf("files = %#v, want one justified SyncQueue hit", list.Files) + } + if got := list.Files[0].Hits[0]; got.Node.QualifiedName != "reposync.SyncQueue" || len(got.Matched) == 0 { + t.Fatalf("hit = %#v, want justified reposync.SyncQueue", got) + } +} + +func TestReaderQuery_ShortMultiTermQueryRemainsStrict(t *testing.T) { + db := setupTestDB(t) + seedNodes(t, db) + backend := NewSQLiteBackend() + if err := backend.Migrate(db); err != nil { + t.Fatal(err) + } + if err := backend.Rebuild(context.Background(), db); err != nil { + t.Fatal(err) + } + + got, err := NewReader(db, backend).Query(context.Background(), "session credentials", 10) + if err != nil { + t.Fatal(err) + } + if len(got) != 0 { + t.Fatalf("strict query returned %d candidates, want none", len(got)) + } +} + +func TestDistinctTermCount_DoesNotCountRepeatedWordsTwice(t *testing.T) { + if got := distinctTermCount([]string{"graph", "graph", "updates"}); got != 2 { + t.Fatalf("distinctTermCount = %d, want 2", got) + } +} diff --git a/internal/adapters/outbound/searchsql/sanitize.go b/internal/adapters/outbound/searchsql/sanitize.go index a7fd9ceb..6b1f22c2 100644 --- a/internal/adapters/outbound/searchsql/sanitize.go +++ b/internal/adapters/outbound/searchsql/sanitize.go @@ -16,30 +16,28 @@ import ( // @intent build SQLite FTS queries that preserve prefix matching without exposing parser-breaking characters. // @domainRule empty or fully stripped input returns an empty query string. func SanitizeFTS5(query string) string { - return buildPrefixQuery(query, alwaysPrefix(`"`+"%s"+`"*`), " AND ", " OR ", " ") + return buildPrefixQuery(query, alwaysPrefix(`"`+"%s"+`"*`), " AND ", " OR ", " AND ") } -// SanitizeIntentFTS5 converts a question into an any-term FTS5 prefix query for -// the intent index. +// SanitizeNaturalFTS5 converts prose into an any-term FTS5 query. // // SanitizeFTS5 requires every term because the searcher there typed identifiers, // and an identifier a caller half-remembers is still worth demanding in full. -// An intent question is the opposite: it is a sentence aimed at another sentence, -// and no recorded reason will contain all of "why do we verify the signature on a -// push". Requiring every term would answer almost nothing. -// -// Any-term is only safe here because this index holds nothing but recorded -// reasons. The same widening was measured on the shared index and rejected: there -// a common word could match an identifier, a path segment, or a language alias, -// so widening pulled in fifty unrelated nodes. With the names removed, a term can -// only match prose somebody wrote on purpose, and bm25 discounts the words that -// appear in many of those. -// @intent let a sentence-shaped question match a sentence-shaped reason. -// @domainRule any-term matching is confined to the intent index, which holds no identifier text. -func SanitizeIntentFTS5(query string) string { +// Prose is the opposite: no useful code document will contain every word of +// "why do we verify the signature on a push". Stopword removal, any-term +// retrieval, and shared BM25/IDF scoring together let distinctive terms win +// without requiring every term to occur in one document. +// @intent retrieve candidates for a sentence-shaped query without requiring every content word in one document. +func SanitizeNaturalFTS5(query string) string { return buildPrefixQuery(query, intentTerm(`"`+"%s"+`"*`, `"`+"%s"+`"`), " AND ", " OR ", " OR ") } +// SanitizeIntentFTS5 applies the natural-language query shape to recorded reasons. +// @intent keep intent retrieval on the same soft-matching syntax as general prose retrieval. +func SanitizeIntentFTS5(query string) string { + return SanitizeNaturalFTS5(query) +} + // SanitizePostgresTSQuery converts raw user input into a safe prefix tsquery, // mirroring SanitizeFTS5 including camelCase sub-token expansion. // @intent translate free-form user input into a PostgreSQL tsquery that mirrors the SQLite prefix search behavior. @@ -48,31 +46,32 @@ func SanitizePostgresTSQuery(query string) string { return buildPrefixQuery(query, alwaysPrefix("%s:*"), " & ", " | ", " & ") } -// SanitizePostgresIntentTSQuery is the PostgreSQL twin of SanitizeIntentFTS5: -// any term may match, because no recorded reason contains every word of a -// question. See SanitizeIntentFTS5 for why that widening is confined to the -// intent index. -// @intent let a sentence-shaped question match a sentence-shaped reason on PostgreSQL. -// @domainRule any-term matching is confined to the intent index, which holds no identifier text. -func SanitizePostgresIntentTSQuery(query string) string { +// SanitizePostgresNaturalTSQuery is the PostgreSQL twin of SanitizeNaturalFTS5. +// @intent retrieve PostgreSQL candidates for prose with any-term matching. +func SanitizePostgresNaturalTSQuery(query string) string { return buildPrefixQuery(query, intentTerm("%s:*", "%s"), " & ", " | ", " | ") } +// SanitizePostgresIntentTSQuery applies the natural-language query shape to recorded reasons. +// @intent keep PostgreSQL intent retrieval on the same soft-matching syntax as general prose retrieval. +func SanitizePostgresIntentTSQuery(query string) string { + return SanitizePostgresNaturalTSQuery(query) +} + // alwaysPrefix renders every term as a prefix, whatever it looks like. // @intent keep prefix expansion the default for the shared search index. func alwaysPrefix(termFmt string) func(string) string { return func(tok string) string { return strings.Replace(termFmt, "%s", tok, 1) } } -// intentTerm renders one term of an intent question, choosing prefix or exact +// intentTerm renders one term of a natural-language question, choosing prefix or exact // matching by intentrank.MatchesByPrefix. // // The rule lives with the scorer rather than here because both have to apply the // same one: the index decides what a term matches, the scorer decides what that // match is worth, and a term matched one way and scored the other would order // the answer by evidence from a query that never ran. -// @intent keep a short question word from reaching an identifier spelled inside a recorded reason. -// @domainRule only the intent index narrows a term to an exact match. +// @intent keep a short question word from reaching a longer identifier only because it shares a prefix. func intentTerm(prefixFmt, exactFmt string) func(string) string { return func(tok string) string { if intentrank.MatchesByPrefix(tok) { diff --git a/internal/adapters/outbound/searchsql/sanitize_test.go b/internal/adapters/outbound/searchsql/sanitize_test.go index 4f964596..aa9aae41 100644 --- a/internal/adapters/outbound/searchsql/sanitize_test.go +++ b/internal/adapters/outbound/searchsql/sanitize_test.go @@ -13,7 +13,7 @@ func TestSanitizeFTS5_SplitsCamelCaseTokens(t *testing.T) { }{ {query: "", want: ""}, {query: "user", want: `"user"*`}, // 단일 단어 불변 - {query: "get user", want: `"get"* "user"*`}, // 소문자 멀티토큰 불변 + {query: "get user", want: `"get"* AND "user"*`}, // 소문자 멀티토큰 불변 {query: "getUser", want: `("getuser"* OR ("get"* AND "user"*))`}, // camelCase 분할 {query: "UserService", want: `("userservice"* OR ("user"* AND "service"*))`}, } @@ -51,19 +51,19 @@ func TestSanitize_DropsFunctionWords(t *testing.T) { { name: "a question keeps only its content words", query: "what stops the server", - fts5: `"stops"* "server"*`, + fts5: `"stops"* AND "server"*`, pg: "stops:* & server:*", }, { name: "a query made only of function words keeps them", query: "how does the", - fts5: `"how"* "does"* "the"*`, + fts5: `"how"* AND "does"* AND "the"*`, pg: "how:* & does:* & the:*", }, { name: "code words that read like function words survive", query: "get set list new", - fts5: `"get"* "set"* "list"* "new"*`, + fts5: `"get"* AND "set"* AND "list"* AND "new"*`, pg: "get:* & set:* & list:* & new:*", }, } @@ -129,11 +129,21 @@ func TestSanitizeIntent_KeepsShortLatinTermsExact(t *testing.T) { } } +func TestSanitizeNatural_UsesAnyMeaningfulTerm(t *testing.T) { + query := "why are graph updates processed as a background job" + if got, want := SanitizeNaturalFTS5(query), `"graph"* OR "updates"* OR "processed"* OR "background"* OR "job"`; got != want { + t.Errorf("SanitizeNaturalFTS5 = %q, want %q", got, want) + } + if got, want := SanitizePostgresNaturalTSQuery(query), "graph:* | updates:* | processed:* | background:* | job"; got != want { + t.Errorf("SanitizePostgresNaturalTSQuery = %q, want %q", got, want) + } +} + // The shared index keeps prefix expansion for every term. A caller there typed a // symbol out of code it already read, so `get` reaching getAnnotation is the // answer rather than the mistake. func TestSanitizeSearch_KeepsShortTermsAsPrefixes(t *testing.T) { - if got, want := SanitizeFTS5("lock get keeps"), `"lock"* "get"* "keeps"*`; got != want { + if got, want := SanitizeFTS5("lock get keeps"), `"lock"* AND "get"* AND "keeps"*`; got != want { t.Errorf("SanitizeFTS5 = %q, want %q", got, want) } if got, want := SanitizePostgresTSQuery("lock get keeps"), "lock:* & get:* & keeps:*"; got != want { diff --git a/internal/adapters/outbound/searchsql/sqlite.go b/internal/adapters/outbound/searchsql/sqlite.go index b7f0435b..9e6ee7ad 100644 --- a/internal/adapters/outbound/searchsql/sqlite.go +++ b/internal/adapters/outbound/searchsql/sqlite.go @@ -13,6 +13,7 @@ import ( "github.com/tae2089/trace" "github.com/tae2089/code-context-graph/internal/app/search/intentrank" + "github.com/tae2089/code-context-graph/internal/app/search/queryterm" requestctx "github.com/tae2089/code-context-graph/internal/ctx" "github.com/tae2089/code-context-graph/internal/domain/graph" ) @@ -318,19 +319,15 @@ func (s *SQLiteBackend) matchRows(ctx context.Context, db *gorm.DB, ftsQuery, ns // Query searches for related nodes using FTS5 MATCH queries. // -// Every term is required, and SanitizeFTS5 decides what counts as a term. That -// pairing is the whole retrieval policy: requiring all terms is right when the -// searcher typed identifiers, and it only became wrong for sentences because -// ordinary English words were being required too. -// -// Widening to any-term when all-terms matches nothing was measured and -// rejected. It answered no query the narrow expression missed — the two extra -// nodes it retrieved never reached the top ten — and it filled the deliberate -// nonsense query in the golden set with fifty unrelated hits. +// Every meaningful term is required first. That preserves the precision of +// identifier queries and sentence queries whose content words really do meet +// in one document. Only a sentence-shaped query with no strict hit widens to +// any-term retrieval and shared BM25/IDF scoring; compact identifier queries +// never widen. // // @intent Converts the user's search term into a SQLite FTS prefix query to find nodes. // @requires limit must be greater than 0 to get meaningful results. -// @return Returns a list of nodes sorted by FTS rank. +// @return Returns strict matches by FTS rank, or soft sentence matches by shared relevance rank when strict retrieval is empty. func (s *SQLiteBackend) Query(ctx context.Context, db *gorm.DB, query string, limit int) ([]graph.Node, error) { if limit <= 0 { return nil, fmt.Errorf("limit must be > 0, got %d", limit) @@ -345,6 +342,13 @@ func (s *SQLiteBackend) Query(ctx context.Context, db *gorm.DB, query string, li if err != nil { return nil, err } + if len(rows) == 0 && queryterm.IsNaturalLanguage(query) { + docs, matchErr := s.matchContent(ctx, db, query, maxNaturalCandidates) + if matchErr != nil { + return nil, matchErr + } + return rankContentCandidates(ctx, db, query, docs, limit) + } if len(rows) == 0 { return nil, nil } @@ -360,6 +364,32 @@ func (s *SQLiteBackend) Query(ctx context.Context, db *gorm.DB, query string, li return promoteExactNameMatch(nodes, query), nil } +// matchContent retrieves every indexed code document containing any meaningful +// term of a natural-language query. It deliberately leaves ordering to shared +// Go scoring so SQLite and PostgreSQL cannot disagree about relevance. +// @intent retrieve soft-matched general search candidates for backend-neutral ranking. +func (s *SQLiteBackend) matchContent(ctx context.Context, db *gorm.DB, query string, maxCandidates int) ([]intentrank.Doc, error) { + if maxCandidates <= 0 { + return nil, fmt.Errorf("maxCandidates must be > 0, got %d", maxCandidates) + } + ftsQuery := SanitizeNaturalFTS5(query) + if ftsQuery == "" { + return nil, nil + } + + var docs []intentrank.Doc + if err := db.WithContext(ctx).Raw( + `SELECT CAST(search_fts.node_id AS INTEGER) AS node_id, search_fts.content, + n.file_path, n.qualified_name, n.kind, n.namespace, n.start_line + FROM search_fts + JOIN nodes n ON n.id = CAST(search_fts.node_id AS INTEGER) + WHERE search_fts MATCH ? AND search_fts.namespace = ? + LIMIT ?`, ftsQuery, requestctx.FromContext(ctx), maxCandidates).Scan(&docs).Error; err != nil { + return nil, trace.Wrap(err, "natural-language fts query") + } + return docs, nil +} + // MatchIntent finds every recorded reason holding any term of the question. // // FTS5 could order these by bm25 and used to, but the ordering moved to diff --git a/internal/app/search/queryterm/queryterm.go b/internal/app/search/queryterm/queryterm.go index 3e2f01b6..a6811d55 100644 --- a/internal/app/search/queryterm/queryterm.go +++ b/internal/app/search/queryterm/queryterm.go @@ -5,7 +5,11 @@ // the searcher asked for. package queryterm -import "strings" +import ( + "strings" + + "github.com/tae2089/code-context-graph/internal/app/search/identtoken" +) // functionWords are English words that carry no meaning in a corpus of code // identifiers and one-line annotations. They hurt both searches, in opposite @@ -34,6 +38,15 @@ var functionWords = map[string]bool{ "would": true, "should": true, "will": true, "shall": true, "there": true, } +// questionWords identify queries whose grammar says they are prose even when +// only a few content words survive stopword removal. They are kept separate +// from functionWords because the two sets answer different questions: one +// chooses a retrieval shape, while the other chooses terms worth matching. +var questionWords = map[string]bool{ + "how": true, "what": true, "where": true, "when": true, + "why": true, "who": true, "which": true, +} + // IsFunctionWord reports whether a term is one of the meaningless words. // @intent let a caller judge one term without copying the list. func IsFunctionWord(term string) bool { @@ -57,3 +70,24 @@ func DropFunctionWords(tokens []string) []string { } return kept } + +// IsNaturalLanguage reports whether a query should retrieve documents by any +// meaningful term and let shared scoring combine the evidence. +// +// Two or three code terms are normally a partially remembered identifier and +// retain strict all-term matching. Prose either announces itself with a +// question word or contains more than three meaningful terms. The +// count happens after function-word removal so "get user by id" remains the +// compact identifier-shaped query "get user id". +// @intent choose soft retrieval only for sentence-shaped queries while preserving precise identifier lookup. +// @domainRule compact queries of at most three meaningful terms remain strict unless their grammar explicitly asks a question. +func IsNaturalLanguage(query string) bool { + tokens := identtoken.FieldsLower(query) + meaningful := DropFunctionWords(tokens) + for _, token := range tokens { + if questionWords[token] && len(meaningful) >= 2 { + return true + } + } + return len(meaningful) > 3 +} diff --git a/internal/app/search/queryterm/queryterm_test.go b/internal/app/search/queryterm/queryterm_test.go new file mode 100644 index 00000000..e8bd3419 --- /dev/null +++ b/internal/app/search/queryterm/queryterm_test.go @@ -0,0 +1,27 @@ +package queryterm + +import "testing" + +func TestIsNaturalLanguage(t *testing.T) { + tests := []struct { + name string + query string + want bool + }{ + {name: "question marker", query: "why graph updates", want: true}, + {name: "explanation with question marker", query: "explain why graph updates", want: true}, + {name: "long descriptive phrase", query: "graph updates use a background job", want: true}, + {name: "three code terms", query: "session token credentials", want: false}, + {name: "function word does not widen", query: "get user by id", want: false}, + {name: "single identifier", query: "buildOrUpdateGraph", want: false}, + {name: "ambiguous command name", query: "explain", want: false}, + } + + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + if got := IsNaturalLanguage(tt.query); got != tt.want { + t.Fatalf("IsNaturalLanguage(%q) = %v, want %v", tt.query, got, tt.want) + } + }) + } +} diff --git a/internal/app/search/service.go b/internal/app/search/service.go index 5bd1b6a5..8eccc94b 100644 --- a/internal/app/search/service.go +++ b/internal/app/search/service.go @@ -8,11 +8,13 @@ package search import ( "context" + "sort" "github.com/tae2089/trace" "github.com/tae2089/code-context-graph/internal/app/search/evidence" intentapp "github.com/tae2089/code-context-graph/internal/app/search/intent" + "github.com/tae2089/code-context-graph/internal/app/search/queryterm" searchrank "github.com/tae2089/code-context-graph/internal/app/search/rank" requestctx "github.com/tae2089/code-context-graph/internal/ctx" "github.com/tae2089/code-context-graph/internal/domain/graph" @@ -232,6 +234,9 @@ func (s *Service) fetch(ctx context.Context, p Params) (pool, error) { // @ensures the order of the first n rows does not depend on any row after them, at every n that is a multiple of block. // @intent keep a page already delivered from being reshuffled by the wider pool the next page fetches. func orderPool(query string, nodes []graph.Node, block int) []graph.Node { + if queryterm.IsNaturalLanguage(query) { + return nodes + } if block <= 0 || len(nodes) <= block { return searchrank.Rerank(query, nodes, 0) } @@ -255,6 +260,9 @@ func orderPool(query string, nodes []graph.Node, block int) []graph.Node { // @requires each group is that repository's rank-ordered pool, fetched in whole multiples of block. // @intent give federated paging the same fixed prefix a single repository's paging has. func orderGroupedPool(query string, groups [][]graph.Node, block int) []graph.Node { + if queryterm.IsNaturalLanguage(query) { + return mergeRankedGroups(groups) + } widest, total := 0, 0 for _, g := range groups { widest = max(widest, len(g)) @@ -277,6 +285,33 @@ func orderGroupedPool(query string, groups [][]graph.Node, block int) []graph.No return out } +// mergeRankedGroups combines backend-neutral natural-language rankings without +// replacing them with identifier-oriented structural scores. Items at the same +// rank across repositories use the canonical node identity as their stable tie +// break, so changing namespace input order cannot change the answer. +// @intent preserve soft-match relevance while merging independently ranked namespace results. +func mergeRankedGroups(groups [][]graph.Node) []graph.Node { + widest, total := 0, 0 + for _, group := range groups { + widest = max(widest, len(group)) + total += len(group) + } + out := make([]graph.Node, 0, total) + for position := range widest { + level := make([]graph.Node, 0, len(groups)) + for _, group := range groups { + if position < len(group) { + level = append(level, group[position]) + } + } + sort.SliceStable(level, func(i, j int) bool { + return graph.CompareIdentity(level[i].Identity(), level[j].Identity()) < 0 + }) + out = append(out, level...) + } + return out +} + // keepPathPrefix drops the candidates that live outside the caller's path // filter, once the answer's order is already decided. // diff --git a/internal/app/search/service_test.go b/internal/app/search/service_test.go index 6fa367c4..169d8b0b 100644 --- a/internal/app/search/service_test.go +++ b/internal/app/search/service_test.go @@ -66,6 +66,39 @@ func node(id uint, name, path string) graph.Node { return graph.Node{ID: id, Name: name, QualifiedName: name, FilePath: path, Kind: "function"} } +func TestOrderPool_NaturalLanguagePreservesSoftMatchRank(t *testing.T) { + query := "why are graph updates processed as a separate background job" + softRanked := []graph.Node{ + node(1, "SyncQueue", "internal/app/reposync/queue.go"), + node(2, "GraphUpdates", "internal/graph/updates.go"), + } + + got := orderPool(query, softRanked, 10) + if got[0].ID != 1 { + t.Fatalf("first node = %d, want soft-ranked node 1", got[0].ID) + } +} + +func TestMergeRankedGroups_NaturalLanguagePreservesRanksAndStableTies(t *testing.T) { + query := "explain graph update background processing" + alpha := []graph.Node{ + node(1, "A1", "z/a1.go"), + node(2, "A2", "a/a2.go"), + } + beta := []graph.Node{ + node(3, "B1", "a/b1.go"), + node(4, "B2", "z/b2.go"), + } + + got := orderGroupedPool(query, [][]graph.Node{alpha, beta}, 10) + want := []uint{3, 1, 2, 4} + for i, id := range want { + if got[i].ID != id { + t.Fatalf("result[%d] = %d, want %d", i, got[i].ID, id) + } + } +} + func TestSearch_OverfetchesThenCutsToLimit(t *testing.T) { pool := []graph.Node{ node(1, "alpha", "a/alpha.go"), diff --git a/internal/app/search/wire/wire.go b/internal/app/search/wire/wire.go index 5f10cd34..171fa862 100644 --- a/internal/app/search/wire/wire.go +++ b/internal/app/search/wire/wire.go @@ -92,6 +92,99 @@ type Response struct { Note string `json:"note,omitempty"` } +// CompactResultItem keeps the evidence an agent needs to choose and open a hit, +// without repeating data already present in its file group. +// @intent reduce search-response context while preserving rank evidence and exact source bounds. +type CompactResultItem struct { + QualifiedName string `json:"qualified_name"` + Kind graph.NodeKind `json:"kind"` + StartLine int `json:"start_line"` + EndLine int `json:"end_line"` + Intent string `json:"intent,omitempty"` + Matched []evidence.Match `json:"matched,omitempty"` + Reason string `json:"reason,omitempty"` + MatchedTerms []string `json:"matched_terms,omitempty"` +} + +// CompactFileGroup groups compact hits under the one copy of their file path. +// @intent avoid repeating file identity on every hit while keeping federated namespace labels. +type CompactFileGroup struct { + FilePath string `json:"file_path"` + Namespace string `json:"namespace,omitempty"` + Hits []CompactResultItem `json:"hits"` +} + +// CompactResponse is the token-efficient search view used by agents that do +// not need storage ids or redundant per-hit names and paths. +// @intent preserve search decisions, completion signals, and continuations in a smaller wire payload. +type CompactResponse struct { + Files []CompactFileGroup `json:"files"` + WeakFiltered int `json:"weak_filtered"` + Truncated bool `json:"truncated"` + PoolTruncated bool `json:"pool_truncated"` + Limits Limits `json:"limits"` + AnnotationCoverage evidence.Coverage `json:"annotation_coverage"` + Next []NextAction `json:"next,omitempty"` + Note string `json:"note,omitempty"` +} + +// Compact returns a smaller view without changing the full response contract. +// Search continuations retain compact mode so later pages do not grow again. +// @ensures every search action in Next carries compact=true. +// @intent let CLI and MCP share one loss-aware compact representation. +func (r Response) Compact() CompactResponse { + files := make([]CompactFileGroup, len(r.Files)) + for i, file := range r.Files { + hits := make([]CompactResultItem, len(file.Hits)) + for j, hit := range file.Hits { + intent := hit.Intent + if intent == hit.Reason { + intent = "" + } + hits[j] = CompactResultItem{ + QualifiedName: hit.QualifiedName, + Kind: hit.Kind, + StartLine: hit.StartLine, + EndLine: hit.EndLine, + Intent: intent, + Matched: hit.Matched, + Reason: hit.Reason, + MatchedTerms: hit.MatchedTerms, + } + } + files[i] = CompactFileGroup{ + FilePath: file.FilePath, + Namespace: file.Namespace, + Hits: hits, + } + } + + next := make([]NextAction, len(r.Next)) + for i, action := range r.Next { + next[i] = action + if action.Tool != "search" { + continue + } + args := make(map[string]any, len(action.Args)+1) + for key, value := range action.Args { + args[key] = value + } + args["compact"] = true + next[i].Args = args + } + + return CompactResponse{ + Files: files, + WeakFiltered: r.WeakFiltered, + Truncated: r.Truncated, + PoolTruncated: r.PoolTruncated, + Limits: r.Limits, + AnnotationCoverage: r.AnnotationCoverage, + Next: next, + Note: r.Note, + } +} + // Limits states the bounds that shaped this page, all counted in files // except the budget, which only decides whether one more file joins. // @intent let a caller tell a short answer from the first page of a long one. diff --git a/internal/app/search/wire/wire_test.go b/internal/app/search/wire/wire_test.go index 6e72d92f..641c4c2a 100644 --- a/internal/app/search/wire/wire_test.go +++ b/internal/app/search/wire/wire_test.go @@ -1,6 +1,7 @@ package wire import ( + "encoding/json" "os" "path/filepath" "runtime" @@ -12,6 +13,78 @@ import ( "github.com/tae2089/code-context-graph/internal/domain/graph" ) +func TestResponse_CompactKeepsDecisionEvidenceWithoutDuplicateFields(t *testing.T) { + list := evidence.List{ + Files: []evidence.File{{ + Namespace: "repo-a", + FilePath: "internal/app/search/service.go", + Hits: []evidence.Result{{ + Node: graph.Node{ + ID: 41, + QualifiedName: "search.Service.Search", + Kind: graph.NodeKindFunction, + Name: "Search", + FilePath: "internal/app/search/service.go", + StartLine: 73, + EndLine: 121, + }, + Intent: "rank precise matches before widening a natural-language question", + Matched: []evidence.Match{evidence.MatchName, evidence.MatchIntent}, + Reason: "widen a natural-language question only after the precise match is empty", + MatchedTerms: []string{"natural", "question"}, + }}, + }}, + OverflowFiles: 2, + NextOffset: 1, + Coverage: evidence.Coverage{WithReason: 12, Declarations: 40}, + } + seed := list.Files[0].Hits[0] + list.Files[0].Hits = append(list.Files[0].Hits, seed, seed, seed, seed) + + full := NewResponse(list, "natural question", 1, 0, true) + compact := full.Compact() + raw, err := json.Marshal(compact) + if err != nil { + t.Fatal(err) + } + + var payload map[string]any + if err := json.Unmarshal(raw, &payload); err != nil { + t.Fatal(err) + } + files := payload["files"].([]any) + file := files[0].(map[string]any) + if file["namespace"] != "repo-a" { + t.Errorf("compact file namespace = %v, want repo-a", file["namespace"]) + } + hit := file["hits"].([]any)[0].(map[string]any) + for _, key := range []string{"qualified_name", "kind", "start_line", "end_line", "intent", "matched", "reason", "matched_terms"} { + if _, ok := hit[key]; !ok { + t.Errorf("compact hit is missing %q: %s", key, raw) + } + } + for _, duplicate := range []string{"id", "name", "file_path", "namespace"} { + if _, ok := hit[duplicate]; ok { + t.Errorf("compact hit kept duplicate field %q: %s", duplicate, raw) + } + } + if payload["truncated"] != true || payload["annotation_coverage"] == nil { + t.Errorf("compact response lost completion metadata: %s", raw) + } + next := payload["next"].([]any)[0].(map[string]any) + args := next["args"].(map[string]any) + if args["compact"] != true { + t.Errorf("next args = %v, want compact=true so paging stays compact", args) + } + fullRaw, err := json.Marshal(full) + if err != nil { + t.Fatal(err) + } + if len(raw)*10 >= len(fullRaw)*9 { + t.Errorf("compact response is %d bytes versus %d full bytes, want at least 10%% smaller", len(raw), len(fullRaw)) + } +} + func file(path string, hits int) evidence.File { f := evidence.File{FilePath: path} for i := range hits { diff --git a/skills/ccg-build/SKILL.md b/skills/ccg-build/SKILL.md index c7742793..e649f36c 100644 --- a/skills/ccg-build/SKILL.md +++ b/skills/ccg-build/SKILL.md @@ -54,8 +54,8 @@ ccg migrate # Existing database schema upgrade when required Over MCP: -- `build_or_update_graph(full_rebuild=false)` requests an incremental update. -- `build_or_update_graph(full_rebuild=true)` requests a full rebuild. +- Use `build_or_update_graph` with `full_rebuild=false` for an incremental update. +- Use `build_or_update_graph` with `full_rebuild=true` for a full rebuild. - `parse_project` writes parsed graph state without search postprocessing. - `run_postprocess` refreshes selected derived artifacts after graph changes. diff --git a/skills/ccg/SKILL.md b/skills/ccg/SKILL.md index d9655c36..1b779b42 100644 --- a/skills/ccg/SKILL.md +++ b/skills/ccg/SKILL.md @@ -2,7 +2,7 @@ name: ccg description: "Fast read-only code discovery with a bounded code-context-graph search and targeted source verification. Use when an ordinary positive lookup or explanation needs an entry point, recorded intent, known-path inventory, or direct relationship evidence. Do not use for absence, completeness, exhaustive inventory, deep flow or impact analysis, or graph writes; use ccg-search-verify for defensible negative claims, and require explicit invocation for ccg-analyze or ccg-build." metadata: - version: 3.0.0 + version: 4.9.0 openclaw: category: "code-intelligence" domain: "core" @@ -14,97 +14,120 @@ metadata: # ccg — Fast Search -Use CCG to find one credible entry point, then verify it in current source. Keep -ordinary discovery small enough that it beats broad grep exploration. +Find the smallest source-verified evidence set that answers an ordinary positive +code question. -## Task Routing and Entry +## Route -| User intent | Start with | +| Request | Route | | --- | --- | -| Known filename, identifier, literal, or error text | Grep + Read | -| Unknown entry point, recorded intent, or keyword | One bounded `search` | -| Contents of a known file or folder | `describe` | -| One direct caller/callee fact | One bounded `query_graph` lookup | -| Absence, completeness, or exhaustive inventory | `ccg-search-verify` skill if available | -| Deep flow, change impact, or blast radius | Stop and report that the user must explicitly invoke `ccg-analyze` | -| Generated documentation | `ccg-docs` skill if available | -| Write or repair annotations | `ccg-annotate` skill if available | -| Multiple repositories or services | `ccg-namespace` skill if available | -| Build, update, migrate, postprocess, or scoped graph write | Stop and report that the user must explicitly invoke `ccg-build` | - -An ordinary miss is not permission to make a negative claim. When the answer -would become “not found,” “does not exist,” “complete,” or “all,” switch to -`ccg-search-verify` instead of widening this workflow. - -Do not invoke `ccg-analyze` automatically. A request about algorithms, -pipelines, impact, callers, or flow does not itself authorize its traversal and -impact-analysis procedure. - -## Fast Workflow - -1. Reuse a namespace already supplied by repository instructions, configuration, - or the user. Do not call `get_minimal_context`, `list_namespaces`, or - `list_graph_stats` for an ordinary search when that information is already - known. -2. With an exact clue, use grep/read. Otherwise call `search` at most once with - `limit: 5`: - - known thing: one short identifier or rare keyword; - - unknown symbol: one concise plain-language question that can match recorded - `@intent` or `@domainRule` reasons. -3. Verify the best credible candidate in one or two source ranges. Stop when - those ranges answer the question. -4. Use `get_node`, `describe`, or one bounded `query_graph` call only when the - answer needs exact identity, an unranked known-path inventory, or one direct - relationship fact. - -Every query word must occur in the same indexed document, so do not concatenate -the prompt’s examples into a long query. If the first query has no credible hit, -do not fan out into synonyms inside this fast workflow. - -```bash -ccg search --limit 5 "" -ccg status # population only; not proof of freshness -``` - -Use `ccg --help` rather than relying on remembered flags. - -## Freshness Boundary - -Current source is authoritative for text, branches, runtime semantics, and -location. CCG supplies indexed intent and relationship candidates. - -Positive evidence verified in current source can answer an ordinary question -without a separate freshness preflight. A graph miss, stale result, or missing -namespace cannot support a negative claim; route that task to -`ccg-search-verify`. Call `list_graph_stats` only when the user asks about graph -population or graph state itself. - -This skill is read-only. Report a missing or stale graph instead of invoking -`ccg-build`. The user must explicitly name `ccg-build` before any graph write. - -## Response Budget Rule - -- One CCG `search` at most, with `limit: 5`. -- Verify no more than one or two source ranges unless the selected source itself - points to one necessary continuation. -- Do not page an ordinary ranked answer. Disclose truncation only when it limits - the answer. -- Do not echo raw result lists or mandatory operational reports. Return the - answer and its relevant paths or symbols. -- Read [`references/supported-languages.md`](references/supported-languages.md) - only for a supported-language or extension question. - -## Boundary - -- `search` returns ranked candidates, not proof of absence or completeness. -- Reserve impact-radius and flow-tracing workflows for an explicit +| Source path already known | Targeted Grep + Read | +| Source path unknown, including an exact identifier | Core Loop | +| Known file or folder inventory | One `describe` call | +| One direct caller/callee fact | One bounded `query_graph` call | +| Absence, completeness, or exhaustive inventory | `ccg-search-verify` | +| Deep flow or impact analysis | Require explicit `ccg-analyze` invocation | +| Graph write or refresh | Require explicit `ccg-build` invocation | + +An ordinary miss is not evidence that code does not exist. Route any negative or +complete claim to `ccg-search-verify`. + +## Core Loop + +1. **Search first when the path is unknown.** Run one initial structured CCG + `search` with `limit: 5`. Use an exact identifier, literal, or error text as a + compact query. If the request already contains one focused code question, use + it verbatim after removing only command wrappers or output instructions. Do + not compress a behavior or reason question into keywords. Otherwise ask one + focused natural-language question in the repository's vocabulary. CCG tries + the precise match first, then uses OR + matching with BM25/IDF, rewards more distinct terms, and rejects single-term + coincidences. Choose the structured surface from available routing evidence: + use MCP when repository instructions provide explicit MCP or server-visible + routing; otherwise, when local `ccg` and a repository-local `.ccg.yaml` exist, + use JSON CLI with `ccg search --json --compact --limit 5 ""`; use MCP + when no usable local configuration exists. Pass `compact: true` to MCP search. + Compact mode keeps paths, declaration bounds, evidence, and continuations + while omitting redundant storage fields. MCP tool availability or MCP + documentation alone is not routing evidence; explicit routing names the + target namespace together with MCP or a server-visible repository path. JSON + CLI reads its namespace and database from `.ccg.yaml`. For MCP, when + configuration supplies a namespace, include it in the initial search + arguments and every continuation. When a namespace must be extracted, read + only the `namespace:` field, never the full configuration. +2. **Choose evidence from the whole page.** Compare the returned file paths, + matched signals, reasons, and declaration hits. Start with the production hit + that most directly addresses the question, not automatically the first row. + Read each chosen declaration's exact range separately; do not span unrelated + hits by reading from a file's earliest result to its latest. Do not re-locate + a hit with grep when `start_line` and `end_line` are present. Use a targeted + in-file locator only for missing bounds or a helper named by the verified + declaration. For a production-behavior question, do not read tests to + corroborate production behavior already established by current source. Tests + become evidence only when the user asks about them or the production source + leaves a material ambiguity. +3. **Stop by claim sufficiency.** Before another tool call, name the specific + evidence gap in the user's question and state that gap in one sentence. If + no material evidence gap can be named, answer. A verified current-source call + site plus the invoked declaration or contract establishes that mechanism; do + not trace constructor or dependency-injection wiring unless the user asks + which runtime implementation or configuration is selected. Use these + completion rules: + - **Where:** the current-source declaration is enough. + - **How/what:** the branch or contract directly implementing the requested + behavior is enough; include public input or output only when asked. + - **Why:** one directly relevant author-recorded design reason plus current + source confirming its mechanism closes the rationale gap unless the user + requests several reasons; secondary consequences are optional. Do not + trace downstream work merely to prove optional consequences. + - **One relationship:** the edge and the endpoint declaration needed to + interpret it are enough. + Mandatory stop: when the matching condition is satisfied, the next action + must be the final answer. Another tool call is allowed only when current + source contradicts the recorded evidence or the user explicitly requested + additional distinct reasons or details. +4. **Follow a new exact clue before semantic paging.** When verified source + reveals an identifier or literal that names a remaining usage, caller, or + exposure gap, run a path-only reference lookup such as `rg -l` before semantic + paging. Print file names only, exclude test-like paths unless relevant, then + read the necessary production declarations. Do not create a new evidence gap + from identifiers encountered after the original question is already answered. + Never print repository-wide match context. +5. **Page only for an unresolved semantic gap.** A truncation flag makes `next` + available; it does not require paging. If the current page and source-derived + exact clues cannot close the named gap, follow the returned `next` call + verbatim. Preserve query, limit, namespace, and offsets. Use at most three + `next` calls. +6. **Handle a miss once.** An exact-identifier miss goes directly to a path-only + source lookup. For a natural-language miss, retry local JSON CLI once only + when it demonstrably uses a different graph or runtime than MCP. Do not repeat + the same local graph query through another interface. If direct source search + yields no new clue, stop without making a negative claim. + +## Conditional Details + +Do not load a reference merely because declaration bounds are missing; Step 2's +targeted in-file locator is the complete recovery rule. Read +[`references/search-execution.md`](references/search-execution.md) only for +language translation, uncertain test classification, or ambiguous continuation +metadata. Read +[`references/supported-languages.md`](references/supported-languages.md) only for +a language-support question. + +## Boundaries + +- Current source is authoritative for location and runtime semantics; CCG ranks + entry points, intent, and relationships. +- Do not call `get_minimal_context`, namespace-list, or graph-stat tools when the + namespace is already configured. +- After starting Core Loop search, do not add `query_graph`, `get_node`, flow, or + impact calls merely to enrich an already supported answer. +- This workflow is read-only. Never build, update, migrate, or postprocess. + Do not invoke `ccg-analyze` automatically; deeper analysis requires explicit `ccg-analyze` invocation. -- Use `ccg-search-verify` for defensible negative or exhaustive claims. -- Use `ccg-build` only when explicitly invoked. ## Completion -Answer from the smallest verified evidence set. Name the selected path or -qualified symbol and disclose only uncertainty that materially limits the -answer. Do not append namespace, freshness, limit, or tool-call accounting -unless the user requests it or it changes the conclusion. +Return the answer with relevant paths or qualified symbols. Mention truncation +only when the continuation cap limits the answer. Do not append namespace, +freshness, or call-count reports unless requested. diff --git a/skills/ccg/references/search-execution.md b/skills/ccg/references/search-execution.md new file mode 100644 index 00000000..1c456cbd --- /dev/null +++ b/skills/ccg/references/search-execution.md @@ -0,0 +1,148 @@ +# Search Execution Details + +Use only the section required by the active branch of the `ccg` Core Loop. + +## Structured Search Interfaces + +Use MCP when repository instructions provide explicit MCP or server-visible +routing. Otherwise prefer JSON CLI when local `ccg` and a repository-local +`.ccg.yaml` are available, so the current binary and repository configuration +answer directly. Fast discovery needs declaration bounds, rank evidence, +truncation, and continuation arguments; storage IDs are not required. Use the +compact response on either surface. The CLI form is: + +```bash +ccg search --json --compact --limit 5 "" +``` + +For MCP, pass `compact: true` and execute the returned `next` call verbatim. For +CLI JSON, execute the command represented by `next.args` without calculating offsets or changing the +query, limit, or namespace. Plain CLI output is not a substitute because it can +hide structured continuation and declaration data. + +For MCP, include the namespace supplied by repository instructions or +`.ccg.yaml` in the initial call. This is not a freshness or namespace-list +preflight: it is the routing key a shared server needs to select the repository. +Keep that namespace in every returned continuation. If the namespace must be +read from `.ccg.yaml`, extract only the `namespace:` field (for example with a +targeted `rg`) rather than opening the whole file; database configuration may +contain credentials and is irrelevant to read-only search routing. + +## Query Language + +When the source path is unknown, use structured CCG search even when an exact +identifier, literal, or error fragment is available. Those clues make excellent +compact CCG queries; they do not justify an unbounded source scan. Direct grep +may start only when its source path is already known. + +Use one exact identifier, path, literal, or error fragment for a known thing. +When the symbol is unknown, ask a focused natural-language question that names +the component, behavior, and distinguishing operational context. The search +engine removes English function words and first tries a precise all-term match. +If that is empty, it uses OR retrieval to collect candidates sharing any +meaningful term, rejects single-term coincidences, and applies shared BM25/IDF +ranking. A candidate ranks higher when it matches more distinct query terms and +when those terms are rarer in the indexed corpus. This is lexical matching, not +semantic synonym expansion. Preserve the question's useful evidence instead of +manually compressing it into an AND-friendly keyword list, and use vocabulary +that actually appears in the repository. + +Write natural-language terms in the dominant language of repository comments +and annotations. Translate the user's intent when needed, but preserve exact +identifiers, paths, literals, and error messages. Do not mix translations in one +query; one focused query in the repository's vocabulary is cheaper and easier +to verify than parallel synonyms. + +If an exact-identifier query returns no files and exposes no continuation, use a +path-only source lookup for that identifier. Do not repeat a lexical index miss +through another interface to the same graph. For a natural-language miss, a +single structured JSON CLI retry is useful only when it is evidenceably a +different runtime or graph — for example, remote MCP versus a repository-local +`.ccg.yaml`. Treat that retry as a replacement result stream and do not page both +MCP and CLI results. When MCP already runs the local `ccg` binary against the +same configuration, go directly to current-source fallback. + +## Sufficiency Before Paging + +One returned file can contain several declaration hits. Verify the relevant hit +ranges on the current page before following `next`; do not treat only the first +declaration in the first file as the whole page. A positive explanation is +complete when ranked intent or domain-rule evidence identifies the design reason +and current production source confirms the behavior being described. Additional +wiring, retries, tests, or examples may be useful context, but they are not a +reason to page or grep after the requested explanation is already supported. +The amount of evidence depends on the claim. Begin with the highest-ranked +relevant production evidence, then inspect another declaration, file, or helper +only when a material part of the requested answer remains unsupported. Before +each extra lookup, state the evidence gap and choose the narrowest operation that +can close it. If no such gap exists, answer instead of collecting optional +detail. + +Treat a declaration hit's `start_line` and `end_line` as its ready-to-read source +range. Read selected ranges directly; do not spend another grep call locating a +symbol whose bounds CCG already returned. Use an in-file locator only for a file +hit, missing bounds, or a newly referenced helper outside the returned range. + +After CCG returns a usable page, ordinary source verification stays within paths +on that page and paths directly referenced by a declaration already verified +there. There is one hybrid handoff: when verified source reveals an exact +identifier or literal that names a remaining usage, caller, or exposure question, +locate its references with a path-only lookup such as `rg -l`. Return file names, +not repository-wide match context; exclude test-like paths unless they are the +subject, then read the relevant production declarations. Prefer this exact +reference lookup before semantic paging because it follows a source-proven clue +instead of asking the broad question again. A truncation flag only says more +candidates are available; follow `next` when a specific part of the user's +question remains unsupported and neither the current page nor an exact +source-derived clue can close it. + +When a returned production file is clearly relevant but its declaration bounds +do not include one referenced helper, run a targeted grep for that exact symbol +or phrase inside the returned file and read its declaration. This remains page +verification. Follow `next` only when the current page failed to identify a +usable production path or symbol, or when current-source verification disproved +the candidates; paging is not a substitute for looking precisely inside a known +candidate. + +## Candidate Classification + +Classify candidates as `production`, `test`, or `unknown`. + +Confirm `test` only from one of these signals, in priority order: + +1. CCG test-node metadata; +2. explicit repository source-set or build rules; +3. established repository or language test path and filename conventions. + +Fixtures, golden files, snapshots, testdata, and generated mocks are test +support only when those same signals confirm the role. A generic directory name +is not enough. Ambiguous candidates remain `unknown`. + +For a non-test question, inspect production and unknown candidates first. If a +page contains only confirmed tests and has `next`, continue before reading the +tests. Tests may be used later as supporting evidence when production evidence +is insufficient. + +## Declaration Evidence + +When a result supplies a node ID, use `get_node` for exact declaration bounds. +Otherwise use the returned line, language-aware symbol navigation, or a +targeted locator search restricted to the returned paths. A locator grep should +not include broad context ranges. + +Read the complete declaration and attached annotation or documentation. Read +multiple declarations as separate ranges rather than one large range containing +unrelated code. Preserve line numbers on the first read when citations may be +needed. + +## Direct-Source Fallback + +A source search that includes paths not returned by the current CCG page is a +fallback, even when the scope looks small. + +For non-test questions, exclude confirmed test and test-support paths before +fallback matches are returned. Inspect production and unknown files first. The +initial fallback may search a task-relevant scope; every subsequent search must +narrow using a new concrete clue from the preceding result. Stop if the result +provides no new clue. Never turn a fallback miss into an absence or completeness +claim; route such claims to `ccg-search-verify`.