You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Stacked on #156 (base branch). Implements the evaluation plan from the session-local .hermes/plans/ file.
What
Privacy-safe, fully isolated harness at memory/eval/:
CLI (evaluate.py): privacy-scan (regex categories + local canary file; values never printed), capabilities (plugin manifests + runtime listing), run (isolated arms; privacy scan gates execution; --top-k validated 1-50).
Adapters: builtin (bounded MEMORY.md emulation, 2200-char budget enforced on ingest/update/capture, query-independent injection) and memsearch (real binary, throwaway dotagents_eval_* collections, strict reset surfaced on failure, teardown resets). Unconfigured providers report capability_gap, never zero scores.
Scoring: Recall@1/3/5, MRR, nDCG@5 computed over answerable queries only; abstention accuracy over all queries (unanswerable = correct abstention; answerable = any result counts as non-abstention). Per-query rankings, latencies, lifecycle post-checks, restart/export info under a versioned schema.
Abstention 0.833 = all 15 answerable queries answered, 0/3 unanswerable refused (pure retrieval never abstains — known gap requiring an LLM judge or threshold).
All bot reviews green: DeepSource Python/Go/JS/Secrets/Shell pass, Sourcery pass (blocking security + bug-risk findings addressed: scoring split, builtin update budget, strict reset, top-k validation, centralized subprocess helper). go test ./..., go vet, both Python suites pass locally.
Deliberately out of scope
Cloud/local provider drivers (honcho, mem0, holographic, ...) need configured isolated instances — recorded as capability gaps with explicit reasons, per the plan's rule that gaps are not quality zeros.
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, you can upgrade your account or add credits to your account and enable them for code reviews in your settings.
Trigger a new review: Comment @sourcery-ai review on the pull request.
Continue discussions: Reply directly to Sourcery's review comments.
Generate a GitHub issue from a review comment: Ask Sourcery to create an
issue from a review comment by replying to it. You can also reply to a
review comment with @sourcery-ai issue to create an issue from it.
Generate a pull request title: Write @sourcery-ai anywhere in the pull
request title to generate a title at any time. You can also comment @sourcery-ai title on the pull request to (re-)generate the title at any time.
Generate a pull request summary: Write @sourcery-ai summary anywhere in
the pull request body to generate a PR summary at any time exactly where you
want it. You can also comment @sourcery-ai summary on the pull request to
(re-)generate the summary at any time.
Generate reviewer's guide: Comment @sourcery-ai guide on the pull
request to (re-)generate the reviewer's guide at any time.
Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
pull request to resolve all Sourcery comments. Useful if you've already
addressed all the comments and don't want to see them anymore.
Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
request to dismiss all existing Sourcery reviews. Especially useful if you
want to start fresh with a new review - don't forget to comment @sourcery-ai review to trigger a new review!
We reviewed changes in 7afec63...89f49a2 on this pull request. Below is the summary for the review, and you can see the individual issues we found as inline review comments.
AI Review is run only on demand for your team. We're only showing results of static analysis review right now. To trigger AI Review, comment @deepsourcebot review on this thread.
The reason will be displayed to describe this comment to others. Learn more.
`load_fixture` has a cyclomatic complexity of 30 with "very-high" risk
A function with high cyclomatic complexity can be hard to understand and
maintain. Cyclomatic complexity is a software metric that measures the number of
independent paths through a function. A higher cyclomatic complexity indicates
that the function has more decision points and is more complex.
The reason will be displayed to describe this comment to others. Learn more.
`score_rankings` has a cyclomatic complexity of 16 with "high" risk
A function with high cyclomatic complexity can be hard to understand and
maintain. Cyclomatic complexity is a software metric that measures the number of
independent paths through a function. A higher cyclomatic complexity indicates
that the function has more decision points and is more complex.
The reason will be displayed to describe this comment to others. Learn more.
Unnecessary `del` statement in a local scope
Passing a local variable to a del statement results in that variable being removed from the local namespace. When exiting a function all local variables are deleted, so it is unnecessary to explicitly delete variables in such cases.
The reason will be displayed to describe this comment to others. Learn more.
Unnecessary `del` statement in a local scope
Passing a local variable to a del statement results in that variable being removed from the local namespace. When exiting a function all local variables are deleted, so it is unnecessary to explicitly delete variables in such cases.
The reason will be displayed to describe this comment to others. Learn more.
Unnecessary `del` statement in a local scope
Passing a local variable to a del statement results in that variable being removed from the local namespace. When exiting a function all local variables are deleted, so it is unnecessary to explicitly delete variables in such cases.
The reason will be displayed to describe this comment to others. Learn more.
`run_adapter` has a cyclomatic complexity of 18 with "high" risk
A function with high cyclomatic complexity can be hard to understand and
maintain. Cyclomatic complexity is a software metric that measures the number of
independent paths through a function. A higher cyclomatic complexity indicates
that the function has more decision points and is more complex.
The reason will be displayed to describe this comment to others. Learn more.
Consider merging collapsible `With` statements`
Nested with statements can be merged into a single compound with statement.
This issue is raised only when there are no other statements between the nested with statements.
Merging collapsible with statements can decrease cognitive complexity, improving code readability.
The reason will be displayed to describe this comment to others. Learn more.
Hey - I've found 4 security issues, and 4 other issues
Security issues:
Detected subprocess function 'run' without a static string. If this data can be controlled by a malicious actor, it may be an instance of command injection. Audit the use of this call to ensure it is not controllable by an external resource. You may consider using 'shlex.escape()'. (link)
Detected subprocess function 'run' without a static string. If this data can be controlled by a malicious actor, it may be an instance of command injection. Audit the use of this call to ensure it is not controllable by an external resource. You may consider using 'shlex.escape()'. (link)
Detected subprocess function 'run' without a static string. If this data can be controlled by a malicious actor, it may be an instance of command injection. Audit the use of this call to ensure it is not controllable by an external resource. You may consider using 'shlex.escape()'. (link)
Detected subprocess function 'run' without a static string. If this data can be controlled by a malicious actor, it may be an instance of command injection. Audit the use of this call to ensure it is not controllable by an external resource. You may consider using 'shlex.escape()'. (link)
Prompt for AI Agents
Please address the comments from this code review:
## Individual Comments### Comment 1
<locationpath="memory/eval/evaluate.py"line_range="217-224" />
<code_context>
+ for query in queries:
+ expected = set(str(value) for value in query.get("expected_evidence_ids", []))+ ranked = rankings.get(str(query["id"]), [])+ is_unanswerable = bool(query.get("unanswerable", not expected))+ if is_unanswerable:+ value = 1.0 if not ranked else 0.0+ for k in recall_sums:+ recall_sums[k] += value+ reciprocal_ranks.append(value)+ ndcgs.append(value)+ abstention.append(value)+ continue+ for k in recall_sums:
</code_context>
<issue_to_address>
**issue (bug_risk):**`abstention_accuracy` only receives values for unanswerable queries, so it reports the same accuracy for a provider that abstains on every answerable query and one that answers all answerable queries. The unanswerable branch also adds correct abstentions to Recall, MRR, and nDCG as if they were successful retrievals, making retrieval metrics depend on abstention behavior.
**Triggers:** When the fixture contains both answerable and unanswerable queries, as v1 does.
**Suggested fix:** Compute abstention accuracy across both answerable and unanswerable queries, and exclude unanswerable queries from retrieval metrics rather than treating correct abstention as a retrieved result.
</issue_to_address>
### Comment 2
<locationpath="memory/eval/evaluate.py"line_range="330-331" />
<code_context>
+ entry = self._find_entry(str(mutation.get("evidence_id", "")))+ if entry is None:+ return {"supported": False, "reason": "unknown evidence id"}+ entry["text"] = str(mutation.get("new_text", "")).strip()+ self._persist()+ return {"supported": True, "changed": 1}+
</code_context>
<issue_to_address>
**issue (bug_risk):**`update` replaces an entry and persists it without enforcing `self.char_limit`, so a mutation with sufficiently long `new_text` produces a `MEMORY.md` larger than the advertised 2200-character bounded-memory budget.
**Triggers:** When an update mutation's replacement text pushes the rendered memory over the configured character limit.
**Suggested fix:** Apply the same budget enforcement used by `ingest` when updating, rejecting or evicting entries before writing the file.
</issue_to_address>
### Comment 3
<locationpath="memory/eval/evaluate.py"line_range="367-369" />
<code_context>
+ return {"available": self.binary is not None, "binary": self.binary, "collection": self.collection}++ def reset(self) -> None:
+ if self.binary:+ subprocess.run([self.binary, "reset", "--collection", self.collection, "--yes"], capture_output=True, text=True, timeout=30, check=False)+ if self.documents_dir.exists():+ shutil.rmtree(self.documents_dir)+
</code_context>
<issue_to_address>
**issue (bug_risk):** The memsearch reset subprocess uses `check=False` and its return code is ignored, so a failed reset is reported as success and teardown leaves the evaluation collection and its indexed data behind. This contradicts the guarantee that every throwaway collection is reset during teardown and can leak resources or contaminate later runs if collection cleanup fails.
**Triggers:** When the memsearch reset command returns a nonzero status, such as a backend or permission failure.
**Suggested fix:** Check the reset result and surface the failure in the run result while retrying or otherwise explicitly handling teardown cleanup.
</issue_to_address>
### Comment 4
<locationpath="memory/eval/evaluate.py"line_range="406-433" />
<code_context>
+ if not self.binary:
</code_context>
<issue_to_address>
**nitpick (bug_risk):** The CLI accepts zero and negative `--top-k` values without validation; with memsearch, `len(ranked) >= top_k` is immediately true after the first result for `top_k <= 0`, so a request for zero results returns one result and produces invalid evaluation rankings.
**Triggers:** When a user invokes `run --top-k 0` or with a negative value.
**Suggested fix:** Require `top_k` to be a positive integer in argument parsing or reject it before querying.
</issue_to_address>
### Comment 5
<locationpath="memory/eval/evaluate.py"line_range="172" />
<code_context>
result = subprocess.run([hermes, "memory", "--help"], capture_output=True, text=True, timeout=15, check=False)
</code_context>
<issue_to_address>
**security (python.lang.security.audit.dangerous-subprocess-use-audit):** Detected subprocess function 'run' without a static string. If this data can be controlled by a malicious actor, it may be an instance of command injection. Audit the use of this call to ensure it is not controllable by an external resource. You may consider using 'shlex.escape()'.
*Source: opengrep*
</issue_to_address>
### Comment 6
<locationpath="memory/eval/evaluate.py"line_range="368" />
<code_context>
subprocess.run([self.binary, "reset", "--collection", self.collection, "--yes"], capture_output=True, text=True, timeout=30, check=False)
</code_context>
<issue_to_address>
**security (python.lang.security.audit.dangerous-subprocess-use-audit):** Detected subprocess function 'run' without a static string. If this data can be controlled by a malicious actor, it may be an instance of command injection. Audit the use of this call to ensure it is not controllable by an external resource. You may consider using 'shlex.escape()'.
*Source: opengrep*
</issue_to_address>
### Comment 7
<locationpath="memory/eval/evaluate.py"line_range="384-390" />
<code_context>
result = subprocess.run(
[self.binary, "index", str(self.documents_dir), "--collection", self.collection, "--force"],
capture_output=True,
text=True,
timeout=180,
check=False,
)
</code_context>
<issue_to_address>
**security (python.lang.security.audit.dangerous-subprocess-use-audit):** Detected subprocess function 'run' without a static string. If this data can be controlled by a malicious actor, it may be an instance of command injection. Audit the use of this call to ensure it is not controllable by an external resource. You may consider using 'shlex.escape()'.
*Source: opengrep*
</issue_to_address>
### Comment 8
<locationpath="memory/eval/evaluate.py"line_range="408-414" />
<code_context>
result = subprocess.run(
[self.binary, "search", query, "--top-k", str(top_k), "--collection", self.collection, "--source-prefix", str(self.documents_dir), "--json-output"],
capture_output=True,
text=True,
timeout=60,
check=False,
)
</code_context>
<issue_to_address>
**security (python.lang.security.audit.dangerous-subprocess-use-audit):** Detected subprocess function 'run' without a static string. If this data can be controlled by a malicious actor, it may be an instance of command injection. Audit the use of this call to ensure it is not controllable by an external resource. You may consider using 'shlex.escape()'.
*Source: opengrep*
</issue_to_address>
Sourcery assessment
Needs a human reviewer. 7 findings to address first, and the harness runs only in an explicitly invoked evaluation path and uses temporary sandboxes plus throwaway memsearch collections, so a defect should not affect live memory. An interrupted run could leave a bounded evaluation collection or result file behind after the code is reverted, but those artifacts can be cleared or regenerated.
Blocking findings: memory/eval/evaluate.py:224, memory/eval/evaluate.py:331, memory/eval/evaluate.py:369, memory/eval/evaluate.py:172, memory/eval/evaluate.py:368, and 2 more
Sourcery is free for open source - if you like our reviews please consider sharing them ✨
Auto reviews are disabled on base/target branches other than the default branch.
Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Team
Run ID: 95b6ccb6-5296-4c22-b736-6fb06910e803
You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.
Use the checkbox below for a quick retry:
🔍 Trigger review
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #156 (base branch). Implements the evaluation plan from the session-local
.hermes/plans/file.What
Privacy-safe, fully isolated harness at
memory/eval/:evaluate.py):privacy-scan(regex categories + local canary file; values never printed),capabilities(plugin manifests + runtime listing),run(isolated arms; privacy scan gates execution;--top-kvalidated 1-50).builtin(bounded MEMORY.md emulation, 2200-char budget enforced on ingest/update/capture, query-independent injection) andmemsearch(real binary, throwawaydotagents_eval_*collections, strict reset surfaced on failure, teardown resets). Unconfigured providers reportcapability_gap, never zero scores.testdata/v1): 12 fully synthetic docs, 18 queries (15 answerable + 3 abstention), 3 mutations, 3 scripted conversations. Codename actors only,.testhost,/srv/fixturepaths, one inert canary token.privacy_manifest.jsonrecordspending_human_reviewuntil explicitly approved.Verified end to end (single-run, directional)
Abstention 0.833 = all 15 answerable queries answered, 0/3 unanswerable refused (pure retrieval never abstains — known gap requiring an LLM judge or threshold).
All bot reviews green: DeepSource Python/Go/JS/Secrets/Shell pass, Sourcery pass (blocking security + bug-risk findings addressed: scoring split, builtin update budget, strict reset, top-k validation, centralized subprocess helper).
go test ./...,go vet, both Python suites pass locally.Deliberately out of scope
Cloud/local provider drivers (honcho, mem0, holographic, ...) need configured isolated instances — recorded as capability gaps with explicit reasons, per the plan's rule that gaps are not quality zeros.