Repository navigation
Version2.1 - #8
Merged
Merged
Conversation
…architectures (ARM/CPU). Introduce agent subsystem configuration and update environment handling for CORS and ACP integration. Update README with new agent features and migration instructions.
Fetch agent configuration after authorization and keep provider credentials on the backend.
… tool failure back to agent
Toollist bug in code4me-agent
- Add f3c4d5e6a7b9 policy-snapshot migration, crud/schemas/manager support - Add ACP managed endpoints (capabilities/readiness/runs/inference) alongside chat/completions relay - Enforce policy fail-closed, provider creds stay server-side - Add managed backend tests
- AgentConfig managed_mode/allowed_tools/approval, frozenset allowlist with mcp__* wildcard - ToolRegistry approval policies (auto/per_step/suggestion_only), definitions filtering, denied-continues loop - acp_runtime managed sessions via plugin bridge, apply_config rebuild; runtime_auth managed inference transport - Backend relay priority managed_backend>code4me_backend>openai, dual tools/allowed_tools compat - Add policy enforcement and CLI tests
* feat: add study-scoped agent assignments * feat: manage agent arms and assignments by study
- real HTTP + PostgreSQL TestClient suite driving black-box POST /studies then POST /join, a genuinely qualified packaged macos/arm64 release, bootstrap revision-free manifest, session create/reuse/heartbeat, telemetry accepted/duplicate/retryable(tampered capability)/terminal, receipt, close, revoke - repair test_research_schema_lifecycle fixtures: canonical uppercase join codes and a release_id on the lock-study profile (stale vs the uppercased lookup and RELEASE_UNRESOLVED requirement)
…release downloads - Add an opt-in `sign` input to build-managed-runtime, so publishing a runtime release no longer needs the signing secrets. - Accept artifact download URLs only from AISE-TUDelft/code4me2-server release assets. - Seed the IDE's agent cache in the e2e UI layer instead of bundling the agent into the plugin. - Require an explicit producer manifest in seed_local_dev.sh and admin_flow.js. - Update the release, setup and packaging docs for the agent-free plugin.
…when the backend changes - Add `stack.stub_host` (default 127.0.0.1). CI passes 0.0.0.0: a Linux Docker engine maps host.docker.internal to the bridge gateway, so the backend never reached the loopback-only stub and send_message failed with a 502. - Trigger the workflow on backend, packaging and dependency changes, and run the Docker stack for pushes to test_draft/main, pull requests and dispatches. - Declare and export CODE4ME_E2E_PYTHON separately (SC2155).
…ite keeps the snapshot check - The workflow set none of the backend's required settings, so it failed while importing the app and never compared the snapshot. - tests/backend_tests/test_openapi_contract_snapshot.py runs the same check.
Pilot feedback (studies, data access, capture, Revise): - New studies assign arms by a salted SHA-256 hash of the enrollment (formula and golden vector in docs/RESEARCH_ANALYTICS_SCOPE.md); an opt-in manual override before first use is audited and disclosed in consent. - Duplicate any study; custom consent document with tick-box statements, frozen per study, and the accepted version recorded on the enrollment (migration c5e8f1a2d3b4). - Analytics: filters by arm and participant, collapsible metric categories, a per-chat table and the history-dependence view; chat ends never count as cancels. - Per-chat trace view and a raw data export (CSV or JSONL, with content only for content-capture studies). - Built-in agent: "Revise..." on approvals through ACP elicitation (keep some hunks, instructions for the rest). - Shared ACP normalizer: prompt capture, chat start/reopen/end events, and session/load replay suppressed. Fixes from local acceptance testing and the pre-deployment audit: - A managed run that names its window's idle-ended session continues in that window's live session, so the built-in agent keeps answering after an idle rotation. - Session time: never-started sessions expire on their last heartbeat, a placeholder that waited past the idle timeout starts at its first activity, overlapping windows are merged, and sessions the server closes end at their last heartbeat. Studies refuse an idle timeout within one heartbeat. - Self-reported calls under content capture drop a secret-shaped field instead of the whole event; ingestion never rewrites a stored enrollment status. - Logins: reuse only sessions live in Redis and keep the account link alive with them; an expiring old session no longer removes the new link; a backend shutdown no longer flushes Redis. - Agent: unique fallback tool-call ids; replayed prompts show the typed text. - Website: traces name IDE attachments instead of printing them, the agent catalogue shows a just-imported version, the join page points to the Marketplace. nginx re-resolves the backend (image pinned to 1.27.5). - Export carries last_heartbeat_at and session_seconds; the e2e report calls only an HTTP 500 an application bug.
The packaged runtime's Python has no usable CA store on macOS, so every urlopen to an HTTPS backend failed with CERTIFICATE_VERIFY_FAILED: the grant exchange never reached the server and IDE chats reported "auth required". All agent HTTP calls now use one verifying context that adds the bundled certifi roots to the platform defaults (SSL_CERT_FILE and system stores still apply). Earlier tests only used http://localhost, which needs no trust.
OpenCode Zen/Go rejects provider calls without x-opencode-session (400 MissingSessionID) and asks clients for their own user agent. Calls to an opencode.ai upstream now carry code4me-research/1.0 and a hashed, per-agent- session id from both the ACP relay and the inference gateway. Every other provider, OpenRouter included, still receives only the bearer key.
Rewrite landing page: researcher focus, JetBrains theme, neutral feat…
Remove Overview, Usage, Model performance, Agent telemetry and Calibration from the sidebar and stop routing them; study analytics and the participant's own figures on My studies supersede them, and Agent telemetry exposed the model/profile names that My studies keeps blind. The components stay in components/analytics/ but are no longer imported. Signed-in users now land on Studies (researchers, admins) or My studies (participants) after login, on / and from the brand link; old /dashboard?view= links to archived views redirect there. Admin views, Config management and Agent profiles keep their ?view= deep links.
The Cloudflare edge in front of api.code4me.me answers urllib's default Python-urllib/3.x agent with 403 (error 1010) before the request reaches the server, so the grant exchange failed and IDE chats reported "auth required". The urllib calls in runtime_auth and the telemetry upload now send code4me2-agent/<runtime version> from one tls.USER_AGENT constant, which the OpenAI-compatible provider also uses instead of its hardcoded code4me2-agent/0.1. Calls to the plugin's loopback bridge are unchanged.
Fixes found while running the agent on SWE-bench tasks. IDE/ACP
behaviour changes only where it was wrong; the headless-only parts are
opt-in through AgentConfig.autonomous.
- Provider: direct OpenCode calls send a hashed x-opencode-session
header (400 MissingSessionID otherwise), and an optional
request_deadline_seconds abandons a request that a keep-alive stream
would hold open forever (retried like a timeout).
- Context: the token estimate is calibrated against the provider's
prompt_tokens, and a turn that outgrows the budget has its older tool
output elided once, down to 60%, instead of being trimmed on every
call, so the request prefix stays cacheable.
- Reasoning: reasoning_content (or reasoning) stays on the assistant
message that produced it and is sent back unchanged. DeepSeek V4
models get "" on every assistant message when tools are sent, and a
400 that rejects or demands the field is retried once with the other
policy.
- Stopping: only commands that run code count as verification (git,
ls, cat, installs and the like do not; bash -c scripts are checked
per segment). Headless turns whose final text announces more work
("Let me...") or was cut off are continued at most twice, with a tool
run in between.
- Tools: out-of-range size arguments are clamped and a plain argv
string is split with shlex, both reported in argument_notes; shell
syntax is still refused. Unknown tools are reported as unknown, with
the available tools listed.
- Test output: pytest -q summaries and runners inside bash -c scripts
are parsed.
- Prompt: the run_command description and the system prompt mention
["bash", "-c", ...] only when an allowlisted shell is installed.
- Telemetry stays on the existing event types: main-loop provider
failures add call_purpose, error_type and error_message (payload) and
status_code, attempts and duration_ms (metrics) to
agent.adapter.loop_failed; failed side calls are listed on
agent.run.completed (metrics.failed_side_calls); compaction adds
integer context metrics to the next agent.model.requested.
…aged agent benchmarks/swebench runs the packaged Python agent (no plugin or ACP) inside each SWE-bench Verified or SWE-rebench task container and grades the patches with the official harness. It is its own Python project; nothing in the server imports it. - The agent runs from a content-hashed runtime volume (python-build-standalone 3.13, the runtime lock and the agent wheel) and writes a code4me.agent.event.v1 trace per task. `check` validates every trace: ordering, every model request answered or failed, token metrics equal to provider usage, no provider key. - Network: task containers sit on an internal Docker network, and a small CONNECT proxy allows only the model host on port 443 and logs every denial. The mini-swe-agent baseline runs with --network none. - Provider outages (429 and 5xx) are recorded as infrastructure errors, stop the run after three in a row and can be requeued instead of being graded as failures. - Defaults: deepseek-v4.1-flash through OpenCode Go, 100 iterations, a 400k-token context and self-review off, all recorded in the run manifest. - Bundled subsets: Verified mini, stratified 20 and 50, and the 50 newest SWE-rebench 2026-03 tasks. `compare` reports Wilson intervals and McNemar's test against another run or the mini-swe-agent baseline.
The agents layer needs a host Goose, which the harness deliberately never installs (participants bring their own), so `layer=all` on GitHub always ended "blocked by prerequisites" even with every other layer green. The all run now downloads Goose 1.51.0, the release the harness probes target, checks it against the SHA-256 GitHub publishes for the asset, and points the harness at it through CODE4ME_E2E_GOOSE_EXECUTABLE.
A direct provider (kind "openai") can now carry provider.extra_body: extra top-level JSON fields sent with every request, such as OpenRouter's provider routing that pins one upstream or an explicit reasoning effort. The fields go out through the OpenAI SDK's extra_body and are recorded top-level in the request telemetry, as they appear on the wire. The option is local configuration only. A server-assigned provider drops it, the backend relays never send it, and the agent's own request fields (model, messages, tools, sampling, token limits) cannot be overridden. An invalid value is a configuration error rather than being ignored, because silently losing an upstream pin would change which model answers.
…a OpenRouter The SWE-bench harness now compares against the official SWE-rebench leaderboard instead of a baseline of our own: same model, same task window, same number of runs, same metrics. - Protocol defaults follow the leaderboard (read from swe-rebench.com on 2026-10-14): up to 500 model calls, a 128k context, 300 s per command, and a 4 h guard against hangs. The dataset revision is pinned. - `run --repeats N` makes N independent runs task by task, so each image is pulled once. `aggregate` computes the leaderboard's metrics (mean resolved rate over runs with every task in the denominator, SEM, pass@n, tokens, cache share and cost per problem) overall and per time window, next to a published result from references.json, with the difference and its z score. - `select --window` writes leaderboard windows. The bundled subsets cover split 2026_03: the full window (110 tasks) and its windows of 46, 64, 39 and 21 tasks. - OpenRouter: `--openrouter-upstream SLUG` pins every request to one host, and an unpinned openrouter.ai run is refused. OpenRouter runs state temperature 1.0, top_p 1.0 and reasoning effort high explicitly. Telemetry records the serving host and the billed cost of every call. `check` fails a task that any other host served. - A refused or exhausted key and a missing model or host (401, 402, 403, 404) now count as provider outages, like rate limits, instead of grading every remaining task as unresolved. - Fix: test runs were counted from a field the agent never writes, so test_runs was always 0.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.