Release v0.43.0 - #6333
Conversation
Release-Triggered-By: jerm-dro
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #6333 +/- ##
==========================================
+ Coverage 72.94% 72.98% +0.04%
==========================================
Files 742 742
Lines 78236 78236
==========================================
+ Hits 57070 57103 +33
+ Misses 17188 17137 -51
- Partials 3978 3996 +18 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
📝 Generated release notes for
|
| Attribute | Type | Values |
|---|---|---|
rate_limit.decision |
string | allowed, rejected |
rate_limit.rejected_by |
string | none, shared_server, shared_tool, per_user_server, per_user_tool |
rate_limit.fail_open |
bool | always false in this release |
The span is now created as "{HTTP method} {url.path}" and renamed after the inner chain completes, so head samplers or SpanProcessor.OnStart logic keyed on span name see POST /mcp rather than tools/call <tool>. Tail sampling in the collector is unaffected. For users of mutating webhooks, http.request.body.size now reports the pre-mutation body length.
Operator upgrade side effect: because the middleware order is serialized into the runconfig ConfigMap and hashed into the pod template's toolhive.stacklok.dev/runconfig-checksum annotation, every MCPServer / MCPRemoteProxy with telemetry enabled gets a changed runconfig and a rolling restart on upgrade — even if you pin toolhiveRunnerImage.
Known gap: a Redis failure returns before any span annotation, so fail-open requests get none of the three attributes and rate_limit.fail_open is a constant false. Detect fail-open via toolhive_rate_limit_redis_errors until the follow-up counter lands.
Before
# Error-budget alert — counted only genuine backend errors on v0.42.1
sum(rate(toolhive_mcp_requests_total{status="error"}[5m]))
/ sum(rate(toolhive_mcp_requests_total[5m])) > 0.05
After
# Exclude throttling and authz denials from the error budget
sum(rate(toolhive_mcp_requests_total{status="error", status_code!~"429|403"}[5m]))
/ sum(rate(toolhive_mcp_requests_total[5m])) > 0.05
Migration steps
- Identify affected deployments: operator-managed
MCPServer/MCPRemoteProxywithspec.rateLimit,spec.mutatingWebhooksorspec.validatingWebhooks, plus telemetry enabled. - Exclude throttling and denials from error-rate alerts as shown above, or split them into a separate throttling panel.
- Rebase latency SLOs onto
toolhive_mcp_request_duration_seconds{status_code!~"429|403"}—mcp_server_operation_duration_secondscannot filter out rejections — or re-baseline the thresholds after upgrading. - Re-baseline request-count and tool-call-count panels; expect a step change in totals at upgrade time.
- If you run a name-based head sampler, key it on attributes (
mcp.method.name) or move the decision to the collector. - Do not alert on
rate_limit.fail_open="true"in this release; usetoolhive_rate_limit_redis_errors. - Plan for the pod roll described above, or schedule the operator upgrade in a maintenance window.
Migration guide: client-controlled metric labels are capped at 128 bytes
Affects dashboard owners and alert authors whose queries exact-match mcp_resource_id on toolhive_mcp_requests_total / toolhive_mcp_request_duration_seconds, and anyone using label_values(mcp_resource_id) as a Grafana template variable. Trace consumers, tool-name panels and method-name panels are unaffected.
Cumulative metric readers keep every distinct attribute set resident for the process lifetime, and MCP method, tool and prompt names arrive verbatim from the client request bounded only by the 8 MB body cap — so roughly 64 crafted requests could retain ~512 MB permanently and OOM a pod. Client-controlled metric label values are now clamped to 128 bytes on a UTF-8 rune boundary with a trailing ....
128 bytes is comfortably enough for real tool names, method names and prompt names. The problem is mcp_resource_id, which also carries the full resource URI for resources/read and the pagination cursor for */list — both of which routinely exceed 128 bytes.
Labels affected: mcp_method and mcp_resource_id on toolhive_mcp_requests_total and toolhive_mcp_request_duration_seconds; tool on toolhive_mcp_tool_calls_total; mcp.method.name, gen_ai.tool.name and gen_ai.prompt.name on mcp_server_operation_duration_seconds. No metric names, label keys or span attributes changed — span attributes deliberately keep the full value, since spans are sampled and ephemeral. That does mean a metric label and its corresponding span attribute no longer match for truncated values, breaking label-equality "jump to traces" workflows for exactly those series.
Before
toolhive_mcp_requests_total{mcp_method="resources/read", mcp_resource_id="https://api.example.com/…<long URI>…"}
After
sum by (mcp_method) (
toolhive_mcp_requests_total{mcp_method="resources/read", mcp_resource_id=~"https://api\\.example\\.com/.*"}
)
Migration steps
- Grep dashboards and recording rules for exact
mcp_resource_id=matches. Forresources/readand paginated*/listpanels, switch to a prefix regex or drop the label from the selector and aggregate bymcp_method. - Expect a one-time counter discontinuity for affected series at upgrade: the full-value series goes stale and a
...-suffixed series starts from zero. Usesum without (mcp_resource_id)across the transition window, or accept the reset inrate()/increase(). - Refresh Grafana template variables built from
label_values(mcp_resource_id), and any recording rules that pin the label. - Be aware that two resource URIs sharing a 125-byte prefix now collapse into one series. This is unreachable for tool names but plausible for deep paths and presigned URLs, and it silently sums distinct resources in per-resource panels.
- No action needed for trace queries, tool-name panels or method-name panels.
Migration guide: the proxy /health response no longer includes the version object
Affects anyone parsing the JSON body of a workload proxy's unauthenticated /health endpoint — monitoring scripts, dashboards, deployment gates, curl … | jq .version.version checks. The endpoint must stay unauthenticated so Kubernetes probes can reach it, so the fix was to stop disclosing the build fingerprint rather than to add auth.
The version object (version, commit, build_date, go_version, platform) is gone from all three proxies — transparent, httpsse and streamable. A client reading .version.version now gets null with no error and no flag to restore it. status, timestamp, transport and mcp are unchanged, so liveness/readiness probes and degraded-state detection need no change (/health still returns 503 when unhealthy). This aligns the proxies with the minimal shape vMCP and the v1 API already used.
Note the replacement lives on a different server and port than the proxy that lost the field, so it is not a drop-in substitute.
Before
curl -s http://localhost:8080/health | jq -r .version.versionAfter
thv version --format json | jq -r .version
# or, against the thv serve API (authenticated, different port)
curl -s http://127.0.0.1:8080/api/v1beta/version | jq -r .versionMigration steps
- Stop reading version data from a workload proxy's
/health. - Use
thv version --format json— this is the only remaining source ofcommit,build_date,go_versionandplatform. TheGET /api/v1beta/versionendpoint on thethv serveAPI returnsversiononly. - Keep using
/healthforstatusandmcp.available; probes need no change.
Migration guide: thv llm teardown of the last tool clears the gateway connection settings
Affects thv llm gateway users who run a full thv llm teardown, or tear down their last remaining tool, and then expect thv llm setup to work from persisted settings.
Settings like Bedrock and Models were persisted so a plain setup re-run kept them, but nothing ever ended that stickiness — a value outlived every tool that read it and got re-applied by the next setup, even against a gateway since repointed elsewhere. Teardown now resets the LLM config once no configured tool remains, matching what thv llm config reset already did.
On v0.42.1, thv llm teardown && thv llm setup re-applied the persisted connection settings. On v0.43.0 the second command fails with LLM gateway is not configured — run "thv llm config set" first. The whole llm: block is cleared: gateway_url, oidc.issuer / client_id / callback_port, proxy.listen_port, models and bedrock.compat / enable_1m. Cached OIDC token references survive unless --purge-tokens is passed, so the keyring secret is never left orphaned. A teardown that leaves other tools configured is unchanged.
Before
thv llm teardown
thv llm setup # re-used persisted settings on v0.42.1After
thv llm teardown
thv llm setup \
--gateway-url https://llm.example.com \
--issuer https://auth.example.com \
--client-id my-client-idMigration steps
- Capture your settings before tearing down if you need them:
thv llm config show --format json > llm-config.json. - Re-supply the connection settings on the next setup, either inline as above or via
thv llm config set --gateway-url … --issuer … --client-id …first. - Re-add any non-default extras that were also cleared:
--models,--bedrock-compat,--enable-1m,--tls-skip-verify, and the proxy listen port. - To keep the old behaviour, tear down one tool at a time and leave at least one configured —
thv llm teardown claude-codepreserves the config while any tool remains.
PR: #6295
🆕 New Features
- Operators can declare pre-provisioned confidential OAuth clients via
delegateClientsonMCPExternalAuthConfig/VirtualMCPServer, making RFC 8693 token-exchange delegation reachable end to end for agentic use cases — the secret is always by reference and never copied into a ConfigMap or status (#6320) - The embedded auth server can issue confidential clients through dynamic client registration behind a default-off
allowConfidentialClientRegistration, withforceConfidentialRedirectURIsfor clients that declare themselves public and then require a secret (#6252) thv skill pushsigns artifacts with a cosign key via--key, and a failed signing fails the push so nothing is ever silently published unsigned (#6139)- Skill installs and
thv skill infonow display the recorded trust state — the pinned signer identity, certificate issuer and unsigned exceptions — instead of the user first meeting it weeks later inside a 403 (#6137) - Skill verification now records and enforces the signing workflow's git ref and runner class from the Fulcio certificate, so a lock entry pinning "this workflow in this repository" is no longer satisfied by the same workflow run from an attacker's branch or moved to a self-hosted runner (#6315)
- The lock file provenance block records
repositoryRefandrunnerEnvironment, with absence meaning unconstrained so existing lock files keep verifying unchanged (#6312) - macOS
thvbinaries in release archives can now carry aDeveloper ID Application: Stacklok, Incsignature and a Team ID that managed-macOS fleets can allowlist by code-signing identity, instead of only Go's ad-hoc signature (#6156) - Traces for Redis-backed rate-limited tool calls now expose whether a request was allowed or rejected and which bounded bucket rejected it (#5800)
toolhive.lock.yamlgains a typed siblingplugins:key so project-scoped AI-plugin installs can be pinned in the same version-1 lock file as skills, without a schema bump that would hard-fail mixed-version teams (#6303)- Groundwork for plugin sync and upgrade: a
PluginLockServiceinterface plus anInstalledPlugin.managedmarker sosync --prunecan tell lock-managed plugins from hand-installed ones (#6311)
🐛 Bug Fixes
- MCP clients that negotiate a Legacy protocol version while including reserved
_metakeys — the ChatGPT connector among them — can call tools again; previously everytools/callfrom such a client was rejected with-32020before reaching the backend, a regression introduced in v0.41.0 (#6231) - MCP clients whose client-metadata document declares grant types the embedded auth server does not support — VS Code's
device_codeamong them — now resolve successfully with the unsupported entries ignored, instead of failing with an opaqueinvalid_client(#6297) - vMCP correctly detects conformant Modern backends that split one JSON-RPC response across multiple SSE
data:lines, instead of silently downgrading them to Legacy and dropping interleaved progress notifications (#6126) thv llm setupworks on Windows for Claude Code, where it previously failed on every machine because the executable path contained backslashes (#6326)- The interactive-login OAuth callback server binds to loopback only, so a host on your LAN can no longer observe or abort a ToolHive login in progress (#6238)
- The private-IP guard that protects against SSRF now decodes 6to4 addresses and blocks Teredo wholesale, closing two IPv6 transition families that were previously treated as ordinary public addresses (#6249)
🧹 Misc
- Cleared a 111-finding GitHub Actions hardening backlog to zero and made the zizmor check blocking at
mediumseverity, so a PR that introduces a workflow finding now fails CI instead of reporting quietly (#6251, #6274, #6262) - Removed shell-injection sinks across the workflows by binding expressions through
env:rather than interpolating them intorun:blocks, including thegithub.refhandling in the image publish workflow (#6248, #6258, #6275, #6281) - Stopped persisting the checkout credential in
.git/configacross 20 checkouts in 12 CI workflows, where every subsequent step and the third-party code it invokes could read it (#6255) - Scoped release app tokens and workflow permissions to what each job actually needs, and stopped handing every repository secret to the test workflows via
secrets: inherit(#6263, #6266, #6269, #6270, #6272) - The PR size labeler derives its target from the consuming workflow's own trigger event instead of trusting an artifact produced by a
pull_requestworkflow (#6259, #6261) @claudenow responds only to accounts with write access to the repository, replacing a deny-list that permanently admitted anyone who had ever landed a single PR (#6260)- Pinned all actions to commit SHAs and corrected version comments that named a different version than the pinned commit (#6254, #6313)
- Deflaked
TestRoundTripDoesNotReplayInitializeOnDialError, which shared a keep-alive transport with this package's parallel tests (#6264)
📦 Dependencies
| Module | Version |
|---|---|
github.com/stacklok/toolhive-catalog |
v0.20260810.0 |
golang.org/x/crypto |
v0.55.0 |
golang.org/x/mod |
v0.40.0 |
golang.org/x/net |
v0.58.0 |
golang.org/x/text |
v0.41.0 |
golang.org/x/tools |
v0.49.0 |
The golang.org/x/* bumps carry CVE fixes (#6327); the catalog update is #6257.
👋 Welcome to our newest contributor: @kocaemre 🎉
Full commit log
What's Changed
- Guard against shell injection via github.ref in image workflow by @ChrisJBurns in Guard against shell injection via github.ref in image workflow #6248
- Extend private-IP guard to 6to4/Teredo addresses by @ChrisJBurns in Extend private-IP guard to 6to4/Teredo addresses #6249
- Parse multi-line Modern SSE events by @kocaemre in Parse multi-line Modern SSE events #6126
- Add zizmor to CI and stop double-running the security scan by @ChrisJBurns in Add zizmor to CI and stop double-running the security scan #6251
- Pin actions to commit SHAs and fix stale version comments by @ChrisJBurns in Pin actions to commit SHAs and fix stale version comments #6254
- Stop persisting git credentials in CI workflow checkouts by @ChrisJBurns in Stop persisting git credentials in CI workflow checkouts #6255
- Bind workflow expressions and drop persisted credentials by @ChrisJBurns in Bind workflow expressions and drop persisted credentials #6258
- Derive PR number from the triggering run in the size labeler by @ChrisJBurns in Derive PR number from the triggering run in the size labeler #6259
- Gate the Claude workflow on write access, not contribution history by @ChrisJBurns in Gate the Claude workflow on write access, not contribution history #6260
- Stop writing the PR number into the size label artifact by @ChrisJBurns in Stop writing the PR number into the size label artifact #6261
- Suppress four zizmor findings that cannot be fixed by @ChrisJBurns in Suppress four zizmor findings that cannot be fixed #6262
- Scope the release app token to what releaseo needs by @ChrisJBurns in Scope the release app token to what releaseo needs #6266
- Scope releaser permissions and bind the tag expression by @ChrisJBurns in Scope releaser permissions and bind the tag expression #6263
- Pass only the secret the test workflow needs by @ChrisJBurns in Pass only the secret the test workflow needs #6269
- Bind the GHCR credentials and chart version in helm-publish by @ChrisJBurns in Bind the GHCR credentials and chart version in helm-publish #6270
- Scope the remaining release app tokens and bind a trailer by @ChrisJBurns in Scope the remaining release app tokens and bind a trailer #6272
- Rate limiting observability (metrics and tracing) PR B by @Sanskarzz in Rate limiting observability (metrics and tracing) PR B #5800
- Clear the remaining zizmor findings in helm-publish by @ChrisJBurns in Clear the remaining zizmor findings in helm-publish #6275
- Make the zizmor check blocking by @ChrisJBurns in Make the zizmor check blocking #6274
- Clear the last zizmor findings in the release workflows by @ChrisJBurns in Clear the last zizmor findings in the release workflows #6281
- Drop build fingerprint from proxy /health response by @Nashon-Steffen in Drop build fingerprint from proxy /health response #6280
- Let an explicit non-Modern header outrank a reserved key by @amirejaz in Let an explicit non-Modern header outrank a reserved key #6231
- Deflake the initialize dial-error proxy test by @amirejaz in Deflake the initialize dial-error proxy test #6264
- Support confidential clients in dynamic client registration by @jhrozek in Support confidential clients in dynamic client registration #6252
- fix(vmcp): reject Authorization and Cookie in passthroughHeaders, as documented by @SashaMIT in fix(vmcp): reject Authorization and Cookie in passthroughHeaders, as documented #6235
- fix(auth): bind OAuth callback listener to loopback only by @SashaMIT in fix(auth): bind OAuth callback listener to loopback only #6238
- Display recorded trust state to the user by @samuv in Display recorded trust state to the user #6137
- Bound client-controlled metric label length by @Nashon-Steffen in Bound client-controlled metric label length #6279
- Sign the macOS thv binary with a Developer ID certificate by @eleftherias in Sign the macOS thv binary with a Developer ID certificate #6156
- Sign pushes by default and remove the lock feature gate by @samuv in Sign pushes by default and remove the lock feature gate #6139
- Ignore unsupported grant types in CIMD documents by @amirejaz in Ignore unsupported grant types in CIMD documents #6297
- fix(authz): make Cedar URI entity IDs collision-free by @SashaMIT in fix(authz): make Cedar URI entity IDs collision-free #6239
- Add plugins key to lock file schema by @samuv in Add plugins key to lock file schema #6303
- Update module github.com/stacklok/toolhive-catalog to v0.20260810.0 by @renovate[bot] in Update module github.com/stacklok/toolhive-catalog to v0.20260810.0 #6257
- Add PluginLockService and managed install flag by @samuv in Add PluginLockService and managed install flag #6311
- Record certificate ref and runner in lock provenance by @samuv in Record certificate ref and runner in lock provenance #6312
- Pin CodeQL SARIF action comment to v4.37.6 by @samuv in Pin CodeQL SARIF action comment to v4.37.6 #6313
- Reset the LLM config when the last tool is torn down by @jerm-dro in Reset the LLM config when the last tool is torn down #6295
- Use a bare thv command as the LLM token helper by @jerm-dro in Use a bare thv command as the LLM token helper #6326
- Bump go.mod to get CVE fixes by @jhrozek in Bump go.mod to get CVE fixes #6327
- Make RFC 8693 delegate clients reachable and usable by @jhrozek in Make RFC 8693 delegate clients reachable and usable #6320
- Enforce recorded ref and runner on skill verification by @samuv in Enforce recorded ref and runner on skill verification #6315
- Release v0.43.0 by @toolhive-release-app[bot] in Release v0.43.0 #6333
New Contributors
- @kocaemre made their first contribution in Parse multi-line Modern SSE events #6126
Full Changelog: v0.42.1...v0.43.0
🔗 Full changelog: v0.42.1...v0.43.0
1 participant
Release v0.43.0
Version Bump
minor release
Files Updated
VERSIONdeploy/charts/operator-crds/Chart.yaml(path:version)deploy/charts/operator-crds/Chart.yaml(path:appVersion)deploy/charts/operator/Chart.yaml(path:version)deploy/charts/operator/Chart.yaml(path:appVersion)deploy/charts/operator/values.yaml(path:operator.image)deploy/charts/operator/values.yaml(path:operator.toolhiveRunnerImage)deploy/charts/operator/values.yaml(path:operator.vmcpImage)Next Steps
Checklist