Skip to content

Add live-stack MCP 2026-07-28 conformance testing - #82

Merged
lucarlig merged 12 commits into
mainfrom
user/luca/mcp-conformance-ci
Aug 14, 2026
Merged

Add live-stack MCP 2026-07-28 conformance testing#82
lucarlig merged 12 commits into
mainfrom
user/luca/mcp-conformance-ci

Conversation

@lucarlig

@lucarlig lucarlig commented Aug 11, 2026

Copy link
Copy Markdown
Contributor

What this adds

  • runs the official @modelcontextprotocol/conformance@0.2.0-alpha.11 server suite for MCP 2026-07-28
  • tests the real public path: official runner → nginx → dataplane → official fixture
  • builds the dataplane image from the checked-out source so pull requests test the merge candidate before it reaches main
  • provides make conformance to build the current checkout and run the same live harness locally
  • provisions the fixture through the real control plane and publishes its configuration through Redis
  • keeps the harness self-contained under tests/conformance/
  • records detailed counts in the Actions log and enforces a checked-in expected-failure baseline

Live stack

The job uses pinned Redis and nginx images, the published latest control-plane image, and a dataplane image built locally from the checked-out source. The control plane uses ephemeral SQLite, so PostgreSQL and PgBouncer are not required.

The alpha.11 runner cannot send a bearer header. Nginx therefore adds an ephemeral control-plane-issued token only when the request has no authorization header. There is no auth proxy, score calculator, or repository-owned JavaScript helper. A route probe confirms traffic reached the dataplane before conformance starts.

Current baseline

Total: 76 passed, 80 failed
Baseline check passed: all failures are expected.

The conformance output remains in the Actions log; no separate conformance artifact is uploaded. Pull-request runs build and test the checked-out merge candidate, while the control plane remains the currently published latest image. New failures and stale expected failures both fail CI.

Additional maintenance

  • bumps rmcp and rmcp-macros from 3.1.1 to the latest release, 3.1.2
  • updates the Pages workflow to Node 24-compatible upload/deploy actions
  • installs mdBook 0.5.3 directly without the cargo-binstall fallback warning

Validation

  • hosted MCP conformance workflow
  • MCP_CONFORMANCE_PORT=18080 make conformance
  • hosted Pages build without Node 20 or mdBook fallback warnings
  • all repository PR checks
  • actionlint, ShellCheck, Docker Compose validation, and mdBook build/tests
  • workspace formatting, Clippy, and nextest with rmcp 3.1.2

Signed-off-by: lucarlig <luca.carlig@ibm.com>
Signed-off-by: lucarlig <luca.carlig@ibm.com>
@lucarlig
lucarlig force-pushed the user/luca/mcp-conformance-ci branch from 16486da to 3dd0371 Compare August 11, 2026 10:34
Signed-off-by: lucarlig <luca.carlig@ibm.com>
Signed-off-by: lucarlig <luca.carlig@ibm.com>
@lucarlig lucarlig changed the title Add live MCP 2026-07-28 conformance CI Add live-stack MCP 2026-07-28 conformance testing Aug 11, 2026
@lucarlig lucarlig self-assigned this Aug 11, 2026
@lucarlig
lucarlig requested a review from Lang-Akshay August 11, 2026 11:03
@lucarlig
lucarlig marked this pull request as ready for review August 11, 2026 11:26
Signed-off-by: lucarlig <luca.carlig@ibm.com>
Signed-off-by: lucarlig <luca.carlig@ibm.com>
@lucarlig
lucarlig marked this pull request as draft August 11, 2026 11:36
@lucarlig
lucarlig marked this pull request as ready for review August 11, 2026 12:15
@lucarlig
lucarlig requested review from madhu-mohan-jaishankar and removed request for Lang-Akshay August 11, 2026 13:14
Comment thread .github/workflows/mcp_conformance.yml
Comment thread .github/workflows/mcp_conformance.yml
Comment thread tests/conformance/docker-compose.yml Outdated
Comment thread tests/conformance/docker-compose.yml Outdated
Comment thread tests/conformance/docker-compose.yml Outdated
Comment thread tests/conformance/docker-compose.yml
Comment thread tests/conformance/register-fixture.sh

@madhu-mohan-jaishankar madhu-mohan-jaishankar left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

some minor changes and questions.

Signed-off-by: lucarlig <luca.carlig@ibm.com>
Signed-off-by: lucarlig <luca.carlig@ibm.com>
Signed-off-by: lucarlig <luca.carlig@ibm.com>
Signed-off-by: lucarlig <luca.carlig@ibm.com>
Signed-off-by: lucarlig <luca.carlig@ibm.com>
Comment thread .github/workflows/mcp_conformance.yml Outdated
Signed-off-by: lucarlig <luca.carlig@ibm.com>

@madhu-mohan-jaishankar madhu-mohan-jaishankar left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@lucarlig
lucarlig merged commit 051bee6 into main Aug 14, 2026
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants