agent-plugins-conformance-kit: a fixture corpus for testing client loaders #81
Booyaka101
started this conversation in
Show and tell
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
There are a few tools now for checking whether a plugin you wrote is valid. This one
points the other way: it checks whether a client that loads plugins behaves the way 1.0.0
requires.
The unit is a pair. A real plugin directory on disk, and the load report a conformant
client has to produce for it. 133 of them, covering sections 4 through 11, each carrying
the normative sentence verbatim so a fixture cannot drift from the text it claims to
encode.
Hooking up a client is one script that takes a directory as
argv[2]and prints thatreport as JSON to stdout. No imports and no language requirement, so a Go or Rust client
works the same way as a Node one.
Three things that might be of interest here rather than just to client authors:
The two cases in #77 are fixtures,
core/AP-5.2-UNKNOWN-FIELDandcore/AP-8.1-EXTENSIONS-NOT-OBJECT, both expectingrejected: null. A validate-and-rejectloader fails both, which is that issue as a test rather than as prose.
Four fixtures are marked
disputedbecause the spec permits more than one answer and Idid not want the corpus picking sides.
ssesupport is OPTIONAL, §4.1 symlinks MAYresolve to in-root targets, Agent Skills does not say its frontmatter set is closed, and
the §8.1 member-value question I raised on #77. Those record which way a client went
instead of grading it.
Nine are marked
partial, where a load report genuinely cannot observe the whole rule,mostly §9 subprocess behaviour. Each says in writing what it does and does not assert
rather than implying coverage it does not have.
It is run against two independently written clients on every push, which is what found
three bugs in the corpus before release rather than in it.
https://github.com/Booyaka101/agent-plugins-conformance-kit
Feedback welcome, particularly on the four disputed ones. Those are where a second
implementer's reading is worth more than mine.
All reactions