Skip to content

Add conformance/negative/: vectors, failure codes, and runner (closes the suite gap for the enforcement layer) - #30

Open
avp9-nexus wants to merge 1 commit into
GenAI-Security-Project:mainfrom
avp9-nexus:feature/negative-conformance-vectors
Open

Add conformance/negative/: vectors, failure codes, and runner (closes the suite gap for the enforcement layer)#30
avp9-nexus wants to merge 1 commit into
GenAI-Security-Project:mainfrom
avp9-nexus:feature/negative-conformance-vectors

Conversation

@avp9-nexus

@avp9-nexus avp9-nexus commented Aug 17, 2026

Copy link
Copy Markdown

Implements the proposal of #29 and begins closing the conformance-suite gap of #19.

What is in here:

  • 18 framework-agnostic vectors in JSON across the six categories of Proposal: negative conformance vectors for the enforcement layer (valid-but-unauthorized, replayed, and malformed agent actions) #29: 12 must-reject inputs and 6 positive controls, one per category
  • a proposed failure code enumeration. Its backbone is one distinction: REJECT (evidence measured and found wrong) is a different verdict from UNMEASURABLE (evidence could not be measured; the action must still be refused, under a distinct code). Collapsing the two either lets unmeasurable inputs fail open, or sends operators hunting for frauds that never happened
  • runner.py, which fails the suite on any verdict mismatch, code mismatch, missing reason substring, or any category lacking its positive control. That last rule is structural: a suite made only of must-reject inputs cannot distinguish an enforcement layer that works from one that rejects everything
  • reference_adapter.py, a minimal passing implementation, so the suite is executable end to end as shipped: python conformance/negative/runner.py --adapter conformance/negative/reference_adapter.py

Category 3 (expired or revoked mandates) is covered for expiry only. Revocation vectors are a declared gap: the contributing implementation has no revocation mechanism, by documented decision, and vectors for a path never exercised in production would be design fiction. Contributions from implementations that exercise revocation are the way to close it.

Every failure code in the enumeration was produced by a real refusal in a production enforcement layer before it was named here. Vector wording is deliberately implementation-neutral; naming, directory layout and the enumeration itself are all open to review.


Type of change

  • Specification change (schema, hooks, events, AgBOM)
  • Documentation
  • Tooling or CI
  • Governance (licensing, security policy, contributor docs)

(Nothing under specification/ or docs/spec/ is touched; the new directory sits at conformance/negative/.)

Checklist

  • Commits are signed off with git commit -s (required by the DCO)
  • Prose follows STYLE.md
  • uv run mkdocs build --strict passes (not run locally; conformance/ sits outside the docs tree, happy to add the check if maintainers want it wired into CI)
  • No secrets, tokens, or internal URLs in the diff

Security

  • This change has no security impact

(It adds test vectors and a runner; it changes no normative text and no schema.)

…AI-Security-Project#29, GenAI-Security-Project#19)

Implements the negative conformance proposal of GenAI-Security-Project#29 and begins closing
the conformance-suite gap of GenAI-Security-Project#19.

- 18 framework-agnostic vectors across the six categories of GenAI-Security-Project#29
  (12 must-reject, 6 positive controls)
- a proposed failure code enumeration whose backbone separates
  REJECT (measured and found wrong) from UNMEASURABLE (could not
  be measured; must still refuse, under a distinct code)
- runner.py fails on verdict, code, missing reason substring, or
  any category lacking its positive control
- reference_adapter.py: a minimal passing implementation, so the
  suite is executable end to end out of the box
- category 3 covers expiry only; revocation vectors are a declared
  gap, per the contributing implementation's documented scope

Each failure code was produced by a real refusal in a production
enforcement layer before it was named here.

Signed-off-by: avp9-nexus <avp9pro@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant