Add hosted-agent PR policy checks - #983
Open
Antriksh Jain (antriksh30) wants to merge 1 commit into
Open
Antriksh Jain (antriksh30) wants to merge 1 commit into
Antriksh Jain (antriksh30) wants to merge 1 commit into
Conversation
Port credential-free hosted-agent contract and Python dependency policies to public pull requests. Keep shared test contracts and guidance under .azure-pipelines/hosted-agent-tests. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Antriksh Jain (antriksh30)
deployed
to
L4-validation
September 14, 2026 09:57 — with 
GitHub Actions
Active
GitHub Actions
Active
Copilot started reviewing on behalf of
Antriksh Jain (antriksh30)
September 14, 2026 10:03
View session
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
Validation gaps permit assertion-free contracts, malformed approval evidence, and inadequately specified dependency exceptions.
Get a fresh assessment by requesting another Copilot review.
Pull request overview
Adds credential-free hosted-agent policy enforcement for pull requests.
Changes:
- Adds contract and Python dependency policy workflows/checkers.
- Adds schema, fixture-path, exception, and regression-test support.
- Updates contributor documentation and ownership routing.
File summaries
| File | Description |
|---|---|
samples/python/hosted-agents/DEPENDENCY_POLICY.md |
Documents public dependency validation. |
samples/python/hosted-agents/AGENTS.md |
Updates Python contract guidance. |
samples/csharp/hosted-agents/AGENTS.md |
Updates C# contract guidance. |
CONTRIBUTING.md |
Describes new PR checks. |
.github/workflows/hosted-agent-policies.yml |
Defines policy jobs. |
.github/scripts/test/test_hosted_agent_policies_workflow.py |
Tests workflow security properties. |
.github/scripts/test/test_hosted_agent_fixture.py |
Tests fixture path mapping. |
.github/scripts/test/test_check_hosted_agent_python_requirements.py |
Tests dependency policy. |
.github/scripts/test/test_check_hosted_agent_contracts.py |
Tests contract coverage. |
.github/scripts/requirements.txt |
Adds packaging dependency. |
.github/scripts/hosted_agent_test_spec.py |
Implements contract validation and evaluation. |
.github/scripts/hosted_agent_fixture.py |
Maps samples to fixtures. |
.github/scripts/check-hosted-agent-python-requirements.py |
Enforces dependency reproducibility. |
.github/scripts/check_hosted_agent_contracts.py |
Enforces new-sample contracts. |
.github/CODEOWNERS |
Routes fixture ownership. |
.azure-pipelines/hosted-agent-tests/README.md |
Documents contract schema and workflow. |
.azure-pipelines/hosted-agent-tests/python-requirements-exceptions.toml |
Defines dependency exceptions. |
Review details
Suppressed comments (2)
.github/scripts/hosted_agent_test_spec.py:492
- Falsey non-array values such as
{},"", or0are coerced to an empty list here, so malformed Responses evidence can be reported as having no approval requests (and even as sequence-complete) instead of failing closed. Default only a missing key, then let the existing type check reject every non-list value.
output = response.get("output") or []
.github/scripts/hosted_agent_test_spec.py:368
- The validator permits a contract with no turn- or test-level assertions because both assertion collections default to empty. Such a contract passes the new PR policy but only checks execution success, contrary to the required “assertions for its defining behavior” in
samples/python/hosted-agents/AGENTS.md:52-55; require at least one assertion somewhere in the document.
return document
- Files reviewed: 17/17 changed files
- Comments generated: 3
- Review effort level: Balanced
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+507
to
+509
| missing = {"path", "code", "reason", "owner", "issue", "expires"} - raw.keys() | ||
| if missing: | ||
| raise CheckError(f"{path}: exception {index} is missing {sorted(missing)}") |
Comment on lines
+167
to
+170
| if "exists" in predicate and not isinstance(predicate["exists"], bool): | ||
| raise SpecError(f"{context}.exists must be a boolean") | ||
| if "regex" in predicate: | ||
| _compile_regex(predicate["regex"], f"{context}.regex") |
Comment on lines
+100
to
+101
| CI rejects any fixture whose full path does not map back to a sample containing | ||
| `azure.yaml`. |
|
|
||
| findings: list[Finding] = [] | ||
| for root, root_triggers in sorted(triggers.items(), key=lambda item: str(item[0])): | ||
| source = root / "requirements.txt" |
Contributor
There was a problem hiding this comment.
The required policy currently unconditionally selects requirements.txt. A new or dependency-updated uv-native runtime with pyproject.toml + uv.lock therefore fails PYREQ001/PYREQ006, even though that is the intended future format. Because this check will block merges, it should accept either a fully resolved requirements.txt or a valid pyproject.toml + uv.lock.
| _validate_assertion( | ||
| assertion, "test", f"{context}.assertions[{assertion_index}]" | ||
| ) | ||
| return document |
Contributor
There was a problem hiding this comment.
Assertion collections currently default to empty, so an assertion-free contract passes. This does not need to block this PR, but it is worth tracking a follow-up to require at least one defining-behavior assertion.
| group: hosted-agent-policies-${{ github.event.pull_request.number }} | ||
| cancel-in-progress: true | ||
|
|
||
| jobs: |
Contributor
There was a problem hiding this comment.
These checks are currently advisory: the public main ruleset still requires only trusted. Please track adding both new job contexts as required checks after the workflow merges and passes a proof PR.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
.azure-pipelines/hosted-agent-tests/, preserving the full language/framework/sample path.Execution boundary
Both jobs use
pull_request, a read-only token, and checkout without persisted credentials. They can run on fork PRs subject to GitHub approval and require no Azure secrets or OIDC access. This change does not enable cloud deployment, modify the existingtrustedgate or merge rules, or re-enable disabled workflows.Historical cloud fixtures and ADO pipeline wiring are separate follow-up work.