Repository navigation
test(harness): fix the nine hosted-GitHub tests and guard what their fixes left unchecked - #272
Conversation
…fixes left unchecked Hosted jobs must carry a pinned commit, run on a dedicated VM and take secrets from the platform vault (job.py:400-426). These nine tests were updated for those rules in August, but a merge kept the older test file, so they have failed on dev ever since, before reaching their own assertions. The fixes to the nine are taken from #186 by Karthik, TH-8768, byte for byte, so the two PRs merge cleanly in either order. New tests cover what those fixes leave unchecked: - a laptop GitHub job's commit is frozen from the checkout, and a revision that can't be resolved fails the run; - the token stays out of every git command, and out of the pin check; - a checkout that isn't the pinned commit is refused; - a hosted job with any isolation but a dedicated VM is refused. Part of TH-9444. Signed-off-by: khushalsonawat <khushalsonawat@gmail.com>
KarthikAvinashFI
left a comment
There was a problem hiding this comment.
Looks good. Checked the nine hunks against #186 (identical apart from the import), both merges come out clean, and the focused run gives 61 passed on 3.11. Also mutated the guarded paths (freeze skipped or run in the wrong folder, returncode and commit-shape checks dropped, token env passed to the pin check, prefix-only pin compare, isolation rule dropped or widened to pod/external) and all 17 were caught by the new tests. One small naming nit inline.
Small one on the body: since the repo is public, the #186 reference could drop the person's name and keep just the PR number.
| ] | ||
|
|
||
|
|
||
| def _recording_git(pinned_output: str): |
There was a problem hiding this comment.
Small naming nit: this is what git rev-parse HEAD prints for the checkout, not the pin, and the mismatch test passes a value that isn't the pin. Maybe head_output would read a bit truer?
Refs TH-9444 Signed-off-by: khushalsonawat <khushalsonawat@gmail.com>
c1685ec
into
feat/TH-8804-source-subfolder
#272 (TH-9444) fixed them and is merged into this branch, so sdk-smoke no longer deselects them. The two sandbox-server tests stay deselected under TH-8978. Signed-off-by: khushalsonawat <khushalsonawat@gmail.com>
Summary
Nine tests in
tests/test_harness_architecture.pyfail ondev, so #225 skips them insdk-smoke.yml. One of them is the only check that a GitHub token stays out of git's command line. This makes all nine pass and adds tests for what their fixes leave unchecked.Part of TH-9444. It isn't closed by this PR, because the nine
--deselectlines live only on #225's branch. Whichever of the two lands second removes them, keeping the comment's TH-8978 clause for the two sandbox-server skips.Why they fail. The tests are wrong; the code is right.
job.py:400-426). The contract tests intest_hosted_azain_contract.pypass. That change updated three of these tests and deleted two.The fixes to the nine are taken from #186 (TH-8768), byte for byte.
dedicated_vm,platform-vault, and a fake git that answersrev-parse.New tests, for what those fixes leave unchecked:
..._freezes_the_checked_out_commit_for_a_local_github_job: fix(harness): restore ask routing on the Claude backend and repair stale tests on dev #186 pins that test's job, so the freeze code never runs. Only a laptop job reaches it now...._fails_a_local_github_job_whose_commit_cannot_be_resolved(x2): a failed or non-commitrev-parsefails the run...._keeps_token_out_of_git_arguments_and_the_pin_check: fix(harness): restore ask routing on the Claude backend and repair stale tests on dev #186's fakes record only the clone. This records every call...._refuses_a_checkout_that_is_not_the_pinned_commit: nothing tested the mismatch refusal...._refuses_any_isolation_but_a_dedicated_vm(x4): the old isolation assert now reads back its own input.The new tests stay in this file with every other job and acquirer test. They sit in one block after the clone tests, because placed next to a test #186 edits they conflict with it.
Verification
uv run ruff check .:ruff check tests/test_harness_architecture.pypassespytest -q tests/test_harness_architecture.py tests/test_hosted_azain_contract.pygives 61 passed (52 + 9), on Python 3.11. The 49 the file had before the review fixes also passed on 3.12.14.uv run pytest -qwhen touching shared runtime, CLI, SDK, or release gates: not run; tests onlytypescript/: n/aagent-learn release-proofwhen touching release-candidate behavior: n/aAlso checked:
git merge-fileagainst fix(harness): restore ask routing on the Claude backend and repair stale tests on dev #186 (ef5b4d41) and feat(harness): build an agent from a repo's subfolder #225 (22bcbb87), in every order: 6 of 6 exit 0 and give the expected file. That's local git 2.39.5, a stand-in for GitHub's merge.Checklist
agent_learning.*,agent-learn, or@future-agi/agent-learning-kit: unchangedKnown limits
dev. fix(harness): restore ask routing on the Claude backend and repair stale tests on dev #186'sharness-tests.ymldoesn't include it.Pre-existing issues (not changed here)
executor.pybugs::103);