Skip to content

test(harness): fix the nine hosted-GitHub tests and guard what their fixes left unchecked - #272

Merged
khushalsonawat merged 2 commits into
feat/TH-8804-source-subfolderfrom
feat/TH-9444-hosted-github-tests
Nov 10, 2026
Merged

khushalsonawat merged 2 commits into
feat/TH-8804-source-subfolderfrom
feat/TH-9444-hosted-github-tests

Conversation

@khushalsonawat

@khushalsonawat khushalsonawat commented Nov 10, 2026 •

Copy link
Copy Markdown
Member

Summary

Nine tests in tests/test_harness_architecture.py fail on dev, so #225 skips them in sdk-smoke.yml. One of them is the only check that a GitHub token stays out of git's command line. This makes all nine pass and adds tests for what their fixes leave unchecked.

Part of TH-9444. It isn't closed by this PR, because the nine --deselect lines live only on #225's branch. Whichever of the two lands second removes them, keeping the comment's TH-8978 clause for the two sandbox-server skips.

Why they fail. The tests are wrong; the code is right.

  • On 25 Aug, hosted jobs started requiring a pinned commit, a dedicated VM and vault secrets (job.py:400-426). The contract tests in test_hosted_azain_contract.py pass. That change updated three of these tests and deleted two.
  • A merge four minutes later kept the older test file, and the others predate the rules. So each fails at job setup, before its own assertion.
  • These are intended behaviour changes, edited into the tests. A code-owner review of the test change is requested, per standard 12.

The fixes to the nine are taken from #186 (TH-8768), byte for byte.

New tests, for what those fixes leave unchecked:

The new tests stay in this file with every other job and acquirer test. They sit in one block after the clone tests, because placed next to a test #186 edits they conflict with it.

Verification

  • uv run ruff check .: ruff check tests/test_harness_architecture.py passes
  • Focused pytest target: pytest -q tests/test_harness_architecture.py tests/test_hosted_azain_contract.py gives 61 passed (52 + 9), on Python 3.11. The 49 the file had before the review fixes also passed on 3.12.14.
  • Full uv run pytest -q when touching shared runtime, CLI, SDK, or release gates: not run; tests only
  • TypeScript build/test when touching typescript/: n/a
  • agent-learn release-proof when touching release-candidate behavior: n/a

Also checked:

Checklist

  • Public API names stay under agent_learning.*, agent-learn, or @future-agi/agent-learning-kit: unchanged
  • New user-facing workflow has docs or an example: n/a
  • New release claim has executable gate coverage: n/a
  • No hosted-service dependency was added to release-gated examples
  • Security-sensitive output is redacted or justified: token values are made up

Known limits

Pre-existing issues (not changed here)

  • TH-8978 and TH-9503: the sandbox server builds hosted jobs these rules refuse.
  • TH-9499: a pin that isn't the branch tip fails at clone.
  • A new ticket for three executor.py bugs:
    • a public-repo clone error is garbled by the empty-token redaction (:103);
    • a git timeout in the pin check escapes untyped;
    • a git timeout or missing binary in the commit freeze escapes untyped.

…fixes left unchecked

Hosted jobs must carry a pinned commit, run on a dedicated VM and take secrets
from the platform vault (job.py:400-426). These nine tests were updated for
those rules in August, but a merge kept the older test file, so they have
failed on dev ever since, before reaching their own assertions.

The fixes to the nine are taken from #186 by Karthik, TH-8768, byte for byte,
so the two PRs merge cleanly in either order.

New tests cover what those fixes leave unchecked:
- a laptop GitHub job's commit is frozen from the checkout, and a revision
  that can't be resolved fails the run;
- the token stays out of every git command, and out of the pin check;
- a checkout that isn't the pinned commit is refused;
- a hosted job with any isolation but a dedicated VM is refused.

Part of TH-9444.

Signed-off-by: khushalsonawat <khushalsonawat@gmail.com>

@KarthikAvinashFI KarthikAvinashFI left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good. Checked the nine hunks against #186 (identical apart from the import), both merges come out clean, and the focused run gives 61 passed on 3.11. Also mutated the guarded paths (freeze skipped or run in the wrong folder, returncode and commit-shape checks dropped, token env passed to the pin check, prefix-only pin compare, isolation rule dropped or widened to pod/external) and all 17 were caught by the new tests. One small naming nit inline.

Small one on the body: since the repo is public, the #186 reference could drop the person's name and keep just the PR number.

Comment thread tests/test_harness_architecture.py Outdated
]


def _recording_git(pinned_output: str):

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Small naming nit: this is what git rev-parse HEAD prints for the checkout, not the pin, and the mismatch test passes a value that isn't the pin. Maybe head_output would read a bit truer?

Refs TH-9444

Signed-off-by: khushalsonawat <khushalsonawat@gmail.com>
@khushalsonawat
khushalsonawat changed the base branch from dev to feat/TH-8804-source-subfolder November 10, 2026 15:15
@khushalsonawat
khushalsonawat merged commit c1685ec into feat/TH-8804-source-subfolder Nov 10, 2026
6 checks passed
@khushalsonawat
khushalsonawat deleted the feat/TH-9444-hosted-github-tests branch November 10, 2026 15:17
khushalsonawat added a commit that referenced this pull request Nov 10, 2026
…leep

Signed-off-by: khushalsonawat <khushalsonawat@gmail.com>
khushalsonawat added a commit that referenced this pull request Nov 10, 2026
#272 (TH-9444) fixed them and is merged into this branch, so
sdk-smoke no longer deselects them. The two sandbox-server tests stay
deselected under TH-8978.

Signed-off-by: khushalsonawat <khushalsonawat@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants