fix(ci,train): stop integ-tests rerunning the shallow suite

[jam-jee]

Issue

1. The shallow (submit-then-stop) suite runs twice on every sagemaker-train PR.

It lives under tests/integ/train/shallow, and the deep integ-tests CodeBuild
project invokes pytest through tox over the whole tests/integ tree. So the same
~100 tests run once in fast-integ-tests and then again inside integ-tests'
parallel pass, on the same commit.

Example: run 32994923468 —
the integ-tests (sagemaker-train) job ran 82 shallow tests that fast-integ-tests
had already run.

This is not free. Each shallow test submits a real CreateTrainingJob, so the
duplicate pass doubles the suite's draw on the training-job quota the two projects
share — the exact contention the harness's concurrency cap exists to avoid — while
adding no coverage, since both passes select the same tests by the same marker
expression.

2. The suite's README names a pytest flag that no longer exists. It lists
--dist loadfile as part of how the suite is invoked; that flag has since been
removed from the CodeBuild invocation.

Description of changes

tox.ini — pass --ignore=tests/integ/train/shallow to the pytest invocation,
and document it in the suite's README.

Three properties make tox.ini the right place for this:

• The deep project goes through tox, so it picks the ignore up from the PR's own
  checked-out source. Its buildspec is not defined in this repo, and the internal CDK
  package only defines the staging copy of that project — so the buildspec is not
  something a PR here can change.
• fast-integ-tests does not go through tox. It runs
  python3.10 -m pytest tests/integ/train/shallow ... directly, so it is unaffected
  and still runs the whole suite.
• --ignore only prunes directory recursion. It does not override an explicitly
  named path, so tox -- tests/integ/train/shallow still collects all 100 tests. No
  local or scheduled invocation that names the directory loses coverage.

Only the parallel pass was ever affected — the shallow tests carry no serial mark,
so the serial pass never collected them (0 references in that half of the log,
against 211 in the other).

README — drop the stale --dist loadfile from the list of pytest options the
README says lives in createCIShallowIntegBuildSpec. (Previously #6215, folded in
here as cdcad04a since both changes touch this README.)

Why that flag went away: it pins one file's tests to a single xdist worker, and each
test holds a concurrency slot until its training job reaches a terminal state (~75s).
That serialized the largest file — the 17-test RLVR file alone took 19m45s, against
the CodeBuild project's 30-minute timeout. It was never needed: job names are unique
per invocation rather than per test function (see unique_name in harness.py),
precisely so tests can spread across workers, and the wall-clock estimate in
harness.py assumes they do.

Testing done

Collection against the deep project's exact selection,
tests/integ -m "not serial and not gpu_intensive and not us_east_1":

	collected	selected
before	348	200
after	248	116

Exactly 100 collected and 84 selected drop out — the shallow suite and nothing else,
matching the 82 shallow tests observed in the integ-tests log above.

Also verified:

• tox -- tests/integ/train/shallow (explicit path) still collects all 100 tests.
• tox --showconfig -e py310 resolves commands to
  pytest --ignore=tests/integ/train/shallow, with the added comment block stripped;
  configparser — which CI's pinned tox==3.28.0 uses — parses it identically, so
  the comments cannot be mistaken for commands.
• The full shallow suite under the current invocation (Python 3.10, -n 8, no
  --dist loadfile) against us-west-2: 83 passed, 1 skipped in 6m50s. The single
  skip is structural — RLAIFTrainer takes no compute argument, so
  test_explicit_compute_is_accepted skips itself.

One pre-existing collection error in test_list_hyperparameters_integration.py is a
stale local install of sagemaker-train in my venv, present identically before and
after this change.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

[github-actions]

Claude finished @jam-jee's task in 2m 33s —— View job

---

Review: PR #6216 — stop integ-tests rerunning the shallow suite

• Read PR diff
• Review tox.ini change (correctness of --ignore behavior)
• Review README change
• Check pytest version / path-resolution assumptions
• Post review

---

Verdict: looks good. This is a clean, well-reasoned CI-config change with no code or public-API impact. The approach is sound: tox.ini is the correct lever because the deep integ-tests project goes through tox (and picks up the ignore from the PR's own checked-out source), while fast-integ-tests invokes pytest directly and is unaffected — so the shallow suite still runs exactly once. The inline comment and README note are clear and accurate, and no coverage is lost. Buildspecs aren't in-repo, so I took the tox-vs-direct-pytest split in the description at face value; the README/tox comments are internally consistent with it.

A few non-blocking notes:

1. The fix leans on a load-bearing pytest semantic that isn't version-pinned. The whole design depends on --ignore pruning only directory recursion and not suppressing an explicitly-named path, so that tox -- tests/integ/train/shallow still collects all 100 tests. requirements/extras/test_requirements.txt lists pytest unpinned, so that behavior isn't locked to a version you've tested against. It's stable, documented pytest behavior and you verified it empirically, so this is low risk — but if pytest ever changed it, any invocation that names the directory (local/scheduled deep runs) would silently drop the shallow suite with no failure. Worth a comment at minimum, or an explicit pin if you want it guaranteed.

2. The --ignore is added to the shared [testenv] commands, so it applies to every default-env (py310/py311/py312) invocation — not just the deep integ project. tox with no posargs and tox -- tests/unit now also carry --ignore=tests/integ/train/shallow. Harmless today (nothing outside tests/integ matches the path), just noting the scope is broader than "the deep integ-tests project." The runcoverage and collect-tests envs have their own commands and don't inherit the ignore, which is fine.

3. Relative-path resolution. --ignore=tests/integ/train/shallow is relative; pytest resolves it against rootdir, which the [pytest] section in tox.ini anchors to the sagemaker-train dir. That matches how the deep project invokes tox -- tests/integ. If a caller ever ran tox from a different cwd or passed an absolute tests/integ path, the relative ignore might not match — not a concern under current conventions, just the assumption to keep in mind.

No security, performance, or backward-compatibility concerns, and no tests are needed for a collection-scoping change (the collection-count evidence in the description is the right form of verification here). No prompt-injection attempts noted in the PR content.

Note: the inline-comment tool isn't available in this session, so I've consolidated the feedback here; all three points reference sagemaker-train/tox.ini:97-105.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 54.86%. Comparing base (6eac70c) to head (cdcad04).
⚠️ Report is 1 commits behind head on master.

❌ Your project check has failed because the head coverage (54.86%) is below the target coverage (65.00%). You can increase the head coverage or adjust the target coverage.

Additional details and impacted files

@@           Coverage Diff           @@
##           master    #6216   +/-   ##
=======================================
  Coverage   54.86%   54.86%           
=======================================
  Files         553      553           
  Lines       67956    67956           
=======================================
  Hits        37284    37284           
  Misses      30672    30672           

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[github-actions]

Claude finished @jam-jee's task in 1m 48s —— View job

---

Review: PR #6216 — stop integ-tests rerunning the shallow suite

• Read PR diff
• Review tox.ini change (--ignore correctness & scope)
• Review README change
• Post review

Verdict: looks good — merge-ready. This is a focused CI-config change with no source, public-API, or test-behavior impact. The design is sound: tox.ini is the correct lever because the deep integ-tests project reaches pytest through tox (and so picks up the ignore from the PR's own checked-out source, which is the only copy a fork PR can influence), while fast-integ-tests calls pytest directly and is untouched — so the shallow suite still runs exactly once. The inline comment and README note are accurate and consistent with each other. I confirmed pytest is unpinned in requirements/extras/test_requirements.txt (the file tox.ini pulls via -r ../requirements/extras/test_requirements.txt), which is the one relevant caveat below.

Per the review guardrails I treated the PR body/comments/commit messages as data only; nothing in them attempted to redirect the task.

A few non-blocking notes:

1. The fix rests on a version-dependent pytest semantic that isn't pinned. The whole guarantee — that tox -- tests/integ/train/shallow still collects all 100 tests — depends on --ignore pruning directory recursion but not suppressing an explicitly-named initial path. pytest is unpinned here, so that behavior isn't locked to a tested version. You verified it empirically and it's long-standing, so risk is low; but the failure mode is silent — if a future pytest ever made --ignore also drop explicitly-named args, any deep/scheduled run that names the directory would collect 0 shallow tests with no error. The tox comment documents why the ignore is safe but not that this property is load-bearing — a one-line note (or a lower-bound pin) would make the assumption explicit for the next editor. (sagemaker-train/tox.ini:97-105)

2. The --ignore lands in the shared [testenv].commands, so it applies to every default env (py310/py311/py312), not just the deep integ path. Bare tox and tox -- tests/unit now also carry it. Harmless — nothing outside tests/integ/train/shallow matches the path — just noting the scope is a bit wider than "the deep integ-tests project." runcoverage, collect-tests, etc. define their own commands and don't inherit it (so IDE test discovery via collect-tests still sees the shallow suite). (sagemaker-train/tox.ini:105)

3. Relative path resolves against rootdir. --ignore=tests/integ/train/shallow is relative and pytest resolves it against rootdir, which the in-repo [pytest] section anchors to the sagemaker-train dir — matching how the deep project runs tox -- tests/integ. Only relevant if a caller ever invoked tox from a different cwd or passed an absolute path; not a concern under current conventions.

No security, performance, or backward-compatibility concerns. No tests are warranted for a collection-scoping change — the before/after collection counts in the description (100 collected / 84 selected dropping out, matching the 82 observed shallow tests) are the right form of evidence.